Driving EdTech Systems: Build Research and Measurement Capacity

Overview

From designing and running pilot programs to evaluating tools, data is an essential indicator and a critical driver of equitable edtech systems, like those described in the EdTech Systems Guide created by The Learning Accelerator (TLA) in partnership with the Massachusetts Department of Elementary and Secondary Education’s Office of Educational Technology. Rooting edtech leadership decisions in data limits the impact of bias and cuts through the noise to ensure efforts focus on the right levers to address real challenges rather than red herrings. Edtech leaders must develop and strengthen crucial data collection and analysis skills that allow them to approach critical decisions from the standpoint of researchers, and for many, this can feel overwhelming. To make research and measurement approachable for users of the EdTech Systems Guide, this resource distills TLA’s Research and Measurement Guide into an overview of foundational evaluation practices that should guide edtech systems.

Before You Begin: Understanding what we mean by equitable edtech systems.

We believe that unlocking the potential of edtech systems to promote equality requires intentional selection, implementation, and evaluation processes that:

Center equity by systematically seeking out and elevating the perspectives and needs of groups historically positioned furthest from opportunity,
Support school and system priorities by rooting decisions in a student-centered vision for teaching and learning; and,
Strive to improve continuously, recognizing that technology and the needs of the stakeholders who use it are constantly changing.

Practices

Develop Research Questions

Research questions sit at the heart of any measurement project. Crafting clear and purposeful research questions is important because these questions guide your data collection and shape the insights you glean from analysis. In the context of your edtech systems, well-formulated research questions are:

Clear and Focused: Research questions should directly address the context of your research, including what you are trying to learn or measure, where and when the research will occur, and which stakeholders will be involved.
Open-Ended: Research questions should not be framed as “yes or no” but rather should ask broader questions that lead to a deeper exploration of relationships between inputs, practices, and outcomes.
Measurable: Respondents must be able to answer the research questions with existing data or by collecting additional information.

Examples of effective research questions related to developing and strengthening edtech systems include:

How do specific middle school ELA Tier-II curriculum tools impact the learning outcomes of students who are learning English?
What is the relationship between the usage of implementation supports provided to high school math teachers and the adoption and implementation of specific instructional support tools?
To what extent does our system-wide edtech evaluation process engage and elevate the needs of stakeholders from groups historically positioned furthest from opportunity?

Collect Data

Research questions may drive measurement projects, but answering these questions relies on data collection. You do not have to be trained in research methodology to collect data successfully. However, you do have to collect the right data to answer your question(s) accurately. Whether you use a data story strategy or develop a data collection plan like the one provided in the Driving EdTech Systems Continuous Improvement Workbook, you should be able to answer the following questions before you begin:

What types of data will you collect? To answer your research question(s), do you need qualitative data like stories or stakeholder perspectives collected from interviews, focus groups, surveys, observations of practice, or student work samples? Do you need quantitative data like student test scores or edtech tool usage statistics? Some combination of both? What data already exists, and where can you access the data? What data do you need to collect that has yet to be captured elsewhere?
Who will collect the data? Which of your stakeholders is best positioned to collect the information you need? For example, teachers might be able to collect information from their students, your edtech leadership team may be able to collect information from their school leaders or members of smaller communities within your school or system, or you may be best positioned to conduct classroom observations to collect information about the implementation of edtech tools.
How often will you collect this data? Does your research question require a one-time snapshot of your edtech system’s reality or do you need to collect data to identify changes over time? When is the best time to collect the necessary information (e.g., before and after you intervene, once each quarter, once at the end of the school year)?
Where will you collect and organize your data? What tools will you use to keep track of your data? What is the best way to organize your data to facilitate your analysis? Who needs access to this data, and how will you provide it to them while preventing others from accessing it?

Addressing these questions before you begin data collection ensures that you anticipate and mitigate potential challenges, and enables a smoother collection process.

Analyze Your Data

Analyzing data is crucial in turning raw data into meaningful insights that inform your edtech decisions. Effective analysis helps you understand the patterns, relationships, and impacts of your intervention(s). When analyzing your data, consider the following to ensure your data analysis is thorough, rigorous, and provides valuable insights:

For Quantitative Data

Data Cleaning: Ensure your data is accurate and complete by removing duplicates, correcting errors, and handling missing values.
Descriptive Statistics and Counts: For larger data sets, use measures like mean, median, mode, and standard deviation to summarize your data and identify trends; for smaller data sets, opt for counts or percentages.

For more information about quantitative research methods, see this resource from Dissertation by Design.

For Qualitative Data

Coding: Assign labels to segments of your qualitative data (e.g., interview transcripts, open-ended survey responses) to categorize and identify themes.
Thematic Analysis: Identify and analyze patterns and themes that emerge from your data. This can be done inductively (letting themes emerge naturally) or deductively (using predefined themes).
Triangulation: Use multiple data sources or methods to confirm findings and ensure the reliability and validity of your analysis.

For more information about qualitative research methods, see this resource from Qualtrics.

For Both Quantitative and Qualitative Data

Integrating Data: Combine quantitative and qualitative data to view your research questions comprehensively. This can involve comparing and contrasting data from different sources or using qualitative data to explain quantitative findings.
Cross-Validating: Use one data type to validate or support findings from another type. For example, qualitative insights can be used to explain trends observed in quantitative data.

For more information on mixed-methods research, see this resource from Harvard Catalyst.

Regardless of the kinds of data that you analyze, you can ensure rigor by documenting your analysis procedures, having colleagues review your analysis and provide feedback, and reflecting on how your biases, perspectives, or assumptions might have influenced your analysis.

Communicate Your Findings

While your data analysis should directly inform your edtech leadership decisions, you should also consider how to share your findings with relevant stakeholders. To ensure that you communicate your findings clearly and coherently, consider what your audience already knows and what you want them to learn. Whether providing findings through a written report, presentation, infographic, or video, you should aim to succinctly and accurately communicate your findings, and be sure to do the following:

Represent Your Data Visually: Raw data can be overwhelming and hard to understand. To help your stakeholders comprehend your findings or see the trends you identified through your analysis, you might build meaningful data visualizations that translate key findings into easy-to-read graphics, tables, and charts.
Identify and Present Key Information: Your presentation of your findings should directly answer your research questions, identify key trends, provide the context of your research, and make recommendations or summarize how you acted on this learning.
Share Findings in Multiple Formats: Depending on your audience and their preferences, you can share your findings using research reports, data sheets, strategy memos, or other formats. It is important that the format is easy to follow and clearly and concisely represents your findings.

Research and measurement can feel daunting, but collecting and analyzing data to inform edtech leadership decisions is essential. By strengthening your abilities to develop clear research questions, collect accurate data, conduct thorough analyses, and effectively communicate findings, you can drive meaningful improvements in your edtech systems and make stronger decisions related to edtech selection, implementation, and evaluation. Return to this resource often to refine your strategies and ensure that your efforts continue to focus on addressing real challenges and promote equity in education.

This strategy is a part of TLA's Driving EdTech Systems series, which accompanies the EdTech Systems Guide developed in partnership with MA DESE OET. Explore the full guide to find additional strategies, insights, and resources.

Strategy Resources

Mount Greylock Regional School District’s EdTech Rubric and Stakeholder Survey Planning

Mount Greylock Regional School District created this edtech evaluation rubric and aligned survey questions to... Learn More