Executive Summary
Out-of-school-time (OST) programs and their funders rely on sound data to make decisions about everything from professional development and student recruitment to the selection of activities to offer students. Programs operate at a range of times (before and after school, weekends, summer) and in a variety of locations (e.g., schools, community-based organizations, city parks and recreation centers), are run by a variety of entities (e.g., government agencies, private community organizations), and receive funding from a variety of sources (e.g., government, philanthropy)—each of which may be interested in a different set of data and come with its own reporting requirements. This means there is a great deal of variation in the types of data programs collect.
In 2019, The Wallace Foundation (Wallace) commissioned Child Trends to conduct a study of the kinds of youth outcomes OST programs are interested in measuring, the tools they use to measure those outcomes, and the challenges they experience in doing so. The study included a literature scan and interviews with leaders and staff members at 28 OST programs. Twelve of the 28 also completed surveys; a separate group of 10 provided information by survey only. The study expands on past research by a) focusing on programs that work in specific content areas (e.g., the arts, civic engagement and social justice, career and workforce development) and b) covering both quantitative approaches (i.e., tracking numerical data) and qualitative approaches (gathering descriptive information through surveys, interviews, etc.) to data collection.
What outcomes do programs measure?
- Programs in the study measured outcomes that were closely related to the content they delivered (e.g., the arts, career and workforce readiness, civic engagement and social justice). They also generally measured SEL outcomes and other outcomes required by funders.
- Programs thought critically about how to measure progress toward achieving equitable outcomes for the youth they serve. Approaches included disaggregating data by race and other demographic variables and tracking the development of equity-related knowledge and skills.
- Programs used a number of criteria to decide what outcomes to track, including whether a given outcome was consistent with the program’s logic model or theory of change, usability and shareability of the data, the effort and capacity required to collect the data, availability of valid and reliable measures, youth interest, and whether measuring a given outcome would promote equity.
- In addition to outcomes, programs consistently measured outputs (i.e., steps that lead to desired outcomes), particularly program participation and quality. This finding aligns with our literature scan that revealed that public and private funders have invested heavily in quality assessment tools and participation tracking systems. Comparatively few programs used qualitative methods to understand the factors that contribute to program participation and quality.
What methods do programs use to measure progress?
- Programs reported using a variety of traditional methods to measure outcomes, including quantitative methods—like administering surveys and questionnaires, conducting formal assessments, and tracking attendance rates—and qualitative methods such as conducting interviews and focus groups with young people. Their use of quantitative methods was well-documented, but their use of qualitative methods less so.
- Programs also used nontraditional methods, such as regular check-ins with participants and reviewing participant journals and portfolios, along with creative, informal ways to track SEL outcomes, including games and award systems. As with their use of qualitative measures, use of these nontraditional methods was not well-documented.