The communities we serve deserve it.Above all, evaluation matters because we have a duty to the students we serve. Without funding, staff, or program improvements, we shortchange the communities that stand to benefit from them the most. If society is implementing programs designed to fix historic inequities, then we have an obligation to the students and families we serve to find out whether these programs are actually doing that. We need to rigorously evaluate these programs to hold ourselves accountable to improving outcomes for low-income and disadvantaged students and their families.
What are common challenges in evaluating college access and success programs?
The good news is that many of the programs outlined above require or strongly encourage program providers to conduct an external assessment, or evaluation, of the extent to which their programs impact college readiness and participation outcomes for low-income and disadvantaged students. But addressing the many challenges that disadvantaged students face is neither easy nor quick. The changes these programs generate take time to emerge, and real impact will require years to manifest. Here are some common issues evaluators face — and ideas about how to resolve them:
Programs tend to be longitudinal in nature—that is, they occur over a long period of time—which causes many evaluation challenges.
As much as we want to make a difference for students the moment they enter a program, counteracting the effects of historic and entrenched inequities will require more than a year or two of program services. Programs like GEAR UP recognize that, and provide up to seven years of continuous service to students. And while that level of support helps students, it complicates evaluation because those students will have to be tracked over not only time, but across institutional boundaries (from high school to college, for example, and then perhaps to another college or to employment).
In these instances, program staff should think about capitalizing on the data available in national databases such as the National Student Clearinghouse, a leading provider of educational reporting, data exchange, verification, and research services in which most public and private colleges and universities participate, as well as thousands of high schools. The NSC Research Center provides longitudinal data on some important student outcomes like college enrollment, persistence, and completion.
State longitudinal data systems are another source of information on student outcomes that cross institutional boundaries. Over the course of several federal grant cycles, states developed and invested in these systems to address specific education-related policy and research questions by integrating data from early childhood, public pre-Kindergarten through grade 12, public higher education, and workforce data systems.
Programs may serve small populations of students, leading to other types of evaluation challenges.
While working with fewer students may help program staff provide more individual attention and support, studying smaller groups of students makes it more difficult to show statistically significant effects. This issue especially confronts programs operated in rural schools that might have very few students in each grade or programs that serve very small populations of low-income or disadvantaged students, like American Indians.
Programs of any size, furthermore, are expensive to operate, staff, and expand. But even among these smaller programs, it’s possible to improve statistical power without substantially increasing costs. For example, imagine a program that purchases technology for students, or a mentoring program with the capacity to serve only 15 students per school.
One very useful strategy might be to select comparatively more comparison group students or schools. For example, in the mentoring example above, program staff could serve 15 students in each school, but collect comparison group data from 45 students not served by the program. This increases the total sample size to 60 students per school.
Attempting to establish a control condition for these programs raises serious ethical issues.
The most rigorous evaluation studies use randomized controlled trial (RCT) designs that include a random process to assign students to treatment or control conditions. In these studies, treatment students receive the intervention while control students do not. Likewise, quasi-experimental design (QED) studies require the identification of a comparison group of students who do not receive program services. If all or many of the students in a school face serious disadvantages, it would be seriously unethical to withhold the program from half of them so that it could be studied.
For some projects, one option is to delay the treatment approach so that the control group receives the intervention later—but with college access programs, how long can one delay the intervention without seriously reducing the changes of achieving a good outcome for those students?
Another option is to provide a suite of basic program services to all students, but to assign one group (the one being studied) to receive an extra intervention. This design provides the added benefit of offering an additional activity that program staff wish to study only to treatment students, while the control/comparison group still receives core services. Of course, the absence of a true control group means that the program impact for the added activity may appear to be weaker than it is in actuality.
Students Can’t Wait
At the end of the day, the purpose of evaluation in this context is to ensure that the college access and success programs we offer help accelerate achievement and reduce inequities for low-income and disadvantaged students. We owe it to students like Ikie Brooks and thousands of others across the U.S. to study these important programs and find out how we are doing in meeting our goals.
We know that undertaking evaluations of these programs will undoubtedly challenge us in ways we could not have imagined when we set out on this course. That’s why it’s so important to remain flexible and open-minded to innovative solutions if we wish to overcome these challenges. While it can seem overwhelming at times, we must keep in mind that the reward is great — and there’s no time to waste.