No matter how fantastic your program is, unless it is completely comprehensive (i.e., the Mars isolation experiment), other, external factors will influence how much your participants succeed. Their families might be extremely supportive – or extremely distracting. They may watch telenovelas to supplement a Spanish language class. They may receive an inheritance shortly after completing a down-payment assistance program. Impact evaluation attempts to control for all of those other factors and determine what outcomes can specifically be attributed to a given policy or program. It does so in one of two ways:
  • Randomly assigning people to get the service, and assuming that the comparison population shares many of those same external factors, at least across a large enough group
  • Using statistical or qualitative methods to control for the external factors as you are analyzing a program’s impact
Either of these options can be daunting, particularly if your resources for evaluation are limited. Here is a quick story of how one client gained an understanding of their impact on a limited budget. A local organization wanted to understand how people’s lives, relationships, careers and health were impacted by living in their safe, stable and sober housing development. We talked about the types of information that would be most meaningful and settled on a plan that would gather impacts without undue burden to the organization or their participants. We learned that participants would be very comfortable in an interview format. In 1:1 conversations, participants described how their lives changed over the previous several months. They then shared the degree to which the program helped them in that change, and what other factors contributed to the change. All of the interviews were coded and analyzed, so we ultimately could attribute specific results to the program, such as a strong sense of community supporting sobriety.