At the annual American Evaluation Association conference last year, I went to a compelling panel session that focused on how using both quantitative and qualitative data together is a powerful combination that helps evaluators capture both intended and unintended consequences of a program or project. I think we have all heard, and been convinced, about the increased depth of understanding from using both qualitative and quantitative methods. We often hear how they can tell two sides of the story, with qualitative data explaining the why and how of quantitative data. But, the presenters shared the power of this combination in a new light; qualitative methods may be the only way to open an evaluation to what evaluators and program developers DON’T expect to happen.
The presenters actually used one of the Improve Group projects as an example for the audience. We used our Image Grouping tool to gather information from program participants in Niamey, Niger. This pictorial approach to data collection gathers both quantitative (survey-like) results as well as qualitative insights. Its innovative format helps those who may be put off by traditional data collection interaction contribute their perspectives to the evaluation. A traditional survey could have found that the informal savings groups started by the program were continuing to operate, and could have collected data on current income/assets of participants. But the groups weren’t consistently powerful for all participants; it wasn’t solely the collective savings that were at work. Image Grouping showed us that these groups were a “first stop” for women trying to manage a financial crisis, providing not only cash but in-kind and emotional support. However, men used the groups solely as a place to deposit group savings. Women participants were better able to maintain assets than men because of this difference. This was an unintended impact of the program.
In a different project in the Caribbean, we used Image Grouping to understand the impact of a pre-natal health program for teen mothers. Image Grouping revealed that the program was achieving its intended impacts on the issues it targeted (such as healthy eating); however, participants pointed to other issues that were not impacted much by the program but were more of a problem for them (such as relationship issues with the father).
Unintended consequences can be small or large, negative or positive. It is critical to open your evaluation to them, as they may point out that a program should be significantly changed, set off in a new direction or stopped right away. If you set out only to prove the impact you intend to have, and structure your data collection only around that, you may have your “head down” in a narrow dataset and miss the bigger picture of change that reverberates around your program.