Several recent articles, listserv discussions, and American Evaluation Association conference sessions have explored where evaluation fits in complex systems. In fact, the theme of the 2012 evaluation conference was complex ecologies. Complex systems involve multiple aspects that interact in nonlinear, dynamic, emerging ways. In a 2007 Harvard Business Review article, David Snowden and Mary Boone state:
"Though a complex system may, in retrospect, appear to be ordered and predictable, hindsight does not lead to foresight because the external conditions and systems constantly change."
With this caveat, how can what you learn from evaluation be used to improve and refine systems? Learning needs to focus on experimentation and testing ideas, articulating what happened, and helping to uncover patterns that can help define the system. Approaches to evaluation can be:
- Experimental: testing one new idea while holding other factors as constant as possible (often using random assignment or other quasi-experimental methods) in order to determine its impact
- Empowering: giving more stakeholders in the system an opportunity to decide what to learn, how, and what to do with the information once it’s available
- Developmental: functioning like R&D in the for-profit sector, with questions embedded in each new innovation and shared responsibility for learning and reflection
One complex system that has recently reviewed and modified its approach to evaluation is USAID. Supported by a learning portal, USAID recognizes that projects will need to adjust as new issues emerge, and that ongoing learning is critical. While multiple approaches to learning can be used, they include the experience of experts and local community members, ongoing monitoring of data, and more in-depth analysis where new issues are emerging.USAID recommends that evaluation be planned during project design, but recognizes that while questions may be defined early, once they are answered, new questions will arise.
USAID has crafted a number of resources to help its evaluation practitioners that could be useful in other complex systems, including these:
- The Learning Lab has a number of training modules, including Module 17, focused on evaluating complex programs
- An online presentation, focused on what complex systems are, is available.
Because the Improve Group works in so many different fields, we find that complexity can provide a unifying framework for how we approach each project. We track resources, such as those above, on our Pinterest Board.