Hi! I’m Christen Pentek and in this month’s IG-ology I want to talk about how evaluators can ensure validity in their work. For me validation brings to mind conversation about details that matter. And it evokes the African proverb: “If you want to go fast, go alone. If you want to go far, go together.”
There are two concepts in evaluation that help to clue people into whether a study is of quality or not. One of these pieces is reliability, the other is validity. They work together to create believable, credible studies.
Reliability means that if the study was repeated, you could expect roughly the same results. So, for example, you might do a survey with a random sample of people, and then repeat the survey with another random sample of people, and the second group would respond in a similar way to the first group. If the study was set up to be reliable through its survey design, administration, and sampling methods, you could expect similar results.
Validity means that the methods you use result in data that actually mean what you (or stakeholders) think it does. For example, if you ask a question about joy, your respondents understand what your question means, answer it in the way you intended, and data users find meaning in the way you’ve articulated your measures of joy.
There are several practices that we have as evaluators to help increase and ensure the validity of our work:
- Working in an evaluation team. People have different perspectives and live different lives. This can mean that our assumptions about the way the world works may be different. Having multiple people involved, especially in analysis, can help expand the understanding and mitigate assumptions made about the data.
- Validation checks. A validation check is when two or more evaluation team members do analysis together and are in conversation about what they are interpreting about the data. Validation checks ensure that evaluation team members stay in communication about the larger goals of the evaluation project, such as the guiding evaluation questions.
- Emerging findings meetings with stakeholders. As external evaluators, we understand that we may not have a full picture of the field and the context of the project. Our Community-Responsive Approach prioritizes bringing findings back to the stakeholders for their input, as our clients often provide community expertise and are responsible for bringing the voice of the community.
- Empathy. Increasing use of empathy includes emotions as a valid part of the data and evaluation process. Bringing empathy to the evaluation process increases validity because it allows people to clarify the response from an experiential frame, utilizing a frame for quality that is broader than logical reasoning.