Any profession has its own set of terrors and spooky stories... Just for Halloween the Improve Group brings you an evaluator’s terrifying trifecta!
Zombie focus group
It was a dark, cold October evening. Leaves rustled as the evaluator, Connie Versation, rushed from her car, arms laden with cider, pumpkin bars, and a just-charged voice recorder. Pushing the community center door open, she noticed the lights were flickering and halls were empty. “That’s strange,” she thought, “I thought I was running late and that at least some of my participants would be here.”
The room she had reserved was dark, the only light coming through the window from the streetlight outside. As she flicked the switch, a voice said from right beside her “I don’t think that light is working.”
Startled, the evaluator nearly dropped all her supplies before mustering a smile for the group she now noticed sitting around the table. Apologizing, she thought fast… could cell phones provide enough light? She asked all the participants if they had a cell phone, would they please turn on their flashlights.
Now illuminated by the ghostly glow of LED flashlights, the evaluator welcomed everyone and asked them to please enjoy some food. No one moved, and answered just in short, one-syllable sentences. As she introduced the purpose of the focus group, and started asking her questions, the group didn’t get more verbose. In fact, they all just stared at her blankly, barely answering the questions and just repeating each other when they did. Was this a case of bad group-dynamics luck… or something more sinister? Shaking off her nerves, she reached for her cup of cider for a quick refresh before one last attempt at an ice breaker – and noticed her cider had turned to a deep, thick red. Screaming, she ran from the room, never to conduct a focus group in late October again.
Purgatory
Quan T. Tatuv’s meeting with a new client was going well. The client shared that one of their biggest challenges was clarity about what they were accomplishing and how that aligned with funder expectations. Luckily, Quan had brought some handouts about logic models. He talked about how they can be one method for describing goals, activities, outputs and outcomes that can be clear for staff and funders alike.
The client confided that one of his biggest funders had made several comments that his grant reports seemed more focused on outputs than outcomes. What could they do about it? Quan gave some context about how an output describes what you’ve accomplished, and an outcome describes how participants have changed. Several examples later, the client felt ready to begin drafting a logic model.
“What are some of the goals of your program?” Quan asked. “To reduce hunger and increase food security” the client said immediately. Great! Quan thought to himself. We are on the right track. “And your activities?” The client immediately rattled off five activities of the organization.
Next, Quan asked about output. “That’s easy!” said the client. “We distribute over 4,000 pounds of food each month.” “Awesome!” Quan said, encouragingly. “So, what are your outcomes?”
Suddenly, there was dead silence. After a few moments, the client quietly said “People have food.” Quan immediately grasped the hesitation. People didn’t have food before, and now they have food. That’s a change, so it’s an outcome. BUT, the food itself is an output. Which is it? Output or outcome?
Quan and the client talked. And talked, and talked. After half an hour, they were still spinning. What are the outputs? What are the outcomes? Finally, Quan knew they had to find a way out of this purgatory. Grasping for straws, he asked whether the change represented knowledge, attitudes or behavior. As Quan and the client blinked at each other, they knew they would be trapped in a logic model underworld…. forever.
Missing!
Dee Ata sat at her desk late one night. Lit by the glow of an excel workbook, she prepared to dig into the analysis that had been waiting for her all day. Humming “Thriller” to herself as she worked to get into the holiday spirit, she coded variables, transforming all of the “strongly agrees” to 1, “agrees” to 2, and so on until all her data was ready for analysis.
Dee was assessing the effectiveness of a grant to increase school outcomes. Her first calculation was to count how many respondents there were; and was pleased to see that her final tally was 13,013 surveys completed. Smiling, she started her frequencies. Boys and girls were evenly distributed and matched the school district demographics. So did different ages and ethnicities. Finally, she got to the questions about outcomes. More than 75% of kids had strong peer and teacher relationships in school.
The last question was about safety. A startling 0% of students said they felt safe in school. Puzzled, Dee started to look at the data. Record after record had no response to that question. It seemed like all of her data was… missing! Clicking to open the original survey instrument, she saw that the question had been included. Next, she opened the individual survey responses she had scanned in the day before. Her screen flicked out, just as a howling wind blew outside.
Frowning, Dee opened the box beside her, where she had the tenth grade responses. Pulling the top one out, and turning to the last page, she saw a black sticky ooze smeared over the question about safety. “Was I eating a donut?” she thought. But as she searched through the box, she found the ooze over every one of the safety responses. Eyes growing wide, Dee closed the box, taped it shut, and fled. To this day, no one knows how safe students feel in that school.
We hope these stories do not cause any nightmares for you. Happy Halloween from the Improve Group!