I am often asked to help organizations decide
when to gather data to demonstrate outcomes. Some of the models suggested are pre-and-post (usually a survey at the beginning of a program and at the end), post-only (often with questions built in to assess change over time), and post-then-follow up (to see whether outcomes are sustained). We usually spend time discussing when it is likely participants will experience outcomes, how aware they will be of any outcomes, and whether they will experience the infamous “response-shift bias” – a bias that results when a participating in a program makes them aware of how much more they could improve.
Recently, I hit upon a good analogy I can use in this discussion that would be familiar to many: exercise. I realized that I am a PERFECT example of (1) not being very aware of my own outcomes and (2) having a response-shift bias. I did not set a specific goal when I started a new exercise regimen about 18 months ago – I just wanted to exercise more. So I haven’t been tracking – do I indeed exercise more than I did when I first started? Because of this, a pre-and-post test model would have been appropriate. I could have recorded the amount of exercise I got in a week before making the change, and the amount of exercise I get now.
As to response-shift bias, I thought of myself as somewhat fit when I first started, and I think of myself as somewhat fit now. Do I really have the same fitness level? Probably not! I just realize how much more fit I could be. A good way to address this would be through a post-and-then-test model. For example, I could answer a survey about how fit I feel now (somewhat) and how fit I think I was 18 months ago (not at all). This would be a pretty realistic measure of change. Another way to measure change without my own perceptions changing would be to give me specific fitness questions – like how long I can sustain vigorous activity, or how much weight can I carry, etc.
I’ll use this analogy in the future with clients – I’d love to hear if you have some of your own!
I am often asked to help organizations decide when to gather data to demonstrate outcomes. Some of the models suggested are pre-and-post (usually a survey at the beginning of a program and at the end), post-only (often with questions built in to assess change over time), and post-then-follow up (to see whether outcomes are sustained). We usually spend time discussing when it is likely participants will experience outcomes, how aware they will be of any outcomes, and whether they will experience the infamous “response-shift bias” – a bias that results when a participating in a program makes them aware of how much more they could improve.
Recently, I hit upon a good analogy I can use in this discussion that would be familiar to many: exercise. I realized that I am a PERFECT example of (1) not being very aware of my own outcomes and (2) having a response-shift bias. I did not set a specific goal when I started a new exercise regimen about 18 months ago – I just wanted to exercise more. So I haven’t been tracking – do I indeed exercise more than I did when I first started? Because of this, a pre-and-post test model would have been appropriate. I could have recorded the amount of exercise I got in a week before making the change, and the amount of exercise I get now.
As to response-shift bias, I thought of myself as somewhat fit when I first started, and I think of myself as somewhat fit now. Do I really have the same fitness level? Probably not! I just realize how much more fit I could be. A good way to address this would be through a post-and-then-test model. For example, I could answer a survey about how fit I feel now (somewhat) and how fit I think I was 18 months ago (not at all). This would be a pretty realistic measure of change. Another way to measure change without my own perceptions changing would be to give me specific fitness questions – like how long I can sustain vigorous activity, or how much weight can I carry, etc.
I’ll use this analogy in the future with clients – I’d love to hear if you have some of your own!