Clients often ask us how they can be sure they are getting the "true" answer to their research questions. This question comes up when developing a survey, reviewing participant files, or even recruiting for a focus group. The answer comes down to both sampling strategy and sample size. You want to structure both the strategy and the size so that (1) you are including the people most likely to have the information you need; (2) you are respecting the variety of ideas and information that exists within a group; (3) you are balancing costs with need for valid information; and (4) you are realistic about what you can learn from the group.
A sample strategy is the method you use to select people or documents who will provide you with information. The sampling strategy can vary greatly depending on the situation. Some situations call for a random or stratified random sampling strategy. In these cases, a random formula is applied to all potential respondents, or subsets of respondents. We use this strategy in our work with the Department of Human Services, in which we obtain a random sample of participants in each of six programs.
You can also use a targeted sampling strategy, in which you specifically recruit people you think best suited to provide you with the information you need. For example, in our work for the United Jewish Fund and Council, we are trying to reach two specific groups (families with young children and people who are not very involved in the Jewish community) to explore new outreach ideas and need for childcare. Because this group will be self-identified, we are asking people to recommend friends and relatives to participate and then asking them to identify others as well (we are calling this a "drill-down" sampling strategy). We will supplement this with some random selection from a phone list.
Once your strategy is identified for your project, you should begin to think about the size of the sample. For this you want to take into account the confidence level you hope to achieve and the type of information you are collecting. The confidence level tells you how sure you can be that the responses are the "true" responses, within a percentage basis. So, if your confidence level is 95% (the most commonly used in social science research), you can be 95% sure that your responses are accurate. You can only use confidence levels to help determine sample sizes in randomly selected groups; it doesn't make sense to be confident about your results if you specifically selected respondents for targeted characteristics.
When working with a targeted group, you should use different methods for gaining confidence about your results. One way is to directly ask respondents how representative they feel they are of the general population as a whole. If you ask this question repeatedly to several different respondents, you can begin to assess how well your respondents match the population at large.
Some resources on sampling size, strategies and calculators can be found on the Internet:
University of Florida Extension
Vanderbilt University
Harvard University
