How to Reduce Selection Bias

Co-authored by Michael Simpson, PhD and Jennifer Mueller, JD

Last Updated: December 25, 2022 References

When you conduct a scientific study, you're typically trying to determine the effect one thing has on something else. Since you can't study a whole population, you instead take a sample of that population. You then divide that sample into the number of groups called for by your research design. The only difference between those groups should be the thing you're trying to measure. Selection bias occurs if there are other differences between the groups that could affect your results. When that happens, you can't apply the results of your study to the larger population. The main way researchers reduce selection bias is by conducting randomized controlled studies. However, randomized controlled studies can be cost-prohibitive and, in some types of studies, such as social science studies, they aren't feasible. If you can't do a randomized controlled study to prevent potential selection bias, you can still adjust your results to take selection bias into account.^{[1]
X
Research source}

Method 1

Method 1 of 3:

Conducting Randomized Controlled Studies

1
Enroll study participants who reflect your target population. Your target population is the one to which you'll apply the results of your study. Draw all of your study participants from that single population. Even in a randomized controlled study, selection bias may occur if your study participants don't accurately reflect your target population.^{[2]
X
Research source}
- For example, suppose your target population is college students. However, you advertised for volunteers off-campus and attracted some locals as well. Locals who don't attend the college may not have the same characteristics as your target population and including them could lead to selection bias.
- The number of participants in your study must also be of adequate size for you to be able to apply the results of your study to the population at large. The necessary sample size will vary depending on a variety of factors, such as the magnitude of the effect you're studying and its variability within the population.
- You might also get some help from an online calculator that helps you determine your sample size, such as the one available at https://clincalc.com/stats/samplesize.aspx.
2
Select study participants at random who meet your criteria. While recruiting volunteers can be less expensive, you also run the risk of volunteer bias. This happens when people willing to volunteer in your study have a personal interest in the outcome. The reason for their interest could mean they don't completely represent your target population.^{[3]
X
Research source}
- Create a questionnaire with inclusion and exclusion criteria. For example, if you're studying the effect of sleep on college student grades, you might want to ensure you have a balance of students with a lot of early morning courses and night students. In that case, you would include a question about the volunteer's class schedule. If you only wanted to include full-time students, you would ask how many course hours the volunteer was taking.
- Once you have approximately 2-3 times the number of potential participants you need for the study, assign them each a random number. Then choose your study participants randomly based on those numbers. This helps reduce selection bias as well as volunteer bias.
Tip: If your randomization is strong throughout your study, you can effectively eliminate any selection bias that may have existed.

Advertisement
3
Do a pilot study to identify potential problems. In a pilot study, you practice your participant recruitment techniques and do a basic run-through of at least the first part of the study. Any flaws in your study design or in your selection criteria for study participants will become apparent. This gives you the opportunity to correct any flaws before you do the full study.^{[4]
X
Research source}
- Since it's not the real thing, your sample size doesn't have to be as large as it would be for the full study, which helps reduce costs.
- Pilot studies also give you an idea of how quickly you'll be able to recruit participants for your study and which methods of recruitment seem to work the best.
4
Create an operations manual to standardize all study procedures. Selection bias can slip through the cracks of your carefully designed study if others involved in the study are using different methods to recruit participants or measure data. If all study procedures are standard, you can also be certain that another researcher could reproduce your study results.^{[5]
X
Research source}
- For example, if your investigators were asking the participants a series of questions, your operations manual would include the exact questions asked. Then, you could coach your investigators on their tone of voice and other factors that might skew the participants' responses.
- If you have multiple people involved in the study, train them on the methods you want them to use during the study and test them to make sure they're all doing everything the same.
- If your study is going to take place over the course of months or years, it might be necessary to have "refresher" courses to keep investigators up to speed on your protocol, especially if they are away from the study for a while.
5
Assign participants to intervention or placebo groups randomly. If you're doing randomization on your own, use random numbers to identify your study participants. The person who assigns the random numbers should be someone who is not working on the study as an investigator. Once random numbers are assigned, you can randomly split the participants between the two groups.^{[6]
X
Research source}
- Most universities have research support units to assist with randomization. There are also computer programs that will do randomization for you. If you don't have access to research support, use a free random number generator, such as the one at https://www.random.org/.
- Larger studies typically use a remote randomization facility to ensure that there's no way anyone involved with the study could know which group any given participant was in.
6
Keep each participant's group assignment double-blind. In a double-blind study, neither the participant nor the investigator knows which group the participant is in. However, sometimes this process is not possible or would be cost-prohibitive.^{[7]
X
Research source}
- For example, if your study included surgery, it would be impossible for your participants not to know if surgery was being performed on them. In that case, your investigators could be blind as to a particular subject's group while taking their measurements and compiling data, but the participant could not because they would have to consent to the surgical procedure.
- Even if you have double-blinding in place, it might break down. For example, if you're studying a drug that turns out to have dangerous side effects, you might need to know which participants were taking the drug so you could monitor them or warn them of the side effects.

Method 2

Method 2 of 3:

Minimizing Selection Bias in Case-Control Studies

1
Collect basic demographic information from potential participants. In a case-control study, you have people who have contracted the disease or condition (your cases) and people who didn't (your controls), despite being exposed to the same thing. Choosing participants from both groups who have similar backgrounds and biographic data helps eliminate other factors that could potentially bias your result.^{[8]
X
Research source}
- For example, if you're studying a population's likelihood of contracting a disease after exposure to the virus that causes it, you would want a sample that was similar in age, socio-economic status, and access to healthcare. Maintaining these similarities reduces the possibility that some participants' outcome was affected by their health or medical treatment.
2
Select controls using the same process as your cases. In a case-control study, identify your cases first. Then, follow the same or similar process to enroll controls in your study. This ensures that you have an accurate measure of exposure in the population you want to study.^{[9]
X
Research source}
- For example, if your case population comes from patients referred to a particular hospital for treatment, you might seek out your controls from the healthcare providers who made those referrals.
3
Avoid choosing controls from hospital populations. It's okay if your cases are hospitalized. However, if your controls are also hospitalized, the resulting association between the exposure and the disease will be weakened.^{[10]
X
Research source}
- For example, if you're doing a study on smoking and chronic heart disease, having hospitalized controls would weaken the association because smoking is a factor that leads to many health problems that could also result in hospitalization.
4
Match controls with cases based on similar demographics. Include any factors that might affect the results of your study as criteria when you choose controls for your case-control study. Use the demographic information you got from your cases as a profile for your controls.^{[11]
X
Research source}
- For example, suppose a local restaurant is responsible for a viral outbreak, but you don't know which one. The local population who contracted the virus are your cases. To identify which restaurant is responsible, you could enroll people from the local area who matched your cases in terms of neighborhood, age, and gender, but didn't contract the virus, as your controls.
5
Use population data instead of recruiting participants as controls. In a case-control study, people who didn't come down with the disease or condition you're studying will generally be less likely to participate in your study. However, if you have population information available from a national, regional, or local database, using that information as your control solves this problem. Additionally, using data from a publicly accessible database decreases the cost of your study.^{[12]
X
Research source}
- Choose a population dataset for your control that matches the population of the cases you're studying. For example, if all of your cases are located in the state of California, you might use a state database to get your population data. However, you wouldn't want to use a national database.
Tip: Keeping your geographic area as small as possible also enables you to have a smaller sample size, which increases the accuracy of your study as well as decreasing the cost.

Method 3

Method 3 of 3:

Adjusting Results to Account for Bias

1
Include the variable associated with the selection bias in your analysis. Look for variables that could potentially cause selection bias and record that information from each of your participants. Then, analyze your results based specifically on that variable in addition to your overall analysis.^{[13]
X
Research source}
- For example, suppose you are studying the connection between coffee and migraines. You sent out postal surveys to households in the state of California. However, you're aware of previous studies that have shown older people are typically more interested in participating in postal surveys than younger people, so this could bias your study by age.
- To adjust for bias in the study of the connection between coffee and migraines, you could separate your data so that it measured the connection in different age groups separately (stratification). This would reduce the selection bias that would occur by having too many older people in your sample.
2
Weight participant responses to correct a biased sample. If your participants don't appropriately match the demographics of your target population, allow the results from the underrepresented group to be more valuable than the results from the other group. This adjusts your sample so that you can apply your results to the entire population.^{[14]
X
Research source}
- For example, suppose you were studying the effect of sleep on grades among college students. The student population at the school you're studying is 40% male and 60% female. However, your sample is only 20% male. To weight the male responses, divide the population percentage by your sample percentage (40% divided by 20%). The result is 2, so each male's response counts double.
Warning: If your sample is too distinct from the population you're attempting to study, your results may not be accurate for the population as a whole, even with weighting, because you have too few representatives in the sample.
3
Discuss the potential for selection bias in your report. If there's no effective way to adjust your results to sufficiently reduce selection bias, simply acknowledge that selection bias exists. Mention any ways that you tried to correct for the bias or describe why correcting for the bias wasn't possible given the circumstances of the study.^{[15]
X
Research source}
- For example, suppose you want to evaluate the association between working the night shift and having a particular health problem by comparing people who work at the same factory doing the same work, with the only difference being that some work during the day and some work at night. However, there are likely to be many other differences between these groups that you can't possibly account for, such as their socio-economic status or access to health care.
- In the report of your study, acknowledge that there are a lot of other differences that your study didn't take into account. You might also mention what some of those differences could be and include references to other studies that have analyzed those variables in depth.
- To conduct effective scientific research, you should have a clear idea of your objectives before starting the project.
- Knowing what specific question you are asking will make it easier to target your research.
- Include experiments with the hypothesis to provide an answer.