Survey design often feels like detective work. You want to extract the truth from a respondent without tricking them into an answer or getting them to divulge details you wish to hear.
It is important to structure your survey and wordsmith your questions carefully. Failing to do so runs the risk of biased responses.
Survey bias can occur when a poorly crafted survey influences respondents’ answers whereby they provide inaccurate or inconsistent information.
There’s no magic formula to design the perfect survey - it’s a mix of best practice, experience, and intuition. To help you avoid the trappings of biased questions, we highlight some of the most common biases that can occur while crafting a survey. For each, we then provide some examples of how a poorly designed question can bias responses, and some tips on how you can fix them!
What you ask a respondent before a particular question often influences how they respond to it.
Let’s take a simple example. Consider asking your respondents the following two questions in this order :
Q1. What is your favourite TV show?
Q2. What is your favourite hobby?
There is a good chance that they might be tempted to answer “watching TV” for Q2 even if it wasn’t an answer on top of their minds. This phenomenon is called priming and it underscores how important it is to take into account the context that directly precedes a question.
Question order bias may creep into a survey in other non-obvious ways too.
Consider this famous example from back in the 1950s when Americans were polled on these questions :
Q1. Should American reporters be allowed to enter Russia and report back what they see?
Q2. Should reporters from communist countries be allowed to enter the US and report back what they see?
When these questions were shown in the above order, support for communist reporters in Q2 increased by 37 percentage points as opposed to when the order was reversed. This is because the respondents being Americans were more likely to answer Q1 favourably, and it’s a common human tendency to maintain consistency and fairness in your answers which explains the higher ratings for the subsequent question (read more : assimilation-contrast theory).
Acquiescence bias, also known as "yea-saying", refers to the tendency of agreeing with statements or saying yes to questions, irrespective of the content and what one believes in. Questions that ask respondents to confirm an opinion are especially susceptible to this bias e.g., yes/no, agree/disagree, true/false.
This can be explained by a few reasons.
Respondents may often believe that saying “yes” to a question will prevent them from being screened-out, and will qualify them for more questions, and hence earn them more points or rewards in exchange. This is particularly true for questions that appear at the start of a survey.
For some, coming across as agreeable is an inherent trait, making them less likely to challenge a statement.
In some instances, it could be a sign of satisficing i.e. putting in low effort in your answers. This is especially the case in lengthy, tedious surveys when respondent fatigue kicks in. Agreeing with something is cognitively less effortful than disagreeing with or challenging a presented notion.
As the name suggests, leading questions sway respondents towards answering questions in a certain way due to the manner in which they are framed. This can harm your data because respondents may lose objectivity and be unknowingly biased towards picking an answer. A good question is neutrally phrased and does not sway respondents to select one response over another.
A double-barreled question is one that clubs two separate issues or topics together but allows for only one answer. This introduces a bias because respondents are allowed to provide only one opinion when in fact they may have a difference stance on both.
An easy way to confuse and tire your respondents would be to ask double negative questions i.e. use two negative words in the same question. This can not only frustrate respondents, but also cause them to misinterpret the question such that they answer in the opposite way that they intended to.
The solution is to phrase the question in a simple and neutral manner. Avoid clubbing negative words such as "do not", “no,” and “not,” in one question. Also be wary of negatively worded sentences in the question when paired with words with prefixes like “dis,” “non,” "un" or “mis” in the response scale.
Social desirability bias refers to the tendency to respond in ways that respondents feel are more appropriate or socially acceptable to others. This can come at the cost of being dishonest.
This can especially be an issue when the content of the question is sensitive or political in nature. They may avoid providing answers that portray them in what they perceive to be a negative light. This leads to answers being inflated to reflect “good behaviour” or under-inflated to hide “poor behaviour”.
A good question should be interpreted the same way by all respondents. Always use simple, clear and precise wordings that everyone can understand. Steer clear from using jargon, technical terms, and acronyms that only a few will understand. It’s also good to ensure that the words or phrases you use are not ambiguous or open to interpretation.
If you must introduce terms or acronyms that may not be easily understood by an average layman or if they may be open to multiple interpretations, it is best to provide them with a definition before or in the question so that everyone is on the same page.
Feel free to check out more survey design best practices under our Learn Section.
If you have questions related to Milieu's survey platform and/or other product/service offerings feel free to reach out to firstname.lastname@example.org.