How to Write Survey Questions
11 Tips to write survey questions and avoid reponse bias
In this article
- Aims & Reponse Bias
- See What’s Been Done Before
- Choose Your Question Type
- Number and Length
- Consider the Phrasing
- Ensure Variety in Options
- Keep It Focussed
- Avoid Repetition
- Keep Things Flexible
- Types of Survey Questions
- Examples of bad survey questions?
Writing good survey questions is an important part of survey research to help make sure that you avoid response bias. How you construct your survey questions will depend on what your research objectives are, and how much detail you would like to obtain. We will explore 11 tips for writing survey questions which are also highly applicable when making online forms.
1. Identify Your Aims and Avoid Reponse Bias:
Before you can write your survey questions, it’s helpful to have a clear idea about what your survey’s purpose is and who you intend to give your survey to. The aim of your survey, and your population of interest, will impact the questions that you ask and therefore influence the results that you obtain and help you avoid response bias. You as a researcher need to make sure that the questions you ask are relevant to the aim of the survey and who will be answering it.
A useful approach at this stage is to write a draft list of everything about your survey – what the aim is, who will complete it, and what you want to cover. Having a draft will let you see the overall picture and make sure that there are no ‘plot holes’. Drafts also help ensure that everything you’ve written down is necessary, and that you’re not missing anything.
2 – See What’s Been Done Before:
Before you start writing your own survey questions, it’s a good idea (and potentially helpful) to do a quick search of the literature to check what questions other researchers in your area have use already. There’s a good chance that published questions have been validated, or tested in a variety of different situations to ensure that the information they provide is what we want to obtain. You may consider using already validated questions, or you may use those questions as models for writing your own – it ultimately depends on your research objectives, and whether relevant existing questions are available.
3 – Choose Your Question Type:
The two different types of questions used in survey research are open-ended and closed questions. An open-ended question is an open-response style of question, where participants can answer in any volume of text based on their own experience, opinions, or level of understanding. In contrast, a closed question is where possible answers are limited to a selected number of pre-determined options, and often occur in dichotomous or multiple-choice formats, such as ‘Yes/No’ or ‘Please select…’ questions.
Closed questions are generally easier to administer in large-scale survey research because participants only need to consider and choose from options provided to them. However, closed questions restrict answers and don’t allow for elaboration. Open-ended questions allow for greater insight into the reasons behind a particular answer, but these questions tend to be more demanding of participants because answers need to be more articulated.
Using an open-ended or closed question depends on the information that you want to obtain from your participants. It is becoming more common in research to rely on closed questions to obtain a snapshot of the information needed, and then place a select few open-ended questions at the end of the survey to give participants an opportunity to elaborate on anything they answered earlier. It’s important though that you don’t shy away from either question type if that’s the method that is needed.
4 – Consider Number and Length:
The number of questions you ask in your survey, and therefore its length, are an important consideration. You will need to consider how many questions you need to ask to achieve your research aim, making sure that the number of questions included is reasonable. Participants who complete surveys often do so without compensation – so shorter surveys are often preferred so they don’t create too much of a burden on those who complete it. Generally, shorter surveys have higher completion rates and lower attrition rates. However, it’s important that you don’t compromise the length of the survey if it is necessary that a specific amount of information needs to be obtained.
5 – Consider the Order:
The order that you present your questions in will have important implications for how the survey is completed. It’s best practice to ease participants into your survey by starting with some basic questions, such as demographic information, before moving on to more specific questions which can be a bit more personal or detailed in nature. Outside the basic starting questions, it is important to consider the order that the information needs to be obtained in. Attrition, or withdrawing, occurs regularly in surveys and so if there is one set of questions that are of a higher priority to complete, consider putting those closer to the beginning rather than at the end of the survey.
6 – Consider the Phrasing:
With survey questions, it’s critical that you choose your phrasing carefully because this will indicate what it is you want to know. Several factors need to be considered, and include:
- Clarity: questions should be phrased clearly and directly. Avoid ambiguity where possible to ensure that your answers are relevant to what you want to know. Technical language should also be avoided to keep things clear. If you need to use a technical term, it’s helpful to include a brief explanation or definition.
- Neutrality: To avoid potentially influencing the results, questions should be objective in nature and in tone, and should not use leading phrasing. In addition, to minimise any bias, rather than giving only one option to consider, provide a range of alternatives as well so you don’t encourage participants towards one answer.
- Avoid Absolutes: Absolutes in questions include words such as ‘never’ or ‘always’. Responses however are generally not as black and white as this – there is every possibility that an answer may be close to one of these categories but not exact. Absolutes should be avoided where possible to avoid making participants feel as if they must agree or disagree with what has been said.
7 – Ensure Variety in Your Response Options:
In the case of closed questions, particularly ones that are reliant on a rating scale, there is always a possibility of researcher bias occurring because you as the researcher decided what the response options were. It is always preferable to have a balanced variety of options available for participants to choose from to reduce this possibility of bias and let participants choose answers that are representative of them. Generally, having at least 5 options to choose from provides enough scope. The first and fifth answers should be the upper and lower limits, and the third should be the neutral middle ground. This can be expanded if more than 5 options are chosen.
8 – Keep It Focussed:
Each question that you include in your survey should be done with the intention of obtaining a specific piece of information. Each question should clearly ask one thing at a time. It is always recommended to avoid asking two questions at once, referred to as double-barrelled questions, because it may make participants confused about which part of the question to address. To avoid this, you can separate double-barrelled questions into individual ones. Be mindful that doing this repeatedly can add length to your survey, but in doing this you minimise confusion and keep things clear.
9 – Avoid Repetition:
Being asked similar questions repeatedly in a survey can lead to participants getting confused or potentially frustrated. It’s important that your questions are clear and ask about one concept once. If there are similarities between questions, it is helpful to make the differences between questions obvious. One useful strategy is to use italics to highlight unique points of emphasis or differentiation from previous questions.
10 – Keep Things Flexible:
While it is always preferable to have a complete dataset, sometimes participants may not know the answers to the questions, or may not feel comfortable answering. To address this, for closed questions it’s a good idea to provide an option such as ‘Not Sure’ to give participants an out if they don’t want to answer. To help participants feel more comfortable, responses can be anonymous or de-identifiable unless necessary (for example, following up at a later timepoint if they consent to). Online survey platforms now have features which will ‘force responses’, or not allow participants to move on from the question unless an answer has been given. This feature should be used sparingly to make sure that your participants feel comfortable and avoid providing a random answer to move on.
11 – Pilot, Pilot, Pilot:
Once you’ve written your survey, it’s useful to do a test run on a small group of people to see if there are any sticking points with any of the questions in your survey. Piloting will give you an indication of how the questions are received and allow you to test how the data comes out – which in turn will give you the chance to refine anything before distributing your survey more widely. Piloting is also a good way to spot any mistakes or typos, which have been known to happen – having fresh sets of eyes on your questions help!
Types of Survey Questions
Survey questions can broadly be categorized as Open Ended and Closed Question types.
An open-ended question is an open response-style of question, where participants can answer in as much or as little text as they choose.
When to use:
Use when you want your participants to be able to respond with as much or as little detail as possible. Open ended questions provide primarily qualitative data and are used in exploratory research.
A closed question is one where the possible answers are restricted to a select number of choices.
When to use:
Use when you need a degree of consistent measurement in your results. Closed-ended questions provide primarily quantitative data, and are frequently used in confirmatory research
A Likert scale is usually a 5, 7 or 9 option scale which is used to measure how much particpants agree with a statement.
The scale may move from strongly disagree to strongly agree.
When to use:
Use a Likert-scale question when you are trying to determine respondents’ attitudes or feelings about something
What are examples of bad survey questions?
It's surprisingly easy to ask bad survey questions. The following example from pew research highlights how great an impact question design can have on results.
When people were asked whether they would favor or oppose taking military action in Iraq to end Saddam Hussein's rule, 68% said they favored military action while 25% said they opposed military action. However, when asked whether they would favor or oppose taking military action in Iraq to end Saddam Hussein's rule even if it meant that U.S. forces might suffer thousands of casualties, responses were dramatically different; only 43% said they favored military action, while 48% said they opposed it.
Bad survey questions do not measure what it is you're intending to learn. They're at best unreliable measures and often completely invalid measures. The follow is common in bad survey questions
- Include loaded assumptions "Please rate how much you enjoyed..."
- Use vague language and are not to the point "What do you think of the president?"
- Don't include all possible answers
- Do not precisely assist in understanding the research topic
- Asks redundant and unnecessary questions
If you're considering a career in qualitative research take a look at questions about questions
- Australian Bureau of Statistics (2021). Questionnaire Design.
- Harvard University. Questionnaire Design Tip Sheet
- FAQ. Questionnaire Design
- How to make a survey
- Survey Questions | Ultimate Guide
- How to increase survey response rates
- What are demographics?
- Correlation coefficients
- What is sampling?
- What is sample size?
- Open Ended vs Close Questions
- How to conduct research using surveys
- Quantitative Data
- Statistical Significance
- Multiple Choice Questions