White Papers

Attrition Patterns in Online Surveys

Analysis and Guidance for Industry

Michael Braun Hamilton, Online Survey Analyst
support@supersurvey.com

Ipathia, Inc. / SuperSurvey
http://www.supersurvey.com
800-241-0279

Executive Summary

This white paper analyzes meta-data for 281 surveys conducted using the SuperSurvey^® cluster at http://www.supersurvey.com. These surveys were all premium (paid) surveys of at least 2 pages that had received at least 10 responses and did not employ skip logic. The meta-data were analyzed to determine factors influencing survey completion rates and to elucidate attrition patterns over the course of surveys.

Conclusions:

The main factors correlating with survey attrition rates are those measuring survey length. The fewer pages or questions a survey has, the higher the completion rates are.
For shorter surveys (fewer than 7 pages), use of the SuperSurvey^® "Percent Done" Indicator correlates with significantly higher completion rates.
Surveys that used a unique-key based tracking system (Auto-Generated Keys or User Specified keys) show higher completion rates than surveys with no access control or just a Single Key.
The vast majority of break-offs occur at the very beginning of the survey. The highest number of break-offs occur after only completing the first page.

Recommendations:

Be concise. The fewer pages and questions there are in your survey, the more people will make it through.
Use the SuperSurvey^® percent done indicator to increase completions.
Use SuperSurvey's^® Auto-Generated or User-Specified unique key features to reduce accidental survey break-offs.
Place critical questions at the beginning of the survey to minimize the impact of break-offs.
Contact SuperSurvey^® for guidance on conducting online surveys for high-value business, government, and research applications.

Introduction

SuperSurvey^® (http://www.supersurvey.com) has conducted online surveys for clients across industry and local and federal government, primarily in the US and Canada. In this white paper we examine meta-data on a subset of those surveys to determine factors influencing survey completion and to investigate survey attrition patterns. The analysis in this white paper provides industry with guidance on what to expect with online surveys and provides a baseline for comparison with the completion patterns that they see on those surveys, along with recommendations for best practices.

Method

This paper analyzes a subset of the response meta-data compiled by the SuperSurvey^® system. The data was limited to premium (paid) surveys with at least 2 pages and at least 10 complete responses. To simplify analysis, surveys employing skip logic were excluded. Information from the 281 surveys remaining in our sample was then analyzed to determine basic attrition rates and to identify correlations between attrition rates and possible influencing factors. In addition, respondent meta-data for this subset of surveys was extracted from server logs and analyzed to determine attrition patterns over the course of the survey (i.e. the pages at which break-offs occur.)

Completion Rates

Average survey completion rate - 84.75%

Median survey completion rate - 89.06%

Completion rates were defined as the percentage of survey respondents that begin a given survey (submit the first page) who also complete that survey (submit the final page). Completion rates varied greatly from survey to survey in our sample, ranging from 26.09% to 100%, with an average completion rate of 84.75% and a median of 89.06%. (The discrepancy between these figures is mainly due to the large number of surveys with an completion rate close to 100%. These surveys are generally surveys in which participation can be required, e.g. employee surveys.)

Conclusions and recommendations:

Attrition rates vary greatly, but in our experience, the majority of surveys will have completion rates greater than 89%.

Factors affecting Attrition

To determine factors that may play a role in survey attrition, attrition rates (the percentage of respondents who began but did not finish the survey) were looked at in conjunction with various possible contributing factors. Variables examined at were:

the number of people who took the survey
whether the survey was run as a SuperSurvey^® Anonymous survey ¹
whether the survey was sun as a SuperSurvey^® Secure survey ²
whether a survey employs unique-key based access control or not ³
whether the survey includes the SuperSurvey percent-completed indicator
the number of pages in a survey
the number of questions in a survey

The data was analyzed to determine the degree of correlation between these variables and the survey attrition rates. The factors correlating most strongly with attrition were the measures of survey length. The number of pages and the number of questions both showed significant correlation (R2 = 0.137, p<0.0001 for pages and R2=0.128, p<0.0001 for questions)⁴ Figure 1 shows a plot of survey attrition rates vs. the number of questions on a survey. Generally, as the number of questions goes up, survey attrition rates rise. (This is especially true if you ignore those surveys with extremely high or low attrition rates, which most likely represent special cases in which other factors played a strong role in attrition rates.)

The next correlation that proved statistically significant was between attrition and the use of unique-key based access control. Figure 2 shows the attrition rates for both unique-key surveys and surveys with no key or just a single key. Unique Key systems, such as SuperSurvey's Auto-Generated and User-Specified Keys assign a sperate key to each individual invited to complete a survey. Amongst other things, this allows the system to track individual respondents as they progress through the survey, and if they are disconnected from the survey, return to the page that they left off on. Users empoying these features showed significantly lower attrition rates than those surveys with no keys or with a single key (R2=0.035, p=0.0017). This correlation is probably a least partly due to the ability for customers to resume taking these surveys after leaving, minimizing attrition due to accidental disconnects.

Another correlation examined was between attrition and the use of the SuperSurvey Progress Bar. This is a graphical indicator included on the page that indicates how far a respondent has progressed through the survey. Many researchers have postulated that the use of such a progress indicator will contribute to increased completion rates, though the research results have been inconclusive.1 In our data, it was found that the use of the indicator did have a positive effect, but only for surveys up to 6 pages. (R2=0.029, p=0.0261 for surveys of 6 or less pages). Figure 3 shows a graph of average attrition rates versus pages for surveys with and without the progress indicator. You can see how the attrition rates are lower with the progress bar for smaller surveys, and how the correlation disappears with longer surveys. It is unclear whether the fact this this correlation disappears with larger surveys is indicative of a relationship between the progress bar and survey length, or whether it is simply an artifact of the smaller sample size for longer surveys in the dataset examined. It should be noted, however, that there was no evidence that the progress bar has a negative effect on completion at higher survey lengths.

None of the other variables examined showed a statistically significant correlation to attrition rates.

Conclusions and recommendations:

Longer surveys (surveys with more pages or more questions) show significantly higher attrition rates. Clients concerned about attrition should strive to make surveys as concise as possible to minimize break-offs.
The use of unique key based tracking methods such as SuperSurvey's^® Auto- Generated Keys or User-Specified Keys corresponds to higher completion rates. This is probably at least partly because respondents who are disconnected from or leave a survey can resume where they left off, reducing attrition due to accidental disconnects.
For surveys up to 6 pages long, there is a significant correlation between the use of the SuperSurvey percent-done indicator and lower attrition rates. For longer surveys the relationship is unclear, though there is no evidence that the use of the indicator will have negetive effect on completion rates.

Attrition Patterns within Surveys

To look at attrition characteristics over the course of surveys, individual respondent meta-data corresponding to the sample surveys were extracted from the SuperSurvey logs. In order limit the variables present, attrition patterns within surveys of given page lengths were analyzed separately. Specifically, surveys of 7, 9 and 15 pages were examined, as they had enough surveys to form a useful sample and had average survey respondent numbers close to the norm ⁶.

The data was analyzed to determine at what point in the survey break-offs (people leaving the survey) occur. Figure 4 shows the number of break-offs for each of these groups in relation to the number of pages completed of the survey. There are a couple of things to notice. First, for all these groups it is apparent that the majority of survey break-offs occur at the beginning of the surveys, with the highest number of dropout occurring after only a single page has been completed. Second, there seems to be a general trend in all of these groups where the number of people who leave the survey goes down as the survey progresses.

Conclusions and recommendations:

Generally, respondents are more likely to drop out of a survey at the beginning. The most break-offs occur after first page. After the initial break-offs, attrition declines slowly as pages go up.

Next Step

Contact SuperSurvey^® for more information on successfully conducting your survey online, or visit our web application to begin building surveys right away at http://www.supersurvey.com.

SuperSurvey^®
800-241-0279
service@supersurvey.com
http://www.supersurvey.com

Notes:
1. For Anonymous Surveys, identifying information (e-mail addresses, IP addresses) are stripped from the results by the SuperSurvey^® system. Additionally, the surveys are presented with a header and footer indicating the to the respondent that they will remain anonymous.

2. The SuperSurvey^® “Secure” survey feature runs surveys under SSL security. (SSL is the encryption used for online financial transactions, amongst other applications).

3. SuperSurvey's^® Auto-Generated Keys and User-Specified Keys access control options use unique keys to track individual respondents. It was postulated that these might correlate with higher completion rates as they offer people who leave or are disconnected from the survey the ability to return and finish the survey.

4. R2 is a measure of the amount of the variation in one variable that can be predicted by another. The accompanying p value is a indication of the probability that a given result would arise simply by chance. Lower values will indicate that the correlation is more likely to be true. For example, the R2 of 0.137 for the correlation between the number of pages and attrition rates indicates that 13.7% of variation of attrition can be predicted by the number of pages in the survey. The corresponding p value of p<.0001 indicates that the probability of this strong of a correlation being found in a random sample would be less than 0.01%.

5. For a good overview, see Heerwegh, Dirk. (2004) Using Progress Indicators in Web Surveys. Paper presented at the 59th AAPOR Conference. (The paper has also been made available by the author at http://www.kuleuven.ac.be/dvz/lazsite/heerwegh.pdf.)

6. Overly high average respondents could indicate the presence of one or more very large surveys, which would be overly weighted in any aggregate analysis.

Welcome!