Q. What is Survey Methodology?
A. "Survey Methodology" refers to the set of practices and principles which help set survey
research apart from unscientific questioning. The goal of good methodology is to get as much data as
is needed, and to ensure that the results are true, accurate, and helpful to the survey maker.
Good methodology includes writing questions well, selecting and inviting respondents properly, and
having a standardized, easy survey-taking experience for respondents. We strive to help you at each
of these steps, with help available on respondent selection,
writing effective
questions, and a list of specific features
that SuperSurvey offers to enhance your survey's methodology.
Q. Is the Internet a good medium for surveying people?
A. Yes -- if the people you want to survey are online. Naturally, if you want to
survey people who are not users of the Internet, the Web will not work well. For
this reason, Web-based surveys do not represent the general population. It would
be improper, for example, to take the U.S. Census on the Web (yet).
However, most survey makers find that the populations they are interested in
do use the Web. For instance, surveys of executives, high-income consumers,
corporate purchasers, college students, technology businesses, and of course,
computer owners, are all well-suited to the Web, because these groups are likely to
use the Web.
Q. Are Web survey results "valid?"
A. Just like with paper surveys, the results of a well-designed and executed survey
will be more "valid" than those of a poorly done survey. The information in this
FAQ is provided to help you make excellent surveys, using SuperSurvey's services to
their fullest potential.
Q. What specific methodology features does SuperSurvey offer?
Good methodology is more than just a catchphrase at SuperSurvey; it is a guiding principle at every
step of our process. But don't take our word for it -- check out this list of actual methodology
features and options:
Q. Whom should I survey to get the best results?
A. The information from a survey is most useful for learning about the survey
respondents and people who are similar to them. Therefore, the results from a
survey of retired persons will be less useful for making decisions about working
people or students.
Most business and professional users have a good idea about who they want to
question. Ideally, survey makers can use SuperSurvey to selectively target those
people. For example, you can solicit responses directly from your customers,
website users, or employees.
Some Web survey services charge outrageous fees to provide "survey pools" who give
"valid" answers. This is a very suspect claim, because the "professional"
survey respondents that make up these pools are usually not similar to your
populations of interest. In fact, survey pool members are mostly concerned with
getting through as many surveys as possible in order to get their rewards -- not
with honestly answering questions.
Q. How can I write a question most effectively?
A. The most important thing in question design is knowing what information you
want. Many survey makers find it helpful to write down what specific information
they are seeking first, before deciding on how to phrase their questions.
In general, a question should be stated as simply and clearly as possible.
Also, while you want to keep your questions short and sweet, you also want everyone
who reads the question to understand it the same way.
For example, a question like "If the Presidential election were today, who would
you vote for?" is better than "Who would you choose?", because the latter could
have different meanings (who would you choose to sit next to, who would you choose
to impeach, etc.). At the same time, "Assuming the Presidential election were to
be held today, for whom would you cast your ballot?" is too long and wordy.
Beware of using loaded words that may influence your respondent's choices. For
example, "How much do you like liver-and-onions flavor ice cream?" is a better
question than "How disgusted are you by the revolting taste of liver-and-onions ice
cream?" Finally, be consistent -- if you ask a question about Fords, you should
ask the question about Chevrolets in the same way.
Q. What question type should I use?
A. The simplest type of data to analyze are those which have one and only one
answer to each question. The question types which resemble that are
multiple-choice with single answer, and pull-down box.
If you want to rank a large number of choices (i.e., rating your opinions from 1
to 5 on different soda flavors), a matrix question may be right for you. Matrix
questions also allow for easy comparisons and analysis of the data.
Free-response questions are invaluable for gaining feedback in a respondent's own
words, but are difficult to analyze.
Q. How many answer choices should I give? Should questions be required?
A. You should give the most flexibility to respondents while making the data useful
to you. Most people are used to responding to between three and six choices on a
multiple-choice question. If you give too many options, respondents may not give
optimal answers. On the other hand, if respondents feel they are being required to
"pigeonhole" into a choice, they may give up.
Studies show that people are more likely to give you fully completed surveys if
there is a "none of the above" or "no opinion" choice on multiple-choice questions.
Also, your data are more likely to give an accurate picture of the situation that
way (e.g., if you give only two choices in a political poll, you could get
misleading responses from third-party respondents).
Q. What is LOGIQ™ (Linking, Ordering, and Grouping of Interdependent
Questions)? Should I use it?
A. LOGIQ™ is a way of grouping your survey's questions into pages, and linking
these pages together based on conditions (e.g., If the answer to question 3 is
"Yes," skip to question 9...).
These conditions, known as skip logic, can be very helpful if you have a long or
complex survey, in order to reduce the amount of work each survey taker must do and
therefore increase response rates. However, if you do not require skip logic, you
can still use LOGIQ™ to map out the order of your questions.
Q. Should I have a one-page or multi-page survey?
A. If your survey is only a few questions, it is a good idea to keep them all on
one page. Experts have found that response rates are highest when respondents have
to make the fewest number of clicks to finish.
However, if your survey scrolls down a page with many questions, you should
consider breaking it up into a number of short, manageable pages. We offer the
option of a "percent completed" indicator, which has been shown in scientific
studies to increase the willingness of respondents to complete a multi-page survey.
Q. I want to make sure I have a "valid" set of responses.
A. The biggest concern with sampling is that your sample resemble the population
that you are studying. For example, if you wanted data about web users in general,
but you only asked computer programmers to take your survey, you would get
information that is more relevant to programmers than to general users.
However, if you are surveying not just a sample, but most or all of your population
of interest, you don't need to worry as much. For example, if you are surveying
all your employees to learn about employee satisfaction, you automatically have an
excellent sample.
An issue that is often present is response/non-response bias. If some people aren't taking
a survey, is it because they are different in some important way from the people who do respond? To
help address this concern, surveys with Auto-Generated
keys permit tracking and follow-up of non-respondents, in order to help minimize the bias.
Q. What about "statistical significance?"
A. Statistical significance is basically a mathematician's bet that the results of a
survey are not random. Results are called significant if there is a very high
chance that things would come up the same if you did the survey again.
Statistical significance of results, while important for determining if a scientific survey deserves
publication, is usually not a critical concern for business surveys. If your survey need is rigorous
in this aspect, however, SuperSurvey offers an excellent platform upon which to seek significant
results.
If you have need of a statistically rigorous survey project, and you require highly
skilled consulting knowledge, SuperSurvey can refer you to a statistically trained
professional who can provide guidance for a fee. Contact us at
service@supersurvey.com.
Q. Can you recommend some good publications on Web surveying?
Dillman,
Don A. Mail and Internet Surveys: The Tailored Design Method. John
Wiley and Sons, 1999.
Salant,
Pricilla and Don A. Dillman. How to Conduct Your Own Survey. John Wiley
and Sons, 1994.
Sudman,
Seymour and Norman M. Bradburn. Asking Questions. Jossey-Bass, 1982.
Q. Where can I get professional help with survey methodology?
A. SuperSurvey focuses on providing the very best technology, so we do not directly provide
consulting. However, we are happy to provide our customers with referrals to experts in statistics
and the social sciences, who can provide telephone and e-mail consultations on methodological and
statistical matters at an hourly rate. Please email inquiries to service@supersurvey.com.
An introduction to Internet based surveying from a respected name in traditional
surveying.
This is a good general introduction to the idea of surveying for research, although it
does not treat more recent issues of Internet surveying.
Although this book is older, it remains the gold standard for question design, including
numerous examples and checklists to help design good questions.