Chapter 8 Survey Research

1
Chapter 8Survey Research
2
Introduction

• Survey research involves the collection of
  information from a sample of individuals through
  their responses to questions.
• Surveys are the most popular form of social
  research because of their versatility,
  efficiency, and generalizability.
• Many survey datasets, like the General Social
  Survey, are available for social scientists to
  use in teaching and research.

3
Attractions of Survey Research

• Survey research owes its popularity to three
  features versatility, efficiency, and
  generalizability.
• Each of these features is changing as a result of
  new technologies.

4
Versatility

• First and foremost, survey methods are versatile.
• Although a survey is not the ideal method for
  testing all hypotheses or learning about every
  social process, a well-designed survey can
  enhance our understanding of just about any
  social issue.
• Computer technology has made surveys even more
  versatile. Computers can be programmed so that
  different types of respondents are asked
  different questions. Short videos or pictures can
  be presented to respondents on a computer screen.

5
Efficiency

• Surveys also are popular because data can be
  collected from many people at relatively low cost
  and, depending on the survey design, relatively
  quickly.
• Surveys are efficient because many variables can
  be measured without substantially increasing the
  time or cost.
• Modern information technology has been a mixed
  blessing for survey efficiency. The Internet
  makes it easier to survey some populations, but
  it leaves out important segments.

6
Exhibit 8.1
7
Generalizability

• Survey methods lend themselves to probability
  sampling from large populations.
• Thus, survey research is very appealing when
  sample generalizability is a central research
  goal.
• In fact, survey research is often the only means
  available for developing a representative picture
  of the attitudes and characteristics of a large
  population.

8
The Omnibus Survey

• An omnibus survey shows just how versatile,
  efficient, and generalizable a survey can be.
• An omnibus survey covers a range of topics of
  interest to different social scientists, in
  contrast to the typical survey that is directed
  at a specific research question.
• It has multiple sponsors or is designed to
  generate data useful to a broad segment of the
  social science community rather than to answer a
  particular research question. It is usually
  directed to a sample of some general population,
  so the questions, about a range of different
  issues, are appropriate to at least some sample
  members.

9
Errors in Survey Research

• It might be said that surveys are too easy to
  conduct.
• Organizations and individuals often decide that a
  survey will help to solve some important problem
  because it seems so easy to write up some
  questions and distribute them.
• But without careful attention to sampling,
  measurement, and overall survey design, the
  effort is likely to be a failure.

10
Errors in Survey Research, cont.

• For a survey to succeed, it must minimize four
  types of error (Groves 1989vi, 1012)
• Poor measurement. Presenting clear and
  interesting questions in a well-organized
  questionnaire will help to reduce measurement
  error by encouraging respondents to answer
  questions carefully and to take seriously the
  request to participate in the survey. Tailoring
  questions to the specific population surveyed is
  also important.

11
Errors in Survey Research, cont.

• Nonresponse. Nonresponse is a major and growing
  problem in survey research, although it is a
  problem that varies between particular survey
  designs.
• Inadequate coverage of the population. A poor
  sampling frame can invalidate the results of an
  otherwise well-designed survey.
• Sampling error. The process of random sampling
  can result in differences between the
  characteristics of the sample members and the
  population simply on the basis of chance.

12
Writing Questions

• Questions are the centerpiece of survey research.
  
• Because the way they are worded can have a great
  effect on the way they are answered, selecting
  good questions is the single most important
  concern for survey researchers.
• All hope for achieving measurement validity is
  lost unless the questions in a survey are clear
  and convey the intended meaning to respondents.

13
Writing Questions, cont.

• Consider just a few of the differences between
  everyday conversations and standardized surveys
  that make writing survey questions much more
  difficult
• Survey questions must be asked of many people,
  not just one.
• The same survey question must be used with each
  person, not tailored to the specifics of a given
  conversation.
• Survey questions must be understood in the same
  way by people who differ in many ways.

14
Writing Questions, cont.

• You will not be able to rephrase a survey
  question if someone doesnt understand it because
  that would result in a different question for
  that person.
• Survey respondents dont know you and so cant be
  expected to share the nuances of expression that
  help you and your friends and family to
  communicate.
• Adherence to a few basic principles will go a
  long way toward ensuring clear and meaningful
  questions.

15
Writing Questions, cont.

• Avoid Confusing Phrasing. A simple direct
  approach to asking a question minimizes
  confusion.
• Use shorter rather than longer words and
  sentences.
• Breaking up complex issues into simple parts also
  reduces confusion.
• A sure way to muddy the meaning of a question is
  to use double negatives.
• So-called double-barreled questions are also
  guaranteed to produce uninterpretable results
  because they actually ask two questions but allow
  only one answer.

16
Writing Questions, cont.

• Minimize the Risk of Bias. Specific words in
  survey questions should not trigger biases,
  unless that is the researchers conscious intent.
  
• Biased or loaded words and phrases tend to
  produce misleading answers.
• Answers can also be biased by more subtle
  problems in phrasing that make certain responses
  more or less attractive to particular groups.
• Responses can also be biased when response
  alternatives do not reflect the full range of
  possible sentiment on an issue.

17
Exhibit 8.2
18
Writing Questions, cont.

• Avoid Making Either Disagreement or Agreement
  Disagreeable. People often tend to agree with a
  statement just to avoid seeming disagreeable.
• This is termed agreement bias, social
  desirability bias, or an acquiescence effect.
• The response choices themselves should be phrased
  to make each one seem as socially approved, as
  agreeable, as the others.

19
Writing Questions, cont.

• Minimize Fence-Sitting and Floating. Two related
  problems in question writing also stem from
  peoples desire to choose an acceptable answer.
• There is no uniformly correct solution to these
  problems
• Fence-sitters, people who see themselves as being
  neutral, may skew the results if you force them
  to choose between opposites.

20
Writing Questions, cont.

• Even more people can be termed floaters
  respondents who choose a substantive answer when
  they really dont know or have no opinion.
• In spite of the prevalence of floating, people
  often have an opinion but are reluctant to
  express it.
• Because there are so many floaters in the typical
  survey sample, the decision to include an
  explicit Dont know option for a question is
  important.

21
Exhibit 8.3
22
Writing Questions, cont.
Maximize the Utility of Response Categories.
Questions with fixed response choices must
provide one and only one possible response for
everyone who is asked the questionthat is, the
response choices must be exhaustive and mutually
exclusive. There are two exceptions to this
principle (1) Filter questions may tell some
respondents to skip over a question (the response
choices do not have to be exhaustive), and (2)
respondents may be asked to check all that
apply (the response choices are not mutually
exclusive).
23
Writing Questions, cont.

• Vagueness in the response choices is also to be
  avoided.
• Questions about thoughts and feelings will be
  more reliable if they refer to specific times or
  events.
• Sometimes, problems with response choices can be
  corrected by adding questions.
• How many response categories are desirable? Five
  categories work well for unipolar ratings, while
  seven will capture most variation on bipolar
  ratings (Krosnick 2006 Schaeffer and Presser
  200378-79).

24
Combining Questions in Indexes

• Writing single questions that yield usable
  answers is always a challenge.
• Simple though they may seem, single questions are
  prone to error due to idiosyncratic variation,
  which occurs when individuals responses vary
  because of their reactions to particular words or
  ideas in the question.
• Differences in respondents backgrounds,
  knowledge, and beliefs almost guarantee that some
  will understand the same question differently.

25
Combining Questions in Indexes, cont.

• But the best option is often to develop multiple
  questions about a concept and then to average the
  responses to those questions in a composite
  measure termed an index or scale.
• Index The sum or average of responses to a set of
  questions about a concept.
• The idea is that idiosyncratic variation in
  response to particular questions will average
  out, so that the main influence on the combined
  measure will be the concept that all the
  questions focus on.
• The index can be considered a more complete
  measure of the concept than can any one of the
  component questions.

26
Combining Questions in Indexes, cont.

• Because of the popularity of survey research,
  indexes already have been developed to measure
  many concepts, and some of these indexes have
  proved to be reliable in a range of studies.
• It usually is much better to use such an index to
  measure a concept than to try to devise questions
  to form a new index.
• Use of a preexisting index both simplifies the
  work involved in designing a study and
  facilitates comparison of findings to those
  obtained in other studies.

27
Exhibit 8.6
28
Combining Questions in Indexes, cont.

• Three cautions are in order
• Our presupposition that each component question
  is indeed measuring the same concept may be
  mistaken.
• Combining responses to specific questions can
  obscure important differences in meaning among
  the questions.
• The questions in an index may cluster together in
  subsets. (User factor analysis)

29
Exhibit 8.7
30
Designing Questionnaires

• Questionnaire The survey instrument containing
  the questions in a self-administered survey.
• Interview schedule The survey instrument
  containing the questions asked by the interviewer
  in an in-person or phone survey.
• Survey researchers must give very careful
  attention to the design of the questionnaire as
  well as to the individual questions that it
  includes.

31
Designing Questionnaires, cont.

• The way a questionnaire should be designed varies
  with the specific survey method used and with
  other particulars of a survey project.
• There can be no precise formula for identifying
  questionnaire features that reduce error.
• Nonetheless, some key principles should guide the
  design of any questionnaire, and some systematic
  procedures should be considered for refining it.

32
Designing Questionnaires, cont.

• Build on Existing Instruments. If another
  researcher already has designed a set of
  questions to measure a key concept, and evidence
  from previous surveys indicates that this measure
  is reliable and valid, then use that instrument.
• Refine and Test Questions. Adhering to the
  preceding question-writing guidelines will go a
  long way toward producing a useful questionnaire.
  However, simply asking what appear to you to be
  clear questions does not ensure that people have
  a consistent understanding of what you are
  asking.

33
Designing Questionnaires, cont.

• Add Interpretive Questions. These will help the
  researcher understand what the respondent meant
  by his or her responses to particular questions.
• Consider five issues when developing interpretive
  questionsor when you review survey results and
  need to consider what the answers tell you.

34
Designing Questionnaires, cont.

1. What do the respondents know?
2. What relevant experiences do the respondents
   have?
3. How consistent are the respondents attitudes,
   and do they express some larger perspective or
   ideology?
4. Are respondents actions consistent with their
   expressed attitudes?
5. How strongly are the attitudes held?

35
Maintain Consistent Focus

• A survey (with the exception of an omnibus
  survey) should be guided by a clear conception of
  the research problem under investigation and the
  population to be sampled.
• Until the research objective is formulated
  clearly, survey design cannot begin.
• Throughout the process of questionnaire design,
  this objective should be the primary basis for
  making decisions about what to include and
  exclude and what to emphasize or treat in a
  cursory fashion.

36
Order the Questions

• The order in which questions are presented will
  influence how respondents react to the
  questionnaire as a whole and how they may answer
  some questions.
• As a first step, the individual questions should
  be sorted into broad thematic categories, which
  then become separate sections in the
  questionnaire.
• Throughout the design process, the grouping of
  questions in sections and the ordering of
  questions within sections should be adjusted to
  maximize the questionnaires overall coherence.

37
Order the Questions, cont.

• The first question deserves special attention,
  particularly if the questionnaire is to be
  self-administered.
• This question signals to the respondent what the
  survey is about, whether it will be interesting,
  and how easy it will be to complete.
• For these reasons, the first question should be
  connected to the primary purpose of the survey,
  it should be interesting, it should be easy, and
  it should apply to everyone in the sample
  (Dillman 20009294).

38
Order the Questions, cont.

• One or more filter or screening questions may
  also appear early in the survey in order to
  identify respondents for whom the questionnaire
  is not intended or perhaps to determine which
  sections of a multipart questionnaire a
  respondent is to skip (Peterson 2000106107).
• Prior questions can influence how questions are
  comprehended, what beliefs shape responses, and
  whether comparative judgments are made
  (Tourangeau 1999).

39
Make the Questionnaire Attractive

• An attractive questionnaire is more likely to be
  completed and less likely to confuse either the
  respondent or, in an interview, the interviewer.
• An attractive questionnaire also should increase
  the likelihood that different respondents
  interpret the same questions in the same way.
• Printing a multipage questionnaire in booklet
  form usually results in the most attractive and
  simple-to-use questionnaire.

40
Exhibit 8.8
41
Consider Translation

• Should the survey be translated into one or more
  languages?
• In the 21st century, no survey plan in the United
  States or many other countries can be considered
  complete until this issue has been considered.
• When immigrants are a sizeable portion of a
  population, omitting them from a survey can
  result in a misleading description of the
  population.

42
Consider Translation, cont.

• A properly translated questionnaire will be
• Reliable convey the intended meaning of the
  original text
• Complete do not add any new information nor omit
  any information in the source document
• Accurate free of spelling and grammatical
  errors
• Culturally appropriate convey a message that is
  appropriate for the target population
• Equivalent maintain in the target language the
  same terms and sentence structures and concepts
  and adhere to the same societal rules as in the
  source language and culture.

43
Organizing Surveys

• There are five basic social science survey
  designs.
• Survey researchers are now also combining
  elements of two or more of these basic designs in
  mixed mode surveys.
• Manner of administration. The five survey designs
  differ in the manner in which the questionnaire
  is administered.

44
Organizing Surveys, cont.

• Questionnaire structure. Survey designs also
  differ in the extent to which the content and
  order of questions are structured in advance by
  the researcher.
• Setting. Most surveys are conducted in settings
  where only one respondent completes the survey at
  a time, but some are administered to groups.
• Cost. The expense of different types of surveys
  can vary greatly, with phone surveys being the
  least expensive.

45
External Validity

• Because of their different features, the five
  designs vary in the types of error to which they
  are most prone and the situations in which they
  are most appropriate.
• They can also be improved in different ways by
  adding some features of the other designs.

46
Five Basic Social Science Survey Designs

• Mailed, Self-Administered Surveys
• Group-Administered Surveys
• Telephone Surveys
• In-Person Interviews
• Web Surveys

47
Mixed-Mode Surveys

• Mixed-mode surveys allow the strengths of one
  survey design to compensate for the weaknesses of
  another and they can maximize the likelihood of
  securing data from different types of respondents
  (Dillman 2007451-453 Selm and Jankowski 2006).
• The mixed mode approach is not a perfect
  solution. Respondents to the same question may
  give different answers because of the survey
  mode, rather than because they actually have
  different opinions.

48
A Comparison of Survey Designs

• Which survey design should be used when?
• The most important consideration in comparing the
  advantages and disadvantages of the methods is
  the likely response rate they will generate.

49
A Comparison of Survey Designs, cont.

• Various points about the different survey designs
  lead to two general conclusions
• First, in-person interviews are the strongest
  design and generally preferable when sufficient
  resources and a trained interview staff are
  available telephone surveys have many of the
  advantages of in-person interviews at much less
  cost, but response rates are an increasing
  problem.
• Second, the best survey design for any
  particular study will be determined by the
  studys unique features and goals rather than by
  any absolute standard of what the best survey
  design is.

50
Ethical Issues in Survey Research

• Survey research usually poses fewer ethical
  dilemmas than do experimental or field research
  designs.
• Potential respondents to a survey can easily
  decline to participate, and a cover letter or
  introductory statement that identifies the
  sponsors of, and motivations for, the survey
  gives them the information required to make this
  decision.
• The methods of data collection are quite obvious
  in a survey, so little is concealed from the
  respondents.

51
Ethical Issues in Survey Research, cont.

• Current federal regulations to protect human
  subjects allow survey research to be exempted
  from formal review unless respondents can be
  identified and disclosure of their responses
  could place them at risk.
• Confidentiality is most often the primary focus
  of ethical concern in survey research.
• Many surveys include some essential questions
  that might, in some way, prove damaging to the
  subjects if their answers were disclosed.
• To prevent any possibility of harm to subjects
  due to disclosure of such information, the
  researcher must preserve subject confidentiality.
  

52
Conclusions

• Survey research is an exceptionally efficient and
  productive method for investigating a wide array
  of social research questions.
• In addition to the potential benefits for social
  science, considerations of time and expense
  frequently make a survey the preferred
  data-collection method.
• The relative ease of conducting at least some
  types of survey research leads many people to
  imagine that no particular training or systematic
  procedures are required.

53
Conclusions, cont.

• Nothing could be further from the truth.
• But as a result of this widespread misconception,
  you will encounter a great many nearly worthless
  survey results.
• You must be prepared to examine carefully the
  procedures used in any survey before accepting
  its findings as credible.
• And if you decide to conduct a survey, you must
  be prepared to invest the time and effort
  required by proper procedures.