Chapter 8 Survey Research - PowerPoint PPT Presentation

1 / 53
About This Presentation

Chapter 8 Survey Research


Chapter 8 Survey Research * * * * * * * * * * * * * * * * * * * * * * * * * Designing Questionnaires, cont. What do the respondents know? What relevant experiences do ... – PowerPoint PPT presentation

Number of Views:44
Avg rating:3.0/5.0
Slides: 54
Provided by: carole164
Learn more at:


Transcript and Presenter's Notes

Title: Chapter 8 Survey Research

Chapter 8Survey Research
  • Survey research involves the collection of
    information from a sample of individuals through
    their responses to questions.
  • Surveys are the most popular form of social
    research because of their versatility,
    efficiency, and generalizability.
  • Many survey datasets, like the General Social
    Survey, are available for social scientists to
    use in teaching and research.

Attractions of Survey Research
  • Survey research owes its popularity to three
    features versatility, efficiency, and
  • Each of these features is changing as a result of
    new technologies.

  • First and foremost, survey methods are versatile.
  • Although a survey is not the ideal method for
    testing all hypotheses or learning about every
    social process, a well-designed survey can
    enhance our understanding of just about any
    social issue.
  • Computer technology has made surveys even more
    versatile. Computers can be programmed so that
    different types of respondents are asked
    different questions. Short videos or pictures can
    be presented to respondents on a computer screen.

  • Surveys also are popular because data can be
    collected from many people at relatively low cost
    and, depending on the survey design, relatively
  • Surveys are efficient because many variables can
    be measured without substantially increasing the
    time or cost.
  • Modern information technology has been a mixed
    blessing for survey efficiency. The Internet
    makes it easier to survey some populations, but
    it leaves out important segments.

Exhibit 8.1
  • Survey methods lend themselves to probability
    sampling from large populations.
  • Thus, survey research is very appealing when
    sample generalizability is a central research
  • In fact, survey research is often the only means
    available for developing a representative picture
    of the attitudes and characteristics of a large

The Omnibus Survey
  • An omnibus survey shows just how versatile,
    efficient, and generalizable a survey can be.
  • An omnibus survey covers a range of topics of
    interest to different social scientists, in
    contrast to the typical survey that is directed
    at a specific research question.
  • It has multiple sponsors or is designed to
    generate data useful to a broad segment of the
    social science community rather than to answer a
    particular research question. It is usually
    directed to a sample of some general population,
    so the questions, about a range of different
    issues, are appropriate to at least some sample

Errors in Survey Research
  • It might be said that surveys are too easy to
  • Organizations and individuals often decide that a
    survey will help to solve some important problem
    because it seems so easy to write up some
    questions and distribute them.
  • But without careful attention to sampling,
    measurement, and overall survey design, the
    effort is likely to be a failure.

Errors in Survey Research, cont.
  • For a survey to succeed, it must minimize four
    types of error (Groves 1989vi, 1012)
  • Poor measurement. Presenting clear and
    interesting questions in a well-organized
    questionnaire will help to reduce measurement
    error by encouraging respondents to answer
    questions carefully and to take seriously the
    request to participate in the survey. Tailoring
    questions to the specific population surveyed is
    also important.

Errors in Survey Research, cont.
  • Nonresponse. Nonresponse is a major and growing
    problem in survey research, although it is a
    problem that varies between particular survey
  • Inadequate coverage of the population. A poor
    sampling frame can invalidate the results of an
    otherwise well-designed survey.
  • Sampling error. The process of random sampling
    can result in differences between the
    characteristics of the sample members and the
    population simply on the basis of chance.

Writing Questions
  • Questions are the centerpiece of survey research.
  • Because the way they are worded can have a great
    effect on the way they are answered, selecting
    good questions is the single most important
    concern for survey researchers.
  • All hope for achieving measurement validity is
    lost unless the questions in a survey are clear
    and convey the intended meaning to respondents.

Writing Questions, cont.
  • Consider just a few of the differences between
    everyday conversations and standardized surveys
    that make writing survey questions much more
  • Survey questions must be asked of many people,
    not just one.
  • The same survey question must be used with each
    person, not tailored to the specifics of a given
  • Survey questions must be understood in the same
    way by people who differ in many ways.

Writing Questions, cont.
  • You will not be able to rephrase a survey
    question if someone doesnt understand it because
    that would result in a different question for
    that person.
  • Survey respondents dont know you and so cant be
    expected to share the nuances of expression that
    help you and your friends and family to
  • Adherence to a few basic principles will go a
    long way toward ensuring clear and meaningful

Writing Questions, cont.
  • Avoid Confusing Phrasing. A simple direct
    approach to asking a question minimizes
  • Use shorter rather than longer words and
  • Breaking up complex issues into simple parts also
    reduces confusion.
  • A sure way to muddy the meaning of a question is
    to use double negatives.
  • So-called double-barreled questions are also
    guaranteed to produce uninterpretable results
    because they actually ask two questions but allow
    only one answer.

Writing Questions, cont.
  • Minimize the Risk of Bias. Specific words in
    survey questions should not trigger biases,
    unless that is the researchers conscious intent.
  • Biased or loaded words and phrases tend to
    produce misleading answers.
  • Answers can also be biased by more subtle
    problems in phrasing that make certain responses
    more or less attractive to particular groups.
  • Responses can also be biased when response
    alternatives do not reflect the full range of
    possible sentiment on an issue.

Exhibit 8.2
Writing Questions, cont.
  • Avoid Making Either Disagreement or Agreement
    Disagreeable. People often tend to agree with a
    statement just to avoid seeming disagreeable.
  • This is termed agreement bias, social
    desirability bias, or an acquiescence effect.
  • The response choices themselves should be phrased
    to make each one seem as socially approved, as
    agreeable, as the others.

Writing Questions, cont.
  • Minimize Fence-Sitting and Floating. Two related
    problems in question writing also stem from
    peoples desire to choose an acceptable answer.
  • There is no uniformly correct solution to these
  • Fence-sitters, people who see themselves as being
    neutral, may skew the results if you force them
    to choose between opposites.

Writing Questions, cont.
  • Even more people can be termed floaters
    respondents who choose a substantive answer when
    they really dont know or have no opinion.
  • In spite of the prevalence of floating, people
    often have an opinion but are reluctant to
    express it.
  • Because there are so many floaters in the typical
    survey sample, the decision to include an
    explicit Dont know option for a question is

Exhibit 8.3
Writing Questions, cont.
Maximize the Utility of Response Categories.
Questions with fixed response choices must
provide one and only one possible response for
everyone who is asked the questionthat is, the
response choices must be exhaustive and mutually
exclusive. There are two exceptions to this
principle (1) Filter questions may tell some
respondents to skip over a question (the response
choices do not have to be exhaustive), and (2)
respondents may be asked to check all that
apply (the response choices are not mutually
Writing Questions, cont.
  • Vagueness in the response choices is also to be
  • Questions about thoughts and feelings will be
    more reliable if they refer to specific times or
  • Sometimes, problems with response choices can be
    corrected by adding questions.
  • How many response categories are desirable? Five
    categories work well for unipolar ratings, while
    seven will capture most variation on bipolar
    ratings (Krosnick 2006 Schaeffer and Presser

Combining Questions in Indexes
  • Writing single questions that yield usable
    answers is always a challenge.
  • Simple though they may seem, single questions are
    prone to error due to idiosyncratic variation,
    which occurs when individuals responses vary
    because of their reactions to particular words or
    ideas in the question.
  • Differences in respondents backgrounds,
    knowledge, and beliefs almost guarantee that some
    will understand the same question differently.

Combining Questions in Indexes, cont.
  • But the best option is often to develop multiple
    questions about a concept and then to average the
    responses to those questions in a composite
    measure termed an index or scale.
  • Index The sum or average of responses to a set of
    questions about a concept.
  • The idea is that idiosyncratic variation in
    response to particular questions will average
    out, so that the main influence on the combined
    measure will be the concept that all the
    questions focus on.
  • The index can be considered a more complete
    measure of the concept than can any one of the
    component questions.

Combining Questions in Indexes, cont.
  • Because of the popularity of survey research,
    indexes already have been developed to measure
    many concepts, and some of these indexes have
    proved to be reliable in a range of studies.
  • It usually is much better to use such an index to
    measure a concept than to try to devise questions
    to form a new index.
  • Use of a preexisting index both simplifies the
    work involved in designing a study and
    facilitates comparison of findings to those
    obtained in other studies.

Exhibit 8.6
Combining Questions in Indexes, cont.
  • Three cautions are in order
  • Our presupposition that each component question
    is indeed measuring the same concept may be
  • Combining responses to specific questions can
    obscure important differences in meaning among
    the questions.
  • The questions in an index may cluster together in
    subsets. (User factor analysis)

Exhibit 8.7
Designing Questionnaires
  • Questionnaire The survey instrument containing
    the questions in a self-administered survey.
  • Interview schedule The survey instrument
    containing the questions asked by the interviewer
    in an in-person or phone survey.
  • Survey researchers must give very careful
    attention to the design of the questionnaire as
    well as to the individual questions that it

Designing Questionnaires, cont.
  • The way a questionnaire should be designed varies
    with the specific survey method used and with
    other particulars of a survey project.
  • There can be no precise formula for identifying
    questionnaire features that reduce error.
  • Nonetheless, some key principles should guide the
    design of any questionnaire, and some systematic
    procedures should be considered for refining it.

Designing Questionnaires, cont.
  • Build on Existing Instruments. If another
    researcher already has designed a set of
    questions to measure a key concept, and evidence
    from previous surveys indicates that this measure
    is reliable and valid, then use that instrument.
  • Refine and Test Questions. Adhering to the
    preceding question-writing guidelines will go a
    long way toward producing a useful questionnaire.
    However, simply asking what appear to you to be
    clear questions does not ensure that people have
    a consistent understanding of what you are

Designing Questionnaires, cont.
  • Add Interpretive Questions. These will help the
    researcher understand what the respondent meant
    by his or her responses to particular questions.
  • Consider five issues when developing interpretive
    questionsor when you review survey results and
    need to consider what the answers tell you.

Designing Questionnaires, cont.
  1. What do the respondents know?
  2. What relevant experiences do the respondents
  3. How consistent are the respondents attitudes,
    and do they express some larger perspective or
  4. Are respondents actions consistent with their
    expressed attitudes?
  5. How strongly are the attitudes held?

Maintain Consistent Focus
  • A survey (with the exception of an omnibus
    survey) should be guided by a clear conception of
    the research problem under investigation and the
    population to be sampled.
  • Until the research objective is formulated
    clearly, survey design cannot begin.
  • Throughout the process of questionnaire design,
    this objective should be the primary basis for
    making decisions about what to include and
    exclude and what to emphasize or treat in a
    cursory fashion.

Order the Questions
  • The order in which questions are presented will
    influence how respondents react to the
    questionnaire as a whole and how they may answer
    some questions.
  • As a first step, the individual questions should
    be sorted into broad thematic categories, which
    then become separate sections in the
  • Throughout the design process, the grouping of
    questions in sections and the ordering of
    questions within sections should be adjusted to
    maximize the questionnaires overall coherence.

Order the Questions, cont.
  • The first question deserves special attention,
    particularly if the questionnaire is to be
  • This question signals to the respondent what the
    survey is about, whether it will be interesting,
    and how easy it will be to complete.
  • For these reasons, the first question should be
    connected to the primary purpose of the survey,
    it should be interesting, it should be easy, and
    it should apply to everyone in the sample
    (Dillman 20009294).

Order the Questions, cont.
  • One or more filter or screening questions may
    also appear early in the survey in order to
    identify respondents for whom the questionnaire
    is not intended or perhaps to determine which
    sections of a multipart questionnaire a
    respondent is to skip (Peterson 2000106107).
  • Prior questions can influence how questions are
    comprehended, what beliefs shape responses, and
    whether comparative judgments are made
    (Tourangeau 1999).

Make the Questionnaire Attractive
  • An attractive questionnaire is more likely to be
    completed and less likely to confuse either the
    respondent or, in an interview, the interviewer.
  • An attractive questionnaire also should increase
    the likelihood that different respondents
    interpret the same questions in the same way.
  • Printing a multipage questionnaire in booklet
    form usually results in the most attractive and
    simple-to-use questionnaire.

Exhibit 8.8
Consider Translation
  • Should the survey be translated into one or more
  • In the 21st century, no survey plan in the United
    States or many other countries can be considered
    complete until this issue has been considered.
  • When immigrants are a sizeable portion of a
    population, omitting them from a survey can
    result in a misleading description of the

Consider Translation, cont.
  • A properly translated questionnaire will be
  • Reliable convey the intended meaning of the
    original text
  • Complete do not add any new information nor omit
    any information in the source document
  • Accurate free of spelling and grammatical
  • Culturally appropriate convey a message that is
    appropriate for the target population
  • Equivalent maintain in the target language the
    same terms and sentence structures and concepts
    and adhere to the same societal rules as in the
    source language and culture.

Organizing Surveys
  • There are five basic social science survey
  • Survey researchers are now also combining
    elements of two or more of these basic designs in
    mixed mode surveys.
  • Manner of administration. The five survey designs
    differ in the manner in which the questionnaire
    is administered.

Organizing Surveys, cont.
  • Questionnaire structure. Survey designs also
    differ in the extent to which the content and
    order of questions are structured in advance by
    the researcher.
  • Setting. Most surveys are conducted in settings
    where only one respondent completes the survey at
    a time, but some are administered to groups.
  • Cost. The expense of different types of surveys
    can vary greatly, with phone surveys being the
    least expensive.

External Validity
  • Because of their different features, the five
    designs vary in the types of error to which they
    are most prone and the situations in which they
    are most appropriate.
  • They can also be improved in different ways by
    adding some features of the other designs.

Five Basic Social Science Survey Designs
  • Mailed, Self-Administered Surveys
  • Group-Administered Surveys
  • Telephone Surveys
  • In-Person Interviews
  • Web Surveys

Mixed-Mode Surveys
  • Mixed-mode surveys allow the strengths of one
    survey design to compensate for the weaknesses of
    another and they can maximize the likelihood of
    securing data from different types of respondents
    (Dillman 2007451-453 Selm and Jankowski 2006).
  • The mixed mode approach is not a perfect
    solution. Respondents to the same question may
    give different answers because of the survey
    mode, rather than because they actually have
    different opinions.

A Comparison of Survey Designs
  • Which survey design should be used when?
  • The most important consideration in comparing the
    advantages and disadvantages of the methods is
    the likely response rate they will generate.

A Comparison of Survey Designs, cont.
  • Various points about the different survey designs
    lead to two general conclusions
  • First, in-person interviews are the strongest
    design and generally preferable when sufficient
    resources and a trained interview staff are
    available telephone surveys have many of the
    advantages of in-person interviews at much less
    cost, but response rates are an increasing
  • Second, the best survey design for any
    particular study will be determined by the
    studys unique features and goals rather than by
    any absolute standard of what the best survey
    design is.

Ethical Issues in Survey Research
  • Survey research usually poses fewer ethical
    dilemmas than do experimental or field research
  • Potential respondents to a survey can easily
    decline to participate, and a cover letter or
    introductory statement that identifies the
    sponsors of, and motivations for, the survey
    gives them the information required to make this
  • The methods of data collection are quite obvious
    in a survey, so little is concealed from the

Ethical Issues in Survey Research, cont.
  • Current federal regulations to protect human
    subjects allow survey research to be exempted
    from formal review unless respondents can be
    identified and disclosure of their responses
    could place them at risk.
  • Confidentiality is most often the primary focus
    of ethical concern in survey research.
  • Many surveys include some essential questions
    that might, in some way, prove damaging to the
    subjects if their answers were disclosed.
  • To prevent any possibility of harm to subjects
    due to disclosure of such information, the
    researcher must preserve subject confidentiality.

  • Survey research is an exceptionally efficient and
    productive method for investigating a wide array
    of social research questions.
  • In addition to the potential benefits for social
    science, considerations of time and expense
    frequently make a survey the preferred
    data-collection method.
  • The relative ease of conducting at least some
    types of survey research leads many people to
    imagine that no particular training or systematic
    procedures are required.

Conclusions, cont.
  • Nothing could be further from the truth.
  • But as a result of this widespread misconception,
    you will encounter a great many nearly worthless
    survey results.
  • You must be prepared to examine carefully the
    procedures used in any survey before accepting
    its findings as credible.
  • And if you decide to conduct a survey, you must
    be prepared to invest the time and effort
    required by proper procedures.
Write a Comment
User Comments (0)