Survey research techniques: Matching the method with the research question - PowerPoint PPT Presentation

1 / 21
About This Presentation

Survey research techniques: Matching the method with the research question


Title: Survey research techniques: Different Ways of asking your research subjects to answer your research questions Author: stein Last modified by – PowerPoint PPT presentation

Number of Views:114
Avg rating:3.0/5.0
Slides: 22
Provided by: stei91


Transcript and Presenter's Notes

Title: Survey research techniques: Matching the method with the research question

Survey research techniques Matching the method
with the research question
  • Robert M. Stein
  • Department of Political Science
  • Rice University
  • March 2015

Popular survey techniques
  • Face-to-face interviews
  • Mail surveys
  • Self-administered surveys
  • Telephone interviews
  • Live interviewer
  • Interactive voice recorded
  • Web based surveys (with/without telephone
    recruitment. e.g., Mechanical Turk, Knowledge
    networks, Goggle Consumer Surveys
  • Focus groups

Major research issues with surveys
  • Sampling
  • Cost
  • Reactivity
  • Psychology of the survey response
  • The dont know response
  • Response time
  • Social desirability
  • Question wording and placement
  • Meaning of survey responses

Survey methods for studying who votes
  • Web based surveys
  • Strengths Cost, sample size, timely data
    collection, survey panels
  • Weaknesses Selection bias e.g., digital divide
  • Telephone, face-to-face surveys
  • Strengths Sample size, limited to registered
  • Weaknesses Social desirability, reactivity,
    recall problems, sample selection bias (unlisted
    phone number
  • Exit poll
  • Strengths Validated voters, limited number of
  • Weaknesses Reactivity, unable to study
  • IVR poll
  • Strengths Large sample, less invasive, lower
  • Weaknesses Low response rate, limited number of
    questions, skewed sample
  • Surveyless Survey (e.g., annotated voter
  • Strengths Unambiguous, valid and reliable
  • Weaknesses Limited sample (i.e. registered
    voters) and bias time series

Overreporting voting
  • Robert Bernstein, Anita Chadha and Robert
    Montjoy, Overreporting Voting Why it Happens
    and Why it Matters. Public Opinion Quarterly
  • Allyson L. Holbrook and Jon A. Krosnick,
    Measuring voter turnout by using the randomized
    response technique Evidence calling into
    question the methods validity, Public Opinion
    Quarterly 74(2010)328-343.
  • Brian D. Silver, Paul R. Abramson and Barbara A.
    Anderson, The presence of others and
    overreporting of voting in American National
    Elections, Public Opinion Quarterly
  • Robert F. Belli, Sean E. Moore and John
    VanHoewyk, An experimental comparison of
    question forms used to reduced vote
    overreporting, Electoral Studies 25(2006)751-759

Social expectation and over reporting of voting
  • Treatment 1
  • In talking to people about elections, we often
    find that a lot of people were not able to vote
    because they weren't registered, they were sick,
    or they just didn't have time. How about you--did
    you vote in the elections this November?
  • Treatment 2
  • In talking to people about elections, we often
    find that a lot of people were not able to vote
    because they weren't registered, they were sick,
    or they just didn't have time. Which of the
    following statements best describes you One, I
    did not vote (in the election this November)
    Two, I thought about voting this time - but
    didn't Three, I usually vote, but didn't this
    time or Four, I am sure I voted?

Live telephone and IVR surveys of 2009 Houston
mayoral election voters
  • Live telephone interview
  • 500 registered voters interviewed 11/15/09
  • Error rate /- 4.5
  • Interview length 7.5-12 minutes (95)
  • Response rate 32
  • IVR interview
  • 586 registered voters interviewed 11/15/09
  • Error rate /- 3.3
  • Interview length 1- 3.5 minutes (95)
  • Response rate 8.5

(No Transcript)
(No Transcript)
Response time (latency) measures of contextual
effects on voting behavior
  • Problem Studies of contextual processes have
    always involved the possibility that if
    individuals aggregation into geographic units is
    not exogenous to their values on the dependent
    variable, then what appear to be contextual
    processes may be due solely to selection
  • Solution measure contextual variables separately
    from the individual-level variables, and later to
    connect them to the survey data by means of
    linkage variables. Latency measures of response
    time to survey generated questions.
  • Methodology Cati (computer assisted telephone
    interviewing) technology used to generate latency
    measures to query to respondents assessment of
    the partisanship of their neighbors
  • M. Johnson, W. Phillips Shively and R.M. Stein,
    Contextual data and the study of elections and
    voting behavior connecting individuals to
    environments. Electoral Studies

  1. When perception of partisanship of ones
    neighbors (i.e., (generally speaking, do you
    usually think of your neighborhood as Republican,
    Democratic or Independent?) is latent (i.e.,
    faster response time) context sould have a
    significant and positive effect on vote choice.
  2. Republicans (Democrats) residing in neighborhoods
    that perceive to be Democratic (Republican) will
    be more likely to vote for Democrats
    (Republicans) when their perception of the
    partisan make-up of their neighborhood is readily
  3. The accessibility of ones context will be
    greater when that context is congruent with a
    respondents personal preference.

Research design
  • Live telephone interview with 750 registered
    voters in Harris County, Texas, conducted
    September 23-29, 1999
  • Retrospective vote choice in the 1996
    Presidential election, partisanship and
    perception of partisanship of neighborhood
  • Response times collected on Cati system similar
    to Bassli and Fletcher (1991).

(No Transcript)
(No Transcript)
(No Transcript)
Analyzing surveys
  • When are answers to survey questions revealing of
    individual perceptions, preferences and behavior?
  • How might multiple survey responses reveal more
    information than single responses?
  • An Answer Risk perceptions and evacuations from

Google consumer surveys(GCS)
  • GCS is a new tool developed by Google that
    surveys a sample of Internet users as they
    attempt to view content on the websites of online
    publishers who are part of the program.
  • These sites require users to complete a survey
    created by researchers in order to access premium
    content provided by the publishers.
  • Currently, GCS is available in four countries and
    it takes about 48 hours to field a survey of any
    size. The 4 countries are USA, Canada, UK, and
  • The current cost-structure of GCS makes very
    short surveys (even as short as one question) the
    most economically attractive.
  • Since the respondents are not part of an online
    panel, GCS has not collected this information at
    an earlier date (as is the case in many panels).
    Instead, GCS provides researchers with a set of
    respondents inferred demographics (location,
    income, age, gender) that are derived from their
    browsing histories and IP addresses .

Example of Survey Wall
Cost structure of GCS
Limitations of GCS
  • GCS recommends the question length to be 125
    characters and sets the maximum limit to 175
  • Censored questions and populations GCS places
    restrictions on sensitive demographics
    information by prohibiting researchers from
    asking respondents for their age, gender,
    ethnicity, religion, and immigration status.
    Researchers can only ask these questions if the
    response choices include "I prefer not to say" as
    the opt out answer.
  • GCS also requires an option for the respondents
    to not answer the question by either clicking an
    I dont know, show me another question link.
  • Observational studies that require statistical
    models including a long (or even small list of
    control variables can likely be accomplished as
    efficiently using other survey tools.

Strengths of GCS
  • First, if these experiments can be implemented in
    one question, then the considerable cost
    advantages of GCS over its competitors can be
  • Second, If true randomization of treatments can
    be achieved (and we argue that GCS methodology
    does so - and demonstrate resulting balance in a
    set of measured covariates then no additional
    control variables are necessary to make strong
  • Finally, given the fact that most experimenters
    are much more concerned with the internal
    validity of their inferences than with tier
    external validity, unresolved questions about the
    representativeness of the GCS sample may be of
    secondary concern and certainly the sample is
    better  than that of most lab experiments.
Write a Comment
User Comments (0)