Week 2 Survey Research Experiment - PowerPoint PPT Presentation

1 / 54
About This Presentation

Week 2 Survey Research Experiment


SURVEY RESEARCH Components Questionnaires & Interviews Week 2 Survey Research Experiment – PowerPoint PPT presentation

Number of Views:198
Avg rating:3.0/5.0
Slides: 55
Provided by: Sandy307
Learn more at: http://www.unm.edu


Transcript and Presenter's Notes

Title: Week 2 Survey Research Experiment

Week 2Survey ResearchExperiment

  • Components Questionnaires Interviews

Surveys are the most Popular method for
information gathering in Social Research
  • Why?
  • Versatile appropriate for studying most issues
  • Efficient appropriate for use with large
    populations, relatively low cost, quick, can use
    many variables, can be geographically diverse
  • Generalizability appropriate for probability
    sampling- can generalize from sample to
  • Omnibus studies good for large studies like
    General Social Survey

Problems in Conducting Social Research
  • Sampling, measurement and Design issues
  • Use of Survey as Design Society is dynamic
    survey is one shot deal
  • Two types of Data Errors in Survey
  • 1. poor measurement of cases that are surveyed
  • 2. omission of cases that should have been
  • Why? Individuals dont respond, poor sampling
    frame, sampling error (characteristics of sample
    members dont match the characteristics of

Survey Instrument Types
  • Questionnaire Self Administered
  • Interview Schedule- Questions asked face-to-face
    and answers recorded by researcher.
  • Both must be well thought out, clear, and
    integrated as a whole with questions
    complementing each other
  • Questions must be asked of many people, not just
    one person
  • Must use same instrument for all participants
  • Questions must be understood in the same way by
    all participants

Principles of Question Writingfor Survey research
  • Avoid confusion (double negatives and
    double-barreled questions)
  • Screening filter questions to move participants
    to other parts of survey based on their answers
    (contingency questions)
  • Bias- Avoid words that trigger bias (loaded) i.e.
    communist take-over like Vietnam
  • Instead of in a situation like Vietnam

  • Dont lead or mislead with questions
  • Response range deficient response range must
    reflect full range of possible answers
  • Continuums number of positive and negative
    categories must be balanced
  • Minimize fence sitting neutral category may
    attract too many only use neutral when you need
    to know how many fence-sitters you have.

  • Dont Know/No Opinion May omit if it keeps
    people from expressing their real opinion
  • People float to no opinion categories
  • However, forcing people to choose an answer
    even if they dont know anything about the topic
    is also a problem
  • Make sure you know why you are using these
    categories and have a purpose for them

Response Categories for Questions
  • Can be closed ended or open-ended
  • Closed-ended- Answer categories are provided by
    researcher. Respondents check or circle their
    choice Were you abused? 1.yes 2.no
  • Open-ended Respondent provides the answer in
    his/her own words. Can be lengthy and
    disorganized. Used to get at the meaning the
    subjects attach to their answers How were you

Best Response Category for Questionnaires
  • Closed Ended (Fixed-choice) (one and only one
    possible response for everyone who is asked the
    question) 1. Male 2. Female 3. Other
  • Categories must be exhaustive (if you cant list
    all possible choices then provide an other
  • Categories must be mutually exclusive (Ranges of
    ages, incomes, years of school etc. should not
    overlap and should not leave out any value
    ranges) 1. 20-30 2. 21-40.
  • Check all that apply should be used with caution
    and kept to a minimum.

  • If a variable cant be measured with one question
    then create an index or scale
  • ( Several questions to measure the same
    variable) Depression cant be measured simply by
    asking Are you depressed?
  • If some questions in an index carry more weight
    than others then you have a scale

Guttman Scale
  • It is okay for Aliens to work where I work
  • It is okay for Aliens to live in my neighborhood
  • It is okay for Aliens to live next door.
  • It is okay to be friends with Aliens
  • It is okay to marry Aliens
  • It is okay to have children with Aliens

Multi-item Index
  • Several questions for one concept
  • Use statistical testing of index to make sure all
    questions fit in index (i.e. correlation or
    factor analysis)
  • Popular Likert Index (often called scale)
  • Ordinal measure ranging from low to high. Is
    the subject liberal about ethnic differences?
  • Aliens are bad
  • 1. Strongly Agree, Agree, Disagree, 4.
    Strongly Disagree
  • Can sum the responses for individual
    statements to get a summated rating scale for
    each subject

Design Issues
  • Pay attention to the order of questions
    (questions can influence each other)
  • Sort the questions into thematic categories
  • Start with easy and interesting questions
  • Maintain consistent focus on the research
    problem eliminate irrelevant questions
  • Have a clear conception of the population to be

  • Make the Questionnaire attractive
  • Indicate path through the questionnaire with
    words, arrows or graphics
  • Mailed questionnaires must include a cover letter
    and stamped, self-addressed return envelop

Response Categories and other issues for
Interview Schedules
  • Looks like a Questionnaire, except it is read to
    the respondent by the interviewer
  • A mix of Open-ended and Fixed Choice
  • Interviewer negotiates through the questionnaire
    path for the subject
  • Interviewer must seem to be asking questions from
    memory and float seamlessly along
  • Interviewer must make the interview seem personal
    and must seem engaged and interested
  • Must begin with an introductory statement to draw
    the respondent in
  • May want to send a copy of questions to the
    interviewee before the interview appointment

Refine and Test Questions
  • All questionnaires and interview schedules must
    be pre-tested on a small sample
  • Focus groups can be used to formulate content of
    questions for surveys

Most Popular 5 Designsfor Administering Surveys
  • Mailed
  • Group-administered
  • Phone
  • In-person Interview
  • Electronic

Important Issues for Administering Surveys
  • Manner of Administration Is the survey
    completed by the respondent or does the
    researcher ask the questions and record the
  • Structure Is the instrument highly structured
    or relatively unstructured
  • Setting Is the survey being answered in an
    individual or group setting
  • Cost What are the costs and time expectations
    of each of the 5 basic designs?

Mailed, Self Administered Surveys
  • Respondents administer the survey themselves
  • Low response rates
  • Have to do follow up, include self-addressed and
    stamped envelope, maybe token reward
  • Mostly structured
  • Lower cost
  • Subjects must be literate
  • Cant be sure who answers it

Group-Administered Surveys
  • Researcher distributes or provides access
  • and it is administered in group setting
  • Individual Group members administer surveys to
  • High response rate
  • Mostly structured
  • lower cost
  • Works well with students, employees, members of
    military or institutionalized populations
  • Participants often feel coerced, less likely to
    answer honestly, Believe researcher is not
    independent of sponsoring organization

Telephone Surveys
  • Respondents interviewed via phone so is not
    self-administered but by professionals
  • May be computerized and responses automatically
  • Can be structured or unstructured
  • Cost is high
  • Safe, efficient, large samples, fast turnaround
  • Validity questionable may not reach proper
    sampling frame
  • Lots of incomplete responses

  • Most phone surveys use random digit dialing in
    sampling process
  • CATI (Computer-assisted Telephone Interviewing
    software) often used
  • May require multiple call backs
  • 10-15 time max
  • People have negative bias toward telemarketing
  • Impersonal nature of phone interviewing (need
    careful training) (Also could be computer asking

In-Person Interview Surveys
  • Face-to-face interaction
  • More, Longer and more complex questions
  • Administered by professional
  • Mostly structured
  • Cost very high and time consuming
  • High response rates
  • Interviewer can monitor conditions
  • Interviewer can probe for meaning
  • Can use more open-ended Qs

  • Order in which questions read and answered can be
  • But, in spite of flexibility, all respondents
    should have same interview experience
  • Interview should seem personalized
  • Computer-assisted personal interviewing software
    (CAPI) can be used. Can display interview
    schedule on laptop. The software processes the
    answers and checks to make sure answers fall into
    allowed ranges
  • Difficult to record answers and appear to be

Electronic Surveys
  • Self-Administered via computer over Internet
  • Delivered via Web or Email
  • Mostly structured
  • Cost very low
  • Email surveys usually have to be short
  • Web-based surveys can be longer
  • Require programming expertise by researcher
  • Web-based can include graphics, links, pull-down
    menus, pictures, audio
  • Responses recorded directly into researchers
    database (eliminates data entry error quick
    report generation)
  • Could be problems with sampling frame
  • Fast growing technique

IVR (Less Popular)Interactive Voice Response
  • Allows Internet surveys to be achieved with
    telephone-based technology
  • Respondent received automated call
  • Answer questions by pressing numbers on
    touch-tone phone or speaking numbers that are
    interpreted by computerized voice recognition
  • Is a very impersonal approach

Mixed Mode Surveys
  • Combinations of 5 basic survey administration
    designs (group,mailed, interview, phone,
  • Maximizes quantity and quality of data
  • Allow strengths of one design to compensate for
    weaknesses in another
  • Maximizes likelihood of securing data from
    different types of respondents

  • Mailed surveys
  • worst response rate
  • best for large, diverse populations
  • inexpensive
  • good for sensitive issues
  • Best alternative Phone survey

  • In-person Survey
  • Best in terms of possible length and
  • Not as good for sensitive questions
  • Require a lot of training
  • Best response rate
  • Expensive and Time consuming

  • Electronic Surveys
  • Not everyone has Internet connection or
    capacity to display complex Web pages
  • Conclusions
  • In-person Interview is strongest design and
    generally preferable
  • Phone surveys have many of the advantages of
    in-person interview but at much less cost.
    Response rates a problem.
  • A decision about the best survey design must take
    into account the unique features and goals of the
    particular study.
  • Survey research which is mostly Quantitative can
    be enhanced by combining with Qualitative
    research such as Case Studies, Focus Groups or

Ethical Issues in Survey Research
  • Fewer ethical issues than Experiment or
    Participant Observation
  • In group-administered survey the question of
    participation being voluntary arises
  • Primary ethical concern is Confidentiality
  • Answers to questions could be damaging to
    subject if disclosed
  • Only research personnel should have access to
    information that could link subjects identity to
    answers. Use ID numbers not names
  • Anonymity But, if you have no identifying info
    linking subjects to their surveys then follow-up
    is not possible

  • Answers questions about the effect of a treatment
    or intervention on some other variable whose
    values can be manipulated by the researcher. If
    Clarinex given then symptoms reduced. 1. Clarinex
    given 2. placebo given 3. nothing given
  • Experiment is powerful design for testing
    hypotheses (A research question in testable
  • Hypothesis is explanatory - quantitative

  • A specific expectation deduced from a more
    general theory
  • We test the hypotheses not the more general
  • Variation in one variable in the hypothesis
  • is supposed to predict, influence or cause
    variation in another variable in the hypothesis

  • Deterrence Theory Punishment deters crime
  • Deterrence Hypothesis If abusers are arrested
    on first offense then abusers will be less likely
    to abuse again.
  • or
  • As arrests for abuse increase, recidivism for
    abuse decreases

Components of Hypotheses
  • Independent variable considered to be the
    causal variable. It is manipulated by researcher
    to effect (influence or cause) change in
    Dependent (another) variable
  • Independent variableCause
  • Dependent variableEffect

  • A cause is an explanation for some
    characteristic, attitude or behavior
  • Causation is often shown by experiment
  • A variation in the IV results in variation in
    the DV

Essential Components of Experimental Design
  • At least two comparison groups of subjects
    (usually experimental and control groups)
  • Variation in the independent variable occurs
    before assessment of change in the dependent
  • Random assignment to the two (or more) comparison
  • A combination of these components gives us more
    confidence in the validity of causal conclusions

  • Confidence is increased with the addition of two
    other components
  • 1. Control over the context of an experiment
  • 2. Identification of the causal mechanism

Conditions of a True Experiment
  • Must have at least one experimental group
    (subjects who receive some treatment or
  • At least one comparison group (subjects to whom
    the experimental group can be compared) who
    receive a different treatment or no treatment
  • If no treatment is given to the comparison group
    it is called a control group

  • All true experiments have a post-test
    (measurement of the outcome in both groups after
    the experimental group has received the
  • Does not require a pre-test but it can be
  • Pre-tests provide a measure of how much the
    experimental and comparison groups change over

Popular Pre-test/Post-test Control Group Design
  • Two or more groups (at least one experimental and
    one control)
  • Pre-test and post-tests
  • Random Assignment to Groups (Randomization)
  • Note This is not the same thing as Random
    Sampling to be discussed in Sampling Lecture

  • Random Assignment places pre-designated subjects
    into two or more groups on the basis of chance
  • If comparison group differs from the
    experimental group in anyway besides not
    receiving treatment or receiving a different
    treatment, cant determine for sure what the
    unique effects of the treatments are.

  • In true experiment subjects must be randomly
    assigned to the comparison and experimental
  • Eliminates systematic bias
  • When used, the odds of a difference between the
    comparison and experimental groups by chance can
    be calculated.
  • Larger the group the better this works (over 30
    subjects is best)

  • Matching subjects are assigned to the different
    groups based on similarity of variables such as
    gender, age, year in school, or other important
  • researcher identifies in advance all the
    important variables on which to make a match of
    the assignment to groups (should be used in
    conjunction with random assignment not instead

Limitations of True Experiment
  • Difficult to isolate the actual mechanism by
    which treatments have their effect
  • Difficult to guarantee that control has been
    maintained (more likely to maintain control in
    lab than in field)

  • Often testing a hypothesis with a true
    experimental design is not feasible
  • It may be too costly, time consuming or desired
    setting or subjects not available
  • In Quasi-experiment- subjects in groups may not
    be randomly assigned

Popular Quasi-Experiment Designs
  • Nonequivalent Control Group Experimental and
    control groups are designated before the
    treatment occurs, not by random assignment
  • Before and After Designs Has a pretest and
    posttest, but no comparison group. The subjects
    exposed to treatment serve as own controls by
    comparison of pre and post tests
  • 1. Multiple Group Before and After Designs
    several before and after comparisons are made
    involving the same variables but different groups
    and then the groups are compared
  • 2. Repeated Measures panel Designs- the same
    group is observed many times (30 or more),
    receiving many pre and post tests. Why? To study
    the process by which an intervention or treatment
    has an impact over time.

Identifying Cause
  • Quasi-Experiments only partially meet the
    criteria for identifying a cause.
  • But, the association between the IV and DV and
    the Context in which change occurs can still be
    met in this type of experiment

Validity in Experiment
  • Internal Validity
  • 1. Selection bias characteristics of
    treatment groups differ (attrition maybe)
  • 2. Endogenous change subjects develop or
    change during the experiment (maybe experiment
  • 3. History something occurs during treatment
    which influences results
  • 4. Contamination one of the groups is aware
    of the others and is influenced by this knowledge

  • 5. Treatment misidentification researcher is
    not aware of some process that affects treatment
    . For example the way staff administers
  • Hawthorn Effect patients perform better because
    feel special. In Evaluation Research, clients
    know research findings may affect chances of
    further funding
  • Double Blind procedures can offset this (staff
    doesnt know who is getting what)
  • Placebo Effect- patient improves on placebo
    because they think they are getting the real thing

External Validity - Irony
  • The more that assignment to treatments is
    randomized and all experimental conditions are
    controlled, the less likely it is that the
    research subjects and setting will be
    representative of the larger population.

Ethics in Experiment
  • Deception subjects may be misled, but deception
    is a critical component of many social
  • If used, debriefing may be a good idea
  • Deception can mean harm to subject, lack
    of voluntary
    participation and informed consent
  • Selective Distribution of Benefits Can cause
    harm to subjects (but researcher may not know
    that one treatment is more effective)

Advantages and Disadvantages
  • Positives
  • Can study change over time
  • Control, Control, Control
  • Can Replicate
  • Can Quantify

  • Negatives
  • Artificiality
  • Major Ethical issues
  • Small subject groups (cant generalize)
  • Sampling problems (not random)
  • Internal Validity (Experiment effects)
  • External Validity (history)
Write a Comment
User Comments (0)
About PowerShow.com