Week 2 Survey Research Experiment

1
Week 2Survey ResearchExperiment

• SURVEY RESEARCH
• Components Questionnaires Interviews

2
Surveys are the most Popular method for
information gathering in Social Research

• Why?
• Versatile appropriate for studying most issues
• Efficient appropriate for use with large
  populations, relatively low cost, quick, can use
  many variables, can be geographically diverse
• Generalizability appropriate for probability
  sampling- can generalize from sample to
  population.
• Omnibus studies good for large studies like
  General Social Survey

3
Problems in Conducting Social Research

• Sampling, measurement and Design issues
• Use of Survey as Design Society is dynamic
  survey is one shot deal
• Two types of Data Errors in Survey
• 1. poor measurement of cases that are surveyed
• 2. omission of cases that should have been
  surveyed
• Why? Individuals dont respond, poor sampling
  frame, sampling error (characteristics of sample
  members dont match the characteristics of
  population)

4
Survey Instrument Types

• Questionnaire Self Administered
• Interview Schedule- Questions asked face-to-face
  and answers recorded by researcher.
• Both must be well thought out, clear, and
  integrated as a whole with questions
  complementing each other
• Questions must be asked of many people, not just
  one person
• Must use same instrument for all participants
• Questions must be understood in the same way by
  all participants

5
Principles of Question Writingfor Survey research

• Avoid confusion (double negatives and
  double-barreled questions)
• Screening filter questions to move participants
  to other parts of survey based on their answers
  (contingency questions)
• Bias- Avoid words that trigger bias (loaded) i.e.
  communist take-over like Vietnam
• Instead of in a situation like Vietnam

6

• Dont lead or mislead with questions
• Response range deficient response range must
  reflect full range of possible answers
• Continuums number of positive and negative
  categories must be balanced
• Minimize fence sitting neutral category may
  attract too many only use neutral when you need
  to know how many fence-sitters you have.

7

• Dont Know/No Opinion May omit if it keeps
  people from expressing their real opinion
• People float to no opinion categories
• However, forcing people to choose an answer
  even if they dont know anything about the topic
  is also a problem
• Make sure you know why you are using these
  categories and have a purpose for them

8
Response Categories for Questions

• Can be closed ended or open-ended
• Closed-ended- Answer categories are provided by
  researcher. Respondents check or circle their
  choice Were you abused? 1.yes 2.no
• Open-ended Respondent provides the answer in
  his/her own words. Can be lengthy and
  disorganized. Used to get at the meaning the
  subjects attach to their answers How were you
  abused?_____

9
Best Response Category for Questionnaires

• Closed Ended (Fixed-choice) (one and only one
  possible response for everyone who is asked the
  question) 1. Male 2. Female 3. Other
• Categories must be exhaustive (if you cant list
  all possible choices then provide an other
  category
• Categories must be mutually exclusive (Ranges of
  ages, incomes, years of school etc. should not
  overlap and should not leave out any value
  ranges) 1. 20-30 2. 21-40.
• Check all that apply should be used with caution
  and kept to a minimum.

10

• If a variable cant be measured with one question
  then create an index or scale
• ( Several questions to measure the same
  variable) Depression cant be measured simply by
  asking Are you depressed?
• If some questions in an index carry more weight
  than others then you have a scale

11
Guttman Scale

• It is okay for Aliens to work where I work
• It is okay for Aliens to live in my neighborhood
• It is okay for Aliens to live next door.
• It is okay to be friends with Aliens
• It is okay to marry Aliens
• It is okay to have children with Aliens

12
Multi-item Index

• Several questions for one concept
• Use statistical testing of index to make sure all
  questions fit in index (i.e. correlation or
  factor analysis)
• Popular Likert Index (often called scale)
• Ordinal measure ranging from low to high. Is
  the subject liberal about ethnic differences?
• Aliens are bad
• 1. Strongly Agree, Agree, Disagree, 4.
  Strongly Disagree
• Can sum the responses for individual
  statements to get a summated rating scale for
  each subject

13
Design Issues

• Pay attention to the order of questions
  (questions can influence each other)
• Sort the questions into thematic categories
• Start with easy and interesting questions
• Maintain consistent focus on the research
  problem eliminate irrelevant questions
• Have a clear conception of the population to be
  sampled

14

• Make the Questionnaire attractive
• Indicate path through the questionnaire with
  words, arrows or graphics
• Mailed questionnaires must include a cover letter
  and stamped, self-addressed return envelop

15
Response Categories and other issues for
Interview Schedules

• Looks like a Questionnaire, except it is read to
  the respondent by the interviewer
• A mix of Open-ended and Fixed Choice
• Interviewer negotiates through the questionnaire
  path for the subject
• Interviewer must seem to be asking questions from
  memory and float seamlessly along
• Interviewer must make the interview seem personal
  and must seem engaged and interested
• Must begin with an introductory statement to draw
  the respondent in
• May want to send a copy of questions to the
  interviewee before the interview appointment

16
Refine and Test Questions

• All questionnaires and interview schedules must
  be pre-tested on a small sample
• Focus groups can be used to formulate content of
  questions for surveys

17
Most Popular 5 Designsfor Administering Surveys

• Mailed
• Group-administered
• Phone
• In-person Interview
• Electronic

18
Important Issues for Administering Surveys

• Manner of Administration Is the survey
  completed by the respondent or does the
  researcher ask the questions and record the
  answers
• Structure Is the instrument highly structured
  or relatively unstructured
• Setting Is the survey being answered in an
  individual or group setting
• Cost What are the costs and time expectations
  of each of the 5 basic designs?

19
Mailed, Self Administered Surveys

• Respondents administer the survey themselves
• Low response rates
• Have to do follow up, include self-addressed and
  stamped envelope, maybe token reward
• Mostly structured
• Lower cost
• Subjects must be literate
• Cant be sure who answers it

20
Group-Administered Surveys

• Researcher distributes or provides access
• and it is administered in group setting
• Individual Group members administer surveys to
  selves
• High response rate
• Mostly structured
• lower cost
• Works well with students, employees, members of
  military or institutionalized populations
• Participants often feel coerced, less likely to
  answer honestly, Believe researcher is not
  independent of sponsoring organization

21
Telephone Surveys

• Respondents interviewed via phone so is not
  self-administered but by professionals
• May be computerized and responses automatically
  recorded
• Can be structured or unstructured
• Cost is high
• Safe, efficient, large samples, fast turnaround
• Validity questionable may not reach proper
  sampling frame
• Lots of incomplete responses

22

• Most phone surveys use random digit dialing in
  sampling process
• CATI (Computer-assisted Telephone Interviewing
  software) often used
• May require multiple call backs
• 10-15 time max
• People have negative bias toward telemarketing
• Impersonal nature of phone interviewing (need
  careful training) (Also could be computer asking
  questions)

23
In-Person Interview Surveys

• Face-to-face interaction
• More, Longer and more complex questions
• Administered by professional
• Mostly structured
• Cost very high and time consuming
• High response rates
• Interviewer can monitor conditions
• Interviewer can probe for meaning
• Can use more open-ended Qs

24

• Order in which questions read and answered can be
  controlled
• But, in spite of flexibility, all respondents
  should have same interview experience
• Interview should seem personalized
• Computer-assisted personal interviewing software
  (CAPI) can be used. Can display interview
  schedule on laptop. The software processes the
  answers and checks to make sure answers fall into
  allowed ranges
• Difficult to record answers and appear to be
  engaged

25
Electronic Surveys

• Self-Administered via computer over Internet
• Delivered via Web or Email
• Mostly structured
• Cost very low
• Email surveys usually have to be short
• Web-based surveys can be longer
• Require programming expertise by researcher
• Web-based can include graphics, links, pull-down
  menus, pictures, audio
• Responses recorded directly into researchers
  database (eliminates data entry error quick
  report generation)
• Could be problems with sampling frame
• Fast growing technique

26
IVR (Less Popular)Interactive Voice Response
System

• Allows Internet surveys to be achieved with
  telephone-based technology
• Respondent received automated call
• Answer questions by pressing numbers on
  touch-tone phone or speaking numbers that are
  interpreted by computerized voice recognition
  software
• Is a very impersonal approach

27
Mixed Mode Surveys

• Combinations of 5 basic survey administration
  designs (group,mailed, interview, phone,
  electronic)
• Maximizes quantity and quality of data
• Allow strengths of one design to compensate for
  weaknesses in another
• Maximizes likelihood of securing data from
  different types of respondents

28
Comparisons

• Mailed surveys
• worst response rate
• best for large, diverse populations
• inexpensive
• good for sensitive issues
• Best alternative Phone survey

29

• In-person Survey
• Best in terms of possible length and
  complexity
• Not as good for sensitive questions
• Require a lot of training
• Best response rate
• Expensive and Time consuming

30

• Electronic Surveys
• Not everyone has Internet connection or
  capacity to display complex Web pages
• Conclusions
• In-person Interview is strongest design and
  generally preferable
• Phone surveys have many of the advantages of
  in-person interview but at much less cost.
  Response rates a problem.
• A decision about the best survey design must take
  into account the unique features and goals of the
  particular study.
• Survey research which is mostly Quantitative can
  be enhanced by combining with Qualitative
  research such as Case Studies, Focus Groups or
  Observation

31
Ethical Issues in Survey Research

• Fewer ethical issues than Experiment or
  Participant Observation
• In group-administered survey the question of
  participation being voluntary arises
• Primary ethical concern is Confidentiality
• Answers to questions could be damaging to
  subject if disclosed
• Only research personnel should have access to
  information that could link subjects identity to
  answers. Use ID numbers not names
• Anonymity But, if you have no identifying info
  linking subjects to their surveys then follow-up
  is not possible

32
EXPERIMENT IN RESEARCH

• Answers questions about the effect of a treatment
  or intervention on some other variable whose
  values can be manipulated by the researcher. If
  Clarinex given then symptoms reduced. 1. Clarinex
  given 2. placebo given 3. nothing given
• Experiment is powerful design for testing
  hypotheses (A research question in testable
  format)
• Hypothesis is explanatory - quantitative

33
Hypotheses

• A specific expectation deduced from a more
  general theory
• We test the hypotheses not the more general
  theories
• Variation in one variable in the hypothesis
• is supposed to predict, influence or cause
  variation in another variable in the hypothesis

34
Example

• Deterrence Theory Punishment deters crime
• Deterrence Hypothesis If abusers are arrested
  on first offense then abusers will be less likely
  to abuse again.
• or
• As arrests for abuse increase, recidivism for
  abuse decreases

35
Components of Hypotheses

• Independent variable considered to be the
  causal variable. It is manipulated by researcher
  to effect (influence or cause) change in
  Dependent (another) variable
• Independent variableCause
• Dependent variableEffect

36
CAUSATION

• A cause is an explanation for some
  characteristic, attitude or behavior
• Causation is often shown by experiment
• A variation in the IV results in variation in
  the DV

37
Essential Components of Experimental Design

• At least two comparison groups of subjects
  (usually experimental and control groups)
• Variation in the independent variable occurs
  before assessment of change in the dependent
  variable
• Random assignment to the two (or more) comparison
  groups
• A combination of these components gives us more
  confidence in the validity of causal conclusions

38

• Confidence is increased with the addition of two
  other components
• 1. Control over the context of an experiment
• 2. Identification of the causal mechanism

39
Conditions of a True Experiment

• Must have at least one experimental group
  (subjects who receive some treatment or
  manipulation)
• At least one comparison group (subjects to whom
  the experimental group can be compared) who
  receive a different treatment or no treatment
• If no treatment is given to the comparison group
  it is called a control group

40

• All true experiments have a post-test
  (measurement of the outcome in both groups after
  the experimental group has received the
  treatment)
• Does not require a pre-test but it can be
  advantageous
• Pre-tests provide a measure of how much the
  experimental and comparison groups change over
  time.

41
Popular Pre-test/Post-test Control Group Design

• Two or more groups (at least one experimental and
  one control)
• Pre-test and post-tests
• Random Assignment to Groups (Randomization)
• Note This is not the same thing as Random
  Sampling to be discussed in Sampling Lecture

42

• Random Assignment places pre-designated subjects
  into two or more groups on the basis of chance
• If comparison group differs from the
  experimental group in anyway besides not
  receiving treatment or receiving a different
  treatment, cant determine for sure what the
  unique effects of the treatments are.

43

• In true experiment subjects must be randomly
  assigned to the comparison and experimental
  groups.
• Eliminates systematic bias
• When used, the odds of a difference between the
  comparison and experimental groups by chance can
  be calculated.
• Larger the group the better this works (over 30
  subjects is best)

44

• Matching subjects are assigned to the different
  groups based on similarity of variables such as
  gender, age, year in school, or other important
  characteristics
• researcher identifies in advance all the
  important variables on which to make a match of
  the assignment to groups (should be used in
  conjunction with random assignment not instead
  of

45
Limitations of True Experiment

• Difficult to isolate the actual mechanism by
  which treatments have their effect
• Difficult to guarantee that control has been
  maintained (more likely to maintain control in
  lab than in field)

46
Quasi-Experiments

• Often testing a hypothesis with a true
  experimental design is not feasible
• It may be too costly, time consuming or desired
  setting or subjects not available
• In Quasi-experiment- subjects in groups may not
  be randomly assigned

47
Popular Quasi-Experiment Designs

• Nonequivalent Control Group Experimental and
  control groups are designated before the
  treatment occurs, not by random assignment
• Before and After Designs Has a pretest and
  posttest, but no comparison group. The subjects
  exposed to treatment serve as own controls by
  comparison of pre and post tests
• 1. Multiple Group Before and After Designs
  several before and after comparisons are made
  involving the same variables but different groups
  and then the groups are compared
• 2. Repeated Measures panel Designs- the same
  group is observed many times (30 or more),
  receiving many pre and post tests. Why? To study
  the process by which an intervention or treatment
  has an impact over time.

48
Identifying Cause

• Quasi-Experiments only partially meet the
  criteria for identifying a cause.
• But, the association between the IV and DV and
  the Context in which change occurs can still be
  met in this type of experiment

49
Validity in Experiment

• Internal Validity
• 1. Selection bias characteristics of
  treatment groups differ (attrition maybe)
• 2. Endogenous change subjects develop or
  change during the experiment (maybe experiment
  effect)
• 3. History something occurs during treatment
  which influences results
• 4. Contamination one of the groups is aware
  of the others and is influenced by this knowledge

50

• 5. Treatment misidentification researcher is
  not aware of some process that affects treatment
  . For example the way staff administers
  treatment.
• Hawthorn Effect patients perform better because
  feel special. In Evaluation Research, clients
  know research findings may affect chances of
  further funding
• Double Blind procedures can offset this (staff
  doesnt know who is getting what)
• Placebo Effect- patient improves on placebo
  because they think they are getting the real thing

51
External Validity - Irony

• The more that assignment to treatments is
  randomized and all experimental conditions are
  controlled, the less likely it is that the
  research subjects and setting will be
  representative of the larger population.

52
Ethics in Experiment

• Deception subjects may be misled, but deception
  is a critical component of many social
  experiments
• If used, debriefing may be a good idea
• Deception can mean harm to subject, lack
  of voluntary
  participation and informed consent
• Selective Distribution of Benefits Can cause
  harm to subjects (but researcher may not know
  that one treatment is more effective)

53
Advantages and Disadvantages

• Positives
• Can study change over time
• Control, Control, Control
• Can Replicate
• Can Quantify

54

• Negatives
• Artificiality
• Major Ethical issues
• Small subject groups (cant generalize)
• Sampling problems (not random)
• Internal Validity (Experiment effects)
• External Validity (history)