PP279: RESEARCH DESIGN AND DATA COLLECTION

FOR PUBLIC POLICY ANALYSIS (CCN 77244)

Professor Robert MacCoun

Goldman School of Public Policy, 2607 Hearst Ave.

642-7518, maccoun@ berkeley.edu

Fall 2009: Tuesdays and Thursdays, 2-3:30, 105 GSPP (the old bldg.)

(Office Hours: right after class or by appt.)

 

The online version of this syllabus is at http://conium.org/~maccoun/PP279_f09.html 
Please see the online version for the most up-to-date version; I will announce any revisions in class. 

COURSE DESCRIPTION

Empirical arguments and counterarguments play a central role in policy debates, thus public policy analysis requires a sophisticated understanding of a variety of types and sources of data. Quantitative analysis courses teach you how to analyze data; this course will introduce you to strategies of data collection and principles for critically evaluating data collected by others. Topics include measurement reliability and validity, questionnaire design, sampling, experimental and quasi-experimental program evaluation designs, qualitative research methods, and the politics of data in public policy.

 

READINGS

DeVellis, R. F. (2003).  Scale Development: Theory and Applications.  Sage.

Groves, Fowler, Couper, Lepkowski, Singer, and Tourangeau (2004).  Survey Methodology.  Wiley.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs For Generalized Causal Inference. Houghton Mifflin.  [If you are interested, my review of this book appears here: http://conium.org/~maccoun/JPAM_2003_BookRev.pdf.]

Many of the additional web-based readings are marked with * to indicate that they are optional.

If a link is bad, email me (maccoun@berkeley.edu) and I’ll try to fix it ASAP.  But every reading (except my handouts) is available on the web – that’s where I found them! – and you should be able to hunt them down yourself (e.g., using Google Scholar (http://scholar.google.com/).  The ANNUAL REVIEW essays are available online but only from a UC computer account because we have a site license.

 

ASSIGNMENTS (see due dates in Schedule at end of syllabus)

  • Two draft research proposals:  Each worth 40%. Like most research proposals, these will be developed in small groups of 2-3 people (size will depend on class enrollment). Teams will give in-class briefings on the second proposals at the end of the term.

1) Survey proposal (http://conium.org/~maccoun/PP279_proposal1.html)

2) Program evaluation proposal/group briefing:  http://conium.org/~maccoun/PP279_proposal2.html

READINGS - Schedule appears on last page

NOTE: The readings are NOT in a reader; they are online at http://conium.org/~maccoun/PP279.html

The Philosophy of Science and the Politics of Data (First Look)

MacCoun, R. (1998). Biases in the interpretation and use of research results, Annual Review of Psychology, 49, 259-287. http://conium.org/~maccoun/MacCoun_AnnualReview98.pdf

 

MacCoun R. J., & Paletz, S. (2009).  Citizens’ perceptions of ideological bias in research on public policy controversies.  Political Psychology, 30, 43-65.  [Note: During the editing process, we accidentally used a formatting command that put negative signs on all the standard errors while moving them into parentheses.  Oops!  Standard errors can only take on a positive value, and the absolute values of the displayed standard errors are all correct.]

http://conium.org/~maccoun/MacCounPaletz2009PoliPsy.pdf

Describing the World: Surveys and Other Measures

Asking Questions

Chapters 7, 8, and 9 of Survey Methodology text

 

OPTIONAL READINGS:

 

*Krosnick, Jon A. (1999). Survey research. Annual Review of Psychology, 50, 537-567. http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.50.1.537 

 

*Schaeffer, Nora Cate, & Presser, Stanley (2003).  The science of asking questions.  Annual Review of Sociology Aug 2003, Vol. 29: 65-88.

http://conium.org/~maccoun/PP279_SchaefferPresser2004.pdf

 

Reliability and Validity (basic psychometrics)

First, skim the entire Scale Development book to get a general feel for the topic.  Then take a second pass through, reading Chapters 2 through 6 more carefully.  These chapters are directly relevant to the homework and to the Proposal 1 assignment.

 

Rob’s memo on coefficient alpha: 

http://conium.org/~maccoun/CoefAlpha.pdf

 

Rob’s memo on how low reliability weakens the ability to detect relationships (e.g., program effects)

http://conium.org/~maccoun/PredictiveValidity.pdf

 

 

OPTIONAL READINGS (can be found using Google Scholar):

 

*Schmitt, Neal, Uses and abuses of coefficient alpha. Psychological Assessment. 1996 Dec Vol 8(4) 350-353.
http://conium.org/~maccoun/PP279_Schmitt.pdf

 

*Neisser, U., et al. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.

http://conium.org/~maccoun/PP279_Neisser1.pdf

and his replies to critics:

http://conium.org/~maccoun/PP279_Neisser2.html

 

*Flynn, James R., Searching for justice: The discovery of IQ gains over time. American Psychologist. 1999 Jan Vol 54(1) 5-20.

http://conium.org/~maccoun/PP279_Flynn.pdf

 

*Kuncel, N. R., Hezlett, S. A., Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin. 127(1) 162-181.
http://conium.org/~maccoun/PP279_GRE.pdf

 

*Lubinski, D. (2000). Scientific and social significance of assessing individual differences: "Sinking shafts at a few critical points." Annual Review of Psychology, 51, 405-444.   

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.51.1.405

 

Survey Sampling

Chapters 1 through 6 of Survey Methodology. 

Rob’s memo on computing sample sizes.  http://conium.org/~maccoun/pp279_samplesize.pdf

OPTIONAL READINGS:

 

*Magnani, R., Sabin, K., Saidel, T., & Heckathorn, D. (2005). Review of sampling hard-to-reach and hidden populations for HIV surveillance.  AIDS 2005, 19, S67-S72.  [This has a good discussion of a sophisticated version of snowball sampling]

http://conium.org/~maccoun/PP279_Magnani.pdf

 

*Birnbaum, Michael H. (2004).  Human research and data collection via the internet.  Annual Review of Psychology, 55, 803-832.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.55.090902.141601

 

* Marshall, G.N., Burnam, M. A., Koegel, P., Sullivan, G., & Benjamin B. (1996).  Objective Life Circumstances and Life Satisfaction: Results from the Course of Homelessness Study.  Journal of Health and Social Behavior, Vol. 37, No. 1. (Mar., 1996), pp. 44-58.  [Note: This paper has a very interesting sampling strategy for a difficult to sample population; it also has a nice example of multiple-indicator measurement of latent constructs.  And two of the authors attended my wedding!]

http://conium.org/~maccoun/PP279_Marshall.pdf

 

Inferring Cause and Effect: Experimental and Quasi-Experimental Design

 

Dealing with Threats to Internal Validity

 

Shadish, Cook, & Campbell text, Chapter 1, pp. 53-62 of Chapter 2, and Chapter 8.

 

OPTIONAL READINGS: 

 

* Sander Greenland and Hal Morgenstern (2001). Confounding in health research. Annual Review of Public Health, 22, 189-212.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.22.1.189

 

* A. G. Barnett, J. C. Van der Pols, & A. J. Dobson (2005).  Regression to the mean: What it is and how to deal with it.  Int. J. Epidemiology, 34, 215-220. 

http://conium.org/~maccoun/PP279_Barnett.pdf

 

Quasi-Experiments

Shadish, Cook, & Campbell text, Chapters 4-7

OPTIONAL READINGS:

* Shadish, W. R., & Cook, T. D. (2008).  The renaissance of field experimentation in evaluating interventions.  Annual Review of Psychology.

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.60.110707.163544

*Mark W. Lipsey and David S. Cordray (2000). Evaluation Methods for Social Intervention.  Annual Review of Psychology, 51, 345-375. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.51.1.345

*Joseph P. Newhouse and Mark McClellan (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17-34.   http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.19.1.17

 

Dealing with Threats to Statistical Conclusion Validity

REQUIRED READINGS:

 

Rosnow, R. L., and Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284. http://conium.org/~maccoun/PP279_Rosnow.pdf


Cohen, Jacob  The earth is round (p < .05). American Psychologist. 1994 Dec Vol 49(12) 997-1003

http://conium.org/~maccoun/PP279_Cohen1.pdf

 

Cohen, J. (1992b). A power primer. Psychological Bulletin, 112, 155-159.

http://conium.org/~maccoun/PP279_Cohen2.pdf

 

 

OPTIONAL READINGS: 

*Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008).  Sample size planning for statistical power and accuracy in parameter estimation.  Annual Review of Psychology, 59, 537-563,

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.59.103006.093735

 

* Lenth R. (2001).  Some practical guidelines for effective sample size determination.  The American Statistician, 55, 187-193.

http://conium.org/~maccoun/PP279_Lenth.pdf  [NOTE: LENTH AND COHEN DISAGREE ABOUT SOME ISSUES; WE CAN DISCUSS THE POINTS OF DISAGREEMENT IN CLASS]

* Christopher Winship and Stephen L. Morgan (1999).  The estimation of causal effects from observational data Annu. Rev. Sociol., 25, 659-706.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.25.1.659

* Roderick J. Little and Donald B. Rubin (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121-145.  http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.21.1.121

Here are some useful web-based power calculators.  I recommend you verify your results using more than one calculator.  I also recommend that you create a table and solve for N under a variety of assumptions about alpha and effect size.  These links seem to come and go a lot; let me know if you find one that doesn't work, or if you find a good one that I haven't listed.  DO NOT confuse power calculators with survey sample size calculators – they answer different (but related) questions.

 

http://www.dssresearch.com/toolkit/spcalc/power.asp

 

http://home.clara.net/sisa/power.htm

 

http://www.stat.uiowa.edu/~rlenth/Power/index.html

 

http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/

 

http://statpages.org/

 

Dealing with Threats to External Validity

Shadish, Cook, & Campbell text, Chapters 13 and 14

 

 

OPTIONAL READINGS:

 

* Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181. 

http://conium.org/~maccoun/PP279_Schmidt.pdf

 

 * Hunter, John E.; Schmidt, Frank L., Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, & Law. 1996 Jun Vol 2(2) 324-347.

http://conium.org/~maccoun/PP279_Hunter.pdf


* R. Rosenthal and M. R. DiMatteo (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.52.1.59

 

 

Qualitative Methods

 

REQUIRED:

 

Shadish, W. R. (1995).  Philosophy of science and the quantitative-qualitative debates: Thirteen common errors.  Evaluation and Program Planning, 18, 63-75.

http://conium.org/~maccoun/PP279_Shadish.pdf

 

OPTIONAL:

*March, J. G., Sproull, L. S., & Tamuz, M. (2003).  Learning from samples of one or fewer.  Quality and Safety in Health Care, 12, 465-471.

http://conium.org/~maccoun/LearningfromSamplesofOne.pdf

 

* Morgan, G., & Smircich, L. (1980).  The case for qualitative research. The Academy of Management Review, 5, 491-500.

http://conium.org/~maccoun/PP279_Morgan.pdf

 

* David L. Morgan (1996). Focus groups. Annual Review of Sociology, 22, 129-152.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.22.1.129

 

PP279 SCHEDULE – FALL 2009

Week

Day

Date

Topic

 

1

 

 

 

 

 

Thurs

Aug 27

Course introduction

 

2

Tues

Sep 1

Philosophy vs. politics

 

 

Thurs

Sep 3

Asking questions

 

3

Tues

Sep 8

Asking questions

 

 

Thurs

Sep 10

Asking questions

 

4

Tues

Sep 15

Intro to psychometrics

 

 

Thurs

Sep 17

Measurement reliability

 

5

Tues

Sep 22

Measurement validity

 

 

Thurs

Sep 24

Measurement validity

1 page PR#1 preposal due

6

Tues

Sep 29

Survey sampling

 

 

Thurs

Oct 1

Survey sampling

HW due, bring to class

7

Tues

Oct 6

Special populations

 

 

Thurs

Oct 9

Special populations

 

8

Tues

Oct 8

Threats to internal validity

 

 

Thurs

Oct 15

Threats to internal validity

 

9

Tues

Oct 20

Experimentation

 

 

Thurs

Oct 22

Experimentation

PR#1 due Fri Oct 23, 5pm via email attachment

10

Tues

Oct 27

Quasi-experimentation

 

 

Thurs

Oct 29

Quasi-experimentation

 

11

Tues

Nov 3

Stat conclusion validity

 

 

Thurs

Nov 5

Stat conclusion validity

1-page PR#2 preposal due

12

Tues

Nov 10

Stat conclusion validity

 

 

Thurs

Nov 12

Threats to external validity

 

13

Tues

Nov 17

Threats to external validity

 

 

Thurs

Nov 19

Qualitative methods

 

14

Tues

Nov 24

See note*

 

 

Thurs

Nov 26

NO CLASS:  THANKSGIVING

 

15

Tues

Dec 1

See note*

 

 

Thurs

Dec 3

See note*

 

16

Tues

Dec 8

See note*

 

 

Thu

Dec 10

See note*

 

FinalsWk

 

 

 

PR#2 due Dec 19 by 5pm, via email attachment

 

 

 

 

This schedule may shift a bit depending on how class discussions go, but the order of topics will not change.

 

*NOTE:  I had planned to (a) cancel class on Tues Nov 24th and (b) add a session on Thu Dec 10th.  The idea was to give us four consecutive sessions after the Thanksgiving holiday to devote to interim briefings on the final team project.  (I knew this would cut into your final exam period, but we don't have a final exam scheduled and the briefing is a big part of putting together your final assignment.)  But now we have an added complication, which is that on 8/20 the campus administration announced that December 7-9 will be converted from class time to a "Reading/Review/Recitation" period.  So during the first day of class, we need to discuss how we will handle this.


Last revised on 8/26/09