PP279: RESEARCH DESIGN AND DATA COLLECTION

PP279: RESEARCH DESIGN AND DATA COLLECTION

FOR PUBLIC POLICY ANALYSIS (CCN 77256)

Professor Robert MacCoun

Goldman School of Public Policy, 2607 Hearst Ave.

642-7518, maccoun@ berkeley.edu

Fall 2010: Tuesdays and Thursdays, 2-3:30, 105 GSPP (the old bldg.)

(Office Hours: right after class or Mondays @ 11-noon)

The online version of this syllabus is at http://conium.org/~maccoun/pp279_f10.html
Please see the online version for the most up-to-date version; I will announce any revisions in class.

COURSE DESCRIPTION

Empirical arguments and counterarguments play a central role in policy debates, thus public policy analysis requires a sophisticated understanding of a variety of types and sources of data. Quantitative analysis courses teach you how to analyze data; this course will introduce you to strategies of data collection and principles for critically evaluating data collected by others. Topics include measurement reliability and validity, questionnaire design, sampling, experimental and quasi-experimental program evaluation designs, qualitative research methods, and the politics of data in public policy.

READINGS

DeVellis, R. F. (2003). Scale Development: Theory and Applications. Sage.

Groves, Fowler, Couper, Lepkowski, Singer, and Tourangeau (2009). Survey Methodology (2^nd ed.). Wiley.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs For Generalized Causal Inference. Houghton Mifflin. [If you are interested, my review of this book appears here: http://conium.org/~maccoun/JPAM_2003_BookRev.pdf.]

Many of the additional web-based readings are marked with * to indicate that they are optional.

If a link is bad, email me (maccoun@berkeley.edu) and I’ll try to fix it ASAP. But every reading (except my handouts) is available on the web – that’s where I found them! – and you should be able to hunt them down yourself (e.g., using Google Scholar (http://scholar.google.com/). The ANNUAL REVIEW essays are available online but only from a UC computer account because we have a site license.

ASSIGNMENTS (see due dates in Schedule at end of syllabus)

Team assignment on psychometrics, to be completed during a class session (see Schedule) by your Proposal 1 Team (Around 4-5 pg., worth 20%) (Download the excel workbook here: http://conium.org/~maccoun/PP279_homework.xls)

Two draft research proposals: Each worth 40%. Like most research proposals, these will be developed in small groups of 2-3 people (size will depend on class enrollment). Teams will give in-class briefings on the second proposals at the end of the term.

1) Survey proposal (http://conium.org/~maccoun/PP279_proposal1.html)

2) Program evaluation proposal/group briefing: http://conium.org/~maccoun/PP279_proposal2.html

READINGS - Schedule appears on last page

NOTE: The readings are NOT in a reader; they are online at http://conium.org/~maccoun/pp279_f10.html

The Philosophy of Science and the Politics of Data (First Look)

MacCoun, R. (1998). Biases in the interpretation and use of research results, Annual Review of Psychology, 49, 259-287. http://conium.org/~maccoun/MacCoun_AnnualReview98.pdf

MacCoun R. J., & Paletz, S. (2009). Citizens’ perceptions of ideological bias in research on public policy controversies. Political Psychology, 30, 43-65. [Note: During the editing process, we accidentally used a formatting command that put negative signs on all the standard errors while moving them into parentheses. Oops! Standard errors can only take on a positive value, and the absolute values of the displayed standard errors are all correct.] http://conium.org/~maccoun/MacCounPaletz2009PoliPsy.pdf

Describing the World: Surveys and Other Measures

Asking Questions

Chapters 7, 8, and 9 of Survey Methodology text

OPTIONAL READINGS:

*Krosnick, Jon A. (1999). Survey research. Annual Review of Psychology, 50, 537-567. http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.50.1.537

*Schaeffer, Nora Cate, & Presser, Stanley (2003). The science of asking questions. Annual Review of Sociology Aug 2003, Vol. 29: 65-88. http://conium.org/~maccoun/PP279_SchaefferPresser2004.pdf

Reliability and Validity (basic psychometrics)

First, skim the entire Scale Development book to get a general feel for the topic. Then take a second pass through, reading Chapters 2 through 6 more carefully. These chapters are directly relevant to the homework and to the Proposal 1 assignment.

Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
http://conium.org/~maccoun/PP279_Schmitt.pdf

Rob’s memo on coefficient alpha:

http://conium.org/~maccoun/CoefAlpha.pdf

Rob’s memo on how low reliability weakens the ability to detect relationships (e.g., program effects)

http://conium.org/~maccoun/PredictiveValidity.pdf

OPTIONAL READINGS (can be found using Google Scholar):

*Neisser, U., et al. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.

http://conium.org/~maccoun/PP279_Neisser1.pdf

and his replies to critics:

http://conium.org/~maccoun/PP279_Neisser2.html

*Flynn, J. R. (1999). Searching for justice: The discovery of IQ gains over time. American Psychologist, 54, 5-20.

http://conium.org/~maccoun/PP279_Flynn.pdf

*Kuncel, N. R., Hezlett, S. A., Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162-181.
http://conium.org/~maccoun/PP279_GRE.pdf

*Lubinski, D. (2000). Scientific and social significance of assessing individual differences: "Sinking shafts at a few critical points." Annual Review of Psychology, 51, 405-444.

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.51.1.405

Survey Sampling

Chapters 1 through 6 of Survey Methodology.

Rob’s memo on computing sample sizes. http://conium.org/~maccoun/pp279_samplesize.pdf

OPTIONAL READINGS:

*Magnani, R., Sabin, K., Saidel, T., & Heckathorn, D. (2005). Review of sampling hard-to-reach and hidden populations for HIV surveillance. AIDS 2005, 19, S67-S72. [This has a good discussion of a sophisticated version of snowball sampling]

http://conium.org/~maccoun/PP279_Magnani.pdf

*Birnbaum, Michael H. (2004). Human research and data collection via the internet. Annual Review of Psychology, 55, 803-832.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.55.090902.141601

* Marshall, G.N., Burnam, M. A., Koegel, P., Sullivan, G., & Benjamin B. (1996). Objective Life Circumstances and Life Satisfaction: Results from the Course of Homelessness Study. Journal of Health and Social Behavior, 37, 44-58. [Note: This paper has a very interesting sampling strategy for a difficult to sample population; it also has a nice example of multiple-indicator measurement of latent constructs. And two of the authors attended my wedding!]

http://conium.org/~maccoun/PP279_Marshall.pdf

Inferring Cause and Effect: Experimental and Quasi-Experimental Design

Dealing with Threats to Internal Validity

Shadish, Cook, & Campbell text, Chapter 1, pp. 53-62 of Chapter 2, and Chapter 8.

Falk, A., & Heckman, J. J. (2009). Lab experiments are a major source of knowledge in the social sciences. Science, 326, 535-539. http://conium.org/~maccoun/FalkHeckman2009.pdf

OPTIONAL READINGS:

* Brady, H. E. (2008). Causation and explanation in social science. In the Oxford Handbook of Political Science.

* Greenland, S., & Morgenstern, H. (2001). Confounding in health research. Annual Review of Public Health, 22, 189-212. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.22.1.189

* Barnett, A. G. , Van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: What it is and how to deal with it. Int. J. Epidemiology, 34, 215-220. http://conium.org/~maccoun/PP279_Barnett.pdf

Quasi-Experiments

Shadish, Cook, & Campbell text, Chapters 4-7

OPTIONAL READINGS:

* Shadish, W. R., & Cook, T. D. (2008). The renaissance of field experimentation in evaluating interventions. Annual Review of Psychology. http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.60.110707.163544

* Lipsey, M. W., & Cordray, D. S. (2000). Evaluation Methods for Social Intervention. Annual Review of Psychology, 51, 345-375. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.51.1.345

*Newhouse, J. P., & McClellan, M. (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17-34. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.19.1.17

Dealing with Threats to Statistical Conclusion Validity

REQUIRED READINGS:

Rosnow, R. L., and Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284. http://conium.org/~maccoun/PP279_Rosnow.pdf

Cohen, Jacob The earth is round (p < .05). American Psychologist, 49, 997-1003.

http://conium.org/~maccoun/PP279_Cohen1.pdf

Cohen, J. (1992b). A power primer. Psychological Bulletin, 112, 155-159.

http://conium.org/~maccoun/PP279_Cohen2.pdf

OPTIONAL READINGS:

*Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537-563,

http://arjournals.annualreviews.org/doi/pdf/10.1146/annurev.psych.59.103006.093735

* Lenth R. (2001). Some practical guidelines for effective sample size determination. The American Statistician, 55, 187-193.

http://conium.org/~maccoun/PP279_Lenth.pdf [NOTE: LENTH AND COHEN DISAGREE ABOUT SOME ISSUES; WE CAN DISCUSS THE POINTS OF DISAGREEMENT IN CLASS]

* Christopher Winship and Stephen L. Morgan (1999). The estimation of causal effects from observational data Annu. Rev. Sociol., 25, 659-706. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.25.1.659

* Roderick J. Little and Donald B. Rubin (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121-145. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.publhealth.21.1.121

There are many web-based power calculators; I've stopped posting links because they change so often. (It is also possible, but less easy, to do power calculations in R, Stata, and other packages.) I recommend you verify your results using more than one power calculator. I also recommend that you create a table and solve for N under a variety of assumptions about alpha and effect size. These links seem to come and go a lot; let me know if you find one that doesn't work, or if you find a good one that I haven't listed. DO NOT confuse power calculations/calculators with survey sample size calculations/calculators – they answer different (but related) questions.

Dealing with Threats to External Validity

Shadish, Cook, & Campbell text, Chapters 13 and 14

OPTIONAL READINGS:

* Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.

http://conium.org/~maccoun/PP279_Schmidt.pdf

*Hunter, J. E., & Schmidt, F. L. (1996). Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, & Law, 2, 324-347.

http://conium.org/~maccoun/PP279_Hunter.pdf

*Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. http://arjournals.annualreviews.org/doi/full/10.1146/annurev.psych.52.1.59

Qualitative Methods

REQUIRED:

Shadish, W. R. (1995). Philosophy of science and the quantitative-qualitative debates: Thirteen common errors. Evaluation and Program Planning, 18, 63-75.

http://conium.org/~maccoun/PP279_Shadish.pdf

OPTIONAL:

*March, J. G., Sproull, L. S., & Tamuz, M. (2003). Learning from samples of one or fewer. Quality and Safety in Health Care, 12, 465-471.

http://conium.org/~maccoun/LearningfromSamplesofOne.pdf

*Morgan, G., & Smircich, L. (1980). The case for qualitative research. The Academy of Management Review, 5, 491-500.

http://conium.org/~maccoun/PP279_Morgan.pdf

*Morgan, D. L. (1996). Focus groups. Annual Review of Sociology, 22, 129-152.

http://arjournals.annualreviews.org/doi/full/10.1146/annurev.soc.22.1.129

PP279 SCHEDULE – FALL 2010 (REVISED)

Thu Aug 26	Course introduction
Tue Aug 31	Cognitive and motivational biases in interpreting data
Thu Sep 2	Asking questions
Tue Sep 7	Asking questions
Thu Sep 9	Intro to psychometrics
Tue Sep 14	Measurement reliability
Thu Sep 16	Measurement validity
Tue Sep 21	Measurement validity	*1 page PR#1 preposal due*
Thu Sep 22	IN-CLASS EXERCISE
Tue Sep 28	Survey sampling
Thu Sep 30	Survey sampling
Tue Oct 5	Special populations
Thu Oct 7	Special populations
Tue Oct 12	Threats to internal validity
Thu Oct 14	Threats to internal validity
Tue Oct 19	Experimentation
Thu Oct 21	Experimentation	*PR#1 due Fri Oct 21@5pm via email attachment*
Tue Oct 26	Quasi-experimentation
Thu Oct 28	Quasi-experimentation
Tue Nov 2	Stat conclusion validity	*1-page PR#2 preposal due*
Thu Nov 4	Stat conclusion validity
Tue Nov 9	Stat conclusion validity
Thu Nov 11	NO CLASS:VETERAN'S DAY
Tue Nov 16	Threats to external validity
Thu Nov 18	Threats to external validity
Tue Nov 23	Qualitative methods
Thu Nov 25	NO CLASS: THANKSGIVING
Tue Nov 30	Team briefings*
Thu Dec 3	Team briefings*
TBA	Team briefings*
TBA	Team briefings*	*PR#2 due Wed Dec 15@5pm via email attachment*

* Team briefing sessions usually involve around 4 teams, with time slots determined by random assignment. You will be required to come to your entire session, and to ask questions and offer feedback to the other teams in your session, and you are encouraged to come to the other sessions as well.

Last revised on 10/27/10