dr. g. johnson, introduction to research methods research methods for public administrators dr....
DESCRIPTION
Dr. G. Johnson, In the News According to a 2009 study done for the American Health Insurance Plans by Price Waterhouse and Coopers: The average family health insurance coverage costs approximately $12,300 today Key finding: The overall impact of the proposals for health care reform will be to increase the cost of private insurance coverage for individuals, families, and businesses above what these costs would be in the absence of reform.TRANSCRIPT
Dr. G. Johnson, www.researchdemystified.org
1
Introduction to Research Methods
Research Methods for Public Administrators
Dr. Gail Johnson
Dr. G. Johnson, www.researchdemystified.org
2
Numbers and Facts
Any policy debate is awash in numbers that are presented as absolute fact.
But are they?Or are we being tricked into “premature
certainty” because numbers appear concrete?
Dr. G. Johnson, www.researchdemystified.org
3
In the News According to a 2009 study done for the American
Health Insurance Plans by Price Waterhouse and Coopers:
The average family health insurance coverage costs approximately $12,300 today
Key finding: The overall impact of the proposals for health care reform will be to increase the cost of private insurance coverage for individuals, families, and businesses above what these costs would be in the absence of reform.
Dr. G. Johnson, www.researchdemystified.org
4
Their Numbers
Health Insurance premiums could be expected to increase to approximately: $15,500 in 2013 under current law and to
$17,200 if these provisions are implemented. $18,400 in 2016 under current law and to
$21,300 if these provisions are implemented. $21,900 in 2019 under current law and to
$25,900 if these provisions are implemented.
Dr. G. Johnson, www.researchdemystified.org
5
Policy Debate What does this data mean in terms of
Congressional proposals to require everyone purchase health insurance?
What would you conclude based on this research result?
What do you want Congress to do based on these numbers?
What do you want Congress to do based on your beliefs?
Dr. G. Johnson, www.researchdemystified.org
6
Paradox
Research: the search for rationalityBut:“We must be aware that our knowledge and
experiences are finite, and always imperfect.”
Hitoshi Kume
Dr. G. Johnson, www.researchdemystified.org
7
Assessing Credibility
Requires Basic knowledge of research methods
Maintaining a research perspective Knowing how data was collected is necessary to help
assess whether the results are believable Critical thinking skills
A hint of detachment and skepticism Being aware of our beliefs yet setting them aside to
see things clearly
Dr. G. Johnson, www.researchdemystified.org
8
The Tough Questions
Is it true? How do you know? What is the logic of the argument? What is the evidence?
Remember: Much of what is presented as fact melts away
under scrutiny.Beware of premature certainty.
Dr. G. Johnson, www.researchdemystified.org
9
Critical Thinking Ask Yourself:
“How do I know that the things I believe are true?” “What would it take to convince me otherwise?”
The greatest challenge comes when looking at research that reflects what we already believe
Critical thinking challenges our most cherished beliefs
To see clearly, we must be aware of our beliefs while also remaining detached from them
Dr. G. Johnson, www.researchdemystified.org
10
The Role of Research in the Public Sector Guide policy Target programs Find out what works and what doesn’t Convince funders Provide feedback from customers Help plan interventions Advocate for change
Dr. G. Johnson, www.researchdemystified.org
11
Research in the Public Sector
The Intersection of Data, Values and Beliefs and Politics
Does welfare help or hurt people? Should oil companies be prevented from
earning too much profit? Is investment in primary prevention more
effective in reducing crime than spending on law enforcement?
Dr. G. Johnson, www.researchdemystified.org
12
Different Research Approaches
Qualitative: Stories Observations No numbers
Quantitative: What you can count.
Dr. G. Johnson, www.researchdemystified.org
13
Qualitative Data: Stories
AnecdotesExample: Michael Moore’s “Sicko”
Example: Barbara Erhen “Nickled and Dimed”
Paints a picture, makes it feel real Limited in scope:
Is this a small or large problem?
Dr. G. Johnson, www.researchdemystified.org
14
Quantitative Data: Numbers
Percents, Counts, Correlations Example: Study: U.S. Ranking for Preventable
Deaths: U.S. lowest of industrialized countriesBut need to check how they defined and
measured preventable deaths
Dr. G. Johnson, www.researchdemystified.org
15
Quantitative Data: Numbers
Be aware that some things are not really knowable—so how did they come up with the numbers? Number of illegal immigrants Amount of money spent on illegal drugs 10-year projection of annual U.S. budget deficits (even
1-year is not precise) The true cost of mandatory health insurance Percent of carbon from raising animals
Dr. G. Johnson, www.researchdemystified.org
16
As An Advocate
Which would be more effective?
Voice your concerns about a problem? Or
Show data documenting the problem?
Dr. G. Johnson, www.researchdemystified.org
17
As a Citizen
What do you base decisions on? Data? Emotional appeals? Personal opinion? Political demands?
Do some work better than others in different situations?
Dr. G. Johnson, www.researchdemystified.org
18
As a Decision Maker
What do you base decisions on? Data? Emotional appeals? Personal opinion? Political demands?
Do some work better than others in different situations?
Dr. G. Johnson, www.researchdemystified.org
19
The Tough Question
Generally, we prefer data, facts, hard evidence
But the question is: How good is the research? Meaning: does the research yield accurate,
reliable, valid, and unbiased information?
Dr. G. Johnson, www.researchdemystified.org
20
What is Research?
A systematic search for answers to questions.
Search: to uncover, examine, find by exploration, to investigate, to inquire.
Research: "the systematic inquiry into a subject in order to discover or revise facts, theories
Dr. G. Johnson, www.researchdemystified.org
21
What is Research?
Research: Root of work means “to know.”Re---Search:
conveys the idea of searching again, from a different perspective, using different approaches, looking at different data.
Dr. G. Johnson, www.researchdemystified.org
22
What is Research?
Empirical: derived from experience or experiment; observation and experience.
Science: "a branch of knowledge or study dealing with a body of facts or truths systematically arranged and showing the operation of general laws; systematic knowledge"
Dr. G. Johnson, www.researchdemystified.org
23
This Thing Called Science
Scientific Method Curiosity Systematic observation Systematic experimentation
Theories and Hypotheses: Used to Explain Relationships
Dr. G. Johnson, www.researchdemystified.org
24
Different Goals
Scientists: More interested in creating knowledge Not concerned about utility or application of
the results Practitioners:
More interested in application and problem solving
If the research isn’t useful, then so what?
Dr. G. Johnson, www.researchdemystified.org
25
Research Process: Simplified 3-step Plan Do Report
Dr. G. Johnson, www.researchdemystified.org
26
Research Planning Process
Determine the issues Decide on the research question(s) Select measures Identify the ideal design given the type of
research question Develop data collection methods and
instruments
Dr. G. Johnson, www.researchdemystified.org
27
Research Planning Process
Decide on sampling frame and strategy Develop analysis plan Articulate your entire research plan Review research plan and test all data
collection instruments Prepare work plan with resource and time
requirements
Dr. G. Johnson, www.researchdemystified.org
28
Planning Matters
When people think about research, they tend to immediately focus on statistics
But no amount of statistical wizardry can save mistakes made in the planning process
Planning is harder and takes longer than most people expect
Dr. G. Johnson, www.researchdemystified.org
29
Doing Research
Gathering the data Preparing data for analysis Analyzing and interpreting the data
Dr. G. Johnson, www.researchdemystified.org
30
Reporting
Executive Summary Reports Charts and Tables Oral Briefings
Dr. G. Johnson, www.researchdemystified.org
31
Research Planning Process
Presented as linear but in practice, it is not The word is iterative: meaning researchers make
some initial assumptions about how to proceed and then go back and make changes to the plan as new information shows that their initial plans will not work
There is a lot of back and forth before it all clicks into place (and researchers live with the limitations of the situation)
Dr. G. Johnson, www.researchdemystified.org
32
Ethical Research Do No Harm
Protect participants from being harmed by the research Guard the confidentiality of participants Do not coerce people to participants: they must freely
consent No one should be denied benefits to which they are
entitled because they refuse to participate in research project
Do not quickly conclude a program does not work just because there does not appear to be an impact
Dr. G. Johnson, www.researchdemystified.org
33
Objective: Strive to See Clearly
Objectivity is a worthy goal even if not 100% attainable Think in terms of being free of ideological or
political blinders or desired outcomes Be aware of biases
Researchers should state biases and build in quality controls to minimize bias
Quality Control: Have others with different perspectives review the research design, analysis and results.
Dr. G. Johnson, www.researchdemystified.org
34
Principles of Good Research
Honest Do not lie Do not distort or “spin” the data Fully disclose methods, definitions,
assumptions, biases Fully disclose limitations of the research and
implications for making any conclusions
Dr. G. Johnson, www.researchdemystified.org
35
Principles of Good Research
Accurate: Strive to be error free Build quality control procedures into data
collection, analysis, interpretation, and written product
Verify data entry and analysis. Have someone check your numbers.
Dr. G. Johnson, www.researchdemystified.org
36
Principles of Good Research
Technically Correct Use appropriate designs, data collection
methods, analysis, statistics and charts The right data collection instruments. The right statistics.
Dr. G. Johnson, www.researchdemystified.org
37
Principles of Good Research
Reliable and Valid Measures Use reliable and valid measures Measure what matters using systematic
approaches with fixed measurement rules Ask: do they measures actually measure what
they claim to measure? Are they measuring reported crime but claiming to
be measuring all crime?
Dr. G. Johnson, www.researchdemystified.org
38
Research Flaws
Perfection is not the standard to assess the credibility of research results. All research is flawed, so the trick is to
distinguish minor flaws from major ones. Skill: to assess the relative strengths and
limitations of research to determine the credibility of the results.
Dr. G. Johnson, www.researchdemystified.org
39
Research Flaws
It means working in the gray area. If the research is the best given the
circumstances, flawed though it may be, it should be considered.
However, decisions “to do something” based on that data should be made with caution, and implemented incrementally with built-in feedback.
Dr. G. Johnson, www.researchdemystified.org
40
Seriously Flawed Research
While you might not want to make decisions based on seriously flawed research, it still may actually be correct, provide useful insights, or some guidance about how to do the next study. It might provoke important conversations.
Remember: perfection is not a useful standard.
Dr. G. Johnson, www.researchdemystified.org
41
Takeaway Lesson
Sophisticated Users of Research Question Numbers!“Many a statistic is false on its face. It gets by only because the magic of numbers brings out a suspension of common sense."
---How to Lie With Statistics, Huff, p. 138.
Dr. G. Johnson, www.researchdemystified.org
42
Ask Tough Questions
Does the researcher have axe to grind?Particular political ideology? Desire to
get more funding? Who was included and excluded in the
study? Is the data really knowable?
Number of actual crimes is not the same as the number of reported crimes.
Dr. G. Johnson, www.researchdemystified.org
43
Ask Tough Questions
Is reported behavior based on self-reports?How many people used illegal drugs in
the past week? Does the research make a giant leap to
conclusion? Did a study with all male participants but
generalizes to everyone?
Dr. G. Johnson, www.researchdemystified.org
44
Creative Commons
This powerpoint is meant to be used and shared with attribution
Please provide feedback If you make changes, please share
freely and send me a copy of changes: [email protected]
Visit www.creativecommons.org for more information