Data Collection andReliability

All this data,

but can I really count on it??

Agenda

• Data collection – review and continue

• Reliability • What is it?• How do we achieve?

Questionnaires

• Controlling problems• Equal numbers of positively and negatively

worded statements• Alternating positive and negative statements • Providing confidentiality or anonymity to

respondents

Designing Questionnaires

• Online resources• www.surveymonkey.com• http://pareonline.net/getvn.asp?v=5&n=3• http://www.peecworks.org/PEEC/PEEC_Inst/I0

004E536• http://www.statpac.com/surveys/

Questionnaires

• Take the following questionnaire

http://www.surveymonkey.com/s/PNBKFJ7

• Strengths? Weaknesses?

Observations

• Observations - direct observations of behaviors• Provide first hand account (ameliorates issues

of self-reporting in questionnaires)• Natural or controlled settings

• Ex – classroom vs. lab (child attachment studies)

• Structured or unstructured observations• Ex – frequency counts vs. narrative record

• Detached or involved observers

Observations

• Inference• Low inference - involves little if any inference

on the observers part• On-task/Off-task behavior instrument

• High inference - involves high levels of inference on the observers part

• Teacher effectiveness – PDE form 430

Observations

• Controlling observer effects• Observer bias

• Training• Inter-rater reliability (Cronbach’s alpha)• Multiple observers

• Contamination - knowledge of the study influences the observation

• Training• Targeting specific behaviors• Observers do not know of the expected outcomes• Observers are “blind” to which group is which

Observations

• Observer effects• Halo effectHalo effect - initial ratings influence subsequent

ratings

• Hawthorne effectHawthorne effect - increased performance results from awareness of being part of study

• LeniencyLeniency - wanting everyone to do well

• Central TendencyCentral Tendency - measuring in the middle

• Observer DriftObserver Drift - failing to record pertinent information

Examples to critique

• Measures• Questionnaire – Psychological School

Membership Survey used with middle school students

• Interview protocol – for teachers & counselors regarding professional development issues

• Observation instrument – PDE 430 for student teachers

• What are 2 benefits and 2 limitations of this measure?

Validity/Reliability and Trustworthiness

• Why do we need validity and reliability in quantitative studies and “trustworthiness” in qualitative studies?

We can’t trust the results if we can’t trust the

methods!

Thought Question

• On the ACT and SAT assessments, there is a definitive script that test administrators are required to follow exactly. What measurement issue are the test makers addressing?

Reliability of Measurement

• Reliability - The extent to which measures are free from error

• Error is measured by consistency

Reliability of Measurement

• Sources of error• Test construction and administration

• Ambiguous questions, confusing directions, changes in scoring, interrupted testing, etc.

• Subject’s characteristics• Test anxiety, lack of motivation, fatigue,

guessing, etc.

Reliability of Measurement

• Reliability• Measurement

• 0.00 indicates no reliability or consistency• 1.00 indicates total reliability or consistency• < .60 = weak reliability• > .80 = sufficient reliability

Reliability of Measurement

• Types of reliability evidence• Test-retest

• Testing the same subject using the same test on two occasions

• Limitation - carryover effects from the first to second administration of the test

• Equivalence (i.e. parallel form)• Testing the same subject with two parallel (i.e. equal)

forms of the same test taken at the same time• Limitation - difficulty in creating parallel forms

Reliability of Measurement

• Internal consistency• Testing the same subject with one test and

“artificially” splitting the test into two halves

• Limitations - must have a minimum of ten (10) questions

• Often see “Chronbach’s alpha” for reliability coefficient

Reliability of Measurement

• Agreement/ Inter-rater reliability• Observational measures• Multiple observers coding similarly

Reliability of Measurement

• Enhancing reliability• Standardized administration procedures

(e.g. directions, conditions, etc.)• Appropriate reading level• Reasonable length of the testing period• Counterbalancing the order of testing if

several tests are being given

Next Week

• Tuesday – Validity and Review• Thursday – Opportunity to display brilliance!

• Multiple choice

• Short Answer

• Essay – TAKE HOME. Critique Triplett article (see next slide).

Take Home Exam

Critique the article based on the following:• Introduction and research problem• Review of literature/ theoretical framework• Methods of data collection (including participants)

and data analysis• Results and conclusions including issues of

trustworthiness. Be sure to address whether we should trust the claims that the authors have made and why we should or should not trust the claims.