validity + reliability

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Jonathan HallettEvidence + Effectiveness in Health PromotionVALIDITY + RELIABILITY


Lecture Overview

Revisit questionnaires

Reliability + validity

Methods for ensuring reliability + validity

Assessment check-in

Evidence and Effectiveness in Health Promotion


Friendly Schools, Friendly Families

The primary outcome of FSFF was the prevalence of being bullied at the end of the three year intervention trial.

The purpose of FSFF was to assess the effectiveness of a parent and whole school bullying intervention, that includes policy review and enhancement, staff training, learning activities and parent and community involvement, aimed at preventing, reducing and managing bullying in primary schools.

Based on the Health Promoting Schools concept



Friendly Schools, Friendly Families

A three year group randomised control intervention trial

20 Perth metropolitan primary schools

Schools randomly allocated to high, moderate and low intervention groups

Followed cohorts of Year 2, 4 and 6 students and their teachers and parents



FSFF Research Objectives

Some of the objectives included:

Reduce reported frequency of children’s unresolved conflict/bullying

Improve children’s perceptions of positive outcome expectancies should they become involved in a bullying situation



Activity – Find questions that measure:

Prevalence of being bullied

Prevalence of bullying others

Telling someone about bullying (not parents)

Perception of prevalence of bullying at school



Answers

Prevalence of being bullied 7, 8, 9, 10, 11

Prevalence of bullying others 23, 24

Telling someone about bullying (not parents)

12, 13, 14, 15, 34g

Perception of prevalence of bullying at school

21, 22



Class Activity

Developing a Participant Satisfaction Sheet

The following page provides an example of a participation satisfaction sheet for a session conducted on responsible drinking.

- What are some noticeable features about this questionnaire? (e.g. size, layout, types of questions).

- What changes do you think you would make to improve on this feedback sheet if you were conducting a workshop on drinking behaviour?



Group activity

Review your case study

Write three SMART objectives

Develop a questionnaire with at least three different formats of questions (Likert, dichotomous, multi-choice etc.)

Identify the most appropriate mode of survey

Consider ethical issues related to your data collection



Reliability and Validity

What do we mean when we say something is reliable?

What about if something is valid?



Reliability + Validity

Reliability - does the questionnaire measure yield the same results on repeated trials? (stability - need this for validity)

Validity - does the questionnaire measure what you are actually trying to measure? (relationship between measure and indicator)



Reliability

The extent to which a measurement instrument is dependable, stable and consistent when repeated under identical conditions.

A reliable instrument will ascertain the same result on two or more occasions.



Types of reliability

Test-retest reliability describes the extent to which a stable evaluation of the attribute or behaviour can be obtained on two different occasions when no change is expected

Intra-rater reliability describes the extent to which the same person can rate the same performance consistently

Inter-rater reliability is the extent to which different people rate the same performance consistently



Sources of reliability error

Intra-rater reliability is higher than inter-rater reliability

For self-rated measures (self-report), intra- and inter-rater reliability measures are not relevant, so test-retest reliability is important



Testing Reliability

Observer dependent instrument- Inter-rater reliability (when measure applied only once each by different

observers)

- Intra-rater reliability (when measure taken more than once by same observer)

Self-report instrument- Test-retest reliability (most efficient method – give same measure to same

group at two points in time [about 1 to 2 weeks apart]; correlate results

- Split-half reliability (divide instrument questions in half; sum the scores and correlate the two results



Test-retest Method

Instrument administered to same participants twice (two weeks apart) under similar conditions.

Problems:- Non-independent respondents- Remember the answers- Testing effect- Take less effort the second time around- Change in characteristic itself, e.g. ever smoked vs number of

cigarettes in last seven days



Split-half Method

The items on the scale are divided into two matched halves and the scores for the two halves are correlated

Items need to be compatible / comparable



Split-half Method

1. On how many days in the last week did you drink alcohol?

2. On average, how many drinks did you consume when you drunk alcohol?



Split-half Method

3. Starting yesterday, going backwards through the last week, how many alcoholic drinks (minutes of exercise) did you consume on each day?

Monday __

Saturday __

Sunday __

Friday __ Thursday __

Wednesday __

Tuesday __



Validity

The degree to which a scale measures what it is intended to measure



Measurement Validity

Face validity

Content validity

Construct validity

Criterion validity

Predictive validity



Types of validity

Content validity refers to the extent to which an instrument covers all the important aspects or domains of interest that it intends to measure, and does not include items that are irrelevant.

Face validity indicates that an instrument appears to evaluate what it is supposed to and seems plausible to the users of the tool (evaluators and respondents)



Types of validity

Construct validity refers to the ability of an instrument to measure an abstract concept. Establishing this involves:

1.Explicitly describing the theoretical concepts and how they relate to each other.

2.Developing an assessment too that can measure this.

3.Testing the relationship between the constructs and the observed behaviours that it represents.



Types of validity

Criterion validity concerns the correlation of one tool with another, ideally a ‘gold standard’ that has been widely used and accepted in the field of interest.

But is there a ‘gold standard’?



Types of validity

Predictive validity is established by evaluating performance using the instrument at a specified time-point (e.g. baseline) and following the participants over time until the outcome to be predicted occurs. These studies determine the accuracy of the measurement tool to predict future outcomes.



Instrumentation processes which reduce validity1. Response sets. For example: tendency of respondents to

agree (pleasers) or tendency towards socially acceptable answers.

2. Interviews. For example: interviewer contamination – differential emphasis on questions or respondent may know objectives of study and answer accordingly.

3. Content validity – always a factor because no instrument can cover all aspects of a problem.



Improving Reliability + Validity

Having colleagues and experts review and make suggestions to the survey.

Interview a few participants as they are completing a survey which involves asking them to verbalise how they are answering the question. This will provide some indication of whether the questions are being interpreted as designed.

Conduct a pilot test of the questionnaire on a smaller sample

Have the survey proof read by an individual(s) unrelated to the project.



29

Valid / Reliable Questions

Valid and reliable results need valid and reliable approaches.

Minimum requirements:- Target group / community participation in face & content validity

- Test/retest approaches for reliability



Triangulation

Method of cross-checking data from multiple sources:

Methodological triangulation: use of more than one method of data collection (e.g. self-report survey and observation)

Investigator triangulation: use of more than one evaluator/researcher

Data triangulation: multiple sources of data (e.g. collecting data over different times and from different people)

Theory triangulation: multiple theories in interpreting phenomenon



Next Week

Focus Groups


validity + reliability

Documents

bullied prevalence of

j friendly schools

bullying situation evidence

j answers prevalence

school bullying intervention

j activity

school evidence

parents evidence