patient reported outcome measures (proms) for …pediatric diabetes 11, 364–375cooper h., cooper...

Patient Reported Outcome

Measures (PROMs) for use

with children and adolescents:

a view across disciplines

Dr Gillian Lancaster

Postgraduate Statistics Centre

Lancaster University, UK [email protected]

Funded by Diabetes UK

Supported by NIHR Research Network: Children

Acknowledgments

Principal Investigator: Prof Helen Cooper

Research Associates: Joy Spencer, Sara Wheeler Department of Community Health and Wellbeing, University of Chester

Statisticians: Gill Lancaster, Andrew Titman Department of Mathematics and Statistics, Lancaster University

IT Specialist: Mark Johnson Dept of Games Computing/Creative Technologies, University of Bolton

Clinical Psychologist: Rebekah Lwin Alder Hey Children’s NHS Trust and University of Chester

DIFFERENT by Clere Parsons (1908-31)

“Not to say what everyone else was saying

not to believe what everyone else believed

not to do what everybody did,

then to refute what everyone else was saying

then to disprove what everyone else believed

then to deprecate what everybody did,

was his way to come by understanding

how everyone else was saying the same as he was saying

believing what he believed

and did what doing”.

-ICT improved knowledge, psychosocial well-being and self-care skills, with trends toward improving glycaemic control.

4

3

1Cooper et al (2009) Pediatric Diabetes; 2Spencer, Cooper & Milton (2009) Pediatric Diabetes; 3Spencer, Cooper & Milton (2013). Diabetic Medicine; 4Spencer & Cooper (2011) Practical Diabetes.

Adolescent Diabetes Management: Complex Adaptive System (CAS)

Blood Glucose Friends

Puberty

Exercise

Family

Alcohol

School

Emotions

Body image

Blood glucose tests

Lifestyle

Health &

illness

Independence &

autonomy

Insulin Diet

Friends Family

Alcohol Emotions

Blood glucose tests

Insulin Diet

Successive UK National Paediatric DiabeBlood Gtes lucoseAudits show (here for 2013-14):

Only Independence 15.8% achieved recomme& nded glycaemic targets, with 25.9% at high risk of future complicaautonomytions Puberty

90% of glycaemic variability is due to service related factors, inc. standards Body imageand delivery of diabetes self-care Lifestyleeducation, only 45.2% receiving structured education

27.5% Exerciseof those aged 12 or over had high blood Schoolpressure, > 7% had excess protein in their urine, >14% had early signs of eye disease, and nearly 1 in 4 was obese Health &

Just 16% received all 7 of the recommended health care processes illness

Adolescent Diabetes Management: Complex Adaptive System (CAS)

5

COMPLEXITY SCIENCE/EXPERIENTIAL LEARNING/CHANGE CYCLE

Cooper & Geyer. (2007) Riding the Diabetes Roller Coaster: Oxford: Radcliffe Publishing.

Cooper & Geyer (2008) Social Science and Medicine.

Fitness Landscape

Troughs Poor fitness

(poor control)

Peaks

High fitness

(good control)

In-between Neutral fitness (mod. Control)

Designed to model

chemical & species

interactions as

evolutionary processes

Basic rules for survival

on a fitness landscape:

– Adaptability

– Flexibility

– Learning

– Balance

Gill T.G. (2008) Reflections on Researching the Rugged Fitness Landscape, Int. Jl. Emerging Transdiscipline, vol.11.

Study Aims • To create a Diabetes Fitness Landscape

• Primary goal = adaptation + balance

• To develop capability to manage condition

where

• Main participants are the patients/families

• Experts are helpers on their journeys

• No endpoint to learning

• Mistakes are a learning tool i.e. learning from

experience is key

Paediatric Diabetes Education*

Age and maturity appropriate programmes

Individual Needs Assessment → Tailored Diabetes Self Management Education

Onset of DM

Paediatric long term care

< 5 years, primary, adolescents

Adult

long term care

*Adapted from a slide by Sheridan Waldron

SE

D

EV

EL

OP

ME

NT

PH

A

TESTING PHASE

RESPONDANT VALIDATION PHASE

SE

D

EV

EL

OP

ME

NT

PH

A

70 items

7 domains

109 items

8 domains

112 items

7 domains

194 items

6 domains

129 items

6 domains

TESTING PHASE

117 items

6 domains

RESPONDANT VALIDATION PHASE

Study participants

Sample: • Mersey Region Children's Diabetes Network - 4 paediatric diabetes

centres: Alder Hey, Chester, Leighton and Whiston

Inclusion criteria: • Age 12-18 years

• Able to read and write

• Minimum diabetes duration of 1 year

• Ability to complete a questionnaire unaided

• Computer literate

Exclusion criteria: • Significant medical disease in addition to diabetes

• Diagnosed psychiatric disorder or neuro-cognitive disorder

• Drug and/or alcohol dependency

Data Analysis

Pilot study (n=52) graphs to assess item performance, HbA1c

Main study (n=155) • Item performance eg. graphs, levels of missing data, skewness of

responses (low and high response rates)

• Devised 3-point Likert scale to give traffic-light feedback (red, amber, green), simple signposting

Reliability • Internal consistency – item total correlations

• Test-retest reliability in subgroup of young people (kappa statistics)

Validity • Concurrent - correlation with SMOD-A tool domains

• Convergent – correlation with HbA1c

• Item response analysis to validate additive score

• Respondent – focus groups at end of study

ADNAT: 117 questions divided between 6 domains

No. of quests. answered depends on insulin regime and lifestyle choices

Questions prompt self-awareness: education focus

Assessment of core needs: 20 questions = overall self-care score & 16 self-

perception questions = self-evaluation of psycho-social health

Positively evaluated for appropriateness & readability.

Tests for validity: SMOD-A* (r = -0.41, p < 0.001); weak correlation with

HbA1c (r = -0.16, p=0.056) but this compared favourably with SMOD-A (r= -

0.06, p=0.52).

Item Response Analysis validated use of a simple additive score

ADNAT DOMAINS 1. ALL ABOUT ME 2. PHYSICAL ACTIVITY 3. EATING 4. MEDICATION

5. BLOOD GLUCOSE MONITORING 6. LIVING WITH DIABETES.

*Self-management of Diabetes in Adolescence/USA

BG testing,

managing BG,

hypo/hyperglycaemia

Frequency,

responsibilities,

forgetfulness, motivators

& barriers to injecting,

use of set programs

(pump)

Illness, going out for the day,

sleep over, smoking, alcohol,

going to parties, eye & foot

care, personal support &

understanding, self-appraisal

of life quality

Age, gender,

diagnosis, home

situation, height,

weight, insulin

regime, feelings

Types, intensity,

duration, setting,

barriers, reasons,

managing blood

glucose levels

Meal & food

choices,

responsibilities,

management of

diabetes, weight

Consensus Meeting

117 Items:

81 Educational

20 self-care score

16 Psychosocial

(self-perception)

20 Self-care score items

20 Self-care score items

Previously asked:

How many portions of fruit and vegetables do you normally eat in a day?

ADNAT SCORING ALGORITHM FOR 20 GENERIC QUESTIONS

Low score = low educational need, high score = high educational need Scoring range: 0 – 40

Do

ma

in Item

No.

Question Red (score = 2)

>25

Overall = 26-40

Amber (score = 1)

>12

Overall = 13-25

Green (score = 0)

Overall = 0-12

2

16 How many hours of pulse-raising exercise

or physical activity did you do last week?

Less than 1 hour

1-2 hours

3-4 hours 5-6 hours

7-8 hours More than 8 hours

18 What stops or prevents you from starting

to do exercise or physical activity?

I am not very good at it

I don’t like/enjoy

exercise or sport

Would rather watch TV

or play computer

games

My weight

I don’t have time

If no red options ticked + at

least one of:

It is difficult to manage my diabetes when I do

exercise

I don’t have the opportunity

It is difficult to do exercise with an insulin

pump

If no red or amber options

ticked + at least one of:

Nothing Being Ill eg. infection If my blood glucose is

low

21 What makes it difficult to manage your

blood glucose levels when exercising or

doing physical activity?

Exercise makes my

diabetes control worse

I am frightened of

getting a hypo

I’d need to take more

insulin

It is difficult with my

pump

If no red options ticked + at

least 1 of:

I’d need to eat more If my blood sugar is low

Only ticked:

I don’t find it difficult

22 What usually happens to your blood

glucose levels when you do exercise or

physical activity?

They usually go higher

I don’t know

I end up having a hypo

They vary They stay the same

They usually go lower but not so I am hypo

3

34 Do you eat fruit and/or vegetables? No or rarely Sometimes but not every day

Yes, usually every day

35 Do you eat treats such as sweets,

chocolate, fast food, takeaways?

Yes usually every day Sometimes but not every day

No or rarely

Proposed Path of learning

ADNAT combines reflective awareness raising with needs assessment to

facilitate patient-centred clinical consultations.

“I think it raised awareness cause obviously I know where I’m going wrong but I think it raised further awareness/you need to properly register and accept you need to change something about your diet or your lifestyle”

“a private space to address my illness”.

“..when I’m on the computer I feel as though it’s just me on that computer, not everyone’s got access to it....so I feel as though I’m safe”

“We had an actual area to talk on instead of going in, just having a chit chat about general health”

6. CREATING AN APP

http://blog.williamferriter.com/2013/07/11/technology-is-a-tool-not-a-learning-outcome/


6. CREATING AN APP


Assess Needs

(ADNAT)

Carers/Choice/

Talking Diabetes

etc.

Young person

led


Version 1

Website: www.adnat.co.uk

http://www.myadnat.co.uk/

Version 2

ADNAT App Website: www.myadnat.co.uk

http://www.myadnat.co.uk/

Version 3

Future development

Red Ninja Studios - Design and Technology

We

design

the

future

http://www.redninja.co.uk/

7. FUTURE PLANS

DEVELOPMENT PHASE √

TESTING PHASE √

RESPONDANT VALIDATION PHASE √

ADNAT √

Website √ www.myadnat.co.uk

APP for mobile devices √

EVALUATION PHASE √

Longitudinal cohort study of ADNAT

References • Cooper H, Spencer J, Lancaster G, Titman A, Johnson M, Wheeler S, Lwin R. (2014)

Development and psychometric testing of the on-line ‘Adolescent Diabetes Needs

Assessment Tool’ instrument. Journal of Advanced Nursing 70(2), 454–468.

• Cooper H., Spencer J., Lancaster J., Johnson M. & Lwin R. (2012) Perceptions of the clinical usefulness of the Adolescent Diabetes Needs Assessment Tool (ADNAT).

Diabetes Care for Children and Young People 1(2), 55–61.

• Spencer J., Cooper H. & Milton B. (2010) Qualitative studies of Type 1 diabetes in adolescence: a systematic literature review. Pediatric Diabetes 11, 364–375Cooper H.,

Cooper J. & Milton B. (2009) Technology-based approaches to patient education for

young people living with diabetes: a systematic literature review. Pediatric Diabetes 10,

474–483.

• Spencer J. & Cooper H. (2011) A multidisciplinary paediatric diabetes health care team: perspectives on adolescent care. Practical Diabetes 28, 210–2.

• Spencer J., Cooper H. & Milton B. (2013) A qualitative phenomenological study to explore the lived experiences of young people (13-16 years) with type 1 diabetes and

their parents. Diabetic Medicine 30, e17–e24.

• Cooper H. & Geyer R. (2007) Riding the Diabetes Roller Coaster: A New Perspective for Patients, Carers and Health Professionals. Radcliffe Publishing, Oxford.

• Cooper H. & Geyer R. (2008) Using ‘Complexity’ for improving educational research in health care. Social Science and Medicine 67, 177–182.

Malawi Developmental

Assessment Tool (MDAT)

Acknowledgments Principal Investigator: Melissa Gladstone

MD supervisors: Gill Lancaster, Prof Ros Smyth Institute of Child Health, University of Liverpool

Ashley Jones Centre for Medical Statistics and Health Evaluation, University of Liverpool

Ed Umar, Maggie Nyirenda, Edith Kayira, Ken Maleta Depts of Comm Health/Paediatrics/Wellcome Trust lab, Blantyre College of Medicine

APPLe trial: Prof Nynke van den Broek Liverpool School of Tropical Medicine

Lungwena Cohort study: Prof Per Ashorn, Ed Mtitimila Dept International Health, Tampere University Medical School, Finland

1. Background

•80% of the worlds disabled population (600 million) live in low income countries, many in Africa

•Early identification of children with disabilities can reduce the impact of impairment later in life –World Health Organisation - priority area

•Developmental delays in infants are often impossible to detect through routine physical examination –may only come to light once children start school

• Developmental assessment tools are therefore vital for identifying these children as early as possible

Developmental assessment

•Developmental tools in current use are: –Denver Developmental Screening Test, Denver II –Bayley’s Scales of Infant Development (BSID) –Griffith’s Developmental Scales –Kaufman Assessment Battery for Children (K-ABC)

•But they are designed and mainly validated in Western countries – may include tasks and materials alien to African

children

•Tools may fail to identify and assess children appropriately in an African setting

Example: Denver II tool American tool assessing 4 main sections:

a) Gross Motor skills eg. rolls over (3 months), walks

well (12 months), hops (3-4 years)

b) Fine Motor skills eg. grasps rattle (3 months),

scribbles (12 months), copies circle (3-4 years)

c) Social skills eg. smiles responsively (1-2 months),

drinks from cup (12 months), plays board/card

games (3-4 years)

d) Language eg. laughs (2 months), speaks 3 words

(12 months), names 4 colours (3-4 years)

Setting milestones in rural Africa

Culturally important milestones may differ in Africa from the West

For example:

In Malawi: tells stories to friends and parents, knows songs and dances, carries water on head (Kambalametore et al., 2000)

In Kenya: works in fields, integrates well into community (Levine et al., 1994)

More emphasis on social intelligence (eg. social duties and roles, obedience and good behaviour)

Aims Aim: To develop a more culturally acceptable child developmental assessment

tool for use in rural Africa

Preliminary Study: Lungwena in 2000/1

(n=1130 children, aged 0-6 years)

Main Study: Blantyre area in 2006/7

(n=1446 children)

Research Teams

Main Study: Blantyre area (as part of Azithromycin Presumptive

Preterm Labour Trial (APPLe trial))

Preliminary Study: Lungwena (as part of Lungwena Child Survival

Cohort Study)

2. Methodology Item selection

Examine existing evidence & tools in literature;

focus groups to develop themes; translation of

items; build on work done in preliminary study.

Item review and reduction Group items into hypothesised domains; PPI

involvement – research midwives and villagers; review

items by language expert and research team.

Pre-testing (n=6-19) Face and content validity and

comprehensiveness of items; Item

modification/elimination.

Piloting (n=80) Testing procedures with 6 research midwives;

review meeting every 2 weeks for 6 weeks;

wording, ceiling effects, item gaps, use timers.

Quantitative Item Reduction and Validation (n=1446) Assess ch ildren; Item by ite m plots, item reduction, item invariance;

reliability; consensus meeting; produce ‘norms’; scoring and validity

testing.

Main stages of development

Preliminary study • Western tools (Denver, Griffiths) assessed by MG, Malawian

doctors and language expert from University of Malawi

– culturally appropriate questions translated into Yao; some items modified

• Focus groups (8 female research workers) - new items added

• Piloting - items removed, modified, re-translated

Items excluded: ‘prepares cereal’, ‘uses a spoon and fork’

Items modified: ‘uses stick to draw on ground’ or ‘piece of chalk on

concrete floor’ instead of ‘uses a pencil and paper’; used objects such

as plastic cup, spoon, plate for recognition rather than pictures of them

• 1130 children 0-6 yrs were assessed; methodology developed

• Validity (face and content, respondent) was evaluated

• 97% GM, 79% FM, 91% Language, 49% Social items reliable

Qualitative study - focus groups • Malawian Professionals in English (physiotherapy, occupational health, nursing, special needs, paediatrics)

• Villagers in Yao or Chichewa (men, women, mothers, fathers, grandparents)

Mothers focus group in Namitambo Grandmothers focus group

Area of

development

discussed:

Quotes relating to social development from village

focus groups:

Area of

village:

Social duties and

chores:

“If it is a girl, can go and fetch water. If it is a boy, can go to

buy salt from the grocery” (Grandparents)

“When it’s a male child and you have taken him to the

garden, he will cry for the hoe and you know that he is

really a man. If you have not taken 2 hoes, you can not

work”. (Grandparents)

1

1

Community

roles:

“my child is 2 years 8 months....everything that has

happened at the house he tells the relatives when they are

back from school... showing that he is growing well”

(women).

2

Good behaviour

and obedience:

“When a child of this age is sent to draw drinking water in

the house and when giving it, the child kneels down, you

know that this child is intelligent (women)”

“When you call the child to come, she kneels down and

says what are you calling me for?” (women)

2

3

Feeding and

toileting:

“At 1 year 2 months, any mother knows that her child is

giving signs of visiting the toilet, so you undress the child

and tell them to pass stool” (mothers)

“you go with the child to the toilet and hold the child to

avoid him falling inside” (grandparents).

3

4

Intelligence and

cognition:

“If it is a female child, when sees a mother putting a pot on

the fire, she takes a tin and places it somewhere and

pretends cooking nsima and we also see that she is

intelligent”(mothers)

2

1 Bangwe (semi–urban)

2 Nguludi (rural)

3 Namitambo (rural)

4 Mikolongwe (rural)

Main study participants • Quota sample – 1446 children aged 0-6 years from Blantyre

in Southern Malawi (preparation on pilot sample of n=80)

• Recruited from Azithromycin Presumptive Preterm Labour Trial (APPLe) trial

– Pregnant women presenting to local antenatal clinics between June 2006 and July 2007 with normal healthy child aged 0-6 years; asked to bring child to next clinic

–3 rural areas, 1 semi-urban area

• Children assessed at clinic by 6 local research assistants

• Exclusion criteria: children born prematurely (

Data Analysis

Pilot study (n=80) over 6 weeks, discussions with research midwives

Main study (n=1446) • Item by item performance - logistic regression graphs, covariate gender

• Pass/fail scoring for each item and domain

Reliability

• Intra-observer reliability, same child, same observer, 2 weeks apart (n=124)

• Inter-observer reliability (immediate) same child, same day (n=56)

• Inter-observer reliability (delayed) same child, later same day (n=52)

Validity

• Face validity - Do the items look acceptable to untrained judges?

• Content validity - Does the tool examine all the domains it is meant to measure?

• Construct validity – extreme groups (disabled n=80, malnourished n=120, compared to age matched normal children)

3. Outcomes and results • 1446 of 1657 children approached, were eligible & assessed

• Focus groups (professionals and village) useful for

developing the gross motor, language and social domains

• Piloting - questions removed, modified, re-translate

• Excellent reliability (inter-observer immediate kappa>0.75

for 99% items, inter-observer delayed for 89% items, intra-

observer 2-wk for 71% items; remaining items k>0.4, 2 poor)

• Consensus meeting - final set of items agreed (34 items in

each of the 4 domains)

• Validity - high sensitivity/specificity across domains; disabled

vs normals 63.9 points lower, malnourished 14.9 points lower

http:kappa>0.75

Gross motor skills (draft MDAT III)

Assessment Basket of props

Item by Item Analysis • Logistic regression was carried out for each item with decimal age as a covariate in the model to obtain

predicted probabilities

• The observed percentage of children in each of the 33

age groups passing an item was found to obtain

observed grouped probabilities.

• Graphs were then drawn of decimal age (for the 33 age

groups) against the predicted probability of passing, the

observed probability of passing and the observed 0/1

responses.

• This was done for all the question in the four domains

Example: Fine Motor Question ‘Transfers from hand to hand’

0.2

.4.6

.81

Pro

bability

0 1 2 3 4 5 6 7Age (years)

Predicted Observed grouped Observed

FM08

Assessing the fit of the models

•For each question the ‘goodness of fit’ of the model was tested using the Hosmer and Lemeshow test statistic.

•A p-value of less than 0.05 was taken as statistically significant.

• If the model fitted the data well, then the ages corresponding to 25%, 50%, 75% & 95% percent passing, were determined

•Questions that gave a poor fit to the data, were refitted using triple split joined spline regression.

Example: Gross Motor Question

‘Walks backwards’

0.2

.4.6

.81

Pro

bab

ility

0 1 2 3 4 5 6 7Age (years)

Predicted Observed grouped Predicted Spline

GM17

Examples of poor questions

0.2

.4.6

.81

Pro

bability

0 1 2 3 4 5 6 7Age (years)


FM17

0.2

.4.6

.81

Pro

ba

bili

ty

0 1 2 3 4 5 6 7Age (years)


GM08

‘Stands holding on’ ‘Moulds ball with clay’

0.2

.4.6

.81

Pro

ba

bili

ty

0 1 2 3 4 5 6 7Age (years)


SOC 16

0.2

.4.6

.81

Pro

babili

ty

0 1 2 3 4 5 6 7Age (years)


SOC 04

‘Can put clothes on with help’ ‘Spends most of time on mum’s back’

Reliability

Consensus Meeting

•Once all questions had been analysed, an ‘expert’ panel met to review which questions

should remain, be modified or removed

•Each question was judged on: –Graphical representation

–Goodness of fit on logistic and spline regression

–Inter and intra-observer reliability

–Interpretability by participants and researchers

Charts of ‘norms’ – main study

Scoring ‘Fail’ if failed 2 items before line at 90th percentile ‘norm’

Validity

5. Conclusion •Malawi Developmental Assessment Tool

(MDAT) with normal reference values to

assess child development up to 6 years of

age in a rural Africa setting

•Useful as: – clinical tool for the early identification

of neuro-disabilities

– school assessments by teachers

– outcome measure in international intervention

programmes designed to improve child

development

6. Future Work – scoring methods

• PhD student (Phillip Gichuru)

• Item by item scoring – Estimate the age of a ‘normal’ child passing an item

at a specified probability using: GLM - Logistic Regression

GAM - Generalized Additive Models Unconstrained Generalized Additive Models

SCAM - Shape Constrained Additive Models

• Overall scoring – Using all the responses of a child to obtain an overall score to characterise a child’s developmental status given their age using: Simple score versus model based methods (IRT)

Z-score versus smoothed Z-score

GAMLSS Smoothed Z-Scores

Figure a) : Empirical Z-score means and standard

deviations Figure b): GAMLSS Smoothed Z-means and standard

deviations

Density Comparison – Smoothed Z-Scores

Fig. a): Scatter plot of smoothed scores versus age in normal, disabled

and malnourished samples in Gross Motor domain.

Fig. b): Density plot of the smoothed Z-scores in Gross Motor

domain for Normal, disabled and malnourished samples.

GENERAL PRINCIPLES AND

STATISTICAL ISSUES

Construct a Research Plan Item selection

xamine existing evidence & tools in literature,

olicy documents, guidelines; focus groups,

terviews – develop themes; systematic review.

E

p

in

Item review and reduction Group items into hypothesised domains; PPI

involvement - young people/carers on steering group;

Delphi study to review/rate items (esp.if newly created)

Pre-testing (n=10-20) Cognitive interviews; Face and content

validity and comprehensiveness of

items; Item modification/elimination.

Piloting (n=50) Test procedures; mean, range of responses,

% missing data, floor and ceiling effects; item

by item plots; initial reliability assessment.

Quantitative Item Reduction and Validation (n=300) Distribute questionnaire; item performance, item-total correlations,

item reduction, item invariance; consensus meeting to finalise tool;

devise scoring method; internal consistency, reliability, validity.

Scoring of items

•Theoretically driven – domain scores and/or overall scores, relationship with age

eg. MDAT (gross motor, fine motor, language and

social domains)

•Logistic regression – weighting of items, predictive score eg. EARLI

•Item-response analysis (eg. Rasch model, graded response model)

eg. ADNAT, new work on MDAT (GAMLSS, IRT)

Reliability Concerned with the consistency of measurement

•Test-retest reliability –assesses the repeatability of the method – are assessments made by the same person on two

separate occasions under the same conditions reliable?

• Inter-rater reliability –assesses the reproducibility of the method – do two different raters agree in their assessments using

the same method under the same conditions?

• Internal reliability (eg. item total correlations, Cronbach’s alpha)

Validity Concerned with the accuracy of measurement

• Construct validity

– Convergent validity - should perform similarly to other

known constructs eg. ADNAT to HbA1c levels

– Divergent validity eg. should differentiate between

normal/abnormal people (extreme groups)

–Criterion validity •How well does the method compare to an established ‘gold standard’ accurate method in terms of sensitivity/specificity

•Concurrent or predictive

• Respondent validity

PROM Design - a wider perspective Stage Issues Study design Methods of analysis

1. Developing the

instrument

(a) Physiological Device Does this device agree

sufficiently well with

one in current use?

How do I handle

repeated measures

and duplicates?

How do I compare this

device to a known gold

standard?

Method comparison

study

Diagnostic test study

1Mean difference, 95% limits of

agreement 2Kappa with 95% CI

1ANOVA or confirmatory factor

analysis 1,2Multi-level modelling 1ROC curve, Area under Curve

with 95% CI 2Sensitivity, specificity, positive

predicted value, 95% CI

(b) Questionnaire Creation of items

Do the items (eg.

measured on Likert

scale) work reasonably

well?

Focus groups,

interviews, expert

review

Pilot study to evaluate

item performance on a

small sample

Qualitative analysis to identify

themes and assess face, content

validity

Mean, minimum, maximum

responses; % missing; floor and

ceiling effects; item-total

correlations

1measurements taken on a continuous scale (can include ordinal data with many categories) satisfying the assumptions of the method, 2binary and/or ordered categorical measurements

Stage Issues Study design Methods of analysis

2. Establishing

reliability and validity

a) Reliability Does the instrument

give stability and

consistency of

measurement?

Does the instrument

have good internal

properties

(reliability)?

(for questionnaire

type instruments

typically using

dichotomous items

or Likert scales)

Intra-rater/test-retest

(using the same

assessor on two

different occasions)

Inter-rater (using 2

different assessors on

the same occasion)

Item performance

evaluation on larger

sample

Examine internal

structure and number

of domains

1ICC (also mean difference, 95%

limits of agreement may be

useful) 2Kappa, weighted kappa with

95% CI, latent class analysis

As in Stage 1 above for pilot

study.

Cronbach’s alpha, split-half

coefficient, Kuder-Richardson 20

method 1,2Factor analysis, 2item-response

analysis

Reference: Lancaster GA, Statistical issues in the assessment of health outcomes in children: a methodological review. JRSS A 2009, vol 172.


b) Validity Does the instrument

measure what it is

supposed to be

measuring?

Convergent/divergent

validity

Extreme groups;

known-groups;

concurrent validity

using external criterion;

predictive validity

Criterion validity (with

gold standard)

1Correlation (Pearson or

Spearman rank)

1T-test, Mann-Whitney U, Cuzick

test for trend, ANOVA, Kruskal-

Wallis etc. 2Chi-squared test

As in Stage 1 above for diagnostic

test study


3. Measuring change over

time and discriminating

between subjects

Magnitude of change

Measurement error

Multiple measurements

How do I assess the

magnitude of change

from baseline to

endpoint?

Is there a better way of

taking into account pre-

test/baseline measures

(eg. to avoid regression

to mean)?

How do I handle

measurement error

(particularly in dietary

data)?

How do I account for

multiple measurements

taken over time?

Pilot study and/or

main intervention

study

Randomised or non-

randomised two group

comparison on large

sample

Take additional

measurements using a

reference instrument

1Cohen’s effect size, Guyatt’s

responsiveness statistic, standard

error of measurement

1ANCOVA to compare groups 2Logistic/ordinal regression

Use reference instrument (eg. diet

diary, biomarker, till receipts) to

calibrate/adjust results

Select the baseline and most

important follow up time point only

and analyse as above;

Use summary measures (eg. mean,

peak, area under curve, gradient)

to obtain one measure for each

child; 1,2Longitudinal data analysis,

growth curve modelling


4. Reference values for

a healthy population

Reference range

Gender-specific

Age-specific

Z-scores

Anthropometric

measurements

How do I create reference

values for a normal,

healthy population?

What if the

measurements vary by

gender (usually bimodal

distribution)?

What if the

measurements vary for

children of different

ages?

How do I measure and

compare deviations from

the average across

different groups of

children?

Is there a way of dealing

with variability in growth

and non-linear change

eg. in weight for age,

weight for height, BMI?

Take measurements from

a large consecutive or

random sample of

children from school or

community

Separate the sample

measurements into those

for boys and girls

Separate sample

measurements into age

groups (could take quota

sample);

Use whole sample

1Mean, 95% reference range

or 1Median, 2.5th and 97.5th centiles

Calculate reference range for each

gender group separately

Calculate age-specific reference ranges

for each age group separately 1Linear regression, fractional

polynomials 2Logistic regression, regression splines

Express measurements in terms of z-

score units from the mean (ie.

reference mean for that age and

gender) and compare summary

statistics for z-scores across groups

Z-scores, LMS (Lambda, Mu, Sigma)

method (skewed data), GAMLSS

(Generalised Additive Models for

Location, Shape and Scale)

References • Gladstone M., Lancaster G.A., Umar E., Nyirenda M., Kayira E., Van Den

Broek N. and Smyth R.L. (2010). The Malawi Developmental Assessment

Tool (MDAT): creation, validation, and reliability of a tool to assess child

development in rural African settings. Public Library of Science (PLoS)

Medicine, 7(5): e1000273.

• Gladstone M., Lancaster G.A., Umar E., Nyirenda M., Kayira E., Van Den Broek N. and Smyth R.L. (2010). Perspectives of normal child development

in rural Malawi - a qualitative analysis to create a more culturally appropriate

developmental assessment tool. Child: care, health and development 36(3):

346-353.

• Gladstone M., Lancaster G.A., Jones A.P., Maleta K., Mtitimila E., Ashorn P. and Smyth R.L. (2008). Can Western developmental screening tools be

modified for use in a rural Malawian setting? Archives of Disease in

Childhood 93: 23 – 29.

• Lancaster G.A. (2009) Statistical issues in the assessment of health outcomes in children: a methodological review. JRSS A 172(4): 707-727.

Patient Reported Outcome Measures (PROMs) for use with children and adolescents: a view across disclipinesPatient Reported Outcome Measures (PROMAcknowledgments. Statisticians: Gill Lancaster, Andrew TiIT Specialist: Mark Johnson DIFFERENT. Adolescent Diabetes Management: Complex Fitness Landscape. Study Aims. Paediatric Diabetes Education*. Study participants .Data Analysis .20 Self-care score items. ADNAT SCORING ALGORITHM FOR 20 GENERIC QProposed Path of learning. 6. CREATING AN APP Version 1 Website: www.adnat.co.uk Version 2 ADNAT App Version 3. Future development. 7. FUTURE PLANS References. Malawi Developmental .Assessment Tool (MAcknowledgments. Ashley Jones Ed Umar, Maggie Nyirenda, Edith Kayira, 1. Background. Developmental assessment. Example: Denver II tool. Setting milestones in rural Africa. Aims. Research Teams. 2. Methodology. Main stages of development. Preliminary study. Qualitative study -focus groups. Main study participants. Data Analysis .3. Outcomes and results. Gross motor skills (draft MDAT III). Assessment. Item by Item Analysis. Example: Fine Motor Question. ‘TransfersAssessing the fit of the models. Example: Gross Motor Question .‘Walks baReliability. Consensus Meeting. Charts of ‘norms’ – main study. Scoring Validity. 5. Conclusion. 6. Future Work – scoring methods. GAMLSS Smoothed Z-Scores. Density Comparison – Smoothed Z-Scores GENERAL PRINCIPLES. AND .STATISTICAL ISSConstruct a Research Plan. Scoring of items. Reliability. Validity. PROM Design -a wider perspective. References.

patient reported outcome measures (proms) for …pediatric diabetes 11, 364–375cooper h., cooper...

Documents