Brunel University London

2nd International Symposium on

Factorial Survey Methods in Health, Social and

Ageing Research

Wednesday 15th June 2016

Darwin Room, Hamilton Centre,

Brunel University London

Hosted by:

Professor Ken Gilhooly, Brunel University London

Brunel University London

Contents

1. Using the factorial survey design to study individual judgement

Prof Brian Taylor, Ulster University

2. Using a fractional factorial survey to develop an effective training tool to

improve fitness-to-drive decisions

Prof Cilla Harries, Brunel University London

3. Using a fractional factorial survey to develop an effective training tool to

improve dietetic referral prioritisations

Dr Hulya Gokalp, Brunel University London

4. The importance of work willingness in social assistance eligibility

assessments

Dr Marjolijn de Wilde, University of Antwerp

5. Administration of an online factorial survey using Qualtrics software

Helena McElhinney, Ulster University (contact details only)

6. Synthesis of three data sources to determine independent variables in an online

factorial survey of implantable cardioverter defibrillator deactivation decisions

Loreena Hill, Ulster University

7. A tool that produces judgment study stimulus case sets with specified cue-cue

correlations

Dr Robert Hamm, University of Oklahoma Health Sciences Center

Using the factorial survey

design to study judgement

Brian Taylor

Ulster University

Northern Ireland

Factorial Survey - Essence

Present a random sample of respondents

with a random selection of vignettes

with randomised characteristics

that represent familiar type of situations

to make a familiar type of judgement;

analyse results to measure effect of

vignette factors on the judgements

of this type of respondent

Juni 2016 ISFS 4

Judgements vs decisions

Judgment = the considered evaluation of

evidence by an individual using their cognitive

faculties so as to reach an opinion on a preferred

course of action based on available information,

knowledge and values. (Taylor, 2013, p179)

Decision = a conscious process (individually or

with one or more others) leading to the selection

of a course of action from among two or more

alternatives (Taylor, 2013, p178)

Taylor BJ (2013) Professional Decision Making

and Risk in Social Work (2nd ed) London: Sage

Juni 2016 ISFS 5

Approaches to Studying Judgements

Normative

= how people OUGHT to make judgements (to be ‘rational’)

Descriptive

= how people DO make judgements in reality - FS

Juni 2016 ISFS 6

FS=studying decision inputs & output

IN = factors considered, context, etc

Box = respondent cognitive process

OUT = judgement to be made

NB can incorporate respondent

characteristics in FS but this is not a strength

Juni 2016 ISFS 7

JudgementIN OUT

Basic factorial survey research

question:

What is the size of effect of factors

<X, Y, Z> [IVs]

on judgements by

<type of respondent>

regarding

<judgement to be made>? [DV]

Juni 2016 ISFS 8

Vignettes, Case Scenarios, Paper Cases

BASIC

QUESTION

for

respondent:

What would

you do in this

(familiar type

of) situation?

Juni 2016 ISFS 9

What factors to use within vignettes?

Identify factors from

previous research AND/OR

preliminary quaLitative study AND/OR

an expert group

> Factors of most interest YET

realistic vignettes for respondents

> Select for hypotheses in analysis

Juni 2016 ISFS 10

What levels of each vignette factor?

Interval variables

Ordinal variables

Nominal variables

Measurable effect on this judgement

Realistic levels for respondents

Meaningful anchors at end of scales

Pilot variables & levels

Juni 2016 ISFS 11

Vignette structure – IVs- example

Create one standard series of sentences to

encapsulate the dimensions of interest

Include one level of each element in the

sentence in each vignette e.g.

Mrs Smith is <70, 75, 80, 85, 90, 95>. Her

mobility level is <bedfast, uses a zimmer

frame, ‘furniture walks’, needs supervision,

fully mobile unaided>. etc etc

Juni 2016 ISFS 12

Client, Family AND Service Variables

Son visits <daily, weekly, monthly, never>

Home care services are currently provided for

<1,2,3,4,5,6,7> days per week

The Day Centre staff are <very stressed,

stressed, able for new admissions, eager to

redress falling enrolment>

NB variables must be familiar to

respondents and realistic to them

Juni 2016 ISFS 13

Juni 2016 ISFS 14

Independent Variable Levels No.

Levels

Source

Age Under 16 yrs. old

17-20 yrs. old

In her late 20’s

3 Focus Group 1,

FG 2, FG3, FG4,

article A

Drug Use Has never taken illegal drugs

Takes prescribed methadone

Takes illegal drugs

3 FG1, FG2, FG3,

FG4

A, P

Domestic Violence Has a supportive partner

Feels afraid at home

2 FG1, FG2, FG3,

FG4

W, A, P, M

Alcohol use Does not drink alcohol

Drinks 1-2 units per week

Drinks 5-6 units per week

3 FG1, FG3, FG4

P

Demonstrating source of IVs & Levels

(extract: Helena Mc Elhinney)

Vignette structure

– Dependent Variables Create 1, 2, or 3 standard judgements to be

made at the end of each vignette:

E.g. choose one of: “admit to nursing or

residential home”, “provide homecare”, “refer

for further assessment”, “send letter

regretting refusal of service”

E.g. rate likelihood of admission to

institutional care during next 12 months on a

scale of 1 to 100

Juni 2016 ISFS 15

The Question(s) = DVs

Juni 2016 ISFS 16

To what extent do you perceive there to be a risk of harm to the unborn child?No Harm 0 1 2 3 4 5 6 7 8 9 Significant Harm

To what extent do you think a child safeguarding referral should be made?No 0 1 2 3 4 5 6 7 8 9 Yes

If referred, how confident are you that a there will be a good outcome to this case?Not Confident 0 1 2 3 4 5 6 7 8 9 Very Confident

Categorical, Ordinal and Interval

Independent Variables

Nominal (categories)

Co-morbidity;

Ordinal (ordered)

Activities of Daily Living - level;

Interval (measured)

Number of ICD shocks; score on MMSE;

number of hospital admissions

Juni 2016 ISFS 17

Categorical, Ordinal and Interval

Dependent Variables

Nominal (categories)

Refer for investigation or not

Ordinal (ordered)

Outcomes in accepted order of severity

Interval (measured)

Likelihood to discuss de-activation of ICD;

estimate of likelihood of X occurring in

time period Y

Juni 2016 ISFS 18

How many vignettes per respondent

Typically 15 – 20

May be 6 to 40

Consider

respondent fatigue

Will they still be

taking vignettes

seriously at the

end?!

Check in piloting

Juni 2016 ISFS 19

Factorial DesignS Research methods textbooks that include

factorial designs normally include only:

what I call Factorial Experiments

(called Factorial Designs)

DO NOT YET USUALLY INCLUDE:

Factorial Surveys

= another type of Factorial Design

NB neither of these are factor analysis!

Juni 2016 ISFS 20

Use Factorial Design (Factorial

Survey OR Factorial Experiment)

Interested in two or more variables

Context means variables cannot be separated

We are interested in interactions

E.g. chronic illness (IV) and age (IV) on need for health & social care service (DV)

E.g. age at admission of child to state care (IV) and length of time in care (IV) on educational attainment (DV)

Factorial Design (Factorial Survey OR

Factorial Experiment)

Juni 2016 ISFS 22

Mobility limited A B

Mobility not limited C D

Cognitive functionMobility

Alert Confused

DV: Would you recommend client as able

to live in sheltered housing arrangement?

Independent Variables

More Complex: 3*4 Factorial Design

Juni 2016 ISFS 23

Shaken vigorously A B C

Struck with stick D E F

Banged on wall while shaking G H J

Slapped on face K L M

Child’s AgeAbuse

0 - 5 years 6 - 10 years 11 - 15 years

DV: Is a child protection investigation appropriate? -

adapted from O'Toole R, Webster SW, O'Toole AW & Lucal B

(1999) Teachers' recognition and reporting of child abuse: A

factorial survey. Child Abuse and Neglect, 23(11), 1083-101

Factorial Experiments compared to Factorial Surveys

In a factorial experiment, the number of

vignette factors in EACH cell of the above

tables must be sufficient to allow us to

measure the effect of each variable on every

other variable

HENCE in a factorial experiment

vignette must be (unrealistically) simple to

give enough responses for each combination

Juni 2016 ISFS 24

Factorial Survey - external validity

through randomisation of factors

The factorial survey, in contrast, randomises

the selection of factors within the vignettes

HENCE

It is not necessary to have large numbers for

every possible combination

HENCE

Vignettes may be made more realistic by

including more factors & more levels

Juni 2016 ISFS 25

Eg. IVs in Factorial Experiment

Social workers’ support for older person’s

autonomy (DV) in relation to 3 factors (IVs):

Cognitive status [2 levels]

Care status [2 levels]

Caregiver burden [2 levels]

NB factorial experiments allow modelling

of the judgements of individuals, unlike FSTara Healy (1998) ‘The complexity of everyday ethics in home health

are: An analysis of social workers’ decisions regarding frail elders’

autonomy’, Social Work in Health Care, 27(4): 19-37

Juni 2016 ISFS 26

Eg. IVs in Factorial Survey

DV: Long term care plan in older age

3 factors with 2 levels

15 factors with 3 levels

3 factors with 4 levels

2 factors with 5 levels

1 factor with 6 levels

> 1,099,496,032,600 possible vignettes!

Catherine Hagan Hennessy (1993) ‘Modeling case

management decision-making in a consolidated long-term

care program’, The Gerontologist, 33(3): 333-41

Juni 2016 ISFS 27

Basic Analysis Regression measures effect of significant IVs

Need measure independent of sample size

Pearson’s R2 = amount of variance in the

decision that is attributable to that factor

One-way ANOVA to measure size of

relationship between each IV & DV in turn

Can use pairs of two-way ANOVA to establish

if there are first-order interaction effects

>Taylor BJ & Zeller RA (2007) Getting robust and valid data

on decision policies: The factorial survey. Irish Journal of

Psychology, 28(1-2), 27-42

Juni 2016 ISFS 28

Refinements

Initial 2-3 standard vignettes to acclimatise

respondent and to control for tendency to

score high or low before randomised ones

These may be chosen as ~low/medium/high

Repeat standard vignette - consistency

Ask to describe ‘a recent case’ to validate

external validity of vignettes

Questionnaire to give demographic data on

respondents – can be combined with vignette

data

Juni 2016 ISFS 29

Can combine levels of data:

Juni 2016 ISFS 30

Organisation Data

Respondent Data

Vignette Data

BUT hierarchical regression may lose more than is

gained! Recommend standard regression >

Taylor BJ & Zeller RA (2007) Getting robust and valid

data on decision policies: The factorial survey. Irish

Journal of Psychology, 28(1-2), 27-42

Independence of Decisions

If we can assume for statistical purposes that

the vignette decisions may all be treated as

independent even though each respondent

has completed several

Then can use standard regression, giving

Much greater scope to combine the results

from different studies

Can demonstrate error with Monte Carlo Analysis

Juni 2016 ISFS 31

Factorial Survey – Advantages

High internal validity of experimental method

because vignette variables are random and

are controlled by the researcher

High external validity of survey method

because vignettes may be realistic in terms

of complexity of factors

Efficiency of survey method to access large

numbers giving robust results

Juni 2016 ISFS 32

Prospects for Factorial Surveys

Computers greatly simplify creation of

vignettes with randomised characteristics

Need to develop more efficient process of

data extraction & web-based administration

Developing alternative presentations

visual, auditory, referral form, typical presentation

Develop time-sequence judgements?

Juni 2016 ISFS 33

34

Theory: What is Judgement Analysis?

(its) “.. use as a common descriptor for any paradigm employing multiple regression equations to model human judgement is now widespread.”*

Tenets:

human judgement is probabilistic &

can be expressed as a linear model**

*Ray Cooksey (1996) Judgment Analysis: Theory, Methods &

Applications. San Diego, CA: Academic Press (p. xi)

**Robert S Wigton (1988) Use of linear models to analyze

physicians’ decisions. Medical Decision Making, 8, 241-252

35

Applying Brunswik Lens ModelBrunswik E (1952) The Conceptual Framework of Psychology, Chicago:

University of Chicago

REALITY

CUE 1

CUE 2

CUE 3

JUDGE

Observed Cues

Cues can be modelled egFS study?!

Cues can be

modelled

eg risk factors

Factorial Surveys in relation to

Brunswik’s key points

1. Probabilistic view of the world

vs then dominant deterministic paradigm

2. Representative design

= real life not dominant laboratory focus

3. Focus on individuals

Factorial Surveys fit the first two

points but not the third

36

37

What is missing from JA?

Judgement Analysis does not include explicitly

the concept of modelling (with multiple

regression) dispersed ‘groups’ (= types) of

people such as in Factorial Surveys

JA either samples the whole universe of

possible vignettes (full factorial) or cases

selected from reality (‘representative’ in sense

of real-life, not laboratory-based)

JA currently lacks the concept of random

selection from the full factor(ial) universe

References Origins:

Rossi PH & Nock SL (1982) Measuring Social

Judgments – The Factorial Survey Approach.

Beverley Hills, California, Sage

A straightforward ‘how to’ description:

Taylor BJ (2006) Factorial surveys: Using

vignettes to study professional judgement. British

Journal of Social Work, 36(7), 1187-1207

Detailed recent text by Katrin & Thomas:

Auspurg K & Hinz T (2014) Factorial Survey

Experiments. London: Sage [with web resources]

Juni 2016 ISFS 38THE END

Professor Priscilla Harries

Professor Carolyn Unsworth

Dr. Hulya Gokalp

Dr. Miranda Davies

Educating Novice

Occupational Therapy Driver

Assessors To Make Fitness-

to-drive Recommendations

For Older and / or Disabled

Clients

Driving is an important and valued

IADL. It promotes independence, mobility & freedom

Health conditions and disability can impair driving

Occupational therapists are well placed to assess fitness-to-drive, and usually conduct off & on-road assessments.

It is not clear what judgement policies OTDAs are using when making fitness-to-drive recommendations

Evidence based training is needed to enhance workforce capacity (Classen, 2010; Kortling & Kaptein 1996; Unsworth, 2007)

Background

Project advisors:

Service user group

Tamalina Al-Dakkak

Ann Bunce

Jon and Rodney Hutchings

Judith Sinclair

Hilary Strickland

Project advisory group

Professor Mary Gilhooly (Brunel University London)

Professor Mark Williams (Brunel University London)

Professor Peter Ayton (City University)

Professor Richard Tay (La Trobe University)

Dr Catherine Lant (Rosemead GP surgery)

Experienced occupational therapy driver

assessors

Meg Marmo (Australia)

Jenny Wise (Australia)

Kirsty Flint (UK)

Objectives 1-3

Develop a consensus policy on fitness-to-drive from experts.

Method

Results

Objectives 3-6

Use consensus policy in RCT to determine if we can train novices and then develop decision aid.

Method

Results

Structure of the presentation

Task 1Case scenarios

Task 3Training Package

Task 2Policy Capturing

Study

Decision

Training

Aid

Study Design

Task 4RCT

Task 5Training aid Website

Task 5Training aid Website

1 Create ‘driving’ case scenarios of older people and people with disabilities who want to learn to drive, or resume driving for assessment;

2 Identify what fitness-to-drive (fit, not fit) decisions are made;

3 Statistically model and obtain consensus on how experienced occupational therapy driver assessors make optimal fitness-to-drive decisions for older people and people with disabilities who want to learn to drive, or resume driving;

Objectives

In relation to Objectives 1-3

1.

Case background (same for each case)

The client you are assessing, who is a private car driver, lives on

the outskirts of the city / metropolitan area. Their doctor, who

would like them to have a comprehensive driver assessment, has

referred them to you. They are suitable for on-road assessment as

they are medically fit to drive, they do not have unilateral spatial

neglect (lack of awareness of one side) and they meet the legal

vision standards for driving. Initially you undertake a client-based

assessment using a battery of tests in the clinic e.g. range of

movement testing, pain questionnaire and medication screen, etc.

You decide to proceed to an on-road assessment, which involves

you taking the client out, with a driving instructor in a dual-

controlled car, in an area not familiar to them. You sit in the back to

observe their actual driving behaviour and are going to make your

final recommendation from the findings of the on-road assessment.

The results of the on-road assessment are as shown below.

Cue Client Cue Level

Age 60 years old

Driving experience Client has been driving 3-7 years

Driving history Client has had a few minor scrapes in the last 12-months

Current driving needs Client drives predominantly in the local / familiar area

Physical Skills Physical skills support safe driving

Cognitive & / or perceptual skills

Minor cognitive &/or perceptual problems identified but demonstrates capacity for learning & improvement

Sensory functions Sensory functions support safe driving

Driver behaviour Some behaviour problems identified

Road law knowledge & / or road craft

Road law knowledge and / or road craft support safe driving

Vehicle handling skills Vehicle handling supports safe driving

Driving instructor interventions Driving instructor provides one physical intervention

Medical prognosis Medically stable

Fit-to-drive– Unrestricted licence

Fit-to-drive– With conditions. For example, using an automatic car.

Not fit-to-drive– Driver rehabilitation to be completed (may require reassessment)

Not fit-to-drive– Suspend or cancel licence.

What is your recommendation for this client?

Please click on one of the boxes below to make

your recommendation:

Social Judgment Theory method

45 experienced occupational therapy driver assessors from the UK, Australasia

Mean number of years they worked as an occupational therapist was 21, with a mean of 11 years’ experience working as a driving assessor.

Methods

Fitness-to-drive recommendations

made for a series of 64 case scenarios, on-line;

Data were analysed using discriminant function analysis and an intraclass correlation coefficient;

ICC (type 2,1) was used to determine the level of agreement between decisions made by OTDAs.

Results

Recommendation Number of cases

Not fit-to-drive: Suspend or cancel licence

569 (20%)

Not fit-to-drive – driver rehabilitation to be completed

1529 (53%)

Fit-to-drive: With conditions

415 (14%)

Fit-to-drive: Unrestricted licence

367 (13%)

Total 64 scenarios x 45 OTDAs

2880

Discriminant function analysis.

Information central to fitness-to-drive recommendations are:

physical skills,

cognitive and perceptual skills,

road law craft skills,

vehicle handling skills

the number of driving instructor interventions.

Results How do OTDAs weight different types of information when making fitness-to-drive recommendations?

The three functions produced by the discriminant function analysis, showing the correlations between cues and the fitness-to-drive recommendation (Structure Matrix).

Functions

Cues 1Discriminated clients who were Fit-to-drive from Not fit-to-drive

2Discriminated clients who were Not fit-to-drive - require rehab, from the other three

outcomes

3Discriminated clients who were Fit-to-drive

or Not fit-to-drive from the middle two

outcomes of Fit to drive with conditions,

or Not fit-to-drive -require rehab.

Physical skills .48* .43 .07

Instructor interventions .44* -.38 -.42

Road law/road craft .39* -.03 -.03

Cognitive & perceptual skills .39* .37 -.09

Sensory functions .33* .07 .20

Driving experience .14* .05 .06

Medical prognosis .12 .48* -.27

Driving Need .07 -.37* .10

Driving History .04 -.19* .10

Vehicle handling skills .39 -.29 .53*

Age .07 .15 .41*

Driver behaviour .32 -.08 -.38*

Agreement (consensus) between fitness-to-drive recommendations was very high ICC= .97(95% CI .96-.98).

Results Do experienced OTDAs agree (level of

consensus) about the fitness-to-drive

recommendations made?

4. Produce a training package and develop an experimental website to test the effectiveness of the web-based decision aid;

5. Test the effectiveness of this aid on novice occupational therapists’ capacity to make fitness-to-drive decisions;

6. Host open access web-based decision aid designed to promote optimal occupational therapy assessment for use by the profession internationally.

Objectives

Randomised Control Design

Case scenario

decisions

Baseline

Case scenario

decisions

Post training

Training

(Intervention)

No training

(Control)

Eligibile for invitation (n=XX )

Consented to participate

(n=289)

Analysed at the baseline (n=70)

Analysed at immediate post-test (n=70)

Completed immediate post-training test

(n=70 )

Allocated to intervention (n=120)

completed the task (n=70)

received training intervention

(n=70)

Completed immediate post-training test

(n=96 )

Allocated to control (n=169)

completed the task (n=96)

received no training (n=96)

Analysed at the baseline (n=96)

Analysed at immediate post-test (n=96)

Allocation

Follow-Up

Randomized (n=289)

.

Results

Bland-Altman test was used to measure agreement between decisions made by a novice and experts’ consensus.

The equality graphs below demonstrate this trend

Control Group

Experimental Group

The effect size is calculated using the mean BA statistics from the post-training for control (.31, sd=.41) and for intervention (-.02, sd=.56); this demonstrated a moderate effect (d=.69, r=.32).

Results

Signal detection theory (SDT) was used to predict accuracy of decisions made by novices and to identify any change in decision strategy.

Although not all novices were able to align their recommendations to align with the expert consensus, their recommendation strategy changed and they adopted a lower risk strategy (ie recommended more clients to receive rehabilitation or not drive) which was more aligned with the experts (same as viewed in Equality Graphs above)

Web link: https://cisbic.bioinformatics.ic.ac.uk/fitness_to_drive_decision_aid/

(About to be launched!)

This tool is a training aid only. The developers are not liable for practice based decision making in the workplace.

Share expert capacity to optimize decision making

Can increase workforce capacity among novices in enhancing skilled fitness-to-drive recommendations

Can lead to a reduction in the number of unsafe drivers

Can lead to maintenance quality of life for those drivers who are fit-to-drive

Conclusion &

Contribution to practice

Unsworth, CU. , Harries, PA. and Davies, M. (2015) 'Using Social Judgment Theory method to examine how experienced occupational therapy driver assessors use information to make fitness-to-drive'. British Journal of Occupational Therapy, 2 pp. 71 - 72.

doi: 10.1177/0308022614562396

Publications

c.unsworth@cqu.edu.au

priscilla.harries@brunel.ac.uk

Professor Priscilla Harries

Professor Mary Hickson

Dr. Miranda Davies

Dr. Hulya Gokalp

Using fractional factorial

survey to develop an

effective training tool to

improve dietetic referral

prioritisations

Background

Study design

Expert policy capturing study →Training materials

Randomised Controlled Trial (RCT)

Final decision training aid

Content

Nutrition condition affects Effectiveness of medical treatment, quality and cost of care

Large numbers of referrals in adult acute care settings ; they need to be prioritised

Novice dietitians may struggle to prioritise referrals

An evidence based decision training tool can be developed and used to train the novices

Background

Case scenarios

Training Package

Policy Capturing

Study stages

RCTTraining aid

Website

i) What type of information is available to dietitians in dietetic referrals

ii) What kind of prioritization decisions are made?

iii) How do experienced dietitians weight the different information provided in dietetic referrals in order to prioritize them?

Phase I – Expert policy capturing

Questions to answer:

Case scenarios

Training Package

Policy Capturing

Cues / Levels

Cues / Cue Levels for dietetic referrals

Cue Cue Levels

Presenting complaint COPD*; Gastrointestinal C; Dementia; Pneumonia; UTI; Falls; Stroke; Dysphagia; Pressure ulcers

Nutrition status high-risk, at-risk or no-risk* of malnutrition

Reason for referral oral nutrition support*;specialist diet; enteral tube feeding; parenteral nutrition;dietary education

Previous food intake not eating; poor food intake; eating well*

Biochemistry picture Normal biochemistry; liver impairment; abnormal K+; re-feeding syndrome

Weight history Lost weight; gained weight; stableweight*

• In SPSS

• Unlikely cases were excluded• E.g. eating well, stable weight, BUT biochemistry picture indicating

refeeding syndrome

• 60 test cases

Fractional factorial design of test

cases

Sample case scenario

You have received a referral for a 65-year-old patient who may require dietetic assessment. The patient’s presenting complaint is Dysphagia; and they have screened as “High risk of malnutrition”. They have been referred for enteral tube feeding. The referrer reports that the patient is not eating and has stable weight. The biochemistry results

show normal biochemistry.

Non urgent – refer on to community dietetics

Non urgent – assess before discharge

Non urgent – assess within two working days

Urgent – assess on next working day

Urgent – assess today

When does this patient need to be seen for

assessment?:

Prioritisation options

Policy capturing study website

50 experienced dietitians to prioritise the 60 dietetic referral scenarios Inclusion criteria: min 6 months of experience as a dietitian

Multiple regression analysis was used to estimate weight of each cue level on the prioritisation decisions Dummy coding → Full regression model and reduced models

Analysis of experts’ decisions

• 50 dietitians : mean age 32 years, mean experience 7 years

• Agreement (consensus) between prioritisation decisions was very high: ICC= .98 (95% CI .97-.99).

Results

Results Full-regression results for influential cue levels

Cue Cue level B sr2

Presenting complaint Dysphagia .48 0.02

Nutrition status High risk .51 0.07

Reason for referral

Enteral tube feeding 1.01 0.28

Parenteral nutrition 1.18 0.29

Previous intake Not eating .39 0.04

Biochemistry picture

Re-feeding syndrome .74 0.13

Abnormal K+ .32 0.03

Weight history Lost W .23 0.02

model R2=0.957

Results

0

0.1

0.2

0.3

0.4

0.5

0.6

Reason Biochemistrypicture

Nutritionstatus

Presentingcomplaint

Previousintake

Weight history

Squ

ared

sem

i-p

arti

al c

orr

elat

ion

*

Cue

Results Importance of cue levels for the two most influential cues

Cue Cue Levels

Reason for referral 1. enteral tube feeding ≈ parenteral nutrition;2. specialist diet ≈ dietary education ≈ oral nutrition

support

Biochemistry picture 1. re-feeding syndrome2. liver impairment ≈ abnormal K+; 3. Normal biochemistry;

Phase 2

Training Package

RCT Training aid Website

• Development of study website

• Randomised Controlled Trial and analysis of data

• Development of final decision training website

Do the training materials improve capacity of novices

for making dietetic prioritisation decisions?

Randomised Control Design

Case scenario

decisions

Baseline40 scenarios

Case scenario

decisions

Post training20 scenarios

TrainingCase scenario

decisions

Case scenario

decisions

part

icip

ants

N=

197

115

82

intervention

control

82 82

69 69

Participants: pre-registration dietitians and recent graduates (1 year)

Results – Agreement between experts’ and novices’ decisions

ICC(2,1) between each novice’s decisions and experts’ consensus decisions

mean ICC(2,1) values for intervention and control at the two time points

Results

Pre-training

Post-training

Mean (pre –post)

Control 0.58 ↓ 0.53 -0.049

Intervention 0.59 ↑ 0.63 0.037

Cohen’s d (r) 0.54

Mean of ICC(2,1) values

• Significant main effect of group factor: F(1,149)=4.7, p<.01

• Significant interaction between time-point and group: F(1,149)=8.30, p<.01

• Simple effects: significant effect of group at post-training: F(1,149)=9.5,

p<0.01

ANOVA

Significant differences between the study groups at the post-training could be

attributed to the training intervention.

Decision training website

Dedicated website for training on dietetic referral prioritisation

Available since Jan 2016

TrainingPost

TrainingBaseline Feedback

https://cisbic.bioinformatics.ic.ac.uk/dietitian_training_aid/

• Evidence-based training materials are effective

• An open-access website for translation of the study into practice and for sharing best practice among novices

Conclusion

Thank you

&

ulster.ac.uk

The Administration of an Online Factorial Survey Using Qualtrics

SoftwareHelena Mc Elhinney (email: mc_elhinney-h@email.ulster.ac.uk)

Professor Marlene Sinclair

Professor Brian Taylor

SYNTHESIS OF THREE DATA

SOURCES TO DETERMINE

INDEPENDENT VARIABLES IN AN

ONLINE FACTORIAL SURVEY OF

IMPLANTABLE CARDIOVERTER

DEFIBRILLATOR DEACTIVATION

DECISIONSPresenter: Loreena Hill

Supervision team: Prof. B Taylor, Prof. S McIlfatrick & Prof. D Fitzsimons

Venue: Factorial Survey Methods in Health, Social and Ageing Research, London

15th June 2016

Outline

Clinical Context

Rationale for factorial survey design

Development of vignettes

Electronic dissemination of the survey

Results to date

Implications for clinical practice & methodology

Conclusions

Clinical Context

Implantable Cardioverter Defibrillator:

Implantations rates

Portrayed as ‘life-saving’ & ‘life-transforming’

Therapeutic role at end-of-life is contentious

Aging society: Multiple comorbidities such as renal failure or cancer

ICD prevents natural death

Decision-making: Patients’ choice limited due to lack of knowledge

Aim & Objectives

Aim:

To explore the perspectives of patients’, carers’ and professionals’ regarding ICD

deactivation and to examine the impact these have on clinical judgments about end-

of-life management.

Objectives:

1. To describe the understanding of heart failure patients with an ICD and carers

have regarding its deactivation at end-of-life.

2. To explore healthcare professionals’ practice and attitude towards ICD

deactivation.

3. To use this qualitative analysis to define the salient factors in ICD deactivation.

4. To evaluate the impact of these factors on professional judgement regarding

deactivation of an ICD at end-of-life.

Sequential Mixed Methods Design

Objective 3

9 Independent variables

Objective 1Systematic review of the literature (n=19)

Semi-structured interviews: patients (n=10) and carers (n=10)

Objective 2Retrospective case note review (n=44)

Focus Groups (n=9)

Objective 4

UK & Ireland factorial Survey

Rationale for Methodology

Ideal methodology

40+ years within research (Jasso & Rossi 1977, Ludwick et al. 2004, Taylor & Killick 2012)

Judgements are assumed to be “socially and individually structured”,

composed of guiding principles and values (Wallander 2009).

Sensitive research question

Factorial Study Design:

Novel & innovative to enhance recruitment

Manipulation of patient factors would be unethical & impossible within real-

life

Robust quantitative data

Complete anonymity ensuring authentic data collection

Electronic dissemination

Project Implicit:

US non-profit organisation, founded through by University of Washington,

Harvard & Virginia (1988)

Responsible for development of web-link and in real-time randomise IVs to

the stem vignette

Study instrument developed by the research team & uploaded onto

platform

200 vignettes were checked prior to dissemination

Online platform

Advantages:

Ease of access for busy healthcare professionals

Diverse sample in terms of geography disciplines

Minimal time to complete (7-10 mins), facilitated by “drop-down” menus

(Murray 2009)

Disadvantages:

Cost associated with using Project Implicit

Legal contract between Ulster University & Project Implicit

Vignette development

Independent variables (IV):

Seven (+/- two) within each vignette

Derived from practice knowledge, previous research & qualitative study (Ludwick 1999, Taylor 2006, Rattray et al. 2011)

Orthogonal (Wallander 2009) and clinically relevant

Each IV had between 2 - 5 levels, enhancing external validity.

(Excessive numbers can add unnecessary complexity and reduce statistical

power of analysis (Taylor & Zeller 2007))

Nine IVs in mathematical terms (5x3x3x3x4x3x3x3x3) equated to a potential

of 43,740 different vignettes= vignette universe

Vignette development (2)

Dependent variables (DV):

All participants answered 2 questions or DV on the clinical scenario

presented

Questions addressed both “likelihood” to make a decision and the

“confidence” in the decision just made

Corresponding responses were recorded on an ordered Likert scale with

scores ranging from zero to ten, anchored by descriptive terms (Lauder 2002)

Preliminary Analysis

Possible

variables

Levels Values Evidence from:

Literature Retrospective Case studies

case note review

Age 4 29-49

50-69

70-79

80-90

Null

Thylen, et al. 2013,

Fluur,, et al. 2013,

Groarke, et al. 2012,

Tajouri, et al. 2012,

Strachan, et al. 2011

Mean age 73

years

Focus Group 2 Dr:

“She is a bit

younger so it

would be difficult”

49 years is too

young

Co-Morbidities 4 Bowel cancer

Advanced renal disease

Dementia

Null

Stewart et al 2010,

Dodson et al 2013,

Buchalter et al 2014

Tajouri, et al 2014;

Kelley, et al 2009;

Marinski, et al 2010

12 patients had a

malignancy

Patient 4: Suffering

equated with

cancer and pain

with no mention of

HF symptoms

(NYHA IV)

Number of shocks 3 None

1-5

>10

Raphael et al. 2011,

Goldstein et al. 2008,

Stewart et al 2010

Ten patients had a

documented shock

and 4 multiple

shocks. Patients

who had never

experienced a

shock were more

likely (45% :15/33)

to undergo

deactivation.

Focus Group 5

Nurse: “ he thinks

he is getting

benefit from it and

he would like it left

on”. Carer 2:

linked shocks at

EOL to suffering.

Patient 3: “I would

hate to die

prematurely over

the sake of

Example of a vignette

You review a 59 year old female with moderate heart failure (NYHA III),

advanced renal failure. She has had 1 admission over the past year and

has experienced more than 1 shock. Medical records show no previous

discussion about deactivation with documented management plan to be

continue present treatment. The patient lives alone with no family or

friends.

1. What is the likelihood that you would discuss ICD deactivation with this

patient?

Not at all likely 0 1 2 3 4 5 6 7 8 9 10 very likely

2. How confident are you in the decision you have just made?

Not at all confident 0 1 2 3 4 5 6 7 8 9 10 very confident

Study instrument

Study instrument:

Short demographic questionnaire

One standard vignette (monitor the range of participants’ responses,

increasing internal & external validity; Lauder 2002)

Six vignettes with randomly assigned nine independent variables

Last case scenario (evaluate clinical validity of the study’s vignettes)

Sample of Questionnaire

Vignette & DV One within survey

Vignette & DV Two within survey

Analysis

Each vignette is a unit of analysis

Software: SPSS (version 21), significance was p=<0.05

Two levels of analysis:

Patient factors (vignette data)

Participant factors ( questionnaire data)

Multivariate regression analysis (Wallander 2009)

i.e. variance in the DV that can be explained by a IV

ANOVA with Least Significant Difference (LSD) posthoc

i.e. strength & direction of significance

Implications for methodology

To my knowledge 1st FS to synthesis 3 sources of evidence to develop

Independent variables

Systematic, rigorous and transparent process

Outcome: robust quantitative data on professional attitudes to

deactivation

International interest, with post-doc data to enable comparisons across

countries

A Tool That Produces Judgment Study Stimulus Case Sets

With Specified Cue-Cue Correlations.

Robert M. Hamm, PhD

Presentation at Factorial Survey Methods Conference, Brunel University, Uxbridge UK, June 15 2016

Department of Family and Preventive MedicineUniversity of Oklahoma Health Sciences Center

Overview

• Judgment studies

– We model judgments of sets of stimulus cases

• Stimulus sets

– Independent or correlated cues?

• Construction of stimulus sets with specified cue correlations

– Demonstration of spreadsheet

Judgment studies• Policy capturing

– Describe subject’s/judge’s use of cues– Possibly compare to “right way” to use cues

• Brunswik Lens Model – Describe cue use – Relate judge’s cue use to cue usefulness measured in

model of the ecology – The ecology model needs a criterion

• The “right answer” for each case

Stimulus sets

• Select appropriate cues– Cues relevant and important for task

– Vary over realistic ranges

– Number of levels for each cue

• Optional: definition of correct judgment– Statement of principles of good judgment

– Observation of true answer (“criterion”) • allows cue-criterion correlations (needed for lens model studies)

• Cue-cue correlations– Independent, orthogonal – factorial design?

– Or correlated?

Cue-Cue Correlations

• Efficiency

– Use of uncorrelated cues is more efficient

• “Design efficiency: (Glossary, Auspurg and Hinz, 2015)– Measure of the goodness of an experimental design.

– Good designs provide maximum information (minimum variance and covariance) for intended parameter estimates.”

• Easier to fit models– Factorial and “fractional factorial” ANOVA

• Smaller N of cases needed for a good fit

Cue-Cue Correlations

• Representative of the task ecology– Cue-cue correlations in stimulus set should be similar

to cue-cue correlations in the ecology.• Orthogonal design may produce unrealistic cases.

• “Vicarious functioning” – when cues are correlated, does not matter which one is attended.

• Judge may experience stimuli from orthogonal stimulus set as anomalous

– unusual cue combinations –

and adopt a different judgment strategy than usual.

• The descriptive analysis many not be able to reveal successful judgment strategies.

Analysis of Judgment of Stimulus Sets with Representative Cue-Cue Correlations

• Typically uses multiple linear regression.

– Could also use ANOVA

• But regression specifies relative weights

– Could also use multiple logistic regression (if judgment is dichotomous)

Example: Risk that ER patient will deteriorate

• Patient very ill, in emergency room– What is risk he or she will need resuscitation?

• Guidelines name the key predictors– Heart rate, respiration rate, temperature, level of

consciousness, systolic blood pressure– Could make up the cases, varying cue values

independently• Easy analysis

• Set of cases available (Subbe, 2001)– Could sample from them

• Representative of ecology

Example (continued)

• A blend between the “making up the cases with desirable cue correlations” and “sampling from real cases” was chosen:– For efficiency, 50% of the stimulus cases had a bad

outcome– Sample within outcome to get the cases

• What do you think of this strategy? How do you think it worked?– With respect to efficiency?– With respect to representativeness of cue

intercorrelations?

Cue Correlations in the EcologyData set – 555 patients, 57 “at risk”

• Cue correlations in data setCues Systolic

BP

Heart

rate

Resp.

rate

Temper

- ature

Conscious-

ness

Ecology

Criterion

Systolic BP ~

Heart rate .004 ~

Respiration

rate

.014 .284*** ~

Temperature -.047 .223*** .126** ~

Consciousness -.145** .069 .116** .088* ~

Ecology Criter. -.171*** .124** .179*** -.031 .161*** ~

Cue Correlations in Sample20 cases, 10 “at risk”

• Tolerance is 1 – R2, the complement of the proportion of variance explained in a multiple linear regression predicting the cue from all the other cues.

Cues Systolic

BP

Heart

rate

Resp.

rate

Temper

- ature

Conscious-

ness

Ecology

Criterion

Tolerance

Systolic BP ~ .072 .032 .558* -.616** -.147 .25

Heart rate .004 ~ .507* .493* .485* .455* .47

Respiration

rate

.014 .284*** ~ .353 .542* .594** .45

Temperature -.047 .223*** .126** ~ .007 .023 .43

Consciousness -.145** .069 .116** .088* ~ .523* .18

Ecology Criter. -.171*** .124** .179*** -.031 .161*** ~

Tolerance .98 .88 .91 .94 .96

Strategy did not reach either goal

• Failed to have efficiency

– very high cue intercorrelation

– Some subjects’ dichotomous judgments could not be modeled with logistic regression

• Failed to have representativeness

– More “at risk” cases than would typically be seen

Why were the cue-cue correlations so high?

• When we adjusted the prevalence of the criterion

– from 10% to 50%

it increased the cue-criterion correlations.

• When cues are correlated with a common variable (the criterion), we can expect the cues to be correlated with each other.

Sample or Construct the Stimulus Set?

• Barriers to using stimulus sets with correlated cues– Sampling: You need data on cues in the ecology

• Who has such data?

• If sample from data set, check the sample’s correlations.

• Look at the cue–cue correlations in the sample (and the cue-criterion correlations, if it is to be a lens model study),

• Make sure they are close to the correlations in the whole data set; if not, sample again.

Barriers to constructing a stimulus set with correlated cues

• Construction

– Who knows what the cue-cue correlations should be, without data?

• Ask experts? Refer to publications?

– Who knows how to construct a set of cases with the specified correlations?

• I developed an Excel spreadsheet to do it.

Spreadsheet strategy

• User specifies stimulus set features

– Including cue-criterion correlations

• Excel projects cue-cue correlations

– User can specify adjustments to cue-cue correlations

• Excel constructs stimulus set, random generation

– Computes cue-criterion and cue-cue correlations

– Compares with expected and projected correlations

• User inspects and uses it if “close enough”

Specify stimulus set

– N cases

• # with each level of criterion

– # cues

• # levels per cue

– Cue-criterion correlations

– Cue-cue correlations

Basics for any stimulus set

• Features

• Correlations with criterion

– “Cue 1” is the criterion.

Cue-cue correlations

• One factor

• Several factors

• Also correlated factors

Spreadsheet projects cue-cue correlations

• If two cues are each correlated with the criterion, they’ll be correlated with each other

r(cue 2, cue 3)=r(cue 2,crit)*r(cue 3,crit)

• E.g., if only specify cue-criterion correlations

Spreadsheet also projects cue-cue correlations

• E.g., if specify some cue-cue correlations

Spreadsheet constructs cases• For cue-criterion correlations only

• Each cue value a mix of the criterion value and random.

• Continuous values (scale -0.5 to +0.5) chopped into ordered levels

Spreadsheet computes correlations• Correlation among continuous cues

• Correlations among ordered category cues

• Comparison with specified and projected correlations

Is this sample good enough?

• A random sample’s cue-cue correlations do not necessarily resemble the intended correlations– Among all the possible samples, one could even

find one where all cue-cue correlations are 0. • Extremely rare.

– (but this is what factorial surveys often use)

– So you have to inspect. • Is this close enough to expected/desire cue-cue

correlations?

Extensions, use, availability

• Extension: – Class–conditional dependence. In principle and

probably in practice, the intercorrelations among “cases” may differ from the cue-cue correlations among “non-cases”.

– The worksheet has a sheet that accommodates this.

• Use:– One study has used this, so far. Data not in.

• Availability: – CueCorrels.xlsx– Robert-Hamm@ouhsc.edu

Opportunities

• Opens up possibility of producing stimulus sets with chosen cue-cue correlations

– (approximately)

• Allows researchers to address considerations of representative design of stimulus sets

– (also called “ecological validity” of the stimuli)