September 28, 2015 Webinar Leslie Cunningham-Sabo, PhD, RDN

Associate Professor Dept. Food Science & Human Nutrition, Colorado State University

At the end of this webinar attendees will be able to:

Describe at least 3 challenges researchers face when developing valid measures of child dietary intake

Distinguish the definitions and processes to measure face and content validity

Identify the purpose and effective strategies for conducting cognitive interviews with children

Describe at least 1 strategy to assess the sensitivity of an instrument to measure real change

Validation of a group-administered pictorial dietary recall with 9- to 11-year-old children

Instrument: Day in the Life Questionnaire – Colorado (DILQ -CO)

Child’s age ◦ Recall and cognitive ability; 7+ Y.O. generally accepted

Errors include omissions (exclusions) and intrusions; amount eaten

Retention interval and prompts (Baxter, et al., 2015)

◦ Parent/caregiver as proxy – may not observe all meals/snacks (Byers, et

al., 1993)

Usual methods ◦ 24-hr recall, usually 2 week days, 1 weekend day, multiple pass

◦ FFQ and screeners (Hunsberger, et al, 2012)

◦ Method/s selected dependent upon nutrients or dietary groups/patterns of interest

Instrument validity – ability of the instrument to measure what is intended ◦ 4 types ◦ Subsequent webinars will discuss construct and criterion validity

Content – confirm all items reasonably represent larger domain ◦ Clarify concept/construct through literature review ◦ expert (panel) review

Face – determine language, format & procedures are understandable & reasonable to target audience ◦ testing with target audience ◦ When assessing for children can also include adults (parents, teachers)

(Contento, 2015; Sharma & Petosa, 2014)

Purpose ◦ Assess language, format & processes are understood by target audience ◦ One way to assess face validity

Processes ◦ Done individually, not in groups ◦ Administer instrument as intended ◦ Followed by ‘think-aloud’ and probing questions to assess comprehension

and any missing content (e.g., response options)

(Beatty & Willis, 2007; Drennan, 2003; Shafer & Lohse, 2005)

Awareness of developmental stages of cognitive abilities is critical; influenced by age, experience ◦ Age of 7 generally accepted as earliest to interview ◦ Language expands, reading skills acquired, distinguish points of view

Youth continue to develop cognition and communication skills through adolescence and early adulthood

Establish trust; undistracted & unhurried environment

Model type of questions to be asked; probing necessary (de Leeuw, Borgers & Smits, 2004)

Does the instrument detect a true condition? ◦ Often discussed for biological tests in comparison to specificity

◦ Determined by ‘true positives’ divided by sum of true positives and false negatives (TP / TP + FN)

(Sharma & Petosa, 2014)

Rationale – identify feasible method to collect children’s individual self-report of FV intake in a group (classroom) setting

Original DILQ ◦ Development – yesterday’s report of food intake and physical activities

◦ Face validity with focus groups

◦ Performed well/acceptably for

reliability, validity, sensitivity

(Edmunds & Ziebland, 2002)

Similar rationale to original DILQ; assess all food groups

Purpose – ◦ Assess content & face validity of DILQ adapted to Denver, CO 4th-

graders – DILQ-CO

◦ Assess feasibility of administration and analysis of DILQ-CO

◦ Determine DILQ-CO’s sensitivity to novel eating occasion & accuracy of children’s recall of school lunch

Procedures ◦ Child nutrition education research expert review

Content, order of items, language & graphics (US v. UK)

How to include estimation of amounts of foods eaten?

Is forward or reverse recall (end/start after lunch/recess) more comprehensible? (Baxter et al., 2003; Moore et al., 2007)

◦ Elementary teacher review Language, format, feasibility of classroom administration

◦ Cognitive interviews with children of similar age & demographics DILQ item interpretation & comprehension

Format, instructions

Ability to quantify amounts of foods eaten

Wording, phrasing & item order ◦ For example replaced “pudding” with “dessert” & “school dinner, pack lunch” with

“lunch from home, school lunch”

◦ Made embedded instructions more direct (e.g., “check box”)

Added semi-quantitative scale for portions of recalled items ◦ Circle = ate all, √ = ate more than 1/2, x = ate less than 1/2, line through = ate

none

Created 2 versions – either could be administered after lunch/recess ◦ Forward recall order (yesterday to today); reverse recall order (today to yesterday)

Adapted original DILQ administration protocol

10 4th-grade students from school with similar demographic ◦ 8 girls, 2 boys; ½ received forward v reverse format

Applied standardized interview protocol

In general students enjoyed & comprehended DILQ-CO ◦ Including quantification scale

◦ Memory perceived as better with reverse recall order (from lunch/recess today backwards to after lunch/recess yesterday); supported by literature (Baxter, et al. 2003, Baxter, et al., 2009)

Results indicated minor revisions to wording and graphics ◦ E.g., changed front yard gate to house door; evening meal to dinner

2 low-income Denver, CO schools; 1 randomly assigned to receive morning fresh fruit snack (FSS, NFSS)

125 4th graders in 6 classrooms ◦ Power calculation to detect moderate effect size indicated sample >

115

> 77.5% eligible for F/R-priced lunches; 51% male; > 77% Hispanic

DILQ-CO completed after lunch/recess 35 minutes to complete per classroom; few problems or

questions

Procedures ◦ FSS students were offered pre-portioned banana or grapes for morning

snack

◦ We observed and recorded amount wasted

Analysis ◦ Averaged reported instances of fruit snack between schools

◦ Man-Whitney U test

(Penkilo & Hoelscher, 2008)

Procedures ◦ Observation of lunch (recorded item selection/brought from home)

◦ Weighed plate waste

◦ Compared against standard weight samples (school lunches)

Analysis ◦ Food selection/plate waste compared to DILQ-CO for school lunch

◦ Matches = item recorded/not recorded on both data sources

◦ Exclusions = item missing from DILQ-CO

◦ Intrusions = item only recorded on DILQ-CO (inaccurate recall)

◦ Sensitivity = percent matches/(percent matches + percent exclusions)

Students accurately recalled “instances” of morning fruit intake:

FSS 1.18 + 0.93 v NFF 0.02 + 0.13 (mean + SD; P < .001)

DILQ-CO demonstrated high level of validity & sensitivity compared to plate waste (Table 1) ◦ K coefficients of substantial to almost perfect agreement for entrees,

fruit, vegetables, bread, and milk

◦ Only dessert (cookie) had poorer agreement

Accurate recall for specific fruit & vegetable types

Significant correlations between quantified & weighed amounts for 6 of 10 food items (P < .05)

Poorer correlations with vegetables, turkey wrap, milk, bread

No differences by gender or school

Feasible DILQ-CO classroom administration

Students were quite accurate with recall of recent lunch & fruit snack selections ◦ recall of more distal timeframes unknown

Amounts consumed were less accurate & varied by food type ◦ No clear pattern; this warrants further investigation

Challenges with analyses – extensive time to score/code

Inconsistent validation (studies) using original DILQ Moore, et al., 2007 reported comparable k coefficients & percent matches

Lim, et al., 2015, results do not support validity of DILQ

Recommendations for collecting children’s self-report ◦ Reverse recall for immediate 24 hour period

◦ Parent-involved report for non-school meals

Strategies to measure FV change that don’t rely on self-report Plate waste (Smith & Cunningham-Sabo, 2014)

Skin carotenoid assessment of FV intake (Mayne, 2013; Aguilar, 2014)

Article co-authors ◦ Victoria (Wallen) Lee

◦ Garry Auld

◦ Cathy Romaniello

Graduate students ◦ Tessa Komine – cognitive interviewing resources

◦ Alicia Grove – presentation preparation

Aguilar, Sheryl S., et al. "Skin Carotenoids: A Biomarker of Fruit and Vegetable Intake in Children." Journal of the Academy of Nutrition and Dietetics 114.8 (2014): 1174-1180.

Baxter, Suzanne D., et al. "Effectiveness of prompts on fourth-grade children’s dietary recall accuracy depends on retention interval and varies by gender." The Journal of nutrition 145.9 (2015): 2185-2192.

Baxter, Suzanne Domel, et al. "Fourth-grade children's dietary recall accuracy is influenced by retention interval (target period and interview time)." Journal of the American Dietetic Association 109.5 (2009): 846-856.

Baxter, Suzanne Domel, et al. "Reverse versus forward order reporting and the accuracy of fourth-graders’ recalls of school breakfast and school lunch."Preventive medicine 36.5 (2003): 601-614.

Beatty, Paul C., and Gordon B. Willis. "Research synthesis: The practice of cognitive interviewing." Public Opinion Quarterly 71.2 (2007): 287-311.

Byers, Tim, et al. "The accuracy of parental reports of their children's intake of fruits and vegetables: validation of a food frequency questionnaire with serum levels of carotenoids and vitamins C, A, and E." Epidemiology 4.4 (1993): 350-355.

Contento, Isobel R. Nutrition education: linking research, theory, and practice. 3rd ed. Jones & Bartlett Learning, 2015.

de Leeuw, E., Borgers, N., & Smits, A. (2004). Pretesting questionnaires for children and adolescents. In S. Presser, J. M. Rothgeb, M. P. Cooper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.). Methods for testing and evaluating survey questionnaires. Hoboken,NJ: Wiley & Sons. 409-29.

Drennan, Jonathan. "Cognitive interviewing: verbal data in the design and pretesting of questionnaires." Journal of advanced nursing 42.1 (2003): 57-63.

Edmunds, L. D., and S. Ziebland. "Development and validation of the Day in the Life Questionnaire (DILQ) as a measure of fruit and vegetable questionnaire for 7–9 year

olds." Health Education Research 17.2 (2002): 211-220.

Hunsberger, Monica, et al. "Relative validation of Block Kids Food Screener for dietary assessment in children and adolescents." Maternal & child nutrition 11.2 (2015): 260-270.

Lim, Sue Sing, et al. "Validation of 2 Brief Fruit and Vegetable Assessment Instruments Among Third-Grade Students." Journal of nutrition education and behavior 47.5

(2015): 446-451.

Mayne, Susan, et al. “Resonance Raman spectrosopic evaluation of skin carotenoids as a biomarker of carotenoid status for human studies.” Archive of Biochemistry and

Biophysics. 539 (2013) 163-170.

Moore, G. F., et al. "Validation of a self-completion measure of breakfast foods, snacks and fruits and vegetables consumed by 9-to 11-year-old

schoolchildren." European journal of clinical nutrition 61.3 (2007): 420-430.

Penkilo, Monica, Goldy Chacko George, and Deanna M. Hoelscher. "Reproducibility of the School-Based Nutrition Monitoring Questionnaire among fourth-grade

students in Texas." Journal of nutrition education and behavior 40.1 (2008): 20-27.

Shafer, K., and B. Lohse. "How to conduct a cognitive interview: A nutrition education example." US Department of Agriculture, National Institute of Food and

Agriculture (2005).

Sharma, Manoj, and Rick Lingyak Petosa. Measurement and evaluation for health educators. 2nd ed. Jones & Bartlett Publishers, 2014.

Smith, Stephanie L., and Leslie Cunningham-Sabo. "Food choice, plate waste and nutrient intake of elementary-and middle-school students participating in the US

National School Lunch Program." Public health nutrition 17.6 (2014): 1255-1263.

Wallen, Victoria, et al. "Validation of a group-administered pictorial dietary recall with 9-to 11-year-old children." Journal of nutrition education and behavior 43.1

(2011): 50-54.

Cognitive interview protocol and

DILQ administration protocol available upon

request

Leslie.Cunningham-Sabo@Colostate.Edu