Inter-observer Reliability

Despite the well-known benefits of consuming fruits and vegetables (F/V),1 only about 20% of children report consuming the recommended five or more servings of F/V per day.2,3 Schools are frequently targeted as a place to improve children’s dietary habits, and multi-component school interventions such as Farm to School aim to increase children’s F/V consumption. However, methodology is needed that can accurately assess children’s F/V consumption in school. Children’s self-reports using instruments such as surveys, questionnaires, and dietary recalls are subject to error since children may misidentify or forget to report foods (omissions), identify uneaten foods (intrusions), and have a difficult time estimating portion sizes.4 Overweight and obese children and their parents may be especially prone to misreporting energy and F/V intake.5,6 Accurate measurement of children’s food intake can be obtained using the “gold standard” of weighed plate waste, but this method is time and labor intensive.7 Two other methods – direct observation and digital imaging – have been used to estimate consumption by observation, but validation of these methods has generally been limited to studies in environments more controlled than the busy school cafeteria, such as the laboratory8 or hospital.9 Earlier studies of direct observation validated the method against weighed plate waste in a school cafeteria,10 but present-day school cafeterias pose unique challenges.

•  A research team of undergraduate nutrition students (n=19) completed 12 hours of training on three methods for measuring children’s F/V consumption during school lunch: weighed plate waste (WPW), direct observation (DO), and digital imaging (DI).

•  The feasibility of each method and inter-observer reliability of DO and DI were tested in the laboratory and school cafeteria.

•  Data were collected from school lunch trays (n=521) of third-, fourth-, and fifth-graders in two Vermont school cafeterias. Each lunch tray was assigned a colored, numbered sticker, with one color observed during each visit.

•  F/V consumption was determined as the difference between mean baseline weights of the foods selected and plate waste for each lunch tray. For WPW, plate waste was weighed to the nearest gram. For DO and DI, a six-point scale was used to estimate the percent consumed of each F/V item selected.

•  The University for Vermont Institutional Review Board approved the study.

The development of a reliable and feasible measure of school children's fruit and vegetable consumption

Jennifer C Taylor, BS, Bethany A Yon, PhD and Rachel K Johnson, PhD, MPH, RD Nutrition and Food Sciences Dept., University of Vermont, Burlington, VT

ABSTRACT

Most children fail to meet recommended intakes for fruits and vegetables (F/V). Farm to School programs have been proposed as a strategy to increase consumption of these foods, but methodology is needed that can accurately measure intake. F/V consumption can be accurately measured during school lunch using weighed plate waste (WPW), but this method is rarely adopted because it is time and labor intensive. Direct observation (DO) and digital imaging (DI) are more feasible, but these methods have not been validated against WPW in present-day cafeterias. This study tested the reliability and feasibility of DO and DI for measuring F/V consumption against the gold standard of WPW in two Vermont school cafeterias. 521 lunch trays were collected over ten school visits; 339 of the lunch trays were observed using WPW, 218 using DO, and 278 using DI. Inter-observer reliability (IOR) was high for both DO and DI; percent agreement scores for IOR were ≥89% and intraclass correlations were ≥0.70. DO and DI are feasible and reliable methods for measuring F/V consumption during school lunch. Further data analyses will test the validity of DO and DI against WPW.

BACKGROUND

PRELIMINARY RESULTS

This study was funded by the USDA - Vermont Agricultural Experiment Station. The authors thank Alan Howard and Leah Conchieri for their support with data and statistical analyses.

METHODS

References

1.  Bazzano LA. The high cost of not consuming fruits and vegetables. J Am Diet Assoc. 2006;106(9):1364-8. 2.  Robinson-O’Brien R, Burgess-Champoux T, Haines J, Hannan P, Neumark-Sztainer D. Associations between school meals

offered through the national school lunch program and the school breakfast program and fruit and vegetable intake among ethnically diverse, low-income children. J Sch Health. 2010;80(10):487-492.

3.  Guenther PM, Dodd KW, Reedy J, Krebs-Smith SM. Most Americans eat much less than recommended amounts of fruits and vegetables. J Am Diet Assoc. 2006;106(9):1371-9.

4.  Livingstone MBE, Robson PJ, Wallace JMW. Issues in dietary intake assessment of children and adolescents. Br J Nutr. 2004;92(Suppl. 2):S213-S222.

5.  Burrows TL, Martin RJ, Collins CE. A systematic review of the validity of dietary assessment methods in children when compared with the method of doubly labeled water. J Am Diet Assoc. 2010;110(10):1501-10.

6.  Burrows TL, Warren JM, Colyvas K, Garg ML, Collins CE. Validation of overweight children’s fruit and vegetable intake using plasma carotenoids. Obesity (Silver Spring). 2009;17(1):162-8.

7.  Kirks B, Wolff H. A comparison of methods for plate waste determinations. J Am Diet Assoc. 1985;85(3):328-31. 8.  Williamson DA, Allen HR, Martin PD, et al. Comparison of digital photography to weighed and visual estimation of portion sizes.

J Am Diet Assoc. 2003;103(9):1139-45. 9.  Connors PL, Rozell SB. Using a visual plate waste study to monitor menu performance. J Am Diet Assoc. 2004;104(1):94-6. 10. Comstock EM, St Pierre RG, Mackiernan YD. Measuring individual plate waste in school lunches. Visual estimation and

children’s ratings vs. actual weighing of plate waste. J Am Diet Assoc. 1981;79(3):290-296.

CONCLUSIONS

RESEARCH OBJECTIVE To establish a reliable and feasible method for measuring school children’s consumption of F/V during school lunch in a variety of school cafeteria settings.

Data on F/V consumption were successfully collected using three assessment methods, and DO and DI were reliable for measuring F/V consumption. Unique challenges in each cafeteria affected the final number of lunch trays collected. Challenges included students’ selection of second servings of soup, F/V items missing at the end of the meal without evidence of consumption (e.g. apple cores), and students’ and lunch aides’ disposal of trays prior to final imaging, estimations, or weighing. Additional analyses will be conducted to test the validity of DO and DI estimations against the gold standard of WPW. DI may be especially promising since it requires the least time and labor to collect data on F/V consumption.

Data collected during the ten school visits

Distributed 662 lunch trays distributed

1104 maximum possible observations Incomplete Data

141 lunch trays removed (21.3% of distributed)

269 tray observations removed (24.4% of distributed)

No F/V Selected 85 lunch trays removed (12.8% of distributed)

126 tray observations removed (11.4% of distributed)

Weighed Plate Waste 277 F/V tray

observations collected

Direct Observation 199 F/V tray

observations collected

Digital Imaging 233 F/V tray

observations collected

Tray Observations 521 lunch trays collected

835 tray observations collected

F/V Tray Observations 436 F/V lunch trays collected

709 F/V tray observations collected

Raters Intraclass correlations Percent agreement Traysa Reliability coefficient

(confidence interval) F/V itemsb

(comparisons)c IOR

scored Direct observation

Training 19 10 0.99 (0.98, 1.00) 51 (2493) 94% Visit 1 4 30 0.93 (0.86, 0.97) 57 (103) 96% Visit 2 5 35 0.73 (0.55, 0.85) 61 (65) 89% Visit 3 3 25 0.98 (0.95, 0.99) 45 (45) 98% Visit 4 2 21 0.95 (0.88, 0.98) 48 (48) 100%

Digital imaging

Visit 1 2 31 0.98 (0.96, 0.99) 55 (55) 98% Visit 2 2 24 0.97 (0.93, 0.99) 44 (44) 96% Visit 3 2 30 0.98 (0.96, 0.99) 69 (69) 100% Visit 4 2 47 0.99 (0.98, 0.99) 47 (47) 98% Visit 5 Visit 6

3 44 0.99 (0.98, 0.99) 73 (219) 98% 3 55 0.97 (0.95, 0.98) 113 (339) 92%

aObserver estimations for total F/V consumption per lunch tray bObserver estimations for individual F/V items cIn many instances, the same F/V item was estimated by more than two observers, allowing for multiple comparisons dPercentage of estimations within one-quarter serving of another observer’s estimation

Analyses

Tests of inter-observer reliability (IOR): •  Percent agreement assessed the consistency of researchers’ plate

waste estimations for individual F/V items using DO or DI. Observers were in agreement if estimations were within one-quarter serving of each other. A score of at least 85% agreement was expected during each school visit.

•  Intraclass correlations tested reliability of researchers’ estimations for F/V consumption per lunch tray using DO or DI.

Analyses were conducted using SAS System for Windows (version 9.3, 2010), and Microsoft Excel (Redmond, WA, 2007)