GEM No.Assessment of Student Experience in a Problem-BasedLearning Course Using the Course Experience QuestionnaireCristen Harris, PhD, RD, CSSD; June Kloubec, PhD

INTRODUCTION

Problem-based learning (PBL) hasbeen described as ‘‘the learning thatresults from the process of working to-wards the understanding of a resolu-tion of a problem.’’1 This pedagogicalapproach, based in the theoreticalframework of constructivism,2 is botha method and a philosophy that aimsto build knowledge by workingthrough (in an active, iterative, andself-directed way) a progressive frame-work of problems via facilitated small-group work and independent study.3

The PBL approach uses open-endedproblems or case scenarios that areconsidered ill-structured in that theymay havemultiple solutions or requiremore information for understandingthe problem than is initially avail-able.4 This compels students to buildon prior knowledge, integration, andcritical thinking3 and to practiceacting as professionals in seeking solu-tions. Because the problems in thesecase scenarios often pose no singlecorrect solution, an atmosphere ofcooperative learning can be facilitated.

Because nutrition professionals takeon a wide variety of occupations, stu-dents require the development of aglobal skill set that may not be specificto nutrition. A PBL approachmay facil-itate the development of self-directedlearning, reflective thinking,5 leader-ship, teamwork, cooperation, self-confidence, critical thinking, resourceidentification, and knowledge reten-tion.6 These skills may help nutritiongraduates enter the job market.

Department of Nutrition and Exercise Science,Bastyr University, Kenmore, WAAddress for correspondence: Cristen Harris, Phand Exercise Science, School of Natural Hea14500 Juanita Dr NE, Kenmore, WA 98028; PE-mail: charris@bastyr.eduJ Nutr Educ Behav. 2014;-:1-5�2014 SOCIETY FOR NUTRITION EDUChttp://dx.doi.org/10.1016/j.jneb.2013.12.002

Journal of Nutrition Education and Behav

Although the use of PBL is wide-spread in the preclinical curricula ofUS medical schools,7 there is limitedpublished literature on the use ofthis approach and outcomes in nutri-tion curricula. Furthermore, long-term assessments of the perceivedeffectiveness and satisfaction ofPBL—after graduates are in the jobmarket for a year—are lacking. Previ-ous research suggests that a problem-based education may help nutritionprofessionals respond successfully totheir own professional developmentneeds, in addition to those of otherhealth care professionals8,9 whosetraining would more likely include aPBL approach.5 Therefore, the objec-tives of this project were to (1)uniquely implement a PBL approachin a capstone course in an undergrad-uate nutrition program; and (2)collect both short-term (eg, grades)and long-term post-course data evalu-ating students' perceptions and satis-faction with the teaching andlearning in such a course.

IMPLEMENTATION

A capstone course, ‘‘Nutrition, Phys-ical Activity and Disease,’’ was taughtusing the PBL method during thespring quarter of 2012. Since 2011,this particular course was requiredof students majoring in 3 undergradu-ate nutrition programs—Nutrition,Nutrition and Exercise Science, andNutrition and Culinary Arts—at a pri-vate university in the northwestern

School of Natural Health Arts and Sciences,

D, RD, CSSD, Department of Nutritionlth Arts and Sciences, Bastyr University,hone: (425) 602-3098; Fax: (425) 823-6222;

ATION AND BEHAVIOR

ior � Volume -, Number -, 2014

US. The learners were primarily youngadults and the PBL approach was thebasis for the entire 11-week course.At the first class session, the instruc-tors facilitated the development ofclassroom norms and ground rulesfor group work, including the estab-lishment of attendance policies andthe consequences for rule violation.Each week of the term, students werepresented with a different topic, ordisease state (Table 1). Instructorscreated multiple case studies on eachtopic each week. For each topic, stu-dents (n ¼ 36) were arbitrarily dividedinto 6–9 groups of 4-6 students each,and each group was assigned to a spe-cific case. Although there was noexpectation regarding where thework was to be done, groups weretypically given 2 2.5-hour class pe-riods to complete each assignment,during which time instructors rotatedamong groups to pose or answer ques-tions and facilitate progress. Assign-ments for each case varied andincluded written reports, oral presen-tations, posters, presentations withadded technology (eg, video), andgame creation. For each case, studentswere encouraged to answer the ques-tions ‘‘What do I already know?’’ and‘‘What do I need to find out?’’Detailed elements, including nutri-tion and physical activity preventionand management components, wererequired for each assignment relatedto each case study (Table 2). Processand summative assessments that inte-grated course objectives were adaptedfor instructor, peer, and self-evalua-tions.10 Therefore, eachweek, studentsworked on a different topic with adifferent group of peers; and presentedand were evaluated in a differentmanner (Table 1). Course grades foreach student were calculated frominstructor, peer, and self-evaluationsof assignments throughout thequarter.

Varied evaluation methods meantthat students were challenged tointerpret and present their resultsin different formats each week.

1

Table 1. Example Schedule of Topics for an Undergraduate Nutrition Capstone Course

Topic/Disease Specific Case Topics Assignment Evaluation of Student Effort

Cardiovascular disease Hypertension Written report InstructorCongestive heart failureStroke

Diabetes Type 1 diabetes Poster PeerType 2 diabetesGestational diabetes

Lower gastrointestinal disease Celiac disease Presentation SelfCrohn diseaseIrritable bowel syndrome

Upper gastrointestinal disease Peptic ulcer disease Written report PeerGastroesophageal refluxHiatal hernia

Food allergies and intolerances Gluten intolerance Game Instructor, peer, and selfLactose intoleranceShellfish allergy

2 Harris and Kloubec Journal of Nutrition Education and Behavior � Volume -, Number -, 2014

One unique example of this was thecreation of a game designed for thefood allergies/intolerances topic. In-structors set several parameters forthe games students created including(1) number of players (5–6), (2) timeto complete the game (10–15 mi-nutes), and (3) integration of multipleterms related to food allergies or intol-erances from a list provided (eg,anaphylaxis, food challenge, naturalkiller T-cells, mast cells). One classperiod was devoted to game creationand the subsequent class was used toplay the games. Each group playedthe other groups' games and ratedthem on scientific accuracy andassimilation into the game, as well ascreativity, fun, flow of material, andease of interpretation and play. Theprofessors rated each game according

Table 2. Required Elements for Each Assi

Element

Etiology of the disease (Who gets it and wh

Prevention (How can it be avoided?)

Pathophysiology (What went wrong?)

Management (How can I help?)

to scientific accuracy and inclusionof requisite terms. At the conclusionof the second class period, studentscompleted a self-assessment of theirperformance in their group.

The original course took place be-tween April 2 and June 13, 2012. TheCourse Experience Questionnaire(CEQ) was administered in an onlineformat 1 year after the start of thecourse, between March 20 and April5, 2013. The 23-item, short-formCEQ was chosen for a 1-year, post-course assessment because the itemsalign well with the PBL strategiesand outcomes that instructors at-tempted to facilitate (eg, the abilityto be an effective team member, ana-lytic skills, problem-solving skills,research skills, and oral and writtencommunication skills).11 In addition,

gnment in an Undergraduate Nutrition Capsto

Sub-compon

y?) Structure and function of a hIncidence, prevalence, and eNutritional and lifestyle risk faconditions—general and c

Nutrition—primary and seconPhysical activity—primary an

Diagnosis—health history, reClinical findings—signs and s

Nutrition—case-specific, tertPhysical activity—case-specMedications—drugs or dietar(Special attention on natural/because this is aligned wit

the CEQ is a recommended evaluationtool for PBL, because since a student-centered learning approach logicallydictates the evaluation of studentexperiences and perceptions.12 The23-item CEQ, which contains 5 scales,is a validated survey13 that is adminis-tered nationally in Australia in theyear after completion of a course ofstudy.14 The version of the CEQ thathas been used in the Australian Grad-uate Survey since 2010,15 which isfully labeled and includes the vali-dated Graduate Qualities scale,16 wasused for this study. Five questionswere added to facilitate open-endedinquiry (Table 3). The resulting 28-item, adapted CEQ was administeredonline to all students who took thecourse (n ¼ 36), using REDCap soft-ware,17 which enabled anonymous

ne Course

ents of Each Element

ealthy organ or organ systempidemiology of the conditionctors and predisposingase-specific

dary prevention ideasd secondary prevention ideas

levant lab results of the caseymptoms of the case

iary prevention ideasific, tertiary prevention ideasy supplements typically prescribedholistic management methods,h the school’s general curricular focus.)

Table 3. Course Experience Questionnaire Items, Scale, and Mean Response of Each Item (n ¼ 17)

Item Question Scale

Response,Mean ± SD(Range)

1 The instructors put a lot of time into commenting on my work. GT 4.3 � 0.6 (4–5)

2 The instructors normally gave me helpful feedback on how I was doing. GT 4.1 � 0.7 (3–5)

3 The course helped me develop my ability to work as a team member. GS 4.1 � 1.0 (2–5)

4 It was always easy to know the standard of work expected in this course. CG 2.7 � 1.1 (1–5)

5 The instructors of this course motivated me to do my best work. GT 4.1 � 1.0 (2–5)

6 The course provided me with a broad overview of my field of knowledge. GQ 4.0 � 1.1 (1–5)

7 The course sharpened my analytic skills. GS 4.1 � 1.0 (2–5)

8 The instructors were extremely good at explaining things in this course. GT 3.9 � 0.7 (3–5)

9 The instructors worked hard to make the course content interesting. GT 4.3 � 0.7 (3–5)

10 The course developed my confidence to investigate new ideas. GQ 4.0 � 1.0 (1–5)

11 The course developed my problem-solving skills. GS 4.0 � 1.0 (1–5)

12 The instructors made a real effort to understand difficulties I might behaving with my coursework.

GT 3.9 � 0.8 (2–5)

13 I usually had a clear idea of where I was going and what was expectedof me in this course.

CG 3.4 � 0.8 (1–5)

14 The course stimulated my enthusiasm for further learning. GQ 4.1 � 1.1 (1–5)

15 The course improved my skills in written communication. GS 3.7 � 1.0 (1–5)

16 I learned to apply principles from this course to new situations. GQ 3.7 � 0.8 (2–5)

17 I consider what I learned in this course valuable for my future. GQ 4.4 � 0.7 (3–5)

18 As a result of this course, I feel confident about tackling unfamiliar problems. GS 3.6 � 1.2 (1–5)

19 This course helped me to develop the ability to plan my own work. GS 3.5 � 1.0 (1–5)

20 The instructors made it clear right from the start what they expected from students. CG 3.1 � 1.1 (2–5)

21 The course helped me develop my oral presentation skills. GS 3.9 � 0.9 (3–5)

22 The overall course experience encouraged me to value perspectivesother than my own.

GQ 4.0 � 0.6 (3–5)

23 Overall, I was satisfied with the quality of this course. OS 4.2 � 1.1 (1–5)

24 What were the best aspects of this course? AD n/a

25 What aspects of this course were most in need of improvement? AD n/a

26 What skills from this course have helped you get where you are today in your career? AD n/a

27 This course was taught using the problem-based learning approach rather than atraditional lecture format. What did you like about the problem-based learning format,compared with other courses you took that were traditional lecture format?

AD n/a

28 What did you dislike about the problem-based learning format, compared withother courses you took that were traditional lecture format?

AD n/a

AD indicates added questions that require open-ended responses; CG, clear goals and standards scale; GQ, graduate qualitiesscale; GS, generic skills scale; GT, good teaching scale; n/a, not available; OS, overall satisfaction item.Note: Responses were 5-point options, from 1 (‘‘strongly disagree’’) to 5 (‘‘strongly agree’’).

Journal of Nutrition Education and Behavior � Volume -, Number -, 2014 Harris and Kloubec 3

data collection. Three recruitment e-mails with a Web link to the surveywere sent to prospective participants.For each item, participants rated theirlevel of agreement on a 5-point Likertscale (ie, from ‘‘strongly disagree’’to ‘‘strongly agree’’). Descriptive statis-tics were generated for each itemof the CEQ and each of its scales. In-

ternal reliability of the scales wasmeasured by the Cronbach alpha co-efficient.

OUTCOMES

Of the 36 students who took thecourse, 22 responded and 17completed the online survey, repre-

senting a 47% response rate. Resultsare displayed in Tables 3 and 4. Inter-nal consistency of the CEQ scaleswas considered to be adequate.18 Themean responses for the first 23 items(closed-ended) were $ 3.4, indicatinggenerally positive perceptions relatedto course experience. Mean scoresof $ 4.0 on scales related to ‘‘good

Table 4. Characteristics of Course Experience Questionnaire Scales Used in Study

Scale Items, n Mean SD Cronbach a

Good teaching 6 4.0 0.8 .84

Clear goals and standards 3 3.0 1.1 .76

Generic skills 7 3.8 1.0 .92

Graduate qualities 6 4.0 0.9 .91

Overall satisfaction 1 4.2 1.1 n/a

n/a indicates not available.Note: The clear goals and standards and generic skills scales were modified; typi-cally, clear goals and standards includes 4 items and generic skills includes 6.

4 Harris and Kloubec Journal of Nutrition Education and Behavior � Volume -, Number -, 2014

teaching,’’ ‘‘graduate qualities,’’ and‘‘overall satisfaction’’ with the qualityof the course corroborated thisfinding. Respondents reported thatthe best aspects of the course included‘‘learning in a different way,’’ the ‘‘va-riety of topics covered,’’ the ‘‘free-dom. to investigate. problems,’’‘‘teamwork,’’ ‘‘constantly changinginformation,’’ ‘‘interesting assign-ments,’’ feeling ‘‘challenged by thecoursework,’’ and ‘‘applied learning.’’Students also liked the ‘‘detectivework,’’ the ‘‘social’’ aspect of PBL,simulation of ‘‘real-life working situa-tions,’’ the ‘‘relaxed environment,’’‘‘doing research,’’ feeling ‘‘engaged,’’and ‘‘independence.’’

Despite overall satisfaction withthe course, 2 main areas of dissatisfac-tion emerged. The first issue had todo with clarity of expectations. Themean response for item 4 (2.7 � 1.1)revealed that students thought it wasnot ‘‘always easy to know the standardof work expected.’’ This issue was re-flected in the mean score of the ‘‘cleargoals and standards’’ scale, which stu-dents rated at only 3.0 � 1.1 (‘‘neitheragree nor disagree’’), the lowest meanscore of all the scales. Responses toopen-ended questions regarding as-pects of the course ‘‘most in need ofimprovement’’ and what respondentsdisliked about the PBL format werecongruent with these findings. A sec-ond area of dissatisfaction evidentfrom responses to open-ended ques-tions involved challenges related togroup work (eg, underperformingmembers).

IMPACT

Students' responses suggested thatthe PBL approach to a capstone nutri-tion course helped them gain skills

in collaboration and teamwork, pub-lic speaking, leadership, prioritizingtasks, time management, researchand problem solving. These are globalskills that have high transferability ina wide variety of health science profes-sions. Specific to nutrition, respon-dents also reported improved abilityto see the ‘‘big picture when doingnutrition assessment’’ and improvedfamiliarity with evidence-based nutri-tion resources.

The issue of unclear expectationson assignments may have beenrelated to the nature of the problem-based case scenarios, in that some ofthem contained multiple solutionpaths or required more informationfor understanding the problem thanwas initially given. Providing moreclearly defined purposes, procedures,and expectations before the first PBLsession—related to both the courseand individual assignments—is para-mount to the success of PBL19 andmay help alleviate this potential prob-lem in the future.

The second issue related todysfunctional small group work isinherent to PBL.20 To prevent this, in-structors could spend time before theinitial PBL session training studentson how to work effectively in groups,including role assignment, communi-cation, and conflict management, inaddition to offering more expertiseon process during PBL sessions.

There were a number of limitationsto this study. First, generalizabilityof results is limited because PBL wasimplemented in only 1 undergraduatenutrition course at a small privateuniversity, reflecting a small samplesize. Second, the CEQ is a self-administered survey, which may beprone to rater biases (eg, it is possiblethat those responding had a more

favorable impression of the coursethan nonrespondents). Third, dataon the number of participants whohad entered the job market at thetime the CEQ was administered werenot available, which limited long-term impact evaluation of the PBLapproach. Finally, this study did notevaluate the importance of other fac-tors such as motivation or additionalbaseline personal or academic charac-teristics that may have influenced theresults. Strengths of the study includethe application of a broadly publishededucational method (PBL), the use ofa validated and widely used survey(CEQ), and a response rate of nearly50%.

The results of this pilot studysupport the use of a PBL approach inan undergraduate capstone nutritioncourse. By promoting cooperation,leadership, teamwork, self-confidence,independence, critical thinking, re-source identification, informationretention, improved public speakingskills, and active lifelong learning,6

PBL has the potential to help preparenutrition students to enter a competi-tive job market.

SUMMARY

Problem-solving skills remain thecore of actual professional experiencethat many nutrition students willgo on to perform; yet, these skillsmay not be practiced in a traditionallytaught curriculum. Nutrition studentsneed to have experience using mul-tidisciplinary and lateral-thinkingapproaches to problems; and to prac-tice prioritizing, asking questions,and finding evidence-based solutionsto nutrition-related issues. Problem-based learning provides a frameworkfor instructors to facilitate these skills,and preliminary evidence suggeststhat undergraduate nutrition studentsalso find success and satisfaction fromthis approach.

NOTE

This project was evaluated by the uni-versity's institutional review boardadministrator, who determined thatit was exempt from full institutionalreview board review as per Code ofFederal Regulations, Title 45, Part46.101(b): Survey/Interview/Observa-tional Research: ‘‘Research involving

Journal of Nutrition Education and Behavior � Volume -, Number -, 2014 Harris and Kloubec 5

the use of educational tests (cognitive,diagnostic. aptitude, achievement),survey procedures, interview proce-dures or observation of publicbehavior unless: (i) information ob-tained is recorded in such a mannerthat human subjects can be identified,directly or through identifiers linkedto the subjects; and (ii) any disclosureof the human subjects' responsesoutside the research could reasonablyplace the subjects at risk of criminalor civil liability or be damaging tothe subjects' financial standing,employability, or reputation.’’ The au-thors thank their Department Chair,Debra Boutin, for supporting themin trialing this innovative instruc-tional method as a capstone experi-ence for undergraduate nutritionstudents.

REFERENCES

1. Barrows H, Tamblyn R. Problem-BasedLearning: An Approach to Medical Educa-tion. New York, NY: Springer; 1980.

2. McCarlie VW, Orr DL. Health scienceeducation: reviewing a framework forproblem-based learning. J Dental Educ.2010;74:480-488.

3. Maudsley G. Do we all mean the samething by ‘‘problem-based learning’’? Areview of the concepts and a formula-tion of ground rules. Acad Med. 1999;74:178-185.

4. Pople HE. Heuristic methods forimposing structure on ill-structuredproblems: the structuring of medical di-agnostics In: Szolovits P, ed. ArtificialIntelligence in Medicine. Boulder, CO:

Westview Press; 1982. Chapter 5.http://groups.csail.mit.edu/medg/ftp/psz/AIM82/ch0.html. Accessed April 28,2013.

5. Lohse B, Nitzke S, Ney DM. Intro-ducing a problem-based unit into alifespan nutrition class using a ran-domized design produces equivocaloutcomes. J Am Diet Assoc. 2003;103:1020-1025.

6. Terry PH, Seibels DR. Incorporatingproblem-based learning into an under-graduate community nutrition class. JNutr Educ Behav. 2006;38:121-122.

7. Kinkade S. A snapshot of the status ofproblem-based learning inU.S. medicalschools, 2003-2004. Acad Med. 2005;80:300-301.

8. Touger-Decker R, BenedictBarracato JM, O-Sullivan-Maillet J.Nutrition education in health profes-sions programs: a survey of dental,physician assistant, nurse practitioner,and nurse midwifery programs. J AmDiet Assoc. 2001;101:63-69.

9. Freeman TK, Shanahan M, Banz WJ.Incorporating nutrition into the curric-ulum of a Midwestern medical school. JNutr Educ Behav. 2005;37:273-274.

10. Elizondo-Montemayor LL. Formativeand summative assessment of theproblem-based learning tutorial sessionusing a criterion-referenced system. JInt Assoc Med Sci Educ. 2004;14:8-14.

11. Ramsden P. A performance indicatorof teaching quality in higher educa-tion: the course experience question-naire. Studies Higher Educ. 1991;19:129-150.

12. Gibbon C. Problem-Based LearningEvaluation Toolkit [HSAP RepositoryWeb site]. http://repos.hsap.kcl.ac.uk/

content/m10074/1.5/. Accessed April30, 2013.

13. Wilson KL, Lizzio A, Ramsden P. Thedevelopment, validation and applica-tion of the course experience question-naire. Studies Higher Educ. 1997;22:33-52.

14. Tucker B, Jones S, Straker L. Onlinestudent evaluation improves courseexperience questionnaire results in aphysiotherapy program. Higher Ed ResDev. 2008;27:281-296.

15. Course Experience Questionnaire. Of-fice of Strategy and Planning, CurtinUniversity. http://planning.curtin.edu.au/mir/ceq.cfm. Accessed April 20,2013.

16. Griffin P, Coates H, McInnis C,James R. The development of anextended course experience question-naire. Quality Higher Educ. 2003;9:259-266.

17. Harris PA,TaylorR,ThielkeR,Payne J,Gonzalez N, Conde JG. Research elec-tronic data capture (REDCap)—Ametadata-driven methodology andworkflow process for providing transla-tional research informatics support. J Bio-med Inform. 2009;42:377-381.

18. Tavakol M, Dennick R. Making senseof Cronbach’s alpha. Int J Med Educ.2011;2:53-55.

19. Dion L. About Teaching #50: But Iteach a large class. Newsletter of the Cen-ter for Teaching Effectiveness, Spring,1996. http://www.udel.edu/pbl/cte/spr96-bisc2.html. Accessed April 24,2013.

20. Tarnvik A. Revival of the case method:a way to retain student-centeredlearning in a post-PBL era. MedicalTeacher. 2007;29:e32-e36.