student perspectives of tbl in a cs course: summary of ... · student perspectives of team-based...

Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick

Student Perspectives of TBL in a CS Course: Summary of Qualitative Findings

Michael S. Kirkpatrick SIGCSE Technical Symposium

March 2017


TBL implementation

Team-based Learning (TBL)• Flipped pedagogy developed by Larry Michaelsen • Course divided into 5-7 modules • Founded on constructivism and group process dynamics

Preparation (Pre-Class)

Readiness Assurance Diagnostic - Feedback Application of Course Concepts

45-75 minutes class time 1-4 hours class time

1 2 3 4 5 6Individual

Study

Individual TestTeam Test

Written AppealsInstructor Feedback Application

Oriented Activities


Pre-module preparationModule 1 Reading Guide

Required readings:• Watch the video lectures linked on Canvas on C programming basics• Practical C Programming: sections 4.8, 5.2, 9.2, 13.1, 13.2, 13.3, 17.1, 17.2

• Skim additional sections for reference as needed

Objectives and questions:• Summarize the structure of a typical C program

• What types of #include statements are most similar to Java API imports?• What types of #include statements are most similar to Java imports of classes in

the same package?• What is the purpose of a function prototype?• Every executable program must have what function defined?• If a function is defined in one source code file but used in another, how do you

expose the function so the compiler can check for the correct parameter types?• If assume that counter is a global variable of type int. How would you declare

this variable in a header file to make it visible to multiple source files?• Compile and execute a C program on the command line

• What would you type on the command line to compile hello_world.c into an object file?

• What would you type on the command line to link hello.o and goodbye.o into an executable called greetings?

• What are the 3 components of a Makefile rule?• Assume that a Makefile has a rule with the target name clean. What would you

type on the command line to execute that rule?• Demonstrate basic C programming techniques, including stdio and string processing

• If x is a uint32_t (unsigned 32-bit integer), how would you print its value in C?• If x is a string in C, how would you print its value?• How many bytes are required to store the string ”my name” in C?• Assume you have an array int arr[5]. What would happen if you did arr[10] = 7?

• What would be the C equivalent of the following Java class definition?! class Junk {

! ! int x;! ! char y[4];! }

• Summarize the fundamental segments of the C memory model• Global variables are stored in what segment of memory?• Local variables are stored in what segment of memory?

JMU CS 350: Computer Organization • Spring 2015


Readiness Assurance Process (RAP)

CS 350 – Spring 2015 Name:

By submitting this paper, I affirm my compliance with the JMU Honor Code.

Module 1 iRAT

1. Which of the following lines of C code is most like these lines of Java?

import java.io.*;import System.out.*;

(A) #include "stdio.h" (C) #include <stdio.h>(B) #include <stdio.c> (D) #include "stdio.c"

2. Which of the following command lines will compile and link rudolph.c intoan executable called reindeer.exe?

(A) gcc -c rudolph.c -o reindeer.exe(B) gcc rudolph.c -o reindeer.exe -O0(C) gcc -g reindeer.exe rudolph.c(D) gcc -c -o reindeer.exe rudolph.c

3. Which phase of the compilation process creates a single executable file bymerging one or more object files?

(A) assembling (C) linking(B) loading (D) preprocessing

4. Which of the following statements about Makefiles is FALSE?

(A) All dependencies for a target must be satisfied before that target’srecipe will be run.(B) The recipe is the sequence of system commands that will be executedwhen make runs a target.(C) If no target is indicated, make will execute the all target by default.(D) The clean target has no inherent meaning, but is commonly used fordeleting object files.

5. Which of the following would be stored on the stack?

(A) A dynamically allocated string that is a duplicate of the string declaredas char str[] = "hi";(B) A function’s local variable node that is a pointer to a part of a datastructure(C) A global variable references that keeps track of how many times afunction gets called(D) The character ’h’ in the local string char *str = "hello world";


Application activities

C code Assembly language Machine language

x = ~(x >> 2);

y = x + 10;

printf (“%c\n”,y);

x += (char)y;

Two-pass assemblyThe following program has gone through the first pass of a two-pass assembler. Complete the symbol table that is used as input to the second pass. Below that table is the output from the first pass. Write the final bytes of the executable on the line above the first pass bytes.0000! C10000! main:! LDA x,d0003! 710000! ! ADDA y,d0006! E10000! L1:! STA x,d0009! 390000! ! DECO x,d000C! 00! ! STOP000D! 0005! x:! .WORD 5000F! 0002! y:! .WORD 2

C1 00 00 71 00 00 E1 00 00 39 00 00 00 00 05 00 02

Memory layout - C vs. Pep/8Using the memory layout of the C memory model to the right, fill in the missing addresses for the following assembly language program. Are these values realistic for a normal C program? What difficulty would arise, particularly in regard to pointers?

JMU CS 350: Computer Organization • Spring 2015

1239 461238 451237 441236 431235 421234 41

FFFFKernel

FBCFKernel

FBCE

--

--

Address Depend Symbol Value

0000 LDA x,d0003 ANDA y,d0006 STBYTEA z,d0009 CHARO z,d000C STOP000D z: .BYTE 0000E x: .WORD 0xA94F0010 y: .WORD 0xFFFD


Major assessments

Module 1 Module 2 Module 3 Module 4 Module 5 Module 6

Team Exam

Individual Exam

Team Exam

Final Exam

Project 1 Project 2 Project 3 Project 4


Prior results (ITiCSE 2015)

4050

6070

80CS 350 Grade

CS

450

Fin

al E

xam

Sco

re

D C B A

Fall 2013 C experienceFall 2013 No CFall 2012 C experienceFall 2012 No C

Figure 5: Visual representation of the linear regres-sion model of final exam score

the primary goal was to find a way to bring active learninginto our systems courses.Finally, we also need to address a commonly cited concern

about switching to active learning techniques: its impact oncoverage. That is, instructors often worry that adopting anew pedagogy will interfere with their ability to cover all re-quired material. In this instance, we do not feel that such aconcern is warranted. After doing an informal retrospectivemapping to the ACM 2013 curriculum recommendations,we found no differences between coverage of Core Tier-1and Core Tier-2 topics. Any differences between coverageof Elective topics was negligible. Consequently, we find thatadopting these techniques will not have any measurable im-pact on coverage.

5. CONCLUSION AND FUTURE WORKIn Fall 2013, we introduced two new techniques into our

OS course in an attempt to increase the amount of activelearning that happened in the classroom. These two tech-niques, the RAP and the RSQC2, require a very modestamount of effort on the part of the instructor, but initial ev-idence indicates that they can be powerful first steps towardimproving student learning. Our statistical analysis, whichcompares the Fall 2012 and Fall 2013 offerings of OS, showsthat there were statistically significant improvements in a va-riety of measurements, even when demographic differenceswere considered. Thus, for any instructor contemplating aswitch to an active learning pedagogy, we recommend thesetwo techniques as a low-cost transitional step.

6. REFERENCES[1] T. A. Angelo and K. P. Cross. Classroom Assessment

Techniques: A Handbook for Faculty. Jossey-Bass, SanFrancisco, CA, 1993.

[2] J. Campbell, D. Horton, M. Craig, and P. Gries.Evaluating an inverted cs1. In Proceedings of the 45thACM Technical Symposium on Computer Science

Education, SIGCSE ’14, pages 307–312, New York,NY, USA, 2014. ACM.

[3] R. M. Felder and R. Brent. Active learning: Anintroduction. ASQ Higher Education Brief, 2(4),August 2009.

[4] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith,N. Okoroafor, H. Jordt, and M. P. Wenderoth. Activelearning increases student performance in science,engineering, and mathematics. Proceedings of theNational Academy of Sciences, 2014.

[5] R. R. Hake. Interactive-engagement vs. traditionalmethods: A six-thousand-student survey of mechanicstest data for introductory physics courses. AmericanJournal of Physics, 66(1):64–74, 1998.

[6] S. Horwitz, S. H. Rodger, M. Biggers, D. Binkley,C. K. Frantz, D. Gundermann, S. Hambrusch,S. Huss-Lederman, E. Munson, B. Ryder, andM. Sweat. Using peer-led team learning to increaseparticipation and success of under-represented groupsin introductory computer science. In Proceedings ofthe 40th ACM Technical Symposium on ComputerScience Education, SIGCSE ’09, pages 163–167, NewYork, NY, USA, 2009. ACM.

[7] H. H. Hu and T. D. Shepherd. Using pogil to helpstudents learn to program. Trans. Comput. Educ.,13(3):13:1–13:23, Aug. 2013.

[8] P. Lasserre and C. Szostak. Effects of team-basedlearning on a cs1 course. In Proceedings of the 16thAnnual Joint Conference on Innovation andTechnology in Computer Science Education, ITiCSE’11, pages 133–137, New York, NY, USA, 2011. ACM.

[9] C. B. Lee, S. Garcia, and L. Porter. Can peerinstruction be effective in upper-division computerscience courses? Trans. Comput. Educ.,13(3):12:1–12:22, Aug. 2013.

[10] M. Mani and Q. Mazumder. Incorporatingmetacognition into learning. In Proceeding of the 44thACM Technical Symposium on Computer ScienceEducation, SIGCSE ’13, pages 53–58, New York, NY,USA, 2013. ACM.

[11] L. K. Michaelsen, A. B. Knight, and L. D. Fink.Team-Based Learning: A Transformative Use of SmallGroups in College Teaching. Stylus Publishing,Sterling, VA, 2002.

[12] B. Pfaff, A. Romano, and G. Back. The pintosinstructional operating system kernel. In Proceedingsof the 40th ACM Technical Symposium on ComputerScience Education, SIGCSE ’09, pages 453–457, NewYork, NY, USA, 2009. ACM.

[13] L. Porter, C. Bailey Lee, and B. Simon. Halving failrates using peer instruction: A study of four computerscience courses. In Proceeding of the 44th ACMTechnical Symposium on Computer Science Education,SIGCSE ’13, pages 177–182, New York, 2013. ACM.

[14] M. Prince. Does active learning work? a review of theresearch. Journal of Engineering Education, 93(3),July 2004.

[15] M. K. Smith, W. B. Wood, W. K. Adams,C. Wieman, J. K. Knight, N. Guild, and T. T. Su.Why peer discussion improves student performance onin-class concept questions. Science, 323, January 2009.

0%

20%

40%

60%

None Limited Small Large

Offering

Fall 2012

Fall 2013

Figure 3: Self-reported C programming experience

performed significantly better in CS 450 than those in thecontrol group. Figure 4 shows the grade distributions forthe two offerings. The WDF rate improved from 13/49 inFall 2012 to 3/52 in Fall 2013; a one-sided χ2 test yieldedp = 0.004893, indicating significance. Similarly, if we lookat the students who did well in CS 450 (those earning Asand Bs, and only considering the subset of students whoresponded), we see that the ratio of As and Bs improvedfrom 17/44 in Fall 2012 to 35/43 in Fall 2013, which yieldsp = 0.0000596 in a one-sided χ2 test. Thus, the interventiongroup showed a significant improvement in course grades.In addition to overall course grades, we also observed a

significant improvement on programming projects. For in-stance, both offerings were required to complete (workingin groups) portions of the Pintos OS projects [12]. In thesecourses, the userprog project was completed at the end ofthe semester. The average number of test cases (only asubset were required) that the teams passed increased from28/50 to 40/50. A t test yields p = 0.02391, thus indicat-ing significance. As these were team projects, we attemptedto perform a linear regression that modeled the number oftest cases passed as a function of the team’s average C pro-gramming experience and the intervention. The only factorfound to be significant was the intervention (p = 0.0678),but the adjusted R2 value was 0.1535 (overall p = 0.09181for the model), indicating the model produced by the regres-sion is rather weak. However, what is interesting about thisregression is that teams’ average C programming experienceis not a significant predictor of test case success, regardlessof whether or not the intervention was present. This findingis counterintuitive, as the Pintos projects are all written inC, but it is welcome: It suggests that teams with less C ex-perience in both groups were able to catch up with those withmore experience by the end of the semester.

3.3 Impact on final exam performanceAs noted above, our intervention group (Fall 2013) demon-

strated significant improvements in the WDF rate, overall

0%

10%

20%

30%

40%

50%

WDF C B A

Offering

Fall 2012

Fall 2013

Figure 4: Fall 2012 vs. Fall 2013 CS 450 grade dis-tribution

course grade, and programming projects. We also observedthat students performed better on the exams, as well. Toanalyze exam performance in more detail, we performed amultiple linear regression analysis aimed at describing finalexam scores as a function of demographics (C programmingexperience, community college equivalency of the prerequi-site, and prerequisite grade) and our intervention. Linearrelationships between final exam scores and each of the de-mographic and intervention variables were reasonable, andresidual diagnostics for the final model (described below)revealed no issues with model form or assumptions. Wefound the community college equivalency factor to be in-significant with the other factors present, and removed itbased on ANOVA results (p = 0.4958).

Recall that the C programming factor was self-reportedLikert-scale data. While we initially considered all four Lik-ert responses as factors in the regression, we found the in-crease to the third level, “Written several small programs,”relative to having “No experience,” to be the only significantfactor (p = 0.0551). Consequently, we found it appropri-ate to group the responses into two categories: those withlimited or no experience, versus those with at least someexperience with C programming.

Table 1 shows our model, which is significant with an over-all p = 1.153 ∗ 10−12 and R2 = 0.5114, indicating that thismodel captures the behavior well. According to this model,the y-intercept for students in the intervention group was 24points higher on the final exam when all other factors areheld constant. Furthermore, there was a significant inter-action between the prerequisite grade and the intervention,which changes the slope of the line for that class. The resultis that the weakest students (those who scored a D in theprerequisite) in the intervention group benefited the most.At the same time, though, the strongest students in the in-tervention group also benefited but to a lesser extent.

Figure 5 illustrates the model in graphical form. In thisfigure, the dotted lines indicate the model for the studentswho had limited or no C programming experience, whereas

Kirkpatrick, M.S. and Prins, S. “Using the Readiness Assurance Process and Metacognition in an Operating Systems Course.” 20th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), July 2015, http://dx.doi.org/10.1145/2729094.2742594.


Do students perceive value in using TBL in CS classes?

Does this perception change?


Setting

• Master’s university, very high undergraduate • More selective, full-time, lower transfer-in • CS department is primarily undergraduate

• 300-level Computer Organization • 30 sophomores, 47 juniors, 34 seniors (111 total) • 28+27 using TBL (one instructor) • 27+29 using traditional lecture (different instructor)


Mixed-methods design

Quantitative • Value of Teams Survey

(VTS) • Working with Peers • Value of Group Work

Qualitative • Semi-structured focus

groups in 12th week • 8+6 participants (each team

represented) • Anonymized transcripts • Open & axial coding • No inter-rater reliability


Transcript coding results

10% 15%

15%

20%20%

20%

Value of in-class collaboration Team formationFactors influencing success RAP difficultiesTeam tests Other


Value of in-class collaboration

Greater learning, but difficult

Correct answers emerged from

discussion

Mixed feelings on making projects

collaborative

“[I]t’s easier to ask your peers that are in your immediate group if you don’t understand something.”

"Even if everyone didn’t know the answer exactly, like when we talk about it we somehow come up with an answer.”

“I mean as programmers we’re gonna be working on teams and developing software programs.”

“I would say it was harder to do things out of class, but in class was a lot more valuable time.”

“I made a joke along the lines of as a group we work really well together, but as an individual entity we all suck.”

“I think there are others that might use that as a way to cruise.”


Team formation is problematic

Split opinions on balanced vs. unbalanced teams

Survey-based team assignment can be manipulated, but no better

choice

“I think maybe if that was improved then the teams could be more balanced... I feel like there could be better ways to balance the groups and that would really enhance the experience.”

“[I]t all depends how honest you are on the survey too. Cuz if you say that, ‘Okay, I’m not proficient in this,’ and then you actually are, and then you’re put in the group with a bunch of a people who are really good at it.”

If “he puts you in the group [with] people who don’t know, I feel like then you guys will work harder… You guys will teach each other, like, ‘Hey, guys, we don’t have anyone that really knows the stuff in the group, so we’re gonna work hard to get it.’”

“That’s not to say that I’d like to choose the groups on my own because I feel like that could be unfair to the people who don’t know anybody.”


Factors that influence success

Preferred effort and formative evaluation

over ability

Team dynamics create risks

Team structure mostly encourages

accountability

“I’d rather have you try and get the answer wrong than just to sit there and not do anything at all.”

“[I]f you have [mixed ability groups], they’re supposed to try and teach them, but they’re introverted or don’t care, they’re not gonna teach anything to us.”

“If you feel that they’re gonna rely on you, then you might be more inclined to make sure that you read and make sure you know.”

“[A:] “I feel like [peer evaluations] should be maybe twice during the semester.” [B:] “I know. Cuz at the end of the semester what good is it gonna do [chuckles]?”

“They would go into that stuff so fast, and I was literally so lost. I had so much anxiety the first couple of weeks because I didn’t know what was going on. They didn’t wanna take the time to explain it to me.”

“[I]t’s kind of a self-correcting system. I mean eventually they might kind of reap the rewards for a while, but you’re gonna face what’s happening in the class.”


RAP frustrations

Reading assignments are insufficient

preparation

RAP reveals misunderstandings

Sometimes RAP is insufficient feedback

“He gives you a reading, but it’s not...this is not like a history class where you just read it and get all the materials there. It’s kinda hard.”

“I feel a certain point of this is to show you how wrong you are in thinking whatever you thought you knew from studying…”

“They’re not gonna–not everyone’s gonna explain every single answer that they put, and then they’re like, ‘Okay, we got that right,’ but you still have no idea why that was the right answer.”

“We basically had to teach ourselves and then we got graded on that. Then we wouldn’t learn...until we did the activity, which was after the quiz, so not very effective.”

“[T]he iRAT would show me what I think I know versus what I don’t know, which is usually 70 percent or whatever. Then the tRAT sort of allows us to collectively come together and reason out why.”


Team exams

New process skills and new test-writing approach required

“[I]f we start a problem and then I say what I think the answer is and then my group, they disagree, then we have to talk about it.”

“If we split up the team exam into different sections so certain people do certain parts, that defeats the point of it.”

“[W]e had that idea in the back of our mind, like we practiced that the day before and we had a strategy. That was something we vocally talked about the day before our second exam cuz we didn’t do quite as well on our first exam. Talking about pacing ourselves.”


Recommendations• Be transparent about the process of assigning students to teams. Use

objective metrics to the extent possible. • Provide explicit instruction for TBL process skills, including how to prepare

for team exams and the iRAT. • Supplement reading assignments with scaffolding materials to facilitate

achieving desired learning objectives for the RAP. • Consider replacing or augmenting the iRAT with a pre-class online quiz. • Ask for additional points of confusion after addressing commonly missed

tRAT questions. • Make team exams shorter than individual exams to allow for discussion and

to be mindful of the impact of absences. • Provide time estimates for team exam problems.


Thank you

[email protected]

student perspectives of tbl in a cs course: summary of ... · student perspectives of team-based...

Documents