What Novice Programmers Don’t Know

Gary Lewandowski

Alicia Gutschow

Robert McCartney

Kate Sanders

Dermot Shinners-Kennedy

1 October 2005 ICER 2005 2

Which programming concepts have little or no meaning for students?

• How frequently do students indicate a concept has less meaning?

• Does “don’t know” differ from “not applicable”?

• Does classification vary based upon course performance?

• Is it the concept or the vocabulary?

1 October 2005 ICER 2005 3

Outline

• Bootstrapping

• Card Sorts and Ragbags

• Methodology

• Results

• Conclusions

1 October 2005 ICER 2005 4

Bootstrapping• NSF sponsored workshop: Sally, Marian

and Josh

• Multi-national, multi-institutional study

• 22 researchers from four continents, six countries

• Data from 247 students

• How do first-competency students organize programming concepts?

1 October 2005 ICER 2005 5

Elements in data flow

Data itselfWhat modifies the data flow Not applicableDon’t know

Things contained in a program

Things that are Things that are not

The way things work in programs

FunctionsThings about variables

Things you can do

On its own

Card Sorting

1 October 2005 ICER 2005 6

Ragbags

• Don’t Know (DK): never knows it

• Not Sure (NS): doesn’t know in this sort

• Not Applicable (NA): not applicable in this sort– Boolean sorts may or may not have an NA

1 October 2005 ICER 2005 7

categories

On its own MNS

leftovers

Things that are not

Concepts associated with programming

NA NA

NA

M

M

M

Can’t fit them inNANS

Agreed on 4280 of 4410 categories

Sometimes you need alawyer to settle yourdisputes…

Methodology: Tagging the data

NS

NA

NS

1 October 2005 ICER 2005 8

Students rated in each course1 = failing…5 = Picasso

Average Performance Rating

Low[1,3)

50 students

Mid[3,4)

106 students

High[4,5]

63 students

Methodology: Performance

28 students not rated

1 October 2005 ICER 2005 9

Methodology: Abstract Concepts

S SS SS S S S S S S S

11 Scholars categorized concepts as

Abstract or Concrete

Dependency

Decomposition

Abstraction

Encapsulation

State

Tree7 of 11 described as abstract

1 October 2005 ICER 2005 10

Methodology: MeaningfulnessHow meaningful is a concept to a student?

Breakdown of Classes

Largest componentsmethods/functions

Not applicable

Levels of how specific each one is

Smallest items

loops Didn’t really know

boolean

If-then-else scope

How things can be operated with by using other thingsThings that get acted on

Things that act on other things

Things I don’t know

boolean If-then-else scope

If-then-elseboolean

scope

If-then-else is alwaysMeaningful (A)

Boolean isusually meaningful (U)

Type is limited meaningful (L)

Scope is never meaningful (N)

type

type

type

1 October 2005 ICER 2005 11

Data: Frequency of Use#students 247 total 50 low 106 mid 63 high

At least 1 ragbag

96% 94% 92% 98%

NA

Bool/not

75%

(20%/169%)

68%

(24%/58%)

69%

(15%/63%)

90%

(25%/86%)

DK 32% 14% 37% 40%

NS 48% 64% 44% 33%

1 October 2005 ICER 2005 12

Data: Which Concepts?

0% 20% 40% 60% 80% 100%

Variable

Array

Loop

Parameter

Boolean

If-then-else

Object

Constant

Type

Function

List

Method

Expression

Recursion

Procedure

Iteration

Tree*

Abstraction*

Event

Scope

Choice

State*

Encapsulation*

Dependency*

Thread

Decomposition*

Concepts ordered by frequency in Don't Know ragbag

Don't Know

Not Sure

1 October 2005 ICER 2005 13

Data: Not Applicables

0% 20% 40% 60% 80% 100%

Variable

Array

Loop

Parameter

Boolean

If-then-else

Object

Constant

Type

Function

List

Method

Expression

Recursion

Procedure

Iteration

Tree*

Abstraction*

Event

Scope

Choice

State*

Encapsulation*

Dependency*

Thread

Decomposition*

Concepts ordered by frequency in Don't Know ragbag

Boolean NA

Non-boolean NA

1 October 2005 ICER 2005 14

Data: Meaningful or Familiar?Variable is always or usually known to 238 of 247 students

State is always or usually known to 165 students.

Recursion is always or usually known to 223 of 247 students

Iteration:Procedure:

Expression:

217212

204

Object is always or usually known to 234 students

Deomposition, the least well known concept, is always or usually known by 126 of the 247 students.

1 October 2005 ICER 2005 15

Life, the universe, and ragbags• Ragbags used frequently, dominated by abstract

concepts.• Ragbag categories are used for different purposes.• Low performers less likely to place concepts into

don’t know. (Murphy/Tenenberg, do computer science students know what they know?) How do we help them see their confusion?

• Dichotomous pairs: a problem and opportunity?• Yet again, programming language and paradigm

are not the issue.