using pla to liberate learning (pla: participatory learning approach)

1Bieber et al., NJIT ©2003

Using PLA to Liberate Learning(PLA: participatory learning approach)

Michael Bieber, Jia Shen, Dezhi Wu, Vikas Achhpiliya

Information Systems Department

College of Computing Sciences

New Jersey Institute of Technologyhttp://web.njit.edu/~bieber

November 2003


Outline

• Motivation

• PLA: Participatory Learning Approach

• A bit of theory

• Experimental results

• Interesting issues


Motivation

• To increase learning of course content

• Learning through active engagement– involve students as active participants– with the full problem life-cycle– through peer evaluation

• Minimize overhead for instructors


Outline

• Motivation


• A bit of theory




PLA Process• Each student creates 2 exam problems• Instructor edits the problems if necessary• Each student solves 2 problems• Students evaluate (grade) the solutions to the problems

they authored, writing detailed justifications• Ph.D. students evaluate each problem a second time• Instructor gives a final grade• optional: Students can dispute their solution’s grade, by

evaluating it themselves and writing detailed justifications

• Instructor resolves the dispute

All entriesposted on-line


Exam Process ControlAssign ID

Edit questionsAssign who answers questions

Assign level-2 graders

Course Design

Determine Final Grades

Set up on-line environment

Dispute final grade

Level-1 and Level-2 gradersgrade solutions

Make up problems

Read- other problems- other solutions

- grade justifications- disputes

Solveproblems

Instructor Control Process Student Learning Process

Resolve Disputes

Process Flow:Learning from doing the PLA activities

additional learning from reading everything peers write


Exam Process ControlAssign ID

Edit problemsAssign who solves problems

Assign level-2 graders

Course Design

Determine Final Grades

Set up on-line environment

Dispute final grade

Level-1 and Level-2 gradersgrade solutions

Make up problems

ConfirmationID, understand process

Read- other problems- other solutions

- grade justifications- disputes

Solveproblems

Instructor Control Process Student Learning Process

Resolve Disputes


Evaluation (grading)

• Evaluation includes:

– Written critique or “justification” (positive or negative)

– Optional: separate sub-criteria to critique

• Solution result is correct and complete (40%)

• Solution was well explained (30%)

• Solution demonstrated class materials well (10%)

• Solution cited appropriate references (20%)

– Grade (optional; recommended to save instructor time)

• Evaluation/grade may be disputed (optional)

– Student must re-evaluate own solution when disputing


Instructor should provide…

• Detailed instructions and timetable

• Solution: what is expected

• Critiquing and grading guidelines


Outline

• Motivation


• A bit of theory




Constructivism(Learning Theory)

• The central idea is that human learning is constructed, that learners build new knowledge upon the foundation of previous learning{learning throughout the exam process}

• Two classic categorizations– Cognitive Constructivism (Piaget’s theory)– Social Constructivism (Vygotsky’s theory)


Cognitive Constructivism (Piaget 1924)

• Knowledge is constructed and made meaningful through individual’s interactions and analyses of the environment.

--> knowledge is constructed in the mind of individual

• Knowledge construction is totally student-centered.


Learning

• Learning is a constructivist, often social activity occurring through knowledge building (Vygotsky, 1978)

• Knowledge building activities include contributing to, authoring within, discussing, sharing, exploring, deploying a collective knowledge base (O’Neill & Gomez 1994; Perkins 1993).


Learning

• People learn as they navigate to solve problems (Koschmann et al, 1996) and design representations of their understanding (Suthers 1999)

• Learning requires cognitive flexibility (Spiro et al.

1991), and results from interaction with people having different experiences and perspectives (Goldman-Segall et al. 1998)


Expert-like Deep Learning

• Categorizing knowledge and constructing relationships between concepts are likely to promote expert-like thinking about a domain (Bransford 2000).

• To design appropriate problems for their peers, students must organize and synthesize their ideas and learn to recognize the important concepts in the domain.

• This results in deep learning (Entwistle 2000): – seeing relationships and patterns among pieces of information,

– recognizing the logic behind the organization of material

– achieving a sense of understanding


Where is Knowledge Constructed in PLA?

• In all PLA stages:constructing problems, solutions, grade justifications, dispute justifications

• When reading everything their peers write– Students also are motivated to learn more when

peers will read their work (McConnell, 1999).


Assessment & Learning

• Main goals of tests:– To measure student achievement– To motivate and direct student learning

• The process of taking a test and discussing its grading should be a richly rewarding learning experience (Ebel and Frisbie 1986)

• Assessment should be a fundamental part of the learning process (Shepard 2000)


Outline

• Motivation


• A bit of theory




Course Information

NJIT CIS677: Information System Principles• Graduate level core course (Masters/Ph.D.)• Aim: study how IS/IT can be used effectively• Both on-campus and distance-learning sections• software: Virtual Classroom/WebBoard• Traditional Exam:

– Three-hour, in class, 3-4 essay questions, 6 pages of notes

• Used PLA 5 times between Fall 1999 and Summer 2002• We compared control groups without PLA and treatment groups

with PLA

• Also, we used with shorter essay questions in CIS 365, undergraduate course on file structures in Fall 2002, with similar survey results.


Enjoyability

Questions SA A N D SD Mean S.D. #

I enjoyed the flexibility in organizing my resources

26.2% 48.9% 16.7% 3.6% 4.6% 3.88 1.00 221

I was motivated to do my best work 23.5% 42.9% 28.2% 3.4% 2.1% 3.82 .92 238

I enjoyed the examination process

17.2% 42.3% 22.6% 10.5% 7.4% 3.51 1.13 239

SA - strongly agree (5 points); A - agree (4); N - neutral (3); D - disagree (2); SD - strongly disagree (1); the mean is out of 5 points; S.D. - standard deviation

Cronbach’s Alpha=0.68


Perceived Learning

Questions SA A N D SD Mean S.D. #

I learned from making up questions

17.9% 42.5% 21.3% 13.8% 4.5% 3.55 1.08 240

I learned from grading other students answers

17.7% 48.1% 19.4% 9.3% 5.5% 3.63 1.06 237

I learned from reading other people’s answers

15.8% 45.0% 22.1% 11.3% 5.8% 3.54 1.07 240

I demonstrated what I learned in class 13.6% 50.2% 22.6% 10.9% 2.7% 3.61 .95 221

My ability to integrate facts and develop generalizations improved

21.8% 49.2% 25.6% 2.1% 1.3% 3.88 .83 238

I learned to value other points of view

17.6% 51.9% 27.6% 1.3% 1.6% 3.82 .81 239

I mastered the course materials

7.4% 51.6% 31.4% 6.9% 2.7% 3.54 .84 188

Cronbach’s Alpha=0.88


Recommendation: Do Again!

Question SA A N D SD Mean S.D. #

Would you recommend in the future that this exam process used?

20.7% 40.1% 24.5% 8.9% 5.8% 3.60 1.10 237

Similar results for CIS365: undergraduate file structures course using short essay questions (Fall 2002)


Outline

• Motivation


• A bit of theory




What students liked best

• Active involvement in the exam process

• Flexibility

• Reduction in tension


Trade-offs

• Trade-offs for students (traditional vs. PLA)– Timing: Concentrated vs. drawn-out (2.5 weeks)– Access to information: limited vs. the Internet– Experimental integrity: we couldn’t justify the

process to the students fully

• Trade-offs for professors– Fewer solutions to evaluate, but each is different– Timing: Concentrated vs. drawn-out process– Much more administration


Timing

• PLA for exams took 2.5 weeks

• For frequent activities PLA processes could overlap – e.g., quizzes, homeworks– Students could be creating problems for one quiz,

while solving problems for the prior quiz, while evaluating solutions from the quiz before that

• Benefits to overlapping PLA activities: – working with materials from several classes at the same time– could reinforce class materials– could result in synthesis (combined understanding)


Scope

• Which activities?– so far: exams– what about: quizzes, homeworks, larger projects, in-class

projects

• Which problem types?– so far: short and long essay questions– what about: multiple choice, short answer, computer

programs, semester projects– Sub-problems:

• computer program design & implementation• semester project outline & execution


Scope, cont.

• Course Level– Graduate, undergraduate, secondary school (high

school, junior high)

• Disciplines– IS/IT, business, science, engineering, humanities,

medical, all of secondary school


Scope, cont.

• Degree of Evaluation (assigning grades)– Currently: solutions– What about:

• quality of problems• quality of evaluations/grades

– All could be disputed

• Degree of Participation– students could evaluate each– students could arbitrate disputes


Full Collaboration

• Groups for:– Problems, solutions, evaluation, dispute arbitration

• Requires group process support– Group roles: leader, scheduler, etc.– Process: work on each activity together or separately,

internal review– Grading of individual group members– Process Tools: brainstorming, voting, etc.


What can go wrong

• Students are late; students drop the course

• Entries posted in wrong place

• Inadequate critiques– “Good”– “I agree with the other evaluator”

• and of course, technical difficulties…


PLA Environment Software• Guide the process• Form groups• Assign problem solvers, evaluators, dispute arbitrators• On-line templates to ensure full entries• Guide people to post entries in correct place• Incorporate group process tools• Handle problems as much as possible

– Remind people who are late– Reallocate who does what

• Based on a workflow management tool…


Anonymity/Privacy Issues

• Should student entries be anonymous?

• Will students reveal their IDs?

• Is it fair to post critiques if not anonymous?

• Is it fair to post grades if not anonymous?

• Will anonymity work in small classes?


Issue: Perceived Fairness

• Should students evaluate/grade peers?– But they must evaluate others in the workplace…

• It’s the instructor’s job to evaluate and grade– PLA is a (constructivist) learning technique

• Students have no training in evaluation– Evaluation is a skill that must be learnt (and taught)

• Many evaluators = inconsistent quality– safeguards in the PLA process


Grading Issues

• Disputing high grades:– Award bonus points if students dispute (and justify

with a critique) grades that are too high

• Encouraging honest grading:– For successful disputes, deduct points from

evaluators


Grade Inflation

• Detailed grading guidelines for sub-criteria:• great: 20 points

• very good: 18 points

• good: 14 points

• OK: 10 points

• poor: 6 points

• Student does “good” on 5 problems, grade = 70• U.S. students will protest vigorously• Evaluators will hesitate to assign “good”• Result: pressure for highly skewed grading rubrics


Other Cross-Cultural Issues

• In some cultures:– Students are so competitive, they would only give

failing grades to peers– Students would not hurt peers’ feelings, and would

only give good evaluations

• Some systems only have pass/fail, so numeric grades are mostly irrelevant


PLA: Contributions

• Systematic technique to increase learning– Constructivist approach, actively engaging students

in the entire problem life-cycle– Minimizes overhead for students and instructors

• Experimental evaluation• Supporting software

• PLA liberates learning from its traditional instructor-controlled structure!

Thank you! Questions, please?

using pla to liberate learning (pla: participatory learning approach)

Documents