pilot implementation of active learning at an engineering department

Pilot implementation of Active Learning at an Engineering

Department for the development of generic skills of students

The pilot course Decision Support Systems (DSS)

Technical University of Crete, Department of Production Engineering & Management, Evangelia (Lia) Krassadaki, PhD candidate

Indicatively presentation of practices applied in universities for the improvement of generic skills of students

• A’ trend: as part of the curricular – Through courses (without activities for skills assessment) of the curriculum. A certification-

transcript is given at the end of studies to students (the case of Luton University) – Autonomous courses for generic skills improvement. The grade at the course certifies the level

of the skills possessed by students. For example, the case of General Education program at college level-USA.

– Writing/speaking intensive courses. The case of USA universities. – E-Portfolios (University of Denver, Virginia Tech, Queensland University of Technology, etc)

• B’ trend: as extracurricular activities – Activities of Career Offices sometimes by giving a certificate and sometimes without it. For

example, the transcript ‘Nottingham Advantage Award’ from Nottingham University. – Surveys for skills improvement for graduates or students. Characteristic case the yearly

graduate survey in Australia ‘Graduate Course Experience’-GSA (118 thousand graduates participated the year 2009).

– Specially designed tests for the evaluation of generic skills, like the national level test in Australia ‘Graduate Skills Assessment’-GSA for assessing critical thinking, problem solving, interpersonal understanding and written communication.


The educational context of the pilot implementation at Technical University of Crete

Goals:

Performance improvement at the content of the course DSS (content)

Generic skills improvement: Communication (written-oral) & Team-working

Active participation of students

Satisfaction of students

Technical University of Crete, Department

of Production Engineering & Management, Evangelia (Lia) Krassadaki, PhD candidate

Criteria for the verification of the successful redesign of the course


The Constructive Alignment Model which is used for the redesign of the course

Source: Biggs, 2003; Felder & Brent, 2003.


Redesigning a course: (1) Learning Outcomes (students) - Learning Objectives (tutor)

A model for course provision based on 3 areas: 1. Disciplinary content – content of the course 2. Disciplinary skills - practical skills associated with the content of the course 3.

Generic skills – informal/transferable skills or abilities

ΑΒΕΤ criteria

by italics


Redesigning a course: (2) Teaching (strengthening participation & interaction)

Short lecture technique (vs lectures)

Brainstorming technique

Discussion technique

Question-answer technique

Giving a speech – presentation of a subject (students)

Technique of 10-minute exercises in groups of two

Technique of short written assignments (every week)

Digital self-evaluation material for the knowledge evaluation of the content of the course (outside class)

Research questionnaires (inside/outside class)


Redesigning a course: (3) Assessment (summative – formative)

Summative assessment: • Short written-oral assignments (every week) • Oral performance on disciplinary content & disciplinary skills Formative assessment: • 10-minute group exercises (group of two) • Digital self-evaluation material for the knowledge evaluation

(outside class) • Peer-assessment technique of written assignments & oral

presentations (inside/outside class) • Individual feedback for every written (oral) assignment (tutor) • Self-evaluation of students’ generic/informal skills (before-after

scheme)


Feedback


Treatment group (students of the pilot course)

2009: 1 group, 27 students

2010: 2 groups, 46 students

2011: 2 groups, 44 students

Control group (students of the traditional offered course)

2009: 57 students 2010: 30 students 2011: 40 students


Pilot Course characteristics

• Meeting: once per week • Hours: 2 hours/week • Duration: 13 weeks • Tutors: 1 professor & 1 coordinator • Classroom: room (not an amphitheater) of total capacity 30 persons • Room with circular arrangement • Technological equipment: WiFi, video-projector, laptop(s), board • Participation: compulsory • Submission of assignments: Ε-Class (E-learning platform) with time

constraint • Groups of two-students: form the groups by lot


Plan of 2nd meeting with topic Information & Information Systems (2 teaching hours)

Educational Techniques: • Short lecture (15 minutes) • Brainstorming (20 minutes) • Questions-answers (10 minutes) • Group exercise (10 minutes) • Discussion about the exercise (10 minutes) • Short lecture (5 minutes) • Discussion (15 minutes) • Overview-short lecture (5 minutes)



6.4

6.4

6.7

5.7

5.4

5.2

5 5.2 5.4 5.6 5.8 6 6.2 6.4 6.6 6.8 7

Average grades 2009

Average grades 2010

Average grades 2011

Comparison of average performance on disciplinary content & skills for three years

Students of the Pilot course

Students of the Traditional offeredcourse

Higher average grades for the students of the pilot course

Results for t-test & Effect size

2009: • [t (66.33) = 2.12, p <0.05], 26 students pilot course, 52

students traditional course • Cohen’s d = 0.48, r = 0.23 (medium effect) 2010: • [t (73) = 2.601, p <0.05], 45 students pilot course, 30

students traditional course • Cohen’s d = 0.6, r = 0.29 (medium-large effect) 2011: • [t (79) = 4.029, p <0.00], 42 students pilot course, 39

students traditional course • Cohen’s d = 0.89, r = 0.41 (large effect)


Self-evaluation of Writing skill (before-after scheme for the students of the pilot course)

Self-evaluation criteria:

1. Organise a text (report)

2. Use of literature

3. Synthesis & presentation of information in texts

4. Online information retrieval for documentation purposes

5. Word processing

Technical University of Crete, Department


Writing skill – Statistical & Multicriteria Results of the self-evaluation survey (year 2010)

Wilcoxon Test

Weight (before)

SI index (before)

Weight (after)

SI index (after)

% improvement

Organise a text -3.924 (0.000) 32.6% 16.2% 20% 72.9% 350

Use of literature -3.848 (0.000) 20% 19% 20% 41.5% 118

Synthesis & presentation of information

-3.646 (0.000) 20% 25.2% 30.4% 31.8% 26

Information retrieval from internet

-2.358 (0.018) 13.3% 55.2% 15.6% 73.5% 33

Word processing -3.689 (0.000) 14.1% 43.1% 14% 72.7% 69

Global self-evaluation index -4.482 (0.000) 26.9% 56.6% 110.4

Strong Weakness Significant improvement despite the weaknesses

Weakness with improvement Technical University of Crete, Department


Writing skill – comparison of the indices at the before-after scheme (2010)


16.2% 19%

25.2%

55.2%

43.1%

26.9%

72.9%

41.5% 31.8%

73.5%

72.7%

56.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

organise a text use of literature synthesis &presentation of

information

informationretrieval from

internet

word processing Global self-evaluation index

SI (before) SI (after)

Self-evaluation of speaking skill (before-after scheme for the students of the pilot course)

Self-evaluation criteria:

1. Oral argumentation

2. Formulation of questions-answers in class

3. Participation in discussion in class

4. Speaking in class (speech)

5. Computer-supported presentation of a topic


Speaking skill – Statistical & Multicriteria Results of the self-evaluation survey (year 2010)

Wilcoxon Test

Weight (before)

SI index (before)

Weight (after)

SI index (after)

% improvem

ent

Oral argumentation -2.461 (0.014) 14.6% 51.6% 37% 23.3% (-54.8)

Formulation of questions-answers in class

-2.953 (0.003) 30.4% 17.3% 13.5% 51.2% 196

Participation in discussion in class -2.051 (0.040) 19.6% 33.9% 16.5% 71.2% 110

Speaking in class (speech) -3.075 (0.002) 20.6% 18.8% 16.5% 33% 75.5

Computer-supported presentation of a topic

-3.798 (0.000) 14.7% 27.2% 16.5% 63.6% 134


Strong weakness Weakness with improvement

Significant improvement despite the weaknesses


Speaking skill – comparison of the indices at the before-after scheme (2010)


23.3%

51.2%

71.2%

33%

63.6%

43.2%

51.6%

17.3%

33.9%

18.8%

27.2% 28.4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

oral argumentation formulation ofquestions-answers

participation indiscussion in class

speaking in class(speech)

computer-supportedpresentation of a

topic

Global self-evaluation index


Self-evaluation of teamworking skill (before-after scheme for the students of the pilot course)

Self-evaluation criteria: 1. Organisation & planning activities with

colleagues 2. Collaboration with colleagues for project

completion 3. Respect for different opinions 4. Consensus politics for teamwork 5. Creative criticism to team members 6. Leading a team 7. Mediating for conflicts among members


Teamworking skill – Statistical & Multicriteria Results of the self-evaluation survey (year 2010)

Wilcoxon Test

Weight (before)

SI index (before)

Weight (after)

SI index (after)

% improveme

nt Organisation & planning activities with colleagues

-3.807 (0.000) 14.3% 58.1% 14.3% 72.3% 24.4

Collaboration with colleagues for project completion

-2.457 (0.014) 15.8% 68.9% 14.3% 72.9% 5.7

Respect for different opinions -1.014 (0.311) 14.3% 72.5% 14.3% 75.9% 4.8

Consensus politics for teamwork

-3.852 (0.000) 13.1% 59.3% 14.3% 75.9% 28

Creative criticism to team members

-3.586 (0.000) 14.3% 60.6% 14.3% 71.8% 18.5

Leading a team -2.696 (0.007) 13.9% 51.1% 14.3% 65.5% 28.2

Mediating for conflicts among team members

-1.272 (0.203) 14.3% 67.2% 14.3% 70.9% 5.5


Lower room for improvement – higher self-confidence for the teamworking skill Technical University of Crete, Department


Teamworking skill – comparison of the indices at the before-after scheme (2010)


72.3% 72.9% 75.9% 75.9% 71.8%

65.5% 70.9% 76.6%

58.1%

66.8% 72.5%

59.3% 61.1%

51.1%

69.6% 64.70%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

organisation &planning of

activities withcolleagues

collaborationwith colleagues

for projectcompletion

respect fordifferentopinions

consensuspolitics forteamwork

creativecriticism to

team members

leading a team mediating forconflicts amongteam members

Global self-evaluation

index


Contribution of different courses at the development of generic skills of students (a hypothetical example – results

from the pilot course)

Writing skill Speaking skill Teamworking skill

Course % GLOBAL IMPROVEMEN

T INDEX

CONTRIBUTION

% GLOBAL IMPROVEME

NT INDEX

CONTRIBUTION

% GLOBAL IMPROVEMEN

T INDEX

CONTRIBUTION

(2010) 110.4 Substantively 52.1 Slightly 18.4 Moderately

(2011) 140.3 Substantively 277.7 Substantively

36.3 Substantively

Low threshold

14.95 112.8 8.95

High threshold

77.63 195.25 22.63


The 3-level contribution scale (slightly, moderately, substantively) : a proposal from Felder & Brent (2003)

Implementation of Peer-assessment of written/oral assignments

Written assignments • Collect 6 assignments of different quality (coordinator) • Delete of personal information-details of the authors • Distribution: copies of 6 assignments & a specially designed

evaluation sheet • Distribution: same assignments at the years 2010 & 2011 • Distribution: at the beginning of the pilot course as a homework • When peer-assessment completed a discussion followed for the

findings and results Oral assignments • Distribution of the evaluation sheet before the presentations • Completion of the evaluation sheet after the presentation-speech • When peer-assessment completed a discussion followed for the

findings


Multicriteria clustering of students as assessors for written assignments based on the assessed weights of criteria

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8

2011 (19 students)

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8

2010 (38 students)

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8

ALL (57 students)

• Criteria weights: UTASTAR • Clusters: Global K-means • Clustering quality:

Silhouette coefficient Technical University of Crete, Department


Content completeness Syntax-spelling Text organisation

Population (students-assessors) distribution at the 8 clusters according to criteria weights

2 2 3

10

1 5

29

5 54% 9% 38%

4% 93% 4%

9% 27% 65%

3% 49% 49%

100% 0% 0%

44% 46% 10%

33% 34% 33%

2% 4% 94%

The most populated group: almost equal weight for each of the 3 criteria


Students’ behavior in peer-assessment: the multicriteria approach for the 8 groups at this case study


Implementation of integrated feedback policy for students-tutors

From tutor (coordinator of the pilot course) to students • Informative personal comments on written

assignments • Informative personal comments to oral

presentations From students to tutor (coordinator of the pilot course) • Students’ satisfaction survey • Teaching techniques preferences survey


Results of the students’ satisfaction survey for the years 2010 & 2011. (Application of multicriteria

method MUSA) Criteria Assessed Weight

(%) Assessed satisfaction

index-SI (%)

Content comprehension 16.7 92.3

Connection with prior knowledge and experiences

16.7 94

Quality of the material of the pilot course 16.7 91

Learning & teaching approaches applied 16.7 89.7

Room and recourses used during pilot course

16.7 82

Coordinator’s preparation for the course 16.7 97

GLOBAL SATISFACTION 97.2


All satisfaction indices (partial & global) are very

high

Teaching techniques preference survey using Conjoint Analysis (for the development of communication & teamworking skills)

no Factor Level

1 Written assignments Every week Every two weeks

2 Oral presentations Every week Every two weeks

3 Completion of a group project during semester

Yes (project technique is applied) No (project technique is not applied)

4 Discussion through questions-answers Yes (the technique is applied) No (the technique is not applied)

5 10-minute group exercises by cooperation of two students in every meeting

Yes (the technique is applied) No (the technique is not applied)

The survey is based at a specially designed questionnaire using 10 scenarios via orthogonal design (8 design, 2 holdout scenarios)


Results of the teaching techniques preference survey


13.91%

11.81%

21.48%

29.55%

23.25%

0%

5%

10%

15%

20%

25%

30%

35%

written assignments oral presentations group project discussion throughquestions-answers

10-minute groupexercises

I

m

p

o

r

t

a

n

c

e

Factor

The 2 most preferred techniques

Criteria for the verification of the successful redesign of the pilot course

• Students’ performance: higher than that of the students who attended traditional offered course

• Writing skill: improvement despite the weaknesses

• Speaking skill: improvement despite the weaknesses

• Teamworking skill: improvement despite the lower room for improvement

• Satisfaction survey: high satisfaction partially & globally

• CONCLUSION: successful design of the new pilot course


Some more benefits of the pilot course design & implementation for 3 years

Active participation of students at the learning process Enhancement of students interaction, who are not well-known, friends, etc Connection course’s content with prior learning (knowledge & experience) Development of generic skills of students through a course (major trend applied

universally) Implementation of Formative Assessment techniques (not for grade) Implementation of the peer-assessment technique (new experience for students) Implementation of oral presentation technique on a week basis (new experience for

students at this level of studies) Determining the behavior of students as assessors in peer-assessment process Positive effect of the Learning Objectives of the tutor at the behavior-value system of

the students as assessors Determining the criteria significance weights at the peer-assessment process as

opposed to negotiation process among students and the tutor(s). Determining the preferences of students for teaching & learning approaches applied


Difficulties for the implementation of active learning

At general level: • Low content flow in comparison with the traditional offered course. • Low number of students per group when active learning is applied, which

practically implies more tutors per course or more time spent by one tutor. • High initial investment in time for the design of the course, new materials, etc. • Substantive investment in time during the course which compete other roles of

tutors, like: research, administration tasks, etc. • Change of the applied low level (or no) feedback. Usually, feedback is an under-

estimated process during teaching. • etc

At Department level:

• No room with round table for the meetings of the students-tutor. Technical University of Crete, Department


Difficulties for implementation of active learning (cont.)

At Greek Universities level:

• Absence of a supportive policy for universities in order to describe the Learning Outcomes (Objectives), as opposed to content-based curriculum already applied.

• Staff shortage.

• Educational adequacy of the staff that implements a different technique than the usual lecture.

• Assessment: an under-estimated process usually synonym to grading.

• Generic-transferable skills: unknown words at Higher Education Level.

• The usual assessment technique: summative. New other forms of assessment either formative or summative like e-portfolios are unknown.

• etc


Difficulties for implementation of active learning (cont.)

At methodology level: • Literature review demonstrates the dominance of statistics due to the great interest in such a field

of researchers coming from education, sociology, etc thus the use of multicriteria methods is a novelty. Therefore is not easy to ‘compare’ the usefulness of the different approaches applied with previous research.

• Moreover, is not easy to compare the exact results achieved with other researches due to one more reason: the uniqueness of each course even if the compared courses deal with the same subject. Of course, general speaking the results show the same weak and strong points for our students, which agree in a great extent with literature review.

• Data shortage for testing purposes of methodology and techniques. Each researcher owns data concerning his/her students but these data are not available for more persons even for research purposes.

• There is an extended literature about the skills developed during studies. Many synonyms or nearly synonyms terms are used alternatively, like generic skills, graduate skills, key-skills, transferable skills, soft skills, etc. Similarly, classification attempts have produced a large number of different lists of attributes/skills at international level . Mason (2010) says that ‘a definitive agreed set of skills that students in higher education should be fortunate enough to be born with or expected to acquire simply does not exist. This is not only true but also desirable’.

• etc


A proposal for implementation

As an extra like the pilot course, offered in parallel with the traditional form of the course or at a group

of courses where tutors want as volunteers to act innovatively by applying new methods of teaching

and learning.

Key-point for an ‘innovative’ approach: awareness of the academic community!!


pilot implementation of active learning at an engineering department

Documents