LEARNING ANALYTICS TO EXPLORE TEACHING AND LEARNING DISCOURSE

(OR)

LET’S PUT LANGUAGE INTO LEARNING

ANALYTICS

Jenny McDonald

Higher Education Development Centre (HEDC)

University of Otago, Dunedin, NZ

17th March, 2016. ACODE 70, Charles Sturt University, Orange, NSW

Acknowledgements:

Dr Rebecca Bird, Dr Ruth Napper, Dr Jeff Erickson, Dr Greg Jones (HUBS192)

Adon Moskal and Richard Zeng (HEDC)

Dr Amal Zouaq (University of Ottawa), Dr Michel Gagnon (Ecole Polytechnique, Montreal)

Argument

• Learning isn’t something that happens because you are

retained!

• Learning doesn’t happen in a straight line

Learning is being a non-native speaker in a

new land

Therefore:

If we are to teach….

Argument

• Need to capture more than:

• Traveller attributes

• Footprints in the sand/snow/…

• Resting and reflection places

• Pictures taken

• Diary notes

• Departure from well-worn tracks…

Argument

• Need to capture understanding in context:

• The student in the landscape

• Why they go where they do

• Where and why they choose to rest and reflect

• How they see the world

• How they interact with the world

• The tracks they form for themselves …

Language and learning

Piaget

=> We construct meaning

Vygotsky

=> We (co)construct meaning in a social context

Halliday and others..

=> Language is the essential condition through

which we construct meaning

Biber and others…

=> Distinct disciplinary differences in language

Context: HUBS 192

• Core 1st year undergraduate health science course

• Enrolment 1600-2000 students => ~ 1300 complete

• 5 modules => basic human anatomy and physiology

• 4 lectures/week, 7 labs/semester plus self-study modules

• => Information-dense, competitive course

• Final exam (72%) includes short-answer questions

(SAQ)….

• Students perform poorly on SAQ

• How to provide students with feedback on practice

questions?

Objective: Go back a step

• How to understand student responses in order to provide

meaningful feedback?

• Should situate student responses in context

• Aim to identify sources of student responses or part-

responses from curricula material (e.g. lecture

transcripts, course-notes, textbook etc)

=> Inform pedagogic action

Review student responses

Is it possible to group them? Are some responses:1. Both good and poor?/correct and incorrect?

2. Incomplete?

3. Ambiguous?

4. Indicating that question was ambiguous?

5. Self-referential?

6. Confused or don’t understand?

7. Naïve

8. …

Overlap within and between groups? 1. Common words/phrasing

2. Belong to more than one group

3. …

Automated Short-Answer Response Analysis

• Recent progress in automatic short-answer grading

(Burrows, Gurevych, & Stein, 2015).

• Analysing student responses and automatically assigning

them to meaningful categories to provide individualised

feedback to students is still a work in progress

(Dzikovska, Nielsen, & Leacock, 2015)

• A long way from widespread application in the broader

higher education setting. Why?

Automated Short-Answer Response Analysis

• Response classification for 1 x question (at best) ~ 90% accuracy

• => Over say, 5 questions only a ~ 60% chance that all are classified correctly

• => individualised feedback on short answers is a very hard problem!!

c.f. Teacher dashboard

Classification or grouping even 60-80% accurate (compared to human analyst) => still see patterns and can check context

=> Informed pedagogical response ✓

Future work

• Automate the teacher dashboard: S2, 2016 in 3 classes

• Evaluate the dashboard in use

• Can we devise a measure (or qualitative categories) of

disciplinary fluency development?

• Contribute to both practice and theory of language and

learning?

ReferencesBiggs, J. B. and Tang, C. (2011). Teaching for quality learning at university: What the student does.

McGraw-Hill Education (UK).

Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading.

International Journal of Artificial Intelligence in Education, 25(1), 60-117.

Cortes, V. (2004). Lexical bundles in published and student disciplinary writing: Examples from history and

biology. English for specific purposes, 23(4), 397-423.

Csomay, E. (2013). Lexical bundles in discourse structure: A corpus-based study of classroom discourse.

Applied linguistics, 34(3), 369-388.

Dzikovska, M. O., Nielsen, R. D., & Leacock, C. (2015). The joint student response analysis and

recognizing textual entailment challenge: making sense of student responses in educational applications.

Language Resources and Evaluation, 1-27.

Halliday, M.A.K. (1993) Towards a language-based theory of learning. Linguistics and Education Vol 5. 93 -

116

Hattie, J. (2009). Visible Learning: A synthesis of over 800 meta-analyses relating to achievement. New

York: Routledge.

Jurafsky, D. and Martin, J. (2009). Speech and language Processing. New Jersey:Prentice Hall.

Lakoff, G. (1993). The contemporary theory of metaphor.

Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of

learning technologies (2nd ed.). London: Routledge falmer.

Marton, F., & Säaljö, R. (1976). On qualitative differences in learning—ii Outcome as a function of the

learner's conception of the task. British Journal of educational Psychology, 46(2), 115-127.

Reddy, M. J. (1979). The conduit metaphor: A case of frame conflict in our language about language. In A.

Ortony (Ed.), Metaphor and thought (pp. 284-324). Cambridge: Cambridge University Press.