reflecting on reflective writing analytics (lak16)

Reflec%ng on Reflec%ve Wri%ng Analy%cs Assessment Challenges and Itera0ve Evalua0on of a Prototype Tool Simon Buckingham Shum, Ágnes Sándor, Rosalie Goldsmith, Xiaolong Wang, Randall Bass, Mindy McWilliams University of Technology Sydney, Xerox Research Centre Europe, Georgetown University

Who we are and what we contribute…

Ágnes Sándor Xerox Research Centre Europe

Grenoble (linguist: modelling academic wri0ng)

Natural Language Processing

service

Simon Buckingham Shum & Shawn Wang Connected Intelligence Centre

(learning analy0cs researcher and web developer)

learning analy0cs, user requirements, web applica0on, user interface design and

evalua0on

Rosalie Goldsmith Ins0tute for Mul0media Learning (researcher in reflec0ve wri0ng in engineering)

Reflec0ve wri0ng scholarship and coding

Randall Bass, Mindy McWilliams Georgetown University (Forma0on by Design Project – student reflec0ve wri0ng)

…IF you know how to write to be1er reflect …AND IF you know how to

make your thinking visible using language

WHY FOCUS ON REFLECTION? SOME OF THE GRADUATE ATTRIBUTES WE WANT TO BUILD

Authen%c Performance

Professional Disposi%on

Emo%onally Integrated Reflec%on

Theory & Prac%ce reflec=ve wri=ng is a window onto the mind

Sense of Calling

Epistemological ShiRs

REFLECTIVE WRITING EXAMPLE: NURSING PRACTICE

Price, B and Harrington, A (2013) Cri=cal Thinking and Wri=ng for Nursing Students. London: Sage/Learning Ma_ers. (Example Reflec0ve Essay for Cri0cal Thinking and Wri0ng for Nursing Students) h_ps://au.sagepub.com/sites/default/files/upm-‐binaries/54814_Example_reflec0ve_essay.pdf

“To help structure this essay I use the framework described by Gibbs (1988). Whilst the episode concerned relates a stage in Mrs Drew's illness when she challenged her treatment protocol, it also includes some of the memories and thoughts that this pa0ent refers to regarding her earlier illness and past ways of coping with pain. In par0cular, it prompted me to ques0on to what extent I as a nurse should recommend analgesia, drawing on what I had been taught about the effec0ve control of pain. I had learned that it was be_er to control rather than to chase pain (e.g. Mann and Carr, 2006 ; Forbes, 2007).”

REFLECTIVE WRITING EXAMPLE: NURSING PRACTICE

Price, B and Harrington, A (2013) Cri=cal Thinking and Wri=ng for Nursing Students. London: Sage/Learning Ma_ers. (Example Reflec0ve Essay for Cri0cal Thinking and Wri0ng for Nursing Students) h_ps://au.sagepub.com/sites/default/files/upm-‐binaries/54814_Example_reflec0ve_essay.pdf

Reflec0on Analy0cs from AWA: Academic Wri0ng Analy0cs web app h_ps://utscic.edu.au/tools/awa

REFLECTIVE WRITING EXAMPLE: TEACHING PRACTICE

N. Ha_on and D. Smith, (1995) Reflec0on in Teacher Educa0on: Towards Defini0on and Implementa0on. Teacher and Teacher Educa=on, Vol. 11, No 1 pp. 33-‐49

“While I had planned to use mainly wri_en text materials I became aware very quickly that a number of students did not respond to these. Thinking about this now there may have been several reasons for this. A number of students, while reasonably proficient in English, even though they had been NESB learners, may s0ll have lacked some confidence in handling the level of language in the text. Alterna0vely, a number of students may have been visual and tac0le learners. In any case I found that I had to employ more concrete ac0vi0es in my teaching.”

WHAT MAKES REFLECTING WRITING DIFFERENT TO DEMONSTRATING YOUR ACADEMIC MASTERY OF A TOPIC?

Students are encouraged to say what they don’t know

Expressions of internal conflict and uncertainty are valued

Insights into one’s changing understanding of oneself, studies and purpose

Experience and one’s personal responses (including emo0onal) to it is central

Oqen looking to demonstrate growth of one’s professional disposi%on and interpersonal quali%es

STARTING POINT — RUBRICS: LANGUAGE + DISCOURSE FEATURES + TYPE OF INFORMATION

Context of the event that triggers the reflec0on (why, when, where, who, how much, what)

Specific informa0on about the event; ac0ons , ideas, learning experiences

Changes in assump0ons about learning, or prior knowledge and actual experience/performance

References to the past: 0me markers and use of past tense (when I started; before my internship); shiq between habitual past tense (I used to) and the present or the recent past (since then I have …)

REFLECTIVE WRITING IS CHALLENGING FOR EDUCATORS AS WELL AS STUDENTS

Challenging to teach Different interpreta0ons of what reflec0ve wri0ng is/what it looks like A novel genre for many educators and students

Challenging to assess How do you iden0fy a deep or a superficial reflec0on? How important is content mastery vs depth of reflec0on? Can some students be advantaged/disadvantaged?

FROM INFORMAL RUBRICS TO RHETORICAL PATTERNS IN AN ANALYTICS TOOL

10

"I am grateful for the practical component that this internship has offered as I feel I have achieved a greater knowledge of just simply 'how things work'."

Academic’s Rubric: “Verbs that show awareness or shifts in perception (I began to understand, I could see, I could visualise, I could perceive, I became aware, I became, I grew, I realised, I recognised”

FROM INFORMAL RUBRIC TO RHETORICAL PATTERN

Reflection label: shift in perception


FROM INFORMAL RUBRIC TO RHETORICAL PATTERN Academic’s Rubric: “Verbs that show awareness or shifts in perception (I began to understand, I could see, I could visualise, I could perceive, I became aware, I became, I grew, I realised, I recognised”


"Understanding how the entire company works grants a holistic overview of business operations and often allows me to understand the office procedures and processers."


FROM INFORMAL RUBRIC TO RHETORICAL PATTERN Academic’s Rubric: “Verbs that show awareness or shifts in perception (I began to understand, I could see, I could visualise, I could perceive, I became aware, I became, I grew, I realised, I recognised”



"I can already see that my attitude towards University has changed."

AUTHOR MENTAL

FROM INFORMAL RUBRIC TO RHETORICAL PATTERN

Rhetorical Pattern: CHANGE


Academic’s Rubric: “Verbs that show awareness or shifts in perception (I began to understand, I could see, I could visualise, I could perceive, I became aware, I became, I grew, I realised, I recognised”





IMPLEMENTATION: XEROX INCREMENTAL PARSER

syntactic dependency parsing


lexical database

+




lexical database

co-occurrence rules

+

+



STUDENT CONTEXTS

18

UTS ENGINEERING PRACTICE STUDENTS §  4 year degree; two 6-‐month internships

§  Graded reflec0ve report on changes in their professional, personal and technical awareness.

§  Approx 200 students per semester, 40-‐50 page reports — very 0me-‐consuming to mark.

§  So very difficult for tutors to provide forma0ve feedback on draqs

§  New requirement for finer-‐grained assessment and grading of reflec0ve wri0ng — wri0ng analy0cs could make a significant impact

19 Course details h_p://handbook.uts.edu.au/subjects/48121.html

GEORGETOWN UNIVERSITY WELL-‐BEING PROJECT

20

Reflective Writing Prompt: Georgetown has a commitment to educating the whole person. Some students find that this course has affected them as a person, other students find that it has not. Tell us about your experience in this course.

GU Forma0on by Design Project h_ps://futures.georgetown.edu/forma0on

GEORGETOWN UNIVERSITY WELL-‐BEING PROJECT

21

Example response: “This class is the first course I have taken at Georgetown that has made me really feel like I can make a difference in the world and that has helped me to grow as a whole person. It has caused me to reflect on several harms our world is facing and I left class most days feeling very passionate and motivated to make a difference in the world. This course often helped me to put life in perspective and helped me realize what really matters to me as an individual in society.”

EVALUATION

22

COMPARISON OF HUMAN AND MACHINE ANNOTATION

23

human machine

This counts as a “True Posi0ve”

COMPARISON OF HUMAN AND MACHINE ANNOTATION

24

human highligh0ng automated highligh0ng

25

METRICS TO QUANTIFY HUMAN-‐MACHINE MATCH

v0.1: 30 pieces of wri%ng, with 382 sentences from GU students Coded without knowledge of the parser Promising given the parser was developed for engineering reflec0on

v0.2: expanded lexicon, disambiguated words, new sentence categories (See Sec. 6.1 for details)

26

METRICS TO QUANTIFY HUMAN-‐MACHINE MATCH

v0.3: 312 extracts and 2366 coded sentences accuracy did not decrease significantly: promising since the new evalua0on corpus had almost ten 0mes as many sentences as the first, increasing the number of poten0al new words that might not have been recognised. A closer look at False Nega=ves and False Posi=ves was needed…

A CLOSER LOOK AT FALSE NEGATIVES

Missing words from reflec0ve lexicon •  “Over the past year I have come to realize that many of my close friends seek support and counseling through campus support and outside healthcare providers.”

Human coding for ac0on inspired by reflec0on •  “When I walk into a lecture hall, I look for a familiar face, perhaps one that I met during [course name].”

27

A CLOSER LOOK AT FALSE NEGATIVES

Human coding for ac0on inspired by reflec0on “Looking back on the semester, I don’t think I could have felt as comfortable and at ease as I do now without this class. When I walk into a lecture hall, I look for a familiar face, perhaps one that I met during [course name].”

28

Poten=al future work: looking at what comes aQer reflec=ve markers. Classifying inten=ons to act in the future.

A CLOSER LOOK AT FALSE POSITIVES

29

0 = surface self-‐reflec0on

Human considera0ons: •  text length counts – this was all the student wrote •  lack of detail or explana0on = lack of reflec0on

depth

Machine annota0on: •  sentence by sentence •  surface/deep reflec0on

dis0nc0on not implemented

Future work: paragraph-‐level analysis

FUTURE WORK: FROM VALIDATING AWA WITH EDUCATORS, TO SUPPORTING STUDENT REFLECTION

From highligh0ng to ac0onable reports •  How to bridge the gap between the current ability to highlight sentences, and

capability to generate a meaningful, ac0onable report “Does this highligh0ng mean it’s good?” •  Without gevng into automated grading, students (and educators) are keen to know

if there are signature pa_erns that are proxies for quality From “Algorithmic Accountability” to “Analy0cs System Integrity” •  How is trust forged between all the stakeholders? Educators, Analysts, Students 30

reflecting on reflective writing analytics (lak16)

Education