Writing Studies and Innovation in Designing

NLP Educational Applications:

A Multidisciplinary PerspectiveWorkshop 3: 9th Workshop on Innovative Use of NLP

for Building Educational Applications (BEA)Norbert Elliot

Professor of EnglishJune 26, 2014

Computational Linguistics, 2014

Cortes, Mohri, & Rostamizadeh Lee, Liu, & Popović

Rhetoric and Composition/Writing Studies, 2014Weigle Dikli & Bleyle

Outline

• Part 1: Automated Writing Evaluation—Perspectives from Rhetoric and Composition/Writing Studies

• Part 2: Case Study, 2010 to 2014• Part 3: Directions for Multidisciplinary,

Collaborative Research

Part 1: Automated Writing Evaluation—Perspectives from Rhetoric and Composition/Writing Studies

What is the field of Writing Studies?What is the relationship between Writing

Studies and AWE?

Writing Studies: A Chronology

• Modern Language Association Founded: 1883

• National Council of Teachers of English Founded: 1911

• Conference on College Composition and Communication Founded: 1948

• The Visibility Project: 2004-2010– 23 English Language and

Literature/Letters– 23.13 Rhetoric and

Composition/Writing Studies– 23.1301 Writing, General– 23.1302 Creative Writing– 23.1303 Professional,

Technical, Business and Scientific Writing

– 23.1304 Rhetoric and Composition

– 23.1399 Rhetoric and Composition/Writing Studies, Other

Writing Studies: Traditional Beliefs

• Aristotle (335 BCE)– “Rhetoric may be

defined as the faculty of observing in any given case the available means of persuasion. This is not a function of any other art.” (Rhetoric, Book 1, Chapter 2, p. 24)

• Roberta Camp (1992)– Writing is “a rich,

multifaceted, meaning-making activity that occurs over time and in a social context, an activity that varies with purpose, situation, and audience and is improved by reflection on the written product and on the strategies used in creating it.” (p. 135)

Writing Studies: Emerging Beliefs

• White, Elliot, Peckham (forthcoming)– writing within a digital

environment is an activity that emphasizes audiences within and beyond the classroom, an activity that resonates with the writer’s use of image and voice and improves through collaboration and experimentation.

• Council of Writing Program Administrators, National Council of Teachers of English, and National Writing Project (2011)– Habits of mind– Writing, reading, and

critical analysis experiences

Writing Studies: Construct Span

Communication in Digital, Print, Blurred Environments

Writing and Reading, Visualization, and Speaking

Rhetorical Conceptualization

Cognitive Domain: Genre, Task, Audience, Writing Process, Problem Solving, Information Literacy, Conventions, Metacognition

Interpersonal Domain: Collaboration, Social Networking, Leadership, Diversity, Ethics

Intrapersonal Domain: Openess, Conscientiousness, Extraversion, Agreeableness, Stability

Neurological, Attention, and Visual Domain

Intersection: Automated Writing Evaluation and Writing Studies

Digression: Multidisciplinary Research• Fish (1989)

– “The blurring of existing authoritative disciplinary lines and boundaries will only create new lines and new authorities; the interdisciplinary impulse finally does not liberate us from the narrow confides of academic ghettos to something more capacious; it merely redomiciles us in enclosures that do not advertise themselves as such” (237).

• Elliot & Klobucar (2013)– Importance of

multidisciplinary research– CIP Codes: Teams for

Multidisciplinary AWE Research

• 11: Computer and Information Sciences

• 13: Education• 16: Foreign Languages,

Literatures, and Linguistics• 23: English Language and

Literature• 27: Mathematics and

Statistics• 42: Psychology• 52: Business, Management,

and Marketing

Determining Directions for Future Research

• Not in our control– Interpretations

inherent in technological determinism

• Within our control– Patterns of

adoption associated with technological transfer

Part 2: Case Study, 2010 to 2014

What work has been done to bridge the gap between Writing Studies and AWE?

Measures of the Writing Construct

Klobucar, Deane, Elliot, Ramineni, Dees, Rudniy (2012)

Portfolio and course grade: r = .43Combined AWE score and portfolio relationship: r = .39Combined AWE score and course grade: r = .29

Klobucar, Elliot, Dees, Rudniy, Joshi (2013)

Statistically significant relationships

Variation in group predictions

Elliot & Klobucar, 2013

AWE within an environment of instruction and assessment

Burstein & Elliot, Holtzman, Lentini, Molloy, Shore, Steinberg, & Vezzu, 2014

Commonality = black Disjuncture = red (lower)/green (higher)

Two-proportion Z-test used to evaluate statistically significance (p < .000)

Alvarez, Biswal, Elliot, Klobucar (in process, IRB F 183-13); ETS partnership

• Subject: A neural basis of writing

• Aim: Examination of modes of discourse scored by e-rater®

• Sampling plan: High performing writers and emerging writers

• Analysis: Resting-state functional MRI (R-fMRI)

Part 3: Directions for Multidisciplinary, Collaborative Research

What might the future hold for multidisciplinary collaboration?

Future Direction 1: A Multidisciplinary Program of Research• Identify consequence as a

primary category of evidence• Distinguish between mode

(exposition, description, narration, and argumentation [EDNA]) and aim

• Reconsider hierarchy of the variables of writing ability

• Engage in robust construct modeling, with machines and humans serving designated roles based on known assessment limits.

Future Direction 2: Domain Models

Future Directions 3: Embedded Systems of Response

Future Direction 4: A Body of Knowledge Initiative for AWE

Future Direction 5: Contingency