PIER Research Methods Protocol Analysis Module

Hua Ai

Language Technologies Institute/ PSLC

Questions? "There was a significant negative

correlation between the first and third metrics used to compute a Power score for the partner's conversational contributions and this question's numeric value, and a marginal negative correlation in the case of the second metric.“What are we supposed to interpret/learn from

this statement?

Rose at el. Automatic Analysis

By Machine Learning

What is machine learning?

Machine learning is aboutautomatically finding meaningful

patterns in data

Example for medical data:Rule predicts who is more likely to have problems with their teeth as they get older.

Why Machine Learning? We use machine-learning

products every day Weather forecast Spelling checker Automated voice response system …

It has been successfully applied to many research areas Natural language processing Market analysis Bioinformatics …

Note: Search engines use machine learning to personalize search results and suggest related sites or queries.

How does machine learning work?

The simplest rule learner willlearn to predict whatever isthe most frequent result class.This is called the majorityClass.

What will the rule be in this case?

It will always predict yes.

A slightly more sophisticated rule learner will find the feature that gives the mostinformation about the result class. Whatdo you think that would be in this case?

Outlook:Sunny -> NoOvercast -> YesRainy-> Yes

<Feature Name>:<value> -> <prediction><value> -> <prediction>…

What is machine learning?

Automatically or semi-automatically Inducing concepts (i.e., rules) from dataFinding patterns in dataExplaining dataMaking predictions

Data Learning Algorithm Model

New Data

PredictionClassification Engine

What will be the prediction?

Outlook:Sunny -> NoOvercast -> YesRainy-> Yes

Model

New Data

Yes

Terminology

Concept: the rule you want to learn

Instance: one data point from your training or testing data (row in table)

Attribute: one of the features that an instance is composed of (column in table)

* Compute the predicted value.

bad

Task Assign labels to a collaborative learning

corpus using the Weinberger and Fischer’s coding scheme Text classification task

Two approach categories The Feature Based Approach

Basic feature Thread Structure feature – depthFSM featuresLIWC features (Linguistic Inquiry Word Count)

The Algorithm Based ApproachCascaded Binary ClassificationConfidence Restricted Cascaded Binary

Classification Supervised/Unsupervised

Methodology issues related to automatic corpus analysis Validity

Whether the automatic coding accomplished by the computer captures human analysts’ intention

ReliabilityHow faithfully the automatic codes match those

of human experts’ Efficiency

Are we saving time by using the automatic classifier?

Ai et al. Manual Analysis

By statistical tests Interaction between tutor’s social behaviors

and tutor’s bias towards one student’s stance or the other

Topic modeling: sub-dialog level instead of utterance level

Study design 3*3

Social: how, low, noneGoal Match: yes, no, neutral

The ccLDA model

Green Collection Power Collection

Topic 1

Topic 2 Topic 3 Topic 1 Topic 2

Topic 3

A topic is a distribution of words.

We computed a score for each utterance based on the words contained in the utterance. This score stands for to what extend this utterance is biased towards this topic.

Results Learning Gain

Student learned most in Low-social + Yes-matched

Perception (Questionnaire & Topic detection)Effect on Goal-match manipulation

Conversational DataMore social turns in social conditionsMore off-task turns in non-social conditionMore Jokes on tutor in the high social condition

Important message Understand the data is important

Design good features Both in automatic and manual analysis