displayed bias as a reflection of both speaker and intended hearer in conversational settings

Carolyn Penstein RoséLanguage Technologies Institute

and Human-Computer Interaction Institute

With funding from the National Science Foundation and the Office of Naval Research

Acknowledgements

DongNguyen

Elijah Mayfield

HuaAi

Rohit Kumar

IrisHowley

Nguyen, D., Mayfield, E., & Rosé, C. P. (2010). An analysis of perspectives in interactive settings, in Proceedings of the KDD Workshop on Social Media Analytics.

Kumar, R. & Rosé, C. P. (2010). Engaging learning groups using Social Interaction Strategies, In Proceedings of the North American Chapter of the Association for Computational Linguistics.

Howley, I., Mayfield, E. & Rosé, C. P. (to appear). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.

Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., Rosé, C. P. (2010). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, in Proceedings of Intelligent Tutoring Systems.

DongNguyen

Elijah Mayfield

HuaAi

Rohit Kumar

IrisHowley

For more information about my group:http://www.cs.cmu.edu/~cprose

OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and Discourse

AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design

discussionsCurrent Directions

Typical paradigm for sentiment analysis of product reviews:Make a prediction based on text of single reviews taken out

of context

Some evidence of group effects in product review blogs based on numerical ratings (Wu et al., 2008)

KEY ASSUMPTION: language is a reflection of the speaker’s perspective

Are product reviews conversational?


Work towards weakening the assumptionSometimes use syntactic cues to reverse polarity on some terms

(Somasunderan & Wiebe, 2009; Wijaya & Bressan, 2008)Factoring out the effect of context rather than modeling it

Aggregation over all reviews posted by the same individual Taking opinions of similar individuals into account (e.g.,

collaborative filtering)

Falls short of modeling conversational aspects of product reviews




Are product reviews conversational?“After many MANY weeks of research, gathering

information from several sites, reviews etc I decided that the Britax Boulevard was definitely the safest bet available on the market. The things that sold me: All the safety gadgets that other seats don't have like the side impact wings, the HUGS system, the LATCH system and 5 point harness and also the fact that it lasts up to 29Kg. “

Are product reviews conversational?“I did most of my research on the net, picking my

top 3 choices I went and had a look at them in the shops. I looked at one the Graco Comfort Sport, the Britax Boulevard and the Decathlon and Marathon seats. By far it seems that Britax have the upper hand safely wise on the market, many professional reviews and crash tests agree on this so Britax was the clear choice for us. “

Are product reviews conversational?“I have the seat front facing in my Camry (2007) I

worried about the size of the chair from reading other reviews but that is NO problem in my car, my son has plenty of leg room and can see perfectly out the window.”


http://www9.georgetown.edu/faculty/irvinem/theory/Bakhtin-MainTheory.html

OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and

Discourse AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design


Discourse and IdentityIdentity is reflected in the way we present ourselves

in conversational interactionsReflects who we are, how we think, and where we

belongAlso reflects how we think of our audience

ExamplesRegional dialect: shows my identification with where

I am from, but also shows I am comfortable letting you identify me that way

Jargon and technical terms: shows my identification with a work community, but also shows I expect you to be able to relate to that part of my life

Level of formality: shows where we stand in relation to one another

Explicitness in reference: shows whether I am treating you like an insider or an outsider

Discourse and Identity

Lave & Wenger, 1991

Discourse is text above the clause level (Martin & Rose, 2007)

A Discourse is an ongoing conversation [type]Socialization is the process of joining a

Discourse (Lave & Wenger, 1991; Sfard, 2010)

We join Discourses that match our core identity (de Fina, Schiffrin, & Bamberg, 2006)

In moving from the periphery to the core of a Discourse community, we sound more and more like the community (Arguello et al., 2006)

A discourse is one instance of it [token]All discourses contain echoes of

previous discourses (Bakhtin, 1983)

Lakoff & Johnson, 1980

Metaphors Structure our Experience

We describe arguments using terms related to warUsing a typical war ‘script’ to

structure a story about an argument

We orient towards arguments as though they were warsOur conversational partner is

our opponentWe may feel that we won or

lostWe may feel wounded as a

result

Discourses, Frames, and Metaphors

Frame: A portion of a discourse belonging to distinct Discourse

Metaphor : One linguistic device that can be used to define a set of discourse practices that constitute a frame

Topic models: a technical approach that makes sense for identifying frames within a discourse

A discourse could be drawn from a mixture of DiscoursesWithin the same conversation, we may wear a

variety of “hats”E.g., the same discourse with a co-worker may

contain exchanges pertaining to our relationship as colleagues and others to our relationship as friends

Model of Communication from Rhetoric

Implied author: Communication style is a projection of identityImpression management, not

necessarily the ground truthImplied reader: What we assume

about who is listeningReal assumptions, possibly incorrectWhat we want recipients or

overhearers to think are our assumptions

Reader: may or may not understand the text the way it was intended

Author

ImpliedAuthor

ImpliedReader

Text

Effect

Reader

Engagement: Social positioning in conversational style

The message: Most contributions express some content

Implied author: How I phrase it says something about my stance with respect to that content

Implied reader: Also says something about what I assume is your stance and my stance in relation to you

Reader: The hearer may respond either to the message or its positioning

Author

ImpliedAuthor

ImpliedReader

Text

Effect

Reader

Engagement: Social positioning in conversational style

Speaker 1: I want chocolate for dessert.Speaker 2: [you can’t have chocolate]

Options with different implications about author and reader You can’t have chocolate. You’re allergic to chocolate. You’re allergic to chocolate, so eating it

would be a bad idea. Your mom said you’re allergic to chocolate Having chocolate might be a poor choice for

you. Having chocolate might be a poor choice for

you for a great number of reasons.

Author

ImpliedAuthor

ImpliedReader

Text

Effect

Reader

Even Scientific Writing is Social and Conversational

Implied author: Rhetorical style in academic writing gives an impression of who we are as researchers

Implied reader: targeting writing to community standardsAbstracts and literature reviews

position us in research communityReader: research papers teach us

both about the content of our field and its politics

Author

ImpliedAuthor

ImpliedReader

Text

Effect

Reader


AnalysisStudy one: Political bias in a political discussion

forumStudy two: Goal orientation in chat based design


Bias Estimation

Start with LDA model of politics dataset with 15 topicsThen separate the texts into two collections, one left

affiliated, and one right affiliatedWe then have a Left model and a Right model

We can then compute a rank for each word w in each topic t in each modelIntuition: a word is more distinguishing for a particular point

of view if it has a high probability within the associated model and a low probability in the opposite model

Bias(w,t) = log(rankright(w,t) + 1) – log(rankleft(w,t) + 1)The bias of a text is the average bias over the terms within

the text Left scores positive, right scores negative

Qualitative Analysis

Terror Language (Right): evokes emotional response to thread of attack. Define target as evil and as a threat. Provokes a defensive posture.

Imperialist rhetoric (Right): racial prejudice, attitude of superiority.

Web of concern (Left): focus on opposition as individuals with a culture and history, concern for wellbeing of all people, focus on potential negative effects of war

Quantitative AnalysisRight BiasLeft Bias

Score of posterScore of quoted

messageScore of full postScore of words

that appear in both messages

Score of words that appear only in quoted message

Score of words that appear only in the post

Investigation of Quoting behavior

Negative correlation between words only in quoted message and words only in post (r=-0.1, p < 0.05)

Positive correlation between score quoted words and score of the whole post (r=0.18, p < 0.02)

Score of words only in post are significantly more reflective of the affiliation of the poster than that of the author of the quoted messageSimilar result with score of words only in quote with

affiliation of author of quoted message


Which words are quoted?

by pointing out the inflation of Saddam’s body count by neocons in an effort to further vilify him and thus further justify our invasion we are not DEFENDING saddam....just pointing out how neocons rarely

let facts get in the way of a good war.

by pointing out the inflation of Saddam’s body count by neocons in an effort to further vilify him and thus further justify our invasion we are not DEFENDING saddam....just pointing out how neocons rarely

let facts get in the way of a good war.

So wait, how many do you think Saddam killed or oppressed? You’re trying to make him look better than he actually was.

You’re the one inflating the casualties we’ve caused! Seriously, what estimates (with a link) are there that we’ve killed over 100,000 civilians. Not some crack pot geocities page either.

So wait, how many do you think Saddam killed or oppressed? You’re trying to make him look better than he actually was.

You’re the one inflating the casualties we’ve caused! Seriously, what estimates (with a link) are there that we’ve killed over 100,000 civilians. Not some crack pot geocities page either.

Thread level analysis

Effect of initial postCorrelation between score thread (without first post)

and first post = 0.210 (p<0.01)Effect of Prior posts

Aggregate score of previous postsDifference in score of current post and average score of

user Small correlation (r=0.133, p < 0.01) indicating that

users talk more left than they usually do when previous posts are left and similarly for right.

Overview of Findings

Quotes from opposite point of view include the words that are less strongly associated with the opposite perspective

Because of quotes, displayed bias shifts towards the bias of the person to whom the message is directed

Personal bias of the speaker is most strongly represented by non-quoted portions of text

The effect of a post extends past just the immediate response

Collaborative Design TaskGoal: Design a power plantCompeting Student Goals:

Power: Design a power plant that achieves maximum power output

Green: Design a power plant that has the minimum impact on the environment

Competing goals encourages deeper discussionExploration of design spaceExplicit articulation of reasoning

30> Experimental Design >Task

31

Classroom Study

Chat room style interaction ConcertChat

106 studentsCMU undergrads in ME

Classroom sessionDuring the semester• Instruction Pretest

Session Posttest Questionnaire

31> Experimental Design >Study Procedure

Experimental Design

HypothesesStudents will be more engaged with agents displaying

social behaviorsStudents will be sensitive to tutor goal orientationInteraction effect

FrequentGreen

InfrequentGreen

FrequentNeutral

NonePower

NoneNeutral

FrequentPower

InfrequentNeutral

InfrequentPower

NoneGreen• Social Behavior

– Frequent, Infrequent, None

• Goal Alignment– Green, Power, Neutral

32> Experimental Design >Manipulations

An Example of Displaying Bias

33> Experimental Design >Agent Design

Green BiasGreen: What is bad about increasing heat input to the cycle is that

more waste heat is rejected to the environment.Neutral and Power: Increasing heat input to the cycle increases

waste heat rejected to the environment.

Power Bias:Power: What is good about increasing heat input to the cycle is

that more power output is produced.Neutral and Green: Increasing heat input to the cycle increases

power output produced.

34

Example of Social Behaviors

34> Experimental Design >Agent Design

1. Showing Solidarity: Raises other's status, gives help, reward

2. Showing Tension Release: Jokes, laughs, shows satisfaction

3. Agreeing: Shows passive acceptance, understands, concurs, complies

Tutor: Let’s Introduce ourselves. My name is Avis. Tutor: Be nice to your teammates!

Tutor: I’m happy to work with our team :-)

Tutor: m-hmm (showing attention)

Adapted from Bales’ IPA (Bales, 195)

Experimental Design

HypothesesStudents will be more engaged with agents displaying

social behaviorsStudents will be sensitive to tutor goal orientationInteraction effect

FrequentGreen

InfrequentGreen

FrequentNeutral

NonePower

NoneNeutral

FrequentPower

InfrequentNeutral

InfrequentPower

NoneGreen• Social Behavior

– Frequent, Infrequent, None

• Goal Alignment– Green, Power, Neutral

35> Experimental Design >Manipulations

36

Measuring Student BiasUsing a topic modeling tool – ccLDA [Paul and Girju, 2009]

36> Experimental Design >Displaying Bias

CorpusCollection1 Collection2

Topic1

Topic 2

Topic 3

Example of Extracted Topics Heat quality right max decrease possible goes efficiency need gas graph say natural want goal fuel Tmax min sounds temp going friendly turbine kpa mean

11000 values different makes larger graphs bit green large kind produce hate steam team step solid 6574 split bored nat geo instead happens plant love

yah blades sir dunno kk x85 rejected guessing starts FINAL life helping compromise nd depends corresponding teammate stays tmin new hard sitting afk tmax500 bec

power decreases nuclear make 85 cycle work guess high pmin want pmax wait 570 lower green 40 tmax Pmin value low best point pick environment

low 500 12800 sort 1 tutors effeciency 440 coool ecofriendly half fun 105 Nuclear sweeet maximized cooler question boy 6000 worked creepy Goes 16250 maxes

generates makes 085 different 7000 12000 qdot becuase decreasing click leads liquid gues doubt 10790 meet POWER 6574 DESIGN transfer hope Qin 11000 discussion km

TOPIC 1 TOPIC 2Background Green Power Background Green Power


Bias Measurement MetricsMax Topic-word bias: count the number of words in the list of

the N most strongly associated words, and take the maximum across topics

Average Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the average across topics

Weighted Topic-Word Average bias: Same but weight each word by its association within the background model first

All three measures highly correlated both for Green and for Power perspectives

Students in the Green condition got higher Green scores on average than Power scores and vice versa in the Power conditionOnly statistically significant for the first two metrics

Measuring Influence

Within pairs, the Green score of the Green student and the Green score of the Power student were significantly correlatedSame story for Power scoresResult consistent with analysis of Politics dataset


Operationalization of Authoritativeness

Negotiation coding scheme (Martin & Rose, 2007, Chapter 7)

Agreement on K1/K2/Other .72 Kappa

Coded all transcripts from Infrequent Social conditionAuthoritativeness score

= K1/[K1 + K2]Within pairs, one

Authoritative student and one NonAuthoritative student

Balance Effect• Alignment

• Align: Authoritative partner shares affiliation with agent

• Neutral: Agent is neutral• NoAlign: Non-Authoritative partner shares

affiliation with agent• Affiliated agent has polarizing effect on

displayed bias• Difference in bias scores was significantly higher

in conditions with affiliated agents• Direction of polarization depends on alignment

• Balance Effect• Authoritative student shows less of his own

bias when he’s in the minority• NonAuthoritative student is less non-

authoritative when he’s in the minority

Overview of Findings

Topic models can display differences in goal orientation in chat data

Confirmation of influence of partner speech on displayed bias

Complex relationship between personal orientation, authoritativeness, and ingroup/outgroup effects

Current DirectionsFull circle for opinion miningContinuing to operationalize multiple dimensions of

relational codesHowley, I., Mayfield, E., & Rosé (to appear). Linguistic Analysis

Methods for Studying Small Groups, in Hmelo-Silver, O’Donnel, & Chan (Eds.) International Handbook of Collaborative Learning, Taylor & Francis, Inc.

Collaboration with Bob Kraut: investigating how exchange of social support is reflected in relational codes

Collaboration with Bhiksha Raj: investigating evidence of relational codes like Negotiation in speech

CHI submission in progress: Analysis of effects of bullying behavior on distribution of relational codes and learning

New grant!: studying the emergence of leadership in ad-hoc teams

Computational Models of Discourse AnalysisFocus on literature from the field of Discourse

AnalysisInvestigating issues such as Conversational Structure,

Attitude, Perspective, Persuasion and Positioning Critical reflection on the state-of-the art in language

technologiesHands-on programming assignments, fun contests

Carolyn Penstein Roséhttp://www.cs.cmu.edu/~cprose

[email protected] Center 5415

displayed bias as a reflection of both speaker and intended hearer in conversational settings

Documents