1 language, science and data science kathleen mckeown department of computer science columbia...

1

Language, Science and Data Science

Kathleen McKeownDepartment of Computer Science

Columbia University

2

3

Science and Language

Predicting impact of scientific journal articles

Generating updates on disaster

4

Machine learning framework

Data (often labeled)

Extraction of “features” from text data

Prediction of output

5

Machine learning framework

Data (often labeled)

Extraction of “features” from text data

Prediction of output

What data is available for learning?

6

Machine learning framework

Data (often labeled)

Extraction of “features” from text data

Prediction of output

What features yield good predictions?

7

Science and Language

Predicting impact of scientific journal articles

Generating updates on disaster

8

9

What can articles yield?

10

Alexandrescu and Kirchhoff, 2007

Lee and Ng, 2002

Ando, 2006Niu et al., 2005

Stevenson and Wilks, 2001

Ng et al., 2003

Rigau et al., 1997

Yarowsky, 1995

Stetina et al., 1998

Ng and Lee, 1996

Peh and Ng, 1997

Yarowsky, 1992

Khapra et al., 2011

Chan et al., 2007

Tim

e

Qazvinian et al, JAIR 2013Citation Network

11

What can articles yield? Argumentative zoning (Teufel, 1996)

“Here, we present quantitative estimates of the global biological impacts of climate changes.” [AIM]

Citation sentiment and scope “The approach of economists takes a

broader view. … We argue that this approach misses biologically important phenomena.”

12

Prediction of Scientific Impact

Climate change, Climate model

Input: term, document

Extract features from full text and metadata

Predict prominence of

term, document

13

Approach

Metadata features Citation network Author collaboration

network

Text features Argumentative

zoning Citation sentiment Primary concepts Relations

Logistic regression to predict future impact

14

Data

4 million full-text Elsevier journal articles

48 million Web of Science metadata records

Fields: medical, chemistry, biology, computer science, …

Ground truth function

15

ExperimentsExperimental comparisons of text features vs metadata features

Data: Selected Elsevier articles and counterparts in WOS

16

What have We Learned?

17

What have we learned?

18

Text features alone outperform metadataElsevier data

Text

Metadata

19

Metadata adds value when combined with textElsevier data

Text

Text+metadata

20

ExperimentsExperimental comparisons of text features vs metadata features

Data: All WOS and Elsevier data

21

What have We Learned?

22

Text better than metadata on Web of Science

Text

Metadata

23

Science and Language

Predicting impact of scientific journal articles

Generating updates on disaster

24

It’s October 28th, 2012

25

26

27

28

29

30

Monitor events over time

Input: streaming data

News, social media, web pages

At every hour, what’s new

31

Data NIST: Trec Temporal Summarization

Track hourly web crawl October 2011 - February 2013 16.1TB

Natural disasters, industrial disasters, social unrest, …

Training Data drawn from Wikipedia

32

33

The U.S. Pacific Tsunami Warning Center said there was a possibility of a local tsunami, within 100 or 200 miles of the epicenter,

34

Would you like to contribute to this story?Start a discussion.

35

Add to Digg. Add to del.icio.us. Add to Facebook.

36

System Update

37

Temporal Summarization Approach

At time t:

1. Predict salience for input sentences Disaster-specific features for predicting

salience

2. Remove redundant sentences

3. Cluster and select exemplar sentences for t

Incorporate salience prediction as a prior

Kedzie & al, Bloomberg Social Good Workshop, KDD 2014Kedzie & al, ACL 2015

38

Predicting Salience: Model FeaturesBasic sentence level features

sentence length punctuation count number of capitalized words number of event type synonyms, hypernyms, and hyponyms

39

Predicting Salience: Model FeaturesBasic sentence level features

sentence length punctuation count number of capitalized words number of event type synonyms, hypernyms, and hyponyms

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

40

Predicting Salience: Model FeaturesBasic sentence level features

sentence length punctuation count number of capitalized words number of event type synonyms, hypernyms, and hyponyms

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

41

Predicting Salience: Model FeaturesBasic sentence level features

sentence length punctuation count number of capitalized words number of event type synonyms, hypernyms, and hyponyms

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

42

Predicting Salience: Model FeaturesLanguage Models (5-gram Kneser-Ney model)

generic news corpus (10 years AP and NY Times articles) domain specific corpus (disaster related Wikipedia

articles)

43

Predicting Salience: Model FeaturesLanguage Models (5-gram Kneser-Ney model)

generic news corpus (10 years AP and NY Times articles) domain specific corpus (disaster related Wikipedia

articles)

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

44

Predicting Salience: Model FeaturesLanguage Models (5-gram Kneser-Ney model)

generic news corpus (10 years AP and NY Times articles) domain specific corpus (disaster related Wikipedia

articles)

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

45

Predicting Salience: Model FeaturesLanguage Models (5-gram Kneser-Ney model)

generic news corpus (10 years AP and NY Times articles) domain specific corpus (disaster related Wikipedia

articles)

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

46

Predicting Salience: Model FeaturesGeographic Features

tag input with Named-Entity tagger get coordinates for locations and mean distance to event

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

47

Predicting Salience: Model FeaturesGeographic Features

tag input with Named-Entity tagger get coordinates for locations and mean distance to event

High SalienceNicaragua's disaster management said it had issued a local tsunami alert.

Medium SaliencePeople streamed out of homes, schools and oce buildings as far north as Mexico City.

Low SalienceAdd to Digg Add to del.icio.us Add to Facebook Add to Myspace

48

Predicting Salience: Model FeaturesTemporal Features measuring burstiness of words

average tf-idf at the current time t0 difference of average tf-idf at time t0 and t - 1 difference of average tf-idf at time t0 and t - 5 hours since event start time

49

SubEvent Identification

➔Decompose articles on a main event into related sub-events:

Manhattan Blackout Breezy Point fire Public Transit Outage

Hurricane Sandy

50

What have We Learned?

51

What have We Learned?

Salience predictions increase precision

52

What have We Learned?

Salience predictions increase precision

Penalizing redundancy removes too many important facts

53

Integrate lens of the media and scientific knowledge

To better predict unfolding impact oof disaster

Social media to feed into modelsof resilience

54

In Sum

Can exploit the words on the web

Language analysis can help the world of science

55

Thank You!

The research presented here has been supported in part by DARPA BOLT, IARPA FUSE, and NSF.