grounding language with points and paths in continuous spaces

Berkeley

N L P

Grounding Language with Points and Paths in Continuous Spaces

Jacob Andreas and Dan KleinUC Berkeley

Berkeley

N L P

Formal grounding

On June 26th, Facebook stock cost $65 per share

quote {date: 2014-06-26,stock: FB,price: $65

}

Perceptual grounding

On June 26th, Facebook stock reboundedafter a bruising swoon

?

Perceptual grounding

On June 26th, Facebook stock reboundedafter a bruising swoon

Perceptual grounding

On June 26th, Facebook stock reboundedafter a bruising swoon

A after B A, B A before B B, A rebounded { sgn(slope) = +1 }

bruising { sgn(slope) = -1, abs(slope) = +2.3 }

Continuous spaces everywhere

On June 26th, Facebook stock reboundedafter a bruising swoon

A deep red sunset

Keep a little to the left of the post

Beat the eggs gently, until they form stiff peaks

Three tasks

Color Time series Navigation

Predicting colors

bluepastel bluedark pastel blue

H

V S

Regression model

bluepastel bluedark pastel blue

H

V S

Regression model

dark pastel blue

0

0

-40

0

-37

-25

216

80

90

H

S

V

216

43

75

+ + =

Regression model

dark pastel blue

dark pastel blue{dark, pastel, blue}

Regression model

H 216S 43V 75

Experiment setup

Sample predictions

electric green pale blue dark brown

pale green indigo

Prediction error

Hue Sat Val Mean0

0.1

0.2

0.3

0.4

BaselineLast wordFull model

A guessing game

pale blue

A guessing game

Prediction accuracy0

0.2

0.4

0.6

0.8

1

0.50

0.78 0.810.86

BaselineLast wordFull modelHuman

Predicting time series

stocks rebounded after a bruising swoon

1 2

2 1

Predicting time series

stocks rebounded after a bruising swoon

Predicting time series

stocks rebounded after a bruising swoon{stocks, rebounded} {after, a, bruising, swoon}

2 1

sgn(slope): -1abs(slope): 3.1curvature: 0.5

sgn(slope): 1abs(slope): 2.7curvature: -0.1

Learning & inference

• Need parameters for linear prediction model & log-linear alignment model: easy with EM

• For small number of path segments, possible to sum exactly over latent alignments

• Otherwise, approximation of your choice

Experiment setup

Market rallies to new highs

Sample predictions

Reference

Predicted

U.S. stocks end lower as economic worries persist[U.S. stocks end lower]2 [as economic worries persist]1

A guessing game

Prediction accuracy0

0.2

0.4

0.6

0.8

0.5

0.59 0.61

0.72

BaselineNo alignmentFull modelHuman

Peeking at parameters

sgn(slope) abs(slope)

rise

swoon

sharply

0.27

-0.57

-0.22

-0.78

0

0.28

Following instructions…

and then we're going to turn north again

and immediat-- well a distance below that turning point there's a fenced meadow

but you should be avoiding that by quite a distance

okay so we've turned and we're going up north again

continue straight up north

and then we're going to turn to the west on a curvature right sort of

…

Navigation results

Precision Recall F-measure0

0.1

0.2

0.3

0.4

0.5

0.6

BranavanVogelThis work

Conclusions

• New model for predicting grounded representations of meaning in arbitrary real-valued spaces

• Beats strong baselines on a diverse range of tasks

• Code and data available online athttp://cs.berkeley.edu/~jda