grounding language with points and paths in continuous spaces
DESCRIPTION
Grounding Language with Points and Paths in Continuous Spaces. B erkeley. N L P. Jacob Andreas and Dan Klein UC Berkeley. Formal grounding. On June 26 th , Facebook stock cost $65 per share. quote { date: 2014-06-26, stock: FB, price: $65 }. Perceptual grounding. - PowerPoint PPT PresentationTRANSCRIPT
Berkeley
N L P
Grounding Language with Points and Paths in Continuous Spaces
Jacob Andreas and Dan KleinUC Berkeley
Berkeley
N L P
Formal grounding
On June 26th, Facebook stock cost $65 per share
quote {date: 2014-06-26,stock: FB,price: $65
}
Perceptual grounding
On June 26th, Facebook stock reboundedafter a bruising swoon
?
Perceptual grounding
On June 26th, Facebook stock reboundedafter a bruising swoon
Perceptual grounding
On June 26th, Facebook stock reboundedafter a bruising swoon
A after B A, B A before B B, A rebounded { sgn(slope) = +1 }
bruising { sgn(slope) = -1, abs(slope) = +2.3 }
Continuous spaces everywhere
On June 26th, Facebook stock reboundedafter a bruising swoon
A deep red sunset
Keep a little to the left of the post
Beat the eggs gently, until they form stiff peaks
Three tasks
Color Time series Navigation
Predicting colors
bluepastel bluedark pastel blue
H
V S
Regression model
bluepastel bluedark pastel blue
H
V S
Regression model
dark pastel blue
0
0
-40
0
-37
-25
216
80
90
H
S
V
216
43
75
+ + =
Regression model
dark pastel blue
dark pastel blue{dark, pastel, blue}
Regression model
H 216S 43V 75
Experiment setup
Sample predictions
electric green pale blue dark brown
pale green indigo
Prediction error
Hue Sat Val Mean0
0.1
0.2
0.3
0.4
BaselineLast wordFull model
A guessing game
pale blue
A guessing game
Prediction accuracy0
0.2
0.4
0.6
0.8
1
0.50
0.78 0.810.86
BaselineLast wordFull modelHuman
Predicting time series
stocks rebounded after a bruising swoon
1 2
2 1
Predicting time series
stocks rebounded after a bruising swoon
Predicting time series
stocks rebounded after a bruising swoon{stocks, rebounded} {after, a, bruising, swoon}
2 1
sgn(slope): -1abs(slope): 3.1curvature: 0.5
sgn(slope): 1abs(slope): 2.7curvature: -0.1
Learning & inference
• Need parameters for linear prediction model & log-linear alignment model: easy with EM
• For small number of path segments, possible to sum exactly over latent alignments
• Otherwise, approximation of your choice
Experiment setup
Market rallies to new highs
Sample predictions
Reference
Predicted
U.S. stocks end lower as economic worries persist[U.S. stocks end lower]2 [as economic worries persist]1
A guessing game
Prediction accuracy0
0.2
0.4
0.6
0.8
0.5
0.59 0.61
0.72
BaselineNo alignmentFull modelHuman
Peeking at parameters
sgn(slope) abs(slope)
rise
swoon
sharply
0.27
-0.57
-0.22
-0.78
0
0.28
Following instructions…
and then we're going to turn north again
and immediat-- well a distance below that turning point there's a fenced meadow
but you should be avoiding that by quite a distance
okay so we've turned and we're going up north again
continue straight up north
and then we're going to turn to the west on a curvature right sort of
…
Navigation results
Precision Recall F-measure0
0.1
0.2
0.3
0.4
0.5
0.6
BranavanVogelThis work
Conclusions
• New model for predicting grounded representations of meaning in arbitrary real-valued spaces
• Beats strong baselines on a diverse range of tasks
• Code and data available online athttp://cs.berkeley.edu/~jda