representations of language in a model of visually ... · pdf filea major commercial success...
TRANSCRIPT
![Page 1: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/1.jpg)
Representations of language in a model of visually
grounded speech signal
Grzegorz Chrupała Lieke Gelderloos Afra Alishahi
![Page 2: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/2.jpg)
Automatic Speech Recognition
A major commercial success story in Language Technology
![Page 3: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/3.jpg)
Very heavy-handed supervision
I can see you
![Page 4: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/4.jpg)
Grounded speech perception
![Page 5: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/5.jpg)
Data
Flickr8K Audio (Harwath & Glass 2015) 8K images, five audio captions each
MS COCO Synthetic Spoken Captions
300K images, five synthetically spoken captions each
![Page 6: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/6.jpg)
Project speech and image to joint space
a bird walks on a beam
bears play in water
![Page 7: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/7.jpg)
Image model
BOAT
BIRD
BOAR
Pre
-cla ssifcat io
n lay e
r
![Page 8: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/8.jpg)
Speech model
Input: MFCC Subsampling CNN Recurrent
Highway Network (Zilly et al 2016)
Attention
![Page 9: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/9.jpg)
Model settings
![Page 10: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/10.jpg)
Image retrieval
Flickr8K
MSCOCO
Newer CNN architecture: Harwath et al 2016 (NIPS), Harwath and Glass 2017 (ACL)
![Page 11: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/11.jpg)
Levels of representation
What aspects of sentences are encoded?
Which layers encode form, which encode meaning?
Auxiliary tasks (Adi et al 2017)
![Page 12: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/12.jpg)
Form-related aspects
Use activation vectors to decode Utterance length in words Presence of specific words
![Page 13: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/13.jpg)
Number of words
Input Activations for
utterance Model
Linear regression
![Page 14: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/14.jpg)
Word presence
Input Activations for
utterance MFCC for word
Model MLP
![Page 15: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/15.jpg)
Semantic aspects
![Page 16: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/16.jpg)
Representational Similarity
Correlations between sets of pairwise similarities according to Activations
AND Edit ops on written
sentences Human judgments
(SICK dataset)
![Page 17: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/17.jpg)
Homonym disambiguation
![Page 18: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/18.jpg)
Follow-up work
Afra Alishahi, Marie Barking and Grzegorz Chrupała. Encoding of phonology in a recurrent neural model of grounded speech
Friday, session #4 at CoNLL
![Page 19: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/19.jpg)
Conclusion
Encodings of form and meaning emerge and evolve in hidden layers of stacked RHN listening to grounded speech
Code: github.com/gchrupala/visually-grounded-speechData: doi.org/10.5281/zenodo.400926
![Page 20: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/20.jpg)
![Page 21: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/21.jpg)
![Page 22: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/22.jpg)
Error analysis
Text usually better Speech better:
Long descriptions
Misspellings
Text Speech
a yellow and white birtd is in flight
![Page 23: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/23.jpg)
Length
![Page 24: Representations of language in a model of visually ... · PDF fileA major commercial success story in Language Technology. Very heavy-handed supervision I can see you. ... Auxiliary](https://reader031.vdocuments.site/reader031/viewer/2022030401/5a78984b7f8b9a7b698d4cab/html5/thumbnails/24.jpg)
Text model
Convolution word embedding→ No attention