Upload File

supervised learning of universal sentence representations from natural language inference data

  • Home
  • Engineering
  • Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
14
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes EMNLP 2017, pages 681-691. 論文 コード（エンコーダ） コード（評価） 読み手: 岡崎 直観 （東京工業 大学情報理工学院） （@ chokkanorg） ACL2017 読み会@すずかけ台 ※但し書きがある場合を除き，スライド中の図・表はConneau+ (2017) の引用

Upload: naoaki-okazaki

Post on 23-Jan-2018

433 views

Category:

Engineering


5 download

Report

TRANSCRIPT

Page 1: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Supervised Learning of Universal Sentence Representations from

Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk,Loïc Barrault, and Antoine Bordes

EMNLP 2017, pages 681-691.論文 コード（エンコーダ） コード（評価）

読み手: 岡崎直観 （東京工業大学情報理工学院） （@chokkanorg）

ACL2017読み会@すずかけ台

※但し書きがある場合を除き，スライド中の図・表はConneau+ (2017) の引用

http://aclweb.org/anthology/D17-1071
https://github.com/facebookresearch/InferSent
https://github.com/facebookresearch/SentEval
http://chokkan.org/
http://www.c.titech.ac.jp/
https://twitter.com/chokkanorg
http://www.lr.pi.titech.ac.jp/%7Ehaseshun/acl2017suzukake/
Page 2: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

汎用的な文エンコーダは作れる！

• 構成方法• 双方向LSTMにmax-poolingを組み合わせたDNN• Stanford Natural Language Inference (SNLI) で訓練

• 検証方法• SNLIデータで様々なエンコーダの性能を比較

• SkipThoughtやFastSentなどの既存手法を含め，訓練済みのエンコーダを12個のタスクに適用して性能を比較

• 検証結果• 提案手法の文エンコーダは，様々なタスクへ転用することが可能で，かつ既存手法よりも高性能

• 提案手法の訓練は，既存手法の訓練よりも高速

• 文の特徴抽出器として利用可能（モデルを公開）

2Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 3: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

本研究で用いたタスク:Stanford Natural Language Inference (SNLI) (Bowman+ 15)

• Flickr30kのキャプションを基に，含意（entailment），矛盾（contradiction），中立（neutral）する文をクラウドソーシングで収集したもの（57万文対）

https://nlp.stanford.edu/projects/snli/

3Conneau+ (2017) Supervised Learning of Universal Sentence Representations

https://nlp.stanford.edu/projects/snli/
Page 4: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

本研究で訓練するDNNの構成:２文のエンコーダに基づく３値分類

A soccer game with multiple males playing. Some men are playing a sport.

文のエンコーダ 文のエンコーダ

(Premise) (Hypothesis)

𝑢𝑢 𝑣𝑣

× ー

512次元の隠れ層とsoftmax層（3値分類）

Premiseのベクトル

Hypothesisのベクトル

含意，矛盾，中立のスコア（教師信号）

2文のエンコード結果を統合したベクトル

×は要素ごとの積－は要素ごとの差の絶対値

4Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 5: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

文のエンコーダの構成

• 以下の7種類の手法を比較・検討• Long Short-Term Memory (LSTM)• Gated Recurrent Unit (GRU)• 双方向GRU （両方向の最終内部ベクトルを結合）

• 双方向LSTM• 各単語に対応する内部ベクトルの平均をとる (mean pooling)• 各単語に対応する内部ベクトルの最大値を取る (max pooling)

• Self-attentive network• Hierarchical Convolutional Network

• 単語埋め込みはGloVeで固定• 公開されている単語ベクトル（300次元）を利用

5Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 6: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

双方向GRU

Skip-thoughtで用いられているエンコーダと

同じアーキテクチャを採用

The movie was great𝑥𝑥𝑡𝑡

ℎ𝑡𝑡

𝑢𝑢または 𝑣𝑣

ℎ𝑡𝑡

𝑤𝑤𝑡𝑡

ℎ𝑡𝑡 = GRU (𝑤𝑤1, … ,𝑤𝑤𝑇𝑇)

ℎ𝑡𝑡 = GRU (𝑤𝑤1, … ,𝑤𝑤𝑇𝑇)

𝑢𝑢 = ℎ𝑇𝑇 , ℎ1

6Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 7: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

双方向LSTMとプーリング

The movie was great𝑥𝑥𝑡𝑡

ℎ𝑡𝑡

ℎ𝑡𝑡

𝑤𝑤𝑡𝑡

ℎ𝑡𝑡 = LSTM (𝑤𝑤1, … ,𝑤𝑤𝑇𝑇)

ℎ𝑡𝑡 = LSTM (𝑤𝑤1, … ,𝑤𝑤𝑇𝑇)

ℎ𝑡𝑡 , ℎ𝑡𝑡

各次元について，全時刻における最大値もしくは平均値をとる

7Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 8: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Self-attentive network (Liu+ 16; Lin+ 17)

The movie was great𝑥𝑥𝑡𝑡

ℎ𝑡𝑡

ℎ𝑡𝑡

𝑤𝑤𝑡𝑡

ℎ𝑡𝑡 = LSTM (𝑤𝑤1, … ,𝑤𝑤𝑇𝑇)

ℎ𝑡𝑡 = LSTM (𝑤𝑤1, … ,𝑤𝑤𝑇𝑇)

ℎ𝑡𝑡 = ℎ𝑡𝑡, ℎ𝑡𝑡

𝑞𝑞𝑖𝑖

𝑎𝑎𝑖𝑖(𝑡𝑡) =exp( �ℎ𝑡𝑡 ⋅ 𝑞𝑞𝑖𝑖)

∑𝑡𝑡𝑡 exp(ℎ𝑡𝑡𝑡 ⋅ 𝑞𝑞𝑖𝑖)�ℎ𝑡𝑡 = tanh(𝑊𝑊ℎ𝑡𝑡 + 𝑏𝑏𝑖𝑖)

𝑢𝑢𝑖𝑖 = �𝑡𝑡𝑎𝑎𝑖𝑖(𝑡𝑡)ℎ𝑡𝑡

※実際は𝑞𝑞1, 𝑞𝑞2,𝑞𝑞3,𝑞𝑞4の4種類のベクトル（観点）を学習し，4種類の𝑢𝑢1,𝑢𝑢2,𝑢𝑢3,𝑢𝑢4を計算して，連結する

（アテンション計算用）（学習で求める）

8Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 9: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Hierarchical convolutional network (Zhao+ 15)

The movie was the best of all𝑥𝑥𝑡𝑡

𝑐𝑐𝑡𝑡1

𝑐𝑐𝑡𝑡2

𝑐𝑐𝑡𝑡3

𝑐𝑐𝑡𝑡4※実際には4層まで畳み込みを適用するが，この例では単語数が足りないため4層まで到達していない

最大値プーリング

4層分のプーリング結果の連結を文ベクトルとする

9Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 10: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

評価に用いたタスク

• 分類問題• 評判分析（MR, SST）• 質問タイプ分類（TREC）• 商品レビュー分析（CR）• 主観・客観分類（SUBJ）• 意見極性分類（MPQA）

• 意味的関係• 含意関係認識（SICK-E）• 意味的関連度（SICK-R）• SemEvalの意味的類似度（STS14）

• 言い換えの認識• 言い換えの認識（Microsoft Research Paraphrase Corpus）

• キャプション／画像検索• キャプションからの画像検索• 画像からのキャプション検索

10Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 11: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

エンコーダ構成の比較転用先12タスクでの性能の平均

11

※ NLIで高い性能のものが転用先でも高い性能を示すとは限らない

Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 12: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

タスク横断の比較

12Conneau+ (2017) Supervised Learning of Universal Sentence Representations

6,400万文で訓練したSkipThoughtに対し，57万文対で

訓練した提案手法が上回る（SUBJとTRECを除く）

STS14の結果より，SkipThoughtよりも

提案手法の文ベクトルの方がコサイン類似度を正確に計算できている（提案手法の中に要素積や要素差の情報が取り込まれているため？）

(AllNLI = SNLI + MultiGenre NLI)

機械翻訳や辞書定義文などで訓練するよりも高性能

Page 13: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

画像・キャプション検索の性能

• 上段は画像と言語の対応を直接的に学習するもの

• 下段は訓練済みの画像と言語の特徴抽出器を用い，それらを同じ空間にマップしてランキング学習を行った

• 直接訓練する手法に迫る性能を示す

13Conneau+ (2017) Supervised Learning of Universal Sentence Representations

Page 14: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

議論

• SNLIデータは文エンコーダの訓練に適している• NLIは文の深い意味や関係の理解が必須だから？

• 教師なし（単なる生コーパス）の大量データよりも，SNLIの教師ありデータの方が，性能・学習時間の観点から優れている

• 今後の展開• 実際にはどのようなタスクに転用可能なのか？ 機械翻訳の性能は向上の余地があるのか？

• 日本語のNLIのデータを作る？

14Conneau+ (2017) Supervised Learning of Universal Sentence Representations


Part 3: Latent representations and unsupervised learning · 2012-07-11 · Part 3: Latent representations and unsupervised learning Dale Schuurmans University of Alberta. Supervised

Word representations: A simple and general method for semi-supervised learning

Composing graphical models with neural networks for structured representations and fast inference

b r a c e - Carnegie Mellon School of Computer Scienceepxing/Class/10701/slides/lecture18-Inference... · Inference and Learning We now have compact representations of probability

Abstract - finale.seas.harvard.edu · these pathologies affects key downstream tasks, including learning compressed and disentangled representations, adversarial robustness and semi-supervised

Factorized Graph Representations for semi-supervised

Semi-supervised inference: General theory and estimation

Supervised Inference of Gene Regulatory Networks by Linear

GraphTER: Unsupervised Learning of Graph Transformation ... · supervised learning. Auto-encoders aim to train an encoder to learn feature representations by reconstructing input

Deep Learning Tutorial - University of Edinburgh · 2017-07-19 · Conventional Pattern recognition Learning Feature Representations Supervised Learning with ... Backpropagation in

Improved Bayesian Logistic Supervised Topic Models with ... · mulation of logistic supervised topic models as an instance of regularized Bayesian inference pro-vides a direct comparison

Learning representations for efficient inferenceapproximateinference.org/2016/schedule/Johnson2016.pdf · Learning representations for efficient inference Matt Johnson, David Duvenaud,

Concurrent Inference of Topic Models and Distributed Vector … · 2015. 7. 26. · Concurrent Inference of Topic Models and Distributed Vector Representations Debakar Shamanta 1,

SUPERVISED EVALUATION OF REPRESENTATIONS A …web.stanford.edu/~hastie/THESES/charlesZheng_thesis.… ·  · 2017-06-08and computation made him a valuable resource to consult whenever

Self-Supervised Hyperboloid Representations from Logical

Supervised Learning of Universal Sentence Representations ... · tations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform

Tractable Representations Probabilistic Inference Learning ...guyvdb/slides/TPMTutorialUAI19.pdf · Tractable Probabilistic Models Representations Inference Learning Applications

Learning Action Representations for Self-supervised Visual ...andrea/papers/2019_ICRA__LearningAction... · Learning Action Representations for Self-supervised Visual Exploration

Word representations: A simple and general method for semi-supervised learning Joseph Turian with Lev Ratinov and Yoshua Bengio Goodies:

Using Self-Supervised Learning Can Improve Model ... · Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches

Using Self-Supervised Learning Can Improve Model ......Self-supervised learning holds great promise for improving representations when labeled data are scarce. In semi-supervised learning,

Inference in Supervised Spectral Classifiers for On-Board Hyperspectral Imaging… · 2020-02-06 · remote sensing Article Inference in Supervised Spectral Classifiers for On-Board

Factorized Graph Representations for Semi-Supervised ......Factorized Graph Representations for Semi-Supervised Learning from Sparse Data Krishna Kumar P. IIT Madras Paul Langton Northeastern

Quantum Statistical Inference - arXiv · in classical statistical inference and supervised machine learning. A crucial component of the quantum GP algorithm is solving linear systems

Language Models as Representations for Weakly-Supervised NLP Tasks (CoNLL2011)

Word representations: a simple and general method for semi-supervised learning · 2016-01-07 · Word representations: a simple and general method for semi-supervised learning. In

Unsupervised Learning of Feature Hierarchies · 2015. 10. 28. · representations, and invariant feature hierarchies from unlabeled data. These methods go beyond traditional supervised

Learning Self-Supervised Representations of Code Functionality

Universal Semi-Supervised Semantic SegmentationUniversal Semi-Supervised Semantic Segmentation A novel approach towards learning efficient and transferable representations for dense

GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING … · GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING WITH OBJECT-CENTRIC LATENT REPRESENTATIONS Martin Engelcke z, Adam R. Kosiorekzr,

Self-Supervised Learning of Face Representations for Video ...Self-Supervised Learning of Face Representations for Video Face Clustering Vivek Sharma 1, Makarand Tapaswi2, M. Saquib

Supervised inference of biological networks and ...members.cbio.mines-paristech.fr/~jvert/talks/070214singapore/singa… · Supervised inference of biological networks and Classification

“Unsupervised Learning of Visual Representations by ...yjlee/teaching/ecs289g-fall2016/chenshan.pdfRepresentations by Solving ... - Problem: Self-supervised learning CNN - Solution:

Do Supervised Distributional Methods Really Learn Lexical Inference Relations? Omer Levy Ido Dagan Bar-Ilan University Israel Steffen Remus Chris Biemann

Learning representations for counterfactual inference...Learning representations for counterfactual inference Fredrik D. Johansson*2, Uri Shalit*1, David Sontag1 *Equal Contribution