rebecca knowles and philipp koehnrknowles/presentations/knowleskoehnamta... · 2016. 11. 24. ·...

Neural Interactive Translation Prediction

Rebecca Knowles and Philipp Koehn

–––

Interactive Translation Prediction

••

•

http://www.casmacat.eu/

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month

The company said it will start interviewing applicants this month ,

The company said it will start interviewing talks this

The company said it will start interviewing applicants this month ,

The company said it will start

The company said it will start interviewing applicants this month ,

The company said it will start

The company said it will start interviewing applicants this month ,

The company said it will start job applicants this month ,

The company said it will start interviewing applicants this month ,

The company said it will start job talks this month ,

The company said it will start interviewing applicants this month ,

with staff levels rising from October through December .

Interactive Translation Prediction using Search Graph

planned monthsfor

has for months

monthssincehas

he

it

planned monthsfor

has for months

monthssincehas

he

it

planned monthsfor

has for months

monthssincehas

he

it

planned monthsfor

has for months

monthssincehas

he

it

Related Work

•–

•–

…•

–•

–

Ein Auftakt in dersouveräner Saison

Neural Machine Translation

Ein Auftakt in dersouveräner Saison

Ein in dersouveräner Auftakt Saison

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS>

Ein in dersouveräner Auftakt Saison

.

.

.

Distribution Over Vocabulary

0.6 a

0.3 an

0.05 the

0.01 that

... ...

1e-9 halloween

1e-10 trilobite

<BOS>

Ein in dersouveräner Auftakt Saison

.

.

.

Attention Mechanism

<BOS>

Ein in dersouveräner Auftakt Saison

.

.

.

Condition on Previous Word

<BOS>

Ein in dersouveräner Auftakt Saison

.

.

.

Select Most Probable Word

0.6 a

0.3 an

0.05 the

0.01 that

... ...

1e-9 halloween

1e-10 trilobite

<BOS> a

Ein in dersouveräner Auftakt Saison

.

.

.

New Distribution, Conditioned on Previous Selection

<BOS> a

.

.

.

Ein in dersouveräner Auftakt Saison

.

.

.

New Distribution, Conditioned on Previous Selection

Highest probability word:

“sovereign”

<BOS> a

.

.

.

Ein in dersouveräner Auftakt Saison

.

.

.

New Distribution, Conditioned on Previous Selection

Highest probability word:

“sovereign”

<BOS> a

.

.

.

sovereign

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

pre- lude

.

.

.

in

.

.

.

the

.

.

.

season

A sovereign prelude in the season

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

pre- lude

.

.

.

in

.

.

.

the

.

.

.

season

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

pre- lude

.

.

.

in

.

.

.

the

.

.

.

season

a

✔

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

pre- lude

.

.

.

in

.

.

.

the

.

.

.

season

a

✔

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

pre- lude

.

.

.

in

.

.

.

the

.

.

.

season

a

✔

con-fident

✘

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

start

.

.

.

in

.

.

.

the

.

.

.

season

a

✔

con-fident

✘

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

start

.

.

.

to

.

.

.

the

.

.

.

season

a

✔

con-fident

✘

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

start

.

.

.

to

.

.

.

the

.

.

.

season

a

✔

con-fident

✘

start

✔

Ein in dersouveräner Auftakt Saison

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

start

.

.

.

to

.

.

.

the

.

.

.

season

a

✔

con-fident

✘

start

✔to the season

✔ ✔ ✔

.

.

.

<BOS> a

.

.

.

sovereign

.

.

.

start

.

.

.

to

.

.

.

the

.

.

.

season

a

✔

con-fident

✘

start

✔to the season

✔ ✔ ✔

Word Prediction Accuracy

In this example, what fraction of suggestions were accepted?

5/6

Interactive Translation Prediction with Neural MT

•–

Interactive Translation Prediction with Neural MT

•–

•

Evaluations

••••

Evaluation Data

•–––

MT Systems

•–

•–

System Quality

System Configuration BLEU

Neural no beam searchbeam size 12+ensemble+r2l reranking

34.536.237.538.6

Phrase-based 34.5

System Quality

System Configuration BLEU

Neural no beam searchbeam size 12+ensemble+r2l reranking

34.536.237.538.6

Phrase-based 34.5

Word Prediction Accuracy

Correct Prediction

✔✔✔✔✔✔✗✗✔✔✗✗✔✗✗✔✔✗✔✔

thecompanysaiditwillstartinterviewingapplicantsthismonth,withstafflevelsrisingfromOctoberthroughDecember.

thecompanysaiditwillstartjobtalksthismonthandandstaffnumbersincreasingfromOctobertoDecember.

Example with about-average word prediction accuracy.

(13/20)

Word Prediction Accuracy

System Configuration Word Prediction Accuracy

Neural no beam searchbeam size 12

61.6%63.6%

Phrase-based - 43.3%

DistributionsCorrect Prediction

✔✔✔✔✔✔✗✗✔✔✗✗✔✗✗✔✔✗✔✔

thecompanysaiditwillstartinterviewingapplicantsthismonth,withstafflevelsrisingfromOctoberthroughDecember.

thecompanysaiditwillstartjobtalksthismonthandandstaffnumbersincreasingfromOctobertoDecember.

DistributionsCorrect Prediction

✔✔✔✔✔✔✗✗✔✔✗✗✔✗✗✔✔✗✔✔

thecompanysaiditwillstartinterviewingapplicantsthismonth,withstafflevelsrisingfromOctoberthroughDecember.

thecompanysaiditwillstartjobtalksthismonthandandstaffnumbersincreasingfromOctobertoDecember.

DistributionsCorrect Prediction

✔✔✔✔✔✔✗✗✔✔✗✗✔✗✗✔✔✗✔✔

thecompanysaiditwillstartinterviewingapplicantsthismonth,withstafflevelsrisingfromOctoberthroughDecember.

thecompanysaiditwillstartjobtalksthismonthandandstaffnumbersincreasingfromOctobertoDecember.

DistributionsCorrect Prediction

✔✔✔✔✔✔✗✗✔✔✗✗✔✗✗✔✔✗✔✔

thecompanysaiditwillstartinterviewingapplicantsthismonth,withstafflevelsrisingfromOctoberthroughDecember.

thecompanysaiditwillstartjobtalksthismonthandandstaffnumbersincreasingfromOctobertoDecember.

Recovery from First Failure

Recovery from First FailureCorrect Prediction

✔✔✔✗✗✔

andsomeareholdingoutfor...

andsomearewaitingontofor...

Correct Prediction

✔✗✔✔✔

atthistimelastcycle...

atthattimelastcycle...

Sequences of MispredictionsCorrect Prediction

✔✔✔✔✔✔✗✗✔✔✗✗✔✗✗✔✔✗✔✔

thecompanysaiditwillstartinterviewingapplicantsthismonth,withstafflevelsrisingfromOctoberthroughDecember.

thecompanysaiditwillstartjobtalksthismonthandandstaffnumbersincreasingfromOctobertoDecember.

Deployment and Speed

CASMACAT: http://www.casmacat.eu/

Lilt: https://lilt.com/

Speed

Patching Partial Predictions

Patching Partial Predictions

Patching Partial Predictions

Patching Partial Predictions

Patching Partial Predictions

Patching Partial Predictions

→

Conclusions

•

–––

Thank You

Detailed Recovery

Length of Error Sequences