61. sequential data - mlvu · models for sequential data machine learning 2020 mlvu.github.io the...

Models for Sequential Data

Machine Learning 2020 mlvu.github.io

the plan

part 1: Sequences

Markov models

Embedding models Word2Vec

part 2: Recurrent Neural Nets

LSTMs

2

numeric 1-dimensional

3

Before we start looking at different models to learn from sequential data, it pays to think a little bit about the different types of sequential datasets we might encounter.

As with the traditional setting (a table of independently sampled instances) we can divide our features into numeric and discrete.

A single 1D sequence might look like this. We could this of a stock price over time, trafBic to a webserver, or atmospheric pressure over Amsterdam. In this case, the data shows the number of sunspots observed over time.

4

numeric n-dimensional Sequential numeric data can also be multidimensional. In this case, we see the closing index of the AEX and the FTSE100 over time. This data is a sequence of 2D vectors.

symbolic (categorical) 1-dimensional

5

the, cat, sat, on, the, mat

t, h, e, _, c, a, t, _, s, a, t, _, o, n, _, t, h, e, _, m, a, t

If the elements of our data are discrete (analogous to a categorical feature), it becomes a sequence of symbols. Language is a prime example. In fact, we can model language as a sequence in two different ways: as a sequence of words, or as a sequence of characters.

symbolic n-dimensional

6

the/ART cat/NOUN sat/VERB on/PREP the/ART mat/NOUN

We can also encounter sequences with multiple categorical features per timestamp, like music, or tagged language. These can often be difBicult to represent.

single sequence vs set-of-sequences

7

Dear recipient, Avangar Technologies announces the beginning of a new unprecendented global employment campaign … reviser yeller winers butchery twenties Due to company's exploding growth Avangar is expanding business to the European region. During last employment campaign over 1500 people worldwide took part in Avangar's business and more than half of them are currently employed by the company. And now we are offering you one more opportunity to earn extra money working with Avangar Technologies.

spam

BENCHMARK SUPPLY 7540 BRIDGEGATE COURT ATLANTA GA 30350 ***LASER PRINTER TONER CARTRIDGES*** ***FAX AND COPIER TONER*** CHECK OUT OUR NEW CARTRIDGE PRICES :

spam

On 08/08/02 15:26 -0700, Chip Paswater wrote: ) > Well, a little more than one bit -- you have to transmit the signature, plus now the entire … ID. ) > Plus the rights to all intellectual property contained in any ) > email message you submit (I guess technically there are no bits ) > in the rights assignment) ) ) Perhaps a feature can be added to razor-report, so that it checks whether a

ham

If you own a travel related website, why not submit your site to our directory. Just select the appropriate category and subcategory and enter your title … description.

Click here to start: http://www.holprop-travel-directory.com

This footnote conBirms that this email message has been swept by Anti Virus

spam

| hmmm, I assume you're going to report this to the nmh folks? Yes, I will, sometime, after I look at the nmh sources and see what they have managed to break, …why.

But we really want exmh to operate with all the versions of nmh that exist, don't we? The patch to have exmh do the right thing, whether this bug exists, or not, is trivial, so I'd suggest including it.

ham

Hello I am emailing you as we found your holiday property at [Source] and would love you to list it on our website for free:

http://www.holidayrentals.org

This web site now has over 1000 holiday rentals listed in less than 2 months, so please register and then add as many properties as you wish. All for free forever

spam

Hello again self-catering.co.uk was recently launched and if you have not yet added your property for free then please do as we have had many properties added in the past few days. spam

Then there is the question of what we’re trying to predict. Do we have one sequence per instance, and are trying to, for instance, classify the sequences? That’s what we see here: each email is a sequence (of words or or characters).

The instances themselves do not have a very sequential ordering (emails do have a timestamp, but this ordering is usually ignored).

single sequence

8

?

An entirely different setting is one where the dataset as a whole is a sequence, and the instances are ordered.

In that case, we often want to predict the future values of the sequence based on what we’ve seen in the past. To keep things simple, let's stick with 1D numeric data, like our sunspots example.

single sequence: feature extraction

9

t-3 t-2 t-1 t

5 2 3 3

3 5 2 3

3 3 5 2

5 3 3 5

4 5 3 3

2 4 5 3

3 2 4 5

3 3 2 4

1 1 3 2

2 3 3 3

1 3 5 3 4 2 3 4 5 5 4 3 2 1 3 2 2

One simple way to achieve this, is to translate it to a classic regression problem by representing each point by the m values before it. This gives us a table with a target label (the value at time t) and m features (the m preceding values). If we successfully train a regression model on this task, we can then use it to predict future values of the sequence.

Many other features are possible: mean over the whole history. Mean over the last 10 points, variance one the last 10 points, etc.

But, there’s a problem. If we shufBle this data, and pick a test and training set, the classiBier is trained on data that happens in the future to data it will be tested on!

Think about the REAL-WORLD use case.

10

Whenever you have questions about how to approach something like this, it’s best to think about the real world setting where you might apply your trained model.

In this case, we are going to apply the prediction model on the three most recent values, to predict the next value.The test data should simulate that situation. That means that the training data should not contain instances that occur later than the test data.

walk-forward validation

11

training data

time

train v 0.4

0.1train v

0.3train v

average0.4

0.3

train v

… …

test

To check your algorithm, to do cross validation.

testing

12

training data

time

test

train

train

train

train

…

Note, only if the target labels have a meaningful ordering in time. In spam classiBication, for instance, this is not necessary (emails do have a timestamp, but seeing an email from the future does not usually give you an unrealistic advantage).

summary: sequential data

Sequences: consisting of numbers, vectors or symbols

Dataset: consisting of a sequence per instance, or a sequence of instances. For a sequence of instances, carful with your test and validation

Both can be converted to fit classic machine learning through feature extraction.

Rest of the lecture: 1D symbolic sequences. Extension to nD and/or numeric is often trivial

13

But, often it’s better to take a model that can consume sequences natively.

modeling sequences: probability

14

p(“congratulations you have won a prize”)= p(W1=congratulations,W2=you,W3=have,

W4=won,W5=a,W6=prize)<latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit><latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit><latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit><latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit>

p(W1,W2,W3,W4,W5,W6)<latexit sha1_base64="81laJBbsbhBvwmoiZCx1Q4GVAFM=">AAAHgnicfVVdb9MwFA1j0FEYDHjkgYgKCaZqSlr2gcSksQ9AArYxreuktaoc97aN6iSW7XQNVh75NbzCj+HfYPdjSuqAH5yre8658T22bI8SnwvH+XNr6fbynbullXvl+w9WHz5ae/zkgkcxw9DAEYnYpYc4ED+EhvAFgUvKAAUegaY3PNB4cwSM+1F4LhIK7QD1Q7/nYyRUqrP2nL5qdtyq3ezU9FTX0xs9bepp63VnreJsOJNhm4E7CyrWbJx2Ht/93OpGOA4gFJggzq9ch4q2REz4mEBabsUcKMJD1IerWPR22tIPaSwgxKn9UmG9mNgisvVi7a7PAAuSqABh5qsKNh4ghrBQLZXzpTiEKABe7Y58yqchH/WngUDKj7YcT/xKV3NK2WeIDnw8zi1NooAHSAyMJE8CL5+EmAAbBfmkXqZa5AJzDAz7XJtwqpw5oXoP+Hl0OsMHCR1AyFMZM5JmhQoAxqCnhJOQg4ipnHSjNn7IdwWLoarDSW73ELHhGXSrqk4ukV9Oj0RI5FOeakO5E8I1joIAhV3ZoqlsCRgL2apupBPvsuhZKmVLG+V59pmGc+hxBj1O0zx4lAGPFJhHGzdoz24sSi8y4IXx12YGbS5KvTiDxgY6yqAjo7J3nYGvDXicQccGmmTQxEC/Z9Dvps9IHYurWltO92KyqfKE+CP4yADCVFZq6WIvTO33lZuX6DMgK246sbsLPXVrTIEg0XT56fzrl1Qe7NQ2na10keGRGOYUp761eeAYlP50NTOOs7NT2zc4EUNh/6bQ4dHWe9csRGNGyQ1pe7v+4a1ZKQFCouubSgf7h7X64jli2DBh1qtdcW3DtH4RfdZVocArEkydKuQPTf5HhpJ/sKOi6nMDCxW0SDF3s1CRFCnm1s4VC01QfVqHWP1O3+uITCmHoG58Bl/VKT5RtxQSEVtXR5f1A1/Zp76tqo7+R0TjOVFF5bJ6ftzFx8YMLmobrrPhfntT2dufPUQr1jPrhfXKcq1ta8/6ZJ1aDQtbP6yf1i/rd2m5tF5yS/UpdenWTPPUyo3Su79FS7O/</latexit><latexit sha1_base64="81laJBbsbhBvwmoiZCx1Q4GVAFM=">AAAHgnicfVVdb9MwFA1j0FEYDHjkgYgKCaZqSlr2gcSksQ9AArYxreuktaoc97aN6iSW7XQNVh75NbzCj+HfYPdjSuqAH5yre8658T22bI8SnwvH+XNr6fbynbullXvl+w9WHz5ae/zkgkcxw9DAEYnYpYc4ED+EhvAFgUvKAAUegaY3PNB4cwSM+1F4LhIK7QD1Q7/nYyRUqrP2nL5qdtyq3ezU9FTX0xs9bepp63VnreJsOJNhm4E7CyrWbJx2Ht/93OpGOA4gFJggzq9ch4q2REz4mEBabsUcKMJD1IerWPR22tIPaSwgxKn9UmG9mNgisvVi7a7PAAuSqABh5qsKNh4ghrBQLZXzpTiEKABe7Y58yqchH/WngUDKj7YcT/xKV3NK2WeIDnw8zi1NooAHSAyMJE8CL5+EmAAbBfmkXqZa5AJzDAz7XJtwqpw5oXoP+Hl0OsMHCR1AyFMZM5JmhQoAxqCnhJOQg4ipnHSjNn7IdwWLoarDSW73ELHhGXSrqk4ukV9Oj0RI5FOeakO5E8I1joIAhV3ZoqlsCRgL2apupBPvsuhZKmVLG+V59pmGc+hxBj1O0zx4lAGPFJhHGzdoz24sSi8y4IXx12YGbS5KvTiDxgY6yqAjo7J3nYGvDXicQccGmmTQxEC/Z9Dvps9IHYurWltO92KyqfKE+CP4yADCVFZq6WIvTO33lZuX6DMgK246sbsLPXVrTIEg0XT56fzrl1Qe7NQ2na10keGRGOYUp761eeAYlP50NTOOs7NT2zc4EUNh/6bQ4dHWe9csRGNGyQ1pe7v+4a1ZKQFCouubSgf7h7X64jli2DBh1qtdcW3DtH4RfdZVocArEkydKuQPTf5HhpJ/sKOi6nMDCxW0SDF3s1CRFCnm1s4VC01QfVqHWP1O3+uITCmHoG58Bl/VKT5RtxQSEVtXR5f1A1/Zp76tqo7+R0TjOVFF5bJ6ftzFx8YMLmobrrPhfntT2dufPUQr1jPrhfXKcq1ta8/6ZJ1aDQtbP6yf1i/rd2m5tF5yS/UpdenWTPPUyo3Su79FS7O/</latexit><latexit sha1_base64="81laJBbsbhBvwmoiZCx1Q4GVAFM=">AAAHgnicfVVdb9MwFA1j0FEYDHjkgYgKCaZqSlr2gcSksQ9AArYxreuktaoc97aN6iSW7XQNVh75NbzCj+HfYPdjSuqAH5yre8658T22bI8SnwvH+XNr6fbynbullXvl+w9WHz5ae/zkgkcxw9DAEYnYpYc4ED+EhvAFgUvKAAUegaY3PNB4cwSM+1F4LhIK7QD1Q7/nYyRUqrP2nL5qdtyq3ezU9FTX0xs9bepp63VnreJsOJNhm4E7CyrWbJx2Ht/93OpGOA4gFJggzq9ch4q2REz4mEBabsUcKMJD1IerWPR22tIPaSwgxKn9UmG9mNgisvVi7a7PAAuSqABh5qsKNh4ghrBQLZXzpTiEKABe7Y58yqchH/WngUDKj7YcT/xKV3NK2WeIDnw8zi1NooAHSAyMJE8CL5+EmAAbBfmkXqZa5AJzDAz7XJtwqpw5oXoP+Hl0OsMHCR1AyFMZM5JmhQoAxqCnhJOQg4ipnHSjNn7IdwWLoarDSW73ELHhGXSrqk4ukV9Oj0RI5FOeakO5E8I1joIAhV3ZoqlsCRgL2apupBPvsuhZKmVLG+V59pmGc+hxBj1O0zx4lAGPFJhHGzdoz24sSi8y4IXx12YGbS5KvTiDxgY6yqAjo7J3nYGvDXicQccGmmTQxEC/Z9Dvps9IHYurWltO92KyqfKE+CP4yADCVFZq6WIvTO33lZuX6DMgK246sbsLPXVrTIEg0XT56fzrl1Qe7NQ2na10keGRGOYUp761eeAYlP50NTOOs7NT2zc4EUNh/6bQ4dHWe9csRGNGyQ1pe7v+4a1ZKQFCouubSgf7h7X64jli2DBh1qtdcW3DtH4RfdZVocArEkydKuQPTf5HhpJ/sKOi6nMDCxW0SDF3s1CRFCnm1s4VC01QfVqHWP1O3+uITCmHoG58Bl/VKT5RtxQSEVtXR5f1A1/Zp76tqo7+R0TjOVFF5bJ6ftzFx8YMLmobrrPhfntT2dufPUQr1jPrhfXKcq1ta8/6ZJ1aDQtbP6yf1i/rd2m5tF5yS/UpdenWTPPUyo3Su79FS7O/</latexit><latexit sha1_base64="81laJBbsbhBvwmoiZCx1Q4GVAFM=">AAAHgnicfVVdb9MwFA1j0FEYDHjkgYgKCaZqSlr2gcSksQ9AArYxreuktaoc97aN6iSW7XQNVh75NbzCj+HfYPdjSuqAH5yre8658T22bI8SnwvH+XNr6fbynbullXvl+w9WHz5ae/zkgkcxw9DAEYnYpYc4ED+EhvAFgUvKAAUegaY3PNB4cwSM+1F4LhIK7QD1Q7/nYyRUqrP2nL5qdtyq3ezU9FTX0xs9bepp63VnreJsOJNhm4E7CyrWbJx2Ht/93OpGOA4gFJggzq9ch4q2REz4mEBabsUcKMJD1IerWPR22tIPaSwgxKn9UmG9mNgisvVi7a7PAAuSqABh5qsKNh4ghrBQLZXzpTiEKABe7Y58yqchH/WngUDKj7YcT/xKV3NK2WeIDnw8zi1NooAHSAyMJE8CL5+EmAAbBfmkXqZa5AJzDAz7XJtwqpw5oXoP+Hl0OsMHCR1AyFMZM5JmhQoAxqCnhJOQg4ipnHSjNn7IdwWLoarDSW73ELHhGXSrqk4ukV9Oj0RI5FOeakO5E8I1joIAhV3ZoqlsCRgL2apupBPvsuhZKmVLG+V59pmGc+hxBj1O0zx4lAGPFJhHGzdoz24sSi8y4IXx12YGbS5KvTiDxgY6yqAjo7J3nYGvDXicQccGmmTQxEC/Z9Dvps9IHYurWltO92KyqfKE+CP4yADCVFZq6WIvTO33lZuX6DMgK246sbsLPXVrTIEg0XT56fzrl1Qe7NQ2na10keGRGOYUp761eeAYlP50NTOOs7NT2zc4EUNh/6bQ4dHWe9csRGNGyQ1pe7v+4a1ZKQFCouubSgf7h7X64jli2DBh1qtdcW3DtH4RfdZVocArEkydKuQPTf5HhpJ/sKOi6nMDCxW0SDF3s1CRFCnm1s4VC01QfVqHWP1O3+uITCmHoG58Bl/VKT5RtxQSEVtXR5f1A1/Zp76tqo7+R0TjOVFF5bJ6ftzFx8YMLmobrrPhfntT2dufPUQr1jPrhfXKcq1ta8/6ZJ1aDQtbP6yf1i/rd2m5tF5yS/UpdenWTPPUyo3Su79FS7O/</latexit>

p(“congratulations you have won a prize”)= p(W1=congratulations,W2=you,W3=have,

W4=won,W5=a,W6=prize)<latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit><latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit><latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit><latexit sha1_base64="0ZDmEqPpMWzEAI0xqXQ9UZAXRSA=">AAAIQnicfVVLj9s2EJbTNE7cptm0x16IGM2jMAzJ3ldRLJDsownQJrtdrNcBVoZD0WNbMCURFGVbS/Df9U/01HtvRa89lJLshV6tLh7M933j4TcD0mHUDYVp/t6499n9zx80Hz5qffHl46+e7Dz9+joMIk5gQAIa8I8ODoG6PgyEKyh8ZByw51AYOouTBB8ugYdu4F+JmMHIwzPfnboEC50a7/zxnL20BayF/PSJBP6MYxHRFAtRHERojpeAVoGPMGLcvYUXL9QrZNutI/tHrRyOLWQviDxSKCtSKqE6aDjuFSm6apruZ9ksmfyN6ui6z+15yDABaXV3+8RTmriL8kzdSyrfyydxmtrPp9Ju1avxTtvsmumHqoG1CdrG5rsYP33wsz0JSOSBLwjFYXhjmUyMJObCJRRUy45C0B0u8AxuIjE9HEnXZ5EAnyj0ncamEUUiQInZaOJyIILGOsCEu7oCInPMMRF6JK1iqRB87EHYmSxdFmZhuJxlgcB6niO5TuetHheUUhvO5i5ZF1qT2As9LOaVZBh7TjEJEQW+9IrJpE3dZIm5Bk7cMDHhQjtzztIhXwUXG3wesznoqcuIU5UXagA4h6kWpmEIImIyPY1e3EV4JHgEnSRMc0enmC8uYdLRdQqJYjtTGmBRTDn6GNodH1Yk8DzsT6TNlMw2wu50VepdHr1UUtqJUY6DLhO4gH7IoR+UKoJnOfBMg0V0cIdO0aAsvc6B15V/HebQYVnqRDk0qqDLHLqsVHZWOXhVgdc5dF1B4xwaV9DbHHpb9RnrtbjpjWQ2i3So8py6S3jLAXwl2z1VPgvX876xipJkB2TbUqndE5jqWy8DvDihy3dX739R8uSwt2fuqzLDoRFsKWZ/f+/ErFBmWTcbjnl42DuucAKO/dldodOz/TdWtRCLOKN3pIOD/k8/VCvFQGmwuqt0cnza65f3iJOKCZuzoraFKqbN6uibU9UKnDpB5lQtf1Hlv+U4/g92UFd9a2CtgtUptm7WKuI6xdbaraJ0CJZsq34tbJbc65hmlFPQNz6H93qLz/UthUXAv9ery2eeq+3Tv3Ynif6PiNdboo5aLf38WOXHphpc97qW2bV+3W2/Pt48RA+Nb41nxkvDMg6M18Y748IYGKRx1lg0RCNq/tb8s/lX8++Meq+x0XxjFL7mP/8CWbkBDA==</latexit>

We’ll start by looking at some ways to model sequences in probabilistic terms.

When modelling probability, we usually break the sequence up into its tokens (in this case the words of the sentence) and model each as a random variable. Note that these random variables are decidedly not independent.

This leaves us with a joint distribution over 6 variables, which we would somehow like to model and Bit to a dataset.

slide 22, Probabilistic Methods 1

p(x,y) = p(x | y)p(y)<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>

Remember this rule from the Birst Probabilistic Methods lecture. If we have a joint distribution over more than two variables on the left, we can apply this rule multiple times.

chain rule of probability

16

p(W4,W3,W2,W1)

= p(W4,W3,W2 | W1)p(W1)

= p(W4,W3 | W2,W1)p(W2 | W1)p(W1)

= p(W4 | W3,W2,W1)p(W3 | W2,W1)p(W2 | W1)p(W1)<latexit sha1_base64="QoDUbgA3RE7TskrO8/tAOSdfo08=">AAAIlnicnVXRbts2FJXbrfO8dW3Wp6EvwowV2WAEkt0m6UPQrkmWYlsbL4jjApERUPS1LJiSCJJyrBL6kX5NX7c/2N+MtOxMMtVgmF58cM85l7yX16RPSciF4/zduHP3s8/vfdH8svXV1/e/efBw69sLnqQMwwAnJGHvfMSBhDEMRCgIvKMMUOQTGPqzQ80P58B4mMTnIqMwilAQh5MQI6FCV1uNHt1+4jEsh1dP847tBRr1NEo06mrka+TmP9qe13pyYNPt2/ReFI7/tSjtGn7aXFjM9ZS2lPS/5Cwy3VaD0pZW/Z+LXj1sOzvO8rNN4K5A21p9/aute7954wSnEcQCE8T5petQMZKIiRATyFteyoEiPEMBXKZisj+SYUxTATHO7R8UN0mJLRJbn6A9DhlgQTIFEGahymDjKWIIC3XOrWoqDjGKgHfG85DyAvJ5UACB1JCM5GI5RPn9ilMGDNFpiBeVrUkU8QiJqRHkWeRXg5ASYPOoGtTbVJvcUC6A4ZDrJvRVZ06pHkx+nvRX/DSjU4h5LlNG8rJREcAYTJRxCTmIlMplNerfMOMHgqXQ0XAZOzhCbHYG447KUwlUtzMhCRLVkK/KUN2J4RonUYTisfRoLj0BCyG9zk6+7F2ZPcul9HSjfN8+03SFfVti3+Z5lTwukceKrLKDG3ZiDzatFyXywlh1WGKHm1Y/LbGpwc5L7NzI7F+X6GuDXpTYhcFmJTYz2Pcl9r3ZZ6TG4rI7ksVZLA9VnpJwDicMIM5lu5tv1sLUeV+6VYueAdl282W7xzBRV2lBRJmWy9fnb37P5eF+95mzm28qfJLCWuL0dp8dOoYkKHaz0jj7+91XhiZhKA5uEh0d7/7smoloyii5Ee3t9X55bmbKgJDk+ibT4aujbm9zjhg2mrCq1W67ttG0oE6+qqrW4NcZik7V6mem/oSh7BPqpC77uoG1DlrnWHez1pHVOdatXTs2iqB6WmdYLafvdUQKyRGoG5/BGzXFp+qWQiJhP6nRZUEUqvapX6+j0W1CtFgLFWq11PPjbj42Jhh0d57vuH88bb/8dfUONa3H1vfWtuVae9ZL67XVtwYWbnxofGz82fir+V3zRfO4eVJI7zRWnkdW5Wv2/wFl9BTU</latexit><latexit sha1_base64="QoDUbgA3RE7TskrO8/tAOSdfo08=">AAAIlnicnVXRbts2FJXbrfO8dW3Wp6EvwowV2WAEkt0m6UPQrkmWYlsbL4jjApERUPS1LJiSCJJyrBL6kX5NX7c/2N+MtOxMMtVgmF58cM85l7yX16RPSciF4/zduHP3s8/vfdH8svXV1/e/efBw69sLnqQMwwAnJGHvfMSBhDEMRCgIvKMMUOQTGPqzQ80P58B4mMTnIqMwilAQh5MQI6FCV1uNHt1+4jEsh1dP847tBRr1NEo06mrka+TmP9qe13pyYNPt2/ReFI7/tSjtGn7aXFjM9ZS2lPS/5Cwy3VaD0pZW/Z+LXj1sOzvO8rNN4K5A21p9/aute7954wSnEcQCE8T5petQMZKIiRATyFteyoEiPEMBXKZisj+SYUxTATHO7R8UN0mJLRJbn6A9DhlgQTIFEGahymDjKWIIC3XOrWoqDjGKgHfG85DyAvJ5UACB1JCM5GI5RPn9ilMGDNFpiBeVrUkU8QiJqRHkWeRXg5ASYPOoGtTbVJvcUC6A4ZDrJvRVZ06pHkx+nvRX/DSjU4h5LlNG8rJREcAYTJRxCTmIlMplNerfMOMHgqXQ0XAZOzhCbHYG447KUwlUtzMhCRLVkK/KUN2J4RonUYTisfRoLj0BCyG9zk6+7F2ZPcul9HSjfN8+03SFfVti3+Z5lTwukceKrLKDG3ZiDzatFyXywlh1WGKHm1Y/LbGpwc5L7NzI7F+X6GuDXpTYhcFmJTYz2Pcl9r3ZZ6TG4rI7ksVZLA9VnpJwDicMIM5lu5tv1sLUeV+6VYueAdl282W7xzBRV2lBRJmWy9fnb37P5eF+95mzm28qfJLCWuL0dp8dOoYkKHaz0jj7+91XhiZhKA5uEh0d7/7smoloyii5Ee3t9X55bmbKgJDk+ibT4aujbm9zjhg2mrCq1W67ttG0oE6+qqrW4NcZik7V6mem/oSh7BPqpC77uoG1DlrnWHez1pHVOdatXTs2iqB6WmdYLafvdUQKyRGoG5/BGzXFp+qWQiJhP6nRZUEUqvapX6+j0W1CtFgLFWq11PPjbj42Jhh0d57vuH88bb/8dfUONa3H1vfWtuVae9ZL67XVtwYWbnxofGz82fir+V3zRfO4eVJI7zRWnkdW5Wv2/wFl9BTU</latexit><latexit sha1_base64="QoDUbgA3RE7TskrO8/tAOSdfo08=">AAAIlnicnVXRbts2FJXbrfO8dW3Wp6EvwowV2WAEkt0m6UPQrkmWYlsbL4jjApERUPS1LJiSCJJyrBL6kX5NX7c/2N+MtOxMMtVgmF58cM85l7yX16RPSciF4/zduHP3s8/vfdH8svXV1/e/efBw69sLnqQMwwAnJGHvfMSBhDEMRCgIvKMMUOQTGPqzQ80P58B4mMTnIqMwilAQh5MQI6FCV1uNHt1+4jEsh1dP847tBRr1NEo06mrka+TmP9qe13pyYNPt2/ReFI7/tSjtGn7aXFjM9ZS2lPS/5Cwy3VaD0pZW/Z+LXj1sOzvO8rNN4K5A21p9/aute7954wSnEcQCE8T5petQMZKIiRATyFteyoEiPEMBXKZisj+SYUxTATHO7R8UN0mJLRJbn6A9DhlgQTIFEGahymDjKWIIC3XOrWoqDjGKgHfG85DyAvJ5UACB1JCM5GI5RPn9ilMGDNFpiBeVrUkU8QiJqRHkWeRXg5ASYPOoGtTbVJvcUC6A4ZDrJvRVZ06pHkx+nvRX/DSjU4h5LlNG8rJREcAYTJRxCTmIlMplNerfMOMHgqXQ0XAZOzhCbHYG447KUwlUtzMhCRLVkK/KUN2J4RonUYTisfRoLj0BCyG9zk6+7F2ZPcul9HSjfN8+03SFfVti3+Z5lTwukceKrLKDG3ZiDzatFyXywlh1WGKHm1Y/LbGpwc5L7NzI7F+X6GuDXpTYhcFmJTYz2Pcl9r3ZZ6TG4rI7ksVZLA9VnpJwDicMIM5lu5tv1sLUeV+6VYueAdl282W7xzBRV2lBRJmWy9fnb37P5eF+95mzm28qfJLCWuL0dp8dOoYkKHaz0jj7+91XhiZhKA5uEh0d7/7smoloyii5Ee3t9X55bmbKgJDk+ibT4aujbm9zjhg2mrCq1W67ttG0oE6+qqrW4NcZik7V6mem/oSh7BPqpC77uoG1DlrnWHez1pHVOdatXTs2iqB6WmdYLafvdUQKyRGoG5/BGzXFp+qWQiJhP6nRZUEUqvapX6+j0W1CtFgLFWq11PPjbj42Jhh0d57vuH88bb/8dfUONa3H1vfWtuVae9ZL67XVtwYWbnxofGz82fir+V3zRfO4eVJI7zRWnkdW5Wv2/wFl9BTU</latexit><latexit sha1_base64="QoDUbgA3RE7TskrO8/tAOSdfo08=">AAAIlnicnVXRbts2FJXbrfO8dW3Wp6EvwowV2WAEkt0m6UPQrkmWYlsbL4jjApERUPS1LJiSCJJyrBL6kX5NX7c/2N+MtOxMMtVgmF58cM85l7yX16RPSciF4/zduHP3s8/vfdH8svXV1/e/efBw69sLnqQMwwAnJGHvfMSBhDEMRCgIvKMMUOQTGPqzQ80P58B4mMTnIqMwilAQh5MQI6FCV1uNHt1+4jEsh1dP847tBRr1NEo06mrka+TmP9qe13pyYNPt2/ReFI7/tSjtGn7aXFjM9ZS2lPS/5Cwy3VaD0pZW/Z+LXj1sOzvO8rNN4K5A21p9/aute7954wSnEcQCE8T5petQMZKIiRATyFteyoEiPEMBXKZisj+SYUxTATHO7R8UN0mJLRJbn6A9DhlgQTIFEGahymDjKWIIC3XOrWoqDjGKgHfG85DyAvJ5UACB1JCM5GI5RPn9ilMGDNFpiBeVrUkU8QiJqRHkWeRXg5ASYPOoGtTbVJvcUC6A4ZDrJvRVZ06pHkx+nvRX/DSjU4h5LlNG8rJREcAYTJRxCTmIlMplNerfMOMHgqXQ0XAZOzhCbHYG447KUwlUtzMhCRLVkK/KUN2J4RonUYTisfRoLj0BCyG9zk6+7F2ZPcul9HSjfN8+03SFfVti3+Z5lTwukceKrLKDG3ZiDzatFyXywlh1WGKHm1Y/LbGpwc5L7NzI7F+X6GuDXpTYhcFmJTYz2Pcl9r3ZZ6TG4rI7ksVZLA9VnpJwDicMIM5lu5tv1sLUeV+6VYueAdl282W7xzBRV2lBRJmWy9fnb37P5eF+95mzm28qfJLCWuL0dp8dOoYkKHaz0jj7+91XhiZhKA5uEh0d7/7smoloyii5Ee3t9X55bmbKgJDk+ibT4aujbm9zjhg2mrCq1W67ttG0oE6+qqrW4NcZik7V6mem/oSh7BPqpC77uoG1DlrnWHez1pHVOdatXTs2iqB6WmdYLafvdUQKyRGoG5/BGzXFp+qWQiJhP6nRZUEUqvapX6+j0W1CtFgLFWq11PPjbj42Jhh0d57vuH88bb/8dfUONa3H1vfWtuVae9ZL67XVtwYWbnxofGz82fir+V3zRfO4eVJI7zRWnkdW5Wv2/wFl9BTU</latexit>

p(prize | a,won,have, you, congratulations)<latexit sha1_base64="66CPN2K8ZInKJ4Xu/jjW2bwfsFM=">AAAH2nicfVXPb9s2FJa7tu68pku34y7CjBXtYASSsybpoUDXJGux/kgWxEmByAgo+lkmTIkERdlWCJ52GnbdP7fL/pZRshRIllpe/Px933vi+/hA+pySWDrOv507X929d7/74OveNw+3Hn27/fi7i5glAsMIM8rEJx/FQEkEI0kkhU9cAAp9Cpf+/DDjLxcgYsKic5lyGIcoiMiUYCQNdL29esKfegIrT8JKKi7IDWjthWRie0GJIq0HtsfKv0sW5YBfAjO0gBxJSyRlSQ7wEsAsCgSSCc0/G2v97Hq77+w4+bKbgVsEfatYp9eP77/zJgwnIUQSUxTHV67D5VghIQmmoHteEgNHeI4CuErk9GCsSMQTCRHW9k+GmybUlszOXLAnRACWNDUBwoKYCjaeIYGwNF716qViiFAI8WCyIDxeh/EiWAcSGaPHapUfhN6qZSrTMZ8RvKptTaEwDpGcNcA4Df06CAkFsQjrYLZNs8kN5QoEJnFmwqlx5oTnLp+z04KfpXwGxnaVCKqriYYAIWBqEvMwBplwlXdjJmoev5QigUEW5tjLIyTmZzAZmDo1oL6dKWVI1iHftGHciWCJWRiiaKI8rovh8AY7Oveuyp5ppbzMKN+3zzK6xn6ssB+1rpPHFfLYkHV2dMtO7dFm6kWFvGh89bLCXm6m+kmFTRrsosIuGpX9ZYVeNuhVhV012LTCpg32psLeNH1GZiyuhmO1Pov8UNUJJQt4IwAirfpDvdmLMOd95dZTshlQfVfndk9gaq6jNRGmmVy9Pf/wXqvDg+FzZ09vKnyaQClxdveeHzoNSbDeTaFxDg6GrxsaJlAU3BY6Ot771W0W4ong9Fa0v7/724tmpRQoZcvbSoevj4a7m3MkcMOEole779oN04I2edFVa4LflrB2qlU/b+rfCJR+Rs3aqpcGtmbwtozSzdaMtC2jtLbM2GiCZ9M6N88Gz+51RNeSIzA3voAPZopPzC2FJBM/m9EVQUiMfebXG2TRl4RoVQpN1OuZ58fdfGyawWi482LH/eOX/qvfi3fogfWD9aP11HKtfeuV9dY6tUYWtv7r3O1sdR51x90/u391/15L73SKnO+t2ur+8z/ATtzf</latexit><latexit sha1_base64="66CPN2K8ZInKJ4Xu/jjW2bwfsFM=">AAAH2nicfVXPb9s2FJa7tu68pku34y7CjBXtYASSsybpoUDXJGux/kgWxEmByAgo+lkmTIkERdlWCJ52GnbdP7fL/pZRshRIllpe/Px933vi+/hA+pySWDrOv507X929d7/74OveNw+3Hn27/fi7i5glAsMIM8rEJx/FQEkEI0kkhU9cAAp9Cpf+/DDjLxcgYsKic5lyGIcoiMiUYCQNdL29esKfegIrT8JKKi7IDWjthWRie0GJIq0HtsfKv0sW5YBfAjO0gBxJSyRlSQ7wEsAsCgSSCc0/G2v97Hq77+w4+bKbgVsEfatYp9eP77/zJgwnIUQSUxTHV67D5VghIQmmoHteEgNHeI4CuErk9GCsSMQTCRHW9k+GmybUlszOXLAnRACWNDUBwoKYCjaeIYGwNF716qViiFAI8WCyIDxeh/EiWAcSGaPHapUfhN6qZSrTMZ8RvKptTaEwDpGcNcA4Df06CAkFsQjrYLZNs8kN5QoEJnFmwqlx5oTnLp+z04KfpXwGxnaVCKqriYYAIWBqEvMwBplwlXdjJmoev5QigUEW5tjLIyTmZzAZmDo1oL6dKWVI1iHftGHciWCJWRiiaKI8rovh8AY7Oveuyp5ppbzMKN+3zzK6xn6ssB+1rpPHFfLYkHV2dMtO7dFm6kWFvGh89bLCXm6m+kmFTRrsosIuGpX9ZYVeNuhVhV012LTCpg32psLeNH1GZiyuhmO1Pov8UNUJJQt4IwAirfpDvdmLMOd95dZTshlQfVfndk9gaq6jNRGmmVy9Pf/wXqvDg+FzZ09vKnyaQClxdveeHzoNSbDeTaFxDg6GrxsaJlAU3BY6Ot771W0W4ong9Fa0v7/724tmpRQoZcvbSoevj4a7m3MkcMOEole779oN04I2edFVa4LflrB2qlU/b+rfCJR+Rs3aqpcGtmbwtozSzdaMtC2jtLbM2GiCZ9M6N88Gz+51RNeSIzA3voAPZopPzC2FJBM/m9EVQUiMfebXG2TRl4RoVQpN1OuZ58fdfGyawWi482LH/eOX/qvfi3fogfWD9aP11HKtfeuV9dY6tUYWtv7r3O1sdR51x90/u391/15L73SKnO+t2ur+8z/ATtzf</latexit><latexit sha1_base64="66CPN2K8ZInKJ4Xu/jjW2bwfsFM=">AAAH2nicfVXPb9s2FJa7tu68pku34y7CjBXtYASSsybpoUDXJGux/kgWxEmByAgo+lkmTIkERdlWCJ52GnbdP7fL/pZRshRIllpe/Px933vi+/hA+pySWDrOv507X929d7/74OveNw+3Hn27/fi7i5glAsMIM8rEJx/FQEkEI0kkhU9cAAp9Cpf+/DDjLxcgYsKic5lyGIcoiMiUYCQNdL29esKfegIrT8JKKi7IDWjthWRie0GJIq0HtsfKv0sW5YBfAjO0gBxJSyRlSQ7wEsAsCgSSCc0/G2v97Hq77+w4+bKbgVsEfatYp9eP77/zJgwnIUQSUxTHV67D5VghIQmmoHteEgNHeI4CuErk9GCsSMQTCRHW9k+GmybUlszOXLAnRACWNDUBwoKYCjaeIYGwNF716qViiFAI8WCyIDxeh/EiWAcSGaPHapUfhN6qZSrTMZ8RvKptTaEwDpGcNcA4Df06CAkFsQjrYLZNs8kN5QoEJnFmwqlx5oTnLp+z04KfpXwGxnaVCKqriYYAIWBqEvMwBplwlXdjJmoev5QigUEW5tjLIyTmZzAZmDo1oL6dKWVI1iHftGHciWCJWRiiaKI8rovh8AY7Oveuyp5ppbzMKN+3zzK6xn6ssB+1rpPHFfLYkHV2dMtO7dFm6kWFvGh89bLCXm6m+kmFTRrsosIuGpX9ZYVeNuhVhV012LTCpg32psLeNH1GZiyuhmO1Pov8UNUJJQt4IwAirfpDvdmLMOd95dZTshlQfVfndk9gaq6jNRGmmVy9Pf/wXqvDg+FzZ09vKnyaQClxdveeHzoNSbDeTaFxDg6GrxsaJlAU3BY6Ot771W0W4ong9Fa0v7/724tmpRQoZcvbSoevj4a7m3MkcMOEole779oN04I2edFVa4LflrB2qlU/b+rfCJR+Rs3aqpcGtmbwtozSzdaMtC2jtLbM2GiCZ9M6N88Gz+51RNeSIzA3voAPZopPzC2FJBM/m9EVQUiMfebXG2TRl4RoVQpN1OuZ58fdfGyawWi482LH/eOX/qvfi3fogfWD9aP11HKtfeuV9dY6tUYWtv7r3O1sdR51x90/u391/15L73SKnO+t2ur+8z/ATtzf</latexit><latexit sha1_base64="66CPN2K8ZInKJ4Xu/jjW2bwfsFM=">AAAH2nicfVXPb9s2FJa7tu68pku34y7CjBXtYASSsybpoUDXJGux/kgWxEmByAgo+lkmTIkERdlWCJ52GnbdP7fL/pZRshRIllpe/Px933vi+/hA+pySWDrOv507X929d7/74OveNw+3Hn27/fi7i5glAsMIM8rEJx/FQEkEI0kkhU9cAAp9Cpf+/DDjLxcgYsKic5lyGIcoiMiUYCQNdL29esKfegIrT8JKKi7IDWjthWRie0GJIq0HtsfKv0sW5YBfAjO0gBxJSyRlSQ7wEsAsCgSSCc0/G2v97Hq77+w4+bKbgVsEfatYp9eP77/zJgwnIUQSUxTHV67D5VghIQmmoHteEgNHeI4CuErk9GCsSMQTCRHW9k+GmybUlszOXLAnRACWNDUBwoKYCjaeIYGwNF716qViiFAI8WCyIDxeh/EiWAcSGaPHapUfhN6qZSrTMZ8RvKptTaEwDpGcNcA4Df06CAkFsQjrYLZNs8kN5QoEJnFmwqlx5oTnLp+z04KfpXwGxnaVCKqriYYAIWBqEvMwBplwlXdjJmoev5QigUEW5tjLIyTmZzAZmDo1oL6dKWVI1iHftGHciWCJWRiiaKI8rovh8AY7Oveuyp5ppbzMKN+3zzK6xn6ssB+1rpPHFfLYkHV2dMtO7dFm6kWFvGh89bLCXm6m+kmFTRrsosIuGpX9ZYVeNuhVhV012LTCpg32psLeNH1GZiyuhmO1Pov8UNUJJQt4IwAirfpDvdmLMOd95dZTshlQfVfndk9gaq6jNRGmmVy9Pf/wXqvDg+FzZ09vKnyaQClxdveeHzoNSbDeTaFxDg6GrxsaJlAU3BY6Ot771W0W4ong9Fa0v7/724tmpRQoZcvbSoevj4a7m3MkcMOEole779oN04I2edFVa4LflrB2qlU/b+rfCJR+Rs3aqpcGtmbwtozSzdaMtC2jtLbM2GiCZ9M6N88Gz+51RNeSIzA3voAPZopPzC2FJBM/m9EVQUiMfebXG2TRl4RoVQpN1OuZ58fdfGyawWi482LH/eOX/qvfi3fogfWD9aP11HKtfeuV9dY6tUYWtv7r3O1sdR51x90/u391/15L73SKnO+t2ur+8z/ATtzf</latexit>

p(W4,W3,W2,W1)

= p(W4,W3,W2 | W1)p(W1)

= p(W4,W3 | W2,W1)p(W2 | W1)p(W1)


p(W4,W3,W2,W1)

= p(W4,W3,W2 | W1)p(W1)

= p(W4,W3 | W2,W1)p(W2 | W1)p(W1)


p(W4,W3,W2,W1)

= p(W4,W3,W2 | W1)p(W1)

= p(W4,W3 | W2,W1)p(W2 | W1)p(W1)


This gives us the chain rule of probability (not to be confused with the chain rule of calculus, which is entirely different), which is often used in modelling sequences.

The chain rule allows us to break a joint distribution on many variables into a product of conditional distributions. In sequences, we often apply it so that each word becomes conditioned on the words before it.

This tells us that if we build a model that can estimate the probability p(x|y, z) of a word x based on the words y, z that precede it, we can then chain this estimator to give us the joint probability of the whole sentence x, y, z.

17

log p(sentence) =

Σword log p(word| all words before word)

In other words, we can rewrite the probability of a sentences as the product of the probability of each word, conditioned on its history. If we use the log probability, this becomes a sum.

Note that applying the chain rule in a different order would allow us to condition any word on any other word, but conditioning on the history Bits well with the sequential nature of the data, and will allow us to make some useful simpliBications later.

language model

18

the man fell out of the …aquarium

pool

window

cycling

p(W | the,man, fell, out, of, the)<latexit sha1_base64="YK7RHBilRFNd/Jkjf1X5lNUGScc=">AAAHuHicfVVdj9NGFDXQEkhLWeCRF6sREkXRys7C7rYSEuxHQWpht6vNBmkdLePJtTPK2B7NjLMxo/ll/BIe+9r+Ccaxk9qxi19ycs851zNnrsY+o0RIx/ly4+at776/3blzt/vDj/d+ur/14OGFSFKOYYgTmvAPPhJASQxDSSSFD4wDinwKI392mPOjOXBBkvhcZgzGEQpjEhCMpCldbQ3ZU8/HaqRtLyIT25OwkEpOQfdLHKF4jQOgdP0nSeV/OFjD3PvL1VbP2XaWj90Ebgl6VvmcXj24/Yc3SXAaQSwxRUJcug6TY4W4JJiC7nqpAIbwDIVwmcpgf6xIzFIJMdb2E8MFKbVlYudbtCeEA5Y0MwBhTkwHG08RR1iaILr1VgJiFIHoT+aEiQKKeVgAiUyKY7VYpqzv1Zwq5IhNCV7UlqZQJCIkp42iyCK/XoSUAp9H9WK+TLPIDeUCOCYiD+HUJHPC8pMT58lpyU8zNoVYaJVyqqtGQwDnEBjjEgqQKVPL3ZhxmYmXkqfQz+Gy9vII8dkZTPqmT61QX05AEyTrJd9sw6QTwzVOIjMwE+UxrYp58PrbepldlT3TSnl5UL5vn+V0jX1fYd9rXSePK+SxIevscM0G9nDTelEhLxpvHVXY0abVTyts2mDnFXbe6OxfV+jrBr2osIsGm1XYrMF+qrCfmjkjMxaXg7EqzmJ5qOqEkjm84QCxVr2B3twLN+d96dYt+QyonquXcU8gMHdNQURZLldvz9/9qdXh/uCFs6s3FT5NYSVxdnZfHDoNSVisptQ4+/uDg4Ym4SgO142Ojndfu81GLOWMrkV7ezu//9rslJlbLLledzo8OBrsbM4Rx40Qyr3aPdduhBa2yctdtRr8NkORVKt+1tS/4Sj7H3XS1n0VYKuDtTlWabY6sjbHKtqVY2MTLJ/WGTavy+91RAvJEZgbn8M7M8Un5pZCMuHPzOjyMCImPvPr9XP0LSFarIQGdbvm8+Nufmya4GKw7Trb7l/Pe68Oyg/RHeux9bP11HKtPeuV9dY6tYYWtj5bf1v/WP92fut87IQdUkhv3ig9j6za0+FfAZhKzsw=</latexit><latexit sha1_base64="YK7RHBilRFNd/Jkjf1X5lNUGScc=">AAAHuHicfVVdj9NGFDXQEkhLWeCRF6sREkXRys7C7rYSEuxHQWpht6vNBmkdLePJtTPK2B7NjLMxo/ll/BIe+9r+Ccaxk9qxi19ycs851zNnrsY+o0RIx/ly4+at776/3blzt/vDj/d+ur/14OGFSFKOYYgTmvAPPhJASQxDSSSFD4wDinwKI392mPOjOXBBkvhcZgzGEQpjEhCMpCldbQ3ZU8/HaqRtLyIT25OwkEpOQfdLHKF4jQOgdP0nSeV/OFjD3PvL1VbP2XaWj90Ebgl6VvmcXj24/Yc3SXAaQSwxRUJcug6TY4W4JJiC7nqpAIbwDIVwmcpgf6xIzFIJMdb2E8MFKbVlYudbtCeEA5Y0MwBhTkwHG08RR1iaILr1VgJiFIHoT+aEiQKKeVgAiUyKY7VYpqzv1Zwq5IhNCV7UlqZQJCIkp42iyCK/XoSUAp9H9WK+TLPIDeUCOCYiD+HUJHPC8pMT58lpyU8zNoVYaJVyqqtGQwDnEBjjEgqQKVPL3ZhxmYmXkqfQz+Gy9vII8dkZTPqmT61QX05AEyTrJd9sw6QTwzVOIjMwE+UxrYp58PrbepldlT3TSnl5UL5vn+V0jX1fYd9rXSePK+SxIevscM0G9nDTelEhLxpvHVXY0abVTyts2mDnFXbe6OxfV+jrBr2osIsGm1XYrMF+qrCfmjkjMxaXg7EqzmJ5qOqEkjm84QCxVr2B3twLN+d96dYt+QyonquXcU8gMHdNQURZLldvz9/9qdXh/uCFs6s3FT5NYSVxdnZfHDoNSVisptQ4+/uDg4Ym4SgO142Ojndfu81GLOWMrkV7ezu//9rslJlbLLledzo8OBrsbM4Rx40Qyr3aPdduhBa2yctdtRr8NkORVKt+1tS/4Sj7H3XS1n0VYKuDtTlWabY6sjbHKtqVY2MTLJ/WGTavy+91RAvJEZgbn8M7M8Un5pZCMuHPzOjyMCImPvPr9XP0LSFarIQGdbvm8+Nufmya4GKw7Trb7l/Pe68Oyg/RHeux9bP11HKtPeuV9dY6tYYWtj5bf1v/WP92fut87IQdUkhv3ig9j6za0+FfAZhKzsw=</latexit><latexit sha1_base64="YK7RHBilRFNd/Jkjf1X5lNUGScc=">AAAHuHicfVVdj9NGFDXQEkhLWeCRF6sREkXRys7C7rYSEuxHQWpht6vNBmkdLePJtTPK2B7NjLMxo/ll/BIe+9r+Ccaxk9qxi19ycs851zNnrsY+o0RIx/ly4+at776/3blzt/vDj/d+ur/14OGFSFKOYYgTmvAPPhJASQxDSSSFD4wDinwKI392mPOjOXBBkvhcZgzGEQpjEhCMpCldbQ3ZU8/HaqRtLyIT25OwkEpOQfdLHKF4jQOgdP0nSeV/OFjD3PvL1VbP2XaWj90Ebgl6VvmcXj24/Yc3SXAaQSwxRUJcug6TY4W4JJiC7nqpAIbwDIVwmcpgf6xIzFIJMdb2E8MFKbVlYudbtCeEA5Y0MwBhTkwHG08RR1iaILr1VgJiFIHoT+aEiQKKeVgAiUyKY7VYpqzv1Zwq5IhNCV7UlqZQJCIkp42iyCK/XoSUAp9H9WK+TLPIDeUCOCYiD+HUJHPC8pMT58lpyU8zNoVYaJVyqqtGQwDnEBjjEgqQKVPL3ZhxmYmXkqfQz+Gy9vII8dkZTPqmT61QX05AEyTrJd9sw6QTwzVOIjMwE+UxrYp58PrbepldlT3TSnl5UL5vn+V0jX1fYd9rXSePK+SxIevscM0G9nDTelEhLxpvHVXY0abVTyts2mDnFXbe6OxfV+jrBr2osIsGm1XYrMF+qrCfmjkjMxaXg7EqzmJ5qOqEkjm84QCxVr2B3twLN+d96dYt+QyonquXcU8gMHdNQURZLldvz9/9qdXh/uCFs6s3FT5NYSVxdnZfHDoNSVisptQ4+/uDg4Ym4SgO142Ojndfu81GLOWMrkV7ezu//9rslJlbLLledzo8OBrsbM4Rx40Qyr3aPdduhBa2yctdtRr8NkORVKt+1tS/4Sj7H3XS1n0VYKuDtTlWabY6sjbHKtqVY2MTLJ/WGTavy+91RAvJEZgbn8M7M8Un5pZCMuHPzOjyMCImPvPr9XP0LSFarIQGdbvm8+Nufmya4GKw7Trb7l/Pe68Oyg/RHeux9bP11HKtPeuV9dY6tYYWtj5bf1v/WP92fut87IQdUkhv3ig9j6za0+FfAZhKzsw=</latexit><latexit sha1_base64="YK7RHBilRFNd/Jkjf1X5lNUGScc=">AAAHuHicfVVdj9NGFDXQEkhLWeCRF6sREkXRys7C7rYSEuxHQWpht6vNBmkdLePJtTPK2B7NjLMxo/ll/BIe+9r+Ccaxk9qxi19ycs851zNnrsY+o0RIx/ly4+at776/3blzt/vDj/d+ur/14OGFSFKOYYgTmvAPPhJASQxDSSSFD4wDinwKI392mPOjOXBBkvhcZgzGEQpjEhCMpCldbQ3ZU8/HaqRtLyIT25OwkEpOQfdLHKF4jQOgdP0nSeV/OFjD3PvL1VbP2XaWj90Ebgl6VvmcXj24/Yc3SXAaQSwxRUJcug6TY4W4JJiC7nqpAIbwDIVwmcpgf6xIzFIJMdb2E8MFKbVlYudbtCeEA5Y0MwBhTkwHG08RR1iaILr1VgJiFIHoT+aEiQKKeVgAiUyKY7VYpqzv1Zwq5IhNCV7UlqZQJCIkp42iyCK/XoSUAp9H9WK+TLPIDeUCOCYiD+HUJHPC8pMT58lpyU8zNoVYaJVyqqtGQwDnEBjjEgqQKVPL3ZhxmYmXkqfQz+Gy9vII8dkZTPqmT61QX05AEyTrJd9sw6QTwzVOIjMwE+UxrYp58PrbepldlT3TSnl5UL5vn+V0jX1fYd9rXSePK+SxIevscM0G9nDTelEhLxpvHVXY0abVTyts2mDnFXbe6OxfV+jrBr2osIsGm1XYrMF+qrCfmjkjMxaXg7EqzmJ5qOqEkjm84QCxVr2B3twLN+d96dYt+QyonquXcU8gMHdNQURZLldvz9/9qdXh/uCFs6s3FT5NYSVxdnZfHDoNSVisptQ4+/uDg4Ym4SgO142Ojndfu81GLOWMrkV7ezu//9rslJlbLLledzo8OBrsbM4Rx40Qyr3aPdduhBa2yctdtRr8NkORVKt+1tS/4Sj7H3XS1n0VYKuDtTlWabY6sjbHKtqVY2MTLJ/WGTavy+91RAvJEZgbn8M7M8Un5pZCMuHPzOjyMCImPvPr9XP0LSFarIQGdbvm8+Nufmya4GKw7Trb7l/Pe68Oyg/RHeux9bP11HKtPeuV9dY6tYYWtj5bf1v/WP92fut87IQdUkhv3ig9j6za0+FfAZhKzsw=</latexit>

A perfect language model would encompass everything we know about language: the grammar, the idiom and the physical reality it describes. For instance, it would give window a very high probability, since that is a very reasonable way to complete the sentence. Aquarium is less likely, but still physically possible and grammatically correct. A very clever language model might know that falling out of a pool is not physically possible (except under unusual circumstances), so that should get a lower probability, and Binally cycling is ungrammatical, so that should get very low probability (perhaps even zero).

The problem is most sentences of this length will never have been seen before in their entirety. A simple way to get a basic model is to limit how far back we look in the sentence.

Markov assumption

19

p(prize | a,won,have, you, congratulations) =p(prize | a,won)

<latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit><latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit><latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit><latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit>

This is called a Markov assumption. It’s a bit like the naive Bayes assumption: we know it’s incorrect, but it still yields a very usable model. The number of words we retain in the conditional is called the order of the Markov model: this is a Markov assumption for a second-order Markov model.

Markov model

20

p(prize, a,won,have, you, congratulations)=p(prize | a,won) p(a | won,have) p(won | have, you)p(have | you, congratulations) p(you | congratulations) p(congratulations)

<latexit sha1_base64="CQ3QhuygUQh3w1H35+vTtFEG2mo=">AAAJJniclVXbbttGEKXVNlHUpo3Tx74QFRo4gWCQcmI7KAwksd0EbROrhmWnMQVjuRpRhJbkYrmURBP7Qf2OfkCfiqJv/ZQubwbJZWOULxrOOWc4czha2pS4ITeMvzc6n3z62Z273Xu9z7+4/+VXDzYfnodBxDCMcUAC9t5GIRDXhzF3OYH3lAHybAIX9uIwxS+WwEI38M94TGHiIcd3Zy5GXKauNjd+p1uPLIYTi8OaJ5S51yDEQLecMoWy26C8XQV+lrDLxBwtc0VcZuIgyhK0TODAdxjiEckeGgrx2LJ6B/ojuqU82fLc6S0Pf2x9T7fqnFx1W4+FsE7LpbdNkzacttvg5eL/NbhsoSHIi9yi+Sjh6kHf2DayS1cDswj6WnGNrjbv/GRNAxx54HNMUBhemgblkwQx7mIComdFIVCEF8iBy4jP9ieJ69OIg4+F/p3EZhHReaCnG6VPXQaYk1gGCDNXVtDxHDGEudy7Xr1UCD7yIBxMly4N8zBcOnnAkVzaSbLOllrcrykTOTGdu3hday1BXughPleSYezZ9SREBNjSqyfTNmWTDeYaGHbD1ISRdOaEZi6fBaMCn8d0DtL2JGJEVIUSAMZgJoVZGAKPaJJNI/+di/CAswgGaZjlDo4QW5zCdCDr1BL1dmYkQLyesuUY0h0fVjjwPORPE4uKYjmswbbIvKuipyJJrNQo29ZPU7iGvqug74Sog8cV8FiCdXR8g870cVN6XgHPladeVNCLptSOKmikoMsKulQq26sKvFLgdQVdK2hcQWMFva6g16rPSK7F5XCS5O8ie6nJCXGX8JoB+CLpD0VzFibf96VZl6Q7kPRNkdk9hZk82nPAi1N68ubs7c8iOdwfPjN2RZNhkwhKirGz++zQUChO3k3BMfb3h68UTsCQ79wUOjrefWmqhWjEKLkh7e3t/PBcrRQDIcHqptLhq6PhTnOPGFZMKGbV+6aumOa00YupWgV2myB3qpW/UPmvGYr/gx20VS8NbFXQNkXpZqsiblOU1paKxhA03daF/GzQ9FxHJKccgTzxGbyVW3wiTynEA/ZEri5zPFfaJ3+tQRp9jIjWJVFGvZ78/JjNj40ajIfbz7fNX572X/xYfIe62jfat9qWZmp72gvtjTbSxhruPOmMOr92PnR/6/7R/bP7V07tbBSar7Xa1f3nXwvBYlQ=</latexit><latexit sha1_base64="CQ3QhuygUQh3w1H35+vTtFEG2mo=">AAAJJniclVXbbttGEKXVNlHUpo3Tx74QFRo4gWCQcmI7KAwksd0EbROrhmWnMQVjuRpRhJbkYrmURBP7Qf2OfkCfiqJv/ZQubwbJZWOULxrOOWc4czha2pS4ITeMvzc6n3z62Z273Xu9z7+4/+VXDzYfnodBxDCMcUAC9t5GIRDXhzF3OYH3lAHybAIX9uIwxS+WwEI38M94TGHiIcd3Zy5GXKauNjd+p1uPLIYTi8OaJ5S51yDEQLecMoWy26C8XQV+lrDLxBwtc0VcZuIgyhK0TODAdxjiEckeGgrx2LJ6B/ojuqU82fLc6S0Pf2x9T7fqnFx1W4+FsE7LpbdNkzacttvg5eL/NbhsoSHIi9yi+Sjh6kHf2DayS1cDswj6WnGNrjbv/GRNAxx54HNMUBhemgblkwQx7mIComdFIVCEF8iBy4jP9ieJ69OIg4+F/p3EZhHReaCnG6VPXQaYk1gGCDNXVtDxHDGEudy7Xr1UCD7yIBxMly4N8zBcOnnAkVzaSbLOllrcrykTOTGdu3hday1BXughPleSYezZ9SREBNjSqyfTNmWTDeYaGHbD1ISRdOaEZi6fBaMCn8d0DtL2JGJEVIUSAMZgJoVZGAKPaJJNI/+di/CAswgGaZjlDo4QW5zCdCDr1BL1dmYkQLyesuUY0h0fVjjwPORPE4uKYjmswbbIvKuipyJJrNQo29ZPU7iGvqug74Sog8cV8FiCdXR8g870cVN6XgHPladeVNCLptSOKmikoMsKulQq26sKvFLgdQVdK2hcQWMFva6g16rPSK7F5XCS5O8ie6nJCXGX8JoB+CLpD0VzFibf96VZl6Q7kPRNkdk9hZk82nPAi1N68ubs7c8iOdwfPjN2RZNhkwhKirGz++zQUChO3k3BMfb3h68UTsCQ79wUOjrefWmqhWjEKLkh7e3t/PBcrRQDIcHqptLhq6PhTnOPGFZMKGbV+6aumOa00YupWgV2myB3qpW/UPmvGYr/gx20VS8NbFXQNkXpZqsiblOU1paKxhA03daF/GzQ9FxHJKccgTzxGbyVW3wiTynEA/ZEri5zPFfaJ3+tQRp9jIjWJVFGvZ78/JjNj40ajIfbz7fNX572X/xYfIe62jfat9qWZmp72gvtjTbSxhruPOmMOr92PnR/6/7R/bP7V07tbBSar7Xa1f3nXwvBYlQ=</latexit><latexit sha1_base64="CQ3QhuygUQh3w1H35+vTtFEG2mo=">AAAJJniclVXbbttGEKXVNlHUpo3Tx74QFRo4gWCQcmI7KAwksd0EbROrhmWnMQVjuRpRhJbkYrmURBP7Qf2OfkCfiqJv/ZQubwbJZWOULxrOOWc4czha2pS4ITeMvzc6n3z62Z273Xu9z7+4/+VXDzYfnodBxDCMcUAC9t5GIRDXhzF3OYH3lAHybAIX9uIwxS+WwEI38M94TGHiIcd3Zy5GXKauNjd+p1uPLIYTi8OaJ5S51yDEQLecMoWy26C8XQV+lrDLxBwtc0VcZuIgyhK0TODAdxjiEckeGgrx2LJ6B/ojuqU82fLc6S0Pf2x9T7fqnFx1W4+FsE7LpbdNkzacttvg5eL/NbhsoSHIi9yi+Sjh6kHf2DayS1cDswj6WnGNrjbv/GRNAxx54HNMUBhemgblkwQx7mIComdFIVCEF8iBy4jP9ieJ69OIg4+F/p3EZhHReaCnG6VPXQaYk1gGCDNXVtDxHDGEudy7Xr1UCD7yIBxMly4N8zBcOnnAkVzaSbLOllrcrykTOTGdu3hday1BXughPleSYezZ9SREBNjSqyfTNmWTDeYaGHbD1ISRdOaEZi6fBaMCn8d0DtL2JGJEVIUSAMZgJoVZGAKPaJJNI/+di/CAswgGaZjlDo4QW5zCdCDr1BL1dmYkQLyesuUY0h0fVjjwPORPE4uKYjmswbbIvKuipyJJrNQo29ZPU7iGvqug74Sog8cV8FiCdXR8g870cVN6XgHPladeVNCLptSOKmikoMsKulQq26sKvFLgdQVdK2hcQWMFva6g16rPSK7F5XCS5O8ie6nJCXGX8JoB+CLpD0VzFibf96VZl6Q7kPRNkdk9hZk82nPAi1N68ubs7c8iOdwfPjN2RZNhkwhKirGz++zQUChO3k3BMfb3h68UTsCQ79wUOjrefWmqhWjEKLkh7e3t/PBcrRQDIcHqptLhq6PhTnOPGFZMKGbV+6aumOa00YupWgV2myB3qpW/UPmvGYr/gx20VS8NbFXQNkXpZqsiblOU1paKxhA03daF/GzQ9FxHJKccgTzxGbyVW3wiTynEA/ZEri5zPFfaJ3+tQRp9jIjWJVFGvZ78/JjNj40ajIfbz7fNX572X/xYfIe62jfat9qWZmp72gvtjTbSxhruPOmMOr92PnR/6/7R/bP7V07tbBSar7Xa1f3nXwvBYlQ=</latexit><latexit sha1_base64="CQ3QhuygUQh3w1H35+vTtFEG2mo=">AAAJJniclVXbbttGEKXVNlHUpo3Tx74QFRo4gWCQcmI7KAwksd0EbROrhmWnMQVjuRpRhJbkYrmURBP7Qf2OfkCfiqJv/ZQubwbJZWOULxrOOWc4czha2pS4ITeMvzc6n3z62Z273Xu9z7+4/+VXDzYfnodBxDCMcUAC9t5GIRDXhzF3OYH3lAHybAIX9uIwxS+WwEI38M94TGHiIcd3Zy5GXKauNjd+p1uPLIYTi8OaJ5S51yDEQLecMoWy26C8XQV+lrDLxBwtc0VcZuIgyhK0TODAdxjiEckeGgrx2LJ6B/ojuqU82fLc6S0Pf2x9T7fqnFx1W4+FsE7LpbdNkzacttvg5eL/NbhsoSHIi9yi+Sjh6kHf2DayS1cDswj6WnGNrjbv/GRNAxx54HNMUBhemgblkwQx7mIComdFIVCEF8iBy4jP9ieJ69OIg4+F/p3EZhHReaCnG6VPXQaYk1gGCDNXVtDxHDGEudy7Xr1UCD7yIBxMly4N8zBcOnnAkVzaSbLOllrcrykTOTGdu3hday1BXughPleSYezZ9SREBNjSqyfTNmWTDeYaGHbD1ISRdOaEZi6fBaMCn8d0DtL2JGJEVIUSAMZgJoVZGAKPaJJNI/+di/CAswgGaZjlDo4QW5zCdCDr1BL1dmYkQLyesuUY0h0fVjjwPORPE4uKYjmswbbIvKuipyJJrNQo29ZPU7iGvqug74Sog8cV8FiCdXR8g870cVN6XgHPladeVNCLptSOKmikoMsKulQq26sKvFLgdQVdK2hcQWMFva6g16rPSK7F5XCS5O8ie6nJCXGX8JoB+CLpD0VzFibf96VZl6Q7kPRNkdk9hZk82nPAi1N68ubs7c8iOdwfPjN2RZNhkwhKirGz++zQUChO3k3BMfb3h68UTsCQ79wUOjrefWmqhWjEKLkh7e3t/PBcrRQDIcHqptLhq6PhTnOPGFZMKGbV+6aumOa00YupWgV2myB3qpW/UPmvGYr/gx20VS8NbFXQNkXpZqsiblOU1paKxhA03daF/GzQ9FxHJKccgTzxGbyVW3wiTynEA/ZEri5zPFfaJ3+tQRp9jIjWJVFGvZ78/JjNj40ajIfbz7fNX572X/xYfIe62jfat9qWZmp72gvtjTbSxhruPOmMOr92PnR/6/7R/bP7V07tbBSar7Xa1f3nXwvBYlQ=</latexit>

p(prize | a,won) ⇡ #“won a prize”#“won a”

<latexit sha1_base64="TEn9cwN6ZDLtIN8rQ1vBPZO/Syk=">AAAIK3icjVVdb9s2FJW7ru68dU23x70QM4Z2g5FKzpqkDwW6JlmLbW2yIE4KRG5K0deyYEoiSMq2QvAv9X/sfS8b9rr/MUqWA31tmF50dc85V7yHF6THaCCkbf/RufXR7Y/vdO9+0vv0s3uf39968MW5iBNOYERiGvO3HhZAgwhGMpAU3jIOOPQoXHjzgwy/WAAXQRydyZTBOMR+FEwDgqVJXW39xh65nChXwkoqxoNr0NoNgwly/U0Waz1Abrz5XMaR1t8iFzPG4xVypxwThdw+WsPv3+sqGWldLpZ/1v9YaB8+1Ahp9b9rlYX6aqtvb9v5g5qBUwR9q3hOrh7c+dmdxCQJIZKEYiEuHZvJscJcBoSC7rmJAIbJHPtwmcjp/lgFEUskRESjbww2TSiSMcpsRZOAA5E0NQEmPDAVEJlhY4005veqpQREOAQxmCwCJtahWPjrQGKzc2O1yndW36solc8xmwVkVVmawqEIsZw1kiINvWoSEgp8EVaT2TLNImvMFXASiMyEE+PMMcumRZzFJwU+S9kMIqFVwqkuCw0AnMPUCPNQgEyYyrsxIzoXzyRPYJCFee7ZIebzU5gMTJ1KorqcKY2xrKY804ZxJ4IlicMQRxPlMl0MhjvY1rl3ZfRUK+VmRnkeOs3gCvqmhL7RugoelcAjA1bR0Q06RaO69LwEnjf+elFCL+pSLymhSQNdlNBFo7K3LMHLBrwqoasGmpbQtIFel9Drps/YjMXlcKzWe5FvqjqmwQJecoBIq/5Q13vhZr8vnaokmwHVd3Ru9wSm5nxbA2Ga0dWrs9e/aHWwP3xi7+o6w6MJbCj2zu6TA7tB8derKTj2/v7wRYMTcxz5N4UOj3Z/cJqFWMIZvSHt7e38+LRZKQVK4+VNpYMXh8Od+hxx0jCh6BX1HdQwzW+jF121Crw2wdqpVv68yX/Jcfov7Lit+sbAVgVrU2zcbFWkbYqNtRtFrQmWTevcXBksO9cxXVMOwZz4HF6bKT42pxSWMf/OjC73w8DYZ97uIIv+i4hXG6KJej1z/Tj1y6YZjIbbT7edX7/vP/+puIfuWl9ZX1uPLMfas55br6wTa2SRzuPOqPOuc9X90P29+2f3rzX1VqfQfGlVnu7f/wCMhv0l</latexit><latexit sha1_base64="TEn9cwN6ZDLtIN8rQ1vBPZO/Syk=">AAAIK3icjVVdb9s2FJW7ru68dU23x70QM4Z2g5FKzpqkDwW6JlmLbW2yIE4KRG5K0deyYEoiSMq2QvAv9X/sfS8b9rr/MUqWA31tmF50dc85V7yHF6THaCCkbf/RufXR7Y/vdO9+0vv0s3uf39968MW5iBNOYERiGvO3HhZAgwhGMpAU3jIOOPQoXHjzgwy/WAAXQRydyZTBOMR+FEwDgqVJXW39xh65nChXwkoqxoNr0NoNgwly/U0Waz1Abrz5XMaR1t8iFzPG4xVypxwThdw+WsPv3+sqGWldLpZ/1v9YaB8+1Ahp9b9rlYX6aqtvb9v5g5qBUwR9q3hOrh7c+dmdxCQJIZKEYiEuHZvJscJcBoSC7rmJAIbJHPtwmcjp/lgFEUskRESjbww2TSiSMcpsRZOAA5E0NQEmPDAVEJlhY4005veqpQREOAQxmCwCJtahWPjrQGKzc2O1yndW36solc8xmwVkVVmawqEIsZw1kiINvWoSEgp8EVaT2TLNImvMFXASiMyEE+PMMcumRZzFJwU+S9kMIqFVwqkuCw0AnMPUCPNQgEyYyrsxIzoXzyRPYJCFee7ZIebzU5gMTJ1KorqcKY2xrKY804ZxJ4IlicMQRxPlMl0MhjvY1rl3ZfRUK+VmRnkeOs3gCvqmhL7RugoelcAjA1bR0Q06RaO69LwEnjf+elFCL+pSLymhSQNdlNBFo7K3LMHLBrwqoasGmpbQtIFel9Drps/YjMXlcKzWe5FvqjqmwQJecoBIq/5Q13vhZr8vnaokmwHVd3Ru9wSm5nxbA2Ga0dWrs9e/aHWwP3xi7+o6w6MJbCj2zu6TA7tB8derKTj2/v7wRYMTcxz5N4UOj3Z/cJqFWMIZvSHt7e38+LRZKQVK4+VNpYMXh8Od+hxx0jCh6BX1HdQwzW+jF121Crw2wdqpVv68yX/Jcfov7Lit+sbAVgVrU2zcbFWkbYqNtRtFrQmWTevcXBksO9cxXVMOwZz4HF6bKT42pxSWMf/OjC73w8DYZ97uIIv+i4hXG6KJej1z/Tj1y6YZjIbbT7edX7/vP/+puIfuWl9ZX1uPLMfas55br6wTa2SRzuPOqPOuc9X90P29+2f3rzX1VqfQfGlVnu7f/wCMhv0l</latexit><latexit sha1_base64="TEn9cwN6ZDLtIN8rQ1vBPZO/Syk=">AAAIK3icjVVdb9s2FJW7ru68dU23x70QM4Z2g5FKzpqkDwW6JlmLbW2yIE4KRG5K0deyYEoiSMq2QvAv9X/sfS8b9rr/MUqWA31tmF50dc85V7yHF6THaCCkbf/RufXR7Y/vdO9+0vv0s3uf39968MW5iBNOYERiGvO3HhZAgwhGMpAU3jIOOPQoXHjzgwy/WAAXQRydyZTBOMR+FEwDgqVJXW39xh65nChXwkoqxoNr0NoNgwly/U0Waz1Abrz5XMaR1t8iFzPG4xVypxwThdw+WsPv3+sqGWldLpZ/1v9YaB8+1Ahp9b9rlYX6aqtvb9v5g5qBUwR9q3hOrh7c+dmdxCQJIZKEYiEuHZvJscJcBoSC7rmJAIbJHPtwmcjp/lgFEUskRESjbww2TSiSMcpsRZOAA5E0NQEmPDAVEJlhY4005veqpQREOAQxmCwCJtahWPjrQGKzc2O1yndW36solc8xmwVkVVmawqEIsZw1kiINvWoSEgp8EVaT2TLNImvMFXASiMyEE+PMMcumRZzFJwU+S9kMIqFVwqkuCw0AnMPUCPNQgEyYyrsxIzoXzyRPYJCFee7ZIebzU5gMTJ1KorqcKY2xrKY804ZxJ4IlicMQRxPlMl0MhjvY1rl3ZfRUK+VmRnkeOs3gCvqmhL7RugoelcAjA1bR0Q06RaO69LwEnjf+elFCL+pSLymhSQNdlNBFo7K3LMHLBrwqoasGmpbQtIFel9Drps/YjMXlcKzWe5FvqjqmwQJecoBIq/5Q13vhZr8vnaokmwHVd3Ru9wSm5nxbA2Ga0dWrs9e/aHWwP3xi7+o6w6MJbCj2zu6TA7tB8derKTj2/v7wRYMTcxz5N4UOj3Z/cJqFWMIZvSHt7e38+LRZKQVK4+VNpYMXh8Od+hxx0jCh6BX1HdQwzW+jF121Crw2wdqpVv68yX/Jcfov7Lit+sbAVgVrU2zcbFWkbYqNtRtFrQmWTevcXBksO9cxXVMOwZz4HF6bKT42pxSWMf/OjC73w8DYZ97uIIv+i4hXG6KJej1z/Tj1y6YZjIbbT7edX7/vP/+puIfuWl9ZX1uPLMfas55br6wTa2SRzuPOqPOuc9X90P29+2f3rzX1VqfQfGlVnu7f/wCMhv0l</latexit><latexit sha1_base64="TEn9cwN6ZDLtIN8rQ1vBPZO/Syk=">AAAIK3icjVVdb9s2FJW7ru68dU23x70QM4Z2g5FKzpqkDwW6JlmLbW2yIE4KRG5K0deyYEoiSMq2QvAv9X/sfS8b9rr/MUqWA31tmF50dc85V7yHF6THaCCkbf/RufXR7Y/vdO9+0vv0s3uf39968MW5iBNOYERiGvO3HhZAgwhGMpAU3jIOOPQoXHjzgwy/WAAXQRydyZTBOMR+FEwDgqVJXW39xh65nChXwkoqxoNr0NoNgwly/U0Waz1Abrz5XMaR1t8iFzPG4xVypxwThdw+WsPv3+sqGWldLpZ/1v9YaB8+1Ahp9b9rlYX6aqtvb9v5g5qBUwR9q3hOrh7c+dmdxCQJIZKEYiEuHZvJscJcBoSC7rmJAIbJHPtwmcjp/lgFEUskRESjbww2TSiSMcpsRZOAA5E0NQEmPDAVEJlhY4005veqpQREOAQxmCwCJtahWPjrQGKzc2O1yndW36solc8xmwVkVVmawqEIsZw1kiINvWoSEgp8EVaT2TLNImvMFXASiMyEE+PMMcumRZzFJwU+S9kMIqFVwqkuCw0AnMPUCPNQgEyYyrsxIzoXzyRPYJCFee7ZIebzU5gMTJ1KorqcKY2xrKY804ZxJ4IlicMQRxPlMl0MhjvY1rl3ZfRUK+VmRnkeOs3gCvqmhL7RugoelcAjA1bR0Q06RaO69LwEnjf+elFCL+pSLymhSQNdlNBFo7K3LMHLBrwqoasGmpbQtIFel9Drps/YjMXlcKzWe5FvqjqmwQJecoBIq/5Q13vhZr8vnaokmwHVd3Ru9wSm5nxbA2Ga0dWrs9e/aHWwP3xi7+o6w6MJbCj2zu6TA7tB8derKTj2/v7wRYMTcxz5N4UOj3Z/cJqFWMIZvSHt7e38+LRZKQVK4+VNpYMXh8Od+hxx0jCh6BX1HdQwzW+jF121Crw2wdqpVv68yX/Jcfov7Lit+sbAVgVrU2zcbFWkbYqNtRtFrQmWTevcXBksO9cxXVMOwZz4HF6bKT42pxSWMf/OjC73w8DYZ97uIIv+i4hXG6KJej1z/Tj1y6YZjIbbT7edX7/vP/+puIfuWl9ZX1uPLMfas55br6wTa2SRzuPOqPOuc9X90P29+2f3rzX1VqfQfGlVnu7f/wCMhv0l</latexit>











2nd order Markov model



Using the Markov assumption and the chain rule together, we can model a sequence as limited-memory conditional probabilities. These probabilities can then be very simply estimated from a large dataset of text (called a corpus).

To estimate the probability of prize given “won a” we just count how often “won a prize” occurs as a proportion of the times “won a” occurs. In other words, how often “won a” is followed by prize.

These n-word snippets are called n-grams. “won a prize” is a trigram, and “won a” is a bigram.

This type of language model is often called a Markov model, because of the Markov assumption of limited memory. The size of the memory is referred to as the order of the Markov model. The higher the order of your model, the more you can model,

but the more data you’ll need for accurate estimates if the probability.

sequential sampling from a language model

start with a small seed sequence s = [w1, w2, w3] of tokens.

loop:

Sample next word w according to p(W = w | w1, w2, …)

append w to s

21

One interesting thing we can do with a Markov model, is to sample from it, step by step. We start with a seed of a few words, and then work out the probability distribution over the next word, given the last n words. We sample from this distribution, append the sample to our text and repeat the process.

Note that with the Markov assumption, we only need the last n elements of the sequence to work out the probabilities.

Sequential sampling is also known as autoregressive

sampling. In the context of Markov models, the sampling process is often called a Markov chain.

sequential sampling: Markov chain

22source http://www.schmipsum.com/

Here is is a bit of text sampled from a Markov model trained on the works of Shakespeare.

(We’re essentially training a very simple probabilistic predictor in the fashion of slide 9, and then sampling from it sequentially)

sequence classification

23

p(spam | “congratulations, you have won a prize”)= p(“congratulations, you have won a prize” | spam)p(spam)

<latexit sha1_base64="6ttO1rJEyiT+M9E6UG9tahnsDVc=">AAAIDHiclVVdb9s2FJW7rW69dUu3x70INdY2gxFIzpqkD0HTfKwFtjZZECcFIiOl6GtbMCURJGVbIfgb9j/2vqdhr/sP+zejPhxIojdgfNHFPedcXR5ekD4lAReO83fr3ieffna//eBh5/MvHn351cbjry95nDAMAxyTmH3wEQcSRDAQgSDwgTJAoU/gyp8dZfjVHBgP4uhCpBSGIZpEwTjASOjUzcZvT+lzj2HpCVgKySkKlfLCYGQXiY8fcRxNGBIJyQW8Z6dxYk/RHOxFHNnIpiy4hWfP1KbndZ7u27ra/xQWf2u0sGl2tXmz0XW2nHzZZuCWQdcq19nN4/s/eaMYJyFEAhPE+bXrUDGUiIkAE1AdL+FAEZ6hCVwnYrw3lEFEEwERVvZ3GhsnxBaxnflmjwIGWJBUBwizQFew8RQxhIV2t1MvxSFCIfDeaB5QXoR8PikCgfTRDOUyPzr1qKaU2jA6DfCy1ppEIQ+RmBpJnoZ+PQkJATYP68msTd1kg7kEhgOemXCmnTml+SFdxGclPk3pFCKuZMKIqgo1AIzBWAvzkINIqMx3o2dwxvcFS6CXhXlu/xix2TmMerpOLVFvZ0xiJOopX29DuxPBAsdhiKKR9KgqJ8Lrbancuyp6rqT0MqN83z7P4Br6voK+V6oOnlTAEw3W0cEdOrYHTellBbw0/npVQa+aUj+poImBzivo3KjsLyrwwoCXFXRpoGkFTQ30toLemj4jPRbX/aEsziI/VHlKgjm8YQCRkt2+au6F6fO+duuSbAZk11W53SMY6wusAMI0o8u3F+9+VvJor//C2VFNhk8SWFGc7Z0XR45BmRTdlBxnb69/aHBihqLJXaHjk53XrlmIJoySO9Lu7vaPL81KKRASL+4qHR0e97ebc8SwYUK5V7vr2oZpk3X0cldrBf46QeHUWv7M5L9hKP0Xdryu+srAtQq6TrFyc60iXadYWbtSNDZBs2md6beCZvc6IgXlGPSNz+CdnuJTfUshEbPv9eiySRho+/TX62XRfxHRckXUUaejnx+3+diYwaC/9XLL/eWH7sFh+Q49sL61nljPLdfatQ6st9aZNbBwa6O103rVOmj/2v69/Uf7z4J6r1VqvrFqq/3XP8g67aI=</latexit><latexit sha1_base64="6ttO1rJEyiT+M9E6UG9tahnsDVc=">AAAIDHiclVVdb9s2FJW7rW69dUu3x70INdY2gxFIzpqkD0HTfKwFtjZZECcFIiOl6GtbMCURJGVbIfgb9j/2vqdhr/sP+zejPhxIojdgfNHFPedcXR5ekD4lAReO83fr3ieffna//eBh5/MvHn351cbjry95nDAMAxyTmH3wEQcSRDAQgSDwgTJAoU/gyp8dZfjVHBgP4uhCpBSGIZpEwTjASOjUzcZvT+lzj2HpCVgKySkKlfLCYGQXiY8fcRxNGBIJyQW8Z6dxYk/RHOxFHNnIpiy4hWfP1KbndZ7u27ra/xQWf2u0sGl2tXmz0XW2nHzZZuCWQdcq19nN4/s/eaMYJyFEAhPE+bXrUDGUiIkAE1AdL+FAEZ6hCVwnYrw3lEFEEwERVvZ3GhsnxBaxnflmjwIGWJBUBwizQFew8RQxhIV2t1MvxSFCIfDeaB5QXoR8PikCgfTRDOUyPzr1qKaU2jA6DfCy1ppEIQ+RmBpJnoZ+PQkJATYP68msTd1kg7kEhgOemXCmnTml+SFdxGclPk3pFCKuZMKIqgo1AIzBWAvzkINIqMx3o2dwxvcFS6CXhXlu/xix2TmMerpOLVFvZ0xiJOopX29DuxPBAsdhiKKR9KgqJ8Lrbancuyp6rqT0MqN83z7P4Br6voK+V6oOnlTAEw3W0cEdOrYHTellBbw0/npVQa+aUj+poImBzivo3KjsLyrwwoCXFXRpoGkFTQ30toLemj4jPRbX/aEsziI/VHlKgjm8YQCRkt2+au6F6fO+duuSbAZk11W53SMY6wusAMI0o8u3F+9+VvJor//C2VFNhk8SWFGc7Z0XR45BmRTdlBxnb69/aHBihqLJXaHjk53XrlmIJoySO9Lu7vaPL81KKRASL+4qHR0e97ebc8SwYUK5V7vr2oZpk3X0cldrBf46QeHUWv7M5L9hKP0Xdryu+srAtQq6TrFyc60iXadYWbtSNDZBs2md6beCZvc6IgXlGPSNz+CdnuJTfUshEbPv9eiySRho+/TX62XRfxHRckXUUaejnx+3+diYwaC/9XLL/eWH7sFh+Q49sL61nljPLdfatQ6st9aZNbBwa6O103rVOmj/2v69/Uf7z4J6r1VqvrFqq/3XP8g67aI=</latexit><latexit sha1_base64="6ttO1rJEyiT+M9E6UG9tahnsDVc=">AAAIDHiclVVdb9s2FJW7rW69dUu3x70INdY2gxFIzpqkD0HTfKwFtjZZECcFIiOl6GtbMCURJGVbIfgb9j/2vqdhr/sP+zejPhxIojdgfNHFPedcXR5ekD4lAReO83fr3ieffna//eBh5/MvHn351cbjry95nDAMAxyTmH3wEQcSRDAQgSDwgTJAoU/gyp8dZfjVHBgP4uhCpBSGIZpEwTjASOjUzcZvT+lzj2HpCVgKySkKlfLCYGQXiY8fcRxNGBIJyQW8Z6dxYk/RHOxFHNnIpiy4hWfP1KbndZ7u27ra/xQWf2u0sGl2tXmz0XW2nHzZZuCWQdcq19nN4/s/eaMYJyFEAhPE+bXrUDGUiIkAE1AdL+FAEZ6hCVwnYrw3lEFEEwERVvZ3GhsnxBaxnflmjwIGWJBUBwizQFew8RQxhIV2t1MvxSFCIfDeaB5QXoR8PikCgfTRDOUyPzr1qKaU2jA6DfCy1ppEIQ+RmBpJnoZ+PQkJATYP68msTd1kg7kEhgOemXCmnTml+SFdxGclPk3pFCKuZMKIqgo1AIzBWAvzkINIqMx3o2dwxvcFS6CXhXlu/xix2TmMerpOLVFvZ0xiJOopX29DuxPBAsdhiKKR9KgqJ8Lrbancuyp6rqT0MqN83z7P4Br6voK+V6oOnlTAEw3W0cEdOrYHTellBbw0/npVQa+aUj+poImBzivo3KjsLyrwwoCXFXRpoGkFTQ30toLemj4jPRbX/aEsziI/VHlKgjm8YQCRkt2+au6F6fO+duuSbAZk11W53SMY6wusAMI0o8u3F+9+VvJor//C2VFNhk8SWFGc7Z0XR45BmRTdlBxnb69/aHBihqLJXaHjk53XrlmIJoySO9Lu7vaPL81KKRASL+4qHR0e97ebc8SwYUK5V7vr2oZpk3X0cldrBf46QeHUWv7M5L9hKP0Xdryu+srAtQq6TrFyc60iXadYWbtSNDZBs2md6beCZvc6IgXlGPSNz+CdnuJTfUshEbPv9eiySRho+/TX62XRfxHRckXUUaejnx+3+diYwaC/9XLL/eWH7sFh+Q49sL61nljPLdfatQ6st9aZNbBwa6O103rVOmj/2v69/Uf7z4J6r1VqvrFqq/3XP8g67aI=</latexit><latexit sha1_base64="6ttO1rJEyiT+M9E6UG9tahnsDVc=">AAAIDHiclVVdb9s2FJW7rW69dUu3x70INdY2gxFIzpqkD0HTfKwFtjZZECcFIiOl6GtbMCURJGVbIfgb9j/2vqdhr/sP+zejPhxIojdgfNHFPedcXR5ekD4lAReO83fr3ieffna//eBh5/MvHn351cbjry95nDAMAxyTmH3wEQcSRDAQgSDwgTJAoU/gyp8dZfjVHBgP4uhCpBSGIZpEwTjASOjUzcZvT+lzj2HpCVgKySkKlfLCYGQXiY8fcRxNGBIJyQW8Z6dxYk/RHOxFHNnIpiy4hWfP1KbndZ7u27ra/xQWf2u0sGl2tXmz0XW2nHzZZuCWQdcq19nN4/s/eaMYJyFEAhPE+bXrUDGUiIkAE1AdL+FAEZ6hCVwnYrw3lEFEEwERVvZ3GhsnxBaxnflmjwIGWJBUBwizQFew8RQxhIV2t1MvxSFCIfDeaB5QXoR8PikCgfTRDOUyPzr1qKaU2jA6DfCy1ppEIQ+RmBpJnoZ+PQkJATYP68msTd1kg7kEhgOemXCmnTml+SFdxGclPk3pFCKuZMKIqgo1AIzBWAvzkINIqMx3o2dwxvcFS6CXhXlu/xix2TmMerpOLVFvZ0xiJOopX29DuxPBAsdhiKKR9KgqJ8Lrbancuyp6rqT0MqN83z7P4Br6voK+V6oOnlTAEw3W0cEdOrYHTellBbw0/npVQa+aUj+poImBzivo3KjsLyrwwoCXFXRpoGkFTQ30toLemj4jPRbX/aEsziI/VHlKgjm8YQCRkt2+au6F6fO+duuSbAZk11W53SMY6wusAMI0o8u3F+9+VvJor//C2VFNhk8SWFGc7Z0XR45BmRTdlBxnb69/aHBihqLJXaHjk53XrlmIJoySO9Lu7vaPL81KKRASL+4qHR0e97ebc8SwYUK5V7vr2oZpk3X0cldrBf46QeHUWv7M5L9hKP0Xdryu+srAtQq6TrFyc60iXadYWbtSNDZBs2md6beCZvc6IgXlGPSNz+CdnuJTfUshEbPv9eiySRho+/TX62XRfxHRckXUUaejnx+3+diYwaC/9XLL/eWH7sFh+Q49sL61nljPLdfatQ6st9aZNbBwa6O103rVOmj/2v69/Uf7z4J6r1VqvrFqq/3XP8g67aI=</latexit>

Let’s now look at a sequence-per-instance example: the spam classiBication task we saw earlier. We’ll see how to approach this with a Markov model.

We are after the probability of the class, given the contents of the message.

Bayes classifier

24

p(spam | W1, . . . ,Wn) / p(W1, . . . ,Wn | spam)p(spam)<latexit sha1_base64="buqRjVcb3kyzY+gE+5YiQLpjFtw=">AAAH0XicfVVBb9s2FFaatk69dk23Yy9CjQHJYASS0ybZoUDXJG2xrU2axXGAyAgo+tkWTEkESTlWCQJFr/17veyv7DTKkgNJ1MqLn973fc98Hx9In5KAC8f5Z+3O+t1791sbD9o/PHz04+PNJz9d8DhhGPo4JjG79BEHEkTQF4EgcEkZoNAnMPBnhxk+mAPjQRydi5TCMESTKBgHGAmdut6M6JbHsPQELITkFIVK2V4YjOzBtdu1PTKKBe/qj2jb9iiLqYhtulXHckW9zrZZevt6s+PsOMtlm4FbBB2rWKfXT+7/6Y1inIQQCUwQ51euQ8VQIiYCTEC1vYQDRXiGJnCViPHBUAYRTQREWNm/aGycEFtvOuvdHgUMsCCpDhBmga5g4yliCAvtULtaikOEQuDd0TygPA/5fJIHAml7h3KxtF89qijlhCE6DfCisjWJQh4iMTWSPA39ahISAmweVpPZNvUma8wFMBzwzIRT7cwJzY6Un8enBT5N6RQirmTCiCoLNQCMwVgLlyEHkVC57EbP0Yy/FCyBbhYucy+PEJudwair61QS1e2MSYxENeXrNrQ7EdzgOAxRNJIeVcVEeN0dtfSujJ4pKb3MKN+3zzK4gn4ooR+UqoLHJfBYg1W0f4uO7X5delECL4x/HZTQQV3qJyU0MdB5CZ0blf2bEnxjwIsSujDQtISmBvqphH4yfUZ6LK56Q5mfxfJQ5QkJ5vCWAURKdnqq3gvT533lViXZDMiOq5Z2j2CsL6EcCNOMLt+dv/9LycOD3gtnT9UZPklgRXF2914cOgZlku+m4DgHB73XBidmKJrcFjo63vvdNQvRhFFyS9rf333zm1kpBULim9tKh6+Perv1OWLYMKHo1e64tmHapIledNUo8JsEuVON/JnJf8tQ+j/suKn6ysBGBW1SrNxsVKRNipW1K0WtCZpN60y/FTS71xHJKUegb3wG7/UUn+hbComY/apHl03CQNunf71uFn2PiBYroo7abf38uPXHxgwuejuus+N+fN559UfxEG1YT61n1pblWvvWK+uddWr1LWx9s/5dW1+72/q7lbY+t77k1DtrheZnq7JaX/8DParVew==</latexit><latexit sha1_base64="buqRjVcb3kyzY+gE+5YiQLpjFtw=">AAAH0XicfVVBb9s2FFaatk69dk23Yy9CjQHJYASS0ybZoUDXJG2xrU2axXGAyAgo+tkWTEkESTlWCQJFr/17veyv7DTKkgNJ1MqLn973fc98Hx9In5KAC8f5Z+3O+t1791sbD9o/PHz04+PNJz9d8DhhGPo4JjG79BEHEkTQF4EgcEkZoNAnMPBnhxk+mAPjQRydi5TCMESTKBgHGAmdut6M6JbHsPQELITkFIVK2V4YjOzBtdu1PTKKBe/qj2jb9iiLqYhtulXHckW9zrZZevt6s+PsOMtlm4FbBB2rWKfXT+7/6Y1inIQQCUwQ51euQ8VQIiYCTEC1vYQDRXiGJnCViPHBUAYRTQREWNm/aGycEFtvOuvdHgUMsCCpDhBmga5g4yliCAvtULtaikOEQuDd0TygPA/5fJIHAml7h3KxtF89qijlhCE6DfCisjWJQh4iMTWSPA39ahISAmweVpPZNvUma8wFMBzwzIRT7cwJzY6Un8enBT5N6RQirmTCiCoLNQCMwVgLlyEHkVC57EbP0Yy/FCyBbhYucy+PEJudwair61QS1e2MSYxENeXrNrQ7EdzgOAxRNJIeVcVEeN0dtfSujJ4pKb3MKN+3zzK4gn4ooR+UqoLHJfBYg1W0f4uO7X5delECL4x/HZTQQV3qJyU0MdB5CZ0blf2bEnxjwIsSujDQtISmBvqphH4yfUZ6LK56Q5mfxfJQ5QkJ5vCWAURKdnqq3gvT533lViXZDMiOq5Z2j2CsL6EcCNOMLt+dv/9LycOD3gtnT9UZPklgRXF2914cOgZlku+m4DgHB73XBidmKJrcFjo63vvdNQvRhFFyS9rf333zm1kpBULim9tKh6+Perv1OWLYMKHo1e64tmHapIledNUo8JsEuVON/JnJf8tQ+j/suKn6ysBGBW1SrNxsVKRNipW1K0WtCZpN60y/FTS71xHJKUegb3wG7/UUn+hbComY/apHl03CQNunf71uFn2PiBYroo7abf38uPXHxgwuejuus+N+fN559UfxEG1YT61n1pblWvvWK+uddWr1LWx9s/5dW1+72/q7lbY+t77k1DtrheZnq7JaX/8DParVew==</latexit><latexit sha1_base64="buqRjVcb3kyzY+gE+5YiQLpjFtw=">AAAH0XicfVVBb9s2FFaatk69dk23Yy9CjQHJYASS0ybZoUDXJG2xrU2axXGAyAgo+tkWTEkESTlWCQJFr/17veyv7DTKkgNJ1MqLn973fc98Hx9In5KAC8f5Z+3O+t1791sbD9o/PHz04+PNJz9d8DhhGPo4JjG79BEHEkTQF4EgcEkZoNAnMPBnhxk+mAPjQRydi5TCMESTKBgHGAmdut6M6JbHsPQELITkFIVK2V4YjOzBtdu1PTKKBe/qj2jb9iiLqYhtulXHckW9zrZZevt6s+PsOMtlm4FbBB2rWKfXT+7/6Y1inIQQCUwQ51euQ8VQIiYCTEC1vYQDRXiGJnCViPHBUAYRTQREWNm/aGycEFtvOuvdHgUMsCCpDhBmga5g4yliCAvtULtaikOEQuDd0TygPA/5fJIHAml7h3KxtF89qijlhCE6DfCisjWJQh4iMTWSPA39ahISAmweVpPZNvUma8wFMBzwzIRT7cwJzY6Un8enBT5N6RQirmTCiCoLNQCMwVgLlyEHkVC57EbP0Yy/FCyBbhYucy+PEJudwair61QS1e2MSYxENeXrNrQ7EdzgOAxRNJIeVcVEeN0dtfSujJ4pKb3MKN+3zzK4gn4ooR+UqoLHJfBYg1W0f4uO7X5delECL4x/HZTQQV3qJyU0MdB5CZ0blf2bEnxjwIsSujDQtISmBvqphH4yfUZ6LK56Q5mfxfJQ5QkJ5vCWAURKdnqq3gvT533lViXZDMiOq5Z2j2CsL6EcCNOMLt+dv/9LycOD3gtnT9UZPklgRXF2914cOgZlku+m4DgHB73XBidmKJrcFjo63vvdNQvRhFFyS9rf333zm1kpBULim9tKh6+Perv1OWLYMKHo1e64tmHapIledNUo8JsEuVON/JnJf8tQ+j/suKn6ysBGBW1SrNxsVKRNipW1K0WtCZpN60y/FTS71xHJKUegb3wG7/UUn+hbComY/apHl03CQNunf71uFn2PiBYroo7abf38uPXHxgwuejuus+N+fN559UfxEG1YT61n1pblWvvWK+uddWr1LWx9s/5dW1+72/q7lbY+t77k1DtrheZnq7JaX/8DParVew==</latexit><latexit sha1_base64="buqRjVcb3kyzY+gE+5YiQLpjFtw=">AAAH0XicfVVBb9s2FFaatk69dk23Yy9CjQHJYASS0ybZoUDXJG2xrU2axXGAyAgo+tkWTEkESTlWCQJFr/17veyv7DTKkgNJ1MqLn973fc98Hx9In5KAC8f5Z+3O+t1791sbD9o/PHz04+PNJz9d8DhhGPo4JjG79BEHEkTQF4EgcEkZoNAnMPBnhxk+mAPjQRydi5TCMESTKBgHGAmdut6M6JbHsPQELITkFIVK2V4YjOzBtdu1PTKKBe/qj2jb9iiLqYhtulXHckW9zrZZevt6s+PsOMtlm4FbBB2rWKfXT+7/6Y1inIQQCUwQ51euQ8VQIiYCTEC1vYQDRXiGJnCViPHBUAYRTQREWNm/aGycEFtvOuvdHgUMsCCpDhBmga5g4yliCAvtULtaikOEQuDd0TygPA/5fJIHAml7h3KxtF89qijlhCE6DfCisjWJQh4iMTWSPA39ahISAmweVpPZNvUma8wFMBzwzIRT7cwJzY6Un8enBT5N6RQirmTCiCoLNQCMwVgLlyEHkVC57EbP0Yy/FCyBbhYucy+PEJudwair61QS1e2MSYxENeXrNrQ7EdzgOAxRNJIeVcVEeN0dtfSujJ4pKb3MKN+3zzK4gn4ooR+UqoLHJfBYg1W0f4uO7X5delECL4x/HZTQQV3qJyU0MdB5CZ0blf2bEnxjwIsSujDQtISmBvqphH4yfUZ6LK56Q5mfxfJQ5QkJ5vCWAURKdnqq3gvT533lViXZDMiOq5Z2j2CsL6EcCNOMLt+dv/9LycOD3gtnT9UZPklgRXF2914cOgZlku+m4DgHB73XBidmKJrcFjo63vvdNQvRhFFyS9rf333zm1kpBULim9tKh6+Perv1OWLYMKHo1e64tmHapIledNUo8JsEuVON/JnJf8tQ+j/suKn6ysBGBW1SrNxsVKRNipW1K0WtCZpN60y/FTS71xHJKUegb3wG7/UUn+hbComY/apHl03CQNunf71uFn2PiBYroo7abf38uPXHxgwuejuus+N+fN559UfxEG1YT61n1pblWvvWK+uddWr1LWx9s/5dW1+72/q7lbY+t77k1DtrheZnq7JaX/8DParVew==</latexit>

We’ll train a generative/Bayes classiBier and use Bayes rule to Blip around the probabilties.

p(spam) can be estimated as the proportion of spam emails in our data set. For the probability of the message given the class, we’ll use our language model.

conditional on class

25

p(prize, a,won,have, you, congratulations | spam)

=p(prize | a,won, spam) p(a | won,have, spam)

p(won | have, you, spam) p(have | you, congratulations, spam)

p(you | congratulations, spam) p(congratulations, spam)<latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit><latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit><latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit><latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit>

p(prize | a,won, spam) ⇡#spam“won a prize”

#spam“won a”<latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit><latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit><latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit><latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit>

p(prize, a,won,have, you, congratulations | spam)

=p(prize | a,won, spam) p(a | won,have, spam)

p(won | have, you, spam) p(have | you, congratulations, spam)

p(you | congratulations, spam) p(congratulations, spam)<latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit><latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit><latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit><latexit sha1_base64="jyDCx02ztpxdlJTNV24KeBq0u+E=">AAAJTXicjVVdb9s2FFW8rnW9dWu2x74INVZkgxFIzpqkGAK0TbIW29qkQZwUiIyAomlbMCURJGVbIQhs/3Cv+wv7AXsahlEfdilRDaoXX/Occ+/l4ZXoExww7jh/bbQ+u/P53Xvt+50vvnzw1dcPN7+5YHFCIRrAGMf0vQ8YwkGEBjzgGL0nFIHQx+jSnx1m+OUcURbE0TlPCRqGYBIF4wACrpauNzf+JltPPI6WXBAa3CDZs4t/YB0t4mgdT8H8AyWNk3UM42hCAU9wnpdJLwxGtkehKGBGQCjl957XObCf2GRLr1hQm4vWE9i29xPZWrM1aVOXDeXtvLymyVJ8fG9G/ay8RtcauM2N23r5oNaSfUKCihWfVPD6YdfZdvLHNgO3DLpW+Zxeb9791RvFMAlRxCEGjF25DuFDASgPIEay4yUMEQBnYIKuEj7eH4ogIglHEZT2dwobJ9jmsZ0Nnj0KKIIcpyoAkAYqgw2ngALI1Xh2qqkYikCIWG80DwgrQjafFAEHaraHYpnPvnxQUQplApkGcFlpTYCQhYBPjUWWhn51ESUY0XlYXczaVE3WmEtEYcAyE06VMyckN/48Pi3xaUqmSJ2ESCiWulABiFI0VsI8ZIgnROS7US/xjB1wmqBeFuZrB0eAzs7QqKfyVBaq7YxxDHh1yVfbUO5EaAHjMATRSHhEljPh9bZl7p2OnkkhvMwo37fPMriCvtXQt1JWwWMNPFZgFR2s0bE9qEsvNPDCqHqpoZd1qZ9oaGKgcw2dG5n9hQYvDHipoUsDTTU0NdAbDb0xfQZqLK76Q1GcRX6o4gQHc/SKIhRJ0e3L+l6oOu8rtyrJZkB0XZnbPUJjdQMUQJhmdPH6/M1vUhzu9586u7LO8HGCVhRnZ/fpoWNQJkU3JcfZ3++/NDgxBdFknejoePeFayYiCSV4Tdrb2/n5mZkpRRjHi3Wmw5dH/Z36HFFomFDu1e66tmHapIle7qpR4DcJCqca+TOT/4qC9CPsuCn7ysBGBWlSrNxsVKRNipW1K0VtEySb1pm6LUj2XQe4oBwh9cWn6I2a4hP1lQI8pj+o0aWTMFD2qV+vl0W3EcFyRVRRp6OuH7d+2ZjBoL/9bNt992P3+S/lPdS2HlmPrS3Ltfas59Zr69QaWLD1rrVo/d76o/1n+5/2v+3/Cmpro9R8a1We+/f+B90ybGM=</latexit>

p(prize | a,won, spam) ⇡#spam“won a prize”

#spam“won a”<latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit><latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit><latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit><latexit sha1_base64="Bir0Qe5+wc+6uKICqKlUhXw94aw=">AAAIGHicjVXLbhs3FB2lTZSoSes0y3QxiBAkKQRjRm5sZxEg8aMJmoddw7IDeASHM7qSCHFmCJIjaUzwS/If2WdVdNtd/6ach4R5JSg3urrnnDu8hxekSwnmwrL+bV377vvrN9o3b3V+uH3nx5827v58xsOIeTDwQhKyDy7iQHAAA4EFgQ+UAfJdAufubD/Bz+fAOA6DUxFTGPpoEuAx9pDQqcuNz/SxI2ApJGX4CpTj45GZJZDq5dEiDJKYeTL7zynylXpiOohSFi5NZ8yQxrqXVUqu//hRrQuZyMy/lGYePVLqfysLmsuNrrVppcusB3YedI18HV/evfHGGYVe5EMgPII4v7AtKoYSMYE9AqrjRBwo8mZoAheRGO8OJQ5oJCDwlPlQY+OImCI0Ew/NEWbgCRLrAHkM6wqmN0XaBKGd7pRLcQiQD7w3mmPKs5DPJ1kgkD6moVymx6julJRywhCdYm9Z2ppEPveRmNaSPPbdchIiAmzul5PJNvUmK8wlMA/zxIRj7cwRTUaDn4bHOT6N6RQCrmTEiCoKNQCMwVgL05CDiKhMu9HzOOPPBYugl4Rp7vkBYrMTGPV0nVKivJ0xCZEop1zdhnYngIUX+j4KRtKhKh8Yp7epUu+K6ImS0kmMcl3zJIFL6PsC+l6pMnhYAA81WEYHa3RsDqrSswJ4VvvqeQE9r0rdqIBGNXReQOe1yu6iAC9q8LKALmtoXEDjGnpVQK/qPiM9Fhf9oczOIj1UeUTwHF4xgEDJbl9Ve2H6vC/ssiSZAdm1VWr3CMb6MssAP07o8vXpu7dK7u/2n1rbqspwSQQrirW1/XTfqlEm2W5yjrW729+rcUKGgsm60MHh9ku7XohGjJI1aWdn6/dn9UoxEBIu1pX29w76W9U5Yl7NhLxXs2ubNdMmTfS8q0aB2yTInGrkz+r8VwzFX2GHTdVXBjYqaJNi5WajIm5SrKxdKSpN0GRaZ/opocm9jkhGOQB94zN4p6f4SN9SSITsVz26bOJjbZ/+dXpJ9C0iWq6IOup09PNjVx+bejDobz7btP/8rfvij/wdumncNx4Yjw3b2DFeGK+NY2NgeK1fWnutN6237U/tL+2/2n9n1GutXHPPKK32P/8Brqz2oQ==</latexit>

We use the chain rule and the Markov assumption to deBine the probability that a message occurs (given that it’s spam).

We then estimate the different conditional probabilities by computing the relative frequencies of bigrams and trigrams, as before, but we compute them only over the spam part of our data.

algorithm: 2nd-order markov model

training:

for each class c:

for n in 1, 2, 3:

count n-grams in all text belonging to class c

26

classification:

given text w1, …, wm

p(c | w1, . . . ,wm) / p(w1, . . . ,wm | c)p(c)

p(w1, . . . ,wm | c) =

p(w1 | c) p(w2 | w1, c)Y

i2[3,...,m]

p(wi | wi-1,wi-2, c)

<latexit sha1_base64="3gvS/EvBT8T3/6iR30SYefI41AI=">AAAIT3ichVVbb9s2FJZ7S+q1a9o99kWYsSIdvEBymstQBOhyWQtsbbIgTgpYhkFRtE2YkgiSsq0SfOj+4R73H/bet2LUxa5u3fjio/N93/HhxwPSpQRzYVl/t27dvnP33sbm/fY3Dx5++2jr8ZNrHkYMoj4MScjeu4AjggPUF1gQ9J4yBHyXoBt3dpLgN3PEOA6DKxFTNPTBJMBjDIHQqdHWP8/othNCCZXp+NgzFyO7azrECwXv6g//uelQFlIRmnS7iqWCTPzcXJXRAqf97H/IR2tOKe28THK9QiNrIOnCG0lnBiU2HRyYg90v5f2hUml/OJdK/JOtulnQU+s6o62OtWOly6wHdh50jHxdjB7f+83xQhj5KBCQAM4HtkXFUAImMCRItZ2IIwrgDEzQIBLjw6HEAY0ECrSfP2hsHBFTe5c4b3qYIShIrAMAGdYVTDgFDEChz6ddLsVRAHzEu94cU56FfD7JAgH04Q7lMj189bCklBMG6BTDZak1CXzuAzGtJXnsu+Ukighic7+cTNrUTVaYS8Qg5okJF9qZc5oMFL8KL3J8GtMpCriSESOqKNQAYgyNtTANORIRlelu9BTP+JFgEeomYZo7OgVsdom8rq5TSpTbGZMQiHLK1dvQ7gRoAUPfB4EnHaqkI9BSSKe7o1LviuilktJJjHJd8zKBS+i7AvpOqTJ4VgDPNFhG+2t0bPar0usCeF3715sCelOVulEBjWrovIDOa5XdRQFe1OBlAV3W0LiAxjX0QwH9UPcZ6LEY9IYyO4v0UOU5wXP0miEUKNnpqepemD7vgV2WJDMgO7ZK7fbQWF+BGeDHCV2+uXr7u5Inh709a19VGS6J0Ipi7e7vnVg1yiTrJudYh4e94xonZCCYrAudnu3/YtcL0YhRsiYdHOz++nO9UowICRfrSifHp73d6hwxWDMh36vZsc2aaZMmer6rRoHbJMicauTP6vzXDMRfYYdN1VcGNipok2LlZqMiblKsrF0pKpugybTqh8Whyb0OSEY5RfrGZ+itnuJzfUsBEbIf9eiyiY+1ffrX6SbRfxHBckXUUbutnx+7+tjUg+vejv1iZ++PF51Xx/lDtGk8Nb43tg3bODBeGW+MC6NvwNZla9n62Ppz46+NTxufN3PqrVYefGeU1ub9fwEUnv1M</latexit>

p(c | w1, . . . ,wm) / p(w1, . . . ,wm | c)p(c)

p(w1, . . . ,wm | c) =

p(w1 | c) p(w2 | w1, c)Y

i2[3,...,m]



p(c | w1, . . . ,wm) / p(w1, . . . ,wm | c)p(c)

p(w1, . . . ,wm | c) =

p(w1 | c) p(w2 | w1, c)Y

i2[3,...,m]



p(c | w1, . . . ,wm) / p(w1, . . . ,wm | c)p(c)

p(w1, . . . ,wm | c) =

p(w1 | c) p(w2 | w1, c)Y

i2[3,...,m]



p(c | w1, . . . ,wm) / p(w1, . . . ,wm | c)p(c)

p(w1, . . . ,wm | c) =

p(w1 | c) p(w2 | w1, c)Y

i2[3,...,m]



p(c | w1, . . . ,wm) / p(w1, . . . ,wm | c)p(c)

p(w1, . . . ,wm | c) =

p(w1 | c) p(w2 | w1, c)Y

i2[3,...,m]



Markov modeling: final comments

• 0-order Markov model: Naive Bayes For spam classification higher orders don’t improve performance. For other tasks they do.

• Short documents are vastly more likely than long ones Doesn’t matter for classification. In other settings, conditioning on length may be necessary.

• Laplace smoothing Same as before: adapt the estimator by adding pseudo-observations

27

p(prize | a,won) ⇡ 1+#“won a prize”|V |+#“won a”

<latexit sha1_base64="/zwZuU6glvNbv6lOl+/jKvYnKOs=">AAAINXiclVVdj9tEFHUKNCVQuoVHXkZEqAWilZ3t7oaHSqW7Syug3WW1yVZah+14cuNYGduj8TiJdzq/i//BGw+8IV75C4w/svIXSPjF1/eccz33zNWMw6gXCdP8vXPnvfc/uNu992Hvo4/vf/Jg5+GnkyiMOYExCWnI3zg4AuoFMBaeoPCGccC+Q+HSWR6l+OUKeOSFwYVIGEx97Abe3CNY6NT1zm/ssc2JtAVshGTcuwGlbN+bIdvdZrFSA2SH2891GCj1FbIxYzzcIHvOMZEW+gbZfZQz3r5VVT5Sqlwv+6z/tNA+eqQQUvLd5N3/qViWq+udvrlrZg9qBlYR9I3iObt+ePdHexaS2IdAEIqj6MoymZhKzIVHKKieHUfAMFliF65iMR9NpRewWEBAFPpSY/OYIhGi1F808zgQQRMdYMI9XQGRBdYeCb0LvWqpCALsQzSYrTwW5WG0cvNAYL2FU7nJtljdryilyzFbeGRTWZrEfuRjsWgko8R3qkmIKfCVX02my9SLrDE3wIkXpSacaWdOWTo20UV4VuCLhC0giJSMOVVloQaAc5hrYRZGIGIms270rC6jp4LHMEjDLPf0GPPlOcwGuk4lUV3OnIZYVFOObkO7E8CahL6Pg5m0mSoGwx7sqsy7MnqupLRToxwHnadwBX1dQl8rVQVPSuCJBqvo+Bado3FdOimBk8ZfL0voZV3qxCU0bqCrErpqVHbWJXjdgDcldNNAkxKaNNCbEnrT9BnrsbgaTmW+F9mmylPqreAFBwiU7A9VvReu9/vKqkrSGZB9S2V2z2CuD7oc8JOULl9evPpJyaPRcN88UHWGQ2PYUsy9g/0js0Fx89UUHHM0Gj5vcEKOA/e20PHJwXdWsxCLOaO3pMPDve+/bVZKgNJwfVvp6PnxcK8+R5w0TCh6RX0LNUxz2+hFV60Cp02QO9XKXzb5LzhO/oUdtlXfGtiqYG2KrZutiqRNsbV2q6g1wdJpXeorg6XnOqY55Rj0ic/hlZ7iU31KYRHyr/Xoctf3tH36bQ/S6L+IeLMl6qjX09ePVb9smsFkuGs92d3/+Un/2Q/FRXTP+Nz4wnhsWMah8cx4aZwZY4N0Rp1fOm5n0f21+0f3z+5fOfVOp9B8ZlSe7t//AN/c/+g=</latexit>

V is the vocabulary (the set of all words known to the language model).

As before, we can give the pseudo observations a smaller weight than 1, to have less impact on the estimate.

28

“congratulations you have won a prize”

p(spam|“congrats, a reward has been awarded to you”)

the man fell out of the …

pool

gazebo

cycling

These kinds of probability models, and many other models for symbolic sequential data, is that they model symbols as entirely separate objects. They don’t exploit the fact that some words have similar meanings or syntactic functions.

For instance, the Markov assumption helps us with the man falling out of the pool, because “out of the pool” is a common phrase, even if nobody falls out of it. However, for rarer cases like “out of the gazebo” vs “out of the cycling” it may be that both trigrams never appear in the data. In that case, we might still say that the Birst is more likely because it at least ends in a noun, which is expected.

Similarly, if our training data give us little information about the message shown here, we would want a model that can conclude that the words in the sentences are very similar to words that do occur in the spam training corpus.

To deal with both cases, we need to somehow model word similarity. There are many ways to do this. One popular choice is to use an embedding model.

embedding models

Model object x by embedding vector ex.

If ex and ey are “similar,” so are x and y

Learn the parameters of {ex} from data.

Cf: latent vectors from autoencoders, PCA, etc.

29

Embedding models take a set of discrete objects, and assign vectors to them. These vectors are then learned from data, based on some loss over the embeddings.

(This is a bit like mapping images to latent vectors in an autoencoder, except we learn the latent representation directly, instead of learning a function from some representation into the late representation).

distributional hypothesis

Words that occur in the same context often have similar meanings.

30

The distributional hypothesis provides us with a sound way to train these embeddings.

To paraphrase: if we have some word X, and we create a probability distribution P modelling how likely a word Y is to occur in the context of X, we can use that distribution P as a proxy for the meaning of X.

The algorithm we will discuss, Word2Vec, provides learns a simple mapping from the embedding to a probability distribution.

1-hot vector

31

cat

vocabulary

mat

the

To start with, we’ll represent words as atomic objects: in a single, very large one-hot vector.

These are very big, but we can usually implement things in a clever way so that we won’t explicitly need to store these in memory.

Word2Vec (skipgram version)

32

x ycompare shall compare icompare theecompare to

thee i thee comparethee tothee a

to compareto thee to ato summersa theea toa summersa day

summers to

shall i compare thee to a summers day thou art more lovely …

x

y

linear

softmax

10000

300

10000

x

We slide a context window over the sequence. The task is to predict the distribution p(y|x): that predict which words are likely to occur in the context window given the middle word.

We create a dataset of word pairs from the entire text and feed this to a very simple two-layer network. This is a bit like an autoencoder, except we’re not reconstructing the output, but predicting the context.

The softmax activation over 10k outputs is very expensive to compute, and you need some clever tricks to make this feasible (called hierarchical softmax or negative sampling). We won’t go into them here.

after training

33

compare

ecompare

After training, we discard the second layer, and use only the embeddings produced by the Birst layer.

34

x

linear

embedding

Wx

compare

e com

pare

Because the input layer is just a matrix multiplication, and the input is just a one-hot vector, what we end up doing when we compute the embedding for word i, is just extracting the i-th column from W.

In other words, we’re not really training a function that computes an embedding for each word, we are actually learning the embeddings directly: every element of every embedding vector is a separate parameter.

35

v(king) + v(woman) - v(man) ≈ v(queen)

“feminine” vector

Famously, Word2Vec produces not just an informative embedding, where similar words are close together, but for many concepts, there seems to be a kind of algebraic structure, similar to the smile vector example from the autoencoder.

word2vec summary

Word2Vec creates embedding vectors for words.

In the embedding space distances and directions reflect semantic meaning.

Word2Vec embeddings are a great starting point for deep learning projects on natural language.

Standard W2V embeddings can be downloaded from Google.

36

break

Garfield generated by a Markov model

37source: https://blog.codinghorror.com/markov-and-you/

sequences in deep learning

Sequence models: operate on inputs of different lengths (using the same weights).

input: raw sequence data deep learning is end-to-end learning

output: classificiation, regression, token prediction

layers: sequence-to-sequence

38

inputs

39

a

b

c

d

e

f

g

As we’ve seen, when we want to do deep learning, our input should be represented as a tensor. Preferably in a way that retains all information (i.e. we want to be learning from the raw data, or something as close to it as possible).

Here is an example: to encode a simple monophonic musical sequence, we just one-hot encode the notes, and encode note sequence as a matrix: one dimension for time, one dimension for the notes. We can do the same thing for characters.

source: https://violinsheetmusic.org

instance

s

40

time

voca

bula

ry

If we have multiple sequences of different lengths, this leads to a data set of matrices of different sizes.

batching sequences

41

paddingoriginal length

batch

In principle, this is not a problem, we want to build models that can deal with sequences of any length (and can generalise over sequences of variable lengths).

However, within a batch it’s usually required that all sequences have the same length. Often, we put sequences of similar length together (for instance by sorting the data by length) and then pad the shorter sequences with zero vectors, so that all sequences are the same length.

recurrent neural network

42

x

y

h

h

copy previous hidden layer

One of the most popular neural networks for sequences is the

recurrent neural network. This is a generic name for any neural network with cycles in it.

The Bigure shows a popular conBiguration. It’s a basic fully connected network, except that its input x is extended by three nodes to which the hidden layer is copied.

visual shorthand

Adapted from http://colah.github.io/posts/2015-08-Understanding-LSTMs/

43

W

x

V

y

To keep things clear we will adopt this visual shorthand: a rectangle represents a vector of nodes, and an arrow feeding into such a rectangle annotated with a weight matrix represents a fully connected transformation.

We will assume bias nodes are included without drawing them.

visual shorthand

44

W

x

V

y

<- concatenation

h

A line with no weight matrix represents a copy of the input vector. When two lines Blow into each other, we concatenate their vectors.

Here, the added line copies h, concatenates it to x, and applies weight matrix W.

RNNs on sequences

45

W

x1

V

y1

h1

x2 x3 …

h0 = (0, 0, …, 0)

We can now apply this neural network to a sequence. We feed it the Birst input, x1, result in a Birst value for the hidden layer, h1, and retrieve the Birst output y1.

The hidden nodes are initialise to zero, so at Birst the network behaves just like a fully connected network.

RNNs on sequences

46

W

x2

V

y2

h2

x3 x4 …

h1

x1

y1

We then feed the second input in the sequence, x1. We now receive the previous hidden layer, h1, concatenate it to the input, and multiply it by W, to produce the second hidden layer h2. This is multiplied by V to produce the second output.

RNNs on sequences

47

W

x3

V

y3

h3

x4 x5 …

h2

x2

y2

x1

y1

And so on.

how to train RRNs?

Provide an input sequence x and a target sequence t.

Backpropagation Through Time (BPTT).

48

In principle, we

how to train RNNs? unrolling

49

W

x1

V

y1

x2 x3 …

y2 y3 …

h1 h2 h3

W

V V

W

x4

V

y4

x5 x6

y5 y6

h4 h5 h6

W

V V

…W

h0

t1 t2 t3 …t4 t5 t6

Instead of visualising a single small network, applied at every time step, we can unroll the network. Every step in the sequence is applied in parallel to a copy of the network, and the recurrent connection Blows from the previous copy to the next.

Now the whole network is just one big, complicated

feedforward net, that is, a network without cycles. Note that we have a lot of shared weights, but we know how to deal with those.

The hidden layer is initialised to the zero vector.

backpropagation through time

50

W

x1

V

y1

x2 x3 …

y2 y3 …

h1 h2 h3

W

V V

W

x4

V

y4

x5 x6

y5 y6

h4 h5 h6

W

V V

…W

Now the whole network is just one big, complicated feedforward net. Note that we have a lot of shared weights, but we know how to deal with those.

truncated backpropagation through time

51

W

x1

V

y1

x2 x3 …

y2 y3 …

h1 h2 h3

W

V V

W

x4

V

y4

x5 x6

y5 y6

h4 h5 h6

W

V V

…W

In truncated backpropagation through time, we limit how far back the backpropagation goes, to save memory. The output is still entirely dependent on the whole sequence, but the weights are only trained based on the last few steps. Note that the weights are still affected everywhere, because the are shared between timesteps.

RNNs: thing to note

sequence-to-sequence model.

no Markov assumption (potentially infinite memory).

fixed set of weights, variable input length.

52

what can we use RNNs for?

Sequence to sequence: Eg. Translating English to French

Sequence to label Eg. sequence classification, regression

Label to sequence E.g. (conditional) sentence generation

53

sequence to sequence

54

x1 x2 x3 x4 x5 x6

y1 y2 y3 y4 y5 y6

t1 t2 t3 t4 t5 t6loss

If is the most straightforward way of doing sequence to sequence modeling. Note that the model is now forced to produce the Birst output before seeing the rest of the input.

55

x1 x2 x3 x4 x5 x6

y1 y2 y3 y4 y5 y6

t1 t2 t3 t4 t5 t6

If we feed the input and output to the model like this (where the greyed out boxed represent zero vectors), the model gets to read the whole input before having to decide on an output.

Sometimes people even put a Bixed number of zero vectors in between the input and the output to allow the model some “time” to process.

sequence to label

56

x1 x2 x3 x4 x5 x6

L

If we want to map sequences to labels, we could just ignore all outputs except the last, and take that to be the label. We can compare it to the target label and back propagate the error.

However, this puts a long distance between the Birst input and the label, and a very short distance between the last input and the label. This asymmetry may mean the model ignores information at the start of the sequence.

sequence to label

57x1 x2 x3 x4 x5 x6

average

L

A better approach is to average the outputs over the whole sequence.This makes the distance to the label the same for every input.

Note that the model should be able to handle sequences of variable lengths: this means we can’t put a fully connected layer on the top to combine the whole sequence into a single label.

label to sequence

58L

y1 y2 y3 y4 y5 y6

label to sequence

59

L

y1 y2 y3 y4 y5 y6

We can also use the hidden layer at timestep 0 as an input for the label, and zero the sequence inputs (or use them for something else).

the problem of long-term dependence

60

I was born in France, as matter of fact in a little village near Paris, it’s famous for its pain-au-chocolat, I lived there until I was 16, when I moved to Amsterdam, so I’m fluent in…

French

Dutch

Aquarium

Basic RNNs work pretty well, but they do not learn to remember information for very long. Technically they can, but the gradient vanished too quickly over the timesteps.

You can’t have a long term memory for everything. You need to be selective, and you need to learn to select words to be stored for the long term when you Birst see them.

In order to remember things long term you need to forget many other things.

LSTM (1997)

Long short-term memory

Selective forgetting and remembering, controlled by learnable “gates”

Possibly the first successful deep neural network

61

An enduring solution to the problem are LSTMs. LSTMs have a complex mechanism, which we’ll go through step by step, but the main component is a gating mechanism.

gate

62

+

input

1

0

1

-1

sigmoid

tanh

vector to add

Ws Wt

The gate combines the sigmoid and tanh activations. The sigmoid we’ve seen already. The tanh is just a the sigmoid rescaled so that its outputs are between -1 and 1.

The gating mechanism takes two input vectors, and combines them using a sigmoid and a tanh activation. The gate is best understand as producing an additive value: we want to Bigure out how much of the input to add to some other vector (if it’s import, we want to add most of if, otherwise, we want to forget it, and keep the original value).

The input is Birst transformed by two weight metrics and then passed though a sigmoid and a tanh. The tanh should be though of as a mapping of the input to the range [-1, 1]. This ensures that the effect of the addition vector can’t be too much. The

sigmoid acts as a selection vector. For elements of the input that are important, it outputs 1, retaining all the input in the addition vector. For elements of the input that are not important, it outputs 0, so that they are zeroed out. The sigmoid and tanh vectors are element-wise multiplied.

Note that if we initialise Wt and Ws to zero, the input is entirely ignored.

x7

y5

cells

63

yt

x6

y6

x5

y5

x4

y4

x5

y3

C5

y5

C4

y4

C3

y3

C6

y6

The basic operation of the LSTM is called a cell (the orange square, which we’ll detail later). Between cells, there are two recurrent connections, y, the current output, and C the cell

state.

notation

64

Wsigmoid activation

tanh activationW

+element-wise operation

apply weights W

concatenate

Here is our visual notation.

65

xt xt+1

yt-1 yt

+

+

+

+

Wf Wi WC Wo

Here is what happens inside the cell. It looks complicated, but we’ll go through all the elements step by step.

66

xt xt+1

yt-1 yt

+

+

+

+

Wf Wi WC Wo

Ct-1 Ct

The Birst is the “conveyor belt”. It passes the previous cell state to the next cell. Along the way, the current input can be used to manipulate it.

Note that the connection from the previous cell to the next has no activations. This means that along this path, gradients do not decay. It’s also very easy for an LSTM cell to ignore the current information and just pass the information along the conveyor belt.

67

xt xt+1

yt-1 yt

+

+

+

+

Wf Wi WC Wo

Here is the Birst manipulation of the conveyor belt. This is called the forget gate.

It looks at the current input, concatenated with the previous output, and applies an element-wise scaling to the current value in the conveyor belt. Outputting all 1s will keep the current value on the belt what it is, and outputting all values near 0, will decay the values (forgetting what we’ve seen so far, and allowing it to be replaces by our new values in the next step).

68

xt xt+1

yt-1 yt

+

+

+

+

Wf Wi WC Wo

in the next step, we pass the input through a generic gate, as described earlier, and add the resulting vector to the value on the conveyor belt.

69

xt xt+1

yt-1 yt

+

+

+

+

Wf Wi WC Wo

Finally, we need to decide what to output now. We take the current value of the conveyor belt, tanh it to rescale, and element-wise multiply it by another sigmoid activated layer. This layer is sent out as the current output, and sent to the next cell along the second recurrent connection.

LSTM language model

70

p(prize | a,won,have, you, congratulations) =p(prize | a,won)

<latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit><latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit><latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit><latexit sha1_base64="znPXRgDzYjmhSZkkF35R5GnrkEc=">AAAIGHicnVVdb9s2FJW7re68dUvXx+5BmLGhHYxActYkfQjQ5WMt1o9kQZwUiIyAoq9lwZRIkJRtheAv2f/Y+56Gve5t/2aULAWSpe1hfPH1OedekedekD4joZCO83fn3kcff3K/++DT3mefP/ziy61HX10KmnAMI0wJ5R98JICEMYxkKAl8YBxQ5BO48udHGX+1AC5CGl/IlME4QkEcTkOMpIFutn77jj31OFaehJVUjIe3oLUXhRPbC0oUaT2wPVr+XdI4B/wSmKEF5EhaIilNcoCVAKZxwJFMSP5ZofUz+8D2vN7//Pqzm62+s+3ky24GbhH0rWKd3Ty6/8abUJxEEEtMkBDXrsPkWCEuQ0xA97xEAEN4jgK4TuR0f6zCmCUSYqztbw03TYgtqZ15aE9CDliS1AQI89BUsPEMcYSlcbpXLyUgRhGIwWQRMrEOxSJYBxKZNo3VKm+jfljLVMYvNgvxqrY1hSIRITlrgCKN/DoICQG+iOpgtk2zyQ3lCjgORWbCmXHmlOU9uqBnBT9L2QxM01TCia4mGgI4h6lJzEMBMmEqP42Zx7k4kDyBQRbm2MEx4vNzmAxMnRpQ386UUCTrkG+OYdyJYYlpFKF4ojymi4HwBts6967KnmulvMwo37fPM7rGvq+w77WukycV8sSQdXZ0x07t0WbqZYW8bHz1qsJebab6SYVNGuyiwi4alf1lhV426FWFXTXYtMKmDfa2wt42fUZmLK6HY7XuRd5UdUrCBbziALFW/aHePAs3/b526ynZDKi+q3O7JzA1l9maiNJMrl5fvHur1dH+8LmzqzcVPkmglDg7u8+PnIYkWO+m0Dj7+8PDhoZyFAd3hY5Pdn90m4VYwhm5E+3t7fz0olkpBULo8q7S0eHxcGdzjjhumFCc1e67dsO0oE1enKo1wW9LWDvVqp839a84Sv9FTduqlwa2ZrC2jNLN1oy0LaO0tszYOATLpnVungqW3euIrCXHYG58Du/MFJ+aWwpJyr83o8uDKDT2mV9vkEX/JUSrUmiiXs88P+7mY9MMRsPtF9vuLz/0X/5cvEMPrCfWN9ZTy7X2rJfWa+vMGlm483XnsPOm87b7a/f37h/dP9fSe50i57FVW92//gGKpPYL</latexit>

NB: no Markov assumption!

That’s a lot of complexity. Let’s see what it buys us. The best way, by far, to illustrate the power of LSTMs is to apply the sequential sampling trick.

LSTM language model

71

you have won a prize

have won a prize

We train the language model to predict the next token in the sequence. This is easy to do: we just us the sequence shifted by 1 as the training sequence.

character level language model

72

y o u h a v e w o

o u h a v e w o

softmax ->

p(X| o,u,h,a,v,e,w,o )

We’ll do this at charter level, to make it more challenging, and we ensure that he network outputs a softmax activated vector over all characters at each timestep. We can then interpret the output of the model at step t as the probability of the character at that point, conditoned on the entire history.

sequential sampling from a language model

start with a small seed sequence s = [c1, c2, c3] of tokens.

loop:

Sample next char c according to p(C = c | c1, c2, …) feed the whole seed to the network

append c to s

also known as a autoregressive RNN

73

Note that this time, as there is no Markov assumption, the network has to see the whole sequence so far to predict the next character.

some examples

source: The Unreasonable Effectiveness of Recurrent Neural Networks Andrej Karpathy

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

74

Shakespeare

75

PANDARUS:Alas, I think he shall be come approached and the dayWhen little srain would be attain'd into being never fed,And who is but a chain and subjects of his death,I should not sleep.

Second Senator:They are away this miseries, produced upon my soul,Breaking and strongly should be buried, when I perishThe earth and thoughts of many states.

Remember, this is a character level language model.

Wikipedia

76

Naturalism and decision for the majority of Arab countries' capitalide was groundedby the Irish language by [[John Clair]], [[An Imperial Japanese Revolt]], associated with Guangzham's sovereignty. His generals were the powerful ruler of the Portugal in the [[Protestant Immineners]], which could be said to be directly in Cantonese Communication, which followed a ceremony and set inspired prison, training. The emperor travelled back to [[Antioch, Perth, October 25|21]] to note, the Kingdom of Costa Rica, unsuccessful fashioned the [[Thrales]], [[Cynth's Dajoard]], known in western [[Scotland]], near Italy to the conquest of India with the conflict. Copyright was the succession of independence in the slop of Syrian influence that was a famous German movement based on a more popular servicious, non-doctrinal

Note that not only is the language natural, the wikipedia markup is also correct (link brackets are closed properly, and contain key concepts).

77

October 25|21]] to note, the Kingdom of Costa Rica, unsuccessful fashioned the [[Thrales]], [[Cynth's Dajoard]], known in western [[Scotland]], near Italy to the conquest of India with the conflict. Copyright was the succession of independence in the slop of Syrian influence that was a famous German movement based on a more popular servicious, non-doctrinal and sexual power post. Many governments recognize the military housing of the [[Civil Liberalization and Infantry Resolution 265 National Party in Hungary]], that is sympathetic to be to the [[Punjab Resolution]](PJS)[http://www.humah.yahoo.com/guardian.cfm/7754800786d17551963s89.htm Official economics Adjoint for the Nazism, Montgomery was swear to advance to the resources for those Socialism's rule, was starting to signing a major tripad of aid exile.]]

The network can even learn to generate valid (looking) URLs for external links.

78

<page> <title>Antichrist</title> <id>865</id> <revision> <id>15900676</id> <timestamp>2002-08-03T18:14:12Z</timestamp> <contributor> <username>Paris</username> <id>23</id> </contributor> <minor /> <comment>Automated conversion</comment> <text xml:space="preserve">#REDIRECT [[Christianity]]</text> </revision></page>

Sometimes wikipedia text contains bits of XML for structured information. The model can generate these Blawlessly.

LaTeX

79

80

https://twitter.com/deepdrumpf?lang=en

generative sequence model

81

generative sequence model

82

h e l l o

sequence to sequence (variational) AE

83

h e l l o

h e l l o

µz Az<latexit sha1_base64="US5A12l3Bu7RUka+T73Vg/HSnJg=">AAAHa3icfVXbbtNKFDW3BgIcyuUJeLCIODo6iiI7pTchJGjL5QFoT9W0SHVUjSc7iZWxPZoZp3FH8wd8Da/wI/wE38DYTio7Y47zkK291trae+3RjE9JwIXj/Lxy9dr1GyuNm7eat+/c/eve6v0HxzxOGIYejknMvviIAwki6IlAEPhCGaDQJ3DiT3Yz/GQKjAdxdCRSCv0QjaJgGGAkdOps9W8vxtLzQ+mFiVJn3oXtvcx+tucX+Td59my15XSc/LPNwJ0HLWv+HZzdX+l4gxgnIUQCE8T5qetQ0ZeIiQATUE0v4UARnqARnCZiuNWXQUQTARFW9nONDRNii9jOerYHAQMsSKoDhFmgK9h4jBjCQk/WrJbiEKEQeHswDSgvQj4dFYFA2pa+nOW2qbsVpRwxRMcBnlVakyjkIRJjI8nT0K8mISHApmE1mbWpm1xizoDhgGcmHGhn9mm2Cn4UH8zxcUrHEHElE0ZUWagBYAyGWpiHHERCZT6N3v+EvxIsgXYW5rlXe4hNDmHQ1nUqiWo7QxIjUU35egztTgTnOA5DFA2kR5X0BMyE9NodlXtXRg+V1CdIG+X79mEGV9DPJfSzUkva3iU6tHsarYDHJfDYKHxSQk+WpX5SQhMDnZbQqVHZPy/B5wY8K6EzA01LaGqgFyX0wrQS6c2fdvuysDvfm9wnwRTeM4BIyVZXLc/C9EpP3aokW7NsuSq3ewBDfT8UQJhmdPnh6NNHJXe3uuvOhlpm+CSBBcVZ21jfdQzKqOhmznG2tro7BidmKBpdFtp7u/HGNQvRhFFySdrcXHu3bVZKgZD4/LLS7s5ed235HDFsmDCf1W65tmHaqI4+n6pW4NcJCqdq+ROT/56h9A/suK76wsBaBa1TLNysVaR1ioW1C8XSEDQ7rRP9FtDs6kakoOyBvtQZfNKneF9fREjE7F99dNkoDLR9+t9rZ9H/EdFsQdRRs6lfGHf5PTGDXrez3XH/e9F6vTN/am5aT6xn1j+Wa21ar60P1oHVs7D11fpmfbd+rPxqPGo8bjwtqFevzDUPrcrXeP4bPmavoA==</latexit><latexit sha1_base64="US5A12l3Bu7RUka+T73Vg/HSnJg=">AAAHa3icfVXbbtNKFDW3BgIcyuUJeLCIODo6iiI7pTchJGjL5QFoT9W0SHVUjSc7iZWxPZoZp3FH8wd8Da/wI/wE38DYTio7Y47zkK291trae+3RjE9JwIXj/Lxy9dr1GyuNm7eat+/c/eve6v0HxzxOGIYejknMvviIAwki6IlAEPhCGaDQJ3DiT3Yz/GQKjAdxdCRSCv0QjaJgGGAkdOps9W8vxtLzQ+mFiVJn3oXtvcx+tucX+Td59my15XSc/LPNwJ0HLWv+HZzdX+l4gxgnIUQCE8T5qetQ0ZeIiQATUE0v4UARnqARnCZiuNWXQUQTARFW9nONDRNii9jOerYHAQMsSKoDhFmgK9h4jBjCQk/WrJbiEKEQeHswDSgvQj4dFYFA2pa+nOW2qbsVpRwxRMcBnlVakyjkIRJjI8nT0K8mISHApmE1mbWpm1xizoDhgGcmHGhn9mm2Cn4UH8zxcUrHEHElE0ZUWagBYAyGWpiHHERCZT6N3v+EvxIsgXYW5rlXe4hNDmHQ1nUqiWo7QxIjUU35egztTgTnOA5DFA2kR5X0BMyE9NodlXtXRg+V1CdIG+X79mEGV9DPJfSzUkva3iU6tHsarYDHJfDYKHxSQk+WpX5SQhMDnZbQqVHZPy/B5wY8K6EzA01LaGqgFyX0wrQS6c2fdvuysDvfm9wnwRTeM4BIyVZXLc/C9EpP3aokW7NsuSq3ewBDfT8UQJhmdPnh6NNHJXe3uuvOhlpm+CSBBcVZ21jfdQzKqOhmznG2tro7BidmKBpdFtp7u/HGNQvRhFFySdrcXHu3bVZKgZD4/LLS7s5ed235HDFsmDCf1W65tmHaqI4+n6pW4NcJCqdq+ROT/56h9A/suK76wsBaBa1TLNysVaR1ioW1C8XSEDQ7rRP9FtDs6kakoOyBvtQZfNKneF9fREjE7F99dNkoDLR9+t9rZ9H/EdFsQdRRs6lfGHf5PTGDXrez3XH/e9F6vTN/am5aT6xn1j+Wa21ar60P1oHVs7D11fpmfbd+rPxqPGo8bjwtqFevzDUPrcrXeP4bPmavoA==</latexit><latexit sha1_base64="US5A12l3Bu7RUka+T73Vg/HSnJg=">AAAHa3icfVXbbtNKFDW3BgIcyuUJeLCIODo6iiI7pTchJGjL5QFoT9W0SHVUjSc7iZWxPZoZp3FH8wd8Da/wI/wE38DYTio7Y47zkK291trae+3RjE9JwIXj/Lxy9dr1GyuNm7eat+/c/eve6v0HxzxOGIYejknMvviIAwki6IlAEPhCGaDQJ3DiT3Yz/GQKjAdxdCRSCv0QjaJgGGAkdOps9W8vxtLzQ+mFiVJn3oXtvcx+tucX+Td59my15XSc/LPNwJ0HLWv+HZzdX+l4gxgnIUQCE8T5qetQ0ZeIiQATUE0v4UARnqARnCZiuNWXQUQTARFW9nONDRNii9jOerYHAQMsSKoDhFmgK9h4jBjCQk/WrJbiEKEQeHswDSgvQj4dFYFA2pa+nOW2qbsVpRwxRMcBnlVakyjkIRJjI8nT0K8mISHApmE1mbWpm1xizoDhgGcmHGhn9mm2Cn4UH8zxcUrHEHElE0ZUWagBYAyGWpiHHERCZT6N3v+EvxIsgXYW5rlXe4hNDmHQ1nUqiWo7QxIjUU35egztTgTnOA5DFA2kR5X0BMyE9NodlXtXRg+V1CdIG+X79mEGV9DPJfSzUkva3iU6tHsarYDHJfDYKHxSQk+WpX5SQhMDnZbQqVHZPy/B5wY8K6EzA01LaGqgFyX0wrQS6c2fdvuysDvfm9wnwRTeM4BIyVZXLc/C9EpP3aokW7NsuSq3ewBDfT8UQJhmdPnh6NNHJXe3uuvOhlpm+CSBBcVZ21jfdQzKqOhmznG2tro7BidmKBpdFtp7u/HGNQvRhFFySdrcXHu3bVZKgZD4/LLS7s5ed235HDFsmDCf1W65tmHaqI4+n6pW4NcJCqdq+ROT/56h9A/suK76wsBaBa1TLNysVaR1ioW1C8XSEDQ7rRP9FtDs6kakoOyBvtQZfNKneF9fREjE7F99dNkoDLR9+t9rZ9H/EdFsQdRRs6lfGHf5PTGDXrez3XH/e9F6vTN/am5aT6xn1j+Wa21ar60P1oHVs7D11fpmfbd+rPxqPGo8bjwtqFevzDUPrcrXeP4bPmavoA==</latexit><latexit sha1_base64="US5A12l3Bu7RUka+T73Vg/HSnJg=">AAAHa3icfVXbbtNKFDW3BgIcyuUJeLCIODo6iiI7pTchJGjL5QFoT9W0SHVUjSc7iZWxPZoZp3FH8wd8Da/wI/wE38DYTio7Y47zkK291trae+3RjE9JwIXj/Lxy9dr1GyuNm7eat+/c/eve6v0HxzxOGIYejknMvviIAwki6IlAEPhCGaDQJ3DiT3Yz/GQKjAdxdCRSCv0QjaJgGGAkdOps9W8vxtLzQ+mFiVJn3oXtvcx+tucX+Td59my15XSc/LPNwJ0HLWv+HZzdX+l4gxgnIUQCE8T5qetQ0ZeIiQATUE0v4UARnqARnCZiuNWXQUQTARFW9nONDRNii9jOerYHAQMsSKoDhFmgK9h4jBjCQk/WrJbiEKEQeHswDSgvQj4dFYFA2pa+nOW2qbsVpRwxRMcBnlVakyjkIRJjI8nT0K8mISHApmE1mbWpm1xizoDhgGcmHGhn9mm2Cn4UH8zxcUrHEHElE0ZUWagBYAyGWpiHHERCZT6N3v+EvxIsgXYW5rlXe4hNDmHQ1nUqiWo7QxIjUU35egztTgTnOA5DFA2kR5X0BMyE9NodlXtXRg+V1CdIG+X79mEGV9DPJfSzUkva3iU6tHsarYDHJfDYKHxSQk+WpX5SQhMDnZbQqVHZPy/B5wY8K6EzA01LaGqgFyX0wrQS6c2fdvuysDvfm9wnwRTeM4BIyVZXLc/C9EpP3aokW7NsuSq3ewBDfT8UQJhmdPnh6NNHJXe3uuvOhlpm+CSBBcVZ21jfdQzKqOhmznG2tro7BidmKBpdFtp7u/HGNQvRhFFySdrcXHu3bVZKgZD4/LLS7s5ed235HDFsmDCf1W65tmHaqI4+n6pW4NcJCqdq+ROT/56h9A/suK76wsBaBa1TLNysVaR1ioW1C8XSEDQ7rRP9FtDs6kakoOyBvtQZfNKneF9fREjE7F99dNkoDLR9+t9rZ9H/EdFsQdRRs6lfGHf5PTGDXrez3XH/e9F6vTN/am5aT6xn1j+Wa21ar60P1oHVs7D11fpmfbd+rPxqPGo8bjwtqFevzDUPrcrXeP4bPmavoA==</latexit>


e ⇠ N(0, I)<latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit><latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit><latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit><latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit>


MusicVAE

84source: https://magenta.tensorflow.org/music-vae

source: https://magenta.tensorBlow.org/music-vae

teacher forcing

85

h e l l o

h e l l o





h e l l_

sketch-rnn

86

Figure 7: Interpolation of (pig, rabbit, crab and face), yoga poses, mosquitoes and mermaids.We also interpolate between four distinct colors for visual effect.

We also construct latent space interpolation examples for the mosquito class and the mermaid class,in the last two grids Figure 7. We see that the model can interpolate between concepts such as styleof wings, leg counts, and orientation. In Figure 8 below, we show more interpolation examples ofother classes from the dataset.

Figure 8: Latent space interpolation between four generated gardens, owls, cats, and firetrucks.

6 Which Loss Controls Image Coherency?

We would like to question the relative importance of the reconstruction loss term LR, relative to theKL loss term LKL, when our goal is to produce higher quality image reconstructions. While ourreconstruction loss term LR optimizes for the log-likelihood of the set of strokes that make up asketch, this metric alone does not give us any guarantee that a model with a lower LR number willproduce higher quality reconstructions compared to a model with a higher LR number.

For example, imagine a simple sketch of an face, ,, where most of the data points of S are be used torepresent the head, and only a minority of points represent facial features such as the eyes and mouth.It is possible to reconstruct the face with incoherent facial features, and yet still score a lower LR

number compared to another reconstruction with a coherent and similar face, if the edges around theincoherent face are generated more precisely.

In Figure 9, we compare the reconstructed images generated using models trained with various wKL

settings. In the first three examples from the left, we train our model on a dataset consisting of fourimage classes (crab, face, pig, rabbit). We deliberately sketch input drawings that contain features oftwo classes, such as a rabbit with a pig mouth and pig tail, a person with animal ears, and a rabbit withcrab claws. We see that the model trained using higher wKL weights, tend to generate sketches withfeatures of a single class that look more coherent, despite having lower LKL numbers. For instance,the model with wKL = 1.00 omit pig features, animal ears, and crab claws from its reconstructions.In contrast, the model with wKL = 0.25, with higher LKL, but lower LR numbers tries to keep bothinconsistent features, while generating sketches that look less coherent.

In the last three examples in Figure 9, we repeat the experiment on models trained on single-classimages, and see similar results even when we deliberately choose input samples from the test set withnoisier lines.

If we look at the interpolations produced in the Latent Space Interpolation section in the main text,models with better KL loss terms also generate more meaningful reconstructions from the interpolated

5

source: A Neural Representation of Sketch Drawings, Ha and Eck (2017)

Here we see two interpolation grids for a sequence-to-sequence VAE trained on a a dataset of millions of quickly drawn sketches.

interpolation

87

VAE

AE

source: Generating Sentences from a Continuous Space by Samuel R. Bowman

Here is an example of interpolation on sentences. First using a regular autoencoder, and then using a VAE. Note that the intermediate sentences for the AE are non-grammatical, but the intermediate sentences for the VAE are all grammatical.

source: Generating Sentences from a Continuous Space by Samuel

R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal

Jozefowicz, Samy Bengio

https://arxiv.org/abs/1511.06349

natural language descriptions (2014)

88

source: https://www.engadget.com/2014/11/18/google-

natural-language-image-description/

recommended: https://twitter.com/picdescbot

MSCOCO

89

cocodataset.org

automatic captioning

90

a dog on a skateboard

Here’s a simple approach to the image captioning problem, which you . We take a pretrained image classiBication network (see the transfer learning slid from Deep Learning 2) and feed the output to an LSTM. We train end-to-end and use sequential sampling to generate outputs. This is model that you should

be able to train on a high end laptop with a basic GPU.

This will give you a pretty decently performing model. To reach the state of the art, you also need something called an “attention mechanism.” But that’s outside the scope of this lecture.

final notes

Word-level LSTMs: use embeddings on the input layer Either pre-trained w2v embeddings, or an embedding layer that is trained end-to-end

Other, similar architecture: GRUs

Bidirectional RNNs Two RNNs, in both directions, concatenate the outputs.

91

summary

Sequence modeling: we can use existing methods through feature extraction, but be careful about validation.

Markov models: simple, finite-memory sequence modeling For classification or generation

Word2Vec: word embeddings reflecting similarity, semantics

LSTMs: incredibly powerful language models. Tricky to train, very opaque.

92

[email protected] 93

61. sequential data - mlvu · models for sequential data machine learning 2020 mlvu.github.io the...

Documents