nlp. text similarity typos: –brittany spears -> britney spears –catherine hepburn ->...
DESCRIPTION
Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater -> theatreTRANSCRIPT
![Page 1: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/1.jpg)
NLP
![Page 2: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/2.jpg)
Text Similarity
Spelling Similarity:Edit Distance
![Page 3: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/3.jpg)
Spelling Similarity
• Typos:– Brittany Spears -> Britney Spears– Catherine Hepburn -> Katharine Hepburn– Reciept -> receipt
• Variants in spelling:– Theater -> theatre
![Page 4: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/4.jpg)
Who is this?
معمر القذافي
![Page 5: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/5.jpg)
Hints
معمر القذافيM
![Page 6: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/6.jpg)
Hints
معمر القذافيMF
![Page 7: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/7.jpg)
Hints
معمر القذافيMF AL
![Page 8: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/8.jpg)
Hints
معمر القذافيMF AL
Muammar (al-)Gaddafi, or Moamar Khadafi, or …
![Page 9: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/9.jpg)
QuizHow many different transliterations can there be?
el al El Al ø
Q G Gh K Kha e u d dh ddh dhdh th zz a f ffi y
m u o a m mm a er
![Page 10: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/10.jpg)
A lot!
m u o a m mm a er
el al El Al ø
Q G Gh K Kha e u d dh ddh dhdh th zz a f ffi y
8 5 360 14,400x x =
![Page 11: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/11.jpg)
Edit Operations
• behaviour - behavior (insertion/deletion) (“al”)• string - spring (substitution) (“k”-”q”)• sleep - slept (multiple edits)
![Page 12: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/12.jpg)
Levenshtein Method
• Based on dynamic programming• Insertions, deletions, and substitutions usually
all have a cost of 1.
![Page 13: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/13.jpg)
Examples t r e n g t h
0 1 2 3 4 5 6 7 8
t 1
r 2
e 3
n 4
d 5
![Page 14: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/14.jpg)
Recurrence relation• Recursive dependencies
D(i,0)=iD(0,j)=jD(i,j)=min[
D(i-1,j)+1D(i,j-1)+1D(i-1,j-1)+t(i,j)]
• Simple edit distance: t(i,j)=0 iff s1(i)=s2(j)t(i,j)=1, otherwise
• Definitions– s1(i) – ith character in string s1
– s2(j) – jth character in string s2
– D(i,j) – edit distance between a prefix of s1 of length i and a prefix of s2 of length j
– t(i,j) – cost of aligning the ith character in string s1 with the jth character in string s2
![Page 15: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/15.jpg)
Example s t r e n g t h
0 1 2 3 4 5 6 7 8
t 1 1
r 2
e 3
n 4
d 5
![Page 16: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/16.jpg)
Example s t r e n g t h
0 1 2 3 4 5 6 7 8
t 1 1 1
r 2
e 3
n 4
d 5
![Page 17: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/17.jpg)
Example s t r e n g t h
0 1 2 3 4 5 6 7 8
t 1 1 1 2 3 4 5 6 7
r 2 2 2
e 3
n 4
d 5
![Page 18: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/18.jpg)
Example s t r e n g t h
0 1 2 3 4 5 6 7 8
t 1 1 1 2 3 4 5 6 7
r 2 2 2
e 3
n 4
d 5
![Page 19: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/19.jpg)
Examples t r e n g t h
0 1 2 3 4 5 6 7 8
t 1 1 1 2 3 4 5 6 7
r 2 2 2 1 2 3 4 5 6
e 3 3 3 2 1 2 3 4 5
n 4 4 4 3 2 1 2 3 4
d 5 5 5 4 3 2 2 3 4
![Page 20: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/20.jpg)
Edit Transcripts t r e n g t h
0 1 2 3 4 5 6 7 8
t 1 1 1 2 3 4 5 6 7
r 2 2 2 1 2 3 4 5 6
e 3 3 3 2 1 2 3 4 5
n 4 4 4 3 2 1 2 3 4
d 5 5 5 4 3 2 2 3 4
![Page 21: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/21.jpg)
Other Costs
• Damerau modification– Swaps of two adjacent characters also have a cost of 1– E.g., Lev(“cats”,”cast”) = 2, Dam(“cats”,”cast”) = 1
![Page 22: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/22.jpg)
Quiz
• Some distance functions can be more specialized.
• Why do you think that the edit distances for these pairs are as follows?– Dist (“sit clown”,“sit down”) = 1– Dist (“qeather”,”weather”) = 1, but Dist
(“leather”,”weather”) = 2
![Page 23: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/23.jpg)
Quiz Answers
• Dist(“sit down”,”sit clown”) is lower in this example because we want to model the type of errors common with optical character recognition (OCR)
• Dist(“qeather”,”weather”) < Dist(“leather”,”weather”) because we want to model spelling errors introduced by “fat fingers” (clicking on an adjacent key on the keyboard)
![Page 24: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/24.jpg)
Quiz: Guess the Language
AACCTGCGGAAGGATCATTACCGAGTGCGGGTCCTTTGGGCCCAACCTCCCATCCGTGTCTATTGTACCC TGTTGCTTCGGCGGGCCCGCCGCTTGTCGGCCGCCGGGGGGGCGCCTCTGCCCCCCGGGCCCGTGCCCGC CGGAGACCCCAACACGAACACTGTCTGAAAGCGTGCAGTCTGAGTTGATTGAATGCAATCAGTTAAAACT TTCAACAATGGATCTCTTGGTTCCGGC
![Page 25: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/25.jpg)
Quiz Answer
• This is a genetic sequence (nucleotides AGCT)
>U03518 Aspergillus awamori internal transcribed spacer 1 (ITS1) AACCTGCGGAAGGATCATTACCGAGTGCGGGTCCTTTGGGCCCAACCTCCCATCCGTGTCTATTGTACCC TGTTGCTTCGGCGGGCCCGCCGCTTGTCGGCCGCCGGGGGGGCGCCTCTGCCCCCCGGGCCCGTGCCCGC CGGAGACCCCAACACGAACACTGTCTGAAAGCGTGCAGTCTGAGTTGATTGAATGCAATCAGTTAAAACT TTCAACAATGGATCTCTTGGTTCCGGC
![Page 26: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/26.jpg)
Other uses of Edit Distance
• In biology, similar methods are used for aligning non-textual sequences– Nucleotide sequences, e.g., GTTCGTGATGGAGCG, where A=adenine, C=cytosine,
G=guanine, T=thymine, U=uracil, “-”=gap of any length, N=any one of ACGTU, etc.
– Amino acid sequences, e.g., FMELSEDGIEMAGSTGVI, where A=alanine, C=cystine, D=aspartate, E=glutamate, F=phenylalanine, Q=glutamine, Z=either glutamate or glutamine, X=“any”, etc. The costs of alignment are determined empirically and reflect evolutionary divergence between protein sequences. For example, aligning V (valine) and I (isoleucine) is lower-cost than aligning V and H (histidine). Valine Isoleucine Histidine
![Page 27: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/27.jpg)
External URLs
• Levenshtein demo• http://www.let.rug.nl/~kleiweg/lev/
• Biological sequence alignment– http://www.bioinformatics.org/sms2/pairwise_align_dna.html – http://www.sequence-alignment.com/sequence-alignment-soft
ware.html
– http://www.ebi.ac.uk/Tools/msa/clustalw2/ – http://www.animalgenome.org/bioinfo/resources/manuals/seqfo
rmats
![Page 28: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/28.jpg)
NACLO Problem
• “Nok-Nok”, NACLO 2009 problem by Eugene Fink:– http://www.naclo.cs.cmu.edu/problems2009/N2009-B.pdf
![Page 29: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/29.jpg)
Solution to the NACLO Problem
• “Nok-Nok”– http://www.naclo.cs.cmu.edu/problems2009/N2009-BS.pdf
![Page 30: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/30.jpg)
NACLO Problem
• “The Lost Tram” - NACLO 2007 problem by Boris Iomdin:
– http://www.naclo.cs.cmu.edu/problems2007/N2007-F.pdf
![Page 31: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/31.jpg)
Solution to the NACLO problem
• “The Lost Tram”– http://www.naclo.cs.cmu.edu/problems2007/N2007-FS.pdf
![Page 32: NLP. Text Similarity Typos: –Brittany Spears -> Britney Spears –Catherine Hepburn -> Katharine Hepburn –Reciept -> receipt Variants in spelling: –Theater](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a4d1b5b7f8b9ab0599ab33b/html5/thumbnails/32.jpg)
NLP