introduction to information retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like...
TRANSCRIPT
![Page 1: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/1.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Introduction to Information Retrievalhttp://informationretrieval.org
IIR 18: Latent Semantic Indexing
Hinrich Schutze
Institute for Natural Language Processing, Universitat Stuttgart
2011-08-29
Schutze: Latent Semantic Indexing 1 / 31
![Page 2: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/2.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Models and Methods
1 Boolean model and its limitations (30)
2 Vector space model (30)
3 Probabilistic models (30)
4 Language model-based retrieval (30)
5 Latent semantic indexing (30)
6 Learning to rank (30)
Schutze: Latent Semantic Indexing 3 / 31
![Page 3: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/3.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Take-away
Schutze: Latent Semantic Indexing 4 / 31
![Page 4: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/4.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Take-away
Singular Value Decomposition (SVD): The math behind LSI
Schutze: Latent Semantic Indexing 4 / 31
![Page 5: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/5.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Take-away
Singular Value Decomposition (SVD): The math behind LSI
SVD used for dimensionality reduction
Schutze: Latent Semantic Indexing 4 / 31
![Page 6: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/6.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Take-away
Singular Value Decomposition (SVD): The math behind LSI
SVD used for dimensionality reduction
Latent Semantic Indexing (LSI): SVD used in informationretrieval
Schutze: Latent Semantic Indexing 4 / 31
![Page 7: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/7.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Outline
1 Singular Value Decomposition
2 Dimensionality reduction
3 Latent Semantic Indexing
Schutze: Latent Semantic Indexing 5 / 31
![Page 8: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/8.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Recall: Term-document matrix
Anthony Julius The Hamlet Othello Macbethand Caesar Tempest
Cleopatraanthony 5.25 3.18 0.0 0.0 0.0 0.35brutus 1.21 6.10 0.0 1.0 0.0 0.0caesar 8.59 2.54 0.0 1.51 0.25 0.0calpurnia 0.0 1.54 0.0 0.0 0.0 0.0cleopatra 2.85 0.0 0.0 0.0 0.0 0.0mercy 1.51 0.0 1.90 0.12 5.25 0.88worser 1.37 0.0 0.11 4.15 0.25 1.95. . .
Schutze: Latent Semantic Indexing 6 / 31
![Page 9: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/9.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Recall: Term-document matrix
Anthony Julius The Hamlet Othello Macbethand Caesar Tempest
Cleopatraanthony 5.25 3.18 0.0 0.0 0.0 0.35brutus 1.21 6.10 0.0 1.0 0.0 0.0caesar 8.59 2.54 0.0 1.51 0.25 0.0calpurnia 0.0 1.54 0.0 0.0 0.0 0.0cleopatra 2.85 0.0 0.0 0.0 0.0 0.0mercy 1.51 0.0 1.90 0.12 5.25 0.88worser 1.37 0.0 0.11 4.15 0.25 1.95. . .
This matrix is the basis for computing the similarity betweendocuments and queries.
Schutze: Latent Semantic Indexing 6 / 31
![Page 10: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/10.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Recall: Term-document matrix
Anthony Julius The Hamlet Othello Macbethand Caesar Tempest
Cleopatraanthony 5.25 3.18 0.0 0.0 0.0 0.35brutus 1.21 6.10 0.0 1.0 0.0 0.0caesar 8.59 2.54 0.0 1.51 0.25 0.0calpurnia 0.0 1.54 0.0 0.0 0.0 0.0cleopatra 2.85 0.0 0.0 0.0 0.0 0.0mercy 1.51 0.0 1.90 0.12 5.25 0.88worser 1.37 0.0 0.11 4.15 0.25 1.95. . .
This matrix is the basis for computing the similarity betweendocuments and queries.
This lecture: Can we transform this matrix, so that we get a bettermeasure of similarity between documents and queries?
Schutze: Latent Semantic Indexing 6 / 31
![Page 11: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/11.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
Schutze: Latent Semantic Indexing 7 / 31
![Page 12: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/12.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
We will decompose the term-document matrix into a productof matrices.
Schutze: Latent Semantic Indexing 7 / 31
![Page 13: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/13.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
We will decompose the term-document matrix into a productof matrices.
The particular decomposition we’ll use: singular valuedecomposition (SVD).
Schutze: Latent Semantic Indexing 7 / 31
![Page 14: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/14.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
We will decompose the term-document matrix into a productof matrices.
The particular decomposition we’ll use: singular valuedecomposition (SVD).
SVD: C = UΣV T (where C = term-document matrix)
Schutze: Latent Semantic Indexing 7 / 31
![Page 15: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/15.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
We will decompose the term-document matrix into a productof matrices.
The particular decomposition we’ll use: singular valuedecomposition (SVD).
SVD: C = UΣV T (where C = term-document matrix)
We will then use the SVD to compute a new, improvedterm-document matrix C ′.
Schutze: Latent Semantic Indexing 7 / 31
![Page 16: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/16.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
We will decompose the term-document matrix into a productof matrices.
The particular decomposition we’ll use: singular valuedecomposition (SVD).
SVD: C = UΣV T (where C = term-document matrix)
We will then use the SVD to compute a new, improvedterm-document matrix C ′.
We’ll get better similarity values out of C ′ (compared to C ).
Schutze: Latent Semantic Indexing 7 / 31
![Page 17: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/17.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Latent semantic indexing: Overview
We will decompose the term-document matrix into a productof matrices.
The particular decomposition we’ll use: singular valuedecomposition (SVD).
SVD: C = UΣV T (where C = term-document matrix)
We will then use the SVD to compute a new, improvedterm-document matrix C ′.
We’ll get better similarity values out of C ′ (compared to C ).
Using SVD for this purpose is called latent semantic indexingor LSI.
Schutze: Latent Semantic Indexing 7 / 31
![Page 18: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/18.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix C
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
This is a standard term-document matrix.
Schutze: Latent Semantic Indexing 8 / 31
![Page 19: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/19.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix C
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
This is a standard term-document matrix.
Actually, we use a non-weighted matrix here to simplify theexample.
Schutze: Latent Semantic Indexing 8 / 31
![Page 20: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/20.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix U
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
Schutze: Latent Semantic Indexing 9 / 31
![Page 21: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/21.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix U
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
Square matrix, M × M
Schutze: Latent Semantic Indexing 9 / 31
![Page 22: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/22.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix U
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
Square matrix, M × M
This is an orthonormal matrix: (i) Row vectors have unit length.(ii) Any two distinct row vectors are orthogonal to each other.
Schutze: Latent Semantic Indexing 9 / 31
![Page 23: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/23.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix U
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
Square matrix, M × M
This is an orthonormal matrix: (i) Row vectors have unit length.(ii) Any two distinct row vectors are orthogonal to each other.
Think of the dimensions as “semantic” dimensions that capturedistinct topics like politics, sports, economics. 2 = water/land
Schutze: Latent Semantic Indexing 9 / 31
![Page 24: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/24.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix U
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
Square matrix, M × M
This is an orthonormal matrix: (i) Row vectors have unit length.(ii) Any two distinct row vectors are orthogonal to each other.
Think of the dimensions as “semantic” dimensions that capturedistinct topics like politics, sports, economics. 2 = water/land
Each number uij in the matrix indicates how strongly related termi is to the topic represented by semantic dimension j .
Schutze: Latent Semantic Indexing 9 / 31
![Page 25: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/25.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix Σ
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
Schutze: Latent Semantic Indexing 10 / 31
![Page 26: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/26.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix Σ
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
This is a square, diagonal matrix of dimensionalitymin(M,N) × min(M,N).
Schutze: Latent Semantic Indexing 10 / 31
![Page 27: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/27.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix Σ
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
This is a square, diagonal matrix of dimensionalitymin(M,N) × min(M,N).
The diagonal consists of the singular values of C .
Schutze: Latent Semantic Indexing 10 / 31
![Page 28: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/28.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix Σ
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
This is a square, diagonal matrix of dimensionalitymin(M,N) × min(M,N).
The diagonal consists of the singular values of C .
The magnitude of the singular value measures the importance ofthe corresponding semantic dimension.
Schutze: Latent Semantic Indexing 10 / 31
![Page 29: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/29.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix Σ
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
This is a square, diagonal matrix of dimensionalitymin(M,N) × min(M,N).
The diagonal consists of the singular values of C .
The magnitude of the singular value measures the importance ofthe corresponding semantic dimension.
We’ll make use of this by omitting unimportant dimensions.
Schutze: Latent Semantic Indexing 10 / 31
![Page 30: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/30.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix V T
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.226 0.00 0.00 0.00 -0.58 0.58 0.58
Schutze: Latent Semantic Indexing 11 / 31
![Page 31: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/31.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix V T
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.226 0.00 0.00 0.00 -0.58 0.58 0.58
N ×N square matrix.
Schutze: Latent Semantic Indexing 11 / 31
![Page 32: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/32.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix V T
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.226 0.00 0.00 0.00 -0.58 0.58 0.58
N ×N square matrix. Drop row 6 – only want min(M,N) LSI dims.
Schutze: Latent Semantic Indexing 11 / 31
![Page 33: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/33.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix V T
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.226 0.00 0.00 0.00 -0.58 0.58 0.58
N ×N square matrix. Drop row 6 – only want min(M,N) LSI dims.
Again: This is an orthonormal matrix: (i) Column vectors haveunit length. (ii) Any two distinct column vectors are orthogonal toeach other.
Schutze: Latent Semantic Indexing 11 / 31
![Page 34: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/34.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix V T
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.226 0.00 0.00 0.00 -0.58 0.58 0.58
N ×N square matrix. Drop row 6 – only want min(M,N) LSI dims.
Again: This is an orthonormal matrix: (i) Column vectors haveunit length. (ii) Any two distinct column vectors are orthogonal toeach other.
These are again the semantic dimensions from matrices U and Σthat capture distinct topics like politics, sports, economics.
Schutze: Latent Semantic Indexing 11 / 31
![Page 35: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/35.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : The matrix V T
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.226 0.00 0.00 0.00 -0.58 0.58 0.58
N ×N square matrix. Drop row 6 – only want min(M,N) LSI dims.
Again: This is an orthonormal matrix: (i) Column vectors haveunit length. (ii) Any two distinct column vectors are orthogonal toeach other.
These are again the semantic dimensions from matrices U and Σthat capture distinct topics like politics, sports, economics.
Each vij in the matrix indicates how strongly related document i isto the topic represented by semantic dimension j .
Schutze: Latent Semantic Indexing 11 / 31
![Page 36: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/36.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Example of C = UΣV T : All four matrices
C d1 d2 d3 d4 d5 d6
ship 1.00 0.00 1.00 0.00 0.00 0.00boat 0.00 1.00 0.00 0.00 0.00 0.00ocean 1.00 1.00 0.00 0.00 0.00 0.00wood 1.00 0.00 0.00 1.00 1.00 0.00tree 0.00 0.00 0.00 1.00 0.00 1.00
=
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
×
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
×
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.22
LSI is decomposition of C into a representation of the terms, a representation of the documentsand a representation of the importance of the “semantic” dimensions.
Schutze: Latent Semantic Indexing 12 / 31
![Page 37: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/37.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Summary
Schutze: Latent Semantic Indexing 13 / 31
![Page 38: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/38.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Summary
We’ve decomposed the term-document matrix C into aproduct of three matrices: UΣV T .
Schutze: Latent Semantic Indexing 13 / 31
![Page 39: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/39.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Summary
We’ve decomposed the term-document matrix C into aproduct of three matrices: UΣV T .
The term matrix U – consists of one (row) vector for eachterm
Schutze: Latent Semantic Indexing 13 / 31
![Page 40: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/40.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Summary
We’ve decomposed the term-document matrix C into aproduct of three matrices: UΣV T .
The term matrix U – consists of one (row) vector for eachterm
The document matrix V T – consists of one (column) vectorfor each document
Schutze: Latent Semantic Indexing 13 / 31
![Page 41: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/41.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Summary
We’ve decomposed the term-document matrix C into aproduct of three matrices: UΣV T .
The term matrix U – consists of one (row) vector for eachterm
The document matrix V T – consists of one (column) vectorfor each document
The singular value matrix Σ – diagonal matrix with singularvalues, reflecting importance of each dimension
Schutze: Latent Semantic Indexing 13 / 31
![Page 42: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/42.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Summary
We’ve decomposed the term-document matrix C into aproduct of three matrices: UΣV T .
The term matrix U – consists of one (row) vector for eachterm
The document matrix V T – consists of one (column) vectorfor each document
The singular value matrix Σ – diagonal matrix with singularvalues, reflecting importance of each dimension
Next: Why are we doing this?
Schutze: Latent Semantic Indexing 13 / 31
![Page 43: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/43.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Outline
1 Singular Value Decomposition
2 Dimensionality reduction
3 Latent Semantic Indexing
Schutze: Latent Semantic Indexing 14 / 31
![Page 44: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/44.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Schutze: Latent Semantic Indexing 15 / 31
![Page 45: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/45.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
Schutze: Latent Semantic Indexing 15 / 31
![Page 46: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/46.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
Schutze: Latent Semantic Indexing 15 / 31
![Page 47: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/47.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
Schutze: Latent Semantic Indexing 15 / 31
![Page 48: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/48.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
be noise – in that case, reduced LSI is a better representationbecause it is less noisy.
Schutze: Latent Semantic Indexing 15 / 31
![Page 49: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/49.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
be noise – in that case, reduced LSI is a better representationbecause it is less noisy.make things dissimilar that should be similar – again, thereduced LSI representation is a better representation because itrepresents similarity better.
Schutze: Latent Semantic Indexing 15 / 31
![Page 50: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/50.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
be noise – in that case, reduced LSI is a better representationbecause it is less noisy.make things dissimilar that should be similar – again, thereduced LSI representation is a better representation because itrepresents similarity better.
Analogy for “fewer details is better”
Schutze: Latent Semantic Indexing 15 / 31
![Page 51: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/51.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
be noise – in that case, reduced LSI is a better representationbecause it is less noisy.make things dissimilar that should be similar – again, thereduced LSI representation is a better representation because itrepresents similarity better.
Analogy for “fewer details is better”
Image of a blue flower
Schutze: Latent Semantic Indexing 15 / 31
![Page 52: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/52.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
be noise – in that case, reduced LSI is a better representationbecause it is less noisy.make things dissimilar that should be similar – again, thereduced LSI representation is a better representation because itrepresents similarity better.
Analogy for “fewer details is better”
Image of a blue flowerImage of a yellow flower
Schutze: Latent Semantic Indexing 15 / 31
![Page 53: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/53.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How we use the SVD in LSI
Key property: Each singular value tells us how important itsdimension is.
By setting less important dimensions to zero, we keep theimportant information, but get rid of the “details”.
These details may
be noise – in that case, reduced LSI is a better representationbecause it is less noisy.make things dissimilar that should be similar – again, thereduced LSI representation is a better representation because itrepresents similarity better.
Analogy for “fewer details is better”
Image of a blue flowerImage of a yellow flowerOmitting color makes is easier to see the similarity
Schutze: Latent Semantic Indexing 15 / 31
![Page 54: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/54.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Reducing the dimensionality to 2
U 1 2 3 4 5
ship −0.44 −0.30 0.00 0.00 0.00boat −0.13 −0.33 0.00 0.00 0.00ocean −0.48 −0.51 0.00 0.00 0.00wood −0.70 0.35 0.00 0.00 0.00tree −0.26 0.65 0.00 0.00 0.00
Σ2 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 0.00 0.00 0.004 0.00 0.00 0.00 0.00 0.005 0.00 0.00 0.00 0.00 0.00
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.00 0.00 0.00 0.00 0.00 0.004 0.00 0.00 0.00 0.00 0.00 0.005 0.00 0.00 0.00 0.00 0.00 0.00
Schutze: Latent Semantic Indexing 16 / 31
![Page 55: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/55.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Reducing the dimensionality to 2
U 1 2 3 4 5
ship −0.44 −0.30 0.00 0.00 0.00boat −0.13 −0.33 0.00 0.00 0.00ocean −0.48 −0.51 0.00 0.00 0.00wood −0.70 0.35 0.00 0.00 0.00tree −0.26 0.65 0.00 0.00 0.00
Σ2 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 0.00 0.00 0.004 0.00 0.00 0.00 0.00 0.005 0.00 0.00 0.00 0.00 0.00
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.00 0.00 0.00 0.00 0.00 0.004 0.00 0.00 0.00 0.00 0.00 0.005 0.00 0.00 0.00 0.00 0.00 0.00
Actually, we
only zero out
singular values
in Σ. This has
the effect of
setting the
corresponding
dimensions in
U and VT to
zero when
computing the
product C =
UΣVT .
Schutze: Latent Semantic Indexing 16 / 31
![Page 56: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/56.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Reducing the dimensionality to 2
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
=
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
×
Σ2 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 0.00 0.00 0.004 0.00 0.00 0.00 0.00 0.005 0.00 0.00 0.00 0.00 0.00
×
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.22
Schutze: Latent Semantic Indexing 17 / 31
![Page 57: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/57.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Recall unreduced decomposition C = UΣV T
C d1 d2 d3 d4 d5 d6
ship 1.00 0.00 1.00 0.00 0.00 0.00boat 0.00 1.00 0.00 0.00 0.00 0.00ocean 1.00 1.00 0.00 0.00 0.00 0.00wood 1.00 0.00 0.00 1.00 1.00 0.00tree 0.00 0.00 0.00 1.00 0.00 1.00
=
U 1 2 3 4 5
ship −0.44 −0.30 0.57 0.58 0.25boat −0.13 −0.33 −0.59 0.00 0.73ocean −0.48 −0.51 −0.37 0.00 −0.61wood −0.70 0.35 0.15 −0.58 0.16tree −0.26 0.65 −0.41 0.58 −0.09
×
Σ 1 2 3 4 5
1 2.16 0.00 0.00 0.00 0.002 0.00 1.59 0.00 0.00 0.003 0.00 0.00 1.28 0.00 0.004 0.00 0.00 0.00 1.00 0.005 0.00 0.00 0.00 0.00 0.39
×
V T d1 d2 d3 d4 d5 d6
1 −0.75 −0.28 −0.20 −0.45 −0.33 −0.122 −0.29 −0.53 −0.19 0.63 0.22 0.413 0.28 −0.75 0.45 −0.20 0.12 −0.334 0.00 0.00 0.58 0.00 −0.58 0.585 −0.53 0.29 0.63 0.19 0.41 −0.22
Schutze: Latent Semantic Indexing 18 / 31
![Page 58: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/58.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Original matrix C vs. reduced C2 = UΣ2VT
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
Schutze: Latent Semantic Indexing 19 / 31
![Page 59: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/59.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Original matrix C vs. reduced C2 = UΣ2VT
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
We can view
C2 as a two-
dimensional
representation
of the matrix
C . We have
performed a
dimensionality
reduction to
two
dimensions.
Schutze: Latent Semantic Indexing 19 / 31
![Page 60: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/60.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why the reduced matrix C2 is better than C
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
Schutze: Latent Semantic Indexing 20 / 31
![Page 61: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/61.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why the reduced matrix C2 is better than C
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
Similarity of d2
and d3 in the
original space:
0.
Schutze: Latent Semantic Indexing 20 / 31
![Page 62: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/62.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why the reduced matrix C2 is better than C
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
Similarity of d2
and d3 in the
original space:
0.
Similarity of
d2 and d3 in
the reduced
space: 0.52 ∗
0.28 + 0.36 ∗0.16 + 0.72 ∗
0.36 + 0.12 ∗0.20+−0.39∗−0.08 ≈
0.52
Schutze: Latent Semantic Indexing 20 / 31
![Page 63: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/63.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why the reduced matrix C2 is better than C
C d1 d2 d3 d4 d5 d6
ship 1 0 1 0 0 0boat 0 1 0 0 0 0ocean 1 1 0 0 0 0wood 1 0 0 1 1 0tree 0 0 0 1 0 1
C2 d1 d2 d3 d4 d5 d6
ship 0.85 0.52 0.28 0.13 0.21 −0.08boat 0.36 0.36 0.16 −0.20 −0.02 −0.18ocean 1.01 0.72 0.36 −0.04 0.16 −0.21wood 0.97 0.12 0.20 1.03 0.62 0.41tree 0.12 −0.39 −0.08 0.90 0.41 0.49
“boat” and
“ship” are
semantically
similar. The
“reduced”
similarity mea-
sure reflects
this.
Schutze: Latent Semantic Indexing 20 / 31
![Page 64: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/64.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Outline
1 Singular Value Decomposition
2 Dimensionality reduction
3 Latent Semantic Indexing
Schutze: Latent Semantic Indexing 21 / 31
![Page 65: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/65.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
Schutze: Latent Semantic Indexing 22 / 31
![Page 66: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/66.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
Schutze: Latent Semantic Indexing 22 / 31
![Page 67: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/67.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
. . . but are not similar in the vector space (because they usedifferent words) . . .
Schutze: Latent Semantic Indexing 22 / 31
![Page 68: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/68.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
. . . but are not similar in the vector space (because they usedifferent words) . . .
. . . and re-represents them in a reduced vector space . . .
Schutze: Latent Semantic Indexing 22 / 31
![Page 69: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/69.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
. . . but are not similar in the vector space (because they usedifferent words) . . .
. . . and re-represents them in a reduced vector space . . .
. . . in which they have higher similarity.
Schutze: Latent Semantic Indexing 22 / 31
![Page 70: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/70.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
. . . but are not similar in the vector space (because they usedifferent words) . . .
. . . and re-represents them in a reduced vector space . . .
. . . in which they have higher similarity.
Thus, LSI addresses the problems of synonymy and semanticrelatedness.
Schutze: Latent Semantic Indexing 22 / 31
![Page 71: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/71.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
. . . but are not similar in the vector space (because they usedifferent words) . . .
. . . and re-represents them in a reduced vector space . . .
. . . in which they have higher similarity.
Thus, LSI addresses the problems of synonymy and semanticrelatedness.
Standard vector space: Synonyms contribute nothing todocument similarity.
Schutze: Latent Semantic Indexing 22 / 31
![Page 72: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/72.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Why we use LSI in information retrieval
LSI takes documents that are semantically similar (= talkabout the same topics), . . .
. . . but are not similar in the vector space (because they usedifferent words) . . .
. . . and re-represents them in a reduced vector space . . .
. . . in which they have higher similarity.
Thus, LSI addresses the problems of synonymy and semanticrelatedness.
Standard vector space: Synonyms contribute nothing todocument similarity.
Desired effect of LSI: Synonyms contribute strongly todocument similarity.
Schutze: Latent Semantic Indexing 22 / 31
![Page 73: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/73.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How LSI addresses synonymy and semantic relatedness
The dimensionality reduction forces us to omit a lot of“detail”.
Schutze: Latent Semantic Indexing 23 / 31
![Page 74: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/74.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How LSI addresses synonymy and semantic relatedness
The dimensionality reduction forces us to omit a lot of“detail”.
We have to map differents words (= different dimensions ofthe full space) to the same dimension in the reduced space.
Schutze: Latent Semantic Indexing 23 / 31
![Page 75: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/75.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How LSI addresses synonymy and semantic relatedness
The dimensionality reduction forces us to omit a lot of“detail”.
We have to map differents words (= different dimensions ofthe full space) to the same dimension in the reduced space.
The “cost” of mapping synonyms to the same dimension ismuch less than the cost of collapsing unrelated words.
Schutze: Latent Semantic Indexing 23 / 31
![Page 76: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/76.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How LSI addresses synonymy and semantic relatedness
The dimensionality reduction forces us to omit a lot of“detail”.
We have to map differents words (= different dimensions ofthe full space) to the same dimension in the reduced space.
The “cost” of mapping synonyms to the same dimension ismuch less than the cost of collapsing unrelated words.
SVD selects the “least costly” mapping (see below).
Schutze: Latent Semantic Indexing 23 / 31
![Page 77: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/77.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How LSI addresses synonymy and semantic relatedness
The dimensionality reduction forces us to omit a lot of“detail”.
We have to map differents words (= different dimensions ofthe full space) to the same dimension in the reduced space.
The “cost” of mapping synonyms to the same dimension ismuch less than the cost of collapsing unrelated words.
SVD selects the “least costly” mapping (see below).
Thus, it will map synonyms to the same dimension.
Schutze: Latent Semantic Indexing 23 / 31
![Page 78: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/78.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
How LSI addresses synonymy and semantic relatedness
The dimensionality reduction forces us to omit a lot of“detail”.
We have to map differents words (= different dimensions ofthe full space) to the same dimension in the reduced space.
The “cost” of mapping synonyms to the same dimension ismuch less than the cost of collapsing unrelated words.
SVD selects the “least costly” mapping (see below).
Thus, it will map synonyms to the same dimension.
But it will avoid doing that for unrelated words.
Schutze: Latent Semantic Indexing 23 / 31
![Page 79: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/79.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Comparison to other approaches
Schutze: Latent Semantic Indexing 24 / 31
![Page 80: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/80.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Comparison to other approaches
Relevance feedback and query expansion are used to increaserecall in information retrieval – if query and documents haveno terms in common.
Schutze: Latent Semantic Indexing 24 / 31
![Page 81: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/81.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Comparison to other approaches
Relevance feedback and query expansion are used to increaserecall in information retrieval – if query and documents haveno terms in common.
LSI increases recall and hurts precision.
Schutze: Latent Semantic Indexing 24 / 31
![Page 82: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/82.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Comparison to other approaches
Relevance feedback and query expansion are used to increaserecall in information retrieval – if query and documents haveno terms in common.
LSI increases recall and hurts precision.
Thus, it addresses the same problems as (pseudo) relevancefeedback and query expansion . . .
Schutze: Latent Semantic Indexing 24 / 31
![Page 83: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/83.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI: Comparison to other approaches
Relevance feedback and query expansion are used to increaserecall in information retrieval – if query and documents haveno terms in common.
LSI increases recall and hurts precision.
Thus, it addresses the same problems as (pseudo) relevancefeedback and query expansion . . .
. . . and it has the same problems.
Schutze: Latent Semantic Indexing 24 / 31
![Page 84: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/84.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Schutze: Latent Semantic Indexing 25 / 31
![Page 85: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/85.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Schutze: Latent Semantic Indexing 25 / 31
![Page 86: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/86.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Reduce the space and compute reduced documentrepresentations
Schutze: Latent Semantic Indexing 25 / 31
![Page 87: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/87.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Reduce the space and compute reduced documentrepresentations
Map the query into the reduced space ~qk = Σ−1k
UTk ~q.
Schutze: Latent Semantic Indexing 25 / 31
![Page 88: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/88.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Reduce the space and compute reduced documentrepresentations
Map the query into the reduced space ~qk = Σ−1k
UTk ~q.
This follows from: Ck = UΣkV T ⇒ Σ−1k
UTC = V Tk
Schutze: Latent Semantic Indexing 25 / 31
![Page 89: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/89.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Reduce the space and compute reduced documentrepresentations
Map the query into the reduced space ~qk = Σ−1k
UTk ~q.
This follows from: Ck = UΣkV T ⇒ Σ−1k
UTC = V Tk
Compute similarity of qk with all reduced documents in Vk .
Schutze: Latent Semantic Indexing 25 / 31
![Page 90: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/90.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Reduce the space and compute reduced documentrepresentations
Map the query into the reduced space ~qk = Σ−1k
UTk ~q.
This follows from: Ck = UΣkV T ⇒ Σ−1k
UTC = V Tk
Compute similarity of qk with all reduced documents in Vk .
Output ranked list of documents as usual
Schutze: Latent Semantic Indexing 25 / 31
![Page 91: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/91.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Implementation
Compute SVD of term-document matrix
Reduce the space and compute reduced documentrepresentations
Map the query into the reduced space ~qk = Σ−1k
UTk ~q.
This follows from: Ck = UΣkV T ⇒ Σ−1k
UTC = V Tk
Compute similarity of qk with all reduced documents in Vk .
Output ranked list of documents as usual
Exercise: What is the fundamental problem with thisapproach?
Schutze: Latent Semantic Indexing 25 / 31
![Page 92: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/92.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
Schutze: Latent Semantic Indexing 26 / 31
![Page 93: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/93.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Schutze: Latent Semantic Indexing 26 / 31
![Page 94: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/94.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Keeping the k largest singular values and setting all others tozero gives you the optimal approximation of the originalmatrix C . Eckart-Young theorem
Schutze: Latent Semantic Indexing 26 / 31
![Page 95: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/95.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Keeping the k largest singular values and setting all others tozero gives you the optimal approximation of the originalmatrix C . Eckart-Young theorem
Optimal: no other matrix of the same rank (= with the sameunderlying dimensionality) approximates C better.
Schutze: Latent Semantic Indexing 26 / 31
![Page 96: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/96.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Keeping the k largest singular values and setting all others tozero gives you the optimal approximation of the originalmatrix C . Eckart-Young theorem
Optimal: no other matrix of the same rank (= with the sameunderlying dimensionality) approximates C better.
Measure of approximation is Frobenius norm:
||C ||F =√
∑
i
∑
j c2ij
Schutze: Latent Semantic Indexing 26 / 31
![Page 97: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/97.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Keeping the k largest singular values and setting all others tozero gives you the optimal approximation of the originalmatrix C . Eckart-Young theorem
Optimal: no other matrix of the same rank (= with the sameunderlying dimensionality) approximates C better.
Measure of approximation is Frobenius norm:
||C ||F =√
∑
i
∑
j c2ij
So LSI uses the “best possible” matrix.
Schutze: Latent Semantic Indexing 26 / 31
![Page 98: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/98.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Keeping the k largest singular values and setting all others tozero gives you the optimal approximation of the originalmatrix C . Eckart-Young theorem
Optimal: no other matrix of the same rank (= with the sameunderlying dimensionality) approximates C better.
Measure of approximation is Frobenius norm:
||C ||F =√
∑
i
∑
j c2ij
So LSI uses the “best possible” matrix.
There is only one best possible matrix – unique solution(modulo signs).
Schutze: Latent Semantic Indexing 26 / 31
![Page 99: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/99.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Optimality
SVD is optimal in the following sense.
Keeping the k largest singular values and setting all others tozero gives you the optimal approximation of the originalmatrix C . Eckart-Young theorem
Optimal: no other matrix of the same rank (= with the sameunderlying dimensionality) approximates C better.
Measure of approximation is Frobenius norm:
||C ||F =√
∑
i
∑
j c2ij
So LSI uses the “best possible” matrix.
There is only one best possible matrix – unique solution(modulo signs).
Caveat: There is only a tenuous relationship between theFrobenius norm and cosine similarity between documents.
Schutze: Latent Semantic Indexing 26 / 31
![Page 100: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/100.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Data for graphical illustration of LSI
Schutze: Latent Semantic Indexing 27 / 31
![Page 101: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/101.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Data for graphical illustration of LSI
c1 Human machine interface for lab abc computer applicationsc2 A survey of user opinion of computer system response timec3 The EPS user interface management systemc4 System and human system engineering testing of EPSc5 Relation of user perceived response time to error measurementm1 The generation of random binary unordered treesm2 The intersection graph of paths in treesm3 Graph minors IV Widths of trees and well quasi orderingm4 Graph minors A survey
Schutze: Latent Semantic Indexing 27 / 31
![Page 102: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/102.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Data for graphical illustration of LSI
c1 Human machine interface for lab abc computer applicationsc2 A survey of user opinion of computer system response timec3 The EPS user interface management systemc4 System and human system engineering testing of EPSc5 Relation of user perceived response time to error measurementm1 The generation of random binary unordered treesm2 The intersection graph of paths in treesm3 Graph minors IV Widths of trees and well quasi orderingm4 Graph minors A survey
The matrix C
c1 c2 c3 c4 c5 m1 m2 m3 m4human 1 0 0 1 0 0 0 0 0interface 1 0 1 0 0 0 0 0 0computer 1 1 0 0 0 0 0 0 0user 0 1 1 0 1 0 0 0 0system 0 1 1 2 0 0 0 0 0response 0 1 0 0 1 0 0 0 0time 0 1 0 0 1 0 0 0 0EPS 0 0 1 1 0 0 0 0 0survey 0 1 0 0 0 0 0 0 1trees 0 0 0 0 0 1 1 1 0graph 0 0 0 0 0 0 1 1 1minors 0 0 0 0 0 0 0 1 1
Schutze: Latent Semantic Indexing 27 / 31
![Page 103: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/103.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Graphical illustration of LSI: Plot of C2
2-dimensional plot ofC2 (scaled dimensions).Circles = terms. Opensquares = documents(component terms inparentheses). q = query“human computer inter-action”.
The dotted cone represents the region whose points are within a cosine of.9 from q . All documents about human-computer documents (c1-c5) arenear q, even c3/c5 although they share no terms. None of the graph theorydocuments (m1-m4) are near q.
Schutze: Latent Semantic Indexing 28 / 31
![Page 104: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/104.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
LSI performs better than vector space on MED collection
LSI-100 = LSI reduced to 100 dimensions; SMART = SMARTimplementation of vector space model
Schutze: Latent Semantic Indexing 29 / 31
![Page 105: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/105.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Take-away
Singular Value Decomposition (SVD): The math behind LSI
SVD used for dimensionality reduction
Latent Semantic Indexing (LSI): SVD used in informationretrieval
Schutze: Latent Semantic Indexing 30 / 31
![Page 106: Introduction to Information Retrieval ` `%%%`# ` ~~~false [0.5cm] … · distinct topics like politics, sports, economics. 2 = water/land Each number uij in the matrix indicates how](https://reader035.vdocuments.site/reader035/viewer/2022070802/5f02dde57e708231d40666d9/html5/thumbnails/106.jpg)
Singular Value Decomposition Dimensionality reduction Latent Semantic Indexing
Resources
Chapter 18 of Introduction to Information Retrieval
Resources at http://informationretrieval.org/essir2011
Latent semantic indexing by Deerwester et al. (original paper)Probabilistic LSI by HofmannWord space: LSI for words
Schutze: Latent Semantic Indexing 31 / 31