ranking with high-order and missing information m. pawan kumar ecole centrale paris aseem behlpuneet...
TRANSCRIPT
![Page 1: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/1.jpg)
Ranking with High-Orderand Missing Information
M. Pawan Kumar
Ecole Centrale Paris
Aseem Behl Puneet Dokania Pritish Mohapatra C. V. Jawahar
![Page 2: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/2.jpg)
PASCAL VOC“Jumping” Classification
Features
Processing
Training
Classifier
![Page 3: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/3.jpg)
PASCAL VOC
Features
Processing
Training
Classifier
Think of a classifier !!!
“Jumping” Classification
✗
![Page 4: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/4.jpg)
PASCAL VOC
Features
Processing
Training
Classifier
Think of a classifier !!!✗
“Jumping” Ranking
![Page 5: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/5.jpg)
Ranking vs. ClassificationRank 1 Rank 2 Rank 3
Rank 4 Rank 5 Rank 6
Average Precision = 1
![Page 6: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/6.jpg)
Ranking vs. ClassificationRank 1 Rank 2 Rank 3
Rank 4 Rank 5 Rank 6
Average Precision = 1 Accuracy = 1= 0.92 = 0.67= 0.81
![Page 7: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/7.jpg)
Ranking vs. Classification
Ranking is not the same as classification
Average precision is not the same as accuracy
Should we use 0-1 loss based classifiers?
Or should we use AP loss based rankers?
![Page 8: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/8.jpg)
• Optimizing Average Precision (AP-SVM)
• High-Order Information
• Missing Information
Yue, Finley, Radlinski and Joachims, SIGIR 2007
Outline
![Page 9: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/9.jpg)
Problem FormulationSingle Input X
Φ(xi)for all i P
Φ(xk)for all k N
![Page 10: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/10.jpg)
Problem FormulationSingle Output R
Rik =
+1 if i is better ranked than k
-1 if k is better ranked than i
![Page 11: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/11.jpg)
Problem FormulationScoring Function
si(w) = wTΦ(xi) for all i P
sk(w) = wTΦ(xk) for all k N
S(X,R;w) = Σi P Σk N Rik(si(w) - sk(w))
![Page 12: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/12.jpg)
Ranking at Test-Time
R(w) = maxR S(X,R;w)
x1
Sort samples according to individual scores s i(w)
x2 x3 x4 x5 x6 x7 x8
![Page 13: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/13.jpg)
Learning FormulationLoss Function
Δ(R*,R(w))
= 1 – AP of rank R(w)
Non-convex
Parameter cannot be regularized
![Page 14: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/14.jpg)
Learning FormulationUpper Bound of Loss Function
Δ(R*,R(w))S(X,R(w);w) + - S(X,R(w);w)
![Page 15: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/15.jpg)
Learning FormulationUpper Bound of Loss Function
Δ(R*,R(w))S(X,R(w);w) + - S(X,R*;w)
![Page 16: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/16.jpg)
Learning FormulationUpper Bound of Loss Function
Δ(R*,R)S(X,R;w) + - S(X,R*;w)maxR
Convex Parameter can be regularized
minw ||w||2 + C ξ
S(X,R;w) + Δ(R*,R) - S(X,R*;w) ≤ ξ, for all R
![Page 17: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/17.jpg)
Optimization for LearningCutting Plane Computation
maxR S(X,R;w) + Δ(R*,R)
x1 x2 x3 x4 x5 x6 x7 x8
Sort positive samples according to scores si(w)
Sort negative samples according to scores sk(w)
Find best rank of each negative sample independently
![Page 18: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/18.jpg)
Optimization for LearningCutting Plane Computation
Train
ing
Tim
e
0-1
AP
5x slowerAP
Slightly faster
Mohapatra, Jawahar and Kumar, NIPS 2014
![Page 19: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/19.jpg)
Experiments
PASCAL VOC 2011Jumping
Phoning
Playing Instrument
Reading
Riding Bike
Riding Horse
Running
Taking Photo
Using Computer
Walking
Images Classes
10 ranking tasks
Cross-validation
Poselets Features
![Page 20: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/20.jpg)
AP-SVM vs. SVMPASCAL VOC ‘test’ Dataset
Differencein AP
Better in 8 classes, tied in 2 classes
![Page 21: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/21.jpg)
AP-SVM vs. SVMFolds of PASCAL VOC ‘trainval’ Dataset
Differencein AP
AP-SVM is statistically better in 3 classes
SVM is statistically better in 0 classes
![Page 22: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/22.jpg)
• Optimizing Average Precision
• High-Order Information (HOAP-SVM)
• Missing Information
Dokania, Behl, Jawahar and Kumar, ECCV 2014
Outline
![Page 23: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/23.jpg)
![Page 24: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/24.jpg)
![Page 25: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/25.jpg)
High-Order Information
• People perform similar actions
• People strike similar poses
• Objects are of same/similar sizes
• “Friends” have similar habits
• How can we use them for ranking? classification
![Page 26: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/26.jpg)
Problem Formulationx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
Ψ(x,y) = Ψ1(x,y)
Ψ2(x,y)
Unary Features
Pairwise Features
![Page 27: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/27.jpg)
Learning Formulationx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
Δ(y*,y) = Fraction of incorrectly classified persons
![Page 28: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/28.jpg)
Optimization for Learningx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
maxy wTΨ(x,y) + Δ(y*,y)
Graph Cuts (if supermodular)
LP Relaxation, or exhaustive search
![Page 29: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/29.jpg)
Classificationx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
maxy wTΨ(x,y)
Graph Cuts (if supermodular)
LP Relaxation, or exhaustive search
![Page 30: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/30.jpg)
Ranking?x
Input x = {x1,x2,x3}
Output y = {-1,+1}3
Use difference of max-marginals
![Page 31: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/31.jpg)
Max-Marginal for Positive Classx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
mm+(i;w) = maxy,yi=+1 wTΨ(x,y)
Best possible score when person i is positive
Convex in w
![Page 32: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/32.jpg)
Max-Marginal for Negative Classx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
mm-(i;w) = maxy,yi=-1 wTΨ(x,y)
Best possible score when person i is negative
Convex in w
![Page 33: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/33.jpg)
Rankingx
Input x = {x1,x2,x3}
Output y = {-1,+1}3
si(w) = mm+(i;w) – mm-(i;w)
Difference-of-Convex in w
Use difference of max-marginals HOB-SVM
![Page 34: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/34.jpg)
Ranking
si(w) = mm+(i;w) – mm-(i;w)
Why not optimize AP directly?
High Order AP-SVM
HOAP-SVM
![Page 35: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/35.jpg)
Problem FormulationSingle Input X
Φ(xi)for all i P
Φ(xk)for all k N
![Page 36: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/36.jpg)
Problem FormulationSingle Input R
Rik =
+1 if i is better ranked than k
-1 if k is better ranked than i
![Page 37: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/37.jpg)
Problem FormulationScoring Function
si(w) = mm+(i;w) – mm-(i;w) for all i P
sk(w) = mm+(k;w) – mm-(k;w) for all k N
S(X,R;w) = Σi P Σk N Rik(si(w) - sk(w))
![Page 38: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/38.jpg)
Ranking at Test-Time
R(w) = maxR S(X,R;w)
x1
Sort samples according to individual scores s i(w)
x2 x3 x4 x5 x6 x7 x8
![Page 39: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/39.jpg)
Learning FormulationLoss Function
Δ(R*,R(w)) = 1 – AP of rank R(w)
![Page 40: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/40.jpg)
Learning FormulationUpper Bound of Loss Function
minw ||w||2 + C ξ
S(X,R;w) + Δ(R*,R) - S(X,R*;w) ≤ ξ, for all R
![Page 41: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/41.jpg)
Optimization for Learning
Difference-of-convex program
Kohli and Torr, ECCV 2006
Very efficient CCCP
Linearization step by Dynamic Graph Cuts
Update step equivalent to AP-SVM
![Page 42: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/42.jpg)
Experiments
PASCAL VOC 2011Jumping
Phoning
Playing Instrument
Reading
Riding Bike
Riding Horse
Running
Taking Photo
Using Computer
Walking
Images Classes
10 ranking tasks
Cross-validation
Poselets Features
![Page 43: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/43.jpg)
HOB-SVM vs. AP-SVMPASCAL VOC ‘test’ Dataset
Differencein AP
Better in 4, worse in 3 and tied in 3 classes
![Page 44: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/44.jpg)
HOB-SVM vs. AP-SVMFolds of PASCAL VOC ‘trainval’ Dataset
Differencein AP
HOB-SVM is statistically better in 0 classes
AP-SVM is statistically better in 0 classes
![Page 45: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/45.jpg)
HOAP-SVM vs. AP-SVMPASCAL VOC ‘test’ Dataset
Better in 7, worse in 2 and tied in 1 class
Differencein AP
![Page 46: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/46.jpg)
HOAP-SVM vs. AP-SVMFolds of PASCAL VOC ‘trainval’ Dataset
HOAP-SVM is statistically better in 4 classes
AP-SVM is statistically better in 0 classes
Differencein AP
![Page 47: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/47.jpg)
• Optimizing Average Precision
• High-Order Information
• Missing Information (Latent-AP-SVM)
Outline
Behl, Jawahar and Kumar, CVPR 2014
![Page 48: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/48.jpg)
Fully Supervised Learning
![Page 49: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/49.jpg)
Weakly Supervised Learning
Rank images by relevance to ‘jumping’
![Page 50: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/50.jpg)
• Use Latent Structured SVM with AP loss– Unintuitive Prediction– Loose Upper Bound on Loss– NP-hard Optimization for Cutting Planes
• Carefully design a Latent-AP-SVM– Intuitive Prediction– Tight Upper Bound on Loss– Optimal Efficient Cutting Plane Computation
Two Approaches
![Page 51: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/51.jpg)
Results
![Page 52: Ranking with High-Order and Missing Information M. Pawan Kumar Ecole Centrale Paris Aseem BehlPuneet DokaniaPritish MohapatraC. V. Jawahar](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649dd05503460f94ac5814/html5/thumbnails/52.jpg)
Questions?
Code + Data Available