pairwise decomposition of image sequences for active...
TRANSCRIPT
![Page 1: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/1.jpg)
PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE MULTI-VIEW RECOGNITION(EXPERIMENT)
Dongguang You
1
![Page 2: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/2.jpg)
RECAP
➤ Pairwise Classification
2
![Page 3: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/3.jpg)
RECAP
➤ Pairwise Classification
➤ Next Best View selection/Trajectory Optimisation
3
![Page 4: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/4.jpg)
TRAJECTORY OPTIMISATION
➤ Goal: maximize
➤ At each step: find a trajectory that maximizes
4
X
i,j2Sequence
predictedCrossEntropy(i, j)
X
i2Observed,j2unobserved
predictedCrossEntropy(i, j)
![Page 5: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/5.jpg)
MOTIVATION
➤ Recall lambda in
➤ lambda only depends on the relative pose
Failure case:
➤ Predicted cross entropy of pairs in two trajectories: [1, 10, 1] and [3, 3, 3]
➤ Choose [1, 10, 1] over [3, 3, 3]
➤ Lambda for the three pairs in [1, 10, 1]: 0.4, 0.2, 0.4
➤ Sadly a small weight is assigned to the critical pair during classification
5
![Page 6: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/6.jpg)
Failure case
6
lambda = 0.2
lambda = 0.4 lambda = 0.4
predicted cross entropy = 10
predicted cross entropy = 1 predicted cross entropy = 1
V1
V3V2
![Page 7: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/7.jpg)
MOTIVATION
➤ Problem: lambda and predicted cross entropy may conflict
➤ Solution1: incorporate lambda into trajectory optimisation
➤ choose [3,3,3] over [1,10,1] given lambda = [0.4,0.2,0.4]
7
X
i2Observed,j2unobserved
�(i, j) ⇤ predictedCrossEntropy(i, j)
![Page 8: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/8.jpg)
8
lambda = 0.2
lambda = 0.4 lambda = 0.4
predicted cross entropy = 10
predicted cross entropy = 1 predicted cross entropy = 1
X
i2Observed,j2unobserved
�(i, j) ⇤ predictedCrossEntropy(i, j)
V1
V3V2
![Page 9: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/9.jpg)
MOTIVATION
➤ Problem: lambda and predicted cross entropy conflict
➤ Solution2: replace lambda with predicted cross entropy
➤ choose [1,10,1] over [3,3,3], and assign a weight = [1,10,1]/12 to the 3 pairs
9
f(y|w1...wN ) =i=NX
i=1
predictedCE(wi) ⇤ p(y|wi)
![Page 10: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/10.jpg)
10
predicted cross entropy = 10
predicted cross entropy = 1 predicted cross entropy = 1
V1
V3V2
f(y|w1...wN ) =i=NX
i=1
predictedCE(wi) ⇤ p(y|wi)
lambda = 0.2
lambda = 0.4 lambda = 0.4
![Page 11: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/11.jpg)
EXPERIMENT SETUP
➤ Simplified setting
➤ binary classification
➤ relative poses are either good or bad
➤ consider testing data of one label
➤ Simulate the activation of the pairwise classification net
➤ assuming the activation follows Gaussian distribution
11
![Page 12: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/12.jpg)
ACTIVATION SIMULATION
12
Simulated Activation of True label
Simulated Activation of False label
![Page 13: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/13.jpg)
Good relative pose
13
For True label: Gaussian(10, 0.5)
For False label: Gaussian(0, 0.5)
Good
![Page 14: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/14.jpg)
Bad relative pose
14
For True label: Gaussian(0.5, 0.5)
For False label: Gaussian(0, 0.5)
bad
![Page 15: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/15.jpg)
RELATIVE POSE SIMULATION
For each test sample
➤ 4*4 grids of viewpoints
➤ 120 pairs
➤ 60 pairs in good relative pose, 60 pairs in bad relative pose
15
![Page 16: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/16.jpg)
CROSS ENTROPY PREDICTION SIMULATION
➤ Compute ground-truth cross entropy for each pair
➤ Predicted cross entropy ~ Gaussian(truth cross entropy, 0.5)
16
![Page 17: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/17.jpg)
CONVERTING LAMBDA AND CROSS ENTROPY
➤ lambda and cross entropy are negative
➤ converted lambda = lambda - min(lambda) - max(lambda)
➤ [-1.5, -1] -> [1, 1.5]
➤ [-2, -1.2 , -0.6] -> [0.6, 1.4, 2]
➤ Same for cross entropy
17
The author didn’t make this clear. He pick the pairs that are good by maximising the cross-entropy, so I assume he is using sum(p(x) * log(p’(x))), which is nonpositive
![Page 18: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/18.jpg)
EXPERIMENT 1
➤ Proposed: incorporate lambda into trajectory optimisation
➤ Baselines:
➤ Baseline 1: averaged classification
➤ Baseline 2: classification weighted with lambda
18
X
i2Observed,j2unobserved
�(i, j) ⇤ predictedCrossEntropy(i, j)
X
i2Observed,j2unobserved
predictedCrossEntropy(i, j)
![Page 19: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/19.jpg)
RESULT1
19
Baseline1: classification on average
Baseline2: classification weighted with lambdas
Proposed: Baseline2 + trajectory optimisation with lambdas
0.89 0.902 0.914 0.926 0.938 0.95
average softmax across 1000 samples
![Page 20: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/20.jpg)
EXPERIMENT2
➤ Proposed: use the predicted cross entropy as the weight, instead of lambda
➤ Baseline 1: averaged classification result
➤ Baseline 2: classification result weighted with lambda
➤ Baseline 3: classification result weighted with ground truth cross entropy
20
f(y|w1...wN ) =i=NX
i=1
predictedCE(wi) ⇤ p(y|wi)
![Page 21: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/21.jpg)
RESULT2
21
Baseline1: classification on average
Baseline2: classification weighted with lambdas
Baseline3: classification weighted with ground truth cross entropy
Proposed: classification weighted with predicted cross entropy
0.89 0.9 0.91 0.92 0.93 0.94
average softmax across 1000 samples
![Page 22: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/22.jpg)
EXPERIMENT2*
➤ What if the effect of relative pose is weaker?
The activation of correct label is modified:
➤ Good relative pose ~ Gaussian(1, 0.5) instead of Gaussian(10, 0.5)
➤ Bad relative pose ~ Gaussian(0.5,0.5), same as before
➤ What would the comparisons look like?
22
![Page 23: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/23.jpg)
RESULT2*
23
Baseline1: classification on average
Baseline2: classification weighted with lambdas
Baseline3: classification weighted with ground truth cross entropy
Proposed: classification weighted with predicted cross entropy
0.72 0.728 0.736 0.744 0.752 0.76
average softmax across 1000 samples
![Page 24: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/24.jpg)
LIMITATION OF THE PAIRWISE METHOD
➤ do not have a global view(as compared to “Look ahead before you leap”)
➤ range of entropy is (-inf, 0), hard to guarantee the accuracy of regression
24
![Page 25: PAIRWISE DECOMPOSITION OF IMAGE SEQUENCES FOR ACTIVE …vision.cs.utexas.edu/381V-fall2016/slides/you_expt.pdf · EXPERIMENT2* What if the effect of relative pose is weaker? The activation](https://reader033.vdocuments.site/reader033/viewer/2022041923/5e6cd6aab035d5735f726550/html5/thumbnails/25.jpg)
CONCLUSION
➤ When the effect of relative pose is strong
➤ incorporating lambda into trajectory optimisation might improve the prediction
➤ When the effect of relative pose is weak
➤ predicted cross entropy could be a better choice for weight than lambda
25