efficient sparse coding algorithms liang sun [email protected] arizona state university
Post on 19-Dec-2015
220 views
TRANSCRIPT
![Page 2: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/2.jpg)
Paper Source
Honglak Lee, Alexis Battle, Rajat Raina, Andrew Y. Ng. Efficient Sparse Coding Algorithms. NIPS 2007.
![Page 3: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/3.jpg)
Main Contributions
It proposes a new efficient algorithm to solve LASSO
A two stage optimization algorithm is proposed to the coding problem
![Page 4: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/4.jpg)
Sparse Coding Problem Sparse coding is a method for discovering good ba
sis vectors automatically using only unlabeled data
It is similar to PCA Given a training set of m vectors where
, we attempt to find a succinct representation for each xi using basis vectors
and a sparse vector such that Note that the basis can be overcomplete, i.e., n>k
1 2, ,[ ], mx xX x k
ix R
1 2, , , knbb b R nsR
1 21
[ , , , ]n
i j j nj
x s b b b b s
![Page 5: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/5.jpg)
The basis act as the principal components in PCA, and they capture a large number of patterns in the input data
The optimization problem in sparse coding
where and is a sparse penalty function
1 1, , , , ,[ ], [ ], [ 1, ]m n mx bB b S sX x s
![Page 6: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/6.jpg)
A New Algorithm to Solve LASSO
The formulation of LASSO
where x, y are vectors and A is a matrix
Basic idea of the new algorithm The difficulty of this problem lies in We guess the sign of each component
of x
1x
![Page 7: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/7.jpg)
Feature-sign Search Algorithm
![Page 8: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/8.jpg)
Proof of Feature-sign Search Algorithm – (I)
![Page 9: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/9.jpg)
Proof of Feature-sign Search Algorithm – (II)
![Page 10: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/10.jpg)
Proof of Feature-sign Search Algorithm – (III)
![Page 11: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/11.jpg)
Use Dual to simplify computation
Original Problem
![Page 12: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/12.jpg)
Experiment – (I) The comparison of Feature-sign search
algorithm and other algorithms for LASSO
![Page 13: Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649d3e5503460f94a16d33/html5/thumbnails/13.jpg)
Experiment – (II) The comparison of the two-stage
algorithm and other algorithms