bmc 2012

Foreground Detection via Robust Low Rank MatrixDecomposition including spatio-temporal Constraint

C. Guyon, T. Bouwmans and E. Zahzah

MIA Laboratory (Mathematics Images & Applications), University of La Rochelle, France—

Workshop BMC, Daejong Korea

November 5, 2012

C. Guyon, T. Bouwmans and E. Zahzah (MIA Laboratory (Mathematics Images & Applications), University of La Rochelle, France — Workshop BMC, Daejong Korea )Foreground Detection via Robust Low Rank Matrix Decomposition including spatio-temporal ConstraintNovember 5, 2012 1 / 27

Summary

1 Introduction and motivations on IRLS

2 Temporal constraint with an adapted Norm

3 Diagram flow and Spatial constraint

4 Experimental Results

5 Conclusion


Introduction and motivations

PurposeForeground detection : Segmentation of moving objects in videosequence acquired by a fixed camera.Background modeling : Modelization of all that is not movingobject.

Involved applicationsSurveillance cameraMotion capture

On the importanceCrucial Task : Often the first step of a full video surveillance system.

Strategy usedEigenbackground decomposition.


Eigenbackgrounds

Find an « ideal » subspace of the video sequence, which describes the bestas possible the (dynamic) background.

Fig.1 The common process of background subtraction via PCA (Principal ComponentAnalysis). At final step, an adaptative threshold is used to get a binary image.

Without a robust framework, the moving object may be absorbed in the model !C. Guyon, T. Bouwmans and E. Zahzah (MIA Laboratory (Mathematics Images & Applications), University of La Rochelle, France — Workshop BMC, Daejong Korea )Foreground Detection via Robust Low Rank Matrix Decomposition including spatio-temporal ConstraintNovember 5, 2012 4 / 27

Data Structure Transformation

First, we consider a video sequence as a matrix A ∈ Rn×m

n is the amount of pixels in a frame (∼ 106)m is the number of frames considered (∼ 200)


IRLS : Vector version (1)

The usual IRLS (Iteratively reweighted least squares) scheme for solveargmin

x||Ax − b||α is given by

D(i) = diag((ε+ |b − Ax (i)|)α−2)x (i+1) = (AtD(i)A)−1AtD(i)b

(1)

that a suitable IRLS method is convergent for 1 ≤ α < 3Other formulation,

r (i) = b − Ax (i)

D = diag((ε+ |r (i)|)α−2)y (i) = (A′DA)−1A′Dr (i)

x (i+1) = x (i) + (1+ λopt)y (i)

(2)

λopt computed in a second inner loop and convergent for 1 ≤ α < +∞


IRLS : Vector version (2)

For Spatio/temporal RPCA, needs to solve the following general problem :

argminx

||Ax − b||α + λ||Cx − d ||β (3)

By derivation, the associated IRLS scheme is,

r1 = b − Ax (i), r2 = d − Cx (i), e1 = ε+ |r1|, e2 = ε+ |r2|D1 = (

∑eα1 )

1α−1diag(eα−21 ),D2 = λ(

∑eβ2 )

1β−1diag(eβ−22 )

y (i) = (A′D1A + C ′D2C)−1(A′D1r1 + C ′D2r2)

x (i+1) = x (i) + (1 + λopt)y (i)

(4)

Good news : Just few line of matlab code !


Matrix Version

More generally, we consider the following matrix regression problem withtwo parameters norm (α, β) and a weighted matrix (W ),

minX||AX − B||α,β

Wwith ||Mij ||α,β

W= (

n∑i=1

(m∑

j=1

Wij |Mij |β)αβ )

1α (5)

The problem is solved in the same manner on matrices with a reweightedregression strategy,

Until X is stable, repeat on each k-columnsR ← B − AXS ← ε+ |R|Dk ← diag(Sβ−2ik ◦ (

∑j (S

βij ◦Wij ))

αβ−1 ◦Wik)k

Xik ← Xik +(1+Λ(max(α, β)))(AtDkA)−1AtDkRik

(6)


Exemple : Geometric median (convex problem)

Fig.1 Find the x-axis points which minimize the ||.||α distance between the 7 otherspoints (black stars). Minimum of Curves is corresponding to this optimal value for α =2, 1.66, 1.33 and 1 (blue, green, red, black).

Fig.1 Same in Two Dimension. By continuously changing the minization problem (i.e.varying α) during each iterations, this trick accelerates the convergence of IRLS.


Various RPCA formulation (only for α = 1)

PCA with fixed rank is : minL,S

||S ||Fs.t. Rank(L) = k

A = L + S(7)

R(obust)PCA is (Non convex and NP-hard) :

minL,S

||σ(L)||0 + λ||S ||0s.t. A = L + S

(8)

Convex relaxed problem of (2) is RPCA-PCP proposed by Candès et al. [1] :

minL,S

||σ(L)||1 + λ||S ||1s.t. A = L + S

(9)

Where σ(L) means singular values of L.A mix is Stable PCP of Zhou et al. [2] (both entry-wise and sparse noise) :

minL,S

||σ(L)||1 + λ||S ||1s.t. ||A− L− S ||F < δ

(10)

All of them could be solved by Augmented Lagrangian Multipliers (ALM).


Video examples

Some examples, temporal RPCA and ideal RPCA with groundtruth fittingand optimal parameters fitting.


Summary





5 Conclusion


Sparse solution

In RPCA, residual error is sparse.Using the RPCA decomposition on a synthetic low-rank random matrixplus noise, the error looks like :

Same principle with video. Sparse noise (or outliers) are the moving objects.


Let’s play with norms

Varying the α, β norm → Different kind of recovering pattern error.


Let’s play with norms...(2)

Some issuesWhat is the best specific norm for temporal constrain ?Initial assumption is ||.||2,1. Confirmed experimentally ?


Validation

If ideal eigenbakgrounds are that, best norm must be ...

Let’s denote Lopt , the ideal low-rank subspace which outliers do not contribute to PCA


Experimental results

Let’s denote Lα,β , the low-rankrecovered matrix with a ||.||α,β-PCA.The plot show the error between||Lopt − Lα,β||F for parameters αand β chosen freely. The darkestvalue means that the error is thesmallest here.

||S ||2,1 is not optimal, but for convenience we use it.The benefit of the ad hoc block-sparse hypothesis is confirmed bytesting its efficiency directly on video dataset.

Experimentation done on dynamic category of dataset change detectionworkshop 2012 : http://www.changedetection.net/


http://www.changedetection.net/

Summary





5 Conclusion


Overview and addition of the spatial constraint

Figure: Overview of the learning and evaluation process. Learning process needsGT (Groundtruth) for better fits the eigenbackground components.

Spatial ConstrainSuppose A = L+ S and L and S are computed via some kind ofRPCA technique with the addtion of Total Variation penalty on S .This increase connected (or connexe) shapes.


Exemple with a synthetic 1-D signal


Summary





5 Conclusion


Experimental Protocol

Optimal threshold is chosen for maximizing F-measure criterionwhich is based 2× 2 histogram of True/false/positive/negative :

DR =TP

TP + FN, Prec =

TP

TP + FP, F =

2 DR PrecDR + Prec

Good performance is then obtained when the F-measure is closed to 1Time consumption is not take into account in the evaluation process.

RPCA-LBD is compared with the following two Robust methods :Robust Subspace Learning (RSL) De La Torre et al. [4]Principal Component Pursuit (RPCA-PCP) [1]


Quantitative Results

Here, we show experimental results on the real dataset of BMC,

Video Recall Precision F-measure PSNR Visual Results1 0.9139 0.7170 0.8036 38.24252 0.8785 0.8656 0.8720 26.77213 0.9658 0.8120 0.8822 37.70534 0.9550 0.7187 0.8202 39.36995 0.9102 0.5589 0.6925 30.58766 0.9002 0.7727 0.8316 29.99947 0.9116 0.8401 0.8744 26.83508 0.8651 0.6710 0.7558 30.50409 0.9309 0.8239 0.8741 55.1163

Table: Quantitative results with common criterions. Last column show theoriginal, GT and result of the first four real video sequences.


Other results

Figure: First eigenBackground of the fifths sequence of Rotary (BMC) with thenorms ||.||opt , ||.||1,1 and ||.||2,1. Last row shows the first five eigenBackground onreal dataset with ||.||2,1


Summary





5 Conclusion


Qualitative analysis

AdvantagesExperiments on video surveillance datasets show that this approach ismore robust than RSL and RPCA-PCP in presence of dynamicbackgrounds and illumination changes.Well suited for video with spatially spread and temporarily sparseoutliers.

DisadvantagesNot efficient on sequences with outliers always presents.For small local variations, like « wind tree » are not (yet) wellmodelized by this kind global PCA. Probably needs more eigencomponents !


Future Works & References

Future WorksLack in computation time : Further research consists in developping anincremental version to update the model at every frame and to achievethe real-time requirements.

References[1] E. Candes, X. Li, Y. Ma, and J. Wright, Robust principal component

analysis, International Journal of ACM, vol. 58, no. 3, May 2011.[2] Z. Zhou, X. Li, J. Wright, E. Candes, and Y. Ma, Stable principal

component pursuit,IEEE ISIT Proceedings, pp. 1518-1522, Jun. 2010.[3] G. Tang and A. Nehorai, Robust principal component analysis based

on low-rank and block-sparse matrix decomposition, CISS 2011, 2011.


bmc 2012

Business

france workshop bmc

spatiotemporal constraintc

robust framework

daejong koreanovember

franceworkshop bmc

irls2 temporal constraint

video sequence

b ax id