ℓ1-norm methods for convex-cardinality problems part ii ?· ℓ1-norm methods for...

Download ℓ1-norm Methods for Convex-Cardinality Problems Part II ?· ℓ1-norm Methods for Convex-Cardinality…

Post on 03-Jul-2018

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • 1-norm Methods for

    Convex-Cardinality Problems

    Part II

    total variation

    iterated weighted 1 heuristic

    matrix rank constraints

    Prof. S. Boyd, EE364b, Stanford University

  • Total variation reconstruction

    fit xcor with piecewise constant x, no more than k jumps

    convex-cardinality problem: minimize x xcor2 subject tocard(Dx) k (D is first order difference matrix)

    heuristic: minimize x xcor2 + Dx1; vary to adjust number ofjumps

    Dx1 is total variation of signal x

    method is called total variation reconstruction

    unlike 2 based reconstruction, TVR filters high frequency noise outwhile preserving sharp jumps

    Prof. S. Boyd, EE364b, Stanford University 1

  • Example (6.3.3 in BV book)

    signal x R2000 and corrupted signal xcor R2000

    0 500 1000 1500 20002

    1

    0

    1

    2

    0 500 1000 1500 20002

    1

    0

    1

    2

    i

    xx

    cor

    Prof. S. Boyd, EE364b, Stanford University 2

  • Total variation reconstruction

    for three values of

    0 500 1000 1500 20002

    0

    2

    0 500 1000 1500 20002

    0

    2

    0 500 1000 1500 20002

    0

    2

    i

    xx

    x

    Prof. S. Boyd, EE364b, Stanford University 3

  • 2 reconstruction

    for three values of

    0 500 1000 1500 20002

    0

    2

    0 500 1000 1500 20002

    0

    2

    0 500 1000 1500 20002

    0

    2

    i

    xx

    x

    Prof. S. Boyd, EE364b, Stanford University 4

  • Example: 2D total variation reconstruction

    x Rn are values of pixels on N N grid (N = 31, so n = 961)

    assumption: x has relatively few big changes in value (i.e., boundaries)

    we have m = 120 linear measurements, y = Fx (Fij N (0, 1))

    as convex-cardinality problem:

    minimize card(xi,j xi+1,j) + card(xi,j xi,j+1)subject to y = Fx

    1 heuristic (objective is a 2D version of total variation)

    minimize

    |xi,j xi+1,j| +

    |xi,j xi,j+1|subject to y = Fx

    Prof. S. Boyd, EE364b, Stanford University 5

  • TV reconstruction

    510

    1520

    2530

    0

    5

    10

    15

    20

    25

    30

    35

    0.5

    0

    0.5

    1

    1.5

    original

    510

    1520

    2530

    0

    5

    10

    15

    20

    25

    30

    35

    0.5

    0

    0.5

    1

    1.5

    TV reconstruction

    . . . not bad for 8 more variables than measurements!

    Prof. S. Boyd, EE364b, Stanford University 6

  • 2 reconstruction

    510

    1520

    2530

    0

    5

    10

    15

    20

    25

    30

    35

    0.5

    0

    0.5

    1

    1.5

    original

    510

    1520

    2530

    0

    5

    10

    15

    20

    25

    30

    35

    0.5

    0

    0.5

    1

    1.5

    2 reconstruction

    . . . this is what youd expect with 8 more variables than measurements

    Prof. S. Boyd, EE364b, Stanford University 7

  • Iterated weighted 1 heuristic

    to minimize card(x) over x C

    w := 1repeat

    minimize diag(w)x1 over x Cwi := 1/( + |xi|)

    first iteration is basic 1 heuristic

    increases relative weight on small xi

    typically converges in 5 or fewer steps

    often gives a modest improvement (i.e., reduction in card(x)) overbasic 1 heuristic

    Prof. S. Boyd, EE364b, Stanford University 8

  • Interpretation

    wlog we can take x 0 (by writing x = x+ x, x+, x 0, andreplacing card(x) with card(x+) + card(x))

    well use approximation card(z) log(1 + z/), where > 0, z R+

    using this approximation, we get (nonconvex) problem

    minimizen

    i=1 log(1 + xi/)subject to x C, x 0

    well find a local solution by linearizing objective at current point,

    n

    i=1

    log(1 + xi/) n

    i=1

    log(1 + x(k)i /) +

    n

    i=1

    xi x(k)i

    + x(k)i

    Prof. S. Boyd, EE364b, Stanford University 9

  • and solving resulting convex problem

    minimizen

    i=1 wixisubject to x C, x 0

    with wi = 1/( + xi), to get next iterate

    repeat until convergence to get a local solution

    Prof. S. Boyd, EE364b, Stanford University 10

  • Sparse solution of linear inequalities

    minimize card(x) over polyhedron {x | Ax b}, A R10050

    1 heuristic finds x R50 with card(x) = 44

    iterated weighted 1 heuristic finds x with card(x) = 36(global solution, via branch & bound, is card(x) = 32)

    1 2 3 4 5 60

    10

    20

    30

    40

    50

    iteration

    card(x

    )

    iterated 11

    Prof. S. Boyd, EE364b, Stanford University 11

  • Detecting changes in time series model

    AR(2) scalar time-series model

    y(t + 2) = a(t)y(t + 1) + b(t)y(t) + v(t), v(t) IID N (0, 0.52)

    assumption: a(t) and b(t) are piecewise constant, change infrequently

    given y(t), t = 1, . . . , T , estimate a(t), b(t), t = 1, . . . , T 2

    heuristic: minimize over variables a(t), b(t), t = 1, . . . , T 1

    T2t=1 (y(t + 2) a(t)y(t + 1) b(t)y(t))

    2

    +T2

    t=1 (|a(t + 1) a(t)| + |b(t + 1) b(t)|)

    vary to trade off fit versus number of changes in a, b

    Prof. S. Boyd, EE364b, Stanford University 12

  • Time series and true coefficients

    0 50 100 150 200 250 3003

    2

    1

    0

    1

    2

    3

    t

    y(t

    )

    50 100 150 200 250 300

    1

    0.8

    0.6

    0.4

    0.2

    0

    0.2

    0.4

    0.6

    0.8

    1

    t

    a(t)b(t)

    Prof. S. Boyd, EE364b, Stanford University 13

  • TV heuristic and iterated TV heuristic

    left: TV with = 10; right: iterated TV, 5 iterations, = 0.005

    50 100 150 200 250 300

    1

    0.8

    0.6

    0.4

    0.2

    0

    0.2

    0.4

    0.6

    0.8

    1

    t50 100 150 200 250 300

    1

    0.8

    0.6

    0.4

    0.2

    0

    0.2

    0.4

    0.6

    0.8

    1

    t

    Prof. S. Boyd, EE364b, Stanford University 14

  • Extension to matrices

    Rank is natural analog of card for matrices

    convex-rank problem: convex, except for Rank in objective orconstraints

    rank problem reduces to card problem when matrices are diagonal:Rank(diag(x)) = card(x)

    analog of 1 heuristic: use nuclear norm, X =

    i i(X)(sum of singular values; dual of spectral norm)

    for X 0, reduces to TrX (for x 0, x1 reduces to 1Tx)

    Prof. S. Boyd, EE364b, Stanford University 15

  • Factor modeling

    given matrix Sn+, find approximation of form = FFT + D, where

    F Rnr, D is diagonal nonnegative

    gives underlying factor model (with r factors)

    x = Fz + v, v N (0, D), z N (0, I)

    model with fewest factors:

    minimize RankXsubject to X 0, D 0 diagonal

    X + D C

    with variables D, X Sn

    C is convex set of acceptable approximations to

    Prof. S. Boyd, EE364b, Stanford University 16

  • e.g., via KL divergence

    C = { | log det(1/21/2 + Tr(1/21/2) n }

    trace heuristic:

    minimize TrXsubject to X 0, D 0 diagonal

    X + D C

    with variables d Rn, X Sn

    Prof. S. Boyd, EE364b, Stanford University 17

  • Example

    x = Fz + v, z N (0, I), v N (0, D), D diagonal; F R203

    is empirical covariance matrix from N = 3000 samples

    set of acceptable approximations

    C = { | 1/2( )1/2 }

    trace heuristic

    minimize TrXsubject to X 0, d 0

    1/2(X + diag(d) )1/2

    Prof. S. Boyd, EE364b, Stanford University 18

  • Trace approximation results

    102

    101

    100

    2

    4

    6

    8

    10

    12

    14

    16

    Rank(X

    )

    102

    101

    100

    104

    102

    100

    102

    i(

    X)

    Prof. S. Boyd, EE364b, Stanford University 19

  • for = 0.1357 (knee of the tradeoff curve) we find

    6(

    range(X), range(FFT ))

    = 6.8

    d diag(D)/diag(D) = 0.07

    i.e., we have recovered the factor model from the empirical covariance

    Prof. S. Boyd, EE364b, Stanford University 20

Recommended

View more >