Upload File

non-negative matrix factorization via alternating updates · 2020-01-03 · non-negative matrix...

  • Home
  • Documents
  • Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,
34
Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop, Berkeley, Nov 2016 Andrej Risteski Yuanzhi Li Joint work with

Upload: others

Post on 19-Jun-2020

3 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-negative Matrix Factorization via Alternating Updates

Yingyu Liang, Princeton University

Simons workshop, Berkeley, Nov 2016

Andrej RisteskiYuanzhi Li

Joint work with

Page 2: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-convex Problems in ML

Dictionary learningMatrix completion

Deep learning

Page 3: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-convex Problems in ML

Dictionary learningMatrix completion

Deep learning

NP-hard to solve in the worst case

Page 4: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-convex Problems in ML

• In practice, often solved by “local improvement”

• Gradient descent and variants

Page 5: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-convex Problems in ML

• In practice, often solved by “local improvement”

• Gradient descent and variants

• Alternating update

Page 6: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-convex Problems in ML

• In practice, often solved by “local improvement”

• Gradient descent and variants

• Alternating update

Goal: provable guarantees of simple algos under natural assumptions

When and why do such simple algos work for the hard problems?

Page 7: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-convex Problems in ML

• In practice, often solved by “local improvement”

• Gradient descent and variants

• Alternating update

Goal: provable guarantees of simple algos under natural assumptions

When and why do such simple algos work for the hard problems?

This work: alternating update for Non-negative Matrix Factorization

Page 8: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Non-negative Matrix Factorization (NMF)

• Given: matrix

• Find: non-negative matrices

Page 9: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

NMF in ML Applications

Basic tool in machine learning

• Topic modeling [Blei-Ng-Jordan03, Arora-Ge-Kannan-Moitra12, Arora-Ge-Moitra12,…]

0.05

0.04

0.08

0.05 0.05 0.05

0.2

0.3

0.1

0.2

0.8

the

physics

soccer

……

……

Doc1 Doc2 Doc3 ……

rain

Weather Sport Science ……

the

soccer

……

……

physics

rain

Doc1 Doc2 Doc3 ……

Page 10: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

NMF in ML Applications

Basic tool in machine learning

• Topic modeling [Blei-Ng-Jordan03, Arora-Ge-Kannan-Moitra12, Arora-Ge-Moitra12,…]

• Computer vision [Lee-Seung97, Lee-Seung99, Buchsbaum-Bloch02,…]

Page 11: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

NMF in ML Applications

Basic tool in machine learning

• Topic modeling [Blei-Ng-Jordan03, Arora-Ge-Kannan-Moitra12, Arora-Ge-Moitra12,…]

• Computer vision [Lee-Seung97, Lee-Seung99, Buchsbaum-Bloch02,…]

• Many others: network analysis, information retrieval, ……

Page 12: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Worst Case v.s. Practical Heuristic

Worst case analysis [Arora-Ge-Kannan-Moitra12]

• Upper bound:

• Lower bound: no algo, assuming ETH

Alternating updates: typical heuristic, suggested by Lee-Seung

Page 13: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Analyzing Non-convex Problems

• Generative model: the input data is generated from (a distribution defined by) a ground-truth solution

• Warm start: good initialization not far away from the ground-truth

Warm start

Update

Page 14: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Beyond Worst Case for NMF

• Separability-based assumptions [Arora-Ge-Kannan-Moitra12]

• Motivated by topic modeling: each column of (topic) has an anchor word

• Lots of subsequent work [Arora-Ge-Moitra12, Arora-Ge-Halpern-Mimno-Moitra-Sontag-

Wu-Zhu12, Gillis-Vavasis14, Ge-Zhou15, Bhattacharyya-Goyal-Kannan-Pani16, …]

0.05 0.05 0.05

0.2 0 0

0.3

0.1

Weather Sport Science

the

soccer

……

physics

rain

Page 15: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Beyond Worst Case for NMF

• Separability-based assumptions [Arora-Ge-Kannan-Moitra12]

• Motivated by topic modeling: each column of (topic) has an anchor word

• Lots of subsequent work [Arora-Ge-Moitra12, Arora-Ge-Halpern-Mimno-Moitra-Sontag-

Wu-Zhu12, Ge-Zhou15, Bhattacharyya-Goyal-Kannan-Pani16,…]

• Variational inference [Awasthi-Risteski15]

• Alternating update method on objective

• Requires relatively strong assumptions on , and/or a warm start depending on its dynamic range (not realistic)

Page 16: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Outline

• Introduction

• Our model, algorithm and main results

• Analysis of the algorithm

Page 17: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Generative Model

Each column of is i.i.d. example from

Page 18: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Generative Model

Each column of is i.i.d. example from

• (A1): columns of are linearly independent

Page 19: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Generative Model

Each column of is i.i.d. example from

• (A1): columns of are linearly independent

• (A2): are independent random variables

where is a parameter

Page 20: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Our Algorithm

Parameters:

0

Known as Rectified Linear Units (ReLU) in Deep Learning

1

Page 21: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Our Algorithm

Parameters:

are two independent examples, and are their decodings

Page 22: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Our Algorithm

Parameters:

Page 23: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Our Algorithm

Parameters:

Page 24: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Warm Start

• (A3): warm start with error

Page 25: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Warm Start

• (A3): warm start with error

Aligned with truth Not too much

Page 26: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Main Result

Page 27: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Analysis: Overview

Show the algo will

1. Maintain and

2. Decrease the potential function

Page 28: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Analysis: Effect of ReLU

•

Page 29: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Analysis: Effect of ReLU

•

Page 30: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Analysis: Effect of ReLU

•

Much less noise after

Page 31: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Analysis: Change of Error Matrix

•

• Therefore,

Page 32: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

More General Results

1. Each column of is generated i.i.d. by

• The decoding is

• So can* tolerate large adversarial noise,

• and tolerate zero-mean noise much larger than signal

2. Distribution of only needs to satisfy some moment conditions

Page 33: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Conclusion

• Beyond worse case analysis of NMF

• generative model with mild condition on the feature matrix

• Provable guarantee of alternating update algorithm

• Strong denoising effect by ReLU + non-negativity

Page 34: Non-negative Matrix Factorization via Alternating Updates · 2020-01-03 · Non-negative Matrix Factorization via Alternating Updates Yingyu Liang, Princeton University Simons workshop,

Conclusion

• Beyond worse case analysis of NMF

• generative model with mild condition on the feature matrix

• Provable guarantee of alternating update algorithm

• Strong denoising effect by ReLU + non-negativity

Thanks! Q&A


DISCRIMINATIVE NON-NEGATIVE MATRIX FACTORIZATION FOR ...boulanni/ISMIR2012.pdf · the multiple pitch estimation accuracy of the non-negative matrix factorization (NMF) algorithm

Ranking by Alternating SVM and Factorization Machine · Ranking by Alternating SVM and Factorization Machine Shanshan Wu, ... Next, we try Quadprog, which is a Matlab toolbox specifically

Single-channel source separation using non-negative matrix ...mikkelschmidt.dk/presentations/thesis_presentation.pdf · Non-negative matrix factorization Non-negative bilinear model

Augmented Lagrangian alternating direction …zhang/reports/tr1102.pdfAugmented Lagrangian alternating direction method for matrix separation based on low-rank factorization Yuan Shen

A Particle-Based Variational Approach to Bayesian Non-negative Matrix Factorization · 2020-04-09 · Keywords: Bayesian, Non-negative Matrix Factorization, Stein discrepancy, Non-identi

Non-negative matrix factorization with Gaussian process priorsmikkelschmidt.dk/presentations/nmfgppcam.pdf · 2015. 4. 15. · Non-negative Matrix Factorization D. D. Lee and H. S

Non-negative matrix factorization for analysing high ... · Experts in numerical software and High Performance Computing Non-negative matrix factorization for analysing high-dimensional

Beginners Guide to Non-Negative Matrix Factorization

Non-negative Matrix Factorization for Face Recognition

NON-NEGATIVE MATRIX FACTORIZATION FOR REAL …cseweb.ucsd.edu/~dhu/docs/icassp08_poster.pdf · non-negative matrix factorization for real time musical analysis and sight-reading evaluation

Projected Gradient Methods for Non-negative Matrix Factorization

Truncated Cauchy Non-Negative Matrix Factorization · Index Terms—Non-negative matrix factorization, truncated cauchy loss, robust statistics, half-quadratic programming Ç 1INTRODUCTION

A novel non-negative matrix factorization technique for

Age estimation based on extended non negative factorization

Graph Regularized Non-negative Matrix Factorization for ... · 1 Graph Regularized Non-negative Matrix Factorization for Data Representation Deng Cai, Member, IEEE, Xiaofei He, Senior

Non-negative matrix factorization for classifying the

1861 Algorithms for Non Negative Matrix Factorization

NON-NEGATIVE MATRIX FACTORIZATION FOR DRUM …zduan/teaching/ece477/projects/2015/... · NON-NEGATIVE MATRIX FACTORIZATION FOR DRUM SOURCE SEPARATION AND TRANSCRIPTION Jon Downing

NON-NEGATIVE MATRIX FACTORIZATION FOR DRUM …zduan/teaching/ece477/projects/2015/Jon... · NON-NEGATIVE MATRIX FACTORIZATION FOR DRUM SOURCE SEPARATION AND TRANSCRIPTION Jon Downing

A Novel Non-Negative Matrix Factorization Method for ......A Novel Non-Negative Matrix Factorization Method for Recommender Systems Mehdi Hosseinzadeh Aghdam, Morteza Analoui∗and

Non-negative matrix factorization (NMF) - Amazon S3 · 2017-01-31 · Unsupervised Learning in Python Non-negative matrix factorization NMF = "non-negative matrix factorization" Dimension

Structured Non-Negative Matrix Factorization

Non-negative matrix factorization for semi-supervised …mdong/kais08.pdf · Non-negative matrix factorization for semi-supervised data clustering modified so that the available

Non-negative Matrix Factorization with Sparseness Constraints

Extensions of Non-Negative Matrix Factorization (NMF) to Higher Order Data HONMF (Higher Order Non-negative Matrix Factorization) NTF2D/SNTF2D ((Sparse)

Extensions of Non-negative Matrix Factorization to Higher Order data

Matrix Factorization For Topic Models - Derek Greene · Matrix Factorization For Topic Models Dr. Derek Greene Insight Latent Space Workshop. Non-negative Matrix Factorization •

Non Negative Matrix Factorization Hamdi Jenzri. Outline Introduction Non-Negative Matrix Factorization (NMF) Cost functions Algorithms Multiplicative

Blind Source Separation using Non-negative Matrix Factorization: FASSTjordipons.me/media/NMF_presentation.pdf · 2015-11-25 · Blind Source Separation using Non-negative Matrix Factorization:

Position-dependent motif characterization using Non-negative matrix Factorization (NMF)

Non-negative Multiple Matrix Factorization

Using separable non-negative matrix factorization techniques for the

Non-negative matrix factorization of multimodal MRI, fMRI ...jxie/personalpage_file/publications/nmf_fMRI.pdf · Non-negative matrix factorization of multimodal MRI, fMRI and phenotypic

Non-Negative Matrix Factorization And Its Application to Audiobhiksha/courses/mlsp.fall2009/class16/nmf.pdfNon-Negative Matrix Factorization And Its Application to Audio ... – With

Algorithms for Non-negative Matrix Factorization