on the spectral bias of neural networks13-11...2000/11/13 · on the spectral bias of neural...

On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A. Hamprecht Yoshua Bengio Aaron Courville

Upload: others

Post on 17-Jun-2020

5 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

On the Spectral Bias of

Neural NetworksNasim Rahaman* Aristide Baratin* Devansh Arpit

Felix Draxler Min Lin

Fred A. Hamprecht Yoshua Bengio Aaron Courville

Page 2: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

The good old question:

Why do massive neural

networks generalize when they

can learn random labels?

Page 3: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

The good old question:

Why do massive neural

networks generalize when they

can learn random labels?

Page 4: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Our proposal:

Neural networks learn simpler

functions first.

Page 5: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

But how do we quantify simplicity?

We use the (Fourier) Spectrum.Our approach:

Lower Frequency Functions

Higher Frequency Functions

Page 6: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Our proposal becomes:

Neural networks learn lower

frequencies first.

Page 7: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Fully learned

Not learned

Colorbar shows the Fourier amplitude

of the network relative to the

target.

Trai

ning

Iter

atio

Page 8: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Why should I care?

One of the many reasons:

NN training is vulnerable against low frequency label noise.

Page 9: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Training with label noise

High frequency label noise leads to

a dip in the validation loss.

Validation loss on MNIST (w.r.t pure targets)

Low frequency label noise does

not… :(

Page 10: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

To learn how the manifold complexity

attenuates the spectral bias,

drop by at our poster!

#72

Page 11: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Spoilers

Learning gets easier with

increasing manifold

complexity.

To express complex functions, the parameters must “work

together in harmony”.

Page 12: On the Spectral Bias of Neural Networks13-11...2000/11/13 · On the Spectral Bias of Neural Networks Nasim Rahaman* Aristide Baratin* Devansh Arpit Felix Draxler Min Lin Fred A

Thank you for your

attention!

Spectral Graph Theory and Graph CNN - qdata.github.io€¦ · Reference 1 Laplacian Operator - Wikipedia 2 An introduction to spectral graph theory Jiang Jiaqi 3 Convolutional Neural

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Peter Tino and Barbara Hammer- Architectural Bias in Recurrent Neural Networks - Fractal Analysis

Graph Convolutional Neural Networks• Introduction to Graph Convolutional Neural Networks (Graph CNN) • Background in Graph Signal Processing for analysis on graphs • Spectral

Asymmetries in behavioral and neural responses to spectral ... · Asymmetries in behavioral and neural responses to spectral cues demonstrate the generality of auditory looming bias

Towards Understanding the Spectral Bias of Deep Learning · Towards Understanding the Spectral Bias of Deep Learning Yuan Caoy, Zhiying Fangz, Yue Wux, Ding-Xuan Zhou{, Quanquan Guk

CONVOLUTIONAL NEURAL NETWORKS FOR MESH-BASED … · "Convolutional neural networks on graphs with fast localized spectral filtering." In Advances in Neural Information Processing

Graph Convolutional Neural Networks · M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in NIPS

Review: Spectral Representations for Convolutional Neural ... · Spectral pooling truncates the spectral representation of a image-kernel product. Simply put, spectral pooling is

The Pitfalls of Simplicity Bias in Neural Networks

SPECTRAL MAPPING USING ARTIFICIAL NEURAL … MAPPING USING ARTIFICIAL NEURAL NETWORKS FOR ... Hyderabad, India CERTIFICATE ... Centre, IIIT-H for providing

Walczak_Artificial Neural Networks in Classification of NIR Spectral Data

On the Language Coverage Bias for Neural Machine Translation

Bias-Free Neural Predictor

Spectral Clustering with Graph Neural Networks for Graph …Spectral Clustering with Graph Neural Networks for Graph Pooling eigenvalues, and O 2R K is an orthogonal transforma- tion

Spectral Representations for Convolutional Neural …papers.nips.cc/paper/5649-spectral-representations-for...Spectral Representations for Convolutional Neural Networks Oren Rippel

Spectral Representations for Convolutional Neural Networks

Automated Spectral Classification Using Neural Networks · CEFET-PR, 13 a 15 de Setembro de 1995 Curitiba Paraná Automated Spectral Classification Using Neural Networks E. F. Vieira1

Qualifying Oral Exam: Representation Learning on Graphs · Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering [De errard et al., 2016] Convolutional neural

A neural mass model of spectral responses in electrophysiology · 2008-09-15 · A neural mass model of spectral responses in electrophysiology ... transfer function. ... We have

Spectral Analysis for Neural Signals - Department of … Analysis for Neural Signals Bijan Pesaran, PhD Center for Neural Science, New York University ... in neural signal analysis

Spectral Analysis for Neural Signals · spectral analysis and applies spectral analysis to char-acterize neural signals. Spectral analysis is a form of time series analysis and concerns

Spectral Pruning: Compressing Deep Neural Networks via Spectral … · 2020. 7. 20. · that the bias–variance tradeoff induced by model com-pression appears. The obtained bound

Generalizing Convolutional Neural Networks to Graph ...dzeng/BIOS740/Walker_Bios740.pdf · "Convolutional neural networks on graphs with fast localized spectral ﬁltering." Advances

BENN: Bias Estimation Using Deep Neural Network · 2020. 12. 24. · BENN: Bias Estimation Using Deep Neural Network Amit Giloni,*1 Edita Grolman,1 Tanja Hagemann,2,3 Ronald Fromm,3

Pinpointing the peripheral bias in neural scene-processing

Peter Tino, Barbara Hammer and Mikael Boden- Markovian bias of neural-based architectures with feedback connections

Exposure Bias and State Matching in Recurrent Neural

The neural correlates of rule implementation and ... · 1 The neural correlates of rule implementation and attentional bias in the task-cuing paradigm Yiquan Shi

SAFE: Spectral Evolution Analysis Feature Extraction for ... · Index Terms—non-stationary, time series, deep neural network, spectral analysis. I. INTRODUCTION Time series analysis

Deep Spectral Descriptors: Learning the point-wise ... · Deep Spectral Descriptors: Learning the point-wise correspondence metric via Siamese deep neural networks Zhiyu Sun 1; 2Yusen

A neural mechanism of response bias Johan Lauwereyns Laboratory of Sensorimotor Research National Eye Institute, NIH

Cognitive Psychology for Deep Neural Networks: A Shape ...proceedings.mlr.press/v70/ritter17a/ritter17a.pdfCognitive Psychology for Deep Neural Networks: A Shape Bias Case Study gai”,

Monaural Spectral Contrast Mechanism for Neural ... · Monaural Spectral Contrast Mechanism for Neural Sensitivity to Sound Direction in the Medial Geniculate Body of the Cat THOMAS

Convolutional Neural Networks on Graphs with Fast ...web.eng.tau.ac.il/deep_learn/wp-content/uploads/... · Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering