"designing and implementing camera isp algorithms using deep learning and computer...
TRANSCRIPT
Copyright © 2017 Motorola Mobility LLC 1
Val Marchevsky
May 2017
Designing and Implementing Camera ISP
Algorithms Using Deep Learning and
Computer Vision
Copyright © 2017 Motorola Mobility LLC 2
Quality Image Preserves Your Memories
Examples
Copyright © 2017 Motorola Mobility LLC 3
Subtle Differences Matter
Copyright © 2017 Motorola Mobility LLC 4
People Want Top Notch Cameras
Camera is a top priority (Unaided)2015 PUF Study
BrazilUS
Copyright © 2017 Motorola Mobility LLC 5
When Your Camera Is Good, Phone is Good
• Customers get it. They
want a good camera on
their smartphones.
Copyright © 2017 Motorola Mobility LLC 6
What is Image Quality?
• Focus
• Sharpness
• Preservation of Texture
• White Balance
• Contrast
• Exposure
• Noise
• Artifacts
• Stabilization
Image Credit DXO Labs
Copyright © 2017 Motorola Mobility LLC 7
What makes Image Quality Challenging?
• Subjectivity
• Competing goals (sharpness /
texture)
• Lab performance vs. real-
world performance
• Corner cases (wrong focus,
green dogs)
Copyright © 2017 Motorola Mobility LLC 8
How does Lenovo/Motorola use machine
learning to make better cameras?
• No Reference Image Quality Analysis (NR-IQA)
• SVM-based HDR trigger
• Focus Failure Detection
• Estimated MOS
• DxOMark analytics
• Neural Network AWB
Copyright © 2017 Motorola Mobility LLC 9
• Optimizers
• AdaBoost
• Support Vector Machines
• Linear Regression
• Neural Networks, CNNs and Stochastic Gradient Descent
• Frameworks
• Caffe
• TorchFlow
• MXNet
• Moto-proprietary
What Machine Learning Technologies Does Lenovo/Motorola Use?
Copyright © 2017 Motorola Mobility LLC 10
• Motorola developed General
Image Quality Index (GIQI),
a regression model that
computes an estimate of
mean opinion score (MOS).
GIQI is a CNN application,
with its own network
definition, “MOSNet”
No Reference Image Quality Analysis (NR-IQA)
MOS Per Scene
Copyright © 2017 Motorola Mobility LLC 11
GIQI Examples
Copyright © 2017 Motorola Mobility LLC 12
GIQI and MOSNet
MOSNet is trained on a large body of
artificially distorted images
To speed convergence, it uses transfer-
learning and borrows its initial weight values
from a standard pre-trained Caffe model
x 2
Copyright © 2017 Motorola Mobility LLC 13
GIQI Performance on Public Datasets
• Comparison against the top performing algorithms on the public datasets
including LIVE in the Wild Challenge Database
• 100 random train-test (80/20) splits
Copyright © 2017 Motorola Mobility LLC 14
Using GIQI Estimated MOS for Product Evaluation
• Motorola uses GIQI to
compute probability
distributions of estimated
MOS for comparing
product performance
Copyright © 2017 Motorola Mobility LLC 15
GIQI Estimated MOS vs. Psychometric Evaluation
Psychometric
Evaluation
(man)
GIQI
(machine)
Man vs. Machine
Copyright © 2017 Motorola Mobility LLC 16
Machine Assessed Image Quality
• Effective Analysis Of Corner Cases
• Rapid Iteration of Development Process
• Competitive Analysis
• Relative Parity with Human Observers
Copyright © 2017 Motorola Mobility LLC 17
Auto Focus Failure Detection
Problem: We spend way too much time analyzing
perceived Auto Focus failures. Can we have a machine filter
gross or all failures out?
Vector: Software
Solution: Use machine-learning based classifier to improve auto-trigger
performance. Initial support-vector based solution improves recall by over
30% with no degradation in precision. Results improved with expanded
dataset.
Markets: all
End Users:
• Better quality focus solution where we can analyze true failures and
concentrate on real issues
Iterations Chart credit DXO Labs
Copyright © 2017 Motorola Mobility LLC 18
AF-FD Examples
Image
Quality
Details:
Focus Class:
Out of focus
Focus Score:
0.6843
Image
Quality
Details:
Focus Class:
Out of focus
Focus Score:
0.7237
Copyright © 2017 Motorola Mobility LLC 19
• Motorola developed an
autofocus failure detection
classifier
• Based on engineered features
including natural scene
statistics
• Best results were obtained
with AdaBoost optimization
GIQI and MOSNet
Image distortions deform the Gaussian
shape of natural scene statistics
Figure credit: Mittal, Anish, Anush Krishna Moorthy, and
Alan Conrad Bovik. "No-reference image quality
assessment in the spatial domain." IEEE Transactions on
Image Processing 21.12 (2012): 4695-4708.
Copyright © 2017 Motorola Mobility LLC 20
AF-FD: Sample Results after Training
Copyright © 2017 Motorola Mobility LLC 21
AF-FD: Real-world Results on User Trial Data
Copyright © 2017 Motorola Mobility LLC 22
• Huge reduction in escaped focus defects
• Higher complexity algorithms without higher risk penalty
• Error classification and data mining
• Best focus software stack in industry
Auto Focus Failure Detection
Copyright © 2017 Motorola Mobility LLC 23
HDR Trigger
Problem: Image quality suffers
when HDR trigger is too
conservative (poor recall)
Solution:
Use machine-learning based classifier to
improve auto-trigger performance.
Initial support-vector based solution improves
recall by over 30% with no degradation in
precision.
Results improved with expanded dataset.
Copyright © 2017 Motorola Mobility LLC 24
We want to use HDR when we can!
Copyright © 2017 Motorola Mobility LLC 25
• Sometimes best
technology to use
depends on scene
content!
HDR Trigger Comparison
% o
f corr
ect
HD
R T
riggers
Copyright © 2017 Motorola Mobility LLC 26
• Laboratory for Image and Video Engineering (http://live.ece.utexas.edu)
• Mittal, Anish, Anush Krishna Moorthy, and Alan Conrad Bovik. "No-
reference image quality assessment in the spatial domain." IEEE
Transactions on Image Processing 21, no. 12 (2012): 4695-4708.
Resources
Copyright © 2017 Motorola Mobility LLC 27
• Motorola Mobility LLC is a proud partner of the Laboratory of Image and Video Engineering
(LIVE) at the University of Texas @ Austin
• Professor Alan Bovik and his students have generously shared their ideas and talents with
Motorola. Their expertise was crucial in the development and to the success of GIQI and
AF-FD
• LIVE developed Natural Scene Statistics (NSS) and continues to pioneer research and
advances in the field of NR-IQA
• Motorola Mobility thanks DXO for being an advocate for consumer Image Quality and letting
us use their public data
Acknowledgements
Copyright © 2017 Motorola Mobility LLC 28
Note: Moto branded products are designed and
manufactured by or for Motorola Mobility LLC,
a wholly owned subsidiary of Lenovo.