ee645: independent component analysis elliot taniguchi advisor: dr. kuh may 16, 2003
TRANSCRIPT
![Page 1: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/1.jpg)
EE645: Independent Component Analysis
Elliot Taniguchi
Advisor: Dr. Kuh
May 16, 2003
![Page 2: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/2.jpg)
Presentation Overview
ICA Motivation Mathematical Formulation Fast ICA Algorithm Applications
Noise Separation and Feature Extraction Digital Watermarking
![Page 3: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/3.jpg)
Motivation for ICA
Cocktail Party Problem Suppose you are in a crowded room with
many people. How do you understand what any one person is saying?
Separation of Independent Signals Similar to Blind Source Separation Little knowledge of the signals Access to mixed signals only
![Page 4: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/4.jpg)
Cocktail Party Problem??
![Page 5: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/5.jpg)
Cocktail Party Problem
ICA Separation Algorithm Separation of Speech
Signals Humans can separate
multiple signals with only two ears/sensors
ICA needs as many ears/sensors as message signals
Here we assume he has four ears!
![Page 6: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/6.jpg)
Recovered Messages
![Page 7: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/7.jpg)
Mathematical Formulation Overview
ICA Definition ICA Assumptions
Independent Signals Non-Gaussian
ICA Limitations Scaling Permutations No. of Sensors
![Page 8: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/8.jpg)
ICA Definition
Mixed Signals in Matrix Notation
n
i
is1
iasAx
Variable Definitions
Signalt Independen i ~
Matrix ngMultiplexi ~
Signal dMultiplexe~
this
A
xj
![Page 9: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/9.jpg)
ICA Solution
Signal Separation
Find using the ICA Algorithm
ssIsA)(W
s)(AWxWs
ˆ
W
![Page 10: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/10.jpg)
ICA Block Diagram (2 Signals)
Signal #1 Signal #2
MultiplexedSignal #2
MultiplexedSignal #1
a11
a12 a21
a22
![Page 11: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/11.jpg)
ICA Assumption #1: Independence Probability Density Definition
Expected Value Definition
functions are ,
)}({)}({)}()({
21
22112211
hh
yhEyhEyhyhE
spdf' are ,
)()()(
21
22112,1
pp
ypypyyp
![Page 12: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/12.jpg)
ICA Assumption #2: Non-Gaussian Property of Gaussian signals
Addition of two independent Gaussian random variables is another single Gaussian random variable.
Information Lost! Kurtosis Function
Special Case: kurt(N) = 0
224 }){(3}{)( yEyEykurt
![Page 13: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/13.jpg)
Limitation #1: Scaling
ICA maximizes independence between signals.
)}ˆ({*)}ˆ({
)}ˆ(*)ˆ({)}ˆ(*)ˆ({
222111
221121222111
XhEXhE
XhXhEXhXhE
Assuming theSeparated Signalsare Independent
Scaling Factorfor Signal #1
Scaling Factorfor Signal #2
![Page 14: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/14.jpg)
Limitation #2: Signal Permutations
The mixing matrix and independent components are unknown.
sAsPPA
sPPAsIAsAx1
1
*)*(*)*(
*)*(***
sPssIsPP
sPIPsPAWP
sPAWPsAWx
1
11
1
***)*(
*))*(*(*)*)*(*(
*)*(*)*(**ˆ
![Page 15: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/15.jpg)
Limitation #3: No. of Sensors
Sensor Requirement The number of separated signals cannot be
larger than the number of inputs. Current research is being done to reduce this
constraint.
![Page 16: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/16.jpg)
ICA Separation Technique
Central Limit Theorem If two random (non-Gaussian) signals added,
the resulting signal will be more Gaussian than the original two random signals
ICA Separation Concept Central Limit Theorem (in Reverse) Maximizing Non-Gaussianity
Results in separating the two signals
![Page 17: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/17.jpg)
Fast ICA Algorithm Overview
“Fixed-Point” Algorithm Implementation
Fast ICA algorithm Extensions
Algorithm Speed & Performance Currently the fastest Most Commonly Used
![Page 18: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/18.jpg)
Fast ICA Algorithm
1. Choose a random initial weight vector.
2. Let,
3. Let,
4. Repeat until converges.
www /wxwxwxw TT )}(g{)}g({ EE
w
![Page 19: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/19.jpg)
Fast ICA Extensions
Preprocessing Normalize mean to zero Pre-Whitening
Activation Functions g(u)=u^3 g(u)=u^2 g(u)=tanh(a1*u) g(u)=u*exp(-a2*u^2/2)
![Page 20: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/20.jpg)
Noise Separation Example
Separation of Noise Impulsive Noise Additive White Gaussian Noise
Implementation Two Sensor Setup Fast ICA Algorithm
![Page 21: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/21.jpg)
Noise Generation
AWGN Gaussian R.V.
Impulsive Noise Poisson R.V. Gaussian R.V.
)(*),( PmNI
),( mN
![Page 22: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/22.jpg)
Physical Setup
Receiver #1Receiver #2
ImpulsiveNoise
GaussianNoise
![Page 23: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/23.jpg)
Noise Separation Example
0 100 200 300 400 500 600 700 800 900 1000-30
-20
-10
0
10
20
30Noise Separation Simulation
P(
= 1
)*N
(m =
0,
= ,
5)
0 100 200 300 400 500 600 700 800 900 1000-3
-2
-1
0
1
2
3
N(m
= 0
, =
1)
![Page 24: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/24.jpg)
Noise Separation Example
0 100 200 300 400 500 600 700 800 900 1000-30
-20
-10
0
10
20
30Noise Separation Simulation
Mul
tiple
xed
Sig
nal #
1
0 100 200 300 400 500 600 700 800 900 1000-30
-20
-10
0
10
20
30
Mul
tiple
xed
Sig
nal #
2
![Page 25: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/25.jpg)
Noise Separation Example
0 100 200 300 400 500 600 700 800 900 1000-20
-10
0
10
20Noise Separation Simulation
Sep
arat
ed S
igna
l #1
0 100 200 300 400 500 600 700 800 900 1000-3
-2
-1
0
1
2
3
Sep
arat
ed S
igna
l #2
![Page 26: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/26.jpg)
Noise Separation Performance
0 100 200 300 400 500 600 700 800 900 1000-40
-20
0
20
40
Orig
inal
Sig
nal #
1
Noise Separation Simulation
0 100 200 300 400 500 600 700 800 900 1000-40
-20
0
20
40
Nor
mal
ized
IC
A S
igna
l #1
0 100 200 300 400 500 600 700 800 900 1000-0.1
-0.05
0
0.05
0.1
Err
or :
Sig
nal #
1 MSE = 0.00051558
![Page 27: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/27.jpg)
Noise Separation Performance
0 100 200 300 400 500 600 700 800 900 1000-4
-2
0
2
4
Orig
inal
Sig
nal #
2
Noise Separation Simulation
0 100 200 300 400 500 600 700 800 900 1000-4
-2
0
2
4
Nor
mal
ized
IC
A S
igna
l #2
0 100 200 300 400 500 600 700 800 900 1000-0.2
-0.1
0
0.1
0.2
Err
or :
Sig
nal #
2 MSE = 7.3829e-005
![Page 28: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/28.jpg)
Noise Separation & Feature Extraction
ICA performs well in Blind Source Separation ICA for Feature Extraction
Reduce Complexity of the Neural Network Train only on the appropriate signal
Detection and Estimation of Hidden Signals
![Page 29: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/29.jpg)
Noise Detection and Estimation: Method #1
FastICASensor #3
Sensor #4
Sensor #2
Sensor #1
Feature #3
Feature #4
Feature #2
Feature #1
N.N.(BP,RBF,SVM)
![Page 30: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/30.jpg)
Noise Detection and Estimation: Method #2
FastICASensor #3
Sensor #4
Sensor #2
Sensor #1
Feature #3
Feature #4
Feature #2
Feature #1
N.N.(BP,RBF,SVM)
DecisionDevice
![Page 31: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/31.jpg)
Noise Detection and Estimation:Conclusion Additional Preprocessing
Segmentation of Impulsive Noise (Time-Limited) Possible Inputs to the Neural Network
Statistical Moments Signal Samples
Possible Neural Networks Back Propagation SVM Radial Basis Functions
Problem Need to train Neural Network in Matlab
![Page 32: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/32.jpg)
Digital Watermarking of Music
Motivation Popularity of Digital Storage Devices
Reliable, Fast, Ease of duplication, etc. How to protect copyrighted information?
Leaving digital signatures of its artist
Essential Properties for Watermarking Undetectable Irremovable Resilient
![Page 33: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/33.jpg)
Detection & Estimation of Watermarks Detection of Watermark
Authenticate copyrighted music Estimation of Watermark
Authenticate copyrighted music Information on artist, producer, etc.
![Page 34: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/34.jpg)
Watermarking Model
Process Mix the original musical data with watermark Keep watermark Power relatively low
Ensure high quality of the watermarked music Watermark is better hidden
hidden is watermark that theensure to :Note
WatermarkEmbedded
Data MusicalCopyright
Key Encryption
Data Musical dWatermarke
12 11
2221
1211
aa
aa
aa
![Page 35: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/35.jpg)
Popular Digital Formats
Wave Format Mp3 Format
Bit Rate 1411 [kbps] 128 [kbps]
Channels 2 2
Audio Sample Rate
44 [kHz] 44 [kHz]
![Page 36: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/36.jpg)
Detection of Watermark
Watermarking Detection Algorithm1. Use the ICA model to randomly mix the watermark
and music file.
2. Save the watermarked music in the popular *.wav format
3. Read the saved *.wave file. Separate the watermark and the music file.
4. Identify the watermark using statistical methods (mean, std, etc.)
Performance Statistic Correlation Coefficient (Absolute Value)
![Page 37: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/37.jpg)
Detection Performance
3.85 3.852 3.854 3.856 3.858 3.86 3.862 3.864 3.866 3.868 3.87
x 10-6
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Signal-to-Noise Ratio
Cor
rela
tion
Coe
ffic
ient
Performance of ICA Watermarking: 16 Bit, 44100 Hz
3.853 3.854 3.855 3.856 3.857 3.858 3.859 3.86 3.861
x 10-6
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Signal-to-Noise RatioC
orre
latio
n C
oeff
icie
nt
Performance of ICA Watermarking: 8 Bit, 44100 Hz
![Page 38: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/38.jpg)
Estimation of Watermark
Watermarking Estimation Algorithm1. Use the ICA model to randomly mix the watermark
and music file.2. Save the watermarked music in the popular *.wav
format3. Read the saved *.wave file. Separate the watermark
and the music file.4. Identify the watermark using statistical methods
(mean, std, etc.)5. Digitize the watermark signal.
Performance Statistic Bit Error Rate
![Page 39: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/39.jpg)
Estimation Performance
3.85 3.855 3.86 3.865
x 10-6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1Performance of ICA Watermarking: 16 Bit, 44100 Hz
Signal-to-Noise Ratio
Bit
Err
or R
ate
3.853 3.854 3.855 3.856 3.857 3.858 3.859 3.86 3.861 3.862 3.863
x 10-6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Signal-to-Noise RatioB
it E
rror
Rat
e
Performance of ICA Watermarking: 8 Bit, 44100 Hz
![Page 40: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/40.jpg)
Is the Music Content Preserved?
“Hero” – Mariah CareyOriginal Wave file from CD
Hero.wav29 [s]
“Hero” – Mariah CareyWatermarked Wave fileHeroWatermarked.wav
29 [s]
![Page 41: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/41.jpg)
Resilience of Proposed Watermark
Resiliency Previous Simulations show that wav format is
resilient to 8-bit and 16-bit quantization. Can the watermark be detected after Mp3
compression and decompression?
![Page 42: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/42.jpg)
Mp3 Compression/Decompression
Actual Mp3 compression program used CDex Version 1.40 Release
Mp3 (lossy) Compression Down Sampling Filter banks
Mp3 Decompression Up Sampling Reconstruction Filter
![Page 43: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/43.jpg)
Detection and Estimation Performance
Bad News Correlation Coefficients 0 MSE 0.5
Possible Problems with Mp3 Compression Down Sampling
Watermark information is lost Quantization Noise
Watermark information absorbs into the quantization noise
Lossy Compression 11:1 Compression Rate
![Page 44: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/44.jpg)
How to Improve its Resilience?
Alternative Approaches Synchronization of the music data
Time shift in Mp3 compression? Storing watermark in certain frequencies
(where less quantization occurs) Error Coding
Hamming Reed-Solomon
![Page 45: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/45.jpg)
Conclusion
Wave to Wave Format Very good performance (even for 8-bit wave
files) SNR is very low. Music Integrity is excellent
Mp3 Compression Very bad performance Alternative methods need to be found! Need a greater understanding of current Mp3
Compression Algorithms
![Page 46: EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c765503460f9492b484/html5/thumbnails/46.jpg)
References
[1] Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.
[2] Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000.
[3] Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis. http://www.cis.hut.fi/~aapo/.
[4] Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. http://www.cis.hut.fi/~aapo/. April 1999.
[5] Hyvarinen and Oja. Independent Component Analysis: A Tutorial. http://www.cis.hut.fi/projects/ica/. April 1999.
[6] Introduction to Blind Source Separation. http://www.cnl.salk.edu/~tewon/Blind/. [7] Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th
International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.
[8] Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998.[9] Shen and others. A Method for Digital Image Watermarking Using ICA. 4th
International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.