![Page 1: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/1.jpg)
Thomas Abraham, PhD([email protected])
![Page 2: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/2.jpg)
• Deconvolution, also termed as “Restoration” or “Deblurring” is an image processing technique used in a wide variety of fields from 1D spectroscopy to 3D biomedical imaging.
• It has been rigorously applied in the field of astronomy, because so much expense and time are spent in acquiring astronomical images.
What is Deconvolution?
Raw Image Deconvolved
![Page 3: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/3.jpg)
• An image is produced by an instrument. It is not the same as the object but is some “photocopy” of the object.
What makes an image?
Convolution
• The PSF describes what the instrument does to the object to produce an image.
• In mathematical terms, the image is the object convolved by the ‘point spread function’ (PSF).
Image = Object PSF
![Page 4: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/4.jpg)
• Every lens element in the optical path alters the object (specimen) in some way.
• We call this the point spread function (PSF).
• The PSF of the system is a convolution of all of the PSF’s of all the lens elements
and apertures in the optical path.
Point Spread Function {PSF}
![Page 5: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/5.jpg)
• The image of that macromolecular object can be considered as the superposition of large number of points convolved with the PSF.
• Likewise, a macromolecular object can be considered as a large number of points arranged in 3D space.
• For instance, an image of point source is not a point but a point spread in 3D space.
Point Spread Function {PSF}
Point Source
Image
![Page 6: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/6.jpg)
Image(x,y,z) = ∫∫∫Object(x’,y’,z’)psf(x-x’,y-y’,z-z’)dx’dy’dz’
Convolution Theorem
In mathematical terms, for each point, take the object and multiply it by the entire PSF centered at that point in the image and sum the results of the multiplication.
![Page 7: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/7.jpg)
Convolution Theorem
Whatever we record as the image intensity is the convoluted product of the PSF and the object intensity.
Restoring the object intensity from the smear of PSF by some mathematical operations
Image = Object PSF
Image(x,y,z) = ∫∫∫Object(x’,y’,z’)psf(x-x’,y-y’,z-z’)dx’dy’dz’
Deconvolution
![Page 8: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/8.jpg)
The Fourier Transform
I(x) ai(coskix) ibi(sinkix)i
Amplitudes Phase
From spatial domain to Frequency Domain
•Any image can be decomposed into a series of sine and cosine wave functions.
![Page 9: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/9.jpg)
Low frequency(Coarse structural details)
High frequency (Fine structural details)
Fourier Transform of the elastin collagen fiber bundle in lung alveoli
FT
Closely related to the diffraction pattern of the object
Spatial Domain Frequency Domain
PSD
![Page 10: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/10.jpg)
Fourier Transform, Examples
FT
FT
Dude (Big Labowski)
Spatial Domain Frequency Domain
Bad looking Vs Good Looking
![Page 11: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/11.jpg)
Fourier Transform, Examples
FT
FT
Spatial Domain Frequency Domain
Unidirectional Objects Vs Randomly Oriented ObjectsFrequency distribution
is orthogonal
![Page 12: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/12.jpg)
Fourier Transform, Examples
FFT
FFT
Spatial Domain Frequency Domain
Fine Vs Fuzzy Objects
![Page 13: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/13.jpg)
Inverse Fourier Transform of the FourierTransform Returns the original Image
F -1
A powerful mathematical tool for image processing
=
![Page 14: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/14.jpg)
Deconvolution using Fourier Transformation
It has been proved that the Fourier transform of the image is just the Fourier transform of object times the Fourier transform
of the PSF in frequency domain
So, to obtain the object, we simply divide Fourier transform of Image by the Fourier transform of the PSF
F {Image} = F {Object PSF}
F {Object} = F {Image} / F {PSF}
This is known as Optical Transfer Function or OTF or CTF
F {Image} = F {Object} X F {PSF}
![Page 15: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/15.jpg)
By applying an inverse Fourier transform, We can restore object
(specimen)
Transforms back to spatial domain
Object (or sample) = F {Object} -1
![Page 16: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/16.jpg)
Step 1: Acquire the image and obtain its Fourier transform
Convoluted Image{Object
Fourier Transform of Image ⊗ PSF}
Steps in Deconvolution
![Page 17: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/17.jpg)
Step 2: Generate PSF and obtain its Fourier transform
Steps in Deconvolution
Point spread function Fourier transform of PSF
![Page 18: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/18.jpg)
Step 3: Divide and inverse Fourier transform
F -1
Steps in Deconvolution
Object
![Page 19: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/19.jpg)
FT
FT
Raw Image
Deconvolved Image( Restored Object)
![Page 20: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/20.jpg)
F {Object} = F{Image} / F{PSF}
OTF or CTF
Object (or sample) = F.T. {Object} -1
Works perfectly if there is no noise in the image. But there is always noise
![Page 21: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/21.jpg)
Various Sources of NOISEPhoton (or shot) noise: Photon detection is a quantum mechanical event. So, it is subjected to random fluctuations (or random noise) which is governed by Poisson statistics.
Electronic (thermal) noise: The light is detected electronically and all detectors have thermal noise.
Bit noise: Digital images have noise due to the number of bits (~8-14) used during A/D conversion.
Processing noise: The computer is operating on digital data with limited precision.
![Page 22: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/22.jpg)
Effect of noise in data
More Noise
Relatively Noise FreeLess Noise
Each spatial frequency has an associated noise component resulting from various sources of the noise, and at high spatial frequencies (corresponds fine structural details) this can be very large.
![Page 23: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/23.jpg)
Non-iterative Deconvolution
So simply dividing F{Image} by OTF will magnify noise components - this obscures real structural details.
Alternate approach is to modify OTF (or CTF) on the basis of local signal levels –Wiener filter.
F {Object} = F{Image} / F{PSF}
OTF or CTF
![Page 24: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/24.jpg)
Deconvolution using Wiener filter
• Here an extra noise dependent factor (K) is added to the OTF values, so that in noisy regions the divisor is larger, and the noisy components are not boosted.
• One major drawback here is that we don’t actually know what is real signal and what is noise.
• This often results in rejecting genuinely real spatial information. Consequently the reconstructed image can be of lower resolution.
![Page 25: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/25.jpg)
![Page 26: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/26.jpg)
![Page 27: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/27.jpg)
![Page 28: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/28.jpg)
Deconvolution by Iteration1) Take a guess at the object2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too.3) If blurred guess = image then stop iteration, otherwise refine guess.
This is also known as maximum likelihood approach.
Richardson-Lucy algorithm (which includes Poisson Noise model) :
![Page 29: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/29.jpg)
xy plane
xz plane
A
F
CB
D E
Abraham et al. (2010) Micron
![Page 30: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/30.jpg)
xy plane
xz plane
A CB
Abraham et al. (2010) Micron
![Page 31: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/31.jpg)
A B
C D
Abraham et al. (2010) Micron
![Page 32: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/32.jpg)
• Imaging is a convolution of the object by the microscope’s PSF and an image is a convoluted product of the object and PSF.
• Deconvolution attempts to restore the various spatial frequency components to something nearer what they were in the original object before being attenuated by the optical path.
Summary
![Page 33: Thomas Abraham, PhD · 1) Take a guess at the object 2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too. 3) If blurred guess = image then stop iteration,](https://reader033.vdocuments.site/reader033/viewer/2022060409/5f100eb77e708231d4473c0a/html5/thumbnails/33.jpg)
Recommended Readings:
Cannell, et al. (2006) Image Enhancement by Deconvolution, In Handbook of Biological Confocal Microscopy, Third Edition, Pawley, J. B, Ed., Springer Science + Business Media, LLC, New York
Shaw, P. (1994) Deconvolution in 3-D optical microscopy, HistochemicalJournal 26, 687-694.
Agard, D. A and Sedat, J. W. (1983) Three-dimensional architecture of a polythene nucleus. Nature 302, 676-81.
Sindelar, C.V and Grigorieff,N. (2011) An adaptation of the Wiener filter suitable for analyzing images of isolated single particles, J. Struct. Biol. 176, 60-74.
Sigworth, et al., (2010) An Introduction to Maximum-Likelihood Methods in Cryo-EM, Methods in Enzymology, Volume 482, 263-294