Imaging through scattering

Ivo Vellekoop

Ivo Vellekoop

Biomedical imaging with light

0.1 mm 1 mm 10 mm 100 mm

2

2

< 50 nm resolution

~ 1 µm resolution

~ 0.5mm resolution

Ivo Vellekoop

More than tissue?

Insert your favorite specimen:

• Integrated circuits

• Microfluidic systems

• Organ on a chip

• Opaque catalysts

• Powders

All ideas are welcome!

Ivo Vellekoop

Newtonian Ray Optics

4

lens

All microscopes rely on Newtonian ray optics basic principle has not changed in 400 years:

Ivo Vellekoop

Scattering

In tissue light does not propagate along straight rays

Ivo Vellekoop

Biomedical imaging with light

0.1 mm 1 mm 10 mm 100 mm

6

2

Paradigm 1: microscopy

Paradigm 2: diffuse tomography

full body microscopy of an intact mouse

complete full mouse brain at dendritic resolution Dream on …

… or get to work!

Ivo Vellekoop

Looking through scattering

All microscopic approaches are aimed at: • Reducing scattering:

– Chemical clearing – Skull thinning – Multi-photon microscopy

• Rejecting scattered light – Optical coherence tomography – Confocal microscopy – Multi-photon microscopy

How about using scattered light?

Ivo Vellekoop

From speckle to focus

Vellekoop & Mosk. Opt. Lett. 31, 2309 (2007)

spatial light modulator (LCD)

Ivo Vellekoop

Before

detector

total field in target

Ivo Vellekoop

After

detector

total field in target

Ivo Vellekoop

Ivo Vellekoop

Focusing better than with a lens

Vellekoop, Lagendijk, Mosk, Nature Photonics 2010.

Ivo Vellekoop

Focusing light inside

Vellekoop, van Putten, Lagendijk & Mosk. Opt. Expr. 16, 67 (2008)

Ivo Vellekoop

Focus + scanning + fluorescence

Focus + scanning + fluorescence = microscope

Vellekoop & Aegerter. Opt. Lett. 35, 1245 (2010)

Ivo Vellekoop

The optical memory effect

Maximum scan angle (HWHM): α ≈ 0.24 λ/ L

Ivo Vellekoop

Imaging results

Used scattered light for imaging!

Ivo Vellekoop

Resolution

300 nm resolution

with multiply scattered light!

HWHM:300 nm

Vellekoop & Aegerter. Opt. Lett. 35, 1245 (2010)

Ivo Vellekoop

Inside?

How do you get the initial focus?

Need some form of feedback

We are not allowed to peek

Ivo Vellekoop

Now we have many options:

• Iterative – 1-photon fluorescence, ultrasound tagging,

photo acoustic effect, non-linear PA, 2-photon fluorescence, …

• Phase conjugation – Ultrasound tagging (TRUE)

– Movement tagging (TRACK)

– Fluorescence, 2nd harmonic, …

Ivo Vellekoop

Imaging through a mouse skull

Park, Sun & Cui. PNAS 2015

Feedback comes from strength of 2-photon signal!

Park, Sun & Cui. PNAS 2015

Ivo Vellekoop

“High-resolution in vivo imaging of mouse brain through the intact skull” Park, Sun & Cui. PNAS 112, 9236 (2015) [scale bar = 5 µm]

Ivo Vellekoop

Ivo Vellekoop

Are we already there?

Challenge: - Problem for looking through very thin scattering

layer more or less solved

- How about looking inside a thick scattering material? - Focusing possible. - Need ways to scan!

- fundamental understanding of wave correlations inside forward scattering materials (tissue)

- experimental validation - numerical validation

Ivo Vellekoop

Anisotropic scattering

Ivo Vellekoop

Memory effects

Gerwin Osnabrugge

Tilt/Tilt memory effect (thin layers only)

L

Feng et al. PRL 1988.

Shift/Shift memory effect (tissue-like scattering only)

Judkewitz et al. Nat. Phys. 2015.

Ivo Vellekoop

Combining shift and tilt

Feng et al. PRL 1988. Judkewitz et al. Nat. Phys. 2015. Osnabrugge et. al. (in preperation)

Ivo Vellekoop

Joint memory effect

Experiment Simplified analytical model

Osnabrugge et. al. (in preperation)

Ivo Vellekoop

Combining shift and tilt

Osnabrugge et. al. (in preperation)

Rotate one way, shift the other way Maximizing overlap ‘volume’

Ivo Vellekoop

How does it look inside the medium?

Ivo Vellekoop

Shift memory effect inside?

Ivo Vellekoop

About the simulations

Pack

er e

t al

. Nat

ure

Met

ho

ds

6, 2

10

2 (

20

12

)

Ivo Vellekoop

Validation of simulation method

Osnabrugge, Leedumrongwatthanakun, Vellekoop. J. Comp. Physics 2016.

Pseudospectral time domain: More accurate? slower

Our method: Alwayst fast & extremely accurate

Ivo Vellekoop

Almost perfect focus

Ivo Vellekoop

Aberrated focus

Ivo Vellekoop

• No ballistic component distinguishable

• No homogeneous diffuse background

• All features are sharp (speckle)

Ivo Vellekoop

Ivo Vellekoop

Ivo Vellekoop

Ordinary scanning

x [µm]

y [µ

m]

39

Ivo Vellekoop

Exploiting the memory effects

optimally (simulation)

40

x [µm]

y [µ

m]

Ivo Vellekoop

Conclusion

• Scattered light can be used for microscopic imaging

• Imaging through a thin slab of material is possible

• Imaging inside tissue still challenging, but progress:

– Anisotropic memory effect can be used

– Theoretical understanding of joint shift/tilt effects

– Extremely efficient & accurate simulation tools

– Promising results in simulation

Ivo Vellekoop

Biomedical Photonic Imaging (University of Twente)

Gerwin Osnabrugge

Saroch Leedumrongwatthanakun

Changhuei Yang lab (Caltech)

Benjamin Judkewitz lab (Charité Berlin)

Roarke Horstmeyer, Ioannis Papadopoulos

Allard Mosk (University of Utrecht)

Complex Photonics Systems (University of Twente)

Willem Vos, Ad Lagendijk

Christoph Aegerter lab (University of Zurich)

Acknowledgements

PhD and PD positions available

Ivo Vellekoop

Imagine…

Elke Vockenhuber

SCATTERING AND ABERRATIONS IN MICROSCOPY

Ivo Vellekoop

Aberrations

• Lenses – Spherical aberrations – Chromatic aberrations – Astigmatism

• Cover-glass • Misalignment • Sample

Ivo Vellekoop

Adaptive optics

Ray optics, smooth aberrations

Correct for: • System aberrations • Turbulent atmosphere • Lens of human eye • Cover slip thickness • …

Ivo Vellekoop

Aberrations vs scattering

Ray optics, smooth aberrations

Multi-path interference, wavelength-scale disorder speckle

Ivo Vellekoop

Atmosphere Paint

Ray optics Wave optics: diffraction, multiple scattering, interference

1 ray in = 1 ray out 1 ray in = multi-path interference

1 optical path length Distribution of optical path lengths -> diffusion in both space and time

Start with low-quality image Start with speckle

Ivo Vellekoop

Are scattering and aberrations fundamentally different things?

No: they are different ends of a spectrum of distortions

(like ice and steam are basically the same thing)

Ivo Vellekoop

Is this Adaptive Optics?

Absolutely: there is optics and there is adaptation

However: • For decades, the term Adaptive Optics has

been used far to narrowly, dealing mostly with aberrations in optical imaging systems

• Until 2007, a layer of paint was not considered to be an optical imaging system

Magic You can do the same without first

creating a focus!

Ivo Vellekoop

Raster scanning with unknown PSF

Bertolotti, van Putten, Blum, Lagendijk, Vos & Mosk, Nature 491, 232–234 (2012).

Ivo Vellekoop

How does the magic work?

• Raster scanning microscope

• With unknown PSF and unknown object

• Use statistical properties of PSF together with phase retrieval algorithm to recover image

Ivo Vellekoop

Single shot magic

Katz, Heidmann, Fink & Gigan, Nature Photonics 8, 784–790 (2014)

Ivo Vellekoop

Applications?

• Very cool ideas • For small objects far behind a thin screen

Applications for microscopy limited: – Through thin aberrating layer (skull) – Imaging inside eggs?

• Without memory effect, you need a different wavefront for every given scan point.

• This wavefront is a-priori unknown

Ivo Vellekoop

Novel memory effect

Judkewitz, Horstmeyer, Papadopoulos, Vellekoop & Yang, Nature Physics 2015.

Ivo Vellekoop

Exploiting the memory effect

THE MICROSCOPE OF THE FUTURE?

Ivo Vellekoop

Scanner

Future

Laser

Primary information: fluorescence signal

Secondary information: • wavefront distortion • OCT • background signal • sample movement/distortion

Smart Scanner

a-priori information: • system aberrations • memory-effect ranges • guide-star shape & distribution • refractive index models • recognizable features

Spatial Light Modulator

Ivo Vellekoop

Where are we now?

(based on Ntziachristos, V. Nature Methods, 7, 603–614 (2010))

1 MFP ~ 30 µm 1 TMFP ~ 1 mm

Paradigm 1: microscopy

Paradigm 2: diffuse tomography

Scattered light fluorescence microscopy

TROVE

TRUE

Brave new world: scattered light microscopy

full body microscopy of an intact mouse

Ivo Vellekoop

Conclusion

High-resolution imaging with scattered waves

A largely unexplored territory that requires a completely different way of thinking about imaging.

• Focus light anywhere if ‘only’ the correct wavefront is known

• First applications are already a fact.

Ivo Vellekoop

The Future

Microscopic endoscopy through a thin injection needle Full body microscopy of living animals Seeing through ‘walls’, fog, skin. Looking inside eggs, paint, composites, granular materials, IC’s, etc., etc.

Ivo Vellekoop

More reading

Review of feedback-based wavefront shaping:

• Vellekoop, Optics Express 23, 12189 (2015)

Review of digital optical phase conjugation:

• Horstmeyer et al., Nature Photonics (2015)

New memory effect

• Judkewitz et al., Nature Physics 11, 684 (2015)

Ivo Vellekoop

Time reversal of ultrasound encoded

light (TRUE)

• X. Xu et al. Nat. Photonics (2011).

• Y. M. Wang et al. Nat. Communications (2012). K. Si et al. Nat. Photonics (2012).

ultrasound lock-in 3D fluorescence microscopy

Ivo Vellekoop

TRUE Deep tissue fluorescence

microscopy

Deep-tissue focal fluorescence imaging with digitally time-reversed ultrasound-encoded light Ying Min Wang, Benjamin Judkewitz, Charles A. DiMarzio & Changhuei Yang Nature Communications 3, 928 (2012). Scale bars 50 μm.

Ivo Vellekoop

The problem

• US-assisted focusing (TRUE) is a great solution. Problem:

– Low resolution (~ 20 µm).

– Worse: contrast (~ 1:20)

• Contrast inversely proportional to number of modes in the target focus [Vellekoop & Mosk. Opt. Commun 281 (2008)]

Ivo Vellekoop

Solutions: • More pixels • Smaller ultrasound tagging volume • Longer wavelength • TROVE (Benjamin Judkewitz) / more efficient methods?

Not as bad as thought before: - intensity correlations result in more efficient focusing [Cui2016]

Ivo Vellekoop

Ivo Vellekoop

Example: wavefront-shaping in fatty tissue FDTD

71

Our novel approach FDTD

Computationally constructing transmission matrix of scattering tissue is becoming feasible

Ivo Vellekoop

Graphical representation

Re E

Im E

contribution of

segment 1 contribution of

segment 2

contribution of

segment N

Maximize Eb

Ivo Vellekoop

Algorithm

Adjust phase of individual segments

until contribution is in phase with total field

Ivo Vellekoop

Global maximum

before after

all the same phase

global maximum