Medical University of Lübeck

Institute for Signal Processing

Nonlinear visual coding from an intrinsic-geometry perspective

E. Barth* & A. B. Watson

NASA Ames Research Centerhttp://vision.arc.nasa.gov

Supported by DFG grant Ba 1176/4-1 to EB and NASA grant 199-06-12-39 to ABW

Medical University of Lübeck

Institute for Signal Processing

Intrinsic dimensionality in 2D

• i0D: constant in all directions

• i1D: constant in one direction

• i2D: no constant direction

Medical University of Lübeck

Institute for Signal Processing

i0D

f (x, y) =const.FT

Medical University of Lübeck

Institute for Signal Processing

i1D

• e.g. straight lines and edges, gratings

FTf (x, y) =g(ξ)

Medical University of Lübeck

Institute for Signal Processing

i2D

• e.g. corners, line ends, curved edges and lines

FTf (x, y) =g(ξ,ζ )

Medical University of Lübeck

Institute for Signal Processing

i2Di1D

i0D

i0D

i1D

i2D

Medical University of Lübeck

Institute for Signal Processing

Intrinsic dimensionality in 3D

• i0D: constant in all (space-time) directions

• i1D: constant in 2 directions

• i2D: constant in one direction

• i3D: no constant direction

Medical University of Lübeck

Institute for Signal Processing

i0D

f (x, y, t) =const.

FT

Medical University of Lübeck

Institute for Signal Processing

i1D

• e.g. drifting spatial grating

f (x, y, t) =g(ξ)FT

Medical University of Lübeck

Institute for Signal Processing

i2D

e.g. drifting corner, flashed grating

f (x, y, t) =g(ξ,ζ )FT

Medical University of Lübeck

Institute for Signal Processing

i3D

e.g. flow discontinuities, flashed corners

f (x, y, t) =g(ξ,ζ,τ)FT

Medical University of Lübeck

Institute for Signal Processing

Intrinsic dimensionality and motion

• FT of (rigid) motion signal is in a plane

motion ⇔ i2D

The visual input as a hypersurface

luminance

( x , y , t , f ( x , y , t ))

f ( x , y , t )

hypersurface

Visualization of surfaces is easier:(x, y, f (x, y))

Medical University of Lübeck

Institute for Signal Processing

Geometric view on intrinsic dimensionality

i1D++

mean curvature

i2D+

Riemann curvature tensor

i3D

Gaussian curvature

Hypersurface Geometry

H ≠0 R ≠0 K ≠0

Medical University of Lübeck

Institute for Signal Processing

Curvature and motion

(“plane” = “more than line but no volume”)

motion ⇔ R≠0 ∧¬ K ≠0

Medical University of Lübeck

Institute for Signal Processing

The Riemann tensor R

• most important property of (hyper)surfaces

• measures the curvature of the (hyper)surface

• has 6 independent components in 3D

• vanishes in 1D.

Medical University of Lübeck

Institute for Signal Processing

The Riemann tensor components

R1 =fxxfyy− fxy

2

1+ fx2 + fy

2 + ft2

R2 =fxxftt − fxt

2

1+ fx2 + fy

2 + ft2 R3 =

fyyftt − fyt2

1 + fx2 + fy

2 + ft2

R6 =fxyftt− fxtfyt

1+ fx2 + fy

2 + ft2 R4 =

fxxfyt − fxtfxy

1+ fx2 + fy

2 + ft2 R5 =

fxyfyt − fyyfxt

1 + fx2 + fy

2 + ft2

are nonlinear combinations of derivatives, i.e., of linear filters with various spatio-temporal orientations.

Medical University of Lübeck

Institute for Signal Processing

R components and speed v

Multiple representation of speed.

R3

R1

=v2 (cosθ)2

R2

R4

=−v (sinθ)R4

R1

=−v (sinθ)

R3

R5

=v (cosθ)

R2

R1

=v2 (sinθ)2

R5

R1

=v (cosθ)R6

R4

=v (cosθ)

R6

R5

=−v (sinθ)

f : ′ f (x−tvcosθ,y−tvsinθ)

Medical University of Lübeck

Institute for Signal Processing

R and direction of motion q

R6

R3

=− (tanθ )R2

R6

= (tanθ)

R2

R3

= (tanθ)2

Multiple, distributed representation of direction.

R4

R5

=− (tanθ )

f : ′ f (x−tvcosθ,y−tvsinθ)

Medical University of Lübeck

Institute for Signal Processing

Sectional curvatures

x

′ t ′ x t

yvx =

Ky ′ t −Ky ′ x

2 Kxy

(R5 ) : R3221 =fxyfyt − fyyfxt

1 + fx2 + fy

2 + ft2 =

12

( fyyf ′ t ′ t − fy ′ t2 )−( fyyf ′ x ′ x − fy ′ x

2 )

1 + fx2 + fy

2 + ft2

Medical University of Lübeck

Institute for Signal Processing

Direction tunings of R componentsverticalmotion

horizontalmotion

Wallach, 1935

QuickTime™ and aAnimation decompressorare needed to see this picture.

Barber pole

Kooi, 1993

QuickTime™ and aAnimation decompressorare needed to see this picture.

“abolished illusion”

Rodman & Albright, 1989

Analytical predictions based on R components

Typical Type II MT neuron, macaque monkey

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1-1

-0.5

0

0.5

1-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

Direction tuning

Orientation tuning

Orthogonal orientation and direction tunings

Recanzone, Wurtz, & Schwarz, 1997

Analytical predictions based on R components

Typical MT neuron, macaque monkey

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

-1

-0.5

0

0.5

1

Multiple motions

Medical University of Lübeck

Institute for Signal Processing

(Reference to 3D world of moving objects is not needed.)

Conclusion

Hypothesis that a basic (geometric) signal property (the intrinsic dimensionality) is encoded in early- and mid-level vision explains– orientation selectivity (derivatives, and R2, R3)– endstopping

(all R components are endstopped for translations)– velocity selectivity

– direction selectivity– some global-motion percepts (by integration)– some properties reported for MT neurons.