Avalanche

Ski-Resort

Snow-Clad Mountain

Moving Vistas: Exploiting Motion for Describing ScenesNitesh Shroff, Pavan Turaga, Rama Chellappa

University of Maryland, College Park

Problem Definition and Motivation

Contributions

Dynamic Attributes

Dynamic Attributesmotion information from a global perspective.

Characterize the unconstrained dynamics of scenes using Chaotic Invariants.

Does not require localization or tracking of scene elements.Unconstrained real world Dynamic Scene dataset.

Dynamic Scene Recognition

Dynamics of scene reveals further information !!

Motion of scene elements improve or deteriorate

classification?

How to expand the scope of scene classification to videos?

What makes it difficult?

Scenes are unconstrained and ‘in-the-wild’ -- Large variation in scale, view, illumination, background

Underlying physics of motion -- too complicated or very little is understood of them.

Ray of hope !!!

Underlying process not entirely random but has deterministic component

Can we characterize motion at a global level ??

Yes using dynamic attributes and chaotic invariants

Modeling Dynamics

Requires No assumptions

Purely from the sequence of observations.

Fundamental notion -- all variables in a influence one another.

Constructs state variables from given time series

Estimate embedding dimension and delay

Chaotic Invariants[2,4]

ClassLDS[3]

(GIST)

Bag of

Words

Mean

(GIST)Dynamics

(Chaos)

Statics+

Dynamics

Toranado 70 10 70 60 90

Waves 40 50 70 80 90

Chaotic Traffic 10 20 30 50 70

Whirlpool 20 30 40 30 40

Total 25 24 40 36 52

[1] A.Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV, 2001

[2] M. Perc. The dynamics of human gait. European journal of physics, 26(3):525–534, 2005

[3] G. Doretto, A. Chiuso, Y. Wu, and S. Soatto. Dynamic textures, IJCV, 2003

[4]S. Ali, A. Basharat, and M. Shah. Chaotic Invariants for Human Action Recognition. ICCV, 2007.

References

Degree of Busyness: Amount of activity in the video. Highly busy: Sea-waves or Traffic scene --high degree of detailed motion patterns. Low busyness: Waterfall -- largely unchanging and motion typically in a small portion

Degree of Flow Granularity of the structural elements that undergo motion.

Degree of RegularityDegree of Busyness

Reconstruct the phase space.

Characterize it using invariants

Lyapunov Exponent: Rate of separation of nearby trajectories.

Correlation Integral: Density of phase space.

Correlation Dimension: Change in the density of phase space

Coarse: falling rocks in a landslide . Fine: waves in an ocean

Degree of Regularity of motion of structural elements.Irregular or random motion: chaotic traffic Regular motion: smooth traffic

Algorithmic Layout

GIST [1] for

each frame

Each dimension as time series

Chaotic Invariants

Classification&

Learn Attributes

Unconstrained YouTube videos Large Intra-class variation Available at

http://www.umiacs.umd.edu/users/nshroff/DynamicScene.html

Dynamic Scene Dataset

Recognition Accuracy

Linear Separation using Attributes

18 out of 20 correctly classified

Whirlpool Waves

Busyness