Perceptually Consistent Example-based Human Motion Retrieval

Zhigang Deng*, Qin Gu, Qing Li University of Houston

Introduction

Popularization of human motion capture data in animation and gaming applications

Efficient retrieval of similar motions from a large data repository Fundamental basis for many motion data based

applications

e.g. CMU motion capture library. http://mocap.cs.cmu.edu.2605 trials in 6 categories and 23 subcategories.

Related Work – Motion retrieval Transform original high-dimensional human motion data

to a reduced representation [Agrawal et al. 1993; Faloutsos et al. 1994; Chan and Fu 1999; Liu et al. 2003; Chiu et al. 2004; Baciu 2006; Lin 2006] .

Match webs [Kovar and Gleicher 2004]

Describe potential subsequence matches between any pair of motion sequences.

Semantics-based motion retrieval [Muller et al. 2005; Muller and Roder 2006] Users provide a query motion as a set of time-varying geometric

feature relationships.

Human hierarchy construction

Motion segmentation and normalization

Motion pattern detection and indexing

Hierarchical pattern matching

Search result ranking

Our Approach PipelineMotion Data Preprocessing

Runtime Motion Query

Data Preprocessing - Motion Hierarchy Construction Decompose human motion into a hierarchical

structure [Gu et al. 08]

Local control granularity Correlations among different human parts are embedded in

different layers 4 layers, 18 parts are used in this work.

Data Preprocessing - Motion Segmentation and Normalization Existing human motion segmentation techniques

Angular acceleration [Zhao 01, Fod et al. 02, Kim et al.03], SVM

classifier [Li et al. 07], weighted sum of marker velocities [Gu et al. 08], PCA/PPCA [Barbic et al.04].

Probabilistic PCA [Barbic et al. 04] is used to segment motion into short motion segments for each body part in the hierarchy.

Parts Head LHand LArm RArm

Ave Frm + Var

18.32 + 2.32

8.43 + 6.43

11.39 + 6.54

12.75 + 7.34

Parts Torso RLeg LFoot RFoot

Ave Frm + Var

13.43 + 5.65

11.24 + 5.12

6.75 + 5.35

6.05 + 5.88

Average Frame Information of segments

Data Preprocessing - Clustering

Motion Pattern for each body part A representative motion segment for a node (i.e.,a body

part) in the constructed human hierarchy Normalization of motion segments

Adaptive K-Means clustering Increase K when the clustering error metric is larger than a

threshold Resulting data structures

(1) Motion Pattern Library, (2) Pattern Index Lists, (3) Pattern Dissimilarity Maps.

Review of Motion Preprocessing

Runtime Motion Query Query motion transformation

Map the query motion into a motion pattern index list for each hierarchy node

Fast (no clustering, just database matching) Motion similarity score computing

Local motion similarity between two index lists Extended Knuth-Morris-Pratt (KMP) string matching

algorithm [Knuth et al. 77] Global motion similarity computing and ranking

Hierarchical propagation

Local Motion Similarity

Similarity between two pattern index lists Different length of index lists Matching of two integer lists

Extended KMP String match algorithm Introducing “Quasi-Match” based on the pre-constructed

pattern dissimilarity maps Large numbers of different motion segments Distance is less than a threshold

Update matching score If the number of consecutive quasi-matches is larger than a

threshold, otherwise decrease.. Score normalization based on the length of index lists

Global Motion Similarity

Hierarchical Score Propagation High local motion similarity does not mean global

motion similarity Nodes in the upper levels encode more global motion

information From bottom to top

Ranking of the final scores at the root node

Review of Runtime Motion Retrieval

Results and Evaluation

Time and Storage Search Accuracy Search Quality Perceptual Consistency Experiment

Results and Evaluations – Time and Storage

We tested our method on four datasets with different sizes

The test computer with a Intel Duo Core 2GHz CPU and 2GB memory.

The average duration of used query motions is 10 seconds.

56MB, 170 motions,68,293 frames456MB, 396 motions, 556,097 frames976MB, 542 motions, 1,190,243 frames1452MB, 941 motions, 1,770,731 frames

Results and Evaluations – Search Accuracy Accuracy evaluation scheme [Kovar and Gleicher 04]

Two different types of datasets: single-type motion datasets (pre-labeled dataset with the same semantic category, walking) – Ground truth, mixed motion dataset (unlabeled, mixed of various types).

True-positive accuracy ratio is defined top N (=20) results from mixed motion datasets are in the correct/expected single-type motion dataset.

56M test dataset: 170 sequences, 68,293 frames, five categories – walking, running, jumping, kicking, basket-playing.

Results and Evaluation – Comparative User Studies

Compare our approach with match-webs approach [Kovar and Gleicher 04], piecewise linear space [Liu et al. 05], weighted PCA [Forbes and Fiume 05]. Semantic-based motion retrieval [Muller et al. 05] was not

chosen, because of significant differences in input requirements.

Two usability questions (a) Perceptual Consistency: Retrieved results (motions) are

ranked in a perceptually consistent order? (b) Search Quality: Motion similarities of retrieved results?

Results and Evaluation – Comparative User Studies Perceptual-consistency

Computer algorithms rank motions in a certain order, C. Humans rank these (the same) motions in another order, H. Relationship/consistency between C and H?

Study Experiments 3 query motions (walking, running, basketball-playing),Top-ranked

N (=6) results for query, 4 approaches, total 72 = 3*6*4 results. Side-by-side comparison and user rating (one is a searched

motion, the other is the query motion), in a random order. Rating is from 1 (“completely different”) to 10 (“identical”). 24 experiment participants

Results and Evaluation – Comparative User Studies

Quality of searched motions Compute average similar ratings and standard

deviation Higher the average similar rating is, the better

quality of search it achieves.

Results and Evaluation – Comparative User Studies

Perceptual-Consistency Plot human-rankings vs computer-

rankings in a 2D space. Ideal consistency is shown as a straight

line.

Canonical Correlation Analysis Scale-invariant optimum linear

framework

CCA Coefficient results

Walking

Running

Basketball-playing

Review of User Studies

Conclusions An efficient, example-based human motion retrieval

technique Major distinctions of our approach

Efficiency Linear to the size of query motion and database size

Flexible search query A human motion subsequence, or a hybrid of multiple motion sequences

Perceptually consistent search outcomes Comparative user studies to find out the correlations between the result-

ranking by computer algorithms and the result-ranking by humans

Discussion and Limitations

Current approach does not consider the path/motion trajectory of the root of the human in the retrieval algorithm. The search results may enclose different paths/trajectories.

Current approach can only search for single-character motion sequences.

Future Work

A number of empirical parameters of current approach may critically affect the search accuracy and outcomes. Establish quantitative correlations between

“parameter setting” and “search accuracy and outcomes”.

Graphics hardware accelerated, motion query processing.

Thank You!

Questions?