authors: zhe cao, tomas simon, shih-en wei, yaser …yjlee/teaching/ecs289g...presented by: suraj...

Presented by: Suraj Kesavan, Priscilla Jennifer

Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

Authors: Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh

ECS 289G: Visual Recognition 02/27/2018

Introduction to Pose Estimation and Association

https://www.youtube.com/watch?v=pW6nZXeWlGM&t=54s

Challenges

● Unknown number of people that can occur in a frame.

● Complex Spatial Interference - Contact, Occlusion between people.

● Variance in person scales

● Run time Complexity.

Top-down Approach: Person Detection + Pose Estimation

Faster R-CNN(Person Detector) ResNet

Papandreou, George, et al. "Towards accurate multiperson pose estimation in the wild." arXiv preprint arXiv:1701.01779 8 (2017).

CNN

Parts Detection

Image

Bottom-up Approach: Parts Detection and Parts Association

Parts Association

Sub-network 1: Part Detection

S = (S1, S2, … SJ), Si is a confidence map - for each part (j )

http://image-net.org/challenges/talks/2016/Multi-person%20pose%20estimation-CMU.pdf

Sub-network 1: Part Detection p - (x, y) in an imageW(p) - Binary mask ⥺ {0, 1}St

j(p) - Confidence score for joint J ⥺ {1….J} at stage t S*

j(p) - Ground truth confidence map

xj,k - Ground truth of body part j for person kS*

j,k(p) - confidence score for joint j for person k

Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No. 2. 2017.

Sub-network 1: Parts Detection


Sub-network 2: Part Association using Part Affinity Fields

Part Affinity Fieldsencodes Orientationhttp://image-net.org/challenges/talks/2016/Multi-person%20pose%20estimation-CMU.pdf

v - normalized unit vector along Xj1,kXj2,klc.k - distance between Xj1,kXj2,kσl - limb width

nc(p) - No of non-zero vectors at point p for all k people

L = (L1, L2, … LC), Li is a vector field - for each limb (c )

Sub-network 2: Part Association using Part Affinity Fields


CNN Architecture

https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation

CNN

P

CNN

Stage 1

P……

Stage TP

CNN

……

Sequential Prediction with Learned Spatial Context

Right Wrist - Stage 1 Right Wrist - Stage 2 Right Wrist - Stage T

Stage 2


CNN

P

CNN

Stage 1

P

Stage TP

CNN

CNN

Stage 2

P

Stage T

CNN

2nd Branch Part Affinity

Fields

Jointly Learning Parts Detection and Parts Association

Stage 2


OpenPose Pipeline

https://arvrjourney.com/human-pose-estimation-using-openpose-with-tensorflow-part-2-e78ab9104fc8

Testing - Non-maximum Suppression

NMS

https://arvrjourney.com/human-pose-estimation-using-openpose-with-tensorflow-part-2-e78ab9104fc8

Testing - Line Integral

n1 n2 n3 n4

l1 0/1 0/1 0/1 0/1

l2 0/1 0/1 0/1 0/1

l3 0/1 0/1 0/1 0/1

n1 n2 n3 n4

l1 .1 2.5 2 .3

l2 .4 5 1 .5

l3 3 .3 .5 .1

Line integral

PAF

Bipartite graph Weighted Bipartite graph

Midpoint Score Map for Part-to-Part Association


Spatial Ambiguity of the Midpoint Representation

— Correct Connection— Wrong Connection


Increasing Midpoint Number Cannot Solve The Problem

— Correct Connection— Wrong Connection


Part Affinity Fields Avoid Spatial Ambiguity

Correct Connection Wrong Connection

Elbow Wrist


Greedy algorithm for Graph matching

Shoulder

Wrist

Elbow

Hungarian algorithm for Graph matchingShoulder

Wrist

Elbow

2.8

1 2

1 2

1 2

Wrist

Elbow

0.8 1.1

2.6

E1 E2

S1 2.8 0.8

S2 2.6 1.1

1 2

2.62.9

2.4

1.9E1 E2

W1 2.9 2.6

W2 2.4 1.9


Wrist

Elbow

2.8

1 2

1 2

1 2

Wrist

Elbow

0.8 1.1

2.6

E1 E2

S1 2.8 0.8

S2 2.6 1.1

1 2

2.62.9

2.4

1.9E1 E2

W1 2.9 2.6

W2 2.4 1.9

Maximum E = 2.8 + 2.9 = 5.7


Wrist

Elbow

2.8

1 2

1 2

1 2

Wrist

Elbow

2.6 1.1

0.8

E1 E2

S1 2.8 0.8

S2 2.6 1.1

1 2

2.62.92.4

1.9E1 E2

W1 2.9 2.6

W2 2.4 1.9

Maximum E = 1.1 + 1.9 = 3.0

https://docs.google.com/file/d/1RqRAMOkrB1R1V43Wtrjz7t177BIUhSJU/preview

Results on the MPII Multi Person Dataset

Comparison of mAP across other implementations on MPII Dataset.

Comparison of the graph matching algorithms on validation set.


Results on the MPII Multi Person Dataset

mAP curves of different experiments mAP curves at different stages of the experiment.


Results on COCO Challenge Validation Set

Comparison of results from the top-down approach with this approach.

Comparison of techniques which use Convolutional Pose machines(CPM) with this

approach.


Strength

Runtime Analysis

● Robustness to early commitment

● Run time for this method only increases slowly with the no of people in the image.


Weakness

● Greedy algorithm can fail to give a perfect matching.● It fails in certain cases of rare posture, false positives for statues, overlapping limbs.

Failure cases


https://docs.google.com/file/d/1KoIwhyiQxBqfGi3ert75FA3qwF0ZDk9r/preview

Questions and

Discussion.

authors: zhe cao, tomas simon, shih-en wei, yaser …yjlee/teaching/ecs289g...presented by: suraj...

Documents