soccer trajectory

73
Trajectory Analysis of Broadcast Soccer Videos Computer Science and Engineering Department Computer Science and Engineering Department Indian Institute of Technology, Kharagpur Indian Institute of Technology, Kharagpur by Prof. Jayanta Mukherjee [email protected]

Upload: akashag11111

Post on 25-Jan-2015

780 views

Category:

Documents


1 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Soccer Trajectory

Trajectory Analysis of Broadcast Soccer Videos

Computer Science and Engineering DepartmentComputer Science and Engineering Department

Indian Institute of Technology, KharagpurIndian Institute of Technology, Kharagpur

by

Prof. Jayanta Mukherjee

[email protected]

Page 2: Soccer Trajectory

Collaborators

•V. Pallavi --- research scholar. •Prof. A.K. Majumdar, CSE •Prof. Shamik Sural, SIT

Page 3: Soccer Trajectory

OUTLINE• Motivation and Objective

• State Based Video Model

• Extraction of Features

• Trajectory Detection

• States and Event Detection

Page 4: Soccer Trajectory

MotivationIncreasing availability of soccer videos

Soccer videos appeal to a large audience

Processing of soccer videos to deliver it over narrow band networks

Relevance of soccer videos drops significantly after a short period of time

Therefore soccer video analysis needs to be made automatic and the results must be semantically meaningful

Page 5: Soccer Trajectory

State based Video Model

Video data model : representation of information contained in the unstructured video in order to support users’ queries.

State based model: states of soccer video objects and their transitions (due to some event).

Page 6: Soccer Trajectory

State Chart Diagram for Ball Possession

Page 7: Soccer Trajectory

Immediate Goal

Our objective is to identify these states and their transitions by analyzing the unstructured video.

Page 8: Soccer Trajectory

Cinematic Features

– Shot Transitions

– Shot Types

– Shot Durations

Object Based Features

– Players

– Ball

– Billboards

– Field Descriptors

Features UsedFeatures Used

Page 9: Soccer Trajectory

Shots can be classified into:

• Long shot

– Captures a global view of the field

• Medium shot

– Shows close up view of one or more players in a specific part of the field

• Close shot

– Shows an above-waist view of a single player

Shot classification

Page 10: Soccer Trajectory

Typical long views in soccer videos

Grass covering entire frame Grass covering partial frame

Page 11: Soccer Trajectory

Shot Classification (contd..)Shot Classification (contd..)

Soccer Video Sequence

If dominant color is green

Dominant color ratio >0.75 and

<=1.0

Long Shot Medium Shot

Dominant color ratio >0.5 and

<=0.75

Close Shot

Dominant color ratio >0.25 and

<=0.5

Page 12: Soccer Trajectory

Shot Classification Results

20147211078Close Shot

73841837Medium Shot

14424325830Long Shot

Unclassified Shot

(No of frames)

Close Shot

(No of frames)

Medium Shot

(No of frames)

Long Shot

(No of frames)

Predicted ClassTrue Class

Page 13: Soccer Trajectory

87.6387.63Close ShotClose Shot

83.7683.76Medium Medium ShotShot

96.6896.68Long ShotLong Shot

% of True % of True ClassificationClassification

Shot TypeShot Type

Shot Classification Results

Page 14: Soccer Trajectory

Shot detection improved by shot classification

Page 15: Soccer Trajectory

Object Based FeaturesFeature extraction for grass pixels

Each frame is processed in YIQ color space.

It is found experimentally that grass pixels have ‘I’ values ranging between 25 and 55 while ‘Q’ values range

between 0 and 12.

0 50 100 150 200 2500

50

100

150

200

250

Q v

alu

es

I values

Grass Values

Page 16: Soccer Trajectory

Playfield region detected

Grass pixels detected for a long view frame

Page 17: Soccer Trajectory

Object Based Features (contd..)

Playfield Line Detection

A playfield line separates playfield from the non playfield background which are usually the billboards (also called advertisement boards).

Hough transform is used to detect the playfield line.

Page 18: Soccer Trajectory

Object Based Features (contd..)

Midfield line is the line that divides the playfield in half along its width. Hough transform is applied to detect the midfield line.

Midfield Line Detection

Page 19: Soccer Trajectory

Ball DetectionObject Based Features (contd..)

Challenges :

Features of the ball (color, size, shape) vary with time

Relative size of the ball is very small

Ball may not be an ideal circle because of fast motion and illumination conditions

Objects in the field or in the crowd may look similar to a ball

Field appearance changes from place to place and time to time

No definite property to uniquely identify ball in a frame

Page 20: Soccer Trajectory

Detecting Ball Candidates in Long Shots

• Obtain ball candidates by detecting circular regions by using circular Hough Transform

• Filter the non ball candidates by :

– Removing candidates from channel’s logo

– Removing candidates from gallery region

– Removing candidates from midfield line

– Filtering out the candidates moving against the camera

Object Based Features (contd..)

Page 21: Soccer Trajectory

Ball candidates before and after filtering

Ball candidates before filtering Ball candidates after filtering

Object Based Features (contd..)

Page 22: Soccer Trajectory

Detecting Players in Long Shots

Challenges

• Features of the players (color, texture, size, motion) are neither static nor uniform

• Players appear very small in size

• Size of players changes with their position and zooming of cameras

• Color and texture of the jersey and shorts vary from team to team

• Players in the field do not have constant motion

Object Based Features (contd..)

Page 23: Soccer Trajectory

• Obtain player pixels by removing non player pixels :

– Removing grass pixels

– Removing the broadcasting channel’s logo

– Removing the extra field region (billboards and gallery)

– Removing pixels from the midfield line

• Segment the image containing player pixels to isolated player regions by :

– Region growing algorithm

– Center of the bounding rectangle of each region is said to be the location of the player

Detecting Player Regions

Object Based Features (contd..)

Page 24: Soccer Trajectory

A Long Shot View

Object Based Features (contd..)

Page 25: Soccer Trajectory

Player pixels detected

Object Based Features (contd..)

Page 26: Soccer Trajectory

Players detected in long shot views

Object Based Features (contd..)

Page 27: Soccer Trajectory

Team Identification in Soccer Videos

Players in a soccer videos are classified using a supervisory classification method.

Mean I and Q values of the player regions are obtained by randomly selecting a few frames

The minimum and maximum I and Q values are set as the range for classifying player regions

Feature Detection (Contd.)

Page 28: Soccer Trajectory

Team Classification in Soccer Videos

Experiments were performed on two different matches:

Real Madrid and Manchester United (UEFA Champions League 2003)

Chelsea and Liverpool (UEFA Champions League 2007)

Feature detection (contd.)

Page 29: Soccer Trajectory

Team Classification Results

72

(11.51)

6520Team B

50

(25)

3

(17.86)

Unclassified

173

(16.29)

0725Team A

UnclassifiedNo of players (%)

Team BNo of players (%)

Team ANo of players (%)

Predicted ClassTrue Class

Real Madrid and Manchester United

Page 30: Soccer Trajectory

Chelsea and Liverpool

72

(9.94)

6520Team B

503

(37.5)

Unclassified

173

(19.27)

0725Team A

UnclassifiedNo of players (%)

Team BNo of players (%)

Team ANo of players (%)

Predicted ClassTrue Class

Team Classification Results (contd..)

Page 31: Soccer Trajectory

Camera Related FeatureObject Based Features (contd..)

Camera Direction Estimation :

1. Optical Flow velocities and their directions are computed using Horn and Shunck’s method.

2. Based on the sign of the horizontal component of the majority pixels in a frame, the direction of movement (left or right) of the camera is estimated.

Page 32: Soccer Trajectory

Camera Direction Estimation (contd..)

Optical flow velocities for the camera moving towards right

Page 33: Soccer Trajectory

Tracking of Broadcast Video Objects

Challenges

Camera parameters are unknown

Cameras are not fixed

Cameras are zoomed and rotated

Broadcast video is an edited video

Page 34: Soccer Trajectory

Construction of a Directed Weighted Graph

Objects in a frame form nodes.

Between two correlated objects in two different framesan arc (edge) is formed.

The measure of correlation or similarity provide the weight.

Temporal direction provides the direction of the edge.

Page 35: Soccer Trajectory

Object Trajectory Detection

Given a source node, longest path of the graph obtained by dynamic programming gives the path of the object.

Tracking of Broadcast Video Objects (contd..)

Page 36: Soccer Trajectory

Ball detection results for long shots

10010027272741400-41940

91.8598.03609597650Liang et al * sequence 1

95.8398.26689677719Liang et al * sequence 2

94.7494.7419181937020-37400

10010020202040500-40900

92.7396.2353515535900-37000

93.3393.3330283034800-35400

91.395.4522212330300-30760

9094.7319182023800-24200

PrecisionRecallBall present in

(number of frames)

Ball identified in

(number of frames)

Total

(number of frames)

Frame Range

Average Recall is 96.75 % and Average Precision is 94.42 % * Liang D., Liu Y., Huang Q. \& Gao W., A Scheme for Ball Detection and Tracking in Broadcast Soccer Video, Pacific Rim Conference on Multimedia, 2005, 1, LNCS 3767, 864-875.

Tracking of Broadcast Video Objects (contd..)

Page 37: Soccer Trajectory

Results for ball detection in long shots (contd..)

Page 38: Soccer Trajectory

Given a source node (player in the first frame), longest path of the graph obtained by dynamic programming gives the path of the player in the whole sequence.

Tracking a Single Player

Player being tracked

Tracking of Broadcast Video Objects (contd..)

Page 39: Soccer Trajectory

Tracking Multiple Players

Longest path from each node (represented by players in the first frame) of the graph obtained by dynamic programming gives the trajectories of the players for the sequence of frames.

Limitations :

Occlusion between players

Players in contact

Similarity between players belonging to same team

Tracking of Broadcast Video Objects (contd..)

Page 40: Soccer Trajectory

Resolving Conflicting Player Trajectories

If more than one player has more than two common nodes in its trajectory then only one amongst them is true.

The path having maximum weight is said to be the true trajectory

Nodes constituting the paths of correctly detected players are removed and a graph is again constructed

Mistracked players are again tracked

Tracking Multiple Players (contd..)

Page 41: Soccer Trajectory

Multiple Player Detection Results

99.3799.48158815781586317Soccer 6

95.2810053050550568Soccer 5

83.58100670560560100Soccer 4

10085.7136036042060Soccer 3

81.81100220018001800200Soccer 2

81.03100311025202520180Soccer 1

Precision (%)

Recall (%)

SOP detected SOTP detectedSOP presentNo of FramesVideo file

Average Recall is 97.53 % and Average Precision is 90.18%

Tracking Multiple Players (contd..)

Page 42: Soccer Trajectory

Multi - Player Tracking Results

96.59153212901586317Soccer 6

91.4946235650568Soccer 5

100560540560100Soccer 4

90.4838030042060Soccer 3

100180012001800200Soccer 2

85.71216018002520180Soccer 1

Accuracy (%)

SOTP tracked by tracking

and retracking algorithm

SOTP tracked by tracking algorithm

SOP presentNo of FramesVideo file

Average Accuracy is 94.05 % .

Tracking Multiple Players (contd..)

Page 43: Soccer Trajectory

Occlusion Results

1002020Soccer 6

801620Soccer 5

1004444Soccer 3

55.552036Soccer 1

Accuracy

(%)

No of cases that could be solved

No of occlusion and contact cases

Video file

Average Accuracy is 83.89 % .

Tracking Multiple Players (contd..)

Page 44: Soccer Trajectory

Multi - Player Tracking Results

Page 45: Soccer Trajectory

Multi - Player Tracking with Occlusion Results

Page 46: Soccer Trajectory

Multi - Player Tracking with Occlusion Results (contd..)

Page 47: Soccer Trajectory

Multi - Player Tracking with Occlusion Results (contd..)

Page 48: Soccer Trajectory

Multi - Player Tracking with Occlusion Results (contd..)

Page 49: Soccer Trajectory

Tracking the Mistracked Player (contd..)

Page 50: Soccer Trajectory

Tracking the Mistracked Player (contd..)

Page 51: Soccer Trajectory

Tracking the Mistracked Player (contd..)

Page 52: Soccer Trajectory

Tracking the Mistracked Player (contd..)

Page 53: Soccer Trajectory

Detection of States and Events

The features extracted and the trajectories detected are used to detect states and events based on the proposed state based video model.

States identified

- Ball possession states

Events detected

- Ball passing events

Page 54: Soccer Trajectory

State Chart Diagram for Ball Possession

Detection of States and Events (contd..)

Page 55: Soccer Trajectory

State Detection

Ball possession states are obtained based on

Spatial proximity analysis:

Distance between nearest player and second nearest player to the ball

Spatial arrangements between the players and the ball

Page 56: Soccer Trajectory

Ball Possession State Detection

Ball in possession of player 1’s team

Ball in possession of player 1’s team

Page 57: Soccer Trajectory

Ball Possession State Detection

Ball in possession of player 1’s team

Ball in a fight state

Page 58: Soccer Trajectory

Ball Possession Results

8

(1.92)

39216

(3.85)

Team B

5412

(17.14)

4

(5.71)

Fight

8

(3.17)

2

(0.93)

206Team A

FightNo of frames

(% of misclassified

frames)

Team BNo of frames

(% of misclassified

frames)

Team ANo of frames

(% of misclassified

frames)

Predicted ClassTrue Class

Page 59: Soccer Trajectory

Edit Distance as performance measure for ball possession states

If the actual state sequence for a sequence of frames is:

AAAAFFFFFFFFFFBBBB

And if the state sequence obtained by the proposed algorithm is:

AAAAFFFFFFFFBBBBBB

Both the sequences are represented as strings S1 and S2.

Edit distance D(S1, S2 ) is defined as the minimum number of point mutations required to change S1 to S2 where a point mutation is one of:

replacing an alphabet

inserting an alphabet

deleting an alphabet

Edit distance for the above sequence is 2. While normalized edit distance is:

D(S1, S2 )/| S1 |

|S|)S,S(D 121

Page 60: Soccer Trajectory

Shot wise ball possession results

Page 61: Soccer Trajectory

Event Detection

The event detected in this work is the ball passing event. It can be:

Forward pass

Reverse pass

Page 62: Soccer Trajectory

Event Detection (contd..)

The ball passing event cannot be detected

from state transition graphs because: Ball is usually passed between players of the same team State transition graphs show the change in ball possession states from Team A-Team B, Team B - Team A, Team B – Fight , Fight – Team B, Team A – Fight or Fight – Team A

Page 63: Soccer Trajectory

Schematic diagram for ball passing events

Ball is said to be passed in a sequence of frames, if:

Nearest player in the initial frames of the sequence is the second nearest player to the ball in the subsequent frames

Nearest and the second nearest players to the ball belong to the same team

Page 64: Soccer Trajectory

Example of a ball passing event:

Page 65: Soccer Trajectory

Example of a ball passing event (contd..)

Page 66: Soccer Trajectory

Example of a ball passing event (contd..)

Page 67: Soccer Trajectory

Example of a ball passing event (contd..)

Page 68: Soccer Trajectory

Classifying ball passing events

Forward pass:

Direction of camera motion is towards the goal post of the team opposite to that of the nearest player

Reverse pass:

Direction of camera motion is towards the goal post of the team of the nearest player

Page 69: Soccer Trajectory

Results for ball passing events

Average Recall = 100% and Precision = 60%

Page 70: Soccer Trajectory

Classification of ball passing events:

5-Reverse

13Forward

Reverse

(no of passes)

Forward

(no of passes)

False Ball PassesTrue Ball Passes

Page 71: Soccer Trajectory

Graphs for ball possession and ball passing

Graphs illustrating ball possession states and ball passing events for Sequence 7

Page 72: Soccer Trajectory

Graphs for ball possession and ball passing

Graphs illustrating ball possession states and ball passing events for Sequence 10

Page 73: Soccer Trajectory