design of an expert - george mason university · 6 design of an expert system coach for complex...

Design of an Expert System Coach for Complex Team Sports

Group Lead:

Brice Colcombe

Group Members:

Lindsay Horton

Muhammad Ommer

Julia Teng

Sponsor:

Dr. Lance Sherry

Tactical Adjustment

Expert Coaching System

Game Video

Statistical Report

Winning

A10 Champs

Agenda

• Context

• Stakeholder Analysis

• Problem/ Need Statements

• Concept of Operations

• Mission/Functional/Design Requirements

• Simulation

• Risks

• Project Plan

• Next Steps

2


Objective of Game

• Soccer: Game between two teams of ten field players and one goalie per team

• Main Focus: George Mason University Men’s Soccer Team

• Field: A standard NCAA regulated field is 115-120 yds (length) by 70-75 yds (width) including two goals that are defended on either side of the field.

• Objective: The object of the game is to score more goals than the other team

3


Strategy

•Offense: ▫ Connect a series of passes to get around and

through the opposing team’s defense by using numerical advantage

• Defense: ▫ Win the ball back from opposing team ▫ Do not allow the opposing team to get near

your goal with the ball causing them to not score any goals

4

Strategies: Common Formations 4-4-2 4-3-3

5

Figure 3: 4-4-2 Formation Figure 4: 4-3-3 Formation

Types: Numerical Advantages: Types: Numerical Advantages:

• Flat • Wide Diamond • Narrow Diamond

• Numbers out wide • Support for attacking

• Flat • 2 Holding Midfielders • 2 Attack Midfielders • High/Low Wings

• Attack-centric • Strong central

midfield presence

Disadvantages: Disadvantages:

• Outnumbered in middle • Open to counterattack • Open on outside


Inverting the Pyramid: A History of Football Tactics (Jonathan Wilson) Coaching Rules

Complexity of Soccer • Baseball vs. Soccer: Discrete vs. Flowing

• Players have to make own decisions in order to

benefit the team

• Players have to adapt to the situations around them

• No instructions/stoppages until halftime

• Network Complexity of Soccer: Netcentricity

6


How Coaching is Done

• Traditional ▫ Use coaching expertise

▫ Watch practices/games, make changes accordingly

▫ Cannot ‘see’ full complexity of game

• New Generation ▫ Use data analysis to find holes in strategy, make

changes accordingly

▫ Able to ‘see’ patterns and complexity of game

• Hybrid Mix of Traditional and New Generation ▫ Use a mix of the two coaching strategies

7


8


Current Process

NCAA Tournament

• It is a single elimination tournament with a total of 48 teams

• 22 teams get an automatic bid from winning their conference championships

• The remaining 26 teams get an at large bid based on their teams rating percentage index (RPI) score

9


Source: NCAA (2015)

Rating Percentage Index [RPI] • Consists of Three Parts:

1. Teams Winning Percentage (Expert Coaching System)

𝑊+ 𝑇 2

𝑊+𝐿+𝑇 W = win, L = loss, and T = tie

2. Opponents Average Winning Percentage (Schedule Planning)

𝑂𝑊+ 𝑂𝑇 2

𝑂𝑊+ 𝑂𝐿−1 +𝑂𝑇,

𝑂𝑊+ 𝑂𝑇 −1 2

𝑂𝑊+𝑂𝐿+(𝑂𝑇−1),(𝑂𝑊−1)+ 𝑂𝑇 2

(𝑂𝑊−1)+𝑂𝐿+𝑂𝑇

(win , tie, loss) OW = Opponent’s win, OL = opponents loss, and OT = opponents tie

3. Teams A’s Opponent’s Opponents Average Winning Percentage (Schedule Planning)

𝑥 of Team A’s opponents’ part 2 Combine the three parts:

𝑹𝑷𝑰 =𝑝𝑎𝑟𝑡 1 + 2 ∙ 𝑝𝑎𝑟𝑡 2 + 𝑝𝑎𝑟𝑡 3

4

10


Historical Atlantic 10 Data (2004-2014)

11

23%

37% 37%

44% 46% 48%

51% 52% 52% 55%

57% 60%

74%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Win

nin

g P

erc

en

tage

(p

ast

ten

ye

ars)

A-10 Historic Winning Percentages

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Win

nin

g P

erc

en

tage

GMU vs SLU Winning Percentage

GMU Saint Louis GMU Average Saint Louis

Winning Percentage: 𝑊+ 𝑇 2

𝑊+𝐿+𝑇


Source: Atlantic 10 Team Archives (2015)

What a Win is Worth

12

School Salary Win % Conference

Wins NCAA

Championships Years Coaching

at School Total

Experience

UCLA 205,000 75% 4 4 12 14

Maryland 206,000 71% 4 2 23 25

UI 176,225 62% 1 3 6 12

UNC 88,044 58% 1 2 5 23

URI 72,500 55% 2 0 3 12

VCU 90,640 55% 0 0 6 15

UVA 115,400 52% 2 2 20 26

GMU 90,855 51% 2 0 11 24

UMASS 108,681 47% 1 0 1 24

Average Salary: $158,133.80 Average Atlantic 10 Coach: $90,676.50 GMU Head Coach worth per win: $9,085

A national championship level coach gets paid on average $67,457.30 per year more than an Atlantic 10 coach. Design of an Expert System Coach for Complex Team Sports

Gap Analysis

Replicate Saint Louis Success By: 1. Win Atlantic 10 Conference Championship 2 times

every 5 years 2. Receive an NCAA Bid 6 times every 10 years 3. Average RPI Score of .56

Resulting success will help close head coaches salary gap of $67,000

13


Agenda

• Context





• Simulation

• Risks

• Project Plan

• Next Steps

14


15


Stakeholder Analysis Class Stakeholder Goals Tensions

Primary

Head Coach Produce a team that gives highest probability of winning Keep job, get next big job

Pressure from University to win Old school coaching ideals

Assistant Coaches Scout best players for the team Prepare the team for games Become head coach

Disagreements with head coach Doesn’t always have a say in decisions

Player Perform at highest level Get to next level (pro)

Not getting played by coach Playing on a losing team

Trainers Have as few injuries as possible Rehab injured players

Can’t always control how hard coach pushes players

Secondary

Investors (University)

Fund a winning team Make money through ticket and memorabilia sales Market the University in a positive fashion

Want immediate success (not always possible) Budget is only so big

Parents Invest money in youth development in hopes of college scholarships and professional contracts

Can’t be involved with coaches like at the youth level

Academies Train youth players in hopes of signing with professional clubs

Players move, go onto college

Tertiary Data Analytic Companies Earn revenue from data software sales Multiple data companies competing with one another

16


Problem Statement

George Mason University Men’s Soccer Team is not consistently achieving NCAA Tournament bids at a high rate (2 bids out of last 10 years).

17


Need Statement

There is a need for a coaching tool that uses coaching expertise and uses soccer game data to understand the complexity of soccer and seek a competitive advantage in order to increase the probability of an NCAA Tournament bid to 3 bids out of every 5 years.

Solution Concept of Operations

18


Mission Requirements

19


Mission Requirement # Requirement Description

MR. 1.0 The Expert Coaching System (ECS) shall recommend XX possible

strategies based on game data gathered in real time.

MR. 2.0 The ECS shall recommend XX possible strategies based on gameplay

data gathered by half-time

MR. 3.0 The ECS shall recommend XX possible strategies based on gameplay

data gathered by overtime.

Functional Requirements

20


Functional Requirement # Requirement Description

FR. 1.0

The ECS shall run a simulated soccer game derived from

collected gameplay data.

FR. 2.0 The ECS shall accurately recognize soccer field patterns using

probability maps 90% of the time.

FR. 3.0 The ECS shall accurately quantify pass rate probabilities in-

between the 14 zones from game data 95% of the time.

FR. 4.0 The ECS shall accurately determine goal probability rates that

can be made from each of the 14 zones.

Design Requirements (Input/Output)

21


Input Requirement # Requirement Description

IR. 1.0 The ECS shall input statistics from Sports Data company providers.

IR. 2.0 The ECS shall input zone graph percentages from InStat to run the

simulation.

IR. 3.0 The ECS shall input XX coaching rules to be used to make

adjustments.

Output Requirement # Requirement Description

OR. 1.0

The ECS shall output a table showing the relationship between

current formations utilized and possible adjustments that are

recommended by the system.

OR. 2.0 The ECS shall output correct coaching adjustments that result in

30% greater chance of winning

OR. 3.0 The ECS shall output tactical adjustments xx amount of hours after

receiving game data from Data analytics’ providers.

Agenda

• Context





• Simulation

• Risks

• Next Steps

• Project Plan

22


Simulation • Objective: To simulate a game based on game data and

see if the coaching system rules will impact the outcome of the game.

• Input: Input passing data that is transformed into probability maps that are inputted into the simulation

• Output: table of passing percentages and strategy changes

• Simulates a 90 minute game (two 45 minute halves) using passing probability maps and different ball movements

• Simulation based on previous refereeing senior design project: Assessment of soccer Referee Proficiency in Time-Sensitive Decision Making Jones et.al.(2013)

• Currently over 10,000 lines of code

23


Simulation Requirements

Simulation Requirements # Requirement Description

SR 1.0 The simulation shall input zone graph data consisting of 14 zones of

a standard NCAA soccer field.

SR 2.0 The simulation shall follow strategies of George Mason University

Men’s Soccer Team formations.

SR 3.0 The simulation shall follow 1 set of probability maps per strategy.

SR 4.0 The simulation shall update the probability map of successful passing

rate percentages once the ball moves from one zone to another.

SR 5.0

The simulation shall change the possession of the ball after a shot or

intercepted pass.

SR 6.0

The simulation shall calculate average player location within 1 of the

14 zones in order to determine feasible pass rates from zone to zone.

24


Probability Maps

• Total of 15 probability maps per strategy

• 14 cell grid

• 13 possible passing opportunities per map

• Once the ball has successfully been passed to a new zone the probability map will change

25


Netcentricity – (Flow Centrality)

SHOT

*Data collected from 5 games played in 2015 season

26


Ball Movement Interaction Diagram

27


Simulation Interface

28


Output Table Time From Zone To Zone Possession Action Successful

80:08 7 6 Home Pass Yes


80:11 7 7 Home Dribble Yes

80:15 7 10 Home Pass No

80:17 10 13 Away Pass No

80:21 13 13 Home Dribble Yes


80.29 14 14 Home SHOT GOAL

29


Field Functions

Function Zones 4-3-3 Flat 4-3-3 Triangle 4-4-2 Flat 4-4-2 narrow

Diamond

Right Wing Zone 10 - 13 40% 30% 30% 25%

Left Wing Zone 0-3 40% 30% 30% 25%

Middle Zone 4-9 20% 40% 40% 50%

Defense Zone 0,4,5,10 20% 25% 20% 20%

Midfield Zone 1,2,6,7,11,12 50% 55% 50% 55%

Forward Zone 3,8,9,13 30% 20% 30% 25%

30

Coaching Rules (Numerical Advantage)

31


Current Strategy 4-3-3 Flat Tactical Change RWZone <= 40% and LWZone<= 40%

4-3-3 Triangle

Message: Ball needs to get out wide because passing in wing zones is xx change

formation

Dzone > 20% and Fzone is <= 30% 4-4-2 Flat

Message: Ball is being passed to much in the defensive zone and not enough

attacking is occurring. Switch formation to 422 flat.

Current Strategy meets criteria Message: continue to play with current strategy.

Current Strategy 4-3-3 Flat Tactical Change RWZone <= 40% and LWZone<= 40% 4-3-3 Triangle


formation

Dzone > 20% and Fzone is <= 30% 4-4-2 Flat

Message: Ball is being passed to much in the defensive zone and not enough

attacking is occurring. Switch formation to 422 flat.


Coaching Rules (Numerical Advantage)

32

Current Strategy 4-4-2 Flat Tactical Change

RWZone <= 30% and LWZone<=

30%

4-2-2 Diamond


formation

Dzone > 20% and Mzone is <=

50%

4-4-2 Diamond

Message: Ball is being passed too much in the defensive zone and not enough

attacking is occurring. Switch formation to 4-4-2 diamond to get the ball in the

midfield and forward.


Current Strategy 4-4-2

Narrow Diamond

Tactical Change

MidZone<= 50% 4-3-3 Triangle

Message: Need to find central midfielders with the ball. Change to a 4-3-3 triangle

and get ball inside.

Midzone <= 50% 4-4-2 Flat

Message: Ball needs to go through the middle more. Switch to a 4-4-2 flat to get the

ball to the wider flanks.


Validation of Concept of Operations

33


Score Differential from Monte Carlo Simulation

34

Design of Experiment Formations Replications

Right Wing Zone

Left Wing Zone Middle Zone Message Tactical Change

4-4-2

Flat 250

≤ 30% ≤ 30% ≤ 40% Get ball wide 4-4-2

Diamond

> 30% > 30% > 40% Continue Strategy

None

Diamond 250

≤ 25% ≤ 25% ≤ 50% Find central

players 4-3-3 Triangle


None

4-3-3

Flat 250

≤ 40% ≤ 40% ≤ 20% Get ball wide 4-3-3 Triangle


None

Triangle 250

≤ 30% ≤ 30% ≤ 40% Find central

players 4-4-2 Flat


None

35

The above design of experiment focuses on the various formations and the equivalent right, left and middle zone passing percentages

Design of Experiment

36

Formations Replications Defense

Zone Midfield

Zone Forward Zone Message Tactical Change

4-4-2

Flat 250

≥ 20% ≤ 50% ≤ 30% Get ball higher 4-4-2 Diamond

< 20% > 50% > 30% Continue Strategy

None

Diamond 250

≥ 20% ≤ 55% ≤ 25% Find central

players 4-4-2 Flat


None

4-3-3

Flat 250

≥ 20% ≤ 50% ≤ 30% Get ball high 4-4-2 Flat


None

Triangle 250

≥ 25% ≤ 55% ≤ 20% Find higher

players 4-4-2 Diamond


None

The above design of experiment focuses more on the defensive, midfield and forward zone passing percentages in order to run the various different formations

System Risks Risk Category What is the risk Risk Mitigation

Safety Will this system hurt someone? Verify the system doesn’t put human life into risk

Money Will this system lose money?

Compare win probability using system with how much each win is worth

Bad Advice Will this system give bad advice?

Test system with XX scenarios and verify accurate results with coaches

Expert System Rules Will accurate rules input into the system?

Talk to XX coaches to get various opinions of accurate rules

Simulation Will the simulation of the system be accurate?

Create XX scenarios to test simulation

37


Agenda

• Context





• Simulation

• Risks

• Project Plan

• Next Steps

38


Work Breakdown Structure

39


Critical Path

40


Budget Task Number Tasks Predicted Hours Cost

1 Expert Coaching System 2326 $ 116,300.00

1.1 Management 70 $ 3,500.00

1.2 Research 308 $ 15,400.00

1.3 CONOPS 93 $ 4,650.00

1.4 Originating Requirements 80 $ 4,000.00

1.5 Design Alternatives 85 $ 4,250.00

1.6 Design Of Experiment 88 $ 4,400.00

1.7 Simulation 995 $ 49,750.00

1.8 Analysis 130 $ 6,500.00

1.9 Conclusion 100 $ 5,000.00

1.1 Presentations 107 $ 5,350.00

1.11 Documentation 270 $ 13,500.00

41

Total Cost = (Average Junior SE salary per hour) * Overhead Cost * Predicted Hours

Source : GMU Office of Budget and Planning Design of an Expert System Coach for Complex Team Sports

Project Cost Analysis (week 1-14)

42


$-

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

0 2 4 6 8 10 12 14 16

Co

st (

Tho

usa

nd

s)

Time (weeks)

Planned Value Actual Cost Earned Value

Performance Indices (week 1-14)

43


0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

0 2 4 6 8 10 12 14 16

Per

form

ance

Ind

ex

Time (weeks)

Cost Performance Index (CPI) Schedule Performance Index (SPI)

Project Risks Critical Tasks Foreseeable Risks Mitigation

Data Collection • Unable to get GMU Men’s

Soccer Team data

• Use Instat dashboard to look

at other college Team’s data

Coaching Rules • Inaccurate rules input in the

system

• Talk to XX coaches to get

various opinions of accurate

rules

Develop Simulation

• Unable to run simulation

correctly

• The simulation of the system is

inaccurate

• Test and debug while

developing simulation

• Using Monte Carlos simulation

to test it XX times

Cost Analysis • Schedule Variance is negative

• Cost Variance is negative

• Work more hours than planed

to ensure project is on

schedule

• Work more hours than

planned to ensure actual cost

is under budget

44


Next Steps (WBS)

• Design of Experiment (1.6)

• Continue to add complexity to simulation (1.7.2.3)

• Analysis (1.8)

• Validation and Verification (1.9.2)

• Recommendations (1.9.4)

• Documentation (1.11)

45


46


“Only those who can see the invisible can do the impossible”

-Frank L. Gaines

Special Thanks • 2014 Referee Proficiency Senior Design Project

(Jones, Cann, Almashhadi, Popal) • Tom Morrell

References • [1] Virginiasports.com, 'VIRGINIASPORTS.COM - The University of Virginia Official Athletic Site - UVa Men's

Soccer', 2015. [Online]. Available: http://www.virginiasports.com/sports/m-soccer/archive/va-m-soccer-archive.html. [Accessed: 25- Oct- 2015].

• [2] Uclabruins.com, 'Men's Soccer - Schedule - UCLA Bruins Official Athletic Site | UCLABruins.com', 2015. [Online]. Available: http://www.uclabruins.com/SportSelect.dbml?SPSID=749841&SPID=126914&DB_OEM_ID=30500&Q_SEASON=2013. [Accessed: 25- Oct- 2015].

• [3] Atlantic10.com, 'Men's Soccer - News - Atlantic 10 Conference Official Athletic Site', 2015. [Online]. Available: http://www.atlantic10.com/SportSelect.dbml?DB_OEM_ID=31600&SPID=136213&SPSID=799925. [Accessed: 25- Oct- 2015].

• [4] Soccer-training-guide.com, 'The 4-3-3 is a Highly Popular Formation', 2015. [Online]. Available: http://www.soccer-training-guide.com/4-3-3.html#.Vi1ul36rTIU. [Accessed: 25- Oct- 2015].

• [5] J. Duch, J. Waitzman and L. Amaral, 'Quantifying the Performance of Individual Players in a Team Activity', PLoS ONE, vol. 5, no. 6, p. e10937, 2010.

• [6] J. Perez, Concepts and Coaching Guidelines, 1st ed. U.S Soccer Curriculum, 2015 [Online]. Available: http://resources.ussoccer.com/n7v8b8j3/cds/downloads/Part%202%20-%20Concepts%20and%20Coaching%20Guidelines%20U.S.%20Soccer%20Coaching%20Curriculum.pdf. [Accessed: 20- Oct- 2015]

• [7] J. Maxcy and J. Drayer, Sports Analytics: Advancing Decision Making Through Technology and Data, 1st ed. Philadelphia: Fox School of Business, 2014 [Online]. Available: http://ibit.temple.edu/wp-content/uploads/2014/04/IBITSportsanalytics.pdf. [Accessed: 20- Oct- 2015]

• [8] Sites.google.com, 'RPI: Formula - RPI for Division I Women's Soccer', 2015. [Online]. Available: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-formula. [Accessed: 20- Oct- 2015]

• [9] Budget, 'Fiscal Year 2015-2016', 2015. [Online]. Available: http://budget.gmu.edu/fiscal-year-2015-2016/. [Accessed: 16- Nov- 2015].

47

Back-Up

Complexity of Coaching

Figure 2: Coaches role in sports team dynamics

Source : Concepts and Coaching Guidelines

49


10/21/2015

50

Soccer Analytics

• Not much is known about Soccer Analytics before 1996. – Influenced by the movement of

data analytics in other major sports

– The Sabermetrics model in MLB got the ball rolling in the 1970’s (Oakland Athletics first used) • Using big data to make decisions in

areas such as : pricing, management, player development, and training.

Source : Sports Analytic: Advancing Decision Making Through Technology and Data


Figure 5: Percentages of sports that employ data analysis

10/21/2015

51

• The emergence of modern sports data companies

– Opta – 1996 – began collecting data for English Premier league teams • Initial Data Collection approach

– Number of passes

– Number of tackles

– Distance travelled

– Prozone – 1998 – providing multiple performance analysis packages to major football clubs including:

» Manchester United (1999)

» Real Madrid (2006)

» US Major League Soccer (2008)

Soccer Analytics (cont’d)


52

Current Process

52

Challenges: • Patterns are inconsistent from game to game • A lot of pressure on coaches to make successful decisions • Limited time between games

What a Win is Worth 10/21/2015

53


y = 1E-06x + 0.3963 R² = 0.6994

0%

10%

20%

30%

40%

50%

60%

70%

80%

0 50,000 100,000 150,000 200,000 250,000

Co

fere

nce

Win

Pe

rce

nta

ge

Head Coaches Salary

Winning Percentage vs. Salary

54

Statistical Data (InStat) Individual Player Stats

Challenges, lost ball, recoveries 130

Participating Attacks 52

Set pieces 13

Passes 13

Action Maps 12

Action Graph (time x actions) 12

Successful Actions Graph (time x actions) 12

Average Position Player Maps 3

Pass Distribution Table 1

Pass Distribution Maps 1

Dangerous attack maps 10

Overall Player tables 104

Overall Player Maps 16

Overall Player Zones 97

54

Team

Total actions 25

Challenges 9

Indexes 4

Passes 11

Ball Possession 10

Attacks 14

Above data time in game (15min) 102

Zones

Actions Successful 14

Passes Accurate 14

Challenges won 14

% Challenges won 14

Total of 707 Statistics for each team evaluated

InStat Data

55

Input Data

56

Statement of Work

• Scope of work includes all planning, design, implementation, risk mitigation and validation of the expert coach system

• Responsible for allocating the proper resources to the proper phases in creating the system.

57

10/21/2015


Cost Analysis

58

$-

$20.00

$40.00

$60.00

$80.00

$100.00

$120.00

$140.00

0 5 10 15 20 25 30 35 40

Co

st (

Tho

usa

nd

s)

Time (weeks)

Planned Value Actual Cost Earned Value

Critical Path

59

60

Critical Path

61

Critical Path

62

Critical Path

design of an expert - george mason university · 6 design of an expert system coach for complex...

Documents