2010 simulated car racing championship @ wcci-2010

2010 Simulated Car Racing Championship @ WCCI-2010

The 2010 Simulated Car Racing Championship @ WCCI-2010Daniele Loiacono, Luigi Cardamone,

Martin V. Butz, and Pier Luca Lanzi


2010 Simulated Car Racing Championship9 races during 3 conferences

ACM GECCO-2010, Portland, OR (USA), July 7-IEEE WCCI-2010, Barcelona (Spain), July 18-23

IEEE CIG-2010, Copenhagen (Denmark), August 1821

Develop a driver for TORCS(hand-coded, learned, evolved, )

Drivers will be awarded based on their score in each conference competition

At the end, the team with highest overall scorewins the championship

2


What is the structure of a race?

Three stages: warm up, qualifiers, actual race

During warm-up, each driver can explore the track and learn something useful

During qualifiers, each driver races alone against the clock (the best 8 drivers move to the race)

During the race all the drivers race together

3


Motivations

Proposing a relevant game-based competition

more representative of commercial games AI

more similar to a real-world problem

Proposing a funny and exciting competition

you can see and play with the entries of this competition

human players can interact with AI

a lot of programmed AI available for comparison

Proposing a challenging competition

not designed with Machine Learning in mind

computationally expensive

real-time

dealing with a lot of practical issues


Whats new?

If everything seems under control, you're not going fast enough Mario Andretti

Warm-up stage

Before qualifying stage, competitors have 100000 game ticks to race on the track

Allows track learning and optimization of parameters

Noisy sensors

Track sensors and opponent sensors are affected by a Gaussian noise (standard deviation equal to 10% of the readings)

Extended sensor model

Focus sensors

Z position and speed

Direction of track sensors fully customizable

Added clutch control and focus command

The Open Racing Car Simulator


The Open Racing Car Simulator

TORCS is a state of the art open source simulator written in C++

Main features

Sophisticated dynamics

Provided with several cars, tracks, and controllers

Active community of users and developers

Easy to develop your own controller

OS Support

Linux: binaries and building from sources

Windows: binaries and limited bulding from sources support

OSX: legacy binaries and no building from sources support


The Open Racing Car Simulator & the Competition Software

TORCS

BOT BOT BOT

TORCS

PATCH

SBOT SBOT SBOT

BOT BOTBOT

UDP UDPUDP The competition server

Separates the bots from TORCS

Build a well-defined sensor model

Works in real-time


Sensors and actuators

Rangefinders for edges on the track and opponents

Speed, RPM, fuel, damage, angle with track, distance race, position on track, etc.

Six effectors: steering wheel [-1,+1], gas pedal [0, +1], brake pedal [0,+1], gearbox {-1,0,1,2,3,4,5,6}, clutch [0,+1], focus direction

Competitors


The competitors

Five entries in the second leg

AUTOPIA, Madrid and Granada

J. Muoz, Carlos III University of Madrid

S.Pohl, J. Quadflieg and T. Delbrgger, TU Dortmund

Joseph Alton, University of Birmingham

Timothy Alford (Xiaodong Li), RMIT University, Melbourne

Two more entries from the 2009 championship

COBOSTAR (T. Lnneker & M.V. Butz, University ofWrzburg)

POLIMI (Cardamone, Politecnico di Milano)


Industrial Computer Science Department.

Centro de Automtica y Robtica

Consejo Superior de Investigaciones Cientficas

Madrid, Spain

Contact:E. Onieva ([email protected])

AUTOPIA


Architecture Schema

Three basic modules for gear, steering and speed control

Steady state genetic algorithm to compute the best weights to combine parameters for steering and target speed control

Opponents module

Acts on steering and brake signal to overtake opponents and avoid collisions

Learning Module in Warm-up Stage

Factors over the target speed in certain track segments


Learning Module (Warm-Up)

Running normally in warm-up stage.

Maintain a vector with as many real values as tracklengthin meters.

Vector initialized to 1.0

If the vehicle goes out of the track or suffers damage then multiply vector positions from 250 meters before the current position by 0.95.

Vector is multiplied by F to make the driver more cautious in function of the damage:

F=1-0.02*round(damage/1000)


Susanna Pohl, Jan Quadflieg and Tim DelbrggerTU Dortmund

Mr Racer


Mr. Racer 2009-2010

Mr Racer 2009

Good classifier which identifies six situations

Acceleration/brake learned offline using an EA

Model of the track learned online

Simple heuristic to use the model: override the learned behaviour on straights and in full speed corners to drive flat out

Mr Racer 2010

Save the model after warmup, use it during qualifying and the race

Use the model to derive a plan consisting of target speeds and a racing line

Optimize the parameter set of the planning module, left to be done for the next round


Mr. Racer - Classification

Angle based measure mapped to six situations (straight, fast corner, slow corner, etc)


Mr. Racer 2010 The catch

Noise completely breaks our classifier

Without a descent classifier we cant learn the track

Without a trackmodel we cant drive

Workaround for GECCO-2010: Classify the whole track as being straight

New classifier for the next leg at WCCI-2010


Department of Computer ScienceCarlos III University of Madrid

Jorge Muoz


Jorge Muoz

Build a model of the track during the warm-up stage.

Two neural networks to predict the trajectory using the track model. Two neural networks to predict the target speed given the model of the track and the current car position

The four neural networks are trained with backpropagation using data retrieved from a human player.

The controller tries to imitate a human player.

A scripted policy is used to follow the trajectory (steering value), set the speed (accelerate and brake values), set the clutch and the gear


Jorge Muoz

Other optimizations performed during the warm-up and used in the race:

The car remember where it goes out of the car or drives far form the trajectory and in the next laps goes slower in those points

The car remember where it follows the trajectory perfectly and tries to go faster in the next laps.

Overtaking is made by means of modifying the predicted trajectory, the modification is bigger in straighs than in turns

To avoid being overtaken the car also modifies the trajectoy, trying to stay in front of the opponent car.


Joseph Alton

Joseph Alton


Steering

4 sensors at -30 and 30

Each sensors is mapped to the steering as:

Steering += left feeler * 0.005

Steering -= right feeler * 0.005

Noise filtering is done through having multiple sensors at the same position

The effect of repeating these calculations for each sensor means the steering grows exponentially when the feeler proportion changes, so the driver can turn sharply enough

No left steer

Full right steer No right steer

Full left steer


Warm-up

During warm-up we go around the track at 60 km/h (which is fast enough to complete all possible road tracks and obtain good readings)

During this time for each segment (meter) of the track we record the biggest turn and map this to a speed.

The speeds are as follows:

Type Value Speed ( km/h)

Sharp turn > Absolute 0.1 60

Turn > Absolute 0.05 100

Straight 200


Competitive phase look-ahead

During the competitive phase the circular array of target speeds for each segment is loaded.

For each game tick we look ahead 40 segments (metres) and pick the lowest speed as our Target Speed.

The effect is that the driver is always prepared for the worst case scenario and drives safely.

Adjusting to target speeds are handled by the Speed Control module. The module aims to keep the car within a range 5 km/h of a given target speed (which constantly changes).

If the car falls below this range acceleration is gradually applied until this range is met. Likewise if we are above this range the brake is gradually applied until we fall within this ideal speed range.


Timothy Alford (supervised by Xiaodong Li) RMIT University, Melbourne, Victoria


Overview

Y

N

Action object

Action object

on track ?

Recovery

Fuzzy system

Gears

Sensors

All components of the car are controlled by Fuzzy Logic (excluding gears and recovery )

Recovery, Gears are controlled with simple rules

GA is exploited in the Warm-up


Controlling the car

Input is collected from the sensors and fuzzified this is achieved by using membership functions.

Input from sensors (speed, angle, distance to track edge ...)

Membership function (calculate membersip[how true fuzzy variables are] for each set)

0

1

Fuzzy values (slow=0.2 normal=0.8 fast=0)


Using the membership values calculated previously, 'fired' rules are determined and output(s) can be inferred.

Controlling the car, continued

Membership values

Collection of rules

IF distance=close THEN speed=slowIF RPM=high THEN gear=up

IF angle=negative AND speed=fast THEN turn=hard_right

Fired rulesInferencing

process

Real world output (crisp)


Warming up

(1+1) ES (one parent, one child)

Parent uses already 'good' values

Generates children based on these values

Child becomes new parent if better, else rejected

Optimisations stored in XML file

Controller will optimise when warm-up stage is selected


COBOSTAR

Thies Lnneker and Martin V. Butz

University of Wrzburg

http://www.coboslab.psychologie.uni-wuerzburg.de


CIG-2008 Champ

Luigi Cardamone

Politecnico di Milano


References

Loiacono, D.; Lanzi, P. L.; Togelius, J.; Onieva, E.; Pelta, D. A.; Butz, M. V.; Lonneker, T. D.; Cardamone, L.; Perez, D.; Saez, Y.; Preuss, M.; Quadflieg, J.; , The 2009 Simulated Car Racing Championship, Computational Intelligence and AI in Games, IEEE Transactions on , vol.2, no.2, pp.131-147, June 2010

Enrique Onieva, David A. Pelta, Javier Alonso, Vicente Milans and Joshu Prez. A Modular Parametric Architecture for the TORCS Racing Engine. In Proc. of IEEE Symposium on Computational Intelligence and Games (CIG'09), pag. 256-262, 2009

Butz, M.V., & Lnneker, T. Optimized sensory-motor couplings plus strategy extensions for the TORCS car racing challenge. IEEE Symposium on Computational Intelligence in Games, IEEE CIG 2009, 317-324.

CIG-2009 (papers of session on racing games)

Qualifying


Scoring process: Warm-up Qualifying

Scoring process involves three road tracks:

Wild-Speed

Petit

Brondehach

All the tracks are not provided with the standard TORCS distribution:

Petit and Brondehach are designed by TORCS users

Wild-Speed has been provided by the organizers

Each controller raced for 100000 game ticks in the warm-up stage and then its performance is computed in the qualifying stage as the distance covered within 10000 game ticks


Qualifying: Wild-Speed

10417

8906.23

10483.9

10032.9

2693.95

7897.91

8824.99

0 2000 4000 6000 8000 10000 12000

Autopia

Cardamone

COBOSTAR

Jorge Muoz

Joseph Alton

MR. Racer

Timothy Alford


Qualifying: Petit

7733.64

7668.49

9541.39

8164.82

3986.65

5439.3

8572.59

0 2000 4000 6000 8000 10000 12000

Autopia

Cardamone

COBOSTAR

Jorge Muoz

Joseph Alton

MR. Racer

Timothy Alford


Qualifying: Brondehach

7010.28

798.197

3597.55

2623.54

9134

1645.72

787.932

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Autopia

Jorge Muoz

Joseph Alton

Cardamone

COBOSTAR

MR. Racer

Timothy Alford


Qualifyng summary

Competitor Wild-Speed Petit Brondehach Total

COBOSTAR 10 10 10 30

Autopia 8 5 8 21

Jorge Muoz 6 6 3 15

Cardamone 5 4 5 14

Timothy Alford 4 8 2 14

Mr. Racer 3 3 4 10

Joseph Alton 2 2 6 10


How much does noise affect the performance?


Petit with and without noise

7733.64

7668.49

9541.39

8164.82

3986.65

5439.3

8572.59

7734.76

7705.83

9197.55

9896.16

5467.67

9451.8

8736.34

0 2000 4000 6000 8000 10000 12000

Autopia

Cardamone

COBOSTAR

Jorge Muoz

Joseph Alton

MR. Racer

Timothy Alford

Without Noise With Noise

What about qualifying?

COBOSTAR is the fastest driver

On complex track noise seems to affect significantly the performance

However, some controllers are able to reach almost the same performance even with noise

The Race


The Three GPs

For each track we run 7 races with random starting grids

Each race is scored using the F1 point system (10 to first, 8 to second, 6 to third, )

Two points to the controller with lesser damage

Two points for the fastest lap of the race

44


Competitor Wild-Speed

Jorge Muoz 10

COBOSTAR 8

Autopia 10

Cardamone 5

Joseph 2

Tim Alford 4

Mr. Racer 4

Race: Wild-Speed


Competitor Wild-Speed Petit

Jorge Muoz 10 10

COBOSTAR 8 10

Autopia 10 8

Cardamone 5 5

Joseph 2 4

Tim Alford 4 3

Mr. Racer 4 3

Race: Petit


Competitor Wild-Speed Petit Brondehach

Jorge Muoz 10 10 8

COBOSTAR 8 10 10

Autopia 10 8 6

Cardamone 5 5 6

Joseph 2 4 5

Tim Alford 4 3 3

Mr. Racer 4 3 2

Race: Brondehach



Jorge Muoz 10 10 8 28

COBOSTAR 8 10 10 28

Autopia 10 8 6 24

Cardamone 5 5 6 16

Joseph 2 4 5 11

Tim Alford 4 3 3 10

Mr. Racer 4 3 2 9

Race: Overall


How much does noise affect the performance in races?



Jorge Muoz 8 (-2) 8 (-2) 5 (-3) 21 (-7)

COBOSTAR 10 (+2) 10 (=) 12 (+2) 32 (+4)

Autopia 12 (+2) 6 (-2) 8 (+2) 26 (+2)

Cardamone 2 (-3) 5 (=) 6 (=) 13 (-3)

Joseph 3 (+1) 2 (-2) 5 (=) 10 (-1)

Tim Alford 4 (=) 3 (=) 3 (=) 10 (=)

Mr. Racer 5 (+1) 5 (+2) 4 (+2) 14 (+5)

Race without noise

What about the race results?

Race and Qualifying have similar outcomes

COBOSTAR still very competitive

Track learning not so effective

Opponent manegement on complex track isless important and very difficult with noise


Championship Standings

Competitor GECCO WCCI Total

Autopia 34 24 58

Jorge Muoz 22.5 28 50.5

COBOSTAR 14 28 42

Cardamone 16 16 32

Joseph 15.5 11 26.5

Mr. Racer 16 9 25

Tim Alford - 10 10

2010 simulated car racing championship @ wcci-2010

Technology

ieee wcci

trackallows track learning

opponents speed

learning module warmup

distance race

actual race

driver races

warmup stagefactors