Kazuya Kotani†1, Weihua Sun†1, Tomoya Kitani†2,

Naoki Shibata†3, Keiichi Yasumoto†1, Minoru Ito†1

†1 Nara Institute of Science and Technology (NAIST), Japan†2 Shizuoka University, Japan

†3 Shiga University, Japan

1

Inter-Vehicle Communication Protocol forCooperatively Capturing andSharing Intersection Video

(revised slides)

2

Goal: Allow drivers to grasp vehicle positionsin blind spots at intersections

Overview

Proposed Method• Vehicles shoot videos from onboard cameras• Each vehicle decides if it should send its video wirelessly

• according to available bandwidth, etc.• Each vehicle receives videos from multiple vehicles and

compose them into a Bird's Eye View Video (BEVV)• In order to reduce latency, It does not use multi-hop

communication

1. Background

2. Related Work

3. Proposed Method

4. Experimental Results

5. Conclusion

3

Outline

4

We propose a system that allows drivers to seevehicle positions in blind spots in an intuitive way

Intersection43.9 ％

Local Street26.8 ％

Vehicles in blind spots

[1] ITS: “Ministry of Land, Infrastructure, Transport and 　Tourism,”http://www.mlit.go.jp/road/ITS/.

Background

City area74.8 ％

43.9% of traffic accidents happen at◦ intersections in city area[1]

They are collisions between◦ vehicle and vehicle◦ vehicle and pedestrian

Most of these are caused by

1. Background

2. Related work

3. Proposed method

4. Experimental results

5. Conclusion

5

Outline

DSSS(Driving Safety Support Systems) project [2]◦ Gathers other vehicles’ position with inter-vehicle comm.◦ Vehicles without communication devices are not displayed◦ Positions are not accurate because of GPS positioning error

6

[3] Ota et al. : ``Visual Reconstruction of an Intersection by Integrating Cameras on Multiple Vehicles,’’ Proc. on Machine Vision Applications (MVA2007), pp.335–338(2007).

[2] Honda Motor Co., Ltd.: “Honda Technology,” http://www.honda.co.jp/news/2009/4090219.html?from=rss.

Method to compose a birds-eye-view video(BEVV)[3]

◦ Multiple videos are taken from multiple cars from different angles◦ Videos can be composed into one BEVV◦ The method includes no mechanism for exchanging videos

Related Work

7

IEEE802.11b, 200 Kbps

1 2 3 4 5 6 7 8 9 100

102030405060708090

100

Number of Vehicles

Pa

ck

et

Arr

ive

d R

ati

o

[%]

Problems with this method• Vehicle density can be too high• Wireless bandwidth is not insufficient

We need to carefully select vehicles that send videos to save wireless bandwidth

Problem with a straightforward approach forexchanging captured video

1. Vehicles broadcast request messages before turning right at intersection2. All vehicles in intersection capture and send videos wirelessly3. Vehicles receive videos and compose BEVV

The straightforward method

Objectvie◦Drivers need high quality view of the blind spots

Approach◦Assign priority to each vehicle for sending video◦The following vehicles are assigned high priorities

Vehicles that can capture the requested areas Vehicles that can capture high quality videos

◦Vehicles send out the captured video according to their priorities

8

Key Idea for Selecting Vehicles

07

56

0

4

86

5

1. Background

2. Related work

3. Proposed method

4. Experimental results

5. Conclusion

9

Outline

We assume that it is possible to compose a BEVV from videos taken from any different angles

We conducted preliminary experiment with scaled down model of intersection

convert

10

compose

Assumptions

Captured on camera

Converted video frame

Multiple video frames

Bird’s eye view frame

An intersection is divided into rectangular cells◦ Each cell is defined by its geographical position◦ Each vehicle knows which cell it is located◦ Each vehicle know cells that can be captured by its camera

Finer position can be determined by visually analyzing captured video[4]

11

Requesting vehicle

[4] Alberto et al. : `` Multi-Resolution Vehicle Detection using Artificial Vision,’’ Proc. on IEEE INTELLIGENT VEHICLES SYMPOSIUM (IV’04), pp.310–314(2004).

Dividing Intersection into Cells

Request cells

Vehicles have the following equipment◦ On-board camera◦ Car navigation system with GPS◦ Wireless LAN communication device◦ On-board computer

On-board computer can sense the state of indicator (going straight, turning left or right)

12

Assumptions on Vehicle

13

1. All vehicles periodically exchange Hello Message

2. Requesting vehicle broadcasts Request Message

3. Each vehicle which received Request Message calculates all vehicles’ Priority

86

7

6

0

00

70

0

7

4. Each vehicle decides if it should send video frames

Requesting vehicle

How Proposed Method Works

There are two phases in the proposed method

Information Exchange Phase◦ Vehicles exchange messages to get the information from

the surrounding vehicles

Vehicle Selection Phase◦ Videos are sent utilizing available bandwidth as much as

possible◦ Priority is calculated in this phase

14

Two phases in the proposed method

Goal : Each vehicle knows other vehicles’ information

Vehicles close to an intersection exchange the following messages

Hello Message◦ All vehicles inform their own status to other vehicles

position, speed, 　 capturing cells, video quality, number of video frames already sent

Request Message◦ Requesting vehicle sends requested cells to other vehicles

items of Hello Message, requested cells◦ When a vehicle receives a Request Message, it transits to Vehicle

Selection Phase

15

Information Exchange Phase

Goal : Making vehicles send videos utilizing available bandwidth as much as possible

Vehicles need to know the usage of communication bandwidth◦ The usage of communication bandwidth is defined by the ratio

of

To decide if each vehicle should send videos◦ Calculate the priority based on the following items

Requested cells that can be captured by the camera on vehicle Video quality for each cell

◦ Priority is the weighted sum of these values

Preliminary experiment We conducted preliminary experiment to determine these weights 16

Vehicle Selection Phase

Number of actually received video framesNumber of video frames informed in Hello messages

Deciding if each vehicle sends video Each vehicle calculates priorities of vehicles

independently◦ Usually, the results of calculations are almost same

For each direction, the vehicle with the highest priority decides to send the video

◦ For each direction, usually the car closest to the center of intersection starts sending video

◦ Then, the cars following the first car sends their videos in turn

◦ The priority is dynamically changed according to the positions of moving vehicle

17

Vehicle Selection Phase(contd.)

1. Background

2. Related work

3. Proposed method

4. Experimental results

5. Conclusion

18

Outline

Goal◦ To see if the proposed method can select vehicles that capture

requested cells in high quality

Evaluation criteria1. The number of video frames covering each cell in requested cells2. The number of video frames covering requested cells from each

driving direction3. The priority of video frame received by request vehicle

19

Experimental Evaluation

We used QualNet[5]

◦ Created terrain data to imitate actual intersection in Kyoto, Japan

◦ Requesting vehicle goes into intersection ⇒ turns to the right

◦ The number of requesting vehicle is one

Transmitting bit rate: 200 [Kbps]

20

Settings

Simulation parameters

Packet parameters

21

item value

The number of vehicles 60

Requested cells 3 cells

Wireless LAN standard IEEE802.11b

Vehicle density Dense, Middle, Sparse

Carrying rate of equipments 100%, 60%, 30%

Packet name Packet sizeTransmission

interval

Hello Message 300[byte] 0.5[s]

Request Message 300[byte] 0.5[s]

Video frame 1666[byte] 0.067[s]

Parameter setting of experiment

We compared the following methods for selecting vehicles to send their videos i. Proposed method

ii. All vehicles

All vehicles near intersection send video

iii. Vehicles closest to the center of intersection

The vehicle closest to the center of intersection from each

driving direction sends video

Measured evaluation criteria with these three methods

22

Comparison of Three Methods

23

Request vehicle

Request cells

green1

green2

red2

red1

1 2

3

Evaluation experimentDriving directions and request cells

In order to see vehicles in blind spot by video, video quality of at least 10[fps] in average is necessary

RNF is the minimum number of received frames that satisfies the above condition for each cell◦ Simulation time and RNF value in each density setting are

below

24

Required Number of Frames (RNF)

Vehicle density Dense Middle Sparse

Simulation time 35[s] 58[s] 88[s]

RNF(each direction) 350[frame] 580[frame] 880[frame]

RNF(each cell) 1400[frame]

2400[frame]

3520[frame]

×4 (driving directions)

In Dense setting with 100% carrying rate◦ Proposed method achieves RNF on all request cells◦ All vehicles doesn’t achieves RNF on any cells◦ Vehicles near the center achieves RNF on two cells

25

Number of video frames (Request cells) 1/3

call1 cell2 cell30

500

1000

1500

2000

2500

3000

Proposed methodAll vehiclesVehicles near the centerRNF

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

es

In Dense Middle and Sparse setting with 100% carrying rate◦ Proposed method sends more frames than any other method

regardless of vehicle density

26

Number of video frames (Request cells) 2/3

call1 cell2 cell30

500

1000

1500

2000

2500

3000

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

es

cell1 cell2 cell30

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

e

cell1 cell2 cell30

1000

2000

3000

4000

5000

6000

7000

8000

Proposed method

All vehicles

Vehicles near the cen-ter

RNF

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

e

Dense Middle Sparse

In Dense setting, with 100, 60, 30% carrying ratio◦ Proposed method provides more frames than other methods ◦ Request cells achieving RNF decreases as carrying rate decreases

27

Number of video frames (Request cells) 3/3

call1 cell2 cell30

500

1000

1500

2000

2500

3000

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

es

call1 cell2 cell30

500

1000

1500

2000

2500

3000

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

e

call1 cell2 cell30

500

1000

1500

2000

2500

3000

Proposed method

All vehicles

Vehicles near the cen-ter

RNF

Request cells

Nu

mb

er

of

Re

ce

ive

d F

ram

e

100% 60% 30%

In Dense setting with 100% carrying rate◦ Proposed method achieves RNF from all directions◦ Other methods achieve RNF from one or two directions

28

Number of video frames (Driving direction)

green1 green2 red1 red20

200

400

600

800

1000

1200

1400

1600

1800

Proposed methodAll vehiclesVehicles near the centerRNF

Driving Direction

Nu

mb

er

of

Re

cie

ve

d F

ram

e

In Dense setting with 100% carrying rate◦ There are more high priority frames by proposed method than other

methods

29

24~30 18~24 12~18 6~12 0~60

10

20

30

40

50

60

70

80

90

100

Proposed method 100%All vehicles 100%Vehicles near the center 100%

Priority

Cu

mm

ula

tive

Den

sity

of

Frm

ae[%

]

Effective in Dense setting with 60% or more carrying rate More efficient than other methods in all environments

Conclusions of experiment results about proposed method

PriorityCumulative distribution function (CDF) graph

Proposed method can make BEVV which contains request cells in high quality

Vehicle selection method◦ For creating a bird's eye view video by letting selected

vehicles efficiently capture and exchange video◦ It assigns priorities to send videos to other vehicles◦ Vehicle with higher priority sends videos within the

available communication bandwidth Results

◦ Proposed method is especially effective in the dense setting with 60% or more arrival rate

30

Conclusion

Kotani, K., Sun, W., Kitani, T., Shibata, N., Yasumoto, K., Ito, M.:Inter-Vehicle Communication Protocol for Cooperatively Capturing and Sharing Intersection Video, Proc. of 2nd IEEE Intelligent Vehicular Communications System Workshop (IVCS'10), (CD-ROM), Jan. 9th, 2010.

DOI:10.1109/CCNC.2010.5421635 [ PDF ]

31