1 an experimental system for the collaborative control of unmanned air vehicles raja sengupta, cee...

1

An Experimental System for the Collaborative Control of Unmanned Air

Vehicles

Raja Sengupta, CEE Systems, UC Berkeley

Joint work with Karl Hedrick, ME UC Berkeley

Graduate Students- Tim McGee, Elaine Shaw, Xiao Xiao, Jack Tisdale, Dan Coatta, David Nguyen, Allison Ryan, Mark Godwin, Sivakumar Rathinam , Marco Zennaro, John Cason, and Dan Prull

Post Docs- Derek Caveney, Zu Kim, Stephen Spry

Engineers- Aram Soghikian, Susan Dickey, Dave Nelson

2

Collaboration Research Goals

Study distributed mechanisms for the collaboration of Unmanned Vehicles• Air to air communication used

Generalize a large number of missions under one framework• Surveillance/Mapping • Border Patrol• Search & Rescue• Convoy Protection, etc.

Create a system that:• Can accommodate a large number of agents (UVs)• Displays tolerance to communcation, hardware failure faults• Capable of running in real time

3

In Action

Mission Control

5

Commander View

6

Current UAV Platform Configuration

Wing-Mounted Camera allowing for vision-based control, surveillance, and obstacle avoidance

Ground-to-Air UHF Antenna for ground operator interface

GPS Antenna for navigation

802.11b Antenna for A-2-A comm.

Payload Tray for on-board computations and devices

Payload Switch Access Door for enabling / disabling on-board devices

7

Current Payload Configuration

Off-the-shelf PC-104 with custom Vibration Isolation

Orinoco 802.11b Card and Amplifier for A-2-A comm.

Analog Video Transmitter for surveillance purposes

Printed Circuit Board for Power and Signal Distribution among devices.

Umbilical Cord Mass Disconnect for single point attachment of electronics to aircraft.

Keyboard, Mouse, Monitor Mass Disconnect for access to PC-104 through trap door while on the ground.

8

MLB Bat IVAircraft

• Improved payload weight (25lbs) and volume • Improved logistics: 7.5 hour duration, onboard generator

9

Autopilots and SensorsPiccolo II autopilots

• 4Hz GPS updates (compared to 1 Hz in the old system)• Improved gyros, leading to much better attitude estimation• Analog I/O ports, allowing integration of user-specified sensor inputs into the core autopilot

structure• Satellite communication capability

Sensors• several new types: IR camera, radar, IMU, gimbals • will allow expansion of efforts in mapping, vision-based tracking, and control based on other

sensor types. • will allow testing and comparison of the effectiveness of various sensors for particular tasks • will allow exploration of how sensor types in a heterogeneous UAV team can be used together in a

complementary way.

10

Communications and Video Link

New air-to-air communications system• amplified 802.11b • testing of collaborative team control concepts using short-range air-air comm.• will retain long-range, low-bandwidth, air-ground links

New video downlink system• better monitoring of aircraft video streams• will allow ground-based testing of image processing algorithms and human-machine interface systems.

Ground

Station

11

Future Experimental System

DURIP funded multi-aircraft testbed

Six Primary Components• A. Five new aircraft with improved payload capacity and configuration• B. Upgraded autopilots with improved autopilot functions• C. New sensors-(bullet cameras, fisheye lenses, IR camera, radar, IMU, gimbals)• D. New air-to-air communications system• E. New video downlink system• F. Trailer/operations center

12

System Architecture

Task allocation/ Conflict Resolution

Mission to task decomposer

UAV UAV

Team Level

Mission Control

missions

UAV

13

BLCC- Berkeley Language for Collaborative Control

Define the mission and communicate it to team members

Define the “state” of each agent

Define the mission “state”

Allow for faults

Allow for conflict resolution

Define the information to be communicated between agents.

14

Current Collaborative Architecture

Task 1

Task 2

AllocatedTask 1

AllocatedTask 1

Tasking Conflict

Border Patrol

LocationVisit

ReallocatedTo Task 2

Conflict Resolution

15

Example Scenario

fault

subtasks start

obstacles UAVi

indicates comm.

16

Mission State

Each agent communicates primarily through a list of tasks that is shared between UVs.

A task is often described by a location, such as a GPS position.

1

2

3

4

17

Current Collaborative Architecture

Mission statements are decomposed into tasks and relayed from the ground to all aircraft.

Each plane without a task picks the closest available task for itself. Each plane allocates tasks only for itself.

Conflicts in task allocation are resolved using Euclidean distance.

Each aircraft broadcasts its current state and its knowledge of other vehicles’ states. It only has overwriting permissions for its own state.

Current objective• If the airplanes communicate sufficiently often each

task will eventually be done

More involved communication, tasking, and conflict resolution protocols are currently under development for future system integration

18

Simulation Results

19

Database

PiccoloPayload

Orbit Control Waypt Control Vision Control

Switchboard

Vision Process.

Orinoco Comm.

Camera

A-2-A

GroundCommands

AircraftAvionics

Vision ControlFor Turn-Rate Based Path

Following

Waypoint ControlFor Single-Point Visits

Orbit ControlFor Closed-Loop Multi-Point Paths

SwitchboardTask Allocation, Conflict Resolution,

and Controller Switching

OrinocoInter-vehicular communication

protocol

DatabasePermits inter-process

communication

Vision ProcessingFrame Grabbing Capabilities

PayloadResponsible for Relaying

Commands between the PC-104 and Mission Control

PiccoloResponsible for Relaying

Commands between the PC-104 and Aircraft Avionics

Aircraft Level Architecture

20

Generalization: Vision Based Following of Locally Linear Structures(Closed Loop on the California Aqueduct, June 2005)

21

Results – Canal Following

The road detection algorithm runs at 5 Hz (takes < 200 ms) or faster on the PC104 (700 MHz, Intel Pentium III).

No visible error was found from video sequences of over 100 frames containing the canal

22

Cal Road Detection on MLB Video(No Control)

Generic corridor detection by one-dimensional learning•Roads•Aqueducts•Perimeters•Pipelines•Power Lines

23

Vision Based Obstacle Avoidance System

24

Conclusion

Built an unmanned air vehicle system for experimental work on collaboration• Currently four airplanes• Five more planned

Current missions• Visit location and send picture• Border patrol

GPS based Vision based

Collaboration• Mission to task decomposition

Each mission should have its own semantics of decomposition

• Autonomous in-air task division and conflict resolution• Currently limited to static tasks

25

Geographic Data Management

26

Scalable Information Management:Target Map and Risk Map

Target distribution map• P(A, N, t); probability of N

targets of type t in area A

Target distribution update • Fuses measurements from

different kinds of sensors (SAR and EO)

• Bayesian update

Risk map computation• Integral of threat model with

respect to the measure P(A, N, t)• Generates the value function for

navigation Example: Target Map

Risk Map

UCB Rathinam 2003

27

Scalable Information Management:Distributing the Publisher Service

Geographic Data Management Network

Euclidean Space

Voronoi tessellation

Data objects

Publishing ServersSengupta AINS 2003

28


Metric Space

desired data

server

User

delivery

29

Movie of Implementation

4 laptops over wireless

One publisher per laptop

Start with one publisher

Three others come up

Some die

Data redistributes as publishers join and leave

Total data is this map

30






Some die


Total data made of manydata objects

31






Some die


Voronoi tessellation

32






Some die


33






Some die


34


Movie of ourImplementation 4 servers on 4 laptops over wireless

35

Data Consistency in the Publisher:Inconsistent copies are detected whp

Data Location

Wrong location copy 1

Wrong location copy 2

36

Data Consistency in the Publisher:Drift in a 2-D Markov Process

37

Geographic Data Management Network:Survivable Information for UAV Swarms

The server backbone dynamically tracks the client agent organization

Servers move in and out while the information survives

38

Tracking the Agent Organization:Dynamic GDMN backbone Control

Design a distributed control algorithm for the servers to partition the data and the clients to minimize the total bit-meters (Kumar etal.) of work done in the system and balance the load on the servers.

Let the load generated in each client be bi. If the locations of the points are denoted by pi and the location of the servers are denoted by cj, then the total cost is:

bi ( min dist(pi, cj) )

i j

The control algorithm updates server positions to reduce this cost

39

Simulation This example involves 100 clients and 6 servers

40

Control algorithm

In each sampling interval, each server • Measures the positions and the traffic generated by its clients

GDML routing protocols make the client set the Voronoi cell

• Calculates the weighted centroid of all the clients it serves• Moves towards its weighted centroid

Works well if the servers travel faster than the clients

The algorithm is based on the k-means algorithm (MacQueen ,1967 )

41

Publications: 2005 T.G. McGee, S. Spry, and J .K. Hedrick, "Optimal path planning in a constant wind with a

bounded turning rate," Accepted to 2005 AIAA Conference on Guidance, Navigation, and Control.

T.G. McGee, R. Sengupta, and J .K. Hedrick, "Obstacle detection for small autonomous aircraft using sky segmentation,” Proc. of the 2005 IEEE International Conference on Robotics & Automation.

T.G. McGee and J .K. Hedrick, "Guaranteed Strategies to Search for Mobile Intruders or Evaders in the Plane," submitted to AIAA J ournal of Guidance, Navigation, and Control.

T.G. McGee, G. Gonzalez, and J .K. Hedrick, "Temporal and Spatial Sliding Surface Control for Path Following," submitted to IEEE Transactions on Automatic Control.

Z. Kim, "Realtime Road Detection by Learning from One Example," Proc. of the IEEE Workshop on Application of Computer Vision, 2005, pp. 455-460.

M. Zennaro and R. Sengupta, "Distributing Synchronous Programs Using Bounded Queues," To appear in 5th ACM International Conference on Embedded Software (EMSOFT'05).

M. Zennaro and R. Sengupta, "Distributing Synchronous Programs Using Bounded Queues, a Coordinated Traffic Signal Application", University of California at Berkeley, Intelligent Transportation Studies, UCB-ITS-RR-2005-4, May 2005

A. Ryan, D. Nguyen, and J .K. Hedrick, "Hybrid Control for UAV-Assisted Search and Rescue," Submitted to the International Mechanical Engineering Congress and Exposition (IMECE), Orlando, Nov. 2005.

A. Ryan and J .K. Hedrick, "A Mode-Switching Path Planner for UAV-Assisted Search and Rescue," Submitted to the IEEE Conference on Decision and Control, Seville, Dec 2005.

42

Publications: 2005 cont’d M. Godwin, S. Spry, and J .K. Hedrick, “A Distributed System for Collaboration and

Control of UAV Groups: Experiments and Analysis,” Proc. of the 5th International Conference on Cooperative Control and Optimization, Gainesville, FL, J an. 2005.

S. Spry, A. Girard, and J .K. Hedrick, “Convoy Protection using Multiple Unmanned Aerial Vehicles: Organization and Coordination,” Proc. of the 2005 American Control Conference, Portland, OR, J une 2005.

D. Caveney and R. Sengupta, “Architecture and Application Abstractions for Multi-Agent Collaboration Projects,” Submitted to the IEEE Conference on Decision and Control, Seville, Spain, Dec. 2005.

D. Caveney, Y. Kang, and J .K. Hedrick, “Probabilistic Mapping For Unmanned Rotorcraft using Point-Mass Targets and Quadtree Structures,” Accepted to ASME IMECE, Orlando, FL, Nov. 2005.

D. Caveney and J .K. Hedrick, “Path Planning for Targets in Close Proximity with a Bounded Turn-Rate Aircraft,” Accepted to AIAA Guidance, Navigation, and Control Conference, San Francisco, CA, Aug. 2005.

E. Shaw, H. Chung, J .K. Hedrick, and S. Sastry, “Unmanned Helicopter Formation Flight Experiment for the Study of Mesh Stability,” submitted to the 5th International Conference on Cooperative Control and Optimization, 2005.

S. Rathinam, R. Sengupta, S. Darbha, “A Resource Allocation Algorithm for Multi-Vehicle Systems with Non-Holonomic Constraints,” Institute of Transportation Studies Research report, Berkeley UCB-ITS-RR-2005-2. Also submitted to the IEEE Transactions of Automation Science and Engineering.

43

Publications: 2005 cont’d S. Rathinam, Z. Kim, A. Soghaikain and R. Sengupta, “Vision Based Following of Locally

Linear Structures using an Unmanned Aerial Vehicle,” Submitted to the IEEE Conference on Decision and Control, 2005.

J . Tisdale, “A UAV Trajectory Planning Algorithm for Simultaneous Search and Track,” Accepted to ASME IMECE, Orlando, FL, Nov. 2005.

E. Frew and J . Langelan, “Receding Time Horizon Control Using Random Search for UAV Navigation with Passive, Non-cooperative Sensing,” To appear in AIAA Guidance, Navigation, and Control Conference, San Francisco, CA, August 2005.

A.R. Girard, J . Sousa and J .K. Hedrick, “A Selection of Recent Advances in Networked Multi-Vehicle Systems,” IMECE J ournal of Systems and Control Engineering, Part I, 219(1), 2005, pp. 1-14.

44

Publications: 2004 A.R. Girard, A.S. Howell and J .K. Hedrick, “Border Patrol and Surveillance Missions using

Multiple Unmanned Air Vehicles,” Submitted to IEEE Control Systems Technology, 2004 M. Zennaro and R. Sengupta, “Distributing Synchronous Systems with Modular

Structure,” Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

A. Ryan, M. Zennaro, A.S. Howell, R. Sengupta, J .K. Hedrick, “An Overview of Emerging Results in Cooperative UAV Control,” Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

Anouck R. Girard, Adam S. Howell and J . Karl Hedrick, “Border Patrol and Surveillance Missions using Multiple Unmanned Air Vehicles,” Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

J oão Sousa, Anouck R. Girard and J . Karl Hedrick, “Elemental Maneuvers and Coordination Structures for Unmanned Air Vehicles,” Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

S. Spry and J .K. Hedrick, “Formation Control Using Generalized Coordinates”, Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

S. Rathinam, M. Zennaro, T. Mak, R. Sengupta, “An Architecture for UAV Team Control”, IFAC Symposium on Autonomous Vehicles, Lisbon, Portugal, J uly 2004.

45

The End

46

Publications: 2004 cont’d S. Rathinam and R. Sengupta, “A Safe Flight Algorithm for Unmanned Aerial Vehicles”,

IEEE Aero Conference, March 6-9, Montana. S. Rathinam and R. Sengupta, “UAV Navigation in an Unknown Environment,” Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

Eric W. Frew and Raja Sengupta. “Obstacle Avoidance with Sensor Uncertainty for Small Unmanned Aircraft,” Proc. of the 43rd IEEE Conference on Decision and Control, Bahamas, Dec. 2004.

Frew et. al. “Vision-Based Road Following Using a Small Autonomous Aircraft.” In Proc. of the 2004 IEEE Aerospace Conference, Big Sky, MT, March 2004.

Frew et. al. “Stereo-Vision-Based Control of a Small Autonomous Aircraft Following a Road.” Second Annual Swarming Conference, Crystal City, MD, J une 2004.

E. Frew, X. Xiao, S. Spry, T. McGee, Z. Kim, J . Tisdale, R. Sengupta, and J .K. Hedrick, “Flight Demonstrations of Self-Directed Collaborative Navigation of Small Unmanned Aircraft,” Proc. of the 3rd AIAA Unmanned Unlimited Technical Conference, Workshop, & Exhibit, Chicago, IL, September 2004.

J . Sousa, A.R. Girard and J .E. Silva, “Elemental Maneuvers for Unmanned Air Vehicles”, Proceedings of Robotica 2004, pp. 63-70, Portugal.

47

Publications: 2003

S. Spry, A. Vaughn, X. Xiao, and J .K. Hedrick. “A Vehicle Following Methodology for UAV Formations”. Proceedings of the fourth International Conference on Cooperative Control and Optimization, Destin, FL., Nov. 2003.

S. Spry and J .K. Hedrick, “Coordinated Formation Control”, Poster, AINS Symposium 2003, Palo Alto, CA.

J . Lee, R. Huang, A. Vaughn, X. Xiao, M. Zennaro, J .K. Hedrick, R. Sengupta “Strategies of Path-Planning for a UAV to Track a Ground Vehicle”, Proceedings of the AINS Conference, 2003

48

Theses and Dissertations

D. Prull, “Flight Dynamics of Fixed-Wing Unmanned Aerial Vehicle Formations in Vortex Wakes,” Masters Thesis, University of California, Berkeley, May 2005

A. Ryan, “A Mode-Switching Path Planner for UAV-Assisted Search and Rescue,” Masters Thesis, University of California, Berkeley, May 2005

J . Tisdale, “A UAV Trajectory Planning Algorithm for Simultaneous Search and Track,” Masters Thesis, University of California, Berkeley, May 2005

A. Williams, “Search and Rescue Augmentation using Unmanned Aircraft,” Masters Thesis, University of California, Berkeley, May 2004.

A.C. Vaughn, “Path Planning and Control of Unmanned Aerial Vehicles in the Presence of Wind,” Master's Thesis, University of California, Berkeley, Dec. 2003.

49

The End

1 an experimental system for the collaborative control of unmanned air vehicles raja sengupta, cee...

Documents

action slide

mission control slide

shortrange airair

dave nelson slide

commander view slide

real time slide

onboard generator slide

new aircraft