nghi-

tht

ec-fsxi-

o-

av-

m-.fab-nh

inm,ofg-tllyi-mtoll,

ce,rm

dnaledraertri-

Autonomous Guidance and Control for an Underwater Robotic Vehicle

David Wettergreen, Chris Gaskett, and Alex Zelinsky

Robotic Systems Laboratory Department of Systems Engineering, RSISE

Australian National University Canberra, ACT 0200 Australia

[dsw | cg | alex]@syseng.anu.edu.au

AbstractUnderwater robots require adequate guidanceand control to perform useful tasks. Visualinformation is important to these tasks andvisual servo control is one method by whichguidance can be obtained. To coordinate andcontrol thrusters, complex models and controlschemes can be replaced by a connectionistlearning approach. Reinforcement learning usesa reward signal and much interaction with theenvironment to form a policy of correct behav-ior. By combining vision-based guidance with aneurocontroller trained by reinforcement learn-ing our aim is to enable an underwater robot tohold station on a reef or swim along a pipe.

1 IntroductionAt the Australian National University we are develop-ing technologies for underwater exploration and obser-vation. Our objectives are to enable underwater robotsto autonomously search in regular patterns, followalong fixed natural and artificial features, and swimafter dynamic targets. These capabilities are essentialto tasks like exploring geologic features, catalogingreefs, and studying marine creatures, as well asinspecting pipes and cables, and assisting divers. Forunderwater tasks, robots offer advantages in safety,accuracy, and robustness.

We have designed a guidance and control architec-ture to enable an underwater robot to perform usefultasks. The architecture links sensing, particularlyvisual, to action for fast, smooth control. It alsoallows operators or high-level planners to guide therobot’s behavior. The architecture is designed toallow autonomy of at various levels: at the signallevel for thruster control, at the tactical level for com-petent performance of primitive behaviors and at thestrategic level for complete mission autonomy.

We use visual information, not to build maps tonavigate, but to guide the robot’s motion using visualservo control. We have implemented techniques forarea-based correlation to track features from frame toframe and to estimate range by matching between ste-reo pairs. A mobile robot can track features and usetheir motion to guide itself. Simple behaviors regu-late position and velocity relative to tracked features.

Approaches to motion control for underwater vehi-cles, range from traditional control to modern control[1][2] to a variety of neural network-based architec-tures [3]. Most existing systems control limiteddegrees-of-freedom and ignore coupling betweenmotions. They use dynamic models of the vehicle andmake simplifying assumptions that can limit the oper-ating regime and/or robustness. The modeling processis expensive, sensitive, and unsatisfactory.

We have sought an alternative. We are developia method by which an autonomous underwater vecle (AUV) learns to control its behavior directly fromexperience of its actions in the world. We start wino explicit model of the vehicle or of the effect thaany action may produce. Our approach is a conntionist (artificial neural network) implementation omodel-free reinforcement learning. The AUV learnin response to a reward signal, attempting to mamize its total reward over time.

By combining vision-based guidance with a neurcontroller trained by reinforcement learning our aimis to enable an underwater robot to hold station onreef, swim along a pipe, and eventually follow a moing object.1.1 Kambara Underwater VehicleWe are developing a underwater robot named Kabara, an Australian Aboriginal word for crocodileKambara's mechanical structure was designed and ricated by the University of Sydney. At the AustraliaNational University we are equipping Kambara witpower, electronics, computing and sensing.

Kambara's mechanical structure, shown Figure 1, has length, width, and height of 1.2m, 1.5and 0.9m, respectively and displaced volume approximately 110 liters. The open-frame design riidly supports five thrusters and two watertighenclosures. Kambara’s thrusters are commerciaavailable electric trolling motors that have been modfied with ducts to improve thrust and have custopower amplifiers designed to provide high current the brushed DC motors. The five thrusters enable ropitch, yaw, heave, and surge maneuvers. HenKambara is underactuated and not able to perfodirect sway (lateral) motion; it is non-holonomic.

A real-time computing system including main ansecondary processors, video digitizers, analog sigdigitizers, and communication component is mountin the upper enclosures. A pan-tilt-zoom camelooks out through the front endcap. Also in the uppenclosure are proprioceptive sensors including a

Figure 1: Kambara

-etorri-

forori-

tord-r

anbynom-le

atethelyft

forinrdn-r

arectt of

mor-hi-ngr-

r-ra-toalle.ote-t,r-

oa-eer-

asis ans-gan]

axial accelerometer, triaxial gyro, magnetic headingcompass, and inclinometers. All of these sensors arewired via analog-to-digital converter to the mainprocessor.

The lower enclosure, connected to the upper by aflexible coupling, contains batteries as well as powerdistribution and charging circuitry. The batteries aresealed lead-acid with a total capacity of 1200W. Alsomounted below are depth and leakage sensors.

In addition to the pan-tilt-zoom camera mounted inthe upper enclosure, two cameras are mounted inindependent sealed enclosures attached to the frame.Images from these cameras are digitized for process-ing by the vision-based guidance processes.

2 Architecture for Vehicle GuidanceKambara’s software architecture is designed to allowautonomy at various levels: at the signal level for adap-tive thruster control, at the tactical level for competentperformance of primitive behaviors, and at the strate-gic level for complete mission autonomy.

The software modules are designed as indepen-dent computational processes that communicate overan anonymous broadcast protocol, organized asshown in Figure 2. The Vehicle Manager is the soledownstream communication module, directing com-mands to modules running on-board. The FeatureTracker is comprised of a feature motion tracker anda feature range estimator as described in section 3. Ituses visual sensing to follow targets in the environ-ment and uses their relative motion to guide theVehicle Neurocontroller. The Vehicle Neurocontrol-ler, described in 4, learns an appropriate valuation ofstates and possible actions so that it can produce con-trol signals for the thrusters to move the vehicle to itsgoal. The Thruster Controller runs closed-loop servocontrol over the commanded thruster forces. ThePeripheral Controller drives all other devices on thevehicle, for example cameras or scientific instru-ments. The Sensor Sampler collects sensorinformation and updates the controllers and the StateEstimator. The State Estimator filters sensor informa-tion to generate estimates of vehicle position,orientation and velocities. The Telemetry Routermoves vehicle state and acquired image and sciencedata off-board.

The Visualization Interface will transform telemetry into a description of vehicle state that can brendered as a three-dimensional view. The OperaInterface interprets telemetry and presents a numecal expression of vehicle state. It provides method generating commands to the Vehicle Interface fdirect teleoperation of vehicle motion and for supervsory control of the on-board modules.

The Swim Planner interprets vehicle telemetry analyze performance and adjust behavior accoingly, for example adjusting velocity profiles to bettetrack a pattern. A Terrain Mapper would transformdata (like visual and range images) into maps that cbe rendered by the Visualization Interface or used the Swim Planner to modify behavior. The MissioPlanner sequences course changes to produce cplex trajectories to autonomously navigate the vehicto goal locations and carry out complete missions.2.1 Operational ModesThe software architecture is designed to accommoda spectrum of operational modes. Teleoperation of vehicle with commands fed from the operator directto the controllers provides the most explicit control ovehicle action. While invaluable during developmenand some operations, this mode is not practical long-duration operations. Supervised autonomy, which complex commands are sequenced off-boaand then interpreted over time by the modules oboard, will be our nominal operating mode. Undesupervised autonomy, the operator’s commands infrequent and provide guidance rather than direaction commands. The operator gives the equivalen“swim to that feature” and “remain on station”. In fullyautonomous operation, the operator is removed frothe primary control cycle and planners use state infmation to generate infrequent commands for the vecle. The planners may guide the vehicle over a lotraverse, moving from one target to another, or thooughly exploring a site with no human intervention

3 Vision-based Guidance of an Underwater Vehicle

Many tasks for which an AUV would be useful owhere autonomous capability would improve effectiveness, are currently teleoperated by human opetors. These operators rely on visual information perform tasks making a strong argument that visuimagery could be used to guide an underwater vehic

Detailed models of the environment are often nrequired. There are some situations in which a thredimensional environment model might be useful bufor many tasks, fast visual tracking of features or tagets is necessary and sufficient.

Visual servoing is the use of visual imagery tcontrol the pose of the robot relative to (a set of) fetures.[4] It applies fast feature tracking to providclosed-loop position control of the robot. We arapplying visual servoing to the control of an undewater robot. 3.1 Area-based Correlation for Feature TrackingThe feature tracking technique that we use as the bfor visual servoing applies area-based correlation toimage transformed by a sign of the difference of Gausians (SDOG) operation. A similar feature trackintechnique was used in the visual-servo control of autonomous land vehicle to track natural features.[5

VehicleNeurocontroller

On-board controlOff-board telemetryOff-board guidance

SensorSampler

ThrusterController

ImageArchive

Telemetry

PeripheralController

Archive

VehicleManager Feature

Tracker

VisualizationInterface

MissionPlanner

OperatorInterface

SwimPlanner

TerrainMapper

TelemetryRouter

Figure 2: Architecture for vehicle guidance and control

StateEstimator

re-heeenninere.a-iro-m-

sti-ingga-s a tom

fedrngerds

eringer-andd

Input images are subsampled and processed usinga difference of Gaussian (DOG) operator. This opera-tor offers many of the same stability properties of theLaplacian operator, but is faster to compute.[6] Theblurred sub-images are then subtracted and binarizedbased on sign information. This binary image is thencorrelated with an SDOG feature template matching asmall window of a template image either from a pre-vious frame or from the paired stereo frame. A logicalexclusive OR (XOR) operation is used to correlatethe feature template with the transformed sub-image;matching pixels give a value of zero, while non-matching pixels will give a value of one. A lookuptable is then used to compute the Hamming distance(the number of pixels which differ), the minimum ofwhich indicates the best match. 3.2 Tracking Underwater FeaturesWe are verifying our feature tracking method withactual underwater imagery. Figure 3 shows trackingthree features through 250 images of a support pile.

The orientation and distance to the pile changesthrough this 17 second sequence. Some features arelost and then reacquired while the scene undergoesnoticeable change in appearance. The changing posi-tion of the features provides precisely the data neededto inform the Vehicle Neurocontroller of Kambara’sposition relative to the target.3.3 Vehicle Guidance from Tracked FeaturesGuidance of an AUV using our feature trackingmethod requires two correlation operations within theFeature Tracker, as seen in Figure 4. The first, the fea-

ture motion tracker, follows each feature between pvious and current images from one camera while tother, the feature range estimator, correlates betwleft and right camera images. The feature motiotracker correlates stored feature templates to determthe image location and thus direction to each featuRange to a feature is determined by correlating fetures in both left and right stereo images to find thepixel disparity. This disparity is then related to an abslute range using camera intrinsic and extrinsic paraeters which are determined by calibration.

The appearance of the features can change dracally as the vehicle moves so managing and updatfeature templates is crucial part in reliably trackinfeatures. We found empirically that updating the feture template at the rate at which the vehicle movedistance equal to the size of the feature is sufficienthandle appearance change without suffering froexcessive accumulated correlation error.[5]

The direction and distance to each feature are the Vehicle Neurocontroller, The neurocontrollerequires vehicle state, from the State Estimator, alowith feature positions to determine a set of thrustcommands. To guide the AUV, thruster commanbecome a function of the position of visual features.

4 Learned Control of an Underwater Vehicle Many approaches to motion control for underwatvehicles have been proposed, and although worksystems exist, there is still a need to improve their pformance and to adapt them to new vehicles, tasks, environments. Most existing systems control limite

Figure 3: Every tenth frame (top left across to bottom right) in a sequence of 250 images of an underwater support pile recorded at 15Hz. Boxes indicate three features tracked from the first frame through the sequence.

rsin-f-ea-geet-derale

roal-

csg aen

of

disfs.

r-e

ar-hardivehee,

rd

r-r

:

esg-

ee-

ndndate.ingndh iso- 5

act.

degrees-of-freedom, for example yaw and surge, andassume motion along some dimensions can be con-trolled independently. These controllers usuallyrequire a dynamic model and simplifying assumptionsthat may limit operating regime and robustness.

Traditional methods of control for vehicle systemsproceed from dynamic modelling to the design of afeedback control law that compensates for deviationfrom the desired motion. This is predicated on theassumption that the system is well-modelled and thatspecific desired motions can be determined.

Small, slow-moving underwater vehicles present aparticularly challenging control problem. The dynam-ics of such vehicles are nonlinear because of inertial,buoyancy and hydrodynamic effects. Linear approxi-mations are insufficient, nonlinear control techniquesare needed to obtain high performance.[7]

Nonlinear models of underwater vehicles havecoefficients that must be identified and some remainunknown because they are unobservable or becausethey vary with un-modelled conditions. To date, mostcontrollers are developed off-line and only with con-siderable effort and expense are applied to a specificvehicle with restrictions on its operating regime.[8]4.1 Neurocontrol of Underwater VehiclesControl using artificial neural networks, neurocontrol,[9] offers a promising method of designing a nonlinearcontroller with less reliance on developing accuratedynamic models. Controllers implemented as neuralnetworks can be more flexible and are suitable for deal-ing with multi-variable problems.

A model of system dynamics is not required. Anappropriate controller is developed slowly throughlearning. Control of low-level actuators as well ashigh-level navigation can potentially be incorporatedin one neurocontroller.

Several different neural network based controllefor AUVs have been proposed. [10] Sanner and Ak[11] developed a pitch controller trained by backpropagation. Training of the controller was done ofline in with a fixed system model. Output error at thsingle output node was estimated by a critic eqution. Ishii, Fujii and Ura [12] developed a headincontroller based on indirect inverse modelling. Thmodel was implemented as a recursive neural nwork which was trained offline using data acquireby experimentation with the vehicle and then furthtraining occurred on-line. Yuh [10] proposed severneural network based AUV controllers. Error at thoutput of the controller is also based on a critic.4.2 Reinforcement Learning for ControlIn creating a control system for an AUV, our aim is fothe vehicle to be able to achieve and maintain a gstate, for example station keeping or trajectory following, regardless of the complexities of its own dynamior the disturbances it experiences. We are developinmethod for model-free reinforcement learning. Thlack of an explicit a priori model reduces reliance oknowledge of the system to be controlled.

Reinforcement learning addresses the problemforming a policy of correct behavior through observeinteraction with the environment. [13] The strategy to continuously refine an estimate of the utility operforming specific actions while in specific stateThe value of an action is the reward received for carying out that action, plus a discounted sum of threwards which are expected if optimal actions are cried out the future. The reward follows, often witsome delay, an action or sequence of actions. Rewcould be based on distance from a target, roll relatto vertical or any other measure of performance. Tcontroller learns to choose actions which, over timwill give the greatest total reward.

Q-learning [14] is an implementation method foreinforcement learning in which a mapping is learnefrom a state-action pair to its value (Q). The mappingeventually represents the utility of performing an paticular action from that state. The neurocontrolleexecutes the action which has the highest Q value inthe current state. The Q value is updated according to

where Q is the expected value of performing action uin state x; R is the reward; α is a learning rate and γ isthe discount factor. Initially Q(x,u) is strongly influ-enced by the immediate reward but, over time, it comto reflect the potential for future reward and the lonterm utility of the action.

Q-learning is normally considered in a discretsense. High-performance control cannot be adquately carried out with coarsely coded inputs aoutputs. Motor commands need to vary smoothly aaccurately in response to continuous changes in stWhen states and actions are continuous, the learnsystem must generalize between similar states aactions. To generalize between states, one approacto use a neural network.[15] An interpolator can prvide generalization between actions.[16] Figureshows the general structure of such a system.

A problem with applying Q-learning to AUV con-trol is that a single suboptimal thruster action in long sequence does not have noticeable effeAdvantage learning [17] is a variation of Q-learning

Feature

Estimator

Directionto Feature

Rangeto Feature

UpdateFeature

VehicleMotion

Figure 4: Diagram of the AUV visual servoing system

Left

RangeFeature

TrackerMotion

ThrusterController

StateEstimator

VehicleNeurocontroller

ImageRightImage

Position & Velocity

Thruster Forces

Templates

Q x u,( ) 1 α–( )Q x u,( ) α R γmaxuQt 1+ x u,( )+[+=

eesll-

esr

tohes-

ctt,n.gill

ern-

ar-l

e to.ntt-ai-

nden

ng-n-nt,terol- by

to.

which addresses this by emphasizing the difference invalue between actions and assigning more reward tocorrect actions whose individual effect is small.

Kambara’s neurocontroller [18] is based on advan-tage learning coupled with an interpolation method[16] for producing continuous output signals.4.3 Evolving a NeurocontrollerWe have created a simulated non-holonomic, twodegree-of-freedom AUV with thrusters on its left andright sides, shown in Figure 6. The simulation includeslinear and angular momentum, and frictional effects.Virtual sensors give the location of targets in bodycoordinates as well as linear and angular velocity.

The simulated AUV is given a goal at 1 units ofdistance away in a random direction. For 200 timesteps the controller receives reward based upon itsability to move to and then maintain position at thegoal. A purely random controller achieves an averagedistance of 1.0. A hand-coded controller, which pro-duces apparently good behavior by moving to thetarget and stopping, achieves 0.25 in average dis-tance to the goal over the training period.

Every 200 time steps, a new goal is randomly gen-erated until the controller has experienced 40 goals. Agraph showing the performance of 140 neurocontrol-lers, trained with advantage learning is shown in thebox-and-whisker plot of Figure 7. All controllers(100%) learn to reach each goal although some dis-play occasionally erratic behavior, as seen by theoutlying “+” marks. Half of the controllers perform

within the box regions, and all except outliers liwithin the whiskers. This learning method convergto good performance quickly and with few and smamagnitude spurious actions.

The next experiments are to add additional degreof freedom to the simulation so that the controllemust learn to dive and maintain roll and pitch, and repeat the procedure in the water, on-line, with treal Kambara. Experiments in linking the vision sytem to the controller can then commence.

A significant challenge lies in the nature and effeof live sensor information. We anticipate bias, drifand non-white noise in our vehicle state estimatioHow this will effect learning we can guess by addinnoise to our virtual sensors but real experiments wbe most revealing.

5 Commanding Thruster ActionThe task of Vehicle Neurocontroller is simplified if itscommanded output is the desired thrust force raththan motor voltage and current values. The neurocotroller need not learn to compensate for the non-lineities of the thruster, its motor and amplifier. Individuathruster controllers use force as a desired referenccontrol average motor voltage and current internally

Considerable effort has been applied in receyears to developing models of underwater thrusers.[19][20][21] This is because thrusters are dominant source of nonlinearity in underwater vehcle motion.[19] Every thruster is different either indesign or, among similar types, due to tolerances awear, so parameter identification must be undertakfor each one.

We have measured motor parameters includifriction coefficients and motor inertia and begun intank tests measure propeller efficiency and relatioships between average input voltage and curremotor torque, and output thrust force. Using a thrusmodel [21] and these parameters, the neurocontrlers force commands can be accurately producedthe thrusters.

6 Estimating Vehicle StateIn order to guide and control Kambara we need know where it was. where it is, and how it is moving

Figure 5: A Q-learning system with continuous states and actions as implemented in the neurocontroller.

Q

RNeural

Network

u

x

Inter-

polator

u0,q

0u

1,q

1

k,q

k

u ,qn n

u

Figure 6: Kambara simulator while learning to control motion and navigate from position to position. The path between goals becomes increasingly direct.

5 10 15 20 25 30 35 400

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Ave

rage

Dis

tanc

e T

o T

arge

t

Target Number

Figure 7: Performance of 140 neurocontrollers trained using advantage learning. Box and whisker plots with median line when attempting reach and maintain 40 target positions each for 200 time steps.

ofa-le

n-o-L

r-l.

e3,

al

6,

od

sr,s-

”,

-p.

-

r-

:,

-n-

-

dal

tern,

ero-ic

dr-0,

d-

This is necessary for long-term guidance of the vehicleas it navigates between goals and for short-term con-trol of thruster actions. Continuous state information isessential to the reinforcement learning method thatKambara uses to learn to control its actions.

Kambara carries a rate gyro to measure its threeangular velocities and a triaxial accelerometer to mea-sure its three linear accelerations. A pressure depthsensor provides absolute vertical position, an incli-nometer pair provide roll and pitch angles and amagnetic heading compass measures yaw angle in afixed inertial frame. Motor voltages and currents arealso relevant state information. The Feature Trackercould also provide relative position, orientation andvelocity of observable features.

These sensor signals, as well as input control sig-nals, are processed by a Kalman filter in the StateEstimator to estimate Kambara’s current state. Fromten sensed values (linear accelerations, angular veloc-ities, roll, pitch, yaw and depth) the filter estimatesand innovates twelve values: position, orientation andlinear and angular velocities.

The Kalman filter requires models of both the sen-sors and the vehicle dynamics to produce its estimate.Absolute sensors are straightforward, producing aprecise measure plus white Gaussian noise. The gyromodels are more complex to account for bias anddrift. A vehicle dynamic model, as described previ-ously, is complex, non-linear, and inaccurate. All ofour models are linear approximations.

There is an apparent contradiction in applyingmodel-free learning to develop a vehicle neurocon-troller and then estimating state with a dynamicmodel. Similarly, individual thruster controllersmight be redundant with the vehicle neurocontroller.We have not fully reconciled this but believe that aspractical matter partitioning sensor filtering and inte-gration, and thruster control from vehicle control willfacilitate learning. Both filtering and motor servo-control can be achieved with simple linear approxi-mations leaving all the non-linearities to be resolvedby the neurocontroller.

If the neurocontroller is successful in doing this,we can increase the complexity (and flexibility) byreducing reliance on modelling. The first step is toremove the vehicle model from the state estimator,using it only to integrate and filter data using sensormodels. Direct motor commands (average voltages)could also be produced by the neurocontroller,removing the need for the individual thruster control-lers and the thruster model. Without the assistance ofa model-based state estimator and individual thrustercontrollers the neurocontroller will have to learn fromless accurate data and form more complex mappings.

7 ConclusionMany important underwater tasks are based on visualinformation. We are developing robust feature trackingmethods and a vehicle guidance scheme that are basedon visual servo control. We have obtained initial resultsin reliably tracking features in underwater imagery andhave adapted a proven architecture for visual servocontrol of a mobile robot.

There are many approaches to the problem ofunderwater vehicle control, we have chosen to pur-sue reinforcement learning. Our reinforcementlearning method seeks to overcome some of the limi-

tations of existing AUV controllers and theirdevelopment, as well as some of the limitations existing reinforcement learning methods. In simultion we have shown reliable development of stabneurocontrollers.

AcknowledgementsWe thank Wind River Systems and BEI Systron Doner for their support and Pacific Marine Group for prviding underwater imagery. We also thank the RSUnderwater Robotics team for their contributions.

References[1] D. Yoerger, J-J. Slotine, “Robust Trajectory Control of Unde

water Vehicles,” IEEE Journal of Oceanic Engineering, voOE-10, no. 4, pp.462-470, October1985.

[2] R. Cristi, F. Papoulias, A. Healey, “Adaptive Sliding ModeControl of Autonomous Underwater Vehicles in the DivPlane,” IEEE Journal of Oceanic Engineering, vol. 15, no. pp. 152-159, July 1990.

[3] J. Lorentz, J. Yuh, “A survey and experimental study of neurnetwork AUV control,” IEEE Symposium on AutonomousUnderwater Vehicle Technology, Monterey, USA, pp 109-11June 1996.

[4] S. Hutchinson, G. Hager, P. Corke, “A Tutorial on Visual ServControl,” IEEE International Conference on Robotics anAutomation, Tutorial, Minneapolis, USA, May 1996.

[5] D. Wettergreen, H. Thomas, and M. Bualat, “Initial Resultfrom Vision-based Control of the Ames Marsokhod Rove“IEEE International Conference on Intelligent Robots and Sytems, Grenoble, France,1997.

[6] K. Nishihara, “Practical Real-Time Imaging Stereo MatcherOptical Engineering, vol. 23, pp. 536-545, 1984.

[7] T. Fossen, “Underwater Vehicle Dynamics,” UnderwaterRobotic Vehicles: Design and Control, J. Yuh (Editor), TSIPress, pp.15-40, 1995.

[8] K. Goheen, “Techniques for URV Modeling,” UnderwaterRobotic Vehicles: Design and Control, J. Yuh (Ed), TSI Press,pp.99-126, 1995.

[9] P. Werbos, “Control,” Handbook of Neural Computation,F1.9:1-10, Oxford University Press, 1997.

[10] J. Yuh, “A Neural Net Controller for Underwater Robotic Vehicles,” IEEE Journal of Oceanic Engineering, vol. 15, no. 3, p161-166, 1990.

[11] R. M. Sanner and D. L. Akin, “Neuromorphic Pitch AttitudeRegulation of an Underwater Telerobot,” IEEE Control Systems Magazine, April 1990.

[12] K. Ishii, T. Fujii, T. Ura, “An On-line Adaptation Method in aNeural Network-based Control System for AUV's,” IEEE Jounal of Oceanic Engineering, vol. 20, no. 3, July 1995.

[13] L. Kaebling, M. Littman, A. Moore, “Reinforcement LearningA Survey,” Journal of Artificial Intelligence Research, vol. 4pp. 237-285, 1996.

[14] C. Watkins, Learning from Delayed Rewards, Ph.D. Thesis,University of Cambridge, England,1989.

[15] L.-J. Lin. “Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching” Machine Learing Journal, 8(3/4), 1992.

[16] L. Baird, A. Klopf, “Reinforcement Learning with High-dimensional, Continuous Actions,” Technical Report WL-TR93-1147, Wright Laboratory, 1993.

[17] M. Harmon, L. Baird, “Residual Advantage Learning Applieto a Differential Game,” International Conference on NeurNetworks, Washington D.C, USA, June 1995.

[18] C. Gaskett, D. Wettergreen, A. Zelinsky, “ReinforcemenLearning applied to the control of an Autonomous UnderwatVehicle,” Australian Conference on Robotics and AutomatioBrisbane, Australia, pp. 125-131, March 1999.

[19] D. Yoerger, J. Cooke, J-J Slotine, “The Influence of ThrustDynamics on Underwater Vehicle Behavior and Their Incorpration Into Control System Design,” IEEE Journal of OceanEngineering, vol. 15, no. 3, pp. 167-178, July 1990.

[20] A. Healey, S. Rock, S. Cody, D. Miles, and J. Brown, “Towaran Improved Understanding of Thruster Dynamics for Undewater Vehicles,” IEEE Journal of Oceanic Engineering, vol. 2no. 4., pp. 354-361, July 1995.

[21] R. Bachmayer, L. Whitcomb, M. Grosenbaugh, “A Four-Quarant Finite Dimensional Thruster Model,” IEEE OCEANS’98Conference, Nice, France, pp. 263-266, September 1998.