an introduction to cube and the navigation surface r.m. hare july 2004

An Introduction to An Introduction to CUBECUBE and the and the Navigation SurfaceNavigation Surface

R.M. HareR.M. Hare

July 2004July 2004

An intro to What?An intro to What? CUBECUBE = Combined Uncertainty and Bathymetry = Combined Uncertainty and Bathymetry

Estimator. Developed by Dr. Brian Calder – Estimator. Developed by Dr. Brian Calder – Electrical Engineer with a background in digital Electrical Engineer with a background in digital signal processingsignal processing

Navigation SurfaceNavigation Surface – a database of those – a database of those bathymetry and uncertainty estimates that can be bathymetry and uncertainty estimates that can be used to generate chart and ENC soundings and used to generate chart and ENC soundings and contours (safety-biased) in an automated way. contours (safety-biased) in an automated way. Developed by Lt. Sheppard Smith – NOAA Corps Developed by Lt. Sheppard Smith – NOAA Corps officer Masters thesis projectofficer Masters thesis project

University of New Hampshire, Center for Coastal University of New Hampshire, Center for Coastal and Ocean Mapping and Joint Hydrographic Center and Ocean Mapping and Joint Hydrographic Center (NOAA, Larry Mayer, Andy Armstrong)(NOAA, Larry Mayer, Andy Armstrong)

Why the need?Why the need?

Most promising developments in Most promising developments in hydrography in at least a decadehydrography in at least a decade

NOAA Workshop at IOS July 28-29NOAA Workshop at IOS July 28-29 CUBECUBE will be implemented in HIPS 6.0 will be implemented in HIPS 6.0

It’s already in Fledermaus 6.1It’s already in Fledermaus 6.1 Navigation SurfaceNavigation Surface will be part of CARIS will be part of CARIS

toolbox for creating field sheet/chart toolbox for creating field sheet/chart soundings and contours (safety biased)soundings and contours (safety biased)

So what is CUBE?So what is CUBE?

a fresh attempt to “tell the truth” about data a fresh attempt to “tell the truth” about data acquired during a multibeam surveyacquired during a multibeam surveyss

Dave Well’s uses 7 axioms to describe Dave Well’s uses 7 axioms to describe what what CUBECUBE is all about is all about

There are 8 steps required (also according There are 8 steps required (also according to Dave Wells)to Dave Wells)

The 7 axioms of The 7 axioms of CUBECUBE

1.1. Most data, most of the time, are mostly okayMost data, most of the time, are mostly okay2.2. A sounding is not a depthA sounding is not a depth 3.3. The appropriate question to ask is “What is the depth The appropriate question to ask is “What is the depth

(and its uncertainty) here?” NOT “How good is this (and its uncertainty) here?” NOT “How good is this sounding?”sounding?”

4.4. Data cleaning is a Triage OperationData cleaning is a Triage Operation5.5. Every sounding solution must include estimates of its Every sounding solution must include estimates of its

uncertaintyuncertainty 6.6. MBES data density and spatial resolution may not be MBES data density and spatial resolution may not be

sufficient to capture all bathymetric features of interestsufficient to capture all bathymetric features of interest 7.7. Models do not represent truthModels do not represent truth

Axiom 1Axiom 1

Most data, most of the time, are mostly Most data, most of the time, are mostly okayokay

has has two practical consequencestwo practical consequences::1.1. Blunders (outliers, inappropriate soundings) Blunders (outliers, inappropriate soundings)

exist, but not commonlyexist, but not commonly

2.2. Systematic uncertainties have been removed (to Systematic uncertainties have been removed (to the extent possible) in advance of applying the extent possible) in advance of applying CUBECUBE

Axiom 2Axiom 2

A sounding is not a depthA sounding is not a depth highlights the fallacy of the “golden highlights the fallacy of the “golden

sounding”, based on the belief that sounding”, based on the belief that soundings are “real”soundings are “real”

All soundings are results of measurements All soundings are results of measurements (e.g. of time, of sound speed, of tide, …)(e.g. of time, of sound speed, of tide, …)

Axiom 3 (my paraphrase)Axiom 3 (my paraphrase)

Mariners want to know: “What is the depth (and Mariners want to know: “What is the depth (and its uncertainty) where I am?” NOT “How good is its uncertainty) where I am?” NOT “How good is that (selected) sounding over there?”that (selected) sounding over there?” intent of data cleaning is the estimation of derived intent of data cleaning is the estimation of derived

parameters (depths at nodes), rather than parameters (depths at nodes), rather than assessment / flagging of measured soundingsassessment / flagging of measured soundings

what we call “soundings” these days are themselves what we call “soundings” these days are themselves estimated derived parameters, based on a multitude estimated derived parameters, based on a multitude of measurements (two-way travel time, beam of measurements (two-way travel time, beam direction, roll, pitch, heave, dynamic draft, etc.), and direction, roll, pitch, heave, dynamic draft, etc.), and processes (ray tracing using an imperfectly known processes (ray tracing using an imperfectly known sound speed profile, etc.)sound speed profile, etc.)

Axiom 4Axiom 4

Data cleaning is a Triage OperationData cleaning is a Triage Operation, so…, so… provide power tools to reduce workloadprovide power tools to reduce workload replace subjectivity with objectivityreplace subjectivity with objectivity algorithm should produce repeatable resultsalgorithm should produce repeatable results

• Same cannot be said about human hydrographic data Same cannot be said about human hydrographic data cleanerscleaners

Algorithmic Hydrography tools always divide data into: Algorithmic Hydrography tools always divide data into:

a) clearly good;a) clearly good;

b) clearly bad; and b) clearly bad; and c) questionable - requiring further attentionc) questionable - requiring further attention

Axiom 5Axiom 5

Every sounding solution must include Every sounding solution must include estimates of its uncertaintyestimates of its uncertainty essential to any form of uncertainty essential to any form of uncertainty

propagationpropagation (including (including CUBECUBE)) key feature of IHO S-44, 4key feature of IHO S-44, 4 thth Edition, and of the Edition, and of the

ISO 191xx family of standardsISO 191xx family of standards CUBECUBE demonstrates how this information demonstrates how this information can can

be exploited to our advantagebe exploited to our advantage

Axiom 6Axiom 6

MBES data density and spatial resolution MBES data density and spatial resolution may not be sufficient to capture all may not be sufficient to capture all bathymetric features of interestbathymetric features of interest you can’t make a silk purse out of a sow’s earyou can’t make a silk purse out of a sow’s ear there will always be features (sand ripples, there will always be features (sand ripples,

scours, pebbles, boulders) that are too small scours, pebbles, boulders) that are too small for the MBES to detectfor the MBES to detect

Axiom 7Axiom 7

Models do not represent truthModels do not represent truth simplified description of the complexity of the simplified description of the complexity of the

real worldreal world estimation models are based on mathematical estimation models are based on mathematical

processesprocesses Sometimes human experience and gut Sometimes human experience and gut

feelings are more valid than model resultsfeelings are more valid than model results A prudent navigator never depends upon a A prudent navigator never depends upon a

single source of informationsingle source of information

The 8 steps of The 8 steps of CUBECUBE1.1. Establish a network of “nodes” or “depth estimation Establish a network of “nodes” or “depth estimation

points” over the survey areapoints” over the survey area 2.2. At each node, maintain sequentially-determined At each node, maintain sequentially-determined

estimates of depth, and depth uncertainty (variance)estimates of depth, and depth uncertainty (variance) 3.3. Compare incoming sounding measurements against Compare incoming sounding measurements against

these existing nodal estimatesthese existing nodal estimates4.4. Establish a new “hypothesis” or alternative surface Establish a new “hypothesis” or alternative surface

model, on a node by node basis, when incoming model, on a node by node basis, when incoming soundings are not statistically consistent with previous soundings are not statistically consistent with previous nodal estimatesnodal estimates

5.5. Run the “disambiguation engine”Run the “disambiguation engine” 6.6. CUBECUBE output = the nodal point-set plus four surfaces output = the nodal point-set plus four surfaces 7.7. Caution One: Rinse, spit and repeat as necessaryCaution One: Rinse, spit and repeat as necessary 8.8. Caution Two: Consider hydrographic safety (aka Caution Two: Consider hydrographic safety (aka

Algorithmic Hydrography don’t know Jack!)Algorithmic Hydrography don’t know Jack!)

Step 1 – create a gridStep 1 – create a grid

Establish a network of “nodes” (depth Establish a network of “nodes” (depth estimation points) over the survey areaestimation points) over the survey area

Select nodes to be:Select nodes to be:1.1. dense enough to completely capture all features dense enough to completely capture all features

of interestof interest

2.2. no denser than neededno denser than needed

3.3. regularly-spacedregularly-spaced (or perhaps quad tree) (or perhaps quad tree)

4.4. at horizontal locations that are “perfect” since at horizontal locations that are “perfect” since they are selected by convention, and do not they are selected by convention, and do not involve any measurementsinvolve any measurements

Appropriate ResolutionAppropriate Resolution

Slide credit: Brian Calder

Estimation at a PointEstimation at a PointEstimation at a point means: - Only one true depth - No horizontal uncertainty - Soundings should agree on depth

Implications: - Only need to estimate a constant - Must propagate information - Uncertainty gives measure of expected consistency


Step 2 – initialize the gridStep 2 – initialize the grid

At each node, maintain sequentially-determined At each node, maintain sequentially-determined estimates of depth, and depth uncertainty estimates of depth, and depth uncertainty (variance)(variance) Need a starting pointNeed a starting point

• Could be derived from digitized historic field sheetsCould be derived from digitized historic field sheets• Normally, start with depth=0, uncertainty = 1000 mNormally, start with depth=0, uncertainty = 1000 m

current estimate can be updated by new local current estimate can be updated by new local sounding measurementssounding measurements

means that, in principle, means that, in principle, CUBECUBE can be used as a real- can be used as a real-time estimation tool (built into the MBES data time estimation tool (built into the MBES data acquisition software), rather than only in post-acquisition software), rather than only in post-processing software like CARIS HIPSprocessing software like CARIS HIPS

Step 3 – start loading soundingsStep 3 – start loading soundings

Compare incoming sounding measurements against Compare incoming sounding measurements against these existing nodal estimatesthese existing nodal estimates Each incoming sounding is considered to contain some Each incoming sounding is considered to contain some

(imperfect) information about the depth at each of the nodes (imperfect) information about the depth at each of the nodes surrounding itsurrounding it

propagation strategy involves propagation strategy involves three processesthree processes that that appropriately dilute the 3D sounding uncertainty based on the appropriately dilute the 3D sounding uncertainty based on the sounding / node separationsounding / node separation 1.1. give closer soundings and soundings with smaller depth give closer soundings and soundings with smaller depth

uncertainty higher uncertainty higher weightweight2.2. convert the convert the horizontalhorizontal sounding uncertainty into a vertical sounding uncertainty into a vertical

uncertainty at the nodeuncertainty at the node 3.3. make a make a statisticalstatistical comparisoncomparison between the incoming sounding between the incoming sounding

(propagated to the node location), and the already-assimilated (propagated to the node location), and the already-assimilated information at the nodeinformation at the node

Error Model: UncertaintiesError Model: UncertaintiesRoll

YawPitch

Heave

v

Beamwidth

Range Sampling

v

z

SVP

Tx/Rx

GPS

VRU

time

VRU

SonarAtt

itude

Latency


Error Model: VerticalError Model: Vertical


Error Model: HorizontalError Model: Horizontal


Error ModelingError Modeling


Step 4 – check nodal consistencyStep 4 – check nodal consistency

Establish a new “hypothesis” or alternative Establish a new “hypothesis” or alternative surface model, on a node by node basis, when surface model, on a node by node basis, when incoming soundings are not statistically incoming soundings are not statistically consistent with previous nodal estimatesconsistent with previous nodal estimates Why is this so innovative? Two reasons:Why is this so innovative? Two reasons:

a) It differs from previous approaches by deferring final a) It differs from previous approaches by deferring final judgment on appropriateness of a sounding until all the judgment on appropriateness of a sounding until all the evidence is inevidence is in

b) It recognizes the possibility that soundings in the same area b) It recognizes the possibility that soundings in the same area may be sensing more than one depth trend (e.g. the top of a may be sensing more than one depth trend (e.g. the top of a school of fish as well as the bathymetry beneath them).school of fish as well as the bathymetry beneath them).

Robustness: Multiple HypothesesRobustness: Multiple Hypotheses Robustness is a:Robustness is a:

““judicious and grudging elaboration of the model to judicious and grudging elaboration of the model to ensure against particular hazards” (G.E.P. Box).ensure against particular hazards” (G.E.P. Box).

Observed problems:Observed problems: Spontaneous outliers (e.g., missolved beams)Spontaneous outliers (e.g., missolved beams) Burst mode sounder failuresBurst mode sounder failures Systematic artifacts (e.g., dynamic draft)Systematic artifacts (e.g., dynamic draft)

We track multiple potential depths per node:We track multiple potential depths per node: Piecewise defined Dynamic Linear Model (DLM)Piecewise defined Dynamic Linear Model (DLM) Incorporates failures in separate hypothesesIncorporates failures in separate hypotheses Extends ‘memory range’ of estimatorExtends ‘memory range’ of estimator


CUBE: Multiple HypothesesCUBE: Multiple Hypotheses


Step 5 – resolve inconsistenciesStep 5 – resolve inconsistencies Run the “disambiguation engine”Run the “disambiguation engine”

decide which of several hypotheses that may exist at decide which of several hypotheses that may exist at a node is most likely to be correcta node is most likely to be correct

Subject to ongoing refinement, engine uses three Subject to ongoing refinement, engine uses three “metrics” to resolve most probable hypothesis:“metrics” to resolve most probable hypothesis:

a) The popularity contest. Which hypothesis is consistent with a) The popularity contest. Which hypothesis is consistent with the greatest number of incoming soundings at that node?the greatest number of incoming soundings at that node?

b) The local consistency test. Which hypothesis is closest to a b) The local consistency test. Which hypothesis is closest to a prediction based on nearest neighbor nodes that have only prediction based on nearest neighbor nodes that have only one hypothesis?one hypothesis?

c) The external consistency test. Which hypothesis is closest c) The external consistency test. Which hypothesis is closest to an “external” (probably lower resolution) reference to an “external” (probably lower resolution) reference surface. This surface could be generated by a median filter surface. This surface could be generated by a median filter through all the soundings being processed, or based on an through all the soundings being processed, or based on an older chart or survey data.older chart or survey data.

CUBE Data AssimilationCUBE Data Assimilation


Step 6 – take a lookStep 6 – take a look

CUBECUBE output = the nodal point-set plus four output = the nodal point-set plus four surfacessurfaces::1.1. DepthDepth – the final depth value at each node, after the – the final depth value at each node, after the

competing hypotheses have been disambiguatedcompeting hypotheses have been disambiguated2.2. UncertaintyUncertainty – the final uncertainty associated each – the final uncertainty associated each

nodal depthnodal depth3.3. The The number of hypothesesnumber of hypotheses at each node at each node4.4. The uncertainty associated with the selection of the The uncertainty associated with the selection of the

correct hypothesis by the disambiguation engine. correct hypothesis by the disambiguation engine. This “This “hypothesis strengthhypothesis strength”, is a measure of ”, is a measure of algorithmic certainty - how convinced algorithmic certainty - how convinced CUBECUBE is that is that it's telling you the real story.it's telling you the real story.

Output Surface DataOutput Surface DataD

epth

Est

imat

eN

um. H

ypot

hese

s

Unc

erta

inty

Est

imat

eH

ypot

hesi

s S

tren

gth


Step 7 – iterateStep 7 – iterate

Caution One: Rinse, spit and repeat as Caution One: Rinse, spit and repeat as necessarynecessary Axiom 4 - Axiom 4 - CUBECUBE is a triage tool is a triage tool CUBECUBE does its best to get as much right as does its best to get as much right as

possible, but always needs a user at the end possible, but always needs a user at the end to resolve problems that the algorithm cannotto resolve problems that the algorithm cannot

makes the user's job different: not so much makes the user's job different: not so much about about killing dotskilling dots, more about , more about quality quality assuranceassurance

Implies adaptive sampling if used in real-timeImplies adaptive sampling if used in real-time

CUBE: Per-Day Pass 1CUBE: Per-Day Pass 1


Step 8 – retain human controlStep 8 – retain human control

Caution Two: Consider hydrographic safety (aka Caution Two: Consider hydrographic safety (aka Algorithmic Hydrography don’t know Jack!)Algorithmic Hydrography don’t know Jack!) Axiom 7 - Axiom 7 - CUBECUBE inhabits the model world, not the real inhabits the model world, not the real

worldworld Users should not take raw Users should not take raw CUBECUBE grids and use them grids and use them

for chartingfor charting hydrographer should be allowed to override the hydrographer should be allowed to override the

algorithm, and retain those (rare) "critical to algorithm, and retain those (rare) "critical to navigation" sounding values in any given surveynavigation" sounding values in any given survey

And that leads us to …And that leads us to …

CUBE: Cumulative Pass 2CUBE: Cumulative Pass 2


Total Time ComparisonTotal Time Comparison

Traditional Process: 305.06 hrs. CUBE Process: 124.84 hrs.


Total Time ComparisonTotal Time Comparison

0.00 50.00 100.00 150.00 200.00 250.00 300.00 350.00

Traditional

CUBE

Time Expended (hrs)

In Survey Post Survey Reporting Troubleshooting


The The Navigation SurfaceNavigation Surface

Redefining the Field SheetRedefining the Field Sheet Creation of a databaseCreation of a database A process to create multiple products from A process to create multiple products from

a single sourcea single source Comparison of traditional field sheet to Comparison of traditional field sheet to

CUBECUBE and and Navigation SurfaceNavigation Surface output output product.product.

Redefining Field SheetsRedefining Field Sheets Field sheet is typical primary archive of a Field sheet is typical primary archive of a

hydrographic surveyhydrographic survey But:But:

It represents the data at only It represents the data at only one scaleone scale … … … … and and notnot at the at the best achievablebest achievable resolution resolution

And, it’s not adequate:And, it’s not adequate: For other derived products (e.g., marine geology, For other derived products (e.g., marine geology,

habitat studies)habitat studies) For automatic cartographic processing (e.g., For automatic cartographic processing (e.g.,

automatic contour generation)automatic contour generation)

Slide credit: Brian Calder and Shep Smith

A possible alternativeA possible alternative

Replace the sheet with a database:Replace the sheet with a database: ‘‘Best Available’ representation of the sea-floor as a Best Available’ representation of the sea-floor as a

(possibly irregular) grid(possibly irregular) grid Attribute each grid node with an estimate of the Attribute each grid node with an estimate of the

measurement uncertaintymeasurement uncertainty Generate products directly from the database:Generate products directly from the database:

On demand automatic processingOn demand automatic processing Generalize to required resolution from grid with Generalize to required resolution from grid with

techniques to ensure hydrographic safetytechniques to ensure hydrographic safety Construct other components from generalized grid Construct other components from generalized grid

(e.g., selected soundings or contours)(e.g., selected soundings or contours)


Proposed ProcessProposed Process

SourceSoundings

NauticalCharts

Survey ProductsDatabase

NavigationSurface

HydroVector

Database

ENC

Other


Proposed ProcessProposed Process A A Navigation SurfaceNavigation Surface is: is:

A ‘best resolution’ regular or irregular grid of depth A ‘best resolution’ regular or irregular grid of depth estimates, plus uncertainty associated with the estimates, plus uncertainty associated with the estimateestimate

A methodology for generalization of the grid to any A methodology for generalization of the grid to any scale with hydrographic safetyscale with hydrographic safety

Processing Goals:Processing Goals: Speed up interactive time in MBES processingSpeed up interactive time in MBES processing Speed up down-stream processes with automatic Speed up down-stream processes with automatic

cartographic methodscartographic methods Maintain survey in digital database format for ease Maintain survey in digital database format for ease

of use and maintenanceof use and maintenance


Snow Passage, AKSnow Passage, AK


Proposed ProcessProposed Process

SourceSoundings

NauticalCharts

Survey ProductsDatabase

NavigationSurface

HydroVector

Database

ENC

Other


Grid ManipulationGrid Manipulation


Generalize to Scale of ProductGeneralize to Scale of Product Detail must be sacrificed for clarity of a particular product at a Detail must be sacrificed for clarity of a particular product at a

particular scaleparticular scale Multiple products may be defined – don’t want to have to fix at any Multiple products may be defined – don’t want to have to fix at any

particular survey scale in the databaseparticular survey scale in the database Significant features have to be preserved, e.g. shoal depthsSignificant features have to be preserved, e.g. shoal depths

Current Process:Current Process: Subsampling (binning & suppressing soundings) to maximum Subsampling (binning & suppressing soundings) to maximum

resolution (based on survey instructions)resolution (based on survey instructions) Smoothing LineworkSmoothing Linework Combination of small featuresCombination of small features

Defocussing Process:Defocussing Process: Maintain database at highest resolutionMaintain database at highest resolution Manipulate the model, not the cartographic objectsManipulate the model, not the cartographic objects Keep relevant details (shoals, etc.) while simplifying data to scale of Keep relevant details (shoals, etc.) while simplifying data to scale of

final productfinal product


Cartographic Objects from Cartographic Objects from Defocused SurfaceDefocused Surface


Smooth Sheet SoundingsSmooth Sheet Soundings


Full Surface Multi-ResolutionFull Surface Multi-ResolutionSnow Passage Selected Sounding Comparison

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Fraction of IHO Survey Order Vertical Error Budget

Cu

mu

lati

ve P

rob

abil

ity


ConclusionsConclusions CUBE can achieve equivalent performanceCUBE can achieve equivalent performance

Does not resolve systematic errors, but does indicate that they Does not resolve systematic errors, but does indicate that they existexist

Computes good estimates even in severe noiseComputes good estimates even in severe noise

Operator interaction time is heavily reducedOperator interaction time is heavily reduced Two runs (and one setup) are requiredTwo runs (and one setup) are required Final time reduction appears to be between 10-20:1Final time reduction appears to be between 10-20:1 Operator’s task is primarily data quality control rather than Operator’s task is primarily data quality control rather than

editing each outlierediting each outlier

Resolution is an essential issueResolution is an essential issue Current implementation uses masks and multiple MapSheets to Current implementation uses masks and multiple MapSheets to

implement a crude multi-resolutionimplement a crude multi-resolution Production implementation will have a more sophisticated Production implementation will have a more sophisticated

solution, probably quad-tree basedsolution, probably quad-tree based


an introduction to cube and the navigation surface r.m. hare july 2004

Documents

cube axiom

uncertainty estimates

uncertainty mbes data

sounding solution

intent of data cleaning

inappropriate soundings

axioms of cubemost data

enc soundings