geometric enhancement to physics-based target detection mike foster 15 aug 06

Proposal OverviewProposal Overview

Geometric Enhancement to Physics-based Target Detection

Mike Foster

15 Aug 06

Digital Imaging and Remote Sensing Laboratory

OverviewOverview

• Motivation & Hypothesis

• Methodology– Ground plane orientation correction– Mixing fraction prediction

• Preliminary results


Target DetectionTarget Detection

ReflectanceImage(s)

ReflectanceImage(s)

Radiance Domain

Reflectance Domain

Target Reflectance

Radiance Image

Traditional Approach

AtmosphericCompensationAtmospheric

Compensation

DetectionDetection

Physics-Based Approach

Radiance Image

Target Space

Physical ModelThe “Big Equation”

Physical ModelThe “Big Equation”

DetectionDetection

Target Reflectance


Physics-Based DetectionPhysics-Based Detection

• Advantages– Physical variations (illumination, target contamination,

adjacency) can be integrated in process.• On the atmospheric compensation side, would have to generated

multiple cubes

• Disadvantages– Requires moderately complex infrastructure to create

estimates (i.e., target space)– Run times can be long

• Lack of geometric knowledge drives the span of Target Spaces

– Incorporation of unnecessary physical variation can be detrimental for detection for a given pixel


The “Big Equation”The “Big Equation”

Predicts spectral radiance at a sensor based on a target reflectance for a specific atmosphere and geometry

• Inherent geometric terms (F, • Inherent spectral terms (everything else)

• Pure target pixel implied in prediction


The “Big Equation”The “Big Equation”

(Modeled “Full Pixel” Vector)


Physics-based ModelingPhysics-based Modeling

• Messinger’s physical model for mixed pixels



• Geometric model variables – Linear mixing fraction, M: 0.2-1.0 – Ground plane orientation, cos(: 0-1– Sky dome shape factor, F: 0-1

• Spectral model variables– Various probable atmospheres (i.e. constituent

levels) based on time of year and location

• Massive target space– Target variability vs vector space confusion



• Possible to constrain the geometric model variables using spatial information from 3D Lidar data

• Constrained on a per-pixel basis• Assumptions

– Co-registered hyperspectral and Lidar data– High Lidar spatial sampling relative to spectral pixel IFOV

• Providing physical model with accurate constraints does 2 things:– Potential to speed up run times associated with generating target

space– Reduces target space noise or confusion = better detection

performance


OverviewOverview





Ground Plane Orientation CorrectionGround Plane Orientation Correction

• Ground plane extraction– Estimated using Lidar last pulse return logic, in combination with

global filtering – Trees and buildings removed using various filtering techniques– Resulting data gaps are filled using interpolation– Leverage technique developed by PhD-candidate Steve Lach



Hyperspectral FPA

Extracted

ground plane

• Ortho-project ground plane points to FPA

• Determine which 3D points fall in each FPA bin



• Ortho-project the points associated with ground plane onto the hyperspectral sensor focal plane

• Compute the ground plane normal for each point location using eigenvalue decomposition

• Compute the mean ground plane normal vector for each hyperspectral pixel

• Compute between solar declination vector (i.e. unit vector from sun to global coordinate origin) and mean ground plane normal vector for every pixel


Mixing Fraction PredictionMixing Fraction Prediction

• Challenging problem

• Requires means of recognizing 3D shapes (i.e. targets) in point clouds

• No a priori knowledge of target pose

• Knowledge of sensor location and ground plane does provide a priori knowledge of target scale

• Must be robust in the presence of occlusion



• 3D target matched filter problematic– 6 Degrees of Freedom

• X, y, z, tip, tilt, pan

– Target articulation – Not robust in presence of occlusion

• Spin-Images is potential solution– Adapted technique from Robotic Vision


Spin-ImagesSpin-Images

• 2D parametric space image– Capture 3D shape information about a single point in

3D point cloud– Pose invariant

• Based on local geometry relative to a single point normal• i.e. invariant to tip, tilt, pan

– Scale variant• Estimate scale from ground plane/sensor position

– Graceful detection degradation in the presence of occlusion


Spin-Image FormationSpin-Image Formation

• Step 1: For a given point p in a 3D cloud, estimate point/surface normal– p referred to as spin image basis point– Normal estimate referred to as spin image basis normal– Coordinate space is localized for a single point p– Points are spatial samples of a 3D surface– Define voxel with p at origin– Use eigenvalue decomposition to determine surface

normal– Eigenvector associated with smallest eigenvalue

estimates surface normal



• Step 2: Calculate 2D parameter space coordinates – Radial distance to

local point normal

– Signed vertical distance along basis normal

22)( pxnpx )( pxn



• Step 3: Build Spin “Image”– 2D histogram of all points’ alpha and beta

coordinates relative to generation basis point, p

• Spin Image generation variables– Only points within a set range of p are allowed

to contribute to spin image (aka spin support)– Histogram bin size should be ~3-4 times larger

than mean point spacing– Spin angle (described later)


Spin-Image ExamplesSpin-Image Examples

• 3 spin image pairs corresponding to 3 different points on the model

• Left image is high resolution spin image (small bin size)

• Right image is low resolution image (larger bin size) post bilinear interpolation


Spin Image Library MatchingSpin Image Library Matching

• How do I find the model in a real scene?– Build a library of spin images using the model

• Spin image for every point on the model– Note generation variables: spin support, bin size, and spin

angle

• Only has to be done once

– Build a spin image for a point in the scene using same generation variables

– Compare scene spin image to all spin images in model-based library



• Will a spin image from a target point in my scene match a spin image from the model library?– Model library generated from 3D CAD model

• Points on all sides of model• High sampling density• No occlusion

– Scene• Target and background present• Points only from Lidar illumination direction• Self-occlusion and background occlusion• Not necessarily at same spatial sampling as model library



• Intelligent model library generation– Typically model has many more points than scene data

• Normalize scene and model spin images

– Scene has points from only one illumination direction• Spin angle limits model points that can contribute to a spin

image when building model library• Compute normal for every point in model• Pick spin image basis point p• Allow only normals within 90° angle relative to spin image

basis normal to contribute to model spin image• This builds self-occlusion effects into model library



All model points allowed to contribute to spin image



Only model points (pink) with normals within 85° of spin image basis normal allowed to contribute to spin image



• Matching with occluded scenes– Compute correlation based on overlapping bins* in the

model spin image and scene spin image– N = number of overlapping bins – Similarity metric, S

– Model library spin image with highest S-score is best match

– Considered a point correspondence between model and scene

S = N × Correlation(Spin*m, Spin*s)



• Geometric consistency– Ensures similar locations between scene to model

point correspondences– Ensures proper basis normal orientation– Filters out bad correspondences


Spin Image LimitationsSpin Image Limitations

• Need sufficient point density on a surface to accurately estimate normal– May not work for targets under

trees

• Does not adequately address target symmetry– Most common “targets” have a

plane of symmetry– Points symmetric about plane have

identical spin images– May result in bad point

correspondences


Spin Image LimitationsSpin Image Limitations

• Bottom line: spin images may not be a robust target detector for point clouds with low spatial sampling (i.e. most Lidar systems)

• May be good enough to estimate mixing fractions

• Need to quantify mixing fraction uncertainty


Lidar Target DetectionLidar Target Detection

Range-gate scene data

Scene data

Scale 3D CAD Model

Extract ground plane

Create model spin image

library

Select point & create scene spin image

Compare scene spin image to

model spin image library

Sort & filter based on Similarity

Sort & filter Geometric

Consistency

Extract target 3D point locations based on best

model/scene spin images

repeat for n points



• For the best model/scene correspondence– Store spin image bin locations that are non-

zero in both the scene spin image and model spin image (i.e. overlapping bins)

– Extract the 3D point coordinates for the scene points that contributed to the overlapping bins

– Orthoproject 3D target points on to the hyperspectral FPA

– Mixing fraction, M = % of FPA pixel filled by target points



• Final ortho-projection of scene target points onto hyperspectral FPA necessary to estimate mixing fraction M on a per-pixel basis

9.0M

5.0M



• Create M uncertainty bounds – Perform linear unmixing on hyperspectral

image using spectral target vector as an end member

– Requires knowledge of background end members and target spectrum

– Compare unmixing target fraction to M, as predicted from spin image process


AnalysisAnalysis

• Test geometrically-enhanced target detect versus other established methods

• Present results in the form of multiple ROC curves– Compare ROC curves for Rx, Dr. Messinger’s

unconstrained mixed pixel model, traditional physics-based results, and my constrained model

– Compare ROC curves for my model using various degrees of spatial oversampling

– Compare ROC curves for my model for various viewing geometries

• Coincident LIDAR and hyperspectral platforms • Separate platforms with varying pointing geometries


OverviewOverview





Preliminary ResultsPreliminary Results

• Simulated Lidar point clouds using Rhino

• Fully coded spin image process

• Created high resolution model spin image library

• Created low resolution “scene” point clouds

• Matched scene points to model based on similarity and geometric consistency



• Use Rhino to convert facetized model of tank to sampled point cloud – Import obj/3ds model > DrapePt function >

Export point cloud with new vertex locations



• Normal estimation via eigenvector decomposition – Select a voxel containing small number of

points– Represent all points in voxel in 2D array (N x 3)

• N rows = point observations• 3 columns = point locations in x, y, z space

– Compute covariance matrix of 2D array– Eigenvector associated with smallest

eigenvalue is estimated normal vector


Spin Image Matching ExamplesSpin Image Matching Examples

• Scene 1 = Model translated, rotated, sampled from off NADIR

VIEW 1Model Scene



VIEW 2Model Scene

• Scene 1 = Model translated, rotated, sampled from off NADIR


Spin Image Matching ExamplesSpin Image Matching ExamplesScene 1 = Model translated, rotated, sampled from off NADIR

MODEL SPIN IMAGE SCENE 1 SPIN IMAGE


Spin Image Matching ExamplesSpin Image Matching Examples• Scene 2 = Model translated, rotated, sampled from off

NADIR 50% occluded



Model

• Scene 2 = Model translated, rotated, sampled from off NADIR - 50% occluded + tree clutter

SceneVIEW 1


Spin Image Matching ExamplesSpin Image Matching Examples• Scene 2 = Model translated, rotated, sampled from off

NADIR - 50% occluded + tree clutter

ModelVIEW 2

Scene



MODEL SPIN IMAGE SCENE 2 SPIN IMAGE

• Scene 2 = Model translated, rotated, sampled from off NADIR - 50% occluded + tree clutter


ConclusionsConclusions

• Constraining the target space associated with physics-based modeling may improve target detection performance

• 3D spatial information from Lidar may provide a means of estimating geometric parameter bounds

• Target detection based on fused spectral and spatial information may prove to be more robust than each sensing modality alone


Spin Image Surface MatchingSpin Image Surface Matching


Model / Scene RegistrationModel / Scene Registration

• Pink points represent occluded scene points of the bunny• Green 3D model of the bunny has been registered to

scene points


A

B

C

Sensor-reaching Radiance PathsSensor-reaching Radiance Paths


Mixing FractionsMixing Fractions

25.0M

1M

geometric enhancement to physics-based target detection mike foster 15 aug 06

Documents

predictiondigital imaging

pixel vectordigital

elsepure target pixel

target contamination

mixed pixelsdigital

fpa bin digital imaging

target spacerun times

d lidar data