PDSMC November 29-30, 2012 1

PDS Imaging Node PDS4 Migration

Lisa Gaddis (USGS)

Sue LaVoie (JPL)

November 30, 2012PDS Management Council Meeting

UCLA

PDSMC November 29-30, 2012 2

• Migrate only past missions & newly awarded missions (InSight)– Justification: for active missions, community analysis is ongoing with existing

software & tools– Prioritize based on need for class development/follow-on missions and popularity– Retain PDS3 labels & metadata within PDS4 structure

• Migrate mission datasets requiring no (or simple) data manipulation– Retain copy of any data requiring manipulation

• Migrate metadata only of mission datasets requiring data manipulation– Justification: complex data migration is too risky– Retain PDS3 data format & label– Create PDS4 labels and metadata to enable complex data product searches

across all holdings– Decisions/Action required by MC & Data Modelers

• Policy allowing waiving of data migration for complex legacy data sets• Dictionary terms/keywords enabling description of legacy formats

• Modify IMG systems to support both PDS3 & PDS4 data sets & formats

PDS4 Migration Strategy

PDSMC November 29-30, 2012 3

• Process– Identify data set for migration– Create/update Discipline Classes, provide input to PDS-wide Classes– Map PDS3 to PDS4 labels and ancillary files/data– Create schema– Modify existing software tools & scripts to automatically create labels, collection

and bundles– Generate Sample Bundle(s)– Verify– Iterate– Informal IMG internal review & testing– Load Bundle into IMG Registry– Identify next data set– …..

– Repeat iterative process until Data Model and Classes are “final”, refining all along the way. Then perform the full data set migration on priority data sets

PDS4 Migration Approach

PDSMC November 29-30, 2012 4

PDS4 Migration - Top Priority Data Sets

PDSMC November 29-30, 2012 5

PDS4 Migration – Waive Data Format Change

PDSMC November 29-30, 2012 6

• Clementine 750nm Basemap (Chris)– Sample Bundle validated, reviewed by EN, updated and redelivered for ingestion

• Folders in bundle: data (6 map projected single band images & ISIS detached labels), document (.txt, .doc, .htm, .pdf versions of “volinfo” suite), miscellaneous (delivery & transfer manifests), xml_schema

– Cartography Class - Incorporating Federal Geographic Data Committee (FDGC) model. Still needs fine tuning

– Tools used: oXygen XML Editor, IMG Generate Tool, EN Validate Tool, ad hoc scripts

• Mars Pathfinder (MPF) IMP EDR (Elizabeth)– Sample Bundle validated and delivered to EN for review and feedback

• Folders in bundle: data (200 files), browse (browse for each data file), context (instrument, instrumet host, investigation, resources, etc.), document (EDR SIS), geometry (gazetteer), xml_schema

– Imaging Instrument Parameter Class – in good shape for this data set & Clementine data set

– Tools used: oXygen XML Editor, IMG Generate Tool, EN Validate Tool, ad hoc scripts

Migration Progress

PDSMC November 29-30, 2012 7

• Clementine UVVIS 5-band Mosaic (Chris)– Multispectral data set– Preliminary Bundle layout defined

• Mars Pathfinder (MPF) RVRCAM (Elizabeth)– Tables and Engineering data sets

• Phoenix OM, RAC, SSI EDR (Amy)– Precursor for InSight (launch 2016; Geo is lead node)

• Landed mission products and metadata• Phoenix, MER, MSL heritage• Work closely with Geo throughout

– Class work: Camera Model, Landed Geometry, Mars InSitu, likely more

Migration – Next Up

PDSMC November 29-30, 2012 8

• Accomplishments– Installed all PDS2010 software on IMG operational machine, including Registry,

Registry UI, Harvest, and Validate• Used Validate tool on test bundles prior to delivery to EN for review

– Continued development of IMG Generate Tool (label generation) to support ongoing data migration activities

– Created internal IMG Wiki to track PDS4 migration details, progress, lessons learned, open items, versions of dictionaries and schemas used, etc.

• Plans– Harvest test & completed Bundles into IMG Registry– Work with Engineering Node to communicate with central Registry

System & Tools Status

PDSMC November 29-30, 2012 9

Back-up

PDSMC November 29-30, 2012 10

Imaging Node: PDS4 Data Migration• IMG manages a lot of

complex data with a wide variety of formats

– 203 data sets, ~13 M products

– Many of these are not PDS4-compliant

• 90% of IMG data sets require

– Assessment & revision of classes of keywords

• Discipline Classes• PDS-wide Classes

– Manipulation of data to achieve compliance

• 25% of IMG data sets

• But, we assume that migration of all data sets is not required

PDSMC November 29-30, 2012 11

IMG Data Migration: Data Manipulation• Data Manipulation

– 25% of IMG data sets• 52 of 203 data sets

– Potentially very time consuming and costly

– Process:• Engage experts• Software development & testing• V&V• Peer Review?• Release

– Issues:• Retain PDS3 version – forever?

PDSMC November 29-30, 2012 12

IMG Data Migration: Classes• Class development will require

– IMG for Discipline-specific classes– Cross-node participation for many others

• Classes needed for IMG - prelim analysis (not all included on chart)– Calibration – PDS-wide– Compression – PDS-wide– Geometry

• Camera Model – IMG• Surface Geometry – PDS-wide• SPICE Geometry – PDS-wide

– Cartography – IMG– Imaging Instrument Parameter – IMG– Telemetry Parameters – IMG– Command Parameters – IMG– Processing History – IMG– Imaging Instrument Compression – IMG– Mosaic – IMG– Anaglyph (?) - IMG– Spectral Qube (?) - IMG

PDSMC November 29-30, 2012 13

PastMissions

PDSMC November 29-30, 2012 14

ActiveMissions