wsdc cdr ingest - california institute of...
TRANSCRIPT
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 1WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Ingest Subsystem
Tim ConrowIPAC
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 2WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
WSDC Functional BlockDiagram
HRP
White Sands
MOS
JPL
SOC
UCLA
Ingest
Scan/FramePipelines
- Level-0,1b frames- Coadds- Frame index- Calibration data- Ancillary data- 2MASS ref. catalog- QA Data- Meta-data
Ops Archive
-Working Source DB . Coadd . Frame (epoch)- QA Meta-data- Image Meta-data- Final Products . Catalogs . Atlas
IRSA Archive
Archive
Multi-FramePipeline
QA
FPG
Images
H/K Maneuvers
Ancillary
dataMission
Status
WebWSDCCaltech
Science Team
Expectant Public
FTP Site
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 3WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Driving Requirements
• Key WSDC System Level Functional Requirements– Ingest functions
• Create Level-0 archive (L4WSDC-29)– Decompress– Depacketize– Make FITS files– Correlate with ancillary data
• Create ancillary data archive (L4WSDC-33)• Validate data (L4WSDC-35)• Notify MOS of receipt (L4WSDC-36)
– Throughput and Latency• Data volume: 25GB/day (L4WSDC-30), 50GB/day peak (L4WSDC-31)• 6 months of data acquisition (L4WSDC-82)• Quicklook: QA report within 24 hours (L4WSDC-32)• Ingest+Scan/Frame Pipelines: Level-1 available within 3 days (L4WSDC-34,
L4WSDC-39)
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 4WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Ingest Status
• Prototype Ingest supporting HRP testing– Depacketize realistic CCSDS source packet telemetry– Decompress images compressed with USES chip
• Uses standard rice decompression software– Meta-data extractino from packet headers– Save FITS file with meta-data– Compare to truth image (not relevent to ops, of course)– Error checking and handling– Extracts 100 framesets in ~1 minute, <1 second/frameset
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 5WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Overview
Imagebuffer
Ground-station
White Sands. H/K as CSV - Temp.s - SPICE kernel - etc.. Orbit events
MOS (JPL) SOC (UCLA)
Mission plan
. Dedicated ~4 Mbit/s
. 25 Gbyte/day = ~14 hours. Data PUSH WSDC (IPAC)
. ~1GB/day
. Data PUSH . Weekly. Low volume
AncillaryData
archiveIngest-1
Level-0 Frames
. Science telem
. Manifest
Internet
HRP
Scan pipelineScan pipelineScan pipelineScan pipeline
Generatelevel-0frames
Framepipeline
Ingest-2Raw
image staging
FrameIndex
Metadata Raw
telemarchive
QuicklookpipelineFrame
pipeline
Pipelines
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 6WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Image Telemetry Receipt
DeliveryComplete
?No
Yes
Pre-requisitessatisfied
?
Pause
Wait for an HRP file
No
Pause
. Read manifest
. Confirm all files present
. Ancillary data
StageData
TelemetryStaging
Level-0 framecreation
Yes
DeliveryDB
HRP SFtp site
White Sands
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 7WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Ancillary Data Receipt
DeliveryComplete
?No
Yes
Pause
Wait for an MOS file
. Read manifest
. Confirm all files present
ArchiveData
AncillaryData Deep
Archive
Meta-dataDB
ParseData
. PEF
. CSV H/K
. Clock/C-kernels
. Etc.
MOS SFtp site
DeliveryDB
JPL
EventDB
Create level-0frames
Clock/C-kernels
NAIFToolkit
VTC=>UTCconversion
ADCS PointingEstimates
Meta-data
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 8WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Level-0 Frame Creation
EventDB
DeliveryDB
Meta-dataDB
TelemetryStaging
Image TelemetryReceipt Ancillary Data
Receipt
DepacketizeDecompress
Assign frame IDRetrieve meta-data
. VTC => UTC
. UTC => Events, Meta-data
Write level-0 archive
Level-0Archive
PipelinesQuicklook, Scan/frame
pipeline kickoff
Rice Library
Frame Index
Asynchronous
CFITSIO
4 concurrent jobs
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 9WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Image Telemetry Handling
• Corrupted images are discarded– If an image is damaged in any way in transmission between detector tand ground,
it cannot be used– It is unlikely such images will be replayed
• Incomplete images are discarded; they cannot be reconstructed– Downlink gaps due to transmission dropouts will be recovered if the gap size
exceeds a replay threshold• Replaying is scheduled without WSDC input since it must happen quickly• Exact replay protocol still to be negotiated between MOS and WSDC
– Partial images at the start/end of downlinks will be covered by other downlinks
• Duplicate images (as per the frame index) are discarded• Framesets with missing bands are processed as fully as possible
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 10WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Frame Index
• Used for– Deciding on completeness of a scan– Frame searches
• Position, time, grade (Coadd)• Position, time, image stats, events, grade (dynamic ICal)• Delivery, time, image stats, events, grade (Quicklook)• Scan (QA)
• Columns (* = indexed)– Identification: vtc*, utc*, deliv_id*, scan_id*, frame_num, frame_id*, band– Delivery: deliv_utc*, deliv_sz, deliv_file, level0_file– Position: ra_raw, dec_raw, pa_raw, ra_raw, dec_raw, pa_raw,
hpbin_coarse*, hpbin_fine*– Events: orbn, saa_stat, sci_stat, anneal_stat, anneal_dt, frame_dt, tdrss_dt– Image stats: ave, min,max,med, sig, psig, p1, p5, p16, p84, p95, p99, p999, etc.– Other: hk_utc, pipe_rel, pipe_utc, pipe_stat, ingest_grade, qa_grade
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 11WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Quicklook Kickoff
Ingest
ImageTelem.
White Sands
AncillaryData
JPL
- Frame times- Frame pixels
- scanID (12345a)- time- orientation (ra,dec,PA)- events (anneal? SAA? Etc.)
- ScanID + Frame# = FrameID (12345a123)- FrameID: scanID, t, ra, dec, events, pixel stat.s, …- ScanID: state: (queued? quicklook?, etc.)
FrameIndex
QuicklookpipelineFrame
pipeline
• Choose frames for quicklook processing based on info in the frame index– In current delivery– Position (away from GP, LMC, etc.)– Events (away from SAA, anneals, etc.)– Image statistics (normal looking pixel histogram, normal source density)– Etc.
MetaData
Level-0Frames
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 12WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Scan Pipeline Kickoff
Ingest
ImageTelem.
White Sands
AncillaryData
JPL
- Frame times- Frame pixels
- scanID (12345a)- time- orientation (ra,dec,PA)- events (anneal? SAA? Etc.)
- ScanID + Frame# = FrameID (12345a123)- FrameID: t, ra, dec, events, pixel stat.s, …- ScanID: state: (queued? quicklook?, etc.)
Ingest
FrameIndex
Delivery A
Delivery B
NeedDynamic
Cal?
ScanComplete?
Yes
No
YesImageTelem.
AncillaryData
No
Scan/FramePipeline
Check next scan
ScanPipeline:
Dynamic Cal
Queue scan
Wait for data
National Aeronautics and SpaceAdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
TPC - 13WISE Science Data Center CDR – January 29-30, 2008
WSDS Design
Schedule
• WSDS v0: Oct. 15 ‘07– Prototype: depacketize, decompress, FITS files
• WSDS v1: July 17 ‘08– Prelim: depacketize, decompress, FITS files– Prototype: image/ancillary receipt, frame index,
delivery/event/meta-data DBs, kickoff logic• WSDS v2: Feb. 28 ‘09
– Complete: All• WSDS v3: Aug. 4 ‘09
– Mature: All• WSDS v3.5: Jan. 26 ‘10
– No change• WSDS v4: Oct. 18 ‘10
– No change