CDAAC Status and StrategyBill Schreiner

and CDAAC Team

COSMIC Retreat

Nov 4, 2010

• CDAAC Objectives• CDAAC Overview• CDAAC Status• Summary of CDAAC mini-review• Processing COSMIC Follow-on missions and missions of

opportunity

Outline

• Provide high quality GNSS RO data to the community in NRT, post-processing, re-processing (re-analysis)

• Improve data processing algorithms to maximize impact of RO data on research and operations

• Support data users, TACC

• Conduct scientific investigations when appropriate and efficient

CDAAC Objectives

CDAAC Personnel

• Program Director: Bill Kuo

• Chief Scientist: Chris Rocken

• CDAAC Manager: Bill Schreiner

• S/W Engineers

– Doug Hunt, Lead

– Maggie Sleziak-Sallee, Web

– Teresa Vanhove, GPS

– Jason Lin, SE

– Karl Hudnut, System Admin/Engineer

• Scientists

– Bill Schreiner, GPS Lead

– Sergey Sokolovskiy, RO Inversions

– Xinan Yue, Ionosphere

– Janet Zeng, RO Inversions

– Tae-Kwon Wee, 1DVAR

Main CDAAC Functions

• RO Payload Operation– Configuration control (firmware and tracking configuration)– Scientific/technical guidance for commanding, operation– Near real-time monitoring, trouble-shooting, and Q/C data analysis

• Data Processing and Analysis– Level0 unformatting and QC– GPS ground processing (ZTD, site estimation, clock estimation)– LEO POD, and atmospheric excess phase– Absolute TEC generation– Inversions (neutral atmosphere and ionosphere)– Retrievals (1DVAR)– NWP and correlative data handling– Product QC and analysis

CDAAC Functions (cont)

• System Operation and Monitoring- H/W, O/S and NFS filesystem- System fail-overs- CDAAC operations

• Input data stream monitoring– RTS downlink– GPS Bit-grabber operation and monitoring– GPS ground data (IGS, NRCan, EUMETSAT, COSMIC

sites)– IGS and IGU orbits, clocks and EOP– Bernese configuration– NCEP GFS, ECMWF, radiosonde, ionosonde

• Data Management, Dissemination, and Archival• Support Data Users

ActiveI/OComputer

CDAAC Hardware

Real-Time Setup– Dual String system– Redundant Input/Output IO-

computers – Redundant Real-Time Operational

cluster computers (with RAIDS)– Real-Time staging cluster to stage

(test) software updates– Backup power and cooling

Post-Processing Setup– R&D cluster– Re-Analysis cluster for climate re-

analysis– FTP Server with RAID

BackupI/OComputer

CDAAC Processing Flow

LEO data Level 0--level 1

POD (Orbits and clocks)

Excess PhaseFull SpectrumAbel Inversion

1-D VarMoisture Correction

Fiducial data

Atmospheric processing

Excess PhaseProcessing Abel Inversion

Combinationwith other data

Ionospheric processing

Profiles

Real time Task Scheduling Software

atmPhs atmPrf(S4)

ionPrf(EDPs)

ionPhs(L1,L2 amp/phase)

bfrPrf

podTec (TEC)scnLv1 (S4)tipLv1 (Radiance)

wetPrf

CDAAC File types in RED

Best Effort Monitoring

• Monitor the system regularly throughout the day M-F 8am-8pm

• CDAAC Ops team monitors the system 3 times per day on weekends and holidays

• Available by email and cell phone

• CDAAC reliability estimated > 99.5%

• Near Real-Time- COSMIC- SAC-C and C/NOFS

• Post-Processing (2007.3200-2009.2650)- GPSMET, GPSMET-AS, CHAMP, SAC-C, GRACE, COSMIC- Will attempt to understand 3 “noisy” CHAMP months (2007.3200)

• Re-Processing (2009.2650)- Published new CHAMP re-processing- COSMIC re-processing finished, published soon- Then start re-processing for GPSMET, GRACE, TerraSAR-X, SAC-

C, Metop/GRAS, C/NOFS

CDAAC Data Processing Status

• Output file saved from each process• Hard failures logged in cosmic_badprf table• Most products have BAD flags in files and database• Some product files include error characteristics

• LEO Orbit Determination (Post-fit residual test, Orbit overlap test)• Neutral atmospheric excess phase files (SNR and phase noise tests)• Neutral atmospheric profiles (Bending angle noise and bias tests, checks for L2

quality, presence of scintillation, gross checks on N/T/P)• Absolute TEC data arcs (cycle slip detection/correction, outliers)• Electron Density Profiles (under development)• Scintillation Indices (under development)• TIP Night-side Radiances (BAD flag for incomplete integration time, motor not at

proper destination, ill-defined sensitivity)

COSMIC Data Product QC

• To be released by late November 2010• RPM-based install system• Improved data access, website, DB interface, monitoring,

configuration management, archiving, documentation• All other packages updated to latest versions: perl,

postgres, apache, Bernese, ..• Real-time clock subsystem• Improved excess phase for SD processing• Use of higher resolution real-time and post-processing first

guess models (GFS high resolution and ERA-40 interim)

CDAAC 3.0 Status

• Held Oct 27, 2010

• What CDAAC development efforts are required for COSMIC and other missions? How much effort?

• How do we know our CDAAC GPS RO data processing is optimal?

–Review of algorithms and software internally and via publication–Comparison of results (POD, BA, N, T, WV) with other data

processing software and investigate and understand differences (i.e. more detailed GRAS study)

–Comparison with correlative data, models?

CDAAC Review

CDAAC System changes

Better system for managing code for production/researchSimplify addition of new missionsnames in CDAAC codeRestructure ROAMRefactor genXphaseAdd more test suitesClean up unused codeContinue to add missionsImproved QC, bad flags, error estimatesMulti-GNSS capability, new observables!Low latency processingBetter systems managementNeed to develop CDAAC Level0-Level1 processing modulesImproved monitoring scripts

GNSS Processing • Add GNSS capability, new observables• Improve NRT ZTD

- Compare algorithm with Suominet

• Improve NRT GNSS clock estimation- 5-sec GNSS clocks

• Improve NRT and PP LEO POD (to 0.1 mm/sec)– Process data from both POD antennas

– Compute/Apply PCV maps

– Implement SLR processing for validation

• Improve excess phase processing- Review algorithms, fix bugs,

- 5-sec GNSS clocks

- refactor

• Perform additional validation studies

Neutral Atmospheric Inversions

• Restructure ROAM• Add GNSS capability, new observables• Improve wave optics processing

– looking for better filtering approaches– looking for alternative methods (like recently introduced WDF)

• Improve bending angle optimization- testing methods with reduced weight of climatology- validation by independent data sources in the stratosphere

• Improve 1DVAR retrieval code and documentation• Perform additional validation studies (e.g. integrate ROPP package, RO-

Trends+)- Requires investigation and understanding the sources of the differences

1) Phase Connection

2) Bending AngleGeneration (WO,GO)

3) Bending AngleCorrection, Connection

4) Bending AngleOptimization, Inversion

atmPhs

conPhs

benPrf

bcnPrf

atmPrf

1) Mission-dependent; 2-4) Mission-independent.

1) Input positions, velocities, raw phase and amplitude,clim. model.Processing removal of NDM, connection of the phase,down-sampling to one rate. Output positions, velocities, connected phase,amplitude, HSL, lat, lon, height of TP.

2) Input output from 1Processing retrieval of WO (Phase Matching and FSIor CT2) and GO bending angles for L1, L2, (L5)Output GO and WO bending angles for L1,L2,(L5)

3) Input output from 2Processing ionospheric correction (incl. additionalsmoothing of L4), connection of GO and WO bendingangles.Output ionosphere free connected bending angles

4) Input output from 3, clim. model, atm. model.Processing optimization of bending angles,inversion of N,T,P.Output N,T,P.

Comparison of COSMIC retrieved temperatures to ECMWF for Different optimizations of BA 2009.015-045

Current CDAAC approach:- clim. fitted to obs. BA- mixed with weighting function - WF different for different occs.- on average fitted clim. >50% above h~45 km

Tested approach:- obs. BA smoothed with window increasing from 1.5 km at h=30 km to 15 km at h=60 km- clim. >50% above h=60 km for all occs.

• Absolute TEC uncertainty under investigation- DCB estimation, Code/phase leveling uncertainty

• Electron Density Profiles- Improving EDPs by using information on horizontal gradients with DA

• Improve scintillation products• Add GNSS capability, new observables• Perform additional validation studies

- With Paul Straus of Aerospace, JPL

Ionospheric Processing

1-2 TECu uncertainty can be ignored for ground based GPS or low elevation LEO GPS observations. But it might have problem if we want to study the plasmasphere using COSMIC sTEC [eg,2006.230-245]

TEC Precision from Collocated TracksCOSMIC (2006.230-245)

Comparison of the latitude and altitude Ne and its retrieval error from simulation

LT=13

Manmade plasma cave disappears in DA retrieval.

Data assimilation retrieval

Processing NRT data from MOOs at CDAAC

• development, testing, monitoring of RTS downlink• decoding and formatting binary level 0 science payload data.

Effort depends on payload architecture• system modification, configuration and management• data processing modifications• testing and validation• NRT operations and monitoring, • post-processing, archival, maintenance • program management

• Recent experience with C/NOFS and SAC-C valuable

C-2 Operational Data Processing Requirements

• Reliable and low latency input data streams• Primary and Backup DPC (Data Processing Center)• Development system (GNSS capable)• Staging system to test processing changes• Configuration management of systems• Communication access between systems (DPCs, SOCC,

RTSs). Increased IT Security?• 24/7 monitoring of near real-time operations• Maintenance and on call technical support• Operator Training• Secondary Science payload processing• Post-Processing

Configuration Management for C-2

Cfengine controls• Log rotation• Shared file systems• Processing Daemons• Software integrity• Fail-over between I/Os and processing clusters• Disk space/cleanup• Syncing

SVN configuration repository

Cfengine ConfigurationControlMachine

Primary DPC

Backup DPC

StagingSingle string

DevelopmentSingle string

Post-Processing

C-2 DPC Options with CDAAC

Option A Option B

Primary NOAA (single string hw ~$100K) NOAA (single string hw ~$100K)

Hot Backup UCAR (single string hw ~$100K)UCAR staffing (TBD)

TDPC (single string hw ~$100K)TDPC staffing (TBD)

Staging NOAA (single string hw ~$100K) NOAA (single string hw ~$100K)

Monitoring 24/7 at NOAA (~5 FTE)Best effort at UCAR (TBD)

24/7 at NOAA (~5 FTE)24/7 at TDPC (~5 FTE)

R&D UCAR (single string hw ~$100K)UCAR staffing (TBD)

UCAR (single string hw ~$100K)UCAR staffing (TBD)

On Call Tech Support

UCAR staffing (TBD) UCAR staffing (TBD)

Post-Processing

UCAR (single string hw ~$150K) UCAR (single string hw ~$150K)

CDAAC R&D Schedule for C-2

• June 30th 2010 meeting with NOAA/OSDPD (Kathy Kelly) - Review of GPS RO processing at CDAAC- HW/SW data processing architecture at NSOF- Held additional telecon- Working to test CDAAC within NSOF architecture - Providing info for NOAA CLASS archive

• Oct 14th 2010 meeting with AF/Aerospace - Review of CDAAC and SW architecture- Strong interest in using CDAAC to process space weather

payloads- Additional meetings in planning

• F-7/C-2 JPMO Meeting Dec 7-8, 2010- Discuss data processing issues

Recent Activities

• Investigated structural uncertainty of RO in LT. Published in JGR• Comparing different methods of estimation of the ABL depth from RO.

Investigating global variability. Publishing results soon.• Detection of the sporadic E clouds from RO. In press at GRL• Investigation of the response of the UTLS in terms of gravity waves

and tidal variations from COSMIC data to the sudden stratospheric warming event in 2009

• COSMIC EDPs used to estimate ionosphere High Transition Heights (HTH). In press at JASTP

• Investigated global ionospheric response observed by COSMIC Satellites during the January 2009 Stratospheric Sudden Warming Event. In press at JGR-SP

• Submitted data assimilation paper to improve EDPs. In revision at JGR

Recent RO Research and Science Apps

• Global Evaluation of Radiosonde Water Vapor with RO data, Published in J of Remote Sensing

• Evaluation of AMSU and AIRS with RO• Impacts of GPS radio occultation data assimilation on regional weather

predictions. Published in GPS Solutions• Working on Round-2 of RO-Trends inter-agency comparison• UCAR funded by EUMETSAT to evaluate Metop/GRAS RO data. Final

Report due in early Nov

Recent RO Research and Science Apps