level 2 ocean salinity l2os v610 & ott post-processor
DESCRIPTION
Level 2 Ocean Salinity L2OS v610 & OTT post-processor. 5 July 2013. ARGANS & SMOS L2OS ESL. L2OS v610 status. Significant changes from v600 OTT post-processor OTT running average tracking L1 drift, updated daily in DPGS & reprocessing no more compareTBs! TEC from Stokes 3 - PowerPoint PPT PresentationTRANSCRIPT
SMOS L2 Ocean Salinity
www.argans.co.uk
Level 2 Ocean Salinity
L2OS v610 & OTT post-processor
5 July 2013
ARGANS & SMOS L2OS ESL
SMOS L2 Ocean Salinity
www.argans.co.uk
Significant changes from v600
• OTT post-processor• OTT running average tracking L1 drift, updated daily in DPGS & reprocessing
• no more compareTBs!
• TEC from Stokes 3• impact on high TEC descending orbits; ignore L1c IRI TEC
• Multi-threading• > 2 x speed increase
• Simplified slim-line UDP; extended DAP• remove unnecessary flags & fields, make it easy for users to extract good quality data
L2OS v610 status
SMOS L2 Ocean Salinity
www.argans.co.uk
Real-time salinity retrieval requires near real-time OTT to track short term drift: while processing L1c,
L2OS v610 extracts deltaTBs for South Pacific region and writes them to AUX_DTBXY_. The OTT post-
processor runs once per day, reads AUX_DTBXY_ and a set of current deltaTBs (from AUX_DTBCUR),
writes a new set of OTTs, and updates AUX_DTBCUR. The L2OS processor always uses the latest set of
OTT.
deltaTBs(DTBXY_)
current deltaTBs
(DTBCUR)
OTT post-
processor
OTTsL2OS
processorL1c
L2OSUDP & DAP
OTT post-processor
By-products:
• daily drift statistics available in AUX_DTBCUR.HDR for analysis & monitoring
• other (non-OTT) regions may be specified (in AUX_CNFOSF) for monitoring (output in DTBXY_)
SMOS L2 Ocean Salinity
www.argans.co.uk
• L2OS v610for each region in AUX_CNFOSF
apply region snapshot filter // use snapshot if enough measurements & all within region
if OTT region
apply OTT snapshot filter // ignore if TBs out-of-range, RFI, L1c error
apply OTT grid point filter // not ocean, hi/lo WS, ice
if count(snapshots) > Min_Snapshots & count(gridpoints) > Min_Grid_Points
if OTT region apply OTT measurement filter // ignore sun_point, rfi_tails
deltaTB[region] = use forward model to compute deltaTBs & statistics
write deltaTB & statistics to AUX_DTBXY_
• OTT post-processormerge all AUX_DTBXY_ from previous day with current AUX_DTBCUR
discard old sets of deltaTBs (last-in/last-out)
compute & write new OTT using current set of deltaTBs
write current set of deltaTBs & statistics to new AUX_DTBCUR
OTT post-processor overview
SMOS L2 Ocean Salinity
www.argans.co.uk
for each selected measurement
map xi,eta onto x,y grid
count[x,y] += 1
sumDeltaTB[x,y] += TBsmos – Tbmodel
sumDeltaRA[x,y] += (TBsmos – Tbmodel) / ra
sumDeltaRA2[x,y] += ((TBsmos – Tbmodel) / ra)2
flags[x,y] |= selected L1c & L2OS flags
for x = 0 to 128
for y = 0 to 128
n = count[x,y]
write n
mean_deltaTB[x,y] = sumDeltaTB[x,y] / n
write mean_deltaTB[x,y]
stdra[x,y] = sqrt(((n * sumDeltaRA2[x,y] – sumDeltaRA[x,y]2) / (n * (n – 1)))
write stdra[x,y]
write flags[x,y]
AUX_DTBXYcount[x,y]deltaTB[x,y]stdra[x,y]flags[x,y]
Statistics for fov, affov, eaffov, & filtered by flags
Flags for off-line analysis:
OTT post-processor: L2OS v610 algorithm
SMOS L2 Ocean Salinity
www.argans.co.uk
deltaTB referencemax_orbits (nominally 10)
quality thresholds (RFI, L1c errors)
ott_strategy (mean/gaussian)
ott_validity_start (first/mean/last)
List of regionsName, IDtype (OTT/REG)orbit dir (A/D/?)lat/longsnapshot thresholdsgrid point thresholds
AUX_CNFOSF <OTTPP> section
Source of geophysical reference parameters for computing deltaTBs:
0 = climatology, L1c TEC & ECMWF priors
1 = model 1 retrieved parameters
And new filters:
OTT_SNAPSHOT_FILTER
OTT_REGION_FILTER
OTT_STATS_FILTER
New OTT post-processor section in AUX_CNFOSF
SMOS L2 Ocean Salinity
www.argans.co.uk
n x regions:IDsnapshot start/stop timessnapshot start/stop ID
for 3 modelsfor 8 polarisations
4 x statisticscount[x,y]deltaTB[x,y]stdra[x,y]flags[x,y]
AUX_DTBXY_ datablock
Statistics for FOV, AFFOV, EAFFOV, & filtered by flags[x,y]: mean, median, min, max, std.
OTT post-processor components: AUX_DTBXY_
SMOS L2 Ocean Salinity
www.argans.co.uk
read all deltaTB orbits from AUX_DTBCUR // ignore
ottA/D
for each AUX_DTBXY_
if region = South Pacific OTT region & good quality
read deltaTB for this orbit
// keep n ascending & n descending deltaTBs...
trim deltaTB // discard oldest first
ottA = average(deltaTB, ascending)
ottD = average(deltaTB, descending)
for model = 1 to 3
write ottA(model) & ottD(model) to AUX_OTT1/2/3
deltaTB += ottA
deltaTB += ottD
write deltaTB to new AUX_DTBCUR // including ottA/D
Options:
number of deltaTB orbits (nominally 10+10)
mean or weighted mean (nominally mean)
filter DTBXY by quality
OTT post-processor: OSCOTT algorithm
SMOS L2 Ocean Salinity
www.argans.co.uk
Header contains quality information for each region:
Grid point quality informationtotal grid points in region, number of grid points used after filteringnumber of grid points:
too near to land/coast; rain/ice contaminationmissing ECMWF data; low WS (<3m/s), high WS (>7m/s)
Measurement quality informationtotal measurements in region; number of measurements used after filteringnumber of measurements flagged:
as sun point, sun tails, with sun/moon/galactic glintcontaminated by RFI (by L1 or L2), L1c error flags
Quality information is from forward model 1 only:
used by OSCOTT to select ‘good’ sets of deltaTBs.
OTT post-processor components: AUX_DTBXY_
Snapshot quality informationtotal snapshots in region, number of snapshots used after filteringnumber of XX/YY/XXY/YYX snapshotsnumber of snapshots with each of 4 L1c error flags setnumber of snapshots with TB out-of-range, high std/std_Stokes3/4
SMOS L2 Ocean Salinity
www.argans.co.uk
Header contains quality information for each region:
Grid point quality informationtotal grid points in region, number of grid points used after filteringnumber of grid points:
too near to land/coast; rain/ice contaminationmissing ECMWF data; low WS (<3m/s), high WS (>7m/s)
Measurement quality informationtotal measurements in region; number of measurements used after filteringnumber of measurements flagged:
as sun point, sun tails, with sun/moon/galactic glintcontaminated by RFI (by L1 or L2), L1c error flags
Quality information is from forward model 1 only:
used by OSCOTT to select ‘good’ sets of deltaTBs.
OTT post-processor components: AUX_DTBXY_
Snapshot quality informationtotal snapshots in region, number of snapshots used after filteringnumber of XX/YY/XXY/YYX snapshotsnumber of snapshots with each of 4 L1c error flags setnumber of snapshots with TB out-of-range, high std/std_Stokes3/4
SMOS L2 Ocean Salinity
www.argans.co.uk
AUX_DTBXY_ datablock
m x AUX_DTBXY + extra sets:set type (L1c, OTT...)region IDorbit direction (A/D)orbit polarisation (D/F)snapshot start/stop timessnapshot start/stop IDL1c filename
AUX_DTBCUR datablock
OSCOTT
m x AUX_DTBXYn x regions:
IDsnapshot start/stop timessnapshot start/stop ID
for 3 modelsfor 8 polarisations
4 x statisticscount[x,y]deltaTB[x,y]stdra[x,y]flags[x,y]
for 3 modelsfor 8 polarisations
4 x statisticscount[x,y]deltaTB[x,y]stdra[x,y]flags[x,y]
OTT post-processor components: AUX_DTBCUR
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor components: AUX_DTBCUR
Header contains quality information statistics for ascending & descending OTTs generated by
the last OTT post-processor run:
Ascending OTT quality informationcount of orbits used to make OTTsnapshot start/stop timesfor 3 polarisations
for 8 polarisations4 x statistics (same format as AUX_DTBXY_ statistics(count, mean, median, std for fov, affov, eaffov, & filtered)
Descending OTT quality information(same fields as above)
List of L1c datasets
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor components: AUX_DTBCUR
Header contains quality information statistics for ascending & descending OTTs generated by
the last OTT post-processor run:
Ascending OTT quality informationcount of orbits used to make OTTsnapshot start/stop timesfor 3 polarisations
for 8 polarisations4 x statistics (same format as AUX_DTBXY_ statistics(count, mean, median, std for fov, affov, eaffov, & filtered)
Descending OTT quality information(same fields as above)
List of L1c datasets
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor preliminary results
AUX_DTBXY_ x 10
AUX_OTT1F_
deltaTB, XX pol, ascending
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor preliminary results
AUX_DTBXY_ x 10
AUX_OTT1F_
Median = 0.58 K
Histograms of deltaTB in XX FOV, ascending orbits, AUX_DTBCUR 20110501
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor preliminary results
AUX_DTBXY_ x 10
AUX_OTT1F_
std/ra, XX pol, ascending
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor preliminary results
AUX_DTBXY_ x 10
AUX_OTT1F_
Histograms of std/ra in XX FOV, ascending orbits, AUX_DTBCUR 20110501
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor preliminary results
Comparing OTT from DPGS with OTTPP: re-sampled grid
DPGS OTT (20110504) OSCOTT (20110501) delta = OSCOTT – DPGS
Differences along border-belt-suspenders & tails
XX
YY
XX
YY
SMOS L2 Ocean Salinity
www.argans.co.uk
• Which statistics should be used to select/reject DTBXY deltaTB data?• Current implementation has thresholds for maximum % measurements contaminated
by L1 RFI, L2 RFI, or with L1c error flags set
OTT post-processor issues: good AUX_DTBXY
SMOS L2 Ocean Salinity
www.argans.co.uk
• What is the optimal running average window?• Averaging many sets of deltaTBs gives better quality OTTs but with a slower response to
rapid TB drift
OTT post-processor issues: averaging window
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor issues: priors or retrievals?
std(priors) std(model 1 retrievals)
deltaTB(priors-model1)
deltaTBs are computed as the difference
between forward model TBs and L1c TBs
Should we run the forward model using
priors (SSS from climatology, SST/WS
from ECMWF, TEC from L1c or A3TEC)?
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor issues: xi/eta grid sampling
-1.0 xi 1.0
1.0
eta
-1.0
-0.7 xi 0.7
0.4
eta
-0.7
v600 v610
v600: 129x129 (xi ±1.0, eta ±1.0): sampling interval 0.016
v610: 129x129 (xi ±0.7, eta -0.7 to 0.4): sampling interval 0.011 in xi, 0.008 in eta
What grid sampling interval is best for OTTs?
Grid point density varies within snapshot – highest at leading edge of AFFOV
Typical total 5600 grid points, >100 along leading edge (±0.25 xi): ~0.005 sampling?
Blackman pixels ~0.04 diameter: with pixel overlap 1.4, sampling interval 0.028
Small sampling interval leads to higher noise in OTTs.
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor issues: merge OTTs?
long integration time short integration time delta
OTTs are similar – should they be merged?
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor issues: merge OTTs?
long integration time short integration time
OTTs are similar – should they be merged to reduce noise
(especially for short integration time)?
SMOS L2 Ocean Salinity
www.argans.co.uk
OTT post-processor issues: merge OTTs?
XXY YYX delta
OTTs are similar – should they be merged?
SMOS L2 Ocean Salinity
www.argans.co.uk
L1PP v600 20110505T040402
tecu
tecu
latitude
latitude
L1cTEC & A3TEC
Cross-verification between Matlab breadboard & v610: delta = breadboard-v610
TEC from Stokes 3: descending orbit, high TEC
SMOS L2 Ocean Salinity
www.argans.co.uk
L1PP v600 20110505T040402
tecu
latitude
L1cTEC & A3TEC
TEC from Stokes 3: descending orbit, high TEC
Tuning:
decreased sigma prior, increased gaussian smoothing, added outlier filtering, TEC height changed from 450 to 400 km
SMOS L2 Ocean Salinity
www.argans.co.uk
L1PP v600 20110505T040402
tecu
latitude
L1cTEC & A3TEC
TEC from Stokes 3: descending orbit, high TEC
L2OS retrieved TEC follows A3TEC better than L1c TEC
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v601 20110505T040448
tecu
latitude
L1cTEC & A3TEC
TEC from Stokes 3: descending orbit, high TEC
L1OP v601 using consolidated VTEC
SMOS L2 Ocean Salinity
www.argans.co.uk
Ascending orbit with low TEC – noisy A3 TEC retrieval, but good match with L1c
L1PP v600 20100802T153815
tecu
L1cTEC & A3TEC
TEC from Stokes 3: ascending orbit
latitude
SMOS L2 Ocean Salinity
www.argans.co.uk
L1cTEC & A3TEC
TEC from Stokes 3: ascending orbit
L1PP v600 20100802T153815
tecu
latitude
Tuning:
decreased sigma prior, increased gaussian smoothing, added outlier filtering, TEC height changed from 450 to 400 km
SMOS L2 Ocean Salinity
www.argans.co.uk
L1cTEC & A3TEC
TEC from Stokes 3: ascending orbit
L1PP v600 20100802T153815
tecu
latitude
L2OS retrieved TEC follows A3TEC better than L1c TEC?
SMOS L2 Ocean Salinity
www.argans.co.uk
L1cTEC & A3TEC
TEC from Stokes 3: ascending orbit
L1OP v601 20100802T153855
tecu
L1OP v601 using consolidated VTEC
latitude
SMOS L2 Ocean Salinity
www.argans.co.uk
Faraday rotation depends on TEC altitude (not only on TEC amplitude).
Tomography using Stokes 3:first analysis shows mean altitude of electronic layer is about 390 kmelectron density not located at same altitude along the orbittomography depends on TEC prior
TEC from Stokes 3: altitude of TEC
Estimated
Prior
SMOS L2 Ocean Salinity
www.argans.co.uk
Changing TEC altitude in v610 from 450 to 400km
TEC from Stokes 3: altitude of TEC
SMOS L2 Ocean Salinity
www.argans.co.uk
Motivation
• run OTT post-processor per-day during reprocessing (same as DPGS)• average L2OS run time in DPGS/reprocessing is 2.5 – 3 hours (TBC)
• x3 speed-up with 4 threads would match SM performance (Antonio de la Fuente)
• 15% faster CPU clock in reprocessing platform
• L2OS maximum run time per half-orbit is ~370 minutes (Pacific Ocean)(on 5 year old original specification server)
• L2OS v610 uses OpenMP to multi-thread time-consuming loops• Pacific Ocean test orbit results (includes 25 minutes I/O) :
3 threads: 160 minutes (5.3G) – 2.3 x speed-up
4 threads: 144 minutes (5.1G) – 2.6 x speed-up
• average run-time (20110416) with 4 threads 124 minutes (max 166, min 77)
Multi-threading L2OS
SMOS L2 Ocean Salinity
www.argans.co.uk
Motivation
• UDP includes too many flags & fields – confusing• make it easy for users to extract good quality data from UDP
• move unnecessary UDP flags & fields to DAP
• UDP is for oceanographers, not for analysis
• make the DAP easier to use• variable length grid point records in v600 makes reading DAP grid point data very slow
• restructure DAP into a block of all grid point data (fixed size records), followed by a block of all measurement data (variable size records)
• users/ESL can read just the grid point data & ignore measurements (measurement data can still be read slowly if needed)
Summary
• new UDP only contains geophysical outputs from L2OS processor
• simplified UDP quality index & quality flags for each model
• no compatibility with existing UDP/DAP readers• requires new versions of SDV, GMT, Beam toolbox & bespoke tools
Simplified UDP & extended DAP
SMOS L2 Ocean Salinity
www.argans.co.uk
Improving the UDP
n x grid points:
Grid point dataID, latitude, longitude, footprint, time x_swath, distance to coast
Science flags (4)near landsuspect rain, ice (Acard), RFI
Geophysical retrievals (2 x 7)
SSS1, SSS2, SSS3, AcardWS, SST, TEC,
sigmas on all retrieved parameters
Product confidencequality index for SSS1/2/3/Acard
Product confidence flags (4 x 3)
High/medium/low for SSS1/2/3/Acard
RFI probabilitycounters for DGGRFI
v600 (176 flags & 57 fields) v610 (16 flags & 29 fields)
n x grid points:
Grid point dataID, latitude, longitude, footprint, time
Science flags (4 x 30)land/sea/coast, rainice (4 flags: climatology, ECMWF, TBs,
Acard)high/low wind/SSS/SST, sea state
Geophysical priorsWS & SST from ECMWF
Geophysical retrievalsSSS1/2/3/Acard & sigmas
TB42.5 (8)modelled BTs & sigmas at surface & antenna
Control flags (4 x 28)range, sigma, chi, chi2, sun/moon/gal glintmax iterations, low/min #measurements,too many outliers, convergence limit, poor retrieval, poor geophysical, RFI suspect, RFI prone
Product confidence (34)
chi, chi2, #iterations, quality for SSS1/2/3/Acardvalid, border, affov, tails, sun/moon/gal glint, iceRFI probability counters for DGGRFIx_swath
SMOS L2 Ocean Salinity
www.argans.co.uk
Fg_high_quality
High quality flag is set if Fg_ctrl_poor_retrieval = 0 & Fg_ctrl_poor_geophysical = 0 & Dg_af_fov > 130 (grid points with lots of measurements in the AFFOV). Users filtering with this flag will get the best quality, suitable for cal-val.
Fg_medium_quality
Medium quality flag is set if Fg_ctrl_poor_retrieval = 0 & Fg_ctrl_poor_geophysical = 0. Users filtering with this flag will get all the high quality retrievals, plus those with Dg_af_fov <= 130. This set is suitable for oceanography, , since we know there were no retrieval or geophysical problems.
Fg_low_quality
Low quality flag is set if Fg_ctrl_poor_retrieval = 0. Users filtering with this flag will get all the high & medium quality retrievals, plus those without any retrieval problems (but there may be geophysical problems). This is the lowest quality we would recommend to users: these retrievals have passed the chi2P/max iterations/convergence tests, so may be ok.
Proposed UDP v610 quality flags (per model)
SMOS L2 Ocean Salinity
www.argans.co.uk
n x grid points:
ID, lat/longpriors & sigmas (4 x 14)
posts & sigmas (4 x 14)
oor flags (4 x 27)
fields (8)
m x measurements:
flags (4 x 30)
fieldsdeltaTB (x4)
TBgal (x2)
n x grid points:
ID, lat/longpriors & sigmas (12)
posts & sigmas (24)
science flags (4 x 30) control flags (4 x 28)
oor flags (4 x 27)
fields (45)
n x grid points:
IDm x measurements:
snapshot IDxi, etaflags (4 x 30)
fieldsdeltaTB (x4)
TBgal (x2)
Fixed block size grid pointsVariable
block size grid points
Variable block size
measurements
Improving DAP readabilityv600 v610
Extra fields & flags from
v600 UDP
v600 priors, posts & sigmas (112 fields) allow for 7 parameters per model – only 5+4+3+2 parameters retrieved: 50% unused.
For a large DAP (eg South Pacific) with typically 150k grid points & 120 measurements/grid point:
v600 = 360M (70M grid points + 290M measurements)
v610 = 400M (50M grid points + 350M measurements)
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v601 & L2OS v610
32 ascending & 25 descending L1OP v601 orbits processed using L2OS v610, deltaTBs extracted from AUX_DTBXY_
Orbits don’t intersect OTT region, so deltaTB statistics extracted for South Pacific test region 122 (30S to 0, 160W to 120W)
Some correlation, but:climatology for 122OTT shifted (late)
v601 reduces long term drift?
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v601 & L2OS v610: RFI test orbit 20100709T034234
L1OP v504 REPR L1OP v601Fm_L1c_RFI set mostly by Fm_RFI_XX & Fm_RFI_YY
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v601 & L2OS v610: RFI test orbit 20100709T034234
L1OP v504 REPR L1OP v601(using L2OS v600 L1OP v504 REPR OTTs)
For RFI (& other) tests we need to ensure OTTs match L1c...
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v504 & v601: RFI test orbit 20100802T153855
L1OP v504 REPR L1OP v601Region ID 126 - Equatorial Ocean ±10: small islands?
>1K delta in mean/median
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v504 & v601: RFI test orbit 20100802T153855
L1OP v504 REPR L1OP v601 delta = v504 – v601
v601 has larger spatial bias than v504 – why?
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v504 & v601: RFI test orbit 20110128T125103
L1OP v504 REPR L1OP v601Region ID 126 - Equatorial Ocean ±10: RFI @ 5N
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v504 & v601: RFI test orbit 20110128T125103
L1OP v504 REPR L1OP v601 delta = v504 – v601
SMOS L2 Ocean Salinity
www.argans.co.uk
L1OP v504 & v601: RFI test orbit 20110128T125103
L1OP v504 REPR L1OP v601Improved Stokes 3 & 4
SMOS L2 Ocean Salinity
www.argans.co.uk
Verification scheduled August/September
• requires at least 1 month reprocessed L1c to verify (May 2011)• OTT post-processor – run at ARGANS
• TEC from Stokes 3 – run on G-POD?
• multi-threading
Remaining work before delivery
• Simplified slim-line UDP; extended DAPneed specification of weighted distance-to-coast (TBC)
• New galactic map LUTs?
Schedule• delivery mid-September, FAT early October?
L2OS v610 status summary