a collocation service for in situ and remotely sensed measurements

The National Center for Atmospheric Research is operated by the University Corporation for Atmospheric Research under sponsorship of the National Science Foundation

MARCDAT–III, ESRIN, Frascati, Italy 1

A Collocation Service for In Situ and Remotely Sensed Measurements

Steven J. WorleyNational Center for Atmospheric Research

Andrew W. BinghamPO.DAAC JPL NASA

Zaihua JiNational Center for Atmospheric Research

5/2-6/11



Topics

1. Motivation 2. Web Service and Use Cases3. Infrastructure and Testing Results4. Record Content for Web Service5. Next Steps6. Request to the Community7. Conclusions

5/2-6/11



Motivation

Support new research opportunities• Compare and Contrast Independent Observing

Systems (in situ .vs. remote)• Reveal biases and systematic errors – in both

• Leads to improved algorithms and QCCommunity benefits

• More accurate independent data products• More accurate combined synergistic analyses• Easy to get research started in this domain

5/2-6/11



Motivation

Data Management Benefits• 24x7 (nearly) availability• Most current version of data are always

available• Easy to serve a variety & many clients

5/2-6/11



Web Service and Use Cases

5/2-6/11

ICOADSIn situ

Remotely Sensed DataSatellite

Data Request Protocol

Matched DataProtocol

Value Added DataSatellite + In SituInternet

Internet

Enhance a Satellite Archive- Phase I

Client

Server




5/2-6/11

ICOADSIn situ



Internet

Internet


Value Added DataIn Situ +Satellite

Enhance an In situ archive- Phase II

Client

Server




5/2-6/11

ICOADSIn situ



Internet



Enhance an individual’s archive- Phase III

Client

Server

Server



Use Case Development

Phase I system development vision

1. Create an ICOADS Server at NCAR 2. Install Client software and create value-added

datasets at PO.DAAC and/or another agency3. Tune and ensure system scaling4. Open service worldwide

5/2-6/11



Infrastructure and Testing Results

ICOADS prototype server• MySQL DB, 300M+ IMMA records

• R2.5 + 2008-April 2011, updated monthly• Hierarchical tables enable fast response for L2

dataSatellite L2 data – orbit swath, from PO.DAAC• AMSRE, SST• ASCAT, Wind

5/2-6/11




5/2-6/11

Terms:Orbit – One Earth Revolution (blue)Scan – Perpendicular Sampling Line (green)Footprint – One Earth Surface Sample (grey)

AMSRE ASCATOrbit Time (min) ~ 95 ~ 95Scans / orbit 4193 3259Footprints / scan 243 82Foot prints / orbit 1018899 267238




Challenges for Phase I• Create value-added satellite datasets

• Fast Internet-based DB queries

Influencing Factors1. Data packet size2. DB response speed

• Impact of time and space ranges on queries3. Internet transfer rate using 'wget' with HTTP

protocol

5/2-6/11



Infrastructure and Testing ResultsHow much data to send in each packet?

5/2-6/11

FP 1S 50S 100S 200S 500S Orbit0

5001000150020002500

Orbit Time Completion By Data Packet Size

AMSRE (243 FP/Scan)ASCAT (82 FP/Scan)

Packet Size (FP, Scans, Full Orbit)

Min

utes

1S 50S 100S 200S 500S Orbit0

20406080

Orbit Time Completion By Data Packet Size

AMSRE (243 FP/Scan)ASCAT (82 FP/Scan)

Packet Size (Scans, Full Orbit)

Min

utes

1 Foot Print per Data Packet is not viable• AMSRE 34 hours to complete an orbit• ASCAT 10 hours

1 Scan (or more) per Data Packet is viable• AMSRE 65-46 minutes• ASCAT 40-19 minutes

Hit upper limit > 200S per packet, AMSRE• Server time out (receipt and

processing time too long)



Infrastructure and Testing ResultsTime and Space Range Impact on DB performance• Locally run – no Internet, LAN

5/2-6/11

0.2 0.6 1.0 1.4 1.80

10

20

30

40

50

AMSRE, 1 Scan Req., Fixed Space 25 km

Delta Time (Hours)

Orb

it Co

mpl

etion

Tim

e (M

inut

es)

10 20 30 500

10

20

30

40

50

60

AMSRE, 1 Scan Req., Fixed Time 1.0 hr

Delta Space (km)

Orb

it Co

mpl

etion

Tim

e (M

inut

es)

Time to completion can double as ranges grow

Space (km)

Time (hr) Completion Time (min)

10 0.2 22

20 0.6 29

30 1.4 48

50 1.8 72




5/2-6/11

1 1000

10

20

30

40

50

60

70

Mean Orbit Completion Time, Network & Scan Impacts

AMSRE, PO.DAAC -> NCAR AMSRE, BLDR -> NCAR AMSRE, @ NCARASCAT, PO.DAAC -> NCAR ASCAT, BDLR -> NCAR ASCAT, @ NCAR

Number of Scans per Request

Tim

e (m

in)

•ASCAT faster than AMSRE – less data•Number of scans per request – little impact at NCAR, LAN•100 Scans better than 1 for Internet, PO.DAAC to NCAR ~ 20% improvement•Internet PO.DAAC to NCAR better than local Boulder




5/2-6/11

5 10 15 20 25 30 35 400

20

40

60

80

100

120

Simultaneous Mulit-orbit Processing TimesAMSRE, 1 Scan Request AMSRE, 100 Scan RequestAMSRE, 1 Scan Request, JPL - NCAR

Orbit Count

Tim

e to

Com

pleti

on (M

inut

es)

Fixed Time and Space Range (1 hr. & 25km)Good Scaling, relative to orbit period, to 40 simul.Expected relationship from PO.DAAC to NCAR

Internet-based Tests




How to manage performance?

• Plan to pair Perl client software with data process software at the client location• Via system calls• Client functions: form packets, send to server,

receive results from server • Use dedicated web and DB server

• Optimized for the problem

5/2-6/11



Record Content for Web Service

Client provided data elements:• Observed Satellite data

• Date & time, latitude & longitude• Treat as point measurements

• Selectable specifications• Time range• Space range• Parameter to be matched • ICOADS trimming (standard, enhanced, none)• Code Indicators of additional variables (TBD)

OR• Code to request IMMA Core + ICOADS attm – standard data

5/2-6/11



Server data returned to client• All records, with the requested parameter, in time and space

range• Record content (TBD):

• Resend client record for identification• Selected data fields from ICOADS IMMA

OR• ICOADS Core and ICOADS attm

• All the standard data fields• All the record provenance information (DCK, SID, PT, etc)

5/2-6/11

Record Content for Web Service



Next Steps

Write a white paper based these tests, ideas, and your comments & recommendations

Seek funding to support:• Development, implementation, testing, and staffing

at Server and Client locations• Dedicated DB and web server

• Expand capacity to handle more simultaneous orbits• Design a user web interface for manually specified

Client requests

5/2-6/11



Requests to the Community

• Should we develop this service?• Are there technological or system

requirements we have overlooked?• Are there additional data requirements?• Would you use such a service?• Would you use datasets produced from this

service?

5/2-6/11



Conclusions

• Demonstrated the potential to use a web service to bring in situ and remotely sensed data together

• The Phase I development would instantiate an ICOADS DB server and client software package

• Phase I would lead to a matched in situ and satellite archive dataset(s)

• There is more work to be done formulating the data to be exchanged

5/2-6/11

a collocation service for in situ and remotely sensed measurements

Documents

atmospheric researchandrew

italythe national center

italy2the national center

italy3the national center

italy4the national center

university corporation

worleynational center

use cases52