06 ncar rda integration cram · rda)usage) 2014: • 17+pb)virtual)processing) •...
TRANSCRIPT
![Page 1: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/1.jpg)
Integra(ng Globus into the NCAR Research Data Archive User Services
Thomas Cram Doug Schuster Steven Worley
NCAR, Boulder, CO
![Page 2: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/2.jpg)
What is the RDA?
• 600+ datasets for climate and weather research
• Free and open access • Worldwide usage • Various data access pathways – HTTP (wget, cURL, etc.) – OPeNDAP, WCS, WMS – Web services (CLI, API) – Direct analysis on HPC systems (NCAR users)
![Page 3: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/3.jpg)
RDA Usage
2014: • 17+ PB virtual processing • Web downloads: 7300 users, 750 TB served • Custom orders: 4000 users, 380 TB served, 45,000 orders
![Page 4: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/4.jpg)
That’s a lot of data. How can we improve download
efficiency and reliability?
![Page 5: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/5.jpg)
Globus @ RDA: How we do it
• Single shared endpoint – Data copied to subdirectories under endpoint source path
– ACLs managed programa(cally via Globus CLI – Allow read permission to subdirectories under the shared endpoint
![Page 6: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/6.jpg)
RDA example user interface
acl-add rda#data_request --perm=r --path=/download.auto/CRAM80279/ [email protected]
![Page 7: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/7.jpg)
RDA Alternate Iden(ty login
![Page 8: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/8.jpg)
RDA Alternate Iden(ty login
![Page 9: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/9.jpg)
Collec(ng usage metrics
Globus Endpoint Manager API • Method = GET • URL = https://transfer.api.globusonline.org/v0.10/endpoint_manager/task_list
• Headers = {‘Authorization’:’Globus-Goauthtoken $GOTOKEN’}
• Filters = {‘filter_endpoint’ : ‘rda#data_request’}
![Page 10: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/10.jpg)
What’s next / improvements
• Globus access for general file lists (not just custom orders) – Files are hard-‐linked to enable data shares – Globus groups to manage access to restricted datasets
• Migra(on to “data cart” func(onality • Map Globus user ID to RDA user in API metrics – Easier linkage to individual user ac(vity
![Page 11: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/11.jpg)
Upcoming
• GlobusWorld 2015 poster — this evening
• ESIP Summer Mee(ng • 14-‐17 July 2015, Pacific Grove, CA • Globus breakout session: Speakers wanted • hOp://esipfed.org/
![Page 13: 06 NCAR RDA Integration Cram · RDA)Usage) 2014: • 17+PB)virtual)processing) • Web)downloads:)7300)users,)750)TB)served) • Custom)orders:)4000)users,)380)TB)served,)45,000)orders)](https://reader036.vdocuments.site/reader036/viewer/2022081517/5fdd0658d4310a7b59712719/html5/thumbnails/13.jpg)
Collec(ng usage metrics (cont.)
Metrics stored in RDA DB – Globus user ID – Transfer volume – Timestamps – Source & des(na(on endpoints – # files transferred/skipped