open science data cloud - jpgrid.org · nsf‐pire osdc data challenge the image cannot be...
TRANSCRIPT
Open Science Data Cloud
Robert Grossman Open Cloud Consor7um University of Chicago Open Data Group
March 10, 2011
Astronomical data Biological data (Bionimbus)
NSF‐PIRE OSDC Data Challenge
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
Earth science data (& disaster relief)
Open Science Data Cloud
Who are we?
4 www.opencloudconsor7um.org
• U.S based not‐for‐profit corpora7on. • Manages cloud compu7ng infrastructure to
support scien7fic research: Open Science Data Cloud.
• Manages cloud compu7ng testbeds: Open Cloud Testbed.
• Develop reference implementa7ons, benchmarks and standards.
OCC Members
• Companies: Cisco, Citrix, Yahoo!, … • Universi7es: University of Chicago, Calit2, Johns Hopkins, Northwestern Univ., ORNL, University of Illinois at Chicago, …
• Federal agencies: NASA • Other: Na7onal Lambda Rail • Beginning to add interna7onal partners in 2011.
5
Infrastructure
• 2010 Proof‐of‐Concept Infrastructure – 450+ nodes – 3000+ cores – 2+ PB – Four data centers (two more to come in 2011) – Data centers have 10G network connec7ons to StarLight (some 100G links in 2011)
• Plan to add approximately 1 PB of data in 2011. • With current funding, we will refresh 1/3 of the infrastructure in 2011 and 2012.
Why Another Cloud Project?
Small Medium to Large Very Large
Data Size
Low
Med
Wide
Variety of analysis
No infrastructure Dedicated infrastructure General infrastructure
Scien7st with laptop
Open Science Data Cloud
High energy physics, astronomy
Single worksta7ons
Small to medium clusters
HPC
Cycles
Small
Med
Large
Persistent data
data clouds
Large & spec. clusters
databases
What is the Open Science Data Cloud?
Hosted, managed, distributed facility to: • Manage & archive your medium and large datasets • Provide computa7onal resources to analyze it • Provide networking to share it with your colleagues and the public.
Long Time Goal
Build a (small) data center for science.
And preserve your data the same way that libraries preserve books &
museums preserve art.
OSDC Perspec7ve • Take a long term point of view (think like a library not a cloud service provider)
• Operate infrastructure at the scale of a small data center
• Interoperate with public clouds
• Open, interoperable architecture
• Experiment at scale • Vendor neutral
OSDC Projects
Project 1. Bionimbus
www.bionimbus.org
Case Study: Public Datasets in Bionimbus
What Could You Do With 1 PB of Genomics Data?
• The NIH in the U.S. currently makes available for download approximately 2PB of data.
• Bionimbus today consists of 6 racks, 212 nodes, 1568 cores and 0.9 PB of storage.
• We plan to add approximately 1 PB of genomics and other data from the biological sciences to Bionimbus in 2011.
Case Study: ModENCODE
• Bionimbus is used to process the modENCODE data from the White lab (over 1000 experiments).
• Bionimbus VMs were used for some of the integra7ve analysis.
• Bionimbus is used as a backup for the modENCODE DCC
Project Matsu 2: An Elastic Cloud For Disaster Response
Daniel Mandl - NASA/GSFC, Lead
20
Provide Fire / Flood Data to Rescue Workers
Short Term Pilot for 2011 • Colored areas represent catchments where rainfall collects and drains to river basins • River gauges displayed as small circles • Detailed measurements are available on the display by clicking on the river gauge sta7ons.
21
Note blue bars indicating a surge of rainfall upstream
Then a flood wave appears downstream at Rundu river gauge days later
Flood Dashboard
Zambezi basin consisting of upper, middle and lower catchments
Project 3: OSDC PIRE Project
OSDC PIRE Project Overview
• Research – Cloud middleware for data intensive compu7ng – Wide area clouds
• Training and educa7on workshops – Data intensive compu7ng using the OSDC – Cloud compu7ng for scien7fic compu7ng
• Outreach – OSDC Data Challenge
Foreign Partners
• Na7onal Ins7tute of Advanced Industrial Science and Technology (AIST), Japan
• Beijing Ins7tute of Genomics (BIG) • Edinburgh University • Korea Ins7tute of Science & Technology • San Paulo State University • Universidade Federal Fluminense, Brasil • University of Amsterdam
OSDC Data Challenge
• Annual contest to select 3 to 4 datasets each year to add to the OSDC.
• Looking for the most interes7ng datasets to add.
Research Focus
• Cloud architectures for data intensive compu7ng
• Wide area clouds • Con7nuous learning • Scanning queries
Ways to Par7cipate
• Nominate one of your graduate students to spend a summer working with one of the OSDC PIRE Foreign Partners
• Send one of your graduate students to hands‐on Workshops, such as Introduc7on to Data Intensive Compu7ng
• Submit your most impressive dataset to the OSDC Data Challenge
• Buy a container of computers and join the OSDC
Open Science Data Cloud Sustainability Model
Towards a Long Term, Sustainable Model
• Capital Exp about $1M/year • Opera7ng Exp about $1M/year • Moore Founda7on providing $1M/year for 2011 and 2012 to support the Cap Exp.
! "#$$! "#$"! "#$%! "#$&! "#$'!
!"#$%&'()$ $ *$ *$ *$ *$
+,-&.$%&'()$ /$ 01$ 12$ 1*$ 1*$
!"#$3&4&'5-6$789:$ 2$ ;<=>$ ></?$ *<10$ 01<;0$
+,-&.$3&4&'5-6$789:$ 0<11$ /<*=$ 02<;/$ 0?<;;$ 1/<?*$
!@AB"%$3,%")$ 01*2$ ;*/2$ 01*22$ 0?C12$ ;>*/2$
$
Who do you most trust to manage your data for 100 years?
Companies may not be here tomorrow.
Think of a not for profit with that mission.
Government agencies have a role, but not always easy to use.
Buy A Container and Join the OCC
• Use 2/3 of the container for your own purposes.
• Provide 1/3 of the container to the OCC for a share replica space.
To Get Involved
Join the Open Cloud Consor7um: www.opencloudconsor7um.org
Ques7ons?