adoption of cloud computing in scientific research

19
Adoption of Cloud Computing in Scientific Research Yehia El-khatib School of Computing & Communications Lancaster University

Upload: yehia-el-khatib

Post on 18-Nov-2014

1.380 views

Category:

Technology


1 download

DESCRIPTION

Some might say the scientific research community is somewhat behind the curve of adopting the cloud. In this talk, I present a few examples of adopting the cloud from the wider research community. I also highlight some of the aspects by which cloud computing could affect scientific research in the near future and the associated challenges.

TRANSCRIPT

Page 1: Adoption of Cloud Computing in Scientific Research

Adoption of Cloud Computing in Scientific Research

Yehia El-khatib School of Computing & Communications

Lancaster University

Page 2: Adoption of Cloud Computing in Scientific Research

Obligatory cloud image…

Page 3: Adoption of Cloud Computing in Scientific Research

Outline

• Cloud Computing in Business

• Cloud Computing in Research

– What does it offer

– Comparison with other distributed paradigms

– Different solutions

– Examples

– Challenges

• Conclusions

Page 4: Adoption of Cloud Computing in Scientific Research

Cloud Computing

• Computational and storage resources provided in an on-demand fashion by large clusters of commodity computers.

• Offers opportunities:

– Customised and isolated computing resources are obtained as and when required to handle user demand.

– Pay per use model allows feasibility and sustainability through harnessing economies of scale.

– Management via web service APIs.

– Universal Internet-based access (all you need is / / / / … ).

Page 5: Adoption of Cloud Computing in Scientific Research

Cloud Computing in Business

• Used to curb computing expenses without restricting the business. – Scale to meet user demand. – Dynamically mitigate system failures. – Seamlessly roll out new capabilities.

• Numerous users:

• Cloud computing market – Worth $40.7bn in 2010 – Expected $177bn in 2015

– Expected $241bn in 2020

http://www.forrester.com/rb/Research/sizing_cloud/q/id/58161/t/2 http://www.gartner.com/it/page.jsp?id=1735214

Page 6: Adoption of Cloud Computing in Scientific Research

Academic Research

• Researchers do not spend their entire time in the lab, field, etc.

• Collected data needs to be processed in order to distil some meaning.

• Such analysis processes range from scripts and spreadsheets to very complex computationally-intensive workflows.

• More data is being gathered using innovative methods (e.g. remote sensing).

Page 7: Adoption of Cloud Computing in Scientific Research

Cloud in Academia

• People in academic circles are slowly adopting cloud computing for particular applications.

• What does the cloud offer? – ‘Everything as a service’ promotes integration and

relatively easy collaboration across institutions, communities and disciplines.

– Customised environments. – Elastic computing infrastructure. – More load off the users, i.e. scientists. More time to focus on their scientific processes.

Page 8: Adoption of Cloud Computing in Scientific Research

Distributed Computing Paradigms HPC Grid P2P Cloud

Ownership (management)

My university Our universities Our partners 3rd party

Trust Very High High Trust in partners

?

Reliability High High Depends on

size & partners Very high

Accounting Individual

Quotas

Individual & Organisational

quotas Difficult… Pay per use

Customisation Very bad Bad Fairly flexible Very flexible

Access Easy Complicated Complicated Easy

Support Local

sysadmin Remote

sysadmin Local/Remote

sysadmin 24x7 support

Page 9: Adoption of Cloud Computing in Scientific Research

What solutions do clouds offer?

• Generic solutions: – Infrastructure (e.g. EmuLab)

– Analysis (e.g. Biocep-R, CloudNumbers)

– Space to discover (e.g. Academia.edu), share (e.g. myExperiment) and collaborate (e.g. Mendeley)

• Domain-driven solutions: – Workflow execution

– Data normalisation

– Data discovery, based on content rather than problem area

Research Support

Research

Page 10: Adoption of Cloud Computing in Scientific Research

Domain-driven Cloud Solutions

• Environmental Virtual Observatory pilot (EVOp)

http://www.EnvironmentalVirtualObservatory.org

– To help: • Environmental scientists solve ‘big questions’.

• Policy makers understand implications of decisions.

• Raise awareness in and interact with local communities.

– Use case for pilot phase: hydrology.

– Deal with both geospatial and time series data.

– Customisable modelling workflows for scientists.

– Predefined analysis tools for non-specialists.

Page 11: Adoption of Cloud Computing in Scientific Research

Domain-driven Cloud Solutions

• Penn State Integrated Hydrologic Model (PIHM) http://slidesha.re/pFFMWp

– Terrestrial watershed modelling in order to predict water distribution.

– Data is sourced through a repository.

– Cloud offers seamless access to abundant resources to carry out modelling workflows and simulations.

– Results are delivered using bespoke visualisation (SaaS).

Page 12: Adoption of Cloud Computing in Scientific Research

Domain-driven Cloud Solutions

• Coaddition of SDSS Astronomical Images http://arxiv.org/abs/1010.1015

– Using Apache Hadoop for coaddition of images from the Sloan Digital Sky Survey. (Coaddition increases the signal-to-noise ratio).

– Runs over NSF cloud maintained by Google and IBM.

– Experimented different approaches to coaddition using the MapReduce framework.

– Improved performance was achieved by reducing job initialisation overhead using index files.

– 300 million pixels processed in 3 minutes.

Page 13: Adoption of Cloud Computing in Scientific Research

Domain-driven Cloud Solutions

• Cell structure analysis http://books.google.co.uk/books?id=C_aQqAa6rEoC

– Hadoop jobs to analyse videos of single cell structures under varying conditions.

• European Space Agency http://www.esa.int

– Uses AWS EC2 & S3 to deliver data about the current state of the planet to scientists, governmental agencies and other organizations worldwide.

• MD Anderson Cancer Center http://bit.ly/o0zDwl

– Large private cloud (8,000 processors) maintained by The University of Texas.

– Used to execute genomic processes against large clinical datasets (~1.4PB) on cancer.

Page 14: Adoption of Cloud Computing in Scientific Research

Domain-driven Cloud Solutions

• NSF http://www.nsf.gov/news/news_summ.jsp?cntn_id=119248

– Approx. $4.5m to fund 13 research projects.

– Mostly CS, but also bioinformatics & earth sciences.

• VENUS-C http://www.venus-c.eu

– 15 year-long pilots in different disciplines: architecture, biology, bioinformatics, chemistry, earth sciences, healthcare, maritime surveillance, mathematics, physics and social media.

• Masters @ SCC Lancaster – Corpus linguistics

– Hydrological modelling

– 3D imaging (volcanology)

Page 15: Adoption of Cloud Computing in Scientific Research

Challenges

• Trust: security and privacy (even by law in some circumstances).

• Great divide between different disciplines.

• Data ownership. – Most data producers don’t mind sharing as long as they

retain ownership.

• Software licenses.

• Belief that cloud/grid/etc is only for certain app’s.

• Investment into delivering cloud-based solutions to scientists. – Legacy applications & infrastructures.

Page 16: Adoption of Cloud Computing in Scientific Research

Challenges

• Trust: security and privacy (even by law in some circumstances).

• Great divide between different disciplines.

• Data ownership. – Most data producers don’t mind sharing as long as they

retain ownership.

• Software licenses.

• Belief that cloud/grid/etc is only for certain app’s.

• Investment into delivering cloud-based solutions to scientists. – Legacy applications & infrastructures.

Page 17: Adoption of Cloud Computing in Scientific Research

Conclusions

• Need for cloud computing for scientific research: – Mainly: “I need more number crunching!” – Also: “I need to bridge data/discipline gaps.”

• Overall adoption is still relatively limited. – Various reasons, including trust. But also cloud-unrelated

problems such as data ownership and software licensing.

• Investment into cloud-enabled research is important. – Not to browse articles via a mobile app while on the tube… – But for the added value of building and nurturing

relationships. – And the economic model (less up front costs).

• Impact: – Better scientific tools, with less overhead on the scientists. – Potential for more integration.

Page 18: Adoption of Cloud Computing in Scientific Research

Thank you!

Questions

http://www.EnvironmentalVirtualObservatory.org

@EVOpilot

Yehia El-khatib http://www.comp.lancs.ac.uk/~elkhatib/

@yelkhatib

Flickr credits: • theaucitron • theplanetdotcom • Pnnl

• stacylynn • bpamerica • soilscience

Page 19: Adoption of Cloud Computing in Scientific Research

Discussion

• Trust is not the problem; it is the perception of trust.

• Different academic communities have varying attitudes

towards new technologies such as the cloud.

• More examples of funding to adopt cloud computing:

o research: http://www.jisc.ac.uk/news/stories/2011/02/umf.aspx

o Gov’t: http://www.cabinetoffice.gov.uk/content/government-ict-strategy