Hosting Cloud, HPC and Grid Educational Activities on FutureGrid

Renato Figueiredo – U. of Florida

Geoffrey Fox, Barbara Ann O’Leary – Indiana University

Introduction

Some of the pressing needs faced by educators when teaching Cloud, Grid, HPC:• Availability of suitable resources

• A flexible environment

• Fast turn-around times

• Reuse and sharing of educational materials

FutureGrid sees education as one of its main use cases and seeks to address these needs• Goals of this BOF:

• Highlight key aspects of FG and its use to date

• Benefit from your input and discussions to make it a primary resource used by you and the community for education activities

2

Needs addressed by FutureGrid

Resources:• FutureGrid is an XSEDE resource available to the

community at large which provides unique opportunities for education

• Thousands of cores, multiple sites• Heterogeneous: supporting HPC, cloud, Grid

Flexibility:• Cloud, virtualization, dynamic provisioning

• Environment can adapt to the user, rather than expect user to adapt to the environment

3

Needs addressed by FutureGrid

Fast turn-around times• Reduce barriers to entry and engage new users

• Web 2.0 portal to create class project and manage users; fast turn-around

Reuse and sharing of educational materials• Use of encapsulated “appliances” as a primary

delivery mechanism of education/training modules• Promoting reuse, replication, and sharing

4

Guiding principles

Fidelity: activities should use full-fledged, executable software: education/training modules• Learn using the proper tools

Reproducibility: Creators of content should be able to install, configure, and test their modules once, and be assured of the same functional behavior regardless of where the module is deployed• Incentive to invest effort in developing, testing and

documenting new modules

5

Guiding principles

Deployability: Students and users should be able to deploy modules in a simple manner, and in a variety of resources• Reduce barriers to entry; avoid dependences

upon a particular infrastructure

Community-oriented: Modules should be simple to share, discover, reuse, and expand• Create conditions for “viral” growth

6

FutureGrid Web Portal

How classes are setup and managed • Create a Web account

• Submit a project proposal: define a class, workshop, short course, tutorial• Education is one of the key use cases of FG; eager

to have more projects!!!

• Needs approval, but not tied to a fixed schedule

• Typically, turn-around of days

• Add users to project• Users create own accounts using the portal

• Project leaders authorize them to gain access

• Students can then interactively use FG resources (e.g. to start VMs)

• Share and reuse – appliances, documents7

Educational appliances

A flexible, extensible platform for hands-on, lab-oriented education on FutureGrid

Support clustering of resources• Virtual private clusters: Virtual machines + virtual

networking to create isolated, repeatable environments• Virtual appliances: self-contained, pre-packaged VMs

• Virtual networks: simple management of virtual clusters by students and educators

8

9

Virtual appliance example

Virtual appliance + virtual network

copy

instantiate

Hadoop+

VirtualNetwork A Hadoop worker Another Hadoop worker

Repeat…

Virtual machine

Virtual network

Use of FutureGrid in classes

Cloud/distributed computing classes• U.of Florida, U. Central Florida, U. of Puerto

Rico, Univ. of Piemonte Orientale (Italy), Univ. of Mostar (Croatia)

Distributed scientific computing• Louisiana State University

Tutorials, workshops:• Big Data for Science summer school

• A cloudy view on computing

• SC’11 tutorial – Clouds for science

• Science Cloud Summer School (July/Aug’12)

10

Cloud computing classes

Massimo Canonico, U. Piemonte Orientale Difficulties to overcome:

• Hardware issues: find enough free physical machines able to host virtual machines

• Software issues: time to install/configure as many as possible different cloud platforms

• University was not able to provide me the necessary hardware and software support

Students started to play with FutureGrid• After attending few lessons, they were able to

start/stop virtual instance with several Cloud Computing platforms

11

Cloud computing classes

Students used Eucalyptus, OpenStack and Nimbus• Half were not computer scientists.

As FutureGrid freely shares their physical machines and their cloud platforms, decided to freely share all materials of my class.• Hands-out, configuration files and link to

useful documentation are available

• https://portal.futuregrid.org/contrib/cloud-computing-class

12

Cloud computing classes

Graduate-level “Cloud computing for Data-Intensive Sciences” (Judy Qiu, Fall 2010)• Virtualization technologies and tools

• Infrastructure as a service

• Parallel programming (MPI, Hadoop)

• FutureGrid provided a set of software options that made it possible for students to work on different projects along the system stack.

13

Cloud Storage#8 Cloud Storage Survey (Xiaoming, Nixiaogang)

Cloud Storage#8 Cloud Storage Survey (Xiaoming, Nixiaogang)

Iterative MapReduce#3 LDA (Changsi, Yang)

#4 MemCache (Saliya, Yiming ,Jerome)#5 Avro (Yuduo, Yuan, patanachai)#6 PageRank (Shuo-Huan,Parag)

Iterative MapReduce#3 LDA (Changsi, Yang)

#4 MemCache (Saliya, Yiming ,Jerome)#5 Avro (Yuduo, Yuan, patanachai)#6 PageRank (Shuo-Huan,Parag)

Virtualization#9 Hypervisor Performance Analysis Project (James ,

Andrew)

Virtualization#9 Hypervisor Performance Analysis Project (James ,

Andrew)

Cloud Platfor

m

CloudInfrastruc

ture

Cloud Infrastructure #7 Nimbus, Eucalyptus (Stephen, Sonali, Shakeela)

Cloud Infrastructure #7 Nimbus, Eucalyptus (Stephen, Sonali, Shakeela)

Hypervisor/

Virtualization

Dryad/DryadLINQ#1 Matrix Multiplication (Swapnil,Amit,Pradnay)

#2 PhyloD (Ratul,Adrija,Chengming)

Dryad/DryadLINQ#1 Matrix Multiplication (Swapnil,Amit,Pradnay)

#2 PhyloD (Ratul,Adrija,Chengming)

Higher Level

Languages

Term Projects

(Slide courtesy of Judy Qiu)

University ofArkansas

Indiana University

University ofCalifornia atLos Angeles

Penn State

IowaState

Univ.Illinois at Chicago

University ofMinnesota Michigan

State

NotreDame

University of Texas at El Paso

IBM AlmadenResearch Center

WashingtonUniversity

San DiegoSupercomputer

Center

Universityof Florida

Johns Hopkins

July 26-30, 2010 NCSA Summer School Workshophttp://salsahpc.indiana.edu/tutorial

300+ Students (200 on sites from 10 institutes; 100 online)IU MapReduce and UF Virtual Appliance technologies are supported by FutureGrid.

(Slide courtesy of Judy Qiu)

Big Data for Science

16

Uploading and sharing images

APIs available to upload images, customize, save, and share images

Community education pages are available• FutureGrid Web portal allows users to publish

their own content

• Tutorials, presentations on Web portal; VMs on image repositories

Discussion starters

At a high-level, does FutureGrid provide capabilities that are aligned with your expectations for an CI useful to hosting educational activities?

What would you need to evaluate whether using FutureGrid is a good idea?

17

Cyberinfrastructure

What cyber-infrastructure features does FutureGrid have that may help you with educational needs?

What cyber-infrastructure features may be missing which would be important?

18

Educational materials

What documentation do educators, students need to use this infrastructure?

What formats are effective? Tutorials, manuals, videos?

19

Topics

IaaS cloud computing? Map/reduce? HPC? Grid? Applications?

20

Outreach, Community-building

What information may be missing, unclear, hindering ability to reach out?

Ways to attract/foster a community that will share and benefit from sharing educational materials?• Within and beyond XSEDE

21

22

Questions?

More information:• http://www.futuregrid.org

• http://grid-appliance.org

This document was developed with support from the National Science Foundation (NSF) under Grant No. 0910812 to Indiana University for "FutureGrid: An Experimental, High-Performance Grid Test-bed." Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF