conflicts and synergies between industry and academiaevent.twgrid.org/isgc2008/presentation...

© 2006 Open Grid Forum

Conflicts and Synergiesbetween

Industry and AcademiaCraig A. Lee, President, OGF

International Symposium on Grid ComputingAcademia Sinica, Taipei, Taiwan -- April 9, 2008

2© 2008 Open Grid Forum

Conflicts -- Different Goals

• Academia:• Computer Scientists want to

"design the world" -- define the entire framework

• Discovery, workflow, security• Technical Transparency• Publications and grants• Explore fundamental issues,

properties, and design space for a given topic area

• Industry:• Industry wants to build just

what they need for immediate goals

• Cluster scheduling• Competitive advantage• Short time to market• Identify specific capabilities

-- points in the design space -- around which to build products and markets


Innovation “no man’s

land”OGF

Driving Innovation

Research push Market pull

Phase 1Solution proposal

Phase 2Prototype

Phase 3Pre-commercial product/service

Phase 4Commercial

product/service

Phase 0Research

Market Pull

Managing the Technology Maturation Process

Technology Readiness Levels

Ulf Ulf DahlstenDahlsten, Director , Director ‘‘EmergingEmergingTechnologies and InfrastructuresTechnologies and Infrastructures’’


The Goal: Synergy!

How to manage “technology maturation”?How to:

1. Facilitate the development of exciting research directions and the desire for thorough technical understanding, while

2. Coordinating the wide-scale application and adoption of specific capabilities that provide value to IT organizations and build markets?

Where are we now?• What have we accomplished?

Where are OGF and the grid community going?• What does the technical landscape look like and

where are the opportunities for synergy?


What Have We Accomplished?

• Many, many operational grids comprised of hundreds of thousands of nodes

• Brand Identity• Strong world-wide community• Growing adoption of standards

• DRMAA, HPC Basic Profile, JSDL, BES• Others in pipeline

• Vibrant research topics• Web 2.0, Virtualization, Cloud Computing,

Green IT


An Example: EGEE

QuickTime?and aTIFF (LZW) decompressor

are needed to see this picture.

Bob Jones, EGEE Project Director


European Grid Initiative




Important Activities at OGF

• HPC-Basic Profile• JSDL & BES for HPC job submission• Interoperability layer for cluster schedulers• Demos at SC06 and SC07 with MS, HP, IBM, Platform…

• GLUE: Resource Information Model• On-going work w/ DMTF on harmonizing GLUE and CIM

• OGSA-WG: Open Grid Services Architecture• Specifications developed & now driving adoption

• GridRPC-WG: Grid Programming Model• Recently progressed to full recommendation

• SAGA-WG: Simple API for Grid Applications• Reference Model WG

• Infrastructure Management -- Data Centers8


OGSA Spec AdoptionProject/Spec

WS-

Nam

ing

RN

S

OG

SA-B

ES

OG

SA-B

yteI

O

WS-

DA

I

OG

SA-B

SP 2

.0

JSD

LW

SRF

Bas

ic

Prof

ileH

PC B

asic

Pr

ofile

Globus ? yes yes yesUnicore 6 no yes yes yesFujitsu USMT will yes yes yes yesMicrosoft CCS yes yes yesGenesis II yes yes yes yes no will yes yes yesMarty yes yes yesOMII-UK yes yes yes yesGridSAM yes yes yesCrown yes yes yesPlatform yes yes yesOGSA-DAI yes yesNAREGI No May Yes May Yes yesOsamu yes


OGF Reference Model:a Grid Component Instantiation Dependency Graph

OGF Reference Model, v2.0E Draft, D. OGF Reference Model, v2.0E Draft, D. SnellingSnelling, Fujitsu, P. Strong, eBay, Fujitsu, P. Strong, eBay


Grid Management Entities

OGF Reference Model, v2.0E Draft, D. OGF Reference Model, v2.0E Draft, D. SnellingSnelling, Fujitsu, P. Strong, eBay, Fujitsu, P. Strong, eBay


Some OGF-22 Workshops

• Financial Services• Hedging, Risk Mgmt, Protocol pricing• Wall Street firm just approved $100M for next generation grid• Scaling, storage, power, cooling, app development

• Cloud Computing• Keynote by Irving Wladawsky-Berger of IBM and MIT• Workshop room packed• Cloud Interoperability, Reliability, Security

• Data• Digital Libraries, Digital Repositories -- data explosion• Metadata Exchange key topic• Potential collaboration w/ SNIA

• Spec Adoption Track• GIN, SAGA, DRMAA, HPC-BP• Managed efforts for key stakeholders and potential adopters

13© 2008 Open Grid Forum 13

OGF-Europe Funded by EU!

• The EU recognises interoperable software standardsinteroperable software standards are important for broader, more mainstream adoption

• OGF-EUROPE crucial for realization of past 8 years of European investments in Grid computing into actionable and sustainable results, targeting the Small-to-Medium Enterprise (SME) sector

• EC Funding: Service & Software Architectures, Infrastructures & Eng.

• FP6, 2002‐2006, €€78 Million78 Million

• FP7, 2007‐2013, €€120 Million120 Million



14 14

Some specifics …

5 Technical Expert Fellowships5 Student Scholars2 Surveys on trends & grid practices 3 Reports supporting OGF Tech. Strategy & Roadmap 4 Grid adoption challenge reports 6 Community best practice reports 4 Workshops on standardization challenges6 Community outreach seminars

Including 4 in-depth tutorials

2 OGF-Europe international events

Dave Wallom, OERC

First International Event• OGF‐23, June 2‐6, 2008, Barcelona

– Hosted by Barcelona Supercomputing Center– 500 + participants: EU projects, industry leaders, standards developers and end users

• Community Outreach Seminars– Digital Repositories– Geospatial, GENESI‐DR

• Co‐located with BEinGRID Industry Days– EU project with 23 experiments with 75 partners in applied research and industry

– Financial modeling, retail mgmt, design and simulation, earth observation, groundwater modeling, fluid dynamics, and seismic processing

Who’s in OGF‐Europe ?

QuickTime?and aFF (LZW) decompreseeded to see this pict

Dave Wallom, OERC

Contact [email protected] to learn more


The OGCThe OGC--OGF CollaborationOGF Collaboration

• Promote best practices and international standardization for distributed geospatial data processing capabilities that is:•• TransparentTransparent -- users is not aware of the infrastructure•• InteroperableInteroperable -- the resources work together•• ScalableScalable -- small local, to massive distributed platforms


Some Geospatial Factoids

• Open Geospatial Consortium• www.opengeospatial.org

• “Helping the World to Communicate Geographically”• Any type of geospatial data• Anything that goes on a map

• A Few Current OGC Standards• Web Map Server (WMS)• Web Feature Server (WFS)• Web Coverage Server (WCS)• Catalog Service for Web (CSW)

• Half of all Web 2.0 URLs registered on programmableweb.com are geospatially related

• Google pushing KML through OGC process

http://www.opengeospatial.org/


Immense Applicability of Geo Data



Image: E. Gennai, C. Terbough, ESRI

Estimates vary, but ~80Estimates vary, but ~80--90% of all data collected or 90% of all data collected or produced by the human race is geospatially referencedproduced by the human race is geospatially referenced




Carl Reed, OGC CTO:Carl Reed, OGC CTO:““GRID Technology must GRID Technology must be an integral component be an integral component of the of the GeoWebGeoWeb!!!!””


State of the Collaboration• Initial Goals

• Integrate OGC's Web Processing Service (WPS) with a range of"back-end" processing environments• Web 2.0, basic web services, local/national grid infrastructures

• Integration of WPS with programming and workflow tools• Swift, SAGA, DAGMan, Pegasus, Kepler, Taverna, and Triana• … just to name a few

• Integration of OGC Federated Catalogues and Data Repositories with grid data movement tools

• Outcome from OGF-22 Collaboration Workshop, Feb. 26, 2008• Workshop report, set-up project web site, email list• Geospatial grid sessions at Euro Geophysical Union in April in Vienna• Coordinate June meetings (June 2-6)

• OGF-23 in Barcelona with OGC-TC in Potsdam• OGF-Europe sessions on Digital Repositories

• Invited paper in November IEEE Computer (just in time for SC08)•• OGF invited to participate in OGC OWSOGF invited to participate in OGC OWS--6 6 testbedtestbed planning processplanning process


What is OWS?• OGC Web Service Testbed• Annual process where sponsors identify specific

demonstration targets• Larger number of participants supply in-kind resources

• OWS-5 recently finished (March, 2008)• OWS-5 Sponsors

• BAE Systems - National Security Solutions• Federal Geographic Data Committee (FGDC/USGS)• Google• Lockheed Martin Integrated Systems & Solutions (lead org)• Northrup Grumman• US National Aeronautic and Space Administration (NASA)• US National Geospatial-Intelligence Agency (NGA)

• ~70 Participants•• 4x4x--5x Return on Investment5x Return on Investment

•• Strength of CollaborationStrength of Collaboration


OWS-5• Five Threads

• Sensor Web Enablement (SWE)• Geo Processing Workflow (GPW)• Information Communities and Semantics (ICS)• Agile Geography• Compliance Testing (CITE)

• Many videos presented at final outbrief• Sensor Web 2.0 Experiments• Data Architecture Views• Conflation Processing• Web Coverage Processing Service


OWS-5 NASASensor Web Demo



EO-1 tasked to collect imagery on Northern San Diego Countywildfires, UAV track data integrated in Google Earth


Planning for OWS-6

• “Whiteboard” session• March 28 in St. Louis• All manner of ideas aired

• FAA AIXM (Aeronautical Info Exchange Model)• SOA for GEO• Service Lifecycle• Geo Rights Management• WMS rehosting• Workflow and WPS• Sensor Web Enablement• OGF INFOD for event pub/sub

•• OWSOWS--6 Sponsors Meeting6 Sponsors Meeting•• April 30, 2008, Herndon, Virginia, USAApril 30, 2008, Herndon, Virginia, USA•• WPSWPS--toto--Grid is a key goal on the tableGrid is a key goal on the table


Switching Gears:“To Distribute or Not To Distribute”

• Prof. Satoshi Matsuoka, TITech• Keynote at Mardi Gras Conference, Baton

Rouge, Jan.31, 2008





• In the late 90s, petaflops were considered very hard and at least 20 years off …

• After 10 years (around now) petaflops are “real close” but there's still no "global grid”

• What happened?

• … while grids were supposed to happen right away


What Happened?• It was easier to put together massive clusters than to get

people to agree about how to share their resources• For tightly coupled HPC applications, tightly coupled

machines are still necessary• Grids are inherently suited for loosely coupled apps

(e.g., Monte Carlo, Parameter Sweep), or enabling access to machines and data, and the integration of the two



• With Gilder's Law, bandwidth to the compute resources will promote thin client approach

• Example: Tsubame machine in Tokyo

Phil Papadopoulos @ SDSC




S. Matsuoka, TITech


So How to Use a Large Machine?



Question:Question: So whatSo whatdoes this job sizedoes this job sizedistribution distribution meanmean

for the userfor the userdata distribution? data distribution?

Satoshi Matsuoka, TITech

• How to feed a large machine with lots of users w/ lots of decentralized data?

• What about data interoperability?• How to make it easy to use?• How to virtualize it?


Virtualization and Cloud Computing

•• ““ThereThere’’s Grid in them s Grid in them tharthar clouds!clouds!””• I. Foster’s blog, ANL & UC, Jan. 8, 2008

• Clouds have a very simple user API effectively hiding all the complexity of an ad hoc grid on the back-end• e.g., Amazon’s EC2 & S3, IBM’s Blue Cloud

and others …• If so, will this enable mass-market grids?

• Users don’t have to be aware of using “a grid”• If so, what does “cloud interoperability” require?

• Is virtualization a means of achieving this?•• Major opportunity for synergyMajor opportunity for synergy


How to Manage Sets of VMs in Distributed Environments?

• Virtual Workspaces?• Dynamically provisioned environments

• Implementation using VMs• Encapsulated configuration and fine-grained

enforcement• Easy way to build VMs?

• rPath, OSFarm (CERN), Bcfg2 (ANL)• Managing Virtual Clusters

•• ContextualizationContextualization


Deploying Workspaces Remotely

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Workspace-Workspace metadata

-Pointer to the image-Logistics information

-Deployment request-CPU, memory, node count, etc.

VWSService

K. Keahey, ANL, SC07


Interacting with Workspaces

Poolnode

Trusted Computing Base (TCB)

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

The workspace service publishesinformation on each workspace

as standard WSRF ResourceProperties.

Users can query thoseproperties to find out

information about theirworkspace (e.g. what IP

the workspace was bound to)

Users can interact directly with their

workspaces the same way the would with a

physical machine.

VWSService

K. Keahey, ANL, SC07


Opportunities for Synergy• Resolving VM deployment issues

• Do they have public IP addresses and can find each other?• Can they share storage?• Can I integrate them into my Virtual Organization?

• Virtual Cluster and Contextualization• Networking, shared storage, name resolution, etc. that will be

portable across sites and implementations• Put VMs in a deployment context of the Grid, site, and other VMs• Common understanding among the image “vendor”, deployer,

and user• Continuing work with DMTF

• Identify/write use cases to feed DMTF virtualization work, e.g. portals, aggregated virtualized resources, cloud computing, etc,leveraging OGF’s Grid-Virt use case work as appropriate

• Packaging: OVF vs rPath. Evaluate, identify, and input changes to potentially merge OVF and rPath so there is a single packaging format for the next version of the OVF spec


Spaceship Earth








… and Climate Change





With thanks to Carl Reed


Global demand outstripping supply 3:1

Rich media and social computing

New York City

Demand for Data Centers

LondonSource: www.Bastionhost.com

Bill St. Arnaud


Zero Carbon Strategyfor Cyber-infrastructure

1. More efficient computers and network equipment

2. Virtualization of computer and networks and relocated to “zero carbon” data centers

3. Deployment of “zero carbon” data centers at remote renewable energy sites linked by optical networks

4. New network and computer architectures that take advantage of remote zero carbon data centers

Bill St. Arnaud


Example:Grid in a Box at a Windmill

Source: www.Bastionhost.com

Bill St. Arnaud

• Extensive use of virtualizationvirtualization and fiber opticsfiber optics to put the computing where the power is

• Enable the workload to “follow the wind”or

“follow the sun”


Solar Computing Tiles


If we make some assumptions about performance levels…


How Much Can a Building Compute?


• Typical House• Assume 500m2 surface area• 20% efficiency• ~ 2 TeraFlops / house

• 200m Skyscraper• Assume 50m x 50m base,

40,000 m2 surface area• 25% efficiency• ~ 1/2 BG/L compute capacity


A Call to Action

• Look for those opportunities for synergy• Nurture the grid research community

• Good events on hot topics• Broad community involvement• Let a thousand flowers bloom (innovation)

• While targeting key efforts that produce high impact results•• Efforts that connect the investment of time, money Efforts that connect the investment of time, money

and people with concrete resultsand people with concrete results• For example, HPC-Basic Profile

• Seek natural adjacencies for the development and use of grid technology


A Summary of Opportunities

• Technical• Data access and integration• Geospatial data and applications• Data centers• Virtualization• Cloud Computing• Green IT, …

• Organizational• OGF-Europe• DMTF, SNIA, OGC• IBM, eBay, MS, Platform, …


A Shameless Plug

• OGF-23 in Barcelona• Barcelo-Sants Hotel• June 2-6, 2008

• OGF-24 in Singapore• Biopolis• September 15-19, 2008

• OGF-25 with EGEE• Somewhere in Europe, Spring 2009

• And a final bit of wisdom …


The Real Reason why there areno Penguins at the North Pole

Thank you. Any Questions?Thank you. Any Questions?

conflicts and synergies between industry and academiaevent.twgrid.org/isgc2008/presentation...

Documents