the grid: the first 50 years ian foster argonne national laboratory university of chicago carl...
TRANSCRIPT
-
The Grid: The First 50 YearsIan Foster
Argonne National LaboratoryUniversity of Chicago
Carl Kesselman
Information Sciences InstituteUniversity of Southern California
Grid, Globus Toolkit, and OGSA
*
AbstractThe Internet and Web have had a major impact on society: by allowing us to discover and access information on a global scale, they have created entirely new businesses and brought new meaning to the term "surf." Yet simply being able to "show you stuff" and "know stuff" are ultimately unsatisfactory: we want to "do stuff"and increasingly, to "do stuff together" within distributed teams.This need has lead to the creation of the Grid, an infrastructure that enables us to share capabilities, integrating services and resources within and across enterprises, and allowing active collaborations across distributed, multi-organizational collaborations. Powered by on-demand access to computing, seamless access to data, and dynamic composition of distributed services, the Grid promises to enable fundamentally new ways of interacting with our information technology infrastructure, of doing business and practicing science. It represents perhaps the final step in the great disappearing act that will take computing out of our homes and machine rooms and into the fabric of society, where it will stand alongside telephone switches, power generators, and the other invisible technologies that drive the modern world.In this talk, we trace the history of Grid ideas, starting with the first days of the Internet in the early 1970s, proceeding to the work in the mid to late 1990s that created today's Grid, and finally looking some years into the future. We talk about accomplishments, opportunities, obstacles, and challenges--and, we hope, provide a compelling picture of the dynamic and open mix of ideas, technology, and community that are turning the Grid into reality.
Grid, Globus Toolkit, and OGSA
*
Licklider (1960):Man-Computer Symbiosis is an expected development in cooperative interaction between men and electronic computers. The main aims are
to let computers facilitate formulative thinking as they now facilitate the solution of formulated problems, and
to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs.
Grid, Globus Toolkit, and OGSA
*
Problem Solving in the 21st CenturyTeams organized around common goalsCommunities: Virtual organizationsWith diverse membership & capabilitiesHeterogeneity is a strength not a weaknessAnd geographic and political distributionNo location/organization possesses all required skills and resourcesMust adapt as a function of the situationAdjust membership, reallocate responsibilities, renegotiate resources
Grid, Globus Toolkit, and OGSA
*
Context (1):Revolution in SciencePre-InternetTheorize &/or experiment, alone or in small teams; publish paperPost-InternetConstruct and mine large databases of observational or simulation dataDevelop simulations & analysesAccess specialized devices remotelyExchange information within distributed multidisciplinary teams
Grid, Globus Toolkit, and OGSA
*
Context (2):Revolution in BusinessPre-InternetCentral data processing facilityPost-InternetEnterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)Business processes increasingly computing- & data-richOutsourcing becomes feasible => service providers of various sorts
Grid, Globus Toolkit, and OGSA
*
The (Power) Grid:On-Demand Access to ElectricityTimeQuality, economies of scale
Grid, Globus Toolkit, and OGSA
*
By Analogy, A Computing GridDecouple production and consumptionEnable on-demand accessAchieve economies of scaleEnhance consumer flexibilityEnable new devicesOn a variety of scalesDepartmentCampusEnterpriseInternet
Grid, Globus Toolkit, and OGSA
*
Not Exactly a New Idea The time-sharing computer system can unite a group of investigators . one can conceive of such a facility as an intellectual public utility.Fernando Corbato and Robert Fano, 1966We will perhaps see the spread of computer utilities, which, like present electric and telephone utilities, will service individual homes and offices across the country.Len Kleinrock, 1967
Grid, Globus Toolkit, and OGSA
*
But Things are Different Now
Grid, Globus Toolkit, and OGSA
*
Computing isnt Really Like ElectricityI import electricity but must export dataComputing is not interchangeable but highly heterogeneous: data, sensors, services, This complicates things; but also means that the sum can be greater than the parts Real opportunity: Construct new capabilities dynamically from distributed servicesRaises fundamental questionsAchieving economies of scaleQuality of service across distributed servicesApplications that exploit synergies
Grid, Globus Toolkit, and OGSA
*
New OpportunitiesDemand New Technology Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations
When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances (George Gilder)
Grid, Globus Toolkit, and OGSA
*
Taking Sharing to the Next LevelSharing of communicationTelephones, mailing lists, collaboration toolsSharing of data and knowledgeWeb, semantic webWhat about the rest of the infrastructure?Services, computers, programs, sensors,
Grid, Globus Toolkit, and OGSA
*
Existing Technologies are Helpful,but Not Complete Solutions Peer-to-peer technologiesLimited scope and mechanismsEnterprise-level distributed computingLimited cross-organizational supportDatabasesVertically integrated solutionsWeb servicesNot dynamicSemantic webLimited focus
Grid, Globus Toolkit, and OGSA
*
Whats Missing is Support for Sharing & integration of resources, viaDiscoveryProvisioningAccess (computation, data, )Security PolicyFault toleranceManagementIn dynamic, scalable, multi-organizational settings
Grid, Globus Toolkit, and OGSA
*
Enter the GridInfrastructure (middleware) for establishing, managing, and evolving multi-organizational federationsDynamic, autonomous, domain independentOn-demand, ubiquitous access to computing, data, and servicesMechanisms for creating and managing workflow within such federationsNew capabilities constructed dynamically and transparently from distributed servicesService-oriented, virtualization
Grid, Globus Toolkit, and OGSA
*
Building the GridOpen source softwareGlobus Toolkit , UK OGSA DAI, Condor, Open standardsOGSA, other GGF, IETF, W3C standards, Open communitiesGlobal Grid Forum, Globus International, collaborative projects, Open infrastructureUK eScience, NSF Cyberinfrastructure, StarLight, AP-Grid,
Grid, Globus Toolkit, and OGSA
*
Open SourceEncourage adoption of standards by reducing barriers to entryOvercome new technology Catch-22Enable broad Grid technology ecosystemKey to international cooperationKey to science-commerce partnershipJumpstart Grid industry and allow vendors to focus on value-addE.g., IBM, Avaki use GT3; Platform GlobusOpen source is industry friendly!
Grid, Globus Toolkit, and OGSA
*
Globus Toolkit HistoryNASA beginsfunding Grid work, DOE adds supportThe Grid: Blueprint for a New ComputingInfrastructure publishedGT 1.0.0ReleasedEarly ApplicationSuccesses ReportedNSF & European CommissionInitiate Many New Grid ProjectsAnatomy of the GridPaper ReleasedSignificantCommercialInterest inGridsPhysiology of the GridPaper ReleasedGT 2.0ReleasedDoes not include downloads from: NMI, UK eScience, EU Datagrid, IBM, Platform, etc.
Grid, Globus Toolkit, and OGSA
Chart4
0
0
0
0
0
0
40
40
40
40
40
40
50
50
50
50
50
50
50
50
50
75
75
75
75
75
75
75
100
100
100
150
200
350
350
400
425
450
498
1481
1385
2509
1448
1563
1506
1884
1801
1217
1896
3086
2916
2107
2587
1796
2254
11016
9897
14318
13050
6646
11270
8247
10607
19430
15900
11911
12004
16517
18790
26640
24371
20788
Downloads per Month from ftp.globus.org
Sheet1
Jan-990
Feb-990
Mar-990
Apr-990
May-990
Jun-990
Jul-9740
Aug-9740
Sep-9740
Oct-9740
Nov-9740
Dec-9740
Jan-9850
Feb-9850
Mar-9850
Apr-9850
May-9850
Jun-9850
Jul-9850
Aug-9850
Sep-9850
Oct-9875
Nov-9875
Dec-9875
Jan-9975
Feb-9975
Mar-9975
Apr-9975
May-99100
Jun-99100
Jul-99100
Aug-99150
Sep-99200
Oct-99350GT 1.0.0 released
Nov-99350
Dec-99400GT 1.1.1 released
Jan-00425
Feb-00450
Mar-00498GT 1.1.2 released
Apr-001481
May-001385
Jun-002509GT 1.1.3 released
Jul-001448
Aug-001563
Sep-001506GT 1.1.4 and MPICH-G2 released
Oct-001884
Nov-001801
Dec-001217
Jan-011896Anatomy of the Grid paper released
Feb-013086
Mar-012916
Apr-012107
May-012587
Jun-011796
Jul-012254
Aug-0111016
Sep-019897
Oct-0114318
Nov-0113050GT 2.0 beta released
Dec-016646
Jan-0211270
Feb-028247
Mar-0210607
Apr-0219430GT 2.0 released
May-0215900
Jun-0211911
Jul-0212004
Aug-0216517
Sep-0218790
Oct-0226640GT 2.2 released
Nov-0224371
Dec-0220788
Sheet1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Globus Toolkit Downloads
Sheet2
Sheet3
*
The Emergence ofOpen Grid StandardsIncreased functionality,standardizationCustomsolutions19901995200020052010
Grid, Globus Toolkit, and OGSA
*
Grid CommunitiesGlobal Grid ForumStandards, information exchange, advocacy1000+ participants in tri-annual meetingsApplication communitiesE.g., physics, earthquake engineering, biomedical, etc.Software development and supportNSF Middleware Initiative, UK eScience, Globus Toolkit, EGEE,
Grid, Globus Toolkit, and OGSA
*
Data Grids for High Energy PhysicsEnable international community of 1000s to access & analyze petabytes of dataHarness computing & storage worldwideVirtual data concepts: manage programs, data, workflowDistributed system management
Grid, Globus Toolkit, and OGSA
*
myGrid(Goble, De Roure, Shadbolt, et al.)Imminent data deluge in bioinformaticsHeterogeneous, complex, and inter-related data sourcesIntegrated, community -wide treatment of data, literature, computational services
Grid, Globus Toolkit, and OGSA
*
NEESgrid Earthquake Engineering CollaboratoryU.Nevada Renowww.neesgrid.org
Grid, Globus Toolkit, and OGSA
-
Network for Earthquake Engineering Simulation
Field Equipment
Laboratory Equipment
Remote Users
Remote Users: (K-12 Faculty and Students)
High-Performance Network(s)
Instrumented Structures and Sites
Leading Edge Computation
Curated Data Repository
Laboratory Equipment (Faculty and Students)
Global Connections(fully developed FY 2005 FY 2014)
(Faculty, Students, Practitioners)
12/06/01 MRE Panel
*
Distributed Aircraft Maintenance Environment (Austin et al.)In flight dataAirlineMaintenance CentreGround StationGlobal Networkeg: SITAInternet, e-mail, pagerDS&S Engine Health CenterData centre
Grid, Globus Toolkit, and OGSA
*
Industrial Perspective on Grids:A Wide Range of ApplicationsSources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAIGrid Services Market Opportunity 2005Unique by Industry with Common Characteristics
Grid, Globus Toolkit, and OGSA
*
Open InfrastructureBroadly deployed services in support of fundamental collaborative activitiesFormation & operation of virtual organizationsAuthentication, authorization, discovery, Services, software, and policies enabling on-demand access to critical resourcesComputers, databases, networks, storage, software services,Operational support for 24x7 availabilityIntegration with campus and commercial infrastructures
Grid, Globus Toolkit, and OGSA
*
Open Infrastructure
Grid, Globus Toolkit, and OGSA
*
Grid Communities & TechnologiesYesterdaySmall, static communities, primarily in scienceFocus on sharing of computing resourcesGlobus Toolkit as technology baseTodayLarger communities in science; early industryFocused on sharing of data and computingOpen Grid Services ArchitectureTomorrowLarge, dynamic, diverse communities that share a wide variety of services, resources, dataChallenging computer science research issues
Grid, Globus Toolkit, and OGSA
*
Summary: The Grid Explained,via the BCS Lovelace Medal
Grid, Globus Toolkit, and OGSA
-
Questions?
Grid, Globus Toolkit, and OGSA
There is an increasing need in the military to support virtual organizations: human / agent communities assembled as needed out of existing resources in order to pursue common goals. Virtual organizations frequently must be created to carry out multinational peace keeping operations, joint force operations, or to respond to natural disasters.
Virtual organizations pose a range of problems, some of which are potentially amenable to technological solutions. Commanders need to locate the resources that the virtual organization requires. Elements of the VO may be part of multiple real organizations and may be distributed geographically, so proper coordination is paramount. The composition and function of the virtual organization must change as the situation changes. People may need help in adapting their work practices to meet the needs of the virtual organization.Climate Assumptions about the models
Science is unstable.