the grid: past, present, and future tom defanti university of illinois at chicago as ian foster...
TRANSCRIPT
![Page 1: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/1.jpg)
The Grid:Past, Present, and Future
Tom DeFantiUniversity of Illinois at Chicago
as Ian Foster
Mathematics and Computer Science DivisionArgonne National Laboratory andDepartment of Computer Science
The University of Chicago
![Page 2: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/2.jpg)
ARGONNE CHICAGO
On-demand creation of powerful virtual computing systems
The Grid: The Web on Steroidshttp://
http://
Web: Uniform access to HTML documents
Grid: Flexible, high-perf access to all significant resources
Sensor nets
Data archives
Computers
Softwarecatalogs
Colleagues
![Page 3: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/3.jpg)
ARGONNE CHICAGO
Why Now?
The Internet as infrastructure– Increasing bandwidth, advanced services
Advances in storage capacity– You can buy a Terabyte for < $15,000
Increased availability of compute resources– Clusters, supercomputers, etc.
Advances in application concepts– Simulation-based design, advanced scientific
instruments, collaborative engineering, ...
![Page 4: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/4.jpg)
ARGONNE CHICAGO
Talk in a Nutshell … Where this all started
– Special purpose networks, stunts Where we are
– Grid services as unifying concept– Limited but growing application set– Persistent testbeds
Where we’re going– Production grids– New user communities and applications– Convergence of Grid and Info Infrastructure– High-level services and development tools
![Page 5: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/5.jpg)
ARGONNE CHICAGO
Disclaimers
Describing work of many people– ANL, ISI, NCSA, CIT, SDSC, NASA, EVL, etc.
– In particular, Carl Kesselman, Globus Co-PI
– In U.S., Europe, Asia, Australia Much important work by others is left out
– I have picked just a representative sample
![Page 6: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/6.jpg)
Where This All Started
![Page 7: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/7.jpg)
ARGONNE CHICAGO
High-Performance Networks:E.g., CASA Gigabit Testbed
One of four U.S. testbeds in early 90s Dedicated network
– HIPPI over SONET, custom hardware– Caltech, SDSC, LANL
Focus on distributed supercomputing applications– Large distributed memory & vector supercomputers
CASA: Paul Messina et al.
![Page 8: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/8.jpg)
ARGONNE CHICAGO
The I-Way @ SC’95 Conference
Focus on application demonstrations– 60+ groups
OC-3 backbone Large-scale use of
immersive displays– CAVE and I-Desk
I-Soft programming environment– Pioneered security,
scheduling ideas
I-WAY leaders: Tom DeFanti, Tim Kuhfuss, Rick Stevens, Maxine Brown
![Page 9: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/9.jpg)
ARGONNE CHICAGO
The I-Soft SoftwareEnvironment
Kerberos authentication– I-POP initiated rsh to local
resources Andrew File System to
distribute software & state Central scheduler
– Dedicated I-WAY nodes on resource
– Interface to local scheduler Nexus-based comm libraries
– MPI, CAVEcomm, CC++
I-Soft development: Warren Smith, Jonathon Geisler, Steve Tuecke
I-POPAFS KerberosScheduler
ATMSwitch
Resource 1
Resource N
Possible Firewall
ATM
Internet
![Page 10: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/10.jpg)
ARGONNE CHICAGO
Combustion System Modeling
A shared collaborative space– Link people at multiple
locations– Share and steer scientific
simulations on supercomputer Prototypical of new applns
– Traditional HPC an element of a more complex whole
Substantial development effort– Yet insecure, no QoS
Chicago
San Diego
Lori Freitag et al., Argonne National Laboratory and NALCO
DC
![Page 11: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/11.jpg)
Where We Are Now
![Page 12: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/12.jpg)
ARGONNE CHICAGO
The Need for Grid Services
Remoteaccess
Remotemonitor
Informationservices
Faultdetection
. . .Resourcemgmt
CollaborationTools
Data MgmtTools
Distributedsimulation
. . .
net
![Page 13: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/13.jpg)
ARGONNE CHICAGO
The Grid Services Concept
Standard services that– Provide uniform, high-level access to a wide
range of resources (including networks)– Address interdomain issues: security, policy– Permit application-level management and
monitoring of end-to-end performance Broadly deployed, like Internet Protocols Enabler of application-specific tools as well as
applications themselves Integrate across domains to support the
creation of scalable virtual organizations
![Page 14: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/14.jpg)
ARGONNE CHICAGO
The Globus Project: Developinga Grid Services Architecture
Developed as a key enabling mechanism the Grid Security Infrastructure– Uniform authentication & authorization
mechanisms in multi-institutional setting– Single sign-on, delegation, identity mapping– Public key technology, SSL, X.509, GSS-API
Used to construct Grid resource managers that provide secure remote access to– Computers: GRAM server (HTTP), secure shell– Storage: storage resource managers, GSIFTP– Networks: differentiated service mechanisms
Globus project Co-PI: Carl Kesselman
![Page 15: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/15.jpg)
ARGONNE CHICAGO
Developing a Grid Services Architecture
Developed as a 2nd key enabling mechanism the Grid Information Service (aka MDS)– Lightweight Directory Access Protocol (LDAP)
used to access info about specific resources– Index servers provide for flexible indexing– Uniform resource discovery & characterization
mechanisms in highly distributed systems– A basis for resource brokering, adaptation,
autoconfiguration Other services include fault detection and
communication
GIS: Steve Tuecke, Steve Fitzgerald, Gregor von Laszewski
![Page 16: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/16.jpg)
ARGONNE CHICAGO
Status of Grid Services
Core Grid services & others (e.g., Network Weather Service) have been deployed in large-scale testbeds
Availability of these services is enabling tool & application development projects
Major challenges certainly remain: e.g.– Advance reservation, policy, accounting– End-to-end application adaptation (events?)– Integration with commodity technology
Grid Forum: http://www.gridforum.org
![Page 17: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/17.jpg)
ARGONNE CHICAGO
Examples of Grid-Enabled Tools
Message Passing
– MPICH-G2 Distributed collaboration
– CAVERNsoft, Access Grid High-throughput computing
– Condor-G, Nimrod-G Distributed data management
& analysis
– Data Grid toolkits Desktop access to Grid
resources
– Commodity Grid Toolkits (CoG Kits)
![Page 18: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/18.jpg)
ARGONNE CHICAGO
tomographic reconstruction
real-timecollection
wide-areadissemination
desktop & VR clients with shared controls
Advanced Photon Source
Online Instrumentation
archival storage
DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago
![Page 19: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/19.jpg)
ARGONNE CHICAGO
Distributed Supercomputing
SF-Express Distributed Interactive Simulation: Caltech, USC/ISI
Starting point: SF-Express parallel simulation code
Globus mechanisms for– Resource allocation
– Distributed startup
– I/O and configuration
– Fault detection 100K vehicles using 13 computers, 1386
nodes, 9 sites (5 years early!)
NCSAOrigin
CaltechExemplar
CEWESSP
MauiSP
![Page 20: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/20.jpg)
ARGONNE CHICAGO
Problem solving environment for comp. chemistry
Globus services used for authentication, remote job submission, monitoring, and control
Future: distributed data archive, resource discovery, charging
Problem Solving Environments
ECCE’: Pacific Northwest National Laboratory
![Page 21: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/21.jpg)
Where We Are Going
![Page 22: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/22.jpg)
ARGONNE CHICAGO
EmergingProduction
Grids
NASA Information Power Grid
NSF National Technology Grid
![Page 23: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/23.jpg)
ARGONNE CHICAGO
Emerging User Communities
NSF Network for Earthquake Engineering Simulation (NEES)– Integrated instrumentation,
collaboration, simulation Grid Physics Network (GriPhyN)
– ATLAS, CMS, LIGO, SDSS– World-wide distributed analysis
of Petascale data Access Grid: supporting group
based collaboration
![Page 24: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/24.jpg)
ARGONNE CHICAGO
Today’s Information Infrastructure
Network-centric: simple, fixed end systems; few embedded capabilities; few services; no user-level quality of service
O(106) nodes
![Page 25: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/25.jpg)
ARGONNE CHICAGO
Tomorrow’s Information Infrastructure:Not Just “Faster and More Reliable”
Application-centric: heterogeneous, mobile end-systems; many embedded capabilities; rich services; user-level quality of service
QoS
Resource Discovery
Caching
O(109) nodes
![Page 26: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/26.jpg)
ARGONNE CHICAGO
Optical MREN (OM)
ANL3 pr
1 pr
(2) OC-3
3 pr
OC
-3
2 pr
Bell Nexxiato CA*net4
I-WIRE
OC-3
OC-3
?
Optical STAR TAP(AUP free)
AADS switch
6 pr
FranklinCO
Central Office
UIUC
UIC
UC
IIT
MREN
NUChicago
BN
NUEvanston GigE
GigE
IUBloom-ington
UIPUI
Purdue
STARLIGHT
![Page 27: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/27.jpg)
ARGONNE CHICAGO
I-WIRE: Computing
Urbana
Argonne Chicago
Topology
Research Areas• Latency-Tolerant
Algorithms• Interaction of
SAN/LAN/WAN technologies
• Cluster Architectures
![Page 28: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/28.jpg)
ARGONNE CHICAGO
Scalable Global Services:The InterGrid
Do for the Grid what routers do for the network
Wide variety of services– Resource trading– Policy enforcement– Agent based scheduling– Agent based monitoring– Virtual data
Exploit emerging network services, e.g. QoS
Data Cachin
g
Data Cachin
g
Scheduling
Scheduling
Fault Recover
y
Fault Recover
y
RoutingRouting
![Page 29: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/29.jpg)
ARGONNE CHICAGO
“Megacomputing” (Larry Smarr)– Very large numbers of computers
– Opportunistic, dynamic behaviors
– Bandwidths variable, ultimately to Gb/s SETI@home, distributed.net, entropia.com
![Page 30: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/30.jpg)
ARGONNE CHICAGO
The Emergence of theApplication Service Provider
ASP model: “rent” access to applications that execute on remote computers
Decouples interface, application, computing, hardware– Each can be optimized & sold separately
Highly disruptive impact on all aspects of the computer industry
May also be key to a dramatic increase in the penetration of HPC– E.g., see NetSolve (U.Tenn), PUNCH (Purdue)
![Page 31: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/31.jpg)
ARGONNE CHICAGO
Summary
Grids will change the way we do science and engineering
Key services and concepts have been identified, although major challenges remain
Transition of services and applications to production use is starting to occur
Future will see increased sophistication and scope of services, tools, and applications
Potentially disruptive trends: ASPs, megacomputing
![Page 32: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/32.jpg)
ARGONNE CHICAGO
For More Information
Book published by Morgan Kaufman– www.mkp.com/grids
Globus– www.globus.org
Grid Forum– www.gridforum.org
![Page 33: The Grid: Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National](https://reader036.vdocuments.site/reader036/viewer/2022081518/55149e83550346ea6e8b5897/html5/thumbnails/33.jpg)
ARGONNE CHICAGO
We are Hungry for the Grid
www.evl.uic.edu