san diego supercomputer center inca 2.0 shava smallen grid development group san diego supercomputer...
TRANSCRIPT
![Page 1: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/1.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Inca 2.0
Shava SmallenGrid Development Group
San Diego Supercomputer Center
June 26, 2006
![Page 2: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/2.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
“You know you have [a distributed system] when the crash of a computer you've never heard of stops you from getting any work done.” -- Leslie Lamport
Simple Grid application
Grid Reliability
• Grid computing: The ability to dynamically link resources together as an ensemble to support the execution of large-scale, resource-intensive, and distributed applications
![Page 3: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/3.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Is the Grid up?
• Can I login?• Are Grid services the
application[s] use available? Compatible versions?
• Are dataset[s] N accessible to user X? Credentials?
• …
• Can user X run application[s] Y on Grid[s] Z? Access dataset[s] N?
![Page 4: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/4.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Testing a Grid
1. Iteratively define a set of concrete requirements 2. Write tests to verify requirements3. Periodically run tests and collect data4. Publish data
Automate Steps 3 and 4
![Page 5: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/5.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
What type of testing?
• Deployment testing• Automated, continuous
checking of Grid services, software, and environment
• Installed? Configured correctly? Running? Accessible to users? Acceptable performance?
• E.g., gatekeeper ping or scaled down application
Software Package(unit, integrated)
Software Stack(interoperability)
Software
Deployment
NMI
Junit,PyUnit,
Tinderbox
![Page 6: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/6.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Who are the consumers?
• Grid/VO management• Responsible for designing & maintaining requirements• Verify requirements are fulfilled by resource providers
• System administrators• Notified of problems• Enough information to understand context of problem
• End users• View results and compare to problems they are having• Debug user account/environment issues • Advanced users: feedback to Grid/VO
![Page 7: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/7.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Inca
• Inca is a framework for the automated testing, benchmarking and monitoring of Grid resources
• Inca provides:• Scheduled execution of information gathering
scripts (reporters)• Data management
• collection• archiving• publishing
![Page 8: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/8.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Inca’s primary objective: user-level Grid functionality testing and performance measurement
Related Grid monitoring tools
HawkeyeMDS
![Page 9: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/9.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Unique features of Inca
• Debugging• Runs under a regular user account• Flexibly expresses results • Captures reporter execution context• Securely re-runs reporters• Archives full reports • Reporters can be run outside framework
![Page 10: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/10.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Unique features of Inca (cont.)
• Compares results to a specification• Easily and securely configured
• Data collection• Installation
• Profiles and logs reporter resource use
![Page 11: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/11.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Outline
• Inca in use
• Architecture overview
• Software Status
![Page 12: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/12.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Inca today
• Version 1 • aka 0.10.3• available from website and
NMI distribution
• Version 2 pre-release• Available as of 02/06• Production version available
in late summer
• Both versions of Inca are currently being used in production environments
![Page 13: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/13.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Inca in use
1) Software stack validation and verification (v1)
2) Network bandwidth measurements (v1)
3) Grid benchmarking
![Page 14: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/14.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
1) Inca in use: TeraGrid software stack V&V
• TeraGrid - an “enabling cyberinfrastructure” for scientific research• ANL, Indiana Univ., NCSA, ORNL, PSC, Purdue Univ., SDSC,
TACC• 40+ TF, 1+ PB, 40Gb/s net
• Common TeraGrid Software & Services• Common user environment
across heterogeneous resources
• TeraGrid VO service agreement
![Page 15: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/15.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
1) Inca in use: TeraGrid software stack V&V
• Common software stack:• 20 core packages: Globus, SRB, Condor-G, MPICH-G2, OpenSSH,
SoftEnv, etc.• 9 viz package/builds: Chromium, ImageMagick, Mesa, VTK,
NetPBM, etc.• 21 IA-64/Intel/Linux packages: glibc, GPFS, PVFS, OpenPBS, intel
compilers, etc.
50 version reporters: compatible versions of SW
123 tests/resource: package functionality• Services: Globus GRAM, GridFTP, MDS, SRB, DB2, MyProxy,
OpenSSH• Cross-site: Globus GRAM, GridFTP, OpenSSH
![Page 16: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/16.jpg)
1) Inca in use: TeraGrid deployment
• 8 sites/17 resources• Run under user account inca
![Page 17: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/17.jpg)
1) Inca in use: Summary status page
History of percentage History of percentage of tests passed in of tests passed in
“Grid” category for a 6 “Grid” category for a 6 month periodmonth period
All tests passed: 100%All tests passed: 100%
One or more tests One or more tests failed: < 100%failed: < 100%
KeyKey
Tests not applicable to Tests not applicable to machine or have not machine or have not
yet been portedyet been ported
![Page 18: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/18.jpg)
1) Inca in use: Detailed Status View
SWpackages
Resources
![Page 19: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/19.jpg)
1) Inca in use: Detailed view
![Page 20: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/20.jpg)
2) Inca in use: Comparison of end-to-end bandwidth measurement tools
• Joint work with Margaret Murray (TACC) and Martin Swany (UDel)
• Deployed to TeraGrid, GEON
• Compare bandwidth measurement tools:• Pathload
[Dovrolis] • Pathchirp
[Ribeiro]• NWS ping
[Wolski]
![Page 21: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/21.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
3) Inca in use: Grid benchmarks
• GrASP: Grid Assessment Probes
• Omid Khalili, et al., Acquiring and Using Benchmark Data from Computational Grids, accepted for Grid 2006
• Set of probes designed to emulate Grid applications
• Deployed to GEON and TeraGrid
Gather probe on TeraGrid
![Page 22: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/22.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
3) Inca in use: Measuring Grid middleware performance
![Page 23: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/23.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
3) Inca in use: Monitoring Grid middleware reliability
![Page 24: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/24.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
3) Inca in use: Error tracking over time
![Page 25: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/25.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Outline
• Inca in use
• Architecture overview
• Software Status
![Page 26: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/26.jpg)
Architecture overview
Resource 2Resource 1Resource N
…ReporterManager
ReporterManager
ReporterManager
ReporterRepositoryIncat
ReporterAgent
Depot
1. Create a “suite”
2. Submit suite to Reporter Agent
Suite
Depot3. Reporter Agent
invokes Reporter Managers and distributes suite and reporters
R
RRS
R
S S
4. Reporter Managers
send data to Depot
5. GUIs can display collected data by querying Depot
Data Consumers
![Page 27: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/27.jpg)
Architecture overview: Scalable design
Resource 2Resource 1 Resource 3
IncatReporter
Repository
ReporterAgent Depot
ReporterManager
ReporterManager
ReporterManager
VO A
Resource 4 Resource 5
Incat ReporterRepository
ReporterAgent
Depot
ReporterManager
ReporterManager
VO B
Forward suite
Forward results
![Page 28: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/28.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Outline
• Inca in use
• Architecture overview
• Software Status
![Page 29: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/29.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
New features of v2
• Full report archiving• Flexible querying interface• Improved installation and configuration control
• GUI tool for centralized administration• Proxy management via MyProxy• Reporter sharing via repositories• Binary distribution
• Profile reporter system usage• Inca components communicate using SSL
![Page 30: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/30.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Software Status
• 2.0 Pre-release• Available as of February 6, 2006• More integration/stability testing• Not recommended for production deployments• Binary distribution
• 2.0 Production release in August• Source and binary distributions
http://inca.sdsc.edu/prerelease.html
![Page 31: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/31.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
• Supported by:
Inca Information
• Announcements:[email protected]
• Bugs/Feature Requests:http://inca.sdsc.edu/bugs
• Email: [email protected]
• Website: http://inca.sdsc.edu
![Page 32: SAN DIEGO SUPERCOMPUTER CENTER Inca 2.0 Shava Smallen Grid Development Group San Diego Supercomputer Center June 26, 2006](https://reader036.vdocuments.site/reader036/viewer/2022062417/551521e055034673228b5106/html5/thumbnails/32.jpg)
SAN DIEGO SUPERCOMPUTER CENTER
Add Interactive Reporter Details
Submit informationor suggestions
View error history
View historical graphsof run time, CPU time,memory usage