the interactive european grid project
Post on 31-Dec-2015
35 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Interactive European Grid Project
introduced by Dr. Jesús Marco de LucasCSIC, Instituto de Física de Cantabria, Santander, SPAIN
EELA Conference, 4 September 2006, Santiago de Chile
J. Marco , EELA Conference, 4 September, Santiago de Chile2
Example: Ultrasound Computer TomographyA new method of medical imaging based on the reconstruction by numerical techniques of an image, using as input the data measured by a scanner of ultrasounds which surrounds the object of interest. The application requires analyzing about 20 Gb of data, which would take order of one month in a workstation…
Researchers demand resources
Data Acquisition System
Site A
Site B
Site C
~ 20 GBytes
========
Total < 30 minutes
~ 4 GBytes
+
+
Bandwidths ~100 Mb/s
MatlabOctave Java
Data Acquisition System
Site A
Site B
Site C
~ 20 GBytes
========
Total < 30 minutes
~ 4 GBytes
+
+
Bandwidths ~100 Mb/s
MatlabOctave Java
“Interoperable production-level e-Infrastructure for
demanding interactive applications to impact the daily work of researchers”
• Distributed Parallel (MPI) Interactive Computing & Storage at the Tera level • User Friendly Access• Grid Interactive Desktop
int.eu.grid VISION
int.eu.grid in three slides
J. Marco , EELA Conference, 4 September, Santiago de Chile3
A Real Challenge!
the int.eu.grid project aims to change the way researchers can use the available e-Infrastructure, exploiting the interactivity and collaboration possibilities
Researchers need to be convinced that they can: Transfer and process gigabytes of information in minutesForesee more complex algorithms on larger statistics, test and tune them, use more powerful visualization techniquesCollaborate across the network in a rewarding mode, from sharing information to discussing and presenting remotely through enhanced videoconference environments.
J. Marco , EELA Conference, 4 September, Santiago de Chile4
Interactive European Grid(http://www.interactive-grid.eu)
Project acronym int.eu.gri
d
Contract number 031857
Instrumen
t I3
Duration 2 years
may ´06-april ´08
“providing transparently the researcher’s desktop with the power of a supercomputer, using distributed resources”
http://www.interactive-grid.euCoordinator: CSIC, Jesús Marco, IFCA, Santander, SPAIN
[marco@ifca.unican.es]
J. Marco , EELA Conference, 4 September, Santiago de Chile5
My summary of the GRID(testbed/integrator/developer/user view)
The Grid “idea” is as good as difficult to put in practiceUsing the network, and open standards, join resources from different sites and make them available to user communities in a transparent way:
Sit users in front of a 1000 GHz/ 100 Tbytes computer Theoretically, it makes sense:
The network is there now (and evolves even faster than CPUs!)Basic middleware exists (for a “secure” communication sharing resources)
KEY point: “SHARING” & “COLLABORATION”This is EXTREMELY difficult, why should we just bother?????Why not use just resources at one site (clusters, SC centers)?The answer is two-fold:
e-Science scenarioMeta-computing possibilities
J. Marco , EELA Conference, 4 September, Santiago de Chile6
About a 1000GHz/100Tbytes shared computer…
We have a clear idea of how to setup and also of how to use it!
LARGE CLUSTER
• Get an account/password from administrator• Develop your code, parallelize/distribute• Use a repository• Compile using MPI, make executable, store
it • Set links to data• Submit script to PBS or LSF queue• Monitor job status• Retrieve output & logs from queue• Display output
GRID FRAMEWORK
• Get certificate, register in Virtual Organization• Develop your code, use Grid API• Use a CVS repository to share with others• Compile using MPI, make executable, store
in Virtual directory (SE) • Define data lfn (will be stored in SE)• Submit to WMS/RB from a Grid Desktop• Check jobstatus• Get output, store in Virtual directory• Display output in Grid Desktop
J. Marco , EELA Conference, 4 September, Santiago de Chile7
Executing Applications: HTC vs.
HPC High Throughput Computing :
Maximize the number of program executionsExample: 1.000.000 independent simulations
Use 1000 processors, do 1000 simulations in each oneAggregate for final result
Works perfectly in a clusterExamples: Condor, batch queues: LSF, PBS
Works perfectly in a GRID with a Workload Management System
High Performance Computing:Minimize the answer timeExample: Train a complex Neural Network
Use 1000 processors in parallelThe information of all of them is required along the jobNeeds a “parallelized description” (e.g. MPI) + and a topology (master-workers)
How to handle network latency in a GRID framework? (or in general QoS)Metacomputing “issues”
J. Marco , EELA Conference, 4 September, Santiago de Chile8
Benefit for applications
y = 17180x-0.971
R2 = 0.9994
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
0 10 20 30 40 50 60 70
number of computing nodes
time
(seg
)
Time reduction: from 5 hours down to 5 min
using 60 nodes!
Scale as 1/N
412980 events,16 variables16-10-10-1 architecture1000 epochs for training
Modeling including latency <300 ms needed !
Parallelization techniques reduce waiting time in Neural Network Application
J. Marco , EELA Conference, 4 September, Santiago de Chile9
Innovation
What innovation do we need?
The CrossGrid project demonstrated the feasibility of an interactive Grid framework showing applications running in a testbed using middleware developed on top of EDG/LCG services. int.eu.grid will provide a production quality service supporting MPI and interactivity on top of EGEE services
Web Services approach will be followed (gLite as underlying middleware)
MPI support will be based on PACX-MPI + OPEN-MPIEuropean MPI software (HLRS) + International effortMPI across sites without the need for public IP on WNs
Interactivity support will integrate: User friendly interface to the infrastructure Powerful visualization capabilities
Consideration of QoS4
J. Marco , EELA Conference, 4 September, Santiago de Chile10
Single sign-on / authorisation
Platform independent
Batch jobs MPI jobs Running
interactive applications using java plugins or VNC
Monitoring grid applications
Flexible Application framework
User profile management
Easy application add on
Local and grid file management
Desktop tools: Job Wizard Job Monitor Application Container and Plugin
Grid Desktop
GridFTP Commander User Profile Manager Private Storage Management VNC/SSH console
J. Marco , EELA Conference, 4 September, Santiago de Chile11
Project Resources
Grid-empowered Infrastructure Initial core: 13 sites in Europe
linked through GEANT-2
ComputersClusters with 10-100 Pentium/Xeon nodesbut also Itanium, Opteron, Power5
2-4 processors nodes, multi core, large RAM
InteroperableEGEE batch jobs will be able to execute in background mode
3
Human resources Excellent subteams already “in place”, examples:
CrossGrid WP4-sites & Integration Team, driven by FZK & LIPPSNC-UAB-CSIC on interactivity, MPI use, and applications
Operation of infrastructureContinuous effort along the projectBenefit from support expertise for middleware Integration (experience from CrossGrid)
e-Infrastructure
J. Marco , EELA Conference, 4 September, Santiago de Chile12
Summary: the int.eu.grid Mission
“To deploy and operate a production-quality Grid-empowered eInfrastructure oriented to service research communities
supporting demanding interactive applications.”
Deployment of e-InfrastructureOriented to interactive useSite integration supportGrid operations service
Middleware for interactivity and MPIAdapt/integrate existing middlewareguarantee interoperability with EGEE
Provide a complete interactivity suite
Desktoproaming accessscheduler with prioritization servicescomplex visualization.
Sub-objectives through the I3 instrument:
Support for interactive applications:setup of collaborative environment and VOconsideration of performanceinteractivity and visualization requirementsidentification and selection of research
oriented interactive applications
Support remote collaboration activities:research, management, integration, training
Approach target research communities Provide security measures for interactivity
2
J. Marco , EELA Conference, 4 September, Santiago de Chile13
Potential Impact
INPUT TO/FROM OTHER PROJECTS
On experience, middleware, application support, training, test of infrastructure
Input to National Grid initiativesDGRID, AustriaGrid, GridIreland
Collaboration with EGEEAim for interoperabilityObjective: two way synergy: transfer experience and middleware to/from EGEETraining and DisseminationOffer complementary services to a large user base
Dissemination effortPresence in EGEE/GGF meetingsAddressing different research communities at their workshops (national and international wide contacts)
Ex. FZK, PSNC, LIP, TCDProfit from multidisciplinary framework
Ex. CSIC: horizontal program
11
APPLICATIONS:Medicine:
Ultrasound Computer Tomography Simulation, Visualization, Remote
AccessEnvironmental studies:
Landscape modelling , PollutionPhysics:
Astrophysics: Planck missionFusion: visualization and 3D transport codes HEP: filtering of data produced by ATLAS
Attracting others:Through dissemination activitiesIdentification of Applications
Outreach: workshops, seminars, coursesDefinition of requirements (and use cases) for a given application/communityAnalyze the possibilities for use of the eInfrastructureTest and train on real infrastructure
Profiting from links with National Grid InitiativesEGEEOther international projects (like EELA)
J. Marco , EELA Conference, 4 September, Santiago de Chile14
Collaboration with EELA
The Interactive European Grid open to collaboration
Listen/Support researchers needs for interactive demanding applications
Offer MPI expertise, in particular for distributed applicationsRun on resources adapted for MPI (multi cpu, large RAM, etc)
Interactive oriented setup
Test collaborative and user friendly frameworkTry Grid DesktopAdd Powerful visualizationJoin Access Grid collaborative experience
Collaborate on development/deployment issuesSimplified installation and virtualizationApplication Development framework
Common partners: LIP, CSIC
Hope this collaboration will be fruitful! Thanks! 4
top related