part i web portals for scientific and engineering communities
TRANSCRIPT
![Page 1: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/1.jpg)
Part I
Web Portalsfor Scientific and Engineering communities
![Page 2: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/2.jpg)
Portals for scientific and engineering communities
• Seamless access to HPC resources• Seamless access to instruments• Data storage• Application specific data base• Visualization Tools• Collaboratory• Scientific notepads
![Page 3: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/3.jpg)
Remote Resources
Front-End
FRONT-END:high-level user friendly - visual programming and authoring tools - application GUI
RESOURCES:all hardware and software components needed to complete the user task, including, but not limited to, compute engines from workstations to supercomputers, storage, databases, instruments, codes, libraries, and licenses.
Desktop/Laptop
Seamless Access
![Page 4: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/4.jpg)
Seamless Access
• Create an illusion that all resources needed to complete the user tasks are available locally.
• In particular, an authorized user can allocate the resources she needs without explicit login to the host controlling the resources.
• An analogy: NSF mounted disk or a network printer.
![Page 5: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/5.jpg)
Example: Globus
Advantages:- platform independent mini-language (RSL) for specification of resources- can be layered on top of different schedulers- enables interoperability between resources (can allocate many resources at a time, file transfer, monitoring, etc.)
Disadvantage: - a bag of low level tools
GRAMClient
Gatekeeper Gatekeeper Gatekeeper
Contact addressResource Language Specification
MDSDirectoryService
GSS-API
![Page 6: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/6.jpg)
Towards a complete solution ...PSE: problem description (physics, chemistry, ...)
Task description: I need 64 nodes of SP-2 at Argonne to run my MPI-based executable “a.out” you can find in “/tmp/users/haupt” on
marylin.npac.syr.edu. In addition, I need any idle workstationwith jdk1.1 installed. Make sure that the output of my a.out is
transferred to that workstation
Middle-Tier: map the user’s task description onto the resource specification; this may include resource discovery, and other services
Resource Specification
Resource Allocation: run, transfer data, run
![Page 7: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/7.jpg)
Remote Resources
Front-End Front-End
Middle-Tier
Resource Specification
Abstract Task Specification
We need a third tier!
![Page 8: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/8.jpg)
Target Architecture
Middle-Tier
Resource Specification
Abstract Task Specification
CTA specificknowledgedatabases
VisualAuthoring
Tools
User andGroup
Profiles
ResourceIdentificationand Access
VisualizationsCollaboration
WebFlow
Back-End Resources
Problem Solving Environment
![Page 9: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/9.jpg)
Example of a portalNavigate and choose an existing application
to solve the problem at hand.Import all necessary data.
Retrieve data
Pre/post-processing
Run simulations
Select host
Select model
Set parameters
Run
![Page 10: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/10.jpg)
PSE Example: CCM IPSE
Ken Flurchick, http://www.osc.edu/~kenf/Gateway
1 .Define your problem2. Identify resources (software and hardware)3. Create input file4. Run your application5. Analyze results
![Page 11: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/11.jpg)
QS Front End
Compose interactivelyyour applicationfrom pre-existing
modules
Data-Flow Front-End
![Page 12: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/12.jpg)
Part II
WebFlow
![Page 13: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/13.jpg)
WebFlow design
• Object Oriented, follows JavaBeans model– everything is an object
– objects interact through events
Object A(event source)
Object B(event target)
Fire event EMethod M(){…}
Firing event E by object A causes invocation of method M of object B.The association of event E and method M is achieved by an eventregistration mechanism. An event is also an object and it carries data.
![Page 14: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/14.jpg)
A few words about CORBA
(a digression)
more information on Java, Corba, Distributed Object:
http://www.npac.syr.edu/projects/cps616spring96/index.html
![Page 15: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/15.jpg)
Distributed objects
• Typically WebFlow objects live in different address spaces. We use CORBA to invoke methods of the remote objects.
Object A(event source)
Object B(event target)
Fire event EMethod M(){…}
ORB
![Page 16: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/16.jpg)
How is this possible?
ORB2
Object A(event source)
Object B(event target)
Fire event EMethod M(){…}
ORB1IIOP
- Objects A and B are CORBA objects (thus not Java objects)- Objects are defined in IDL (Interface Definition Language)- IDL definitions are compiled using (Java)IDL compiler- The IDL compiler generates new classes to be used by the Java compiler (javac) instead of the original ones, on both the client and server side- The IDL compiler generates either classes to be extended, or interfaces to be implemented
Object Adapterserves also as a daemon
![Page 17: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/17.jpg)
Example of IDL definition#include “..\BC.idl”module WebFlow { module lms{ interface runEdys:BeanContextChild { void run(); void receiveData(); void setParameter(in string p); }; interface runCasc2d:BeanContextChild{ void run(); void runAgain(); }; interface DoneEvent{ Object getSource(); }; };};
We will create 3 CORBA objects
* two modules: - runEdys - runCasc2d * one event - DoneEvent
They will be added to packageWebFlow.lms
![Page 18: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/18.jpg)
We need more flexibility...
• WebFlow objects are developed independently of each other(reusable modules): we cannot assume that the event source knows anything about the event target and vice versa
![Page 19: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/19.jpg)
Event binding
addEventListenerrmEventListenerfireEvent(E,M)
method M
Event Source Event TargetAdapter
Event
ORB
binding table
DII DSI
![Page 20: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/20.jpg)
Controlling a moduleApplet
ModuleControls
Proxy Module
ModuleActionButton1ActionButton2
….
IIOP
Another complication:Java sandbox!
![Page 21: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/21.jpg)
Adding a remote moduleLocal Host
Add module
Module Factory
Proxy Module
Remote Host
FE
requestAdd module
Module Factory
Module
![Page 22: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/22.jpg)
Back to WebFlow design
![Page 23: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/23.jpg)
CORBA Based Middle-Tier
Mesh of WebFlow Serversimplemented as CORBA objects
that manage and coordinate
distributed computation.
Front End
GatekeeperAuthenticationAuthorization
![Page 24: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/24.jpg)
WebFlow Server
• The WebFlow server is a container object, a.k.a. context - in fact it implements JavaBeanContext class (Java1.2)
• The BeanContext acts as a logical container for JavaBeans (“WebFlow modules and services”) and BeanContexts.
![Page 25: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/25.jpg)
User 1 User 2
Application 1
Application 2
App 2App 1
WebFlow Server
WebFlow server is given by a hierarchy of containers
and components
WebFlow server hosts users and services
Each user maintainsa number of applications
composed of custom modules
and common services
WebFlow Services
![Page 26: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/26.jpg)
WebFlow Context HierarchyMaster Server (Gatekeeper)
Slave Server
Slave Server
User Context
Application Context
Module
Slave Server Proxy
![Page 27: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/27.jpg)
Gatekeeper
![Page 28: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/28.jpg)
Services User Modules
Browserbased
Front-End
Middle-Tier modulesserve as proxies ofBack-End Services
Browserbased
Front-End
User Space Definition and Task Specification
Metacomputing Services
Back-End Resources
![Page 29: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/29.jpg)
Modules
• Similar to JavaBeans– full power of Java (or C++) to implement functionality
– can encapsulate legacy applications
• May serve as Proxies– JDBC
– metacomputing services (such as Globus)
– schedulers (such as PBS, CONDOR, etc)
![Page 30: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/30.jpg)
Services
• Services are modules provided by the system and offers a generic functionality– job services (submit,monitor,kill,... a job)
– file services (edit,copy,move,… a file)
– XML parser
– database access
– mass storage access
– ...
![Page 31: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/31.jpg)
Example of a proxy module
&(rsl_substitution = (MYDIR “/tmp/haupt”))(DATADIR $(MYDIR)/data)(EXECDIR) $MYDIR)/bin))(executable = $(EXECDIR)/a.out)(arguments=$(DATADIR)/file1)(stdout=(MYDIR)/result.dat))(count=1)
GRAM resource descriptionGenerate Data
Run Job
Analyze
The Run Job module is a proxy module. It generates the RSL on-the-fly and submits the job for execution using globusrun function.
The module knows only exec name, location and its arguments/parameters.
![Page 32: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/32.jpg)
WebFlow over Globus
• In order to run WebFlow over Globus there must be at least one WebFlow node capable of executing Globus commands, such as globusrun
• Jobs that require computational power of massively parallel computers are directed to the Globus domain, while other jobs can be launched on much more modest platforms, such as the user’s desktop or even a laptop
running Windows NT.
Bridge between WebFlow and Globus
![Page 33: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/33.jpg)
Secure Access: terminology• Access Control (or Authorization)
– Assurance that the person or computer at the other end of the session is permitted to do what he asks for.
• Authentication– Assurance that the resource (human or machine) at the other end of the session is
what it claims to be
• Integrity– Assurance that the information that arrives is the same as when it was sent
• Accountability (or non-repudiation)– Assurance that any transaction that takes place can subsequently proved to have
taken place
• Privacy– Assurance that sensitive information is not visible to an eavesdropper (usually
achieved using encryption)
![Page 34: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/34.jpg)
Secure Access
• Mutual authentication of servers and users– Certificates, Keberos/SecurID
• Access control– Full autonomy of the resources owner(s)
– Akenti
• Privacy• Integrity
![Page 35: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/35.jpg)
SECIOP
Security Model
Front End Applet
https
authentication& authorization
Gatekeeper
delegation
Stakeholders
HPCC resources
GSSAPIGSSAPI
Layer 1: secure Web
Layer 2: secure CORBA
Layer 3: Secure access to resources
Policies defined by resource owners
https (SSL)AKENTI
CORBA security service
GSSAPI (Globus)
![Page 36: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/36.jpg)
Distributed Objects are less secure
• can play both client and server– in client/server you trust the server, but not the clients
• evolve continually– objects delegate parts of their implementation to the other objects (also dynamically composed at runtime).
Because of subclassing, the implementation of an object may change over time
• interactions are not well defined– because of encapsulation, you cannot understand all the interactions between objects
• are polymorphic (ideal for Trojan horses!)
• can scale without limit – how do you manage the access right to millions of servers?
• are very dynamic
![Page 37: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/37.jpg)
CORBA security is built into ORB
Secure Communications
Authentication
ClientUser
Encryption Audit Authorization
Server
Encryption
Credentials
ObjectAdapterORB
![Page 38: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/38.jpg)
Authentication
• A principal is authenticated once by ORB and given a set of credentials, including one or more roles, privileges, and an authenticated ID.
• An authenticated ID is automatically propagated by a secure ORB; it is part of the caller context
Principal Credentials
Current
Client Server
set_credentials get_attributes
authenticate
![Page 39: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/39.jpg)
Privilege Delegation
• No delegation– The intermediary uses its own credentials
• Simple delegation– The intermediary impersonates the client
• Composite delegation– The intermediary uses both
ClientT
arge
t
Clie
nt
Tar
get
Clie
nt
Tar
get
Clie
nt TargetObject
IIOP
![Page 40: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/40.jpg)
CORBA access model
• Based on a trusted ORB model:you must trust that your ORB will enforce the access policy on the server resource
• The ORB determines:if this client on behalf of this principal can do this operation on this object
• Server uses Access Control Lists (ACL) to control user access
Principal Role Rights Operation
![Page 41: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/41.jpg)
Part III
WebFlow Applications
![Page 42: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/42.jpg)
• Applications vary by the functionality of their Front-Ends– Front-End Applications
• must be pre-installed
• run fast, no restrictions
– Front-End Applets• no installation, but may take time to download
• sandbox restrictions apply, unless signed
![Page 43: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/43.jpg)
• Applications vary by how they are composed from modules– statically
• can by prepared in the Middle-Tier
– dynamically• the user composes them from reusable components
![Page 44: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/44.jpg)
• The modules can interact with each other in different ways:– through events (object oriented approach)
– through ports (data flow model)
– through message passing
![Page 45: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/45.jpg)
• Applications vary on how the Front-End interacts with the Middle-Tier– A complete task description is sent to the middle-tier
• composed of reusable modules
• predefined
– Objects are added to the user context one at a time, and Front-End keeps their references
![Page 46: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/46.jpg)
Landscape Management System
(CEWES)
![Page 47: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/47.jpg)
LMS Objectives To develop a web based system that implements a
“navigate-and-choose” paradigm and allows the end user to: – Select (a set of) computational modules that provide answers to
the problem at hand
– Retrieve input data sets from remote sources
– Use adequate (remote) computational resources
– Visualize and analyze output data on the local host
Anytime, anywhere, using any platform
(e.g., a connected to the Internet laptop PC)
![Page 48: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/48.jpg)
LMS: Changes in Vegetation A decision maker (the end user of the system) wants to evaluate
changes in vegetation in a geographical region over a long time period caused by short term disturbances such as a fire or human activity.
One of the critical parameters of the vegetation model (EDYS) is soil condition at the time of the disturbance.
This in turn is dominated by rainfall that possibly occurs at that time (CASC2D simulation)
Input data for the simulations are available from the Internet, such as Data Elevation Models (DEM) from USGS web site or from custom databases (spices characteristics)
![Page 49: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/49.jpg)
LMS: Changes in Vegetation
Data retrieval Data preprocessing Simulation: two
interacting codes EDYS CASC2D
Visualization
WMS
EDYS CASC2D
DEM Land UseSoil
TextureVegetation
EDYS: vegetation model CASC2D: watershed modelWMS: Watershed Modeling System
![Page 50: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/50.jpg)
LMS Front End
Data retrieval Data pre- and post-processing Simulations
![Page 51: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/51.jpg)
Data RetrievalThe data wizard allows the user to
interactivelyselect the data anddownload them tothe local machine.The raw data arethen fed to the WMS system
launched from the browser to
generate input filesfor simulations.
![Page 52: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/52.jpg)
Launching coupled simulations on different Back-End computational resources
Select host
Select model
Set parameters
Run
![Page 53: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/53.jpg)
WMS based Visualizations
The results of the simulations are send
back to the Front-End, and can be visualized using tools included
in WMS package
![Page 54: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/54.jpg)
Implementation of LMS
• Front-End (client) is a Java application– Data wizard, EDYS and WMS are run locally
• “navigate and choose” - no interactive composition of applications– EDYS, CASC2D, EDYS and CASC2D
• modules exchange data through message passing mediated by WebFlow
• client keeps the module references
![Page 55: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/55.jpg)
slave
Running LMS
runCasc2d
master
UNIXWinNT
slave
runEdys
lms.class
Data wizardWMS
exeCasc2d
WebServer
WebServer
WebFlow ServersClient
- WebFlow modules
![Page 56: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/56.jpg)
Client code
try { //add modules p1 = slaveNT.addNewModule("runEdys"); //as defined in conf.file runEdys re = runEdysHelper.narrow(p1); p2 = slaveUNIX.addNewModule("runCasc2d"); //as defined in conf.file runCasc2d rc = runCasc2dHelper.narrow(p2); //bind events master.attachEvent(p2,"Casc2dDone","Casc2dDone",p1,"run"); master.attachEvent(p1,"EdysStarted","EdysStarted",p2,"run"); master.attachEvent(p1,"EdysDone","EdysDone",p2,"runAgain"); //invoke methods of runCasc2dImp rc.run(); } catch(COMM_FAILURE ex) {System.err.println(ex.getMessage()); System.exit(1);}
![Page 57: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/57.jpg)
Write
slave
Interactions between components
runCasc2d
master
UNIXWinNT
slave
runEdys
lms.class
Data wizardWMS
exeCasc2d
WebServer
WebServer
casc2dIIOP
http
Write
http
![Page 58: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/58.jpg)
Quantum Simulations
![Page 59: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/59.jpg)
Quantum Simulations
![Page 60: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/60.jpg)
QS: WebFlow implementation
![Page 61: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/61.jpg)
![Page 62: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/62.jpg)
Implementation of QS
• Front-End (client) is a Java applet• applications are created dynamically from pre-
existing modules• modules exchange data through ports
(data flow model)• server keeps the module references;
the references are published on a web site
![Page 63: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/63.jpg)
QS: Front-End
![Page 64: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/64.jpg)
Building an application
XMLA visual representation
is converted into a XMLdocument
XMLservice
WebServer
save
parse
ApplContext
Generates Java code to add modules to ApplContextPublishes IOR
Front-End Applet
Middle-Tier
![Page 65: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/65.jpg)
Document Type Definition <!DOCTYPE taskspec [ <!ELEMENT taskspec (task)+> <!ATTLIST taskspec UserContextRef CDATA #REQUIRED AppName CDATA #REQUIRED><!ELEMENT task ((task | module)*,connection*) > <!ELEMENT module (#PCDATA) > <!ATTLIST module modulename CDATA #REQUIRED host CDATA #REQUIRED > <!ELEMENT connection (out,in)> <!ELEMENT in EMPTY> <!ELEMENT out EMPTY> <!ATTLIST out modulename CDATA #REQUIRED eventname CDATA #REQUIRED<!ATTLIST in modulename CDATA #REQUIRED method CDATA #REQUIRED > ]>
![Page 66: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/66.jpg)
Example XML document
<taskspec UserContextRef="123as321" AppName="TestApplication"><task> <module modulename="FileBrowser" host="localhost"> </module> <module modulename="FileEditor" host="localhost"> </module> <module modulename="Gaussian" host="localhost"> </module> <connection> <out modulename="FileBrowser" eventname="FileEvent" event="File"/> <in modulename="FileEditor" method="run"/> </connection> <connection> <out modulename="FileEditor" eventname="FileEvent" event="File"/> <in modulename="Gaussian" method="run"/> </connection></task></taskspec>
![Page 67: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/67.jpg)
Object Oriented Applications
![Page 68: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/68.jpg)
Mobility System’s Applications
Coordinatestransformations
databases
Remote HPCC resources
- object oriented approach - implementation:- CORBA based Middle-Tier - bean-box type API - JDBC proxy modules
- Web interface to store data in DB in variable format- Data transfer from DB to a visualization engine- Coordinates transformations on a remote server- Launching simulations on remote hosts with interactive input
![Page 69: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/69.jpg)
Part IV
Gateway:Portal for Computing
![Page 70: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/70.jpg)
Target Architecture
Middle-Tier
Resource Specification
Abstract Task Specification
CTA specificknowledgedatabases
VisualAuthoring
Tools
User andGroup
Profiles
ResourceIdentificationand Access
VisualizationsCollaboration
WebFlow
Back-End Resources
Problem Solving Environment
![Page 71: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/71.jpg)
Design Issues
• Support for a seamless access (security)
• Support for distributed, heterogeneous Back-End services (HPCC, DBMS, Internet, ...) managed independently from Gateway
• Variable pool of resources: support for discovery and dynamical incorporation into the system
• Scalable, extensible, low-maintenance Middle Tier
• Web-based, extensible, customizable, self-adjusting to varying capacities and capabilities of clients (humans, software and hardware)
front end
![Page 72: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/72.jpg)
Gateway Implementation
• Distributed, object-oriented middle tier– CORBA objects (Gateway Containers, Gateway
Modules and Gateway Services) implemented in Java. [Scalable, extensible, low-maintenance middle tier]
– Containers define the user environment.– Modules and Services serve as proxies: they accept the
user requests (Front End) and delegate them to the Back End. [Support for distributed, heterogeneous back-end services managed independently from Gateway]
Note: modules can be implemented in C++; also can be DCOM components
![Page 73: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/73.jpg)
Gateway Implementation (2)
• Gateway operates in a keberized environment [Support for a seamless access]
– tickets are generated on the client side
– Keberos-based CORBA security service is used to manage the user sessions
– Globus GSSAPI implemented over Keberos is used for resource allocation
![Page 74: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/74.jpg)
Gateway Implementation (3)
• Task Specification is expressed in XML– CTA independent
– Decouples implementation of the Front End and the Middle Tier
– Allows for an abstract (platform independent) task specification, and thus the Middle Tier may act as a resource broker
• Resource Specification is expressed in XML– Simplifies match-making and resource discovery
– Simplifies generating Globus RSL in-the-fly
[Support for distributed, heterogeneous Back-End services; Variable pool of resources; Scalable, extensible, low-maintenance Middle Tier]
![Page 75: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/75.jpg)
Gateway Implementation (4)
• Component-based Front-End [extensible]
• Front-End Components (“toolbox interfaces”) are – applets (interfaces for common services)– XML pages or frames
[Web-based, extensible, customizable, self-adjusting]
• All components (Front End, Middle-Tier) are defined in XML and contain metadata (used for component mining)
![Page 76: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/76.jpg)
Front End
![Page 77: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/77.jpg)
CTA specific knowledge database
– requires server side support (both the middle tier and the back-end) through well defined interfaces
– should be constructed from reusable or cloneable components
– allows for identification of software components best suited to solve the problem at hand
![Page 78: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/78.jpg)
Visual Authoring Tools
• Allows for composition of the computational task from components (reusable modules)
• Different tools to support various programming models such as data parallel, task parallel, data flow, object oriented
• No assumption on granularity
• Metadata about components and support for archiving and mining the components
• Support for instrumentation and steering
![Page 79: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/79.jpg)
Example: Data Flow
![Page 80: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/80.jpg)
Example: DARP
![Page 81: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/81.jpg)
User and Group Profile
• Controls the user/group environment– file access
– job monitoring
– ...
• Allows for customization– preferences
– users with disabilities
– ...
• History of actions
• Scientific notebook
![Page 82: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/82.jpg)
Resource Identification and Access
• Computational resources– hardware, software, licenses
– desktop applications
• Data– file systems, mass storage, distributed databases
– Internet data repositories
• Networks
![Page 83: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/83.jpg)
Front-End infrastructure
![Page 84: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/84.jpg)
Front-End Support
• Portal Page
• User Context
• Control Applet
• Navigator (extensible, customizable)
• PSE specific toolboxes– A placeholder for the Problem Description toolboxes
– A placeholder for the code toolbox
– Resource request toolbox – Data postprocessing toolbox
• Other (Collaboration, Visualizations, …)
![Page 85: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/85.jpg)
User Context
• Represents a Gateway session.
• The session is associated with a user (or group) profile.
• WebFlow extends the notion of the UNIX profile via the 'User Data Base' (UDB). This UDB contains information about submitted jobs, history of the users actions, and other user state information. The user context may also contain application/front-end specific information.
![Page 86: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/86.jpg)
![Page 87: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/87.jpg)
Control Applet• The control applet is responsible for maintaining the
session, and direct communication with the middle-tier. • Direct communication is the most efficient, but since it is
buried into an applet, this mechanism is not readily customizable.
• The generic services, such as file service (upload, download, edit, copy, move, delete) and job services (show current jobs/show queues/kill jobs) will be supported this way. [combination of the user context and a query]
• The Gateway will also support a non-direct communication with the middle-tier through servelts.
![Page 88: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/88.jpg)
Screen Dump of the Control Applet
![Page 89: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/89.jpg)
Navigator
• The navigator allows the user to select and customize toolboxes.
• Embedded in a separate frame, it consists of menus, buttons, links, etc, derived from an XML document.
• The navigator is a hierarchical, extensible and customizable.
![Page 90: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/90.jpg)
![Page 91: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/91.jpg)
Problem description toolboxes
• The problem description is application specific, and the Gateway only provides a general framework for creating a PSE.
• The most important part is the specification of what services (middle and back tier) are needed, what is their API, and how to add new services.
• Example services: access to databases, XML parsing, generating HTML in-the-fly, file services.
![Page 92: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/92.jpg)
Code toolboxes
• The end user see it as a mapping between the problem description and software to be used to solve the problem. Actually, it identifies WebFlow modules and their parameters to be used to construct the application (see resource request toolbox below).
• The module parameters may include input files, and if necessary, the input files are generated at this stage (using this or a separate toolbox). In addition, some parameters will be constructed from information stored in data bases, including UDB, and other sources.
![Page 93: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/93.jpg)
Resource Request Toolbox
• The front-end activities result in an abstract task specification.
• Abstract in the sense that the user may not know nor care what actual resources are used.
• The task is composed of independently developed modules and services following different programming models.
![Page 94: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/94.jpg)
Other toolboxes
• Visualizations• Collaboration• Scientific notebook• ...
![Page 95: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/95.jpg)
Middle-Tier
![Page 96: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/96.jpg)
User 1 User 2
Application 1
Application 2
App 2App 1
WebFlow Server
WebFlow server is given by a hierarchy of containers
and components
WebFlow server hosts users and services
Each user maintainsa number of applications
composed of custom modules
and common services
WebFlow Services
![Page 97: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/97.jpg)
CORBA Based Middle-Tier
Mesh of WebFlow Serversimplemented as CORBA objects
that manage and coordinate
distributed computation.
Front End
GatekeeperAuthenticationAuthorization
![Page 98: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/98.jpg)
Back End
![Page 99: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/99.jpg)
Back End Services
• Access to HPCC (via Globus)• Access to distributed databases (via JDBC)• Access to mass storage• Access to the Internet resources• Access to desktop application and local data• Access to code repositories
![Page 100: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/100.jpg)
Gateway Security
![Page 101: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/101.jpg)
SECIOP
Security Model (Keberos)
Front End Applet
SECIOP
authentication& authorization
Gatekeeper
delegation
HPCC resources
GSSAPIGSSAPI
Layer 1: secure Web
Layer 2: secure CORBA
Layer 3: Secure access to resources
Policies defined by resource owners
![Page 102: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/102.jpg)
Building Gateway Components
![Page 103: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/103.jpg)
Middle-Tier is given by a mesh of WebFlow Servers that manage
and coordinate distributed computation
.
• Gateway applications are composed of independent reusable modules• Modules are written by module developers who have only limited knowledge of the system on which the modules will run.• The WebFlow system hides module management and coordination functions
![Page 104: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/104.jpg)
How to develop a Gateway component (or a toolbox)
• Back-end service• Middle-tier proxy• Front-end controls
![Page 105: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/105.jpg)
How the Back-End interacts with the rest of the system?
• Often, your job do not need to interact.– Using GRAM and GASS you stage data and
executable, submit the job and retrieve output.
– Using DUROC you can coallocate resources and run MPI-based parallel/distributed codes. The messages between nodes are sent outside Gateway control or support.
– HPF runtime will distribute your job and facilitate interprocess communication.
![Page 106: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/106.jpg)
Implementing Back-End Services
• If you need to interact– Using a separate module, you may move files between
nodes while your jobs are executing– Your job may be a server (e.g., database, GRAM)
[if socket listener - be careful about security!]
– Your job my be a CORBA client (Java, C++)
– ...
![Page 107: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/107.jpg)
What does it take to develop a
Gateway module (a proxy)? • Many come as a standard Gateway modules• User’s modules
– Are CORBA objects
• Define IDL (as an XML document)• Compile IDL (in the tie mode)
• Implement the functionality of the module
• Implement events
• Develop Front-End controls that invoke methods of the module
![Page 108: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/108.jpg)
Selecting a Predefined Task
![Page 109: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/109.jpg)
Gateway/WebFlow Mission
• seamless access to remote resources– through a Web based user interface
– customized application GUI
• high-level user friendly visual programming and runtime environment for HPDC
• portable system based on industry standards and commodity software components
![Page 110: Part I Web Portals for Scientific and Engineering communities](https://reader035.vdocuments.site/reader035/viewer/2022062408/56649eab5503460f94bb1891/html5/thumbnails/110.jpg)
Updates
• Contact person: Tomasz Haupt
• voice (315) 443-2087
• http://www.npac.syr.edu/users/haupt/WebFlow/