collaboration and grid technologies
DESCRIPTION
Collaboration and Grid Technologies. Geoffrey Fox Professor of Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 [email protected] http://www.infomall.org http://www.grid2002.org. What we can discuss. - PowerPoint PPT PresentationTRANSCRIPT
Collaboration and GridTechnologies
Geoffrey FoxProfessor of Computer Science, Informatics, Physics
Pervasive Technology Laboratories
Indiana University Bloomington IN 47401
http://www.infomall.org
http://www.grid2002.org
What we can discuss Grid and Collaboration Tools and Architectures which
can address some difficulties encountered upto now in areas like distance education
Audio-Video conferencing addressing network difficulties and the integration of different approaches
Sharing material over the Internet• Shared display collaboration
Producing web pages using “re-usable” portlet architecture
Tools are in “alpha” release at moment• So they might be less reliable than unreliable tools they are
trying to improve
2/22/2001 JSUFall97Master http://www.npac.syr.edu
Programming for the WebProgramming for the WebGeneral IntroductionGeneral Introduction
Course at Jackson State University Spring98 Course at Jackson State University Spring98 and Fall 97and Fall 97
http://www.npac.syr.edu/users/gcf/jsufall97introNancy McCracken
Geoffrey Fox, Tom Scavo
Syracuse University NPAC
111 College PlaceSyracuse NY 13244 4100
3154432163
JSU
Syracuse
Teaching Jackson State Fall 97 to Fall 2001
Spring 2004: Course on “e-Science”/Grid
Distance Learning …..• Linking teachers to students remotely• Linking students/mentors to students/mentors for
collaboration at a distance on a project• Collaboration is sharing
– People (audio, video, text chat, instant messenger)– Web Pages– Other electronic resources (e.g. Word running on laptop)– Homework– Grades
• Asynchronous: I post a web page; you access it– I send email– Can recover from temporary network glitches
• Synchronous: I post an electronic document; you see what I post (in a second or two)– As I update, you get update– Very sensitive to network glitches
Problems from Tango and all Others Clients were unreliable – addressed by better windows and
moving collaboration from Browser to an application Networks were unreliable and firewalls are a problem
• Not a lot of progress with QoS at network level• Some QoS problems are due to different collaboration
streams interfering• Use application level QoS with highly robust managed
messaging Very hard to customize each application in “shared state event
model”• Offer shared display• Convert Applications to Web Services
Many different standards H323, SIP, Access Grid, T120 …• Unify as single XML standard
Inconvenient to customize user interfaces• Use portlet technology supporting desktop and PDA clients
Collaboration and Web Services Collaboration has
a) Mechanism to set up members (people, devices) of a “collaborative sessions”
b) Shared generic tools such as text chat, white boards, audio-video conferencing
c) Shared applications such as Web Pages, PowerPoint, Visualization, maps, (medical) instruments ….
b) and c) are “just shared objects” where objects could be Web Services but rarely are at moment
• We can port objects to Web Services and build a general approach for making Web services collaborative
a) is a “Service” which is set up in many different ways (H323 SIP JXTA are standards supported by multiple implementations) – we should make it a WS
Shared Event Collaboration All collaboration is about sharing events defining state changes
• Audio/Video conferencing shares events specifying in compressed form audio or video
• Shared display shares events corresponding to change in pixels of a frame buffer
• Instant Messengers share updates to text message streams
• Microsoft events for shared PowerPoint (file replicated between clients) as in Access Grid
Finite State Change NOT Finite State Machine architecture Using Web services allows one to expose updates of all kinds as
messages• “Event service” for collaboration is similar to Grid notification service and we
effectively define SDE’s (service data elements) in OGSI
Group (Session) communication service is needed for the delivery of the update events• Using Event Messaging middleware makes messaging universal
Global-MMCS 2.0 XGSP based MCU We are building an open source protocol independent Web
Service “MCU” which will scale to an arbitrary number of users and provide integrated thousands of simultaneous users collaboration services.
We will deploy it globally and hope to test with later this year. The function of A/V media server will be distributed using
NaradaBrokering architecture.• Media Servers mix and convert A/V streams
Open Global-MMCS MCU based on the following open source projects• openh323 is basis of H323 Gateway• NIST SIP stack is basis of SIP Gateway• NaradaBrokering is open source messaging from Indiana• Java Media Framework basis of Media Servers (needs
performance enhancement)
XGSP Web Service MCU Architecture
SIP H323 Access Grid Native XGSPAdmire
Gateways convert to uniform XGSP Messaging
High Performance (RTP)and XML/SOAP and ..
Media ServersFilters
Session ServerXGSP-based Control
NaradaBrokeringAll Messaging
Use Multiple Media servers to scale to many codecs and manyversions of audio/video mixing
NB Scales asdistributed
WebServices
NaradaBrokering
Polycom, Access Grid and RealVideo views of multiple streams using
A/V Web Service integrating SIP and H323
Features of A/V Conferencing Need a video camera – Polycom ViaVideo or Logicon Plantronics or similar echo-canceling microphone such
as Viavideo VRVS or GlobalMMCS does not need hardware
multicast• Messenging in GlobalMMCS very powerful• Software multicast will IMHO replace hardware multicast
Access Grid supports multiple video cameras but no special reason one needs four – depends on application
Web service will mosaic multiple streams into one• Allow Polycom to receive multiple streams
RealVideo for most reliability as 10 second buffer
NaradaBrokering
Minicomputer
Firewall
ComputerServer
PDA
Modem
Laptop computerWorkstationPeers
Peers
Audio/VideoConferencing Client
Audio/VideoConferencing Client
NaradaBrokering BrokerNetwork
BrokersONLYprocessmessages
Grid Messaging Substrate
Consumer Service
SOAP+HTTPRMI TCP/IPRTP ….
Messaging Substrate
Consumer Service
Standard client-serverstyle communication.
Substrate mediatedcommunication removestransport protocoldependence.
SOAP+HTTPRMI TCP/IPRTP ….
Any Protocols satisfying QoS
Messaging Substrate has “intelligence”, “any desired fault tolerance” “logging” etc.
“GridMPI” v. NaradaBrokering In parallel computing, MPI and PVM provided “all the features
one needed’ for inter-node messaging NB aims to play same role for the Grid but the requirements and
constraints are very different• NB is not MPI ported to a Grid/Globus environment
Typically MPI aiming at microsecond latency but for Grid, time scales are different• 100 millisecond quite normal network latency• 30 millisecond typical packet time sensitivity (this is one audio or video
frame) but even here can buffer 10-100 frames on client (conferencing to streaming)
• <1 millisecond is time for a Java server to “think” Jitter in latency (transit time through broker) due to routing,
processing (in NB) or packet loss recovery is important property Grids need and can use software supported message functions and
trade-offs between hardware and software routing different from parallel computing
NaradaBrokering Based on a network of cooperating broker nodes
• Cluster based architecture allows system to scale in size• Grid routing topologies are open research issue?
Originally designed to provide uniform software multicast to support real-time collaboration linked to publish-subscribe for asynchronous systems.
Perhaps better thought of as stream not message handler Now has several core functions
• Reliable order-preserving “Optimized” Message transport (based on performance measurement) in heterogeneous multi-link fashion with TCP, UDP, SSL, HTTP, and will add GridFTP
• General publish-subscribe including JMS & JXTA and support for RTP-based audio/video conferencing
• General software routing to avoid network problem• Distributed XML event selection using XPATH metaphor• QoS, Security profiles for sent and received messages• Interface with reliable storage for persistent events
Laudable Features of NaradaBrokering Is open source http://www.naradabrokering.org available now;
major new release for SC03 Will have end-point “plug-in” as well as standalone brokers
• end-point is service or user-interface machine Will have a discovery service to find nearest brokers Does tunnel through most firewalls without requiring ports to be
opened Links to NWS (Network Weather Service) style performance
estimation systems Supports JXTA (peer-to-peer network), JMS (Java Message
Service) and more powerful native mode Transit time < 1 millisecond per broker Will have setup and broker network administration module
NaradaBrokering Naturally Supports Filtering of events (streams) to support different end-point
requirements (e.g,. PDA versus desktop, slow lines, different A/V codecs)
Virtualization of addressing, routing, interfaces (OGSI versus pure Web services for example)
Federation and Mediation of multiple instances of Grid services as illustrated by • Composition of Gridlets into full Grids (Gridlets are single computers in
P2P case)
• JXTA with peer-group forming a Gridlet
Monitoring of messages for Service management and general autonomic functions
Fault tolerant data transport Virtual Private Grid with fine-grain Security model
Performance Test : GlobalMMCS1.0 We conducted extensive performance tests on audio and video
servers. Video:
• The test shows that our video server is capable of supporting 100-300 clients if there is only one video sender.
• Video Server Machine : 1.2GHz Intel Pentium III dual CPU, 1GB MEM, RedHat Linux 7.3
Audio: • Our tests show that audio server can support 5 concurrent
sessions (250 participants in total) without any packet droppings. • Audio Server Machine: 2.5GHz Pentium 4 CPU, 512MB memory,
Windows XP machine Scale with logarithmic Broker network
0
10
20
30
40
50
60
0 200 400 600 800 1000 1200 1400 1600 1800 2000
De
lay
(Mill
ise
con
ds)
Packet Number
Average delays per packet for 50 video-clients NaradaBrokering Avg=2.23 ms, JMF Avg=3.08 ms
NaradaBrokering-RTP JMF-RTP
0
1
2
3
4
5
6
7
8
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Jitte
r (
Mill
ise
con
ds)
Packet Number
Average jitter (std. dev) for 50 video clients.
NaradaBrokering Avg=0.95 ms, JMF Avg=1.10 ms
NaradaBrokering-RTP JMF-RTP
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Jitt
er
(M
illis
eco
nd
s)
Packet Number
900 audio clientsNaradaBrokering-RTP
JMF-RTP
Commercial Collaboration
Systems
Placeware
WebEx
Centra Anabas
Robust Web Service Model However hard we work, there will be disconnects and problems Typically difficulty is client and client/server network So we are designing a backup with a web page version of
collaborative page – shared display, text chat etc.• And using instant messenger as backup control function
Have this working (roughly) for Shared PowerPoint but need to extend to other functions
SynchronizationServer (NB)
BackupWeb Service
Fancy Collaboration Backup HTMLInstant Messenger
Collaboration
Application Web Service Model Anabas provides shared display (frame-buffer) as this
works for all applications Shared event is how Tango used to it and this is Access
Grid PowerPoint model• Share events like “New Slide” or “New File”
Our strategy is to convert applications to Web Services and use these to produce shared event• Using NaradaBrokering as system to share events
Have implemented for Scalable Vector Graphics (SVG) and PowerPoint (partially as Microsoft makes hard)
Will extend to OpenOffice if pratical
Web Service Model for Application Development
User Interface
Raw (UI) Events
Logic of Application
Semantic Events
Data
User FacingPorts
Resource Facing Ports
Events as Messages
Rendering as Messages
View
Control
Model
MessagingSystem
Interrupts in traditional monolithic applications become“real messages” not directly method callsNatural for collaboration and universal access
Application as a Web serviceApplication as a Web service
Participating Client
RenderingRendering
User Interface
W3C DOM Events
From Master
FromCollaborationAs a WS
Events
Application as a Web serviceApplication as a Web service
Master Client
RenderingRendering
User Interface
W3C DOM Events
To Collaborative Clients
FromCollaborationAs a WS
Events
Control flow for collaborative SVG clients
Figure 3 Control flow for collaborative SVG clients
Collaborative SVG As A Web Service
NaradaBrokering
SVGBrowser
SVGBrowser
SVGBrowser
SVGBrowser
Identical Programs receiving identical eventsToken determines if browser is moving, waiting for opponent or an observer
SIMD Collaboration
SVGViewer
SVGViewer
SVGViewer
SVGViewer
SVG Model (DOM)
NaradaBrokering
Shared Output portSIMD CollaborativeWeb Service
Non Web ServiceImplementation
NaradaBrokering
MIMD Collaboration
SVGViewer
SVGViewer
SVGViewer
SVGViewer
NaradaBrokering
Shared Input portMIMD CollaborativeWeb Service
SVG Model
SVG Model
SVG Model
SVG Model
NaradaBrokering
PowerPoint as a Sort of Web Service
MasterClient
ConnectableObject
Sink
NaradaBrokering Message Service
User1
1
User2
Usern
PPT as a Web Service
As in Access Grid CollaborativeClient
Holds meta-data(sample of high level semanticevents) like slide number and title
Integration of PDA, Cell phone and Desktop Grid Access
Portals and Web Services Web Services allow us to build a component model (see
CCA) for resources. Each resource naturally has a user interface (which
might be customized for user) Web Service <--> Portlet Natural to use a component model for portal building
displayed web page from collection of portlets• So can customize each portlet and customize which portlets
you want Apache Jetspeed seems good open source technology
supporting this model • Being used by NCSA and DoD HPCMO
WSRP Structure of a Portlet Each Web Service naturally has a user interface specified as “just another
port” This gives each Web Service a Portlet view specified (in XML as always) by
WSRP (Web services for Remote Portals) So component model for resources “automatically” gives a component model
for user interfaces• When you build your application, you define portlet at same time
Application orContent source
WSDL
Web Service
S
R
W
P
Application as a WSGeneral Application Ports Interface with other Web Services
User Face ofWeb ServiceWSRP Ports define WS as a Portlet
PortalUser ProfileAggregate
UI Fragments
Client
WSRP isWeb Services for Remote Portals1st Meeting OASIS March 18 2002
WSDisplay
WSViewer
WS Display
WS Viewer
Event(Message)
Service
Master
WSDisplay
WS Viewer
Web Service MessageInterceptor
Collaboration as a WSSet up Session with XGSP
Application orContent source
WSDL
Web Service
F
I
U
O
F
I
R
O
Shared Output Port Collaboration
OtherParticipants
Text ChatWhiteboardMultiplemasters
WSDisplay
WSViewer
WS Display
WS ViewerEvent
(Message)Service
Master
WSDisplay
WS Viewer
Collaboration as a WSSet up Session with XGSP
WebServic
e
F
I
U
O
F
I
R
O
Shared Input Port (Replicated WS) Collaboration
OtherParticipants
WebServic
e
F
I
U
O
F
I
R
O
WebServic
e
F
I
U
O
F
I
R
O
Collage of Portals
Earthquakes – NASAFusion – DoEComputing Info – DoDPublications -- CGL
Web Services as a Portlet• Each Web Service naturally has a
user interface specified as “just another port” – Customizable for universal access
• This gives each Web Service a Portlet view specified (in XML as always) by WSRP (Web services for Remote Portals)
• So component model for resources “automatically” gives a component model for user interfaces– When you build your
application, you define portletat same time
Application orContent source
WSDL
Web Service
S
R
W
P
Application as a WSGeneral Application PortsInterface with other WebServices
User Face ofWeb ServiceWSRP Ports define WS as a Portlet
Web Services have other ports (Grid Service) to be OGSI compliant
Online Knowledge Center built from Portlets
• Web Services provide a component model for the middleware (see large “common component architecture” effort in Dept. of Energy)
• Should match each WSDL component with a corresponding user interface component
• Thus one “must use” a component model for the portal with again an XML specification (portalML) of portal component
A set of UIComponents
PortletPortlet PortletPortlet PortletPortlet PortletPortlet
XMLRSS, OCS, or otherLocal or remote
HTMLLocal files
JSP or VMLocal templates
WebPageRemote HTML
PortletPortlet
PortletsUser implementedusing Portal API
Portlets
Data
PortletController PortletController
Screen Manager
HTML
PSML
PortletControl
ECS
JSP template
ECS ECS ECS ECS
ECS ECS ECS
ECS Root to HTML
ECS
Turbine ServletJetspeedArchitecture
Portlets and Portal Stacks
• User interfaces to Portal services (Code Submission, Job Monitoring, File Management for Host X) are all managed as portlets.
• Users, administrators can customize their portal interfaces to just precisely the services they want. Core Grid Services
User facing Web Service Ports
Application Grid Web Services
Aggregation Portals(Jetspeed)
Messa
ge S
ecu
rity, In
form
atio
n
Serv
ices
IU and OGCE Portal ArchitectureC
lient
s (P
ure
HT
ML,
Jav
a A
pple
t ..
)
Agg
rega
tion
and
Ren
derin
g
JetspeedInternalServices
Portlet Class:IFramePortlet
Portlet Class:VelocityPortlet
Portlet Class:JspPortlet
Portlet Class:WebForm
Gateway(IU)
Web/Gridservice
Web/Gridservice
Web/Gridservice
Computing
Data Stores
Instruments
GridPortTexas
(Java)COG Kit
Clients Portal Portlets Libraries Services Resources
LocalPortlets
Remoteor ProxyPortlets
EmphasisLargely taken
from other projects
(Jetspeed)
Hierarchical arrangement
Jetspeed Computing Portal: Choose Portlets
4 available portletslinking to Web ServicesI choose two
Choose Portlet Layout
Choose 1-column Layout
Original 2-column Layout
Lists user files on selected host, noahsark.File operations include
Upload, download, Copy, rename, crossload
Tabs indicate availableportlet interfaces.
File management
Sample page with several portlets:
proxy credential manager,submission, monitoring
Provide information about application
andhost parameters
Select applicationto edit
Administer Grid Portal