chep 2003 iosif legrand diamonds distributed agents for mobile & dynamic services iosif c....
TRANSCRIPT
CHEP 2003 Iosif Legrand
DIAMONDS DIstributed Agents for MObile & Dynamic
Services
Iosif C. LegrandIosif C. Legrand
California Institute of Technology
CHEP 2003 Iosif Legrand
This project was developed in collaboration with This project was developed in collaboration with
a team from NUST, Pakistan :a team from NUST, Pakistan :
Saad KianiSaad Kiani
Maria RiazMaria Riaz
Aamir ShafiAamir Shafi
Anjum ShehzadAnjum Shehzad
Umer FarooqUmer Farooq
coordinated by : Arshad Alicoordinated by : Arshad Ali
CHEP 2003 Iosif Legrand
Distributed Service Architecture Distributed Service Architecture Design Considerations Design Considerations
The prototype for Distributed Dynamic Services we are developing gather, disseminate and coordinate time-dependent information and services for specific HEP applications.
It aims to extend and improve the (centralized) client – server architectures on large scale distributed systems.
Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols.
These services should facilitate an efficient use of the distributed resources and provide adaptability.
Reliable on a large scale network distributed environment
Avoid single points of failure
Automatic re-activation of components and services
Scalable & Flexible for adding dynamically new services and automatically replicate existing ones to cope with time dependent load
CHEP 2003 Iosif Legrand
JAVA RMI & JINI JAVA RMI & JINI Distributed Object Systems Distributed Object Systems
LookupServiceStubLookup
Service Skeleton CLIENTServer
“Traditional” Distributed Object Models(CORBA, DCOM)
“IDL” Compiler
Dynamic Code Loading Less Protocols !!
LookupServiceStub
(Proxy)CLIENT
LookupServiceStub
(Proxy)
Service
The Stub is linked to the Client.The Client must know about theservice from the beginning and needs the right stub for it
Any service can be used dynamically
CHEP 2003 Iosif Legrand
Jini & Code MobilityJini & Code Mobility
The purpose of the Jini architecture is to federate groups of devices and soft-ware components into a single, dynamic distributed system.
Jini is a set of specifications that enables services to find each other on a network and allows these services to participate within certain types of operations within the framework. It allows these services to interact in a dynamic and robust way: no user intervention when devices are added or removed, adaptability when services come and go, no prior knowledge of the services implementation are needed simplifying administrative duties.
It provides support for a service oriented architecture with code mobility
JINI extends the Java Platform providing support for Distributed Applications
CHEP 2003 Iosif Legrand
JINI OverviewJINI Overview
The purpose of the Jini architecture is to federate groups of devices and soft-ware components into a single, dynamic distributed system.
Lookup Service
Services are found and resolved by a lookup service. The lookup service is the central bootstrapping mechanism for the system and provides the major point of contact between the system and users of the system.
Leasing
Access to many of the services in the Jini system environment is lease based. A lease is a grant of guaranteed access over a time period. Each lease is negotiated between the user of the service and the provider of the service as part of the service protocol.
Remote Events
The Jini architecture supports distributed events. An object may allow other objects to register interest in events in the object and receive a notification of the occurrence of such an event. This enables distributed event-based programs to be written with a variety of reliability and scalability guarantees.
CHEP 2003 Iosif Legrand
JINI Overview(2)JINI Overview(2)
Transactions Manager
A series of operations, either within a single service or spanning multiple ser-vices, can be wrapped in a transaction. The Jini Transaction interfaces supply a service protocol needed to coordinate a two-phase commit.
The JavaSpaces Service
which can be used for simple communication and for storage of related groups of objects written in the Java programming language
The Mailbox Service
Can be used to provide asynchronous communications (based on any type of messages ) between distributed services
Security
The design of the security model for Jini technology is built on the twin notions of a principal and an access control list. Jini services are accessed on behalf of some entity which generally traces back to a particular user of the system. Services themselves may request access to other services based on the identity of the object that implements the service.
CHEP 2003 Iosif Legrand
Lookup Discovery Service Lookup Discovery Service
A Service Registers with at least one Lookup Service using the same ID.
It provides information about its functionality and the URL addressed from where interested clients may get the dynamic code to use it.
The Service must ask each Lookup Service for a lease and periodically renew it.
If a Service fails to renew the lease, it is removed form the Lookup Service Directory. When problems are solved, it can re-register.
The lease mechanism allows the Lookup Service to keep an up to date directory of services and correctly handle network problems. Discovery mechanism implementation can be enhanced with UDDI , JXTA protocols.Discovery mechanism implementation can be enhanced with UDDI , JXTA protocols.
Service
LookupServiceLookupService
LookupServiceLookupService
CLIENT
Register
Service IDRegister with ID
Ask for a leaseGet a lease for T
jar
Web Server
Publish the“Interface” jar
jar
Web Server
Publish the“Interface” jar
CHEP 2003 Iosif Legrand
Remote EventsRemote Events
Any Interested Service or Client in certain types of events must register. It must ask for an event notification lease and periodically renew it.
If the Event Provider or the Event Listener fails the system will correctly identify such a state as well as the case when such a unit becomes again available.
The lease mechanism provides for any Event Producer a real list of active Listeners. Any Event Consumer is informed if the Producer is alive .
Service
LookupServiceLookupService
CLIENT
Register for Event Notification
Ask for a leaseGet a lease for T
ServiceEvent Provider
CHEP 2003 Iosif Legrand
JavaSpacesJavaSpaces
JavaSpace ->Tuplespace computation model (LINDA) & Distributed Object Model
Loosely coupled/anonymous communication. Participants are no aware of each others identity and the message objects are selected and retrieved using associative lookup based on pattern matching
Persistence of the Tuple Space does not require that participants exist concurrently (timely uncoupled)
Allows Code and Data Migration (tuples=code & data)
Well suited for many parallel algorithms in dynamic configurations.
Process
Process
Process Tuplespace
Take Write
CHEP 2003 Iosif Legrand
Transaction Services Transaction Services
Transactions enable operational grouping into all successful or all failed. It allows a two phase commit operations in a distributed environment.
A Client hands a factory method to a Transactions Manager reference and a lease duration and receives a Transaction Object and a Lease.
The Client pass the Transaction Object to the Services it wants to use asking to do the task “under the transaction”.
If the Client or any participant aborts the transaction the Transaction Manager instruct all the participates to “rollback”.
If the Client asks to commit the transaction, the Transaction Manager queries all the participants and if all are “prepared” it asks all of them for “ roll forward” and commit the transaction.
CHEP 2003 Iosif Legrand
The System FunctionallyThe System Functionally
This prototype system, based on Jini, aims to provide:
Customized dynamic services
“Smart Proxies”
“Mobile Agents”
These conceptual entities should be able to work together and interact in a distributed environment by:
Remote Event Subscription/ NotificationSynchronous and asynchronous communicationCode Mobility Persistent Objects
CHEP 2003 Iosif Legrand
Dynamic Services
The Station Servers allows to dynamical load into the system local services. This procedure may be initiated by a user or by an other service.
The Station Servers register the dynamic services with the lookup services, test them and continuously keep them available by renewing the leases.
The additional “mobile code”, partners may need to use a service is either loaded on a http server or on a JavaSapace.
StationServer
Service Discovery
StationServer
LookupServiceLookupService
Dynamic Service
Dynamic Service
ClientClient
Registration
jar
Web Server
Publish the“Interface” jar
jar
Web Server
Publish the“Interface” jar
Dynamic Service
Dynamic Service
Tuplespace
CHEP 2003 Iosif Legrand
Dynamic Proxies
Smart proxies may :
• Dynamically load part of the service code to the client to improve performance.
• Cache locally frequently used data from the remote service and keep them synchronized
• Discover and select the most convenient places to access common services from the “client” perspective.
LookupServiceProxy
CLIENTStationServer Lookup
ServiceService
“Smart” Proxies extend the functionality of the network services by trying to provide an optimized performance based on the locally available resources, services or data.
CHEP 2003 Iosif Legrand
Mobile Agents
Mobile Agents are automatons entities able to migrate between Station Servers.
Each Station Server offers the runtime environment, priority and controls the access permission for the hosted agents.
Mobile Agents interact with the Station Servers network to get access to the necessary information to migrate and to perform a certain task.
The agent migration is done as a weak migration ( e.g. the agent prepare itself before it is moved and save its state into structured which a serialized)
Mobile Agents should remain small and simple entities. To achieve this, and provide complex functionality they must use and access “static” services hosted by the Station Server network.
CHEP 2003 Iosif Legrand
Mobile Agent Framework DevelopmentMobile Agent Framework DevelopmentDIAMONDSDIAMONDS
Develop a framework that allows for hosting dynamic services in the Develop a framework that allows for hosting dynamic services in the form of Mobile Agentsform of Mobile Agents
Allows for registration/publishing and discovery of servicesAllows for registration/publishing and discovery of services Support for remote event notification Support for remote event notification Trust Establishment between servicesTrust Establishment between services Enabling access to remote resources through AgentsEnabling access to remote resources through Agents Reliability over a large distributed networkReliability over a large distributed network
Avoid loss/duplication of agents – transactions Avoid loss/duplication of agents – transactions Dynamic Interaction with other services toDynamic Interaction with other services to
Use other service’s generated dataUse other service’s generated data Help another service in decision makingHelp another service in decision making
CHEP 2003 Iosif Legrand
Mobile Agents and Code MobilityMobile Agents and Code Mobility
Object moves with its code definitionObject moves with its code definitionAgents are packed in Jar files which serve as the ‘codebase’ for Agents are packed in Jar files which serve as the ‘codebase’ for
the agentsthe agents Marshalled Object for the agent is created which stores the Marshalled Object for the agent is created which stores the
codebase information.codebase information. On un-marshalling, the agent’s definition is requested from its On un-marshalling, the agent’s definition is requested from its
codebase.codebase.Weak MobilityWeak Mobility
Code and Variable state is transferred.Code and Variable state is transferred.
Dynamic Services register their proxies with the discovery services.Dynamic Services register their proxies with the discovery services. Client interested in using these services needs to download the proxyClient interested in using these services needs to download the proxy Client need not be aware of the actual location of the service. Client need not be aware of the actual location of the service. Service’s proxy communicates back to its service for the client.Service’s proxy communicates back to its service for the client.
Redundant codebases for downloading proxy definitionRedundant codebases for downloading proxy definition
““Proxy” is the entire service Proxy” is the entire service
CHEP 2003 Iosif Legrand
Dynamic Services in this framework may have a GUI Dynamic Services in this framework may have a GUI component (act as complex proxy) that is created at runtime, component (act as complex proxy) that is created at runtime, on request by a client.on request by a client.
These user interfaces are transferred to the client and These user interfaces are transferred to the client and opened at the Client sideopened at the Client side
Clients can perform operations on the services, remotely, Clients can perform operations on the services, remotely, through their User Interfaces.through their User Interfaces.
Each client has a unique state for the GUI depending on the Each client has a unique state for the GUI depending on the operations it is performing.operations it is performing.
When a service un-registers, its remote GUI is closed and on When a service un-registers, its remote GUI is closed and on re-registration, it is re-opened automatically at the clients re-registration, it is re-opened automatically at the clients that were previously viewing the GUI.that were previously viewing the GUI.
Remote User Interfaces for Dynamic ServicesRemote User Interfaces for Dynamic Services
CHEP 2003 Iosif Legrand
Transaction Service controls Mobile AgentsTransaction Service controls Mobile Agents
Transactions are used in the framework wherever atomic operations Transactions are used in the framework wherever atomic operations are requiredare required
Agent Movement from one Station to another needs to be atomicAgent Movement from one Station to another needs to be atomic (Un) registration(Un) registration Code and state transfer / Migration Code and state transfer / Migration Re registrationRe registration
Either the move operation is carried out completely and successfully, Either the move operation is carried out completely and successfully, otherwise, it stays where it was.otherwise, it stays where it was. Avoid agents loss and duplicationAvoid agents loss and duplication
Implemented using JINI Transaction ServiceImplemented using JINI Transaction Service
CHEP 2003 Iosif Legrand
SecuritySecurity
Asymmetric keys and CertificatesAsymmetric keys and Certificates Each service signed with owner/administrator’sEach service signed with owner/administrator’s
private keyprivate key Hosting Services (Stations) maintain a trust store Hosting Services (Stations) maintain a trust store
filled with trusted certificatesfilled with trusted certificates
Trust between AgentStation and Agent’s OwnerTrust between AgentStation and Agent’s Owner Key StoresKey Stores
Signing Jars, Generating CertificatesSigning Jars, Generating Certificates Trust StoresTrust Stores
Trusted CertificatesTrusted Certificates Import, Send, Delete CertificatesImport, Send, Delete Certificates
Certificate verification on loading and moveCertificate verification on loading and move
Trust between Agent and Remote ClientTrust between Agent and Remote Client Open Agent – Remotely accessible To All ClientsOpen Agent – Remotely accessible To All Clients Close Agent – Remotely accessible To Owner OnlyClose Agent – Remotely accessible To Owner Only
Agent Verifies if the requesting client is its true ownerAgent Verifies if the requesting client is its true owner
Agent Station
PrivateKeystoreTrust
Store
CHEP 2003 Iosif Legrand
Agent StationAgent Station
An Agent Station is a network service that can host Dynamic Services.Station Servers are dynamically interconnected (peer-to-peer) providinga distributed frame for Services.
Handling Remote events
Keeps a dynamic list of available services
Performs security authentification / encrypted communication
Support for Inter-Service communications (Roaming for mobile Agents)
StationServer
StationServer
StationServer
LookupService
LookupService
Proxy Exchange
Registration
Service ListenerRemote Notification
CHEP 2003 Iosif Legrand
Agent StationsAgent Stations
Agent Stations are deployed at various Agent Stations are deployed at various locations across the network (LAN/WAN) locations across the network (LAN/WAN)
Agents can be loaded by a user.Agents can be loaded by a user.
The agent can move to other stations.The agent can move to other stations.
Where ever the agent goes, it always registers Where ever the agent goes, it always registers itself as a service, making remote itself as a service, making remote administration/monitoring possible for the administration/monitoring possible for the user user
Agent may be open or closedAgent may be open or closed Open – Anyone can view what the Open – Anyone can view what the
agent is doing i.e.it’s a free serviceagent is doing i.e.it’s a free service Closed – The activities of this agent Closed – The activities of this agent
can only be monitored and results can only be monitored and results retrieved by its owner.retrieved by its owner.
The agent mobility and its monitoring are all The agent mobility and its monitoring are all subject to its security credentialssubject to its security credentials
A
B
Registration Service
Agent Station B
Agent Station A
Agent
Agent Proxy
Station Proxy
CHEP 2003 Iosif Legrand
Mobile Agents ServicesMobile Agents Services
Agents are Dynamic & Mobile ServicesAgents are Dynamic & Mobile Services Generic Agent with mobility mechanism to carry the Generic Agent with mobility mechanism to carry the softsoft
statestate Service Agents with registration mechanism & remote UI, Service Agents with registration mechanism & remote UI,
extended from Generic Agentsextended from Generic Agents
Basic Application AgentsBasic Application Agents Test Service AgentTest Service Agent Database Access AgentDatabase Access Agent File System Access AgentFile System Access Agent Search agent Search agent
The Basic Application Agents should be extended for specific The Basic Application Agents should be extended for specific applications applications
CHEP 2003 Iosif Legrand
Database Access AgentDatabase Access Agent
Provides access to databases nearby the host AgentStation.Provides access to databases nearby the host AgentStation. MySQLMySQL MS-AccessMS-Access RmiJDBCRmiJDBC
Host AgentStation keeps record of the dbs on the LAN.Host AgentStation keeps record of the dbs on the LAN.
Load and send the Agent to an AgentStation and then access the Load and send the Agent to an AgentStation and then access the databases there remotely through the Agent’s UI.databases there remotely through the Agent’s UI.
Remote DB BrowserRemote DB Browser
CHEP 2003 Iosif Legrand
Remote File Access AgentRemote File Access Agent
Provides access to the host Agent Station's File System Provides access to the host Agent Station's File System depending on security parametersdepending on security parameters View, Edit filesView, Edit files Remote File BrowserRemote File Browser
CHEP 2003 Iosif Legrand
Read Performance of Agent
0
1
2
3
4
5
6
7
8
9
10
1.95 4.88 7.81 11.72 14.65 17.58
File Size (MB)
Sp
ee
d (
MB
/se
c)
Local Agent
LAN(100Mbps)
LAN(10Mbps)
CHEP 2003 Iosif Legrand
Write Performance of Agent
0
1
2
3
4
5
6
7
8
1.95 4.88 7.81 11.72 14.65 17.58
File Size (MB)
Tra
ns
fer
Ra
te(M
B/s
ec
)
Local Agent
LAN(100Mbps)
LAN(10Mbps)
CHEP 2003 Iosif Legrand
SummarySummary
A prototype system, based on Java & Jini is currently under development to provide a distributed mobile agents service system to help the physics data analysis in a multi-tier Regional Centre architecture.
Code mobility paradigm provides the mechanism for a consistent, Code mobility paradigm provides the mechanism for a consistent, correct invocation of components in large, distributed systems.correct invocation of components in large, distributed systems.
These services need to use Monitoring information services (MonaLisa) in decision making and the strategy they use to adapted to available resources.
This prototype aims prove the advantages of such an approach compared with a client-server model for handling dynamic information in a large scale distributed system