networked systems and services - courses · networked systems and services fall 2018 chapter 6:...
TRANSCRIPT
Networked Systems and ServicesFall 2018
Chapter 6: Cross-organisationalcollaboration and interoperability
Jussi Kangasharju
Markku Kojo
Lea Kutvonen
9/19/2018 Lea Kutvonen 1
Chapter 6: Cross-organisationalcollaboration and interoperability
• 4moves• Interoperation and collaboration of business services at OSI 7th
• Autonomy of organisations --> plug and "pray" --> required utilities for interoperability
• Reliability concepts widened to business level
• How middleware hides lower levels and supports software engineering• How to build a business service
• different levels of computing and communication middleware to choose from
• Communication channel becomes 1st class citizen, to be managed even at runtime
• Composition of services into collaborations: glued together with business processes
• What kind of contract breaches are to be expected, fixed by whom and how
• Reliability concepts in cross-organisational collaborations
• Business process modeling advise in brief (design task involved)
9/19/2018 Lea Kutvonen 2
Outline version
# 1
Some comments on the material• 1st part focuses on current trend vision of interoperability between independent organisations
• Viewpoints cover platform support, software engineering, runtime control , and application domain
• Take home point: everyone should be open for discussing all aspects, make teams with different expert role members
• Overview, no short reading material (multiple active research areas)
• Context in which the design exercise takes place
• 2dn part focuses on the middleware stack as growing concept
• Each layer provides additional concepts and patterns known by experts of that layer
• Key pattern that keeps repeating: open implementation
• 3rd part on reliability requirements design ideas, some shared patterns for alleviating or recovering from risks and failures
• 4th part gives a bit of help for your first business process modeling task
• Article reading assumption: open implementation paper (continues from end-to-end paper)
• Gregor Kiczales, John Lamping, Christina Videira Lopes, Chris Maeda, Anurag Mendhekar, and Gail Murphy. 1997. Open implementation design guidelines. In Proceedings of the 19th international conference on Software engineering (ICSE '97). ACM, New York, NY, USA, 481-490. DOI=http://dx.doi.org/10.1145/253228.253431. Scientific conference paper.
• Design assignment: Business process design in breach recovery situation
• Slides presenting the case and providing some business process modeling hints included in the end of these slides (not presented, questions answered though)
9/19/2018 Lea Kutvonen 3
Outline version #
2
Contents1. Introduction
• System view at OSI 7th layer for networked business• Requirements: generic facilities for aligning business and computing in networked business• Goals of OSI 7th layer for this
2. Layering of the OSI level 7 environment with middleware• Middleware layers: overall approach• Facilities for service-oriented computing• Extending the end-to-end argument to open platforms
→ open middleware → open bindings• Change in software engineering• Summarising: Inter-enterprise collaboration management requirements
3. Reliablity in inter-enterprise collaboration cases• Reliability aspects• Reliability support
4. Intro to BPM modeling for the design task
9/19/2018 Lea Kutvonen 4
How to build up from sockets and RPC
Extending with new arguments (patterns) to SE and platformsEnvironment
+reliabiilty
Outline version #
3
Role of more detailed elements in the big picture
Q1: How middleware hides socket programming and supports software engineering
• Featuring middleware types• (RPC MW already done)
• MOM MW
• Transactional MW
• Object-oriented MW
• Component-oriented MW
• Service-oriented MW
• Model-driven MW
Q2: Open platform (mw, binding) argument
• Reflective systems • connecting abstract, self-managing
model and real system management
• Ways of guiding such system: parameters incl. models, alternative implementations
• Open implementation patterns
9/19/2018 Lea Kutvonen 5
Solving Q1
One agreed learningpoint: transaction mgmt
Business process modeling
• Basic concepts• How to proceed• Relationship to transactions • Opportunities: verification, validation,
generation, execution, …
Extending end-to-end argument
New basicpattern
Current workenvironment
Outline version
# 3
Q3: We have so many levels to choose from, competing solutions on each level. Which one is the right one to use?
Networked Systems and ServicesFall 2018
Chapter 6.1:Interoperation and collaboration of business
services at OSI 7th
Jussi Kangasharju
Markku Kojo
Lea Kutvonen
9/19/2018 Lea Kutvonen 6
1. Introduction • Different environment: large scale networked systems• Endpoints are business services, provided by
autonomous organisations
• Collaboration arranged by interoperabilityinstead of integration
• Applying to various business areas
• Capable of hearing the clients, discussing within the team of business people, software engineers, modeling experts, enterprise architects, global solution architects, competing product engineers, …
• Different meanings for reliability• Building reliable interoperability instead of
assuming it, business level NFP
9/19/2018 Lea Kutvonen 7
End-point
Channel
N-to-M Communication
End-point
End-pointEnd-
point
Endpoint Endpoint
Channel
EndpointEndpoint
Consortia, organisation, end-user agent
Business / computational service
Enhanced middleware stack
Ecosystem infrastructure services
Interoperability utilities
9/19/2018 Lea Kutvonen 8
Goals of OSI level 7 approachesfor networked business
• Separate business solutions from computing solutions• Current software engineering practices tend to choose
computing and engineering technique for everything(efficiency) (PSM = platform specific model)
• Some decisions cannot be judged by software engineers• business decisions, regulatory rules etc → prepare for
variation and evolution in design• Situational busienss decisions→ runtime decisions
• Use of open solutions (PIM = platform independent m, CIM = computation independent m)
• Who decides in the collaboration?• single resposible partner → single decision-maker for
business aspect• → orchestrated solutions• Peers independent as business partners →
choreographied , federated solutions→ collaboration control and management must takeplace at CIM level
• Aim for platform-agnostic solutions• abstract away from the
technical middlewares (open platform?)
• Minimize the expectations on platform facilities required (avoiddomino effect of changes)
• Who are the end-points in thenetworked system?• Not hosts, but for
example applications running• Individual services → instance per
client or collaboration group
9/19/2018 Lea Kutvonen 9
End-point End-point
Chan-nel
End-pointEnd-point
Application layer solutions in use
• For computing the business services …
• … and the channel (here, we call it binding object)• Use a stack of middlewares
• Use them as “open platform” (modifiable behaviour)
• Also middleware remains within OSI layer 7
Physical
Link
Network
Transport
Application
User
9/19/2018 Lea Kutvonen 10
Layers of infrastructure / middleware / utilities
How can computing and communicationbe organised effectively?
global transport of messages, media streams, and massive data sets
global network of computing nodes●sources of data●usable as computing resources
● What concepts are supported in programming environments?
● Which system qualities become transparently supported?
● Which properties can be considered configurable?
● How to implement these effectively?
● How are organisational issues, autonomy and dynamics supported by system models and management facilities?
●How are architecture quality properties, such as privacy preservation, supported?
pervasive services at each node: ●programming and management platform (middleware)●OS, network protocol stack
● collaboratively provided ● management for collaborative application systems● reflects relationship of independent system and its environment
Distribution middleware
How are independent, reusable modules or applications composed to form context-sensitive and adaptive services?
Internet and wireless devices
Collaboration coordination infrastructure
Lea Kutvonen
Runtime / Config composition ofbusiness services , Application level(with business processes)
9/19/2018 11
What is different?Solutions built on• low levels (oversimplification):
• programmers decide on technical choises, business methods, information formats
• software engineering team a single decisionmaking unit
• top level (overcomplication):• software engineers work independently• business and information representation
defined by someone else• Business rules and policies defined elsewhere
and modified as the system is used• middleware support dynamic adaptation to
hetegogeneous, dynamic environment
● collaboratively provided ●management for collaborative application systems● reflects relationship of independent system and its environment
Collaboration coordination infrastructure
Runtime / Config composition ofbusiness services, Application level(with business processes)
9/19/2018 Lea Kutvonen 12
Networked business needs in society
• Social networks
• Contents (e.g. Flickr)
• Expert nets
• Scientific societies, standardisation
• Multicasting, crowdsourcing
• Entertainment, nets of newsagens
• Business networks
• Supply chains, value nets, ...
• Virtual organisations
• Creation supported by breeding environments
• Business service collaborations
• Roles, gains, responsibilities
• In addition, runtime supported by ecosystem facilities
Example domains
• Chrisis management
• National heath-care
• Big data networks
• Innovative nets
AU
TOM
ATE
D C
ON
TRO
L OF C
OLLA
BO
RATIO
N
Trends in enterprise / business computing
• Networked business pull• Value networks• Business networks• Service science
• Technology push• Web Services• Internet of Things /
Services• BPM• Service-oriented
computing
Business processes
Business applications
Enterprise system
Business processes &
services
Business processes &
services
Ecosysteminfrastructure
Ecosysteminfrastructure
General purpose
platforms
General purpose
platforms
9/19/2018 14Lea Kutvonen
Moving towards inter-enterprise collaborations (compositions of business services)
Examples from networked businesspoint of view
• eCommerce• Supply chain
→
Multiple partners + (in)tangible values →
• Value net• Virtual organization
breeding environment• Collaboration (case)
Key concepts• Business process
• Business service
• Exchanged documents (msgs)
• Vocabulary
→ i.e.x-ecosystem rules→
Individual adaptation• Case management
9/19/2018 Lea Kutvonen 15
Examples of ecosystemssingle driver organisation vs democratic peers
Software ecosystem
• software producers make sharedchoise of platform, e.g. Apple, Microsoft, Android
• easy, focus only on development of software-based application services• Shared set of standard protocols in use
for communication transport• Often shared agreement on business
processes, ontologies for informationexchange → additional middleware to support this knowledge
• Middlewares help developers in theirwork
Supporting evolution amongst peers
• More support for evolution of newbusiness ideas and applications• Companies develop new business
ideas and tackle them together
• Free choise for platforms, encapsulation, open middleware
• Enterprise architecture management kind of support• models of business processes
• key terminology of the business field,
• contracting on the new business area
9/19/2018 Lea Kutvonen 17
Features supported Concepts supported
2. Layering the OSI level 7 with middleware
9/19/2018 Lea Kutvonen 18
RPC mw
MOM mw
Transacti-onal mw
Object-oriented mw
Componenent mw
Service-oriented mw
Dockers etcWith dynamicConnections,Deployments, Bindings
Additional layerswithin application layer
* See arch
Model-driven mw
WhateverReliablity not achieved here,May bite you on the upper layers
Adding“genericbusinesssupport”
Separationof PIMand platform→middleware-agnisticity
Morelater
Service-oriented computing architecturesuggests middleware for business domains in general
S1
S2
S3
S1
S2
S3
Portfolio of services
Service selection and interoperability testing
Process
model 2
ST1
ST2
ST3
ST4
ST5
Processmodel
3
ST4
ST5
Portfolio of business process / network types
Network type selection and interoperability enforcing / testing
IdentityCapabil-
itiesTrust /
reputationPortfolio of partners
Selection of partners
OntologiesProcessmodel
3
ST4
ST5
SOASOCStatefull services
Key concepts:ServiceService ecosystems+ others, likeeContracting,Breach management,Trust, privacy
9/19/2018 Lea Kutvonen 19
Service sciences definition of service is more elaborate
◼ eContract management functions
− Service selection
− Negotiation
− Monitoring, breach detection
− Breach management
− Trust management
− Reputation management
− Identification management
− Privacy management
◼ Meta-information repositories
− Service type repository
− Business process model repository
− Service offer repository
– Reputation information flow / repository
Example view onService ecosystem infrastructure
* Platform issues
- ESB, WS*, cloud, agent systems
- Open binding platform
Private support
Private support
Global ecosystem infrastructure
9/19/2018 Lea Kutvonen 20
Takes into consideration:who decides?
Explicit binding: selective & adaptable properties for each channel
9/19/2018 Lea Kutvonen 21
interface → server interface, client interface, peer interface;
may differ but be mapped
late bindingno addresses available at compilation or deployment time; recall DNS
open bindingcommunication channel between objects is explicit, configurable object
service (object, component)
interface
client(object,agent, ...)
interface
binding managementinterface
Recall
9/19/2018 Lea Kutvonen 22
Service engineering on top of the new middlewareDoes it differ from the traditional software engineering practices?
Changes:• Service focus: horisontal interoperability, dynamic• Platform independent operation support
©IJH
Service focus: verticalPortability, reusability
Summarising:
Inter-enterprise collaboration management reqs
• Requires middleware that is • Service-oriented,• Model-driven and• Component-driven
• Requires business domain in which definitions are engineered and made available for standard (best practice) • Business processes• Vocabulary for service types and
collaboration types• Information schema• Contracting processes• Identities for known partners, known
tangibles
• Examples• Swift (banking)• RosettaNet (Nokia, mobiles)• Health-care (Obamacare systems)• Crisis management, international
• Functions• Middleware utilities above• Alternative, selectable business
processes, implementations of different business services, open bindings and binding architecture elements
9/19/2018 Lea Kutvonen 23
3.Reliablity in inter-enterprise collaboration cases• Collaboration
• late, dynamic integration of business services through business processes• Reliability requirements include
• Safety in plug and play / pray patterns since failures become manageable• Detecting unexpected situations and considering business level consequences
• Interoperability• Technical interoperability allow messages exchanged reliably
• Transactionality as needed, understanding of reliability of the platform• Bridging between platforms
• Semantic interoperability support document/message and service identification• Accuracy of communication, transformations (translations)• Heterogeneity support + open bindings
• Pragmatic interoperability enforce shared understanding on business processes and protect from unwanted access to service (use, provision)• Trust, privacy• Only valid collaborations within the ecosystem, acceptable for partners
9/19/2018 Lea Kutvonen 24
Reliablity in inter-enterprise collaboration cases• Recognition and control of non-functional properties
• Extension of QoS (such as throughput, jitter) to business level concepts (in contracts)
• Examples: • non-repudiation, privacy, timeliness, …
• business transactions (different from database transactions, always dirty)
• Conformance to regulatory systems, …
• Defined metrics for monitoring → additional concepts for reliability
• New set can be defined separately for each business domain
• Support mechanisms• New concepts → ontology support for vocabulary
→ new middleware facility implementations
• Protocols for negotiation, business transaction coordination
• Intercepting operations + Monitoring + policies to monitor
• Hidden binding architecture partners
9/19/2018 Lea Kutvonen 25
Middleware supports reliability in multiple ways
• Reliablity between case / collaboration partners’ business services• Trust management seen as part of pragmatic
interoperability• Subjective acceptance criteria by each partner
(organization not forced by others)• Market position bases on feedback: regular breaches
not accepted
• Reliable binging objects• Properties selectable by peers• Properties may include non-repudiation, encryption,
…• Collaborations may be monitored based on the
models available in repositories → breach detection• Failures can be monitored normally → recovery
processes modeled → self-recovery, self-management, … → area of autonomic computing
• Reliability of X-ecosystem as provider of a certain type of business service• Only interface (service type) matters as a contract on
syntax and behaviour tender• Service offers by multiple members
• Partners can be changed (recovery by choosing a new partner with same service type) - note statefulness!
• coopetition• “Automation” of composition
• hiding monolithic vs modular composed services
• Monitoring and breach detection• Collaborations may be monitored based on the models
available in repositories → breach detection• Failures can be monitored normally → recovery
processes modeled → self-recovery, self-management, … → area of autonomic computing
9/19/2018 Lea Kutvonen Open system paradigm applied several times over
26
• Case management approaches• Case (instance) has only part of its business processes known • Decision-points allow insertion of next chunks• Traditional way: business process defined fully, but includes alternative
paths, decision points as variation points
Networked Systems and ServicesFall 2018
Chapter 6.2: How middleware hides lower levels and supports software engineering
Jussi Kangasharju
Markku Kojo
Lea Kutvonen
9/19/2018 Lea Kutvonen 27
Background: Featuring middleware types recollected
• RPC MW • MOM MW
• Transactional MW + transaction management techniques (because above this failures on preceeding layers hurt)
• Object-oriented MW
• Component-based MW + Q1 + facilities for enabling Q2
• Service-oriented MW
• Model-driven MW + facities needed for using open platform use (Q2)
• Reflective MW + connecting abstract, self-managing model and real system management, guiding open platform use (Q2)
9/19/2018 Lea Kutvonen 28
Background
9/19/2018 Lea Kutvonen 29
Featuring middleware types• RPC MW
• MOM MW
• Transactional MW
• Object-oriented MW
• Component-oriented MW
• Service-oriented MW
• Model-driven MW
• Reflective MWRPC mw
MOM mw
Transacti-onal mw
Object-oriented mw
Componenent mw
Service-oriented mw
Dockers etcWith dynamicConnections,Deployments, Bindings
* See arch
Model-driven mw
WhateverReliablity not achieved here,May bite you on the upper layers
Separationof PIMand platform→middleware-agnisticity
Questions to consider• Observation 1:
• MW support for software engineering• component MW adds horizontal view for interoperability and composability,
• developers focus on business logic, componentization (microservices) and late composition
• Q2: Extending end-to-end argument to open implementation (open platform) argument• Component mw, (e.g. dockers) support deployable application units and local mw
support• Q3: Socket programming hidden in open component mw and open binding concepts• Q4: Supporting reliability
• How do I design a reliable business process?• Reliability of collaborations? ecosystem? business processes? business services?
9/19/2018 Lea Kutvonen 30
The answers first ….
• … but we need to take a look on the MW stack first to understand what actually is going on
• Course members have very different backgrounds on this area
• We return to these in the end
9/19/2018 Lea Kutvonen ‹#›
Q3: How middleware hides socket programming
9/19/2018 Lea Kutvonen 31
Find service with type SBind (C, S1, bindingarch)
Formulatemessage& splitto writesize parts
FindChannelArchitecture&Msgschema Intercept for
monitoring
marshall
ESB
encrypt
T
T
TTT
Intercept for monitoring
marshall
ESB
encrypt
Distributiontransparencies
Peer servicefailures
Click to add text
Having first made the problem more difficult than it was ...
Q2: Use of reflective MW and model-hierarchy facilities for open binding pattern
• Recall open binding
• Runtime check for adaptation coherence based on model-hierarchy
• Binding architecture specifies• Structural rules• Behavior restrictions and
obligations• Policies to be followed by
partners (e.g. using same security protocol)
9/19/2018 Lea Kutvonen / Service ecosystems 32
Choises
• Programmatic adaptation (not so safe, additions through model space for check-up)
• Policy-based adaptation (model names)
• Automatic adaptation (reflective mechanisms)
Q4: Autonomy of partners andrequirements of reliability• In your designs, consider
• Start from the need of service, remember to stick with a clearly cut abstraction level• A shared external business process between partners is a good place to fix first
• Consider the correctness of your business processes as a verifiable protocol
• Your services have state, so you can utilise theories on extended state machines
• What are the interoperability challenges in your case• Why and when partners may trust each others, or each others' services
• Current technology choices may not be final
• How well trusted is your source of vocabulary in messaging?
9/19/2018 Lea Kutvonen 33
Autonomy of partners andrequirements of reliability
• For interactions with external services• Plan for check points for policies (rules to restrict or modify behaviours at borders)
• Design recovery processes
• Keep the policies stored, not part of the process definitions, in case of (regular) changes
• Consider replacing access granted / denied being replaced by prohibited/obligated/permitted - and plan reactions in all three cases (negations are not useful, inadequate logics behind)
• On suitable levels, further down the implementation path, remember to check the "call semantics" failures the assumed platform may have – but this is not to be solved on business process model level
9/19/2018 Lea Kutvonen 34
Autonomy of partners andrequirements of reliability
• Following regulations (legal aspects, best business practices, cultural habits)• A software agent capable to several services may be wrapped behind a decision-
making wall that under certain conditions refuses the service to be run
• Note deontic logic in policies (rules): permitted, obligated, prohibited
• In business context, additional considerations include any contracted aspects, nonrepudiation ("cannot deny"), "transactionality" (joint understanding of the outcome), timeliness, trust, privacy
• Application domain usually has some specific quality metrics to consider
9/19/2018 Lea Kutvonen 35
Autonomy of partners andrequirements of reliability
• Reliability of collaborations? ecosystem? business processes? business services?• Define reliability metrics and concepts in ontologis / models• Use quality ensuring methods: verification and validation, search of bad information flow
patterns, analyse regulatory conformance
• Use runtime monitoring (contracts, reliability metrics, enterprise and ecosystem policies)
9/19/2018 Lea Kutvonen 36
RPC middleware summary
9/19/2018 ©Lea Kutvonen / Service Ecosystems 37
• Various versions, recall call semantics
• Essentially, can always cause partial executions
• We cannot be sure whether a full, exactly once semantics has been reached or not
• Need for more exactness → transactions
Transactional middleware
• What is a transaction?
• ACID Properties
• How transactional middleware works?
• Two-Phase Commit
• Two-Phase Locking
1.
What’s a transaction?
• The execution of a program that performs an action by accessing a shared database, usually on behalf of an on-line user.
• For example• Withdraw money from ATM
• Verify credit card sale
• Place bid at an on-line auction
• Submit purchase order
• Further, it is required (as a technical term) refer to ACID properties of the action
ACID Properties
40
• Atomicity – compute fully or not at all
• Consistency - preserve database integrity
• Isolation – execution results are as if computations were run in agreed sequence
• Durability - results keep, failures do not change
Atomicity
41
• No partial results.
• Commit and abort: irrevocable actions
• Abort rolls back operations• Restore data values from before
• For example, temporary private copy that becomes official only at commit, otherwise discarded
• Business services often break atomicity by “dirty actions” that effect the real world or change things outside of the system scope• Money given out from ATM cannot be retrieved
Compensating Transactions
42
• A transaction that reverses the effect of another transaction (that committed) or could not be rolled back properly
• Not always available• Like for real world effects, business transactions
• Still, well-designed applications try to have this
• Also possible to do “forward recovery”• Seek a new acceptable state of affairs
• Flight overbooked → ticket discount, accommodation for delay, etc for volunteers to wait for next flight
Consistency
43
Transactions maintain DB consistency
When each transaction preserves consistency, then the DB stays consistent Referential integrity (no incomplete operations/tables)
Invariants over key data (debits=credits)
Serial execution preserves consistency
Isolation
44
• effect of a set of transactions the same as if they ran independently:• an interleaved execution of transactions is serializable if its effect is equivalent to
a serial one• interleaved computing that just reads or modifies separate datasets can take place with any
interleaved sequence
• Modifications and reads of modified values for further computing should be excluded from interleaved sequences
• Possible methods include pessimistic and optimistic methods• Locking
• Checking sequences for improper orderings and aborting
Durability
45
• Committeed transaction results are permanent
• Methods usually utilise a log• All updates + commits
• When log data physically written on stable storage, results permanent
Two phase locking for avoiding deadlocks
• In transaction systems, resources may become locked as need for writes occur (effects on serialisation)
• Risk of deadlocks arise: processes wait for resources to be released by each other
• Deadlock resolution requires aborting at least one of the transactions
• Deadlocks can be avoided by using 2PL: • Phase 1: collect your resources in given order
• Phase 2: release all resources
• Do not mix phases → no circular waiting situations occur
9/19/2018 Lea Kutvonen / Service ecosystems 46
Two-Phase Commit
47
• Distributed DB system needs control of all DB units succeeding on commits• Pre-commit ensure willingness and stores data• After that, technical failures recoverable
• Phase 1:• Coordinator asks all to “prepare to commit”• Others store data and ack “prepared”
• Phase 2:• When everyone has “prepared”, coordinator tells everyone to do the actual
commit• Any “abort” responses cause global abort
Two phase commit protocol for distributed systems
Application program
Transaction manager
Resource manager (RM)
commit
prepare tocommit response
acknowledgecommitabort
Resource manager (RM)
Two phase commit protocol for distributed systems
Application program
Transaction manager
Resource manager (RM)
commit
prepare tocommit response
acknowledgecommitabort
Resource manager (RM)
Transaction manager
Application program
prepare tocommit etc...
Resource manager (RM)
Transaction processing systemTransactional middleware
Front End Program
Request Controller(routes requests and
supervises their execution)
Database System
Client
Back-End
(Server)
End-User
Transaction Server
requests
Transaction monitors
◼ Transaction processing monitor◼ Supports ACID properties
◼ atomicity – performed completely no not at all◼ consistency – no confused data around◼ isolation – intermediate states not visible, locking ◼ durability – effects of operations permanent
◼ can be dealt with◼ recoverable processes◼ 2PC – two phase commit protocol (commit-prepare, commit)◼ 2PL for locking in strict order to avoid deadlocks at resource reservation
◼ Service aspects◼ management interfaces◼ connections to a variety of databases
Traditional mainframe processing
9/19/2018 Lea Kutvonen / Service Ecosystems 52
network
message manager
request controller
application server
database system
BERNSTEIN198x?
collects transaction input
constructs standard msgsends transaction output
starts and commits transaction
determines type of requestroutes request to proper appl.
executes requested application
Manages shared data
Properties
• Reliability - system should rarely fail
• Availability - system must be up almost all the time
• Response time – second(s)
• Throughput - thousands of transactions/second
• Scalability – #clients → internet size, transaction sizes varies a lot (amount of data, messages, code)
• Load – limited number of transaction types / system
• Security – confidentiality
• Configurability
• Distribution - of users and data
Transactional middleware summary
9/19/2018 ©Lea Kutvonen / Service Ecosystems 54
• Transactions provide exactly once semantics when they are “clean”• Protected area of program between begin and commit
• Transaction management relies on two protocols• 2 phase locking for avoiding deadlocks on resources
• 2 phase commitment for surviving distribution and independent decision making by distributed database nodes
• For real use, also need stable snapshots as recovery points in case the system crashes• Snapshot collected from all database nodes between the transactions
• Can be done because of serialisability
• Database protocols assume atomic, failure free messaging → MOM
Message-Oriented Middleware (MOM)
• Communication using messages
• Messages stored in persistent storage based message queues
55
Client App.
local messagequeues
Server App.
local messagequeues
messagequeues
Network Network Network
Message Servers
Properties of MOM
• asyncronous messages• reliable, fault-tolerant• no loss, duplication,
permutation, cluttering
• persistent subscriptions
• models supported• message queue• request-response• multicast• publish-subscribe
◼ violates access transparency
◼ no support for data heterogenity unless in programming language mapping
◼ no support for transactions
◼ persistent message queues support fault tolerance
56
9/19/2018 Lea Kutvonen / Service Ecosystems 57
Properties of MOM
◼ Basic model: client-server, pipe◼ compare with socket level programming
◼ Topics for variation and development◼ persistent/transient msgs◼ FIFO/priority queues◼ translations of msgs◼ abstractions on msg ordering◼ multithreading, automatic load balancing◼ msg routing (source, cost, changes in topology etc) ◼ secure transfer of msgs (at least between msg servers)
Example: IBM MQSeries
Middleware 58
• One-to-one reliable message passing using queues• Persistent and non-persistent messages
• Message priorities, message notification
• Queue Managers• Responsible for queues
• Transfer messages from input to output queues
• Keep routing tables
• Message Channels• Reliable connections between queue managers
• Messaging API: MQopen Open a queue
MQclose Close a queue
MQput Put message into opened queue
MQget Get message from local queue
9/19/2018 ©Lea Kutvonen / Service Ecosystems 59
MOM not very flexible, provides only the low level transport of messages. Does not suit into situations where multiple independent parties around the distributed system wants to listen to particular notifications.
A more flexible model is needed → publish/subscribe model
Publish/Subscribe communication patternEvent-based middleware
• Publishers (advertise and) publish events (messages)
• Subscribers express interest in events with subscriptions
• Event Service notifies interested subscribers of published events
• Events can have arbitrary content (typed) or name/value pairs
• Consumer event filtering, event batching, event priority, event expiration, logging, internationalization, flow control mechanism
Event Service(event-broker
network)
Subscriber
Subscriber
Subscriber
Publisher
Publisher
Publisher
publish
publish
publish
subscribe
subscribe
subscribe
notify
notify
notify
Filter n
Typed events
constraint
Composite patterns
Topic-Based and Content-Based Pub/Sub
• Event Service matches events against subscriptions• Publishers do not know subscribers + vice-versa
• Dynamic subscriptions and withdrawals
Topic-Based Publish/Subscribe
– Publishers publish events belonging to a topic or subject
– Subscribers subscribe to a topicsubscribe(PrintJobFinishedTopic, …)
(Topic and) Content-Based Publish/Subscribe
– Publishers publish events belonging to topics and
– Subscribers provide a filter based on content of eventssubscribe(type=printjobfinished, printer=‘aspen’, …)
9/19/2018 ©Lea Kutvonen / Service Ecosystems 62
Implementation
• Controller with knowledge on • the subscription rules called filters• registered subscribers for each filter
• Interfaces for• subscribing and adding a filter• withdrawing a subscription
• Operation• Message sent to controller• All appropriate filters found (method critical for performance)
(hash table, for example)• Message forwarded to all subscribers
Object-oriented middleware
• local or remote objects, identified by object references• Remote objects located by remote interfaces, represented by
proxy objects and used in RPC way through Remote method invocation
• Designer view dominated by server interface
object A
proxy object B
OOM OOM
skeleton object B
object B
local remote
objectrequestbroker /object
manager
objectrequestbroker/object
manager
Examples:Java RMICORBADCOM.NET
Object-oriented middleware
9/19/2018 ©Lea Kutvonen / Service Ecosystems 64
• Middleware services• Naming (remote/interoperable object references)
• Trading service for locating based on attributes
• Persistency service
• Transactdion service
• Event and notification service
• Pros and cons• Platform and language independence
• Object constellations static
• remote interface specification by server provider ( only 1 view)
Distribution middleware
9/19/2018 ©Lea Kutvonen / Service Ecosystems 65
• Distribution middleware avoids hard-coding client & server application dependencies on object location, language, OS, protocols, & hardware
Interface
Repository
IDL
Compiler
Implementation
Repository
ClientOBJ
REF
Object
(Servant)in args
operation()
out args +
return
DIIIDL
STUBS
ORB
INTERFACE
IDL
SKELDSI
Object Adapter
ORB CORE GIOP/IIOP/ESIOPS
Object request broker core
Interface
repositoryCORBA services
Object
adapter
Server objects
Client
objects
CORBA architecture
9/19/2018 ©Lea Kutvonen / Service Ecosystems 67
•Common middleware services support many recurring distributed system capabilities, e.g.,
•Transactional behavior
•Authentication & authorization,
•Database connection pooling & concurrency control
•Active replication
•Dynamic resource management
•Examples•CORBA Component Model & Object Services, Sun’s J2EE, Microsoft’s .NET
68
Component middleware
•Components encapsulate application “business” logic
•Components interact via ports•Provided interfaces, e.g.,facets•Required connection points, e.g., receptacles
•Event sinks & sources•Attributes
•Containers provide execution environment for components with common operating requirements
•Components/containers can also
•Communicate via a middleware bus and
•Reuse common middleware servicesSecurityReplication NotificationPersistence
SchedulingA/V Streaming Load Balancing
…
Container
… …
Middleware Bus
Container
…
Component middleware from software developer perspective:Docker as example
9/19/2018 ©Lea Kutvonen / Service Ecosystems 69
• Containers and micro-services (technical)
• Platform services• Building and pushing images to image repository
• Pulling images, provisioning and scheduling containers
• Discovering and binding to services running in containers
• Containers discovering and binding to other containers
• Operating and managing services in containers
Integratable to•Amazon Web Services, •Google Cloud Platform,•IBM Bluemix,•Microsoft Azure,•OpenStack,•Cloud Foundry PaaS
Q1: How middleware supports software engineering
9/19/2018 Lea Kutvonen
• Container role• Deployment of applications on to hosts or registries for clusters or clouds• instantiating application code and maintaining private resources, namespace, dependencies,
connections in+out• platform element that hides underlying technology, for software engineers isolates
• Application logic above bindings, service calls and streams and selected middleware facilities (instead of direct use of sockets)
• Middleware functionality on each technology platform separately
• Docker• Application lifecycle management• Packaging, deyployment, portability• Deployment of IaaS, PaaS, SaaS packages
• Sources, for example:• Kai Wähner, Relation of Middleware to Microservices, Docker, and Cloud-Native Architectures.
(Practitioners)
Open implementation and its adaptation strategies
9/19/2018 Lea Kutvonen 73
Programmatic adaptation• Insert new program elements• Allows introduction of new functionality• Risk for errors, security gaps
Policy-based adaptation• use parameters to indicate strategic
decisions on which modules to utilise• no user code insertion, easy
Automatic adaptation• The implementation includes strategies for
selecting the right configuration for each situation
Q2: Component MW and dockers support open platform and open binding argument
• Instead of fixed ESB products components and dockers allow local selection of • communication channel type
• local technology choices
• For interoperability and ease of deployment• Encapsulation of private
dependencies and resources
• Ease of introduction of new code, registering it, redeploying it
9/19/2018 Lea Kutvonen 74
Service-oriented middleware• Service is composable
• Providers• Produce implementations• Provide service descriptions• Provide runtime environment
• Service description• Advertises capabilities• Interface signature + behaviour• QoS, security, availability, ..
• Technology neutral
• Technology transparency
75
Service directory
Service consumer Service
FindRegister
Invoke
9/19/2018 Lea Kutvonen / Service ecosystems 76
Service-oriented middleware
Within service middlware: Workflow / Business process engines
9/19/2018 ©Lea Kutvonen / Service Ecosystems 77
van der AalstComprehensiveSurvey 2013
ObtainingProcessmodels
Process repository activities
van der AalstComprehensiveSurvey 2013
BP enactment in WS stack
van der AalstComprehensiveSurvey 2013
WfMC reference model,operations and interfaces
9/19/2018 ©Lea Kutvonen / Service Ecosystems 80
WfMC RM1999
WfMC reference model
9/19/2018 ©Lea Kutvonen / Service Ecosystems 81
WfMC RM1999
WfMC distributed solution
9/19/2018 ©Lea Kutvonen / Service Ecosystems 82
WfMC RM1999
9/19/2018 ©Lea Kutvonen / Service Ecosystems 83
van der AalstComprehensiveSurvey 2013
Data flow &Command flow
Explicit binding concepts
9/19/2018 Lea Kutvonen / Service ecosystems 84
Enterprise service bus ESB
• Take the following incredients and mix• MOM mw + RPC feel• Transactional mw• routing, identities, name service• SOA, i.e. service registry, service-type based
routing• Publish/subscribe• Business process engines for orchestration• Semantic interoperability support (incl.
ontology)• Bridges for the essential vendor specific
communication platforms• SLA management (QoS monitoring and
control)• Encryption• Multi-language support
9/19/2018 Lea Kutvonen 85
Something missing? New themes?
• Replace by Cloud?• Communication protocols and SOA services missing
• Storage and transaction processing on DB natural?
• Bridge ESBs to I-ESB• Joining ontologies?
• Security solutions? Limits between intra- and inter-enteprise collaborations
• Architecting communication patterns separately and using open binding concept for injecting all that into the ecosystem
• In comparison, IoT protocols more specific to context, still grasping some of the same (old) patterns
Distributed system support mechanisms
Consensus formation• Agreeing on a decision requiring preferrably
all patners (but some can be omitted, due to unreachability or non-cooperative state)
• Collect required Quorum• Quorum = (n/2) +1 • Necessary in case of network / group
partititioning situations → at recovery state, the decision cannot change, all will eventually obey → eventual consistency
• Ensure leader available for reporting decision state• Leader (or contract agent replica) always present• When leader fails, re-election of leader in
identity order
9/19/2018 Lea Kutvonen 87
Distributed system support mechanisms
Ontology is• formal, explicit specification of a shared
conceptualization
• semantic structure which encodes the implicit rules constraining the structure of a piece of reality
• Ontology defines• A common vocabulary• The meaning of terms• Interrelation of terms
• extensional class• if and only if it is characterized solely by its
membership
• intensional class• Created by categorization rules
Ontologies are used in computer science
• Expert systems with knowledge databases (facts, axioms) for example for• Diagnostics• Business decisions• Predictions
• Decision-making support based on “big data”• Business intelligence• Innovation support• Social knowledge management
9/19/2018 Lea Kutvonen 88
Model-driven middleware key elements
Model Model is about• Structure:
• Data, architecture, configuration
• Behaviour• User interface• Access control• Business process
• On any system aspect
9/19/2018 Lea Kutvonen 89
Interpretation:
Mental model
Reality
(observed
or imagined)
Described model
Model specification• Precise instructions for construction, for example,
code generation
Metamodel hierarchy and model hierarchy
Metamodel • Metamodel is a model of a model• Metamodel determines for a category
of models• structure, properties and relationships or• representation
• Metamodel enables• Comparison or categorisation of models• Determination of belonging to such a
category• Refinement of meta-model to
alternative models• Transformation of representational
formats of models of the same category
Model hierarchy (MOF) is an ontology too
9/19/2018 Lea Kutvonen 90
Ongologies on entitities and their representation for interoperable comptuing
Model-driven middleware key elements
Model driven architecture
Model driven hierarchy
and engineering• Metamodel enforces
conformance
• Model transformations• Potentially automated• For example, code generation, or
reversing: changes in code reflected back to more abstract model
• Models can be used for monitoring too (runtime use)
• Transformations• require mapping rules on
metalevel• transformation specifications
expressed in specific language
9/19/2018 Lea Kutvonen 91
Reflective middleware (reflective system pattern)
9/19/2018 Lea Kutvonen 92
Reflective systems include
• Model of their own structure, state, behaviour
• Self control• Deduction of need for structure or
state change• Capability of performing such
change
Source: P Grace, G Blair, Reflective middleware. In the Handbook of Mobile Middleware. CRC Press 2006.
Networked Systems and ServicesFall 2018
Chapter 6.3: Business process modeling advise in brief for the design task
Jussi Kangasharju
Markku Kojo
Lea Kutvonen
9/19/2018 Lea Kutvonen 93
Outline• Why business processes are
needed?
• Business process model vs information flow model
• How a business transaction differs from a transaction (with ACID properties)?
• My own BPM• Basic concepts of BPML• Requirements on autonomy• Reliability definitions:
• by whom & how
Networked business needs in society
• Social networks
• Contents (e.g. Flickr)
• Expert nets
• Scientific societies, standardisation
• Multicasting, crowdsourcing
• Entertainment, nets of newsagens
• Business networks
• Supply chains, value nets, ...
• Virtual organisations
• Creation supported by breeding environments
• Business service collaborations
• Roles, gains, responsibilities
• In addition, runtime supported by ecosystem facilities
Example domains
• Chrisis management
• National heath-care
• Big data networks
• Innovative nets
AU
TOM
ATE
D C
ON
TRO
L OF C
OLLA
BO
RATIO
N
From enterprise centric automation to value nets
9/19/2018 Lea Kutvonen / Service ecosystems 96
Business process properties
• Why business processes are needed?• Societal and business needs• Technical opportunities:
verification, validation, generation, execution, model-driven engineering
• Business process model vs information flow model• Languages for both, focus choice• E.g., cumulating nationwide big data, utilising
adaptive case management
• How a "business transaction" differs from a (database style) transaction (with ACID properties)?• Business processes do have real world
consequences --> no rollback, but compensations
• Sometimes just agreed acceptable future state possible to reach
• e.g. passanger left on an airport mid-trip due to an oversale of tickets
• Resolution may have wider scope
• Business transaction may mean too many things, including running an individual case (instance) of business process (and whatever supporting processes and services it associates with)
9/19/2018 Lea Kutvonen 97
Concepts for inter-enterprise collaborations (I.e., concepts for composition of business services)
Examples from networked businesspoint of view
• eCommerce
• Supply chain
• Multiple partners + (in)tangible values
• Value net
• Virtual organization breeding environment
• Collaboration (case)
Key concepts• Business process
• Business service
• Exchanged documents (msgs)
• Vocabulary
i.e.
• x-ecosystem rules
• Individual adaptation
• Case management
9/19/2018 Lea Kutvonen 98
My own BPM:
Languages: BPMN as an example
9/19/2018 Lea Kutvonen / Service ecosystems 99
• Version numbers and alternative languages have the same key concepts
• Business Process Modeling Notation
• By BPMNI (... Initiative) → OMG (by org. merger)
• Does:• Supports business process management for technical and business users• Bridge communication gap between business process design and implementation
• Does not:• No state transitions, no functional decomposition, no organisational hierarchies, no
data modeling
• Process flows can be either executable or non-executable
• Finnish public sector agencies must use for their enterprise architecture work and for publishing their business service interface behaviour descriptions
Basic concepts• Process
• Flow of activities, decisions, data and events
• Collaboration• Conversations and interactions (also processes)
• Choreography– Between pools– Participant
interactions using atomic tasks
Basic concepts
9/19/2018 Lea Kutvonen / Service ecosystems 101
• Pools• Independent
organisation
• Black box option
• Lane• Independent actor
(role) within organisation
• Organise and categorise activities
Private and public processes
9/19/2018 Lea Kutvonen / Service ecosystems 102
Collaboration modeling aspects
9/19/2018 Lea Kutvonen / Service ecosystems 103
Pools with a gap in between: pools represent independent "organisations"
Within each pool, orchestrated workflows (single decision-making & responsibiilty domain) running (note begin and end circles, compare with "daemons" in computing system)
Exchange of messages (information or control notifications) across "gap"forms choreography, I.e. cross-organisational discussion
Process: Activity sequence & “token”
9/19/2018 Lea Kutvonen / Service ecosystems 104
• Activity, process, task
• Normal flow
• Uncotnrolled flow
• Conditional flow
• Exception flow
• Default flow
• Compensation flow
• Fork – merge; sequence flow looping
• Multiple instances
• Process break
• Transactions, nested, ...
Activities
9/19/2018 Lea Kutvonen / Service ecosystems 105
• atomic, non-atomic (compound)
• no need to show further detail
• variations: service task; send, receive; User, manual; Business rule; script, ...
• Has self-contained start and end events, sequence flows from parent process must not cross boundary (no overall reusability)
• Completed fully or compensated
Gateways
9/19/2018 Lea Kutvonen / Service ecosystems 106
• Controls sequence, decisions for branching, merging,
• Can be based on events or evaluation of expression
• Exclusive, inclusive, parallel gateways• One or more alternative parallel
executions allowed
Events: start, intermediate, end
9/19/2018 Lea Kutvonen / Service ecosystems 107
• Error: throw error, close all parent activities hierarchically
• Escalation: code to be passed to nearest parent Activity
• Cancel: send cancel to all involved entities in transaction
• Compensation: indicates which task to perform for compensating the target activity (transaction)
Message flows
9/19/2018 Lea Kutvonen / Service ecosystems 108
• Processes are abstract
• Runtime performances are concrete instances• Correlation, correlation keys to keep isolated operations
associated
How to design business processes and independent services?
9/19/2018 Lea Kutvonen / Service ecosystems 109
• Process definition example video (4 mins)• capturing business process model from a use case; not enough
• Between collaborating processes (5 mins)• Want each organisation (or role) to have independent business processes with
start and stop, and choreographies in between of the swimlines
• Seek for a suitable level of abstraction: Abstract models must• Suit the ecosystem environment in which the service must collaborate with
others, be adaptable for multiple partners,• Be adaptable to multiple heterogeneous environments by wrappers,
transformations of messages, intelligent bridges etc• Complete enough for verification ()• Support the engineering methodology (e.g. MDE, model-driven engineering,
• Notation poster v.1
• Notation poster v.2
Verification:Finite automata and global state analysishelp seeing whether your process makes sense
9/19/2018 Lea Kutvonen / Service ecosystems 111
a b
start
0?
1!
A B
start0!
1?
aA – bB – bB – bB – bB - ....
D C
start0?
1!
0!
DC - BC – BC – BC –!
D RR (locked, likely to be a design error)
Shared state graph for 1st and 2nd
Shared state graph for 1st and 3rd
Must check all paths:All reached states acceptable and sufficient?
9/19/2018 Service Ecosystems / Lea Kutvonen 112
Service and collaboration model engineeringDoes it differ from the traditional engineering practices?
9/19/2018 Service Ecosystems / Lea Kutvonen 113
Service and collaboration model engineeringDoes it differ from the traditional engineering practices?
Design task 3:Breach recovery in cross-
organizational business process
Lea Kutvonen
Fall 2017
Notes on the design task
• The situation is not realistic, for a few reasons• The limitations bring up a conditions under which collaboration reliability
threats become present
• The limitations keep it small enough to handle
• The limitations make the solutions similar enough to walk through the intended learning processes
• See the grading hints
9/19/2018 Lea Kutvonen 115
Outline
• Context: Networked business and open business service ecosystem
• Business scenario: cross-organizational collaboration
• Design task
• Grading hints
• Recall challenge:• Balancing of system reliability and business expectations
Context: Networked business
• Autonomy (independence) of organisations• Platform• Business services provided• Decision on whom to contract collaborations
with• Setting internal (private) enterprise policies
(related to strategies): what business processes to invest in, which orgs to trust/distrust or through which process to decide on trusting, loyalty program member identification and variations in business process for the members etc)
• Local regulatory policy availability
• X-ecosystem support facilities• Service discovery by type• Repository of business process models (and
chunks for them)• Ecosystem policies
• Instead of access-rights, organisationsuse deontic model in contracting• Negotiated contract on the collaboration
states:• Orgs X and Y are obligated to provide
service Type1 to other collaboration members
• Org ZY is permitted to provide service Type1 (identified as “S1-byZY-policiesP1”)
• Org ZN is prohibited from providing service Type1
• Org A is permitted to use service Type1 (but not prohibited or obligated from using it from X)
• Contract shows that things may work, but do not quarantee it in all runtime situations and circumstances → runtime breach detection and recovery needed
Business scenario:cross-organizational collaboration• Group of organizations from different countries sell and transport bulky tangible
items across national boundaries• Business model realistic (existing): international selling and transportation requires knowledge
of international and multiple national regulatory systems and failures to follow those laws have sanctions (loss of merchandize, prison sentence, huge cost)• Regulatory system → directives, legal system, sometimes even business best practices
• Collaboration focuses on knowledge of contracting, regulation and support facilities for import and export• Organizations make contracts themselves, bill clients, split costs and profits• However, they outsource transportation
• Outsourcing: external service provider takes care of that part of operation• Transportation can only be arranged by national carriers by national client
• Collaboration may involve several business processes between required partners• Internal business processes of the partners may be hidden or schetched only in the “need to
know” basis from the other partners point of view
Design task
• Work in steps:1. Design the key business processes between the partners using BPMN
(business process modeling notation) • When identifying the partners, note also the outsourcing needs and regulators (through
stored rules)• In BPMN modeling
• an autonomous organization must have a role (separate set of swimlines) while a unit or computing system may have a swimline associated to that role
• Each role (and swimline, unless you have a justified reason) must have a start node and an end node as the behaviours of separate services of organisations and units are independent (choreographies → starts and stops per role, orchestrations → may have sequences crossing swimlines within a single role)
• Supporting processes can be left out for simplicity and size, also use composed task blocks for parts that do not include interesting elements• exchange of key information, exchange of control signals (start, your turn, stop, did you start
processing my request?, etc as you design)
Design task (cont)
2. Identify potential breaches • Breach = failure to operate according to the contract• Note that the business transactions (requests and activities belonging together between
business partners) are not clean in the database transaction sense• If your transport fails, the bulky goods are in wrong place, and you need activities for moving them
to an acceptable location and the cost and sanctions needs to be determined in a way that depends on the situation, partners, country in which the breach took place, etc.
3. Design a couple of alternative recovery business processes4. Consider where in your system architecture you can place them and why. What
benefits and drawbacks there are for these choises?5. Did you remember to consider who in the collaboration scenario gets to decide
which of your recovery processes is used? 6. Complete your overall design by ensuring all collaboration partners know when (if)
the purchase reached the final location and know whether any breach recovery (with potential sanctions) took place during the collaboration operation.
7. Make sure you discuss the choices you made and assumptions behind them.
Recall challenge + grading hints
• Remember to include into your submission commentary on balancing of system reliability and business expectations in the scenario• What is your definition of reliability here?• What are the facilities for supporting reliability?• What kind of costs is caused? • While the cost seems huge, the same facilites support engineering, correct local operation,
agility on market place, following and adapting to regulatory systems and other well worth goals in addition to the reliability goal.
• There is a lot of space for using your imagination here, no need to check any laws etc for this design. The scenario is not fully realistic.
• Focus on the use of the business scenario page ideas
• From BPMN, very basic use skills suffice, syntax errors do not matter. • May use paper and pen or some BPMN free tools• Some tool examples here