soa4all d7.7 use case evaluation 1 - cordis...0.8 25.02.2011 updated sections 4.2 and 4.3, executive...

Project Number: 215219

Project Acronym: SOA4All

Project Title: Service Oriented Architectures for All

Instrument: Integrated Project

Thematic Priority: Information and Communication Technologies

D7.7 Use Case Evaluation

Activity 3: Use Case Activities

Work Package 7: End-user Integrated Enterprise Service Delivery Platform

Due Date: 30/04/2011

Submission Date: 28/04/2011

Start Date of Project: 01/03/2008

Duration of Project: 38 Months

Organization Responsible of Deliverable: UNIMAN

Revision: 1.6

Authors: Juergen Vogel (SAP), Georgi Pavlov (SAP), Sonja Meyer (SAP), Abdallah Namoun (UNIMAN), Nikolay Mehandjiev (UNIMAN), Ali Owrak (UNIMAN), Usman Wajid (UNIMAN), Yosu Gorroñogoitia (ATOS)

Reviewers:

Project co -funded by the European Commission within the Seventh Framework Programme (2007 -2013)

Dissemination Level

PU Public X

SOA4All –FP7 – 215219 – D7.7 Use Case Evaluation

© SOA4All consortium of 132

Version History

Version Date Comments, Changes, Status Authors, contributors, reviewers

0.1 11.01.2011 ToC Juergen Vogel (SAP)

0.2 14.01.2011 ToC update Juergen Vogel (SAP)

0.3 27.01.2011 Update Chapters 1, 5 Georgi Pavlov (SAP), Juergen Vogel (SAP)

0.4 31.01.2011 Update Chapter 2, 3, 5 and 6, Section 4.4

Juergen Vogel (SAP), Nikolay Mehandjiev (UNIMAN)

0.5 31.01.2011 Update Chapter 2, 3, and 5 Juergen Vogel (SAP), Georgi Pavlov (SAP), Yosu Gorroñogoitia (ATOS)

0.6 11.02.2011 Update section 6.3 Abdallah Namoun (UNIMAN), Ali Owrak (UNIMAN)

0.7 21.02.2011 Update Sections 6.2, 6.3, added Appendix A

Juergen Vogel (SAP)

0.8 25.02.2011 Updated Sections 4.2 and 4.3, Executive Summary Juergen Vogel (SAP)

0.9 09.03.2011 Updated Sections 4.5, 4.6, and 4.7 Sonja Meyer (SAP)

1.0 14.04.2011 Updated Section 6, added Appendix D Usman Wajid (UNIMAN)

1.1 15.04.2011 Overall review and updates in various sections

Nikolay Mehandjiev and Usman Wajid (UNIMAN)

1.1 18.04.2011 Review Sven Abels (TIE)

1.1 24.04.2011 Review Sandra Stincic (BT)

1.2 26.04.2011 Address reviewer comments and minor fixes Usman Wajid (UNIMAN)

1.3-1.4 27.04.2011 Final polishing and minor additions Nikolay Mehandjiev (UNIMAN)

Ali Owrak (UNIMAN)

1.5/1.6 28/04/2011 Final check and corrections for submission

Julia Wells (ATOS)



Table of Contents

LIST OF FIGURES ________________________________________________________________ 5

LIST OF TABLES ____________________________________ _____________________________ 6

GLOSSARY OF ACRONYMS ______________________________ _________________________ 8

EXECUTIVE SUMMARY ___________________________________________________________ 10

1. INTRODUCTION ____________________________________________________________ 11

1.1 PURPOSE AND STRUCTURE OF THE DOCUMENT ___________________________ 11 1.2 FUTURE WORK ________________________________________________________ 12

2. USE CASE STORYBOARDS ______________________________ ____________________ 13

2.1 STORYBOARD 4: WIZARD-BASED SERVICE COMPOSITION ___________________ 13 2.1.1 Overview ____________________________________________________________ 13 2.1.2 Detailed Storyboard ____________________________________________________ 14

2.2 STORYBOARD 5: CITY OF X – SERVICE PROVISIONING AND COMPOSITION ____ 16 2.2.1 Overview ____________________________________________________________ 16 2.2.2 Detailed Storyboard ____________________________________________________ 17

2.3 STORYBOARD 6: UNIVERSITY – SERVICE COMPOSITION _____________________ 21 2.3.1 Overview of Student Registration Scenario __________________________________ 21 2.3.2 Detailed Storyboard ____________________________________________________ 23 Overall Tasks for Users _______________________________________________________ 23

3. DEMONSTRATOR ARCHITECTURE ____________________________________________ 27

3.1 COMPONENTS OVERVIEW _______________________________________________ 27 3.1.1 Storyboard 4 _________________________________________________________ 27 3.1.2 Storyboard 5 _________________________________________________________ 28 3.1.3 Storyboard 6 _________________________________________________________ 28

4. ECONOMIC EVALUATION: STAKEHOLDER CONSULTATIONS ____ _________________ 29

4.1 M11: PUBLIC SECTOR EXPERT INTERVIEW _________________________________ 29 4.2 M18: SAP WORLDTOUR _________________________________________________ 29 4.3 M22: PUBLIC SECTOR EXPERT GROUP DISCUSSION ________________________ 31

4.3.1 Setup _______________________________________________________________ 31 4.3.2 Discussion Notes ______________________________________________________ 31 4.3.3 Conclusion ___________________________________________________________ 32

4.4 M26: CONSULTATIONS WITH GENERAL USERS OF E-GOVERNMENT SERVICES _ 33 4.4.1 Setup _______________________________________________________________ 33 4.4.2 Analysis of results _____________________________________________________ 33

4.5 M30: PUBLIC SECTOR EXPERT ___________________________________________ 33 4.5.1 Setup _______________________________________________________________ 34 4.5.2 Content _____________________________________________________________ 34 4.5.3 Analysis _____________________________________________________________ 35

4.6 M31: JSWEB ___________________________________________________________ 35 4.6.1 Setup _______________________________________________________________ 35 4.6.2 Content _____________________________________________________________ 35 4.6.3 Analysis _____________________________________________________________ 35

4.7 M33: SAP RESEARCH DAY _______________________________________________ 36 4.7.1 Setup _______________________________________________________________ 36 4.7.2 Analysis _____________________________________________________________ 36

5. TECHNICAL EVALUATION ______________________________ ______________________ 39

5.1 M18: STORYBOARD 1 ___________________________________________________ 40 5.2 M33: STORYBOARDS 2 – 3 _______________________________________________ 43 5.3 M34: STORYBOARD 5 ___________________________________________________ 46



5.4 M37: STORYBOARD 6 ___________________________________________________ 48

6. USABILITY EVALUATION ______________________________ ______________________ 49

6.1 M24: STORYBOARD 4 ___________________________________________________ 49 6.1.1 Method ______________________________________________________________ 49 6.1.2 Results ______________________________________________________________ 50 6.1.3 Discussion ___________________________________________________________ 51

6.2 M28: STORYBOARD 3 ___________________________________________________ 51 6.2.1 Method ______________________________________________________________ 52 6.2.2 Results and Discussion _________________________________________________ 53

6.3 M34: STORYBOARD 5 ___________________________________________________ 55 6.3.1 Aim and Objectives of Study _____________________________________________ 55 6.3.2 Participants’ Background ________________________________________________ 55 6.3.3 Evaluation Procedure and Scenario _______________________________________ 56 6.3.4 Analysis Method_______________________________________________________ 58 6.3.5 Results ______________________________________________________________ 58

6.4 M37: STORYBOARD 6 ___________________________________________________ 69 6.4.1 Participants’ Background ________________________________________________ 69 6.4.2 Evaluation Procedure and Scenario _______________________________________ 70 6.4.3 Analysis Method_______________________________________________________ 71 6.4.4 Results ______________________________________________________________ 71

7. VALUE PROPOSITION OF SOA4ALL ______________________ _____________________ 78

7.1 STATE-OF-THE-ART BEFORE SOA4ALL ____________________________________ 78 7.2 VALUE PROPOSITION OF SOA4ALL _______________________________________ 78

8. CONCLUSIONS _____________________________________________________________ 79

9. REFERENCES ______________________________________________________________ 80

APPENDIX A: M28 USER STUDY WORKBOOK _______________ ________________________ 81

APPENDIX B: M33: SAP RESEARCH DAY _________________ __________________________ 90

APPENDIX C: M34 USER STUDY WORKBOOK _______________ ________________________ 91

APPENDIX D: M37 USER STUDY WORKBOOK _______________ ________________________ 99

APPENDIX E: USE CASE STORYBOARDS __________________ ________________________ 110



List of Figures

Figure 1: Process Model for Storyboard 5 ............................................................................................. 17

Figure 2: Student registration process model........................................................................................ 22

Figure 3: SOA4All Overall Architecture with WP7 modifications ........................................................... 27

Figure 4: Background of survey stakeholders ....................................................................................... 36

Figure 5: Software development experience ......................................................................................... 37

Figure 6: SOA4All concept .................................................................................................................... 37

Figure 7: SOA4All in the Public Sector .................................................................................................. 38

Figure 8: Exploiting SOA4All at SAP ..................................................................................................... 38

Figure 9: Evaluation Method.................................................................................................................. 52

Figure 10: Participants Achieving the Optimal Solution ........................................................................ 54

Figure 11: Participants Knowing What to Do ......................................................................................... 54

Figure 12: Task Completion Times ....................................................................................................... 55

Figure 13: Participants’ software development and design experience ................................................ 56

Figure 14: Participants’ semantic annotation, process modelling, and service lifecycle management experience .................................................................................................................................... 56

Figure 15: Participants’ skills and experience summed up ................................................................... 57

Figure 16: Average rating of WSMO-Lite Editor .................................................................................... 60

Figure 17: Average rating of Consumption Platform ............................................................................. 61

Figure 18: Average rating of Process Editor ......................................................................................... 62

Figure 19: Average usability rating of SOA4All Studio .......................................................................... 63

Figure 20: Average rating of WSMO-Lite Editor .................................................................................... 72

Figure 21: Average rating of Consumption platform ............................................................................. 72

Figure 22: Average rating of Process Editor ......................................................................................... 73

Figure 23: Average ratings for Overall Soa4All Studio .......................................................................... 74

Figure 24: Annotation Satisfaction Rating ............................................................................................. 75

Figure 25: Consumption Satisfaction Rating ......................................................................................... 76

Figure 26: Process Modelling Satisfaction Rating ................................................................................. 76

Figure 27: SOA4All Studio Satisfaction Rating ..................................................................................... 77

Figure 28: SOA4All Components and Users ....................................................................................... 110

Figure 29: Storyboard 1 Initial Process Model (Screenshot from SOA4All Process Editor) ............... 112

Figure 30: Process Modeling with SOA4All ......................................................................................... 113

Figure 31: Process Execution with SOA4All ....................................................................................... 113

Figure 32: Service Provisioning in SOA4All ........................................................................................ 119

Figure 33: Storyboard 3 Process Model (snapshot from Process Editor) ........................................... 128



List of Tables

Table 1: Detailed User Manual for Storyboard 4 ................................................................................... 15

Table 2: Services for Storyboard 5 ........................................................................................................ 17


Table 4: Services for Storyboard 6 ........................................................................................................ 23


Table 6: SOA4All Components used in Storyboard 4 ........................................................................... 27



Table 9: Attitudes to Benefits, Risks and Supporting Approaches ........................................................ 30

Table 10: Overview of Use Case Requirements ................................................................................... 40

Table 11: M18 Fit for Purpose Evaluation Results ................................................................................ 42




Table 15: Questionnaire for Wizard-based BPM ................................................................................... 50

Table 16: Evaluation time results (time in minutes) .............................................................................. 51

Table 17: Evaluation errors results (number of errors) ......................................................................... 51

Table 18: Results from Questionnaire ................................................................................................... 53

Table 19: Participants Background ....................................................................................................... 53

Table 20: Positive user comments in regard to WSMO-Lite Editor ....................................................... 59

Table 21: Positive user comments in regard to Consumption Platform ................................................ 60

Table 22: Positive user comments in regard to Process Editor ............................................................ 61

Table 23: Positive user comments in regard to Assisted Modelling (Process Editor) ........................... 61

Table 24: Positive user comments in regard to the SOA4All Studio ..................................................... 62

Table 25: Usability problems within the SOA4All Studio and its tools .................................................. 64

Table 26: User suggestions to improve the WSMO-Lite Studio ............................................................ 65

Table 27: User suggestions to improve the Consumption Platform ...................................................... 67

Table 28: User suggestions to improve the Process Editor .................................................................. 68

Table 29: User suggestions to improve the Assisted Modelling............................................................ 69

Table 30: User suggestions to improve the SOA4All Studio ................................................................. 69

Table 31: Positive user comments for WSMO-Lite Editor ..................................................................... 71

Table 32: Positive user comments for Consumption platform ............................................................... 72

Table 33: Positive user comments for Process editor ........................................................................... 73

Table 34: Positive user comments for SOA4All Studio ......................................................................... 74

Table 35: User suggestions ................................................................................................................... 75

Table 36: Actors in the different storyboards....................................................................................... 112

Table 37: Detailed User Manual for Storyboard 1 ............................................................................... 118



Glossary of Acronyms

Acronym Definition

API Application Programming Interface

BPEL Business Process Execution Language

BP Business Process

BPM Business Process Modeling

BPMN Business Process Modeling Notation

BU Business User

CMS Content Management System

CRM Customer-Relationship Management

D Deliverable

DSB Distributed Service Bus

EC European Commission

EJB Enterprise Java Beans

EP Enterprise Portal

ERP Enterprise Resource Planning

ES Enterprise Service

ESR Enterprise Service Repository

ESB Enterprise Service Bus

EU European Union

EUD End User Development

GDT Global Data Type

GUI Graphical User Interface

HCM Human Capital Management

HTML HyperText Markup Language

HTTP Hypertext Transfer Protocol

ID Identifier

ISO International Organization for Standardization

IT Information Technology

M Month

NLP Natural Language Processing

OWL Web Ontology Language

QoS Quality of Service

RDF Resource Description Framework

RDFS RDF Schema

REST REpresentational State Transfer

SaaS Software as a Service



SAWSDL Semantic Annotations for WSDL

SCM Supply Chain Management

SC Service Composition

SD Standard Deviation

SEE Semantic Execution Environment

SEI Service Endpoint Interface

SME Small and Medium Enterprise

SOA Service-Oriented Architecture

SOA4All Service-Oriented Architectures for All

SOAP Simple Object Access Protocol

SPARQL SPARQL Protocol and RDF Query Language

SRM Supplier Relationship Management

TCO Total Costs of Ownership

UI User Interface

URI Uniform Resource Identifier

URL Uniform Resource Locator

USD United States of America Dollars

WP Work Package

WS Web Service

WSDL Web Services Description Language

WSML Web Service Modeling Language

WSMO Web Service Modeling Ontology

XI Exchange Infrastructure

XML Extensible Markup Language

XQuery XML Query Language

XSPARQL XQuery for SPARQL



Executive Summary WP7 is one of the three SOA4All use cases and has the public sector as its target domain. It envisions an integrated service delivery platform that has been realized based on the technologies and tools developed in SOA4All. This platform allows non-technical end users in public administrations to handle typical administrative procedures (such as a permit approval process). More specifically, using the Web-based tools of the SOA4All Studio, public servants of various governmental organizations can search, model, annotate, modify, share, analyze, and execute administrative procedures in the form of lightweight business processes. These processes may be composed of SAP Enterprise Services, public Web services (hosted by 3rd party service providers), and human activities (to be executed by end users). Thus, the main result of WP7 is an integrated demonstrator that addresses the specific needs of public administrations. In this document, we summarize the main achievements of this work package during the entire duration of SOA4All by (1) describing the different use case storyboards that have successfully been implemented on the basis of SOA4All technologies, and (2) reporting on the advantages of the SOA4All approach together with open issues and recommendations for future work by conducting different evaluation and validation studies from an economical, technical, and usability point of view. As a summary, we can conclude that the SOA4All technologies in general and a service delivery platform for the public sector specifically have a strong potential for achieving high impact during and after the project.



1. Introduction WP7 is one of the three SOA4All use cases and has the public sector as its target domain. Public administrations nowadays have to deal with hundreds of different procedures (e.g., for handling a parking permit request) that are typically implemented in one or more legacy systems or are even executed manually. At the same time, the increasing number of regulatory changes and new regulations, including an increasing number of international, bilateral agreements, asks public administrations to constantly adapt their procedures in a flexible and cost-efficient way. For instance, the EU Services Directive requires administrations to implement a one stop e-Government approach where constituents can file requests for public services via a single point of contact. This single point of contact then coordinates all necessary activities, which is contrary to the current setup where the constituents themselves have the main responsibility and need to manage on their own. As a consequence, public administrations now need to adapt their service offerings flexibly to the needs of their constituents.

SOA4All investigates different key technologies (Semantic Web services, context adaptation, Web 2.0 principles) that can help to address such challenges on the basis of an advanced service-oriented architecture. WP7 envisions an open and flexible service delivery platform where administrative procedures are handled over a central Internet portal as single point of contact. Administrative procedures are composed of Semantic Web services. These services can be combined in different ways so that new procedures can be created or existing ones can be adapted easily. A key element for creating the content for this service delivery platform in an efficient and scalable way is the enablement of end (or business) users that resemble the large majority of employees in public administrations (and other organizations). Such business users have a detailed understanding of the procedures in their field of expertise but lack the specific programming skills required to actively consume and compose Web services. The SOA4All approach therefore is to provide end users with simple Web-based tools on top of Semantic Web technology so that they can search, model, annotate, modify, share, analyze, and execute administrative procedures on the basis of Web services. This Web 2.0 approach is also a main differentiator where SOA4All advances the current state of the art.

The main goal of WP7 is to realize this service delivery platform as an integrated demonstrator. This demonstrator is built from the components developed by the technical WPs of the project: the communication and data storage infrastructure is provided by WP1, formalisms and tools for the semantic handling of services by WP3, services discovery and a service registry by WP5, lightweight process modeling and execution by WP6, and different Web-based end user tools by WP2. Besides the technical integration and validation, the main contribution of WP7 from a technological point of view is to investigate how so-called SAP Enterprise Services can be integrated into the open, dynamic, lightweight, and end user-driven service platform that is envisioned by SOA4All. Such Enterprise Services offer complex business functionality like the management of resources or relationships with customers. But at the same time, they have large syntactic (i.e., WSDL-based) service interface descriptions and need to be composed in specific sequences, which makes them difficult to understand for non-expert service consumers. Thus, by investigating how to make Enterprise Services consumable for non-experts, WP7 increases the number of services that can be handled by SOA4All.

Previously, D7.2 motivated the public sector as a target domain for the SOA4All project and described how the research of SOA4All can support SAP in realizing a novel service delivery platform for the public sector by means of an exemplary use case scenario. The resulting technical and business requirements for this service delivery platform were described in D7.1. Following, D7.3 illustrated how the SOA4All components are leveraged to build the service delivery platform and which components are developed additionally by WP7 in order to realize a complete demonstrator for the given scenario. D7.4 and D7.5 described the status of the WP7 demonstrator at M18 and M33, respectively. In D7.6, we covered the different semantic artifacts required to integrate the syntactic SAP Enterprise Services into the SOA4All technology stack.

1.1 Purpose and Structure of the Document

In this deliverable, we summarize the results of all evaluation and validation activities that were conducted in WP7 over the course of SOA4All, and we report on the advantages and benefits of the SOA4All approach as well as on open issues that require further investigations and improvements.



The overall evaluation methodology for SOA4All has been outlined previously in the deliverable D2.5.1. The overall goal of the different evaluation activities is to measure the SOA4All project with respect to its objectives of scalability, usability, discovery, dynamic composition, robustness, and context awareness (see Section 2.1 of D2.5.1). Thus, the evaluation strategy includes three different types of evaluation:

1. Fit-for-purpose Evaluation (see Section 2.2 of D2.5.1): The techniques and models developed in every WP are tested against the requirements of the use cases (i.e. requirements validation)

2. Technical Evaluation (see Section 2.3 of D2.5.1): The performance of the software modules and algorithms/techniques developed by each WP are tested upon integration in the main SOA4All Deliverables.

3. Usability Evaluation (see Section 2.4 of D2.5.1): Several techniques are employed to evaluate the usability of the interfaces produced by the project, using representative samples of the target end users

In Chapter 2, we first summarize the different user storyboards that have been developed by WP7 over the course of the project and that are the basis for all evaluation activities. In the following Chapter 3, we give an overview on which SOA4All components have been used for which storyboard implementation. In Chapter 4, we present the results of the economic validity as part of the fit for purpose evaluation of our general use case scenario. The technical part of the fit for purpose evaluation covering the use case requirements as well as the technical evaluation were conducted through implementing the use case storyboards and is described in Chapter 5. Lastly, Chapter 6 reports on the different usability evaluations that have been executed within WP7.

Please note that this document integrates and summarizes relevant results over the entire duration of the SOA4All project in order to give a comprehensive overview. Sections that repeat results, which have been reported in other deliverables before, reference these deliverables.

1.2 Future Work

D7.7 is the final deliverable of WP7.



2. Use Case Storyboards The use case storyboards 1, 2 and 3 have been described previously in deliverables D7.5, an updated summary of these use cases is also included in appendix E. This chapter presents the latest storyboards 4, 5 and 6.

2.1 Storyboard 4: Wizard-based Service Composition

This storyboard was introduced for evaluating the concept of wizard-based service composition (or BPM) and has not been described before in another deliverable. The focus of our work on this topic in SOA4All was to develop the wizard concept and evaluate it. For this purpose, we first developed three different wizards for typical process modeling tasks and integrated them into the SOA4All Composer. The evaluation results are described in Section 6.1. Based on these results and as a further proof of concept, we then implemented a more elaborated wizard for guiding the user through the necessary steps of KPI-based BPM of the Storyboard 3 (see appendix E).

2.1.1 Overview

There are many software tools in the market for business process modeling (BPM). However, a general issue with these tools is the lack of modeling methodologies that cater for non-IT savvy business users (BUs):

• Workshop results in the SOA4All project and studies in research literature [Bandara2007, Filipowska2009] discovered that current BPM tools do not guide users in any sensible way. Supported by an analysis of Gartner [Rosser2008] that shows that BPM tools are too complex and error-prone for average users. These BPM tools are as such a significant barrier for non IT-savvy BUs both in usability and complexity. They prevent BUs to participate in the procedure of BPM.

• Creating business process models demand a certain level of domain expertise in both business and IT [Ko2004].

To allow BUs to perform modeling tasks efficiently and error-free, it is necessary to offer them an appropriate, more user-friendly BPM approach. For WP7, we have therefore researched on End User Development (EUD) for BPM, which aims at improving the ability of BUs to effectively modify process models according to their individual needs. In particular, we have developed a step-by-step instruction design pattern (“wizard approach”). A wizard is a special interaction paradigm where the user of a software is guided actively in small steps to achieve a specific task rather than leaving the user the initiative. Wizards are particularly successful when the corresponding task is performed rarely, is complex, requires several dependent steps, or cannot be undone easily. Subsequently, we anticipated that a wizard can help to ease the development of lightweight service compositions using the SOA4All Composer especially for inexperienced business users. Starting from simple process editing operations (create, modify, or delete a single process activity or connection), a wizard can help the user also in complex process modifications, e.g., when a process model has to be updated due to new business regulations.

The developed wizard-concept consists of two steps where the first step enables IT experts to describe the BPM activity, annotate this description, and publish it as a wizard-model representation to a central repository (wizard production). In a second step, BUs are able to browse through the repository and load the descriptions as needed. Relying on this description, a wizard can be automatically generated. This generated wizard guides the BU in an intuitive way through the BPM activity and performs the necessary changes on the process model (wizard consumption).

1. Wizard production: For the description of lightweight BPM activities, a formalism is applied in the first step that has the advantage of using an XML-based language. Thus, it is intuitive for IT experts who are experienced in lightweight BPM. Moreover, the formalism is suitable to document knowledge on how to perform certain BPM tasks. Additionally, a tool has been developed that allows IT experts to easily create, annotate and publish wizard-models. It stores the models in a central repository.

2. Wizard consumption: Wizard-models can be loaded by BUs and serve as input for the SOA4All Composer to automatically generate a wizard. The wizard is a fully functional tool to guide BUs through the lightweight BPM activities and to give them support, based on the



knowledge of IT experts. Whenever possible, the tool performs the necessary changes on the process model to serve the convenience of the BUs. As a result, even BUs that are not necessarily IT professionals, are able to create and modify lightweight process models according to their needs.

Our concept has been judged suitable for the considered users (see Section 6.1). More details can be found in [Lombardi2010]. In the next section, we describe the storyboard and the process model that has been used for the evaluation purpose (please note that the process model itself is purely fictional).

2.1.2 Detailed Storyboard

# Actor Screenshot Description Components

1 Egon

Egon’s task is to model the following process: take a walk and eat a chewing gum at the same time. If the walking person manages this task without stumbling, he receives a multitasking award. First Egon uses the SOA4All composer without any assistance during modeling.

WP2

• Storage Services

• Composer

WP7

• Wizard plug-in

• services

2a

Egon

In comparison, Egon uses the wizard-based BPM method. For this, he opens the wizard tab and starts the appropriate wizard. A welcome screen gives general instructions on how to use the wizard.

WP2


• Composer

WP7

• Wizard plug-in

• services

2b

Egon

In the first step, the wizard guides Egon through the required actions to model the first parallel activity.

WP2


• Composer

WP7

• Wizard plug-in

• services




2c

Egon

In the second step, the wizard guides Egon through the required actions to model the second parallel activity.

WP2


• Composer

WP7

• Wizard plug-in

• services

2d

Egon

In the third step, the wizard guides Egon through the required actions to define the conditions.

WP2


• Composer

WP7

• Wizard plug-in

• services

2e

Egon

In the next step, the wizard helps Egon to create the last activity required for this process.

WP2


• Composer

WP7

• Wizard plug-in

• services

2f

Egon

In the final step, the wizard congratulates Egon on finalizing the task.

WP2


• Composer

WP7

• Wizard plug-in

• services

Table 1: Detailed User Manual for Storyboard 4



2.2 Storyboard 5: City of X – Service Provisioning and Composition

For the usability evaluation at M34 at SAP Labs Bulgaria (see Section 5.3), we have designed and implemented a new storyboard based on the activities of Storyboards 1 (see Appendix E) and 2 (see Appendix E) using the process model and services of Storyboard 3 (see Section E). Due to the limited timeframe for the actual evaluation as well as technical issues reported in Section 5.3, this storyboard is shorter than the previous ones.

2.2.1 Overview

The storyboard covers the three phases in a typical service and process modeling lifecycle of the City of X: (1) first the evaluation user needs to semantically annotate a pre-defined service email_accessibility_check with a WSDL interface using the SOWER tool and then check the service availability in the iServe service registry, (2) then the user needs to check this/her annotations for correctness by searching the annotated service via the SPICES tool, and (3) the user needs to model a complete process, which handles the sending of a survey to the citizens of the City of X either by mail or via post (see Figure 1), with the SOA4All composer. Besides a general introduction to SOA4All and task instructions for these three phases (see Section 6.2.2), the evaluation users also were provided with a textual description of all relevant services (see Table 2).

Service Description

CitizenEmailAvailabilityScanService This service checks how many citizens from a certain street could be reached by email. The input of the service is the country, the zip code, the city and the street. It returns, aside from a Boolean value which indicates whether citizens with emails have been found, a percentageOfAvailableEmails which signifies the percentage of citizens reachable by mail and with absolute values the number of email addresses (numberOfAvailableEmails ) and the number of all residents from this street (numberOfResidents ).

GetCitizenEmailsService

This service returns all email addresses for the citizens from a certain street. The inputs will be the same as CitizenEmailAvailabilityScanService’s and the return would be, aside from a Boolean value indicating whether the service had worked properly, a List of emails.

GetOnlyPostalReachableCitizensService

This service finds all citizens of a street which are not reachable via email and returns a List of them. The inputs and outputs are the same as CitizenEmailAvailabilityScanService’s and the return would be a List of citizens with names and addresses and as additional information the number of citizens found.

OnlineSurveyDeliveryService The service takes as input a List of email addresses and a String which points to the questionnaire and sends the messages to the email addresses of the citizens.

PostalDeliveryServiceService This service sends the questionnaire to the citizens. The input of this service is a List of citizens’ data



containing names and addresses.

PrintSurveyQuestionnaireService

This service needs a link to the questionnaire and the number of citizens to decide how many surveys should be printed.

Table 2: Services for Storyboard 5



0 PREPARATION

services

• deploy business services on DSB

service annotations

• store ontology via storage service • annotate WSDL in SOWER to create SAWSDLs and upload them to

iServe

process

• create initial process with Functional classification to show advanced features like assisted modeling

WP2


• SOWER • iServe • Composer

WP3

• Grounding Editor

WP6

• LPML • DTC

WP7

• services • ontologies

Figure 1: Process Model for Storyboard 5




1a

User

The user should create semantic annotations for one of the services. The first part was to annotate the type part by drag the elements from the left side, which represents the ontology to the matching elements on the right side, which showed the service description and drop them there. After this the prepared lowering and lifting schemas should be added, this could be done by right clicking on the message and select the wanted operation.

WP2


• SOWER

WP7


1b

User

After the lowering schema has been added to the query message and the lifting to the response message the last thing to annotate was the operation with the fitting functional classification to make it discoverable with the chosen classification. With the finished annotations the SWASDL was ready and could be uploaded to iServe.

WP2

• SOWER • iServe

WP7


2a

User

The next task was to find the services, which were annotated and deployed in the previous step. This was done with SPICES, which offers a field on the left side to tip in a keyword and with pressing the “Search” button all services containing the keyword are listed. By clicking on one of them a new window opens for the chosen service

WP2

• SPICES • iServe

WP7





2b

User

To have a closer look on the services the window can be maximized and in the “Details” tap additional information are provided, as the services was just created there were now rating or comments yet. Then the button with the label: ”Add services to favorites” was used to add this services to the favorites list and so this services could be selected in the Process Editor from that list.

WP2



WP7


3a

User

For task 3 the Process Editor was required as a process should be modeled with this tool of the SOA4All Studio. A written description of the process and what should be achieved was provided to the participants.

This process should cover several different model possibilities: conditions, split, parallel flow, activities and human task.

WP2


• Composer

WP6

• LPML • DTC

WP7

• services • ontologies • human

tasks

3b

User

After doing the whole modeling process manual the user should make some experience with advanced features of the Process Editor like assisted modeling to complete a modeling task. For that a process is loaded into the Composer, which is a blueprint process with defines how in general the process should look like. This process has no concrete bindings, but it features a functional semantic

WP2

• Composer

WP6

• LPML • DTC




classification, which gave a hint to the tools which kind of service is required. As that is enough at first Resolve Activity are chosen to resolve the Activity with the name “Email survey”.

3c

User

The tools behind will make use of existing knowledge and enhance this process by changing the structure and adding fitting bindings. When this process is send to the Design Time Composer (DTC), so that this automated support could be done it takes a short while.

WP2

• Composer

WP6

• LPML • DTC

WP7


3d

User

In this case the one Activity has been split into two and both have more specific in and output parameters. The same can be done for the whole process which is the next step by selecting “Resolve process” this time. Again is takes a short while until the resolved process is presented in the Composer.

WP2

• Composer

WP6

• LPML • DTC

WP7


3e

User

The resolved process is now more complex and provides for each necessary action an activity and this activity also contain bindings, which represent possibly candidate services with which this process could be executed.

WP2

• Composer

WP6

• LPML • DTC

WP7





3f

User

For the execution of the user needed to choose which of the candidate services should be used during the execution of the process. In this complex step the user is also supported by a tool called Optimizer, all that is needed to be done is select the “Optimize” option and it will be done.

WP2


• iServe • Composer

WP6

• Optimizer

WP7


3g

User

The Optimizer will choose the best suitable services out of all, which had been proposed by the DTC based on predefined criteria. Now this process is ready to be sent to deployment and execution.

WP2

• Composer


2.3 Storyboard 6: University – Service Composition

For the usability evaluation at M37, we have designed and implemented a new storyboard based on a new process model and set of services.

2.3.1 Overview of Student Registration Scenario

This scenario describes the registration process that oversees students go through while getting admission in UK universities. In this respect, students start by searching for suitable universities and after choosing a university they register for a course in the University of their choice. The next step is to pay the university fee. At this stage students open a bank account and get the funds transferred into that account. The bank account is then used to set up payment of university fee. In the other case, students get a letter from their sponsor (funding body) and submit that letter to the university. The next step is to find accommodation and registering with NHS.



Figure 2: Student registration process model

There are dependencies between the different registration tasks, often unknown to the overseas students, which can cause problems leading to frustration. A composite reusable service can alleviate this by guiding the process, which is shown in Figure 1.

The storyboard covers the three phases in a typical service and process modeling lifecycle of the overseas student registration (1) first the evaluation user needs to semantically annotate a pre-defined service UniversityService with a WSDL interface using the SOWER tool and then check the service availability in the iServe service registry, (2) then the user needs to check this/her annotations for correctness by searching the annotated service via the SPICES tool, and (3) the user needs to model a complete process, which handles the various steps involved in university registration with the SOA4All composer. Besides a general introduction to SOA4All and task instructions for these three phases the evaluation users also were provided with a textual description of all relevant services (see Table 2).

Service Description

SearchUniversity Given the name of a city the service returns a list of universities in that city. The input is a string indicating city name, the output is a list of strings or ‘0’ if no university is found.

CourseRegistration

This service registers a course in the university. The inputs are the course details, the output is registration confirmation.

FindAccomodation

The input to this service is the accommodation name (that can be found on the accommodation list on university website), the output is booking confirmation for the accommodation (keep in mind oversees students are guaranteed a place in university accommodation of their choice).

BankAccount The service can be used to open a bank account. This service takes the account request as input and the output is the account details.

TransferFunds The service can be used to transfer funds between two accounts. The input is the transfer request, the output is transfer conformation.



SponsorLetter

Get letter from sponsor. The input is the letter request. The output is a sponsor letter.

PayUniversityFee The input for this service is a request, which can be either bank account details from where money can be debited or a letter from sponsor. The output is payment confirmation corresponding to the university registration.

RegisterNHS The service can be used to register with NHS. The input is registration request and the output is registration confirmation.

Table 4: Services for Storyboard 6


Overall Tasks for Users

• Service annotations • Annotate WSDL in SOWER to create SAWSDLs and upload them to iServe process • create student registration process model and bind services with relevant • Establish dataflow between various activities/services in the process model • Optimize the process model based on KPIs


1a

User

The user should create semantic annotations for one of the services in Table 4. This involves annotating the type part by drag the elements from the left side, which represents the ontology to the matching elements on the right side and drop them there.

WP2


• SOWER

WP7


1b

User

The other thing to annotate is the service operations with the fitting functional classification to make it discoverable with the chosen classification. With the finished annotations the SWASDL is now ready and can be uploaded to iServe.

WP2


WP7





2a

User In the Consumption platform the task for user is to find the services, which were annotated and deployed in the previous step. This is done with SPICES, which offers a search field to enter keyword and with pressing the “Search” button all services containing the keyword are listed. By clicking on one of the services a new window opens for the chosen service

WP2


WP7


2b

User From the previous step, the “Details” tap on the new window shows additional information for each service. Newly created services are assigned a default rating of 3 (medium). The user can”Add services to favorites” by clicking on the button shown at the bottom of the window. Once in the favorite the services can be selected within Process Editor.

WP2



WP7


3a

User

For task 3 the Process Editor is required to model the student registration process in the SOA4All Studio. A written description of the process and what should be achieved was provided to the participants.

The student registration process covers several different model possibilities: conditions, split, parallel flow and activities.

WP2


• Composer

WP6

• LPML • DTC

WP7

• services • ontologies • human

tasks

3 User In this case one Activity WP2




b

(Pay University Fee) can be performed in two ways and both have more specific in and output parameters. The same can be done for the whole process.

• Composer

WP6

• LPML • DTC

WP7


3d

User

Once the design phase of process model is finished user can bind services with relevant activities and specify conditions for multi-output connections in the model.

A binding represents possibly candidate service with which a activity could be executed.

WP2

• Composer

WP6

• LPML • DTC

WP7


3e

User

For execution of the process user need to establish dataflow connections between the activities. This is done by clicking on an activity and clicking ‘Data Flow’ button from the left menu. In the data flow window user can drag the output of a service from the left column to the middle column and then click on the input(s) of the other service in the right column to establish a link between output and input of two services.

WP2


• iServe • Composer

WP6

• Optimizer

WP7





3f

User

The last thing to do on the process model is to run KPI-based optimization using the KPI wizard which can be found under the ‘Wizards’ tab in the left menu of the Editor.

The Optimizer will choose the best suitable services out of all based on user selected KPIs. After optimization the process is ready for deployment and execution.

WP2

• Composer • KPI Wizard




3. Demonstrator Architecture The architecture of the WP7 demonstrator is previously discussed in the deliverables D7.3, D7.4, and D7.5. This chapter summarizes the use of SOA4All components (as shown in Figure 3) per WP in the storyboards 4, 5 and 6. For further details of the SOA4All components the reader is referred to Appendix E or Section 4 in D7.5.

3.1 Components Overview

3.1.1 Storyboard 4

The following table summarizes the use of SOA4All components per WP in Storyboard 4:

WP Component SB 3 use

2 Composer SOA4All process compositions design-time tool

2 Storage Service Maintains processes, ontologies, service favorites

6 LPML API SOA4All process compositions formal language

7 Services (WSDL) services in the storyboard

7 Composer Wizard Guided process modeling

Table 6: SOA4All Components used in Storyboard 4

Figure 3: SOA4All Overall Architecture with WP7 modifications

Consumption Platform

Human Task

Semantic Adaptation

& Integration Layer for SAP ES



3.1.2 Storyboard 5

The following table summarizes the use of SOA4All components per WP in Storyboard 5:




2 iServe Maintains the semantic annotations for SAP ES

2 SOWER (WSMO-Lite Editor) SA-WSDL annotation tool for SAP ES

2 SPICES SOA4All consumption platform for process compositions

2 Authentication Integrated authentication b/n components with OpenID

3 Data Grounding Editor LiLo mapping schemas tool for SAP ES


6 DTC Automated process composition assistance

6 Optimizer Automated process composition assistance


7 Service Ontologies Taxonomy and Ontologies for Service Interface


3.1.3 Storyboard 6

The following table summarizes the use of SOA4All components per WP in Storyboard 6.




2 iServe Maintains the semantic annotations for SAP ES

2 SOWER (WSMO-Lite Editor) SA-WSDL annotation tool

2 SPICES SOA4All consumption platform for process compositions

2 Authentication Integrated authentication b/n components with OpenID


6 Optimizer Automated process composition assistance


7 Service Ontologies Taxonomy and Ontologies for Service Interface




4. Economic Evaluation: Stakeholder Consultations In order to evaluate the economic potential of a SOA4All/based service delivery platform for the public sector as described in Chapter 2 (also see deliverables D7.2, D7.3, 27.4, D7.5, and D7.6), we have conducted a number of quantitative and qualitative studies with business stakeholders. In the following sections, we describe the results of these studies.

4.1 M11: Public Sector Expert Interview

In January 2009, we conducted an expert interview with a senior representative from the SAP sales department in Switzerland who was responsible for the public sector. As such, the representative had intensive contact to customers from the Swiss public sector including Canton and city administrations as well as universities. Before the actual interview, we first introduced the general ideas of a service delivery platform for the public sector based on SOA4All technologies. For better illustration, we also showed a selection of user interface mockups following Storyboard 1 (see Section 0) that were designed in order to define the general look and feel of the SOA4All studio as well as the basic functionality of the WP7 demonstrator.

Our first set of questions aimed to investigate the current state-of-the-art of a typical SAP customer from the public sector. The expert described that even small public administrations usually have their own local IT department that in principle could handle small software development projects such as implementing new requirements for a business process using SAP technology. However, in practice these IT departments often need to invest many resources for routine maintenance work and first level support so that almost all development projects need to be contracted to external solution providers. Typically such projects are then executed in a very formal way and take at least several weeks to complete.

On the business user side, there exists to basic user groups: first, domain experts in the front office, who directly intact with the citizens and who usually handle up to 15 different administrative procedures. Second, in the back office process experts support the infrastructure of an administration and handle vertical functions such as finance and HR. These process experts usually have a detailed understanding of business process modeling and SAP applications. However, any required change to a business process or the development of a new business process currently needs to be handled by external solution providers as described above. As a result, there is currently a strict role model distinction between process experts, process modelers, and process executors, which often results in a large communication and coordination overhead.

As a summary, we can deduct that there is a high potential for user-friendly business process development tools, such as the ones envisioned by SOA4All, which would allow public administrations to quickly implement smaller changes in existing or new administrative procedures in-house, resulting in significantly reduced costs and helping administrations to adapt faster to changing requirements or even offering new services.

In the second phase of the interview, we discussed specific requirements for a public sector service delivery platform. These requirements they used as a basis for the initial design of the WP7 demonstrator described in D7.3. The expert pointed out that a service delivery platform for the public sector needs to support

• both human tasks and automated activities, • different user roles with different rights and functionalities, • intuitive management of process models, • process lifecycle management such as the approval of process changes by a process owner, • easy to use, graphical process modeling with support for inexperienced users, • simple consumption of services from external service providers, and • process monitoring and history keeping.

4.2 M18: SAP WorldTour

In August 2009, the SAP event “SAP World Tour” took place at SAP headquarters in Regensdorf, Switzerland (also see the deliverable D10.3.1). The purpose of the event is to showcase the latest



SAP solutions to customers and technology partners. We used this opportunity to present the conceptual ideas, the scenarios, and the benefits of the SOA4All technologies with respect to the WP7 use case. We also demonstrated a first version of the SOA4All studio tools. Following the presentation, we asked the participants to fill out a questionnaire to investigate the participants’ opinions regarding their requirements towards a flexible service delivery platform for the public sector as envisioned in WP7 (see D7.2 and D7.3) in their particular organization. With respect to the business model, the participants largely represented potential customers of such a service delivery platform and therefore provided excellent feedback on the economic potential of SOA4All.

The results of this evaluation have been reported before in the appendix of deliverable D7.4. Here, we summarize the most important findings.

Regarding their background, the 17 participants of the questionnaire were mainly mature professionals, aged between 32 and 60 with an average age of 41 (those who provided their age). They were involved mainly in managerial IT roles, with all but two having IT degrees. The majority worked in the Public Sector. Whilst the majority respondents were experts with internet and standard applications such as MS Office and SAP, and many have used Web Services (WS), they were largely inexperienced in WS development.

One set of questions was to investigate the current IT landscape of the participants’ companies in order to evaluate the market potential of the SOA4All platform. The participants mostly represented large companies, all having their own IT departments that in the vast majority were also responsible for software development. Typically, all companies also had a high number of IT solutions (more than 50), that are integrated to some extent but not completely. Participants also reported that business process change often, resulting in change requests to the IT solutions. As a consequence, we can say that in all organizations reported there is a high potential for a SOA4All-based service delivery platform, which aims to simplify the integration of different IT solutions based on services and to shorten the task completion time the client for implementing change requests.

To estimate the likelihood of uptake of the SOA4All ideas, we asked participants about their attitudes towards benefits, risks and proposed support approaches. The results obtained are shown in Table 9. It compares the responses from the SAP World Tour with the results from other SOA4All workshops.

Group of Responses WP7/SAP Other

Number of Responses 17 64

SC is Useful (all responses are from 1-disagree to 5-agree, 3 is neutral) 4.1 4.4

SC is Easy 2.9 2.9

SC is Efficient 3.6 4.0

SC is Unfeasible 2.6 2.2

SC is Error-prone 3.7 3.0

SC may be abused 3.9 3.5

Examples of successful SCU can stimulate one to try it. 3.7 4.5

Recognising and rewarding SCU effort will make people more willing to try it. 2.4 4.0

Attending a training course could help people to start SCU. 4.0 4.0

SCU quality standards and testing will decrease risks. 3.7 4.0

Table 9: Attitudes to Benefits, Risks and Supporting Approaches

Their answers demonstrate a positive balance between benefits and risks, which is considered a



necessary pre-requisite for uptake of such user-driven service composition. Evidently, service composition by business users was considered to be potentially very useful, bringing about a more efficient way of conducting on-line activities. At the same time, it was not regarded as easy to achieve, yet its achievement was considered feasible. It was considered somehow error-prone with a non-zero risk to using it for breaking organisational rules and policies.

Providing examples and training were identified as the approaches with most potential for supporting user-driven service development.

When asked about the potential business model of a SOA4All service delivery platform, the results were inconclusive: in principle, the participants did not favor a licensing model over a pay-per-use model and vice versa.

4.3 M22: Public Sector Expert Group Discussion

On December 2nd, 2009, a Focus Group Discussion on “The Internet of Services and eGovernment Innovation” took place at Manchester Business School, UK. In this discussion, WP7 collected valuable feedback about the general idea of a Service Delivery Platform for Public Sector Users.

4.3.1 Setup

The participation in the workshop was solicited using the following abstract: “The Internet of Services is the next wave of the internet taking us beyond the internet of data and information. This workshop offers an opportunity to learn about recent developments in service-based technologies and to explore how these can be used for innovation in local government.”

The workshop was designed for those actively engaged in local government or with recent experience, and was organized by the Centre for Service Research, Manchester Business School.

Participants : 10 participants attended, with varying degrees of involvement in eGovernment initiative, including people from Manchester City Council, Manchester Knowledge, consultants to eGovernment projects and academics conducting research in the area of eGovernment.

Method: After an introductory presentation about the general concepts of the WP7 Service Delivery Platform for Public Sector Users, a lively group discussion was triggered, with some topics explicitly brought into the discussion by the SOA4All representatives.

4.3.2 Discussion Notes

The following notes summarise the comments made by the participants.

(a) Benefits and Risks from eGovernment 2.0:

Differences between users: some users (the majority) will be happy to just use information and eGovernment services, others would want to compose and modify such services. A parallel can be drawn with Wikipedia – most people use the knowledge, some prefer to contribute. Passive and Active use will depend on the value provided by the service to the consumers – if the information I need is there, I am more likely to use it. The service Web seems quite useful to ordinary users.

Value can be generated in participative fashion – example of citizens collecting information about minor crimes on urban maps, and then one can profile their daily routine and be warned if necessary. Also traffic congestion and bike accidents. However, these ideas may get negative reception in eGovermment area, mainly based on concerns about privacy, malicious use, falsifying information, etc. Example could be an application where criminals can put patterns of police control. Also criminals may provide services which look like very useful benign services but are actually malicious, designed to trick people to provide information they should not. Another risk is citizens providing wrong negative information about their neighbors, the feedback would be heavily polluted with misinformation. The Government in Switzerland is now providing some legislation to ward off against misuse in the services area, this example is not widely followed in the EU.

At local government level there are two concerns with the idea of publishing services:

• Security: enable police (tools but also resources) to monitor and stop misuse; • Education: educate people about the risks.

UK is different from EU in that the data about cities does not belong to local government, only 10% does. For example health service data is collected by NHS, etc. There is a wealth of data collected by



the local government for Westminster. Campaigns like “free the data” try to change the way data is treated by local government(s) but still the public data is not sufficiently open yet, for example the noise pollution data you can see but not use in mashups, etc.

In terms of encouraging people to contribute, you can consider returning some of the accrued financial benefits to the contributors, share benefits with individual contributors. Attitudes also change with the time and education, now we are all obsessed with saving electricity, water, carbon impact, etc.

For success with eGovernment 2.0, one should really consider the attitudes of different groups of people with different models of thinking, and how to impact some of these. One needs to share the benefits, for example Amazon keeps your purchasing history for you so you can look it back. At eGovernment level, this could be facilitating interfaces to multiple service providers so you only need to enter your data once.

(b) Organisational Issues and Business Models

Top-down vs bottom-up: eGovernment 2.0 can be seen as making consultants redundant by empowering users to map processes and link activities in the required fashion. But experience shows that the users in the organization are quite myopic, and cannot see the big picture of all processes. Indeed, eGov2.0 proposes the opposite of the consultancy-style top-down model, a bottom-up model where fragments of knowledge hosted by individuals and individual units are collated up in a big picture.

Social Media: Related to this is the customer feedback through social media, for example Manchester City Council (MCC) use twitter feeds to collect feedback from the Manchester Art Gallery regarding special exhibitions, etc. MCC is training its workers to do this kind of analysis rather than outsource it to other organizations, in the drive to improve efficiency of operations.

Organisational Structures: Different councils organize their IT and BMP functions in different ways. MCC have Department of Transformation, which brings together IT specialists, BPM specialists and Customer Service Specialists. This Dept does all mapping of Business Processes and IT support, interfacing with individual units within MCC via specialists Contacts in both DT and each such unit. Other councils have gone different routes – in BCC they have kept IT central and separate from BPM issues, which are outsourced.

EU Services Directive: Its implementation is impacted by realities of funding and legacy applications. In the UK the organic approach is prevalent, where a single point of contact is a central switchboard, and the linking of different processes and activities is done there. There is a wealth of legacy systems and they cannot be replaced because of financial reasons, rather they are integrated in a gradual fashion.

Empowerment and Social Networks: Junior IT staff is actually too enthusiastic about this, the middle layer is more conservative since they realize the risks involved in this. Firewalls actually stop any access to social networking sites, so people have to go to internal “internet cafes” to do this kind of work.

Control of EUD: Any workflow created bottom-up would have to subject to some control and testing procedure. Service efficiency considerations will prevail over the fear that staff will misuse the power provided by EUD. Otherwise the public sector may find itself bankrupt.

Cooperative EUD: Currently relying on direct communication, facilitated by the physical proximity of staff.

Ideas for further case studies: Students arriving in Manchester is a good case study since it involves interaction with multiple organizations, and we can map the study quite easily, and we have the test subjects also. This has been reflected in the creation of a new storyboard 6.

4.3.3 Conclusion

In general, discussions within the focus group confirmed the solid basis for the WP7 vision of eGovernment and the Internet of Services, whilst highlighting the highly situated nature of take-up measures and initiatives if they are to be successful in the existing variety of local government organizational structures and cultures.

One positive way of generating further feedback data to facilitate uptake measures would be to use a set of example case studies. It would be best to try them with smaller authorities since the big ones



have very slow decision-making dynamics and take far too much time to initiate positive responses and operations. In doing this we should explore existing links between agents of change, such as the University of Manchester, and local authorities, for example Rochdale.

Finally, many of the ideas behind WP7 would also have fruitful application to state-controlled health service organizations, such as the National Health Service within the UK. Indeed, they share similar concerns with local government authorities whilst having somewhat different funding stream and mechanisms may facilitate faster decision-making and willingness to change rapidly.

4.4 M26: Consultations with general users of e-Gove rnment Services

Acting on the conclusions of the Public Sector Experts group discussion in M22 (Section 4.3), the University of Manchester developed a new storyboard (StoryBoard 6), based on a student registration for a study in a University in a new city. This was designed to facilitate access to student users, which reflect the characteristics of the general users of e-Government services.

4.4.1 Setup

The Centre for Service Research at the University of Manchester organized a focus group workshop where 15 students who are not programmers were invited to discuss their opinions of SOA4All vision and prototype tools. The workshop lasted approximately 3 hours and was organized using the following procedure:

1) introduction to SOA4All and WP7 use case study, 2) discussion of risks and benefits of consuming and composing e-government services, 3) notational studies to understand natural ways of composing applications, and 4) discussion of 5 alternative composition approaches.

4.4.2 Analysis of results

The participants in the workshop were asked about their opinion regarding what public sector services they use. The most populate ones turned out to be health, education, transport, mail, and utility bills.

When the participants were asked to articulate their main concerns with current e-government services, they focused on the technical difficulties and performance constraints: long waiting times, finding the right service, language barriers, complexity, too many passwords.

A wider variety of services and richer features for each service were desired.

When presented with the SOA4All vision of e-Government services, the participants found the ability to adapt existing public sector services to their personal needs, and building new ones from scratch, to be a valuable idea and were positive overall on it. The main reason for their positive attitude was highlighted to be the potential saving in time and efforts, and the ability to extend applications defined by experts for personal needs.

We also discussed the risks perceived by the participants. The following risks were identified:

• The risks related to data privacy, especially the exposure of sensitive information (e.g bank details) to third parties such as service brokers;

• The risks related to unreliable applications and lack of transparency in service operations and results;

• Knowledge of process modeling is needed for application creation, this may prevent a number of potential users to access service composition facilities;

• Services are not necessarily compatible with each other on the basis of the information their consume and produce, and on their behavioral and functional models.

Finally, the participants were asked to express their preference to five alternative representations expressing service composition. They preferred the simplest representation which hides details of inter-service dependencies and uses a template-based selection of suitable services by the end user , also known as “assisted composition”. This was followed by control flow-style composition, due to its ease of use and ease of understanding.

4.5 M30: Public Sector Expert

On August 10th, 2010, a Public Sector expert interview focusing on the German public report “Single Point of Contact: Where are we now and how can it go on” took place at the chamber of trade in a city



of Baden-Wuerttemberg. This second expert interview built on the results obtained by the first interview (see Section 4.1). In this expert interview, WP7 gained qualitative feedback about the storyboards of the use case and the general idea of introducing an end-user adapted Service Delivery Platform for the Public Sector like presented in [Meyer2010].

The setup section presents the applied setup of the interview, while the content section summarizes the gained expert results, which have been reported in detail in D10.3.2.

4.5.1 Setup

For the interview, an employee of a Public Administration of Germany was selected, directly interacting between new information technology solutions and public servants. This expert stakeholder is an active participant of the EU Services Directive implementation in the German state of Baden-Wuerttemberg and has detailed insights about the status and appeared opportunities of the German implementation. For the appointment, we selected the expert interview methodology as a specific form of applying a semi-structured interview [Meuser2002], focusing on the interviewee as a whole person in his role as an expert for the EU Services Directive field. To address all issues within 2 hours, we pre-formulated parts of the questions in a guideline consisting of different areas. For reacting in a flexible way during the conversation, additionally individual questions were added to the prepared interview part. The interview was moderated by one WP7 member.

4.5.2 Content

Parts of the interviewees’ duties consist of representing the chamber of trade within the project for the electronic implementation of the EU Services Directive in Baden-Wuerttemberg. This includes the content wise involvement in the project such as the provisioning of written descriptions for sub-processes that need to be performed (e.g. registering a new business in the City of X, see Section 0). The project of Baden-Wuerttemberg currently accounts 70 participants and includes public administrations as well as external partner companies. The aim of the project is to create an administration portal for Baden-Wuerttemberg that is accessible on the Web. Core content of the website is the representation of municipal public administration services over a single point of entry. The Web portal requires to be constantly updated and legally secure on all available administrative services for every citizen in the state Baden-Wuerttemberg. A known IT company in cooperation with the German Ministry of Interior and partners of the public administration of the different state cities implemented the electronic procedure for the EU Services Directive. Partly, the EU Services Directive was implemented in a uniformed way. A single process description was achieved at federal level for all cities in Baden-Wuerttemberg. In addition, for realizing this uniformed solution even new laws were passed. Unfortunately, each of the 16 German federal states implemented the actual electronic procedures individually. Furthermore, all EU Member States realized the implementation of the EU Services Directive in a different way, which lead to a multitude of implementations instead of having a central and shared platform. The interviewee reported that the new electronic procedures of registering a new business became too complicated for public servants. One reason is the graphical user interface of the application which was not realized in an end user-friendly manner. End users in this context refer to both the employees of the public administrations as well as to the citizens that use the developed service platform. According to the interviewee, the graphical user interface should be more intuitive, following the example of new Internet applications. Known online marketplaces are already very end-user friendly, and address a large set of users. In terms of usability, these web portals should serve as a model for the public sector. The chamber of trade of this city already offered a digital Web solution for registering a new business before the new implementation at federal level. Via this application, about 200 of the yearly 1000 enterprise procedures were processed. Since the new implementation was realized, not a single procedure was processed digitally. The interviewee reported that he is still convinced of the electronic procedures but that it is not realistic to automate this process completely. In public administration, there are discretionary decisions, which need to be taken manually by the desk officers, when aspects of a process are not or not clearly defined. This aspect of simplifying and unifying procedures at European level was not in the spotlight of the project implementation in Baden-Wuerttemberg, since no exchange took place at European level. Nevertheless, by the enactment of new laws a gradual alignment of this process took place at federal state level. The new application was realized on time to the end of December 2009. This covers the single point of contact and the process for the registration of new businesses. Currently, the full implementation of the electronic process is still in work and was expected to be terminated towards the end of 2010. At this time, the new Web portal does not promise a relief as it brings new labor-intensive aspects. These aspects include the training of employees and citizen as well as the too rare



use of the electronic procedures. In addition, a fundamental change of the organizational structure emerged in the last recent years. The number of processes that implement the offered public services doubled, while the number of employees has decreased proportionally within the last 10 years.

4.5.3 Analysis

Transferring the research vision of the SOA4All WP7 Service Delivery Platform to this concrete scenario, some facilitation in relation to the EU Services Directive could be achieved. The EU Services Directive is presently not fully implemented within all EU countries. The implementation was realized by using different technologies and methodologies. This yield to different solutions within all European countries, sometimes even at federal state level like in Germany. WP7 envisions one central Service Delivery Platform that supports besides further aspects the modeling and execution of processes between different public authorities at EU level, but in this case also on federal state level. In this way, administrative staff of various authorities could collaborate by exchanging, reusing, or adapting already implemented processes from other European administrations. The SOA4All Service Delivery Platform ensures a unified technology for the implementation within the various provinces, but also within the different EU countries. If new processes are introduced in the different countries, an overall and comprehensive implementation through a central platform approach can be realized. This minimizes the effort for the introduction and adaption of procedures. The implementation in Baden-Wuerttemberg is not realized in an end-user friendly way. This causes big problems for public servants as well as citizens and leads to the total rejection of the electronic procedures from end-user side. The Service Delivery Platform envisioned by WP7 explicitly addresses public servants. The goal is to provide administrative staff with the help of an end-user-friendly platform and an easily understandable process modeling notation, the ability to model, adapt and execute processes themselves and on demand.

4.6 M31: JSWEB

On September 12th 2010, a workshop was conducted during the JSWEB 2010 event in Valencia as part of the scientific-technical days on Web Services and SOA. The workshop focused on SOA4All and aimed at promoting, disseminating and transferring SOA4All development achievements to potential users and to get valuable feedback from scientific experts having a related technical background as well as public sector experts in order to improve the development stage of current prototypes and to discuss new exploitation ideas. A detailed report can be found in D10.3.2.

4.6.1 Setup

The workshop was split into four main sections. First, an introduction to the project contents was provided. Second, a conceptual overview about the SOA4All Studio components was given and WP7 demonstrated a live process optimization scenario from the public sector. Third, an open discussion round followed with several technical und public sector experts. Finally, all workshop participants filled in a prepared questionnaire.

4.6.2 Content

WP7 was mainly involved in the live demonstration of the SOA4All Studio. Therefore, a demonstration based on the KPI storyboard was developed in order to give some insights into the idea how a public administration could use the presented technologies and how they interact with each other (see Section 0). The demonstration was focused on the optimal selection of services of a business process. Acting in the role of an employee of the public administration, we showed how to optimize the costs of the created business process via selecting the service set with the minimum overall costs. After manually optimizing the service selection, the automated optimize functionality of the process was demonstrated as support for the employee of the public administration. Concluding the demonstration, an open discussion round was started to ask the workshop participants for their opinion (for discussion details see D10.3.2). To get further detailed feedback, a questionnaire was delivered after the discussion (see Appendix of D10.3.2).

4.6.3 Analysis

The analysis of the questionnaire revealed that most of the stakeholders agree that existing IT solutions do not satisfactorily assist to deal with problems of the Public Sector. Furthermore, the majority of the workshop participants agreed to the statement that SOA4All generally is a relevant technology for industries in a business context and also that it is an applicable technology for the Public Sector. The participants agreed further that SOA4All could help to reduce IT costs in Public



Administrations and that collaborative work between different public institutions could be supported. Additionally most of the workshop participants believe that SOA4All provides a technical basis to improve the communication between customers (citizen / companies) and Public Authorities.

The obtained questionnaire results with respect to WP7 corresponded with the previous expectations and confirmed the usability of the first prototype of the developed demonstration of the KPI based storyboard. The obtained results from the questions and discussions with the workshop participants fed finally into the preparation of an updated version of the corresponding storyboard (see Sections 0 and 2.1) and supported our following use case evaluation activities (see Sections 4.7 and 6.3).

4.7 M33: SAP Research Day

On June 14th, 2010, and on November 1st, 2010, a so-called SAP Research Day was held at SAP headquarters in Switzerland, Regensdorf. The first event focused on publishing and demonstrating research results, prototypes and further WP7 achievements to the academic network of SAP Research in order to get valuable feedback and ideas from university professors, researchers of other industry companies and universities as well as to position SAP for future projects. The second event focused on the promotion of concrete research results for getting direct feedback from SAP consultants working close to SAP customers in various industries including the public sector.

4.7.1 Setup

For both events the WP7 participants prepared theoretical and live demonstrations as well as posters for familiarizing the stakeholders with the developed functionalities of the SOA4All Service Delivery Platform. At the second event, the stakeholders were asked to fill out a survey based on a previous presentation of the SOA4All Service Delivery Platform (see Appendix B: M33: SAP Research Day). The paper-based survey is based on the general SOA4All questionnaire developed in WP10 (see D10.3.2), which was modified in order to capture the WP7 specific background of stakeholders within the SAP organization.

4.7.2 Analysis

This section presents the results of the survey. In general, 16 stakeholders provided feedback to the WP7 focused functionalities of the SOA4All Service Delivery Platform. Figure shows their background.

Figure 4: Background of survey stakeholders

Since the stakeholders come from different SAP departments and each of the departments covers several technical and business aspects, the following chart provides information about the detailed development expertise to SOA4All-related fields of the attendees.

25%

25%19%

19%

6%6%

(Pre-)Sales

(Business) Consulting

Service/ Support

Training

Customer

Development



Figure 5: Software development experience

Overall, the survey participants had an intermediate knowledge of the technical background of SOA4All related technologies and architecture approaches covering specifically the areas of process modeling tools, SOA, Web Service development as well as application development. This result is a good starting point to get valuable feedback in a business related context.

Regarding the SOA4All concept, different statements can be obtained from Figure 6. As an average, the participants slightly agree that it is not necessary to have technical skills for using SOA4All and that SOA4All is likely to provide tools that are easy to use und to make the services composition easy to achieve. Neither an agreement nor a disagreement can be obtained by the statements whether the participants would like to use the SOA4All platform for sharing services and processes and whether SOA4All could reduce application development costs.

Figure 6: SOA4All concept

Applying the SOA4All concept to the Public Sector in particular result in a positive feedback, gained as presented in Figure 7. SAP stakeholders agree that SOA4All supports the work between Public Authorities from a collaborative perspective and may lead in this context to a reduction of IT costs. Furthermore, the Public Sector is seen as a more relevant industry for applying SOA4All technology than the application in other industries in a business context.

4

5

2

1

3

4

5

6

5

4

6

4

3

2

2

5

1

1

1

0

Application development using

programming languages (e.g. Java)

Web service development

Analysis and design of service

architectures

Experience with Process Modeling

tools

very low low intermediate high expert

1

1

3

1

5

5

8

4

7

10

7

6

6

6

3

2

2

2

SOA4All is likely to make the services

composition easy to achieve

SOA4All is likely to provide tools that are

easy to use

SOA4All could reduce application

development cost

I would like to use SOA4All platform for

sharing services and processes

It is not necessary to have technical skills

to use the SOA4All tool

disagree slightly disagree intermediate agree strongly agree



Figure 7: SOA4All in the Public Sector

For exploiting the results of the SOA4All project within the SAP organization, different statements were confirmed and rejected by the SAP internal stakeholder group. A very positive reaction was obtained when asking if SAP solutions based on SOA4All could enable new markets especially for small and medium enterprises. In this context SOA4All was seen in order to complement and enhance existing SAP solutions. The participants saw a small potential for current SAP solutions in order to apply the SOA4All approach for allowing fast and business user-friendly BPM. A more critical response was gained asking if SAP customers are willing to invest in new BPM and services technologies like SOA4All (see Figure 8).

Figure 8: Exploiting SOA4All at SAP

5

2

7

5

8

8

8

6

2

5

4

SOA4All is applicable in the Public Sector

and other industries in a business context

The Public Sector is a relevant industry for

SOA4All

SOA4All may lead to an IT costs reduction

in Public Administrations

SOA4All supports collaborative work

between different public institution


2 4

1

1

4

1

6

3

3

9

5

8

2

4

2

5

Current SAP solutions allow fast and

business user-friendly BPM

SOA4All could complement or enhance

existing SAP solutions

SAP customers are willing to invest in new

BPM and services technologies such as

SOA4All

SAP solutions based on SOA4All could

open new markets (e.g., SMEs)




5. Technical Evaluation The purpose of the technical validation is to assess the compliance of SOA4All with the initially defined use case requirements and usage. This validation is also known as “fit-for-purpose” validation in the overall SOA4All evaluation approach (see D2.5.1).The validation discussed here is performed in the context of WP7 storyboards. The storyboards are detailed in different deliverables and are described in Chapter 2. The requirements that define the assessment frame of the technical validation are detailed in deliverable D7.3 (Section 2). It specifies the functional and non-functional requirements from the storyboard use-cases and domain (public sector) perspective. The following table summarizes these requirements. They will be referenced further in this document where the following subsections outline the storyboards as they were provided in each project milestone and the corresponding technical validation report.

Requirement Description

Shared Service Registry and Process Repository

The service delivery platform should provide a central and shared registry that lists all services and processes as well as a shared repository for processes, including those that are added by the target users so that existing processes can be executed, re-used, and modified by users other than the original author.

Support SAP Enterprise Services

The service delivery platform should allow accessing SAP Enterprise Services that provide rich business functionality for administrative procedures such as the management of the documents associated with a certain case. Enterprise Services are typically hosted by the internal IT department of a public administration, but could also be hosted by centralized shared services centers for several administrations or independent 3rd providers. Currently, SAP Enterprise Services have complex, WSDL-based service interfaces that can be handled by service experts only.

The integration of SAP Enterprise Service into the SOA4All platform therefore requires (1) a semantic adaptation and integration layer that is compatible with the semantic formalisms used in SOA4All and (2) a front end that makes the handling of such services feasible for the target user group.

Support Public Web Services

The service delivery platform should support the integration of Web services that are offered by external public or private service providers so that a public administration can outsource certain functionality.

Support Human Tasks

Business processes in public administrations often involve activities that are executed by humans such as checking requests for completeness and correctness. The envisioned service delivery platform therefore needed to support the modeling and execution of human tasks .

Manage and Search Services and Processes

In a typical public administration, the service delivery platform will contain several hundred services and processes so that intelligent search and organization techniques are required to minimize the access times for users.

Execute Services and Processes

The service delivery platform should allow the seamless execution (i.e., consumption) of services and processes.

Capture User Knowledge on

Services and Processes

The main efficiency gain from social Web 2.0 applications results from capturing and structuring knowledge from all users in an organized way and from making this knowledge easily accessible to all users. In addition to managing the services and processes in a shared registry, the service delivery platform should allow to manage different types of meta-data, which supports the handling of services and processes for the SOA4All tools or the SOA4All users:



• Semantic annotations for services and processes to support automatic service discovery and service composition

• Categories to organize services and processes • Tags to organize services and processes • Ratings as user feedback on the quality of services and processes • Textual Comments for user discussions, guidelines, or qualitative feedback

Compose New Processes

Users of the service delivery platform should be able to compose new processes that resemble administrative procedures by connecting activities with control flow elements (i.e., business logic).

Activities may be SAP Enterprise Services, public Web services, or human tasks.

Modify Existing Processes

The modification of existing processes is needed so that they can be reused as parts for new ones or in order to adapt them to new requirements.

Analyze and Access Status

Information on Services and Processes

The service delivery platform should provide status information on selected process instances so that users can identify the active activity of a running instance or can retrieve information about the success or failure of a completed or aborted instance. Moreover, users can evaluate the performance of a service before placing it in a process model, which is of particular interest for public Web services from 3rd party providers.

Support User Roles

Like in other professional organizations, employees in public administrations have different positions with specific rights and responsibilities. The service delivery platform should reflect this internal organization of administrations by supporting a user and role model that allows assigning specific access rights to data or functionalities to certain users or roles.

Usability The main target user group of the service delivery platform will be business users, i.e., civil servants in public administrations with sufficient business knowledge to handle their assigned working tasks, but with limited IT knowledge (usage of office, Internet, and business applications but no programming or SOA skills). The user experience of the tools available via the service delivery platform should meet the specific needs of this target group by using a minimalist screen design with direct manipulation of all elements, by using human-computer interaction paradigms known from other applications like an Internet browser, by hiding the underlying technical complexity, by pre-filtering and sorting large amounts of information , by error tolerance, by offering explicit support and guidance for specific tasks, etc.

Low Administrative Overhead

A key business goal of the service delivery platform is to achieve a smaller total cost of ownership (TCO) when compared to traditional software solutions. Besides enabling business users with end user development (EUD) facilities and thus shifting efforts from IT specialists to end users, this requirement can be accomplished by low installation and maintenance costs for all software components of the platform.

Table 10: Overview of Use Case Requirements

5.1 M18: Storyboard 1

The details for the demonstrator state can be explored in detail in the D7.4 Deliverable. In summary and emphasizing on the technical support of SOA4All to the demonstrator in the different phases, the validation reveals the following:

• Annotation of services is not supported well yet as the tool for WSDL annotations (SOWER) is not stable yet and, more severe, the SOA4All Service Model (defined by WP3) is still under



development and has been redesigned several times. Overall, service annotation is a time-consuming task because complex services require complex ontologies and because there is no tool to validate results automatically for correctness.

• Service discovery is not employed yet for this Storyboard but services are preselected / provided via the favorites list which is integrated with the SOA4All Composer.

• The modeling phase is supported by a fully functional editor for manual modeling. Assisted modeling works for selected examples. The complete loading-modeling-save cycle for modifications of existing processes is supported. Automatic deployment to the execution engine phase is not functional yet in this first prototype.

• Process invocation is implemented in the demonstrator as a tailored client (web application City of X Portal) that triggers the process execution, which showcases how third party software can extend the out-of-the-box SOA4All toolset.

• The Process Monitoring tool is not fully functional yet and does not provide access to live process monitoring data.

In conclusion, the different SOA4All components partially provide the required functionality, but a seamless integration has not been achieved yet. Implementation of a fully flexible, dynamic end-to-end scenario suffers in result. The main obstacles at achieving lower TCO are the time-consuming service limitations, the low level of integration between the tools, and the low level of automation and guidance in some cases.


Rating

1..5

Comments


3 The requirement is technically supported by the WP2 Storage Services (SS) for shared registry support. Processes are shared in a dedicated repository maintained by the SS (process-models).

Exploring and retrieval of processes (for modeling) is implemented in WP2 Composer .


3 At this stage, ontologies and service annotations for SAP ES have been modeled manually because the SOA4All modeling tools were not stable yet and also not yet integrated. Due to the complex service interfaces of the SAP ES the manual modeling is very labor-intensive and error prone. Modeling errors can be discovered and fixed only in a trial and error fashion. The first version of the WP7 SAP ES integration layer has been implemented, together with a number of simplified SAP ES test services.

The published semantic descriptions for SAP ES cannot be successfully consumed and executed by WP2 SPICES at this stage.


3 Even though public Web services tend to have less complex service interfaces when compared to SAP ES, the manual modeling of semantic descriptions is nevertheless labor-intensive and error prone.

Support Human Tasks

2 For supporting human tasks, the basic architecture has been defined with a dedicated WP7 human tasks serve r for human task execution and management, specialized WP7 human tasks activities for process modeling, and a WP7 human task client in the SOA4All studio for human task consumption. However, the implementation at just started at M18 so that's human tasks not yet supported in process models natively.




Rating

1..5

Comments


2 Service discovery is not employed yet for this Storyboard but services are preselected / provided via the favorites list which is integrated with the SOA4All Composer. Currently the only facility to browse design-time processes is the WP2 Composer ’s open dialog. It offers capability to list processes form different repositories with a simple set of details accompanying the file.


2 WP2 SPICES provides for service invocation. However as of this milestone it was not fit to execute SAP ES.

Processes can be executed in the Execution Engine after they have been transformed and deployed semi-manually. Since it is a platform service it requires tailored facilities to invoke a deployed process. This costs some undesirable integration effort that prolongs the scenario development. A more seamless method of invoking a process at least for test purposes should be elaborated instead in order to fulfill the requirement fully.

Capture User Knowledge on Services and Processes

2 These requirements will be supported by the annotation tools WP2 SOWER/ SWEET and the WP3 Grounding editor , as well as by WP5 Service Discovery (Semantic search) and WP2 SPICES (web 2.0). The previous comments on the level of the tools support should be considered and influence this assessment as well.


3 Process composition is supported by WP2 Composer to a large extent. Limited dataflow and control flow modeling supported (WSDL only and error prone). Bindings through Favorites. WP6 DTC (resolve process only) and WP6 Optimizer services not yet integrated into the SOA4All process editor. Deploy not integrated. No WP7–specific modeling capabilities yet, e.g., wizard-based modeling.


4 WP2 Composer supports opening an existing process for modification and saving back in a process repository.

Process monitoring 1 Process monitoring with the provided by the WP2 Analysis platform in the SOA4All studio, while a client prototype exists, live data from the execution engine cannot be accessed yet.

Support User Roles 3 User authentication is implemented by the WP2 Profile studio module. The concept for roles however is not covered. There’s a notion of roles in the WP7 Human task client which should personalize the human tasks for the logged in user. The notion of a human task role is also available in the WP2 Composer when modeling human tasks. By this milestone Human task role is available only conceptually.


3 Overall SOA4All have the potential to lower the entrance barrier and to diminish the integration costs. On the other hand it shifts the focus to a different approach at service platforms but there’s still a good way to go in terms of guidance and assistance that the tools provide to end users.

Table 11: M18 Fit for Purpose Evaluation Results



5.2 M33: Storyboards 2 – 3

The detailed description of Storyboard 1, 2 and 3 in their state as of the M33 milestone can be examined in D7.5 sections 2.1, 2.2 and 2.3, respectively. In summary and emphasizing on the technical support of SOA4All to the demonstrator in the different phases, the validation reveals the following:

• Annotation of services is technically supported by soa4all but is a time-consuming task and results are not validated for correctness. The iServe registry is integrated with the annotation tools via the publish functionality and an end-to-end service description provisioning cycle spanning over starting from scratch (raw WSDL) to published description on iServe is available.

• Service discovery can find services from iServe and supports complex search criteria. Some search criteria must be typed manually, which is prone to error, and there is no way to add additional functional classification taxonomies. It provides also means to filter results by selecting functional classification categories for the services returned in the result set. It needs to be more stable and dynamic concerning changes in the knowledge base (iServe) and its support for users though. The UI comprises a Favorites tool to integrate with the modeling phase.

• In this final prototype, the modeling phase is fully connected with the previous phases as the Composer now is fully integrated with the iServe registry. It is also connected with next phases as the deployment is also fully integrated. The set of assisted modeling functions also grew considerably lowering significantly the efforts by automation. A KPI based modeling was introduced to model business requirements to automated optimization of process.

• The execution phase is in the same state as in previous milestone described in D7.5. Same comments apply. Service invocation in the consumption platform is not functional for partonomized service descriptions on this stage.

• The monitoring did not change since last milestone. Same comments apply.

In conclusion, the tools are supporting the demonstrator technically to a large extent in this final prototype. The main obstacles at achieving lower TCO are the redundant, labor-intensive, and error prone tasks in all phases of the service lifecycle, as well as the lack of integration in some cases.


Rating

1..5

Comments


4 The requirement is technically supported by two components:

- WP2 iServe service repository for service registry support

- WP2 Storage Services (SS) for shared registry support. Processes are shared in a dedicated repository maintained by the SS (process-models).

Exploring and retrieval of processes (for modeling) is implemented in WP2 Composer .


3 The WP7 demonstrator features a fully functional WP7 SAP ES Integration Layer to simplify the interaction with the original SAP ES. It mediates the communication with SAP ES and provides simplified service interfaces, which are the actual subject to annotation.

The integration of SAP ES services in SOA4All is enabled by the provisioning tools for annotation of services, which make them consumable in the SOA4All platform




Rating

1..5

Comments

The provisioning requirement is technically supported by two modules integrated in the SOA4All studio:

• WP2 SOWER for WSDL annotations • WP3 Grounding Editor for Li/Lo schemas development

Finally, the WP2 iServe is used as integrated in SOA4All tools service registry to publish SAP ES descriptions.

With the help of these, SAP ES are provisioned to SOA4All for execution and service compositions, i.e. technically it is possible to complete an end-to-end annotation and publishing of annotations as Linked Open Data in the iServe registry from where they can be consumed as knowledge.

However it still takes a considerable amount of time and effort to accomplish the annotation process as it requires not complex but numerous and redundant operation with limited tools support. The tools, for instance, are completely missing automation capabilities which, translated to the SAP ES complexity of data types means a very manual labor-intensive task. The tools are also missing validations to enforce all constraints on the used ontologies input or the produced description output. Thus, the identification of an erroneous description that cannot be consumed by the tools comes only very late and finding this out is unnecessarily hard.

The published semantic descriptions for SAP ES cannot be successfully consumed and executed by WP2 SPICES at this stage but its support improved as it renders the input form for the service correctly now.


4 In addition to the technical support for SOAP services, the public web services might be REST enabled. The corresponding technical support for annotation of such services is provided by WP2 SWEET editor.

The comments for SAP ES support apply here as well, but the lack of modeling support is less severe because most public web services tend to have much simpler service interfaces.

Support Human Tasks

4 The WP2 Composer supports modeling a predefined set of human tasks types. The set of tasks changed to a more flexible form of templates which can be updated on demand. A dedicated platform WP7 Human Task Server is integrated with the Execution Engine and serves requests for human tasks invocation during process execution. The human tasks are consumed by users via the WP7 Human Task Client .

One limitation is that the human task server currently does not support dynamic role management, that is, changes on the list of roles supported requires server restarts and synchronization with Process Editor and HT Client.


4 WP5 Service Discovery provides UI for searching services via dynamically provided criteria. The user assistance has shortcomings that make it usable primarily for technical users. For example the query construction assistance is not dynamically updated and one has to learn the query language specifics and deal with long, not very user friendly ontology concepts URIs. Besides, it is not possible




Rating

1..5

Comments

to select any domain-specific functional classification taxonomy and/or a data type ontology that helps to set up the query criteria. It also feels like not very scalable as even for a moderate service repository like iServe it takes some visible amount of time to fetch and retrieve all services details even for a small result set. It is also not very stable as one can often stumble upon an application error. WP2 iServe Browser has a friendlier interface for crawling the service repository but does not follow the corporate design of the SOA4All studio. WP2 SPICES features a simplified, Google-like search facility which serves the purpose of finding a service by name but is also limited just to that, i.e. cannot fulfill entirely the requirement. Deployed processes are available for discovery after they are provisioned just like any other service in SOA4All. Currently the only facility to browse design-time processes is the WP2 Composer ’s open dialog. It offers capability to list processes form different repositories with a simple set of details accompanying the files, alternatively the processes on which the user has been working lately and processes that have been bookmarked as ‘favorite’. The latter unfortunately is not functional yet and the simple set of properties is not always sufficient to detect the required process which tends to prolong the browse time. The WP6 DTC offers a search-like functionality where the user can model a query process consisting of one or more semantically described activities and then invokes the DTC that retrieves fitting processes.

Invoke Services and Processes

3 WP2 SPICES provides for service invocation. In this milestone it renders correctly the input form for the services but it still cannot execute SAP ES.

Processes can be executed in the WP6 Execution Engine after they have been deployed. Since it is a platform service it requires tailored facilities to invoke a deployed process. This costs some undesirable integration effort that prolongs the scenario development. A more seamless method of invoking a process at least for test purposes should be elaborated instead in order to fulfill the requirement fully. To execute the demonstrator process the prototype delivers a tailored frontend – a web application WP7 City of X Portal.


4 The requirement is supported by the annotation tools WP2 SOWER/ SWEET and the WP3 Grounding editor , as well as by WP5 Service Discovery (Semantic search) and WP2 SPICES (web 2.0), which now integrates with the recommender system. The previous comments on the level of the tools support should be considered and influence this assessment as well.


5 In this final prototype, the modeling phase is fully connected with the previous phases as the WP2 Composer now is fully integrated with the WP2 iServe registry. Now all bindings, dataflow mapping parameters and control flow parameters are fetched form iServe




Rating

1..5

Comments

service descriptions. The integration with WP5 Service Discovery is via Favorites. There’s a newly added integration with WP2 SPICES via favorites for service operations found via the consumption platform. It is also connected with next phases as the deployment is also fully integrated. The set of assisted modeling functions also grew considerably (WP6 DTC – resolve process, resolve activity, bind activity, generate dataflow, check semantic compatibility) lowering significantly the efforts by automation. A WP7 KPI-based modeling was introduced to model business requirements to automated optimization of process.


5 Modification and sharing of existing processes is fully supported by WP2 Composer .



1 Process monitoring is provided by the WP2 Analysis platform in the SOA4All studio. However it was not possible to consistently retrieve such information by this milestone.

The integration with monitored performance during modeling should be provided by WP2 Composer but could not be implemented by this milestone.

Support User Roles 3 User authentication is implemented by the WP2 Profile studio module. The concept for roles however is not covered. But roles are supported in the WP7 Human task client which personalizes the human tasks for the logged in user. The notion of a human task role is also available in the WP2 Composer when modeling human tasks. WP7 Human task client and server synchronize the same role management configuration (handcrafted) so the roles of logged users in Human Task Client are used to filter assigned human tasks. However, those roles as completely unrelated with the roles managed by the WP2 profile.


4 Overall SOA4All lowers the entry barrier and diminishes the integration costs. On the other hand it shifts the focus to a different approach at service platforms but there’s still a good way to go in terms of guidance and assistance that the tools provide to end users.



In M34, the implementation of Storyboard 5 was conducted in order to prepare the usability evaluation at SAP Labs Sofia, Bulgaria (see Section 6.2.2). In summary, the technical fit for purpose validation reveals the following aspects:

• Annotation of services is technically supported by SOA4All but it is a time consuming and error-prone task. The iServe registry is integrated with the annotation tools via the publish functionality.

• All major concepts required for modeling processes are implemented. Optimizer service is integrated in Composer. DTC service is integrated with its main operation- resolve process in Composer. Loading-modeling-save cycle is supported. However, seamless deployment of process to the execution engine is not possible to you to open implementation issues in the LPML2BPEL transformation component.

• Integration of monitoring information in the modeling phase is elaborated on conceptual level.



• The main obstacles at achieving lower TCO are the low level of integration between the tools and the low level of automation and guidance in some cases (see user evaluation report in section 6.2.2).


Rating

1..5

Comments


5 see Table 12 above





Support Human Tasks



3 Compared to the state as described in Table 12 above no additional features are available. However, while preparing the new scenario the service taxonomy could not be updated easily because it was statically bound within the service discovery component. Therefore, service discovery could not be evaluated. For all other comments, see Table 12 above.












Support User Roles 3 see Table 12 above







The implementation of Storyboard 6 (student registration) was conducted in M37 in order to prepare the usability evaluation at Manchester Business School, UK. In summary, the technical fit for purpose validation reveals the following aspects:

• SOA4All supports annotation of services but it is a rather complex process that requires know-how of service components e.g. types, operations etc. The iServe registry is integrated with the annotation tools via the publish functionality.

• All major concepts required for modeling processes are implemented. KPI-based optimizer wizard is integrated in Composer.

Requirement Description Rating

1..5

Comments

Shared Service Registry and Process Repository 5 see Table 12 above

Support SAP Enterprise Services 4 see Table 12 above

Support Public Web Services 5 see Table 12 above

Support Human Tasks 4 see Table 12 above

Manage and Search Services and Processes 4 see Table 12 above

Execute Services and Processes 1 see Table 12 above



Compose New Processes 5 see Table 12 above

Modify Existing Processes 5 see Table 12 above




Support User Roles 2 see Table 12 above

Low Administrative Overhead 3 see Table 12 above




6. Usability Evaluation The usability evaluations were carried by employing task-based evaluations technique. The aim of the evaluation is to test the usability of the final products of SOA4All. Usually user testing involves measuring end users’ performance while carrying out typical tasks. The test moderators or evaluators analyse the number and type of problems users make while carrying out predefined tasks, and record the time spent to perform these tasks. While performing the tasks, user interaction is recorded via video and logging programs. At the end of the experiment, users’ opinion is elicited using questionnaires and interviews. The results of the usability testing will be analysed using a usability post analysis process (i.e. Model Mismatch Analysis (MMA)) to gain a further understanding of the identified problems.

In the following subsections, we describe the usability evaluations for the various storyboards in WP7.


In order to evaluate the „wizard consumption“ phase of the wizard concept introduced in Section 2.1, we carried out a preliminary task-based evaluation to test the usability of the wizard-based modeling method and measured the performance of users while carrying out typical tasks. We analyzed the number and type of errors users made and recorded the time spent to perform these tasks. At the end of the experiment, the users’ option was elicited using questionnaires. To gain further understanding of the identified problems, the results were analyzed using as usability post analysis process [Sutcliffe2000]. The following two hypotheses were tested:

H1. Faster lightweight BPM (less time)

We hypothecate that the average user will have a faster task completion time when modeling a process with wizard support when compared to unsupported BPM.

H2. More accurate lightweight BPM (less errors)

We hypothecate that the average user will make less errors when modeling a process with wizard support when compared to unsupported BPM.

In the following sections, we first describe the evaluation method and then report on the most interesting findings.

6.1.1 Method

12 volunteers aged between 24 and 52 years from SAP Research participated in the study. Three were female, nine were male, and ten had experience with visual modeling tools like MS Visio. Nine had experience with UML tools, and 10 had programming experience.

Each participant conducted two study sessions: the first session using the traditional drag-and-drop modeling method, and the second using the wizard-based method described in section 2.1. The independent variable in this test was the BPM method. The dependent variables were the time spent and the number of errors produced to perform each task, as well as the usability rating of the test persons.

All study sessions were conducted using a desktop computer running the latest version of MS Internet Explorer. Each session started with giving the participants an introduction on how to use the modeling tools and an explanation about the tasks they would have to perform. This briefing took approx. 5 minutes. After the briefing, the subjects were asked to model three process models on their own, each model with and without the support of a wizard (also see section 2.1.2). The three process models were:

1. Task 1: A simple linear BP Please model a process where you buy a cookie. You can use the following service: http://www.cookieworld.com/getCookies.wsdl.

2. Task 2: A BP with one loop and two conditional flows Please model the following process: you have a cookie and a glass of milk. Dip the cookie into the milk and take a bite. Repeat that until you have eaten the entire cookie. After you've eaten the cookie drink the rest of the milk. In this task, you do not need to bind actual services to



each activity.

3. Task 3: A BP with two parallel activities and two conditional flows Now you need to model a very difficult task: take a walk and eat a chewing gum at the same time. If you manage this task without stumbling you receive a multitasking award. Please use the following services: http://www.kaugummi.de/kauDenGummi.wsdl, http://www.running.com/ runForYourLife.wsdl, http://www.multitaskingaward.com/superRunner.wsdl (also see step 2 of the storyboard described in section 2.1.2).

The modeling part of each session took approximately 20 min overall. We specified the parameters of the activities that should be used for the task consisting of an activity name and a URI. The experimenter was available for help, but only when the subject explicitly requested it.

After the session, the subjects were asked to fill out a questionnaire with 10 different questions, see Table 15.

Question Measure Result

How supportive were the instructions given in each wizard step?

5-point Likert scale from 1 (not supportive) to 5 (very supportive)

4.2

How do you rate the arrangement of UI elements?

5-point Likert scale from 1 (not clear) to 5 (very clear)

4.3

Did the SOA4All Composer always react how you anticipated it?

5-point Likert scale from 1 (never) to 5 (always)

4.3

How easy was the wizard to use? 5-point Likert scale from 1 (very difficult) to 5 (very easy)

4.5

Please describe problems you had while using the wizard, if any.

Textual answer

Did the wizard cover all required functionality?

5-point Likert scale from 1 (no functionality provided) to 5 (all functionality provided)

4.4

Please describe missing functionality of the wizard, if any.

Textual answer

Did the wizard force you to make unnecessary actions?

5-point Likert scale from 1 (many unnecessary actions) to 5 (no unnecessary actions)

4.6

Please describe unnecessary actions, if any. Textual answer

Do you have general remarks for improving or changing the visit?

Textual answer

Table 15: Questionnaire for Wizard-based BPM

6.1.2 Results

The results of the user study showed that both hypothesis (H1) and (H2) could be validated in the test setting. As can be seen in Table 16, we can observe that the more complex the task was, the more prominent is the difference between the average task completion times for all users achieved with the wizard-based BPM versus the unassisted BPM.



BPM Method Task 1 Task 2 Task 3

Unassisted 1.4 4.0 3.5

Wizard-based 1.1 2.6 1.3

Table 16: Evaluation time results (time in minutes)

Further, we observed a decreased error-rate during wizard-based modeling versus the unassisted modeling: as can be seen in Table 17, the average number of modeling errors for all users is significantly lower for the visit-based BPM.

BPM method Task 1 Task 2 Task 3

Unassisted 0.25 1.1 0.9

Wizard-based 0.2 0.5 0.1

Table 17: Evaluation errors results (number of errors)

Considering that most participants were very experienced computer users and almost all of them had some previous experience in formal modeling or programming, these results are even more remarkable because even these users could improve their efficiency and effectiveness significantly using the wizard-based BPM approach. For users with lower computer skills it is therefore to be expected that the wizard-based BPM approach produces even better results.

The results of the questionnaire are given in Table 15. Over all, the usage and support through the wizard was assessed as mainly above-average. On a scale of 1-5 all features and benefits were between 4.2 (usefulness of the instructions) and 4.6 (no unnecessary actions needed).

6.1.3 Discussion

Some of the difficulties that arose during the user study were associated to textual and user interface details that can easily be improved in upcoming versions of the wizard design. For example, the wizard start button is not visible enough (2x) and textual descriptions were perceived as too long (4x) and not precise enough (2x). Another problem arose because the wizard interface within the SOA4All Composer primary interface does not support all drag-and-drop interactions. The user is informed about the actions he can carry out, though more work is needed to integrate the wizard-based interface with the main drag-and-drop interface. Despite these minor problems, the users report that by the end of their session, they were comfortable with the wizard.


In order to verify the general idea of supporting business users during process modeling by selecting services for a certain process activity based on business KPIs rather than technical service properties, we have conducted a qualitative and a quantitative user study based on Storyboard 3 (see Section 0 and D7.6). However, please note that at the time of the user study the wizard-based version of the KPI tool was not ready yet and the testing was performed using a first version of the KPI tool where users had to select KPIs, call the WP6 Optimizer, and analyze the results in a non-guided way.

Besides the investigation of general usability issues of the non-wizard KPI tool, the aim of the study was to test the following hypothesis:

1. improved process quality: a process modeled with the help of the proposed KPI tool is better aligned to the business goals (requirements) than a process modeled without support

2. lower task completion time: a process can be modeled faster with the help of the proposed



KPI tool than a process modeled without support

6.2.1 Method

Based on the advantages and disadvantages of the different evaluation methods a two phase approach was selected for the evaluation of the designed artifact. An assisted and a participative approach were combined and applied based on a use case scenario. A detailed procedure plan is shown in Figure 9. Evaluation participants were assigned to two different groups, a theoretical group and a practical one. For both groups the evaluation started with a short introduction of SOA4All and the developed KPI tool. Trainings about the functionality and usage of the designed artifact followed for all participants, whereas for the theoretical group these were based on the use case description and for the practical group general information was given. The practical group was next part of the

assisted evaluation. Based on the underlying use case the participants had to establish an optimal service selection according to the predefined key performance indicators with and without the use of the KPI tool. The first task for the user was to get familiar with the underlying process and with the KPI tool. He/she had to then select a combination of services to minimize the costs of the process (Task 1.1) and as a second exercises the users had to optimize the process in a way that a minimal number of service providers is established (Task 1.2). In the second part of the practical evaluation the same optimization exercises are solved with the help of the KPI tool (Task 2.1, Task 2.2). A detailed description of the practical tasks can be found in Appendix A: M28 User Study Workbook.

For the execution of the evaluation the following materials were used:

• A short video presenting the SOA4All platform and the KPI tool was created and used as training material.

• The SOA4All platform was deployed on a local computer, which was then used for the practical part of the evaluation.

• General observations about the usability of the KPI tool and the problem understanding were made and documented in an observation sheet. User thoughts expressed aloud and remarks were recorded and subsequently transferred into the observation sheet (see Appendix A: M28

Figure 9: Evaluation Method



User Study Workbook) • The Unipark site (http://www.unipark.info) was used for an online questionnaire that was filled in

at the end of both sessions (see Appendix A: M28 User Study Workbook).

In total, 57 participants provided feedback to the KPI tool. For the identification of different user categories, some general background information was enquired as depicted in Table 19.

6.2.2 Results and Discussion

Based on the online questionnaire three aspects of the user acceptance of the KPI tool were analyzed (see Table 18). The first aspect was the general usefulness of the application that should answer the question whether the KPI tool is able to support the end users of the SOA4All platform in their daily work as of enabling them to transform their strategically goals in the operative tasks. The next section contains questions about the practical usability of the application covering aspects like understandability and user friendliness. And last but not least the functionality of the optimizer is enquired.

The results of the online questionnaire give general hints of the overall impression of the KPI tool. End users identified the usefulness of the application and its supportive functions in terms of business process modeling. All participants considered the application as innovative and as a means to implement business goals within the organization. With regard to the usability the KPI tool is considered neutral. Some of the reasons for this rating is the design of the user interface and also missing functionality, which were identified through the answers of the open questions. The results of this part of the usability study served as input for the design of the wizard-based approach described in Storyboards 3 (see Section 0) and 4 (see Section 2.1).

The optimization functionality of the KPI tool was definitely realized and regarded as trustful. However, half of the participants still believe that they could achieve the same results also without the application. For the evaluation of this statement the results of the practical part can be taken into account. As can be seen in Table 24, the number of participants who achieved the optimal solution is for the tasks without the KPI tool support less than the one for the tasks executed with the help of the KPI tool, which were obviously solved optimally at 100%.

Table 19: Participants Background Table 18: Results from Questionnaire



The hypothesized statements described above can be partly confirmed. The acceleration of the process modeling can be definitely proved by the results of the assisted evaluation. As depicted in Error! Reference source not found. , the tasks which contained a services selection by hand needed n average longer than the ones in which the users were supported by the KPI tool. Based on the practical exercises processes designed with the help of the KPI tool are more aligned to the strategic goals of the organization than the ones designed manually by the users. Nevertheless, due to the user's uncertainty with respect to this thesis, the statement cannot be confirmed nor refused.

Figure 10: Participants Achieving the Optimal Solution

Figure 11: Participants Knowing What to Do

The observation of the practical exercise provided some further results which verify the usefulness of the application. As shown in Figure 16, even the participants, who were judges as not familiar with the topic and did not really know how to solve the optimization tasks, could complete the exercise faster with the help of the KPI tool(Task 1.1, Task 1.2) than without it (Task 2.1, Task 2.2).



Figure 12: Task Completion Times


6.3.1 Aim and Objectives of Study

In November 2010 (M34), we have conducted a large WP7 usability study at SAP Labs in Sofia, Bulgaria, based on Storyboard 5 (see Sections 2.2 and 3.1.2). Specific sub-goals of this evaluation comprised:

(1) evaluating an end-to-end WP7 Use Case; (2) gauging user acceptance of the SOA4All Studio; (3) gathering users’ opinions and attitudes towards the SOA4All Studio and its tools in the form of

positive comments, usability problems, and recommendations to remedy these problems.

This user study took the form of a contextual inquiry in which we collected data about how users interact with the annotation tool (WSMO-Lite Editor), the consumption platform, and the SOA4All Process Editor in natural situations. This evaluation approach allows the experimenter and participants to cooperate together to identify usability issues.

6.3.2 Participants’ Background

A background questionnaire, designed to capture users’ skills and experience in relation to programming and development experience including annotation, composition, and monitoring, and prior knowledge of various annotation and modeling tools, was distributed to and completed by a total of 12 participants. Of those, 8 participants had completed a Master degree.

User experience in various service development activities was rated on a 5-point Likert scale, where 1 corresponds to poor and 5 corresponds to excellent. 10 participants (out of 12) indicated that their development/programming experience ranges from average to excellent (see Figure 13). However, 11 participants indicated that their semantic annotation experience was below average (see Figure 14). In regard to process modeling experience, 8 participants rated their experience from average to excellent. Finally, 8 users rated their experience in monitoring, deploying and executing services equal to or greater than average.



Figure 13: Participants’ software development and design experience

Figure 14: Participants’ semantic annotation, process modelling, and service lifecycle management experience

We totaled users’ scores for all of the above skill dimensions and visualized the results in the subsequent graph in order to be able to view and compare users’ skills (see Figure 15). The length of each series represents the total rating received by the participant. The longer the series the more experience the participant has.

In respect to the annotation and composition tools, the majority of our participants (11 out of 12 participants) have used SAP NetWeaver, a process modeling platform. However, none of the participants was aware or had used the SOA4All Studio before.

9 participants mentioned using WSDL language, and 4 participants indicated using BPLM. Only one participant reported using Protégé. In summary, most of the WP7 participants are experienced process modelers but have no experience in annotating services.

6.3.3 Evaluation Procedure and Scenario

In this WP7 usability testing we aimed to gather users’ opinions and attitudes toward the SOA4All Studio and its tools and to identify usability problems alongside recommendations to encounter these problems. The usability testing consisted of three main parts and their constituent tasks: training phase, development phase, and rating phase (Appendix C: M34 User Study Workbook). A detailed description of each phase is presented below.

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10 11 12

SoftDevelopmentExperience SoftDesignExperience

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10 11 12

SemanticTechnologies BPM ServComposition Lifecycle(AnalysisExecution)



Figure 15: Participants’ skills and experience summed up

6.3.3.1 Training Phase

During this training phase, representative users attended a group presentation which explained and demonstrated (1) the notion of service composition and (2) the various features and aspects of each of the SOA4All tools. This included a tutorial demonstrating one service development example of moderate complexity. Users were then invited to comment and ask questions about what they have seen. In detail, the training phase encompassed the subsequent steps:

1- A group presentation and training providing detailed explanation on how to use the service development tools under test

2- A post-training session for participants to ask questions about the service development tools 3- Giving out hands-on documentation / notes about the tools after the tutorial

Following the training session, each participant performed a set of development tasks independently as explained in the development phase.

6.3.3.2 Development Phase

At the beginning of this session, users were allowed to explore and interact with the SOA4All Studio alongside its constituent tools for 5 minutes with the aim of familiarizing them with the available functionality and getting their initial impressions. Immediately after that, participants were given an end-to-end WP7 scenario description and instructed to develop a service-based application covering three aspects of the development lifecycle: annotation, consumption, and modeling (also see Section 2.2). In this usability testing, we opted the think-aloud protocol through which participants verbalized their thoughts during the development process to unravel their design strategies and discover the types of usability and conceptual problems they ran into. We recorded user interaction behavior with each tool and their vocal discourse using a screen capturing software (i.e., Snag it) for post-experiment analysis. The precise steps performed by the participants in the development phase are as follows:

• Pre-test Interview: participants discussed their existing experience and opinions about Software and Service Development using different software development environments

• Interaction with the SOA4All Studio and its tools: participants interacted with the service development environment “SOA4All Studio” and completed a set of diverse development tasks (annotation, consumption, and modelling) in order to fulfill the test scenario

• Post-test Questionnaire and Interview: participated rated different usability dimensions of the SOA4All Studio and its tools. Subsequently, participants discussed their experience and opinions about the SOA4All Studio in a post-test interview.

6.3.3.2.1 Scenario and Task Descriptions

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11 12

SoftDevelopmentExperience SoftDesignExperience

SemanticTechnologies BPM

ServComposition Lifecycle(AnalysisExecution)



Prior to conducting the usability testing, we ran a set of pilot studies to ensure that the development tasks are well described and easily understood by the participants; otherwise it would not be possible to perform the tasks correctly. We refined the description of the tasks so that they do not detail the fine-grained steps required to solve a particular development task. A balance should be struck between clarity and specificity. We also ensured that the development tasks are complex enough to enable full interaction with at least the main features of the Studio. The tasks included annotation, consumption and modelling tasks. The annotation tasks instructed the participants to annotate the data types and one operation of the check email accessibility service, and export the resulting annotations to iServe. In the consumption platform, participants had to search for two relevant services and add their operations to the list of favourites. The modelling tasks focused on modelling the process of surveying citizens, binding services to the right activities in the process model, adding semantic descriptions to the activities, and resolving and optimising a complete process model. Refer to the appendix for the list of tasks and their description.

The overall description of the test scenario is as follows: “Your goal is to conduct a survey among a selected number of citizens of a particular street in order to gather feedback about a planned playground in their neighbourhood. For this you need to develop a software application which allows you to find citizens’ email and postal addresses and send them the survey to fill in. To realise this application you will use the SOA4All Studio which allows you to annotate, discover, and model a process both manually and using the help of the system (i.e. assisted modelling)”.

6.3.3.3 Rating Phase

At the end of the usability testing, participants were invited to rate the differing features of the tools by filling in a questionnaire and report their feedback in a debriefing interview. In this final phase participants:

• rated their development experience, usability of and satisfaction towards each of the SOA4All tools (WSMO-Lite Editor, Consumption Platform, Process Editor). They completed a paper-based questionnaire containing question items about usability and preferences such as “ease of use … etc” to which they expressed their the degree of agreement on a 1 to 5 rating scale, where (1 =disagree and 5 =agree)

• reported their final feedback and opinions about the SOA4All Studio in a de-briefing interview, addressing the positive and negative features and ways of improving them

6.3.4 Analysis Method

Two Human Factors’ experts (one of which had conducted the user studies) watched the videos of user interaction with the Studio, listened to users’ vocal discourse and transcribed the conversations into an excel file. This step is important as it allowed the experts to familiarise themselves with the data. Each expert worked independently by going through, preparing and analysing the transcripts for each task category (i.e. annotation, consumption, modelling). In this analysis phase, the experts coded the opinion data and identified corresponding themes following thematic analysis; a qualitative analysis method for analysing data involving the identification of a number of themes which emerge from the opinion data. Once finished, the two experts sat together and discussed the emerged themes; they removed the reoccurring themes and refined/paraphrased some themes for better accuracy.

6.3.5 Results

In the results section, we primarily focus on:

1. the positive attitudes and comments made by the end users toward the SOA4All Studio, as well as the subjective rating of the tools;

2. the main issues that WP7 end users encountered and reported during their interaction with the SOA4All Studio;

3. the user suggestions to improve the usability of the SOA4All Studio and its tools.

WP7 end users walked through an end-to-end scenario which aimed at creating a process model that retrieves email and postal addresses of citizens before sending a survey to be completed. The scenario focused on three tools of the SOA4All Studio, namely WSMO-Lite Editor, Consumption Platform and Process Editor. In total 12 participants took part in this evaluation, and each user testing



lasted approximately two to three hours. It is worthwhile to note that prior to usability testing all participants attended a 2-hour workshop explaining the various features of the SOA4All Studio. Participants were given a presentation followed by a tutorial and encouraged to ask elaborating questions.

6.3.5.1 Users Positive Comments and Attitudes toward the SOA4All Studio

User comments were analysed and categorised according to emerging themes for each tool, i.e. WSMO-Lite Editor, Consumption Platform, and Process Editor. Positive user feedback are summarised in the tables below followed by average rating of usability dimensions for the tools.

Theme Positive attitude

Ontology concepts

• A participant referred to a few concepts as straight forward and easy to understand, e.g. Zip code

• Participants praised the navigational features (e.g. ± signs) for expanding and collapsing the concepts of the ontology as it facilitates browsing the whole tree of concepts more easily

Navigation • Participants were able to find and start the annotation editor quickly and without difficulty

• Participants have stated that following initial use of the Studio they were able to easily recognise and navigate their way through the menus.

Mapping • Similarity between the names of ontology concepts and some data types of the service elements was praised as it enabled participants to perform the mapping task

Annotations • Participants liked that annotations are visualised at the bottom of the relevant service elements

• Right mouse click for removing added annotations was regarded as intuitive

iServe • Exporting annotations to iServe was regarded as an easy task • Participants found iServe browser more easier to browse and check their

annotated services than assessing a list of annotated services

Visual presentation - aesthetics

• Participants appreciated the opaqueness of some elements, they commented: “this is very eye catching”

• Participants liked the large buttons presented on the initial screen. Some participants also commented favourably on the side menu and the perceived sense of familiarity that is conveyed from the Studio following initial use

Terminology • A participant praised and described some of the terminology used in the side panel menu as descriptive

System feedback

• Participants appreciated that the system showed the green status (success notification) to inform them of the possibility of mapping a selected concept to a particular service element and more importantly to warn them in the case of invalid mappings (i.e. to prevent errors)

Drag and drop

• Participants liked the ability to graphically perform certain tasks such as the mapping task; this task can be performed by simply dragging a concept and dropping it at a particular service element. Participants have stated that they found the drag & drop process of annotation both simple and easy to interact with.

Table 20: Positive user comments in regard to WSMO-Lite Editor

On average, users perceived service annotation as a rewarding task in the service development lifecycle and an easy to achieve task. However, the rating of the WSMO-Lite Editor was less than the average value (3) which could be attributed to the cosmetic problems. All ratings were performed on 1-5 Likert scale where 1= disagree and 5= agree.



Figure 16: Average rating of WSMO-Lite Editor


Navigation • Participants were able to find and start the consumption platform successfully and quickly since it is visible at the first level of the menu

Service quality

• Participants appreciated that services contain a section dedicated to describing the quality and rating of the services; however this section needs to be populated with descriptions and details

System feedback

• Participants liked that the system reacts to their input by showing them feedback messages at the bottom of the consumption platform

Terminology • Participants indicated that some terms are clear and easy to understand, e.g. Search

Drag and drop

• Participants liked the ability to graphically move widgets (i.e. windows) around and arrange them in the consumption platform

Service Discovery

• A participant was pleased with the manner in which the search process was conducted.

Table 21: Positive user comments in regard to Consumption Platform

Rating of different question items related to the Consumption Platform fluctuated around the neutral value (3) apart from the reliability question which was rated higher (mean = 3.6, standard deviation= 0.70).

Average rating of WSMO-Lite Editor

1

2

3

4

5

Serviceannotation is

easy to achieve

Serviceannotation is arewarding task

WSMO-LiteEditor is easy to

use

WSMO-LiteEditor prevents

errors

WSMO-LiteEditor provides

sufficientinstructions in

case ofproblems



Figure 17: Average rating of Consumption Platform


Navigation • Participants were able to find and start the process editor successfully and without difficulty

Level of abstraction

• Participants liked the pretty view – when specifying an outgoing flow- which showed URIs of the parameters as easy to understand and read names

Layout • Participants praised the ability to minimize the left menu in order to gain more space

Visual presentation

• Participants liked that the process editor contained white horizontal lines to help them align their notations and follow the logic of the flow

• A participant stated that the notations are simple and easy to interact with throughout the modelling phase.

System guidance and help

• Participants appreciated that the process editor provides a set of pre-defined operators (e.g. lessThan( , )) for expressing conditions (outgoing flows)

Terminology • A participant stated that they are familiar with and understand the terms used in the operators during the modelling phase.

Table 22: Positive user comments in regard to Process Editor

Themes Positive attitude

System Feedback

• Participants liked the provision of a status progress bar which kept them informed during the ‘resolve’ process model task.

Assisted Modelling

• Participants were impressed with the capability of the Studio to undertake assisted modelling and the manner in which a complex model can be derived from a simple model.

Efficiency • Participants commented that they liked the proficient manner in which the Studio enabled the development and utilisations of models.

Table 23: Positive user comments in regard to Assisted Modelling (Process Editor)

Average rating of different aspects of the Process Editor was higher than those of the WSMO-Lite Editor and Consumption Platform. Users agreed that process modelling is a rewarding task which could be attributed to their prior experience with process modelling using SAP NetWeaver. The assisted modelling feature and optimise were highly rated since they reduce efforts and save time.

1

2

3

4

5

Consumption platform responds quickly to search

queries

Consumption platform is easy to

use

Consumption platform is reliable

Using the Consumption

platform, I can easily retrieve the services I

need

Average rating of Consumption Platform



Figure 18: Average rating of Process Editor

Aspect / Feature

Positive attitude

Visual Presentation

• Participants stated that they liked the side panel menu as it provided access to their options throughout the entirety of their Studio experience.

Level of Expertise

• Participants felt that the Studio would be appropriate for those with only a competent level of expertise and familiarity in developing notations.

Modelling Familiarity

• Participants liked that the shapes used in the modelling process are similar to those found in prominent process modelling environments.

System Accessibility

• Participants liked that the accessibility of the Studio was through the Web browser rather than a traditional download and install platform.

Assisted Modelling

• Participants stated that they liked the utility of the Studio in allowing users to develop process models by clicking and drawing objects. Furthermore, they appraised the simplicity with which the Studio derives complex process models from simple process models.

Tool Relevance

• Participants commented on the applicability of the Studio to annotate and deploy simple services and how this concept is currently absent within the field of software engineering.

Level of Abstraction

• Participants liked the format in which the annotation levels are presented since it is not undertaken in raw XML form.

Table 24: Positive user comments in regard to the SOA4All Studio

Finally, the overall rating of the SOA4All Studio was rather encouraging as most ratings averaged about the neutral value, for example users expressed willingness to user the SOA4All Studio in the future (mean= 3.54, standard deviation= 0.93):

Average rating of Process Editor

1

2

3

4

5

ProcessModelling isrewarding

Process Editoris easy to use

PE can createapplicationsthat facilitate

my job activities

AsssitedModelling saves

time & efforts

The optimiser isuseful



Figure 19: Average usability rating of SOA4All Studio

6.3.5.2 Usability Problems

The majority of the negative features of the SOA4All Studio reported were mainly cosmetic and related to the look and feel of the Studio. These usability problems are easily fixable. We summarise the main issues users encountered when performing the development tasks in Table 25. All usability problems are rated by the two experts on a 3-point likert scale [low, medium, high], where ‘low’ signifies low severity and ‘high’ signifies high severity for users’ successful completion of the task.

Tool Usability problem Severity rating

WSMO-Lite Annotator

Participants spent a lot of time to find services, ontologies, and concepts, largely due to the random way in which services and ontologies are listed in the repositories and the concepts in the ontology

Medium

WSMO-Lite Annotator

Participants were not able to understand some ontology concepts and their corresponding nodes (i.e. properties) as their names were not self-descriptive (e.g. sbcountry and addrcountry) which made the selection of concepts more difficult

High

WSMO-Lite Annotator

were overwhelmed with the amount of information presented in the WSDL section which made it difficult for them to locate the elements /operations they need to annotate

High

All tools of the Studio

Participants struggled to read error and feedback messages due to the short display time or poor visibility/readability of some user interface features which employed white writing on grey background (i.e. have low color contrast).

Medium


Participants complained that no help section /wizard is present in the Studio to assist them in case of problems especially that some tasks require technical knowledge

High


Participants complained about the names and structure used within the left hand side menu pane. They sometimes found difficult to navigate.

Low

Average usability rating of SOA4All Studio

1

2

3

4

5

SOA4All Studio iseasy to use

SOA4All studio iseasy to navigate

I am willing to useSOA4All studio in

the future

SOA4All Studio is agood substitute to

traditional tools (e.g.eclipse)



Consumption platform

Participants found it difficult to select the appropriate services in the consumption platform since no information was present to describe the functionality and features of the services

High

Consumption platform

Participants were frustrated by the inefficient search engine of the consumption platform and that upon entering a new search criteria, a new list of services is created at the bottom of the page instead of updating the existing list of services

Medium

Table 25: Usability problems within the SOA4All Studio and its tools

6.3.5.3 User Suggestions to Improve the SOA4All Studio and its Tools

To overcome the aforementioned problems and improve the usability of the Studio in the future, design suggestions provided by WP7 end users are enumerated in the following tables per tool type.

Theme Suggestion

Service / ontology sorting in repository

• Participants suggested ordering the list of services and ontologies in the repositories alphabetically. This facility will aid users in identifying their target folders more quickly

• Participants suggested adding a filtering mechanism to the repositories dialog window to enable efficient search for particular services and ontologies

• Participants suggested that the repository list should have a quick filter which enables the end user to quickly identify their target destination.

Concepts definition

• Participants suggested to add a description/definition to ontology concepts and properties. This feature is important to enable annotators to understand the various types of concepts and select the appropriate concepts and properties

Concepts sorting

• Participants suggested to group concepts with relatively close semantic meaning together (e.g. group Country and Street concepts under Address concept), or / and order the list of concepts alphabetically

Search facility

• They also asked to enrich search boxes with automatic suggestion so that the relevant concepts / services are suggested to the users as they type

• Participants requested to add a search facility by criteria on the iServe to enable them to find services according to specific features like the name of the provider

Representation of annotations on editor

• Participants suggested to show the annotations to the side of the service elements and change the icons of the annotated elements to indicate they have been annotated

Representation of annotations on iServe

• Participants suggested to group different elements together to make it easier to locate and make sense of their annotations on iServe

• Participants suggested that the presentation of information in iServe needs to be improved through the implementation of a time stamp or the opportunity for users to name services.

Redundant information and steps

• Participants felt that some parts of the annotation editor show some parts which are not required for the task of annotation. For example, the binding part of in the WSDL browser. Therefore they advised to remove such sections as they confuse them

• Participants requested that the system should list the available services and ontologies without the need to press on the ‘List’ button when opening a service or ontology; they felt this is an unnecessary step

Persistent sessions

• Participants requested to keep their annotation sessions alive when they navigate to different areas of the SOA4All platform or when they visit iServe to check their annotations



• Participants suggested that the system starting point should begin with where the last session was ended.

• Participants suggested that when loading or viewing repository folders the system should refer the user back to the last folder viewed and remember the last point of history.

Navigation • Participants wondered whether it is possible to make menus hovering-sensitive so that they can invoke sub-menus without having to click on menu items

• Participants requested to place all editors in one place (e.g. under one category). However, they also specified that it would be helpful to keep the current type of categorisation (i.e. task-based menu). Users can switch between the two types of menus according to their preference and proficiency

Visual presentation

• Participants suggested that the menu contrast and spacing should be enhanced to improve both visibility and navigation. A participant felt that there is far too much empty space on the left hand menu.

• A participant suggested that not all elements or types should be represented by an icon as this overcomplicates visual presentation.

Terminology • Participants suggested improving the menu titles as they are not very descriptive and do not correspond with their application.

Level of abstraction

• Participants suggested that the level of information displayed during the process of annotation needs to be context dependent so that end users are not overloaded with descriptions which they view as both excessive and confusing. Furthermore, a participant suggested that this could be achieved through a filtering option that enables end users to view types as and when they are needed.

• A participant suggested that the process of annotation should not be XML based and that end users should be presented with a more abstract view when mapping service elements to types.

Visualisation of mapping

• Participants wanted to toggle on and off / visualise mapping connections from ontology concepts to service elements in order to see the whole picture and evaluate their mapping as they go along

System help and guidance

• Participants suggested to provide some system hints along with system suggestions to help them with the non-trivial task of mapping; in this respect they wanted to select particular service elements and expect the system to highlight the potential concepts that match their selection

• Participants asked for more information and hints about the different parts of the annotation editor. These hints should be precise and accurate; for instance participants requested to add hints to differentiate between repositories and services

• Participants suggested the implementation of a ‘help file’ to provide additional information as to the elements presented, what they are and how they interact with one another.

• A participant also suggested that the annotation process should be linked to supporting documentation.

• Participants proposed the implementation of a ‘wizard guide’ to enable users to identify the ‘big picture’ of the annotation process and the order in which tasks need to be completed.

Simple and advanced view

• Participants proposed to incorporate a simple view of the annotation editor for novice users as well as an advanced view for more experienced users to match their skills and expectations. Users should be able to switch between the two types of views as necessary

Automatic Mapping

• Participants suggested the semi/automations of the mappings as it is difficult for the end user to know which types need matching with which service models

Table 26: User suggestions to improve the WSMO-Lite Studio



Theme Suggestion

Visual presentation

• Participants requested to make the position of the ‘AddToFavorites’ button more prominent by moving it, for example, to the top of the window

• A participant suggested having an independent button under the consumption menu to add favourite services.

• A participant suggested moving the buttons (i.e. minimise and close) to the other side of the screen.

System feedback

• Participants suggested to update the list of services for every new search instead of creating a new result window at the bottom of platform

• Participants requested to show feedback messages in order to inform them of the output of their search queries, especially in the case where no match is found

• Participants requested to prolong the exposure of the popup messages at the bottom of the consumption platform as they are shown only for a short interval and it was difficult for them to read them

• A participant suggested the implementation of a tutorial guide to aid end users in deciphering what is expected from them and in what format.

Result representation

• Participants requested to present the results of their search queries in the form of a cloud in case they do not know the exact name of the target service

• Participants requested to show the result of their search queries at the bottom of the search box rather than to the right

• Participants have suggested that new search results should be displayed at the top of the screen above the old search results and the old screen minimised to enhance the end users view.

• A participant suggested that the pop up search screens should be expanded automatically.

• A participant suggested that there should be a representation of the relationships between services found and services searched for.

Service description and quality

Many users have stated that the information accompanying services is insufficient and does not provide the end user with noteworthy data that can be used to differentiate between candidate services. In regards to this matter the following suggestions have been made:

• Participants asked to add a description / definition to each service to illustrate its purpose, functionality and elements/features

• Participants asked to add user rating and comments to each service, such as services used by Facebook friends or name of providers, to assert the quality of services and facilitate the process of selecting the relevant and correct services from a list of services. Rating of service features should to be self-descriptive and meaningful such as the rating of reliability, rating of security, and number of downloads

• Participants have suggested that services should be accompanied with functional and non-functional data.

• Participants suggested that all services should be accompanied with detailed service descriptions.

• Participants suggested that by hovering over the service star rating more information should be made available.

• Participants suggested that the ‘details’ tab should be replaced with technical detail, QoS or a name that provides meaningful information.

• Participants suggested that they would like more information about services and their providers.

• Participants suggested that service descriptions should be human readable and not presented in WSDL form.

• Participants suggested that they would like information as to whether services can be trialled.

• Participants suggested that although a rating system can be applied, it has to be backed up with comments and other valuable information.



• Participants suggested that functional data must be provided for critical services.

• Participants suggested that for important services there must be an SLA to ensure that the user receives a standard level of service.

• Participants suggested that public services should attain some level of certification, whilst for private services there should be mechanisms to provide direct communication between the user and the provider

Service discovery

• Participants wanted to find services using particular ontology concepts; for example it would have been beneficial if they could retrieve services that are annotated with the concept ‘Address’

Direct of manipulation of services

• Participants requested to be able to resize the windows (i.e. services) to see all the details of the service

Search facility - Automatic suggestion

• Participants suggested that partial service results should be shown in parallel as the search word is typed.

Service Comparison

• Participants suggested that they would like to compare services and would like service information to be presented in a manner that can facilitate assessment.

• Participants suggested that there should be an option to compare services side by side within a table that outlines both their functional and non-functional characteristics so that they can ascertain their differences.

Access to list of favourite services

• Participants requested to make the list of favourite services more prominent and accessible

• Participants suggested that a more logical location for ‘Favorites’ is the ‘Recommendation’ menu; alternatively to create a favorites menu similar to Firefox’s

Terminology • Participants asked to use the appropriate and precise terminology to describe features and options of the consumption platform

Input mode • Participants suggested to enable search using the ‘Enter’ keyboard key instead of restricting that to Search button only

Table 27: User suggestions to improve the Consumption Platform

Theme Suggestion

Visual Representation

• Participants suggested that the menu system should be made clearer. • Participants suggested that the parameters should be set on the flow rather than

in the activity.

Naming of Activities/Connections

• Participants suggested the development of simpler way in which annotations can be named so that they are easily recognisable.

System Guidance and Help

• Participants suggested that they would like more information to be provided in regards to what the different types are during the semantic annotation phase.

• A participant suggested that definitions should be made available for the operators and parameters.

• Participants suggested that there should be a hint guide in the concept to aid with selection.

• Participants suggested that the technical aspect of the left hand menu should be changed to a more abstract level.



Service Identification Through Criteria

• Participants suggested that criteria should be made available to aid the identification of services.

Visual Alignment

• Participants suggested the automatic ordering and aligning of objects during the design phase.

Redundant Step

• Participants suggested that both the ‘load ontology’ and ‘file upload’ buttons are unnecessary and should be removed.

System Feedback

• Participants suggested a form of system acknowledgement following the creation of conditions.

Drag & Drop • Participants suggested using the standard format of ‘drag and drop’ when creating activities since this is what they are used too.

Context Menu

• Participants suggested that it would be better if they were able to interact with the properties of icons through a menu system that would appear when right clicking on that icon.

Terminology • Participants suggested changing the names of the menu options so that they correspond better with the tasks they represent.

• Participants suggested that the editor should have a more specific name which represents its purpose.

Personalisation

• Participants suggested that they would like the development approach to be personal rather than constrained.

Delete Feature

• Participants suggested that the end user should be able to move connections between activities rather than having to delete them and potentially risk losing information.

Automatic Modelling

• Participants suggested automating the model development process as currently it is problematic having to move, tag and name activities.

Table 28: User suggestions to improve the Process Editor

Theme Suggestion

Visual Presentation

• Participants suggested changing the colour schema of the pop up messages (e.g. completion message should not be written in white).

Assisted Modelling

• Participants suggested that they would like to be able to group and breakdown tasks and diagrams so that they can be specifically designed in times where many services need to be modelled.

• Participants suggested that they would like to see the different elements grouped so that they can view the model more easily.

• Participants suggested that the initial and complete model should be presented together so that they can see the changes that have taken place.

• Participants suggested that as long as the initial model works then they should not be presented with the complete model.

Service Comparison

• Participants suggested that they would like to be presented with a choice when selecting services.

System Guidance

• Participants suggested that they would like to see definitions outlining the different functions available.

• Participants suggested that they would like to be presented with an explanation



and Help of each activity. • Participants suggested that they would like an improvement in the quality of

messages that inform users on their activities.

Drag & Drop • Participants suggested that they would like the tool to have consistency with other popular modelling tools and use ‘drag & drop’ in addition to the use of ‘multiple selection’ and ‘move’ of objects.

Table 29: User suggestions to improve the Assisted Modelling

Theme Suggestion

Level of Expertise

• Participants suggested that the ideal end user for the Studio would be those with knowledge of the environment.

• Participants suggested that the ideal end user for the Studio would be non-programmers with an IT background.

Service Description

• Participants suggested that services should provide guarantees and follow protocols.

• Participants suggested that they would prefer to look for services which are certified, in particular when dealing with sensitive data.

• Participants suggested that the issue of data security is an important factor when dealing with financial services.

System Guidance and Help

• Participants suggested the need for a help wizard to guide users. • Participants suggested the presentation of the different steps so that end users

can be guided through the complete process in the correct sequence. • Participants suggested that they would like to see how parameters can be

created.

Categorisation of Favourites

• Participants suggested that the selection process of favourite services should be context dependent and manageable.

Service Classification

• Participants suggested creating permissions and conditions that will allow specific people to work on specific services.

Mapping • Participants suggested that they would like the annotation process to include automatic (suggestion of) mapping.

Service Discovery

• Participants suggested that they would like services to be presented with a greater depth of information so that end users can make informed decisions as to their capability.

Table 30: User suggestions to improve the SOA4All Studio


In March/April 2011 (M37) the last W7 usability study was carried out in Manchester Business School, based on storyboard 6. The purpose of this evaluation was to carry out end-to-end evaluation of a (WP7) use case and to gather user opinion about SOA4All studio and its tool.

The study involved the use of annotation tool (WSMO-Lite Editor), consumption platform (SPICES), SOA4All process editor and associated tools such KPI wizard. In this respect, the study allowed us to gather data about user interaction with the SOA4All tools and identify any usability issues that users might experience during their interaction with the tools.

6.4.1 Participants’ Background

In total 6 users took part in this evaluation study. At the start of the study we asked the user to complete a background questionnaire, which was designed to capture users’ skills and experience in relation to IT training, development of service-based applications, use of service development languages and prior experience of using SOA4All Studio.



The results of the background questionnaire reveal that all participants had IT background and were aware of service development languages and systems.

6.4.2 Evaluation Procedure and Scenario

The aim of the Storyboard 6 evaluation study was to test SOA4All tools in a realistic scenario, gather user opinion about using the tools and identify usability problems that user might face during the course of user-based testing.

Like the previous (Storyboard 5) usability testing, the Storyboard 6 usability testing consisted of three main parts preceded by a training session. A detailed description of the training session and the three main parts is given below:

6.4.2.1 Training Session

The training session was organized as one-to-one session where each participant was briefed about the notion of software services, web services in particular, the various tools that constitute the SOA4All Studio and their features and an overview of the upcoming tasks. After the introduction of the SOA4All Studio participants were given an overview of the student registration scenario and were instructed to developed a service-based application based on the scenario using the three key aspects of SOA4All Studio i.e. annotation, consumption and process modeling. As per previous studies, users were asked to think-aloud while performing their tasks so that their thoughts about various aspects of the Studio and design strategies can be recorded for analysis at a later stage.

Participants were encouraged to ask questions and raise their concerns about any aspects discussed in the training session.

After the training session participants were referred to the workbook where the main tasks of the usability testing were elaborated.

6.4.2.2 Development Phase

After the training session the participants were asked to follow the tasks prescribed in the workbook. According to the workbook, before commencing to perform any task users were required to explore and interact with the SOA4All Studio in order to familiarize themselves with the available functionality and the interface. User interaction with the Studio was recorded using a screen capture software for post-study analysis.

6.4.2.3 Scenario and Task Description

The tasks that participants were asked to perform involved: annotation, consumption and modeling. Starting from annotation the task required participants to open a service and an ontology in the WMSO-Lite annotation platform and annotate the data types and operation of the service using concepts from the ontology. Once the annotation is complete participants were required to upload the annotation service to iServce repository. In the consumption task participants were asked to retrieve the previously annotated service in the consumption platform (SPICES) and add its operation to their favorites. Off course this required participants to be already registered/logged in to the SOA4All Studio. Once a service operation was added to their favorites participants were asked to model the student registration process in the process editor tool. Process modeling also involved binding appropriate services with respective activities in the process model and establishing conditions and data flow connections between various activities/services in the process model. Once the participants completed the process modeling task, they were asked to optimize the process using the KPI-based optimization wizard.

The description of the task scenario is as follows: “Your goal is to complete an overseas student registration process. For this, you need to develop a software application which allows you to search for a UK university, register for a course in the university and make payment for the university fee. There are two ways for paying the university fee, the first way is to open a bank account and get funds transferred into that account. The bank account can be used to make payment for a university fee. In the second way you can request a letter from a sponsor and submit that letter to the university. You must choose only one way to pay the university fee. After paying the university fee you will make booking for university accommodation and register with NHS”.



6.4.2.4 Rating Phase

After perform each task (i.e. annotation, consumption and modeling) participants were asked to rate the different features of the tools by filling in a questionnaire and report the feed in a post-study interview. At the end of the study participants were asked to report their final feedback about the overall experience of interacting with SOA4All Studio and write down negative and positive feature of the Studio.

6.4.3 Analysis Method

Two Human Factors’ experts (one of which conducted the user studies) watched the videos of user interaction with the Studio, listened to users’ vocal discourse and transcribed the conversations into an excel file. This step is important as it allowed the experts to familiarize themselves with the data. Each expert worked independently by going through, preparing and analyzing the transcripts for each task category (i.e. annotation, consumption, modeling). In this analysis phase, the experts coded the opinion data and identified corresponding themes following thematic analysis; a qualitative analysis method for analyzing data involving the identification of a number of themes which emerge from the opinion data. Once finished, the two experts sat together and discussed the emerged themes; they removed the reoccurring themes and refined/paraphrased some themes for better accuracy.

6.4.4 Results

This section presents the main outcomes of the usability testing based on Storyboard 6 scenario. For each tool (tested in this scenario) the comments of the participants are organized according to the emerging themes.

6.4.4.1 Positive Comments about the WSMO-Lite Editor


Opening Services and Ontologies

• Participants found it easy to look for desired services and ontology within the WSMO-Lite platform

• Participants liked the way a service and ontology could be opened side by side for the annotation.

Mapping • The simplicity of drag and drop feature was praised by the participants since it enabled participants to perform the mapping/annotation task quickly

iServe • Exporting annotations to iServe was regarded as an easy task • Participants found iServe browser easy to browse

Visual presentation - GUI

• Participants liked the GUI of the SOA4All studio. • Participants liked the layout of WSMO-Lite platform along with the feedback

that it provides while exporting annotated services to the iServe.

Terminology • Participants found the terminology, used to define various elements and functions, simple and easy to understand

Wizard • Participants found the KPI wizard very informative as it provides guidance about various steps of optimization

Table 31: Positive user comments for WSMO-Lite Editor

Figure 20 shows the average user rating for the WSMO-Lite editor. All ratings are performed on 1-5 Likert scale, where 1=disagree and 5=agree. As shown in Figure 20 in this case most of the ratings are above average, which represents a positive attitude of users about the annotation tool i.e. WSMO-Lite editor. In particular the question ‘service annotation is easy to achieve’ was rated above 4 (on average).



Figure 20: Average rating of WSMO-Lite Editor

6.4.4.2 Positive Comments about the WSMO-Lite Editor

Search • Participants found searching for services easy and straight forward • Participants liked the way search results can be opened in several windows

where each window shows the details of a particular service

GUI • Participants found the service descriptions clear and easy to understand • The status messages or pop-ups shown on the screen were considered

informative and useful by some participants •

Drag and drop

• Participants appreciated the drag and drop feature in the consumption platform • Participants considered it useful to be able to organize search result window(s)

by drag and drop

Wizard • Participants liked the informative nature of wizard and found it interesting

Table 32: Positive user comments for Consumption platform

The consumption platform received positive ratings from the participants of the study owing to its ease of use and simple and straight forward GUI. As shown in Figure 21 participants rated all questions above average.

Figure 21: Average rating of Consumption platform



6.4.4.3 Positive Comments about the Process Editor

Modeling • Participants liked the simple and easy to use way of modeling a process • Participants like the flexibility of organizing the various shapes in the process

editor

GUI • Participants found top menu useful and easy to easy to understand. • Participants praised the layout of the editor

Drag and drop

• Participants appreciated the drag and drop feature in the process editor • Participants considered it useful to connect activities by simple drag and drop

Wizard • Participants liked the informative nature of KPI wizard

Adding Functionality

• Participants found it easy to bind services with activities and in this way add functionality in the process model.

Table 33: Positive user comments for Process editor

The user ratings for the process editor are above average. In particular, on average 4.5 users considered process modelling useful and the same number of users believed that they can create powerful applications using the process editor.

Figure 22: Average rating of Process Editor

6.4.4.4 Comments about Overall SOA4All Studio

At the end of the study participants were asked to rate the SOA4All Studio based on the their overall experience of interacting with its key tools. The comments received from the user are listed in Table 34.



Visual Representation

• Participants liked the visual representations within the various tools of the studio.

• Participants liked the top menus and side panels of the studio.

Level of Abstraction

• Participants liked the simple terminologies and visual representations were used to hide the underlying complexity in the annotation and process modeling phases.

Drag and drop • Participants appreciated the use of drag and drop feature in the studio.

Navigation • Participants liked the simplicity of left menu and the main screen (switchboard) of the studio.

• Participants also liked that the left menu was in every tool to ease navigation.

Functionality Offered

• Participants found the functionality offered by the studio adequate to perform their tasks easily and efficiently.

Table 34: Positive user comments for SOA4All Studio

The average ratings for the overall SOA4All Studio show a positive attitude of participants as most of the ratings are well above average, as shown in Figure 23.

Figure 23: Average ratings for Overall Soa4All Studio

6.4.4.5 User Suggestions for Potential Improvements in the SOA4All Studio

The participants of our study also made some suggestions for future improvements to SOA4All Studio. These suggestions relate to various aspects of the Studio and are listed below in Table 35.

Them Suggestion

KPI Wizard Participants suggested that when a wizard is running the user should be allowed to do anything else with the Studio.

GUI – Consumption Platform

A participant suggested that the GUI of consumption platform should be changed in order to show full description of search results rather than just service names



Search Results in Consumption Platform

Participants suggested that when they click on a search result it should be shown in a new full-size window

Editor Participants suggested introducing pop-up help in the editor to guide users in carrying out their process modeling task.

Terminology Participants suggested that explanation of technical terms (e.g. WSMO-Lite, SPICES etc) should be given in some form in order to ease the understanding for non-technical user.

Editor - Organization of Activities

Participants suggested that it would be easy for them to organize activities in a process model if they can group various activities and drag multiple activities at the same time.

Table 35: User suggestions

6.4.4.6 Evaluation Improvements

For the two usability evaluation studies (conducted month 34 and 37) participants were asked to score their satisfaction by rating a number of criteria on a 5-point Likert scale, where 1 signifies “disagree” and 5 signifies “agree”. Feedback results from the participants enabled the developers to measure the progress made in the user interface of the SOA4All tools between the two evaluation studies undertaken.

The thick line in the centre of each spider-graph below represents neutral opinion, the outer circle signifies most positive, whilst the centre line indicates most negative. Furthermore, the usability questions which contain ‘NOT’ have been inverted from negative to positive statements. It is evident that on most criteria the last evaluation in Manchester represents significant improvement versus the status of the tool during the last-but-one evaluation in Sofia. We have conducted analysis to discover those improvements which were statistically significant, these are marked with asterisk behind the criteria statement.

Figure 24: Annotation Satisfaction Rating

1,00

3,00

5,00

Service annotation is

a rewarding task

Service annotation is

easy to achieve *

The annotation

editor is easy to use

*

Service Annotation is

NOT time consuming

*

I know which parts

of the service I need

to annotate

I know which

concepts I need to

annotate a service …

The annotation

editor provides

sufficient …

I need specialist skills

and knowledge to

annotate services

Manchester

Sofia



Figure 25: Consumption Satisfaction Rating

Figure 26: Process Modelling Satisfaction Rating

1

3

5

Consumption

platform is NOT

difficult to use

It is easy to learn to

find services using

the Consumption …

Finding services is

easy to achieve *

Consumption

platform responds

quickly to my …

Consumption

platform is reliable

Consuming (i.e.

executing) services is

useful

Manchester

Sofia

1,00

3,00

5,00Process modelling is useful

Process modelling is easy to achieve

Process Editor is easy to use *

I would like to use the Process Editor frequently in the future

There were NOT too many steps required to model a process

I can create applications that facilitate my job activities using the Process

Editor *

It is NOT too difficult to understand the modelling notations

I feel confident using the Process Editor *

I can create powerful applications using the Process Editor *

I do NOT feel confused using the Process Editor

I know which services/operations to bind to the activities of the process model *

Process Modelling is NOT time consuming

Binding service operations to activities is NOT a difficult task

It is easy to define conditions using the Process Editor *

You do NOT need to be a programmer to use the Process Editor

M S

*

*



Figure 27: SOA4All Studio Satisfaction Rating

1

3

5

I would like to use the

SOA4All Studio frequently

in the future *

SOA4All Studio is easy to

use

SOA4All Studio is NOT

difficult to navigate

M

S



7. Value Proposition of SOA4All 7.1 State-Of-The-Art before SOA4All

Our studies have established that whilst our target group of users was already undertaking a wide spectrum of EUD activities, there was no support tuned to the area of software services as used for delivery of eGovernment services. The existing eGovernment portals and platforms were tailor-made and did not provide any customization facilities. They were also poor in relation to personalization and context-aware services.

At the same time the demand for adaptation and customization activities was brought up by our end users, who can see the added value of these activities in terms of improved efficiency and tuning the e-Government services to their needs. Indeed some 75% of our “IT-naïve” respondents have declared they use advanced customization features for their office organizational software such as setting filtering and forwarding rules for email, which are considered EUD activities. In the wider context, EUD practices were widely reported, with 50% of our “IT-naïve” respondents indicating they have done at least five EUD-style activities.

7.2 Value Proposition of SOA4All

SOA4All delivered advanced provisioning, consumption, customization, composition and monitoring facilities to the hands of the target population of users and providers of e-Government services. This is unique value proposition which is not currently replicated by any other tool and development system in existence.

The WP7 adaptations of the SOA4All platform provide customized access to existing eGovernment services and ability to optimize service compositions using Key Performance Indicators (KPIs), which serve to match the requirements of the target group of users and the specific features of the target domain of e-Government.



8. Conclusions WP7 implements an open service delivery platform that allows civil servants to handle typical administrative procedures (such as an application for registering a new business). More specifically, using the Web-based tools of the SOA4All Studio, civil servants can search, model, annotate, modify, share, analyze, and execute administrative procedures in the form of lightweight processes. These processes may be composed of Enterprise Services (hosted by SAP), public Web services (hosted by 3rd party service providers), and human activities (to be executed by end users).

This service delivery platform was largely implemented by leveraging the functional components provided by the technical work packages of SOA4All. But in order to meet all requirements of the use case scenario, some customizations and extensions had to be developed within WP7. Thus, in addition to investigating the public sector business case and to providing a basis for the technical validation and end user evaluation of the project results, the key technical contributions of WP7 to SOA4All are an adaptation and integration layer together with semantic artifacts so that SAP Enterprise Services can be consumed in SOA4All, additional tools and services for handling human tasks, a wizard plugin for supporting users of the WP2 Composer, an extension of the WP6 Optimizer realizing the KPl-based process modeling of Storyboard 3, and a City Portal as frontend for consuming administrative procedures. The scientific concepts behind the WP7 demonstrator have successfully been published in conference papers and a book chapter, with some journal papers in preparation.

In the final phase of the project, the WP7 scenario and demonstrator were evaluated from a number of perspectives as reported in this deliverable.



9. References

[Lombardi2010] Jean-Philippe Lombardi, Juergen Vogel, Karl Duebon, Wizard-based Process Modeling for Business Users, EUD4Services, Rome, Italy, May 2010

[Bandara2007] Bandara, W., Indulska, M., Chong, S., and Sadiq, S. Major issues in business process management: An expert perspective. In ECIS 2007. University of St. Gallen, 2007

[Filipowska2009] Filipowska, A., Kaczmarek, M., Kowalkiewicz, M., Zhou, X., and Born, M. Procedure and guidelines for evaluation of bpm methodologies. Business Process Management Journal 15, 3 (2009), 336–357.

[Meuser2002] Meuser, M. and Nagel, U. (2002) „ExpertInneninterviews – vielfach erprobt, wenig bedacht. Ein Beitrag zur qualitativen Methodendiskussion,“ in A. Bogner, B. Littig, and W. Menz (eds.), Das Experteninterview. Opladen: Leske & Budrich. pp. 71-95.

[Meyer2010] Sonja Meyer, Juergen Vogel, Nikolay Mehandjiev, A Case Study: e-Government Service Provisioning and Adaptation by End Users, Case Studies in Service Innovation (hosted by SSMEnetUK) pp. 63-65, June 14-15, 2010, Manchester, England.

[Rosser2008] Rosser, B. Taking advantage of user-friendly business process modeling. Gartner Research Note, G00156919 (2008).

[Ko2004] Ko, A. J., Myers, B. A., and Aung, H. H. Six learning barriers in end-user programming systems. In VLHCC ’04: Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing (Washington, DC, USA, 2004), IEEE Computer Society, pp. 199–206.

[Sutcliffe2000] Sutcliffe, A., Ryan, M., Doubleday, A., and Springett, M. Model mismatch analysis: towards a deeper explanation of users’ usability problems. Behaviour and Information Technology 19, 1 (2000), 43–55.



Appendix A: M28 User Study Workbook Dear participant,

this exercise is part of the evaluation procedure of the Key Performance Indicator (KPI) based modelling panel of the SOA4All research

project. The developed application simplifies the handling of the SOA4All

Editor from a business point of view which is part of the research solution SOA4All Studio. The SOA4All Editor enables end users to model intuitively

lightweight business processes. During the process modeling different process activities are connected with each other. With the help of the so

called “conservations” several services can be bound to one activity. But how can you find the right service? The KPI tool supports you to select

a certain service that is assigned to a particular Key Performance Indicator (For instance “select a service that is searching for a customer with the

minimum price”). After once a process is modelled and you have selected all services for

each activity the KPI panel provides you the option to optimize your process regarding certain KPIs (For instance “model a customer

registration process that is sustainable and has the minimum price”). This task is not as easy as it sounds, because the dependencies between the

different services have to be considered.

Please read once the exercise on the next page. After this you will

have the opportunity to ask questions and to clarify ambiguities.

Thank you, your SOA4All Team!



1. Process modelling without KPI Panel

Task 1.1 Cheapest Business Process (guided)

� Please load the process for creation of a public opinion survey by clicking on the “Load Process” button in the “Key Performance

Indicator Panel”.

� Please get familiar with the process by going through the activities and observing the bound conversations and services. Now begins your real

work! Your task is to provide the cheapest possible business

process! Please consider the fact that the price of some services may

contain more than one price component!

� Please select only one single service per conversation by deleting the ones which do not contribute to fulfil your task. By clicking on a

conversation you are able to observe the bound services and their

different service properties. By clicking on the “Calculate” button you

can see the values of the key performance indicators you are

interested in for a combination of services. (You can add these

indicators by clicking on “Add key performance indicators to the

process”.) Please keep in mind that after deleting a service you won’t

be able to add it again to the conversation, therefore you’ll need to

load the process again and start again!

Task 1.2 Minimum Number of Service Providers

� Please repeat the steps of task 1 with the following goal: You have to provide the business process of a public opinion survey with the

minimum number of different service providers.

2. Process modelling with KPI Panel + Optimise Function

Task 2.1 Cheapest Business Process (guided)

� Please load the process for creation of a public opinion survey by clicking on the “Load Process” button in the “Key Performance

Indicator Panel”.

� Please select the appropriate key performance indicator for minimizing the price of your business process.

� Please click on “Optimise” and have a look at the different conversations.

Task 2.2 Minimum Number of Service Providers

� Please repeat the steps of task 3 with the following goal: You have to provide the business process with the minimum number of different

service providers.



Thank you for your participation!

Evaluation Information

This observation guideline is for the observers for the SOA4All KPI-Tool

evaluation activity. Please use this guideline to document your impression about the evaluation.

Date:

Time:

Location:

Participant:

Background:

9 Business Administration 9 Computer Science

9 Information Systems 9 Other

___________________________________



KPI-BASED MODELLING - EXERCISE

Did you have problems to understand the exercise? 9 No

9 Yes

Please describe the questions of the participant and the provided help



TASK 1.1: PROCESS MODELLING WITHOUT KPI PANEL

Needed time:

Number of steps to get the final solution:

Did the participant have problems to load the process?

9 No 9 Yes

Did the participant have problems to understand the presentation of the

process? 9 No

9 Yes

Did the participant have problems to navigate through the activities, the

bound conversations and services? 9 No

9 Yes

Did the participant use the calculate button? 9 No

9 Yes

Did the participant use the right KPI? 9 No

9 Yes

Did the participant achieve the optimal solution? 9 Yes

9 No Distance from optimal solution:

Number of mistakes to get final solution:

Please describe problems the participant had with this task:



TASK 1.2: PROCESS MODELLING WITHOUT KPI PANEL Needed time:


Did the participant have problems to load the process? 9 No

9 Yes


9 Yes




Please describe problems the participant has with this task:



TASK 2.1: PROCESS MODELLING WITH KPI PANEL + OPTIMISE FUNCTION

Needed time:



9 No

9 Yes


9 Yes

Did the participant use the optimize function? 9 No

9 Yes







TASK 2.2: PROCESS MODELLING WITH KPI PANEL + OPTIMISE FUNCTION

Needed time:



9 No

9 Yes


9 Yes

Did the participant use the optimize function? 9 No

9 Yes







KPI-BASED MODELLING – QUESTIONNAIRE, FEEDBACK

Please answer the following questions: Do not ask the participant – make estimations.

The participant knew what to do (clear description of the

evaluation task)? 9 Strongly disagree

9 Disagree 9 Neutral

9 Agree 9 Strongly agree

The participant knew how to use the KPI panel of the SOA4All

Platform? 9 Strongly disagree

9 Disagree

9 Neutral 9 Agree

9 Strongly agree

Please ask the participant for feedback.

Are there positive or negative things you like to mention regarding

the KPI-based modeling panel?



Appendix B: M33: SAP Research Day Questionnaire SOA4All Project

Please provide the following information by ticking (�) where appropriate:

1. My role at SAP

(Pre-)Sales (Business)Consulting Service/Support

Training Custom Development Administration

2. Please rank your software development experience :

• Application development using programming languages (e.g. Java)

• Web service development

• Analysis and design of service architectures

• Experience with Process Modeling tools

None 1 2 3 4 5 Expert

3. How do you evaluate the SOA4All concept?

• SOA4All is likely to make the services composition easy to achieve

• SOA4All is likely to provide tools that are easy to use

• SOA4All could reduce application development cost

• I would like to use SOA4All platform for sharing services and processes

• It is not necessary to have technical skills to use the SOA4All tool

Disagree 1 2 3 4 5 Agree

4. SOA4All in the Public Sector

•••• SOA4All is applicable in the Public Sector and other industries in a business context

•••• The Public Sector is a relevant industry for SOA4All

•••• SOA4All may lead to an IT costs reduction in Public Administrations

•••• SOA4All supports collaborative work between different public institution


5. Exploiting SOA4All for SAP

• Current SAP solutions allow fast and business user-friendly BPM

• SOA4All could complement or enhance existing SAP solutions

• SAP customers are willing to invest in new BPM and services technologies such as SOA4All

• SAP solutions based on SOA4All could open new markets (e.g., SMEs)




Appendix C: M34 User Study Workbook

EVALUATION OF SOA4ALL STUDIO

Participant Information Sheet

The aim of this software testing is to collect your opinions about the SOA4All Studio, an integrated service development environment, for opening up software service development to people without technical and programming skills and making it as easy as creating a personal blog or a customised Facebook page. The current experiment consists of three main parts and will take approximately 1 hour and a half to complete.

PART A – Pre-test Questionnaire and Interview (~ 15 min)

1. Complete a Participant Background Form 2. Discuss your experience and opinions about Software and Service Development

PART B – Interaction with the SOA4All Studio (~ 1 hour)

1. Interact with a service development environment “SOA4All Studio” and complete a

set of development tasks

PART C – Post-test Questionnaire and Interview (~ 20 min)

1. Rate your service development experience and satisfaction towards the SOA4All

Studio 2. Discuss your experience and opinions about the SOA4All Studio



PART A – Participant Background Questionnaire

Please state your agreement with the following questions by ticking the appropriate box (�) Personal Details: Age: _______________ Gender: M F Current job / Course of Studies: _________________________________ Background: Please indicate the highest level of education you have completed.

High school Undergraduate Diploma/Certificate Masters PhD Others, please specify: _________________________________________________ Which of the options below best describes the level of your IT training?

none self-taught Introduction to office software or similar

non-IT degree non-IT degree with significant IT training

IT-focused degree or significant vocational training such as Microsoft Certified Professional (MCP), etc.

other Please rank your software experience according to the following criteria, where 1 corresponds to poor and 5 corresponds to excellent: • My experience in using software development

environments (e.g. Eclipse, NetBeans, Microsoft Visual Studio) is:

Please list the software analysis and design notations (e.g. UML) you have experience with:

1 2 3 4 5

poor excellent

…………………………….

…………………………….

……………………………. How often do you develop service-based applications?

daily weekly monthly less often never What are your favourite service development languages or systems? 1.________________________________________ 2.________________________________________ 3.________________________________________ Please indicate your knowledge and expertise with the SOA4All Studio: Service Development Tool

I have used it before? Experience

1 2 3 4 5

SOA4All Studio Yes No poor excellent



PART B – Interaction with the SOA4All Studio

- Before starting the tasks of the scenario, please freely explore and interact with the SOA4All Studio for 5 minutes and report your initial impressions and opinions.

Scenario Description

Your goal is to conduct a survey among a selected number of citizens of a particular street in order to gather feedback about a planned playground in their neighborhood. For this you need to develop a software application which allows you to find citizens’ email and postal addresses and send them the survey to fill in. To realise this application you will use the SOA4All Studio which allows you to annotate, discover, and model a process both manually and using the help of the system (i.e. assisted modelling).

Task 1:

- Start the WSMO-Lite Editor - Annotate the WSDL of service “email_accessibility_check” using the data domain ontology “wp7-sap-m33-eval-ontology”. Both the service and the ontology are located in the “wp7-evaluation” repository.

• You need to annotate the data types from the request/response message input/output data structures with the corresponding concepts from the ontology.

• You need to annotate the operation(s) in the service WSDL with concept(s) from the functional classification in the ontology to make them discoverable by Functional Classification criterion.

- Export the annotations to iServe.

- Access the iServe using the link (http://iserve.kmi.open.ac.uk/page/services/) and ensure that your annotated service exists in iServe.

EmailAccessibilityCheckService

Task 2:

- Start the Consumption Platform - Find service “CitizenEmailAvailabilityScanService1”

1, 2 Refer to the description of this service in the services’ descriptions sheet to know what it does and what kind of parameters it is expected to have



- Add operation to your list of favourites. - Find service “PrintSurveyQuestionnaireService2” - Add operation to your list of favourites.



Task 3.1:

- Start the Composer - Build a process model (“Surveying of citizens”) which retrieves email and postal addresses of citizens to send them a survey to fill in. The survey shall be sent following these conditions:

1. There need to be at least 20 people who can be contacted using e-mail or postal address in a particular street; otherwise no survey should be sent out (i.e. the process should be ended).

2. If a person can be contacted by both e-mail and postal address, e-mail has to be

preferred, as e-mail surveys are 5 times cheaper than postal survey. This should be a human decision and implemented using a human task element in the process model. It should be called “survey decision”. During this task an office clerk at the municipality will decide whether to pursue exclusively an-email survey campaign (in case enough residents have emails) or whether to pursue a postal and email survey campaign simultaneously (in case not enough residents have email).

3. Depending on the office clerk decision, the survey will be sent to the residents. Postal

surveys need to be printed before posting them. - Manually bind two activities of the process model (“Surveying of citizens”) with the appropriate services/operations. - Add semantic annotations to one of the activities in your process model. - Upload the process model (“Complete_survey_model”) - Discuss how the complete survey model differs from your solution

Task 3.2:

- Upload the initial process model “InitialWP7EvaluationProcessMIO.xml” - Resolve the initial process model

Task 3.3:

- Optimise the new process model.

Task 3.4:

- Deploy the new process model.



Services’ descriptions

• CitizenEmailAvailabilityScanService

This service checks how many citizens from a certain street could be reached by email. The input of the service is the country, the zip code, the city and the street. It returns, aside from a Boolean value which indicates whether citizens with emails have been found, a percentageOfAvailableEmails which signifies the percentage of citizens reachable by mail and with absolute values the number of email addresses (numberOfAvailableEmails) and the number of all residents from this street (numberOfResidents).

• GetCitizenEmailsService

This service returns all email addresses for the citizens from a certain street. The inputs will be the same as CitizenEmailAvailabilityScanService’s and the return would be, aside from a Boolean value indicating whether the service had worked properly, a List of emails.

• GetOnlyPostalReachableCitizensService

This service finds all citizens of a street which are not reachable via email and returns a List of them. The inputs and outputs are the same as CitizenEmailAvailabilityScanService’s and the return would be a List of citizens with names and addresses and as additional information the number of citizens found.

• OnlineSurveyDeliveryService

The service takes as input a List of email addresses and a String which points to the questionnaire and sends the messages to the email addresses of the citizens.

• PostalDeliveryServiceService

This service sends the questionnaire to the citizens. The input of this service is a List of citizens’ data containing names and addresses.

• PrintSurveyQuestionnaireService

This service needs a link to the questionnaire and the number of citizens to decide how many surveys should be printed.



PART C – Rating of the SOA4All Studio

Rating of the SOA4All Studio

Please rate the following questions by expressing your agreement to each of them on a 5-point scale, where 1= disagree and 5= agree.

Rating of the SOA4All Studio Annotation 1 2 3 4 5 Service annotation is easy to achieve disagree agree Service annotation is time consuming disagree agree Service annotation is a rewarding task disagree agree I know which parts of the service I need to annotate disagree agree I know which concepts I need to annotate a service with disagree agree I need specialist skills and knowledge to annotate services disagree agree The annotation editor is easy to use disagree agree The annotation editor prevents erroneous annotations disagree agree The annotation editor provides sufficient instructions in case of problems

disagree agree

Consumption 1 2 3 4 5 Finding services is easy to achieve disagree agree Consuming (i.e. executing) services is useful disagree agree Consumption platform responds quickly to my search queries disagree agree Consumption platform is difficult to use disagree agree Consumption platform is reliable disagree agree Using the consumption platform, I can easily retrieve the services I want

disagree agree

Process Modelling 1 2 3 4 5 Process modelling is easy to achieve disagree agree Process modelling is time consuming disagree agree Process modelling is a rewarding activity disagree agree The process editor is easy to use disagree agree I feel confused using the process editor disagree agree I feel confident using the process editor disagree agree There were too many steps required to model a process disagree agree It is difficult to understand the modelling notations (e.g. merge, human task) used in the process editor

disagree agree

It is easy to express conditions using the process editor disagree agree I can create powerful applications using the process editor disagree agree I can create applications that facilitate my job activities using the process editor

disagree agree

You do not need to be a programmer to use the process editor disagree agree Binding services/operations to activities is a difficult task disagree agree I know which services/operations to bind to the activities of my process model

disagree agree



It is easy to annotate activities of a process model disagree agree Assisted modelling simplifies process modelling disagree agree Assisted modelling is efficient disagree agree Assisted modelling saves time and efforts disagree agree The optimiser facilitates the binding of services to activities disagree agree The optimiser is useful disagree agree I am willing to use the process editor to model processes in the future

disagree agree

Overall Rating of the SOA4All Studio 1 2 3 4 5 Overall, the SOA4All Studio is easy to use disagree agree Overall, the SOA4All Studio is difficult to navigate disagree agree I am willing to use SOA4All Studio in the future disagree agree The SOA4All Studio is a good substitute to traditional software development environments such as Eclipse

disagree agree

A) Please list the top five features you like about the SOA4All Studio

1. _________________________________________________________________ 2. _________________________________________________________________ 3. _________________________________________________________________ 4. _________________________________________________________________ 5. _________________________________________________________________

B) Please list the worst five features you dislike about the SOA4All Studio

1. _________________________________________________________________ 2. _________________________________________________________________ 3. _________________________________________________________________ 4. _________________________________________________________________ 5. _________________________________________________________________



Appendix D: M37 User Study Workbook

EVALUATION OF SOA4ALL STUDIO

Participant Information Sheet

The aim of this software testing is to collect your opinions about the SOA4All Studio, an integrated service development environment, for opening up software service development to people without technical and programming skills and making it as easy as creating a personal blog or a customised Facebook page.

The current experiment consists of three main parts and will take approximately 1 hour and a half to complete.

PART A – Pre-test Questionnaire and Interview (~ 15 min)

3. Complete a Participant Background Form 4. Discuss your experience and opinions about Software and Service Development

PART B – Interaction with the SOA4All Studio (~ 1 hour)

2. Interact with a service development environment “SOA4All Studio” and complete a set of development tasks

PART C – Post-test Questionnaire and Interview (~ 15 min)

3. Rate your service development experience and satisfaction towards the SOA4All Studio

4. Discuss your experience and opinions about the SOA4All Studio



PART A – Participant Background Questionnaire

Please indicate your knowledge and expertise with the SOA4All Studio, where 1 corresponds to poor and 5 corresponds to excellent:

Service Development Tool I have used it before? Experience

1 2 3 4 5

SOA4All Studio Yes No poor excellent



PART B – Interaction with the SOA4All Studio

- Before starting the tasks of the scenario, please freely explore and interact with the SOA4All Studio for 5 minutes and report your initial impressions and opinions.

Scenario Description

This scenario describes the registration process that overseas students go through while getting admission in UK universities.

Your goal is to complete an overseas student registration process. For this you need to develop a software application which allows you to search for a UK university, register for a course in the university and make payment for the university fee. There are two ways for paying the university fee, the first way is to open a bank account and get funds transferred into that account. The bank account can be used to make payment for the university fee. In the second way you can request a letter from a sponsor and submit that letter to the university. You must choose only one way to pay the university fee. After paying the university fee you will make booking for university accommodation and register with NHS.

To realise this application you will use the SOA4All Studio which allows you to annotate, discover, and model a process both manually and using the help of the system (i.e. assisted modelling).

Task 1:

- Start the WSMO-Lite Editor

- Annotate the WSDL of service “SearchUniversity” using the domain ontology “StudentRegistration_Ontology.wsml”.

(Your moderator will help you locate the wsdl service and the ontology)

• You need to annotate the message input/output in the operation with the corresponding concepts from the ontology.

• You also need to annotate the “SearchUniversity” operation in the service WSDL with concept(s) from the ontology to make it discoverable.

- Export the annotations to iServe.

(use following details for logging in to iServe: username=”Liwei Liu” password= “111”)

- Copy the URL shown on the confirmation message and paste it in the address bar of a new browser window to see you annotated service in iServe



Task 2:

- Log into the studio using the ‘Profile’ tab.

(login details will be provided by your moderator)

- Start the Consumption Platform

Find the following services and add their operations in your list of favourites – one by one

- Find service “SearchUniversity3”

- Find service “PayUniversityFee”

1 Refer to the description of services in the services’ descriptions sheet to know what it does and what kind of parameters it is expected to have



Task 3.1:

- Start the Editor

- Build a process model (“Student Registration”) which registers overseas students in a UK university and implement the following conditions (the scenario description at the beginning of Part B may help you in this task)

(a) You can pay university fee in one of two ways: you can either pay the fee using a bank account or submit a sponsor letter. Specify appropriate conditions so that University fee can be paid in only one of the above options.

(b) If you want to open a bank account then you must transfer funds in that account in order to make the fee payment.

- Manually bind two activities of the process model with the appropriate services/operations (SearchUniversity and PayUniversityFee).

- Add semantic annotations to one of the activities in your process model.

(use available ontology)

- Save your process model

- Open and analyse the completed process model (Student Registration).

- Discuss how the complete registration model differs from your solution

Task 3.2:

- Optimize the completed process model using the KPI wizard.



Services’ descriptions

University Service

Operation: SearchUniversity- given the name of a city the operation returns a list of universities in that city. The input is a string indicating city name, the output is a list of strings or ‘0’ if no university is found.

CourseRegistration Service

Operation: CourseRegistration- register a course in the university. The inputs are the course details, the output is registration confirmation.

FindAccomodation Service

Operations: FindAccomodation–The input to this operation is the accommodation name (that can be found on the accommodation list on university website), the output is booking confirmation for the accommodation (keep in mind oversees students are guaranteed a place in university accommodation of their choice).

BankAccount Service

Operation: OpenBankAccount – open a bank account. This operation takes the account request as input and the output is the account details.

TransferFunds Service

Operation: TransferFunds – Transfer funds between two accounts. The input is the transfer request, the output is transfer conformation.

SponsorLetter Service

Operation: SponsorLetter- Get letter from sponsor. The input is the letter request. The output is a sponsor letter.

PayUniversityFee Service

Operation: PayUniversityFee- Pay university fee. The input for this operation is a request, which can be either bank account details from where money can be debited or a letter from sponsor. The output is payment confirmation corresponding to the university registration.

RegisterNHS Service

Operation: RegisterNHS - Register with NHS. The input is registration request and the output is registration confirmation.



PART C – Rating of the SOA4All Studio


Please rate the following questions by expressing your agreement to each of them on a 5-point scale, where 1 = Strongly disagree and 5 = Strongly agree.


Annotation 1 2 3 4 5

Service annotation is easy to achieve Strongly disagree Strongly agree

Service annotation is time consuming Strongly disagree Strongly agree

Service annotation is a rewarding task Strongly disagree Strongly agree

I know which parts of the service I need to annotate Strongly disagree Strongly agree

I know which concepts I need to annotate a service with Strongly disagree Strongly agree

I need specialist skills and knowledge to annotate services

Strongly disagree Strongly agree

The annotation editor is easy to use Strongly disagree Strongly agree

The annotation editor prevents erroneous annotations Strongly disagree Strongly agree

The annotation editor provides sufficient instructions in case of problems


Overall, I am satisfied with the Annotation platform Strongly disagree Strongly agree

Consumption 1 2 3 4 5

Finding services is easy to achieve Strongly disagree Strongly



agree

Consuming (i.e. executing) services is useful Strongly disagree Strongly agree

Consumption platform responds quickly to my search queries


Consumption platform is difficult to use Strongly disagree Strongly agree

Consumption platform is reliable Strongly disagree Strongly agree

Consumption platform provides sufficient help instructions


It is easy to learn to find services using the Consumption platform


Overall, I am satisfied with the Consumption platform Strongly disagree Strongly agree

Process Modelling 1 2 3 4 5

Process modelling is easy to achieve Strongly disagree Strongly agree

Process modelling is time consuming Strongly disagree Strongly agree

Process modelling is useful Strongly disagree Strongly agree

Process Editor is easy to use Strongly disagree Strongly agree

I feel confused using the Process Editor Strongly disagree Strongly agree

It is easy to learn to model a process using the Process EditorStrongly disagree Strongly agree

I feel confident using the Process Editor Strongly disagree Strongly agree

There were too many steps required to model a process Strongly disagree Strongly



agree

It is difficult to understand the modelling notations

(e.g. activity, split) used in the Process Editor


It is easy to define conditions using the Process Editor Strongly disagree Strongly agree

I can create powerful applications using the Process EditorStrongly disagree Strongly agree

I can create applications that facilitate my job activities using

the Process Editor


You need to be a programmer to use the Process Editor Strongly disagree Strongly agree

Binding services/operations to activities is a difficult task Strongly disagree Strongly agree

I know which services/operations to bind to the activities of

the process model


The Dataflow editor is difficult to use Strongly disagree Strongly agree

I know which input of a service to map to which output of a

preceding service in the Dataflow editor


I would like to use the Process Editor frequently in the futureStrongly disagree Strongly agree

Overall, I am satisfied with the Process Editor Strongly disagree Strongly agree

Overall Rating of the SOA4All Studio 1 2 3 4 5

SOA4All Studio is easy to use Strongly disagree Strongly agree

SOA4All Studio is difficult to navigate Strongly disagree Strongly agree

The interface of the SOA4All Studio is pleasant Strongly disagree Strongly agree



It is easy to learn to use the SOA4All Studio Strongly disagree Strongly agree

SOA4All Studio contains various tools and features that are

well integrated


SOA4All Studio provides sufficient help and documentation Strongly disagree Strongly agree

People would need the support of technical people to be able

to use the SOA4All Studio


I would like to use the SOA4All Studio frequently in the

future


Overall, I am satisfied with the SOA4All Studio Strongly disagree Strongly agree

A) Please list the top five features you like about the SOA4All Studio

1. _________________________________________________________________

2. _________________________________________________________________

3. _________________________________________________________________

4. _________________________________________________________________

5. _________________________________________________________________

B) Please list the worst five features you dislike about the SOA4All Studio

1. _________________________________________________________________

2. _________________________________________________________________

3. _________________________________________________________________



4. _________________________________________________________________

5. _________________________________________________________________



Appendix E: Use Case Storyboards This appendix summarizes the use case storyboards described previously in deliverables D7.2, D7.3, D7.4, D7.5 and D7.6.

WP7 investigates the potential of SOA4All concepts and technologies. Of particular interest hereby is whether business users with limited IT skills (i.e., users who are able to surf the Internet and use Office and business applications for their daily tasks but cannot program) can leverage SOA4All to search, consume, compose, and monitor Web services (see D7.2). For this purpose, we have defined several scenarios with typical usage patterns. The scenarios relate to the public sector domain where civil servants of the fictional “City of X” are to handle administrative procedures over the combined SOA4All Service Delivery Platform4. All scenarios involve different actors with different roles and responsibilities to reflect the typical division of work in larger organizations (see Figure 28 and Table 36). For small organizations some or all of these roles could be combined.

Actor Role(s) Profile Story-boards

Barbara Process Expert (Service Composer)

Barbara is a process expert at the City of X (back office) and helps to create process models using the process editor of the SOA4All Studio. She also writes the manuals for the process models and annotates them with different meta data so that they become searchable and usable for the service consumers.

Barbara has good business knowledge of the administrative

1

4 Please note that similar scenarios can also be found in other domains and functional units such as supply chain management in the automotive industry.

Barbara

process expert

Egon

domain expertJose

citizen

SOA4All Studio

SS/DSB/Monitoring

SOA4All Platform Services

SOA4All technology in the Public Sector –

Administrative Services in a City Portal

Claudia

legal expert

ProcessService

annotation

Process

modelingProcess

executionlifecycle

Adam

service developer

Figure 28: SOA4All Components and Users




procedures and the administration’s interactions with constituents and service providers. She has typical end user IT skills such as using office applications, or a browser to access information from the Internet. She is also an experienced end user.

Claudia Legal Expert (Service Composer)

Claudia is a legal expert at the City of X (back office) and needs to check the compliance of a process before it becomes executable.

Claudia has good business and legal knowledge about administrative procedures. She has typical end user IT skills such as using office applications, or a browser to access information from the Internet.

1

Egon Domain Expert (Service Composer, Service Consumer)

Egon is a domain expert for handling administrative procedures at the City of X related to (new and existing) small and medium enterprises. He resides in the administration’s front office and uses the service platform to handle specific service request by constituents and executes some of the human tasks involved in such procedures.

Egon has good business knowledge about his own field of duty and limited business knowledge about other fields. He has limited IT skills but is familiar with Internet browsers in general and the SOA4All Studio in particular.

1, 2, 3, 4, 5

Jose Citizen (Service Consumer)

Jose is a Spanish citizen that requests and handles the administrative procedure to register a business via the Internet portal of the City of X. Thus, Jose is an indirect service consumer.

Jose is an experienced Manager of his coffee store chain and has some knowledge about administrative procedures in his country of origin (Spain). He is an advanced end user of IT technologies, in particular Internet applications like wikis, online stores, and interactive maps.

1

Adam Service Developer

Adam is a service developer at SAP. He prepares the services (incl. SAP Enterprise Services) that should be available via SOA4All by modeling the required service and domain ontologies as well as the lifting and lowering schemas for services discovery, consumption and composition. For this purpose he is using the Service Provisioning tools and the Grounding Editor.

2, 5

SAP and others

Service Provider

SAP and other service providers make their web services accessible via the service platform. They employ teams of IT specialists covering different skills (service development, service provisioning)

1, 2, 3, 4, 5

SAP and SOA4All

Platform Vendor

The SOA4All consortium develops the components of the service platform that are customized by SAP to the specific needs of the use case. They employ teams of IT specialists covering different skills (platforms, programming, service development, service provisioning)

1, 2, 3, 4, 5




Public Admini-stration City of X

Platform Provider, Service Provider

The City of X hosts the SOA4All service delivery platform within the city’s Intranet. It employs one or more IT administrator(s).

1, 2, 3, 4, 5, 6

Table 36: Actors in the different storyboards

Storyboard 1: City of X - Administrative Processes

The goal of Storyboard 1 is to allow civil servants of a (fictitious) city to model, manage, and execute administrative procedures, which are composed of Web services incl. SAP Enterprise Services and human tasks, via the SOA4All tools. This allows a public administration to efficiently implement new and adapt existing administrative procedures such that e.g., the requirements of the EU Services Directive can be met. As an example, the process for registering a new business at the City of X is modeled in SOA4All (see D7.2 for a detailed description). This storyboard has been introduced with D7.2 and been continuously updated in deliverables D7.3, D7.4, D7.5, and D7.6, respectively. It was the main storyboard for the technical evaluation described in sections 5.1 and 5.2, and the basis for the ideas of the other storyboards. It was also used for the economic evaluations in the first two years of the project.

Overview

The City of X aims to implement the EU Services Directive and realize a constituent-friendly, one-stop e-Government solution based on the SOA4All service delivery platform. Barbara is a process expert in the City of X and her task is to create process models for selected standardized administrative procedures for the City of X. In our example, Barbara models the process displayed in Figure 29 of registering a business using the process editor (the so-called composer) of the SOA4All Studio (also see D2.6.3) and the related SOA4All components depicted in (also see Chapter 3). Before this process is ready to be deployed, the responsible manager Claudia verifies that it actually complies with all legal requirements and regulations.

Sometime later, the City of X decides to simplify payment procedures for its citizens and now wants to support payment by credit card in addition to the traditional invoicing scheme. Consequently, all administrative processes that involve a payment need to be adapted. Thanks to SOA4All, this modification is rather simple and therefore the workload for these adjustments can be distributed among the employees of the City of X. In our example, Egon usually handles the incoming “Registration of a Business” requests and as a domain expert therefore is able to modify the existing process model to include the new payment method.

Figure 29: Storyboard 1 Initial Process Model (Screenshot from SOA4All Process Editor)



In the final phase of the sample scenario, the citizen Jose uses the Internet portal of the City of X to register his newest coffee shop: Jose is a Spanish citizen who owns the coffee store chain “Café Solo” that has been founded and located in Madrid, Spain. The coffee chain was very successful in recent years due to the overall boom for luxurious and trendy coffee stores in Spain. In order to further expand his business and enter a new regional market, Jose decides to invest into a new branch in the City of X, Germany. Usually this process would be weary and take up a considerable amount of time and money for travelling and for researching information. With the single point of contact principle implemented by the City of X, Jose is able to manage the entire procedure via the web interface from his office in Madrid. In the corresponding form of the city’s Internet portal, Jose fills in all required information. The according administrative procedure is then handled by Egon, who executes the human tasks and uses the SOA4All Studio to monitor the process status (see Figure 31).

The execution of the sample processes includes human tasks and SAP Enterprise Services. Documentation about the SAP Enterprise Services is available at https:// www.sdn.sap.com/irj/scn/wiki?path=/display/ESpackag es/Home.

Detailed Storyboard


0 PREPARATION

services

• deploy business services (SAP ESI Layer) on DSB node • add business services to favorites list

service annotations

• store service ontologies via storage services • annotate service interfaces (WSDL) in Studio Editor (SOWER) ->

SAWSDL is automatically uploaded to iServe • create lifting / lowering schemas

WP2



WP3


WP7

• services

create process

model Co

mp

ose

r SS

/DS

B

iServe

Storage Service

Human Task Server

Re

aso

ne

r

Design Time Composer

Optimizer

Execution Engine SA

P E

S

LPM

L A

PI

DiscoveryBarbara

process expert

Figure 30: Process Modeling with SOA4All

Egon

domain expert monitor, interact

assess request Analysis Tool (Monitoring)

HT Client HT Server

Execution Engine

SS / DSB

SAP ES

Figure 31: Process Execution with SOA4All




• ontologies

1a

Barbara

Barbara models the first version of the “Registration of a Business” process as depicted in Figure 16, Deliverable D7.2. She logs in to the SOA4All service platform (i.e., the SOA4All Studio) with her browser and starts the SOA4All Composer by selecting the corresponding icon in the SOA4All Dashboard

WP2

• Studio and Dashboard

• User Authentication

1b

Barbara

Barbara draws a new process model for the “Registration of a Business” Process V1: First, she creates a new process, adds the start element, and activities. For each activity, she can define properties such as the name. She connects these elements (and thus defines the control flow)

WP2

• Composer

WP6

• LPML

1c

Barbara

From her favorites list containing the services she most frequently uses, Barbara binds a service to the first activity.

WP2


• Composer • iServe

WP6

• LPML

WP7

• services




1d

Barbara

For the other activities, Barbara adds semantic annotations such as functional classifications and constraints that specify the activity. Invoking the Design Time Composer via the “Resolve” button, Barbara lets the SOA4All tools automatically find fitting services for all activities She can check the result via the “Process Elements Properties” window in the left hand panel.

WP2



WP3

• Reasoner

WP5

• Discovery

WP6

• LPML • DTC

WP7


1e

Barbara

Barbara knows that new processes need to be checked by Claudia for legal compliance before it can be released. She leaves a comment for Claudia for a part of the process where she is unsure. Finally, Barbara saves the process. A unique URI is created to reference the process, and the process becomes available in the shared repository of the City of X so that it can be accessed by her colleagues.

WP2


• Composer

WP6

• LPML




2a

Claudia

Claudia has the task to check the legal compliance of new or updated processes before they can be deployed. She logs in to the SOA4All service platform (i.e., the SOA4All Studio) with her browser. First, she starts the SOA4All Composer by selecting the corresponding icon in the SOA4All Dashboard. In the “Load Dialog” she can see that there is a new process, created by Barbara that she has to check.

WP2




• Composer

WP6

• LPML

2b

Claudia

Claudia reviews the process details in the SOA4All Composer. Satisfied with the compliance of the process, she leaves a textual comment for her colleagues. Finally, she saves the process back to the repository.

WP2


• Composer

WP6

• LPML

3a

Egon

Egon has the task to adjust the payment part of the “Registration of a Business” process model to include credit card payments. First, he logs into the SOA4All Studio, selects the composer, and opens the “register business” process.

WP2




• Composer

WP6

• LPML




3b

Egon

Egon replaces the existing semantic annotation for the payment activity with an updated one that can be obtained by browsing a domain specific contextual ontological model. Egon then invokes the DTC via Resolve Process entry, which may replace activities with process fragments, bind services, generate data flow connectors, and check I/O semantic compatibility. The result is a complete (executable) process model. Once a composition has been generated, the optimizer aims to assign the best services in the composition based on their KPIs. Therefore the most optimal composition is automatically generated without involving Egon in this complex process. Next, Egon invokes the “Deployer” to transfer the process mould to the Process Execution Engine- the process is now ready to run. Finally, Egon saves the modified process model.

WP2



WP3

• Reasoner

WP5

• Discovery

WP6

• LPML • DTC • Optimizer • Deployer • Execution

Engine

WP7

• services




4 Jose

Jose accesses the Internet portal of the City of X to register his business. Jose opens the appropriate service request and enters the required information. When he sends the request, a new instance of the “Registration of a Business” process is instantiated at the server side and Jose’s data is handed over.

WP1

• DSB • Semantic

Space • Monitoring

WP6

• Execution Engine

WP7

• services • City Portal

5a

Egon

Within the services platform, Egon handles all incoming requests by constituents that are assigned automatically to him (considering his area of expertise and availability). In our example, Egon handles the process for Jose and executes the human tasks in the process. Switching to the Human Task Client, Egon can retrieve all open tasks which are assigned to him. First, he needs to authenticate.

WP7

• Human Task Client

• Human Task Server

5b

Egon

After a successful login, the Human Task Client application is shown.

Then he selects an active task and enters the result once he has completed it. The result is handed back to the Execution Engine which had halted the process instance in the meantime.

WP6

• Execution Engine

WP7

• Human Task Client

• Human Task Server




Storyboard 2: City of X - Service Provisioning and Discovery

The goals of Storyboard 2 are (1) to prepare existing services such as the SAP Enterprise Services so that they are accessible at the SOA4All platform e.g., to introduce new functionalities that can be used to model new administrative procedures, (2) to let users search for services that match their requirements, and (3) to let users consume individual services. Storyboard 2 was introduced in deliverable D7.5 and has been implemented for the M33 milestone. The results with respect to the technical validation are described in Section 5.2.

Overview

The storyboard has the following three phases: (1) service discovery, (2) service preparation or provisioning, and (3) service consumption. In the first phase, the service developer Adam checks via the SOA4All Service Discovery whether all services required to implement the processes of the first and third storyboard (see Sections 0 and 0) are already available. In the second phase, Adam uses the SOA4All tools SOWER and Grounding Editor to prepare the missing services such that they become available for discovery, consumption, and composition within SOA4All. He achieves this by adding the required semantic descriptions to the WSDL-based Web services (see Figure 32). In the third phase, the process expert Barbara uses the SPICES consumption platform to find a specific service for the process of the first storyboard, to invoke it for test purposes, and to bookmark it for later reuse in process modeling.

Detailed Storyboard


0 PREPARATION

services

• deploy business services (SAP ESI Layer) on DSB node

service annotations

• store domain and service ontologies via storage services • store xsd data schemas via storage services

WP2


WP7


Adam

service developer

SOUR

SWEET

Grounding Editor

SAP ES

Analysis tool SS / DSB

deploy, annotate,

monitor

prepare services

iSe

rve

Figure 32: Service Provisioning in SOA4All




1a

Adam

First, Adam logs into the SOA4All Studio. Then he opens the Service Discovery Tool.

WP2


WP5

• Service Discovery

1b

Adam

The Service Discovery Tool provides different possibilities to search for a service. Adam first explores the functional classification which represents a taxonomy for the SAP ES. Selecting, e.g., whether Adam is interested in services with read or write access (BusinessDomainOperationalDimension ), he creates a filter.

WP2



WP5


1c

Adam

Here Adam looks for services which are classified for the public sector. The result list is displayed in the middle panel.

WP2



• iServe

WP3

• Reasoner

WP5





1d

Adam

In order to further narrow down the result list, Adam can specify preconditions that the service should fulfill, e.g., which input parameters it should take (here: name and address of a citizen). In our case, Adam does not find what he is looking for.

WP2



• iServe

WP3

• Reasoner

WP5


2a

Adam

Adam now needs to prepare the missing services such that they become available for discovery, consumption, and composition within SOA4All. In the first step, he wants to create the lifting and lowering schemas that are required for data mapping when invoking a service through SOA4All. More concise, these schemas map the data objects of the syntactic WSDL service interface to the semantic instances of the data ontology. For this, he opens the Grounding Editor via the SOA4All Studio Dashboard.

WP2



WP3





2b

Adam

For each SAP ES, Adam needs to create a mapping: He selects “New Mapping” in the File menu and selects the appropriate ontology and xsd schema (that have been uploaded to the WP2 storage services in the preparation step).

WP2


WP3


WP7

• services • service

ontologies

2c

Adam

After loading both the ontology and the xsd schema, they are displayed in the left and right panels of the Grounding Editor, respectively. Now Adam can do the mappings by dragging the schema elements to their ontological counterparts.

WP2


WP3


WP7


ontologies

2d

Adam

As the structure of the schema is with parent and child types and the ontology has this layout too in the ontology are properties in this example the name has two of them one is hasSurname and the other hasFirstname which also needs to be defined by drag and drop the child elements into the property field.

Property fields are all the fields which do not have a simple data type like string.

WP2


WP3


WP7


ontologies




2e

Adam

In case the in- and output have the same type like ServiceParameters here and the user has already added both to the ontology element and tried to add another property the editor offers both possibilities and let the user decide to which of this class mappings the property should be added.

WP2


WP3


WP7


ontologies

2f

Adam Now, Adam is done and can save the created mapping. As a last step, he exports both the lifting and the lowering schema via the Tool menu such that they become available on iServe.

WP2


• iServe

WP3


2g

Adam

In the next step, Adam needs to annotate the different parts of each SAP ES service interface with the corresponding semantic concepts in order to create a semantic service description according to the WSMO-Lite based service model of WP3. For this, he opens the WSMO-lite Editor (aka SOWER) in the SOA4All Studio.

WP2



• SOWER




2h

Adam

To begin with service annotation, Adam loads a service definition, e.g. a WSDL endpoint definition, which is either stored in storage services or taken from a live service endpoint directly. To load a WSDL in storage services, he chooses File, Open and Service Description; to load a live WSDL endpoint, he chooses Service Description from URL. The USDL becomes visible in the right hand panel.

WP2

• Storage services

• SOWER

WP7


ontologies

2i

Adam Next, Adam loads a WSMO-Lite service ontology representing a semantic view of the service. It is used to annotate the elements such as messages, operations, etc. Adam can load it by using either the Ontology from URL item or the Ontology item in the File menu. The loaded ontology becomes visible in the left hand panel. Adam clicks the items in the list of semantic concepts and finds different instances of items corresponding to the respective semantic concepts, e.g. for service operation, messages, etc. These instances can be dragged and dropped to the right panel of the loaded WSDL definition, onto the corresponding structural parts.

WP2


• SOWER

WP7


ontologies




2j

Adam

In order to annotate the XML Schema data types of the service (represented by the wsdl:types subtree), Adam loads a domain ontology containing the concepts corresponding to the data types. This domain ontology is also visible in the left hand panel.

WP2


• SOWER

WP7


ontologies

2k

Adam

Adam now can create the links between the semantic and the syntactic model via drag and drop. Here he starts by dragging the service concept to the WSDL service element. As a result, a SAWSDL model reference will be added to the WSDL service element, pointing to the concept contained in the semantic model. Adam continues to link the operation, request and response messages, data types…

WP2


• SOWER

WP7


ontologies

2l

Adam

Finally, Adam exports the resulting SAWSDL service description to iServe using the Export tool of SOWER. For this, Adam needs to provide his iServe credentials. As a result of a successful export, the unique iServe ID for the service is displayed.

WP2



WP7


ontologies




2m

Adam

Adam can switch to iServe and access the annotated service in detail using the service’s identifier or the iServe search functionality.

WP2

• iServe

3a

Barbara

In the third phase, the process expert Barbara searches for a specific service for the process of the first storyboard: an SAP ES to retrieve the customer data of a citizen based on the citizen’s address. Barbara opens the SPICES service consumption platform via the SOA4All Studio.

WP2



• SPICES

3b

Barbara

SPICES offers a simplified keyword-based search interface on top of iServe, integrates a recommender system that can suggest related services, displays meta information about services such as user ratings, and allows to directly execute service. Here, Barbara searches for “ES” and “Citizen” (SPICES can display several search results at the same time).

WP2







3c

Barbara

Barbara further investigates the details of the ESILayerESCitizenERPBasidDataByAddressQueryResponseService and, satisfied, bookmarks the service to her personal favorite’s list.

WP2






Storyboard 3: City of X - KPI-based Process Modelin g

The third storyboard was introduced with D7.6, which gives a detailed description. It’s focus is on supporting civil servants during service selection and administrative process modeling: Civil servants (and other business users) are primarily not interested in the technical details of a Web service such as the definition of parameters (which is often the focus in service annotations) but on the business aspects associated with that service, e.g., the pricing model, the service quality, the security level etc. Thus, in this storyboard a civil servant can specify certain business requirements in the form of Key Performance Indicators (KPIs) for the business process at hand. Then, the best Web services are selected automatically by SOA4All so that the overall process is optimal with respect to these KPIs. From a technological point of view, the Web services in this scenario have been annotated semantically with business properties following the novel USDL notation (see http://internet-of-services.com ).

We have verified the general idea of supporting business users with KPI-based service selection during process modeling in a quantitative and qualitative usability study described in Section 6.2.

In order to simplify the process modeling task for the targeted end users even further, we have additionally implemented a wizard-plugin for the SOA4All Composer that guides the user through the required steps. The conceptual model for the wizard plugin was developed in a detailed design study (see Storyboard 4 in Section 2.1) with evaluation results described in Section 6.1.

Overview

Egon is a domain expert at the City of X. The City wants to build several new playgrounds and intends to gather feedback from its citizens in order to gather ideas and general feedback. Thus, Egon has the task to conduct a survey among a selected number of citizens about the planned playground in their neighborhood. For this Egon, uses the predefined process depicted in : Figure 6. First, the process retrieves the contact data of all citizens in a certain neighbourhood, then it retrieves the email addresses of those citizens that have one and sends an invitation to an online survey to these. The other citizens are contact via mail. Because the City of X may use different service providers for these activities, Egon has to decide which service to use for which activity. This service selection can be done based on different KPIs, and Egon uses the wizard plugin of the SOA4All composer in order to get support for this complex task.



Detailed Storyboard


0 PREPARATION

services

• deploy business services (SAP ESI Layer) on DSB node

service annotations

• store service ontologies via storage services • annotate service interfaces (WSDL) in Studio Editor (SOWER) ->

SAWSDL is automatically uploaded to iServe • create lifting / lowering schemas

process

• model process • annotate activities • create bindings via DTC or manually • save process

WP1

• DSB

WP2


• SOWER • iServe • Composer

WP3


WP6

• LPML • DTC • Optimizer

WP7


1 Egon

Egon wants to conduct a survey among selected citizens. Because he urgently needs the results, his main concern is the overall response time of the process. At the same time, also the costs should be as low as possible.

First, he logs in to the SOA4All service platform (i.e., the SOA4All Studio) and starts the SOA4All Composer by selecting

WP2



Figure 33: Storyboard 3 Process Model (snapshot from Process Editor)




the corresponding icon in the SOA4All Dashboard

2 Egon

Egon loads the administrations standard survey process, which had been predefined by an experienced colleague, from the internal process repository.

WP2


• Composer

WP6

• LPML

3 Egon

For each activity, the process already contains several possible Web services. Thus, Egon needs to pick one Web service for each activity in order to implement the process. For this task, which can become quite complex and error-prone depending on the number of service providers and available Web services, he starts the KPI wizard provided by the SOA4All Composer.

WP2


• Composer

WP7

• KPI wizard




4 Egon

The wizard follows a stepwise approach in order to break down a complex task. The user can go back and forth between these steps until he is satisfied with the result. Thus, Egon can try several alternatives. He can also cancel the wizard at any point in time, and start over if desired. The wizard first displays a step by step guidelines to assist user through the optimization process.

WP2

• Composer

WP7

• KPI wizard

5 Egon

If wanted, Egon could also select services for some activities manually by deleting the unwanted bindings in the "Process Elements" tab. Here, he simply clicks "Next".

WP2

• Composer

WP7

• KPI wizard

6a

Egon

Now Egon selects those KPIs for the given process, which are relevant for his business requirements, i.e., multi dimensional optimization is also possible. Which KPIs are available depends on the available semantic service annotations.

WP2

• Composer • iServe • Storage

Services

WP7

• KPI wizard




6b

Egon

In this example, Egon wants to minimize both the response time of the process as well as the overall costs.

WP2

• Composer

WP7

• KPI wizard

6c

Egon

The wizard has now collected all information required for the optimization and gives a detailed summary.

WP2

• Composer

WP7

• KPI wizard

7 Egon

Switching to the next step, the actual service selection is executed and the resulting values for the available KPIs are calculated and displayed in the form of bar charts. In that way, Egon may check whether he is satisfied with the achieved result. If not, he can go back and change the KPI selection and thus compare the results and consequences of different KPI selections.

WP2

• Composer • iServe • Storage

Services

WP6

• Optimizer

WP7

• KPI wizard • services




8 Egon

Once Egon is satisfied with the result, he can close the wizard. Even at this stage it is possible to cancel everything and revert the process to its initial state.

WP2

• Composer

WP7

• KPI wizard

9 Egon

Finally, Egon can save, deploy, and execute his process.

WP2



WP6

• LPML • Deployer • Execution

Engine

WP7

• services


soa4all d7.7 use case evaluation 1 - cordis...0.8 25.02.2011 updated sections 4.2 and 4.3, executive...

Documents