merlin 2g - highqsoft€¦ · web application avalon ods server merlin ... 21.05.2015 26 visions...

HighQSoft GmbH | www.highqsoft.de | 21.05.2015

Merlin 2GHighQSoft Evaluation Server

Andreas Hofmann, Constantin BadescuApplication Development


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

2

Content


3

Motivation for Merlin 2G DevelopmentAny test/measurement is subject to analysis

Domain-Experts lose time and resources developing duplicate analysis programs which also run next door.

The analysis programs

• contain a high level on domain specific know-how

• will only run once (locally) and can not be contributed to a automated process


4

Motivation for Merlin 2G DevelopmentStandard Validation, Evaluation & Report

Automated Data validation Evaluation to create new measurements

Evaluation to create attached standard reports

Executed on different reasons

After import

User action

Event driven

Manual Data ValidationCreating data set on request only (e.g. statistic counting)


5


Standard Evaluation ResultsAdministrator/User can only influence the result by defined options

Centralized evaluation administration

Scalable in horizontal (network) as well in vertical (hardware) direction

Separation of concerns

Participate and use of big-data solutions


6


Fat ClientEvaluation Tool

Avalon ODS Server

Merlin Evaluation Server


7



ServerWeb Application

Avalon ODS Server



8



ServerWeb Application

Batch ProcessingImporter

Avalon ODS Server



1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

9

Content


10

Requirement for Merlin 2GEasy to use and implement

AdministrationConfiguration can be done just by file copy operation.Deployment can be done just be file copy operation.

SystemCommunication protocol must be agreeable with firewall boundaries. No CORBAREST interface must be provided to ensure connectivity for everybody and from everywhere.

End-user interfaceEvaluation parameter are described by interface to be able to create generic UIEvaluations can be monitored at runtimeEvaluation input and output is passed by stream


11

Requirement for Merlin 2GEasy to deploy, connect, ….

DevelopmentDifferent evaluation tools / programming languages must be allowed

Data can be read from ODS interface -- ODS security is required

Evaluation can be a collection of evaluations -- Reuse of Evaluations

In case of multiple evaluations are involved, context must be able to transport information from one evaluation to the next

Evaluation scheduler must be defined for different purposes

Evaluations are bundles in context of the OSGi specification

Configuration storage can be switch (ODS, XML File, …)


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

12

Content


13

Merlin 2G Demo

Based on Basic Web Weather Demo Data

Avalon and Merlin Servers are running in the HighQSoft office Glashütten

One measurement describes a set of channels (temperature, wind speed, …) who’s values are each collected hourly and per station and for one month

There are 81 registered stations

We have ~900 measurements per year

There are ~51000 measurements available

The demo is using Advanced REST Client Chrome App as a GUI

Facts and Data


14

Merlin 2G Demo

By HTTP method

1) OPTIONS against the /merlin resource ,we get the WEB service functionality description

2) GET against the /evaluations resource, we get the list of available evaluations

3) POST against the /connection resource, we establish an ASAM ODS connection

4) POST against the /job resource, we execute the evaluation to build ahistogram of the air temperature of all stations in 2014

5) GET against the /jobs resource, we retrieve the list of all managed evaluations and their status

Live demo…

In steps…


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

15

Content


16

Roles

Development and Maintenance

Done by HighQSoft GmbH

Development of domain specific evaluations

This is done on customer side by a team that has the domain knowledge. Normally this is done by a third party company that is located close to customer. In case of ODS storage, ASAM ODS know-how is required.

Evaluation Requirement

This is a group of end users that are qualified to define the requirements for evaluations.Because the requirements are domain specific and independent on any technology, the members of that group must not have any experience with ASAM ODS or the final programming language.


17

Roles

Using the EvaluationsStarted by the end-user from WEB or other program that is in contact to the Merlin server.

Scheduled and started by a time based condition, that runs evaluation in background or in a time when

server has no load. This kind of evaluations are checking consistency of the data or creating complex

reports.

Started automatically as part of a complex process like import.

Any other startup reason (events / cron) is possible.


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

18

Content


Merlin 2G InfrastructureSystem Architecture View

ODS ServerImporter Notification

Indexer

MerlinAuswerteserverWEB Server


Merlin 2G InfrastructureApplication Architecture View


21

Merlin 2G InfrastructureConfiguration of Evaluations

Call

Analysis Program Parameter

1:n

n

1

1:1 1:n

Structure


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

22

Content


23

Implementation

Merlin implementation is based (on top) of HighQSoft GmbH Integration Platform.

Configuration is based on XML files

A JAVA reflection based factory is available only

HighQSoft Integration Platform provides a scheduler that is not used for the demo

Deployment can be done by simple XML Editor and a file transfer tool

Development can be done JAVA IDE that supports OSGi definitions and HighQSoft Integration Platform

What are the details done for the demo


24

Implementation

….

What is currently available?

detaillieren


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

25

Content


26

Visions & Ideas

ASAM ODS support for evaluation configuration

Existing XML configuration to be used for export and import

A Groovy, MATLAB and C/C++ factory implementation

Merlin engine based on Apache YARN for job execution and distributionWIKIPEDIA: A resource-management platform responsible for managing computing resources in clusters and using them for scheduling of users' applications.

Merlin running in a cluster or in a cloud

An evaluation can use a big-data infrastructure like Hadoop or Spark

An evaluation can be a producer for big-data, e.g. histogram of columns or indexing of meta-data.

What’s next?


1) Motivation

2) Requirements

3) Demo

4) Roles

5) Infrastructure

6) Implementation

7) Visions & Ideas

8) Back to Demo

27

Content


28

Merlin 2G Demo

By HTTP method

1) OPTIONS against the /merlin resource ,we get the WEB service functionality description

2) GET against the /evaluations resource, we get the list of available evaluations

3) POST against the /connection resource, we establish an ASAM ODS connection

4) POST against the /job resource, we execute the evaluation to build ahistogram of the air temperature of all stations in 2014

5) GET against the /jobs resource, we retrieve the list of all managed evaluations and their status

Live demo…

In steps…


29

Thank you very much!

merlin 2g - highqsoft€¦ · web application avalon ods server merlin ... 21.05.2015 26 visions...

Documents