agents and semantics for future internet applications 2015_keynote-klusch.pdf · of data, streams,...

56
Agents and Semantics for Future Internet Applications PD. Dr. Matthias Klusch German Research Center for Artificial Intelligence Saarbrücken, Germany 30th ACM Symposium on Applied Computing 16.4.2015, Salamanca, Spain

Upload: others

Post on 20-May-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Agents and Semanticsfor Future Internet Applications

PD. Dr. Matthias KluschGerman Research Center for Artificial Intelligence

Saarbrücken, Germany

30th ACM Symposium on Applied Computing16.4.2015, Salamanca, Spain

Page 2: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Agenda

Future Internet

Perspectives

Agents and Semantic Technologies

Intelligent Applications

Showcases in Manufacturing, Retail,

Private and Social Life

Selected Challenges

2

Page 3: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Internet of Today

Number of users grows2015: 3 billion (1 billion 2005), 50 billion connected devices

Internet economy grows2016: 5% of GDP in G-20 countries

Variety and number ofapplications grows2017: 268 billion mobile apps

Internet usage growsPer minute: 200M emails, 100K tweets, 2M+ search queries, 3K photo uploads, …

Number of cyber attacks grows91% increase in 2013 (62% successful)

3

Page 4: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Future Internet Perspectives …

Internet of ThingsIP-connected, resource-aware, autonomous,self-coordinating smart things (aka smart objects)

Internet of ServicesEverything as a (Web) Service.IP-accessible, interoperable, reusable assets.Coordination in SOAs, in clouds (I/P/SaaS).

Internet of InteractionSocial networks and media sharing.Multimodal, virtual 3D & AR-based.

… each with its own• technological resources, standards

• institutions, research agenda

2017: 10B mobiles, 2050: 50B IP-connected things

4

Page 5: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

… Including a Quantum Internet

Today deployed quantum networks … in USA … in China

Based on quantum computing „Quantum big data“ processing

beyond boundaries of classical computational power

Based on quantum cryptography and networking Perfectly private communication (wired and wireless)

2003: DARPA-fundedQI backbone in USA(Boston, BBN, Harvard)

2014: USA nationwide(incl. Google, IBM, Microsoft,Quantum Data Centers/Labs)

2014 - 2016: China QI backbone(Beijing - Shanghai)

~ 2030: planned

Extension toworldwide QI

2000 km

Seth Lloyd et al. (2004). Infrastructure for the Quantum Internet. ACM SIGCOMM Computer Communications Review Volume 34(5)5

Page 6: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

NSF (USA)

European Commission

Korea

Future Internet Convergence

• Towards what, how, when?

Nobody knows yet …• Several FI initiatives with different

research focus and proposals– FI architecture & infrastructure,

abstractions (SDN)– FI testbeds (FIRE), technologies,

generic enablers (FIWare) for networking, security, etc.

Expectationson FI applications

6

Page 7: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Future Internet Applications: The A* Vision

Smart TransportationSmart Cities

Smart Travel Smart

Home

Smart ManufacturingSmart Retail

Smart Energy

Real-time access to, andassisted coordination

• T Kelly, ITU Report 2005: The A4 vision: Anywhere, anytime, by anyone and anything.• J Hafkesbrink & M Schroll, TII conference 2010: A5 - Anything, Anytime, Anywhere, Anyway, Anyone. • L Castaneda et al. 2012: The Future of Internet Applications. Verizon 2015: State of the Market report. FIRA, FIWARE websites.

Smart Health

Anything, Anywhere, Anytime, on Any device, by Anyone, Intelligent, Safe & Secure

7

Page 8: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Intelligent FI Applications: Agents

From an AI perspective: Intelligent FI applications exhibitautonomous, rational, proactive, adaptive behaviorfor • Problem-solving and decision support

• Data, process and service coordination• Human-agent interaction

Intelligent agent technologies

• Agent modeling, execution platforms(e.g. Bochica, JACK, JADEX)

• Individual or joint AI planning, learning

• Inter-agent communication(e.g. FIPA-ACL, JADE)

• Inter-agent coordination(e.g. eCNP, Negotiation mechanisms, Swarm rules)

software, or robotic, or animalagent

8

Page 9: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Examples: Intelligent Agent Applications

Multiagent system for decentralized steel production lines control and optimisation.

• Multiagent systems for optimal fleet management, logistics.

• Trading Agents for negotiation support on B2B markets,

product recommendation, service brokerage.

Multiagent systemsfor soccer games

Semi-autonomous, cooperativeplanetary surface exploration

9

Page 10: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Intelligent FI Applications: Semantics

• Semantic interoperation and analysisof data, streams, services, processes, actions

• Semantic explanation to human user

Semantic technologies

• Semantic modelling, management, reusewith formal ontologies or linked data cloud sets(e.g. W3C OWL2, W3C SSN, OWLIM, SAWSDL, OWL-S)

• Semantic search, analysis, mediation and compositionof linked data, data streams, processes and services(e.g. Hermit, Pellet, FSPARQL, C-SPARQL, Ztreamy, iSeM, OWLS-Xplan)

lod-cloud.net

10

Page 11: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Sorry …No general selection and usage strategy

for agents and semantic technologiesin intelligent Internet applications

… but lots of individual illustrating showcases

in different application areasIEEE IC March/April 2015

11

Page 12: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

SHOWCASES IN MANUFACTURING AND RETAIL

Agents and Semantics for Intelligent Internet Applications

12

Page 13: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Mechanical Electrical power Electronics IoT+IoS+IoI

Related Vision: Industry 4.0

Vertical and horizontal online integration of all IP-connected data and services

for situated optimal adaptation and execution of manufacturing processes

at runtime, in mixed reality.

Industry 4.0 apps

are IoT+IoS+IoI apps

with (some) A* properties.

In what manufacturing areascan agents and semantics help

13

Page 14: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Area: Condition-Based Machine Maintenance

Condition monitoring systems (CMS)

• Support detection of onset of faults

• Quantitative (statistical) analysis of data streams from IP-networked machine sensors

Results interpretable only by human expertsfor diagnosis and maintenance decisions

Semantic explanation of machine conditions and faults for human experts and non-experts (anyone)

Fast combined quantitative and semantic data analysisonline (anywhere, anytime)

14

Page 15: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: iCM-Hydraulic App

Mobile iCMH web client

PLC

iCMH-System

MachineSensor

Network

Combines statistical, semantic and probabilistic analysis for fault detection and condition diagnosis with human understandable explanations.

Implementation: Java; OWLIM store; C-SPARQL stream analysis engine; reasoners STAR, Hermit; BN tool GeNIe, (MATLAB)

15

Page 16: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

iCMH Domain Ontology

Fact base: RDF encoded state and history data• Component / sensor instances• sensor measurements• detected conditions, faults, symptoms • fault probability values

Concept base:OWL2-DL encoded semantics of

• Machine components, sensors,measured properties

• Component faults, symptoms,condition, operational factors

• Condition-fault-symptom relations

(279 concepts, 184 relations)Modelling

• Expert interviews at HYDAC

• CM standards ISO 2041, 13372,

17359:2011

• W3C Semantic Sensor Network Ontology

(part of concept base)

16

Page 17: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

iCMH Belief Network

Pump Leakage Fault F P(F|C)

Pump Leakage Symptoms S e.g. Pressure Level After Load P(S|F)

Pump Condition C P(C|EF)

External Factors P(EF):e.g. PLC Signal, Operational Pump State

Probabilistic relations:

(36 variable nodes, 60 relations)Dynamic update after eachsymptom, fault detection.

Fault detectionF: Max P(F|S)

Condition diagnosis{S}: P(C|S) >

17

Page 18: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

P( Pump_Leakage = Onset | SPAL = low, POS = active ) = 0.7Probabilistic analysisMax. probable fault state:

EvidencesEvidences

Semantic annotation

Semantic feature streamanalysis• Logical inference of symptoms

(C-SPARQL rules) and diagnosis (Hermit, STAR reasoner)[ Pump_Leakage = Onset, Valve_Op_Degradation = No,Cooling_Op_Degradation = No,Accumulator_Gas_Leakage = No ]

Feature extraction50K 90

Symptom: (Static_Pressure_After_Load hasState “low”), Factor: (Pump_Op_State hasState “active”)

Statistical feature streamanalysis• Trained fault state classifier (LDA)

e.g.

Note: Wrong statistical fault state but correct semantic symptom detection is recognized by BN.

Hybrid Fault Detection Online

Multi-variate sensor data stream

(from PLC for 20 sensors) [ ][ TS = “22.10.2014T22:10:23”; cool_temp = (.., 21.0, 34.0,..); valve_pressure = (11.0, 1.0,..); … ]

1 min machine work cycle

326 KB

18

Page 19: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Semantic Explanation

1. Most likely explanation of machine [component] condition?19

Page 20: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Semantic Explanation (2)

Semantic diagnosis online

(in parallel over stream)

2. Semantic relation between

detected [component] faults?

3. Other components affected

by detected [component] fault?

PLUS: Semantic diagnosis offline.

Query-specific pattern-basedgeneration of explanation.

20

Page 21: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Example

Semantic relation between component faults detected at same time ?

1. Find shortest path in ontology between

faults (ipl_234, agl_456) with STAR reasoner

2. Rule-based interpretation of object relations

in this path, e.g. „object location“ rule:

(?x connectedTo ?a, ?a ... ?b, ?b connectedTo ?y)

(?x before ?y)

3. Aggregation of results into an explanation text pattern for this query type:

Pump pump_123 with internal pump leakage ipl_234 is located before faulty componentaccumulator accu_3457 with gas leakage agl_456, detected at time 12.03.2015 23:00:09.Therefore, detected internal pump leakage might have caused detection of accumulatorgas leakage.

internal pump leakageevent ipl_234

accumulator gas leakageevent agl_456

21

Page 22: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Online Analysis Performance

2015

• Fast average response• Semantic stream: 0.3 sec

• Hybrid fault detection: 25 sec

• Statistic classification: 0.5 sec

• Fault relations: 0.6 sec

• Affected components: 40 sec

• BN exact update: 1 sec

• App-specific fixed throughput • 600 RDF triples/min or 2K triples/min (w/ or w/o feature reduction)

• Scales up to > 80k triples/sec (C-SPARQL SparkWave w/reasoning)

SparkWave

C-SPARQL

? in cloud with STORM-based SPARQLstream[UP Madrid 2014]

22

Page 23: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Next: Online Assisted Maintenance

Extreme case: Self-maintaining „smart objects“ e.g. smart product with embedded semantic memory

(in XML-OMM) of its state and handling across lifecycle

iCM Agent automatically

• Generates maintenance plan based on its

hybrid semantic diagnosis online

– Semantic text retrieval (digital handbook)

– Semantic matching of similar cases (history data)– Semantic state-based planning (world state)

• Informs and instructs mobile worker in mixed reality

… tells machines about its state and how to grasp it (lifting points) –robot decides with which service to handle it in its current state.

av.dfki.de

DFKI FemBot„AILA“

23

Page 24: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Area: Virtual Factory Simulation & Training

Intelligent virtual human-machine interaction (anyone) with

integrated semantic explanation and verification (safe)

in a 3D web space (anytime, anywhere)

Virtual factory simulates manufacturing processes

of real-world factory in 3D environment for

• Testing of optimal functionality, safety,

ergonomics issues of assembly-lines

• Training of workers on (new) machines

Ford, BMW, Volvo, Yamaha, …

24

Page 25: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: ISReal-SmartFactory App

Virtual factory model in annotated XML3D scene

AnnotatedXML3D scene Verification HAVLE

(Hybrid Automata Verification by Location Elimination)

Semantic planningOWLS-Xplan2

Semantic reasoningDL-based, object relational

25

Page 26: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Example

Semantic planning of machine services and

handling actions for given goal

Time verification of created action plan

Explanation (text2speech, simulated plan execution)

User query-answering on functionality of machine:

Can I produce 20 pills

with this machine

within 30 secs ?

Yes, you can. I show you how to handle the machine for that ….26

Page 27: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Example (2)

Time-based verification of plan execution

3D visualization of failure trace + T2S (explanation)

Building of alternative plan with revised state

Why did my handling of

this machine fail ?

Sorry, you lifted the carriage stopper eight seconds too late (max 2.5s) !

Here is your correct control plan for handling this machine…

• Explanation of detected failures of machines, or their handling by user

• Show alternatives to user

27

Page 28: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Area: Collaborative Virtual Factory Engineering

Collaborative virtual design

and simulation of factory

by multiple IP-connected engineers

online.

• Real-time synchronized multiple interactions and simulations

• Support of high-precision, fast joint search for 3D models

• in shared 3D web space

28

Page 29: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: C3D-SmartFactory App

Near-real time synchronized view

• 3D factory model and design actions by multiple engineers• multiple avatars (BDI agents) in scene

in 3D scene of shared XML3D web space

XML3D & FiVeSmiddleware web client

I. Zinnikus et al. (2013): 9th IEEE Intern. Conf. on Collaborative Computing (CollaborateCom)29

Page 30: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Semantic 3D Object Search

Local search with semantic 3D index iRep3D

• Hybrid semantic matchingof annotated XML3D, X3D, COLLADA objects(geometry, text, concept, semantic services)

Distributed P2P search with S2P2P

• Semantic expert-driven query routing

- Local learning of semantic overlay- Joint routing path generation with maximum

#expert peers for query topic within TTL

• Dynamic semantic 3D object replication

• Local alignment of individual annotationontologies to query semantics

X. Cao, M. Klusch (2013): S2P2P. 15th IEEE Intern. Conf. on High-Performance Computing and Communication HPCC. iRep3D. 8th Intern. Conf. on Computer Vision Theory and Applications.

High-precision, robust, fast

• P2P (1M, 3DS-TC): 0.8 AP in 3s • Local:

30

Page 31: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Simulation & Verification

Formal verification of safety properties of designed factory components (with HAVLE)

3D visualisation of failure traces to human user

Simulation of human user behaviorswith configurable BDI-agent types

31

Page 32: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: C3D-Retail App

• Collaborative product placement in virtual supermarket by

• Layout designers • Test customers via instrumented

supermarket (Dual reality)

• Agents simulate types of customerbehavior in virtual supermarket

• Optimization of supermarket layout[customer preferences / sales revenue]

32

Page 33: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: ADIGE App

Product alert: selection, execution

Instrumented supermarket (RFID, EBS)

Productre-orderingprocesses

Virtual supermarket in dual reality management dashboard

2014

Process adaptation to dynamically changing process services(availability, new, SLA) at runtime.

Reactive semantic re-/planning of servicesof annotated process models in OWL2.

DemoVid

33

Page 34: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Challenge: Cloud Manufacturing

Large-scale

• Semantic integration, analysis, search, composition,

negotiation of manufacuring process data & services

in the cloud

large-scale semantic data analysis in the cloud (WebPIE, etc.)

Xu, X. (2012): From cloud computing to cloud manufacturing. Robotics and Computer-Integrated Manufacturing, 28

• Elastic process services execution (wrt. resources, quality, costs)

• Resource-aware semantic services coordination (mediation, planning)

• Process optimisation online (semantic stream reasoning/CEP + reactive service composition + Business analytics)

crema-project.eu (2015 – 2017)

34

Page 35: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Challenge: Human-Agent Teamwork

• Semantic modelling, reasoning and interaction

mutual understanding of behaviors, functional capabilities, incentives, intentions, goals across all team members – fast … very fast …

• Integrated hybrid team planning, negotiation, execution, aggregation

@DFKI (ongoing)orchid.ac.uk (ongoing)

Agent types

1. Software

2. Robots

3. Animals

4. Animoids

interactive.mit.edu

Advanced Manufacturing

charm.sites.olt.ubc.ca (ongoing)

Desaster Rescue

35

Page 36: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Challenge: Quantum Internet of Services

Quantum computing and information processing

assets of QI-connected quantum computers

are provided as quantum services Search, Scientific Simulations (Weather, Life

Sciences, Finance, Defences, Energy),

De-/Encryption, etc.

M Klusch (2008): Toward Quantum Matchmaker Agents. ACM/IEEE Intern. IAT Conference

M Klusch (2004): Toward Quantum Computational Agents. LNAI 2969, Springer.

How to describe their semantics

How to coordinate (select, compose) them for QI apps

with agents on hybrid quantum computers

Source: market research media512qbit (2012), 4.2 PFLOPs, 10M$

36

Page 37: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

SHOWCASES INPRIVATE AND SOCIAL LIFE

Agents and Semantics for Intelligent Internet Applications

37

Page 38: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Area:Intelligent Recommendation

Exploit• Semantic model of domain, user profile,

sensed external data (streams)• Semantic item relevance computation

– Knowledge graph analytical heuristics, DL-reasoning, social network data analysis, etc.

R Yan et al. (2013): Using semantic technology to improve recommender systems based on Slope One. JW Ha et al. (2014): EPE – An embedded personalization engine for mobile users. IEEE Internet Computing, 18(1) J Pazos Arias et al. (2012): Recommender systems for the social Web. R De Virgilio et al. (2012): Semantic search over the Web.

NewsTV channels

MoviesVideos

Music

Improved accuracy of item recommendations compared to non-semantic approaches

Semantic explanation of recommendations− Most relevant properties of item, or N-item property paths in knowledge graph, etc.

Mitigation of cold start problem

38

Page 39: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: SPrank

(1) Computes heuristic path-based item relevancein profile-extended semantic LOD knowledge graph Frequency of different user-item path types (features) j

Semantic relevance ofitem i1 for user u3 (for |p| < 5):

• Collaborative path types#path(1) = (likes, likes, likes): 2

• Content-based path types#path(2) = (likes, p2, p1): 2#path(3) = (likes, p2, p3, p1): 1

• Hybrid path types: |p|>=5#path(4) = (likes,p2,p1,likes,likes)

x31 = (2/5, 2/5, 1/5)

likes i1

i2

u3

i3u2

u1

u4

u: user, i: iteme: entity (new user/item)p: property

e2e4

e1p1

p1p2 p2p2p1

p2

e5

p1 p2

p3

p2

p4

i4

User-item profile

?

p3

e3

Knowledge graph

T. Di Noia et al. (2013): ACM 7th Conf. on Recommender Systems (RecSys)

(2) Simple regression-based learning of rank f(xui) yieldshigher accuracy than with common standards!

(up to 0.6 recall >> BPRLin, SLIM, SMRMF with test data from MovieLens, Last.fm)

D

Ddui

uiui R

dpathjpathjx ∈=

∑∈

)(#)(#)(

39

Page 40: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Example: Semantic Explanation

For N-Item relational queries:

Display the shortest item-item paths

in the knowledge graph ….

(Corr. NP-hard Steiner tree problem solution)

…. and learn to improve accuracy based on implicit/explicit user feedback:

horst.dfki.de

40

Page 41: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: SmartCar-Agent

M Moniri, M Feld, C Müller (2012): Proc. 8th IEEE Conf. on Intelligent Environments.

That looks nice, is it worth a visit?

Human context-sensitive POI recommendation

− Object recognition and cognitive activity level of driver (eye tracking and cognitive load analysis system, semantic image matching)

− Semantic CB recommendation and multimodal presentationbased on actual driver profile and activity level to reduce distraction

Driver profileLocal knowledge graphSemantic image index

Yes, this baroque church, called Ludwigskirche, can be of interest to you + [Info, Nav]

42

Page 42: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcases: Smart Shopping and Cooking Assistance

Instrumented shopping cart with agent

• Searches in huge RDF knowledge graph, text databases, etc.

• Recommends recipes which combination of ingredients have

properties „tasty“, „combinable with each other“,

match user preferences and with novelty bias.

Shopping cart agent

− Reasons on sensed semantic memory of productsin shopping cart and surrounding

− Validates and recommends alternatives wrt. uploaded user profile (e.g. lactose intolerance),

product state (e.g. was opened before, crushed)

43

Page 43: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Area: Smart City and Social Events

Real-time tracking of crowds Identification of newly attractive,

and underserved areas,

Explanation of attraction shifts

Real-time weather forecasts

Real-time traffic monitoring Traffic prediction and diagnosis

Social Listening Milano

Semantic analysis and explanation• Streamed social network, sensor data• Non-streamed public (linked) data sources

may require computational complex combinationof stream querying / CEP, semantic IR, DL-based reasoning, etc.

Olympic Games 2012 London

44

Page 44: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Trading Off Real-Time for Accuracy ?

STAR-City traffic diagnosis system (deployed in Dublin, Bologna, Miami, Rio)

combines fast stream querying with DL-based and probabilistic reasoningto diagnose traffic congestions given sensed actual and history data in cloud.

Diagnosis response times worse than in current TMS but way more accurate

2.7hrs

F Lecue (2014): Semantic Traffic Diagnosis with STAR-CITY. Intern. Semantic Web Conference (ISWC)

Experiments with 6 OWL/LOD ontologies (~140 KB), online & offline data (~70 GBs/day) on vehicleactivity, traffic stats, incidents, road works, social network analysis, social media events planned, etc.

45

Page 45: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Alternative: Brute Force in Cloud

experiments with (petabyte scale) traffic data from highways in Bavaria achieved an avg.

95% prediction accurracy of traffic congestions within 30 minutes

Experiments with using lower expressive semantics e.g. in OWL2-RL

for reasoning revealed that accurate prediction of traffic congestion

events can be achieved in near-real time (within 5 secs).

A Pascale et al. (2015): 21st Sympos. Transporation and Traffic Theory.B Gorman et al. (2014): Traffic Management using RTEC in OWL 2 RL. ISWC

insight-ict.eu

Even without any semantics but e.g.

• Conditional probabilistic analysis of stateevolutions in traffic network graphs

• Multiagent (vehicle) path planning

46

Page 46: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Showcase: MyMedia App

P2P network

• Semantic P2P search & live streaming in one app

• Resource-adaptive streaming for HTTP with MPEG-DASH standard

Real mobile-to-mobile search and sharing of live recodings and videosat social events (festivals, sports parades, theme parks) with friends.

47

Page 47: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Video Semantics

Name: ID1_vds.owl

ProfilehasDescription:

“Keanu Reeves in front oft Cinestar!“

hasInput: -hasOutput: LiveStream,

Topic: Actor, CinemahasPrecondition: -hasEffect: -

GroundingAndroid Activity „Play“

REST InterfaceVLC Player

http:// …/ID1_vds.mpd

Semantics: Service in OWL-S

User tagsfrom imported

ontologyin OWL2

User comment

Klusch, M.; Kapahnke, P. (2012): The iSeM Matchmaker. Web Semantics, 15, Elsevier

Semantic video relevance:

Hybrid semantic service matching (logic, text, structural similarity-based)

Content: MPEG-DASH file

48

Page 48: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

P2P Search & Streaming

User selects video v from returned top-n vids

(2) P2P live streaming of v (with pDASH)

• Initiate P2P streaming session of peer group G for v• Get MPEG-DASH media profile descriptor of v with peer resource info

from each peer p‘ known to p in G

• Make decision: From which p‘ to best download which next segment(s) of v wrt. maximum available resources for each of them ?

• Parallel download & play segments of v from target peers• Update MPD(v); (4)

D: Experts C, B for q; C: Experts E, F for q

Each MyMedia peer p

(1) Semantic expert-driven P2P searchfor given query q (with S2P2P)

49

Page 49: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Performance & User Acceptance

Android 3, Java• @sourceforge.net/projects/mymedia-peer/• pDASH @Uklagenfurt

M Klusch et al. (2014): MyMedia. ACM Intern. Conf. on Mobile and Ubiquitous Computing (Mobiquitous)

User tests: 50 in groups of 2-10; TIFF WLAN; HTC One, Nexus7, S2/S3/S4

• Easy to use; improved user experience

• Issues: Live movie piracy out of theatre; no secured user profiles

Experimental setting: P2P network w/ RPL topology, 1M peers, TTL=10, k=2; uniform at random/sparse distribution of 400 vids,1.4-6Mbps bitrates

• P2P search: AP 0.80, CR10 0.32, AQRT 3s(vs. k-random: 0.35, 0.22, 1.6s, MsgOvh -1.22)

• P2P streaming: 4s avg. latency, BW savings 25%

• Energy [Ws]: Search 1.4, Record 575, Stream 644

Up to 4.7 hrs of usage on Samsung S3

2013, 2014

50

Page 50: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Other FIA Areas for Agents and Semantics …

Smart HomePersonal agent on mobile app interacts with IP-connectedsmart appliances at home

• Remote coordination and execution control of human commands

• Proactive sensing, semantic reasoning and action planningin unprecedented situations at home

Smart home market uncertain: User acceptance still low.

Smart Micro-Grids• Dynamic distributed optimization of energy consumption

• Individual or group rational energy trading and sharingin consumer agent coalitions

- with specialized services and tarifs by provider

51

Page 51: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Challenges

Agents for self-driving cars• Decision-making and navigation based on OWN

semantic perception and understanding of environment

Vs. exact semantic city street map (incl. all traffic signs, lights) produced, updated, distributed by manufacturer

Vs. Human irrational interference, well-being, legal liability

Agents for elderly care

2015 - 2020: Audi A7 „Jack“, Volvo, Ford,Mercedes F015, Tesla, Google, Rinspeed, etc.

Fast building of trusted relationship by human and(non-stupid) humanoid care agent required

Semantics of emotional stances, interaction, etc.

vrworld.com 2015

vrworld.com 2015

52

Page 52: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Social Challenges

… may attract ever more manufacturing companies and public servicesto become vertically/horizontally connected in the FI.

Internet of Everything (IoE) Market Value; Source: CISCO report 2013

Prognosed excessiveIoT/IoS market value …

How to keep them secure?

How to keep the related critical infrastructuresof our public life secure?

53

Page 53: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Social Challenges

As intelligent agent-based and semantics-empoweredapplications are ever more invisibly permeating our everyday life:

How to balance our increasing dependencyon them with our human nature ofself-determination, and our social life ?

Joseph Weizenbaum (1923 – 2008)AI pioneer and critic

Privacy of human communication in the FI ?

Societal response to FIA inflictedjob losses and creation ?

Report 2014

In remembrance of:

54

Page 54: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Take-Home Messages

Intelligent agents and semantic technologies aregeneric key enablers for building FI applications.

Agents for intelligent human-agent interactionand distributed action coordination.

Semantic technologies for semantic interoperation,semantic analysis and explanationof data, streams, processes, and actions.

Combination of both can be quite powerfulin many FIA domains.

„Good-enough“ flexible, fast semantic reasoners andself-coordinating agents on resource-constrained devices.

Incorporation of privacy, trust, and security in practice.

55

Page 55: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Special Thanks

Intelligent Information Systems (I2S) Research Team@ DFKI Agents and Simulated Reality Department, Saarbrücken

Manuel Anglet Patrick KapahnkeXiaoqi Cao Christian MathieuJosenildo Costa Da Silva Luca MazzolaAndreas Frische Hanna MousaMaximilian Harz David WeissAnkush Meshram Ingo ZinnikusMatthias Klusch (head)

Visit us at www.dfki.de/~klusch/i2s

56

Page 56: Agents and Semantics for Future Internet Applications 2015_Keynote-Klusch.pdf · of data, streams, services, processes, actions • Semantic explanation to human user Semantic technologies

Thank you very much for your attention !

Questions ?

[email protected]

www.dfki.de/~klusch

57