data processing and semantics for advanced internet of things (iot) applications: modeling,...

Data Processing and Semantics for Advanced Internet of Things (IoT)

Applications:modeling, annotation, integration, and perception

Pramod Anantharam1, Payam Barnaghi2, Amit Sheth1

1Kno.e.sis Center, Wright State University2Centre for Communication Systems Research, University of Surrey

Madrid, Spain, June 12-14, 2013http://aida.ii.uam.es/wims13/keynotes.php#tutorials

Special Thanks to: Cory Henson, Kno.e.sis Research Center, Wright State UniversityWei Wang, Centre for Communication Systems Research, University of Surrey

http://aida.ii.uam.es/wims13/keynotes.php#tutorials

2

Part 1: Introduction to Internet of “Things”

Image source: CISCO

Internet of Things

“sensors and actuators embedded in physical objects — from containers to pacemakers — are linked through both wired and wireless networks to the Internet.”

“When objects in the IoT can sense the environment, interpret the data, and communicate with each other, they become tools for understanding complexity and for responding to events and irregularities swiftly”

source: http://www.iot2012.org/

“Thing” connected to the internet

“In 2013 cumulative shipments of Bluetooth-enabled devices will surpass 10 billion and Wi-Fi enabled devices will surpass 10 billion cumulative shipments in 2015,” - Peter Cooney, wireless analyst with ABI Research

“73% of Countries with 4G Services Have Dropped Their 4G Tariffs by an Average of 30% in the Past 6 Months”1 - ABI Research (07 Feb 2013)

1http://www.abiresearch.com/

http://www.abiresearch.com/

http://www.abiresearch.com/

5

Network connected Things and Devices

Image courtesy: CISCO

6

Sensor devices are becoming widely available

- Programmable devices- Off-the-shelf gadgets/tools

7

More “Things” are being connected

Home/daily-life devicesBusiness and Public infrastructureHealth-care…

8

People Connecting to Things

Motion sensorMotion sensor

Motion sensor

ECG sensor

Internet

9

Things Connecting to Things

- Complex and heterogeneous resources and networks

10

Wireless Sensor Networks (WSN)

Sinknode Gateway

Core networke.g. InternetCore networke.g. InternetGateway

End-userEnd-user

Computer servicesComputer services

- The networks typically run Low Power Devices- Consist of one or more sensors, could be different type of sensors (or actuators)- The networks typically run Low Power Devices- Consist of one or more sensors, could be different type of sensors (or actuators)

11

Key characteristics of IoT devices

Often inexpensive sensors (actuators) equipped with a radio transceiver for various applications, typically low data rate ~ 10-250 kbps.

Deployed in large numbers The sensors should coordinate to perform the desired task. The acquired information (periodic or event-based) is

reported back to the information processing centre (or sometimes in-network processing is required)

Solutions are application-dependent.

11

12

Beyond conventional sensors

Human as a sensor (citizen sensors) e.g. tweeting real world data and/or events

Virtual (software) sensors e.g. Software agents/services

generating/representing data

Road block, A3Road block, A3

Road block, A3Road block, A3

Suggest a different routeSuggest a different route

13

Cyber, Physical and Social Data

14

Citizen Sensors

Source: How Crisis Mapping Saved Lives in Haiti, Ushahidi Haiti Project (UHP).

15

Cosm- Air Quality Egg

16

Cosm- data readings

Tags

Data formats

Location

17

Making Sense of Data

In the next few years, sensor networks will produce 10-20 time the amount of data generated by social media. (source: GigaOmni Media)

18

Things, Data, and lots of it

image courtesy: Smarter Data - I.03_C by Gwen Vanhee

19

Big Data and IoT

"Big data" is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.” (wikipedia)

Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. (source IBM)

20

The seduction of data

Turn 12 terabytes of Tweets created each day into sentiment analysis related to different events/occurrences or relate them to products and services.

Convert (billions of) smart meter readings to better predict and balance power consumption.

Analyze thousands of traffic, pollution, weather, congestion, public transport and event sensory data to provide better traffic management.

Monitor patients, elderly care and much more…

Adapted from: What is Bog Data?, IBM

21

Do we need all these data?

22

“Raw data is both an oxymoron and bad data”

Geoff Bowker, 2005

Source: Kate Crawford, "Algorithmic Illusions: Hidden Biases of Big Data", Strata 2013.

23

IoT Data in the Cloud

Image courtesy: http://images.mathrubhumi.comhttp://www.anacostiaws.org/userfiles/image/Blog-Photos/river2.jpg

http://images.mathrubhumi.com/

24

Perceptions and Intelligence

Data

Information

Knowledge

Wisdom

Raw sensory data

Structured data (with semantics)

Abstraction and perceptions

Actionable intelligence

25

Change in communication paradigm

Sinknode Gateway

Core networke.g. InternetCore networke.g. Internet End-userEnd-user

DataData

SenderSender

DataData

ReceiverReceiver

A sample data communication in conventional networksA sample data communication in conventional networks

A sample data communication in WSNA sample data communication in WSN

Fire!Fire! Some bits01100011100Some bits01100011100

26

Collaboration and in-network processing In some applications a single sensor node is not able to handle

the given task or provide the requested information. Instead of sending the information form various source to an

external network/node, the information can be processed in the network itself. e.g. data aggregation, summarisation and then propagating the

processed data with reduced size (hence improving energy efficiency by reducing the amount of data to be transmitted).

Data-centric Conventional networks often focus on sending data between

two specific nodes each equipped with an address. Here what is important is data and the observations and

measurements not the node that provides it.

Required mechanisms

“People want answers, not numbers” (Steven Glaser, UC Berkley)

Sinknode Gateway

Core networke.g. InternetCore networke.g. Internet

What is the temperature at home?What is the temperature at home?Freezing!Freezing!

28

IoT Data alone is not enough

Domain knowledge Machine interpretable meta data Delivery, sharing and representation services Query, discovery, aggregation services Publish, subscribe, notification, and access

interfaces/services

29

Storing, Handling and Processing the Data

Image courtesy: IEEE Spectrum

30

IoT Data Challenges

Discovery: finding appropriate device and data sources Access: Availability and (open) access to IoT resources and

data Search: querying for data Integration: dealing with heterogeneous device, networks

and data Interpretation: translating data to knowledge usable by

people and applications Scalability: dealing with large number of devices and

myriad of data and computational complexity of interpreting the data.

Energy consumption of the nodes

Batteries have small capacity and recharging could be complex (if not impossible) in some cases.

The main consumers of the energy are: the controller, radio, to some extent memory and depending on the type, the sensor(s).

A controller can go to: “active”, “idle” and “sleep”

A radio modem could turn transmitter, receiver, or both on or off,

sensors and memory can be also turned on and off.

Actuators

Stepper Motor [1]

Image credits:[1] http://directory.ac/telco-motion.html[2] http://bruce.pennypacker.org/category/theater/[3] http://www.busytrade.com/products/1195641/TG-100-Linear-Actuator.html[4] http://www.arbworx.com/services/fencing-garden-fencing/

[2]

[3][4]

Wireless Sensor Networks (WSN)- gateway connection

SunSpots

Information channel

Control channel

Directory server

Gateway

Web user/application

Distributed WSN

What are the main issues?

Heterogeneity Interoperability Mobility Energy efficiency Scalability Security

What is important?

Robustness Quality of Service Scalability Seamless integration Security, privacy, Trust

In-network processing

Mobile Ad-hoc Networks are supposed to deliver bits from one end to the other

WSNs, on the other end, are expected to provide information, not necessarily original bits Gives addition options E.g., manipulate or process the data in the network

Main example: aggregation Applying aggregation functions to a obtain an average

value of measurement data Typical functions: minimum, maximum, average, sum,

… Not amenable functions: median

source: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .

In-network processing- example

Applying Symbolic Aggregate Approximation (SAX)

SAX Pattern (blue) with word length of 20 and a vocabulary of 10 symbolsover the original sensor time-series data (green)

Data-centric networking

In typical networks (including ad hoc networks), network transactions are addressed to the identities of specific nodes A “node-centric” or “address-centric” networking paradigm

In a redundantly deployed sensor networks, specific source of an event, alarm, etc. might not be important Redundancy: e.g., several nodes can observe the same area

Thus: focus networking transactions on the data directly instead of their senders and transmitters ! data-centric networking Principal design change

source: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .

Implementation options for data-centric networking Overlay networks & distributed hash tables (DHT)

Hash table: content-addressable memory Retrieve data from an unknown source, like in peer-to-peer networking – with

efficient implementation Some disparities remain

Static key in DHT, dynamic changes in WSN DHTs typically ignore issues like hop count or distance between nodes when

performing a lookup operation

Publish/subscribe Different interaction paradigm Nodes can publish data, can subscribe to any particular kind of data Once data of a certain type has been published, it is delivered to all

subscribes Subscription and publication are decoupled in time; subscriber and published

are agnostic of each other (decoupled in identity); There is concepts of Semantic Sensor Networks- to annotate sensor

resources and observation and measurement data!

Adapted from: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .

IoT and Semantic technologies

The sensors (and in general “Things”) are increasingly being integrated into the Internet/Web.

This can be supported by embedded devices that directly support IP and web-based connection (e.g. 6LowPAN and CoAp) or devices that are connected via gateway components. Broadening the IoT to the concept of “Web of Things”

There are already Sensor Web Enablement (SWE) standards developed by the Open Geospatial Consortium that are widely being adopted in industry, government and academia.

While such frameworks provide some interoperability, semantic technologies are increasingly seen as key enabler for integration of IoT data and broader Web information systems.

Semantics and IoT resources and data Semantics are machine-interpretable metadata (for mark-up),

logical inference mechanisms, query mechanism, linked data solutions

For IoT this means: ontologies for: resource (e.g. sensors), observation and

measurement data (e.g. sensor readings), domain concepts (e.g. unit of measurement, location), services (e.g. IoT services) and other data sources (e.g. those available on linked open data)

Semantic annotation should also supports data represented using existing forms

Reasoning /processing to infer relationships and hierarchies between different resources, data

Semantics (/ontologies) as meta-data (to describe the IoT resources/data) / knowledge bases (domain knowledge).

43

A Few Words on

Semantic Web

SSW Introduction

lives in

has petis ahas pet

Person

Person

Animal

Animal

Concrete Facts Resource Description Framework

Concrete Facts Resource Description Framework

Semantic Web(according to Farside)

General Knowledge Web Ontology Language General Knowledge Web Ontology Language

“Now! – That should clear up a few things around here!”

is a

Semantic Web Stack

Linked Open Data

Linked Open Data

~ 50 Billion Statements~ 50 Billion Statements

SW is moving from academia to industry

In the last few years, we have seen many successes …

Knowledge Graph

Watson

Apple Siri

Google Knowledge Graph

51

Sensors and the Web

Sensors are ubiquitous

Enabling the Internet of Things

Situational awareness enables:

Devices/things to function and adapt within their environment

Devices/things to work together

Sensor systems are too often stovepiped.

Closed centralized management of sensing resources

Closed inaccessible data and sensors

We want to set this data free

With freedom comes responsibilityDiscovery, access, and searchIntegration and interpretationScalability

Drowning in Data

A cross-country flight from New York to Los Angeles on a Boeing 737 plane generates a massive 240 terabytes of data

- GigaOmni Media

Drowning in Data

Challenges

To fulfill this vision, there are difficult challenges to overcome such as the discovery, access, search, integration, and interpretation of sensors and sensor data at scale

Discovery finding appropriate sensing resources and data sources

Access sensing resources and data are open and available

Search querying for sensor data

Integration dealing with heterogeneous sensors and sensor data

Interpretation translating sensor data to knowledge usable by people and

applications

Scalability dealing with data overload and computational complexity

of interpreting the data

Solution

Semantic Sensor WebInternet Computing, July/Aug. 2008

Uses the Web as platform for managing sensor resources and data

Uses semantic technologies for representing data and knowledge, integration, and interpretation

Solution

Discovery, access, and search Using standard Web services

OGC Sensor Web Enablement

Solution

Integration Using shared domain models / data

representation


W3C Semantic Sensor Networks

Solution

Interpretation Abstraction – converting low-level data to high-level

knowledge

Machine Perception – w/ prior knowledge and abductive reasoning

IntellegO – Ontology of Perception

Solution

Scalability Data overload – sensors produce too much data

Computational complexity of semantic interpretation

“Intelligence at the edge” – local and distributed integration and interpretation of sensor data

SSW Adoption and Applications

65

Part 2: Data and knowledge modelling requirements,

semantic annotation

Image source: CISCO

Recall of the Internet of Things

A primary goal of interconnecting devices and collecting/processing data from them is to create situation awareness and enable applications, machines, and human users to better understand their surrounding environments.

The understanding of a situation, or context, potentially enables services and applications to make intelligent decisions and to respond to the dynamics of their environments.

Barnaghi et al 2012, “Semantics for the Internet of Things: early progress and back to the future”

IoT challenges

Numbers of devices and different users and interactions required. Challenge: Scalability

Heterogeneity of enabling devices and platforms Challenge: Interoperability

Low power sensors, wireless transceivers, communication, and networking for M2M Challenge: Efficiency in communications

Huge volumes of data emerging from the physical world, M2M and new communications Challenge: Processing and mining the data, Providing secure access and

preserving and controlling privacy. Timeliness of data

Challenge: Freshness of the data and supporting temporal requirements in accessing the data

Ubiquity Challenge: addressing mobility, ad-hoc access and service continuity

Global access and discovery Challenge: Naming, Resolution and discovery

IoT: one paradigm, many visions

Diagram adapted from L. Atzori et al., 2010, “the Internet of Things: a Survey”

Semantic oriented vision

“The object unique addressing and the representation and storing of the exchanged information become the most challenging issues, bringing directly to a ‘‘Semantic oriented”, perspective of IoT”, [Atzori et al., 2010]

Data collected by different sensors and devices is usually multi-modal (temperature, light, sound, video, etc.) and diverse in nature (quality of data can vary with different devices through time and it is mostly location and time dependent [Barnaghi et al, 2012]

some of challenging issues: representation, storage, and search/discovery/query/addressing, and processing IoT resources and data.

What is expected?

Unified access to data: unified descriptions Deriving additional knowledge (data mining) Reasoning support and association to other entities

and resources Self-descriptive data an re-usable knowledge In general: Large-scale platforms to support discovery

and access to the resources, to enable autonomous interactions with the resources, to provide self-descriptive data and association mechanisms to reason the emerging data and to integrate it into the existing applications and services.

Semantic technologies and IoT

There are already Sensor Web Enablement (SWE) standards developed by the Open Geospatial Consortium that are widely adopted.

While such frameworks provide certain levels of interoperability, semantic technologies are seen as key enabler for integration of IoT data and and existing business information systems.

Semantic technologies provide potential support for: Interoperability and machine automation IoT resource and data annotation, logical inference, query

and discovery, linked IoT data

Identify IoT domain concepts

Users Physical entities Virtual entities Devices Resource Services …

Diagram adapted from IoT-A project D2.1

IoT domain concepts - Entity

Physical entities (or entity of interests): objects in the physical world, features of interest that are of interests to users (human users or any digital artifacts). Virtual entities: virtual representation of

the physical entities. Entities are the main focus of interactions

between humans and/or software agents. This interaction is made possible by a

hardware component called Device.Definition adapted from De et al, 2012, “Service modeling for the Internet of Things”

IoT domain concepts – Device, Resource and Service

A Device mediates the interactions between users and entities.

The software component that provides information on the entity or enables controlling of the device, is called a Resource.

A Service provides well-defined and standardised interfaces, offering all necessary functionalities for interacting with entities and related processes.

Definition adapted from De et al, 2012, “Service modeling for the Internet of Things”

Other concepts need to considered

Gateways Directories Platforms Systems Subsystems … Relationships among them And links to existing knowledge base

and linked data

Don’t forget the IoT data

Sensors and devices provide observation and measurement data about the physical world objects which also need to be semantically described and can be related to an event, situation in the physical world.

The processing of data into knowledge/ perception and using it for decision making, automated control, etc.

Huge amount of data from our physical world that need to be Annotated Published Stored (temporary or for longer term) Discovered Accessed Proceeded Utilised in different applications

Semantics for IoT resources and data Semantics are machine-interpretable metadata, logical

inference mechanisms, query and search mechanism, linked data…

For IoT this means: ontologies for: resource (e.g. sensors), observation and

measurement data (e.g. sensor readings), services (e.g. IoT services), domain concepts (e.g. unit of measurement, location) and other data sources (e.g. those available on linked open data)

Semantic annotation should also supports data represented using existing forms

Reasoning/processing to infer relationships between different resources and services, detecting patterns from IoT data

Characteristics of IoT resources

Extraordinarily large number Limited computing capabilities Limited memory Resource constrained environments

(e.g., battery life, signal coverage) Location is important Dynamism in the physical environments Unexpected disruption of services …

Characteristics of IoT data

Stream data (depends on time) Transient nature Almost always related to a phenomenon or

quality in our physical environments Large amount Quality in many situations cannot be

assured (e.g., accuracy and precision) Abstraction levels (e.g., raw, inferred or

derived) …

Utilise semantics

Find all available resources (which can provide data) and data related to “Room A” (which is an object in the linked data)? What is “Room A”? What is its location? returns “location” data What type of data is available for “Room A” or that “location”?

(sensor category types)

Predefined Rules can be applied based on available data (TempRoom_A > 80°C) AND (SmokeDetectedRoom_A position==TRUE)

FireEventRoom_A

Learning these rules needs data mining or pattern recognition techniques

81

Part 3: Semantics and data modelling for IoT

Image source: CISCO

Semantic modelling

Lightweight: experiences show that a lightweight ontology model that well balances expressiveness and inference complexity is more likely to be widely adopted and reused; also large number of IoT resources and huge amount of data need efficient processing

Compatibility: an ontology needs to be consistent with those well designed, existing ontologies to ensure compatibility wherever possible.

Modularity: modular approach to facilitate ontology evolution, extension and integration with external ontologies.

Existing models for resources and data

W3C Semantic Sensor Network Incubator Group’s SSN ontology (mainly for sensors and sensor networks, observation and measurement, and platforms and systems)

Quantity Kinds and Units Used together with the SSN ontology based on QUDV model OMG SysML(TM) Working group of the SysML 1.2 Revision Task

Force (RTF) and W3C Semantic Sensor Network Incubator Group

Existing models for services

OWL-S and WSMO are heavy weight models: practical use?

Minimal service model Deprecated Procedure-Oriented Service Model (POSM) and Resource-

Oriented Service Model (ROSM): two different models for different service technologies

Defines Operations and Messages No profile, no grounding

SAWSDL: mixture of XML, XML schema, RDF and OWL hRESTS and SA-REST: mixture of HTML and reference

to a semantic model; sensor services are not anticipated to have HTML

W3C’S SSN ontology

Diagram adapted from SSN report

Some existing IoT models and ontologies

FP7 IoT-A project’s Entity-Resource-Service ontology A set of ontologies for entities, resources, devices

and services Based on the SSN and OWL-S ontology

FP7 IoT.est project’s service description framework A modular approach for designing a description

framework A set of ontologies for IoT services, testing and

QoS/QoI Technology independent modelling for services

IoT-A resource model


IoT-A resource description


IoT-A service model


IoT-A service description


Service modelling in IoT.est

Diagrams adapted from Iot.est D3.1

IoT.est service profile highlight

ServiceType class represents the service technologies: RESTful and SOAP/WSDL services.

serviceQos and serviceQoI are defined as subproperty of serviceParameter; they link to concepts in the QoS/QoI ontology.

serviceArea: the area where the service is provided; different from the sensor observation area

Links to the IoT resources through “exposedBy” property

Future extension: serviceNetwork, servicePlatform and

serviceDeployment Service lifecycle, SLA…

93

Part 4: Linked-data and semantic enabled systems

Image source: CISCO

Linked data principles

using URI’s as names for things: Everything is addressed using unique URI’s.

using HTTP URI’s to enable people to look up those names: All the URI’s are accessible via HTTP interfaces.

provide useful RDF information related to URI’s that are looked up by machine or people;

including RDF statements that link to other URI’s to enable discovery of other related concepts of the Web of Data: The URI’s are linked to other URI’s.

Linked data in IoT

Using URI’s as names for things;- URI’s for naming IoT resources and data (and also streaming

data); Using HTTP URI’s to enable people to look up those names;

- Web-level access to low level sensor data and real world resource descriptions (gateway and middleware solutions);

Providing useful RDF information related to URI’s that are looked up by machine or people;- publishing semantically enriched resource and data descriptions

in the form of linked RDF data; Including RDF statements that link to other URI’s to enable discovery

of other related things of the web of data;- linking and associating the real world data to the existing data on

the Web;

Linked data layer for not only IoT…

Images from Stefan Decker, http://fi-ghent.fi-week.eu/files/2010/10/Linked-Data-scheme1.png; linked data diagram: http://richard.cyganiak.de/2007/10/lod/

Creating and using linked sensor data

http://ccsriottb3.ee.surrey.ac.uk:8080/IOTA/

Sensor discovery using linked sensor data

Semantics in IoT - reality

If we create an Ontology our data is interoperable Reality: there are/could be a number of ontologies for a domain

Ontology mapping Reference ontologies Standardisation efforts

Semantic data will make my data machine-understandable and my system will be intelligent. Reality: it is still meta-data, machines don’t understand it but can interpret it.

It still does need intelligent processing, reasoning mechanism to process and interpret the data.

It’s a Hype! Ontologies and semantic data are too much overhead; we deal with tiny devices in IoT. Reality: Ontologies are a way to share and agree on a common vocabulary and

knowledge; at the same time there are machine-interpretable and represented in interoperable and re-usable forms;

You don’t necessarily need to add semantic metadata in the source- it could be added to the data at a later stage (e.g. in a gateway);

100

Part 5: Semantic Sensor Web and

Perception

Image source: semanticweb.com; CISCO

What is the Sensor Web?

Sensor Web is an additional layer connecting sensor networks to the World Wide Web.

Enables an interoperable usage of sensor resources by enabling web based discovery, access, tasking, and alerting.

Enables the advancement of

cyber-physical applications through

improved situation awareness.

Why is the Sensor Web important?

In general Enable tight coupling of the cyber and

physical world

In relation to IoT Enable shared situation awareness (or

context) between devices/things

Bridging the Cyber-Physical Divide

Psyleron’s Mind-Lamp (Princeton U), connections between the mind and the physical world.

Neuro Sky's mind-controlled headset to play a video game.

MIT’s Fluid Interface Group: wearable device with a projector for deep interactions with the environment


Foursquare is an online application which integrates a persons physical location and social network.

Community of enthusiasts that share experiences of self-tracking and measurement.

FitBit Community allows the automated collection and sharing of health-related data, goals, and achievements


Tweeting Sensorssensors are becoming social

How do we design the Sensor Web?

Integration through shared semantics OGC Sensor Web Enablement W3C SSN ontology and Semantic Annotation

Interpretation through integration of heterogeneous data and reasoning with prior knowledge

Semantic Perception/Abstraction Linked Open Data as prior knowledge

Scale through distributed local interpretation

“intelligence at the edge”

Role of OGC SWE

Vision of Sensor Web

Quickly discover sensors (secure or public) that can meet my needs – location, observables, quality, ability to task

Obtain sensor information in a standard encoding that is understandable by me and my software

Readily access sensor observations in a common manner, and in a form specific to my needs

Task sensors, when possible, to meet my specific needs

Subscribe to and receive alerts when a sensor measures a particular phenomenon

Principles of Sensor Web

Sensors will be web accessible

Sensors and sensor data will be discoverable

Sensors will be self-describing to humans and

software (using a standard encoding)

Most sensor observations will be easily accessible

in real time over the web

OGC SWE Services

Sensor Observation Service (SOS) access sensor information (SensorML) and sensor

observations (O&M

Sensor Planning Service (SPS) task sensors or sensor systems

Sensor Alert Service (SAS) asynchronous notification of sensor events (tasks,

observation of phenomena)

Sensor Registries discovery of sensors and sensor data

OGC SWE Services

OGC SWE Languages

Sensor Model Language (SensorML)

Models and schema for describing sensor

characteristics

Observation & Measurement (O&M)

Models and schema for encoding sensor

observations

OCG SWE Observation

Semantic Sensor Web

RDF OWL


Sensor Web + Semantic Web

Semantic Web

The web of data where web content is processed by machines, with human actors at the end of the chain.

The web as a huge, dynamic, evolving database of facts, rather than pages, that can be interpreted and presented in many ways (mashups).

Fundamental importance of ontologies to describe the fact that represents the data. RDF(S) emphasises labelled links as the source of meaning: essentially a graph model . A label (URI) uniquely identifies a concept.

OWL emphasises inference as the source of meaning: a label also refers to a package of logical axioms with a proof theory.

Usually, the two notions of meaning fit.

Goal to combine information and services for targeted purpose and new knowledge

Sensor Web

The internet of things made up of Wireless Sensor Networks, RFID, stream gauges, orbiting satellites, weather stations, GPS, traffic sensors, ocean buoys, animal and fish tags, cameras, habitat monitors, recording data from the physical world.

Today there are 4 billion mobile sensing devices plus even more fixed sensors. The US National Research Council predicts that this may grow to trillions by 2020, and they are increasingly connected by internet and Web protocols.

Record observations of a wide variety of modalities: but a big part is time-series‟ of numeric measurements.

The Open Geospatial Consortium has some web-service standards for shared data access (Sensor Web Enablement).

Goal is to open up access to real-time and archival data, and to combine in applications.

So, what is a Semantic Sensor Web?

Reduce the difficulty and open up sensor networks by:

Allowing high-level specification of the data collection process;

Across separately deployed sensor networks; Across heterogeneous sensor types; and Across heterogeneous sensor network platforms; Using high-level descriptions of sensor network

capability; and Interfacing to data integration methods using similar

query and capability descriptions.

To create a Web of Real Time Meaning!

W3C SSN Incubator Group

SSN-XG commenced: 1 March 2009

Chairs: Amit Sheth, Kno.e.sis Center, Wright State University Kerry Taylor, CSIRO Amit Parashar Holger Neuhaus Laurent Lefort, CSIRO

Participants: 39 people from 20 organizations, including: Universities in: US, Germany, Finland, Spain, Britain,

Ireland Multinationals: Boeing, Ericsson Small companies in semantics, communications,

software Research institutes: DERI (Ireland), Fraunhofer

(Germany), ETRI (Korea), MBARI (US), SRI International (US), MITRE (US), US Defense, CTIC (Spain), CSIRO (Australia), CESI (China)

W3C SSN Incubator Group

Two main objectives:

The development of an ontology for describing sensing resources and data, andThe extension of the SWE languages to support semantic annotations.

Sensor Standards Landscape

SSN Ontology

OWL 2 DL ontology

Authored by the XG participants

Edited by Michael Compton

Driven by Use Cases

Terminology carefully tracked to sources through annotation properties

Metrics Classes: 117 Properties: 148 DL Expressivity:

SIQ(D)

SSN Ontology – http://purl.oclc.org/NET/ssnx/ssn

SSN Use Cases

SSN Ontology

Stimulus-Sensor-Observation

The SSO Ontology Design Pattern is developed following the principle of minimal ontological commitments to make it reusable for a variety of application areas.

Introduces a minimal set of classes and relations centered around the notions of stimuli, sensor, and observations. Defines stimuli as the (only) link to the physical environment.

Empirical science observes these stimuli using sensors to infer information about environmental properties and construct features of interest.

SSN Ontology Modules

SSN Sensor

A sensor can do (implements) sensing: that is, a sensor is any entity that can follow a sensing method and thus observe some Property of a FeatureOfInterest.

Sensors may be physical devices, computational methods, a laboratory setup with a person following a method, or any other thing that can follow a Sensing Method to observe a Property.

SSN Measurement Capability

Collects together measurement properties (accuracy, range, precision, etc) and the environmental conditions in which those properties hold, representing a specification of a sensor's capability in those conditions.

SSN Observation

An Observation is a Situation in which a Sensing method has been used to estimate or calculate a value of a Property.

Links to Sensing and Sensor describe what made the Observation and how; links to Property and Feature detail what was sensed; the result is the output of a Sensor; other metadata gives the time(s) and the quality.

Different from OGC’s O&M, in which an “observation” is an act or event, although it also provides the record of the event.

Alignment with DOLCE

What SSN does not model

Sensor types and models

Networks: communication, topology

Representation of data and units of measurement

Location, mobility or other dynamic behaviours

Animate sensors

Control and actuation

….

Semantic Annotation of SWE

Recommended technique via Xlink attributes requires no change to SWE

xlink:href - link to ontology individual

xlink:role - link to ontology class

xlink:arcrole - link to ontology object property

136

Interpretation of data

A primary goal of interconnecting devices and collecting/processing data from them is to create situation awareness and enable applications, machines, and human users to better understand their surrounding environments.

The understanding of a situation, or context, potentially enables services and applications to make intelligent decisions and to respond to the dynamics of their environments.

137

Observation and measurement data

Source: W3C Semantic Sensor Networks, SSN Ontology presentation, Laurent Lefort et al.

138

How to say what a sensor is and what it measures?

Sinknode

Gateway

139

Data/Service description frameworks

There are standards such as Sensor Web Enablement (SWE) set developed by the Open Geospatial Consortium that are widely being adopted in industry, government and academia.

While such frameworks provide some interoperability, semantic technologies are increasingly seen as key enabler for integration of IoT data and broader Web information systems.

140

Sensor Markup Language (SensorML)

Source: http://www.mitre.org/

141

W3C SSN Ontology

makes observations of this type

Where it is

What it measures

units

SSN-XG ontologies

SSN-XG annotations

SSN-XG Ontology Scope

142

Semantics and IoT data

Creating ontologies and defining data models is not enough tools to create and annotate data data handling components

Complex models and ontologies look good, but design lightweight versions for constrained environments think of practical issues make it as compatible as possible and/or link it to the other

existing ontologies Domain knowledge and instances

Common terms and vocabularies Location, unit of measurement, type, theme, …

Link it to other resources Linked-data URIs and naming

143

Semantics and sensor data

Source: W. Wang, P. Barnaghi, "Semantic Annotation and Reasoning for Sensor Data", In proceedings of the 4th European Conference on Smart Sensing and Context (EuroSSC2009), 2009.

144

Semantics and Linked-data

The principles in designing the linked data are defined as: using URI’s as names for things; using HTTP URI’s to enable people to look up

those names; provide useful RDF information related to

URI’s that are looked up by machine or people;

including RDF statements that link to other URI’s to enable discovery of other related concepts of the Web of Data;

145

Linked Sensor data

146

Myth and reality

#1: If we create an Ontology our data is interoperable Reality: there are/could be a number of ontologies for a domain

Ontology mapping Reference ontologies Standardisation efforts

#2: Semantic data will make my data machine-understandable and my system will be intelligent. Reality: it is still meta-data, machines don’t understand it but can interpret it. It still

does need intelligent processing, reasoning mechanism to process and interpret the data.

#3: It’s a Hype! Ontologies and semantic data are too much overhead; we deal with tiny devices in IoT. Reality: Ontologies are a way to share and agree on a common vocabulary and

knowledge; at the same time there are machine-interpretable and represented in interoperable and re-usable forms;

You don’t necessarily need to add semantic metadata in the source- it could be added to the data at a later stage (e.g. in a gateway);

Legacy applications can ignore it or to be extended to work with it.

147

Processing Streaming Sensor Data

148148

Symbolic Aggregate Approximation (SAX)

Variable String Length and Vocabulary size.

Length: 10, VocSize: 10 Length: 10, VocSize: 4

“gijigdbabd” “cdddcbaaab”

Green Curve: consists of 100 Samples, Blue Curve: SAX

149

SAX representation

SAX Pattern (blue) with word length of 20 and a vocabulary of 10 symbolsover the original sensor time-series data (green)

P. Barnaghi, F. Ganz, C. Henson, A. Sheth, "Computing Perception from Sensor Data", in Proc. of the IEEE Sensors 2012, Oct. 2012.

fggfffhfffffgjhghfff

jfhiggfffhfffffgjhgi

fggfffhfffffgjhghfff

150

Data Processing Framework

fggfffhfffffgjhghfff dddfffffffffffddd cccddddccccdddccc aaaacccaaaaaaaaccccdddcdcdcdcddasddd

PIR Sensor Light Sensor Temperature Sensor

Raw sensor data stream



Attendance

PhoneHot Temperature

Cold Temperature

Bright

Day-time

Night-time

Office room BA0121

On going meeting

Window has been left open

….

Temporal data(extracted from SSN descriptions)

Spatial data(extracted from SSN descriptions)

Thematic data(low level abstractions)

Parsimonious Covering Theory

Observations

Perceptions

Domain knowledge

SAX Patterns

Raw Sensor Data(Annotated with SSN Ontology)

…

….

Perception Computation

High-level Perceptions

151

SensorSAX

F. Ganz, P. Barnaghi, F. Carrez, “Information Abstraction for Heterogeneous Real World Internet Data”, Feb. 2013.

152

Evaluation results of abstraction creation


153

Data size reduction


154

Enabling the Internet of Things

- Diversity range of applications- Interacting with large number of devices with various types-Multiple heterogeneous networks-Deluge of data-Processing and interpretation of the IoT data

155

Challenges and opportunities

Providing infrastructure Publishing, sharing, and access solutions on a global scale Indexing and discovery (data and resources) Aggregation and fusion Trust, privacy and security Data mining and creating actionable knowledge

Integration into services and applications in e-health, the public sector, retail, manufacturing and personalized apps. Mobile apps, location-based services, monitoring control etc.

New business models

156

Part 6: Cognitive aspects of knowledge representation, reasoning and perception

Image source: CISCO

Abstraction provides the ability to interpret and synthesize information in a way that affords effective understanding and communication of ideas, feelings, perceptions, etc. between machines and people.

Abstraction

People are excellent at abstraction; of sensing and interpreting stimuli to understand and interact with the world.

The process of interpreting stimuli is called perception; and studying this extraordinary human capability can lead to insights for developing effective machine perception.

Abstraction

observe perceive

conceptualizationof “real-world”

“real-world”

Abstraction

Semantic Perception/Abstraction

Fundamental Questions

What is perception, and how can we design machines to perceive?

What can we learn from cognitive models of perception?

Is the Semantic Web up to the task of modeling perception?

What is Perception?

Perception is the act of

Abstracting

Explaining

Discriminating

Choosing

What can we learn from Cognitive Models of Perception?

A-priori background knowledge is a key enabler

Perception is a cyclical, active process

Ulric Neisser (1976)Ulric Neisser (1976) Richard Gregory (1997)Richard Gregory (1997)

Is Semantic Web up to the task of modeling perception?

RepresentationHeterogeneous sensors, sensing, and observation recordsBackground knowledge (observable properties, objects/events, etc.)

InferenceExplain observations (hypothesis building)Focus attention by seeking additional stimuli (that discriminate between explanations)

Difficult Issues to OvercomePerception is an inference to the best explanationHandle streaming dataReal-time processing (or nearly)

Both people and machines are capable of observing qualities, such as redness.

* Formally described in a sensor/ontology (SSN ontology)

observesObserver

Quality

The ability to perceive is afforded through the use of background knowledge, relating observable qualities to entities in the world.

* Formally described in domain ontologies

(and knowledge bases)

inheres in

Quality

Entity

With the help of sophisticated inference, both people and machines are also capable of perceiving entities, such as apples.

the ability to degrade gracefully with incomplete

information

the ability to minimize explanations based on new

information

the ability to reason over data on the Web

fast (tractable)

perceivesEntity

Perceiver

Perceptual Inference

minimizeexplanations

degrade gracefully

tractable

Abductive Logic (e.g., PCT)

high complexity

Deductive Logic (e.g., OWL)

(relatively) low complexity

Web reasoning

Perceptual Inference(i.e., abstraction)

The ability to perceive efficiently is afforded through the cyclical exchange of information between observers and perceivers.

Traditionally called the Perceptual Cycle

(or Active Perception)

sendsfocus

sends observation

Observer

Perceiver

Neisser’s Perceptual Cycle

1970’s – Perception is an active, cyclical process of exploration and interpretation.

- Nessier’s Perception Cycle

1980’s – The perception cycle is driven by background knowledge in order to generate and test hypotheses.

- Richard Gregory (optical illusions)

1990’s – In order to effectively test hypotheses, some observations are more informative than others.

- Norwich’s Entropy Theory of Perception

Cognitive Theories of Perception

Key InsightsBackground knowledge plays a crucial role in perception; what we know (or think we know/believe) influences our perception of the world.Semantics will allow us to realize computational models of perception based on background knowledge.

Contemporary IssuesInternet/Web expands our background knowledge to a global scope; thus our perception is global in scopeSocial networks influence our knowledge and beliefs, thus influencing our perception

observes

inheres in

Integrated together, we have an general model – capable of abstraction – relating observers, perceivers, and background knowledge.

perceives

sendsfocus

sends observation

Observer

Quality

EntityPerceive

r

Ontology of Perception – as an extension of SSN

Provides abstraction of sensor data through perceptual inference of semantically annotated data

Prior Knowledge

W3C SSN Ontology Bi-partite Graph

Prior knowledge conformant to SSN ontology (left), structured as a bipartite graph (right)

Explanation is the act of accounting for sensory observations (i.e., abstraction); often referred to as hypothesis building.

Observed Property: A property that has been observed.

ObservedProperty ≡ ∃ssn:observedProperty—.{o1} ⊔ … ⊔ ∃ssn:observedProperty—.{on} Explanatory Feature: A feature that explains the set of observed properties.

ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{p1} ⊓ … ⊓ ∃ssn:isPropertyOf—.{pn}

Semantics of Explanation

ExampleAssume the properties elevated blood pressure and palpitations have been observed, and encoded in RDF (conformant with SSN): ssn:Observation(o1), ssn:observedProperty(o1, elevated blood pressure)ssn:Observation(o2), ssn:observedProperty(o2, palpitations) Given these observations, the following ExplanatoryFeature class is constructed:

ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{elevated blood pressure} ⊓ ∃ssn:isPropertyOf—.{palpitations}

Given the KB, executing the query ExplanatoryFeature(?y) can infer the features, Hypertension and Hyperthyroidism, as explanations:

ExplanatoryFeature(Hypertension) ExplanatoryFeature(Hyperthyroidism)

Semantics of Explanation

Discrimination is the act of deciding how to narrow down the multitude of explanatory features through further observation.

Expected Property: A property is expected with respect to (w.r.t.) a set of features if it is a property of every feature in the set.

ExpectedProperty ≡ ∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ∃ssn:isPropertyOf.{fn}

NotApplicable Property: A property is not-applicable w.r.t. a set of features if it is not a property of any feature in the set.

NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ¬∃ssn:isPropertyOf.{fn}

Discriminating Property: A property is discriminating w.r.t. a set of features if it is neither expected nor not-applicable.

DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicableProperty

Semantics of Discrimination

Example Given the explanatory features from the previous example, Hypertension and Hyperthyroidism, the following classes are constructed:

ExpectedProperty ≡ ∃ssn:isPropertyOf.{Hypertension} ⊓ ∃ssn:isPropertyOf.

{Hyperthyroidism} NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{Hypertension} ⊓ ¬∃ssn:isPropertyOf.{Hyperthyroidism} Given the KB, executing the query DiscriminatingProperty(?x) can infer the property clammy skin as discriminating: DiscriminatingProperty(clammy skin)

Semantics of Discrimination

Efficient Algorithms for IntellegO

Use of OWL-DL reasoner too resource-intensive for use in resource constrained devices (such as sensor nodes, mobile phones, IoT devices)

Runs out of resources for problem size (prior knowledge) > 20 concepts

Asymptotic complexity: O(n3) [Experimentally determined]

To enable their use on resource-constrained devices, we now describe algorithms for efficient inference of explanation and discrimination.

These algorithms use bit vector encodings and operations, leveraging a-priori knowledge of the environment.


Semantic (RDF) Encoding

Bit Vector Encoding

Lower

Lift

First, developed lifting and lowering algorithms to translate between RDF and bit vector encodings of observations.


Explanation Algorithm

Discrimination Algorithm

Utilize bit vector operators to efficiently compute explanation and discrimination

Explanation: Use of the bit vector AND operation to discover and dismiss those features that cannot explain the set of observed properties

Discrimination: Use of the bit vector AND operation to discover and indirectly assemble those properties that discriminate between a set of explanatory features. The discriminating properties are those that are determined to be neither expected nor not-applicable


Evaluation: The bit vector encodings and algorithms yield significant and necessary computational enhancements – including asymptotic order of magnitude improvement, with running times reduced from minutes to milliseconds, and problem size increased from 10’s to 1000’s.

Adoption of SSN

185

Part 7: Physical-Cyber-Social Computing

Image source: CISCO

186

J. McCarth

y

J. McCarth

y

M. Weiser

M. Weiser

D. Engelba

rt

D. Engelba

rt

J. C. R. LickliderJ. C. R.

Licklider

Similar Visions of Computing

187

If people do not believe that mathematics is simple, it is only because they do not realize how

complicated life is.

- John von Neumann

If people do not believe that mathematics is simple, it is only because they do not realize how

complicated life is.

- John von NeumannComputational paradigms have always

dealt with a simplified representations of the real-world…

Computational paradigms have always dealt with a simplified representations of

the real-world…

Algorithms work on these simplified representations

Algorithms work on these simplified representations

Solutions from these algorithms are transcended back to the real-world by

humans as actions

Solutions from these algorithms are transcended back to the real-world by

humans as actions

http://ngs.ics.uci.edu/blog/?p=1501

Grand challenges in the real-world are complex

What has changed now?

Data Processing and Semantics for Advanced Internet of Things (IoT) Applications, WIMS 2013, Madrid, Spain

There are more devices connected to the internet thanthe entire human population.There are more devices connected to the internet thanthe entire human population.

Today we have around 10 billion devices connected to the internet making it an era of IoT (Internet of Things)Today we have around 10 billion devices connected to the internet making it an era of IoT (Internet of Things)

http://www.cisco.com/web/about/ac79/docs/innov/IoT_IBSG_0411FINAL.pdf

189

What has not changed?

What has not changed?

We need computational paradigms to tap into the rich pulse of the

human populace

We need computational paradigms to tap into the rich pulse of the

human populace

We are still working on the simpler representations of the real-world!

We are still working on the simpler representations of the real-world!

Represent, capture, and compute with richer and fine-grained representations of

real-world problems

Represent, capture, and compute with richer and fine-grained representations of

real-world problems

What should change?What should change?

PCS Computing: Asthma Scenario

190

Sensordrone – for monitoring environmental air quality

Wheezometer – for monitoringwheezing sounds

Can I reduce my asthma attacks at night?

What are the triggers?What is the wheezing level?

What is the propensity toward asthma?

What is the exposure level over a day?

What is the air quality indoors?

Commute to Work

Personal

Public Health

Population Level

Closing the window at homein the morning and taking analternate route to office may

lead to reduced asthma attacks

Actionable Informatio

n

Actionable Informatio

n

Personal, Public Health, and Population Level Signals for Monitoring Asthma


ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist ; *consider referral to specialist

Asthma Control and Actionable Information

Sensors and their observations for understanding asthma

Asthma Early Warning Model


192

Personal Level

Signals

Personal Level

Signals

Societal Level Signals

Societal Level Signals

(Personal Level Signals)

(Personalized Societal Level Signal)

(Societal Level Signals)

Societal Level Signals Relevant to the Personal

Level

Societal Level Signals Relevant to the Personal

Level

Personal Level Sensors

(mHealth**)

QualifyQualify QuantifyQuantify

Action Recommendati

on

Action Recommendati

on

What are the features influencing my asthma?What is the contribution of each of these features?

How controlled is my asthma? (risk score)What will be my action plan to manage asthma?

StorageStorage

Societal Level Sensors

Asthma Early Warning Model (AEWM)

Query AEWM

Verify & augmentdomain knowledge

Recommended Action

Action Justification

*http://www.slideshare.net/jain49/eventshop-120721, ** http://www.youtube.com/watch?v=btnRi64hJp4

(EventShop*)

Health Signal Extraction to Understanding


Population Level

Personal

Wheeze – YesDo you have tightness of chest? –Yes

Observations Physical-Cyber-Social System Health Signal Extraction Health Signal Understanding

<Wheezing=Yes, time, location>

<ChectTightness=Yes, time, location>

<PollenLevel=Medium, time, location>

<Pollution=Yes, time, location>

<Activity=High, time, location>

Wheezing

ChectTightness

PollenLevel

Pollution

Activity

Wheezing

ChectTightness

PollenLevel

Pollution

Activity

RiskCategory

<PollenLevel, ChectTightness, Pollution,Activity, Wheezing, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory>

.

.

.

Expert Knowledge

Background Knowledge

tweet reporting pollution level and asthma attacks

Acceleration readings fromon-phone sensors

Sensor and personal observations

Signals from personal, personal spaces, and community spaces

Risk Category assigned by doctors

Qualify

Quantify

Enrich

Outdoor pollen and pollution

Public Health

Well Controlled - continueNot Well Controlled – contact nursePoor Controlled – contact doctor

194

J. McCarth

y

J. McCarth

y

M. Weiser

M. Weiser

D. Engelba

rt

D. Engelba

rt

J. C. R. LickliderJ. C. R.

Licklider

195

Part 8: IoT/PCS Systems: Applications

Image source: CISCO

SSN Applications

Applications of SSN

HealthcareWeather Rescue

Traffic Fire Fighting Logistics

SSN Application: Weather

50% savings in sensing resource requirements during the detection of a blizzard

Order of magnitude resource savings between storing observations vs. relevant abstractions

Linked Sensor Data

Linked Sensor Data(~2 Billion Statements)

Sensor Discovery Application

Query w/ location name to find nearby sensors

Real-Time Feature Streams

Demo: http://www.youtube.com/watch?v=_ews4w_eCpg


SSN Application: Fire Detection

Weather Application

SECURE: Semantics-empowered Rescue Environment(detect different types of fires)

DEMO: http://www.youtube.com/watch?v=in2KMkD_uqg

SSN Application: Health Care

MOBILEMD: Mobile app to help reduce re-admission of patients with Chronic Heart Failure


Passive Monitoring Phase Passive Monitoring Phase

• Abnormal heart rate• Clammy skin

• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic Shock

Observed Symptoms Possible Explanations

Passive Sensors – heart rate, galvanic skin response


Active Monitoring Phase Active Monitoring Phase

Are you feeling lightheaded?Are you feeling lightheaded?

Are you have trouble taking deep breaths?

Are you have trouble taking deep breaths?

yesyes

yesyes

Have you taken your Methimazole medication?

Have you taken your Methimazole medication?

Do you have low blood pressure?Do you have low blood pressure?

yesyes

• Abnormal heart rate• Clammy skin• Lightheaded• Trouble breathing• Low blood pressure

• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic Shock

Observed Symptoms Possible Explanations

nono

Active Sensors – blood pressure, weight scale, pulse oxymeter Demo: http://www.youtube.com/watch?v=btnRi64hJp4

Domain ExpertsDomain Experts

ColdWeatherColdWeather

PoorVisibilityPoorVisibility

SlowTrafficSlowTraffic

IcyRoadIcyRoad

Declarative domain knowledge

Causal knowledge

Linked Open Data

ColdWeather(YES/NO)IcyRoad (ON/OFF) PoorVisibility (YES/NO)SlowTraffic (YES/NO)

1 0 1 1 1 1 1 0 1 1 1 1 1 0 1 0

Domain ObservationsDomain Observations

Domain KnowledgeDomain Knowledge

Structure and parameters

205

WinterSeasonWinterSeason

Otherknowledge

Correlations to causations using Declarative knowledge on the Semantic Web

kTraffic

Traffic jam

Traffic jam

Link Descriptio

n

Link Descriptio

nScheduled

EventScheduled

Event

traffic jamtraffic jam

baseball gamebaseball game

Add missing random variables

Time of day

Time of day

bad weather CapableOf slow traffic

bad weatherbad weather

Traffic data from sensors deployed on road network in San Francisco

Bay Area

time of daytime of day


baseball gamebaseball gametime of daytime of day

slow trafficslow traffic

Three Operations: Complementing graphical model structure extraction

Add missing links bad weatherbad weather




Add link directionbad weatherbad weather




go to baseball game Causes traffic jam

Knowledge from ConceptNet5

traffic jam CapableOfoccur twice each daytraffic jam CapableOf slow traffic

206

207

Scheduled EventScheduled Event

Active EventActive Event

Day of weekDay of week Time of

dayTime of day

delaydelay

Travel timeTravel time

speedspeed

volumevolume

Structure extracted formtraffic observations (sensors + textual) using statistical techniques

Scheduled EventScheduled Event

Active EventActive Event

Day of weekDay of week

Time of dayTime of day

delaydelayTravel

timeTravel time

speedspeed

volumevolume

Bad WeatherBad Weather

Enriched structure which has link directions and new nodes such as “Bad Weather” potentially leading to better delay predictions

kTraffic

SemMOB


First responders have limited time to analyzesensor (on and around them) observations. First responders have limited time to analyzesensor (on and around them) observations.

Fire fighters need to combine prior knowledge of fires and its behavior, extinguishers, and floorplan of the building for rescue strategy and operations.

Fire fighters need to combine prior knowledge of fires and its behavior, extinguishers, and floorplan of the building for rescue strategy and operations.

There are a variety of sensors used to monitorvitals of firefighters, location, and poisonous gases.There are a variety of sensors used to monitorvitals of firefighters, location, and poisonous gases.

O2O2

Heart rateHeart rate

COCO

CO2CO2

GPSGPS

AccelerometerAccelerometer CompassCompass

FootstepsFootsteps

There is a team of them makingit further difficult for analysisThere is a team of them makingit further difficult for analysis

Semantic Web allow us to describe the domain, sensors, andfirst responders -- apply reasoning techniques to derive actionableinsights

Semantic Web allow us to describe the domain, sensors, andfirst responders -- apply reasoning techniques to derive actionableinsights

Sensors on each responder provides different view of the event and they needto register dynamically as the firefightersarrive.

Sensors on each responder provides different view of the event and they needto register dynamically as the firefightersarrive.

SemMOB


PlatformPlatform SensingDevice

SensingDevice

System_AGTC1System_AGTC1

Sensor_LSM11Sensor_LSM11

SensorOutput

SensorOutput

O2Concentraction_outputO2Concentraction_output

MeasurementCapabili

ty

MeasurementCapabili

ty

MeasurementCapability_LSM11

MeasurementCapability_LSM11

Property

Property

concentrationconcentration

onPlatform

hasOutput

hasMeasurementCapability

Observes

Observation

Observation

O2ConcentrationO2Concentration

ObservedProperty

O2Obs_15052012O2Obs_15052012

observationResultTimehttp://geonames.org/6298640http://geonames.org/6298640

hasLocation

name

xsd:floatxsd:float

latlong

Dayton, Dayton-Wright Brothers Airport^^xsd:string

FeatureFeature

Demo: http://www.youtube.com/watch?v=nX11OqgtQlw

LOKILAS (LOst Key Identification, Localization and Alert System)


LOKILAS


Future work

Creating ontologies and defining data models are not enough tools to create and annotate data Tools for publishing linked IoT data

Designing lightweight versions for constrained environments think of practical issues make it as much as possible compatible and/or link it to

the other existing ontologies Linking to domain knowledge and other resources

Location, unit of measurement, type, theme, … Linked-data URIs and naming

Some of the open issues

Efficient real-time IoT resource/service query/discovery Directory Indexing

Abstraction of IoT data Pattern extraction Perception creation

IoT service composition and compensation Integration with existing Web services Service adaptation

Selected references

Payam Barnaghi, Wei Wang, Cory Henson, Kerry Taylor, "Semantics for the Internet of Things: early progress and back to the future", (to appear) International Journal on Semantic Web and Information Systems (special issue on sensor networks, Internet of Things and smart devices), 2012.

Atzori, L., Iera, A. & Morabito, G. , “The Internet of Things: A survey”, Computer Networks, Volume 54, Issue 15, 28 October 2010, 2787-2805.

Suparna De, Tarek Elsaleh, Payam Barnaghi , Stefan Meissner, "An Internet of Things Platform for Real-World and Digital Objects", Journal of Scalable Computing: Practice and Experience, vol 13, no.1, 2012.

Suparna De, Payam Barnaghi, Martin Bauer, Stefan Meissner, "Service modelling for the Internet of Things", in Proceedings of the Conference on Computer Science and Information Systems (FedCSIS), pp.949-955, Sept. 2011.

Cory Henson, Amit Sheth, and Krishnaprasad Thirunarayan, “Semantic Perception: Converting Sensory Observations to Abstractions”, IEEE Internet Computing, Special Issue on Context-Aware Computing, March/April 2012.

Payam Barnaghi, Frieder Ganz, Cory Henson, Amit Sheth, “Computing Perception from Sensor Data”, In proceedings of the 2012 IEEE Sensors Conference, Taipei, Taiwan, October 28-31, 2012.

Michael Compton et al, “The SSN Ontology of the W3C Semantic Sensor Network Incubator Group”, Journal of Web Semantics, 2012.

Harshal Patni, Cory Henson, and Amit Sheth , “Linked Sensor Data”, in Proceedings of 2010 International Symposium on Collaborative Technologies and Systems (CTS 2010), Chicago, IL, May 17-21, 2010.

Amit Sheth, Cory Henson, and Satya Sahoo , “Semantic Sensor Web IEEE Internet Computing”, vol. 12, no. 4, July/August 2008, pp. 78-83.

Wei Wang, Payam Barnaghi, Gilbert Cassar, Frieder Ganz, Pirabakaran Navaratnam, "Semantic Sensor Service Networks", (to appear) in Proceedings of the IEEE Sensors 2012 Conference, Taipei, Taiwan, October 2012.

Wang W, De S, Toenjes R, Reetz E, Moessner K, "A Comprehensive Ontology for Knowledge Representation in the Internet of Things", International Workshop on Knowledge Acquisition and Management in the Internet of Things (KAMIoT 2012) in conjunction with IEE IUCC-2012, Liverpool, UK. Liverpool. 25-27 June, 2012.

Some useful links related to IoT

Internet of Things, ITU

http://www.itu.int/osg/spu/publications/internetofthings/InternetofThings_summary.pdf

IoT Comic Book

http://www.theinternetofthings.eu/content/mirko-presser-iot-comic-book

Internet of Things Europe, http://www.internet-of-things.eu/

Internet of Things Architecture (IOT-A)

http://www.iot-a.eu/public/public-documents

W3C Semantic Sensor Networks

http://www.w3.org/2005/Incubator/ssn/XGR-ssn-20110628/

data processing and semantics for advanced internet of things (iot) applications: modeling,...

Technology

iot data

data processing

smarter data

social data

ofthe data

raw data

iot big data

big data sizes