smap 2.0: integrating and managing physical data stephen dawson-haggerty computer science division,...

sMAP 2.0: Integrating and Managing Physical Datahttp://code.google.com/p/smap-data

Stephen Dawson-Haggerty

Computer Science Division, University of California, [email protected]

sMAP Redesign

• Design iteration based on 2 years’ experience• Case studies

– Building management system integration– Residential monitoring– Retrofit system– Repository integration– Transient instrumentation

• Ecosystem – Real-time– Slicing– Analysis

January 9, 2012 Local Winter Retreat 2012 2

Services Communicating with sMAP

January 9, 2012 Local Winter Retreat 2012

Archiver

RDBMS TSDB

sMA

PsM

AP

sMA

P

con

trol

web

6lowpan networks

RS-458 bus

BacNET/IP

mo

de

ls

mg

mt

Many deployments share common infrastructure

Points of decoupling

3

sMAP Principles

• Decoupled, multi-site design– Instruments, archive, and front-ends can all

cross administrative boundaries• Common representation of Timeseries

– Between all parties• Import all data and metadata

– Don’t be tied to particular schemas– Allow for slicing and dicing

• Fix common pain points once


Why Not an Existing System?

• We are built on existing systems– postgresql, django, twisted, bdb

• Many existing systems are vertical stovepipes or proprietary – OSISoft, BMSs, BI software

• Open source software is still struggling to take timeseries seriously– OpenTSDB, Flume/FlumeBase– PubSubHub, pachube

• Be modular and efficient and share code in an open ecosystem– Decoupled architecture allows independent development,

optimization


SMAP SOURCE CASES


Case 1: Building Management System

• Sutardja-Dai (SDH) Hall– Managed with Siemens Apogee

System• Shared serial bus, polled by master• Large number of points, high data

volume• Similarly complex data

relationships


SDH Apogee System

• 1358 control settings– Set points, Relays (lights, pumps, etc), Schedules

• 2291 meters/sensors– Power (building, floor, lights, chiller, pumps, etc)

• Current, voltage, apparent, real, reactive, peak

– Temp (rooms, chilled water, hot water)– Air volume– Alarms, Errors

• 2165 control outputs– Dampers, valves, min/max flow, fan speed, PID

parameters• 72 other


Case 2: Residential Plug-Loads

Dense plug-load metering using wireless mesh networks

Goal: understand the use patterns of appliances

• Deployment methodology for inventory and stratified sampling of loads

• Takeaways: unreliability of home internet, NATed operation


Edge Router

(With LBNL MELS group)

12

Network Complexity & Reachability


(Re)design Takeaways

• Integrating heterogeneous communication substrates is [still] a must– Was a goal of sMAP 1– Better support is possible

• Intermittent connectivity is unavoidable– Don’t lose data when not necessary– Design for reliability

• Existing repositories are unavoidable– Efficient import/export of data (same as a trickle load)– Well known requirement of existing BI/decision support

products• Accept metadata from existing systems when available


Local Winter Retreat 2012

sMAP 1 Overview

/ # list resource under URI root [GET] /data # list sense points under resource data [GET] / [sense_point] # select a sense point [GET] /meter # meters provide this service [GET] / [channel] # a particular channel [GET] /reading # meter reading [GET] /format # calibration and units [GET/POST] /parameter # sampling parameter [GET/POST] /profile # history of readings [GET]

Represent units, sampling parameters of the underlying device, for each channel

Expose over HTTP

January 9, 2012 15

Change 1: Represent Timeseries

• Objects are Timeseries and Collections• Timeseries are durably identified

– A collection of structured key-value pairs, data– Data are (time, value[, seqno]) vectors

• Collections group timeseries with shared metadata


{ "/" : { "Contents" : ["sensor0"], “Metadata” : { “SourceName” : “Example sMAP Source” }, }, "/sensor0" : { "Contents” :["channel1"] }, "/sensor0/channel0" : { "uuid" : "a7f63910-ddc6-11e0-8ab9-13c4da852bbc", "Readings" : [ [1315890624000, 12.5 ] ] }}

colle

ctio

nstim

esei

res

16

/true_power

Properties/ReadingType: doubleProperties/Timezone: America/Los_AngelesProperties/UnitofMeasure: kW

/ABC

Extra/Phase: ABC

Change 2: Simple Metadata

• Each stream can be “tagged” with key-value pairs• Easy to tag related collections of points• Natural way to represent “subset of” relation


/

Location/Campus: UCBLocation/Building: Cory Hall

gpe

Instrument/Manufacturer: Dent IndustriesInstrument/Model: PowerScout 18Instrument/SamplingPeriod: 20

/elt-A

Extra/DentElement: elt-AExtra/Circuit: 1Extra/System: lightingExtra/ServiceDetail: lighting floors 0-4

17

OSS sMAP Source Implementation

• Implementation of new design– Focus on ease of writing new source

• Reliable delivery with local buffering• Multiple configuration options for adding metadata,

setting up drivers• Growing driver library• Documentation, tutorials, etc.• Focus on minimum effort to add new source


Den

t ci

rcui

t m

eter

Tem

pera

ture

/PA

R/TS

R

sMA

PsM

AP

http://code.google.com/p/smap-data/

18

Example Teaser

caiso.ini[/]uuid = dd6a4ae4-b17c-11e0-9d53-0026bb56ec92Metadata/SourceName = CA ISO price and consumption feed

[/demand]type = smap.drivers.caiso.CaIsoDriver

[/oakland]type = smap.drivers.caiso_price.CaIsoPrice# this is passed to the driverLocation = OAKLAND_1_N001


INFRASTRUCTURE


sMAP Ecosystem Architecture


Services Communicating with sMAP


Archiver

RDBMS TSDB

sMA

PsM

AP

sMA

P

con

trol

web

6lowpan networks

RS-458 bus

BacNET/IP

mo

de

ls

mg

mt

Many deployments share common infrastructure

Points of decoupling

22

Archiver

• Application server functionality on top of back-end databases– Add data and metadata from sMAP sources– Query stream data and metadata– Enforce security policy


/add/KEY # add new data to system/republish # data firehose/api/query # look up timeseries by tags/api/[data,next,prev] # retrieve timeseries data/api/tags # return tagsets

23

High-performance Storage Service


Page CacheLock

Manager

Key-Value Store

Storage Alloc.

Time-series Interface

Bucketing RPC Compression

read

ingd

b

SQL

Storage mapper

MySQL

• Low-latency, high speed access to archived readings• 5 billion readings, 3.5 bytes/reading on disk

24

Archiver Query Language

• Column-like store with SQL-like interfaceselect distinct Metadata/SourceName


Tag names look like columns

Arc

hive

r Q

uery

Lan

guag

e Q

uery

Sec

urity

che

ck a

dditi

on

Tra

nsfo

rmat

ion

to s

et o

pera

tions

Exe

cute

on

post

gres

25

Archiver Query Language Examples

query > select distinct Metadata/SourceNameSELECT DISTINCT m.tagval FROM metadata2 m, stream s, subscription sub WHERE s.id = m.stream_id AND s.subscription_id = sub.id AND tagname = 'Metadata/SourceName' AND (sub.public OR sub.key = 'jNiUiSNvb2A4ZCWrbqJMcMCblvcwosStiV71’)

query > select distinct Metadata/SourceName where Metadata/Instrument/Manufacturer ~ '^Dent'SELECT DISTINCT m.tagval FROM metadata2 m, stream s WHERE stream_id IN ( (SELECT stream_id AS cnt FROM metadata2 mi, subscription sub, stream si WHERE ((tagname = 'Metadata/Instrument/Manufacturer' AND tagval ~ E'^Dent')) AND (sub.public OR sub.key = 'jNiUiSNvb2A4ZCWrbqJMcMCblvcwosStiV71') AND mi.stream_id = si.id AND si.subscription_id = sub.id)) AND tagname = 'Metadata/SourceName' AND s.id = m.stream_id


powerdb2: Plotting and Exploration


slicr: Tags Generate Multiple Views


[ { tag : "Metadata/SourceName", restrict: "has Metadata/Extra/EndUse"},

{ tag: "Metadata/Extra/EndUse"},

{ tag: "Metadata/Extra/Category", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Location/Room", "Metadata/Extra/Load"]}, { tag: "Metadata/Extra/ProductType", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Location/Room", "Metadata/Extra/Load"]}, { tag: "Metadata/Instrument/PartNumber", defaultSubStream: "Properties/UnitofMeasure = 'mW'", seriesLabel:["Metadata/Instrument/PartNumber", "Metadata/Location/Room", "Metadata/Extra/Load”]}, "Properties/UnitofMeasure”]

28

Status

• Nearly all sMAP 1 sources moved– Dent + Veris modbus electric– Siemens + Johnson BACnet – pQube, campus obvius electric + steam– Vaisala weather, wunderground, NOAA, NWS– Steam, condensate– BPA Hydro, CAISO price + consumption– ACme, Dust, Labjack, TED

• Sweet new sources– Real-time resampling, campus baselining

• Applications, too:– Data closet PUE– Lighting– Dashboards galore– Lots of undergrad involvement, projects in the works

• Entire system released: readingdb, smap-data, powerdb2


Adoption & The Future

• CBE site– Shared backend– Store and organize BMS data from study participants– Mix of some realtime with mostly loaded data

• LBNL site– Full separate install

• CIEE Energy and Climate Research Innovation Seed Fund Grant

• Energy data application contest (thanks, Quanta!)


DEMO & QUESTIONS


http://new.openbms.org


31

http://new.openbms.org/


smap 2.0: integrating and managing physical data stephen dawson-haggerty computer science division,...

Documents