is4: a storage system for organizing physical data
DESCRIPTION
IS4: A Storage System for Organizing Physical Data. Jorge Ortiz University of California, Berkeley. Talk overview. Integrated sensor stream storage system (IS4) Recap from last retreat Physical data Representation of: Physical devices Measurements Environments Management challenges - PowerPoint PPT PresentationTRANSCRIPT
IS4:A Storage System for
Organizing Physical Data
Jorge OrtizUniversity of California, Berkeley
Talk overview
• Integrated sensor stream storage system (IS4)– Recap from last retreat
• Physical data– Representation of:
• Physical devices• Measurements• Environments
– Management challenges
• Current features and future work
3
Inst
rum
enta
tion
Inst
rum
enta
tion
Mod
els
Mod
els
ControlsControls
Building OSBuilding OS
Plug LoadsPlug Loads
LightingLightingFacilitiesFacilities
BuildingIn
stru
men
tatio
nIn
stru
men
tatio
nM
odel
sM
odel
s
Routing/ControlRouting/Control
Grid OSGrid OS
Demand ResponseDemand Response
Load FollowingLoad FollowingSupply FollowingSupply Following
Grid
Facility-to-Building
Facility-to-Building
Facility-to-Building
Facility-to-Building
Facility-to-Building
Facility-to-Building
Gen-to-BuildingGen-to-Building
Inst
rum
enta
tion
Inst
rum
enta
tion
Mod
els
Mod
els
ControlControl
CompressorScheduling
CompressorScheduling
Temperature MaintenanceTemperature Maintenance
Fridge as a model Thermal storage system
Storage-to-Building
Storage-to-Building
Inst
rum
enta
tion
Inst
rum
enta
tion
Mod
els
Mod
els
Power-AwareCluster Manager
Power-AwareCluster Manager
Load Balancer/Scheduler
Load Balancer/Scheduler
Web ServerWeb Server
Web App LogicWeb App LogicDB/StorageDB/Storage
Machine RoomMR-to-BuildingMR-to-Building
Multi-scale Energy Network
Gen-to-Grid
Gen-to-Grid
uGrid-to-GriduGrid-to-Grid
Building-to-Grid
Building-to-Grid Building-
to-GridBuilding-to-Grid
Wind Modeling
IS4
SCADA SystemsSupervisory Control and Data Acquisition
+SODA4R787__ART
+Data
Problems with SCADA
• No systematic organization of data and metadata– Metadata embedded in UI– Difficult to find context information
• Incomplete data and metadata– Mostly 0’s– Missing type descriptions, location information
Capturing building layout
Chiller
Pump
Chiller
Pump
AHUSF EF
Vent Vent
Zone
Entity relationships Building Systems Spaces/Zones Structural relationship Sensors/Actuators
Inside systems Inside zones/spaces
Types
Chiller
Pump
Chiller
Pump
AHUSF EF
Vent Vent
Zone
System
Space
Building entity schema
Relevant queries
• List temperature sensors on 4th floor of Soda hall– SELECT id FROM zone_sensors WHERE
building=‘Soda’ and floor=4 AND id LIKE ‘%ART’
• Get all temperature sensor data in the last day on the 4th floor of Soda– SELECT * FROM sensor_data WHERE id=[list of
sensors] AND timestamp >= ’ 2009-12-16 00:00:00’ and timestamp<=now()
System objectives
• Physical data storage repository– Append only, forever
• Management of data and metadata– Changes occur over time
• Physical-data queries– Historical and real time
• Security– Secure access to data streams, distillates, metadata
What is physical data?
• Data collected from the environment• Why is it different from other kinds of data?
– Data is produced by sensing devices– Devices are placed in the physical environment– Placement matters
• Deployment changes over time– Devices, configuration, environment
Data, metadata relationship-managementis challenging
Sensing device metadata
• Make– Dent SmartLogger
• Model– CTLogger TOU-CT
• url– http://www.dentinstruments.com/media/
PDF/SMARTware2009_Manual.pdf
• Make– ACme
• Model– Revision 1
• url– http://smote.cs.berkeley.edu:8000/
tracenv/wiki/ACME
Object schema
{ "description":"object_stream_schema", "type":"object", "properties": {
“type":{"type":"string","optional":false, "options":[{"value":"object_stream"}]}, "device_name":{"type":"string", "optional":"false"}, "model":{"type":"string","optional":false}, "desc":{"type":"string","optional":true},“url":{"type":"string",”format”:”url”,"optional":true
}}
{ "description":"object_stream_schema", "type":"object", "properties": {
“type":{"type":"string","optional":false, "options":[{"value":"object_stream"}]}, "device_name":{"type":"string", "optional":"false"}, "model":{"type":"string","optional":false}, "desc":{"type":"string","optional":true},“url":{"type":"string",”format”:”url”,"optional":true
}}
{ “type“:"object_stream", “$schema”:”object_stream_schema.json”,
“device_name":“dentMeter123", "model":“CTLogger TOU-CT", "desc":“dent meter”,“url”:” http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf”
}
{ “type“:"object_stream", “$schema”:”object_stream_schema.json”,
“device_name":“dentMeter123", "model":“CTLogger TOU-CT", "desc":“dent meter”,“url”:” http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf”
}
inst
ance
inst
ance
sche
ma
sche
ma
Communication information
• Network address or URL
• Function calls, parameter description
• Functional schema or url documentation
2001:252:0:1::2008:6 and 2001:252:0:1::2008:8
http://coryacme.cs.berkeley.edu/data/33
Logic schema
{ "description":“logic_stream_schema", "type":"object", "properties": {
“type":{"type":"string","optional":false, "options":[{"value":“logic"}]}, “$schema”:”logic_stream_schema.json”,“address”:{“type”:”string”, “format”:[“string”, “ipv4”, “ipv6”, “url”],
“optional”:”false”},“functions”:{“type”:”array”, “options”:[{
“function_name”:”{“type”:”string”},“operation”:{“type”:”string”, “optional”:”true”},“parameters”:{“type”:”array”, “options”:[”object”, “string”, “$ref”]}}
}}
{ "description":“logic_stream_schema", "type":"object", "properties": {
“type":{"type":"string","optional":false, "options":[{"value":“logic"}]}, “$schema”:”logic_stream_schema.json”,“address”:{“type”:”string”, “format”:[“string”, “ipv4”, “ipv6”, “url”],
“optional”:”false”},“functions”:{“type”:”array”, “options”:[{
“function_name”:”{“type”:”string”},“operation”:{“type”:”string”, “optional”:”true”},“parameters”:{“type”:”array”, “options”:[”object”, “string”, “$ref”]}}
}}
{ “type“:“logic", “$schema”:”logic_stream_schema.json”,“address”:”http://is4/resource”,“function”:[{
“function_name”:”a”, “opertation”:”GET”,“parameters”:[{“field”, “name”}]}
}
{ “type“:“logic", “$schema”:”logic_stream_schema.json”,“address”:”http://is4/resource”,“function”:[{
“function_name”:”a”, “opertation”:”GET”,“parameters”:[{“field”, “name”}]}
}
inst
ance
inst
ance
sche
ma
sche
ma
Sensor context information
Temperature
Humidity
Vibration
Pressure
Electric load tree+
Electrical Physical
Humidity
PAR/TSR
Temperature
Panel 1 Panel 2
A
B
A
B
Panel 1 Panel 2
A
B
A
B
1
5
9
13
17
21
25
29
33
37
41
3
7
11
15
19
23
27
31
35
39
A1
5
9
13
17
21
25
29
33
37
41
3
7
11
15
19
23
27
31
35
39
1
5
9
13
17
21
25
29
33
37
41
3
7
11
15
19
23
27
31
35
39
A2
6
10
14
18
22
26
30
34
38
42
4
8
12
16
20
24
28
32
36
40
B2
6
10
14
18
22
26
30
34
38
42
4
8
12
16
20
24
28
32
36
40
2
6
10
14
18
22
26
30
34
38
42
4
8
12
16
20
24
28
32
36
40
B
1
5
9
13
17
21
25
29
33
37
41
3
7
11
15
19
23
27
31
35
39
A1
5
9
13
17
21
25
29
33
37
41
3
7
11
15
19
23
27
31
35
39
1
5
9
13
17
21
25
29
33
37
41
3
7
11
15
19
23
27
31
35
39
A2
6
10
14
18
22
26
30
34
38
42
4
8
12
16
20
24
28
32
36
40
B2
6
10
14
18
22
26
30
34
38
42
4
8
12
16
20
24
28
32
36
40
2
6
10
14
18
22
26
30
34
38
42
4
8
12
16
20
24
28
32
36
40
B
Circuit/breaker panel level power monitoring
Vibration
Load Tree
CT mains power monitoring
ACme: plug load energy monitor and controller
+ Slide courtesy of Xiaofan Jiang
Context description
• General description• Google JSON GeoCoding object
– http://code.google.com/apis/maps/documentation/javascript/v2/services.html#Geocoding_Object
• IS4 context graph– Schema that expresses contextual objects and
relationships as a graph
IS4 Context graph schemasc
hem
asc
hem
a
Context Node:{ "label":{"type":"string", "optional":false},
"name":{"type":"string", "optional":false},"type":{"type":"string","options":[{"value":"context_node"}], "optional":false},"description":{"type":"string", "optional":false}, "properties": {
"AssociatedDevices":{ "type":"array","options":[{"value":"string","format":"UUID"}], "optional":true },
"parents":{ "type":"array", "options":[{"value":"string","properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "children":{ "type":"array", "options":[{"value":"string",
"properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "cnid": { "type":[“string"] "description":"local node identifier", "optional":false }}}
Context Edge{…
"label":{"type":"string", "optional":false, "description":"String with spaces"}, "name":{"type":"string", "optional":false, "description":"String without space"}, …"sourceNode":{"type":[“string"], "description":"local node identifier",
"optional":false}, "destinationNode":{"type":[“string"], "description":"local node identifier",
"optional":false}, "AssociatedDevices":{ "type":"array",
"options":[{"value":"string","format":"UUID"}], "optional":true }}
Context Node:{ "label":{"type":"string", "optional":false},
"name":{"type":"string", "optional":false},"type":{"type":"string","options":[{"value":"context_node"}], "optional":false},"description":{"type":"string", "optional":false}, "properties": {
"AssociatedDevices":{ "type":"array","options":[{"value":"string","format":"UUID"}], "optional":true },
"parents":{ "type":"array", "options":[{"value":"string","properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "children":{ "type":"array", "options":[{"value":"string",
"properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "cnid": { "type":[“string"] "description":"local node identifier", "optional":false }}}
Context Edge{…
"label":{"type":"string", "optional":false, "description":"String with spaces"}, "name":{"type":"string", "optional":false, "description":"String without space"}, …"sourceNode":{"type":[“string"], "description":"local node identifier",
"optional":false}, "destinationNode":{"type":[“string"], "description":"local node identifier",
"optional":false}, "AssociatedDevices":{ "type":"array",
"options":[{"value":"string","format":"UUID"}], "optional":true }}
Cory hall load tree
Load Tree Image Graphical representation
Context state snapshots
Time t1 Time t2
Queries
• What floor consumes the most power per month?
• Send alert if the air conditioner in room 465 consumes more than Y kW in a 24 hour period
• Send alert if independent zones become dependent
• What is the most common state of the air/water flow graph?
Expressing physical-data queries
• Semantic graph + temporal data– Physical and logical relationships– Data streams associated with graphical elements
• Easily express queries across both data types• Efficient storage• Efficient access
IS4 Current Status
• Data publishers identified by unique id (and IS4 resource URI).
• Publishing process includes object and logic JSON objects as input
• Context data managed separately• Publisher may belong to multiple contexts• Data and metadata stream explicitly bound
Current Architecture
StorageStorage
HTTP/RESTHTTP/REST
Publish ManagerPublish Manager Subscribe ManagerSubscribe ManagerProxy
ManagerProxy
Manager
HumidityPAR/TSRTemperature VibrationCT mains power monitoring
ACme: plug load energy monitor and controller
PublishPublish
SubscribeSubscribe
Context ManagerContext Manager
IS4 Interface Overview
/ #root – status information [GET] /is4 # a particular channel [GET] /info # statistical system information [GET] /publish # publishing resource [GET/PUT]
/all # list of all publishers [GET] /id # sub-children [GET]
/<id> # id of publisher [GET/POST] /name # name alias for this id [GET]
/mysubs # list/add to pub subscriptions [GET/PUT/DELETE]
/metadata# returns publisher-associate metadata [GET]
POST requests supply JSON objects as arguments:POST:
http://is4.cs.berkeley.edu/is4/publish/id/123456/mysubs?username=jortiz&pw=8832749823{“SubscribeTo":[13241324,234213,45456745], “UnsubscribeFrom":[45432,567365,21465765,2435786]}
Publisher Id Username and pw hash
Ongoing work
• Security– Token-based security– Tokens encapsulate access rights to resources
• Resources referenced by IS4 URI• Permissions: GET, PUT, POST, DELETE
– GET = read– PUT = create– POST = update– DELETE = delete
• Query interface– Semantic graph + timeseries
Future Work
• Incorporate security model into current implementation
• Move the data store into Amazon cloud– 14 Dents, 1 min period = ~15 MB/hour 130
GB/year (uncompressed)
• Data processing functionality– Interpolation, extrapolation, other time series
processing
• JSON streaming query engine
Summary
• Current building management systems poorly handle physical data
• Physical data is tightly bound metadata and context
• IS4 provides schemas to express metadata• IS4 implements mechanisms to manage
data/metadata binding
Thank you
• More information– http://smote.cs.berkeley.edu:8000/tracenv/wiki/is4
• Jorge Ortiz– [email protected]
• Questions?
Extra Slides
Snapshots of physical information
• Time t1 • Time t2
Publishers and Subscribers
Open design issues
• Doesn’t PowerMeter already do (some) of this?– They also deal with the data stream; context is not
used
• How different from pubsubhubbub?– Explicit context management
Interpreting data across contextual changes
• All the data is timeseries• Real time and historical querying
– User can be notified if context changes have occurred over query-time interval
General notes
• Talk about what I presented at the last retreat• Include previous work• Include a summary slide at the end• Update the opening slide (missing sponsors)• Look at Stephen’s slides and figure out how to
incorporate them– You have to make connections between the two
• Ask Gil for his slides to see how to segway these into that
General notes (2)