research on the framework and data fusion of an energy big-data … · 2017-08-25 · research on...

Research on the Framework and Data Fusion of an Energy Big-data Platform

Gengfeng Li, Zhaohong Bie, Jiang Wu, Cheng [email protected]

Xi’an Jiaotong University

21 July 2017

1

Panel: Big data for Integrated Energy Systems

Paper Number: 17PESGM2652

2

Content

Integrated Energy System

Framework of an Energy Big-data Platform

Multi-source Heterogeneous Data Fusion

Conclusions and Future Work

01

02

03

04

3

Integrated Energy System(Energy Internet)


4

01

New challenges

Sustainable development

Energy crisis

Environmental degradation

Global climate change


Revolution of energy production and consumption

1st Industrial

Revolution:

Appearance

of steam

engine

2st Industrial

Revolution:

Wide use of

electricity

3st Industrial

Revolution:

Nuclear power,

computers

4st Industrial

Revolution:

Cyber-

physical

system

Time: end of 18th century start of 20th century 70s of 20th century now

This new concept attracts a lot of attention

Highly integrated and interdependent energy and cyber systems

There is no consensus on an exact definition

5

Integrated Energy System01

A new generation of energy system with electricity at core

Integrated Energy System - seen from power system field

Power system at core

Maximum integration

of renewable energy

Centralized and

distributed resources

Integrated energy

system considering

demand response

Centralized

resources

renewableenergy

renewableenergy

Demand

response


6

01

Integrated Energy System - seen from cyber system field

To innovate the current energy system based on the benefits

of the cyber system, including openness, free flow of energy

and peer-access


7

01

State Grid: to build a

Global Integrated Energy

System, based on ultra

high-voltage electric

transmission system,

sharing and utilization

renewable energy

Integrated Energy System - seen from industry field

Global Integrated Energy System


8

01

8

Integrated Energy System - seen from finance field

Business mode of Integrated Energy System

Based on new technology and development trend

Create revenue for customers and service providers

Innovation for energy supply

cloud platform


9

01

Finance field

Industry field

Electricity field

Automation field

Cyber field

……

Integrated

Energy System


10

01

“Researches on basic theory of planning, operation and tra-

ding for Integrated Energy System” (2016YFB0901900)

(Principal Investigator)

2016 National Key Research and Development

Program of China

Power Plant

Substation Substation

Tower

Tower

Wind Farm

PV Power Station

Tower

Tower

Factory

Resident Users

Residential Building

Business Building

Transmission Line


11

01

Traditional power system

generation

transmission

utilization

distribution

Hydrogen Production Plant

Product H2

H2 Storage Container

Store H2

Generate Electricity Using Natural Gas

Heat Storage

Heating StationPressurizer

Natural Gas Pipeline

EV

Gas Power Plant

Power Plant


Tower

Tower

Wind Farm

PV Power Station

Tower

Tower

Factory

Resident Users


Business Building

Transmission Line

Gas Power Plant

Natural Gas PipelinePressurizer

Product H2

Product CH4

Product CH4

Information Exchange




Cloud Computing Equipment

the Internet

Charging Pile

Electrified Traffic System


12

01

1. A variety of primary energy including wind, solar, gas and coal;

2. Spatial and temporal distribution analysis of different resources;

3. Intermittent of renewable energy;

Source

Architecture


Product H2


Store H2


Heat Storage



EV

Gas Power Plant

Power Plant


Tower

Tower

Wind Farm

PV Power Station

Tower

Tower

Factory

Resident Users


Business Building

Transmission Line

Gas Power Plant


Product H2

Product CH4

Product CH4






the Internet

Charging Pile



13

01

1. Coupled Natural Gas and Electric Power Network;

2. Large scale, different time scale flows: electricity and gas flow;

3. Big data from both Natural Gas and Electric Power Systems.

Architecture

Source Network


Product H2


Store H2


Heat Storage



EV

Gas Power Plant

Power Plant


Tower

Tower

Wind Farm

PV Power Station

Tower

Tower

Factory

Resident Users


Business Building

Transmission Line

Gas Power Plant


Product H2

Product CH4

Product CH4






the Internet

Charging Pile



14

01

1. Distributed sources: CHP, DG, etc.;

2. Coupled with Transportation Systems via electric vehicles;

3. Multi-energy demand and transform: cooling, heating, gas, electricity.

Architecture

Source Network Demand

Optimal operation and dispatch

for Energy Internet

Comprehensive demonstration system of Energy Internet

Modeling of cyber-physical system based

on Energy Internet

Operation simulation platform

Trading simulation platform

Planning

platform

Big data

center

System planning and business

model for Energy Internet

Mixed dynamic system

modeing

Stochastic system simulation and

evaluation

United Interface

system modeling

stochastic modeling of

compatibility

Integrated risk

analysis

Distributed

optimization

Coordinated

planning

Distributed game theory

analysis

Integrated marketStochastic

optimization


15

01

modeling

algorithm

simulation

demonstration

Modeling of Integrated Energy Systems

System Planning and Trading

MechanismOptimal operation and dispatch

Comprehensive demonstration platform of Integrated Energy Systems

16


Data for demonstration

17

Electric Power Network of Northwest China

Jiuquan wind power base

Hydropower base of the Yellow River

Energy base of northern Shaanxi

Electric Power Network of Shaanxi

Energy system of Xi’an

high energy-consuming enterprises

Distributed multiple energy system

Framework of an Energy Big-data Platform02


18

02

Big-data Platform

Planning Platform Trading Platform

Operation Platform

Features of the Energy Big-data Platform

Collection of multi-type

energy data

Fast Query of multi-type

energy data

Data processing and

computing

19

(1) Collection of multi-type energy data

Real-time Data Collection1

Wind Farm PV StationH2 Production

Plant

KafkaCluster

Real-time Monitoring

Hadoop Cluster Database

...

Real-time data flow collection based on Kafka


20

Storage2 Data Fusion3

Challenge

Big data volume(TB level)

Solution

Hadoop Distributed File System (HDFS)

Challenge

Sources of data various

Heterogeneous characteristics

Solution


Semantic heterogeneous data fusion

&

System heterogeneous data fusion

Big-data Platform

Raw Data

(1) Collection of multi-type energy data


21

(2) Fast Query of multi-type energy data

Hadoop Data Dase

Hbase

Basic storage support

HDFS

Computing power support

MapReduce

Coordination services & Failover

ZooKeeper

High level language support

Hive

Challenge

Big data rapid indexing

Solution

A distributed, scalable, big data store:

Apache Hbase.


22

(3) Data processing and computing

Mining Optimization Visualization

Load

Forecasting

State

Assessment

Power Quality

Monitoring

Planning

Optimization

Operation

Optimization

Market

Optimization

Report Form

Summary

Graph

User

Interface

Planning

Platform

Trading

Platform

Operation

Platform


23

01

02

03

Big-data

Platform

04

Theoretical

Architecture

Technology

Architecture

Physical

Architecture

Core Technology

and Function

Framework


24

(1) Theoretical Architecture

Power system

data

Natural gas

system data

Heating system

data

Meteorological

data

Financial

market data

Data Source

Com

munic

ation N

etw

ork

Data Storage

Offline

Computing

Rapid Index

Online

Computing

Optimization

Resource

Management

Big-data Platform

Inte

ractive inte

rface

Planning

Platform

Trading

Platform

Operation

Platform


25

User Application Layer

User Gateway Layer

Base Platform Layer

Infrastructure LayerHardware Resources Operating Environment

Server #1 Server #n Redhat Linux 6.2

Wind Power Data

Simulation

PV DataSimulation

Building Users Data Simulation

Factory Users Data Simulation

Kafka

HDFS Hive

Universal Optimiz--ation Platform

MapreduceSpark

Surveillan

ce System

Distrib

uted

Co

llabo

rative Fram

ewo

rk

（ZOO

KEEP

ER

）

Offline Computing Online Computing OptimizationResource

Management

Spark Client

Hive Client

Hadoop Client

Yarn Standalone

Ap

plication

Gatew

ayIn

frastructure

Base

Platfo

rm

Storm

Storm Client

Universal Optimization Platform Client

(2) Technology Architecture


26

(3) Physical Architecture The layers of the Big-data platform are

deployed to the physical nodes, which

are connected as a whole through the

LAN, providing physical support for

applications.


27

(4) Core Technology and Function

Hardware Resources

Server #1 Server #2 Server #n

Data CollectionKafka

Data StorageHDFS Hive

Data Processing and Computing

Resource and Job Management

Yarn Standalone

Offline Computing Realtime Computing Optimization

Distributed Optimization

Universal Optimiz--ation Platform

Realtime Analysis

Spark StreamingStorm

Random Characteristic Analysis

Spark MllibMapreduceSu

rveillance System

Distrib

uted

Co

llabo

rative Framew

ork

（ZOO

KEEP

ER

）

Wind Power Data

Simulation

PV DataSimulation

Building Users Data Simulation

Factory Users Data Simulation


28


The large amount of data is a big challenge

Variety means the increasing complex of data types

Data itself is meaningless unless valuable knowledge

Refers to the speed requirement for processing data

29

Multi-source Heterogeneous Data Fusion03

Energy Big-data

Volume

Variety

Value

Velocity

30


System heterogeneous

data are stored in different system or database

Semantic heterogeneous

two records of the same entity have different

express

Structured heterogeneousthe data are not only

structured data but also semi-structured data and unstructured data

Grammar heterogeneous

data have different formats such as units of data

Heterogeneous data

1. Semantic heterogeneous data fusion

(1) Duplicate database records：

(2) Fusion methods

① Field Matching Method

② Sorted-neighborhood Method

31


name date of birth terminal name terminal address power

K.X. Huang 1994.10.14 Taoyuan 3166 2439.71

K.X. Huang 1994.10.14 Taoyuan 3166 439.71

① Field Matching Method(FMM)

32


• Calculate the weight of each attribute of records

Weight

• Calculate the similaritybetween records

Similarity• Records

matching and detection

Detection

33

𝑌 = 𝑦1, 𝑦2, ⋯ , 𝑦𝑛 , where 𝑦𝑖 = 𝑦𝑖1, 𝑦𝑖2, ⋯ , 𝑦𝑖𝑚 , 𝑖 ∈ 1,2,⋯ , 𝑛

Y means n records in a database with m attributes

𝑇𝑖𝑘 ∈ 𝑇𝑖1, 𝑇𝑖2, ⋯ , 𝑇𝑖𝑚 , 𝑖 ∈ 1,2,⋯ ,𝑁 , 𝑇𝑖𝑘 ≥ 1

𝑇 = 𝑇1, 𝑇2, ⋯ , 𝑇𝑚 , where 𝑇𝑘 = Τσ𝑖=1𝑁 𝑇𝑖𝑘 𝑁 , 𝑘 ∈ 1,2,⋯ ,𝑚

𝑇𝑘 → 𝑇𝑘′ , 𝑆 = 𝑚𝑎𝑥 𝑇1

′, 𝑇2′, ⋯ , 𝑇𝑚

′ , Convert 𝑇𝑘 to an integer 𝑇𝑘′

𝑊𝑘′ =

1

𝑆

𝑖=𝑇𝑘′

𝑆1

𝑖, 𝑘 ∈ 1,2,⋯ ,𝑚

𝑊𝑗 = ൘𝑊𝑗′

𝑘=1

𝑚

𝑊𝑘′ , 𝑗 ∈ 1,2,⋯ ,𝑚

33


a) Weight

3434

b) Similarity𝑦𝑖𝑘 = 𝑦𝑖1, 𝑦𝑖2, ⋯ , 𝑦𝑖𝑝

𝑦𝑗𝑘 = 𝑦𝑗1, 𝑦𝑗2, ⋯ , 𝑦𝑗𝑞

𝑠𝑖𝑚 𝑦𝑖𝑘 , 𝑦𝑗𝑘 = ൘

𝑏=1

𝑞

𝑚𝑎𝑥 𝑠𝑐𝑜𝑟𝑒 𝑦𝑖𝑘𝑎, 𝑦𝑗𝑘𝑏 𝑝 , 𝑎 ∈ 1,2,⋯ , 𝑝

𝑠𝑖𝑚 𝑦𝑖 , 𝑦𝑗 =

𝑘=1

𝑚

𝑊𝑘 ∗ 𝑠𝑖𝑚(𝑦𝑖𝑘 , 𝑦𝑗𝑘)

c) Detection

𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 =𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑢𝑚𝑏𝑒𝑟𝑠

𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑛𝑢𝑚𝑏𝑒𝑟𝑠

𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑢𝑚𝑏𝑒𝑟𝑠

𝑡𝑟𝑢𝑒 𝑛𝑢𝑚𝑏𝑒𝑟𝑠

34


the k-th attribute of the record 𝑦𝑖 has p strings

the k-th attribute of the record 𝑦𝑗 has q strings

35

② Sorted-neighborhood Method(SNM)

……

……

35


Sort

Window

Detection

………………

current windownext window

(3) Example

36

name date of birth terminal name terminal address power

K.X. Huang 1994.10.14 Taoyuan 3166 2439.71

K.X. Huang 1994.10.14 Taoyuan 3166 439.71

D.J. Zhang 1995.40.28 Taoyuan 3166 510.2

D.J. Zhang 1995.4.28 Taoyuan 3166 51.2


Detected number correct number completeness precision time(s)

FMM 237 230 0.9465 0.9705 3.555

SNM 238 233 0.9588 0.9790 0.249

Power Data of Taoyuan residential community

Results of two methods

2. System heterogeneous data fusion

(1) Data are stored in different database

(2) Fusion method

Open Database Connectivity

(3) Example

37

Excel

ODBCOracle

databaseText

SQL


38



39

04

Conclusions

1. Integrated energy system has drawn widely attention

around the world. Researches from various of fields

greatly promote the development of Integrated energy

system.

2. Energy Big-data Platform is the foundation of an

Integrated Energy Platform, and is a significant research

field.

3. Methods for multi-source heterogeneous data fusion is

introduced, furthermore, establishment of an Energy Big-

data Platform framework is on going.


40

04

Future Work

An Integrated Energy Platform based on the Energy Big-data

Platform

Energy Big-data Platform

Integrated Energy Platform

Integrated Energy

Planning Platform

Integrated Energy

Operation Platform

Integrated Energy

Trading Platform

41

Thanks for Your Attention!

research on the framework and data fusion of an energy big-data … · 2017-08-25 · research on...

Documents