![Page 1: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/1.jpg)
Research on the Framework and Data Fusion of an Energy Big-data Platform
Gengfeng Li, Zhaohong Bie, Jiang Wu, Cheng [email protected]
Xi’an Jiaotong University
21 July 2017
1
Panel: Big data for Integrated Energy Systems
Paper Number: 17PESGM2652
![Page 2: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/2.jpg)
2
Content
Integrated Energy System
Framework of an Energy Big-data Platform
Multi-source Heterogeneous Data Fusion
Conclusions and Future Work
01
02
03
04
![Page 3: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/3.jpg)
3
Integrated Energy System(Energy Internet)
![Page 4: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/4.jpg)
Integrated Energy System
4
01
New challenges
Sustainable development
Energy crisis
Environmental degradation
Global climate change
Integrated Energy System
Revolution of energy production and consumption
1st Industrial
Revolution:
Appearance
of steam
engine
2st Industrial
Revolution:
Wide use of
electricity
3st Industrial
Revolution:
Nuclear power,
computers
4st Industrial
Revolution:
Cyber-
physical
system
Time: end of 18th century start of 20th century 70s of 20th century now
This new concept attracts a lot of attention
Highly integrated and interdependent energy and cyber systems
There is no consensus on an exact definition
![Page 5: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/5.jpg)
5
Integrated Energy System01
A new generation of energy system with electricity at core
Integrated Energy System - seen from power system field
Power system at core
Maximum integration
of renewable energy
Centralized and
distributed resources
Integrated energy
system considering
demand response
Centralized
resources
renewableenergy
renewableenergy
Demand
response
![Page 6: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/6.jpg)
Integrated Energy System
6
01
Integrated Energy System - seen from cyber system field
To innovate the current energy system based on the benefits
of the cyber system, including openness, free flow of energy
and peer-access
![Page 7: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/7.jpg)
Integrated Energy System
7
01
State Grid: to build a
Global Integrated Energy
System, based on ultra
high-voltage electric
transmission system,
sharing and utilization
renewable energy
Integrated Energy System - seen from industry field
Global Integrated Energy System
![Page 8: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/8.jpg)
Integrated Energy System
8
01
8
Integrated Energy System - seen from finance field
Business mode of Integrated Energy System
Based on new technology and development trend
Create revenue for customers and service providers
Innovation for energy supply
cloud platform
![Page 9: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/9.jpg)
Integrated Energy System
9
01
Finance field
Industry field
Electricity field
Automation field
Cyber field
……
Integrated
Energy System
![Page 10: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/10.jpg)
Integrated Energy System
10
01
“Researches on basic theory of planning, operation and tra-
ding for Integrated Energy System” (2016YFB0901900)
(Principal Investigator)
2016 National Key Research and Development
Program of China
![Page 11: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/11.jpg)
Power Plant
Substation Substation
Tower
Tower
Wind Farm
PV Power Station
Tower
Tower
Factory
Resident Users
Residential Building
Business Building
Transmission Line
Integrated Energy System
11
01
Traditional power system
generation
transmission
utilization
distribution
![Page 12: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/12.jpg)
Hydrogen Production Plant
Product H2
H2 Storage Container
Store H2
Generate Electricity Using Natural Gas
Heat Storage
Heating StationPressurizer
Natural Gas Pipeline
EV
Gas Power Plant
Power Plant
Substation Substation
Tower
Tower
Wind Farm
PV Power Station
Tower
Tower
Factory
Resident Users
Residential Building
Business Building
Transmission Line
Gas Power Plant
Natural Gas PipelinePressurizer
Product H2
Product CH4
Product CH4
Information Exchange
Information Exchange
Information Exchange
Information Exchange
Cloud Computing Equipment
the Internet
Charging Pile
Electrified Traffic System
Integrated Energy System
12
01
1. A variety of primary energy including wind, solar, gas and coal;
2. Spatial and temporal distribution analysis of different resources;
3. Intermittent of renewable energy;
Source
Architecture
![Page 13: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/13.jpg)
Hydrogen Production Plant
Product H2
H2 Storage Container
Store H2
Generate Electricity Using Natural Gas
Heat Storage
Heating StationPressurizer
Natural Gas Pipeline
EV
Gas Power Plant
Power Plant
Substation Substation
Tower
Tower
Wind Farm
PV Power Station
Tower
Tower
Factory
Resident Users
Residential Building
Business Building
Transmission Line
Gas Power Plant
Natural Gas PipelinePressurizer
Product H2
Product CH4
Product CH4
Information Exchange
Information Exchange
Information Exchange
Information Exchange
Cloud Computing Equipment
the Internet
Charging Pile
Electrified Traffic System
Integrated Energy System
13
01
1. Coupled Natural Gas and Electric Power Network;
2. Large scale, different time scale flows: electricity and gas flow;
3. Big data from both Natural Gas and Electric Power Systems.
Architecture
Source Network
![Page 14: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/14.jpg)
Hydrogen Production Plant
Product H2
H2 Storage Container
Store H2
Generate Electricity Using Natural Gas
Heat Storage
Heating StationPressurizer
Natural Gas Pipeline
EV
Gas Power Plant
Power Plant
Substation Substation
Tower
Tower
Wind Farm
PV Power Station
Tower
Tower
Factory
Resident Users
Residential Building
Business Building
Transmission Line
Gas Power Plant
Natural Gas PipelinePressurizer
Product H2
Product CH4
Product CH4
Information Exchange
Information Exchange
Information Exchange
Information Exchange
Cloud Computing Equipment
the Internet
Charging Pile
Electrified Traffic System
Integrated Energy System
14
01
1. Distributed sources: CHP, DG, etc.;
2. Coupled with Transportation Systems via electric vehicles;
3. Multi-energy demand and transform: cooling, heating, gas, electricity.
Architecture
Source Network Demand
![Page 15: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/15.jpg)
Optimal operation and dispatch
for Energy Internet
Comprehensive demonstration system of Energy Internet
Modeling of cyber-physical system based
on Energy Internet
Operation simulation platform
Trading simulation platform
Planning
platform
Big data
center
System planning and business
model for Energy Internet
Mixed dynamic system
modeing
Stochastic system simulation and
evaluation
United Interface
system modeling
stochastic modeling of
compatibility
Integrated risk
analysis
Distributed
optimization
Coordinated
planning
Distributed game theory
analysis
Integrated marketStochastic
optimization
Integrated Energy System
15
01
modeling
algorithm
simulation
demonstration
Modeling of Integrated Energy Systems
System Planning and Trading
MechanismOptimal operation and dispatch
Comprehensive demonstration platform of Integrated Energy Systems
![Page 16: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/16.jpg)
16
Framework of an Energy Big-data Platform
![Page 17: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/17.jpg)
Data for demonstration
17
Electric Power Network of Northwest China
Jiuquan wind power base
Hydropower base of the Yellow River
Energy base of northern Shaanxi
Electric Power Network of Shaanxi
Energy system of Xi’an
high energy-consuming enterprises
Distributed multiple energy system
Framework of an Energy Big-data Platform02
![Page 18: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/18.jpg)
Framework of an Energy Big-data Platform
18
02
Big-data Platform
Planning Platform Trading Platform
Operation Platform
Features of the Energy Big-data Platform
Collection of multi-type
energy data
Fast Query of multi-type
energy data
Data processing and
computing
![Page 19: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/19.jpg)
19
(1) Collection of multi-type energy data
Real-time Data Collection1
Wind Farm PV StationH2 Production
Plant
KafkaCluster
Real-time Monitoring
Hadoop Cluster Database
...
Real-time data flow collection based on Kafka
Framework of an Energy Big-data Platform02
![Page 20: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/20.jpg)
20
Storage2 Data Fusion3
Challenge
Big data volume(TB level)
Solution
Hadoop Distributed File System (HDFS)
Challenge
Sources of data various
Heterogeneous characteristics
Solution
Multi-source Heterogeneous Data Fusion
Semantic heterogeneous data fusion
&
System heterogeneous data fusion
Big-data Platform
Raw Data
(1) Collection of multi-type energy data
Framework of an Energy Big-data Platform02
![Page 21: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/21.jpg)
21
(2) Fast Query of multi-type energy data
Hadoop Data Dase
Hbase
Basic storage support
HDFS
Computing power support
MapReduce
Coordination services & Failover
ZooKeeper
High level language support
Hive
Challenge
Big data rapid indexing
Solution
A distributed, scalable, big data store:
Apache Hbase.
Framework of an Energy Big-data Platform02
![Page 22: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/22.jpg)
22
(3) Data processing and computing
Mining Optimization Visualization
Load
Forecasting
State
Assessment
Power Quality
Monitoring
Planning
Optimization
Operation
Optimization
Market
Optimization
Report Form
Summary
Graph
User
Interface
Planning
Platform
Trading
Platform
Operation
Platform
Framework of an Energy Big-data Platform02
![Page 23: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/23.jpg)
23
01
02
03
Big-data
Platform
04
Theoretical
Architecture
Technology
Architecture
Physical
Architecture
Core Technology
and Function
Framework
Framework of an Energy Big-data Platform02
![Page 24: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/24.jpg)
24
(1) Theoretical Architecture
Power system
data
Natural gas
system data
Heating system
data
Meteorological
data
Financial
market data
Data Source
Com
munic
ation N
etw
ork
Data Storage
Offline
Computing
Rapid Index
Online
Computing
Optimization
Resource
Management
Big-data Platform
Inte
ractive inte
rface
Planning
Platform
Trading
Platform
Operation
Platform
Framework of an Energy Big-data Platform02
![Page 25: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/25.jpg)
25
User Application Layer
User Gateway Layer
Base Platform Layer
Infrastructure LayerHardware Resources Operating Environment
Server #1 Server #n Redhat Linux 6.2
Wind Power Data
Simulation
PV DataSimulation
Building Users Data Simulation
Factory Users Data Simulation
Kafka
HDFS Hive
Universal Optimiz--ation Platform
MapreduceSpark
Surveillan
ce System
Distrib
uted
Co
llabo
rative Fram
ewo
rk
(ZOO
KEEP
ER
)
Offline Computing Online Computing OptimizationResource
Management
Spark Client
Hive Client
Hadoop Client
Yarn Standalone
Ap
plication
Gatew
ayIn
frastructure
Base
Platfo
rm
Storm
Storm Client
Universal Optimization Platform Client
(2) Technology Architecture
Framework of an Energy Big-data Platform02
![Page 26: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/26.jpg)
26
(3) Physical Architecture The layers of the Big-data platform are
deployed to the physical nodes, which
are connected as a whole through the
LAN, providing physical support for
applications.
Framework of an Energy Big-data Platform02
![Page 27: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/27.jpg)
27
(4) Core Technology and Function
Hardware Resources
Server #1 Server #2 Server #n
Data CollectionKafka
Data StorageHDFS Hive
Data Processing and Computing
Resource and Job Management
Yarn Standalone
Offline Computing Realtime Computing Optimization
Distributed Optimization
Universal Optimiz--ation Platform
Realtime Analysis
Spark StreamingStorm
Random Characteristic Analysis
Spark MllibMapreduceSu
rveillance System
Distrib
uted
Co
llabo
rative Framew
ork
(ZOO
KEEP
ER
)
Wind Power Data
Simulation
PV DataSimulation
Building Users Data Simulation
Factory Users Data Simulation
Framework of an Energy Big-data Platform02
![Page 28: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/28.jpg)
28
Multi-source Heterogeneous Data Fusion
![Page 29: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/29.jpg)
The large amount of data is a big challenge
Variety means the increasing complex of data types
Data itself is meaningless unless valuable knowledge
Refers to the speed requirement for processing data
29
Multi-source Heterogeneous Data Fusion03
Energy Big-data
Volume
Variety
Value
Velocity
![Page 30: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/30.jpg)
30
Multi-source Heterogeneous Data Fusion03
System heterogeneous
data are stored in different system or database
Semantic heterogeneous
two records of the same entity have different
express
Structured heterogeneousthe data are not only
structured data but also semi-structured data and unstructured data
Grammar heterogeneous
data have different formats such as units of data
Heterogeneous data
![Page 31: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/31.jpg)
1. Semantic heterogeneous data fusion
(1) Duplicate database records:
(2) Fusion methods
① Field Matching Method
② Sorted-neighborhood Method
31
Multi-source Heterogeneous Data Fusion03
name date of birth terminal name terminal address power
K.X. Huang 1994.10.14 Taoyuan 3166 2439.71
K.X. Huang 1994.10.14 Taoyuan 3166 439.71
![Page 32: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/32.jpg)
① Field Matching Method(FMM)
32
Multi-source Heterogeneous Data Fusion03
• Calculate the weight of each attribute of records
Weight
• Calculate the similaritybetween records
Similarity• Records
matching and detection
Detection
![Page 33: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/33.jpg)
33
𝑌 = 𝑦1, 𝑦2, ⋯ , 𝑦𝑛 , where 𝑦𝑖 = 𝑦𝑖1, 𝑦𝑖2, ⋯ , 𝑦𝑖𝑚 , 𝑖 ∈ 1,2,⋯ , 𝑛
Y means n records in a database with m attributes
𝑇𝑖𝑘 ∈ 𝑇𝑖1, 𝑇𝑖2, ⋯ , 𝑇𝑖𝑚 , 𝑖 ∈ 1,2,⋯ ,𝑁 , 𝑇𝑖𝑘 ≥ 1
𝑇 = 𝑇1, 𝑇2, ⋯ , 𝑇𝑚 , where 𝑇𝑘 = Τσ𝑖=1𝑁 𝑇𝑖𝑘 𝑁 , 𝑘 ∈ 1,2,⋯ ,𝑚
𝑇𝑘 → 𝑇𝑘′ , 𝑆 = 𝑚𝑎𝑥 𝑇1
′, 𝑇2′, ⋯ , 𝑇𝑚
′ , Convert 𝑇𝑘 to an integer 𝑇𝑘′
𝑊𝑘′ =
1
𝑆
𝑖=𝑇𝑘′
𝑆1
𝑖, 𝑘 ∈ 1,2,⋯ ,𝑚
𝑊𝑗 = ൘𝑊𝑗′
𝑘=1
𝑚
𝑊𝑘′ , 𝑗 ∈ 1,2,⋯ ,𝑚
33
Multi-source Heterogeneous Data Fusion03
a) Weight
![Page 34: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/34.jpg)
3434
b) Similarity𝑦𝑖𝑘 = 𝑦𝑖1, 𝑦𝑖2, ⋯ , 𝑦𝑖𝑝
𝑦𝑗𝑘 = 𝑦𝑗1, 𝑦𝑗2, ⋯ , 𝑦𝑗𝑞
𝑠𝑖𝑚 𝑦𝑖𝑘 , 𝑦𝑗𝑘 = ൘
𝑏=1
𝑞
𝑚𝑎𝑥 𝑠𝑐𝑜𝑟𝑒 𝑦𝑖𝑘𝑎, 𝑦𝑗𝑘𝑏 𝑝 , 𝑎 ∈ 1,2,⋯ , 𝑝
𝑠𝑖𝑚 𝑦𝑖 , 𝑦𝑗 =
𝑘=1
𝑚
𝑊𝑘 ∗ 𝑠𝑖𝑚(𝑦𝑖𝑘 , 𝑦𝑗𝑘)
c) Detection
𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 =𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑢𝑚𝑏𝑒𝑟𝑠
𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑛𝑢𝑚𝑏𝑒𝑟𝑠
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑢𝑚𝑏𝑒𝑟𝑠
𝑡𝑟𝑢𝑒 𝑛𝑢𝑚𝑏𝑒𝑟𝑠
34
Multi-source Heterogeneous Data Fusion03
the k-th attribute of the record 𝑦𝑖 has p strings
the k-th attribute of the record 𝑦𝑗 has q strings
![Page 35: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/35.jpg)
35
② Sorted-neighborhood Method(SNM)
……
……
35
Multi-source Heterogeneous Data Fusion03
Sort
Window
Detection
………………
current windownext window
![Page 36: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/36.jpg)
(3) Example
36
name date of birth terminal name terminal address power
K.X. Huang 1994.10.14 Taoyuan 3166 2439.71
K.X. Huang 1994.10.14 Taoyuan 3166 439.71
D.J. Zhang 1995.40.28 Taoyuan 3166 510.2
D.J. Zhang 1995.4.28 Taoyuan 3166 51.2
Multi-source Heterogeneous Data Fusion03
Detected number correct number completeness precision time(s)
FMM 237 230 0.9465 0.9705 3.555
SNM 238 233 0.9588 0.9790 0.249
Power Data of Taoyuan residential community
Results of two methods
![Page 37: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/37.jpg)
2. System heterogeneous data fusion
(1) Data are stored in different database
(2) Fusion method
Open Database Connectivity
(3) Example
37
Excel
ODBCOracle
databaseText
SQL
Multi-source Heterogeneous Data Fusion03
![Page 38: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/38.jpg)
38
Conclusions and Future Work
![Page 39: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/39.jpg)
Conclusions and Future Work
39
04
Conclusions
1. Integrated energy system has drawn widely attention
around the world. Researches from various of fields
greatly promote the development of Integrated energy
system.
2. Energy Big-data Platform is the foundation of an
Integrated Energy Platform, and is a significant research
field.
3. Methods for multi-source heterogeneous data fusion is
introduced, furthermore, establishment of an Energy Big-
data Platform framework is on going.
![Page 40: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/40.jpg)
Conclusions and Future Work
40
04
Future Work
An Integrated Energy Platform based on the Energy Big-data
Platform
Energy Big-data Platform
Integrated Energy Platform
Integrated Energy
Planning Platform
Integrated Energy
Operation Platform
Integrated Energy
Trading Platform
![Page 41: Research on the Framework and Data Fusion of an Energy Big-data … · 2017-08-25 · Research on the Framework and Data Fusion of an Energy Big-data Platform Gengfeng Li, Zhaohong](https://reader033.vdocuments.site/reader033/viewer/2022042416/5f318e9911c3d3375e749c4c/html5/thumbnails/41.jpg)
41
Thanks for Your Attention!