physics’guided,machine,learningclasp-research.engin.umich.edu/groups/admg/agu_2018/07_karpatn… ·...
TRANSCRIPT
![Page 1: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/1.jpg)
Physics-‐guided Machine Learning:Opportunities in Combining Physical Knowledge with
Data Science for Weather and Climate Sciences
Anuj KarpatneAssistant Professor, Computer Science
Virginia Tech
Torgersen Hall 3160Q,[email protected]
https://people.cs.vt.edu/karpatne/ 1
![Page 2: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/2.jpg)
Theory-‐based vs. Data Science Models
Contain knowledge gaps in describing certain processes
(turbulence, groundwater flow)
Use of Data
Use
of S
cien
tific
The
ory
Theo
ry-b
ased
Mod
els
Data Science Models
Theory-guidedData Science Models
Low High
High
Low
Theory-‐based M
odels
2
![Page 3: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/3.jpg)
Theory-‐based vs. Data Science Models
Contain knowledge gaps in describing certain processes
(turbulence, groundwater flow)
Require large number of representative samples
Take full advantage of data science methods without ignoring
the treasure of accumulated knowledge in scientific “theories”
Use of Data
Use
of S
cien
tific
The
ory
Theo
ry-b
ased
Mod
els
Data Science Models
Theory-guidedData Science Models
Low High
High
Low
Theory-‐based M
odels
Data Science Models
Theory-‐guided Data Science Models
(TGDS)1
1 Karpatne et al. “Theory-‐guided data science: A new paradigm for scientific discovery,” TKDE 2017
3
![Page 4: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/4.jpg)
• Motivation:
• 1-‐D Model of Temperature:Target: Temperature of water at every depth in a lake
Growth and survival of fisheries Harmful Algal Blooms Chemical Constituents:O2, C, N
TempShort-‐wave Radiation,Long-‐wave Radiation,Air Temperature,
Relative Humidity,Wind Speed,
Rain, …
Input Drivers (observed via meteorology):
Illustrative Case Study:Modeling Lake Quality (Temperature)
4
![Page 5: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/5.jpg)
Modeling Temperature using Physics-‐based Models
• Standard Paradigm for Scientific Computing
PHY
𝒛"𝜽
𝐹, 𝐺
𝒙"
𝑦"
solar radiation, air temp., …
Input Drivers:
temperatureTarget:
Hidden States
water clarity, salinity, …
Parameters:
Physics-‐based Equations
𝜕𝜌𝜕𝑡 = −𝛻. 𝜌𝐮
𝜕𝜌𝐮𝜕𝑡 = −𝛻.
1𝜌 𝜌𝐮 ⨂ 𝜌𝐮 +𝑝𝐈 + 𝜌𝐠
𝜕𝐸𝜕𝑡 = −𝛻.
1𝜌 𝐸 +𝑝 𝜌𝐮 + 𝐮. 𝜌𝐠
Conservation of mass, momentum, energy
General Lake Model1 (GLM)
1Hipsey et al., 2014 5
![Page 6: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/6.jpg)
Limitations of Physics-‐based Models• Unknown parameters (𝜽) need to be “calibrated”
– Computationally Expensive– Easy to overfit: large number of parameter choices, small
number of samplesPHY
𝒛"𝜽𝐹, 𝐺
𝒙"
𝑦"
Errors
𝜃:, % water clarity
𝜃 ;, %
salinity
𝜃< to 𝜃=
For everylake Number of
parameter choices: ~ 𝑂(2=)
6
![Page 7: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/7.jpg)
Limitations of Physics-‐based Models• Unknown parameters (𝜽) need to be “calibrated”
– Computationally Expensive– Easy to overfit: large number of parameter choices, small
number of samples
• Incomplete or missing physics (𝑭, 𝑮)– Physics-‐based models often use approximate forms to meet
“scale-‐accuracy” trade-‐off– Results in inherent model bias
PHY
𝒛"
𝒙"
𝑦"
𝜽
𝑭, 𝑮
Lake bathymetry often simplified using approximate forms
7
![Page 8: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/8.jpg)
“Black-‐box” Data Science Models• An alternative to physics-‐based modeling?
PHY
𝒛"𝜽
𝐹, 𝐺
𝒙"
𝑦"
DS
𝒄"LSTM Gates,
Attention,...
𝒙"
𝑦"
• Requires calibration of model parameters
• Incomplete physics
Support Vector Machine
Deep Learning
…
Choice of model family not governed by physics
• Require lots of data• Ignorant of physical
knowledge
8
![Page 9: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/9.jpg)
Hybrid-‐Physics-‐Data Models• A paradigm shift in data-‐intensive scientific modeling
PHY
𝒛"𝜽
𝐹, 𝐺
𝒙"
𝑦"
DS
𝒄"LSTM Gates,
Attention,...
𝒙"
𝑦"
• Requires calibration of model parameters
• Incomplete physics
• Require lots of data• Incoherent with
physical knowledge
PHY
𝒛"𝜽
𝐹, 𝐺
𝒙"
𝑦"
DS
𝒄"LSTM Gates,
Attention,...
Overcomes complementary weaknesses of PHY and DS by combining them together
9
![Page 10: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/10.jpg)
!…
"#"$ "%
A Generic Framework forHybrid-‐Physics-‐Data (HPD) Modeling:
Input Drivers(Solar Radiation,Air Temperature,Relative Humidity,Wind Speed, …)
Black-‐Box Model
PHY(GLM)
𝒀𝐏𝐇𝐘(Temperature)
𝒀JKLM(Temperature)
10
![Page 11: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/11.jpg)
!…
"#"$ "%
A Generic Framework forHybrid-‐Physics-‐Data (HPD) Modeling:
Input Drivers(Solar Radiation,Air Temperature,Relative Humidity,Wind Speed, …)
Hybrid-‐Physics-‐Data Model
PHY(GLM)
𝒀𝐏𝐇𝐘(Temperature)
𝒀JKLM(Temperature)
Challenge with training AI methods:• 𝑌JKLM may violate physical relationships
b/w 𝑌 and other variables11
![Page 12: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/12.jpg)
DensityDe
pth
Physical Relationships of Temperature
-10 -5 0 4 10 15 20 25 30
996
997
998
999
1000
Denser water is at higher depthTemperature directly related
to density of water
Use physics-‐based loss functions:• Measure violations of physical relationships b/w 𝑌JKLM and other variables• Does not require labeled data!
12
![Page 13: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/13.jpg)
Physics-‐guided Neural Network (PGNN)1
!…
"#"$ "%
Labeled Data
Drivers+
Unlabeled Data
Drivers+
Training Loss
𝑌"KOL,𝑌JKLM+
𝜆 R 𝑊
Physics-‐based Loss 𝑌JKLM
𝑌JKLM
𝑌JKLM
𝒀𝐏𝐇𝐘
𝒀𝐏𝐇𝐘
Objective Function ∶=Training Loss 𝑌"KOL,𝑌JKLM + 𝜆 R 𝑊 +
𝜆TUV Physics-‐based Loss 𝑌JKLM
1Karpatne et al., “Physics-‐guided neural networks (PGNN): An Application in Lake Temperature Modeling,” arXiv: 1710.11431, 2017. 13
![Page 14: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/14.jpg)
Experimental Results
Lake Mille Lacs, MN
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Te
st R
MS
E
14
![Page 15: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/15.jpg)
Experimental Results
Lake Mille Lacs, MN
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Te
st R
MS
E
PHY
15
![Page 16: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/16.jpg)
Experimental Results
Lake Mille Lacs, MN
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Te
st R
MS
E
PHY
NN
16
![Page 17: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/17.jpg)
Experimental Results
Lake Mille Lacs, MN
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Te
st R
MS
E
PHY
NN
PGNN0
17
![Page 18: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/18.jpg)
Experimental Results
Lake Mille Lacs, MN
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Te
st R
MS
E
PHY
NN
PGNN0PGNN
18
![Page 19: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/19.jpg)
Experimental Results
Lake Mille Lacs, MN
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Te
st R
MS
E
PHY
NN
PGNN0PGNN
Lake Mendota, Wisconsin
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
1.6
1.8
2
2.2
2.4
2.6
2.8
3
Te
st R
MS
E
PHY
NN
PGNN0PGNN
PGNN ensures Generalizability + Physical Consistency
19
![Page 20: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/20.jpg)
Analyzing Physical Inconsistency
20
Lake Mendota, Wisconsin
0 0.2 0.4 0.6 0.8 1
Physical Inconsistency
1.6
1.8
2
2.2
2.4
2.6
2.8
3
Te
st R
MS
E
PHY
NN
PGNN0PGNN
998.4 998.6 998.8 999 999.2 999.4 999.6 999.8 1000 1000.2
0
5
10
15
20
25
Density
Depth
27−May−2003
Obs
PGNN
PGNN0
NN
PHY
Includephysical consistency as another evaluation criterion, going beyond standard metrics for test error
![Page 21: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/21.jpg)
Alternate Ways of Incorporating Physics in ML
• Other Physics-‐based Loss Functions:
• Pre-‐training ML models using Physics
– Train ML methods using physical simulations– Fine-‐tune using observational data 21
Density
Depth
-10 -5 0 4 10 15 20 25 30
996
997
998
999
1000
Physical)Constraint:)Denser&water&is&at&higher&depthPhysical&relationship&b/w&
temperature&and&density
Conservation of Energy in Recurrent Neural Networks
Depth-‐Density Constraint in Multi-‐layer Perceptron Network
dUt/dt = RSW (1-‐ αSW) + RLWin(1-‐ αLW) -‐ RLWout -‐ E– H
![Page 22: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/22.jpg)
Physics-‐guided Recurrent Neural Networks (PGRNN)
22
Use of Data
Use
of S
cien
tific
The
ory
Theo
ry-b
ased
Mod
els
Data Science Models
Theory-guidedData Science Models
Low High
High
Low
Theory'based,M
odels
Data,Science,Models
Theory'guided,Data,Science,Models
(TGDS)1
Number of training samples
PGRNN
AGU 2018 Poster IN41D-‐0872: Read et al., Process-Guided Data-Driven modeling of water temperature: Anchoring predictions with thermodynamic constraints in the Big Data era
Jia et al., Physics Guided RNNs for Modeling Dynamical Systems: A Case Study in Simulating Lake Temperature Profiles, arxiv: 1810.13075, 2018.
![Page 23: Physics’guided,Machine,Learningclasp-research.engin.umich.edu/groups/admg/AGU_2018/07_Karpatn… · Physics’guided,Machine,Learning: Opportunities+in+Combining+Physical+Knowledge+with+](https://reader034.vdocuments.site/reader034/viewer/2022052014/602ac88952834b1be5489466/html5/thumbnails/23.jpg)
Summary and Future Directions• Research themes in TGDS
– Physics-‐guided Learning of ML models• Loss Functions, Priors, Constraints, …
– Physics-‐guided Design of ML models• Architecture of NN models, LSTM connections, activation functions
– Pre-‐training ML models using Physical Simulations• Train Using Combination of Physical Simulations + Observations
– Building Hybrid-‐Physics-‐Data Models• Rectify Model Outputs, Replace Model Components using ML
– Inferring Parameters/States in Physics-‐based Models• Parameter Calibration, Data Assimilation
23