parallel adaptive pde simulation with libmeshroystgnr/cleaned.pdf · introduction background...
TRANSCRIPT
![Page 1: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/1.jpg)
Parallel Adaptive PDE Simulation with libMesh
Roy H. Stogner
University of Texas at Austin
March 2, 2016
R. H. Stogner libMesh PDE Simulation March 2, 2016 1 / 124
![Page 2: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/2.jpg)
Outline
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 2 / 124
![Page 3: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/3.jpg)
Introduction
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 3 / 124
![Page 4: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/4.jpg)
Introduction
Background
• Modern simulation software is complex:
• Implicit numerical methods
• Massively parallel computers
• Adaptive methods
• Multiple, coupled physical processes
• There are a host of existing software libraries that excel at treating
various aspects of this complexity.
• Leveraging existing software whenever possible is the most efficient
way to manage this complexity.
R. H. Stogner libMesh PDE Simulation March 2, 2016 4 / 124
![Page 5: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/5.jpg)
Introduction
Background
• Modern simulation software is multidisciplinary:
• Physical Sciences
• Engineering
• Computer Science
• Applied Mathematics
• . . .
• It is not reasonable to expect a single person to have all the
necessary skills for developing & implementing high-performance
numerical algorithms on modern computing architectures.
• Teaming is a prerequisite for success.
R. H. Stogner libMesh PDE Simulation March 2, 2016 5 / 124
![Page 6: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/6.jpg)
Introduction
Background
• A large class of problems are amenable to mesh based simulation
techniques.
• Consider some of the major components such a simulation:
R. H. Stogner libMesh PDE Simulation March 2, 2016 6 / 124
![Page 7: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/7.jpg)
Introduction
Background
• A large class of problems are amenable to mesh based simulation
techniques.
• Consider some of the major components such a simulation:
1 Read the mesh from file
2 Initialize data structures
3 Construct a discrete representation of the governing equations
4 Solve the discrete system
5 Write out results
6 Optionally estimate error, refine the mesh, and repeat
R. H. Stogner libMesh PDE Simulation March 2, 2016 6 / 124
![Page 8: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/8.jpg)
Introduction
Background
• A large class of problems are amenable to mesh based simulation
techniques.
• Consider some of the major components such a simulation:
1 Read the mesh from file
2 Initialize data structures
3 Construct a discrete representation of the governing equations
4 Solve the discrete system
5 Write out results
6 Optionally estimate error, refine the mesh, and repeat
• With the exception of step 3, the rest is independent of the class of
problems being solved.
R. H. Stogner libMesh PDE Simulation March 2, 2016 6 / 124
![Page 9: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/9.jpg)
Introduction
Background
• A large class of problems are amenable to mesh based simulation
techniques.
• Consider some of the major components such a simulation:
1 Read the mesh from file
2 Initialize data structures
3 Construct a discrete representation of the governing equations
4 Solve the discrete system
5 Write out results
6 Optionally estimate error, refine the mesh, and repeat
• With the exception of step 3, the rest is independent of the class of
problems being solved.
• This allows the major components of such a simulation to be
abstracted & implemented in a reusable software library.
R. H. Stogner libMesh PDE Simulation March 2, 2016 6 / 124
![Page 10: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/10.jpg)
Introduction The libMesh Software Library
The libMesh Software Library
• In 2002, the libMesh library began with these ideas in mind.
• Primary goal is to provide data structures and algorithms that can beshared by disparate physical applications, that may need somecombination of
• Implicit numerical methods
• Adaptive mesh refinement techniques
• Parallel computing
• Unifying theme: mesh-based simulation of partial differential
equations (PDEs).
R. H. Stogner libMesh PDE Simulation March 2, 2016 7 / 124
![Page 11: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/11.jpg)
Introduction Software Reusability
The libMesh Software Library
Key Point
• The libMesh library is designed to be used by students,
researchers, scientists, and engineers as a tool for developing
simulation codes or as a tool for rapidly implementing a numerical
method.
• libMesh is not an application code.
• It does not “solve problem XYZ.”
• It can be used to help you develop an application to solve problem
XYZ, and to do so quickly with advanced numerical algorithms on
high-performance computing platforms.
R. H. Stogner libMesh PDE Simulation March 2, 2016 8 / 124
![Page 12: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/12.jpg)
Introduction Software Reusability
libMesh Community
Scope
• Free, Open source
• LGPL2 for core
• 35 Ph.D. theses, 393
papers (58 in 2015)
• ∼ 10 current developers
• 160− 240 current users?
Jan 2010 Jul 2011 Oct 2013 Feb 201650
100
150
200
250
300libmesh-devellibmesh-users
LibMesh Mailing List Membership Size
Challenges
• Radically different application types
• Widely dispersed core developers
• INL, UT-Austin, U.Buffalo, JSC, MIT,
Harvard, Argonne
• OSS, commercial, private applications
R. H. Stogner libMesh PDE Simulation March 2, 2016 9 / 124
![Page 13: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/13.jpg)
Introduction Software Reusability
PetscMetis
Mpich
Laspack
LibMesh
STL
RBM
TumorAngiogenesis
CompressibleNS
DD
• Foundational
(typically optional)
library access via
LibMesh’s “roots”.
• Application
“branches” built off
the library “trunk”.
• Additional
middleware layers
(e.g. Akselos,
GRINS, MOOSE) for
more complex
applications
R. H. Stogner libMesh PDE Simulation March 2, 2016 10 / 124
![Page 14: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/14.jpg)
Introduction Software Reusability
Software Reusability
• At the inception of libMesh in 2002, there were many high-qualitysoftware libraries that implemented some aspect of the end-to-endPDE simulation process:
• Parallel linear algebra
• Partitioning algorithms for domain decomposition
• Visualization formats
• . . .
• A design goal of libMesh has always been to provide flexible &
extensible interfaces to existing software whenever possible.
• We implement the “glue” to these pieces, as well as what we viewedas the missing infrastructure:
• Flexible data structures for the discretization of spatial domains and
systems of PDEs posed on these domains.
R. H. Stogner libMesh PDE Simulation March 2, 2016 11 / 124
![Page 15: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/15.jpg)
Library Design
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 12 / 124
![Page 16: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/16.jpg)
Library Design
Geometric Element Classes
Elem
#_nodes: Node **
#_neighbors: Elem **
#_parent: Elem *
#_children: Elem **
#_*flag: RefinementState
#_p_level: unsigned char
#_subdomain_id: unsigned char
+n_faces,sides,vertices,edges,children(): unsigned int
+centroid(): Point
+hmin,hmax(): Real
NodeElem FaceEdge Cell InfQuad InfCell
Prism Hex Pyramid Tet
Hex8 Hex20 Hex27
DofObject
-_n_systems: unsigned char
-_n_vars: unsigned char *
-_n_comp: unsigned char **
-_dof_ids: unsigned int **
-_id: unsigned int
-_processor_id: unsigned short int
Node
• Abstract interface gives
mesh topology
• Concrete instantiations of
mesh geometry
• Hides element type from
most applications
• Runtime polymorphism
allows mixed element types,
dimensions
• Base class data arrays allow
more optimization, inlining
R. H. Stogner libMesh PDE Simulation March 2, 2016 13 / 124
![Page 17: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/17.jpg)
Library Design
Linear Algebra
libMesh::SparseMatrix< T >
libMesh::EigenSparseMatrix< T >
libMesh::EpetraMatrix< T >
libMesh::LaspackMatrix< T >
libMesh::PetscMatrix< T >
libMesh::ReferenceCountedObject< SparseMatrix< T > >libMesh::ReferenceCounter
R. H. Stogner libMesh PDE Simulation March 2, 2016 14 / 124
![Page 18: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/18.jpg)
Library Design
I/O formats
libMesh::MeshInput< MeshBase >
libMesh::AbaqusIO
libMesh::ExodusII_IO
libMesh::GmshIO
libMesh::GMVIO
libMesh::LegacyXdrIO
libMesh::Mat labIO
libMesh::Nemesis_IO
libMesh::OFFIO
libMesh::TetGenIO
libMesh::UCDIO
libMesh::UNVIO
libMesh::VTKIO
libMesh::XdrIO
libMesh::MeshOutput< MeshBase >
libMesh::DivaIO
libMesh::Ensight IO
libMesh::FroIO
libMesh::GnuPlot IO
libMesh::MEDITIO
libMesh::Postscript IO
libMesh::Tecplot IO
R. H. Stogner libMesh PDE Simulation March 2, 2016 15 / 124
![Page 19: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/19.jpg)
Library Design
Domain Partitioning
libMesh::Part it ioner
libMesh::CentroidPart it ioner
libMesh::LinearPart it ioner
libMesh::Met isPart it ioner
libMesh::Parmet isPart it ioner
libMesh::SFCPart it ioner
libMesh::HilbertSFCPart it ioner
libMesh::MortonSFCPart it ioner
R. H. Stogner libMesh PDE Simulation March 2, 2016 16 / 124
![Page 20: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/20.jpg)
Library Design
Mesh Data Structures
MeshBase
+boundary_info: AutoPtr<BoundaryInfo>
+elem(unsigned int)
+node(unsigned int)
+elements_begin/end()
+nodes_begin/end()
+active_local_elements_begin/end()
+read()
+write()
ParallelMesh
#_elements: mapvector<Elem*>
+allgather()
+delete_remote_elements()
UnstructuredMesh
+find_neighbors()
+read/write()
+add/delete_elem()
SerialMesh
#_elements: vector<Elem*>
+elements_begin()
• MeshBase gives node
or element iterators, all
or active, global or local
• SerialMesh or
ParallelMesh
manages synchronized
or distributed data
R. H. Stogner libMesh PDE Simulation March 2, 2016 17 / 124
![Page 21: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/21.jpg)
Library Design
Discretization: Finite Elements
libMesh::FEAbstract
libMesh::FEGenericBase< FEOutputType< T >::type >
libMesh::FEGenericBase< Real >
libMesh::FEGenericBase< OutputType >
libMesh::ReferenceCountedObject< FEAbstract >libMesh::ReferenceCounter
libMesh::FE< Dim, CLOUGH >
libMesh::FE< Dim, HERMITE >
libMesh::FE< Dim, HIERARCHIC >
libMesh::FE< Dim, L2_HIERARCHIC >
libMesh::FE< Dim, L2_LAGRANGE >
libMesh::FE< Dim, LAGRANGE >
libMesh::FE< Dim, LAGRANGE_VEC >
libMesh::FE< Dim, MONOMIAL >
libMesh::FE< Dim, NEDELEC_ONE >
libMesh::FE< Dim, SCALAR >
libMesh::FE< Dim, XYZ >
libMesh::FE< Dim, T >
libMesh::FEClough< Dim >
libMesh::FEHermite< Dim >
libMesh::FEHierarchic< Dim >
libMesh::FEL2Hierarchic< Dim >
libMesh::FEL2Lagrange< Dim >
libMesh::FELagrange< Dim >
libMesh::FELagrangeVec< Dim >
libMesh::FEMonomial< Dim >
libMesh::FENedelecOne< Dim >
libMesh::FEScalar< Dim >
libMesh::FEXYZ< Dim >
libMesh::InfFE< Real >
R. H. Stogner libMesh PDE Simulation March 2, 2016 18 / 124
![Page 22: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/22.jpg)
Library Design
Algorithms: Error Estimation
libMesh::ErrorEst imator
libMesh::AdjointRefinementEst imator
libMesh::AdjointResidualErrorEst imator
libMesh::ExactErrorEst imator
libMesh::JumpErrorEst imator
libMesh::PatchRecoveryErrorEst imator
libMesh::UniformRefinementEst imator
libMesh::Discont inuityMeasure
libMesh::KellyErrorEst imator
libMesh::LaplacianErrorEst imator
libMesh::WeightedPatchRecoveryErrorEst imator
R. H. Stogner libMesh PDE Simulation March 2, 2016 19 / 124
![Page 23: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/23.jpg)
Application Examples
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 20 / 124
![Page 24: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/24.jpg)
Application Examples Application Results
FIN-S: Fully Implicit Navier-Stokes
• Kirk, Stogner, Bauman,
Oliver, Computers & Fluids,
2014
• Application for high-speed
(including reentry)
compressible flows in
thermochemical
nonequilibrium
• Including AMR capabilities
• Fully implicit coupling with
surface response models
R. H. Stogner libMesh PDE Simulation March 2, 2016 21 / 124
![Page 25: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/25.jpg)
Application Examples Application Results
FIN-S: Fully Implicit Navier-Stokes
• Kirk, Stogner, Bauman,
Oliver, Computers & Fluids,
2014
• Application for high-speed
(including reentry)
compressible flows in
thermochemical
nonequilibrium
• Including AMR capabilities
• Fully implicit coupling with
surface response models
R. H. Stogner libMesh PDE Simulation March 2, 2016 21 / 124
![Page 26: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/26.jpg)
Application Examples Application Results
FIN-S: Fully Implicit Navier-Stokes
• Kirk, Stogner, Bauman,
Oliver, Computers & Fluids,
2014
• Application for high-speed
(including reentry)
compressible flows in
thermochemical
nonequilibrium
• Including AMR capabilities
• Fully implicit coupling with
surface response models
R. H. Stogner libMesh PDE Simulation March 2, 2016 21 / 124
![Page 27: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/27.jpg)
Application Examples Application Results
FIN-S: Fully Implicit Navier-Stokes
• Kirk, Stogner, Bauman,
Oliver, Computers & Fluids,
2014
• Application for high-speed
(including reentry)
compressible flows in
thermochemical
nonequilibrium
• Including AMR capabilities
• Fully implicit coupling with
surface response models
R. H. Stogner libMesh PDE Simulation March 2, 2016 21 / 124
![Page 28: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/28.jpg)
Application Examples Application Results
FIN-S: Fully Implicit Navier-Stokes
• Kirk, Stogner, Bauman,
Oliver, Computers & Fluids,
2014
• Application for high-speed
(including reentry)
compressible flows in
thermochemical
nonequilibrium
• Including AMR capabilities
• Fully implicit coupling with
surface response models
R. H. Stogner libMesh PDE Simulation March 2, 2016 21 / 124
![Page 29: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/29.jpg)
Application Examples Application Results
GRINS
https://github.com/grinsfem/grins
• Multiphysics FEM platform built on libMesh
• Modular structure for “Physics”, solvers, QoIs, etc.
• Key feature: automatically enabled discrete adjoints (AMR,
sensitivities)
R. H. Stogner libMesh PDE Simulation March 2, 2016 22 / 124
![Page 30: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/30.jpg)
Application Examples Application Results
GRINS
https://github.com/grinsfem/grins
• Multiphysics FEM platform built on libMesh
• Modular structure for “Physics”, solvers, QoIs, etc.
• Key feature: automatically enabled discrete adjoints (AMR,
sensitivities)
R. H. Stogner libMesh PDE Simulation March 2, 2016 22 / 124
![Page 31: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/31.jpg)
Application Examples Application Results
GRINS
https://github.com/grinsfem/grins
• Multiphysics FEM platform built on libMesh
• Modular structure for “Physics”, solvers, QoIs, etc.
• Key feature: automatically enabled discrete adjoints (AMR,
sensitivities)
R. H. Stogner libMesh PDE Simulation March 2, 2016 22 / 124
![Page 32: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/32.jpg)
Application Examples Application Results
GRINS
https://github.com/grinsfem/grins
• Multiphysics FEM platform built on libMesh
• Modular structure for “Physics”, solvers, QoIs, etc.
• Key feature: automatically enabled discrete adjoints (AMR,
sensitivities)
R. H. Stogner libMesh PDE Simulation March 2, 2016 22 / 124
![Page 33: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/33.jpg)
Application Examples Application Results
GRINS
https://github.com/grinsfem/grins
• Multiphysics FEM platform built on libMesh
• Modular structure for “Physics”, solvers, QoIs, etc.
• Key feature: automatically enabled discrete adjoints (AMR,
sensitivities)
Courtesy Nick Malaya, UT
Austin
R. H. Stogner libMesh PDE Simulation March 2, 2016 22 / 124
![Page 34: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/34.jpg)
Application Examples Application Results
The MOOSE Framework - Gaston et al., INL
MOOSE – Multiphysics Object Oriented Simulation Environment
• A framework for solving computational nuclear engineering problems in a well planned, managed, and coordinated way – Leveraged across multiple programs
• Designed to significantly reduce the expense and time required to develop new applications
• Designed to develop analysis tools – Uses very robust solution methods
– Designed to be easily extended and maintained
– Efficient on both a few and many processors
• Currently supports ~7 applications which are developed and used by ~20 scientists.
R. H. Stogner libMesh PDE Simulation March 2, 2016 23 / 124
![Page 35: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/35.jpg)
Application Examples Application Results
The MOOSE Framework - Gaston et al., INL
BISON fuel performance
• LWR, Triso, and TRU fuel performance code
• Parallel 1D-3D thermomechanics code
• Thermal, mechanical, and chemical models for FCI
• Constituent redistribution
• Material, fission product swelling, fission gas release models
• Mesoscale-informed material models
R. H. Stogner libMesh PDE Simulation March 2, 2016 24 / 124
![Page 36: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/36.jpg)
Application Examples Application Results
Coupled Thermal/Solid Mechanics
R. H. Stogner libMesh PDE Simulation March 2, 2016 25 / 124
![Page 37: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/37.jpg)
Application Examples Application Results
Coupled Thermal/Solid Mechanics
R. H. Stogner libMesh PDE Simulation March 2, 2016 25 / 124
![Page 38: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/38.jpg)
Application Examples Application Results
FALCON - Fracturing And Liquid CONservation
R. H. Stogner libMesh PDE Simulation March 2, 2016 26 / 124
![Page 39: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/39.jpg)
Application Examples Application Results
FALCON - Fracturing And Liquid CONservation
R. H. Stogner libMesh PDE Simulation March 2, 2016 26 / 124
![Page 40: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/40.jpg)
Application Examples Application Results
FALCON - Fracturing And Liquid CONservation
R. H. Stogner libMesh PDE Simulation March 2, 2016 27 / 124
![Page 41: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/41.jpg)
Application Examples Application Results
FALCON - Fracturing And Liquid CONservation
R. H. Stogner libMesh PDE Simulation March 2, 2016 27 / 124
![Page 42: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/42.jpg)
Software Installation & Ecosystem
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 28 / 124
![Page 43: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/43.jpg)
Software Installation & Ecosystem Important Websites
Important Websites
• Primary website
• Revision Control & Collaboration with GitHub
• Continuous Integration with MooseBuild (from INL)
R. H. Stogner libMesh PDE Simulation March 2, 2016 29 / 124
![Page 44: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/44.jpg)
Software Installation & Ecosystem Important Websites
http://libmesh.github.io
R. H. Stogner libMesh PDE Simulation March 2, 2016 30 / 124
![Page 45: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/45.jpg)
Software Installation & Ecosystem Important Websites
http://github.com/libMesh/libmesh
R. H. Stogner libMesh PDE Simulation March 2, 2016 31 / 124
![Page 46: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/46.jpg)
Software Installation & Ecosystem Important Websites
MooseBuild Testing Integrated with GitHub
R. H. Stogner libMesh PDE Simulation March 2, 2016 32 / 124
![Page 47: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/47.jpg)
Software Installation & Ecosystem Compiling the library
Building libMesh
R. H. Stogner libMesh PDE Simulation March 2, 2016 33 / 124
![Page 48: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/48.jpg)
Software Installation & Ecosystem Compiling the library
Getting the libMesh Source
• Blessed, Stable releases:
Download prepackaged releases from
http://github.com/libMesh/libmesh/releases
• Development tree:
Grab the latest source tree from GitHub:
$ git clone git://github.com/libMesh/libmesh.git
R. H. Stogner libMesh PDE Simulation March 2, 2016 34 / 124
![Page 49: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/49.jpg)
Software Installation & Ecosystem Compiling the library
libMesh Suggested Dependencies
• MPI is of course required for shared-memory parallelism.
• Out of the box, libMesh will build with support for serial linear
systems.
• Highly recommended you first install PETSc and/or Trilinos,
which libMesh uses for solving linear systems in parallel.
• Other recommended, optional packages are:
• SLEPc: eigenvalue support on top of PETSc.
• Intel’s Threading Building Blocks for shared-memory multithreading.
R. H. Stogner libMesh PDE Simulation March 2, 2016 35 / 124
![Page 50: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/50.jpg)
Software Installation & Ecosystem Compiling the library
Building libMesh from source
Unpack, Configure, Build, Install, & Test
# unpack the distribution
$ tar jxf libmesh-0.9.5.tar.bz2 && cd libmesh-0.9.5
# configure, install into the current directory
$ ./configure --prefix=$PWD/install
# build & install
$ make -j 4 && make -j 4 install
# run all the examples, but only the optimized flavor
$ make -j 4 check METHODS=opt
R. H. Stogner libMesh PDE Simulation March 2, 2016 36 / 124
![Page 51: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/51.jpg)
A Generic Boundary Value Problem
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 37 / 124
![Page 52: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/52.jpg)
A Generic Boundary Value Problem A Generic BVP
Typical Boundary Value Problem
• Common BVP components:
M∂u
∂t= F(u) ∈ Ω ⊂ R
n
G(u) = 0 ∈ Ω
u = uD ∈ ∂ΩD
N(u) = 0 ∈ ∂ΩN
u(x, 0) = u0(x)
• Less common components:
• Moving domain Ω(t), Ω(u, t)• Multi-dimensional manifolds
• Self-overlapping, contact
• Acceleration ∂2u/∂t2
• Integro-differential equations
Ω
∂ΩD
∂ΩN
R. H. Stogner libMesh PDE Simulation March 2, 2016 38 / 124
![Page 53: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/53.jpg)
A Generic Boundary Value Problem A Generic BVP
• Associated to the problem
domain Ω is a libMesh data
structure called a Mesh
• A Mesh is essentially a
collection of finite elements
Ωh :=⋃
e
Ωe
Ωh
R. H. Stogner libMesh PDE Simulation March 2, 2016 39 / 124
![Page 54: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/54.jpg)
A Generic Boundary Value Problem A Generic BVP
• Associated to the problem
domain Ω is a libMesh data
structure called a Mesh
• A Mesh is essentially a
collection of finite elements
Ωh :=⋃
e
Ωe
Ωh
• libMesh provides some simple structured mesh generation
routines, interfaces to Triangle and TetGen, and supports a rich set of
input file formats.
R. H. Stogner libMesh PDE Simulation March 2, 2016 39 / 124
![Page 55: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/55.jpg)
Key Data Structures
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 40 / 124
![Page 56: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/56.jpg)
Key Data Structures The Mesh Class
Operations on Objects in the Mesh
• From a Mesh it is trivial to access ranges of objects of interest
through iterators.
• Iterators are simply a mechanism for accessing a range of objects.
• libMesh makes extensive use of predictated iterators to access, forexample,
• All elements in the mesh.
• The “active” elements in the mesh assigned to the local processor in a
parallel simulation.
• The nodes in the mesh.
R. H. Stogner libMesh PDE Simulation March 2, 2016 41 / 124
![Page 57: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/57.jpg)
Key Data Structures The Mesh Class
Mesh Iteratorsvoid foo (const MeshBase &mesh)
// Now we will loop over all the elements in the mesh that
// live on the local processor. We will compute the element
// matrix and right-hand-side contribution. Since the mesh
// may be refined we want to only consider the ACTIVE elements,
// hence we use a variant of the \p active_elem_iterator.
MeshBase::const_element_iterator
el = mesh.active_local_elements_begin();
const MeshBase::const_element_iterator
end_el = mesh.active_local_elements_end();
for ( ; el != end_el; ++el)
// Store a pointer to the element we are currently
// working on. This allows for nicer syntax later.
const Elem* elem = *el;
...
...
R. H. Stogner libMesh PDE Simulation March 2, 2016 42 / 124
![Page 58: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/58.jpg)
Key Data Structures The Mesh Class
Mesh Iteratorsvoid foo (const MeshBase &mesh)
// We will now loop over all nodes.
MeshBase::const_node_iterator node_it = mesh.nodes_begin();
const MeshBase::const_node_iterator node_end = mesh.nodes_end();
for ( ; node_it != node_end; ++node_it)
// the current node pointer
const Node* node = *node_it;
...
...
R. H. Stogner libMesh PDE Simulation March 2, 2016 43 / 124
![Page 59: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/59.jpg)
Key Data Structures The EquationSystems Class
EquationSystems
• The Mesh is a discrete representation of the geometry for a problem.
• For a given Mesh, there can be an EquationSystems object,which represents one or more coupled system of equations posed onthe Mesh.
• There is only one EquationSystems object per Mesh object.
• The EquationSystems object can hold many System objects, each
representing a logical system of equations.
• High-level operations such as solution input/output is usually handled
at the EquationSystems level.
R. H. Stogner libMesh PDE Simulation March 2, 2016 44 / 124
![Page 60: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/60.jpg)
Key Data Structures The EquationSystems Class
EquationSystems...
// Create an equation systems object. This object can contain
// multiple systems of different flavors for solving loosely coupled
// physics. Each system can contain multiple variables of different
// approximation orders. The EquationSystems object needs a
// reference to the mesh object, so the order of construction here
// is important.
EquationSystems equation_systems (mesh);
// Now we declare the system and its variables. We begin by adding
// a "TransientLinearImplicitSystem" to the EquationSystems object,
// and we give it the name "Simple System".
equation_systems.add_system<TransientLinearImplicitSystem> ("Simple System");
// Adds the variable "u" to "Simple System". "u" will be
// approximated using first-order approximation.
equation_systems.get_system("Simple System").add_variable("u", FIRST);
// Next we’ll by add an "ExplicitSystem" to the EquationSystems
// object, and we give it the name "Complex System".
equation_systems.add_system<ExplicitSystem> ("Complex System");
// Give "Complex System" three variables -- each with a different
// approximation order. Variables "c" and "T" will use first-order
// Lagrange approximation, while variable "dv" will use a
// second-order discontinuous approximation space.
equation_systems.get_system("Complex System").add_variable("c", FIRST);
equation_systems.get_system("Complex System").add_variable("T", FIRST);
equation_systems.get_system("Complex System").add_variable("dv", SECOND, MONOMIAL);
// Initialize the data structures for the equation system.
equation_systems.init();
// Prints information about the system to the screen.
equation_systems.print_info();
...
R. H. Stogner libMesh PDE Simulation March 2, 2016 45 / 124
![Page 61: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/61.jpg)
Key Data Structures The Elem Class
Elements
• The Elem base class defines a geometric element in libMesh.
• An Elem is defined by Nodes, Edges (2D,3D) and Faces (3D).
• An Elem is sufficiently rich that in many cases it is the only argument
required to provide to a function.
R. H. Stogner libMesh PDE Simulation March 2, 2016 46 / 124
![Page 62: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/62.jpg)
Key Data Structures The Elem Class
Elements// access each Node on the element
for (unsigned int n=0; n<elem->n_nodes(); n++)
const Node *node = elem->get_node(n);
// get a user-specified material property, based on
// the subdomain the element belongs to
const Real k_diff = my_matprop_func (elem->subdomain_id(), *node);
...
// Perform some operation for elements on the boundary
for (unsigned int side=0; side<elem->n_sides(); side++)
// Every element knows its neigbor. If it has no neighbor,
// then it lies on a physical boundary.
if (elem->neighbor(side) == NULL)
// Construct the side as a lower dimensional element
AutoPtr<Elem> elem_side (elem->build_side(side));
...
...
R. H. Stogner libMesh PDE Simulation March 2, 2016 47 / 124
![Page 63: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/63.jpg)
Key Data Structures The Node Class
Nodes
• Nodes define spatial locations in arbitrary dimensions.
• Logically, a Node is a point in N-space plus metadata:
• Global ID.
• Processor ownership.
• Degree of freedom indexing data.
R. H. Stogner libMesh PDE Simulation March 2, 2016 48 / 124
![Page 64: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/64.jpg)
Key Data Structures The Node Class
Nodes
// loop over a range and determine the bounding box
void bounding_box(const ConstNodeRange &range)
...
for (ConstNodeRange::const_iterator it = range.begin();
it != range.end(); ++it)
const Node *node = *it;
for (unsigned int i=0; i<LIBMESH_DIM; i++)
_vmin[i] = std::min(_vmin[i], (*node)(i));
_vmax[i] = std::max(_vmax[i], (*node)(i));
...
...
// Query the number of DOFs for a particular node in a system
const unsigned int n_dofs_per_node = node->n_dofs(sys_num);
R. H. Stogner libMesh PDE Simulation March 2, 2016 49 / 124
![Page 65: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/65.jpg)
Weighted Residuals
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 50 / 124
![Page 66: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/66.jpg)
Weighted Residuals
• The point of departure in any FE analysis which uses libMesh is the
weighted residual statement
(F(u), v) = 0 ∀v ∈ V
R. H. Stogner libMesh PDE Simulation March 2, 2016 51 / 124
![Page 67: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/67.jpg)
Weighted Residuals
• The point of departure in any FE analysis which uses libMesh is the
weighted residual statement
(F(u), v) = 0 ∀v ∈ V
• Or, more precisely, the weighted residual statement associated with
the finite-dimensional space Vh ⊂ V
(F(uh), vh) = 0 ∀vh ∈ Vh
R. H. Stogner libMesh PDE Simulation March 2, 2016 51 / 124
![Page 68: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/68.jpg)
Weighted Residuals Some Examples
Poisson Equation
−∆u = f ∈ Ω
R. H. Stogner libMesh PDE Simulation March 2, 2016 52 / 124
![Page 69: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/69.jpg)
Weighted Residuals Some Examples
Poisson Equation
−∆u = f ∈ Ω
Weighted Residual Statement
(F(u), v) :=
∫
Ω
[∇u · ∇v− fv] dx
+
∫
∂ΩN
(∇u · n) v ds
R. H. Stogner libMesh PDE Simulation March 2, 2016 52 / 124
![Page 70: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/70.jpg)
Weighted Residuals Some Examples
Linear Convection-Diffusion
−k∆u + b · ∇u = f ∈ Ω
R. H. Stogner libMesh PDE Simulation March 2, 2016 52 / 124
![Page 71: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/71.jpg)
Weighted Residuals Some Examples
Linear Convection-Diffusion
−k∆u + b · ∇u = f ∈ Ω
Weighted Residual Statement
(F(u), v) :=
∫
Ω
[k∇u · ∇v + (b · ∇u)v− fv] dx
+
∫
∂ΩN
k (∇u · n) v ds
R. H. Stogner libMesh PDE Simulation March 2, 2016 52 / 124
![Page 72: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/72.jpg)
Weighted Residuals Some Examples
Stokes Flow
∇p− ν∆u = f
∇ · u = 0∈ Ω
R. H. Stogner libMesh PDE Simulation March 2, 2016 52 / 124
![Page 73: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/73.jpg)
Weighted Residuals Some Examples
Stokes Flow
∇p− ν∆u = f
∇ · u = 0∈ Ω
Weighted Residual Statement
u := [u, p] , v := [v, q]
(F(u), v) :=
∫
Ω
[−p (∇ · v) + ν∇u :∇v− f · v
+ (∇ · u) q] dx +
∫
∂ΩN
(ν∇u− pI) n · v ds
R. H. Stogner libMesh PDE Simulation March 2, 2016 52 / 124
![Page 74: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/74.jpg)
Weighted Residuals Some Examples
• To obtain the approximate problem, we simply replace u← uh,
v← vh, and Ω← Ωh in the weighted residual statement.
R. H. Stogner libMesh PDE Simulation March 2, 2016 53 / 124
![Page 75: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/75.jpg)
Weighted Residuals Weighted Residual Statement
• For simplicity we start with the weighted residual statement arising
from the Poisson equation, with ∂ΩN = ∅,
(F(uh), vh) :=∫
Ωh
[
∇uh · ∇vh − fvh]
dx = 0 ∀vh ∈ Vh
R. H. Stogner libMesh PDE Simulation March 2, 2016 54 / 124
![Page 76: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/76.jpg)
Weighted Residuals Element Integrals
• The integral over Ωh . . .
0 =
∫
Ωh
[
∇uh · ∇vh − fvh]
dx ∀vh ∈ Vh
R. H. Stogner libMesh PDE Simulation March 2, 2016 55 / 124
![Page 77: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/77.jpg)
Weighted Residuals Element Integrals
• The integral over Ωh . . . is written as a sum of integrals over the Ne
finite elements:
0 =
∫
Ωh
[
∇uh · ∇vh − fvh]
dx ∀vh ∈ Vh
=
Ne∑
e=1
∫
Ωe
[
∇uh · ∇vh − fvh]
dx ∀vh ∈ Vh
R. H. Stogner libMesh PDE Simulation March 2, 2016 55 / 124
![Page 78: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/78.jpg)
Weighted Residuals Finite Element Basis Functions
• An element integral will have
contributions only from the
global basis functions
corresponding to its nodes.
• We call these local basis
functions φi, 0 ≤ i ≤ Ns.
vh∣
∣
Ωe=
Ns∑
i=1
ciφi
Ωe
R. H. Stogner libMesh PDE Simulation March 2, 2016 56 / 124
![Page 79: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/79.jpg)
Weighted Residuals Finite Element Basis Functions
• An element integral will have
contributions only from the
global basis functions
corresponding to its nodes.
• We call these local basis
functions φi, 0 ≤ i ≤ Ns.
vh∣
∣
Ωe=
Ns∑
i=1
ciφi
∫
Ωe
vh dx =
Ns∑
i=1
ci
∫
Ωe
φi dx
Ωe
R. H. Stogner libMesh PDE Simulation March 2, 2016 56 / 124
![Page 80: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/80.jpg)
Weighted Residuals Element Matrix and Load Vector
• The element integrals . . .
∫
Ωe
[
∇uh · ∇vh − fvh]
dx
R. H. Stogner libMesh PDE Simulation March 2, 2016 57 / 124
![Page 81: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/81.jpg)
Weighted Residuals Element Matrix and Load Vector
• The element integrals . . .
∫
Ωe
[
∇uh · ∇vh − fvh]
dx
• are written in terms of the local “φi” basis functions
Ns∑
j=1
uj
∫
Ωe
∇φj · ∇φi dx−∫
Ωe
fφi dx , i = 1, . . . ,Ns
R. H. Stogner libMesh PDE Simulation March 2, 2016 57 / 124
![Page 82: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/82.jpg)
Weighted Residuals Element Matrix and Load Vector
• The element integrals . . .
∫
Ωe
[
∇uh · ∇vh − fvh]
dx
• are written in terms of the local “φi” basis functions
Ns∑
j=1
uj
∫
Ωe
∇φj · ∇φi dx−∫
Ωe
fφi dx , i = 1, . . . ,Ns
• This can be expressed naturally in matrix notation as
KeUe − Fe
R. H. Stogner libMesh PDE Simulation March 2, 2016 57 / 124
![Page 83: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/83.jpg)
Weighted Residuals Global Linear System
• The entries of the element stiffness matrix are the integrals
Keij :=
∫
Ωe
∇φj · ∇φi dx
R. H. Stogner libMesh PDE Simulation March 2, 2016 58 / 124
![Page 84: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/84.jpg)
Weighted Residuals Global Linear System
• The entries of the element stiffness matrix are the integrals
Keij :=
∫
Ωe
∇φj · ∇φi dx
• While for the element right-hand side we have
Fei :=
∫
Ωe
fφi dx
R. H. Stogner libMesh PDE Simulation March 2, 2016 58 / 124
![Page 85: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/85.jpg)
Weighted Residuals Global Linear System
• The entries of the element stiffness matrix are the integrals
Keij :=
∫
Ωe
∇φj · ∇φi dx
• While for the element right-hand side we have
Fei :=
∫
Ωe
fφi dx
• The element stiffness matrices and right-hand sides can be
“assembled” to obtain the global system of equations
KU = F
R. H. Stogner libMesh PDE Simulation March 2, 2016 58 / 124
![Page 86: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/86.jpg)
Weighted Residuals Reference Element Map
• The integrals are performed on a “reference” element Ωe
eΩx (ξ)x ξ
Ωe
R. H. Stogner libMesh PDE Simulation March 2, 2016 59 / 124
![Page 87: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/87.jpg)
Weighted Residuals Reference Element Map
• The integrals are performed on a “reference” element Ωe
eΩx (ξ)x
Ωe
ξ
• The Jacobian of the map x(ξ) is J.
Fei =
∫
Ωe
fφidx =
∫
Ωe
f (x(ξ))φi|J|dξ
R. H. Stogner libMesh PDE Simulation March 2, 2016 59 / 124
![Page 88: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/88.jpg)
Weighted Residuals Reference Element Map
• The integrals are performed on a “reference” element Ωe
eΩx (ξ)x
Ωe
ξ
• Chain rule: ∇ = J−1∇ξ := ∇ξ
Keij =
∫
Ωe
∇φj · ∇φi dx =
∫
Ωe
∇ξφj · ∇ξφi |J|dξ
R. H. Stogner libMesh PDE Simulation March 2, 2016 59 / 124
![Page 89: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/89.jpg)
Weighted Residuals Element Quadrature
• The integrals on the “reference” element are approximated via
numerical quadrature.
R. H. Stogner libMesh PDE Simulation March 2, 2016 60 / 124
![Page 90: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/90.jpg)
Weighted Residuals Element Quadrature
• The integrals on the “reference” element are approximated via
numerical quadrature.
• The quadrature rule has Nq points “ξq” and weights “wq”.
R. H. Stogner libMesh PDE Simulation March 2, 2016 60 / 124
![Page 91: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/91.jpg)
Weighted Residuals Element Quadrature
• The integrals on the “reference” element are approximated via
numerical quadrature.
• The quadrature rule has Nq points “ξq” and weights “wq”.
Fei =
∫
Ωe
fφi|J|dξ
≈Nq∑
q=1
f (x(ξq))φi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 60 / 124
![Page 92: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/92.jpg)
Weighted Residuals Element Quadrature
• The integrals on the “reference” element are approximated via
numerical quadrature.
• The quadrature rule has Nq points “ξq” and weights “wq”.
Keij =
∫
Ωe
∇ξφj · ∇ξφi |J|dξ
≈Nq∑
q=1
∇ξφj(ξq) · ∇ξφi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 60 / 124
![Page 93: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/93.jpg)
Weighted Residuals libMesh Quadrature Point Data
• libMesh provides the following variables at each quadrature point q
Code Math Description
JxW[q] |J(ξq)|wq Jacobian times weight
phi[i][q] φi(ξq) value of ith shape fn.
dphi[i][q] ∇ξφi(ξq) value of ith shape fn. gradient
d2phi[i][q] ∇2ξ φi(ξq) value of ith shape fn. Hessian
xyz[q] x(ξq) location of ξq in physical space
R. H. Stogner libMesh PDE Simulation March 2, 2016 61 / 124
![Page 94: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/94.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 95: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/95.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Fei =
Nq∑
q=1
f (x(ξq))φi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 96: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/96.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Fei =
Nq∑
q=1
f (x(ξq))φi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 97: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/97.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Fei =
Nq∑
q=1
f (x(ξq))φi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 98: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/98.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Fei =
Nq∑
q=1
f (x(ξq))φi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 99: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/99.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Keij =
Nq∑
q=1
∇ξφj(ξq) · ∇ξφi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 100: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/100.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Keij =
Nq∑
q=1
∇ξφj(ξq) · ∇ξφi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 101: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/101.jpg)
Weighted Residuals Matrix Assembly Loops
• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Keij =
Nq∑
q=1
∇ξφj(ξq) · ∇ξφi(ξq)|J(ξq)|wq
R. H. Stogner libMesh PDE Simulation March 2, 2016 62 / 124
![Page 102: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/102.jpg)
Weighted Residuals Convection-Diffusion Equation
• The matrix assembly routine for the linear convection-diffusion
equation,
−k∆u + b · ∇u = f
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(k*(dphi[j][q]*dphi[i][q])
+(b*dphi[j][q])*phi[i][q]);
R. H. Stogner libMesh PDE Simulation March 2, 2016 63 / 124
![Page 103: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/103.jpg)
Weighted Residuals Convection-Diffusion Equation
• The matrix assembly routine for the linear convection-diffusion
equation,
−k∆u + b · ∇u = f
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(k*(dphi[j][q]*dphi[i][q])
+(b*dphi[j][q])*phi[i][q]);
R. H. Stogner libMesh PDE Simulation March 2, 2016 63 / 124
![Page 104: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/104.jpg)
Weighted Residuals Convection-Diffusion Equation
• The matrix assembly routine for the linear convection-diffusion
equation,
−k∆u + b · ∇u = f
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(k*(dphi[j][q]*dphi[i][q])
+(b*dphi[j][q])*phi[i][q]);
R. H. Stogner libMesh PDE Simulation March 2, 2016 63 / 124
![Page 105: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/105.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• We have an array of submatrices: Ke[ ][ ]
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 106: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/106.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• We have an array of submatrices: Ke[0][0]
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 107: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/107.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• We have an array of submatrices: Ke[1][1]
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 108: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/108.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• We have an array of submatrices: Ke[2][1]
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 109: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/109.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• And an array of right-hand sides: Fe[].
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 110: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/110.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• And an array of right-hand sides: Fe[0].
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 111: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/111.jpg)
Weighted Residuals Stokes Flow
• For multi-variable systems like Stokes flow,
∇p− ν∆u = f
∇ · u = 0∈ Ω ⊂ R
2
• The element stiffness matrix concept can extended to include
sub-matrices
Keu1u1
Keu1u2
Keu1p
Keu2u1
Keu2u2
Keu2p
Kepu1
Kepu2
Kepp
Ueu1
Ueu2
Uep
−
Feu1
Feu2
Fep
• And an array of right-hand sides: Fe[1].
R. H. Stogner libMesh PDE Simulation March 2, 2016 64 / 124
![Page 112: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/112.jpg)
Weighted Residuals Stokes Flow
• The matrix assembly can proceed in essentially the same way.
• For the momentum equations:
for (q=0; q<Nq; ++q)
for (d=0; d<2; ++d)
for (i=0; i<Ns; ++i)
Fe[d](i) += JxW[q]*f(xyz[q],d)*phi[i][q];
for (j=0; j<Ns; ++j)
Ke[d][d](i,j) +=
JxW[q]*nu*(dphi[j][q]*dphi[i][q]);
R. H. Stogner libMesh PDE Simulation March 2, 2016 65 / 124
![Page 113: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/113.jpg)
Adaptive Mesh Refinement
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 66 / 124
![Page 114: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/114.jpg)
Adaptive Mesh Refinement
Model Problem
• Consider the 1D model ODE
−u′′ + bu′ + cu = f ∈ Ω = (0, L)u(0) = u0
u(L) = uL
(1)
• with weak form∫
Ω
(
u′v′ + bu′v + cuv)
dx =
∫
Ω
fv dx (2)
for every v ∈ H10(Ω).
R. H. Stogner libMesh PDE Simulation March 2, 2016 67 / 124
![Page 115: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/115.jpg)
Adaptive Mesh Refinement
Model Problem (cont.)
• The analogous d-dimensional problem with Ω ⊂ Rd and boundary ∂Ω
is
−∆u + b · ∇u + cu = f ∈ Ωu = g ∈ ∂Ω
(3)
• with weak form∫
Ω
(∇u · ∇v + (b · ∇u)v + cuv) dx =
∫
Ω
fv dx (4)
R. H. Stogner libMesh PDE Simulation March 2, 2016 68 / 124
![Page 116: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/116.jpg)
Adaptive Mesh Refinement
Model Problem (cont.)
• The finite element method works with the weak form, replacing the
trial and test functions u, v with their approximations uh, vh, and
summing the contributions of the element integrals
Ne∑
e=1
∫
Ωe
(
∇uh · ∇vh + (b · ∇uh)vh + cuhvh − fvh)
dx = 0 (5)
• Remark: We considered here a standard piecewise continuous finite
element basis. In general, ∇uh will have a jump discontinuity across
element boundaries.
R. H. Stogner libMesh PDE Simulation March 2, 2016 69 / 124
![Page 117: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/117.jpg)
Adaptive Mesh Refinement
Galerkin FE Method
• Expressing uh and vh in our chosen piecewise continuous polynomial
basis
uh =N∑
j=1
ujϕj vh =N∑
i=1
ciϕi (6)
we obtain on each element Ωe
N∑
j=1
uj
[∫
Ωe
(∇ϕj · ∇ϕi + (b · ∇ϕj)ϕi + cϕjϕi) dx
]
=
∫
Ωe
fϕi dx (7)
for i = 1 . . .N.
• In the standard element-stiffness matrix form,
KeU = Fe (8)
R. H. Stogner libMesh PDE Simulation March 2, 2016 70 / 124
![Page 118: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/118.jpg)
Adaptive Mesh Refinement
LibMesh Representation (cont.)
for (q=0; q<Nq; ++q)
// Compute b, c, f at this quadrature point
// ...
for (i=0; i<N; ++i)
Fe(i) += JxW[q]*f*phi[i][q];
for (j=0; j<N; ++j)
Ke(i,j) += JxW[q]*(
(dphi[i][q]*dphi[j][q]) +
(b*dphi[j][q])*phi[i][q] +
c*phi[j][q]*phi[i][q]
);
R. H. Stogner libMesh PDE Simulation March 2, 2016 71 / 124
![Page 119: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/119.jpg)
Adaptive Mesh Refinement
Natural Refinement Patterns
Triangle Quadrilateral
Tetrahedron Prism
R. H. Stogner libMesh PDE Simulation March 2, 2016 72 / 124
![Page 120: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/120.jpg)
Adaptive Mesh Refinement
Flux-Jump Error Indicator
• The flux-jump error indicator is derived starting from the element
integrals
Ne∑
e=1
∫
Ωe
(
∇uh · ∇vh + (b · ∇uh)vh + cuhvh − fvh)
dx = 0 (5)
• Applying the divergence theorem “in reverse” obtains
Ne∑
e=1
∫
Ωe
(
−∆uh + (b · ∇uh) + cuh − f)
vh dx + (9)
∑
∂Ωe 6⊂∂Ω
∫
∂Ωe
s
∂uh
∂n
vh dx = 0
R. H. Stogner libMesh PDE Simulation March 2, 2016 73 / 124
![Page 121: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/121.jpg)
Adaptive Mesh Refinement
Flux-Jump Error Indicator (cont.)
• Defining the cell residual
r(uh) = −∆uh + (b · ∇uh) + cuh − f (10)
we have
Ne∑
e=1
∫
Ωe
r(uh)vh dx +∑
∂Ωe 6⊂∂Ω
∫
∂Ωe
s
∂uh
∂n
vh dx = 0 (11)
• Clearly, the exact solution u satisfies (11) identically.
• Computing r(uh) requires knowledge of the differential operator (i.e.
knowledge of the “physics”).
• The second sum leads to a physics-independent method for
estimating the error in the approximate solution uh.
R. H. Stogner libMesh PDE Simulation March 2, 2016 74 / 124
![Page 122: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/122.jpg)
Adaptive Mesh Refinement
1D Example
• Consider the function
u =1− exp(10x)
1− exp(10)
which is a solution of the classic 1D
advection-diffusion boundary layer
equation.
• We assume here that the finite element
solution is the linear interpolant of u, and
compute the error indicator for a sequence
of uniformly refined grids.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
R. H. Stogner libMesh PDE Simulation March 2, 2016 75 / 124
![Page 123: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/123.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
0 0.2 0.4 0.6 0.8 10
2
4
6
8
10
12
x
u’exact
1 2 3 40
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Element Number
η
4 elements
||e||L2= 0.09
R. H. Stogner libMesh PDE Simulation March 2, 2016 76 / 124
![Page 124: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/124.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
0 0.2 0.4 0.6 0.8 10
2
4
6
8
10
12
x
u’exact
1 2 3 4 5 6 7 80
0.5
1
1.5
Element Number
η
8 elements
||e||L2= 0.027
R. H. Stogner libMesh PDE Simulation March 2, 2016 76 / 124
![Page 125: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/125.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
0 0.2 0.4 0.6 0.8 10
2
4
6
8
10
12
x
u’exact
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 160
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Element Number
η
16 elements
||e||L2= 0.0071
R. H. Stogner libMesh PDE Simulation March 2, 2016 76 / 124
![Page 126: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/126.jpg)
Adaptive Mesh Refinement
A Simple Refinement Strategy
• A simple adaptive refinement strategy with r max refinement steps
for this 1D example problem is:
r=0;
while (r < r_max)
Compute the FE solution (linear interpolant)
Estimate the error (using flux-jump indicator)
Refine the elements with error in top 10%
Increment r
end
R. H. Stogner libMesh PDE Simulation March 2, 2016 77 / 124
![Page 127: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/127.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 128: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/128.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 129: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/129.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 130: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/130.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 131: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/131.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 132: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/132.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 133: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/133.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 134: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/134.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 135: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/135.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 136: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/136.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 137: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/137.jpg)
Adaptive Mesh Refinement
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
uexact
u
R. H. Stogner libMesh PDE Simulation March 2, 2016 78 / 124
![Page 138: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/138.jpg)
Adaptive Mesh Refinement
A Simple Refinement Strategy (cont.)
0 0.5 1 1.5−3
−2.5
−2
−1.5
−1
−0.5
0
log10
Nnodes
log
10 ||e
||UniformAdaptive
R. H. Stogner libMesh PDE Simulation March 2, 2016 79 / 124
![Page 139: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/139.jpg)
Parallelism on Adaptive Unstructured Meshes
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 80 / 124
![Page 140: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/140.jpg)
Parallelism on Adaptive Unstructured Meshes Parallelism
Parallelism Goals
Reduced CPU time
• Distributed CPU usage
• Asynchronous I/O
Reduced memory requirements
• Larger attainable problem size
R. H. Stogner libMesh PDE Simulation March 2, 2016 81 / 124
![Page 141: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/141.jpg)
Parallelism on Adaptive Unstructured Meshes Parallelism
Parallelism Levels
SIMD Instructions
• Assemblies, MatVec operations
Shared-memory Threads
• Mesh Partitioning
R. H. Stogner libMesh PDE Simulation March 2, 2016 82 / 124
![Page 142: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/142.jpg)
Parallelism on Adaptive Unstructured Meshes Parallelism
Parallelism Levels
Separate Simulations
• Parametric studies
• Uncertainty analysis
Distributed-memory Processes
• Asynchronous I/O
• Mesh Partitioning
R. H. Stogner libMesh PDE Simulation March 2, 2016 83 / 124
![Page 143: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/143.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
SerialMesh Partitioning
• Each element, node is “local” to
one processor
• Each processor has an identical
Mesh copy
• Mesh stays in sync through
redundant work
• FEM data synced on “ghost”
elements only
R. H. Stogner libMesh PDE Simulation March 2, 2016 84 / 124
![Page 144: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/144.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
ParallelMesh Partitioning
• Processors store only local and
ghost objects
• Each processor has a small
Mesh subset
• Mesh stays in sync through MPI
communication
R. H. Stogner libMesh PDE Simulation March 2, 2016 85 / 124
![Page 145: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/145.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
ParallelMesh Partitioning
Pros
• Reduced memory use
• O(NE/NP) CPU costs
R. H. Stogner libMesh PDE Simulation March 2, 2016 86 / 124
![Page 146: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/146.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
ParallelMesh Partitioning
Cons
• Increased code complexity
• Increased synchronization
“bookkeeping”
• Greater debugging difficulty
R. H. Stogner libMesh PDE Simulation March 2, 2016 87 / 124
![Page 147: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/147.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
“Typical” PDE example
Transient Cahn-Hilliard, Bogner-Fox-Schmidt quads or hexes
101
102
103
104
103
104
105
106
107
RA
M (
MB
)
Mesh Size (quad elements)
2D SerialMesh vs. ParallelMesh Per-Node Memory Usage
1 Node Serial 2D
1 Node Parallel 2D
2 Node Serial 2D
2 Node Parallel 2D
4 Node Serial 2D
4 Node Parallel 2D
8 Node Serial 2D
8 Node Parallel 2D
16 Node Serial 2D
16 Node Parallel 2D
32 Node Serial 2D
32 Node Parallel 2D
101
102
103
104
103
104
105
106
RA
M (
MB
)
Mesh Size (hex elements)
3D SerialMesh vs. ParallelMesh Per-Node Memory Usage
1 Node Serial 3D
1 Node Parallel 3D
2 Node Serial 3D
2 Node Parallel 3D
4 Node Serial 3D
4 Node Parallel 3D
8 Node Serial 3D
8 Node Parallel 3D
16 Node Serial 3D
16 Node Parallel 3D
32 Node Serial 3D
32 Node Parallel 3D
Results
• Parallel codes using SerialMesh are unchanged for ParallelMesh
• Overhead, distributed sparse matrix costs are unchanged
• Serialized mesh, indexing once dominated RAM use
R. H. Stogner libMesh PDE Simulation March 2, 2016 88 / 124
![Page 148: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/148.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Distributed Mesh Refinement
Elem, Node creation
R. H. Stogner libMesh PDE Simulation March 2, 2016 89 / 124
![Page 149: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/149.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Distributed Mesh Refinement
Elem, Node creation
• Ids i : i mod (NP + 1) = p are owned
by processor p
R. H. Stogner libMesh PDE Simulation March 2, 2016 89 / 124
![Page 150: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/150.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Distributed Mesh Refinement
Elem, Node creation
• Ids i : i mod (NP + 1) = p are owned
by processor p
Synchronization
• Refinement Flags
• New ghost child elements, nodes
• Hanging node constraint equations
R. H. Stogner libMesh PDE Simulation March 2, 2016 89 / 124
![Page 151: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/151.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Distributed Mesh Refinement
Elem, Node creation
• Ids i : i mod (NP + 1) = p are owned
by processor p
Synchronization
• Refinement Flags
• Data requested by id
• Iteratively to enforce smoothing
• New ghost child elements, nodes
• Id requested by data
• Hanging node constraint equations
• Iteratively through subconstraints,
subconstraints-of-subconstraints...
R. H. Stogner libMesh PDE Simulation March 2, 2016 89 / 124
![Page 152: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/152.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Adaptivity and ParallelMesh
Challenges
• Reindexing elements, nodes,
DoFs
• Synchronization of ghost objects
• Load balancing, Repartitioning
R. H. Stogner libMesh PDE Simulation March 2, 2016 90 / 124
![Page 153: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/153.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
ParallelMesh Data Structure
std::vector fails
• Not sparse
• O(NE) storage cost
std::map
• “mapvector” interface provides iterators
• O(log(NE/NP)) lookup time without std::unsorted map
• O(1) lookup time with std::unsorted map
R. H. Stogner libMesh PDE Simulation March 2, 2016 91 / 124
![Page 154: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/154.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Parallel:: API
Encapsulating MPI
• Improvement over MPI C++ interface
• Makes code shorter, more legible
Example:
std::vector<Real> send, recv;
...
send_receive(dest_processor_id, send,
source_processor_id, recv);
R. H. Stogner libMesh PDE Simulation March 2, 2016 92 / 124
![Page 155: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/155.jpg)
Parallelism on Adaptive Unstructured Meshes ParallelMesh
Parallel:: APIInstead of:
if (dest_processor_id == libMesh::processor_id() &&
source_processor_id == libMesh::processor_id())
recv = send;
#ifdef HAVE_MPI
else
unsigned int sendsize = send.size(), recvsize;
MPI_Status status;
MPI_Sendrecv(&sendsize, 1, datatype<unsigned int>(),
dest_processor_id, 0,
&recvsize, 1, datatype<unsigned int>(),
source_processor_id, 0,
libMesh::COMM_WORLD,
&status);
recv.resize(recvsize);
MPI_Sendrecv(sendsize ? &send[0] : NULL, sendsize, MPI_DOUBLE,
dest_processor_id, 0,
recvsize ? &recv[0] : NULL, recvsize, MPI_DOUBLE,
source_processor_id, 0,
libMesh::COMM_WORLD,
&status);
#endif // HAVE_MPI
R. H. Stogner libMesh PDE Simulation March 2, 2016 93 / 124
![Page 156: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/156.jpg)
Verification
Outline
1 Introduction
2 Library Design
3 Application Examples
4 Software Installation & Ecosystem
5 A Generic Boundary Value Problem
6 Key Data Structures
7 Weighted Residuals
8 Adaptive Mesh Refinement
9 Parallelism on Adaptive Unstructured Meshes
10 Verification
R. H. Stogner libMesh PDE Simulation March 2, 2016 94 / 124
![Page 157: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/157.jpg)
Verification
Verification: Finding The Unknown Unknowns
Solution Verification
Estimating Numerical Error:
• Discretization Error
• Iterative Error
• Round-off Error
These errors cannot be avoided, but can be quantified.
Code Verification
• Software bugs
• Numerical algorithm weaknesses
• Model implementation mistakes
Codes cannot practically be proven error-free, but can be proven to have
errors. So we try our best to do the latter...
R. H. Stogner libMesh PDE Simulation March 2, 2016 95 / 124
![Page 158: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/158.jpg)
Verification Designing for Reuse
Code Reuse
Examples
• MPI, BLAS
• Aztec, MUMPS, PETSc
• deal.II, FEniCS, libMesh
• Don’t reinvent the wheel
unnecessarily!
• Time spent rewriting something
old is time that could have been
spent writing something new.
• More eyes == fewer bugs
• Extend existing capabilities
where possible.
libMesh
STL
Parallel::
MPI BLAS
GetPot
TBB
PETSc/
Trilinos
FIN-S
R. H. Stogner libMesh PDE Simulation March 2, 2016 96 / 124
![Page 159: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/159.jpg)
Verification Designing for Reuse
Modular Programming
FE
Elem
Control
Matrix,
Vector
Linear
Solvers
Mesh
Physics
Nonlinear
Solvers
Transient
Integrators
Discrete Components, Interfaces
• Linear, nonlinear solvers are
discretization-independent
• System assembly, solution I/O &
postprocessing can be
discretization-independent
• Time, space discretizations should be
physics-independent
• Error analysis, sensitivity methods can be
physics-independent
• Reusable components get re-tested
• Errors too subtle to find in complex physics
are easy to spot in benchmark problems.
R. H. Stogner libMesh PDE Simulation March 2, 2016 97 / 124
![Page 160: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/160.jpg)
Verification Designing for Reuse
Assertions
“When in trouble when in doubt, run in circles scream and shout.”
- old Army War College football team slogan
R. H. Stogner libMesh PDE Simulation March 2, 2016 98 / 124
![Page 161: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/161.jpg)
Verification Designing for Reuse
Assertions
“When in trouble when in doubt, run in circles scream and shout.”
- old Army War College football team slogan
#if IN DOUBT if (in trouble()) run in circles();
scream and shout();
#endif
R. H. Stogner libMesh PDE Simulation March 2, 2016 98 / 124
![Page 162: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/162.jpg)
Verification Designing for Reuse
Assertions
“When in trouble when in doubt, run in circles scream and shout.”
- old Army War College football team slogan
#if IN DOUBT if (in trouble()) run in circles();
scream and shout();
#endif
assert(!in trouble());
R. H. Stogner libMesh PDE Simulation March 2, 2016 98 / 124
![Page 163: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/163.jpg)
Verification Designing for Reuse
High-level Assertions
libmesh assert(), PETSc debug mode
• Active only in “debug” runs
• Function preconditions
• Are function arguments all valid?
• Function postconditions
• Does function result satisfy requirements?
• Approx. 7000 assertions in libMesh alone
Assertion types
Standard assert() prints failed assertion, prints file/line numbers, exits
libmesh assert() adds:
• Per-processor stack trace files
• Line numbers via gdb
• C++ exception thrown
R. H. Stogner libMesh PDE Simulation March 2, 2016 99 / 124
![Page 164: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/164.jpg)
Verification Designing for Reuse
High-level Assertions Examples
libmesh_assert(c < _variables.size());
libmesh_assert(s < elem->n_sides());
libmesh_assert((ig >= Ug.first_local_index()) &&
(ig < Ug.last_local_index()));
libmesh_assert(requested_ids[p].size() == ghost_objects_from_proc[p]);
libmesh_assert(obj_procid != DofObject::invalid_processor_id);
MeshTools::libmesh_assert_valid_node_procids(mesh);
libmesh_assert(neigh->has_children());
libmesh_assert(this->closed());
libmesh_assert(this->initialized());
libmesh_assert(mesh.is_prepared());
libmesh_assert(error_estimator.error_norm.type(var) == H1_SEMINORM
error_estimator.error_norm.type(var) == W1_INF_SEMINORM)
libmesh_assert(number_h_refinements > 0 || number_p_refinements >
R. H. Stogner libMesh PDE Simulation March 2, 2016 100 / 124
![Page 165: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/165.jpg)
Verification Designing for Reuse
Low-level Assertions
Defining GLIBCXX DEBUG
• Runtime bounds-checking of standard vector, iterator use
• Similar to ifort “-check bounds”
• Out Of Bounds errors otherwise lead to corrupt data, not just
segfaults!
R. H. Stogner libMesh PDE Simulation March 2, 2016 101 / 124
![Page 166: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/166.jpg)
Verification Designing for Reuse
Unit Tests
Testing One Object At A Time
• Reusable modules interact with all other code through a limited API
• That API can be tested directly outside of application code
• Test one method at a time, isolate problems locally
• 108 unit tests currently in libMesh
R. H. Stogner libMesh PDE Simulation March 2, 2016 102 / 124
![Page 167: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/167.jpg)
Verification Designing for Reuse
Unit Tests
Testing One Object At A Time
• Reusable modules interact with all other code through a limited API
• That API can be tested directly outside of application code
• Test one method at a time, isolate problems locally
• 108 unit tests currently in libMesh
Tests are useful only when run!
• Regression, example, application tests had more coverage
• Unit tests became useful only when added to CI
• Most useful: Test-driven development
• Write tests first
• Write code to fit
R. H. Stogner libMesh PDE Simulation March 2, 2016 102 / 124
![Page 168: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/168.jpg)
Verification Designing for Reuse
Unit Tests Example
#include <quadrature.h>
class QuadratureTest : public CppUnit::TestCase
public:
CPPUNIT_TEST_SUITE( QuadratureTest );
CPPUNIT_TEST( test3DWeights<FIFTH> ); // etc.
template <Order order>
void test3DWeights ()
AutoPtr<QBase> qrule = QBase::build(QGAUSS, 3, order);
qrule->init (TET10);
sum = 0;
for (unsigned int qp=0; qp<qrule->n_points(); qp++)
sum += qrule->w(qp);
CPPUNIT_ASSERT_DOUBLES_EQUAL( 1./6., sum , TOLERANCE*TOLERANCE
;
R. H. Stogner libMesh PDE Simulation March 2, 2016 103 / 124
![Page 169: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/169.jpg)
Verification Designing for Reuse
Parametric Testing
One Test Code, Many Tests
• Keep test codes generic
• Execute with many different parameter choices
• libMesh compile time examples:
• Algebraic solver interface
• Real/complex arithmetic
• Mesh data structure
• libMesh run time examples:
• Geometric element type
• Finite element type
• Polynomial degree
• Error indicator type
• Processor count
• Partitioner
• Adaptive refinement strategy
• I/O format
R. H. Stogner libMesh PDE Simulation March 2, 2016 104 / 124
![Page 170: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/170.jpg)
Verification Library Verification
Regression Tests
Revise Software⇒ Rerun Tests
• Example applications, unit tests, benchmark tests
• Catches unintended consequences of revisions
• Continuous Build System automation
• Tests once run “by hand” by libMesh developers
• BuildBot tests for wide configuration coverage
• Civet tests on every Pull Request
R. H. Stogner libMesh PDE Simulation March 2, 2016 105 / 124
![Page 171: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/171.jpg)
Verification Library Verification
Verification Benchmark Problems
Choosing Test Problems
Capitalize on anything you know a priori:
• Known solutions
• Exact solution to discretized problem
• Limit solution of continuous problem
• Known quantities of interest
• Known asymptotic convergence rates
• Known residuals
R. H. Stogner libMesh PDE Simulation March 2, 2016 106 / 124
![Page 172: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/172.jpg)
Verification Library Verification
Known Solutions, Functionals
Examples
• Incompressible flow around a cusp
• Wetting angle in Laplace-Young surface tension
R. H. Stogner libMesh PDE Simulation March 2, 2016 107 / 124
![Page 173: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/173.jpg)
Verification Library Verification
Manufactured Solutions
Any Solution for Any Equation
• Real Physics: R(u) = 0 for u = ur
• Choose manufactured solution um
• Desired Physics: Rm(u) = 0 for u = um
• Construct Rm(u) ≡ R(u)− R(um)
R. H. Stogner libMesh PDE Simulation March 2, 2016 108 / 124
![Page 174: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/174.jpg)
Verification Library Verification
Manufactured Solution Example
Convection-Diffusion Problem with Adjoints
Residual equation:
R(u) = ∇ · α∇u + β~ex · ∇u + f = 0
Manufactured solution:
u ≡ 4(1− e−αx − (1− e−α)x)y(1− y)
• Homogeneous Dirichlet boundary
• α controls flux strength, layer
• Choose any convection strength β, solve for f
• β = 0 gives simple series adjoint solutions
R. H. Stogner libMesh PDE Simulation March 2, 2016 109 / 124
![Page 175: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/175.jpg)
Verification Library Verification
Manufactured Solution Example
Goal-Oriented Refinement
• Superconvergence on some grids
• Convergence “plateaus” found in multiple
refinement strategies
• UniformRefinementEstimator required
new code to solve for adjoint solution errors
• PatchRecoveryErrorEstimatorrequired new seminorm integration
(H1/L2/mixed vs. W1,inf) to give compatible
error subestimates
5 6 7 8 9 10 11 12−10
−9
−8
−7
−6
−5
−4
−3
−2
Log(Dofs), Degrees of Freedom
Lo
g(%
ag
e R
ela
tive
Err
or)
Uniform
Exact
Kelly
Flux BCs AR (Kelly)
Flux BCs AR (Uniform)
Flux BCs AR (Patch Recovery)
R. H. Stogner libMesh PDE Simulation March 2, 2016 110 / 124
![Page 176: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/176.jpg)
Verification Library Verification
Manufactured Solution Example
Adjoint-based Parameter Sensitivity
• Convergence to analytic sensitivity
plateaus at 2% relative error in
every refinement strategy
• Finite differenced partial derivatives
not responsible
• Manufactured solution allowed
sensitivity subcomponent
comparison to analytic solutions
• Sign errors in libMesh parameter
sensitivity method
5 6 7 8 9 10 11 12−4.2
−4
−3.8
−3.6
−3.4
−3.2
−3
−2.8
−2.6
Log(Dofs), Degrees of Freedom
Lo
g(R
ela
tive
Err
or)
Uniform (10−4
)
ARPP (10−0
)
ARPP (10−2
)
ARPP (10−4
)
R. H. Stogner libMesh PDE Simulation March 2, 2016 111 / 124
![Page 177: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/177.jpg)
Verification Library Verification
Manufactured Solution Example
3 3.5 4 4.5 5 5.5−4
−3.5
−3
−2.5
−2
−1.5
Log(Dofs), Degrees of Freedom
Lo
g(R
ela
tive
Err
or)
Uniform (10−6
)
ARPP (10−6
)
Adjoint-based Parameter Sensitivity
• “Off by 100%” error remaining in
one subterm of equations
• Switch to u′′ = f , 1D quadratic
solutions, manufactured residual
test
• Identified bug in repeated
adjoint solve rhs assembly
• Returned to manufactured solution
benchmark: now converges to true
solution
R. H. Stogner libMesh PDE Simulation March 2, 2016 112 / 124
![Page 178: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/178.jpg)
Verification Library Verification
Manufactured Solution Issues
Compressible Inviscid Perfect Gas Flow
Qet= −
γ
γ − 1
πapxpx
Lsin
πxapx
L
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
+
+γ
γ − 1
πapypy
Lcos
πyapy
L
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
+
−
γ
γ − 1
πapzpz
Lsin
πzapz
L
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
+
+πaρxρx
2Lcos
πxaρx
L
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
2
+
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
2
+
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
2
+
−
πaρyρy
2Lsin
πyaρy
L
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
2
+
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
2
+
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
2
+
+πaρzρz
2Lcos
πzaρz
L
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
2
+
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
2
+
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
2
+
+πauxux
2Lcos
πxaux
L
3
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
2
+
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
2
+
+
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
2
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
px cos
πxapx
L
+ py sin
πyapy
L
+ pz cos
πzapz
L
+ p0
2γ
γ − 1
+
−
πauyuy
Lsin
πyauy
L
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
·
·
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
−
πauzuz
Lsin
πzauz
L
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
·
·
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
−
πavxvx
Lsin
πxavx
L
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
·
·
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
+πavyvy
2Lcos
πyavy
L
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
2
+ 3
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
2
+
+
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
2
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
+
px cos
πxapx
L
+ py sin
πyapy
L
+ pz cos
πzapz
L
+ p0
2γ
γ − 1
+
+πavzvz
Lcos
πzavz
L
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
·
·
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
+πawxwx
Lcos
πxawx
L
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
·
·
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
+πawywy
Lcos
πyawy
L
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
·
·
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
−
πawzwz
2Lsin
πzawz
L
ux sin
πxaux
L
+ uy cos
πyauy
L
+ uz cos
πzauz
L
+ u0
2
+
vx cos
πxavx
L
+ vy sin
πyavy
L
+ vz sin
πzavz
L
+ v0
2
+
+ 3
wx sin
πxawx
L
+ wy sin
πyawy
L
+ wz cos
πzawz
L
+ w0
2
ρx sin
πxaρx
L
+ ρy cos
πyaρy
L
+ ρz sin
πzaρz
L
+ ρ0
+
+
px cos
πxapx
L
+ py sin
πyapy
L
+ pz cos
πzapz
L
+ p0
2γ
γ − 1
• Use Maple, Mathematica, automatic differentiation
• Manufactured Analytic Solution Abstraction library:
https://manufactured-solutions.github.io/MASA/
R. H. Stogner libMesh PDE Simulation March 2, 2016 113 / 124
![Page 179: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/179.jpg)
Verification Library Verification
Manufactured Residuals
The Last Resort
• Manufactured solution tests fail
• The bug location isn’t obvious
• You can’t invert the problem by hand
R. H. Stogner libMesh PDE Simulation March 2, 2016 114 / 124
![Page 180: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/180.jpg)
Verification Library Verification
Manufactured Residuals
The Last Resort
• Manufactured solution tests fail
• The bug location isn’t obvious
• You can’t invert the problem by hand
Test R(U)
• Small (one, two element!?) meshes
• Cartesian grids
R. H. Stogner libMesh PDE Simulation March 2, 2016 114 / 124
![Page 181: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/181.jpg)
Verification Application Verification
Code-to-Code Comparisons
“Lumped” Verification
• Differing results from:
• Different models
• Different formulations
• Different discretizations
R. H. Stogner libMesh PDE Simulation March 2, 2016 115 / 124
![Page 182: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/182.jpg)
Verification Application Verification
Hierarchic Models
Per-Operator Tesing
• Coefficients selectively “turn off” parts of equations
• Allows easier construction of analytic solutions
• Assists with “narrowing down” other bugs
Model Simplification
• Model linearity can be tested in solver
• Reduce complex physics to simple physics case
• Code-to-code testing
R. H. Stogner libMesh PDE Simulation March 2, 2016 116 / 124
![Page 183: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/183.jpg)
Verification Application Verification
Symmetry Tests
Symmetry In, Symmetry Out
• Mirror, radial symmetries
• Beware unstable solution modes!
R. H. Stogner libMesh PDE Simulation March 2, 2016 117 / 124
![Page 184: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/184.jpg)
Verification Application Verification
Jacobian Verification
Inexact Newton Step
J(un−1)(
un − un−1)
≡ −R(un−1)
J ≡ ∂R
∂u
• Library code handles inexact solve tolerances, line search, etc.
• R, J are application-dependent
R. H. Stogner libMesh PDE Simulation March 2, 2016 118 / 124
![Page 185: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/185.jpg)
Verification Application Verification
Library Jacobian Construction
Finite Differencing
Jij ≈Ri(u + εej)− Ri(u− εej)
2ε
Greedy or element-wise algorithms handle sparsity
Complex-Step Perturbations
Jij ≈ℑ[Ri(u + εej
√−1)]
ε
Avoids floating point subtractive cancellation error
Automatic Differentiation
• Variable constructors seed derivatives
• Variable operations evaluate derivatives
R. H. Stogner libMesh PDE Simulation March 2, 2016 119 / 124
![Page 186: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/186.jpg)
Verification Application Verification
Jacobian Verification
Test Analytic vs. Numeric Jacobians
• Relative error in matrix norm
• If match isn’t within tolerance, either:
• The discretization or floating point error has overwhelmed the finitedifferenced Jacobian
• Unlikely for good choices of finite difference perturbations
• Can be investigated
• The residual is non-differentiable at that iterate
• Can be checked analytically
• The Jacobian calculation is wrong
• The residual calculation is wrong
R. H. Stogner libMesh PDE Simulation March 2, 2016 120 / 124
![Page 187: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/187.jpg)
Verification Application Verification
A Priori Asymptotic Convergence Rates
Biharmonic Problem, Manufactured Solution
• ∆2u = f
• C1 Macroelement bases
101
102
103
104
105
10−6
10−5
10−4
10−3
10−2
10−1
100
101
Number of Degrees of Freedom
Err
or
Fra
ction
Error Convergence, Manufactured Benchmark
Powell−Sabin 6−split L2 error
Powell−Sabin 6−split H1 error
Powell−Sabin 12−split L2 error
Powell−Sabin 12−split H1 error
Clough−Tocher 3−split L2 error
Clough−Tocher 3−split H1 error
Verification Example
• Code verification failures - bugs
in basis transformations
• Solution verification “failure” -
higher order Nitsche lift fails for
L2 error with quadratic elements
for fourth order problems
R. H. Stogner libMesh PDE Simulation March 2, 2016 121 / 124
![Page 188: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/188.jpg)
Verification Application Verification
Asymptotic Convergence Rate Examples
Cahn-Hilliard Phase Evolution
0 0.005 0.01 0.01510
−4
10−3
10−2
10−1
100
Time
L2 e
rro
r
coarse
medium
fine
Gives some confidence in even highly
nonlinear, transient, stochastic problems
R. H. Stogner libMesh PDE Simulation March 2, 2016 122 / 124
![Page 189: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/189.jpg)
Verification Application Verification
Thanks to Dr. Graham F. Carey
The original development team was
heavily influenced by Professor Graham F.
Carey, professor of aerospace engineering
and engineering mechanics at The
University of Texas at Austin, director of
the ICES Computational Fluid Dynamics
Laboratory, and holder of the Richard B.
Curran Chair in Engineering.
Many of the technologies employed in
libMesh were implemented because Dr.
Carey taught them to us, we went back to
the lab, and immediately began coding. In
a very real way, he was ultimately
responsible for this library that we hope
you may find useful, despite his continued
insistence that “no one ever got a PhD
from here for writing a code.”
R. H. Stogner libMesh PDE Simulation March 2, 2016 123 / 124
![Page 190: Parallel Adaptive PDE Simulation with libMeshroystgnr/cleaned.pdf · Introduction Background •Modern simulation software is complex: • Implicit numerical methods • Massively](https://reader036.vdocuments.site/reader036/viewer/2022070716/5edaa0b409f66a09130ba7cb/html5/thumbnails/190.jpg)
Verification Application Verification
AcknowledgementsRecent libMesh contributors:
• David Andrs
• Paul Bauman
• Vikram Garg
• Derek Gaston
• Dmitry Karpeev
• Benjamin Kirk
• David Knezevic
• Cody Permann
• John Peterson
• Sylvain Vallaghe
Useful resources:
• libMesh: https://libmesh.github.io/
• MOOSE: https://mooseframework.org/
• FALCON: https://github.com/idaholab/falcon
• MASA:
https://manufactured-solutions.github.io/MASA/
• GRINS: https://grinsfem.github.io/
Questions?R. H. Stogner libMesh PDE Simulation March 2, 2016 124 / 124