post-seismic relaxation process in southeastern asia - mines … · 2013-09-06 · december 9, 2008...

December 9, 2008

Centre desMatériauxP.-M. Fourt

Post-seismic relaxation process in southeastern Asia(my first steps as a user of Z-Set in parallel)

Jean-Didier GaraudClub ZéBuLoN

Club ZéBuLoN

2


Outline

Sumatra, December 26th 2004

General presentation

2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

3


Context & objectives

ANR between CdM and ENS-géologie (Luce Fleitout & Christophe Vigny)

Fit finite element model to GPS data

Large computations (>106 nodes)

coco @ CdM (112x2 cores)

jade @ cines (1536x8 cores)

Club ZéBuLoN

4



Club ZéBuLoN

5



Giant earthquake, mag>9, causing the Tsunami

Epicenter ~200 km away from the coast

GPS measures:

27 cm at Phuket (500 km away)

1 cm at Chengdu (3000 km away)

Club ZéBuLoN

6


Available data

Seismic cartography [Engdhal]

Subduction & transform faults

Club ZéBuLoN

7


Available data

~60 GPS measurements [Vigny & al. 2005]

Co-seismic displacement

Inverse, elasticcomputation

Club ZéBuLoN

8


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

9


Material model

Elastic lithosphere (0►-80 km)

E = 400 GPa

= 0.3

Visco-elastic asthenosphere (-80►-1500 km)

E = 400 GPa

= 0.3

n = 1

K = 1019 -> 1022 Pa.s (depends on temperature & depth)

Club ZéBuLoN

10


Geometry

Elsets :

Crust

Mantle

Serpentine

MPC

1500 km

4000 km

Club ZéBuLoN

11


Input earthquake

Through MPC-2

Impose co-seismic displacement

Debonding elements (cohesive zone)

Gravity

Initial stress: gh

Club ZéBuLoN

12


Results @3yr for different material parameters

Drift:40 cm

Club ZéBuLoN

13


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

14


Parallel computation: step-by-step

1- mesh & split

2- compute

3- glue

4- visu & post

Club ZéBuLoN

15


3D meshing

Engdhal data too noisy

Extension along fault line

**xyz_to_r

**extension_along_nset_jd

**por

cupin

e

Club ZéBuLoN

16


3D meshing: difference

On a flat Earth

“Hand-made” boolean operation

Yams

Club ZéBuLoN

17


3D meshing: difference

Special care at -80km such that the union process goes fine

Yams 3d gives unsatisfying results

Laplacian on z-coordinate

Club ZéBuLoN

18


3D meshing: union

Club ZéBuLoN

19


3D meshing: ghs3d

c Requires 32 Gb RAM

ghs3d

Club ZéBuLoN

20


Mesh splitting

Splitmesh

**split *domains 60 *mincon 3

**rigid_body *nset est est est ouest *dof U1 U2 U3 U1

Domain 36

Forbids insufficiently attached elements

Detects rigid body movements caused by split& creates the ***bc bloc in .cut file

Club ZéBuLoN

21


Mesh splitting

Metis split (**metis_split)

Optimized mode is not reproducible !!!

**metis_split *domains 60 *check_domains

**rigid_body *nset est est est ouest *dof U1 U2 U3 U1

Club ZéBuLoN

22


Mesh splitting

Split by elset ?

Rigid body movements ?

Load balancing ?

**elset_split *domain lithosphere *domain asthenosphere *domain serpentine *domain subduction

Club ZéBuLoN

23


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

24


BCs & MPCs

Local / not problematic:

**impose_nodal_dof

**gravity

Problematic:

**mpc2 currently requires the global mesh (old format)roundabout if coupled dofs are in the same subdomain

Club ZéBuLoN

25


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

26


Linear solvers

***linear_solver rigid **n_last 15 **local_solver sparse_dscpack

***linear_solver sparse_dscpack ***linear_solver sparse_direct

***linear_solver dissection **ldlt

Detects RBMSlower (x3)Memory consuming

Doesn’t detect rigid body movements (RBM)

Detects RBMStill in development

Detects RBMSlowerUnnecessary after 1st iteration

Club ZéBuLoN

27


FETI solver

Global solver on the interface

Iterative (Conjugate Gradient)

Depends on conditioning number

Mesh quality

Splitting

Material heterogeneity

MPCs

...

Club ZéBuLoN

28


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

29


Elastic computations (fake)

To test the convergence / mesh / inp

Simple traction, 16 domains

FETI converges in

296 iterations (dscpack + mesher_rigid)

403 iterations (sparse_direct)

Pb with dissection (segfault in mkl, during 1st FETI iteration)

***feti **precond lumped **projector direct **reprojection **max_iteration 1000 **precision 1.e-9

**precond dirichlet doesn’t work (?)

Club ZéBuLoN

30


Visco-elastic computations (fake)

Visco-elastic behavior in asthenosphere

Simple traction, 16 domains

FETI converges in

~500 it per increment (dscpack)

Club ZéBuLoN

31


Visco-elastic computation

Realistic BCs

fixed outer boundaries

Earthquake localized in 1 sub-domain

Cohesive Zone Elements

**mpc2

Club ZéBuLoN

32



Attempt 1: dscpack / 60+ domains

FETI ratio doesn’t decrease

Club ZéBuLoN

33





Attempt 2: dscpack / 60 domains / handmade .cut


Matrix becomes singular at iter 5 in domain 30 ( ?? )

Club ZéBuLoN

34





Attempt 2: dscpack / 60 domains / handmade .cut


Matrix becomes singular at iter 5 in domain 30 ( ?? )

Attempt 3: dissection / 60 domains

std::bad_alloc

Club ZéBuLoN

35


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

36


Visualization: Zglue

Most commonly used

Main drawbacks

Large output files

Zmaster becomes memory consuming

Club ZéBuLoN

37


Visualization: Zmaster -PP

xhost +

mpirun -np 1 -hostfile mymachines Zrun -PP -s MPI mpi2 -s Parallel.Mpmd.RunMode -G -s Parallel.Display x3755 mycalcul.inp

Zmaster in parallel mode

Rendering is done on computation nodes

No longer requires the glue process

Experimental (?)

See SQ

Club ZéBuLoN

38


Visualization: Paraview

Paraview

open source equivalent of Ensight

Vtk based

Features

Cut, probe, calculator

~100 other filters

Visualization by sub-domain(s)

Misses Zmaster’s posts & plots

***output Z7 [**other options]

***output ensight_gold

Club ZéBuLoN

39


Outline



2D models

Parallel approach

Meshing & Splitting

BCs & MPCs

Linear solvers

Computations

Visualization

Conclusion

Club ZéBuLoN

40


Conclusion

Parallel computation is a routine on simple cases (elastic / small size)

Becomes difficult when:

Large size (> 50 sub-domains)

Non linearities

Mesh & split quality

Source(s) of problem is not clear

User-time consuming !

Club ZéBuLoN

41


Useful (?) stuff

Zman parallel: troubleshooting

Obscure error messages

***verify_parallel_splitting

Error handler

Attaches a gdb to a dying process

Zrun -s Error.HandlingPolicy auto [...]

// MPMD - Waiting for ACK: 1 2 3 4 5 6 7 8 9 10 [stuck] Output directory doesn’t exist

// MPMD - 4 tasks spawned on sisulu. [stuck] export PVM_ARCH=LINUX

post-seismic relaxation process in southeastern asia - mines … · 2013-09-06 · december 9, 2008...

Documents