LES

1) 2) / 3) 4)

OpenFOAM , 2013927() UDX 6 F

1),2),3), 1),4)

HPCI () (283)

Knowledge DataBase, KDBHPC HPC/PF

FrontFlow/violet (FFV) KDB

OpenFOAM (104106

) LES

() OpenFOAM LES [1]

AICS (276) LES

LES RIAM-COMPACT OpenFOAMFFV-C OpenFOAMFFV-C()

257127630

( )

( )

( )

[1] , - 2 -, J. of Wind Engineering, Vol.29, No.3, 35 (2004).

RIAM-COMPACT (, ) (33622667=5M) OpenFOAM

9h 3h

12.2h

10h

h

h = 0.1 [m]

U = 1.5 [m/s]

(Side)

:(1.5[m/s])::::

2

:2

()210-4

10-5[m2/s] (1.5104)

()20

()5(15 20 )

(a) Smagorinsky (Cs = 0.2, van driest ) OpenFOAM (b)WALE (Cw = 0.5) (c)k- SST OpenFOAM (d)Spalart-Allmaras OpenFOAM

LES

RANS

(a) Smagorinsky (b) WALE

(c) k- SST (d) Spalart-Allmaras

()

(a) Smagorinsky (b) WALE

(c) k- SST (d) Spalart-Allmaras

[1] ,- 2 -, J. of Wind Engineering, Vol.29, No.3, 35 (2004).

x/h = 8

(x )

x/h =8.6

x/h = 6.5x/h = 5.3

x/h = 10

(b) WALE (x/h = 0.8)

(a) Smagorinsky (b) WALE

(c) k- SST (d) Spalart-Allmaras

(a) (b)

(c) (d)

( Smagorinsky)

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

(WALE) (b)

(c) (d)

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

(k- SST)

(b)

(c) (d)

LES

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

(Spalart-Allmaras)

(b)

(c) (d)

(c) LES

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

(a) (b)

(c) (d)

(LES )

2

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

T = 20

8 672452134=40M (refineMesh )

y+min

(a) (b)

(c)(d)

(WALE)

(a) 4000

(b) 500

t = 20

(a) (b)

(c) (d)

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

(a) (b)

(c) (d)

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k)

LES

PI*h 2.5*PI*h

2*h

Ret = 300

Kasagi, DNS (CH12_PG.WL8)

(a) : (Nx, Ny, Nz) = (1024, 64, 1024) N = 6710 (b) : (Nx, Ny, Nz) = (2048, 64, 2048) N = 2.68

2048 90%

(a) 6710(b) 2.68

91.3%

63.7%

MPI X( Y Z 3Dmapping*

128 16 4 2 4x25x32 8* 2x4x2 256 16 4 4 4x25x16 8* 2x4x4 512 16 4 8 4x25x8 8* 2x4x81024 32 4 8 2x25x8 8* 4x4x8

X8MPI( XYZD MPI TOFU X

TOFU 3D

MPI

TOFU 3D

64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 7980 81 82 83 84 85 86 87 88 89 90 91 92 93 94 9596 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127

x

yz

(16x4x2)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1516 17 18 19 20 21 22 23 24 25 26 27 28 29 30 3132 33 34 35 36 37 38 39 40 41 42 43 44 45 46 4748 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

0 1 2 34 5 6 7

(a) 8* 4x2x2

8 9 10 1112 13 14 15

0 1 X0 8 Z 0 2 Y

3D (128 MPI )

X 8 rank 1node ( In node node )

8 910 1112 1314 15

0 12 34 56 7

(b) 8* 2x4x2 ( mapping)

y z

x

y z

x

ExecutionTime = 578.67 [s] ExecutionTime = 456.36 [s]

*

------------------------- MPI Information ------------------------- ------------------------- MPI Information -------------------------Dimension 3 Dimension 3

Shape 4x2x2 Shape 2x4x2---------------------- MPI Memory Usage (MiB) --------------------- ---------------------- MPI Memory Usage (MiB) --------------------- MAX MIN AVE MAX MIN AVEEstimated_Memory_Size 38.64 [ 16] 31.95 [ 1] 35.21 Estimated_Memory_Size 38.64 [ 16] 31.95 [ 1] 35.21------------------- Per-peer Communication Count ------------------ ------------------- Per-peer Communication Count ------------------ MAX MIN AVE MAX MIN AVEIn_Node 1459760 [ 4] 726340 [ 15] 1271994.6 24% In_Node 1459760 [ 4] 726340 [ 15] 1271994.6 24%Neighbor 2903450 [ 0] 1442631 [ 1] 1714077.0 32% Neighbor 4340001 [ 32] 2163948 [ 1] 3247048.7 61%Not_Neighbor 4332923 [ 16] 721317 [ 1] 2344627.1 44% Not_Neighbor 2163949 [ 39] 0 [ 1] 811655.5 15%Total_Count 7242392 [ 16] 3611606 [ 1] 5330698.8 100% Total_Count 7242392 [ 16] 3611606 [ 1] 5330698.8 100%Connection 120 [ 0] 4 [ 1] 8.2 Connection 120 [ 0] 4 [ 1] 8.2Max_Hop 4 [ 0] 2 [ 1] 3.2 Max_Hop 4 [ 0] 2 [ 1] 2.4Average_Hop 2.25 [ 81] 1.17 [ 7] 1.77 Average_Hop 2.13 [ 0] 1.17 [ 8] 1.41----------------- Per-peer Transmission Size (MiB) ---------------- ----------------- Per-peer Transmission Size (MiB) ---------------- MAX MIN AVE MAX MIN AVEIn_Node 11389.36 [ 66] 5279.81 [ 112] 9476.69 In_Node 11389.36 [ 66] 5279.81 [ 112] 9476.69Neighbor 7544.89 [ 63] 1416.72 [ 5] 2849.56 Neighbor 8271.16 [ 16] 1934.60 [ 5] 3686.03Not_Neighbor 1389.00 [ 16] 507.50 [ 2] 870.23 Not_Neighbor 187.47 [ 96] 0.00 [ 1] 33.75Total_Size 13850.11 [ 111] 12582.21 [ 5] 13196.48 Total_Size 13850.11 [ 111] 12582.21 [ 5] 13196.48----------------- Per-protocol Communication Count ---------------- ----------------- Per-protocol Communication Count ---------------- MAX MIN AVE MAX MIN AVEEager 6837492 [ 16] 2769920 [ 126] 4623577.4 Eager 6837491 [ 16] 2769920 [ 126] 4623577.9Rendezvous 911739 [ 66] 404877 [ 15] 707121.3 Rendezvous 911738 [ 66] 404877 [ 15] 707120.9Unexpected_Message 2 [ 0] 2 [ 0] 2.0 Unexpected_Message 2 [ 0] 2 [ 0] 2.0----------- Tofu Barrier Collective Communication Count ----------- ----------- Tofu Barrier Collective Communication Count ----------- MAX MIN AVE MAX MIN AVEAllreduce 1565271 [ 0] 1565271 [ 0] 1565271.0 Allreduce 1565271 [ 0] 1565271 [ 0] 1565271.0--------- Non-Tofu-specific Collective Communication Count -------- --------- Non-Tofu-specific Collective Communication Count --------Allreduce 25003 [ 0] 25003 [ 0] 25003.0 Allreduce 25003 [ 0] 25003 [ 0] 25003.0------------------------- Process Mapping ------------------------- ------------------------- Process Mapping -------------------------(0,0,0) 0,1,2,3,4,5,6,7 (0,0,0) 0,1,2,3,4,5,6,7(1,0,0) 8,9,10,11,12,13,14,15 (1,0,0) 8,9,10,11,12,13,14,15(2,0,0) 16,17,18,19,20,21,22,23 (0,1,0) 16,17,18,19,20,21,22,23

3D

(b) 8*2x4x2 ( mapping)(a) 8*4x2x2

20%

LES (Cs = 0.2 ) Smagorinsky WALE

() OpenFOAM LES

RANS LES

2.68 2048 90%TOFU 3D MPI 20

OpenFOAM ()

100 TF/s (6400 core ) 1OpenFOAM

7-8

100TF/s (6400core)

500GF/s (32core)

10PF/s (640000 core)

10TF/s (640core)

http://www.top500.org/

HPCI

hp130018 (

CAE V&V ()

OpenFOAM ()

Kasagi DNS (CH12_PG.WL8)

PI*h 2.5*PI*h

2*h

Ret = 300

http://thtlab.jp/index-orig-j.html

(256, 64, 256)

: ()::

LES (a) Smagorinsky (Cs = 0.1, van driest )(b) Smagorinsky (c) WALE ()(d) CSM ()

2

2

0.2, 50000

DNS 5%