gpu-accelerated scalable solver for banded linear systems

Computational Fluid Dynamics

Solving the Navier-Stokes!Equations in Primitive

Variables!

Grétar Tryggvason !Spring 2013!

http://www.nd.edu/~gtryggva/CFD-Course/!Computational Fluid Dynamics

The projection method—review!Methods for the Navier-Stokes Equations!

!Moin and Kim!!Bell, et al !!

Colocated grids!

Outline!


∇h ⋅ui , jn +1 = 0

Discretization in time!

�

ui, jn +1 − ui, j

n

Δt= −A i, j

n − ∇hPi, j +Di, jn

Summary of discrete vector equations !

No explicit equation for the pressure! !

Evolution of the velocity!

Constraint on velocity!


ui , jn +1 − ui , j

t

Δt= −∇hPi , j ⇒ ui , j

n +1 = ui, jt − Δt ∇hPi, j

ui , jt − ui, j

n

Δt= −Ai, j

n +Di, jn ⇒ ui, j

t = ui, jn + Δt −Ai, j

n + Di, jn( )

�

ui, jn +1 − ui, j

n

Δt= −A i, j

n − ∇hPi, j +Di, jn

Split !

by introducing the temporary velocity ut!

into!

and!

Projection Method!



�

ui, jn +1 = ui, j

t − Δt ∇ hPi, j

∇h ⋅ui , jn +1 = 0

∇h ⋅ui , jn +1 =∇h ⋅ui, j

t − Δt ∇h ⋅∇hPi, j0!

To derive an equation for the pressure we take the divergence of!

and use the mass conservation equation!

∇h2Pi, j =

1Δt

∇h ⋅ui, jt

The result is!

Discretization in time!Computational Fluid Dynamics

ui , jn +1 = ui , j

t − Δt ∇hPi, j

ui , jt = ui, j

n + Δt −Ai , jn +Di, j

n( )

∇h2Pi, j =

1Δt

∇h ⋅ui, jt

1. Find a temporary velocity using the advection and the diffusion terms only:!

2. Find the pressure needed to make the velocity field incompressible!

3. Correct the velocity by adding the pressure gradient:!


Computational Fluid Dynamics Algorithm!

Determine u, v boundary conditions!

Poisson equation for (iteration)!

Initial field given!

t=t+Δt!

Projection!

( )nji

nji

nji

tji t ,,,, DAuu +−Δ+=

jiP ,

ui , jn +1 = ui , j

t − Δt ∇hPi, j

Advect!


pi,j+1! pi+1,j+1! pnx,j+1! pnx+1,j+1!pi-1,j+1!p2,j+1!p1,j+1!

pi,j! pi+1,j! pnx,j! pnx+1,j!pi-1,j!p2,j!p1,j!

pi,j-1! pi+1,j-1! pnx,j-1! pnx+1,j-1!pi-1,j-1!p2,j-1!p1,j-1!

pi,2! pi+1,2! pnx,2! pnx+1,2!pi-1,2!p2,2!p1,2!

pi,1! pi+1,1! Pnx,1! pnx+1.1!pi-1,1!p2,1!p1,1!

vi,j!

vi,j-1! vi+1,j-1! vnx,j-1! vnx+1,j-1!vi-1,j-1!v2,j-1!v1,j-1!

vi,2! vi+1,2! vnx,2! vnx+1,2!vi-1,2!v2,2!v1,2!

vi,1! vi+1,1! vnx,1! vnx+1,1!vi-1,1!v2,1!v1,1!

u i,j!

vi,j! vi+1,j! vnx,j! vnx+1,j!vi-1,j!v2,j!v1,j!

vi,j+1! vi+1,j+1! vnx,j+1! vnx+1,j+1!vi-1,j+1!v2,j+1!v1,j+1!

u i-1,

j!

u 2,j!u 1

,j!

u i+1,

j!

u nx,

j!

u i,j-1!

u i-1,

j,-1!

u 2,j-

1!u 1,j-

1!

u i+1,

-1j!

u nx,

j-1!

u i,2!

u i-1,

2!

u 2,2!u 1,2!

u i+12!

u nx,

2!

u i,1!

u i-1,

1!

u 2,1!u 1,1!

u i+1,

1! u nx,

1!

pi,ny! pi+1,ny! pnx,ny! pnx+1,ny!pi-1,ny!p2,ny!p1,ny!

pi,ny+1! pi+1,ny+1! pnx,ny+1! pnx+1,ny+1!pi-1,ny+1!p2,ny+1!p1,ny+1!

vi,ny! vi+1,ny! vnx,ny! vnx+1,ny!vi-1,ny!v2,ny!v1,ny!

u i,ny!

u i-1,

ny!

u 2,n

y!U1,

ny!

u i+1,

ny!

u nx,

ny!

u i,ny

+1!

u i-1,

ny+1!

u 2,n

y+1!

U1,

ny+1!

u i+1,

ny+1!

u nx,

ny+1!

u i,j+

1!

u i-1,

j+1!

u 2,j+

1!

u 1,j+

1!

u i+1,

j+1!

u nx,

j+1!

�

P(1: nx +1,1: ny +1)u(1: nx,1: ny +1)v(1: nx +1,1: ny)

Array Dimension:!

Computational Grid!

Computational Fluid Dynamics Algorithm!

nx=16;ny=16;dt=0.005;nstep=200;mu=0.1;maxit=100;beta=1.2;h=1/nx; !u=zeros(nx+1,ny+2);v=zeros(nx+2,ny+1);p=zeros(nx+2,ny+2); !ut=zeros(nx+1,ny+2);vt=zeros(nx+2,ny+1);c=zeros(nx+2,ny+2)+0.25; !uu=zeros(nx+1,ny+1);vv=zeros(nx+1,ny+1);w=zeros(nx+1,ny+1); !c(2,3:ny)=1/3;c(nx+1,3:ny)=1/3;c(3:nx,2)=1/3;c(3:nx,ny+1)=1/3; !c(2,2)=1/2;c(2,ny+1)=1/2;c(nx+1,2)=1/2;c(nx+1,ny+1)=1/2; !un=1;us=0;ve=0;vw=0;time=0.0;!!for is=1:nstep !u(1:nx+1,1)=2*us-u(1:nx+1,2);u(1:nx+1,ny+2)=2*un-u(1:nx+1,ny+1); !v(1,1:ny+1)=2*vw-v(2,1:ny+1);v(nx+2,1:ny+1)=2*ve-v(nx+1,1:ny+1);!!for i=2:nx,for j=2:ny+1 % temporary u-velocity ! ut(i,j)=u(i,j)+dt*(-(0.25/h)*((u(i+1,j)+u(i,j))^2-(u(i,j)+u(i-1,j))^2+(u(i,j+1)+u(i,j))*(v(i+1,j)+...! v(i,j))-(u(i,j)+u(i,j-1))*(v(i+1,j-1)+v(i,j-1)))+ (mu/h^2)*(u(i+1,j)+u(i-1,j)+u(i,j+1)+u(i,j-1)-4*u(i,j)));!end,end!!for i=2:nx+1,for j=2:ny % temporary v-velocity ! vt(i,j)=v(i,j)+dt*(-(0.25/h)*((u(i,j+1)+u(i,j))*(v(i+1,j)+ v(i,j))-(u(i-1,j+1)+u(i-1,j))*(v(i,j)+v(i-1,j))+... ! (v(i,j+1)+v(i,j))^2-(v(i,j)+v(i,j-1))^2)+(mu/h^2)*(v(i+1,j)+v(i-1,j)+v(i,j+1)+v(i,j-1)-4*v(i,j)));!end,end!!for it=1:maxit % solve for pressure! for i=2:nx+1,for j=2:ny+1!p(i,j)=beta*c(i,j)*(p(i+1,j)+p(i-1,j)+p(i,j+1)+p(i,j-1)-(h/dt)*(ut(i,j)-ut(i-1,j)+vt(i,j)-vt(i,j-1)))+(1-beta)*p(i,j); !end,end!end!!% correct the velocity!u(2:nx,2:ny+1)=... ut(2:nx,2:ny+1)-(dt/h)*(p(3:nx+1,2:ny+1)-p(2:nx,2:ny+1));!v(2:nx+1,2:ny)=... vt(2:nx+1,2:ny)-(dt/h)*(p(2:nx+1,3:ny+1)-p(2:nx+1,2:ny));!time=time+dt !% plot the results uu(1:nx+1,1:ny+1)=0.5*(u(1:nx+1,2:ny+2)+u(1:nx+1,1:ny+1)); !vv(1:nx+1,1:ny+1)=0.5*(v(2:nx+2,1:ny+1)+v(1:nx+1,1:ny+1)); !w(1:nx+1,1:ny+1)=(u(1:nx+1,2:ny+2)-u(1:nx+1,1:ny+1)-...!v(2:nx+2,1:ny+1)+v(1:nx+1,1:ny+1))/(2*h);!hold off,quiver(flipud(rot90(uu)),flipud(rot90(vv)),'r');!hold on;contour(flipud(rot90(w)),20),axis equal,pause(0.01)!end!


Velocity and pressure! Velocity and vorticity!


Forward in time, centered in space (summary):!

�

u* −u n

Δt= −A u n( ) + ν∇2u n

∇2p = 1Δt

∇ ⋅u*

u n+1 = u* −Δt∇p

�

Δt ≤ 2νU 2 & Δt ≤ 1

4h2

ν

�

U 2 =max(u2 + v 2)

Time step limitations!

where!


Forward in time, centered in space:!

�

Δtadv ≤2νU 2 & Δtdiff ≤

14h2

ν

�

τ = LU

�

Δt '= Δtτ

≤ 2νU 2

UL

= 2Re

�

Δt '= Δtτ

≤ 14h2

νUL

= 14

hL

⎛ ⎝ ⎜

⎞ ⎠ ⎟ 2

Re

Define!

Therefore the nondimensional time step is:!

And Δt à 0 for Re à 0 and Re à ∞!

and!


Forward in time, centered in space:!

�

Δt '= 2Re

�

Δt '= 14hL

⎛ ⎝ ⎜

⎞ ⎠ ⎟ 2

Re

What is the maximum timestep?!

Δt à 0 for Re à 0 and Re à ∞!

and!

Increasing h!

h à 0! Re!

Δt’!

�

Δt 'max = 12hL�

ReΔt ' max = 8 Lh


Advanced Solvers!

For low Re, use implicit methods for diffusion term!For high Re, use stable advection schemes!!Combine both for schemes intended for all Re !


Higher Order in Time!


Predictor-Corrector!

�

u* −u n

Δt= −∇u nu n + ν∇2u n

∇2p* = 1Δt

∇ ⋅u*

u1 = u* −Δt∇p*

�

u ** − u1

Δt= −∇u1u1+ν∇2u1

∇2p** = 1Δt

∇⋅u **

u 2 = u ** − Δt∇p**

�

u n+1 = 12u n + u 2( )

Then average the results!

Backward step using the predicted velocity:!

A second order method can be developed by first taking a forward step, then a backward step and average the results:!


�

u * − u n

12 Δt

= −∇u nu n +ν∇2u n

∇2p* = 2Δt

∇⋅u *

u1 = u * − Δt2∇p*

�

u ** − u n

12 Δt

= −∇u1u1+ν∇2u1

∇2p** = 2Δt

∇⋅u **

u 2 = u ** − Δt2∇p**

continue for the second step:!

First a half step:!A complete Runge-Kutta time integration (Weinan E.) !


�

u *** − u n

Δt= −∇u 2u 2 +ν∇2u 2

∇2p*** = 1Δt

∇⋅u ***

u 3 = u *** − Δt∇p***

�

k = Δt −∇u 3u 3 +ν∇2u 3( )

�

u+ = 13 −u n + u1+ 2u 2 + u 3( ) + 1

6 k

∇2p+ = 1Δt

∇⋅u+

u n+1 = u+ − Δt∇p+

And finally!

Then compute !

Take a full step using the predicted velocity!

A complete Runge-Kutta time integration (continued) !


�

u * − u n

12 Δt

= −∇u nu n +ν∇2u n

u ** − u *12 Δt

= −∇u *u * +ν∇2u *

u *** − u n

Δt= −∇u *u * +ν∇2u *

u+ = 13 −u n + u * + 2u ** + u ***( ) + Δt

6−∇u ***u *** +ν∇2u ***( )

∇2p+ = 1Δt

∇⋅u+

u n+1 = u+ − Δt∇p+

Simplified Fourth order method!


Other Methods!


Fully Implicit!

�

un +1 − un

Δt=12

− A (un ) +A(un +1)( ) +ν ∇h2un + ∇ h

2un+1( )[ ] −∇p∇ ⋅un+1 = 0

Solve by iteration!

Rarely used due to the complications of the nonlinear system that must be solved for the advection terms!


Adam-Bashford/Crank-Nicolson!

�

un +1 − un

Δt= −

32A(un ) − 1

2A(un−1)⎛

⎝ ⎞ ⎠ +

ν2

∇h2un + ∇ h

2un+1( ) −∇p∇ ⋅un+1 = 0

�

˜ u −un

Δt= − 3

2A (un ) − 1

2A (un−1)⎛

⎝ ⎞ ⎠ +

ν2∇ h

2un

un +1 - ˜ u Δt

= −∇p +ν2∇ h

2un+1

∇ ⋅un+1 = 0

�

∇ h2p =

1Δt

∇ ⋅ ˜ u

Split:!

The correction equation is implicit and must be solved by an iteration in the same way as the pressure equation!


Method of Kim and Moin (JCP 59 (1985), 8-23)!

�

˜ u −un

Δt= − 3

2A (un ) − 1

2A (un−1)⎛

⎝ ⎞ ⎠ +

ν2

∇ h2un + ∇h

2 ˜ u ( )un +1 − ˜ u

Δt= −∇φ

∇ ⋅un+1 = 0

�

∇ h2φ =

1Δt

∇⋅ ˜ u

The first equation is implicit and must be solved by an iteration in the same way as the pressure equation!


�

un +1 − un

Δt= −

32

A(un ) − 12

A(un−1)⎛ ⎝

⎞ ⎠ +

ν2

∇h2un + ∇ h

2un+1( ) +ν2

∇h2 ˜ u −∇ h

2un+1( ) −∇φ

�

ν2∇h

2 ˜ u −∇ h2un+1( ) −∇φ =

ν2∇ h

2φ −∇φ = ∇p

Where we have added and subtracted an implicit diffusion term.!

�

un +1 − ˜ u Δt

= −∇φ

we can rewrite the last terms as:!

�

∇p

Using!

Notice that Φ is not exactly p. Adding the first two equations gives!


Method of Bell, Colella and Glaz (JCP 85 (1989), 7-83)!

�

un +1 − un

Δt= −A (un+1/ 2) +

ν2

∇ h2un + ∇h

2un +1( ) − ∇p∇ ⋅un+1 = 0

A Godunov method is used for the advection terms!


Many other methods have been proposed!!SIMPLE (Semi-Implicit Method for Pressure-Linked Equations). One of the earliest method for steady state problems. Many extensions.!!PISO (Pressure Implicit with Split Operator). Similar to the projection method but iterates to enforce incompressibility!


Colocated grids!


Colocated grids!Although staggered grids have been very successful, in some cases it is desirable to use co-located (or colocated) grids where all variables are located at the same physical point. !


vi,j+1/2!

Pi,j!ui-1/2,j! ui+1/2,j!

vn!

uw! ue!

vs!vi,j-1/2!

Pi,j,ui,j,vi,j!

Colocated grids!Staggered grids!

All variables are stored at the same location!


First idea: use averaging for the variables on the edges:!�

ui, j* = ui, j − ΔtA(ui, j

n )

ui, jn+1 = ui, j

* − Δth(pe − pw )

uen+1 − uw

n+1 + vnn+1 − vs

n+1 = 0

�

pe =12pi+1, j + pi, j( )

�

ui, jn+1 = ui, j

* − Δth12pi+1, j + pi, j( ) − 12 pi, j + pi−1, j( )⎛

⎝ ⎜

⎞ ⎠ ⎟

�

ue =12ui+1, j + ui, j( )

�

ui, jn+1 = ui, j

* − Δt2h

pi+1, j − pi−1, j( )

Split equations for the u velocity!


Substituting!

�

ui, jn +1 = ui, j

* −Δt2h

pi+1, j − pi, j−1( )into!

�

uen +1 − uw

n+1 + vnn +1 − vs

n +1 = 0

yields!

�

pi+2, j + pi− 2, j + pi, j+2 + pi, j− 2 − 4 pi, j =2hΔt

ui+1, j* − ui−1, j

* + vi, j+1* −vi, j−1

*( )

�

ue =12ui+1, j + ui, j( ) and!


The pressure points are also uncoupled and the pressure field can develop oscillations!

A straight forward application discretization on colocated grids results in a very wide stencil for the pressure!


The remedy is to find the pressures that make the edge velocities incompressible!

Rhie and Chow. AIAA Journal. 21 (1983), 1525-1532.!


vn!

uw! ue!

vs!

Pi,j,ui,j,vi,j!

Instead of interpolating (the final velocity)!

�

uen +1 = ue

* −Δth

pi+1, j − pi, j( )

�

ue* =12ui+1, j* + ui, j

*( )

�

uen +1 =

12ui+1, jn +1 + ui, j

n+1( )

interpolate (the intermediate velocity)!

and then find!

In effect, “pretend” we are using a staggered grid!!

The Rhie and Chow method!


�

uen +1 = ue

* −Δth

pi+1, j − pi, j( )

�

uen +1 − uw

n+1 + vnn +1 − vs

n +1 = 0

�

ue* =12ui+1, j* + ui, j

*( )

�

ue* −Δth

pi+1, j − pi, j( )⎛ ⎝

⎞ ⎠ − uw* −

Δth

pi, j − pi−1, j( )⎛ ⎝

⎞ ⎠

+ vn* −Δth

pi, j+1 − pi, j( )⎛ ⎝

⎞ ⎠ − vs* −

Δth

pi, j − pi, j−1( )⎛ ⎝

⎞ ⎠ = 0

�

pi+1, j + pi−1, j + pi, j+1 + pi, j−1 − 4 pi, j =hΔt

ue* − uw

* + vn* − vs

*( )Rearrange:!

Substitute!

where!into!

giving!


Substituting for the velocities:!

�

pi+1, j + pi−1, j + pi, j+1 + pi, j−1 − 4 pi, j =h2Δt

ui+1, j* − ui−1, j

* + vi, j+1* −vi, j−1

*( )

�

ui, jn +1 = ui, j

* −Δt2h

pi+1, j − pi−1, j( )

�

pe =12pi+1, j + pi, j( )

For the correction of the momentum equation we still use the average of the pressures!

giving!


The boundary conditions for the velocity are now very simple: The velocity at nodes on the wall is simply the wall velocity.!!The pressure boundary is more complex:!


Find the pressure gradient by applying the Navier-Stokes equations at a point at the boundary!

�

ρ∂v∂t

+ u∂v∂x

+ v∂v∂y

⎛ ⎝ ⎜ ⎞

⎠ = −

∂p∂y

+ µ∂ 2v∂x 2

+∂ 2v∂y2

⎛ ⎝ ⎜ ⎞

⎠ ⎟

At the wall, most of the terms are zero!

�

∂p∂y

= µ∂ 2v∂y 2

�

pi,2 − pi,1h

= µ∂ 2v∂y 2

⎞ ⎠ ⎟ i,1

Evaluated by one-sided differences!

�

i,1

�

i,2

�

pi,2 − pi,1h

= µ vi,1n − 2vi,2

n + vi,3n

h2


Define:!

We can write:!

�

i,1

�

i,2

�

pi,2 − pi,1h

= µ vi,1n − 2vi,2

n + vi,3n

h2

�

pi,2 − pi,1 = hΔtvi,1*

�

vi,1* = Δt

hpi,2 − pi,1( ) = µΔt

h2vi,1n − 2vi,2

n + vi,3n( )


Write the pressure equation for j=2!

�

pi,2 − pi,1 =hΔtvi,1*

�

pi+1,2 + pi−1,2 + pi,3 + pi,1 − 4 pi,2 =h2Δt

ui+1,2* − ui−1,2

* + vi,3* −vi,1

*( )

And use!

For the pressure at j=1!


The algorithm is therefore:!

1. First find predicted velocities:!

�

ui, j* = ui, j − ΔtA(un )

2. Find pressure by solving:!

�

pi+1, j + pi−1, j + pi, j+1 + pi, j−1 − 4 pi, j =h2Δt

ui+1, j* − ui−1, j

* + vi, j+1* −vi, j−1

*( )

3. Correct the velocities:!

and!

�

ui, jn +1 = ui, j

* −Δt2h

pi+1, j − pi−1, j( ) and!

suitably modified at the boundaries!

�

vi, j* = vi, j − ΔtA(vn )

�

vi, jn+1 = vi, j

* −Δt2h

pi, j+1 − pi. j−1( )


% projection method in primitive variables on a colocated mesh nx=19;ny=19;dt=0.0025;nstep=300;mu=0.1;maxit=100;beta=1.2;h=1/nx; time=0; u=zeros(nx,ny);v=zeros(nx,ny);p=zeros(nx,ny); ut=zeros(nx,ny);vt=zeros(nx,ny); u(2:nx-1,ny)=1.0; for is=1:nstep for i=2:nx-1,for j=2:ny-1 ut(i,j)=u(i,j)+dt*(-(0.5/h)*(u(i,j)*(u(i+1,j)-u(i-1,j))+v(i,j)*(u(i,j+1)-u(i,j-1)))+(mu/h^2)*(u(i+1,j)+u(i-1,j)+u(i,j+1)+u(i,j-1)-4*u(i,j))); vt(i,j)=v(i,j)+dt*(-(0.5/h)*(u(i,j)*(v(i+1,j)-v(i-1,j))+v(i,j)*(v(i,j+1)-v(i,j-1)))+ (mu/h^2)*(v(i+1,j)+v(i-1,j)+v(i,j+1)+v(i,j-1)-4*v(i,j))); end,end vt(2:nx-1,1)=(mu*dt/h^2)*(v(2:nx-1,3)-2*v(2:nx-1,2)); vt(2:nx-1,ny)=(mu*dt/h^2)*(v(2:nx-1,ny-2)-2*v(2:nx-1,ny-1)); ut(1,2:ny-1)=(mu*dt/h^2)*(u(3,2:ny-1)-2*u(2,2:ny-1)); ut(nx,2:ny-1)=(mu*dt/h^2)*(u(nx-2,2:ny-1)-2*u(nx-1,2:ny-1)); for it=1:maxit % solving for pressure p(2:nx-1,1)=p(2:nx-1,2)+(h/dt)*vt(2:nx-1,1); p(2:nx-1,ny)=p(2:nx-1,ny-1)-(h/dt)*vt(2:nx-1,ny); p(1,2:ny-1)=p(2,2:ny-1)+(h/dt)*ut(1,2:ny-1); p(nx,2:ny-1)=p(nx-1,2:ny-1)-(h/dt)*ut(nx,2:ny-1); for i=2:nx-1,for j=2:ny-1 p(i,j)=beta*0.25*(p(i+1,j)+p(i-1,j)+p(i,j+1)+p(i,j-1)- (0.5*h/dt)*(ut(i+1,j)-ut(i-1,j)+vt(i,j+1)-vt(i,j-1)))+(1-beta)*p(i,j); end,end

p(floor(nx/2),floor(ny/2))=0.0; % set the pressure in the center. Needed since bc is not incorporated into eq end u(2:nx-1,2:ny-1)=ut(2:nx-1,2:ny-1)-(0.5*dt/h)*(p(3:nx,2:ny-1)-p(1:nx-2,2:ny-1)); v(2:nx-1,2:ny-1)=vt(2:nx-1,2:ny-1)-(0.5*dt/h)*(p(2:nx-1,3:ny)-p(2:nx-1,1:ny-2)); time=time+dt hold off,quiver(flipud(rot90(u)),flipud(rot90(v)),'r'); hold on;axis([1 nx 1 ny]);axis square,pause(0.01) end plot(10*u(floor(ny/2),1:nx)+floor(nx/2),1:nx),hold on; plot([1,nx],[floor(ny/2),floor(ny/2)]),plot([floor(nx/2),floor(nx/2)],[1,ny]) plot(10*v(1:nx,floor(ny/2))+floor(ny/2));axis square,axis([1,nx, 1,ny])


Colocated grids!

2 4 6 8 10 12 14 16 18

2

4

6

8

10

12

14

16

18


Why colocated grids:!!Sometimes simpler for body fitted grids!!Easy to use methods for hyperbolic equations!!Easier to implement AMR!!Some people just don’t like staggered grids!!


The two-dimensional programs developed in the project and shown here can be extended to fully three-dimensional flows in a relatively straight forward way, replacing u(i,j) by u(i,j,k), etc. The time required to run the code increases significantly and visualizing the output becomes more challenging. !

gpu-accelerated scalable solver for banded linear systems

Documents