high performance computing (planing-usage-application)

8/3/2019 High Performance Computing (Planing-Usage-Application)

1/41

High Performance Computing:Planing, usage, and Application

Yudi Rosandi

Workshop on High Performance Computing:from design to application


2/41

Outline

Planing

Hardware components

Infrastructures

Operating system

Usage

Basic knowledgeCluster Application

Simulation, research and development

Data processing, forecasting


3/41

Meet some Beowulf clusters....


4/41



5/41



6/41



7/41

Planing


8/41

Cluster

collection of computers

interconected one and another via network

working together on a single programor interdependent different programs


9/41

Planing of a cluster

What is the purpose?

How much budget do we have?

What is the expected outcome?


10/41

High availibility cluster


11/41

Load Balancing

multi-user environtment


12/41

Compute cluster (HPC)

single parallel program run in many procs


13/41

Budget...LOW

build from scratch

chalengingprecious experience

troublesome but interresting

freedom

Pitfallsmost problems comes from networking

cable mesh

high maintenance cost

high power consumtion(depends on hardware selection)


14/41

Budget...HIGH

high-end cluster

blade serverready built racks

What we get

production ready

integrated high-speed lan (infiny-band)good cooling system

low power consumption


15/41

HPC Outcome

Research

General sciencesfast publication

many ready to use programs

ability to solve complex problems

insight to unknown aspects in science

Computer science

parallel programming

computer administration

networking

user managementbenchmarking and optimization

Low research cost!


16/41

HPC Outcome

Production

fast computationfast decission making

increasing services

high availability


17/41

Hardware components


18/41

What kind of hardware do we need?

Anything...


19/41

Hardware requirements

Computing nodes

fast computerfast network interface

sufficient memory

local scratch storage

Storage serverdedicated, reliable

large storage

sufficient memory (+8Gb)

high-speed network connection(channel bonding, etc)


20/41

Hardware requirements

Head server(s):reliable, not necessarily fast

dhcp

nis

login

queuing

note:

these servers may be separated individual servers,combined servers in one machine, or virtualized(e.g. Xen)

Login server may need special consideration!


21/41

Common clusterfor high performance computing


22/41

Needed infrastructure

Room temperature control

Stabil electricityPower consumption

networking backbone


23/41

Operating system

Things to consider

stabilityreliability

managability

accessibility

security


24/41

Operating system: LINUX

Why????


25/41

Operating system: LINUX

stability (OK)

reliability (OK)managability (OK)

accessibility (OK)

security (OK)


26/41

Cluster Usage


27/41

Basic knowledge to usea cluster computing facility

know how to:make remote connection

use a shell (csh,sh,bash,etc)

file-system: locate directories, file permission

submit/manage programswrite a parallel program

don't do:

hacking

running malicious programrobbing memory

spamming file-system


28/41

Know your limits!Respect other users!


29/41

Remote login

Accesible from internet

static addressdedicated

Secured internet connection

key authentication

firewall


30/41

Remote login


31/41

Remote login

Files for ssh connection (under ~/.ssh/)

id_rsa (@client)identity file, must be kept 'only' inthe connecting computerSECRET! don't transfer via email, etc!

id_rsa.pub

public key, to be distributed to allservers

known_hosts (@client)

list of host keys of known servers

authorized_keys, authorized_keys2 (@server)

list of public keys of clients/user those allowed to connect.

config (@client)

local user specific configuration


32/41

command to create id keys:ssh-keygen


33/41

Shell

elements

variablesprogram controls

redirection

pipes

customizing shellmake your self convenient

personal commands, path, etc

basic scripting knowledge

creating job-scriptbuilding embarrassingly parallel programs

automating processes


34/41

Shell: bash

Standard in Linux

CompleteFlexible

Advanced

Demo/example....

declaring variable

simple program controls

redirection and pipes


35/41

Application


36/41

Animation movies!(@ pixar)


37/41

Animation movies!(@ pixar)


38/41

Material science (@ TU-KL)


39/41

Material science (@ TU-KL)

a glance at computing center


40/41

Application

incomplete examples...

Research:modelling

nano-sciences

complex-system

Industryimage rendering

data processing

Many others...


41/41

Thank you

high performance computing (planing-usage-application)

Documents