reaching the goal with the regensburg marathon cluster · 2002-09-05 · reaching the goal with the...
TRANSCRIPT
withon Cluster
ect -
er.de>
Reaching the Goalthe Regensburg Marath
- A NetBSD Cluster Proj
Hubert Feyrer <hubert@feyr
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
2/32
and R-KOM
Hubert Feyrer <[email protected]>
Introduction
• 5.500 runners
• Cooperation between FH Regensburg
• 45 machines
• Video rendering
• 100% Open Source based
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
3/32
ardware
install
arddisk
arddisk
Hubert Feyrer <[email protected]>
Cluster Client Setup: H
• Four public rooms with 15 machines
• 15 machines with Solaris preinstalled
• Remaining machines available for re
• Hardware: Dell OptiPlex PCs
- PII-500MHz, 64MB RAM, 4GB h
- PIII-1GHz, 256MB RAM, 10GB h
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
4/32
oftware
ftware collection
Hubert Feyrer <[email protected]>
Cluster Client Setup: S
• Chosen node OS: NetBSD
- Supports the hardware
- Easy to install
- Know-how available in-house
- Software available in 3rd party so
• Cluster software:
- dumpmpeg, mpeg_encode
- tload, ucd_snmp, statd
• Image cloning: g4u
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
5/32
ployment
Hubert Feyrer <[email protected]>
Cluster Client Setup: De
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
6/32
ter
Hubert Feyrer <[email protected]>
Tasks of the Clus
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
7/32
EG Sequences
into single images
Hubert Feyrer <[email protected]>
Cluster Task #1: Splitting MP
• Splitting sequences of the input video
• 11 minutes per sequence
• 16.500 resulting images
• 45 minutes on 1GHz machines
• Software: dumpmpeg
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
8/32
sations (I)
d smpeg
tools
Hubert Feyrer <[email protected]>
Cluster Task #1: Optimi
• dumpmpeg writes BMP per default
- we needed JPG for the 2nd step
- sizeof(BMP) >> sizeof(JPG)
• No JPEG-writing routines in SDL an
• Source code changed to use NetPBM
• After 250 BMPs written to disk,
batch conversion to JPG in one run
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
9/32
ations (II)
e expensive)
not done
0 images each in
Hubert Feyrer <[email protected]>
Cluster Task #1: Optimis
• Replacing external calls (fork/exec ar
with NetPBM and jpeg lib functions
(ENOTIME)
• Improving access times by placing 25
their own directory
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
10/32
p
of first and last
quence
c due to thermal
curacies
usable results over
Hubert Feyrer <[email protected]>
Intermediate Ste
• For each sequence, record exact time
image into a MySQL datebase
• Calculate actual framerate for this se
• Framerate is not always 25 frames/se
effects and resulting mechanical inac
• A small difference could add up to un
5 hours of video material
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
11/32
g videos (I)
ing the goal
000 starters)
d men
Hubert Feyrer <[email protected]>
Cluster Task #2: renderin
• Render videos for each runner reach
• 5.500 runners (reaching the goal; >7.
• Three disciplines:
- Marathon (42km)
- Half-marathon (21km)
- Speed skating (21km)
• Seperate lists of results for women an
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
12/32
videos (II)
ctory
Hubert Feyrer <[email protected]>
Cluster Task #2: rendering
• Image selection:
• Images were copied to a working dire
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
13/32
videos (III)
nner,
Hubert Feyrer <[email protected]>
Cluster Task #2: rendering
• Credit frames include data for the ru
written into a template:
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
14/32
videos (IV)
l:
Hubert Feyrer <[email protected]>
Cluster Task #2: rendering
• Image of the runner reaching the goa
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
15/32
videos (V)
ine,
gly
the nodes
k to NFS storage
n collects and
Hubert Feyrer <[email protected]>
Cluster Task #2: rendering
• Software: mpeg_encode
• First send a few images to each mach
to estimate machine speed
• Distribute remaining images accordin
• Images are read from NFS storage by
• Resulting video-parts are written bac
• The master mpeg_encode process the
merges the video-parts at the end
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
16/32
g videos (VI)
o four subclusters:
each subcluster
Hubert Feyrer <[email protected]>
* Cluster Task #2: renderin
• The available machines were split int
• Seperate mpeg_encode config file for
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
17/32
videos (VII)
le, containing
e goal
Hubert Feyrer <[email protected]>
Cluster Task #2: rendering
• List of results was available as CSV fi
name, place and time
• For each runner:
- Prepare working dir with images
- Render video
- Store video
- Store image of runner reaching th
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
18/32
videos (VIII)
ccessing the cluster
ead:
Hubert Feyrer <[email protected]>
Cluster Task #2: rendering
• mpeg_encode used rsh (not ssh!) for a
nodes to prevent authentication overh
- rendering MPEG: 3-8 s
- ssh authentication: 2 s
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
19/32
d
Hubert Feyrer <[email protected]>
Experiences
• Deployment took longer than expecte
• dumpmpeg has problems on Solaris
• dumpmpeg ran longer than expected
• mpeg_encode doesn‘t scale infinitely
• mpeg_encode sometimes hangs
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
20/32
ment
30min (for setup of
one room server
achines fighting
e same switch,
r room
Hubert Feyrer <[email protected]>
Experiences: Deploy
• Image size: 650MB
• Deployment of one image took about
room server)
• Deployment of 11 / 14 machines from
took rather long (>2h) due to many m
over network bandwidth and disk IO
• All client nodes were connected to th
possible improvement: one switch pe
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
21/32
Solaris (I)
nd Linux
e on Solaris
malloc(3)
ers more serious
15 machines!
loc would probably
Hubert Feyrer <[email protected]>
Experiences: dumpmpeg &
• dumpmpeg worked fine on NetBSD a
• dumpmpeg sporadically dumped cor
• some poking in gdb shows crashes in
• probably overwritten memory
• Guess: Solaris takes overwritten buff
than NetBSD and Linux
• No quick fix was available, so we lost
• In retrospect, linking with libbsdmal
have helped
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
22/32
Solaris (II)
al target platform,
Hubert Feyrer <[email protected]>
Experiences: dumpmpeg &
• With more time and testing on the re
this could have been avoided.
• Not all the world is Linux!
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
23/32
too slow
lit w/ 1GHz
rs of video input,
isk IO on the local
tc.
Hubert Feyrer <[email protected]>
Experiences: dumpmpeg
• 18min test sequence took 60min to sp
• For 12 machines running through 5 h
we estimated 5 hours.
• In reality, the machines took 8 hours.
• Possible reasons here are related to d
disk and NFS storage, network load e
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
24/32
& # of nodes
omputed
machines into
handling scripts
hich subcluster
Hubert Feyrer <[email protected]>
Experiences: mpeg_encode
• A sequence of 156 images cannot be c
on more than about 15 machines
• As a result, we did split the available
several subclusters
• Minor adjustments of config files and
was needed
• Scheduling of which lists to run on w
was done manually.
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
25/32
de hangs
peg_encode
re‘s no obvious
Hubert Feyrer <[email protected]>
Experiences: mpeg_enco
• After printing „Wrote 160 frames“, m
• sometimes hangs
• After some quick code inspection, the
• reason what‘s happening.
• Workaround was to
- ^C the program
- edit the list of runners to process,
removing the ones already done
- restart the subcluster in question
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
26/32
ite=blue,
achines and the
een=client write):
Hubert Feyrer <[email protected]>
Some stats
• Disk utilisation of the NFS server (wr
read=green):
• Network traffic between the cluster m
control machine (blue=client read, gr
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
27/32
tting sequences:
Hubert Feyrer <[email protected]>
More stats (I)
• System load (load average) while spli
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
28/32
all eng^Wnodes:
Hubert Feyrer <[email protected]>
More stats (II)
• The cluster running at full steam on
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
29/32
936
.5 GB
B
Hubert Feyrer <[email protected]>
Some numbers
• Participants: 5.501
• Available computers: 57
• Running time of video tapes: 5 h
• Number of images after step #1: 669.
• Diskspace of images after step #1: 17
• Average size of image (JPEG): 27 kB
• Average size of video (MPEG): 987 k
• Overall data images: 150 MB
• Overall data video: 5.4 GB
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
30/32
EGs
m JPEGs
g)
ies
hines
Hubert Feyrer <[email protected]>
Software
• dumpmpeg: splittimg MPEG into JP
• mpeg_encode: rendering MPEGs fro
• SDL, smpeg, NetPBM: for dumpmpe
• perl, gimp, ImageMagick: misc utilit
• tload, xmeter: node monitoring
• g4u: image deployment
• NetBSD: OS of the cluster client mac
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
31/32
r Team
Hubert Feyrer <[email protected]>
The Marathon Cluste
• Hubert Feyrer
• Jürgen Mayerhofer
• Oliver Melzer
• Daniel Ettle
• Christian Krauss
• Tino Hirschmann
• Fabian Abke
• Udo Steinegger
Reaching the Goal with the Regensburg Marathon-Cluster - A NetBSD Cluster Project
32/32
Hubert Feyrer <[email protected]>Thanks!
Questions?
- Hubert Feyrer