image re-slicing for parallel computing im&t advanced scientific computing mark sedrak |...
TRANSCRIPT
Image Re-slicing for Parallel Computing
IM&T ADVANCED SCIENTIFIC COMPUTING
Mark Sedrak | Supervised by Darren Thompson & Sam Moskwa13 February 2013 | Big Day In - Summer Vacation Project
• Aim: To improve the existing cluster re-slicing routine in X-TRACT with Parallel Computing.
• Moore’s Law: Hardware and Data expansion.
• To be covered:– Image Re-Slicing.– Parallel Computing and the use of Super Computers.– My work through-out the project
Image Re-Slicing for Parallel Computing| Mark Sedrak
Project Introduction
2 |
• What is it?– Slices from a CT reconstruction
• Synchrotron• MRI
Image Re-slicing
3 | Image Re-Slicing for Parallel Computing| Mark Sedrak
X-ray imaging tools for HPC clusters and the Cloud | Darren Thompson
Reconstructed Image
4 |
• Uses:– Medical Imaging – Image reconstruction (Materials, Objects, etc)
• XTRACT– Software developed by CSIRO
• Data Sizes
Image Re-slicing
5 |
N / M*
N2 float(projection /
slice)
NM float (sinogram)
N2M float (all
sinograms)
N3 float (all slices)
1k / 720 4 MB 2.8 MB 2.8 GB 4 GB2k / 1,440 16 MB 11¼ MB 22½ GB 32 GB4k / 2,880 64 MB 45 MB 180 GB 256 GB8k / 5,760 256 MB 180 MB 1.4 TB 2 TB
Image Re-Slicing for Parallel Computing| Mark Sedrak
• Serial vs. Parallel Programming– Serial: Instructions are executed one-by-one in sequence.– Parallel: Instructions can be executed simultaneously.
• Splits the work
• Aspects of Parallel Systems• Communication
– Embarrassingly Parallel, Coarse-Grain Parallel, Fine-grain Parallel
• Memory– Shared Memory, Distributed Memory
• Problem Definition– Data Parallel, Task Parallel
Parallel Computing
6 | Image Re-Slicing for Parallel Computing| Mark Sedrak
• Super Computers– Clusters
• TBI “Mini Cluster”, MASSIVE• Bragg: Dual 8-Core CPU’s, 128GB RAM, 40Gb/s InfiniBand• Burnet (Specs): Dual 6-Core CPU’s, 48/96 GB RAM, 40Gb/s InfiniBand
– File Systems• GPFS, HNAS
• Message Passing Interface (MPI)– A Framework for sharing information between distributed memory
processes– Different communication types: 1-1, 1-Many, Many-to-Many– Synchronous vs. Asynchronous Communication
Supercomputers and Message Parsing
7 | Image Re-Slicing for Parallel Computing| Mark Sedrak
• Optimising the re-slicing routine– Generic and portable
• Three main aspects– Communication– Computation/Shuffling– File I/O (Input/Output)
• Developed Three Main Methods – Method 1: (Single Mass communication, High Memory)– Method 2: (Multiple Smaller Communication, High Memory)– Method 3: (Multiple Smaller Communication, Low Memory)
My Project
8 | Image Re-Slicing for Parallel Computing| Mark Sedrak
Results - 2k dataset
9 | Image Re-Slicing for Parallel Computing| Mark Sedrak
0 5 10 15 20 25 30 350
50
100
150
200
250
300
350
Method1 - Bragg0 10 20 30 40 50 60 70
0
50
100
150
200
250
300
350
400
Method 1 - Burnet
0 5 10 15 20 25 300
50
100
150
200
250
300
350
Method 2- Bragg
0 5 10 15 20 25 30 350
50
100
150
200
250
300
350
400
Method 2 - Burnet
Results - Overview
10 | Image Re-Slicing for Parallel Computing| Mark Sedrak
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
50
100
150
200
250
Image Re-slicing routine - 2k data set
M1 - Burnet M2 - Burnet M1 - Bragg M2 - Bragg
Nodes
Tim
e(s)
M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3Bragg Burnet Bragg Burnet Bragg Burnet
1k 2k 4k
0123456789 Re-slicing Routine Optimisation Results
•Shows the effectiveness of outputs for the Different Data Sizes, of M1,2,3 on both Burnet and Bragg
•4k Data Set, 256 GB• 7-10 min
•Shows method 1 compared with M2, on Both Burnt and Bragg
•Issues• Shared users• Resource Limits• Bottlenecks (File System)
• Image Re-Slicing
• Using Parallel Computing to Solve the Data Problem
• File I/O bottleneck, recommend a parallel file system.
Summary
11 | Image Re-Slicing for Parallel Computing| Mark Sedrak
IM&T ASCMark SedrakStudent
e [email protected] www.csiro.au
IM&T ADVANCED SCIENTIFIC COMPUTING
Thank you