image re-slicing for parallel computing im&t advanced scientific computing mark sedrak |...

Image Re-slicing for Parallel Computing

IM&T ADVANCED SCIENTIFIC COMPUTING

Mark Sedrak | Supervised by Darren Thompson & Sam Moskwa13 February 2013 | Big Day In - Summer Vacation Project

• Aim: To improve the existing cluster re-slicing routine in X-TRACT with Parallel Computing.

• Moore’s Law: Hardware and Data expansion.

• To be covered:– Image Re-Slicing.– Parallel Computing and the use of Super Computers.– My work through-out the project

Image Re-Slicing for Parallel Computing| Mark Sedrak

Project Introduction

2 |

• What is it?– Slices from a CT reconstruction

• Synchrotron• MRI

Image Re-slicing

3 | Image Re-Slicing for Parallel Computing| Mark Sedrak

X-ray imaging tools for HPC clusters and the Cloud | Darren Thompson

Reconstructed Image

4 |

• Uses:– Medical Imaging – Image reconstruction (Materials, Objects, etc)

• XTRACT– Software developed by CSIRO

• Data Sizes

Image Re-slicing

5 |

N / M*

N2 float(projection /

slice)

NM float (sinogram)

N2M float (all

sinograms)

N3 float (all slices)

1k / 720 4 MB 2.8 MB 2.8 GB 4 GB2k / 1,440 16 MB 11¼ MB 22½ GB 32 GB4k / 2,880 64 MB 45 MB 180 GB 256 GB8k / 5,760 256 MB 180 MB 1.4 TB 2 TB

Image Re-Slicing for Parallel Computing| Mark Sedrak

• Serial vs. Parallel Programming– Serial: Instructions are executed one-by-one in sequence.– Parallel: Instructions can be executed simultaneously.

• Splits the work

• Aspects of Parallel Systems• Communication

– Embarrassingly Parallel, Coarse-Grain Parallel, Fine-grain Parallel

• Memory– Shared Memory, Distributed Memory

• Problem Definition– Data Parallel, Task Parallel

Parallel Computing

6 | Image Re-Slicing for Parallel Computing| Mark Sedrak

• Super Computers– Clusters

• TBI “Mini Cluster”, MASSIVE• Bragg: Dual 8-Core CPU’s, 128GB RAM, 40Gb/s InfiniBand• Burnet (Specs): Dual 6-Core CPU’s, 48/96 GB RAM, 40Gb/s InfiniBand

– File Systems• GPFS, HNAS

• Message Passing Interface (MPI)– A Framework for sharing information between distributed memory

processes– Different communication types: 1-1, 1-Many, Many-to-Many– Synchronous vs. Asynchronous Communication

Supercomputers and Message Parsing

7 | Image Re-Slicing for Parallel Computing| Mark Sedrak

• Optimising the re-slicing routine– Generic and portable

• Three main aspects– Communication– Computation/Shuffling– File I/O (Input/Output)

• Developed Three Main Methods – Method 1: (Single Mass communication, High Memory)– Method 2: (Multiple Smaller Communication, High Memory)– Method 3: (Multiple Smaller Communication, Low Memory)

My Project

8 | Image Re-Slicing for Parallel Computing| Mark Sedrak

Results - 2k dataset

9 | Image Re-Slicing for Parallel Computing| Mark Sedrak

0 5 10 15 20 25 30 350

50

100

150

200

250

300

350

Method1 - Bragg0 10 20 30 40 50 60 70

0

50

100

150

200

250

300

350

400

Method 1 - Burnet

0 5 10 15 20 25 300

50

100

150

200

250

300

350

Method 2- Bragg

0 5 10 15 20 25 30 350

50

100

150

200

250

300

350

400

Method 2 - Burnet

Results - Overview

10 | Image Re-Slicing for Parallel Computing| Mark Sedrak

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

50

100

150

200

250

Image Re-slicing routine - 2k data set

M1 - Burnet M2 - Burnet M1 - Bragg M2 - Bragg

Nodes

Tim

e(s)

M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3Bragg Burnet Bragg Burnet Bragg Burnet

1k 2k 4k

0123456789 Re-slicing Routine Optimisation Results

•Shows the effectiveness of outputs for the Different Data Sizes, of M1,2,3 on both Burnet and Bragg

•4k Data Set, 256 GB• 7-10 min

•Shows method 1 compared with M2, on Both Burnt and Bragg

•Issues• Shared users• Resource Limits• Bottlenecks (File System)

• Image Re-Slicing

• Using Parallel Computing to Solve the Data Problem

• File I/O bottleneck, recommend a parallel file system.

Summary

11 | Image Re-Slicing for Parallel Computing| Mark Sedrak

IM&T ASCMark SedrakStudent

e Mark.Sedrak@csiro.auw www.csiro.au

IM&T ADVANCED SCIENTIFIC COMPUTING

Thank you