``parallel medical and genomics applications on power3 and

SP SciComp 6, Berkeley, August 23, 2002

Parallel Medical and Genomics Applications on Power3 and Power4 Machines

Amit MajumdarSan Diego Supercomputer Center - UCSD

Application I : Brain Deformation Simulation in Image Guided Neurosurgery Simon K. Warfield1, Florin Talos1,2, Alida Tei1,3, Aditya Bharatha1,4, Arya Nabavi1,2, Matthieu Ferrante1,5, Peter McL.

Black2, Ferenc A. Jolesz1, Ron Kikinis1, Corey Kemper1

1Surgical Planing Laboratory and 2Dept. of SurgeryBrigham and Women’s Hospital and Harvard Medical School

3Massashusetts Institute of Technology4University of Toronto Medical School

5Telecom. Lab., Universite’ Catholique de Louvain, Belgium

Application II : Monte Carlo SPECT ImagingYuni Dewaraja1, Kenneth Koral1, Abhijit Bose2, Michael Ljungberg3

1Nuclear Medicine, University of Michigan2Center for Advanced Computing, University of Michigan3Department of Radiation Physics, Univ. of Lund, Sweden

Application III : Parallel Proteomics ApplicationJohn R. Yates1, Daniel J. Carucci2 ,Giri Chukkapalli3, Robert Sinkovits3

1The Scripps Research Institute2Naval Medical Research Center, US NAVY

3SDSC


IBM Parallel Machines : compute nodes

• Power2 and Power3 at CAC University of Michigan : • 176 160Mhz Power2; 1Gbytes and 256 Mbytes memory; three

110Mbytes/sec HP switches• 3 Power3 nodes, 8 cpu/node, 375 Mhz Power3,8 Gbytes/node; 420

Mbytes/sec Colony switch

• Power3 at SDSC : 144 nodes with 8zcpu/node, 375 Mhz Power3; 4 Gbytes/node; 350 Mbytes/sec Colony switch

• Power3 at NAVO MSRC : 334 nodes with 4 cpu/node, 375 Mhs

• Power4 at TACC University of Texas : 4 Regatta-HPC frames; 16 cpu/node; 1.3 Ghz ; 32 Gbytes/node (can be 64 procs 128 Gbytes memory machine by LL); high-speed dual-plane IBM SP Switch2


Application I : Brain Deformation Simulation in Image Guided Neurosurgery


Brain Deformation Simulation in Image Guided Neurosurgery

• Challenge faced by neurosurgeons• Remove as much as possible tumor tissue while minimizing

removal of healthy tissue • Avoid critical anatomical structures

• Real Time Brain Mapping• Enhanced visualization of tumor and critical brain structures• Align preoperatively acquired image data with intraoperative

images of patient’s brain during surgery• Real time constraints

• The code must meet real-time constraints of neurosurgery – provide images within few minutes few times during surgery lasting few hours


Algorithm

• Project preoperative image onto intraoperative images• Allows fusion of images from multiple imaging

modalities and with multiple contrast types• Tracks surfaces of key structures in intraoperatively

acquired images – allows projection of preoperative images into the patient’s brain configuration during surgery

• A volumetric deformation field is inferred from the surface changes

• The field captures nonrigid deformations of the shape of the brain due to brain swelling, cerebrospinal fluid loss, anaesthetic agents and actions of neurosurgeon

• Current model uses linear elastic material model to represent brain


Overall Process• Before Image Guided Neurosurgery :

• During Image Guided Neurosurgery :

Segmentation and Visualization

Preoperative Planning ofSurgical Trajectory

Preoperative

Data Acquisition

Preoperative data

Intraoperative MRISegmentation Registration

Surfacematching

Solve biomechanicalModel for volumetricdeformation

Visualization Surgicalprocess


Volumetric Biomechanical Simulation of Brain Deformation

• During surgery brain shape changes due to surgical intervention

• During surgery surgeon can acquire new volumetric MRI to review current configuration of the entire brain

• Volumetric Biomechanical Simulation of Brain Deformation• Match surface from earlier acquisition to the new acquisition• Infer volumetric deformation based upon the surface

correspondences• Apply forces to the volumetric model that will produce the same

displacement field at the surface as was obtained by the surface matching

• Biomechanical model allows the computation of the deformation throughout the volume


Biomechanical Simulation Equations

ndeformatiobrain

c volumetrifor the solve tominimized isequation Energy

meshes ahedral with tetrdone istion discretizaelement Finite

operatorlinear a is L ; uL

property material ngrepresentimatrix elastic theis D

D),,,,,(

vectorstraintheis;vectorstresstheis

computetofieldvectorntdisplacemethe:)z,y,x(uu

bodyelasticthetoappliedforces :)z,y,x(FF

udTFdT21

E

T

T

xzyzxyzyx


Biomechanical Simulation Equations• Mathematical operations, plugging in of interpolation of

nodes in terms of linear functions etc. etc. finally gives : Ku = -F

• K is the stiffness matrix.• Displacement at the boundary surface nodes are fixed to

match those generated by the active surface model• The force vector F is set equal to the displacement

vector for the boundary nodes : F =

• Now solving matrix system for unknown displacement produces deformation field for the entire mesh that matches prescribed displacements at the boundary

u~


Signa SP (GE Medical Systems)

R. Pergolizzi


Brain shift (1)

F. Talos


Brain shift (2)

F. Talos


Linear System Solver

• The PETSc package is used to solve the linear system • Generalized Minimal Residual (GMRES) solver with block

Jacobi preconditioning

• The rows of matrix are divided equally amongst CPUs

• Global matrix is assembled in parallel• Each CPU assembles the local matrix for each element in its

subdomain• Each CPU has equal # of rows to process• Due to irregular connectivity of the meshes some CPUs may do

more work than others


Performance on Power3 and Power4 Machines

0.10

1.00

10.00

100.00

1 2 4 8 16 32

# of CPU

Tim

e (s

ec)

tot time P4

assem time P4

solve time P4

tot time P3

assem time P3

solve time P3


Timing Table

1 cpu 2 cpu 4 cpu 8 cpu 16 cpu 32 cpu

Tot time p4 30.01 21.11 13.77 9.52 15.28 12.63

Assem time p4 10.62 5.5 2.5 1.33 0.77 0.57

Solve time p4 8.75 5.9 2.28 1.23 2.08 2.32

Tot time p3 52.2 39.45 30.31 25.43 21.58 18.95

Assem time p3 19.89 11.43 6.19 3.71 2.08 1.24

Solve time p3 16.43 11.71 7.48 5.1 2.83 1.05


Observations

• Power4 timings:• Unexpected timings on Power4 16 and 32 processors• Scheduler gives exclusive access to nodes and CPUs• Cache , network ?

• Power3 timing is consistent (with other machines)• Overall scaling is not good beyond few processors

• Serial I/O part contributes to this• Petsc performance : (GMRES with block jacobi precond)

• Linear system solver MFLOPS scale well with # of procs on Power3 (have not checked on Power4 yet)

• Petsc sparse matrix storage allocation is efficient• # of GMRES iterations increase with # of processors – 41 to 135 iterations

on 1 to 16 procs respectively - contributes partially to scaling• Future plan is to investigate scaling further• Investigate viscoelastic modelling ($funding$)• Thanks to Petsc group (Barry Smith) for valuable discussions


Application II : Monte Carlo SPECT Imaging


• Radionuclide therapy

• SPECT imaging in radionuclide therapy

• Monte Carlo simulation of SPECT imaging• SIMIND code• Applications

• Parallel Monte Carlo code and performance


Radionuclide therapy

• Cancer cells are sterilized using internally administered ionizing radiation

• Some therapeutic isotopes, ex. I-131, produce both beta particles and gamma ray photons • Beta particles kill tumor cells. Beta pathlength span several

cells.• Photons used to image radioactivity distribution within patient

• Radionuclide therapy has less toxic effect on normal tissue than chemotherapy.


I-131 Radionuclide therapy

• I-131 Radioimmunotherapy (RIT) : I-131 labeled antibodies selectively target radioactivity to tumor cells while sparing normal tissue.• Shows promise for the treatment of non-Hodgkin’s lymphoma

(NHL). NHL is the fifth leading cause of cancer death. Median survival 6-10 years.

• I-131 MIBG• Shows promise for the treatment of metastatic neuroblastoma

which is a childhood cancer with poor long term survival.


I-131 RIT for NHL at University of Michigan

• Phase II clinical trial: Out of 76 patients with no previous treatment, 48 achieved a complete response and 26 achieved a partial response.

• Patient-specific infusion• Tracer dose: ~ 5 mCi for dosimetry studies to determine

therapeutic dose for each patient• Therapeutic dose: 50 -100 mCi one week later


I-131 imaging

• Single photon emission computed tomography (SPECT) imaging using a rotating gamma camera

• Components of the gamma camera• Lead collimator• Detection medium - scintillation crystal• Electronics

• Tomographic reconstruction of SPECT data produces a 3-D image of the radioactivity distribution within the patient.


Activity quantification

• For accurate quantification, SPECT data has to be compensated for• Patient attenuation• Patient scatter• Camera response

NaI CrystalCollimator

scatterattenuation Patient


What is the role of M.C. simulation in SPECT Imaging?

• Monte Carlo is used primarily to evaluate compensation methods for scatter, attenuation and camera response and to evaluate the overall accuracy of our clinical activity quantification.

• M.C. is ideal for such evaluations because unlike in experiments, the details of photon histories are known.


M.C. simulation of SPECT imaging• SIMIND Monte Carlo code

• Complete photon transport in phantom and SPECT camera

• Complex source distributions: analytical or digital phantoms

• Relatively fast• Code has been verified by experiment

NaI CrystalCollimator

Digital Phantom


SIMIND verification using thorax phantom

• Measurement with experimental phantom

• Simulation with byte-coded digital phantom based on CT images


SIMIND verification: thorax phantom

0

2 5 0

5 0 0

7 5 0

1 0 0 0

Inte

nsit

y

1 5 2 0 2 5 3 0 3 5 4 0 4 5 5 0

Pixel

SIMIND

Measured


M.C. applications: SPECT quantification accuracy using voxel-man phantom

• Clinically realistic case• Voxel-man is based on

patient CT• Realistic activity

distribution in organs and tumor

• Quantification error for large, spherical tumors < 3%

• Simulation time: 220 hours using 16 SP2 processors (60 projections; 1 billion photon per projection)


Work in progress: Monte Carlo generated patient-specific recovery coefficients

Patient CTwith tumor outline

RC=estimated activity

true activity

Activity =estimated activity x 1/ RC

Patient voxel phantom

Monte Carlo data

Projections Reconstructedimage

Apply VOI and quantify

Measured data

Measuredprojections

Reconstructed image

Apply VOI and quantify


A parallel M.C. code for SPECT: motivation• Fast code for accurate I-131 Monte Carlo simulations

• I-131 M.C. simulations are computationally tedious• Physical modeling of collimator

• Variance reduction limited• SPECT require a large number of projections• Realistic simulations using high resolution voxel phantoms

• When all of the above are included in a simulation, CPU time can be several months using the serial SIMIND code


SIMIND parallelization

• In the present application the photon histories are independent of each other - “inherently parallel”

• Critical to have a good parallel RNG (SPRNG)• Each processor performs entire simulation and reports

results to host processor• Code replicated in each of the N processors

• Host sums N partial results and calculates the final result• Standard deviation of the combined result is improved by

1/sqrt(N)• Minimal changes to original SIMIND code


Start

Read data

Reset history counter

Start history

photon transport

More projs?

MPI_RECEIVE

Sum data sets

Store image

Results from N-1 procs. ?

New proj.

More histories?

End

Yes

Yes

Yes

No

No

No

New proj.

Start

Read data

Reset history counter

Start history

photon transport

More projs?

MPI_SEND

More histories?

End

Yes

Yes

No

No

host processor

other processors

SP2 timing results of one SPECT projection of the voxel-man phantom

Small (8.4x107 photons/projection)

Medium (8.4x108 photons/projection)

N Time (sec)

Speedup

Efficiency

Time (sec)

Speedup

Efficiency

1 17131 1.000 1.000 171640

1.000 1.000

2 8581 1.997 0.998 85922 1.998 0.999 4 4298 3.986 0.996 42959 3.995 0.999 8 2162 7.923 0.990 21598 7.947 0.993 16 1085 15.785 0.987 10749 15.968 0.998 32 547 31.304 0.978 5408 31.739 0.992


Power3 timing results of one SPECT projection of the voxel-man phantom

(16.8x107 photons/projection)

N Time (sec) Speedup Efficiency

8 25354 1.00 1.00

64 3167 8.00 1.00

512 402 63.06 0.98


Power4 Timing of a different Monte Carlo photon transport code

May, 2002 on IBM San Mateo center 32 procs Power4 with large page set ( early access to machine may have contributed to results below)

#proc Time P4 (sec)

Speedup P4 (P3)

1 455 1.00 (1.00)2 227 2.00 (1.99)4 114 3.99(3.99)8 57 7.98 (7.98)16 29 15.68

(15.64)29 16 28.4332 19 23.94

(31.54)


More Realistic Simulations

• A 60-projection SPECT simulation of the voxel-man phantom simulation of the voxel-phantom.

• Realistic values were used for the activity concentration ratios in several organs and tumors (based on typical I-131 RIT patient studies at U. Michigan clinic): kidneys, 81; liver, 26; lungs, 26; spleen 53; blood pool, 48; 100 cc spherical tumor, 100; 50 cc spherical tumor, 100; 20 cc spherical tumor, 100; all other structures, 4. The SPECT matrix size was 64x64x64 with pixel size of 0.8 cm x 0.8 cm x 0.4 cm. Up to 3 orders of scatter was modeled. 1 billion photons were simulated for each projection.

• The simulation time on Power3 using 512 processors was 6.5 hours for all 60 projections (time for each projection was 6.5 min).


Effect of Parallel Computing• Monte Carlo has enabled to evaluate and

improve the quantification of I-131 tumor uptake for dosimetry in NHL patients undergoing RIT at U. Michigan clinic.• May lead to statistically significant dose-response

relationships

• Speed-up due to the parallel SIMIND code has enabled us to carry out clinically realistic simulations using voxel-phantoms.

• In the future we will carry out M.C. based dose calculations• More accurate• Tumor and organ dose distributions


Application III : Parallel Proteomics Application


• Sequest is a proteomics application used to analyze mass spectrometer output and match to protein database to identify proteins

• The Naval Medical Research Center (NMRC) is using Sequest in the research to develop a malaria vaccine based on the expression of proteins in the various stages of the malaria parasite.

• Performance of the serial Sequest is currently was a major bottleneck in malaria vaccine project

• SDSC computational scientists developed a parallel version of the Sequest code to reduce simulation time significantly


Chloroquine-resistant

Chloroquine-sensitive

More individuals on the planet with malaria today then ever before in history

300-500 million people become ill with malaria each year

1-3 million children die each year from malaria (200-300 per hour)

Drug resistance is spreading rapidly

There is no licensed vaccine available anywhere in the world

Malaria is a major cause of illness in US troops overseas

Facts about Malaria

An efficient vaccine should be achievable:

Immunity can be acquired naturally

Irradiated sporozoites provide > 95% protection

Vaccines targeting single proteins were disappointing

Current strategy: multistage multicomponent vaccine


A Proteomics View of the Malaria Parasite Life Cycle

OOne Genome: ~6,000 genesDDifferent Proteomes: Distinct Stages

Comprehensively Analyze Protein Complements from 4 P. falciparum Cell Types

Identify Stage-specific Targets for Drug and Vaccine Development


Importance of a malaria vaccine

The battle against malaria hampered by the emergence of drug resistant strains of Plasmodium falciparum, the parasite responsible for majority of malaria infections.

Most cases of malaria are concentrated in the world's poorest countries. Malaria vaccine likely to be affordable alternative to expensive drugs.


SEQUEST®

DTASelect & Contrast

> 1,000 Proteins Identified

Tandem Mass Spectrometer

Peptide MixturePlasmodium falciparum Sporozoites, Trophozoites, Merozoites, Gametocytes

Digestion

Lysis

Proteins

High-Throughput Proteomics: MudPIT

SCXRP

2D Chromatography

48,000 MS/MS SpectraPySpzS5609 #2438 RT: 66.03 AV: 1 NL: 8.37E6T: + c d Full ms2 [email protected] [ 190.00-1470.00]

200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400

m/z

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

85

90

95

100

Re

lativ

e A

bu

nd

an

ce

545.31

658.36

900.36

1031.40

913.421240.53

782.23

896.29

1032.43895.33546.19 771.24

1028.41

721.31

431.15 801.38

1241.39914.34427.27 559.13

1258.56317.17 669.39 1033.60 1312.35651.14408.74 1027.221142.43

915.53432.40 882.07600.24399.24986.50 1123.49217.91 1356.10481.13 869.23 1195.44


Single Processor Optimization

• ( Timings on 375 MHz IBM Power3 procs.TF:ThermoFinnigan;TSRI:The Scripps Research institute)

Case Original –O2

Optimized –O2

Original –O3

Optimized –O3

TF1 86 43 83 45

TF2 140 63 137 63

TF3 209 110 207 111

TF4 388 169 381 173

TSRI 304 145 269 139


Code parallelization•Parallel version of Sequest has been developed using MPI and incorporating all single processor optimizations.

•Parallelization was done so that all the processors work on a different file simultaneously; files are picked up from a list in round robin distribution by the processors

•Tests show that good load balancing is achieved by distributing units of work in a round robin fashion.

•Benchmarks show almost linear scaling on thousands of files

•Database file (~30 Mb) currently read in once per input file (one test case has ~30,000 input file ; ~1000 procs)

•Future plan : once per MPI process (reduces I/O from ~ 1 Tb to 30 Gb); eventually database file will be read once by single node


Power 3 Timing and Speedup

# files # proc Time (sec) Speedup

2227 32 2724 – 3031 1.0 – 1.0

2227 64 5531 – 5744 2.0 – 1.89


Power3 versus Power4 Speedup on 32 procs

# files Machine Time (sec) Speedup

2227 Power3 2724 – 3031 1.0 – 1.0

2227 Power4 1869 – 1939 1.45 – 1.50


Impact on science

Calculations (anlysis of part of a whole cell lysate of a merozoite sample: late-blood stage in the malaria lifecycle) which would have required 30 days on a single processor now require less than an hour on ~1000 processors of IBM SP Power3 NAVO machine

2x speedup due to single processor tuning observedand ~1000x speedup from parallelization observed

Sequest is estimated to be in use at 500 laboratories worldwide – this work impacts entire proteomics community


Final Comments

• Genomics community has already extensively used supercomputing capabilities and will continue to use

• Now the proteomics community will use supercomputing more and more

• Medical community is starting to use parallel computing in clinical and operation room procedures such as imaging, modeling of physical organs and their functions etc.

``parallel medical and genomics applications on power3 and

Health & Medicine

brain deformation simulation

sp scicomp

power3 nodes

volumetric model

critical brain structures

volumetric deformation

patients brain configuration

surgery brain shape