Distributed MRI reconstruction using gadgetron-based cloud computing

Download Distributed MRI reconstruction using gadgetron-based cloud computing

Post on 11-Mar-2017




1 download

Embed Size (px)


<ul><li><p>FULL PAPER</p><p>Distributed MRI Reconstruction Using Gadgetron-BasedCloud Computing</p><p>Hui Xue,1* Souheil Inati,2 Thomas Sangild Srensen,3 Peter Kellman,4 andMichael S. Hansen1</p><p>Purpose: To expand the open source Gadgetron reconstructionframework to support distributed computing and to demonstratethat a multinode version of the Gadgetron can be used to provide</p><p>nonlinear reconstruction with clinically acceptable latency.Methods: The Gadgetron framework was extended with newsoftware components that enable an arbitrary number of</p><p>Gadgetron instances to collaborate on a reconstruction task.This cloud-enabled version of the Gadgetron was deployed onthree different distributed computing platforms ranging from a</p><p>heterogeneous collection of commodity computers to thecommercial Amazon Elastic Compute Cloud. The Gadgetron</p><p>cloud was used to provide nonlinear, compressed sensingreconstruction on a clinical scanner with low reconstructionlatency (eg, cardiac and neuroimaging applications).</p><p>Results: The proposed setup was able to handle acquisitionand 11-SPIRiT reconstruction of nine high temporal resolution</p><p>real-time, cardiac short axis cine acquisitions, covering theventricles for functional evaluation, in under 1 min. A three-dimensional high-resolution brain acquisition with 1 mm3 iso-</p><p>tropic pixel size was acquired and reconstructed with nonlin-ear reconstruction in less than 5 min.</p><p>Conclusion: A distributed computing enabled Gadgetron pro-vides a scalable way to improve reconstruction performanceusing commodity cluster computing. Nonlinear, compressed</p><p>sensing reconstruction can be deployed clinically with lowimage reconstruction latency. Magn Reson Med 000:000000, 2014. VC 2014 Wiley Periodicals, Inc.</p><p>Key words: Gadgetron; distributed computing; nonlinear MRIreconstruction; open-source software</p><p>INTRODUCTION</p><p>Image reconstruction algorithms are an essential part ofmodern medical imaging devices. The complexity of thereconstruction software is increasing due to the compet-</p><p>ing demands of improved image quality and shortened</p><p>acquisition time. In the field of MRI, in particular, the</p><p>image reconstruction has advanced well beyond simple</p><p>fast Fourier transforms to include parallel imaging (15),</p><p>nonlinear reconstruction (6,7), and real-time reconstruc-</p><p>tion (8,9). The increased interest in three-dimensional</p><p>(3D) acquisitions and non-Cartesian sampling has</p><p>boosted the computational demands further. For appli-</p><p>cations where lengthy acquisition and reconstruction</p><p>time is prohibited by physiological motion, biological</p><p>processes, or limited patient cooperation, the reconstruc-</p><p>tion system is under further pressure to deliver images</p><p>with low latency in order to keep up with the ongoing</p><p>study.While the need for fast image reconstruction is grow-</p><p>ing, most published reconstruction algorithms, especiallythose relying on iterative solverssuch as image domaincompressive sensing (6,1012) and k-space SPIRiT andits regularized versions (7,13,14)do not come with theefficient reference implementations that would enableclinical use. In many cases, the algorithms are not imple-mented for online use on clinical scanners. Even if thedevelopers would like to integrate their reconstructionalgorithms for online use, the vendor-provided hardwareand software platform may have inadequate specifica-tions for a demanding reconstruction, or the availableprogramming window may be unsuited for integration ofnew reconstruction schemes. Consequently, there is agap between the number of new algorithms being devel-oped and published and the clinical testing and valida-tion of these algorithms. Undoubtedly, this is having animpact on the clinical adoption of novel nonlinear recon-struction approaches (eg, compressed sensing).</p><p>We have previously introduced an open-source plat-form for medical imaging reconstruction algorithmscalled the Gadgetron (15), which aims to partiallyaddress the aforementioned concerns. This platform isfreely available to the research community and industrypartners. It is platform independent and flexible for bothprototyping and commercial development. Moreover,interfaces to several commercial MR scanners have beendeveloped and are being shared in the research commu-nity. This simplifies the online integration of new recon-struction algorithms significantly, and the newalgorithms in research papers can be tested in clinicalsettings with less implementation effort. As a result,some groups have used Gadgetron for online implemen-tation and evaluation of their reconstruction methods(1618). Since the publication of the first version of theGadgetron, the framework has adopted a vendor-independent raw data format, the ISMRM Raw Data</p><p>1Magnetic Resonance Technology Program, National Heart, Lung andBlood Institute, National Institutes of Health, Bethesda, Maryland, USA.2National Institute of Mental Health, National Institutes of Health, Bethesda,Maryland, USA.3Department of Computer Science and Department of Clinical Medicine,Aarhus University, Aarhus, Denmark.4Medical Signal and Image Processing Program, National Heart, Lung andBlood Institute, National Institutes of Health, Bethesda, Maryland, USA.</p><p>Grant sponsor: National Heart, Lung and Blood Institute IntramuralResearch Program.</p><p>*Correspondence to: Hui Xue, Magnetic Resonance Technology Program,National Heart, Lung and Blood Institute, National Institutes of Health, 10Center Drive, Bethesda, MD 20814. E-mail: hui.xue@nih.gov</p><p>Additional Supporting Information may be found in the online version ofthis article.</p><p>Received 11 December 2013; revised 28 January 2014; accepted 18February 2014</p><p>DOI 10.1002/mrm.25213Published online 00 Month 2014 in Wiley Online Library (wileyonlinelibrary.com).</p><p>Magnetic Resonance in Medicine 00:0000 (2014)</p><p>VC 2014 Wiley Periodicals, Inc. 1</p></li><li><p>(ISMRMRD) format (19). This further enables sharing ofreconstruction algorithms.</p><p>Although this concept of an open-source platform anda unified ISMRMRD format shows great potential, theoriginal Gadgetron framework did not support distrib-uted computing across multiple computational nodes.Although the Gadgetron was designed for high perform-ance (using multiple CPU cores or graphics processingunits [GPUs]), it was originally implemented to operatewithin a single node or process. Distributed computingwas not integral to the design. Because reconstructionalgorithms increase in complexity, they may need com-putational power that would not be economical toassemble in a single node. The same considerations haveled to the development of commodity computing clusterswherein a group of relatively modest computers areassembled to form a powerful computing cluster. Anexample of such a cluster system is the National Insti-tutes of Health (NIH) Biowulf Cluster (http://biowulf.nih.gov). Recently, commercial cloud-based services havealso offered the ability to configure such commoditycomputing clusters on demand and rent them by thehour. The Amazon Elastic Compute Cloud (EC2) is anexample of such a service (http://aws.amazon.com/ec2).</p><p>In this paper, we propose to extend the Gadgetronframework to enable cloud computing on multiplenodes. With this extension, named Gadgetron Plus (GT-Plus), any number of Gadgetron processes can be startedat multiple computers (referred to as nodes), and adedicated interprocess controlling scheme has beenimplemented to coordinate the Gadgetron processes torun on multiple nodes. A large MRI reconstruction taskcan be split and run in parallel across these nodes. Thisextension to distributed computing maintains the origi-nal advantages of the Gadgetron framework. It is freelyavailable and remains platform-independent. As demon-strated in this paper, the nodes can even run differentoperating systems (eg, Windows or different distributionsof Linux) and have different hardware configurations.</p><p>The implemented architecture allows the user to setup a GT-Plus cloud in several different ways. Specifi-cally, it does not require a dedicated professional cloudcomputing platform. The GT-Plus cloud can be deployedon setups ranging from an arbitrary collection of net-worked computers in a laboratory (we refer to this as acasual cloud) to high-end commercial cloud systems,as demonstrated herein. In this paper, we demonstratethe GT-Plus cloud setup in three different scenarios anddemonstrate its flexibility to cope with variable computa-tional environments. The first cloud setup is a casualcloud consisting of seven personal computers situated inour laboratory at the National Institutes of Health. Thesecond setup uses NIHs Biowulf Cluster (20). The lastconfiguration is deployed on the commercial AmazonElastic Compute Cloud (Amazon EC2) (21).</p><p>To demonstrate the benefits of this extension, we usedthe cloud-enabled version of the Gadgetron to implementnonlinear 11-SPIRiT (7) for two-dimensional (2D) time-resolved (2Dt) and 3D imaging applications. We dem-onstrate that cardiac cine imaging with nine slices cover-ing the entire left ventricle (32 channels; acquiredtemporal resolution50 ms; matrix size 192 100;</p><p>acceleration factor5, 1.5 s per slice) can be recon-structed with 11-SPIRiT with a reconstruction time(latency </p></li><li><p>6. The client sends every piece of readout data to theGadgetron in the ISMRMRD format. These data aredeserialized by the Reader and passed through theGadget chain.</p><p>7. The reconstruction results are serialized by theWriter and sent back to the client via the socketconnection.</p><p>8. When the end of acquisition is reached, the Gadget-ron closes down the Gadget chain and closes theconnection when the last reconstruction result hasbeen passed back to the client.</p><p>GT-Plus: Distributed Computing Extension of Gadgetron</p><p>A schematic outline of GT-Plus extension is shown inFig. 2. A distributed Gadgetron process has at least oneGadgetron running on a specific port for each node (mul-tiple Gadgetron processes can run within the same node</p><p>at different ports). A software module, GadgetCloudCon-troller, manages the communication between nodes. Typ-ically, the gateway node receives the readout data fromthe client and deserializes them using a Reader. Depend-ing on the reconstruction workflow, the gateway nodemay buffer the incoming readouts and perform someprocessing before sending reconstruction jobs to the con-nected nodes, or data can be forwarded directly to theclient nodes. GadgetCloudController maintains multipleTCP/IP socket connections with every connected nodevia a set of GadgetronCloudConnector objects (one foreach connected node). Each GadgetronCloudConnectorhas a Reader thread (CloudReaderTask) and a Writerthread (CloudWriterTask) that are responsible for receiv-ing and sending data (to the node), respectively. There isa Gadgetron Gadget chain running on every connectednode. The gateway node will send the XML configura-tion to the connected node to assemble the chain. For</p><p>FIG. 2. Example presentation of GT-Plus distributed computing architecture for Gadgetron. In this example, at least one Gadgetron pro-cess is running on a node (Multiple Gadgetron processes can run in one node on different ports). The gateway node communicates</p><p>with the client application (eg, MRI scanner). It manages the connections to each of the computing nodes via the software moduleGadgetCloudController and GadgetronCloudConnector. Whenever sufficient data is buffered on the gateway node, the package can be</p><p>sent to the computing node for processing. Different computing nodes can run a completely different reconstruction chain. [Color figurecan be viewed in the online issue, which is available at wileyonlinelibrary.com.]</p><p>FIG. 1. Schematic presentation of Gadgetron architecture. The client application communicates with the Gadgetron process via theTCP/IP connection. The Reader deserializes the incoming data and passes them through the Gadget chain. The results are serialized by</p><p>the Writer and sent back to the client application. The Gadgetron process resides on one computer, the Gadgetron server. The clientcan be an MRI scanner or other customer processes. Note that the algorithm modules implemented in the Gadgetron toolboxes areindependent of the streaming framework; therefore, these functionalities can be called by stand-alone applications that are not using</p><p>the streaming framework. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]</p><p>Gadgetron C-Bud Computing 3</p></li><li><p>different cloud nodes, different Gadget chains can beassembled. In fact, the connected nodes can also be gate-way nodes, thus creating a multi-tiered cloud. The typi-cal protocol of GT-Plus distributed computing is asfollows:</p><p>1. The client issues the connection request to the GT-Plus gateway at a specific network port. Once theconnection is established, the client will send theXML based configuration and parameter files toestablish and initialize the gadget chain at the gate-way node.</p><p>2. The client starts sending readout data to the gate-way node.</p><p>3. The Gateway node establishes a connection to cloudnodes and supplies them with XML configurationand parameter files. As mentioned, different con-nected nodes can be configured with differentchains if needed.</p><p>4. Job packages are sent to connected cloud nodes forprocessing via the corresponding CloudWriterTask.</p><p>5. When all jobs are sent to nodes (ie, when the acqui-sition is done), the gateway Gadgetron either waitsfor reconstruction results or conducts extra compu-tation. The ReaderTask objects, at the same time,listen for reconstruction results. Whenever thereconstruction results are sent back from a cloudnode, the gateway Gadgetron is notified by theReaderTask object and will take user-definedactions (eg, passing the results downstream or wait-ing for the completion of all jobs). Finally, the gate-way node will proceed to finish other processingsteps (if any) and send the images down the remain-ing part of the gateway Gadget chain and eventuallyback to the client.</p><p>6. If one or more connected nodes fail to complete ajob successfully, the gateway node will be notifiedby either receiving an invalid result package ordetecting a shutdown message on the socket. TheGadgetCloudController on the gateway node willkeep a record of uncompleted jobs and processthem locally. In this way, the GT-Plus gains robust-ness against network instability.</p><p>The software modules mentioned above are imple-mented as C template classes and are capable of han-dling any type of custom job packages (ie, the user isable to configure the data types that are sent andreceived from the cloud nodes). The only requirement isto implement appropriate functions for serialization anddeserialization of work packages and results. This designis a straightforward extension of the Readers and Writersin the original Gadgetron.</p><p>In the current implementation, every computing nodecan be given an index indicating its computational power.This index is used to allow the gateway node to apply ascheduling algorithm where the workload is distributedto the cloud nodes in proportion to their...</p></li></ul>