Aggregated Motion Estimation for Real-Time MRIReconstruction

Housen Li,1,2* Markus Haltmeier,3 Shuo Zhang,4 Jens Frahm,4 Axel Munk1,2

Purpose: In real-time MRI serial images are generally recon-

structed from highly undersampled datasets as the iterativesolutions of an inverse problem. While practical realizations

based on regularized nonlinear inversion (NLINV) have hithertobeen surprisingly successful, strong assumptions about thecontinuity of image features may affect the temporal fidelity of

the estimated reconstructions.Theory and Methods: The proposed method for real-timeimage reconstruction integrates the deformations between

nearby frames into the data consistency term of the inverseproblem. The aggregated motion estimation (AME) is not

required to be affine or rigid and does not need additionalmeasurements. Moreover, it handles multi-channel MRI databy simultaneously determining the image and its coil sensitivity

profiles in a nonlinear formulation which also adapts to non-Cartesian (e.g., radial) sampling schemes. The new method

was evaluated for real-time MRI studies using highly under-sampled radial gradient-echo sequences.Results: AME reconstructions for a motion phantom with con-

trolled speed as well as for measurements of human heart andtongue movements demonstrate improved temporal fidelity

and reduced residual undersampling artifacts when comparedwith NLINV reconstructions without motion estimation.Conclusion: Nonlinear inverse reconstructions with aggre-

gated motion estimation offer improved image quality andtemporal acuity for visualizing rapid dynamic processes by

real-time MRI. Magn Reson Med 72:1039–1048, 2014.VC 2013 Wiley Periodicals, Inc.

Key words: inverse problems; motion estimation; aggregatedimaging; nonlinear inversion; real-time MRI; parallel imaging

INTRODUCTION

Speed is crucial in real-time MRI studies of physiologic proc-esses, ranging from monitoring of interventional proceduresto cardiovascular imaging. Because the physical acceleration

of the data acquisition process is limited by regulations to

prevent peripheral nerve stimulation, most strategies achieve

sufficiently short measuring times by acquiring less data. As

a consequence, the task to be accomplished is to preserve the

quality of the reconstructed images.A basic development along this line is the parallel imaging

concept which takes advantage of multiple receiver coils to

acquire data simultaneously. Such techniques benefit from

spatially complementary coil sensitivities, which are gener-

ally unknown and also depend on the actual experimental

condition. Therefore, coil sensitivity profiles are either explic-

itly precalibrated in image space, like SENSE (1), or implicitly

in k-space, like SMASH (2) and GRAPPA (3). Unfortunately,

however, such precalibration techniques make only subopti-

mal use of the available data from multiple receiver channels,

so that errors in the estimated coil profiles may lead to arti-

facts in the iteratively optimized image—already for moderate

acceleration factors of approximately 2 to 3.An improved strategy is to compute spin density maps

and coil profiles at the same time by means of a nonlin-

ear formulation of the inverse reconstruction problem

(4,5). In this case, the use of highly undersampled data

renders the reconstruction problem ill-posed. To stabi-

lize the problem and obtain plausible solutions, it is nec-

essary to incorporate a priori information about the

unknown object (described by its spin density) and the

coil sensitivities into the reconstruction process. For

example, previous implementations of the NLINV

method (6–8) applied a temporal L2-regularization on the

object, which is usually too weak to completely avoid

residual artifacts for high degrees of undersampling. In

particular, temporally flickering artifacts are observed for

radial sampling schemes which use complementary sets

of spatial encodings in consecutively acquired datasets.

In practice, a temporal median filter may effectively

diminish such streaking artifacts though at the expense

of degrading the true temporal resolution. Alternatively,

the total variation or total generalized variation may be

used for regularization (9). These approaches also reduce

streaking artifacts, but fail to recover small-scale details

of the object and, therefore, sacrifice spatial resolution.

In general, regularization methods alone seem to be

unable to provide artifact-free images with both high spa-

tial and temporal resolution in real-time MRI scenarios

with pronounced undersampling. It has also been pro-

posed to use a Kalman filter to unaliase undersampled

reconstructions (10,11). Although the method is causal

and very fast, it so far has not found wider applications.An alternative idea for improved image quality is to

integrate information about the features of a movingobject into the reconstruction. This may be accomplishedby exploiting multiple measurements at different (neigh-boring) time points. One of the most effective means for

1Institute for Mathematical Stochastics, University of G€ottingen, G€ottingen,Germany.2Max Planck Institute for Biophysical Chemistry, G€ottingen, Germany.3Department of Mathematics, University of Innsbruck, Innsbruck, Austria.4Biomedizinische NMR Forschungs GmbH am Max-Planck-Institut f€ur bio-physikalische Chemie, G€ottingen, Germany.

*Correspondence to: Housen Li, M.S., Max Planck Institute for BiophysicalChemistry, Am Fassberg 11, 37077 G€ottingen, Germany. E-mail:housen.li@mpibpc.mpg.de

Additional Supporting Information may be found in the online version ofthis article.

Received 18 July 2013; revised 24 September 2013; accepted 8 October2013

DOI 10.1002/mrm.25020Published online 18 November 2013 in Wiley Online Library(wileyonlinelibrary.com).

Magnetic Resonance in Medicine 72:1039–1048 (2014)

VC 2013 Wiley Periodicals, Inc. 1039

motion compensation in MRI is the “navigator echo”technique and its variants (12–19). In these methods, anavigator signal is repetitively acquired to extract spe-cific motion information. The need to insert multiplenavigator modules into the MRI sequence may beavoided by “self-navigating” techniques which determinemotions from the actual data. Early applications (20,21)used partial or full k-space data in a block-based or para-metric manner, but failed to detect complex motionssuch as elastic deformations. Lately, more flexiblemotion-detection techniques were developed for free-breathing cine MRI studies (22–26), but most of themcan only compensate for slow (e.g., respiratory) move-ments that affect a faster (e.g., cardiac) motion of interest.Motion compensation was also combined with conven-tional parallel MRI reconstructions (27–31).

To overcome some of the aforementioned limitations,this work presents a novel reconstruction method forreal-time MRI which incorporates the idea of a “self-navigating” motion into a nonlinear formulation of theinverse problem jointly estimating spin density and coilsensitivity profiles. The method is based on a nonpara-metric aggregated motion estimation (AME) and gener-ates images with high temporal fidelity and low residualartifacts. The experimental validation used real-time MRIstudies of a motion phantom with controlled speed aswell as of human heart motions and rapid tongue move-ments during sound production. See Supplementary Vid-eos 1–3, which are available online.

THEORY

The proposed AME approach is based on the NLINVmethod for reconstructions from highly undersampledradial MRI acquisitions with multiple receiver coils(5,6). It generalizes the respective data consistency termto incorporate an aggregated reconstruction from multi-ple frames with nonparametric motion correction. Asoutlined in Figure 1, the scheme first involves conven-tional NLINV reconstructions of serial datasets withinterleaved sets of spokes in five successive acquisitions.Respective image series are then used to estimate rele-vant motion features for a central frame from five consec-utive NLINV reconstructions. Final AME reconstructionsadd the motion information into the data consistencyterm of the original problem and compute a new nonlin-ear inverse solution. Throughout this study, we use nota-tions given in Table 1.

Real-Time MRI

The real-time MRI data acquisition of the t-th (t 2 N)frame from multiple receiver coils is given by

yt;l ¼ StFðrt � ct;lÞ þ et;l; l 2 L: [1]

The goal is to obtain a serial stream of images ðrtÞt2Nwith high spatial and temporal resolution from the meas-ured data ðyt;lÞt2N;l2L. The imaging time per frame isdetermined by the physical repetition time TR neededfor radiofrequency excitation and spatial encoding timesthe number of k-space samples M , which, therefore, iskept as small as possible. On the other hand, for pro-

nounced undersampling conditions, Eq. [1] becomesincreasingly ill-conditioned. As a consequence, the inver-sion of the system leads to an amplification of noisewhich in turn results in low-resolution images. Thus, a

FIG. 1. AME reconstruction method. The example refers to thet-th frame using G ¼ f�2;�1;0; 1; 2g.

Table 1Glossary of Notations

rt Spin density of frame t

ct;l Sensitivity profile of coil l of frame t

yt;l Measurement by coil l of frame t

et;l Noise of the measurement by coil l of frame t

Fð�Þ Fourier transform FðfÞ ¼R

fðxÞe�2pihx;kidx

Stð�Þ Sampling operator StðfÞ ¼ ðfðkt; jÞÞMj¼1 at ðkt; jÞMj¼1

in k-space

M Number of samples in k-space per frame

L Indices of coils L ¼ f1; . . . ;NgN Number of coils

G Indices of used frames, e.g. G ¼ f�2; �1;0;1; 2gft;sð�Þ Motion (or deformation) from rt to rtþs,

i.e. ft;sðrtÞ � rtþs

Ftþsð�Þ FtþsðxÞ ¼ ðStþsFðft;sðrtÞ � ctþs;lÞÞl2L

for x ¼ ðrt; ðctþr;lÞr2G;l2LÞT

jj � jj L2 norm

jj � jj1 L1 norm

jj � jjm Sobolev norm jjf jjm :¼ jjað1þ bjj � jj2Þm=2ðF fÞð�Þjjfor a;b;m 2 Rþ

ð�ÞH Conjugate transpose

ð�ÞT Transpose

ð�Þ’ Derivative

rð�Þ Gradient

1040 Li et al.

proper choice of M should be a sensible trade-off betweentemporal and spatial resolution.

Reconstructions Using Aggregated Motion Estimations

In this section, we first assume that the motion (or defor-mation) ft;s from rt to rtþs, i.e. ft;sðrtÞ � rtþs, is knownfor every s 2 G, with some G � Z. For example, G can bef�2; �1;0;1;2g, which corresponds to five successiveacquisitions. By variable substitution, the spin density ofthe t-th frame satisfies

StþsFðft;sðrtÞ � ctþs;lÞ ¼ ytþs;l; l 2 L; s 2 G: [2]

Thus, if successive frames rely on complementary datasamples in k-space, the reconstruction takes advantage ofjGj �M samples for recovering rt. The approach may,therefore, obtain images with higher spatial resolutionfrom Eq. [2], while the temporal resolution remainsunchanged, i.e. M � TR. For sake of clarity, we rewriteEq. [2] by an abstract nonlinear operator equation

FtþsðxÞ ¼ ytþs; for s 2 G;

where x ¼ ðrt; ðctþr;lÞr2G;l2LÞT ; ytþs ¼ ðytþs;lÞl2L and Ftþs :

x 7!ðStþsFðft;sðrtÞ � ctþs;lÞÞl2Lfor every s 2 G:These equations are solved for the unknown x by a

Newton-type method. Its key idea consists in repeatedlylinearizing the operator equation FtþsðxÞ ¼ ytþs, s 2 G,around some approximate solution xn, and solving thelinearized problems

F 0tþsðxnÞðx � xnÞ ¼ ytþs � FtþsðxnÞ; s 2 G: [3]

As the real-time MRI problem is highly ill-posed, the

inverse of�

F 0tþsðxnÞ�

s2Gis unbounded and hence any

discretization of�

F 0tþsðxnÞ�

s2Gis seriously ill-

conditioned. The standard Newton method is, therefore,not applicable and may not even be well defined fornoise-free data, because the residual ytþs � FtþsðxnÞ is notguaranteed to lie in the range of F 0tþsðxnÞ. As a conse-

quence, some regularization has to be used for solvingthe linearized Eq. [3].

Because only the product of the spin density and coilprofiles is determined, the real-time MRI problem isunderdetermined even in the fully sampled case. Whilethe image usually contains fine structures, the coil pro-files are generally rather smooth. As in Uecker et al (5),this can be ensured by introducing a term promotingsmoothness which may be given by a Sobolev normjjf jjm :¼ jjað1þ bjj � jj2Þm=2ðF f Þð�Þjj, for some a; b;m 2 Rþ.It penalizes high spatial frequencies by a polynomial ofdegree m as a function of the distance to the center of k-space. Moreover, the object (i.e., its spin density) usuallydeforms continuously and smoothly from frame to frame,so that an efficient regularization penalizes the differen-ces between neighboring frames to ensure temporal con-tinuity. By combining the standard Newton method andthe aforementioned regularization, we obtain the well-known iteratively regularized Gauss-Newton method(IRGNM) (32,33) for solving Eq. [2]

hn ¼ argminh

Xs2G

jjF 0tþsðxnÞh� ðytþs � FtþsðxnÞÞjj2

þ anjjrðnÞt þ hrt� r0

t jj2 þ an

�X

s2G;l2L

jjcðnÞtþs;l þ hctþs;l� c0

tþs;ljj2m;

xnþ1 ¼ xn þ hn

[4]

where xn ¼ ðrðnÞt ; ðcðnÞtþs;lÞs2G;l2LÞT , h ¼ ðhrt

; ðhctþs;lÞs2G;l2LÞ

T ;

hn ¼ ðhðnÞrt; ðhðnÞctþs;l

Þs2G;l2LÞT and the initial guess

x0 ¼ ðr0t ; ðc0

tþs;lÞs2G;l2LÞT . If the initialization is close to the

true solution, the choice an :¼ a0qn, with 0 < q < 1, isusually sufficient for convergence (34). Because theIRGNM method reduces to a Gauss-Newton method foran ¼ 0, it uses the more robust descent direction at thebeginning of the iterative process (far from the solution)and the faster convergent algorithm at the end (near thesolution). The choice of parameter a in the Sobolev normserves as a balance between penalization of the spin den-sity and coil profiles. The quadratic optimization [4] canbe numerically solved by inverting its normal equationwith proper preconditioning and discretization. Thetechnical details are provided in the Appendix. Togetherwith the motion estimation described in the next section,this strategy represents the proposed AME-based nonlin-ear reconstruction method for high temporal and spatialresolution (see Fig. 1). In this manner, multiple acquisi-tion frames are exploited for reconstruction with propermotion correction, implicitly increasing the samplingrate while preserving temporal sharpness.

Aggregated Motion Estimation

The positional displacements between nearby frames aregenerally not known. Therefore, each frame is initiallyprecomputed by the NLINV method, which correspondsto G ¼ f0g in the reconstruction method defined in theprevious subsection. Subsequently, these images serve toestimate the motion using the TV-L1 optical flow model(35,36). In detail, the motion ft;s from rt to rtþs is esti-mated by fs;tðrtÞðxÞ ¼ rtðx þ uðxÞÞ, with u given by thesolution to

minu;vjjrt þrrt � u� rtþs þ vjj1 þ ljjrujj1 þ mjjrvjj1:

Because ft;sðrtÞðxÞ :¼ rtðx þ uðxÞÞ � rt þrrt � u, theauxiliary variable v models the varying reconstruction(i.e., undersampling) artifacts in different frames. Forradial MRI acquisitions, the residual streaking artifactshave a relatively low total variation in comparison tothe object which contains all local structures. There-fore, it is expected that u can only capture the truemotion of the object instead of the moving artifacts inthe precomputed NLINV images. This will empiricallybe verified in the Results section (cf. Fig. 2). To avoidthe impact of outliers on the motion estimation, weonly used the L1 norm. This nonsmooth minimizationcan efficiently be solved by the first-order primal-dualalgorithm (36).

Aggregated Motion Estimation for Image Reconstruction 1041

METHODS

Data Acquisition

All studies were conducted on a 3 Tesla (T) MRI system(Magnetom TIM Trio, Siemens, Erlangen, Germany). Forhuman studies, healthy subjects with no known illnesswere recruited among the students of the local Univer-sity. Written informed consent was obtained in all casesbefore each examination.

The proposed AME reconstruction was evaluated forreal-time MRI of a motion phantom with objects (i.e.,water-filled tubes) moving at predefined velocities (37)as well as for movements of the human heart and tongue.Continuous data acquisition for real-time MRI was

achieved with use of a highly undersampled radialFLASH sequence which used an interleaved arrangementof spokes for five successive frames (7,38). The spokeswere equally distributed over a full 360� circle to homo-geneously sample k-space. To prevent aliasing effectsfrom object structures outside the selected FOV, spokeswere acquired with two-fold oversampling. T1-weightedimages were generated by a short repetition time TR anda low radiofrequency (RF) flip angle as detailed inTable 2.

The motion phantom consisted of a polyacetal disc(100 mm diameter) rotating with respect to its geometriccenter. It was studied using the 32-channel head coil(Siemens Healthcare, Erlangen, Germany). Three water-

FIG. 2. Principle demonstration

of the motion estimation. Twoframes of a serial NLINV recon-struction of the human heart

(short-axis view, 30 ms resolu-tion) at end diastole (NLINV1)

and end systole (NLINV2) wereused to estimate motion fieldsthat were applied to NLINV1 to

yield DEFORM. DIFF representsthe difference between NLINV2and DEFORM (intensities scaled

by a factor of 6).

Table 2

Acquisition parameters for Real-Time MRI

Parameter Motion phantom Tongue movements Cardiac movements

Repetition time/msec 2.28 2.22 2.00Echo time/msec 1.48 1.40 1.29

Flip angle/� 8 5 8Field-of-view/mm2 256 � 256 192 � 192 256 � 256

Base resolution 128 � 128 128 � 128 160 � 160Voxel size/mm3 2.0 � 2.0 � 5.0 1.5 � 1.5 � 10.0 1.6 � 1.6 � 8.0Spokes per frame 9 15 15

Resolution/msec 20.5 33.3 30.0

1042 Li et al.

filled tubes with 10 mm diameter were fixed on the discwith distances to the center of 25 mm, 37.75 mm, and 50mm, respectively. Real-time MRI at 20 ms temporal reso-lution was performed for three different rotation frequen-cies (0.5 Hz, 1.0 Hz, and 1.5 Hz) corresponding toangular velocities ranging from 8 cm s�1 (innermost tubefor 0.5 Hz) to 47 cm s�1 (outermost tube for 1.5 Hz).

Real-time MRI of the human heart and tongue wereperformed in a supine position. Sound generationreferred to playing a plastic mouthpiece of a brass instru-ment, while subjects were asked to perform rapid tonguemovements (staccato) at a rate of approximately 5 Hz. Amid-sagittal image was chosen to cover the oropharyngo-laryngeal area at 33.3 ms temporal resolution by combin-ing a four-channel small flexible receiver coil (SiemensHealthcare, Erlangen, Germany) and a bilateral 2 � 4array coil (NORAS MRI products, Hoechberg, Germany).The study used the same setup as recently reported forreal-time MRI of speech generation (39). Cardiac MRIwas performed during free breathing and without syn-chronization to the electrocardiogram (7,8) using a 32-channel body coil consisting of an anterior and posteriorarray with 16 elements each. Online image control usedconventional NLINV reconstructions at a rate of at least17 frames per second (40,41) with a postprocessing tem-poral median filter (NLINV-MED).

Image Reconstruction

All reconstructions shown here were done offline usingan in-house software package written in Matlab (R2012a,The MathWorks, Natick, MA). Respective parameters aresummarized in Table 3. In a first step, data from up to32 receiver channels were combined into a set of 10 vir-tual channels based on a principal component analysisas described (8,42,43). The interpolation in k-space fromradial spokes to Cartesian grids involved a Kaiser-Besselfunction with L ¼ 6, b ¼ 13:8551 and 1.5-fold oversam-pling (44). To speed up computations, the interpolationcoefficients were precalculated and stored in a look-uptable. In the next step the interpolated data were normal-ized such that the L2 norm equaled 100. This allows forchoosing the reconstruction parameters independentfrom the data acquisition parameters, which minimizesthe operator interference and also maintains the qualityof the results.

For AME reconstructions, G ¼ f�2;�1;0;1;2g wasempirically found to take full advantage of the comple-mentary information from five successive acquisitionswith interleaved radial encodings. Numerically, G ¼f�3;�2;�1;0;1;2;3g gives similar results for both simu-lated and real data (not shown), but increases the com-

putational complexity. For motion estimation, the spatialdeformation of the object was implemented with a bicu-bic interpolation, and parameters of the model were setto l ¼ 0:02 and m ¼ 1:0. For comparison, the same datawere also reconstructed by standard NLINV (7,8) whichwas implemented in the same software environment. Inaddition, a temporal median filter covering five neigh-boring images was applied as a postprocessing step toreduce residual streaking artifacts (NLINV-MED).

RESULTS

The principle of the motion estimation is demonstratedin Figure 2 using data of the human heart at 30 ms tem-poral resolution. Two frames of a serial NLINV recon-struction at end diastole (NLINV1) and end systole(NLINV2) were selected to obtain large differences dueto pronounced contraction and thickening of the myocar-dial wall during systole (arrow). DEFORM refers to thedeformation of NLINV1 and has been calculated by usingthe estimated motion. It clearly identifies the contractionof the myocardium, whereas the streaking artifacts at thetop-left corner of the image remain similar as in NLINV1.The example demonstrates that the information of themoving object has correctly been captured by the motionestimation, while the image artifacts, which may alsochange with time, are largely excluded. This can also bevisualized in the difference image DIFF betweenDEFORM and NLINV2, where the artifacts are dominantand the structure of the heart is less visible.

Figure 3 compares selected frames from serial NLINV,NLINV-MED, and AME reconstructions at 20 ms resolu-tion (50 frames per second) for three water-filled tubesmoving at different velocities. For the lowest velocitiesof 8 to 16 cm s�1 (top row), all three methods produceacceptable results, although the latter two surpassNLINV in reducing streaking artifacts (arrow). Moderatevelocities (middle row) refer to velocities from 16 to 31cm s�1. Here, conventional NLINV leads to geometricdistortions for the fastest motion of the outermost tube(i.e., worm-shaped extensions along the motion direc-tion), while NLINV-MED even destroys the true struc-tural information (arrow). The resulting shape of thecircular tube is a typical effect from the temporal medianfilter. In contrast, the AME reconstruction offers properimage estimations with almost no motion or streakingartifacts. Finally, for the highest velocites of up to 47 cms�1 (lower row), both NLINV and NLINV-MED result inseverely deformed shapes for almost all tubes as well aspronounced streaking artifacts. Again, the AME methodshows the best performance with only very mild distor-tions and barely visible artifacts. Furthermore, because

Table 3Reconstruction Parameters for Real-Time MRI

Parameter Newton Steps

Reg. Coeff.an ¼ a0qn Sobolev Norm

Used Frames Ga0 q a b m

AME 5 0.8 0.5 0.8 440 32 {-2,-1,0,1,2}

NLINV 6 1.0 0.5 1.0 440 32 {0}

Aggregated Motion Estimation for Image Reconstruction 1043

of the absence of distributed object representations, thesignal-to-noise ratio of the AME reconstructions is gener-ally higher compared with results obtained by the othertwo methods.

Figure 4 compares NLINV, NLINV-MED, and AMEreconstructions for rapid tongue movements at 33.3 msresolution (30 frames per second). To blow staccatophrases through the mouthpiece of a brass instrument,the tongue tip of the subject had to rapidly move forwardand backward touching the upper teeth ridge. The tem-poral evolution of this motion is best demonstrated byspatiotemporal intensity profiles taken along a referenceline (top left) placed at the tongue tip. For NLINV recon-structions (top right), the flickering of residual streakingartifacts in the upper portion of the image/profiles isclearly visible. NLINV-MED effectively removes theseartifacts at the expense of blurring the tongue movementsby the temporal median filter (middle right). On the con-trary, the proposed AME method (bottom right) not onlyminimizes residual streakings, but more importantly

enhances the temporal acuity by preserving the sharpintensity changes that are associated with the rapid pro-trusion of the tongue. In fact, while the original NLINVreconstructions cover the movement within approxi-mately four to five frames (t1 � 130–170 ms), the tempo-ral profiles of the AME reconstructions exhibit only 3 to4 frames (t2 � 100–130 ms).

Similar comparisons of NLINV, NLINV-MED, andAME reconstructions for real-time cardiac MRI at 30 msresolution (33 frames per second) are shown in Figure 5.Most likely due to high-intensity fat signals, NLINVreconstructions contain temporally flickering streakingartifacts, which in the images (top left) are mainly seenoutside the body. Nevertheless, they also affect varioustissues as shown in the temporal intensity profiles (topright) taken along the marked vertical line. The situationmay be alleviated by adding a temporal median filter,but NLINV-MED reconstructions sacrifice temporal reso-lution as demonstrated for local structures in both theimage (middle left, arrow) and temporal intensity profile

FIG. 3. (Left) NLINV, (middle) NLINV-MED, and (right) AME reconstructions of a phantom (20 ms resolution) comprising three tubes ofwater moving with different angular velocities: Slow¼8, 12, and 16 cm s�1, medium¼16, 24, and 31 cm s�1, and fast¼24, 36, and 47

cm s�1 (inner to outer tube, respectively). For details, see text.

1044 Li et al.

(middle right, arrow). In contrast, AME reconstructions(bottom row) reveal no loss in temporal fidelity, butinstead offer both enhanced structural sharpness and areduction of flickering streaking artifacts.

Finally, the average computation times per frame forthe three consecutive steps of the AME reconstruction(i.e., preliminary NLINV reconstruction, motion estima-tion and final AME reconstruction) are given in Figure 6for MRI studies of tongue and heart movements at twodifferent matrix resolutions (compare Table 2).

DISCUSSION

In comparison to NLINV reconstructions with and with-out temporal median filter, the proposed AME recon-struction for real-time MRI acquisitions yields serialimages with improved temporal acuity and less residualartifacts. The new method emerges as an extension of

the previously introduced NLINV reconstruction by esti-mating motion fields from multiple consecutive data setswith complementary spatial encodings. The additionalinformation is incorporated into the data consistencyterm of the nonlinear inverse problem for a simultaneousdetermination of both the image and coil sensitivities.

Extending other approaches for motion estimation inMRI, the present work is not limited to affine or rigidmotions. Moreover, the combination of AME with non-linear reconstruction permits an arbitrary choice of G

which defines the set of frames used for reconstructingthe actual frame. Extensive experimental studies (notshown) demonstrate that a choice of jGj smaller than thenumber of frames with complementary spatial encodingsfails to remove the temporally flickering artifacts. On theother hand, choosing jGj greater than the number of dif-ferently encoded frames does not further improve thereconstruction, but yields comparable image quality.Because the computational complexity increases as jGj

FIG. 4. (Left) NLINV, NLINV-MED, and AME reconstructions oftongue movements during generation of staccato sounds whileblowing the mouthpiece of a brass instrument (mid-sagittal plane,

33 ms resolution). (Right) Spatiotemporal profiles (vertical line inupper left image) of signal intensities for a selected 1.0 s period.

For details, see text.

FIG. 5. (Left) NLINV, NLINV-MED, and AME reconstructions of the

human heart (three-chamber view, 30 ms resolution). (Right) Spa-tiotemporal profiles (vertical line in upper left image) of signalintensities for a selected 2.7 s period. For details, see text.

Aggregated Motion Estimation for Image Reconstruction 1045

increases, this behavior explains the present choice ofG ¼ f�2;�1; 0; 1; 2g in all experiments.

As a stopping criterion for the iterations in NLINV,NLINV-MED, and AME, the well-known Morozov’s dis-crepancy principle was initially considered. Unfortunately,it forbids a unique choice of the threshold value for everyframe because the energy of the signal slightly changeswith time even for normalized k-space data. In the currentapplications, we, therefore, selected a fixed number of iter-ations (i.e., Newton steps) for each method, respectively,which gives satisfactory results. However, because data-driven choices are expected to be more sensitive, suchtechniques might be considered in future. An alternativeapproach to solve Eq. [4] might be optimization transfermethods (a.k.a. MM algorithms), which do not improvethe convergence rate, but are appealing with respect tosoftware modularization and algorithm maintenance (45).

A putative limitation of the AME method, which maydeteriorate its performance, stems from errors in the ini-tial NLINV reconstructions that may lead to suboptimalmotion estimations. A natural way to overcome thisproblem would be to run the AME procedure twice usingprevious AME (rather than NLINV) reconstructions formore accurate motion estimations – obviously at theexpense of further increasing the computational demand.In fact, in this proof-of-principle study, we do notaddress the issue of speeding up the AME algorithm,which is currently approximately 10 times slower than acomparable NLINV implementation on a laptop withMATLAB. However, because the computations for eachreceiver coil and of different frames are independent,AME is highly adaptable to parallel computing. Apartfrom interpolation, the involved calculations are simpli-fied to point-wise operations, fast Fourier transform, andscalar products. Moreover, the interpolation for non-Cartesian data may be separated from the iterative opti-mization, through a convolution with the point-spreadfunction. These features further ensure a possible speed-up by an implementation on graphical processing units(GPU). Recently, a corresponding GPU implementationof NLINV has already been demonstrated to yield onlinereconstructions with at least 17 frames per second(40,41).

CONCLUSIONS

This work introduces a new reconstruction method forreal-time MRI that offers improved temporal fidelity forvisualizing rapid dynamic changes. Preliminary experi-mental results for a motion phantom and in vivo humandata demonstrate the practical performance andimproved quality, in particular for fast moving objects.The concept is based on the incorporation of estimatedobject motions into the nonlinear inverse reconstructionprocess. Future improvements are expected by exploitingnew regularization methods and by accelerating the com-putational speed.

ACKNOWLEDGMENTS

The authors thank Sebastian Schaetz and Aaron Nieber-gall for the design and construction of the motion phan-tom. H.L. thanks for financial support by the ChinaScholarship Council and SFB 755. S.Z. thanks for finan-cial support by the DZHK (German Centre for Cardiovas-cular Research) and BMBF (German Ministry ofEducation and Research). A.M. gratefully acknowledgesthe support of DFG FOR 916. The support of the FelixBernstein Institute for Mathematical Statistics in the Bio-sciences and the Niedersachsen Vorab of the VolkswagenFoundation is gratefully acknowledged.

APPENDIX:

In this appendix, we will discuss the numerical methodfor solving the minimization [4].

Preconditioning

It is known that the quadratic optimization [4] is equivalentto solve its normal equation, precisely a linear equationwith symmetric positively definite matrix. Unfortunately,this linear equation is numerically ill-conditioned for largem. A simple preconditioning by the following variable sub-stitution can significantly reduce its condition number,making it numerically stable. Let

FIG. 6. Average computation times perframe for the preliminary NLINV recon-

struction, the motion estimation (ME)and the final AME reconstruction. Thevalues correspond to real-time MRI

studies of (left) tongue and (right) heartmovements at different matrix resolu-tions (see Table 2).

1046 Li et al.

W :rt

ðctþs;lÞs2G;l2L

!7!

rt

Fðað1þbjj � jj2Þm=2ctþs;lÞ� �

s2G;l2L

0@

1A;

If x :¼ ðrt; ðctþr;lÞr2G;l2LÞT ; ~x :¼Wx;Gtþs :¼ FtþsW

�1; s2 G,then an equivalent form of Eq. [4] is given by

~hn ¼ argmin~h

Xs2G

jjG0tþsð~xnÞ~h � ðytþs � Gtþsð~xnÞÞjj2

þ anjj~xn þ ~h � ~x0jj2;~xnþ1 ¼ ~xn þ ~hn:

Explicitly, the optimality condition for this quadraticoptimization is

Xs2G

G0tþsð~xnÞHG0tþsð~xnÞ þ anI

!~hn

¼Xs2G

G0tþsð~xnÞH ytþs � Gtþsð~xnÞð Þ � anð~xn � ~x0Þ; [A1]

where I is the identity operator.

Discretization

For s � G, l � L, let us denote

Ctþs;lð~xÞ :¼ ft;sðrÞ � FH a�1ð1þ bjj � jj�m=2Þ~ctþs;l

� �;

where ~x :¼ ðr; ð~ctþr;jÞr2G;j2LÞT . A detailed formula of Eq.

[A1] is

Xs2G;l2L

C 0tþs;l ~xnð ÞHFH SHtþsStþsFC 0tþs;l ~xnð Þ þ anI

!~dn

¼X

s2G;l2L

C 0tþs;lð~xnÞHFH SHtþs ytþs;l � StþsFCtþs;lð~xnÞ� �

� anð~xn � ~x0Þ:

This equation will be solved by the conjugate gradientCG algorithm which requires repeated application of the

operations StþsF and FH SHtþs. For numerical computa-

tion, every function involved needs to be approximatedby a discretized form of points on a rectangular grid.Because spin density and coil profiles are compactlysupported, the Fourier transform F can be computed byfast Fourier transform (FFT) with proper periodicextension. If the sampling trajectory represents anon-Cartesian radial scheme as used for real-time MRIin (6,7), the computation involving the sampling

operators Stþs and SHtþs is not straightforward. With

hStþsf ; gi ¼ hf ;SHtþsgi and Stþsf ¼ ðf ðktþs;jÞÞMj¼1, we have

SHtþsg ¼

XM

j¼1d � � ktþs;j

� �g ktþs;j

� �: Then, FH SH

tþsytþs;l ¼XM

j¼1ytþs;le

2pihktþs;j ;�i can be computed by inverse FFT

after gridding (44,46,47) or nonuniform FFT (48,49).

With respect to FH SHtþsStþsF , we have

FHSHtþsStþsF f

� �xð Þ

¼Z XM

j¼1

d y � ktþs;j

� �e2pihx;yi

Zf zð Þe�2pihktþs;j ;zidzdy

¼Z XM

j¼1

e2pihktþs;j x�zif zð Þdz

¼ q � f ¼ FH Fqð Þ F fð Þð Þ;

where qðxÞ :¼XM

j¼1e2pihktþs ;xi. It can be computed by two

FFT and one inverse FFT, with Fq given by the griddingalgorithm. It is worth to notice that in Cartesian case Fqequals ones at measured points and zeros elsewhere. Tosum up, Eq. [A1] can be numerically solved in an effi-cient way.

REFERENCES

1. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sen-

sitivity encoding for fast MRI. Magn Reson Med 1999;42:952–962.

2. Sodickson DK, Manning WJ. Simultaneous acquisition of spatial har-

monics (SMASH): fast imaging with radiofrequency coil arrays. Magn

Reson Med 1997;38:591–603.

3. Griswold MA, Jakob PM, Heidemann RM, Nittka M, Jellus V, Wang

JM, Kiefer B, Haase A. Generalized autocalibrating partially parallel

acquisitions (GRAPPA). Magn Reson Med 2002;47:1202–1210.

4. Ying L, Sheng JH. Joint image reconstruction and sensitivity estima-

tion in SENSE (JSENSE). Magn Reson Med 2007;57:1196–1202.

5. Uecker M, Hohage T, Block KT, Frahm J. Image reconstruction by

regularized nonlinear inversion - joint estimation of coil sensitivities

and image content. Magn Reson Med 2008;60:674–682.

6. Uecker M, Zhang S, Frahm J. Nonlinear inverse reconstruction for

real-time MRI of the human heart using undersampled radial FLASH.

Magn Reson Med 2010;63:1456–1462.

7. Uecker M, Zhang S, Voit D, Karaus A, Merboldt KD, Frahm J. Real-

time MRI at a resolution of 20 ms. NMR Biomed 2010;23:986–994.

8. Zhang S, Uecker M, Voit D, Merboldt KD, Frahm J. Real-time cardio-

vascular magnetic resonance at high temporal resolution: radial

FLASH with nonlinear inverse reconstruction. J Cardiovasc Magn

Reson 2010;12:39–45.

9. Knoll F, Clason C, Bredies K, Uecker M, Stollberger R. Parallel imag-

ing with nonlinear reconstruction using variational penalties. Magn

Reson Med 2012;67:34–41.

10. S€umb€ul U, Santos JM, Pauly JM. Improved time series reconstruction

for dynamic magnetic resonance imaging. IEEE Trans Med Imaging

2009;28:1093–1104.

11. S€umb€ul U, Santos JM, Pauly JM. A practical acceleration algorithm

for real-time imaging. IEEE Trans Med Imaging 2009;28:2042–2051.

12. Ehman RL, Felmlee JP. Adaptive technique for high-definition MR

imaging of moving structures. Radiology 1989;173:255–263.

13. Ward HA, Riederer SJ, Grimm RC, Ehman RL, Felmlee JP, Jack CR.

Prospective multiaxial motion correction for fMRI. Magn Reson Med

2000;43:459–469.

14. Welch EB, Manduca A, Grimm RC, Ward HA, Jack CR Jr. Spherical

navigator echoes for full 3D rigid body motion measurement in MRI.

Magn Reson Med 2001;47:32–41.

15. van der Kouwe AJ, Benner T, Dale AM. Real-time rigid body motion

correction and shimming using cloverleaf navigators. Magn Reson

Med 2006;56:1019–1032.

16. White N, Roddey C, Shankaranarayanan A, Han E, Rettmann D,

Santos J, Kuperman J, Dale A. PROMO: real-time prospective motion

correction in MRI using image-based tracking. Magn Reson Med

2009;63:91–105.

17. Ooi MB, Krueger S, Thomas WJ, Swaminathan SV, Brown TR. Pro-

spective real-time correction for arbitrary head motion using active

markers. Magn Reson Med 2009;62:943–954.

18. Lin W, Huang F, B€ornert P, Li Y, Reykowski A. Motion correction

using an enhanced floating navigator and GRAPPA operations. Magn

Reson Med 2009;63:339–348.

Aggregated Motion Estimation for Image Reconstruction 1047

19. Nielsen T, Bornert P. Iterative motion compensated reconstruction for

parallel imaging using an orbital navigator. Magn Reson Med 2011;

66:1339–1345.

20. Schaffter T, Rasche V, Carlsen IC. Motion compensated projection

reconstruction. Magn Reson Med 1999;41:954–963.

21. Batchelor PG, Atkinson D, Irarrazaval P, Hill DL, Hajnal J, Larkman

D. Matrix description of general motion correction applied to multi-

shot images. Magn Reson Med 2005;54:1273–1280.

22. Kellman P, Chefd’hotel C, Lorenz CH, Mancini C, Arai AE, McVeigh

ER. Fully automatic, retrospective enhancement of real-time acquired

cardiac cine MR images using image-based navigators and respiratory

motion-corrected averaging. Magn Reson Med 2008;59:771–778.

23. Kellman P, Chefd’hotel C, Lorenz CH, Mancini C, Arai AE, McVeigh

ER. High spatial and temporal resolution cardiac cine MRI from retro-

spective reconstruction of data acquired in real time using motion

correction and resorting. Magn Reson Med 2009;62:1557–1564.

24. Hansen MS, Sorensen TS, Arai AE, Kellman P. Retrospective recon-

struction of high temporal resolution cine images from real-time MRI

using iterative motion correction. Magn Reson Med 2012;68:741–750.

25. Usman M, Atkinson D, Odille F, Kolbitsch C, Vaillant G, Schaeffter

T, Batchelor PG, Prieto C. Motion corrected compressed sensing for

free-breathing dynamic cardiac MRI. Magn Reson Med 2013;70:504–

516.

26. Vuissoz PA, Odille F, Fernandez B, Lohezic M, Benhadid A, Mandry

D, Felblinger J. Free-breathing imaging of the heart using 2D cine-

GRICS (generalized reconstruction by inversion of coupled systems)

with assessment of ventricular volumes and function. Magn Reson

Med 2012;35:340–351.

27. Wei L, Feng H, Duensing GR, Reykowski A. High temporal resolution

retrospective motion correction with radial parallel imaging. Magn

Reson Med 2012;67:1097–1105.

28. Asif MS, Hamilton L, Brummer M, Romberg J. Motion-adaptive

spatio-temporal regularization for accelerated dynamic MRI. Magn

Reson Med 2013;70:800–812.

29. Anderson AG III, Velikina J, Block W, Wieben O, Samsonov A.

Adaptive retrospective correction of motion artifacts in cranial MRI

with multicoil three-dimensional radial acquisitions. Magn Reson

Med 2013;69:1094–1103.

30. Odille F, Cindea N, Mandry D, Pasquier C, Vuissoz PA, Felblinger J.

Generalized MRI reconstruction including elastic physiological motion

and coil sensitivity encoding. Magn Reson Med 2008;59:1401–1411.

31. Odille F, Vuissoz PA, Marie PY, Felblinger J. Generalized reconstruc-

tion by inversion of coupled systems (GRICS) applied to free-

breathing MRI. Magn Reson Med 2008;60:146–157.

32. Bakushinskii AB. The problem of the convergence of the iteratively

regularized Gauss-Newton methods. Comput Math Math Phys 1992;

32:1353–1359.

33. Bauer F, Hohage T, Munk A. Iteratively regularized Gauss-Newton

method for nonlinear inverse problems with random noise. SIAM J

Num Anal 2009;47:1827–1846.

34. Hohage T. Iterative methods in inverse obstacle scattering: regulariza-

tion theory of linear and nonlinear exponentially ill-posed problems.

Ph.D. dissertation. Johannes-Kepler-Universitat; 1999.

35. Wedel A, Pock T, Zach C, Bischof H, Cremers D. An improved

algorithm for tv-l1 optical flow. In: Cremers D, Rosenhahn B,

Yuille AL, Schmidt FR, editors. Statistical and geometrical

approaches to visual motion analysis. Berlin, Heidelberg: Springer;

2009. p 23–45.

36. Chambolle A, Pock T. A first-order primal-dual algorithm for convex

problems with applications to imaging. J Math Imag Vis 2011;40:120–

145.

37. Schaetz S, Untenberger M, Niebergall A, Frahm J. Motion phantom

for real-time MRI. In Proceedings of the 21st Annual Meeting of

ISMRM, Salt Lake City, Utah, USA, 2013. Abstract 4340.

38. Zhang S, Block KT, Frahm J. Magnetic resonance imaging in real

time: advances using radial FLASH. J Magn Reson Imaging 2010;31:

101–109.

39. Niebergall A, Zhang S, Kunay E, Keydana G, Job M, Uecker M,

Frahm J. Real-time MRI of speaking at a resolution of 33 ms: under-

sampled radial FLASH with nonlinear inverse reconstruction. Magn

Reson Med 2012;69:477–485.

40. Schaetz S, Uecker M. A multi-GPU programming library for real-time

applications. In: Algorithms and architectures for parallel processing

(Springer). Lect Notes Comp Sci 2012;7439:114–128.

41. Uecker M, Zhang S, Voit D, Merboldt KD, Frahm J. Real-time

MRI—recent advances using radial FLASH. Imaging Med 2012;4:461–

476.

42. Buehrer M, Pruessmann KP, Boesiger P, Kozerke S. Array compres-

sion for MRI with large coil arrays. Magn Reson Med 2007;57:1131–

1139.

43. Hunag F, Vijayakumar S, Li Y, Hertel S, Duensing GR. A software

channel compression technique for faster reconstruction with many

channels. Magn Reson Med 2008;26:133–141.

44. Beatty PJ, Nishimura DG, Pauly JM. Rapid gridding reconstruction

with a minimal oversampling ratio. IEEE Trans Med Imag 2005;24:

799–808.

45. Fessler JA. Optimization transfer approach to joint registration/recon-

struction for motion-compensated image reconstruction. Proc IEEE

Int Symp Biomed Imaging 2010:596–599.

46. Osullivan JD. A fast sinc function gridding algorithm for Fourier

inversion in computer-tomography. IEEE Trans Med Imaging 1985;4:

200–207.

47. Jackson JI, Meyer CH, Nishimura DG, Macovski A. Selection of a con-

volution function for Fourier inversion using gridding. IEEE Trans

Med Imaging 1991;10:473–478.

48. Fessler JA, Sutton BP. Nonuniform fast Fourier transforms using min-

max interpolation. IEEE Trans Signal Proc 2003;51:560–574.

49. Keiner J, Kunis S, Potts D. Using nfft 3—a software library for various

nonequispaced fast fourier transforms. ACM Trans Math Software

2009;36:1–30.

1048 Li et al.