1

Deep-learning-based Optimization of theUnder-sampling Pattern in MRI

Cagla D. Bahadir*, Alan Q. Wang*, Adrian V. Dalca, and Mert R. Sabuncu

In compressed sensing MRI (CS-MRI), k-space measurementsare under-sampled to achieve accelerated scan times. CS-MRIpresents two fundamental problems: (1) where to sample and(2) how to reconstruct an under-sampled scan. In this paper,we tackle both problems simultaneously for the specific caseof 2D Cartesian sampling, using a novel end-to-end learningframework that we call LOUPE (Learning-based Optimizationof the Under-sampling PattErn). Our method trains a neuralnetwork model on a set of full-resolution MRI scans, whichare retrospectively under-sampled on a 2D Cartesian grid andforwarded to an anti-aliasing (a.k.a. reconstruction) model thatcomputes a reconstruction, which is in turn compared with theinput. This formulation enables a data-driven optimized under-sampling pattern at a given sparsity level. In our experiments, wedemonstrate that LOUPE-optimized under-sampling masks aredata-dependent, varying significantly with the imaged anatomy,and perform well with different reconstruction methods. Wepresent empirical results obtained with a large-scale, publiclyavailable knee MRI dataset, where LOUPE offered superiorreconstruction quality across different conditions. Even withan aggressive 8-fold acceleration rate, LOUPE’s reconstructionscontained much of the anatomical detail that was missed byalternative masks and reconstruction methods. Our experimentsalso show how LOUPE yielded optimal under-sampling patternsthat were significantly different for brain vs knee MRI scans. Ourcode is made freely available at https://github.com/cagladbahadir/LOUPE/.

Index Terms—Compressed Sensing, Magnetic ResonanceImaging, Deep Learning

I. INTRODUCTION

MAGNETIC Resonance Imaging (MRI) is a ubiquitous,non-invasive, and versatile biomedical imaging tech-

nology. A central challenge in MRI is long scan times, whichconstrains accessibility and leads to high costs. One remedyis to accelerate MRI via compressed sensing [1], [2]. In com-pressed sensing MRI, k-space data (i.e., the Fourier transformof the image) is sampled below the Nyquist-Shannon rate [1],which is often referred to as “under-sampling.” Given anunder-sampled set of measurements, the objective is to “recon-struct” the full-resolution MRI. This is the main problem thatmost of the compressed sensing literature is focused on andis conventionally formulated as an optimization problem thattrades off two objectives: one that quantifies the fit between themeasurements and the reconstruction (sometimes referred to as

Cagla D. Bahadir and Mert R. Sabuncu are with the Meinig Schoolof Biomedical Engineering, Cornell University. Alan Q. Wang and MertR. Sabuncu are with the School of Electrical and Computer Engineering,Cornell University. Adrian V. Dalca is with the Computer Science andArtificial Intelligence Lab at the Massachusetts Institute of Technology and theA.A. Martinos Center for Biomedical Imaging at the Massachusetts GeneralHospital. * indicates equal contribution. Cagla D. Bahadir and Alan Q. Wangare co-first authors.

data consistency), and another that captures prior knowledgeon the distribution of MRI data. This latter objective is oftenachieved via the incorporation of regularization terms, suchas the total variation penalty and/or a sparsity-inducing normon transformation coefficients, like wavelets or a dictionarydecomposition [3]. Such regularization functions aim to cap-ture different properties of real-world MR images, which inturn help the ill-posed reconstruction problem by guiding tomore realistic solutions. One approach to develop a data-drivenregularization function is to construct a sparsifying dictionary,for example, based on image patches [4]–[6].

Once the optimization problem is set up, the reconstructionalgorithm often iteratively minimizes the regularization term(s)and enforces data consistency in k-space. This is classicallysolved for each acquired dataset, independently, and fromscratch - a process that can be computationally demanding.As we describe in the following section, there has been arecent surge in machine learning based methods that take adifferent, and often computationally more efficient approachto solving the reconstruction problem. These techniques aregradually becoming more widespread and expected to com-plement existing regularized optimization based approaches.

Another critical component of compressed sensing MRI isthe under-sampling pattern. For a given acceleration rate, thereis an exponentially large number of possible patterns one canimplement for under-sampling. Each of these under-samplingpatterns will in general lead to different reconstruction per-formance that will depend on the statistics of the data andthe utilized reconstruction method. One way to view this isto regard the reconstruction model as imputing the parts ofthe Fourier spectrum that was not sampled. For example, afrequency component that is constant for all possible datasetswe might observe, does not need to be sampled when we usea reconstruction model that can leverage this information. Onthe other hand, highly variable parts of the spectrum will likelyneed to be measured to achieve accurate reconstructions. Thissimple viewpoint ignores the potentially multi-variate nature ofthe distribution of k-space measurements. For instance, miss-ing parts of the Fourier spectrum might be reliably imputedfrom other acquired parts, due to strong statistical dependency.Widely used under-sampling strategies in compressed sensingMRI include Random Uniform [2], Variable Density [7] andequi-spaced Cartesian [8] with skipped lines. These tech-niques, however, are often implemented heuristically, not ina data-driven adaptive fashion, and their popularity is largelydue to their ease of implementation and good performancewhen coupled with popular reconstruction methods. As wediscuss below, some recent efforts compute optimized under-sampling patterns - a literature that is closely related to our

arX

iv:1

907.

1137

4v3

[ee

ss.I

V]

18

Jun

2020

2

primary objective.In this paper, we propose to merge the two core problems of

compressed sensing MRI: (1) optimizing the under-samplingpattern and (2) reconstruction. We consider these two prob-lems simultaneously because they are closely inter-related.Specifically, the optimal under-sampling pattern should, ingeneral, depend on the reconstruction method and vice versa.Inspired by recent developments in machine learning, wepropose a novel end-to-end deep learning strategy to solvethe combined problem. We call our method LOUPE, whichstands for Learning-based Optimization of the Under-samplingPattErn. A preliminary version of LOUPE was published asa conference paper [9]. In this journal paper, we present anextended treatment of the literature, an important modificationto the LOUPE objective that enables us to directly set thedesired sparsity level while obviating the need to identifythe value of an extra hyper-parameter, more details on themethods, and new experimental results.

The rest of the paper is organized as follows. In the fol-lowing two sections, we review two closely related bodies ofwork: a rapidly growing list of recent papers that use machinelearning for efficient and accurate reconstruction; and severalproposed approaches to optimize the under-sampling pattern.Section IV then presents the mathematical and implementationdetails of the proposed method, LOUPE. Section V presentsour experiments, where we compare the performance obtainedwith several benchmark reconstruction methods and under-sampling patterns. Section VI concludes with a discussion.

II. MACHINE LEARNING FOR UNDER-SAMPLED IMAGERECONSTRUCTION

Over the last few years, machine learning methods havebeen increasingly used for medical image reconstruction [10]–[15], including compressed sensing MRI. An earlier exam-ple is the Bayesian non-parametric dictionary learning ap-proach [16]. In this framework, dictionary learning is used inthe reconstruction problem to obtain a regularization objectivethat is customized to the image at hand. The reconstructionproblem is solved via a traditional optimization approach.

More recently, fueled by the success of deep learning,several machine learning techniques have been proposed toimplement efficient and accurate reconstruction models. Forinstance, in a high profile paper, Zhu et al. demonstrated a ver-satile supervised learning approach, called AUTOMAP [13].This method uses a neural network model to map sensor mea-surements directly to reconstruction images, via a manifoldrepresentation. The experimental results show that AUTOMAPcan produce high quality reconstructions that are robust tonoise and other artifacts. Another machine learning basedreconstruction approach involves using a neural network toefficiently execute the iterations of the Alternating DirectionMethod of Multipliers (ADMM) method that solves a conven-tional optimization problem [10], [17]. This technique, calledADDM-Net, uses an “unrolled” neural network architectureto implement the iterative optimization procedure. In a similarapproach, an unrolled neural network is used to implement theiterations of the Landweber method [15].

Another class of methods rely on the U-Net architecture [18]or its variants. For example U-Net-like models have beentrained in a supervised fashion to remove aliasing artifacts inthe imaging domain [19], [20]. The U-Net architecture has alsobeen used by appending with a forward physics based modelto learn phase masks for depth estimation in other computervision applications [21]. In these methods, the neural networktakes as input a poor quality reconstruction (e.g., obtained via asimple inverse Fourier transform applied to zero-filled k-spacedata) to compute a high quality output, where the objectiveis to minimize the loss function (e.g., squared difference)between the output and the ground truth provided duringtraining. Inspired by the success of Generative AdversarialNetworks (GANs) [22], several groups have proposed the useof an adversarial loss, in addition to the more conventional lossfunctions, to obtain better quality reconstructions [23]–[26].

The method we propose in this paper builds on the neuralnet-based reconstruction framework, which offers a compu-tationally efficient and differentiable anti-aliasing model. Wecombine the anti-aliasing model with a retrospective under-sampling module, and learn both blocks simultaneously.

III. DATA-DRIVEN UNDER-SAMPLING IN COMPRESSEDSENSING MRI

The under-sampling pattern in k-space is closely related toreconstruction performance. Several under-sampling patternsare widely used, including Random Uniform [2], VariableDensity [7], [27] and equi-spaced Cartesian [8] with skippedlines. Since the early work by Lustig et al. [27], random orstochastic under-sampling strategies are commonly used, asthey induce noise-like artifacts that are often easier to removeduring reconstruction. However, as several papers have pointedout, the quality of reconstruction can also be improved byadapting the under-sampling pattern to the data and applica-tion. We underscore that there is a related, but more generalproblem of optimizing projections in compressed sensing,which has received considerable attention [28]–[31]. Below,we review adaptive under-sampling strategies in compressedsensing MRI, which is the focus of our paper.

Knoll et al. proposed an under-sampling strategy that fol-lows the power spectrum of a provided example (template)image [32]. This method collects denser samples in parts ofk-space that has most of the power concentrated, such as thelow-frequency components of real-world MRIs. The authorsargue that most of the anatomical variability is reflected in thephase and not magnitude of the Fourier spectrum, justifyingthe decision to rely on the magnitude to construct the under-sampling pattern. In experiments, they apply their methodto both knee and brain MRI scans, demonstrating betterperformance than parametric variable density under-samplingpatterns [27].

Kumar Anand et al. presented a second-order cone op-timization technique for finding the optimal under-samplingtrajectory in volumetric MRI scans [33]. Their heuristic strat-egy accounts for coverage, hardware limitations and signalgeneration in a single convex optimization problem. Buildingon this approach, Curtis et al. employed a genetic algorithm

3

to optimize sampling trajectories in k-space, while accountingfor multi-coil configurations [34].

Seeger et al. employed a Bayesian inference approach tooptimize Cartesian and spiral trajectories [35]. Their iterative,greedy technique seeks the under-sampling pattern in k-spacethat minimizes the uncertainty in the computed reconstruction,conditioned on the acquired measurements. In a recent paperthat explores a similar direction, Haldar et al. proposed anew framework called Oracle-based Experiment Design forImaging Parsimoniously Under Sparsity constraints (OEDI-PUS) [36]. OEDIPUS solves an integer programming problemto minimize a lower (Cramer-Rao) bound on the variance ofthe reconstruction.

Roman et al. offered a novel theoretical perspective oncompressed sensing, which underscores the importance ofadapting the under-sampling strategy to the structure in thedata [37]. The authors also presented an innovative multilevelsampling approach that depends on the resolution and thestructure of the data that empirically outperforms competingunder-sampling schemes.

An alternative class of data-driven approaches aims to findthe optimal under-sampling pattern that yields the best re-construction or imputation quality, using some full-resolutiontraining data that is retrospectively under-sampled [38]. Thecentral challenge in this framework is to efficiently solvetwo computationally challenging nested optimization prob-lems. The first, outer problem is to identify the optimalunder-sampling mask or sampling trajectory that achievesbest reconstruction quality, which is in turn the result ofan inner optimization step. Several authors have proposedto solve this nested pair of problems via heuristic, greedyalgorithms [39]–[43]. In an empirical study, Zijlstra et al.demonstrated that a data-driven optimization approach [40],[44] can yield better reconstructions than those obtained withmore conventional methods [27], [32]. However, prior data-driven methods mostly lack the computational efficiency tohandle large-scale full-resolution (training) data. Furthermore,they are often not flexible enough to deal with different typesof under-sampling schemes.

In this paper, we present a novel, flexible, and computa-tionally efficient data-driven approach to optimize the under-sampling pattern. Instead of formulating the problem as twonested optimization problems, we leverage modern machinelearning techniques and take an end-to-end learning perspec-tive. Similar to recent deep learning based reconstruction tech-niques [19], [20], our implementation employs the U-Net [18]architecture, which we append with a probabilistic under-sampling step that is also learned. We assume we are pro-vided with full-resolution MRI data, which we retrospectivelyunder-sample. We note that there have been contemporaneousefforts [45]–[47] that build on our prior work [9] to take adeep learning-based approach similar to ours for optimizingthe under-sampling pattern in compressed sensing.

IV. METHOD

A. Learning-based Optimization of Under-sampling PatternLOUPE solves the two problems of compressed sensing

simultaneously: (1) identifying the optimal under-sampling

pattern; and (2) reconstructing from the under-sampled mea-surements. Building on stochastic strategies of compressedsensing, we consider a probabilistic mask P , which describesan independent Bernoulli random variable at each k-spacepoint. The probabilistic mask P is defined on the full-resolution k-space grid, and at each point takes on non-negative continuous probability values, i.e., P ∈ [0, 1]d, whered is the total number of grid points. For example, for a100 × 100 image, d = 10, 000. We parameterize P with anunconstrained image O ∈ Rd, such that P = σt(O). Here,σt denotes an element-wise sigmoid, with slope t, which istreated as a hyper-parameter. Pi = 1

1+e−tOi, where the sub-

script i indexes the grid points. Binary realizations drawnfrom P represent an under-sampling mask M ∈ {0, 1}d: i.e.,M ∼

∏di=1 B(Pi), where B(p) denotes a Bernoulli random

variable with parameter p. The binary mask M has value 1for grid points that are acquired and 0 for points that were notacquired.

Suppose we are provided with a collection of complex-valued full-resolution images, denoted as {xj ∈ Cd}nj=1,where j corresponds to the scan number and n is the totalnumber of scans. In our following treatment, without loss ofgenerality, we will assume the provided data are in imagedomain. This can be modified to accept raw k-space measure-ments, by simply removing a forward Fourier transform. Inthe LOUPE framework, we solve the following problem:

minO,θ

EM∼∏i=1 B(σt(Oi))

n∑j=1

‖Aθ(FHdiag(M)Fxj)−xj‖22,

such that1

d‖σt(O)‖1 = α (1)

where F ∈ Cd×d stands for the (forward) Fourier transformmatrix, FH is the inverse Fourier transform matrix, Aθ denotesan anti-aliasing reconstruction function parameterized with θ,‖·‖2 denotes the L2 norm, ‖·‖1 denotes the L1 norm, α ∈ [0, 1]is the desired sparsity level, and diag(·) denotes a diagonalmatrix obtained by setting the diagonal to the argument vector.The loss function of Equation (1) quantifies the average qualityof the reconstruction. The constraint 1

d‖σt(O)‖1 = 1d‖P ‖1 =

α ensures that the probabilistic under-sampling mask has anaverage value of α, which will correspond to an accelerationfactor of R = 1/α. We emphasize that the problem formula-tion of Equation (1) is slightly different than the formulationin our conference paper [9], where we employed a sparsityinducing regularization instead of a hard constraint. Thatformulation required fine-tuning a hyper-parameter in order toobtain the desired sparsity level, which was computationallyinefficient. In this paper we implement a practical strategy,described below, to solve the problem with the hard sparsityconstraint directly.

The loss function involves an expectation over the randombinary mask M . We approximate this expectation using a

4

Monte Carlo-based sample averaging strategy:

minO,θ

n∑j=1

1

K

K∑k=1

‖Aθ(FHdiag(m(k))Fxj)− xj‖22,

such that1

d‖σt(O)‖1 = α (2)

where m(k) are independent realizations drawn from∏i B(σt(Oi)). Similar to the re-parameterization trick used

in variational techniques, such as the VAE [48], we can re-write Equation (2):

minO,θ

n∑j=1

1

K

K∑k=1

‖Aθ(FHdiag(U (k) ≤ σt(O))Fxj)−xj‖22,

such that1

d‖σt(O)‖1 = α (3)

where U (k) are independent realizations drawn from∏di=1 U(0, 1), a spatially independent set of uniform random

variables on [0, 1]. The result of the inequality operation is 1if the condition is satisfied and 0 otherwise. In Equation (3)the random draws are thus from a constant distribution (inde-pendent uniform) and the probabilistic mask P (through itsparameterization O), only effects the thresholding operation.

B. Implementation

In our first version of LOUPE, we used a 2D U-Netarchitecture [18] for the anti-aliasing function, as depicted inFigure 1 and similar to other prior work [19], [20]. To enable adifferentiable loss function, we relaxed the non-differentiablethreshold operation of Equation (3) with a sigmoid. This relax-ation is similar to the one used in recent Gumbel-softmax [49]and concrete distributions [50] in training neural networks withdiscrete representations. Our final objective is:

minO,θ

n∑j=1

1

K

K∑k=1

‖Aθ(FHdiag(σs(σt(O)−U (k)))Fxj)−xj‖22,

such that1

d‖σt(O)‖1 = α (4)

where σs(a) = 11+e−sa is a sigmoid with slope s.

A critical issue for LOUPE is the enforcement of the hardconstraint 1

d‖σt(O)‖1 = 1d‖P ‖1 = α. To achieve this we

introduce a normalization layer, which rescales P to satisfythe constraint. We first define p = ‖P ‖1

d , which is the averagevalue of the pre-normalization probabilistic mask. Note that1 − p is the average value of 1 − ‖P ‖1. We define thenormalization layer as:

Nα(P ) =

{αpP , if p ≥ α1− 1−α

1−p (1− P ), otherwise.(5)

It can be shown that Equation (5) yields Nα(P ) ∈ [0, 1]d and‖Nα(P )‖1

d = α. This normalization trick allows us to convert

the constrained problem of Equation (4) to the followingunconstrained problem:

minO,θ

n∑j=1

1

K

K∑k=1

‖Aθ(FHdiag(σs(Nα(σt(O))−U (k)))Fxj)− xj‖22. (6)

Figure 1 illustrates an instantiation of the LOUPE model.In the depicted version of LOUPE, we used a U-Net archi-tecture to implement Aθ, the reconstruction network. We havealso experimented with an alternative reconstruction model,namely Cascade-Net proposed in [51]. In the supplementarymaterial, we present LOUPE results obtained with this alter-native implementation. The LOUPE network accepts complex2D images represented as two channels, corresponding tothe imaginary and real components. The network applies aforward discrete Fourier transform, F , converting the datainto k-space. The k-space measurements are under-sampledby m = σs(P − u(k)), which approximates a Monte Carlorealization of the probabilistic mask P . The values of theprobabilistic mask are computed using the sigmoid operationof pixel-wise parameters O that are learned during the trainingprocess. The mask m is multiplied element-wise with thek-space data (approximating retrospective under-sampling byinserting zeros at missing points), which is then passed to aninverse discrete Fourier transform FH .

This image, which will typically contain aliasing artifacts,is then provided as input to the anti-aliasing network Aθ(·),also called the reconstruction network. This network takesa complex-valued under-sampled image, represented as two-channels, and aims to minimize the reconstruction error, suchas the squared difference between the magnitude images ofground truth and the reconstruction. We can view the entirepipeline to be made up of two building blocks: the firstoptimizing the under-sampling pattern, and the second solvingthe reconstruction problem.

The model was implemented in Keras [52], with Tensor-flow [53] in the back-end. Custom functions were adaptedfrom the Neuron library [54]. ADAM [55] with an initial learn-ing rate of 0.001 was used for optimization, and learning wasterminated when validation loss plateaued. Consistent withprior work [48]–[50], a single Monte Carlo realization wasdrawn for each training datapoint. This is common practice invariational neural networks as it is a computationally efficientapproach that yields an unbiased estimate of the gradient that isused in stochastic gradient descent. We used a batch size of 16.We used a slope value of s = 200 for σs that approximates thethresholding operation and a slope value of t = 5 for σt thatsquashes the values of O to the range [0, 1]. These values forthe slope hyper-parameters were chosen based on a grid searchstrategy, where we identified the pair of values that yieldsthe smallest validation loss (see Supplementary Material). Wenote, however, that we did not observe a significant amountof sensitivity to these values. Our code is freely available at:https://github.com/cagladbahadir/LOUPE/

5

Fig. 1. LOUPE architecture with two building blocks: The Under-sampling Pattern Optimization Network and the Anti-aliasing Network. The model implementsan end-to-end learning framework for optimizing the under-sampling pattern while learning the parameters for faithful reconstructions. The normalizationlayer for the probabilistic mask computes Equation (5). Red arrows denote 2D convolutional layers with a kernel size of 3 × 3, followed by Leaky ReLUactivation and Batch Normalization. Green vertical arrows represent average pooling operations and yellow vertical arrows refer to up-sampling. The initialnumber of channels in the U-Net is 64, which doubles after each average pooling layer. The U-Net also uses skip connections depicted with gray horizontalarrows, that concatenate the corresponding layers to facilitate better information flow through the network.

V. EMPIRICAL ANALYSIS

A. Data

We conducted our experiments using the NYU fastMRIdataset [56] (fastmri.med.nyu.edu). This is a freely available,large-scale, public data set of knee MRI scans. The datasetoriginally comprises of 2D coronal knee scans acquired withtwo pulse sequences that result with Proton Density (PD) andProton Density Fat Supressed (PDFS) weighted images. Inour experiments, we used the provided emulated single-coil(ESC) k-space data from the Biograph mMR (3T) scanner,which were derived from raw 15-channel multi-coil data. Weused 100 volumes from the provided official training datasetas our training dataset, and split the provided validation datainto two halves, where we used 10 volumes for validationand 10 volumes for test. We could not rely on the providedtest data for evaluation as they were not fully sampled. Forthe scans we analyzed, the sequence parameters were: echotrain length of 4, matrix size of 320x320, in plane resolutionof 0.5mm × 0.5mm, slice thickness of 3mm and no gapbetween slices. The time of repetition (TR) varied between2200 and 3000ms and the Echo Time (TE) ranged between27 and 34ms. Training volumes had 38± 4 slices, where thevalidation volumes had 37± 3 and test volumes had 38± 4.

Each set (training, validation and test) had differing slicesizes across volumes. After taking the Inverse Fourier Trans-

form of the ESC k-space data, and rotating and flipping theimages to match the orientation in the fastMRI paper [56], wecropped the central 320x320 and normalized by dividing tothe maximum magnitude within each volume.

The U-Net architecture we employed was also used in theoriginal NYU fastMRI study [56] for reconstruction, with theonly difference that their implementation accepted a single-channel, real-valued input image.

B. Evaluation

We used three metrics in the quantitative evaluations of thereconstructions with respect to the ground truth images createdfrom fully sampled measurements: (1) peak signal to noiseratio (PSNR), (2) structural similarity index (SSIM), and (3)high frequency error norm (HFEN).

PSNR is a widely used metric for evaluating the quality ofreconstructions in compressed sensing applications [10], [57],defined as:

PSNR(x, x) = 10 log10

max(x)2d

‖x− x‖22, (7)

where d is, as above, the total number of pixels in the full-resolution grid, and x denotes the reconstruction of the groundtruth full-resolution image x.

6

SSIM aims to quantify the perceived image quality [58]:

SSIM(x, x) =(2µxµx + c1) + (2σxx + c2)

(µ2x + µ2

x + c1)(σ2x + σ2

x + c2), (8)

where µx and µx are defined as the “local” average values forthe original and reconstruction images, respectively, computedwithin an N × N neighborhood. Similarly, σ2

x and σ2x are

the local variances; and σxx is local covariance between thereconstruction and ground truth images. c1 = (k1L)2 and c2 =(k2L)2 are constants that numerically stabilize the division,where L is the dynamic range of the pixel values and k1 andk2 are user defined. We use the parameters provided in [56]for computing SSIM values: k1 = 0.01, k2 = 0.03, and awindow size of 7× 7. The dynamic range of pixel values wascomputed over the entire volume.

Finally, we also report the high-frequency error norm(HFEN), which is used to quantify the quality of reconstruc-tion of edges and fine features. Following [5], we use a 15×15Laplacian of Gaussian (LoG) filter with a standard deviationof 1.5 pixels. The HFEN is then computed as the L2 differencebetween LoG filtered ground truth and reconstruction images.

C. Benchmarks

We compared the LOUPE-optimized mask and other maskconfigurations, together with one machine learning (ML) basedand three non-ML based reconstruction techniques.

1) Reconstruction MethodsEach reconstruction method was optimized on a represen-

tative slice from the data-set to find the values of hyper-parameters that yielded the best quality reconstruction foreach of the individual masks. In other words, hyper-parametertuning was done separately for each mask.

The first benchmark reconstruction method is BM3D [59],which is an iterative algorithm that alternates between de-noising and reconstructions steps, and was shown to yieldfaithful reconstructions in the under-sampled cases.1 A gridsearch on the parameters: σ, noise and λ were conducted,over the range 3− 3× 102 and 0− 102, respectively.

The second reconstruction benchmark method, called P-LORAKS, is based on low-rank modeling of local k-spaceneighborhoods [60], [61].2 A grid search on the parameters:λ, VCC (Virtual Conjugate Coils) and rank was conducted.The regularization parameter λ was searched between 10−2

and 1. The rank value which is related to the non-convexregularization penalty was searched between 1 to 45 and theVCC, which is a Boolean parameter was searched for bothcases: 1 and 0.

The third benchmark method is Total Generalized Variation(TGV) based reconstruction with Shearlet Transform [62].The method regularizes image regions while keeping theedge information intact.3 We conducted a grid search forthe parameters β, λ, α0, α1 and µ1, adopting a range foreach parameter based on the suggestions from the original

1We used the code at: http://web.itu.edu.tr/eksioglue/pubs/BM3D MRI.htm.2 We used the code available at: https://mr.usc.edu/download/loraks2/.3We used the code at http://www.math.ucla.edu/∼wotaoyin/papers/tgv

shearlet.html.

paper [62]. λ was searched between 10−3 and 5 × 10−1, βwas searched between 102 and 103. The parameters µ1, α0

and α1 were respectively searched between values: 3 × 102

and 10× 3, 8× 10−4 to 10−2 and 10−3 to 10−1.The final benchmark reconstruction method is the residual

U-Net [19], which is also a building block in our model. In thisframework, the U-Net is used as an anti-aliasing neural net-work widely used for biomedical image applications. Unlikein LOUPE, the benchmark U-Net implementation is trainedfor a fixed under-sampling mask, that is provided by the user.In our experiments the U-Net models were individually trainedand tested for each mask configuration.

2) Under-sampling MasksWe implemented several widely used benchmark under-

sampling masks, shown in Figure 2. The first set of maskswe considered are what we refer to as 2D Cartesian under-sampling masks. These are physically feasible for 2D imagingwith (i) two phase-encoding dimensions and no frequency-encoding gradient; or (ii) for 3D acquisition with one fre-quency encoding dimension and two phase-encoding dimen-sions. In this category, we have “Random Uniform” [2],which exhibits the benefits of stochastic under-sampling increating incoherent, noise-like artifacts [1], thus facilitatingthe discrimination of artifacts from the signal present in theimage. We also employed a parametric Variable Density [7](VD) mask that samples lower frequencies more densely,while still enjoying the benefits of creating noise-like artifacts.We used the publicly available code of [27] to identify theoptimal values of the VD mask that maximizes incoherence (ofaliasing artifacts) in the wavelet domain. Another 2D under-sampling mask we implemented was a data-driven strategythat was based on [63]. In our implementation, we averagedthe magnitude spectrum of all the fully-sampled training data.The average spectrum was then thresholded to keep the k-space points that have the largest average magnitude. We referto this mask as “spectrum-based.”

Next, we considered 1D under-sampling masks, whichare physically feasible for 2D imaging with one frequency-encoding dimension and one phase-encoding dimension.Cartesian [8] with skipped lines orthogonal to the providedread-out direction is a popular mask in this category. Inaddition, we implemented a version of LOUPE where themask was constrained to be made up of lines along the read-out direction, which is similar to a strategy proposed in [45].We achieved this with a simple modification in our code bysharing the weights corresponding to the entries of O alongeach read-out line. This “line-constrained version” of LOUPE,we believe, is one step closer to being physically realistic.When we need to be explicit, we refer to the first LOUPEversion as unconstrained.

We underscore that it might not be fair to compare the 1Dand 2D under-sampling masks, as the two categories wouldrely on different acquisition protocols. Thus, any comparisonswe present below need to account for this difference.

D. ResultsFigure 2 shows the LOUPE optimized and benchmark

masks, for two different acceleration rates R = 4 and R = 8.

7

Fig. 2. LOUPE-optimized and benchmark masks for two levels of acceleration rates for NYU fastMRI data set: R = 4 and R = 8. Black dots representk-space points that are sampled and white areas correspond to measurements that are not acquired. The two right-most masks represent 1D under-samplingstrategies, whereas the remaining four represent 2D under-sampling. These two categories of under-sampling rely on different acquisition protocols.

The LOUPE, spectrum-based, and variable density masksshare the behavior of a drop in density from lower to higherfrequencies, however differ significantly in their symmetryand shape. Although the unconstrained LOUPE-optimizedmasks are similar to the VD masks in terms of emphasizinglower frequencies, importantly lateral frequencies are favoredsignificantly more than ventral/dorsal frequencies. The data-driven spectrum-based masks, on the other hand, exhibit asimilar tendency to pick out more lateral frequencies, yet witha dramatically different overall shape that concentrates aroundthe two axes. The line-constrained LOUPE masks also showa strong preference for lower frequencies.

In an earlier conference paper [9], we had reported anunconstrained LOUPE-optimized mask for T1-weighted brainMRI scans, obtained from a public dataset [64]. Figure 3shows a side by side comparison of the two LOUPE-optimizedmasks for the knee and brain anatomies, respectively, em-phasizing the asymmetry present in the knee dataset. Theknee mask favors lateral frequencies more than ventral/dorsalfrequencies due to the unique features of knee anatomy,where there is significantly more tissue contrast in the lateraldirection. Brain scans, on the other hand, exhibit a moreradially symmetric behavior. This comparison highlights theimportance of a data-driven approach in identifying the under-sampling mask.

Figure 4 shows subject-level quantitative reconstructionquality values computed for the different masks and recon-struction methods, under two different acceleration conditions.We present the results for the 6 test subjects with PD weightedimages and 4 test subjects with PDFS weighted images,separately. The fat suppression operation inherently lowers thesignal level, as fat has the highest levels of signal in an MRIscan, thus yielding a noisier image, where small details aremore apparent. In contrast, regular PD weighted scans thatcontain fat tissue have inherently higher SNR. Therefore, thePD scans yield better quality reconstructions compared to thePDFS scans.

Figure 4 reveals that overall LOUPE yields significantlybetter results than competing masks, when used with any of thetested reconstruction methods and under the two accelerationrates. Furthermore, the U-Net reconstruction model, coupled

Fig. 3. LOUPE-optimized under-sampling masks (for R = 8) compared sideby side for the knee and brain anatomies. The brain mask was derived usingthe data described in our prior work [9]. There is a striking difference in thesymmetry, which can be appreciated visually. Note that black dots representk-space points that are sampled and white areas correspond to measurementsthat are not acquired.

with LOUPE-optimized masks yields the best reconstructions,for both acceleration rates and for all test subjects. Unsur-prisingly, the unconstrained version of LOUPE is generallybetter than the line-constrained version, with the exception ofthe TGV reconstruction method, where the line-constrainedLOUPE outperformed unconstrained LOUPE. These resultssuggest that LOUPE-optimized masks can offer a performancegain even when not used with the U-Net based reconstructionmethod. We further observe that the Variable Density andspectrum-based masks outperforms the uniform mask, whichis consistent with prior literature.

Table I lists the summary quantitative values for each of

8

the reconstruction method and mask configurations, whereaverages within the PD and PDFS subjects are presentedseparately. Supplementary Tables 1 and 2 includes standard de-viations for these results. The lowest HFEN and highest PSNRand SSIM values are bolded for each reconstruction method,metric and subject group. We observe that the LOUPE-optimized masks yield the highest reconstruction quality underall considered conditions, and the line-constrained and uncon-strained versions are often quite similar. For reference, wealso added the best values reported in [56] that was achievedwith a U-Net reconstruction model and a Variable DensityCartesian mask. We underscore that their model was trainedon the full training set and evaluated on the full validationset. Nevertheless, we observe that their results are consistentwith ours, often falling between our U-Net models trainedfor Cartesian and VD masks, but always under-performingcompared to the U-Net model trained with LOUPE masks.

Figures 5 and 6 show example PD weighted slices andU-Net based reconstructions, obtained after under-samplingwith R = 4 and R = 8, using different masks. As canbe appreciated from these figures, much of the structuraldetails that correspond to the femur, tibia, tendons, ligaments,muscle tissue, and meniscus are more faithfully capturedin the reconstructions obtained with the LOUPE-optimizedmasks, whereas reconstructions computed with other maskswere more blurry. For example, edges of the posterior cruciateligament and lateral meniscus suffer from blurring artifacts inthe benchmark mask configurations compared to the LOUPE-optimized masks. Overall, in agreement with the quantitativeresults presented above, the LOUPE-optimized mask yields thebest-looking reconstruction results and these reconstructionsappear less prone to artifacts and blurring, while preservingmore of the anatomical detail.

In the Supplementary Material, we present our resultsobtained with a version of LOUPE where the U-Net recon-struction model was replaced with Cascade-Net [51] with thesame architecture details of the reference paper. Everythingelse in LOUPE, including all optimization settings were keptthe same. We observe that the optimized masks are verysimilar between U-Net-LOUPE and Cascade-Net-LOUPE. Wealso quantified the quality of reconstructions with these masksand found that there was no substantial difference. Theseresults, we believe, demonstrate that LOUPE can be relativelyrobust to the choice of the reconstruction architecture. We note,however, that we have not experimented with more advancedreconstruction networks, such as the recently proposed [65],which might yield different results. We consider the explo-ration of different architectural designs for LOUPE as animportant direction for future research.

VI. DISCUSSION

Compressed sensing is a promising approach to accelerateMRI and make the technology more accessible and affordable.Since its introduction over a decade ago, it has gained inpopularity and been used in a range of application domains.Compressed sensing MRI has two fundamental challenges.The first is identifying where in k-space to sample (acquire)

from. One approach to solve this problem is to take a data-driven strategy, where the optimal under-sampling pattern isidentified using, for example, a collection of full-resolutionscans that are retrospectively under-sampled. The second coreproblem of compressed sensing MRI is solving the ill-posedinverse problem of reconstructing a high quality image fromunder-sampled measurements. This is conventionally solvedvia a regularized optimization approach, and more recentlydeep learning techniques have been proposed to achieve supe-rior performance.

In this paper, we tackled both of these problems simultane-ously. We leveraged recently proposed deep learning basedreconstruction techniques to design an end-to-end learningtechnique that enabled us to optimize the under-samplingpattern on given training data. Our approach relies on full-resolution data that is under-sampled retrospectively, which isthen fed to a reconstruction network. The final output is thencompared to the original input to assess overall quality. Weformulate an objective that optimizes reconstruction qualitywith a constrained number of collected measurements. Thestronger the constraint, the more aggressive acceleration rateswe can achieve.

Our results indicate that LOUPE, the proposed method,can compute an optimized under-sampling pattern that isdata-dependent and performs well with different reconstruc-tion methods. In this paper, we present empirical resultsobtained with a large-scale, publicly available knee MRIdataset. Across all conditions we experimented with, LOUPE-optimized masks coupled with the U-Net based reconstructionmodel offered the most superior reconstruction quality. Evenwith an aggressive 8-fold acceleration rate, LOUPE’s recon-structions contained much of the anatomical detail that wasmissed by alternative masks as seen in Figure 6.

In our primary experiments, we used a U-Net based recon-struction method in the implementation of LOUPE. Addition-ally, we investigated the use of an alternative reconstructionnetwork, namely Cascade-Net. Our results, which we presentin the Supplementary Material, indicate that both versions ofLOUPE achieved similar optimized masks and yielded similarreconstruction quality. However, we consider the explorationof different architectural designs (such as ADMM-Net [10] orMoDL [14]) for LOUPE as an important direction for futureresearch.

The LOUPE-optimized under-sampling mask for the ana-lyzed knee MRI scans exhibited an interesting asymmetricpattern, which emphasized higher frequencies along the lateraldirection much more than those along the vertical direction.This is likely due the anisotropy in the field of view ofsagittal knee images. This is in stark contrast to the LOUPE-optimized mask that we recently presented at a conference forT1-weighted brain MRI scans [9]. The brain mask exhibiteda radially symmetric nature, very similar to a variable densitymask, which is widely used in compressed sensing MRI. Thiscomparison highlights the importance of adapting the under-sampling pattern to the data and application at hand.

An important caveat of the presented LOUPE framework isthat we largely ignored the costs of physically implement-ing the under-sampling pattern in an MR pulse sequence.

9

Fig. 4. Quantitative evaluation of reconstruction quality for the NYU fastMRI data set. Top: PSNR values (higher is better), middle: SSIM results(higher is better), bottom: high-frequency error norm (HFEN) results (lower is better). For each plot, we show four reconstruction methods using differentacquisition masks, including LOUPE-optimized masks in green. Note that 2D (first four in each group) and 1D (last two in each group) under-sampling masksshould be interpreted separately and any cross-category comparison should be done with caution. The slice-averaged values for each test subject are connectedwith lines. For each box, the blue straight line shows the median value. Patients with Proton Density (PD) images and Proton Density Fat Suppressed (PDFS)images are shown separately. The whiskers indicate the the most extreme (non-outlier) data points.

10

Fig. 5. U-Net based reconstructions for an example PD slice from NYU fastMRI experiments with 4-fold acceleration rate (α = 0.25). Each columncorresponds to an under-sampling mask. Corresponding PSNR values are listed above each image.

Fig. 6. U-Net based reconstructions for an example PD slice from NYU fastMRI experiments with 8-fold acceleration rate (α = 0.125). Each columncorresponds to an under-sampling mask. Corresponding PSNR values are listed above each image.

In the unconstrained version of LOUPE, the cost of anunder-sampling pattern is merely captured by the numberof collected measurements. We underscore that in 3D, theLOUPE-optimized mask can be implemented as a Cartesiantrajectory, since the isolated measurement points in the coronalslices line up along the z-direction. We also presented resultsfor a readout-line-constrained version of LOUPE, which isconsistent with widely used 2D acquisition protocols. Theseresults, we believe, demonstrate the utility and flexibility ofLOUPE. Nonetheless, integrating the actual physical cost of aspecific under-sampling pattern in an MR pulse sequence is adirection that we leave for future research. One way to achievethis would be to constrain the sub-sampling mask to a classof viable trajectories, for example, by directly parameterizingthose trajectories. Alternatively, one can incorporate samplingconstraints that obey a set of predefined hardware requirements(as defined in terms of, e.g., peak currents and maximum slewrates of magnetic gradients), as described recently in [66].

Our paper restricted its treatment to Cartesian samplingand largely ignored non-Cartesian (off-the-grid) acquisitionschemes, which can offer important benefits [67]–[69]. Ex-tension of LOUPE to non-Cartesian settings will involve theuse of the non-uniform (inverse) Fourier transform operatoror an approximation of it. Furthermore, one would need tointerpolate off-grid measurements in retrospective sampling.

Another weakness of the presented LOUPE implementation

is due to the relaxation of the thresholding operation, whichallowed us to use a back-propagation-based learning strategy.However, due to this relaxation, the reconstruction model seesinput data that have been rescaled with a continuous valuebetween 0 and 1. By adopting a relatively large slope pa-rameter (s = 200), we ensure that these continuous values arealmost always very close to zero or 1. However, this relaxationmeans that once we have obtained the optimized mask, weneed to retrain a reconstruction model with a binary mask.Furthermore, we are likely paying a performance penaltydue to the relaxation, since it will undoubtedly influence theoptimization. We see two ways to address this issue. Thefirst approach is to implement the so-called “straight-through”gradient estimation strategy, where the continuous relaxation ismerely used in the backward pass for computing the gradientand not in the forward pass [70]. Another approach wouldinvolve using non back-propagation based techniques, such asthe REINFORCE estimator [71]. This technique is known tosuffer from high variance and there are several recent papersthat have attempted to address this issue [72], [73]. We planto explore this direction in future work.

In this paper, we used an L2 difference between the mag-nitude of the ground truth and reconstruction images to trainLOUPE. Our experiments with an L2-loss computed on two-channel (complex-valued) reconstructions produced inferiorresults. We also noticed that our LOUPE-optimized masks did

11

not emphasize high frequency content, which can also explainsome of the blurriness in reconstructions. This is particularlynoticeable in the line-constrained LOUPE results. We believethat our choice of the L2 loss is the main cause of thiseffect, which can be remedied with alternate loss functionsthat care more about high-frequency content. We note thatwe experimented with L1 reconstruction loss, as we reportedin our conference paper [9]. Our experiments (results notshown) revealed that L1 loss can yield better quality results,particularly as measured by the high frequency error norm(HFEN). We intend to explore such differences in future work.While these loss functions (L1 or L2) are a widely used globalmetrics of reconstruction quality, they might miss clinicallyimportant anatomical details. An alternative approach couldbe to devise loss functions that emphasize structural featuresthat are relevant to the clinical application. This is anotherfuture direction we will explore.

In the current version of LOUPE, we did not considerparallel imaging via multiple coils. Such hardware-based tech-niques, in combination with compressed sensing, promise toyield even higher degrees of acceleration. Investigating how tocombine LOUPE with multi-coil imaging will be an importantarea of research, as was recently demonstrated [74].

Finally, we would like to emphasize that LOUPE, as inany other data-driven optimization strategy, will perform sub-optimally if test time data differ significantly from trainingdata. For example, if we train only on healthy subjects andthen, at test time, are presented with pathological cases,the reconstructed scans might not be clinically useful. Thus,carefully monitoring reconstruction quality and ensuring themodel is adapted to the new data will be of utmost importancein deploying this approach in the real world. For example,this could be achieved via fine-tuning the pre-trained LOUPEmodel on new data, as recently described in [75]. We alsoconsider machine learning paradigms that don’t rely on highquality full-resolution data as an important direction. Therehave been some recent pre-prints in this domain [26], [76],[77]. Integrating such techniques into LOUPE will be criticalfor certain applications. For instance, in some anatomicalregions such as the abdomen, a fully sampled scenario may notbe possible due to organ motion, breathing, the cardiac cycleor other factors. This is also true for populations such as youngchildren or patient groups, where subject motion can make theacquisition of high quality fully sampled data impossible.

ACKNOWLEDGMENT

This work was, in part, supported by NIH R01 grants(R01LM012719 and R01AG053949, to MRS), the NSF Neu-roNex grant (1707312, to MRS), and an NSF CAREER grant(1748377, to MRS). This work was also in part supported bya Fulbright Scholarship (to CDB).

Data used in the preparation of this article were obtainedfrom the NYU fastMRI Initiative database [56]. As such,NYU fastMRI investigators provided data but did not par-ticipate in analysis or writing of this report. A listing ofNYU fastMRI investigators, subject to updates, can be foundat:fastmri.med.nyu.edu. The primary goal of fastMRI is to

test whether machine learning can aid in the reconstructionof medical images.

The concepts and information presented in this paper arebased on research results that are not commercially available.

Finally, we would like to express our gratitude for theextremely thorough, thoughtful, and constructive feedback wereceived from the Associate Editor Justin Haldar, and threeanonymous reviewers.

REFERENCES

[1] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressedsensing MRI,” IEEE signal processing magazine, vol. 25, no. 2, p. 72,2008.

[2] U. Gamper, P. Boesiger, and S. Kozerke, “Compressed sensing indynamic MRI,” Magnetic Resonance in Medicine, vol. 59, no. 2, pp.365–373, 2008.

[3] S. Ma, W. Yin, Y. Zhang, and A. Chakraborty, “An efficient algorithmfor compressed MR imaging using total variation and wavelets,” in 2008IEEE Conference on Computer Vision and Pattern Recognition. IEEE,2008, pp. 1–8.

[4] X. Qu, Y. Hou, F. Lam, D. Guo, J. Zhong, and Z. Chen, “Magneticresonance image reconstruction from undersampled measurements usinga patch-based nonlocal operator,” Medical image analysis, vol. 18, no. 6,pp. 843–856, 2014.

[5] S. Ravishankar and Y. Bresler, “MR image reconstruction from highlyundersampled k-space data by dictionary learning,” IEEE Transactionson Medical Imaging, vol. 30, no. 5, pp. 1028–1041, 2011.

[6] Z. Zhan, J.-F. Cai, D. Guo, Y. Liu, Z. Chen, and X. Qu, “Fast multiclassdictionaries learning with geometrical directions in MRI reconstruction,”IEEE Transactions on Biomedical Engineering, vol. 63, no. 9, pp. 1850–1861, 2016.

[7] Z. Wang and G. R. Arce, “Variable density compressed image sampling,”IEEE Transactions on Image Processing, vol. 19, no. 1, pp. 264–270,2010.

[8] J. P. Haldar, D. Hernando, and Z.-P. Liang, “Compressed-sensing MRIwith random encoding,” IEEE Transactions on Medical Imaging, vol. 30,no. 4, pp. 893–903, 2011.

[9] C. D. Bahadir, A. V. Dalca, and M. R. Sabuncu, “Learning-basedoptimization of the under-sampling pattern in MRI,” Proc of InformationProcessing in Medical Imaging, 2019.

[10] J. Sun, H. Li, Z. Xu et al., “Deep ADMM-Net for compressive sensingMRI,” in Advances in neural information processing systems, 2016, pp.10–18.

[11] K. C. Tezcan, C. F. Baumgartner, R. Luechinger, K. P. Pruessmann, andE. Konukoglu, “MR image reconstruction using deep density priors,”IEEE transactions on medical imaging, 2018.

[12] G. Wang, J. C. Ye, K. Mueller, and J. A. Fessler, “Image reconstructionis a new frontier of machine learning,” IEEE transactions on medicalimaging, vol. 37, no. 6, pp. 1289–1296, 2018.

[13] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen, “Imagereconstruction by domain-transform manifold learning,” Nature, vol.555, no. 7697, p. 487, 2018.

[14] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Modl: Model-baseddeep learning architecture for inverse problems,” IEEE transactions onmedical imaging, vol. 38, no. 2, pp. 394–405, 2018.

[15] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson,T. Pock, and F. Knoll, “Learning a variational network for reconstructionof accelerated MRI data,” Magnetic resonance in medicine, vol. 79,no. 6, pp. 3055–3071, 2018.

[16] Y. Huang, J. Paisley, Q. Lin, X. Ding, X. Fu, and X.-P. Zhang, “Bayesiannonparametric dictionary learning for compressed sensing MRI,” IEEETransactions on Image Processing, vol. 23, no. 12, pp. 5007–5019, 2014.

[17] Y. Yang, J. Sun, H. Li, and Z. Xu, “ADMM-Net: A deep learning ap-proach for compressive sensing MRI,” arXiv preprint arXiv:1705.06869,2017.

[18] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networksfor biomedical image segmentation,” in International Conference onMedical image computing and computer-assisted intervention. Springer,2015, pp. 234–241.

[19] D. Lee, J. Yoo, and J. C. Ye, “Deep residual learning for compressedsensing MRI,” in 2017 IEEE 14th International Symposium on Biomed-ical Imaging (ISBI 2017). IEEE, 2017, pp. 15–18.

12

[20] C. M. Hyun, H. P. Kim, S. M. Lee, S. Lee, and J. K. Seo, “Deep learningfor undersampled MRI reconstruction,” Physics in Medicine & Biology,vol. 63, no. 13, p. 135007, 2018.

[21] Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan, and A. Veer-araghavan, “Phasecam3dlearning phase masks for passive single viewdepth estimation,” in 2019 IEEE International Conference on Compu-tational Photography (ICCP). IEEE, 2019, pp. 1–12.

[22] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” inAdvances in neural information processing systems, 2014, pp. 2672–2680.

[23] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L. Dragotti, X. Ye, F. Liu,S. Arridge, J. Keegan, Y. Guo et al., “DAGAN: deep de-aliasinggenerative adversarial networks for fast compressed sensing MRI re-construction,” IEEE Transactions on Medical Imaging, vol. 37, no. 6,pp. 1310–1321, 2018.

[24] T. M. Quan, T. Nguyen-Duc, and W.-K. Jeong, “Compressed sensingmri reconstruction using a generative adversarial network with a cyclicloss,” IEEE Transactions on Medical Imaging, vol. 37, no. 6, pp. 1488–1497, 2018.

[25] M. Mardani, E. Gong, J. Y. Cheng, S. Vasanawala, G. Zaharchuk, M. Al-ley, N. Thakur, S. Han, W. Dally, J. M. Pauly et al., “Deep generativeadversarial networks for compressed sensing automates MRI,” arXivpreprint arXiv:1706.00051, 2017.

[26] K. Lei, M. Mardani, J. M. Pauly, and S. S. Vasawanala, “WassersteinGANs for MR imaging: from paired to unpaired training,” arXiv preprintarXiv:1910.07048, 2019.

[27] M. Lustig, D. Donoho, and J. M. Pauly, “Sparse MRI: the applicationof compressed sensing for rapid mr imaging,” Magnetic Resonance inMedicine: An Official Journal of the International Society for MagneticResonance in Medicine, vol. 58, no. 6, pp. 1182–1195, 2007.

[28] M. Elad, “Optimized projections for compressed sensing,” IEEE Trans-actions on Signal Processing, vol. 55, no. 12, pp. 5695–5702, 2007.

[29] J. Xu, Y. Pi, and Z. Cao, “Optimized projection matrix for compressivesensing,” EURASIP Journal on Advances in Signal Processing, vol.2010, no. 1, p. 560349, 2010.

[30] G. Puy, P. Vandergheynst, and Y. Wiaux, “On variable density com-pressive sampling,” IEEE Signal Processing Letters, vol. 18, no. 10, pp.595–598, 2011.

[31] G. Li, Z. Zhu, D. Yang, L. Chang, and H. Bai, “On projection matrixoptimization for compressive sensing systems,” IEEE Transactions onSignal Processing, vol. 61, no. 11, pp. 2887–2898, 2013.

[32] F. Knoll, C. Clason, C. Diwoky, and R. Stollberger, “Adapted randomsampling patterns for accelerated MRI,” Magnetic resonance materialsin physics, biology and medicine, vol. 24, no. 1, pp. 43–50, 2011.

[33] C. Kumar Anand, A. Thomas Curtis, and R. Kumar, “Durga: Aheuristically-optimized data collection strategy for volumetric magneticresonance imaging,” Engineering Optimization, vol. 40, no. 2, pp. 117–136, 2008.

[34] A. T. Curtis and C. K. Anand, “Random volumetric mri trajectories viagenetic algorithms,” Journal of Biomedical Imaging, vol. 2008, p. 6,2008.

[35] M. Seeger, H. Nickisch, R. Pohmann, and B. Scholkopf, “Optimizationof k-space trajectories for compressed sensing by Bayesian experimentaldesign,” Magnetic Resonance in Medicine, vol. 63, no. 1, pp. 116–126,2010.

[36] J. P. Haldar and D. Kim, “OEDIPUS: an experiment design frameworkfor sparsity-constrained MRI,” IEEE Transactions on Medical Imaging,2019.

[37] B. Roman, A. Hansen, and B. Adcock, “On asymptotic structure incompressed sensing,” arXiv preprint arXiv:1406.4178, 2014.

[38] F. Sherry, M. Benning, J. C. D. l. Reyes, M. J. Graves, G. Maierhofer,G. Williams, C.-B. Schonlieb, and M. J. Ehrhardt, “Learning thesampling pattern for mri,” arXiv preprint arXiv:1906.08754, 2019.

[39] S. Ravishankar and Y. Bresler, “Adaptive sampling design for com-pressed sensing MRI,” in 2011 Annual International Conference of theIEEE Engineering in Medicine and Biology Society. IEEE, 2011, pp.3751–3755.

[40] D.-d. Liu, D. Liang, X. Liu, and Y.-t. Zhang, “Under-sampling trajectorydesign for compressed sensing MRI,” in 2012 Annual InternationalConference of the IEEE Engineering in Medicine and Biology Society.IEEE, 2012, pp. 73–76.

[41] L. Baldassarre, Y.-H. Li, J. Scarlett, B. Gozcu, I. Bogunovic, andV. Cevher, “Learning-based compressive subsampling,” IEEE Journalof Selected Topics in Signal Processing, vol. 10, no. 4, pp. 809–822,2016.

[42] B. Gozcu, R. K. Mahabadi, Y.-H. Li, E. Ilıcak, T. Cukur, J. Scarlett,and V. Cevher, “Learning-based compressive MRI,” IEEE Transactionson Medical Imaging, vol. 37, no. 6, pp. 1394–1406, 2018.

[43] T. Sanchez, B. Gozcu, R. B. van Heeswijk, A. Eftekhari, E. Ilıcak,T. Cukur, and V. Cevher, “Scalable learning-based sampling optimiza-tion for compressive dynamic MRI,” in ICASSP 2020-2020 IEEEInternational Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2020, pp. 8584–8588.

[44] F. Zijlstra, M. A. Viergever, and P. R. Seevinck, “Evaluation of variabledensity and data-driven k-space undersampling for compressed sensingmagnetic resonance imaging,” Investigative radiology, vol. 51, no. 6, pp.410–419, 2016.

[45] T. Weiss, S. Vedula, O. Senouf, A. Bronstein, O. Michailovich, andM. Zibulevsky, “Learning fast magnetic resonance imaging,” arXivpreprint arXiv:1905.09324, 2019.

[46] H. K. Aggarwal and M. Jacob, “Joint optimization of samplingpatterns and deep priors for improved parallel mri,” arXiv preprintarXiv:1911.02945, 2019.

[47] I. A. Huijben, B. S. Veeling, and R. J. van Sloun, “Learning samplingand model-based signal recovery for compressed sensing MRI,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP). IEEE, 2020, pp. 8906–8910.

[48] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” arXivpreprint arXiv:1312.6114, 2013.

[49] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization withGumbel-softmax,” arXiv preprint arXiv:1611.01144, 2016.

[50] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution:A continuous relaxation of discrete random variables,” arXiv preprintarXiv:1611.00712, 2016.

[51] J. Schlemper, J. Caballero, J. V. Hajnal, A. Price, and D. Rueckert,“A deep cascade of convolutional neural networks for MR imagereconstruction,” in International Conference on Information Processingin Medical Imaging. Springer, 2017, pp. 647–658.

[52] F. Chollet et al., “Keras,” 2015.[53] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,

S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-scale machine learning,” in 12th {USENIX} Symposium on OperatingSystems Design and Implementation ({OSDI} 16), 2016, pp. 265–283.

[54] A. V. Dalca, J. Guttag, and M. R. Sabuncu, “Anatomical priors inconvolutional networks for unsupervised biomedical segmentation,” inProceedings of the IEEE Conference on Computer Vision and PatternRecognition, 2018, pp. 9290–9299.

[55] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014.

[56] J. Zbontar, F. Knoll, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio,M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana, Z. Zhang,M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, J. Pinkerton, D. Wang,N. Yakubova, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, andY. W. Lui, “FastMRI: An open dataset and benchmarks for acceleratedMRI,” arXiv preprint arXiv:1811.08839, 2018.

[57] S. T. Welstead, Fractal and wavelet image compression techniques.SPIE Optical Engineering Press Bellingham, Washington, 1999.

[58] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli et al., “Imagequality assessment: from error visibility to structural similarity,” IEEETransactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.

[59] E. M. Eksioglu, “Decoupled algorithm for MRI reconstruction usingnonlocal block matching model: BM3D-MRI,” Journal of MathematicalImaging and Vision, vol. 56, no. 3, pp. 430–440, 2016.

[60] J. P. Haldar, “Low-rank modeling of local k-space neighborhoods (LO-RAKS) for constrained MRI,” IEEE Transactions on Medical Imaging,vol. 33, no. 3, pp. 668–681, 2014.

[61] J. P. Haldar and J. Zhuo, “P-LORAKS: Low-rank modeling of local k-space neighborhoods with parallel imaging data,” Magnetic Resonancein Medicine, vol. 75, no. 4, pp. 1499–1514, 2016.

[62] W. Guo, J. Qin, and W. Yin, “A new detail-preserving regularizationscheme,” SIAM Journal on Imaging Sciences, vol. 7, no. 2, pp. 1309–1334, 2014.

[63] J. Vellagoundar and R. R. Machireddy, “A robust adaptive samplingmethod for faster acquisition of MR images,” Magnetic resonanceimaging, vol. 33, no. 5, pp. 635–643, 2015.

[64] A. Di Martino, C.-G. Yan, Q. Li, E. Denio, F. X. Castellanos, K. Alaerts,J. S. Anderson, M. Assaf, S. Y. Bookheimer, M. Dapretto et al., “Theautism brain imaging data exchange: towards a large-scale evaluation ofthe intrinsic brain architecture in autism,” Molecular psychiatry, vol. 19,no. 6, p. 659, 2014.

13

[65] Z. Ramzi, P. Ciuciu, and J.-L. Starck, “Benchmarking MRI reconstruc-tion neural networks on large public datasets,” Applied Sciences, vol. 10,no. 5, p. 1816, 2020.

[66] T. Weiss, O. Senouf, S. Vedula, O. Michailovich, M. Zibulevsky, andA. Bronstein, “Pilot: Physics-informed learned optimal trajectories foraccelerated MRI,” arXiv preprint arXiv:1909.05773, 2019.

[67] S. Zhang, M. Uecker, D. Voit, K.-D. Merboldt, and J. Frahm, “Real-timecardiovascular magnetic resonance at high temporal resolution: radialFLASH with nonlinear inverse reconstruction,” Journal of Cardiovascu-lar Magnetic Resonance, vol. 12, no. 1, p. 39, 2010.

[68] L. Feng, R. Grimm, K. T. Block, H. Chandarana, S. Kim, J. Xu, L. Axel,D. K. Sodickson, and R. Otazo, “Golden-angle radial sparse parallelMRI: combination of compressed sensing, parallel imaging, and golden-angle radial sampling for fast and flexible dynamic volumetric MRI,”Magnetic resonance in medicine, vol. 72, no. 3, pp. 707–717, 2014.

[69] C. Lazarus, P. Weiss, N. Chauffert, F. Mauconduit, L. El Gueddari,C. Destrieux, I. Zemmoura, A. Vignaud, and P. Ciuciu, “SPARKLING:variable-density k-space filling curves for accelerated T2*-weightedMRI,” Magnetic resonance in medicine, vol. 81, no. 6, pp. 3643–3661,2019.

[70] Y. Bengio, N. Leonard, and A. Courville, “Estimating or propagatinggradients through stochastic neurons for conditional computation,” arXivpreprint arXiv:1308.3432, 2013.

[71] R. J. Williams, “Simple statistical gradient-following algorithms forconnectionist reinforcement learning,” Machine learning, vol. 8, no. 3-4,pp. 229–256, 1992.

[72] G. Tucker, A. Mnih, C. J. Maddison, J. Lawson, and J. Sohl-Dickstein,“Rebar: Low-variance, unbiased gradient estimates for discrete latentvariable models,” in Advances in Neural Information Processing Sys-tems, 2017, pp. 2627–2636.

[73] M. Papini, D. Binaghi, G. Canonaco, M. Pirotta, and M. Restelli,“Stochastic variance-reduced policy gradient,” arXiv preprintarXiv:1806.05618, 2018.

[74] B. Gozcu, T. Sanchez, and V. Cevher, “Rethinking sampling in parallelMRI: A data-driven approach,” in 2019 27th European Signal ProcessingConference (EUSIPCO). IEEE, 2019, pp. 1–5.

[75] J. Zhang, Z. Liu, S. Zhang, H. Zhang, P. Spincemaille, T. D.Nguyen, M. R. Sabuncu, and Y. Wang, “Fidelity imposed networkedit (FINE) for solving ill-posed image reconstruction,” arXiv preprintarXiv:1905.07284, 2019.

[76] J. Liu, Y. Sun, C. Eldeniz, W. Gan, H. An, and U. S. Kamilov, “RARE:Image reconstruction using deep priors learned without ground truth,”arXiv preprint arXiv:1912.05854, 2019.

[77] B. Yaman, S. A. H. Hosseini, S. Moeller, J. Ellermann, K. Ugurbil, andM. Akcakaya, “Self-supervised learning of physics-based reconstructionneural networks without fully-sampled reference data,” arXiv preprintarXiv:1912.07669, 2019.

VII. SUPPLEMENTARY MATERIAL

A. Fine-tuning the slope hyperparameters

Figure S3 shows the validation set MAE and MSE forLOUPE with masks optimized using a variety of hyper-parameter values for the two sigmoidal slopes s and t. Theseerrors are relatively stable compared to the variation of resultsamong the baselines in Table I. For example, for MAE, chang-ing s by ten-fold and t by 2-fold results in a loss variation ofless than ±0.02, which is well within the variability of theresults produced by the benchmark masks.

B. Details of the Cascade-Net Implementation

We implemented the identical architecture described in theoriginal Cascade-Net paper. Specifically, for the regularizationlayers of Cascade-Net, a 5-layer model was used with channelsize 64 at each layer. Each layer consists of convolutionfollowed by a ReLU activation function. We used a residuallearning strategy which adds the zero-filled input to the outputof the CNN. The overall architecture is unrolled such thatK=5. In the original paper and in our case, lambda is notlearnable but is a hyperparameter that we set to 0.001, whichwas found by grid search over validation loss and further opti-mized using a Bayesian Optimization hyper-parameter tuningpackage: https://iopscience.iop.org/article/10.1088/1749-4699/8/1/014008/meta.

14

TAB

LE

IQ

UA

NT

ITA

TIV

ER

ES

ULT

SO

FR

EC

ON

ST

RU

CT

ION

QU

AL

ITY

,AV

ER

AG

ED

OV

ER

TE

ST

CA

SE

S.1

DM

AS

KS

AR

ES

EPA

RA

TE

DF

RO

M2D

MA

SK

SA

ND

CO

MPA

RIS

ON

SS

HO

UL

DB

ED

ON

EA

CC

OR

DIN

GLY

.MS

E:M

EA

NS

QU

AR

ED

ER

RO

R.M

AE

:ME

AN

AB

SO

LU

TE

ER

RO

R.H

FE

N:H

IGH

FR

EQ

UE

NC

YE

RR

OR

NO

RM

.PS

RN

:PE

AK

SIG

NA

LT

ON

OIS

ER

AT

IO.S

SIM

:ST

RU

CT

UR

AL

SIM

ILA

RIT

YM

ET

RIC

.

Acc

eler

atio

nM

etho

dM

asks

MSE

PDM

SEPD

FSM

AE

PDM

AE

PDFS

HFE

NPD

HFE

NPD

FSPS

NR

PDPS

NR

PDFS

SSIM

PDSS

IMPD

FS

4-fo

ld

BM

3D

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

0079

60.

0062

90.

0646

0.05

870.

9484

1.05

9621

.122

.10.

590.

63L

OU

PEL

ine

Con

stra

ined

0.00

023

0.00

089

0.01

050.

0234

0.54

80.

7096

36.6

930

.50.

950.

82U

nifo

rmR

ando

m0.

0107

40.

0090

20.

0707

0.06

950.

9467

1.14

1619

.95

20.4

70.

630.

58V

aria

ble

Den

sity

0.00

032

0.00

137

0.01

280.

029

0.57

720.

8288

35.2

28.6

60.

930.

75Sp

ectr

um0.

0002

10.

0013

50.

0107

0.02

870.

537

0.84

6437

.05

28.7

20.

940.

75L

OU

PEU

ncon

stra

ined

0.00

014

0.00

094

0.00

880.

0241

0.29

720.

5298

38.8

130

.28

0.96

0.81

LO

RA

KS

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

2271

70.

2611

30.

1785

0.18

962.

8759

3.50

018.

456.

930.

460.

28L

OU

PEL

ine

Con

stra

ined

0.00

025

0.00

097

0.01

120.

0246

0.58

010.

7668

36.3

230

.14

0.94

0.8

Uni

form

Ran

dom

0.01

692

0.01

861

0.09

130.

1052

1.18

861.

2677

17.9

917

.30.

420.

26V

aria

ble

Den

sity

0.00

041

0.00

231

0.01

520.

0372

0.63

641.

1334

34.1

26.4

0.89

0.64

Spec

trum

0.00

036

0.00

208

0.01

420.

0355

0.66

911.

184

34.6

926

.85

0.91

0.67

LO

UPE

Unc

onst

rain

ed0.

0001

90.

0011

30.

0105

0.02

660.

3754

0.68

5437

.42

29.4

80.

940.

78

TV

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

0078

90.

0058

20.

0643

0.05

640.

8938

0.91

2621

.14

22.4

60.

550.

55L

OU

PEL

ine

Con

stra

ined

0.00

089

0.00

120.

0217

0.02

70.

9029

0.78

2330

.57

29.2

20.

80.

73U

nifo

rmR

ando

m0.

0128

60.

0117

80.

0755

0.08

180.

9303

0.91

8419

.33

19.3

30.

620.

55V

aria

ble

Den

sity

0.00

30.

0033

30.

0417

0.04

541.

2104

1.02

2125

.26

24.7

80.

60.

56Sp

ectr

um0.

0014

40.

0018

80.

0289

0.03

440.

8695

0.84

9228

.48

27.2

70.

710.

62L

OU

PEU

ncon

stra

ined

0.00

196

0.00

218

0.03

320.

0366

0.89

910.

7559

27.1

326

.63

0.71

0.65

U-N

et

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

0040

60.

0034

50.

0381

0.04

010.

8265

0.86

4824

.11

24.6

50.

830.

79L

OU

PEL

ine

Con

stra

ined

0.00

015

0.00

058

0.00

840.

0186

0.35

630.

4567

38.5

332

.36

0.97

0.89

Uni

form

Ran

dom

0.00

220.

0020

40.

0303

0.03

230.

8342

0.76

8926

.89

26.9

30.

870.

85V

aria

ble

Den

sity

0.00

014

0.00

053

0.00

860.

018

0.40

880.

5186

38.7

932

.74

0.97

0.9

Spec

trum

0.00

013

0.00

053

0.00

840.

0179

0.42

660.

5727

39.0

532

.77

0.97

0.9

LO

UPE

Unc

onst

rain

ed0.

0000

90.

0004

80.

0071

0.01

730.

1681

0.24

640

.71

33.1

60.

980.

91U

-Net

NY

UV

aria

ble

Den

sity

Car

tesi

an34

.01

29.9

50.

820.

64

8-fo

ld

BM

3D

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

0087

10.

0067

10.

0665

0.06

030.

9855

1.06

0420

.71

21.8

40.

530.

6L

OU

PEL

ine

Con

stra

ined

0.00

059

0.00

124

0.01

520.

0273

0.78

430.

8844

32.6

629

.08

0.9

0.75

Uni

form

Ran

dom

0.01

248

0.00

854

0.07

590.

0683

1.03

311.

1762

19.5

620

.81

0.59

0.5

Var

iabl

eD

ensi

ty0.

0009

70.

0015

30.

0217

0.03

060.

706

0.91

4730

.45

28.1

60.

870.

73Sp

ectr

um0.

0003

20.

0014

80.

0128

0.03

010.

6758

0.92

7835

.26

28.3

20.

920.

7L

OU

PEU

ncon

stra

ined

0.00

020.

0010

40.

0103

0.02

560.

4684

0.70

2237

.16

29.8

50.

930.

76

LO

RA

KS

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

4987

0.34

232

0.28

750.

2739

4.20

924.

1481

4.05

4.77

0.44

0.31

LO

UPE

Lin

eC

onst

rain

ed0.

0004

60.

0013

80.

0147

0.02

90.

8339

0.99

7933

.56

28.6

30.

90.

73U

nifo

rmR

ando

m1.

1511

80.

8648

30.

3866

0.32

219.

7904

6.52

990.

260.

770.

350.

17V

aria

ble

Den

sity

0.00

041

0.00

188

0.01

520.

0343

0.76

711.

2851

34.1

227

.27

0.89

0.65

Spec

trum

0.00

038

0.00

175

0.01

450.

033

0.75

551.

1352

34.4

527

.59

0.9

0.66

LO

UPE

Unc

onst

rain

ed0.

0002

20.

0011

10.

0111

0.02

660.

5038

0.80

3736

.79

29.5

60.

920.

75

TV

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

0086

0.00

631

0.06

640.

0585

0.95

080.

9606

20.7

622

.11

0.46

0.45

LO

UPE

Lin

eC

onst

rain

ed0.

0011

40.

0015

30.

0245

0.03

021.

1597

1.01

0629

.48

28.1

50.

790.

71U

nifo

rmR

ando

m0.

0139

20.

0109

70.

0792

0.07

850.

975

0.97

2919

.15

19.7

0.6

0.55

Var

iabl

eD

ensi

ty0.

0028

60.

0032

0.04

140.

0445

1.24

871.

076

25.4

824

.95

0.58

0.56

Spec

trum

0.00

199

0.00

240.

034

0.03

881.

0703

0.98

6827

.07

26.2

10.

660.

59L

OU

PEU

ncon

stra

ined

0.00

229

0.00

270.

0364

0.04

071.

2016

1.03

5126

.44

25.6

90.

660.

57

U-N

et

Car

tesi

an(1

D)

Skip

ped

Lin

es0.

007

0.00

509

0.05

170.

0489

1.02

440.

988

21.7

222

.97

0.73

0.68

LO

UPE

Lin

eC

onst

rain

ed0.

0002

70.

0007

90.

0108

0.02

140.

6327

0.73

0136

.031

.01

0.95

0.85

Uni

form

Ran

dom

0.00

316

0.00

279

0.03

630.

0376

0.81

890.

8267

25.1

725

.56

0.84

0.8

Var

iabl

eD

ensi

ty0.

0002

70.

0007

40.

0118

0.02

110.

4746

0.57

5935

.82

31.3

20.

960.

87Sp

ectr

um0.

0002

30.

0007

50.

0105

0.02

10.

5828

0.72

3336

.69

31.2

80.

960.

86L

OU

PEU

ncon

stra

ined

0.00

014

0.00

066

0.00

850.

0199

0.30

440.

4303

38.8

531

.83

0.97

0.87

U-N

etN

YU

31.5

28.7

10.

760.

56

15

Fig. S1. Learned LOUPE masks using NYU fastMRI dataset with 4-fold (α = 0.25) and 8-fold (α = 0.125)acceleration rates. Two versions of LOUPE arecompared. One with U-Net as its reconstruction network (top row), and the other with Cascade-Net (bottom row).

16

Fig. S2. Reconstruction performance of U-Net and Cascade-Net models with different LOUPE optimized masks and acceleration rates on the test knee scans.Masks were computed with the two different versions of LOUPE - one with U-Net as its reconstruction network, and the other with Cascade-Net. We observethat the performance metrics are very close between different masks and reconstruction networks.

17

Fig. S3. Grid search for the slope hyper-parameters s and t, where MAE (L1 loss) and MSE (L2 loss) on the validation data are listed. N/A indicates failureof optimization due to numerical issues.

18

TAB

LE

IIS

UP

PL

EM

EN

TAR

YTA

BL

E,M

EA

NV

AL

UE

S±

STA

ND

AR

DD

EV

IAT

ION

S

Acc

eler

atio

nM

etho

dM

asks

MSE

PDM

SEPD

FSM

AE

PDM

AE

PDFS

HFE

NPD

HFE

NPD

FSPS

NR

PDPS

NR

PDFS

SSIM

PDSS

IMPD

FS

4-fo

ld

BM

3DC

arte

sian

0.00

796±

0.00

176

0.00

629±

0.00

127

0.06

46±

0.00

920.

0587

±0.

0051

0.94

84±

0.00

911.

0596

±0.

0199

21.1

±0.

9922

.1±

0.9

0.59

±0.

030.

63±

0.01

BM

3DL

OU

PEL

ine

Con

stra

ined

0.00

023±

8e-0

50.

0008

9±

7e-0

50.

0105

±0.

0018

0.02

34±

0.00

10.

548±

0.04

210.

7096

±0.

0715

36.6

9±

1.62

30.5

±0.

360.

95±

0.02

0.82

±0.

02B

M3D

Uni

form

Ran

dom

0.01

074±

0.00

350.

0090

2±

0.00

081

0.07

07±

0.01

760.

0695

±0.

0063

0.94

67±

0.07

381.

1416

±0.

1027

19.9

5±

1.56

20.4

7±

0.39

0.63

±0.

060.

58±

0.02

BM

3DV

aria

ble

Den

sity

0.00

032±

0.00

011

0.00

137±

0.00

013

0.01

28±

0.00

230.

029±

0.00

130.

5772

±0.

0449

0.82

88±

0.08

4635

.2±

1.54

28.6

6±

0.41

0.93

±0.

020.

75±

0.02

BM

3DSp

ectr

um0.

0002

1±

9e-0

50.

0013

5±

0.00

012

0.01

07±

0.00

210.

0287

±0.

0013

0.53

7±

0.04

640.

8464

±0.

0831

37.0

5±

1.78

28.7

2±

0.4

0.94

±0.

020.

75±

0.02

BM

3DL

OU

PEU

ncon

stra

ined

0.00

014±

5e-0

50.

0009

4±

8e-0

50.

0088

±0.

0015

0.02

41±

0.00

110.

2972

±0.

049

0.52

98±

0.08

4538

.81±

1.49

30.2

8±

0.39

0.96

±0.

010.

81±

0.02

LO

RA

KS

Car

tesi

an0.

2271

7±

0.25

121

0.26

113±

0.13

396

0.17

85±

0.10

430.

1896

±0.

0521

2.87

59±

1.09

523.

5001

±0.

9479

8.45

±3.

866.

93±

3.64

0.46

±0.

070.

28±

0.06

LO

RA

KS

LO

UPE

Lin

eC

onst

rain

ed0.

0002

5±

7e-0

50.

0009

7±

0.00

010.

0112

±0.

0018

0.02

46±

0.00

130.

5801

±0.

0295

0.76

68±

0.07

8836

.32±

1.41

30.1

4±

0.43

0.94

±0.

020.

8±

0.02

LO

RA

KS

Uni

form

Ran

dom

0.01

692±

0.00

581

0.01

861±

0.00

020.

0913

±0.

0231

0.10

52±

0.00

621.

1886

±0.

0497

1.26

77±

0.04

3417

.99±

1.57

17.3

±0.

050.

42±

0.08

0.26

±0.

02L

OR

AK

SV

aria

ble

Den

sity

0.00

041±

0.00

014

0.00

231±

0.00

028

0.01

52±

0.00

250.

0372

±0.

0024

0.63

64±

0.08

811.

1334

±0.

1639

34.1

±1.

4526

.4±

0.57

0.89

±0.

040.

64±

0.04

LO

RA

KS

Spec

trum

0.00

036±

0.00

013

0.00

208±

0.00

021

0.01

42±

0.00

240.

0355

±0.

0017

0.66

91±

0.07

431.

184±

0.10

9434

.69±

1.45

26.8

5±

0.43

0.91

±0.

030.

67±

0.03

LO

RA

KS

LO

UPE

Unc

onst

rain

ed0.

0001

9±

6e-0

50.

0011

3±

9e-0

50.

0105

±0.

0017

0.02

66±

0.00

110.

3754

±0.

0745

0.68

54±

0.08

5337

.42±

1.37

29.4

8±

0.35

0.94

±0.

020.

78±

0.02

TV

Car

tesi

an0.

0078

9±

0.00

176

0.00

582±

0.00

129

0.06

43±

0.00

920.

0564

±0.

0055

0.89

38±

0.00

280.

9126

±0.

0141

21.1

4±

1.01

22.4

6±

0.99

0.55

±0.

030.

55±

0.03

TV

LO

UPE

Lin

eC

onst

rain

ed0.

0008

9±

0.00

012

0.00

12±

2e-0

50.

0217

±0.

0019

0.02

7±

0.00

020.

9029

±0.

1649

0.78

23±

0.06

3630

.57±

0.6

29.2

2±

0.07

0.8±

0.03

0.73

±0.

02T

VU

nifo

rmR

ando

m0.

0128

6±

0.00

505

0.01

178±

0.00

170.

0755

±0.

0229

0.08

18±

0.01

0.93

03±

0.01

110.

9184

±0.

0134

19.3

3±

2.01

19.3

3±

0.61

0.62

±0.

080.

55±

0.01

TV

Var

iabl

eD

ensi

ty0.

003±

0.00

037

0.00

333±

0.00

013

0.04

17±

0.00

410.

0454

±0.

0012

1.21

04±

0.28

331.

0221

±0.

0692

25.2

6±

0.54

24.7

8±

0.18

0.6±

0.04

0.56

±0.

06T

VSp

ectr

um0.

0014

4±

0.00

025

0.00

188±

6e-0

50.

0289

±0.

0032

0.03

44±

0.00

070.

8695

±0.

1871

0.84

92±

0.06

6328

.48±

0.72

27.2

7±

0.14

0.71

±0.

060.

62±

0.02

TV

LO

UPE

Unc

onst

rain

ed0.

0019

6±

0.00

032

0.00

218±

0.00

011

0.03

32±

0.00

370.

0366

±0.

0011

0.89

91±

0.26

340.

7559

±0.

0998

27.1

3±

0.75

26.6

3±

0.22

0.71

±0.

030.

65±

0.02

U-N

etC

arte

sian

0.00

406±

0.00

112

0.00

345±

0.00

038

0.03

81±

0.00

810.

0401

±0.

0018

0.82

65±

0.03

950.

8648

±0.

0182

24.1

1±

1.37

24.6

5±

0.47

0.83

±0.

040.

79±

0.01

U-N

etL

OU

PEL

ine

Con

stra

ined

0.00

015±

5e-0

50.

0005

8±

3e-0

50.

0084

±0.

0014

0.01

86±

0.00

040.

3563

±0.

0148

0.45

67±

0.01

838

.53±

1.52

32.3

6±

0.19

0.97

±0.

010.

89±

0.0

U-N

etU

nifo

rmR

ando

m0.

0022

±0.

0007

80.

0020

4±

0.00

017

0.03

03±

0.00

780.

0323

±0.

0009

0.83

42±

0.09

740.

7689

±0.

0149

26.8

9±

1.7

26.9

3±

0.35

0.87

±0.

030.

85±

0.01

U-N

etV

aria

ble

Den

sity

0.00

014±

4e-0

50.

0005

3±

3e-0

50.

0086

±0.

0015

0.01

8±

0.00

050.

4088

±0.

0231

0.51

86±

0.01

5838

.79±

1.5

32.7

4±

0.21

0.97

±0.

010.

9±

0.0

U-N

etSp

ectr

um0.

0001

3±

4e-0

50.

0005

3±

3e-0

50.

0084

±0.

0014

0.01

79±

0.00

050.

4266

±0.

020.

5727

±0.

019

39.0

5±

1.52

32.7

7±

0.22

0.97

±0.

010.

9±

0.0

U-N

etL

OU

PEU

ncon

stra

ined

9e-0

5±

3e-0

50.

0004

8±

1e-0

50.

0071

±0.

0011

0.01

73±

0.00

020.

1681

±0.

010.

246±

0.01

6240

.71±

1.44

33.1

6±

0.09

0.98

±0.

010.

91±

0.0

8-fo

ld

BM

3DC

arte

sian

0.00

871±

0.00

189

0.00

671±

0.00

146

0.06

65±

0.00

940.

0603

±0.

0054

0.98

55±

0.00

691.

0604

±0.

012

20.7

1±

0.99

21.8

4±

0.95

0.53

±0.

020.

6±

0.02

BM

3DL

OU

PEL

ine

Con

stra

ined

0.00

059±

0.00

024

0.00

124±

0.00

010.

0152

±0.

0029

0.02

73±

0.00

120.

7843

±0.

0395

0.88

44±

0.05

5932

.66±

1.7

29.0

8±

0.36

0.9±

0.02

0.75

±0.

02B

M3D

Uni

form

Ran

dom

0.01

248±

0.00

567

0.00

854±

0.00

211

0.07

59±

0.02

40.

0683

±0.

0112

1.03

31±

0.07

441.

1762

±0.

0649

19.5

6±

2.22

20.8

1±

0.98

0.59

±0.

090.

5±

0.04

BM

3DV

aria

ble

Den

sity

0.00

097±

0.00

038

0.00

153±

0.00

012

0.02

17±

0.00

450.

0306

±0.

0011

0.70

6±

0.06

390.

9147

±0.

0872

30.4

5±

1.6

28.1

6±

0.34

0.87

±0.

030.

73±

0.03

BM

3DSp

ectr

um0.

0003

2±

0.00

012

0.00

148±

0.00

012

0.01

28±

0.00

230.

0301

±0.

0013

0.67

58±

0.04

630.

9278

±0.

071

35.2

6±

1.65

28.3

2±

0.36

0.92

±0.

020.

7±

0.03

BM

3DL

OU

PEU

ncon

stra

ined

0.00

02±

7e-0

50.

0010

4±

9e-0

50.

0103

±0.

0017

0.02

56±

0.00

120.

4684

±0.

0444

0.70

22±

0.08

3537

.16±

1.52

29.8

5±

0.39

0.93

±0.

020.

76±

0.03

LO

RA

KS

Car

tesi

an0.

4987

±0.

3226

80.

3423

2±

0.07

458

0.28

75±

0.08

710.

2739

±0.

0318

4.20

92±

1.07

14.

1481

±0.

9071

4.05

±3.

224.

77±

0.99

0.44

±0.

060.

31±

0.03

LO

RA

KS

LO

UPE

Lin

eC

onst

rain

ed0.

0004

6±

0.00

013

0.00

138±

0.00

011

0.01

47±

0.00

20.

029±

0.00

130.

8339

±0.

0421

0.99

79±

0.07

0133

.56±

1.27

28.6

3±

0.36

0.9±

0.02

0.73

±0.

02L

OR

AK

SU

nifo

rmR

ando

m1.

1511

8±

0.80

465

0.86

483±

0.21

083

0.38

66±

0.15

350.

3221

±0.

0275

9.79

04±

3.02

26.

5299

±0.

6076

0.26

±2.

620.

77±

1.13

0.35

±0.

070.

17±

0.03

LO

RA

KS

Var

iabl

eD

ensi

ty0.

0004

1±

0.00

013

0.00

188±

0.00

014

0.01

52±

0.00

250.

0343

±0.

0014

0.76

71±

0.07

311.

2851

±0.

1822

34.1

2±

1.5

27.2

7±

0.34

0.89

±0.

030.

65±

0.04

LO

RA

KS

Spec

trum

0.00

038±

0.00

011

0.00

175±

0.00

015

0.01

45±

0.00

230.

033±

0.00

150.

7555

±0.

0334

1.13

52±

0.09

0734

.45±

1.36

27.5

9±

0.37

0.9±

0.03

0.66

±0.

03L

OR

AK

SL

OU

PEU

ncon

stra

ined

0.00

022±

7e-0

50.

0011

1±

0.00

011

0.01

11±

0.00

180.

0266

±0.

0014

0.50

38±

0.05

40.

8037

±0.

0823

36.7

9±

1.41

29.5

6±

0.45

0.92

±0.

020.

75±

0.02

TV

Car

tesi

an0.

0086

±0.

0018

90.

0063

1±

0.00

143

0.06

64±

0.00

940.

0585

±0.

0055

0.95

08±

0.00

160.

9606

±0.

008

20.7

6±

1.0

22.1

1±

0.99

0.46

±0.

040.

45±

0.06

TV

LO

UPE

Lin

eC

onst

rain

ed0.

0011

4±

0.00

018

0.00

153±

5e-0

50.

0245

±0.

0021

0.03

02±

0.00

041.

1597

±0.

244

1.01

06±

0.05

829

.48±

0.67

28.1

5±

0.15

0.79

±0.

030.

71±

0.02

TV

Uni

form

Ran

dom

0.01

392±

0.00

634

0.01

097±

0.00

251

0.07

92±

0.02

740.

0785

±0.

0128

0.97

5±

0.00

790.

9729

±0.

0058

19.1

5±

2.41

19.7

±0.

910.

6±

0.08

0.55

±0.

03T

VV

aria

ble

Den

sity

0.00

286±

0.00

043

0.00

32±

0.00

012

0.04

14±

0.00

430.

0445

±0.

0011

1.24

87±

0.30

411.

076±

0.06

9725

.48±

0.69

24.9

5±

0.17

0.58

±0.

050.

56±

0.07

TV

Spec

trum

0.00

199±

0.00

032

0.00

24±

9e-0

50.

034±

0.00

360.

0388

±0.

0008

1.07

03±

0.19

890.

9868

±0.

0546

27.0

7±

0.73

26.2

1±

0.17

0.66

±0.

040.

59±

0.07

TV

LO

UPE

Unc

onst

rain

ed0.

0022

9±

0.00

031

0.00

27±

0.00

012

0.03

64±

0.00

360.

0407

±0.

0011

1.20

16±

0.37

561.

0351

±0.

1059

26.4

4±

0.61

25.6

9±

0.19

0.66

±0.

040.

57±

0.01

U-N

etC

arte

sian

0.00

7±

0.00

191

0.00

509±

0.00

075

0.05

17±

0.01

010.

0489

±0.

0023

1.02

44±

0.02

830.

988±

0.00

4121

.72±

1.26

22.9

7±

0.6

0.73

±0.

020.

68±

0.02

U-N

etL

OU

PEL

ine

Con

stra

ined

0.00

027±

9e-0

50.

0007

9±

6e-0

50.

0108

±0.

0019

0.02

14±

0.00

070.

6327

±0.

0499

0.73

01±

0.02

2536

.0±

1.53

31.0

1±

0.3

0.95

±0.

010.

85±

0.01

U-N

etU

nifo

rmR

ando

m0.

0031

6±

0.00

089

0.00

279±

0.00

026

0.03

63±

0.00

810.

0376

±0.

0022

0.81

89±

0.04

720.

8267

±0.

0133

25.1

7±

1.24

25.5

6±

0.39

0.84

±0.

030.

8±

0.02

U-N

etV

aria

ble

Den

sity

0.00

027±

8e-0

50.

0007

4±

4e-0

50.

0118

±0.

0019

0.02

11±

0.00

060.

4746

±0.

0268

0.57

59±

0.01

9335

.82±

1.31

31.3

2±

0.24

0.96

±0.

010.

87±

0.0

U-N

etSp

ectr

um0.

0002

3±

8e-0

50.

0007

5±

5e-0

50.

0105

±0.

0019

0.02

1±

0.00

070.

5828

±0.

0394

0.72

33±

0.01

9936

.69±

1.59

31.2

8±

0.29

0.96

±0.

010.

86±

0.01

U-N

etL

OU

PEU

ncon

stra

ined

0.00

014±

5e-0

50.

0006

6±

2e-0

50.

0085

±0.

0014

0.01

99±

0.00

030.

3044

±0.

0175

0.43

03±

0.02

2938

.85±

1.59

31.8

3±

0.15

0.97

±0.

010.

87±

0.0