the computer journal 2009 greenspan 43 63

7/29/2019 The Computer Journal 2009 Greenspan 43 63

1/21

Super-Resolution in Medical Imaging

HAYIT GREENSPAN*

Faculty of Engineering, Biomedical Engineering Department, Tel-Aviv University, Tel-Aviv, Israel

*Corresponding author: [email protected]

This paper provides an overview on super-resolution (SR) research in medical imaging

applications. Many imaging modalities exist. Some provide anatomical information and reveal

information about the structure of the human body, and others provide functional information,

locations of activity for specific activities and specified tasks. Each imaging system has a character-

istic resolution, which is determined based on physical constraints of the system detectors that are in

turn tuned to signal-to-noise and timing considerations. A common goal across systems is to

increase the resolution, and as much as possible achieve true isotropic 3-D imaging. SR technology

can serve to advance this goal. Research on SR in key medical imaging modalities, including MRI,

fMRI and PET, has started to emerge in recent years and is reviewed herein. The algorithms used

are mostly based on standard SR algorithms. Results demonstrate the potential in introducing SR

techniques into practical medical applications.

Keywords: medical imaging; super-resolution; MRI; PET

Received 6 August 2006; revised 6 June 2007

1. INTRODUCTION

The main goal of medical imaging is to extract a 3-D modeling

of the human body or specific organs within it. To accomplish

this goal, various imaging modalities have been developed

over the years, each based on a particular energy source that

passes through the body. Many imaging modalities exist,

with some providing anatomical information and revealinginformation about the structure, and others providing func-

tional information, e.g. revealing locations of activity within

the brain for specific activities and specified tasks. The

medical imaging field is rapidly evolving in increased resol-

ution machines and advanced content-processing tools [1].

Medical imaging system developers strive to increase resolu-

tion since higher resolution is the key to more accurate under-

standing of the anatomy, it can support early detection of

abnormalities and can increase the accuracy in the assessment

of size and morphology of organs and pathologies.

In recent years, several research groups have started to

address the goal of resolution augmentation in medical

imagery as a software post-processing challenge, rather than

a medical hardware-engineering task. The motivation for

this initiative emerged following major advances in the

domains of image and video processing that indicated the

possibility of augmenting resolution using what are known

as super-resolution (SR) algorithms. SR deals with the task

of using several low-resolution (LR) images from a particular

imaging system to estimate, or reconstruct, the high-resolution

(HR) source. Each LR input image focuses on a slightly

shifted field-of-view [or point-of-view (POV)] of the HR

scene. A variety of reconstruction algorithms have been pro-

posed in the literature, where the common goal is to estimate

the HR source as accurately as possible, while minimizing

noise and preserving important image constraints, including

image smoothness and more recently, additional prior-

knowledge about the source.In 2001 and 2002, initial attempts were made to adapt SR

algorithms from the computer-vision community to medical

imagery applications. Initial research dealt with the magnetic

resonance imaging (MRI) modality [2, 3]. Results were

encouraging and were reproduced around the world within

the same modalities as well as additional ones, including func-

tional MRI (fMRI) [4] and positron emission tomography

(PET) [5, 6]. The goal of the current paper is to review

several of the key studies that focus on SR algorithms in

medical applications.

Following are the key observations that may be made from

the current overview.

Research in the field has so far been in what can be defined

as a hypothesis testing phase: the investigation of a selected

imaging modality and its evaluation as a candidate for SR,

and the application of a standard SR algorithm (selected

from the literature) to a specific medical task.

Results are encouraging, for the MRI variants, and PET

modalities.

Experiments are conducted on phantoms as well as real

patient data. Results are more substantial on the phantom

THE COMPUTER JOURNAL, Vol. 52 No. 1, 2009

# The Author 2008. Published by Oxford University Press on behalf of The British Computer Society. All rights reserved.For Permissions, please email: [email protected]

Advance Access publication on February 19, 2008 doi:10.1093/comjnl/bxm075


2/21

data. Real data scenarios introduce various artifacts that need

to be addressed, patient motion difficulties as well as acqui-

sition time constraints.

A comparison across several currently published SR var-

iants algorithmic variants does not reveal major differences.

Future major challenges in this field include: on the theor-

etical front, the development of novel SR algorithms thatcombine medical image prior knowledge as regularization

terms in the SR process. On the applicative front, the shift to

the clinical settings, an important next step to define the con-

tribution of SR in the medical field.

In the rest of this introduction we introduce the non-medical

reader to the variety of medical imaging modalities as well as

introduce resolution-related challenges in the medical imaging

field. A brief overview of the SR algorithms, SR parameteriza-

tion, and evaluation schemes are given in Section 2. Section 3

focuses on the application of SR in MRI, and Section 4

describes major works on the application of SR in PET. Key

issues and an overview of the outstanding challenges in this

domain finalize this overview paper, in Section 5.

1.1. Medical imaging acquisition process and clinical

significance

Figure 1 illustrates the general medical imagery acquisition

process. Energy is acquired as it transverses the body (or

organ) as part of a transmission or an emission process. For

example, in X-ray computer tomography (CT), X-rays are

transmitted through the body and captured by an array of

detectors, following attenuation by the imaged object. In

PET, photons are emitted from within the body (from radio-

active molecules inserted into the body) and are then detected

by an array of detectors. The energy is captured by an array of

detectors that are designed per specific imaging modality [e.g.

radio frequency (RF) coils in MRI, crystal detectors in PET].

Figure 2 presents a schematic description of an X-ray trans-

mission (a) and a photon emission process (b). Following

the energy acquisition process, various algorithmic transform-

ations are used, such as the inverse Radon transform (e.g. in

CT), to reconstruct a visual image. The target of a successful

acquisition process is a high spatial-resolution visual image of

the organ of interest. Additional characteristics of interest

include a high-contrast and a high dynamic-range as well as

a strong signal-to-noise ratio (SNR) output from the imagingsystem.

A complete physical and signal-processing review of the

various medical imaging modalities is beyond the scope of

this paper (several books can be found, including [710]).

Below, we briefly introduce several key imaging modalities

and present representative output images. Additional detail

for each modality is given in Appendix A.

MRI is an imaging technique used primarily in medical set-

tings to produce high-quality images of the human body. It is

based on the absorption and emission of energy in the

RF range of the electromagnetic spectrum, producing images

based on spatial variations in the frequency of the RF

energy being absorbed and emitted by the imaged object.A sample MRI scanner and an MR reconstructed image of

the human head are shown in Fig. 3a and b, respectively.

A set of MRI brain scans, from three standard MR imaging

sequences, termed T1, T2 and Proton-Density (Pd) (see

Appendix A), are shown in Fig. 4. We note that each such

sequence displays a unique image of the imaged organ

(brain). A standard coordinate system was developed to rep-

resent three (2-D) slice directions, as displayed in the figure.

The defined slice directions will be used throughout the

experiments reviewed in this paper.

In addition to the standard, anatomical MR imaging, several

variants exist, including the following: MRI Angiography

deals with the imaging of the flowing blood in the arteries

and veins of the body, with intensity proportional to the velo-

city of the flow. MRI Angiography can be used to evaluate

abnormal narrowing of the blood vessels (stenosis) and their

risk of rapture (aneurysms). Diffusion-weighted imaging

(DWI) is an MRI modality that produces the in vivo MR

images of biological tissues weighted with the local character-

istics of water diffusion. DWIs are very useful in diagnosing

vascular strokes in the brain and to study white matter dis-

eases. Diffusion tensor imaging (DTI) is a variation of DWI

in which at least seven images are acquired for every slice,

with at least six directions of diffusion weighting. DTI

serves as a unique tool for visualization of the direction and

intactness of white matter fiber tracts in vivo by identifying

the preferred direction of diffusion. HR DTI combined with

algorithms for tracing fibers in three dimensions in tensor

fields has the potential to enable fiber tract mapping of critical

functional pathways in the brain. The clinical applications are

the tract-specific localization of white matter lesions, the

localization of tumors relative to the white matter tracts

and the localization of the main white matter tracts for neuro-

surgical planning. fMRI measures the changes in blood flowFIGURE 1. Illustration of the general medical imagery acquisition

process.

44 H. GREENSPAN



3/21

and blood oxygenation in the brain (hemodynamics), which iscorrelated to neural activity in the brain or spinal cord of

humans or other animals. The magnetic resonance (MR)

signal of blood is slightly different depending on the level of

oxygenation. These differential signals can be detected using

an appropriate MR pulse sequence.

Figure 3 presents examples of an angiographic image (c), a

DTI image (d) and an fMRI image (e).

X-ray CT is based on the fact that X-rays can traverse a cross-

section of an object along straight lines, be attenuated by the

object, and detected outside it (Fig. 2a). Since its introduction

in the 1970s, CT has become an important tool in medical

imaging in the diagnosis of a large number of differentdisease entities. CT is currently a standard diagnosis tool in

several domains, including: head imaging in particular for diag-

nosis of cerebrovascular accidents and intracranial hemorrhage;

facial and skull fractures evaluation; surgical planning for cra-

niofacial and dentofacial deformities; detecting acute and

chronic chest diseases; imaging of coronary arteries (cardiac

CT angiography); abdominal diseases; imaging complex frac-

tures especially around joints, and more. Recently, CT is also

being considered for preventive medicine or screening for

disease, for example CT colonography for patients with a

high risk of colon cancer. The CT scanner can image complete

organs and volumes using a large series of two-dimensional

x-ray images taken around a single axis of rotation. Over

recent years, a transition has been made from slice-by-slice

imaging to volume imaging, with the introduction of spiral

scan modes. An example CT image is shown in Fig. 6a.

PET belongs to the radiology specialty of nuclear medicine,

and it provides information on the distribution of a chosen

molecule inside the human body (Fig. 2b). PET provides func-

tional information that is complimentary to the anatomical

information from other radiological imaging techniques such

as the MRI and CT. In particular, PET is emerging as an

important tool to detect tumors and to evaluate their degree

of malignancy, based on differences in biochemistry and

metabolism between tumors and their surrounding normal

tissues [11]. PET is also used for functional brain imaging

and in cardiology, to assess myocardial viability and efficiency

[1215]. PET images are commonly fused with anatomical

images such as CT. The need to combine PET and CT has

evolved into specialized hardware that makes the task of

fusing the two modalities much easier. A combined PET/CT

machine is shown in Fig. 5. An example of PET image is

shown in Fig. 6b and a combined PET/CT image is shown

in Fig. 6c. For an overview on PET, see [16].

FIGURE 2. Schematic description of x-ray transmission and PET acquisitions. (a) X-ray CT acquisition via X-ray transmission. (b) Schematics

of PET acquisition via photon emission. The PET camera is comprised of a ring of discrete detectors. Shown on the left is a pair of opposing PET

detectors. Each such detector pair detects coincident photon pair emission.

FIGURE 3. The MRI modality. (a) MRI scanner (GE medical

systems). (b) Anatomical image of a head. T1-weighted. (c) Angio-

graphy. (d) DTI. 150 gradient directions. (e) fMRI. Activated areas

overlayed on the anatomical image (for color see Figure 12).

SR IN MEDICAL IMAGING 45



4/21

FIGURE 4. Brain slices in varying MRI sequences (taken from BrainWeb [17]). Transverse slices are in the xy plane. Sagittal slices are in the zy

plane. Coronal slices are in the zx plane.

FIGURE 5. Combined CT/PET scanners. (a) Illustration. (b) Scanner (GE medical systems).

FIGURE 6. Example brain scans. (a) CT image. (b) PET image. (c) PET and CT images fused together.

46 H. GREENSPAN



5/21

1.2. Resolution limitations and challenges in medical

imaging

A common goal in all medical imaging systems is to increase

the resolution and, to the extent possible, achieve true isotro-

pic 3-D imaging. To capture the entire frequency content of

the imaged object, a sampling rate at the Nyquist frequency

is required, defined as twice the highest frequency present inthe imaged object. Correspondingly, the sampling distance

must be one-half the spatial resolution, defined as the distance

between half-value points of the system impulse response, or

the full-width-at-half-maximum (FWHM).

In practice, the range of frequencies captured is limited by

the maximal sampling frequency of the imaging device detec-

tors, as defined by the detector pitch, or the detector spacing.

Reduced size (width) of detectors and smaller inter-detector

distances can provide increased resolution, but this is at the

cost of an increase in the noise, thus results in a much-reduced

SNR. The sensor resolution of a general imaging system is

determined according to the physical constraints of the detec-

tors, which are in turn tuned to SNR and timing considerations

in the system. Within each imaging modality, specific physical

laws are in control, defining the meaning of noise and the sen-

sitivity of the imaging process. Signal processing rules govern

the system design in an attempt to achieve an acceptable com-

promise between resolution and SNR. The resolution limit-

ations in MRI and PET will be defined in depth in Sections

3 and 4, respectively.

A resolution-related challenge in medical image processing

is known as partial volume effect (PVE), which arises when an

interface between two different tissues occur within a single

voxel. The PVE is a direct consequence of limited resolution

during the acquisition process. In general, PVE blurs theboundary between tissues and adds complexity to tissue

characterizations. In MRI, the resulting image pixels display

a gray level proportional to the weighted average of the

signals stemming from neighboring tissues. The exact location

of boundaries may be shifted thus introducing a major obstacle

for anatomical MRI brain segmentation. In a CT image, each

voxel represents the attenuation properties of a specific

volume. When more than a single tissue is present within

the voxel, the value will be some (nonlinear) average of the

tissues attenuation properties. In DTI, where isotropic resolu-

tion is particularly important, PVEs are a limiting factor in the

analysis of directional and structural axonal connectivity.

Increased resolution can help overcome or reduce the pro-blems associated with PVE.

1.3. Can SR support the medical imaging challenges?

SR reconstruction deals with combining several LR images to

create a HR image. SR techniques have been suggested in

recent years as a means for increasing resolution without alter-

ing the existing imaging hardware. Thus, they can be seen as a

means for extending current medical imaging resolution

limitations. The goal of SR algorithms is to improve the

image resolution in cases in which the image was under-

sampled. Such cases involve the following: first, the imaged

object has high-frequency content. Second, the sampling fre-

quency as defined by the detectors, does not fulfill the

Nyquist frequency; thus, aliasing and degradation in the high-frequency content can be observed. The SR process helps

overcome the detector sampling limitations by practically

increasing the sampling rate, and thus utilizing additional

high-frequency information and reducing the aliasing

effects. Note that in cases in which no frequencies higher

than half of the detectors sampling frequency exist, SR will

in effect result in the averaging of noise; in such cases, no

additional improvements in the image resolution can be

obtained by SR.

2. SR ALGORITHMS: BRIEF OVERVIEW

The term SR refers here to a technique in which several LR

images, from different POV relative to the image object, are

combined to obtain a higher-resolution image. Several SR

reconstruction methodologies have been developed in the

last two decades [18]. In initial works [e.g., 19], the frequency

domain was used to demonstrate the ability to reconstruct one

improved resolution image from several down-sampled noise-

free versions of it, based on the spatial aliasing effect. The fre-

quency domain approach was further generalized to noisy and

blurred images in [20] and a spatial domain alternative was

suggested in [21]. Further non-iterative spatial domain data

fusion approaches were proposed in [22, 23].

An iterative back-projection (IBP) method was proposed in

[24]. This method starts with an initial guess of the outcome

image, projects the initial result to simulate the LR measure-

ments, and updates the temporary guess according to the simu-

lation error. Further detail on the IBP method is provided in

the following subsections.

A set theoretic approach to SR was suggested in [25]. Here,

convex sets are defined which represent tight constraints on

the required image. Nonlinear constraints are combined

within the restoration process and a projection onto convex

sets (POCS) algorithm is utilized. A hybrid model that com-

bines maximum-likelihood (ML) and POCS was suggested

in [26]. More recent SR works aim at combining the SR

approaches with regularization terms, e.g. in [27] fast and

robust multi-frame SR is proposed using L1 norm minimiz-

ation and robust regularization based on a bilateral prior to

deal with different data and noise models.

In this section, we mathematically define the general image

acquisition procedure, and present the general formalism for

SR. It is our goal to provide the reader with a brief overview

of specific algorithms and related parameterization issues, as

related to the research works reviewed herein.




6/21

2.1. Modeling the image acquisition process

Given a set fykgk1N of LR images and assuming all LR images

are degraded versions of the same original HR image, x, SR

algorithms reconstruct a super-resolved HR image which by

simulating the imaging process, results in images that best

describe the LR measurements via some SR criterion.

Mathematically, we can express each LR image yk as theresult of a sequence of operators on the original HR image

source, x, consisting of geometrical warp, blurring and deci-

mation, as in equation (1),

yk fkxh # s vk; k f1; . . . ;Ng; 1

where fk is the kth geometrical transformation of the image x to

the same reference frame of acquisition for yk, h is a blur

kerneloften referred to as the point spread function (PSF),

defined by the properties of the lens and the imaging device,

and vk is an additive noise. The symbol (*) is the convolution

operator and #s represents the down sampling of an HR image

to a LR grid by a factor s.

2.2. The general formalism of SR

We present next the general formalism for SR, as relevant both

for the IBP approach of Irani and Peleg [24], as well as to the

more recent, ML formalism of [27]. An initial estimate of the

HR image, x(0), is taken as the average of the set of LR acqui-

sitions brought to the same reference point and up-sampled:

x0 1

kX

K

k1

f1k yk " s; 2

where yk is one of K acquisitions, fk the geometric transform-

ation to a common reference frame and "s the up-sampling

operator from LR to the HR representation.

Given an image-acquisition model, and an HR estimate (of the

nth iteration), x(n), a set of synthetically generated LR images,

fyk(n)g, can be extracted. The process involves shifting the HR

image to the kth POV, blurring to account for the psf and down-

sampling to the systems sampling rate. The nth LR synthetically

sampled set of images fyk(n)g is thus obtained from the nth

approximation of the HR image x(n) (ignoring the noise), by:

~ynk fkx

nh # s: 3

The current iteration x(n) is updated according to the differ-

ence between the synthetically generated set of LR images

fyk(n)g and the actual acquired set of LR images fykg:

xn1 xn 1

k

XKk1

f1k yk ~ynk " sp: 4

The differences between the original image and the syn-

thetically generated image are up-sampled to achieve the

smaller SR pixel size, moved to a common reference frame,

and averaged over K acquisitions. The symbol p is termed

the BP kernel and is related to h [27] (often h is assumed

to be symmetric which results in p h). The defined iterative

process [equations (3) and (4)] is repeated until a predefinederror measure, such as the mean square error in the

maximum-likelihood (ML) sense:

12ML

x XKk1

yk ~ynk

25

has been minimized or has reached a predefined threshold. Alter-

natively, a maximum number of iterations has been reached.

Equation (1) defines a classical restoration problem for the

original HR image x. The solution to this problem depends on

the minimization criterion. Equations [3, 4] describe a solution

in the sense of ML estimator that minimizes theL2

normcriterion [equation (5)], which assumes an additive indepen-

dent noise with normal distribution in the forward model.

In [27], the solution for the general Lp norm is derived. It is

shown that the L1 norm is more robust and can be used for

strong sporadic noise.

In SR reconstruction with MAP estimation, prior know-

ledge on the imaged data, or the desired reconstructed

solution, is incorporated as a regularization term in the mini-

mization process:

12MAP

xn XKk1

yk ~ynk

2lA xn; 6

where A(x) is the regularization term providing prior infor-

mation on the desired SR image x, and l is the regularization

coefficient specifying the weight of the regularization term.

Several regularization terms from the literature have been

used in SR, including the Tikhonov cost function [26] and

the bilateral filter [27].

2.3. SR parameters and performance evaluation:

general and medical domain considerations

Each SR algorithm has certain key parameters that need to be

determined to most closely match with the true imaging

system characteristics and specific application scenario. Two

key parameters are the transformation and blur: the transform-

ation parameter needs to enable precise image registration,

accurate to a small fraction of a pixel, capable of bringing

all input images to a common reference frame. In medical

systems, the transformation is typically the physical shift

(from an original position) between the object (patient bed)

and the imager.

48 H. GREENSPAN



7/21

The blur, h, is defined by the imaging system PSF. In

medical imaging systems, the PSF is often defined as the

slice excitation profile. Experiments indicate that the typical

slice profiles are well approximated by Gaussian functions,

where the FWHM is the originally selected slice width. Two

PSFs are thus commonly used to represent the blur: the first

is a rectangular pulse PSF which is a crude estimation forthe slice profile (termed Box-PSF), where the box width is

taken as the selected slice width (in the desired HR pixel

units). The more accurate representation is the use of a Gaus-

sian PSF (Gaussian-PSF), with FWHM set to the selected

slice width.

The BP parameter of equation (4), p, is ideally the inverse of

the blur kernel. In [24], it was proven that the proposed SR

algorithm converges with a convergence condition (for 2-D

translations and rotations) given by kd2 h *pk, 1, where

d is the unity impulse function. The smaller kd2 h * pk is

the faster the algorithm converges. This criterion allows the

kernel p to be other than the exact inverse of the blurring func-

tion h. For the typical slice profiles mentioned above, the pfilter can be taken as an impulse function, satisfying the

Irani Peleg requirement for convergence. In both MRI [3]

and PET studies [5], the blur h and BP kernel p were corre-

spondingly set to unity. In [6], an increase in model accuracy

is attempted by defining both the blur kernel as well as the BP

kernel, to be modeled as a Gaussian PSF.

In regularization-based schemes, an additional minimiz-

ation parameter exists per defined objective function. Optimi-

zation of this parameter can be performed in all three spatial

directions, denoted as 3-D anisotropic filtering, or in the

slice select direction only, known as 1-D anisotropic

filtering.

Quantitative measures for any image enhancement pro-

cedure are a challenge. In general, image enhancement can

be expressed as the increase in edge slope in the image

plane as well as the increase in frequency content as viewed

via the image power spectrum. When considering a method

for resolution improvement, it is important to ensure that the

SNR is not compromised. The SNR is measured by taking

the mean of a high-intensity region of interest and dividing

by the standard deviation of a region of noise outside the

imaged object. A variation of the above measure is a contrast

ratio measure, C, which defines the ratio between the average

signals to the average background.

Traditional alternatives to SR reconstruction include zero-

padding, or sinc interpolation, and interleaving. In zero-

padding, the spectral resolution is augmented by adding

zeros embedded between the given samples. Padding the

data with zeroes provides more frequency-domain points

(improved spectral resolution), but does not improve the resol-

ution limits as established by the given sampling rate, nor does

it alter the effects of aliasing error. Interleaving is a method for

achieving an HR image from a set of shifted LR images by

combining the pixels, one by one, from alternating LR

image inputs, to generate a single large image. In the studies

reviewed below, the two alternatives are utilized for compari-

son and evaluation of the SR methodology in varying medical

imaging modalities.

3. SR IN MRI

In this section, we focus on the application of SR method-

ologies to the MR imaging modality. We start by highlighting

the key factors that limit resolution in MRI. We then present

an analysis into what can be termed the SR dimensionality

in MRI. An extended overview of recent works that explore

the possibility of augmenting MRI resolution, using SR algori-

thms, is presented next. Both anatomical as well as functional

MR images have been used with promising initial results. In

[3], the IBP SR method was used for anatomical MRI.

Results were shown on both phantom as well as human

brain data and demonstrated that isotropic resolution can be

achieved while preserving SNR. SR reconstruction based onthe use of discontinuity-preserving regularization methods

was proposed in [4] for HR fMRI image reconstruction.

Results demonstrate that the use of SR may increase the

ability to detect and visualize small regions of neuronal

activity; moreover, the activated regions appear sharper and

provide better information regarding their morphological

limits and structure.

3.1. Resolution challenges in MRI

High resolution, isotropic 3-D MRI images are important for

visualization of 3-D volumes in the imaged object and for

early medical diagnosis. In practice, true 3-D acquisition

methods are frequently not effective or possible, as is often

the case in T2-weighted imaging, DWI and occasionally in

MR angiography (MRA). True T2-weighting is difficult to

obtain in reasonable imaging times by 3-D acquisition

methods. The problem arises due to the need for long signal

recovery between excitations to enable the operation of

the spin-echo mechanism that provides T2 contrast (see

Appendix A). Since all the spins are excited by every pulse,

the recovery time cannot be utilized and the sequence takes

a long time. In DWI, no 3-D technique for humans currently

exists. Sequences that acquire raw data pertaining to the

same slice or volume over many excitations cannot be modi-

fied to provide diffusion-weighted contrast because of phase

inconsistencies resulting from physiological motion. MRA is

another popular application that sometimes performs better

in the 2-D rather than the 3-D version. In fMRI, temporal res-

olution as well as spatial resolution is important. Most 3-D

acquisition procedures cannot reach the required temporal res-

olution necessary for appropriate statistical analysis.

When the true 3-D image acquisition is not effective or

possible, it is common practice to acquire a set of 2-D




8/21

slices. The problem, as illustrated in Fig. 7, is that a set of 2-D

slices does not give a good isotropic 3-D image. A recon-

structed MR image is commonly of HR in-plane (x, y) and

of much reduced resolution in the slice-select (z) direction.

For example, it is common to find reconstructed MR images

of size 1 1 3 mm3. The spatial resolution in-plane (x, y)

is determined by several factors, including the gradients

intensity, the imaging bandwidth, the number of readout

points and phase encoding steps (see Appendix A and

related references). The slice thickness in MRI is determined

by what is termed the slice-selection pulse, which is in turn

determined by hardware limitations coupled with pulse

sequence timing considerations. The challenge for SR in

MRI is to increase the resolution in the slice-select dimension

so as to achieve HR, isotropic, 3-D images. A further chal-

lenge is to achieve the HR outcome without decreasing the

SNR.

3.2. SR dimensionality

Several works have used theoretical analysis and experimental

validation, to reveal the potential for SR in MRI along with the

appropriate task dimensionality [3, 28]. Signal-processing

principles underlying MRI acquisition, in particular, in

Fourier-encoded MRI data sets, have led researchers to

acknowledge the distinct characteristics of the in-plane

versus slice-select encoding. This fact, in turn, affects the

effective dimensionality of the SR task. In particular, Fourier-

encoded in-plane MRI data is inherently band limited. This is

due to the time limit of the acquisition process and the fact that

the information is gathered in the frequency domain (known as

k-space acquisition). In-plane shifting is thus equivalent to a

global phase shift in the acquisition space (k-space), the orig-

inal temporal domain, which does not affect the inherent

spatial frequency resolution of the acquired data. In other

words, increasing the in-plane resolution by in-plane shifting

of the image is equivalent to zero-padding of the raw data in

the temporal domain. A different scenario exists in the

slice-select direction of a Fourier-encoded MRI. There is

sufficient information in the slice-select dimension such that

under-sampling of the data in that direction results in aliasing.

A less sharp cut-off can thus be observed when viewing thespatial frequencies in the slice-select (z) direction in Fourier-

encoded MRI. The existing aliasing in the slice-select direc-

tion provides the basis for using SR algorithms in enhancing

the resolution.

The illustration shown in Fig. 8 (taken from [3]) is used to

validate the above claims: multislice 2-D image data sets were

acquired, with half-voxel shifts in all three spatial directions.

A 3-D iterative SR algorithm was applied [3]. The original

LR image is shown (top left) with its original power spectrum

(top right). The output of the SR process (double size in each

dimension) is shown (bottom left) with its power spectrum

(bottom right). The sharp frequency cut-off in the y

(in-plane) direction is evident. A spreading out of the power-spectrum is present in the slice-select (z) direction, indicating

an effective augmentation in the resolution. The conclusion

regarding the dimensionality of the SR task is the following:

the best that can be done in the in-plane (x, y) is to interpolate,

via zero-padding, the given data to the desired resolution. In

the slice-select dimension, sub-voxel spatial shifts can in

fact be used to increase the resolution. In current Fourier-

encoded MRI systems, the task is inherently a 1-D, slice-select

task. An interesting point is that if successful in this dimen-

sion, the goal of 3-D isotropy can in fact be achieved.

3.3. Experiments and results: anatomical MRI

In [3], the possibility of using SR for inter-slice MRI data was

explored. Several key results are reviewed herein. The SR

algorithm used was the Irani Peleg IBP method of [24]

(Section 2). Anatomical MRI results are shown on a

phantom, on inanimate objects and on a human brain. Quali-

tative results are shown, followed by quantitative evaluation.

All imaging was performed with an RF head coil on either a

1.5 Tesla GE Signa MRI system or a 3 Tesla GE MRI

FIGURE 7. MRI slice acquisition. The resolution in the slice-select

direction is lower than in the in-plane directions [3]. FIGURE 8. Investigating SR dimensionality in MRI. Spectrum

analysis (y,z) plane, Horizontal axis is the slice-select (z) direction.

Top row: LR input (left) with spectrum (right). Bottom row: HR

output (left) and spectrum (right) [3].

50 H. GREENSPAN



9/21

system. The phantom used for the experiment consisted of

long thin plastic partitions (teeth), lodged in a plastic

block, placed 4 mm apart, surrounded by Gd-DTPA-doped

water. The imaging sequence consisted of multislice fast

spin-echo (FSE) with 16 slices, 3-mm thick, approximately

parallel to the plastic partitions. Three sets of multislice data

were acquired, with 1 mm shifts in the slice-select direction.

The LR input voxel size was 1 1 3 mm3. Following the

SR procedure, an output voxel is a 1 mm isotropic cube.

Results are shown in Fig. 9. The visibility of the comb teeth

has greatly improved by using SR rather than zero-padding

interpolation. Moreover, more information is evident with

SR than with interleaving. The implementation with a

GaussianPSF (e) seems to give slightly better results than

when using a box-PSF (d): Better estimation of the HR

image is achieved by using a blurring filter, h, that more

closely matches with the MRI system and the MR image

characteristics in the slice-select dimension.

Results on an inanimate object are shown next. A papaya

image (zoomed-in) is shown in Fig. 10. The x-axis is the slice-

select axis. The input LR image, shown in Fig. 10a, is of res-

olution 1 1 3 mm3, whereas the image following SR

shown in Fig. 10b has a resolution of 1 1 1 mm3. To

quantify the resolution augmentation, both the image and fre-

quency domains are used. Figure 10c and d shows a

comparison of edges (two examples) within the two images,

while Fig. 10e and f shows the augmentation of the frequency

spectrum in the slice-select direction.

Figure 11 shows initial SR results on human brain data. An

FSE imaging sequence was used with three shifts in the slice

direction. The original slice thickness was 4.5 mm, in-plane

resolution was 1.5 mm and the number of slices was 22. SRoutput resolution was 1.5 mm (cubed). Results show a clear

improvement in the progression from the LR input

(Fig. 11a) or the zero-padded input (Fig. 11b) to the SR

results (Fig. 11c and d).

Quantitative evaluation was carried out on an apple input

source (in [3] and revalidated in [4]). The resolution was quan-

tified by the measurement of edge widths (see [3] for details on

measuring width). Timing required and measured SNR were

recorded as well. Table 1 summarizes the main results.

Looking across the rows it is clear that the resolution in edge-

width improves as a shift is made from the zero-padded input

to the HR source. The SR result is much better than the zero-

padded or interleaved result, with the mean edge width of theSR result almost identical to the mean width of the HR source.

SNR values decrease with the increased resolution. Note that

the SNR of the SR result is higher than the SNR of 2-D thin

slice acquisitions. In an MRI process, the goal is to obtain

HR images with a high-SNR efficiency, which is the ratio

between the SNR of the result and the square root of the

time length of the data acquisition sequence. The SNR effi-

ciency measure displays a similar trend: an SNR efficiency

of 7.63 is achieved for the SR result, compared with 6.13 for

the HR acquisition.

3.4. Experiments and results: fMRI

Peeters et al. [4] propose an optimization approach for HR

fMRI reconstruction using SR. The fMRI acquisition is

adapted to acquire two image stacks with low slice resolution,

shifted over half-a-slice thickness. Two separate slice-shifted

overlapping volumes are acquired, each obtained at half the

acquisition time of the HR volume. The shifted volumes are

combined via SR to reconstruct a stack of slices with half

the acquisition thickness. The SR methodology used in this

work is based on edge-preserving approaches and conver-

gence rate studies [29].

fMRI data differs from anatomical MRI data in that it

involves dynamical data; it images the hemodynamic response

function (hrf). It is important to note that the image data

acquired during the initial and final portion of the hrf is not

in a stationary state (plateau), and thus cannot be utilized in

SR algorithms, which assume a combination of two sets of

volumes acquired with an identical neurophysiological

response condition. In [4], the solution suggested to this

dynamics issue is the elimination of specific volumes that

were acquired during the non-stationary state of the response.

FIGURE 9. SR on a phantom image. (a) The original LR data. (b)

Zero-padding interpolation. (c) Interleaving the slices. (d) SR with

box-PSF. (e) SR with Gaussian-PSF. Horizontal axis is the slice-

select axis [3].




10/21

Results of using SR on fMRI datasets, both simulated and real

data are shown next, based on [4].

In the real fMRI datasets, a visual stimulation paradigm for

retinotopic mapping was used. The stimuli used were designed

to stimulate the horizontal (HM) and vertical (VM) visual field

meridian, using horizontally and vertically oriented wedge-

shaped checkerboards alternating at 4 Hz. The HM and VM

stimuli were alternated in blocks of 10 brain volume scans.

In this experiment, a total of 10 sessions of 12 blocks each,

i.e. 120 scans per session, were performed on the same

subject. During the experiment two different acquisition strat-

egies were interleaved: the HR fMRI (ground truth) acqui-

sition protocol and the LR (slice shifted) fMRI acquisition

protocol, yielding five high resolution (ground truth) and

FIGURE 10. Papaya example. (a) LR data. (b) SR result. (c) and (d) Comparison between two corresponding edges of the input image (dotted line)

and the SR image (solid line). (e) Power spectrum of input image. (f) Power spectrum of SR image.

FIGURE 11. Human brain MRI. (a) The original LR data. (b) Zero-padding interpolation. (c) SR with box-PSF. (d) SR with Gaussian-PSF [3].

52 H. GREENSPAN



11/21

five slice-shifted LR volume datasets. In the slice-shifted

mode, the slice thickness was doubled as compared to the stan-

dard sequence. The two shifted volumes were acquired con-secutively, with the second volume shifted in a slice position

over a distance equal to the slice thickness of the HR

images. The high-resolution data were collected on a

Siemens Sonata 1.5 Tesla MR system. A matrix of 128

128 was acquired. Voxel size was taken as 2 2 2 mm3

for the ground-truth (HR) images and 2 2 4 mm3 for the

slice-shifted images. The global acquisition time of the high-

resolution volume was 3328 ms and each of the slice-shifted

interleaved volumes was 1664 ms (for additional acquisition

protocol details see [4]).

Figure 12 shows statistical parametric mapping (SPM) acti-

vation maps [30], displaying the activated areas above a stati-

stical threshold (pcorr, 0.05), overlaid on the mean EPI slices

of the interpolated datasets. Patches of color on the MRI brain

slice shows differences in brain activity, with the colors repre-

senting the location of voxels that have shown statistically sig-

nificant differences between experimental conditions (see

Appendix A). Figure 12a displays the images of the retino-

topic mapping fMRI experiment in the acquisition plane

with the corresponding activation superposed, and Fig. 12b

shows activation images perpendicular to the slice direction.

Cases compared include the following (top to bottom, left to

right): original HR and LR inputs, an average dataset in

which each HR voxel is computed as the average of all LR

voxels that contain it (equivalent to the initial guess step of

Irani Peleg), two SR results, and a composed result,

which is equivalent to interleaving. In the SR algorithm, the

regularization term was applied in all three dimensions (Ani-

sotropic 3-D) or in the slice-select only (Anisotropic 1-D).

Qualitative analysis of the results indicates overall simi-

larity between the interpolated datasets and the reference

HR dataset. A closer look at the data reveals that the 1-D

and 3-D anisotropic SR datasets show higher t values at the

foci of the activated areas than the reference HR, the original

LR and the average and composed data sets. Also the size of

the activated clusters with a t value above threshold appears

to be larger in the SR datasets compared to the original data.

When comparing the two different SR datasets (3-D versus

1-D anisotropic regularization), the following differences are

observed: the 3-D anisotropic images display more intense

but less sharp activation patches than the 1-D anisotropicinterpolated and HR dataset both in- and through-plane. The

1-D anisotropic SR dataset demonstrates a higher resolution

of the activated patches in the slice direction than the LR

dataset and the 3-D SR anisotropic dataset. These results can

be explained via the larger smoothing inherent to the 3-D ani-

sotropic interpolation algorithm in all directions.

A second synthetic database was generated with known acti-

vation areas inserted, as shown in Fig. 13. The mean EPI MR

volume was used as a template with a base resolution of 3

3 4 mm3. This template was duplicated 120 times to gener-

ate a dynamic time series, with different activated regions

inserted in an interleaved mode of 10 rest volumes and 10

activated volumes. These activation regions consisted ofdifferent spheres with different radii and an irregularly

shaped area at carefully chosen positions. The intensity of acti-

vation was set to a maximum of 8% peak signal change. Two

slice-shifted LR datasets were generated by addition of the

adjacent slices of the HR dataset. Gaussian noise was inserted

with a standard deviation of 2% for the HR set and 1% for the

LR volumes. Figure 13 shows the statistical analysis on

the synthetic dataset, with similar conclusions obtained as in

the real dataset. Both 1-D and 3-D SR anisotropic datasets

extract the activated areas in good agreement with the position

and size of the simulated activated areas.

Quantitative measures of the ability to separate two

close-by-activated areas were extracted. Figure 14 shows the

line graphs of calculated t-values of a cut in the slice direction

through two activated regions, with a separation of two slices

in Fig. 14a and a single slice in Fig. 14b. The results demon-

strate that the SR algorithms show a good separation, closely

resembling the HR dataset and much better that in the LR

input and average or interleaved datasets.

4. SR IN PET

In this section, we focus on the application of SR to PET. We

discuss the resolution limitations in PET and the challenges for

SR in PET. Recent works that have started to investigate the

potential of SR in this domain are reviewed herein.

PET resolution is limited by physical properties, such as

scatter, counting statistics, positron range and patient

motion, as well as by the detector array geometry and the

implemented acquisition protocol. Detector widths are

limited to a certain minimal size, due to SNR considerations.

A width that is too small will reduce detection efficiency and

will increase intercrystal scatter and penetration. Resolution in

TABLE 1. Quantitative measures of SNR and resolution on apple

FSE sequence

MRI on an apple

input source

Acquisition

time (min:s)

Mean

edge width

(pixels)

SNR SNR

efficiency

(s21/2

)

Zero-padded

reconstruction

1:28 3.7 287 30.6

Interleaved

reconstruction

4:24 3.7 276 16.97

SR reconstruction

box PSF

4:24 2.9 170 10.46

SR reconstruction

Gaussian PSF

4:24 2.2 124 7.63

High resolution 4:00 2.3 95 6.13

Original slice width of 4.5 mm; three shifts.




12/21

PET scanners is often degraded in order to achieve an accep-

table image variance, where the variance is largely determined

by the number of counts (counting statistics) collected during a

scan. The noise that affects the counting statistics is comprised

of several factors, as illustrated in Fig. 15. The first one is the

angular uncertainty of the photons created in the annihilation

process. Although the photons emitted in this process should

move in a straight line of 1808 with respect to each other,

there is a small angular divergence. The second limiting

factor is the scatter events that one or both of the photons

may pass before they reach the detector. The scatter event

causes miss-estimation of the line where the annihilation

process took place. Also present are random events that

occur simultaneously and introduce wrong information to

the reconstructed image.

Current clinical PET scanners consist of 18 39 rings of

detectors, which are aligned axially. A volumetric PET data

set is commonly reconstructed by collecting a stack of 2-D

transaxial images perpendicular to the axial (bed) direction.

Many PET scanners have the option of restricting the

line-of-response for gamma-ray coincident pair detection to

the transverse plane perpendicular to the axial direction.

This restrictive acquisition mode is termed the 2-D acquisition

mode. A 3-D acquisition mode is one in which no restrictions

apply during the acquisition process thus maximizing the

number of events detected. In this scenario, the data are typi-

cally rebinned into transverse planes [32] and then recon-

structed using a 2-D algorithm that generates images from

projection data [33]. In either mode, a reconstructed 3-D

PET data set consists of a stack of 2-D transverse images

FIGURE 12. Slices of activated areas resulting from the visual stimulation paradigm overlaid on mean EPI images for different datasets.

(a) Transversal acquisition plane. (b) Sagittal slices [4].

54 H. GREENSPAN



13/21

along an axial direction. Coronal or sagittal images are gene-

rated by re-sampling the voxel matrix along these planes.

Accordingly, the spatial resolution in the transaxial plane is

largely limited by the detector width, whereas the resolution

along the axial direction is affected by the spacing of the

detector rings. In practice, the final reconstructed resolution

of a PET image is usually poorer than the best obtainable,

intrinsic resolution, because reconstruction algorithms typi-

cally trade-off resolution for reduced noise. In [16], an

example is given where the intrinsic resolution is ,5 mm

yet the final resolution of the image is greater than 8 mm.

A typical resolution in clinical scanners is between 4 and

7 mm FWHM [31].

The SNR considerations, as briefly outlined above result in

an under-sampling of available data. Wider detectors define a

lower sampling frequency, while preserving a high-SNR ratio.

This trade-off ensures that PET is a good candidate for SR.

Higher resolution PET images may have several implications

in research and clinical practice. The imaging of small cer-

ebral structures such as the cortical sub-layers and nuclei

may need PET spatial resolutions of ,2 mm [34, 35].

Higher PET resolution would also be beneficial for improving

FIGURE 13. Comparison between activated areas observed in the high, low and interpolated datasets for the synthetic data. (a) Transversal

plane. (b) Sagittal slices [4].

FIGURE 14. Line graphs of a cut in the slice direction showing the z-score for activated areas separated by two slices (a) and one slice (b) for

different reconstructions. Real boundaries of the activated areas are shown with a solid black line (for color see [4]).




14/21

sensitivity for detection of small tumors [36]. Cancer lesions

need to be of diameters equal or larger than the resolution of

the PET scanner to be identified provided they also have ahigh-glucose metabolism. Finally, higher resolution PET

images may show a more differentiated anatomical structure.

The increase in anatomical detail may aid in the registration of

a PET image with a corresponding anatomical image from

another modality, such as CT or MRI.

In recent work, the possibility of augmenting PET resol-

ution using SR algorithms was explored [5, 6]. In [5], the

use of SR to improve PET resolution using shifts and rotations

in the transaxial plane as well as along the axial direction was

demonstrated (directions illustrated in Fig. 15). Motivated by

the results of SR in MRI [3], the IraniPeleg SR algorithm

was used, with results demonstrated on a phantom as well as

initial patient data. In a phantom study, the SR technique

was shown to improve resolution and increase the contrast

ratio, using a commercially available PET scanner, without

increasing total scan time. In the patient study, an increase

in scan time for one field of view (FOV) demonstrated that

it is feasible to apply SR axially in a clinical setting without

increasing the radiation dosage and without the need for any

modification to the PET scanner hardware.

4.1. Experiments and results

In [5], an SR scheme based on the Irani Peleg iterative algor-

ithm is proposed and experimentally confirmed to improve the

resolution of PET. SR attenuation corrected PET scans of a

phantom were obtained using the 2-D and 3-D acquisition

modes of a clinical PET/CT scanner (Discovery-LS PET/CT

scanner, GE Medical Systems). A special phantom was con-

structed as shown in Fig. 16. The phantom contained holes

of sizes 1, 1.5, 2, 4, 6 and 8 mm in diameter. Several exper-

imental scenarios were tested: 1-D axial shifts (equivalent to

bed-shifts), combined axial and transaxial shifts, and a

patient study for the detection of small lung lesions.

In the 1-D mode, SR via Irani and Peleg [24] was

implemented along the axial direction by combining the sets

of four LR acquisitions, each shifted by one-fourth of a LR

pixel relative to the previous one [similar method used in [3]

for the anatomical MRI (Section 3)]. In a combination of1-D (axial) and 2-D (transaxial) shifts, otherwise termed the

3-D brain acquisition mode, SR was applied to transverse

images using rotation and translation in the transverse plane

as the geometric shift between successive acquisitions.

Small CT markers were attached to the phantom in order to

track the shifts.

In the patient study, CT images were evaluated to identify

regions-of-interest exhibiting the suspicious small lung

lesions. Following a PET/CT scan, the patient was requested

to remain still and bed was positioned so that the ROI was cen-

tered in the FOV of the PET scanner. The scanner was pro-

grammed for four additional PET acquisitions of this FOV,

lasting 4 min each. Between each acquisition, the bed wasautomatically shifted by 1 mm. The patient was not exposed

to any additional radiation since the X-ray CT component of

the PET/CT study was not repeated. Both standard and SR

images were processed, with additional re-slicing to provide

coronal, sagittal and transverse images through the lesion of

interest. In all the scenarios above, the process of synthetically

generating LR acquisitions, comparing them to the four

measured acquisitions, and updating the HR estimate was

repeated until a predefined error was reached, or 16 iterations.

The final estimated HR image was referred to as the SR result.

Figure 17 shows the 3-D brain acquisition mode in which

both axial as well as transaxial shifts are conducted (combined

1-D and 2-D). CT markers provided the data needed to deter-

mine the geometric shifts between successive images. In this

example, between the initial PET image (Fig. 17b) and the

FIGURE 15. PET events. (a) Graphic representation of true.

(b) Scatter. (c) Random events [31].

FIGURE 16. Phantom disk for PET experimentations ([5], #2006

IEEE).

56 H. GREENSPAN



15/21

fourth PET image (Fig. 17c), there is a rotation of 7.28 and a

translation of 9.4 mm. Reconstructed images are shown in

Fig. 18. The top two images are coronal images and the

bottom two images are transaxial images. On the left are

images generated using a standard PET acquisition and on

the right are SR images. All images were taken with the

same total acquisition time. Both 4 (gray arrow) and 3 mm

(black arrow) holes are more distinct in the SR images than

in the standard acquisition images, for both coronary as well

as the transaxial cases. Table 2 indicates that the SR image

consistently provides a better contrast ratio than the other

methods (see definition of Contrast in Section 2).

An additional computational measure is the PSF FWHM,

which can be computed from an approximate point source.

Table 3 shows the case of a 1-mm diameter circular hole.

The axial resolution in the 2-D whole-body mode was calcu-

lated to be 4.1 mm FWHM with the SR algorithm, which is

superior to both interleaving (4.9 mm FWHM) and the stan-

dard reconstruction (4.8 8.6 mm FWHM). Thus, it can be

concluded, that for the 3-D brain scenario, SR provides

better resolution than the other methods both axially andtransaxially.

Figure 19 shows the SR data in a study of a patient with a

suspicious small lung lesion on CT. The uptake in the small

lesion seen in the SR image of Fig. 19c is more localized

than that seen on the corresponding standard original PET

image in Fig. 19b. The second uptake (gray arrow) does not

appear clearly in the SR image of Fig. 19c; rather it falls in

an adjacent image plane, shown in Fig. 19d.

FIGURE 17. 3-D brain mode acquisition. (a) CT with transaxial

rotation and translation, four acquisitions, gray arrow indicates one

of the CT markers. (b) Initial transaxial PET image. (c) Fourth PET

image ([5],#2006 IEEE).

TABLE 2. Contrast ratio C for the PET signals in 3-D AC brain

mode acquisition phantom trials ([5] #2006 IEEE)

Image type 3-mm

holes

4-mm

holes

6-mm

holes

8-mm

holes

Coronal

No SR N/R

a

N/R 2.1 2.9Four acquisitions

interleaved

1.1 1.3 2.4 3.0

SR 1.1 1.5 2.8 3.5

Transaxial

No SR N/R 1.2 2.2 2.7

SR 1.2 1.8 3.2 3.8

aN/R, not resolved.

TABLE 3. PSF FWHM values for phantom trials ([5]#2006 IEEE)

Acquisitionmode

Axis Standard(mm)

Interleave(mm)

SR(mm)

2-D whole body Axial 4.8 to 8.6a

4.1

3-D brain Axial 5.3 to 8.7 4.8

3-D brain Radial 5.2 to 5.5 N/Ab

4.3

3-D brain Tangential 4.9 to 5.2 N/A 4.3

aPoint source either centered in a pixel (lower value) or between

two pixels (upper value).bN/A, not applicable.

FIGURE 18. 3-D brain mode acquisitionresults. (a) Standard

coronal PET image. (b) Coronal image with SR. (c) Standard transax-

ial PET image. (d) Transaxial with SR ([5], #2006 IEEE).




16/21

5. SUMMARY AND CHALLENGES AHEAD

Recent studies using SR in medical applications have demon-

strated that using SR technology enables the limits on slice

thickness as posed by the physical properties of existing

imaging hardware to be effectively broken. Higher resolution

in the image plane usually means acquisition with a smaller

sampling distance, by using a smaller detector (e.g. in PET)

or by using higher magnetic field scanners and shorter

sampling distances (in MRI). Due to physical constraints,

HR image acquisition results in a lower SNR, i.e. a trade-off

exists between resolution and SNR. One of the key features

in SR technology is the ability to obtain an HR image with

almost the same SNR as the original LR images from which

it is constructed. In the current overview, we have demon-

strated this fact for MRI, fMRI and PET.

In both MRI and fMRI, reconstructed SR images displayed

a close resemblance to the HR data, while improving the SNR.

Two different SR algorithms were used in the reviewed study:

the Irani Peleg iterative BP algorithm [2, 3, 5] and a

minimization algorithm with a constraint term on the smooth-

ness of the solution [4]. Overall, the two approaches display a

similar set of results and conclusions. In the anisotropic SR

technique of [4], an investigation was conducted into theeffect of 1-D versus 3-D smoothing. Quantitative comparison

of the activation maps indicates that the 3-D anisotropic diffu-

sion SR data set provides the largest response and the largest

activated areas, with the extracted regions much smoother

than the 1-D case. Thus, for increasing the detection capability

of small-activated areas, a 1-D smoothing filter is to be chosen.

In the PET phantom study [5], smaller features were

resolved with SR than without (3-mm features, as opposed

to the minimum of 4-mm in standard techniques), furthermore

the features that were resolved have a higher SNR. The

phantom trials showed improvement in both the axial and

transaxial resolutions. The axial resolution was improved by

952% compared to the standard method and by 1416%

compared to the interleaving reconstruction method. In the

3-D brain mode transaxial images, SR improved the resolution

by 12%. In the patient study, SR displayed more accurate

(more localized) 18F-FDG uptake, without using any hardware

changes or any increase in the patient radiation exposure. In a

recently published study [6], two main extensions were inves-

tigated: first, a more accurate SR algorithm was defined, in

which both the blur kernel as well as the BP kernel are

FIGURE 19. Coronal (left), sagittal (middle) and transaxial (right) sections of one FOV of a patient. The white arrows and black arrows denote

the lesion of interest. (a) X-ray CT scan. (b) Original non-AC18

FFDG PET image using clinical reconstruction protocols. Image planes dis-

played are 4-mm thick. (c) Non-AC PET image through the center of the lesion of interest using SR. (d) Non-AC PET image using SR. The sec-

ondary foci of uptake (gray arrows) seen in the original images (b) are evident here, in super-resolution images of planes adjacent to those depicted

in (c) ([5], #2006 IEEE).

58 H. GREENSPAN



17/21

modeled with a Gaussian PSF. This, more accurate modeling,

improves the above phantom results by an additional 24%.

A second contribution of [6] is the fusion of PET and CT

data: in addition to augmenting the PET resolution via SR, a

post-processing step combines smoothing and edge-

enhancement of the resultant image, based on the HR border

information from the CT. The combined SR and anatomicaledge information were evaluated in phantom and patient

studies using a clinical PET scanner. In both cases, a substan-

tial increase in contrast was demonstrated.

SR, for images and image sequences, has customarily been

treated as a 2-D problem. In the overviewed studies, SR has

often been applied to 1-D signals. In the general medical

arena, the extension of the SR concept to 3-D is strongly motiv-

ated, as has been recently proposed in [3]. Practical constraints

have so far limited actual usage of true 3-D SR algorithms. In

Fourier-encoded MRI, in-plane resolution is constrained

(Section 3). The 3-D problem is thus downgraded to a 1-D

task. It may be the case that with the new MRI technology

that is not Fourier-based, the possibility for 3-D SR mayarise. The PET SR example demonstrates SR in more than a

single dimension. It seems that true 3-D may be applicable in

this domain and may be a worthwhile effort for the future.

Regardless of the dimensionality of the task, an important con-

tribution of SR is the reduction of PVEs in the reconstructed

image. In this respect, even if a SR algorithm is applied in a

single (axial) dimension, it in effect contributes to the increased

resolution in additional (transaxial) dimensions as well.

5.1. Spiral CT: a case where SR does not work?

An important question to address is the applicability of SR to a

given medical imaging system. It is important to note that SR

can augment the resolution as acquired by the system

detectors, in cases in which the detectors have under-sampled

the input data. In other words, high frequencies exist in the

signal that reaches the detectors, and the detectors sampling

limit leads to aliasing and degradation in the high-spatial fre-

quency content, as output in the reconstructed image. SR

reconstructs the aliased high frequency information thus pro-

viding a higher resolution output and minimizing the aliasing

problem. In cases in which no frequencies exist that are higher

than half of the detectors sampling frequency, no additional

improvements in the image resolution can be obtained by

SR technology.

Such is the case in recently developed spiral CT systems.

Today it is possible to scan the complete body trunk with sub-

millimeter isotropic resolution in less than 30 seconds, with an

effective slice thickness of 0.10.2 mm. Although the detec-

tors sampling looks very promising, it is difficult to achieve

this resolution in the reconstructed image. The main reason

is the PSF of the spiral CT acquisition system, which has

an equivalent bandwidth of 12 mm (for an elaborate discus-

sion on the PSF of the CT system, see [37]). Thus, in spiral CT,

the main obstacle for achieving increased resolution is the

relatively large PSF of the system. To summarize, in spiral

CT, a 3-D volume can be generated which is over-sampled

(by decreasing the spiral pitchthe density of the helicon

turns) and is heavily blurred across all dimensions. Various

de-blurring algorithms can be utilized in this scenario. These

in turn will not provide the sub-pixel level resolutiondesired. It is de-blurring, and not SR.

5.2. Additional issues and future challenges

Newly emerging hardware may provide additional means for

resolution augmentation. In the MRI field, new parallel

imaging techniques are currently being developed. Such tech-

niques will allow faster acquisition and higher in-plane

resolution. Yet, in many of the developed techniques, the

added resolution comes at the expense of SNR. The ability to

use SR post-processing of thick slices may provide the boost

needed for the SNR. Novel encoding methodologies, such as

non-Fourier methods (e.g. hadamard wavelets) are starting toemerge in MRIfor encoding the third dimension [38].Suchtech-

nologies may enable the utilization of true 3-D SR techniques.

The current overview was aimed at summarizing the key

published results of SR in medical applications. As such, not

all specific details per modality were presented, including

certain pre-processing and post-processing steps. The reader

is advised to consult the related literature for more specific

implementation details. The overview is definitely not exhaus-

tive: additional studies are currently emerging in the MRI

research community [39, 40]. Additional modalities exist

that have not been covered, including ultrasound and

microscopy. Finally, the review is based on the published

research and does not reveal the state-of-the-art in existing

medical hardware.

Futurework canadvance thetopic in twomain directions: On

the clinical frontfinding the applications that may gain most

from the SR technology and implementing the theory in the

clinical practice. On the SR algorithmic frontextending the

investigation into additional medical imaging modalities as

well as comparing between SR algorithms to find the advan-

tages of each per modality. A small number of studies have

recently attempted to compare the performance of SR algor-

ithms on MRI data. Results seem to indicate that no major

difference exists between the Irani Peleg results and the

ML-based frameworks.

5.3. SR versus segmentation versus registration

SR cannot be viewed as an isolated domain. A strong three-

way relationship exists between SR, image segmentation and

image registration. It is our conjecture that future research in

the field will focus on strengthening this three-way relation-

ship. Augmented resolution of an image can augment its regis-

tration to another image or to an atlas, and it can, of course,




18/21

greatly contribute to its segmentation (e.g. resolving PVE).

We would like to suggest that registration and segmentation

are also critical contributors in advancing the SR field, in

general, and in particular in the medical domain. Accurate

registration is a key factor in achieving satisfactory SR

results and is required to facilitate the use of SR algorithms

in real-world clinical settings. In the studies presented,patient motion was largely ignored. In both MRI as well as

PET, the focus was on head scans, given that the head is con-

strained to a cradle and is less likely to move during the scan.

For the more general case of a moving subject or organ, accu-

rate image registration methods need to be incorporated.

Strong registration schemes will enable the development of

true 3-dimensional algorithms that involve 3D patient motion.

Image segmentation and image modeling can advance the

field by introducing important prior knowledge, which in

turn can be included as part of the regularization terms

within the SR formalism. Bi-lateral total variation regulariz-

ation was shown to provide a within-region smoothing

effect while preserving strong transitions (edges) within theimage. A variation on this theme was demonstrated as a

post-processing step in [6] with improved results. Utilizing

image segmentation further may entail using region, or

tissue properties (such as characteristic intensity), as key infor-

mation within the regularization framework. Important infor-

mation exists in the medical domain, such as statistical

atlases for location prior, and tissue modeling for intensity

priors, all of which provide additional key information and

opportunities for advanced SR algorithms in medical

applications.

ACKNOWLEDGEMENTS

I would like to thank my colleagues and collaborators on

SR related research: Dr. Sharon Peled, Prof. Nahum Kiryati,

and Dr. Yossi Rubner. Thanks to Prof. Azhari and

Dr Kennedy for discussions on SR in PET applications.

Supporting the current study: Uri Marias, Oren Friefeld, Avi

Ben-Ezra. The author is grateful for the suggestions by the

anonymous referees to this study.

FUNDING

Research was supported in part by the Ela Kodesz Institute

for Medical Engineering and Physical Sciences, and by

the Adams Super-Center for Brain Studies, Tel-Aviv

University.

REFERENCES

[1] Roux, C. and Udupa, K. (2003) Special issue on emerging

medical imaging technology. Proc. IEEE, 91, 1479 1482.

[2] Peled, S. and Yeshurun, Y. (2001) Superresolution in MRI:

application to human white matter fiber tract visualization by

diffusion tensor imaging. Magn. Reson. Med., 45, 2935.

[3] Greenspan, H., Oz, G., Kiryati, N. and Peled, S. (2002) MRI

inter-slice reconstruction using super-resolution. Magn. Reson.

Imaging, 20, 437446.

[4] Peeters, R.R. et al. (2004) The use of super-resolutiontechniques to reduce slice thickness in functional MRI.

Int. J. Imaging Syst. Technol., 14, 131138.

[5] Kennedy, J.A. et al. (2006) Super resolution in PET imaging.

IEEE Trans. Med. Imaging, 25, 137147.

[6] Kennedy, A., Israel, O., Frenkel, A., Bar-Shalom, R.

and Azhari, H. (2007) Improved image fusion in PET/CT

using hybrid image reconstruction and super-resolution.

Int. J. Biomed. Imaging, Article ID 46846.

[7] Beutel, J., Kundel, H.L. and Van Metter, R.L. (2000) Handbook

of Medical Imaging. SPIE Press, Bellingham.

[8] Cho, Z.H., Jones, J.P. and Singh, M. (1993) Foundations of

Medical Imaging. Wiley-Interscience, New York.

[9] Liang, Z.P. and Lauterbur, P.C. (2000) Principles of MagneticResonance Imaging. IEEE Press.

[10] Epstein, C.L. (2003) Mathematics of Medical Imaging. Prentice

Hall, Upper-Saddle River, NJ.

[11] Gambhir, S.S. et al. (2001) A tabulated summary of the FDG

PET literature. J. Nucl. Med., 42, 1S93S.

[12] Moretti, A., Gorini, A. and Villa, F. (2003) Affective disorders,

antidepressant drugs and brain metabolism. Mol. Psychiatry, 8,

773785.

[13] Matsunari, I. et al. (2001) Phantom studies for estimation of

detect size on cardiac18

F SPECT and PET: implications for

myocardial viability assessment. J. Nucl. Med., 42, 1579 1585.

[14] Bax, J.J. et al. (2000) 18-Fluorodeoxyglucose imaging with

positron emission tomography and single photon emissioncomputed tomography: cardiac application. Semin. Nucl.

Med., 30, 281298.

[15] Bengel, F.M. et al. (2000) Non-invasive estimation of

myocardial efficiency using positron emission tomography

and carbon-11 acetate-comparison between the normal and

failing human heart. Eur. J. Nucl. Med., 27, 319326.

[16] Ollinger, J.M. and Fessler, J.A. (1997) Positron-emission

tomography. IEEE Signal Process. Mag., 14, 4355.

[17] BrainWeb. http://www.bic.mni.mcgill.ca/brainweb/

[18] Park, S.C., Park, M.K. and Kang, M.G. (2003) Super-resolution

image reconstruction: a technical overview. IEEE Signal

Process. Mag., 20, 2135.

[19] Tsai, R.Y. and Huang, T.S. (1984) Multiframe Image Restoration

and Registration. Advances in Computer Vision and Image

Processing, pp. 317339. JAI Press Inc., Greenwich, CT.

[20] Kim, S.P., Bose, N.K. and Valenzuela, H.M. (1990) Recursive

reconstruction of high resolution image from noisy undersampled

multiframes. IEEE Trans. Acoust. Speech, 38, 1013 1027.

[21] Ur, H. and Gross, D. (1992) Improved resolution from subpixel

shifted pictures. Comput. Vis. Graph. Model. Image Process.,

54, 181186.

60 H. GREENSPAN



19/21

[22] Elad, M. and Hel-Or, Y. (2001) A fast super-resolution

reconstruction algorithm for pure translational motion and

common space invariant blur. IEEE Trans. Image Process.,

10, 11871193.

[23] Chiang, M.C. and Boult, T.E. (2000) Efficient super-resolution

via image warping. Image Vis. Comput., 18, 761771.

[24] Irani, M. and Peleg, S. (1993) Motion analysis for image

enhancement: resolution, occlusion, and transparency. J. Vis.

Commun. Image Represent, 4, 324335.

[25] Patti, A.J., Sezan, M.I. and Tekalp, A.M. (1994) High-resolution

image reconstruction from a low-resolution image sequence in

the presence of time-varying motion blur. Proc. ICIP, Austin,

TX, pp. 343 347.

[26] Elad, M. and Feuer, A. (1997) Restoration of single

super-resolution image from several blurred, noisy and

down-sampled measured images. IEEE Trans. Image

Process., 6, 16461658.

[27] Farsiu, S., Robinson, M.D., Elad, M. and Milanfar, P. (2004)

Fast and robust multiframe super resolution. IEEE Trans.

Image Process., 13, 1327 1344.

[28] Scheffler, K. (2002) Superresolution in MRI? Magn. Reson.

Med., 48, 408.

[29] Nikolova, M. and Ng, M. (2001) Fast image reconstruction

algorithms combining half-quadratic-regularization and

preconditioning. Proc. ICIP, Thessaloniki, Greece, pp. 277

280.

[30] Statistical Parametric Mapping (SPM). Available at http://www.

fil.ion.ucl.ac.uk/spm/

[31] Tarantola, G., Zito, F. and Geundini, P. (2003) PET

implementation and reconstruction algorithms in whole-body

applications. J. Nucl. Med., 44, 756769.

[32] Defrise, M. et al. (1997) Exact and approximate rebinning

algorithms for 3-D PET data. IEEE Trans. Med. Imaging, 16,145148.

[33] Hudson, H.M. and Larkin, R.S. (1994) Accelerated image

reconstruction using ordered subsets of projection data. IEEE

Trans. Med. Imaging, 13, 601609.

[34] Kessler, R.M. (2003) Imaging methods for evaluating brain

function in man. Neurobiol. Aging, 24, S21S35.

[35] Shmand, M. et al. (1998) Performance results of a new DOI

detector block for a high resolution PET-LSO research

tomography HRRT. IEEE Trans. Nucl. Sci., 45, 3000 3006.

[36] Fukui, M.B., Blodgett, T.M. and Meltzer, C.C. (2003) PET/CT

imaging in recurrent head and neck cancer. Semin. Ultrasound

CT MR, 24, 157163.

[37] Schwarzband, G. and Kiryati, N. (2005) The point spreadfunction of spiral CT. Phys. Med. Biol., 50, 5307 5322.

[38] Goelman, G. (2000) Fast 3D T2 weighted MRI with Hadamard

encoding in the slice select direction. Magn Reson Imag., 18,

939945.

[39] Carmi, E. et al. (2006) Resolution enhancement in MRI. Magn.

Reson. Imaging, 24, 133154.

[40] Hsu, J.T. et al. (2004) Application of wavelet-based POCS

superresolution for cardiocascular MRI image enhancement.

Proc. 3rd ICIG, IEEE, pp. 572575.

APPENDIX A: A BRIEF INTRODUCTION TO

MEDICAL IMAGING MODALITIES

A.1. Magnetic resonance imaging

MRI utilizes the principles of nuclear magnetic resonance, a

spectroscopic technique used to obtain microscopic chemicaland physical information about molecules containing atomic

nuclei posessing non-zero magnetic moment, or spin. The

hydrogen nucleus in water is the most common nucleus used

for MRI. Placed in a magnetic field, the particles spin can

absorb external energy in the form of a photon, and thus

move to an energy level higher than the original equilibrium

energy level. Once the external magnetic field is stopped,

the spins start moving back to the lower energy level. The

shift between energy states results in an amount of energy

that is emitted from the system and which can be detected

using wire coils. The measured induced current in the coils

is called free induction decay (FID) signal. The magnitude

of the signal is proportional to the density of the protons inthe specimen.

The MR image is formed by extracting the locations of the

different spins from the FID signals. By using a magnetic field,

which varies spatially (e.g. changes linearly in all three direc-

tions), spins in different locations will yield FID signals with

different frequencies. Frequency encoding then enables to

extract the spatial information. In three dimensions, a plane

can be defined by slice selection, in which an RF pulse of

defined bandwidth is applied in the presence of a magnetic

field gradient in order to reduce spatial encoding to two dimen-

sions. Spatial encoding can then be applied in 2-D after slice

selection, or in 3-D without slice selection. In either case, a

2-D or 3-D matrix of spatially encoded phases is acquired,

and these data represent the spatial frequencies of the image

object. Images can be created from the acquired data using

the discrete Fourier transform. In conventional 2-D multi-slice

MR imaging, the RF pulse first selects the 2-D slice to be ana-

lyzed, following which a combination of frequency encoding

of the signal in one direction and phase encoding in the

other direction enables to encode the 2-D spatial information

within the slice (so called in-plane). A single slice may

require multiple RF excitations, with predefined repetition

time (TR) in order to be fully encoded.

The MR image intensities (per voxel) are proportional to the

number of nuclei in each voxel. MR image contrast is deter-

mined by several time constants, each of which reflects a

certain relaxation process that establishes equilibrium follow-

ing the RF excitation. As the high-energy nuclei relax and

realign, they emit energy at rates which are recorded to

provide information about their environment. The realignment

of nuclear spins with the magnetic field is termed longitudinal

relaxation and the time required for a certain percentage of the

tissue nuclei to realign (typically about 1 s for tissue water) is

termed Time 1 or T1 (spin-lattice relaxation). T2-weighted




20/21

imaging relies upon local dephasing of spins in the transverse

plane; the transverse relaxation time (typically ,100 ms for

tissue water) is termed Time 2 or T2 (spin-spin relaxation).

T2 imaging employs a spin echo technique in which spins are

refocused to compensate for local magnetic field inhomogene-

ities. A subtle but important variant of the T2 technique is

called T2* imaging, which is performed without the refocus-ing. This sacrifices some image integrity in order to provide

additional sensitivity to relaxation processes that cause inco-

herence of transverse magnetization.

Conventional image contrast is created by using a selection

of image acquisition parameters that weights the signal by T1,

T2 or T2*, or no relaxation time (proton-density images). In

the brain, T1-weighting causes fiber tracts (nerve connections)

to appear white, congregations of neurons to appear gray and

cerebrospinal fluid to appear dark. The contrast of white

matter, gray matter and cerebrospinal fluid is reversed

using T2 or T2* imaging, whereas proton-weighted imaging

provides less contrast in normal subjects. Various MRI

sequences exist, each defined by a set of sequence parametersthat determine the selected compromised between contrast,

spatial resolution and speed. Among the frequently used

MRI sequences are fast spin echo (FSE) and echo planar

imaging (EPI). The essential components for any imaging

sequence include: An RF excitation pulse, required for the

phenomenon of magnetic resonance, gradients for spatial

encoding (2D or 3D), and a signal reading, that combines

one or a number of echo types (e.g. spin echo, gradient

echo) and determines the type of contrast (the varying influ-

ence of relaxation times T1, T2 and T2*).

A.1.1. MRI angiography

Contrast enhanced angiography is based on the difference in

the T1 relaxation time of blood and the surrounding tissue

when a paramagnetic contrast agent is injected into the

blood. This agent reduces the T1 relaxation times of the fluid

in the blood vessels relative to surrounding tissues. When the

data are collected with a short TR value, the signal from the

tissues surrounding the blood vessels is very small due to its

long T1 and the short TR. Images of a region of interest are

recorded with rapid volume imaging sequences. An example

angiographic image is shown in Fig. 3c.

A.1.2. Diffusion-weighted imaging

DWI uses very fast scans with an addit

the computer journal 2009 greenspan 43 63

Documents