Advanced Image Processing Techniques for Physics Studies

T. Craciunescu and A. Murari

with contribution from:

G. Kocsis, P. Lang, I. Tiseanu, J. Vegaand JET EFDA Contributors *

*See the Appendix of F. Romanelli et al., Fusion Energy Conference 2008 (Proc. 22nd Int. FEC Geneva, 2008) IAEA, (2008)

Workshop on Fusion Data Processing Validation and Analysis, ENEA- Frascati (26-28 March 2012)

- Implementation (CLG, MPEG)

- Application to pellets and instability tracking

Optical flow - extraction of advanced information for control and physics studies

Automatic instability detection

- Phase congruency image classification

- Sparse image representation for disruption prediction

- Interest points and local features for image identification

attempt to find the vector field which describes how the image is changing with time

Basic assumptions:

Ill-posed problem:

- small perturbations in the signal can create large fluctuations in its derivatives

- undetermined set of equations

0),,()1,,( tyxftvyuxf

the grey values of image objects in subsequent frames do not change over time

small displacements: 0 tyx fvfuf

Optical flow

frame t frame (t+1)optical flow

Etlin

ger T

or s

eque

nce

A sufficiently large value for ρ is very successful in rendering the method robust under noise.

in flat regions where the image gradient vanishes, the problem become again undetermined.

Assumes that the unknown optic flow vector is constant within some neighbourhood of size ρ.

Incorporates a global smoothness assumption for the estimated flow field.

Larger values for α result in a stronger penalisation of large flow gradients and lead to smoother flow fields.

At locations with | f| ≈ 0, no ∇reliable local flow estimate is possible, but the regulariser | u|2 + | v|2 ∇ ∇fills in information from the neighbourhood - the filling-in effect.

dxdywwfJwwE TCLG

2231

Combined local-global (CLG) method

Coarse-to-fine multi-resolution approach

Recent investigations revealed that pellet ablation is a complex 3D process taking place on the μs timescale

→ pellet cloud dynamics (expansion, instabilities and drifts)

Analysis of pellet cloud dynamics and drifts by observing the visible radiation with fast framing cameras and by applying image processing algorithms

Experiments with sophisticated diagnostic settings performed during the 2011 campaign of AUG

__________________________________________________________________________________

*detailed results will be presented at EPS conference (G. Kocsis et al)

Predictive understanding of the underlying processes of the pellet-plasma interaction

Line profiles through the images and its reconstruction (bottom)

showing the extruded deuterium ice in case of JET pulse #76379

Image sequences provided by a CCD camera viewing the ice at the exit of the nozzles of the extrusion cryostat

Determination of ice extrusion velocity by optical flow method Illustration of optical flow calculations

(I)- Intrinsic frames - coded using only information present in the picture itself by discrete cosine transform (DCT)

– Processing at the level of MB8 blocks– DCT concentrates the energy into the low-

frequency coefficients (spatial redundancy)

MPEG-2 compressed space

• Statistical redundancies in both temporal and spatial directions:

- inter-pixel correlation - simple translation motion between

consecutive frames

→ neglecting the low value coefficients→ High-frequency coefficients are more coarsely

quantized than the low-frequency coefficients

(P)- Predicted frames are coded with forward motion compensation, using the nearest previous reference (of type I or P) images.

(B) - Bi-directional frames are also motion compensated, this time with respect to both past and future reference frames.

Encoding is implemented using MB14 macroblocks

The parts of the image that do not change significantly are simply copied from other areas or other frames.

In case of the other parts, for each MB16, the best matching block is searched in the reference frame(s).

→ Motion is represented by a field of motion vectors (MV)

→ one MV per macroblock

MV field used:

as a crude initial estimation for optical flow recovery

for image segmentation

Confidence measure to ensure that the MV field is meaningful

Assumption: areas with strong edges exhibit better correlation with real motion than textureless ones

weighted averages of the image gradients can be expressed using DCT coefficients:

• eigenvalue decomposition• size of the eigenvalue is a measure of uncertainty in the direction

of the corresponding eigenvector (the stronger the eigenvalue, the lower the error variance)

Total optical flow computation time: ~16.4 ms.

Image acquisition framing rate:50 Hz

Image processing step Time (ms)

Segmentation using information from MPEG video compressed domain

Applying the regularization rules for the

MV field0.6

Median filtering + Morphological operations 0.8

Optical flow calculation performed on the segmented image region

Image derivatives 1.8SOR iteration 4.1

Median filtering 0.3 Peak signal-to-noise ratio (PSNR) of the

residual image > 14 dB

The difference between the speeds of the different pellets in the same ribbon structure below 12.5%.

Error estimation

Computing time

Optical flow evaluation can be engrafted in MPEG compressing routines in case of real time estimation of the speed of moving objects.

Tracking of plasma instabilities

MARFEs can reduce confinement leading to harmful disruption → a risk for the integrity of the devices

MARFEs determine a significant increase in impurity radiation → a clear signature in the video data

Image processing step Time (ms)Applying the regularization

rules for the MV field 0.5

Median filtering 0.7Dilation/Erosion 3.0

Labeling 2.4Objection centroid

determination 0.3

Visually discernable features coincide with those points where the Fourier waves, at different frequencies, have congruent phases

Extraction of highly informative features at points of high PC

Lateral inhibition vs.

Pahse congruency

Mach bands

black – measued luminancered –brightnesses as perceived

M.C. Morrone et al., Mach bands are phase dependent, Nature 324(1986)250.

construction of PC from the Fourier components

Phase congruency

Approximations of F and G by convolving the signal with a quadrature pair of filters(linear-phase filters for phase information preservation)

t

An appropriate choice for constructing the symmetric/antisymmetric quadrature pairs of filters are the Gabor filters.

symmetric/antisymmetric quadrature pairs of nonorthogonal wavelets

nodd

neven QQ ,

S.N.Prasad,

J.Domke, Gabor

Filter Visualization,Technical

Report, University of M

aryland (2005)

Gabor filters with different frequencies and orientations

Response for Gabor filter oriented vertically

SIM map, pooled into a single similarity score

96.2% were correctly interpreted. From the misclassified events 0.03% were false positives and 3.5% false negatives.

Combine all the orientations

Sparse learned representations of video images

D – fixed, general (DCT, wavelet) or it can be adapted to suit the application domain.

Learning both D and in an efficient way has been the focus of much of recent published work.

• D initialized from random patches of natural images.• Then learned adaptively from the data such that the

decomposition is sparse:

Sparse image representation

Since images are usually large, the decomposition is implemented on overlapping patches instead of whole images.

LtsDx

iiikiD i

0

2

2,...,1,..min

Each patch written as a column vector

Sparsity: - counting the number of non-zero elements in a vector

Matching pursuit algorithm • first find the one D atom that has

the biggest inner product with the signal

• then subtract the contribution due to that atom

• repeat the process until the signal is satisfactorily decomposed

N different classes Si of signals

– learn separate dictionaries, one per class– a signal belonging to one class is reconstructed

poorly by a dictionary corresponding to another class.

– classification is performed by using residual reconstruction errors of a signal by the dictionary belonging to a class

as a discriminative operator for classification.

iDxR ,Limited results: ~ 85 % success classification rate

The dictionary trained on patches from natural images 2

, DXDXR

Decomposition error:

‘Good’ for one class ‘bad’ for the orher by incorporating discriminative components:

i ji

FjTiiiikiD

DDDxi

2

1

2

2,...,1,min

i j

ijFaA

2

Dictionary incoherence term

Encourages dictionaries associated to different classes to be as independent as possible, while still allowing for different classes to share features.

Improved (preliminary) results: ~ 92 % success clasification rate

Learning discriminative dictionaries

Further tuning adjustable parameters – mainly size of patches multiscale

Multiscale framework to capture first a global appearance of objects

→ atoms representing common features in all classes tend to appear repeated almost exactly in dictionaries corresponding to different classes

→ False similar reconstruction decomposition error

• Detect by inspecting the inner product of dictionary atoms.

• Threshold for controlling the sharing atoms

Bag-of-words model - represention of a ‘sentence’ as an unordered collection of words, disregarding grammar and even word order.

Image retrieval

→ algorithms to detect and describe local features in images: – MSER (Efficient Maximally Stable Extremal Region)

Transforms an image into a large collection of feature vectors invariant to:

• illumination changes• local geometric distortion

• image translation• scaling• rotation

extremal region iR qpRboundaryqRp ii ImIm ,

Component tree

Rooted, connected tree constructed by successive thresholdings taking into account hierarchic image inclusion

→ maximally stable regions are those regions which have approximately the same region size across 2Δ neighboring threshold images

Features calculated: mean gray value, region size, center

of mass, width, dimension of the bounding box

(weights for the features can be used to adapt to different

kinds of input data).

Matching criteria: smallest Euclidean distance between

feature vectors

MSE

R e

xam

ple

Reference image Image identification using maximally stable extremal regions

Various MARFE images

Conclusions

Phase congruency as a highly localized operator for automatic MARFE identification with a good prediction rate.

Optical flow method for the study of several fusion plasma relevant issues, able to provide the real velocity for objects moving close to structures.– MPEG Motion segmentation - a key contrivance to allow very fast optical flow estimation– Application to pellet injection and pellet dynamic

Sparse image representation for disruption prediction. Encouraging preliminary results. Improvements expected mainly from a more efficient definition and implementation of the discriminative operator.

Image retrieval by image local feature detection.