turbulence-induced 2d correlated image...
TRANSCRIPT
Turbulence-Induced 2D Correlated Image Distortion
Armin SchwartzmanDiv. Biostatistics
UC San DiegoLa Jolla, CA, [email protected]
Marina AltermanDept. Elect. Eng. & Comp. Sci.
Northwestern UniversityEvanston, IL, USA
Rotem Zamir, Yoav Y. SchechnerViterbi Faculty of Electrical Eng.Technion - Israel Inst. Technology
Haifa, [email protected]
Abstract
Due to atmospheric turbulence, light randomly refractsin three dimensions (3D), eventually entering a camera at aperturbed angle. Each viewed object point thus has a dis-torted projection in a two-dimensional (2D) image. Simu-lating 3D random refraction for all viewed points via com-plex simulated 3D random turbulence is computationallyexpensive. We derive an efficient way to render 2D imagedistortions, consistent with turbulence. Our approach by-passes 3D numerical calculations altogether. We directlycreate 2D random physics-based distortion vector fields,where correlations are derived in closed form from turbu-lence theory. The correlations are nontrivial: they dependon the perturbation directions relative to the orientationof all object-pairs, simultaneously. Hence, we develop atheory characterizing and rendering such a distortion field.The theory is turned to a few simple 2D operations, whichrender images based on camera and atmospheric proper-ties.
1. Introduction
Imaging through refractive media [5, 6, 25, 40, 42, 45]is of interest both for rendering [11, 47, 58] and scene anal-ysis [2, 27, 37, 54, 56]. It is studied in computer visionand graphics, as the importance of participating media isacknowledged [3, 8, 16, 18, 22, 24, 35, 36, 41, 46, 49, 55].Complex, random refraction is created by turbulent me-dia [13, 17, 29], often encountered in long range observa-tions through the atmosphere [23, 28, 48, 57] and ground-based astronomy [39, 44]. There, random perturbations ofthe refractive index [38] follow a complicated fractal multi-scale structure of eddies in the three dimensional (3D) do-main. This structure, in turn, creates highly complex re-fraction of propagating light, in the 5D plenoptic domain(space and direction). This leads to random perturbations
Figure 1: [Top] Computationally complex rendering of imagedistortion, requiring a 3D simulated random turbulent field, thenray tracing in 3D for all object points in the field of view. [Bottom]Our approach. Efficient rendering of turbulence-induced distortionas 2D image processing. It relies on closed-form physics-basedcovariance of the distortion.
of all light rays passing through this medium. Finally, thecomplexly refracted light is projected by a camera, forminga distorted image of the scene.
Now, suppose one seeks to render images that are dis-torted as if they are taken through atmospheric turbulence.A motivation for rendering can be computer graphics. An-other motivation is to form a database on which to testand develop recovery algorithms, to correct for turbulence-induced distortion [60, 61]. A database can also train alearning system to recognize objects via turbulence. Fromthe description above, apparently, rendering should includea series of computationally complex steps (Fig. 1[Top]):
Simulate a huge 3D turbulent random field (scale of kilo-meters) at a 3D resolution that is relevant to optics; Ray-
trace refraction through this 3D medium, from an objectpoint to the camera; Repeat this propagation process for allresolvable points in the field of view. In large scales, such arendering approach poses a computational burden. It is alsounnecessary.
We believe that for some applications, there is no needfor 3D simulations, in order to render turbulence-inducedimage distortion. There is no need to simulate a fractal ran-dom refractive index 3D field, or ray-trace through such afield. Basically, image distortion is an operation in just twodimensions (2D). The input is a 2D image, free of randomdistortion, i.e. the view in the absence of turbulence. Theoutput is a 2D distorted image (Fig. 1[Bottom]). Render-ing boils down, thus, to creating a 2D distortion operation,which is consistent with distortion that turbulence can in-duce.
Random 3D turbulence eventually leads to a randomlydistorted 2D projection. The random distorted projectionmust be drawn from a distribution that is characterisedby a covariance function. The covariance function deter-mines how the distortion of any pixel is correlated to anyother pixel, and what the variance is. The covariance func-tion of turbulence-induced distortion is defined by physics.In other words, the physics of turbulence in 3D (a ran-dom process), and 5D refraction in it, dictate the image-distortion covariance function, in 2D. The probability dis-tribution of distortion had already been derived using thetheory of turbulence, for pairs of object points (not full-fieldimages) [7, 51]. This pair-wise function had also been veri-fied empirically, using field experiments [7], where correla-tions between image points were measured. We thus use thepair-wise covariance function of turbulence-induced imagedistortion, to create 2D distortion fields.
Transferring the physics-based pair-wise covariancefunction to a full distortion field is nontrivial in turbulence.Distortion is a vector-valued spatial field. The covarianceof this field is a matrix-valued function. It is a function ofrelative coordinates that vary for each pair of pixels, includ-ing the relative orientation of each pair, and the distortionorientation in each pixel. We derive theoretically the solu-tion: a full-field covariance of a 2D distortion field, basedon the physics-based pair-wise orientation-sensitive covari-ance. We then give a recipe how to render random 2D dis-tortion fields that satisfy the physical model. The recipe iscomposed of several simple 2D image operations.
2. Related Work
2.1. Phase Screen Propagation
A common method for simulating imaging through tur-bulence is based on light propagation through multiple ran-dom 2D phase screens [33], approximating a 3D turbu-lent medium. Phase-shifting layers are generated using ei-
ther fast Fourier transform (FFT), the Zernike polynomialmethod or the fractal interpolation method [59]. This ap-proach requires simulating a 3D refractive field and lightpropagation in 3D.
2.2. 3D Ray Tracing for Graphics
There are rendering techniques that trace rays through a3D randomly refractive media. Physically based simulationof atmospheric phenomena is done in [19, 20]. Renderingcomplicated lighting effects through various refractive ob-jects is presented in [26]. These 3D methods often requirespecialized hardware such as GPU and extensive computa-tion time.
2.3. 2D Image Distortion Simulation
2D image distortion is simpler to implement, and doesnot require extensive computational power. Usually, para-metric models are used. These models are based on analysisof real empirical distortions observed through various atmo-spheric conditions [15, 43]. Other methods, as in [14, 61],use simple Gaussian random functions to generate imagedistortion fields. The results resemble turbulence distortion.However, these methods do not use a physical model andare not set to have physically consistent spatial correlations.
3. Background3.1. Turbulence and Variance
3D turbulent flow is characterized by independent fluideddies in a range of scales. An eddy at the largest (outer)spatial scale L0 break into smaller eddies, which in turnbreak further. The process continues until the smallest ed-dies reach the inner scale of the turbulence, l0. The inertialrange is defined by [lo, Lo]. There, kinetic energy is trans-ferred to eddies of decreasing scale. Below l0, energy in thefluid motion dissipates fully into heat. The outer scale ofthe turbulence is often modelled [52] as L0 = 0.4h, whereh is the height above ground.
A turbulent flow is accompanied by pressure and tem-perature fluctuations, that affects the refractive index of themedium [4, 32]. In statistically homogeneous and locallyisotropic turbulence [30, 31] refractive fluctuations are char-acterized by a refractive index structure constant C2
n. It ex-presses turbulence strength [1, 53] and measured in unitsof m−2/3. A high value of C2
n indicates strong turbulence,while C2
n = 0 represents a medium free of turbulence. Inair, C2
n is typically the range 10−17− 10−13 m−2/3. A highvalue of 8× 10−13 is typical [1] of summer daytime. Sinceturbulence strength depends on the environment, in generalC2n can vary over the imaging path. In particular it is af-
fected by high pressure at low altitudes and strong winds athigh altitudes, both of which affect placement and perfor-mance of sky-observing telescopes. In this paper, we focus
Figure 2: Without turbulence, the line of sight to an object pointo is o. The point is projected then to p. Here L is the path length,f is focal length and D is the lens aperture diameter. Random re-fractions in the turbulent medium creates an angle of arrival (AA)perturbation, leading to a distorted projection to p + e(p).
on rather horizontal views, which are typical in applicationssuch as long distance observations.
A camera having focal length f observes an object pointo at distance L. Without turbulence, the line of sight is ex-pressed by unit vector o. As illustrated in Fig. 2, due torefractive index fluctuations, turbulence leads to random re-fraction of light emanating from the object point. Thus, thislight arrives at the camera at a random angle of arrival (AA),which is a perturbation around the direction of o. The AAperturbation distorts the projection of o at the image plane.
The AA variance for a spherical light wavefront1 is [10]:
σ2AA = γ(q)C2
nLD−1/3, (1)
where D is the diameter of the camera lens and q is theFresnel number:
q =D√λL
. (2)
Here λ is the light wavelength. The coefficient γ(q) for aspherical wave [10] is
γ(q) =
√3
16Γ
(1
6
)Γ
(8
3
)(2
β
) 13
[1 +
6
17
8
3·
Re
{(−π (βq)
2
2j
) 16
2F1
(1
6,
17
6;
23
6; 1 +
π (βq)2
2j
)}](3)
where β = 0.5216 and
Γ(µ) =
∫ ∞0
τµ−1e−τdτ (4)
is the Gamma function. In Eq. (3), 2F1() is the hypergeo-metric function
2F1(a, d; c; ζ) =
∞∑m=0
(a)m(d)m(c)m
ζm
m!. (5)
1We deal with light emanating from an object point. Light from dif-ferent object points is assumed to be incoherent. Hence, light from anyobserved object point is modelled as having a spherical wavefront.
Figure 3: Without turbulence, the lines of sight to two objectpoints o, o′ are respectively o,o′. The points then project to p andp + v, respectively.
For q � 1, Eq. (3) simplifies [12] to
γ(q) ≈ 1.09 . (6)
3.2. Pair-wise Distortion Correlation
Now, consider two object points, o, o′. Without turbu-lence, their corresponding lines of sight are defined by unitvectors o,o′, from the camera’s center of projection, as canbe seen in Fig. 3. These lines form a plane Π = o×o′. Theangle between them is
θ = arccos(o · o′) . (7)
Turbulence-induced random refractive disturbance causeseach of these lines of sight to deviate, correspondingly, byangles of arrival AA(o),AA(o′). There is correlation be-tween AA(o),AA(o′). The AA correlation function is
b(θ) =Cov[AA(o),AA(o′)]
σ2AA
, (8)
where Cov is the covariance operation. An expression forb(θ) is derived in Refs. [7] and [51]. The correlation gen-erally decreases with θ. Moreover, the correlation dependson the directions of the AAs, relative to Π. Suppose an AAperturbation is within the plane Π, which includes o,o′.Then, this perturbation is termed parallel and denoted ‖.If an AA perturbation is perpendicular to this plane, it isdenoted ⊥. In general, an AA perturbation at o has bothparallel and perpendicular components.
Note that for a fixed o, the ‖,⊥ axes and components aredefined depending on another point o′. Since object point ocoexists with a large number W of other points in variousrelative directions, there are O(W ) definitions of ‖,⊥ forany particular o. We deal, thus, with a non-trivial field.
Let us return, for the moment, to analyzing two partic-ular object points o, o′. Use the following dimensionlessvariables. From the object’s point of view, the angular sizeof the aperture is
θD =D
L. (9)
The inter-object view angle is normalized to
η =θ
θD. (10)
The outer scale of the turbulence is normalized to
ρ =L0
D. (11)
For the dimensionless parameter 0 ≤ ξ ≤ 1, define thefunctions
Q(ξ) = (1− ξ)5/3 , Mη = 2ηξ(1− ξ) . (12)
Following Ref. [7], the correlation function of each ofthe two components is
b‖,⊥(η) =∫ 1
0
{∫∞0H(κ)[J0(Mηκ)∓ J2(Mηκ)]dκ
}C2n(Lξ)Q(ξ)dξ
∫ 1
0
{∫∞0H(κ)dκ
}C2n(Lξ)Q(ξ)dξ
(13)Here, κ is a dimensionless integration variable included
in real non negative numbers, C2n is the refractive index
structure constant (see Sec. 3.1), J0 and J2 are Bessel func-tions of the first kind, while
H(κ) =[(2ρκ)2 + 1]−11/6J2
2 (κ)
κ. (14)
The negative sign in Eq. (13) corresponds to b‖(η), whilethe positive sign corresponds b⊥(η). Notice that the denom-inator of (13) is simply the numerator evaluated at η = 0,so that b‖,⊥(0) = 1.
In a horizontal path, for which C2n is constant, Eq. (13)
simplifies to
b‖,⊥(η) =
∫ 1
0
{∫∞0H(κ)[J0(Mηκ)∓ J2(Mηκ)]dκ
}Q(ξ)dξ
∫ 1
0
{∫∞0H(κ)dκ
}Q(ξ)dξ
(15)The correlation function (15) no longer depends on C2
n noron the range L, but depends on ρ and η. The correlationfunction (15) can be easily computed, as plotted in Fig. 4.
Figure 4: The correlation function (15) between a pair of pixels,for the two distortion components (‖,⊥), plotted for ρ = 4 andρ = 10.
Figure 5: The image plane. In absence of distortion, two objectpoints project to p and p + v. Turbulence induces projection dis-placements, e(p) and e(p + v), respectively. Any displacementvector e(p) can be divided to components parallel and perpendic-ular to v. These components are denoted e‖(p), e⊥(p).
4. Designing the Field Autocorrelation
4.1. Autocorrelation of a Stationary Vector Field
In this section, we work directly in the 2D image domain.We define the distortion vector field, and a matrix-valuedcovariance function in this field. Consider Fig. 5. Imagepixel location p is the undistorted projection of o. With-out turbulence, the line of sight o′ from object o′ projects to2D pixel location p + v. Turbulence displaces the respec-tive projections by the random vectors e(p) and e(p + v).Hence, e is a random vector field in the image plane. Allvectors, p,v, e are given in the (x, y) coordinates of the im-age domain. Specifically, the cartesian components of v arevx, vy . All spatial components, as well as the focal length fhave pixel-units. For narrow view angles,
θ ≈ |v|f
=r
f, (16)
where r = |v|. Thus, referring to Eqs. (10,15),
b‖,⊥(η) = b‖,⊥
(r
fθD
). (17)
To simulate a turbulence distorted image, we seek a ran-dom field e with correct spatial correlations. The displace-ment e is a spatially stationary vector field with mean zero.Since we are interested in the autocorrelation function ofthis field, we may assume without loss of generality thatthe variance of e is 1: the variance can be adjusted later asneeded, using Eq. (1). The autocorrelation is equal to theauto-covariance function and is given by the 2× 2 matrix
C(v) = E[e(p)eT(p + v)
]. (18)
Here E denotes expectation and T denotes transposition.Note that C(v) is a matrix-valued function of vx, vy . Bythe stationarity assumption, the autocorrelation function de-pends only on the difference v between two points and it issymmetric in the sense that C(−v) = CT(v).
We wish to find a correlation function (18) such that theinduced longitudinal and lateral autocorrelation functionsmatch Eq. (15). Define the unit vector
v = v/r. (19)
From Fig. 5, the longitudinal component of the displace-ment vector field e(p) is the projection of e(p) onto v:
e‖(p) = 〈v, e(p)〉 = vTe(p), (20)
where 〈., .〉 is the inner product operator. Let
R =
(0 −11 0
)(21)
denote a rotation matrix, which rotates v by 90◦ in the im-age plane. Thus
vTRv = 0. (22)
The perpendicular displacement e⊥(p) is the projection ofe(p) onto the perpendicular vector Rv:
e⊥(p) = 〈Rv, e(p)〉 = vTRTe(p). (23)
From Eq. (18), the autocorrelation scalar functions of theparallel and perpendicular displacements are, respectively
C‖(v) = E[e‖(p)eT‖(p + v)
]=
E[vTe(p) eT(p + v)v
]= vTC(v)v (24)
C⊥(v) = E[e⊥(p)eT⊥(p + v)
]= vTRTC(v)Rv. (25)
4.2. A Solution to the Autocorrelation Function
Given parallel and perpendicular autocorrelation scalarfunctions C‖(v) and C⊥(v), we seek a matrix-valued au-tocorrelation function C(v) that is consistent with Eq. (24)and Eq. (25), respectively. The solution to such a problemis not unique. Let us use the solution class:
C(v) = A(v)I−B(v)vvT, (26)
where A(v), B(v) are scalar functions and I is the 2 × 2identity matrix. In the class defined by Eq. (26), paralleland perpendicular displacements are uncorrelated. Indeed,using (26),
E[e‖(p)eT⊥(p + v)
]= E
[vTe(p)eT(p + v)Rv
]=
vTC(v)Rv = vTA(v)Rv − vTB(v)vvTRv = 0,(27)
where we have used Eq. (22) and the scalar-valued natureof the functions A(v) and B(v).
To determine the required functions A, B, we substituteEq. (26) into Eqs. (24) and (25), yielding
C‖(v) = A(v)vTv −B(v)(vTv)2 = A(v)−B(v) (28)
C⊥(v) = A(v)vTRTRv −B(v)vTRTvvTRv = A(v).(29)
Here we used RTR = I and Eq. (22). Eqs. (26,28,29) yield
C(v) = C⊥(v)I−[C⊥(v)− C‖(v)
]vvT . (30)
4.3. Consistency with Turbulence Correlation
Given Eqs. (10,15,17), the parallel and perpendicularscalar autocorrelation functions must be set to satisfy
C‖(v) = b‖
(r
fθD
), C⊥(v) = b⊥
(r
fθD
). (31)
Thus, the desired autocorrelation function can be written as
C(v) = A(r)I−B(r)vvT (32)
using the positive functions
A(r) = b⊥
(r
fθD
)B(r) = b⊥
(r
fθD
)− b‖
(r
fθD
).
(33)
In a 2D spatial image domain of size U ×U , the autocorre-lation domain v has size 2U × 2U .
We show an example using the following parameters:λ = 550nm, pixel size 4µm, U = 1024 pixels, NikonAF-S Nikkor lens, with f = 300mm and f# = 5.6,at height h = 4m. Object distance is L = 2km and
Cxx
Cyx
-0.1
0
0.1
Cxy
Cyy
0
0.5
1
Figure 6: The 2× 2 matrix-valued C(v), based on parameters ofthe example detailed in Sec. 4.3. In each sub-plot, the spatial di-mensions of the v domain are 2048× 2048 pixels, correspondingto an image of size 1024 × 1024. The top-right scale-bar corre-sponds to Cxy, Cyx, having negative as well as positive values.The bottom-right scale-bar corresponds to Cxx, Cyy , having non-negative values.
C2n = 3.6 · 10−13m−2/3. The AA variance and correla-
tion are then calculated using Eqs. (9-15). Figs. 6 and 7 ploteach element of the 2 × 2 matrix C(v), as a function of v.Using the same fixed imaging parameters, this example isfurther followed in this paper when creating Figs. 8,9,10,13,as explained next.
5. Creating a Distortion FieldIn this section, we derive a recipe for creating a random
distortion field e(p), whose auto-correlation is consistentwith C(v), given in Eqs. (32,33). Some of the expressionshere are matrix-valued generalizations of simple, textbookrelations of scalar-valued fields.
Let z(p) be a white noise vector field such that each ele-ment in it is a 2× 1 vector and E[z(p)zT(p + v)] = Iδ(v).A random vector field having an autocorrelation functionC(v) can be obtained by a convolution of z(p) with a de-terministic matrix-valued kernel K(p):
e(p) =
∫K(p− p′)z(p′)dp′. (34)
Here K(p) is a matrix-valued field: there is a 2 × 2 matrixK(p) for each vector p. The goal is to find a kernel K(p)such that the field e(p) has the desired autocorrelation func-tion C(v), given in Eqs (32,33).
Cxx
Cyx
-0.1
-0.05
0
0.05
0.1
Cxy
Cyy
0
0.5
1
Figure 7: A ×10 zoom-in on C(v) of Fig. 6.
The 2×2 matrix-valued autocorrelation function C(v) isrelated to the 2×2 matrix-valued convolution kernel K(p),through
C(v) =
∫K(p)KT(p + v) dp. (35)
Deriving K(p) based on Eq. (35) is a deconvolutionproblem. This deconvolution can be solved in the Fourierdomain. Let ω be a 2D spatial frequency vector. From (34),the spatial Fourier transform of the vector field e(p) is thevector field
e(ω) =
∫e(p)e−jω
Tpdp = K(ω)z(ω). (36)
On the other hand, the spatial Fourier transform of C(v) isthe matrix-valued spectral density function
C(ω) =
∫C(v)e−jω
Tvdv = E[e∗(ω)eT(ω)
], (37)
where ∗ denotes the complex conjugation. There is a 2× 2matrix C(ω) for each ω. Substituting Eq. (36) into Eq. (37):
C(ω) = E[K∗(ω)z∗(ω)zT(ω)KT(ω)
]= K∗(ω)E
[z∗(ω)zT(ω)
]KT(ω) = K∗(ω)KT(ω).
(38)
Eq. (38) uses the fact that the spectral density of any whitenoise field is E [z∗(ω)zT(ω)] = I, ∀ω. Eq. (38) is thus thespatial Fourier transform of (35).
Figure 8: The 2 × 2 matrix-valued K(ω), that corresponds tothe function C(v) shown in Figs. 6,7. It was calculated usingFFT, as described in Sec. 6. The plots here are a ×10 zoom-in onthe low-frequency domain. The top-right scale-bar corresponds toKxy, Kyx. The bottom-right scale-bar corresponds to Kxx, Kyy .
The autocorrelation function C(v) is real, symmetricas a matrix, symmetric with respect to sign changes of itsargument v and its diagonal entries are positive definitefunctions. As a result, its Fourier transform C(ω) is alsoreal, symmetric as a matrix, symmetric with respect to signchanges of its argument ω, and its diagonal entries are posi-tive functions. While there are many 2×2 matrices K satis-fying (38), it is natural to choose the symmetric square rootof C(ω), which enjoys the same properties. Given C(ω),its matrix square root can be computed per ω using the for-mula [34]:
K(ω) =1
t(ω)
[C(ω) + s(ω)I
]. (39)
Here s(ω) =√
det C(ω) and t(ω) =√
trC(ω) + 2s(ω).For example, Fig. 8 plots each element of the 2 × 2 ma-trix K(ω), as a function of ω, in correspondence with thefunction C(v) shown in Figs. 6,7.
To generate a random field e(p) with cross-spectraldensity C(ω), it is not actually necessary to find K(p),the inverse Fourier transform of K(ω). It suffices to dothe filtering in the Fourier domain, similarly to [50]. Thetheoretical recipe is thus:1 Calculate C(v) using Eqs (32,33).2 Fourier transform C(v) to C(ω).3 Calculate filter K(ω) using Eq.(39).4 Sample a random Gaussian white noise field z(p).
0 10 20 30 40 50 60 70 800
10
20
30
40
50
60
70
80
Figure 9: Part of a random distortion vector field e(p), derivedfrom the autocorrelation function shown in Figs. 6,7. This is a sim-ulated turbulence-induced distortion. For clarity, this plot zooms-in on a 80× 80 pixel sub-domain of the image
5 Fourier transform z(p) to z(ω).6 Use K(ω) in a simple multiplication per ω, as written inEq. (36), to obtain e(ω).7 Inverse Fourier transform e(ω), to obtain the distortionfield e(p), of variance 1.8 Amplify e(p) so its variance is consistent with Eq. (1).
For example, Fig. 9 plots a field e(p) created by thisrecipe, in correspondence with the autocorrelation and ker-nel functions shown in Figs. 6,7,8. The plot in Fig. 9 zoomson a 80× 80 pixel sub-domain of the image, for clarity.
6. Practical ConsiderationsIn practice, a computer does not perform a continuous
Fourier transform, but a discrete, fast Fourier transform(FFT). This has several implications. In a square grid, thespatial locations and frequencies are sampled. Using FFTas in steps 2,5 above induces a multiplicative scale rela-tive to a continuous-domain Fourier transform that is sam-pled. This factor is compensated for: in step 7 the field isamplified to have unit variance.
Moreover, FFT is cyclic: elements in one side of its inputaffect the other side. This wrap-around can cause problemsin functions that do not spatially decay fast enough. Asseen in Fig 4, b⊥ may decay very slowly. Despite havingvery low values at the periphery, wrap-around issues pro-duce negative oscillations in the Fourier domain that render
Table 1: Parameters of our examples
Example C2n
[m−2/3
]L [km] f [mm] f# h [m]
Fig. 10 3.6 · 10−13 2.0 300 5.6 4Fig. 11 3.6 · 10−13 1.3 380 5.6 4Fig. 12 3.6 · 10−13 2.0 200 3.2 20
the supposedly positive spectral density invalid. This canbe mitigated by extending the spatial domain, at a highercomputational cost. In practice, we feathered C(v) using aBlackman window that spans the domain of v.
7. Using the Distortion FieldA source turbulence-free image is graw(p), where p is
rectified on a regular grid. Due to turbulence, any pixel lo-cation p is displaced to p + e(p), creating the set of valueson a nonuniform grid
gnonuniform(p + e(p)) = graw(p) . (40)
The eventual distorted image gdistorted(p) is rendered by in-terpolating gnonuniform(p+e(p)) to the uniform pixel grid.For example, Fig. 10[Top] represents an undistorted view,graw(p). Since Fig. 10[Top] is in color, we first calculatedγ(q) for λ = 450, 550, 650nm. Practically, we saw thatγ(q) is insensitive to λ in visible light, thus a single randomdistortion field applies to all color channels. Based on theexample parameters that lead to Figs. 6,7,8,9, we rendereda distorted image, gdistorted(p), shown in Fig. 10[Bottom].
We show additional examples. They use most parametervalues as in the example detailed in Sec. 4.3. The parame-ters that were changed are detailed in Table 1. Correspond-ingly, the results are shown in Figs. 11 and 12. Specifi-cally, in Fig. 11, the focal length and object range changedto f = 380mm and L = 1.3km, respectively. The dis-tortion variance in pixel units is nearly the same, but thecorrelation-range doubles. Hence, distortions have lowerspatial frequencies, yet similar amplitude. The process wasthen run on a new random white noise field z(p) in step 4.
8. VerificationIt is possible to verify empirically that distortion statis-
tics are consistent with the theory. Let us render N distor-tion fields, {ei(p)}Ni=1, each based on a different sampledwhite noise field, in step 4 above. An empirical version ofEqs. (24,25) is
C‖(v) =1
N |Λ|
N∑i=1
∑p∈Λ
vTei(p)eTi (p + v)v
C⊥(v) =1
N |Λ|
N∑i=1
∑p∈Λ
vTRTei(p)eTi (p + v)Rv,
(41)
Figure 10: [Top] A raw source 1024× 1024 image, without ran-dom distortion. [Bottom] The image after undergoing distortionconsistent with turbulence. Part of the distortion field is shown inFig. 9. This distortion was derived from the autocorrelation func-tion shown in Figs. 6,7.
where Λ is a random subset of pixels.
Continuing with the example described in Figs. 6, 7, 8,we renderedN = 1500 distortion fields, as the one in Fig. 9.We used horizontal v vectors and |Λ| = 100. From Eqs.
Figure 11: [Top] A raw source 1024 × 1024 image, graw(p),without random distortion. [Bottom] gdistorted(p), the image afterundergoing distortion consistent with turbulence. The parametersare described in Sec. 4.3 and Table 1. Here 2D spatial correlationsof the distortion field have about twice the pixel-range, comparedto Fig. 10. Hence, the turbulence-distortion field is smoother here.
(10,16)
η =θ
θD=|v|fθD
. (42)
Using Eqs. (41,42), C‖(v), C⊥(v) are converted to b‖(η),
Figure 12: [Top] A raw source 1024 × 1024 image, graw(p).Photo courtesy of [http://pixabay.com]. [Bottom] gdistorted(p),the image after undergoing distortion consistent with turbulence.The parameters are described in Sec. 4.3 and Table 1.
b⊥(η). Figure 13 compares the empirical b‖, b⊥ to the the-oretical b‖, b⊥. From Fig. 13, generally, there is very goodconsistency between the theoretical AA correlation and ourrendered fields. For high |v|, i.e. η, the empirical correla-tion is a little smaller than the theoretical one. We believethis is because of the FFT wrap-around, whose repercus-
Figure 13: Plot of the empirical b‖(η) and b⊥(η), using N =1500 fields, |Λ| = 100 random pixels and setup parameters asdetailed in Sec. 4.3. The results are compared to the theoreticalcorrelations b‖(η) and b⊥(η).
sions are discussed in Sec. 6. A wrap-around has a higherdeterioration effect as a vector p + v is closer to the imageboundary, i.e. at large |v|.
9. Discussion of Generalizations
The recipe we give is by no means the only possible one.For example, Fourier domain processing can be bypassed:the kernel K(p) can be derived in Eq. (35) using other de-convolution methods, and then be applied in the spatial do-main (34). It should be beneficial to explore such variations.
This work focused on a particular turbulence-inducedimage effect: distortion in a 2D image, that views ascene via a stationary turbulent field. This approach maybe extended, to express other turbulence-induced imageeffects:
• Turbulence creates image blur [9, 21, 23]. As distor-tion, blur is a 2D operation on a 2D image. Similarlyto distortion, blur parameters form a vector, e.g. widthsof the blur kernel in two components. Moreover, theturbulence-induced blur-parameter field is random, havingcorrelation between adjacent points. There are closed-formphysical models for the blur that can result from turbulence.Hence, we believe that a similar recipe can be devised forrendering random 2D blur fields that are consistent withturbulence. The same hypothesis applies to turbulence-induced scintillation.
• Turbulence is dynamic, and so are its image effects. The2D distortion field is correlated both in 2D and in time. Torender videos, a physics-based spatiotemporal correlation
function should be incorporated.
• In some cases, the turbulence field is not spatially station-ary. Specifically, oblique views in steep angles as in as-tronomy pass through atmospheric layers having significantspatial variations of C2
n. Extending the approach to suchcases is important. It may be aided by assumptions that areoften used in astronomy, based on a layered structure.
Acknowledgments
We thank Johanan Erez, Ina Talmon, Dani Yagodin fortechnical support. Yoav Schechner is a Landau Fellow -supported by the Taub Foundation. The work was supportedby the Israel Science Foundation (ISF Grant 1467/12). Thiswork was conducted in the Ollendorff Minerva Center.Minerva is funded through the BMBF. Part of this work wasconducted while Armin Schwartzman and Marina Altermanwere with the Faculty of Electrical Engineering, Technion.
References[1] M. Alterman, Y. Y. Schechner, M. Vo and S. G. Narasimhan.
Passive tomography of turbulence strength. In Proc. ECCV,pages 47-60, 2014.
[2] M. Alterman, Y. Swirski and Y. Schechner. STELLAMARIS: Stellar marine refractive imaging sensor. In Proc.IEEE ICCP, 2014.
[3] D. Akkaynak, T. Treibitz, T. Shlesinger, R. Tamir, Y. Loya,and D. Iluz. What is the space of attenuation coefficients inunderwater computer vision? In Proc. CVPR, 2017.
[4] L. C. Andrews and R. L. Phillips. Laser Beam PropagationThrough Random Media. volume 52. SPIE press Belling-ham, WA, 2005.
[5] B. Atcheson, W. Heidrich and I. Ihrke. An evaluation of opti-cal flow algorithms for background oriented Schlieren imag-ing. Experiments in Fluids 46(3), 467–476, 2009.
[6] B. Atcheson, I. Ihrke, W. Heidrich, A. Tevs, D. Bradley,M. Magnor, H. P. Seidel. Time-resolved 3D capture of non-stationary gas flows. ACM TOG 27(5), 132:1–132:9, 2008.
[7] M. S. Belen’kii, J. M. Stewart, and P. Gillespie. Turbulence-induced edge image waviness: theory and experiment. Ap-plied Optics, 40(9):1321–1328, 2001.
[8] D. Berman, T. Treibitz and S. Avidan. Air-light estimationusing haze-lines. In Proc. ICCP, 2017.
[9] T. Bishop and P. Favaro. Blind deconvolution. In Encylopeiaof Computer Vision, pages 52–57. Springer, 2014.
[10] Y. Cheon and A. Muschinski. Closed-form approximationsfor the angle-of-arrival variance of plane and spherical wavespropagating through homogeneous and isotropic turbulence.JOSA A, 24(2):415–422, 2007.
[11] Y. Chuang, D. E. Zongker, J. Hindorff, B. Curless, D. H.Salesin, and R. Szeliski. Environment matting extensions:Towards higher accuracy and real-time capture. In ACM Sig-graph, pages 121–130, 2000.
[12] J. H. Churnside and R. J. Lataitis. Angle-of-arrival fluctua-tions of a reflected beam in atmospheric turbulence. JOSA A,4(7):1264–1272, 1987.
[13] S. Dong, T. Haist, W. Osten, T. Ruppel, and O. Sawodny. Re-sponse analysis of holography-based modal wavefront sen-sor. Applied Optics, 51(9):1318–1327, 2012.
[14] B. Fishbain, L. P. Yaroslavsky, and I. A. Ideses. Real-time stabilization of long range observation system turbulentvideo. Journal of Real-Time Image Processing, 2(1):11–22,2007.
[15] G. Potvin. J. L. Forand and D. Dion. A parametric modelfor simulating turbulence effects on imaging systems. De-fense Research and Development Canada (DRDC) Val-cartier Technical Report 2006-787, 2007.
[16] I. Gkioulekas, S. Zhao, K. Bala, T. Zickler, and A. Levin.Inverse volume rendering with material dictionaries. ACMTOG, 32(6):162, 2013.
[17] E. Grossman, R. Tzioni, A. Gur, E. Gur, and Z. Zalevsky.Optical through-turbulence imaging configuration: experi-mental validation. Optics Letters, 35(4):453–455, 2010.
[18] M. Gupta and S. G. Narasimhan. Legendre fluids: a uni-fied framework for analytic reduced space modeling andrendering of participating media. In Proc. ACM SIG-GRAPH/Eurographics Symposium on Computer Animation,pages 17–25, 2007.
[19] D. Gutierrez, A. Munoz, O. Anson, and F. J. Seron. Non-linear volume photon mapping. In Rendering Techniques,pages 291–300, 2005.
[20] D. Gutierrez, F. J. Seron, A. Munoz, and O. Anson. Simu-lation of atmospheric phenomena. Computers & Graphics,30(6):994–1010, 2006.
[21] S. Harmeling, M. Hirsch, S. Sra, and B. Scholkopf. On-line blind deconvolution for astronomy. In Proc. IEEE ICCP,2009.
[22] F. Heide, L. Xiao, A. Kolb, M. B. Hullin, and W. Hei-drich. Imaging in scattering media using correlation imagesensors and sparse convolutional coding. Optics Express,22(21):26338–26350, 2014.
[23] J. Holloway, S. Asif, M. Sharma, N. Matsuda,R. Horstmeyer, O. Cossairt, and A. Veeraraghavan.Toward long distance, sub-diffraction imaging using co-herent camera arrays. IEEE T. Computational Photography2(3):251 - 265, 2016.
[24] R. Horisaki, R. Takagi, and J. Tanida. Learning-based imag-ing through scattering media. Optics Express, 24(13):13738–13743, 2016.
[25] M. B. Hullin, M. Fuchs, I. Ihrke, H.-P. Seidel, and H. P.Lensch. Fluorescent immersion range scanning. ACM TOG,27(3):87, 2008.
[26] I. Ihrke, G. Ziegler, A. Tevs, C. Theobalt, M. Magnor, andH. P. Seidel. Eikonal rendering: Efficient light transport inrefractive objects. In ACM TOG, 26(3):59,2007.
[27] Y. Ji, J. Ye, and J. Yu. Reconstructing gas flows using light-path approximation. In Proc. IEEE CVPR, pages 2507–2514,2013.
[28] N. Joshi, M. F. Cohen. Seeing Mt. Rainier: Lucky imagingfor multi-image denoising, sharpening, and haze removal. InProc. IEEE ICCP, 2010.
[29] J. Kim, D. Lanman, Y. Mukaigawa, and R. Raskar. Descat-tering transmission via angular filtering. In Proc. ECCV,pages 86–99, 2010.
[30] A. N. Kolmogorov. Dissipation of energy in locally isotropicturbulence. In Dokl. Akad. Nauk SSSR, volume 32, pages16–18. JSTOR, 1941.
[31] A. N. Kolmogorov. The local structure of turbulence in in-compressible viscous fluid for very large reynolds numbers.In Dokl. Akad. Nauk SSSR, volume 30, pages 301–305. JS-TOR, 1941.
[32] N. S. Kopeika, A System Engineering Approach to Imaging.SPIE Press, 1998.
[33] R. G. Lane, A. Glindemann, and L. C. Dainty. Simulationof a Kolmogorov phase screen. Waves in Random media,2(3):209–224, 1992.
[34] B. W. Levinger. The square root of a 2× 2 matrix. Mathe-matics Magazine, 53(4):222–224, 1980.
[35] A. Levis, Y. Y. Schechner and A. B. Davis. Multiple-scattering microphysics tomography. In Proc. IEEE CVPR,2017.
[36] H. Y. Liu, E. Jonas, L. Tian, J. Zhong, B. Recht, andL. Waller. 3D imaging in volumetric scattering media usingphase-space measurements. Optics Express, 23(11):14461–14471, 2015.
[37] C. Ma, X. Lin, J. Suo, Q. Dai, and G. Wetzstein. Transpar-ent object reconstruction via coded transport of intensity. InProc. IEEE CVPR 3238–3245, 2014.
[38] H. Ma, Z. Y, X. Wang, Y. Ma, P. Zhou, X. Xu, and Z. Liu.Adaptive conversion of a wavefront-distortion beam to near-diffraction-limited flattop beam based on stochastic parallelgradient descent algorithm. In Proc. SPIE Photonics Asia2010, pages 78430P–78430P, 2010
[39] M. Mendillo, S. Laurent, J. Wilson, J. Baumgardner, J. Kon-rad, and W. C. Karl. The sources of sodium escaping fromio revealed by spectral high definition imaging. Nature,448(7151):330–332, 2007.
[40] D. Miyazaki, M. Saito, Y. Sato, and K. Ikeuchi. Determiningsurface orientations of transparent objects based on polariza-tion degrees in visible and infrared wavelengths. JOSA A,19(4):687–694, 2002.
[41] S. G. Narasimhan, S. K. Nayar, B. Sun, and S. J. Koppal.Structured light in scattering media. In Proc. IEEE ICCV,pages 420–427, 2005.
[42] M. O’Toole, J. Mather, and K. N. Kutulakos. 3D shape andindirect appearance by structured light transport. In Proc.IEEE CVPR, pages 3246–3253, 2014.
[43] E. Repasi and R. Weiss. Computer simulation of im-age degradations by atmospheric turbulence for horizontalviews. In Proc. SPIE Defense, Security, and Sensing, pages80140U–80140U, 2011.
[44] M.C., Roggemann, B. Welsh, Imaging Through Turbulence,CRC Press, 1996.
[45] A. Sedlazeck, K. Koser, R. Koch, 3D reconstruction basedon underwater video from ROV Kiel 6000 considering un-derwater imaging conditions. In Proc. IEEE OCEANS EU-ROPE, 2009.
[46] M. Sheinin and Y. Y. Schechner. The next best underwaterview. In Proc. IEEE CVPR, 2016.
[47] F. J. Seron, D. Gutierrez, G. Gutierrez, and E. Cerezo. Imple-mentation of a method of curved ray tracing for inhomoge-neous atmospheres. Computers & Graphics, 29(1):95–108,2005.
[48] M. Shimizu, S. Yoshimura, M. Tanaka, M. Okutomi, Super-resolution from image sequence under influence of hot-airoptical turbulence. In Proc. IEEE CVPR, 2008.
[49] O. Spier, T. Treibitz and G. Gilboa. In situ target-less cali-bration of turbid media. In Proc. IEEE ICCP, 2017.
[50] J. Stam and E. Fiume. Turbulent wind fields for gaseousphenomena. ACM Siggraph pages 369–376, 1993.
[51] J. Stone, P. Hu, S. Mills, and S. Ma. Anisoplanatic effectsin finite-aperture optical systems. JOSA A, 11(1):347–357,1994.
[52] V. I. Tatarskii. Wave Propagation in a Turbulent Medium.McGraw-Hill, 1961.
[53] Y. Tian, S. Narasimhan and A. Vannevel. Depth from opticalturbulence. In Proc. IEEE CVPR, 2012.
[54] B. Trifonov, D. Bradley, W, Heidrich. Tomographic recon-struction of transparent objects. In Proc. Eurographics Sym-posium on Rendering, 51–60, 2006.
[55] A. T. Watnik and P. S. Lebow. Weak-signal iterative holog-raphy. Applied Optics, 54(10):2615–2619, 2015.
[56] G. Wetzstein, D. Roodnick, W. Heidrich, and R. Raskar.Refractive shape from light field distortion. In Proc. IEEEICCV, pages 1180–1186, 2011.
[57] T. Xue, M. Rubinstein, N. Wadhwa, A. Levin, F. Durand,and W. T. Freeman. Refraction wiggles for measuring fluiddepth and velocity from video. In Proc. ECCV, pages 767–782, 2014.
[58] S. K. Yeung, C. K. Tang, M. S. Brown, and S. B. Kang. Mat-ting and compositing of transparent and refractive objects.ACM TOG, 30(1):2, 2011.
[59] Z. Yu, T. Qiuyan, W. Jin, and S. Quan. Numerical simula-tor of atmospherically distorted phase screen for multibeamtime-dependent scenario. Applied Optics, 53(22):5008–5015, 2014.
[60] X. Zhu and P. Milanfar. Stabilizing and deblurring atmo-spheric turbulence. In Proc. IEEE ICCP, 2011.
[61] X. Zhu and P. Milanfar. Removing atmospheric turbu-lence via space-invariant deconvolution. IEEE T. PAMI,35(1):157–170, 2013.
Table of notationsA A coefficient function of the autocorrelation matrixa Parameter of 2F1()AA Angle of arrivalb AA correlationb⊥ b component perpendicular to plane Πb‖ b component parallel to plane Π
b⊥ Empirical b⊥b‖ Empirical b‖B A coefficient function of the autocorrelation matrixβ A coefficient that equals 0.5216C Matrix valued autocorrelation functionC Spatial Fourier transform of CCxx Correlation of the horizontal components of field eCxy Correlation of the horizontal and vertical compo-
nents of field eCyx Correlation of the vertical and horizontal compo-
nents of field eCyy Correlation of the vertical components of field ec Parameter of 2F1()C2n Turbulence structure constantCov The covariance operationD Diameter of the camera lens.d Parameter of 2F1()e A random vector fielde⊥ Field component perpendicular to plane Πe‖ Field component parallel to plane Πe Spatial Fourier transform of eE Expectationη Normalized inter-object view anglef Focal length of the camera2F1() Hypergeometric functiongraw Source turbulence-free imagegdistorted Image distorted by turbulencegnonuniform graw on nonuniform grid, created by turbulenceγ AA variance coefficientΓ Gamma functionh Height above groundH An integrated function to derive b⊥ and b‖i Index of distortion fieldI Identity matrixj
√−1
J0, J2 Bessel functions of the first kindK Matrix valued kernel fieldK Spatial Fourier transform of K
Kxx First row and first column component of K
Kxy First row and second column component of K
Kyx Second row and first column component of K
Kyy Second row and second column component of Kκ Integration parameterl0 Turbulence inner scaleL0 Turbulence outer scaleL Distance between the camera and the object
Continuation of table of notationsλ Light wavelengthΛ Random subset of pixelsMη An integrated function to derive b⊥ and b‖m Summation index in 2F1()µ Parameter of ΓN Number of distortion fieldso, o′ object pointso,o′ Unit vectors of lines of sight to o, o′, respectivelyω 2D spatial frequency vectorp Pixel location vectorΠ A plane defined by o,o′
q Fresnel numberQ An integrated function to derive b⊥ and b‖r The length |v|R Rotation matrixρ Normalized outer scales Frequency domain scalar functionσ2AA AA variancet Frequency domain scalar functionT Transposition operationθ Inter-object view angleθD Angular size of the aperture, from the object’s point of
viewU Spatial image sizev Vector between pixels to which o,o′ are projected with-
out distortionW Number of object points(x, y) Cartesian coordinatesξ Dimensionless parameterz White noise vector fieldζ Parameter of 2F1()z Spatial Fourier transform of z