streaming multigrid for gradient-domain operations on large images
Post on 24-Feb-2016
72 Views
Preview:
DESCRIPTION
TRANSCRIPT
Streaming Multigridfor Gradient-Domain Operations
on Large Images
Johns Hopkins University Misha Kazhdan
Microsoft ResearchHugues Hoppeand
Stitching Image PanoramasDifferent exposures Seams in the panorama
Courtesy of Uyttendaele
Gradient Domain Fix:Set seam-crossing gradients to zero.
High Dynamic Range CompressionGradient Domain Fix:Amplify small gradients and dampen large ones.
Medium ExposureShort ExposureCourtesy of Industrial Light & Magic
Long Exposure
No single image exposes all the detail
Tone-Mapped Image
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.
Removing Lighting EffectsRange CompressionImage CompositingImage StitchingImage Authoring
[Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05][Fattal ’02, Weyrich ’07][Perez ’03, Agarwala ’04, Jia ’06][Levin ’04, Agarwala ’07][McCann ’08]
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 SolverGrad
Removing Lighting EffectsRange CompressionImage CompositingImage StitchingImage Authoring
[Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05][Fattal ’02, Weyrich ’07][Perez ’03, Agarwala ’04, Jia ’06][Levin ’04, Agarwala ’07][McCann ’08]
Outline
• Introduction• Solving for the Image– The Poisson Equation– Solving the Linear System
• Choosing the System• Solving for Big Images• Results and Discussion
The Poisson EquationGradient-domain processing requires fitting an image to local-difference constraints:
1
gu
fu
g uf=div(g)
)(div)(div gu
1
averaged^
Gauss-Seidel SolversGradient-domain processing requires fitting an image to local-difference constraints.
Iterative Algorithm:
For each pixel (i,j)1. Assume the rest of the pixel values are correct2. Update u[i][j] to satisfy the Laplacian:
u[i][j] (Average of Neighbors) + f[i][j]
gu
fu
averaged^
Gauss-Seidel Solvers
This quickly solves for the high frequencies, but the low frequencies converge slowly.
Desired Solution
10 Iterations 100 Iterations10 IterationResidual
100 IterationResidual
2 Iterations2 IterationResidual
Multigrid SolversV-Cycle Algorithm:The process is recursed for the low-res problem.
DesiredBest-Guess
Iterated
Residual
1. Perform a few updates on the high-res image2. Compute the residual constraints
Desired
3. Down-sample, solve, and up-sample
SolvedUp-Sampled
Best-Guess
4. Sum the low-res and high-res results
Iterated
5. Refine the high-res image
+
-
Outline
• Introduction• Solving for the Image• Choosing the System• Solving for Big Images• Results and Discussion
Choosing the SystemThe choice depends on the interpretation of pixels as elements of a continuous function.
Pixel values 0th-order interpretation1st-order interpretation2nd-order interpretation3rd-order interpretation
Elements
Choosing the SystemImplementing Up-/Down-Sampling:The nesting of the elements determines the transition between the levels of the solver.
0th-order elements 1st-order elements 2nd-order elements 3rd-order elements1 1
1/2
1
1/21/4
3/4 3/4
1/4 1/81/2
3/41/2
1/8
Choosing the SystemDefining the Laplacian:The neighborhood of the Laplacian is defined by the element overlap.
Undefined
0th-order elements 1st-order elements 2nd-order elements 3rd-order elements
1-Ring Average 2-Ring Average 3-Ring Average
Second-Order ElementsWhich order elementconverges fastest to anaccurate solution?
Original Image Image Gradients
1st-order error
2nd-order error
3rd-order error0 5
1st-Order2nd-Order3rd Order
256
1
1/256
Average Error
GSUpdates
Second-Order Elements
Spectral Analysis of Convergence:The eigenvalues of the multigrid operator determine how efficiently the residual is dampened.
0 320.0
0.2
0.4
Two-Grid Spectra (k=5)
Eigenvalue Index
Eigenvalue
1st-order2nd-order3rd-order
G.S Update
G.S Update
G.S Update
Two-Grid Operator
Down-Sample
Up-Sample
Outline
• Introduction• Solving for the Image• Choosing the System• Solving for Big Images– Naïve Out-of-Core Multigrid– Temporal Blocking– Interleaved Multilevel Streaming
• Results and Discussion
Naïve Out-of-Core MultigridComponents of the solver:
1. Gauss-Seidel updates2. Down-sampling3. Up-sampling
The multigrid solvercan be streamed byadvancing a fixed-size window.
Local data access
On Disk
In Memory
Naïve Out-of-Core MultigridThe large linear system can be solved by streaming each step of the V-cycle.
Low-Res Solve
Down-Sample
Down-Sample
Up-Sample
Up-Sample
G.S.Update
Solution
G.S. Update
G.S. Update
k k
kk
G.S. Update
Constraints
Disk Streaming
k Gauss-Seidel Updates
2(k+2) Streaming Passes
Residual
Residual
Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.neighbor
Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.
000
0 1
0
0 1
0
0 1
1 2
0
0 1
1 2
0
0 1
1 2
2
neighbor
Win
dow
k=2 Gauss-Seidel Updates
2nd-order elements
Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.
SolutionConstraints
neighbor
Low-Res Solve
Down-Sample
Down-Sample
Up-Sample
Up-Sample
G.S.Update
G.S. Update
G.S. Update
G.S. Update
Residual
Residual
Disk Streaming
Interleaved Multilevel StreamingProcessing at one level is buffered for the next.
piped into
Processing at one level is buffered for the next.
1/4
3/4 3/4
1/4
¾¾¼
¼
Interleaved Multilevel Streaming
000
0 1
0
0 1
0
0 1
1 2
0
0 1
1 2
0
0 1
1 2
0
2
0 1
1 2
0
0 1
1 2
2
0
0 1
1 2
0
2
0 1
1 2
00
0 11 2
0 1
1 2
20000 1
¾¾¼
¼¾¾¼
¼
Resid
ual
High-Resolution Low-Resolution
2nd-order elements
piped into
Interleaved Multilevel StreamingProcessing at one level is buffered for the next.
SolutionConstraints
Low-Res Solve
Down-Sample
Down-Sample
Up-Sample
Up-Sample
G.S.Update
G.S. Update
G.S. Update
G.S. Update
Residual
Residual
Streaming Pass 1 Streaming Pass 2
Globality of the Poisson equation
Two streaming passes is optimal
Disk StreamingMemory Streaming
Disk Streaming
piped into
For gradient-domain operators that are local, processing only takes two streaming passes.
+1 streaming pass for each additional V-cycle.
Streaming through the Gradient-Domain
Pass 3uu
Update
Updatef
Solve
Update…
u0
u1
g0
g1 g
un gn
Op…
Update
…
Memory StreamingDisk Streaming
Pass 1 Pass 2
Solve
Update Update
UpdateUpdate
Outline
• Introduction• Solving for the Image• Choosing the System• Solving for Big Images• Results and Discussion
Stitching Big Images
Courtesy of Agarwala
Agarwala ’07:A high-res solution is only required near the seams, so solve over an adapted quadtree.
Stitching Results
Courtesy of Agarwala and Curless
Beynac: 6,0461,920 Rainier: 10,9602,096
Edinburgh: 16,9502,956
Redrock: 19,5884,457
Dataset Pixels ImagesError [0,256) Memory (MB) Time (seconds)
Quadtree Streaming Quadtree Streaming Quadtree Streaming
Beynac 12x106 3 0.01 0.01 16 190 8 17
Rainier 23x106 5 0.02 0.01 17 110 14 33
Edinburgh 50x106 25 0.01 123 203 122 79
Redrock 87x106 9 0.01 112 133 118 118
Tone-Mapping Results
Bring out subtle details by amplifying small gradients and dampening large ones.
Large Image ResultsSt James:• Stitched from 643 photographs• Contains 3.3 billion (88,309 x 37,842) pixels
Tone-MappedMemory: 224 MBTime: 45:10Max Error [0,256): 0.5 Courtesy of Uyttendaele
StitchedMemory: 408 MBTime: 1:27:50Max Error [0,256): 0.15
Large Image Results
Courtesy of Uyttendaele
St James:• Peak Memory: 408 MB• Solver Time: 1:27:50
Total disk I/O:• Down-Sampling
– Read in the gradient constraints: 40 GB– Write out solution and Laplacian: 53 GB
• Up-Sampling– Read in solution and Laplacian: 53 GB– Write out the image: 20 GB
53GB in-core1:19:02 I/O Time
Summary
Streaming multigrid for big image processing:– 2nd order elements– Temporal blocking– Interleaved streaming
Issues addressed in the paper:– Integration with finite-differences– Setting the average color–Non power-of-two images
Accurate solution in two streaming passes
Issues for Future Work
1. Non-trivial boundaries and constraints?e.g. cut-and-paste, matting
2. Adaptive weighting?e.g. local tone adjustment
3. Extensions to higher dimensions?for incompressible fluid simulation
4. Less I/O in the streaming passes?on-the-fly compression
5. Really big images?parallelized and distributed solvers
AcknowledgmentsDatasets:
Aseem AgarwalaBrian CurlessMatt Uyttendaele
Discussions and Ideas:Ketan Dalal Rick SzeliskiBill Bolosky Ann WolvertonCAVGRAPH Reviewers SIGGRAPH Reviewers
Code & Images @ http://www.cs.jhu.edu/~misha/Code/SMG
Thank You!
General Temporal BlockingTo perform k updates:
Updating the 3rd row from front,The row two behind that,The row two behind that,
000
0 1
0
0 1
0
0 1
1 2
0
0 1
1 2
0
0 1
1 2
2 33
k=3 Gauss-Seidel Updates
00
0 1
1 2
2 3
0 1
1 2
2 3
k
Win
dow
…
General Temporal BlockingTo perform k updates:
Updating the 3rd row from front,The row two behind that,The row two behind that,…
k=3 Gauss-Seidel Updates
k
with d-th order elements^
3
00
0 1
1 2
2 3
0 1
1 2
2 3
Win
dow
(d+1)-stdd
Finite Differences to Finite Elements
The derivative of a d-th order element is the difference of elements of order d-1.
Finite Differences
1st-order Element
0th-order Elements
Coefficients of Derivative ElementsDifferentiation
Second-Order ElementsWhich order elementconverges fastest to anaccurate solution?
Original Image Image Gradients
1st-order error
2nd-order error
3rd-order error0 5 10 15 20
1st-Order2nd-Order3rd-Order
GS Updates
256
1
1/256
Average Error256
Average Error
0 1000
1st-Order2nd-Order3rd-Order
Operations
256Average Error256
1
1/256
Average Error
1/2561/256
top related