A System forHigh-Volume Acquisition and Matching of Fresco Fragments

Reassembling Theran Wall Paintings

Benedict Brown1,2, Corey Toler-Franklin1, Diego Nehab1,3,Michael Burns1, Andreas Vlachopoulos4, Christos Doumas4,5,

David Dobkin1, Szymon Rusinkiewicz1, Tim Weyrich1,6

1Princeton University3Microsoft Research5National University of Athens2Katholieke Universiteit Leuven

4Akrotiri Excavations, Thera6University College, London

Bronze Age Thera• Modern day Santorini• Aegean civilization: c. 1700 BC• Traded with other

Mediterranean civilizations • Evidence of fishing, agriculture,

and livestock

• Volcanic eruption c. 1650 BC

NASA Visible Earth

Akrotiri• Major archaeological

excavation since 1967• Well-preserved by ash• Most significant find:

plaster wall paintings– Pigments excellently

preserved

Thera Foundation

Akrotiri• Major archaeological

excavation since 1967• Well-preserved by ash• Most significant find:

plaster wall paintings– Pigments excellently

preserved

– But shattered in pieces by earthquake

The Akrotiri Jigsaw• Current assembly

process is laborious

The Akrotiri Jigsaw• Current assembly

process is laborious• Enough work for

another century

Fragment Characteristics• Conservators consider:

– size, thickness

– level of erosion

– discoloration and fading

– set of pigments

– curvature / flatness

– texture of the back

– string impressions

Constrained 3-D Acquisition Protocol• Automatic turntable control• Acquire scans at 45°• Two 360° scan sequences

– Face-down: front face at known plane

– Face-up: front face visible

Color and Normals: 2-D Acquisition• Custom scan software

– One-click acquisition

– Preview scan locates fragment

• Five scans– Four front orientations

(photometric normals)

– One back orientation

Scan Alignment with Multi-Way ICP• Align fragments scanned on turntable

– Axis of rotation gives initial guess

– Standard algorithm to improve alignments:Iterative Closest Points [BESL 1992], [CHEN 1992]

• Flat front surfaces lead to instability• Improved algorithm: Multi-way ICP

– Constrain all scan-to-scantransformations to be identical

– Equivalent to solving fora single rotation axis

Front/Back Alignment• Flipping fragment is

uncalibrated• Little overlap between

front and back scans• Front/back alignment is

vertically unstable

Front/Back Alignment• Use front face to determine

vertical alignment– Visible in front scans

– On (calibrated) turntablesurface in back scans

• Initial guess and ICP forwithin-plane alignment

2-D/3-D Alignment• Flatbed scanner has superior color• Can’t use calibration [LEVOY 2000], reliable silhouette

[LENSCH 2000], or features [LIU 2006] [CHEN 2007]

• Use image alignment: PCA + downhill simplex

Projected 3-D Color Flatbed Scan

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Ribbon Matching• Try all possible alignments• Update alignment

incrementally• Regular edge

parameterization:similar to image correlation

Fragment MatchingICP Matching

– Nearest neighbor correspondence search

– Iterate to find matches

– 45 seconds per fragment pair

Ribbon Matching

– Regular edge sampling for correspondences

– Exhaustive search with incremental update

– 2 seconds per pair

Original (irregular) mesh Resampled ribbon

Erosion Detection• Erosion causes incorrect alignments• Detected on ribbons with normal constraint

Fragment Back

Fragment Front

No Erosion Detection

Erosion Detection• Erosion causes incorrect alignments• Detected on ribbons with normal constraint

Fragment Back

Fragment Front

No Erosion Detection With Erosion Detection

Outline• System design• Processing pipeline• Matching• Results

Ribbon Matching Results

Synthetic Fresco25 mm strip width12.5 mm strip width50 mm strip width

Future Work (Matching)• Multi-cue matching• Improved ribbon matching/

Handling gaps– Dynamic programming can probably

handle gaps

– Record all possible alignments instead of only best candidates to do saliency analysis

• Global matching– Fuse matched fragments and re-match

– Do global consistency checks on networks of matches

Future Work (Scanners)We want to scan:• large fragments• assembled edges?• edge and back normals

Approach:• Hand-held scanner• Two cameras and a projector/fixed pattern• Alignment similar to in-hand scanner• Should be able to get normals from mutiple views

Future Work (Scanners)We want to scan:• large fragments• assembled edges?• edge and back normals

Approach:• Hand-held scanner• Two cameras and a projector/fixed pattern• Alignment similar to in-hand scanner• Should be able to get normals from mutiple views

Acknowledgments• Princeton University: Tom Funkhouser, Dimitris Gondicas,

Matt Plough, Phil Shilane, Xiaojuan Ma• Akrotiri Excavation, Laboratory of Wall Paintings:

Manolis Hamaoui, Litsa Kalambouki, Marina Papapetrou, Panagiotis Vlachos, Alexandros Zokos, Iakovos Michailidis, Fragoula Georma, Niki Spanou

• Special thanks to David Koller (University of Viriginia),Misha Kazhdan (Johns Hopkins University), and Peter Nomikos Jr.

• Funding: Thera Foundation, Kress Foundation,Seeger Foundation, Cotsen Family Foundation, andNSF Grants CCF-0347427 and CCF-0702580