Principles of 30 Image Analysis and Synthesis

THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

PRINCIPLES OF 3D IMAGE ANALYSIS AND SYNTHESIS

EDITED BY

BERND GIROD Information Systems Laboratory Stanford University Stanford, CA, USA

GUNTHER GREINER Lehrstuhl fur Graphische Datenverarbeitung Universitat Erlangen-Nurnberg Erlangen, Germany

HEINRICH NIEMANN Lehrstuhl fOr Mustererkennung Universitat Erlangen-Nurnberg Erlangen, Germany

Springer Science+Business Media, LLC

Library of Congress Cataloging-in-Publication Data

Principles of 3D image analysis and synthesis I edited by Bernd Girod, GUnther Greiner, Heinrich Niemann.

p. em - (The Kluwer international series in engineering and computer science ; SECS 556)

Includes bibliographical references and index.

I. Image processing-digital techniques. 2. Image analysis. I. Girod, Bernd. II. Greiner, GUnther. III. Niemann, Heinrich. IV. Series.

TA1637.P75 2000 621.36'7-dc21

Copyright © 2002 by Springer Science+ Business Media New York

Originally published by Kluwer Academic Publishers in 2002

00-029626

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work.

Permissions for books published in Europe: permissions@wkap.nl Permissions for books published in the United States of America: permissions@wkap.com

Printed on acid-free paper.

ISBN 978-1-4419-4982-0 ISBN 978-1-4757-3186-6 (eBook) DOI 10.1007/978-1-4757-3186-6

Contents

Contributing Authors

Preface

1. OPTICAL 3D SENSORS

1.1 2D Image Acquisition for 'Perfect' 3D Sensors

1.2 3D Sensors-Principles, Potentials and Limitations

2. MULTIPLE VIEWS AND IMAGE SEQUENCES

2.1 Coordinate Systems and Camera Models

2.2 Motion Models

2.3 Structure From Multiple Views

2.4 Object Tracking in Image Sequences

2.5 Illumination Estimation

3. RECOGNITION AND INTERPRETATION

3.1 Segmentation

3.2 Semantic Models

3.3 Statistical Models

3.4 Object Recognition

3.5 Image Understanding

3.6 Active Vision

4. REPRESENTATION AND PROCESSING OF SURFACE DATA

4.1 Polygon Meshes

4.2 Feature Extraction and Registration

4.3 Discrete Modeling of Point Clouds

4.4 Fusion of Discrete Models

4.5 Splines

4.6 Fitting Free Form Surfaces

vii

xi

2

5

25

26

30

38

57

67

79

80

93

101

108

122

131

141

142

153

166

175

181

192

vi PRINCIPLES OF 3D IMAGE ANALYSIS AND SYNTHESIS

5. REALISTIC RENDERING

5.1 Models for Illumination and Reflection

5.2 Global Illumination

5.3 Texture Models

5.4 Image-Based Rendering

6. VOLUME VISUALIZATION

6.1 Volume Data

6.2 Volume Models

6.3 Data Acquisition

6.4 Visualization of Scalar Volume Data

6.5 Visualization of Vector Fields

7. ACOUSTIC IMAGING, RENDERING, AND LOCALIZATION

7.1 3D Acoustical Imaging

7.2 Aurealization

7.3 Fusion of Multisensor Data

7.4 Object Localization Using Audio and Video Signals

8. SELECTED APPLICATIONS

8.1 Digitizing 3D Objects for Reverse Engineering and Virtual Reality

8.2 Surface Interrogation in Automotive Design

8.3 Fluid Dynamics

8.4 Diagnosis Support of Patients with Facial Paresis

8.5 Surgical Planning

8.6 Image Communication

References

Index

203

204

213

223

232

243

244

245

252

261

270

279

280

295

309

322

335

336

347

353

365

378

389

399

457

Contributing Authors

Michael Breuer Lehrstuhl fiir Stromungsmechanik, Universitat Erlangen-Ni.irnberg

Swen Campagna Lehrstuhl fi.ir Graphische Datenverarbeitung, Universitat Erlangen-Ni.irnberg

Katja Daubert AG 4 Computergraphik, Max-Planck-Institut fi.ir Informatik, Saarbri.icken

Joachim Denzler Lehrstuhl fi.ir Mustererkennung, Universitat Erlangen-Ni.irnberg

Peter Eisert Lehrstuhl fiir Nachrichtentechnik, Universitat Erlangen-Ni.irnberg

Thomas Ertl Visualisierung und Interaktive Systeme, Universitat Stuttgart

Arnd Gebhard Lehrstuhl fi.ir Mustererkennung, Universitat Erlangen-Ni.irnberg

Bernd Girod Information Systems Laboratory, Stanford University

Sabine Girod Department of Functional Restoration, Stanford University School of Medicine

Gunther Greiner Lehrstuhl fi.ir Graphische Datenverarbeitung, Universitat Erlangen-Ni.irnberg

Peter Hastreiter Lehrstuhl fi.ir Graphische Datenverarbeitung, Universitat Erlangen-Ni.irnberg

viii PRINCIPLES OF 30 IMAGE ANALYSIS AND SYNTHESIS

Gerd Hausler Lehrstuhl fi.ir Optik, Universitat Erlangen-Ni.imberg

Wolfgang Heidrich AG 4 Computergraphik, Max-Pianck-Institut fi.ir Informatik, Saarbri.icken

Benno Heigl Lehrstuhl fi.ir Mustererkennung, Universitat Erlangen-Ni.imberg

Matthias Hopf Visualisierung und Interaktive Systeme, Universitat Stuttgart

KaiHormann Lehrstuhl fi.ir Graphische Datenverarbeitung, Universitat Erlangen-Ni.imberg

Manfred Kaltenbacher Lehrstuhl fiir Sensorik, Universitat Erlangen-Niimberg

Stefan Karbacher Lehrstuhl fiir Optik, Universitat Erlangen-Ni.imberg

Xavier Laboureux Lehrstuhl fi.ir Optik, Universitat Erlangen-Niimberg

Hermann Landes Lehrstuhl fiir Sensorik, Universitat Erlangen-Niimberg

Reinhard Lerch Lehrstuhl fiir Sensorik, Universitat Erlangen-Ni.imberg

Marcus Magnor Lehrstuhl fi.ir Nachrichtentechnik, Universitat Erlangen-Niimberg

Heinrich Niemann Lehrstuhl fiir Mustererkennung, Universitat Erlangen-Niimberg

Dietrich Paulus Lehrstuhl fiir Mustererkennung, Universitat Erlangen-Niimberg

Rudolf Rabenstein Lehrstuhl fi.ir Nachrichtentechnik, Universitat Erlangen-Niimberg

Frank Schafer Lehrstuhl fiir Stromungsmechanik, Universitat Erlangen-Niimberg

CONTRIBUTING AUTHORS ix

Annette Scheel AG 4 Computergraphik, Max-Planck-Institut fiir Informatik, Saarbrticken

Hartmut Schirmacher AG 4 Computergraphik, Max-Planck-Institut ftir Informatik, Saarbrticken

Nikolaus Schon Lehrstuhl fiir Optik, Universitat Erlangen-Ntimberg

Harald Schonfeld Lehrstuhl fiir Optik, Universitat Erlangen-Niimberg

Stephan Seeger Lehrstuhl ftir Optik, Universitat Erlangen-Niimberg

Hans-Peter Seidel AG 4 Computergraphik, Max-Planck-Institut fiir Informatik, Saarbrticken

Marc Stamminger AG 4 Computergraphik, Max-Planck-Institut fiir Informatik, Saarbriicken

Eckehard Steinbach Lehrstuhl fiir Nachrichtentechnik, Universitat Erlangen-Ntimberg

Norbert Strobel Lehrstuhl fiir Nachrichtentechnik, Universitat Erlangen-Niimberg

Matthias Teschner Lehrstuhl ftir Nachrichtentechnik, Universitat Erlangen-Niimberg

Christian Teitzel Lehrstuhl fiir Graphische Datenverarbeitung, Universitat Erlangen-Ntimberg

Lutz Trautmann Lehrstuhl ftir Nachrichtentechnik, Universitat Erlangen-Ntimberg

Christian Vogelgsang Lehrstuhl fiir Graphische Datenverarbeitung, Universitat Erlangen-Niimberg

Riidiger Westermann Visualisierung und Interaktive Systeme, Universitat Stuttgart

Matthias Zobel Lehrstuhl ftir Mustererkennung, Universitat Erlangen-Niimberg

Preface

Traditionally, say 15 years ago, three-dimensional image analysis (aka computer vi­sion) and three-dimensional image synthesis (aka computer graphics) were separate fields. Rarely were expert<; working in one area interested in and aware of the advances in the other field. Over the last decade, this has changed dramatically. Maybe it is a result of the growing maturity of each of these areas that they are less concerned with themselves. Vision and graphics communities are today engaged in a mutually beneficial exchange, learning from each other and coming up with new ideas and tech­niques that build on the state-of-the-art in both fields. Many of us today believe that we will ultimately have one unified field, that, besides vision and graphics, also might encompass traditional image processing and image communication, and names such as 'Visual Computing', 'Imaging Sciences', or 'Image Systems Engineering' have been proposed.

Without doubt, three-dimensional image analysis and synthesis is very much an application-driven field. The declining cost of processors, memory, and sensors con­tinues to expand the scope of viable applications at a breath-taking speed. New systems solutions are in reach by combining state-of-the-art techniques from vision and graph­ics. The thorough scientific treatment of the underlying principles, for example, the limitations of sensors, the reliability, accuracy, and complexity of image processing algorithms, the fidelity of interactive visualization schemes, or the appropriate math­ematical formulation of the interaction of diverse methods from computer vision and computer graphics, is the prerequisite for such advanced systems.

This book is the result of a most fruitful collaboration between scientists at the University of Erlangen-Ntimberg, Germany, that, coming from diverse fields, are working together propelled by the vision of a unified area of three-dimensional im­age analysis and synthesis. As a formal framework for this collaboration, we set up the "Graduiertenkolleg Dreidimensionale Bildanalyse und -synthese" (Graduate Re­search Center 3D Image Analysis and Synthesis) initially, which is being supported by the Deutsche Forschungsgemeinschaft (German Research Foundation) since Jan­uary 1996. The Graduiertenkolleg comprises a program of research and advanced studies for doctoral students, with special emphasis on problems of 3D image acquisi­tion, computer vision, 3D graphics, and selected applications ranging from medicine to manufacturing. In January 1998, this program was substantially augmented and broad-

xii PRINOPLES OF 3D IMAGE ANALYSIS AND SYNTHESIS

ened by the addition of a "Sonderforschungsbereich (SFB)" (similar to an NSF Center of Excellence in the US). The SFB 603 "Model-bac;ed Analysis and Visualization of Complex Scenes and Sensor Data," again funded by the Deutc;che Forschungsgemein­schaft, complement<; the Graduate Research Center through currently 12 long-term collaborative research projects. These SFB projects address much more ambitious goals, often involving implementation and testing of large systems and investigations with a time-line exceeding five years. Within the SFB, a working group was formed that set out to systematically review and compile the state-of-the-art in image anal­ysis and synthesis, and quickly we realized that publishing the results of our effort in the fonn of a book and thus sharing them with the scientific community might be worthwhile.

It is our long-term vision to develop a theoretically well-founded general methodol­ogy for the design of systems for image synthesis and interpretation. This methodology should be based on the formulation of relevant problems ac; optimization problems, the use of model-based techniques, sensor data fusion, and hierarchical processing algorithms and data structures. Models are the backbone of 3D image analysis and synthesis. Modeling 3D scenes is central to computer graphics, as it is for 3D vision techniques. Often, these models are implicitly built into the flow of an algorithm, but more and more we are seeing techniques that keep the model explicit and separate from the algorithm, thus deserving the label 'model-bac;ed' . Initially we considered to organize the contents of this book around the various types of models used, ranging from surface geometry, reflectance and illumination to statistical models or semantic networks. But since many important state-of-the-art techniques are not model-bac;ed in the sense of providing and using an explicit model, we decided to follow a more conventional outline which startc; out at the image acquisition end of a hypothetical processing chain, proceeds with analysis, recognition and interpretation of images, towards the representation scenes by 3D geometry, then back to images via rendering and visualization techniques.

3D image analysis and synthesis systems can process regular camera images, but often optical 3D sensors are a viable and more powerful alternative that can acquire 3D data directly. Chapter 1 discusses the principles, the potential and the limitations of such sensors. Chapter 2 considers the processing of multiples views of a scenes acquired by several camerae; simultaneously or by one camera ac; a video sequence over time. Discussion includes the famous structure-from-motion problem and approaches to incorporate illumination estimation into motion estimation techniques. Chapter 3 deals with image recognition and interpretation, including segmentation, statistical models for recognition and image understanding, and active vision, where interpretation results are fed back to the sensor. Chapter 4 "Representation and Processing of Surface Data" is a central chapter since it interfaces both to traditional image analysis and image synthesis techniques. The discussion includes polygon meshes and their reduction, splines, and techniques for reverse engineering. The next two chapters address topics from computer graphics. Chapter 5 presentc; techniques for synthesizing realistic images. The traditional approach bac;ed on scene geometric descriptions and on models for reflection and illumination is discussed, including texturing techniques and global illumination. In addition, the recent developments in image-based rendering are presented. Scientific visualization, discussed in Chapter 6, would not be possible

PREFACE xiii

without combining image analysis and synthesis techniques, and thus can serve a<> an excellent example for the spirit of this book. It treat<; the state-of-the-art in volume visualization and includes a presentation of flow visualization techniques. We have also included Chapter 7 on Acoustic Imaging, Rendering, and Localization, partly because there are many interesting parallels to the material presented in the previous chapters, partly because we expect to see more applications in the future that can benefit from joint audio-video analysis and rendering. The concluding Chapter 8 is a collection of selected applications that some of us are involved in, ranging from automotive design to surgical planning.

We hope that scientist<>, engineers, graduate students and educators will find this book a useful reference for their work and a readable, but concise introduction to areas that they might be less familiar with. We have included references to the important seminal papers in each area, and thus the book should be a reasonable starting point for in-depth study of a selected problem. There is far too much material to cover in a one­or two-semester course, but individual chapters might be suitable as supplementary reading material in an advanced graduate level course.

The comprehensive treatment of such a variety of research topics wa<; possible only by a close cooperation of many expert<;. This book is the result of a true team effort. As editors, we express our sincere appreciation to all the contributing authors listed above. In addition, our special thanks go to M. Magnor, K. Hormann and C. Vogelgsang for the technical assistance in preparing the manuscript. Finally we are grateful to the Gennan Research Foundation DFG for financially supporting many of the contributors.

Stanford and Erlangen, February 2000

BERND GIROD, GUNTHER GREINER, HEINRICH NIEMANN