fraunhofer institute for production systems and design ... · the fraunhofer institute for...

F r a u n h o F e r I n s t I t u t e F o r P r o d u c t I o n s y s t e m s a n d d e s I g n t e c h n o l o g y I P K

AutomAted VirtuAl reconstruction

Virtual reconstruction connects high technology with social-cultural relevance.

c o n t e n t s

3 Preface

4 Interview: Jigsaw for re-evaluation

6 The entire process

10 Preservation and restoration of cultural

assets

14 Strategic partners

15 Imprint

Dr.-Ing. Bertram Nickolay, head of the Security Technology department

2

P r e F a c e

For more than 30 years Fraunhofer IPK in Berlin has been

promoting excellence in production science. It delivers applied

research and development services for the entire range of

industrial tasks – from product development, the production

process and product recycling to design and management of

factory plants. The institute is divided into six business divi-

sions: Corporate Management, Virtual Product Creation, Pro-

duction Systems, Automation Technology, Joining and Coating

Technology, Medical Technology. The institute sets particular

value in applying production-related solutions beyond the

industrial setting, for example in the areas of medicine, traffic

and security.

Since the early 1980s the Security Technology department

within the Automation Technology division has been develop-

ing image-analysis systems for use in difficult environmental

conditions. Through systematic research its research engineers

have been able to develop techniques for complex pattern

recognition tasks. The department is an international pioneer

in implementing systems for detection of biometric features,

monitoring movement of people and vehicles, and document

interpretation.

The most distinguished advancement in image analysis is a

technology that will be a valuable contribution to the preser-

vation of cultural assets, historical research and reappraising,

as well as forensics: automated virtual reconstruction of rup-

tured, shredded or otherwise damaged documents. Fragments

of documents are digitized and re-assembled on the computer

monitor through complex automated image processing pro-

cedures. The system has been developed to make ruptured

G.D.R. secret service files readable again. But also experts in

other fields – such as Ancient Oriental Studies, Archivistics and

Criminalistics – are watching the development of the system

with great interest. The technology promises enormous gains

of information and advancement in these fields. Throughout

the ages, paper, papyrus and parchment have played a promi-

nent role as cultural mediums. Until now, when documents

made of these materials were intentionally or unintention-

ally damaged, the information recorded on them had to be

considered lost, because restoration – if possible at all – was

either too time-consuming or too expensive. Fraunhofer IPK’s

research and development will fundamentally change this

dilemma.

In this context it is important that the Security Technology de-

partment does more than develop and implement algorithms

for the virtual reconstruction of two-dimensional objects. The

team considers all related processes, exploring for example

concepts for time-efficient and high-resolution digitization

of the initial material, as well as long-term backup, analysis

and evaluation of reconstructed documents. Furthermore,

the department is concerned with scenarios where experts

are aiming for a physical reconstruction of the initial material,

whether that material is paper or a three-dimensional object

such as a fresco or sculpture. To support physical reconstruc-

tion, robotics solutions that hold, carefully fit together and

join fragments are considered, as well as systems for expert-

assisted virtual (pre-)reconstruction.

The Fraunhofer Inst i tute for Product ion Systems and Design Technology IPK is the global leader in auto-

mated v i r tual reconstruct ion of destroyed documents, thus restor ing damaged archival data or ev idence

for evaluat ion. Future projects inc lude technology for repl icat ion and restorat ion of damaged artworks,

such as frescos.

3

the world’s largest jigsaw puzzle

In fall of 1989, members of the G.D.R.’s

Department of State Security (MfS)

began destroying files on spying ac-

tivities. A large part of the material was

»pre-destroyed«, i.e. ruptured by hand,

when the file shredders overheated. The

fragments were to be destroyed for good

later on. But it didn’t get that far. Rumors

of the document destruction activities

leaked out and G.D.R. citizens stormed

the MfS offices on December 4th, 1989.

More than 15 000 sacks containing some

600 million fragments of paper were

salvaged. In 1995, the Federal Author-

ity for the Records of the State Security

Service of the former G.D.R. (BStU) began

the task of manually reconstructing the

documents. It soon became clear that the

ruptured files contained highly impor-

tant information. They bear witness, for

instance, to psychological subversion of

dissidents and document the cooperation

of so-called Unofficial Collaborators (IM)

with the Communist Party and the secret

service. The destruction of the files mostly

aimed at protecting collaborators from

exposure and prosecution. Therefore, re-

construction of the damaged documents

is a matter of urgency toward assessment

of the injustices of the G.D.R.

I n t e r v I e w

Conversation with Dr. Bertram Nickolay, head of the Security Technology department

4

JigsAw for re-eVAluAtion

Dr. Bertram Nickolay is head of the Secur i ty Technology department. Born in Saar land, with a doctorate in

engineer ing, he has been working at Fraunhofer IPK s ince 1977. He came up with the idea to develop a

system for automated v i r tual reconstruct ion of Stas i f i les , which resulted in the ePuzzler. The ePuzzler i s

a computer system based on complex a lgor i thms that wi l l make a fundamental contr ibut ion to coming to

terms with recent German history.

how did you get the idea for the ePuzzler?

In the mid-nineties I heard from media reports and conversa-

tions with Stasi victims that BStU employees in Zirndorf, Ba-

varia, were manually piecing back together the ruptured secret

service files. I asked, ›isn’t there an automated solution for this

problem?‹ It was estimated that reconstructing the files manu-

ally would take several hundreds of years. We were sure that

automated virtual reconstruction would drastically reduce the

time necessary for this task.

why do you think you in particular are fit to solve the

stasi puzzle?

My team has spent years acquiring competencies in automa-

tion of visual processes based on pattern recognition and

digital image processing. As a result we had the necessary

know-how to face this complex job. Also, I’ve got a personal

motivation: I have many friends that were civil rights activists

in the time of the former G.D.R. One of them was the writer

Jürgen Fuchs who died of cancer at the age of just 48. It is

possible that his illness was caused by clandestine radiation

during his Stasi imprisonment. It is assumed that dissidents

were secretly exposed to radiation in prisons. If this could be

proved by a reconstructed piece of paper, I’d be pleased.

how did the project materialize, and has there been any

opposition?

The project materialized through my initiative, but it wasn’t

immediately successful. Today it is experiencing huge interna-

tional media response, but it took constant work to get the

ministries with authority over BStU and representatives of vari-

ous German parliament committees to one table and convince

them of the feasibility and innovation of the project. When in

2002 the German parliament put the project internationally

out to tender, we won the tender: of all the candidates, we

were the only one that could prove that the problem could

be solved using fragments of original Stasi documents. But it

wasn’t until 2007 that the Procurement Agency of the Federal

Ministry of the Interior mandated us to develop a procedure

for virtual reconstruction in a pilot project. During this project,

400 sacks are being processed. The pilot stage is going well;

we want to finish it by the end of 2011.

how important are the ruptured stasi files?

These files are an important legacy of the Peaceful Revolution.

The BStU states that they are mainly from the last 20 years

of the G.D.R. and contain especially explosive and instructive

information. Moreover, if a political power goes to so much

trouble to destroy something, it must be important.

are there systems comparable to the ePuzzler? and is

this technology applicable in other areas?

Fraunhofer IPK’s reconstruction technology is probably unique.

Even though there is isolated development work going on

elsewhere – no one has gotten as far with it as we have. This

is astonishing, because the inquiries coming from all over

the world prove that there is an enormous demand for this

technology. At the beginning we thought the Stasi files were

an isolated case. But then other countries such as Poland and

the Czech Republic asked us for help reconstructing ruptured

and shredded files. We’ve also reconstructed documents and

even license plates for criminal investigation departments and

tax authorities. Also, we started applying the technology to re-

constructing objects, damaged frescos for instance. In order to

do so, we already cooperate with archaeologists and experts in

art restoration.

how important is this work for you personally?

Very important. I’m a scientist with heart and soul, the techni-

cal aspect of the project is an exciting challenge for me as an

engineer. But above all, my friendship with Jürgen Fuchs and

other victims of G.D.R. persecution is what keeps encouraging

me to carry the project through. I want to make a contribution

to clarification of the facts. In my team’s work, advanced tech-

nology and social relevance work together in a unique way.

5

Merkmaleextrahieren

Suchraumreduktion Virtuelles Puzzeln

DigitalisierungVorbereitung

Automatisierte Aktenformierung

InhaltlicheErschließung

Langzeit-Archivierung

2. REKONSTRUKTION

1. DIGITALISIERUNG

3. AUFBEREITEN UND VERWENDEN

Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Preparation

digitization

ePuzzler

Processing & analysis

Digitization

Feature extraction Search space reduction Virtual jigsaw

Automated fileforming

Contentanalysis Long-term backup

With Fraunhofer IPK’s automated v i r tual reconstruct ion system, digita l repro-

duct ions of fragments of ruptured and damaged documents can be arranged

with the a id of the computer: for instance to make documents readable in

cases where manual reconstruct ion is e i ther imposs ib le or too t ime-consuming.

At present, the procedure is used mainly for restorat ion of two-dimensional

objects such as paper or paperboard.

Reconstruct ion is done in three steps: to process fragments v i r tual ly , f i rst they

must be digit ized. Then ePuzzler evaluates the digita l images of the paper

fragments and merges them gradual ly in an automated process to complete

pages. The third step goes beyond reconstruct ion of s ingle pages: i t comprises

mechanisms for automated formation of reconstructed pages to ent i re f i les as

wel l as techniques to interpret content. These tools are essent ia l especia l ly for

the evaluat ion of large amounts of source mater ia l , as in the case of the rup-

tured Stas i f i les .

the entire process

Before virtual reconstruction, digital images of the source ma-

terial have to be created, which means the fragments must be

scanned into a computer. This requires a series of preliminary

steps. The fragments must be isolated and cleaned, if neces-

sary; wrinkled and/or crushed material should be smoothed

out; paper clips and staples must be removed.

During digitization, metadata important for reconstruction is

collected. For instance, if paper snippets are clipped together,

this information is stored so the reconstructed pages will be

recognized as belonging together. Also, information on mate-

rial, rough content and temporal classification are fixed. This

»pre-knowledge« considerably increases the performance of

the matching process and renders possible to assemble multi-

page documents from single pages.

A high quality scanning process is required in order to be able

to evaluate and to arrange the digitized fragments precisely.

Both sides of fragments of any size – even very small ones –

must be scanned without reflections and shadows, true to

scale and color according to the original. Otherwise relation-

ships between the pieces, such as identical pigmentation or

neighbored edges, cannot be identified. Furthermore, the

fragments must be digitized on a background they can eas-

ily be separated from, similar to blue-screen-techniques. The

black and white backgrounds of common paper scanners are

not suitable for this. With large amounts of source material –

as with the ruptured Stasi documents – the goal is to process

several thousand pieces of paper per hour, while putting a

minimum of strain on the originals. At present, no tool exists

that lives up to all demands. Working with manufacturers,

scanning devices are being adapted specifically for the virtual

reconstruction process.

One of the biggest challenges is to limit deviations in color

and shape which can either result from aging of the scanner’s

lighting source, the scanning technique itself or the use of sev-

eral different scanners. Color differences can be balanced by

using color management systems, but inaccuracies concerning

the image shapes are much harder to overcome. Moreover,

different types of scanning techniques produce different

resolution errors. Line scanners capture the source material

line-by-line as it moves over a camera. A constant resolution

requires an extremely constant motion which is mechanically

hard to attain. By contrast, matrix scanners capture an entire

object at once. Resolution errors that occur by this scanning

technique can be compensated for through calculation to a

certain extent.

Scanning both sides of a piece of paper also requires thought-

out procedures. Alignment of both sides of the piece of paper

must be constant. This can be achieved by simultaneously

scanning both sides or using a special procedure by which the

fragments of paper stay fixed during turn over.

Furthermore, carrier materials are required to protect frag-

ments from friction and pressure, such as plastic film

envelopes. However, image defects can occur because of

these devices. Shadow preventing digitization requires very

thin film. Illumination is particularly difficult when extremely

fragile fragments have to be protected by solid glass panels

while they are being fed through the scanning device. Thus,

digitization is still a laborious field of research the institute is

actively involved in, in addition to development of reconstruc-

tion software.

digitizAtion

the entire process

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

7

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

The core of the automated virtual reconstruction system is

the so-called ePuzzler; here, the actual virtual restoration of

ruptured pages takes place. ePuzzler is a reconstruction soft-

ware developed by Fraunhofer IPK which uses complex image

processing and pattern recognition algorithms to construct

complete pages from digitized paper fragments. In addition, it

provides tools to examine and correct questionable or equivo-

cal puzzle results.

ePuzzler is structured into three main modules: feature extrac-

tion, search space reduction, and matching. Since there are

no two identical paper fragments, it is impossible to forecast

a straight calling sequence indicating how these modules

have to interact during a single reconstruction task. Thus, they

are embedded in a non-deterministic, adaptive workflow by

means of a complex software framework. The methodology of

virtual reconstruction is comparable to that of a human being

working on a jigsaw puzzle. On the basis of a number of de-

scribing features, he decides whether two pieces do match or

epuzzler

special mission: shredded documents

Reconstruction of shredded documents is a specific task

within virtual reconstruction of two-dimensional objects. In

this case, feature extraction and matching procedures have

to handle tremendous challenges: Shredded fragments

have a uniform shape, which means reconstruction must

rely only on content features and color gradients. More-

over, the fragments are just a few millimeters in width and

a few centimeters in length; therefore, content features

must be derived from a few sets of pixels. Hence, recon-

don’t. Similar to human procedure, ePuzzler initially calculates

different features of the paper fragments such as outlines,

paper color, writing, or ruling. These features are used to re-

duce the computational effort during puzzling – particularly

important with large amounts of data. Similar fragments are

clustered into subsets by means of intelligent search space re-

duction. The actual reconstruction, the matching, takes place

within these reduced quantities. Corresponding features are

investigated along the snippet contours. When two fragments

match, they are digitally glued and viewed as a larger frag-

ment during further reconstruction.

Ambiguities sometimes leave ePuzzler undecided about

whether two pieces really match. In this case partial recon-

structions available so far are collected and automatically

directed to so-called post-processing work stations. Here,

the system presents a range of most likely reconstruction

partners to a human editor so the results can be completed

interactively.

struction of shredded documents does not classify content

into obvious classes as »writing« or »ruling«. Instead all

elements on each shred – including paper color – are con-

sidered as geometric objects by whose allocation suitable

neighbors are being determined using rules of binary string

matching and probability. So the result is one that geo-

metrically suggests itself, rather than one that is correct in

respect of content. Therefore, reconstructed paper shreds

are currently always examined by a human supervisor.

t h e e n t I r e P r o c e s s

8

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosse

Merkmaleextrahieren






2. REKONSTRUKTION

1. DIGITALISIERUNG


Genosseprocessing & AnAlysis

If a reconstruction process results in a correctly and completely

reassembled page, this is already a very valuable result. But

for content evaluation, single pages usually have little value.

Therefore, mechanisms are developed in cooperation with

partners that will in the future help form documents or even

entire files from reconstructed pages. They will be comple-

mented by tools that facilitate the development of large quan-

tities of virtual material.

»File formation« means combining single pages to reconstruct

complete documents and organizing several documents into

larger files. Presently, in archives this process is performed

manually without support of software tools. By using digital

image processing methods like those used in ePuzzler, file for-

mation can be automated to a certain degree.

First attempts in this direction are based on metadata that are

gathered in advance or calculated during the reconstruction

process: information on type of paper, paper color, typeface,

and type color or ruling. At Fraunhofer IPK, algorithms for ta-

ble and form identification have already been developed that

recognize known types of forms on reconstructed pages with

the help of patterns. It is intended to expand these procedures

so that they can autonomously derive other, unknown types

of forms from recurring patterns. Additionally the extraction

of layout features and handwriting comparison is advanced

in order to identify recurrent elements such as page numbers

or page headers, so that relationships between single pages

can be established. The goal is ultimately to be able to identify

pages written by the same hand or on the same typewriter.

This would considerably speed up the formation of files.

Digital image processing also holds great potential for formal

and content analysis of reconstructed documents. Intelligent

character recognition (ICR) could make it possible to search for

specific catch phrases and thereby assign documents to a cer-

tain context. This would not only facilitate file formation but

also specification of catalogue entries. However, these meth-

ods still require a lot of research. Current OCR/ICR software

is limited to a small number of typefaces – for example, the

technology cannot cope with handwriting or Gothic print.

Even before all the above-mentioned procedures become ap-

plicable in archive management, it will be necessary to consult

single reconstructed pages for research and examination, for

example in trials touching activities of the G.D.R. State Secu-

rity. In this context it is important to ensure the legal capacity

of the digitally reconstructed material. Hence the entire pro-

cess of virtual reconstruction from digitization to long-term

archiving must be designed to be clear–cut and transparent.

Furthermore, all processed data – and particularly the snippet

images – must be protected against manipulation by means of

common hash procedures so that the integrity and complete-

ness of the data can be verified.

Despite these measures the virtual material itself will not be

enough in every context. Sometimes it may be necessary to

present a reconstructed original. This case is also taken into

consideration in the reconstruction process. Already during

digitization every fragment is marked with a definite code that

refers to the storage place of the original. This makes it easy

to find the original fragments of a virtually reconstructed page

and, if necessary, to physically reassemble them.

9

assistance-based reconstruction in egyptology

Dr. Friederike Seyfried, director of the Egyptian Collection of the

Staatliche Museen zu Berlin, is planning a unique pilot project

in cooperation with Fraunhofer IPK to put papyrus fragments

together with computer assistance.

what do you expect from this project?

Currently there is only one restorer taking care of the papyrus

collection within the Egyptian Collection. The number of papyrus

fragments she can put together during conservation work is

limited. Imagine the amount of information we would gain, if

she could be supported by virtual procedures! We also own parts

of documents whose fragments are dispersed over collections at

different locations. To put them together at least virtually, would

be a great help to research.

why does the system need the assistance of an expert?

Our restorer knows the highly sensitive material papyrus like no

one else. On the basis of fiber direction and style she can narrow

down place and date of origin. Such knowledge can hardly be

taught to a computer. Since the restorer’s expertise or a scientist’s

ability to decipher the characters is essential, it has to flow into

the necessary software development. But it cannot be substituted

by a computer.

why wouldn’t virtual reconstruction be sufficient?

We want to read what is written on the reconstructed papyrus,

but analysis of material is also an important part of the work. The

process for making papyrus is not known from contemporary

sources. We will only understand the procedure through study

of the material itself. Virtual images do not reflect every detail;

therefore, we have to assemble the originals – or at least the

parts we have. Hopefully virtual reconstruction will be a great

help with this process: It should be able to tell us, if we actually

have all the pieces of a document, or if there are important pieces

that have been lost or are perhaps in other collections.

P r e s e r v a t I o n a n d r e s t o r a t I o n o F

c u l t u r a l a s s e t s

Extremely damaged but

historically unique papyri

from the Egyptian Collec-

tion of Staatliche Museen

zu Berlin

Right: fragments of

destroyed documents from

the Cologne City Archive

high-tech meets expert knowledge

There are two approaches in assistance-based reconstruc-

tion. On one hand there is the concept of assistance-based

virtual reconstruction, which just like fully automated virtual

reconstruction aims at the virtual restoration of contents. After

digitization physical fragments are usually not required any-

more and can be archived conventionally. But in contrast to

fully automated restoration, assistance-based reconstruction

deals with fragments that for technical reasons cannot be re-

constructed in a fully automated manner – the reconstruction

process requires the active collaboration of a human supervi-

sor. Examples are the restoration of shredded documents and

restoration of damaged license plates to make them legible as

described on page 8.

The second approach follows the vision of an assistant system

to support physical reconstruction. Here, virtual reconstruction

will serve as a model for a subsequent manual reconstruction

or restoration. Such a system makes high demands on the

workflow before digitization and after virtual reconstruction.

Since the goal of this approach is the physical reconstruction

of the objects previously digitized, it requires the use of an ef-

ficient tracking system designed for the job. Also, depending

on the task at hand, many cases of physical reconstruction de-

mand the consideration of expert knowledge and evaluation.

Such knowledge may be the professional skills and insight of

the archivist or restorer – information on origin, age, or specif-

ic contemporary features – which can be applied to the virtual

reconstruction process via metadata. Other knowledge may be

gained during the process of virtual reconstruction itself – for

example when content features not visible on small fragments

though become visible to an expert when several pieces are

joined to make a larger piece. The computer will not be able

to depict expert knowledge that’s based on long-term experi-

ence, but it must be available for the process of restoration in

order to make it faster and more effective.

This approach is currently being tested. Fraunhofer IPK, in co-

operation with the Egyptian Collection of Stiftung Preußischer

Kulturbesitz Foundation – Staatliche Museen zu Berlin is plan-

ning a system to enable the physical reconstruction of a set of

extremely fragmented but historically unique papyri. Recon-

struction by hand would take an untold number of years be-

cause of the number of tiny fragments. This task can only be

solved through interaction of expert knowledge with a virtual

reconstruction solution.

The same applies to the Cologne City Archive building that

collapsed in 2009. Here too, Fraunhofer IPK’s systems and

methods can significantly support the reconstruction and res-

toration of the historically important archival records that sus-

tained considerable damage from the mechanical forces of the

collapse. The City of Cologne requested a test on 1000 pieces.

This test proved that the procedure can be applied to the

problem and it works.

Automated v i r tual reconstruct ion is a very versat i le tool . To use i t in an even more eff ic ient way in the

process of preserv ing and restor ing documents and other objects important to culture and society, sc ien-

t ists at Fraunhofer IPK are developing ass istance-based reconstruct ion systems that re ly on expert knowl-

edge for the reconstruct ion process and that wi l l ass ist phys ica l reconstruct ion of cultural assets.

11

P r e s e r v a t I o n a n d r e s t o r a t I o n o F

c u l t u r a l a s s e t s

Frescos broken into small

fragments can be re-

arranged with the aid of

virtual 3D reconstruction.

3d VirtuAl reconstruction

Archaeology and preservation of historical monuments has

for a long time made use of virtual 3D representation of de-

stroyed buildings and objects. But there hasn’t yet been an

automated procedure for assembling digitized components of

three-dimensional objects. So far, virtual reconstruction used

for presentation in a museum is a manual process which is

based on full knowledge of which part belongs where. But

automated reconstruction procedures are especially useful for

three-dimensional objects. When reconstructing these, restor-

ers struggle with multiple difficulties. First, the arrangement

of fragments in space presents various challenges. It is not

only difficult to practically do so, but often times additionally

complicated by the fact that the original form of the object to

be reconstructed is not known. Second, extensive excavations

or large-scale objects such as frescos render great numbers of

fragments. Virtual procedures can greatly assist and speed up

the process of classification and arrangement of such large

volumes of object fragments.

Fraunhofer IPK develops concepts for automated virtual re-

construction of three-dimensional objects using the example

of 100 000 marble slab fragments that are being excavated

in the ancient city of Ephesus in Turkey. The project is ideal

for first attempts in this field because no complex three-

dimensional registration of the fragments is required. Instead

a reconstruction by means of surface information is attempted

by evaluating contours and textures, similar to the process

described above for two-dimensional surfaces. In addition,

the thickness of the slabs at the breaking edges is taken into

consideration to help find fragment pairings – this is where

the procedure goes beyond that for reconstruction of two-

dimensional objects.

This method offers an intermediate step to »true« 3D recon-

struction: a 2D image of one side of the object is evaluated,

supplemented by information on depth. This procedure can be

considered a 2.5D reconstruction. In the next step entire 3D

objects can be constructed from several 2.5D images that por-

tray all areas of an object’s surface. One major challenge

in this context is to precisely distinguish between the object

and its break-up regions in order to generate respective views

of both.

High-quality 3D image acquisition devices designed for the

material to be reconstructed are a necessity for any kind of

spatial virtual reconstruction. State of the art are laser scanners

that capture a surface pointwise as well as devices using laser

cutting technology that gets depth information by layer cuts,

or topometric measuring systems that obtain depth informa-

tion via triangulation of patterns projected onto the object.

Additional procedures to gain 3D information, such as stereo-

photogrammetry, could also be taken into account.

Image acquisition is a major time factor in the processing of

three-dimensional objects. Therefore, Fraunhofer IPK first

develops concepts to execute this step as automated and ef-

ficiently as possible. Another field of work is the development

of algorithms for 3D reconstruction; here, successfully-tested

methods from 2D reconstruction are adapted and expanded.

Appl icat ion of automated v i r tual reconstruct ion is not l imited to two-dimensional objects such as paper

or papyrus. Three-dimensional reconstruct ion tasks can also great ly benef i t f rom the technology.

13

s t r a t e g I c P a r t n e r s

research and development partners

In the field of application-oriented basic research, Fraunhofer

IPK closely cooperates with the Institute for Computer Aided

Automation, CAA, of Vienna Technical University. Head of the

institute is Prof. Sablatnig, an internationally recognized figure

in the field of pattern recognition. His research team concen-

trates on the field of computer vision, image processing and

pattern recognition. They are international leaders in the use

of digital image processing for archaeology. Because their re-

search on methods of automated optical character recognition

has been so successful, they develop modules for character

extraction and classification for Fraunhofer IPK’s projects on

reconstruction.

Industry partners

Fraunhofer IPK’s important industry partners in the field of

reconstruction are arvato services and SAP Deutschland AG

& Co. KG (SAP Germany). arvato services is part of the global

Bertelsmann group and one of the largest suppliers of profes-

sional document management services in Germany. A success-

ful service enterprise for efficient solutions, arvato services has

longstanding experience in control and coordination of nation-

al and international services with complex business models.

The field of document management comprises a large spec-

trum of services; virtual reconstruction benefits from arvato

services’ elementary scan and logistics know-how. Having

provided similar services to other customers in the past, they

are experts on scanning, electronic storing, image processing

as well as transmission of electronically-processed image data.

arvato services has been Fraunhofer IPK’s strategic partner for

digitization within reconstruction projects since 2005. With

the start of the pilot phase of the BStU project, arvato services,

as a subcontractor of Fraunhofer IPK, is involved in digitizing

ruptured documents. Faced with masses of paper fragments,

the company manages a huge logistic challenge.

In terms of market capitalization, SAP is the world’s third-

largest independent software manufacturer for enterprises

and administrations. SAP has also been developing integrated

solutions, particularly in the public sector, since the beginning

of the 1990s (SAP Public Sector). With this, organizations are

able to successfully organize their business processes both in-

ternally and with customers, partners, and suppliers while sig-

nificantly improving the operational value. A leading software

manufacturer, SAP will in the future support Fraunhofer IPK in

product development and acquisition, performance and dis-

tribution for 2D digitization and reconstruction projects. SAP

also supports development of process-related graphical user

interfaces for automated virtual reconstruction and integration

of textual analysis procedures into document interpretation for

file forming and analysis.

clients as innovation partners

But not only partners from research, development, and indus-

try were in the past and still are relevant for bringing forward

reAching for higher goAls together

Virtual reconstruct ion is a technical ly and socia l ly re levant chal lenge for sc ience and research. Within

a global network of dist inguished partners in research and industry, Fraunhofer IPK has the goal of

thoroughly explor ing the potent ia l for reconstruct ion technology, thus highl ight ing Germany’s capabi l i -

t ies as a base for innovat ion in technology.

14

Kontakt

automated virtual reconstruction technologies. Fraunhofer

IPK’s client and a close partner in the Stasi documents frag-

ments project is »BStU«, the Office of the Federal Commis-

sioner Preserving the Records of the State Security Service of

the former G.D.R. More than 15 000 bags with ruptured Stasi

documents are in their archives, waiting for reconstruction

so that they can be used for archival development and for

tasks according to the Stasi Documents Law (StUG). The huge

amount of pre-destroyed archival materials and the manual

reconstruction of the Stasi documents that began in Zirndorf

near Nuremberg in February, 1995, is what inspired Fraun-

hofer IPK to its innovative technological developments. There

is a close cooperation between Fraunhofer IPK and BStU in

the interest of reviewing violent regimes. Together, they have

entered new territory in technology again and again. The chal-

lenges posed by the damaged Stasi files project have inspired

important advancements in the fields of computer-assisted for-

mation and analysis of the virtually reconstructed documents.

Issuer

Dr.-Ing. Bertram Nickolay

Fraunhofer Institute for Production Systems

and Design Technology IPK

Pascalstrasse 8-9, 10587 Berlin

www.ipk.fraunhofer.de

contact

Dr.-Ing. Bertram Nickolay, head of Security

Technology department

Phone: +49 30 39006-201

Fax: +49 30 391-7517

E-Mail: [email protected]

Dipl.-Ing. Jan Schneider, project manager

Phone: +49 30 39006-203

E-Mail: [email protected]

editorial staff: Nicole Kaiser, Jan Schneider,

Katharina Strohmeier

layout: Konstantin Heß

Photos: Fraunhofer IPK: Gerold Baumhauer (1,

2, 3, 4, 7, 8, 9, 15), Jan Schneider (12,13)

Staatliche Museen zu Berlin, Egyptian Museum

and Papyrus Collection: Myriam Krutzsch (10)

Imprint

© Fraunhofer IPK, 2010