augmented reality, virtual reality, and computer...

Lucio Tommaso De Paolis • Antonio Mongelli (Eds.)

Augmented Reality,Virtual Reality,and Computer GraphicsThird International Conference, AVR 2016Lecce, Italy, June 15–18, 2016Proceedings, Part II

123

EditorsLucio Tommaso De PaolisUniversity of SalentoLecceItaly

Antonio MongelliUniversity of SalentoLecceItaly

ISSN 0302-9743 ISSN 1611-3349 (electronic)Lecture Notes in Computer ScienceISBN 978-3-319-40650-3 ISBN 978-3-319-40651-0 (eBook)DOI 10.1007/978-3-319-40651-0

Library of Congress Control Number: 2016941288

LNCS Sublibrary: SL6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics

© Springer International Publishing Switzerland 2016This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of thematerial is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,broadcasting, reproduction on microfilms or in any other physical way, and transmission or informationstorage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology nowknown or hereafter developed.The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoes not imply, even in the absence of a specific statement, that such names are exempt from the relevantprotective laws and regulations and therefore free for general use.The publisher, the authors and the editors are safe to assume that the advice and information in this book arebelieved to be true and accurate at the date of publication. Neither the publisher nor the authors or the editorsgive a warranty, express or implied, with respect to the material contained herein or for any errors oromissions that may have been made.

Printed on acid-free paper

This Springer imprint is published by Springer NatureThe registered company is Springer International Publishing AG Switzerland

Santo Stefano in Soleto (Lecce, Italy):The Presentation of Heterogeneous Data

Using Hybrid Platform

Francesco Gabellone1(&), Ivan Ferrari1, Francesco Giuri1,Paola Durante2, and Sofia Giammarruco2

1 CNR - IBAM (Istituto per i Beni Archeologici e Monumentali del ConsiglioNazionale delle Ricerche), Via prov.le Monteroni, 73100 Lecce, Italy

{f.gabellone,itlab.giuri}@ibam.cnr.it2 In-Cul.Tu.Re Project, Lecce, Italy

[email protected], [email protected],

[email protected]

Abstract. The paper illustrates the results of the research carried out by PONIn-Cul.Tu.Re. Project (INnovation in CULture, TUrism and REstoration) on thechurch of Santo Stefano in Soleto (Lecce, Italy). A multidisciplinary study hasbeen carried out in order to increase the knowledge of the church, throughnon-invasive diagnostic tests, and to monitor the state of conservation of thebuilding. An Augmented Reality virtual tour has been created through the use ofInformation and Communications Technologies in order to enhance the churchknowledge and make it more accessible to the public.

Keywords: Virtual archaeology � Hybrid platform � 3D modelling � App �Medieval church

1 Introduction

The paper illustrates the results of the research carried out by In-Cul.Tu.Re. Project(INnovation in CULture, TUrism and REstoration) on the Church of Santo Stefano inSoleto (Lecce, Italy).

The In-Cul.Tu.Re. Project, winner of the competition announcement Smart Citiesand Communities and Social Innovation (d.d. 84/Ric 02/03/2012), was founded byMiur (Italian Minister of University and Scientific Research).

The activities of the Project took place for the last three years 2012-2015 in GrecìaSalentina, an area that comprises twelve municipalities in the province of Lecce (theUnion of Grecìa Salentina), and have been supported by Ibam-Cnr Institute of Lecce,the Politecnico of Torino “Non Destructive Diagnostic Laboratory”, the Mario BoellaHigh Institute of Torino and CRESCo cooperative. During the course of In-Cul.Tu.Re.,twelve cases of study have been identified in line with three main aims: to applynon-destructive and non-invasive diagnostic tests for knowledge and conservation ofcultural heritage; to study energy efficiency of historic buildings; to develop Infor-mation and Communication Technologies for enhancement of cultural heritage.

© Springer International Publishing Switzerland 2016L.T. De Paolis and A. Mongelli (Eds.): AVR 2016, Part II, LNCS 9769, pp. 205–216, 2016.DOI: 10.1007/978-3-319-40651-0_16

The results are available on-line and could be downloaded through the www.inculture.eu web platform, an open-data solution having GIS (geographical and informationsystem).

One of the cases of study of In-Cul.Tu.Re. was the Church of S. Stefano in Soleto(see Fig. 1).

A multidisciplinary study has been carried out in order to increase the knowledge ofthe Church, through non-invasive diagnostic tests, and to monitor the state of con-servation of the building, with the aim to prevent decay problems after latest restorationrealized in 2012 by ex-Soprintendenza Beni Storici, Artistici ed Etnoantropologici ofthe region of Puglia. Moreover, an Augmented Reality virtual tour has been createdthrough the use of Information and Communications Technologies in order to enhancethe Church and make it more accessible to the public.

The Church of S. Stefano, with its ornamental and iconographic scheme, representsone of the most significant examples of late Gothic painting in the region of Puglia. It issituated in the historical centre of Soleto, an important Greek Byzantine cultural andreligious site from XIII to XVI century. The Church was consecrated to S. Stefano andS.Sofia and probably built before 1385 by Raimondello del Balzo Orsini, who was oneof the most important feudatories of the Naples kingdom. A sequence of pictorialcycles covers completely the walls of the building. The iconographic scheme iscoherent with a homogeneous project, but the executive phases are different andseparated.

Fig. 1. The Church of S. Stefano in Soleto, Lecce, Italy (Ph. by P.Colaiocco for In-Cul.Tu.Re.,2014)

206 F. Gabellone et al.

http://www.inculture.eu

http://www.inculture.eu

The bibliographic research and the study of archives allowed to create the icono-graphic interpretation diagrams, in which the different theories about pictorial cycles ofthe Church are detailed (see Fig. 2).

Non-invasive diagnostic tests were carried out in order to monitor the conservationconditions, to prevent decay problems (through IR Termography) and to characterizethe pigments employed in mural paintings (through Portable Micro X-Ray Fluores-cence Analysis).

Passive IR Termography, carried out by the Politecnico of Torino “Non DestructiveDiagnostic Laboratory”, was useful to map the degradation phenomena, to investigatethe masonry and the historical stratification of the building. The researchers plannedthree surveys in a year, in different seasons (Spring 2013, Autumn 2013, Late Sum-mer), hours (early afternoon, evening and midday) and climatic conditions (sunny, afterrains).

They investigated the presence of rising damp and infiltrations, responsible for thepast partial paintings loss. The first survey revealed the presence of thermal anomalyinduced by rising damp that wasn’t confirmed by next surveys (see Fig. 3).

In addition to this, it has also been possible to study the weaving of the masonry,identifying some walled up windows and the juxtaposition of different materialsattributed to the several restoration works done over the time.

Thanks to an investigation carried out by Ibam-Cnr Institute of Lecce, the portableMicro X-Ray Fluorescence technique (see Fig. 4), it has been possible to extend thepigment characterization also to the western wall and to the upper parts of the northernand southern partitions, whose pictorial cycles had never been analysed by any

Fig. 2. The iconographic interpretation diagrams.

Santo Stefano in Soleto (Lecce, Italy) 207

diagnostic programs realized before the restoration. The pigments and the possibleblends used for the main colour fields (blue, red, yellow, green, white, black and fleshypink) have been found in nearly 70 measuring points.

The results identified pigments usually employed in frescos such as red and yellowochre, green earth but also pigments less common in this technique such as cinnabar,

Fig. 3. The IR Termography mapped the decay problems and investigated the masonry of theChurch of S.Stefano.

Fig. 4. The portable Micro X-Ray Fluorescence testing. (Color figure online)


azurite, etc. The study need to be improved by the use of diagnostic tests comple-mentary to Micro-XRF.

Moreover, an Augmented Reality virtual tour has been created through the use ofInformation and Communications Technologies in order to enhance the Church andmake it more accessible to the public.

To develop the platform of AR virtual tour for both mobile and remote devices,Ibam-Cnr IT-Lab researchers first realized the Church 3D model employing laserscanning relief, digital photogrammetry and high resolution paintings mapping. The 3Dmodel was exploited as a support for information about paintings iconographic analysisand techniques, virtual restoration and for the results of the latest diagnostic surveys.This useful tool for the Church protection and conservation allows an interactive,immediate and efficient tour and approaches in an easy way the general public to topicswhich are often for experts only (Fig. 5).

2 An Hybrid Platform

The virtual archaeology aims to give to the public the results of interpretation formonuments and artwork whose appearance has been damaged or compromised, a goalpursued through a systematic study, which must be traceable in its evolution andpossibly must be transparent and intelligible. In this study, all the information gained inthe different disciplines of archaeological and historical research converge in a“knowledge model” that can be identified as a “synthesis” of the collected data. Asoften happens, the reconstructive study or the digital presentation of an ancient mon-ument is strongly affected by information gaps, such as the misspelling of the sources,the wrong translation or worse by subjective interpretations that can drastically affectthe final result, leading to reconstructions and interpretations many time very differentfrom each other. The main objective that we intend to always pursue, therefore, is the

Fig. 5. Screenshot of some available contents.


development of reconstructive studies through a strict philological path, highlightingthe results achieved with the whole path from which the proposed reconstruction starts.All data collected are generally available individually on specific publications, inscientific paper form, generally too complex and detailed for the visitor who wishes tostart a journey of knowledge on that cultural object. Starting from this premises, in theIn-Cul.Tu.Re. project we have been developed digital output intended to be used fromsmartphones, focused on logics of efficient on-site use ad on-site experience, throughthe use of technologies mainly based on Augmented Reality (AR). These technologiesallow the creation of an overlap between the actual experience and virtual informationelements (media information, geo-data, analytical data, historical, archaeometric, etc.)In an environment where the media elements that “increase” the information on thereality can be added and viewed via devices, such as the latest generation of mobilephones (Fig. 6).

As known, there are different approaches to use AR technologies. The most classicway provides a simple superposition of information directly over the object displayedon portable device. In recent graphics libraries implementations, you can add a sim-plified three-dimensional models that help the understanding of archaeological struc-tures, or contexts of cultural and tourist interest, directly over the real site. The use ofthese libraries, however, is strongly influenced by the computational limitations ondifferent devices, effectively compromising the enjoyment on many poorly performingequipments. Another possible limitation concerns the realism of the represented scenes,which for the reasons described above, do not allow to observe the complex 3D objectswith its realistic textures. A solution called 3DHOP recently experienced by ISTI CNRof Pisa in collaboration with our laboratory (CNR IBAM ITLab), developed only for

Fig. 6. User experience on iPad device. In evidence the different point of view.


iOS, has allowed to represent with high quality rendering of the Hypogeum Palmieri inLecce, but with a navigation only on fixed path (rails). One solution to this limit isgiven by an AR hybrid mode, where ultra-realistic three-dimensional reconstructionsare mixed in spherical high resolution VR panoramas. In this process special attentionis given to lighting the scene and to different set-up that re-creates the same lightingconditions of the actual site, to produce a convincing result perfectly superimposed onthe real site. This process allows to view several monuments in their original context,but also to enjoy a better and efficient management of the reconstructions even incomplex environmental conditions. A emblematic case, unfortunately very common, isrepresented by the reconstructions where the adoption of a simplified 3D model,without shadows, radiosity and low-resolution textures would result in a poor inte-gration and a qualitatively acceptable results. Regarding the church of Santo Stefano inSoleto we have been used technologies that could be called semi-AR, where the basisof the navigation is a three-dimensional model in which are “hooked” heterogeneouscontents, not simply the usual photographic restitution of the status quo. The utility ofthis approach has elements of interest because, through a panorama based on a 3Dmodel, you can explore the church from unusual perspectives, very difficult to repro-duce in reality. Another useful element is the easy overlapping of scenes where thesame model is represented with different or modified textures, or even with architec-tural elements added or subtracted to the original model (Fig. 7).

In general this technique is useful for the effective control of environmental impact,since its “direct” vision from a real point of view, allows to easily evaluate anyvolumetric disproportions, to control the actual spatial contribution and its relationshipto the whole. Certainly the use of easy and portable technologies, like spherical VR

Fig. 7. 3D model of church.


panoramas based on 3D scenes, are an effective solution for small architectural spacessuch as St. Stefano church. The use of a 3D model explorable in real time would notadd significant functionality, but, on the contrary, it would increase the complexity ofthe platform, thus decreasing the usability. The architectural space, based on a singlenave, is very simple and small, consequently two or three points of view cover all thespace (Fig. 8).

3 Multiple Contents Accessible from the Platform

The contents accessible from the platform concern mainly the figurative themes of thepaintings, but the most original aspect is definitely related to read the archaeometricdata within the virtual environment. Many scientific information, from IR, XRF andmicroscope observations, can be certainly defined as out of “range of visible” and needsuitable tools for their correct interpretation and visualization. These data are generallyavailable within specialist technical reports and are not published for the general user,the so called “general public”. For this reason has been developed the idea to placethese information within a virtual platform able to make “visible the invisible”, to lookbeyond the surface of the object, to reveal the presence of cavities, hidden object, to

Fig. 8. Screenshot of figurative themes of the paintings.


observe the behavior of materials under the temperature variation, highlighting cracksor plaster detachments, or to emphasize and display various types of deterioration(Fig. 9).

With the creation of a three-dimensional model, used for the generation of sphericalpanoramas, it was possible to obtain a perfect projection of the textures generated fromthe IR analysis on the inner walls of the church. This approach has solved the problemsarising from the resolution of the single IR frame and lens distortion of images capturedin different measurement campaigns and on different points of view.

The result is a reading, in a fading transition, of the visible color, with the possi-bility of comparison between the two different stages (Real-IR). An application con-ceived on this way immediately transfer scientific and technical information to usersnot prepared to read archaeometric data. We propose a “branched” reading, in which itis consulted only what you really matters. This visiting system combines the historicaland humanistic knowledge with the technical and scientific disciplines: the object, asvalues container, is analyzed in its mineralogical, petrographic, chemical and physicalcomponents, but also in its historical and formal features related with the ancientcontext. The virtual space accelerates and enhances cognitive performance. It is capableof generating extremely effective learning processes, easy to use and understand. Thesesystems improves understanding of cultural works and take advantage of the inter-disciplinary contributions from several researchers, who in different ways contribute totrace the historical, artistic and archaeometrical values (Fig. 10).

The technology is an important tool, especially to represent a complex and dia-chronic vision of the monument. It allows to elaborate articulated script onmulti-platform languages, such as JavaScript, which provide advanced solutions, butrequire adequate skills in programming. Despite this, the creativity and the ability to

Fig. 9. Spherical panoramas with IR analysis (left) perfectly matched over visible colors. (Colorfigure online)


solve problems suggest some interesting answers, such as the representation of thephases before and after the restoration of some paintings present on the walls of thechurch. The solution used is based on a open jQuery library. The result implemented inthis enjoyment platform has allowed to obtain an interactive comparison that returns inan even more “transparent” way the restoration work. It makes the user aware of thegreat work of figurative recovery, that the purely conservative operation would seemoverlook. The main output of the project is an HTML5 application compatible with PCand smartphone, with gyroscope function enabled on-site. This architecture allowsfuture developments with significant enrichment of all types of information necessaryto the understanding of the monument features and its transformations over the time(Figs. 11 and 12).

Fig. 10. a-b: IR and XRF analysis accessible on demand.


References

1. Gabellone, F.: Integrated technologies for museum communication and interactive apps in thePON DiCet project. In: De Paolis, L.T., Mongelli, A. (eds.) AVR 2015. LNCS, vol. 9254,pp. 3–16. Springer, Heidelberg (2015)

2. Malomo, L., Banterle, F., Pingi, P., Gabellone, F., Scopigno, R.: VirtualTour: a system forexploring cultural heritage sites in an immersive way. In: Proceeding of 2015 Digital HeritageInternational Congress, vol. 1, pp. 309–312, September 2015

3. Berger, M., Jacob, A.: La Chiesa di S. Stefano a Soleto, Argo Editrice4. Zacchino, V., Berger, M.: Paesi e figure del vecchio Salento, vol. secondo. Congedo editore,

Galatina (1980)

Fig. 11. Example of digital restoration of paintings. (thanks to Maida Leo, Fine Arts academy ofLecce)

Fig. 12. Synoptic view of church


5. Ortese, S.: Pittura tardogotica nel Salento, Galatina (2014)6. Ortese, S.: Sequenza dei lavori in Santo Stefano a Soleto. In: Dal giglio all’orso: i principi

D’Angiò e Orsini del Balzo nel Salento, A. Cassiano, B. Vetere, Galatina (Lecce) (2006)7. Potenziani, M., Callieri, M., Dellepiane, M., Corsini, M., Ponchio, F., Scopigno, R.: Com-

puters & Graphics, vol. 52, pp. 129–141, November 2015. ISSN: 0097-8493


augmented reality, virtual reality, and computer...

Documents