Day 1 | British School at Rome (BSR) 9:00-17:00

9:00-9:05

Saluti by BSR Director | Stephen Milner

9:05-9:10

Introduction | Peter Campbell, Chris Stewart, and Iris Kramer

Session 1 | Archaeology and Culture

9:10-9:30

Graph Convolutional Neural Networks for Cultural Heritage: Applications in RS recognition, numismatics and epigraphyTraviglia, Arianna and Marco Fiorucci

9:30-9:50

ArchAIDE: A Neural Network for automated recognition

of archaeological pottery | Gattiglia, Gabriele, and Francesca Anichini

9:50-10:10

Machine Learning for the Classification of Stone-Age ArtefactsTziotas, Christos

10:10-10:30

Techniques of Machine learning for sex determination in human remains: When more advanced doesn’t mean betterPalomeque-Gonzalez, Juan F.

10:30-11:00 | Coffee break

Session 2 | Archaeology and Culture

11:00-11:20

Using Machine Learning for Named Entity Recognition in Dutch Excavation Reports | Brandsen, Alex, Karsten Lambers, Suzan Verberne, and Milco Wansleeben

11:20-11:40Machine Learning, Remote Sensing, and Archaeology: Tasks, Tools, Resources and Needs | Dzeroski, Saso and Ziga Kokalj

11:40-12:00

Digital Phrenology? An Experimental Digital ArchaeologyGraham, Shawn and Damien Huffer

12:00-12:20

Restoring ancient text using deep learning: a case study on Greek epigraphy | Sommerschield, Thea and Yannis Assael

12:20-13:00 Discussion

13:00-14:00 | Lunch

Session 3 | Remote Sensing 1 (LiDAR)

14:00-14:20

Historical landscapes and Machine Learning: (Re)Creating the hinterland of Tarragona, SpainMoreno Escobar, Maria del Carmen and Saul Armendariz

14:20-14:40

Learning to See LiDAR Pixel-by-Pixel | Schneider, Agnes

14:40-15:00

Classifying objects from ALS-derived visualizations of ancient Maya settlements using convolutional neuralnetworks | Somrak, Maja, Žiga Kokalj, and Sašo Džeroski

15:00-15:20

The use of R- CNNs in the automated detection of archaeological objects in LiDAR dataVerschoof-van der Vaart, Wouter Baernd and Karsten Lambers

15:20-15:40

Automated detection of grave mounds, deer hunting systems and charcoal burning platforms from airborne lidar data using faster- RCNN | Trier, Øivind Due and Kristian Løseth

15:40-16:40

Keynote: Tracking changes in meaning over time: how can machines learn from humans | Barbara McGillivray

16:40-17:00 Discussion

20:00 | Private Dinner and Drinks Reception for Conference Presenters at Villa Wolkonsky

Day 2 | European Space Agency (ESA) Centre for Earth Observation 9:30-17:20

Session 4 | Remote Sensing 2 (Machine learning for geospatial analysis in cultural heritage)

9:30-10:00 Welcome to ESA/ESRIN | Stewart, Chris

10:00-11:00 Keynote: Big Data Sources and Deep Learning Methods in Archaeology: A critical overview | Juan A. Barceló

11:00-11:20 Coffee break

11:20-11:40 Classification of Heritage 3D Data with Machine and Deep Learning Strategies | Remondino, Fabio, Emre Ozdemir, Eleonora Grilli

11:40-12:00 Arran: a benchmark dataset for automated detection of archaeological sites on LiDAR dataKramer, Iris, Jonathon Hare, and Dave Cowley

12:00-12:20 Machine Learning with Earth Observation for Cultural Heritage at the ESA Phi-Lab | Stewart, Chris

12:20-12:40 Deep learning for automatic feature detection and extraction on the archaeological landscape of Centocelle neighborhood in Rome using optical and radar remote sensing imagesMarsella, M.A., J.F. Guerrero Tello, and A. Celauro

12:40-13:00 Detection of Archaeological Sites using Artificial Intelligence

and Deep Learning Techniques | Karamitrou, Alexandra, Petros Bogiatzis and Fraser Sturt

13:00-14:00 Lunch

14:00-14:20 Mapping Threats to Cultural Heritage of the Middle East and North Africa | Rayne, Louise

14:20-14:40 InSAR Coherence Patch Classification using ML: Towards Automatic Looting Detection of Archaeological Sites | el-Hajj, Hassan

14:40-15:00 U-net for Archaeo-Geophysical Image SegmentationKüçükdemirci, Melda and Apostolos Sarris

15:00-15:20 Machine Learning in Space ArchaeologyLinstead, Erik, Alice Gorman, and Justin St. P. Walsh

15:20-15:40 As above so below: artificial intelligence-based detection and analysis of archaeological sites and features at a continental scaleOrengo, Hector A., Arnau Garcia-Molsosa, Francesc C. Conesa and Cameron A. Petrie

15:40-16:00 Discussion

16:00-16:20 Coffee break

16:20-17:20 Visit to Phi-Experience

Optional November 9 Tour Visit to Archaeological Sites

ABSTRACTS

Big Data Sources and Deep Learning Methods in Archaeology: A critical overview

Barceló, Juan A.

Keynote LectureComputational Intelligence in Archaeology began more than 30 years ago with the pioneering work of Jim Doran on simulation and Jean Claude Gardin on Expert Systems. The radical critique of Archaeological Theory prevented further development of formal reasoning methods, although some applications of Neural Networks and related methods of Machine Learning were published at the end of 20th century.

Nowadays, the popularity of so called “Artificial Intelligence”, and the new approaches opened by Virtual Reality have revitalized the field. In this paper I intend a critical overview of the historical development of the subdiscipline, as viewed from the domain of Archaeology and of Computer Science and Philosophy of Science.

The lack of recognition of Big Data sources in archaeology is one of the main obstacles for the dissemination of methods like case-based reasoning, rule-based systems, artificial neural networks, genetic algorithms, cellular automata, fuzzy models, multi-agent systems, automated induction and deduction algorithms, swarm intelligence, reinforcement learning, Bayesian networks and other hybrid systems.

In this paper I present an overview of main Big Data sources, both from field data and from image recognition. Reverse engineering and archaeological experimentation is another domain of application for computational intelligence. The needs of new and advanced temporal reasoning systems is also discussed, as the importance of the analysis of huge quantities of textual information from monographs and archaeological site reports.

This critical overview of available but underused techniques and technologies suggests the need of a new theoretical approach to understand the past from its remains in the present.

Using Machine Learning for Named Entity Recognition in Dutch Excavation Reports

Brandsen, Alex, Karsten Lambers, Suzan Verberne, and Milco Wansleeben

Over 60.000 Dutch archaeological research reports are available online, and this number is growing rapidly. The information in this grey literature can be of immense value, but is underused at the moment.

Currently it is only possible to search through the metadata of these documents, mainly via the Archis database and DANS repository. However, these metadata are often limited and sometimes inconsistent, and don’t capture the ‘by-catch opportunity’; i.e. a single Bronze Age find within a large Medieval excavation.

To effectively index these texts, Named Entity Recognition (NER) is needed to correctly identify and distinguish between archaeological concepts. Previously we have focused on using ‘classic’ machine learning such as Conditional Random Fields. Recently, Deep Learning (DL) methods, including transfer learning, have been introduced to the field of natural language processing, promising the same improvements they had on the field of computer vision. In this paper, we will present our initial results using DL techniques for archaeological NER, in the context of creating an online system, AGNES, that allows search on both the full text and these detected entities.

Machine Learning, Remote Sensing, and Archaeology: Tasks, Tools, Resources and Needs

Dzeroski, Saso and Ziga Kokalj

Archaeological use of remote sensing data has become standard practice, based on direct observation of sites in open landscapes from very high resolution UAS and satellite optical data, and on investigations with technologies revealing landscapes hidden below foliage, such as airborne laser scanning and InSaR. We

will first discuss the types of remotely sensed data that are increasingly often used in archaeology. We will then turn to the machine learning tasks that can be defined on such data, such as the detection and classification of archaeological objects or sites. We will finally discuss the tools, available and needed, for solving such tasks, as well as the resources needed (such as ontologies and annotated benchmark datasets) for the field to develop fast and follow the direction of open and reproducible science.

Digital Phrenology? An Experimental Digital Archaeology

Graham, Shawn and Damien Huffer

Not a week goes by without another buzzword crashing from tech journalism into mainstream consciousness. The hype and marketing surrounding ‘machine learning’ ‘deep networks’ ‘big data’ can be overwhelming. What’s an archaeologist to do? In this talk, I describe an experimental approach built around the idea of deliberately breaking the algorithms, to understand the potentials of a particular kind of machine learning in terms of computer vision and neural networks.

The BoneTrade Project uses convolutional neural networks and adjacent technologies to try to understand the trade in human remains mediated through visual social media. While scraping via particular keywords can produce volumes of materials, there is a disconnect between what people say versus what they show. There is also evidence of deliberate action to hide activities from keyword searches. While in dollar value this trade pales compared to other illicit antiquities trades, the damage to descendent communities is vast.

The resurgence of this trade replicates many of the sins of early archaeological research. To date we have been using ‘vanilla’ or unmodified versions of various ML techniques like transfer learning to surface various patterns. However, this introduces other ethical issues, which we can see more clearly when we deliberately ‘break’ the machine.

Machine Learning therefore requires an experimental digital archaeology, a pedagogy of breaking things, so that we see in

the edges and ruptures the ethical challenges for archaeology. Without these experiments, what we end up teaching the machine / teaching ourselves might only be a glorified digital phrenology

ArchAIDE: A Neural Network for automated recognition of archaeological pottery

Gattiglia, Gabriele, and Francesca Anichini

ArchAIDE is a European Union’s H2020 RIA programme funded project (2016-2019), aimed to create a new system for the automated recognition of archaeological pottery from excavations. Pottery is of fundamental importance for the comprehension and dating of archaeological contexts, and for understanding the dynamics of production, trade flows, and social interactions.

Today, this characterisation and classification of ceramics is carried out manually, through the expertise of specialists and the use of analogue catalogues held in archives and libraries. The goal of ArchAIDE is to optimise and economise this process, making knowledge accessible wherever archaeologists are working.

The ArchAIDE project developed an innovative app designed for desktop and mobile devices that aims to change the global practice of archaeology, thanks to the development of two neural networks for supporting appearance-based and shape-based recognition. Pottery fragments can be photographed, their characteristics sent to the neural network model, which activates the automatic object recognition system, resulting in a response where all relevant information is linked, and ultimately stored, within a database that allows each new discovery to be shared online. Currently, ArchAIDE represents a proof of concept which permits the classification of three pottery classes: Majolica of Montelupo, Roman Amphorae and Terra Sigillata. This presentation will focus on the results achieved by the project and on the main technical aspects: the (semi) automated digitisation of paper catalogues, problems encountered, and solutions found in developing the neural networks, creation of the mobile application.

InSAR Coherence Patch Classification using ML: Towards Automatic Looting

el-Hajj, Hassan

Detection of Archaeological Sites Artefact looting in the Near East has been a common feature on the archaeological scene and has been on the rise since the 1990s. The vast expanses of land along which these sites are distributed, as well as the remoteness of many means physical monitoring is impractical.

In recent years, many projects are relying on VHR imagery to monitor archaeological sites. However, this comes with a hefty price-tag, and requires human operators to visually detect looting. This method is inefficient when confronted with ‘big data’. In this paper, I propose a twofold automatic approach using Synthetic Aperture Radar data from the European Space Agency Sentinel-1 satellites to detect ground changes, and machine learning techniques to classify the disturbances into looting/non-looting.

The SAR data is used to generate Coherence maps from Repeat-Pass-Interferometry processed from two consecutive Sentinel-1 looks. These maps highlight areas of change occurring between the two acquisitions, which could be associated to archaeological looting. However, countless other processes can be at the root of InSAR Coherence disturbances. For this reason, Support Vector Machines are used to classify the patches of high disturbances (extracted from the segmented Coherence Map). These patches are associated with feature vectors constructed from coherence data, difference of NDVI values among others.

The training/validation data is acquired from Sentinel-1 InSAR over several areas in the Near East form the past couple of years. This paper presents the above work-flow with examples of detection at several test sites in Syria.

Detection of Archaeological Sites using Artificial Intelligence and Deep Learning Techniques

Karamitrou, Alexandra, Petros Bogiatzis and Fraser Sturt

Remote sensing data have seen wide use in the archaeological prospection for locating and mapping archaeological sites over broad regions of interest. However, the task of examining such data can be time consuming and requires large number of experienced and specialized analysts therefore could be immensely benefitted from automation.

In this work, we examine the potential of deep learning methods for the detection and mapping of three archaeological sites in Peru, South America, consisting of features of different sizes and shapes. Some of the main challenges of such methods is that they require the availability of large amount of labeled data to achieve high accuracy levels, and that the training stage can require significant computational resources. We show that even with a relatively limited amount of data and a simple eight layer, fully convolutional network that can be trained efficiently with minimal computational resources, the trained models can successfully identify and classify archaeological sites and distinguish them from features of similar characteristics.

The next step is to explore how techniques such as transfer learning and artificially augmentation can improve the accuracy of the network.

Arran: a benchmark dataset for automated detection of archaeological sites on LiDAR data

Kramer, Iris, Jonathon Hare, and Dave Cowley

Over recent years, machine learning (ML) has increasingly been applied to the problem of automated detection of archaeological sites. With each new approach, researchers have created their own unique datasets, which has made objective cross-study measurement and comparison extremely challenging. To solve this problem of non-standardization, we are introducing a new benchmark database containing approximately 1000 archaeological

sites (including roundhouses, shielings and cairns), and a selection of confusion objects on LiDAR data.

The database structure was inspired by the popular ‘Cars Overhead With Context’ benchmark which enables tasks ranging from simple scene classification to more advanced object detection. The known sites have mainly been manually detected by Historic Environment Scotland and the collection has been expanded through human-in-the-loop automation efforts.We will demonstrate ML baselines for the dataset on each of the prominent tasks.

Most notably, we will use the popular RetinaNet for our object detection task. This neural network is widely known for its introduction of Focal Loss, which addresses the foreground-background imbalance that trails accuracy in two-stage detectors like R-CNN. Additionally, the Feature Pyramid Network backbone of RetinaNet creates a multi-scale feature pyramid that fully utilises the information in the low-level and high-level features, which allows for the detection of objects in their wider context and improves the detection of small objects.

The dataset and all related material will be made publicly available. By doing so we hope to encourage others to publish their data in a similar format, leading to an increase in cross-study comparison and the reusability of data and ML approaches.

U-net for Archaeo-Geophysical Image Segmentation

Küçükdemirci, Melda and Apostolos Sarris

Thanks to the recent advancements on Deep Learning (DL) and the increasing availability of large annotated and trained image datasets, there are impressive improvements in the automated analysis of images using deep neural networks from different scientific domains such as medical and dental sciences, microbiology and genetics, physics and astronomy, geophysics and satellite remote sensing.

The automated analysis of archaeological features is also considered as an important approach for archaeo-geophysical applications because of the large spatial extent of areas covered by the surveys and the increasing quantity of the collected data with high-resolution sampling. This work constitutes one of the first

attempts of deep learning-based approach using convolutional neural networks (CNN) for the automated image segmentation of archaeo-geophysical features. A convolutional neural network is built by Python 3.6 programming language and the Deep Learning Library of Keras with Tensorflow backends and network trained from the scratch by adopting U-Net architecture. Here, network architecture, experimental results on choosing hyper-parameters and prediction results on images obtained in different archaeological site are discussed. Considering the accuracy of the results, this network obtained a promising information combining localization and context, which is necessary to predict a good segmentation map.

Machine Learning in Space Archaeology

Linstead, Erik, Alice Gorman, and Justin St. P. Walsh

The International Space Station Archaeological Project (ISSAP) is an international collaboration using machine learning (ML) and deep learning to enable archaeological research in new contexts. Recent advances in ML and computing infrastructures have opened new directions in large-scale mining of multimedia data. ISSAP’s use of ML was necessitated by the difficulty of studying an archaeological site that cannot be visited by the researchers. In 18 years of continuous habitation, however, ISS crew have taken millions of digital photos, which today are held in NASA archives. The aim is to analyse this massive collection of images showing life on board the International Space Station (ISS), using ML to recognize people, places, and objects, and the relationships between them.

Understanding how astronauts use material culture to adapt to life in space has applications in the design of future long duration missions. ML presented a possible way to automate the recording of objects, spaces, and individuals. Convolutional neural networks are a well-established, state-of-the-art tool for this work. A hybrid relational-graph database will be used for further analysis of patterns, such as presence/absence and mapping objects through time. Our paper will describe some challenges and solutions that we have encountered so far. Beyond ISS, we are assessing and developing ML technologies for their utility in classifying features in other large historic and ethnographic photograph collections.

Our project applies these techniques to a domain that has yet to benefit fully from the power of AI, enhancing archaeological research both on Earth and in space.

Deep learning for automatic feature detection and extraction on the archaeological landscape of Centocelle neighborhood in Rome using optical and radar remote sensing images

Marsella, M.A., J.F. Guerrero Tello, and A. Celauro

The Archaeological Park of Centocelle in Rome, with the concomitant presence of several roman villas and pre-roman and mediaeval settlements, which vestiges have been mainly reburied after excavation, and the roman subterranean quarries of tuff and pozzolana which extension is not fully known, is a perfect test zone for the setting of a tool for automatic feature extraction, based on (CNN) convolutional neuronal network as the feature classifier from aerial and satellite data.

This study underwent a preliminary phase, carried out to test the functionality of different geomatic products for the identification and monitoring of complex archaeological evidences in urban areas. The methods used were the digital photogrammetry, 3D modelling, remote sensing interpretation and digital cartography. Archaeological buried or semi-buried features and traces related to subterranean mining have been identified by processing aerial and satellite optical and SAR dataset to enhance the contrast of archaeological features from the background. For the detection and extraction of archaeological features it is convenient to apply the CNN architecture, applying one or more layers of convolution units. A convolution unit receives its input from multiple units from the previous layer which together create a proximity. Therefore, the input units (that form a small neighbourhood) share their weights.The convolutions units are especially beneficial to reduce the number of units in the network. This means, there are fewer parameters to learn which reduces the chance of overfitting as the model would be less complex than a fully connected network ( improving processing time, quality of results). The tool is proposed as a method to promote a better knowledge and protection of thearchaeological context, allowing the establishment of limits to urban enlargement in the areas of archaeological respect.

Tracking changes in meaning over time: how can machines learn from humans

McGillivray, Barbara

Keynote LectureOver time new words enter the language, others become obsolete, and existing words acquire new meanings. This is a phenomenon inherent in all languages, and it is affected not only by cognitive, linguistic and literary factors, but also by interactions between languages and cultures and social changes affecting communities and individuals. For example, over the course of the rise of Rome and the conversion of the Roman world to Christianity, Latin words such as uirtus and passio underwent profound alterations as the values and conceptual systems of Latin speakers changed. Virtus, originally meaning ‘manliness’, extended to comprise violence, Christian virtue, and miracles. Passio, originally meaning ‘suffering’ or ‘experience’, extended its range to mean ‘emotion’ and the particular suffering and death of Christ and the martyrs.

The recent digitization efforts have now made it possible to access and mine large digital collections of historical texts using machine learning methods, and to investigate the question of word meaning change at an unprecedented scale.

In this talk I will discuss my research on developing computational models for word meaning change in ancient languages. This work paves the way to a hybrid Artificial Intelligence expert system in Humanities research which uniquely combines human expertise and algorithms.

Historical landscapes and Machine Learning: (Re)Creating the hinterland of Tarragona, Spain

Moreno Escobar, Maria del Carmen and Saul Armendariz

Archaeology is increasingly using models of the surface of the Earth to investigate issues and dynamics from the past. Digital Elevation Models, Digital Surface Models, and Digital Terrain Models (produced by LiDAR, photogrammetry and other methods)

are becoming more accurate and precise. Despite their increased spatial resolution, these models still represent the surface of the Earth at present, and as such they ignore the profound trans- formations that landscapes might have experienced from Antiquity.

These transformations can be, in turn, explored through means of geoarchaeological and paleobotanical methods, as well as through the analysis of historical maps and other sources of historical cartography. However, the integration of these different sources of information remains problematic, due to the high cost of processing data from historical maps and the low special resolution of the archaeobotanical analysis, amongst other issues.

Using the highly transformed area of Tarragona (Spain) as study case, this paper compares and discusses several approaches to the construction of historical digital terrain models through the application of different techniques of machine learning for developing more efficient methods to process historical maps.

As such, this contribution intends to highlight the issues and offer possible solutions to the reconstruction of historical landscapes, thus extending its application in archaeological and temporal studies.

As above so below: artificial intelligence-based detection and analysis of archaeological sites and features at a continental scale

Orengo, Hector A., Arnau Garcia-Molsosa, Francesc C. Conesa, Cameron A. Petrie

The last few years have seen an important increase of new AI-based methods for the detection and analysis of archaeological sites and features. However, many of these methods provide satisfactory results that are largely contingent upon specific cultural and environmental circumstances. This paper will present current efforts by a group of researchers from the Catalan Institute of Classical Archaeology and the University of Cambridge to develop a semi-automated workflow for the largescale detection of archaeological sites within a range of environments.

These efforts have focussed on: 1) the development of global site detection algorithm; and 2) an automated drone-based survey method. The global site detection algorithm uses the location of known sites to train a machine learning probabilistic classifier, which is able to adapt to specific local environmental conditions. The classifier uses multi-temporal and multi-sensor satellite data in combination with specific raster products that can boost the detection of archaeological sites. The algorithm self-evaluates its performance against the training dataset and creates a purpose-built data composite before classifying the whole study area. The automated drone-based intensive survey method is able to map and extract as vectors a high percentage of all visible pottery sherd fragments distributed across the surface of a given study area.This method makes use of photogrammetric processes, CNN-based deep learning algorithms and geospatial analyses. It was designed for use with standard commercial drones, though the implementation of new drone technologies is able to boost detection rates and significantly reduce alse positives. In the future, it will be possible to combine these two methods in a web-based open source tool to aid the location and analysis of archaeological sites world-wide.

Techniques of Machine learning for sex determination in human remains: when more advanced doesn’t mean better

Palomeque-Gonzalez, Juan F.

The development of new statistical and data science techniques open a new and exciting world of opportunities in all the sciences, including archaeology. These new methods have the need for high amounts of standardised data and for this reason, osteoarchaeology is especially suitable to trythese new advances. But one of the difficulties that we can find when we apply this new approach is to decide which specific test to use in each specific case. In this study, we wanted to answer two main questions, how accurate can be these new techniques in compassion with the classical statistical methods, and how can we know which method suits better to answer the specific question that we want to answer.

We have tried to determine the sex of the individuals in the Goldmann Osteological Database but using only the anthropological measures made in the humerus. We have compared a number of 10 different methods of machine learning to answer the question and different combinations of measures, trying to simulate the possible states of conservation that we can find in the real archaeological practice. We have demonstrated that not always the most advanced, complex and expensive in computer power terms is the most accurate test to apply in a specific problem.

Mapping Threats to Cultural Heritage of the Middle East and North Africa

Rayne, Louise

Since 2015, the Endangered Archaeology in the Middle East and North Africa (EAMENA) project has been using Earth observation to document archaeological sites across the MENA region and the threats posed to them in an online database. In this paper we present a discussion of the trends relating to damage and threats which we have identified and our strategies for risk monitoring and mitigation. While damage to archaeological sites due to conflict (e.g. vandalism, looting) is often high-profile, significant damage is caused by the impacts of modern land use activities. We discuss selected case studies from our region of interest, for example the ancient oases of Al-Jufra in Libya which have been damaged by agricultural expansion. Multispectral data such as Sentinel allows changes to sites to be monitored and higher-resolution imagery is used for digitising archaeology. Over 250,000 sites have been documented in detail using these data and this process is ongoing. We use open-source software and data so that the EAMENA database and methodology is as widely available as possible to cultural heritage professionals in the MENA region. Strategies for minimising risk to archaeological sites include regular automated change detection. EAMENA is utilising classification and machine learning algorithms to monitor the main threats posed by modern land use at each site. We have developed methodologies using Google Earth Engine to map previous and ongoing change widely, mainly using Copernicus Sentinel data. Since 2016 we have been delivering

training courses for heritage professionals in the MENA region in these methods and in the use of databases, GIS and remote sensing for recording archaeological sites and assessing risk to their preservation.

Classification of Heritage 3D Data with Machine and Deep Learning Strategies

Remondino, Fabio, Emre Ozdemir, Eleonora Grilli

The production of 3D point clouds or meshes, coming from photogrammetry and laser scanning surveys, is nowadays broadly diffused for documentation purposes of Cultural Heritage (CH) scenarios. As the evolution of the technologies and digital tools allowed to multiply photogrammetric and laser scanning acquisitions, the need for rapid methods to classify point clouds or meshes is becoming fundamental. Among the possible and interesting applications provided by the classification of heritage models there are, for example, the possibility to identify and distinguish structural and decorative architectural elements, or the identification of different states of conservation and materials, or the automatic recognition of similar architectural elements in very large dataset as a propaedeutic phase for Building Information Modelling (BIM), etc. In the literature, different classification methods were proposed like edge and region-based approaches (originally applied for image segmentation) or also model fitting approaches, based on the possibility to fit geometric primitives to the 3D shapes.

With the blooming of Machine and Deep Learning (ML/DL), data classification methods got much attention and interest, also in archaeology and cultural heritage. Deep learning can be considered an evolution of machine learning as it structures algorithms in layers in order to create an “artificial neural network” that can learn from some training data and make intelligent predictions or decisions on its own.

In the last decade the use of ML/DL techniques for point clouds classification has been successfully investigated in urban and geospatial environments whereas, in the field of cultural heritage, it has only recently started to be explored. We will report our developments and results in the use of ML/DL techniques with heritage 3D data. In particular, we will present various

supervised methods, i.e. approaches where the given input data contain associated labels (classes) information provided by an operator. Up to the authors’ knowledge, in the archaeological and heritage field is hard to reach good results with unsupervised methods, that creates self-evaluation to identify different groups based on similar features without knowing the class of interest. This is mainly due to two different reasons:

- the delineation of the classes in the heritage field can be really variegate: for the same case study several classes can be identified based upon different purposes (e.g. materials, deterioration areas, construction phases, etc.)

- not always a precise shape/colour correspond to a semantic architectural class.

Therefore, for the classification of 3D heritage data, case-by-case different geometric and radiometric features are generally extracted to train a model able to predict then the classification of the entire dataset. The geometric ones are normally extracted at different scales, then just the ones that better highlight the heritage elements under consideration are selected and used during the classification processes. These features are based on the covariance matrix (and its eigenvalues and eigenvectors) computed within a local neighbourhood of a 3D point and they describe the local distribution of the 3D points. In addition to these ones, the elevation of the points as well as the RGB/HSV values are also taken into consideration in some experiments.

These features are used to train a specific classifier (Random Forest, One-versus-One, CNN, Bi-LSTM, etc.) which then predicts all classes on the entire dataset. In order to evaluate the performance of the classification methods, the label predicted by the classifier is compared with the same manually annotated. Confusion matrices are then generated and for all the classes F1-Scores and balanced accuracy are computed. Example from various 3D data (point clouds and meshes) will be shown, including the Bartoccini Etruscan tomb, the Spouses Sarcophagus, the Neptun temple in Paestum and the Bologna porticoes. The retrieved semantic information is also used to quantify specific evidence (e.g. square meters of ruined surfaces) useful for particular applications (e.g. restoration).

Learning to See LiDAR Pixel-by-Pixel

Schneider, Agnes

Methods borrowed from other disciplines have pushed Archaeology to reinvent its methods from time-to-time, opening always new and broader horizons. This is the case of computer applications which are present in one or another since the commercialization of computers in the archaeological tool set.

Also other disciplines lent concepts, approaches and most of all methods and tools to cope with always new kinds of data and challenges, like archaeological big data. Such a discipline is Computer Vision which methods have revolutionized the methods for e.g. of Environmental Informatics, Climatology and Remote Sensing. Because Archaeological Remote Sensing is intertwined with these disciplines it is obvious that we do not have to reinvent the wheel but serve ourselves from a pretested toolset. In this light this paper would like to present the results of a project, which goal is to detect archaeological objects in LiDAR data by pixel-based image analysis. Throughout this projects several machine learning algorithms are applied to a dataset – in the region around the Dünsberg, Germany – and compared to each other in R. Furthermore ways to improve prediction results are shown and discussed. As a last point the viability of pixel based image analysis of LiDAR data is assessed.

Restoring ancient text using deep learning: a case study on Greek epigraphy

Sommerschield, Thea and Yannis Assael

Epigraphy is primary evidence for reconstructing the history and thought of the ancient world. Restoring missing epigraphic text is one of the central undertakings of the discipline. Each step in this historical exercise necessitates accessing vast repositories of information, offering textual and contextual parallels. These repositories primarily consist in a researcher’s mnemonic repertoire of parallels, and in digital corpora for performing character-sequence searches that match word patterns.

Our research presents a novel assistive method for providing text

restorations using Deep Learning, a research field of Machine Learning which allows statistical models to discover patterns in data, and meaningfully represent them.

To the best of our knowledge, our deep neural network PYTHIA is the first ancient text restoration model that recovers missing characters from a damaged text input. Its architecture is carefully designed to handle long-term context information, and deal efficiently with missing or corrupted character and word representations. To train it, we wrote a non-trivial pipeline to convert PHI, the largest digital corpus of ancient Greek inscriptions, to machine actionable text, which we call PHIML.

On PHI-ML, PYTHIA’s predictions achieve a 30.1% character error rate (CER), compared to 57.3% of human epigraphists. Moreover, in 3.5% of cases the ground-truth sequence was among the Top-20 hypotheses of PYTHIA, which effectively demonstrates the impact of such an assistive method on the field of digital epigraphy, and sets the state-of-the-art in ancient text restoration.

Classifying objects from ALS-derived visualizations of ancient Maya settlements using convolutionalneural networks

Somrak, Maja, Žiga Kokalj, and Sašo Džeroski

Archaeologists engaging with remote sensing data rely heavily on manual interpretation. This presents a major bottleneck in the data analysis pipeline, preventing them from keeping up with ever increasing data volumes, and creating a pressing need for computational methods that can automate data annotation and analysis and thus facilitate interpretation. Critics of computational methods like to stress the superiority of human vision for identification of archaeological objects from remote sensing data.

Therefore, raw data is often converted to a representation that is most intuitive for visual interpretation. Deep convolutional neural networks (CNNs), which mimic the human visual system, are now dominant in the field of computer vision. While there have already been applications of CNNs in remote sensing, only a handful of them concern airborne laser scanning data (ALS) data

in archaeological prospection.

Our own survey acquired 220 km2 of ALS data around Chactun, an ancient Maya city, in Campeche, Mexico. Manual annotation of over 12.000 anthropogenic objects, condensed on 130 km2 of prospected area, took several man-months. Having in mind that thousands of square kilometres of similar data have already been collected in other surveys, further manual work can be minimized if we manage to automate the annotation process. We therefore aim to develop a robust and adaptive method, capable of distinguishing and localizing various types of anthropogenic objects in ALS visualizations. In this paper, we will present a CNNbased image recognition method. This will represent the first stage of a three-stage process, which will be followed by the stages of object detection and semantic segmentation.

Using our annotated data we generated training and testing sets of thousands of images (patches), corresponding to different object types. The abundance of labelled data was sufficient to successfully train a deep CNN. High accuracy was reached for multiclass classification, distinguishing among three types of anthropogenic objects and natural terrain.

Machine Learning with Earth Observation for Cultural Heritage at the ESA Phi-Lab

Stewart, Chris

To fully embrace New Space, the Φ-lab was created at the European Space Agency with the objective to become a catalyst for European based innovation in Earth observation. This paper outlines research carried out in the Φ-lab in the application of Machine Learning with Earth observation satellite data for Cultural Heritage. Two projects will be presented: the first combines crowdsourcing with Deep Learning for the systematic prospection of archaeological crop-marks, the second involves monitoring of historic infrastructure in desert regions applying Machine Learning to Synthetic Aperture Radar (SAR) input data.

The method of applying crowdsourcing as a means to collect training data to train Machine Learning models is well established. Applying this method to archaeological prospection is challenging, given the heterogeneity of archaeological crop-marks and the difficulty

in distinguishing these from natural crop-marks, such as roddons, and other features like paths, tractor marks and field boundaries. A crowdsourcing project has been developed in the Φ-lab to extract archaeological crop-marks appearing in very high resolution Birds Eye oblique air photos, available on Bing Maps. The project focuses on the area surrounding Rome in order to manage the number of tasks. Each task comprises the classification of one Birds Eye tile, which needs to be completed at least three times for redundancy. Results so far appear promising, with many positive identifications of archaeological features, including buried roads, buildings and urban areas. Once a critical mass of training data is obtained, this will be enhanced with data augmentation techniques, and finally fed into a Deep Convolutional Neural Network for automatic classification scaled over a larger area.

The second project on the monitoring of historical infrastructure is still at an early stage of development. In many desert regions, such as in North Africa and the Middle East, a significant problem is the burial of infrastructure by sand drift. In some cases previously excavated archaeological sites are once again lost beneath the sand in such areas. Many historical structures, such as abandoned roads, or military objects, are also frequently buried beneath wind-blown sand. Given the sensitivity of SAR to the relative permittivity of man-made objects, and to topographic effects, it is often more suitable than optical data for man-made object detection in desert regions. However, the backscatter of such objects can be very similar to natural features, like sand dune ridges and rock formations. The Φ-lab, in collaboration with the EU Satellite Center, aims to use Sentinel-1 data with cloud processing and machine learning in an attempt to develop an automatic algorithm to retrieve man-made objects in a number of desert areas of interest. If successful, the intention is to scale to larger areas.

Graph Convolutional Neural Networks for Cultural Heritage: Applications in RS recognition, numismatics and epigraphy

Traviglia, Arianna and Marco Fiorucci

Convolutional neural networks (CNNs) have emerged as a powerful tool for the detection and the recognition of objects in images and video. However, CNNs may lose spatial relationships between parts of an image. A natural way to encode these relationships is to model an image as a graph, where the nodes represent image regions and the edges represent the relationships between pair of regions. In this talk, we first describe a generalization of CNNs onto graphs, called Graph Neural Networks (GNNs), we then provide an overview of the ongoing projects of the Center for Cultural Heritage Technology (CCHT) and, finally, we discuss how GNNs may play an important role in these projects.

A GNN is a neural network that takes as input a graph and learns node features by taking into account both the initial node features (characteristics of image regions) and the relationships between pairs of nodes. Indeed, GNNs are well suited to process spatial vectors representing features of groups of similar objects (e.g. burial mounds) and for capturing geometric variations of different object regions in remote sensing images.

Furthermore, GNNs can be used for clustering the nodes of an ancient coin graph, where each node is a coin and each edge connects pair of visually similar coins, or a cuneiform tablet bipartite graph constructed to study the function of specific epigraphic marks.

Automated detection of grave mounds, deer hunting systems and charcoal burning platforms from airborne lidar data using faster-RCNN

Trier, Øivind Due and Kristian Løseth

We present a new method for automated mapping of historic monuments such as grave mounds, pitfall traps and charcoal kilns. The method is based on a region-proposal convolutional neural network called “simple faster R-CNN”. The network was pre-trained on a large database of natural scene images. Each image had annotations in the form of bounding boxes with associated class labels. Then the network was trained on images derived from airborne lidar data.

The lidar point cloud data was converted to a digital terrain model (DTM) by keeping all points that were labelled as ‘ground’. The DTM was then converted to a simplified local relief model by subtracting a smoothed version of the DTM. The local relief model enhances local detail in the DTM while suppressing the general landscape topography. Thus, cultural heritage remains such as grave mounds, pitfall traps and charcoal kilns are often visible. Each geographic area was divided into disjoint areas for training, validation and testing. Training, validation and test images of sizes 150 m × 150 m were extracted from the local relief model data. Each image contained one or more cultural heritage objects clearly visible.

For the test images, the overall correct classification rate was 83%, and for the specific classes: grave mound 81%, pitfall trap 78% and charcoal platform 95%. 16% of the true cultural heritage objects were missed by the method, while 1% were detected with wrong class. 21% of the objects that the method predicted as being cultural heritage were in fact not.

Machine Learning for the Classification of Stone-Age Artefacts

Tziotas, Christos

During the last year I have researched how machine learning (hereafter ML) can be used to classify stone-age artefacts, more specifically, stone age axes from Southern Norway and Sweden - called Nøstvet/Lihult- and -axes. There has not been performed much research on ML-approaches regarding stone age artefacts in digital archaeology in the last decade, and much has changed regarding technology and available data.

In my research on Nøstvet/Lihult- and Trinnaxes I have trained multiple ML with several different ML-approaches. I have also programmed new custom ML`s and used pre-trained models in my research for the optimal accuracy in training, validation and prediction. My dataset has varied from 500 to 1300 pictures of stone age axes. After the training, validation and test sets

I used the ML to give its prediction on 10 unknown stone age axes, to showcase the ML’s further accuracy. The pretrained model managed to learn much faster, as was expected, and managed to produce an accuracy of 94.7 %. What was especially interesting was to see whether the ML managed to learn, through its training, to classify between the two similar axes that also can vary within its class. Thereby it was a great challenge for the ML, but an important one to research whether it can learn and predict with a high accuracy. The researched showcased and raised important questions regarding future work within classification of artefacts with an ML-approach and potentially future and alternative usage of MLapplications. I have also started researching how the model predicts on fragments of the two axes with interesting results.

The use of R-CNNs in the automated detection of archaeological objects in LiDAR data

Verschoof-van der Vaart, Wouter Baernd and Karsten Lambers

To cope with the ever growing set of largely digitally and easily available remotely sensed data, computer-aided methods for the (semi-) automatic detection of archaeological objects are needed. This research project explores the implementation of R-CNNs (Region-based Convolutional Neural Networks) in order to develop a generic, flexible, and robust automated detection method for archaeological objects in LiDAR data.

In this paper our improved workflow will be presented. This workflow is based on our prior developed WODAN workflow, that served as a proof of concept. The improved workflow has been trained and tested on LiDAR data gathered from a forested area in the central part of the Netherlands, the Veluwe.

This area contains a multitude of archaeological objects, including (Prehistoric) barrows, Celtic fields, and (Medieval) charcoal kilns. By implementing this improved workflow we have been able to develop a method to automatically detect and categorize these archaeological objects. We will present the results of the tests done with the improved workflow on the developed testing datasets. A comparison will be made between the results of the prior workflow and the results of a large scale citizen science project, called Heritage Quest, run in the same area.

Notes