HOW TO COMBINE LIDAR AND VERY HIGH RESOLUTION MULTISPECTRAL IMAGESFOR FOREST STAND SEGMENTATION?

Clément Dechesne, Clément Mallet, Arnaud Le Bris, Valérie Gouet-Brunet

Univ. Paris-Est, LASTIG MATIS, IGN, ENSG, F-94160 Saint-Mande, France

ABSTRACT

Forest stands are a basic unit of analysis for forest inventoryand mapping. Stands are defined as large forested areas of ho-mogeneous tree species composition and age. Their accuratedelineation is usually performed by human operators throughvisual analysis of very high resolution (VHR) infra-red andvisible images. This task is tedious, highly time consum-ing, and needs to be automated for scalability and efficientupdating purposes. The most appropriate fusion of two re-mote sensing modalities (lidar and multispectral images) isinvestigated here. The multispectral images give informationabout the tree species while 3D lidar point clouds providegeometric information. The fusion is operated at three dif-ferent levels within a semantic segmentation workflow: over-segmentation, classification, and regularization. Results showthat over-segmentation can be performed either on lidar or op-tical images without performance loss or gain, whereas fusionis mandatory for efficient semantic segmentation. Eventually,the fusion strategy dictates the composition and nature of theforest stands, assessing the high versatility of our approach.

Index Terms— Lidar, multispectral imagery, fusion, for-est stands, classification, segmentation.

1. INTRODUCTION

The fusion of multimodal data has been widely investigatedin the remote sensing literature [1, 2, 3]. Recent surveys dis-cuss the different levels of fusion (original measured domain,feature spaces and/or classification level) and underline thesuperior performance of the joint use of lidar and hyperspec-tral data for various classification challenges. While urbanareas are heavily documented, no genuine study has been car-ried out for forested environments so far.The study of forested areas from a remote sensing point ofview can be operated at different levels: pixel, object (mainlytrees) or stand. Forest stands are among the basic units for for-est analysis. It can be defined in many different terms (specie,age, height, maturity). Most of the time in national forestinventories, for reliability purposes, each area is manually in-terpreted by human operators using VHR geospatial imageswith an infra-red channel [4].Airborne laser scanning (ALS) and VHR multispectral im-ages are both well adapted remote sensing data for stand seg-

mentation. ALS provides information about the vertical dis-tribution of the trees while multispectral images are useful forthe tree species discrimination. Surprisingly, only a few meth-ods have addressed the forest stand segmentation issue.The analysis of the lidar and multispectral data is performedat three levels in [5], following the hierarchical nomenclatureof forest classes. The multi-scale analysis offers the advan-tage of alleviating the standard limitations of individual treecrown detection, and of retrieving labels related to forest de-velopment stage. Nevertheless, the pipeline is highly heuris-tic, under-exploits lidar data, and significant confusions be-tween classes are reported.The automatic segmentation of forests in [6] is also performedwith lidar and VHR multispectral images. The idea is to di-vide the forests into higher and lower sections with the heightinformation provided by lidar point clouds. An unsupervisedclassification process is applied and pre-defined thresholds al-low to obtain the desired delineation of stands. The resultsare improved using standard morphological operators. Thismethod is efficient if the canopy structure is homogeneousand requires a strong knowledge on the area of interest. Basedon height information only, it cannot differentiate two standsof similar height but different species.In [7], a stand segmentation technique for a forest composedof Scots Pine, Norway Spruce and Hardwood is defined. A hi-erarchical segmentation on the Crown Height Model followedby region growing is performed on images composed of ras-terized lidar data and colored infra-red images. The processwas only applied on a limited area, preventing from drawingstrong conclusions. However, the quantitative analysis car-ried out by the authors shows that lidar data can help to definestatistically meaningful stands and that multispectral imagesare inevitable inputs for tree species discrimination.This paper will focus on the method proposed in [8], as it al-ready delineates accurately the forest stands, according to thetree species, and investigates the possibilities to fuse ALS andmultispectral data at different levels (classification and regu-larization). Here, the approach has been improved in orderto experiment the fusion at the over-segmentation model (ba-sic tree extraction being substituted by any kind of superpixelsegmentation technique). The aim of this paper is to deter-mine which modality is the most relevant in each step, or ifboth are needed.

2. METHODS

The investigated method is composed of three main steps(Figure 1, more details in [8]). First, 25 features are derivedfrom the airborne lidar point clouds and 70 from the multi-spectral images. They are computed at the pixel level, forthe subsequent energy minimization process. They are alsogenerated at the object level (see below). Indeed the featuresare more consistent at this scale which improves the discrimi-nation process. Then, the classification of the vegetation type(mainly tree species) is performed using a Random Forestclassifier. Training samples are automatically extracted fromthe French national land-cover (LC) forest database [8]. Last,a smoothing regularization process based on an energy mini-mization framework is carried out in order to obtain the finalsegments.

Pixel-basedlidar features

Pixel-basedspectral features

Pixel-basedfeatures

Object-basedlidar features

Object-basedspectral features

Object-basedfeatures

Over-segmentation

1

Feature selectionClassification

2

Regularization

3

1, 3,4, 5

2, 6,7, 8

1

2

3, 4, 5,6, 7, 8

5, 8

2, 4, 7

1, 3, 6

Fig. 1. Flowchart of the investigated method. The purplenumbers correspond to the scenarii investigated (Table 1), thered boxes to the nodes where the fusion is operated.

In the investigated method, the two different data typescan be used together or separately at the three main steps;

1. Over-segmentation. It can be operated either on the li-dar point cloud (individual tree crown extraction –ITC–or any segmentation based on a rasterized lidar feature)or on the image itself (superpixels). The aim is justto efficiently reduce the spatial information. It is com-monly assumed that ITC is mandatory. However, here,we investigated whether it leads to superior accuracywith respect to standard superpixel techniques.

2. Classification. It can be performed using only the lidarfeatures, only the spectral features or using both.

3. Regularization. Conversely to a standard MarkovRandom Field, our smoothing procedure integratesdata features. The pixel-based features used in the reg-ularization can also be only the lidar features, spectralfeature or both.

With regard to all the 18 possibilities, 8 scenarii are investi-gated in this work since they allow to determine efficientlywhere the fusion is the most appropriate. They are summa-rized in Table 1.

# Over-seg. Classification Regularization1 Lidar Lidar Lidar2 Spectral Spectral Spectral3 Lidar Lidar+Spectral Lidar4 Lidar Lidar+Spectral Spectral5 Lidar Lidar+Spectral Lidar+Spectral6 Spectral Lidar+Spectral Lidar7 Spectral Lidar+Spectral Spectral8 Spectral Lidar+Spectral Lidar+Spectral

Table 1. The 8 scenarii investigated for optimal data fusion.

Four segmentation methods were tested:

SITC : A coarse 3D-based single tree extraction developed ina previous study (for lidar only) [8];

SMSS : A hierarchical 2D multiscale segmentation approach(applied on any rasterized lidar feature) [9];

SPFF : A standard graph-based image segmentation, re-stricted to optical images (PFF) [10];

SSLIC : A widely-adopted superpixel segmentation method(SLIC) on optical images [11].

The classification is carried out with the Random Forest (RF)classifier, fed with features extracted from the 95 attribute set,depending on the scenario. However, in order to reduce thecomputation times and to validate the complementarity of thedata sources, a selection of 20 feature is carried out with theSFFS algorithm [12] before the classification process.

3. DATA

The experiments are conducted over a mountainous forest inthe East of France, which exhibits a high diversity of land-scapes and tree species. We focus on four areas of 1 km2.The airborne multispectral images were captured by the IGNdigital cameras [13]. They have 4 bands: 430-550 nm (blue),490-610 nm (green), 600-720 nm (red) and 750-950 nm (nearinfra-red) at 0.5m ground sample distance. The airborne lidardata were collected using an Optech 3100EA device (0.8 mfootprint). The point density ranges from 2 to 4 points/m2.Multispectral and lidar data fit with the standards used inmany countries for large-scale operational forest mappingpurposes [14]. Data were acquired under leaf-on conditions,in May and June 2011 for the multispectral images and thelidar data, respectively.The forest LC geospatial database is composed of 2D poly-gons delineated by photo-interpreters. It is the French na-tional LC datum for forests, available for any end-user1. Fourtraining sets were generated, depending on the input of theover-segmentation and classification.

1http://inventaire-forestier.ign.fr/spip/?rubrique67

4. RESULTS

The overall accuracy is computed by comparing each pixel ofthe forest LC database with the classification results.The segmentation results of the scenarii 5 and 8 over a singlezone are presented in Table 2. The lidar-based segmenta-tion are SITC (performed on the lidar point cloud) and SMSS(conducted on the lidar Digital Surface Model obtained).SPFF and SSLIC are both applied on the three visible channelsof the multispectal image. The results show that the over-segmentation has a weak impact on the final result. A simplesuperpixel approach is sufficient to have a good segmentationand regularization accuracy. However, this step has a sig-nificant impact on the classification results (∼ +5% for thespectral based over-segmentation). Same results are observedon the other zones.

Scenario 5 Scenario 8ITC MSS PFF SLIC

Classification 80.9% 81.8% 87.6% 86.5%Regularization 94.1% 95.8% 96.2% 95.7%

Table 2. Classification and stand segmentation accuracy withdifferent segmentation methods. The ground truth is providedby the forest LC database.

The feature selection step confirms that both data are im-portant for classification: over 40 runs of the SFFS algorithmon the different areas, 60% of the selected features are derivedfrom the multispectral images and 40% are derived from lidar.This shows the complementarity of the data sources. It is im-portant to keep both in the subsequent classification task.The results of the scenarii 1, 2, 5 and 8 over the 4 zonesare presented in Figure 2. From a visual point of view, theresults are satisfactory. Similar results are observed for theother zones. The results of the 8 proposed scenarii over a sin-gle zones are presented in Table 3. In terms of performance,lidar data provide worse results: the classification has an over-all accuracy of 74.8%. The regularization step improves theresults, with an overall accuracy of 92.2%. Multispectral dataalone produce better results: the classification has an overallaccuracy of 79.1% and the regularization reaches 95.2%.The fusion of the two data sources has a significant impact onthe classification step and on the final results (scenario 1 vs 3and 2 vs 7). Adding the lidar data increases the classifica-tion accuracy of ∼ 8% (79.1% → 87.6%) and adding thespectral data increases the classification accuracy of ∼ 7%(74.8% → 81.8%). The gain is less significant for the finalresults; + ∼ 2.5% (92.2% → 94.8%) when adding spectralinformation and + ∼ 1% (95.2% → 96.1%) when addinglidar information. These results confirm that it is meaningfulto use both data sources for classification. They provide com-plementary information for vegetation type discrimination.The spectral information is more beneficial than the lidar in-formation for the regularization step. Fusing the two datasources or using only the spectral information in the regu-

larization step does not significantly change the final results.Same results are observed on the other zones.

Scenario Overall accuracy (%)Classification Regularization Gain

1 74.8 92.2 17.42 79.1 95.2 16.13

81.894.8 13.0

4 95.8 14.05 95.8 14.06

87.696.0 8.4

7 96.1 8.58 96.2 8.6

Table 3. Classification and stand segmentation accuracies forthe 8 scenarii investigated.

5. CONCLUSIONS

The contribution of two major very high resolution remotesensing data sources for forest analysis has been investigatedfor the specific problem of forest stand segmentation. Theover-segmentation step can be carried out by any data type,since the obtained objects are consistent for both scenarios(individual tree crown extraction vs superpixels). No spe-cific effort should be put on 3D analysis for tree extraction.One should select techniques with limited parameter tun-ing. The classification must be carried out on both remotesensing sources, otherwise, the segmentation quality greatlydecreases. The regularization can be operated with only onedata type or two. In both cases, segments are spatially con-sistent with the existing forest geodatabase. When using onlylidar data, the final stand will have an homogeneous verti-cal structure while inserting multispectral-based features willlead to stands more homogeneous in term of species. Never-theless, when using both remote sensing sources, the standsare better delimited compared to the Forest LC map.

6. REFERENCES

[1] M. Dalla Mura, S. Prasad, F. Pacifici, P. Gamba,J. Chanussot, and J.A. Benediktsson, “Challenges andOpportunities of Multimodality and Data Fusion in Re-mote Sensing,” Proceedings of the IEEE, vol. 103, no.9, pp. 1585–1601, 2015.

[2] L. Gomez-Chova, D. Tuia, G. Moser, and G. Camps-Valls, “Multimodal classification of remote sensing im-ages: A review and future directions,” Proceedings ofthe IEEE, vol. 103, no. 9, pp. 1560–1584, 2015.

[3] Michael Schmitt and Xiao Xiang Zhu, “Data fusion andremote sensing: An ever-growing relationship,” IEEEGeoscience and Remote Sensing Magazine, vol. 4, no.4, pp. 6–23, 2016.

[4] A. Kangas and M. Maltamo, Forest inventory: method-ology and applications, vol. 10, Springer Science &Business Media, 2006.

[5] D. Tiede, T. Blaschke, and M. Heurich, “Object-basedsemi automatic mapping of forest stands with laser scan-ner and multi-spectral data,” International Archives of

(A1) Orthoimage/DSM. (A2) Forest LC database. (A3) Scenario 1 result. (A4) Scenario 2 result. (A5) Scenario 5 result. (A6) Scenario 8 result.

(B1) Orthoimage/DSM. (B2) Forest LC database. (B3) Scenario 1 result. (B4) Scenario 2 result. (B5) Scenario 5 result. (B6) Scenario 8 result.

(C1) Orthoimage/DSM. (C2) Forest LC database. (C3) Scenario 1 result. (C4) Scenario 2 result. (C5) Scenario 5 result. (C6) Scenario 8 result.

(D1) Orthoimage/DSM. (D2) Forest LC database. (D3) Scenario 1 result. (D4) Scenario 2 result. (D5) Scenario 5 result. (D6) Scenario 8 result.

Fig. 2. Stand segmentation results for different scenarii over 4 zones of 1 km2. Color code : deciduous oaks, beech,chestnut, robinia, Scots pine, black pine, fir or spruce, larch, Douglas fir, non-pectinated fir, herbaceous

formation, No data.

Photogrammetry, Remote Sensing and Spatial Informa-tion Sciences, vol. 36(8/W2), pp. 328–333, 2004.

[6] O. Diedershagen, B. Koch, and H. Weinacker, “Auto-matic segmentation and characterisation of forest standparameters using airborne lidar data, multispectral andfogis data,” International Archives of Photogrammetry,Remote Sensing and Spatial Information Sciences, vol.36(8/W2), pp. 208–212, 2004.

[7] V. Leppänen, T. Tokola, M. Maltamo, L. Mehtätalo,T. Pusa, and J. Mustonen, “Automatic delineation offorest stands from lidar data,” International Archives ofthe Photogrammetry, Remote Sensing and Spatial Infor-mation Sciences 38(4/C1), pp. 5–8, 2008.

[8] C. Dechesne, C. Mallet, A. Le Bris, and V. Gouet-Brunet, “Semantic segmentation of forest stands of purespecies combining airborne lidar data and very high res-olution multispectral imagery,” ISPRS Journal of Pho-togrammetry and Remote Sensing, vol. 126, pp. 129–145, 2017.

[9] L. Guigues, J.-P. Cocquerez, and H. Le Men, “Scale-sets image analysis,” IJCV, vol. 68, no. 3, pp. 289–317,2006.

[10] P. F Felzenszwalb and D. P Huttenlocher, “Efficientgraph-based image segmentation,” IJCV, vol. 59, no.2, pp. 167–181, 2004.

[11] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, andS. Süsstrunk, “SLIC superpixels compared to state-of-the-art superpixel methods,” IEEE TPAMI, vol. 34, no.11, pp. 2274–2282, 2012.

[12] P. Pudil, J. Novovicová, and J. Kittler, “Floating searchmethods in feature selection,” Pattern Recognition Let-ters, vol. 15, no. 11, pp. 1119–1125, 1994.

[13] J.-P. Souchon, C. Thom, C. Meynard, and O. Martin, “Alarge format camera system for national mapping pur-poses,” RFPT, vol. 200, pp. 48–53, 2012.

[14] J.C. White, N.C. Coops, M.A. Wulder, M. Vastaranta,T. Hilker, and P. Tompalski, “Remote sensing technolo-gies for enhancing forest inventories: A review,” CJRS,vol. 42, no. 5, pp. 619–641, 2016.