skull identification via correlation measure between skull and face shape

1322 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 8, AUGUST 2014

Skull Identification via Correlation MeasureBetween Skull and Face Shape

Fuqing Duan, Yanchao Yang, Yan Li, Yun Tian, Ke Lu, Zhongke Wu, and Mingquan Zhou

Abstract— Skull identification is an important subject forresearch in forensic medicine. Current research can bedivided into two categories: 1) craniofacial superimposition and2) craniofacial reconstruction. Both categories rely essentiallyon the accurate extraction and representation of the intrinsicrelationship between the skull and face in terms of the morphol-ogy, which still remain unsolved. They have high uncertaintyand a low identification capability. This paper proposes a novelskull identification method that matches an unknown skull withenrolled 3D faces, in which the mapping between the skulland face is obtained using canonical correlation analysis. Unlikeexisting techniques, this method needs no accurate relationshipbetween the skull and face, and measures only the correlationbetween them. In order to measure the correlation more reliablyand improve the identification capability of the correlationanalysis model, a region fusion strategy is adopted. Experimentalresults validate the proposed method, and show that the region-based method can significantly boost the matching accuracy. Thecorrect identification rate reaches 94% when using a CT dataset. This paper can provide a theory support for research oncraniofacial superimposition and craniofacial reconstruction.

Index Terms— Forensic medicine, canonical correlationanalysis, skull identification.

I. INTRODUCTION

IN MANY forensic cases, only the skulls of the victims areavailable, and there is no other evidence. This circumstance

makes some classical identification techniques unusable. Skullidentification [1], one of the most important tasks in forensicmedicine, is to determine the identity of a victim by theskull. Skull identification has been researched for over acentury. Since the successful identification of relics of thefamous German composer Bach in 1895, skull identificationhas drawn wide attention and been applied in huge numberof forensic cases, ranging from the identification of victims

Manuscript received October 18, 2013; revised February 11, 2014 andMay 6, 2014; accepted June 18, 2014. Date of publciation June 25, 2014;date of current version July 11, 2014. This work was supported in part bythe National Natural Science Foundation of China under Grant 61272363 andin part by the Program for New Century Excellent Talents in University,under Grant NCET-13-0051. The associate editor coordinating the reviewof this manuscript and approving it for publication was Prof. Sviatoslav S.Voloshynovskiy.

F. Duan, Y. Yang, Y. Li, Y. Tian, Z. Wu, and M. Zhou are with theCollege of Information Science and Technology, Beijing Normal University,Beijing 100875, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]).

K. Lu is with the College of Engineering and Information Technology,University of Chinese Academy of Sciences, Beijing 100049, China (e-mail:[email protected]).

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TIFS.2014.2332981

of the Indian Ocean tsunami [2] to the identification ofterrorists [3]. With the development of digitization tech-nologies such as CT, 3D scanners etc., data acquisitionbecomes easier and easier, and human identification byskulls has been an interdisciplinary research focus of infor-matics, anthropology, forensic sciences, and so on. Unlikeother biological features such as DNA and fingerprinting,skull data for a living person still cannot be acquired ina convenient and non-intrusive way. Thus, skull identifi-cation cannot be realized in the same way as used byother biological feature identification technology, i.e., match-ing an unknown skull with a large skull database. Currentskull identification research focuses mainly on two cate-gories. One is craniofacial superimposition [6], [7], [14]–[18],and the other is craniofacial reconstruction [4], [5], [19]–[24].

Craniofacial superimposition firstly overlays the 2Dprojection of a 3D model of the unknown skull with a facephoto of the missing person according to the same pose, andthen, it makes an identification decision by assessment ofthe anatomical consistency of the cephalometric landmarks onthe face photo and the skull projection. The superimpositionis usually performed manually by forensic anthropologistsand requires special utilities and professional qualifications.Thus, the procedure of the identification is time-consumingand easily influenced by the subjectivity of the practitioners.Recently, a few automatic skull-face overlay methods havebeen proposed. These methods are based on evolutionaryalgorithms [14], [15], fuzzy logic [17], [18], and so on. Forexample, Ballerini et al [14] used a genetic algorithm to findthe optimal transformation to match the landmarks on the3D skull model and the face photo. Ibanez et al [18] usedfuzzy logic to solve the uncertainty involved in the locationof the cephalometric landmarks. Automatic methods needno tedious manual operation and can be easily reproduced.However, regardless of whether the method is automatic ornon-automatic, it is difficult to fit a 2D image (the face photo)onto a 3D object (the skull). The reason is that the skull andface are two objects of different natures [16], which causesome inherent uncertainty in the matching. Moreover, accuratelocalization of craniofacial landmarks has been a longstand-ing problem in the field, and the uncertainty introduced bylandmark localization also affects the matching.

Craniofacial reconstruction aims to estimate an individual’sface appearance from its skull using the relationship betweensoft tissues and the underlying bone structure. It can providea clue and trigger recognition by the victim’s relatives, sothat further identification evidence can be collected on a

1556-6013 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

DUAN et al.: SKULL IDENTIFICATION VIA CORRELATION MEASURE 1323

restricted list of candidates. Currently, craniofacial reconstruc-tion is implemented manually by anatomists or artists. Theyphysically model a face by adding clay or plasticine to askull replica, relying on their experience. The reconstructionprocedure is time consuming and prone to subjectivity. Mostearly computerized reconstruction methods [19], [20] obtainedthe face of the unknown target skull by deforming a cranio-facial reference. They deform the reference skull to the targetskull according to some skull features, and subsequently applyan extrapolation of the skull deformation to the referenceface to obtain the reconstructed face. They assume that thereference and the unknown target individual have similartissue thickness distributions. A model bias or unrealisticreconstructions always appear when an inappropriate referenceis chosen [5]. Recent methods use statistical learning tech-niques, such as statistical shape models [21], [22], regressionmodels [23], [24], etc., to explore the relationship between theskull and face, and the reconstruction is based on the learnedrelationship. However, the obtained statistical models depictonly the craniofacial variation in a statistical sense and cannotreflect the personal variation. Thus, most reconstructed facesare not sufficiently accurate for identification.

In essence, both craniofacial superimposition and craniofa-cial reconstruction are based on a good grasp or representationof the relationship between the skull and face in terms of themorphology. However, how to accurately extract and representthe relationship is intractable and is still unsolved. Theyhave a high uncertainty and a low identification capability.With the progress of 3D imaging technology, we expect thatnon-intrusive 3D face data capture [8], [9] will become readilyavailable and cost efficient. More importantly, 3D face model-ing from images has received substantial attention [10]–[12].So we believe that the acquisition of 3D face geometric modelswill become easier and easier.

In this paper, we propose to identify an unknown skullthrough using a correlation measure between the 3D skulland 3D face in terms of the morphology, and measure thecorrelation using canonical correlation analysis (CCA) [13].We use the 3D skull data as the probe and 3D face geometricdata as the gallery, and match the unknown skull with enrolled3D faces by the correlation measure between the probe andthe gallery. Considering that the strength of the correlationbetween the skull and face in different craniofacial regions isnot the same, we propose a region fusion strategy to measurethe correlation between the skull and face more reliably andto boost the identification capability. The proposed methodreduces the complexity of the skull identification problem.Unlike craniofacial superimposition and craniofacial recon-struction, it is not necessary to have an accurate relationshipbetween the skull and face; the system measures only thecorrelation between them. The contributions of this paper areas follows: Firstly, a novel and reliable skull identificationtechnique is presented, and to the best of our knowledge,it is the first time that such a technique has been reported.Secondly, this paper proves that there is indeed a close relationbetween the skull and face in terms of the morphology, andthus, it provides a theory support for related research, i.e.craniofacial superimposition and craniofacial reconstruction.

Thirdly, we show that the region-based strategy is betterthan the holistic strategy in terms of the extraction of therelationship between the skull and face; this finding supportsfurther research on regional craniofacial reconstruction [22].

The remainder of this paper is organized as follows.Section II introduces the materials. Section III describes theproposed skull identification method. Experimental results andanalysis are shown in Section IV. A discussion is provided inSection V. Finally, we give the conclusions in Section VI.

II. MATERIALS

Our study has been approved by the Institutional ReviewBoard (IRB) of the Image Center for Brain Research, NationalKey Laboratory of Cognitive Neuroscience and Learning,Beijing Normal University. This study was conducted on adatabase of 208 whole head CT scans from voluntary personswho mostly came from the Han ethnic group in the Northof China, ages 19–75 years for females and 21-67 years formales. There were 93 females and 115 males. The CT imageswere obtained by a clinical multi-slice CT scanner system(Siemens Sensation 16) in the Xianyang Hospital located inwestern China. The images of each subject were restored inDICOM format with a size of approximately 512×512×250.The original CT slice images were processed by the Sobeloperator model after filtering out the noise to extract theskull and face borders. The 3D skull and skin surfaces arereconstructed by a marching cubes algorithm [25], and theyare represented as triangle meshes that include approximately1 50,000 and 2 20,000 vertices, respectively. All of the headsare substantially complete. In detail, each skull contains allof the bones from calvarias to jaw and has a full mouthof teeth, and no face has a missing part. In addition, thesubject’s properties for each head, such as the age, gender, andbody mass index (BMI), are stored. All of the information isself-reported by the participants. More of the details on theprocedure for the data processing can be found in [26].

To eliminate the inconsistency in the position, pose andscale caused by data acquirement, all of the samples aretransformed into a uniform coordinate system. The uniformcoordinate system is determined by four skull landmarks, theleft porion, right porion, the left (or right) orbitale and theglabella (denote as Lp, Rp, Lo, G). From three points, Lp,Rp, Lo, the Frankfurt plane is determined [22]. The coordinateorigin (denoted as O) is the intersection point of the line LpRpand the plane that contains point G and orthogonally intersectsthe line LpRp. We take the line ORp as the x−axis. Thez−axis is the line through the point O, and it has the directionof the normal of the Frankfurt plane. Then, the y−axis isobtained by the cross product of the z− and x−axis. Oncethe uniform coordinate system is defined, all of the prototypicskins and skulls are transformed into it by a simple rotationand translation transformation. Finally, the scale of all of thesamples is standardized by setting the distance between Lp andRp to a unit, i.e., each vertex (x, y, z) of the skull and skin isscaled by

∥∥L p − Rp

∥∥. The skull and face skin of one sample

in the uniform coordinate system are shown in Figure.1.The original skull and skin meshes have different

connectivity. In statistical learning, a dense point correspon-


Fig. 1. One pair of skull and face in the uniform coordinates system.

Fig. 2. The reference skull and face skin for data registration.

dence must be established across the training set. As is known,the representation capability of statistical models dependscritically on the quality of the correspondence. An inappro-priate correspondence can cause the variation modes of thestatistical model to not reflect the actual variation of theobjects. Although there are many data registration methodsfor dense mesh or point cloud objects to establish the pointcorrespondence, it is still a challenging problem to obtainaccurate registration for dense skull and face meshes becausethere are complex non-rigid deformations among differentcraniofacial subjects. Here, we adopt the dense registrationmethod described in [22]. It includes two steps. The firststep is a non-rigid registration procedure using a fixed ref-erence. The second step is to improve the registration by alinear combination model based on the first-step registeredsamples. This method reduces the model bias caused by afixed reference at the cost of higher computational complexity.Fortunately, we do not need a real-time registration. Similarto the literature [22], we select one skull-and-face pair as areference and cut away their back parts, considering that facerecognition mainly depends on the front part of the head. Asshown in Figure 2, the reference skull and face have 41,059and 40,969 vertices, respectively. After the data registration, allof the skulls and face skins have the same mesh connectivityas the reference one.

III. METHODS

In skull identification, nearly all of the methods dependon accurate extraction and representation of the relationshipbetween the skull and face. However, it is very difficult toextract this complex relationship. Because this work aims toidentify an unknown skull by looking for its correspondingface skin from a 3D face gallery, we measure only thecorrelation between a skull and a face skin, and do not needan accurate relationship.

CCA is a powerful multivariate statistical analysis methodfor measuring the linear relationship between two multidi-mensional variables. It finds two basis vectors that maximizethe data correlation, one for each multidimensional variable.Recently, CCA has been widely applied in many areas, suchas face matching [27], data regression analysis [28], and facialexpression recognition [29]. In this study, we use CCA tomeasure the linear relationship between the skull and face.However, it is difficult to perform CCA in the original high-dimensional feature spaces of skulls and face skins because ofthe computational problem of the eigen-value decompositionof gigantic matrices.

Statistical shape model is a widely used technique inmedical image analysis. It can efficiently describe the shapevariance and ensure that only statistically likely shapes arerepresented. In this work, we firstly build a statistical shapemodel for skulls and face skins respectively, by which theface and skull data are projected into their low-dimensionalshape parameter spaces, and then, CCA is performed in theshape parameter spaces. The skull identification is realizedbased on the CCA model.

A. Statistical Shape Model

Principle component analysis (PCA) is a powerful tool forbuilding statistical shape models. PCA finds the major andminor modes of the shape variation across the training datasetand represents a mean normalized shape as a combination ofvariation modes.

By concatenating the coordinates of all of the vertexes, askull or a face skin can be represented as a high-dimensionalvector. Thus, we obtain a training dataset of skulls {Si =(x S

i1, yS

i1, zS

i1, . . . , x S

im, yS

im, zS

im)T|i = 1, 2, . . . , N } and face

skins {Fi = (x Fi1

, y Fi1

, zFi1, . . . , x F

in, y F

in, zF

in)T|i = 1, 2, . . . , N },

where N, m, n denote the training sample size, the vertexnumbers of the skull and face skin, respectively, and eachcoordinate index labels corresponding points across thetraining set. From the skull dataset, the mean skull dataS and the covariance matrix � of the mean normalizedskulls were calculated. PCA essentially transforms the meannormalized shape data into a subspace spanned by theorthogonal unit eigen-vectors Uk, k = 1, 2, . . . , N − 1 ofthe covariance matrix in descending order according to theirassociated eigen-values λk . Usually, these unit eigen-vectorsrepresent variation modes of the training dataset. Then, theskull statistical shape model is constructed as the followingparameterized model:

S(a) = S +p

∑

k=1

akUk (1)

where the mode number p is determined by a variance contri-bution rate calculated from the cumulative eigen-values, e.g.,98%, and the combination coefficient a = (

a1, a2, . . . , ap)T

is the shape model parameter. Apparently, this statisticalshape model assumes that the shape vector S obeys a normaldistribution with a mean of S and a covariance matrix �, insuch a way that the model parameter a for a plausible skull hasa normal distribution with zero mean and covariance matrix


Fig. 3. Model parameter determination procedure.

diag(

λ1, λ2, . . . , λp)

. Similarly, a face statistical shape modelcan be constructed by using the same procedure:

F(c) = F +q

∑

k=1

ckVk (2)

where F is the mean face data, and Vk, k = 1, 2, . . . , q areorthogonal eigen-vectors of the covariance matrix of the meannormalized face skins.

Statistical shape model matching is to determine the shapemodel parameter for given shape data. If the given skull dataS0 is aligned, the shape model parameter can be determinedby a PCA transform. Let Ps = [

U1, U2, . . . , Up]

denote thePCA transform matrix for the skulls. The model parameter canbe determined according to the statistical shape model (1), asfollows:

a = PTs

(

S0 − S)

(3)

Thus, in fact, the procedure of the model parameter determi-nation for a given skull is the one of skull data registration.Compared with a static reference used in the usual dataregistration [26], a statistical shape model can be seen asa dynamic reference controlled by the model parameters.A dynamic reference can restrain the registration bias causedby a static reference. Figure 3 shows a statistical shape model-based registration algorithm whose output is the shape modelparameter. As shown in Figure 3, for given target prototypicskull data S0, a dynamic reference, denoted as Sr , is updatedby the statistical shape model parameter a in each loop, whichis determined by the PCA transform (i.e., Eq (3)) of the

corresponding aligned sample S0 of the last loop. The initialmodel parameter is set to a = 0, i.e., the initial referenceis the mean skull S. The iterative closest point algorithm(ICP) [30] is used to align the given target prototypic skull datato the reference. ICP is a rigid registration algorithm. It is wellknown that the smaller the deformation between the target andthe reference is, the better the effect of the ICP registration.Apparently, the dynamic reference will be closer and closerto the given skull along with the iterating, and thus, theiteration will converge. When the shape model parameter doesnot change, the iteration stops. In our experiments, usually10 iterations are enough. For the huge data of the skull orface, the ICP searching procedure is very time consuming.Similar to the literature [22], a K dimensional binary searchtree (KD-tree) technique is adopted in the ICP searching toget high efficiency. Similarly, the shape model parameter fora given face can be determined by using the same procedure.

B. CCA in the Shape Parameter Spaces

As described above, each training skull or face skin can beprojected into its shape parameter space. That is, each skullhas p feature variables, while each face skin has q featurevariables. Let XN×p and YN×q denote the training skull andface data matrices, respectively, and let N denote the size ofthe training samples. The aim of CCA is to find two sets ofbasis vectors, wx ∈ Rp and wy ∈ Rq , that maximize thecorrelation coefficient between the components t1 = Xwx andu1 = Ywy, i.e.,

ρ = maxwx,wy

cov(t1, u1)√Var(t1) × Var(u1)

= maxwx,wy

wTx Cxywy

√

wTx Cxxwx × wT

y Cyywy(4)

where the covariance matrices Cxy = XTY, Cxx = XTX andCyy = YTY.

Let W = (

wTx , wT

y)T and Cyx = YTX, then Equation (4)

can be solved by computing a generalized eigen-value decom-position problem, as follows:

AW = λBW (5)

where

A =(

0 CxyCyx 0

)

, B =(

Cxx 00 Cyy

)

. (6)

More details on the derivation and solution of CCA can befound in [13].

Thus, we obtain two subspaces that consist of the basisvectors wx ∈ Rp and wy ∈ Rq , respectively, for the skull andface skin. Assume that Wx denotes the subspace projectionmatrix whose columns are the basis vectors wx, and Wy isthe matrix that corresponds to the basis vectors wy. For askull-and-face pair, let x denote the shape parameter vectorof the skull, and let y denote the shape parameter vector ofthe face. Then, the feature vectors of the skull and face in the


Fig. 4. Framework of CCA based skull identification. The source X isthe skull training dataset, and Y is the corresponding skin training dataset,Wx, Wy are the CCA projection matrices obtained in training phase.

CCA subspaces are

xc = WTx x

yc = WTy y. (7)

We define the matching score between the skull and face asfollows:

r(xc, yc) = 〈xc, yc〉‖xc‖ ‖yc‖ (8)

where 〈·, ·〉 denotes the inner product operation.

C. Framework of the Skull Identification

The aim of this work is to find the most probable face from a3D face gallery for an unknown skull. Thus, we must computethe matching score between the unknown skull and each 3Dface in the gallery and find the face with the highest matchingscore. The general framework for the skull identification isshown in Figure 4. Similar to typical machine learning-basedapplications, the procedure of the skull identification includestwo phases, the training phase and the identification phase.

As Figure 4 shows, in the training phase, the correlationanalysis model is established. First, statistical shape modelsfor skulls and faces are constructed using all of the trainingdata, as Equations (1) and (2) describe; second, all of thetraining data are projected into the shape parameter spacesaccording to Equation (3); and third, CCA is performed forthe shape parameter features of the training skulls and faceskins, and two basis vectors, wx and wy, are obtained. In theidentification phase, the correlation analysis model is used toidentify an unknown skull. First, the unknown skull and theface gallery are projected respectively into the shape parameterspaces by the statistical shape model matching describedabove; second, their shape parameter features are projectedinto the CCA subspaces according to Equation (7). Finally,the matching score between the unknown skull and each facein the gallery is computed according to Equation (8), and theface with the highest matching score is the identification result.

The identification result is surely not correct if the facegallery in a forensic case does not contain the face data of theunknown skull. Thus, it is important to evaluate the confidence

of the identification result. Here, we use Bayes rule to computethe probability that the result is correct. For the identificationresult, we have two class labels, positive and negative. Let w1denote the positive status, and let w2 denote the negative status.According to Bayes rule, we compute the posterior probabilitythat the matching score r belongs to the positive class, asfollows:

P(w1|r) = p(r |w1)P(w1)

2∑

j=1p(r |w j )P(w j )

(9)

where P(w1) and P(w2) denote the prior probabilities, andp(r |w1) and p(r |w2) denote the class conditional probabilitydensity functions. If P(w1|r) > 0.5, the matching is classifiedas a positive matching by the Bayes decision rule.

Let E denote the event that the face data of the unknownskull exists in the face gallery, and let P(E) denote theprobability of the event; then, we have

P(w1) = P(w1, E) = P(E)P(w1|E)

P(w2) = 1 − P(w1) (10)

where P(w1|E) denotes the correct identification probabilitywith the condition that the face data of the unknown skullexists in the face gallery. P(E) is determined by prior informa-tion, and the conditional probability P(w1|E) can be estimatedaccording to the correct identification rate of a test set. Theclass conditional probability density functions p(r |w1) andp(r |w2) can be estimated by the matching score distributionof a data set. A detailed implementation is shown in theexperiments and analysis section.

D. Identification Based on Region Fusion

Human craniofacial morphology is very complex, and thestrength of the correlation between the skull and face indifferent craniofacial regions is not the same. For example,the correlation in the forehead region is strong because thesoft tissue thickness in this region is thin and the morphologyin this region is similar for the skull and face of the samehuman, while the correlation is weak in some regions, such asthe nose, eyes and mouth, because of the complex structure inthese regions. It is obvious that the region of strong correlationhas a high identification capability. However, a single holisticCCA model reflects only the correlation between the skulland face in a global sense and does not take full advantageof this aspect. To measure the correlation between the skulland face reliably and to improve the identification capabilityof the CCA model, we suggest a region-based identificationmethod.

As Figure 5 shows, we divide the skull and face skin intoseveral corresponding physiological feature regions, i.e., theeyes, nose, mouth and remaining features, which is calledthe profile region. In view of the strong correlation, the fore-head region is extracted alone. Because the correspondenceacross the training set has been established, we only need tosegment one sample manually, and the other samples can beautomatically segmented by the correspondence. Using thesesegmented training data, we construct one correlation analysis


Fig. 5. Skull and skin are decomposed into five regions according tophysiological characteristics: profile, forehead, eyes, nose and mouth fromup to down.

model for each of the five regions described above. In theidentification phase, five correlation analysis models are fusedto boost the identification of the unknown skull. Note that inthis phase, we must match the whole unknown skull or facedata to five regional statistical shape models to determine thecorresponding shape parameters.

For a skull-and-face pair, we can obtain five regionalmatching scores using the five correlation analysis models, andthe final matching score is computed by the weighted sum, asfollows:

F = wprp + whrh + were + wnrn + wmrm (11)

where rp, rh , re, rn, rm are the regional matching scores ofthe profile, forehead, eyes, nose and mouth, respectively, andwp, wh, we, wn, wm are the corresponding weights.

How to choose the weight for each region? Because whatthe skull identification relies on is the correlation betweenthe face and skull, a high weight should be assigned to theregion of a high correlation, such as the profile and forehead,in which the face morphology has a strong dependence on theskull. In contrast, the relationship between the face and skullis uncertain in the regions of a weak correlation such as thenose, eyes and mouth, although these regions can provide morediscriminative information for face recognition. Otherwise,if there is a high similarity in the regions of weak correlationbetween two face skins, then this similarity can lead to falseskull identification. In practice, we can find the best fusionweight by using the method of exhaustion.

IV. EXPERIMENTS AND ANALYSIS

The used dataset is the 3D skull-and-face skin pairs of the208 subjects described in Section II. Five-fold cross-validationwas used to evaluate the proposed method. We randomly chose5 non-overlapping data groups from the dataset, and eachgroup included the 3D skull-and-face skin pairs of 40 subjects.For each fold, 40 skulls in one group were used as probesto test the proposed method, and the remaining 168 pairs ofskull-and-face skins constitute the training set. In other words,we have 200 test skulls in all five folds. For each test skull,all of the 208 face skins constitute the gallery. The correctidentification rate reported in the experiments is the averageof the five folds. In this paper, we define that a test skull isidentified at rank n, if the matching score of the correct matchis of rank n. The identification rate at rank n is defined as

the ratio of the cumulative count of the numbers of test skullsidentified at rank n or less to the total number of test skulls.The correct identification rate is defined as the identificationrate at rank 1.

A. Evaluation of Skull Identification by Holistic CorrelationAnalysis Model

For each fold, we built a holistic correlation analysis modelin which all of the 167 eigen-vectors are used to constructthe holistic statistical shape models. A total of 163 test skullswere correctly identified at rank 1. The correct identificationrate attained 81.5%. It verifies that there indeed exists somerelationship between the skull and face, and it also shows thatthe proposed skull identification method is effective.

To describe the confidence evaluation of the identificationresult presented in Section III, four test groups in four foldsare used to estimate the probability distributions of the positivematching (i.e., the matching between a skull with its corre-sponding face) and the negative matching (i.e., the matchingbetween a skull with the faces of other persons), and theidentification results of the 40 test skulls in the remainingone fold are used for the confidence evaluation. In this part,the former four test groups are referred to as the probabilitytraining set, while the latter 40 test skulls are referred toas the evaluation set. We perform statistical analysis for the25600 (4 × 40 × 160) matching scores that were derived fromthe probability training set. The mean and standard deviationare (0.3205, 0.0811) for the positive matching and (0.0009,0.0806) for the negative matching. Figure 6a and 6b show thehistograms of the matching scores for the positive and negativematching, respectively. From Figure 6, we can see that both thepositive and negative matching score approximately follow aGaussian distribution. Thus, the two class conditional probabil-ity density functions can be set as Gaussian distributions, withthe mean and standard deviation computed for the positive andnegative matching, respectively. Figure 6c shows the two stan-dard Gaussian distributions. We can see that the scores of thepositive and negative matching are separable. The conditionalprobability P(w1|E) in Equation (10) can be set as the averagecorrect identification rate of the four folds that the probabilitytraining set belongs to, i.e., 80.6%. The prior probability P(E)can be set to 1 because all of the face data of the test skullsexists in the face gallery in this experiment. According to theBayes decision rule, we evaluate the identification results ofthe evaluation set, i.e., the 40 test skulls in the remaining onefold. Five of the 6 false identification results and all of the34 correct results are classified as positive. If we set P(E) as0.5, i.e., we have no prior information about whether the facedata of the test skull exists in the face gallery, then one correctidentification result is classified as negative and 3 false resultsare classified as positive.

By performing analysis of the false identification results,we find that there are two cases for the false identificationskulls. One case is that both the correct matching score andthe mismatched score are low, and this case can be classified asnegative matching. The other case is that the global morpho-logical similarity between the mismatched face and the true


Fig. 6. Histograms of matching scores for the positive matching (a), negativematching (b), and class conditional probability density functions (c).

face is high, and it can be misclassified as positive matching.Figure 7 shows two samples of the second case. We cansee that the mismatched faces are visually similar to the truefaces. Figure 7 also shows the comparison colored from blueto red according to the geometric distance between the twofaces. The average distances of these two samples are 2.7 mmand 2.56 mm, respectively. From the colored comparisons,we can see that the distance between the mismatched facesand the true faces is very small in most of the regions. Theglobal morphological similarity between two faces leads tosimilar shape model parameters, which contribute to the falsematching due to a holistic model. For this reason, we proposethe region-based method.

B. Evaluation of the Identification by Regional CorrelationAnalysis Models

To validate the regional correlation analysis models, weconstruct the five regional models in each fold. Similar tothe holistic statistical shape model construction, all of the167 eigen-vectors are used to construct each regional statisticalshape model. Table 1 shows the correct identification rateof each regional model. From Table 1, we can see that theresult is in accord with our analysis about the correlationbetween the skull and face in these regions. In other words,

Fig. 7. Two mismatched samples. (a) The first sample. (b) The secondsample.

TABLE I

CORRECT IDENTIFICATION RATES OF THE FIVE REGIONAL MODELS

the larger the correlation is, the better the identification result.The highest correct identification rate is 80.5% and belongsto the forehead region. The second is 75.5% and belongsto the profile region. These results are comparable to theidentification rate of the holistic model. The reason is that theforehead is the region with the largest correlation and plays anoverwhelming role in the correlation measure of the holisticmodel and the profile model. Because the nose is composedmainly of gristle and the lip is separated from the teeth bonein the mouth region, compared with the nose and mouthregions, the morphology of the eyes has a deeper dependenceon the bone shape of the orbit. Hence, the correlation in theeye region is higher than that in the nose or mouth region.As a result, the identification rate of the eye region is higherthan that of the nose and mouth regions. Figure 8 shows thecumulative match curves of the identification results of the fiveregional models and the holistic model, where the horizontalaxis denotes the rank of the matching scores and the verticalaxis denotes the identification rate at each rank. We can seethat the identification rate reaches above 90% at rank 3 and100% at rank 16 for the holistic, forehead and profile regions.

C. The Influence of the Statistical Shape Models

It is well known that the principle component vectors oflarge eigen-values in PCA represent the variation modes ofthe training dataset, while those corresponding to small eigen-values denote noise directions. Until now, we have always


Fig. 8. The cumulative match curve of each model.

TABLE II

CORRECT IDENTIFICATION RATES (%) FOR THE HOLISTIC MODEL

AND REGIONAL MODELS WITH DIFFERENT VARIANCE

CONTRIBUTION RATES (VCR)

used all of the 167 eigen-vectors to construct the statisticalshape models. Here, we discuss the influence caused by thevariation of the mode number of the statistical shape models.As Equation (1) describes, the mode number is determined bya variance contribution rate calculated from the cumulativeeigen-values. We vary the variance contribution rate from95% to 99.5%, and Table 2 shows the correct identificationrate of each model. From Table 2, we can see that with anincrease in the variance contribution rate, the identificationrate ascends gradually and begins descending at some locationnearly for each model. Compared with the identification resultwithout mode selection shown in Table 1, almost all of theidentification rates for the eyes, nose and mouth regions havesizable increases. The identification rate reaches 90.5% and90% at the variance contribution rate of 99% for the profileregion and the holistic model, respectively, and 86% at the rateof 99.5% for the forehead region. All of these results are muchhigher than the identification rate without mode selection,especially for the profile region. However, from Table 2,we can also see that the identification rate for the forehead,profile and holistic models varies quickly when the variancecontribution rate increases, especially for the forehead. This isbecause a large quantity of discarded variation modes containssome distinctive information. Table 3 shows the mode numberof each statistical shape model at the variance contribution rateof 99%. We can see that the mode numbers of the forehead andnose regions are small, especially the forehead. It is becausethe variation in the morphology of the forehead and nose

TABLE III

MODE NUMBER OF EACH STATISTICAL SHAPE MODEL AT

THE VARIANCE CONTRIBUTION RATE OF 99%

Fig. 9. The cumulative match curve of several methods.

regions is relatively simple when compared with that of otherregions. In practice, we can choose different mode numberswhen constructing the statistical shape models.

D. Skull Identification by Regional Fusion

In this section, we evaluate the regional fusiondescribed in Equation (11). Here, we also use all of the167 eigen-vectors to construct the statistical shape models.The correct identification rate reaches 88.5% when a directfusion strategy is adopted, i.e., all of the five weights areequal to 1, while it reaches 94% when we set the weightaccording to the identification rate of each region model, i.e.,

wi = ri/

∑ri

, i = p, h, e, n, m (12)

where ri is the correct identification rate of the corre-sponding region model. Through many attempts, we foundthat the correct identification rate can reach 94% providedthat the selected weights are around the ones computed byEquation (12), which are shown in Table 4. As a result, wesuggest that the fusion weight can be set by Equation (12) inreal applications. Here, we refer to the fusion using the weightcomputed by Equation (12) as the weight fusion. Figure 9shows the cumulative match curves of the identification resultsby using the direct fusion, weight fusion and holistic method.We can see that the identification rate of the weight fusionis higher than the rate of the direct fusion and the holisticmethod at each rank, and it reaches 99% at rank 3 and 100% atrank 9. This experiment verifies that the region-based methodcan indeed boost the identification.

V. DISCUSSION

Our proposed method is different from current skull identifi-cation techniques. Craniofacial reconstruction aims to provide


TABLE IV

A SET OF WEIGHT OF FIVE REGIONS

a clue for a positive identification. Thus, current researchfocuses mainly on how to reconstruct faces from the skulland rarely involves how to determine the identity according tothe reconstructed faces. Tu et al [31] and Huang et al [32]studied the recognition of the reconstructed face against aphoto gallery of potential victims. Both of them estimated theprojection of the reconstructed face to the photos and made anidentification decision by analyzing the projection residuals ofsome of the landmarks. The correct identification rate is onlyapproximately 20%, which is due to the great uncertainty incraniofacial reconstruction and the localization of craniofaciallandmarks. To validate the craniofacial reconstruction meth-ods, Claes et al [21] and Hu et al [33] performed 3D facerecognition by computing the similarity of the reconstructedface and each entry in a gallery of 3D candidate faces.Claes et al [21] combined three skull representations and fourcraniofacial models using different prior knowledge in turn andestimated the most probable face by the Maximum A-posteriorprobability (MAP) estimation. The correct identification rate of12 test cases varies from 0% to 100%, and only the estimationsby two combinations using complex prior knowledge obtainthe correct identification rate of 100%. Hu et al [33] evaluatedtheir reconstruction method using 110 test cases and obtainedthe correct identification rate of approximately 72%. Themain difference between these identifications and ours isthat their probe is the reconstructed face, while ours is theunknown skull. In addition, each craniofacial reconstructionmethod has a degree of uncertainty for leading to inaccuratereconstructions. Thus, in general, the identification capabilityis limited.

Craniofacial superimposition tends to identify or excludeindividuals on the condition that the investigators have an ideaas to the possible identity of the victim. Nearly all of thereports on the reliability of the craniofacial superimpositionin the literature are about a few criminal cases. Gordonand Steyn [34] test the accuracy and reliability in a SouthAfrican sample set of 40 face photos and 10 skulls. Theyassessed each face photo with the 10 skulls and conducted400 superimpositions with the morphological assessment, theanatomical landmarks-based assessment and the combinationof the two techniques respectively. For the morphologicalassessment, the positive identification rate was 85%, i.e., in85% of the cases, the correct skull was included in the possiblematches for a specific photo. However, in all of these cases,between zero and three other skulls out of 10 possibilitiescould also match a specific photo. The false positive ratewas 17.3%. For the landmark-based assessment, the positiveidentification rate was 80%, and the false positive rate was32%. Once again, between one and seven other skulls out of10 possibilities also matched the photo. For a combination of

the two techniques, the positive identification rate was 97.5%,and the false positive rate was 53.6%. Because the false posi-tive rate is relatively high, they think that the superimpositionhas limited use in human skull identification but could beuseful as an initial screening tool. When we have no ideaabout the possible identity of the victim and face a largedatabase of missing persons, craniofacial superimposition doesnot work due to its high false positive rate. In contrast, ourproposed method finds only the most probable faces. Forexample, no more than 9 faces are enough by using theweight fusion method in our experiment. In real applications,it is better to combine these techniques, i.e., craniofacialreconstruction, craniofacial superimposition and our proposedtechnique.

VI. CONCLUSIONS

Skull identification is one of the most important tasks inforensic anthropology. It has been practiced over one hun-dred years. Current research focuses mainly on craniofacialsuperimposition and craniofacial reconstruction. Both of themdepend on accurate extraction and representation of the intrin-sic relationship between the skull and face in terms of themorphology, which is difficult and still unsolved.

In this paper, we propose a novel technique for identifyingan unknown skull based on the correlation measure betweenthe 3D skull and 3D face in terms of the morphology. Usingthe 3D skull data as the probe and 3D face geometric dataas the gallery, this approach matches the unknown skull withenrolled 3D faces by the correlation measure between theprobe and the gallery. Unlike craniofacial superimposition andcraniofacial reconstruction, this method does not require anaccurate relationship between the skull and face. To boost thematching accuracy, we divide the skull and face skin into fivephysiological feature regions, establish five correlation analy-sis models, and make a decision by model fusion. Through thiswork, we prove that there is indeed a close relation between theskull and face, and thus, we provide a theoretical support forcraniofacial superimposition and craniofacial reconstruction.On the other hand, we show that the region-based strategy isbetter than the holistic strategy in the extraction of the relation-ship between the skull and face, which provides an evidencefor some research on regional craniofacial reconstruction.

In this work, we only use CT scan data to validate theproposed method. 3D face data acquisition by CT is infeasiblefor real applications because of the intrinsic radiation forthe livings and the cost of the system. Compared with CTscan, 3D face modeling from 2D images is a convenient andnon-intrusive way. By this way, we can construct a 3D facedatabase from a 2D database of missing persons in a nation.Therefore, future work will focus on applying the proposedtechnique to 3D face models reconstructed from 2D facephotos. On the other hand, with the progress of 3D dataacquisition technology, non-intrusive 3D face data capture willbecome readily available and cost efficient; the construction oflarge 3D face databases will be feasible and possible in thefield of public security. The proposed technique will come intoa wide application.


ACKNOWLEDGMENTS

We would like to thank the anonymous reviewers for theirhelpful comments.

REFERENCES

[1] D. Dirkmaat, A Companion to Forensic Anthropology. Hoboken, NJ,USA: Wiley, 2012.

[2] S. Al-Amad, M. McCullough, J. Graham, J. Clement, and A. Hill,“Craniofacial identification by computer-mediated superimposition,”J. Forensic Odontostomatol., vol. 24, no. 2, pp. 47–52, 2006.

[3] E. Indriati, “Historical perspectives on forensic anthropology inIndonesia,” in Handbook of Forensic Anthropology and Archaeology,S. Blau and D. H. Ubelaker, Eds. Walnut Creek, CA, USA: Left CoastPress, 2009, pp. 115–125.

[4] C. Wilkinson, Forensic Facial Reconstruction. Cambridge, U.K.:Cambridge Univ. Press, 2004.

[5] P. Claes, D. Vandermeulen, S. De Greef, G. Willems, J. G. Clement,and P. Suetens, “Computerized craniofacial reconstruction: Concep-tual framework and review,” Forensic Sci. Int., vol. 201, nos. 1–3,pp. 138–145, 2010.

[6] S. Damas et al., “Forensic identification by computer-aided craniofacialsuperimposition: A survey,” ACM Comput. Surv., vol. 43, no. 4, pp. 1–27,2011.

[7] W. A. Aulsebrook, M. Y. Iscan, J. H. Slabbert, and P. Becker, “Superim-position and reconstruction in forensic facial identification: A survey,”Forensic Sci. Int., vol. 75, nos. 2–3, pp. 101–120, 1995.

[8] T. Beeler, B. Bicke, P. Beardsley, B. Sumner, and M. Gross, “High-quality single-shot capture of facial geometry,” ACM Trans. Graph.,vol. 29, no. 4, pp. 40-1–40-9, 2010.

[9] D. Bradley, W. Heidrich, T. Popa, and A. Sheffer, “High resolutionpassive facial performance capture,” ACM Trans. Graph., vol. 29, no. 4,pp. 41-1–41-10, 2010.

[10] C. J. Kuo, R.-S. Huang, and T.-G. Lin, “3-D facial model estimationfrom single front-view facial image,” IEEE Trans. Circuits Syst. VideoTechnol., vol. 12, no. 3, pp. 183–192, Mar. 2002.

[11] S.-Y. Baek, B.-Y. Kim, and K. Lee, “3D face model reconstructionfrom single 2D frontal image,” in Proc. 8th Int. Conf. Virtual RealityContinuum Appl. Ind., Yokohama, Japan, Dec. 2009, pp. 95–101.

[12] M. Dimitrijevic, S. Ilic, and P. Fua, “Accurate face models from uncal-ibrated and ill-lit video sequences,” in Proc. IEEE Comput. Soc. Conf.Comput. Vis. Pattern Recognit. (CVPR), Jun./Jul. 2004, pp. 1034–1041.

[13] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor “Canonical correlationanalysis: An overview with application to learning methods,” NeuralComput., vol. 16, no. 12, pp. 2639–2664, 2004.

[14] L. Ballerini, O. Cordón, J. Santamaria, S. Damas, I. Aleman, andM. Botella, “Craniofacial superimposition in forensic identification usinggenetic algorithms,” in Proc. 3rd Int. Symp. Inf. Assurance Security,2007, pp. 429–434.

[15] O. Ibáñez, L. Ballerini, O. Cordón, S. Damas, and J. Santamaría, “Anexperimental study on the applicability of evolutionary algorithms tocraniofacial superimposition in forensic identifcation,” Inf. Sci., vol. 179,no. 23, pp. 3998–4028, 2009.

[16] T. W. Fenton, A. N. Heard, and N. J. Sauer, “Skull-photo superimposi-tion and border deaths: Identification through exclusion and the failureto exclude,” J. Forensic Sci., vol. 53, no. 1, pp. 34–40, 2008.

[17] J. Santamaria, O. Cordón, S. Damas, and O. Ibaez, “Tackling thecoplanarity problem in 3D camera calibration by means of fuzzy land-marks: A performance study in forensic craniofacial superimposition,”in Proc. IEEE 12th Int. Conf. Comput. Vis. Workshop (ICCV), Oct. 2009,pp. 1686–1693.

[18] O. Ibáñez, O. Cordón, S. Damas, and J. Santamaría, “Modeling theskull–face overlay uncertainty using fuzzy sets,” IEEE Trans. FuzzySyst., vol. 19, no. 5, pp. 946–959, Oct. 2011.

[19] D. Vandermeulen, P. Claes, D. Loeckx, S. De Greef, G. Willems,and P. Suetens, “Computerized craniofacial reconstruction using CT-derived implicit surface representations,” Forensic Sci. Int., vol. 159,pp. S164–S174, May 2006.

[20] S. Michael and M. Chen, “The 3D reconstruction of facial featuresusing volume distortion,” in Proc. 14th Eurograph. U.K. Conf., 1996,pp. 297–305.

[21] P. Claes, D. Vandermeulen, S. De Greef, G. Willems, J. G. Clement,and P. Suetens, “Bayesian estimation of optimal craniofacial reconstruc-tions,” Forensic Sci. Int., vol. 201, nos. 1–3, pp. 146–152, 2010.

[22] Y. Hu et al., “A hierarchical dense deformable model for 3D facereconstruction from skull,” Multimedia Tools Appl., vol. 64, no. 2,pp. 345–364, 2013.

[23] M. Berar, F. M. Tilotta, J. A. Glaunès, and Y. Rozenholc, “Craniofacialreconstruction as a prediction problem using a latent root regressionmodel,” Forensic Sci. Int., vol. 210, nos. 1–3, pp. 228–236, 2011.

[24] F. Duan, S. Yang, D. Huang, Y. Hu, Z. Wu, and M. Zhou, “Craniofacialreconstruction based on multi-linear subspace analysis,” MultimediaTools Appl., Jan. 2013, doi: 10.1007/s11042-012-1351-2.

[25] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution3D surface construction algorithm,” ACM SIGGRAPH Comput. Graph.,vol. 21, no. 4, pp. 163–169, 1987.

[26] Q. Deng, M. Zhou, W. Shui, Z. Wu, Y. Ji, and R. Bai, “A novelskull registration based on global and local deformations for craniofacialreconstruction,” Forensic Sci. Int., vol. 208, nos. 1–3, pp. 95–102, 2011.

[27] W. Yang, D. Yi, Z. Lei, J. Sang, and S. Z. Li, “2D–3D face matchingusing CCA,” in Proc. 8th IEEE Int. Conf. Autom. Face Gesture Recognit.(FG), Sep. 2008, pp. 1–6.

[28] B. Abraham and G. Merola, “Dimensionality reduction approach tomultivariate prediction,” Comput. Statist. Data Anal., vol. 48, no. 1,pp. 5–16, 2005.

[29] W. Zheng, X. Zhou, C. Zou, and L. Zhao, “Facial expression recognitionusing kernel canonical correlation analysis (KCCA),” IEEE Trans.Neural Netw., vol. 17, no. 1, pp. 233–238, Jan. 2006.

[30] P. J. Besl and N. D. McKay, “A method for registration of 3-D shapes,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 2, pp. 239–256,Feb. 1992.

[31] P. Tu, R. Book, X. Liu, N. Krahnstoever, C. Adrian, and P. Williams,“Automatic face recognition from skeletal remains,” in Proc. IEEE Conf.Comput. Vis. Pattern Recognit. (CVPR), Jun. 2007, pp. 1–7.

[32] J. Huang, M. Zhou, F. Duan, Q. Deng, Z. Wu, and Y. Tian, “Theweighted landmark-based algorithm for skull identification,” in Proc.14th Int. Conf. Comput. Anal. Images Patterns (CAIP), LNCS 6855.2011, pp. 42–48.

[33] Y. Hu, F. Duan, M. Zhou, Y. Sun, and B. Yin, “Craniofacial reconstruc-tion based on a hierarchical dense deformable model,” EURASIP J. Adv.Signal Process., vol. 2012, p. 217, Oct. 2012.

[34] G. M. Gordon and M. Steyn, “An investigation into the accuracy andreliability of skull-photo superimposition in a South African sample,”Forensic Sci. Int., vol. 216, nos. 1–3, pp. 198.e1–198.e6, 2012.

Fuqing Duan received the B.S. and M.S. degreesin mathematics from Northwest University, Xi’an,China, in 1995 and 1998, respectively, and the Ph.D.degree in pattern recognition from the NationalLaboratory of Pattern Recognition, Beijing, China,in 2006. He is currently an Associate Professor withthe College of Information Science and Technol-ogy, Beijing Normal University, Beijing. His currentresearch interests include 3D face reconstruction,skull identification, and machine learning and appli-cations. He has authored more than 80 conference

and journal articles on related topics.

Yanchao Yang received the B.S. degree incommunication engineering from Jilin University,Changchun, China, in 2006, and the M.S. degreefrom the College of Information Science and Tech-nology, Beijing Normal University, Beijing, China,in 2013. Her research interests include patternrecognition, digital image processing, and computervision. She was with Huawei Technologies Com-pany, Ltd., Shenzhen, China, as an Engineer from2007 to 2010. She is currently a Machine Visionand Image Algorithm Research Engineer with Han’s

Laser Technology Company, Ltd., Shenzhen.


Yan Li received the B.S. degree in signal andinformation processing from the North University ofChina, Taiyuan, China, in 2011. She is currently pur-suing the master’s degree with the College of Infor-mation Science and Technology, Beijing NormalUniversity, Beijing, China. Her research interestsinclude pattern recognition and image processing.

Yun Tian received the Ph.D. degree in signal andinformation processing from Northwestern Polytech-nic University, Xi’an, China, in 2007. He is currentlyan Associate Professor with the College of Informa-tion Science and Technology, Beijing Normal Uni-versity, Beijing, China. His research interests includepattern recognition and medical image processing.

Ke Lu received the M.S. and Ph.D. degrees fromthe Department of Mathematics and the Departmentof Computer Science, Northwest University, Xi’an,China, in 1998 and 2003, respectively. He was aPost-Doctoral Fellow with the Institute of Automa-tion, Chinese Academy of Sciences, Beijing, China,from 2003 to 2005. He is currently a Professorwith the University of the Chinese Academy ofSciences, Beijing. His research areas mainly focuson curve matching, 3D reconstruction, and computergraphics.

Zhongke Wu is a Full Professor and the Ph.D.Student Supervisor with the College of InformationScience and Technology, Beijing Normal University(BNU), Beijing, China. Prior to joining BNU, he waswith Nanyang Technological University, Singapore,the Institute National de Recherche en Informatiqueet en Automatique, Paris, France, the Institute ofHigh Performance Computing, Singapore, and theInstitute of Software, Chinese Academy of Sciences,Beijing, from 1995 to 2006. His research interestsare image processing and computer graphics.

Mingquan Zhou is a Professor, the Doctoral Super-visor, and the Dean of the College of InformationScience and Technology at Beijing Normal Uni-versity, Beijing, China. His research interests arecomputer graphics and virtual reality.

skull identification via correlation measure between skull and face shape

Documents