Automatic Face Swapping

Batuhan YamanMehmet Alperen Derin

Hacettepe UniversityAnkara,Turkey

batuhan.yaman@hacettepe.edu.trmehmet.derin@hacettepe.edu.tr

Abstract

Face swapping is a commonly used con-cept in entertainment business. Thisproject aims to achieve to swap face fromtwo different images automatically. Wedetected faces using Viola-Jones algo-rithm and after that we did shifting andre-sizing. Lastly faces are swapped usingPoisson image editing techniques.

1 Introduction

Face detection is a computer technology beingused in a variety of applications that identifies hu-man faces in digital images.[1] Nowadays, facedetection is important part of image processingand computer vision. Addition to its critical usage,face detection is also used in entertainment busi-ness. Combined with image manipulation, bunchof softwares offer face swap applications. In thisproject our goal is to swap faces in two images thatwere given to our program. Every operation willbe automatic. So user only needs to provide targetand source images. To achieve this, we need to de-tect both faces in images. Then extract the neces-sary features and transfer them into target image.We need to transform our source image’s colorsto target image’s color. For better fitting we arere-sizing the source image.

2 Related Work

• Photomontage”The history of photomontage is nearly as oldas the history of photography itself. Pho-tomontage has been practiced at least sincethe mid-nineteenth century, when artists likeOscar Rejlander [1857] and Henry PeachRobinson [1869] began combining multiplephotographs to express greater detail. Muchmore recently, artists like Scott Mutter [1992]

and Jerry Uelsmann [1992] have used a sim-ilar process for a very different purpose: toachieve a surrealistic effect. Whether for re-alism or surrealism, these artists all face thesame challenges of merging multiple imageseffectively.” [2]

• Neural Network-Based Face Detection [3]In this research, they used neural networkbased algorithm to detect faces in gray scaleimages. Their methods designed to be moregeneral, with little differences in faces. Theyused 20x20 pixel regions to determine if theface is present or absent. For larger facesthan this region, sampling is applied. In thefollowing steps, lighting correction and his-togram equalization is applied. With resultimages neural network is trained to give anoutput ranging from -1 to 1, which representsabsence or presence of the face. ”... the sys-tem is able to detect 90.5 percent of the facesover a test set of 130 complex images, withan acceptable number of false positives.”

• Poisson Image Editing [4]This research uses Poisson equation as wellas many other similar works. They proposea guided interpolation, with the guidance be-ing specified by the user. ”The mathematicaltool at the heart of the approach is the Pois-son partial differential equation with Dirich-let boundary conditions which specifies theLaplacian of an unknown function over thedomain of interest, along with the unknownfunction values over the boundary of the do-main.”

• Coordinates for Instant Image Cloning [5]”All gradient domain techniques eventuallysolve a large sparse linear system, the Pois-son equation.”. Instead of solving the Pois-son equation, they propose coordinate-based

approach. Without solving the equation, al-gorithm needs less computational power isrequired and has small memory footprint.With this easy-to-implement algorithm, theyachieved real-time seamless cloning andhealing of still images and video sequenceson the CPU, as well as on the GPU.

3 The Approach

In order to swap faces, first we need to detect them.Here we are using Matlab’s built-in class Cas-cade Object Detector which uses Viola-Jones al-gorithm. This algorithm detects faces rapidly withhigh success rate, but it is weak when the faces arerotated or not directly pointed to the camera.This algorithms uses Haar Features[6] and elim-inates feature redundancy with Ada Boosting[7].When a feature is proven its relevancy for find-ing specific region, in this case we are using ”EyeArea” and ”Mouth”, these features tells us thatthere might be a face. Before we tried using”Left Eye” and ”Right Eye” instead of only ”EyeArea”. Although sometimes the precision of de-tecting eyes individually was better, mostly detec-tion was way harder or was not possible at all.So in order to generalize and increase the possi-ble input image domain, we decided to continuewith ”Eye Area”. For obtaining the final and onlyface, we used MergeThreshold which is a thresh-old value to merge our useful features.First, we found both target and source images eyesand mouth. We compared their width and heightthen re-sized source image accordingly. In orderto do Poisson image editing well we had to re-sizeour source image according to target image but inthe same time size of the face had to remain samefor best swap. So we find the center point of theface in the source image and cropped the cornerswith centering to that point.

Figure 1: Uncropped / Cropped

We obtained 4 different coordinates:

• Left eye’s top left corner - x1,y1

• Right eye’s top right corner - x2,y2

• Mouth’s bottom left corner - x3,y3

• Mouth’s bottom right corner - x4,y4

With these coordinates we formed our mask.We used following formula for applying our maskand transfer that area of source image to target im-age:SourceImage×Mask+TargetImage×(1−

Mask)Transferred image and target image are aligned

to their face centers. Then we applied two Poissonimage editing technique, PoissonJacobi and Pois-sonGaussSeidel.

4 Results

For experimental part, we did many tests but fi-nally obtained 5 test results.

Figure 2: Target / Source

Figure 3: Jacobi / GaussSeidel

Figure 4: Target / Source

Figure 5: Jacobi / GaussSeidel

Figure 6: Target / Source

Figure 7: Jacobi / GaussSeidel

As we can clearly see from the results, Gauss-Seidel algorithm provides better results with nicecolor transition. But at least from our experiments,it takes more time than Jacobi. Our algorithmworks best with similar sized images and as westated before, our weakness is poorly lighted andnot straightly faced images.

Figure 8: Target / Source

Figure 9: Jacobi / GaussSeidel

Figure 10: Target / Source

Figure 11: Jacobi / GaussSeidel

5 Conclusion

Face swapping is one of the most common usedtechniques in image processing, specifically inphotomontage. We implemented a face swappingapplication using MATLAB. We detected faces

with ready-to-use, implemented Viola-Jones algo-rithm, we have done some intermediate processon images to make them ready for Poisson im-age editing techniques and we applied two Poissonimage editing techniques using a 3rd party MAT-LAB toolbox. We got really satisfactory results insufficiently lighted and straightly faced images. Ifthese two are not satisfied, we cannot get a resultsince we fail at detecting faces. Another work canovercome these difficulties.

References

[1] https://facedetection.com/

[2] Agarwala, Aseem, et al. ”Interactive digitalphotomontage.” ACM Transactions on Graph-ics (TOG). Vol. 23. No. 3. ACM, 2004.

[3] Rowley, Henry A., Shumeet Baluja, and TakeoKanade. ”Neural network-based face detec-tion.” IEEE Transactions on pattern analysisand machine intelligence 20.1 (1998): 23-38.

[4] Prez, Patrick, Michel Gangnet, and AndrewBlake. ”Poisson image editing.” ACM Trans-actions on Graphics (TOG). Vol. 22. No. 3.ACM, 2003.

[5] Farbman, Zeev, et al. ”Coordinates for instantimage cloning.” ACM Transactions on Graph-ics (TOG). Vol. 28. No. 3. ACM, 2009.

[6] Whitehill, Jacob, and Christian W. Omlin.”Haar features for facs au recognition.” 7th In-ternational Conference on Automatic Face andGesture Recognition (FGR06). IEEE, 2006.

[7] Freund, Yoav, Robert Schapire, and N. Abe.”A short introduction to boosting.” Journal-Japanese Society For Artificial Intelligence14.771-780 (1999): 1612.