The True Orthophoto Generation Method

Xiao Wang, Xia Zhang, Jinnian Wang Institude of Remote Sensing Applications

Chinese Academy of Sciences name of organization, acronyms acceptable

Beijing, China wangx@irsa.ac.cn

Abstract—This paper presents the process of true orthophoto generation, including the occlusion detection and compensation. The Z-buffer method is applied here to detect the occlusion regions. Then the neighboring images are used to fill up the occlusions. A valid-pixel-based image inpainting method is proposed for compensating the rest occlusions.

Keywords- true orthophoto; occlusion detection; occlusion compensation; image inpainting

I. INTRODUCTION The conventional orthophotos always suffer from two

problems[1]. One is the building lean. The buildings are treated as the terrain and not rectified due to lack of the models of buildings. Another problem is the double mapping, which is caused by the traditional digital rectified method with complete DSM (Digital Surface Model). In the process of true orthophoto generation, most methods recover the relation of the ray between perspective center and objects, and find out where is invisible from the perspective center. If the current object is occluded by other taller ones, it will be marked as invisible point. Finally, the occlusion information of each pixel will be recorded in a visibility map. There are several typical effective methods such as Z-buffer, polygon-based method and angle-based method, etc[1,2].

Once the occlusion region is detected, pixel compensation is required for the completeness of the true orthophoto. The most reliable method is to fill up the holes with the corresponding pixels from the neighboring images[3,4]. Another available method is to apply the image inpainting algorithms[5-8] for complete compensation.

This paper adopts the Z-buffer method for occlusion detection, and an improved image inpainting algorithm is developed for the occlusions compensation.

II. OCCLUSION DETECTION When available building models are available, it will

produce the double mapping problem by conventional differential rectification method. The region around the buildings is invisible from the perspective center, and it will be assigned the pixel value of the roof. In figure 1, OM denotes a ray that goes through the roof. S denotes a point on the roof while the M is the intersection of the OM and the ground. P denotes the intersection point of the OM ray and image plane.

It’s obviously that the S point will be imaged at P, and it will be correctly rectified. However the M point will be also assigned the P value according to the collinearity equation. As a result, the roof texture will appear twice, and that produce the double mapping effect. The researchers propose several algorithms to solve this problem while the Z-buffer is one of the famous methods.

O

M

S

Ground

Image plane

Perspective Center

P

Fig.1. The principle of double mapping

The Z-buffer method is developed gradually by Amhar[1]. The DBM(Digital Building Model) and the DTM(Digital Terrain Model) are used to generate the true orthophoto. The method rectifies the terrain and the buildings separately. Firstly in the process of rectifying the terrain, the terrain occluded by roofs and the walls of the buildings can be masked out in the rectified result, and the terrain orthophoto is produced. Then the buildings are rectified with DBM and the building orthophoto is produced. Finally, the two results will be synthesized and true orthophoto is generated. This method only applies for occlusions existing between the terrain and the buildings. However in the urban areas, buildings occlude other buildings always occurs, and the method above are not able to detect. As a result, the Z-buffer method is proposed by Amhar and it judges the visibilities of the objects by comparing the distances between the perspective center and the objects.

A two dimensional array is generated in the image plane that it has the same resolution as the original image, and it is named “Z-buffer” array. Each array cell records the distance of the corresponding object point to the perspective center. The DBM includes polygons information of roofs and walls, and each polygon includes the relevant identification Code(ID) information. The polygons are project to the source image and

will be rasterized. The corresponding location of the Z-buffer array records the distance between the perspective center and the current object point as well as the ID. Finally, the Z-buffer array will record the distances and ID of visible points by comparing the distances. The occlusions can be detected correctly by the method.paper size. If you are using US letter-sized paper, please close this file and download the file for “MSW_USltr_format”.

III. OCCLUSION COMPENSATION

A. Occlusion compensation with neighboring images

The most reliable occlusion compensation method is to fill the invisible area with the corresponding pixels that selected from the neighboring images[3]. The compensation information is based on the side and forward overlap images. The more abundant of the views of the neighboring images are, the more missing pixels will be obtained.

Once there is abundant information on the adjacent images, the occluding areas on the “master” image may find the available data on distinguished “slave” images [3]. Under this circumstance, it needs to choose the most reliable orthophoto pixels. Zhou uses the reliability of object points. It’s based on the principle that the relief displacement of nadir point is zero. The points with less distance from nadir point will have smaller relief displacement. Follow this principle, the point with shortest distance from nadir point among different slave images will be chosen to compensate on the relevant location.

Yang compensates the occlusions by comparing the angle of view[4]. The smaller the including angle between the view direction and the vertical direction is, the less relief displacement is, which means the corresponding object points are more reliable.This method is based on the similar principle of Zhou’s method.

In fact, the occlusion compensation with neighboring images cannot insure the complete recovery of the occluded regions. The visibilities of the occluded points are decided by the height of the perspective center, the height of the buildings, the distance of the buildings to the nadir point and so forth. It’s not always produce complete orthophoto. The rest of the occlusions can be restored use the image inpainting method according to the requirement.

B. Occlusion compensation with image inpainting method Image inpainting has a wide application in restoration of

the damaged images, the preservation of the architecture and so on. The image inpainting method is generally classified into two categories, the texture inpainting and the structural inpainting. The texture inpainting is more appropriate to restore the relatively larger region while the structural inpainting is expert to finely restore the relatively smaller holes. The most representative method in texture inpainting is the exemplar-based texture synthesis method[5]. In this algorithm, The window that centered each pixel of occlusion region is chosen, and the SSD value of the window with other

windows is calculated. The window that has minimum SSD is selected, and the pixel value will be copied to the corresponding holes. The Partial Differential Equation (PDE) method is one of the algorithms in the structural inpainting methods, and the principle of energy propagating is applied, such as the BSCB (Bertalmio-Sapiro-Caselles- Ballester) method, the Euler elastica model and the CDD method[5, 6]. Another algorithm of structural inpainting is the variation method including BV (Bounded Variation) and TV (Total Variation) [8]. This algorithm mainly converts the image inpainting method to the variation method by constructing the mathematical model and the prior model.

In practical application, the two algorithms have each advantages and weak points. For the texture inpainting method, the recovery color is more similar to the real one, while it is not easy to propagate the structural characteristics, and the texture is more inclined to break off. The structural inpainting method adopts the energy propagation principle, and it is easier to maintain the continuity of the structure, while the restore region is always blurred. Criminisi proposed a method that combines the virtues of the two categories[7]. This method is based on the texture synthesis that selecting texture exemplar patches, and introduces the priority, which is described as:

( ) ( ) ( )P p C p D p= (1) C(p) denotes the confidence of each pixel, D(p) is the data

term, which includes the structural information. The confidence term and the data term are described as:

( )( )

( ) pq I

p

C qC p ψ

ψ∈ −Ω=

∑ ∩

(2)

( ) p pI nD p

α

⊥∇ •=

(3) Ψp denotes the size of the window, and Ι denotes the entire

image. Ω denotes the region prepared to restore, and formula (2) reflects that the current pixel will have a higher confidence if the sum of the confidence of each pixel in the window centered the current pixel is high. pI ⊥∇ denotes the isophote, and np denotes the direction of the normal vector. D(p) is determined by the isophote and the normal vector of the processed district edges, and it has a higher value when there are strong edges. The priority is decided by the confidence and strength of the edge in the window. The pixel with a maximum of priority will be inpainting first, following the principle that searching the best matched window, then the corresponding pixels are copied to the current restoring pixel.

This algorithm considers the texture and structure characteristics simultaneously. When the pixel prepared to restore has obvious structural characteristic as well as high confidence, it’ll be process first. As a result, the strong structural characteristic will be easier to propagate, and it relieves the blur situation. Generally, the algorithm can obtain a satisfied recovery results. However it doesn’t adapt to the compensation of the occlusions of the orthophoto. The occlusions always locate around the buildings, where has

strong structural characteristic. If the principle above is followed, the edges of the buildings will be propagated firstly, and it will lead to the incorrect result. Therefore, the true orthophoto should be compensated according to its unique characteristics.

Criminisi proposed the examplar-based texture synthesis method, which is a representative method. It is improved in this paper for adapting to the compensation of the true orthophoto, it is named “valid-pixel-based”. By introducing the models of the buildings, the buildings and other objects are separated, and the interruption of the strong edge of the buildings will be reduced. The building districts and the occlusion districts are named “invalid pixels”, and the other pixels are “valid” pixels. The interruption of the invalid pixels will be relived while computing the priority and the texture of the buildings’ roofs will not be propagated. When compute the confidence, the C(p) is described as (4), and the IB denotes the building district. When compute D(p), the isophote scalar is determined by the maximum value of the gradient, as describe in (5)

( )( )

( )B

pq I I

p

C qC p ψ

ψ∈ −Ω−=

∑ ∩

(4) max( | ( ) )B

p P pI I p I Iψ⊥∇ = ∇ ∈ − Ω −∩ (5)

In the original algorithm, it searches the most similar

window within the whole image, and it will be time-consuming when processing a large image. The occlusions of the orthophoto are always the terrain, grass, and road, and so on. These objects have consistent texure. Therefore, in our algorithm, we search the window centerd the pixel to be restored within certain scope, and it will obviously improve the efficiency. The steps are described as follows:

1. Rasterize the DBM, and record the columns and rows where is buildings.

2. Initialize the confidence term, and set the valid pixel as 0, and set the others as 1.

3. Compute the priority of each edge point. The C(p) is obtained by (4), and D(p) is calculated by the isophotes and the normal vectors. The isophotes is calculated by (5).

4. The edge point of maximum priority will be process first. The best matched window within a certain range can be found out. The processing pixel is assigned by the value of corresponding valid pixel of the best matched window.

IV. EXPERIMENTS AND ANALYSES Real data is used for testing our methods. The image data is

captured over the city of Kunming, China. The elements of interior orientation and exterior orientation are all available. DBM(Digital Building Model) and DTM are extracted with stereo measurement by VirtuoZo. The resolution of original image is 0.25m, and the resolution of orthophoto is 0.2m.

Figure 2 The original

image Figure 3 The double mapping effect

Figure 4 The occlusion detection result

Figure 5 The occlusion compensation result by the neiboring orthophoto

Figure 2 is the original image, and the figure 3 shows the

double mapping effect by the conventional method. Z-buffer method is adopted to detect the occlusion district and the result is presented by figure 4. The occlusions are compensated by the neighboring images, as showing in figure 5.

However, the occlusions may not be completely filled up by the neighboring images. The proposed valid-pixel-based texture synthesis method is applied to compensate the rest lacking pixels.

The exemplar-based image inpainting method is test to fill the rest occlusions and the result are showed as figure 6. It’s obviously that the boundaries of the building are spread into the region of the terrain because of their strong edge properties. The algorithm is not effective any more for the special image inpainting. Our improved method is experimented with the same data, and the result is showed as figure 7. The disturbance of the boundary of the buildings is reduced and the result is much better.

Figure 6 The result of compensation by exemplar-based image

inpainting method

Figure 7 The results of the compensation by the valid-pixel-

based image inpainting method

V. CONCLUSIONS This paper discusses the occlusion detection and

compensation of the true orthophoto. The Z-buffer method is applied to detect the region occluded by the buildings. The occlusions are compensated by two strategies. Firstly the orthophoto is compensated by the adjacent images, and then a valid-pixels-based method is proposed to fill up the rest occlusion region. By adding the building models, it retains the advantages of the exemplar-based inpainting method. In addition, it relieves the incorrect propagation of the roof texture. Experiments show that the algorithm is effective for the true orthophoto compensation.

REFERENCES [1] F. Amhar, “The generation of true orthophotos using a 3D builing

model in conjunction with a conventional DTM ” . International Archives of Photogrammetry and Remote Sensing, 1998, 32(Part4): 16-22.

[1] P. Kuzmin, A. Korytnik, O. Long. “Polygon-based true orthophoto generation”. Proceedings of ISPRS XXth Congress, Istanbul, 2004.

[2] G. Zhou, W. Chen, J. A. Kelmelis, and D. Zhang “A Comprehensive Study on Urban True Orthorectification”. IEEE Transactions on Geoscience and Remote Sensing, 2005, 43(9): 2138-2147

[3] Y. Sheng, P. Gong., and G. S. Blging (2003). "True Orthoimage Production for Forested Areas from Large-Scale Aerial Photographs." Photogrammetric Engineering & Remote Sensing 69(3): 259-266

[4] Y. Q. Xu, S. C. Zhu, B. N. Guo, and H. Y. Shum, “Asymptotically admissible texture synthesis,” in Proc. Int. Workshop Stat. Comput. Theories Vis., 2001, pp. 1–22.

[5] T. F. Chan and J. Shen, “Non-texture inpainting by curvature driven diffusion,”2001. J. Vis. Commun. Image Represent., vol. 12, no. 4, pp. 436–449, 2001.

[6] A. Criminisi, P.Perez and K.Toyama. “Region filling and object removal by exemplar based image inpainting”. IEEE Transactions on Image Processing, 2004, 13 (9) : 1200 ～1212.

[7] L. Rudin, S. Osher, E. Faterni. “Nonlinear total variation based noise removal algorithms”. Physica D, 1992, 60 (1～4) : 259～268