enhanced pedestrian attitude estimation using vision aiding

This article was downloaded by: [FU Berlin]On: 16 October 2014, At: 06:18Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Location Based ServicesPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/tlbs20

Enhanced pedestrian attitudeestimation using vision aidingLaura Ruotsalainen a , Jared Bancroft b , Gérard Lachapelle b &Heidi Kuusniemi aa Department of Navigation and Positioning , Finnish GeodeticInstitute , Masala , Finlandb Department of Geomatics Engineering , Schulich School ofEngineering, University of Calgary, Calgary , CanadaPublished online: 21 Aug 2013.

To cite this article: Laura Ruotsalainen , Jared Bancroft , Gérard Lachapelle & Heidi Kuusniemi(2013) Enhanced pedestrian attitude estimation using vision aiding, Journal of Location BasedServices, 7:3, 209-222, DOI: 10.1080/17489725.2013.819450

To link to this article: http://dx.doi.org/10.1080/17489725.2013.819450

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/loi/tlbs20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/17489725.2013.819450

http://dx.doi.org/10.1080/17489725.2013.819450

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

Enhanced pedestrian attitude estimation using vision aiding

Laura Ruotsalainena*, Jared Bancroftb, Gerard Lachapelleb and Heidi Kuusniemia

aDepartment of Navigation and Positioning, Finnish Geodetic Institute, Masala, Finland;bDepartment of Geomatics Engineering, Schulich School of Engineering, University of Calgary,

Calgary, Canada

(Received 10 March 2013; final version received 12 June 2013; accepted 19 June 2013)

Inertial Navigation System (INS) sensors are widely used for augmenting GlobalNavigation Satellite System measurements in urban environments and in theindoors. With a known initial position, the current position may be propagatedusing gyroscopes and accelerometers forming the INS for a limited time. Thelimitation of the self-contained sensors is the cumulative measurement errorsthat affect the accuracy of the attitude obtained using the gyroscopes. Visionaiding has proven to be a feasible method for mitigating these errors. This paperintroduces a method to obtain attitude measurements by tracking the motion ofvanishing points in consecutive images and integrating these measurements withthe attitude observed by INS using an extended Kalman filter. The experimentsshow that vision aiding results in significant improvement of the user attitudeand therefore the navigation solution. The challenges in vanishing point-basedvision aiding are the processing time and the method’s lack of capability toperceive sharp turns. These issues are addressed by developing an algorithmbased on the Probabilistic Hough Transform for more efficient vanishing pointcalculation which also provides a means for turn detection. These improvementsadvance the objective of developing a real-time seamless indoor–outdoorpedestrian navigation system utilising vision aiding.

Keywords: vision aiding; vanishing point; Hough Transform; pedestriannavigation; INS

1. Introduction

Pedestrian navigation sets demanding requirements for the equipment and software. The

equipment used has to be light to carry, reasonably priced and easy to use in addition to

providing accurate and real-time navigation. Global Navigation Satellite System (GNSS)

receivers embedded in smartphone succeed in providing the above in open outdoor

environments, but the performance is degraded in indoor and urban areas. Inertial

sensors are widely used for augmenting the GNSS measurements in these environments.

With a known initial position, the current position may be propagated using a triad of

gyroscopes and accelerometers forming the Inertial Navigation System (INS) for a limited

time (Collin 2006). The limitation of the self-contained sensors is the cumulative

measurement errors that affect the accuracy of the attitude obtained using the gyroscopes

q 2013 Taylor & Francis

*Corresponding author. Email: [email protected]

Journal of Location Based Services, 2013

Vol. 7, No. 3, 209–222, http://dx.doi.org/10.1080/17489725.2013.819450

Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

http://dx.doi.org/10.1080/17489725.2013.819450

(Saarinen 2009). Vision aiding has proven to be a feasible method for mitigating the

gyroscope and attitude errors (Prahl and Veth 2011; Ruotsalainen et al. 2012b).

The concept of a ‘visual gyroscope’ has been developed for resolving the user

orientation by tracking the motion of vanishing points in consecutive images

(Ruotsalainen et al. 2012a). The information of the vanishing point locations may be

transformed into the orientation measurements of the camera capturing the images. The

method resolves the accurate locations of the vanishing points by extracting all straight

lines from the image under examination and using a voting algorithm for all intersections

of each line pair. The Hough Transform is a comprehensive method for finding the lines,

but it is found to be the bottleneck in the overall processing time. The visual gyroscope has

been developed for real-time navigation and therefore, the calculation time is crucial.

Earlier, the vanishing point-induced orientation of the camera has been computed

using only information obtained from the images. The information has then been

transformed into heading, pitch and roll information of the user and integrated with

measurements obtained using an inertial measurement unit (IMU) and GNSS to attain a

full navigation solution. The challenges in measuring the orientation information using

vanishing point are that the method fails in sharp turns and is very dependent on the

structure of the environment, namely the method relies on finding straight lines in three

orthogonal directions and fails in situations when this is not fulfilled. When the

environment contains many lines violating the orthogonality requirements, or when there

are too few lines in total, the attitude obtained may be distorted. As the vanishing point

detection fails occasionally and the IMU measurements are too unreliable to evaluate the

magnitude of a turn, updating the attitude using GNSS measurements or indoor floor plans

if available is frequently necessary. This restricts the applicability of the vision-aided

IMU-based navigation in indoor and urban areas and therefore developing methods for

overcoming the situations when vanishing point calculation fails is crucial.

Borkowski and Veth (2010) developed a method for obtaining accurate vanishing

points using a predictive Hough Transform. The attitude information obtained from INS

was utilised to estimate the vanishing point location by calculating a probability density

function for the Hough parameter space. The probability density function was used as a

filter for the Standard Hough Transform (SHT) and resulted in a corrected vanishing point

location. The attitude information from the accurate vanishing point was finally used for

correcting the INS attitude with a Kalman filter and an improved navigation solution was

obtained.

The method presented in this paper is based on Borkowski and Veth (2010), but it also

accommodates for the line detection acceleration needs. Instead of using the probability

density function as a filter for the SHT (Hough 1960), the probabilities are used as weights

in a novel line detecting algorithm derived from the Probabilistic Hough Transform

(Kalviainen et al. 1995). The method presented in this paper provides thus improved

vanishing point estimations in defective situations with reduced computation-time

requirements and therefore improved vision aiding for obtaining user attitude in real time.

2. Mitigation of attitude errors using a visual gyroscope based on SHT

Urban and indoor environments contain many straight lines that are often forming an

orthogonal grid in three dimensions. An image of the parallel straight lines in the direction

of the camera’s principal axis may be used for calculating the location of the central

L. Ruotsalainen et al.210

Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

vanishing point. The change of the vanishing point location in consecutive images

encompasses information about the relative attitude of the camera.

The intersection point v of a ray through the camera centre (principal ray) having

direction d and the image plane is called the vanishing point of the lines having also the

direction d. The vanishing point v is related to the direction d as v ¼ Kd (Hartley and

Zissermann 2003), where K is the camera calibration matrix encompassing the intrinsic

parameters of the camera. Using the assumption that the skew of the camera is 0, the

calibration matrix encompasses the focal length of the camera ( fx, fy) and the principal

point (u, v) and is

K ¼f x 0 u

0 f y v

0 0 1

2664

3775: ð1Þ

The directions d and d0 of two vanishing points in consecutive images are related by

the rotation matrix R as d0 ¼ Rd. The rotation R of the camera may also be thought as the

rotation from the initial position where the camera is aligned with the navigation frame

so that the z-axis of the camera is pointing to the direction of the propagation and the x- and

y-axes are orthogonal to the z-axis as shown in Figure 1. To be able to transform the

measurements into the navigation frame the initial orientation of the camera with respect

to the navigation frame has to be measured in the initialisation phase. To integrate the INS

and visual measurements, the orientation of the camera with respect to the user body has to

Figure 1. Camera and image coordinate frames.

Journal of Location Based Services 211

Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

be evaluated as well. When the rotation of the camera is used as a visual gyroscope and its

measurements are integrated with the initial attitude and INS measurements, the errors in

attitude during the navigation are mitigated and an improved solution is attained

(Ruotsalainen et al. 2012b).

The visual gyroscope used in (Ruotsalainen et al. 2012b) is based on finding all straight

lines in the environment using the SHT (Hough 1960) and the lines being classified based

on their orientation. The central vanishing point is then calculated by voting for the

intersection point of the lines going in the direction of propagation and the vertical

vanishing point location is estimated using the vertical lines. The accuracy of the

vanishing points obtained is evaluated based on the geometry of the lines used for

calculations as explained below.

Because the motion of a pedestrian is unpredictable, the location of the vanishing point

is unrestricted and cannot be estimated without any a priori information. When the user is

turning, the vanishing point may even fall outside the image and still be correct. The

concept of “Line Dilution of Precision (LDOP)” was developed based on the geometry of

the lines used for calculating the central vanishing point. The image is divided into four

sections around the vanishing point obtained. When the lines are found from at least three

sections, the reliability of the vanishing point is high and a small LDOP value is assigned.

When the lines are found from one or two sections, their mutual orientation is evaluated

following the work of Alizahed-Shabdiz and Heidari (2009). The angle difference between

each line pair and the image x-axis is computed. The angle between the x-axis and the first

line (a1) and the second line (a2) is obtained using the starting point of the first and second

lines (x1, y1), (x2, y2), respectively, the estimated central vanishing point (xv, yv) and

distance (Di) of the estimated vanishing point from the starting point of line i (i ¼ 1,2) as

cos ðaiÞ ¼ xv 2 xi

Di

; sin ðaiÞ ¼ yv 2 yi

Di

: ð2Þ

The geometry matrix G is formed as

G ¼cos ða1Þ sin ða1Þcos ða2Þ sin ða2Þ

" #: ð3Þ

The matrix H is formed from the geometry matrix G using H ¼ (GT £ G).

jHj ¼ sin2(a1 2 a2) is the determinant of H and

H21 ¼ 1

Hj jsin 2ða1Þ þ sin 2ða2Þ 2 cos ða1Þ sin ða1Þ þ cos ða2Þ sin ða2Þ

cos ða1Þ sin ða1Þ þ cos ða2Þ sin ða2Þ cos 2ða1Þ þ cos 2ða2Þ

" #:

ð4Þ

The LDOP value is defined as

LDOP ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1

Hð cos 2ða1Þ þ cos 2ða2Þ þ sin 2ða1Þ þ sin 2ða2ÞÞ

r: ð5Þ

For any two angles ða1;a2Þ the LDOP equation may be now written as LDOP ¼ ffiffiffiffiffiffiffiffiffiffiffiffi2=jGjp

.


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

The smallest possible LDOP value isffiffiffi2

pand arises from the maximum angle between the

two lines lying in the same quarter section, namely 90 degrees. When the LDOP value

exceeds a threshold the measurement is evaluated as erroneous and discarded in the

integration phase.

The pixel coordinates of the vanishing points are transformed into a camera

rotation using the relationship v ¼ KR. The attitude obtained using only IMU

measurements suffers from gyro errors, namely the gyro bias, scale factor, non-

orthogonalities, the g-dependent error and noise. The error model is

~vbib ¼ Sgv

bib þ bg þGfbib þ hg; ð6Þ

where ~vbib is the gyroscope angular velocity measurement, Sg is a matrix including the

scale factors and non-orthogonalities, vbib is the body (b) turn rate with respect to the

inertial (i) frame measured by the gyroscope, bg is the gyro biases,G is the 3 £ 3 matrix of

the g-sensitivity coefficients, fbib is the specific force and hg is the noise. These errors are

mitigated using Kalman filtering as explained below.

GNSS, IMU and visual data are fused using a tightly coupled 21-state extended

Kalman filter (EKF) defined as

d_re ¼ dve; d_ve ¼ Nedre 2 2Veiedv

e 2 Fe1 þ RebðbaÞ;

_1e ¼ 2Veie1

e þ Reb ðI 2 SgÞvb

ib þ bg þGfbib� �

; _ba ¼ 2t21a

_ba;

_bg ¼ 2t21g

_bg; _Sg ¼ 0; _G ¼ 0:

ð7Þ

The parameters are the perturbations of position and velocity (re, v e) in the earth centred

earth fixed (ECEF) frame, the Euler angles relating the body frame to the ECEF frame (1)and the biases of the accelerometer and gyro (ba, bg). The inertia tensor is denoted N e, the

skew symmetric forms of the earth rotation vectorVeie and specific force measurement F e.

The rotation matrix Reb rotates the specific force and angular velocity from the body to the

ECEF frame. S is a matrix containing the gyroscope scale factors and G g-sensitivity

coefficients.

Though the vision aiding method provides a significant improvement for the final

attitude of the user, as described in Section 5, the accuracy of the heading obtained using

the visual gyroscope suffers from irregularities of the environment (namely lines violating

the orthogonality requirement) and the calculation is relatively slow for real-time

implementation. Even when using a high-quality camera, 20–30% of the vanishing point

calculations failed during two 30 minutes experiments in challenging environment

(Ruotsalainen et al. 2012c). The method failed to observe the vanishing point and the

attitude measurements were erroneous when the majority of the lines violated the

orthogonality requirement as the visual gyroscope was relying only on visual perception

and received no other information of the motion of the user.

2.1. Performance study of the visual gyroscope’s smartphone implementation

The visual gyroscope has been implemented herein into a Nokia N8 Symbian smartphone.

The implementation was done using Cþþ and the OpenCV open source visual

library. The total processing time for automatically capturing an image, finding the straight

lines, voting for the vanishing point and calculating the orientation of the camera


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

is 2.5 seconds. The bottlenecks of the calculation are the image capturing (1.2 seconds)

and line detection using the SHT algorithm (1 second). The slowness of the Hough

Transform was also acknowledged in Huttunen and Piche (2011), in discussing a different

visual gyroscope implementation in a smartphone. Decreasing the time needed for

extracting the lines and calculating the vanishing point is needed for a real-time navigation

solution.

3. Hough Transform in line detection

This section introduces the SHT algorithm widely used for line detection and its

modifications called Probabilistic Hough Transform methods to reduce the computation

needed.

3.1. Standard Hough Transform

The Hough Transform algorithm (Hough 1960) transforms each input image point into a

line in a parameter space. The original Hough’s algorithm used a slope–intercept

parameterisation of the lines, but the algorithm was further developed by Duda and Hart

(1972) expressing the line by its normal’s angle with the x-axis (u) and its distance from

the origin (r) as

r ¼ x cos ðuÞ þ y sin ðuÞ: ð8Þ

Every point (r, u) in the parameter space corresponds to one line in the image space. All

image points in the input image are examined and a r value for each possible u is

computed. Usually the input image is the output of an edge detection algorithm containing

only the pixels that are estimated to belong to an edge. An accumulator keeps track on the

votes obtained by each (r, u) – pair. After examining all image points the local maximums

in the accumulator are identified and stated to represent lines in the image.

3.2. Probabilistic Hough Transform

SHT finds the relevant lines in an image, but its weaknesses are the processing time and

memory requirements. Applications aimed at pedestrian navigation must perform in real

time and often in platforms with reduced processing capabilities such as smartphones.

Much research for accelerating the Hough Transform computations occurred in the 90s

and a good overview is provided in Kalviainen et al. (1995).

As in the SHT all edge points present in an image are exploited, in the Probabilistic

Hough Transform (Kiriyati, Eldar, and Bruckstein 1991) a subset of image points is

selected randomly and a SHT performed. According to Matas, Galambos, and Kittler

(2000), the original Probabilistic Hough Transform reduces the computation if a priori

information of the number of lines is available. As this is not usually the case they

developed a Progressive Probabilistic Hough Transform. In their method, the image points

used are selected randomly and the parameter pair is used to represent a line when the

votes it has received exceed the number that would be expected by random noise.

The number of image points needed to represent a line is evaluated progressively based on

the rate of the pixels examined and the pixels voting for a certain line.


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

4. Visual gyroscope based on Probabilistic Hough Transform

The visual gyroscope presented in this paper utilises the attitude provided by the INS to

estimate the vanishing point location and the estimation for detecting the lines most

probably intersecting at the vanishing point using the Probabilistic Hough Transform. The

procedure is explained below.

The attitude of the user obtained from the INS may be transformed into an estimate of

the vanishing point using the relation

v ¼ KCcbC

bnR; ð9Þ

where Ccb is the direction cosine matrix (Titterton and Weston 2004) from body to the

camera frame and Cbn from navigation frame to the body frame. The rotation matrix R is

the rotation of the camera position from the initial position and it is normalised.

The expected location of the vanishing point (~v) is characterised by a Gaussian density

function (Borkowski and Veth 2010) as

ru , Nðmr;s2rÞ; ð10Þ

where the distance ru related to a certain angle u is normally distributed. The mean mr is

computed for every possible angle u for lines going through the estimated vanishing point

(~v). This information is further used for finding the lines in the image supporting the

estimated vanishing point as explained below.

A pixel is selected randomly from the set containing all edge pixels of the image. The

distance r is calculated for all possible u and the values in corresponding accumulator cells

are increased by summing the values of the probability density function for the obtained

distance and mean with the existing cell value. The SHT increases all accumulator cells

equally because its objective is to find all straight lines present in the image. Here, the

objective is to find the lines supporting the vanishing point and therefore the closer the

possible line is to the estimated vanishing point, the more the accumulator cell value is

increased. When the value in the accumulator cell exceeds a threshold, a line is found and

all other image points belonging to the line are removed from the pixel set as well as all the

votes in the accumulator arising from the line. In this way, the number of image points

examined and therefore the computation time needed decreases. As the points having a

larger likelihood of belonging to a line going through the estimated vanishing point or a

point close to it are given more weight, the lines found are likely to be in the direction of

supporting the central vanishing point.

A (r, u)-pair in the parameter space represents the intersection point of all collinear

points (xi, yi) in the image points. This is also true the other way around (Duda and Hart

1972); all points (ri, ui) satisfying the equation

ri ¼ x cos ðuiÞ þ y sin ðuiÞ ð11Þ

represent lines intersecting at point (x, y). If the line detection done by emphasising the

lines supporting the estimated vanishing point direction has succeeded, all lines found

intersect at the correct vanishing point and therefore the point is found using a least-

squares estimation technique. The user attitude is found from the vanishing point using

again Equation (9).


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

Two parameters selected for the calculation are crucial for the performance of the

visual gyroscope, namely the threshold for deciding when a line is found and the standard

deviation (SD) of the estimated vanishing point value. When the threshold for finding a

line is deficient, the rate of false positives is large, and when it is too large, the computation

time increases and occasionally too few lines are found from the low-light indoor

environment resulting in an inaccurate vanishing point location. Also, when the SD

assigned for the estimated vanishing point value is too large the errors in IMU-induced

attitude distort the line detection by emphasising points close to the estimated point

probably not even belonging to a line. Results shown herein were obtained using an SD

value of 20 and a threshold of 0.4. In this configuration, the number of points needed is 20,

when they all fully agree with the estimated vanishing point. When the estimated

vanishing point and the image points potentially representing a line are not in such full

agreement, the number of required image points is larger. As a conclusion of numerous

experiments conducted this is the optimal parameter selection for the algorithm. When the

threshold is increased, the execution time grows significantly and the possibility of

extracting too few lines for reliable vanishing point computation hinders the method.

Similarly, when the threshold is decreased, the computation time is reduced, but as the

number of points required to support a line is also decreased, the number of false positives

in the line extraction deteriorates the solution.

Figure 2 shows the result of line detection and vanishing point calculations. The blue

lines are extracted using the Probabilistic Hough Transform, the green point is the

vanishing point estimation based on the IMU attitude and the red point is the vanishing

point corrected using the lines found. The figure shows how the vanishing point is found

reliably even when the IMU-induced attitude and therefore the estimated vanishing point

is distorted.

The IMU and visual attitude measurements calculated from the corrected vanishing

point are integrated using a Kalman filter propagating the attitude. When the algorithm

fails to find any lines, the estimated vanishing point is used for observing the attitude. The

objective of the research was that eventually the vanishing point tracking will be accurate

Figure 2. Estimated vanishing point location (green) corrected (red) using lines extracted with theProbabilistic Hough Transform.


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

enough for also determining the magnitude of a turn and therefore the re-initialisation

phase can be omitted in further developments.

5. Experiments

In (Ruotsalainen et al. 2012b) the visual attitude was calculated using only the vanishing

point location and integrated as visual update with INS measurements using the 21-state

EKF as described previously. The user position was initialised with GNSS measurements

and during the navigation the heading was re-initialised after every steep turn using a floor

plan. The test equipment consisted of Analog Devices ADIS16488 IMU (Analog Devices

2010) and a GoPro Hero helmet camera (GoPro 2012). Images were captured using a

10Hz rate and IMU measurements with 20Hz rate. A NovAtel SPAN-SE GPS/

GLONASS receiver with a Northrop Grumman’s tactical grade LCI IMU was used as a

reference system. All equipment was attached to a backpack single as shown in Figure 3

and carried by a pedestrian.

During the 48-minute experiment conducted mainly indoors the root mean square

(rms) of the heading error decreased from 29.5 degrees when only IMU measurements

Figure 3. Test equipment set up.


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

were used to 2.1 degrees with visual updates. When navigating using an IMU-only

solution the heading error was 90 arc minutes at the end of the experiment. Considering

that the micro-electro-mechanical systems (MEMS) IMU uses Analog Devices’ best gyro,

the fact that GNSS is not available at all after initialisation and the solution does use a

magnetic compass, this heading error is comparable with other state-of-the-art solutions.

The difference between IMU-only and vision-aided attitude accuracy was not as

remarkable for roll and pitch errors, namely the rms of the errors decreased from 2 to 1.4

degrees for roll and from 1.7 to 1.4 degrees for pitch. Figure 4 shows all the attitude errors

when using IMU-only (blue) and vision-aided IMU (green). However, without re-

initialising the heading after sharp turns the vanishing point-induced heading would be

incorrect and eventually distort the user attitude solution. All calculations were done in

post-processing as the computation time of the visual measurements was too long for a

real-time solution. Therefore, there is a need for more tolerant vanishing point detection

running in real time.

To evaluate the concept presented in this paper for accelerated vanishing point

detection, a subset of the collected data was re-processed using the Probabilistic Hough

Transform-based visual gyroscope providing reduced processing time. The data-set

consisted of 80 seconds of data and therefore 800 images were examined. As stated in

Matas, Galambos, and Kittler (2000), the processing time of an algorithm is dependent on

the computer used and the implementation details. Therefore, the effect of the algorithm is

shown by comparing the number of image points examined, in other words iterations of

the parameter calculation. The SHT examines all pixels in the input image and afterwards

searches for local maxima from the accumulator to find the lines. The algorithm presented

uses a fraction of the image points and already detects the lines during the point

examination. The algorithm presented uses on average 45% of all image points and

Figure 4. Attitude errors using no update (blue) and using vision-aided updates (green).


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

therefore the computation is anticipated to be accelerated in the same proportion to the

SHT. Table 1 gives the test iteration statistics. Compared with the SHT, the number of

lines extracted is approximately equal for the algorithm proposed when the given

parameter values are used. The accuracy of the line extraction is difficult to compare, for

the lines found using the Probabilistic Hough Transform are seldom the same lines as

found using the SHT. This is because lines found using the former method are not usually

the lines receiving the maximum number of votes in the SHT, but the lines supporting the

estimated vanishing point. Therefore, the method should be employed only for extracting

lines for computing central vanishing points when IMU attitude measurements are used

and not as a generic line extraction algorithm.

The algorithm succeeds in finding the vanishing points in environments having

restricted light conditions or lines violating the orthogonality requirement. An example of

an image suffering from both situations is shown in Figure 5. The vanishing point

computed by the visual gyroscope using the SHT and voting is shown on left and using the

visual gyroscope with the Probabilistic Hough Transform presented in this paper is on the

right.

The method is also tolerant for large errors in the IMU measurements when the

parameters of the Probabilistic Hough Transform algorithm are carefully selected.

Figure 6 shows how an estimated vanishing point resulting from large temporary errors in

IMU measurements is corrected through the line detection presented. Note that the image

is not corrected from distortion and therefore the lines found do not exactly correspond to

the lines in the original image.

The method gives also promising results for future turn detection that has so far been

one of the most significant obstacles preventing the use of vision-aided inertial sensors

autonomously for navigation in unknown indoor environments. In turning situations the

estimated vanishing point obtained by propagating the attitude from the vision-aided

integrated solution falls outside the image at the same time as the real vanishing point

obtained from the Probabilistic Hough Transform detection is found from the other side of

the image as shown in Figure 7. When this contradiction is used in integration, at least the

existence of a steep turn is observed. Observing the magnitude of the turn is a future

research objective.

6. Conclusions

A novel method was presented for observing the user attitude based on a visual gyroscope

estimating the vanishing point location and utilising the information for extracting lines

using the Probabilistic Hough Transform. Since the calculation used some a priori

information of the vanishing point location from an IMU, the method is more tolerant of

environmental irregularities than when the calculations are done using only visual

information. The search space for the lines can be restricted to contain only the lines going

in the direction of propagation using a probability distribution computed from the

Table 1. Algorithm iteration statistics.

Ratio of image points used compared with all image points (%)

Min. Mean Max. SD

27 45 67 8


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

estimation of the vanishing point location. Therefore, the Probabilistic Hough Transform

method presented should be employed only for extracting lines for computing central

vanishing points when IMU attitude measurements are used and not as a generic line

extraction algorithm.

When the view to the straight lines is blocked for a long time, like when navigating in

crowded open spaces or in environments with a majority of lines violating the

orthogonality requirements set for the vanishing point-based solution, the performance of

the method based on the SHT is equal to the IMU-only navigation solution. All of the

visual heading measurements are discarded by the error detection algorithm when

the LDOP derived from the lines found express the unsuitability of the environment for the

vanishing point calculation. Also, the solution based on the Probabilistic Hough

Transform discards a majority of the vanishing point measurements because in most

situations the irregular lines arising from the dynamic and non-orthogonal objects do not

intersect in an unambiguous point that could be assessed as a central vanishing point.

Therefore, if the environment is mainly unsuitable for the vanishing point calculation the

worst-case scenario is that the performance is equal to the IMU-only solution. However,

previous experiments show that the method is also feasible in an open space and that the

crowds rarely prevent vanishing point calculations (Ruotsalainen et al. 2012a).

Future work includes integration of the attitude measurements obtained using the

novel vanishing point detection method presented herein. The results obtained using the

Figure 5. Evaluation of vanishing point detection in an environment suffering from low lighting andnon-orthogonal lines.

Figure 6. Detected vanishing point (red) may be used to correct large errors in IMU measurementsresulting in erroneous estimated vanishing point location (green).


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

previously developed vision aiding method were shown to provide significantly increased

accuracy for pedestrian navigation in challenging indoor environments. The method

however suffered from slow computation and an inability to detect sharp turns. The novel

method using the Probabilistic Hough Transform is expected to result in reduced

computation time and, in the future, in a real-time pedestrian visual-navigation solution

free from frequent re-initialisation from external information when integrated with INS

measurements.

Acknowledgements

LauraRuotsalainen has received aNokia Foundation Scholarship for the years 2011 and 2012 to supporther PhD thesis work. The authors would like to thank David Garrett, Summer student in the PLANGroup, for his contribution to the experiments. This researchwas performedwhenMs.Ruotsalainenwasa visiting student at the University of Calgary.

References

Alizahed-Shabdiz, F., and M. Heidari. 2009. Method of determining position in a hybrid positioningsystem using a dilution of precision metric. US Patent application 2011/0080317, filedOctober 2.

Analog Devices. 2010. “ADIS16375 Data Sheet (Rev. C).” pp. 28. http://www.analog.com/static/imported-files/data_sheets/ADIS16375.pdf

Borkowski, J. M., and M. J. Veth. 2010. “Passive Indoor Image-Aided Inertial Attitude EstimationUsing a Predictive Hough Transformation.” Proceedings of IEEE ION PLANS 2010, IndianWells, CA, pp. 295–302.

Collin, J. 2006. “Investigations of Self-Contained Sensors for Personal Navigation.” PhD diss.,Tampere University of Technology.

Duda, R. O., and P. E. Hart. 1972. “Use of the Hough Transformation to Detect Lines and Curves inPictures, Graphics and Image Processing.” Communications of ACM 15 (1): 11–15.

GoPro. 2012. Camera Specifications. Accessed June 15. http://www.gopro.com/cameras/hd-helmet-hero-camera/

Figure 7. Conflict between estimated and detected vanishing point locations indicates the existenceof a steep turn.


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

Hartley, R., and A. Zissermann. 2003. Multiple View Geometry in Computer Vision. 2nd ed.Cambridge, UK: Cambridge University Press.

Hough, P. V. C. 1960. Method and means for recognizing complex patterns US patent 3069654, filedMarch 25.

Huttunen, V., and R. Piche. 2011. “A Monocular Camera Gyroscope.” Report 98, Department ofMathematics, Tampere University of Technology.

Kalviainen, H., P. Hirvonen, L. Xu, and E. Oja. 1995. “Probabilistic and Non-Probabilistic HoughTransforms: Overview and Comparisons.” Image and Vision Computing 13: 239–252.

Kiriyati, N., Y. Eldar, and A. M. Bruckstein. 1991. “A Probabilistic Hough Transform.” PatternRecognition 24 (4): 303–316.

Matas, J., C. Galambos, and J. Kittler. 2000. “Robust Detection of Lines Using ProgressiveProbabilistic Hough Transform.” Computer Vision and Image Understanding 78 (1): 119–137.

Prahl, D. M., and M. J. Veth. 2011. “Coupling Vanishing Point Tracking with Inertial Navigation toProduce Drift-Free Attitude Estimates in a Structured Environment.” Proceedings of ION GNSS2011, Portland OR, 3571–3581, Institute of Navigation, September 19–23.

Ruotsalainen, L., H. Kuusniemi, M. Z. H. Bhuiyan, L. Chen, and R. Chen. 2012a. “A TwoDimensional Pedestrian Navigation Solution Aided with a Visual Gyroscope and a VisualOdometer.” GPS Solutions. doi: 10.1007/s10291-012-0302-8.

Ruotsalainen L., J. Bancroft, and G. Lachapelle. 2012b. “Mitigation of Attitude and Gyro ErrorsThrough Vision Aiding.” Proceedings of IPIN 2012, Sydney, Australia, p. 9, IEEE Xplore,November 13–15.

Ruotsalainen, L., J. Bancroft, G. Lachapelle, H. Kuusniemi, and R. Chen. 2012c. “Effect of CameraCharacteristics of the Accuracy of a Visual Gyroscope for Indoor Pedestrian Navigation.”Proceedings of UPINLBS 2012, Helsinki, Finland, p. 8, IEEE Xplore, October 2–4.

Saarinen, J. 2009. “A Sensor-Based Personal Navigation System and Its Application forIncorporating Humans into a Human-Robot Team.” PhD diss., Helsinki University ofTechnology.

Titterton, D., and J. Weston. 2004. Strapdown Inertial Navigation Technology. 2nd ed. Stevenage,UK: IET.


Dow

nloa

ded

by [

FU B

erlin

] at

06:

19 1

6 O

ctob

er 2

014

enhanced pedestrian attitude estimation using vision aiding

Documents