enhanced pedestrian attitude estimation using vision aiding
Post on 10-Feb-2017
216 Views
Preview:
TRANSCRIPT
This article was downloaded by: [FU Berlin]On: 16 October 2014, At: 06:18Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Journal of Location Based ServicesPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/tlbs20
Enhanced pedestrian attitudeestimation using vision aidingLaura Ruotsalainen a , Jared Bancroft b , Gérard Lachapelle b &Heidi Kuusniemi aa Department of Navigation and Positioning , Finnish GeodeticInstitute , Masala , Finlandb Department of Geomatics Engineering , Schulich School ofEngineering, University of Calgary, Calgary , CanadaPublished online: 21 Aug 2013.
To cite this article: Laura Ruotsalainen , Jared Bancroft , Gérard Lachapelle & Heidi Kuusniemi(2013) Enhanced pedestrian attitude estimation using vision aiding, Journal of Location BasedServices, 7:3, 209-222, DOI: 10.1080/17489725.2013.819450
To link to this article: http://dx.doi.org/10.1080/17489725.2013.819450
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Enhanced pedestrian attitude estimation using vision aiding
Laura Ruotsalainena*, Jared Bancroftb, Gerard Lachapelleb and Heidi Kuusniemia
aDepartment of Navigation and Positioning, Finnish Geodetic Institute, Masala, Finland;bDepartment of Geomatics Engineering, Schulich School of Engineering, University of Calgary,
Calgary, Canada
(Received 10 March 2013; final version received 12 June 2013; accepted 19 June 2013)
Inertial Navigation System (INS) sensors are widely used for augmenting GlobalNavigation Satellite System measurements in urban environments and in theindoors. With a known initial position, the current position may be propagatedusing gyroscopes and accelerometers forming the INS for a limited time. Thelimitation of the self-contained sensors is the cumulative measurement errorsthat affect the accuracy of the attitude obtained using the gyroscopes. Visionaiding has proven to be a feasible method for mitigating these errors. This paperintroduces a method to obtain attitude measurements by tracking the motion ofvanishing points in consecutive images and integrating these measurements withthe attitude observed by INS using an extended Kalman filter. The experimentsshow that vision aiding results in significant improvement of the user attitudeand therefore the navigation solution. The challenges in vanishing point-basedvision aiding are the processing time and the method’s lack of capability toperceive sharp turns. These issues are addressed by developing an algorithmbased on the Probabilistic Hough Transform for more efficient vanishing pointcalculation which also provides a means for turn detection. These improvementsadvance the objective of developing a real-time seamless indoor–outdoorpedestrian navigation system utilising vision aiding.
Keywords: vision aiding; vanishing point; Hough Transform; pedestriannavigation; INS
1. Introduction
Pedestrian navigation sets demanding requirements for the equipment and software. The
equipment used has to be light to carry, reasonably priced and easy to use in addition to
providing accurate and real-time navigation. Global Navigation Satellite System (GNSS)
receivers embedded in smartphone succeed in providing the above in open outdoor
environments, but the performance is degraded in indoor and urban areas. Inertial
sensors are widely used for augmenting the GNSS measurements in these environments.
With a known initial position, the current position may be propagated using a triad of
gyroscopes and accelerometers forming the Inertial Navigation System (INS) for a limited
time (Collin 2006). The limitation of the self-contained sensors is the cumulative
measurement errors that affect the accuracy of the attitude obtained using the gyroscopes
q 2013 Taylor & Francis
*Corresponding author. Email: laura.ruotsalainen@fgi.fi
Journal of Location Based Services, 2013
Vol. 7, No. 3, 209–222, http://dx.doi.org/10.1080/17489725.2013.819450
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
(Saarinen 2009). Vision aiding has proven to be a feasible method for mitigating the
gyroscope and attitude errors (Prahl and Veth 2011; Ruotsalainen et al. 2012b).
The concept of a ‘visual gyroscope’ has been developed for resolving the user
orientation by tracking the motion of vanishing points in consecutive images
(Ruotsalainen et al. 2012a). The information of the vanishing point locations may be
transformed into the orientation measurements of the camera capturing the images. The
method resolves the accurate locations of the vanishing points by extracting all straight
lines from the image under examination and using a voting algorithm for all intersections
of each line pair. The Hough Transform is a comprehensive method for finding the lines,
but it is found to be the bottleneck in the overall processing time. The visual gyroscope has
been developed for real-time navigation and therefore, the calculation time is crucial.
Earlier, the vanishing point-induced orientation of the camera has been computed
using only information obtained from the images. The information has then been
transformed into heading, pitch and roll information of the user and integrated with
measurements obtained using an inertial measurement unit (IMU) and GNSS to attain a
full navigation solution. The challenges in measuring the orientation information using
vanishing point are that the method fails in sharp turns and is very dependent on the
structure of the environment, namely the method relies on finding straight lines in three
orthogonal directions and fails in situations when this is not fulfilled. When the
environment contains many lines violating the orthogonality requirements, or when there
are too few lines in total, the attitude obtained may be distorted. As the vanishing point
detection fails occasionally and the IMU measurements are too unreliable to evaluate the
magnitude of a turn, updating the attitude using GNSS measurements or indoor floor plans
if available is frequently necessary. This restricts the applicability of the vision-aided
IMU-based navigation in indoor and urban areas and therefore developing methods for
overcoming the situations when vanishing point calculation fails is crucial.
Borkowski and Veth (2010) developed a method for obtaining accurate vanishing
points using a predictive Hough Transform. The attitude information obtained from INS
was utilised to estimate the vanishing point location by calculating a probability density
function for the Hough parameter space. The probability density function was used as a
filter for the Standard Hough Transform (SHT) and resulted in a corrected vanishing point
location. The attitude information from the accurate vanishing point was finally used for
correcting the INS attitude with a Kalman filter and an improved navigation solution was
obtained.
The method presented in this paper is based on Borkowski and Veth (2010), but it also
accommodates for the line detection acceleration needs. Instead of using the probability
density function as a filter for the SHT (Hough 1960), the probabilities are used as weights
in a novel line detecting algorithm derived from the Probabilistic Hough Transform
(Kalviainen et al. 1995). The method presented in this paper provides thus improved
vanishing point estimations in defective situations with reduced computation-time
requirements and therefore improved vision aiding for obtaining user attitude in real time.
2. Mitigation of attitude errors using a visual gyroscope based on SHT
Urban and indoor environments contain many straight lines that are often forming an
orthogonal grid in three dimensions. An image of the parallel straight lines in the direction
of the camera’s principal axis may be used for calculating the location of the central
L. Ruotsalainen et al.210
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
vanishing point. The change of the vanishing point location in consecutive images
encompasses information about the relative attitude of the camera.
The intersection point v of a ray through the camera centre (principal ray) having
direction d and the image plane is called the vanishing point of the lines having also the
direction d. The vanishing point v is related to the direction d as v ¼ Kd (Hartley and
Zissermann 2003), where K is the camera calibration matrix encompassing the intrinsic
parameters of the camera. Using the assumption that the skew of the camera is 0, the
calibration matrix encompasses the focal length of the camera ( fx, fy) and the principal
point (u, v) and is
K ¼f x 0 u
0 f y v
0 0 1
2664
3775: ð1Þ
The directions d and d0 of two vanishing points in consecutive images are related by
the rotation matrix R as d0 ¼ Rd. The rotation R of the camera may also be thought as the
rotation from the initial position where the camera is aligned with the navigation frame
so that the z-axis of the camera is pointing to the direction of the propagation and the x- and
y-axes are orthogonal to the z-axis as shown in Figure 1. To be able to transform the
measurements into the navigation frame the initial orientation of the camera with respect
to the navigation frame has to be measured in the initialisation phase. To integrate the INS
and visual measurements, the orientation of the camera with respect to the user body has to
Figure 1. Camera and image coordinate frames.
Journal of Location Based Services 211
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
be evaluated as well. When the rotation of the camera is used as a visual gyroscope and its
measurements are integrated with the initial attitude and INS measurements, the errors in
attitude during the navigation are mitigated and an improved solution is attained
(Ruotsalainen et al. 2012b).
The visual gyroscope used in (Ruotsalainen et al. 2012b) is based on finding all straight
lines in the environment using the SHT (Hough 1960) and the lines being classified based
on their orientation. The central vanishing point is then calculated by voting for the
intersection point of the lines going in the direction of propagation and the vertical
vanishing point location is estimated using the vertical lines. The accuracy of the
vanishing points obtained is evaluated based on the geometry of the lines used for
calculations as explained below.
Because the motion of a pedestrian is unpredictable, the location of the vanishing point
is unrestricted and cannot be estimated without any a priori information. When the user is
turning, the vanishing point may even fall outside the image and still be correct. The
concept of “Line Dilution of Precision (LDOP)” was developed based on the geometry of
the lines used for calculating the central vanishing point. The image is divided into four
sections around the vanishing point obtained. When the lines are found from at least three
sections, the reliability of the vanishing point is high and a small LDOP value is assigned.
When the lines are found from one or two sections, their mutual orientation is evaluated
following the work of Alizahed-Shabdiz and Heidari (2009). The angle difference between
each line pair and the image x-axis is computed. The angle between the x-axis and the first
line (a1) and the second line (a2) is obtained using the starting point of the first and second
lines (x1, y1), (x2, y2), respectively, the estimated central vanishing point (xv, yv) and
distance (Di) of the estimated vanishing point from the starting point of line i (i ¼ 1,2) as
cos ðaiÞ ¼ xv 2 xi
Di
; sin ðaiÞ ¼ yv 2 yi
Di
: ð2Þ
The geometry matrix G is formed as
G ¼cos ða1Þ sin ða1Þcos ða2Þ sin ða2Þ
" #: ð3Þ
The matrix H is formed from the geometry matrix G using H ¼ (GT £ G).
jHj ¼ sin2(a1 2 a2) is the determinant of H and
H21 ¼ 1
Hj jsin 2ða1Þ þ sin 2ða2Þ 2 cos ða1Þ sin ða1Þ þ cos ða2Þ sin ða2Þ
cos ða1Þ sin ða1Þ þ cos ða2Þ sin ða2Þ cos 2ða1Þ þ cos 2ða2Þ
" #:
ð4Þ
The LDOP value is defined as
LDOP ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1
Hð cos 2ða1Þ þ cos 2ða2Þ þ sin 2ða1Þ þ sin 2ða2ÞÞ
r: ð5Þ
For any two angles ða1;a2Þ the LDOP equation may be now written as LDOP ¼ ffiffiffiffiffiffiffiffiffiffiffiffi2=jGjp
.
L. Ruotsalainen et al.212
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
The smallest possible LDOP value isffiffiffi2
pand arises from the maximum angle between the
two lines lying in the same quarter section, namely 90 degrees. When the LDOP value
exceeds a threshold the measurement is evaluated as erroneous and discarded in the
integration phase.
The pixel coordinates of the vanishing points are transformed into a camera
rotation using the relationship v ¼ KR. The attitude obtained using only IMU
measurements suffers from gyro errors, namely the gyro bias, scale factor, non-
orthogonalities, the g-dependent error and noise. The error model is
~vbib ¼ Sgv
bib þ bg þGfbib þ hg; ð6Þ
where ~vbib is the gyroscope angular velocity measurement, Sg is a matrix including the
scale factors and non-orthogonalities, vbib is the body (b) turn rate with respect to the
inertial (i) frame measured by the gyroscope, bg is the gyro biases,G is the 3 £ 3 matrix of
the g-sensitivity coefficients, fbib is the specific force and hg is the noise. These errors are
mitigated using Kalman filtering as explained below.
GNSS, IMU and visual data are fused using a tightly coupled 21-state extended
Kalman filter (EKF) defined as
d_re ¼ dve; d_ve ¼ Nedre 2 2Veiedv
e 2 Fe1 þ RebðbaÞ;
_1e ¼ 2Veie1
e þ Reb ðI 2 SgÞvb
ib þ bg þGfbib� �
; _ba ¼ 2t21a
_ba;
_bg ¼ 2t21g
_bg; _Sg ¼ 0; _G ¼ 0:
ð7Þ
The parameters are the perturbations of position and velocity (re, v e) in the earth centred
earth fixed (ECEF) frame, the Euler angles relating the body frame to the ECEF frame (1)and the biases of the accelerometer and gyro (ba, bg). The inertia tensor is denoted N e, the
skew symmetric forms of the earth rotation vectorVeie and specific force measurement F e.
The rotation matrix Reb rotates the specific force and angular velocity from the body to the
ECEF frame. S is a matrix containing the gyroscope scale factors and G g-sensitivity
coefficients.
Though the vision aiding method provides a significant improvement for the final
attitude of the user, as described in Section 5, the accuracy of the heading obtained using
the visual gyroscope suffers from irregularities of the environment (namely lines violating
the orthogonality requirement) and the calculation is relatively slow for real-time
implementation. Even when using a high-quality camera, 20–30% of the vanishing point
calculations failed during two 30 minutes experiments in challenging environment
(Ruotsalainen et al. 2012c). The method failed to observe the vanishing point and the
attitude measurements were erroneous when the majority of the lines violated the
orthogonality requirement as the visual gyroscope was relying only on visual perception
and received no other information of the motion of the user.
2.1. Performance study of the visual gyroscope’s smartphone implementation
The visual gyroscope has been implemented herein into a Nokia N8 Symbian smartphone.
The implementation was done using Cþþ and the OpenCV open source visual
library. The total processing time for automatically capturing an image, finding the straight
lines, voting for the vanishing point and calculating the orientation of the camera
Journal of Location Based Services 213
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
is 2.5 seconds. The bottlenecks of the calculation are the image capturing (1.2 seconds)
and line detection using the SHT algorithm (1 second). The slowness of the Hough
Transform was also acknowledged in Huttunen and Piche (2011), in discussing a different
visual gyroscope implementation in a smartphone. Decreasing the time needed for
extracting the lines and calculating the vanishing point is needed for a real-time navigation
solution.
3. Hough Transform in line detection
This section introduces the SHT algorithm widely used for line detection and its
modifications called Probabilistic Hough Transform methods to reduce the computation
needed.
3.1. Standard Hough Transform
The Hough Transform algorithm (Hough 1960) transforms each input image point into a
line in a parameter space. The original Hough’s algorithm used a slope–intercept
parameterisation of the lines, but the algorithm was further developed by Duda and Hart
(1972) expressing the line by its normal’s angle with the x-axis (u) and its distance from
the origin (r) as
r ¼ x cos ðuÞ þ y sin ðuÞ: ð8Þ
Every point (r, u) in the parameter space corresponds to one line in the image space. All
image points in the input image are examined and a r value for each possible u is
computed. Usually the input image is the output of an edge detection algorithm containing
only the pixels that are estimated to belong to an edge. An accumulator keeps track on the
votes obtained by each (r, u) – pair. After examining all image points the local maximums
in the accumulator are identified and stated to represent lines in the image.
3.2. Probabilistic Hough Transform
SHT finds the relevant lines in an image, but its weaknesses are the processing time and
memory requirements. Applications aimed at pedestrian navigation must perform in real
time and often in platforms with reduced processing capabilities such as smartphones.
Much research for accelerating the Hough Transform computations occurred in the 90s
and a good overview is provided in Kalviainen et al. (1995).
As in the SHT all edge points present in an image are exploited, in the Probabilistic
Hough Transform (Kiriyati, Eldar, and Bruckstein 1991) a subset of image points is
selected randomly and a SHT performed. According to Matas, Galambos, and Kittler
(2000), the original Probabilistic Hough Transform reduces the computation if a priori
information of the number of lines is available. As this is not usually the case they
developed a Progressive Probabilistic Hough Transform. In their method, the image points
used are selected randomly and the parameter pair is used to represent a line when the
votes it has received exceed the number that would be expected by random noise.
The number of image points needed to represent a line is evaluated progressively based on
the rate of the pixels examined and the pixels voting for a certain line.
L. Ruotsalainen et al.214
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
4. Visual gyroscope based on Probabilistic Hough Transform
The visual gyroscope presented in this paper utilises the attitude provided by the INS to
estimate the vanishing point location and the estimation for detecting the lines most
probably intersecting at the vanishing point using the Probabilistic Hough Transform. The
procedure is explained below.
The attitude of the user obtained from the INS may be transformed into an estimate of
the vanishing point using the relation
v ¼ KCcbC
bnR; ð9Þ
where Ccb is the direction cosine matrix (Titterton and Weston 2004) from body to the
camera frame and Cbn from navigation frame to the body frame. The rotation matrix R is
the rotation of the camera position from the initial position and it is normalised.
The expected location of the vanishing point (~v) is characterised by a Gaussian density
function (Borkowski and Veth 2010) as
ru , Nðmr;s2rÞ; ð10Þ
where the distance ru related to a certain angle u is normally distributed. The mean mr is
computed for every possible angle u for lines going through the estimated vanishing point
(~v). This information is further used for finding the lines in the image supporting the
estimated vanishing point as explained below.
A pixel is selected randomly from the set containing all edge pixels of the image. The
distance r is calculated for all possible u and the values in corresponding accumulator cells
are increased by summing the values of the probability density function for the obtained
distance and mean with the existing cell value. The SHT increases all accumulator cells
equally because its objective is to find all straight lines present in the image. Here, the
objective is to find the lines supporting the vanishing point and therefore the closer the
possible line is to the estimated vanishing point, the more the accumulator cell value is
increased. When the value in the accumulator cell exceeds a threshold, a line is found and
all other image points belonging to the line are removed from the pixel set as well as all the
votes in the accumulator arising from the line. In this way, the number of image points
examined and therefore the computation time needed decreases. As the points having a
larger likelihood of belonging to a line going through the estimated vanishing point or a
point close to it are given more weight, the lines found are likely to be in the direction of
supporting the central vanishing point.
A (r, u)-pair in the parameter space represents the intersection point of all collinear
points (xi, yi) in the image points. This is also true the other way around (Duda and Hart
1972); all points (ri, ui) satisfying the equation
ri ¼ x cos ðuiÞ þ y sin ðuiÞ ð11Þ
represent lines intersecting at point (x, y). If the line detection done by emphasising the
lines supporting the estimated vanishing point direction has succeeded, all lines found
intersect at the correct vanishing point and therefore the point is found using a least-
squares estimation technique. The user attitude is found from the vanishing point using
again Equation (9).
Journal of Location Based Services 215
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
Two parameters selected for the calculation are crucial for the performance of the
visual gyroscope, namely the threshold for deciding when a line is found and the standard
deviation (SD) of the estimated vanishing point value. When the threshold for finding a
line is deficient, the rate of false positives is large, and when it is too large, the computation
time increases and occasionally too few lines are found from the low-light indoor
environment resulting in an inaccurate vanishing point location. Also, when the SD
assigned for the estimated vanishing point value is too large the errors in IMU-induced
attitude distort the line detection by emphasising points close to the estimated point
probably not even belonging to a line. Results shown herein were obtained using an SD
value of 20 and a threshold of 0.4. In this configuration, the number of points needed is 20,
when they all fully agree with the estimated vanishing point. When the estimated
vanishing point and the image points potentially representing a line are not in such full
agreement, the number of required image points is larger. As a conclusion of numerous
experiments conducted this is the optimal parameter selection for the algorithm. When the
threshold is increased, the execution time grows significantly and the possibility of
extracting too few lines for reliable vanishing point computation hinders the method.
Similarly, when the threshold is decreased, the computation time is reduced, but as the
number of points required to support a line is also decreased, the number of false positives
in the line extraction deteriorates the solution.
Figure 2 shows the result of line detection and vanishing point calculations. The blue
lines are extracted using the Probabilistic Hough Transform, the green point is the
vanishing point estimation based on the IMU attitude and the red point is the vanishing
point corrected using the lines found. The figure shows how the vanishing point is found
reliably even when the IMU-induced attitude and therefore the estimated vanishing point
is distorted.
The IMU and visual attitude measurements calculated from the corrected vanishing
point are integrated using a Kalman filter propagating the attitude. When the algorithm
fails to find any lines, the estimated vanishing point is used for observing the attitude. The
objective of the research was that eventually the vanishing point tracking will be accurate
Figure 2. Estimated vanishing point location (green) corrected (red) using lines extracted with theProbabilistic Hough Transform.
L. Ruotsalainen et al.216
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
enough for also determining the magnitude of a turn and therefore the re-initialisation
phase can be omitted in further developments.
5. Experiments
In (Ruotsalainen et al. 2012b) the visual attitude was calculated using only the vanishing
point location and integrated as visual update with INS measurements using the 21-state
EKF as described previously. The user position was initialised with GNSS measurements
and during the navigation the heading was re-initialised after every steep turn using a floor
plan. The test equipment consisted of Analog Devices ADIS16488 IMU (Analog Devices
2010) and a GoPro Hero helmet camera (GoPro 2012). Images were captured using a
10Hz rate and IMU measurements with 20Hz rate. A NovAtel SPAN-SE GPS/
GLONASS receiver with a Northrop Grumman’s tactical grade LCI IMU was used as a
reference system. All equipment was attached to a backpack single as shown in Figure 3
and carried by a pedestrian.
During the 48-minute experiment conducted mainly indoors the root mean square
(rms) of the heading error decreased from 29.5 degrees when only IMU measurements
Figure 3. Test equipment set up.
Journal of Location Based Services 217
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
were used to 2.1 degrees with visual updates. When navigating using an IMU-only
solution the heading error was 90 arc minutes at the end of the experiment. Considering
that the micro-electro-mechanical systems (MEMS) IMU uses Analog Devices’ best gyro,
the fact that GNSS is not available at all after initialisation and the solution does use a
magnetic compass, this heading error is comparable with other state-of-the-art solutions.
The difference between IMU-only and vision-aided attitude accuracy was not as
remarkable for roll and pitch errors, namely the rms of the errors decreased from 2 to 1.4
degrees for roll and from 1.7 to 1.4 degrees for pitch. Figure 4 shows all the attitude errors
when using IMU-only (blue) and vision-aided IMU (green). However, without re-
initialising the heading after sharp turns the vanishing point-induced heading would be
incorrect and eventually distort the user attitude solution. All calculations were done in
post-processing as the computation time of the visual measurements was too long for a
real-time solution. Therefore, there is a need for more tolerant vanishing point detection
running in real time.
To evaluate the concept presented in this paper for accelerated vanishing point
detection, a subset of the collected data was re-processed using the Probabilistic Hough
Transform-based visual gyroscope providing reduced processing time. The data-set
consisted of 80 seconds of data and therefore 800 images were examined. As stated in
Matas, Galambos, and Kittler (2000), the processing time of an algorithm is dependent on
the computer used and the implementation details. Therefore, the effect of the algorithm is
shown by comparing the number of image points examined, in other words iterations of
the parameter calculation. The SHT examines all pixels in the input image and afterwards
searches for local maxima from the accumulator to find the lines. The algorithm presented
uses a fraction of the image points and already detects the lines during the point
examination. The algorithm presented uses on average 45% of all image points and
Figure 4. Attitude errors using no update (blue) and using vision-aided updates (green).
L. Ruotsalainen et al.218
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
therefore the computation is anticipated to be accelerated in the same proportion to the
SHT. Table 1 gives the test iteration statistics. Compared with the SHT, the number of
lines extracted is approximately equal for the algorithm proposed when the given
parameter values are used. The accuracy of the line extraction is difficult to compare, for
the lines found using the Probabilistic Hough Transform are seldom the same lines as
found using the SHT. This is because lines found using the former method are not usually
the lines receiving the maximum number of votes in the SHT, but the lines supporting the
estimated vanishing point. Therefore, the method should be employed only for extracting
lines for computing central vanishing points when IMU attitude measurements are used
and not as a generic line extraction algorithm.
The algorithm succeeds in finding the vanishing points in environments having
restricted light conditions or lines violating the orthogonality requirement. An example of
an image suffering from both situations is shown in Figure 5. The vanishing point
computed by the visual gyroscope using the SHT and voting is shown on left and using the
visual gyroscope with the Probabilistic Hough Transform presented in this paper is on the
right.
The method is also tolerant for large errors in the IMU measurements when the
parameters of the Probabilistic Hough Transform algorithm are carefully selected.
Figure 6 shows how an estimated vanishing point resulting from large temporary errors in
IMU measurements is corrected through the line detection presented. Note that the image
is not corrected from distortion and therefore the lines found do not exactly correspond to
the lines in the original image.
The method gives also promising results for future turn detection that has so far been
one of the most significant obstacles preventing the use of vision-aided inertial sensors
autonomously for navigation in unknown indoor environments. In turning situations the
estimated vanishing point obtained by propagating the attitude from the vision-aided
integrated solution falls outside the image at the same time as the real vanishing point
obtained from the Probabilistic Hough Transform detection is found from the other side of
the image as shown in Figure 7. When this contradiction is used in integration, at least the
existence of a steep turn is observed. Observing the magnitude of the turn is a future
research objective.
6. Conclusions
A novel method was presented for observing the user attitude based on a visual gyroscope
estimating the vanishing point location and utilising the information for extracting lines
using the Probabilistic Hough Transform. Since the calculation used some a priori
information of the vanishing point location from an IMU, the method is more tolerant of
environmental irregularities than when the calculations are done using only visual
information. The search space for the lines can be restricted to contain only the lines going
in the direction of propagation using a probability distribution computed from the
Table 1. Algorithm iteration statistics.
Ratio of image points used compared with all image points (%)
Min. Mean Max. SD
27 45 67 8
Journal of Location Based Services 219
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
estimation of the vanishing point location. Therefore, the Probabilistic Hough Transform
method presented should be employed only for extracting lines for computing central
vanishing points when IMU attitude measurements are used and not as a generic line
extraction algorithm.
When the view to the straight lines is blocked for a long time, like when navigating in
crowded open spaces or in environments with a majority of lines violating the
orthogonality requirements set for the vanishing point-based solution, the performance of
the method based on the SHT is equal to the IMU-only navigation solution. All of the
visual heading measurements are discarded by the error detection algorithm when
the LDOP derived from the lines found express the unsuitability of the environment for the
vanishing point calculation. Also, the solution based on the Probabilistic Hough
Transform discards a majority of the vanishing point measurements because in most
situations the irregular lines arising from the dynamic and non-orthogonal objects do not
intersect in an unambiguous point that could be assessed as a central vanishing point.
Therefore, if the environment is mainly unsuitable for the vanishing point calculation the
worst-case scenario is that the performance is equal to the IMU-only solution. However,
previous experiments show that the method is also feasible in an open space and that the
crowds rarely prevent vanishing point calculations (Ruotsalainen et al. 2012a).
Future work includes integration of the attitude measurements obtained using the
novel vanishing point detection method presented herein. The results obtained using the
Figure 5. Evaluation of vanishing point detection in an environment suffering from low lighting andnon-orthogonal lines.
Figure 6. Detected vanishing point (red) may be used to correct large errors in IMU measurementsresulting in erroneous estimated vanishing point location (green).
L. Ruotsalainen et al.220
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
previously developed vision aiding method were shown to provide significantly increased
accuracy for pedestrian navigation in challenging indoor environments. The method
however suffered from slow computation and an inability to detect sharp turns. The novel
method using the Probabilistic Hough Transform is expected to result in reduced
computation time and, in the future, in a real-time pedestrian visual-navigation solution
free from frequent re-initialisation from external information when integrated with INS
measurements.
Acknowledgements
LauraRuotsalainen has received aNokia Foundation Scholarship for the years 2011 and 2012 to supporther PhD thesis work. The authors would like to thank David Garrett, Summer student in the PLANGroup, for his contribution to the experiments. This researchwas performedwhenMs.Ruotsalainenwasa visiting student at the University of Calgary.
References
Alizahed-Shabdiz, F., and M. Heidari. 2009. Method of determining position in a hybrid positioningsystem using a dilution of precision metric. US Patent application 2011/0080317, filedOctober 2.
Analog Devices. 2010. “ADIS16375 Data Sheet (Rev. C).” pp. 28. http://www.analog.com/static/imported-files/data_sheets/ADIS16375.pdf
Borkowski, J. M., and M. J. Veth. 2010. “Passive Indoor Image-Aided Inertial Attitude EstimationUsing a Predictive Hough Transformation.” Proceedings of IEEE ION PLANS 2010, IndianWells, CA, pp. 295–302.
Collin, J. 2006. “Investigations of Self-Contained Sensors for Personal Navigation.” PhD diss.,Tampere University of Technology.
Duda, R. O., and P. E. Hart. 1972. “Use of the Hough Transformation to Detect Lines and Curves inPictures, Graphics and Image Processing.” Communications of ACM 15 (1): 11–15.
GoPro. 2012. Camera Specifications. Accessed June 15. http://www.gopro.com/cameras/hd-helmet-hero-camera/
Figure 7. Conflict between estimated and detected vanishing point locations indicates the existenceof a steep turn.
Journal of Location Based Services 221
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
Hartley, R., and A. Zissermann. 2003. Multiple View Geometry in Computer Vision. 2nd ed.Cambridge, UK: Cambridge University Press.
Hough, P. V. C. 1960. Method and means for recognizing complex patterns US patent 3069654, filedMarch 25.
Huttunen, V., and R. Piche. 2011. “A Monocular Camera Gyroscope.” Report 98, Department ofMathematics, Tampere University of Technology.
Kalviainen, H., P. Hirvonen, L. Xu, and E. Oja. 1995. “Probabilistic and Non-Probabilistic HoughTransforms: Overview and Comparisons.” Image and Vision Computing 13: 239–252.
Kiriyati, N., Y. Eldar, and A. M. Bruckstein. 1991. “A Probabilistic Hough Transform.” PatternRecognition 24 (4): 303–316.
Matas, J., C. Galambos, and J. Kittler. 2000. “Robust Detection of Lines Using ProgressiveProbabilistic Hough Transform.” Computer Vision and Image Understanding 78 (1): 119–137.
Prahl, D. M., and M. J. Veth. 2011. “Coupling Vanishing Point Tracking with Inertial Navigation toProduce Drift-Free Attitude Estimates in a Structured Environment.” Proceedings of ION GNSS2011, Portland OR, 3571–3581, Institute of Navigation, September 19–23.
Ruotsalainen, L., H. Kuusniemi, M. Z. H. Bhuiyan, L. Chen, and R. Chen. 2012a. “A TwoDimensional Pedestrian Navigation Solution Aided with a Visual Gyroscope and a VisualOdometer.” GPS Solutions. doi: 10.1007/s10291-012-0302-8.
Ruotsalainen L., J. Bancroft, and G. Lachapelle. 2012b. “Mitigation of Attitude and Gyro ErrorsThrough Vision Aiding.” Proceedings of IPIN 2012, Sydney, Australia, p. 9, IEEE Xplore,November 13–15.
Ruotsalainen, L., J. Bancroft, G. Lachapelle, H. Kuusniemi, and R. Chen. 2012c. “Effect of CameraCharacteristics of the Accuracy of a Visual Gyroscope for Indoor Pedestrian Navigation.”Proceedings of UPINLBS 2012, Helsinki, Finland, p. 8, IEEE Xplore, October 2–4.
Saarinen, J. 2009. “A Sensor-Based Personal Navigation System and Its Application forIncorporating Humans into a Human-Robot Team.” PhD diss., Helsinki University ofTechnology.
Titterton, D., and J. Weston. 2004. Strapdown Inertial Navigation Technology. 2nd ed. Stevenage,UK: IET.
L. Ruotsalainen et al.222
Dow
nloa
ded
by [
FU B
erlin
] at
06:
19 1
6 O
ctob
er 2
014
top related