application of appropriate technologies to 3d local … international symposium on automation and...
TRANSCRIPT
26th International Symposium on Automation and Robotics in Construction (ISARC 2009)
357
Application of Appropriate Technologies to 3D Local Terrain Modeling in Real-time for Intelligent Excavating System (IES)
Byung-In Yu1, Hyun-seok Yoo2 , Young-suk Kim3, Jong-won Seo4 and Seungwoo Han5
1Graduate Student, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-872-9757; FAX (82) 32-866-4624; e-mail: [email protected] 2Ph.D. Candidate, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-872-9757; FAX (82) 32-866-4624; e-mail: [email protected] 3Associate Professor, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-860-7593; FAX (82) 32-866-4624; e-mail: [email protected] 4Associate Professor, Department of Civil Engineering, Hanyang University, Seoul, 133-791, South Korea; PH (82) 2-2220-1482; FAX (82) 2-2220-4480; e-mail: [email protected] Professor, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-860-7595; FAX (82) 32-866-4624; e-mail: [email protected]
Abstract Automation in construction is considered as one of solutions to fundamental problems which the
construction industry has confronted. The development of real-time 3D local terrain modeling is the core methodology of achievement of IES (Intelligent Excavating System) which is a mega-size research concerning automation in construction. Stereo vision technique was selected as an appropriate method to achieve 3D terrain modeling in real-time. The objective of this study is to suggest a conceptual method by use of the stereo vision and assess feasibility of practical application under various conditions of construction sites. Bumblebee XB-3 is adopted as the stereo vision system in a prototype of a 3D terrain modeling system. This study is to investigate fundamental theories of the stereo vision system and to conduct tests in indoor and outdoor environments for determining appropriate detection range, finding installation location of cameras, doing visual assessment of the technical appropriateness, and checking reliability of 3D visual data. These decomposed tasks present valuable information for establishment of the conceptual method and improvement on the limitations of the use of stereo vision system for 3D terrain modeling in real-time. Keywords: automation, stereo vision, 3D, terrain model
1. Introduction
Many construction planners and site engineers have regarded enhancement of construction performances by high construction productivity and low unit costs as one of crucial factors for determining a success of construction projects. The shortage of experienced laborers, raise of labor costs, and debasement of construction qualities make construction industry in Korea face serious difficulties in achievement of high construction performances. Construction industry has tried to use emerging technologies that are capable of providing excellent construction performances. These efforts are especially witnessed in a field of construction equipment. A research project titled “Intelligent Excavating System (IES)” that is one of research projects supported by Korean government started with professionals in industries and researchers in academic institutes in Korea at 2007 is currently conducting (Ahn et al. 2008; Yu et al. 2008).
Stereo vision technique was selected as an appropriate method to achieve 3D terrain modeling in real-time in previous researches (Ahn et al. 2008; Yu et al. 2008). The objective of this study is to suggest a conceptual method by use of the stereo vision and assess feasibility of practical application under various conditions of construction sites. In this study, Bumblebee XB-3, a state-of-the-art stereo vision system, is adopted as the stereo vision system which is used in a prototype of a 3D terrain modeling system. In accordance with the goal of the research, this study is to investigate fundamental theories of the stereo vision system and findings by past researches related to this study. This study is to conduct tests in indoor and outdoor environments based on theoretical reviews. Indoor test allows us to determine an appropriate detection range by the stereo vision system and gives us basic information about installation location and
installation anof a conceptutechnical apprtest, another osystem with cpresent valuablimitations of
2. Literature
Stereo visacquisition, ocameras placi(Yu et al. 20estimate of thorizontal locmatching poinpresents distacan be highly(Woodfill et a
Identifying
where, f: foc b: bas
d: di dl: thdr: th
As describthe left image
ngle of cameraual method forropriateness ooutdoor test, pcomparison ofble informatio
f the use of ste
Reviews
sion techniquobject identifiing separately,008). The pictthe distances cations and dnt of the objeance data fromy effective foral. 2007; Cho ag distance dat
z: distancecal length seline isparity betwe
d = dl - dr he distance frohe distance frobed the equatie and the right
Figu
Aut
as on a cabin or practical appof the stereo vprovides quanf the result by on for establisereo vision sys
e is currentlyication and tr, that is, thosetures taken bto objects. T
different distanect, a correspom cameras to r detecting thand Park 1997ta from the cam
z
e from the cam
en the distanc
om the left imom the left imion (1), comput image would
ure 1. Diagram
tomation and Ro
3
of an excavatoplication. Outdvision system fntified differenactual measur
shment of the stem for 3D te
y used in a bracking systeme are placed oby two camerThese camerances. These twonding pixel, athe object. Qu
he object and 7; Ahn et al. 20mera to the o
dbfz ×=
mera to the ob
ces from the le
mage to the obmage to the ob
utation of disd be a technica
m of basic ope
obot Application
358
or. It would bedoor test enabfor practical apnces of distancrement. Theseconceptual merrain modelin
broad range ms. This techon different hras are mergeas detect imagwo detected iand then creatuick measuremsegmenting t008; Yu et al. bject is derive
: Equation
bject point
eft image and
ject point bject point sparity based oal issue in stere
eration by ster
ns
e primary infobles to do visupplication to cce data of 3D e decomposed
method and imng in real-time
of various fihnique is imp
horizontal leveed and produges of an ideimages are mtes one mergement in distanthe object and2008).
ed from figure
(1)
the right imag
on the correspeo vision tech
reo vision tech
ormation for eual assessmentconstruction simages by the
d tasks mentiomprovement on
e.
ields such as plemented byel but the samuce a virtuallyentical object
merged based oed image. Thisnce yields thatd gauging its
e 1 and the equ
ge
ponding pixel hnique.
hnique
establishment t of the sites. Accuracye stereo vision
oned previousln the
terrain mody two identic
me vertical levy instantaneou
with differenon an identics merged imagt this techniqusize and shap
uation as belo
in two image
y n ly
del cal vel us nt cal ge ue pe
ow.
es,
26th International Symposium on Automation and Robotics in Construction (ISARC 2009)
359
In addition to computation of the disparity, it would also be critical issue to find an appropriate corresponding pixel in an identical object with two images. Two methods have been introduced on the market: 1) feature-based method, and 2) area-based method to resolve this difficulty.
Primary conditions for achieving the objective of this study is to detect a terrain image in real-time without any interruption of equipment operation. One cycle of excavating is composed of “scooping out” and “unloading the earth”. It generally takes 10 seconds for completing one cycle according to several equipment guide books and site observation. This information notifies us that all procedure including detecting and image processing would be completed within 10 seconds. Detecting terrain image is another limited condition in this study. Detection by cameras would finish as fast as it can so that it would not give any interruption during scooping and unloading the earth on the tray of trucks. These two limitations described previously would be resolved by technical specifications of state-of-the-art stereo vision system. The emerging technique of stereo vision system provides the technical solutions in that interpreting visual data to 3D from distance data takes generally 1 ~ 2 seconds and detecting the image takes milliseconds.
3. Conceptual Method of Stereo Vision for 3D Terrain Modeling
Appropriate conceptual method of stereo vision system requires that detection range of cameras and horizontal excavation range would be primarily determined. Careful study on installation of stereo vision cameras follows. These requirements make this study investigate the detection range of cameras by indoor tests and the excavation range by construction equipment references. The investigation allows this study to determine the appropriate installation location of the stereo vision cameras.
3.1. Estimation of detection angle Available detection range of the stereo vision camera is delivered by an indoor test. According to the
technical specification of Bumblebee XB-3, available angles are vertical viewing angle of 49.5 degree and horizontal viewing angle of 66 degree. Indoor test is required so that appropriate viewing angle for use in excavation operation. The system created 3D images based on two images captured by cameras which were located in 2 meter distance from a screen. The size of projected area in screen was 2.28 meter in width and 1.75 meter height. This result and proportional computation informed that appropriate angles by the stereo vision system were vertical viewing angle of 47 degree and horizontal viewing angle of 59 degree.
3.2. Installation of the stereo vision system An excavator which is under development as a prototype shows 8.2 meter of maximum reach and 5.0
meter of maximum depth below ground. Previous study (Seo et al. 2008) indicated that appropriate horizontal excavating range was 3.4 ~ 7.67 meters when the specification data of the same excavator were delivered.
Location and mounted angle of the stereo vision system are determined by information derived from appropriate detection angle and horizontal excavating range delivered previously. On-site observation of excavating operation informed that the stereo vision system was installed on a cabin of the excavator to obtain effective detection with appropriate coverage of the excavated area. Specification of the excavator used in this study says that that height of prospective installation location on the cabin is 2.8 meter. Figure 1 illustrates basic simulation chart of the excavator for searching appropriate installation location of the stereo vision system.
Figure 2 indicates that the stereo vision system needs to be installed on the cabin with 2.8 meter height and 30 degree incline to cover 3.4 ~ 7.67 meters of appropriate horizontal excavating area.
4. Application Analysis by Field Tests
4.1. 3D terrain modeling There are a few of specific technologies such as Triclops SDK by Point Grey Research Inc., Open GL (Open Graphic Library) by Silicon Graphics Inc., and Microsoft Visual C++ compiler, version 6.0 by Microsoft Corp. which are utilized for 3D terrain modeling. Triclops SDK allows us to control the operation of stereo vision cameras and to transmit image data. It also includes conversion function from the image data to point cloud suing SAD (Sum of Absolute Difference) algorithm. Open GL makes the converted
point clou
functions by T
The imageon the image which is compimages, and aThe conceptuterrain image after matchin
4.2. Field tests Many field
vision systemsites are detec
ud by TriclopsTriclops SDK
Figu
es detected bycaptured by thposed with 3D
a depth map illual method of on the triggerg then is auto
d tests on varim for 3D terrai
cted in field te
Aut
SDK be illusK and Open G
Figure
ure 3. Diagram
y stereo vision he camera on
D terrain modlustrated as a
f stereo vision r signal by excmatically pres
ous constructn modeling. V
ests.
tomation and Ro
3
trated in the uGL.
2. Operation
m of functions
cameras showthe right loca
del display, 2Dgrey scale indsystem condu
cavating systemsented on the
tion sites are cVarious shapes
obot Application
360
user interface.
chart of the e
s by Triclops S
w 640× 480 reation. Figure 4
D stereo imageex which is coucted by this sm automatical3D modeling
conducted to as of terrain an
ns
Figure 3 pres
excavator
SDK and Ope
esolutions in c4 shows a 3D e map created onverted fromstudy enables lly or by manudisplay viewe
assess the pracnd objects whi
sents basic dia
en GL
color and are mmodeling dispby matching t
m the computethe camera to
ually. The merer.
ctical applicatiich may exist i
agram of
merged based play viewer two captured ed distance dato detect the rged image
ion of stereo in constructio
ta.
n
The stehorizon. Tfield tests.developedconceptua
Figuresexisting inFigures 4 aexcavating
Profesrange of ocorrespon
Figure 5
26th Internatio
ereo vision syThe images is d Basically, all
d in this study.al method of ss 5 and 6 show
n constructionand 5 are asseg works. sionals agreed
operation tolernding pixels in
. 3D terrain im
onal Symposium
F
stem is moundetected with captured imag. Consequentlystereo vision sw 3D terrain i sites, respecti
essed and dete
d that most 3Drance of excavterrains while
mages of vario
m on Automati
Figure 4. 3D m
ted on a tripo640× 480 res
ges are convery, assessment ystem feasibleimages of variively. All 3D i
ermined wheth
D images showvation. This fae detecting var
ous shapes of
ion and Roboti
361
modeling displ
od of 2.8 metesolutions in corted to 3D imaof noise cond
e or improvedious shapes ofimages converher these are s
wed appropriaavorable test rrious patterns
terrain Figu
ics in Construc
lay viewer
r height and 3olor. Total 98 age via a speciditions and prd further. f terrain and wrted from the sufficiently app
ate accuracy wesult is due to in terrain.
ure 6. 3D terra
ction (ISARC 2
30 degree incliimages are capific display sys
rocessing time
with various tycaptured imagpropriate to b
with considerato easiness of fi
ain images wit
009)
ine from the ptured by totastem which w
e makes the
ypes of objectsges shown in
be applied in
tion of a broadinding
th various obje
al 5 was
s
d
ects
Unlike mo
determined nwould be resoshows the 3D
Visual asse
images createConsequentlybe compared
Figure 8 sh
As shown
meter height data by stereocamera to the
Comparisoindicates 0.05deviations anagraphs, table
As shown deviations of vision systemfrom the cam
ost 3D imagesot to be used olved by contr
D images of ter
Figure 7
essment showd by stereo vi
y, the accuracywith those byhows an accur
in figure 8, thand 30 degree
o vision systeme target is meaon of total 14
51 meter of analyzed in the 41 presents anain figure 9 anthe distance d
m has limitationmera due to dif
Aut
s, the images thdue to poor qrolling camerarrain with con
. 3D terrain im
wn in figures 4ision system ary test requires y actual measuracy test cond
Figure 8.
he accuracy tee incline detecm are compareasured manuall8 samples of t
n average devia4th and the 5th alysis results ond table 1, it ndata between tn for practicalfficulty in find
tomation and Ro
3
hat concretedquality caused a’s shutter spencrete hume p
mages with co
4, 5, and 6 showre perfectly mthat the distan
urement. ducted in cons
Accuracy test
st is conductects targets whied with those ly. the distance dation and 0.16field tests we
of deviation bynotes that morthe 3D imagesl application toding the corres
obot Application
362
d hume pipes aby not findin
eed and creatinpipes and surfa
oncrete hume
w proper resumatched with a
nce data in 3D
struction sites.
t of stereo vis
ed that the sterch placed randby actual surv
data between 368 meter of a mere illustrated iy distance fromre distance fros and actual mo detecting tersponding pixe
ns
and surface wag corresponding noise remoace water in p
pipes and surf
ult in accuracyactual terrain inD images by st
.
ion system
reo vision systdomly within veying in that
3D images andmaximum devin figure 8. In m the camera
om the camerameasurement. Irrain where is l.
ater were capting pixels. Thi
oval algorithmoor quality
face water
y. It doubts whnformation ortereo vision sy
tem which is l2 ~ 11 metersthe distance f
d actual measuviation. Graphaddition to dand the targe
a to the target It explains thalocated with l
tured in are is limitation . Figure 7
hether 3D r not. ystem need to
located in 2.8 s. The distancfrom the
urements hs where eviation
et. has bigger
at the stereo long distance
o
ce
Di
5. Conclu
This stmodel in eis normallyapplicationchallengesexperimenassessed tefollows:
• Indexcand
• As in 3whimoshometmo
• In tto vandwelon exp
The prexcavator further stu
26th Internatio
stance range
3 ~5 5 ~7 7 ~9 9 ~11
usions and Fu
tudy introduceexcavation opy limited to stn of stereo viss which are totnts using stereechnical chara
door experimeavator. Exper
d 30 degree incresult of seve
3D images. A ite hole is witnre advanced
owed that averter. These devre. this study, thevisual assessmd actual measull-designed exvibration by a
perimented. rimary findingbut also a doz
udies for achie
onal Symposium
Figure 9. D
Table 1. CMaximudeviatio
0.0830.1360.1580.168
uture Works
ed the conceperation. The stationary objecsion system totally different
eo vision systeacteristics inclu
ent conductedrimental resultcline for acquieral field tests few of imagenessed due toalgorithm forrage deviationviations were g
e conducted tement of 3D imurement. Althperiments neean excavator a
gs and conceptzer, a loader aeving accurate
m on Automati
Deviation graph
Comparison rum ons d3 6 8 8
tual method ostereo vision scts in outdoor
o constructionwith past resem in construcuding limitatio
d to find apprts inform thatiring availablein constructio
es detecting coo mismatchingr noise removn of distance dgradually bigg
ests of accuramages and comhough these ted to be contand reliability
tual method suand other conse 3D terrain m
ion and Roboti
363
hs in the 4th a
esult of deviatAverage
deviations 0.034 0.050 0.068 0.065
of stereo visiosystem has beer environment
n operation forearches. Accoction sites in oons. These exp
ropriate installt the camera oe detecting areon sites, mostoncrete hume g two images. val. The accurdata between 3ger while the d
cy of 3D imagmparison assetests providedtinuously deveand accuracy
uggested in thstruction equip
model would b
ics in Construc
nd the 5th field
tions (unit: meStandardeviatio
0.02130.03360.04310.0484
n system to ben deployed ints or steady-str creating 3D rdingly, this st
order to achievperimental res
lation of stereon the excavateas. t targeted terrpipes and surIt, however, wracy test of t3D images anddistance from
ges by stereo essment of disd basic guideleloped. Accur
y of the acquir
his study woulpment. Basede conducted i
ction (ISARC 2
d tests
eter) rd on
V
3 6 1 4
e applied to cn broad rangetate objects in terrain modeltudy conducteve appropriatesults suggest c
eo vision systetor is located i
rain is well derface water hawould be ovethe distance dd actual measuthe camera to
vision systemstance data beline to the coracy and stabired 3D images
ld be applied td on findings inin IES researc
009)
Variance
0.0005 0.0011 0.0019 0.0023
reate 3D terras of area wherindoor. The
l would face med many e applications conclusions as
em on cabin oin 2.8 meter h
etected and creave a problemercome with udata of 3D imurement was 0
o the target be
m were limitedetween 3D imonceptual metility of this sys would be fu
to not only ann this study, h project.
ain re it
many
and s
of an height
eated m that use of mages 0.051 came
d only mages thod, ystem urther
n
Automation and Robot Applications
364
Acknowledgement
This research was supported by a grant from the research titled “Development of Intelligent Excavating System”, a part of Construction Technology Innovation Program(CTIP) by Ministry of Land, Transportation and Maritime Affairs(MLTM) of Korean government.
References
[1] Ahn, J. S., Yu, B. I., Kim, Y. S., and Han, S. W. (2008). “Emerging Technologies for Real-time 3D Terrain Modeling and Automatic Soil Volume Computation in Earthmoving Operation” Proc., 12th ICCCBE &2008 ICITC, 276.
[2] Cho I. S., and Park J. G. (1997). “Extraction of Depth Information using Correspondence Analysis in Stereo Vision.” Basic Science and Engineering, 1(2), 899-905.
[3] Lu, M., Chen, W., shen, X., Lam, H., and Liu, J. (2007) “Positioning and tracking construction vehicles in highly dense urban area and building construction sites.” Automation in Construction, 16(5), 647-656
[4] Woodfill, J. I., Buck, R., Jurasek, D., Gordon, G., Brown, T. (2007). “3D Vision: Developing an Embeded Stereo Vision System.” IEEE Computer.
[5] Yu, B. I., Ahn, J. S., Lee, J. H., Yu, H. S., Kim, Y. S., and Han, S. W. (2008). “Implementation of Real-time 3D Terrain Modeling in Automatic Earthmoving Operations.” Proc., 11th Symposium on Construction Robotics in Japan, 75-80.