Automatic in-line inspection of shape based on photogrammetry

Per Bergstrom1, Michael Fergusson2, Patrik Folkesson2, Anna Runnemalm3, Mattias Ottosson3, Alf Andersson4, and Mikael Sjodahl1

1Lulea University of Technology, Department of Engineering Sciences and Mathematics, Lulea, Sweden2Xtura AB, Kungsbacka, Sweden

3University West, Department of Engineering Sciences, Trollhattan, Sweden4Chalmers University of Technology, Department of Product and Production Development, Gothenburg, Sweden

per.bergstrom@ltu.se

AbstractWe are describing a fully automatic in-line shape inspection system for controlling the shape of mov-ing objects on a conveyor belt. The shapes of the objects are measured using a full-field optical shapemeasurement method based on photogrammetry. The photogrammetry system consists of four cameras,a flash, and a triggering device. When an object to be measured arrives at a given position relative tothe system, the flash and cameras are synchronously triggered to capture images of the moving object.From the captured images a point-cloud representing the measured shape is created. The point-cloud isthen aligned to a CAD-model, which defines the nominal shape of the measured object, using a best-fitmethod and a feature-based alignment method. Deviations between the point-cloud and the CAD-modelare computed giving the output of the inspection process. The computational time to create a point-cloudfrom the captured images is about 30 seconds and the computational time for the comparison with theCAD-model is about ten milliseconds. We report on recent progress with the shape inspection system.Keywords: Automatic, full-field, in-line, photogrammetry, shape inspection, single-shot

1. Introduction

Verification of geometric outcome is a generic problem in manymanufacturing processes, and a problem that has proven to bedifficult to automate. The main reasons for this are that non-contact methods are required that don’t interfere with the ex-isting manufacturing process, that measurements in many sit-uations should be performed on freely moving objects in a dis-turbed environment and that the level of accuracy should bein the order of a few tens of micrometer. Ideally the evalu-ation time should also be short enough to prevent erroneousparts to propagate through the manufacturing system. In thispaper we report on recent progress with a system based on pho-togrammetry and automatic evaluation of the measured shapein comparison to the nominal shape defined by a CAD-model.The system was developed as part of Production 2030 projectfunded by Vinnova, led by Lulea University of Technology,titled SIVPRO (Shape Inspection by Vision in Production).The authors of this paper have all contributed to the SIVPROproject. The photogrammetric rig was designed and devel-oped by Xtura AB, as well as the automated workflow up tothe output of the point-cloud. The alignment, comparison toCAD, and analysis portion of the system was developed byPer Bergstrom. The integration of the two halves of the sys-tem was engineered by Per Bergstrom and Michael Fergusson.The inspection system operates as follows.

1. Shape measurements of moving and arbitrarilyoriented objects.

2. Alignment of the representation of the measured shapeto a CAD-model.

3. Comparison to the CAD-model.A schematic illustration of the repeated inspection process isshown in fig. 1. The measured objects are made of stampedmetal sheets and are moving on a conveyor belt at about 1 m/s.

Shape inspection of manufactured objects is a frequentlydiscussed subject in many scientific journals, see e.g [1–9].What they all have in common is the requirement that the mea-sured objects are non-moving, which is not possible to imple-ment in many manufacturing processes. This lack forms thebasis of the content in this paper. Otherwise, their inspectionprocess is quite similar to our, which is measurement, align-ment, and comparison to a nominal shape. In [1], a time-of flight of a laser pulse as a measurement method is used.However, the precision using time-of flight is far too low fora qualitative shape inspection of stamped metal sheets in vehi-cle manufacturing. More recent papers discuss shape measure-ment methods based on structured light techniques, [4–8]. Pro-jected patterns are illuminating the object to be measured andcameras from different views capture a sequence of imagesof the illuminated object. The captured patterns are analyzedwherefrom a 3D shape is extracted. This method is quite simi-lar to the photogrammetry method that we are using but cannotbe used for moving objects, and requires a highly accurate cal-ibration to operate correctly. In photogrammetry the naturalpattern of the object, i.e. the surface structure, scratches, fea-tures, etc., are used to determine 3D shape. It enables higherresolution shape representations but it also requires that a nat-ural pattern exists.

The output from the measurement is a set of points, i.e. apoint-cloud, representing the measured shape. Since the mea-sured objects are arbitrarily oriented the point-cloud must firstbe aligned to the CAD-model before deviations can be found,see e.g. [10]. Therefore, a registration problem must be solved[11–15]. The same CAD-model is used for all measured ob-jects so in order to speed up the repeated alignments we pre-process the CAD-model as described in [16]. The backgroundis measured as well, which, so as not to affect the compari-son, we need to filter out from the point-cloud, see Section 4.

MAC MAC MAC MAC MACMACMACMACMAC

CAD-model

Measurement:Alignment:Comparison:

PhotogrammetryBest-fit & feature-based alignmentCompute deviations

Fig. 1: Schematic illustration of the repeated inspection process. The same CAD-model is used as a nominal shape referencemodel for all measured objects.

Measurement Point-cloud & CAD-model Alignment Comparison

Fig. 2: Illustration of the process for each measured part. 1) Measurement using photogrammetry. The system consists ofcameras, a flash, and a triggering device. 2) CAD-model and resulting point-cloud in the same coordinate system.

3) Alignment of point-cloud to the CAD-model. 4) Comparison with the CAD-model. Computing deviations.

Single-capture full-field optical shape measurements with shortexposure enables shape measurements on moving objects. Thedisadvantage is that such methods are prone to have errors re-sulting in outliers in the point-cloud. The robust surface reg-istration method described in [17] is used to align the point-cloud to the CAD-model. Outliers in the data are then effi-ciently rejected.

Schematic illustrations of the measurement, alignment, andcomparison are shown in fig. 2. In its present form our mea-surement system consists of four high-resolution colour cam-eras that are synchronously triggered together with a flash byan external LiDAR-based sensor. The total image acquisitiontime is then limited to less than one millisecond that enablesmeasurements on freely moving objects, for example objectsmoving on a conveyor belt. After application of a few ini-tial filters the point-cloud is automatically aligned to its CAD-model and deviations in predefined comparison points are cal-culated. The verification consists of several steps that all con-tribute to the over-all performance of the system: calibration,registration of captured images, transformation of the obtainedpoint-cloud, and calculation of deviations, which all will bebriefly described.

2. History of photogrammetry

Photogrammetry originated during the early years of photog-raphy through the advent of stereo-photography. Essentiallyreplicating the parallax-based depth-perception of humanvision, stereo-photography began as an art, but quickly wasadapted by mathematicians to measure at long distances. Es-pecially useful in geographical mapping by way of aerial pho-tographs, it began to be used underwater in the 1940s and1950s for measuring shipwrecks. Commonly used in earth sci-

ences like mining, geology and geography, it began to be usedin heritage and archaeology.

Film-based Photogrammetry was very complicated, andgradually fell out of use in the late 80s and early 90s whenlasers began to take over as the go-to measurement techniquefor most applications. When digital cameras began to takeover from film, people began to take photogrammetry seri-ously again. Target-based photogrammetry was the most com-mon due to the low resolution of early digital cameras throughthe mid-2000s. Due to highly accurate centroiding algorithms,targets could be measured to 1/100th of a pixel. Dependingon the resolution, these targets could be positioned with sub-millimetric accuracy. These algorithms are still used today be-cause of their high accuracy, and coupled with the ultra-highresolution cameras of the past couple of years, measurementprecision of microns can be achieved [18–21].

The world of computer-vision began to explore photogram-metric techniques with the advent of digital cameras. An en-tirely different field of study, computer vision researchers be-gan to re-invent the photogrammetric wheel [22]. Despite be-ing so parallel to photogrammetry, computer-vision researchersfailed to grasp the significance of half a century of photogram-metric research, if they even acknowledged its existence. Themain difference between the two disciplines is the focus on ac-curacy in photogrammetry and the focus on realistic modellingin computer vision. Due to this, we have reached an point intime where the complexity of computer-vision algorithms andaccuracy of photogrammetric algorithms are beginning to belinked [23–26]. With the introduction of structure from mo-tion (SfM) algorithms from the computer vision world, highlyaccurate photogrammetric techniques such as bundle adjust-ments can be applied to photosets aligned using SfM algo-rithms to achieve results that are impossible to achieve by any

other method. Using low resolution cameras (less than 2 mp),multi-view stereo (MVS) algorithms have been demonstratedto achieve 95 % coverage at 0.5 px accuracy of 10 cm ob-jects [27].

New digital photogrammetry techniques offer distinct ad-vantages over lasers: for each XYZ point, colour is known, andmodern MVS algorithms can also determine the surface nor-mal for a given point. Laser-line techniques are problematicwith edges that introduce beam-splitting, where photogram-metry offers excellent identification of edges because these areprecisely the points of contrast used by the stereo-matching al-gorithms. Laser-line techniques cannot generally capture colo-ur data, and the few systems that can have very slow capturespeeds. Laser-line techniques cannot capture accurate surfacenormal data, because if a surface is angled away from the lasersource, the receiver can only determine the intensity of thebeam based on the return, and cannot measure the directionthe laser beam has traveled after hitting the surface.

Other optical techniques use pattern-projection of variouscolours and wavelengths (white light or blue light) and stereocameras to detect these patterns on the surface of the object.They use essentially the same technique of photo matching andalignment, however a number of patterns need to be projected,so exposure time can be lengthy depending on the colour ofthe object being measured (white is faster, black is slower). Ingeneral the apparatus required for these “structured light tech-niques” - two cameras and a texture projector - requires carefulcalibration and can easily lose accuracy if any of the compo-nents go out of alignment. Even standard stereo-photogrammetry rigs with rigid stereo-pair chart-based cali-brations are at the mercy of camera movement [27]. We pro-pose that camera calibration should only be performed to de-termine scaling, and not the exterior orientation of the cameras.Instead, each successive scan will individually determine theexterior orientation for that set of images and perform a seriesof optimization steps including removal of points with highresiduals and bundle adjustments. In doing so, we would ef-fectively reduce the potential error of camera movement fromthe constant vibration caused by the sheetmetal press. The ef-fect of camera movement on scaling would be negligable at1 % or less.

3. Technical system outline

3.1 Triggering system

The variable aspect of the conveyor belts requires the use ofdual sensors for triggering the system, so that the speed canbe calculated precisely every time. However, the placement ofthese sensors must be exact in order to work properly. Initially,an IR transmitter and receiver were tested. They worked well,and a second pair could be used to determine the speed of theconveyor belt. However, in that case, four arms needed to beprecisely positioned on either side of the conveyor belt, lowenough to sense the part on the conveyor belt. This became aproblem for three reasons.

First, the scanning rig will be installed permanently on thefloor in front of the large sheet-metal stamp. When measuring,the sensors will have to be precisely positioned when preparingto produce the part. Having to align four different sensors willbecome quite tedious.

Second, the conveyor belt is 1.5 m wide, and the IR re-ceiver would need to be set to a high gain setting in order toreceive the transmitter signal at that distance. The sensitivity

Fig. 3: LiDAR trigger test.

of the receiver would allow it to be triggered by stray IR lightfrom welders and other light sources.

Third, when producing the parts, the stamp produces theslightly different left and right parts at the same time, so thatthere are two parts on the conveyor belt. The parts do not endup in the same lateral position on the conveyor belt, so whenmeasuring the right part, the left part may trigger the system.Placement of the transmitters in the centre of the conveyor beltis also not possible, since the system would also be triggeredif the transmitters are bumped by a stray part, and would needto be reset.

Instead, we decided to test a LiDAR based trigger. It canbe used independently, acting as its own transmitter and re-ceiver. It has a range-based feature, that potentially allows itto be positioned directly above the imaging area looking downon the conveyor belt. However, this feature has a 3 % marginof error. At a distance of 1 m (the height of the scanning rig),this means the margin is 3 cm. This is insufficient accuracy forsuch a thin part.

For this reason, the sensor will be placed on one end ofthe conveyor belt and aimed horizontally across the belt. Therange feature will be used to ignore the part on the oppositeside of the conveyor belt, see fig. 3.

3.2 Camera system

Four 36 megapixel Nikon D800 cameras are arrayed above theimaging area. Each camera has a 35 mm lens, which at a heightof 60 cm covers an area of 60 cm by 40 cm. The base, thedistance between each camera is about 20 cm. The base todistance ratio of these cameras is about 1:3, which gives a goodbalance of accuracy and coverage for such a contoured surface,see Subsection 3.6

The system is designed to work with small parts up to25 cm by 25 cm at a thickness no greater than 15 cm. Thisis due to the depth of field limitations of the optical lensesused. The aperture setting chosen through testing was f/9,which gives a balance between depth of field requirements anddiffraction limitations. The depth of field at a focus distance of60 cm is 5 cm, but the images are already diffraction limitedat f/9, with a minimum circle of confusion of 2.5 pixels. Us-ing a smaller aperture will increase the depth of field, but alsoincrease minimum circle of confusion which will effectivelyreduce the image resolution and sharpness. A shutter speed of1/200 s must be used in order to sync the strobe to the shut-ters. An ISO setting of 100 has been chosen for a high signalto noise ratio.

Fig. 4: Set-up and first test at production site.

The two pairs cameras are mounted to two separate barsthat run across the width of the rig. The bars are adjustable inheight and width, while the cameras themselves can be posi-tioned anywhere along the bar. There is a cross-bar that addsstability, and where a fifth camera could be mounted if desired,see fig. 4.

3.3 Lighting system

The lighting is important for two reasons: one, to freeze themovement of the part on the conveyor belt, and two, to controlreflections on the sheet-metal surface. In addition to the metalsurface, the oil used on the tool and dye stamp increases thespecularity of the object, causing more issues than were en-countered during testing with a dry part. The interior of thescanning rig is white so that the lighting is as even as possi-ble. A single strobe is being used here due to expense and sizelimitations.

A low t.1 time is important, which represents the lengthof time it takes the strobe to release 90 % of its light. Moststandard strobes do not specify this value, but it is far morevaluable than the t.5 time that is typically specified, which rep-resents the length of time it takes the strobe to release 50 % ofits light. The strobe we have chosen has a specified t.1 time of1/1600 s at full power, and 1/20000 s at low power. The powersetting required for the system is about 25 %, which will giveus a flash duration of around 1/10000 s, or 0.1 millisecond.This will effectively freeze any movement of the objects beingmeasured.

The strobe is situated on the far side of the scanning rig,centered vertically, and aimed across to the opposite top cornerof the rig. There is a wide reflector so that the strobe light doesnot directly hit the object on the conveyor belt, see fig. 4.

Fig. 5: Testing calibration with target pyramid.

3.4 Calibration, scaling and orientation

When discussing photogrammetry, calibration is one of themost important steps. While the exterior orientation solves theposition of the cameras in the real world, the interior orienta-tion parametrizes the relationship between the camera sensorand the lens. It solves the position of the lens relative to thecentre of the sensor, as well as various distortion types includ-ing radial distortion, decentering distortion, and image flat-ness. While there has been much work on automating the in-terior orientation in-line through SfM algorithms, it is possibleto perform this step ahead-of-time to speed up processing andguarantee accuracy [18–20]. While target-based camera cal-ibration is quite accurate, feature-based matching algorithmscan achieve better coverage of the field of view of a givencamera, because a textured surface will fill the frame, whileonly so many targets can fit within a single image. While indi-vidual point accuracies may not be as high, 0.1-0.3 px against0.01-0.1 px, the sheer number of points compensates for this[24–27]. The interior orientation parameters are saved as XML(Extensible Markup Language) files and stored in a directorythat is directly referenced and accessible to the python scriptthat runs the processing chain. The interior orientation is per-formed after the camera settings have been chosen, since bothfocus and aperture settings can influence the parameters. Eachcamera, cameras are labelled A through D, has its own xmlcalibration file, which can be updated if a camera needs to berefocused, or replaced.

The resulting point-cloud is not within a fixed coordinatesystem. In order to create a fixed coordinate system, we needfixed targets floating above the conveyor belt on cross-bars,or coordinates of the camera positions. The proximity to thestamp would invalidate any calibration quite quickly, espe-cially between four cameras mounted on separate bars.

Calibrationimages

Interiororientation

Images

High-residualpoint removal

Exteriororientation

Bundleadjustment

Scalingreferences

Bundleadjustment

Densereconstruction

Point-cloud

Fig. 6: Flowchart of image processing.

Instead, we have calibrated each pair of cameras on theirrespective bars as a “scale bar”, since a measurement betweentwo cameras on the same bar has less of a chance to varythrough vibration than coordinates of four cameras mountedon separate bars. Insulated mounts should also reduce thechance of movement from the constant slamming of the sheet-metal press. The scaling of the point-cloud is an importantstep, so we use a calibrated pyramid with coded targets cal-ibrated to less than 10 microns to determine the distance be-tween each pair of cameras, in fig. 5. Since scanning the im-ages for coded targets and calibration is time-consuming, as isdetermining the interior orientation, these steps are performedin advance to speed up the workflow. This is shown in fig. 6.

The exact orientation of the point-cloud in a known coor-dinate system is not required. It is enough that the orientationis consistent. Since the coordinate system is based on the firstcamera, Camera A in this case, we know that it will be con-sistent. The reason for consistency is so that the analysis isperformed correctly whether or not the part is right-side-up orupside-down on the conveyor belt. Since we can only measurea single side of the part, and the sheetmetal has a thickness, itis important that the deviation is calculated based on the CADsurface, rather than the solid (with thickness).

The correct surface (top or bottom) for measurement is de-termined by the best-fit; if the orientation of the point-cloud isinverted during the fitting, then we know that the point-cloudis upside-down. This is why consistency of the coordinate sys-tem is important.

3.5 Processing

The 36 megapixel images allow for fine surface detail to becaptured on the sheetmetal, as many micro-scratches and otherimperfections are created during the stamping process in addi-tion to the grain of the metal. There are a number of steps inthe processing chain, beginning with the pre-calibration, out-lined in fig. 6. Feature detection is the first step, when edgesand colours are used to create points of interest in the images.These are to be used as bundle points, used to determine the

d

b

p p

f f

Fig. 7: Diagram of a stereo pair of cameras.

position of each camera relative to one another, the exteriororientation. This feature-based matching method is based onSfM algorithms and is quite efficient at this stage, avoiding theuse of targets.

The camera alignment is strengthened by a bundle ad-justments that increases accuracy through successive steps ofpoor-residual point removal. After this step a semi-globalmatching algorithm, multi-view stereo, is used to determinethe depth maps for each image based on the exterior orienta-tion [27–29]. Once these depth maps are created, points arecreated at a pixel level, and placed in the position determinedby the depth map and initial alignment.

3.6 Resolution and accuracy

Accuracy of image referencing can range between 0.5 and0.1 pixels. During testing, we found that the relative accuracywas consistently around 0.3 pixels [30]. We calculate the res-olution of the images using the formula

GP = (d/f)× p , (1)

where GP is ground pixel size, d is the distance to the object,f is the focal length and p is the pixel size in meters, see fig. 7.The pixel size p can be calculated for each camera by dividingthe width of the sensor by the width of the image in pixels.For a 36 mp sensor and a 35 mm lens at a height of 60 cm,we get GP = (600/35) × 0.00488 = 0.084 mm. So eachpixel represents 0.084 mm on the object. If we estimate imageaccuracy to be 0.3 pixels, multiplying this estimated accuracyby the ground pixel size will give us the planar accuracy of0.025 mm. Depth accuracy can be calculated with the formula

σ = (d/b)× α , (2)

where d is the flying height, b is the base and α is the planaraccuracy. The depth accuracy would thus be 0.061 mm. Itfollows from (2) that the depth accuracy is directly related tothe base-distance ratio.

In order to speed up the dense point-cloud generation stepof the processing chain, we have opted to reduce the resolutionto 1/4 resolution, which is the second level. In doing so, we in-crease our ground pixel size two-fold, bringing it to 0.168 mmand thus our planar accuracy to 0.05 mm and depth accuracyto 0.12 mm. The decrease in processing time is six-fold, fromaround 3 minutes to around 30 seconds. When measuring ahigh volume of parts at high frequency, around one part everytwo seconds, this increase in speed is very significant.

4. Background removal

When using full-field optical shape measurement methods itis inevitable that the background will be measured as well inaddition to the inspected object. Of course, that is undesiredsince the background will disturb the alignment to the CAD-model. To avoid this problem, filters must be used to removethe background from the point-cloud. A first idea one canhave is to remove points outside a region of given coordinates.However, when using photogrammetry the point-cloud will becreated from homologous points in the captured images. Theused algorithm starts to create the point-cloud from the ho-mologous points of best conformance. One and the same pointin the measured volume will have different coordinates in themeasurements. Therefore, a simple filter just based on geom-etry can not be used. The filters we are using are based on

• Colour

• A fitted plane to the flat background

• Luminance

If the conveyor belt was a solid colour, its colour couldsimply be removed and the problem with the background wouldbe solved. However, conveyor belts in industrial environmentare more or less worn where parts land, removing the origi-nal coating and creating a multi-coloured surface. In our case,where the conveyor belt is worn, it has a grey colour very sim-ilar to the colour of the measured objects. Hence, a filter basedon just colours can not be used. That is why a more sophis-ticated filter has been developed. That is by fitting a plane tothe colour of the non-worn region of the conveyor belt. Byremoving points close to and below the fitted plane the pointsin the point-cloud representing the conveyor belt will be re-moved. Luminance is also used to filter out points. Non-illuminated regions in the measured volume will not be accu-rately represented in the point-cloud since no matching pointswill be found in the captured images in these areas; for exam-ple, within holes and slots. That is the reason why we alsoremove points from the point-cloud of luminance value lessthan a given threshold.

5. Alignment

The measured objects on the conveyor belt are arbitrarily ori-ented and positioned. Therefore, the point-cloud must first bealigned to the CAD-model before a comparison can be doneso that the true deviations can be computed. The deviationsare defined from an alignment of three features and these are;the center point of a circular hole, the axis of a slot, and asurface point. Before these features can be identified in thepoint-cloud a best-fit alignment must be accomplished. Fromthe nominal features on the CAD-model and the best-fit align-ment it is roughly known where the measured features are. Thealignment process is given in fig. 8.

Best-fitalignment

Identifycorresponding

features inpoint-cloud

Feature basedalignment

Fig. 8: Automatic alignment process.

5.1 Best-fit

For finding a best-fit of the point-cloud to the CAD-model weneed to estimate a rigid body transformation, consisting of arotation matrix R and a translation vector t, such that ourpoint-cloud of N data points, {pi}Ni=1 , fits to the surface, S,of the CAD-model under the transformation. The rigid bodytransformation is obtained from the minimization problem

minR,t

N∑i=1

%(d(Rpi + t, S)) , (3)

where d(p, S) is the distance between an arbitrary point pand the surface S of the CAD-model. The method used tofind the closest point on the CAD-model is described in [16].The computations are pre-processed adapted for repeated usein registration problems. A search tree is created once and isused for all measured objects. In order to reduce the influenceof outliers, that is remaining background points and measure-ment errors, a robust criterion function % is used. Commonly,least squares minimization is utilized where %LS(r) = 1

2r2.

However, the Squared residual function do not result in robustestimations. The criterion function that we are using is Tukey’sbi-weight function

%Tu(r) =

κ2

Tu6

{1−

(1− r2

κ2Tu

)3} if |r| ≤ κTu ,

κ2Tu6

if |r| > κTu ,

where κTu is a scaling parameter. The graphs of the Squaredresidual function and Tukey’s bi-weight function are shown infig. 9. Outliers will have a very high influence in the alignmentprocess when using the Squared residual function.Tukey’s bi-weight function assigns less influence of deviatingpoints which gives a more reliable result when using data con-taining outliers.

The iterative closest point (ICP) algorithm, see [11], isan algorithm that can be used to solve the registration prob-lem (3). A variant of the ICP algorithm is developed for thespecific application of robust surface registration and is de-scribed in [17]. The ICP-algorithm converges to a local min-imum and to avoid ending up in a local minimum differentfrom the global one, we are testing several various initial po-sitions and orientations of the obtained point-cloud. A methodto determine orientations of the measured objects that is physi-cally possible is developed based on the geometry of the CAD-model. A number of different rotations are tested for each pos-sible orientation. The tested positions are based on the averagepoint of the point-cloud representing the measured shape andthe average point of the surface points S. A few iterations of

r

y

Squared residualTukey's bi-weight

Fig. 9: Two different criterion functions, % .

the ICP-algorithm are used for each one of the tested transfor-mations to roughly find a local alignment of the point-cloud.The transformation that gives the best fit is used as an initialtransformation in further iterations of the ICP-algorithm, andthat is to find an even better fit between the shapes.

5.2 Feature based

The best-fit alignment gives a fairly good matching betweenthe representation of the measured shape and the CAD-model.However, the deviations we want to find are defined from aspecific alignment of some features. These features and whichtransformation they defines are as follows.

1. Center point of circular hole

• Translation in three directions

2. Axis of slot

• Rotation in two directions

3. Point on surface

• Rotation in one direction

An alignment of these features results in the specific alignmentthat defines the true deviations we want to find.

The identification of the corresponding measured featuresin the point-cloud is done by starting from the best-fit align-ment and using data points near the nominal features on theCAD-model and from these points determine the correspond-ing measured features, i.e. the center point, the axis of the slot,and the specific surface point. The measured circular hole andslot are identified by fitting a corresponding nominal feature tothe point-cloud near the best-fit position. The measured sur-face point is found from a fitted plane to the data points nearthe nominal surface point. The thickness is compensated inthis identification depending on which side that is measured.From the best-fit transformation and the known orientation ofthe CAD-model it is determined which side that is measured,that is if the measured object is right-side-up or upside-down.

6. Comparison

The reason for doing the shape measurement and the alignmentto the CAD-model is to compare the measured shape with itsnominal shape, which is defined by the CAD-model. The de-viations are defined from the specific feature alignment. Foreach object there are a number of “comparison points”, whichare points of particular interest where deviations are to be de-termined. The comparison points that we are using are as fol-lows.

• Surface points

• Trimmed edge points

• Center point of circular holes

All comparison points are defined with position and direction.The deviation at a surface comparison point is computed

from the feature aligned point-cloud. A plane is fitted to thedata points near the nominal surface comparison point. Themeasured deviation is obtained from the intersection betweenthe fitted plane and the line given from the comparison pointand its defined direction. The defined direction is a nominalsurface unit normal. The deviation at a trimmed edge compar-ison point is obtained in a similar way. A plane is fitted to thedata points near the nominal comparison point. The “edge” in

Fig. 10: Developed application software for the SIVPROproject. The CAD-model and comparison points are shown tothe left. The deviations between the measured surface and the

CAD-model are shown to the right.

the point-cloud near this fitted plane is used to determine thedeviation in the defined direction of the trimmed edge compar-ison point. The deviation at the center comparison point of acircular hole is computed in a similar way as we did when wedetermined the center point of the circular hole in the featurebased alignment. The deviation at these comparison points arecomputed with high precision.

In addition to the deviation at the comparison point we arealso computing the deviation for all data points representingthe measured shape. The result of this comparison is shownas a deviation map with a colour scale. However, these devi-ations are not computed with the same high precision as thedeviations at the comparison points. That is to get the resultfaster.

Also in the comparison we compensate for the thicknessof the object. Depending on which side we measure, differentpairwise sets of comparison points, offset to each other, areused.

7. Implementation

The used shape analysis algorithms are implemented in a GUI(graphical user interface) Matlab-function which is compiledto a stand-alone application software, see fig. 10. The CAD-model, the features used in the feature based alignment markedin green, and the comparison points marked in blue and red areshown to the left. A color map of the deviations between themeasured surface and the CAD-model is shown to the right.

The input to the application software are setting files con-taining information about search paths to the point-cloud filesand the CAD-file, feature point data, position and direction ofcomparison points, format of output files, etc. A sequence ofcomparisons to the CAD-model is started by just a few mouseclicks, which can easily be done by an operator at a stampingmachine producing the manufactured objects.

The result of the shape comparison is presented in differ-ent ways. The application software produces a computer read-able ASCII file for further analysis. A pdf-report with a colormap of deviations and tables of deviations at the comparisonpoints is also generated. That is the output from the applicationsoftware.

All computational steps described in Section 4–6 are doneusing this software, including the pre-processing of the CAD-model which takes about two minutes and is done only oncefor each CAD-model. The output from the pre-processing is

a file with pre-processed data, which is loaded when a newsequence of comparisons with the CAD-model will be done.The program consists of functions in Matlab-code and func-tions in compiled C-code. The functions in compiled C-code,which is much faster than equivalent Matlab-code, are createdfor repeated intense computations that must be done for eachpoint-cloud obtained from the shape measurements.

The elapsed computational time for the repeated analysisare as follows.

• 1 second to read a point-cloud file andbackground removal.

• 0.01 second for alignment to the CAD-model andcomputing deviations.

That means the time for computing the alignment, i.e. the best-fit and the feature based, and time for computing the deviationscompared to the CAD-model are far less than the computa-tional time for all other processes.

8. Conclusions

We have seen that it is possible to measure the shape of mov-ing objects on a conveyor belt using photogrammetry and doan automatic shape comparison with a nominal shape definedby a CAD-model. The current bottleneck is the computationaltime to create a point-cloud from the captured images whichtakes about 30 seconds depending on the shape of the mea-sured object, precision requirements, and which computer thatis used. Time for doing the analysis takes just about ten mil-liseconds.

References

[1] I. Moring, H. Ailisto, V. Koivunen, and R. Myllyla, “Ac-tive 3-d vision system for automatic model-based shapeinspection,” Optics and Lasers in Engineering, vol. 10,no. 3-4, pp. 149–160, jan 1989.

[2] Y. Li and P. Gu, “Free-form surface inspection tech-niques state of the art review,” Computer-Aided Design,vol. 36, no. 13, pp. 1395 – 1417, 2004.

[3] G. M. Brown, “Overview of three-dimensional shapemeasurement using optical methods,” Optical Engineer-ing, vol. 39, no. 1, p. 10, jan 2000.

[4] K. Wolf, D. Roller, and D. Schafer, “An approach tocomputer-aided quality control based on 3d coordinatemetrology,” Journal of Materials Processing Technology,vol. 107, no. 1, pp. 96–110, 2000.

[5] K. Wang and Q. Yu, “Accurate 3d object measurementand inspection using structured light systems,” in Pro-ceedings of the 12th International Conference on Com-puter Systems and Technologies - CompSysTech ’11.Association for Computing Machinery (ACM), 2011.

[6] J. Xu, N. Xi, C. Zhang, Q. Shi, and J. Gregory, “Real-time 3d shape inspection system of automotive partsbased on structured light pattern,” Optics & Laser Tech-nology, vol. 43, no. 1, pp. 1–8, feb 2011.

[7] J. Xu, N. Xi, C. Zhang, J. Zhao, B. Gao, and Q. Shi,“Rapid 3d surface profile measurement of industrial partsusing two-level structured light patterns,” Optics andLasers in Engineering, vol. 49, no. 7, pp. 907–914, jul2011.

[8] J. Xu, B. Gao, J. Han, J. Zhao, S. Liu, Q. Yi, Z. Zhao,H. Yin, and K. Chen, “Realtime 3d profile measurementby using the composite pattern based on the binary stripepattern,” Optics & Laser Technology, vol. 44, no. 3, pp.587–593, apr 2012.

[9] G. Tan, L. Zhang, S. Liu, and W. Zhang, “A fast and dif-ferentiated localization method for complex surfaces in-spection,” International Journal of Precision Engineer-ing and Manufacturing, vol. 16, no. 13, pp. 2631–2639,dec 2015.

[10] L. Zexiang, G. Jianbo, and C. Yunxian, “Geometric al-gorithms for workpiece localization,” IEEE Transactionson Robotics and Automation, vol. 14, no. 6, pp. 864–878,1998.

[11] P. J. Besl and N. D. McKay, “A method for registrationof 3-d shapes,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 14, no. 2, pp. 239–256, 1992.

[12] Y. Chen and G. Medioni, “Object modeling by registra-tion of multiple range images,” Image and Vision Com-puting, vol. 10, no. 3, pp. 145–155, 1992.

[13] Z. Zhang, “Iterative point matching for registration offree-form curves and surfaces,” International Journal ofComputer Vision, vol. 13, no. 2, pp. 119–152, 1994.

[14] H. Pottmann, Q.-X. Huang, Y.-L. Yang, and S.-M. Hu,“Geometry and convergence analysis of algorithms forregistration of 3d shapes,” Int. J. Comput. Vision, vol. 67,no. 3, pp. 277–296, May 2006.

[15] W. Li and P. Song, “A modified ICP algorithm based ondynamic adjustment factor for registration of point cloudand CAD model,” Pattern Recognition Letters, vol. 65,pp. 88–94, nov 2015.

[16] P. Bergstrom, O. Edlund, and I. Soderkvist, “Repeatedsurface registration for on-line use,” The InternationalJournal of Advanced Manufacturing Technology, vol. 54,pp. 677–689, 2011.

[17] P. Bergstrom and O. Edlund, “Robust registration of sur-faces using a refined iterative closest point algorithmwith a trust region approach,” Numerical Algorithms,July 2016.

[18] C. S. Fraser, “Automatic camera calibration in closerange photogrammetry,” Photogrammetric Engineering& Remote Sensing, vol. 79, no. 4, pp. 381–388, 2013.

[19] S. Cronk, C. Fraser, and H. Hanley, “Automated metriccalibration of colour digital cameras,” The photogram-metric record, vol. 21, no. 116, pp. 355–372, 2006.

[20] F. Remondino and C. Fraser, “Digital camera calibra-tion methods: considerations and comparisons,” Interna-tional Archives of Photogrammetry, Remote Sensing andSpatial Information Sciences, vol. 36, no. 5, pp. 266–272, 2006.

[21] C. S. Fraser, “Digital camera self-calibration,” ISPRSJournal of Photogrammetry and Remote sensing, vol. 52,no. 4, pp. 149–159, 1997.

[22] S. I. Granshaw and C. S. Fraser, “Editorial: Computer vi-sion and photogrammetry: Interaction or introspection?”The Photogrammetric Record, vol. 30, no. 149, pp. 3–7,2015.

[23] F. Remondino, M. G. Spera, E. Nocerino, F. Menna, andF. Nex, “State of the art in high density image matching,”The Photogrammetric Record, vol. 29, no. 146, pp. 144–166, 2014.

[24] G. Caroti, I. M.-E. Zaragoza, and A. Piemonte, “Accu-racy assessment in structure from motion 3d reconstruc-tion from uav-born images: the influence of the dataprocessing methods,” The International Archives of Pho-togrammetry, Remote Sensing and Spatial InformationSciences, vol. 40, no. 1, p. 103, 2015.

[25] C. Strecha, W. von Hansen, L. V. Gool, P. Fua, andU. Thoennessen, “On benchmarking camera calibrationand multi-view stereo for high resolution imagery,” inComputer Vision and Pattern Recognition, 2008. CVPR2008. IEEE Conference on. Ieee, 2008, pp. 1–8.

[26] C. S. Fraser, “Advances in close-range photogrammetry,”in Photogrammetric Week 2015. Wichmann/VDE Ver-lag, Belin & Offenbach, 2015, pp. 257–268.

[27] Y. Furukawa and J. Ponce, “Accurate camera calibrationfrom multi-view stereo and bundle adjustment,” in Com-puter Vision and Pattern Recognition, 2008. CVPR 2008.IEEE Conference on. IEEE, 2008, pp. 1–8.

[28] H. Hirschmuller, “Accurate and efficient stereo process-ing by semi-global matching and mutual information,” inComputer Vision and Pattern Recognition, 2005. CVPR2005. IEEE Computer Society Conference on, vol. 2.IEEE, 2005, pp. 807–814.

[29] F. Bethmann and T. Luhmann, “Semi-global matching inobject space,” The International Archives of Photogram-metry, Remote Sensing and Spatial Information Sciences,vol. 40, no. 3, p. 23, 2015.

[30] K. Wenzel, M. Rothermel, D. Fritsch, and N. Haala,“Image acquisition and model selection for multi-viewstereo,” Int. Arch. Photogramm. Remote Sens. Spatial Inf.Sci, vol. 40, pp. 251–258, 2013.