a short introduction to computer vision-based marker tracking

A Short Introduction toComputer Vision-based

Marker Tracking

Dr. Jens [email protected]

Partly based on materials by Dieter Schmalstieg, Mark Billinghurst, Hartmut Seichter, Daniel Wagner

Dr. Jens Grubert | [email protected]

SpatialRegistrationDefiningRelativePositionofEachElements ofaSceneusingCoordinateSystems(CS)

WorldCS

Camera/SensorCS

Initialposeestimation: RegistrationTemporalupdates givenaninitialpose:Tracking

Imagecourtesy

bySesameWorkshop

http://w

ww.sesam

eworkshop.org/our-b

log/tag/new-technology/


OverviewTrackingMagneticTracker InertialTracker UltrasonicTrackerVision Tracker

Marker-BasedTracking MarkerlessTracking

SpecializedTracking

RecursiveTracking TrackingbyDetection

Edge-BasedTracking Template-BasedTracking

InterestPointTracking

MechanicalTracker

MonocularTracking

DepthBased


MarkerTracking

• Doneformorethan15years• Severalopensourcesolutionsexist• Fairlysimpletoimplement

• Standardcomputervisionmethods

• Arectangularmarkerprovides4cornerpoints• Ú enoughforposeestimation


MarkerBasedTracking– Examples

4. Marker types and identification

90

�

© jamesmarsh.com

Figure 60. Examples of visual designs: On the upper left: QR code used in Red Hot Chilli Peppers’ ‘I’m with you’ album campaign. On the upper right: artwork by James Marsh, ‘Barcode butterfly’ © jamesmarsh.com (Image courtesy of James Marsh). On the bottom: QR codes designed by JESS3 [134] (images courtesy of JESS3).

If the visual appearance plays a very high role in application, it might be a good idea to spend some effort on professional design (see Figure 60).

4.4.5 General marker detection application

Besides augmented reality, markers are widely used for other purposes as well. Our mobile phone application [3, 11, 17] automatically launched the appropriate application depending on the marker content. If a marker contained a phone num-ber, it launched a phone application. Likewise, it launched a web browser if a marker contained a URL. We also proposed GPS as one type of data: users could get the GPS location from a marker. This would enable location-based AR brows-ers on mobile devices that are not equipped with GPS and at indoor locations where GPS is inoperative.

In a general AR browser scenario, we would like to attach hyperlinks to objects (cf. Internet of Things). However, the space of all possible objects is far too big to handle. We see several solutions to overcome this issue. For example, one could limit the browser to one type of data at a time. Current location-based browsers have different environments. In practice, the “bookstore application” would recog-


MarkerTrackingPipelineOverview

Imagecourtesy

byDa

nielW

agner


MarkerTrackingPipelineOverview

0:Input 1:Threshold 2:Contours

3:MarkerCandidates4:PatternChecking5,6:CornerRefinement,Poseestimation


9

1-3:Fiducial Detection1. Thresholdthewholeimage2. Contours

1. Searchscanline perscanline foredges(whitetoblacksteps)

2. Followedgesuntileither:BacktostartingpixelorImageborder

3. Contourapproximationusingpolygons,sanitychecks(4corners,size)


10

4:Patternchecking

• Calculatehomography usingthe4cornerpoints• DirectLinearTransformalgorithm• Mapsnormalized coordinatestomarker coordinates(simpleperspective projection,nocameramodel)

• Extractpatternbysampling• Checkpattern

• Id(implicitencoding)• Template (normalized crosscorrelation)

• 42D-3Dcorrespondences~poseestimation


5:CornerRefinement

• Refinecornercoordinates• Criticalforhighqualitytracking• Remember:4pointsisthebareminimum!• Sothese4pointsshouldbetterbeaccurate…

• Detectsub-pixelcoordinates• E.g.,Harriscornerdetector• Specializedmethodcanbefasterandmoreaccurate• Stronglyreducesjitter!

• Undistortcornercoordinates• Removeradialdistortionfromlens


6:PoseEstimation

• Calculatesmarkerposition/rotationrelativetocamera

• PnPalgorithmwith42D-3Dcorrespondences• Alternative:UseHomography

• Directestimationfrom42D- 2Dcorrespondences• Veryfast,butcoarse• Jittersalot…


Now it‘s your turn

• Downloadmarkertrackerexamplefromhttps://bitbucket.org/jens_grubert/markertracker• Built the code using CMAKE(https://cmake.org/)• RunmarkerTrackerTestwithparameters:

markerTrackerTest <cameraId> <camera.xml> <markerSize>

e.g.:

markerTrackerTest 0 lc910.yml 9.2


MainTrackingMethodbool MarkerDetectorBW::findMarkers(const cv::Mat& frame, std::vector<MarkerBW>& detectedMarkers)

{

// Convert the image to grayscale

prepareImage(frame, m_grayscaleImage);

// Make it binary

performThreshold(m_grayscaleImage, m_thresholdImg);

// Detect contours

findContours(m_thresholdImg, m_contours, m_grayscaleImage.cols / 5);

// Find closed contours that can be approximated with 4 points

findCandidates(m_contours, detectedMarkers);

// Find is them are markers

recognizeMarkers(m_grayscaleImage, detectedMarkers);

// Calculate their poses

estimatePosition(detectedMarkers);

//sort by id

std::sort(detectedMarkers.begin(), detectedMarkers.end());

return false;

}Dr. Jens Grubert | [email protected]

Result


a short introduction to computer vision-based marker tracking

Technology