augmented reality maintenance modelling - mer innovate · augmented reality: definition augmented...
Post on 23-May-2020
20 Views
Preview:
TRANSCRIPT
Augmented Reality maintenance modelling
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Context and problematic
• Wind turbine are off-shore • ~1 hour for accessing a wind farm.
• Accessibility depends on weather conditions.
=> Little time to perform maintenance.
• Expert can’t always be physically on site.
• Expert may not be allowed to be part of the crew.
2
Need to
improve communication
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Summary
Definition of augmented reality
Principles
Display technologies
Existing frameworks
Augmented reality maintenance modelling
Collaboration configuration
Works done
Next steps
3
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality: definition
Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive in real time 3. Registered in 3-D
Augmentation: virtual object added to the real world.
Real
environment
Virtual
environment
Augmented
reality
Augmented
virtuality
Reality-virtuality continuum
Mixed reality
4
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality: blower maintenance demonstrator
Demo done with
• Unity
• Vuforia SDK
5
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality : how it works?
Principle for augmented reality vision-based tracking (Azuma 1997)
• A camera gets the scene Iscene
• Based on the scene, camera pose is computed.
• When cam pose is known, the virtual object is added in the scene.
Scene image
Iscene
Features extraction
Iscene
Features matching between Iref et Iscene
Camera Pose computation
Adding augmentations
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
9
Display Technologies
Optical see-through
Optical Glance
Video see through
Hand held display
Projective worn
Projective fixed
Contact lens
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
10
Ideal configuration from user perspective
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
Tablet
Intuitive interface
Good resolution
Not hand free
11
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
12
Optical Glance Google glass-like
Hand free usage
Secure
Interface
Small display
Only glance
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
Optical see through
Hand free usage
Registered
Security issue
Small display
13
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
Projective
Natural vision
Limited virtual objects
Indoor usage
14
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: display technologies
15
Display Technologies
Optical see-through
Optical Glance
Video see through
Hand held display
Projective worn
Projective fixed
Contact lens
Choice depends on the application
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015 16
How to choose the right display technology?
Augmented reality: display technologies
Device Type Characteristics
Contac lens Optical see-
through Optical Glance
Video see through
Hand held display
Projective worn Projective fixed
Mobile + + + + + Limited Limited
Outdoor use + + + + + Limited Limited
User interface Gesture
or Tactile Gesture
or Tactile Gesture
or Tactile Gesture
or Tactile Tactile
Gesture or Tactile
Gesture or Tactile
Free hand usage + + + + + +
Field Of View + Limited Limited Limited + + +
Occlusion of real world by AR object
Yes Yes Limited Yes Limited No No
Unrestricted Augmentation model
Yes Yes Yes Yes Yes Limited Limited
Eyestrain Not determined Yes Yes Yes No No No
Power usage Limited Limited Limited Limited Limited Limited Unlimited
Available on the white paper by the end of March
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality: the most relevant frameworks
17
AR Framework Characteristics
Metaio Vuforia Wikitude OpenCV
Marker tracking Yes Yes Yes Dev
Image Target tracking
Yes Yes Yes Dev
3D tracking Yes Limited box and cylinder
No Dev
Face Tracking Yes No No Dev
Unity 3D Yes Yes Yes Need
Integration
iOS Yes Yes Yes Yes
Android Yes Yes Yes Yes
Windows Phone 8 No No No Yes
PC Yes Yes Yes Yes
Accelerometer and Gyroscope sensors
Yes Yes Yes Dev
Developer community
Yes Yes Yes Yes
Official Webinar Yes No No No
Free Yes but
Watermark Yes but
Watermark Yes but
Watermark Yes
Open Source No No No Yes
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Expert – operator maintenance problematic
How the expert can quickly create Augmented Reality content for the operator, without development skills?
How can we accelerate AR Player development on current device and future ones?
18
Need to
improve communication
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality content production modelling
19
AR Editor SolidWorks
Authoring tool
Plugin
…
1
3D model
AR player Smart Glasses
or tablet
Standard
XML
file
Export
Ressources
2 3
Expert Opérateur
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality: maintenance modelling
• An AR maintenance is represented by a list of Action performed on Entity of the system (For example: Unscrew the screw 1).
• An Action is a simple Animation played on one of the Entity of the system to maintain. • The Action Unscrew means an animation starting from position, orientation
(x,y,z), (rx, ry, rz) = (0,0,0) (0,0,0) => (0,0,-50) (0,0, -720°) in 800 msec.
• An Action may be composed of Illustration = an Animation on an ExternalEntity.
• An ExternalEntity is a virtual object that is not part of the system. • For example display a screwdriver to show which tool use for disassembling an element.
20
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality: maintenance modelling
• We can define template Action like Unscrew, Screw,… that are reusable for several maintenance operations.
21
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: maintenance configurations
• Guided maintenance 1. Expert prepares the maintenance tasks in AR.
2. Before leaving, operator loads the task he must perform from the DB.
3. Once on site, he performs the maintenance actions.
22
1
2
Operator
Expert
Remote
On site
3
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented reality: maintenance configurations
• Assisted maintenance (or supervised maintenance) 1. Operator shares his view with the expert and explain his issue.
2. Expert augments the operator view in order to explain and assist him.
23
1
2
Operator Expert
Remote On site
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Works done
• State of the art of augmented reality.
24
• State of the art of Augmented Reality devices and frameworks • We chose Metaio as framework and Moverio BT-200 as a smart glasses.
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Works done
• 2 demonstrators of augmented reality
25
AR Demonstrator with marker
created with ARToolKit
(Alexis Barreau EXIA A2)
AR Demonstrator markerless
created with Vuforia and Unity3D
(Gael Lago EXIA A3 and Olivier Massot ECIA A4)
• AR maintenance modelling v1 (Dorian Boulc’h EXIA A4)
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Works done
Conferences
• 4 Mer Innovate Conferences : CESI Rouen 21 of May, Exeter University 29 of October, Technopole Cherbourg 9 December, Portsmouth 18 of March.
Papers
• IEEE VR conference demo (25-27 March 2015, Arles) Augmented Reality maintenance demonstrator and associated modelling.
• Mer innovate white paper available by the end of March on www.merinnovateproject.eu
26
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Perspectives
• Evolution of the AR maintenance modelling. Adding operator review of the process.
• Based on that model, development of the AR Player on the EPSON Moverio BT-200.
• Creating maintenance scenario.
• Testing the 2 maintenance configurations and evaluate augmented reality for maintenance.
• Study the Gesture User Interface usage for smart glasses.
27
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Augmented Reality for maintenance
Thank you
28
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Bibliography
(s.d.). ARToolKit. Récupéré sur www.hitl.washington.edu/artoolkit/
Azuma, R. T., & others. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. Dans Computer Vision--ECCV 2006
(pp. 404-417). Springer.
Bottecchia, S. (2010). Système TAC: Télé-Assistance Collaborative. Réalité augmentée et NTIC au service des
opérateurs et des experts dans le cadre d'une tâche de maintenance industrielle supervisée. Ph.D. dissertation,
Université Paul Sabatier-Toulouse III.
Bottecchia, S., Cieutat, J.-M., & Jessel, J.-P. (2010). TAC: augmented reality system for collaborative tele-assistance
in the field of maintenance through internet. Proceedings of the 1st Augmented Human International Conference,
(p. 14).
Caudell, T. P., & Mizell, D. W. (1992). Augmented reality: An application of heads-up display technology to manual
manufacturing processes. System Sciences, 1992. Proceedings of the Twenty-Fifth Hawaii International Conference
on, 2, pp. 659-669.
Corke, P. (2011). Robotics, vision and control: fundamental algorithms in MATLAB (Vol. 73). Springer Science \&
Business Media.
Grasset, R., Gascuel, J.-D., & others. (2003). Réalité Augmentée et environnement collaboratif: Un tour d'horizon.
AFIG'03.
Harris, C., & Stephens, M. (1988). A combined corner and edge detector. Alvey vision conference, 15, p. 50.
Henderson, S., & Feiner, S. (2011). Exploring the benefits of augmented reality documentation for maintenance and
repair. Visualization and Computer Graphics, IEEE Transactions on, 17(10), 1355-1368.
29
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Bibliography
icons8. (2014). Icons8. Récupéré sur http://icons8.com
Kalkofen, D., Sandor, C., White, S., & Schmalstieg, D. (2011). Visualization techniques for augmented reality.
Springer.
Koch, C., Neges, M., König, M., & Abramovici, M. (2014). Performance Study on Natural Marker Detection for
Augmented Reality Supported Facility Maintenance. Australasian Journal of Construction Economics and Building-
Conference Series, 2, pp. 23-34.
Lieberknecht, S., Benhimane, S., Meier, P., & Navab, N. (2009). A dataset and evaluation methodology for
template-based tracking algorithms. Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEE International
Symposium on, (pp. 145-151).
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International journal of computer
vision, 60(2), 91-110.
Mallem, M., & Roussel, D. (2008). Réalité augmentée: principes, technologies et applications. Techniques de
l'ingénieur. Télécoms(TE5920).
MerInnovate. (2012). Mer Innovate project. Récupéré sur http://merinnovateproject.eu
Metaio. (s.d.). Tracking configuration. Récupéré sur https://dev.metaio.com/sdk/documentation/tracking-config/
30
Mer Innovate final event in Technopole at Portsmouth, 18 March 2015
Bibliography
Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information
and Systems, 77(12), 1321-1329.
Nee, A., Ong, S., Chryssolouris, G., & Mourtzis, D. (2012). Augmented reality applications in design and
manufacturing. CIRP Annals-manufacturing technology, 61(2), 657-679.
Rosten, E., & Drummond, T. (2005). Fusing points and lines for high performance tracking. Computer Vision, 2005.
ICCV 2005. Tenth IEEE International Conference on, 2, pp. 1508-1515.
University, W. (2011). CSE/EE 576: Computer Vision. Récupéré sur
http://courses.cs.washington.edu/courses/cse576/11sp/,
http://courses.cs.washington.edu/courses/cse576/11sp/notes/SIFT_white2011.ppt
Van Krevelen, D., & Poelman, R. (2010). A survey of augmented reality technologies, applications and limitations.
International Journal of Virtual Reality, 9(2), 1.
Zhou, F., Duh, H. B.-L., & Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display: A
review of ten years of ISMAR. Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented
Reality, (pp. 193-202).
31
top related