on-line artificial intelligence applications in monitoring ... · m. di castro, on-line artificial...
TRANSCRIPT
Mario DI CASTRO
On behalf of the
EN-SMM group
On-line Artificial Intelligence Applications in Monitoring,
Mechatronics and Robotics at CERN
EN workshop on Machine Learning and Artificial Intelligence:
Activities and Results in the EN Department, 10 April 2019
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 2
Machine learning
Allows software to learn from training data in order to learn
statistical models of real world problems
Machine learning is not a new technology (first prototypes of
statistical learning from early 1970s)
Latest advances in computing power and hardware development
allow practical execution of machine learning algorithms
Enormous effort in software and hardware development in place
by biggest companies (e.g. Intel, Google, Apple, Nvidia) makes
the research field extremely dynamic
Wide variety of Open Source library for fast prototyping and
development
A machine learning typical pipeline [1]
Application of machine learning to real life scenarios
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 3
Machine learning categories Three main categories [1]:
Supervised
Unsupervised
Reinforcement
Supervised learning learns from a prepared set of
labeled data The algorithm will be able to observe a new, never-
before-seen value and predict its label
Unsupervised learning learns from random data using
tools to understand the properties of this data The algorithm will be able to group, cluster, organize data
Reinforcement learning learns from its own mistakes It uses a “satisfaction” function to distinguish between a
good and a bad behavior
The algorithm will be able to choose and perform actions
Supervised learning
Unsupervised learning
Reinforcement learning
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 4
Needs for Mechatronics/Robotics at CERN
Control, inspection, operation and maintenance of radioactive particle
accelerators devices
Experimental areas and objects not built to be remote handled/inspected Any intervention may lead to “surprises”
Risk of contamination
The LHC tunnelNorth Area experimental zone
Radioactive sample handled by a robot
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 5
Machine learning in robotics #1
Great advances in robot vision thanks to supervised deep
learning techniques Accuracy in object tracking (Fast-RCNN, Mask-RCNN)
Object grasping points calculation
Control of closed chains kinematic robots Still an open issue, Long short-term memory (LSTM) networks
for system dynamic learning
Advances in situation awareness for autonomous behaviors Possibility of learning to predict external changes in the
environment
Human-Robot collaboration Advances in speech recognition, gesture recognition, human
action prediction
Human Robot collaboration for mechanical assembly
Saliency detection (center of attention) in self-driving cars for situational awareness [3]
Grasping points for everyday objects [2]
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 6
Machine learning in robotics #2
Robot still do not appear fast enough Slow in decision making
Difficult to adapt to real world scenarios
Robot still don’t appear fast enough [4]
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 7
Machine learning in Robotics #3
Robotics community is investing strongly in machine learning adapted to social robotics
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 8
Applications of Machine Learning in EN-SMM
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 9
Object Detection and Recognition for Teleoperation #1 Teleoperation is strongly increased during the last years at CERN [5] [19-22]
Telemax robot
Teodor robot
EXTRM robot with single arm (CERN made)
EXTRM robot (CERN made)CERNbot (CERN made)
The TIM (CERN made)
CRANEbot (CERN made)
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 10
Object Detection and Recognition for Teleoperation #1 Teleoperation is strongly increased during the last years at CERN [5] [19-22]
Telemax robot
Teodor robot
EXTRM robot with single arm (CERN made)
EXTRM robot (CERN made)
The TIM (CERN made)
CRANEbot (CERN made)
More than 20 robots in operation
CERNbot (CERN made)
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 11
Robotic Interventions Video
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 12
Object detection and recognition for Teleoperation #2 Machine learning (Faster-RCNN) is used to assist online grasping tasks in teleoperation
Visual servo control endorsed with AI [19]
Object detection embedded in CERN Human-Robot Interface to process live images endorsed
with super resolution techniques [6]
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 13
Learning by Demonstration
Robots can learn complex
tasks
Learning Benefits
Robot fast adaptation to
new tasks and the
environment
Fully autonomous task
implementation
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 14
Learning by demonstration
Dynamic Movement Primitives (DMP) [21]
Mathematical motion
representation
Online motion execution
+
Vision tracking for changing
conditions
Stable adaptation to
new conditions in
dynamic system
Learning by
demonstration
Dynamic Movement
Primitive
Encode user gestures in
primitives
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 15
Learning by demonstration: fusing Fast-RCNN [10] and
DMP for dynamic environment
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 16
HL-LHC RF cavities: proof of concept for internal
welding polishing using a robotic arm (learning by
demonstration techniques)
Picture of a cavity
Preliminary results of a polish on a welded joint.
Before polishing (left) and after polishing (right)
Preliminary integration
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 17
Proof of concept for laser welding for HL-LHC triplets
beam screen using a robotic arm (learning by
demonstration techniques)
Learning by demonstration
Execution of the weldingVery promising
preliminary results
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 18
TIM Survey Wagon alignment to fiducials (Fast-RCNN)
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 19
Autonomous tests of LHC Collimators switches
(Fast R-CNN + learning by demonstration)
Deep learning for
object and pose
recognition
Machine learning for
autonomous
operations
Safety using virtual
fixtures to avoid
collisions
LHC Collimators
LHC Collimators position switches plate
Close view of the LHC Collimators position switchesInternal view of the LHC Collimators
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 20
Autonomous tests of LHC Collimators switches
(Fast R-CNN + learning by demonstration)
Deep learning for
object and pose
recognition
Machine learning for
autonomous
operations
Safety using virtual
fixtures to avoid
collisions
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 21
Online measurement quality prediction during
position displacement
Deep-learning regression network for online
measurement quality and prediction of sensors
going “out-of-range”
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 22
Tunnel structure monitoring with defects detection
Faster-RCNN provides simple bounding boxes around the recognized object
Complex applications require more detailed detection
Mask-RCNN [13] provide accurate segmentation around the recognized object
Examples of object recognition using machine learning
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 23
Online Tunnel Structure Monitoring
Detects defects (cracks, water leaks,
changes [13-14]) using a Mask-RCNN
network.
High-definition picture collection using
TIM and CERNBot
3D reconstruction of wall using Structure
from Motion techniques to compare time
evolution of defects (available on web
browser or virtual reality headset)
Complete application under
commissioning, from data collection to
data analysis
HD cameras mounted on TIM and CERNbot
Scheme of the working principle
HD camera system for tunnel dome view
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 24
Continuous training by live correction improves results as the dataset grows
Tunnel Structure Monitoring
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 25
Visual based RF cavities quality control
Same technique used for defect detection is applied to surface quality control of
the HL-LHC RF cavities
HL-LHC RF cavity Anomaly (burn) Welding cracks
Curtesy of A. Macpherson
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 26
Super resolution for visual online monitoring #1
Generates higher resolution less noisy
images from small resolution compressed
images
Two categories:
Single image super-resolution [7]
Multiple image super-resolution [8]
State-of-the-art neural networks produce
great results but are not suitable for real-
time display
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 27
We merged 2 neural networks : compression noise
reduction and resolution enhancement [9]
Reduce 4G bandwidth consumption for transmitting
images
Generates no lag thanks to real-time capabilities
Little defects in some images are not critical as images
are displayed to the operator at 15 fps
Multiscale super resolution available (2x, 4x, 8x etc.)50% jpeg compression; 14 kb
4X resolution enhancement + noise
reduction; 282 kb; computation time 4 ms
Super resolution for visual online monitoring #2
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 28
Object Detection and Pose Estimation for Measurements
and Calibrations Sensors Approach
Faster-RCNN network for online 2D Beam Loss Monitors (BLM) localization [11]
Multiple RGB-D cameras used for 3D reconstruction of the environment
Bounding boxes generated by Faster-RCNN on each camera allow triangulation of the bounding box in the
space
ICP algorithm [12] on the depth sensor allows accurate 3D localization
3D pose will be used by the robotic arm path planner to calculate a safe approach to the BLM in the
reconstructed environment
INTEL 3D camera on robot arm end effector BLM recognition
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 29
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 30
People recognition and vital monitoring
Machine learning techniques enhance people detection and
vital signals monitoring at distance
People search and rescue is of primary interest in disaster
scenarios
People monitoring during rehabilitationVision system (2D Laser, radar, thermal and 2D-3D camera)
Online respiration monitoring
Online people recognition and tracking
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 31
Quality control for RP sample positioning
RP sample changer enhances throughput for spectrographic
analysis of samples
Supervised deep learning helps in ensuring heterogeneous
sample position for measurement quality control
The RP Sample Changer A sample not precisely positioned
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 32
LSTM for vocal human robot collaboration
Human-Robot Collaboration in tunnel navigation
using vocal commands
e.g. “turn left”, “turn right”, “stop”…
Difficulties due to:
Distance (up to 15 m)
Background noise
Multiple people
Embedded
State-of-the-art speech recognition use cloud
services (Google AI, Microsoft Azure AI, Mozilla
Deep Speech etc.) based on LSTM recurrent
networks
Expensive
Requires internet connection
LSTM network example
Raw speech FBANK features [16]
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 33
Automatic Texturing for Photorealistic Virtual Reality
Automatic texturing using deep-
learning produces photorealistic
texturing during CAD model
importation
Photorealistic texturing enhances
users immersion in virtual reality
environments
Texturing is a long manual process
done with state-of-the-art processes
CERN CAD models contain
information about the material of each
element First row: input image. Second row: retrieved shape and extracted texture
patch. Last row: re-rendered image with our texture transfer output and
estimated illumination. Please note that the proxy shapes differ from the input
images, such as the brim of the cap, handle of the mug, top of the vase, and
aeroplane wings [17]
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 34
Brain-Robot Interface for robot arm control
Online analysis of brain signal
Augmented reality glasses used for commands
display
Eyes focus point detected by CNN processing Steady
State Visual Evoked Potentials (SSVEP [15]) which
are synchronous responses produced in the visual
cortex area when observing flickering stimuli
8 s window:fundamental and 1° harmonic
Raw signal
FFT
Example of brain activity monitoring
Hardware used for the brain monitoring
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 35
Brain-Robot Interface for robot arm control
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 36
Reinforcement learning for safe trajectory planning in RF cavities visual
inspection and BLM positioning
Robot motion planning in cluttered unstructured environment [18]
Reinforcement working learning principle
BLM in a “complicated” position
Conceptual design for autonomous RF
cavities inspection
Using reinforcement
learning for safety critical
trajectory planning
Trajectories are simulated
in a local environment
A reward function is used to
“teach” the robot correct
behaviors
Once the robot “learnt”, it
can perform safe
trajectories in generic
environments
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 37
Helium leak spill detection during LHC warm-up
System already in operation with machine
learning in 5 points in collaboration with TE-
CRG-CE
Web application for operation monitoring
Sensitive image processing with noise filtering
Lots of false positive alarms due to open
access to the area
Machine learning will allow to look only for helium
spill reducing false positive alarm
Possibility to directly connect to valves interlock
system
The web application
Example of false positive during operation
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 38
Bachelor, Master and PhD students
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019
Conclusions
39
The SMM group has acquired knowledge and expertise to provide mechatronics and
robotic support to other CERN departments and groups according to the resources
available
Machine learning technologies have been applied by the SMM group to different
mechatronics/robotics applications within the CERN accelerators complex
Our “unicity” is the fact that we “control” the complete lifecycle of a
mechatronic/robotic system, from the design up to the operation on the field
Artificial intelligence is assisting the current measurements, inspection and
teleoperation tasks reducing the stress of the operators and increasing the
robustness of the robotic intervention and of the mechatronics applications
R&D, machine learning and continuous developments models are fundamental
because ready-to-use mechatronics/robotic solutions that can fulfill CERN needs for
real-time controls, remote inspection and user-friendly teleoperation do not exist
M. Di Castro, On-line Artificial Intelligence Applications in Monitoring, Mechatronics and Robotics at CERN, 10 April 2019 40
References1. Maini, Vishal, and S. Sabri. "Machine learning for humans." Online: https://medium. com/machine-learning-for-humans (2017).
2. Pelossof, Raphael, et al. "An SVM learning approach to robotic grasping." IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA'04. 2004. Vol. 4. IEEE, 2004.
3. Alletto, Stefano; Palazzi, Andrea; Solera, Francesco; Calderara, Simone; Cucchiara, Rita "DR(eye)VE: a Dataset for Attention-Based Tasks with Applications to Autonomous and Assisted
Driving" IEEE Internation Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, 2016, 2016
4. Vogt, David, et al. "A system for learning continuous human-robot interactions from human-human demonstrations." 2017 IEEE International Conference on Robotics and Automation (ICRA).
IEEE, 2017.
5. Di Castro, Mario, et al. "A dual arms robotic platform control for navigation, inspection and telemanipulation." (2018): TUPHA127.
6. Lunghi, Giacomo, Raul Marin Prades, and Mario Di Castro. "An Advanced, Adaptive and Multimodal Graphical User Interface for Human-robot Teleoperation in Radioactive Scenarios."
ICINCO (2). 2016.
7. Yang, Chih-Yuan, Chao Ma, and Ming-Hsuan Yang. "Single-image super-resolution: A benchmark." European Conference on Computer Vision. Springer, Cham, 2014
8. Dong, Chao, et al. "Image super-resolution using deep convolutional networks." IEEE transactions on pattern analysis and machine intelligence 38.2 (2016): 295-307.
9. Zhang, Kai, et al. "Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising." IEEE Transactions on Image Processing 26.7 (2017): 3142-3155.
10. Girshick, Ross. "Fast r-cnn." Proceedings of the IEEE international conference on computer vision. 2015.
11. Ren, Shaoqing, et al. "Faster r-cnn: Towards real-time object detection with region proposal networks." Advances in neural information processing systems. 2015.
12. Chetverikov, Dmitry, Dmitry Stepanov, and Pavel Krsek. "Robust Euclidean alignment of 3D point sets: the trimmed iterative closest point algorithm." Image and vision computing 23.3 (2005):
299-309.
13. He, Kaiming, et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017.
14. Attard, Leanne, et al. "Vision-based change detection for inspection of tunnel liners." Automation in Construction 91 (2018): 142-154.
15. Lin, Zhonglin, et al. "Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs." IEEE transactions on biomedical engineering 53.12 (2006): 2610-2614.
16. Liu, X. "Deep Convolutional and LSTM Neural Networks for Acoustic Modelling in Automatic Speech Recognition." (2018).
17. Wang, Tuanfeng Y., et al. "Unsupervised texture transfer from images to model collections." ACM Trans. Graph. 35.6 (2016): 177-1.
18. Murray, Sean, et al. "Robot Motion Planning on a Chip." Robotics: Science and Systems. 2016.
19. Di Castro, Mario, Manuel Ferre, and Alessandro Masi. "CERNTAURO: A Modular Architecture for Robotic Inspection and Telemanipulation in Harsh and Semi-Structured Environments." IEEE
Access 6 (2018): 37506-37522.
20. https://cds.cern.ch/record/2670732
21. https://cds.cern.ch/record/2670927
22. CERN-THESIS-2015-471
23. https://cds.cern.ch/record/2670925
Thank you for your attention