simulation of visual servo control for robotic harvesting of fruits (in v-rep and ros)
Post on 20-Jan-2017
226 Views
Preview:
TRANSCRIPT
Outline
Visual Servo Control
Simulation approach to robotic harvesting
Research & Developments in Robotic Harvesting
Major Challenges, Variability and occluded fruits
Thinking outside the box
Manipulator mechanism?
Manipulator of choice?
Soft Robotic Grippers/Manipulators,
Lower-cost cameras, robust image processing
Efficient harvesting time
Optimization
Hybrid approach
Conceptual design (Arrays of Actuators Mechanism)
Virtual harvesting approach
Simulation Software
V-REP, Quick Intro
First step: Inverse kinematics
Inverse Kinematics in V-REP,
Simulated Sensors (RGB, Kinect, Proximity, Laser)
Simulation environment and scene objects (V-REP/ROS setup)
Calculation modules
Scan experiments (Plant/fruit scan with manual joint control)
Scan experiments (Plants scan)
Scan Experiments (Automated plant scan in x-y plane)
Scan Experiments (Automated plant scan in x-y-z space)
Simulation of visual servo control (Localization and tracking)
Experiments with path planning
Experiments with depth sensors (Microsoft Kinect)
Experiments with range finder sensors (Hokuyo URG-04LX-UG01)
Experiments with 3D laser scanner sensors (3D Laser scanner fast rangefinder)
Conclusion (Market trend)
Acknowledgement
2 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Visual Servo Control Objective
Objective: A robot end-mounted camera is used to position the robot arm in a plane orthogonal to the axis,
such that the fruit to be harvested is centered in the camera’s field of view.
Control
law
Inverse
Kinematics
Feature extraction Pose determination
Joint
Controllers
Power amplifiers
+
-
𝑪𝒙𝒅
𝑪𝒙
Position-based eye-in-hand visual servo structure
Image processing Feature extraction Pose determination
Inverse kinematics Hardware setup
Reference values CONTROLLER
The system has no trajectory generator, instead a feedback loop, closed visually, is
used to control the robots arm position.
(Two control laws were simulated: join angles and joint velocity)
Input Output
3 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Simulation approach to robotic harvesting Problem statement
Difficulties and hassles of experiments with actual hardware setup in robotic harvesting Time constraints
Unavailability of equipment (i.e., sensors, cameras, and the robot manipulator)
Experiment costs
Some hardware setups may result in actuator saturation
Experiments creating unsafe situation for the operators and for the plants system
Detection rates and visual servo control need to be improved
Sensing from the gripper needs to be performed from multiple viewpoints
Virtual environment
simulating various combinations of plant/fruit/robot/sensors scenarios
4 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Research & Developments in Robotic Harvesting Sweet pepper harvester, CROPS and SWEEPER (Wageningen UR)
Courtesy of CROPS http://www.crops-robots.eu/
Courtesy of Wageningen UR and http://sweeper-robot.eu/
Source: https://www.wageningenur.nl/en/newsarticle/New-Sweet-pepper-harvesting-robot-Sweeper-ready-for-testing-in-commercial-greenhouse-.htm
5 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Research & Developments in Robotic Harvesting Citrus harvester
Courtesy of Univ of Florida. Image source: http://ncr.mae.ufl.edu/index.php?id=research/citrus_harvesting
Courtesy of Energid Citrus Picking System Source: https://sites.google.com/site/cvhanaian/research
6 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Research & Developments in Robotic Harvesting Strawberry harvester
http://www.techradar.com/news/world-of-tech/future-tech/colour-
smart-robot-has-eye-for-strawberries-323902
Strawberry harvester. Courtesy Agricultural machinery maker Shibuya Seiki and the National Agriculture and Food
Research Organization
The arm of Robotic Harvesting's machine picking strawberries. http://mathscinet.ru/robocenter/
7 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Research & Developments in Robotic Harvesting Strawberry harvester (Wageningen UR)
Courtesy of Wageningen UR. Source:
https://www.wageningenur.nl/en/show/Cucumber-harvesting-robot.htm
8 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Research & Developments in Robotic Harvesting Other platforms
The French grape-picking robot
Source: 6-DOF Robotic Arm Páprika Chili-Pepper
Harvester. Courtesy of DIROSE SAC https://www.festo.com/cms/en_corp/14014.htm
http://www.producebusinessuk.com/services/stories/2016
/08/23/robot-harvesting-and-weeding-well-on-the-way-
into-mainstream
MIT Robot Gardener
Berry nice: harvest ripe berries
MIT-CSAIL-DRL-Distributed Robotics Garden.
Photo used with permission from Nikolaus Correll.
Agrobot SW6010
9 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Major Challenges, Variability and occluded fruits Variability in Environment, Fruit color and Orientation
10 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Thinking outside the box
Eliminating the redundant vision feedbacks and mission time
Linearizing the problem
Creating Sub-Problems (smaller task)
Manual fruit detection, Automated harvesting
Re-Thinking the Manipulator
Re-Thinking the Gripper
Re-Thinking the image-processing
Re-Thinking the entire strategy
11 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Manipulator mechanism? How many degrees of Freedom?
12 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Manipulator of choice? Purchase or custom-build?
UR 5 KUKA LBR iiwa 14 R 820
KUKA LBR iiwa 7 R 800
UR 10
PhantomXPincher
Baxter
Jaco Arm
Mico Arm
FANUC LR Mate 200iD
KUKA LBR 4
ABB IRB 360 Adept Quattro 650HS
13 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Soft Robotic Grippers/Manipulators Are they promising?
14 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Soft Robotic Grippers/Manipulators Are they promising?
15 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Lower-cost cameras, robust image processing Fruit detection and tracking
Red Threshold Low=75 Red Threshold Low=60 Red Threshold Low=50
16 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Lower-cost cameras, robust image processing Fruit detection and tracking
Original image (left) and edge detection with fuzzy-logic
image (right) for a typical red-pepper fruit Demonstration of red-pepper fruit detection in MATLAB, (i), Red, Green and Blue bands, (ii): Mask of only
red object, is-red mask, (iii): Border smoothed, regions filled, masked red image, (iv): Masked image
showing only red detected object
R G
B
(i)
(ii)
(iii)
(iv)
(i)
(ii)
(iii)
(iv)
Demonstration of red-pepper tracking, (i):original image, (ii):extracting red component from
grayscale image and median filter to filter out the noise, (iii): Convert the resulting grayscale
image into a binary image and removing pixels less than 3000 pixels, (iv): blob analysis,
bounding the red objects in rectangular box and showing centroid
17 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Lower-cost cameras, robust image processing Fruit detection and tracking
Video Demo Real-time, robust and rapid red-pepper fruit
detection with Matlab © Redmond Ramin Shamshiri:
https://youtu.be/rFV6Y5ivLF8
18 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Efficient harvesting time Path planning, Shortest time
X Y
Z
19
Desired Time: 4 to 6 sec!!
19 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
X Y
Z
Efficient harvesting time Path planning, maximum visibility
20
65% Visible
85% Visible
98% Visible
What about hand-
in-eye approach?
20 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Hybrid approach Simplified VSC, lower cost manipulators/cameras, lesser control effort
Camera View Camera View
20,000$ each
500$, each
22 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
23 Contact: ramin.shamshiri@wur.nl
Sweeper Project, Redmond Sh, Presentation, SEP 2015
23 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Hybrid approach, Linear manipulators, suction gripper
Po
ints
of
eff
ec
t
Suction Gripper
Courtesy of Jochen Hemming, CROPS
24 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Z –Displacement is Controlled
by Laser, or Simple
Impact/Force Sensor
Z
Y
X
Hybrid approach, Arrays of actuator mechanism
Mesa Swissranger SR4000
Time of fly camera AVT Prosilica GC2450
Colour camera
25 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Conceptual design Arrays of Actuators Mechanism
26 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
More conceptual design
Image source: http://www.freshplaza.com/2012/0816/paprikarobot.jpg
Image: Courtesy of Vision Robotics. http://www.wired.com/2007/06/robo-picker/
Courtesy of Hanaian. https://sites.google.com/site/cvhanaian/research
Vision Robotics.
27 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Virtual harvesting approach
Virtual Greenhouse
Virtual Plant / Fruit
Manipulator model
simulation can provide a cheap and safe experiment platform with a faster approach
for development, testing and validating control strategies and algorithms.
28 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Simulation Software
1. Robot Virtual Worlds
2. RoboDK
3. Microsoft Robotics
Developer Studio
4. Webots
5. Workspace
6. V-REP
7. LabVIEW
8. OpenHRP3
9. Player
10. Simbad
11. Algodoo
12. RoboWorks
13. RobotStudio
14. Gazebo
15. Actin Simulation
16. Blender
17. WorkcellSimulator
18. robotSim:Edu
19. 3DSimulate
20. Roboguide
21. RoboLogix
22. Ezphysics
23. SimRobot
29 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
V-REP, Quick Intro
First public release in March 2010, V-REP V2.4.2
Latest version: V-REP V3.3.2 (September 2016)
Has big functionality (the system is developed since March, 2010)
Completely open-source (it is uploaded publicly in 2013)
Cross-platform — windows, mac, linux (works at Qt)
Has API & libraries for work with robots through C/C ++, Python, Java, Lua, Matlab, Octave or Urbi
Free for non-commercial use!
30 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
First step: Inverse kinematics IK recipe in V-REP
1) Importing CAD file (i.e., from SolidWork)
2) Defining Manipulator Links
3) Adding Joints
4) Associating Joints and Links
5) Creating Kinematic chains, described with a "tip" dummy and a "base" object
6) Adding A "target" dummy that the "tip" dummy will be constrained to follow
7) Registering the Inverse Kinematic task and do the adjustments, settings, etc…
Note 1: V-REP can open .STL CAD files, if the CAD file from SolidWork has other extension (i.e., .SLDASM), and you do not
have SolidWork installed, you can use the free software “eDrawings” to do the conversion.
Note 2: Defining manipulator links can be quiet tricky, depending on the imported CAD file
Note 3: Adding joints need a good understanding of the manipulator links and position
In V-REP, an IK task requires specification of at least following elements:
• a kinematic chain described with a "tip" dummy and a "base" object.
• A "target" dummy that the "tip" dummy will be constrained to follow.
Attach the Tip-Dummy to the Last Link, select “Tip", then “last Link", then click [Edit --> Make
last selected object parent
31 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Inverse Kinematics in V-REP Example: 7-DoF Redundant Manipulator
Exploring Sub-Shapes Group relevant Shapes to create Links Joints Added
Building the kinematic chain Define an inverse kinematics task
Video Demo Inverse Kinematics 7 DOF Manipulator, V-REP
©Redmond Ramin Shamshiri
https://youtu.be/mHE_PzD3Y2M
32 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Inverse Kinematics in V-REP Fanuc - LR Mate-200iD
Video Demo Inverse kinematics Fanuc LR Mate200iD for Robotic Pepper
Harvesting, V-REP, ©Redmond Ramin Shamshiri
https://youtu.be/s6UZWDS_Cn8
33 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Simulated Sensors RGB, Kinect, Proximity, Laser
Fish-eye RGB Axis 212 PTZ sensor (left, Source) and V-REP perspective vision-sensor (right)
Microsoft Kinect, Fast Hokuyo URG-04LX-UG01‖ scanning Laser Rangefinder, and ―3D Laser
Scanner Fast‖ rangefinder
An example of Infrared Proximity Sensor Long Range - Sharp GP2Y0A02YK0F,
and Bottom: different proximity sensors types in V-REP, respectively from left to right: Ray-type,
pyramid-type, cylinder-type, disk-type and cone-or randomized ray-type proximity sensors
34 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Simulation environment and scene objects V-REP/ROS setup
3D laser scanner
Kinect Hokuyo URG-04LX-UG01
Arm camera
RGB vision sensor
Proximity sensor
Fixed Camera
A 180° scanning path
Cameras/sensors floating views and graphs
Plant model
Video Demo V-REP, ROS Env Setup
©Redmond Ramin Shamshiri
https://youtu.be/tKagjNQ9FW4
35 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Calculation modules
Minimum distance calculation collision detection
path planning
Control mechanism
Remote
API
Serial
port
Plugins
ROS
nodes
Control
Mechanisms
Embedded
Child Script
inverse kinematics
36 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Scan experiments Plant/fruit scan with manual joint control
Fish-eye RGB Axis 212 PTZ sensor (Source)
and V-REP perspective vision-sensor
Video Demo Fruit/Plant Scan Demo
©Redmond Ramin Shamshiri
https://youtu.be/6EOy1NesvQg
37 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Video Demo Scanning Around/Inside/Outside Red Pepper
Plants with Fanuc LR Mate 200iD ©Redmond Ramin Shamshiri
https://youtu.be/ZstETgfUDLg
Scan experiments Plants scan
38 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Scan Experiments Automated plant scan in x-y plane
Simulated scan of a red sweet pepper fruit. Camera view at 30 degrees increments around
the fruit model and leaves. The robot moves in the x-y plane for finding best angle of attack
Fruit/plant 360-degree scan experiment in x-y plane, and the right image is a top view of the
scanning setup. The RGB sensor mounted on the robot tip moves on the horizontal plane x-
y, to find the best view of the fruit
39 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Scan Experiments Automated plant scan in x-y-z space
Scanning around fruit/plant in x-y-z plane, the manipulator is ―twisted‖ to provide different
viewpoints for the end-effector camera
Vertical scan of fruits/plants in x-y-z plane, the end-effector camera is moved and rotated in
both x-z and y-z directions
40 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Simulation of visual servo control Localization and tracking
Joints position control law (Localization)
1 2 3
4 5 6
Joints velocity control law (Tracking)
Robot and camera with eye-in-hand configuration setup. The RGB camera was mounted on the tip of
the FANUC LR Mate 200iD robot manipulator.
Video Demo #1
Video Demo #2 Visual Servo Control in V-REP for Robotic harvesting of sweet pepper
©Redmond Ramin Shamshiri
https://youtu.be/VupoirQOL0Y
https://youtu.be/eCj9qTTeqzs
41 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Experiments with path planning
Video Demo Path Planning in V-REP with Fanuc LR Mate200iD for robotic harvesting of sweet pepper
©Redmond Ramin Shamshiri
https://youtu.be/mjJhDNNaE-4
42 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Experiments with depth sensors Microsoft Kinect
Depth sensors
Microsoft Kinect
Depth image to work image
Depth image to work image
Depth image to work image
43 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Experiments with range finder sensors Hokuyo URG-04LX-UG01 Rangefinder
Depth image to work image
Extract coordinates from work image
Work image to output image
Filter used in the ―Fast Hokuyo
URG-04LX-UG01‖ sensor
―Fast Hokuyo URG-04LX-UG01‖ scanning
Laser Rangefinder
―Hokuyo URG-04LX-UG01
44 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Experiments with 3D laser scanner sensors 3D Laser scanner fast rangefinder
Original depth image to work image
Extract coordinates from work image
Intensity scale work image
Work image to output image
3D Laser Scanner Fast‖ rangefinder
45 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
Conclusion Market trend
Blueberry picking robot could win $250,000 prize!
Main proposed operating specifications include:
size/color sensing for detecting fruit maturity,
careful handling to avoid any damage to the fruit, ability
to directly fill field trays/baskets, 24/7 harvesting
capability, average production of 500 Kg/day, mobility
capabilities on uneven ground or dusty conditions, and
fabrication costs per unit <$500,000 USD
Read more: http://www.naturipechallenge.com/
46 / 47.
© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting
top related