deep learning for detecting robotic grasps · de nition deep learning for detecting robotic grasps...

University of Hamburg

MIN Faculty

Department of Informatics

Deep Learning for Detecting Robotic Grasps

Deep Learning For Detecting Robotic Grasps

Waleed Mustafa

University of HamburgFaculty of Mathematics, Informatics and Natural SciencesDepartment of Informatics

Technical Aspects of Multimodal Systems

4. Januar 2016

W. Mustafa 1


MIN Faculty


Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

3. Overview of Grasp Process

4. Grasp Representation

5. Detect Grasp from Image

6. Learning Grasp Ranking Function

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 2


MIN Faculty


Definition Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 3


MIN Faculty


Definition Deep Learning for Detecting Robotic Grasps

Definition

Grasp

”A grasp is commonly defined as a set of contacts on the surfaceof the object, which purpose is to constrain the potentialmovements of the object in the event of externaldisturbances”Leon et al. (2014)

Grasp Synthesis

”Grasp synthesis is the problem of finding a suitable set ofcontacts given an object and some constraints on the allowablecontacts”Leon et al. (2014)

W. Mustafa 4


MIN Faculty


Motivation Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 5


MIN Faculty


Motivation Deep Learning for Detecting Robotic Grasps

MotivationWhy do we need robotic grasp?

I Almost all robotic applications include manipulation of objects

I In order to manipulate an object you need first to grasp itI Applications include:

I Exploration

I Household

I Industry robotic hands

W. Mustafa 6


MIN Faculty


Overview of Grasp Process Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 7


MIN Faculty


Overview of Grasp Process Deep Learning for Detecting Robotic Grasps

Overview of Grasp Process

Goal:

Predict Gripper configuration (i.e., Gripper Location,Orientation, and Gripper Opening Width)

Input:

2-D Image, and Depth Map

Output:

Grasp Representation? We need parameters that represent thegripper configuration

W. Mustafa 8


MIN Faculty


Grasp Representation Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 9


MIN Faculty



Grasp Representation

I A Grasp is:I Gripper LocationsI Gripper PoseI Gripper Opening Width

W. Mustafa 10


MIN Faculty



Grasp Representation (cont.)

I A good representation is:I Easily predicted it from sensory dataI The full grasp parameters can be retrieved from it

I Saxena et al. (2008) proposed one point as a representation ofa graspI Easy to predictI Procedure for retrieving Grasp parameters hard and faulty

I Jiang et al. (2011) Represented grasps as an oriented rectangle

W. Mustafa 11


MIN Faculty



Grasp Representation (cont.)

I Grasp is defined by:I rG ,cG position in image plan, mG , nG width and Height of

rectangle θ is the angle of rectangle with respect to X -axis

W. Mustafa 12


MIN Faculty



Grasp Representation

I Positions, Opening Width, and pose around camera axis aredirectly defined

I Other two angles is computed by:I Select the point with lower depth in the middle thirdI Compute average surface norm around this point Jiang et al.

(2011)

W. Mustafa 13


MIN Faculty


Detect Grasp from Image Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 14


MIN Faculty



Detect Grasp from Image

1. Generate every possible grasp rectangle

W. Mustafa 15


MIN Faculty



Detect Grasp from Image (cont.)

2. Using a function f (x |Θ) : Rn → [0, 1] rank the rectangles,where x is features computed from rectangles

3. Choose the rectangle with highest rank

W. Mustafa 16


MIN Faculty


Learning Grasp Ranking Function Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 17


MIN Faculty



Learning Grasp Ranking Function

I Given a training set (Images with Human marking) we cantrain the function f (x |Θ)

I Jiang et al. (2011) proposed extracting feature from potentialrectangle and build an SVM classifierI Their features was histogram of different filters

I Lenz et al. (2015) Used sparse auto-encoder to automaticallylearn features Goodfellow et al. (2009)I Used neural network to learn the rank functionI The input to the network

W. Mustafa 18


MIN Faculty



W ∗ = arg minW

∑Mt=1 (‖x − x‖2

2 + λ∑K

j=1 g(h(t))) + βf (W )

h(t)j = σ(

∑Ni=1 x

(t)i Wij)

x(t)i =

∑Kj=1 h

(t)j Wij

Lenz et al. (2015)W. Mustafa 19


MIN Faculty



I We can repeat the above process to learn N layersI Finally we stack learned layers together with an output decision

layersI Complete the learning with BP

Lenz et al. (2015)

W. Mustafa 20


MIN Faculty



I It is very slow to run the huge network on all the rectangles

I Lenz et al. (2015) propose a cascaded system

Lenz et al. (2015)

W. Mustafa 21


MIN Faculty



Multi-modal Input

I Concatenate the data from different modes

Lenz et al. (2015)

W. Mustafa 22


MIN Faculty



Multi-modal Input (cont.)

I Separate modes at the first layer

Lenz et al. (2015)

W. Mustafa 23


MIN Faculty



Multi-modal Input (cont.)

I Or a mix

Lenz et al. (2015)

W. Mustafa 24


MIN Faculty


Resutls Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 25


MIN Faculty


Resutls Deep Learning for Detecting Robotic Grasps

Resutls

Lenz et al. (2015)

W. Mustafa 26


MIN Faculty


Demo Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 27


MIN Faculty


Demo Deep Learning for Detecting Robotic Grasps

W. Mustafa 28


MIN Faculty


Conclusion Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 29


MIN Faculty


Conclusion Deep Learning for Detecting Robotic Grasps

Conclusion

I We introduced a grasp detection system based on deep learning

I Results shows that it outperforms systems that are based onhuman designed features

I The problem have a lot on common with object detection

I A lot of methods in object detection can be used

I Overfeat will be fasterI CNN might be more suitable for Images

W. Mustafa 30


MIN Faculty


References Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation





7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 31


MIN Faculty



References

I Goodfellow, H Lee, and QV Le. Measuring invariances in deepnetworks. Advances in neural . . . , 2009. URLhttp://papers.nips.cc/paper/

3790-measuring-invariances-in-deep-networks.

Y Jiang, S Moseson, and A Saxena. Efficient grasping from rgbdimages: Learning using a new rectangle representation. Roboticsand Automation ( . . . , 2011. URL http://ieeexplore.ieee.

org/xpls/abs_all.jsp?arnumber=5980145.

I Lenz, H Lee, and A Saxena. Deep learning for detecting roboticgrasps. The International Journal of Robotics, 2015. URLhttp://ijr.sagepub.com/content/34/4-5/705.short.

W. Mustafa 32

http://papers.nips.cc/paper/3790-measuring-invariances-in-deep-networks

http://papers.nips.cc/paper/3790-measuring-invariances-in-deep-networks

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5980145

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5980145

http://ijr.sagepub.com/content/34/4-5/705.short


MIN Faculty



References (cont.)

B Leon, A Morales, and J Sancho-Bru. From robot to humangrasping simulation. 2014. URL http://link.springer.com/

content/pdf/10.1007/978-3-319-01833-1.pdf.

A Saxena, J Driemeyer, and AY Ng. Robotic grasping of novelobjects using vision. International Journal of Robotics, 2008.URL http://ijr.sagepub.com/content/27/2/157.short.

W. Mustafa 33

http://link.springer.com/content/pdf/10.1007/978-3-319-01833-1.pdf

http://link.springer.com/content/pdf/10.1007/978-3-319-01833-1.pdf

http://ijr.sagepub.com/content/27/2/157.short

deep learning for detecting robotic grasps · de nition deep learning for detecting robotic grasps...

Documents