graspit!: a versatile simulator for robotic graspingamiller/thesis.pdfa versatile simulator for...
TRANSCRIPT
GraspIt!: A Versatile Simulator for Robotic
Grasping
Andrew T. Miller
Submitted in partial fulfillment of the
requirements for the degree
of Doctor of Philosophy
in the Graduate School of Arts and Sciences
COLUMBIA UNIVERSITY
2001
c©2001
Andrew T. Miller
All Rights Reserved
GraspIt!: A Versatile Simulator for Robotic
Grasping
Andrew T. Miller
Grasping in an industrial setting is generally accomplished with low complexity
grippers which offer a high degree of reliability within a structured environment.
But, as robots are moved out of the factory into more unstructured settings, they
will need more flexible devices to grasp or manipulate objects. To fill this need,
articulated hands are being designed with more and more degrees of freedom, but
this additional flexibility comes at a price of additional complexity in the design
process and in choosing a grasp from the large number of possibilities. Mechanical
grasping theory, which has been thoroughly studied in the last 20 years, provides
objective methods for evaluating grasps, but the field has lacked a tool that can
apply this analysis to arbitrary grasps of objects using different robotic hands.
This thesis presents a grasping simulator that can aid both robotic hand
designers and those planning grasping tasks. It provides a user with an interactive
environment where he or she can manipulate the degrees of freedom of any given
hand model to form grasps. As contact between the links of the hand and an object
occur, the simulator analyzes the new grasp on the fly and provides the user with
instant feedback concerning the quality of the grasp. This information includes not
only numerical quality measures, but also an indication of the grasp’s weak point
as well as projections of the 6-dimensional space of forces and torques that can be
applied by the grasp.
In addition to its visualization capabilities, the simulator has facilities for
modeling a complete robotic work cell, which is necessary when planning an actual
grasping task. When combined with a real-time, model-based vision system to align
the pose of a simulated object with the pose of an actual object in the workspace,
the simulator can be used to plan, execute, and monitor a complete grasping task.
Examples of this process are presented and serve to demonstrate this application
of the simulator.
Finally, the addition of a module to simulate object dynamics further in-
creases the realism of the simulation. With it, a user can examine the grasp for-
mation process to ensure that a desired grasp can be achieved. Examples in this
section demonstrate the benefit of going beyond a spatial analysis and including
the temporal dimension.
Contents
List of Figures iv
Acknowledgments vii
Chapter 1 Introduction 1
1.1 Why a grasping simulator? . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . 6
Chapter 2 Related Work 8
2.1 High Level Grasp Synthesis . . . . . . . . . . . . . . . . . . . . . . 9
2.2 The Mechanics of Grasping . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Theoretical Bounds on the Necessary and Sufficient Number
of Contacts Required for Form- and Force-Closure . . . . . . 14
2.2.2 Quality Measures . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.3 Contact-level Synthesis Algorithms . . . . . . . . . . . . . . 16
2.2.4 Hand-Level Grasp Synthesis Systems . . . . . . . . . . . . . 17
2.2.5 Optimal Force Distribution . . . . . . . . . . . . . . . . . . 20
2.3 Robot Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
i
Chapter 3 Interacting with GraspIt 22
3.1 Hand Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.1.1 The Hand Library . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 User Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2.1 Modeling a Robotic Work Cell . . . . . . . . . . . . . . . . . 35
3.3 Collision Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4 Contact Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Chapter 4 Grasp Analysis 42
4.1 Contacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.1 Contact Types . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.2 Friction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.2 Collecting contacts into a grasp . . . . . . . . . . . . . . . . . . . . 45
4.2.1 Wrench Basis . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.2.2 The Grasp Map . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 Building the Grasp Wrench Space . . . . . . . . . . . . . . . . . . . 49
4.3.1 Building the Grasp Wrench Space in GraspIt . . . . . . . . . 52
4.4 Quality Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.4.1 Implementation of Quality Measures in GraspIt . . . . . . . 55
4.5 Wrench Space Visualization . . . . . . . . . . . . . . . . . . . . . . 56
4.6 Comparing the Grasps of Two Objects Using Three Different Hands 62
4.6.1 The Mug Object . . . . . . . . . . . . . . . . . . . . . . . . 62
4.6.2 The Telephone Handset . . . . . . . . . . . . . . . . . . . . 62
4.7 Grasp Force Optimization . . . . . . . . . . . . . . . . . . . . . . . 65
4.7.1 The Hand Jacobian . . . . . . . . . . . . . . . . . . . . . . . 69
ii
4.7.2 Optimization of Grasp Forces using Linear Matrix Inequalities 71
4.7.3 Implementation of GFO in GraspIt . . . . . . . . . . . . . . 74
Chapter 5 Planning, Executing, and Monitoring Robotic Grasping
Tasks: An Example Application 78
5.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2 The Vision Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2.1 Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.2.2 Normal Displacement and Affine Motion . . . . . . . . . . . 82
5.2.3 3D Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.3 Planning the Grasp . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.4 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 85
5.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . 85
5.4.2 Results: Pose estimation and grasp generation . . . . . . . . 86
5.4.3 Results: Visual servoing for grasp monitoring . . . . . . . . 90
5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Chapter 6 Adding Object Dynamics to the Simulation 95
6.1 Implementation of the Time-Stepping Algorithm in GraspIt . . . . 102
6.2 Simulating Grasp Formation . . . . . . . . . . . . . . . . . . . . . . 106
Chapter 7 Conclusions and Future Work 112
7.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.2 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . 114
iii
List of Figures
1.1 The GraspIt simulator . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1 Grasp Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1 Configuration file for the mug object . . . . . . . . . . . . . . . . . 24
3.2 Portion of the Barrett hand configuration file . . . . . . . . . . . . . 25
3.3 Parallel jaw gripper model . . . . . . . . . . . . . . . . . . . . . . . 26
3.4 Barrett hand model . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.5 DLR hand model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.6 Robonaut hand model . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.7 Rutgers hand model . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.8 The 3D workspace of GraspIt . . . . . . . . . . . . . . . . . . . . . 32
3.9 Translating a hand . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.10 Rotating a hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.11 Rotating a revolute joint . . . . . . . . . . . . . . . . . . . . . . . . 34
3.12 Translating a prismatic joint . . . . . . . . . . . . . . . . . . . . . . 35
3.13 Robotic work cell model . . . . . . . . . . . . . . . . . . . . . . . . 36
3.14 Bounding volume traversal tree . . . . . . . . . . . . . . . . . . . . 37
iv
3.15 Collision detection and contact location process . . . . . . . . . . . 39
3.16 Possible contact regions between two triangles . . . . . . . . . . . . 40
4.1 Friction cone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Friction cone approximation . . . . . . . . . . . . . . . . . . . . . . 49
4.3 GWS projection dialog box . . . . . . . . . . . . . . . . . . . . . . 56
4.4 Force-closure grasp of the mug using the Barrett hand (top view) . 57
4.5 Force-closure grasp of the mug (side view) . . . . . . . . . . . . . . 58
4.6 Force-closure grasp of the mug using rubber finger surfaces instead
of plastic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.7 Force-closure grasp of the mug using the DLR hand . . . . . . . . . 63
4.8 Force-closure grasp of the mug using a parallel jaw gripper . . . . . 64
4.9 Force-closure grasp of the phone handset using the Barrett hand . . 66
4.10 Force-closure grasp of the phone handset using the DLR hand . . . 67
4.11 Force-closure grasp of the phone handset using the parallel jaw gripper 68
4.12 Fingertip grasp of the phone handset using the Barrett hand . . . . 76
4.13 Optimized contact forces . . . . . . . . . . . . . . . . . . . . . . . . 77
5.1 Block diagram of the grasping system . . . . . . . . . . . . . . . . . 80
5.2 Tracking system diagram . . . . . . . . . . . . . . . . . . . . . . . . 81
5.3 Key components of the visual tracking system . . . . . . . . . . . . 82
5.4 Successfully planned and executed grasp of an L-shaped object . . . 87
5.5 Second successfully planned and executed grasp of an L-shaped object 88
5.6 Successfully planned and executed grasp of a coffee mug object . . . 89
5.7 Planned grasp of the L-shaped object within GraspIt . . . . . . . . 90
v
5.8 Planning the object motion in GraspIt . . . . . . . . . . . . . . . . 92
5.9 The object at the start of the visual servoing process . . . . . . . . 93
5.10 The object at its destination pose . . . . . . . . . . . . . . . . . . . 94
6.1 Initial position of the Barrett hand and handset for the first dynamics
experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.2 Auto-grasp of the static handset . . . . . . . . . . . . . . . . . . . . 108
6.3 Auto-grasp of the dynamic handset . . . . . . . . . . . . . . . . . . 109
6.4 Initial position of the Barrett hand and handset for the second dy-
namics experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.5 Second auto-grasp of the static handset . . . . . . . . . . . . . . . . 110
6.6 Second auto-grasp of the dynamic handset . . . . . . . . . . . . . . 111
vi
Acknowledgments
First off, I would like to thank my dissertation committee—Peter Allen, John
Kender, Andrew Laine, Vijay Modi, and Shree Nayar—for their participation in
my defense and for their helpful comments and suggestions. I give special thanks
to my advisor, Peter Allen, for all of his generous support and guidance over the
years. He took me in when I was fresh out of Hamilton and had never even heard
of a Jacobian, and thanks to his dedication and constant encouragement, he trans-
formed me into a teacher, a robotics researcher, and a major Yankee fan.
It has been a long process, and many other people have helped me along the
way. I would like to thank:
• Professor Gerd Hirzinger and Dr. Max Fischer, Professor Constantinos Mav-
roidis and Kathryn DeLaurentis, and Dr. Myron Diftler for their interest in
my work and for providing me with models of their robotic hands,
• Jeff Trinkle for sharing his knowledge of grasping, for all of the dynamics
discussions, and for not crushing me completely at pool,
• Danica Kragic for pushing me to work hard and for all of her help in imple-
menting the vision system,
vii
• Paul Oh for his help with the early Barrett hand experiments, the discussions
over Chinese buffet, and for all the good times we had at the conferences,
• the other members of the Robotics group—Ioannis Stamos, Michael Reed,
Atanas Georgiev, Paul Blaer, and Ethan Gold—for their insight and support
throughout all the group meetings, demos, and late nights spent together in
the lab,
• and Aya Aner for her love and support and the wonderful times we shared.
Finally, I offer my heartfelt gratitude to all of my other friends and to my family.
I really couldn’t have done this without you.
viii
To Mom and Dad.
ix
1
Chapter 1
Introduction
Grasping is one of the primary ways for a robot to interact with objects in its
environment. Assembly line robots use simple end-effectors such as suction cups
or parallel jaw grippers to accomplish pick and place tasks with a high degree of
reliability. However, the factory environment is highly structured, and each robot
usually only grasps one type of object until it is reprogrammed to handle another
object type. This reprogramming is done simply using operator knowledge or more
recently using software to automatically pick the contact points [Smith et al., 1999].
As robots are being required to work in more unstructured environments, such as
service robots in the home, bomb disposal robots, and robots used in space missions,
more general purpose hands are being designed that are capable of grasping and
manipulating a much wider range of objects.
2
1.1 Why a grasping simulator?
The flexibility of these hands is achieved with additional degrees of freedom, but at
the cost of additional complexity when deciding which grasp is most appropriate
for the task. Given multiple possible grasps of an object, it would be desirable
to analytically determine which is more suitable for the task rather than rely on
operator intuition. Several authors have examined this problem from a theoretical
point of view and have proposed a variety of methods to quantify the quality of a
particular grasp with respect to different criteria. These methods generally exam-
ine the placement of contacts on the surface of the grasped object, and construct
the space of forces and torques that can be imparted to it through the unilateral
contacts. However, an operator wishing to plan a grasp cannot be expected to
perform several grasps of an object with a physical robotic hand and measure the
position of each contact, as this would be a very time consuming process. Instead,
by using a real-time simulation system that includes models of the robotic hand
and the object to be grasped, the operator could test several grasp configurations
in a short amount of time and let the system determine the points of contact and
quality of each grasp.
Designers of robotic hands could also benefit from such a simulation and
analysis system. Currently, designing a robotic hand is a difficult task with many
considerations such as task requirements, mechanism complexity, and physical size
and weight. Often a physical prototype is necessary to truly test a hand’s ability
to perform tasks, but this can be quite costly and design changes are not easy to
make. By using a simulation system, the designer can quickly see how different
choices of kinematic parameters and link geometries affect the hand’s ability to
3
grasp different objects. Simulation has greatly reduced the design cycle time for
countless other mechanical devices from automobile engines to airplane wings; it
seems only natural to apply it to robotic hands.
This thesis presents our work in this area and describes our system called
“GraspIt!” (shown in figure 1.1), which is a unique and powerful tool for grasping
analysis. It is an interactive simulator that allows the user to import an articulated
robotic hand model and an object to be grasped, and then either by individually
manipulating the hand’s degrees of freedom or by initiating an automated grasp
routine, the operator can perform grasps of the object. A real-time collision detec-
tion system determines the points of contact between the links of the hand and the
object, and as each new contact is formed, the system provides the results of its
analyses on the fly.
Another important benefit of our system is its unique method for visualizing
these results. In addition to providing two numeric measures of a grasp’s ability
to resist disturbance forces, it also graphically shows which force-torque combina-
tion would be most difficult for the grasp to resist. By understanding where the
grasp is weak, a user can incrementally improve the quality of the grasp. GraspIt
also has the capability of creating 3-dimensional projections of the 6-dimensional
force-torque space used in the quality computations, which give the user further
information about the characteristics of a particular grasp.
Beyond its visualization tools, GraspIt also has facilities for modeling the
entire robot work cell. The hand of interest can be mounted on a Puma type
arm model, and a workbench or other environment objects can be added to the
simulation so that the user can test reachability constraints when planning the
4
Figure 1.1: The DLR (German Aerospace Center) hand grasping an Erlenmeyerflask within the GraspIt simulator.
5
grasping task. With the addition of a dynamics module, we also allow the user to
examine the grasp formation process, since it is unrealistic to assume that all of
the desired contact points can be reached simultaneously. Finally, once the grasp
is formed, and if a desired task force can be specified, GraspIt can determine the
necessary joint torques (or indicate that it is impossible) to keep the object in
equilibrium while obeying torque limits and friction constraints. By consistently
using accurate models of all elements involved in the task, we ensure that grasps
planned in simulation can be carried out in the real world.
Because GraspIt has applications in both grasp planning and robotic hand
design, several research groups are already using it or have expressed interest in
using it for their work. The Robonaut group at NASA Johnson Space Center is
testing their robotic hand and examining various grasps of tools that will be used
in space missions. Researchers in the Robotics and Mechatronics group at Rutgers
University are building a robotic hand actuated with shape memory alloys, and
they are using GraspIt to test different designs of the hand. A group from the
Biomedical Engineering Department also at Rutgers, is interested in using GraspIt
to aid in the design of a prosthetic hand device that they have previously only
designed using several revisions of mechanical prototypes. In addition, researchers
from the Space Systems Laboratory at the University of Maryland would like to use
GraspIt to help design their own robotic hand for use in EVA space missions. The
reason for the success of GraspIt is that it is the first general grasping simulation
system to incorporate grasping analysis theory into a interactive 3D environment
that allows a user to visualize the grasping capability of virtually any robotic hand
design.
6
1.2 Organization of the Thesis
Having introduced GraspIt, we now describe the remaining contents of this thesis.
In chapter 2, we review the most relevant works in the large body of grasping
research. Many of these papers are theoretical in nature, but on the software side
some researchers have implemented hand-specific grasp synthesis systems as well
as general robot simulators, which we will also review.
Chapter 3 presents the front-end design of the GraspIt simulator. The flex-
ibility of the system is demonstrated by describing how various hand designs can
be imported, and our current library of five existing robotic hands is presented.
The intuitive and visual user interface, which is a primary reason for the simula-
tor’s success as a tool for grasping, is also described along with examples of user
interaction. Finally, we present the collision detection and contact location sys-
tems that determine the contacts between the links of the hand and objects in the
environment as they occur in real-time.
Chapter 4 presents the grasping analysis routines that make up the back-end
of the simulator. The chapter begins by reviewing the mechanics of contact and
friction, and then presents an overview of the relevant pieces of grasping theory. To
demonstrate how GraspIt allows a user to visualize the results of the analyses per-
formed, we provide several example grasps and compare the grasps of two different
objects using three different hands. The work presented in chapters 3 and 4 is also
described in condensed form in [Miller and Allen, 2000].
Grasps planned in simulation are of no use if they cannot be executed by
an actual robot. Chapter 5 shows how GraspIt can be integrated with a real-time
vision system to plan and execute a grasping task. Using a single camera to monitor
7
the workspace, the user begins the process by establishing a few correspondences
between a wire-frame model of the object to be grasped and its image in the scene.
An optimization routine accurately estimates the object’s pose in relation to the
robot and sends this information to GraspIt where the user can manipulate the
robot and plan the grasp. Once a grasp is chosen, the user can execute it directly
from the simulator, and then plan a trajectory for the object by specifying a series
of way points. After the grasp is completed, a model-based visual servoing system
directs the hand and object to the points in the trajectory. After describing this
process in more detail, we present the results of grasps performed on different test
objects and discuss ways that the system can be extended.
In chapter 6, we describe the dynamics module of GraspIt. We begin by
reviewing some of the previous dynamic simulations systems, and discuss differences
in the various methods. We then give a detailed description of our modified version
of the Stewart-Trinkle time stepping algorithm [Stewart and Trinkle, 2000] and
provide examples of the dynamic interaction between a robotic hand and an object
as the hand forms a grasp.
Finally, chapter 7 provides a summary of this work and details our plans for
further research.
8
Chapter 2
Related Work
Because of the versatility of the human hand, there has been a great deal of research
devoted to studying the hand and to building robotic hands that can mimic some
of its functionality. Two large areas of research that are beyond the scope of this
thesis are haptics, which provides robots with the sense of touch, and dextrous
manipulation, which plans the motions of the fingers to move a part within a
grasp. Instead, this thesis focuses on grasp analysis and grasp synthesis as well as
simulation, and this chapter reviews the significant body of work in these areas. A
clear division of approach in grasping was illustrated by Mishra and Silver [1989]
who separated the previous work into higher level physiological studies of the human
hand and lower level studies of the mechanics of contact. While this thesis is
primarily concerned with simulating and analyzing the grasping mechanics, the
synthesis techniques of the higher level approach cannot be ignored. This chapter
begins by reviewing these techniques, and then it presents the large body of previous
work concerned with the mechanics of grasping which is further divided into several
sub-areas. Finally, the chapter concludes by reviewing research in the area of
9
general robotic simulation.
2.1 High Level Grasp Synthesis
One difficulty in synthesizing grasps is the large number of degrees of freedom
possessed by typical robotic hands. This flexibility gives rise to an enormous set
of possible hand configurations. However, in choosing their own grasps, humans
unconsciously simplify the task to selecting one of only a few different prehensile
postures appropriate for the object and for the task to be performed. Medical
literature has attempted to classify these postures into grasp taxonomies as seen
in [McBride, 1942], [Griffiths, 1943], and [Slocum and Pratt, 1946], but the most
well known grasp taxonomy was proposed by Napier [1956]. His work establishes
the fundamental differences between power grips, which envelop the object and
offer maximum stability, and precision grips, which have contacts primarily on the
fingertips and offer maximum manipulability. Cutkosky and Wright [1986] extended
this classification to the types of grips needed in a manufacturing environment and
examined how the task and object geometry affect the choice of grasp. Their tree-
like classification can be seen in figure 2.1.
Iberall [1987], [1997] saw this classification as too rigid, since in practice,
the human hand often uses a combination of grasps to accomplish a task. She
defined grasps with respect to two virtual fingers which apply opposing forces on
the objects, and only later maps these virtual fingers onto physical fingers based
on object characteristics. The basic oppositions of the virtual fingers are divided
into pad opposition, which is between the thumb and finger pads and used for
precision type grasps, palm opposition, which is between the palm and the digits
10
Figure 2.1: Cutkosky and Wright’s [1986] taxonomy of human grasps.
11
and used for power type grasps, and side opposition, which is between the thumb
and the side of the index finger and is a compromise between the flexibility of the
pad opposition grasps and the stability of the palm opposition grasps. Lyons [1985]
uses the concept of the virtual fingers in his development of a grasp index that
selects a grasp on the basis of two object characteristics, shape and size, whether
the grasp should be firm or not and whether the grasp should be precise or not.
Unfortunately, his categories are quite broad and make it difficult to create a grasp
tailored to specific objects.
Stansfield [1991] built these classifications into a rule based system that,
when given a simplified object description from a vision subsystem, will provide
a set of possible hand preshapes and reach directions for the pre-contact stage
of grasping. However, many problems are left unsolved. She only examines five
possible approach directions, she does not try to choose the best grasp from this
set of possibilities, and for any grasp that is chosen, the hand simply closes its
fingers; no attempt is made to optimize the grasp for stability.
Pao and Speeter [1989] developed a method of solving for a transform that
maps human hand poses to poses of the robotic hand, and using a DataGlove to
measure the joint angles of a human hand, they were able to recreate a variety of
poses with the robotic hand. Speeter [1991] later created HPL, Hand Programming
Language, that simplifies the problem of coding robotic grasps and dextrous ma-
nipulation tasks. The language consists of a number of motion primitives that are
related to common human grasps and manipulation motions, providing a high-level
abstraction of the grasping process.
12
2.2 The Mechanics of Grasping
Grasping has also been studied from a mechanical point of view that analyzes how
different configurations of contacts, and how friction occurring at those contacts,
affect the space of forces and torques that can be applied to an object. Additionally,
some of this research has taken into account the kinematics of the hand when
determining what sets of contacts are possible and what forces can be applied at
those contacts.
Before we begin the discussion of these papers, we provide definitions of
three important, independent properties of grasps that have been defined in the
literature:
Equilibrium A grasp is in equilibrium if the sum of the forces and torques acting
on the object is zero.
Force-closure A force-closure grasp can resist all object motions provided that
the end-effector can apply sufficiently large forces at the unilateral contacts.
Manipulable A manipulable grasp can impart arbitrary object velocities without
breaking contact provided that the end-effector can apply sufficiently large
forces at the unilateral contacts.
In some works the term form-closure has been used in place of force-closure and
this has led to some confusion within the community. Form-closure is an older
term borrowed from the mechanics community that does not consider an end-
effector mechanism and only examines the placement of contacts on the surface of
the object. Within this thesis it is defined as follows:
13
Form-closure A form-closure grasp can resist all object motions by relying only
on unilateral, frictionless contacts.
Much of the foundation for the current research in this area was laid out by
Hanafusa and Asada [1982] and by Salisbury [1982]. Hanafusa and Asada developed
a method to compute stable equilibrium grasps in two-dimensions using a three-
fingered gripper that had each finger connected to a motor using a spring. Salisbury
examined the grasping problem from a much more general sense. In his thesis he
classified types of contacts between two bodies, and provided an approach that
accounts for uni-sense contact wrenches1 in determining whether a grasp completely
restrains an object. He also derived the grip transformation (also known as the grasp
map) which relates forces at the contacts points to overall object forces. Finally, he
provided a thorough analysis of the desirable kinematic properties of an articulated
robotic hand which resulted in the creation of what is currently known as the
Salisbury robotic hand. The research that followed these works expanded on their
ideas and developed new methods to synthesize grasps with desirable properties and
new measures to analyze the quality of a given grasp. Given the quantity of relevant
work, they have been divided into the following groups: theoretical bounds on the
necessary and sufficient number of contacts required for form- and force-closure,
quality measures used in grasp analysis, contact-level grasp synthesis algorithms,
hand-level grasp synthesis systems, and methods of computing an optimal grasp-
force distribution.
1Simply put, a wrench can be thought of as a vector that combines a force and a torque. Awrench acting on a 2D object has three values: the amount of force acting in the x and y directionswithin the plane of the object and the amount of torque acting about an axis perpendicular tothe object. A wrench acting on a 3D object has six values. Wrenches are described further inchapter 4, but for an in-depth treatment of wrenches (and their analog in the velocity domain,screws) see [Hunt, 1978].
14
2.2.1 Theoretical Bounds on the Necessary and Sufficient
Number of Contacts Required for Form- and Force-
Closure
Perhaps the earliest contribution to the modern field of grasping can be traced to
Reuleaux [1876] who showed that a minimum of four frictionless point contacts are
required to completely restrain a planar object against arbitrary disturbance forces
acting within that plane. When considering 3-dimensional objects and disturbance
forces, Somoff [1900] and later Lakshminarayana [1978] showed that a minimum
of seven frictionless point contacts are necessary for form-closure. Markenscoff et
al. [1990] proved that four contacts points are sufficient for 2-dimensional form clo-
sure, and for a large class of the non-rotationally symmetric 3-dimensional objects,
seven contact points are sufficient for form-closure. They also proved that in the
presence of friction, three and four contacts are sufficient for force-closure of two
and three dimensional objects respectively. These frictional bounds were lowered
by one contact each by Mirtich and Canny [1994] who assumed rounded finger tips
to provide continuity to the contact normals around the boundary of the object.
When searching for force-closure grasps, the proofs in these papers allow one to
ignore any grasps which have fewer than the required number of contacts, and the
methods used in the proofs of the upper bounds give rise to algorithms which can
find force closure grasps with a minimal number of contacts. This is not considered
useful for the construction of secure grasps but may be useful in finding highly
manipulable grasps.
15
2.2.2 Quality Measures
Quality measures provide a way of ranking grasps beyond the binary identification
of their properties. While measures for several aspects of a grasp exist, the papers
reviewed in this section have examined either the security of force-closure grasps or
the dexterity of a manipulable grasp.
Using a linear program, Trinkle [1992] not only tested for what he termed
frictional and frictionless form-closure, but also provided a quality measure of “how
’far’ a grasp is from losing form closure.” He also showed a method to test for strong
force-closure (grasps which require an external force to maintain equilibrium) which
Lakshminarayana called partial restraint. In an examination of 3- and 4-contact
grasps, Markenscoff and Papadimitriou [1989] proposed an optimality criteria which
seeks to minimize the sum magnitude of the contact forces necessary to counter
the worst-case unit disturbance wrench. Kirkpatrick et al. [1990] generalized this
measure for n-contacts by defining the quality as the radius of the largest wrench
space ball which just fits within the unit grasp wrench space. After developing
this general measure further, Ferrari and Canny [1992] proposed another measure
that gauges the largest disturbance wrench given a limited maximum contact force.
Earlier, Li and Sastry [1988] noted that similar measures are not invariant to the
choice of torque origin, and propose using the volume of the grasp wrench space
as an invariant quality measure. They also develop a quality measure using task
ellipsoids to better model the space of wrenches required for a particular task,
whereas the previously mentioned authors all implicitly assume the task is unknown
by using the wrench space ball (the task wrenches are of equal magnitude in every
direction).
16
Li et al. [1989] also examined the manipulability of a grasp. They designed
a manipulability metric that is the size of the largest twist space ball (or task
ellipsoid) that can be accommodated by a unit ball in the finger joint velocity
space. Then they defined an overall performance measure that used a weighted
combination of the wrench space and twist space measures. The weights could be
adjusted depending on whether the task required stability or manipulability.
2.2.3 Contact-level Synthesis Algorithms
The works in this section present techniques for finding force-closure grasps, that
are optimal with respect to some chosen criteria. In general, this can only be
accomplished by constraining the geometry of the object to be grasped and/or the
number of contacts used to perform the grasp.
The first two papers find grasps whose contacts are maximally separated
in order to efficiently resist externally induced torques. Chen and Burdick [1992]
showed a system that finds maximal antipodal force-closure grasps (grasps with
two opposing contacts separated by a maximum distance) of 2- and 3-dimensional
irregularly shaped objects. Mirtich and Canny [1994] also provided an algorithm
to find optimal grasps of three contacts on any polyhedron.
Markenscoff and Papadimitriou [1989] gave an algorithm for computing 3-
contact, optimal, stable grasps of polygons, where they defined stable as the ability
to resist, through friction, the weight (a force along the third dimension) of an
object whose center of gravity can be anywhere within the polygon. They also
gave an algorithm for computing 4-contact, optimal, form semi-closure grasps of
polygons, where they defined form semi-closure as the ability to resist externally
17
applied forces which are directed through the center of gravity of the object. Ding
et al. [2000] tried to solve the more general case of finding an optimal n-contact
form-closure grasp of a polyhedron given that m < n of the contacts had already
been specified.
Another possible optimality criteria maximizes the size of the independent
contact regions which maintain form closure as long as one contact is placed in
each region. Optimal grasps will therefore be robust to contact positioning er-
rors, but they are not necessarily optimal with respect to their ease of resisting
disturbance wrenches. Nguyen [1986] outlined an algorithm that finds every set
of form-closure contact regions on the surface of 3-dimensional polyhedral objects.
Ponce et al. [1993b] present an linear programming method to find 3- or 4-contact,
form-closure grasps with maximal contact regions on polyhedral objects. With an-
other group, Ponce et al. [1993a] developed a system to compute robust 2-contact,
form-closure grasps of curved 2-dimensional objects represented as collections of
polynomial parametric curves. They also showed results from simulations and a
real system that used a vision subsystem and a PUMA robot equipped with a
parallel jaw gripper.
2.2.4 Hand-Level Grasp Synthesis Systems
The papers in the previous section focused on finding the optimal placement of a
given number of contacts on the surface of an object without considering whether
an actual gripper mechanism could reach those points. The authors in this section
presented systems that designed grips for a given hand.
In one of the earliest works in this area, Jameson [1985], performed local
18
grasp optimization to minimize the coefficient of friction necessary to lift an object
using a three-contact grasp. Since he considered a physical hand and not just the
placement of contact points, he added potential fields to guide the search away from
physical constraints. This meant keeping the contacts away from the ends of the
fingers, keeping the joints within their limits, and keeping the fingers away from
the table.
Smith et al. [1999] developed a polynomial time algorithm for computing
high-quality parallel-jaw grips on an n-sided polygonal slice through the center of
gravity of the part to be grasped. The algorithm took into account robustness (how
close the grip points are to convex vertices), reliance on friction (grip axis should
be closely aligned to contact normals), amount of frictional torque required (grip
axis should pass close to the center of gravity), and the accessibility of the contact
points (the points should be reachable from infinity along the grip axis).
Borst et al. [1999] implemented a precision-grasp planner for their 4-fingered
DLR hand. The system generates candidate grasps by first choosing a starting
point on the object and then finding three other contacts that create a modified
tetrahedral grasp of the object. If the grasp cannot be realized by the hand, it is
thrown out and another is chosen. If the grasp can be accomplished, they check the
robustness of the grasp by examining how close the contacts are to object edges.
If necessary, they perturb the contact points away from these edges. The grasp
is then evaluated using a force-closure quality measure described by Ferrari and
Canny, and after evaluating 100 reachable grasps, they choose the best one.
In designing a precision-grasp synthesis system for NASA’s 14 degree-of-
freedom Robonaut hand, Hester et al. [1999] tried to reduce the size of the 20
19
dimensional search space of all hand configurations. First they assumed that the
wrist position was fixed, immediately eliminating six parameters, and then they
independently optimized fingertip locations with respect to a weighted sum of their
force and velocity transmission ratios. This preliminary grasp served as the starting
point for a local optimization of the grasp quality considered as a whole.
Nancy Pollard [1994] designed a system that found grasps that were within
a certain percentage of the quality of a given prototype grasp. By accepting the
reduced quality, the contact points of the prototype grasp can be grown into contact
regions. These regions can then be projected onto the novel target object, and using
a parallel algorithm, the system optimizes the contact placement independently
within each region. Note that these contact regions are similar to the robust regions
found by Nguyen and Ponce et al., because if one contact is made in each region,
force-closure is guaranteed. However, they also guarantee the grasp of the target
object will have an overall quality that is at least as large as the reduced prototype
quality used to create the regions in the first place. Another feature of her system
is the constraint that the contacts must be kinematically feasible and cannot result
in any collision between the hand and other obstacles.
By considering more than just fingertip grasps, Pollard’s system can be con-
sidered one of the more general grasp synthesis tools available, but it has a few
difficulties. While the prototypes allow her to greatly reduce the complexity of
the search, a system to choose the closest prototype grasp is not given, and she
admits that “the problem of friction is dealt with in an ad-hoc way”. Her system
also roughly approximates the link geometries as a collection of spheres to greatly
simplify the contact analysis. Lastly, it required a Cray supercomputer to compute
20
the grasps.
2.2.5 Optimal Force Distribution
The last group of papers in this section assumed that a grasp had already been
chosen for an object, and they concentrated on the problem of finding the opti-
mal contact forces, subject to various constraints, that would keep the grasp in
equilibrium.
Kerr and Roth [1986] formulated the task of choosing the internal grasping
forces as a linear programming problem. Using linearized friction cones, the algo-
rithm found a grasp force vector that maximized the minimum distance from each
contact force to the surrounding friction cone planes and stayed within the torque
limits of the fingers applying the force.
Buss et al. [1996] showed that friction cone constraints can be rewritten
as positive-definiteness constraints of particular matrices, making it unnecessary to
approximate the friction cone with a linearized model. They then used gradient-flow
techniques to perform the optimization, but they required an admissible starting
point which was a significant drawback to their technique.
Han and Trinkle [2000] discovered that the same friction constraint matrices
could be built into a linear matrix inequality that could first be solved for a feasible
solution to the grasp force equilibrium problem and then could be used to find an
optimal solution. This technique has been implemented within GraspIt and will be
discussed further in chapter 4.
21
2.3 Robot Simulation
Simulation has long been recognized as a valuable tool for designing robots and
planning their motions. Stephen Derby [1983] built a robotic arm simulator called
GRASP that allowed designers to evaluate the performance of robot arms in poten-
tial working environments and attempt to minimize the travel time for a given task.
The operator would describe an arm design kinematically and then plan the robot
motions by using a set of primitives that correspond to velocity and acceleration
profiles. Then the simulator could execute the plan and show plots of the motion
profiles.
Cremer and Stewart [1989] developed a primitive simulator called Newton
that allowed a user to create linked mechanisms and simulate their dynamics within
an environment with other obstacles. It included a collision detection system, a
rudimentary impact resolution scheme, and a method for dealing with continuous
contacts. In addition, the object geometry was quite simple, consisting only of
blocks and spheres.
The commercially available ADAMS package [Mechanical Dynamics, 2001]
takes these ideas much further and is now a professional tool for mechanism design
and simulation. While they have modules that deal with specialized domains such as
the automotive industry and aerospace, they do not have any facilities for studying
grasping.
Ruspini and Khatib’s Simpact [1999], simulates the dynamics of articulated
robotic systems. The focus of their work has not been on grasping or mechanism
design, but rather on finding the best way to simulate the dynamics of constrained
bodies.
22
Chapter 3
Interacting with GraspIt
What makes GraspIt a powerful tool is its interactive nature. All of the analysis
routines described in the next chapter could have been performed using the models
of the hand and object and a file specifying the joint positions and wrist pose.
However, the user interface of GraspIt allows a user to manipulate the hand within
a 3D workspace to form different grasps of an object and then he can see the results
of the analyses generated on the fly for each change in the grasp. Using this visual
feedback, the user can quickly see problems with a grasp and iteratively work to
improve the quality of the grasp. A hand designer also gets a much better sense of
the capabilities of a particular hand design by directly manipulating its degrees of
freedom and observing any limitations of the hand.
In this chapter, we describe the elements of GraspIt’s front-end. We begin by
describing our general method of hand and object specification that allows a user
to import a wide range of hand designs and objects. We also present our current
library of robotic hand models which we have obtained through collaboration with
the research groups building these hands. Next, we describe and provide examples
23
of the intuitive methods of interaction that allow a user to manipulate the hand and
form grasps of the object. Since the user is allowed to freely manipulate all of the
degrees of freedom of a hand, the system must prevent the links of the hand from
passing through the object. Section 3.3 describes the real-time collision detection
system that detects when a link has penetrated the object and adjusts the joint
positions so that the link is within a small (contact threshold) distance away from
the object surface. Finally, section 3.4 describes our method of locating the regions
of contact between two bodies and shows how these areas are indicated visually.
3.1 Hand Construction
We have designed the simulator to be as general as possible so that it is useful for a
variety of hands and objects. However, it was necessary to create standard formats
for the definitions of a hand and an object, so that they can be understood by the
system.
An object configuration file is fairly simple, and consists only of a material
specification and mass properties that are used in the dynamic simulation. In
addition, the name of the object associates it with a CAD model file. Currently the
surface of the object is assumed to be composed entirely of one type of material to
ease the static friction computations.
The hand description file similarly allows us to describe the link geometries as
complete 3D entities, each with an associated material specification. Furthermore,
it fully describes the kinematics of the hand using standard Denavit-Hartenberg pa-
rameters1 with additional information regarding joint limits and finger base trans-
1The Denavit-Hartenberg parameters are joint angle, joint distance, link length, and link twist
24
#material
glass
#mass in g
300
#COG
0 0 0
#Inertia matrix
# Ixx -Ixy -Ixz
#-Ixy Iyy -Iyz
#-Ixz -Iyz Izz
4.753E+5 -5.132E+3 0.0
-5.132E+3 5.732E+5 0.0
0.0 0.0 5.239E+5
Figure 3.1: The configuration file for the mug object. Coordinate axes are placedat the center of mass of the mug.
forms. Each joint, which is either revolute or prismatic, has its free parameter con-
nected to one of the hand’s degree of freedom variables either directly or through
a linear function. This allows for coupled joints like the passively controlled distal
joint found in many anthropomorphic finger designs. We have found our description
method is flexible enough to construct hands as simple as a parallel jaw gripper or
as complicated as many of the fully articulated hands in use today.
After reading the description file, GraspIt reads each of the link model files
and sets the initial transforms based on the kinematics. Once the hand construction
is completed, it turns control over to the user.
angle. The joint angle is variable in a revolute joint, and the joint distance is variable in aprismatic joint.
25
#Palm Description (considered a link)#filename material lastJoint#(lastJoint is -1 since the palm is not affected by any joint)palm plastic -1
#DOF4
#DOF velocities for auto grasp0.00.050.050.05
#Number of fingers3
#-----------------------------------f0----------------------------------------#number of joints3
#number of links3
#Transforms from palm origin to base of the finger#EXAMPLE:#r 90 x (rotation by 90 degrees about x axis)#rr 3.0368729 y (rotation by 3.0368729 radians about y axis)#t 1.92188 134.5 39.0717 (translation in mm)t 25.0 0 -1.0r 180 y
#Joint Descriptions (1 joint per line)#DOFNum theta d a alpha DOFminval DOFmaxval#(joints are ordered from closest to palm outward)#(linear equations are of the form: d#*k+c [no spaces!])#(coupled joints must come after joints they are coupled to)d0+90 0 50 -90 0 180d1+5 0 70 0 0 144d1*0.333333333+40 0 55 0 0 144
#Link Descriptions (1 link per line)#filename material lastJoint#(links are ordered from closest to palm outward)#(lastJoint is the last joint in the chain which can affect this link)link1 plastic 0link2 plastic 1link3 plastic 2
Figure 3.2: Description of the palm and thumb from the Barrett hand configurationfile.
26
Figure 3.3: A parallel jaw gripper model.
3.1.1 The Hand Library
As described above, the ability to easily import new hand designs is a key benefit
of our system. Through collaboration with other research sites we have obtained
CAD models and kinematic descriptions of four different articulated hand designs,
which have now become part of our library of hand designs. In this section we
briefly describe the four hands as well as a simple parallel jaw gripper design we
created ourselves.
A Parallel Jaw Gripper
Our gripper model has the approximate dimensions of an actual Lord gripper. For
ease of use, we changed the kinematics of our gripper so that each plate can be
independently controlled. When only one coupled DOF is used to grip a non-
moving object, the gripper must be centered over the object exactly for both plates
to come in contact. Besides the kinematics, we also specified the surface material
of the palm as metal and the plates as rubber. In some examples, we have used a
27
Figure 3.4: The Barrett hand model.
larger version of this gripper. To make this change is a simple matter of changing
the joint limits in the kinematic description, and modifying the geometry file.
The Barrett Hand
The Barrett hand, produced by Barrett Technology, is based on an innovative
design developed at the University of Pennsylvania [Ulrich et al., 1988]. It is an
eight-axis, three-fingered mechanical hand with each finger having two joints. One
finger is stationary and the other two can spread synchronously up to 180 degrees
about the palm (finger 3 is stationary and fingers 1 and 2 rotate about the palm).
Although there are eight axes, the hand is controlled by four motors. Each of the
three fingers has one actuated proximal link, and a coupled distal link that moves
at a fixed rate with the proximal link. A novel clutch mechanism allows the distal
link to continue to move if the proximal link’s motion is obstructed (referred to as
breakaway). An additional motor controls the synchronous spread of the two fingers
28
Figure 3.5: The DLR hand model.
about the palm. This gives the Barrett hand only 4 internal degrees of freedom: 1
for the spread angle of the fingers, and 3 for the angles of the proximal links. The
links are constructed of high density plastic.
The DLR Hand
The DLR hand [Butterfass et al., 1998], developed at the German Aerospace Cen-
ter, is a four-fingered articulated robotic hand, and unlike the Barrett hand, the
placement of the fingers resembles the human hand structure. However, because
the joint motors are all internally contained, the hand is about 1.5 times the size of
an average human hand. The fingers are identical, and each consists of three links
with two joints at the base, one joint between the proximal and medial links, and
one joint between the medial and distal links. This last joint is coupled in a fixed
ratio to the previous joint in the chain just as it is in the human hand. The DLR
hand has a total of 12 internal degrees of freedom since there are 3 independently
29
Figure 3.6: The Robonaut hand model.
controllable joints in each of the 4 fingers. In the description file for the DLR hand,
we specify its kinematics, including joint limits, and its link materials (metal for
the palm, proximal, and medial links, rubber for distal links). Also specified are
the link models, which are accurate replicas of the actual hand, except we have
omitted unnecessary details at the joints.
The Robonaut Hand
Next, we examine the Robonaut hand which was developed at NASA’s Johnson
Space Center [Lovchik and Diftler, 1999]. This hand has 5 fingers and a total of
14 internal degrees of freedom (see figure 3.6), and it is equivalent in size to a 95th
percentile human male hand. The index and middle fingers along with the thumb
are considered the primary manipulation fingers and are capable of abduction and
adduction. The ring and pinkie are mounted on a squeezing palm joint which make
them good grasping fingers capable of wrapping around a tool or other object.
30
Figure 3.7: The Rutgers hand model.
Although the link geometry only specifies the metal structure of the hand, we
assumed rubber contact surfaces to facilitate stronger grasps.
The Rutgers Hand
The Rutgers Hand is being developed at the Mechanical and Aerospace Engineering
Department at Rutgers University [DeLaurentis et al., 2000]. While this hand is
also a 5 fingered anthropomorphic design, its most novel aspect is the use of shape
memory alloys to actuate the joints. Shape memory alloys are very compact and
lightweight and provide a method of actuation similar to human muscle fibers which
contract when excited. Although one prototype finger has been built, the hand is
still in the design phase, and the research team is using GraspIt to examine how
changes in the kinematic structure of the hand affect the hand’s ability to apply
the contact forces necessary for different tasks. The four fingers each have 3 links
and the thumb has 2, and the base of each finger is connected to the palm with a
31
ball and socket joint that has 2 independently controllable degrees of freedom. The
simulation model allows each joint to be individually controlled, even though the
final model will likely have fewer degrees of freedom. This gives us a current total
of 19 degrees of freedom: 4 for each of the 4 fingers and 3 for the thumb.
3.2 User Interaction
One of the design goals we believe we achieved was to make the user interface
as intuitive and transparent as possible. We began with a standard Inventor
viewer [Wernecke, 1994], which displays, at the user’s choice, a perspective or or-
thographic projection of a 3D world in a 2D window. By using the hand tool found
in the tool bar on the right side of the window, the user can rotate, pan, or zoom
the virtual camera viewing the scene, allowing him to view portions of the scene
that might be physically impossible to see in a real workspace. The seek tool (the
crosshairs) is especially useful for this, since it will smoothly zoom the camera in
to view whatever detail is clicked on by the user.
When GraspIt is started, the user is presented with an empty space and may
import a hand model and an object model using selections from the custom menu
bar at the top of the window. He may also restore a saved grasp, which loads both
hand and object and places them in whatever configuration they were saved in.
Figure 3.8 shows what the workspace looks like after the Robonaut hand and the
Erlenmeyer flask have been imported.
At this point the user may freely maneuver the hand to a desired position.
To translate the hand, the user simply clicks on the palm and a bounding box
appears around the hand. By dragging a face of the box, the user can translate the
32
Figure 3.8: The 3D workspace of GraspIt after the hand and object models havebeen imported. The grasp quality is listed as “unstable” until a force-closed grasphas been formed.
hand within the plane containing that face (see figure 3.9). To rotate the hand, the
user holds the control key down while clicking the palm. This brings up a virtual
sphere displayed as three intersecting circles that surround the hand and is centered
on the base coordinate frame of the hand (see figure 3.10). Dragging one of the
circles rotates the hand about the perpendicular axis, or by dragging elsewhere on
the sphere, the hand can be freely rotated in 3 dimensions. The sphere may also
be recentered by dragging one of the six crosshairs found at the intersections of the
circles. To remove the translation or rotation manipulator, the user simply clicks
outside of its active region.
Manipulating the individual degrees of freedom of the hand is equally intu-
itive. Clicking on a finger of the hand brings up an interactive manipulator for each
actively controllable joint in the finger. Revolute joints are controlled by dragging
a disk whose axis of rotation is coincident with the joint axis (see figure 3.11), and
33
Figure 3.9: The hand is being translated within the plane containing the front faceof the bounding box.
Figure 3.10: The entire hand can be rotated freely in 3 dimensions by dragging onthe virtual sphere that surrounds the hand.
34
Figure 3.11: The angle of a revolute joint can be changed by dragging a disc ma-nipulator located at the joint. The passive distal joint moves in a fixed relationshipwith the medial joint.
prismatic joints are controlled by dragging an arrow which is aligned with the joint
axis (see figure 3.12). These manipulators obey the joint limits defined in the hand
configuration file and prevent the user from moving beyond them. Passive joints
also move in response to changes in the joints they are coupled with.
Since it can be somewhat tedious to form a grasp by manipulating each joint
of a complex hand, we also provide an auto-grasp feature. When initiated, the
active joints of the hand each close at preset velocities which are specified in the
hand configuration file and stop when a link that follows it in the kinematic chain
contacts an object or another finger link. This is often a fast method to perform an
initial grasp of an object which can then be refined by manipulating the individual
joints.
35
Figure 3.12: The position of a prismatic joint can be changed by dragging a doublesided arrow.
3.2.1 Modeling a Robotic Work Cell
This set of manipulators allows the user to explore all physically realizable grasps
of an object. However, if a grasping task is being planned for a real robot, it is
necessary to model the full robot and its environment which will present additional
constraints on the grasps that can be performed. For use in our experiments in
chapter 5, we have modeled the Puma560 robotic arm, its base, and the workbench
found in our robotics lab (see figure 3.13). It would not be difficult, however,
to model other robot work cells. All that is needed are models of environmental
objects, the links of the robot, and a set of inverse kinematics routines to control
the joints of the robot as the hand is translated or rotated.
36
Figure 3.13: A model of a simple robot work cell can be used with GraspIt to testreachability constraints.
3.3 Collision Detection
To prevent the links of the hand from passing through environmental objects or the
object itself as they move, the system performs real-time collision detection using
the PQP system (Proximity Query Package) developed by Larsen et al. [1999].
When a body is loaded into the simulator, it is faceted, and its collection of triangles
is passed to PQP where they become the leaves of a hierarchical tree of bounding
volumes. In fact, the current implementation creates two trees, one using oriented
bounding boxes (OBB) for use in collision queries, and one using rectangle swept
spheres (RSS2) for use in proximity queries. In each case, the tree is created from
the top down. At the top of the tree (also known as the bounding volume hierarchy,
BVH), the set of triangles that make up the body is bounded with a volume that is
oriented along the principal axes of the set, and it is split with a plane perpendicular
2An RSS is a volume that is created by sweeping a sphere over a rectangular area, and it is amore efficient bounding volume for proximity queries than oriented bounding boxes.
37
Y Z
X
BVH
BY BZ CY CZ
CXBX
AX
BVTT
B C
A
BVH
Figure 3.14: An example bounding volume traversal tree (BVTT) created fromtwo bounding volume hierarchies (BVH). The nodes of the BVTT are the recursivesequence of overlap tests between the bounding volumes of the first object andthe bounding volumes of the second object. The root node of the BVTT testsfor overlap between the bounding volumes of the two objects. If they overlap,the search for overlapping triangles continues in a depth first manner through theBVTT. The two children of the root node are tests between the bounding volume ofthe shallower BVH and the two sub-bounding volumes of the deeper BVH. Anytimea pair of bounding volumes are found to be disjoint, the sub-tree below that nodein the BVTT may be pruned from the search.
to the set’s longest axis. Each of the two subgroups become the children of the
root tree node, and they are each recursively bounded and subdivided until the
individual triangles are reached and cannot be divided further.
To determine if two bodies have collided, the system begins with the two root
nodes, the body bounding boxes, and checks for overlap. If the boxes are disjoint
then there is no collision, otherwise the system examines the first child of whichever
tree is deeper and tests it against the root box from the other tree. The hierarchy
of tests can be described by what Larsen calls the bounding volume traversal tree
(BVTT), and a simple example is shown in figure 3.14. The search for a pair of
overlapping triangles continues in a depth first manner down this tree, and anytime
a pair of boxes is disjoint, the subtree below maybe be pruned from the traversal.
If a collision is detected (see figure 3.15(a)), the motion of the bodies must
be reversed back to the point when the contact first occurs. To find this instant,
38
GraspIt begins by moving the objects to their previous locations before the collision
and calls the PQP system to determine the minimum distance between them. PQP
traverses the two RSS trees in a similar fashion as before, except when it reaches
leaf nodes in both trees, rather than testing for overlap, it computes the minimum
distance between them. If that distance is smaller than the distance between the
closest pair of triangles found so far, it records the new minimum distance and
the points on each triangle that are closest to each other. If during the traversal,
the distance between two bounding volumes is greater than the current minimum
distance, the algorithm does not need to descend the current branch of the BVTT
further.
Typically, the links of the hand move only a few millimeters between updates
during an interactive manipulation, so by moving back to the last update before the
collision, the minimum distance returned by PQP is usually in this range. Since
the wrist transformation and the joint positions must be represented with fixed
precision floating point numbers, it is impossible to search for the wrist position or
joint value that results in exact contact between a link and the object. Thus, we
define a thin contact region around the surface of each body, and if the distance
between two bodies is less than this threshold (currently set at 0.1mm) then they
are considered to be in contact. We use a binary search technique to move the
link of the hand to within this distance of the object after it has collided (see
figure 3.15(b)). When the two bodies inter-penetrate, the PQP minimum distance
query returns zero rather than an inter-penetration distance, which is difficult to
define for non-convex objects. It is possible to use a more efficient search algorithm,
since we are provided with more than just binary information when the objects are
39
(a) (b) (c)
Figure 3.15: The collision detection and contact location process: (a) The collisionof a link of the Barrett hand with the side of the phone is detected. (b) A searchis conducted to find the joint angle that will cause the link to be within 0.1mm ofthe surface. (c) The geometry of the contact is determined and friction cones areplaced at the vertices bounding the contact region (in this case a line).
not inter-penetrating, but in practice the binary search only needs a few iterations
to reach the contact threshold distance.
3.4 Contact Location
After the bodies have been moved to within the contact distance, the system must
determine the regions of contact between the two bodies. This is done using a
modification of the PQP distance query algorithm that we have developed. The
process begins by searching the BVTT for pairs of triangles that are separated by
a distance less than the contact threshold. For each pair found, the system must
determine the region of overlap between them. To do this, it examines which pair
of topological features produced the minimum distance between the triangles (see
figure 3.16). Vertex-vertex and vertex-edge contacts are considered unstable and
are ignored (this is explained further in the next chapter). The contact region
40
Figure 3.16: Some of the possible contact regions between a pair of triangles thatdo not actually touch but are within the contact threshold distance of each other(not to scale). Left: an edge-edge contact is located using the closest points on thetwo triangles. Middle: a vertex-face contact is defined by the closest vertex on onetriangle and its projected point on the face of the other triangle. Right: a face-facecontact will have between 3 and 6 contact points (in this case 4). They are foundby projecting one triangle onto the other and finding the intersection points.
for an edge-edge contact is a single point, and on each triangle it is defined by
the closest point to the other triangle. All other topological pairs include a face,
and in these cases the vertices of the other triangle are orthographically projected
onto the plane containing the face. The projected triangle is then used to intersect
the first triangle, and vertices of the resulting region are projected back up to the
other triangle. Of those points that are projected back, only the ones closest to
the face plane are kept. A vertex-face contact will only have one closest point, but
a face-face contact could have a contact region bounded by as many as 6 vertices,
all equidistant from the face plane. Each of these vertices is considered a separate
point contact and is added along with its corresponding point on the other surface
to the set of contact points between the two bodies. This set grows as more triangle
pairs are examined, but it keeps only those contacts that are unique (the position on
the first body or the contact normal differ), since adjacent triangles on one surface
will often have identical contact points with a triangle on the other surface.
41
After the traversal is complete, the system divides the set of contacts up
into subsets that have the same contact normal. If there are more than two contact
points in a subset, the system checks to see if the points are all co-linear (as in an
edge-face contact), and if so, it removes all contacts but the endpoints of the line. If
they are not co-linear (as in a face-face contact), it finds the planar convex hull of the
contact set and removes any interior contact points. These interior contact points
do not affect the mechanics of the grasp and may be ignored [Nguyen, 1986]. Once
the final set of contacts between the two bodies has been determined, the system
draws a red friction cone at each contact point, which serves to visually mark the
position of the contact and its normal (see figure 3.15(c)). The width of this cone
identifies how large any frictional forces can be with respect to a force applied along
the contact normal. As each new contact is formed or broken the system performs
the grasp analysis routines described in the next chapter and displays the results.
42
Chapter 4
Grasp Analysis
Having described the methods of user interaction, we now focus on the grasp anal-
ysis performed by the simulator. This chapter presents necessary background in-
formation on the nature of contact and common methods of characterizing a grasp,
such as the grasp map, the grasp wrench space, and two quality metrics defined on
that space. It also shows how GraspIt uses these analysis tools, and the methods
we developed, to visualize the results. In addition, the simulator can be used to
compare the grasps of objects with different hands, which provides some insights
into the strengths and shortcomings of each hand design. Finally, the chapter con-
cludes by describing an advanced method for synthesizing optimal contact forces
given a known external wrench acting on the object, and it provides an example of
this optimization as carried out in GraspIt.
43
4.1 Contacts
Understanding the nature of contact is paramount to the analysis of grasping. When
two objects touch, it is possible for each of them to transmit forces and velocities
through the regions of contact. We begin this section by reviewing the different
types of contact that can occur between two bodies and how they affect the forces
and torques that can be applied at the contact. Then we examine how friction adds
to the possible contact forces.
4.1.1 Contact Types
When two bodies, A and B, come into contact with each other, a tangent plane is
defined between them. If we consider the forces that body A can apply to body
B, the first thing we note is that body A can only apply forces on the other side
of the tangent plane. A contact is always unilateral in the sense that body A can
push body B but not pull on it. The space of forces that can be applied on the
other side of the tangent plane is defined by the geometry of the contact and the
presence of any frictional forces.
At any point on a 3-dimensional C0 surface there exists two perpendicular
radii of curvature, which are generally expressed along the principal directions of the
surface at that point. The contact type can be determined by examining these radii
at the contact point on each of the contacting surfaces. Planar contacts only occur
when all four of the combined radii are infinite. Line contacts occur when three of
the four radii are infinite, such as when a cylinder contacts a plane that is parallel to
its central axis. As with planar contacts, the tangent plane is defined by the plane
in contact. Point contacts occur when no more than two of the radii are infinite
44
and at least two of them are non-zero. Without two non-zero radii of curvature, a
tangent plane cannot be defined, and the contact is considered unstable, as in the
case of two vertices contacting each other, or a vertex touching a sharp edge. If
either surface has two non-zero radii of curvature, the tangent plane can be defined
by the principal directions of that surface. If both surfaces have only one non-zero
radius of curvature at the point of contact, then a tangent plane can be defined by
the cross product of these two directions, assuming they are not parallel. This case
occurs when edges (straight or curved) touch non-parallel edges.
A frictionless point contact can only apply forces along the normal of the
tangent plane, also referred to as the contact normal. In addition to applying
forces along this normal, higher order contacts can apply resistive moments. A line
contact can resist moments about an axis perpendicular to it within the tangent
plane, and planar contacts can resist all moments with an axis in the tangent
plane. However, in order to simplify our treatment of contacts, we make use of
Ngyuen’s assertion [Nguyen, 1986] that complex contacts (linear and planar) can
be represented as the convex sum of proper point contacts at each of the vertices
of the surface patch.
4.1.2 Friction
Frictionless contacts are purely theoretical. Any two surfaces in contact can sup-
port some amount of friction, which acts to oppose force components within the
tangent plane. Any contact with greater than zero area can also use friction to
resist moments about the contact normal. The limit on the size of the tangential
frictional forces that can arise at a point contact is most commonly determined
45
tan µ-1
ff
Figure 4.1: Frictional forces, which lie in the tangent plane, have a maximummagnitude of µf⊥. To prevent slippage, the total contact force, f , must lie withina cone of possible directions.
using Coulomb’s model, which states:
ft ≤ µf⊥, (4.1)
where ft is the tangential force component, f⊥ is the normal force component, and µ
is the empirically determined coefficient of friction. When working in 3 dimensions,
this can also be written as:
f 2x + f 2
y ≤ µ2f 2⊥, (4.2)
where fx and fy are perpendicular force components within the tangent plane. From
this equation, it is apparent that the forces that may be applied at the contact lie
within a cone aligned with the contact normal, commonly known as a friction cone.
The half angle of this cone is tan−1µ as shown in figure 4.1.
4.2 Collecting contacts into a grasp
The previous section discussed the space of forces that can be applied at a single
point contact. In this section we show how the forces applied at a collection of
46
contacts on an object can be expressed with respect to a single frame of reference
using a matrix called the grasp map. This matrix is also known as a grip transfor-
mation [Salisbury, 1982] because when multiplied by a vector of contact forces, it
will give us the net resultant force and torque acting on the object.
4.2.1 Wrench Basis
A contact is an interface between two bodies through which forces and moments
can be applied. Rather than constantly dealing with these quantities as a pair of
vectors, the two are commonly combined into a single vector called a wrench:
w =
f
τ
. (4.3)
For 2-dimensional problems w ∈ <3, and for 3-dimensional problems, which are
the focus of this work, w ∈ <6.
The geometry of the contact and the presence of friction defines the space
of wrenches that can be transmitted through the contact. Mathematically, this
interface can be described with a wrench basis:
wc = Bcfc, fc ∈ FCc (4.4)
where wc is the transmitted wrench expressed within the contact frame c, Bc is
the wrench basis matrix, fc is a vector of contact force intensities, and FCc are the
friction cone constraints. Contacts in GraspIt are all considered to be point contacts
47
with friction, and have the following wrench basis and friction cone constraints:
Bc =
1 0 0
0 1 0
0 0 1
0 0 0
0 0 0
0 0 0
FCc =f ∈ <3 :
√f1
2 + f22 ≤ µf3, f3 ≥ 0
.
From this wrench basis one can see that torques cannot be transmitted through
a hard point contact. However, soft-finger point contacts, which have a non-zero
contact area, can transmit a torque about the contact normal in addition to the
three force directions. These types of contacts do not occur in GraspIt, but if
needed, they would be trivial to add.
4.2.2 The Grasp Map
To find the affect of applying forces at several different contacts on an object, it is
necessary to express them all with respect to a single frame of reference so that we
can resolve the moments induced by each force. Although it can be arbitrary, the
most commonly chosen reference frame is the center of mass of the object. Given
a fixed transform, (p, R), between each contact frame and the reference frame, we
can express contact wrenches with respect to the object reference frame using the
following transformation:
wo =
R 0
pR R
Bcfc, (4.5)
48
where p is the cross product matrix for vector p. By collecting the transformed
basis wrenches from each contact point into a single matrix, we then have a method
of computing the net wrench acting on the object given a vector of contact wrench
intensities. This matrix is called the grasp map:
G = [Tc1Bc1 , · · · , TcnBcn] . (4.6)
Using the grasp map, the net resultant grasp wrench can be computed:
wo = Gfc, fc ∈ FC, (4.7)
where fc is now a vector of all of the contact wrench intensities and FC combines
all of the friction cone constraints.
In order for the grasp to be in equilibrium, the net grasp wrench must negate
the net external wrench acting on the object:
Gfc + wext = 0. (4.8)
If given any wext ∈ <6, there exists a set of contact forces, fc ∈ FC that satisfy
equation 4.8, then the grasp is said to be a force-closure grasp. To solve for the
contact forces needed to balance an external wrench, we use the pseudo-inverse of
the grasp map:
fc = −G+wext + λN , (4.9)
where N is the null space of G, and it represents the space of internal forces of the
grasp. It is by selecting appropriate values for the vector λ that it may be possible
to satisfy the friction cone constraints, fc ∈ FC. Section 4.7 examines optimization
methods for choosing λ.
49
f3f8 f1 f2f7f
Figure 4.2: We have approximated the friction cone with an 8 sided pyramid. f isrepresented as a convex combination of 8 force vectors around the boundary of thecone.
4.3 Building the Grasp Wrench Space
Obviously, a force-closure grasp is desirable because it can resist arbitrary distur-
bance wrenches, but some disturbances will be easier to resist than others. To more
fully analyze the properties of a grasp, we need to examine the full space of forces
and torques that can be applied by the grasp.
Ferrari and Canny [1992] describe two methods of constructing this space.
In both cases a convex hull bounds the space, but in one, the sum magnitude of the
contact normal forces is bounded, and in the other, the maximum magnitude of the
normal forces is bounded. Both methods begin with a linear approximation of the
quadratic friction cone. This means that any allowable contact force is bounded
by the convex sum of a finite number of forces around the boundary of the friction
cone as shown here,
f ≈m∑
j=1
αjfj, (4.10)
fj = f⊥ + µs cos(
2πj
m
)fx + µs sin
(2πj
m
)fy,
50
αj ≥ 0,m∑
j=1
αj = ‖f⊥‖,
where each fj is a force vector along one edge of an m-sided pyramid described by
the unit vectors f⊥, fx, fy (see figure 4.2). The whole pyramid is scaled by the
magnitude of the normal contact force.
The next step is to find the total space of wrenches that can be exerted on
the object by a grasp. Let g be a grasp force vector composed of the magnitude of
the applied normal contact force of each contact:
g =
‖f⊥1‖
...
‖f⊥n‖
. (4.11)
Given the space of forces that can be applied at contact point i, we must determine
the corresponding space of wrenches that can be exerted on the object by that
contact. To accomplish this, we require a torque multiplier, λ, that relates units of
torque to units of force. In this work, as Pollard [1994] has done, we have chosen
to enforce ‖τ‖ ≤ ‖f‖ by choosing λ = 1
r, where r is the maximum radius from the
wrench space origin, often the center of gravity. This will ensure that the quality of
a grasp is independent of object scale. Each of the m force vectors representing the
boundary of the friction cone is translated to the wrench space origin by computing
the corresponding torque such that
wi,j =
[fi,j
λ(di × fi,j)
], (4.12)
where fi,j is one the m force vectors on the boundary of the friction cone at contact
i, and di is the vector from the wrench space origin to the i-th point of contact.
51
These wrenches form the boundary of the wrenches that can be applied at that
contact point given a unit normal force.
A unit grasp wrench space, which is required to compute the quality measures
described in the next section, is the space of wrenches that can be applied by a
grasp that has a grasp force vector with a magnitude of 1. However, we have not yet
specified how we will determine the magnitude of the grasp forces. Here, Ferrari and
Canny offer two possibilities. One can either limit the maximum allowable normal
contact force (‖g‖ = 1 over an L∞ norm) or one can limit the sum magnitude of
the normal contact forces (‖g‖ = 1 over an L1 norm).
If we assume the contact forces are independent and are individually upper
bounded by using the L∞ norm, then the set of possible wrenches that can be
exerted on the object, WL∞, is bounded by setting each ‖f⊥i
‖ = 1, and we arrive
at the following:
Wi = wi | wi =m∑
j=1
αi,jwi,j, αi,j ≥ 0,m∑
j=1
αi,j ≤ 1, (4.13)
WL∞=
n⊕
i=1
Wi. (4.14)
The last equation is the Minkowski sum of the individual convex sets of wrenches
that can be applied at each contact. An implementation of this would exchange the
convex hull operation with the Minkowski sum, yielding the following equivalent
expression:
WL∞= ConvexHull(
n⊕
i=1
wi,1, . . . , wi,m). (4.15)
The Minkowski sum yields mn 6-dimensional points which are passed to the convex
hull operation.
If, on the other hand, we bound the sum magnitude of the contact normal
52
forces as follows:n∑
i=1
‖f⊥i‖ ≤ 1, (4.16)
we get the following expression for the set of wrenches that can be applied to the
object:
WL1= w | w =
n∑
i=1
m∑
j=1
αi,jwi,j, αi,j ≥ 0,n∑
i=1
m∑
j=1
αi,j ≤ 1, (4.17)
which is equivalent to:
WL1= ConvexHull(
n⋃
i=1
wi,1, . . . , wi,m). (4.18)
This convex hull computation is only over mn points.
Note that WL∞⊇ WL1
because the the set of points that we compute the first
convex hull over is a superset of the set of points in the second convex hull operation.
If the basis set of vectors, (⋃n
i=1wi,1, . . . , wi,m), positively spans <6, then both
WL∞and WL1
will contain the wrench space origin, and we can consider the grasp
to have force-closure because the grasp can counter any disturbance wrench by
applying an infinite amount of force at the contact points.
4.3.1 Building the Grasp Wrench Space in GraspIt
To build the grasp wrench space (GWS), GraspIt approximates the friction cone
at each contact point with 8 force vectors equally spaced around the boundary of
the cone (refer back to figure 4.2). The angle of the force vectors with respect
to the contact normal is determined by the coefficient of static friction, µs. Since
this value can be different for each pair of materials, we have allowed the materials
of the links and object to be individually specified, thus increasing the flexibility
and realism of the system. Table 4.1 is a matrix containing the coefficients we
53
material metal glass plastic wood rubber
metal 0.2 0.2 0.2 0.3 1.0glass 0.2 0.2 0.2 0.3 1.0
plastic 0.2 0.2 0.3 0.4 1.0wood 0.3 0.3 0.4 0.4 1.0rubber 1.0 1.0 1.0 1.0 2.0
Table 4.1: The values of the coefficient of friction for materials defined within GraspIt.
have assumed for the materials currently defined in our simulation. Because the
values can only be determined empirically, we chose values from within the ranges
provided in the CRC Handbook of Chemistry and Physics [Lide, 1995].
The 6D wrench corresponding to each force vector on the boundary of the
each cone approximation is computed using the radius vector from the center of
mass of the object to the point of contact. The torque component of the wrench
vector is scaled by 1/rmax so that the elements of the wrench have the same units
and so that the size of the grasp wrench space is not dependent on the object scale.
After collecting all of the grasp wrenches, the system calls the qhull library [Barber
et al., 1996] to compute the 6-dimensional convex hull of the wrench list. This hull
is the unit grasp wrench space defined under an L1 norm. Currently we are not
building the WL∞space because of the time it would take to compute the convex
hull of 8n points when the number of contacts, n, might be on the order of 6 - 10
for a medium complexity grasp.
4.4 Quality Measures
In order to rate a grasp, we must first consider the properties of a good grasp. In
general, a grasp exists to perform some task which can be described by the space
54
of wrenches that must be applied to an object to carry out that task. This space
could be simply the wrench with a force component in the upward direction and
no torque component, which would be required to suspend an object against the
force of gravity, or it could be a complicated space of wrenches required in some
manipulation task. This space is called the task wrench space. If the task wrench
space is a subspace of the grasp wrench space, then the grasp is valid for that task.
However, it is not necessarily the most efficient grasp that can accomplish the task,
meaning that the internal forces on the object may be much larger than necessary.
Thus one method of defining grasp quality is the ratio of the magnitude of the task
wrenches to the magnitude of the contact forces.
The most general quality measures assume nothing is known about the task
wrench space. Kirkpatrick et al. [1990] and Ferrari and Canny assume that distur-
bance wrenches could come from any direction, and the task wrench space is mod-
eled as a 6-dimensional ball. Thus, the maximum magnitude of any task wrench
within this ball is equal to its radius. Since grasp quality is defined as a ratio of
magnitudes, which scales linearly, we can rephrase our definition of grasp quality so
that it is the radius of the largest wrench space ball, centered at the origin, which
can just fit within the unit grasp wrench space. This radius, ε, will be the same
as the magnitude of the shortest vector, wmin, from the wrench space origin to the
outer boundary of the hull. We define the minimum radius of the L1 unit grasp
wrench space as ε1.
Examining the definition of ε1 more closely, we find a few of its intuitive
characteristics. If an adversary, who knows the shape of the grasp wrench space,
WL1, applies a worst case disturbance wrench to the object, the sum magnitude of
55
the normal contact forces would have to be 1
ε1times the magnitude of the distur-
bance wrench. This means that the closer ε1 is to 1, the more efficient the grasp
is.
Unfortunately, these measures are not invariant to the choice of torque origin
as pointed out by Li and Sastry [1988], so the volume of the unit grasp wrench space
can be used as an invariant average case quality measure for the grasp. We denote
the next quality measure,v1, as the volume of WL1.
4.4.1 Implementation of Quality Measures in GraspIt
Computing the values of these quality measures in GraspIt is a simple matter
once the grasp wrench space has been built as described in section 4.3.1. Qhull
directly reports the volume of the hull, which serves as the average case grasp
quality measure and it is displayed as the “v” quality value. By examining the
offset values of the facet hyperplane equations, the system can determine which
facet of the hull is closest to the origin. The vector that is normal to that facet is
the smallest wrench on the surface of the GWS, wmin, and is considered the grasp’s
weakest point. The magnitude of this vector (the offset of the hyperplane) is used
as the worst case ε1 quality measure and is displayed as the “e” value. Example
grasps with their quality measures are presented in the next section, and further
examples can be found in [Miller and Allen, 1999].
56
Figure 4.3: The grasp wrench space projection dialog box allows a user to fix threeof the six coordinates to specific values. When the user clicks ok, a new wrenchspace projection is created in a separate window.
4.5 Wrench Space Visualization
While the numerical quality measures provide some information about the efficiency
of the grasp, graphical feedback can provide even more information. One reason
3-dimensional examples are often avoided throughout the grasp-quality literature
is due to the difficulty of visualizing the results. A 2D planar object will have a
3D wrench space, but to display the 6D convex hull resulting from a 3D object,
we must project it into 3D space by fixing three of the wrench coordinates. The
GraspIt system allows a user to chose which coordinates should be fixed and at
what value (see figure 4.3). Although some choices for the fixed coordinates may
be useful for particular types of grasps, we have found four projections that are of
general use for most grasps. If we fix the torque coordinates of the wrench space to
0, the resulting projection shows the space of forces that can be applied by the grasp
without imparting a net torque to the object, and if we fix the force coordinates to
0, we can visualize the space of torques that can be applied to the object without
a net force acting on it. By setting the torque coordinates to their values at the
57
Figure 4.4: A completed force-closure grasp of the mug (top view). The magnitudeof the smallest wrench on the surface of the unit grasp wrench space, and thevolume of the unit GWS are displayed as the quality measures “e” and “v”. Thepair of purple indicators show the force and torque components of the worst casedisturbance wrench. In the upper left is a projection of the GWS that shows thespace of forces that can be applied to the mug without creating a moment, and inthe lower left is a projection that shows the space of torques that can be appliedwithout a net force acting on the object.
58
Figure 4.5: A force-closure grasp of the mug (side view).
59
Figure 4.6: A force-closure grasp of the mug using rubber finger surfaces instead ofplastic.
60
point on the hull boundary that is closest to the origin, wτmin, we can display the
space of forces that can be applied if the worst case torque needs to be applied.
Similarly we can display the space of torques that can be applied if the worst case
force, wfmin, must be applied.
If we choose to fix the three torque components, we arrive at a collection of
3-dimensional force halfspaces as follows:
Hn×6
x1
x2
x3
τ1
τ2
τ3
+ b6×1 ≤ 0 ⇒ Hn×3
x1
x2
x3
+ b′3×1 ≤ 0, (4.19)
where H is the matrix of halfspace normal vectors, b is the vector of offsets, and
τ1, τ2, τ3 are fixed values.
In order to construct a 3-dimensional hull, we require the vertices of the
halfspace intersection. Qhull takes advantage of the duality between points and
halfspaces as described in Preparata and Shamos [1985], and computes the convex
hull of the halfspace duals. The facets of this hull are the duals of the vertices in the
halfspace intersection, and the connectivity information is also preserved so that
adjacent facets in the dual hull correspond to adjacent vertices in the halfspace
intersection. Using this information we can construct the hull bodies which are
presented in the following examples. In addition to the projections, GraspIt also
shows the components of the worst case disturbance wrench, −wmin, as a pair of
purple indicators emanating from the object’s frame of reference. These help to
61
show where the grasp is the weakest.
In the example shown in figures 4.4 and 4.5, the Barrett hand is closed around
the outside of a mug. The purple indicators show that the grasp is weakest when
a disturbance force is applied in the upward direction (with relation to the upright
position of the mug), and a torque is applied along an axis almost perpendicular
with the palm of the hand. From the wrench space projection into 3 dimensional
force space (upper projection window), we see that while the grasp can apply a
variety of forces in a horizontal plane, it cannot apply large forces in the vertical
direction. This is because the contacts are arranged around the outer perimeter
of the mug and there are no contacts on the top of bottom of the mug. Thus the
grasp must rely on frictional forces only to resist a disturbance force in the vertical
direction. We also see from the torque space projection (lower projection window)
that this grasp cannot apply large moments to the mug without a net force. This is
because the contact normals all point in the general direction of the center of mass.
Figure 4.6 shows the same grasp as before, but now the finger surface material
has been changed from plastic to rubber. Since the coefficient of friction at the
contacts is much higher, larger frictional forces can be supported by this grasp, and
the quality of the grasp is dramatically increased. Note the size of the force and
torque spaces has also increased greatly.
62
4.6 Comparing the Grasps of Two Objects Using
Three Different Hands
Since GraspIt provides an objective measure of the quality of a grasp, it can be
a useful tool for comparing grasps of an object using different robotic hands. As
more grasps of different objects are performed with the hands, their individual
characteristics become clear. Some hands are simply better suited to certain tasks,
and this sort of information is extremely helpful to a hand designer. In this section
we compare the grasps of two different objects performed with the Barrett and
DLR hands, and a parallel jaw gripper.
4.6.1 The Mug Object
The first object to be grasped is the coffee mug, seen previously. It is fairly large,
having a maximum diameter of 104mm, and it is grasped from the side in each
case. The Barrett hand is able to wrap around the mug, as shown in figure 4.6,
and contact occurs on two of the inner links for additional stability. The fingers of
the DLR hand are longer and able to wrap further around the mug (see figure 4.7),
and the additional finger is able to provide some vertical support underneath the
mug handle. Even using the “palm”, the parallel jaw gripper, seen in figure 4.8,
can only contact the rounded mug in three places resulting in a lower quality grasp.
4.6.2 The Telephone Handset
The second object is a model of a telephone handset with parallel side faces. Its
dimensions are 203mm x 57mm x 51mm and again it is grasped on its sides. It is
63
Figure 4.7: A force-closure grasp of the mug using the DLR hand which has a metalpalm and inner link surfaces and rubber fingertips.
64
Figure 4.8: A force-closure grasp of the mug using a parallel jaw gripper.
65
nearly a perfect fit for the Barrett hand, as shown in figure 4.9. The planar palm
and finger surfaces provide strong contacts along the sides of the phone, and the
inner links of the top two fingers provide contacts on the top face of the phone. The
handset is too small for the DLR hand, which is about 1.5 times the size of a human
hand, to grasp effectively (see figure 4.10). The joint limits on the innermost finger
joints prevent the fingers from wrapping around the phone and only the thumb is
able to provide an inner link contact. The palm is curved and only contacts the side
of the phone at two points. Further experimentation has shown that the DLR hand,
with its 12 internal DOFs, is best suited for manipulation of objects using precision
grasps, but it can also perform a limited range of power grasps of larger objects.
Lastly, the parallel jaw gripper can grasp the parallel sides of the phone much more
effectively than the rounded sides of the coffee mug (see figure 4.11). However, since
the grasp lacks contacts on other faces it is still somewhat susceptible to external
torques about an axis passing through the two contact points, as shown by the
purple worst-case external wrench indicators.
4.7 Grasp Force Optimization
If we are able to find a satisfactory force-closure grasp, then it is possible to produce
any net object wrench by applying different forces at the contacts, and if we know
the external disturbance wrench acting on the object (such as the force of gravity),
we can find a set of contact forces that will keep the object in equilibrium. In
general, however, this is an under determined problem and an optimization method
is required to choose the best values for the internal forces represented by the vector
λ in equation 4.9. The contact forces must lie within the friction cones and are
66
Figure 4.9: A force-closure grasp of the phone handset using the Barrett hand.
67
Figure 4.10: A force-closure grasp of the phone handset using the DLR hand.
68
Figure 4.11: A force-closure grasp of the phone handset using the parallel jawgripper.
69
limited by the maximum torque that can be applied at the joints of the hand.
Because this problem is dependent on the total configuration of the hand and not
just the placement of the contacts, we begin this section by reviewing how we
construct the hand Jacobian that relates contact forces to joint torques. Then we
discuss a method of optimizing the contact forces that uses linear matrix inequalities
and avoids having to use linear approximations of the friction cones. The last
portion describes how this method was implemented in GraspIt and provides an
example of this optimization.
4.7.1 The Hand Jacobian
To this point, we have considered the contact forces independently of the hand
mechanism. However, the hand’s ability to apply force at a contact depends largely
on the finger kinematics and the configuration of the contact with respect to the
finger link. Robotics uses a Jacobian to relate joint velocities of a chain manipu-
lator to its Cartesian endpoint velocity, and the transpose of the Jacobian relates
endpoint forces to joint torques:
v = J(θ)θ, (4.20)
τ = JT (θ)fc. (4.21)
For each contact point, we can compute the traditional Jacobian from the finger
base to the endpoint of the link on which the contact occurs and multiply it with
a fixed Jacobian relating link endpoint velocities with contact point velocities. Fi-
nally, the contact Jacobian is also multiplied by the transpose of the contact’s basis
wrench matrix to select only those velocities which can be imparted on the object
70
through the contact:
Ji(θfk) = BT
ci
RT −RT p
0 RT
Jfk
(θfk), (4.22)
where (p, R) is the transform from the link endpoint to the contact frame.
The individual contact Jacobians, Ji, can be stacked vertically with zeros
placed in columns corresponding to joints not in the chain (i.e. they are part of
another finger) to form the hand Jacobian.:
Jh(θ) =
J1(θf1) 0
J2(θf1)
. . .
0 Jn(θfk)
. (4.23)
The transpose of the grasp map relates object velocity to contact point ve-
locities, so we can set up the following equality:
Jh(θ)θ = GT vo. (4.24)
Using this equation we can also define another grasp property. If for any object
velocity, vo ∈ <6, there exists a set of joint velocities, θ, that satisfies equation 4.24,
then the grasp is said to be manipulable.
Just as with traditional Jacobians, the transpose of the hand Jacobian re-
lates contact forces to joint torques. It can be used to compute the joint torques
associated with the optimal contact forces that are found during the grasp force
optimization process described in the next section.
71
4.7.2 Optimization of Grasp Forces using Linear Matrix In-
equalities
Kerr and Roth’s [1986] method was to linearize the friction cone, as shown earlier,
and define all the constraints as a collection of halfplanes. Then they used a linear
programming method to find the set of forces furthest from all the constraints.
While this works well, the linear approximation of the friction cones tends to be
overly conservative for this application.
Han and Trinkle [2000] avoid this approximation by formulating the contact
force constraints as linear matrix inequalities. They begin with the observation
made by Buss, Hashimoto and Moore [1996] that a friction constraint such as the
one in equation 4.2, is equivalent to the positive semi-definiteness of a particular
matrix, in this case
Pi =
µixi3 0 xi1
0 µixi3 xi2
xi1 xi2 µixi3
, (4.25)
where xi ∈ <miis the wrench applied at contact i (mi is the dimension of the
contact wrench space, which for a point contact with friction, PCWF, is 3), and µ
is the coefficient of friction. Each of the constraints can be compiled into a block
diagonal matrix,
P = Blockdiag(P1, . . . , Pi, . . . , Pk),
which if positive semi-definite, means that all the constraints are satisfied. Since
the constraint matrices are linear in the contact wrench vector xi, they can be
72
rewritten as a linear matrix inequality (LMI):
Pi =mi∑
j=1
xijSij 0, (4.26)
where the individual Sij are real symmetric constant matrices. For the point contact
with friction example these are:
Si1 = E313 + E3
31,
Si2 = E323 + E3
32,
Si3 = µi
(E3
11 + E322 + E3
33
),
where Eabc is a square matrix of dimension a filled with zeros except element (b, c)
which is 1. By rewriting each of the friction constraints this way, they can be
combined into a single LMI:
P (x) =m∑
l=1
xlSl 0, (4.27)
where the double-indexed xij is replaced with xl and Sl = Blockdiag(0, . . . , 0, Sij, 0, . . . , 0).
Similarly, the torque limit constraints,
JT x + gext − τL ≥ 0, (4.28)
−JT x − gext + τU ≥ 0,
where τL and τU are the lower and upper torque limits and gext is a vector of
external loads on the joints such as gravity, can be written as
CL(x) = diag(JTx + gext − τL) = CL0 +
m∑
l=1
CLl xl 0, (4.29)
CU (x) = diag(−JT x − gext + τU) = CU0 +
m∑
l=1
CUl xl 0,
73
where CL0 = diag(gext − τL), CU
0 = diag(−gext + τU), CLl = diag(JT
l ), CUl =
diag(JTl ) and JT
l is column l of matrix JT . These two LMIs can be combined
into a single LMI,
C(x) = Blockdiag(CL(x), CU(x)) = C0 +m∑
l=1
Clxl 0, (4.30)
where Cl = Blockdiag(CLl , CU
l ).
The grasping force optimization problem is then defined as: given a grasp
G with friction cone constraints FC, joint effort constraints T , and a desired net
grasp wrench F , find an optimal grasping force x in the feasible set
Ax = x ∈ <m | P (x) 0, C(x) 0, Gx = F . (4.31)
The measure of optimality was defined
Ψ(x) = wT x + log det P−1(x), (4.32)
where the vector w = [wT1 · · ·wT
i · · ·wTk ]T ∈ <m weights the normal components of
the grasping force x, (i.e. for a point contact with friction wi = [0 0 di]T , di ≥ 0).
The first term of the measure will grow with the contact normal forces, and the
second term grows to infinity as any contact force approaches the boundary of its
friction cone. With this optimality criteria the grasping force optimization can be
written
argminx∈AxΨ(x). (4.33)
Of course, the problem is still stated in terms of x. To optimize the internal
forces, λ, we must transform the LMI:
P (λ) := P (Nλ) =m−6∑
l=1
λlSl 0, (4.34)
Sl =m∑
i=1
nilSi, (4.35)
74
where nil is the element of the null space N in row i and column l. Next we create
the feasible set for λ,
Aλ =λ ∈ <m−r | P (λ) 0, C(λ) 0
, (4.36)
and we rewrite the optimality measure,
Ψ(λ) := Ψ(G+F + Nλ) (4.37)
giving us the internal grasping force optimization problem:
argminλ∈AλΨ(λ). (4.38)
4.7.3 Implementation of GFO in GraspIt
After the contacts have all been determined, the system updates the grasp map and
the hand Jacobian. Next it constructs the linear matrix inequalities from the joint
torque limits and the friction constraints, and attempts to find a feasible solution
to the equilibrium equation as described above. If a feasible solution exists, the
system optimizes the grasping forces using the maxdet1 library and displays them
as varying length pointers within the friction cones of the grasp. It also uses the
hand Jacobian to determine the optimal torques and displays those as well.
To demonstrate this process, we show a fingertip grasp of the phone handset
using the Barrett hand2 in figure 4.12. The mass of the phone is assumed to be
0.5kg and the contact forces will have to resist the force of gravity pulling down on
1The maxdet software package implements a primal-dual interior-point convex optimization al-gorithm. Please refer to [Wu et al., 1996] for details of the algorithm and the choice of parameters.
2In this example, we use simplified fingertip geometry to reduce the total number of contactpoints. This provides for a cleaner example, but the technique can easily handle more complexgrasps with more contact points.
75
the phone. After setting the value of this external wrench to [0, 0,−4.9, 0, 0, 0]T ,
it appears as the green indicator in figure 4.13. Next, using a weighting factor of
1.0 in the optimality measure described by equation 4.32, the system optimizes
the contact forces so that the handset is in equilibrium, the contact forces avoid
the edges of the friction cones, and the joint torques are within their limits. The
resulting contact forces, shown in yellow in figure 4.13, are [0.0N,−1.22N, 6.30N ]T
for the two contacts on the left side of the phone and [0.0N,−0.61N, 3.15N ]T for
the four contacts on the right side, where the z-axis of each contact is the inward
pointing contact normal, and the y-axis of each contact points down in the negative
world-z direction. If the chosen weighting factor was larger, the contact forces would
have been smaller and closer to the edges of the friction cones, and vice versa. By
multiplying by the transpose of the hand Jacobian we arrive at the optimal joint
torques which are 0 N-mm for the finger spread joint, 1486 N-mm for the inner joint
of the finger on the left side, and 743 N-mm for the inner joints of the fingers on the
right side. These torque values are overall joint torque values and do not correspond
directly to motor torques. Further modeling will need to be done to incorporate
gear ratios, gear friction, and backlash in order to compute motor torques.
76
Figure 4.12: A force-closure grasp of the phone handset using only the fingertips ofthe Barrett hand.
77
Figure 4.13: The contact forces (yellow arrows) have been optimized to resist theexternal force of gravity (green arrow) and avoid the boundaries of the frictioncones.
78
Chapter 5
Planning, Executing, and
Monitoring Robotic Grasping
Tasks: An Example Application
Alone, GraspIt is a useful tool for examining how various elements of a hand’s
design affect its ability to grasp a particular object. However, when combined with
a real-time vision system and a robotic arm-hand setup, it can become a powerful
tool for planning and controlling actual grasping tasks. In this setting it is possible
to test the highest quality simulated grasps and determine how well the simulation
corresponds to reality. This chapter describes a system that combines model-based
pose estimation and object tracking with the planning and simulation capabilities of
GraspIt, and demonstrates its use in the execution of a few sample grasping tasks.
This system was built in cooperation with Danica Kragic from the Computer Vision
and Active Perception group at the Royal Institute of Technology in Sweden, and
79
this chapter presents our joint work that was published in [Kragic et al., 2001].
5.1 System Overview
In order for the simulated environment within GraspIt to correspond to the actual
robotic work cell we must know the position and orientation of the object to be
grasped with respect to the robot. Rather than fixing the object in a predetermined
position, which would be a serious limitation, we rely on a calibrated passive vision
system to estimate the pose of the object. This information is relayed to the
simulator which updates the pose of the virtual object. Next the user finds a stable
high-quality grasp of the object using GraspIt and once finished, he can command
the robot to execute the planned grasp with the push of a button. After the grasp
has been completed, the user may then specify a destination position within the
simulator, and this goal pose is sent back to the vision subsystem, which projects
the object features back to the image to define the goal object position in image
space. Then the grasped object is moved to the destination position using a visual
servoing routine. By using visual feedback to monitor the movement task we ensure
the object actually reaches the desired position, which could not be guaranteed if
we relied solely on open-loop control. Figure 5.1 describes this progression of steps
graphically.
5.2 The Vision Subsystem
Closed-loop control of a robot where vision is used in the feedback loop is commonly
referred to as visual servoing. Vision based tracking techniques incorporated in vi-
80
EXECUTION
GRASP
PUMA 560
MONITORING
TASK
OBJECT POSE
ESTIMATION
PLANNINGGRASPGraspIt!
Robot/Hand
Frame
Control
Grabber
Figure 5.1: A block diagram of the grasping system.
sual servoing systems usually facilitate model based or feature based approaches.
A model based approach relies on a CAD model of the object to be tracked [Marc-
hand et al., 2000] [Wunsch and Hirzinger, 1997] while feature based techniques use
distinct image features: corners, lines, regions [Hager and Toyama, 1998]. There
are two categories of visual servoing: position based servoing and image based ser-
voing, [Hutchinson et al., 1996]. A position based approach usually requires the
estimation of a target’s pose in the camera or the robot coordinate system. Image
based visual servoing uses visual information directly from the image. In our work,
we use both approaches: a position based approach is used for driving the arm and
the hand to a pose generated by a simulator, and image based servo control is used
to visually servo a grasped object to the desired pose.
By definition, a necessary requirement of a visual servo system is continuous
tracking. Our tracking system is based on the ideas proposed by Marchand et al..
The method relies on the assumption of the 2D affine motion model of image fea-
81
FITTING
MOTION
POSEINITIALIZATION
MANUAL
PARAMETERSCAMERA
MATRIX
3D MODEL VISIBLE EDGES
FLOW
NORMAL
AFFINE
LOCATE
RENDER MODEL
3DPROJECTION
Figure 5.2: The tracking system.
tures between two successive images. Since this model does not perfectly capture
the complex 3D motion, the affine step is followed by an optimization technique
where the pose space of the target is searched for the most suitable set of pose pa-
rameters given the image data. The general idea and the tracking system operation
are presented in Fig. 5.2.
5.2.1 Pose Estimation
To estimate the position and orientation of the object, Ω(R, t), we use the approach
proposed by Dementhon and Davis [1995], which uses N point correspondences and
relies on linear algebra techniques. The method does not require an initial pose
estimate or matrix inversions like iterative methods based on Newton-Raphson
minimization. It starts by assuming a scaled orthographic projection and iteratively
converges to a perspective projection by minimizing the error between the original
image points and the projected points.
This step is followed by an extension of Lowe’s [1992] nonlinear approach
82
(a) (b) (c)
Figure 5.3: The key components of the visual tracking system: (a) the model withpoints used in POSE to determine the pose, (b) pose estimation (the model isoverlaid in black), and (c) normal flow estimation (the lines represent the directionand not the magnitude).
proposed in [Araujo et al., 1996]. The pose estimation step is called POSE in
Fig. 5.2. The initialization of the tracking system is done manually. A number
of corresponding points on the wire frame model and the image are chosen by the
user, see Fig. 5.3(a) and Fig. 5.3(b).
5.2.2 Normal Displacement and Affine Motion
Using estimated pose, the model of the object is projected onto the image plane. A
simple rendering technique is used to determine the visible edges of the object. For
each visible edge, tracking nodes are assigned at regular intervals in image coordi-
nates along the edge direction. After that, a search is performed for the maximum
discontinuity in the intensity gradient along the normal direction to the edge. The
edge normal is approximated with four directions: 0, 45, 90, and 135 degrees. This
way we obtain a displacement vector, d⊥i (xi, yi) = [∆xi∆yi]
T , representing the nor-
83
mal displacement field of visible edges. A 2D affine transformation is expressed:
xt+1i
yt+1i
=
x y 0 0 1 0
0 0 x y 0 1
Θ = A(xi, yi)Θ, (5.1)
where Θ = (a1, a2, a3, a4, Tx, Ty)T represents the parameters of the affine model.
There is a linear relationship between two consecutive images with respect to Θ:
di(xi, yi) = A(xi, yi)Θ′ (5.2)
= A((xi, yi))[Θ − (1, 0, 0, 1, 0, 0)T ]. (5.3)
From the previous equation follows:
d⊥i = nT
i d(Pi) = nTi A(Pi)Θ
′, (5.4)
where ni is a unit vector orthogonal to the edge at a point Pi. From Eq. 5.4 we
can estimate the parameters of the affine model, Θ′ using an M-estimator ρ:
Θ′ = arg minΘ′
∑
i
ρ(d⊥i − nT
i A(Pi)Θ′). (5.5)
Using the estimate of points in the image at time t+1, the POSE step is performed in
order to obtain the pose of the object in the camera coordinate system, Ω(R, t)init.
5.2.3 3D Tracking
Using the estimated affine parameters Θ′ and positions of edge nodes at time t, we
are able to compute their positions at time t+1 from Eq. 5.1. As already mentioned,
the affine model does not completely account for the 3D motion and perspective
effects that occur during the object motion. Therefore, the pose space, Ω(R, t),
should be searched for a best fit given the image data. As proposed in [Marchand
84
et al., 2000], the projection of the object model is fitted to the spatial intensity
gradients in the image, using Ω(R, t)init as the initialization:
Ω(R, t) = arg minΩ
−
∑
CΩ
‖∇G(t + 1) ‖
, (5.6)
where ∇G(t + 1) is the intensity gradient along the projected model edges and CΩ
are the visible edges of the model for given a pose Ω.
The optimization method uses a discrete hierarchical search with respect to
the pose parameters. However, the correct discrete step of the pose parameters
is crucial in order to find the right minimum value. The affine motion model is
particularly sensitive to the rotational motions of the object and large changes in
rotation usually result in a loss of tracking. In addition to the original algorithm,
we have implemented an adaptive search step determination based on the object’s
3D velocity. The basic idea is to dynamically change the size of the search step
instead of keeping it constant.
5.3 Planning the Grasp
Once the pose of the virtual object has been set using the measurements taken by
the vision subsystem, the user may begin interacting with GraspIt to choose a high
quality grasp of the object. While the user interface and visualization methods
are fully described in chapters 3 and 4, it is important to note the necessity of the
robot and workbench models in planning actual grasps. These elements provide
constraints on the space of possible grasps so that grasps that are impossible to ex-
ecute in the real world are also impossible to reach within the simulation. Dynamic
simulation, described in the next chapter, is also useful in planning how the grasp
85
will be executed since the object might be disturbed from its position if the fingers
do not come in contact with the object simultaneously. However, the experiments
performed in the next section assumed that the object is static and will stay in a
fixed position while the grasp is performed. Future work, will test the accuracy of
the dynamic simulations.
5.4 Experimental Evaluation
To test the system, we set up some simple grasping tasks using two different objects.
The task involved determining the object’s pose from vision, finding an appropriate
and stable grasp for the robotic hand given the object’s configuration, executing
the grasp, and monitoring the grasped object’s trajectory as it is moved.
5.4.1 Experimental Setup
The dextrous robot hand used in this example is the Barrett Hand. It is attached
to a Puma560 arm which operates in a simple work cell. The vision system uses a
standard CCD camera (Sony XC-77) with a focal length of 25mm and is calibrated
with respect to the robot workspace. The camera is mounted on a tripod and
views the robot and work cell from a 2m distance. Both the robot controller and
the frame grabber are connected to a Sparc20 workstation which is linked via a
TCP socket connection to an SGI Indigo 2 running the GraspIt software.
The process of planning and executing a grasping task is as follows (refer
also to the block diagram presented in Fig. 5.1):
1. After an image of the scene is obtained, interactive pose estimation of the
86
object is performed as presented in section 5.2.
2. A pose estimate is sent to GraspIt which assists a user to generate the most
suitable grasp.
3. Through GraspIt the actual arm and hand are controlled in order to perform
the final grasp.
4. Once the object is grasped, the monitoring task begins. Image based visual
servoing is used to place the object into the final desired pose. Through
GraspIt, a few trajectory points (object poses) are manually generated. Each
generated pose is used as an intermediate desired pose for the visual servoing
routine.
5.4.2 Results: Pose estimation and grasp generation
The first three example grasps (figure 5.4, figure 5.5, and figure 5.6) demonstrate
steps 1-3 of the system (pose estimation, grasp planning and grasp execution). After
the first image of the scene is acquired, a manual initialization step is performed as
explained in section 5.2, and the estimated pose of the object is sent to GraspIt,
which aids the user in selecting an appropriate grasp. After a satisfactory grasp is
planned, it is executed by controlling both the robot arm and the robot hand.
Each of the first two examples, show a successfully planned and executed
stable grasp of an L-shaped object. The grasp planned in the third example is not
stable but requires precise estimation of the object pose so that the thumb can
hook through the handle of the mug and it serves to demonstrate the accuracy of
the system.
87
Figure 5.4: Top left: The vision system determines the pose of the L-shaped objectwithin the robot workspace. Top right: A stable grasp planned within GraspIt.Bottom left: The wrist is moved into position. Bottom right: The completedgrasp.
88
Figure 5.5: The user has successfully planned and executed a stable grasp of theobject in a new orientation.
89
Figure 5.6: While the grasp planned in this example is not stable, it demonstratesthe accuracy of the system. The camera is 2m away from the workspace, yet thethumb was able to enter the handle opening which is only 1.5 cm wider than it.
90
Figure 5.7: Left: A stable grasp of the L-shaped object. The axes are positionedat the object’s center of gravity (z axis points into the table). Middle: The 6Dgrasp wrench space projected into force space by setting the net torque to 0. (zaxis points up). Right: The GWS projected into torque space by setting the netforce to 0.
5.4.3 Results: Visual servoing for grasp monitoring
As already mentioned, once the object is grasped the monitoring task begins. We
use image based visual servoing to move the object to a desired position. In general,
this is set up using a “teach–by–showing” method. In many of the existing systems,
[Hutchinson et al., 1996], the object is first placed into the desired position and
feature extraction is performed. After that, the object is put into some initial
position and the servoing sequence is initiated by iteratively minimizing the error
in the image between the desired and the current image positions of the features.
The step of putting the object in the desired position and obtaining the feature
measurement is not needed in our case. The user can easily specify the object
pose at a few different points along the trajectory using the simulator. Using
knowledge of the camera parameters and 3D object pose defined within GraspIt,
it is straightforward to estimate the image positions of a number of control points
91
used to construct the error vector.
Briefly, we present the major steps of the image based servo control. As
already mentioned, the error e is to be regulated to e = 0, corresponding to moving
the features f to fd, as defined by:
e =
u
v
= g(fd − f), (5.7)
where g is a positive scalar that controls the convergence rate of a visual servo law.
The motion of the image features and the motion of the gripper are related using
the image Jacobian in the following way:
u
v
= A
1
Z0 − x
Z−xy 1 + x2 −y
0 1
Z− y
Z−1 − y2 xy x
RGC RGCS(tGC)
0T RGC
vx
vy
vz
ωx
ωy
ωz
(5.8)
where A represents the matrix of internal camera parameters followed by a well
known image Jacobian matrix for point features. Next is the matrix needed to map
the velocities between the gripper and the camera coordinate frame followed by a
velocity screw vector of the gripper. This equation is usually written as:
u
v
= JTG, (5.9)
92
Figure 5.8: The task is planned in GraspIt by moving the object from its originallocation (empty wireframe) to its destination point and saving the new object pose.
Combining equations (Eq. 5.7) and (Eq. 5.9) we obtain:
JTG = g(fd − f). (5.10)
The robot velocity screw is computed as:
TG = g(JTWJ)−1JT W (f − fd) = gJ †(fd − f), (5.11)
where J is a model of J which is used in the control expression and W is a
symmetric positive matrix of rank 6 used to select some preferred points out of k
available ones.
In this experiment we chose a highly stable grasp around the side of the
block, (see Fig. 5.7). The flat palm of the Barrett hand allows for a strong planar
contact on one side of the object and the planar finger surfaces form line contacts
when closed around the edges of the opposite face. Looking at the projection of the
GWS into force space, shown in the middle of Fig. 5.7, we see that this grasp can
apply strong forces along the horizontal plane but is weaker in the vertical direction
since it relies purely on tangential friction forces.
93
Figure 5.9: A snapshot of the servoing near the beginning of the task. The objectis being tracked and the corners of the destination pose are shown as white dots inthe image.
Having planned a stable grasp, we executed it from the simulator as before.
Then we moved the simulated object to a desired destination pose, as shown in
Fig. 5.8, and sent this pose to the visual servoing system. Using the camera param-
eters, the corners of the object in its destination pose were projected to the image
and can be seen in Fig. 5.9, which was recorded soon after the start of the task
execution. Fig. 5.10 shows the object having reached its destination.
5.5 Conclusions
We have implemented a robotic grasping system that integrates real-time vision
with online grasp planning via a simulator to execute stable grasps on objects, and
which can also monitor a grasped object’s trajectory for correctness. The system
represents a step forward in autonomous control of tasks such as grasping.
Although the system is still in an initial phase, we believe it is an impor-
tant step in integrating real-time visual control, grasp planning, and online task
94
Figure 5.10: The object has reached its destination pose.
monitoring. All of the system components are still open research issues in the field
of robotics and we will continue to improve the performance of each of them. In
addition, our previous work in the area of multisensor fusion for grasping tasks,
[Allen et al., 1997], will help us form the planned grasps more reliably.
The visual tracking component has as its major goals improved robustness
and increased speed. The current update rate of 10Hz is far from satisfactory. To
improve the robustness in the case of significant occlusions and natural background,
we will continue to test the two implemented tracking techniques more extensively.
In addition, none of the techniques incorporates any kind of velocity or acceleration
prediction which can be easily added (e.g. a Kalman filter approach).
Currently, the grasp planning function is only semi-automated; the user
needs to place the hand in proximity of the object and an automated finger con-
tact procedure is instituted. In chapter 7, we discuss future plans to expand the
planning capabilities of GraspIt so that it can choose a stable grasp that satisfies
both task and workspace constraints.
95
Chapter 6
Adding Object Dynamics to the
Simulation
The various analyses discussed in chapter 4 are all performed on a static grasp. They
take into account the types and positions of the contacts and the configuration of
the hand, but they do not tell us how to form the grasp or even whether such an
acquisition is possible. In order to answer these questions we must consider how the
object moves over time under the influence of gravity, inertial forces, and in response
to collisions with the links of the hand and other objects in the environment.
Simulating the motion of an object that is only influenced by non-contact
forces such as gravity can be done simply and accurately using only the Newtonian
laws of physics, but when an object is not isolated from the environment and is
allowed to contact other objects, the simulation becomes much more complicated.
A collision between two objects takes place over a finite period of time and even
in the case of hard bodies, it involves a complicated surface deformation before the
objects separate again. One of the more accurate ways of modeling this interaction
96
is with finite element methods (FEMs), but these techniques are extremely com-
putationally expensive. Some of the earliest and cheapest methods of dealing with
collisions use penalty methods, where a small fixed time step is used for each iter-
ation of the simulation and objects are allowed to inter-penetrate slightly [Moore
and Wilhelms, 1988], [McKenna and Zeltzer, 1990], [Raibert and Hodgkins, 1991].
During this inter-penetration, the methods use a virtual spring acting on the two
objects to force them apart, but since this spring has no foundation in reality, the
tuning of the linear constant is usually done by hand so that the simulation “looks
good” in most instances. Unfortunately, the amount of inter-penetration (and thus
the size of the spring’s restoring force) is not only dependent on the velocities of
the bodies but also on the size of the time step which affects how often collision
checks are performed. A delayed check for collision can result in a force applied
that is significantly larger than it would be in reality.
The impulse-based approach, pioneered by Hahn [1988] and thoroughly de-
veloped in [Mirtich, 1996], considers all bodies to be unconstrained and each follows
simple ballistic trajectories. When two bodies collide, a pair of instantaneous equal
and opposite impulses are applied to the two bodies which immediately change
their velocities and prevent inter-penetration from occurring. The Impulse system
developed by Mirtich includes a sophisticated method of collision detection that
maintains a conservative estimate of collision times for each pair of nearby bodies
that prevents a collision from being missed. In each iteration of the system, the
motions of the bodies can be integrated up to the earliest impact time, and if two
bodies have collided, the collision response system uses accurate models of friction
and restitution to compute the impulses that must be applied. This simulator works
97
particularly well for systems, that have many transient contacts, like a vibrational
part feeder. However, Impulse has difficulty handling resting contacts realistically.
A book lying on a table actually bounces continually, but if the collision threshold
is small enough, this is imperceptible visually.
Articulated bodies that have tightly constrained contacts at their joints can-
not be efficiently modeled with a purely impulse-based simulation system. To han-
dle these situations, Mirtich makes use of the Featherstone algorithm [Featherstone,
1983] and creates a hybrid simulation. Ruspini and Khatib [1999] developed a con-
tact space framework that allows them to handle these types of situations more
naturally, and they have included this in their robotic simulation system called
Simpact.
Another approach is to solve for the contact forces analytically. We can
express the non-penetration constraints and frictional forces as inequalities that
can be used to formulate the problem as a linear complementarity problem (LCP).
Then we can take advantage of efficient, proven algorithms to solve the LCP and
find the correct contact forces, thus giving us an accurate simulation of the rigid
body interactions. This approach was first employed by Lotstedt [1984] and devel-
oped further by Baraff [1993]. Stewart and Trinkle [2000] improve upon the earlier
methods by handling multiple simultaneous contacts without any special formula-
tion. Their method can also handle sliding and rolling contacts with the addition
of a new scalar unknown to the traditional LCP formulation.
We found this method most appropriate for use in GraspIt, since prolonged
contact commonly occurs during a grasping task. Unfortunately, it had only previ-
ously been implemented in Matlab simulations with analytical surfaces like spheres
98
and planes. Using the collision detection and contact location framework of our
simulator, we were able to implement their dynamics method to work with our
complex polygonal bodies, a solution is guaranteed to be computable using Lemke’s
algorithm, which is a pivoting method similar to the simplex algorithm for linear
programming [Cottle et al., 1992]. Next, we show how Stewart and Trinkle cast
the Newton-Euler dynamics equations along with the non-penetration constraints
and frictional constraints as an LCP.
They start by defining q(t) and v(t) ∈ <n as the generalized coordinate and
velocity vectors of a system of rigid bodies at time t. The Newton-Euler equations
for such a system with multiple unilateral contacts can be written as follows:
M(q)dv
dt= k(q, v) + ρ, (6.1)
where M is the positive definite, symmetric inertia matrix, k is the vector of
Coriolis, centripetal, and other external generalized forces, and ρ is the sum of the
normal and friction components of the generalized contact forces.
Because the bodies occupy space, there will be some values of q that cor-
respond to inter-penetration. If for the time being we allow only a single contact,
there is an admissible portion of the body configuration space C that can be defined
with a single scalar inequality:
C = q|f(q) ≥ 0 ⊂ <n. (6.2)
We can linearize this non-penetration constraint by approximating it with the half-
space nT q ≥ α0, where n is the gradient of f with respect to q. When only the
motion of a single body is being computed, n is the six-dimensional unit wrench of
the normal component of the contact force.
99
As described earlier, the Coulomb friction law limits the possible contact
forces to a cone of possible values centered around the contact normal with a half-
angle of tan−1 µ, where µ is the coefficient of friction between the contacting sur-
faces. Again, to express this constraint linearly, they approximate the cone with a
convex polyhedron F :
F(q) =cnn + Dβ|cn ≥ 0, β ≥ 0, eTβ ≤ µcn
, (6.3)
where cn is the magnitude of the normal contact force, e = [1, 1, . . . , 1]T ∈ <k with
k being the number of edges in the polyhedral approximation, β ∈ <k is a vector of
weights, and the columns of D are direction vectors that positively span the space
of possible generalized friction forces. With this general friction formulation they
can model anisotropic Coulomb friction and torsional friction simply by modifying
the columns of D.
Contact forces can only arise when the bodies are actually in contact, so the
following constraint (which is the source of the complementarity in this problem)
ensures that the contact force is zero whenever the bodies are separated:
ρT f(q) = 0. (6.4)
Next, the Newton-Euler equations are rewritten in a discrete form so that
the model can be advanced by fixed time steps of size h. Let tl be the current
time and let ql = q(tl) and vl = v(tl) be the current estimates of the position and
velocity of the system. By approximating dvdt
with vl+1−vl
h, and dq
dtwith ql+1−ql
h, and
substituting into the Newton-Euler equation 6.1 yields:
M · (vl+1 − vl) = hncn + hDβ + hk, (6.5)
ql+1 − ql = hvl+1, (6.6)
100
where hncn and hDβ are the impulses of the normal contact force and frictional
contact force over the time period from tl to tl+1.
The following system of linear inequalities:
nT ql+1 ≥ α0 , cn ≥ 0, (6.7)
λe + DT vl+1 ≥ 0 , β ≥ 0, (6.8)
µcn − eT β ≥ 0 , λ ≥ 0, (6.9)
combined with the following complementarity conditions:
(nT ql+1 − α0)cn = 0, (6.10)
(λe + DT vl+1)T β = 0, (6.11)
(µcn − eT β)λ = 0, (6.12)
prevent inter-penetration of objects and ensure that the correct frictional force will
be used. To do this, they have introduced a new scalar unknown, λ, which is used
to satisfy the maximum work inequality that controls the direction of the frictional
force [Goyal, 1989].
The authors examine three possible states for the object, in order to describe
how these constraints work. This first state is separation and occurs when nT ql+1 >
0. Then, equations 6.7 and 6.10 imply that cn = 0, which makes sense since there is
no contact. In a frictionless system, these would be the only constraints necessary,
but in a system with friction, the left-side inequality in 6.9, which essentially encodes
Coulomb’s law, ensures that when the magnitude of the contact normal force is zero,
the magnitude of the frictional force, β, must also be zero.
The next object state is sliding, which means that vl+1 is of nonzero length
and in the span of the columns of D. Since the column vectors of D positively
101
span the space containing vl+1, the vector DT vl+1 must have at least one strictly
negative element. Now, in order to satisfy the left inequality in 6.8, λ must be
strictly positive, and when this is combined with equation 6.12, we see that the
sum of the elements of β must be µcn. Equation 6.11 with equation 6.8 implies
that typically only one element of β will be nonzero, and this will be the one which
maximizes power distribution.
The last object state is rolling, where vl+1 = 0, making the left inequality
in 6.8 equivalent to λ ≥ 0. If we choose λ = 0, the system of inequalities is reduced
to the left one in 6.9 and β ≥ 0. This simply allows the combined contact normal
force and frictional force to range over the interior and surface of the linearized
friction cone. If we choose λ > 0, then β and µcn must both be zero. This non-
generic type of rolling could occur by chance in the absence of a frictional impulse.
At this point the authors simply make the point that this is a mixed linear
complementarity problem, and it is necessary to solve for vl+1 and ql+1 to obtain a
pure LCP, but they leave this to the reader. What follows next is that derivation.
Solving equation 6.5 for vl+1, we get
vl+1 = hM−1ncn + hM−1Dβ + hM−1k + vl. (6.13)
Substituting this into equation 6.6 and solving for ql+1 we get
ql+1 = h(hM−1ncn + hM−1Dβ + hM−1k + vl) + ql. (6.14)
Next we substitute these values into inequalities 6.7 and 6.9, and write them as
follows:
nT M−1Dβ + nT M−1ncn + nT (hM−1k + vl) +ql − α0
h≥ 0, (6.15)
hDT M−1Dβ + hDT M−1ncn + eλ + DT (vl + hM−1k) ≥ 0. (6.16)
102
Finally, by separating out our unknowns from inequalities 6.15, 6.16, and 6.9 and
writing them in matrix form, we arrive at a pure LCP:
nT M−1D nT M−1n 0
hDT M−1D hDT M−1n e
−eT µ 0
cn
β
λ
+
nT (hM−1k + vl) + ql−α0
h
DT (vl + hM−1k)
0
≥ 0,
(6.17)
which may be solved using Lemke’s algorithm. Thus using the contact normal, the
fixed friction directions, the oriented mass matrix, and the current position and
velocity of the object, we can compute the magnitude of the total contact force,
which we can use in equations 6.13 and 6.14 to find the velocity and position of all
objects in the next iteration.
6.1 Implementation of the Time-Stepping Algo-
rithm in GraspIt
While the Stewart-Trinkle method is well equipped to simulate the independent
motions of several rigid bodies, we are currently only simulating the motion of the
object to be grasped, and we consider other elements of the environment such as
the workbench to be static. The hand, while not static, is kinematically controlled
through the interactions of the user or with the fixed closing velocities of an auto-
grasp. The links of the hand can contact the object and impart accelerations, but
reaction forces that would affect the links and joints are currently not considered.
Future work, described in chapter 7, will examine the benefits of simulating such
reaction forces, and will explore extensions to the system that would be required
103
to handle constrained bodies such as jointed fingers.
When the user turns on the dynamics by selecting the appropriate menu
command, the simulator sets up an idle sensor that will continually evolve the
dynamics as long as no other operations are being performed, ensuring that any
other user interactions will not be missed. The state of the object is recorded
after each iteration in two vectors, q and v, where q is a 7x1 vector of the object’s
position and orientation (stored as Euler parameters1 ), and v is a 6x1 vector of the
object’s linear and rotational velocity. The default time step, h, for each iteration
is set to 2.5 milliseconds. In a given iteration, the simulation proceeds as follows:
• If the fingers are being moved as the result of an auto-grasp, each of the joints
of the hand are closed by hθj, where θj is the velocity of joint j.
• The inertial tensor of the object, I, is computed with respect to world coor-
dinates using the current orientation of the object. Then it is combined with
the object mass, m, to form the 6x6 mass matrix:
M =
diag(m) 0
0 I
. (6.18)
• The 6x1 external force vector, k, is filled in with the force gravity exerts on
the object and the inertial moments acting on the object:
k = [0, 0,−9.81m, ω × Iω]T , (6.19)
where ω is the current angular velocity of the object.
1Euler parameters express a orientation using a unit quaternion, unlike Euler angles whichexpress orientation as rotations about the world coordinate axes. The Euler parameters arecommonly used to avoid a phenomenon known as gimbal lock (essentially a singularity) that canaffect Euler angle representations. For more information see [Kuipers, 1998].
104
• The following matrix, which we will label B, relates the angular velocity of
the object (expressed with respect to object’s coordinate frame) to the time
derivatives of the Euler parameters:
[q4, q5, q6, q7]T = B oω
q4
q5
q6
q7
=1
2
−q5 −q6 −q7
q4 −q7 q6
q7 q4 −q5
−q6 q5 q4
oωx
oωy
oωz
. (6.20)
Using this conversion matrix, we build a 7x6 matrix, G, that converts the
object velocity, expressed in world coordinates, to the time derivative of the
object position:
q = Gv
q =
1 0 0
0 1 0 0
0 0 1
0 BRT
v, (6.21)
where R is the current orientation of the object expressed as a 3x3 rotation
matrix.
• If the object was not in contact with any other object at then end of the last
iteration we can simply integrate the equations of motion using the time step,
h, to arrive at a new velocity and position for the object:
vl+1 = hM−1k + vl, (6.22)
ql+1 = hGvl+1. (6.23)
105
• If the object was in contact with another object, we must formulate the LCP
by constructing the matrix and the vector that are described by equation 6.17.
If there are m contacts, n is a 6xm matrix where each column is a contact
normal transformed into generalized wrench space coordinates, and since we
are approximating friction cones as a convex sum of 8 direction vectors, D is
a 6x8m matrix where each set of 8 columns are the friction directions for a
given contact transformed into generalized coordinates. The q l − α0 term is
an mx1 vector that contains the penetration distance (a negative scalar) for
each contact. The LCP solver uses Lemke’s algorithm, and if there exists a
possible set of forces that fulfills the friction and non-penetration constraints,
it returns the values of the three unknowns, cn, β, λ. Plugging cn and β into
equation 6.13 gives the new velocity of the object, and using that velocity
and equation 6.23 the new object position can be determined.
• With the fingers and object in their new positions, the collision detection
system determines if the object has collided with any other body. If there
are no collisions, the iteration is over. If there are, the system moves the
fingers and object back to their positions at the start of the iteration and
begins a binary search for the point at which the bodies are about to collide
but are separated by less than the contact threshold distance (currently 0.1
mm). Once this point is found, the system adds the new contacts and ends
the iteration.
This last step of finding contact points is essentially the same as the collision
detection and contact location steps described for static grasps (see sections 3.3
and 3.4) except the object is moving during the search for the time of impact.
106
Stewart and Trinkle do not perform contact searches because their simulations only
involve analytical surfaces which allow them to directly compute the contact points.
Unfortunately, the contact search slows down the simulation considerably, and fu-
ture research will look at improving the search technique to reduce the number of
iterations necessary to find the time of impact.
Since objects are never allowed to interpenetrate at the end of an iteration of
the dynamic simulation, it would seem the penetration distance should always be 0.
However, in order for the object to move in response to contact by the kinematically
controlled links of the hand, the penetration distance used in the LCP must be
negative, thus indicating penetration and the contact forces returned by the solver
will be sufficient to move the object out of that situation. When two objects are
within the contact threshold, the contact location system returns a new contact
on each body, and the penetration distance is defined as the distance between the
contact point on each body (less than 0.1mm) minus the contact threshold distance
(0.1mm). It is like creating a skin around the object that is offset from its surface by
0.1mm, and when this thin layer is breached, the object moves away. If during the
dynamic creation of a grasp, a force-closure grasp is achieved, the LCP solver will
fail because the object is secure and cannot move to restore the non-interpenetration
constraint. When this happens the dynamics are turned off automatically and the
grasp is analyzed.
6.2 Simulating Grasp Formation
With the object dynamics in place, it is possible to study the temporal formation of
grasps. In this section, we present two examples of this dynamic simulation. In the
107
Figure 6.1: Initial position of the Barrett hand and handset for the first dynamicsexperiment.
first example, the Barrett hand2 is positioned above the phone handset which rests
on the workbench, as shown in figure 6.1. Leaving the object in a static state, an
auto-grasp is performed, and the result is shown in figure 6.2. This is not a force-
closure grasp because the fingers only contact the handset on its bottom surface.
However, by turning on the dynamic simulation, and performing the auto-grasp
again, the handset is pulled up into the hand until it reaches a force-closure state
(figure 6.3). Without using the dynamic simulation, this grasp could not have been
performed since the workbench would block the fingers from wrapping around the
phone.
In the next example, the phone handset rests on its side, and the Barrett
hand is again positioned above it (see figure 6.4). In this configuration, a force-
closure grasp, where the fingers pinch the phone near the top, is possible (see
figure 6.5). However, if we adjust the joint velocity of the thumb to be twice that of
the fingers, the object is disturbed during the grasp formation and the grasp fails
(see figure 6.6). This is an example of how the timing of the contact formation can
2Again, we use a simplified version of this hand to keep the number of contact points at areasonable size during these experiments.
108
Figure 6.2: An auto-grasp of the static handset does not result in a force-closuregrasp.
affect the final outcome of the grasp.
Having the object move in response to contacts adds additional realism to
the grasps that are created, and in some situations makes it easier to perform the
grasp. Grasps created with a parallel jaw gripper are a good example of this.
Typically a parallel jaw gripper has only one degree of freedom, but the model
used in GraspIt requires independently movable plates. The reason for this is if our
model had only one DOF, and the object remained static, the gripper would have
to be perfectly centered over the object for both plates to come into contact with
the object. However, with object dynamics turned on, the object can slide along
the workbench so that it is self centered within the gripper.
109
(a) Iteration 180 (b) Iteration 200
(c) Iteration 300 (d) Iteration 400
(e) Iteration 500 (f) Final Position
Figure 6.3: An auto-grasp of the handset with dynamics turned on results in thehandset being pulled into the hand, and the final grasp has force-closure.
110
Figure 6.4: Initial position of the Barrett hand and handset for the second dynamicsexperiment.
Figure 6.5: An auto-grasp of the static handset results in a force-closure grasp.
111
(a) Iteration 255 (b) Iteration 290
(c) Iteration 350 (d) Iteration 390
(e) Iteration 435 (f) Iteration 470
Figure 6.6: An auto-grasp of the handset with dynamics turned on results in afailure to grasp the handset, because the thumb is closing at twice the rate of theother two fingers. Since the contacts do not occur simultaneously, the object isdisturbed.
112
Chapter 7
Conclusions and Future Work
This thesis has presented a unique and powerful tool for robotic grasping. In this
final chapter, we outline the contributions we have made, and describe our future
plans to extend this work.
7.1 Contributions
In creating GraspIt, several problems were addressed:
• The front-end design of the system, presented in chapter 3, had to fulfill
several requirements. First, it was designed to be flexible so that a wide
variety of hand designs could be imported. Changes in the hand construction
can be made easily by editing parameters in a compact configuration file. The
user interface of the system was designed to be simple and intuitive to use,
and it allows an operator to quickly manipulate the hand to form different
grasps of an object. Existing collision detection software was employed to
prevent links from passing through objects, and it was augmented to find the
113
contact regions between two bodies.
• As each contact between a link of the hand and an object is detected, GraspIt
reanalyzes the grasp and presents the user with the results in real-time. Chap-
ter 4 discusses in detail the types of analyses that are performed. While the
system uses existing quality metrics and grasp force optimization techniques,
it is the first time they have been put to use in a general way to analyze ar-
bitrary grasps. GraspIt also has novel visualization methods. These include
the ability to see where the grasp is the weakest and a method of visualizing
the 6-dimensional grasp wrench space of a particular grasp by allowing the
user to create any 3-dimensional projection of that space.
• Planning grasps in simulation is useless unless the grasps can be performed by
an actual robotic hand. Section 3.2.1 shows how a full robotic work cell can
be modeled in GraspIt, including environmental objects and the robot arm,
and chapter 5 presents a real-time model-based vision system that is used to
bridge the gap between this simulated environment and the real world. Several
experiments in that chapter demonstrate that grasps planned in simulation
can be accurately realized with a physical robotic arm and hand.
• If the simulation keeps the object to be grasped static at all times, it is
possible to plan non-force-closure grasps that would be difficult to perform in
reality. Additionally, as the hand is closed around an object, it might disturb
it from its original position and the resulting grasp will be different from the
one planned. Chapter 6 presents a dynamic simulation system that models
the motion of the object as it is contacted by the moving links, and it can be
114
used to accurately study the grasp formation process.
7.2 Future Research Directions
In its current state, GraspIt is a useful tool for those who are planning grasping
tasks or designing robot hands, but there are some limitations. The following items
describe these limitations which will lead to future extensions of GraspIt.
• The focus of this work has been on force-closure grasps which are useful for
pick and place type tasks. When a task involves manipulation of the object,
it is necessary to plan a manipulable grasp where arbitrary velocities can
be imparted to the object. By incorporating manipulability and dexterity
metrics that are reviewed by Shimoga [1996], these types of grasps could be
analyzed as well. Furthermore, the dynamic simulation would be very useful
in verifying that a manipulation plan will achieve the desired outcome.
• Currently, the force-closure quality metrics assume nothing is known about
the space of external forces that might be applied to a grasped object during
the execution of the task. If certain task forces are more likely to occur than
others, it would be useful to provide an interface for the operator to specify
this task wrench space. Then the quality metrics would more accurately
reflect the quality of the grasp for the given task. One possible method
involving the use of wrench space ellipsoids is described in [Li et al., 1989],
but in an interactive setting there may be other possible approaches.
• When performing the grasps that are planned in simulation, an endpoint
open-loop visual servoing method was used to move the hand into position
115
to grasp the object. Further optimization of the tracking system speed and
an articulated wire frame hand model would allow the use of a closed loop
tracking scheme that would be more robust to system calibration errors.
• By incorporating the Featherstone algorithm [Featherstone, 1983] for com-
puting the dynamics of articulated bodies it would be possible to move from
treating the hand as an completely stiff user-controlled element that is unaf-
fected by external forces, to a more accurate simulation of the joint reactions
as links contact the object. In addition, further work needs to be done to
optimize the dynamic simulation. In a power grasp, where there are several
contacts between the links and the objects, the size of the LCP becomes large
enough to slow the solver down significantly.
• The last and perhaps most exciting area is automatic grasp selection. This
is an extremely difficult problem given the large dimensionality of the search
space of possible grasps, and although it has been studied extensively, little
progress has been made. Perhaps what is necessary is a blending of the high
and low level grasp synthesis techniques described in chapter 2. By using a
high level knowledge based approach to choose an appropriate preshape for
the grasp, we may be able to greatly limit the number of configurations to
be analyzed. A system would provide a small set of possible starting points,
each of which could be locally optimized using a search algorithm that per-
turbs the hand configuration and performs the grasp analysis at each iteration.
The system would then report the best resulting grasp. It will be necessary to
investigate various search techniques that work with non-differentiable func-
tions and to find a way to deal with the rapid changes in the quality surface
116
due to surface normal discontinuities. If the configuration space can be re-
parameterized to keep the links of the hand in contact with the object, thus
limiting the overall dimensionality of the search, it may be possible to use
genetic algorithms to find optimal grasps. During this optimization, which
would probe the quality surface for a given hand and object, we can attempt
to find large regions of high quality. This would allow us to find grasps that
are robust given hand positioning errors.
In conclusion, the combination of grasping analysis with interactive simulation has
resulted in a novel tool for robotic grasping, and our future research will add new
capabilities that will only further extend its usefulness.
117
Bibliography
[Allen et al., 1997] Peter K. Allen, Andrew T. Miller, Paul Y. Oh, and Brian S.
Leibowitz. Using tactile and visual sensing with a robotic hand. In Proc. of the
1997 IEEE International Conference on Robotics and Automation, Albuquerque,
NM, April 1997.
[Araujo et al., 1996] H. Araujo, R. L. Canceroni, and C. M. Brown. A fully pro-
jective formulation for Lowe’s tracking algorithm. Technical report 641, The
University of Rochester, CS Department, Rochester, NY, November 1996.
[Baraff, 1993] D. Baraff. Issues in computing contact forces for non-penetrating
rigid bodies. Algorithmica, pages 292–352, October 1993.
[Barber et al., 1996] C. B. Barber, D. P. Dobkin, and H. Huhdanpaa. The Quick-
hull algorithm for convex hulls. ACM Transactions on Mathematical Software,
22(4):469–483, December 1996.
[Borst et al., 1999] Ch. Borst, M. Fischer, and G. Hirzinger. A fast and robust
grasp planner for arbitrary 3D objects. In Proc. of the 1999 IEEE International
Conference on Robotics and Automation, pages 1890–1896, Detroit, MI, May
1999.
118
[Buss et al., 1996] M. Buss, H. Hashimoto, and J. Moore. Dextrous hand grasping
force optimization. IEEE Transactions on Robotics and Automation, 12(3):406–
418, 1996.
[Butterfass et al., 1998] J. Butterfass, G. Hirzinger, S. Knoch, and H. Liu. DLR’s
multisensory articulated hand, part I: Hard- and software architecture. In Proc.
of the 1998 IEEE International Conference on Robotics and Automation, pages
2081–2086, Leuven, Belgium, 1998.
[Chen and Burdick, 1992] I-Ming Chen and Joel W. Burdick. Finding antipodal
point grasps on irregularly shaped objects. In Proc. of the 1992 IEEE Interna-
tional Conference on Robotics and Automation, pages 2278–2283, Nice, France,
1992.
[Cottle et al., 1992] R. W. Cottle, J. S. Pang, and R. E. Stone. The Linear Com-
plementarity Problem. Academic Press, 1992.
[Cremer and Stewart, 1989] James F. Cremer and A. James Stewart. The archi-
tecture of newton, a general-purpose dynamics simulator. In Proc. of the 1991
IEEE International Conference on Robotics and Automation, pages 1806–1811,
Scottsdale, AZ, May 1989.
[Cutkosky and Wright, 1986] M. R. Cutkosky and P. K. Wright. Modeling manu-
facturing grips and correlation with the design of robotic hands. In Proc. of the
1986 IEEE International Conference on Robotics and Automation, pages 1533–
1539, San Francisco, CA, 1986.
119
[DeLaurentis et al., 2000] K. J. DeLaurentis, C. Pfeiffer, and C. Mavroidis. De-
velopment of a shape memory alloy actuated hand. In Proceedings of the 7th
International Conference on New Actuators, pages 281–284, Bremen, Germany,
June 2000. Messe Bremen GMBH.
[Dementhon and Davis, 1995] D. Dementhon and L. S. Davis. Model–based object
pose in 25 lines of code. International Journal of Computer Vision, 15:123–141,
1995.
[Derby, 1983] Stephen Derby. Simulating motion elements of general-purpose robot
arms. International Journal of Robotic Research, 2(1):3–12, Spring 1983.
[Ding et al., 2000] Dan Ding, Yun-Hui Liu, and Shuguo Wang. Computing 3-D
optimal form-closure grasps. In Proc. of the 2000 IEEE International Conference
on Robotics and Automation, pages 3573–3578, San Fransisco, CA, April 2000.
[Featherstone, 1983] R. Featherstone. The calculation of robot dynamics using
articulated-body inertias. International Journal of Robotics Research, 2(1):13–
30, 1983.
[Ferrari and Canny, 1992] Carlo Ferrari and John Canny. Planning optimal grasps.
In Proc. of the 1992 IEEE International Conference on Robotics and Automation,
pages 2290–2295, Nice, France, 1992.
[Goyal, 1989] S. Goyal. Planar Sliding of a Rigid Body with Dry Friction: Limit
Surfaces and Dynamics of Motion. PhD thesis, Cornell University Department
of Mechanical Engineering, 1989.
120
[Griffiths, 1943] H. E. Griffiths. Treatment of the injured workman. Lancet, i:729,
1943.
[Hager and Toyama, 1998] G. Hager and K. Toyama. The XVision system: A
general-purpose substrate for portable real–time vision applications. Computer
Vision and Image Understanding, 69(1):22–37, January 1998.
[Hahn, 1988] James K. Hahn. Realistic animation of rigid bodies. Computer Graph-
ics, 22(4):299–308, August 1988.
[Han et al., 2000] L. Han, J. C. Trinkle, and Z. Li. Grasp analysis as linear matrix
inequality problems. IEEE Transactions on Robotics and Automation, 16(6):663–
674, December 2000.
[Hanafusa and Asada, 1982] H. Hanafusa and H. Asada. Stable prehension by a
robot hand with elastic fingers. In M. Brady, editor, Robot Motion Planning and
Control, pages 323–336. MIT Press, 1982.
[Hester et al., 1999] R. D. Hester, M. Cetin, C. Kapoor, and D. Tesar. A criteria-
based approach to grasp synthesis. In Proc. of the 1999 IEEE International
Conference on Robotics and Automation, pages 1255–1260, Detroit, MI, May
1999.
[Hunt, 1978] K.H. Hunt. Kinematic Geometry of Mechanisms. Oxford University
Press, 1978.
[Hutchinson et al., 1996] S. Hutchinson, G. D. Hager, and P. I. Corke. A tutorial on
visual servo control. IEEE Transactions on Robotics and Automation, 12(5):651–
670, 1996.
121
[Iberall, 1987] T. Iberall. The nature of human prehension: Three dextrous hands
in one. In Proc. of the 1987 IEEE International Conference on Robotics and
Automation, pages 396–401, 1987.
[Iberall, 1997] Thea Iberall. Human prehension and dexterous robot hands. The
International Journal of Robotics Research, 16(3):285–299, June 1997.
[Jameson, 1985] J. W. Jameson. Analytic Techniques for Automated Grasp. PhD
thesis, Stanford University Department of Mechanical Engineering, 1985.
[Kerr and Roth, 1986] J. Kerr and B. Roth. Analysis of multifingered hands. In-
ternational Journal of Robotics Research, 4(4):3–17, Winter 1986.
[Kirkpatrick et al., 1990] D.G. Kirkpatrick, B. Mishra, and C. K. Yap. Quantita-
tive Steinitz’s theorems with applications to multifingered grasping. In Proc. of
the 20th ACM Symposium on Theory of Computing, pages 341–351, Baltimore,
Maryland, 1990.
[Kragic et al., 2001] D. Kragic, A. Miller, and P. Allen. Real-time tracking meets
online grasp planning. In Proc. of the 2001 IEEE International Conference on
Robotics and Automation, Seoul, Republic of Korea, May 2001.
[Kuipers, 1998] Jack B. Kuipers. Quaternions and Rotation Sequences. Princeton
University Press, 1998.
[Lakshminarayana, 1978] K. Lakshminarayana. Mechanics of form closure. Tech-
nical Report 78-DET-32, ASME, 1978.
122
[Larsen et al., 1999] E. Larsen, S. Gottschalk, M. Lin, and D. Manocha. Fast prox-
imity queries with swept sphere volumes. Technical Report TR99-018, Dept. of
Computer Science, University of North Carolina, Chapel Hill, 1999.
[Li and Sastry, 1988] Z. Li and S. Sastry. Task-oriented optimal grasping by mul-
tifingered robot hands. IEEE Journal of Robotics and Automation, 4(1):32–44,
February 1988.
[Li et al., 1989] Zexiang Li, Ping Hsu, and Shankar Sastry. Grasping and coor-
dinated manipulation by a multifingered robot hand. International Journal of
Robotics Research, 8(4):33–50, August 1989.
[Lide, 1995] David R. Lide, editor. CRC Handbook of Chemistry and Physics, pages
15–40 to 42. CRC Press, seventy-fifth edition, 1995.
[Lotstedt, 1984] Per Lotstedt. Numerical simulation of time-dependent contact
and friction problems in rigid body mechanics. SIAM Journal on Scientific and
Statistical Computing, 5(2):370–393, June 1984.
[Lovchik and Diftler, 1999] C. S. Lovchik and M. A. Diftler. The Robonaut hand:
A dexterous robot hand for space. In Proc. of the 2000 IEEE International
Conference on Robotics and Automation, pages 907–912, Detroit, MI, May 1999.
[Lowe, 1992] D. G. Lowe. Robust model–based motion tracking through the in-
tegration of search and estimation. International Journal of Computer Vision,
8(2):113–122, 1992.
123
[Lyons, 1985] D. Lyons. A simple set of grasps for a dextrous hand. In Proc.
of the 1985 IEEE International Conference on Robotics and Automation, pages
588–593, 1985.
[Marchand et al., 2000] E. Marchand, P. Bouthemy, and F. Chaumette. A 2D–3D
model–based approach to real–time visual tracking. Technical report, INRIA,
March 2000.
[Markenscoff and Papadimitriou, 1989] Xanthippi Markenscoff and Christos H. Pa-
padimitriou. Optimum grip of a polygon. International Journal of Robotics
Research, 8(2):17–29, April 1989.
[Markenscoff et al., 1990] Xanthippi Markenscoff, Luqun Ni, and Christos H. Pa-
padimitriou. The geometry of grasping. International Journal of Robotics Re-
search, 9(1):61–74, February 1990.
[McBride, 1942] E. D. McBride. Disability Evaluation. J.B. Lippincott Co.,
Philadelphia, third edition, 1942.
[McKenna and Zeltzer, 1990] Michael McKenna and David Zeltzer. Dynamic sim-
ulation of autonomous legged locomotion. In Computer Graphics, pages 29–38.
SIGGRAPH Conference Proceedings, 1990.
[Mechanical Dynamics, 2001] Mechanical Dynamics. http://www.adams.com,
2001.
[Miller and Allen, 1999] Andrew T. Miller and Peter K. Allen. Examples of 3D
grasp quality computations. In Proc. of the 1999 IEEE International Conference
on Robotics and Automation, pages 1240–1246, 1999.
124
[Miller and Allen, 2000] Andrew T. Miller and Peter K. Allen. Graspit!: A versatile
simulator for grasping analysis. In Proc. of the ASME Dynamic Systems and
Control Division, volume 2, pages 1251–1258, Orlando, FL, 2000.
[Mirtich and Canny, 1994] Brian Mirtich and John Canny. Easily computable opti-
mum grasps in 2-D and 3-D. In Proc. of the 1994 IEEE International Conference
on Robotics and Automation, pages 739–747, San Diego, CA, May 1994.
[Mirtich, 1996] Brian Mirtich. Impulse-based Dynamic Simulation of Rigid Body
Systems. PhD thesis, Department of Computer Science, University of California
at Berkeley, 1996.
[Mishra and Silver, 1989] Bud Mishra and Naomi Silver. Some discussion of static
gripping and its stability. IEEE Journal on Systems, Man, and Cybernetics,
19(4):783–796, August 1989.
[Moore and Wilhelms, 1988] Matthew Moore and Jane Wilhelms. Collision detec-
tion and response for computer animation. Computer Graphics, 22(4):289–298,
August 1988.
[Napier, 1956] J. Napier. The prehensile movements of the human hand. Journal
of Bone and Joint Surgery, 38B(4):902–913, November 1956.
[Nguyen, 1986] Van-Duc Nguyen. Constructing force-closure grasps. In Proc. of
the 1986 IEEE International Conference on Robotics and Automation, pages
2290–2295, San Francisco, CA, 1986.
[Pao and Speeter, 1989] Lucy Pao and Thomas H. Speeter. Transformation of hu-
man hand positions for robotic hand control. In Proc. of the 1989 IEEE Inter-
125
national Conference on Robotics and Automation, pages 1758–1763, Scottsdale,
AZ, May 1989.
[Pollard, 1994] N. S. Pollard. Parallel Methods for Synthesizing Whole-Hand
Grasps from Generalized Prototypes. PhD thesis, Department of Electrical En-
gineering and Computer Science, Massachusetts Institute of Technology, 1994.
[Ponce et al., 1993a] Jean Ponce, Darrell Stam, and Barnard Faverjon. On com-
puting two-finger force-closure grasps of curved 2D objects. International Journal
of Robotics Research, 12(3):263–273, June 1993.
[Ponce et al., 1993b] Jean Ponce, Steven Sullivan, Jean-Daniel Boissonnat, and
Jean-Pierre Merlet. On characterizing and computing three- and four-finger force-
closure grasps of polyhedral objects. In Proc. of the 1993 IEEE International
Conference on Robotics and Automation, pages 821–827, Atlanta, Georgia, May
1993.
[Preparata and Shamos, 1985] Franco P. Preparata and Michael Ian Shamos. Com-
putational Geometry. Springer-Verlag, New York, 1985.
[Raibert and Hodgkins, 1991] Marc H. Raibert and Jessica K. Hodgkins. Anima-
tion of dynamic legged locomotion. In Computer Graphics, pages 349–358. SIG-
GRAPH Conference Proceedings, 1991.
[Reuleaux, 1876] F. Reuleaux. The Kinematics of Machinery. Macmillan and Com-
pany, 1876. Republished by Dover in 1963.
[Ruspini and Khatib, 1999] Diego C. Ruspini and Oussama Khatib. Colli-
sion/contact models for the dynanmic simulation and haptic interaction. In Proc.
126
of the Ninth International Symposium of Robotics Research, ISRR’99, pages 185–
194, Snowbird, UT, 1999.
[Salisbury, 1982] J. K. Salisbury. Kinematic and Force Analysis of Articulated
Hands. PhD thesis, Department of Mechanical Engineering, Stanford Univer-
sity, 1982.
[Shimoga, 1996] K. B. Shimoga. Robot grasp synthesis algorithms: A survey. In-
ternational Journal of Robotics Research, 15(3):230–266, June 1996.
[Slocum and Pratt, 1946] D. B. Slocum and D. R. Pratt. Disability evaluation for
the hand. Journal of Bone and Joint Surgery, 28:491, 1946.
[Smith et al., 1999] G. Smith, E. Lee, K. Goldberg, K. Bohringer, and J. Craig.
Computing parallel-jaw grips. In Proc. of the 1999 IEEE International Confer-
ence on Robotics and Automation, Detroit, MI, May 1999.
[Somoff, 1900] P. Somoff. Uber gebiete von schraubengeschwindigkeiten eines star-
ren korpers bie verschiedener zahl von stutzflachen. Zeitschrift fur Mathematik
und Physik, 45:245–306, 1900.
[Speeter, 1991] Thomas H. Speeter. Primitive based control of the Utah/MIT dex-
trous hand. In Proc. of the 1991 IEEE International Conference on Robotics and
Automation, pages 866–877, Sacramento, CA, April 1991.
[Stansfield, 1991] S. A. Stansfield. Robotic grasping of unknown objects:
A knowledge-based approach. International Journal of Robotics Research,
10(4):314–326, August 1991.
127
[Stewart and Trinkle, 2000] David Stewart and J. C. Trinkle. An implicit time-
stepping scheme for rigid body dynamics with coulomb friction. In Proc. of the
2000 IEEE International Conference on Robotics and Automation, pages 162–
169, San Fransisco, CA, April 2000.
[Trinkle, 1992] Jeffrey C. Trinkle. A quantitative test for form closure grasps. In
Proc. of the 1992 IEEE/RSJ International Conference on Intelligent Robots and
Systems, pages 1670–1676, Raleigh, North Carolina, 1992.
[Ulrich et al., 1988] Nathan Ulrich, Richard Paul, and Ruzena Bajcsy. A medium-
complexity compliant end effector. In Proc. of the 1988 IEEE International
Conference on Robotics and Automation, pages 434–439, Philadelphia, PA, April
1988.
[Wernecke, 1994] Josie Wernecke. The Inventor Mentor. Addison Wesley, 1994.
[Wu et al., 1996] S.-P. Wu, L. Vandenberghe, and S. Boyd. MAXDET: Software
for Determinant Maximization Problems. User’s guide, Alpha version. Stanford
University, April 1996.
[Wunsch and Hirzinger, 1997] P. Wunsch and G. Hirzinger. Real–time visual track-
ing of 3D objects with dynamic handling of occlusion. In Proc. of the IEEE In-
ternational Conference on Robotics and Automation, pages 2868–2872, Grenoble,
France, 1997.