a spring model for whole-hand virtual grasping

Christoph W. Borst*Arun P. IndugulaCenter for Advanced ComputerStudiesUniversity of Louisiana at LafayetteLafayette LA 70504-4330

*Correspondence [email protected]

Presence, Vol. 15, No. 1, February 2006, 47–61

© 2006 by the Massachusetts Institute of Technology

A Spring Model for Whole-HandVirtual Grasping

Abstract

We present a physically-based approach to grasping and manipulation of virtualobjects that produces visually realistic results, addresses the problem of visual inter-penetration of hand and object models, and performs force rendering for force-feedback gloves, in a single framework. Our approach couples a simulation-controlled articulated hand model to tracked hand configuration using a system oflinear and torsional virtual spring-dampers. We discuss an implementation of ourapproach that uses a widely available simulation tool for collision detection and re-sponse. We pictorially illustrate the resulting behavior of the virtual hand model andof grasped objects, discuss user behavior and difficulties encountered, and showthat the simulation rate is sufficient for control of current force-feedback glove de-signs. We also present a prototype system for natural whole-hand interactions in adesktop-sized workspace.

1 Introduction

1.1 The Grasping Problem

Various interaction techniques are useful in virtual environments. Someof the basic tasks that need to be supported are navigation of an environment,selection of objects or options, exploration of objects, and manipulation ofobject pose or other properties. In some applications, these interactions areadequately supported by specialized devices and techniques that do not neces-sarily duplicate real-world interactions, for example, see the summary by Bow-man, Kruijff, LaViola, and Poupyrev (2001). However, in other applications, itis important to duplicate real-world behavior. For example, a training systemfor a manual assembly task benefits from a realistic system for object graspingand manipulation. A variety of sensing gloves and other hand-tracking tech-nologies has been developed to provide the whole-hand input needed for suchapplications, but the quality of simulated grasping and manipulation has beenlimited.

A main source of difficulty in virtual grasping is that a real hand cannot beexpected to obey the laws of physics with respect to a simulated environment.That is, a virtual environment cannot physically constrain real hand motion likea real environment would. This can result in unrealistic hand motion accompa-nied by visually distracting artifacts such as a hand model sinking into a virtualobject and a virtual grasp appearing frictionless. Visual feedback can be im-proved by keeping the virtual hand model outside of the object, but then thevisual feedback no longer accurately represents the real hand configuration. Arelated problem is that physical sensations of friction and weight are important

Borst and Indugula 47

for human grasp control (Johansson, 1998) but are notprovided by virtual objects, further limiting the realismof a user’s grasping motions. Finally, limited accuracy ofwhole-hand input devices and of geometric hand mod-els also limits achievable interaction quality.

Haptic feedback devices alleviate the first two prob-lems somewhat, but do not solve them. Glove-basedforce feedback devices have been shown to improvegrasping control (Fabiani & Burdea, 1996), but theyhave severely limited degrees of freedom (typically onenormal force per fingertip) and only a limited objectstiffness can be simulated by their actuators due to me-chanical and control characteristics. We therefore do notexpect glove-based devices to provide accurate motionconstraints or sensations in the near future.

1.2 Our Work

In this paper, we present a system for whole-handgrasping and manipulation using dynamic simulation.We prefer the generality of a physically-based approachover approaches that depend on the heuristic analysis ofgrasp stability or user intent, since these tend not tosolve the problem in a general way and only handle se-lected cases. A physically-based approach can also takeadvantage of forthcoming improvements in physicalsimulation tools, which will in turn improve the qualityof the results, while improving a heuristic approachwould require development of more complex heuristicsor further work on the basic grasping mechanism itself.A grasping system that is integrated with physical simu-lation tools will also naturally extend from rigid bodyinteractions to interactions with deformable bodies.

We also sought to develop an approach that would re-duce distracting visual artifacts. Due to the phenomenonof visual capture (Welch & Warren, 1980) and the highsensitivity of users to visual interpenetration artifacts(Burns et al., 2005), we expect visual feedback that is notan exact representation of real hand configuration to beless distracting to users than an exact representation thatsuffers from the visual hand-object interpenetration arti-fact.

Finally, we wanted the grasping system to perform forcerendering for force-feedback gloves. Our grasping ap-

proach includes force rendering for devices such as theImmersion CyberGrasp and the Rutgers Master, but it canalso be used without the force-feedback component.

In the next section, we summarize previous work onvirtual grasping and on haptic rendering for glove-typedevices. In Section 3, we present a system we are devel-oping for natural interactions in a desktop-sized envi-ronment. Section 4 describes our physically-basedgrasping approach. Section 5 discusses our experienceswith the approach and evaluates simulation speed.

The main contributions described in this paper are:

● We present a spring model that, in a single frame-work, supports virtual grasping and manipulation,addresses the problem of visual artifacts describedabove, and provides a new force rendering approachfor force-feedback gloves.

● We evaluate an implementation of our spring modelusing a widely-available simulation tool for collisiondetection and response. Specifically, we illustratethe resulting grasp behavior, discuss user behaviorand difficulties encountered, and analyze simulationrate.

2 Previous Work

The previous section outlined our design philoso-phy and motivated the use of a physically-based ap-proach rather than a heuristic approach. In this section,we discuss previous related work. Note that we do notdiscuss grasping approaches for autonomous control ofrobots or virtual humans because these focus on thedifferent problem of grasp planning. Concepts such asforce closure and form closure from robotics can be use-ful for heuristic approaches that need to identify whengrasping has occurred, and planning concepts can beuseful for systems such as the grasping automata de-scribed by Boulic, Rezzonico, and Thalmann (1996),but our approach falls in neither of these categories.

2.1 Point-Based Grasping Approaches

Early grasping systems typically fixed an object tothe hand’s coordinate system once a simple test de-

48 PRESENCE: VOLUME 15, NUMBER 1

tected a grasp. For example, Iwata’s point-based test(Iwata, 1990) checked 16 points on a hand model forcontact with a virtual object and considered the objectto be captured when it was simultaneously contacted bypoints on both the thumb and any other finger. Berga-masco, Degl’Innocenti, and Bucciarelli (1994) intro-duced physically-based object response with a point-based force calculation. Control point grids placed onpalmar sides of virtual hand segments were used tocompute a force vector that included static friction, dy-namic friction, normal contact forces, and externalforces. Recently, Hirota and Hirose (2003) developed amanipulation system using point-based collision re-sponse forces from a much larger number of points on ahand model. Collision detection and response werebased on the motion of interface points into a tetrahe-dral mesh model of a manipulated object. A video ac-companying their paper demonstrated realistic objectmotion in response to various manipulations.

One problem with point-based approaches is that col-lision detection is not accurate unless a very large num-ber of points is used to densely cover the hand. Anotherproblem is that force applied to the manipulated objectcan be underestimated for sharp or wedge-shaped ob-jects, as pointed out by Hirota and Hirose. We haveobserved a similar problem in the use of grids of pointsfor force rendering for force-feedback gloves (Borst &Volz, 2003). Finally, the point-based approaches de-scribed here did not prevent visual hand-object inter-penetration.

2.2 Approaches to Address Hand-Object Interpenetration

Several methods have been proposed for address-ing hand-object interpenetration. Zachmann and Rettigdescribed a grasping system based on heuristic analysisof contact (Zachmann & Rettig, 2001). Their approachincluded a minimization process to find a joint anglevector that kept fingers outside a grasped object. Boulicet al. instead adjust finger posture by opening the handone joint at a time to move fingers out of a grasped ob-ject (Boulic et al., 1996). In their grasping approach,spherical sensors on the hand model guide a grasping

automaton that controls grasp after user intent is deter-mined, and grasped object motion is kinematic. In asystem for manipulation of virtual objects using virtualchopsticks (Kitamura, Higashi, Masaki, & Kishino,1999), an object was considered grasped when bothchopsticks contacted it, and finger angles were adjustedsuch that the chopsticks did not penetrate the objectvisually.

Ullmann and Sauer (2000) used a physically-moti-vated heuristic for establishing grasps. For two-handedgrasping, their system displayed both “ghost hands”that matched tracked hand configuration and solid handmodels that remained attached to the outside of thegrasped object. Immersion’s Virtual Hand Toolkit in-cludes an algorithm, also termed “ghost hand,” thatadds an offset vector to tracker position to preventhand-object interpenetration. The offset vector is thengradually reduced to zero when interpenetration wouldno longer result (DesRosiers, Gomez, Tremblay, & Ull-rich, 2001). This appears similar in principle to a re-cently proposed Credible Avatar Limb Motion tech-nique (Burns et al., 2005).

None of these approaches incorporated realisticallysimulated dynamics to the extent we would like to see.However, in animation research, interpenetration ofhuman-controlled virtual articulated structures intotheir environments has been addressed by using dy-namic controllers or virtual springs to drive dynamicsimulations based on human input (e.g., see Kokkevis,Metaxas, & Badler, 1996; Westenhofer & Hahn, 1996).Interpenetration is similarly prevented for unarticulatedvirtual tools coupled to haptic device handles by spring-dampers used in several force-feedback rendering algo-rithms (e.g., McNeely, Puterbaugh, & Troy, 1999).This force-feedback rendering technique has also beengeneralized and applied to a low-DOF articulated la-proscopy simulator by per-segment couplings that linksegments of the virtual tool to segments of a physicaldevice (Meseure, Lenoir, Fonteneau, & Chaillou,2004). Outside of haptics research, virtual springs havebeen used for interaction techniques that couple trackedhand-held devices to manipulated objects controlled byphysical simulation and collision response (Froehlich,Tramberend, Beers, Agrawala, & Baraff, 2000; Koutek


& Post, 2001). Our grasping system also couples thephysical world to a dynamic simulation using virtualsprings, although our spring configuration is tailored tothe whole-hand grasping problem and is not a directapplication of any of these methods.

2.3 Force-Feedback Rendering inGrasping Systems

Most work in force-feedback rendering has fo-cused on unarticulated virtual tools. One common tech-nique is a constraint-based approach (e.g., Ruspini, Ko-larov, & Khatib, 1997; Zilles & Salisbury, 1995), tobase force on an interface point’s distance from anotherpoint constrained to move along the surface of a pene-trated object. Other basic approaches include ray-based(Basdogan, Ho, & Srinivasan, 1997) and voxmap (Mc-Neely et al., 1999) techniques.

Iwata’s grasping system added force feedback bytransmitting maximum available torque to fingers thatcontacted a rigid virtual object, and computed force andmoment for palm feedback based on virtual palm andobject configurations. Bergamasco’s point-based systemcomputed interaction forces that included componentsuseful for a hand force-feedback system. Popescu,Burdea, and Bouzit (1999) applied a point-based algo-rithm from other work (Ho, Basdogan, & Srinivasan,1997) to gridpoints of meshes placed on virtual finger-tips, summing mesh forces and applying a force map-ping step that mapped resulting force to an actuator’sline of action. Immersion’s Virtual Hand Toolkit com-putes force for the glove-mounted CyberGrasp device,but details of its rendering method are not available.

3 Desktop Environment

Our grasping approach is part of a prototype sys-tem to support natural whole-hand interactions in adesktop-sized workspace. Figure 1 shows our physicaldisplay. A 21-inch monitor is placed at a 45° angleabove a mirror to effectively place visual feedback in theworkspace under the mirror (a summary of mirror dis-plays and their advantages is found in Mulder & Bosch-

ker, 2004). The monitor refresh rate is 120 Hz to sup-port stereoscopic viewing via CrystalEyes LCD shutterglasses. Our OpenGL-based visual rendering systemincludes projective shadows to provide additional depthcues. Anecdotally, both stereoscopy and shadows appearto improve the user’s ability to locate an object wheninitiating a grasp, as would be expected from work byHu, Gooch, Creem-Regehr, and Thompson (2002) andearlier work summarized therein.

Hand configuration is tracked by a 22-sensor Cyber-Glove and an Ascension MiniBird magnetic tracker that issynchronized with the monitor refresh to stream readingsat 120 Hz. Our hand modeling software uses a 2270-triangle geometric hand model from Viewpoint Data Labsthat we modified to include a deformable palm (other thanthis, it consists of rigid pieces). The deformation supportsthe trapeziometacarpal joint in the joint model. We imple-mented a common joint model: Each of the four fingershas a 2-dof metacarpophalangeal (MCP) joint for abduc-tion and flexion at the first knuckle and a 1-dof interpha-langeal (IP) joint at each of the remaining two knuckles for

Figure 1. Desktop environment.


flexion. The thumb has a 2-dof trapeziometacarpal joint inthe palm, a 1-dof MCP joint for flexion at the firstknuckle, and a 1-dof IP joint for flexion at the secondknuckle. The CyberGlove only provides three abductionsensors for the four fingers, so we compute middle fingerabduction as a linear combination of index and ring fingerabductions.

Accurate hand geometry and good calibration ofglove sensors are important for accurate grasping, par-ticularly for precision grasps. The standard automaticCyberGlove calibration did not provide the desired ac-curacy, although it is valuable when quick calibration isneeded. We implemented a more time-consuming pro-cedure requiring manual calibration of joint gains andoffsets based on visual comparison of our hand model’sjoint angles to those of the real hand. We have noticedthat one remaining factor limiting accuracy is the cross-coupling effect of sensors (recent work by Kahlesz,Zachmann, & Klein, 2004 provides further insight intothis cross-coupling effect). To address the need for ac-curate hand model geometry, we have parameterizedthe hand model geometry to allow manual adjustmentof segment sizes, but an automatic system for capturingaccurate hand geometry is still needed.

For haptic feedback, we have considered a Cyber-Grasp device, a Rutgers Master device, and low-costvibrotactile feedback using our own controller. Figure 1shows the use of the CyberGrasp device. Its exoskeletonsignificantly reduces freedom of motion in our smallworkspace. The Rutgers Master solves this problem, butit has a limited finger motion range that makes certaingrasps difficult (Borst & Volz, 2005), although this maybe improved by visually exaggerating flexions. Sinceboth of these devices are hand-grounded, they supportinternal grasping forces but do not properly simulateexternal forces such as gravity. The use of low-cost vi-brotactile elements placed on the fingertips offers amore comfortable solution—this approach provides noforces but may still provide useful sensations while pre-serving full motion range. Performance data presentedlater in this paper were obtained using the CyberGlovewith no haptic device present. Our grasping system isintended to support interactions both with and withoutthe use of haptic devices.

Our software consists of two main threads, as shown inFigure 2, running on a PC with dual 2.4 GHz Pentium 4processors and an NVIDIA Quadro FX 3000 graphicscard. The graphics thread contains a loop that performsgraphical rendering. The remaining operations are per-formed by the interaction thread, which is assigned a highthread priority. Our custom MiniBird driver obtains aposition-quaternion pair from the palm tracking system.Finger joint angles are obtained using Immersion’s low-level CyberGlove driver, and these are then transformed bythe offset and gain from our calibration data. Our springmodel uses joint angles to compute forces and torques fora dynamic simulation and performs force feedback render-ing as described in Section 4. Force values are reported tothe haptic device driver, if enabled. The dynamic simula-tion is stepped forward by one time interval for each inter-action loop iteration.

4 Grasping Approach

4.1 Overview

Our approach to grasping is to couple a dynamichand simulation to the tracked hand configuration using

Figure 2. Software overview.


a system of virtual linear and torsional spring-dampers.This approach is inspired by the use of virtual couplingsin force feedback rendering and by the use of springmodels to drive articulated structures for animation, assummarized in Section 2.2. We present a spring modelfor an articulated hand that supports whole-hand grasp-ing and manipulation, addresses the problem of visualinterpenetration, and performs force feedback render-ing.

We refer to the two hand configurations as the springhand and the tracked hand. The visual hand model seenby users reflects the simulation-controlled spring handconfiguration. As a result of the simulation, the springhand tends to follow the tracked hand. However, colli-sion detection and response prevent the spring handfrom penetrating grasped objects, in addition to pro-ducing forces and torques for the physically-based re-sponse of grasped objects.

Figure 3 illustrates the tracked hand and the springhand as wireframe and shaded hand models, respec-tively. A subset of the linear and torsional spring-dampers is also shown. We use a total of 21 torsionaland 6 linear spring-dampers. There is one torsional ele-ment for each of the 20 dof provided by finger joints(illustrated only for the index finger), one torsional ele-ment and one linear element for the base of the hand(both illustrated), and one linear element for each of thefive fingertips (not illustrated).

4.2 Torsional Finger Joint Springs

For each degree of freedom of the hand jointmodel described in Section 3, the spring model containsa torsional spring-damper that introduces torque ofmagnitude

� � k��t � �s� � b��s � �t�

wherek, b are the spring and damping constants,�t, �s are tracked joint angle and spring model joint

angle, and�s, �t are angular velocities of these joints.

The direction of the torque vector is the direction ofthe spring model’s joint axis. Note �s and �t are com-puted from differences in successive joint angles, and �s

may alternatively be available directly from the dynamicsimulation. The dynamic simulation generally runs at ahigher rate than the tracker update rate, so �t is a differ-ence divided by the number of simulation iterations per-formed between receipt of two tracker readings. A simi-lar adjustment is made in other tracked angular andlinear velocities described in Section 4.

As a result of these finger joint springs, the springhand’s joint angles tend to resemble those of thetracked hand.

4.3 Torsional and Linear Palm Springs

The base of the spring hand (center of the palm) iscontrolled by both torsional and linear spring-dampers

Figure 3. The tracked hand (left), the spring hand (right), and

some of the virtual spring-dampers used to couple them.


so that it follows the tracked hand. The linear compo-nent computes a force applied at the base of the hand

fpalm � kT�pt � ps� � bT�vs � vt�

wherekT, bT are the translational spring and damping con-

stants,pt, ps are base positions of the tracked and spring

hands, andvs, vt are base velocities of the spring and tracked

models.

The torsional component is more complex and wepresent its restoring torque and damping torque sepa-rately, noting these torque vectors do not generally havethe same direction. Our approach uses a unit quaternionrepresentation for orientations and computes incremen-tal rotations from them in a manner analogous to theuse of differences to compute displacements and veloci-ties from positions. As background, note that for twounit quaternions q1 and q2 representing orientationswith respect to the same fixed frame, (q*1q2) describesthe rotation from q1 to q2 with respect to a q1-rotatedframe (q*1 is the quaternion conjugate of q1). Further-more, (q2q*1) describes the same rotation with respect tothe fixed frame.

Let qt and qs represent current orientations of thetracked palm and spring palm, respectively, with respectto a fixed frame. The system computes the orientationof the tracked palm with respect to the spring palm as:

q � q*sqt

The axis aq and angle �q are extracted from q and arestoring torque is applied to the palm with torque vec-tor direction given by aq and magnitude given by

�restoring � kR��q�

where kR is the rotational spring constant.The resulting torque vector is in the spring hand’s

frame of reference, but conversion to other frames isstraightforward.

To compute the damping term, the system first com-putes angular velocity descriptions for the spring handand tracked hand with respect to a world frame as

q�s � qs1qs0* and q�t � qt1qt0*

whereqs1, qt1 are the current spring and tracked model ori-

entations, andqs0, qt0 are their previous orientations.

Depending on the simulation tool used, better resultsmay be obtained by computing q�s from an angularvelocity description in the current simulation state. Theorientation qt0 refers to a previously received trackerreading, and multiple simulation cycles may have passedbetween readings. Therefore, the velocity q�t is cor-rected by extracting the axis and angle, dividing the an-gle by the number of simulation iterations performedbetween receipt of tracker readings, and returning theresult to quaternion form as q�t,adjusted. A relative ve-locity description is then computed as

q� � q�sq�t,adjusted*

The axis aq� and angle �q� are extracted from q� anda damping torque is applied to the palm with torquevector direction given by aq� (world-referenced) andmagnitude given by

�damping � �bR��q��

where bR is the damping constant.

A caveat must be noted for the torsional springmodel. The calculation of angular velocity finds an in-cremental rotation that rotates from one sampled orien-tation to the next. There are multiple such rotations—this can be seen by adding 2� to a rotation angle or byreplacing a rotation of angle � with a rotation of angle�(2� � �). We choose the smaller magnitude rotation,since per-cycle differences in hand orientation are lim-ited in practice by human motion limits and low-passnoise filtering in the tracking system. With the unitquaternion representation, this is done by modifying thequaternion multiplication operation to negate all termsof the result quaternion if its scalar component is nega-tive (indicating a rotation angle with magnitude above�). Note that only a limited angular velocity magnitudeis therefore represented, and this limitation is notunique to the quaternion representation (for tracked


hand velocity, it is a sampling problem). Quaternionmultiplication is similarly modified for the computationof relative orientation q for restoring torque, but not forthe calculation of relative velocity q�.

4.4 Linear Fingertip Springs

Five additional spring-damper elements wereadded for further control and to improve realism forcertain grasp shapes. Consider, for example, a grasp inwhich the palmar sides of all finger phalanges of thetracked hand intersect with an object. This intersectioncan result in spring hand MCP flexion joints being ex-tended to a point where the fingertips are lifted awayfrom the object surface. The linear fingertip springs de-scribed here pull the spring hand’s fingertips toward thetracked hand’s fingertips to create a more realistic graspshape in which the fingers appear wrapped around theobject.

For each of the fingertips and the thumbtip, there is alinear spring-damper that computes a force applied atthe tip:

ftip � ktip�pt,tip � ps,tip� � btip�vs,tip � vt,tip�

wherektip, btip are the spring and damping constants,pt,tip, ps,tip are a point on the tracked hand’s fingertip

and a corresponding point on the spring hand’s finger-tip, both computed by a forward kinematics routine,and

vs,tip, vt,tip are the velocities of these points.

4.5 Force Rendering for ForceFeedback

An established force rendering technique forsingle-point devices such as the PHANToM is to reflectforces from a virtual spring-damper coupling. We ex-tend this concept to the articulated hand to renderforces for force-feedback gloves. However, due to lim-ited degrees of freedom of these devices, the forces andtorques of our spring model must be reduced for map-ping to the glove actuators. Currently we assume one

degree of freedom per fingertip and compute a scalarforce magnitude at each fingertip (thumbtip) as:

ffeedback � � �a1f�1 � a2f�2 � a3f�3 � a4fls� � n

wherea1, a2, a3, and a4 are user-chosen coefficients,f�1, f�2, and f�3 are three forces related to flexion

torques as described below,fls is a modified ftip as described below, andn is a unit vector describing the actuator’s direction

of force.

The equation, further illustrated by Figure 4, blendsfour force components and maps them onto the actua-tor’s line of action. The simplest component is thefourth component, which projects the linear fingertipspring-damper force onto the actuator’s line of action.The force fls is similar to the force ftip described in Sec-tion 4.4, except that we remove the influence of palmbase by not considering palm base configuration duringforward kinematics for computing fls. The force ftip canbe used instead, depending on user preference.

The first three terms provide further control over sen-sations by mapping flexion torques to force-feedback

Figure 4. Components of force-feedback rendering.


vectors. In our earlier work (Borst & Indugula, 2005),the first three terms were simply proportional to thethree flexion joint torques in the spring model’s finger.We have extended this to further consider the grasp ge-ometry, and illustrate this for the first flexion torque,used to compute f�1 (the others are similar). Let �1 bethe torque magnitude computed for the first flexionjoint as described in Section 4.2, and d be a vector fromthe center of the spring model’s joint to the fingertippoint ps,tip. The vector f�1 is found with direction givenby the cross product of the joint axis and d, and withmagnitude �1/�d�.

4.6 Implementation Notes

In this paper, we do not deal with the details ofcollision detection, collision response, or numerical inte-gration. This is because we obtained satisfactory resultsby using existing simulation tools to perform thesesteps. Basic simulation techniques, including the han-dling of articulated structures, are summarized byCoutinho (2001). We expected to obtain reasonableresults with existing tools provided they include supportfor articulated structures, robust collision detection, agood friction/contact model, and implicit integrationfor stability. We have implemented our spring modelwith two freely available simulation tools: the Open Dy-namics Engine (www.ode.org) and the Novodex PhysicsSDK V2.1.1 (www.novodex.com). The main advantageof ODE is that its source code is available, so its meth-ods can be understood and modified. Novodex cur-rently provides better support for polygon meshes, butminimal information is available about the approaches ituses. In the remainder of the paper, we discuss theNovodex-based implementation, which we found toproduce more stable results.

4.6.1 Novodex-Based Implementation. Novo-dex supports various object representations, which in-clude simple primitives, mesh, convex mesh, and meshwith associated pmap. The mesh and mesh � pmap rep-resentations were chosen for grasped objects based onour desire to support existing polygon mesh objects(the pmap is a voxelized representation used to supple-

ment a mesh-mesh collision detection procedure—im-plementation details are not available). Line-sweptspheres were chosen to represent finger segments afterwe observed that also representing the finger segmentsas mesh or mesh � pmap produced disturbing simula-tion errors (specifically, joint constraints not beingmaintained during grasping). The line-swept spheresrepresent our visual hand model’s segments closely butnot exactly.

Each of the 20 dof provided by finger joints was im-plemented using a Novodex hinge joint. Joints with twodof were implemented as two hinge joints connected bya point mass. The Novodex SDK provides its own hingejoint springs, but they do not directly match our tor-sional spring model for finger joints, as their dampingbehavior is not based on the relative angular velocitiesbetween two models. To combine our desired behaviorwith the Novodex springs, we used the following ap-proach: we computed torque as in Section 4.2 and setthe Novodex spring’s displacement angle to be propor-tional to this torque magnitude, while setting a fixedhigh Novodex spring constant and a fixed zero Novo-dex damping constant (these constants are separatefrom the constants we used to compute torque). Thisresults in the Novodex spring computing a restoringtorque that is proportional to the torque computed byour model, which includes both restoring and dampingterms.

To apply the fingertip spring forces and palm springforce and torque, we used Novodex functions to di-rectly apply the forces and torques computed by ourspring equations.

4.6.2 Parameter Value Selection and RelatedIssues. We tune spring model parameters based oncommon “rules of thumb” for simulation and force ren-dering systems and based on observation of springmodel behavior. For fast response of the spring modelduring free motion, the spring constant should be highand the hand model mass should be low. The masses weassign to hand model segments are roughly proportionalto segment size, with a significant mass added to thepalm since there is no supporting arm model and thecombined finger joint torques otherwise produce unre-


alistic movement or oscillation of the palm. Dampingconstants are set to low values. Spring constants can alsobe tuned to affect hand configuration during grasping.For example, suppose users find errors in MCP joints tobe more tolerable than errors in IP joints; MCP springscan then be made relatively weak.

Parameter values also affect haptic feedback quality.Established heuristics again suggest spring constantsshould be set as high as possible without producing per-ceived instabilities, and for a small amount of dampingto be added. Even during free motion, the coupling canresult in spring hand mass properties being reflected tothe user as force feedback. We adopt the strategy of set-ting feedback force to zero for any finger that does notcollide with an object. Furthermore, tracked hand con-figuration can be used to directly set spring hand con-figuration until contact is established, at which time thespring model can be enabled until no more contact oc-curs and spring hand configuration returns to closelymatch tracked hand configuration.

In the Novodex implementation, much tuning of simu-lation parameters was required to obtain desirable behav-ior, and this included parameters beyond the spring modelparameters. We set friction constants high to provide goodgrasp stability but not so high that excessive sticking re-sulted. We set a restitution coefficient for the hand seg-ments to zero to prevent any elastic bounce of objects onhand surfaces. Other parameters, such as time step, dependheavily on the choice of spring model parameters and de-sired environment characteristics.

5 Results

In this section, we illustrate the behavior of thegrasping system pictorially, discuss observed user behav-iors, and discuss the performance obtained with theNovodex simulation tool.

5.1 Examples of Grasping andManipulation

We used our system to perform interactionswith a variety of rigid virtual objects. Figure 5 shows

the behavior of the tracked and spring hand modelsduring contact with a static object. The tracked hand,shown as a wireframe mesh, intersects the object. Thespring hand, shown as a solid model, remains outsidethe object in a more realistic grasp configuration.Vectors pointing away from the fingernails show theresult of force rendering. Their magnitudes are de-scribed in Section 4.5, and their directions corre-spond to the lines of action typical of force-feedbackglove actuators.

Figure 6 illustrates manipulation of various objects asseen in a video presented at the IEEE VR 2005 confer-ence (see http://www.cacs.louisiana.edu/�cborst/grasp/). The video shows the potential for visually real-istic hand model behavior and object dynamics duringgrasping and manipulation. Readers familiar with work byHirota and Hirose (2003) will note similarities in boththe choice of objects and in the interactions performed—these are intentional, as their work inspired us to use a

Figure 5. Grasps of a static rocker arm showing tracked hand

(mesh), spring hand (solid), and force feedback vectors.


dynamics-based approach and we wanted to compareresults.

5.2 User Behavior and DifficultiesEncountered

We have not conducted a formal user study, butwe have observed the behavior of users during demon-strations in our lab and discuss results based on theseinformal observations. The users were not previouslyexperienced with virtual grasping. Most demonstrationswere given without the use of a haptic feedback device,due primarily to user time constraints. We intended ourapproach to be applicable with or without such devices.

All users of our system have successfully performedbasic grasping and lifting of objects, but there is a widevariation between users in the success of their initialgrasping attempt. Users encountering difficulties appearto improve quickly with subsequent attempts.

Users who encountered difficulty during their initialgrasping attempts did so primarily for one of two reasons.First, some users attempted grasps using unrealistic behav-iors such as pressing the palm into a virtual object ratherthan real-world behaviors such as grasping with the finger-tips. Pressing the palm into an object does not result in asuccessful grasp in our virtual world (or in the real world).Users no longer exhibit this behavior once they have suc-ceeded in grasping with more appropriate techniques. Asecond and more common problematic behavior is thatsome users tended to close their fingertips completely dur-ing grasps, regardless of object size. This complicates therelease of grasps and the manipulation of grasped objects.When the real fingers are deep within the virtual objectand no longer closely match the visual feedback, an objectcan appear to stick to the hand since the user must nowuse exaggerated finger motions to release or manipulatethe object. A similar problem occurs in other grasping ap-proaches.

Figure 6. Grasping and manipulation of various objects.


Users who use a “light touch” to grasp objects arethe most successful at interacting, and some users ex-hibit this behavior even in their initial grasping attempt.Users who encounter the difficulties described abovetypically improve quickly once they are advised to graspobjects with the fingertips and use a light touch. Thisallows users to successfully perform manipulations asshown in Figure 6. The ball, rocker arm, and bunnyobjects appeared to be the simplest to manipulate. Userswere able to grasp these objects naturally with variousgrasp types, cradle the ball object or roll it in the hand,and toss and catch the ball.

Some remaining difficulties were observed duringlifting by small handles such as a horse hoof or the tea-cup handle and in interactions with the pencil. Precisionmanipulations require precise tracking and hand geome-try. Due to imperfections in joint angle sensing andhand model geometry, it can be difficult to achieve athumb, index finger, and middle finger configurationthat allows all three digits to contact the pencil as theywould in real-world pencil use.

5.3 Performance using Novodex

The Novodex-based implementation of our springmodel successfully supports a wide range of graspinginteractions. The simulation is stable for grasping ofbasic objects with the desired light touch. For stress

testing, we intentionally moved fingers deep into com-plex objects and tried to push the hand through objectsresting on the floor. This type of behavior sometimesresulted in fingers popping into the model or movingerratically, particularly for the teacup model when fin-gers were pushed sideways into the rim edge fromabove. This suggests a possible limitation of Novodexfor simulating thin objects such as the teacup sides. Weinitially suspected this was due to the use of a voxelizedpmap representation, but found the same result using amesh without the pmap representation. These problemsare avoided by a proper grasping technique, and thestability of simulation engines is continually improving.

We measured simulation speed for grasps of variousobjects, as summarized in Table 1, using the PC config-uration described in Section 3. The sphere and boxprimitives do not correspond to any previously shownobjects, but were included to consider results obtainablewith simple primitives. We recorded a timestamp onceevery tenth simulation step using a one-msec timer, sothe reported minimum and maximum values are ten-cycle averages. Each table entry corresponds to an inter-action sequence lasting about 60 sec. About one-thirdof this time was spent grasping the object with only thethumb tip and index finger tip, and another third wasspent grasping the object with the whole hand to maxi-mize collisions. We also plotted measured cycle timesfor all interactions as illustrated in the examples of Fig-

Table 1. Object Complexity and Resulting Performance

Object Verts Tris

Cycle Time (msec)

Min Ave Max

Box primitive — — 0.2 0.60 1.1Sphere primitive — — 0.2 0.57 1.4Geosphere (ball) 642 1280 0.2 0.72 2.2Teacup 3749 7494 0.3 0.80 2.5Bunny 7788 15201 0.2 0.85 3.2Rocker arm 10045 20088 0.3 1.10 4.4Hi-res bunny 35947 69451 0.4 1.48 7.6Hi-res rocker 40177 80354 0.3 1.69 14.6


ure 7. The time range of about 5 to 25 sec correspondsto the two-finger grasp. The time range of about 40 to60 sec corresponds to the whole-hand grasp.

The simulation rates obtained for all but the highest-complexity meshes are reasonable for glove-based feed-back. An update rate of 1000 Hz is often cited as a re-quirement for haptic rendering. However, the force-feedback gloves in our lab have low mechanicalbandwidths at the fingertips that relax this requirement.According to Bouzit, Burdea, Popescu, and Boian(2002), mechanical bandwidth at the fingertip is 40 Hzfor the CyberGrasp and 10 Hz for the Rutgers Master,and the Master performs valve control at 300 Hz. TheCyberGrasp is used with a CyberGlove that reportshand configuration at about 150 Hz. We therefore ex-pect force rendering rates of a few hundred updates persecond to be sufficient for these devices and concludethat these were maintained for triangle meshes of up toseveral thousand vertices.

For the purpose of evaluating performance, no mecha-nism was used to control simulation rate. In practice, astable simulation rate is needed to avoid perceivablechanges in simulation speed and to provide consistent con-trol behavior for force feedback. This can be accomplishedby synchronization with an internal timer or an externaldevice such as the tracker, as done by Hirota and Hirose(2003). Our tracker driver supports this synchronization.

6 Conclusion and Future Work

In this paper, we presented a spring model for vir-tual grasping that couples tracked hand configuration to

a virtual hand model controlled by physical simulation.This approach supports whole-hand grasping and ma-nipulation of virtual objects while preventing visuallydistracting hand/object interpenetrations. Such a grasp-ing approach is general in that, unlike most recent ap-proaches, it does not rely on heuristics to estimate userintent or grasp state. We also presented a new force ren-dering equation that uses the forces and torques of thespring model to render forces (or intensity levels) forhaptic feedback gloves. We have used our graspingmodel in a prototype VR system for natural interactionswith virtual objects in a desktop-sized workspace.

We evaluated the performance of the grasping ap-proach when implemented using a widely available sim-ulation tool for collision detection and response. Thesimulation rate was high enough to support control offorce-feedback gloves with grasped objects consisting ofup to several thousand triangles. We graphically illus-trated the quality of the resulting grasps and identifiedsome cases for which difficulties can still be expected.

Since force feedback can improve grasp control (Fabi-ani & Burdea, 1996) and impose at least some motionconstraints, we expect force feedback can aid users inachieving the light touch discussed in Section 5.2. Amore formal evaluation is needed to determine if thisbenefit of force-feedback gloves outweighs the cost ofadded motion limits and discomfort. Alternatives toforce feedback may also be explored in future work.One approach is the use of visual indicators for userswho tend to close the fingers too far. For example, in-stead of constraining the visual model to remain com-pletely outside an object, it could be allowed to sink inslightly when interpenetration exceeds some threshold,or components of tracked configuration could be shownas in Figure 5. Another alternative is to manipulate thetracked hand’s joint angles when interpenetration ex-ceeds some threshold, adding an offset that is main-tained for some time to bring the tracked configurationcloser to the spring configuration so that smaller fingermovements are sufficient to release a grasped object.Unlike the ghost hand technique in DesRosiers et al.(2001), this modified tracked hand should not avoidinterpenetration completely.

Grasping interactions with the pencil model and small

Figure 7. Cycle times during bunny interaction (left) and rocker

arm interaction (right) corresponding to Table 1 entries.


object features can benefit from improved joint sensoraccuracy and from a technique for determining accuratehand model geometry for arbitrary users. Another ap-proach to deal with such cases and possibly improvegrasping quality overall is to reconsider the use of heu-ristics, but to incorporate heuristics in a manner thatpreserves the benefits of the physically-based approach.We propose a hybrid system in which heuristics are usednot to influence the virtual hand model directly or tomake discrete control decisions, but to influence thespring model and simulation parameters in a moretransparent manner to tune them to the particular inter-action being performed.

Formal evaluation of grasp quality and of the hapticfeedback approaches mentioned in Section 3 is desir-able. Finally, more precise guidelines need to be devel-oped for tuning simulation and spring model parame-ters to produce the best possible grasping behavior andforce feedback. Ideally, these would include theoreticaltechniques such as those proposed elsewhere for tuningspring-damper parameters in other force-feedback ap-proaches, for example, as by Adams and Hannaford(1999).

Acknowledgments

The mirror stand in our desktop environment is based on adesign in Colin Ware’s lab at the University of New Hamp-shire. Our MiniBird driver was based on a Flock of Birdsdriver by Greg Schmidt. We obtained geometric models on-line from the Stanford University Computer Graphics Labora-tory (bunny), Cyberware (rocker arm), the Georgia Instituteof Technology (horse originally from Cyberware), andwww.3dcafe.com (uncredited pencil and teacup).

References

Adams, R. J., & Hannaford, B. (1999). Stable haptic interac-tion with virtual environments. IEEE Transaction on Robot-ics and Automation, 15(3), 465–474.

Basdogan, C., Ho, C., & Srinivasan, M. A. (1997). A ray-based haptic rendering technique for displaying shape and

texture of 3D rigid objects in virtual environments. WinterAnnual Meeting of ASME. Dynamic Systems and ControlDivision, 61, 77–84.

Bergamasco, M., Degl’Innocenti, P., & Bucciarelli, D. (1994).A realistic approach for grasping and moving virtual objects.IEEE/RSJ International Conference on Intelligent Robotsand Systems, 717–724.

Borst, C. W., & Indugula, A. P. (2005). Realistic virtualgrasping. IEEE Virtual Reality Conference, 91–98, 320.

Borst, C. W., & Volz, R. A. (2003). Observations on andmodifications to the Rutgers Master to support a mixture ofpassive haptics and active force feedback. Eleventh Sympo-sium on Haptic Interfaces for Virtual Environment and Tele-operator Systems, 430–437.

Borst, C. W., & Volz, R. A. (2005). Evaluation of a hapticmixed reality system for interactions with a virtual controlpanel. Presence: Teleoperators and Virtual Environments,14(6), 677–696.

Boulic, R., Rezzonico, S., & Thalmann, D. (1996). Multi-finger manipulation of virtual objects. ACM Symposium onVirtual Reality Software and Technology, 67–74.

Bouzit, M., Burdea, G., Popescu, G., & Boian, R. (2002).The Rutgers Master II—New design force-feedback glove.IEEE/ASME Transactions on Mechatronics, 7(2), 256–263.

Bowman, D. A., Kruijff, E., LaViola, J. J., & Poupyrev, I.(2001). An introduction to 3-D user interface design. Pres-ence: Teleoperators and Virtual Environments, 10 (1), 96–108.

Burns, E., Razzaque, S., Panter, A. T., Whitton, M. C., Mc-Callus, M. R., & Brooks, F. P. (2005). The hand is slowerthan the eye: A quantitative exploration of visual dominanceover proprioception. IEEE Virtual Reality Conference,3–10.

Coutinho, M. G. (2001). Dynamic simulations of multibodysystems. New York: Springer-Verlag.

DesRosiers, H., Gomez, D., Tremblay, M., & Ullrich, C.(2001). VirtualHand v.25 programmer’s guide. Palo Alto,CA: Virtual Technologies, Inc.

Fabiani, L., & Burdea, G. (1996). Human performance usingthe Rutgers Master II force feedback interface. IEEE Vir-tual Reality Annual International Symposium, 54–56.

Froehlich, B., Tramberend, H., Beers, A., Agrawala, M., &Baraff, D. (2000). Physically-based manipulation on theresponsive workbench. IEEE Virtual Reality Conference,5–11.

Hirota, K., & Hirose, M. (2003). Dexterous object manipula-


tion based on collision response. IEEE Virtual Reality Con-ference, 232–239.

Ho, C., Basdogan, C., & Srinivasan, M. (1997). Haptic ren-dering: Point- and ray-based interactions. Second PHAN-ToM Users Group Workshop.

Hu, H. H., Gooch, A. A., Creem-Regehr, S. H., & Thomp-son, W. B. (2002). Visual cues for perceiving distances fromobjects to surfaces. Presence: Teleoperators and Virtual Envi-ronments, 11(6), 652–664.

Iwata, H. (1990). Artificial reality with force-feedback: Devel-opment of desktop virtual space with compact master ma-nipulator. Computer Graphics, 24(4), 165–170.

Johansson, R. S. (1998). Sensory input and control of grip.Novartis Foundation 218: Sensory Guidance of Movement,45–59.

Kahlesz, F., Zachmann, G., & Klein, R. (2004). Visual-fidelitydataglove calibration. Computer Graphics International,403–410.

Kitamura, Y., Higashi, T., Masaki, T., & Kishino, F. (1999).Virtual chopsticks: Object manipulation using multipleexact interactions. IEEE Virtual Reality Conference, 198–204.

Kokkevis, E., Metaxas, D., & Badler, N. I. (1996). User-controlled physics-based animation for articulated figures.Computer Animation, 16–26.

Koutek, M., & Post, F. H. (2001). Spring-based manipulationtools for virtual environments. Joint IPT/EGVE Conference,61–70.

McNeely, W. A., Puterbaugh, K. D., & Troy, J. J. (1999). Sixdegree-of-freedom haptic rendering using voxel sampling.

International Conference on Computer Graphics and Inter-active Techniques (SIGGRAPH), 401–408.

Meseure, P., Lenoir, J., Fonteneau, S., & Chaillou, C. (2004).Generalized god-objects: A paradigm for interacting withphysically-based virtual worlds. Computer Animation andSocial Agents, 215–222.

Mulder, J. D., & Boschker, B. R. (2004). A modular systemfor collaborative desktop VR/AR with a shared workspace.IEEE Virtual Reality Conference, 75–82.

Popescu, V., Burdea, G., & Bouzit, M. (1999). Virtual realitysimulation modeling for a haptic glove. Computer Anima-tion, 195–200.

Ruspini, D. C., Kolarov, K., & Khatib, O. (1997). Haptic in-teraction in virtual environments. IEEE International Con-ference on Intelligent Robots and Systems, 128–133.

Ullmann, T., & Sauer, J. (2000). Intuitive virtual grasping fornon haptic environments. Pacific Graphics, 373–380.

Welch, R. B., & Warren, D. H. (1980). Immediate perceptualresponse to intersensory discrepancy. Psychological Bulletin,88(3), 638–667.

Westenhofer, W., & Hahn, J. (1996). Using kinematic clonesto control the dynamic simulation of articulated figures.Computer Graphics International, 26–37.

Zachmann, G., & Rettig, A. (2001). Natural and robust inter-action in virtual assembly simulation. Eighth ISPE Interna-tional Conference on Concurrent Engineering: Research andApplications.

Zilles, C. B., & Salisbury, J. K. (1995). A constraint-basedgod-object method for haptic display. IEEE InternationalConference on Intelligent Robots and Systems, 146–151.


a spring model for whole-hand virtual grasping

Documents