an assistive navigation paradigm for semi-autonomous wheelchairs using force feedback and goal...

An Assistive Navigation Paradigm for Semi-Autonomous

Wheelchairs Using Force Feedback and Goal Prediction

Master’s Thesis DefenseCandidate: John Staton

Advisor: Dr. Manfred HuberCommittee Members: Dr. David

Levine, Dr. Gergely Zaruba

John Staton 2008Computer Science & Engineering

Outline

• Introduction• Related Work• Concept Review• Design Methodology• Implementation• Experiments• Concluding Thoughts


Introduction - Motivation

• “49.7 million: Number of people age 5 and over with a disability, according to Census 2000; this is a ratio of nearly 1-in-5 U.S. residents, or 19 percent.”

• 25 million had difficulty walking a quarter mile or climbing a flight of 10 stairs, or used an ambulatory aid, such as a wheelchair (2.2 million) or a cane, crutches or a walker (6.4 million).

• “The rate of power wheelchair prescriptions increased 33 fold from 1994 to 2001”


Introduction – Early WorkSystem Name Sensors

CPWNS Vision, Dead ReckoningThe Intelligent

Wheelchair Vision, Infrared, Sonar

Intelligent Wheelchair System Vision, Sonar, Gesture Recognition

INRO GPS, Sonar, Drop-Off Detector

MAid Sonar, Infrared, Laser Range Finder, Dead Reckoning

OMNI Sonar, Infrared, Bump, Dead Reckoning

RobChair Sonar, Infrared, Bump

Rolland Vision, Sonar, Dead Reckoning, Infrared, Bump Sensors

SENARIO Dead Reckoning, Sonar

SIRIUS Sonar, Dead Reckoning

Smart Wheelchair Line Trackers, Bump Sensors

Smart Wheelchair Ultrasonic Beacons

TetraNauta Vision, Infrared, Sonar, Bump Sensors

VAHM Sonar, Infrared, Dead Reckoning

Wheelesely Vision, Infrared, Sonar


• Intelligent Wheelchair• Started in the 1990’s• Numerous universities

and labs• However

• Only two companies sell smart wheelchairs for research use

• Only one is commercially available, only in Europe

Introduction – Inspiration


“The majority of research and development activity in the field of control and automation applied to powered mobility for people with disabilities has

concerned sophisticated technology and techniques. … A more effective approach is to make use of the most flexible and adaptable

intelligence on the chair – the user. To accomplish this, researchers must design, build

and test their systems with real users and contexts in mind.” - Paul D. Nisbet

University of Edinburgh

Introduction – Thesis


• Something is disconnected– Smart wheelchair projects treat the wheelchair as

an “autonomous unit”– Adults prefer individual independence

• Solution– Semi-Autonomous, assistive wheelchair– Communicate with user– User still provides drive direction

Introduction – Thesis


• What communication technique to use?– Aural and Visual = distracting, both to user and to nearby

observers– Haptics!

• Subtle• Effective• Intuitive• Non-distracting• Already shown to be useful for various tasks (mobility aids,

steering tasks, bio-manipulation in virtual reality, surgical tasks…)

Related Work

• “Luoson III” – Lio, Hu, Chen, Lin• National Chung Cheng University, Taiwan• Specifics:

– Blind user– Ultrasonic sensors– Motion Prediction– MS FF Pro


Related Work

• Wheelchair University – Protho, LoPresti, Brienza

• University of Pittsburgh• Specifics

– Two design philosophies– Passive Assistance– Active Assistance– VR System


Related Work

• Metz University, France – Fattouh, Sahnoun, Bourhis

• Reminiscent of Luoson III– Distance sensors– Averaged feedback forces– VR System– MS FF 2


How Is This Thesis Different?

• Previous research– Obstacle avoidance

• This research project– Obstacle avoidance

&– Goal guidance

• Seeks to intuit the user’s intended goal• Guide the user to that goal & away from obstacles

– Active assistance


Concept Review – Force Feedback

• “Haptics” – anything related to or based on the sense of touch

• Force Feedback – Haptics applied to an I/O device

• Touch Sensations– Vibration– Robust effects

• Emulate the feeling of weight, friction, liquid, and more.




• Simple• Vertical grip that pivots around

a fixed end• Angle of the joystick

Or• Displacement from neutral

position• Intuitive• Effective

• Used in many applications• Flight control• Video games• Electric Powered

Wheelchairs• Therapy



• “Effect” – The encapsulated force-response data sent to the FF device

• Categorized bythree distinctdimensions:

Static Dynamic

One-shot Open-ended

Interactive Time-based

Concept Review – Harmonic Functions

• Formal definition:• “Harmonic Function”

– Real function– Range in the real numbers– With continuous second partial derivatives– Satisfy Laplace’s Equation

• The sum of the second partial derivatives equal zero• No local maxima or minima• Smooth and differentiable



• Used repeatedly, and with great success• Path planning in a known

environment• Potential value =

probability of collision• 1.0 – obstacle, 0.0 – goal• Gradient = direction away

from obstacles and toward goals


Concept Review – Applying Harmonic Functions


Iterate through the entire grid, where for every grid[i,j]:

Re = 0.25 x (nb1+nb2+nb3+nb4) – grid[i,j];

grid[i,j] = grid[i,j] + Re;

nb1 = grid[i-1,j] x w;

nb2 = grid[i+1,j] x w;

nb3 = grid[i,j-1] x w;

nb4 = grid[i,j+1] x w;

The maximum Re is saved for each iteration through the grid.

As long as Re > 10-14 then the process repeats.

Successive Over-Relaxation (SOR)

Iterative, numerical method

Speed up convergence of the Gauss-Seidel method for solving linear systems of equations

Design Methodology

• Objectives– The ability to infer the user’s intention– The ability to help direct the user towards the intended

goal and away from obstacles• Design Methodology

– Two looping procedures– Outer loop

• Infers goal– Inner loop

• Directs user to goal


Design Methodology – Outer Loop


External User

Preference System

Goal Selection

Harmonic Function

Path Planning

Run-Time

System

Goals

Predicted Goal Grid

Location,Orientation,Past Behavior

Outer Loop

Design Methodology – Goal Selection


• Inputs– Series of goals

• Each goal is initially weighted• based on the knowledge of the external system of user preferences

– Recent User Behavior• Current Location• Current Orientation• Series of past user actions

• Output– Predicted user goal– Predicted likelihood for every goal

• Heuristic– “Confidence”– Modified (increased or decreased) based on the

similarity of the user actions to the actions that would lead to the goal(s).

External User

Preference System

Goal Selection

Goals

Predicted Goal


Outer Loop

Design Methodology – Harmonic Function


• Inputs– Predicted Goal– Environmental Data Grid

• Output– Harmonic function as applied to grid

• Potential value for every location• Goal = 0.0• Obstacle = 1.0• All other locations 0.0 < grid[x, y] < 1.0

• Algorithm– Successive Over-Relaxation (SOR)

Harmonic Function

Path Planning

Run-Time

System

Predicted Goal Grid

Outer Loop

Design Methodology – Inner Loop


Run-Time Loop

ForceEffect

Generation

Force Effect

Playback (Joystick)

Wheelchair Location, Orientation

Motors

Risk, Direction

Force Vector Motor Command

Movement

External User

Preference System

Goal Selection

Harmonic Function

Path Planning

Run-Time

System

Goals

Predicted Goal Grid


Outer Loop

Design Methodology – Force Vector Creation


• Inputs– Wheelchair Location– Wheelchair Orientation

• Output– Force Vector

• Heuristic– Force Vector =

direction, amount of force– Direction = away from obstacles,

toward goal– Amount of force = contingent upon the

“riskiness” of user action

Run-Time Loop

ForceEffect

Generation

Wheelchair Location,

Orientation

Risk, Direction

Force Vector

Movement

Design Methodology – Force Direction


• Force Direction– Direction of the harmonic function gradient (slope)

relative to the wheelchair’s current orientation– Compute the angular difference between the

gradient direction and the orientation of the wheelchair for use as the direction of theforce vector

Wheelchair Orientation

Gradient

Design Methodology – Risk


• Amount of force– “Risk” of current action

• Local Risk vs. Future Risk– Local:

• Wheelchair Velocity• Current Potential Value (from Harmonic Function)• Next Potential Value (from Harmonic Function)• Difference between Current and Next

– Future:• Current Potential Value (from Harmonic Function)• Potential Value some distance ahead, calculated based on current velocity• Difference between Current and Future

• Allows for locally “risky” behavior if future risk is minimized

Design Methodology – Risk


• More Formal– (V + Pc + (Pc-Pn)) = Local Risk

• Velocity = V, Pc = Current Potential, Pn = Next Potential

– (Pc - Pf) = Future Risk• Pc = Current Potential, Pf = Future Potential

– grid[i+x, j+y] = Future potential• X & Y are scaled based on V (and are dependant on orientation)

• How was this actually implemented?– “Levels” of risk

• For every risk factor that was present, a “level” of risk was added• Discrete force effects

Design Methodology – Force Feedback Playback


Run-Time Loop

Force Effect

Playback (Joystick)

Motors

Force Vector Motor Command

Movement

• Convert Force Vector to Force-Feedback effect– Thus communicating to the user:

• Severity of the situation/current action– The amount of force

• What action should be performed next– The direction of the force

• Send joystick position to motors as a motor command– Produces movement– Updates wheelchair’s position,

orientation & velocity– Repeat loop

Implementation

• Dell Dimension 8250– Pentium 4 2.66 Ghz– 512 MB RAM– Windows XP– Microsoft Sidewinder FF 2

• Microsoft Visual Studio ‘05– C# – Modifications to Microsoft Robotics Studio


Implementation – Microsoft Robotics Studio


Joystick Path PlanningConsole

Sensor Data

Implementation – Microsoft Robotics Studio


Experiments

• Two loops– Goal Selection/Harmonic Function Path Planning– Force Feedback/Simulation Environment

• Two major sets of experiments– Goal Selection– User Testing of Simulation Environment

• Quantifiable data (time, number of collisions)• Survey data


Experiments – Goal Selection• Experiments tested:

– Similar or clustered goals vs. semi-similar goals vs. one distinct goal

– User actions towards a goal, neutral actions, actions away from goal

– High, medium and low initial goal weight

– System predicts one goal and produces a prediction of it’s likelihood (100%, 50%, etc)


12 3

45

Experiments – Goal Selection Results/Analysis


• When goal is distinct• 100% accuracy (average predicted likelihood: 100%)

• When goals are semi-similar• 100% accuracy (average predicted likelihood: 80%)

• When goals are similar/clustered• Each of the three clustered goals averaged a predicted

likelihood of ~ 33%• The goal of the three with the highest weight was selected

by the system for each run• Predicts the goal properly when it is reasonable to expect so!

Experiments – User Testing• Each test subject was given time to

familiarize themselves with the simulation, both with force-feedback and without

• Six test runs were given, three with FF, three without– Time to complete course– Number of Collisions

• Post Test Survey– Helpful for avoiding obstacles, helpful

for approaching goal– Too forceful <-> Not forceful


Experiments – User Testing Results/Analysis


Without FF Collisions With FF Collisions

Subject 1 59.420 s 0 52.878 s 0

Subject 2 51.927 s 0 45.690 s 0

Subject 3 61.590 s 0.333 55.229 s 0

Subject 4 51.551 s 0 46.133 s 0

Subject 5 87.080 s 0.667 83.116 s 0.333

Subject 6 413.04 s 1 247.25 s 0.333

(average)

• Six Subjects– Three pairs (one male, one

female)– Ages:

• 20’s• 40-50’s• 70’s

• Subject 6 has hand tremors• Three runs without force-

feedback, three with, alternating• All subjects showed:

– Improved time to complete course with Force-Feedback

– Fewer collisions with Force-Feedback

Experiments – Survey Results


• First Question: “Were the force-feedback suggestions helpful in avoiding obstacles?”– All subjects answered “Yes”– Subject 6 (elderly gentleman with hand tremors) indicated that it didn’t necessarily help “at first”,

but he caught on to the idea of the system, and once that happened, it helped “greatly”.• Second Question: “Were the force-feedback suggestions helpful in approaching the goal?”

– 5 of the 6 subjects answered “Yes”– Subject 5, an elderly woman, answered “Somewhat”– When asked why, she indicated that, to her, what held the system back from a full “Yes” was the

strength of the force-feedback effects.• Final Question: “Between too forceful and not forceful enough, where would you rank the

force-feedback effects?”– Answers varied– Subjects 2, 4 and 6, all males, indicated that the force-feedback suggestions could have been more

forceful.– Subjects 1, 3, and 5, all females, indicated that the suggestions were too forceful, Subject 5 wrote

that the joystick was driving her, and not the other way around

Experiments – Survey Analysis


• What can we gather from this survey?– Force-feedback effect “forcefulness”

• Results indicate possible gender bias in response to effect force• More testing needed to expand on this pattern• Potential solutions:

– User-controlled amount of force– Training period

– Not “getting it” at first• Subject 6 needed a longer period of adjustment before truly understanding the

system• Potential solution:

– First –Time Walkthrough/Explanation/Example Training System

• Positive results!– All subjects found it helpful, and were genuinely excited at the potential of

the system

Concluding Thoughts

• Positive early results!– Faster times, fewer collisions– Positive survey answers and excited test subjects

• Future work– Training system, help user “get” the concept, and

determine user strength to adjust effect force– Real world issues: How to get environment data

and user position (GPS?), other issues that come with applying to a real chair


an assistive navigation paradigm for semi-autonomous wheelchairs using force feedback and goal...

Documents