Body Synergy Approach of Exoskeleton RobotControl for Hemiplegia

著者 モダル ハサン内容記述 この博士論文は内容の要約のみの公開（または一部

非公開）になっていますyear 2016学位授与大学 筑波大学 (University of Tsukuba)学位授与年度 2015報告番号 12102甲第7718号URL http://hdl.handle.net/2241/00151524

1

Body Synergy Approach of Exoskeleton RobotControl for Hemiplegia

Graduate School of Systems and Information Engineering, Department of Intelligent Interaction Technologies

Modar HASSAN (201330222)

I. INTRODUCTION

A. Background

Locomotion deficits are fatal to human well being, selfdependence, work and social activities. According to the WorldHealth Organization (WHO) there are annually 5 millionpeople who suffer from permanent disability after stroke,being the biggest cause of major disability in the UnitedKingdom, for example. Technology of wearable exoskeletonrobots offer great potential to assist the locomotion of peoplewith deficiency in gait function due to neuronal injuries, suchas hemiplegia and paraplegia [1]–[7]. Providing assist withan exoskeleton robot to compensate for lost / deficient gaitfunction requires estimation of motion intention [7], and properoperation of the robot in coordination with the remaining gaitfunction of the user [8].

Several methods of human motion intention estimation havebeen investigated and realized for operation of human assistiveexoskeleton robots [3], [7], [9], [10]. These methods varyaccording to the intended assistance level (complete reproduc-tion of movement, support of movement etc.), the level ofinjury or dysfunction of the patient (muscle weakness, partialor complete paralysis etc.), and the structure of the assistiverobot itself. The biological signals are reliable information toestimate human motion intention [9]. However, in the case ofneuronal injury / dysfunction such as stroke related hemiplegiaor spinal cord injury, biological signals are different from thoseof healthy subjects, or even not available. Therefore, referencetrajectory for the assisted limb(s) needs to be computed, andthe motion intention is required to be estimated in differentways [2], [3]. Kawamoto et al. [2] developed a control systemfor single leg version of HAL [1] by using FRF (Floor ReactionForce) sensors to detect the gait phase shifting intended by thepatient. The robot is then operated by assembling segments ofreference trajectories to restitute the motion of the impairedlimb. And more extended work has been realized for paraplegiapatients in [3] with the use of body posture informationto convey the motion intention of paraplegic patients. Foranother exoskeleton robot, eLEGS, Krausser and Kazerooni[7] developed a Human Machine Interface for SCI patientswith two crutches. The patient convey his/her intention to therobot using the two crutches to perform Four-Point walkinggait with assistance from the robot and the crutches. Thesensor suit comprises load measurement mechanism on thecrutches, inertial sensors on the arms, force sensors in theshoe insole, and angle sensors on the robots actuators. the The

robot uses hip and knee angle measurements, foot pressure,arm angle, and crutch load to determine the current state andstate transition in a state machine controller customized forFour-Point gait.

The mentioned paradigms do not consider human inter-limbsynergies in gait. Human gait is not only a function of thelower limbs, but also a coordination between upper and lowerlimbs [11]–[13], adding to balance and cognitive functions aswell. Research on human locomotion have shown evidencefor the existence of a task-dependent neuronal coupling ofupper and lower limbs [14], [15]. Also, research on inter-limbcoordination after stroke [16] indicated that stroke patients inacute stage have close to normal synergies in the unaffectedside, and that synergies in the chronic stage depend on the levelof recovery. It was also demonstrated that high functioningstroke patients preserve the ability to coordinate arm and legmovement during walking [17]. In regard to rehabilitation,Ferris et al. [18] suggested using the arms’ swing to facilitatelower limb muscle activation because of the neuronal couplingbetween upper and lower limbs in rhythmic locomotion tasks.Behrman and Harkema [19] have also shown that reciprocat-ing arm swing in a natural and coordinated form facilitatesstepping.

B. Purpose of this ResearchThe field of robot assisted locomotion is yet to explore

the concept of synergies in the control strategy. In this workwe investigate the use of synergies in robot control for robotassisted locomotion. We define the case of hemiparesis as atarget of this research for the following reasons; First, hemi-paresis is one of the prevailing causes of physical disability.Second, persons with hemiparesis have an affected side, and aless-affected side, which adds extra value to consideration ofsynergies in assisted locomotion, to keep coordination betweenhealthy and assisted limbs. Third, there are several reports thatindicate the preservation of inter-limb coordination ability inpersons with hemiparesis, which makes them good candidatesto use and benefit from prospect synergy based systems. Last,persons with hemiparesis usually use a walking aid caneto help maintain balance and support body weight. Whichrepresents a valuable resource to be harvested, in terms ofhuman robot interface, to instrument the cane with sensorsand use it as a device to capture upper limbs motion, and tocontrol the assistive robot.

For this purpose we set to explore the concept of body syn-ergies in robot assisted locomotion in a series of investigations.

2

First, we investigate the inter-limb synergies of walking witha walking aid cane to verify our initial hypothesis, and acquirethe healthy body synergies required for development of asynergy based robot control system. We then formulate a bodysynergy based motion intention estimation and robot controlmethod considering the case of hemiparesis. Second, Wedevelop a wearable gait measurement and robot control system,a system calibration method to integrate motion sensors withthe user’s body, and a control logic for the system. Finally,we devise and conduct a series of experiments to: (i) Verifythe proposed method and developed hardware with healthysubjects. (ii) Conduct pilot trials with persons with hemiparesisand the developed system to verify their ability to use thesystem. Here we measure improvements in gait variables whileusing the robot, and compare the synergy based method toan alternative autonomous robot control method. (ii) Conductsynergy analysis of robot assisted locomotion on the trialswith hemiparetic subjects. Here we measure and quantifykinematic (body joints) synergies and muscle synergies, inorder to investigate differences between the two systems andinfer human-robot interactions in robot assisted locomotion. Inthe following sections we present each of these steps. Then wediscuss the obtained results, and conclude with a summary ofthis work’s novelty and impacts, and future directions.

II. METHODS

In this section we describe the methods used in this research.The methods include definition and analysis of synergies ingait, the formulation of a human motion intention methodbased on synergistic characteristics, and how it could beapplied to exoskeleton robot control. The technical implemen-tation - development of hardware and software to implementthese methods - is presented in a following chapter.

A. Body Synergies

Synergies in human gait are the manifestation of CentralNervous System (CNS) control of body limbs (muscles) toachieve locomotor tasks. In related literature, synergies areinvestigated in terms of kinematics of lower (and upper) limbs[16], [20], or as muscle synergies extracted from muscleactivation signals, EMG, to describe coordination of limbs(muscles) in locomotion [21].

1) Kinematic Synergies: We use the term Kinematic Syn-ergies to refer to body synergies extracted from kinesiolog-ical information of gait. More precisely, angles and angularvelocities of body joints (Hip, knee, shoulder, etc.) are usedin the literature to quantify kinematic synergies in motion /locomotion tasks [20]. Recorded joint angle trajectories areusually recorded with a motion capture system, and then statis-tical methods such as Principal Component Analysis (PCA) orFactor Analysis is used to extract synergies from the observedvariables. Since forward gait has the most pronounced joints’range of motion in the sagittal plane, most research sufficesby using saggital plane trajectories.

0% 20% 40% 60% 80% 100%

Lower limbs

Cane and

Lower limbs

Upper and

Lower limbs

PC1 PC2 PC3 PC4 PC5~

7 Subjects

Fig. 1. Group mean ratios of the first 4 principal components to the overalldata for each of the three sets of variables.

2) Muscle synergies: Muscle Synergies is the widely usedterm for body synergies extracted from muscle activation sig-nals, measured by means of Electromyography (EMG). EMGsignals of several muscles are recorded in the studied motion/ locomotion task, and statistical methods are used to extractsynergies between EMG signals. In recent literature musclesynergies were used to measure and quantify deviations inCNS motor control brought upon by disability. By comparisonof the number and structure of synergies between paretic andnon-paretic limbs of hemiparetic subjects, the number andstructure of synergies were found to be a good indicator ofthe impairment level [21].

3) Synergies in Robot Assisted Locomotion: Muscle syn-ergies provide a deep insight into the operation of CNS inlocomotion since they represent the direct activation signalson muscle surfaces. Kinematic synergies, on the other hand,are the outcome of muscle activation, passive dynamics, andmechanical constraints of gait. For the purpose of exoskele-ton robot control, we consider kinematic synergies to be anappropriate bases for the following reasons:

1) Current exoskeleton robot technologies mostly rely onactuators fitted on limb joints that only provide assist inthe saggital plane.

2) It is possible to make direct mapping from the motionintention estimation algorithm to the robot’s actuators ifboth are in joint-angle coordinates.

3) It is feasible to measure joint angles on body limbs byusing inertial motion sensors.

In the case of Hemiparesis, patients usually uses a cane inthe healthy arm (contralateral to the affected leg) to supporttheir body weight and balance [22], and the cane’s motion is inphase with the affected leg. The role of cane as an orientationand sensory input [23] motivated us to investigate its role inthe synergistic coordination of limbs in gait. Having a cane, thearm is more significantly incorporated in the coordination ofgait, and the movement of cane can be expected to be naturallya part of the upper-lower limb synergies. For this purpose weproposed to conducted gait analysis with walking aid cane.

4) Body Synergy Analysis of Gait with Walking Aid Cane:We designed an experiment to investigate the kinematic body

3

synergies of gait with a walking aid cane. We recorded gaitpatterns of seven healthy subjects of walking with / without acane using a motion capture system (Motion Analysis .co). Thejoint angles and angular velocities of the shoulder, elbow, hipand knee joints for the right and left side limbs, as well as thetilting angle and angular velocity of the cane were computed.Three cases were inspected: (i) Joint coupling of the lowerlimbs. (ii) Joint coupling of the upper and lower limbs. (iii)Coupling of the cane and the lower limbs.

In this investigation we used Principal Component Analysisto investigate the lower limbs’ inter-joint and cane synergies.PCA was performed on three sets of variables. Two sets ofvariables are the joint angles of lower limbs with their derivedangular velocities, and the joint angles of upper and lowerlimbs with their derived velocities. The other set of variablesis the joint angles of the lower limbs and the tilting angle of thecane with their derived angular velocities. We investigated thenumber of Principal Components (PCs) explaining the majorpercentage of data variation for each set of variables amongthe subjects. The accumulated proportion of the first PrincipalComponents (PCs) that exceeds 95% was considered in thisstudy. The first 4 PCs accounted for more than 95% of the datavariation for all the three sets of variables (Fig.1), except forsubjects 5 and 7 while walking with cane, where the percentageof their first 4 PCs was 94.48% and 94.67% respectively, whichstill represents the major percentage of data variation. Thisresult supports our hypothesis that cane’s motion is part of theupper-lower limb synergies, and thus can be used in a synergybased system for motion intention estimation and robot control.

B. Body Synergy based Exoskeleton Robot Control

We propose a method for exoskeleton robot control based onsynergetic relationship between body limbs. The method canbe devised for exoskeleton control in hemiplegia, paraplegia,and healthy people. Here we describe the algorithm developedfor hemiparesis. The method uses kinesiological informationfrom the walking aid cane and the less affected side’s legto estimate the intended motion on the affected side’s leg.Estimation is performed based on inter-limb gait synergiesextracted from healthy people in real time. Figure 2 shows theconcept of the proposed method and measured joint angles, andan illustrative diagram of signal flow in the proposed method.

Using the synergies extracted from walking with cane ofthe healthy subjects, in the form of principal components,we formulated the method for control using the cane forhemiparetic people [24]. We adapted a method for limb motionestimation called Complementary Limb Motion Estimation(CLME) [8], [10] in which trajectories for affected limbs canbe estimated in real time from other less affected limbs. Weuse the synergies of walking with cane from healthy peopleas means for estimation of motion on the affected side, fromvoluntary input in the form of motion information of the caneand the less affected side, as in the following equation:

A = Γ

(U

C

)All motions described here are in the sagittal plane. A is themotion of the affected side to be estimated: hip and knee

Robot Suit HAL(CYBERDYNE Inc.)

Synergy based

Robot Control

Ground Contact

Unaf

fect

ed S

ide

Aff

ecte

d L

eg /

Ro

bo

t

Ɵ Hip

Ɵ Knee ƟCane

Ɵ RobotHip

Ɵ RobotKnee

Unaffected

LegMotion

Affected Leg

MotionCane

Ground

Contact

Parameter

Modification

ROBOT

CONTROL

Synergy based

Motion

Intention

Estimation

Reference

Trajecotry

Fig. 2. Concept and schematic diagram of synergy based control with awalking aid cane.

joints’s angles and angular velocities, U is the motion ofthe unaffected side: hip and knee joint’s angles and angularvelocities and C is the cane’s motion: tilting angle and angularvelocity. The principal components matrix Γ is matrix ofprincipal components used for estimation, which is alreadyextracted from the gait of 7 healthy subjects. This matrix,Γ, provides linear mapping from unaffected side and canemotion to the affected side’s motion. Γ was also calculatedfrom motion data in the sagittal plane. In this sense thedeveloped method is semi-autonomous. Compared to othercontrol methods of lower limbs exoskeleton robots based onkinesiological information [2], [3], [7], this method capturesinput from both the lower limbs (unaffected leg motion) andthe upper limb (cane motion). Using input from the canerepresents a direct voluntary channel to control the assistedmotion on the affected side. And the use of motion informationfrom the less affected side helps generating assisted motionin coherence with the user’s remaining gait function as well.Therefore, the proposed system is theoretically expected tobe able to adapt to the user’s gait pattern due to the nature ofsynergy based control, while offering a voluntary input channelto modify the assisted motion according to the user’s intentionas well.

III. SYSTEM OVERVIEW

In this section we describe the hardware development andimplementation of the proposed synergy based robot controlmethod. This includes the development of a wearable systemfor gait measurement and robot control, system calibration,and the logic for robot control.

4

A. Wearable SystemWe developed a wearable gait measurement system based on

inertial sensors, force sensors and embedded microprocessorsto control exoskeleton robot [25]. The system consists of threeIMU modules: two modules fitted on the thigh and shank ofthe unaffected leg to acquire its motion (Figure 3.a), and amain unit fixed on the cane (Figure 3.b). Modules on thethigh and the shank acquire the motion (angle and angularvelocity) of the hip and knee joints of the unaffected leg. Theshank module is connected to the thigh module with wiredserial communication, while the thigh module streams motiondata from both thigh and shank modules to the main uniton the cane (Figure 3.a,b). The module on the cane is themain unit (Figure 3.c). It receives motion data via bluetoothfrom the thigh module, acquires the cane’s motion (angleand angular velocity) from its own IMU, acquire the groundcontact information from force sensors in the shoes of the robotthrough wireless communication , acquire the cane’s groundcontact information from FSR sensors, compute the controlcommands for the robot according to the current status, andstream those commands to the robot via WIFI communication.The force sensors embedded in the shoes consist of floorreaction force sensors under the heel and forefoot of eachfoot. The sensors provide continuous measurement of the floorreaction forces, and are used together with the FSR sensors onthe tip of the cane to monitor the ground contact patterns forstart-walk-stop support as well as for modification of controlparameters in stance and swing phases.

B. System CalibrationThe sensor fusion algorithm for each IMU takes read-

ings from 3-axis Gyroscope, 3-axis Accelerometer and 3-axisMagnetometer, and outputs the coordinates of sensor framerelative to reference frame (earth frame) in quaternion form.Performance of the algorithm is described in [26], accuracy;<0.8◦ static RMS error, <1.7◦ dynamic RMS error. In orderto find the joint coordinates from the sensor coordinates atransformation is needed from the sensor frame to the jointframe. For performing this transformation we followed aprocedure similar to that in recent methods [27], [28]. Thetransformation from sensor frame to joint frame is given byEquation (6)

qEJ = qES ⊗ qSJ (6)

The quaternions qEJ , qES and qSJ represent the orientationof joint frame relative to earth frame, sensor frame relativeto earth frame, and joint frame relative to sensor frame,respectively. And operator ⊗ is the quaternion multipli-cation. Therefore, to transform the sensor frame to jointframe we need to find the orientation of joint frame rel-ative to sensor frame qSJ . To do this we assume an ini-tial position where the joint frame is known relative toearth frame. In our system we consider the initial positionas quiet standing with the leg fully extended (leg com-pletely vertical) and the person is roughly facing north.In this pose we assume that the joint frame for both hip andknee joints is identical to earth frame. From this position we

can extract the quaternion of joint frame relative to sensorframe as in Equation (7)

qSJ = (qES )−1 ⊗ qEJ (7)

After calculating qSJ from the initial position we can useit to find the joint coordinates from the sensors coordinatesassuming that the sensor mounting on the limb segment willnot change while walking (sensor is attached firmly on thelimb segment). We find the knee joint coordinates from thesensor fixed on the shank, and the hip joint coordinates fromthe sensor fixed on the thigh. Then we extract the joint anglesin the sagittal plane since only motion in the sagittal plane isrequired in our system (the robot only provides assistance inthe sagittal plane).

For the cane module this procedure was not required sincethe module is permanently fixed to the cane and well aligned toits axis. Therefore, just extracting the angle in the sagittal planfrom the sensor’s frame is adequate to produce the requiredcane’s tilting angle.

C. Robot ControlIn our work we use the single leg version of Robot Suit

HAL. The hybrid control algorithm of Robot Suit HAL [1]consists of a human voluntary control and an autonomouscontrol. The wearer’s voluntary muscle activity is obtainedfrom the bioelectrical signals, detected at the surface of themuscles, and then the required assist torque of the actuatorsis computed from the estimated joint torque. An autonomouscontrol is also implemented based on the pre-determinedmotion primitives, together with the voluntary control method.In this work we provide the control reference to the robot fromthe developed wearable measurement system, and the robot’sembedded motor control algorithm handles the execution. Thismodular approach for robot control allows for stacking addi-tional modules of control in the future, allowing the capacityfor further considerations (e.g. balance monitoring and headorientation).

Robot control with the developed wearable system will beexplained here in detail. The system monitors the status ofa start-stop button fitted on the handle of the cane and theground contact patterns of the feet and the cane to detectstart, walk, and stop conditions (Figure 7). We figured the startand stop conditions for this particular version considering thecase of left side hemiplegia, where the user would be holdingthe cane with the right arm (unaffected side), and the robotwould be fitted on the left leg (affected side). In this case weconsider that the user would typically start with the left legand the cane, since the right (unaffected) leg is more capableof supporting the body weight and balance requirements forstarting. Accordingly, the start assist is triggered when thebutton is on, the right foot ground contact force is large,and the left foot and cane ground contact forces are small(Figure 7). Transmission to the continuous walking mode ismade at the next heel strike of the assisted leg, a state atwhich the unaffected leg is near to toe-off, and the cane isat contact with ground or close to it (Figure 7). From thispoint assistance would be based on synergies based motion

5

Earth[East,North,Up]

y

z

x

Knee

Hip

Shank

Thigh

Cane

�

FSR

WIFI Module Bluetooth Communication

Connectors:

* FSR Sensors

* Control Buttons

* Extensions

Accelerometer,

Gyroscope,

Magnetometer

Main Processor

a) b)

c)

Shank Thigh Cane

≈

≈

85 mm

72 m

m

Weight: 110 grams

Robot

Suit

HAL

Microprocessor

Microprocessor Microprocessor

IMUIMU

BlueTooth

IMU

BlueToothWIFI

Module

FSR

sensors

d)

Fig. 3. Wearable system configuration and frame calibration.

estimation from the cane and unaffected leg, as explained inthe previous section. The estimated trajectories are streamedto the robot, and tracked with the actuators on the robot’ship and knee joints with PD controllers. The ground contactinformation from the robot’s feet are used to modify controlparameters in different conditions (Stance, Swing). To stopwalking the user pushes the handle button again to release,then at the next heel contact of the unaffected leg (Figure7), toe-off of the affected leg, the stop motion would start,leading to quiet standing condition (Figure 7). This patternis also based on the stopping motion being supported by theunaffected leg.

IV. EXPERIMENTS

We devised and conducted a series of experiments to verifyand evaluate the function of the proposed method and de-veloped system in healthy and hemiparetic subjects. Here wepresent these experiments with their goals and outcomes.

A. Verification Experiment with Healthy PeopleObjective: We devised this experiment to verify function

of the developed method with healthy people, and to test thefunction of the developed wearable gait measurement and robotcontrol system against a motion capture system.

Setup: We implemented the proposed synergy based controlmethod using two systems, one is implemented with a 3D-Motion Capture System (MOCAP), and one with the devel-oped wearable systems based on IMU’s. We asked four healthysubjects to walk on a treadmill with the MOCAP and wearablesystems. Experiments were done with a left leg version ofRobot Suit HAL, with the cane being used in the right arm(Figure 4.a). We only used the continuous walk support, toavoid any fall risks that could result from using the start andstop support on a treadmill. We evaluated the resulting gait

Trials with MOCAP control

Trials with Wearable System control

a)

b)

−20

−10

0

10

20

30

0% 100%Cane

[deg]

0

10

20

30

40

50

Robot Knee

−20

−10

0

10

20

30

Robot Hip

0

10

20

30

40

50

60

70

Unassisted Knee

−20

−10

0

10

20

30

Unassisted Hip

Fig. 4. Experimental setup and average trajectories for all subjects. Tra-jectories shown are the averages of the extracted 10 gait cycles for eachtrial, with the gait cycle duration normalized for all trajectories to comparerange differences and trends in those trajectories. The dark lines represent thetrajectories for trials with MOCAP control, and the lighter lines represent thetrajectories with wearable system control.

variables for both cases and compared the results among thetwo.

Results

We extracted and compared the trajectories and step relatedgait variables from the walking trials. For each subject weextracted 10 consequent gait cycles from a trial of walkingwith the wearable system and 10 consequent gait cycles froma trial with the MOCAP system.

The joint angle trajectories show close to normal assistedmotion trajectory on the robot’s hip and knee joints, comparedto that of the unassisted motion on the right leg’s hip andknee trajectories (figure 4.b). However, the range of motionon the robot’s knee was smaller than that on the other side.This observation has several possible underlying causes. Oneis imperfections in the motion estimation algorithm, another isthe change in balance and gait dynamics resulting from wear-ing a robot on one side of the body. From the cane’s trajectorywe note some variation in range between the subjects, as we

6

0

20

40

60

80

100

120

140

160

Subject 1 Subject 2 Subject 3 Subject 4

MOCAP Wearable System

Cad

ence

[st

rid

es /

min

]

Fig. 5. Cadence.

0

100

200

300

400

500

MOCAP IMU MOCAP IMU MOCAP IMU MOCAP IMU

Subject 1 Subject 2 Subject 3 Subject 4

Right Step Left Step

Ste

p L

ength

[m

m]

Fig. 6. Right and Left Step Lengths.

encouraged subjects to adjust the motion of the cane to reachmore comfortable gait.

0

60

0

60

−20

0

20

Right HipRight Knee

Robot HipRobot Knee

Cane

HeelToe

HeelToe

B B

Standing Start StopContinue Standing

Right Foot

Left Foot

Degrees

Degrees

Degrees

Force

Force

Fig. 7. Trajectories of the joint angles, cane, and foot sensors during thestart and stop experiment. Underneath the trajectories are illustrations of themoments of transition.

Though subjects walked at the same speed on the treadmill,they had different body constitutions and walked at their ownpreferred cadence (figure 5). Subjects had slightly varyingstep length between the right and left sides (figure 6). Which

reflected on the symmetry ratio (considered here as the ratioof right step to left step). Subjects 1 and 2 achieved close to1 (more symmetrical gait) ratios for both the wearable systemand MOCAP trials, while subjects 3 and 4 had more varyingsymmetry ratios, closest to 1 is the wearable system trial of 4.

B. On-ground start and stop experimentObjective & Setup: We conducted this experiment to verify

the usability of the start and stop functions in the developedsystem. We asked one healthy subject to walk on ground withthe robot and developed wearable system. Function of thesystem and start stop function were explained to the subject.The subject performed 7 steps of walking on ground includingstart and stop steps.

Result: The subject was able to successfully walk with therobot, and transitions from start to continues gait to stop wereperformed successfully as shown in trajectories and transitionmoments in figure 7.

C. Pilot Study with Hemiparetic SubjectsObjective: In this investigation we set to verify the clinical

applicability of the developed exoskeleton control method withhemiparetic persons. The objective of this experiment is not touse the method in a clinical application such as rehabilitation,rather to verify the applicability of the proposed method anddeveloped system in such a clinical environment. Also, theobjective is to compare the performance of developed methodrelative to an alternative autonomous method based on pre-recorded gait trajectories, and detection of intended phaseshift from shoe insole sensors. The outcomes assessed in thisexperiment are:

1) The subject being able to walk with the system (controlthe robot with the wearable sensors and cane).

2) The step length of right and left legs when walking with/ without the robot (symmetry ratio).

3) The joints’ range of motion when walking with / withoutthe robot (hip and knee joints on both sides in the sagittalplane).

4) The cadence and time for 10 meters when walking with/ without the robot.

Subjects: In collaboration with the Physical RehabilitationDepartment in the University of Tsukuba Hospital we recruitedfive hemiparetic persons as test-pilot volunteers to evaluatethe clinical applicability of the proposed method. The subjectswere all left side hemiparetic who are using a walking aidcane, or have used a cane at some point after acquiringthe hemiparetic condition. All subjects signed an informedconsent, and all procedures were approved for by the ethicscommittee in the University of Tsukuba. The subjects’ clinicalcondition and locomotion ability are shown in Table ??.

Setup: We asked the subjects to walk on flat ground ina 16-camera motion capture room. Each subject walked the10 meters pathway two times without the robot. Those whowere able to use the robot walked with the developed system,and then with the autonomous control system. There was notraining sessions prior to the experiments, and all subjects

7

TABLE I. TABLE OF SUBJECTS AND THEIR RESPECTIVE CLINICAL CONDITION / LOCOMOTION ABILITY.

Subject (SEX:AGE) Stroke Type Affected Brain Side Stroke Location Duration from Stroke Brunnstrom Stage Barthel IndexM:58 Cerebral Hemorrhage Right Putamen 5 years IV 75M:44 Subarachnoid Hemorrhage Right Middle Cerebral Artery 4 years IV 95M:46 Cerebral Infarction Right BrainStem 8 years V 95F:62 Cerebral Hemorrhage Right Thalamus 8 years IV 85M:52 Cerebral Hemorrhage Right Putamen 18 years III 95

had not walked with an assistive exoskeleton before. Sinceall subjects had left side hemiparesis (right side of the brainis affected), they used a left side version of the robot, withthe cane in right arm, and wearable sensors on the right leg.A capture from the experiment is shown in Figure 8 withall mounted devices. Since we asked the subjects to use theinstrumented cane that we developed instead of their usualwalking aid canes, we modified the height of the cane foreach subject to their comfort. Reflexive markers were fixed onthe subjects’ lower limbs, the cane and the robot to capturemotion trajectories in the trials using a 16 camera 3D motioncapture system(Vicon MXTM System). Wireless EMG sensors(DELSYS Trigno TM) were fitted on the surface of iliopsoas,gluteus maximus, rectus femoris and biceps femoris musclesof both legs.

Results: Three participants were able to walk with therobot (M:58, M:44, M:46). Following data analysis and resultsare only for the three subjects who were able to walk withthe robot. The first trial, 10 meters, of each condition wasconsidered as a familiarizing phase and was not used in theanalysis. Only data from the second trial of each condition wasused to create the results.

Joint range of motion: When walking without the robot,subjects had similar range of motion on the hip joint betweenright and left legs, but the left side’s (affected side) kneesuffered a pronounced decrease in range of motion comparedto the less-affected side (Figure 9). When walking with therobot, with both the autonomous system and the synergy basedsystem, there was an increased range of motion on the affectedside’s knee joint compared to walking without the robot, whilethe affected side’s hip joint range of motion varied in decreaseor increase compared to walking without the robot. It can benoted that there was also a difference in joints’ range of motionon the less-affected side as well, though not directly acted onby the robot, which was in general a decrease in range ofmotion of these joints.

Step length: There was no obvious trend in the increaseor decrease of the step length of either legs among the subjects(Figure 10). As for the symmetry ratio (right leg step lengthto left leg step length), subjects had better symmetry ratio(value closer to 1) without the robot and with the synergybased system compared to that with the autonomous system.By comparison of the synergy based system and no-robotconditions, we found that two out of the three subjects hadbetter symmetry ration with the synergy based system, whileone had better symmetry ration without the robot.

Cadence and speed: Subjects were instructed to walk attheir preferred speed in all trials. All subjects walked slower(time for 10 meters) and at a lower cadence (steps per minute)

Robot Suit HAL ®

IMU Sensors

Cane Module

(IMU sensor,

bluetooth/WiFi,

Microcontroller)

Fig. 8. Overview of a subject walking with the robot, walking aid cane andthe wearable sensory system.

with the robot compared to walking without the robot (TableII).

TABLE II. CADENCE AND TIME FOR 10 METERS FOR ALL SUBJECTS.

Cadence 10 Meters TimeS1 No Robot 35 (step/minute) 20.32 (Sec)

S1 Robot (autonomous system) 28.28 30.18S1 Robot (synergy based system) 29.52 22.22

S2 No Robot 41.63 15.64S2 Robot (autonomous system) 34.03 16.96

S2 Robot (synergy based system) 33.5 19.81S3 No Robot 36.75 16.32

S3 Robot (autonomous system) 25.94 21.03S3 Robot (synergy based system) 26.75 22.43

D. Synergy Analysis in Assisted LocomotionObjective: The concept of synergy analysis has surfaced

in recent literature of motion analysis [21], [29]. In assistedlocomotion, kinematics synergies are a result of the finaloutcome of robot assisted gait captured by the motion capturesystem, while the muscle synergies are a direct result of theuser’s intended motion. Therefore, by comparing the kinematicsynergies with corresponding muscle synergies under differentcontrol systems, we can infer the interaction between the user

8

0

300

400

500 [mm]

Subject 1 Subject 2 Subject 3

Right and Left legs’s step length Right leg step length Left leg step length

No Robot Autonomous

System

Synergy based

SystemNo Robot

Autonomous

System

Synergy based

SystemNo Robot

Autonomous

SystemSynergy based

System

Fig. 10. Right (healthy) and left (affected) legs’ step length averaged for 10 gait cycles of the three subjects walking with and without the robot.

0

20

40

60 [degrees]

0

20

40

60

0

20

40

60

Right Hip Right Knee Left Hip Left Knee

No Robot Autonomous System Synergy based System

Su

bje

ct 1

Su

bje

ct 2

Su

bje

ct 3

Average Joint Ranges in all Trials

Fig. 9. Joints’ range of motion averaged for 10 gait cycles of the threesubjects walking with and without the robot.

and the robot. We propose such synergy comparison as acriteria to evaluate the function of different exoskeleton robotcontrol system in regard to synergestic cohesion with the user’sgait.

Method: EMG recordings were acquired from the pilotstudy, as explained in the previous section, of the three hemi-paretic subjects walking without the robot, with the robot usingthe developed control method, and with the robot using anautonomous control method. Recordings were acquired fromthe motion capture data for joint angles, and from wirelessEMG sensors for iliopsoas, gluteus maximus, rectus femoris

and biceps femoris muscles of both legs. EMG signals weredown-sampled to 1 Khz, smoothed with a butterworth band-pass filter (40 400 Hz, 3rd order), rectified, integrated andvariance normalized.

Synergies were extracted using Non-Negative Matrix Fac-torization (NNMF). This method represents target variablesas a linear combination of time-invariant synergies and time-varying activation coefficient. To calculate the kinematic syn-ergies we used NNMF with the joint angle trajectories in thesagittal plane of right and left legs’ hip and knee joints. As formuscle synergies, due to difficulties in fitting the EMG sensorstogether with the robot and other wearable motion sensors,some EMG signals were corrupted. Therefore we excluded thetwo EMG signals that were most affected on each leg, leaving4 signals for the analysis, two from each side for each subject.

Results: We calculated the correlation coefficients (Figure11.c) between synergies of walking without the robot andwalking with the robot under autonomous control and synergybased control respectively. The results for muscle synergies(Figure 11.a) show very little difference between the No robotand autonomous control correlation coefficient, and the Norobot and Synergy based control correlation coefficient (differ-ence <= 0.06), for all subjects. On the other hand, the sameinvestigation on kinematic synergies (Figure 11.b) showedbigger differences, 0.2 for first subject, 0.15 for second subject,and 0.34 for the third subject. The results also show that thesynergy based system had higher correlation coefficients withwalking without the robot in terms of kinematic synergies, thanthe autonomous system did, even though the muscle activationdid not differ as significantly.

V. DISCUSSION

In the body synergy analysis of gait with walking aid werevealed an important observation, that a walking aid in gait isused in coordination with other body limbs, or in other words,the falls within the inter-limb synergies of gait. This result isof importance for engineering applications since it providesthe background to instrument and utilize a walking aid caneas part of an interface with exoskeleton robot. It is also ofimportance from a physiological aspect, as in the theory oftool use, and tool representation in the body schema.

9

S1

S2

No Robot Automatic

Control

Synergy based

Control

S1

S2

S1

S2

0 1

a) Muscle SynergiesS

ub

ject

1S

ub

ject

2S

ub

ject

3b) Kinematic Synergies

No Robot Automatic

Control

Synergy based

Control

Muscle Synergies No robot / Autonomous control No robot / Synergy based control

Subject 1 0.88 0.83

Subject 2 0.73 0.67

Subject 3 0.98 0.98

Kinematic Synergies No robot / Autonomous control No robot / Synergy based control

Subject 1 0.77 0.97

Subject 2 0.78 0.93

Subject 3 0.64 0.98

c) Correlation Coefficients

Fig. 11. a) Muscle and b) kinematic synergies, and c) correlation coefficientsbetween synergies of walking without and with the robot.

In the verification experiment with healthy people we in-spected the function of the developed wearable gait measure-ment and robot control system compared to an optical motioncapture system. We found the function of the wearable systemto be comparable to that of the MOCAP system, which meansit can be used for robot control without serious deteriorationin performance or accuracy. The wearable system also havethe advantage of being much more compact and portable, thuspossible to implement and use in various environments andscenarios. An important observation of this experiment is thatall subjects were able to walk the developed synergy basedsystem at different cadence and step lengths. This outcomereflects the flexible nature of the synergy based system, sinceit generates assisted motion based on motion of the user’scontralateral leg and cane, and thus inherently adapts with theuser’s gait pattern.

In the pilot study with hemiparetic subjects we verified thefeasibility of the proposed method and developed system withhemiparetic persons. Three out of five participants were ableto walk with the system. Although gait speed and cadencedecreased from those of walking without the robot, that is tobe expected from first time users of an exoskeleton robot due toproblems of habituation and altered dynamics due to wearingthe robot on one side of the body. On the other hand, walkingwith the robot and the developed synergy based system leadto noticeable improvement in some other gait variables suchas range of motion on the affected leg’s knee joint, and step-length symmetry ratio in two out of the three subjects. Fromthe results of joints’ range of motion we observe a general

trend that the autonomous system provided a high increase injoints’ range of motion on the affected side that they surpassedthe range of motion on the unaffected side. On the otherhand, with the synergy based system, the joints’ range ofmotion increase on the affected side was present but lesser thanthe range on the unaffected side. The percentage differencebetween right and left sides’ hip joint was lesser with thesynergy based system, and higher with the autonomous system,and vice versa for the knee joint. These results are probablydue to the functional difference between the two systems.The autonomous system provides assist based on the detectedphase shift on the foot sensors solely, and does not take intoconsideration the joints’ motion of the unaffected side. Thesynergy based system, on the other hand, generates assistedmotion based on direct mapping from the unaffected leg andcane motion, and thus has the inherent ability of adapt to theuser’s gait.

In the synergy analysis in robot assisted locomotion weexplored a novel method of gait analysis during robot assistedlocomotion. Depending on the difference between kinematicand muscle synergies in robot assisted locomotion, the earlierbeing a direct result of robot actuation and later being theactual commands of the user’s CNS. We were able to finda simple holistic variable that quantifies consistency betweenthe robot action and the wearer’s action. The two control sys-tems behaved differently in terms of synergetic relationshipsbetween the limbs, even though the muscle activation did notdiffer as significantly. The synergy based system had highercorrelation coefficients with walking without the robot in termsof kinematic synergies, than the autonomous system did. Thisindicates that the synergy based system had higher ability ofoperating in cohesion with the user’s body synergies, yet it wasable to produce enhancements in some gait parameters suchas range of motion on the assisted knee joint and symmetryratio for two out of the three subjects.

We suggest that the proposed method of synergy analysis inassisted locomotion can be used as a holistic criteria for user-robot interaction in robot assisted locomotion. We also suggestthat synergy based robot control / synergy based analysis ofassisted locomotion, can be used in rehabilitation programsto provide assist in cohesion with users’ body synergies, andto quantify human-robot interaction, and adaption (change) ofuser’s body synergies during and after rehabilitation.

VI. CONCLUSIONS AND FUTURE DIRECTIONS

In this research we explored and realized an approach for theuse of body synergies in gait analysis and exoskeleton controlfor robot assisted locomotion.

We formulated an online motion intention estimation algo-rithm using motion of healthy limbs, and an instrumented cane,to estimate the motion of affected limbs. We then developeda sophisticated robot control system with wearable sensors,implemented with the single leg version of Robot Suit HAL.

The main contribution of this research is in introducingthe concept of synergy to the field of exoskeleton robotcontrol and gait analysis in assisted locomotion. This approachhelps bridge the gap between human motor control and robot

10

control, to bring assistive technologies closer to practicalimplementation in rehabilitation and assisted life styles. Whilethe concept of synergies have been investigated in fields suchas physiology, neuro-science, and brain science for over adecade, this research is one of the first real attempts to importthis knowledge into implementation in engineering field.

Traditional gait variables such as gait speed, cadence, steplength and symmetry ratio are the main criteria used in charac-terization of gait. But these criteria are becoming insufficientof describing the interaction during robot assisted locomotion.New exoskeleton robots and control methods are constantlybeing developed. Many of which are based on motion intentionestimation, and aim at a user cooperative scheme. Thus, findingnew criteria to evaluate human-robot interaction in assistedlocomotion, such as the one we suggested in the synergyanalysis, will be of great help in evaluating different robotcontrol methods. Although the analysis of data acquired inthe pilot study was not fully comprehensive, it paves the wayfor future extended studies to evaluate and apprehend bodysynergies in robot assisted locomotion.

REFERENCES

[1] Y. Sankai, “HAL: Hybrid assistive limb based on cybernics,” in RoboticsResearch, ser. Springer Tracts in Advanced Robotics, M. Kaneko andY. Nakamura, Eds. Springer Berlin / Heidelberg, 2011, vol. 66, pp.25–34.

[2] H. Kawamoto, T. Hayashi, T. Sakurai, K. Eguchi, and Y. Sankai, “De-velopment of single leg version of HAL for hemiplegia,” in Engineeringin Medicine and Biology Society, 2009, sept. 2009, pp. 5038 –5043.

[3] K. Suzuki, Y. Kawamura, T. Hayashi, T. Sakurai, Y. Hasegawa, andY. Sankai, “Intention-based walking support for paraplegia patient,” inSystems, Man and Cybernetics, 2005 IEEE International Conferenceon, vol. 3, oct. 2005, pp. 2707 – 2713 Vol. 3.

[4] H. Kawamoto, K. Kamibayashi, Y. Nakata, K. Yamawaki, R. Ariyasu,Y. Sankai, M. Sakane, K. Eguchi, and N. Ochiai, “Pilot study oflocomotion improvement using hybrid assistive limb in chronic strokepatients,” BMC Neurology, vol. 13, no. 1, p. 141, 2013.

[5] S. Kubota, Y. Nakata, K. Eguchi, H. Kawamoto, K. Kamibayashi,M. Sakane, Y. Sankai, and N. Ochiai, “Feasibility of rehabilitationtraining with a newly developed wearable robot for patients with limitedmobility,” Archives of Physical Medicine and Rehabilitation, vol. 94,no. 6, pp. 1080 – 1087, 2013.

[6] J. Veneman, R. Kruidhof, E. Hekman, R. Ekkelenkamp, E. van As-seldonk, and H. van der Kooij, “Design and evaluation of the lopesexoskeleton robot for interactive gait rehabilitation,” Neural Systemsand Rehabilitation Engineering, IEEE Transactions on, vol. 15, no. 3,pp. 379 –386, sept. 2007.

[7] K. A. Strausser and H. Kazerooni, “The development and testing ofa human machine interface for a mobile medical exoskeleton,” inIntelligent Robots and Systems (IROS), 2011 IEEE/RSJ InternationalConference on, sept. 2011, pp. 4911 –4916.

[8] H. Vallery and M. Buss, “Complementary limb motion estimationbased on interjoint coordination using principal components analysis,”in Computer Aided Control System Design, 2006 IEEE InternationalConference on Control Applications, 2006 IEEE International Sympo-sium on Intelligent Control, 2006 IEEE, oct. 2006, pp. 933 –938.

[9] T. Hayashi, H. Kawamoto, and Y. Sankai, “Control method of robotsuit HAL working as operator’s muscle using biological and dynamicalinformation,” in Intelligent Robots and Systems, 2005. (IROS 2005).

2005 IEEE/RSJ International Conference on, aug. 2005, pp. 3063 –3068.

[10] H. Vallery, E. van Asseldonk, M. Buss, and H. van der Kooij, “Ref-erence trajectory generation for rehabilitation robots: Complementarylimb motion estimation,” Neural Systems and Rehabilitation Engineer-ing, IEEE Transactions on, vol. 17, no. 1, pp. 23 –30, feb. 2009.

[11] T. Wannier, C. Bastiaanse, G. Colombo, and Dietz, “Arm to leg coor-dination in humans during walking, creeping and swimming activities,”Experimental brain research, vol. 141, no. 3, pp. 375–379, 2001.

[12] V. Dietz, K. Fouad, and C. M. Bastiaanse, “Neuronal coordinationof arm and leg movements during human locomotion,” The Europeanjournal of neuroscience, vol. 14, pp. 1906–1914, dec. 2001.

[13] E. P. Zehr and J. Duysens, “Regulation of arm and leg movement duringhuman locomotion,” Neuroscientist, vol. 10, no. 4, pp. 347–361, 2004.

[14] J. Duysens and H. W. V. de Crommert, “Neural control of locomotion;part 1: The central pattern generator from cats to humans,” Gait &Posture, vol. 7, no. 2, pp. 131 – 141, 1998.

[15] S. Grillner, Control of Locomotion in Bipeds, Tetrapods, and Fish. JohnWiley & Sons, Inc., 2011.

[16] I. Milovanovi and D. B. Popovi, “Principal component analysis of gaitkinematics data in acute and chronic stroke patients,” Computationaland Mathematical Methods in Medicine, vol. 2012, 2012.

[17] J. L. Stephenson, A. Lamontagne, and S. J. D. Serres, “The coordinationof upper and lower limb movements during gait in healthy and strokeindividuals,” Gait & Posture, vol. 29, no. 1, pp. 11 – 16, 2009.

[18] D. P. Ferris, H. J. Huang, and P.-C. Kao, “Moving the arms to activatethe legs,” Exercise and Sport Sciences Reviews, vol. 34, pp. 113–120,2006.

[19] A. L. Behrman and S. J. Harkema, “Locomotor training after humanspinal cord injury: A series of case studies,” Physical Therapy, vol. 80,no. 7, pp. 688–700, 2000.

[20] N. St-Onge and A. Feldman, “Interjoint coordination in lower limbsduring different movements in humans,” Experimental Brain Research,vol. 148, pp. 139–149, 2003.

[21] V. C. K. Cheung, A. Turolla, M. Agostini, S. Silvoni, C. Bennis, P. Kasi,S. Paganoni, P. Bonato, and E. Bizzi, “Muscle synergy patterns asphysiological markers of motor cortical damage,” Proceedings of theNational Academy of Sciences, vol. 109, no. 36, pp. 14 652–14 656,2012.

[22] T.-S. Kuan, J.-Y. Tsou, and F.-C. Su, “Hemiplegic gait of strokepatients: The effect of using a cane,” Archives of Physical Medicineand Rehabilitation, vol. 80, no. 7, pp. 777 – 784, 1999.

[23] J. J. Jeka, “Light touch contact as a balance aid,” Physical therapy,vol. 77, no. 1, pp. 476 – 487, 1997.

[24] M. Hassan, H. Kadone, K. Suzuki, and Y. Sankai, “Exoskeleton robotcontrol based on cane and body joint synergies,” in Intelligent Robotsand Systems (IROS), oct. 2012, pp. 1609 –1614.

[25] ——, “Wearable gait measurement system with an instrumented canefor exoskeleton control,” Sensors, vol. 14, no. 1, pp. 1705–1722, 2014.

[26] S. Madgwick, A. Harrison, and R. Vaidyanathan, “Estimation of imuand marg orientation using a gradient descent algorithm,” in Rehabilita-tion Robotics (ICORR), 2011 IEEE International Conference on, 2011,pp. 1–7.

[27] J. Bergmann, R. Mayagoitia, and I. Smith, “A portable system forcollecting anatomical joint angles during stair ascent: a comparison withan optical tracking device,” Dynamic Medicine, vol. 8, no. 1, p. 3, 2009.

[28] I. Prayudi and D. Kim, “Design and implementation of imu-basedhuman arm motion capture system,” in Mechatronics and Automation(ICMA), 2012 International Conference on, aug. 2012, pp. 670 –675.

[29] Y. P. Ivanenko, R. E. Poppele, and F. Lacquaniti, “Motor controlprograms and walking,” The Neuroscientist, vol. 12, no. 4, pp. 339–348, 2006.