DEVELOPMENT OF FACIAL MUSCLES

CONTROLLED ROBOTIC SYSTEM

BY

AHMAD JORAIMEE BIN MOHAMAD

A dissertation submitted in partial fulfillment of the

requirements for the degree of Master of Science

(Mechatronic Engineering)

Kulliyyah of Engineering

International Islamic University

Malaysia

MAY 2013

i

ABSTRACT

The work in this dissertation focuses on the development of the facial muscles

controlled robotic system. The main idea of the work is to control the movement of

the small mobile robot by the recorded facial muscle signals. The characteristic of the

recorded facial muscles is discussed thoroughly in the dissertation. The facial muscle

signals used, which are known as Electromyography (EMG) are obtained after passing

the raw signals through EMG system that consist of stage-by-stage amplification and

filtering circuits. A set of three channels of EMG circuit consists of a set of

instrumentation amplifier INA128P and four precision amplifiers OP177 are used to

build the circuit. The raw captured signal is amplified by the gain of 8,415 and band

pass filtered from 50Hz to 500Hz. A microcontroller PIC16F877A starter kit is used

to convert the analogue signal from the circuit into digital form before the signal is

transferred to the receiver on the small mobile robot wirelessly by using XBee module

which can work within 100m radius. The alternative features of the system can be

used to create more interesting facial muscle rehabilitation therapy for the patients

who suffer from strokes.

البحثملخص

. الإنسان الآلييتركزالعمل في هذه الأطروحة على تطوير عضلات الوجه بواسطة السيطرة عن طريق عن طريق الإشارات المحمول الصغير الإنسان الآليالعمل هو السيطرة على حركة الفكرة الرئيسية لهذ

المسجله لعضلات الوجه في هذه الأطروحه تم مناقشة المزايا . عضلات الوجه المسجلة من الصادرةيتم الحصول عليها بعد (EMG)إشارات عضلات الوجه المسجلة والمعروفه بنظام ال . بشكل مستفيظ

والذي يتكون من دوائر التضخخيم والترشيح مرحلة تلو EMG المرور بالإشارات الأصليه خلال نظام التتكون من مجموعة الأجهزه المكبره (EMG)ن من ثلاث قنوات من الدوائر هناك مجموعة تتكو .أخرى

INA128P وآربعة مكبرات ضبطOP177 الإشارة الاصلية المحتجزه تكبر عن . تستخدم لبناء الدائرةوالذي PIC16F877A المتحكم الدقيقهرتز. 500هرتز الى 50موجة ترشيح من 8,415طريق اكتسابها

الإنسان الآليوتنقل إلى المستقبل في هو البادئ يستخدم لتغيير الإشارة الرقمية قبل أن تبدأ الإشارةمميزات هذ النظام م. 100والذي يعمل بنصف قطر XBee المحمول الصغير لاسلكيا بواسطة نموذج

ين يعانون من السكتات هتمام بتأهيل عضلات الوجه لعلاج المرضى الذللإهوامكانية خلق إثارة أكثر الدماغية.

iii

APPROVAL PAGE

I certify that I have supervised and read this study and that in my opinion it conforms

to acceptable standards of scholarly presentation and is fully adequate, in scope and

quality, as a dissertation for the degree of Master of Science in Mechatronics

Engineering.

…………………………………..

Shahrul Naim Sidek

Supervisor

I certify that I have read this study and that in my opinion it conforms to acceptable

standards of scholarly presentation and is fully adequate, in scope and quality, as a

dissertation for the degree of Master of Science in Mechatronics Engineering

…………………………………..

Amir Akramin Shafie

Examiner

This dissertation was submitted to the Advanced Engineering & Innovation Center

and is accepted as a fulfillment of the degree of Master of Science in Mechatronics

Engineering.

…………………………………..

Iskandar Idris Yaacob

Head, Advanced Engineering &

Innovation Center

This dissertation was submitted to the Kulliyyah of Engineering and is accepted as a

fulfillment of the degree of Master of Science in Mechatronics Engineering.

…………………………………..

Md. Noor Bin Salleh

Dean, Kulliyyah of Engineering

iv

DECLARATION

I hereby declare that this dissertation is the result of my own investigations, except

where otherwise stated. I also declare that it has not been previously or concurrently

submitted as a whole for any other degrees at IIUM or other institutions.

Ahmad Joraimee bin Mohamad

Signature: ………………………………………. Date: ………………..

v

INTERNATIONAL ISLAMIC UNIVERSITY MALAYSIA

DECLARATION OF COPYRIGHT AND AFFIRMATION

OF FAIR USE OF UNPUBLISHED RESEARCH

Copyright © 2013 by International Islamic University Malaysia. All rights reserved.

DEVELOPMENT OF FACIAL MUSCLES CONTROLLED

ROBOTIC SYSTEM

No part of this unpublished research may be reproduced , stored in a retrieval

system, or transmitted, in any form or by any means, electronic, mechanical,

photocopying, recording or otherwise without the permission of the copyright holder

except as provided below.

1. Any material contained in or derived from this unpublished research may

only be used by others in their writing with due acknowledgement.

2. IIUM or its library will have the right to make and transmit copies (print

or electronic) for institutional and academic purpose.

3. The IIUM library will have the right to make, store in retrieval system and

supply copies of this unpublished research if requested by other

universities and research libraries.

Affirmed by Ahmad Joraimee bin Mohamad

……………………….. …….……………….

Signature Date

vi

To:

My parent

Dr Norulakma Zainon

Najwan Hakimi

Najlaa Farhani

Nadhirah Kamilia

Nayli Sofiyya

vii

ACKNOWLEDGEMENT

First and foremost, praise to Allah who has given me guidance and support in

completing this research work and receiving His blessing.

I would like to express my deepest appreciation to my supervisor, Dr. Shahrul

Naim Bin Sidek for all the supports he has given towards completing this research

work. His precious time and dedication has pushed me through the difficult gain.

Also a special thanks to the staff of Faculty of Electrical and Automation

Engineering Technology TATI University College especially En Mohd Tarmizi

Ibrahim and En Kharuddin Ali for giving me the technical support and advises, which

has significantly contributed to the success of this research work.

Last but not least, a very special thanks to my parent and wife, Dr Norulakma

bt Zainon for their boundless love and patience to encourage me towards realizing my

goals.

viii

TABLE OF CONTENTS

Abstract .......................................................................................................................... i Abstract in Arabic ........................................................................................................ ii

Approval Page ............................................................................................................. iii Declaration ................................................................................................................... iv Acknowledgement ...................................................................................................... vii List of Figures ............................................................................................................... x List of Tables ............................................................................................................. xiii

List of Symbols .......................................................................................................... xiv

List of Abbreviations ................................................................................................. xv

CHAPTER ONE: INTRODUCTION ........................................................................ 1 1.1 Background ................................................................................................. 1 1.2 Problem statement ....................................................................................... 2 1.3 Research objectives ..................................................................................... 3

1.4 Research Methodology................................................................................ 3 1.5 Research Scope ........................................................................................... 6 1.6 Thesis organization ..................................................................................... 7

CHAPTER TWO: LITERATURE REVIEW ........................................................... 8 2.1 Introduction ................................................................................................. 8

2.1.1 Electrocardiography (ECG) ............................................................. 8

2.1.2 Electroencephalography (EEG) ....................................................... 9 2.1.3 Electrooculography (EOG) ........................................................... 10

2.1.4 Electroretinography (ERG) ........................................................... 10 2.1.5 Electromyography (EMG)............................................................. 11

2.2 Method for Rehabilitation of Facial Muscle for Stroke Patient ................ 12

2.3 Review of previous researches and projects ............................................. 15 2.4 Summary ................................................................................................... 22

CHAPTER THREE: SYSTEM DEVELOPMENT AND DESCRIPTION ......... 23 3.1 Introduction ............................................................................................... 23 3.2 Block Diagram of the Research ................................................................ 23

3.3 Flow Chart of the Research ....................................................................... 24 3.4 Relation between Facial Muscles and Facial Expression ......................... 27

3.5 Method to Acquire EMG signal ................................................................ 29 3.6 Signal conditioning ................................................................................... 30

3.6.1 Electrode ........................................................................................ 31 3.6.2 Amplifier ....................................................................................... 33 3.6.3 Bandpass Filter .............................................................................. 34

3.6.4 Precision full-wave rectifier circuit ............................................... 38 3.7 PIC Microcontroller .................................................................................. 42

3.8 Radio Frequency (RF) communication ..................................................... 44 3.8.1 X-Bee transreceiver ....................................................................... 45

3.9 Small mobile robot .................................................................................... 46

3.10 Summary ................................................................................................... 48

ix

CHAPTER FOUR : RESULTS AND DISCUSSION ............................................. 50 4.1 Introduction ............................................................................................... 50 4.2 Analysis of Signal Conditioning Circuit ................................................... 50

4.3 EMG Classification Algorithm for Muscle Contraction Identification .... 57 4.4 Experiment Setup ...................................................................................... 58

4.4.1 Algorithm for data input and output. ............................................. 58 4.5 Summary ................................................................................................... 63

CHAPTER FIVE : CONCLUSION AND RECOMMENDATION ...................... 64 5.1 Conclusion ................................................................................................ 64 5.2 Recommendation....................................................................................... 65

BIBLIOGRAPHY ...................................................................................................... 67

APPENDIX I: PIC PROGRAMMIMG ................................................................... 69

x

LIST OF FIGURES

Figure No Page No

1.1 Research Methodology workflow 6

2.1 Typical ECG recording illustrating the beating of the heart 9

2.2 Electrodes used to record ERG (Creel, 2008) 11

2.3 (a) Electroretinography (ERG) test and (b) Basic waveform of ERG 11

2.4 EMG block diagram used in the experiment 16

2.5 Scenery testing of the research 16

2.6 Electrode placement in the experiment 17

2.7 A prototype cricket car 18

2.8 The neck of AIBO controlled by the movement of human wrist 18

2.9 Rehabilitation robot configurations 19

2.10 Transmitter and receiver block diagram of the research. 19

3.1 Block diagram of the Facial EMG system to control the small mobile robot 23

3.2 Flowchart of the EMG system implementation 26

3.3 Position of the facial muscles selection for electrodes placement 28

3.4 Muscle contraction: (a) Right cheek muscle contraction (b) Left cheek muscle

contraction (c) Both cheek muscle contraction (d) Forehead muscle

contraction 29

3.5 Electrodes (a) Surface Electrode (b) Needle Electrode 30

3.6 Signal conditioning process 31

3.7 AgCl electrodes 32

3.8 Electrodes position 32

3.9 Schematic for INA 128P connection 34

xi

3.10 Constructed Pre-Amplifier INA128P connection 34

3.11 50Hz High pass filter with unity gain 36

3.12 500Hz Low-pass with 165 gain 37

3.13 Constructed High Pass and Low Pass filter 37

3.14 Precision of full-wave rectifier circuit 39

3.15 Constructed Full wave rectifier 39

3.16 Complete schematic circuit of the Facial EMG interface 40

3.17 Complete EMG circuit simulate using MultiSim 41

3.18 PIC16F877A starter kit with PIC16F877A processor and LCD 42

3.19 PIC16F877A pin configuration for EMG system 43

3.20 PIC16F877A pin configuration located at small mobile robot 44

3.21 Wiring connection between PIC microcontroller starter kit and XBee 46

3.22 Schematic circuit for forward reverse DC motor 48

3.23 (a) Completed three channels EMG circuitry (b) Completed small mobile

robot system 49

3.24 EMG signal captured 49

4.1 Complete system of three channels EMG circuitry, ADC and Transmitter 50

4.2 The oscilloscope show the noise value in EMG system 51

4.3 Waveform observed from Right Zygomaticus major muscle signal has been

passed through band filter 52

4.4 Waveform observed from Left Zygomaticus major muscle signal has been

passed through band filter 52

4.5 Waveform observed from Frontalis muscle signal has been passed through band

filter 53

xii

4.6 Waveform observed from Right Zygomaticus major muscle signal has been

passed through precision full-wave rectifier 54

4.7 Waveform observed from Left Zygomaticus major muscle signal has been

passed through precision full-wave rectifier 54

4.8 Waveform observed from Frontalis muscle signal has been passed through

precision full-wave rectifier 55

4.9 Series of video frames of the small scale mobile robot trajectory by contraction

of the facial muscles. 63

xiii

LIST OF TABLES

Table No Page No

2.1 Common frequencies found in EEG recordings and their associated

conditions 10

2.2 Facial exercises for stroke patient 12

2.3 Summary of the literature of researches and projects done 20

3.1 Facial muscles and location 27

3.2 List of muscle of facial and its actions 28

4.1 Voltage output after band filter with gain 8,415 when the muscle flex 53

4.2 Voltage output after full wave rectifier when the muscles flex 55

4.3 Benchmarking data for algorithm development 56

4.4 Relationship between facial movements, muscle contraction and small

mobile robot movement 57

xiv

LIST OF SYMBOLS

Hz Hertz

k kiloOhm

mV miliVolt

xv

LIST OF ABBREVIATIONS

ADC Analogue to Digital Converter

CMRR Common Mode Rejection Rate

DC Direct current

ECG Electrocardiography

EEG Electroencephalography

EMG Electromyography

EOG Electrooculography

ERG Electroretinography

LCD Liquid Crystal Display

1

CHAPTER ONE

INTRODUCTION

1.1 Background

Stroke occurs when a blood vessel that supplies blood to the brain either is blocked or

bursts. The damage occurred decreases the amount of blood flow and oxygen to the

brain and leads to brain cell death. The symptoms depend on what area of the brain was

affected. Some patients experience difficulty moving their facial muscles, making facial

expressions, speech and eating difficulty after a stroke. Specific exercises can help

regain control and strength in these muscles. In this research work, an alternative

method that is using a small mobile robot as an excitation agent is proposed in order to

have a lighter facial exercise. In addition, it can also be used as an enjoyable motivator.

The method identifies patterns in the electromyography (EMG) signals from the facial

muscles, and commands the small mobile robot to perform certain tasks.

The electromyography (EMG) study is the study of muscle function through

analysis of electrical potential that emanates from the muscle itself and it began as early

as 17th century (Norali, Som, & Kangar-arau, 2009). An argentum chloride surface

electrode (AgCl) is placed on the skin above a superficial muscle that receives electrical

signals emanating from several muscle fibers, which are associated with different motor

units. These electrical signals (EMG) provide an effective means of monitoring muscle

activity. EMG signal obtained by electrode is relatively small with amplitude ranges up

to 10 mV or ±5 mV peak-to-peak (Ibrahim et al., 2008)(Raez, Hussain, & Mohd-Yasin,

2006). Therefore, it is not easily observed. In order to solve the problem, the EMG

2

signals need to be amplified by specifically designed signal conditioning circuits, so that

the signal can be useful and observed.

The EMG signals are picked up by the electrodes and then amplified. Generally,

more than one amplification stages are needed in order to eliminate the low and high

frequency noises or other factors that can affect the signal quality. The frequency of the

signal is between 0 to 500Hz but the strongest amplitudes are to be found in the range of

50 to 150 Hz band (Mei, Ying, & Zheng, 2009). The EMG signals are dependent on

many factors since the signals are susceptible to noise interference such as hum, signal

acquisition such as clipping and baseline drift, skin artifacts, processing errors, and

interpretation problems (Siriprayoonsak, 2005). During the measurement, if there is no

contraction of the muscle, there should not be any electrical potential, but when the

muscle is contracted, the action potentials appears and it depends on the strength of

contraction that determines the level of voltage captured.

Its amplitude can also be affected by parameters such as the type of electrode,

electrode distance placement and electrode orientation. Furthermore, skin preparation,

skin and electrode gel temperature, subcutaneous fat between the electrode and muscle

may also contribute to the amplitude of the signal capture.

1.2 Problem statement

Most of the stroke patients demonstrate lack of enthusiasm in having to exercise for the

stroke recovery process, especially the one that affects the facial muscle. Facial muscle

therapy through regular exercise of the facial muscles had been proven to help in muscle

recovery (May, M., & Schaitkin, 2000) . Hence, an alternative method that is using a

mobile robot as an excitation agent is proposed in this work in order to have a lighter

3

facial exercise. Apart from that, it can also be used as an enjoyable motivator. The

method identifies the patterns in the electromyogram (EMG) signals from the facial

muscles, thus command the mobile robot to perform certain tasks.

1.3 Research objectives

The main objective of this research is to develop a method to control the small mobile

robot by using the facial muscles as the input of the system. To achieve the main

objective, it is necessary:

i. To develop analog signal conditioning circuit that is able to capture and

process EMG signal from the facial muscles to become a useful signal.

ii. To analyze the EMG signal generated from difference facial expressions in

order to develop the algorithm to control robotic system.

iii. To develop a small mobile robotic platform for rehabilitation of facial

muscles.

1.4 Research Methodology

The research is about using facial expression that is governed by a set of face muscles to

control the movement of a small mobile robot. In order to achieve the objectives of the

research, the following steps are taken and the research workflow is shown in Fig 1.1.

i. Review of relevant technical and scientific literatures for available

techniques and methods to control the small mobile robot.

ii. Measurement of EMG signals. It can be done by putting an argentum

chloride surface electrode (AgCl) to the exact selected facial muscles by

using accurate method.

4

iii. Development of analogue circuitry hardware for the signal amplifier and

processing.

iv. Programming of the transceiver between the processing circuitry signal and

small mobile robot.

v. Programming the algorithm to control the small mobile robot movements.

vi. Software and hardware integration.

vii. Implementation and evaluation of the system.

5

Bio-potential signal

capture methods

Method for facial

muscle Rehabilitation

Research and projects

done

Data analysis

EMG

signal

capture

Constructed a set of

three channel analogue

EMG signal

ADC

Fed into PIC

Programming

Data

transfer/received

using XBee

Troubleshooting the

circuit

Report

Troubleshooting the

programming and

connection

Start

Literature Review

Yes

No

Yes

No

A

B

6

Fig 1.1: Research Methodology workflow

1.5 Research Scope

The scope of this research is to use facial expression that is governed by a set of face

muscles to control the movement of the mobile robot. In order to do that, a set of

electrodes are used to measure muscle activities during a particular facial expression.

The data recorded is then processed by an algorithm to control the mobile robot

movement. In this research, the mobile robot’s movements focus on the left, right,

forward and backward movements.

Constructed small

scale mobile robot

Programming the

algorithm

Small mobile robot

move accordingly

Motor Run

Troubleshooting the

programming and

connection

Finish

Report

Yes

No

A

B

7

1.6 Thesis organization

The thesis outline is:

i. Chapter 2: This chapter discusses literature review of EMG signal generated

by facial muscles, as well as hardware and software used in developing the

EMG circuitry and small mobile robot.

ii. Chapter 3: This chapter discusses the development of signal acquisition

circuit and small scale mobile robot.

iii. Chapter 4: This chapter discusses the implementation of EMG system and

the discussion of the results obtained.

iv. Chapter 5: This chapter summarizes all the results obtained in the previous

chapters. Consequently, new development and improvements are suggested

for further study.

8

CHAPTER TWO

LITERATURE REVIEW

2.1 Introduction

Today, the electrical potential of muscle can be measured by using specific tools. All of

these are begun by capturing the electric potential measured between the point in living

cells, tissue and organisms which is called bio-potential. There are many bio-potentials

commonly applied in medical and engineering such as electromyography(EMG),

electrocardiography (ECG), electroencephalography (EEG) electrooculography (EOG)

and electroretinography (ERG).

2.1.1 Electrocardiography (ECG)

Electrocardiography (ECG) is a graphic recording of electrical activity generated by the

heart (Tamil, Kamarudin, Salleh, & Tamil, 2008) shown in Fig 2.1. As the heart pumps

blood, different chambers of cardiac muscle are activated in a special order. The ECG

waveform represents the rhythmical depolarization of each chamber of the heart. The

ECG can be used to determine whether the heart is functioning properly or if there any

problem exists.