University of Arkansas – CSCE DepartmentCSCE 4613 Artificial Intelligence –Final Report – Fall 2009

Smart Devices in a Virtual World

Akihiro Eguchi <aeguchi@uark.edu>Dexter Josh Freeman II <djfreem@uark.edu>

Luis Avila Revollo <lavilare@uark.edu>

Abstract

As technology advances, the reality of smart devices populating our environment is continually approaching us. We must figure out how to implement these smart devices and their interfaces and through a virtual world such as Second Life, we can get a head start on the implementation gaining valuable insight on its capabilities and limits. We will create various smart devices focusing our efforts on those which are likely to be found in medical facilities. We will enable communication between the user and the smart devices both directly and through the use of one or more soft controllers. After our implementation is finished we will be able to test our inventions usefulness and better determine what they are capable of and what the limitations may be, so that the design may be improved or new implementation ideas may spawn from the last.

1. Introduction

1.1 Problem

Technological advances are creating a new environment in which we will be surrounded by ever more powerful smart devices. The new technological advances are likely to change the world we live in. How to correctly implement these new devices is a major cause of concern. As new devices become more powerful they also become increasingly complex. The correct design approach is a key in creating friendly easy to use powerful devices.

What makes smart devices different from existing devices is that the ubiquitous feature of them. The word ubiquitous is a little bit ambiguous term, but we see it as a thing which requires more communication between devices, and devices and people. In other word, there always exists some kind of communication behind the device itself in order it to be called a smart device. For example, a clock itself is just a device to tell you what the time is. However, if the clock sends a command to a radio so that the radio turns on at a certain time to wake you up, it might be called one kind of simple smart devices. The radio might be connected to the internet and gathers the program listing while you are sleeping. Also, for more complex devices, it is very important to make the sophisticated devices easy to control by anybody. Creating a unified standard for the controlling input for the devices can be the one way. This makes it possible to control all the devices from only one controller which the user is familiar with, thus not requiring the user to learn the way to control a new device every time. In order to realize the feature, it can be a good idea to put identity for each device to identify what device the user wants to control and the function for sending back the list of command to the controller. There are enormous ways to

make the devices very smart by establishing the communication between those devices. However, more importantly, we cannot forget the fact that even if the device has many functions, it will be useless if the user cannot establish the fully communication with them. Therefore, it is very important to focus on not only the communication between devices and devices but also the communication between the devices and people.

1.2 Objective

The objective of our project is to gain understanding in the design of smart devices. We will implement several smart devices to be accessed directly or through the use of a soft controller.

1.3 Context

The Second Life world is a 3D virtual world in which various real life experiments can be tested. This environment gives us a pervasive world in which our particular project can be easily implemented and in which we can measure how successful our creations are based on their capabilities and limitations. Our smart devices are particularly easy to implement because of the functionalities given by the SL environment and by the various LSL functions we can use in our scripts to interact with the virtual world.

The Everything is Alive (EiA) concept is truly implemented in the SL virtual world as our scripted creations start to populate it. The smart devices we offer will come to life in a new environment as we give them interaction capabilities both through direct contact with avatars, indirect contact to avatars through soft controllers and through network capabilities which interconnect the devices amongst themselves giving them a new perception of their surrounding environment.

1.4 Potential Impact

The future is very likely going to offer us an environment populated of ever more powerful smart devices. Although the common person or ourselves may not be able to understand the inner workings of all of these, we are still going to need to be able to use them. It is crucial that smart devices are designed appropriately. Smart devices will range from very simple such as a RFID print to very complex such as computers capable of learning and improving their own interfaces. This future offers us great possibilities, but perhaps the most important feature of all is how easy to use and intuitive interfaces will become.

2. Related Work

2.1 Key Technologies

The work on Smart Devices relate quite closely with the theme of the class, Artificial Intelligence. To develop smart or intelligent devices would be to develop some means of artificial intelligence. This development would bring about objects that are able to communicate and reason amongst one another. In a world where everything is alive (ubiquitous computing), all devices would be able to see and communicate with one another on a network. Users would also be able to see and operate such devices (if functionality permits) over a network. Until the day comes where everything is alive, this theory can be implemented using virtual worlds. In a virtual world, objects or devices can be created that are smart, i.e. able to identify self, communicate with users, communicate with other objects or devices, etc. Virtual worlds have their limitations, but can still provided for a broad array of implementations. For example, one

particular limitation would be the absence of feeling. The sensory feeling of touch and smell cannot be implemented in a virtual world.

2.2 Related Work

Relevant work on smart devices include a technology known as Powerline Carrier Systems (PCS), which is used to send coded signals along a home’s existing electric wiring to programmable switches, or outlets. A common protocol for this known as X10 is a signaling technique for remotely controlling any device plugged into an electrical power line [1]. In Europe, technology to equip homes with smart devices is centered on the development of the European Installation Bus (EIB is the world’s leading system for “intelligent” electrical installation networking)[2].

Cell phone is a great example of the smart devices with its many kinds of functions that take advantage of its mobility and network connection. A company called Xanboo provides a home monitoring system which makes it possible for user to easily control and monitor their home right from their cell phone [4]. Also, Biene Electronics provides a remote control module for home over GSM channel via SMS message such as metering of gas, water and oil flow and car remote control such as alarm system, heater, and central door [5]. In those ways many devices are connected via a cell phone.

Home Network System (HNS) is also important base for the smart devices. HNS consists of one or more networked appliances connected to a local area network at home. Every appliance has a set of APIs, by which users or external software agents can control the appliance via network. Here, as the means of communication between human and devices, voice recognition and gaze information are useful means for implementing the intuitive interface of the HNS. Kikino of toy company Bandai is an example of commercial product, which recognizes the voice of people and according to the command, it control the device around [6]. Those kinds of environment and systems helps very complex devices to be easily used by anybody.

2.3 Related EiA Projects

Our project on Smart Devices relates to these other projects, e.g.,

Mirror Worlds project – our project develops smart devices that could fit in a mirror world; smart virtual world device models could sit inside mirror worlds and be used to control the real world device.

Ontology project – what makes a smart device smart is the EiA suite of protocols. This information should be accessible from the ontology data structure. Eventually, we can imagine that all devices in an ontology classification system are more or less smart if the information has a importance in the communication between devices or devices and users.

Soft Controller project – our project’s smart devices can be controlled by a soft con-troller with using common format of input. In addition to the soft controller we created, we are cooperated with the soft controller group so that there will be more than one con-troller.

Workflow – smart devices become active players in workflows as one device can com-municate with other devices to realize the tasks and goals of the workflow.

Search Spider – a spider can find smart devices and add them to the ontology.

Chatbots – smart devices could include a chatbot plugin that lets them chat with humans or other chatbots.

3. Architecture

3.1 Requirements or Use Cases

To create smart devices, which has multiple functions, in a medical field, and by implementing common format of input, making it possible to be controlled by one control device.

All devices which we will make have Listen() running all the time so that they accept the command from outside of the objects.

All devices will be stored in the same room, and one controlling device have an ability to control all the functions of each devices.

The controlling device show the menu of functions for each smart devices in English on its screen, and users can choose from the menu and control the devices.

Since we uses same format of input for each devices and each devices control over their variables, there can be more than one controller.

The term, smart devices, is a very broad concepts; thus, we will have create many variety of things on the second life such as multi functional hospital bed, air conditioner, robot for searching devices, infant warmer, x-ray machine, and nursing dummies.

The multi functional hospital bed has several functions which make the patient feel comfortable on the bed. It allows users to change its angle and has a fan which user can turn on or off. Also, the bed has a table which users can pull out, and more importantly, the controller of other smart devices is integrated into the bed so that the patient can access all devices from the bed.

The controller has a text based instructions on its screen, which is implemented by imported texture which consists of all the characters in English. The names of smart devices are displayed on the screen and users are asked to choose. Then, the functions of your chosen devices are displayed on the screen, and you can choose from the list and send the command to the device.

We gave the input format of the smart devices to the soft controller group so that they also can create another controller to control our built devices.

The air conditioner is attached to a wall of the room, and by using controller on the bed, users can turn it on or off; at the same time, users can choose the air either warm or cool. The particles show the different temperature of the air coming from the AC.

The searching robot is able to roam around the clinic in search of other smart devices. The robot has a remote control and is able to leave the users sight as it traverses the clinic independently.

The G.P.S. Control is a handheld device which has scheduling capabilities. It can store the current location of the user as he enters the checkpoints which the robot

will follow. This way the user can create previously fixed paths which the robot will now be able to traverse by itself.

The RFID tag is an identification tag which responds to a RFID light source. It responds by giving its identification information and it is also possible to approximate it’s position as it signals its response to the RFID light emitter.

Infant warmer is a machine which takes care of infant.

It has arms to give infant oxygen and measure his suction.

This machine contains a X-Ray sheet which projects the X-Ray picture on when the X-Ray machine is used.

It also has a drawer and table which can be pulled out.

Portable X-ray machine is X-ray machine which is moveable. Usually, X-ray machine is deferment, but for the patient who cannot move around, there exists such a machine.

This X-ray machine has a screen where the digital picture of the X-ray taken by this machine is displayed.

When the machine the machine is clicked, it moves its arm upward and approaches to the object. Then, the red particle is shown as it prints the X-ray picture on the X-ray sheet in the infant warmer and displays it on the digital screen.

Then, when it is clicked once more, it goes back to the original place and turn the digital screen off.

Nursing Dummy is an exact same scale of human dummy, which is used for the training of nurse in real life. We assume that each body parts as a smart device which has some unique functions. Our dummy is an infant dummy.

The dummy can be opened its body if the avatar clicks it, and it has most internal organs as a real human.

Each internal organs shows its name when it is clicked so that the avatar can learn which organs are which.

This dummy has a function of PaO2 training, which is a training of saving the patient with losing his O2 in his blood by giving him oxygen.

When the avatar types “/5 start PaO2”, it starts with 100% PaO2 and starts losing its value.

His face will look pale as he loses his PaO2, and if it goes under 75%, he dies.

As PaO2 goes under 95%, it shows the message “problem zone”, and as it goes under 85%, it shows the message “dangerous zone”.

His face will look fine as he obtains O2 by oxygen machine of infant warmer, and when it reaches 100%, the training ends.

IV drip stand and the different IV medicine bottles are used for another nurse training simulation.

The training serves the purpose of virtually allowing nurses to gain practice setting the proper infusion bottles.

Training is began by touching the console on the IV drip stand.

A prompt instructs the nurse to set a certain bottle. If completed successfully, a new bottle is prompted, and this continues until training is complete.

Once training is complete, an overall score is produced.

3.2 Architecture or Design Space

In our project, all smart devices are in one sense connected each other because all the devices have a way to be controlled by one controller. Although each device has a very different use and works independently, by implementing a common format of the input, they have a universal feature. It is very beneficial for the patient because they do not have to learn how to use each smart device but can control them through a controller any time. Also, the controller has to have a very user-friendly interface so that the user can control the objects viscerally.

This projects falls into 3 general categories which is the interface part, smart device in patient room part, and other extended smart devices part such as a nursing dummy. Each member of this project works independently, resulting in several set of smart devices which all can be controlled by a controller.

3.3 Tasks

Akihiro has a responsibility for implementing mainly user interface, which include the controller. Akihiro has created smart bed with a controller which can control all the devices in the room, portable X-ray machine with digital screen and function to print the picture on the X-Ray sheet in the infant warmer, infant warmer which has several functions like oxygen machine, and baby dummy which basic internal organs and a function for PaO2 training.

Dexter focuses on creating several smart devices for the patient such as an air conditioner and an IV drip that functions as another training simulation for nurses.

Luis extends the term of smart devices and creates several other smart devices such as robot, controller and GPS scheduler. All devices share the same command format so that users have a familiar intuitive way to access those devices.

3.4 Testing

Basically, we will fit all devices we created in one room and check their functions one by one. Then, by using controller, we will check if each device correctly works through the controller as well.

4. Results and Analysis

4.1 Multifunctional bed and controller

From the menu displayed on the screen, you can choose smart devices which you want to control.

In the right hand side figure, you are changing the angle of the bed through the controller, and in the other, you are controlling fan attached to the desk and pulling out the table from the desk.

The air conditioner unit featured here has a heating and cooling function that can be controlled using the same controller.

4.2 Infant warmer

Infant warmer has a machine to take care of baby. This time, the baby is a nursing dummy.

This has a drawer, table, and X-ray sheet which can be pulled out. Also, it has arms to give patient oxygen and measure the suction.

4.3 X-ray machine

This X-ray machine is a non-deferment X-ray machine, which can be moved around in the hospital. This has a foldable radial canal.

X-ray sheet where the X-ray picture will be printed is set behind the baby, and the X-Ray machine approaches to the baby to take the X-ray.

The red particle represents the X-ray, and the result is printed on the X-ray sheet behind the baby and displayed on the digital screen.

Then, when the avatar clicks the printed X-ray sheet, it is changed to the new sheet.

4.4 Nursing dummy (infant model)

When avatar clicks body of the infant dummy, it opens the body and shows inner organs. As avatar clicks organs, it shows the name of each organ.

PaO2 training function is available for the nurse to learn how to treat the patient with losing his PaO2, which is the level of oxygen in blood. When avatar clicks the face of the dummy, the

training starts. The PaO2 starts with 100% which is the healthiest level, and then every 10 second, it loses the PaO2, and becomes pale. Same to the real world, when it goes under 95%, it warns some problems on the body. As it goes under 85 %, it goes into dangerous zone. Then, once it goes under 75%, the baby dies and the training fails.

Therefore, the nurse is required to turn the oxygen machine of the infant warmer on, so that the patient can obtain oxygen in his blood. Once the oxygen starts coming out from the machine, the PaO2 becomes increasing, and the face becomes getting healthy color. Then, when PaO2 reaches 100%, the training successfully ends.

4.5 IV Drip Stand and Medicine Bottles

In the image below, the module on the IV drip has been touched and the training has begun. It starts by asking the nurse to set a randomly chosen medicine bottle.

If the proper bottle was chosen you go on to the next one.

However, if the wrong one is chosen, it gives you another chance to select the proper one.

At the end of the training, a nurses score is displayed.

4.6 Hand Held Soft Controller and Smart Device Search Robot

The Hand Held Soft Controller and Smart Device Search Robot are shown in the picture. When it is in Standby Mode, the Robot appears blue and sits still until it receives the order to start the search.

The Blue Button on the lower left indicates that the Soft Controller is currently on and ready to send a Search command to the SD Search Robot. We click the Green Button and the search begins.

The SD Search Robot turns red and starts roaming around the building sending messages to test if a Smart Device answers.

The robot turns green as it encounters a smart device represented in the model by a blue striped cube.

Once the search is complete, the robot returns to the original position and reports the search result with the position of the RFID enabled smart devices found.

Once finished the robot returns to standby as is indicated by the blue color.

A new controller has been added to expand the search robot functionality. This controller is similar to a G.P.S. and records and transmits coordinates to the rover in order to facilitate the creation of fixed paths through the clinic.

The controllers are complemented by four onscreen buttons which are semitransparent boxes fixed to the hub. The new black button on the upper left serves the purpose of sending the

message to store the current position in the G.P.S. controller. The list of positions can later be transmitted to the rover so that it can start traversing the clinic.

5. Conclusions

5.1 Summary

In this project, the work we has done could be classified into three general categories.

The first part is creating several different kinds of smart devices which focus on the communication between different devices. One prominent example is the X-ray machine, which extends its neck, approaches to the patient on the infant warmer, and takes the X-ray picture. When the picture is taken, the particles are shown from the head of the X-ray tube, then the picture is shown on both digital screen of the machine and the X-ray sheet in the infant warmer. Most of the codes are written on different prims depending on its function, but all of them interact with invisible string command for users (like high-frequency ultrasounds in the real world) and work as if they are working as a whole.

The second part is the user interface device which makes the communication between devices and people better. We has introduced a common format of the input for all devices we has created. This makes it possible for users to control all the devices around them only by that one controller. In order to make the interface user friendly, we has implemented the text displaying screen so that the user can control the device with intuition. We has reduced the button as much as possible so that user can use the controller very easily. Similar to the real world example of voice recognizer system, we also make it possible to control the devices over the chat box (PaO2 training system). Even if the devices has many fancy functions, it can be useless when user does not know how to use it; therefore, we can hardly call them as a smart devices. This kinds of effort to improve the communication between devices and user fixes those problems and makes semi-smart devices real smart devices.

For the third part, we extended the meaning of smart devices for the things like nursing dummy, which requires both communication between devices, and devices and people. PaO2 training for nurse of the baby dummy is a good example, which requires to keep track with the PaO2 levels, and depending on the level, the dummy changes its color of the face; at the same time, in order to succeed in the training user has to use the oxygen providing function of infant warmer. In this case, the communication is very important, but more importantly, it can be pointed out that people learn from the devices. In the other word, the devices makes people smarter through the communication.

5.2 Future Work

Future work is to cooperate more with other project groups such as ontology to make it possible to query the ontology for more information and mirror world by placing the devices in the same location. Also, as a corroboration with soft controller group, we could make the controller as much as small as cell phone so that each patient can hold on their hands and control them anytime. Also, the controller can also receive message from the devices so that users can see the command for the functions or see if it really works. Also, it would be a good idea to put more training menu on the nursing dummy.

Bios

Akihiro Eguchi – Eguchi is a sophomore Computer Science major in Computer Science and Psychology Department at the University of Arkansas. He completed courses on Data structure and Algorithm. He is responsible for creating user interface, soft controller, which can control all smart devices and several other smart devices such as infant warmer, X-ray machine, and baby dummy.

Dexter Josh Freeman II – Freeman is a senior Computer Science major in the Computer Science and Computer Engineering Department at the University of Arkansas, with a minor in Mathematics. He has completed courses on Programming Languages and Formal Languages. He is responsible for creating different smart devices with functionalities that can be controlled via a controller or other smart device.

Luis Avila – Avila is a senior Computer Science major in the Computer Science and Computer Engineering Department at the University of Arkansas. He has completed courses on Software Engineering and Computer Graphics. He is responsible of creating a smart device capable of roaming through a user specified area and finding smart devices in the vicinity.

Dr. Craig Thompson, Mentor – Dr. Thompson is a professor in the Computer Science and Computer Engineering Department. He leads the Everything is Alive research project that is currently focusing on how to simulate pervasive computing using 3D virtual worlds. See http://vw.ddns.uark.edu.

Dr. Nan Smith-Blair – Dr. Smith-Blair is an associate professor of Nursing and Interim Director. Her research focuses on the physiologic and biochemical effects of emphysema on the development of diaphragm fatigue and therapeutic interventions to improve diaphragm contractility during diaphragm fatigue. She showed our group the nursing dummy and infant warmer which we built in the virtual world.

References

[1] What is smart home or building?, http://searchcio-midmarket.techtarget.com/sDefinition/0,,sid183_gci540859,00.html

[2] European Installation Bus, http://www.knowledgerush.com/kr/encyclopedia/European_Installation_Bus/

[3] 2009-Fall--EiA-Team-Projects.doc

[4] Control (and Monitor) Your Home via Your Cellphone, http://tech.yahoo.com/blogs/raskin/225

[5] Remote Control and Monitoring, http://www.bieneelectronics.com/products/remotecontrol.htm

[6] Naohiro Yuasa, “Operating networked appliances using gaze information and voice recognition” , Nara Institute of Science and Technology, March 2008.