Exploring the Benefits ofFingernail Displays

Raphael WimmerUniversity of Regensburg93040 Regensburg, Germanyraphael.wimmer@ur.de

Florian EchtlerUniversity of Regensburg93040 Regensburg, Germanyflorian.echtler@ur.de

Figure 1: Fingernail displays - artificial fingernails withembedded displays - may help mitigating the fat finger problemor provide always-available, unobtrusive notifications to theuser.

Copyright is held by the author/owner(s).CHI’13, April 27 – May 2, 2013, Paris, France.ACM 978-1-XXXX-XXXX-X/XX/XX.

AbstractFingers are an important interface both to the physicaland the digital world. We propose research on artificialfingernails which contain tiny displays and sensors. Thesefingernail displays greatly supplement other input andoutput channels, offering novel interaction possibilities.We present three contributions: (1) the general conceptand use cases for fingernail displays, (2) a technique forcapturing touch events at the fingernails and interactionmethods supported by this technique, and (3) an overviewof relevant research questions.

Author Keywordsfingernail, displays, touch input, wearable computing

ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.

IntroductionIn the past few years, touch input has risen to publicprominence - due to technological advances and the riseof smartphones with capacitive touchscreens. Whileresearch on touch interaction has concentrated on thesurface that is being touched, it may be worthwhile toconsider information displayed directly on the fingertips.We propose research on fingernail displays -

fingernail-sized, slightly curved displays that are attachedon top of a person’s fingernails and are flush with the nailgrooves. Fingernail displays offer an unobtrusive,always-available visual output channel on the handwithout interfering with the hand’s main objective,dexterous manipulation. They can be used as statusindicators, magic lenses, or miniature GUIs. The sizedepends on the size of the wearer’s fingernails. We didnot yet find a reliable source for common fingernail sizesand shapes. However, as a first estimate, displays shouldbe between 10 mm x 10 mm and 15 mm x 15 mm.OLED, e-ink and liquid crystal displays are suitable displaytechnologies with significantly different powerconsumption, update rate, color depth, and flexibility.Adding tiny optical sensors between display and fingernailallows for eyes-free input with one or two fingers.In the following we review related work, explain how aninput channel for fingernail displays can be implemented,discuss possible advantages and limitations of fingernaildisplays, and present three exemplary use cases. Finally,we document research questions, current state, and futureplans for this ongoing research project.

Related WorkTo our knowledge, fingernail displays have not beenscientifically investigated so far.In the the science-fiction movie ”Total Recall” (1990)1, areceptionist changes the color of her fingernails bytouching them with a small wand. In the science fictionmovie ”The Fifth Element” (1997)2, another receptionistdoes also recolor her fingernails but uses a small boxinstead, into which she inserts her fingers.For a design competition sponsored by watch makerTimex in 2004, designers Merana, Schubert, and Takacssubmitted concept renderings of a digital watch display to

1http://www.imdb.com/title/tt01008022http://www.imdb.com/title/tt0119116

be worn on the user’s thumbnail [4]. The watch iscontrolled by touching a metallic area at the rim of thefingernail.With Skinput, Harrison et al. [2] present an inputtechnique where the user taps the palm of one hand witha finger of the other hand. The resulting sound waves arecaptured by microphones attached to the user’s arm.From the sound information, the location of the tap iscomputed. The Skinput prototype employs a GUI that isprojected onto the user’s palm.Mascaro et al. [3] found that applying force to the fingerpad changes the coloration of the tissue under thefingernail, because blood gets pressed out of the smallblood vessels in the finger. They showed that differentforce levels and force directions result in unique colorationpatterns. From these coloration patterns, the direction offorces on the finger pad can be estimated. Mascaro et al.implemented a fingernail-mounted sensor employing smallLEDs and photodiodes for measuring the coloration. Theirresearch focused on measuring grasp forces for use inrobotics. It inspired our approach to input on fingernailsand our hardware implementation.

Input on the FingernailWhile information displayed on the fingernail offers severaladvantages, as discussed before, an additional inputchannel on the fingernail might greatly improve theusefulness of fingernail displays. Such direct input on thefingernail would augment systems which track a finger’sposition [1, 5].One option for implementing input might be atouch-sensitive layer (e.g. capacitive, resistive) above thefingernail display. This could allow touch interaction andswiping gestures on the display. As the display is very tiny,the available input area is very limited. Additionally, mostof the fingernail display would be occluded by the fingertouching it. This makes continuous interaction tedious.

While it is possible to attach pressure sensors tofingertips3, such systems inevitably reduce tactilesensitivity of the finger. We propose an input techniquethat employs a sensor which is attached to the fingernailbut senses pressure on the fingerpad. As demonstrated byMascaro et al [3], both perpendicular and shear forcesapplied to the fingerpad can reliably be estimated fromchanges in the coloration of the tissue under thefingernail. Whereas Mascaro et al. utilized this effect formeasuring grasp forces, we plan to detect touch,dragging, and rolling events and use these for explicit andimplicit interaction. To this end we are working on anoptical sensor which is located between display andfingernail and captures changes in nail bed coloration. Wesee two potential input modes:

11:00team meeting

next:

Figure 2: By slightly draggingthe pad of the thumb with theindex finger in differentdirections, users can input text orselect menu items on thefingernail display.

Finger-on-Surface Input Using the approach presentedby Mascaro et al, we can detect when an instrumentedfinger touches a surface with a certain amount of force.This event by itself already allows determining whetherthe user is doing manual work. We assume that it mayalso be possible in some cases to find out what the user isdoing by comparing the time series of touch events topre-recoreded training data for known tasks.Touch-sensitive surfaces allow for further interactionmethods: By correlating the timestamps of events fromthe fingernail with touch events from the surface, it ispossible to identify which finger is used for operating atouchscreen, allowing different digital tools to be bound todifferent fingers. As Mascaro’s approach not only detectsforces perpendicular to the fingerpad but also shear forces,fingernail sensors can be used to make ordinarytouchscreens shear-sensitive. While touching the surfacewith an instrumented finger, the user can initiate differentactions by dragging the finger up/down/left/right. For

3http://www.pressureprofile.com/products-fingertps

example, the user might select an object on the touchscreen by applying a short downward shear impulse. Thisimpulse would be detected by the fingernail sensor andprovided to the touch screen as input event.

Finger-on-Finger Input If two or more fingers areaugmented with fingernail sensors, pressing the fingertipstogether generates simultaneous touch events from allinvolved fingers. Associating each combination of fingerswith a certain action or input symbol allows forunobtrusive, eyes-free mobile input. Measuring shearforces can also be used for turning the thumbpad into adirectional input device. Thereby the index finger touchesthe thumbpad and pulls or pushes the thumb’s tissueup/down/left/right.Some of the proposed approaches have previously beenimplemented using markers on the hands or electricalcontacts on the fingerpads. However, fingernail sensors donot require an external tracking infrastructure and do notobstruct tactile senses.

Potential AdvantagesWhile we are still at the beginning of our research, we seeseveral obvious and potential advantages of fingernaildisplays that prompted us to investigate further. We planto verify our assumptions in future research.

Comfort Fingernails and toenails are the onlycompletely rigid body parts not covered by skin. Directlyaffixing a display to any other body part might make theuser feel uncomfortable or irritate the skin. Fingernaildisplays, in contrast, do not impede tactile senses and doprobably not annoy or hinder the wearer if they are flatand flush with the fingernails edges.

Visibility Moving the fingernails into the focus of ourfield of view is usually very easy and fast. During manymanual tasks, - for example when working at a desk - the

hands are already at least within the peripheral field ofview. Usually, both thumbnails are directed towards oureyes while the other fingernails are directed away from us.We assume that in many cases information displayed onthe thumbnail is not visible to other people even in closeproximity. This makes thumbnails well suited fordisplaying notifications or context information. Theoutward-facing fingernails might be used e.g. fordisplaying status information to other people, or asoptional displays or input buttons.

Effortless Input Putting input and output capabilitiesat the fingertips allows for quick input and intuitive touchinteraction with digital data. As we often look at ourhands while doing manual work, putting displays there -and offering a rapid input method there - may reducedistraction. When using touch input, our fingers act aspointers for manipulating digital data. A fingernail displaycan support these interaction e.g. by making the fingeract as a magical lens or a virtual clipboard.We discuss applications that highlight these potentialadvantages in the next section.

Challenges and LimitationsWe also see a few unique challenges that need to be takeninto account when discussing feasibility and usability offingernail displays:

Anatomy The size and shape of fingernails variesstrongly between persons. Some people have extremelytiny fingernails that would not be suitable for attaching adisplay. Therefore, we need to find ways to buildcustom-shaped fingernail displays and accept that therestill will be users which can not use these.As fingernails slowly grow, the attached display will slowlymove from its ideal position. When the nail plate isair-tightly covered for a long time, it softens as keratincan not solidify without oxygen contact. This is also a

common issue with artificial fingernails. We need toaddress these issues, e.g. by making the display easilyremovable and re-attachable.

Social Acceptance It is not yet clear to us whetherpeople would actually wear fingernail displays. There arecertainly groups with higher affinity to such attachmentsthan others. On the other hand, fingernail displays mightalso function as dynamic jewelry, thereby increasingacceptance among less tech-savvy users.

Miniaturization It is not yet clear, whether and how itis possible to manufacture fingernail displays that are thinenough to not impede dexterous manipulation, especiallyin confined spaces like trousers pockets. At the sametime, the displays should be light enough to not causeadditional exertion and durable enough that they can beworn during everyday tasks.

Power and Data Transfer The smaller a digital artifactgets, the less power it can store, limiting its wireless use.While we are mainly interested in fingernail displays as auser interface, we feel that we also need to find out howfingernail displays can be powered, and how data can betransmitted between fingernail and other computers.

Specific research questions that arise from theseassumptions are discussed in the final section.

Usage ScenariosGiven the interesting properties of fingernail displays, wesee several promising applications of which we shortlydescribe three:

Always-available User Interfaces Having both inputand output capabilities directly at the fingertips allows forimplementing always-available graphical user interfaces.Such systems might present notifications about new

messages, allow control of a media player, offer littlegames, and allow people to change their availability status(Figure 2). As the user can glance at and interact withfingernail displays very quickly, we expect them to reducethe amount of time spent with such non-primary tasks.We are especially interested in the question how fingernaildisplays compare to - and interact with - wrist-worndisplays such as the Sony SmartWatch4 or the Pebble5

watch.

Figure 3: Fingernail displays maybe used for implementinguser-centered clipboards, allowingfor intuitive information transferbetween devices.

Ubiquitous Clipboard While a growing multitude ofsmart objects surround us, intuitively transferring databetween these objects is still a challenge. Fingernaildisplays may help here, serving as a ubiquitous clipboard.When the user wants to transfer a file from their(touch-sensitive) computer screen to a wall-mounteddisplay in an adjacent room, they just touch the file onthe screen and drag the finger slightly upwards, causingshear forces on the fingerpad. The file gets transferredand is shown as a thumbnail preview on the fingernaildisplay. Then the user walks to the target display andpastes the file by performing a similar gesture on it(Figure 3). Users can copy multiple files to their fingerand select the one to be pasted using one of thepreviously described input methods.

Magic Lenses Fingernail displays can show anaugmented image of the area below the finger. This canbe used for overlaying visual information onto a touchscreen or for showing information about an object beingtouched or grasped. The information to be displayedcould be gathered by additional sensors or wirelessinterfaces of the objects. For example, touching a batterypowered device might show the remaining battery life onthe fingernail display, while touching a map might reveal

4http://en.wikipedia.org/wiki/Sony_SmartWatch5http://en.wikipedia.org/wiki/Pebble_(watch)

information about the location under the fingertip. Wealso suspect that a fingernail display that shows the screencontents which are occluded by the finger might mitigatethe fat finger effect that reduces input accuracy on touchscreens (Figure 1).

Research QuestionsWe suspect fingernail displays might be a useful additionto existing user interfaces. There are a few questions,however, that should be investigated to determine whetherfingernail displays have more than just novelty value.

• Which kinds of applications benefit from fingernaildisplays?

• Which fingers are most suited for augmentationwith displays?

• How fast is interaction compared to other UIs suchas mobile phones?

• Can fingernail displays mitigate the fat fingerproblem?

• How can fingernail displays augment other displays,e.g. wrist-worn displays or mobile phones?

• What update rate do displays and sensors need fordifferent applications?

• What is necessary for users to like and wearfingernail displays?

• In which cases are fingernail displays annoying orhindering?

• Which technologies and implementations work bestfor displays and sensors?

• How important is an exact fit of the displays?• How can power and data be transferred wirelessly?

To answer these questions, we are implementingprototypes, measuring key properties, and conductingsurveys and user studies.

ImplementationWe have built a first prototype of a fingernail display(Figure 4) using an off-the-shelf 96x64 pixels (121 dpi)OLED display module (Adafruit product ID 326).

Figure 4: Our first prototype.

The display is connected to a Pinguino PIC32MXprototyping board via SPI. The Pinguino is connected to ahost computer via a serial-over-USB connection. Imagesare transferred from host computer to OLED display in anuncompressed RGB565 format at an (unoptimized)update rate of 12 fps. This prototype does not yet havesensors installed and is quite clunky. However, it helps usin thinking about and discussing applications of fingernaildisplays. We plan to implement three other prototypesthat help us answering our research questions:a) sensors-only - a prototype without display but fivephototransistors. It allows us to refine and test thefingertip input approach without having to bother withdisplay integration.b) miniaturized - a prototype with sensors and display.We will try to get it as small as a thumbnnail (15 mm x15 mm). It will be powered and controlled via a cable.This prototype will be used for most user studies.c) wireless - a proof-of-concept wireless implementation.With it we plan to demonstrate that fingernail displayscan work without cables. We plan to use an e-ink display.Power and data for the display will be transmittedwirelessly.

While none of the prototypes will be a completeimplementation of our proposed concept, we hope to learnmore about which tradeoffs between usability andtechnical feasibility have to be made.

Future WorkUsing our first prototype, we plan to conduct initialexperiments with respect to interaction modalities. In

parallel, we will be developing the other prototypes inorder to a) have higher-fidelity hardware for refinedstudies and b) test the general feasibility of the moreadvanced concepts outlined above.More information can be found athttps://wiki.mi.ur.de/fingernail. We are veryinterested in comments, questions, and cooperations.

References[1] Harrison, C., and Hudson, S. E. Abracadabra:

wireless, high-precision, and unpowered finger inputfor very small mobile devices. In Proceedings of the22nd annual ACM symposium on User interfacesoftware and technology, UIST ’09, ACM (New York,NY, USA, 2009), 121–124.

[2] Harrison, C., Tan, D., and Morris, D. Skinput:appropriating the body as an input surface. InProceedings of the SIGCHI Conference on HumanFactors in Computing Systems, CHI ’10, ACM (NewYork, NY, USA, 2010), 453–462.

[3] Mascaro, S., Chang, K., and Asada, H.Photo-plethysmograph nail sensors: for measuringfinger forces without haptic obstruction: modeling andexperimentation. In Robotics and Automation, 1999.Proceedings. 1999 IEEE International Conference on,vol. 2, IEEE (1999), 962–967.

[4] Merana, N., Schubert, S., and Takacs, D. TX54watch concept. http:

//www.core77.com/timex/winners/wearable.asp,2004.

[5] Yang, X.-D., Grossman, T., Wigdor, D., andFitzmaurice, G. Magic finger: always-available inputthrough finger instrumentation. In Proceedings of the25th annual ACM symposium on User interfacesoftware and technology, UIST ’12, ACM (New York,NY, USA, 2012), 147–156.