USENIX Association 2015 Symposium on Usable Privacy and Security89Usability of Augmented Reality for Revealing SecretMessages to Users but Not Their DevicesSarah J. AndrabiUNC Chapel Hillsandrabi@cs.unc.eduMichael K. ReiterUNC Chapel Hillreiter@cs.unc.eduCynthia SturtonUNC Chapel Hillcsturton@cs.unc.eduABSTRACTWeevaluatethepossibilityof ahumanreceivingasecretmessagewhiletrustingnodevicewiththecontentsofthatmessage, byusingvisual cryptography(VC) implementedwithaugmented-realitydisplays (ARDs). Inapilot userstudyusingGoogleGlassandanimprovedstudyusingtheEpsonMoverio, usersweresuccessfullyabletodecodeVCmessages usingARDs. Inparticular, 26out of 30partic-ipants inthe EpsonMoveriostudydecodednumbers andletters with 100% accuracy. Our studies also tested assump-tionsmadeinpreviousVCresearchaboutusersabilitiestodetect active modication of a ciphertext. While a majorityof the participants could identify that the images were mod-ied, fewer participants could detect all of the modicationsintheciphertextorthedecodedplaintext.1. INTRODUCTIONInthe face of massive surveillance bynation-state-levelactors(e.g.,[22,30]),includingtheimplantationofsurveil-lance functionality in commodity device rmware (e.g., [17]),itappearsthattrulyprivateelectroniccommunicationisaschallenging today as ever. At the core of this problem are thecomplexcryptographicoperationsthatmustbeperformedtoencryptanddecryptmessages: Theseoperationsaretoocomplex for humans to performthemselves, andso theymust use devices to do soperhaps devices that might havealreadyhadtheir privacycompromised. Toachievetrulyprivate communication, then, it wouldseemnecessarytoeliminatedevicesfromthetrustedcomputingbase(TCB),i.e., tohavehumansthemselvesperformthecryptographicoperations.Historyis rife withexamples of private communicationperformedbyhumanswithoutdevices, usuallybecauseca-pabledeviceswerenotyetavailable. Whilemostof theseciphersaretriviallybreakabletoday,anotableexceptionisthe one-time pad, which is both perfectly private and has en-cryptionanddecryptionfunctionsinvolvingonlyexclusive-oroperations[16]. One-timepadswereamethodofchoiceCopyright is held by the author/owner. Permission to make digital or hardcopies of all or part of this work for personal or classroom use is grantedwithout fee.Symposium on Usable Privacy and Security (SOUPS) 2015, July 2224,2015, Ottawa, Canada.for spies inWorldWar II for protecting communicationswiththeir home countries, performingthe exclusive-or ofmessages manuallyagainst keys encodedon, e.g., apaperpadthattheybroughtwiththem(e.g.,[29]). Thisideawasmodernized in 1994 with the invention of visual cryptography(VC)[26], inwhichkeysareencodedonvisual transparen-cies andmanual exclusive-or operations arereplacedwithoverlayingthesetransparenciesonsuitablyencodedcipher-textstorevealtheircontents.The practical diculties associatedwithone-time padsarewell known. Mostnotableisthattheyrequireaquan-tityofkeymaterial (intheaboveproposals, onpapertapeortransparencies)equal tothetotal sizeofall messagestobetransmitted. Inthispaperweexplorethefeasibilityofmakingtheone-timepad, andspecicallyitsimplementa-tion in VC, practical using augmentedrealityhead-mounteddisplays (ARDs) such as Google Glass, Epson Moverio, SonySmartEyeGlass, orMicrosoftHoloLens, whilestillensuringthatnosingledeviceistrustedwiththecontentsofames-sage to the user. We evaluate the ecacy of storing and ren-deringone-timepads(encodedforuseinVC)inanARD,whichtheuservisuallyalignsoveranencodedmessageren-deredonanother displaytoreveal its contents. If work-able, this design should largely eliminate the problems asso-ciated with storage of one-time pads and in fact can supporttheir generation pseudorandomly (e.g., from a secret key persender)withlittlepracticallossofsecurity.In this paper we show that VC via ARDs is in fact possibleandreasonablyusable, rstusingasmall pilotstudywithGoogleGlassandthenaformaluserstudyusingtheEpsonMoverio. Forconveyingmessagesweusedindividuallettersandnumbersencodedasimages. Ofthe30participants,26intheformal userstudydecodednumbersandletterswith100% accuracy. The other four participants decoded at least80% of the alphanumeric plaintexts accurately. The mediantimetakenbyparticipants toidentifythedecodedlettersandnumberswas8.88seconds. Asindicatedbyourstudy,however, several challengesneedtobeaddressedincludingthesmall eldof viewof ARDs, headtracking, andimagestabilization.Moreover, since numerous communicationprotocols in-volvingVChave made assumptions about users abilitiesto detect active modication of a ciphertext or the plaintextthatresultsfromit(e.g.,[25,31,15,9,6]),wealsoevaluateusersabilitiestodoso. ThroughtheEpsonMoveriostudy,we assess whether users were able to identify invalid decodedplaintextsresultingfrommaliciousmodicationstotheci-phertext. Alarge majorityof our participants were able1902015 Symposium on Usable Privacy and SecurityUSENIX Associationtodetectthemodications,eventhoughfewerwereabletoidentifyall of thechangesinanimage. Eveninthepres-ence of the leastnumber of modications (depending on thestudy) tothe recoveredplaintext, more than80%of theparticipantsdetectedthepresenceofthemodications.Wealsoconductedanonlineuserstudytoanalyzeusersabilitiestodetectchangesinciphertextsdirectly. Again, alarge majority of participants detected that the images weremodied. Weexploredtheextent of modicationneededinorder for auser todetect it, andbasedonour currentimplementation,smallmodicationsweredetectedbymostof theparticipants. Ourndingsindicatethataugmentedreality provides a promising use case for visual cryptography.The paper is organized as follows. Section 2 presents back-groundonvisual cryptographyandaugmentedreality. InSection 3,we present our methodology for implementing vi-sual cryptography using augmented reality displays (ARDs).WedescribeouruserstudiesinSection4andpresenttheresultsinSection5. Finally, wepresentthelimitationsofour work in Section 6 and the conclusion and future work inSection7.2. BACKGROUNDInthissectionwegiveabriefintroductiontovisualcryp-tography(Section2.1)andaugmentedreality(Section2.2).Wethendiscussanapproachtocombinethesetechnologiesthat enables message recovery without trusting any individ-ual device with the message (Section 2.3) and that forms thebasisof ourimplementation(whichisfurtherdescribedinSection3).2.1 Visual CryptographyVisual cryptography [26] is a cryptographic secret-sharingscheme where visual information is split into multiple shares,suchthatonesharebyitself isindiscerniblefromrandomnoise. (Equivalently, oneof thesesharesconstitutesaone-timepadasdiscussedinSection1,andtheotherrepresentsthe ciphertext of the secret message.) The humanvisualsystemperformsalogical ORof thesharestodecodethesecretmessagebeingshared.Invisualcryptography,amessageshareisrepresentedbyarectangulargridofblocks,eachblockitselfbeingasquare22 grid of regions. Each block in a message share includestwoblackregionsandtwotransparentregions,asshowninFigure1a. Theciphertextimageisdecodedbyoverlayingtwoimagesharesontopofeachother. ConsidertheblockinFigure1b. Ifonesharehasablockwiththetwotopre-gions black and the two bottom regions transparent and thesecondsharehasthereversetwotopregionstransparentandtwobottomregions blackthenwhenthetwosharesare overlaid the resulting block will appear solid black. Thisisthedecodedimage. Recoveringawhiteblockisdoneinasimilarmanner,asshowninFigure1c. Notethatthere-covered white blockispartiallyblackandpartiallytrans-parent. This gray blockisusedtorepresentwhite.ConsiderFigure2,inwhichthesecretmessageisdecom-posedintotwoshares showninFigures 2a2b, suchthatonstackingthetwo, theuserseesFigure2c. Eventhoughthecontrastof theimagehasbeendegradedby50%fromregularblack-on-whitetextduetothegrayblocks, thehu-man visual system can still identify the content of the secretmessage(123)easily.(a)Validblocksforanimageshare(b) Recovering a blackblock(c) Recovering a whiteblockFigure1: Visualcryptographyblockrepresentations(a)Firstshare (b)Secondshare (c)OverlaidsharesFigure2: ExampleofvisualcryptographyWhenthehumanvisual systemdoesthedecryption, nocomputationisneeded; itisavisual ORoperation. Thatis,ifagivenregionisblackineitheroftheoverlaidshares,the user sees a black region. Otherwise the user sees a whiteregion. Eventhoughthehumaneyeperformsalogical ORoperationper region, theresult is anXORoperationperblock. Hence, the security of the one-time pad is retained. Inparticular,an individual share reveals no information aboutwhether a specic decoded plaintext block is black or white.Adisadvantageoftheprocessisthelossincontrastofthedecryptedmessage, whichinturnaectstheclarityof therecoveredsecret.Whileanadversaryinpossessionofonlyonesharelearnsnoinformationaboutthecorrespondingsecretmessage, hemightstillcompromisetheintegrityofthesecretmessageifheisabletomodifyoneofthesharesinameaningfulway.A method to defend against this attack, introduced by Naorand Pinkas [25], is to use part of the secret message to conveythe desired information and deliberately leave part of the se-cret message as a solid color, i.e., every block consisting oftwoblackregionsandtwotransparentregions(white)oreveryblockconsistingoffourblackregions(black). Iftheadversarymodiesoneoftheblocksinonesharebyswap-ping the white and black regions, the result will be to changethe color of the block in the decoded plaintext image. If thechangedblockhappenstobeinthededicatednon-contentregion, therecipientwill knowthatoneof theshareswascompromised. If half of thesecretmessageisadedicatednon-content region then the adversary has only a 50% chanceof successfullymodifyingasharewithoutnoticeablymodi-fyingthenon-contentregionofthedecodedplaintext. Oneofthequestionsweexamineinouruserstudyiswhetherauserwill alwaysnoticeasingleblockofthewrongcolorinanon-contentregionofthedecodedplaintextimage.ThequestionofhowtoconveyamessageusingVCsuchthatanymodicationtoasharebyanattackerwill bede-tectable has been well studied [6, 31, 9, 15]. These proposalsall make some assumptions about the capabilities of the hu-manvisualsystem(HVS),whichweattempttovalidateinouruserstudies. Thequestionsweexploreare: whethera2USENIX Association 2015 Symposium on Usable Privacy and Security91humanwilldetectablockinamessagesharethatdoesnothaveexactlytwoblackregions; whetherauserwill noticeblocksinthedecodedmessagethathaveeitheroneorthreeblack regions; whether a user will notice blocks of the wrongcolorinanon-contentregion(asdescribedinthepreviousparagraph);andwhetherauserwillnoticeblocksthatsub-tlychangethesemanticmeaningofthedecodedmessage.2.2 Augmented RealityAugmentedreality(AR) systems supplement realitybycombiningareal scene viewedbythe user andavirtualscenegeneratedbythecomputer[24, 5]. TheARsystemsuperimposes a users environment with computer-generatedinformation, making the users environment digitally manip-ulableandinteractive.ThemostcommonrealizationofARiswithsee-throughhead-mounteddisplays(HMDs). AnHMDisadevicethatistypicallyattachedincloseproximitytotheeyeswithadisplaysystemthatcouplesadigital imagewiththeenvi-ronment [11]. HMDs enablethemergingof virtual viewswith a users physical environment, for example by enablingaphysiciantoseethe3Drenderingof CTimagesof apa-tientsuperimposedontothepatientsabdomen[4]. Were-fertotheseaugmentedrealityHMDsasaugmentedrealitydisplays (ARDs) for brevity. ARDs have startedtoshipcommercially, withthe latest examples being MicrosoftsHoloLens[23]andGoogleGlass[13].Thehumanvisual systemformsacoreelementof thesenear-eye ARDs, especially when considering the performanceandusabilityofthesystem. Theeldofview(FOV)ofthehuman eye describes the dimensions that can be seen by theeyeatanygiveninstantof time. Forthehumaneye, thiseldof viewis approximately150horizontallyand120vertically. The binocular overlap region, within which a tar-getisvisibletobotheyes,measuresabout114[11].Many commercially available binocular ARDs like the Lu-musDK-32,theOptinventORA-S,andtheEpsonMoverioBT-200/100 are limited to a FOV of less than 60. Maimoneetal. [21] developedanear-visionsee-throughARDwithaeldof viewof approximately110diagonally, andsofarthisisthemaximumFOVthathasbeenachievedwithARdisplays. GoogleGlasshasa14monocularFOVandtheEpsonMoveriohasanFOVof23[28]. Themajorlimita-tionresultingfromasmall eldof viewistheinabilitytoshownerdetailsonARDs.There are several challenges in overlaying information ontoausersFOV,suchascontrastsensitivityandcolorpercep-tion. ThecolorperceivedbyuserswearinganARDisaf-fectedbythetransparencyof thegraphicsbeingrendered.Inaddition,somedisplayssignicantlydiminishthebright-nessof thereal worldsothatthegraphicsonthedevicesdisplayneednot bebright [11, 20]. Theseissues mayre-sultinchangesincontrastandhuesperceivedbytheuser.For example,since black is rendered as transparent to allowtherealworldtobeseen,darkcolorsmaybeperceivedim-properly,andbrightcolorsthatareintendedtooccludethereal-worldbackgroundmaynotbeabletodoso[20].Anotherchallengeinhead-mountedARDsisheadjitter.As a user tries to focus on an object, slight head movementstranslatetolargemovementsofthedisplayedobject. Thisisparticularlyimportantforourscenario, astheusersaretrying to align two images. However,these problems can beaddressedusingbuilt-inimage stabilizers, motionsensorsandheadtracking[8].2.3 Using Devices to Implement VCWhileourworkis, toourknowledge, thersttoexploreusingaugmentedrealityhead-mounteddisplays toimple-mentVC,otherworkshavesuggestedtheuseofspecializeddevices to replace the transparencies envisioned by the orig-inal VCauthors. Inparticular, Tuylsetal. [32] suggestatransparent displaydevicethat theuser holds or attachestoher computer display. Toreveal aplaintext totheonlooking user,the transparentdisplaydevice renders one im-ageshare,andtheunderlyingcomputerdisplayrenderstheother. Neither the transparent display device nor the under-lyingcomputer displayindividuallygains anyinformationabouttheplaintextimagethattheusersees.Whilethedesignwetestreplacesthetransparentdisplaydevicewithanaugmentedrealitydisplay(ARD),otherwiseitisconceptuallysimilarandborrowsunderlyingelementsfromtheTuylsetal. design. Inparticular, asendergener-atesandsecurelytransmitsimagesharesinourenvisioneddeploymentinthesameway(subjecttosomeARD-specicconstraints, discussedinSection3), andthereceivingusercan read the message without trusting either the ARD or thecomputerdisplaywiththecontentsof themessage. Addi-tionally, Tuyls et al. demonstrate how the user may respondwithoutdivulgingherresponsetoherdevices.Westress, however, that our contributionis not intheconceptual novelty of our design of a VC system using ARDs,but rather inastudyof theusabilityof our approachfordoingso. Toourknowledge, wearethersttoundertakesuchausabilitystudy.3. IMPLEMENTATION OF VC USINGAUGMENTED REALITYAs originally envisioned, visual cryptography used printedtransparencies consistingof black(opaque) andtranspar-ent regions [25, 2]. However, ARDs render images onaglass prismvialight projection. Blackregions are repre-sentedbytheabsenceof illuminationandaretransparent,while formerly transparent regions are now rendered throughtheprojectionof whitelight[11, 20]. Thus, thenotionoftransparencyandopacityinvisual cryptographyachievedthroughARDsisinvertedwithrespecttotraditionalvisualcryptographyusingprintedtransparencies.Inourimplementation, eachshareof thesecretmessageiscomposedof2 2blocks. Eachblockinashareincludestwo illuminated (white) regions and two transparent (black)regions. When two images are overlaid, the regions combineas inFigure3. Inour initial experiments, wetriedusingcolorsotherthanwhitetorendertheopaqueregions, butwefoundwhitewasthemosteective.One share is sent to the users ARD, and the other is sentto a display in the vicinity of the user. To superimpose them,theuserviewsthevisualsharethroughtheARDdisplayingthe second visual share. The two shares thus overlaid revealthesecretmessage. BelowwedescribehowwedesignedoursystemtoworkwithGoogleGlassandtheEpsonMoverio.3.1 Google GlassGoogleGlassisanAndroidbasedaugmentedrealityde-vice that is Wi-Fi andBluetoothenabled. It provides a3922015 Symposium on Usable Privacy and SecurityUSENIX AssociationFigure3: Constructionof visual cryptographyschemeforaugmentedreality. Thesecretblockisobtainedbysuper-imposingthetwocorrespondingshares,lookingthroughanaugmented-realityhead-mounteddisplay.monocular ARexperience withasmall displayscreenfortherighteye. Glassenablesuserstoviewcontentinclud-ingtext, images, andvideos. Itappliestransformationsonimages that aredisplayedonit suchthat theyarescaledinboththehorizontal andvertical directions. Tocounter-balance that transformation, we scale the images to 500600pixels and pad them with a black background, such that ournal imagesare1200 1200pixelsonadesktopmonitor.ThedisplayofGoogleGlasswitharesolutionof640 360pixels(equivalentof a25in(64cm)screenfrom8ft(2.4m)away)[13]isverysmall,withaeldofviewof14. WefoundthemaximumimagesizeforthesecretmessagethatcouldcomfortablybedisplayedonGoogleGlasswasagridof 5 5blocks, witheachblockcontainingtwoblackandtwowhiteregions.In addition to the above transformations, we added align-mentmarkers(redbordersaroundblocks,asshowninFig-ure4)tohelpusersalignthetwoimages. Wealsoblackedout the areas that contained no useful information to furtheraidimagealignmentandaccountforheadjitter.3.2 Epson MoverioOursecondstudyusedanEpsonMoverioBT-200ModelH423AanAndroidbasedaugmentedrealitydevice, withdisplayscreensforbotheyes, i.e., binoculardisplay. Ithasaresolution of960 540pixels (equivalentto80inchimagefrom 16.4 feet away) [28, 10]. The Epson Moverio has a eldof viewof 24andenables 2Dand3Dviewing. The 2DmodeissimilartotheviewingmodeinGoogleGlassanditisthemodeweusedinourstudy. Weusedtwoimagesizescontaining 77 blocks and 95 blocks, respectively. Thesesizeswereselectedbasedontrial anderrorforwhatwasacomfortableandeasyimagesizeforalignmenttobedonebyuntrained, rst-timeparticipants. Theinvestigator forthestudy, becauseof trainingeects, wasabletopackuptotwolettersorfournumbersintoanimageandstill abletoalignandidentifythecharactersintheimages.Eventhoughtheinformationbeingconveyedis limited,there is a markedimprovement over using Google Glass.WebelievethisismostlybecauseoftheEpsonslargereldof view. Withalargereldof viewandhigherresolutionmoreinformationcanbeconveyedthroughoneimage.Onelimitationof thetwoheadsetsused, orsomeof theothersthatarecommerciallyavailable, suchastheOptin-vent ORA-S, Recon Jet, Vuzix M100, and Lumus, is absenceof anyformof built-inheadtrackingandimagestabiliza-tion. This wasoneof themajor challengesthat wefacedduringtheGoogleGlassstudy: evenslightheadmovementmisalignedtheimages. Weelaboratemoreonthis as wedescribetheuserstudies.3.3 Threats introduced by ARDsAsdiscussedinSection1, ourgoal istoimplementVCinsuchawaythatrealisticattackerscanaccess(toreadormodify)onlyoneof thetwovisual sharesneededtorevealthe secret message. While our focus inthis paper is onthe usability of ARDs for this purpose, we pause to considerchallenges in ensuring this property with the two ARDs thatwehaveusedinourimplementations.Onetypeofattackerthatposeschallengeswitheitherofour ARDs is athirdpartyinproximityof the user whomight photographboththephysical share(i.e., theshareintheusersphysical proximity)and, withhighdenition,thesharedisplayedintheARD. Sincethisattackerwouldnot seethesetwoimages alignedas theintendedmessagerecipientdoes,hewouldthenneedtoreconstructthesecretmessageoineusingoptical decodingtechniques or man-ual eort. Suchattacks, however, arenotfar-fetched: thepossibility of observing the image displayed in Google Glassis well-known(e.g., bystanders who see the tinydisplayscreenabovetherighteye [19] and thescreenisjustvis-iblefromtheoutside[12]) andsimilarlyintricateopticaldecodingshavebeendemonstrated(e.g., [18]). Thatsaid,thisthreatisequallypresentifnotmoresointradition-allyenvisioneddeploymentsof VCthatuse, e.g., physicaltransparenciestoreveal asecretmessage. UsingARDs, itis presumablyeasier for theuser toensurethat her ARDisnotbeingobserved(e.g., byshadingitwithherhands)duringmessagerecovery.A related concern is that several ARDs (e.g., Google Glass,HoloLens,EpsonMoverioBT-200,andVuzixM100)haveafront-facing camera. (Several others, like the Epson MoverioBT-100andLasterSee-Thru,donot.) Assuch, ifanARDwithafront-facingcameraiscompromisedbymalware, itcouldpotentiallyusethefront-facingcameratophotographtheothershareandcombineitwiththesharegiventotheARDtodisplay, revealingtheprivatemessage. Itisthere-forenecessarythat,foranARDwithafront-facingcamerasuchasGlass, thecameralensbecoveredphysicallywhilethephysicalshareisdisplayed. Wedidnotincorporatethisstepintoouruserstudybutwouldrecommenddoingsoinactual deployment. Similarly, if thephysical shareis dis-playedona computer display, that computer shouldnothaveacamerafacingtheuserthatmightcapturethesharedisplayedintheusersARD(orelsethatcamerashouldbephysicallycovered).Finally, ARDs like Google Glass are not really standalonedevices; rather, they share their information with the serviceprovidertowhichtheyaretetheredGoogleinthecaseofGlass. Thereisariskthatthisserviceprovidercouldbothextract image shares from the ARD and combine them withthecorrespondingotherimagesharesthatitreceivesoth-erwise(e.g., senttotheusersgmail account). Addressingthisriskpresumablyinvolvesthesenderconveyingcipher-texts totheuser viaadierent serviceprovider thanthe4USENIX Association 2015 Symposium on Usable Privacy and Security93onewithwhichtheARDsharesitscontents,ortoadoptanARD that does not so freely share its contents with a serviceprovider.4. USER STUDIESWeconductedthreeuserstudies: apilotuserstudyusingGoogleGlass, animprovedformal studyusingtheEpsonMoverioARD, andanonlineuserstudyusingMechanicalTurk. All ouruserstudieswerereviewedandapprovedbyour universitys IRB. In the pilot user study we gauged par-ticipantsabilitytoalignvisualsharesanddiscerntheover-laid information from noise. In the Moverio user study, par-ticipantsdecodedlettersandnumbers,andwealsoinvesti-gated their ability to recognize modications to the decodedplaintexts. In the Mechanical Turk study weassessed usersabilitytodetect modications toindividual imageshares.Thepilotuserstudyandtheformal userstudywerecon-ductedover aperiodof twoweekseach. Theformal userstudy was conducted three months after the pilot user study.4.1 Participant DemographicsThe pilot study had 31 participants: 22 male and 9 female.Outof the31participants, 19wereagedbetween18-25, 9between26-35and3between36-45. Ofthe31participants,8woreprescriptionglasses. Intheformalstudy,therewere30participants, of which23weremaleand7werefemale.Theagegroupsincluded17participantsbetween18-26, 11between 26-35, and 2 between 36-45. Of the 30 participants,15woreprescriptionglasses. Bothparticipantgroupscon-sisted mostly of university students,sta,and faculty mem-bers. Recruitmentwasdonethroughemailstodepartmentandstudentgroupmailinglists. Werecruitedparticipantsthiswaybecauseparticipantshadtocometoourocelo-cation(onour universitycampus) for participatinginthestudy.Given the nature of these studies, we believe the most im-portant bias in our participant pools is that toward youngerparticipants. Whilewedonotclaimthattheseparticipantgroups enable us to generalize our results to the general pop-ulation, we are hopeful that they should be representative ofpopulations represented by these age groups. Results wouldvary for other population groups and can be explored as partoffuturework.FortheMechanical Turkstudy, wehad50participants,with28maleand22femaleparticipants. Theagegroupswere 14 between 18-25, 20 between 26-35, 12 between 36-45,and4between46-55.4.2 Pilot User Study4.2.1 TrainingTo acclimate users to the experience of using Google Glass,eachuserrstunderwentabrieftrainingsessiononhowtoperform image navigation and alignment on the device. Thetrainingsetcomprisedtwostages: First, threephotosnotrelated to the study were displayed to demonstrate the act ofnavigation between pictures in Google Glass image browser.Then, threeimages containingblocks (seeFigure4) werepresentedtotheuseraspracticefortheactual task. TheuserswereaskedtoaligntheblocksonGlassandthemoni-torscreenandreportwhichblockswerewhite. Duringthistrainingphase, theparticipants wereprovidedguidanceiftheyrespondedincorrectly.Inthetrainingphaseandintask1below,eachrecoveredmessage was a Braille character. We used Braille charactersbecause they t well in a 53 block grid we used with Glass.ABraille character is a three-row, two-columngridwithraised dots insomecellsthatcanbefelt. Thelocationsofthesedotsinthegridindicateacharacter(e.g., see[1]).Weusedawhiteblocktocorrespondtoaraiseddot andablackblocktocorrespondtotheabsenceofaraiseddot.OurchoiceofBrailleforthispilotstudywasprimarilyduetothe simplicityof this mappingof Braille characters toVC, andofcoursenotbecauseweintendforblindpersons(theprimaryusers of Braille) tomakeuseof VCor AR.Theparticipantswerenottoldthattherecoveredmessagesshould be Braille characters and no knowledge of Braille wasrequired.4.2.2 Task DescriptionsAfter training,the participants were given three tasks. Ineachtask, eachparticipantwasgivenasetof imagepairs(oneinGlassandoneonthecomputermonitor)andaskedto overlay and align the images on Glass and the monitor togureoutwhichblockswerecompletelywhite. Thepartici-pantthenhadtonotethisdownonasheetofpaperbeforemovingontothenextpairof imagesintheset. A3 2gridwasdrawnonthepaperforeachimagepair, andtheparticipantsmarkedtheobservedwhiteblocksonthegrid.Eachparticipantlledoutaquestionnaireaftercompletingeachofthethreetasksandthenanal questionnaireafteralltaskswerecompleted.Each participant was timed for each image pair presented.ThetimeddurationincludedthetimetakentonavigatetotheimageonGoogleGlassaswellasthemonitor,alignthetwoimageshares, identifythewhiteblocks, andnotetheirlocationsonasheetofpaper.(a)Firstshare (b)SecondshareFigure4: ImagesharesforBraillecharacter iTask1: Task1hadve image pairs; anexample imagepair is showninFigure 4. The image pairs inthis taskweresharesof randomlychosenEnglishBraillecharacters.All theparticipantswerepresentedadierentsetofimagepairs. Theparticipantshadtowritedownwhichblocksintherecoveredplaintextwerewhite.Task2: Thistaskwassimilartotheprevioustaskbuttheimages were shares of a 33 grid of blocks with at most onewhiteblock, e.g., seeFigure5. Thelocationof thewhiteblock(ifany)waschosenuniformlyatrandom. Therewereve image pairs in this task, and the participants were askedtowritedownwhichblockineachrecoveredplaintext (ifany)waswhitebywritingthewhiteblocksnumber, withtheupperleftcornerbeingblockone, thelower-rightcor-nerbeingblocknine,andnumbersincrementingalongrows(likeatelephonekeypad). Iftherewasnowhiteblock, the5942015 Symposium on Usable Privacy and SecurityUSENIX Association(a)Firstshare (b)Secondshare (c) Decodedplain-textFigure 5: Image shares for the number 5 in pilot study (Sec-tion4.2)participant was instructedtowriteazero. Wechosethistaskdesignbecausethemajorityofthepopulationisfamil-iarwiththisnumber-padpattern,andgiventhelimitationsofusingVCwithGoogleGlassinitscurrentform,thiswasaviablewaytoconveynumbers.Task 3: The images used in this task had the same layout asintask2. Eachparticipantwasgiventhreepairsofimagesandaskedtoindicatewhichblock(ifany)oftherecoveredplaintext was white. However, unlike intask2, all threevisual sharesweredisplayedside-by-sideatthesametimeonthemonitor. Theparticipantsstill hadtonavigatetheimagesharesonGlass, however, andoverlaythesharesonGlass with the shares on the monitor from left to right untilcompletingthetask. Wechosethistaskdesigntotesttheeaseof alignmentwhenmorethanonephysical sharewasdisplayedonthemonitor. Forthistaskalltheparticipantswere given shares for the same three numbers, which (for noparticularreason)wastheroomnumberoftheroomwherethestudywasconducted.4.2.3 ResultsFigure6presentsthedistributionofaverageplaintextre-coverytime per participant, i.e., averagedover all imagepairsinthetaskindicatedonthehorizontal axis. Atimerrunninginthebackgroundcapturedthetimespentperim-age. Assoonasaparticipantnavigatedtothenextimage,thetimerforthatimagestarted.Eachboxinthis plot consists of three horizontal lines,which indicate the 75th, 50th (median), and 25th percentilesof the average time per participant. Whiskers extendtocoverthelowest(highest)pointwithin1.5theinterquar-tile range of the lower (upper) quartile. There was no signif-icantdecreaseintimespentperimagepairasaparticipantprogressedthroughanindividualtask.Thereisnoticeablevariationintheamountoftimespentbyusersperimagepair, rangingfromafewsecondstoaslarge as 40 seconds. For identifying the white blocks intherecoveredplaintext, theusers not onlyhadtoattunethemselvestowhattheyarelookingforbutalsouseanewtechnologyall oftheuserswererst-timeusersofGoogleGlass.Thereisnorelationshipbetweenerrorratesandtimingdata. Participantswhoidentiedthedecodedplaintextin-correctlymayormaynothavespentlittleormoretimeonthose visual shares. Participants who had a higher error ratedidnotonaveragetakelongerthanparticipantswhohadnoerrors. Atotal of 9participants out of 31decodedatleast one plaintext incorrectly. Table 1 shows the number ofFigure6: Boxplotshowingaverageplaintextrecoverytimeperparticipant, i.e., averagedoverthenumberofplaintextrecoveries inthe taskonthe horizontal axis, inthe pilotstudy(Section4.2).Task1 Task2 Task3Allcorrect 22 28 271incorrect 6 3 42incorrect 1 0 03incorrect 0 0 N/A4incorrect 2 0 N/A5incorrect 0 0 N/ATable1: Numberof participantspererrornumberinpilotstudy(Section4.2)participants that made the number of errors indicated in theleftmostcolumn. Atotalof403imagepairswerepresentedtotheparticipants,outofwhich16occurrenceswereincor-rectlyidentied. Oneinterestingobservationisthatif theparticipantmadenoerrorinthersttask,thentheymadenoerrorsinthesubsequenttasks.It was harder for participants wearing prescription glassestoclearlyseeimagesonGlassandalignthem,asindicatedbythemonthequestionnaires. SomeparticipantsreporteddiscomfortusingGoogleGlassastheywereleft-eyedomi-nant. WemovedtotheEpsonMoveriointhesecondstudytoaddress these issues. Manyparticipants reportedthatthealignmentwashardbecausetheirheadswouldnotstaycompletelystillandtheywerethusunabletogureoutifaparticular block of the recovered plaintext was white or not.As there is no form of head tracking or image stabilization inGoogleGlass, thiswasaproblem. Wemodiedthedesignof theseconduserstudybasedonthisinformation. Someusers also reported that they took longer on images becausetheyhadtorecall whichblockswerewhitewhilemarkingtheir answers down on the sheet. Based on this, we changedthedesignofthenextstudysothattheusersdidnothavetowritedownwhattheysee.4.3 Formal User StudyFor the second user study we used an Epson Moverio BT-100,anAndroid-basedARD[10]. UnlikeGoogleGlass,theMoverio presents content for both eyes, i.e., it is has a binoc-ular display. However, we found the alignment to be tediousinabinocularsetting; therefore, weusedtheMoverioonly6USENIX Association 2015 Symposium on Usable Privacy and Security95(a)Frontview (b)SideviewFigure7: Chinrestsetupfortheformaluserstudy.asamonocularheadset,butallowtheuserstoselecteithertheleft-ortheright-eyedisplay.Similar to Google Glass, the Epson Moverio does not haveimagestabilizationorheadtrackingcapabilitiestoaccountforheadjitter. Tocompensateforthis, weprovidedachinresttotheparticipants(seeFigure7). ThechinreststandalsosituatedtheARDonasmall platformforaddedsup-port.One visual share was displayed on the Epson Moverio andthe other was displayed on a monitor. To enable image align-ment, participants could move and scale the visual share dis-played on the monitor. An Xbox controller was used for thispurpose. The participants reported their observations (whattheywereaskedtoreportvariedbasedonthetask)andtheinvestigatornotedthem. Attheendof thestudy, partici-pants lled out a questionnaire. The questionnaire aimed tocapturetheircondencelevelsasthestudyprogressed, aswellasparticipantdemographics.4.3.1 TrainingAsinthepilotuserstudy, theparticipantsunderwentashorttraining. Participantsweregivenexamplesoftheim-agestheywouldbeviewinganddescriptionsof thetasks.Throughapresentationandpracticesessiononthesetup,participantswerefamiliarizedwiththetasks. Thedecodedplaintext consistedof individual letters andnumbers thatresemble a 14-segment LED display font, as in Figure 8. Af-ter the training, participants took a 1-2 minute break. Eachtaskwasdescribedagainasparticipantsstartedthetasks.Figure8: 14-segmentLEDfontusedfordisplayinglettersandnumbersinthestudydescribedinSection4.34.3.2 Task DescriptionsTherewerethreetasksinthestudy. Eachtaskinvolveda set of ten image pairs. In each of the tasks, partici-pantsalignedtheimagepairs(visualshares)andidentiedthedecodedplaintextcharacter. Theyalsoreportedothercharacteristics of thedecodedplaintext for tasks 2and3.Theparticipantsweretimedforeachimagepairpresented;thisincludedthetimetakentonavigatetothenextimagepair, identify the characteristics of the decoded plaintext re-quested in each task and report their observations. A partic-ipant had to align only the rstimage pair of each task;therestoftheimagepairswouldretainthatalignment, unlesstheusershiftedherposition.Task1: In task 1, users simply identied the decoded char-acters. Eachparticipant was giventenimage pairs thatencodedrandomcharactersvenumbersandveletters.Uponoverlayingthetwovisual sharesontheEpsonsdis-playandthemonitor,theletterornumberwasrevealed,asinFigure9.(a)Firstshare (b)Secondshare (c) DecodedplaintextFigure9: ImagesharesforletterH(Section4.3)Task2: Inthesecondtask, werstreiteratedtothepar-ticipant what constitutesalegal decodedplaintext image,namelyoneinwhichtheblocksthatformacharacterhaveallregionswhite(illuminated)andallotherblockshaveex-actly two black and two white regions. All participants weregiventenimagepairs, wherethedecodedplaintextofeachpair was aletter. For eachimagepair, weaskedpartici-pants to identify the letter in the decoded plaintext, identifywhethertheplaintextislegal,andifnot,tocountthenum-berofblocksthatmadeitillegali.e.,blocksthatarepartoftheletterbutnotwhollywhite,orblocksnotpartoftheletterthatarenothalf-blackandhalf-white. Herewecallsuch blocks nonconforming. All decoded plaintexts in task 2hadbetweenoneandsixnonconformingblocks, thoughwedidnotinformtheparticipantsthatall decodedplaintextsinthetaskwereillegal.All participantsweregiventhesameimagepairsinthistask, and the plaintexts decoded from these image pairs werechosentoinclude confusing letters. Forexample,noncon-formingblocksatparticularlocationscancauseconfusionespeciallybetweenF&E,O&U,andT&I.Sixofthetenimagepairsprovidedtoparticipantswereintention-allychosentoyieldoneof theseletters, andnonconform-ingblockswerepositionedintheirdecodedplaintextssoastomaximizeconfusion. ExamplesofdecodedplaintextsforthesethreeletterpairsareshowninFigure10.Task 3: Intask 3, eachdecodedplaintext hadacon-tentandnon-contentpart, as suggested by Naor andPinkas[25]toincreasethelikelihoodofdetectingadversar-ial manipulationof animageshare(seeSection2.1). Thecontent part carries the message, in this task a number. Thenon-contentpartcontainsnomeaningfulinformationandismadeupof blocksof all thesame color, i.e., blocksthatall haveexactlytwoblackandwhiteregions(i.e., black)or blocks that all have four white regions (i.e., white). Forthe purposes of our study, in a legal decoded plaintext, eachblock of the non-content part has exactly two black and twowhiteregions. Thus, anillegal plaintextissimplyonecon-tainingwhiteblocksinitsnon-contentpart. ThesearethenonconformingblocksasshowninFigure11.The content part of a plaintext could be either on the leftortherightoftheplaintextimage. Therewasashiftfrom7962015 Symposium on Usable Privacy and SecurityUSENIX Association(a) F withfournonconformingblocks inbottomrow, or E withsix nonconform-ing blocks inbottomrow(b) O with threenonconformingblocks in top row,or U withthreenonconformingblocksintoprow(c) T with venonconformingblocks inbottomrow, or I withsix nonconform-ing blocks inbottomrowFigure10: Decodedplaintexts for threeimagepairs withnonconformingblocks tomaximize confusionbetween(a)F &E; (b)O &U; or(c)T &I; usedintask2ofstudydescribedinSection4.3(a) Legal decoded plain-text(b) Illegal decodedplaintextFigure11: Decodedplaintextsobtaineduponoverlayingvi-sual sharesintask3.Thecontentpartisontherightsideandthenon-content is ontheleft. (a) shows alegal im-age with non-content region containing only blocks with twoblack and two white regions; (b) shows an illegal image withmalformedwhite-blocksinthenon-contentregion.righttoleftonceduringthetask. Theparticipantwastoldthat a switch would be signaled by an all-white non-contentregion(afewnonconformingblocksnotwithstanding); i.e.,this was a signal that the next decoded plaintext would haveitsnon-contentregionontheothersidefromitspresentlo-cation. Ifthenon-contentregionwasall-black(again,asidefromafewnonconformingblocks),thenthenon-contentre-gionwouldbeonthesamesideinthenextdecodedplain-text. The participants were asked to report for each decodedplaintextwhetherthecontentwasontheleftortheright,report the alphanumeric character they decoded, and reportifthenon-contentpartwaslegaland,ifnot,thenumberofnonconformingblocksitcontained.Indoingso, the taskevaluatedusers abilitytodetectnonconformingblocksinanon-contentarea. Thenumberof nonconforming blocks in this task ranged from 0 to 10. Wealso observed the minimum number of nonconforming blocksneededinthe non-content part of adecodedplaintext inorder for users to be able to discern the plaintext as illegal.4.4 Mechanical Turk StudyAnadversaryinpossessionof animagesharemightat-tempt tomodifyblocks intheimagesharetochangethemeaningofthedecodedplaintext. Forexample,byippingtheblackandwhiteregions intheimage-shareblock, theadversarychangeswhetherthecorrespondingblockinthedecodedplaintext is blackor white. However, topredictFigure12: Imagesharewithninemalformedblockshares(Section4.4)theresultingblockcolorinthedecodedplaintext, hemustknowthecolorofthatblockintheoriginal decodedplain-text, i.e., priortoippingtheblackandwhiteregions. Ifhedoes not knowthecolor of theoriginal decodedplain-textblock, hecanneverthelessincreasethechancesthatitis turned to white by modifying the block in his image sharetohavethree(orfour)whiteregions, insteadof onlytwo.Inpriorimplementationsof VC, detectingsuchmalformedblocksmaygoundetectedinanindividualsharebecauseofthesmall sizes of theblocks. However, inour implemen-tation,detectingthesemalformedblocksinanimageshare(i.e.,blocksthatdonothaveexactlytwoblackregionsandtwowhiteregions)isfeasibleandcanbeananimportantstep to detecting an adversarys manipulation of that share.WeconductedanonlineuserstudyonAmazonMechani-cal Turk to evaluate users ability to do so. Each participantwasgiventhesame20imagesharesinarandomorderandwasaskedtoidentifywhethereachcontainsanymalformedblocksand, ifso, howmany. Notethateachimagewasanimage share,and so there was no other meaningful informa-tionintheimageshare. AnexampleisshowninFigure12.This image share has nine malformed blocks that are all nexttoeachotherinagroup. Themalformedblockscouldalsobepositionedrandomlyintheimageshare.Oneof thegoalsof thestudywastoobservethedier-encesinidentifyingrandomlypositionedmalformedblocksandgroupedmalformedblocks. Hence, theset of imagesgivento the users was a mix of both: there were eightimageshares withrandomlypositionedmalformedblocks,eightimageshareswithgroupedmalformedblocks,oneim-age share with one malformed block, and three image shareswithnomalformedblocks. Wealsoexploredtheleastnum-berof malformedblocks(bothgroupedandrandom)thatenabledparticipantstoreliablydiscernthatanimageshareis illegal. Thepresenceof groupedmalformedblocks is acharacteristicoftheattacksdescribedbyChenetal.[6].Attheendofthestudytheparticipantslledoutaques-tionnairetocapturedemographicsandtheircondenceasthestudyprogressed.5. RESULTSThis sectionpresents the results andanalysis fromtheformal user study(Section5.1) andthe Mechanical Turkuserstudy(Section5.2).5.1 Formal User Study Results5.1.1 Timing dataFigure13showstheaveragetimefordecodingplaintexttakenbytheusersineachofthetasks. Ofparticularinter-est is the rst task since it reects the amount of time takentoidentifythelettersornumbers. Fortasks2and3, the8USENIX Association 2015 Symposium on Usable Privacy and Security97Figure13: Boxplotshowingtheaveragetimeforplaintextdecoding (excluding the rst) per participant, in each of thethreetasksasdescribedinSection4.3Relationbetween Correlationcoecient p-valueTasks1and2 0.7285 5.0144 1006Tasks2and3 0.6679 5.50604 1005Tasks1and3 0.3867 0.0348Table2: CorrelationbetweentasktimesbasedonPearsoncorrelationinuserstudydescribedinSection4.3reportedtimesincludethetimetakentocountthenumberof nonconformingblocks. Theactual timetoidentifythecharacters and simply the presence of nonconforming blocksinthedecodedplaintextislower. Figure13doesnottakeintoconsiderationthetimetakenfortherstimageineachtask. For therst imageineachtask, users spent acon-siderable amount of time initially aligning the image shares,yieldinganoutlierthatdramaticallyskewstheaveragespertask. The time takento initiallyalignthe image sharesrangedfrom18.49to313.32secondsinthersttask;11.94to303.16seconds inthe secondtask; and27.4to280.79secondsinthethirdtask.Thereisacorrelationbetweenthetimetakeninthedif-ferenttasksonaper-userbasis. Table2showsthePearsoncorrelationcoecientsandcorrespondingp-valuesforeachofthetimingrelations. Participantswhotooklongeronei-thertask1or2werelikelytospendmoretimeintask2or3. Astrongercorrelationwasobservedbetweentiminginconsecutivetasks, i.e., betweentasks1and2andbetweentasks 2and3. Wealsofoundacorrelationbetweentimetakenintask3andimageclarity(seeSection5.3.1).5.1.2 Error RatesIn each task the participants reported the characters theyobserved. Intask1, 26outof30participantsidentiedallimages correctly, and the remaining 4 participants identied9 out of 10 images correctly. In task 2, 28 participants iden-tiedthelettersinalltheimagescorrectly,oneparticipantidentied9imagescorrectly, andoneparticipantidentied8imagescorrectly. Intask3, 27participantsidentiedthenumberintheimagescorrectly, twoparticipantsidentied8imagescorrectly, andoneparticipantidentied9imagescorrectly. This indicates that users were able to clearly iden-tifytheletterornumberbeingconveyed. Theparticipantswho made mistakes identifying characters in task 2 had alsomadeerrors intask1. Twoout of thethreeparticipantswhomadeerrorsinidentifyingthenumbersintask3alsodecodedtheplaintextsincorrectlyintasks1and2.In tasks 2 and 3,we also evaluated users ability to recog-nize illegal decoded plaintexts using the denitions of illegalgiveninSection4.3.2. Inthesetasks,theparticipantswereaskedtoidentifywhetherthedecodedplaintextswereille-gal and, if so, report the number of observed nonconformingblocks.Nonconforming blocks represent an attackers active mod-ications observable in the decoded plaintext (see Section 2.1).Intask2,alldecodedplaintextswereillegal,andintask3,there were sevenillegal plaintexts out of ten. Figure 14showsthepercentageofparticipantresponsesidentifyingadecodedplaintextasillegal asafunctionof thenumberofnonconformingblocks present inthe plaintext. For bothtasks2and3, asthenumberof nonconformingblocksin-creases, moreparticipantsindicatedthatadecodedplain-textwasillegal. Participantswereabletodiscernthatanimagewasillegal evenwhenonlyonenonconformingblockwas present in the plaintext. In task 3, 97% of user responsescorrectlyidentiedthelegal plaintexts.Intask2,participantsweregiven(amongothers)imageswith the letters F, O and T. As discussed in Section 4.3,we suspected that the presence of nonconforming blockscouldeasilycausetheseletters tobeconfusedwithotherletters. All participantsidentiedtheselettersorplausiblealternatives: specically, participantsidentiedeachO aseitherOorU,eachTasT,andeachFasF,E,orP.However,inreportingourresults,weconsideredapar-ticipantsresponseascorrectonlyif shealsoreportedthepresence of nonconformingblocks (i.e., that the plaintextwasillegal).AsseeninFigure14a,whentherewerevenonconform-ingblocksintask2, therewasadecreaseinthenumberofresponsesthatcorrectlyidentiedaplaintextasillegal. Intask2, theonlyletterwithvenonconformingblockswasthe letter K. The nonconforming blocks did not obscure theletter, nor did they cause the K to closely resemble anothercharacter. Thismayhaveledparticipantstoconcludethattheimagewaslegal.Intask3,participantswerealsoaskedtoreportwhetherthecontentwasontheleftortheright. Therewasashiftfromrighttoleftonceduringthetask, whichall 30userswereabletocorrectlyascertain.Figure15presentsthenumberof nonconformingblocksreportedversustheactualnumberofnonconformingblocksinthedecodedplaintext. InFigures15a15b, thethickerredlineindicatesthemedianvalue. Thebottomandtopbarsrepresentthe1standthe3rdquartiles,respectively. Asinglelineindicates that thesequartiles arethesame. Amedianvaluethatisthesameasthe1stquartileorthe3rdquartileindicatesaskewintheresponses.Theplotsindicatethattheparticipantserredtowardre-portingfewer nonconformingblocks thanactuallywereintheplaintexts, andasthenumberofnonconformingblocksincreased, the reported nonconforming blocks also increased.It is important to note that despite not being able to identifyallthenonconformingblocks,89.33%ofdecodedplaintextsin task 2 and 96% in task 3 were correctly identied as illegalbytheparticipants. Inareal-worldscenario,thisabilitytodistinguishbetweenillegal andlegal plaintextscanbecon-sideredmoreimportantthanidentifyingtheexactnumberofnonconformingblocks.9982015 Symposium on Usable Privacy and SecurityUSENIX Association(a)Task2(b)Task3Figure14: Fractionofresponsesindicatingplaintextsasle-galorillegalbasedonthenumberofnonconformingblockspresentinplaintext(Section4.3)The number of participants whowere able tocorrectlyidentifynonconformingblocksintask3ismorethanthatintask2asindicatedbyFigure16. Thisisnotsurprising:Intask3, participantswereaskedtolookforwhiteblocksinaregionexpectedtohaveallblackblocks(orviceversa),whileintask2,thenonconformingblockscouldpotentiallybepartofthecharacter. Thishighlightstheimportanceofusinganon-contentregionasanaidfordetectingpotentialmodicationsmadebyanadversary.Basedonthe observations fromour user study, use ofvisual cryptographywiththeaidof augmentedrealitydis-playsseemspromising, despitethechallengesthatneedtobe overcome. Our participants were able to discern the char-actersbeingconveyedtothem, andtheywerealsoabletorecognizeactivemodicationof thedecodedplaintexts. Itappearstheassumptionsofthevisual cryptographiclitera-tureaboutthecapabilitiesofthehumanvisualsystemholdtrueinanaugmentedrealitysetting.5.2 Mechanical Turk ResultsIn this study, participants were asked to detect malformedblocksinanimageshare, i.e., blocksthatdonothaveex-actlytwoblackregionsandtwowhiteregions. All 50par-ticipantswereabletocorrectlyidentifylegal imageshares,i.e., withnomalformedblocks. However, noneof thepar-ticipantswereabletodetectillegalimagesharescontainingonly one malformed block, except for one user who reportedthattherewerefourmalformedblocksinthatimageshare.Allparticipantscorrectlyidentiedtheremaining25illegalimagesharesasillegal. Therewasoneparticipantwhore-portedalargenumberofmalformedblocksformostimageshares; perhapstheparticipantdidnotclearlyunderstandthe instructions andwas counting the number of regionswithinmalformedblocks,asopposedtothenumberofmal-formedblocks.All theparticipantscorrectlyidentiedall imageshareswithgroupedmalformedblocksasillegal. However,notall(a)Task2(b)Task3Figure 15: Number of nonconformingblocks reportedbyparticipantsversusactual numberofnonconformingblockspresentinplaintext(Section4.3)participants detected that an image share was illegal for thecaseofrandomlypositionedmalformedblocks,asindicatedbyFigure17. Inthiscase, though, asthenumberof mal-formedblocks increased, the number of participants whoindicatedthattheimageshareswereillegal alsoincreased.Mechanical Turkuserstypicallyspentlessthan45secondsperimage.Thenumberofrandomlypositionedmalformedblocksinanimagesharewas3, 4, 5, 6, 7, 9, 10, or12. Thenumberofgroupedmalformedblocksinanimagesharewas4,5,6,7, 9, 10, 12, or13. Figure18showsthenumberof partic-ipantswhoreportedall themalformedblocksinanimageshare, for agivennumber of malformedblocks. It showsthat participants were better at detecting malformed blocksthatweregroupedtogetherintheimageshare, asopposedtorandomlypositionedmalformedblocks. However, thereisaslightdecreaseinthenumberofparticipantswhowereable to detect all the randomly positioned malformed blocksas the number of malformedblocks increased. Figure 18reportsthenumberof participantswhoreportedtheexactnumberof malformedblocks. Thenumberof participantswhoreported12 2malformedblocksfortheimagesharewith12malformedblocksis47. Soeventhoughthepartic-ipantsreportedtheincorrectnumberof malformedblocks,theywerewithinacloserange.10USENIX Association 2015 Symposium on Usable Privacy and Security99Figure16: Fractionofnonconformingblocksreportedbyeachparticipantintask2and3,describedinSection4.3Figure17: Numberof participantswhoindicatedthatim-age shares with randomly positioned malformed blocks wereillegal(Section4.4)Figure18: Numberofparticipantswhocorrectlyidentiedall the malformedblocks inanimage, whether the mal-formedblockswererandomlypositionedorgroupedasde-scribedinSection4.4Figure 19 shows the number of malformed blocks reportedby participants versus the actual number of malformed blockspresent in an image share, for both randomly positioned andgrouped malformed blocks. For both cases,there was an in-crease in reported blocks as the actual number of malformedblocks increased. Figure 19 also shows that the partici-pantserredonthesideofreportingfewermalformedblocksthan were actually present in the image. Again, participantscouldrecognizeillegal imagesharesevenif theycouldnotrecognize all the malformed blocks. In order to detect an at-tackers modication of one share, it would suce for a userto recognize the image share as illegal, rather than count the(a)Randomlypositionedmalformedblocks(b)GroupedmalformedblocksFigure 19: Number of malformedblocks reportedversusnumber of malformed blocks present in the image share (Sec-tion4.4)exactnumberof malformedblocks. However, weexploredthe ability of participants to detect and count dierent num-ber of malformed blocks, to determine the minimum numberofmalformedblocksneededtoidentifyanimagesharethathasbeenmodied.5.3 Questionnaire ResponsesTheformal user studyandtheMechanical Turksurveyposed questions to participants about demographics and theirperceptionsabouttheirownperformanceduringthetasks(seeAppendixAforthequestionnaires).111002015 Symposium on Usable Privacy and SecurityUSENIX Association5.3.1 Formal User StudyWe measured reported condence, character identicationindecodedplaintexts, nonconformingblockrecognitionindecodedplaintexts, andimage claritywithregardtothedecisionprocess. Participants alsoindicatedif theyusedprescriptionglassesduringthestudyand,ifso,howglassesimpactedtheease-of-useoftheARDandimagealignment.Half of our participant population wore prescription glassesduring the study. During the pilot study with Google Glass,prescriptionglasseshadprovedtobeahindranceforimageclarity,alignment,andeaseofuse. Intheformaluserstudythat was not the case. We attribute this improvement to theEpsons form factor,which makes the ARD easier to use,ascomparedtoGoogle Glass, while wearingglasses. Intheformal user study we found no correlation between reportedimageclarityanduseof prescriptionglasses. Participantswearingprescriptionglasses performedneither better norworse interms of accuracyor timing, comparedtootherparticipants.However,amajorityoftheusersindicatedthatclarityoftheimageswasachallenge. InthestudyusingEpson, 11participants indicated that the images were somewhat clear,2 indicated that they were neither clear nor blurred, 13 indi-cated that the images were somewhat blurred, 2 participantsindicatedthattheimageswereblurred, andonly2partic-ipantsindicatedthattheimageswereclear. ThesecanbeexplainedbythechallengesfacedbyARDs, someofwhicharedescribedinSection2.2.Participant perceptionof image claritydidplaya roleintimingintask3: We foundacorrelationbetweenre-portedimage clarityandthe total time takenbypartici-pants intask3(Pearsoncorrelationcoecient of 0.4050,p=0.026). Largertimevaluescorrespondedtolowerimageclarity. However, wefoundnosuchsignicantrelationshipwiththetimestakenintasks1or2, leadingustosuspectthat theparticipants responses tothenal questionnairemayhave beeninuencedprimarilybytheir performanceduring the last task. We found no other signicant relation-shipsbetweentimingresults, accuracy, andotherinforma-tiongatheredinthequestionnaires.5.3.2 Mechanical Turk User StudyPerceivedcondencelevelsandimprovementindetectionofillegal imageswerecapturedbythequestionnaireattheendof theuserstudy. Wefoundnorelationshipsbetweenaccuracy, the informationgatheredinthe questionnaires,anddemographics.6. LIMITATIONSOur participant pool was composed primarily of universitystudentsandyoungerpopulationgroups. Thiscouldhaveintroducedabiasinourresultsourparticipantswerethetypethat understandbasicconcepts of augmentedrealityand secret messaging. Our results may not generalize to theentire population. Future work should examine whether ourndingswouldpersistforwiderpopulationgroups.We conducted the experiments in a dimly lit room,whichistheideal lightingforperceivingbrightwhitelightintheARDas occludingthe background. Stronglylit environ-mentscouldpossiblyinterferewiththeusersrecognitionofthedecodedplaintext. As theaugmentedrealitycommu-nityovercomesthischallenge, lightingshouldnolongerbealimitingfactorfortheuseofVCwithARDs.Duetohardwarelimitations,wewereunabletodoimagestabilization for reducing head jitter,and so the formal userstudywasconductedusingachinrestinordertominimizeheadjitter. Thoughitdoesnotcompletelyeliminatejitter,the chinrest seemedtosubstantiallyimprove the ease ofaligningvisual shares. Assuch, webelievethatheadjitterisachallengethatcanbeovercomewithaugmentedreal-itydisplays(ARDs)thatincludeheadtrackingandimagestabilization[27].The amount of informationconveyedper image is lim-ited. Oureortstoincreasetheblockdensityoftheimageswereconstrainedbythesmalleldofviewandlimitedres-olutionof bothof thedevices. Witha9increaseineldofview, fromGoogleGlasstotheEpsonMoverio, wewereable to increase the block density and convey individual let-ters andnumbers. As ARDs improvetheir elds of view,weexpectuserstobeabletodecodemultiplecharacterssi-multaneously, which would improve the bandwidth at whichmulti-character messages couldbe decoded. That partic-ipants investedamedianof roughly8seconds todecodeand recognize a plaintext character in our formal user studysuggeststhatsuchadvances(andusertraining)willbenec-essaryformessagingviaourapproachtobecomepracticalforanybutthemostsensitivemessages.7. CONCLUSION AND FUTURE WORKUsingone-timepadsforhighlysensitivecommunications(e.g., inintelligenceordiplomaticcontexts)hasalonghis-tory. Inthispaperwesoughttomodernizethistechniqueby coupling visual cryptography (VC) with augmented real-itydisplays(ARDs). Specically,weexploredhowVCandARDscanbeleveragedtoenableausertoreceiveasecretmessage without revealing that message to her ARD (or anyotherdeviceactinginisolation). Ourevaluationfocusedonthe ability of users to decode VC-encrypted plaintexts usingtheirARDs. Throughaninitialpilotstudyandsubsequentformal study, we demonstrated that users were able to eec-tively leverage VC via ARDs to decode a single character atatime, providedthatheadjittercanbeaddressedwhichwe didusingachinrest inour secondstudy, but whichshouldbeaddressableusingimagestabilizationandthatthe block density was not too high. Future studies involvingARDswithimagestabilizationwouldbeuseful.We also measured the ability of users to detect active mod-icationofaVCsharebyanadversary,bothinourformaluserstudy(bydetectingnonconformingblocksindecodedplaintexts) andinaMechanical Turkstudy(bydetectingmalformedblocks inimage shares). Our studies showedthatdetectingsuchactiveattacksisfeasibleformostusers,thoughnotperfectly. Toourknowledge,wearethersttoverify the capabilities that are assumed of the human visualsystembypastworkonmodicationdetectioninVC.ThelimitedbandwidthofVCusingARDsunfortunatelywould appear to limit its use to only the most sensitive con-texts. However, there are various other VC schemes (e.g., [2,3,7,14,33])thatmightoerdierentusabilitycharacteris-ticswhenusedwithARDs. Exploringthesepossibilities,aswell as new devices such as HoloLens and augmented-realitycontactlenses, appeartobefruitful directionsfornewre-search.12USENIX Association 2015 Symposium on Usable Privacy and Security101AcknowledgementsWearegratefultoProf.HenryFuchsforusefuldiscussionsandforlendingustheEpsonMoveriodevice. Weexpressour gratitude to Jim Mahaney for helping build the chin restforourstudies. Wearealsograteful toAndrewMaimone,True Price, andKishore Rathinavel for useful discussionsandtheirassistancewithdocumentingthestudies.8. REFERENCES[1] AmericanFoundationfortheBlind.BrailleAlphabetandNumbers.http://braillebug.afb.org/braille_print.asp/.[2] G.Ateniese,C.Blundo,A.DeSantis,andD.R.Stinson.Extendedcapabilitiesforvisualcryptography.Theoretical ComputerScience,250(1):143161,2001.[3] B.Borchert.Segment-basedvisualcryptography.2007.[4] L.D.BrownandH.Hua.Magiclensesforaugmentedvirtualenvironments.ComputerGraphicsandApplications,IEEE,26(4):6473,2006.[5] T.P.CaudellandD.W.Mizell.Augmentedreality:Anapplicationofheads-updisplaytechnologytomanualmanufacturingprocesses.InSystemSciences,1992.Proceedingsofthe25thHawaiiInternationalConferenceon,volume2,pages659669.IEEE,1992.[6] Y.-C.Chen,D.-S.Tsai,andG.Horng.Anewauthenticationbasedcheatingpreventionschemeinnaorshamirsvisualcryptography.Journal ofVisualCommunicationandImageRepresentation,23(8):12251233,2012.[7] S.Cimato,A.DeSantis,A.L.Ferrara,andB.Masucci.Idealcontrastvisualcryptographyschemeswithreversing.InformationProcessingLetters,93(4):199206,2005.[8] P.Daponte,L.DeVito,F.Picariello,andM.Riccio.Stateoftheartandfuturedevelopmentsoftheaugmentedrealityformeasurementapplications.Measurement,57:5370,2014.[9] R.DePriscoandA.DeSantis.Cheatingimmunethresholdvisualsecretsharing.TheComputerJournal,53(9):14851496,2010.[10] Epson.EpsonMoverioBT100.http://www.epson.com/cgi-bin/Store/jsp/Product.do?sku=V11H423020.[11] B.Furht.Handbookofaugmentedreality,volume71.Springer,2011.[12] S.Gibbs.GoogleGlassreview: useful-butoverpricedandsociallyawkward.http://www.theguardian.com/technology/2014/dec/03/google-glass-review-curiously-useful-overpriced-socially-awkward,Dec.2014.[13] Google.GoogleGlass.http://support.google.com/glass/answer/3064128?hl=en&ref_topic=3063354.[14] Y.-C.Hou.Visualcryptographyforcolorimages.PatternRecognition,36(7):16191629,2003.[15] C.-M.HuandW.-G.Tzeng.Cheatingpreventioninvisualcryptography.ImageProcessing,IEEETransactionson,16(1):3645,2007.[16] D.Kahn.TheCodebreakers: TheStoryofSecretWriting.TheNewAmericanLibrary,Inc.,NewYork,NY,1973.[17] KasperskyLabsGlobalResearch&AnalysisTeam.Equation: Thedeathstarofthemalwaregalaxy.http://securelist.com/blog/research/68750/equation-the-death-star-of-malware-galaxy/,Feb.2015.[18] B.Laxton,K.Wang,andS.Savage.Reconsideringphysicalkeysecrecy: Teleduplicationviaopticaldecoding.In15thACMConferenceonComputerandCommunicationsSecurity,Oct.2008.[19] M.Liedtke.Review: 1stpeekthroughGoogleGlassimpresses.http://www.scmp.com/lifestyle/technology/article/1295459/review-1st-peek-through-google-glass-impresses,Aug.2013.[20] M.A.Livingston,J.L.Gabbard,J.E.SwanII,C.M.Sibley,andJ.H.Barrow.Basicperceptioninhead-wornaugmentedrealitydisplays.InHumanFactorsinAugmentedRealityEnvironments,pages3565.Springer,2013.[21] A.Maimone,D.Lanman,K.Rathinavel,K.Keller,D.Luebke,andH.Fuchs.Pinlightdisplays: wideeldofviewaugmentedrealityeyeglassesusingdefocusedpointlightsources.InACMSIGGRAPH2014EmergingTechnologies,page20.ACM,2014.[22] Mandiant.APT1: ExposingoneofChinascyberespionageunits.http://intelreport.mandiant.com/,2013.[23] Microsoft.MicrosoftHoloLens.http://www.microsoft.com/microsoft-hololens/en-us.[24] P.Milgram,H.Takemura,A.Utsumi,andF.Kishino.Augmentedreality: Aclassofdisplaysonthereality-virtualitycontinuum.InPhotonicsforIndustrial Applications,pages282292.InternationalSocietyforOpticsandPhotonics,1995.[25] M.NaorandB.Pinkas.Visualauthenticationandidentication.InAdvancesinCryptologyCRYPTO97,pages322336.Springer,1997.[26] M.NaorandA.Shamir.Visualcryptography.InAdvancesinCryptologyEUROCRYPT94,pages112.Springer,1995.[27] NelsDzyre.TenForthcomingAugmentedRealityandSmartGlassesYouCanBuy.http://www.hongkiat.com/blog/augmented-reality-smart-glasses/.[28] Optivent.AugmentedRealityHMDBenchmarks.http://optinvent.com/HUD-HMD-benchmark#benchmarkTable.[29] M.J.Ranum.One-time-pad(Vernamscipher)frequentlyaskedquestions.http://www.ranum.com/security/computer_security/papers/otp-faq/,1995.[30] M.Schwartz.WhocancontrolN.S.A.surveillance?TheNewYorker,Jan.2015.[31] D.-S.Tsai,T.-H.Chen,andG.Horng.Acheatingpreventionschemeforbinaryvisualcryptographywithhomogeneoussecretimages.PatternRecognition,40(8):23562366,2007.[32] P.Tuyls,T.Kevenaar,G.-J.Schrijen,T.Staring,andM.vanDijk.Visualcryptodisplaysenablingsecurecommunications.InSecurityinPervasiveComputing,pages271284.Springer,2004.131022015 Symposium on Usable Privacy and SecurityUSENIX Association[33] Z.Zhou,G.R.Arce,andG.DiCrescenzo.Halftonevisualcryptography.ImageProcessing,IEEETransactionson,15(8):24412453,2006.APPENDIXA. USER STUDY QUESTIONNAIRESThequestionnairespresentedtotheparticipantsfortheformaluserstudyusingEpsonMoverioandtheonlineMe-chanical Turkstudyarepresentedinthis section. Inthepaper we referredto modications of blocks as noncon-forming inthe formal user studyandas malformed intheonlineuserstudy. However, inthequestionnairesandwhileexplainingthetaskstotheparticipantsinthestudy,weusedthetermillegal torefertothesame. Thetermsnonconforming andmalformedareusedinthetexttodis-tinguish between the blocks as seen in the decoded plaintextandblocksasseeninoneimageshare.A.1 Formal User Study QuestionnairePleaseanswerthefollowingquestions1. My condence in my responses as I progressed throughthetasksbecame:HigherSomewhatHigherNeitherHighernorLowerSomewhatLowerLower2. AsIprogressedthroughthestudy,myabilitytoiden-tifycharacters/numbersbecame:BetterSomewhatBetterNeitherBetternorWorseSomewhatWorseWorse3. As I progressed through the study, my ability to detectillegalimagesbecame:BetterSomewhatBetterNeitherBetternorWorseSomewhatWorseWorse4. Theclarityoftheoverlaidimageswas:ClearSomewhatClearNeitherClearnorBlurredSomewhatBlurredBlurred5. Had you used any AR technology before taking part inthisstudy?Yes No6. Howwouldyoudescribeyourinteractionwithtechnol-ogyinyourday-to-daylife?Low Moderate High7. Doyouwearprescriptionglasses?Yes No(a) Ifyes,wearingprescriptionglassesmadeusingtheARglasses:EasySomewhatEasyNeitherEasynorDicultSomewhatDicultDicult(b) Wearing prescription glasses made aligning the im-ages:EasySomewhatEasyNeitherEasynorDicultSomewhatDicultDicult8. Gender:Female Male Other9. AgeRange:18-2526-3536-4546-5556andaboveA.2 Mechanical Turk QuestionnairePleaseanswerthefollowingquestions:1. My condence in my responses as I progressed throughthetasksbecame:HigherSomewhatHigherNeitherHighernorLowerSomewhatLowerLower2. As I progressed through the study, my ability to detectillegalimagesbecame:BetterSomewhatBetterNeitherBetternorWorseSomewhatWorseWorse3. Howwouldyoudescribeyourinteractionwithtechnol-ogyinyourday-to-daylife?Low Moderate High4. Pleaseselectyourgender:Female Male Other5. Pleaseselectyouragerange:18-2526-3536-4546-5556andabove14