university of dundee ace-lp black, rolf; kristensson, per ... · christopher norrie1, sophia bano1,...

University of Dundee

ACE-LP

Black, Rolf; Kristensson, Per Ola; Zhang, Jianguo; Waller, Annalu; Bano, Sophia; Rashid,Zulqarnain; Norrie, Christopher

Publication date:2016

Link to publication in Discovery Research Portal

Citation for published version (APA):Black, R., Kristensson, P. O., Zhang, J., Waller, A., Bano, S., Rashid, Z., & Norrie, C. (2016). ACE-LP:Augmenting Communication using Environmental Data to drive Language Prediction. Poster session presentedat Communication Matters - CM2016 National Conference, Leeds, United Kingdom.

General rightsCopyright and moral rights for the publications made accessible in Discovery Research Portal are retained by the authors and/or othercopyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated withthese rights.

• Users may download and print one copy of any publication from Discovery Research Portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain. • You may freely distribute the URL identifying the publication in the public portal.

Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.

Download date: 01. Jul. 2017

Text-basedSpeechGeneratingDevices(SGDs):• Keyboard:directaccess/scanning;• Encoding,expansion,prediction,disambiguation.

University of DundeeAugmenting Communication using Environmental Data to drive Language Prediction

Rolf Black1, Stephen McKenna1, Jianguo Zhang1, Per Ola Kristensson2, Christopher Norrie1, Sophia Bano1, Zulqarnain Rashid1, Annalu Waller1

1University of Dundee, UK, 2University of Cambridge, UK

References ContactDundeeAACResearchGroup,ProfAnnaluWallerComputingintheSchoolofScienceandEngineeringUniversityofDundee,DundeeDD14HN,ScotlandEmail:a.waller@dundee.ac.uk,Projectwebsite:http://ACE-LP.ac.uk

AcknowledgementsThisresearchisfundedbytheEngineeringandPhysicalSciencesResearchCouncil(EPSRC)oftheUKResearchCouncils.GrantReference:EP/N014278/1

Current AAC TechnologyUseofConversationalLanguageModels:• Handcraftedcontextualconversationalitems;• Stagesofconversation(e.g.greetings,farewells);• Data-to-textsentencegeneratorsfornarrativebasedsystems.

UseofSensorData:• Contextprediction;• Storygeneration.

Wordandphraseprediction,on-screenkeyboard,directorscanningswitchaccess.claro-apps.com

Disambiguationwithdirectaccessthrougheyegazewithdwell-freetyping.tobiidynavox.com

T.A.L.K.withhandcraftedconversationalitems.Todman etal.1994

Encoding(SemanticCompaction)withdirectorscanningswitchaccess.minspeak.com

Data-to-textforautomaticphrasegenerationbasedonpersonalexperiencedata.Dempster etal.2010

LocationcontextdeterminationusingGPSandWifi oniPhone;myvoiceaac.com

Limitations:

• Scanningwordlists(prediction)• Accessmethodneedstraining(disambiguation)• Limitedvocabulary(encoding)• Needtoremembertheexistenceandlocationofconversationalitems(encoding,pre-stored);

• Nottimelyorfitting(script,pre-stored)

Our Aim

ProposedSystem

Integratedcontextual,adaptiveSGDforindividualswithphysicaldisabilities:• Useofcomputervisionandcontextualdatacollectiontoinformprediction;

• Developmentofanewuserinterfaceforaccesstopredictedwords,phrasesandstories.

Possiblethrough:• Recentadvancesinenvironmentaldatacollection;

• Abilitytoprocesslargeamountsofdatainreal-time;

• Probabilisticlanguagemodellingandtheavailabilityofmobileplatforms.

Scriptbasedutterancesystemforsituationalconversations.McCoyetal.2010

Egocentricvideodataforcontextrelevantvocabularyidentificationandprovision.Shamdani andPeña 2015

Partners

Dempster M,NAlm andEReiter(2010).AutomaticgenerationofconversationalutterancesandnarrativeforAugmentativeandAlternativeCommunication:aprototypesystem.SLPATWorkshop,LosAngeles,USA.

McCoyKF,JBedrosian andLHoag(2010).ImplicationsofPragmaticandCognitiveTheoriesontheDesignofUtterance-BasedAACSystems.SLPATWorkshop,LosAngeles,USA.

Shamdani SandAPeña(2015).Verse:Contextualaugmentedrealitycommunicationaid.Communicaiton MattersConference,Leeds,UK.

Todman J,NAlm andLElder(1994).Computer-aidedconversation:aprototypesystemfornonspeakingpeoplewithphysicaldisabilities.AppliedPsycholinguistics.

Ouraimistodevelopacommunicationsystemfornonspeakingpeoplewhichwillimproveonthespeedofcommunicationbyautomaticallypopulatingthesystemwithappropriateconversationalitemswithin

Speechratecomparison:Naturalspeech(top),AAC:average(middle),switchaccesswithacceleration,example(bottom)

150words/minute

12w/min

2 w/min

anadaptiveinterfacethatprovidescontrolovertiminganddeliverywhilstminimisingphysicalandcognitiveload.