picto vision - using image recognition to turn sketches into communication

David Wright [email protected] All Rights Reserved 2014

PICTOVISION

mailto:[email protected]

PictoVision turns touch into a new visual communication language.

It is a service that allows anyone to create compelling and expressive graphics – simply by doodling their ideas on a touch screen.

The system takes the sketch data (and motion /touch pressure information) and then renders rich and detailed images and animations. This is important for wearable devices in particular – as there is little room for a keyboard and voice interaction is often not appropriate.

Example:


PICTOVISION |What


Pictovision can also enable – sketch based short-cuts – which look up a users previous sketches and render routines.

PICTOVISION |Touch based shortcuts


Pressure sensitive screensPictoVision – records the speed and pressure (if this data is available). Pressure can be used to add depth.

Hard=distance Soft=close Swipes=many randomly distributed


[email protected]

All objects have properties – which can allow for animation.

A finger swoosh – animates the comet!

Animate the object through touch –swoosh!

Note: The system tries to match the position, size and orientation of your sketch objects.

All objects the system recognizes has an associated taxonomy (based on Natural Language tech)which provides for:• Colour Palette• Animation - movement

PICTOVISION |Depth

moon

rain

PictoVision provides user-feedback via image icons to help the user where there is ambiguity – i.e. is it a ball, the moon, the earth.

Auto-complete can also provide actions/options for the images – based on the previously described object taxonomy.

??

mountains

PICTOVISION | Auto-complete for images



moon

rain

The system then finds suitable real-world images (stored in memory) and merges them seamlessly using image blending routines.

As the system learns – and finds patterns –the image stitching will increase in both speed and efficiency across the population using it. The system also increases efficiency as it builds up a user profile.

Layered image data is sent back to the connected device where it is rendered for further user on-device editing (i.e. animating, positioning, colour filters).

Via deep learning run on a array of GPU’s - PictoVision uses rapid computer vision techniques to identify the objects in the doodle.

PICTOVISION |The Technology



PICTOVISION |As a service


API/SDK

EMBEDED

E-Signatures

SocialKeyboard

Enhancements

DigitalWhiteboards

Messengers

APPs

PICTOVISION


PICTOVISION |Pilot Service I – Communication Aid


“busy happy noodle restaurant”


Susan Bob

Sam Sarah

To tune and promote the service a simple APP could be built that lets users share their sketches (as the sketch/image) as status update.

The app sorts and presents (sorted by latest and most “proded”) the updates in a single image wall.

The pilot will also increase the amount of data contained in the training/machine learning database – making the sketch recognition system ever more accurate and faster.

Via machine learning/profiling the system will also become more efficient and deciding which objects should be stored on an individual users device.

Note: that existing sketch databases already exist so would not be starting from scratch.

Also note reference databases exist for objects and scenes which would form the basis of the systems content.

PICTOVISION |Pilot Service II



PICTOVISION |Pilot II - Social Media Fun


moon


Machine Learning Layer

Image Recognition- Sketch “fingerprint” sent to cloud

for service for matching- *Note sketches are only ~1KB

(tiny)

Image RenderingFind ObjectsBlend – via image recognitionBackground subtraction- Send back layered image – including object data.- Auto-complete data + icons.

PICTOVISION |Architecture


On device memoryReference similar shapes – most common shapesSuggestionsDownloaded FingerprintsFiltersUI – for manipulating images


Business Model

1) SDK for Telco's and OEM's to enhance their messaging platforms. For instance - it would be great in the wearable space where there is no real opportunity for text entry. Charging would be based on volume caps. The benefits to the Telco or OEM would be one - an amazing messaging service and two – a source of meta-data that would be useful to profile users for advertising.

2) SDK - could be quite broad reaching. For instance - education - it would be a great creativity tool - sketch a river and a rainbow - voila - there it is (via a digital whiteboard etc..). As above there would be a volume charge. For education –specific image sets could be compiled so the software runs locally. For instance – the school could buy an astronomy or science pack.

3) Special needs education - communicating visually - overcomes many communications barriers - creating a language independent communication medium. For instance people smiling or dancing - means happiness anywhere on the planet.

So the proposed approach would be 1 and 2 - going through the usual OEM, Telco sales channels.

Pilot stage – release a variety of messaging apps (utilizing Facebook’s social graph for instance). This phase will tune both the technology and the user experience. It will also be used answer questions such as “what is the right mix of cloud + on devicestorage” to ensure the system is lightening fast.

Once the system has been “tuned” in a live environment and training databases full!– the technology lends itself well to publicity.

For instance – a public installation.

Go to market

PICTOVISION

