using intel's real sense to create games with natural user interfaces justin link
TRANSCRIPT
Using Intel’s RealSense to create games with Natural User Interfaces
Justin Link, Chronosapien Interactive
Overview
RealSense Overview• Camera types• System requirements• SDK features• Getting data• Unity toolkit
Our Use Cases• Getting the data• Challenges• What we did
Space Between
The Risen
Photo booth
RealSense Overview
Camera Types
F200 R200 SR300
RealSense Overview
System Requirements
• 4th generation (or later) Intel® Core™ processor• 8GB free hard disk space• Microsoft Windows* 8.1 or 10 OS 64-bit
RealSense Overview
SDK Features• Hand tracking/gesture recognition• Face tracking/recognition• 3D Scanning• Background removal (user segmentation)• Object tracking• Speech recognition
RealSense Overview
Getting Data• The first step for getting any data from the RealSense SDK is creating a
session instance:
https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?pxcsession.html
RealSense Overview
Getting Data• Next you can create an instance of the module you want to use and
(optionally) configure it:
https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_hand_initializing_and_configuring.html
RealSense Overview
Getting Data• Once the module is created and set up, you can poll data:
https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_hand_implementing_the_main_loop.html
RealSense Overview
The Unity Toolkit
• Easy to use• Quick to set up• Covers a lot, but has limited functionality compared to the SDK• Great for prototyping
https://www.youtube.com/watch?v=SK50ncrYq4w
Hand tracking in space between
Hand Tracking in Space Between
What is Space Between?• Underwater themed game that lets you play as five different sea
creatures or machines• Uses RealSense to let players control characters with their hands and
head• Unique input for each character
Hand Tracking in Space Between
Getting Hand Data from RealSense
Let’s check out some of our code
Hand Tracking in Space Between
Challenges
User understanding of the technology• Tracking has bounds• Tracking requires a line of sight• Tracking is based on silhouettes
Hand Tracking in Space Between
Challenges
“Gorilla Arm”, or Consumed Endurance• Constant motion fatigues users’ arms after
about 60 seconds• Might be ok for exercise, but isn’t that fun
Hand Tracking in Space Between
Challenges
Data is noisy• Can create false positives• Limits precise actions• Can look visually unappealing
Hand Tracking in Space Between
What We Did• Cursor for every game to always show the tracked hand/head position• Game instructions to tell players what is being tracked and how it’s
used• Depth feed of the player during gameplay• Instructions/tutorials accessible at all times
Hand Tracking in Space Between
What We Did
Let’s jump into Space Between
Voice Recognition in The Risen
Voice Recognition in The Risen
What is The Risen?• A tech demo we created for the RealSense App Challenge• Let’s players play as a skeleton lord who commands skeletons with
their voice• Let’s players be IN the game via user segmentation• Let’s players use their hands to help their skeleton minions
Voice Recognition in The Risen
Getting Speech Data from RealSense
Let’s check out some of our code
Voice Recognition in The Risen
Challenges
Recognition is based on the player’s environment
• Loud environments kill recognition• Quiet environments may prevent talking
Voice Recognition in The Risen
Challenges
Recognition is slower than players’ expectations• RealSense uses a local solution• Expectations are based on cloud
solutions
Voice Recognition in The Risen
Challenges
Recognition is often inaccurate• False positives• True negatives
Voice Recognition in The Risen
What We Did• Step by step tutorial• UI feedback for when speech was detected• UI feedback for speech too loud or too quiet• Microphone volume slider
Voice Recognition in The Risen
What We Did
Let’s check out The Risen
User Segmentation in our Photo booth
User Segmentation in our Photo booth
What is The Photo booth?• A photo booth that puts users into a virtual environment without the
need for green screen• Uses gestures to control the app• Has print and share functionality
User Segmentation in our Photo booth
Getting Segmentation Data from RealSense
Let’s check out some of our code
User Segmentation in our Photo booth
Challenges
Segmentation quality is highly dependent on lighting• Be careful with external IR sources• Dark environments create grainy color
feeds
User Segmentation in our Photo booth
Challenges
Segmentation often cuts off parts of the user• This is a result of the method of getting a
depth image• Hair almost always gets lost• Reflective materials can be invisible
User Segmentation in our Photo booth
Challenges
Segmentation only supports a single user• This is mostly true, but sometimes
people can trick the software into thinking they are one “blob”
User Segmentation in our Photo booth
Challenges
Color/lighting of the user often does not match the virtual environment• Not necessarily a fault of the SDK, but
still a challenge when making an app
User Segmentation in our Photo booth
What We Did• Used bright lights to illuminate the user• Used curtains/walls to block external light sources• Used hats to hide chopping users’ hair off• Used a plugin to control segmentation color, brightness, and contrast
Voice Recognition in The Risen
What We Did
Let’s jump into the photo booth
Wrapping Up
Take Away’s• Users must be trained to understand how the camera works• User feedback is critical to make up for the loss of haptic feedback• Giving users control over the camera’s properties can improve
recognition and overall UX• This is new tech – it has many quirks and working through them will
be a big part of the work you do
Wrapping Up
References• RealSense SDK download page - https://software.intel.com/en-us/intel-realsense-sdk/download• RealSense Documentation - https://software.intel.com/sites/landingpage/realsense/camera-
sdk/v1.1/documentation/html/index.html• User Feedback for NUI’s with RealSense P1 - https://software.intel.com/en-us/articles/user-
feedback-for-natural-user-interfaces-with-intel-realsense-technology-part-1• User Feedback for NUI’s with RealSense P2 - https://software.intel.com/en-us/articles/user-
feedback-for-natural-user-interfaces-with-intel-realsense-technology-part-2• “Gorilla Arm”, or Consumed Endurance -
https://pdfs.semanticscholar.org/d852/e9749467b06f6cd516ea3f6c894408dea6d4.pdf• Space Between on the RealSense App Showcase - https://appshowcase.intel.com/en-
us/realsense/space-between?cam=f200
Wrapping Up
Contact• http://www.chronosapien.com/• https://www.reddit.com/r/chronosapien• @jlink__ + @_chronosapien• [email protected]• [email protected]
Thanks for Watching!
Time for QA!