yingcai xiao
DESCRIPTION
Interactive Visualization with NUI and Game Engines. Yingcai Xiao. Interactive Visualization. Allow the user to control how data are represented as graphics and the graphics are viewed. Human-Computer Interaction (HCI) is critical to Interactive Visualization (IV). HCI. - PowerPoint PPT PresentationTRANSCRIPT
Yingcai Xiao
Interactive Visualizationwith
NUI and Game Engines
Interactive VisualizationInteractive Visualization
Allow the user to control how data are Allow the user to control how data are represented as graphics and the graphics represented as graphics and the graphics are viewed. are viewed.
Human-Computer Interaction (HCI) is critical to Interactive Visualization (IV).
HCIHCI
Three types of HCI:
CLI: command line interface (with keyboard)
GUI: graphical user interface (mouse)
NUI: natural user interface with A/V (Kinect)
NUI:NUI:
Three parts of NUI:
Hardware: e.g., Kinect
Software: drivers (OpenUI), middleware
Application: integration of HW enabling software with applications.
OpenNIOpenNI
OpenNIOpenNIProduction Nodes:
• a set of components that have a productive role in the data creation process required for Natural Interaction based applications.
• the API of the production nodes only defines the language.
• The logic of data generation must be implemented by the modules that plug into OpenNI.
• E.g. for a production node that represents the functionality of generating hand-point data, the logic of hand-point data generation must come from an external middleware component that is both plugged into OpenNI, and also has the knowledge of how to produce such data.
Production Nodes:
• a set of components that have a productive role in the data creation process required for Natural Interaction based applications.
• the API of the production nodes only defines the language.
• The logic of data generation must be implemented by the modules that plug into OpenNI.
• E.g. for a production node that represents the functionality of generating hand-point data, the logic of hand-point data generation must come from an external middleware component that is both plugged into OpenNI, and also has the knowledge of how to produce such data.
OpenNI OpenNI
(1) body imaging (2) joint recognition (3) hand waving
OpenNI: Sensor-Related Production NodesOpenNI: Sensor-Related Production Nodes
Device: represents a physical device (a depth sensor, or an RGB camera). Its main role is to enable device configuration.
Depth Generator: generates a depth-map. Must be implemented by any 3D sensor that wishes to be certified as OpenNI compliant.
Image Generator: generates colored image-maps. Must be implemented by any color sensor that wishes to be certified as OpenNI compliant
IR Generator: generates IR image-maps. Must be implemented by any IR sensor that wishes to be certified as OpenNI compliant.
Audio Generator: generates an audio stream. Must be implemented by any audio device that wishes to be certified as OpenNI compliant.
OpenNI: Middleware-Related Production NodesOpenNI: Middleware-Related Production Nodes
Gestures Alert Generator: Generates callbacks to the application when specific gestures are identified.
Scene Analyzer: Analyzes a scene, including the separation of the foreground from the background, identification of figures in the scene, and detection of the floor plane. The Scene Analyzer’s main output is a labeled depth map, in which each pixel holds a label that states whether it represents a figure, or it is part of the background.
Hand Point Generator: Supports hand detection and tracking. This node generates callbacks that provide alerts when a hand point (meaning, a palm) is detected, and when a hand point currently being tracked, changes its location.
User Generator: Generates a representation of a (full or partial) body in the 3D scene.
OpenNI: Recording Production NotesOpenNI: Recording Production Notes
Recorder: Implements data recordings
Player: Reads data from a recording and plays it
Codec: Used to compress and decompress data in recordings
OpenNI: CapabilitiesOpenNI: Capabilities
Supports the registration of multiple middleware components and devices. OpenNI is released with a specific set of capabilities, with the option of adding further capabilities in the future. Each module can declare the capabilities it supports.
Currently supported capabilities:
Alternative View: Enables any type of map generator to transform its data to appear as if the sensor is placed in another location.
Cropping: Enables a map generator to output a selected area of the frame.
Frame Sync: Enables two sensors producing frame data (for example, depth and image) to synchronize their frames so that they arrive at the same time.
OpenNI: CapabilitiesOpenNI: Capabilities
Currently supported capabilities:
Mirror: Enables mirroring of the data produced by a generator.
Pose Detection: Enables a user generator to recognize when the user is posed in a specific position.
Skeleton: Enables a user generator to output the skeletal data of the user. This data includes the location of the skeletal joints, the ability to track skeleton positions and the user calibration capabilities.
User Position: Enables a Depth Generator to optimize the output depth map that is generated for a specific area of the scene.
OpenNI: CapabilitiesOpenNI: Capabilities
Currently supported capabilities:
Error State: Enables a node to report that it is in "Error" status, meaning that on a practical level, the node may not function properly.
Lock Aware: Enables a node to be locked outside the context boundary.
Hand Touching FOV Edge: Alert when the hand point reaches the boundaries of the field of view.
OpenNI: Generating and Reading DataOpenNI: Generating and Reading Data
• Production nodes that also produce data are called Generator.
• Once these are created, they do not immediately start generating data, to enable the application to set the required configuration.
• The xn::Generator::StartGenerating() function is used to begin generating data.
• The xn::Generator::StopGenerating stops it.
• Data Generators "hide" new data internally, until explicitly requested to expose the most updated data to the application, using the UpdateData request function.
• OpenNI enables the application to wait for new data to be available, and then update it using the xn::Generator::WaitAndUpdateData() function.
Interactive Visualization Interactive Visualization with awith a
Game EngineGame Engine
Video GameVideo Game
Interactive animation: Interactive animation:
user->user->interface -> interface ->
game object action -> game object action -> feedback (A/V, haptic)feedback (A/V, haptic)
Game objects can represent data.Game objects can represent data.
Video GameVideo Game
UserUser
ControllerController
DisplayDisplay
Game (Software)
Game (Software)
Video GameVideo Game
Input Device Driver
Input Device Driver
DisplayDeviceDriver (GDI)
DisplayDeviceDriver (GDI)
Game (Software)
Game (Software)
Software for Kinect-based game developmentSoftware for Kinect-based game development
OpenNI: a general-purpose framework for obtaining data from 3D sensors
SensorKinect: the driver for interfacing with the Microsoft Kinect
NITE: a skeleton-tracking and gesture-recognition library
Unity3D: a game engine
ZigFu: Unity Package for Kinect (Assets and Scripts)
OpenNI: a general-purpose framework for obtaining data from 3D sensors
SensorKinect: the driver for interfacing with the Microsoft Kinect
NITE: a skeleton-tracking and gesture-recognition library
Unity3D: a game engine
ZigFu: Unity Package for Kinect (Assets and Scripts)
Unity3DUnity3D
Engine
IDE
Assets
Tutorial
Examples
Game Assets for Interactive VisualizationGame Assets for Interactive Visualization
Game Assets: Game Objects and Animation Scripts
Game objects can be real world objects, artistic virtual objects, or data objects.
Scripts can be used to make interactive.
For interactive visualization, we just create assets. The game engine will take care everything else (including physics).
Game Assets CreatorsGame Assets Creators
Blender: blender.orgMaya: AutoDesk.com3ds Max: AutoDesk.comMotionBuilder: AutoDesk.comVisualization and Animation at AutoDesk Student CenterJmol: http://jmol.sourceforge.net/ (for visualization of chemical and biological structures)
All free for students.
Unity 3D IDEUnity 3D IDE
IDEIDE: Integrated Development Environment: Integrated Development Environment
ProjectProject: directory and files for a specific : directory and files for a specific game project.game project.
C:\Users\xiao\Documents\New Unity C:\Users\xiao\Documents\New Unity Project 1Project 1
\\AssetsAssets (anything you can reuse) (anything you can reuse)\\LibraryLibrary (binary files) (binary files)
Unity 3D: Unity 3D: AssetsAssets
C:\Users\xiao\Documents\New Unity C:\Users\xiao\Documents\New Unity Project 1\Assets (anything you can Project 1\Assets (anything you can reusereuse))
\Standard Assets\Standard Assets\OpenNI\OpenNI\Scripts \Scripts \_Scenes\_Scenes\Materials\Materials\Artwork\Artwork
Unity 3D: Unity 3D: Standard AssetsStandard Assets
C:\Users\xiao\Documents\New Unity Project 1\C:\Users\xiao\Documents\New Unity Project 1\Assets\Standard AssetsAssets\Standard AssetsObjectsObjects:: (Look) (Look)
\Tree\Tree\Terrain\Terrain\Charater\Charater
LightsLights: : (Look) (Look)\Light Flares\Light Flares\Light Cookies\Light Cookies
Code: (Feel: control, interaction, animation, …)Code: (Feel: control, interaction, animation, …)\Scripts\Scripts
Unity 3D: Unity 3D: ObjectsObjects
C:\Users\xiao\Documents\New Unity Project 1\C:\Users\xiao\Documents\New Unity Project 1\Assets\Standard Assets\Charater:Assets\Standard Assets\Charater:Prefab: (Predefined Objects)Prefab: (Predefined Objects)
First Person, 3First Person, 3rdrd Person Person\Source:\Source:
\\Prototype (Look)Prototype (Look)Constructor.FBXConstructor.FBX\Materials (properties)\Materials (properties)\Textures (images)\Textures (images)
\Scripts\Scripts (Feel: actions)(Feel: actions)Java Scripts: ThirdPersonController.jsJava Scripts: ThirdPersonController.jsC#: MouseLook.csC#: MouseLook.cs
Unity 3D: Unity 3D: ScriptsScripts
Languages:
Interpreted : Java Script
Compiled: C#
Usages:
General: under Project\Scripts
ExitOnEscape.cs
Objects: attached to objects
ThirdPersonController.js ThirdPersonController.js
Unity 3D: Unity 3D: LibraryLibrary
cashe:
for speeding up processing
metadata:
data that describes data
previews:
for previewing scenes
ScriptAssemblies:
compiled object assemblies for scripts
Unity 3D: Unity 3D: GUIGUI
•Start UnityStart Unity•File->Create ProjectFile->Create Project•Select Assets (Select Assets (Character, Lights, Scripts, Sky, Terrain, Tree)•Assets->Import Package->Custom Package
UnityOpenNIBindings-v1.4.unitypackage• File->New Scene
File->Save Scene
Unity 3D: Unity 3D: GUIGUI
•Terrain->Create Terrain•Terrain->Set Resolution: width = 300; height = 300; length = 300;•GameObject->Create Other->Directional Light•Adjust light direction to the terrain
Unity 3D: Game ObjectsUnity 3D: Game Objects
• Background Objects: Terrain and Sky•Terrain: elevation grid, adjustable height, texture, • Sky: texture, static view• Add-ons: trees, stones, …
• Foreground Objects: • Objects can be animated.
Unity 3D: Game ObjectsUnity 3D: Game Objects
• Rigid Objects: non-deformable with physical properties (gravity, inertial).
• Non-rigid Objects: • Deformable: changeable geometry• Breakable: changeable topology.
• Intangible Objects: No predefined shape.Fire, clouds, …
SummarySummary
Interaction
NUI
Kinect
Visualization
Game Object
Game Engine