motion sensors + audio visual performance: motion sensors + audio visual performance: using kinect...
TRANSCRIPT
![Page 1: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/1.jpg)
MotionMotionSensors Sensors + Audio Visual + Audio Visual Performance:Performance: Using Kinect to Control Live Visuals and Sound
by Joey Bargsten, Ph.D. Florida Atlantic University • SCMS • MTEn
MotionMotionSensors Sensors + Audio Visual + Audio Visual Performance:Performance: Using Kinect to Control Live Visuals and Sound
by Joey Bargsten, Ph.D. Florida Atlantic University • SCMS • MTEn
![Page 2: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/2.jpg)
Thanks!
http://film-gate.org
http://sweatrecordsmiami.com
![Page 3: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/3.jpg)
Format/Overview:
• Presentation / Demo (noon – 12:40)
• Questions? (12:40 – 12:50)
• Break (10 min)
• Workshop on PureData (1pm – 2pm)
![Page 4: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/4.jpg)
Background on Kinect
![Page 5: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/5.jpg)
Background on Kinect
• Released November 2010
![Page 6: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/6.jpg)
Background on Kinect
• Released November 2010
• Sold 8M units in 60 days
![Page 7: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/7.jpg)
Background on Kinect
• Released November 2010
• Sold 8M units in 60 days
• SDK Drama!
![Page 8: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/8.jpg)
Kinect (Development)Community• Gaming and Virtuality
![Page 9: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/9.jpg)
Kinect (Development)Community• Gaming and Virtuality
• Medicine
![Page 10: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/10.jpg)
Kinect (Development)Community
• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
![Page 11: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/11.jpg)
Kinect (Development)Community• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
• Robotics
![Page 12: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/12.jpg)
Kinect (Development)Community• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
• Robotics
• 3D Scanning and Printing
![Page 13: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/13.jpg)
Kinect (Development)Community
• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
• Robotics
• 3D Scanning and Printing
• Avant-Garde Art, Music, and Performance
![Page 14: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/14.jpg)
Suggestions,ApproachesSystems & Configurations
![Page 15: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/15.jpg)
My Configuration
• Kinect Sensor (2011); Model 1414, EN/ES US HD
(this is the most ‘open’ one!)
• Synapse Middleware (converts skeleton to OSC)
• PureData v. 0.43.4-Extended (incl. MrPeach library)
• Quartz Composer v. 4.0 (+ old FW400 SD
camcorder)
• 2011 MBP, 2.4mHz, running OSX 7.5 “Lion”.
![Page 16: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/16.jpg)
Approaches to Performance• One-Person Band (virtuoso!)
– One person, one controller
– Controls both audio and visual
Video on YouTubehttp://www.youtube.com/watch?v=1ge7RcIBWsc
![Page 17: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/17.jpg)
Approaches to Performance• Division of Labor (conventional
ensembles)
![Page 18: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/18.jpg)
Approaches to Performance• Division of Labor (conventional
ensembles)
“For every new technology that comes along,
make an ensemble out of it!”—JB, EMV (2011)
![Page 19: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/19.jpg)
Approaches to Performance• Division of Labor (conventional
ensembles)
“For every new technology that comes along,
make an ensemble out of it!”—JB, EMV (2011)
• Applied to Kinect:–Multiple controllers –Dedicate separate software to audio and visuals–Multiple performers for audio and visuals
![Page 20: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/20.jpg)
What Do You Want to Control?
![Page 21: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/21.jpg)
How Do You Want To Control It?
• Discontinuous, trigger events
–model: keyboard
• Continuous, gradual events
–model: theremin
• Hybrid Control: multiple parameters
–model: sample player
![Page 22: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/22.jpg)
Kinect Control Parameters
![Page 23: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/23.jpg)
Kinect Kombo™:Quad Theremin + Sample Player
• Built in PureData • Free, cc-by-d-nc
![Page 24: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/24.jpg)
Kinect Kombo™:Skeleton View + Performer Interface
![Page 25: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/25.jpg)
Kinect Kombo™
![Page 26: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/26.jpg)
Kinect Kombo™
![Page 27: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/27.jpg)
Controlling Visuals
Kinect and
Quartz
Composer
![Page 28: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/28.jpg)
TheAugmentedDance:
LiveBody—Kinect— VirtualBody
![Page 29: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/29.jpg)
Variations• Figure
– Simple Camera Figure and Skeleton
– Primitives, 3D Objects, Video Mapped to
Skeleton
• Paint Brush/Image Hose (Effects Mapped to
Skeleton)
• Background (Texture or Stylized Video Controlled by Skeleton)
![Page 30: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/30.jpg)
“My Profundis (2012)”
Video on YouTube http://www.youtube.com/watch?v=gT1lbhR1Qj8
![Page 31: Motion Sensors + Audio Visual Performance: Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten,](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649e3b5503460f94b2de75/html5/thumbnails/31.jpg)
Resources
• OpenKinect.org
• OpenNI.org
• PureData.info
• Synapsekinect.tumblr.co
m
• CreateDigitalMusic.com
• CreateDigitalMotion.com
• V002.info (vade &
bangnoise)
• Cycling74.com
• IdeaChannel ‘Kinect’
Episode
• OffBook ‘Creative Coding’
Episode
• www.fau.edu/scms (MFA -
MTEn)
• www.badmindtime.com
• Melancholalaland™ (my
multimedia opera)