augmented reality in web rtc browser
TRANSCRIPT
![Page 2: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/2.jpg)
Who am I ?
★ Btech IT degree from Anna University , Chennai
★ 4 years in telecom industry
★ Frequent contributor to open source software
★ Write at : http://altanaitelecom.wordpress.com
★ Author of book “ WebRTC Integrator’s Guide “
2
Altanai Bisht
![Page 3: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/3.jpg)
Augmented reality (AR) is viewing a real-world environment with elements that are supplemented by computer-generated sensory inputs such as sound, video, graphics , location etc.
What is Augmented Reality ?
3
![Page 4: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/4.jpg)
How is it diff. from Virtual Reality ?
Virtual Reality Augmented Reality
replaces the real world with simulated one
blending of virtual reality and real life
user is isolated from real life user interacts with real world through digital overlays
Oculus RiftKinect
Google glassHolo Lens
4
![Page 5: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/5.jpg)
Methods for rendering Augmented Reality
Computer Vision
Object Recognition
Eye Tracking
Face Detection and substitution
Emotion and gesture picker
Edge Detection5
![Page 6: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/6.jpg)
Web :
WebRTC getusermediaWeb Speech APIWebGLcss svgHTML5 canvassensor API
Components of end -to-end web based Augmented Reality solution
6
H/w :
Graphics drivermicrophone and camerasensors
3D :
Geometry and Math Utilities3D Model Loaders and models Lights, Materials,Shaders, Particles,Animation
![Page 7: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/7.jpg)
WebRTC
7
![Page 8: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/8.jpg)
8
What is WebRTC ?● Web based Real Time communications
● Definition for browser's media stream and data
● Awaiting more standardization , on a API level at the W3C and at the
protocol level at the IETF.
● Enable browser to browser applications for voice calling, video chat and
P2P file sharing without plugins.
● Enables web browsers with Real-Time Communications (RTC) capabilities
● MIT : Free, open project
![Page 9: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/9.jpg)
WebRTC implements 3 APIs:
getUserMediaRTCPeerConnectionRTCDataChannel
9
API Stack
![Page 10: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/10.jpg)
Code snippets for WebRTC
10
1.To begin with WebRTC we first need to validate that the browser has permission to access the webcam.
<video id="webcam" autoplay width="640" height="480"></video>
2. Find out if the user's browser can use the getUserMedia API.
function hasGetUserMedia() { return !!(navigator.webkitGetUserMedia); }
3. Get the stream from the user's webcam.
var video = $('#webcam')[0];if (navigator.webkitGetUserMedia) { navigator.webkitGetUserMedia(
{audio:true, video:true}, function(stream) { video.src = window.webkitURL.createObjectURL(stream); }, function(e) { alert('Webcam error!', e); }
);}
![Page 11: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/11.jpg)
AppRTC
11
Simple WebRTC API maintained by google
https://apprtc.appspot.com
![Page 12: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/12.jpg)
TFX : collaboration and communication with webrtc
12
![Page 13: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/13.jpg)
TFX : collaboration and communication with webrtc
13
![Page 14: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/14.jpg)
14
![Page 15: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/15.jpg)
15
What is WebGL?● Web Graphics Library
● JavaScript API for rendering interactive 2D and 3D computer graphics in
browser
● no plugins
● uses GPU ( Graphics Processing Unit ) acceleration
● can mix with other HTML elements
● uses the HTML5 canvas element and is accessed using Document Object
Model interfaces
● cross platform , works on all major Desktop and mobile browsers
![Page 16: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/16.jpg)
To get started you should know about :
● GLSL, the shading language used by OpenGL and WebGL
● Matrix computation to set up transformations
● Vertex buffers to hold data about vertex positions, normals, colors, and textures
● matrix math to animate shapes
Cleary WebGL is bit tough given the amount of careful coding , mapping and shading it requires .
Proceeding to some JS libraries that can make 3D easy for us .
16
WebGL development
![Page 17: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/17.jpg)
Building 3D graphics with Javascript
17
![Page 18: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/18.jpg)
JS Libraries for 3D graphics
CCV
website : http://libccv.org/
SourceCode : https://github.com/liuliu/ccv
Demo:
Three.js
website : http://threejs.org/
SourceCode : https://github.com/mrdoob/three.js/
Demo: http://www.davidscottlyons.com/threejs/
Awe.js
Website : https://buildar.com/awe/tutorials/intro_to_awe.js/index.html#
SourceCode : https://github.com/buildar/awe.js
18
ArcuCO
SourceCode: https://github.com/jcmellado/js-aruco
Potree
website: http://potree.org/wp/
SourceCode: https://github.com/potree
Demo: http://potree.org/wp/demo/
Karenpeng
emotion & gesture-based arpeggiator and synthesizer
website:
SourceCode : https://github.com/karenpeng/motionEmotion
Demo: http://motionemotion.herokuapp.com/
![Page 19: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/19.jpg)
Features● MIT license
● javascript 3D engine ie ( WebGL
+ more)
● started a year ago
● under active development
● no dependencies or installation
19
1. Set the scene , camera , renderer
2. Set the Mesh with Geometry & materials
3. Lights
4. Object properties
5. Add to scene , set camera position
6. Render with scene and camera
7. Animation
Steps
Three.js
![Page 20: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/20.jpg)
20
1. Spinning Colored Cube
2. Shaded Material on Sphere
3. Complicated materials like a bun
4. 3D on webcam input as texture
5. Motion Detection on webcam input
Lets Code !
![Page 21: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/21.jpg)
1. Get three.js from : http://threejs.org/build/three.min.js
2. Make a empty HTML5 page and import the script + basic styling of page
<html><head>
<title>Spinning colored Cube</title><style>
body { margin: 0; }canvas { width: 100%; height: 100% }
</style></head><body>
<script src="js/three.min.js"></script><script>// Our Javascript will go here. </script>
</body></html>
1. Spinning Colored Cube
21
![Page 22: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/22.jpg)
3. Scene
var scene = new THREE.Scene();
4. Camera Camera types in three.js are CubeCamera , OrthographicCamera, PerspectiveCamera. We are using Perspective camera here . Attributes are field of view , aspect ratio , near and far clipping plane.
var camera = new THREE.PerspectiveCamera( 75, window.innerWidth / window.innerHeight, 0.1, 1000
);
5. Renderer
Renderer uses a <canvas> element to display the scene to us.
var renderer = new THREE.WebGLRenderer();renderer.setSize( window.innerWidth, window.innerHeight );document.body.appendChild( renderer.domElement );
22
![Page 23: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/23.jpg)
6. BoxGeometry object contains all the points (vertices) and fill (faces) of the cube.
var geometry = new THREE.BoxGeometry( 1, 1, 1 );
7. Material
threejs has materials like - LineBasicMaterial , MeshBasicMaterial , MeshPhongMaterial , MeshLambertMaterial
These have their properties like -id, name, color , opacity , transparent etc. Use MeshBasicMaterial and color attribute of 0x00ff00, which is green.
var material = new THREE.MeshBasicMaterial( { color: 0x00ff00 } );
8. Mesh
A mesh is an object that takes a geometry, and applies a material to it, which we then can insert to our scene, and move freely around.
var cube = new THREE.Mesh( geometry, material );
9. By default, when we call scene.add(), the thing we add will be added to the coordinates (0,0,0). This would cause both the camera and the cube to be inside each other. To avoid this, we simply move the camera out a bit.
scene.add( cube );camera.position.z = 5;
23
![Page 24: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/24.jpg)
10. Create a loop to render something on the screen
function render() {requestAnimationFrame( render );renderer.render( scene, camera );
}render();
This will create a loop that causes the renderer to draw the scene 60 times per second.
11. Animating the cube
This will be run every frame (60 times per second), and give the cube a nice rotation animation
cube.rotation.x += 0.1;cube.rotation.y += 0.1;
24
![Page 25: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/25.jpg)
output:
25
![Page 26: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/26.jpg)
2. Shaded Material on Sphere
1. create a empty page and import three.min.js and jquery <html>
<head><title>Shaded Material on Sphere </title><style>
body { margin: 0; }canvas { width: 100%; height: 100% }
</style><script src="js/jquery.min.js"></script><script src="js/three.min.js"></script><script>// Our Javascript will go here.</script>
</head><body>
<div id="container"></div></body>
</html>26
![Page 27: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/27.jpg)
27
2. Repeat the same steps at in previous example
var scene = new THREE.Scene();var camera = new THREE.PerspectiveCamera(45, 600/600 , 0.1, 10000);var renderer = new THREE.WebGLRenderer();
renderer.setSize(600 , 600 );$container.append(renderer.domElement);
scene.add(camera);camera.position.z = 300; // the camera starts at 0,0,0 so pull it back
3. Create the sphere's material as MeshLambertMaterialMeshLambertMaterial is non-shiny (Lambertian) surfaces, evaluated per vertex. Set the color to red .
var sphereMaterial = new THREE.MeshLambertMaterial( { color: 0xCC0000 });
4. create a new mesh with sphere geometry ( radius, segments, rings) and add to scene
var sphere = new THREE.Mesh( new THREE.SphereGeometry( 50, 16, 16 ), sphereMaterial);scene.add(sphere);
![Page 28: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/28.jpg)
5. LightCreate light , set its position and add it to scene as well . Light can be point light , spot light , directional light .
var pointLight = new THREE.PointLight(0xFFFFFF);pointLight.position.x = 10;pointLight.position.y = 50;pointLight.position.z = 130;scene.add(pointLight);
6. Render the whole thing
renderer.render(scene, camera);
28
![Page 29: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/29.jpg)
output:
29
![Page 30: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/30.jpg)
3. Complex objects like Torusknot1.Same as before make scene , camera and renderer
scene = new THREE.Scene();camera = new THREE.PerspectiveCamera(125, window.innerWidth / window.innerHeight, 1, 500);camera.position.set(0, 0, 100);camera.lookAt(new THREE.Vector3(0, 0, 0));var renderer = new THREE.WebGLRenderer(); renderer.setSize( window.innerWidth, window.innerHeight );document.body.appendChild( renderer.domElement );
2. Add the lighting
var light = new THREE.PointLight(0xffffff);light.position.set(0, 250, 0);scene.add(light);var ambientLight = new THREE.AmbientLight(0x111111);scene.add(ambientLight);
30
![Page 31: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/31.jpg)
31
3. add Torusknotgeometry with radius, tube, radialSegments, tubularSegments, arc
var geometry = new THREE.TorusKnotGeometry( 8, 2, 100, 16, 4, 3 ); var material = new THREE.MeshLambertMaterial( { color: 0x2022ff } );var torusKnot = new THREE.Mesh( geometry, material ); torusKnot.position.set(3, 3, 3);scene.add( torusKnot );camera.position.z =25;
4.do the animation and render on screen
var render = function () { requestAnimationFrame( render ); torusKnot.rotation.x += 0.01; torusKnot.rotation.y += 0.01; renderer.render(scene, camera);}; render();
![Page 32: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/32.jpg)
output:
32
![Page 33: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/33.jpg)
4. 3D with webcam texture
Display the video as a plane which can be viewed from various angles in a given background landscape.
1.Use code from slide 10 to get user’s webcam input through getUserMedia
2. Make a Screen , camera and renderer as previously described
3. Give orbital CONTROLS for viewing the media plane from all angles
controls = new THREE.OrbitControls( camera, renderer.domElement );
4. Add point LIGHT to scene
33
![Page 34: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/34.jpg)
34
5. Make the FLOOR with an image texture
var floorTexture = new THREE.ImageUtils.loadTexture( 'imageURL.jpg' );floorTexture.wrapS = floorTexture.wrapT = THREE.RepeatWrapping; floorTexture.repeat.set( 10, 10 );var floorMaterial = new THREE.MeshBasicMaterial({map: floorTexture, side: THREE.DoubleSide});var floorGeometry = new THREE.PlaneGeometry(1000, 1000, 10, 10);var floor = new THREE.Mesh(floorGeometry, floorMaterial);floor.position.y = -0.5;floor.rotation.x = Math.PI / 2;scene.add(floor);
6. Add Fog
scene.fog = new THREE.FogExp2( 0x9999ff, 0.00025 );
![Page 35: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/35.jpg)
35
7.Add video Image Context and Texture.
video = document.getElementById( 'monitor' );videoImage = document.getElementById( 'videoImage' );
videoImageContext = videoImage.getContext( '2d' );videoImageContext.fillStyle = '#000000';videoImageContext.fillRect( 0, 0, videoImage.width, videoImage.height );
videoTexture = new THREE.Texture( videoImage );videoTexture.minFilter = THREE.LinearFilter;videoTexture.magFilter = THREE.LinearFilter;
var movieMaterial=new THREE.MeshBasicMaterial({map:videoTexture,overdraw:true,side:THREE.DoubleSide}); var movieGeometry = new THREE.PlaneGeometry( 100, 100, 1, 1 ); var movieScreen = new THREE.Mesh( movieGeometry, movieMaterial );
movieScreen.position.set(0,50,0); scene.add(movieScreen);
![Page 36: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/36.jpg)
36
8. Set camera position
camera.position.set(0,150,300);camera.lookAt(movieScreen.position);
9. Define the render function
videoImageContext.drawImage( video, 0, 0, videoImage.width, videoImage.height ); renderer.render( scene, camera );
10. Animation
requestAnimationFrame( animate ); render();
![Page 37: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/37.jpg)
output:
37
![Page 38: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/38.jpg)
4. Button touch detector
This example shows the process of triggering Web page activity by detecting motion using difference accuracy
1. Follow the same steps as in previous example
2. Define as many buttons
var buttons = [];var button1 = new Image();button1.src ="https://stemkoski.github.io/Three.js/images/SquareRed.png";var buttonData1 = { name:"red", image:button1, x:320 - 96 - 30, y:10, w:32, h:32 };buttons.push( buttonData1 );
3. make 3 layers of canvas for video , buttons and blended repectively .
38
![Page 39: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/39.jpg)
39
4. Define blend
// get current webcam image datavar sourceData = videoContext.getImageData(0, 0, width, height);// create an image if the previous image doesn’t existif (!lastImageData) lastImageData = videoContext.getImageData(0, 0, width, height);// create a ImageData instance to receive the blended resultvar blendedData = videoContext.createImageData(width, height);// blend the 2 imagesdifferenceAccuracy(blendedData.data, sourceData.data, lastImageData.data);// draw the result in a canvasblendContext.putImageData(blendedData, 0, 0);// store the current webcam imagelastImageData = sourceData;
5. Use mathematical logic for difference accuracy
6. The motion areas will be highlighted with white while no motion areas will become black.
7. check if white portion overlaps with buttons .
![Page 40: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/40.jpg)
output Differential :
40
output:
![Page 41: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/41.jpg)
Other WebRTC WebGL 3D JS applications
41
![Page 42: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/42.jpg)
karenLabs
42
http://www.karenlabs.com/music/#.VUwvYFE4bK4
![Page 43: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/43.jpg)
MotionEmotion
43
http://motionemotion.herokuapp.com/
![Page 44: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/44.jpg)
Mosh.io
44
http://www.karenlabs.com/music/#.VUwvYFE4bK4
![Page 45: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/45.jpg)
DeerHeaven
45
http://hayleejung.com/deerheaven/
motion and gesture detector using webcam inout
![Page 46: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/46.jpg)
Cubeslam
46
https://www.cubeslam.com/mgeumt
WebRTC based 2 party game
![Page 47: Augmented reality in web rtc browser](https://reader030.vdocuments.site/reader030/viewer/2022032503/55bb0245bb61ebc00e8b45db/html5/thumbnails/47.jpg)
47
Thats all folks !