Technologies for Interactive Architecturea guidebook

October 2014

Contents1 - Introduction

3 - Motion tracking

7 - Various sensors

8 - Internet-of-Things

1/9

IntroductionIncreasing popularity of various sensors and tracking devices

allows us to create environments that have an unprecedented

level of interactivity and intelligence. Environments that respond

to movements, gestures, online data or environmental parameters

such as temperature or humidity are now within reach.

The reason behind this interactive movement is not just to use

the technology that happens to be available to us, but to create

environments that adapt and evolve to be simply the best and

most beautiful in any given moment.

This is and should always be the goal.

We realise however, that diving into these, still uncharted, waters

can be a daunting task. A task that is made even more difficult by

the absence of a guide giving a basic insight into what is available

out there.

2/9

This guidebook will make this easier.

3/9

Motion trackingMotion tracking is a technology that is probably most often used in interactive architecture. It tracks movements of people

or objects in a given field of view. There are two prevalent approaches to this: the first (and the oldest) one is using a video

feed, provided by a video camera, while the other is using a depth feed provided by a depth camera. In some cases a thermal

camera can also be used, but each approach has its advantages and disadvantages.

Video feed motion trackingVideo feed motion tracking algorithms use a video stream in

order to track movements of people or objects in it. There are

several ways of doing this but most algorithms subtract the

current frame of the video pixel-by-pixel from the image of a

background. The basic approach is to assume that the pixels

that are different from the background pixels by a significant

margin are the pixels from the object. The main questions that

arise when designing these algorithms are, how is it treating

objects moving very slowly or objects that stop moving after

a while, such as parked vehicles and how is it treating other

moving shapes, such as leaves or moving shadows, that we

don’t want to track.Video feed with superimposed rectangles and paths showing the results of the tracking algorithm.

What to be mindful ofBackground The key challenge of motion tracking based on a video feed is determining what exactly

is a background and how to deal with objects moving differently than expected or moving

objects that we don’t want to track, for example leaves or plastic bags being blown by the

wind or other moving shapes such as moving shadows or lightspots. Changing visual conditions If the visual conditions change rapidly it might influence the reliability of motion tracking

and the algorithm might have difficulties distinguishing between objects that need to be

tracked and the changing background. Examples include video walls in the camera view, car

headlights or strong moving lightspots or similar.Framerate Tracking framerate means how many images per second the tracking algorithm analyses for

moving objects. The faster the algorithm the more precise and responsive it is, but also the

more resources it consumes. Resolution For motion tracking we usually don’t need such high video resolutions as are needed for

security purposes. Also it is beneficial to decrease the resolution of the video, because it

significantly speeds up the performance of the tracking algorithm. However, if you lower it

too much, it might reduce the reliability and accuracy of the motion tracking

4/9

IP cameras can be used with reActor Media (running LightAct Media), which is receiving all the video feeds and performs real-time motion tracking algorithms to generate visual effects.

SmartView modules were designed with analogue video cameras in mind. Each of them can analyse one video stream and send the coordinates of people to reActor V, which uses these to generate a visual effect.

USB cameras can be connected directly to reActor Media through one of its several USB ports.

Using IP cameras for motion trackingIP cameras are rapidly gaining popularity as they are using standard Ethernet equipment such as Cat5 cables, Ethernet routers

and switches for video transmission. They also transmit video in digital packets, which increases the quality of video compared

to analogue cameras. They are available in a wide variety of models, both outdoor and indoor and in many cases can be

powered through Ethernet connection as well (Power-over-Ethernet), which speeds up the installation time.

Using analogue cameras for motion trackingAnalogue cameras are a cheaper alternative to IP cameras, but they do not offer the same ease of installation, high quality

video signal and versatility as IP cameras. In combination with SmartView modules they are suitable for smaller interactive

lighting applications with stable lighting conditions.

Using USB camera for motion trackingUSB cameras (sometimes called also web cameras) are the most affordable way to get a video feed for motion tracking into

the controller. However, they were designed for desktop use, which usually prohibit their use in anything but the simplest

temporary installations.

Analogue cameraIt connects to and is powered by SmartView module

reActor V

SmartView

SmartView

InteractiveInstallation

Coaxial video cable

Coaxial video cable

Cat5 cable

Cat5 cableDMX

Analogue cameraIt connects to and is powered by Smartview module

USB cameraInteractiveInstallation

USB cable

reActor Media orLightAct Media

Art-Net or DVI/HDMI

IP camera

Ethernet switch/router

reActor Media orLightAct Media

InteractiveInstallation

WiFi or Cat5 cable

Cat5 cable Art-Net or DVI/HDMI

IP camera

WiFi or Cat5 cable

5/9

Depth feed motion trackingDepth cameras have gained popularity especially since the introduction of Microsoft Kinect in 2010. Nowadays, however,

there are several competing products as well. Depth cameras use different approaches to obtain a depth image of the space,

meaning that besides reading the colour of a particular pixel, they also measure the distance between that pixel and the

camera itself. This is very useful for motion tracking as it allows the algorithm to track a moving object much more reliably.

Most depth cameras also offer additional features, such as tracking of individual joints or fingers of people or even recognising

facial expressions.

Depth image (left) with the tracking result shown on the right. With depth feed tracking it is much easier to obtain silhouettes of people

What to be mindful ofRange Most depth cameras have a limited range of around 4m. Care has to be taken when

determining the exact location of the depth camera in relation to the area that you want to

track.Protection against

environment

Depth cameras by and large don’t have any weather protection because they have been

designed primarily for indoor operation. Light conditions Depending on the type of depth camera, there may be some limitations of the light

conditions it can be used in. For example, the performance of Microsoft Kinect drops

significantly in direct sunlight.USB connection Most depth cameras are connected to the controller (or a PC) with a USB connection.

As they need a very large bandwidth to transfer all the data, the range of a standard USB

cable can only be a couple of meters. If you need to install your depth camera further than

that from the controller you should use a USB extender.Framerate Similarly to video feed tracking, tracking framerate means how many depth images per

second the tracking algorithm analyses. In addition, whenever we are using a depth camera,

the algorithm usually works on 2 sets of streams, video stream as well as depth stream,

which means framerate has a big effect on the required processing power of the controller.

Depth cameras usually offer additional functionalities such as skeletal tracking. Some offer finger tracking as well.

Depth cameras can be connected directly to a reActor Media through one of its USB 2.0 or 3.0 ports.

reActor Media orLightAct Media

USB cable or an USB extenderDepth camera

InteractiveInstallation

Art-Net or DVI/HDMI

6/9

Thermal feed motion trackingThermal cameras capture thermal energy emitted by the objects

in the scene. This allows the tracking algorithm to distinguish

between people, which have usually a higher temperature, from

their background. The technology has obvious advantages as

it allows for tracking of people that are still for a longer period

of time and it is much less sensitive to can be a problem with

video based motion tracking. It is important to know that if the

temperature of the environment or of the surrounding objects

is equal to that of the people, the tracking won’t be any more

accurate than with video based tracking - it might actually be even

lower.

What to be mindful ofObject temperatures Thermal feed tracking is most commonly used when the goal is to track people (as opposed

to cars for example). Especially in temperate climates, people are usually warmer than

their surroundings. However, it is always necessary to be mindful of the temperature of

the surrounding objects. For example, heating bodies or asphalt heated by the sun could

significantly disrupt the tracking.Camera model Thermal cameras are much more expensive than video cameras and are most commonly

used in completely different types of applications such as military or handheld surveying. It

is important to choose the right model with sufficient protection against the elements.

Feed of a thermal camera showing warmer bodies in red and cooler background in blue.

Comparison of technologiesDescription IP camera Analogue camera USB camera Thermal camera Depth cameraConnection Ethernet

(Cat 5 cable or

Wi-Fi)

Coaxial video cable USB cable or an

USB extender

varies, but

additional video

capture card might

be required

USB cable or an

USB extender

(works only on

USB 2.0)Environment Outdoor or indoor Outdoor or indoor Mostly indoor Outdoor and

indoor

Indoor

Range Infinite* Infinite* Infinite* Infinite* 4m

Limitations Dependent on

network capability

and bandwidth

Video prone to

noise, specific

video cabling

needed

Delicate and non-

robust cameras

mostly designed

for desktop use

Price, most models

designed for

hand-held use,

connectivity

Mostly don’t work

in direct sunlight,

conspicuous shape

unless concealed.* depends on the resolution and zoom

7/9

Various sensorsThere is a myriad of different sensors that can detect almost anything that can be detected or measured. Motion, presence,

temperature, humidity, distance, light levels... - the list goes on and on. Even though most of these sensors were developed

for industrial applications, they can be easily used in interactive architecture as well. They can be divided into two main types:

digital and analogue sensors.

Digital sensorsDigital sensors (also called switching contact sensors) measure quantities that are either

on/off or true/false. For example, the most common digital sensors in architecture are

motion detectors that detect if there is a movement in the room. Digital sensors used

in architecture usually require a 12V power supply, the ones used in electronics and

automation however, usually require 5V power supply. Both of which are available in

every SensorView module.

Analogue sensorsAnalogue sensors are sensors that measure continuous values, such

as distance, light levels, weight, humidity, temperature or other similar

quantities. Most of them communicate the measurement with a voltage

range from 0 to 5V and require a 5V power supply, which is also available in

every SensorView module.

Wall mounted motion sensor

Recessed motion sensor*

Temperature sensor**Humidity sensor**

Luminance sensor** Light sensor**

Up to 8 digital or 4 analogue sensors can be connected to each SensorView module, which are then connected to a reActor V for DMX lighting control. For larger installations or installations where media control is required the reActor V can transmit sensor data to reActor Media over Ethernet network.

* sensor manufactured by Osram** sensor manufactured by Grove

Digital sensor Up to 8 can be connected to and powered by one SensorView through its 5V or 12V power supply.

reActor V

SensorView

SensorView

InteractiveInstallation

InteractiveInstallation

2-wire or RJ9 cable

RJ9 cable

Cat5 cable

Cat5 cableDMX

Analogue sensorUp to 4 can be connected to and powered by one SensorView through its 5V or 12V power supply.

reActor Media orLightAct Media

Art-Net or DVI/HDMI

Cat5 connection over Ethernet

8/9

Internet-of-Things integrationMost modern technology is converging towards connectivity and accessibility through Internet, which is what is commonly

called Internet-of-Things. In interactive architecture it is possible to use any data available on the Internet or local networks

as an input. Examples include dedicated apps on smartphone or tablet, data collected from social network, weather, news or

other data sources, sensors connected to the Internet, traffic data and many others…

Some project examples...

Installation using video feed motion tracking for an interactive shopwindow.

Installation using 6 motion detectors for a low-res motion tracking.

...and ideas in the works:

• tracking of car traffic in order to generate a visual effect for an underpass lighting,

• tracking of people’s silhouettes and displaying them on a large media facade,

• tracking of people’s hand gestures and using the data for a generative effect on a media facade,

• tracking people’s movements & gestures for a dynamic lighting integrated in a 3D logo on a wall,

• tracking of individual finger movements thus allowing people to ‘paint’ on a media facade,

• detecting if there is snow in the park and using this information to control the lighting,

• controlling the lighting integrated in a park bench based on whether someone is sitting on it or not,

• scanning the color of people’s clothes and using it to control lighting & media.

Installation using video feed motion tracking for an interactive gallery.

Interactive suspended luminaire using a motion detector to trigger dynamic scenes.

9/9

Reach out to uslightact-systems.com

ActUp@lightact-systems.com

+386 1 320 44 72

Headquarters:LightAct d.o.o.

Slomškova 23

SI-1000 Ljubljana

Slovenia

With local representatives

in a growing list of countries.

that create

interactive

architecture.

services &

products

LightAct is:

lightact-systems.com