Exploiting cross-modal rhythm for robot perception of objects

A. Arsenio and P. Fitzpatrick

CSAIL, MIT, Cambridge, MA 02139, USA

{arsenio,paulfitz}@csail.mit.edu

Abstract

This paper presents an approach to perceiving rhyth-mically moving objects that generate sound as theymove. The work is implemented on the humanoidrobot Cog [2]. We show selectivity and robustness inthe face of distracting motion and sounds. Our methoddoes not require accurate sound localization, and infact is complementary to it. We are motivated by thefact that objects that move rhythmically are commonand important for a humanoid robot. The humanoidform is often argued for so that the robot can interactwell with tools designed for humans, and such toolsare typically used in a repetitive manner, where thesound is generated by physical abrasion or collision:hammers, chisels, saws etc. We also work with theperception of toys designed for infants – rattles, bellsetc. – which could have utility for entertainment/petrobotics. Our goal is to build the perceptual tools re-quired for a robot to learn to use tools and toys throughdemonstration.

1 Introduction

Tools are often used in a manner that is composedof some repeated motion – consider hammers, saws,brushes, files, etc. This repetition can potentially aida robot to perceive these objects robustly. Our ap-proach is for the robot to detect simple repeated eventsat frequencies relevant for human interaction, usingboth visual and acoustic perception. The advantageof combining rhythmic information across these twomodalities is that they have complementary proper-ties. Since sound waves disperse more readily thanlight, vision retains more spatial structure – but forthe same reason it is sensitive to occlusion and therelative angle of the robot’s sensors, while auditoryperception is quite robust to these factors. The spa-tial trajectory of a moving object can be recoveredquite straightforwardly from visual analysis, but notfrom sound. However, the trajectory in itself is notvery revealing about the nature of the object. We use

cameras on active vision head

microphone array above torso

periodically moving object (hammer)

periodically generated sound (banging)

Figure 1: The experimental platform. The humanoidrobot Cog [2] is equipped with cameras in an activevision head, and a microphone array across the torso.A human demonstrates some repetitive action to therobot, such as using a hammer.

the trajectory to extract visual and acoustic features– patches of pixels, and sound frequency bands – thatare likely to be associated with the object. Both canbe used for recognition. Sound features are easier touse since they are relatively insensitive to spatial pa-rameters such as the relative position and pose of theobject and the robot.

Our work is implemented on Cog [2], an upper-torsohumanoid robot (see figure 1). Previous work on Coghas shown the ability to extract a perceptual bene-fit (object segmentation) from known motion (objecttapping) [4]. Cog has previously been applied to per-forming basic repetitive behaviors [8] including turn-ing a crank, hammering, sawing, playing with a slinky,swinging a pendulum, and so on. In a sense, the cur-rent work is a perceptual analog of this motor ability,which had no sensory component other than directfeedback from the robot’s joints.

2 Detecting rhythmic motion

Perhaps the most direct way to detect periodicityis to use the Short-Time Fourier Transform (STFT).

This transform maps a signal into a two-dimensionalfunction of time and frequency. A STFT is applied toeach input signal available,

I(t, ft) =

N−1∑

t=0

i(t′)h(t′ − t)e−j2πftt′ (1)

where h is a windowing function, and N the number ofsamples. Periodicity is estimated from a periodogramdetermined for all signals from the energy of the win-dowed FFTs over the spectrum of frequencies. Theseperiodograms are then processed to determine whetherthey correspond to a clear unambiguous period. A pe-riodogram is accepted if its energy is concentrated overa narrow peak.

This is a very general method, and is similar tothat adopted in [7]. There are some difficulties withapplying it. We must choose a time span over whichto perform periodicity inference. The longer this span,the larger the number of signal repetitions available,and the larger the set of frequencies that can be pro-cessed, increasing the range of oscillating objects withwhich the robot can interact. But visual tracking per-formance decreases with the increase of the time win-dow, and the assumption of constant period over thiswindow is weakened for larger intervals, for both au-dio and visual signals. We attempted to address thistension by adopting a flexible compromise betweenthe spatial and frequency based views of a signal.The periodicity detection is applied at multiple scales,with long spatial intervals (and hence small windowsizes) providing more precise spatial, local informa-tion, while larger windows increase the frequency res-olution. For objects oscillating during a short periodof time, the movement might not appear periodic at acoarser scale, but does appear as such at a finer scale.If a strong periodicity is found, the points implicatedare used as seeds for object segmentation (discussedin section 4). Otherwise the window size is halved andthe procedure is repeated for each half.

The STFT based-method demonstrated good per-formance after extensive evaluation for visual signalsfrom periodic objects, but it is not appropriate for pe-riodicity detection of acoustic signals. These signalsmay vary considerably in amplitude between periods,which – particularly when combined with variabilityin the length of the periods – suggests that Fourieranalysis is not appropriate. This led us to the devel-opment of a more robust method for periodicity de-tection, which was applied to both acoustic and vi-sual signals. We construct a histogram of the dura-tions between successive instances of particular valuesof the signal, and search for the most common dura-

freq

uenc

y (k

Hz)

0 500 1000 1500 20000

2

4

6

8

0 500 1000 1500 2000 25000

500

1000

1500

ener

gy

0 500 1000 1500 2000 2500−80

−70

−60

−50

time (ms)

ham

mer

pos

ition

Figure 2: Audio and visual information for a hammerhitting a table. The robot monitors the spectrum ofthe sound signal (the top graph shows the completespectrogram, the middle graph shows the overall en-ergy signal) while tracking the trajectory of objectsvisible to its camera (bottom).

tion, which we take to be the signal’s period. To makethis process efficient, we use a hash table indexed bythe signal values (combined with their derivatives) dis-cretized relative to the minimum and maximum valueof the signal. Each index of the hash table correspondsto distinct values for the signal and its derivative, andthe information stored in the table corresponds to thetime instants when each value/derivative pair last oc-curred. After building the table, a period histogramis constructed by the differences found during assign-ments to the table. The histogram maximum corre-sponds to the signal period.

3 Matching sound and vision

Due to physical constraints, the set of sounds thatcan be generated by manipulating a particular objectis often quite small. For tools and toys which aresuited to one specific kind of manipulation – as ham-mers encourage banging – there is even more structureto the sound they generate. We expect that for suchobjects, when sound is produced through motion, theaudio signal will be highly correlated both with themotion of the object and the identity of the tool.

This concept can be illustrated with a basic exam-ple: hammering (see figure 2). Whenever the hammerbangs in the table, a distinctive audio signal is pro-duced, spread all over the frequency bands, with asharp rise of energy at the instant of impact and rapidfall-off thereafter. If we monitor the visual trajectoryof the hammer along the main axis of motion, it oscil-lates at the same frequency as the sound, with approx-

imately zero phase-shift at the moment of impact. Thedetails of this sound may vary with the surface thatthe hammer bangs against, but the overall pattern isconsistent.

For this example and all other experiments de-scribed in this paper, our system tracks moving pix-els in a sequence of images from one of the robot’scameras using a multiple tracking algorithm based onthe a pyramidal implementation of the Lukas-Kanadealgorithm. A microphone array sampled the soundsaround the robot at 16kHz. The Fourier transform ofthis signal is taken with a window size of 512 samplesand a repetition rate of 31.25Hz. The Fourier coeffi-cients are grouped into a set of frequency bands for thepurpose of further analysis, along with the overall en-ergy. ‘Binding’ or grouping of simultaneous audio andvisual signals takes place if the periods of both signalsmatch within a tolerance of approximately 60ms. Nobinding is carried out if a moving object is silent, or ifa noise-making object lies outside of the robot’s fieldof view.

We now work through three cases of cross-modalbinding of increasing complexity, beyond the simplesituation already described for the hammer. Thefirst case is when multiple moving objects are visible,but only one repeating sound is heard. If the soundmatches the motion of one of the objects, it will bebound to that one and not the other. Similarly, iftwo repeating sounds with different periods are heard,and a single moving object is visible, the sound withmatching period can be bound with the visible object– this is the second case examined. Finally, we showthat multiple sound and visual sources can be boundtogether appropriately.

3.1 Matching with visual distraction

In this section we consider the case of multiple ob-jects moving in the robot’s visual field, only one ofwhich is generating sound. The robot uses the soundit hears to filter out uncorrelated moving objects anddetermine a candidate for cross-modal binding. Thisis a form of context priming, in which an external sig-nal (the sound) directs attention towards one of a setof potential candidates.

Figure 3 shows measurements taken during an ex-periment with two objects moving visually, at differ-ent rates, with one - a toy car - generating a rollingsound, while the other - a ball - is moving silently. Theacoustic signal is linked with the object that generatedit (the car) using period matching. The movement ofthe ball is unrelated to the period of the sound, and sothat object is rejected. In contract, for the car there

time (ms)

frequ

ency

(kHz)

0 500 1000 1500 20000

1

2

3

4

5

6

7

8

0 500 1000 1500 2000 25000

5

10

ener

gy

0 500 1000 1500 2000 2500−50

0

50

car p

ositio

n

0 500 1000 1500 2000 250040

50

60

time (ms)

ball p

ositio

n

Figure 3: The top left image shows a car and a ballmoving simultaneously, with their trajectories over-laid. The spectrogram during this event is shown tothe right. Sound is only generated by the rolling car –the ball is silent. A circle is placed on the object (car)with which the sound is bound. The sound energy andthe visual displacements of the objects are given.

is a very definite relationship. In fact, the sound en-ergy signal has two peaks per period of motion, sincethe sound of rolling is loudest during the two momentsof high velocity motion between turning points in thecar’s trajectory. This is a common property of soundsgenerated by mechanical rubbing, so the binding algo-rithm takes this possibility into account by testing forthe occurrence of frequencies at double the expectedvalue. Section 3.4 develops another approach to deal-ing with these and similar situations. Section 4 looksat using the very richness of the relationship betweenaudio and visual signals for the purposes of objectrecognition.

3.2 Matching with acoustic distraction

This section considers the case of one object movingin the robot’s field of view, and one ‘off-stage’, withboth generating sound. Matching the right sound tothe visible object is achieved by mapping the time his-tory of each individual coefficient band of the audiospectrogram (see figure 4) to the visual trajectory ofthe object. We segment the sound of the object fromthe background by clustering the frequency bands withthe same period (or half the period) as the visual tar-get, and assign those bands to the object.

Within the framework being described, visual infor-mation is used to prune the range of frequency bandsof the original sound - the coefficient bands of the au-

freq

uenc

y (k

Hz)

0 500 1000 1500 20000

2

4

6

8

500 1000 1500 2000

50

60

70

80

90

car

posi

tion

freq

uenc

y (k

Hz)

0 500 1000 1500 2000 25000

2

4

6

8

500 1000 1500 2000 2500

4

6

8

10

12

time (ms)

snak

e po

sitio

n

Figure 4: The two spectrogram/trajectory pairs shownare for a shaking toy car and snake rattle. The toppair occurs with only the car visible, and the lowerpair occurs with only the snake visible. The line ineach spectrogram represents the cutoff pitch frequencybetween the car and snake.

dio visual are segmented into clusters of bands thatcharacterize the sound of an object. For the exper-iment shown in the upper part of figure 4, the coef-ficients ranging from 0 to 2.6Hz are assigned to theobject. Afterwards, a band-pass filter is applied tothe audio-signal to filter out the other frequencies, re-sulting the clear sound of the car with the sound ofthe rattle removed or highly attenuated. For the ex-periment shown in the lower part of figure 4 the rolesof the car and snake were switched. A band-pass fil-ter between 2.6-2.8Hz is applied to the audio-signalto filter out the frequencies corresponding to the car,resulting the snakes’ sound.

3.3 Matching multiple sources

This experiment considers two objects moving inthe robot’s field of view and both generating sound,as presented in Figure 5. Each temporal trajectory ofa coefficient group is mapped into one of the visualtrajectories if coherent with its periodicity. For eachobject, the lower and the higher coefficient band are

time (ms)

freq

uenc

y (k

Hz)

0 500 1000 1500 20000

2

4

6

8500 1000 1500 2000

−60

−40

−20

car

posi

tion

500 1000 1500 2000

30

35

40

rattl

e po

sitio

n

Figure 5: The car and the cube, both moving, bothmaking noise. The line overlaid on the spectrogram(bottom) shows the cutoff determined automaticallybetween the high-pitched bell in the cube and the low-patched rolling sound of the car. A spectrogram of thecar alone can be seen on Figure 3. The frequencies ofboth visual signals are half those of the audio signals.

labelled as the lower and higher cut-off frequencies, re-spectively, of a band-pass filter assigned to that object.The complex sound of both the moving car-toy and thecube-rattle are thus segmented into the characteristicsound of the car and sound of the rattle through band-pass filtering. Multiple bindings are thus created formultiple oscillating objects producing distinct sounds.

It is worth stressing that the real world is full of ob-jects making all kinds of noise. However, the systemis robust to such disturbances. On the experimentspresented throughout this paper, people were speak-ing occasionally while interacting with the robot, whileother people were making everyday sounds while work-ing. If the distracting sound occurs at the same rangeof frequencies as the sound of the oscillating object,then a binding might just not occur for that specifictime, but occur after a few seconds when the interfer-ence noise switches to other frequencies or disappears.

3.4 Priming sound detection using vision

Figure 6 shows how well the methods described forbinding sounds with objects work on a series of exper-iments. False bindings occur in only one case, wheretwo objects are moving, a mouse and a plane. Onlythe plane is generating sound. The sound is a roughnoise with silence at the two extrema of the plane’s mo-tion, and hence appears to have a frequency of doublethat of the trajectory. This is close to the frequencyof oscillation of the mouse, so simple period matchingoccasionally gives false results. This is symptomaticof a more general problem: the sound generated by a

Experiment visualperiodfound

soundperiodfound

bindmade

goodbind(%)

badbinds(%)

missedbinds(%)

hammer 6 8 6 100 0 0car & ball 7 11 1 14 0 86car 6 6 5 83 0 17car (snakebackground)

5 16 5 100 0 0

snake (carbackground)

4 9 4 100 0 0

plane andmouse

23 27 16 46 41 13

Figure 6: An evaluation of the efficacy of cross-modalbinding for various objects and situations.

periodically moving object can be much more complexand ambiguous than its visual trajectory. The extremaof an approximately repeating trajectory can be foundwith ease, and used to segment out single periods ofoscillation within an object’s movement. Single peri-ods of the sound signal can be harder to find, sincethere is more ambiguity – for example, some objectsmake noise only at one point in a trajectory (such as ahammer), others make noise at the two extrema (somekinds of bell), others make noise during two times ofhigh velocity between the extrema (such as a saw), andso on. For cases where periodicity detection is difficultusing sound, it makes sense to define the period of anaction in the visual domain based on its trajectory,and match against this period in the sound domain –instead of detecting the period independently in eachdomain. We have developed an approach, where foreach object moving visually, fragments of the soundare taken for periods of that object, aligned, and com-pared. If the fragments are consistent, with sound andvision in phase with each other, then the visual trajec-tory and the sound are bound. This is a more strin-gent test than just matching periods, yet avoids theproblem of determining a period reliably from soundinformation. Figure 7 shows results for the plane-and-mouse example described at the beginning of this sec-tion, showing that it does in fact rectify the problem.The periodicity detection method in section 2 is stillnecessary when only sound information is available.

There is evidence that, for humans, simple visualperiodicity can aid the detection of acoustic period-icity. If a repeating segment of noise is played, therepetition can be detected for much longer periods ifa light is flashing in synchrony with some point in theperiod [1]. More generally, there is evidence that thecues used to detect periodicity can be quite subtle andadaptive [5], suggesting there is a lot of potential forprogress in replicating this ability beyond the ideasalready described.

time (ms)

freq

uenc

y (k

Hz)

0 200 400 6000

2

4

6

8

time (ms)

freq

uenc

y (k

Hz)

0 200 400 600 8000

2

4

6

8

Figure 7: An experiment in which a plane is beingpushed across wood while a mouse is shaken in thebackground. Shown are the highest quality acousticmatches for this sound (right) and the object withwhich they correspond (left). Matches against themouse are much lower and below threshold.

4 Object segmentation and recognition

Different objects have distinct acoustic-visual pat-terns which are a rich source of information for objectrecognition. Our approach can differentiate objectsfrom both their visual and acoustic backgrounds byfinding pixels and frequency bands (respectively) thatare oscillating together. We deal with a fundamen-tal problem in computer vision - object segmentation -by detecting and interpreting natural human behav-ior such as waving or shaking objects. This is impor-tant, because object segmentation on unstructured,non-static, noisy, real-time and low resolution imagesis a hard problem. Results for object-background sep-aration are shown in figure 8 and were obtained withvarying light conditions. The environment was notmanipulated to improve natural occurring elements,such as shadows or light saturation from a light source.All the experiments were made while a human or therobot were performing an activity.

Features extracted from the visual and acoustic seg-mentations are what we need to build an object recog-nition system (in the visual domain see [3], and [6] haslooked at the recognition of sound generated by a sin-gle contact event). Each type of feature is importantfor recognition when the other is absent. But whenboth are present, then we can do better at recognitionby looking at the relationship between visual motionand the sound generated (see figure 9), to which themethod in section 3.4 gives us easy access.

Figure 8: Examples of object segmentations.

5 Discussion and conclusions

We described techniques to detect periodicity ofsignals, identifying their strengths and limitations.Through the detection of visual periodic events, wewere able to localize an object in the visual fieldand extract information concerning its trajectory overtime, as well as to segment a visual representation ofan object from an image. In addition, sound segmen-tation - the identification of the frequency bands thatbest characterize an object - was also possible fromjust acoustic information. A cross-modal strategy toperiod detection proved necessary and advantageous,being more robust to disturbances either in sound orvision, and providing a better characterization of ob-jects. We discussed how to reliably bind the visualappearance of objects to the sound that they gener-ate, and to achieve selectivity: a visual distractor wasfiltered out using sound, and sound was also used toprime the visual field. In addition, we argued that thecross-modal strategy is well suited for integration withobject recognition strategies for searching visually fortools and toys and finding/recognizing them whenevertheir sound is perceived by the robot.

A lot about the world could be communicated to ahumanoid robot through human demonstration. Therobot’s learning process will be facilitated by send-ing it repetitive information through this communica-tion channel. If more than one communication chan-nel is available, such as the visual and auditory chan-nels, both sources of information can be correlated forextracting richer pieces of information. We demon-strated in this paper a specific way to take advantageof correlating multiple perceptual channels at an earlystage, rather than just by analyzing them separately -the whole is truly greater than the sum of the parts.

Acknowledgements

This work was funded by DARPA DABT 63-00-C-10102 (“Natural Tasking of Robots Based on HumanInteraction Cues”), and by NTT under the NTT/MITCollaboration Agreement. Arsenio was supported byPortuguese grant PRAXIS XXI BD/15851/98.

Figure 9: The relationship between object motion andthe sound generated varies in an object-specific way.The hammer causes sound when changing direction af-ter striking an object. The bell typically causes soundat either extreme of motion. A toy truck causes soundwhile moving rapidly with wheels spinning; it is quietwhen changing direction.

References

[1] J. A. Bashford, B. S. Brubaker, and R. M. Warren.Cross-modal enhancement of repetition detectionfor very long period recycling frozen noise. Journalof the Acoustical Soc. of Am., 93(4):2315, 1993.

[2] R. A. Brooks, C. Breazeal, M. Marjanovic, andB. Scassellati. The Cog project: Building a hu-manoid robot. Lect. Notes in Comp. Sci., 1562:52–87, 1999.

[3] P. Fitzpatrick. Object lesson: discovering andlearning to recognize objects. October 2003. Ac-cepted for publication at the 3rd InternationalConference on Humanoid Robots, Karlsruhe, Ger-many.

[4] P. Fitzpatrick and G. Metta. Grounding visionthrough experimental manipulation. PhilosophicalTransactions of the Royal Society: Mathematical,Physical, and Engineering Sciences , 2003. In press.

[5] C. Kaernbach. Temporal and spectral basis of thefeatures perceived in repeated noise. Journal of theAcoustical Soc. of Am., 94(1):91–97, July 1993.

[6] E. Krotkov, R. Klatzky, and N. Zumel. Roboticperception of material: Experiments with shape-invariant acoustic measures of material type. InExperimental Robotics IV . Springer-Verlag, 1996.

[7] R. Polana and R. C. Nelson. Detection and recog-nition of periodic, non-rigid motion. Interna-tional Journal of Computer Vision, 23(3):261–282,June/July 1997.

[8] M. M. Williamson. Exploiting natural dynamics inrobot control. In 14th Eur. Meet. on Cyberneticsand Systems Research. Vienna, Austria, 1998.