Sonderforschungsbereich 314Kunstliche Intelligenz - Wissensbasierte Systeme

KI-Labor am Lehrstuhl fur Informatik IV

Leitung: Prof. Dr. W. Wahlster

VITRAUniversitat des SaarlandesFB 14 Informatik IVPostfach 151150

D-66041 SaarbruckenFed. Rep. of Germany

Tel. 0681 / 302-2363

Bericht Nr. 91

Utilizing Interval-Based Event Representations

for Incremental High-Level Scene Analysis

Gerd Herzog

November 1992

ISSN 0944-7814 91

Utilizing Interval-Based Event Representations for

Incremental High-Level Scene Analysis �

Gerd Herzog

SFB 314, Project VITRA

Universit�at des Saarlandes

D-6600 Saarbr�ucken 11

herzog@cs.uni-sb.de

February 8, 1995

Abstract

Within the project Vitra (VIsual TRAnslator) we are concerned with the design and con-struction of integrated knowledge-based systems capable of translating visual information intonatural language descriptions. In this contribution1 the focus will be on high-level scene analysis,i.e., the step from a geometrical representation, as it might be provided by a vision component,into conceptual descriptions of object motions. In a combined vision and natural language systemaiming at simultaneous natural language descriptions of dynamic imagery the recognition of suchmotion events has to be done incrementally in order to be able to talk about events even while theyare progressing. We present course diagrams as a means for such an incremental motion analysisand show how they can be constructed automatically from interval-based event representationsusing temporal constraint propagation techniques.

�To appear in: Proc. of the 4th International Workshop on Semantics of Time, Space, and Movement and Spatio-Temporal Reasoning, Chateau de Bonas, France, 1992.

1The work described here was partly supported by the Sonderforschungsbereich 314 der Deutschen Forschungsge-meinschaft, \K�unstliche Intelligenz und wissensbasierte Systeme" project N2: VITRA.

1

1 Introduction

Image understanding and natural language processing are two major areas of research within AI thathave generally been studied independently of one another. Collaborating with the vision group atthe Fraunhofer-Institut (IITB, Karlsruhe), the project Vitra (VIsual TRAnslator) is concerned withthe combination of natural language and vision. Di�erent domains of discourse and communicativesituations are examined in order to design an interface between image understanding and naturallanguage systems.

Figure 1: A short tra�c scene

The following scenarios are under investigation:

� Answering questions about observations in tra�c scenes (c.f. [Schirra et al. 87])

� Generating running reports for short sections of soccer games (c.f. [Andr�e et al. 88]).

� Describing routes based on a 3-dimensional model of the Saarbr�ucker University Campus(c.f. [Herzog et al. 93])

� Communicating with an autonomous mobile robot (planned)

Figure 2: Three frames from the Soccer domain

2

Figure 3: 2D Representation

The results already obtained in connecting a vision com-ponent and a natural language access system have beenlimited to a bird's eye view and thus 2-dimensional rep-resentation of the scenes under discussion. Within thenew Vitra system the methods developed so far will beextended in order to cope with 3-dimensional geomet-ric representations (c.f. [Herzog 92], [Wazinski & Her-zog 92]). Advances in low-level vision, as it is carriedout by our project partners, form a promising basis forthis kind of extensions. The system Xtrack (c.f. [Koller92], [Koller et al. 92]) allows for the automatic model-based recognition, tracking, and classi�cation of vehiclesin tra�c scenes. Research described in [Rohr 89] and[Rohr & Nagel 90] concentrates on the model-based 3D-reconstruction of non-rigid bodies. The geometric modelutilized there for the recognition of pedestrians is alreadyintegrated into the Vitra system in order to representthe players in the Soccer domain. Figure 1 and 2 show

some frames taken from real world image sequences, that have been investigated. The previous 2-dimensional geometrical model, with players only represented as centroids, is visualized in �gure 3.The images in �gure 4 have been generated from our internal 3-dimensional models.

Figure 4: Synthesized images

2 High-Level Scene Analysis

The interpretation of object motions in terms of natural-language-oriented concepts constitutes acentral step in the process of translating the results of low-level vision processes into a natural languagedescription. Di�erent approaches for such a high-level scene analysis have been proposed:

3

Badler: Interpretation of simple object motions in wire-frame image sequences (c.f. [Badler75])

SUPP: Feature-based verb selection for picture patterns (c.f. [Okada 79])

ALVEN: Interpretation of left ventricular heart motion (c.f. [Tsotsos 85])

HAM-ANS: Natural language dialog about object motions in a tra�c scene (c.f. [Hoepp-ner et al. 83])

NAOS: Retrospective natural language description of object movements in a tra�c scene(c.f. [Neumann 89], [Novak 87])

SOCCER: Simultaneous natural language description of short sections from soccer games(c.f. [Andr�e et al. 88])

EPEX: Database-assisted extraction of complex motion events in a tra�c scene (c.f.[Walter 89])

Karlsruhe Motion Analysis System: Incremental feature-based computation of mo-tion events in a tra�c scene (c.f. [Heinze et al. 91], [Kollnig 92])

In Alven, Ham-Ans, Naos, Epex, as well as in our own system, the recognition of movements isbased on a hierarchy of event models, i.e., declarative descriptions of classes of higher conceptualunits capturing the spatio-temporal aspects of object motions. Fixed sets of semantic features areemployed instead in Supp and the Karlsruhe system. Since the motion concepts are only de�nedin terms of the features they are semantically independent from each other and thus no abstractionhierarchy exists. From the point of view of a natural language access system, semantic featuresalone do not impose enough structure on the represented knowledge. A conceptual hierarchy basedon specialization (e.g. walking is a moving) and temporal decomposition (e.g. passing consists ofswing-out, drive-beside, and swing-into-line) can be utilized in the language production processin order to guide the selection of the relevant propositions (c.f. [Novak 87], [Andr�e et al. 88]).Natural language access systems like Ham-Ans and Naos concentrate on an a posteriori analysis.Low level vision considers the entire image sequence for the recognition and cueing of moving objects;motion analysis, i.e. event recognition, happens afterwards, based on complete trajectories. Since onlyinformationabout a past scene can be provided, these systems generate retrospective scene descriptions.In Vitra we favour an incremental analysis. Input data is supplied and processed simultaneously asthe scene progresses. Information about the present scene is provided and immediate system reactions(like motor actions of a robot, simultaneous natural language utterances) are possible.

Figure 5: A \passing" event in a tra�c scene

4

3 Incremental Event Recognition

If a real-world image sequence is to be described simultaneously as it is perceived, one has to talkabout events even while they are currently happening and not yet completed. Thus events haveto be recognized stepwise as they progress and event instances must be made availlable for furtherprocessing from the moment they are �rst noticed. Consider the examples given in �gure 5, where awhite station wagon is passing a pick-up truck, and in �gure 2, where a player is transfering the ballto a teammate.Since the distinction between events that have and those that have not occured is insu�cient, wehave introduced the additional event predicates start, proceed, and stop which can be used tocharacterize the progression of an event with greater precision (cf. [Andr�e et al. 88]). By means of anincremental recognition strategy, based on these predicates, events can be recognized simultaneouslyas they occur in the underlying image sequence.

and(Stop(BALL-POSSESSION(p1 b) t) Start(MOVE-FREE(b) t))

and(Start(BALL-POSSESSION(p2 b) t) Stop(MOVE-FREE(b) t))

Proceed(MOVE-FREE(b) t)

:START :STOP

:PROCEED

Figure 6: Course diagram

Labeled directed graphs with typed edges, so called course diagrams, are used to model the prototypicalprogression of an event. Figure 6 shows a simpli�ed course diagram for the concept BALL-TRANSFER.It describes a situation in which a player passes the ball to a teammate. The event starts if aBALL-POSSESSION event stops and the ball is free. The event proceeds as long as the ball is movingfree and stops when the recipient has gained possession of the ball. The recognition of an occurrencecan be thought of as traversing the course diagram, where the edge types are used for the de�nitionof our basic event predicates.

(defevent BALL-TRANSFER(p1*player b*ball p2*player)

:subconcepts

BALL-POSSESSION(p1 b) [I1]

MOVE-FREE(b) [I2]

BALL-POSSESSION(p2 b) [I3]

:temporal-relations

[I1] :m [BALL-TRANSFER] [I1] :m [I2]

[I2] :e [BALL-TRANSFER] [I2] :m [I3])

Figure 7: A simple interval-based concept de�nition

Figure 7 shows the corresponding interval-based representation of the event concept. The de�nitionsof the di�erent temporal relations between two intervals are summarized in �gure 8.Course diagrams rely on a discrete model of time, which is induced by the underlying image sequence.

5

They allow incremental event recognition, since exactly one edge per unit of time (i.e., at video rate| 25 frames per second) is traversed.

X-?-Y Y Y Ybefore b Xmeet m X Xoverlap o X X X�nished by � X X X X Xcontains c X X X X X X Xstart s X Xequal e X X Xstarted by si X X X X Xduring d X�nish f X Xoverlapped by oi X X Xmet by mi X Xafter a X

Figure 8: Possible temporal relations between two intervals (c.f. [Allen 84])

4 Utilizing Temporal Reasoning

In general the speci�cation of a course diagram seems to be more lavish, because an interval-basedconcept de�nition is better readable. Another drawback is that it is di�cult to draw inferences like\the occurrence of an event E1 implies the occurrence of another event E2" (c.f. [Neumann 89]). So theproblem is that reasoning about events and event models based on specialization and decompositionimplies the use of (complete) temporal intervals, but incremental event recognition has to deal withdata related to single points in time.As a solution to this problemwe suggest to use both representation formats2 and to de�ne an automatictranslation procedure from interval-based concept de�nitions into course diagrams.

4.1 Constraint-Based Temporal Reasoning

The automatic construction of course diagrams can be seen as a temporal reasoning problem, whichcan be solved with temporal constraint propagation techniques. So the idea is to utilize a temporalreasoning system for the implementation of the translation procedure. The reasoner we currently useis the TimeLogic system (c.f. [Koomen 87]), that is basically an implementation of the constraintpropagation algorithm described in [Allen & Koomen 83]. Intervals and interval constraints given inthe concept de�nition form the input for the temporal reasoner, that computes the temporal relationsbetween each pair of intervals and detects inconsistencies in the relation network. If there are nodisjunctive relations between the intervals the TimeLogic system is able to compute the temporalorder of all start points and end points of the intervals. This projection onto the temporal axis caneasily be used for the construction of a course diagram.For the example given in �gure 7 two temporal relations [BALL-TRANSFER] :m [I3] and [I1] :b [I3]

will be calculated �rst. Figure 9 shows the �nal ordering of the start and end points. Only those

2A dual representation of motion concepts is also described in [Walter 89]. On the one hand event models andrecognized instances are stored in a KL-ONE-like representation format which is mapped onto a relational database.ATN's (c.f. [Woods 70]), which, however, are not generated from the KL-ONE net, are on the other hand used for therecogniton of events.

6

t0 - START: BALL-POSSESSION(p1 b)

t1 - END: BALL-POSSESSION(p1 b)

START: MOVE-FREE(b) BALL-TRANSFER(p1 b p2)

t2 - PROCEED: MOVE-FREE(b) BALL-TRANSFER(p1 b p2)

t3 - END: MOVE-FREE(b) BALL-TRANSFER(p1 b p2)

START: BALL-POSSESSION(p2 b)

t4 - END: BALL-POSSESSION(p2 b)

Figure 9: Conditions for the course diagram

time points related to the de�ned event have to be considered for the course diagram (c.f. �gure 6).The conditions for t1 and t3 are modelled as transitions from one node to another and the :PROCEEDcondition for t2 results in a loop.

(defevent PENALTY-KICK(p*player b*ball g*goalkeeper)

:subconcepts RUN-UP(p) [I1] SHOOT(p b) [I2]

MOVE(g) [I3] PARRY(g) [I4]

:temporal-relations

[I1] :s [PENALTY-KICK] [I1] (:b :m) [I2]

[I3] :d [PENALTY-KICK] [I3] :m [I4]

[I4] :f [PENALTY-KICK] [I2] :f [I3])

Figure 10: Coping with disjunctive relations

In general, disjunctions of temporal relations might occur explicitly or implicitly in a concept de�nition(c.f. �gure 10). 3 A direct projection onto the temporal axis is not possible if there are any disjunctiverelations between intervals, i.e., if temporal constraints are not strong enough to induce a total orderingon all start and end points. Disjunctive temporal relations imply that there are several ways ofdecomposing an event into subconcepts. Each possible temporal decomposition has to be captured byan own course diagram. A speci�c decomposition can be obtained by adding the appropriate temporalconstraints. TimeLogic provides a context mechanism which can be utilized in order to enumerateall possible temporal decompositions.In the example given in �gure 10 one disjunction is stated explicitly in the de�nition and a second(implicit) disjunction is calculated by the reasoner. Figure 11 and 12 show the context tree and thefour possible decompositions. The solutions of the constraint network correspond to the leafs of thecontext tree, i.e., those contexts in which all disjunctions are resolved (Context 1 3, 4, and 5 in ourexample).Constraint propagation inTimeLogic only ensures path consistency (c.f. [Allen & Koomen 83]). Thusthe reasoner can not detect inconsistencies like in the temporal constraint network given in �gure 13.Anyhow, the process of resolving the disjunctions guarantees that all possible solutions, if there areany, will be enumerated. Despite its computational complexity | the algorithm is exponential | themethod is useful for our application. We can assume that, the number of intervals in a single conceptis quite small (< 10), because of the hierarchical organization of the knowledge base. Run time is alsonot critical, since the translation only has to be done once and before any event recognition happens.

3General disjunctive relations are not allowed in Naos (c.f. [Neumann 89]), i.e., the system is restricted to theso-called point algebra (c.f. [Vilain & Kautz 86]).

7

'&

$%

?

�����������9

Context 0

:b :m

:b :m :o

'&

$%

Context 1

:m

'&

$%

?

�����������9

XXXXXXXXXXXz

Context 2

:b

:b :m :o

'&

$%

Context 3

:b

'&

$%

Context 4

:m

'&

$%

Context 5

:o

Figure 11: Context tree for the example concept

| PENALTY-KICK-C1 | |- PENALTY-KICK-C3 -|

|RUNUP|SHOOT| |RUNUP| |SHOOT|

|- MOVE -|PARRY| |- MOVE -|PARRY|

0 1 2 3 4 0 1 2 3 4 5

|- PENALTY-KICK-C4 -| |- PENALTY-KICK-C5 -|

|RUNUP| |SHOOT| |RUNUP| |SHOOT|

|- MOVE -|PARRY| |- MOVE -|PARRY|

0 1 2 3 4 0 1 2 3 4 5

Figure 12: Solutions of the temporal constraint network

4.2 Extending the framework

In order to rule out \useless" concept de�nitions, additional general temporal constraints might beapplied to event concepts.

� If a decompositional part of an event could happen after the event itself, it would be di�cultto de�ne a clear semantics. Certain statements about present events could only be made in thefuture.

� If the de�nition of an event concept relies on a decompositional part, that must happen before

the event, the system must not only remember the near past but all instances of a certain typeof event.

� In order to avoid strange e�ects during the recognition and simultaneous description of an eventit would be useful to forbid \gaps" in the decomposition, i.e. there must be no sub-interval ofan event during which nothing has to happen.

8

����

?

:o

��

����

��

��

��=

:s :m

ZZZZZZZZZZZZ~

:s :m

D

������������

:d :di

HHHHHHHj:d :di

B

����A -

:f :�

����C

Figure 13: Inconsistent temporal constraint network

Constraints of this type can easily be added automatically to any concept de�nition. The temporalreasoner than takes the burden to satisfy them.

So far only qualitative temporal aspects of an event and its decompositional parts have been considered.Within the TimeLogic system it is possible to specify minimal and maximal relative durations, i.e.,the length of a duration can be measured with respect to the duration of another interval. TheTemporal Inference Engine described in [Tsang 87] even allows for the speci�cation of minimaand maxima for absolute durations.Durational constraints can force stronger relational constraints and may lead to an inconsistent con-straint network. Allthough relational and duration constraints are propagated to one another in thesystems mentioned above, they can only detect some but not all inconsistent situations caused bydurational constraints. In our approach a di�erent strategy is applied. The handling of durationalconstraints is delayed until all solutions of the (qualitative) temporal constraint network have beencomputed. Finally each possible decomposition is treated individually with respect to the remain-ing metric constraints. As proposed in [Malik & Binford 83] this task is carried out using linearprogramming with the simplex method.

5 Summary

Our approach towards simultaneous scene description emphasizes concurrent image sequence evalua-tion and natural language processing, carried out on an incremental basis, an important prerequisitefor real-time performance. High-level scene analysis as it is employed inVitra is based on incrementalevent recognition using course diagrams.In this contribution we have argued that in fact a dual representation of motion concepts is needed,because events are not only to be recognized. During language production the system has to reasonabout implications of speci�c motion events and the relations between di�erent occurrences. Interval-based event models, based on specialization and temporal decomposition, are best suited for this kindof reasoning. We have shown that, given the latter representation format, the automatic constructionof course diagrams can be seen as a temporal reasoning problem. Based on temporal constraintpropagation techniques a translation procedure is de�ned and the representation formalism can beextended in order to cope with disjunctive temporal relations, general qualitative restrictions, anddurational constraints.The current implementation relies on TimeLogic, a basic temporal reasoning system. A more sophis-

9

ticated temporal reasoner, like the one described in [Tolba et al. 91]4, could even simplify the task,since the solutions of the constraint network can be provided directly and a more advanced treatmentof metric constraints is possible.The methods presented here are applied for the investigation of short sections of soccer games and oftra�c scenes. A large knowledge base, with more than 100 event concepts is under development.

Acknowledgements

Discussions with Dieter Koller, Karl Rohr, Hany Tolba, and the participants of the 4th EuropeanWorkshop \Semantics of Time, Space, and Movement and Spatio-Temporal Reasoning" are gratefullyacknowledged. Michael Paul assisted in carrying out the work described here and provided helpfulcomments and suggestions throughout the writing of this paper.

References

[Allen & Koomen 83] J. F. Allen and J. A. Koomen. Planning Using a Temporal World Model. In:Proc. of the 8th IJCAI, pp. 741{747, Karlsruhe, FRG, 1983.

[Allen 84] J. F. Allen. Towards a General Theory of Action and Time. Arti�cial Intelligence,23(2):123{154, 1984.

[Andr�e et al. 88] E. Andr�e, G. Herzog, and T. Rist. On the Simultaneous Interpretation of Real

World Image Sequences and their Natural Language Description: The System SOCCER. In:Proc. of the 8th ECAI, pp. 449{454, Munich, 1988.

[Badler 75] N. I. Badler. Temporal Scene Analysis: Conceptual Description of Object Movements.Technical Report 80, Computer Science Department, Univ. of Toronto, 1975.

[Heinze et al. 91] N. Heinze, W. Kr�uger, und H.-H. Nagel. Berechnung von Bewegungsverben zur

Beschreibung von aus Bildfolgen gewonnenen Fahrzeugtrajektorien in Stra�enverkehrsszenen.Informatik { Forschung und Entwicklung, 6:51{61, 1991.

[Herzog et al. 93] G. Herzog, W. Maa�, and P. Wazinski. VITRA GUIDE: Utilisation du Lan-

gage Naturel et de Repr�esentation Graphiques pour la Description d'Itin�eraires. In: ColloqueInterdisciplinaire du Comit�e National \Images et Langages: Multimodalit�e et Mod�elisationCognitive", pp. 243{251, Paris, 1993.

[Herzog 92] G. Herzog. Visualization Methods for the VITRA Workbench. Memo 53, Universit�atdes Saarlandes, SFB 314 (VITRA), 1992.

[Hoeppner et al. 83] W. Hoeppner, T. Christaller, H. Marburger, K. Morik, M. O'Leary, andW.Wahlster. Beyond Domain-Independence: Experience with the Development of a German

Language Access System to Highly Diverse Background Systems. In: Proc. of the 8th IJCAI,pp. 588{594, Karlsruhe, FRG, 1983.

[Koller et al. 92] D. Koller, K. Daniilidis, T. Th�orhallson, and H.-H. Nagel. Model-based Object

Tracking in Tra�c Scenes. In: G. Sandini (ed.), Proc. of Second European Conf. on ComputerVision, pp. 437{452. Berlin, Heidelberg: Springer, 1992.

[Koller 92] D. Koller. Detektion, Verfolgung und Klassi�kation bewegter Objekte in monokularen

Bildfolgen am Beispiel von Stra�enverkehrsszenen. St. Augustin: In�x, 1992.

4The system is now called TemPro (personal communication).

10

[Kollnig 92] H. W. Kollnig. Berechnung von Bewegungsverben und Ermittlung einfacher Abl�aufe.Diplomarbeit, Fakult�at f�ur Informatik, Univ. Karlsruhe, 1992.

[Koomen 87] J. A. G. Koomen. The TIMELOGIC Temporal Reasoning System. Technical Report231, Department of Computer Science, Univ. of Rochester, NY, 1987.

[Malik & Binford 83] J. Malik and T. O. Binford. Reasoning in Time and Space. In: Proc. of the8th IJCAI, pp. 343{345, Karlsruhe, FRG, 1983.

[Neumann 89] B.Neumann. Natural Language Description of Time-Varying Scenes. In: D. L. Waltz(ed.), Semantic Structures: Advances in Natural Language Processing, pp. 167{207. Hillsdale,NJ: Lawrence Erlbaum, 1989.

[Novak 87] H.-J. Novak. Textgenerierung auf der Grundlage visueller Daten: Beschreibungen von

Stra�enszenen. Berlin, Heidelberg: Springer, 1987.

[Okada 79] N. Okada. SUPP: Understanding Moving Picture Patterns Based on Linguistic Knowl-

edge. In: Proc. of the 6th IJCAI, pp. 690{692, Tokio, Japan, 1979.

[Rohr & Nagel 90] K. Rohr und H. H. Nagel. Modellgest�utzte Bestimmung des Bewegungszustandes

von Fu�g�angern in Realweltbildfolgen. In: R. E. Gro�kopf (Hrsg.), Mustererkennung 1990, pp.52{58. Berlin, Heidelberg: Springer, 1990.

[Rohr 89] K.Rohr. Auf dem Wege zu modellgest�utztem Erkennen von bewegten nicht-starren K�orpern

in Realweltbildfolgen. In: H. Burkhardt, K. H. H�ohne, und B. Neumann (Hrsg.), Mustererken-nung 1989, pp. 324{328. Berlin, Heidelberg: Springer, 1989.

[Schirra et al. 87] J. R. J. Schirra, G. Bosch, C.-K. Sung, and G. Zimmermann. From Image

Sequences to Natural Language: A First Step Towards Automatic Perception and Description

of Motions. Applied Arti�cial Intelligence, 1:287{305, 1987.

[Tolba et al. 91] H. Tolba, F. Charpillet, and J. P. Haton. Representing and Propagating Con-

straints in Temporal Reasoning. AI Communications, 4(4):145{151, 1991.

[Tsang 87] E. P. K. Tsang. Planning in a Temporal Frame: A Partial World Description Approach.PhD thesis, Department of Computer Science, Univ. of Essex, Colchester, UK, 1987.

[Tsotsos 85] J. K.Tsotsos.Knowledge Organization and its Role in Representation and Interpretationfor Time-Varying Data: the ALVEN System. Computational Intelligence, 1:16{32, 1985.

[Vilain & Kautz 86] M. B. Vilain and H. Kautz. Constraint Propagation Algorithms for Temporal

Reasoning. In: Proc. of AAAI-86, pp. 377{382, Philadelphia, PA, 1986.

[Walter 89] I. Walter. Datenbankgest�utzte Repr�asentation und Extraktion von Episodenbeschreibun-

gen aus Bildfolgen. Berlin, Heidelberg: Springer, 1989.

[Wazinski & Herzog 92] P.Wazinski and G.Herzog. Coping with Topological and Directional Rela-

tions Based on 3-Dimensional Geometric Representations. In: Spatial Concepts: ConnectingCognitive Theories with Formal Representations, Workshop Notes, 10th European Conf. onArti�cial Intelligence, Vienna, Austria, 1992.

[Woods 70] W. A. Woods. Transition Network Grammar for Natural Language Analysis. Commu-nications of the ACM, 13(10):591{606, 1970.

11