Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 162–171,Uppsala, Sweden, 15-16 July 2010. c©2010 Association for Computational Linguistics

Cross-caption coreference resolution for automatic image understanding

Micah Hodosh Peter Young Cyrus Rashtchian Julia HockenmaierDepartment of Computer Science

University of Illinois at Urbana-Champaign{mhodosh2, pyoung2, crashtc2, juliahmr}@illinois.edu

AbstractRecent work in computer vision has aimedto associate image regions with keywordsdescribing the depicted entities, but ac-tual image ‘understanding’ would also re-quire identifying their attributes, relationsand activities. Since this information can-not be conveyed by simple keywords, wehave collected a corpus of “action” photoseach associated with five descriptive cap-tions. In order to obtain a consistent se-mantic representation for each image, weneed to first identify which NPs refer tothe same entities. We present three hierar-chical Bayesian models for cross-captioncoreference resolution. We have also cre-ated a simple ontology of entity classesthat appear in images and evaluate howwell these can be recovered.

1 Introduction

Many photos capture a moment in time, telling abrief story of people, animals and objects, their at-tributes, and their relationship to each other. Al-though different people may give different inter-pretations to the same picture, people can read-ily interpret photos and describe the entities andevents they perceive in complex sentences. Thislevel of image understanding still remains an elu-sive goal for computer vision: although currentmethods may be able to identify the overall scene(Quattoni and Torralba, 2009) or some specificclasses of entities (Felzenszwalb et al., 2008), theyare only starting to be able to identify attributesof entities (Farhadi et al., 2009), and are far fromrecovering a complete semantic interpretation ofthe depicted situation. Like natural language pro-cessing, computer vision requires suitable trainingdata, and there are currently no publicly availabledata sets that would enable the development ofsuch systems.

Photo sharing sites such as Flickr allow usersto annotate images with keywords and other de-scriptions, and vision researchers have access tolarge collections of images annotated with key-words (e.g. the Corel collection). A lot of recentwork in computer vision has been aimed at pre-dicting these keywords (Blei et al., 2003; Barnardet al., 2003; Feng and Lapata, 2008; Deschachtand Moens, 2007; Jeon et al., 2003). But key-words alone are not expressive enough to capturerelations between entities. Some research has usedthe text that surrounds an image in a news arti-cle as a proxy (Feng and Lapata, 2008; Deschachtand Moens, 2007). However, in many cases, thesurrounding text or a user-provided caption doesnot simply describe what is depicted in the image(since this is usually obvious to the human readerfor which this text is intended), but provides ad-ditional information. We have collected a corpusof 8108 images associated with several simple de-scriptive captions. In contrast to the text near animage on the web, the captions in our corpus pro-vide direct, if partial and slightly noisy, descrip-tions of the image content. Our data set differsfrom paraphrase corpora (Barzilay and McKeown,2001; Dolan et al., 2004) in that the different cap-tions of an image are produced independently bydifferent writers. There are many ways of describ-ing the same image, because it is often possibleto focus on different aspects of the depicted situ-ation, and because certain aspects of the situationmay be unclear to the human viewer.

One of our goals is to use these captions toobtain a semantic representation of each imagethat is consistent with all of its captions. In or-der to obtain such a representation, it is neces-sary to identify the entities that appear in the im-age, and to perform cross-caption coreference res-olution, i.e. to identify all mentions of the sameentity in the five captions associated with an im-age. In this paper, we compare different meth-

162

A golden retriever (ANIMAL) is playing with a smaller black and brown dog(ANIMAL) in a pink collar (CLOTHING).A smaller black dog (ANIMAL) is fighting with a larger brown dog (ANIMAL) in a forest (NAT_BACKGROUND).A smaller black and brown dog (ANIMAL) is jumping on a large orange dog (ANIMAL).Brown dog (ANIMAL) with mouth (BODY_PART) open near head(BODY_PART) of black and tan dog (ANIMAL).Two dogs (ANIMAL) playing near the woods (NAT_BACKGROUND).

Figure 1: An image with five captions from our corpus. Coreference chains and ontological classes areindicated in color.

ods of cross-caption coreference resolution on ourcorpus. In order to facilitate further computer vi-sion research, we have also defined a set of coarse-grained ontological classes that we use to automat-ically categorize the entities in our data set.

2 A corpus of action images and captions

Image collection and sentence annotation Wehave constructed a corpus consisting of 8108 pho-tographs from Flickr.com, each paired with fiveone-sentence descriptive captions written by Ama-zon’s Mechanical Turk1 workers. We downloadeda few thousand images from each of six selectedFlickr groups2. To facilitate future computer vi-sion research on our data, we filtered out images inblack-and-white or sepia, as well as images withwatermarks, signatures, borders or other obviousediting. Since our collection focuses on imagesdepicting actions, we then filtered out images ofscenery, portraits, and mood photography. Thiswas done independently by two members of ourgroup and adjudicated by a third.

We paid Turk workers $0.10 to write one de-scriptive sentence for each of five distinct and ran-domly chosen images that were displayed one ata time. We required a small qualification testthat examined the workers’ English grammar andspelling and we restricted the task to U.S. work-ers (see Rashtchian et al. (2010) for more details).Our final corpus contains five sentences for eachof our 8108 images, totaling 478,317 word tokens,and an average sentence length of 11.8 words.We first spell-checked3 these sentences, and usedOpenNLP4 to POS-tag them. We identified NPsusing OpenNLP’s chunker, followed by a semi-

1https://www.mturk.com2The groups:“strangers!”, “Wild-Child (Kids in Action)”,

“Dogs in Action (Read the Rules)”, “Outdoor Activities”,“Action Photography”, “Flickr-Social (two or more people inthe photo)”.

3We used Unix’s aspell to generate possible correc-tions and chose between them based on corpus frequencies.

4http://opennlp.sourceforge.net

automatic procedure to correct for a number ofsystematic chunking errors that could easily becorrected. We randomly selected 200 images forfurther manual annotation, to be used as test anddevelopment data in our experiments.

Gold standard coreference annotation Wemanually annotated NP chunks, ontologicalclasses, and cross-caption coreference chains foreach of the 200 images in our test and developmentdata. Each image was annotated independently bytwo annotators and adjudicated by a third.5 Thedevelopment set contains 1604 mentions. On av-erage, each caption has 3.2 mentions, and each im-age has 5.9 coreference chains (distinct entities).

Ontological annotation of entities In order tounderstand the role entities mentioned in the sen-tences play in the image, we have defined a simpleontology of entity classes (Table 1). We distin-guish entities that constitute the background of animage from those that appear in the foreground.These entities can be animate (people or animals)or inanimate. For inanimate objects, we distin-guish static objects from “movable” objects. Wealso distinguish man-made from natural objectsand backgrounds, since this matters for computervision algorithms. We have labeled the entitymentions in our test and development data withclasses from this ontology. Again, two of us an-notated each image’s mentions, and adjudicationwas performed by a single person. Our ontologyis similar to, but smaller than the one proposedby Hollink and Worring (2005) for video retrieval,which in turn is based on Hoogs et al. (2003) andHunter (2001).

3 Predicting image entities from captions

Figure 1 shows an image from our corpus. Dif-ferent captions use different words to refer to the

5We used MMAX2 (Muller and Strube, 2006) both forannotation and adjudication.

163

Ontological Class Examplesanimal dog, horse, cowbackground man-made street, pool, carpetbackground natural ocean, field, airbody part hair, mouth, armsclothing shirt, hat, sunglassesevent trick, sunset, gamefixed object man-made furniture, statue, rampfixed object natural rock, puddle, bushimage attribute camera, picture, closeupmaterial man-made paint, frostingmaterial natural water, snow, dirtmovable man-made ball, toy, bowlmovable natural leaves, snowballnondepictable something, Batmanorientation front, top, [the] distancepart of edge, side, top, tiresperson family, skateboarderproperty of shadow, shade, themevehicle surfboard, bike, boatwriting graffiti, map

Table 1: Our ontology for entities in images.

same entity, or even seemingly contradictory mod-ifiers (“orange” vs. “brown” dog). In order topredict what entities appear in an image from itscaptions, we need to identify how many entitieseach sentence describes, and what role these enti-ties play in the image (e.g. person, animal, back-ground). Because we have five sentences asso-ciated with each image, we also need to identifywhich noun phrases in the different captions ofthe same image refer to the same entity. Becausethe captions were generated independently, thereare no discourse cues such as anaphora to identifycoreference. This creates problems for standardcoreference resolution systems trained on regulartext. Our data also differs from standard corefer-ence data sets in that entities are rarely referred toby proper nouns.

Our first task is to identify which noun phrasesmay refer to the same entity. We do this by identi-fying the set of entity types that each NP may referto. We use WordNet (Fellbaum, 1998) to iden-tify the possible entity types (WordNet synsets) ofeach head noun. Since the salient entities in eachimage are likely to be mentioned by more than onecaption writer, we then aim to restrict those typesto those that may be shared by some head nouns inthe other captions of the same image. This givesus an inventory of entity types for each mention,which we use to identify coreferences, restrictedby the constraint that all coreferent mentions referto an entity of the same type.

4 Using WordNet to identify entity types

WordNet (Fellbaum, 1998) provides a rich ontol-ogy of entity types that facilitates our coreferencetask.6 We use WordNet to obtain a lexicon of pos-sible entity types for each mention (based on theirlexical heads, assumed to be the last word with anominal POS tag7). We first generate a set of can-didate synsets based solely on the lexical heads,and then generate lexicon entries based on rela-tions between the candidates.

WordNet synsets provide us with synonyms,and hypernym/hyponym relations. For each men-tion, we generate a list of candidate synsets.We require that the candidates are one of thefirst four synsets reported and that their fre-quency is to be at least one-tenth of the mostfrequent synset. We limit candidates to oneswith “physical entity#n#1”, “event#n#1”, or “vi-sual property#n#1” as a hypernym, in order to en-sure that the synset describes something that is de-pictable. To avoid word senses that refer to a per-son in a metaphorical fashion, (e.g. pig meaningslovenly person or red meaning communist), weignore synsets that refer to people if the word hasa synset that is an animal or color.8

In general, we would like for mentions to beable to take on more specific word senses. For ex-ample, we would like to be able to identify that“woman” and “person” may refer to the sameentity, whereas “man” and “woman” typicallywould not. However, we also do not want a typeinventory that is too large or too fine-grained.

Once the candidate synsets are generated, weconsider all pairs of nouns (n1, n2) that occur indifferent captions of the same image and exam-ine all corresponding pairs of candidate synsets(s1, s2). If s2 is a synonym or hypernym of s1, itis possible that two captions have different wordsdescribing the same entity, so we add s1 and s29

to the lexicon of n1. Adding s2 to n1’s lexicon al-lows it to act as an umbrella sense covering othernouns describing the same entity.10 We add s2 to

6For the prediction of ontological classes, we use our ownontology because WordNet is too fine-grained for this pur-pose.

7If there are two NP chunks that form a “[NP ... group] of[NP... ]” construction, we only use the second NP chunk.

8An exception list handles cases (diver, blonde), where thehuman sense is more likely than the animal or color sense.

9We don’t add s2 if it is “object#n#1” or “clothing#n#1”.10This is needed when captions use different aspects of

the entity to describe it (for example, “skier” and “a skiingman”).

164

the lexicon of n2 (since if n1 is using the sense s1,then n2 must be using the sense s2) and if n1 oc-curs at least five times in the corpus, we add s1 tothe lexicon of n2.

5 A heuristic coreference algorithm

Based on WordNet candidate synsets, we definea heuristic algorithm that finds the optimal entityassignment for the mentions associated with eachimage. This algorithm is based on the principlesdriving our generative model described below, andon the observation that salient entities will be men-tioned in many captions and that captions tend touse similar words to describe the same entity.

Simple heuristic algorithm:

1. For each noun, choose the synset that appearsin the most number of captions of an image,and break ties by choosing the synset thatcovers the fewest distinct lemmatized nouns.

2. Group all of the noun phrase chunks thatshare a synset into a single coreference chain.

6 Bayesian coreference models

Since we cannot afford to manually annotate ourentire data set with coreference information, wefollow Haghighi and Klein (2007)’s work on un-supervised coreference resolution, and develop aseries of generative Bayesian models for our task.

6.1 Model 0: Simple Mixture Model

In our first model, based on Haghighi and Klein’sbaseline Dirichlet Process model, each image icorresponds to the set of observed mentions wi

from across its captions. Image i has a hiddenglobal topic Ti, drawn from a distribution with aGEM prior with hyperparameter γ as explained byTeh et al. (2006). In a Dirichlet process, the GEMdistribution is an infinite analog of the Dirich-let distribution, allowing for a potentially infinitenumber of mixture components. P (Ti = t) is pro-portional to γ if t is a new component, or to thenumber of times t has been drawn before other-wise. Given a topic choice Ti = t, entity typeassignments Zj for all mentions wj in image iare in turn drawn from a topic-specific multino-mial θt over all possible entity types E that wasdrawn from a Dirichlet prior with hyperparameterβ. Similarly, given an entity type Zi = z, eachcorresponding (observed) head word wj is drawn

from an entity type-specific multinomial φz overall possible words V, drawn from a finite Dirich-let prior with hyperparameter α. The set of all im-ages belonging to the same topic is analogous toan individual document in Haghighi and Klein’sbaseline model.11 All headwords of the same en-tity type are assumed to be coreferent, similar toHaghighi and Klein’s model. As described in sec-tion 4, we use WordNet to identify the subset oftypes that can actually produce the given words.Therefore, similar to the way Andrzejewski andZhu (2009) handled a priori knowledge of topics,we will define an indicator variable δij that is 1iff the WordNet information allows word i to beproduced from entity set j and 0 otherwise.

6.1.1 Sampling Model 0We find argmaxZ,TP (Z,T|X) with Gibbs sam-pling. Here, Z and T are the collection of typeand topic assignments, with Z−j = Z− {Zj} andT−i = T − {Ti}. This style of notation will beextended analogously to other variables. Let ne,x

represent the number of times word x is producedfrom entity e across all topics and let pj be thenumber of images assigned to topic j. Let mt,e

represent the number of times entity type e isgenerated by topic t. Each iteration consists oftwo steps: first, each Zi is resampled, fixing T;and then each Ti is resampled based on Z12.

1. Sampling Zj:

P (Zj =e|wj ∈ wi,Z−j ,T) ∝ P (wj |Zj=e)P (Zj =e|Ti)

P (wj = x|Zj = e) ∝„

n−je,x + αP

x′ n−je,x′ + α

«δxe

P (Zj = e|Ti = t) =m−j

t,e + βPe′ m

−jt,e′ + β

2. Sampling Ti:

P (Ti =j|w,Z,T−i) ∝ P (Ti =j|T−i)P (Z|Ti =j,T−i)

∝ P (Ti = j|T−i)Y

k∈wi

P (Zk|Ti = j)

= P (Ti = j|T−i)Y

k∈wi

m−ij,Zk

+ βPe′ m

−ij,e′ + β

P (Ti = j|T−i) ∝

(γ, If its a new topicpj Otherwise

11Since we do not have multiple images of the same well-known people or places, referred to by their names, we do notperform any cross-image coreference

12Sampling on the exponentiated posterior to find the modeas Haghighi and Klein (2007) did was found to not signifi-cantly affect results on our tasks

165

Caption 1: {x21,1:a golden retriever; x21,2 :a smaller black and brown dog; x21,3:a pink collar}Caption 2: {x21,4:a smaller black dog; x21,5:a larger brown dog; x21,6:a forest}Caption 3: {x21,7:small black and brown dog; x21,8:a large orange dog}Caption 4: {x21,9:brown dog; x21,10:mouth; x21,11:head; x21,12:black and tan dog}Caption 5: {x21,13:two dogs; x21,14:the woods}

DOGattr:5

DOGattr:3

CLOTHINGattr:2

FORESTattr:8

Image 21:

x21,1x21,5

x21,8x21,9

MOUTHattr:0 ...

x21,2 x21,4

x21,7

x21,12

x21,3 x21,6

x21,14

x21,10 Restaurant 211

Figure 2: Models 1 and 2 as Chinese restaurant franchises: each image topic is a franchise, each imageis a restaurant, each entity is a table, each mention is a customer. Model 2 adds attributes (in italics).

6.2 Model 1: Explicit Entities

Model 0 does not have an explicit representationof entities beyond their type and thus cannot dis-tinguish multiple entities of the same type in animage. Although Model 1 also represents men-tions only by their head words (and thus cannotdistinguish black dog from brown dog), it createsexplicit entities based on the Chinese restaurantfranchise interpretation of the hierarchical Dirich-let Process model (Teh et al., 2006). Figure 2 (ig-noring the modifiers / attributes for now) illustratesthe Chinese restaurant franchise interpretation ofour model. Using this metaphor, there are a se-ries of restaurants (= images), each consisting ofa potentially infinite number of tables (= entities),that are frequented by customers (= entity men-tions) who will be seated at the tables. Restau-rants belong to franchises (= image topics). Eachtable is served one dish (= entity type, e.g. DOG,CLOTHING) shared by all the customers. The headword of a mention xi,j is generated in the follow-ing manner: customer j enters restaurant i (be-longing to franchise Ti) and sits down at one ofthe existing tables with probability proportional tothe number of other customers there, or sits at anew table with probability proportional to a con-stant. A dish eia (DOG) from a potentially infinitemenu is served to each new table a, with probabil-ity proportional to the total number of tables it isserved at in the franchise Ti (or to a constant if itis a new dish). The (observed) head word of themention xj,i (dog, retriever) is then drawn fromthe multinomial distribution over words defined bythe entity type (DOG) at the table. The menu (setof dishes) available to each restaurant and table isrestricted by our lexicon of WordNet synsets foreach mention. More formally, each image topict defines a distribution over entities drawn from aglobal GEM prior with hyperparameter κ. There-

fore, the probability of an entity a is proportionalto the number of its existing mentions in images ofthe same topic, or to κ, if it is previoulsy unmen-tioned. The type of each entity, ea, is drawn froma topic-dependent multinomial with global Dirich-let prior. The head words of mentions are gener-ated by their entity type as in Model 0. Mentionsassigned to the same entity are considered to becoreferent. Based on the nature of our corpus, weagain assume that two words cannot be coreferentwithin a sentence, restrict the distribution to notallow inter-sentence coreference and renormalizethe values accordingly.

6.2.1 Sampling Model 1There are three parts to our resampling procedure:resampling the entity assignment for each word,resampling the entity type for each entity, and re-sampling the topic of each image. The kth wordof image i, sentence j, will now be wi

j,k; eia is theentity type of entity a in image i; ai

j,k is the en-tity that word k of sentence j is produced from inimage i, and Zi

j,k represents that entity’s type. ais the set of all current entity assignments and eare the type assignments for entities. m is now de-fined as the number of entities of a certain type be-ing drawn for an image, n is defined as before andci,a is the number of times entity a is expressed inimage i. Topics are resampled as in Model 0.

Entity Assignment Resampling Entity assign-ments for words are resampled one sentence at atime in the order the headwords appear in the sen-tence. For each word in the sentence, entity as-signments are defined by the distribution of Fig-ure 3. The headword is assigned to an existingentity with probability proportional to the numberof entities already assigned to that entity and theprobability that the entity emits that word. Theword is assigned to a new entity with a newly

166

Model 1:

P (eia =e|a,w, Ti = t, e−i,a) ∝ (m−ei,a

t,e + β)Q{wi

j,k=x|ai

j,k=a}

n−ei,a

e,x + αPy(n−ei,a

e,y + α)δx,e

P (aij,k =a|a−(i,j,k′|k′≥k), e,T) ∝

(c−j,k′

i,a P (wij,k|Zi

j,k = eia,a−(i,j,k′|k′≥k))ρi

j,a, if a is not newκP (ei

a|e−i,a, Ti)P (wij,k|Zi

j,k = eia,a−(i,j,k′|k′≥k)), o/w

With:

P (wij,k =x|Zi

j,k =e,a−(i,j,k′|k′≥k)) ∝ n−(i,j,k′|k′≥k)e,x + αP

y(n−(i,j,k′|k′≥k)e,y + α)

δx,e

P (eia =e|e−i,a, Ti = t) ∝

m−ei,a

t,e + βPe′(m

−ei,a

t,e′ + β)Model 2:

P (aij,k =a|a−(i,j,k′|k′≥k), e,T,b) ∝

(c−j,k′

i,a P (wij,k|Zi

j,k = eia,a−(i,j,k′|k′≥k))ρi

j,aP (dij,k|bia), if a is not new

κP (eia|e−i,a, Ti)P (wi

j,k|Zij,k = ei

a,a−(i,j,k′|k′≥k))P (bia)P (di

a|bia), o/w

P (bia =b|Dia,b−i,a)∝P (bia =b|b−i,a)

Qd∈Di

a

(s−i,ab,d + ζ)P

d′(s−i,ab,d′ + ζ)

Figure 3: Sampling equations for Models 1 and 2

drawn entity type with probability proportional κ,the probability that the entity type is for an im-age of the given topic (normalized over WordNet’spossible entities for the word), and the probabilitythe drawn type produces the word. ρi

j,a = 1 iffentity a of image i does not appear in sentence jand ρi

j,a = 0 otherwise. a−(i,j,k′|k′≥k) representsremoving the kth or later words in sentence j ofimage i

Entity Type Resampling Fixing the assign-ments, the type of each entity is redrawn basedon the distribution in Figure 3. It is proportionalto the probability that a certain entity type is in animage of a given topic and, independently for eachof the words, the probability that the given wordexpresses the type. n−ei,a

e,x is the number of timesentity type e is expressed as word x not countingthe words attached to the currently entity being re-sampled and m−ei,a

t,e is the number of times an en-tity of type e appears in an image of topic t notcounting the current entity being resampled. Theprobability of a given image belonging to a topic isproportional the number of images already in thetopic (or γ) followed by the probability that eachof the entities in the image were drawn from thattopic.

6.3 Model 2: Explicit Entities and ModifiersCertain entities cannot be distinguished simply byhead word alone, such in the example in Figure 2.Model 2 augments Model 1 with the ability to gen-erate modifiers. In addition to an entity type, eachentity draws an attribute from a global distributiondrawn from a GEM distribution with hyperparam-

eter η. An attribute is a multinomial distribution,on possible modifier words, drawn from a Dirich-let prior with parameter ζ. From the attribute, eachmodifier word is drawn independently. There-fore given an attribute b and a set of modifiers d:P (d|b) ∝

∏d∈d(sd + ζ) where sd is number of

times modifier d is produced by attribute b. In ad-dition, the probability of a certain attribute b givenall other assignments is given by:

P (bia = b|b−i,a) ∝

(η, If its a new attributerb, Otherwise

where rb is the number of entities with at-tribute b. As in Model 1, mentions assigned tothe same entity are considered coreferent. Con-sider the “smaller black dog” mention in Figure 2.When the mention is being resampled, the at-tribute choice for each table will bias the probabil-ity distribution towards the table whose attributeis more likely to produce “smaller” and “black”.Therefore, the model can now better distinguishthe two dogs in the image.

6.3.1 Sampling Model 2The addition of modifiers only directly effects thedistribution when resampling entity assignmentssince attributes are independent of entity types,image topics, and headwords of noun phrases. Thesampling distribution are again shown in Figure 3.In a separate sampling step, it is now necessary toresample the attribute assigned to each entity: Theprobability of drawing a certain attribute is illus-trated in Figure 3 with Di

a as the set of all the mod-ifiers of all the noun phrases currently assigned to

167

entity a of image i, and s−i,ab,d as the number of

times attribute b produces modifier d without thecurrent assignment of entity a of image i.

6.4 ImplementationThe topic assignments for each image are initial-ized to correspond to the Flickr groups the imageswere taken from. Each mention was initialized asits own entity with type and attribute sampled froma uniform distribution.

As our training is unsupervised, each of themodels were ran on the entire dataset. For Model0, after burn-in, 20 samples of Z were takenspaced 200 iterations apart, while for Model 1samples were taken spaced 100 apart, and 25 apartfor Model 2. The implementation of Model 2 rantoo slow to effectively judge when burn in oc-curred, impacting the results.

The values of parameters α, β, γ, κ, η, ζ, andthe number of initial attributes were hand-tunedbased on the average performance on our anno-tated development subset of 100 images.13

7 Evaluation of coreference resolution

We evaluate each of the generative models and theheuristic coreference algorithm on the annotatedtest subset of our corpus consisting of 100 imageswith both the OpenNLP chunking and the goldstandard chunking. We report our scores basedon the MUC evaluation metric. The results arereported in Table 2 as the average scores acrossall the samples of two independent runs of eachmodel. We also present results on Model 0 with-out using WordNet where every word can be anexpression of one of 200 fake entity sets. Thesame table also shows the performance of a base-line model and the upper bound on performanceimposed by WordNet.

A baseline model: In our baseline model, twonoun phrases in captions of the same image arecoreferent if they share the same head noun.

Upper bound on performance: AlthoughWordNet synsets provide a good indication ofwhether two mentions can refer to the sameentity or not, they may also be overly restrictivein other cases. We measure the upper boundon performance that our reliance on WordNetimposes by finding the best-scoring coreferenceassignment that is consistent with our lexicon.

13(0.1, 0.1, 100, 0.001875, 100, 0.0002, 20) respectively.

This achieves an F-score of 90.2 on the test datawith gold chunks.

Performance increases in each subsequentmodel. The heuristic beats each of the models, butin some sense it is an extreme version of Model1. Both it and Model 1 attempt to produce en-tity sets that cover as many captions as possible,while minimizing the number of distinct words in-volved. The heuristic locally forces this case, atthe expense of no longer being a generative model.

8 Ontological Class Prediction

As a further step towards understanding the se-mantics of images, we develop a model that labelseach entity with one of the ontological classes de-fined in section 2. The immediate difficulty of thistask is that our ontology includes not only seman-tic distinctions, but also spatial and visual ones.While it may be easy to tell which words are an-imals and which are people, there is only a finedistinction at the language level whether an objectis movable, fixed, or part of the background.14

8.1 Model and Features

We define our task as a classification problem,where each entity must be assigned to one oftwenty classes defined by our ontology. We use aMaximum Entropy classifier, implemented in theMALLET toolkit (McCallum, 2002), and definethe following text features:

NP Chunk: We include all the words in the NPchunk, unfiltered.

WordNet Synsets and Hypernyms: The mostlikely synset is either the first one that appears inWordNet or one of the ones predicted by our coref-erence system. For each of these possibilities, weinclude all of that synset’s hypernyms.

Syntactic Role: We parsed our captions with theC&C parser (Clark and Curran, 2007), and recordwhether the word appears as a direct object of averb, as the object of a preposition, as the subjectof the sentence, or as a modifier. If it is a modi-fier, we also add the head word of the phrase beingmodified.

14For example, we deem bowls and silverware to be mov-able objects; furniture, fixed; and carpets, background. More-over, in all three cases, we must correctly distinguish thatthese objects are man-made and not found in nature.

168

Model OpenNLP chunks Gold chunksRec. Prec. F1 Rec. Prec. F1

Baseline 57.3 89.5 69.9 64.1 96.2 77.0Upper bound 82.1 100 90.2WN Heuristic 70.6 84.8 77.0 80.4 90.6 85.2Model 0 w/o WN 79.7 59.8 68.4 85.1 62.7 72.2Model 0 66.8 83.1 74.1 75.9 90.3 82.5Model 1 69.6 83.8 76.0 78.0 90.8 83.9Model 2 69.2 84.4 76.1 77.9 91.5 84.1

Table 2: Coreference resolution results (MUC scores; Models 0-2 are averaged over all samples)

8.2 ExperimentsWe use two baselines. The naive baseline catego-rizes words by selecting the most frequent classof the word. If no instances of the word have oc-curred, it uses the overall most frequent class. TheWordNet baseline works by finding the most fre-quent class amongst the most relevant synsets fora word. It calculates the class frequency for eachsynset by assuming each word has the sense of itsfirst synset and incrementing the frequency of thefirst synset and its hypernyms. When categorizinga word, it finds the set of closest hypernyms of theword that have a non-zero frequency, and choosesthe class with the greatest sum of frequency countsamongst those hypernyms.

We train the MaxEnt classifier using semi-supervised learning. Initially, we train a classifierusing the 500 sentence gold standard developmentset. For each class, we use the top 5%15 of the la-bels to label the unlabeled data and provide addi-tional training data. We then retrain the classifieron the newly labeled examples and the develop-ment set, and run it on the test set. For each coref-erence chain in the test set, we relabel all of thementions in the chain to use the majority class, ifa clear majority exists. If no such majority exists,we leave the labels as is. The MaxEnt classifierexperiments were conducted by varying the sourceof the synset assigned to each word. For each ofour coreference systems, we report two scores (Ta-ble 3). The first is the average accuracy when us-ing the output from two runs of each model withabout 20 samples per run, and the second uses theoutput that performs best on the coreference taskwhen scored on the development data.

Discussion Although we use WordNet to clas-sify our entity mentions, we designed our ontologyby considering only the images and their captions,with no particular mapping to WordNet in mind.

15This was tuned using 10-fold cross-validation of the de-velopment set.

Classifier (synset prediction) Accuracy (gold chunks)Naive Baseline 72.0WordNet Baseline 81.0MaxEnt (1st-synset) 84.4MaxEnt (WN heuristic) 82.7

Avg. σ Best-CorefMaxEnt (Model 1) 83.9 0.5 84.5MaxEnt (Model 2) 84.1 0.4 85.3

Table 3: Prediction of ontological classes

Therefore, these experiments provide of a proof ofconcept for the semi-supervised labeling of a cor-pus using any semantic/visual ontology.

Overall, Model 2 had the best performance forthis task. This demonstrates that the additionalfeatures of Model 2 force synset selections that areconsistent across the entire corpus, and are sen-sitive to the modifiers appearing with them. TheWordNet heuristic selects synsets in a fairly ar-bitrary manner - all other things being equal, thesynsets are chosen without reference to what othersynsets are chosen by similar clusters of nouns.

9 Evaluating entity prediction

Together, the coreference resolution algorithm andontology classification model provide us with a setof distinct, ontologically-categorized entities ap-pearing in each image. We perform a final experi-ment to evaluate how well our models can recoverthe mentioned entities and their ontological typesfor each image. We now represent each entity as atuple (L, c), where L is its coreference chain, andc is the ontological class of these mentions. 16

We compute the precision and recall betweenthe predicted and gold standard tuples for each im-age. We consider a tuple (L′, c′) correctly pre-dicted only when a copy of (L′, c′) occurs bothin the set of predicted tuples and the set of goldstandard tuples.17 Then, as usual, for precision we

16Note that for each image, the tuples of all entities corre-spond to a partition of the set of the head-word mentions inan image.

17We assign no partial credit because incorrect typing or

169

Model Recall Precision F-scoreBaseline 28.4 20.6 23.9

WordNet Heuristic 48.3 43.9 46.0Model 1 (avg) 51.7 42.8 46.8

Model 1 (best-coref) 50.9 45.4 48.0Model 2 (avg) 52.2 42.7 47.0

Model 2 (best-coref) 52.3 46.0 49.0

Table 4: Overall entity recovery. We measurehow many entities we identify correctly (requiringcomplete recovery of their coreference chains andcorrect prediction of their ontological class.

normalize the number of overlapping tuples by thenumber of predicted tuples, and for recall, by thenumber of gold standard tuples. We report averageprecision and recall over all images in our test set.

We report scores for four different pairs of on-tological class and coreference chain predictions.As a baseline, we use the ontological classes pre-dicted by the our naive baseline and the chains pre-dicted by the “same-words-are-coreferent” coref-erence resolution baseline.

We also report results using the classes andchains predicted by Model 1, Model 2, and theWordNet Heuristic Algorithm. The influence ofthe different coreference algorithms comes fromthe entity types that are used to determine corefer-ence chains, and that also correspond to WordNetcandidate synsets. In other words, although thefinal coreference chain may be predicted by twodifferent models, the synsets they use to do so maydiffer, affecting the synset and hypernym featuresused for ontological prediction. We present resultsin Table 4 for these four different set-ups.

The synsets chosen by the different corefer-ence algorithms clearly have different applicabil-ity when it comes to ontological class prediction.Although Model 2 performs comparably to Model1 and does worse than the WordNet heuristic al-gorithm for coreference chain prediction, it cer-tainly does better on this task. Since our end goalis creating a unified semantic representation, thisfinal task judges the effectiveness of our models tocapture the most detailed entity information. Thesuccess of Model 2 means that the incorporationof adjectives informs the proper choice of synsetsthat are useful in predicting ontological classes.

10 Conclusion

As a first step towards automatic image under-standing, we have collected a corpus of images as-

incomplete coreference chaining both completely change thesemantics of an image.

sociated with several simple descriptive captions,which provide more detailed information aboutthe image than simple keywords. We plan to makethis data set available for further research in com-puter vision and natural language processing. Inorder to enable the creation of a semantic repre-sentation of the image content that is consistentwith the captions in our data set, we use Word-Net and a series of Bayesian models to performcross-caption coreference resolution. Similar toHaghighi and Klein (2009), who find that linguis-tic heuristics can provide very strong baselines forstandard coreference resultion, relatively simpleheuristics based on WordNet alone perform sur-prisingly well on our task, although they are out-performed by our Bayesian models for overall en-tity prediction. Since our generative models arebased on Dirichlet Process priors, they are de-signed to favor a small number of unique entitiesper image. In the heuristic algorithm, this biasis built in explicitly, resulting in slightly higherperformance on the coreference resolution task.However, while the generative models can useglobal information to learn what entity type eachword is likely to represent, the heuristic is unableto capture any non-local information about the en-tities, and thus provides less useful input for theprediction of ontological classes.

Future work will aim to improve on these re-sults by overcoming the upper bound on perfor-mance imposed by WordNet, and through a moresophisticated model of modifiers. We will also in-vestigate how image features can be incorporatedinto our model to improve performance on entitydetection. Ultimately, identifying the depicted en-tities from multiple image captions will requirenovel ways to correctly handle the semantics ofplural NPs (i.e. that one caption’s “two dogs” con-sist of another’s “golden retreiver” and “smallerblack dog”). We foresee similar challenges whendealing with verbs and events.

The creation of an actual semantic representa-tion of the image content is a challenging problemin itself, since the different captions often focuson different aspects of the depicted situation, orprovide different interpretation of ambiguous sit-uations. We believe that this poses many inter-esting challenges for natural language processing,and will ultimately require ways to integrate theinformation conveyed in the caption with visualfeatures extracted from the image.

170

Acknowledgements

This research was funded by NSF grant IIS 08-03603 INT2-Medium: Understanding the Mean-ing of Images. We are grateful for David Forsythand Dan Roth’s advice, and for Alex Sorokins sup-port with MTurk.

ReferencesDavid Andrzejewski and Xiaojin Zhu. 2009. Latent

Dirichlet allocation with topic-in-set knowledge. InNAACL HLT 2009 Workshop on Semi-SupervisedLearning for Natural Language Processing, pages43–48.

Kobus Barnard, Pinar Duygulu, David Forsyth,Nando De Freitas, David M. Blei, and Michael I.Jordan. 2003. Matching words and pictures. Jour-nal of Machine Learning Research, 3:1107–1135.

Regina Barzilay and Kathleen R. McKeown. 2001.Extracting paraphrases from a parallel corpus. InProceedings of the 39th annual meeting of the Asso-ciation for Computational Linguistics, pages 50–57,Toulouse, France, July.

David M. Blei, Michael I, David M. Blei, and MichaelI. 2003. Modeling annotated data. In Proceedingsof the 26th International ACM SIGIR Conference,pages 127–134.

Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCGand log-linear models. Computational Linguistics,33(4):493–552.

Koen Deschacht and Marie-Francine Moens. 2007.Text analysis for automatic image annotation. InProceedings of the 45th Annual Meeting of the As-sociation of Computational Linguistics.

Bill Dolan, Chris Quirk, and Chris Brockett. 2004.Unsupervised construction of large paraphrase cor-pora: Exploiting massively parallel news sources.In Proceedings of Coling 2004, pages 350–356,Geneva, Switzerland, August. COLING.

A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. 2009.Describing objects by their attributes. In IEEE Con-ference on Computer Vision and Pattern Recogni-tion, pages 1778–1785, June.

Christiane Fellbaum, editor. 1998. WordNet An Elec-tronic Lexical Database. The MIT Press, Cam-bridge, MA ; London, May.

P. Felzenszwalb, D. McAllester, and D. Ramanan.2008. A discriminatively trained, multiscale, de-formable part model. In IEEE Conference on Com-puter Vision and Pattern Recognition, pages 1–8,June.

Yansong Feng and Mirella Lapata. 2008. Automaticimage annotation using auxiliary text information.In Proceedings of ACL-08: HLT, pages 272–280,Columbus, Ohio, June.

Aria Haghighi and Dan Klein. 2007. Unsupervisedcoreference resolution in a nonparametric Bayesianmodel. In Proceedings of the 45th Annual Meet-ing of the Association of Computational Linguistics,pages 848–855, Prague, Czech Republic.

Aria Haghighi and Dan Klein. 2009. Simple coref-erence resolution with rich syntactic and semanticfeatures. In Proceedings of the 2009 Conference onEmpirical Methods in Natural Language Process-ing, pages 1152–1161, Singapore, August. Associ-ation for Computational Linguistics.

L. Hollink and M. Worring. 2005. Building a vi-sual ontology for video retrieval. In MULTIMEDIA’05: Proceedings of the 13th annual ACM interna-tional conference on Multimedia, pages 479–482,New York, NY, USA. ACM.

A. Hoogs, J. Rittscher, G. Stein, and J. Schmiederer.2003. Video content annotation using visual anal-ysis and a large semantic knowledgebase. In IEEEComputer Society Conference on Computer Visionand Pattern Recognition, volume 2, pages II–327 –II–334 vol.2, June.

Jane Hunter. 2001. Adding multimedia to the semanticweb - building an mpeg-7 ontology. In In Interna-tional Semantic Web Working Symposium (SWWS,pages 261–281.

Lavrenko Manmatha Jeon, V. Lavrenko, R. Manmatha,and J. Jeon. 2003. A model for learning the seman-tics of pictures. In Seventeenth Annual Conferenceon Neural Information Processing Systems (NIPS).MIT Press.

Andrew Kachites McCallum. 2002. Mal-let: A machine learning for language toolkit.http://mallet.cs.umass.edu.

Christoph Muller and Michael Strube. 2006. Multi-level annotation of linguistic data with MMAX2. InSabine Braun et al, editor, Corpus Technology andLanguage Pedagogy, pages 197–214. Peter Lang,Frankfurt a.M., Germany.

Ariadna Quattoni and Antonio B. Torralba. 2009.Recognizing indoor scenes. In IEEE Conferenceon Computer Vision and Pattern Recognition, pages413–420. IEEE.

Cyrus Rashtchian, Peter Young, Micah Hodosh, andJulia Hockenmaier. 2010. Collecting image anno-tations using amazons mechanical turk. In NAACLWorkshop Creating Speech and Language Data WithAmazons Mechanical Turk.

Yee Whye Teh, Michael I Jordan, Matthew J Beal, andDavid M Blei. 2006. Hierarchical dirichlet pro-cesses. Journal of the American Statistical Associa-tion, 101(476):1566–1581.

171