virtual and real-world ontology services

Virt

ual W

orld

Arc

hite

ctur

es

46 Published by the IEEE Computer Society 1089-7801/11/$26.00 © 2011 IEEE IEEE INTERNET COMPUTING

V irtual worlds such as Second Life (http://secondlife.com) and Open-Simulator (http://opensimulator.

org) provide a 3D landscape in which user-controlled avatars traverse a shared, multiplayer world, visiting places, creat-ing objects, and selling land and goods to others. Although seve ral architectural variants exist among virtual worlds, the notion of representing a 3D space con-taining terrain, animate, and inanimate objects is common. These 3D models can represent fantasy places or can model real-world locations.

Whereas many applications of vir-tual worlds (socializing, training, meet-ings, and education) involve humans using the vir tual world platform directly, gaming and simulation appli-cations must augment virtual world objects with information from remote data sources. Often, these applications include computational agents that

interact with the environment or other users.

Augmenting objects with additional semantics is similar to the idea of the Semantic Web, which Tim Berners-Lee, James Hendler, and Ora Lassila pro-posed in 2001 as an extension of the World Wide Web to augment webpages with semantic information that intelli-gent agents could understand and use.1 They envisioned that the Semantic Web would require knowledge representa-tion, ontologies, and agents. Around the same time, the MIT Auto-ID Labs coined the term Internet of Things (IoT) to describe the notion of the real world populated with intelligent objects with semantic attributes that can interact with people or autonomous agents.2

Here, we recognize that we can explore IoT ideas using virtual worlds and focus on how virtual world ontologies

Both augmented-reality and virtual world applications must model semantic

knowledge about real- or virtual world objects. The current generation of

virtual world platforms provides limited facilities for representing this kind

of knowledge, but a next generation will provide the means to tie semantic

information to general or application-specific ontology services. This article

motivates the need for ontology services, outlines several approaches for

associating ontology concepts with objects and locations, and discusses how

to populate common-sense ontologies using data harvested from real and

virtual worlds.

Joshua D. Eno and Craig W. ThompsonUniversity of Arkansas

Virtual and Real-World Ontology Services

IC-15-05-Eno.indd 46 8/18/11 12:28 PM


SEPTEMBER/OCTOBER 2011 47

can borrow ideas from both the Semantic Web and the IoT but can also give back — providing a way to model the real world using virtual world data structures and a way to attach semantics to those data structures — to help develop a unified vision we could call a semantic world or smart world in which objects and avatars are associated with knowledge.

Semantics and the Virtual WorldIn our own work, we’ve explored how to use virtual worlds to model the real world in health-care and retail applications and have developed a collection of prototype smart-world applica-tions (http://vw.ddns.uark.edu), including

• an application that tracks virtual world medical supplies in a supply chain with a remote database, recording a track history of objects’ and avatars’ past locations (related search applications can locate objects — for instance, find a wheelchair not in use);

• a mirror-world application in which we use a real-time location service to track real-world RFID-tagged apparel items and then display the same items moving in a virtual world to create a retail store command post;

• an annotation service for allowing any avatar to annotate unlabeled virtual world objects with descriptions or other attributes (price, calorie count, washing instructions, a link to a repair manual, and so on);

• a recommendation service that compares an avatar’s profile and recent locations to rec-ommend similar locations or avatars; and

• a protocol for service discovery such that, when APIs are associated with virtual world objects (such as a thermostat or a baby monitor), lets users discover, download, and remotely control those objects.3

In each case, we observe the same architec-tural design pattern: virtual world objects are augmented with domain-relevant information that’s then utilized by application-specific logic — for example, to make a refrigerator smarter so it knows the food expiration dates or a bus route smarter so riders can see when the next bus will come. A similar design pattern exists for augmented-reality applications, which provide information about nearby locations based on geospatial coordinates or pictures of landmarks (see www.acrossair.com/apps_nearesttube.htm,

www.yelp.com/yelpmobile, or www.google.com/mobile/goggles).

From a computational viewpoint, 3D vir-tual world platforms represent objects that have explicit identity, a graphical representa-tion, optional text labels, and behavior models (often represented by scripts that are triggered by events). In the future, a corresponding real-world computational model could incorporate analogous data structures, which lets us con-sider the real world as one more type of virtual world.

Current mainstream 3D virtual worlds pro-vide minimal support for semantically label-ing objects. Beyond rendering, the Second Life platform itself doesn’t distinguish functionally whether an object is a door or a castle. It provides text labels for objects but no further descriptions, and these labels aren’t tied to semantic concepts. Similarly, the real world doesn’t provide labels — humans can look at an object and recognize it as a chair, as can computers if the object is at a known location, has an RFID tag, or is other-wise digitally identifiable. Humans can then bring to bear other information they know about that object — about its superclasses, its parts, its function, how to repair it, how to operate it, and so on. Some of this is general common-sense knowledge and some might be application- specific, such as the cost of a particular can of okra at a particular market on a particular day. In this article, we call this kind of knowledge an ontology, by which we just mean a data struc-ture for recording various kinds of information including identity, type, supertype, parts, API, and an open-ended collection of attributes and scripts. We can then describe the semantics of an entity in the world by referencing informa-tion from this ontology, allowing computational agents to interact with and reason about the world more effectively.

3D virtual world platforms don’t currently support an ontology capability (beyond plain-text labels), so applications must provide it. But a new generation of virtual world platforms is beginning to provide extensibility mecha-nisms4,5 that can, among other things, provide ontology services.

Ontology Services and Knowledge SourcesIf we had a more consistent semantic labeling for objects, then we could label some things

IC-15-05-Eno.indd 47 8/18/11 12:28 PM

Virtual World Architectures

48 www.computer.org/internet/ IEEE INTERNET COMPUTING

chair and other things table and begin to associate functional specifications to enable computers to reason about them. If humans can look at an object and recognize it as a chair, it would be useful if computers, including, for instance, our smart phones, could also do so using the same conceptual categories.

Let’s reason about a virtual or real-world architecture with a semantic ontology capability:

• Not all real- or virtual world applications need an ontology layer, so such a capability could be structured as an optional plug-in service or services.

• The same ontology content could be use-ful for modeling a virtual world, but also the real world; an ontology service can be agnostic to which world it’s modeling.

• Similarly, an application might not be aware if it’s operating in the real world or a virtual one. We can build applications and test them in virtual worlds before we install them in the real world.

• General-purpose ontologies are a useful starting point and in many cases are suffi-cient, but because the kinds of metadata and relationships between concepts are open-ended, specific applications often require application-specific ontologies that augment or replace general ontologies.

• Finally, if the same ontology is useful for both real and virtual worlds, perhaps we can build it using data from both real- and vir-tual world data sources.

An ontology service must provide well-structured, standard interfaces that can accom-modate multiple sources and uses. Intelligent agents have difficulty using semantic informa-tion when ontology information sources aren’t structured to be readily available, and the inter-faces for accessing information or controlling objects aren’t standardized. Although central-ized services would standardize sources and interfaces, no single source will likely be suf-ficient, so many service providers would offer private ontology services. As an example, a retailer’s ontology could provide pricing infor-mation while product specifications could come from a manufacturer-provided ontology.

Ontologies could be organized or indexed by various means: type-subtype, location in a 3D world, temporally, by context, or by other means.

Augmented-reality applications already dem-onstrate using a smart phone to view an area, and labeling nearby buildings.

A problem in providing an ontology layer is how to populate the ontology and associate con-cepts with entities. One option is to build fully automated recognizers to identify and cate-gorize objects. In important special cases, as when all objects are labeled with RFID tags or barcodes that smart phones can read, machines can use the labels. More generally, humans use image recognition, but more work in image understanding will be needed for machines to generally recognize objects. Additionally, a fully automated system will need to be able to recognize when the ontology must expand to incorporate new types or relationships discov-ered from the environment.

A second option is for content creators to manually associate concepts with entities by labeling the objects they create. This approach is already available in Second Life and Open-Simulator, but in practice creators label only 20 percent of top-level objects, and these labels are inconsistent. A second, manual method (which we prototyped) is to provide an annota-tion service that lets any user label any object with a semantic label, name, description, rec-ommendation, or other property. This approach uses crowd sourcing to populate ontologies, but is still a manual process.

A third option is to use existing ontologies and link these open datasets together.6 For some classes of objects, building a partial ontology can be automated based on existing databases. The WordNet ontology provides a word/concept-level ontology and can represent taxonomic and compositional (ISA and HASPART) relationships.7 DBpedia harvests ontology templates from Wikipedia, representing roughly 300 ontology classes in RDF (including places, people, orga-nizations, species, vehicles, devices, and works), linked to more than 3.5 million things (see http://dbpedia.org).

Another source of explicit, existing ontolo-gies is retailer databases containing SKU des-ignations for thousands of item types — for example, different kinds of chairs and tables — that include corresponding price and other attributes, such as product descriptions. More generally, the Linked Data project seeks to con-nect a wide range of open datasets (see http://linkeddata.org/home). For objects with interfaces,

IC-15-05-Eno.indd 48 8/18/11 12:28 PM



we can further associate API specifications and provide a consistent way for humans or pro-grams to call the APIs — so we could use our smart phones to recognize a nearby object, download its API, and then generate an inter-face that humans or programs could use to query or control the object (a more universal remote). Several protocols that we could asso-ciate with objects to make them into smart(er) objects are explored elsewhere.3

A fourth option is to mine data from the real or virtual world to create or expand ontologies, which we can then add to the broader linked data community. In the real world, RFID, GPS, image, and sensor data are commonly collected to help model particular real-world applications such as supply chain, battle management, or mapping services, like Google Earth. Especially interesting is recognition of daily living activi-ties (such as setting the table) based on object usage (GPS or RFID traces).8

We can also mine similar information from unstructured text on the Web.9 Smart phones open the door wide to harvesting this sort of information from the real world with humans acting as search spiders. With GPS, they can collect and record where a person has been, their communications (voice and email), and, if RFID readers are added to cell phones, traces of all the objects a person passes. A community of humans could harvest a model of locations and movements, potentially providing a fairly dense model of the world. If graphical and other models were associated with the tags, a virtual earth model could be populated and updated in this way.

In practice, a combination of all four ap proaches provides increasingly accurate and useful ontologies and entity associations. Auto-matically associating attributes with entities is more feasible when contextual information can narrow down the possible concepts to improve accuracy. User-provided tags will be more con-sistent and useful if a means exists for associ-ating existing ontology concepts with entities. Existing ontologies will benefit from additional details derived from virtual or real-world data, and data-derived ontologies will be more accu-rate and useful if they’re based on a scaffold of existing ontologies. All these approaches rely on a combination of existing ontologies, data-derived ontological relationships, and user- provided contextual information.

Although we’re just now entering a time when we can gather dense datasets from the real world, we can already do so using 3D vir-tual worlds. To gather descriptive data from virtual worlds, we created a system that harvests data from OpenSimulator and Second Life.10 In our system, avatarbots (program- rather than human-controlled avatars) navigate the virtual world, storing metadata about the locations and objects they encounter. (We could take a similar approach with other virtual worlds, but many are small and data-sparse).

ExperimentsNext, we describe initial steps for exploring how to use the partial, sparse text labeling in 3D virtual worlds to improve and simplify all four approaches for providing ontolo-gies and semantic associations between onto-logical concepts and virtual world entities. For the first approach, we use probabilistic models to help populate currently unlabeled virtual world entity attributes. To improve user- provided semantic information, we provide lists of likely concepts for newly created objects (such as auto-complete for ontologies). We test the feasibility of linking existing ontologies to objects based on unstructured object names and explore methods for expanding existing ontolo-gies with data-derived relationships.

Determining Location ContextHumans rely on context to guide our ability to reason about the world; in the same way, context can improve computers’ ability to rec-ognize and interact with objects in real and virtual worlds. A computer agent (such as an avatarbot that harvests virtual world content) might recognize that solid objects are obstacles to avoid. However, if the agent knows that the object is a door, it could reason that it can be opened rather than avoided and might further recognize that a door on a residential property separates public from private areas, whereas a front door on a commercial property is a public entrance.

In virtual worlds, context can be explicit in cases where property owners have labeled the property as residential or shopping, but rela-tively few owners do so. However, because the difference between a house and a store is obvi-ous to humans, even in a virtual world, users expect other avatars to respect their privacy

IC-15-05-Eno.indd 49 8/18/11 12:28 PM



and stay out of their residential homes. To act correctly in such cases, an agent must use the same kind of contextual clues that humans do, rather than relying on explicit labels. To enable our crawler agents to avoid being intru-sive in residential areas, we developed a clas-sifier that uses location and object metadata to classify locations, even if they aren’t explicitly labeled.

Using the subset of labeled locations, we trained a classification model to classify loca-tions based on their text and region. We used a support vector machine (SVM) classifier,11 which performs well compared to other algo-rithms such as naïve Bayesian and k-nearest neighbor for text classification with large fea-ture spaces.12 Figure 1 shows a comparison of the SVM classifications with labeled locations’ true classifications.

The average classifier accuracy using five-fold validation was 58.2 percent across 13 parcel Second Life classifications, a result that was 60 percent better than the best naïve classifier based just on the underlying parcel type prob-abilities. Once a location’s general purpose is known, the likelihood of finding certain objects changes. In a fully automated system, an object recognition system can use the new contex-tual probabilities to improve object classifica-tion, providing a way to disambiguate a bed (for flowers) in a park from a bed (for sleeping) in a residence.

Labeling Suggestions for UsersOne difficulty in working with user-assigned names and descriptions is that different users might use different terms for the same object type.

Compounding this is the problem of multiple meanings for the same term. A white house could be a brightly painted home, or it might be a specific government building. One way to minimize these issues is to suggest likely semantic labels for an object. If the likely labels have multiple senses in the ontology, the user can further select the sense of the word that’s most appropriate for the object.

Context is again important in providing relevant suggestions. We developed a proto-type annotation system that focuses on resi-dential locations. The system can suggest a set of likely terms based both on the location type and on other objects found near that loca-tion. The system collects the object names that already exist at a location, finds other locations in the system with the same objects, then sug-gests names that best match the existing set of objects.

The annotation system ranks suggested terms based on a relevance score that incor-porates the existing terms’ importance and the frequency with which the suggested terms occur with the existing terms. The rank score for a suggested term is the sum of the condi-tional probability that each existing term will appear, given that the suggested term is pres-ent. This probability is smoothed by a factor in the denominator to account for low-frequency terms, as discussed elsewhere.13 The prob-abilities are weighted by the existing terms’ inverse-document frequency (idf ), so that co-locations with rare terms receive greater weight than co-locations with common terms. Finally, we normalize the sum by the sum of the idf val-ues. The resulting score is similar to the com-mon TF-IDF score used in information retrieval, except reversed to provide terms for the loca-tion rather than relevant locations for the query terms:

r

stst

idf

E idfst

i

countii

i i=

+∑ α,

where rst is the rank score of suggested term st, sti is the number of locations with co-occurrences of st and existing term i, stcount is the total number of locations with st, and α is a smoothing factor for low-frequency terms. The inverse-document frequency (idf ) is a commonly used means for giving rare terms

00.10.20.30.40.5

Lind

en

Adu

lt

Arts

Business

Education

Gam

ing

Hangout

New

comer

Park

Residential

Shop

ping

Stage

Other

SVM Actual

Figure 1. Support vector machine (SVM) classification results. The classifier performed 60 percent better than simple probability weighting, although it still over-represented the Residential and Shopping categories.

IC-15-05-Eno.indd 50 8/18/11 12:28 PM



in the collection more weight than common terms:

where |L| is the total number of locations and icount is the total number of locations with term i.

The label suggestions generally conform to common-sense terms that we’d expect to find together, particularly for locations that have distinctive objects. For example, the system suggests the terms {hamper, towels, sink, faucet} when given a location with the terms {house, bathroom, sink}. In contrast, when we give it a location with the terms {kitchen, oven, dishwasher}, it returns the suggestions {microwave, freezer, backsplash, utensils}. In locations with more ambiguous terms {chair, door, table}, the suggestions are less focused, resulting in generally common matches. Because the system works on existing plaintext terms, it still has difficulty differentiating between different word senses, but we could adapt and improve this approach as semantic labels become available.

Linking Existing OntologiesAlthough the existing term-suggestion service is helpful in creating a more homogenous set of terms for the set of objects commonly found in similar locations, it still relies on plaintext terms rather than ontological concepts. How-ever, we’re experimenting with tools to associ-ate concepts from WordNet and DBpedia with virtual world entities.

One question to ask is how these existing ontologies cover the objects found in 3D vir-tual worlds. Some terms, such as specific brand names, have no analog in the real world and hence won’t appear in ontologies based on the real world. Likewise, some concepts and terms are specific to virtual world lexicons, such as prims, which are primitive objects used to build 3D models in Second Life. To quantify the cov-erage of the virtual world by two large and widely used ontologies, we matched the 38,000 terms found in the residential location dataset with concept terms in both the WordNet and DBpedia ontologies.

To account for variations in word forms, we analyzed the matches for both raw (or full) and stemmed terms (see Table 1). Stemming the

words increased coverage by eliminating misses caused by plurals, but in some cases might have created less-accurate matching. By examin-ing the terms DBpedia matched that WordNet missed, we found that many newer terms or informal words existed in the DBpedia dataset, which is constructed using Wikipedia entries. Some terms, such as “YouTube,” are unsurpris-ing, but others, such as “media,” were unex-pected. However, some of those matches might have been matching brand names to abbrevia-tions or foreign-language terms that weren’t actually related. Another factor working in DBpedia’s favor is its larger size. DBpedia has 7.5 million titles in its index, covering 3.5 million things (the index contains multiple terms that map to a single thing in some cases). WordNet is smaller, with fewer than 150,000 index entries, so while it had fewer overall matches, it had a higher hit rate as a percentage of its size. Another factor that might indicate a greater utility to the WordNet matches is that only 1.67 million of the DBpedia instances are classified in a con-sistent ontology.

Expanding and Creating OntologiesAlthough high-quality ontologies are created for a range of purposes, individual applications might find that general-purpose ontologies lack needed details. Additionally, ontologies often focus on taxonomic data that provide “is-a” rela-tionships but not necessarily functional or “has-parts” relationships. Researchers have already found that observing use patterns can reveal functional semantic relationships,8 and simi-lar approaches with objects found in images can derive has-part relationships.13 Using an approach similar to that used to suggest likely description terms for content creators, we developed tools to discover common relationships between objects found in the virtual world. For this tool, we rely on the conditional probability P(p|c) that a par-ent term will be found, given that some child

Table 1. Existing ontology coverage.

Ontology coverage Full terms (%) Stemmed (%)

DBpedia percent of Second Life terms

58.0 59.5

WordNet percent of Second Life terms

30.3 33.7

DBpedia percent of total 0.3 0.3

WordNet percent of total 7.9 8.7

IC-15-05-Eno.indd 51 8/18/11 12:28 PM



term is found to identify child terms that are strongly associated with a parent term.

The results for creating or expanding onto-logies varied based on the parent–child relation-ships’ specificity. For example, the probability that a kitchen will be present is highest if the terms cooktop, cabinets, microwave, fridge, or oven are found. For the specific kitchen instance, the system’s precision is high, with 90 percent of the top 20 objects having a real-world relationship to kitchens. For parent terms with less-distinct component parts, the accuracy is mixed, but still provides useful information in terms of expanding the ontology probabilisti-cally. We might be able to improve the results for more general terms by incorporating proxi-mity more directly in the scoring function. This approach’s primary advantage is that it can expand existing ontologies with relationships that are too specific for general-purpose ontolo-gies, or with functional relationships that are difficult to derive from taxonomic ontologies.

N ot all virtual world applications will need an ontology service. Still, such a service makes

sense as a data structure that virtual worlds can use to represent declarative content. Many ontology services will require application- specific content, so, for virtual world architectures, the ability to add ontology plug-ins also makes sense. At the same time, applications can find value in general-purpose semantic information.

We’ve seen that only 20 percent of Second Life objects are labeled by their creator. Even so, a harvester that collects virtual world data can create a database that a classification sys-tem can then mine for semantic information. Even though the information is from a virtual world, for many information types, the vir-tual world provides data that models common-sense aspects of the real world — so kitchens have stoves and refrigerators (and, with a lower probability, microwaves and can openers). The resulting common-sense model can be useful in real or virtual worlds.

References1. T. Berners-Lee, J. Hendler, and O. Lassila, “The Semantic

Web,” Scientific Am., vol. 284, May 2001, pp. 34–43.

2. N. Gershenfeld, R. Krikorian, and D. Cohen, “The Inter-

net of Things,” Scientific Am., vol. 291, no. 4, 2004,

pp. 76–81.

3. A. Eguchi and C. Thompson, “Towards a Semantic

World: Smart Objects in a Virtual World,” Web Vir-

tual Reality and Three Dimensional Worlds Workshop,

Proc. Int’l Assoc. for the Development of the Info. Soc.

(IADIS) Multiconf. Computer Science and Information

Systems, IADIS Press, 2010, pp. 488–493.

4. T. Alatalo, “An Entity-Component Model for Extensible

Virtual Worlds,” IEEE Internet Computing, vol. 15, no. 5,

2011, pp. 30–37.

5. J. Kaplan and N. Yankelovich, “Open Wonderland: An

Extensible Virtual World Architecture,” IEEE Internet

Computing, vol. 15, no. 5, 2011, pp. 38–45.

6. T. Berners-Lee, “Linked Data — Design Issues,” 27 July

2006; www.w3.org/DesignIssues/LinkedData.html.

7. C. Fellbaum, WordNet: An Electronic Lexical Database,

MIT Press, 1998.

8. M. Philipose et al., “Inferring Activities from Interac-

tions with Objects,” IEEE Pervasive Computing, vol. 3,

no. 4, 2004, pp. 10–17.

9. M. Perkowitz et al., “Mining Models of Human Activi-

ties from the Web,” Proc. 13th Int’l Conf. World Wide

Web, ACM Press, 2004, pp. 573–582.

10. J. Eno, S. Gauch, and C. Thompson, “Searching for the

Metaverse,” Proc. ACM Symp. Virtual Reality Software

and Technology, ACM Press, 2009, pp. 223–226.

11. T. Joachims, “Making Large-Scale SVM Learning

Practical,” Advances in Kernel Methods — Support Vec-

tor Learning, C.B.B. Schölkopf and A. Smola, eds., MIT

Press, 1999, pp. 169–184.

12. T. Joachims, “Text Categorization with Support Vec-

tor Machines: Learning with Many Relevant Features,”

Proc. European Conf. Machine Learning, Springer,

1998, pp. 137–142.

13. B. Russel et al., “LabelMe: A Database and Web-Based

Tool for Image Annotation,” Int’l J. Computer Vision,

vol. 77, nos. 1–3, 2008, pp. 157–173.

Joshua D. Eno is a postdoctoral researcher at the University

of Arkansas working on virtual world architectures

and ontologies. His interests include middleware archi-

tectures, 3D virtual worlds, data mining, and health-

care informatics. Eno has a PhD in computer science

from the University of Arkansas. Contact him at jeno@

uark.edu.

Craig W. Thompson is the Charles Morgan chair in the

Computer Science and Computer Engineering Depart-

ment at the University of Arkansas. His research inter-

ests include artificial intelligence, databases, middle ware

architectures, RFID, virtual worlds, and pervasive com-

puting. Thompson has a PhD in computer science from

the University of Texas at Austin. He’s an IEEE fellow.

Contact him at [email protected].

IC-15-05-Eno.indd 52 8/18/11 12:28 PM

virtual and real-world ontology services

Documents