Indagare - Vector Drawings Search Engine

Pedro Sousapedrocrs@gmail.com

September 29, 2008

Abstract

Currently, there are large collections of clip art vector drawings from which users can select the desiredfigures to insert in their documents. However, to locate a particular drawing among thousands is noteasy. This thesis presents a former approach adapted to the human perception model, based in geometricmatching of two drawings, which takes into account the drawing as a whole and each of the shapes init. We developed a web search engine for clip art drawings, where we included this new technique.Experimental evaluation reveals that this new geometric matching algorithm, applied to the problem ofClipArt drawings retrieval based on sketches, conducts to better results than some solutions based ontopological information.

Keywords: Geometric Matching, Drawing Retrieval, Sketch-Based Retrieval

1 Introduction

Nowadays, there are a lot of vector drawings available for integration into documents, either on the Internetor on clip art collections sold in optical media. This large number of drawings makes traditional searchingmechanisms, based on browsing and navigation in directories or complex mazes of categories, inadequate.Furthermore, solutions using keywords or tagging are also impracticable since they have to be generatedmanually. A more adequate solution must take into account information automatically extracted from thecontent of drawings, instead of information manually generated by people. Although there are several solu-tions for Content-Based Image Retrieval (CBIR), they cannot be applied to vector drawings, because theseare described in a structural manner, requiring different approaches from those used for raster images.

In Fonseca’s et al. work [4], in which we based on to develop our approach, they proposed an automaticvisual classification scheme based on topology and geometry, to retrieve vector drawings. Their solutiontakes advantage of users’ visual memory and explores their ability to sketch as a query mechanism. Authorsused a graph-based technique to describe the spatial arrangement of drawing components, coding topologicalrelationships of inclusion and adjacency through the specification of links between nodes of the graph. Thisframework uses a multidimensional indexing method that efficiently supports large sets of drawings, in com-bination with new schemes that allow the hierarchical description of drawings and subparts of drawings bylevel of detail. This way it’s allowed to perform searches using coarse approximations or parts of the desireddrawing. Matching in this solution was performed in two steps: authors first selected the drawings withsimilar topology to the query, then they compared the geometry of this subset of topologically similar draw-ings. Although, this matching sequence produced good results for technical drawings [3], which have a richertopology, the same does not apply to clip art drawings that are more geometric than topological. Indeed,during user evaluation, they observed that users, while searching for clip art drawings, typically draw a smallnumber of shapes, and consequently do not specify topology, but only geometry. Moreover, our most recentexperimental evaluation to test the retrieval performance of this system [15] revealed that this topology plusgeometry combination produces poor results for retrieving clip art drawings. Additionally, we identify theneed to improve the geometry comparison algorithm, since the topology filter ”is not doing its job”. Whilewith CAD drawings we can have a more flexible geometric matching algorithm, with clip art drawings we

1

must have a tighter geometric comparison mechanism to assure good results even without the use of topology.

In this thesis we explain the possible reasons of the poor performance Fonseca’s matching algorithm,and we propose a new geometric matching mechanism, bearing in mind the importance of the human vi-sual perception. To that end we use the similarity of drawings as a whole and the similarity between eachindividual shape in drawings. We developed a prototype for the retrieval of clip art drawings integratingboth algorithms, the prior and the new. This prototype was used to evaluate the performance of the twoalgorithms. Experimental results revealed that our new approach to compare the geometry of drawingsoutperforms the prior solution.

The rest of the paper is organized as follows: Section 2 provides an overview of related work in content-based retrieval of drawings. In Section 3 we present an overview of Fonseca’s framework for sketch-basedretrieval of drawings. Section 4 describes the previous geometric matching technique integrated in Fonseca’sFramework, allowing readers to understand the differences to the new matching algorithm presented in theremainder of this chapter. In Section 5, we describe the experimental evaluation performed to compare bothapproaches. Finally, in Section 6 we conclude and enumerate some future work.

2 Related Work

In the past years there has been a great focus in querying Multimedia databases by content. However, suchwork has focused on image databases disregarding the retrieval of vector drawings. Due to their structurethese require different approaches from image-based methods, which resort to color and texture as mainfeatures to describe content. In the next paragraphs we describe some approaches for content-based retrievalof drawings.A system that supports the retrieval of complex 2D drawings was presented in 1999 by Park and Um [13].Their approach is based on dominant shapes, where objects are described by recursively decomposing itsshape into a dominant shape, auxiliary components and their spatial relationships (inclusion and adjacency).These and the decomposition of blocks are stored in a complex graph structure. Visual elements in drawingsare classified using a set of geometric primitives (triangles, circles, etc.). However, this small set of basegeometric primitives and the not-so-efficient matching algorithm, based on the breadth-first tree matching,make it hard to handle large databases of drawings.

In the work of Beretti and Bimbo [1] shapes from a drawing are decomposed into tokens that correspondto protrusions of the curve. To compute the similarity between shapes, authors verify if the two shapesshare tokens with similar curvature and orientation, within a given threshold. However, the efficiency of thesimilarity computation depends on the number of tokens in each shape and does not take into account thetoken order.

Leung and Chen proposed a sketch retrieval method [7] for general unstructured free-form hand-drawingsstored in the form of multiple strokes. They use shape information from each stroke exploiting the geomet-ric relationship between multiple strokes for matching. Later on, authors improved their system by alsoconsidering spatial relationships between strokes [8]. Authors use a graph based description but describingonly inclusion. Their technique has two draw-backs, complexity, since they use a restricted number of basicshapes (circle, line and polygon) and scalability.

Another approach for matching hand-drawn sketches is the line-based representation of sketches proposedby Namboodiri and Jain [12]. In order to skirt around the problem of identifying basic shapes from a sketch, adrawing is represented as a set of straight lines. However, the conversion into straight lines is very dependentof the way users draw sketches.

Mascio et al. [11] presented an approach for sketch-based retrieval of drawings using a CBIR engine.Although they retrieve vector drawings, authors first convert them into raster images and then apply aset of CBIR techniques. Although authors do not present any evaluation of their solution, we think that

2

converting vector drawings to raster images is not the best approach, since it consumes time and causes thelost of information during the process.

Liang et al. [14] developed a solution for drawing retrieval based on the solution proposed by Fonsecaet al. [4]. Authors included some differences, such as the use of eight topological relationships and rele-vance feedback. Additionally, they segment sketches using vertices, drawing speed and curvature. By usingeight topological relationships, the description and comparison will be more restrictive, producing less results.

Pu and Ramani, developed an approach that analyzes drawings as a whole [6]. Authors proposed twomethods to describe drawings. One uses the 2.5D spherical harmonics to convert a 2D drawing into a 3Drepresentation, which is independent to rotations. The other method, the 2D shape histogram, creates asignature with the shape distribution, by computing values for points in the surface of the shape. Thismethod is independent of transformations, insensible to noise, simple and fast. After experimental evalua-tion, authors decided to combine both methods to get a better descriptor and to increase the system accuracy.

Recently Hou and Ramani [5] presented an approach for contour shape matching of engineering drawings,inspired by the divide and conquer paradigm. They divide the original shape into two levels of represen-tation, a higher level with structure and a lower level with geometry. During matching, they first use thestructure level and then the geometry level, to find similar shapes.

From the content-based retrieval systems described above we can observe that the ones that rely on theshapes’ contour are more popular than the region-based approaches. However, contour-based methods aregenerally sensitive to noise and sketch variations. Besides systems employing region-based approaches aremore robust and precise, they also cope well with shape defection. Furthermore, we can notice that themajority of the presented solutions deal with simple drawings, using mainly the contours to describe thegeometric information of the drawings. These techniques are not suitable for retrieving clip art drawings,since these have a larger complexity and a bigger number of visual elements, requiring other mechanisms todescribe and compare their geometry.

In this thesis, we describe and evaluate a new approach for geometric matching between clip art drawingsand sketched queries, that tries to solve both problems identified above. This algorithm is part of a frameworkfor retrieval of complex vector drawings, which we shortly present in the next section.

3 Framework Overview

The new algorithm developed to compare the geometry between queries and drawings was integrated inFonseca’s framework for sketch-based retrieval of drawings [4]. To give context to the reader and to explainsome of the topics needed to describe our new geometric matching approach, we shortly present an overviewof the overall framework, describing its main components.This framework allows the classification, indexing and retrieval of complex vector drawings, such as CADdrawings or clip art drawings. To that end, it uses spatial relationships, geometric information and indexingmechanisms, as illustrated in Figure 1.

3.1 Classification

In the context of vector drawings, features such as color and texture, used mainly in the domain of digitalimages, are not very expressive. Instead, features related to the shape of objects (geometry) and to theirspatial arrangement (topology) are more descriptive of drawing contents. So, this framework uses topologyand geometry as main features.

The classification process starts by applying a simplification step, to eliminate most useless elements.The majority of drawings contain many details, which are not necessary for a visual query and increase

3

Query

Interface Indexação e Semelhança

Classificação

DesenhoDesenho

Simplificado

Grafo de Topologia

Descritores Geométricos

Extracção Topológica

Extracção Geométrica

Descritores Topológicos

Cálculo de Descritores

Simplificação

NB-tree de geometria

NB-tree de topologia

Inserção

Inserção

QueryRefinada

Topologicamente semelhantes

Desenho Simplificado

Query de geometria

Extracção da geometria

Grafo deTopologia

Query de topologia

Extracção datopologia

Cálculo dosdescritores

interrogação

interrogação

Desenhos Semelhantes

Query

Figure 1: Framework architecture.

the cost of searching. Therefore, this method tries to remove visual details (i.e. small-scale features) whileretaining the perceptually dominant elements and shapes in a drawing. This way the number of entities toanalyze in subsequent steps of the classification process is reduced, speeding up queries.

After simplification, visual elements, namely polygons and lines, are identified and the geometric andtopological information is then extracted from drawings. Topology extraction process uses two relationships,Inclusion and Adjacency, which are a simplified subset of the topological relationships defined by Egenhofer[2]. Relationships thus extracted are compiled in a Topology Graph, where ”parent” edges mean Inclusionand ”sibling” connections mean Adjacency. While these relationships are weakly discriminating, they do notchange with rotation and translation.

In order to acquire geometric information about drawings, it’s used a general shape recognition librarycalled CALI [10]. This enables the use of, not only drawing data as input, but also hand-drawn sketches. Thecomplete description of geometry in a drawing is then obtained by applying this method to each geometryentity of the figure. The geometry and topology descriptors thus computed are inserted into two differentindexing structures, one for topological information and another for geometric information, respectively.

3.2 Query and Matching

This system includes a Calligraphic Interface to support the specification of hand-sketched queries, to sup-plement and overcoming limitations of conventional textual methods. The query component performs thesame steps as the classification process, namely simplification, topological and geometric feature extraction,topology graph creation and descriptor computation. To improve the searching performance while usinglarge databases of drawings, the authors included a multidimensional indexing structure in our framework.This indexing structure, the NB-Tree [9], is a simple, yet efficient indexing structure, which uses dimensionreduction. It maps multidimensional points to a 1D line by computing their Euclidean Norm. In a secondstep points are sorted using a B+-Tree on which all subsequent operations are performed.Computing the similarity between a hand-sketched query and all drawings in a database can entail pro-hibitive costs especially when large sets of drawings are considered.To speed up searching, they divided their matching scheme in a two-step procedure. First, a set of drawings

4

topologically similar to the query is selected, then they use geometric information to further refine the setof candidates.

4 Geometric Algorithm

4.1 Previous Matching Algorithm

The previous algorithm for geometry matching, integrated in Fonseca’s Framework, assumed that most ofthe non relevant drawings were discard by the topology filter. So, this reduced set of drawings were thencompared to the sketched query, yielding a measure of geometric similarity for each drawing. The similaritymeasure was obtained through the computation of distances between geometric query descriptors and geo-metric data stored in an indexing structure. In this approach each drawing has its own indexing structurewith the geometric information, to simplify the matching process. Thus, during polygon comparison onlythe similarity distances between relevant descriptors were computed.

The matching procedure starts by computing a geometric descriptor for each entity specified in the query,using the CALI library. Then, each of these descriptors are used to perform a KNN search (being K thenumber of polygons in the sketched query) to each geometric indexing structure, one for each topologicallysimilar drawing. Returned results have a distance associated that convey similarity. Smaller distances meanmore similar while larger distances mean less similar. Finally, the pair of polygons (one from the query andother from the drawing) that had the smallest distance was iteratively selected. After each selection the pairof polygons was eliminated from the list of results.

We will now explain in detail, resorting to an example, how was the geometric similarity measure com-puted. Considering a query with four polygons and a drawing in the database with four polygons, belongingto a drawing. After performing four 4-NN queries to the correspondent indexing structure (one for eachvisual entity in the query), a matrix representing distances from each polygon in the query (qP1-qP4) to allpolygons in the drawing (dP1-dP4) is built, as depicted in Figure 2 (top-left).

To compute the similarity value, it’s firstly found the row of the matrix with the smallest value (0.1).Then, this value is stored and the corresponding row (qP1) and column (dP4) are deleted. The same stepsare performed within remaining matrix, selecting the new smallest value (0.12) and deleting row qP2 andcolumn dP2. This iterative process continues until all the polygons of the query have a corresponding value.Finally, all the selected values are summed and then divided by the number of polygons in the query. Whenthe drawing has less polygons than the query, the sum of the obtained values is divided by the number ofpolygons in the drawing.This process of computing the geometric similarity value is then applied to the remaining drawings thatpassed through the topology filtering, yielding a list of drawings with a geometric measure of similarity tothe sketched query.

This method does not return the ”optimum” result, since it does not minimize the similarity value. Thefinal result depends on the order by which values are selected. For instance if we select 0.2 in (qP1,dP3)then 0.12 in (qP2, dP2) then 0.16 in (qP4,dP4) and finally 0.2 in (qP3,dP1) we get Γ = (0.2 + 0.12 + 0.16+ 0.2)/4 = 0.17, which is smaller than the value computed before (0.18).As we can see, the number of elements in the query and in the drawing are not relevant for this algorithm.So, the three drawings in Figure 9 right will have the same similarity value for the query in Figure 2 b) left.

4.2 New geometric matching algorithm

To overcome the drawbacks of the previous geometric algorithm, we focused on understanding some of thehuman’s visual perception characteristics. As a result of our study, we developed a new algorithm to comparethe geometry between drawings, which we describe here.

5

0.4qP4 0.30.5 0.16

0.30.2 0.20.4qP3

0.12 0.60.20.3qP2

0.20.3 0.10.4qP1

dP4dP3dP2dP1

0.4qP4 0.30.5 0.16

0.30.2 0.20.4qP3

0.12 0.60.20.3qP2

0.20.3 0.10.4qP1

dP4dP3dP2dP1

0.4qP4 0.30.5 0.16

0.30.2 0.20.4qP3

0.60.20.3qP2

0.20.3 0.10.4qP1

dP4dP3dP2dP1

0.1

0.12

0.4qP4 0.30.5 0.16

0.30.2 0.20.4qP3

0.3 0.60.20.3qP2

0.20.3 0.10.4qP1

dP4dP3dP2dP1

0.1

0.12

0.2

0.4qP4 0.30.5 0.16

0.30.2 0.20.4qP3

0.3 0.60.20.3qP2

0.20.3 0.10.4qP1

dP4dP3dP2dP1

0.1

0.12

0.2

0.3

τ(D1) = (0.1+0.12+0.2+0.3) / 4τ(D1) = 0.72 / 4 = 0.18

(a)

This method does not return the ”optimum” result, since it does not minimizethe similarity value. The final result depends on the order by which valuesare selected. For instance if we select 0.2 in (qP1,dP3) then 0.12 in (qP2,dP2) then 0.16 in (qP4,dP5) and finally 0.2 in (qP3,dP1) we get ! = (0.2 +0.12 + 0.16 + 0.2)/4 = 0.17, which is smaller than the value computed before(0.18). However, the computation of this minimum value is very expensivecomputationally.

Although, this similarity metric for geometry returns approximate results, thealgorithm is simple and has a low computational complexity. Furthermore,from past experimental tests with databases of technical drawings [1], resultsrevealed that it has a good trade-o" between complexity and quality of results.

As we can see, the number of elements in the query and in the drawing arenot relevant for this algorithm. So, the three drawings in Figure 9 right willhave the same similarity value for the query in Figure 9 left.

Fig. 9. Query example (left) and three results with the same geometric similarity(right).

5 New Geometric Matching Algorithm

(AQUI) ¿¿¿

(Rever) ¿¿ The use of proximity to describe the spatial arrangement gets ourmatching algorithm closer to the human perception, and therefore improv-ing the retrieval e"ectiveness of our system. This improvement was validatedthrough experimental evaluation with users. Despite the complete spatial char-acterization of drawings provided by the use of topological relationships andproximity, the improvement achieved was small (only 1%). These results con-firm informal conclusions achieved previously. Clip art drawings, contrarily

13

(b)

Figure 2: Steps to compute the geometric similarity between the sketched query and a drawing a). Queryexample (left) and three results with the same geometric similarity (right) b)

In Fonseca’s approach, as described above, the drawing geometry is described using a set of feature vectors.Each vector yields the geometric information for each given shape of the drawing. With this method, thetask of determining the geometric similarity within two drawings was pretty straightforward. Although thismethod proved to be effective in the content based retrieval of technical drawings, its use for sketch-basedretrieval of clip art drawings is inadequate. When a user draws a sketch to query a drawing database, heexpects results to have a large number of shapes similar to the ones he drew, and not only one of them.This fact, allied to the great complexity of vector drawings led us to the development of a more robust andprecise geometric matching algorithm.Before developing the algorithm, we tried to understand how users draw sketches and what results were theyexpecting to be retrieved by the system. Additionally, from informal conversations with potential users ofthe system, we came across a few subtle details that are implicit in sketches, and therefore in users’ mindwhen they draw a sketch. From these information, we produced two postulates, which we express in twosimilarity measures: the global similarity and the partial similarity. The former compares drawings as awhole, while the latter takes into account each individual shape present in the drawing.The first postulate is that the user normally draws a simplified sketch of what he intends to find, puttingthe smallest number of shapes into his query. So, at this point, the problem is on how to express this ideain terms of an algorithm that better suits users’ needs.Therefore, and within this concept, we defined two restraints. The first one is that drawings with a numberof shapes equal or superior to the query sketch are preferable. In addition, the greater the number of shapespresent in a drawing, the smaller the chance to be relevant to the user’s query. The second one establishesthat if the query sketch has more shapes than a given drawing, this drawing should be penalized.

So, taking these restraints into account, the global similarity of a given database drawing is defined bythe following equation:

globalSimilarity = similarity ∗ Z

where,

6

Z =

#Pq

#Pdbif #Pq ≤ #Pdb,

#Pdb

2#Pq −#Pdbif #Pq > #Pdb

and,

DrawingSimilarity - similarity between two drawings, computed using the prior geometry algorithm

#Pq = number of polygons in the query drawing

#Pdb = number of polygons in a give database drawing

The second postulate is analogous to the previous one but applied to individual shapes instead of thewhole drawing. It states that the user is always expecting that drawings in the result list contain the sameor more instances of a particular shape specified in the query. So, we defined two more restraints to satisfythis requirement. The first one states that drawings with an equal or superior number of a specific shapethan the query sketch should be better classified. The second principle determines that drawings with asmaller number of a particular shape than the query should be penalized. The partial similarity of a givendatabase drawing is given by the following expression:

partialSimilarity = 1#Pq

∗#Pq∑k=1

similarityk ∗M

where

M =

#Pkq

#Pkdbif #Pkq ≤ #Pkdb,

−#Pkq

#Pkdbif #Pkq > #Pkdb,

−1#Pdb

if #Pkdb = 0

andsimilarityk - similarity between shape k in the query and a shape in the drawing

#Pkq = number of polygons in the query similar to the current one (from the query)#Pkdb = number of polygons in a drawing similar to the current one (from the query)Combining the global and the partial similarity of the drawing we reach the following expression, that

defines the total classification of a given database drawing:

classification = similarity ∗ Z +1

#Pq∗

#Pq∑k=1

similarityk ∗M

So, drawing similarity is the sum of the overall drawing similarity with the weighted average of the sim-ilarity of each query shape.

We will now describe in detail how we determine the similarity classification of a drawing using the newgeometry algorithm. To that end we will use the example from Figure 9. Considering the query with threepolygons, and the first drawing in the database with two polygons, we will perform three 3-NN queries to

7

the correspondent indexing structure. Then, and analogously to the previous matching algorithm, we builda matrix, but this time with the similarities from each polygon in the query to all polygons in the drawing,instead of distances 1.

Contrary to our prior algorithm for geometry matching, we first find the row of the matrix with thehighest value (1). The following steps are equally per- formed until we have selected values for all polygonsof the query. Finally, we sum all the selected values, and instead of simply dividing them by the number ofpolygons in the query, we take into account the number of elements of the query drawing and the databasedrawing (see equation for globalSimilarity on page 7). In this case, the query drawing has more elementsthan the first drawing of the database. Thus, we multiply the obtained classification by the second branchof the equation, leading us to a global classification value of 1.1. Since the two remaining drawings in thedatabase have more elements than the query drawing, we use the first branch of the equation. Accordinglyto this, the second drawing will have a global classification of 3, whereas the third one will have 2.25. Inthe following step we calculate the partial similarity classification for all the polygons of the query. The firstthing to do is to determine how many elements of the same type are present in the query drawing. So, wecompare all the polygons between them, and the ones which have a similarity value within a certain thresholdare marked as similar. In this case, the query does not have any similar polygons, since it is composed byone rectangle, one circle and one triangle. Moreover, if we take a look into the drawings in the database wenotice that none of them has similar polygons. This facilitates our task, because both #Pkq and #Pkdb

values are 1. Therefore, in this case, the value of Mk (see equation for partialSimilarity on page 7) will bethe same as Z from the globalSimilarity equation, namely 2.2, 3 and 3 respectively. Now, we divide theresulting value by the number of polygons in the query, obtaining a partial similarity classification of 0.73for the first drawing and 1 for the remaining two. Finally, we sum the global similarity classification and thepartial one, yielding the following values for drawings 1, 2 and 3 respectively: 1.83, 4, 3.25.

This method is specially concerned in retrieving drawings that have an equal or superior number ofpolygons than the query drawing. In addition, it takes into account the number of elements of the sametype. While the previous geometric matching algorithm classifies all the three drawings as similar to thequery, our new geometric matching algorithm distinguishes them, retrieving the identical drawing as a firstmatch. Moreover, this result is closer to the human’s visual perception.

5 Experimental Evaluation

We developed a search engine prototype for vector drawings, using our sketch-based retrieval framework andthe new algorithm to perform geometric matching. The database of the system was filled with a set of SVGclip art drawings and we carried out an experimental evaluation with users to compare the accuracy of thenew algorithm against the previous one.

5.1 Indagare - A Sketch-Based Search Engine for Clip art Drawings

Our prototype for drawing retrieval, called Indagare (see Figure 3), supports the retrieval of SVG clipart drawings, using sketches, an existing SVG drawing or keywords. This prototype integrates all thefunctionalities provided by the framework, namely, simplification mechanisms, an indexing structure tooptimize the search, the topological structure to code spatial arrangement and the new geometric matchingalgorithm.

If the user wants, he can submit an existing drawing in SVG format or search by keywords. Moreover,users can also select one of the results and use it to perform Query-By-Example.

5.2 Experiment Description

To evaluate our new approach for geometric matching, we carried out an experiment with ten users. Six ofthem were male and four were female, with ages between 18 and 58 years old. None of them had previousexperience with tablet devices or any other pen-based system. Our data set of clip art drawings is composedof 20 categories of five drawings each, selected from the OpenClipart library, yielding a total of 100 drawings.

8

Figure 3: Prototype for drawing retrieval.

The tests were conducted in two steps. First we asked each user to draw three sketches, using a digitizingtablet: a balloon, a car and a house. These sketches were stored for further use as queries. Afterwards, weshow all the 100 drawings in the database and requested them to identify the drawings that they consideredmost similar to each of the sketches they drawn, without taking into account the drawing’s semanticalvalue. The second step was carried without users’ intervention. From the similar drawings selected by theparticipants, we identified the five more voted, and considered those as the ”correct” results for each sketch.

5.3 Algorithms Evaluation

To evaluate the new algorithm for geometric classification we submitted the three sketches from each partic-ipant to the system and collected the results. We configured the system to retrieve 30 results for each query.With these results we computed precision and recall values.

5.4 Using just Geometry

In this experimental test, our primary concern was to analyze the new geometry algorithm effectiveness.Therefore, our first step was the comparison between the old geometry algorithm and the new one, withoutthe topology comparison step. We calculated precision & recall levels for the two methods, using the 11-PointInterpolated Average Precision method. The mean precision for each recall value of the two algorithms ispresented in Figure 4 a).

Comparing the precision at each recall level, we can observe that adding implicit information, like thenumber of shapes in a drawing to the geometric similarity calculation, substantially improved the algorithm.We can also notice that with the new geometry algorithm at the 10% and 20% recall levels the precisionsuffers a boost of 54%. Furthermore, fixing the average precision at each recall level of the new geometryalgorithm, we realize that its precision is only overtaken at the 90% and 100% recall levels by the precisionof the old geometry algorithm at the 10% and 20% recall levels. Therefore, and since the 90% and 100%recall levels correspond to the retrieval of the 5 relevant documents, we can conclude that, in average, forevery 4 relevant documents retrieved by the new geometry algorithm, the old geometry algorithm retrievesonly 1 (Figure 4 a)). So, according to the current results, the refinement made in the geometry algorithmproduced undeniable improvements in the system’s effectiveness.

5.5 Using topology and Geometry

Our next step was to evaluate both geometric algorithms in combination with the topology filtering. Fromour prior evaluation of the original geometric matching algorithm [15] we concluded that the best configura-tion for it is to first compare by topology and then compare by geometry. Figure 4 b) presents the resultsfor both geometry algorithms when we first perform the topology filtering. As we can observe, the new

9

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1

Precision

Recall

algoritmo de geometria antigo

algoritmo de geometria novo

(a)

0

0,1

0,2

0,3

0,4

0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1

Precision

Recall

topologia + geometria antigo

topologia + geometria novo

(b)

Figure 4: Precision & Recall for the two geometric matching algorithms alone (a) and for the two geometricalgorithms, when we first compare by topology (b).

method presents better results than the old algorithm. However, the achieved precision is smaller than thatobtained with the new geometric matching alone (see Figure 4 a)). Taking these results into account, andrecalling the previous observation stating that clip art drawings are more geometric than topological, wedecided to perform first a geometric filtering and then the topological comparison. We did this only for thenew algorithm, because this combination was already evaluated for the old one and did not produce goodresults. Results comparing this new sequence with the best combination for the old geometric matchingalgorithm is depicted in Figure 5 a). From this chart, we can identify that comparing first by geometryand then by topology is the matching solution that achieves better precision values. Comparing it with theresults achieved by the first method (topology plus old geometry algorithm), show an improvement superiorto 50%, at the 10% and 20% recall levels. So, at this point we have two possible solutions for the matchingprocess that produce the best results. One using just the new geometric algorithm and another that usesthe new geometric algorithm plus topology.

Figure 5 b) presents a comparison between these two. As we can see, the new geometry algorithmplus topology configuration is slightly better than the new geometry alone. In fact, it presents superiorresults at 10% to 40% recall levels and inferior values for 70% to 100% recall levels. So, we need to checkif the improvements in the final results, caused by the inclusion of another step in the matching process,compensates the additional time spent on it. To answer this, we carried out a new test to measure the timeneeded to execute a query on each algorithm.

5.6 Time Evaluation

To collect query times for both methods, we submitted ten queries to the two different configurations, usingsix databases of 100, 500, 1000, 2000, 5000 and 10000 drawings. We measured the time taken by the systemto retrieve the 30 more similar documents to the query. Results are shown in Figure 6.

From the chart, we notice that in the worst case the configuration that uses both comparison stages takesmore six seconds to perform a query to the database of 10,000 drawings than the other. According to theseresults (see Figures 5 b) and 6) we can configure the matching process for two different types of users: Onefor users that want more accurate results disregarding the time they take to be generated. In this case weshould use both Geometry and Topology comparison stages. Another for most common users that are moreconcerned in getting goodresults in a short period of time.

10

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1

Precision

Recall

geometria novo + topologia

topologia + geometria antigo

(a)

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1

Precision

Recall

algoritmo de geometria novo

geometria novo + topologia

(b)

Figure 5: Precision and recall for topology plus the old geometry algorithm and for the new geometryalgorithm plus topology (a), and Precision and recall for the two best combinations for the matching approach(b).

0

2,2

4,4

6,6

8,8

11,0

100 500 1000 2000 5000 10000

Te

mp

o (

s)

Número de desenhos na base de dados

geometria novo geometria novo + topologia

Figure 6: Time taken to perform a query for the two best configurations.

6 Conclusions and Future Work

In this thesis, we proposed a new geometric algorithm that uses the information about the number of visualelements in the query and in the database drawings to compute a more accurate measure of geometric sim-ilary. This new algorithm was integrated into Fonseca’s framework for sketch-based retrieval of drawings,and its efficiency was compared to the prior geometric algorithm.

Experimental evaluation showed that the new algorithm outperforms the previous one, providing anincrease of 50% in the retrieval effectiveness of the system. Although the most efficient solution is thegeometry filtering followed by the topology matching, it is very time consuming. So, the solution thatprovides a better tradeoff between quality of results and processing time, is the one that uses the newgeometric matching algorithm alone. Currently, the creation and organization of the information extractedfrom drawings is optimized to use topology as the first filter. So, we believe that the overall querying time canbe reduced if we adapt/redesign the framework to first filter by geometry and then by topology. To furtherenhance the system precision, we could use topological information in the computation of the geometric

11

similarity. To that end, we could assign weights to shapes according to its importance in the drawing. Forinstance, a drawing containing a shape that includes a great number of other shapes, should be more relevantthan a drawing that has a shape containing less polygons.

References

[1] Stefano Berretti and Alberto Del Bimbo. Retrieval by shape similarity with perceptual distance andeffective indexing. Multimedia, IEEE Transactions, 2(4):225–239, 2000.

[2] M. J. Egenhofer and K. K. Al-Taha. Reasoning about gradual changes of topological relationships.LNCS, 639:196–219, 1992.

[3] M. J. Fonseca. Sketch-based retrieval in large sets of drawings. PhD thesis, Instituto Superior Tecnico/ Technical University of Lisbon, July 2004.

[4] Manuel J. Fonseca, Alfredo Ferreira Jr., and Joaquim A. Jorge. Content-Based Retrieval of TechnicalDrawings. International Journal of Computer Applications in Technology (IJCAT), 23(2–4):86–100,2005.

[5] Suyu Hou and Karthik Ramani. Structure-oriented contour representation and matching for engineeringshapes. Computer Aided Design, 40(1):94–108, 2008.

[6] Pu Jiantao and Karthik Ramani. On visual similarity based 2d drawing retrieval. Computer AidedDesign, 38(3):249–259, 2006.

[7] Wing Ho Leung and Tsuhan Chen. User-Independent Retrieval of Free-Form Hand-Drawn Sketches.In Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing(ICASSP’02), volume 2, pages 2029–2032, Orlando, Florida, USA, May 2002. IEEE Press.

[8] Wing Ho Leung and Tsuhan Chen. Hierarchical Matching for Retrieval of Hand-Drawn Sketches. InProceedings of the IEEE International Conference on Multimedia and Exposition. (ICME’03), volume 2,pages 29–32, Baltimore, USA, July 2003. IEEE Press.

[9] J. A. Jorge M. J. Fonseca. Indexing high-dimensional data for content-based retrieval in large databases.In 8th International Conference on Database Systems for Advanced Applications.

[10] J. A. Jorge M. J. Fonseca, C. Pimentel. Cali: An online scribble recognizer for calligraphic interfaces.In AAAI Spring Symposium - Sketch Understanding.

[11] Tania Di Mascio, Marco Francesconi, Daniele Frigioni, and Laura Tarantino. Tuning a cbir system forvector images: the interface support. In AVI ’04: Proceedings of the working conference on Advancedvisual interfaces, pages 425–428. ACM, 2004.

[12] Anoop M. Namboodiri and Anil K. Jain. Retrieval of on-line hand-drawn sketches. In Proceedings of the17th International Conference on Pattern Recognition, ICPR’04, Vol. 2, pages 642–645, Washington,DC, USA, 2004. IEEE Computer Society.

[13] Jong Park and Bong Um. A New Approach to Similarity Retrieval of 2D Graphic Objects Based onDominant Shapes. Pattern Recognition Letters, 20:591–616, 1999.

[14] B. Li G.-H. Feng S. Liang, Z.-X. Sun. Effective sketch retrieval based on its contents. In MachineLearning and Cybernetics.

[15] P. Sousa and M. J. Fonseca. Sketch-based retrieval of drawings using topological proximity. In SpecialSession on ”Sketch Computing” at VLC 2008 (To appear).

12