V4Ann:Representa.onandInterlinkingofAtom-based

Annota.onsofDigitalContentGeorgiosMeditskos,StefanosVrochidis,IoannisKompatsiaris

V4Design 15thInterna;onalConferenceonSeman;cSystems(SEMANTiCS)

Karlsruhe,GermanySeptember9-12,2019

Outline

• Overview&Mo;va;on• ProposedFramework

•  V4AnnAnnota;onModel•  InferenceandValida;on•  Assetsearching

•  Evalua;on•  FutureDirec;ons

2

Overview&Mo.va.on

•  Highqualitycontentisnowadayswidelyavailableonthewebandothersources

•  user-generatedcontent,suchasimages,videosandtextpostedbyusersonsocialmedia,wikisandblogs

•  contentprovidedthroughofficialpublishersanddistributors,suchasdigitallibraries,organisa;onsandonlinemuseums

•  Thiscontentremainslargelyunder-exploited•  lackofsolu;onsforitsretrievalandintegra;onintothedesignprocess

•  Ifleveragedappropriately,couldserveasavaluablesourceofinspira.on•  agreatsourceofrevenuefortheCrea;veIndustries,suchasarchitectureandvideogamedesign

•  inspireandsupportthecrea;onofnewcontentandtoproducenewassetsortoenhanceand(re-)usethealreadyexis;ngones

3

Challenge

• Maximisethepoten;alforre-purposingofdigitalcontent• Developmentofinnova;vetechnologiestosystema;cally

•  Analyse•  Combine•  Link•  Fostersearchabilityandreusabilityofheterogeneouscontent

4

V4DesignProjectVisualandtextualcontentre-purposingFOR(4)

architecture,Designandvideovirtualrealitygames

hWp://www.v4design.eu/

5

V4DesignConcept

•  Vision•  Reuseandrepurposehighqualitycontent•  Providerepurposedcontenttotargetedcrea;veindustries

•  Architects,VRandvideogamedesigners•  Providerevenuesforthedataprovidersandcrea;veindustries

•  KeyTechnologies• Webdataminingforcrawlingvisualandtextualdata•  Aesthe;csextrac;onandtextureproposals•  Languageunderstandingandtextgenera;on•  3Dreconstruc;on•  Seman.cknowledgerepresenta.on,linkingandreasoning

6

OverviewofV4DesignPlaQorm

7

V4AnnRoleinV4Design

•  Seman;cmiddleware,capturing,interlinkingandservinganalysisresultstomul;mediaanalysisservices

•  Annota;onpropaga;onandlinking:efficientandinteroperablewaytorepresent,exchangeandfurtherlinkmetadata,bothstructurallyandseman;cally

•  Seman;catom-basedqueryinfrastructuretoretrievegeneratedassets•  Context-awareretrieval:prac;calandefficientretrievalmechanismsontopofthemul;modalannota;ons

•  How?•  WebAnnota;onDataModel(WADM),BuildingTopologyOntology(BOT)•  Domain-specificontologies(EuropeanaDataModel)•  SPIN/SHACLinferenceandvalida;onrules

8

V4Ann Annotation Model

9

KeyConcepts

•  Annota;ons•  resourcecontainers,implemen;ngtheannota;onpaWernofWADM

• Mediatypes•  image,video,text,3Dmodel

•  Views•  containerforaWachingannota;ons(atoms)tomediatypes

•  Atoms•  aesthe;cs,object/buildingtypes,nameden;;es/rela;ons,image/videoframesusedfor3Dreconstruc;on

10

Annota.onresources

• LocalisationAnnotation,TextualAnnotation,AestheticsAnnotationand3DModelAnnotation

•  Extendsoa:Annotationconcept•  hasContext⊑oa:hasBody•  describes⊑ oa:hasTarget

• V4Annannota;onhasacontextthatdescribesamediatypeusingviews

11

Mediatypes

•  Fourmediatypes•  Video•  Text•  Image

•  Mask⊑Image•  Texture⊑Image

•  3DModel

•  Intui;vely,eachmediatyperesourcerepresentsasinglemul;mediaassetforwhichasetofannota;onatomsneedstobecaptured.

12

ViewsandAtoms

•  Encapsulatetheannota;ondata(formediatypes)•  Annota;onarederivedfromcontentanalysis(textanalysis,visualanalysis,etc.)

• Aesthe.cs:categorisa;onoftheaesthe;csofpain;ngsandimagesthatcontainarchitectureobjectsandbuildings

•  Style:e.g.impressionism,cubismandexpressionism•  Creator:mainlyforpain;ngs(schema.org)•  Emo;on:e.g.fear

13

ViewsandAtoms

• ObjectandBuildingLocalisa.on:Buildingandinteriorobjectslocaliza;on,e.g.includetables,vases,aswellasstatues,buildings,etc.

•  Masks:outlineoftheobject•  Tags:computervisionlabelling•  Frames

14

ViewsandAtoms

•  TextAnalysis:annota;onenrichmentwithen;;esandconceptsextractedfrom;tles,cap;ons,descrip;ons,etc.

•  Alreadydisambiguated:WordNet,BabelNetorDbpedia

15

ViewsandAtoms

•  3Dreconstruc.on:Convertsinputvideos/imagesinto3Dpointcloudsandmeshes

•  3D-relatedproper;es(e.g.numberofpoints)•  Sourceof3Dreconstruc;on(veryimportantforannota;onpropaga;onandlinking!)

16

17

Inference and Validation

18

ImplicitRela.ons• Addi;onalinferencesarederivedbycombiningna;veOWL2RLreasoningandcustomrules

•  SPARQL-basedCONSTRUCTgraphpaWernsareusedthatiden;fythevalidinferencesthatcanbemadeontheannota;ongraphs.

•  Example:atompropaga.on•  Propagate/interlinkatomsamongview-basedannota;onprovidedthattheyaresomehowconnected

•  Forexample,theaesthe;csatomsextractedfromvideoframescanbeusedtoannotatethe3Dmodelsthathavebeenreconstructedusingthoseframes

19

20

Valida;onandConsistencyChecking

• Consistency,structuralandsyntac;cqualityofthemetadataA.  Na;veontologyconsistencychecking(e.g.OWL2DLreasoning)

•  TBoxconsistency(e.g.classdisjointness)B.  CustomSHACLvalida;onrules

•  constraintviola;ons,e.g.missingvaluesandcardinalityviola;ons•  e.g.thatall3Dmodelviewsshouldincludereferencestotheatoms(images)usedforthe3Dreconstruc;on.

21

Context-based Asset Retrieval

22

LocalContext•  Enriched,pre-constructedseman;csignatureofthisatom

•  conceptualandlexicalrela;onsfromexis;ngseman;cnetworksanddatasets,suchasWordNet,BabelNetandConceptNet

•  Theretrievalmechanismaimstomatchincominglocalcontextsofqueryatoms(keywords)againstlocalcontextsofannota;onatoms

23

AtomSimilarity(AHMetric)

•  Similarityoftwoatomstakingintoaccounttheirlocalcontext•  Termsimilarityfunc;onS(A,B)∈[0, 1]•  SetFoflocalcontextfilters

•  Filters()•  exact:ThetwoatomsshouldhaveeitherthesameURI,ortheyshouldbeequivalentconcepts

•  plugin:TheatomBshouldbelongtothesetofhypernymsofAortothesetofrelevantconceptsofA

•  subsume:TheatomBshouldbelongtothesetofthehyponymsofA•  TheatomAmatchestheatomB,withrespecttoafiltersetF,ifandonlyifthereisatleastonefilterf∈F

24

SFunc.on•  Heuris;cfunc;onthattakesintoaccounttheinforma;oncaptureinlocalcontexts

•  aandb(a>b)aredefinedmanuallybasedondomainknowledgeregardingthequalityofmul;mediaanalysisthatproducestheatoms(e.g.aesthe;csextrac;on)

•  Theempiricalden;;onofthesevalues(currentlya=0.7andb=0.3)aimstopromotepluginmatches(r2)oversubsumed(r3).

25

Evaluation

26

DigitalContent

• DeutscheWelle(DW)andEuropeanaaretwokeycontentproviders

•  DWprovidestheirdocumentaryandmoviearchives.•  Europeanaprovidestheirlargearchiveofpain;ngs,picturesofcontemporaryartworkandrelatedcri;cs.

•  ThegeneratedV4Annannota;ongraphscontaintheatomsthathavebeenextractedfromtheanalysiscomponents,alongwithinterconnec;onsamongtheannota;onresources.

"description": "Our drone shows you the Bauhaus University in Weimar, Thuringia."

27

User-centredEvalua.on

1.  Collectqualita;vefeedbackontheresults,aswellasonnon-func;onalaspects,suchasqueryresponse;me

2.  (andmostimportant)togenerateanannota;ondatasetandassesstheperformanceofV4Ann

• Alistofrelevantresourceshasbeenprovided,suchassquarenames,monuments,buildingtypes,etc.,inordertohelpthemconductrelevantqueries.

• Usersfilledinafive-pointscaleques;onnaire(1-completelyagree,5-completelydisagree).

28

User-centredEvalua.on

• Qualityofatoms:Thequalityandrelevanceoflocalcontextsdependsontheperformanceofcontentanalysis,e.g.visualandtextualanalysis.Visualanalysisprovides,inprinciple,beWerresultsthantextanalysis

• Retrievalresults:Thesystemachievesgoodtop-rankedaccuracy,howeverthecompletesetoftheresultscontainquitealotirrelevantentries

• Response;me:Theresponse;meofthesystemwasposi;velyassessed.Theaverageresponse;mewas4.1seconds,whichincludesqueryanalysis,buildingoflocalcontextandsearchalgorithmexecu;on

29

SystemEvalua.on

•  Asexpected,thestricterthefilteris,themoreaccurateresultsweobtain(highprecision)withlow,however,recall

•  Ahigherhvalueleadstomoregenericlocalcontextsthataffectprecision

•  Forexample,thethird-levelWordNethypernymof“tower”is“unit”,whichistoogeneric

•  Themorespecificthelabel/atomis,themoreroomforaddi;onalcontextexists

30

Conclusion

•  Ontology-basedframeworkforrepresen;ng,linkingandenrichingresultsofmul;mediaanalysisondigitalcontent

•  Reusesexis;ngstandardsandschemata,buildingtheatom-basedannota;onsgraphsontopofstandardontologies,controlledvocabulariesandpaWerns

•  WADMpaWern• Weevaluatedtheframeworkusingactualmul;mediacontentandatomsprovidedbytheV4Designmodules

•  V4AnnisaccessiblethroughRhinoceros3DandUnitypluginsdevelopedintheV4Designproject.

31

FutureDirec.ons

•  Implementcontext-awarealgorithmstoimprovetheclassifica;onaccuracyofincomingatoms

•  Forexample,ifthewrongstyleforapain;ngisprovidedbyaesthe;cs,thiswillaffectprecision,sinceV4Anndoesnotaimatimprovingtheclassifica;onofincomingatoms

•  Extendthecontext-awareretrievalalgorithmwithmoresophis;catedsimilaritymetricsandfunc;ons

32

Thankyou!

gmeditsk@i;.gr

33

hWp://www.v4design.eu/

hWps://mklab.i;.gr