hobbit project overview @ eswc hobbit workshop

Download HOBBIT Project Overview @ ESWC HOBBIT Workshop

If you can't read please download the document

Post on 14-Jan-2017

240 views

Category:

Engineering

12 download

Embed Size (px)

TRANSCRIPT

  • HOBBITProject Overview

    Axel Ngonga

    Horizon 2020GA No 688227

    01/12/201630/11/2018

    ESWC 2016Crete, GreeceJune 1, 2016

    Axel Ngonga (InfAI) Project Overview June 1, 2016 1 / 13

  • A Lot of Data

    1

    1http://www.ibmbigdatahub.com/infographic/four-vs-big-dataAxel Ngonga (InfAI) Project Overview June 1, 2016 2 / 13

    http://www.ibmbigdatahub.com/infographic/four-vs-big-data

  • A Lot of Tools

    2

    2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.pngAxel Ngonga (InfAI) Project Overview June 1, 2016 3 / 13

    https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.png

  • Core Questions

    Developers: How good is my tool?Vendors: Who is my tool good for?Users: Which tool(s) should I use formy application?

    Axel Ngonga (InfAI) Project Overview June 1, 2016 4 / 13

  • Many Questions

    Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available?Which key performance indicatorsare relevant?How well do or should toolsperform?How do existing solutions performw.r.t. relevant indicators?

    Axel Ngonga (InfAI) Project Overview June 1, 2016 5 / 13

  • GERBIL

    Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10 fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

    Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

  • GERBIL

    Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10 faster

    Archiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

    Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

  • GERBIL

    Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10 fasterArchiving of resultsCiteable URIsAdditional analysis

    Open-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

    Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

  • GERBIL

    Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10 fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

    Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

  • GERBIL

    Annotator TasksNIF-based Annotators 2519Babelfy 958DBpedia Spotlight 922TagMe 2 811WAT 787Kea 763Wikipedia Miner 714NERD-ML 639Dexter 587AGDISTIS 443Entityclassifier.eu NER 410FOX 352Cetus 1

    Axel Ngonga (InfAI) Project Overview June 1, 2016 7 / 13

  • HOBBIT

    Rationale

    A community-driven benchmarking framework for the community

    Focus on Big Linked DataCover all steps of the Linked Data lifecycle

    Used by a growing number of companiesMature and maturing technologies

    Open benchmarks based on industrial dataand use cases

    Axel Ngonga (InfAI) Project Overview June 1, 2016 8 / 13

  • HOBBIT

    Rationale

    A community-driven benchmarking framework for the community

    Focus on Big Linked DataCover all steps of the Linked Data lifecycle

    Used by a growing number of companiesMature and maturing technologies

    Open benchmarks based on industrial dataand use cases

    Axel Ngonga (InfAI) Project Overview June 1, 2016 8 / 13

  • Aims

    1 Gather real requirementsPerformance indicatorsPerformance thresholds

    2 Develop benchmarks based on real data3 Provide universal benchmarking platform

    Standardized hardwareComparable results

    4 Periodic benchmarking challenges5 Periodic reporting6 Found independent Hobbit association

    Axel Ngonga (InfAI) Project Overview June 1, 2016 9 / 13

  • Overview

    Data Collection

    Industrydata

    Measure Collection

    Benchmark Creation

    Benchmark 1

    KPIsTasks

    KPIsTasksKPIsTasks

    KPIsTasks

    KPIsTasks

    KPIsTasks

    Benchmark 2

    Benchmark n

    HOBBITPlatform

    Solution 1

    Solution k

    Solution 2

    Challenges

    Reports

    Participants/Community

    Axel Ngonga (InfAI) Project Overview June 1, 2016 10 / 13

  • Architecture

    Controller

    Data Generator

    Task Generator

    Data Generator

    Data Generator

    Task Generator

    Task Generator

    FrontendSystem Adapter

    System

    data flowcreates component

    Store

    SPARQL Endpoint

    Analysis

    BenchmarkEvaluator Module

    Eval. Store

    Message BusNode Observer

    Logging

    Axel Ngonga (InfAI) Project Overview June 1, 2016 11 / 13

  • We Offer Benchmarks

    Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity

    Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction

    Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised

    Storage and CurationTriple storesVersioningIncl. updates

    Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks

    Axel Ngonga (InfAI) Project Overview June 1, 2016 12 / 13

  • Features of the HOBBIT platform

    Addresses all steps of the LinkedData LifecycleBenchmarks derived from industryuse casesReal data under the bechmarksScalable size of benchmarksOpen-source implementationOnline instance on server clusterUses established deploymenttechnologies

    Axel Ngonga (InfAI) Project Overview June 1, 2016 13 / 13

  • Join HOBBIT

    Participate in the surveyJoin the HOBBIT communityJoin the split sessionsProvide KPIsProvide datasetsJoin the platform development

    Axel Ngonga (InfAI) Project Overview June 1, 2016 14 / 13

  • Thank You

    http://project-hobbit.eu/get-involved/

    http://goo.gl/forms/1iRIoG4Xpb

    https://twitter.com/hobbit_project

    Axel Ngonga (InfAI) Project Overview June 1, 2016 15 / 13

    http://goo.gl/forms/1iRIoG4Xpbhttps://twitter.com/hobbit_projecthttp://project-hobbit.eu/get-involved/

  • Overview