HOBBITSurvey results

Ruben Taelman

Horizon 2020GA No 688227

01/12/2016–30/11/2018

ESWC 2016Crete, GreeceJune 22, 2016

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 1 / 1

Goal

In what areas are organizations active?What do people expect from benchmarks?How are benchmarks being used?

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 2 / 1

Participant profiles

50 participants

Profile CountSolution provider 39Technology user 39Scientific community 40

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 3 / 1

Linked Data areas

Evenly distributed:Storage / QueryingInterlinkingClassification / EnrichmentDiscoveryExtractionReasoning

Solution providers mainly active in storage and querying.

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 4 / 1

Linked Data areas

Other areas:ReportingVisualization

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 5 / 1

Key performance indicators

CorrectnessAccuracy (precision, recall, F-measure, mean reciprocal rank)Runtime / SpeedScalabilityMemory usageCPU usage

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 6 / 1

Benchmarks

48% already evaluates software using a benchmark78% of benchmarking done for storage and querying, much less for otherdomains

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 7 / 1

Datasets

Types of datasets:22% synthetic data46% real data32% both

66% is or can be made public!

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 8 / 1

Dataset sizes

Innumber of triples.

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 9 / 1

Survey still open!

http://goo.gl/forms/1iRIoG4Xpb

Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 10 / 1