1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

Sherlock: A Deep Learning Approach toSemantic Data Type Detection

Anonymous AuthorsInstitutionE-mail

4. Semantic Type Detection2. Sampled Dataset and Features1. Source Corpus 3. Training and Testing Set

DateCountryNigeria

...Germany

Name

Canada

M Buhari 08-18-201806-14-2015

07-20-2017

... ...A Merkel

J Trudeau

UnseenData

Model

PredictedTypes andConfidences

Location: 0.9Name: 0.7Year: 0.8

Prediction

Date

Character Distributions

Word Embeddings

Paragraph Vectors

Global Statistics

Country

Name

...

FeatureExtraction

Join

Join

Training

Feature Categories Types

Neural Network

FeaturesExactMatchingColumn Header

Column Values

...

Figure 1: Data processing and analysis flow, starting from (1) a corpus of real-world datasets, proceeding to (2) feature extrac-tion, (3) mapping from features to ground truth semantic types from column headers, and (4) model training and prediction.

ABSTRACTCorrectly detecting the semantic type of data columns is crucial fordata science tasks such as automated data cleaning, schema match-ing, and data discovery. Existing data preparation and analysis sys-tems rely on dictionary lookups and regular expression matching todetect semantic types. However, these matching-based approachesoften are not robust to dirty data and only detect a limited numberof types. We introduce Sherlock, a multi-input deep neural networkfor detecting semantic types. We train Sherlock on 686, 765 datacolumns retrieved from the VizNet corpus by matching 78 seman-tic types from DBpedia to column headers. We characterize eachmatched column with 1, 588 features describing the statistical prop-erties, character distributions, word embeddings, and paragraphvectors of column values. Sherlock achieves a support-weightedF1 score of 0.89, exceeding that of a decision tree baseline, dictio-nary and regular expression benchmarks, and the consensus ofcrowdsourced annotations.

CCS CONCEPTS• Computing methodologies → Machine learning; Knowl-edge representation and reasoning; • Information systems→ Data mining.

KEYWORDSTabular data, type detection, semantic types, deep learning

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).KDD’19, August 2019, Anchorage, Alaska USA© 2019 Copyright held by the owner/author(s).ACM ISBN xxxx.https://doi.org/xxxx

ACM Reference Format:Anonymous Authors. 2019. Sherlock: A Deep Learning Approach to Se-mantic Data Type Detection. In Proceedings of ACM Knowledge Discoveryand Data Mining conference (KDD’19). ACM, New York, NY, USA, Article x,9 pages. https://doi.org/xxxx

1 INTRODUCTIONData preparation and analysis systems rely on correctly detectingtypes of data columns to enable and constrain functionality. Forexample, automated data cleaning facilitates the generation of cleandata through validation and transformation rules that depend ondata type [15, 26]. Schema matching identifies correspondences be-tween data objects, and frequently uses data types to constrain thesearch space of correspondences [25, 35]. Data discovery surfacesdata relevant to a given query, often relying on semantic similaritiesacross tables and columns [6, 7].

While most systems reliably detect atomic types such as string,integer, and boolean, semantic types are disproportionally morepowerful and in many cases essential. Semantic types provide finer-grained descriptions of the data by establishing correspondencesbetween columns and real-world concepts and as such, can helpwith schema matching to determine which columns refer to thesame real-world concepts, or data cleaning by determining theconceptual domain of a column. In same cases, the detection of asemantic types can be easy. For example, an ISBN or credit cardnumber are generated according to strict validation rules, lendingthemselves to straightforward type detection with just a few rules.But most types, including location, birth date, and name, do notadhere to such structure, as shown in Table 1.

Existing open source and commercial systems take matching-based approaches to semantic type detection. For example, regularexpression matching captures patterns of data values using pre-defined character sequences. Dictionary approaches use matchesbetween data headers and values with internal look-up tables.While sufficient for detecting simple types, these matching-based

1

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

KDD’19, August 2019, Anchorage, Alaska USA Anonymous Authors

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

Table 1: Data values sampled from real-world datasets.

Type Sampled values

location TBA | Chicago, Ill. | Detroit, Mich. | Nashville, Tenn.location UNIVERSITY SUITES | U.S. 27; NA | NORSE HALLlocation Away | Away | Home | Away | Awaydate 27 Dec 1811 | 1852 | 1855 | - | 1848 | 1871 | 1877date – –, 1922 | – –, 1902 | – –, 1913 | – –, 1919date December 06 | August 23 | Nonename Svenack | Svendd | Sveneldritch | Svengöranname HOUSE, BRIAN | HSIAO, AMY | HSU, ASTRIDname D. Korb | K. Moring | J. Albanese | l. dunn

approaches are often not robust to malformed or dirty data, sup-port only a limited number of types, and under-perform for typeswithout strict validations. For example, Figure 2 shows that Tableaudetects a column labeled “Continent Name” as string. After re-moving column headers, no semantic types are detected. Note thatmissing headers or incomprehensible headers are not uncommon.For example, SAP’s system tableT 005 contains country informationand column NMFMT is the standard name field, whereas INTCArefers to the ISO code or XPLZS to zip-code.

Country/Region String Latitude Longitude Country/Region String

String String Decimal Decimal String String

Detected Types Without Column Headers

Detected Types With Column Headers

Remove Headers

Figure 2: Data types detected by Tableau Desktop 2018.3 fora dataset of country capitals, with and without headers.

Machine learning models, coupled with large-scale training andbenchmarking corpora, have proven effective at predictive tasksacross domains. Examples include the AlexNet neural networktrained on ImageNet for visual recognition and the Google NeuralMachine Translation system pre-trained on WMT parallel corporafor language translation. Inspired by these advances, we introduceSherlock, a deep learning approach to semantic type detectiontrained on a large corpus of real-world columns.

To begin, we consider 78 semantic types described by T2Dv2Gold Standard,1 which matches properties from the DBpedia on-tology with column headers from the WebTables corpus. Then, weuse exact matching between semantic types and column headersto extract 686, 765 data columns from the VizNet corpus [14], alarge-scale repository of real world datasets collected from the web,popular visualization systems, and open data portals.1http://webdatacommons.org/webtables/goldstandardV2.html

We consider each column as a mapping from column valuesto a column header. We then extract 1, 588 features from eachcolumn, describing the distribution of characters, semantic contentof words and columns, and global statistics such as cardinality anduniqueness. Treating column headers as ground truth labels of thesemantic type, we formulate semantic type detection as a multiclassclassification problem.

A multi-input neural network architecture achieves a support-weighted F1-score of 0.89, exceeding that of a decision tree base-line model, two matching-based approaches, and the consensusof crowdsourced annotations. We then examine types for whichthe neural network demonstrates high and low performance, inves-tigate the contribution of each feature category to model perfor-mance, extract feature importances from the decision tree baseline,and present an error-reject curve suggesting the potential of com-bining learned models with human annotations.

To conclude, we discuss promising avenues for future researchin semantic type detection, such as assessing training data qualityat scale, enriching feature extraction processes, and establishingshared benchmarks. To support benchmarks for future research,we open source our data, code, and trained model.2 For developerswishing to integrate Sherlock into existing systems, we distribute apretrained model as a Python library.3

Key contributions:(1) Data (§3): Demonstrating a scalable process for match-

ing 686, 675 columns from VizNet corpus for 78 semantictypes, then describing with 1, 588 features

(2) Model (§4): Formulating type detection as a multiclassclassification problem, then contributing a novel multi-input neural network architecture.

(3) Results (§5): Benchmarking predictive performance againsta decision tree baseline, two matching-based models, andcrowdsourced consensus.

2 RELATEDWORKSherlock is informed by existing commercial and open source sys-tems for data preparation and analysis, as well as prior researchwork on ontology-based, feature-based, probabilitic, and synthesizedapproaches to semantic type detection.

Commercial and open source. Semantic type detection enhancesthe functionality of commercial data preparation and analysis sys-tems such as Microsoft Power BI [20], Trifacta [31], and GoogleData Studio [12]. To the best of our knowledge, these commercialtools rely on manually defined regular expression patterns dictio-nary lookups of column headers and values to detect a limited set ofsemantic types. For instance, Trifacta detects around 10 types (e.g.,gender and zip code) and Power BI only supports time-related se-mantic types (e.g., date/time and duration). Open source librariessuch as messytables [10], datalib [9], and csvkit [13] similarly useheuristics to detect a limited set of types. Benchmarking directlyagainst these systems was unfeasible due to the small number ofsupported types and lack of extensibility. However, we compare

2Code and data: https://placeholder.edu/data3Python library: https://placeholder.edu/library

2

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

Sherlock: A Deep Learning Approach to Semantic Data Type Detection KDD’19, August 2019, Anchorage, Alaska USA

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

against learned regular expression and dictionary-based bench-marks representative of the approaches taken by these systems.

Ontology-based. Prior research work, with roots in the semanticweb and schemamatching literature, provide alternative approachesto semantic type detection. One body of work leverages existingdata on the web, such as WebTables [5], and ontologies (or, knowl-edge bases) such as DBPedia [2], Wikitology [30], and Freebase [4].Venetis et al. [33] construct a database of value-type mappings,then assign types using a maximum likelihood estimator based oncolumn values. Syed et al. [30] use column headers and values tobuild a Wikitology query, the result of which maps columns totypes. Informed by these approaches, we looked towards existingontologies to derive the 275 semantic types considered in this paper.

Feature-based. Several approaches capture and compare prop-erties of data in a way that is ontology-agnostic. Ramnandan etal. [27] use heuristics to first separate numerical and textual types,then describe those types using the Kolmogorov-Smirnov (K-S) testand Term Frequency-Inverse Document Frequency (TF-IDF), re-spectively. Pham et al. [23] use slightly more features, including theMann-Whitney test for numerical data and Jaccard similarity for tex-tual data, to train logistic regression and random forest models. Weextend these feature-based approaches with a significantly largerset of features that includes character-level distributions, wordembeddings, and paragraph vectors. Orders of magnitude morefeatures and training samples motivates our use of high-capacitymachine learning models such as neural networks. While code forbenchmarking thesemodels was not available, we include a decisiontree model to represent “simpler” machine learning models.

Probabilistic. The third category of prior work employs a prob-abilistic approach. Goel et al. [11] use conditional random fieldsto predict the semantic type of each value within a column, thencombine these predictions into a prediction for the whole column.Limaye et al. [19] use probabilistic graphical models to annotatevalues with entities, columns with types, and column pairs withrelationships. These predictions simultaneously maximize a po-tential function using a message passing algorithm. Probabilisticapproaches are complementary to our machine learning-based ap-proach by providing a means for combining column-specific pre-dictions. However, as with prior feature-based models, code forretraining these models was not made available for benchmarking.

Synthesized. Puranik [24] proposes a “specialist approach” com-bining the predictions of regular expressions, dictionaries, and ma-chine learning models. More recently, Yan and He [34] developeda system that, given a search keyword and set of positive exam-ples, synthesizes type detection logic from open source GitHubrepositories. This system provides a novel approach to leveragingdomain-specific heuristics for parsing, validating, and transformingsemantic data types. While both approaches are exciting, the codeunderlying these systems was not available for benchmarking.

3 DATAWe describe the semantic types we consider, how we extracted datacolumns from a large repository of real-world datasets, and ourfeature extraction procedure.

3.1 Data CollectionOntologies such asWordNet [32] andDBpedia [2] describe semanticconcepts, properties of such concepts, and relationships betweenthem. To constrain the number of types we consider, we adopt thetypes described by the T2Dv2 Gold Standard,1 the result of a studymatching DBpedia properties [29] with columns from the WebTables web crawl corpus [5]. These 275 DBpedia properties, suchas country, language, and industry, represent semantic typescommonly found in datasets scattered throughout the web.

To expedite the collection of real-world data from diverse sources,we use the VizNet repository [14], which aggregates and character-izes data from two popular online visualization platforms and opendata portals, in addition to the Web Tables corpus. For feasibility,we restricted ourselves to the first 10M Web Tables datasets, butconsidered the remainder of the repository. We then match datacolumns from VizNet that have headers corresponding to our 275types. To accomodate variation in casing and formatting, singleword types matched case-altered modifications (e.g., name = Name =NAME) and multi-word types included concatenations of constituentwords (e.g., release date = releaseDate).

The matching process resulted in 6, 146, 940 columns matchingthe 275 considered types. Manual verification indicated that themajority of columns were plausibly described by the correspond-ing semantic type, as shown in Table 1. In other words, matchingcolumn headers as ground truth labels of the semantic type yieldedhigh quality training data.

3.2 Feature ExtractionTo create fixed-length representations of variable-length columns,aid interpretation of results, and provide “hints” to our neural net-work, we extract features from each column. To capture differentproperties of columns, we extract four categories of features: globalstatistics (27), aggregated character distributions (960), pretrainedword embeddings (200), and self-trained paragraph vectors (400).

Global statistics. The first category of features describes high-level statistical characteristics of columns. For example, the “columnentropy” feature describes how uniformly values are distributed.Such a feature helps differentiate between types that contain morerepeated values, such as gender, from types that contain manyunique values, such as name. Other types, like weight and sales,may consist of many numerical characters, which is captured by the“mean of the number of numerical characters in values.” A completelist of these 27 features can be found in Table 8 in the Appendix.

Character-level distributions. Preliminary analysis indicated thatsimple statistical features such as the “fraction of values with nu-merical characters” provide surprising predictive power. Motivatedby these results and the prevalence of character-based matching ap-proaches such as regular expressions, we extract features describingthe distribution of characters in a column. Specifically, we computethe count of all 96 ASCII-printable characters (i.e., digits, letters,

3

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

KDD’19, August 2019, Anchorage, Alaska USA Anonymous Authors

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

and punctuation characters, but not whitespace) within each valueof a column. We then aggregate these counts with 10 statisticalfunctions (i.e., any, all, mean, variance, min, max, median, sum,kurtosis, skewness), resulting in 960 features. Example featuresinclude “whether all values contain a ‘-’ character” and the “meannumber of ‘/’ characters.”

Word embeddings. For certain semantic types, columns frequentlycontain commonly occurring words. For example, the city typecontains values such as New York City, Paris, and London. To char-acterize the semantic content of these values, we used word embed-dings that map words to high-dimensional fixed-length numericvectors. In particular, we used a pre-trained GloVe dictionary [22]containing 50-dimensional representations of 400K English wordsaggregated from 6B tokens, used for tasks such as text similar-ity [16]. For each value in a column, if the value is a single word, welook up the word embedding from the GloVe dictionary. We omitthe a term if it did not appear in the GloVe dictionary. For valuescontaining multiple words, we looked up each distinct word andrepresented the value with the mean of the distinct word vectors.Then, we computed the mean, mode, median and variance of wordvectors across all values in a column.

Paragraph vectors. To represent entire columns with numericalvectors, we implemented the Distributed Bag of Words versionof Paragraph Vectors (DBoW-PV) [18]. Paragraph vectors wereoriginally developed to learn representations for pieces of texts,but have proven effective for more general tasks, such as documentsimilarity [8]. In our implementation, each column is a “paragraph”while values within a column are “words”: both the entire columnand constituent values are represented by one-hot encoded vectors.We then randomly select a value vector, concatenate the remainingcolumn and value vectors, and train a model to predict the formerfrom the latter. Using the Gensim library [28] and a holdout set,we trained this model for 20 iterations. We used this model to mapcolumns to 400-dimensional paragraph vectors, which provided abalance between predictive power and computational tractability.

3.3 Filtering and PreprocessingCertain types occur more frequently in the VizNet corpus thanothers. For example, description and city are more commonthan collection and continent. To address this heterogeneity,we limited the number of columns to at most 15K per class andexcluded the 10% types containing less than 1K columns.

Other semantic types, especially those describing numerical con-cepts, are unlikely to be represented by word embeddings. To con-tend with this issue, we filtered out the types for which at least15% of the columns did not contain a single word that is present inthe GloVe dictionary. This filter resulted in a final total of 686,765columns corresponding to 78 semantic types, of which a list isincluded in Table 7 in the Appendix. The distribution of number ofcolumns per semantic type is shown in Figure 3.

Before modeling, we preprocess our features by creating an ad-ditional binary feature indicating whether word embeddings weresuccessfully extracted for a given column. Including this featureresults in a total of 1,588 features. Then, we impute missing valuesacross all features with the mean of the respective feature.

DescriptionAlbumWeight

RankCity

LocationProductGrades

PlaysElevation

DepthFamily

CollectionBirth place

CapacityContinent

0 2,500 5,000 7,500 10,000 12,500 15,000Number of Samples

Sem

antic

Typ

es

Figure 3: Number of columns per semantic type extractedfrom VizNet after filtering out the types with less than 1Kcolumns and more than 15% of columns not present in theGloVe dictionary.

4 METHODSWe describe our deep learning model, decision tree baseline, twomatching-based benchmarks, and crowdsourced consensus bench-mark. Then, we explain our training and evaluation procedures.

4.1 Sherlock: A Multi-input Neural NetworkPrior machine learning approaches to semantic type detection [19,33] trained simple models, such as logistic regression, on relativelysmall feature sets. We consider a significantly larger number offeatures and samples, which motivates our use of a feedforwardneural network. Specifically, given the different number of featuresand varying noise levels within each feature category, we use amulti-input architecture with hyperparameters shown in Figure 4.

At a high-level, we train subnetworks for each feature categoryexcept the statistical features, which consist of only 27 features.These subnetworks “compress” input features to an output of fixeddimension. We chose this dimension to be equal to the number oftypes in order to evaluate each subnetwork independently. Then,we concatenate the weights of the three output layers with thestatistical features to form the input layer of the primary network.

Each network consists of two hidden layers with rectified lin-ear unit (ReLU) activation functions. Iteration over hidden layersizes between 100 and 1, 000 (i.e., on the order of the input layerdimension) resulted in hidden layer sizes of 300, 200, and 400 forthe character-level, word embedding, and paragraph vector sub-networks, respectively. To prevent overfitting, we included dropout layers and weight decay terms. The neural network, which werefer to as “Sherlock,” is implemented in TensorFlow [1].

4.2 BenchmarksTo measure the relative performance of Sherlock, we compareagainst four benchmarks.

Decision tree. The first benchmark is a decision tree, a non-parametric machine learning model with reasonable “out-of-the-box” performance and straightforward interpretation. We use the

4

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

Sherlock: A Deep Learning Approach to Semantic Data Type Detection KDD’19, August 2019, Anchorage, Alaska USA

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

Primary Network

Batch Norm(size=128)

Concatenate

ReLU (500 units)

ReLU (500 units)

Output (78 units)

Softmax

Input Features

ReLU (x units)

Batch Norm

Dropout

ReLU (x units)

Output (78 units)

Softmax

Feature-specificSubnetwork

Dropout(rate=0.3)

Output

Character

Output

Word

Output

Paragraph

Statistical

MetricAccuracyLoss FunctionCross-EntropyOptimizerAdam

Epochs100Early Stopping Patience5

Hyperparameters

Learning Rate1e-4Weight Decay Rate1e-4

Figure 4: Architecture and hyperparameters of feature-specific subnetworks and primary network.

decision tree to represent the simpler models found in prior re-search, such as the logistic regression and random forest modelsused in Pham et al. [23]. Learning curves indicated that decisiontree performance plateaued beyond a depth of 50, which we thenused as the maximum depth. For the remaining parameters, weused the default settings in the scikit-learn package [21].

Dictionary. Dictionaries are commonly used to detect seman-tic types that contain a finite set of valid values, such as country,day, and language. The first matching-based benchmark is a dic-tionary that maps column values or headers to semantic types. Foreach type, we collected the 1, 000 most frequently occurring valuesacross all columns, resulting in 78, 000 { value : type } pairs. Forexample, Figure 5 shows examples of entries mapped to the gradestype. Given an unseen data column at test time, we compare 1, 000randomly selected column values to each entry of the dictionary,then classify the column as the most frequently matched type.

Dictionary Entries (20 out of 1000)

Learned Regular Expression\w\w \-(?: \w\w)*+|[06PK][A-Za-z]*+\-\w|\w\w\w\w\w\w \w\w \w\w\w \w\w

9-12K-5PK - 0509 - 12

KG - 05PRESCHOOL-56-8KG-06

06 - 08PK -PRESCHOOL-8PK - 8

KG - 12K-806 - 12K-^

KG - 08- 12PK - 12PK - 08

Figure 5: Examples of dictionary entries and a learned regu-lar expression for the grades type.

Learned regular expressions. Regular expressions are frequentlyused to detect semantic types with common character patterns, suchas address, birth date, and year. The second matching-basedbenchmark uses patterns of characters specified by learned regularexpressions. We learn regular expressions for each type using theevolutionary procedure of Bartoli et al. [3]. Consistent with theoriginal setup, we randomly sampled 50 “positive values” from eachtype, and 50 “negative” values from other types. An example of alearned regular expression in Java format for the grades type isshown in Figure 5. As with the dictionary benchmark, we match1, 000 randomly selected values against learned regular expressions,then use majority vote to determine the final predicted type.

Crowdsourced annotations. To assess the performance of humanannotators at predicting semantic type, we conducted a crowd-sourced experiment. The experiment began by defining the conceptsof data and semantic type, then screened out participants unable toselect a specified semantic type. After the prescreen, participantscompleted three sets of ten questions separated by two attentionchecks. Each question presented a list of data values, asked “Whichone of the following types best describes these data values?”, andrequired participants to select a single type from a scrolling menuwith 78 types. Questions were populated from a pool of 780 samplescontaining 10 randomly selected values from all 78 types.

We used the Mechanical Turk crowdsourcing platform [17] torecruit 390 participants that were native English speakers and had≥95% HIT approval rating, ensuring high-quality annotations. Par-ticipants completed the experiment in 16 minutes and 22 seconds onaverage and were compensated 2 USD, a rate slightly exceeding theUnited States federal minimum wage of 7.25 USD. Detailed workerdemographics are described in Appendix A.2. Overall, 390 partic-ipants annotated 30 samples each, resulting in a total of 11, 700annotations, or an average of 15 annotations per sample. For eachsample, we used the most frequent (i.e., the mode) type from the 15annotations as the crowdsourced consensus annotation.

4.3 Training and EvaluationTo ensure consistent evaluation across benchmarks, we divided thedata into 60/20/20 training/validation/testing splits. To account forclass imbalances, we evaluate model performance using the averageF1-score = 2 × (precision × recall)/(precision + recall), weightedby the number of columns per class in the test set (i.e., the support).To estimate the mean and 95% percentile error of the crowdsourcedconsensus F1 score, we conducted 105 bootstrap simulations byresampling annotations for each sample with replacement.

Computational effort and space required at prediction time arealso important metrics for models incorporated into user-facingsystems. We measure the average time in seconds needed to extractfeatures and generate a prediction for a single sample, and reportthe space required by the models in megabytes.

5 RESULTSWe report the performance of our multi-input neural network andcompare against benchmarks. Then, we examine types for whichSherlock demonstrated high and low performance, the contributionof each feature category in isolation, decision tree feature impor-tances, and the effect of rejection threshold on performance.

5

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

KDD’19, August 2019, Anchorage, Alaska USA Anonymous Authors

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

5.1 Benchmark ResultsWe compare Sherlock against decision tree, dictionary-based, learnedregular expression, and crowdsourced consensus benchmarks. Ta-ble 2 presents the F1 score weighted by support, runtime in secondsper sample, and size in megabytes of each model.

Table 2: Support-weighted F1 score, runtime at prediction,and size of Sherlock and four benchmarks.

Method F1 Score Runtime (s) Size (Mb)Machine Learning

Sherlock 0.89 0.42 (±0.01) 6.2Decision tree 0.76 0.26 (±0.01) 59.1

Matching-based

Dictionary 0.16 0.01 (±0.03) 0.5Regular expression 0.04 0.01 (±0.03) 0.01

Crowdsourced Annotations

Consensus 0.32 (±0.02) 33.74 (±0.86) −

Wefirst note that bothmachine learningmodels significantly out-perform the matching-based and crowdsourced consensus bench-marks, in terms of F1 score. Inspection of the matching-based bench-marks suggests that dictionaries and learned regular expressionsare prone to “overfitting” on the training set. Feedback from crowd-sourced workers suggests that annotating semantic types with alarge number of types is a challenging and ambiguous task.

Comparing the two machine learning models, Sherlock signifi-cantly outperforms the decision tree baseline. However, the deci-sion tree still demonstrates strong performance relative to otherbenchmarks. For cases in which interpretability of features andpredictions are important considerations, decision trees may be asuitable choice of model.

Despite poor predictive performance, matching-based bench-marks are significantly smaller and faster than both machine learn-ing models. For cases in which absolute runtime and model size arecritical, optimizing matching-based models may be a worthwhileapproach. This trade-off also suggests a hybrid approach of combin-ing matching-based models for “easy” types with machine learningmodels for more ambiguous types.

5.2 Performance for Individual TypesTable 3 displays the top and bottom five types, as measured by theF1 score achieved by Sherlock for that type. High performing typessuch as grades and industry frequently contain a finite set ofvalid values, as shown in Figure 5 for grades. Other types such asbirth date and ISBN, often follow consistent character patterns,as shown in Table 1.

To understand types for which Sherlock performs poorly, weinclude incorrectly predicted examples for the lowest precisiontype (sales) and the lowest recall type (ranking) in Table 4. Fromthe three examples incorrectly predicted as sales, we observe thatpurely numerical values or values appearing in multiple classes (e.g.,currency symbols) present a challenge to type detection systems.From the three examples of incorrectly predicted ranking columns,we again note the ambiguity of numerical values.

Table 3: Top five and bottom five types by F1 score.

Type F1 Score Precision Recall Support

Top 5 Types

Grades 0.991 0.989 0.994 1765ISBN 0.986 0.981 0.992 1430Birth Date 0.970 0.965 0.975 479Industry 0.968 0.947 0.989 2958Affiliation 0.961 0.966 0.956 1768

Bottom 5 Types

Brand 0.685 0.760 0.623 574Person 0.630 0.654 0.608 579Director 0.537 0.700 0.436 225Sales 0.514 0.568 0.469 322Ranking 0.468 0.612 0.349 439

Table 4: Examples of low precision and low recall types.

Examples True type Predicted typeLow Precision

81, 13, 3, 1 Rank Sales316, 481, 426, 1, 223 Plays Sales$, $$, $$$, $$$$, $$$$$ Symbol Sales

Low Recall#1, #2, #3, #4, #5, #6 Ranking Rank3, 6, 21, 34, 29, 36, 54 Ranking Plays1st, 2nd, 3rd, 4th, 5th Ranking Position

5.3 Contribution by Feature CategoryWe trained feature-specific subnetworks in isolation and reportthe F1 scores in Table 5. Word embedding, character distribution,and paragraph vector feature sets demonstrate roughly equal per-formance to each other, and significantly above that of the globalstatistics features, though this may be due to fewer features. Eachfeature set in isolation performs significantly worse than the fullmodel, supporting our combining of each feature set.

Table 5: Performance contribution of isolated feature sets.

Feature set Num. Features F1 ScoreWord embeddings 200 0.79Character distributions 960 0.78Paragraph vectors 400 0.73Global statistics 27 0.25

5.4 Feature ImportancesWe measure feature importance by the total reduction of the Giniimpurity criterion brought by that feature to the decision treemodel.The top 10 most important features from the global statistics andcharacter-level distributions sets are shown in Table 6. While wordembedding and paragraph vector features are important, they aredifficult to interpret and are therefore omitted.

6

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

Sherlock: A Deep Learning Approach to Semantic Data Type Detection KDD’19, August 2019, Anchorage, Alaska USA

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

Table 6: Top-10 features for the decision tree model. “Score”denotes normalized gini impurity.

(a) Top-10 global statistics features (out of 27).

Rank Feature Name Score

1 Number of Values 1.002 Maximum Value Length 0.793 Mean # Alphabetic Characters in Cells 0.434 Fraction of Cells with Numeric Characters 0.385 Column Entropy 0.356 Fraction of Cells with Alphabetical Characters 0.337 Number of None Values 0.338 Mean Length of Values 0.289 Proportion of Unique Values 0.2210 Mean # of Numeric Characters in Cells 0.16

(b) Top-10 character-level distribution features (out of 960).

Rank Feature Name Score

1 Sum of ‘D’ across values 1.002 Mean number of ‘M’ 0.773 Minimum number of ‘-’ 0.694 Skewness of ‘,’ 0.595 Whether all values have a ‘,’ 0.476 Maximum number of ‘g’ 0.457 Skewness of ‘]’ 0.458 Mean number of ‘,’ 0.409 Mean number of ‘z’ 0.3710 Sum of ‘n’ 0.36

Inspecting Table 6a, we find that the “number of values” in acolumn is the most important feature. Certain classes like nameand requirements tended to contain fewer values, while otherslike year and family contained significantly more values. Thesecond most important feature is the “maximum value length” incharacters, which may differentiate classes with long values, suchas address and description, from classes with short values, suchas gender and year.

The top character-level distribution features in Table 6b suggestthe importance of specific characters for differentiating betweentypes. The third most important feature, the “minimum number of‘-’ characters”, likely helps determine datetime-related types. Thefifth most important feature, “whether all values have a ‘,’ charac-ter” may also distinguish datetime-related or name-related types.Further study of feature importances for semantic type detection isa promising direction for future research.

5.5 Rejection CurvesGiven unseen data values, Sherlock assesses the probability of thosevalues belonging to each type, then predicts the type with the high-est probability. Interpreting probabilities as a measure of confidence,we may want to only label samples with high confidence of belong-ing to a type. To understand the effect of confidence threshold onpredictive performance, we present the error-rejection curves ofSherlock and the decision tree model in Figure 6.

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Fraction of Samples Rejected

0.6

0.7

0.8

0.9

1.0

F1 S

core

Wei

ghte

d by

Sup

port

Neural Network (Sherlock)Decision Tree Baseline

Model

Figure 6: Rejection curves showing performance while re-jecting all but the top x% highest confidence samples.

By introducing a rejection threshold of 10% of the samples, Sher-lock reaches an F1 score of ∼0.95. This significant increase in pre-dictive performance suggests a hybrid approach in which low con-fidence samples are manually annotated. Note that the higher re-jection threshold, the lower the error we make in predicting labels,at the cost of needing more expert capacity.

6 DISCUSSIONWe began by considering a set of semantic types described byprior work that identifies correspondences between DBPedia [2]and WebTables [5]. Then, we constructed a dataset consisting ofmatches between those types with columns in the VizNet [14]corpus. Inspection of these columns suggests that such an approachyields training samples with few false positives. After extractingfour categories of features describing the values of each column, weformulate type detection as a multiclass classification task. A multi-input neural network demonstrates high predictive performance atthe classification task compared to decision tree, matching-based,and crowdsourced benchmarks.

Developers have multiple avenues to incorporating ML-basedsemantic type detection approaches into systems. To support theuse of Sherlock “out-of-the-box,” we distribute Sherlock as a Pythonlibrary3 that can be easily installed and incorporated into existingcodebases. For developers interested in a different set of seman-tic types, we open source our training and analysis scripts.2 Therepository also supports developers wishing to retrain Sherlockusing data from their specific data ecologies, such as enterprise orresearch settings with domain-specific data.

To close, we identify four promising avenues for future research:(1) enhancing the quantity and quality of the training data, (2)increasing the number of considered types, (3) enriching the setof features extracted from each column, and (4) developing sharedbenchmarks.

Enhancing data quantity and quality. Machine learning modelperformance is limited by the number of training examples. Sher-lock is no exception. Though the VizNet corpus aggregates datasets

7

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

KDD’19, August 2019, Anchorage, Alaska USA Anonymous Authors

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

from four sources, there is an opportunity to incorporate train-ing examples from additional sources, such as Kaggle,4 datasetsincluded alongside the R statistical environment,5 and the ClueWebweb crawl of Excel spreadsheets.6 We expect increases in trainingdata diversity to improve the robustness and generalizability ofSherlock.

Model predictions quality is further determined by the corre-spondence between training data and unseen testing data, such asdatasets uploaded by analysts to a system. Our method of matchingsemantic types with columns from real-world data repositories af-fords both the harvesting of training samples at scale and the abilityto use aspects of “dirty” data (e.g., missing values) as features. Whilewe verified the quality of training data through manual inspection,there is an opportunity to label data quality at scale by combiningcrowdsourcing with active learning. By assessing the quality ofeach training dataset, such an approach would support trainingsemantic type detection models with completely “clean” data atscale.

Increasing number of semantic types. To ground our approach inprior work, this paper considered 78 semantic types described bythe T2Dv2 Gold Standard. While 78 semantic types is a substantialincrease over what is supported in existing systems, it is a smallsubset of entities from existing knowledge bases: the DBPedia on-tology [2] covers 685 classes, WordNet [32] contains 175K synonymsets, and Knowledge Graph7 contains millions of entities. The enti-ties within these knowledge bases, and hierarchical relationshipsbetween entities, provide an abundance of semantic types.

In lieu of a relevant ontology, researchers can count frequencyof column headers in available data to determine which semantictypes to consider. Such a data-driven approach would ensure themaximum number of training samples for each semantic type. Addi-tionally, these surfaced semantic types are potentially more specificto usecase and data ecology, such as data scientists integratingenterprise databases within a company.

Enriching feature extraction. We incorporate four categories offeatures that describe different aspects of column values. A promis-ing approach is to include features that describe relationships be-tween columns (e.g., correlation, number of overlapping values,and name similarity), aspects of the entire dataset (e.g., number ofcolumns), and source context (e.g., webpage title for scraped tables).Additionally, while we used features to aid interpretation of results,neural networks using raw data as input are a promising directionof research. For example, a character-level recurrent neural networkcould classify concatenated column values.

Developing shared benchmarks. Despite rich prior research in se-mantic type detection, we could not find a benchmark with publiclyavailable code that accommodates a larger set of semantic types.We therefore incorporated benchmarks that approximated state-of-the-art data systems, to the best of our knowledge. However,domains such as image classification and language translation havebenefited from shared benchmarks and test sets. Towards this end,

4https://www.kaggle.com/datasets5https://github.com/vincentarelbundock/Rdatasets6http://lemurproject.org/clueweb09.php7https://developers.google.com/knowledge-graph

we hope that open-sourcing the data and code used in this papercan benefit future research.

7 CONCLUSIONCorrectly detecting semantic types is critical to many importantdata science tasks. Machine learning models coupled with large-scale data repositories have demonstrated success across domains,and suggest a promising approach to semantic type detection. Sher-lock provides a step forward towards this direction.

REFERENCES[1] Martín Abadi et al. 2016. TensorFlow: A system for large-scale machine learning.

In 12th USENIX Symposium on Operating Systems Design and Implementation(OSDI 16). 265–283.

[2] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak,and Zachary Ives. 2007. DBpedia: A nucleus for a web of open data. (2007),722–735.

[3] Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. 2016. Infer-ence of regular expressions for text extraction from examples. IEEE Transactionson Knowledge and Data Engineering 28, 5 (2016), 1217–1230.

[4] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.2008. Freebase: A Collaboratively Created Graph Database for Structuring HumanKnowledge. In Proceedings of the 2008 ACM SIGMOD International Conference onManagement of Data (SIGMOD ’08). ACM, New York, NY, USA, 1247–1250.

[5] Michael J. Cafarella, Alon Halevy, Daisy Zhe Wang, Eugene Wu, and Yang Zhang.2008. WebTables: Exploring the Power of Tables on the Web. Proc. VLDB Endow.1, 1 (Aug. 2008), 538–549. https://doi.org/10.14778/1453856.1453916

[6] Raul Castro Fernandez, Ziawasch Abedjan, Famien Koko, Gina Yuan, SamuelMadden, and Michael Stonebraker. 2018. Aurum: A Data Discovery System.1001–1012.

[7] Raul Castro Fernandez, Essam Mansour, Abdulhakim Qahtan, Ahmed Elma-garmid, Ihab Ilyas, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, andNan Tang. 2018. Seeping Semantics: Linking Datasets Using Word Embeddingsfor Data Discovery. https://doi.org/10.1109/ICDE.2018.00093

[8] Andrew M Dai, Christopher Olah, and Quoc V Le. 2015. Document embeddingwith paragraph vectors. arXiv preprint arXiv:1507.07998 (2015).

[9] Interactive Data Lab. 2019. Datalib: JavaScript Data Utilities. http://vega.github.io/datalib

[10] Open Knowledge Foundation. 2019. messytables · PyPi. https://pypi.org/project/messytables

[11] Aman Goel, Craig A Knoblock, and Kristina Lerman. 2012. Exploiting structurewithin data for accurate labeling using conditional random fields. In Proceedingson the International Conference on Artificial Intelligence (ICAI).

[12] Google. 2019. Google Data Studio. https://datastudio.google.com[13] Christopher Groskopf and contributors. 2016. csvkit. https://csvkit.

readthedocs.org[14] Kevin Hu, Neil Gaikwad, Michiel Bakker, Madelon Hulsebos, Emanuel Zgraggen,

César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, and ÇağatayDemiralp. 2019. VizNet: Towards a large-scale visualization learning and bench-marking repository. In Proceedings of the 2019 Conference on Human Factors inComputing Systems (CHI). ACM.

[15] Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wran-gler: Interactive Visual Specification of Data Transformation Scripts. In ACMHuman Factors in Computing Systems (CHI).

[16] Tom Kenter and Maarten De Rijke. 2015. Short text similarity with word embed-dings. In Proceedings of the 24th ACM international on conference on informationand knowledge management. ACM, 1411–1420.

[17] Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studieswith Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factorsin Computing Systems (CHI ’08). ACM, New York, NY, USA, 453–456.

[18] Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences anddocuments. In International Conference on Machine Learning. 1188–1196.

[19] Girija Limaye, Sunita Sarawagi, and Soumen Chakrabarti. 2010. Annotating andsearching web tables using entities, types and relationships. Proceedings of theVLDB Endowment 3, 1-2 (2010), 1338–1347.

[20] Microsoft. 2019. Power BI | Interactive Data Visualization BI. https://powerbi.microsoft.com

[21] Fabian Pedregosa et al. 2011. Scikit-learn: Machine Learning in Python. Journalof Machine Learning Research 12 (2011), 2825–2830.

[22] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove:Global vectors for word representation. In Proceedings of the 2014 conference onempirical methods in natural language processing (EMNLP). 1532–1543.

8

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

Sherlock: A Deep Learning Approach to Semantic Data Type Detection KDD’19, August 2019, Anchorage, Alaska USA

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

[23] Minh Pham, Suresh Alse, Craig A Knoblock, and Pedro Szekely. 2016. Semanticlabeling: a domain-independent approach. In International Semantic Web Confer-ence. Springer, 446–462.

[24] Nikhil Waman Puranik. 2012. A Specialist Approach for Classification of ColumnData. Master’s thesis. University of Maryland, Baltimore County.

[25] Erhard Rahm and Philip A. Bernstein. 2001. A Survey of Approaches to AutomaticSchema Matching. The VLDB Journal 10, 4 (Dec. 2001), 334–350.

[26] Vijayshankar Raman and Joseph M. Hellerstein. 2001. Potter’s Wheel: An Inter-active Data Cleaning System. In Proceedings of the 27th International Conferenceon Very Large Data Bases (VLDB ’01). Morgan Kaufmann Publishers Inc., SanFrancisco, CA, USA, 381–390.

[27] S Krishnamurthy Ramnandan, Amol Mittal, Craig A Knoblock, and Pedro Szekely.2015. Assigning semantic labels to data sources. In European Semantic WebConference. Springer, 403–417.

[28] Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modellingwith Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challengesfor NLP Frameworks. ELRA, 45–50.

[29] Dominique Ritze and Christian Bizer. 2017. Matching web tables to DBpedia – afeature utility study. context 42, 41 (2017), 19.

[30] Zareen Syed, Tim Finin, Varish Mulwad, Anupam Joshi, et al. 2010. Exploitinga web of semantic data for interpreting tables. In Proceedings of the Second WebScience Conference.

[31] Trifacta. 2019. Data Wrangling Tools & Software. https://www.trifacta.com[32] Princeton University. 2010. About WordNet. https://wordnet.princeton.

edu[33] Petros Venetis, Alon Halevy, Jayant Madhavan, Marius Paşca, Warren Shen, Fei

Wu, Gengxin Miao, and Chung Wu. 2011. Recovering semantics of tables on theweb. Proceedings of the VLDB Endowment 4, 9 (2011), 528–538.

[34] Cong Yan and Yeye He. 2018. Synthesizing type-detection logic for rich seman-tic data types using open-source code. In Proceedings of the 2018 InternationalConference on Management of Data. ACM, 35–50.

[35] Benjamin Zapilko, Matthäus Zloch, and Johann Schaible. 2012. Utilizing RegularExpressions for Instance-Based Schema Matching. CEUR Workshop Proceedings946.

A APPENDIX

A.1 Supplemental Tables

Table 7: 78 semantic types included in this study.

Semantic TypesAddress Code Education Notes RequirementAffiliate Collection Elevation Operator ResultAffiliation Command Family Order SalesAge Company File size Organisation ServiceAlbum Component Format Origin SexArea Continent Gender Owner SpeciesArtist Country Genre Person StateBirth date County Grades Plays StatusBirth place Creator Industry Position SymbolBrand Credit ISBN Product TeamCapacity Currency Jockey Publisher Team nameCategory Day Language Range TypeCity Depth Location Rank WeightClass Description Manufacturer Ranking YearClassification Director Name RegionClub Duration Nationality Religion

Table 8: Description of the 27 global statistical features. As-terisks (*) denote features included in Venetis et al. [33].

Feature descriptionNumber of values.Column entropy.Fraction of values with unique content.*Fraction of values with numerical characters.*Fraction of values with alphabetical characters.Mean and std. of the number of numerical characters in values.*Mean and std. of the number of alphabetical characters in values.*Mean and std. of the number special characters in values.*Mean and std. of the number of words in values.*{Percentage, count, only/has-Boolean} of the None values.{Stats, sum, min, max, median, mode, kurtosis, skewness,any/all-Boolean} of length of values.

A.2 Mechanical Turk DemographicsOf the 390 participants, 57.18% were male and 0.43% female. 1.5%completed some high school without attaining a diploma, whileothers had associates (10.5%), bachelor’s (61.0%), master’s (13.1%), ordoctorate or professional degree (1.8%) in addition to a high schooldiploma (12.3%). 26.4% of participants worked with data daily, 33.1%weekly, 17.2% monthly, and 11.0% annually, while 12.3% never workwith data. In terms of age: 10.0% of participants were between 18-23,24-34 (60.3%), 35-40 (13.3%), 41-54 (12.6%), and above 55 (3.8%).

9