© 2013 101companies and Ralf Lämmel

Metadata inferenceCourse "Empirical Software Engineering"

University of Koblenz-Landau

Department of Computer Science

Ralf Lämmel

Software Languages Team

1

© 2013 101companies and Ralf Lämmel

Overall research context

2

© 2013 101companies and Ralf Lämmel 3

Objective

https://commons.wikimedia.org/wiki/File:Prism-rainbow.svg

Your arbitrary program

Languages

Technologies

Concepts

Features

© 2013 101companies and Ralf Lämmel

Raw research questions

How to automatically detect

used software languages,

used software technologies,

relevant software concepts,

implemented system features?

4

... perhaps specifically in the context of the 101companies project

© 2013 101companies and Ralf Lämmel 5

Too many languages.

Too many technologies.

Too many concepts.

Too little time.

EMF

SQL

TENEO

Java

XSD

DOM

Antlr

OWL

UML

XMI

Ecore

SQL DDL

XLSTSaxon

Hibernate

Awk

Json

Yacc

JAXP

RestOWL

RDF

ATOM

SparQLXSLT

DTD

BNF

XSD

OCL

Prolog

grep

MOF

OMG

QVT

jDOMRose

Protegé

XQuery

ODM

XMLSpy

JPA

JAXB

JDBC

ODBC

MySQLArgoUML

Jean

Jena

Jena

Dragan

TXL

VLDB

EMF.gen

ORACLE

TCS

XText

Teneo

Jersey

GWT

Sesame

Stratego

XPATH

JeanBeans

UTF8

ASCII

RDFa

RDF(S)

RDFS

CFG

LALR

ER

SLE2010xerces

xalan

saxonsax

sed

XSD

JMI JMF

SBVR

What’s the problem?

© 2013 101companies and Ralf Lämmel

Research challengesNew 101contributions with unforeseen technologies and concepts and new ways of implementing features.

Different platforms and repositories: Apache, .NET Framework, gem, Maven repository, hackageDB, ...

Lack of an established ontology: sets of tags or categories used by the different platforms and repositories.

Overall: scalability, generality6

© 2013 101companies and Ralf Lämmel 7

Let’s focus on software chrestomathies.

© 2013 101companies and Ralf Lämmel

Software chrestomathies

8

© 2013 101companies and Ralf Lämmel

What’s a software chrestomathy?

• Quick definition

‣ A collection of software systems (‘programs’)✦ designed as an aid in learning a CS subject ✦ exercising diverse languages & technologies

• Examples (more or less)

‣ 99 Bottles of Beer

‣ Rosetta Code

‣ HollingBerries

‣ 101companies

© 2013 101companies and Ralf Lämmel

Company X:Swing + JDBC

Company Y:SWT + Hibernate

Company Z:GWT + MongoDB

...

The 101companies chrestomathyA collection of human-resources management systems:

code + documentation

• Total salaries

• Increase salaries

• Cut salaries

• Edit employee data

• Import / export company data

© 2013 101companies and Ralf Lämmel

“101” covers many languages• Java

• C#

• VB.NET

• Python

• XML

• GIF

• Haskell

• XHTML

• .properties

• .ini

• HTML

• JSON

• Ecore

• PHP

• PHP4

• PHP5

• SQL

• .sh

• PNG

• ATL

• XSD

• Scala

• Text

• Markdown

• Prolog

• F#

• Perl

• CSS

• XMI

• AspectJ

• Smalltalk

• JAR

• Erlang

• WSDL

• Rascal

• Javascript

• 101meta

• Ruby

• C++

• Cobol

• (make)

• (Ant)

• ...

© 2013 101companies and Ralf Lämmel

• make

• Ant

• Maven

• sbt

• JAXB

• xjc

• EF

• xsd.exe

• XmlSerializer

• ANTLR

• javac

• Eclipse

• Hibernate

• Java collections

• VS

• java.util

• java.io

• java.lang.reflect

• LINQ

• System.Xml

• System.Xml.Linq

• ADO

• SOA

• JPA

• EMFCompare

• 101explorer

• Mediawiki

• hackage

• cabal

• ghci

• ghc

• Swing

• AWT

• Java RMI

• JDBC

• zip

• jEdit

• jdom

• w3c.dom

• dom4j

• xom

• SAX

• ...

“101” covers many technologies

© 2013 101companies and Ralf Lämmel

One particular implementation

© 2013 101companies and Ralf Lämmel

Part of the documentation

© 2013 101companies and Ralf Lämmel

Traceabilitybetween code and documentation

15

© 2013 101companies and Ralf Lämmel

The traceability challengefor a general software product

© 2013 101companies and Ralf Lämmel

The traceability challengefor a software chrestomathy

© 2013 101companies and Ralf Lämmel

The traceability challenge

© 2013 101companies and Ralf Lämmel

Metamodels at hand101repo vs. 101wiki

© 2013 101companies and Ralf Lämmel

© 2013 101companies and Ralf Lämmel

© 2013 101companies and Ralf Lämmel

The 101meta language for rule-based metadata specification

• Check condition on files

• Associate metadata with files or fragments

‣ Languages

‣ Technologies

‣ Concepts

‣ Tools for validation, fragment selection, ...

“Metadata inference”

© 2013 101companies and Ralf Lämmel

Links established by 101meta rules

© 2013 101companies and Ralf Lämmel

© 2013 101companies and Ralf Lämmel

© 2013 101companies and Ralf Lämmel

Links established by 101meta rules

© 2013 101companies and Ralf Lämmel

101meta rules for ANTLR

27

© 2013 101companies and Ralf Lämmel

Wanted!101meta rules for ANTLR

When is a file ...

• ... the ANTLR library?

• ... an ANTLR grammar?

• ... an ANTLR-generated parser?

• ... a source that imports ANTLR?

(See more rules online.)

© 2013 101companies and Ralf Lämmel

What’s ANTLR?company :  'company' STRING '{' department* '}' EOF;  department :  'department' STRING '{'     ('manager' employee)    ('employee' employee)*     department*  '}';

employee :  STRING '{'    'address' STRING    'salary' FLOAT  '}';

ANTLR style grammar for companies

Grammars were hot in the 70ies?You are so wrong!

http://www.antlr.org

© 2013 101companies and Ralf Lämmel

[−] ANTLR theme ANTLR-centric grammarware theme

yapg A ANTLR-based generator for text-to-object mappings

antlrAcceptor An ANTLR-based acceptor for textual syntax

antlrLexer Lexer-based processing with ANTLR

antlrObjects Object/Text mapping for Java with ANTLR for parsing

antlrParser Processing textual syntax with semantic actions of ANTLR

antlrTrees Parsing text to trees and walk them with ANTLR

gra2mol Grammar to model transformation with Grammar2Model

xtext An XText- and Eclipse-based DSL editor

http://101companies.org/wiki/Theme:ANTLR

© 2013 101companies and Ralf Lämmel

{ "basename" : "#^antlr-(.*)\\.jar$#", "metadata" : { "partOf" : "ANTLR", "comment" : "The ANTLR library, Version $1" } }

“Condition” of the rule

Assigned metadata

Expressed relationship

A 101meta rule to find the ANTLR library

© 2013 101companies and Ralf Lämmel

{ "suffix" : ".g", "metadata" : { "inputOf" : "ANTLR", "comment" : "An ANTLR grammar" } }

A 101meta rule to find an ANTLR grammar

“Condition” of the rule

Assigned metadata

Expressed relationship

© 2013 101companies and Ralf Lämmel

{ "basename" : "#^.*Parser\\.java$#", "content" : "// \\$ANTLR.*\\.g", "metadata" : [ { "outputOf" : "ANTLR", "comment" : "An ANTLR-generated parser" }, { "concept" : "Parser" } ] }

Condition on text of file

Metadata for software concept

A 101meta rule to find a generated parser

© 2013 101companies and Ralf Lämmel

{ "suffix" : ".java", "predicate" :

"technologies/Java_platform/javaImport.sh", "args" : [ "org.antlr.runtime" ], "metadata" : { "dependsOn" : "ANTLR", "comment" : "A source that imports ANTLR" } }

Programmatic condition

Java package to be tested for

A rule to find a source importing ANTLR

© 2013 101companies and Ralf Lämmel

Metadata management

35

© 2013 101companies and Ralf Lämmel

Metadata management

• Metadata declaration

with rules, as demonstrated

• Metadata assignment

with a rule engine

• Metadata exploration

with a browser (as seen in second)

© 2013 101companies and Ralf Lämmel

Metadata assignment101worker continuously walks

over 101repo to construct data

for files matches 101meta

rules. In fact, each file is

associated with such matches.

http://data.101companies.org/resources/contributions/antlrObjects/org/softlang/parser/CompanyParser.java.summary.json

© 2013 101companies and Ralf Lämmel

Metadata exploration

Disclaimer: This is not even beta.

© 2013 101companies and Ralf Lämmel

© 2013 101companies and Ralf Lämmel

© 2013 101companies and Ralf Lämmel

Metadata exploration

Disclaimer: This is not even beta.

© 2013 101companies and Ralf Lämmel

Platforms

42

© 2013 101companies and Ralf Lämmel

HackageDBhttp://hackage.haskell.org

© 2013 101companies and Ralf Lämmel

HackageDBhttp://hackage.haskell.org

© 2013 101companies and Ralf Lämmel

The Apache Software Foundationhttp://projects.apache.org/

© 2013 101companies and Ralf Lämmel

What’s that?

© 2013 101companies and Ralf Lämmel

PyPi - the Python Package Indexhttps://pypi.python.org/

© 2013 101companies and Ralf Lämmel

Maven Central Repositoryhttp://search.maven.org/

© 2013 101companies and Ralf Lämmel

What’s that?

© 2013 101companies and Ralf Lämmel

http://rubygems.org/

© 2013 101companies and Ralf Lämmel

Microsoft Developer Networkhttp://msdn.microsoft.com/

© 2013 101companies and Ralf Lämmel

Thanks!Questions?

52