distilling the web of data drop by drop (with java)

distilling the Web of Data drop by drop (with Java)

Sourcesense UK “Last Wednesday” - Davide Palmisano @dpalmisano

distilling the Web of Data drop by drop (with Java)

Wednesday, June 29, 2011

the shortest introduction ever to the Web o f Data

Web pages markup technologies are intended for human consumption

extracting valuable data may require fancy scraping techniques

scraping: one size doesn’t fit all

they let machines to present raw data to humans



<div> <div> Canon Rebel T2i (EOS 550D) $899 </div> <div> The Rebel T2i EOS 550D is Cannon's top-of-the-line consumer digital SLR camera. It can shoot up

<div> AN_UCC-13: 013803123784 </div> <div> price: 899 USD </div>

</div></div>




</div></div>


what does this tag mean?




</div></div>


what does this tag mean?is this a

currency or what?


“meaning”, Joseph Kosuth, The Panza Collection, MART - Rovereto, Italy Wednesday, June 29, 2011

Microformats

“Microformats are a way of adding simple markup to human-readable data items such as events, contact details or locations, on web pages”

Andy Mabbett

- community driven initiative- largely adopted- quick & dirty - scarcely extensibility


Microformats

<div class=”hlisting item”> <div> Canon Rebel T2i (EOS 550D) $899< /div> <div class=”description”> The Rebel T2i EOS 550D is Cannon's top-of-the-line consumer digital SLR camera. It can shoot up

<div> AN_UCC-13: 013803123784 </div> <div class=”price”> price: 899 USD </div>

</div></div>


RDFa: RDF in attribute

model your data as they were Web pages connected with named links and properties and embed them in your (X)HTML using @attributes

- RDF, graph-based model- W3C Recommandation- highly extensible

i.e GoodRelations[1], a fully flavored vocabulary for the e-commerce



http://mystore.com/product/5642

http://canon.co.uk

ex:price

ex:description

ex:currency

899

USD

The Rebel T2i EOS 550D blah blah

ex:producer

ex:value

model your data















<div about=”http://mystore.com/product/5642”> <div>Canon Rebel T2i (EOS 550D) $899</div> <div property=”gr:description”>The Rebel T2i EOS 550D is Cannon's blah blah</div>

<div rel=”gr:hasPriceSpecification”> <span> price: <span property=”gr:hasCurrencyValue”>899</span> <span property=”gr:hasCurrency”>USD</span> </span> </div></div>

and then embed them in your (X)HTML pages




HTML5: Microdata

Microdata allows nested groups of name-value pairs to be added to HTML documents, in parallel with the existing content

- W3C Working draft- native of HTML5 specification- serializable in RDF

- Google, Yahoo! and Bing endorsed Schema.org- large adoption expected


HTML5: Microdata

<div itemscop itemtype=”http://schema.org/Offer”> <div itemprop=”name”> Canon Rebel T2i (EOS 550D) $899 </div> <div itemprop=”description”> The Rebel T2i EOS 550D is Cannon's blah blah</div>

<div> <span> price: <span itemprop=”price”> 899 </span> <span itemprop=”priceCurrency”> USD </span> </span> </div></div>




% of marked up Web pages

0

0.5

1

1.5

2

2.5

3

3.5

RDFahCard

adrxfn

hReview09/200803/200910/2010

data from Yahoo! [2]


tie ‘em all together

uniform, reconciled and unified RDF representation


a drop-by-drop distiller

Anything To Triples (any23) is an open source, Apache-licensed:

- Java library,- Web service and- a command-line tool

able to distill RDF triples from a variety of semantically marked up Web documents

http://developers.any23.org




live demo http://any23.org

Web site with ~5000 products description with GoodRelations using RDFa


http://any23.org

http://any23.org

use Any23 in your Java programs

Any23 runner = new Any23();runner.setHTTPUserAgent("test-user-agent");HTTPClient httpClient = runner.getHTTPClient();DocumentSource source = new HTTPDocumentSource( httpClient, "http://test.com/index.html" );ByteArrayOutputStream out = new ByteArrayOutputStream();TripleHandler handler = new NTriplesWriter(out);runner.extract(source, handler);String n3 = out.toString("UTF-8");


http://www.rentalinrome.com/semanticloft/semanticloft.htm

http://www.rentalinrome.com/semanticloft/semanticloft.htm

Any23: Command-Line toolany23-core/bin$ ./any23

usage: any23 [-e <arg>] [-f <arg>] [-l <arg>] [-n] [-o <arg>] [-p] [-s] [-t] [-v] {<url>|<file>} -e <arg> comma-separated list of extractors, e.g. rdf-xml,rdf-turtle -f,--format <arg> Output format [turtle (default), ntriples, rdfxml, quad, uris] -l,--log <arg> logging, please specify a file -n,--nesting disable production of nesting triples -o,--output <arg> ouput file (defaults to stdout) -p,--pedantic validates and fixes HTML content detecting commons issues -s,--stats print out statistics of Any23


Any23: Web Serviceblacky:~ davide$ curl http://any23.org/any23/?format=nquads&url=http://www.bbc.co.uk/programmes/b00kygwh&report=on

<response> <extractors> <extractor>rdf-xml</extractor> </extractors> <report> ... <validationReport> <ruleActivations></ruleActivations> ... </validationReport> </report> <data> <rdf:Description rdf:about="http://www.bbc.co.uk/programmes/b00kygwh#programme"> <rdf:type rdf:resource="http://purl.org/ontology/po/Episode"/> <po:pid>b00kygwh</po:pid> <dc:title>The Terminator</dc:title> </rdf:Description> </data></response>


http://any23.org/any23/?format=nquads&url=http://www.bbc.co.uk/programmes/b00kygwh&report=on




http://www.bbc.co.uk/programmes/b00kygwh#programme




http://purl.org/ontology/po/Episode

http://purl.org/ontology/po/Episode

RDFaExtractor

Microformat Extractors

hReview hCalendar hCard

mimetype detection

hListing

Apache Tika

MicrodataExtractor

DOM extraction

Validator

Cyber Neko HTML

Rule Fix

ExtractionResult

RDF/XML Writer

JSON Writer

NQuadsWriter

Sesame


extractorpublic interface Extractor<Input> {

/** * Executes the extractor. Will be invoked only once, extractors are * not reusable. * * @param in The extractor's input * @param documentURI The document's URI * @param out Sink for extracted data * @throws IOException On error while reading from the input stream * @throws ExtractionException On other error, such as parse errors */ void run(Input in, URI documentURI, ExtractionResult out) throws IOException, ExtractionException;

/** * Returns a {@link org.deri.any23.extractor.ExtractorDescription} of * this extractor. */ ExtractorDescription getDescription();

}


validate and fixpublic interface Rule {

String getHRName();

boolean applyOn( DOMDocument document, RuleContext context, ValidationReportBuilder validationReportBuilder );}

public interface Fix { String getHRName(); void execute(Rule rule, RuleContext context, DOMDocument document); }

void addRule(Class<? extends Rule> rule, Class<? extends Fix> fix);


plugins

@PluginImplementation @Author(name="Michele Mostarda ([email protected])") public class HTMLScraperPlugin implements ExtractorPlugin {

private static final Logger logger = LoggerFactory.getLogger(HTMLScraperPlugin.class);

@Init public void init() { logger.info("Plugin initialization."); }

@Shutdown public void shutdown() { logger.info("Plugin shutdown."); }

public ExtractorFactory getExtractorFactory() { return HTMLScraperExtractor.factory; }

}


mailto:[email protected]

mailto:[email protected]

roadmap

incoming 0.6.0 release- support for Microdata- support for CSV- support for RDFa 1.1 prefix mechanism- improved app configuration- bug fixing

Apache (pre) Incubation process- http://wiki.apache.org/incubator/Any23Proposal- supporters and mentors (thanks guys!)Simone Tripodi (@stripodi)Tommaso Teofili (@tteofili)

- we’re looking for mentors


http://wiki.apache.org/incubator/Any23Proposal

http://wiki.apache.org/incubator/Any23Proposal

closing credits

active committers

Giovanni Tummarello ( @jccq )Michele Mostarda ( @micmos )

Davide Palmisano ( @dpalmisano )Richard Cyganiak ( @cygri )

thanks to the whole Semantic Web community, especially those who tirelessly challenge us

with bugs and features requests


References

[1] http://purl.org/goodrelations/v1

[2] http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/


http://purl.org/goodrelations/v1

http://purl.org/goodrelations/v1

http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/




distilling the web of data drop by drop (with java)

Art & Photos

microdata canon rebel

xhtml pagescanon rebel

web of data drop

data http

rdf triples

any23 e

cannons blah blah price

web serviceblacky