europython 2008: dataportability and python

116
DataPortability and Python

Upload: christian-scholz

Post on 23-Jan-2015

4.529 views

Category:

Technology


3 download

DESCRIPTION

This is the presentation I gave at EuroPython 2008. It introduces people to the concepts of Data Portability and shows some standards which can be of use in that context and explains how to use this in Python.Last but not least I introduce my project pydataportability.

TRANSCRIPT

Page 1: Europython 2008: DataPortability and Python

DataPortabilityand Python

Page 2: Europython 2008: DataPortability and Python

Christian Scholz

Page 3: Europython 2008: DataPortability and Python

http://mrtopf.de

Page 4: Europython 2008: DataPortability and Python

twitter.com/mrtopf

Page 6: Europython 2008: DataPortability and Python

Plone

Page 7: Europython 2008: DataPortability and Python

Python

Page 8: Europython 2008: DataPortability and Python

virtual worlds

Page 9: Europython 2008: DataPortability and Python

DataPortabilityhttp://dataportability.org

Page 10: Europython 2008: DataPortability and Python

Video by Michael Picksmashcut-media.com

Page 11: Europython 2008: DataPortability and Python

Policy

Page 12: Europython 2008: DataPortability and Python

Technology

Page 13: Europython 2008: DataPortability and Python

the ingredients

Page 14: Europython 2008: DataPortability and Python

http://openid.com

Page 15: Europython 2008: DataPortability and Python

Single-Sign-On

Page 16: Europython 2008: DataPortability and Python
Page 17: Europython 2008: DataPortability and Python
Page 18: Europython 2008: DataPortability and Python
Page 19: Europython 2008: DataPortability and Python

http://openidenabled.com

Page 20: Europython 2008: DataPortability and Python

YADIS

Page 21: Europython 2008: DataPortability and Python

?http://mrtopf.myopenid.com

Page 22: Europython 2008: DataPortability and Python

?http://mrtopf.myopenid.com

Page 23: Europython 2008: DataPortability and Python

?http://mrtopf.myopenid.com

X-XRDS-Location: http://mrtopf.myopenid.com/xrds

Page 24: Europython 2008: DataPortability and Python

<?xml version="1.0" encoding="UTF-8"?><xrds:XRDS xmlns:xrds="xri://$xrds" xmlns:openid="http://openid.net/xmlns/1.0" xmlns="xri://$xrd*($v*2.0)"> <XRD>

<Service priority="0"> <Type>http://specs.openid.net/auth/2.0/signon</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant</Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <LocalID>http://mrtopf.myopenid.com/</LocalID> </Service> <Service priority="1"> <Type>http://openid.net/signon/1.1</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant</Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <openid:Delegate>http://mrtopf.myopenid.com/</openid:Delegate> </Service> <Service priority="2"> <Type>http://openid.net/signon/1.0</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant</Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <openid:Delegate>http://mrtopf.myopenid.com/</openid:Delegate> </Service> </XRD></xrds:XRDS>

X-XRDS-Location: http://mrtopf.myopenid.com/xrds

Page 25: Europython 2008: DataPortability and Python

X-XRDS-Location: http://mrtopf.myopenid.com/xrds

<Service priority="0"> <Type>http://specs.openid.net/auth/2.0/signon</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant </Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <LocalID>http://mrtopf.myopenid.com/</LocalID></Service>

XRDSeXtensible Resource Descriptor Sequence

Page 26: Europython 2008: DataPortability and Python

http://oauth.net

Page 27: Europython 2008: DataPortability and Python

Authorization

Page 28: Europython 2008: DataPortability and Python

Service A Service B

Page 29: Europython 2008: DataPortability and Python

Service A Service B

Page 30: Europython 2008: DataPortability and Python

Service A Service B

Page 31: Europython 2008: DataPortability and Python

Service A Service B

Page 32: Europython 2008: DataPortability and Python

Service A Service B

Redirect Redirect

Page 33: Europython 2008: DataPortability and Python

Service A Service B

Redirect Redirect

Token

Page 35: Europython 2008: DataPortability and Python

OAuth Discoveryhttp://oauth.net/discovery/

XRDS-Simplehttp://xrds-simple.net

The discovery protocol provides a way for Consumers to receive

the required configuration data from Service Providers in a

machine-readable format. It creates an extendable framework

for communicating current and future services.

Page 36: Europython 2008: DataPortability and Python

http://foaf-project.org

Friend of a FriendFOAF

Page 37: Europython 2008: DataPortability and Python

foaf:Person

„Christian Scholz“

foaf:name

„male“

foaf:gender

Page 38: Europython 2008: DataPortability and Python

foaf:Person

„Christian Scholz“

foaf:name

„male“

foaf:gender

foaf:Person

„Mary“

foaf:name

Page 39: Europython 2008: DataPortability and Python

foaf:Person

„Christian Scholz“

foaf:name

„male“

foaf:gender

foaf:knows

foaf:Person

„Mary“

foaf:name

Page 40: Europython 2008: DataPortability and Python

<rdf:RDF

xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"

xmlns:foaf="http://xmlns.com/foaf/0.1/">

<foaf:Person rdf:ID="me"> <foaf:name>Christian Scholz</foaf:name>

<foaf:knows>

<foaf:Person> <foaf:name>Mary</foaf:name>

</foaf:Person>

</foaf:knows> </foaf:Person>

</rdf:RDF>

Page 41: Europython 2008: DataPortability and Python

RDFResource Description Framework

http://rdflib.net/

Page 42: Europython 2008: DataPortability and Python

http://microformats.org

Page 43: Europython 2008: DataPortability and Python

Example: XHTML Friends Network (XFN)

Christian Marymet,friend

http://mrtopf.de http://mary.com

Page 44: Europython 2008: DataPortability and Python

Example: XHTML Friends Network (XFN)

Christian Marymet,friend

http://mrtopf.de http://mary.com

<a href=“http://mary.com“>Mary</a>

Page 45: Europython 2008: DataPortability and Python

Example: XHTML Friends Network (XFN)

Christian Marymet,friend

http://mrtopf.de http://mary.com

<a href=“http://mary.com“ rel=“met, friend“>Mary</a>

Page 46: Europython 2008: DataPortability and Python

Example: XHTML Friends Network (XFN)

Christian Christianme

http://mrtopf.de http://seesmic.com/mrtopf

Page 47: Europython 2008: DataPortability and Python

Example: XHTML Friends Network (XFN)

Christian Christianme

http://mrtopf.de http://seesmic.com/mrtopf

<a href=“http://seesmic.com/mrtopf“ rel=“me“>Video</a>

Page 48: Europython 2008: DataPortability and Python

<DIV class="intro vcard" style=""> <A class="url uid" href="/mrtopf/"> <IMG class="user photo" src="2278_large.jpg"/> </A> <H1 class="fn nickname">Christian S.</H1> <P class="description adr">36-year-old gentleman from

<SPAN class="locality">Aachen, Germany</SPAN><SPAN class="country-name">DE</SPAN>

</P> <P class="description note">

Tao Takashi in Second Lifemrtopf on the internet, IRC, TwitterPython/Zope/Plone programmer

</P></DIV>

Example: hCard (VCard)

Page 49: Europython 2008: DataPortability and Python

Example: hCard (VCard)

<DIV class="intro vcard" style=""> <A class="url uid" href="/mrtopf/"> <IMG class="user photo" src="2278_large.jpg"/> </A> <H1 class="fn nickname">Christian S.</H1> <P class="description adr">36-year-old gentleman from

<SPAN class="locality">Aachen, Germany</SPAN><SPAN class="country-name">DE</SPAN>

</P> <P class="description note">

Tao Takashi in Second Lifemrtopf on the internet, IRC, TwitterPython/Zope/Plone programmer

</P></DIV>

Page 50: Europython 2008: DataPortability and Python

in Python

Almost Universal Microformats Parserhttp://code.google.com/p/aump/

pydataportabilityhttp://pydataportability.net/

Page 51: Europython 2008: DataPortability and Python

whereare my

services

?

Page 52: Europython 2008: DataPortability and Python

Service Catalogue

Page 53: Europython 2008: DataPortability and Python

Service Catalogue

http://xrds-simple.net

Page 54: Europython 2008: DataPortability and Python

<XRDS xmlns="xri://$xrds"> <XRD xmlns:simple="http://xrds-simple.net/core/1.0" xmlns="xri://$XRD*($v*2.0)" version="2.0">

<Type>xri://$xrds*simple</Type>

<Service priority="10"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET">http://books.example.com/wishlist</URI> <LocalID>jane</LocalID> </Service>

<Service priority="20"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET" priority="10">https://dvds.example.org/lists/wishes</URI> <URI simple:httpMethod="GET" priority="20">http://dvds.example.org/lists/wishes</URI> <LocalID>janedoe</LocalID> </Service>

</XRD></XRDS>

Page 55: Europython 2008: DataPortability and Python

<Service priority="10">

<Type>http://specs.example.com/wish_list/1.0</Type>

<URI simple:httpMethod="GET">

http://books.example.com/wishlist</URI>

<LocalID>jane</LocalID>

</Service>

Page 56: Europython 2008: DataPortability and Python

Weaving itall together!

Page 57: Europython 2008: DataPortability and Python

Weaving itall together!

Page 58: Europython 2008: DataPortability and Python

goals

Page 59: Europython 2008: DataPortability and Python

goals

Access

Page 60: Europython 2008: DataPortability and Python

goals

Access

Discovery

Page 61: Europython 2008: DataPortability and Python

goals

Access

Discovery

Conversion

Page 62: Europython 2008: DataPortability and Python
Page 63: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces

Page 64: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

Page 65: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()

Page 66: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")

Page 67: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

Page 68: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():

Page 69: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():print "Parser: %s" %name

Page 70: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():print "Parser: %s" %name

for r in result:

Page 71: Europython 2008: DataPortability and Python

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():print "Parser: %s" %name

for r in result: print r

Page 72: Europython 2008: DataPortability and Python

Parser: xfnXFN Relationships for http://mrtopf.de/connect: meXFN Relationships for http://twitter.com/jack: contactXFN Relationships for http://twitter.com/biz: contact...

Parser: hcardVCard for Christian Scholz (fn=Christian Scholz)VCard for None (fn=None, url=http://twitter.com/jack) VCard for None (fn=None, url=http://twitter.com/biz) ...

Page 73: Europython 2008: DataPortability and Python

<XRDS xmlns="xri://$xrds"> <XRD xmlns:simple="http://xrds-simple.net/core/1.0" xmlns="xri://$XRD*($v*2.0)" version="2.0">

<Type>xri://$xrds*simple</Type>

<Service priority="10"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET">http://books.example.com/wishlist</URI> <LocalID>jane</LocalID> </Service>

<Service priority="20"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET" priority="10">https://dvds.example.org/lists/wishes</URI> <URI simple:httpMethod="GET" priority="20">http://dvds.example.org/lists/wishes</URI> <LocalID>janedoe</LocalID> </Service>

</XRD></XRDS>

Page 74: Europython 2008: DataPortability and Python

<XRDS xmlns="xri://$xrds"> <XRD xmlns:simple="http://xrds-simple.net/core/1.0" xmlns="xri://$XRD*($v*2.0)" version="2.0">

<Type>xri://$xrds*simple</Type>

<Service priority="10"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET">http://books.example.com/wishlist</URI> <LocalID>jane</LocalID> </Service>

<Service priority="20"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET" priority="10">https://dvds.example.org/lists/wishes</URI> <URI simple:httpMethod="GET" priority="20">http://dvds.example.org/lists/wishes</URI> <LocalID>janedoe</LocalID> </Service>

</XRD></XRDS>

Page 75: Europython 2008: DataPortability and Python
Page 76: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

Page 77: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)

Page 78: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

Page 79: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:

Page 80: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.type

Page 81: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priority

Page 82: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priorityprint "LocalID:",s.localid

Page 83: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priorityprint "LocalID:",s.localid for uri in s.uris:

Page 84: Europython 2008: DataPortability and Python

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priorityprint "LocalID:",s.localid for uri in s.uris: print " ",uri.uri

Page 85: Europython 2008: DataPortability and Python

Type: http://specs.example.com/wish_list/1.0Prio: 10LocalID: jane http://books.example.com/wishlist

Type: http://specs.example.com/wish_list/1.0Prio: 20LocalID: janedoe https://dvds.example.org/lists/wishes http://dvds.example.org/lists/wishes

Page 86: Europython 2008: DataPortability and Python

XRDS

HTML

Page 87: Europython 2008: DataPortability and Python

XRDS

hCard

XFN

HTML

Page 88: Europython 2008: DataPortability and Python

XRDS

hCard

XFN

XRDS

FOAF

HTML

Page 89: Europython 2008: DataPortability and Python

XRDS

hCard

XFN

XRDS

FOAF

meta/header

HTML

Page 90: Europython 2008: DataPortability and Python
Page 91: Europython 2008: DataPortability and Python

from pydataportability.discovery import discover

Page 92: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

Page 93: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)

Page 94: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

Page 95: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services

Page 96: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

Page 97: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type

Page 98: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

Page 99: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11']

Page 100: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

Page 101: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload

Page 102: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

Page 103: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

>>> services[2].loaded

Page 104: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

>>> services[2].loadedFalse

Page 105: Europython 2008: DataPortability and Python

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

>>> services[2].loadedFalse

>>> services[2].load()

Page 106: Europython 2008: DataPortability and Python
Page 111: Europython 2008: DataPortability and Python

status?

Page 112: Europython 2008: DataPortability and Python

status?

supports hCard/XFN

Page 113: Europython 2008: DataPortability and Python

status?

supports hCard/XFN

supports XRDS

Page 114: Europython 2008: DataPortability and Python

status?

supports hCard/XFN

supports XRDS

supports service discovery