debsources - dive into debian source code! - matthieu caneill · debsources dive into debian source...
TRANSCRIPT
![Page 1: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/1.jpg)
DebsourcesDive into Debian source code!
Matthieu Caneill
Debian contributor, PhD student at LIG
April 11, 2015MiniDebconf (Lyon, France)
![Page 2: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/2.jpg)
Acknowledgements
Debsources
Initially developped at IRILL, by Stefano Zacchiroli and myself. Manypersons have contributed patches since then.
Infrastructure
Debsources’ servers are sponsored by IRILL.
Slides
Inspired by Stefano Zacchiroli’s previous presentations.Licensed CC-BY-SA.
Matthieu Caneill Debsources MiniDebconf, April 2015 2 / 48
![Page 3: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/3.jpg)
Acknowledgements
Debsources
Initially developped at IRILL, by Stefano Zacchiroli and myself. Manypersons have contributed patches since then.
Infrastructure
Debsources’ servers are sponsored by IRILL.
Slides
Inspired by Stefano Zacchiroli’s previous presentations.Licensed CC-BY-SA.
Matthieu Caneill Debsources MiniDebconf, April 2015 2 / 48
![Page 4: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/4.jpg)
Acknowledgements
Debsources
Initially developped at IRILL, by Stefano Zacchiroli and myself. Manypersons have contributed patches since then.
Infrastructure
Debsources’ servers are sponsored by IRILL.
Slides
Inspired by Stefano Zacchiroli’s previous presentations.Licensed CC-BY-SA.
Matthieu Caneill Debsources MiniDebconf, April 2015 2 / 48
![Page 5: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/5.jpg)
Table of contents
1 Introduction
2 FeaturesDebsources’ featuresWhat’s new?Roadmap
3 Technologies
4 Research platform
5 Hacking
Matthieu Caneill Debsources MiniDebconf, April 2015 3 / 48
![Page 6: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/6.jpg)
Table of contents
1 Introduction
2 FeaturesDebsources’ featuresWhat’s new?Roadmap
3 Technologies
4 Research platform
5 Hacking
Matthieu Caneill Debsources MiniDebconf, April 2015 4 / 48
![Page 7: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/7.jpg)
What is Debsources?
A web application to browse the source code of Debian packages
The infrastructure behind: database, plugins, ...
Play with it!
Navigate to http://sources.debian.net
Matthieu Caneill Debsources MiniDebconf, April 2015 5 / 48
![Page 8: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/8.jpg)
What is Debsources?
A web application to browse the source code of Debian packages
The infrastructure behind: database, plugins, ...
Play with it!
Navigate to http://sources.debian.net
Matthieu Caneill Debsources MiniDebconf, April 2015 5 / 48
![Page 9: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/9.jpg)
Home page
Matthieu Caneill Debsources MiniDebconf, April 2015 6 / 48
![Page 10: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/10.jpg)
Source code display
Matthieu Caneill Debsources MiniDebconf, April 2015 7 / 48
![Page 11: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/11.jpg)
So what?Is this really useful?
“I want to check the source code of cowsay. What do?”
The old way
cd /tmp
apt-get source cowsay
cd cowsay-3.03+dfsg1
...
cd ..
rm -r cowsay-3.03+dfsg1
Note that it only works with Debian-based systems, and not with your iPhoneTM.
The new way
lynx http://sources.debian.net/src/cowsay/
Almost runs on your typewriter.
Matthieu Caneill Debsources MiniDebconf, April 2015 8 / 48
![Page 12: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/12.jpg)
So what?Is this really useful?
“I want to check the source code of cowsay. What do?”
The old way
cd /tmp
apt-get source cowsay
cd cowsay-3.03+dfsg1
...
cd ..
rm -r cowsay-3.03+dfsg1
Note that it only works with Debian-based systems, and not with your iPhoneTM.
The new way
lynx http://sources.debian.net/src/cowsay/
Almost runs on your typewriter.
Matthieu Caneill Debsources MiniDebconf, April 2015 8 / 48
![Page 13: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/13.jpg)
Table of contents
1 Introduction
2 FeaturesDebsources’ featuresWhat’s new?Roadmap
3 Technologies
4 Research platform
5 Hacking
Matthieu Caneill Debsources MiniDebconf, April 2015 9 / 48
![Page 14: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/14.jpg)
Source code browsing
Syntax highlighting
For all languages supported by highlight.js: C, C++, Java, Python, Ruby,Makefile... and 112 others.
Included versions
Packages inhamm, sink, potato, woody, sarge, etch, lenny, squeeze,
wheezy, jessie (testing), unstable, experimental,
oldstable-updates, stable-updates, proposed-updates,
testing-proposed-updates, wheezy-backports, squeeze-lts
are in Debsources.
Matthieu Caneill Debsources MiniDebconf, April 2015 10 / 48
![Page 15: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/15.jpg)
Source code browsing
Syntax highlighting
For all languages supported by highlight.js: C, C++, Java, Python, Ruby,Makefile... and 112 others.
Included versions
Packages inhamm, sink, potato, woody, sarge, etch, lenny, squeeze,
wheezy, jessie (testing), unstable, experimental,
oldstable-updates, stable-updates, proposed-updates,
testing-proposed-updates, wheezy-backports, squeeze-lts
are in Debsources.
Matthieu Caneill Debsources MiniDebconf, April 2015 10 / 48
![Page 16: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/16.jpg)
Searching
You can search for:
Packages
Files
File contentI ctagsI regular expressions through codesearch.debian.net
Content indexing
Searching is fast, thanks to PostgreSQL’s indexes.
Matthieu Caneill Debsources MiniDebconf, April 2015 11 / 48
![Page 17: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/17.jpg)
Searching
You can search for:
Packages
Files
File contentI ctagsI regular expressions through codesearch.debian.net
Content indexing
Searching is fast, thanks to PostgreSQL’s indexes.
Matthieu Caneill Debsources MiniDebconf, April 2015 11 / 48
![Page 18: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/18.jpg)
Advanced search
Matthieu Caneill Debsources MiniDebconf, April 2015 12 / 48
![Page 19: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/19.jpg)
Annotations and highlighting
Suppose...
I’m a developer: I want to share a precise location in the source codeof package X.
I’m a user: I can’t compile software X, it fails at line 42 in the file Y.
I’m a static source code analyzer: I found a semantic error in file Y.
Matthieu Caneill Debsources MiniDebconf, April 2015 13 / 48
![Page 20: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/20.jpg)
Annotations and highlighting
http://sources.debian.net/src/
package/version/path/to/file.c?hl=a:b&msg=a:b:c#LXX
package: cowsayversion: 3.03-3path: cowsayhighlight: 32:36message: 30:Debian:rocks
Matthieu Caneill Debsources MiniDebconf, April 2015 14 / 48
![Page 21: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/21.jpg)
Annotations and highlighting
http://sources.debian.net/src/
package/version/path/to/file.c?hl=a:b&msg=a:b:c#LXX
package: cowsayversion: 3.03-3path: cowsayhighlight: 32:36message: 30:Debian:rocks
Matthieu Caneill Debsources MiniDebconf, April 2015 14 / 48
![Page 22: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/22.jpg)
Annotations and highlighting
http://sources.debian.net/src/
package/version/path/to/file.c?hl=a:b&msg=a:b:c#LXX
package: cowsayversion: 3.03-3path: cowsayhighlight: 32:36message: 30:Debian:rocks
Matthieu Caneill Debsources MiniDebconf, April 2015 14 / 48
![Page 23: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/23.jpg)
Annotations and highlightingDeveloper: I want to share a precise location in the source code of package X.
Matthieu Caneill Debsources MiniDebconf, April 2015 15 / 48
![Page 24: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/24.jpg)
Annotations and highlightingUser: I can’t compile software X, it fails at line 42 in the file Y.
Matthieu Caneill Debsources MiniDebconf, April 2015 16 / 48
![Page 25: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/25.jpg)
Annotations and highlightingStatic analyzer: I found a semantic error in file Y.
Matthieu Caneill Debsources MiniDebconf, April 2015 17 / 48
![Page 26: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/26.jpg)
Duplicated files
All the files are in the database, along with their checksum.The duplicates can be computed, for every file.
Matthieu Caneill Debsources MiniDebconf, April 2015 18 / 48
![Page 27: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/27.jpg)
Duplicated files
Matthieu Caneill Debsources MiniDebconf, April 2015 19 / 48
![Page 28: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/28.jpg)
Integration in the ecosystem
Codesearch
http://codesearch.debian.net is used for regular expression searches,and redirects its results back to Debsources.Credits: Michael Stapelberg
Package tracking systems
The old (http://packages.qa.debian.org) and new(http://tracker.debian.org) PTS provide links to Debsources(“browse source code”).Credits: Paul Wise
Need to embed code somewhere?
<iframe>s embedding of files content is supported (see documentation).
Matthieu Caneill Debsources MiniDebconf, April 2015 20 / 48
![Page 29: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/29.jpg)
Integration in the ecosystem
Codesearch
http://codesearch.debian.net is used for regular expression searches,and redirects its results back to Debsources.Credits: Michael Stapelberg
Package tracking systems
The old (http://packages.qa.debian.org) and new(http://tracker.debian.org) PTS provide links to Debsources(“browse source code”).Credits: Paul Wise
Need to embed code somewhere?
<iframe>s embedding of files content is supported (see documentation).
Matthieu Caneill Debsources MiniDebconf, April 2015 20 / 48
![Page 30: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/30.jpg)
Integration in the ecosystem
Codesearch
http://codesearch.debian.net is used for regular expression searches,and redirects its results back to Debsources.Credits: Michael Stapelberg
Package tracking systems
The old (http://packages.qa.debian.org) and new(http://tracker.debian.org) PTS provide links to Debsources(“browse source code”).Credits: Paul Wise
Need to embed code somewhere?
<iframe>s embedding of files content is supported (see documentation).
Matthieu Caneill Debsources MiniDebconf, April 2015 20 / 48
![Page 31: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/31.jpg)
Statistics
Code source metrics for every package.
Plugins: size, ctags, sloccount
Matthieu Caneill Debsources MiniDebconf, April 2015 21 / 48
![Page 32: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/32.jpg)
Statistics
Code source metrics for every package.
Plugins: size, ctags, sloccount
Matthieu Caneill Debsources MiniDebconf, April 2015 21 / 48
![Page 33: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/33.jpg)
Statistics
Aggregated statistics are available athttp://sources.debian.net/stats/.
Metrics
Disk usage
SLOC (source lines of code)
Number of source packages
Number of files
Number of ctags (symbols)
Matthieu Caneill Debsources MiniDebconf, April 2015 22 / 48
![Page 34: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/34.jpg)
Statistics
Currently in the unstable suite:
Disk usage ' 210 GBSLOC (source lines of code) 1,002,329,283Number of source packages 22,763Number of files 10,565,532Number of ctags (symbols) 114,732,392
Matthieu Caneill Debsources MiniDebconf, April 2015 23 / 48
![Page 35: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/35.jpg)
StatisticsFancy graphs
Matthieu Caneill Debsources MiniDebconf, April 2015 24 / 48
![Page 36: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/36.jpg)
API
Along the graphical web interface, an API provides the samefunctionalities.
Examples
curl http://sources.debian.net/api/ping/
{ ‘‘status’’: ‘‘ok’’,
‘‘http status code’’: 200,
‘‘last update’’: ‘‘Fri, 10 Apr 2015 10:16:31 -0000’’ }
curl http://s.d.n/api/info/package/cowsay/3.03-3/
{‘‘pkg infos’’: {
‘‘suites’’: [
‘‘woody’’
...
Documentation at http://sources.debian.net/doc/api/.
Matthieu Caneill Debsources MiniDebconf, April 2015 25 / 48
![Page 37: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/37.jpg)
API
Along the graphical web interface, an API provides the samefunctionalities.
Examples
curl http://sources.debian.net/api/ping/
{ ‘‘status’’: ‘‘ok’’,
‘‘http status code’’: 200,
‘‘last update’’: ‘‘Fri, 10 Apr 2015 10:16:31 -0000’’ }
curl http://s.d.n/api/info/package/cowsay/3.03-3/
{‘‘pkg infos’’: {
‘‘suites’’: [
‘‘woody’’
...
Documentation at http://sources.debian.net/doc/api/.
Matthieu Caneill Debsources MiniDebconf, April 2015 25 / 48
![Page 38: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/38.jpg)
What’s new?
Many new features since DebConf14
OPW student: Jingjie Jiang (sophiejjj)
Many new contributors:
I Stefano Zacchiroli: 688I Matthieu Caneill: 455I sophiejjj: 58I Orestis Ioannou: 10I Jason Pleau: 10I Akshita Jha: 7I Clement Schreiner: 6
I Jingjie Jiang: 5I Luciano Bello: 1I Paul Wise: 1I tessy joseph: 1I Christophe Siraut: 1I sodamatt: 1I Tapasweni Pathak: 1
Matthieu Caneill Debsources MiniDebconf, April 2015 26 / 48
![Page 39: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/39.jpg)
What’s new?
Many new features since DebConf14
OPW student: Jingjie Jiang (sophiejjj)
Many new contributors:
I Stefano Zacchiroli: 688I Matthieu Caneill: 455I sophiejjj: 58I Orestis Ioannou: 10I Jason Pleau: 10I Akshita Jha: 7I Clement Schreiner: 6
I Jingjie Jiang: 5I Luciano Bello: 1I Paul Wise: 1I tessy joseph: 1I Christophe Siraut: 1I sodamatt: 1I Tapasweni Pathak: 1
Matthieu Caneill Debsources MiniDebconf, April 2015 26 / 48
![Page 40: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/40.jpg)
What’s new?
Many new features since DebConf14
OPW student: Jingjie Jiang (sophiejjj)
Many new contributors:
I Stefano Zacchiroli: 688I Matthieu Caneill: 455I sophiejjj: 58I Orestis Ioannou: 10I Jason Pleau: 10I Akshita Jha: 7I Clement Schreiner: 6
I Jingjie Jiang: 5I Luciano Bello: 1I Paul Wise: 1I tessy joseph: 1I Christophe Siraut: 1I sodamatt: 1I Tapasweni Pathak: 1
Matthieu Caneill Debsources MiniDebconf, April 2015 26 / 48
![Page 41: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/41.jpg)
What’s new?Multiple pop-up messages
Credits: Orestis Ioannou and Jason Pleau.
Matthieu Caneill Debsources MiniDebconf, April 2015 27 / 48
![Page 42: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/42.jpg)
What’s new?Blueprints support
Credits: Jingjie Jiang
Blueprints
Flask apps embedded and plugged together
Implied a big refactoring
New features incoming
Matthieu Caneill Debsources MiniDebconf, April 2015 28 / 48
![Page 43: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/43.jpg)
What’s new?Detailed directory listing
Credits: Jingjie Jiang
Matthieu Caneill Debsources MiniDebconf, April 2015 29 / 48
![Page 44: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/44.jpg)
What’s new?File edition in-browser
Credits: Raphael Geissert
File edition
A plugin for Iceweasel and Chromium enables the edition of files in yourbrowser.A patch ready-to-be-sentTM is generated!
Matthieu Caneill Debsources MiniDebconf, April 2015 30 / 48
![Page 45: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/45.jpg)
What’s new?And many many other features...
Refactoring
I Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 46: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/46.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python module
I Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 47: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/47.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loader
I Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 48: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/48.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 49: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/49.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 50: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/50.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 51: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/51.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 52: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/52.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 53: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/53.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 54: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/54.jpg)
What’s new?And many many other features...
RefactoringI Debsources as a top-level Python moduleI Configuration loaderI Flake8 compliance (Zack, Jingjie Jiang, and others)
Test coverage (Jingjie Jiang, Clement Schreiner, and others)
Case-insensitive package name search (Akshita Jha)
?lang=LANG to override detected language (Jingjie Jiang)
Symbolic links handling (Jingjie Jiang)
Better statistics charts (Orestis Ioannou)
Python3 support (Zack)
Matthieu Caneill Debsources MiniDebconf, April 2015 31 / 48
![Page 55: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/55.jpg)
Roadmap
Static analysis
Automatic runs of static analysis tools (e.g. clang, coccinelle) on allDebian packages
Statistics gathering on bugs evolution
→ Debile, Firewoes
copyright.debian.net
Web application
What is the license of package X? Is it compatible with package Y?
Licenses searching/browsing
Statistics gathering
Matthieu Caneill Debsources MiniDebconf, April 2015 32 / 48
![Page 56: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/56.jpg)
Roadmap
Static analysis
Automatic runs of static analysis tools (e.g. clang, coccinelle) on allDebian packages
Statistics gathering on bugs evolution
→ Debile, Firewoes
copyright.debian.net
Web application
What is the license of package X? Is it compatible with package Y?
Licenses searching/browsing
Statistics gathering
Matthieu Caneill Debsources MiniDebconf, April 2015 32 / 48
![Page 57: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/57.jpg)
RoadmapAnd many smaller items
more live stats
file name search
binary package → source package redirection
tarball-in-tarball support
100% test suite coverage
file-level deduplicationI select count(∗) from checksums; → 35’370’653I select count(distinct sha256) from checksums; → 15’822’745
⇒ deduplicated core: ≈ 45%
Matthieu Caneill Debsources MiniDebconf, April 2015 33 / 48
![Page 58: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/58.jpg)
RoadmapAnd many smaller items
more live stats
file name search
binary package → source package redirection
tarball-in-tarball support
100% test suite coverage
file-level deduplicationI select count(∗) from checksums; → 35’370’653I select count(distinct sha256) from checksums; → 15’822’745
⇒ deduplicated core: ≈ 45%
Matthieu Caneill Debsources MiniDebconf, April 2015 33 / 48
![Page 59: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/59.jpg)
RoadmapAnd many smaller items
more live stats
file name search
binary package → source package redirection
tarball-in-tarball support
100% test suite coverage
file-level deduplicationI select count(∗) from checksums; → 35’370’653I select count(distinct sha256) from checksums; → 15’822’745
⇒ deduplicated core: ≈ 45%
Matthieu Caneill Debsources MiniDebconf, April 2015 33 / 48
![Page 60: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/60.jpg)
RoadmapAnd many smaller items
more live stats
file name search
binary package → source package redirection
tarball-in-tarball support
100% test suite coverage
file-level deduplicationI select count(∗) from checksums; → 35’370’653I select count(distinct sha256) from checksums; → 15’822’745
⇒ deduplicated core: ≈ 45%
Matthieu Caneill Debsources MiniDebconf, April 2015 33 / 48
![Page 61: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/61.jpg)
RoadmapAnd many smaller items
more live stats
file name search
binary package → source package redirection
tarball-in-tarball support
100% test suite coverage
file-level deduplicationI select count(∗) from checksums; → 35’370’653I select count(distinct sha256) from checksums; → 15’822’745
⇒ deduplicated core: ≈ 45%
Matthieu Caneill Debsources MiniDebconf, April 2015 33 / 48
![Page 62: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/62.jpg)
RoadmapAnd many smaller items
more live stats
file name search
binary package → source package redirection
tarball-in-tarball support
100% test suite coverage
file-level deduplicationI select count(∗) from checksums; → 35’370’653I select count(distinct sha256) from checksums; → 15’822’745
⇒ deduplicated core: ≈ 45%
Matthieu Caneill Debsources MiniDebconf, April 2015 33 / 48
![Page 63: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/63.jpg)
Table of contents
1 Introduction
2 FeaturesDebsources’ featuresWhat’s new?Roadmap
3 Technologies
4 Research platform
5 Hacking
Matthieu Caneill Debsources MiniDebconf, April 2015 34 / 48
![Page 64: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/64.jpg)
TechnologiesWhat languages and technologies do we use?
Code base: (almost) entirely in Python
Web application: Flask framework, Jinja2 templates,HTML/CSS/Javascript
Database: PostgreSQL
Apache web server, SQLAlchemy, ...
Matthieu Caneill Debsources MiniDebconf, April 2015 35 / 48
![Page 65: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/65.jpg)
TechnologiesWhat languages and technologies do we use?
Code base: (almost) entirely in Python
Web application: Flask framework, Jinja2 templates,HTML/CSS/Javascript
Database: PostgreSQL
Apache web server, SQLAlchemy, ...
Matthieu Caneill Debsources MiniDebconf, April 2015 35 / 48
![Page 66: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/66.jpg)
TechnologiesWhat languages and technologies do we use?
Code base: (almost) entirely in Python
Web application: Flask framework, Jinja2 templates,HTML/CSS/Javascript
Database: PostgreSQL
Apache web server, SQLAlchemy, ...
Matthieu Caneill Debsources MiniDebconf, April 2015 35 / 48
![Page 67: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/67.jpg)
TechnologiesWhat languages and technologies do we use?
Code base: (almost) entirely in Python
Web application: Flask framework, Jinja2 templates,HTML/CSS/Javascript
Database: PostgreSQL
Apache web server, SQLAlchemy, ...
Matthieu Caneill Debsources MiniDebconf, April 2015 35 / 48
![Page 68: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/68.jpg)
OverviewArchitecture
Matthieu Caneill Debsources MiniDebconf, April 2015 36 / 48
![Page 69: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/69.jpg)
Overview
data model (excerpt) checksums*
id serial PK package_id int file_id int FK sha256 varchar files
id serial PK package_id int FK path bytea
packages id serial PK version varchar name_id int FK area varchar sticky boolean
ctags* id serial PK package_id int FK tag varchar file_id int FK line int kind varchar language lang_ctags
metrics id serial PK package_id int FK metric metric_type value_ int
package_names id serial PK name varchar
sloccounts* id serial PK package_id int FK language lang_sloc count int
suites id serial PK package_id int FK suite varchar
table rows1
suites info 16
package names 29,286(source) packages 83,597suites (mapping) 120,550
metrics (e.g., du) 83,597sloccounts 298,360checksums 35,370,653ctags 358,773,259
1snapshot, 31 July 2014Matthieu Caneill Debsources MiniDebconf, April 2015 37 / 48
![Page 70: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/70.jpg)
Disk usage
unpacked sources: 609 GB
PostgreSQL DB: 111 GB
Source mirror: 71 GB
Hosting requirements: ≈ 800 GB
(31 July 2014)
Figure: unpacked sources trend (peekdue to archive.d.o injection)
Matthieu Caneill Debsources MiniDebconf, April 2015 38 / 48
![Page 71: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/71.jpg)
Disk usage
unpacked sources: 609 GB
PostgreSQL DB: 111 GB
Source mirror: 71 GB
Hosting requirements: ≈ 800 GB
(31 July 2014)
Figure: unpacked sources trend (peekdue to archive.d.o injection)
Matthieu Caneill Debsources MiniDebconf, April 2015 38 / 48
![Page 72: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/72.jpg)
Table of contents
1 Introduction
2 FeaturesDebsources’ featuresWhat’s new?Roadmap
3 Technologies
4 Research platform
5 Hacking
Matthieu Caneill Debsources MiniDebconf, April 2015 39 / 48
![Page 73: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/73.jpg)
Research platform
Facts
Debsources is a huge software collection.
Homogeneous: all software follow Debian’s packaging format.
It is up-to-date.
Software evolution
20 years of source code evolution.
Plugins to compute stats.
Nice charts can be computed with this!Example: What are the trending programming languages?
Matthieu Caneill Debsources MiniDebconf, April 2015 40 / 48
![Page 74: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/74.jpg)
Research platform
Facts
Debsources is a huge software collection.
Homogeneous: all software follow Debian’s packaging format.
It is up-to-date.
Software evolution
20 years of source code evolution.
Plugins to compute stats.
Nice charts can be computed with this!Example: What are the trending programming languages?
Matthieu Caneill Debsources MiniDebconf, April 2015 40 / 48
![Page 75: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/75.jpg)
Research platform
Facts
Debsources is a huge software collection.
Homogeneous: all software follow Debian’s packaging format.
It is up-to-date.
Software evolution
20 years of source code evolution.
Plugins to compute stats.
Nice charts can be computed with this!Example: What are the trending programming languages?
Matthieu Caneill Debsources MiniDebconf, April 2015 40 / 48
![Page 76: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/76.jpg)
Research platformSoftware metrics evolution over Debian releases
Matthieu Caneill Debsources MiniDebconf, April 2015 41 / 48
![Page 77: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/77.jpg)
Research platformFile size per language, evolution over Debian releases
Matthieu Caneill Debsources MiniDebconf, April 2015 42 / 48
![Page 78: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/78.jpg)
Research platformAbsolute evolution of SLOC per language, over Debian releases
Matthieu Caneill Debsources MiniDebconf, April 2015 43 / 48
![Page 79: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/79.jpg)
Research platformRelative evolution of SLOC per language, over Debian releases
Matthieu Caneill Debsources MiniDebconf, April 2015 44 / 48
![Page 80: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/80.jpg)
Research platformPublications
Matthieu Caneill, Stefano Zacchiroli. Debsources: Live andHistorical Views on Macro-Level Software Evolution. Inproceedings of ESEM 2014: 8th International Symposium onEmpirical Software Engineering and Measurement.
Stefano Zacchiroli. The Debsources Dataset: Two Decades ofDebian Source Code Metadata. To appear in proceedings of MSR2015: The 12th Working Conference on Mining Software Repositories.
You can find the PDFs of the articles on http://sources.debian.net/doc/.
Matthieu Caneill Debsources MiniDebconf, April 2015 45 / 48
![Page 81: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/81.jpg)
Table of contents
1 Introduction
2 FeaturesDebsources’ featuresWhat’s new?Roadmap
3 Technologies
4 Research platform
5 Hacking
Matthieu Caneill Debsources MiniDebconf, April 2015 46 / 48
![Page 82: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/82.jpg)
HackingHow can I contribute?
Step 1: clone Debsources git repository
git clone git://anonscm.debian.org/qa/debsources.git
Step 2: Set-up a development environment
Follow the instructions in the HACKING file,
Or docker run!https://github.com/matthieucan/Dockerfiles/debsources
Will setup a Docker container with all batteries included:dependencies, database, test data, configuration.
Step 3: open your editor and hack!
Bugs list: https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=
qa.debian.org;tag=debsources
or: implement your own plugin (see examples), add features, etc.
Matthieu Caneill Debsources MiniDebconf, April 2015 47 / 48
![Page 83: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/83.jpg)
HackingHow can I contribute?
Step 1: clone Debsources git repository
git clone git://anonscm.debian.org/qa/debsources.git
Step 2: Set-up a development environment
Follow the instructions in the HACKING file,
Or docker run!https://github.com/matthieucan/Dockerfiles/debsources
Will setup a Docker container with all batteries included:dependencies, database, test data, configuration.
Step 3: open your editor and hack!
Bugs list: https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=
qa.debian.org;tag=debsources
or: implement your own plugin (see examples), add features, etc.
Matthieu Caneill Debsources MiniDebconf, April 2015 47 / 48
![Page 84: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/84.jpg)
HackingHow can I contribute?
Step 1: clone Debsources git repository
git clone git://anonscm.debian.org/qa/debsources.git
Step 2: Set-up a development environment
Follow the instructions in the HACKING file,
Or docker run!https://github.com/matthieucan/Dockerfiles/debsources
Will setup a Docker container with all batteries included:dependencies, database, test data, configuration.
Step 3: open your editor and hack!
Bugs list: https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=
qa.debian.org;tag=debsources
or: implement your own plugin (see examples), add features, etc.
Matthieu Caneill Debsources MiniDebconf, April 2015 47 / 48
![Page 85: Debsources - Dive into Debian source code! - Matthieu Caneill · Debsources Dive into Debian source code! Matthieu Caneill Debian contributor, PhD student at LIG April 11, 2015 MiniDebconf(Lyon,](https://reader031.vdocuments.site/reader031/viewer/2022022118/5cd8272088c993ad7c8d8b9a/html5/thumbnails/85.jpg)
Thanks! Questions?
Matthieu Caneillhttp://matthieu.io
Slides: derived from Stefano Zacchirolihttp://upsilon.cc/~zack/talks/2014/20140826-dc14-debsources.pdf
Licensed CC-BY-SA.