sunlight labs & mongodb @ mongodc
DESCRIPTION
A look at how Sunlight Labs uses MongoDB. Presented at MongoDC on November 18, 2010.TRANSCRIPT
![Page 2: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/2.jpg)
![Page 3: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/3.jpg)
Question? @LuigiMontanez
![Page 4: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/4.jpg)
Question? @LuigiMontanez
Open Source + Open Data =
Open Government
![Page 5: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/5.jpg)
Question? @LuigiMontanez
High Quality Raw Data
✴ First: Raw data in JSON, XML, or CSV✴ Second: RESTful APIs in JSON or XML✴ Third: Nothing else...
![Page 6: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/6.jpg)
Question? @LuigiMontanez
MongoDB enablesopen data
![Page 7: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/7.jpg)
Question? @LuigiMontanez
JSON has won(among developers)
![Page 8: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/8.jpg)
Question? @LuigiMontanez
Opening Up Data
✴ Storing data from disparate sources✴ Data dumps✴ Web scraping✴ Text/PDF parsing
✴ Serving RESTful JSON APIs
![Page 9: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/9.jpg)
Question? @LuigiMontanez
Three Projects
✴ National Data Catalog✴ Real-Time Congress API✴ Open State Project
![Page 10: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/10.jpg)
Question? @LuigiMontanez
Three Projects
✴ National Data Catalog✴ Real-Time Congress API✴ Open State Project
![Page 11: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/11.jpg)
![Page 12: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/12.jpg)
Question? @LuigiMontanez
App designdrives
schema design
![Page 13: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/13.jpg)
![Page 14: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/14.jpg)
Text
{ "title": "Worldwide M1+ Earthquakes, Past Hour"}
![Page 15: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/15.jpg)
![Page 16: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/16.jpg)
Text
{ "title": "Worldwide M1+ Earthquakes, Past Hour", "description": "Real-time, worldwide earthquake list for the past hour", "homepage": "http://data.gov/raw/32", "official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/", "organization": "Department of the Interior", "original_catalog": "data.gov",}
![Page 17: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/17.jpg)
Text
{ "title": "Worldwide M1+ Earthquakes, Past Hour", "description": "Real-time, worldwide earthquake list for the past hour", "homepage": "http://data.gov/raw/32", "official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/", "organization_id": "4cbcc0ff2c34576ba4000001", "catalog_id": "4cbcc0ab2d34d76b97020433",}
![Page 18: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/18.jpg)
{ "title": "Worldwide M1+ Earthquakes, Past Hour", "description": "Real-time, worldwide earthquake list for the past hour", "homepage": "http://data.gov/raw/32", "official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/", "organization": { "name": "Department of the Interior", "id": "4cbcc0ff2c34576ba4000001", "slug": "us-dept-of-interior" }, "original_catalog": { "name": "data.gov", "id": "4cbcc0ab2d34d76b97020433", "slug": "datagov" }}
![Page 19: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/19.jpg)
![Page 20: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/20.jpg)
{ "title": "Worldwide M1+ Earthquakes, Past Hour", "description": "Real-time, worldwide earthquake list for the past hour", "homepage": "http://data.gov/raw/32", "official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/", "organization": { "name": "Department of the Interior", "id": "4cbcc0ff2c34576ba4000001", "slug": "us-dept-of-interior" }, "original_catalog": { "name": "data.gov", "id": "4cbcc0ab2d34d76b97020433", "slug": "datagov" }, "downloads": [ { "type": "csv", "url": "http://data.gov/download/32/csv" } ], "ratings" : { "average_rating": 3.5, "rating_count": 23 }, "comments": []}
![Page 21: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/21.jpg)
![Page 22: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/22.jpg)
Question? @LuigiMontanez
User-centric data?✴ Source document: contains collection of
user data✴ User document: contains collection of
source data✴ UserSource document✴ Rating, Favorite, Note docs
![Page 23: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/23.jpg)
Question? @LuigiMontanez
Freedom of choice
![Page 24: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/24.jpg)
Question? @LuigiMontanez
Three Projects
✴ National Data Catalog✴ Real-Time Congress API✴ Open State Project
![Page 25: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/25.jpg)
Real-Time Congress API(Drumbone)
Credit: vgm8383 on Flickr
![Page 26: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/26.jpg)
Android App: “Congress”
![Page 27: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/27.jpg)
Politiwidgets
![Page 28: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/28.jpg)
Question? @LuigiMontanez
Requirements✴ Aggregate lots of data
Biographical, Bills, Votes, Earmarks, Video Clips, Floor Updates, Legislative Documents, Committee Schedules, Contributions, Interest Group Ratings
✴ Lightweight responses
![Page 29: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/29.jpg)
{legislator: { in_office: true, title: "Rep", nickname: "", district: "9", bioguide_id: "L000551", govtrack_id: "400237", phone: "202-225-2661", website: "http://lee.house.gov/index.html", twitter_id: "", last_name: "Lee", name_suffix: "", last_updated: "2010/04/13 00:00:14 +0000", party: "D", chamber: "house", state: "CA", youtube_url: "http://www.youtube.com/RepLee", first_name: "Barbara", gender: "F", congress_office: "2444 Rayburn House Office Building", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } ...}
![Page 30: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/30.jpg)
// limit selection to a subset of fieldsdb.people.find( { 'first_name' : 'john' }, { 'last_name' : 1, 'address' : 1 } );
// use dot-notation to dig into an objectdb.people.find( { 'state': 'CA' }, { 'address.zip_code': 1 } );
![Page 31: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/31.jpg)
{legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, }}
?sections=last_name,first_name,state,earmarks
![Page 32: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/32.jpg)
{legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { total_amount: 10000000, total_number: 28 }}
?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number
![Page 33: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/33.jpg)
Question? @LuigiMontanez
Partial responses make payloads
smaller
![Page 34: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/34.jpg)
Question? @LuigiMontanez
Three Projects
✴ National Data Catalog✴ Real-Time Congress API✴ Open State Project
![Page 35: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/35.jpg)
![Page 36: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/36.jpg)
Question? @LuigiMontanez
50 States =50 Formats
![Page 37: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/37.jpg)
Question? @LuigiMontanez
Schemalessness allows for
losslessness
![Page 38: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/38.jpg)
Source Scraped JSON PythonTransform PostgreSQL
![Page 39: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/39.jpg)
Source Scraped JSON MongoDB
![Page 40: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/40.jpg)
Question? @LuigiMontanez
Three Projects
✴ National Data Catalog✴ Real-Time Congress API✴ Open State Project
![Page 41: Sunlight Labs & MongoDB @ MongoDC](https://reader033.vdocuments.site/reader033/viewer/2022052504/54bf76b04a7959a9278b4605/html5/thumbnails/41.jpg)
Question? @LuigiMontanez
Thanks!sunlightlabs.com@LuigiMontanez