realtime analytics with mongodb counters (mongonyc 2012)
DESCRIPTION
Real time analytics with pre-aggregation with counters.TRANSCRIPT
![Page 1: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/1.jpg)
Real Time Data Analytics
Pre-aggregation with counters
© Copyright 2010 10gen Inc.
![Page 2: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/2.jpg)
Goals
• Dashboard style reports• (Known) Reports• Real-time numbers
![Page 3: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/3.jpg)
Framework
• Know your metrics/counter• Prepared reports• Calculate during write• Fast queries• Always up to date data• Record time-series collections
![Page 4: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/4.jpg)
Rationale
• Documents are updated in-place*• $inc update operator• Working set is small• Aggregations are much smaller*
![Page 5: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/5.jpg)
Dashboard
JavaScript Java Ruby Python
16 27 42 45
55436497
231435433401
9212342
1234
Projects Lines Events
Monday TuesdayThursday Friday
![Page 6: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/6.jpg)
Demo Dashboard
![Page 7: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/7.jpg)
Roads not traveled
• Map/Reduce• Reprocess raw data• Now possible to do partial reduce
• Aggregation Framework (aggregate in 2.2)• Also reprocess data on operation (initial release)• Optimizations to come
• More costly during reads
![Page 8: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/8.jpg)
Not Appropriate For
• Ad-hoc aggregations (unknown metrics)• One-off reports• Possibly complex calculations
![Page 9: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/9.jpg)
Processing
• Event received• Split into many updates w/$inc• Aggregate
• Input Field(s)• Time periods (hourly, daily, monthly)• Defined Metrics
![Page 10: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/10.jpg)
Example Data: github> db.events.findOne() {
"repository" : {
"url" : "https://github.com/vidageek/games",
...
"open_issues" : 25,
"watchers" : 6,
"pushed_at" : "2012/03/10 08:34:00 -0800",
"language" : "Java"
},
"actor_attributes" : {...},
"created_at" : "2012/03/11 15:20:24 -0700",
"public" : true,
"actor" : "juliano",
"payload" : {...},
"url" : "https://github.com/...",
"type" : "CommitCommentEvent” }
![Page 11: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/11.jpg)
Define Metrics
• “actor”• “repository.name”• “repository.language”• “type”
PushEvent, IssuesEvent, WatchEvent, GistEvent
• “payload.ref” efs/heads/improved_history, refs/heads/master, refs/heads/signs
![Page 12: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/12.jpg)
Aggregations
TimePeriod, type #
TimePeriod, author #
TimePeriod, project #
![Page 13: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/13.jpg)
Stats Collections
stats_[hourly/daily/monthly].actors
stats_[hourly/daily/monthly].projects
stats_[hourly/daily/monthly].langs
stats_[hourly/daily/monthly].types
![Page 14: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/14.jpg)
Stats
> db.stats_hourly.types.find({"_id.type":"GistEvent"}) {
"_id" : {
"p" : ISODate("2012-05-21T00:00:00Z"),
"type" : "GistEvent” },
"hour" : {
"2" : { "count" : 65 },
"3" : { "count" : 2 },
"7" : { ”count" : 130},
"8" : { "count" : 5 } },
"total" : { ”count" : 202 } }
![Page 15: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/15.jpg)
Updates Increment
Query:
{ ”p" : Date(…), "actor" : "neoplastic"}}
Update:
{ "$inc" : { "h.21.c" : 1 , "t.c" : 1}}
Upsert : true
![Page 16: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/16.jpg)
Query/Graphing
• Select by grouping (by date, by type/value)• Documents hold many data points
![Page 17: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/17.jpg)
The Whys
• Writing more data up front, helps with reads• Multiple data points per document• Documents hold many timed points• Good for graphs by time, or types• Nested for improved performance
![Page 18: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/18.jpg)
Thanks for coming… ne questions
![Page 19: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/19.jpg)
© Copyright 2010 10gen Inc.
drivers at mongodb.org
RESTActionScript3C# and .NETClojureColdFusionDelphiErlangF#Go: gomongoGroovyHaskellJavascriptLua
CC#C++ErlangHaskellJavaJavascriptPerlPHPPythonRuby
node.jsObjective CPHPPowerShellBlog postPythonRubyScalaScheme (PLT)Smalltalk: Dolphin Smalltalk
Community Supportedmongodb.org Supported
![Page 20: Realtime Analytics with MongoDB Counters (mongonyc 2012)](https://reader035.vdocuments.site/reader035/viewer/2022070312/5539886b4a79593a698b4a22/html5/thumbnails/20.jpg)
@mongodb
© Copyright 2010 10gen Inc.
conferences, appearances, and meetupshttp://www.10gen.com/events
http://bit.ly/mongofb
Facebook | Twitter | LinkedIn http://linkd.in/joinmongo
download at mongodb.org
support, training, and this talk brought to you by