how we are using bigquery and apps scripts at teowaki
DESCRIPTION
I was invited to speak at the Google Startup Launch Summit in London about how we are using the google cloud to power our startupTRANSCRIPT
![Page 1: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/1.jpg)
javier ramirez@supercoco9
How we are usingBigQuery andApps Scripts
at teowaki
![Page 2: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/2.jpg)
![Page 3: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/3.jpg)
![Page 4: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/4.jpg)
![Page 5: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/5.jpg)
![Page 6: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/6.jpg)
Set a distance.
Set an expiration time.
Bye bye noise.
![Page 7: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/7.jpg)
![Page 8: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/8.jpg)
![Page 9: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/9.jpg)
![Page 10: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/10.jpg)
![Page 11: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/11.jpg)
![Page 12: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/12.jpg)
Analytics flow
![Page 13: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/13.jpg)
Analytics flow, by segment
![Page 14: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/14.jpg)
Automatic Alerts
![Page 15: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/15.jpg)
![Page 16: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/16.jpg)
![Page 17: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/17.jpg)
![Page 18: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/18.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
REST API (Ruby on Rails) +
Web on top (AngularJS)
![Page 19: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/19.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 20: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/20.jpg)
data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the structures of your database architectures.
Ed Dumbill program chair for the O’Reilly Strata Conference
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 21: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/21.jpg)
1. non intrusive metrics2. keep the history3. avoid vendor lock-in4. interactive queries5. cheap6. extra ball: real time
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 22: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/22.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 23: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/23.jpg)
Cloud Storage:Cost-efficient storage of files
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 24: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/24.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 25: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/25.jpg)
HadoopCassandraAmazon Redshift...
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
tools we considered:
![Page 26: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/26.jpg)
Our choice:
Google BigQuery
Data analysis as a service
http://developers.google.com/bigquery
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 27: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/27.jpg)
Based on “Dremel”
Specifically designed for interactive queries over petabytes of real-time data
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 28: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/28.jpg)
loading data
You just send the data intext (or JSON) format
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 29: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/29.jpg)
SQL
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
select name from USERS order by date;
select count(*) from users;
select max(date) from USERS;
select sum(total) from ORDERS group by user;
![Page 30: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/30.jpg)
specific extensions for analytics
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
withinflattennest
stddev
topfirstlastnth
variance
var_popvar_samp
covar_popcovar_samp
quantiles
![Page 31: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/31.jpg)
web console screenshot
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 32: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/32.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
our most active user
![Page 33: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/33.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
country segmented traffic
![Page 34: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/34.jpg)
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
10 request we should be caching
![Page 35: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/35.jpg)
javier ramirez @supercoco9 http://teowaki.com startup launch summit london 14
5 most created resources
![Page 36: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/36.jpg)
new users per month
![Page 37: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/37.jpg)
SELECT repository_name, repository_language, repository_description, COUNT(repository_name) as cnt,repository_urlFROM github.timelineWHERE type="WatchEvent"AND PARSE_UTC_USEC(created_at) >= PARSE_UTC_USEC("#{yesterday} 20:00:00")AND repository_url IN (
SELECT repository_urlFROM github.timelineWHERE type="CreateEvent"AND PARSE_UTC_USEC(repository_created_at) >= PARSE_UTC_USEC('#{yesterday} 20:00:00')AND repository_fork = "false"AND payload_ref_type = "repository"GROUP BY repository_url
)GROUP BY repository_name, repository_language, repository_description, repository_urlHAVING cnt >= 5ORDER BY cnt DESCLIMIT 25
![Page 38: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/38.jpg)
![Page 39: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/39.jpg)
![Page 41: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/41.jpg)
Automation with Apps Script
Read from bigquery
Create a spreadsheet on Drive
E-mail it everyday as a PDF
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 42: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/42.jpg)
![Page 43: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/43.jpg)
![Page 44: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/44.jpg)
![Page 45: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/45.jpg)
![Page 46: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/46.jpg)
![Page 47: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/47.jpg)
cloud storage pricing
$0.032 per GB
a gzipped 4.8 MB file stores 1MM rows
$0.000092 / month per 1MM rows
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 48: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/48.jpg)
bigquery pricing
$26 per stored TB1000000 rows => $0.00416 / month
£0.00243 / month
$5 per processed TB1 full scan = 160 MB1 count = 0 MB1 full scan over 1 column = 5.4 MB100 GB => $0.05 / month £0.03
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 49: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/49.jpg)
£0.054307 / month*
per 1MM rows
*the 1st 100GB every month are free of charge
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 50: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/50.jpg)
1. non intrusive metrics2. keep the history3. avoid vendor lock-in4. interactive queries5. cheap6. extra ball: real time
javier ramirez @supercoco9 https://teowaki.com startup launch summit london 14
![Page 51: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/51.jpg)
ig
![Page 52: How we are using BigQuery and Apps Scripts at teowaki](https://reader034.vdocuments.site/reader034/viewer/2022042814/54c6cba44a795943608b4589/html5/thumbnails/52.jpg)
Find related links at
https://teowaki.com/teams/javier-community/link-categories/bigquery-talk
Thanks!
Javier Ramírez@supercoco9
startup launch summit london 14