elasticsearch at affiliatewindow
TRANSCRIPT
Elasticsearch@Affiliate Window
Richard HeTeam Falcon
A transaction report used by tens of thousands of users every day that took too long to load
Our problem
Near real-time latency Really fast aggregations Accessible individual transaction records Flexible Scalable
What we wanted
The solution - Producing
The solution - Consuming
When shit hits the fan
Next steps
A good architect maximises the number of decisions NOT made
by Uncle Bob Martin
New data fields can be added with almost no work Average response time is <200ms Easy to scale by just adding nodes REST API
Why Elasticsearch
Things that are good to know…
Transaction Log - Refresh - Flush
1.x – fsync every 5 seconds 2.x – on request (default) / fsync every 5 seconds
Transaction log durability
"refresh_interval" : "-1” "refresh_interval" : ”1s” "refresh_interval" : ”15s"
Refresh intervalA refresh makes data searchable and a refresh interval decides how often that happens
Filters vs Queries (v 1.7)A filter askes yes / no questions, a query ask how well does this document match
76ms 1385ms
Filters vs Queries Disable analysed field if not needed Use a filter if a query is not required
Doc Values
Set a smaller heap size (4-16G on a 64G machine) More RAM is now available for File System Cache Enabled by default in version 2 Does not work with analysed field
On-disk data structure that allows fast aggregation without JVM Heap overhead
Aggregations are approximate
Wrong!
The solution Size = 0 The cost
Split Brain
M
4
52
1
1
2
3
4 5
M
2
M
4 5
M
2
M
4 5
1
2
M
4
5
Doc valueshttps://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.htmlhttps://www.elastic.co/guide/en/elasticsearch/guide/current/_deep_dive_on_doc_values.html
Aggregationhttps://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
Update Index settingshttps://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html
Important config changeshttps://www.elastic.co/guide/en/elasticsearch/guide/1.x/_important_configuration_changes.htmlhttps://www.elastic.co/guide/en/elasticsearch/guide/current/translog.htmlhttps://github.com/elastic/elasticsearch/issues/11287
Transaction loghttps://www.elastic.co/guide/en/elasticsearch/guide/current/translog.htmlhttps://github.com/elastic/elasticsearch/issues/11287
Resources
Questions