elasticsearch at affiliatewindow

Elasticsearch@Affiliate Window

Richard HeTeam Falcon

A transaction report used by tens of thousands of users every day that took too long to load

Our problem

Near real-time latency Really fast aggregations Accessible individual transaction records Flexible Scalable

What we wanted

The solution - Producing

The solution - Consuming

When shit hits the fan

Next steps

A good architect maximises the number of decisions NOT made

by Uncle Bob Martin

New data fields can be added with almost no work Average response time is <200ms Easy to scale by just adding nodes REST API

Why Elasticsearch

Things that are good to know…

Transaction Log - Refresh - Flush

1.x – fsync every 5 seconds 2.x – on request (default) / fsync every 5 seconds

Transaction log durability

"refresh_interval" : "-1” "refresh_interval" : ”1s” "refresh_interval" : ”15s"

Refresh intervalA refresh makes data searchable and a refresh interval decides how often that happens

Filters vs Queries (v 1.7)A filter askes yes / no questions, a query ask how well does this document match

76ms 1385ms

Filters vs Queries Disable analysed field if not needed Use a filter if a query is not required

Doc Values

Set a smaller heap size (4-16G on a 64G machine) More RAM is now available for File System Cache Enabled by default in version 2 Does not work with analysed field

On-disk data structure that allows fast aggregation without JVM Heap overhead

Aggregations are approximate

Wrong!

The solution Size = 0 The cost

Split Brain

M

4

52

1

1

2

3

4 5

M

2

M

4 5

M

2

M

4 5

1

2

M

4

5

Doc valueshttps://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.htmlhttps://www.elastic.co/guide/en/elasticsearch/guide/current/_deep_dive_on_doc_values.html

Aggregationhttps://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

Update Index settingshttps://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html

Important config changeshttps://www.elastic.co/guide/en/elasticsearch/guide/1.x/_important_configuration_changes.htmlhttps://www.elastic.co/guide/en/elasticsearch/guide/current/translog.htmlhttps://github.com/elastic/elasticsearch/issues/11287

Transaction loghttps://www.elastic.co/guide/en/elasticsearch/guide/current/translog.htmlhttps://github.com/elastic/elasticsearch/issues/11287

Resources

Questions