mongodb europe 2016 - choosing between 100 billion travel options – instant search from amadeus

Change the Year in the Copyright field 1. Click ‘Insert’ in Top menu 2. Click ’Header & Footer’ 3. Write new Year

in field ‘Footer’ 4. Click ‘Apply to All’

{ “Amadeus”: ”Instant Search for Travel” }

© 2

016

Am

adeu

s S.

A.S

.

Breaking the boundaries of travel inspiration

Loic CRETIN Pierre PETITPREZ

November 15th MongoDB Europe - London

About us…

© 2

015

Am

adeu

s IT

Gro

up S

A

Pierre PETITPREZ Software Engineer,

Middleware

Loic CRETIN Manager Software

Development



© 2

016

Am

adeu

s S.

A.S

.

Agenda

{ “1”: “Who are we?”, “2”: “Data modelling”, “3”: “Sharding”, “4”: “Advanced queries”, “5”: “3.2 Connections”, “6”: “MongoDB @ Amadeus”, “7”: “What is coming…” }



{ “Topic”: “Who are we?” } 1

© 2

016

Am

adeu

s S.

A.S

.

Amadeus is a technology company dedicated to the

global travel industry. We are present in 195 countries with a worldwide team of more than 13,000 people. Our solutions help improve the

business performance of travel agencies, corporations, airlines, airports, hotels, railways and more.

Amadeus in a few words

© 2

015

Am

adeu

s IT

Gro

up S

A

Robust global operations

© 2

015

Am

adeu

s IT

Gro

up S

A

1.6+ billion data requests processed per day

566+ million travel agency bookings processed in 2015

747+ million Passengers Boarded (PBs) in 2015

95% of the world’s scheduled network airline seats



Amadeus Search, Shopping and Pricing solutions

Airlines Travel Agency

Online Web Sites

Search

Shopping Pricing

Re-Pricing



to complex infrastructure

© 2

016

Am

adeu

s S.

A.S

.

Consumer relevance & natural search

Content exhaustiveness

Instant response Time

25% travelers are not decided on destination

42% travelers are not decided on the date

76% travelers have no clear idea on budget to spent

From simple idea…



Use case description

Leaving from London, where can I go with 1200€ from 17th

to 21th of December? NYC

1038 €

SAO 1170 €

CPT 1006 €

SAO 1170 €

CPT 1006 €






1038 €

SAO 1170 €

CPT 1006 €

SAO 1170 €

CPT 1006 €

LON NYC 12/17 12/21

Travel.com

Search 1 adult economy

View deal

1038€ 10:50 LGW -> 18:45 JFK 12h 55m 1 stop (MAD) 22:10 JFK -> 10:00 LHR 6h 50m nonstop

View deal

1041€ 14:05 LHR -> 19:30 JFK 10h 25m 1 stop (AMS) 17:55 JFK -> 08:10 LHR 9h 15m 1 stop (AMS)

View deal

1084€ 11:20 LHR -> 19:02 JFK 12h 44m 1 stop (BOS) 11:25 LGA -> 06:50 LHR 10h 25m 1 stop (RDV)



Deep dive into the architecture

© 2

016

Am

adeu

s S.

A.S

.

Cache

Computation

LON-MAD-12354-IB-AF-123-1200-1354LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-

AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12

3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-123 -1200-1354 LON-MAD-12354-IB-AF-123- 1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354

LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-

MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-13

54 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD- 12354-IB-A1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-I AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354 12354-I













LON-MAD-12354-IB-AF-123-1200-1354LON-MAD-12354-IB-

AF-123-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354

LON-MAD-12354-IB-AF-12 3-1200-1354

LON-MAD-12354-IB-AF-12 3-1200-1354

LON-MAD-12354-IB-AF-12 3-1200-1354 LON-

MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-AF-12

3-1200-1354 LON-MAD-12354-IB-AF-12

3-1200-1354 LON-MAD-12354-IB-AF-12

3-1200-1354 LON-MAD-12354-IB-AF-12 3-1200-1354

LON-MAD-12354-IB-AF-12 3-1200-1354 LON-MAD-12354-IB-

AF-123 -1200-1354 LON-MAD-12354-IB-

AF-123- 1200-1354 LON-MAD-12354-IB-

AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-13

54 LON-MAD-12354-IB-AF-123-1200-1354 LON-

MAD-12354-IB-AF-123-1200-1354 LON-

MAD-12354-IB-AF-123-1200-1354 LON-

MAD-12354-IB-AF-123-1200-1354 LON-MAD-12354-IB-

AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354

LON-MAD-12354-IB-AF-123-1200-1354

LON-MAD-12354-IB-AF-123200-1354 LON-


AF-123-1200-1354 LON-MAD-12354-IB-AF-123-1200-1354

LON-MAD-12354-IB-AF-123-1200-1354

LON-MAD-12354-IB-AF-123-1200-1354 LON-MAD-

12354-IB-A1200-1354 LON-


AF-123-1200-1354 LON-MAD-12354-I AF-123-1200-13

54 LON-MAD-12354-IB-AF-123-1200-1354

12354-I

Indexing

Web Service



In-house NoSQL Store

© 2

016

Am

adeu

s S.

A.S

.

In-house technology

•  Functional data partitioning •  Non blocking updates, eventual consistency •  Indexing for multi-dimensional range queries •  In-memory and/or disk

Scalability limit

•  Use case with more than 1000 recommendations

•  Atomic data update for high cache quality

Move to Standard NoSQL technology



{ “Topic”: “data modeling” } 2

© 2

016

Am

adeu

s S.

A.S

.



Data modeling Principle & goals

© 2

015

Am

adeu

s IT

Gro

up S

A

Why?

Limit index entries, document fetch,

operation

What?

Find the best data model

and query strategy

How?

Go deeper into use case



Data modeling Focus on search use cases

© 2

015

Am

adeu

s IT

Gro

up S

A

Description

•  Get all price results of a cell •  Inputs: Client, Origin, Destination,

Departure and return date

Service Level Agreement

•  Response time O(100)ms •  No down time



Data modeling

•  30 stay durations per departure date •  360 departure dates per city pair •  250 price results per date

combination •  Thousands of city pairs per client •  Hundreds of clients

•  Total >100,000,000,000 results

•  bookable solution

Focus on data domain

© 2

015

Am

adeu

s IT

Gro

up S

A

LAX NYC

Stay Duration

Date

LAX BOS

Stay Duration

Date

NYC LAX

Stay Duration

Date

BOS LAX

Stay Duration

Date

BOS NYC

Stay Duration

Date

NYC BOS

Stay Duration

Date

30

360

O(1000)

ORI DEST OutDate Stay Fares/Flights 1 ORI DEST OutDate Stay Fares/Flights 2 ORI DEST OutDate Stay Fares/Flights 3

…

ORI DEST OutDate Stay Fares/Flights 249 ORI DEST OutDate Stay Fares/Flights 250

Travel Solution

250



Data modeling

•  Updates come per group of cell(s) •  Add/remove one date every day

•  Update could be done several time a day

•  Capacity to update all data in a day

Focus on update use cases

© 2

015

Am

adeu

s IT

Gro

up S

A


…



…



…



…



…



…



…



…


LAX NYC

Stay Duration

Date

LAX NYC

Stay Duration

Date



Data model: 1 document = 1 Cell

© 2

015

Am

adeu

s IT

Gro

up S

A

Document format:{ "_id" : ObjectId("570…2d6"), "Orig" : "NYC", "Dest" : "LAX", "DepDate" : ISODate("2016-06-27…"), "Stay" : 2, "PriceResultList" : [ {…Price result 1…},

{…Price result 2…}, … ]}

Highlights

•  Duplicate the key fields •  Divide by 250 the index

entries and operations •  468kB documents



Bulk.find({ AX ).update(

Queries

© 2

015

Am

adeu

s IT

Gro

up S

A

db.CLI.Update({"Orig" : "NYC", "Dest" : "LAX", "DepDate" : ISODate("2016-06-27…"), "Stay" : 2},{$set:{ PriceResultList:[{…},…] }},{upsert:1})

db.CLI.find({"Orig" : "NYC", "Dest" : "LAX", "DepDate" : ISODate("2016-06-27…"), "Stay" : 2},{_id:0,PriceResultList:1})



Indexes and Shard key

© 2

015

Am

adeu

s IT

Gro

up S

A

{ "Orig" : 1, "Dest" : 1, "DepDate" : 1, "Stay" : 1 }

{ "v" : 1, "key" : { "Orig" : 1, "Dest" : 1,

"DepDate" : 1, "Stay" : 1

}, "name" : "KeyIdx" }

{ "v" : 1, "key" : {

"DepDate" : 1 }, "name" : "expireAt", "expireAfterSeconds" : 172800 }

Data Access index Expiration index Shard key



{ “Topic”: “Sharding - solution for limitations with consequences” }

3

© 2

015

Am

adeu

s IT

Gro

up S

A



Why sharding help you

© 2

015

Am

adeu

s IT

Gro

up S

A

Replication throughput limitation

Initial Sync speed

Update performance (index size reduction)

Replication lag



Sharding consequence for us Split/Migration chunks problems

© 2

015

Am

adeu

s IT

Gro

up S

A

Split & Balancing leads to peak of response time

Shard key important (Date into shard key, Empty chunk)

Split capacity decrease with number of chunks Slow balancing

SplitAt loop 3 days for 200k

Add of 4 shards 1 week of balancing



Split/Migration chunks Solution

© 2

015

Am

adeu

s IT

Gro

up S

A

Shard 1

OldCol Chunks

Chunk1

Shard 2

Chunk2

NewCol

Chunks Chunk1

Chunk2

Feed Search

Free Space

Deactivate autosplit & balancing

Shard key: Hashkey

Manual dispatch (double feed/search)



{ “Topic”: “Advanced queries” } 5

© 2

015

Am

adeu

s IT

Gro

up S

A






1038 €

SAO 1170 €

CPT 1006 €

SAO 1170 €

CPT 1006 €



Insert Co-branding logo 1. Click on placeholder

2. Click ’Insert’ 3. Click ‘Picture’

4. Locate the co-branding logo, click Insert

5. Align with bottom line of amadeus-logo

NYC LON

Stay Duration

Date ORI DEST OutDate Stay Fares/Flights 3

NYC PAR

Stay Duration

Date

ORI DEST OutDate Stay Fares/Flights 3

NYC LAX

Stay Duration

Date

LON-???

© 2

015

Am

adeu

s IT

Gro

up S

A

Open destination







•  Fixed Value: ü Client ü Origin ü Destination

•  Range: ü Departure date ü Return date ü Budget

Return best travel solutions (lowest price) per destinations


…



…



…



…



…



…




Insert Co-branding logo 1. Click on placeholder

2. Click ’Insert’ 3. Click ‘Picture’

4. Locate the co-branding logo, click Insert

5. Align with bottom line of amadeus-logo

LON - ???

© 2

015

Am

adeu

s IT

Gro

up S

A

Open destination db.IFL.aggregate( [{"$match": { "Ori": "NYC", "DepDate": {"$gte": ISODate("2016-06-20T…"), "$lte": ISODate("2016-06-30T…")}, "Stay": {"$gte": 4, "$lte": 10}, "PriceResultList.amount": {"$lte": 900} }}, {"$unwind": "$PriceResultList"}, {"$project": { "Dest": "$Dest", "matcher": { "prices": "$PriceResultList.amount", "doc": "$PriceResultList" } }}, {"$group": { "_id": { "Dest": "$Dest", }, "price": { "$min": "$matcher" }}}]);

Using MongoDB aggregation framework

Performance issue taking minutes



Data model: 1 document = 1 Cell = “best” price

© 2

015

Am

adeu

s IT

Gro

up S

A

{ "_id" : ObjectId("570…2d6"), "Orig" : "NYC", "Dest" : "LAX", "DepDate" : ISODate("2016-06-27…"), "Stay" : 2, "PR" : { "type" : "CHP", "amount" : 215.12, …price result fields… } }}

additional index:{ "v" : 1, "key" : { "Orig" : 1, "PR.type" : 1, "Dest" :1, "PR.amount" : 1, "Stay" : 1,

"DepDate" : 1, }, "name" : "Cheapest" },

New Collection (only cheapest), new index Using MongoDB aggregation framework

Performance issue still above the second

equality

sort range



Query strategy: Open destination

© 2

015

Am

adeu

s IT

Gro

up S

A

Two collections •  Destinations candidate •  City pair prices

Don’t use MongoDB aggregation framework (combination of find) Grouping with Aggregation framework took 350 ms

Better performance



{ “Topic”: “3.2 Connections” } 5

© 2

015

Am

adeu

s IT

Gro

up S

A



3.2 Connection pools Problems

© 2

015

Am

adeu

s IT

Gro

up S

A

_ Open search activation on 3.2 _ Migration from 3.0 to 3.2

_ Search query response time around 5 second (several peaks per minutes)

_ Open/Close connections _ Peaks disappear with higher

search traffic

Max response time

Average response time

Connection activity per shard (ops-manage)



3.2 Connection pools Investigation on-going

© 2

015

Am

adeu

s IT

Gro

up S

A

_ Connection pooling enhancement •  3.0: One Global connection pool •  3.2: Tunable smaller connection pool

_ taskExecutorPoolSize parameter set by default to 0 •  Automatically set depending the number of available cores •  64 cores on machine hosted mongos -> many connections idle

_ SERVER-25027 Server ticket



3.2 Connection pools Investigation on-going

© 2

015

Am

adeu

s IT

Gro

up S

A

_ taskExecutorPoolsize set to •  4 on mongod side •  16 on mongos side

_ Connections parameters tunable into 3.4 (3.4.0-rc3) •  kDefaultMinConns •  kDefaultRefreshTimeout •  kDefaultRefreshRequirement •  kDefaultHostTimeout

Max response time

Average response time



MongoDB Enterprise

© 2

015

Am

adeu

s IT

Gro

up S

A

Automation

Monitoring

Backup / restore

Training / consulting

Commercial support

Security

MongoDB Enterprise



Topology DB Ops Manager

Operations

© 2

015

Am

adeu

s IT

Gro

up S

A

Internal tooling

Monitoring

Alerting

Operability

Configuration deployment



Middleware

© 2

015

Am

adeu

s IT

Gro

up S

A

Enterprise-wide standard

Client side monitoring

Built-in best practices Resiliency (Mongos HA)



Deployment

© 2

015

Am

adeu

s IT

Gro

up S

A

Monitoring / Alerting

MongoDB Cluster

Server 1

Shard 1

Shard 2

Shard 3 …

Shard n

Server 2

Shard 1

Shard 2

Shard 3 …

Shard n

Server 3

Shard 1

Shard 2

Shard 3 …

Shard n

Ops Manager

Corporate tooling

Application servers

BE

S S

BE

S S

BE

S S



Deployment

© 2

015

Am

adeu

s IT

Gro

up S

A

_  Hardware •  24 physical machines

•  1152 CPU •  6 TB of RAM •  72 TB Fusion IO

•  3 members replica set •  56 shards in total •  2 Clusters

_  Traffic •  14.6 Billions Price results •  40 TB Data uncompressed •  5.3 TB Data compressed on disk •  12 Millions transaction Search / day

•  Mongo response time ~0.15ms/PR •  120 Millions transaction Update / day

•  18.6 Billions PR / day •  Mongo response time ~0.10ms/PR

Add shards

Add shards Add

shards

Size per shard



What’s coming… Scaling up

© 2

016

Am

adeu

s S.

A.S

.

Handle huge amount of data

Manageable and Multi cluster environment



What’s coming… Data Store into Cloud Services

© 2

016

Am

adeu

s S.

A.S

.

_ Closer to our customers •  Reduce the latency

_ Remote operations •  Install appliances within customer premises when

appropriate

_ Multiple Cloud/data-center capability



Text starts with NO BULLET To add pre-formatted

bullet text please use the Increase/Decrease Indent

buttons found in the Top-PowerPoint menu

Takeaways

Next step

_ Follow MongoDB evolution (3.2, 3.4++)

_ Scaling up infrastructure

_ Multi Cloud/data-center

© 2

016

Am

adeu

s S.

A.S

.

Production

_ Volume ramp up _ 24 Physical machines _ Increase to > 100

Shards

Performance

_ Benchmark for performance optimization

_ Find, sort, limit vs Aggregate Framework

_ Lessons learned

Collaboration

_ MongoDB JIRA contributor

_ Amadeus MongoDB community

mongodb europe 2016 - choosing between 100 billion travel options – instant search from amadeus

Data & Analytics