go lang migrating billions of documents

GoLang: learning by doing

Case: Migrating nearly 2 billion documents

@jonatasdp

Go is awesome!

2 weeks experience

How it starts?

the challenge: migrate ~1.8 billion from a mongo server to another

WITHOUT STOP!

downtime migrationø

Billion of documents1.8

200K / MINUTE

1Million / 5 MIN

20Millions / hour

400Millions / DAY

5 days MIGRATING

weeks experience2 ● syntax● Basics

●Mgo● TERMUI● CLI

Migration algorithm● Two mongo connections: <origin> and <destiny>● <origin> receive records● <destiny> insert records● When record is on <destiny>● Remove from <origin>

FIRST TASTE● START WITHOUT TESTS :(● LEARNING ABOUT REFERENCES VS VARS● Structs for everything● Interface Comprehension● GO Routines● Locks and Mutex● Channels

MGO EXAMPLESimple insert

MGO EXAMPLEIs Not generic

structs

Structs & Interfaces

Interface {}

"Typecasting" the Interface

GOROUTINES

no wait group = 1.2M rpm

GOROUTINES + wait group = 200k rpm

State less GOROUTINESWith channels

GOROUTINES + channels

Channels

Channels

STRUCT METHODS

TERM UI

TERM UI

TERM UI colors

CLIgithub.com/codegagsta/cli

CLIgithub.com/codegagsta/cli

CLI

What I learned

● simplicity● naturally fast● easy to write async code● easy to maintain (mongo-migration#1)

OPEN SOURCE

https://github.com/ResultadosDigitais/mongo-migration

Thank you!

http://shipit.resultadosdigitais.com.br

@jonatasdp

http://ideia.me