go lang migrating billions of documents
TRANSCRIPT
GoLang: learning by doing
Case: Migrating nearly 2 billion documents
@jonatasdp
Go is awesome!
2 weeks experience
How it starts?
the challenge: migrate ~1.8 billion from a mongo server to another
WITHOUT STOP!
downtime migrationø
Billion of documents1.8
200K / MINUTE
1Million / 5 MIN
20Millions / hour
400Millions / DAY
5 days MIGRATING
weeks experience2 ● syntax● Basics
●Mgo● TERMUI● CLI
Migration algorithm● Two mongo connections: <origin> and <destiny>● <origin> receive records● <destiny> insert records● When record is on <destiny>● Remove from <origin>
FIRST TASTE● START WITHOUT TESTS :(● LEARNING ABOUT REFERENCES VS VARS● Structs for everything● Interface Comprehension● GO Routines● Locks and Mutex● Channels
MGO EXAMPLESimple insert
MGO EXAMPLEIs Not generic
structs
Structs & Interfaces
Interface {}
"Typecasting" the Interface
GOROUTINES
no wait group = 1.2M rpm
GOROUTINES + wait group = 200k rpm
State less GOROUTINESWith channels
GOROUTINES + channels
Channels
Channels
STRUCT METHODS
TERM UI
TERM UI
TERM UI colors
CLIgithub.com/codegagsta/cli
CLIgithub.com/codegagsta/cli
CLI
What I learned
● simplicity● naturally fast● easy to write async code● easy to maintain (mongo-migration#1)
OPEN SOURCE
https://github.com/ResultadosDigitais/mongo-migration
Thank you!
http://shipit.resultadosdigitais.com.br
@jonatasdp
http://ideia.me