networking (containers) in ultra- low-latency environments
TRANSCRIPT
![Page 3: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/3.jpg)
אכסניאAkh-san-ya\?aksnaja?\ n.(ancientAramaic,fromAncientGreekxénos)1:Hospitality,lodging;2:Host.
![Page 4: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/4.jpg)
אכסניאAkh-san-ya\?aksnaja?\ n.(ancientAramaic,fromAncientGreekxénos)1:Hospitality,lodging;2:Host.
פותחים בכבוד אכסניא:AncientJewishcustomtobeginpublicspeakingbyhonouringorthankingthehosts.
![Page 7: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/7.jpg)
WhoAmI?• Lifeintechbusiness:– 10yrsfinancialservicesIT– 10+yrsconsulWng&training– Somestartupsontheway
• Avid(ifnotverygood)icehockeyplayer• Long-Wmeloverofgreatengineering….whenusedtomakearealdifference
• AtomicInc:– ConsulWng– Training
(not24601)
![Page 9: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/9.jpg)
ALi\leHistory
Summer2015
• FintechX:“Helpuscontainerize!”– Hint:Itisharderthanyou
think…andworthit– Culture/process>technology
• QuesWon:Networking?• Answer:ScienWficmethod
![Page 10: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/10.jpg)
ALi\leHistory
Summer2015
• FintechX:“Helpuscontainerize!”– Hint:Itisharderthanyou
think…andworthit– Culture/process>technology
• QuesWon:Networking?• Answer:ScienWficmethod
Summer2016• GoodpracWcedemands:
1. RedotestswithnewopWonsandversions
2. Maketestsavailable3. Explainitallwell
![Page 12: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/12.jpg)
WhatIs“Ultra-Low”Latency?
1. h\p://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt
“every100msofdelaycosts1%ofsales”[1]
![Page 13: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/13.jpg)
WhatIs“Ultra-Low”Latency?
“extra0.5sinsearchpagegeneraWonWmedroppedtrafficby20%”[2]
1. h\p://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt2. h\p://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
“every100msofdelaycosts1%ofsales”[1]
![Page 14: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/14.jpg)
WhatIs“Ultra-Low”Latency?
“extra0.5sinsearchpagegeneraWonWmedroppedtrafficby20%”[2]
1. h\p://home.blarg.net/%7Eglinden/StanfordDataMining.2006-11-29.ppt2. h\p://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
Not.Even.Close.
“every100msofdelaycosts1%ofsales”[1]
![Page 16: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/16.jpg)
NetworkingWorkloads
• NetworkedWorkloads:“thingsthatdoworkandmusttalk”
• Sameprinciplesforallworkloads:– VMs– Cloud– Serverless– Containers
![Page 21: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/21.jpg)
NetworkingOpWonsDirectMetalmacvlanBridge/vSwitch(noNAT)net=hostSR-IOV
OverlayFlannelWeaveDockerOverlayCalico(IPIP)
WorkloadAwarenessDockerbridge(NAT)
FabricAwarenessCalico(NaWve)
![Page 22: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/22.jpg)
OurTests
WhatWeTested• netperf⇒netserver• UDP&TCPround-robin• Sizes:300,500,1024,2048• NoorchestraWon=complete
control• 50000iteraWons
– Lawoflargenumbers• Latency(Avg,%iles),CPU• DifferenRals,notabsolutes
HowWeTested• .net
– Becauseithadtobemetal– Wickedsmartteam
• Completetestrun– Networkchanges– HardwarevariaWons,errors
h\ps://github.com/deitch/[email protected]
![Page 27: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/27.jpg)
LocalNetworkingSummary• SR-IOVhorriblelatencybutgreatCPU– Holdthatthought…
• net=hostonparwithmetal• macvlanclosestvirtualizedtometal• Restinsamerange:– Latency:5-10𝓊-secoverhead– CPU:negligibledifference
• Calico(IPIP&naWve)&Dockeroverlayslightlymoreperformant
• WatchoutforverylargeTCPpackets
![Page 31: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/31.jpg)
RemoteNetworkingSummary
• Weave(sleeve)addslatencyandCPU– Reasonfor“fastdatapath”
• Again,macvlanbestvirtualized• Alltherest:– Latency:within50𝓊-secofeachother,exceptSR-IOVwithverylargeTCPpackets
– CPU:similar,butkeepaneyeonFlannel(UDP)
![Page 33: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/33.jpg)
SR-IOVSR-IOVdoesnotautomaRcallymeanbeXer
• Switchinnetworkcard• TradeshostCPUforcardprocessor• Qualityvariesdrama5cally– EvenMellanoxfarworselocally
• My2¥:SR-IOVfallsfurtherbehinddueto:– SpeedofiteraWon– Open-source– Sowware+CPU
![Page 34: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/34.jpg)
Headaches(andThanks)• Headaches
– WeaveSYN-(nothing)– etcdis“touchy”– PacketL3networkispowerfulbut…unique
• Macvlan,weave,flannel:allrequiredpingsformac• Se{ngupbridgew/oNAT,Calico,macvlanwas“different”
– SR-IOViscomplicatedandflaky,especiallyMellanox– netperfwithUDPpacketscangetstuck(Calico-ipip)– Andawholelotmore(askmeoffline)
• Andthanks:
– BryanBoreham,AdamHarrisonatweave.works– ZacSmith,Adam,Aaron,Andy,Lucas,everyoneatPacket
![Page 35: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/35.jpg)
Whatelsecouldwedo?Ø OtherhardwaretypesØ OthernetworkfabricsØ Dockermacvlannetworkdriver(experimental)Ø IpvlanØ OtherpacketsizesØ KernelandnetworkstacktuningØ Distant(andVPN)networksØ Othertrafficpa\ernsØ Otherhost-to-hostencrypWonØ Awholelotmore…
![Page 36: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/36.jpg)
Conclusions• SR-IOV:mostoftheWme,justnotworthit• Performance:– Metal(+net=host):alwaysperformsbest– Directnetwork++:macvlanisyourfriend– Others:Roughlysimilar,carefulofWeave(sleeve)
• What’syourusecase?– ULL:Metal/net=host>macvlan>calico>overlay– Everythingelse:Focusonyourarchitectureandskills
Pickintelligently:easier,notsimple
![Page 37: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/37.jpg)
Conclusions• SR-IOV:mostoftheWme,justnotworthit• Performance:– Metal(+net=host):alwaysperformsbest– Directnetwork++:macvlanisyourfriend– Others:Roughlysimilar,carefulofWeave(sleeve)
• What’syourusecase?– ULL:Metal/net=host>macvlan>calico>overlay– Everythingelse:Focusonyourarchitectureandskills
Pickintelligently:easier,notsimple
![Page 38: Networking (Containers) in Ultra- Low-Latency Environments](https://reader033.vdocuments.site/reader033/viewer/2022052707/58a1ac7b1a28abbf0b8b9ec8/html5/thumbnails/38.jpg)
QuesWonsandhelp:@[email protected]