2600hz - tuning kazoo to 10,000 handsets - kazoocon 2015
TRANSCRIPT
PRESENTED BY:
Tuning Kazoo to 10,000+ handsets
Luis Azedo
@kazoocon@kazoocon
About the Topic
• People love to talk about scale.
• Customer’s usage patterns are different.
• No cookie-cutter solution.
@kazoocon@kazoocon
Design Overview
KAZOO
@kazoocon@kazoocon
General linux considerations
• File Descriptors
• Sockets
• Processes
• Limits
• Logs
#kazoocon14
Let’s Talk AboutEach Component
@kazoocon@kazoocon
Couch
• Overprovisioning is good• Plan for # of accounts you’ll have in
1-2 years• Z, N, W parameters
• Set these when you first start• Z = Number of Datacenters / Zones• N = Number of copies• W = Number of writes before it’s
“good enough” to move on• Db tool can be used later to redo
@kazoocon@kazoocon
Freeswitch
• Bare metal vs VMs / docker• 5 x 200 calls vs 1 x 1000 calls• Sqlite vs mysql (memory locks)• mod_kazoo filters
@kazoocon@kazoocon
Kamailio
• TCP Workers• Shared Memory• db_text vs mysql (in memory)• Configure Zones• AMQP workers / consumers• Use ASYNC when possible
@kazoocon@kazoocon
Kamailio
#!substdef "!MY_AMQP_URL_1!kazoo://user:[email protected]:5672!g"#!substdef "!MY_AMQP_URL_2! kazoo://user:[email protected]:5672!g”#!substdef "!MY_AMQP_URL_3! kazoo://user:[email protected]:5672!g”
#!substdef "!MY_AMQP_URL_4!zone=lx2;kazoo://user:[email protected]:5672!g"#!substdef "!MY_AMQP_URL_5! zone=lx2;kazoo://user:[email protected]:5672!g”#!substdef "!MY_AMQP_URL_6! zone=lx2; kazoo://user:[email protected]:5672!g”
#!substdef "!MY_AMQP_URL_7!zone=lx3;kazoo://user:[email protected]:5672!g”
modparam("kazoo", "amqp_consumer_processes", 8);modparam("kazoo", "amqp_consumer_workers", 32);
Zone Configuration
@kazoocon@kazoocon
Kamailioroute[KAZOO_AUTHORIZATION]{ xlog("L_INFO", "$ci|log|check $adn"); $var(amqp_payload_request) = '{"Event-Category" : "directory" , "Event-Name" : "authn_req", "Method" : "REGISTER", "Auth-Nonce" : "$adn", "Auth-Realm" : "$fd", "Auth-User" : "$fU", "From" : "$fu", "To" : "$tu" }'; $var(amqp_routing_key) = "authn.req." + $(fd{kz.encode}); $avp(kz_timeout) = 2500; if(kazoo_async_query("callmgr", $var(amqp_routing_key), $var(amqp_payload_request), "KAZOO_AUTHORIZATION_OK", "KAZOO_AUTHORIZATION_ERROR") != 1) { xlog("L_INFO", "$ci|log|failed to send Kazoo query for authentication credentials for $Au $si:$sp"); append_to_reply("Retry-After: 60\r\n"); send_reply("500", "Retry Later"); exit; }}
route[KAZOO_AUTHORIZATION_OK]{ $var(password) = $(kzR{kz.json,Auth-Password}); $var(nonce) = $adn; $xavp(ulattrs=>custom_channel_vars) = $(kzR{kz.json,Custom-Channel-Vars}); xlog("L_INFO", "$ci|log|authenticating $Au via Kazoo query response"); route(CHECK_AUTHORIZATION);}route[KAZOO_AUTHORIZATION_ERROR]{ xlog("L_INFO", "$ci|log|failed to query Kazoo for authentication credentials for $Au $si:$sp"); append_to_reply("Retry-After: 60\r\n"); send_reply("500", "Retry Later"); exit;}
@kazoocon@kazoocon
Whapps
3 Servers
2 Servers
One Servercrossbar, registrar, callflow, fax
crossbar, callflow, fax
crossbar callflow, fax
registrar
registrar
@kazoocon@kazoocon
REST
SmtpCouch
HAProxy
@kazoocon@kazoocon
Rabbit
• Thresholds• Granular statistics (10% gain)• Cluster rabbit• Federation & Zones
@kazoocon@kazoocon
Rabbit Clustering
• High available queues• Publish & consume
from any node• Policies for replication• LAN Only
@kazoocon@kazoocon
Rabbit Federation
• Federation plugin
primary Zone a Zone b
• Kazoo Federation• Granular control
@kazoocon@kazoocon
REGISTER 401 REGISTER AUTHN_REQ COUCH AUTHN_RESP USRLOC REG_SUCCESS ECALLMGR
registration scenario
2 kamailio , 2 x kazoo registrar instances with 4 listeners, 2 x rabbit (clustered)
In this scenario, we are showing 5,800+ registrations per second.
Most phones register every 360 seconds, so:
360 x 5800 = 2,088,000 phones(THEORETICAL!!!)
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbitapps
HAProxy
Couch Couch
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbitapps
HAProxy
Couch Couch
FreeSWITCH
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbitapps
HAProxy
Couch Couch
FreeSWITCH
apps
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
Kamailio
FreeSWITCH
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
Kamailio
FreeSWITCH
apps apps
apps
apps
Rabbit
@kazoocon@kazoocon
PROGRESSING CLUSTER
Kamailio
FreeSWITCH
Couch
Rabbit
apps
HAProxy
Couch Couch
FreeSWITCH
apps
Rabbit
Kamailio
FreeSWITCH
apps apps
apps
apps
Rabbit
Kamailio
FreeSWITCH
Rabbit
apps
HAProxy
Start a New Zone
@kazoocon@kazoocon
The World & Beyond
“A thing called open-source kazoo platform actually make it possible.”
Thank You!
@kazoocon
Thank You!
@kazoocon