mariadb - percona · mysvc start --rid container#03 ... hpux ip fs vg sync hpvm srp openha freebsd...
TRANSCRIPT
1
MariaDB
2
Gcloud - Maxscale - MariaDB NoSQL - DockerGcloud - axscale - MariaDB ColumnStore - Docker
Gcloud - Maxscale - MariaDB Replication Manager - DockerGcloud - axscale - MariaDB Cluster - Docker
1.
2.
3.
Agent Stack••••••
Collector Stack••
••
Step 2 : Agent bootstrap•
wget -O/tmp/opensvc.deb http://repo.opensvc.com/deb/currentsudo dpkg -i /tmp/opensvc.deb
•sudo nodemgr set --param node.repopkg --value http://repo.opensvc.com/
sudo nodemgr set --param node.repocomp --value http://repo.opensvc.com/compliance/
sudo nodemgr set --param node.dbopensvc --value https://collector.opensvc.com
Step 2 : Agent bootstrap is generating a node conf file
•[node]repopkg = http://repo.opensvc.com/repocomp = http://repo.opensvc.com/compliance/dbopensvc = https://collector.opensvc.com/feed/default/call/xmlrpcdbcompliance = https://collector.opensvc.com/init/compliance/call/xmlrpchost_mode = PRDuuid = d5bccb78-a2b2-4809-b036-ac7c7bfa7101
[compliance]auto_update = trueschedule = @1440
[stats]schedule = @60
[gcedisks]scheduler = @120
Step 2 : Agent bootstrap•
sudo nodemgr register --user [email protected]
sudo nodemgr set --param node.uuid --value d5bccb78-a2b2-4809-b036-ac7c7bfa7101
sudo nodemgr pushasset sudo nodemgr pushdiskssudo nodemgr pushpkgsudo nodemgr pushpatchsudo nodemgr checkssudo nodemgr sysreport
Step 3 : Node configuration
sudo nodemgr compliance fix --attach --moduleset mariadb.node
sudo nodemgr compliance attach --moduleset mariadb.node
sudo nodemgr compliance check --moduleset mariadb.node
sudo nodemgr compliance fix --moduleset mariadb.node
Step 4 : Service deployment•
sudo svcmgr -s mysvc pull
•sudo svcmgr -s mysvc provision
Run ops : Reconfigure MariaDB•
mysvc --cluster compliance fix
•mysvc --cluster stopcontainer; mysvc --cluster startcontainer;
•
Trouble Shooting Services •
mysvc --cluster print status
•svcmon -cv
•mysvc --cluster docker ps -a
•mysvc docker logs mysvc.container.03
•mysvc docker exec -ti mysvc.container.03 /bin/bash
•mysvc start --rid container#03
•mysvc logs or /var/log/opensvc/<mysvc>.log*
•nodemgr collector cli -- get /services/spdsvc01/resources --filter "res_type ip%" --filter "res_desc %10.0.0.33%" --filter "res_status up" --props res_desc,res_status,nodes.nodename --meta false
GCloud how to •
• gcloud compute instances set-machine-type mariadb01 --machine-type custom-1-4096 --zone europe-west1-b
13
Linux
iploopfsbind mountslvm2 vgveritas vgsync
lxcopenvzxenkvmvboxovmesxdocker
openhaheartbeatdKeepalived lvm2
Snaps Dds Netapp snapmirrorSymmetrix SRDF-SHP 3Par Remote CopyDatacore replicationDrbdBtrfsZfsJfsSymmetrix cloneHP EVA snapDatacore snapIBM DS8xxx snapNEC ISM, Bull Optima snapCeph RADOS snap
Solaris
iplofszfs poolveritas vgsync
zoneldomxenvbox
openha
HpUx
ipfsvgsync
hpvmsrp openha
FreeBSD
ipmdfsnullfszfs poolsync
jail UNTESTED
GCloud Docker ✔ ✔ ✔ FUTURE ✔
Amazon Docker FUTURE FUTURE FUTURE FUTURE FUTURE
Azure Docker FUTURE FUTURE FUTURE FUTURE FUTURE
Openstack Docker FUTURE FUTURE FUTURE FUTURE FUTURE
•
••
○○○○
14
•
•••
○○○○○
15
MariaDBReplication
Manager
■
■
16
■
●
●
●
17
•• •
• 18
MariaDB & Maxscale Fix size 66 dockers pack in minimum 4 VMShare Disk (POD) for a pair of 1 spider + 1 backend
•
•front
•
Spider Sharding Proxy•
•
•
•front
Shard single table in spdc schema••
+------------+---------------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+------------+---------------------+------+-----+---------+-------+| id | bigint(20) unsigned | NO | PRI | NULL | || collection | bigint(20) unsigned | NO | PRI | NULL | || content | mediumblob | YES | | NULL | || inum | bigint(20) | YES | MUL | NULL | || istr | varbinary(40) | YES | MUL | NULL | |+------------+---------------------+------+-----+---------+-------+
‘t’ linked to local tables per collection •••
| t_c00 || t_c01 || t_c02 || t_c03 |
Each collection table linked to local shard tables in a schema per VM (blackhole)
| spdc34 || spdc36 || spdc38 || spdc40 || spdc42 || spdc44 |. . . . . . . . . . . . . . . . . . . . . . | spdc90 || spdc92 || spdc94 || spdc96 |
Each collection is exposed as a viewCREATE VIEW `dim_pays` AS SELECT column_get(t_ro.content, 'A2CODE' as char) AS A2CODE, spdc.t_ro.istr AS `A3CODE`, /*PK*/ column_get(t_ro.content, 'N3CODE' as char) AS N3CODE, column_get(t_ro.content, 'LIB' as char) AS LIB, column_get(t_ro.content, 'CONTINENT' as char) AS CONTINENT FROM t_ro WHERE t_ro.collection = cast(conv(left(md5('bigdata.dim_pays'),16),16,10) as unsigned)
Multi level spider layer for read to divide number of partitions to manage •
CREATE TABLE `t_ro` ENGINE=SPIDER DEFAULT CHARSET=utf8 COMMENT='mbk "1", mkd "1", alc "1", msi "144" '
(PARTITION p00 VALUES IN (0) COMMENT = 'srv "t_r1d00c00 t_l1d00c00", table "t_c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p01 VALUES IN (1) COMMENT = 'srv "t_r1d00c01 t_l1d00c01", table "t_c01", lst "0 0" ' ENGINE = SPIDER, PARTITION p02 VALUES IN (2) COMMENT = 'srv "t_r1d00c02 t_l1d00c02", table "t_c02", lst "0 0" ' ENGINE = SPIDER, PARTITION p03 VALUES IN (3) COMMENT = 'srv "t_r1d00c03 t_l1d00c03", table "t_c03", lst "0 0" ' ENGINE = SPIDER
•
CREATE TABLE `t_c00` ENGINE=SPIDER DEFAULT CHARSET=utf8 COMMENT='mbk "1", mkd "1", alc "1", msi "49" '
(PARTITION p00 VALUES IN (0) COMMENT = 'srv "t_r1d00c00 t_l1d00c00", table "t_d00c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p01 VALUES IN (1) COMMENT = 'srv "t_r1d01c00 t_l1d01c00", table "t_d01c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p02 VALUES IN (2) COMMENT = 'srv "t_r1d02c00 t_l1d02c00", table "t_d02c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p03 VALUES IN (3) COMMENT = 'srv "t_r1d03c00 t_l1d03c00", table "t_d03c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p04 VALUES IN (4) COMMENT = 'srv "t_r1d04c00 t_l1d04c00", table "t_d04c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p05 VALUES IN (5) COMMENT = 'srv "t_r1d05c00 t_l1d05c00", table "t_d05c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p06 VALUES IN (6) COMMENT = 'srv "t_r1d06c00 t_l1d06c00", table "t_d06c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p07 VALUES IN (7) COMMENT = 'srv "t_r1d07c00 t_l1d07c00", table "t_d07c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p08 VALUES IN (8) COMMENT = 'srv "t_r1d08c00 t_l1d08c00", table "t_d08c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p09 VALUES IN (9) COMMENT = 'srv "t_r1d09c00 t_l1d09c00", table "t_d09c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p10 VALUES IN (10) COMMENT = 'srv "t_r1d10c00 t_l1d10c00", table "t_d10c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p11 VALUES IN (11) COMMENT = 'srv "t_r1d11c00 t_l1d11c00", table "t_d11c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p12 VALUES IN (12) COMMENT = 'srv "t_r1d12c00 t_l1d12c00", table "t_d12c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p13 VALUES IN (13) COMMENT = 'srv "t_r1d13c00 t_l1d13c00", table "t_d13c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p14 VALUES IN (14) COMMENT = 'srv "t_r1d14c00 t_l1d14c00", table "t_d14c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p15 VALUES IN (15) COMMENT = 'srv "t_r1d15c00 t_l1d15c00", table "t_d15c00", lst "0 0" ' ENGINE = SPIDER)
On service front • DO_NOT_DELETE_ME
/spdsvc02/bootstrap# lsbootstrap.sh init_backend init_repli.sh run_middle08.sh sp_frontwrite.sqlDO_NOT_DELETE_ME init_front my_change_master_to run_middle16.sh sp_frontwritetable.sqlfront_file_bootstrap_table.cnf init_middle08 my_ping sp_backend.sql sp_middle08.sqlfront_file_bootstrap_table.kv.cnf init_middle16 run_backend.sh sp_frontread.sql sp_middle16.sqlfront_file_bootstrap_table.sql init_repli run_front.sh sp_frontwritegroup.sql
••
cat front_file_bootstrap_table.kv.cnf ddl=(id bigint unsigned not null primary key auto_increment, collection bigint unsigned , content mediumblob, inum bigint, istr varbinary(40), key idx1(inum ), key idx2(istr))ddlfront=(id bigint unsigned not null, collection bigint unsigned , content mediumblob, inum bigint, istr varbinary(40), key idx1(inum ), key idx2(istr))collection=collectionid=id tbl=tdb=spdcengine=tokudb
•./cluster_check.sh
Boosting •••
set transaction isolation level read uncommitted ; MariaDB [xxx]> select count(*) from xxx ;+----------+| 19598363 |+----------+
Disable background statistics to fetch once every 3600 queries
•| spider_sts_bg_mode | 0 || spider_sts_interval | 3600 || spider_sts_mode | 1 || spider_sts_sync | 1 |
•| spider_crd_bg_mode | 0 || spider_crd_interval | 3600 || spider_crd_mode | 1 || spider_crd_sync | 1 |
●
spider_casual_read=1
Load data••
⅙ to ⅛ compression
••
Small Downtime1.2.
3.
4.
No Downtime with rolling upgrade1.2.
3.
Speed ••
––
Memory ••
Storage Engine •
•
•
GCloud Sizing••
27
Q&A
Thank You
28