Introduction to Troubleshooting PerformanceWhat Affects Query Execution?

Sveta SmirnovaPrincipal Support Engineer

April, 7, 2016

Table of Contents

•Terms of conditions•The query•Controlling optimizer•Concurrency•Hardware•MySQL Options

2 www.percona.com

Terms of conditions

3 www.percona.com

”Slow” is relative• Mind your data

• 1,000,000 rows• 1 row

• Mind use case• Popular website• Admin interface• Weekly cron job

• Mind location• Server, used my multiple connections• Dedicated for OLAP queries

4 www.percona.com

Is MySQL the weak link?• MySQL can run together with other applicationssveta@Thinkie:~$ top -b -n 1 | head -n 20

top - 12:31:58 up 7 days, 16:54, 6 users, load average: 1,04 ...

Tasks: 292 total, 1 running, 290 sleeping, 1 stopped, 0 zombie

%Cpu(s): 18,6 us, 3,3 sy, 0,1 ni, 76,7 id, 1,3 wa, 0,0 hi...

KiB Mem: 16117608 total, 15956604 used, 161004 free, 374776 buffers

KiB Swap: 0 total, 0 used, 0 free. 9316288 cached

PID USER NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

31119 sveta 0 1032316 100608 12056 S 104,1 0,6 0:24.75 mysqld

26014 sveta 0 4645964 2,136g 111084 S 26,0 13,9 487:03.82 firefox

• Query can run fast, but application can be slow• How to find if MySQL is a weak link?<app code>

$start_time = microtime(true);

$result = $mysqli->query($query); - Running query

$end_time = microtime(true);

print("Query took " . ($end_time - $start_time) . " microseconds");

5 www.percona.com

Is MySQL the weak link?• MySQL can run together with other applications• Query can run fast, but application can be slowif ($result = $mysqli->query($query)) { - Database finished its job!

while ($row = $result->fetch_row()) {

$handle = fopen(’my_file.txt’);

fputs($handle, $row);

fclose($handle);

}

...

• How to find if MySQL is a weak link?<app code>

$start_time = microtime(true);

$result = $mysqli->query($query); - Running query

$end_time = microtime(true);

print("Query took " . ($end_time - $start_time) . " microseconds");

5 www.percona.com

Is MySQL the weak link?

• MySQL can run together with other applications

• Query can run fast, but application can be slow

• How to find if MySQL is a weak link?<app code>

$start_time = microtime(true);

$result = $mysqli->query($query); - Running query

$end_time = microtime(true);

print("Query took " . ($end_time - $start_time) . " microseconds");

5 www.percona.com

What affects performance?

• Query

• Concurrency

• Hardware

• MySQL configuration

6 www.percona.com

The query

7 www.percona.com

Heart of the application

• You communicate with database using queries

• Even if you use NoSQL interface• They are not SQL queries, but still queries

• Data, which you request, matters

• 1,000,000,000 rows• 1 row

8 www.percona.com

Troubleshooting vs performance tuning• Query can use bad practice

• SELECT * FROM many columns table• Table can be defined in weird way

• No normalization• Dozens columns like ”Characteristic N”

• Troubleshooting has to live with these• There can be good reasons

• Legacy application• De-normalization to speed up SELECTs

• Usually with price of more locking and slow updates

• You name it9 www.percona.com

Query execution workflow

Query sent

Connection Pool: Authentication, Caches; SQL interface; Parser

Optimizer

Storage engines

Hardware

10 www.percona.com

Controlling optimizer

11 www.percona.com

Pre-requisites• Find queries to optimize

• While designing application• In slow query log• In Performance Schemamysql> select sql_text, (timer_end - timer_start)/1000000000 as time \

-> from events_statements_history_long \

-> where (timer_end - timer_start) > 4*10000000000 \

-> and event_name=’statement/sql/select’ \G

*************************** 1. row ***************************

sql_text: select count(*) from titles join employees using (emp_no) where title=’Senior Engineer’

time: 4493.3863

1 rows in set (0.01 sec)

• Make sure they are sane• Or you really need them

12 www.percona.com

Pre-requisites• Find queries to optimize

• While designing application• In slow query log• In Performance Schema• In sys schema

• statement analysis• statements with full table scans• statements with runtimes in 95th percentile• statements with sorting• statements with temp tables• statements with errors or warnings

• Make sure they are sane• Or you really need them

12 www.percona.com

Pre-requisites• Find queries to optimize• Make sure they are saneSELECT COUNT(*) FROM huge_table LEFT JOIN other_table USING(id) \

LEFT JOIN another_table USING(id) \

LEFT JOIN one_more_table USING(id) \

LEFT JOIN and_yet_more_table USING(id)

WHERE another_field = ’foo’;

vs

SELECT COUNT(*) FROM huge_table WHERE another_field = ’foo’;

• Or you really need them12 www.percona.com

MySQL Indexes

• B-Tree (Mostly)• Fractal Tree• R-Tree (Spatial)• Hash (Memory SE)• Engine-dependent d001

d003d008d009

d003******d009******d008******d009******d001******d003******d009******d008******d009******d001******d008******d008******d001******

13 www.percona.com

Controlling indexes

• SHOW INDEX• Displays index statistics as seen by optimizer• Most important filed is Cardinality• Should be equal to number of distinct values

• Up to 10 % difference is OK

• ANALYZE• Updates index statistics• Recommended to use together with InnoDB

persistent statistics

14 www.percona.com

Use case: Is statistic actual?mysql> show index from sbtest1;

+---------+----------+-------------+-------------+

| Table | Key_name | Column_name | Cardinality |

+---------+----------+-------------+-------------+

| sbtest1 | PRIMARY | id | 98285 |

| sbtest1 | k_1 | k | 49142 |

+---------+----------+-------------+-------------+

mysql> select count(distinct id), count(distinct k) from sbtest1;

+--------------------+-------------------+

| count(distinct id) | count(distinct k) |

+--------------------+-------------------+

| 100000 | 17598 |

+--------------------+-------------------+

1 row in set (0,58 sec)15 www.percona.com

Estimates: EXPLAIN

mysql> explain extended select * from t1 join t2 where t1.int_key=1;

+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+

| id | select_type | table | type | possible_keys | key | key_len | ref | rows | f... | Extra |

+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+

| 1 | SIMPLE | t1 | ref | int_key,ik | int_key | 5 | const | 4 | 100. | NULL |

| 1 | SIMPLE | t2 | index | NULL | pk | 9 | NULL | 6 | 100. | Using index; |

Using join buffer |

(Block Nested Loop) |

+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+

2 rows in set, 1 warning (0.00 sec)

Note (Code 1003): /* select#1 */ select ‘test‘.‘t1‘.‘pk‘ AS ‘pk‘,‘test‘.‘t1‘.‘int_key‘ AS ‘int_key‘,‘test‘.‘t2‘.‘pk‘

AS ‘pk‘,‘test‘.‘t2‘.‘int_key‘ AS ‘int_key‘ from ‘test‘.‘t1‘ join ‘test‘.‘t2‘ where (‘test‘.‘t1‘.‘int_key‘ = 1)

Number of select

Select type

Tables, for which information is printed

How data is accessed

Possible keys

Key, which was actually used

Length of the key

Which columns were compared with the index

Number of examined rows

% of filtered rowsrows x filtered / 100 — number of rows,which will be joined with another table

Additional information

Table, for which information is printedProduct of rows here: how many rows in all tables will be accessed

For this example estimated value is 4*6 = 24

Actual (optimized) query as executed by MySQL Server

16 www.percona.com

EXPLAIN extensions

• EXTENDEDmysql> explain extended select * from t1 join t2 where t1.int_key=1;

...

mysql> SHOW WARNINGS;

Note (Code 1003): /* select#1 */ select ‘test‘.‘t1‘.‘pk‘

AS ‘pk‘,‘test‘.‘t1‘.‘int_key‘ AS ‘int_key‘,

‘test‘.‘t2‘.‘pk‘ AS ‘pk‘,‘test‘.‘t2‘.‘int_key‘ AS ‘int_key‘

from ‘test‘.‘t1‘ join ‘test‘.‘t2‘ where (‘test‘.‘t1‘.‘int_key‘ = 1)

• PARTITIONS• FORMAT=JSON

17 www.percona.com

EXPLAIN extensions

• EXTENDED• PARTITIONSmysql> explain partitions select email, account_id from users

-> where email=’sveta@mysql.com’ and account_id=896;

+----+-------------+-------+------------+-------+----------------------+

| id | select_type | table | partitions | type | possible_keys |

+----+-------------+-------+------------+-------+----------------------+

| 1 | SIMPLE | users | p0 | const | idx_email_account_id |

+----+-------------+-------+------------+-------+----------------------+

1 row in set (0,03 sec)

• FORMAT=JSON17 www.percona.com

EXPLAIN extensions• EXTENDED• PARTITIONS• FORMAT=JSONmysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\G

*************************** 1. row ***************************

EXPLAIN: { "used_key_parts": [

"query_block": { "Host",

"select_id": 1, "User"

"table": { ],

"table_name": "user", "key_length": "228",

"access_type": "index", "rows": 8,

"key": "PRIMARY", "filtered": 100,

"using_index": true } } }

17 www.percona.com

Actual execution• Status variables ’Handler %’mysql> select count(*) from employees join titles using(emp_no) ...

*************************** 1. row ***************************

count(*): 97750

1 row in set (3.24 sec)

mysql> SHOW STATUS LIKE ’Handler_read_%’;

+----------------------------+--------+

| Variable_name | Value |

+----------------------------+--------+

| Handler_read_first | 1 |

| Handler_read_key | 300027 |

| Handler_read_last | 0 |

| Handler_read_next | 397774 |

...

• INFORMATION SCHEMA.OPTIMIZER TRACE• PERFORMANCE SCHEMA.EVENTS STATEMENTS *• PERFORMANCE SCHEMA.EVENTS STAGES *• ANALYZE statement - MariaDB only!

18 www.percona.com

Actual execution

• Status variables ’Handler %’• INFORMATION SCHEMA.OPTIMIZER TRACE

• join preparation, join optimization, join execution• considered execution plans, refine plan, more

• PERFORMANCE SCHEMA.EVENTS STATEMENTS *• PERFORMANCE SCHEMA.EVENTS STAGES *• ANALYZE statement - MariaDB only!

18 www.percona.com

Actual execution• Status variables ’Handler %’• INFORMATION SCHEMA.OPTIMIZER TRACE• PERFORMANCE SCHEMA.EVENTS STATEMENTS *

• events statements * andprepared statements instances: important fields

• CREATED TMP DISK TABLES• CREATED TMP TABLES• SELECT FULL JOIN• SELECT RANGE CHECK• SELECT SCAN• SORT MERGE PASSES• SORT SCAN

• PERFORMANCE SCHEMA.EVENTS STAGES *• ANALYZE statement - MariaDB only!

18 www.percona.com

Actual execution• Status variables ’Handler %’• INFORMATION SCHEMA.OPTIMIZER TRACE• PERFORMANCE SCHEMA.EVENTS STATEMENTS *• PERFORMANCE SCHEMA.EVENTS STAGES *

• EVENT NAME LIKE• ’stage/sql/%tmp%’• ’stage/sql/%lock%’• ’stage/%/Waiting for%’• ’stage/sql/end’• ’stage/sql/freeing items’• ’stage/sql/Sending data’• ’stage/sql/cleaning up’• ’stage/sql/closing tables’

• ANALYZE statement - MariaDB only!18 www.percona.com

Actual execution

• Status variables ’Handler %’• INFORMATION SCHEMA.OPTIMIZER TRACE• PERFORMANCE SCHEMA.EVENTS STATEMENTS *• PERFORMANCE SCHEMA.EVENTS STAGES *• ANALYZE statement - MariaDB only!

• Two new columns• r rows - actual number of read rows• r filtered - actual number of filtered rows

18 www.percona.com

Concurrency

19 www.percona.com

Common concurrency issues

• Query or transaction waits a lock, held by another one

• Fight for system resources

• Symptoms• Many processes are waiting• You see query in the slow log, but it runs fast if

single-thread environment• Query execution time varies

20 www.percona.com

Lock types and transactions

• Lock types• Levels

• MDL• Table-level• Row-level

• What do they lock• Read locks

• Block writes

• Write locks• Block reads and writes

• Transactions• Server-level

• MDL locks• Table locks

• Engine-level• Table locks• Row locks

• AUTOCOMMIT• supported

21 www.percona.com

Locks diagnostic• Table-level

• PROCESSLIST: ”Waiting for table lock”• P S.TABLE HANDLES

• Row-level• InnoDB monitors

• SHOW ENGINE INNODB STATUS

• Tables in INFORMATION SCHEMA• P S.EVENTS TRANSACTIONS• Option –innodb print all deadlocks

• MDL• PROCESSLIST: ”Waiting for metadata lock”• P S.METADATA LOCKS

22 www.percona.com

Hardware

23 www.percona.com

What affects MySQL Performance?

• RAM

• CPU

• IO (Disk)

• Ntework

24 www.percona.com

Common issues with hardware• Too high resource usage

• MySQL is swapping• Disk IO is near the limit of its capacity• Network bandwidth is too small• More

• Too low resource usage• Single CPU core used on 24 cores system• Memory usage is low while disk IO is high due to

temporary tables• More

25 www.percona.com

Tools to work with hardware

• OS-level tools• Performance Schema

• Little support in 5.5• More in 5.6• Even more in 5.7

• Engine status• Limited

26 www.percona.com

Tools to work with hardware• For true sysadmins

• Consult Brendan D. Gregg’s Linux Performancewebpage

• Image License: Attribution-ShareAlike 4.0

• Necessary minimum• Following slides

• pt-stalk• For automatization• To provide for Percona Support for future analysis• Does not use Performance Schema!

27 www.percona.com

Tools to work with hardware

• For true sysadmins• Consult Brendan D. Gregg’s Linux Performance

webpage• Necessary minimum

• Following slides• pt-stalk

• For automatization• To provide for Percona Support for future analysis• Does not use Performance Schema!

27 www.percona.com

Memory

• What uses memory in MySQL

• Buffers

• Temporary tables

• Internal structures - out of user control!

• OS did not show memory as freed yet?

28 www.percona.com

Memory diagnostic before 5.7

• free• top• vmstat• Investigation

• Total size of buffers• Number of temporary tables• Number of parallel connections

• There was no way to know how exactly memory wasallocated

29 www.percona.com

Memory diagnostic in 5.7mysql> select thread_id tid, user, current_allocated ca, total_allocated

-> from sys.memory_by_thread_by_current_bytes;

+-----+-------------------------+-------------+-----------------+

| tid | user | ca | total_allocated |

+-----+-------------------------+-------------+-----------------+

| 1 | sql/main | 2.53 GiB | 2.69 GiB |

| 150 | root@127.0.0.1 | 4.06 MiB | 32.17 MiB |

| 146 | sql/slave_sql | 1.31 MiB | 1.44 MiB |

| 145 | sql/slave_io | 1.08 MiB | 2.79 MiB |

...

| 4 | innodb/io_log_thread | -2880 bytes | 132.38 KiB |

| 72 | innodb/io_write_thread | -7632 bytes | 1.10 MiB |

+-----+-------------------------+-------------+-----------------+

145 rows in set (2.65 sec)30 www.percona.com

Disk diagnostics

• df

• iostat

• ls -l /proc/{PID OF MYSQLD}/fd• InnoDB status

• performance schema.table io waits %

• sys.io %

31 www.percona.com

Disk diagnostics• InnoDB statusmysql> show engine innodb status\G

FILE I/O

--------

I/O thread 0 state: waiting for i/o request (insert buffer thread)

I/O thread 1 state: waiting for i/o request (log thread)

I/O thread 2 state: waiting for i/o request (read thread)

I/O thread 5 state: waiting for i/o request (write thread)

Pending normal aio reads: [0, 0] , aio writes: [0, 0] ,

ibuf aio reads:, log i/o’s:, sync i/o’s:

Pending flushes (fsync) log: 0; buffer pool: 0

11468 OS file reads, 102 OS file writes, 17 OS fsyncs

11.80 reads/s, 16384 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

...31 www.percona.com

Disk diagnostics

• InnoDB status

--------------

ROW OPERATIONS

--------------

0 queries inside InnoDB, 0 queries in queue

0 read views open inside InnoDB

Process ID=26676, Main thread ID=140424906065664, state: sleeping

Number of rows inserted 19, updated 0, deleted 0, read 5843307

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 1584.23 reads/s

31 www.percona.com

Disk diagnosticsmysql> select OBJECT_SCHEMA, OBJECT_NAME, INDEX_NAME \

-> from table_io_waits_summary_by_index_usage where COUNT_STAR=0;

+---------------+--------------+------------+

| OBJECT_SCHEMA | OBJECT_NAME | INDEX_NAME |

+---------------+--------------+------------+

| employees | departments | PRIMARY |

| employees | departments | dept_name |

| employees | dept_emp | PRIMARY |

| employees | dept_emp | emp_no |

| employees | dept_emp | dept_no |

| employees | dept_manager | PRIMARY |

| employees | dept_manager | emp_no |

...

31 www.percona.com

Disk diagnostics• sys.io %mysql> select event_name, count_read, total_read, total_requested \

-> from io_global_by_wait_by_bytes;

+-------------------------+------------+------------+-----------------+

| event_name | count_read | total_read | total_requested |

+-------------------------+------------+------------+-----------------+

| innodb/innodb_data_file | 17349 | 274.36 MiB | 289.22 MiB |

| sql/binlog | 16696 | 130.42 MiB | 130.42 MiB |

| sql/io_cache | 1556 | 37.61 MiB | 100.49 MiB |

| sql/FRM | 1137 | 626.03 KiB | 626.03 KiB |

| innodb/innodb_log_file | 8 | 132.50 KiB | 141.50 KiB |

| sql/ERRMSG | 3 | 72.23 KiB | 72.23 KiB |

...

31 www.percona.com

Disk diagnostics• sys.io %• Other useful views

• host summary by file io• host summary by file io type• io by thread by latency• io global by file by [bytes latency] - Per-file• io global by file by latency• io global by wait by [bytes latency]• latest file io - Latest IO operations• user summary by file io• user summary by file io type

31 www.percona.com

CPU diagnostics• top%Cpu(s): 15.2 us, 2.7 sy, 1.8 ni, 79.4 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

26676 sveta 20 0 1963204 155108 23300 S 100.0 0.960 0:43.73 mysqld

• iostat -xavg-cpu: %user %nice %system %iowait %steal %idle

14.47 1.77 2.54 0.94 0.00 80.28

• pssveta@thinkie> ps -eo pid,user,comm,pcpu,pmem,vsz | grep mysqld

26676 sveta mysqld 1.4 0.9 1963204

26678 sveta mysqld 0.0 0.7 1765772

32 www.percona.com

CPU: problem solving• Low load

• Is your mysqld really active?• Check OS limits

• ulimit• User and global options

• Increase concurrency-related options• innodb thread concurrency• innodb read io threads• innodb write io threads• Max (2 X (number of CPU cores))!

• High load• Move job to memory: increase buffers• Tune queries

33 www.percona.com

Network: what is worth attention?• Stability• Bandwidth• Speed

• RTTsveta@thinkie> ping www.percona.com -c 2

PING www.percona.com (74.121.199.234) 56(84) bytes of data.

64 bytes from www.percona.com (74.121.199.234): icmp_seq=1 ttl=54 time=181 ms

64 bytes from www.percona.com (74.121.199.234): icmp_seq=2 ttl=54 time=181 ms

--- www.percona.com ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1001ms

rtt min/avg/max/mdev = 181.380/181.685/181.991/0.524 ms

34 www.percona.com

Network:diagnostics

• –log-warnings=2

• Send huge file (1G or larger)

• tcpdump

• P S.accounts, hosts, users

• P S.host cache

35 www.percona.com

Network:diagnostics

• P S.accounts, hosts, usersmysql> select user, host, current_connections as cur, \

-> total_connections as total from accounts;

+------+-----------+-----+-------+

| user | host | cur | total |

+------+-----------+-----+-------+

| foo | localhost | 0 | 3 |

| root | localhost | 1 | 3 |

| NULL | NULL | 14 | 17 |

+------+-----------+-----+-------+

3 rows in set (0.01 sec)

35 www.percona.com

Network:diagnostics• P S.host cachemysql> select IP, HOST, SUM_CONNECT_ERRORS, COUNT_NAMEINFO_PERMANENT_ERRORS, \

-> COUNT_PROXY_USER_ERRORS, COUNT_DEFAULT_DATABASE_ERRORS, \

-> FIRST_SEEN, LAST_SEEN from host_cache\G

************************ 1. row ************************

IP: 10.159.169.27

HOST: foo.bar.com

SUM_CONNECT_ERRORS: 1

COUNT_NAMEINFO_PERMANENT_ERRORS: 0

COUNT_PROXY_USER_ERRORS: 0

COUNT_DEFAULT_DATABASE_ERRORS: 1

FIRST_SEEN: 2014-06-10 20:29:45

LAST_SEEN: 2015-11-11 15:32:44

35 www.percona.com

MySQL Options

36 www.percona.com

Types of options

• Not directly affecting performance• Which locks to set

• innodb locks unsafe for binlog

• When to flush to disk• innodb flush log at trx commit

• Directly affecting performance• Concurrency-related

• innodb thread concurrency

• Memory buffers• innodb buffer pool size

37 www.percona.com

System variables and options: scope

• Global• Control parameters, necessary for all server

processes• Location of server files: datadir etc.• Shared buffers• More

• Session• Control connection-specific parameters

• MySQL option tables - direct link

38 www.percona.com

Buffers: when allocated

• Those which control behavior of whole server• Once at server startup• Can start with low values, then grow to specified

• Connection options• For every connection when connection opens

• Operation-specific• For every operation when needed• Can be allocated more than once per query

39 www.percona.com

System variables: control before 5.7• SHOW [GLOBAL] STATUS• Tables in Information Schema

• GLOBAL ,SESSION VARIABLES• GLOBAL ,SESSION STATUS

• GLOBAL• Since server start

• SESSION• For operations in current session• Can be reset

• FLUSH STATUS

40 www.percona.com

System variables: control in 5.7

• Performance Schema tables• variabes by*• user variables by*• status by*

• Statistics grouped by• Global• Session• Thread• Account/Host/User

41 www.percona.com

System variables: best practices

• Record currently used variables• SHOW [GLOBAL] VARIABLES

• Make change dynamically if possible• SET [GLOBAL] var name=NEW VAL

• Test in one session first• Then change global variable• Change configuration file after you are happy with

results42 www.percona.com

When affecting option is not known

• Record currently used variables• SHOW [GLOBAL] VARIABLES

• Start mysqld with option –no-defaults• This option must be first one!

• Check if problem is solved• Change variable values one-by-one until you find one

which leads to the problem

43 www.percona.com

Summary

44 www.percona.com

Summary

• Performance affected by 4 main factors• Query• Concurrency• Hardware• Configuration

• Study all of them• Iterate while troubleshooting

45 www.percona.com

Summary

• Performance affected by 4 main factors• Query – Optimize!• Concurrency• Hardware• Configuration

• Study all of them• Iterate while troubleshooting

45 www.percona.com

Summary

• Performance affected by 4 main factors• Query• Concurrency – Remove conflicts!• Hardware• Configuration

• Study all of them• Iterate while troubleshooting

45 www.percona.com

Summary

• Performance affected by 4 main factors• Query• Concurrency• Hardware – Find bottlenecks!• Configuration

• Study all of them• Iterate while troubleshooting

45 www.percona.com

Summary

• Performance affected by 4 main factors• Query• Concurrency• Hardware• Configuration – Mind the options!

• Study all of them• Iterate while troubleshooting

45 www.percona.com

More information

• MySQL Troubleshooting book• High Performance MySQL book• Planet MySQL• Percona Data Performance Blog• MySQL User Reference Manual• Bug trackers

• http://bugs.mysql.com• https://bugs.launchpad.net/percona-server/

• EXPLAIN FORMAT=JSON is Cool! series46 www.percona.com

Percona LiveData Performance Conference

• April 18-21 in Santa Clara, CA at theSanta Clara Convention Center

• Register with code “WebinarPL” toreceive 15% off at registration

• MongoDB, MySQL, NoSQL, Data in theCloud

www.perconalive.com

47 www.percona.com

Place for your questions

???

48 www.percona.com

Thank you!

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

49 www.percona.com