undelete (and more) rows from the binary log

Hacking Session: undelete (and more) rows from the binary log

Percona Live Europe Amsterdam 2015 Frédéric -lefred- Descamps

Who am I ?● Frédéric Descamps● @lefred - follow me on twitter if you want ;-)

● http://about.me/lefred● Working for Percona since 2011● Managing MySQL since 3.23● devops believer

Why ?

Because of Scott Noyes' blog post : http://thenoyes.com/littlenoise/?p=307

● Because it's faster than point-in-time recovery

● Because I didn't remember the awk script in Scott's blog post

Requirements● Have the binary logs in ROW format● Have binlog_row_image set to FULL● Have access to the binary logs● Have access to MySQL● Have access to the MySQL source code [not mandatory]● Have a brain and being able to use it ;-)

Percona Live Europe Amsterdam 2015

Point-in-Time Recovery

Usually when we need to «undo» even one single statement, we need to perform a point-in-time recovery● restore the last backup (full or several incrementals)● then replay some binary logs (and it can be a bunch of them!)● this is a very slow operation and while the binary logs are

replayed, usually everything needs to be stopped.


Let's hack together a quicker process

Ready ?

Let's delete a row

mysql> select @@version, @@version_comment;+------------+------------------------------+| @@version | @@version_comment |+------------+------------------------------+| 5.6.22-log | MySQL Community Server (GPL) |+------------+------------------------------+

mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 2 | Liz | 2 | 24 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+4 rows in set (0.00 sec)


Let's delete a row (2)

mysql> delete from community_dinner where id=2;Query OK, 1 row affected (0.05 sec)

mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+3 rows in set (0.00 sec)


Let's find the event in the binary logmysql> show master status\G*************************** 1. row *************************** File: mysql-bin.000018 Position: 907 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)

mysql> show binlog events in 'mysql-bin.000018';+------------------+-----+-------------+-----------+-------------+---------------------------+| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |+------------------+-----+-------------+-----------+------------+----------------------------+| mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... || mysql-bin.000018 | 120 | Query | 1 | 220 | create ... || mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; || mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN || mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ || mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu|| mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E|| mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ |+------------------+-----+-------------+-----------+------------+====------------------------+


Let's find the event in the binary logmysql> show master status\G*************************** 1. row *************************** File: mysql-bin.000018 Position: 907 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)


The event we are looking for


binary log's event content # mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;/*!40019 SET @@session.max_insert_delayed_threads=0*/;/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;DELIMITER /*!*/;# at 4#141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v 5.6.22-log created 141221 21:20:55 at startup# Warning: this binlog is either in use or was not closed properly.ROLLBACK/*!*/;BINLOG 'pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMTif8='/*!*/;# at 824#150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags: STMT_END_F

BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;DELIMITER ;# End of log fileROLLBACK /* added by mysqlbinlog */;/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;


binary log's event content decoded...BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;...

● Let's check from the source the type code: https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h

WRITE_ROWS_EVENT = 30,

UPDATE_ROWS_EVENT = 31,

DELETE_ROWS_EVENT = 32,


binary log's event content decoded...BINLOG 'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='/*!*/;...

● Let's check from the source the type code: https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h

WRITE_ROWS_EVENT = 30,

UPDATE_ROWS_EVENT = 31,

DELETE_ROWS_EVENT = 32,

MySQL 5.5 and MariaDB use V1: WRITE_ROWS_EVENT_V1 = 23,UPDATE_ROWS_EVENT_V1 = 24,DELETE_ROWS_EVENT_V1 = 25,


binary log's event content decoded# python -c "print hex(32)"0x20

● Again from the source we can find the event type's offset

#define EVENT_TYPE_OFFSET 4

# python -c "import base64; print ord(base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==')[4])"32

● This is exactly the event we were looking for !!


rebuild a new binlog event● Now we can rebuild a new event by replacing \x32 by \x30# python>>> import base64>>> data=base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==')>>> data'\x05\x11\xa8T \x01\x00\x00\x004\x00\x00\x00l\x03\x00\x00\x00\x00L\x00\x00\x00\x00\x00\x01\x00\x02\x00\x04\xff\xf0\x02\x00\x00\x00\x03Liz\x02\x00\x00\x00\x18\x00\x00\x00\xee)\xc3\xaa'>>> hex(30)'0x1e'>>> data2=base64.b64encode("\x05\x11\xa8T\x1e\x01\x00\x00\x004\x00\x00\x00l\x03\x00\x00\x00\x00L\x00\x00\x00\x00\x00\x01\x00\x02\x00\x04\xff\xf0\x02\x00\x00\x00\x03Liz\x02\x00\x00\x00\x18\x00\x00\x00\xee)\xc3\xaa")>>> data2'BRGoVDABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='

● Let's compare the two lines:BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==BRGoVB4BAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==


rebuild a new binlog event & replay it● ready to replay ?●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 | \ > sed s/BRGoVCAB/BRGoVB4B/ | mysql



rebuild a new binlog event & replay it● ready to replay ?●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 | \ sed s/BRGoVCAB/BRGoVB4B/ | mysql


uuh ? did it fail ?


find what to replay


● We need to replay more, from the BEGIN to the COMMIT


replay it, 2nd try● ready to replay, again ?●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 | \ > sed s/BRGoVCAB/BRGoVB4B/ | mysql

mysql> select * from fosdem.community_dinner;+----+--------+--------+-------+| id | name | pizzas | beers |+----+--------+--------+-------+| 1 | dim0 | 99 | 99 || 2 | Liz | 2 | 24 || 3 | Kenny | 12 | 32 || 4 | lefred | 10 | 10 |+----+--------+--------+-------+4 rows in set (0.00 sec)


More ?● With the same kind of technique, we can also:

– delete rows that were inserted (sql injection?)

– un-update modified rows (more complicated)

● I've created a script that automates all that: MyUndelete


MyUndelete : «un-insert»● We can delete an INSERT


# ./MyUndelete.py -s 41989 -e 42207 -i -b mysql-bin.000004

*** WARNING *** USE WITH CARE ****

Binlog file is /var/lib/mysql/mysqld-bin.000004Start Position file is 41989End Postision file is 42207We also look to undo INSERTsEvent type ('\x1e') is an insert v2Ready to revert the statement ? [y/n]yDone... I hope it worked ;)

MyUndelete : «un-update»● This is a bit more difficult as we need to find the lenght of the

record to be able to split both records and rebuild a working binary event


mysql> select * from community_dinner;+----+-------+--------+-------+| id | name | pizzas | beers |+----+-------+--------+-------+| 1 | fred | 3 | 3 || 2 | lizz | 2 | 10 || 3 | dimi | 99 | 99 || 4 | kenny | 20 | 20 |+----+-------+--------+-------+

MyUndelete : «un-update» (2)


mysql> update community_dinner set beers=0 where id=3;Query OK, 1 row affected (0.01 sec)Rows matched: 1 Changed: 1 Warnings: 0

mysql> select * from community_dinner;+----+-------+--------+-------+| id | name | pizzas | beers |+----+-------+--------+-------+| 1 | fred | 3 | 3 || 2 | lizz | 2 | 10 || 3 | dimi | 99 | 0 || 4 | kenny | 20 | 20 |+----+-------+--------+-------+4 rows in set (0.00 sec)



mysql> show binlog events in 'mysqld-bin.000018';...| mysqld-bin.000018 | 1145 | Query | 1 | 1219 | BEGIN || mysqld-bin.000018 | 1219 | Table_map | 1 | 1285 | table_id: 70 (fosdem.community_dinner) || mysqld-bin.000018 | 1285 | Update_rows | 1 | 1357 | table_id: 70 flags: STMT_END_F | mysqld-bin.000018 | 1357 | Xid | 1 | 1388 | COMMIT /* xid=44 */ ...

# ./MyUndelete.py -s 1145 -e 1388 -u -b /var/lib/mysql/mysqld-bin.000018

*** WARNING *** USE WITH CARE ****

Binlog file is /var/lib/mysql/mysqld-bin.000018Start Position file is 1145End Postision file is 1357Event type ('\x1f') is an update v2We got an update!!Ready to revert the statement ? [y/n]ySending to mysql...Done... I hope it worked ;)



mysql> select * from community_dinner;+----+-------+--------+-------+| id | name | pizzas | beers |+----+-------+--------+-------+| 1 | fred | 3 | 3 || 2 | lizz | 2 | 10 || 3 | dimi | 99 | 99 || 4 | kenny | 20 | 20 |+----+-------+--------+-------+

Resources● Oracle MySQL Source code — https://github.com/mysql/mysql-server

● MariaDB Source Code — https://github.com/MariaDB/server

● Scott Noyes Blog Post - http://thenoyes.com/littlenoise/?p=307

● MyUndelete on github — https://github.com/lefred/MyUndelete


Related topic● MySQL released «Binary Log API» that could be use to decode

in a better way the binary logs– http://mysqlhighavailability.com/mysql-binlog-events-reading-and-handling-

information-from-your-binary-log/

– http://mysqlhighavailability.com/mysql-binlog-events-use-case-and-examples/

● Jeremy Cole wrote a Ruby Library to also parse binary logs– https://github.com/jeremycole/mysql_binlog


Thank you

Questions ?


undelete (and more) rows from the binary log

Engineering