1. STOP SLAVE; on all slaves including B that are attached to A directly.
2. Make sure that log_slave_updates and bin_log is ON on B.
3. If you are not taking A out of rotation and planning to switch back to it later as your master, you can configure it as slave of B forming a master-master pair. Simply take the SHOW MASTER STATUS; coordinates from B and use it to CHANGE MASTER TO for A.
4. Save SHOW MASTER STATUS; output from A.
5. Start replication on all the slaves that are attach to A again, but only until the coordinates you get from #3 using the syntax START SLAVE UNTIL MASTER_LOG_FILE=<log_file>, MASTER_LOG_POS=<log_pos>. This ensures that all slaves are caught up at exactly the same position so you can safely point them to replicate later to B.
6. Once all slaves have caught up on the same coordinates, now its time to reconfigure the rest of the slaves except B to replicate from B. Using the SHOW MASTER STATUS; coordinates from B, execute a CHANGE MASTER TO MASTER_HOST=B ...
7. Now its time to fetch the remaining events from A to B, on B, STOP SLAVE; then START SLAVE; again, this time replication will retrieve all events since the coordinates we stopped at #4.
8. Once B and its slaves has caught up on replication, you can now point your application to send writes to B.
9. Lastly if you are taking A out of rotation but keeping it online, RESET SLAVE; on B after Exec_Master_Log_Pos and Read_Master_Log_Pos on B have stopped on the same position so that no further writes from A is replicated to B.

Did I miss anything? Of course – not all failovers are the same, this guide is meant as an overview and must be reviewed to match your situation always. Comments welcome!

1. After the backup, find the latest backup set on the backup directory using bash or whichever scripting language you are using, by default the resulting directory for new backup sets takes this sample format ’2010-03-13_02-42-44′. Below is how you can achieve this with bash

CB=$(ls -1 | egrep '^[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}-[0-9]{2}-[0-9]{2}$' | sort -n | head -n 1)

2. Use the –no-timestamp option of innobackupex to control the backup directories.

--no-timestamp
This option prevents creation of a time-stamped subdirectory of the
BACKUP-ROOT-DIR given on the command line. When it is specified, the
backup is done in BACKUP-ROOT-DIR instead.

With this method, you will have to create the unique backup directories from your script, which in turn you would already know the resulting name you can use for prepare. You can easily generate one based on current date with the sample bash command below:

CURDATE=$(date +%Y-%m-%d)

This is only one essential part of a good backup procedure, I might blog on more.

SEVERE: java.lang.NumberFormatException: For input string: “true”

This is a default behavior of Connector/J on the property tinyInt1isBit documented here. To fix this you should add tinyInt1isBit=false to your JDBC connection string.

1. Use the –chunk-size with data size option. If you have large tables (a couple of GB), dump and restore may simply consume a lot of RAM. Using –chunk-size like 20M or 50M you use a reasonable amount of memory. The tool manual also specifies more benefits when using chunks http://www.maatkit.org/doc/mk-parallel-dump.html#chunks
2. Make sure your server’s max_allowed_packet is large enough to accommodate large rows. When mk-parallel-restore fails with “Packet too large” error or any other error in that case, it simply emits the whole SQL statements onto the console making it hard to catch the actual error if your console buffer is too small in turn leaving you clueless if this is indeed your first time.

ERROR 1467 (HY000) at line 1: Failed to read auto-increment value from storage engine

1. Your table’s auto increment column datatype is too small i.e. MySQL is not able to generate more auto-increment values beyond TINYINT UNSIGNED because you are trying to insert your 256th record.
2. It is a bug http://bugs.mysql.com/bug.php?id=36411

CREATE TABLE `stories` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`title` VARCHAR( 255 ) NOT NULL ,
`text` TEXT NOT NULL ,
`creationdate` DATETIME NOT NULL ,
`lastupdate` TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE = MYISAM

When you want to store stories, you would also want to record when it was originally created as well track the last time it was updated. The problem here is that adding a `DEFAULT NOW()` clause will not work for the `CREATE TABLE` query above as it is not supported. So when your insert a new story you will have to explicitly add a `NOW()` function for the `creationdate` row so it will reflect the current date as creation date. This should’ve been a simple schema functionality, turns out after more than a year MySQL seems to ignore for some reason.

Go on have yourself a read here http://bugs.mysql.com/bug.php?id=27645

How about you, how many times have you have to work around this from your application code?