mysql [localhost] {msandbox} (employees) > EXPLAIN SELECT
    ->      e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE e.hire_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00' OR
    ->      s.from_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> ORDER BY e.emp_no, hire_date, from_date \G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: e
         type: ALL
possible_keys: PRIMARY,hire_date
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 300547
        Extra: Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: s
         type: ref
possible_keys: PRIMARY,emp_no,from_date
          key: PRIMARY
      key_len: 4
          ref: employees.e.emp_no
         rows: 4
        Extra: Using where
2 rows in set (0.00 sec)

mysql [localhost] {msandbox} (employees) > SHOW CREATE TABLE employees \G
*************************** 1. row ***************************
       Table: employees
Create Table: CREATE TABLE `employees` (
  `emp_no` int(11) NOT NULL,
  `birth_date` date NOT NULL,
  `first_name` varchar(14) NOT NULL,
  `last_name` varchar(16) NOT NULL,
  `gender` enum('M','F') NOT NULL,
  `hire_date` date NOT NULL,
  PRIMARY KEY (`emp_no`),
  KEY `hire_date` (`hire_date`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

mysql [localhost] {msandbox} (employees) > SHOW CREATE TABLE salaries \G
*************************** 1. row ***************************
       Table: salaries
Create Table: CREATE TABLE `salaries` (
  `emp_no` int(11) NOT NULL,
  `salary` int(11) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`from_date`),
  KEY `emp_no` (`emp_no`),
  KEY `from_date` (`from_date`),
  CONSTRAINT `salaries_ibfk_1` FOREIGN KEY (`emp_no`) REFERENCES `employees` (`emp_no`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

If you look at the execution plan, the optimizer will always do a full table scan of the first table even though there is proper indexes on employees.hire_date and salaries.from_date. This is because it cannot know in advance which rows from the second table will match the rows from the second table, the OR conditions needs to match both tables. For the sake of brevity, observe what happens if I change the OR condition to AND instead.

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: e
         type: range
possible_keys: PRIMARY,hire_date
          key: hire_date
      key_len: 3
          ref: NULL
         rows: 2101
        Extra: Using where; Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: s
         type: ref
possible_keys: PRIMARY,emp_no,from_date
          key: PRIMARY
      key_len: 4
          ref: employees.e.emp_no
         rows: 4
        Extra: Using where
2 rows in set (0.00 sec)

As you can see, the optimizer now used the hire_date index in which for every matching row, there is about 4 rows matching on the second table.

Notice how many index reads and the full table scan the original query does. Imagine if you are working on a reporting query based on date ranges that has millions of rows for the first table, you might have to wait the next day for the results!

mysql [localhost] {msandbox} (employees) > SELECT
    ->      e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE e.hire_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00' OR
    ->      s.from_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> ORDER BY e.emp_no, hire_date, from_date;

b5abad9ac152d6289d4cc62b7d71fb83  -
28133 rows in set (1.39 sec)

mysql [localhost] {msandbox} (employees) > NOPAGER; SHOW STATUS LIKE 'Handler%';
PAGER set to stdout
+----------------------------+---------+
| Variable_name              | Value   |
+----------------------------+---------+
...
| Handler_read_key           | 300025  |
| Handler_read_next          | 2844047 |
| Handler_read_prev          | 0       |
| Handler_read_rnd           | 28133   |
| Handler_read_rnd_next      | 328159  |
...
| Handler_write              | 28133   |
+----------------------------+---------+
15 rows in set (0.00 sec)

Because the queries need to return rows matching both tables, we can rewrite it as UNION instead separating the WHERE clauses into 2 different SELECT queries like below.

mysql [localhost] {msandbox} (employees) > PAGER md5sum; FLUSH STATUS;
PAGER set to 'md5sum'
Query OK, 0 rows affected (0.00 sec)

mysql [localhost] {msandbox} (employees) > (
    -> SELECT
    ->      e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE e.hire_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> )
    -> UNION
    -> (
    -> SELECT e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE s.from_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> )
    -> ORDER BY emp_no, hire_date, from_date;
b5abad9ac152d6289d4cc62b7d71fb83  -
28133 rows in set (0.14 sec)

mysql [localhost] {msandbox} (employees) > NOPAGER; SHOW STATUS LIKE 'Handler%';
PAGER set to stdout
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
...
| Handler_read_key           | 11746 |
| Handler_read_next          | 31302 |
| Handler_read_prev          | 0     |
| Handler_read_rnd           | 28133 |
| Handler_read_rnd_next      | 28134 |
...
| Handler_write              | 29200 |
+----------------------------+-------+
15 rows in set (0.00 sec)

With this approach, you are actually limiting the number of rows needed to be examined immediately before being JOINed. There is no real downsize with the new query, you just have to watch out that results from both SELECT’s does not grow too large, nearly the size of the total rows of both tables. In which case, you should not be doing the latter anyway i.e. trying to filter with big date ranges especially if you have very large dataset, instead you should implement summary tables that produces the results you need.

1. STOP SLAVE; on all slaves including B that are attached to A directly.
2. Make sure that log_slave_updates and bin_log is ON on B.
3. If you are not taking A out of rotation and planning to switch back to it later as your master, you can configure it as slave of B forming a master-master pair. Simply take the SHOW MASTER STATUS; coordinates from B and use it to CHANGE MASTER TO for A.
4. Save SHOW MASTER STATUS; output from A.
5. Start replication on all the slaves that are attach to A again, but only until the coordinates you get from #3 using the syntax START SLAVE UNTIL MASTER_LOG_FILE=<log_file>, MASTER_LOG_POS=<log_pos>. This ensures that all slaves are caught up at exactly the same position so you can safely point them to replicate later to B.
6. Once all slaves have caught up on the same coordinates, now its time to reconfigure the rest of the slaves except B to replicate from B. Using the SHOW MASTER STATUS; coordinates from B, execute a CHANGE MASTER TO MASTER_HOST=B ...
7. Now its time to fetch the remaining events from A to B, on B, STOP SLAVE; then START SLAVE; again, this time replication will retrieve all events since the coordinates we stopped at #4.
8. Once B and its slaves has caught up on replication, you can now point your application to send writes to B.
9. Lastly if you are taking A out of rotation but keeping it online, RESET SLAVE; on B after Exec_Master_Log_Pos and Read_Master_Log_Pos on B have stopped on the same position so that no further writes from A is replicated to B.

Did I miss anything? Of course – not all failovers are the same, this guide is meant as an overview and must be reviewed to match your situation always. Comments welcome!

1. After the backup, find the latest backup set on the backup directory using bash or whichever scripting language you are using, by default the resulting directory for new backup sets takes this sample format ’2010-03-13_02-42-44′. Below is how you can achieve this with bash

CB=$(ls -1 | egrep '^[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}-[0-9]{2}-[0-9]{2}$' | sort -n | head -n 1)

2. Use the –no-timestamp option of innobackupex to control the backup directories.

--no-timestamp
This option prevents creation of a time-stamped subdirectory of the
BACKUP-ROOT-DIR given on the command line. When it is specified, the
backup is done in BACKUP-ROOT-DIR instead.

With this method, you will have to create the unique backup directories from your script, which in turn you would already know the resulting name you can use for prepare. You can easily generate one based on current date with the sample bash command below:

CURDATE=$(date +%Y-%m-%d)

This is only one essential part of a good backup procedure, I might blog on more.

SEVERE: java.lang.NumberFormatException: For input string: “true”

This is a default behavior of Connector/J on the property tinyInt1isBit documented here. To fix this you should add tinyInt1isBit=false to your JDBC connection string.

1. Use the –chunk-size with data size option. If you have large tables (a couple of GB), dump and restore may simply consume a lot of RAM. Using –chunk-size like 20M or 50M you use a reasonable amount of memory. The tool manual also specifies more benefits when using chunks http://www.maatkit.org/doc/mk-parallel-dump.html#chunks
2. Make sure your server’s max_allowed_packet is large enough to accommodate large rows. When mk-parallel-restore fails with “Packet too large” error or any other error in that case, it simply emits the whole SQL statements onto the console making it hard to catch the actual error if your console buffer is too small in turn leaving you clueless if this is indeed your first time.

ERROR 1467 (HY000) at line 1: Failed to read auto-increment value from storage engine

1. Your table’s auto increment column datatype is too small i.e. MySQL is not able to generate more auto-increment values beyond TINYINT UNSIGNED because you are trying to insert your 256th record.
2. It is a bug http://bugs.mysql.com/bug.php?id=36411

CREATE TABLE `stories` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`title` VARCHAR( 255 ) NOT NULL ,
`text` TEXT NOT NULL ,
`creationdate` DATETIME NOT NULL ,
`lastupdate` TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE = MYISAM

When you want to store stories, you would also want to record when it was originally created as well track the last time it was updated. The problem here is that adding a `DEFAULT NOW()` clause will not work for the `CREATE TABLE` query above as it is not supported. So when your insert a new story you will have to explicitly add a `NOW()` function for the `creationdate` row so it will reflect the current date as creation date. This should’ve been a simple schema functionality, turns out after more than a year MySQL seems to ignore for some reason.

Go on have yourself a read here http://bugs.mysql.com/bug.php?id=27645

How about you, how many times have you have to work around this from your application code?