mysql [localhost] {msandbox} (employees) > EXPLAIN SELECT
    ->      e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE e.hire_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00' OR
    ->      s.from_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> ORDER BY e.emp_no, hire_date, from_date \G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: e
         type: ALL
possible_keys: PRIMARY,hire_date
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 300547
        Extra: Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: s
         type: ref
possible_keys: PRIMARY,emp_no,from_date
          key: PRIMARY
      key_len: 4
          ref: employees.e.emp_no
         rows: 4
        Extra: Using where
2 rows in set (0.00 sec)

mysql [localhost] {msandbox} (employees) > SHOW CREATE TABLE employees \G
*************************** 1. row ***************************
       Table: employees
Create Table: CREATE TABLE `employees` (
  `emp_no` int(11) NOT NULL,
  `birth_date` date NOT NULL,
  `first_name` varchar(14) NOT NULL,
  `last_name` varchar(16) NOT NULL,
  `gender` enum('M','F') NOT NULL,
  `hire_date` date NOT NULL,
  PRIMARY KEY (`emp_no`),
  KEY `hire_date` (`hire_date`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

mysql [localhost] {msandbox} (employees) > SHOW CREATE TABLE salaries \G
*************************** 1. row ***************************
       Table: salaries
Create Table: CREATE TABLE `salaries` (
  `emp_no` int(11) NOT NULL,
  `salary` int(11) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`from_date`),
  KEY `emp_no` (`emp_no`),
  KEY `from_date` (`from_date`),
  CONSTRAINT `salaries_ibfk_1` FOREIGN KEY (`emp_no`) REFERENCES `employees` (`emp_no`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

If you look at the execution plan, the optimizer will always do a full table scan of the first table even though there is proper indexes on employees.hire_date and salaries.from_date. This is because it cannot know in advance which rows from the second table will match the rows from the second table, the OR conditions needs to match both tables. For the sake of brevity, observe what happens if I change the OR condition to AND instead.

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: e
         type: range
possible_keys: PRIMARY,hire_date
          key: hire_date
      key_len: 3
          ref: NULL
         rows: 2101
        Extra: Using where; Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: s
         type: ref
possible_keys: PRIMARY,emp_no,from_date
          key: PRIMARY
      key_len: 4
          ref: employees.e.emp_no
         rows: 4
        Extra: Using where
2 rows in set (0.00 sec)

As you can see, the optimizer now used the hire_date index in which for every matching row, there is about 4 rows matching on the second table.

Notice how many index reads and the full table scan the original query does. Imagine if you are working on a reporting query based on date ranges that has millions of rows for the first table, you might have to wait the next day for the results!

mysql [localhost] {msandbox} (employees) > SELECT
    ->      e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE e.hire_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00' OR
    ->      s.from_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> ORDER BY e.emp_no, hire_date, from_date;

b5abad9ac152d6289d4cc62b7d71fb83  -
28133 rows in set (1.39 sec)

mysql [localhost] {msandbox} (employees) > NOPAGER; SHOW STATUS LIKE 'Handler%';
PAGER set to stdout
+----------------------------+---------+
| Variable_name              | Value   |
+----------------------------+---------+
...
| Handler_read_key           | 300025  |
| Handler_read_next          | 2844047 |
| Handler_read_prev          | 0       |
| Handler_read_rnd           | 28133   |
| Handler_read_rnd_next      | 328159  |
...
| Handler_write              | 28133   |
+----------------------------+---------+
15 rows in set (0.00 sec)

Because the queries need to return rows matching both tables, we can rewrite it as UNION instead separating the WHERE clauses into 2 different SELECT queries like below.

mysql [localhost] {msandbox} (employees) > PAGER md5sum; FLUSH STATUS;
PAGER set to 'md5sum'
Query OK, 0 rows affected (0.00 sec)

mysql [localhost] {msandbox} (employees) > (
    -> SELECT
    ->      e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE e.hire_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> )
    -> UNION
    -> (
    -> SELECT e.emp_no, birth_date, first_name,
    ->      last_name, gender, hire_date,
    ->      salary, from_date, to_date
    -> FROM employees e
    -> INNER JOIN salaries s ON (e.emp_no = s.emp_no)
    -> WHERE s.from_date BETWEEN '1990-06-01 00:00:00' AND '1990-07-01 00:00:00'
    -> )
    -> ORDER BY emp_no, hire_date, from_date;
b5abad9ac152d6289d4cc62b7d71fb83  -
28133 rows in set (0.14 sec)

mysql [localhost] {msandbox} (employees) > NOPAGER; SHOW STATUS LIKE 'Handler%';
PAGER set to stdout
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
...
| Handler_read_key           | 11746 |
| Handler_read_next          | 31302 |
| Handler_read_prev          | 0     |
| Handler_read_rnd           | 28133 |
| Handler_read_rnd_next      | 28134 |
...
| Handler_write              | 29200 |
+----------------------------+-------+
15 rows in set (0.00 sec)

With this approach, you are actually limiting the number of rows needed to be examined immediately before being JOINed. There is no real downsize with the new query, you just have to watch out that results from both SELECT’s does not grow too large, nearly the size of the total rows of both tables. In which case, you should not be doing the latter anyway i.e. trying to filter with big date ranges especially if you have very large dataset, instead you should implement summary tables that produces the results you need.

1. STOP SLAVE; on all slaves including B that are attached to A directly.
2. Make sure that log_slave_updates and bin_log is ON on B.
3. If you are not taking A out of rotation and planning to switch back to it later as your master, you can configure it as slave of B forming a master-master pair. Simply take the SHOW MASTER STATUS; coordinates from B and use it to CHANGE MASTER TO for A.
4. Save SHOW MASTER STATUS; output from A.
5. Start replication on all the slaves that are attach to A again, but only until the coordinates you get from #3 using the syntax START SLAVE UNTIL MASTER_LOG_FILE=<log_file>, MASTER_LOG_POS=<log_pos>. This ensures that all slaves are caught up at exactly the same position so you can safely point them to replicate later to B.
6. Once all slaves have caught up on the same coordinates, now its time to reconfigure the rest of the slaves except B to replicate from B. Using the SHOW MASTER STATUS; coordinates from B, execute a CHANGE MASTER TO MASTER_HOST=B ...
7. Now its time to fetch the remaining events from A to B, on B, STOP SLAVE; then START SLAVE; again, this time replication will retrieve all events since the coordinates we stopped at #4.
8. Once B and its slaves has caught up on replication, you can now point your application to send writes to B.
9. Lastly if you are taking A out of rotation but keeping it online, RESET SLAVE; on B after Exec_Master_Log_Pos and Read_Master_Log_Pos on B have stopped on the same position so that no further writes from A is replicated to B.

Did I miss anything? Of course – not all failovers are the same, this guide is meant as an overview and must be reviewed to match your situation always. Comments welcome!

In my case if I just specify the “updates” boot option, Anaconda will try to automatically configure networking – too bad if you don not have DHCP available. However, if you specify an alternative install method to Anaconda, it will ask you to manually configure your network, in this case you will be able ti use the updates image as well. In short, you just have to specify something like this on your boot options:

repo=http://mirror.steadfast.net/fedora/releases/16/Fedora/x86_64/os/ updates=http://dlehman.fedorapeople.org/updates/updates-750480.3.img

The only downside is that, obviously, your install media will be over the network. If you have a slow connection then that’s another story. Do you know any workaround?


#19635 0x000000000046b13b in ?? ()
#19636 0x000000000046b231 in ?? ()
#19637 0x000000000046b13b in ?? ()
#19638 0x000000000046b231 in ?? ()
#19639 0x000000000046cb3f in ?? ()
#19640 0x000000000047c76f in php_pcre_exec ()
#19641 0x0000000000480d46 in php_pcre_replace_impl ()
---Type to continue, or q to quit---
#19642 0x0000000000481f0d in ?? ()
#19643 0x0000000000482504 in ?? ()
#19644 0x0000000000482a33 in ?? ()
#19645 0x00000000005e8f19 in ?? ()
#19646 0x00000000005e847b in execute ()
#19647 0x00000000005c1395 in zend_execute_scripts ()
#19648 0x0000000000571418 in php_execute_script ()
#19649 0x000000000064b5a0 in ?? ()
#19650 0x000000314821d994 in __libc_start_main () from /lib64/libc.so.6
#19651 0x00000000004222b9 in _start ()


My appeal to the PHP package builders (CentOS/Fedora mainstream, RPMForge, Atomic) – please please please – include a debuginfo package on your repo

1. After the backup, find the latest backup set on the backup directory using bash or whichever scripting language you are using, by default the resulting directory for new backup sets takes this sample format ’2010-03-13_02-42-44′. Below is how you can achieve this with bash

CB=$(ls -1 | egrep '^[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}-[0-9]{2}-[0-9]{2}$' | sort -n | head -n 1)

2. Use the –no-timestamp option of innobackupex to control the backup directories.

--no-timestamp
This option prevents creation of a time-stamped subdirectory of the
BACKUP-ROOT-DIR given on the command line. When it is specified, the
backup is done in BACKUP-ROOT-DIR instead.

With this method, you will have to create the unique backup directories from your script, which in turn you would already know the resulting name you can use for prepare. You can easily generate one based on current date with the sample bash command below:

CURDATE=$(date +%Y-%m-%d)

This is only one essential part of a good backup procedure, I might blog on more.

SEVERE: java.lang.NumberFormatException: For input string: “true”

This is a default behavior of Connector/J on the property tinyInt1isBit documented here. To fix this you should add tinyInt1isBit=false to your JDBC connection string.

err: Could not retrieve catalog from remote server: certificate verify failed

If you’re new to puppet, this error is a bit tricky to identify. Even after recreating the certificates on the client the error still persist. If this is the case, most likely is that time on your master and the client is different. Install and sync NTP – this should be a standard component for new servers anyway.