Percona Replication Manager – Renaming Cluster Hostnames

Recently I’ve had to rename a node on a production cluster – however, I did not have an experience on this yet so I decided to try on a sandbox and establish a known procedure. Without further ado, assuming we have 3 nodes and we want to rename ha3’s hostname to ha03.

[root@ha01 ~]# crm status
Last updated: Wed Jun  4 21:55:01 2014
Last change: Wed Jun  4 21:53:35 2014 via crm_attribute on ha01.localdomain
Stack: classic openais (with plugin)
Current DC: ha01.localdomain - partition with quorum
Version: 1.1.10-14.el6_5.2-368c726
3 Nodes configured, 3 expected votes
6 Resources configured


Online: [ ha01.localdomain ha02.localdomain ha3.localdomain ]

 Master/Slave Set: ms_MySQL [p_mysql]
     Masters: [ ha01.localdomain ]
     Slaves: [ ha02.localdomain ha3.localdomain ]
 Clone Set: ping_gw_clone [ping_gw]
     Started: [ ha01.localdomain ha02.localdomain ha3.localdomain ]

Here we go:

1. Put ha3 on standby:

   [root@ha01 ~]# crm node standby ha3.localdomain

2. Stop the cluster services on the node and delete it from the cluster, because I was using Pacemaker and Corosync, I followed the steps here, then rename the hostname as you would on any Linux system.

   [root@ha3 mysql]# crm_node -i
   1111009472
   [root@ha3 mysql]# service pacemaker stop
   Signaling Pacemaker Cluster Manager to terminate           [  OK  ]
   Waiting for cluster services to unload.                    [  OK  ]
   [root@ha3 mysql]# service corosync stop
   Signaling Corosync Cluster Engine (corosync) to terminate: [  OK  ]
   Waiting for corosync services to unload:.                  [  OK  ]
   
   [root@ha01 ~]# crm_node --force -R 1111009472
   [root@ha01 ~]# cibadmin --delete --obj_type nodes --crm_xml '<node uname="ha3.localdomain"/>'
   [root@ha01 ~]# cibadmin --delete --obj_type status --crm_xml '<node_state uname="ha3.localdomain"/>'

3. Put the whole cluster in maintenance mode and restart the Pacemaker and Corosync services. I could not find a way to rename cleanly without doing this, if I don’t I always end up with ghost nodes on the cluster. Because ha01 is the current Master, I would cycle all cluster services – stopping at ha01 last and starting with it first.

   [root@ha01 ~]# crm configure property maintenance-mode=true
   
   [root@ha02 ~]# service pacemaker stop
   Signaling Pacemaker Cluster Manager to terminate           [  OK  ]
   Waiting for cluster services to unload                     [  OK  ]
   [root@ha02 ~]# service corosync stop
   Signaling Corosync Cluster Engine (corosync) to terminate: [  OK  ]
   Waiting for corosync services to unload:                   [  OK  ]
   
   [root@ha01 ~]# service pacemaker stop
   Signaling Pacemaker Cluster Manager to terminate           [  OK  ]
   Waiting for cluster services to unload.                    [  OK  ]
   [root@ha01 ~]# service corosync stop
   Signaling Corosync Cluster Engine (corosync) to terminate: [  OK  ]
   Waiting for corosync services to unload:.                  [  OK  ]
   
   [root@ha01 ~]# service corosync start
   Starting Corosync Cluster Engine (corosync):               [  OK  ]
   [root@ha01 ~]# service pacemaker start
   Starting Pacemaker Cluster Manager                         [  OK  ]
   
   [root@ha02 ~]# service corosync start
   Starting Corosync Cluster Engine (corosync):               [  OK  ]
   [root@ha02 ~]# service pacemaker start
   Starting Pacemaker Cluster Manager                         [  OK  ]

4. Disable maintenance-mode and verify the 2 actives nodes are up and running:

   [root@ha01 ~]# crm configure property maintenance-mode=false
   [root@ha01 ~]# crm status
   Last updated: Wed Jun  4 22:56:24 2014
   Last change: Wed Jun  4 22:56:23 2014 via crm_attribute on ha02.localdomain
   Stack: classic openais (with plugin)
   Current DC: ha02.localdomain - partition with quorum
   Version: 1.1.10-14.el6_5.2-368c726
   2 Nodes configured, 3 expected votes
   5 Resources configured
   
   
   Online: [ ha01.localdomain ha02.localdomain ]
   
    Master/Slave Set: ms_MySQL [p_mysql]
        Masters: [ ha01.localdomain ]
        Slaves: [ ha02.localdomain ]
    Clone Set: ping_gw_clone [ping_gw]
        Started: [ ha01.localdomain ha02.localdomain ]

5. Then start the cluster services on ha03 which I renamed:

   [root@ha03 mysql]# service corosync start
   Starting Corosync Cluster Engine (corosync):               [  OK  ]
   [root@ha03 mysql]# service pacemaker start
   Starting Pacemaker Cluster Manager                         [  OK  ]
   [root@ha03 mysql]# crm status
   Last updated: Wed Jun  4 23:14:30 2014
   Last change: Wed Jun  4 23:14:25 2014 via crm_attribute on ha02.localdomain
   Stack: classic openais (with plugin)
   Current DC: ha02.localdomain - partition with quorum
   Version: 1.1.10-14.el6_5.2-368c726
   3 Nodes configured, 3 expected votes
   6 Resources configured
   
   
   Online: [ ha01.localdomain ha02.localdomain ha03.localdomain ]
   
    Master/Slave Set: ms_MySQL [p_mysql]
        Masters: [ ha01.localdomain ]
        Slaves: [ ha02.localdomain ha03.localdomain ]
    Clone Set: ping_gw_clone [ping_gw]
        Started: [ ha01.localdomain ha02.localdomain ha03.localdomain ]

Do you have any more convenient way of doing this?