Percona Replication Manager – Renaming Cluster Hostnames

Recently I’ve had to rename a node on a production cluster – however, I did not have an experience on this yet so I decided to try on a sandbox and establish a known procedure. Without further ado, assuming we have 3 nodes and we want to rename ha3’s hostname to ha03.

Here we go:

1. Put ha3 on standby:
   
   [root@ha01 ~]# crm node standby ha3.localdomain

   1	[root@ha01 ~]# crm node standby ha3.localdomain

2. Stop the cluster services on the node and delete it from the cluster, because I was using Pacemaker and Corosync, I followed the steps here, then rename the hostname as you would on any Linux system.
   
   [root@ha3 mysql]# crm_node -i 1111009472 [root@ha3 mysql]# service pacemaker stop Signaling Pacemaker Cluster Manager to terminate [ OK ] Waiting for cluster services to unload. [ OK ] [root@ha3 mysql]# service corosync stop Signaling Corosync Cluster Engine (corosync) to terminate: [ OK ] Waiting for corosync services to unload:. [ OK ] [root@ha01 ~]# crm_node --force -R 1111009472 [root@ha01 ~]# cibadmin --delete --obj_type nodes --crm_xml '<node uname="ha3.localdomain"/>' [root@ha01 ~]# cibadmin --delete --obj_type status --crm_xml '<node_state uname="ha3.localdomain"/>'

   1 2 3 4 5 6 7 8 9 10 11 12

   [root@ha3 mysql]# crm_node -i 1111009472 [root@ha3 mysql]# service pacemaker stop Signaling Pacemaker Cluster Manager to terminate           [  OK  ] Waiting for cluster services to unload.                    [  OK  ] [root@ha3 mysql]# service corosync stop Signaling Corosync Cluster Engine (corosync) to terminate: [  OK  ] Waiting for corosync services to unload:.                  [  OK  ]   [root@ha01 ~]# crm_node --force -R 1111009472 [root@ha01 ~]# cibadmin --delete --obj_type nodes --crm_xml '<node uname="ha3.localdomain"/>' [root@ha01 ~]# cibadmin --delete --obj_type status --crm_xml '<node_state uname="ha3.localdomain"/>'

3. Put the whole cluster in maintenance mode and restart the Pacemaker and Corosync services. I could not find a way to rename cleanly without doing this, if I don’t I always end up with ghost nodes on the cluster. Because ha01 is the current Master, I would cycle all cluster services – stopping at ha01 last and starting with it first.
   
   [root@ha01 ~]# crm configure property maintenance-mode=true [root@ha02 ~]# service pacemaker stop Signaling Pacemaker Cluster Manager to terminate [ OK ] Waiting for cluster services to unload [ OK ] [root@ha02 ~]# service corosync stop Signaling Corosync Cluster Engine (corosync) to terminate: [ OK ] Waiting for corosync services to unload: [ OK ] [root@ha01 ~]# service pacemaker stop Signaling Pacemaker Cluster Manager to terminate [ OK ] Waiting for cluster services to unload. [ OK ] [root@ha01 ~]# service corosync stop Signaling Corosync Cluster Engine (corosync) to terminate: [ OK ] Waiting for corosync services to unload:. [ OK ] [root@ha01 ~]# service corosync start Starting Corosync Cluster Engine (corosync): [ OK ] [root@ha01 ~]# service pacemaker start Starting Pacemaker Cluster Manager [ OK ] [root@ha02 ~]# service corosync start Starting Corosync Cluster Engine (corosync): [ OK ] [root@ha02 ~]# service pacemaker start Starting Pacemaker Cluster Manager [ OK ]

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

   [root@ha01 ~]# crm configure property maintenance-mode=true   [root@ha02 ~]# service pacemaker stop Signaling Pacemaker Cluster Manager to terminate           [  OK  ] Waiting for cluster services to unload                     [  OK  ] [root@ha02 ~]# service corosync stop Signaling Corosync Cluster Engine (corosync) to terminate: [  OK  ] Waiting for corosync services to unload:                   [  OK  ]   [root@ha01 ~]# service pacemaker stop Signaling Pacemaker Cluster Manager to terminate           [  OK  ] Waiting for cluster services to unload.                    [  OK  ] [root@ha01 ~]# service corosync stop Signaling Corosync Cluster Engine (corosync) to terminate: [  OK  ] Waiting for corosync services to unload:.                  [  OK  ]   [root@ha01 ~]# service corosync start Starting Corosync Cluster Engine (corosync):               [  OK  ] [root@ha01 ~]# service pacemaker start Starting Pacemaker Cluster Manager                         [  OK  ]   [root@ha02 ~]# service corosync start Starting Corosync Cluster Engine (corosync):               [  OK  ] [root@ha02 ~]# service pacemaker start Starting Pacemaker Cluster Manager                         [  OK  ]

4. Disable maintenance-mode and verify the 2 actives nodes are up and running:
   
   [root@ha01 ~]# crm configure property maintenance-mode=false [root@ha01 ~]# crm status Last updated: Wed Jun 4 22:56:24 2014 Last change: Wed Jun 4 22:56:23 2014 via crm_attribute on ha02.localdomain Stack: classic openais (with plugin) Current DC: ha02.localdomain - partition with quorum Version: 1.1.10-14.el6_5.2-368c726 2 Nodes configured, 3 expected votes 5 Resources configured Online: [ ha01.localdomain ha02.localdomain ] Master/Slave Set: ms_MySQL [p_mysql] Masters: [ ha01.localdomain ] Slaves: [ ha02.localdomain ] Clone Set: ping_gw_clone [ping_gw] Started: [ ha01.localdomain ha02.localdomain ]

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   [root@ha01 ~]# crm configure property maintenance-mode=false [root@ha01 ~]# crm status Last updated: Wed Jun  4 22:56:24 2014 Last change: Wed Jun  4 22:56:23 2014 via crm_attribute on ha02.localdomain Stack: classic openais (with plugin) Current DC: ha02.localdomain - partition with quorum Version: 1.1.10-14.el6_5.2-368c726 2 Nodes configured, 3 expected votes 5 Resources configured     Online: [ ha01.localdomain ha02.localdomain ]   Master/Slave Set: ms_MySQL [p_mysql]      Masters: [ ha01.localdomain ]      Slaves: [ ha02.localdomain ] Clone Set: ping_gw_clone [ping_gw]      Started: [ ha01.localdomain ha02.localdomain ]

5. Then start the cluster services on ha03 which I renamed:
   
   [root@ha03 mysql]# service corosync start Starting Corosync Cluster Engine (corosync): [ OK ] [root@ha03 mysql]# service pacemaker start Starting Pacemaker Cluster Manager [ OK ] [root@ha03 mysql]# crm status Last updated: Wed Jun 4 23:14:30 2014 Last change: Wed Jun 4 23:14:25 2014 via crm_attribute on ha02.localdomain Stack: classic openais (with plugin) Current DC: ha02.localdomain - partition with quorum Version: 1.1.10-14.el6_5.2-368c726 3 Nodes configured, 3 expected votes 6 Resources configured Online: [ ha01.localdomain ha02.localdomain ha03.localdomain ] Master/Slave Set: ms_MySQL [p_mysql] Masters: [ ha01.localdomain ] Slaves: [ ha02.localdomain ha03.localdomain ] Clone Set: ping_gw_clone [ping_gw] Started: [ ha01.localdomain ha02.localdomain ha03.localdomain ]

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

   [root@ha03 mysql]# service corosync start Starting Corosync Cluster Engine (corosync):               [  OK  ] [root@ha03 mysql]# service pacemaker start Starting Pacemaker Cluster Manager                         [  OK  ] [root@ha03 mysql]# crm status Last updated: Wed Jun  4 23:14:30 2014 Last change: Wed Jun  4 23:14:25 2014 via crm_attribute on ha02.localdomain Stack: classic openais (with plugin) Current DC: ha02.localdomain - partition with quorum Version: 1.1.10-14.el6_5.2-368c726 3 Nodes configured, 3 expected votes 6 Resources configured     Online: [ ha01.localdomain ha02.localdomain ha03.localdomain ]   Master/Slave Set: ms_MySQL [p_mysql]      Masters: [ ha01.localdomain ]      Slaves: [ ha02.localdomain ha03.localdomain ] Clone Set: ping_gw_clone [ping_gw]      Started: [ ha01.localdomain ha02.localdomain ha03.localdomain ]

Do you have any more convenient way of doing this?