Home › High Performance › Cluster › Heartbeat 3.x Complete Application Guide: Testing Heartbeat HA Functionality

Heartbeat 3.x Complete Application Guide: Testing Heartbeat HA Functionality

Categories: Cluster 2013-09-14 root Comments (0)

I. Starting Heartbeat

1. Start Heartbeat on the Primary Node
After Heartbeat installation completes, a startup script file heartbeat is automatically generated in the /etc/init.d directory. Running /etc/init.d/heartbeat directly shows the usage of the heartbeat script, as shown below:
 
  1. [root@node1 ~]# /etc/init.d/heartbeat   
  2. Usage: /etc/init.d/heartbeat {start|stop|status|restart|reload|force-reload} 
Therefore, Heartbeat can be started with the following command:
 
  1. [root@node1 ~]#service heartbeat start 
  2. Or via 
  3. [root@node1 ~]#/etc/init.d/heartbeat start 
This starts the Heartbeat service on the primary node.
The log information is as follows:
 
  1. Feb  5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: bound send socket to device: eth0 
  2. Feb  5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: bound receive socket to device: eth0 
  3. Feb  5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: started on port 694 interface eth0 to 192.168.12.1 
  4. Feb  5 19:09:48 node1 heartbeat: [22768]: info: glib: ping heartbeat started. 
  5. Feb  5 19:09:48 node1 heartbeat: [22768]: info: glib: ping group heartbeat started. 
  6. Feb  5 19:09:48 node1 heartbeat: [22768]: info: Local status now set to: 'up' 
  7. Feb  5 19:09:49 node1 heartbeat: [22768]: info: Link 192.168.12.1:192.168.12.1 up. 
  8. Feb  5 19:09:49 node1 heartbeat: [22768]: info: Status update for node 192.168.12.1: status ping 
  9. Feb  5 19:09:49 node1 heartbeat: [22768]: info: Link group1:group1 up. 
  10. Feb  5 19:09:49 node1 heartbeat: [22768]: info: Status update for node group1: status ping 
This section of the log shows Heartbeat performing initialization configuration, such as heartbeat interval time, UDP broadcast port, ping node operational status, etc. The log information pauses here and waits 120 seconds before Heartbeat continues to output logs. This 120-second period corresponds exactly to the “initdead” option setting in ha.cf. At this point, Heartbeat’s output information is as follows:
 
  1. Feb  5 19:11:48 node1 heartbeat: [22768]: WARN: node node2: is dead 
  2. Feb  5 19:11:48 node1 heartbeat: [22768]: info: Comm_now_up(): updating status to active 
  3. Feb  5 19:11:48 node1 heartbeat: [22768]: info: Local status now set to: 'active' 
  4. Feb  5 19:11:48 node1 heartbeat: [22768]: info: Starting child client "/usr/local/ha/lib/heartbeat/pingd -m 100 -d 5s" (102,105) 
  5. Feb  5 19:11:49 node1 heartbeat: [22768]: WARN: No STONITH device configured. 
  6. Feb  5 19:11:49 node1 heartbeat: [22768]: WARN: Shared disks are not protected. 
  7. Feb  5 19:11:49 node1 heartbeat: [22768]: info: Resources being acquired from node2. 
  8. Feb  5 19:11:49 node1 heartbeat: [22794]: info: Starting "/usr/local/ha/lib/heartbeat/pingd -m 100 -d 5s" as uid 102  gid 105 (pid 22794) 
     In the above log, since node2 has not yet been started, a “node2: is dead” warning message appears. Next, the heartbeat plugin pingd is started. Since we did not configure STONITH in the ha.cf file, the log also gives a “No STONITH device configured” warning prompt. 
Continue reading the log below:
 
  1. Feb  5 19:11:50 node1 IPaddr[22966]: INFO:  Resource is stopped 
  2. Feb  5 19:11:50 node1 ResourceManager[22938]: info: Running /usr/local/ha/etc/ha.d/resource.d/IPaddr 192.168.12.135 start 
  3. Feb  5 19:11:50 node1 IPaddr[23029]: INFO: Using calculated nic for 192.168.12.135: eth0 
  4. Feb  5 19:11:50 node1 IPaddr[23029]: INFO: Using calculated netmask for 192.168.12.135: 255.255.255.0 
  5. Feb  5 19:11:51 node1 pingd: [22794]: info: attrd_lazy_update: Connecting to cluster… 5 retries remaining 
  6. Feb  5 19:11:51 node1 IPaddr[23029]: INFO: eval ifconfig eth0:0 192.168.12.135 netmask 255.255.255.0 broadcast 192.168.12.255 
  7. Feb  5 19:11:51 node1 avahi-daemon[2455]: Registering new address record for 192.168.12.135 on eth0. 
  8. Feb  5 19:11:51 node1 IPaddr[23015]: INFO:  Success 
  9. Feb  5 19:11:51 node1 Filesystem[23134]: INFO:  Resource is stopped 
  10. Feb  5 19:11:51 node1 ResourceManager[22938]: info: Running /usr/local/ha/etc/ha.d/resource.d/Filesystem /dev/sdf1 /data1 ext3 start 
  11. Feb  5 19:11:52 node1 Filesystem[23213]: INFO: Running start for /dev/sdf1 on /data1 
  12. Feb  5 19:11:52 node1 kernel: kjournald starting.  Commit interval 5 seconds 
  13. Feb  5 19:11:52 node1 kernel: EXT3 FS on sdf1, internal journal 
  14. Feb  5 19:11:52 node1 kernel: EXT3-fs: mounted filesystem with ordered data mode. 
  15. Feb  5 19:11:52 node1 Filesystem[23205]: INFO:  Success 
The above log shows resource monitoring and takeover, primarily executing the settings in the haresources file. In this case, it enables the cluster virtual IP and mounts the disk partition.
At this point, by using the ifconfig command to check the network configuration of the primary node, you can see that the primary node has automatically bound the cluster IP address. Using the ping command from a host outside the HA cluster to test the cluster IP address 192.168.12.135, it is already in a reachable state, meaning the address has become available.
Also, checking the disk partition mount status, the shared disk partition /dev/sdf1 has been automatically mounted. 
 
2. Start Heartbeat on the Standby Node
To start Heartbeat on the backup node, use the same method as the primary node with the following command: 
 
  1. [root@node2 ~]#/etc/init.d/heartbeat start   
  2.  Or execute 
  3. [root@node2 ~]#service heartbeat start  
   This starts the Heartbeat service on the standby node. The heartbeat log output information on the standby node corresponds to that of the primary node. By using “tail -f /var/log/messages”, you can see the following output:
 
  1.  Feb 19 02:52:15 node2 heartbeat: [26880]: info: Pacemaker support: false 
  2. Feb 19 02:52:15 node2 heartbeat: [26880]: info: ************************** 
  3. Feb 19 02:52:15 node2 heartbeat: [26880]: info: Configuration validated. Starting heartbeat 3.0.4 
  4. Feb 19 02:52:15 node2 heartbeat: [26881]: info: heartbeat: version 3.0.4 
  5. Feb 19 02:52:15 node2 heartbeat: [26881]: info: Heartbeat generation: 1297766398 
  6. Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: UDP multicast heartbeat started for group 225.0.0.1 port 694 interface eth0 (ttl=1 loop=0) 
  7. Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0 
  8. Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ucast: bound send socket to device: eth0 
  9. Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ping heartbeat started. 
  10. Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ping group heartbeat started. 
  11. Feb 19 02:52:15 node2 heartbeat: [26881]: info: Local status now set to: 'up' 
  12. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link node1:eth0 up. 
  13. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Status update for node node1: status active 
  14. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link 192.168.12.1:192.168.12.1 up. 
  15. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Status update for node 192.168.12.1: status ping 
  16. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link group1:group1 up. 
  17. Feb 19 02:52:16 node2 harc[26894]: info: Running /usr/local/ha/etc/ha.d//rc.d/status status 
  18. Feb 19 02:52:17 node2 heartbeat: [26881]: info: Comm_now_up(): updating status to active 
  19. Feb 19 02:52:17 node2 heartbeat: [26881]: info: Local status now set to: 'active' 
 
II. Testing Heartbeat High Availability Functionality
How can you determine if the HA cluster is working properly? Simulated environment testing is a good approach. Before deploying the Heartbeat high-availability cluster into a production environment, you need to perform the following test steps to verify that HA is working correctly: 
(1) Gracefully Shut Down and Restart the Primary Node’s Heartbeat
First, execute “service heartbeat stop” on the primary node node1 to gracefully shut down the primary node’s Heartbeat process. At this point, check the primary node’s network interface information using the ifconfig command. Under normal circumstances, you should see that the primary node has released the cluster service IP address and also released the mounted shared disk partition. Then check the backup node, which should now have taken over the cluster service IP and automatically mounted the shared disk partition.
During this process, by testing the cluster service IP with the ping command, you can see that the cluster IP remains continuously reachable without any latency or blocking. In other words, when gracefully shutting down the primary node, the primary-backup node switchover is seamless, and the services provided by HA continue to run uninterrupted.
Next, start the primary node’s Heartbeat normally. After Heartbeat starts, the backup node will automatically release the cluster service IP and unmount the shared disk partition, while the primary node will again take over the cluster service IP and mount the shared disk partition. In fact, the backup node releasing resources and the primary node binding resources happen simultaneously. Therefore, this process is also a seamless switchover.
(2) Unplug the Network Cable from the Primary Node
After unplugging the network cable connecting the primary node to the public network, the heartbeat plugin ipfail can immediately detect the network connection failure through ping testing, and then automatically release resources. At the same moment, the ipfail plugin on the standby node will also detect that the primary node has encountered a network failure. After waiting for the primary node to finish releasing resources, the standby node immediately takes over the cluster resources, thereby ensuring uninterrupted continuous network service operation.
Similarly, when the primary node’s network returns to normal, since the “auto_failback on” option is configured, the cluster resources will automatically switch back from the standby node to the primary node.
(3) Shut Down the Primary Node’s System
After unplugging the primary node’s power, the standby node’s heartbeat process will immediately receive the message that the primary node has shut down. The standby node then begins resource takeover. This situation is actually similar to the primary node network failure scenario.
(4) Cause the Primary Node System Kernel to Crash
When the primary node system crashes, the network loses response, and the standby node’s heartbeat process will immediately detect the primary node network failure and then perform resource switchover. However, since the primary node’s system kernel has crashed, it cannot unmount its own occupied resources, such as the shared disk partition, cluster service IP, etc. In this case, without a Stonith-like device, resource contention will occur. However, if a Stonith device is present, the Stonith device will first power off the failed primary node or restart it, thus forcing the primary node to release the cluster resources. Once the Stonith device completes all operations, the backup node then obtains ownership of the primary node’s resources and takes over them.

Related Posts

Leave a Comment

Your email address will not be published.