1. What is DRBD
DRBD (Distributed Replicated Block Device), often referred to as “Network RAID,” is an open-source software developed by LINBIT.
2. Primary Functions of DRBD
DRBD is essentially a block device implementation, primarily used in High Availability (HA) solutions for the Linux platform. It consists of kernel modules and associated programs, synchronizing and mirroring an entire device over the network, much like a network RAID function. This means when you write data to the filesystem on the local DRBD device, the data is simultaneously sent to another host on the network and recorded in an identical manner on a filesystem there (in fact, the creation of the filesystem is also achieved through DRBD synchronization). Data between the local node (host) and the remote node (host) is guaranteed real-time synchronization, ensuring IO consistency. Therefore, if the local node fails, the remote node retains an identical copy of the data, which can be used to continue operations, achieving high availability.
3. Main Applications of DRBD
The damage caused by a primary server crashing is immeasurable. To ensure uninterrupted service from the primary server, server redundancy is required. Among numerous server redundancy solutions, Heartbeat provides us with an affordable, scalable high-availability clustering solution. By using Heartbeat + DRBD, we can create a High Availability (HA) cluster server on Linux. Using DRBD’s functionality within an HA solution can replace the use of a shared disk array storage device. Because data exists on both the local and remote hosts simultaneously, when a failover is necessary, the remote host only needs to use its backup data copy to continue providing services.
4. The Relationship Between DRBD and MySQL
MySQL and LINBIT have entered a partnership, and the high-profile “12 Days of Scale-Out” campaign is likely driven by this commercial collaboration. DRBD empowers MySQL, boasting up to four nines (99.99%) of reliability, which is no less than that of any commercial database software.
The emergence of DRBD has indeed greatly improved the availability of MySQL clusters. Moreover, it has unique characteristics that are very suitable for Internet-facing applications. Because data block synchronization occurs at the storage layer, it’s easy to achieve IO load balancing at the application layer (with the standby server handling some read pressure). It not only supports database failover but also IP failover, with a takeover time of less than 30 seconds. It is truly an excellent cluster solution for those on a budget.
The test environment mentioned in this article is as follows:
Operating System:
Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
Software:
drbd-8.2.6.tar.gz
heartbeat-2.1.3-3.el4.centos
heartbeat-pils-2.1.3-3.el4.centos
heartbeat-stonith-2.1.3-3.el4.centos
mysql-5.1.26-rc-linux-i686-icc-glibc23.tar.gz
Host Environment:
DRBD Host List IP Address Hostname
Host 1 (primary) 192.168.1.241 drbd-1
Host 2 (secondary) 192.168.1.242 drbd-2
Additionally, both hosts have a reserved empty partition: /dev/sdb1, with no filesystem created.
1. Compiling and Installing DRBD and Heartbeat
DRBD must be installed on both the primary and secondary machines.
[root@drbd-1 ~] tar -xvzf drbd-8.2.6.tar.gz
[root@drbd-1 ~] cd drbd-8.2.6 && make && make rpm
[root@drbd-1 ~] cd dist/RPMS/i386
[root@drbd-1 ~] ls
drbd-8.2.6-3.i386.rpm
drbd-debuginfo-8.2.6-3.i386.rpm
drbd-km-2.6.9_42.EL-8.2.6-3.i386.rpm
[root@drbd-1 ~] rpm -ivh drbd-8.2.6-3.i386.rpm
[root@drbd-1 ~] rpm -ivh drbd-debuginfo-8.2.6-3.i386.rpm
[root@drbd-1 ~] rpm -ivh drbd-km-2.6.9_42.EL-8.2.6-3.i386.rpm
[root@drbd-1 ~] yum install heartbeat
Packages installed via yum upgrade are saved in: /var/cache/yum/extras/packages
Installing MySQL is too simple to elaborate on here!
2. Loading the DRBD Module
[root@drbd-1 ~] modprobe drbd
[root@drbd-1 ~] lsmod |grep drbd
drbd 242924 2
If you see output, it indicates success!
3. Configuring /etc/drbd.conf
Edit the configuration file. The content on both hosts is identical, as follows:
# Allow LINBIT to collect DRBD usage data, yes to participate.
global {
usage-count yes;
}
# Common settings for multiple resources managed by DRBD, mainly for configuring parameters that all DRBD resources can share, such as protocol, syncer, etc.
common {
syncer { rate 100M; }
}
# Create a resource named “db”
resource db {
# Use Protocol C, meaning a write is considered complete once the remote host’s write acknowledgement is received.
protocol C;
startup {
wfc-timeout 0;
degr-wfc-timeout 120;
}
# Since the two servers’ hard disk sizes might differ in the test environment, the DRBD size needs to be set.
disk {
on-io-error detach;
size 6G;
}
net {
max-buffers 2048;
ko-count 4;
}
syncer {
rate 100M;
}
# Define a node, named using its respective hostname
on drbd-1 {
# Set the resource device /dev/drbd0 to point to the actual physical partition /dev/sdb1
device /dev/drbd0;
disk /dev/sdb1;
# Set the listening address and port
address 192.168.1.241:8888;
# Set metadata storage method: can use internal (i.e., stored on the same physical partition)
# or stored on another partition
meta-disk internal;
}
on drbd-2 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.242:8888;
meta-disk internal;
}
}
4. Starting DRBD
Before starting, you need to create the corresponding metadata blocks on the /dev/sdb1 partition on both hosts:
[root@drbd-1 ~] drbdadm create-md db
[root@drbd-2 ~] drbdadm create-md db
Input “yes” twice. If the following prompt appears, it indicates success:
[root@drbd-1 /]# drbdadm create-md db
md_offset 8587153408
al_offset 8587120640
bm_offset 8586858496
Found ext3 filesystem which uses 6291456 kB
current configuration leaves usable 8385604 kB
==> This might destroy existing data! wait forever)
To abort waiting enter 'yes' [ 47]:
At this point, the DRBD services on both machines are up. Check if the processes exist:
[root@drbd-1 /]# ps aux | grep drbd
root 3758 14.5 0.0 0 0 ? S 15:56 0:20 [drbd0_worker]
root 3762 9.6 0.0 0 0 ? S 15:56 0:13 [drbd0_receiver]
root 3787 2.4 0.0 0 0 ? S 15:56 0:03 [drbd0_asender]
root 3794 0.0 0.2 644 128 pts/0 R+ 15:58 0:00 grep drbd
You can see the processes for both nodes are running. Each DRBD device will have three processes: drbd0_worker is the main process for drbd0, drbd0_asender is the data sending process on the primary, and drbd0_receiver is the data receiving process on the secondary.
Check the DRBD status just after startup:
[root@drbd-1 /]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by root@drbd-1, 2008-09-17 17:46:45
0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r—
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:6291456
[root@drbd-2 /]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4/> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:6291456
Note: At this point, both servers are in the secondary state (st:Secondary/Secondary) because we haven’t designated which one is the primary yet.
Next, set one of the started nodes as the primary. We’ll set drbd-1 as the primary:
[root@drbd-1 /]# drbdadm primary db
State change failed: (-2) Refusing to be Primary without at least one UpToDate disk
Command 'drbdsetup /dev/drbd0 primary' terminated with exit code 11
[root@drbd-1 /]# drbdsetup /dev/drbd0 primary -o
As you can see, the drbdadm command fails the first time you try to set the primary node, so use drbdsetup first. After that, you can use drbdadm going forward.
Check the DRBD status on both servers again:
[root@drbd-1 /]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by root@drbd-1, 2008-09-17 17:46:45
0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r—
ns:3483280 nr:0 dw:0 dr:3491456 al:0 bm:212 lo:1 pe:8 ua:256 ap:0 oos:2808416
[==========>………] sync'ed: 55.5% (2742/6144)M
finish: 0:11:24 speed: 4,084 (4,648) K/sec
[root@drbd-2 /]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by root@drbd-2, 2008-09-17 17:51:50
0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r—
ns:0 nr:3556832 dw:3556832 dr:0 al:0 bm:217 lo:1 pe:2464 ua:0 ap:0 oos:2734624
[==========>………] sync'ed: 56.7% (2670/6144)M
finish: 0:07:35 speed: 5,856 (4,128) K/sec
At this point, you can see that data synchronization has begun. The first synchronization after this setup takes a relatively long time because the entire partition’s data needs to be synced.
Once the first synchronization is complete, you can create a filesystem on the DRBD device:
[root@drbd-1 /]# mkfs.ext3 /dev/drbd0
Mount the filesystem:
[root@drbd-1 /]# mount /dev/drbd0 /drbddata
Test data writes on the primary node:
[root@drbd-1 drbddata]# ll
total 4
drwx—— 4 mysql root 4096 Oct 13 16:29 mysql
Then, we demote the primary to secondary and promote the secondary to primary:
[root@drbd-1 /]# umount /drbddata/
[root@drbd-1 /]# drbdadm secondary db
Before demoting the primary, you must unmount the device first. Then promote the secondary:
[root@drbd-2 /]# drbdadm primary db
[root@drbd-2 /]# mount /dev/drbd0 /drbddata/
[root@drbd-2 drbddata]# ll
total 4
drwx—— 4 mysql root 4096 Oct 13 16:29 mysql
As you can see, the data has been fully synchronized over.
Next, we need to integrate with heartbeat. Since heartbeat was already installed earlier, we just need to modify the configuration files.
cp /usr/share/doc/heartbeat-2.1.3/ha.cf .
cp /usr/share/doc/heartbeat-2.1.3/authkeys .
cp /usr/share/doc/heartbeat-2.1.3/haresources .
Configure ha.cf (the main HA configuration file):
[root@drbd-1 ha.d]# more ha.cf
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
bcast eth0
auto_failback off
node drbd-1
node drbd-2
ping_group group1 192.168.1.1 192.168.1.254
respawn root /usr/lib/heartbeat/ipfail
apiauth ipfail gid=root uid=root
Configure authkeys authentication:
[root@drbd-1 ha.d]# more authkeys
auth 1
1 crc
Configure the haresources resource file:
drbd-1 drbddisk Filesystem::/dev/drbd0::/drbddata::ext3 mysql 192.168.1.243
Note:
The resource group configuration file is mainly for configuring the various resources that need to be managed during the failover process. A crucial point is that the order of resources in a resource group matters. When heartbeat manages a resource group, it acquires resources from left to right and releases them from right to left.
The first column in the resource group is one of the nodes from our ha.cf configuration file, and it should be the node currently intended to serve as the primary.
The meaning of each item in the resource group above is as follows:
drbd-1 The current primary node name (uname -n)
drbddisk Tells heartbeat to manage the DRBD resource
Filesystem Tells heartbeat to manage the filesystem resource, which essentially means executing mount/umount commands. The parameters after the “::” are the Filesystem parameters (device name and mount point).
mysql Tells heartbeat to manage MySQL
192.168.1.243 This tells heartbeat to manage a service IP that will float along with the primary node.
Testing failover:
1) Manually invoke heartbeat’s node switching script:
Execute the /usr/lib/heartbeat/hb_standby script to tell heartbeat to notify the peer node that it wants to become the standby node and request the peer to become the primary. The switch completes in about 10 seconds.
2) Unplug the network cable to test failover when the primary node’s network is disconnected:
Through testing, after unplugging the network cable, when the primary node discovers it cannot communicate with the standby node, it will record a warning message in the log. If the duration reaches the time set in ha.cf, it will begin releasing resources. The standby node, after detecting it cannot communicate with the primary for a period (also set in ha.cf), will attempt to start resources and make itself the active primary node. The failover time, apart from the delay configured in ha.cf, is also very short.
3) Shut down the primary host to test if it can fail over normally. This is basically the same as test 2 above.
4) Primary node power loss test. We haven’t actually performed this operation in the server room yet, but will continue testing this operation later.
Test results:
1, Normal failover, data integrity maintained.
2, Normal failover, but primary and secondary data differ by one record.
3, Normal failover, data integrity maintained.
4, Normal failover, data integrity maintained.