LVS clusters support three configuration modes: DR, TUN, and NAT, capable of load balancing WWW, FTP, MAIL, and other services. Below, we demonstrate configuring an LVS cluster in DR mode through a practical example of load balancing a WWW service.
I. Director Server Configuration
There are two methods to configure an LVS load balancing cluster on the Director Server:
• Configuration via the ipvsadm command line
• Configuration via piranha, a tool provided by Red Hat
1. Configuring LVS via ipvsadm Command Line
After installing IPVS, you can configure the LVS cluster. First, bind a virtual IP (VIP) on the Director Server. This IP is used to provide external services. Execute the following command:
[root@localhost ~]#ifconfig eth0:0 192.168.60.200 broadcast 192.168.60.200 /
>netmask 255.255.255.255 up
Here, a virtual device eth0:0 is bound to the eth0 device, and a virtual IP of 192.168.60.200 is set, which is the IP address we planned earlier. The broadcast address is also set to 192.168.60.200. It is particularly important to note that the subnet mask here is 255.255.255.255.
Next, assign a route to the eth0:0 device by executing the following command:
[root@localhost ~]#route add -host 192.168.60.200 dev eth0:0
Then, enable the system’s packet forwarding function so that the system acts as a router. Execute the following command:
[root@localhost ~]#echo "1" >/proc/sys/net/ipv4/ip_forward
In the command, a parameter value of 1 enables IP forwarding, while 0 disables it. Actually, in DR mode, enabling the system’s packet forwarding function is not mandatory, but it is required in NAT mode.
Next, start configuring ipvs. Execute the following operations:
[root@localhost ~]#ipvsadm -C
[root@localhost ~]#ipvsadm -A -t 192.168.60.200:80 -s rr -p 600
[root@localhost ~]#ipvsadm -a -t 192.168.60.200:80 -r 192.168.60.132:80 -g
[root@localhost ~]#ipvsadm -a -t 192.168.60.200:80 -r 192.168.60.144:80 -g
In the operations above, the first line clears all records from the kernel virtual server list. The second line adds a new virtual IP record. This new IP is 192.168.60.200, and a persistent service time of 600 seconds is specified. The third and fourth lines add two new Real Server records to the newly added virtual IP record and specify the LVS working mode as Direct Routing.
Finally, start the LVS service by executing the following command:
[root@localhost ~]#ipvsadm
Thus, the LVS configuration on the Director Server is complete.
For easier management and configuration, you can write the above operations into a script file with the following content:
#!/bin/bash
VIP=192.168.60.200
RIP1=192.168.60.132
RIP2=192.168.60.144
GW=192.168.60.1
# set the Virtual IP Address
/sbin/ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev eth0:0
echo "1" >/proc/sys/net/ipv4/ip_forward
#Clear IPVS table
/sbin/ipvsadm -C
#set LVS
/sbin/ipvsadm -A -t $VIP:80 -s rr -p 600
/sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -g
/sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -g
#Run LVS
/sbin/ipvsadm
#end
You can also write it as a service script that can be started and stopped, with the following content:
#!/bin/sh
# description: Start LVS of Director server
VIP=192.168.60.200
RIP1=192.168.60.132
RIP2=192.168.60.144
./etc/rc.d/init.d/functions
case "$1" in
start)
echo " start LVS of Director Server"
# set the Virtual IP Address and sysctl parameter
/sbin/ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
echo "1" >/proc/sys/net/ipv4/ip_forward
#Clear IPVS table
/sbin/ipvsadm -C
#set LVS
/sbin/ipvsadm -A -t $VIP:80 -s rr -p 600
/sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -g
/sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -g
#Run LVS
/sbin/ipvsadm
;;
stop)
echo "close LVS Directorserver"
echo "0" >/proc/sys/net/ipv4/ip_forward
/sbin/ipvsadm -C
/sbin/ifconfig eth0:0 down
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
Name this script lvsDR, then place the file in /etc/init.d and execute:
[root@localhost ~]#chomd 755 /etc/init.d/lvsDR
Finally, you can start or stop the LVS service using the following command:
service lvsDR {start|stop}
At this point, the command-line configuration of the Director Server is complete.
2. Configuring LVS via Red Hat’s piranha Tool
Piranha is a web-based LVS configuration software provided by Red Hat, which saves the tedious work of manually configuring LVS. Additionally, it can independently provide cluster functionality; for example, you can use Piranha to activate a backup host for the Director Server, meaning configuring a hot-standby dual-machine setup for the Director Server.
Installing the Piranha tool is very simple. Download the Piranha RPM package and install it:
[root@localhost ~]#rpm –ivh piranha-0.8.2-1.i386.rpm
After Piranha is installed, the /etc/sysconfig/ha/lvs.cf file is generated. By default, this file is empty. You can configure this file via the web interface provided by Piranha, or you can edit it directly by hand. The content of an edited lvs.cf file looks similar to the following. Note that the text after the “#” marks are comments.
[root@localhost ~]# more /etc/sysconfig/ha/lvs.cf
serial_no = 18 # Serial number.
primary = 192.168.60.56 # Specifies the real IP address of the primary Director Server, relative to a backup Director Server, i.e., for setting up an HA Cluster for the Director Server.
service = lvs # Specifies the service name for the dual-machine setup.
backup_active = 0 # Whether to activate the backup Director Server. “0” means not activate, “1” means activate.
backup = 0.0.0.0 # Specifies the real IP address of the backup Director Server here. If there is no backup Director Server, use “0.0.0.0” instead.
heartbeat = 0 # Whether to enable heartbeat. 1 means enable, 0 means disable.
heartbeat_port = 539 # Specifies the UDP communication port for heartbeat.
keepalive = 5 # Heartbeat interval in seconds.
deadtime = 10 # If the primary Director Server does not respond after deadtime (seconds), the backup Director Server will take over its service.
network = direct # Specifies the LVS working mode. direct means DR mode, nat means NAT mode, tunnel means TUNL mode.
debug_level = NONE # Defines the debug information level.
virtual www.gaojf.com{ # Specifies the name of the virtual service.
active = 1 # Whether to activate this service.
address = 192.168.60.200 eth0:0 # The virtual IP and network device name bound to the virtual service.
port = 80 # Virtual service port.
send = "GET / HTTP/1.0/r/n/r/n" # Verification string sent to the real server.
expect = "HTTP" # The text response expected from the server when it is operating normally, used to determine if the real server is working properly.
use_regex = 0 # Whether to use regular expressions in the expect option. 0 means no, 1 means yes.
load_monitor = none # The Director Server in LVS can use rup or ruptime to monitor the load status of each real server. This option has three possible values: rup, ruptime, and none. If rup is selected, each real server must run the rstatd service. If ruptime is selected, each real server must run the rwhod service.
scheduler = rr # Specifies the LVS scheduling algorithm.
protocol = tcp # The protocol type used by the virtual service.
timeout = 6 # The time, in seconds, that must elapse after a real server fails before it is removed from the LVS routing list.
reentry = 15 # The time, in seconds, that must elapse after a real server is removed before it can be re-added to the LVS routing list.
quiesce_server = 0 # If this option is set to 1, when a new node joins the cluster, the least connection count will be reset to zero, which may cause LVS to flood the new node with requests, blocking its service. It is recommended to set this to 0.
server RS1 { # Specifies the real server service name.
address = 192.168.60.132 # Specifies the IP address of the real server.
active = 1 # Whether to activate this real server service.
weight = 1 # Specifies the weight of this real server. It is an integer value relative to all real server nodes. A higher weight indicates stronger performance for handling load.
}
server RS2 {
address = 192.168.60.144
active = 1
weight = 1
}
}
After editing, start the pulse service to launch the LVS service:
[root@localhost ~]#service pulse start
Similarly, this method also requires enabling the system’s packet forwarding feature:
[root@localhost ~]#echo "1" >/proc/sys/net/ipv4/ip_forward
This completes the configuration of the Director Server using the Piranha GUI tool.
II. Real Server Configuration
In LVS’s DR and TUN modes, user access requests, after reaching the real servers, are returned directly to the user without passing back through the front-end Director Server. Therefore, it is necessary to add a virtual VIP address on each Real server node so that data can be returned directly to the user. This can be achieved by creating a script. Create the file /etc/init.d/lvsrs with the following content:
#!/bin/bash
VIP=192.168.60.200
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
echo “1″ >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo “2″ >/proc/sys/net/ipv4/conf/lo/arp_announce
echo “1″ >/proc/sys/net/ipv4/conf/all/arp_ignore
echo “2″ >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p
#end
This operation binds a virtual IP address to the loopback device, sets its subnet mask to 255.255.255.255 for communication with the Director Server’s virtual IP, and then suppresses the local machine’s ARP requests.
The script above can also be written as a service script for starting and stopping, as follows:
[root@localhost ~]#more /etc/init.d/lvsrs
#!/bin/bash
#description : Start Real Server
VIP=192.168.60.200
./etc/rc.d/init.d/functions
case "$1" in
start)
echo " Start LVS of Real Server"
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
/sbin/ifconfig lo:0 down
echo "close LVS Director server"
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
Then, make lvsrs executable:
[root@localhost ~]#chomd 755 /etc/init.d/lvsrs
Finally, you can start or stop lvsrs with the following command:
service lvsrs {start|stop}
Since the virtual IP (the VIP address above) is shared by the Director Server and all Real servers, a problem arises if both the Director Server and all Real servers respond to ARP requests for the VIP address. Therefore, it is necessary to prevent the Real servers from responding to ARP requests. The lvsrs script’s purpose is to make the Real Server not respond to ARP requests.
For lower kernel versions below 2.4, the ARP hidden kernel patch must be installed on the Real Server. Fortunately, in the 2.6 kernel, ARP broadcast replies can be ignored by adjusting kernel parameters.
III. Configuring Redundancy Strategy on the Director: Ldirectord
The function of Ldirectord is to monitor Real Servers. When a Real Server fails, it is removed from the virtual server list, and re-added when it recovers. As mentioned in a previous article about heartbeat, Ldirectord is installed by default. Therefore, only configuration is needed here. The configuration file for Ldirectord is /etc/ha.d/ldirectord.cf.
Below are the options to configure. Note: the text after the “#” marks are comments:
# Global Directives
checktimeout=20 # The time interval for determining if a real server has an error.
checkinterval=10 # Specifies the interval time between two checks by ldirectord.
fallback=127.0.0.1:80 # The web service redirect address when all real server nodes are down.
autoreload=yes # Whether to automatically reload the configuration file. If set to yes, the config info is automatically loaded when the config file changes.
logfile="/var/log/ldirectord.log" # Set the ldirectord log output file path.
quiescent=no # When set to no, if a node does not respond within the time period set by checktimeout, ldirectord will directly remove the real server from the LVS routing table. This will interrupt existing client connections and cause LVS to drop all connection tracking records and persistent connection templates. If set to yes, when a real server fails, ldirectord sets the weight of the failed node to 0. New connections will not reach it, but the node is not removed from the LVS routing table. Meanwhile, connection tracking records and persistent connection templates remain on the Director.
Note: The lines above are the “global” settings for the ldirectord.cf file. They can apply to multiple virtual hosts below. The following are configurations for each virtual host.
# Sample for an http virtual service
virtual=192.168.60.200:80 # Specifies the virtual IP address and port number. Note that lines following the virtual line must be indented with 4 spaces or marked with a tab character.
real=192.168.60.132:80 gate # Specifies the Real Server address and port, and sets the LVS working mode. Use gate for DR mode, ipip for TUNL mode, and masq for NAT mode.
real=192.168.60.144:80 gate
fallback=127.0.0.1:80 gate
service=http # Specifies the service type; here, it’s load balancing for the HTTP service.
request="index.html" # ldirectord will send access requests to the specified Real Server address combined with the request path given by this option to check if the service on the Real Server is running normally. Ensure the page address provided here is accessible; otherwise, ldirectord may mistakenly assume the node has failed, causing erroneous monitoring.
receive="Test Page" # Specifies the request and response strings.
scheduler=rr # Specifies the scheduling algorithm; here it is rr (Round-Robin).
protocol=tcp # Specifies the protocol type; LVS supports TCP and UDP protocols.
checktype=negotiate # Specifies the Ldirectord check type; the default is negotiate.
checkport=80 # Specifies the monitoring port number.
virtualhost=www.gaojf.com # The name of the virtual server; can be set arbitrarily.
Once the configuration is complete, you can execute the following commands to start or stop the ldirectord service.
/etc/init.d/ldirectord {start|stop}
The core function of Ldirectord is to monitor the status of Real Server nodes. Additionally, it can automatically call ipvsadm to create the LVS routing table, which can be seen from ldirectord.cf.
It should be noted that both ldirectord and Piranha have the capability to monitor Real Servers. If you want to monitor node status through ldirectord, you only need to start the ldirectord service, and the entire cluster system can run without executing the LVS script we configured earlier. This is because ldirectord automatically calls ipvsadm to create the LVS routing table. The method of configuring LVS using the ipvsadm command line, which we described earlier, is to give readers a deeper understanding of the implementation details and mechanisms of ipvsadm.
If you configure LVS through the Piranha tool, there is no need to use ldirectord. The system process corresponding to the Piranha tool is pluse. This process will also automatically call ipvsadm to create the LVS routing table and simultaneously use its own nanny daemon to monitor the status of real servers!