I. Introduction
rsync is a data mirroring backup tool for Unix-like systems, as can be seen from its naming—remote sync. Its features are as follows:
1. Can mirror and preserve entire directory trees and file systems.
2. Can easily preserve original file permissions, timestamps, soft/hard links, etc.
3. No special privileges required for installation.
4. Optimized workflow with high file transfer efficiency.
5. Can transfer files using rcp, ssh, etc., and of course also via direct socket connections.
6. Supports anonymous transfers.
1. Can mirror and preserve entire directory trees and file systems.
2. Can easily preserve original file permissions, timestamps, soft/hard links, etc.
3. No special privileges required for installation.
4. Optimized workflow with high file transfer efficiency.
5. Can transfer files using rcp, ssh, etc., and of course also via direct socket connections.
6. Supports anonymous transfers.
II. Version Selection
Currently, the rsync versions included in CentOS 5 are all 2.6.x versions. The main issues with these versions are performance-related — for example, memory overflow or sync interruption tends to occur when syncing a large number of small files. This is mainly related to the 2.6 version’s processing mechanism of listing all files first before syncing. When handling large file syncs, incomplete file sync also frequently occurs. Therefore, before using rsync, we should uninstall the old version and install the latest stable version, which is quite effective at fixing various known bugs. For instance, versions 3.0 and later changed the processing mechanism to synchronize files while listing them simultaneously, greatly improving performance and reducing error probability. The latest version currently is 3.0.7.
Currently, the rsync versions included in CentOS 5 are all 2.6.x versions. The main issues with these versions are performance-related — for example, memory overflow or sync interruption tends to occur when syncing a large number of small files. This is mainly related to the 2.6 version’s processing mechanism of listing all files first before syncing. When handling large file syncs, incomplete file sync also frequently occurs. Therefore, before using rsync, we should uninstall the old version and install the latest stable version, which is quite effective at fixing various known bugs. For instance, versions 3.0 and later changed the processing mechanism to synchronize files while listing them simultaneously, greatly improving performance and reducing error probability. The latest version currently is 3.0.7.
III. Upgrade Steps:
1. Uninstall the current rsync package: you can directly use rpm -e rsync.
Of course, a better approach is:
a. First, check the location of the rsync command: which rsync. The which command searches the $PATH environment variable in order to find the first rsync command path. No matter how many rsync commands exist on your system, the result from which is the one used when you type rsync in the command line. For example: /usr/bin/rsync
b. Then check which package this command belongs to: rpm -qf /usr/bin/rsync. For example: rsync-2.6.8-3.1
c. Uninstall: rpm -e rsync-2.6.8-3.1
1. Uninstall the current rsync package: you can directly use rpm -e rsync.
Of course, a better approach is:
a. First, check the location of the rsync command: which rsync. The which command searches the $PATH environment variable in order to find the first rsync command path. No matter how many rsync commands exist on your system, the result from which is the one used when you type rsync in the command line. For example: /usr/bin/rsync
b. Then check which package this command belongs to: rpm -qf /usr/bin/rsync. For example: rsync-2.6.8-3.1
c. Uninstall: rpm -e rsync-2.6.8-3.1
2. Install the new version:
a. Download: normally we use the wget command, but in fact many attacks exploiting weak system passwords use this command to download attack packages. So, it is best to move this command out of the $PATH directory and put it somewhere only you know. In the future, you can type the full path to use it, which does not affect anything.
#wget http://samba.anu.edu.au/ftp/rsync/src/rsync-3.0.7.tar.gz
b. Installation: very simple, no special parameters needed, defaults are fine
#tar zxvf rsync-3.0.7.tar.gz
#cd rsync-3.0.7
#./configure && make && make install
This completes the installation. By default, the rsync command is installed here: /usr/local/bin/rsync
a. Download: normally we use the wget command, but in fact many attacks exploiting weak system passwords use this command to download attack packages. So, it is best to move this command out of the $PATH directory and put it somewhere only you know. In the future, you can type the full path to use it, which does not affect anything.
#wget http://samba.anu.edu.au/ftp/rsync/src/rsync-3.0.7.tar.gz
b. Installation: very simple, no special parameters needed, defaults are fine
#tar zxvf rsync-3.0.7.tar.gz
#cd rsync-3.0.7
#./configure && make && make install
This completes the installation. By default, the rsync command is installed here: /usr/local/bin/rsync
3. Verify:
#which rsync. Check whether the rsync command can be found. Normally, Linux’s $PATH variable places /usr/local/bin before /usr/bin, so even if you do not uninstall the old version, the new version will be used by default. Of course, I still recommend uninstalling the old version for a clean system, hehe.
#rsync –version. Check whether it is the new version. If it says the command cannot be found, that is due to shell caching — logging in again should fix it.
#which rsync. Check whether the rsync command can be found. Normally, Linux’s $PATH variable places /usr/local/bin before /usr/bin, so even if you do not uninstall the old version, the new version will be used by default. Of course, I still recommend uninstalling the old version for a clean system, hehe.
#rsync –version. Check whether it is the new version. If it says the command cannot be found, that is due to shell caching — logging in again should fix it.
IV. Specific Configuration
Generally, there are two configuration methods: push and pull. Usually we choose the former, depending on your situation.
Push: The destination host is configured as an rsync server, and the source host periodically uses the rsync command to push the directories to be synced.
Pull: The source host is configured as an rsync server, and the destination host periodically uses the rsync command to pull the directories to be synced.
Generally, there are two configuration methods: push and pull. Usually we choose the former, depending on your situation.
Push: The destination host is configured as an rsync server, and the source host periodically uses the rsync command to push the directories to be synced.
Pull: The source host is configured as an rsync server, and the destination host periodically uses the rsync command to pull the directories to be synced.
1. Server-side configuration:
Main configuration file, /etc/rsyncd.conf:
motd file=/etc/rsyncd.motd
uid = root
gid = root
max connections = 0 # Unlimited max connections
use chroot = no # Do not use chroot
log file = /var/log/rsyncd.log # Log file
log format = %t %a %m %f %b # Log format
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
timeout = 300
Main configuration file, /etc/rsyncd.conf:
motd file=/etc/rsyncd.motd
uid = root
gid = root
max connections = 0 # Unlimited max connections
use chroot = no # Do not use chroot
log file = /var/log/rsyncd.log # Log file
log format = %t %a %m %f %b # Log format
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
timeout = 300
[optsoftware] # Define a sync service sub-item named optsoftware (if multiple directories need syncing, this block can be added for each one)
path = /opt/software # Directory to be mirrored. Since we use the push method, this directory represents where files pushed from the source host are stored.
list = yes # Allow file listing
comment = rsync software
ignore errors = yes # Can ignore some non-critical IO errors
read only = no # Disable read-only
hosts allow = 192.168.1.37 # Which hosts are allowed to access
hosts deny = * # Which hosts are denied access
auth users = rsyncuser # Authentication username
secrets file = /etc/rsyncd.secrets # Password file
path = /opt/software # Directory to be mirrored. Since we use the push method, this directory represents where files pushed from the source host are stored.
list = yes # Allow file listing
comment = rsync software
ignore errors = yes # Can ignore some non-critical IO errors
read only = no # Disable read-only
hosts allow = 192.168.1.37 # Which hosts are allowed to access
hosts deny = * # Which hosts are denied access
auth users = rsyncuser # Authentication username
secrets file = /etc/rsyncd.secrets # Password file
Password authentication file, /etc/rsyncd.secrets:
rsyncuser:rsyncuser
A. The first part is the username, the second is the password.
B. This file must have 600 permissions.
rsyncuser:rsyncuser
A. The first part is the username, the second is the password.
B. This file must have 600 permissions.
Start the service:
rsync –daemon –config /etc/rsyncd.conf
rsync –daemon –config /etc/rsyncd.conf
Auto-start on boot:
Two different methods, same effect. The second method will leave logs in both the specified log and the system message log, though it is not really necessary.
A. Directly add the startup command to /etc/rc.local, placed on the last line.
B. Add to xinetd, placing the rsync server into the xinetd family for configuration and startup control.
Edit /etc/services, check the rsync service port — default is 873 for both TCP and UDP.
Edit /etc/rsync:
service rsync
{
disable = no # Enable service
socket_type = stream
wait = no
user = root
server = /usr/local/bin/rsync
server_args = –daemon –config /etc/rsyncd.conf
log_on_failure += USERID
}
C. Write your own service script and place it in chkconfig, so you can individually control the rsync service and flexibly adjust its startup order.
A. Directly add the startup command to /etc/rc.local, placed on the last line.
B. Add to xinetd, placing the rsync server into the xinetd family for configuration and startup control.
Edit /etc/services, check the rsync service port — default is 873 for both TCP and UDP.
Edit /etc/rsync:
service rsync
{
disable = no # Enable service
socket_type = stream
wait = no
user = root
server = /usr/local/bin/rsync
server_args = –daemon –config /etc/rsyncd.conf
log_on_failure += USERID
}
C. Write your own service script and place it in chkconfig, so you can individually control the rsync service and flexibly adjust its startup order.
The rc.local startup order is 99, meaning it executes last. The xinetd family startup order is 56. So in practice it depends on your needs — if you need it to start before an application, you should pay attention to this.
2. Client (sync initiator) configuration:
Simply execute the following command. For periodic execution, see crontab.
Push: rsync -vzrtopg –progress –password-file=/etc/rsyncd.secrets /opt/software/* [email protected]::optsoftware
Pull: rsync -vzrtopg –progress –password-file=/etc/rsyncd.secrets [email protected]::optsoftware /opt/software/
Simply execute the following command. For periodic execution, see crontab.
Push: rsync -vzrtopg –progress –password-file=/etc/rsyncd.secrets /opt/software/* [email protected]::optsoftware
Pull: rsync -vzrtopg –progress –password-file=/etc/rsyncd.secrets [email protected]::optsoftware /opt/software/
Parameter overview:
v: –verbose increase verbosity
z: –compress compress file data during the transfer
r: –recursive recurse into directories
t: –times preserve times
o: –owner preserve owner (super-user only)
p: –perms preserve permissions
g: –group preserve group
–progress: show progress during transfer
–delete: delete extraneous files from destination dirs. That is, if files are deleted on the source side, they are also deleted on the destination side.
v: –verbose increase verbosity
z: –compress compress file data during the transfer
r: –recursive recurse into directories
t: –times preserve times
o: –owner preserve owner (super-user only)
p: –perms preserve permissions
g: –group preserve group
–progress: show progress during transfer
–delete: delete extraneous files from destination dirs. That is, if files are deleted on the source side, they are also deleted on the destination side.
Special notes:
1. When using push, do you see that asterisk in the command? The server-side storage location must be a directory. In this example, if you do not include the asterisk, a software directory will be created inside the server-side target directory. With the asterisk, the files inside the source host’s software directory are synced directly. Pay special attention to this and operate according to your actual situation — both approaches have their uses.
2. Be absolutely certain about the sync direction. Do not delete files on the source host — remember this!
1. When using push, do you see that asterisk in the command? The server-side storage location must be a directory. In this example, if you do not include the asterisk, a software directory will be created inside the server-side target directory. With the asterisk, the files inside the source host’s software directory are synced directly. Pay special attention to this and operate according to your actual situation — both approaches have their uses.
2. Be absolutely certain about the sync direction. Do not delete files on the source host — remember this!