System environment: CentOS 6.3 rsync server
CentOS 6.3 rsync client
IP addresses: 10.1.4.44 (server), 10.1.4.41 (client)
Required package: rsync-3.0.9.tar.gz
I. rsync Server
rsync, as the name “remote synchronize” suggests, is a software that implements remote synchronization functionality. While synchronizing files, it can preserve the original file permissions, timestamps, soft and hard links, and other metadata. rsync uses the “rsync algorithm” to provide a fast method for file synchronization between a client and a remote file server, and it can transfer files via SSH, ensuring excellent confidentiality. Additionally, it is free software.
rsync includes the following features:
Can update entire directory trees and file systems;
Selectively preserves symbolic links, hard links, file ownership, permissions, device files, and timestamps;
No special permission requirements for installation;
Internal pipelining reduces file waiting latency for multiple files;
Can use rsh, ssh, or direct port connections as transport channels;
Supports anonymous rsync file synchronization, making it an ideal mirroring tool;
Setting up an rsync server is relatively simple. After installing rsync, you may not find configuration files or an rsync server startup program, because each administrator may use rsync for different purposes. Therefore, typical distributions just install the software and leave it at that, allowing administrators to set up the rsync server according to their own needs and direction. rsync has a wide range of applications—it can perform backup tasks on the same host and also operate between different hosts. For backups between different hosts, setting up an rsync server is essential.
II. rsync Server Installation
rysnc official website: http://rsync.samba.org — you can obtain the latest version from there. The latest version as of writing is 3.0.9.
[root@server ~]# wget http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
[root@server ~]# tar xvf rsync-3.0.9.tar.gz
[root@server ~]# cd rsync-3.0.9
[root@server rsync-3.0.9]# ./configure –prefix=/usr
[root@server rsync-3.0.9]# make
[root@server rsync-3.0.9]# make install Note: Before compiling and installing from source, you must install gcc and other compilation tools.
III. rsync Server Configuration File rsyncd.conf
rsync primarily has the following three configuration files:
rsyncd.conf (main configuration file)
rsyncd.secrets (password file)
rsyncd.motd (rsync server information)
Server configuration file (/etc/rsyncd/rsyncd.conf), which does not exist by default—please create it:
[root@server etc]# mkdir rsyncd Note: Create a directory named rsyncd under /etc to store rsyncd.conf, rsyncd.secrets, and rsyncd.motd files.
[root@server rsyncd]# touch rsyncd.conf Note: Create rsyncd.conf, the rsync server configuration file.
[root@server rsyncd]# touch rsyncd.secrets Note: Create rsyncd.secrets, the user password file.
[root@server rsyncd]# chmod 600 rsyncd.secrets Note: Set permissions to 600 for password security.
[root@server rsyncd]# ls -lh rsyncd.secrets
-rw——- 1 root root 12 Sep 14 11:56 rsyncd.secrets
[root@server rsyncd]# touch rsyncd.motd
[root@server rsyncd]# ll
total 12
-rw-r–r– 1 root root 643 Sep 14 11:55 rsyncd.conf
-rw-r–r– 1 root root 172 Sep 14 11:58 rsyncd.motd
-rw——- 1 root root 12 Sep 14 11:56 rsyncd.secrets
Edit configuration: rsyncd.conf, rsyncd.secrets, rsyncd.motd files;
1. Configure rsyncd.conf
rsyncd.conf is the main configuration file for the rsync server. Let’s look at a simple example:
Backup all files in the /data/share directory on the backup server. Manually add the following content:
[root@server ~]# vim /etc/rsyncd/rsyncd.conf
# Distributed under the terms of the GNU General Public License v2
# Minimal configuration file for rsync daemon
# See rsync(1) and rsyncd.conf(5) man pages for help
# This line is required by the /etc/init.d/rsyncd script
pid file = /var/run/rsyncd.pid Note: Tells the process to write to the /var/run/rsyncd.pid file
port = 873 Note: Specifies the running port. Default is 873; you can specify your own.
address = 10.1.4.44 Note: Specifies the server IP address.
#uid = nobody Note: When the server transfers files, which user and group to execute as; default is nobody.
#gid = nobody Note: When the server transfers files, which user and group to execute as; default is nobody.
uid = root
gid = root
use chroot = yes
read only = yes
#limit access to private LANs
hosts allow=10.1.4.0/255.255.255.0
hosts deny=*
max connections = 5 Note: Maximum number of client connections.
motd file = /etc/rsyncd/rsyncd.motd
#This will give you a separate log file
log file = /var/log/rsync.log
#This will log every file transferred – up to 85,000+ per user, per sync
#transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
[samba_data] Note: Module
path = /data/share Note: Specifies the directory path for file storage.
list=no Note: list determines whether the directories providing sync data on the rsync server are displayed as module listings. Default is yes. If you don’t want them listed, set to no; using no is safer, as at least others won’t know which directories your server provides. Only you need to know.
ignore errors Note: Ignore I/O errors. Refer to documentation for details.
auth users = root Note: The authenticated user is root, which must be a user that actually exists on the server.
secrets file = /etc/rsyncd/rsyncd.secrets Note: Password is stored in the rsyncd.secrets file.
Note: Regarding auth users — they must be real system users that exist on the server. If you want to use multiple users, separate them with commas; e.g., auth users = root, user1.
2. Configure rsyncd.secrets
Password file: rsyncd.secrets content format:
[root@server ~]# vim /etc/rsyncd/rsyncd.secrets
root:111111
user:111111
username:password Note: root:111111
In our example, the content of rsyncd.secrets is as above. The documentation mentions that some systems do not support long passwords—try setting one yourself. Additionally, the rsyncd.secrets file must not be readable by other groups. If configured incorrectly, rsync may not work.
Note:
1. Set the rsyncd.secrets password file to be owned by root with permissions set to 600; otherwise, backups will fail! For security purposes, the file must be readable only by its owner.
#chown root.root rsyncd.secrets #Change owner
#chmod 600 rsyncd.secrets #Change permissions
2. The password here is noteworthy. For security, you should not write the system user’s password here. For example, if your system user root password is abcdefg, for security you can set the rsync root password to 111111. This is similar in principle to Samba user authentication passwords.
3. Configure rsyncd.motd
It defines rsync server information, i.e., user login information. For instance, letting users know who provides this server, etc. It’s similar to the prompt message you see when logging into an FTP server… Of course, in global variable definitions, this is not mandatory—you can comment it out with # or delete it. I wrote the following content for rsyncd.motd:
[root@server ~]# vim /etc/rsyncd/rsyncd.motd
++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002——2012
++++++++++++++++++++++++++++++++++++++++++++++
IV. rsync Server Setup Instructions
1. Global Definitions
In the rsync server, there are several critical global definitions, based on the rsyncd.conf file we provided earlier:
pid file = /var/run/rsyncd.pid Note: Tells the process to write to the /var/run/rsyncd.pid file
port = 873 Note: Specifies the running port. Default is 873; you can specify your own.
address = 10.1.4.44 Note: Specifies the server IP address.
uid = nobody
gid = nobdoy
Note: When the server transfers files, which user and group to execute as. Default is nobody. Using the nobody user and group may cause permission issues—some files may not be pulled from the server. So I took a shortcut and used root for convenience. However, you can resolve permission issues by specifying the user in the module definition when defining the directory to sync.
use chroot = yes Note: Using chroot, before transferring files, the server daemon will chroot into the directory in the file system. The advantage is that it may protect the system from installation vulnerability exploits. The disadvantage is that superuser privileges are required. Additionally, symbolic link files will be excluded. That means, if you have symbolic links on the rsync server, when you run client-side sync on the backup server, only the symbolic link names will be synced, not their contents. This requires your own experimentation.
read only = yes Note: read only is a read-only option, meaning clients are not allowed to upload files to the server. There is also a write only option—try it yourself to see what it does.
#limit access to private LANs
hosts allow=192.168.1.0/255.255.255.0 10.0.1.0/255.255.255.0
Note: You can specify individual IPs or entire network segments to improve security. The format requires spaces between IPs and IPs, IPs and network segments, and network segments and network segments.
max connections = 5 Note: Maximum number of client connections.
motd file = /etc/rsyncd/rsyncd.motd Note: motd file defines server information. You need to write the rsyncd.motd file content yourself. When users log in, they will see this information.
log file = /var/log/rsync.log Note: rsync server logs.
transfer logging = yes Note: This is the transfer file log.
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
2. Module Definitions
What do modules define? They primarily define which server directories should be synced. Each module must be in the [name] format. This name is the name that rsync clients see, somewhat similar to the share name provided by a Samba server. The actual data the server syncs is specified through the path. You can define multiple modules according to your needs. Each module should specify authenticated users and password files, but exclusion is not mandatory.
Let’s look at a simple example:
For instance, we want to back up /data/share and /opt on the server. In /data/share, I want to exclude the beinan and samba directories:
[samba] Note: Module—provides a link name for us.
path = /data/share Note: Specifies the directory path for file storage.
auth users = root Note: The authenticated user is root, which must be a user that actually exists on the server.
list=yes Note: list determines whether the directories providing sync data on the rsync server are displayed as module listings. Default is yes. If you don’t want them listed, set to no; using no is safer, as at least others won’t know which directories your server provides. Only you need to know.
ignore errors Note: Ignore I/O errors. Refer to documentation for details.
secrets file = /etc/rsyncd/rsyncd.secrets Note: Password is stored in the rsyncd.secrets file.
comment =root home data Note: Comments can be defined freely—write whatever you like, just something relevant.
exclude = beinan/ samba/ Note: exclude means exclusion. That is, exclude the beinan and samba directories from /home; beinan/ and samba/ directories are separated by spaces.
[beinan] Note: Module—provides a link name for us.
path = /opt Note: Specifies the directory path for file storage.
list=no
comment = data
auth users = user1 Note: Must be a user that exists on the server.
secrets file = /etc/rsyncd/rsyncd.secrets
ignore errors
V. Starting the rsync Server and Firewall Configuration
1. Starting the rsync Server
Starting the rsync server is quite simple. –daemon makes rsync run in server mode:
[root@server ~]#/usr/bin/rsync –daemon –config=/etc/rsyncd/rsyncd.conf
Note: If you can’t find the rsync command, you should know where rsync is installed. For example, the rsync executable may be installed in the /usr/local/bin directory; then the command would be:
[root@server ~]#/usr/local/bin/rsync –daemon –config=/etc/rsyncd/rsyncd.conf
Of course, you can also write a script to automatically start the rsync server at boot. Check the documentation and try it yourself—it’s simple. Since I use Slackware, I also have a similar script. I feel it’s more convenient to run it manually, or add this command to the rc.local file—that way it will also run automatically.
[root@server ~]# netstat -anp |grep :873
tcp 0 0 10.1.4.44:873 0.0.0.0:* LISTEN 5696/rsync
2. rsync Server and Firewall
Linux firewall uses iptables, so we must at least allow the defined rsync server port through on the server side, and the same on the client side.
[root@server ~]#iptables -A INPUT -p tcp -m state –state NEW -m tcp –dport 873 -j ACCEPT
[root@server ~]#iptables -L Check whether the firewall has opened port 873.
VI. Syncing Data via rsync Client
Command Syntax Details
After configuring the rsync server, you can issue rsync commands from the client to perform various synchronization operations. rsync has many functional options. Below are the commonly used options explained:
The rsync command format can be:
1. rsync [OPTION]… SRC [SRC]… [USER@]HOST:DEST
2. rsync [OPTION]… [USER@]HOST:SRC DEST
3. rsync [OPTION]… SRC [SRC]… DEST
4. rsync [OPTION]… [USER@]HOST::SRC [DEST]
5. rsync [OPTION]… SRC [SRC]… [USER@]HOST::DEST
6. rsync [OPTION]… rsync://[USER@]HOST[:PORT]/SRC [DEST]
rsync has six different working modes:
1. Copy local files; this mode is activated when neither SRC nor DEST path information contains a single colon “:” separator.
2. Use a remote shell program (such as rsh, ssh) to copy local machine contents to a remote machine. This mode is activated when the DST path address contains a single colon “:” separator.
3. Use a remote shell program (such as rsh, ssh) to copy remote machine contents to the local machine. This mode is activated when the SRC address path contains a single colon “:” separator.
4. Copy files from a remote rsync server to the local machine. This mode is activated when the SRC path information contains the “::” separator.
5. Copy files from the local machine to a remote rsync server. This mode is activated when the DST path information contains the “::” separator.
6. List remote machine file lists. This is similar to rsync transfer, but simply omit the local machine information from the command.
-a Archive mode operation, copies directories and symbolic links; equivalent to -rlptgoD
rsync parameters
-r is recursive
-l is for link files, meaning copy link files;
-p means preserve original file permissions;
-t preserve original file timestamps;
-g preserve original file user groups;
-o preserve original file ownership;
-D equivalent to block device files;
-z compress during transfer;
-P transfer progress;
-v transfer progress and other information, somewhat related to -P. Try it yourself. Refer to documentation;
-e ssh parameter to establish an encrypted connection.
-u only update, preventing local new files from being overwritten. Ensure both machines’ clocks are synchronized.
–progress displays detailed progress information.
–delete means if the server side deletes a file, the client side will also delete the corresponding file, maintaining true consistency.
–password-file=/password/path/file specifies the password file, allowing use in scripts without interactive password entry. Note that this password file’s permissions must be set so only the owner can read it.
1. rsync Server Provided Sync Content
View available data sources on the rsync server.
[root@server ~]# rsync –list-only [email protected]::samba_data
++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002——2012
++++++++++++++++++++++++++++++++++++++++++++++
Password:
drwxr-xr-x 4096 2012/09/14 12:04:57 .
drwxrwxrwx 4096 2012/09/14 12:04:57 caiwu
drwxrwxrwx 4096 2012/09/14 12:24:54 jishu
drwxrwxrwx 4096 2012/09/14 12:22:11 public
drwxrwxrwx 4096 2012/09/14 12:04:57 yanfa
Note: The above are the data sources provided by rsync, i.e., the [samba_data] module we wrote in rsyncd.conf. The “linuxsir home data” is provided by the comment = root home in the [samba_data] module. Why is [samba_data] listed as a data source? Because we set list=yes in [samba_data].
2. rsync Client Installation
rsync official website: http://rsync.samba.org — you can obtain the latest version from there. The latest version as of writing is 3.0.9.
[root@server ~]# wget http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
[root@client ~]# tar xvf rsync-3.0.9.tar.gz
[root@client ~]# cd rsync-3.0.9
[root@client rsync-3.0.9]# ./configure –prefix=/usr
[root@client rsync-3.0.9]# make
[root@client rsync-3.0.9]# make install Note: Before compiling and installing from source, you must install gcc and other compilation tools.
View available data sources on the rsync client:
[root@client ~]# rsync -avzP [email protected]::samba_data
++++++++++++++++++++++++++++++++++++++++++++++
Welcome to use the mike.org.cn rsync services!
2002——2012
++++++++++++++++++++++++++++++++++++++++++++++
Password:
receiving incremental file list
drwxr-xr-x 4096 2012/09/14 12:04:57 .
drwxrwxrwx 4096 2012/09/14 12:04:57 caiwu
drwxrwxrwx 4096 2012/09/17 11:57:00 jishu
drwxrwxr-x 4096 2012/09/17 11:31:59 jishu/技术IOS
drwxrwxr-x 4096 2012/09/17 11:31:25 jishu/技术之星
drwxrwxr-x 4096 2012/09/17 16:05:04 jishu/技术备份
drwxrwxr-x 4096 2012/09/17 11:31:50 jishu/技术文档
drwxrwxr-x 4096 2012/09/17 11:31:40 jishu/技术资料完整
drwxrwxr-x 4096 2012/09/17 11:32:07 jishu/技术软件包
drwxrwxrwx 4096 2012/09/17 16:06:34 public
drwxrwxrwx 4096 2012/09/14 12:04:57 yanfa
sent 70 bytes received 489 bytes 74.53 bytes/sec
total size is 0 speedup is 0.00
In root@ip, root refers to the username in the password file, and the ::samba_data that follows is the [samba_data] module name.
3. rsync Client Manual Data Sync
[root@client ~]# rsync -avzP [email protected]::samba_data /data
Password: Enter root’s password here, provided by the server. In our earlier example, we used 111111. The entered password is not displayed; press Enter after typing.
Note: This command means: log in to the server as the root user and sync the [samba_data] data source to the local /data directory. Of course, the local directory can be defined by you, such as /share. When the /data directory does not exist in the current working directory on the client, the system will automatically create one for you. When the /data directory exists, pay attention to its write permissions.
Explanation:
-a parameter, equivalent to -rlptgoD, -r is recursive, -l is for link files (meaning copy link files), -p means preserve original file permissions, -t preserve original file timestamps, -g preserve original file user groups, -o preserve original file ownership, -D equivalent to block device files;
-z compress during transfer;
-P transfer progress;
-v transfer progress and other information, somewhat related to -P. Try it yourself. Refer to documentation;
[root@client ~]# rsync -avzP –delete [email protected]::samba_data /data
–delete option: means the data on the client must be completely consistent with the server. If the [samba_data] directory contains files that do not exist on the server, they will be deleted. The ultimate goal is to keep the /data directory data completely consistent with the server. Use with caution—it’s best not to use a directory that already contains important data as the local update directory, otherwise all your data will be deleted.
[root@client ~]# rsync -avzp –delete –password-file=rsync.password [email protected]::samba_data /data
–password-file=rsync.password option: When logging in as the linuxsir user to the rsync server to sync data, the password will be read from the rsync.password file. This file contains only the linuxsir user’s password. We need to do the following:
[root@client ~]# touch rsync.password
[root@client ~]# chmod 600 rsync.passwod
[root@client ~]# echo “111111”> rsync.password
[root@client ~]# rsync -avzp –delete –password-file=rsync.password [email protected]::samba_data /data
Note: This way, no password is needed. This is actually quite important, because scheduled tasks via crond on the server are still necessary.
4. rsync Client Automatic Data Sync with Server
The server is a mission-critical application, so network data backup is extremely important. We can configure the rsync server on the production server. We can use a machine with rsync installed as the backup server. Have this backup server start syncing server data every morning at 2 AM, with each backup being a full backup. Sometimes hard drives fail, or server data gets deleted—full backups are still very important. This type of backup is essentially creating a daily mirror of the server data. When the production server encounters an incident, we can easily restore data and minimize data loss.
Step 1: Create sync script and password file
Note: We created a file named root.sh in /etc/cron.daily.rsync with permissions set to 755.
[root@client ~]# mkdir /etc/cron.daily.rsync
[root@client ~]# cd /etc/cron.daily.rsync
[root@client cron.daily.rsync]# touch root.sh
[root@client cron.daily.rsync]# chmod 755 root.sh
Edit root.sh, with the following content:
[root@client cron.daily.rsync]#vim root.sh
#!/bin/sh
#10.1.4.44 samba_data backup
/usr/bin/rsync -avzP –delete –password-file=/etc/rsyncd/rsyncroot.password [email protected]::samba_data /data/$(date +’%m-%d-%y’)
:wq
Execute the configured script statement:
[root@client cron.daily.rsync]# sh root.sh
Create a password file. The root user uses rsyncroot.password with permissions set to 600:
[root@client ~]# mkdir /etc/rsyncd/
[root@client ~]# cd /etc/rsyncd/
[root@client rsyncd]# touch rsyncroot.password
[root@client rsyncd]# chmod 600 rsyncroot.password
[root@client rsyncd]# ll
total 4
-rw——- 1 root root 7 Sep 14 17:30 rsyncroot.password
Next, modify the content of rsyncroot.password:
[root@client rsyncd]# echo “111111” > rsyncroot.password
Then create the /data directory under root (/), meaning that the server-side /data/share data will be synced to /data on the backup client, with year-month-day archive directories created; every day’s backup is archived:
[root@client ~]# cd /
[root@client /]# mkdir /data
Step 2: Modify the crond server configuration file and add to scheduled tasks
[root@client ~]# crontab -e
Add the following content:
# Run daily cron jobs at 02:30 every day backup samba data:
30 02 * * * /usr/bin/run-parts /etc/cron.daily.rsync 1> /dev/null
Scheduled task explanation:
# Basic format:
# .—————- Column 1 represents minutes 1~59; use * or */1 for every minute
# | .————- Column 2 represents hours 1~23 (0 means midnight)
# | | .———- Column 3 represents day of month 1~31
# | | | .——- Column 4 represents month 1~12
# | | | | .—- Column 5 represents day of week 0~6 (0 means Sunday) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * *
The first line is a comment—it serves as a description so you can remember it yourself.
The second line means: at 02:30 every morning, run the executable script tasks under /etc/cron.daily.rsync.
Step 3: Restart the crond server:
[root@client ~]# service crond restart
Stopping crond: [ OK ]
Starting crond: [ OK ]
[root@client ~]# killall crond Note: Kill the crond server process;
[root@client ~]# ps aux |grep crond Note: Check if it was killed;
[root@client ~]# /usr/sbin/crond Note: Start the crond server;
[root@client ~]# ps aux |grep crond Note: Check if it started?
root 3872 0.0 0.1 5916 1192 ? Ss 12:02 0:00 crond
root 3951 0.0 0.1 5912 1184 ? Ss 14:06 0:00 /usr/sbin/crond
root 3953 0.0 0.0 4336 760 pts/0 S+ 14:06 0:00 grep crond
View data backup:
[root@client ~]# ll /data
total 24
drwxr-xr-x 6 root root 4096 Sep 14 12:04 09-17-12
drwxr-xr-x 6 root root 4096 Sep 14 12:04 09-18-12
This article is from “运维IT” blog. Please retain this source: http://yanghuawu.blog.51cto.com/2638960/994841