Rsync Usage Tips and Precautions

A recent project required using rsync to push small files. Initially, the efficiency was not great, and too many concurrent push processes caused high load and increased iowait on the target machine. rsync is a great tool, but it must be used properly. Here are a few key considerations I’ve summarized.
1: When syncing, try to use directory-level sync, and avoid having too many files in a single directory; otherwise, building the file list can be quite time-consuming.
2: The configuration of the target machine significantly impacts push efficiency. Testing with 1.5 million files totaling 8.3G (pushing from a 4-core 12G machine to an 8-core 24G machine took 14 minutes, while pushing from a 4-core 12G machine to another 4-core 12G machine took 5 hours).
3: Cross-data-center syncing also affects efficiency. Tests were conducted using China Unicom and China Telecom machines for the 1.5 million files (8.3G size) from 4-core 12G to 8-core 24G (Telecom to Telecom took 14 minutes; Unicom to Telecom took 30 minutes).
4: When transferring small files, using the z flag for compression can cause a several-fold performance loss (this came from another user’s report; I haven’t verified it myself yet).

Here are two simple usage examples for rsync:
One: Using rsync for code deployment

Two: Using rsync to sync files to a CDN. The sync process includes both a real-time incremental sync and a full sync. However, during the full sync, files might be changed or deleted by the real-time sync. In this case, real-time operations must take priority.

Here are several recommended articles related to rsync:
1: What is rsync?
2: rsync Algorithm, Usage, and History
3: Rsync Introduction, Upgrade, and Detailed Configuration
4: Real-Time Linux File Sync with Inotify+Rsync
5: Fastest Deletion of Massive Files Using rsync
6: The Art of rsync Synchronization
7: The Core Algorithm of rsync

Also, here is a detailed explanation of rsync parameters:

-v, --verbose Verbose output mode-q, --quiet Quiet output mode-c, --checksum Enable checksum verification, forcing file transfer verification-a, --archive Archive mode, indicates recursive transfer of files and preservation of all file attributes, equivalent to -rlptgoD-r, --recursive Process subdirectories recursively-R, --relative Use relative path information-b, --backup Create backups. For existing destination files with the same name, renames the old file to ~filename. The --suffix option can specify a different backup file prefix.--backup-dir Store backup files (like ~filename) in a specific directory.-suffix=SUFFIX Define backup file prefix-u, --update Update only; skips files that are newer on the destination than the source.-l, --links Preserve soft links-L, --copy-links Treat soft links as regular files--copy-unsafe-links Only copy links that point outside the SRC path directory tree--safe-links Ignore links that point outside the SRC path directory tree-H, --hard-links Preserve hard links-p, --perms Preserve file permissions-o, --owner Preserve file owner information-g, --group Preserve file group information-D, --devices Preserve device file information-t, --times Preserve file time information-S, --sparse Handle sparse files efficiently to save DST space-n, --dry-run Show which files would be transferred without actually doing so-W, --whole-file Copy whole files without incremental detection-x, --one-file-system Do not cross filesystem boundaries-B, --block-size=SIZE Block size used by the checksum algorithm, default is 700 bytes-e, --rsh=COMMAND Specify rsh or ssh for data synchronization--rsync-path=PATH Specify the path to the rsync command on the remote server-C, --cvs-exclude Auto-ignore files in the same way CVS does, to exclude unwanted files from transfer--existing Only update files that already exist on the destination, not new files--delete Delete files on DST that are not present on SRC--delete-excluded Also delete files on the receiver side that are excluded by this option--delete-after Delete files after the transfer is complete--ignore-errors Proceed with deletion even if IO errors occur--max-delete=NUM Delete at most NUM files--partial Keep partially transferred files to speed up subsequent transfers--force Force deletion of directories even if not empty--numeric-ids Do not map numeric user and group IDs to names--timeout=TIME IP timeout in seconds-I, --ignore-times Do not skip files that have the same time and length--size-only When deciding whether to back up, only compare file size, ignoring time--modify-window=NUM Timestamp window for determining if files have the same time; default is 0-T --temp-dir=DIR Create temporary files in DIR--compare-dest=DIR Also compare files in DIR to decide if a backup is needed-P Equivalent to --partial--progress Show backup progress-z, --compress Compress files during transfer--exclude=PATTERN Specify patterns for files to exclude from transfer--include=PATTERN Specify patterns for files to include in transfer--exclude-from=FILE Exclude files matching patterns listed in FILE--include-from=FILE Include files matching patterns listed in FILE--version Print version info--address Bind to a specific address--config=FILE Specify an alternate configuration file instead of the default rsyncd.conf--port=PORT Specify an alternate rsync service port--blocking-io Use blocking IO for remote shell-stats Give transfer status for certain files--progress Show transfer progress--log-format=formAT Specify log file format--password-file=FILE Get password from FILE--bwlimit=KBPS Limit IO bandwidth, KBytes per second-h, --help Show help information