How to Automate Secure File Synchronization using SSH and rsync

Tom Hilinski
Natural Resource Ecology Laboratory,
Colorado State University
Last updated: Dec 2008

Introduction

In order to automate file transfers between computers without a password, a private-public key identification key simplifies the process. This is useful, for instance, when using rsync to synchronize files in local and remote directories. For instance, after editing files on your local Linux desktop or Microsoft Windows laptop, you want to automatically update the files on the office computer with your modified files. In this case, a utility such as rsync can be used to do the update without prompting you for a password on the office computer.

The process of creating the key is described here in the context of using rsync on a local Windows computer with Cygwin installed. Use the Cygwin setup program to install SSH, rsync, and Bash. Here, I assume that the remote computer, say, your office server, is running Linux, and you have an account on it with the user name yourUserName.

In the examples below, command lines begin with a $ character, comment lines begin with a # character, while text beginning without either is written to the console display. Example text that is italicized means you substitute your own information there; for example, yourUserName is replace by your actual user name.

Create a Key

On your local Windows computer, open a Bash shell console window. If you don't have a directory named .ssh create one by using SSH to connect to your office computer for the first time. You will be prompted to accept the key.

$ ls -d .ssh

# if the directory does not exist, run ssh

$ ssh yourUserName@calypso.nrel.colostate.edu

Go into the .ssh directory and create a key. This key will have two files, a private file and a public file. When prompted, do not enter a password or passphrase.

$ ssh-keygen -t dsa -b 1024 -f yourUserName-rsync-key

Generating public/private dsa key pair.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in yourUserName-rsync-key.

Your public key has been saved in yourUserName-rsync-key.pub.

The key fingerprint is:

(a long string of hexadecimal digits)

Check the permissions on your key files (e.g., ls -l). The permissions should be 600 (or rw-----).

On your office computer, make sure you have in your home directory, a subdirectory named .ssh (note the leading dot).

$ ssh yourUserName@calypso.nrel.colostate.edu

$ ls -d .ssh

# If this directory doesn't exist, create it:

$ mkdir .ssh

# Make sure the permissions are secure:

$ chmod 700 .ssh

Now, log off your office computer.

Next, copy the public key file to your office computer, log onto that computer, then append the key file to the SSH file containing keys it knows about.

# Copy the public key to your office computer:

$ scp yourUserName-rsync-key.pub yourUserName@calypso.nrel.colostate.edu:/home/nrel/yourUserName/.ssh/

# Log on to the remote computer:

$ ssh yourUserName@calypso.nrel.colostate.edu

# If your are not in a bash shell, then start one:

$ bash

# If the key file does not exist, create it:

$ if [ ! -f authorized_keys ]; then touch authorized_keys; chmod 600 authorized_keys; fi

# Append your new public key to the key file:

$ cat yourUserName-rsync-key.pub >> authorized_keys

$ rm yourUserName-rsync-key.pub 

Your key is now ready to use with rsync. Optionally, you can restrict the use of the key to an IP address and a particular process (e.g., rsync). To restrict the key to rsync, create the file listed in Appendix A in your ~/.ssh directory on your office computer. You can use a text editor to paste that text in. Then set the permissions so no one else can read it. For example:

# Use vi to create the file; paste in the script from Appendix A.

$ vi restrict-to-rsync

$ chmod 700 restrict-to-rsync

Next, edit the file authorized_keys so that the line with your key (the last line, since the key was just appended to the file) begins with a command to run that script. The command points to the full path of the script file. The line originally began with:

ssh-dss AAAAB3...

After inserting the script command, the line starts with:

command="/home/nrel/yourUserName/.ssh/restrict-to-rsync" ssh-dss AAAAB3...

Using rsync With SSH and Your Key

Test the use of your new key by copying a junk file from your local computer to your office computer. Here, the local file is junk.txt and the remote directory in your office computer is tmp, and the direction of transfer is local-to-remote. Give SSH the name of your private key file on your local computer, including its path, using the following form:

rsync -auvz -e "ssh -i private-key-file" source destination

Here, source is a file or a directory, and destination has the form yourUserName@remote-computer:/remote-path

A real example, using the file names from the previous examples, is:

rsync -auvz -e "ssh -i /home/yourUserName/.ssh/yourUserName-rsync-key" junk.txt yourUserName@calypso.nrel.colostate.edu

The rsync flags -auvz specify "archive", "update", "verbose messages", and "compress files for transfer", respectively. "Update" means that files on the destination that are newer than your local files are not overwritten. The "-e" flag tells rsync the SSH command.

If you want details on what SSH is doing, add "-v" to the ssh options. To run rsync quietly, remove the "-v" option from both rsync and SSH option list.

To reverse the synchronization so the remote file is updated on your local computer, reverse the source and destinations.

You can store your rsync commands that you use all the time in a script file. Keep this script with your project files or in a script directory that is specified in your PATH environment variable.

Additional Information

rsync document: http://rsync.samba.org/ftp/rsync/rsync.html

rsync web site: http://rsync.samba.org/

Acknowledgements

Many online sources provided the information I used to create this process. A particularly succinct source was http://troy.jdmz.net/rsync/index.html provided the basis of the script in Appendix A. Thanks to all.

Appendix A: File restrict-to-rsync

The following shell script checks that rsync is the process attempting to connect. If it is not, the script fails, and SSH also fails. A log file named validate-rsync.log is created or appended to with each connection.

#!/bin/sh

logfile=/home/nrel/yourUserName/.ssh/restrict-to-rsync.log

case "$SSH_ORIGINAL_COMMAND" in

*\&*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\(*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\{*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\;*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\<*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\`*)

echo `date` "- SSH connection rejected" >> $logfile

;;

*\|*)

echo `date` "- SSH connection rejected" >> $logfile

;;

rsync\ --server*)

{

echo `date` "- SSH connection accepted" >> $logfile

$SSH_ORIGINAL_COMMAND

}

;;

*)

echo `date` "- SSH connection rejected" >> $logfile

;;

esac