Backing up Windows machines using rsync and ssh by Manu Garg

Economical backup solution: rsync and ssh

As all other unix tricks this is also the result of laziness and the need. I wanted to backup data on my windows laptop to a central linux/unix server. I didn't want all the features of available expensive backup solutions. Just a simple updated copy of my data on a central machine which is backed up to the tape daily. rsync is known for fast incremental transfer and was an obvious choice for the purpose.

We have a unix machine at our workplace which has a directory structure /backup/username allocated for backing up user data. rsync has a client/server architecture, where rsync client talks to an rsync daemon at the server side (This statement may not be completely true. I am not sure and don't care also. You can refer to rsync manpage for complete discussion over rsync.). rsync client can connect to rsync server directly or through other remote transport programs like rsh, ssh etc. I decided to use ssh for transport for security and simplicity.

The rsync daemon requires a configuration file rsyncd.conf. For my use, I have set it up like this:

[manu@amusbocldmon01 ~]$ cat rsyncd.conf
use chroot = no
[backup]
        path = /backup
        read only = no
        comment = backup area
 

This says,

-do no chroot (required because I'll run it as a non-root user)
-[backup] specifies a module named backup.
-/backup is the path to backup module on filesystem

That's all we need at the server side. We don't need to keep rsync deamon running on the server. We'll start rsync daemon from the client using ssh before starting the backup.

At Windows side, we need rsync and some ssh client. rsync is available for windows through cygwin port. You can download cygwin from https://www.cygwin.com/. While installing cygwin, remember to select rsync. For ssh client, you can either use ssh that comes with cygwin or plink command line tool that comes with putty. Since, I have already set up my putty for password-less authentication using public/private key pair and pageant, I'll demonstrate this solution using plink. However you can use any other ssh client too. You can download putty and plink from https://www.chiark.greenend.org.uk/~sgtatham/putty/. You can find much information about ssh password less authentication on the web. To keep commands short, add rsync and plink to Windows path. Let's start our backup now.

First, we need to start rsync daemon at the server. It can be started from the client using following command:

plink -v -t -l manu fileserver.local.com rsync --daemon --port=1873 --config=$HOME/rsyncd.conf
 

where, fileserver.local.com is the central server where we are going to store our data. This logs in user 'manu' on fileserver and starts a rsync daemon there at the port 1873. rsync goes to the background and plink returns immediately.

Next we need to setup an ssh transport tunnel using plink:

plink -v -N -L 873:localhost:1873 -l manu fileserver.local.com
 

This sets up the local port forwarding -- forwarding local port 873 to port 1873 on the remote server.

After running this, we have port 873 on our windows box connected to the port 1873 on the fileserver on which rsync daemon is listening. So, now we just need to run rsync on windows machine with localhost as the target server:

rsync -av src 127.0.0.1::backup/manu
 

This command copies file or dir 'src' incrementally to directory 'manu' inside 'backup' module. Since this rsync is the one that comes with cygwin, it understand only cygwin paths for the files. For that reason, 'src' needs to be specified in cygwin terms. For example, D:\project becomes /cygdrive/d/project in cygwin terms.

Putting it all in scripts:

This trick is not much handy, unless you put it in the scripts and make it easy to run. To automate the process, I created 2 small scripts:

plink_rsync.bat: (To start plink for rsync)

REM Start rsync daemon the server
plink -v -t %* rsync --daemon --port=1873 --config=$HOME/rsyncd.conf
REM Setup ssh transport tunnel.
plink -v -N -L 873:localhost:1873 %*

runrsync.bat: (Main script - calls plink_rsync.bat and starts rsync)

REM Start plink_rsync.bat
START /MIN "PLINK_FOR_RSYNC" plink_rsync.bat -l manu fileserver.local.com
REM Sleep for 15 seconds to give plink enough time to finish
sleep 15
REM Iterate through filenames in filelist.txt and rsync them
for /F "delims=" %%i in (filelist.txt) do rsync -av %%i 127.0.0.1::backup/manu
REM Kill plink_rsync.bat window
TASKKILL /T /F /FI "WINDOWTITLE eq PLINK_FOR_RSYNC *"
REM Kill remote rsync daemon
plink -l manu fileserver.local.com pkill rsync
 

The main script starts plink_rsync.bat in another window and sleeps for 15 seconds to make sure that connection is set up. Then it runs rsync over the files and directories list in filelist.txt. After rsyncing is done, it kills plink_rsync.bat window and kills rsync daemon on the remote server by running pkill though plink.

filelist.txt contains the list of files and directories that you want to take backup of. For example, my filelist.txt contains:

filelist.txt:

"/cygdrive/d/Documents and Settings/501106700/My Documents/project"
"/cygdrive/d/Documents and Settings/501106700/My Documents/Outlook"
"/cygdrive/c/Program Files/Lotus/Sametime Client/Chat Transcripts"
 

You can schedule runrsync.bat to run everyday or every week depending on your requirement.

About Me

I am a unix guy, currently working as a sysadmin for a company on a strange planet of the Milky Way. I can be reached at manugarg at gmail dot com.

-Manu Garg
https://www.manugarg.com

Got something to add? Send me email.

7 comments

Mon Feb 23 20:07:03 2009: 5483   Igor


How did you schedule that bat job? I'm trying to use "at" on Windows XP and it won't run.
Thanks!
-- Igor



Mon Feb 23 21:54:11 2009: 5484   TonyLawrence


System Tools -> Scheduled Tasks maybe?



Mon Feb 23 22:09:07 2009: 5485   Igor


I'm trying to automate it. I can put windows "at" command in a batch script. Otherwise it's a manual labor on each PC. :(
So far the script runs fine from DOS command line but does not work if I launch it from "at". I can't launch "ssh.exe" from "at" for some reason, so rsync -e ssh is also broken.
Any ideas?

The PATH to ssh.exe is right. I can run any executable except ssh.exe. Odd...



Mon Feb 23 22:15:56 2009: 5486   TonyLawrence


When you can't run something in a scheduler that you can otherwise, it almost always means that its environment in the scheduler is not what it expects. Either environment variables are missing or set wrongly, the scheduler lacks permissions you have or its input/outut points somewhere the program doesn't like.







Tue May 12 16:48:04 2009: 6356   Igor


I've got the automation resolved. There is no need to install anything on the Windows side, you just need to copy the following files to the client:
cygcrypto-0.9.8.dll
cygiconv-2.dll
cygintl-3.dll
cygminires.dll
cygpath.exe
cygwin1.dll
cygz.dll
exclude.txt
gnudate.exe
rsync.exe
rsync.log
scp.exe
ssh.exe
ssh-keygen.exe
backup.bat
================================================
Setup.bat script will look like this:

@echo off
echo "Setting up backup job for %USERNAME%"
mkdir "c:\Documents and Settings\%USERNAME%\.ssh\"
xcopy \\server\rsync c:\rsync\

set PATH=c:\rsync;%PATH%
set CYGWIN=tty binmode
set TERM=ansi

rem Generating keys:
c:\rsync\ssh-keygen -P "" -t rsa -f "c:\Documents and Settings\%USERNAME%\.ssh\id_rsa"
copy "c:\Documents and Settings\%USERNAME%\.ssh\id_rsa.pub" \\server\%USERNAME%\.ssh\authorized_keys

c:\rsync\ssh server "mkdir backup; chmod 700 backup"
ping 127.0.0.1 -n 5 -w 1000 > null


rem Adding scheduled task:
schtasks /create /ru %USERNAME% /sc hourly /tn backup /tr c:\rsync\backup.bat

echo "All done! Closing in 5 seconds."

rem The folllowing line does 5 seconds sleep:
ping 127.0.0.1 -n 5 -w 5000 > null

================================================
The actual job, backup.bat script will look like this:
@echo off
cd c:\rsync
echo ============================= >>c:\rsync\rsync.log
echo My Documents Backup for %USERNAME% >>c:\rsync\rsync.log
c:\rsync\gnudate >>c:\rsync\rsync.log

c:\rsync\rsync -avzb --delete --exclude-from=exclude.txt -e ssh "/cygdrive/c/Documents and Settings/%USERNAME%/My Documents/" server:/home/%USERNAME%/backup >> c:\rsync\rsync.log
c:\rsync\gnudate >>c:\rsync\rsync.log
==============================================
The only catch is that the user will have to type in his/her windows password during the setup. schtasks requires it for some reason.

==============================================
To avoid pop-up DOS window:
Go to your scheduler:
Start - Setting - Control panel - Scheduled Tasks
Right-click on "backup" job, select properties.
Edit run line to look like:
start c:\rsync\backup.bat /B






Tue Aug 25 17:32:07 2009: 6786   Igor


I learned the hard way that
"start c:\rsync\backup.bat /B" breaks the backup for some reason.

If you want to avoid that pop-up DOS window, you can use this little VBS script and call your batch file from it.

backup.vbs

Set WinScriptHost = CreateObject("WScript.Shell")
WinScriptHost.Run Chr(34) & "C:\rsync\backup.bat" & Chr(34), 0
Set WinScriptHost = Nothing


Ref: (link)






Mon Sep 28 20:13:08 2009: 6997   AlexQuinn


Thank you so much for figuring this out and posting the details.

Many months ago, before you posted this, I several fruitless countless hours trying to do this without the daemon by setting the RSYNC_RSH variable to "plink...". I saw several other sites with people trying to do the same thing unsuccessfully.

Super-helpful. Thanks!

------------------------

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us