Backing up Windows machines using rsync and ssh by Manu Garg

Economical backup solution: rsync and ssh

As all other unix tricks this is also the result of laziness and the need. I wanted to backup data on my windows laptop to a central linux/unix server. I didn't want all the features of available expensive backup solutions. Just a simple updated copy of my data on a central machine which is backed up to the tape daily. rsync is known for fast incremental transfer and was an obvious choice for the purpose.

We have a unix machine at our workplace which has a directory structure /backup/username allocated for backing up user data. rsync has a client/server architecture, where rsync client talks to an rsync daemon at the server side (This statement may not be completely true. I am not sure and don't care also. You can refer to rsync manpage for complete discussion over rsync.). rsync client can connect to rsync server directly or through other remote transport programs like rsh, ssh etc. I decided to use ssh for transport for security and simplicity.

The rsync daemon requires a configuration file rsyncd.conf. For my use, I have set it up like this:

[manu@amusbocldmon01 ~]$ cat rsyncd.conf
use chroot = no
[backup]
        path = /backup
        read only = no
        comment = backup area
 

This says,

-do no chroot (required because I'll run it as a non-root user)
-[backup] specifies a module named backup.
-/backup is the path to backup module on filesystem

That's all we need at the server side. We don't need to keep rsync deamon running on the server. We'll start rsync daemon from the client using ssh before starting the backup.

At Windows side, we need rsync and some ssh client. rsync is available for windows through cygwin port. You can download cygwin from http://www.cygwin.com/. While installing cygwin, remember to select rsync. For ssh client, you can either use ssh that comes with cygwin or plink command line tool that comes with putty. Since, I have already set up my putty for password-less authentication using public/private key pair and pageant, I'll demonstrate this solution using plink. However you can use any other ssh client too. You can download putty and plink from http://www.chiark.greenend.org.uk/~sgtatham/putty/. You can find much information about ssh password less authentication on the web. To keep commands short, add rsync and plink to Windows path. Let's start our backup now.

First, we need to start rsync daemon at the server. It can be started from the client using following command:

plink -v -t -l manu fileserver.local.com rsync --daemon --port=1873 --config=$HOME/rsyncd.conf
 

where, fileserver.local.com is the central server where we are going to store our data. This logs in user 'manu' on fileserver and starts a rsync daemon there at the port 1873. rsync goes to the background and plink returns immediately.

Next we need to setup an ssh transport tunnel using plink:

plink -v -N -L 873:localhost:1873 -l manu fileserver.local.com
 

This sets up the local port forwarding -- forwarding local port 873 to port 1873 on the remote server.

After running this, we have port 873 on our windows box connected to the port 1873 on the fileserver on which rsync daemon is listening. So, now we just need to run rsync on windows machine with localhost as the target server:

rsync -av src 127.0.0.1::backup/manu
 

This command copies file or dir 'src' incrementally to directory 'manu' inside 'backup' module. Since this rsync is the one that comes with cygwin, it understand only cygwin paths for the files. For that reason, 'src' needs to be specified in cygwin terms. For example, D:\project becomes /cygdrive/d/project in cygwin terms.

Putting it all in scripts:

This trick is not much handy, unless you put it in the scripts and make it easy to run. To automate the process, I created 2 small scripts:

plink_rsync.bat: (To start plink for rsync)

REM Start rsync daemon the server
plink -v -t %* rsync --daemon --port=1873 --config=$HOME/rsyncd.conf
REM Setup ssh transport tunnel.
plink -v -N -L 873:localhost:1873 %*

runrsync.bat: (Main script - calls plink_rsync.bat and starts rsync)

REM Start plink_rsync.bat
START /MIN "PLINK_FOR_RSYNC" plink_rsync.bat -l manu fileserver.local.com
REM Sleep for 15 seconds to give plink enough time to finish
sleep 15
REM Iterate through filenames in filelist.txt and rsync them
for /F "delims=" %%i in (filelist.txt) do rsync -av %%i 127.0.0.1::backup/manu
REM Kill plink_rsync.bat window
TASKKILL /T /F /FI "WINDOWTITLE eq PLINK_FOR_RSYNC *"
REM Kill remote rsync daemon
plink -l manu fileserver.local.com pkill rsync
 

The main script starts plink_rsync.bat in another window and sleeps for 15 seconds to make sure that connection is set up. Then it runs rsync over the files and directories list in filelist.txt. After rsyncing is done, it kills plink_rsync.bat window and kills rsync daemon on the remote server by running pkill though plink.

filelist.txt contains the list of files and directories that you want to take backup of. For example, my filelist.txt contains:

filelist.txt:

"/cygdrive/d/Documents and Settings/501106700/My Documents/project"
"/cygdrive/d/Documents and Settings/501106700/My Documents/Outlook"
"/cygdrive/c/Program Files/Lotus/Sametime Client/Chat Transcripts"
 

You can schedule runrsync.bat to run everyday or every week depending on your requirement.

About Me

I am a unix guy, currently working as a sysadmin for a company on a strange planet of the Milky Way. I can be reached at manugarg at gmail dot com.

-Manu Garg
http://www.manugarg.com

More Articles by Manu Garg

Click here to add your comments




Mon Feb 23 20:07:03 2009: Subject:   Igor


How did you schedule that bat job? I'm trying to use "at" on Windows XP and it won't run.
Thanks!
-- Igor



Mon Feb 23 21:54:11 2009: Subject:   TonyLawrence


System Tools -> Scheduled Tasks maybe?



Mon Feb 23 22:09:07 2009: Subject: scheduler   Igor


I'm trying to automate it. I can put windows "at" command in a batch script. Otherwise it's a manual labor on each PC. :(
So far the script runs fine from DOS command line but does not work if I launch it from "at". I can't launch "ssh.exe" from "at" for some reason, so rsync -e ssh is also broken.
Any ideas?

The PATH to ssh.exe is right. I can run any executable except ssh.exe. Odd...



Mon Feb 23 22:15:56 2009: Subject:   TonyLawrence


When you can't run something in a scheduler that you can otherwise, it almost always means that its environment in the scheduler is not what it expects. Either environment variables are missing or set wrongly, the scheduler lacks permissions you have or its input/outut points somewhere the program doesn't like.





Tue May 12 16:48:04 2009: Subject: resolved   Igor


I've got the automation resolved. There is no need to install anything on the Windows side, you just need to copy the following files to the client:
cygcrypto-0.9.8.dll
cygiconv-2.dll
cygintl-3.dll
cygminires.dll
cygpath.exe
cygwin1.dll
cygz.dll
exclude.txt
gnudate.exe
rsync.exe
rsync.log
scp.exe
ssh.exe
ssh-keygen.exe
backup.bat
================================================
Setup.bat script will look like this:

@echo off
echo "Setting up backup job for %USERNAME%"
mkdir "c:\Documents and Settings\%USERNAME%\.ssh\"
xcopy \\server\rsync c:\rsync\

set PATH=c:\rsync;%PATH%
set CYGWIN=tty binmode
set TERM=ansi

rem Generating keys:
c:\rsync\ssh-keygen -P "" -t rsa -f "c:\Documents and Settings\%USERNAME%\.ssh\id_rsa"
copy "c:\Documents and Settings\%USERNAME%\.ssh\id_rsa.pub" \\server\%USERNAME%\.ssh\authorized_keys

c:\rsync\ssh server "mkdir backup; chmod 700 backup"
ping 127.0.0.1 -n 5 -w 1000 > null


rem Adding scheduled task:
schtasks /create /ru %USERNAME% /sc hourly /tn backup /tr c:\rsync\backup.bat

echo "All done! Closing in 5 seconds."

rem The folllowing line does 5 seconds sleep:
ping 127.0.0.1 -n 5 -w 5000 > null

================================================
The actual job, backup.bat script will look like this:
@echo off
cd c:\rsync
echo ============================= >>c:\rsync\rsync.log
echo My Documents Backup for %USERNAME% >>c:\rsync\rsync.log
c:\rsync\gnudate >>c:\rsync\rsync.log

c:\rsync\rsync -avzb --delete --exclude-from=exclude.txt -e ssh "/cygdrive/c/Documents and Settings/%USERNAME%/My Documents/" server:/home/%USERNAME%/backup >> c:\rsync\rsync.log
c:\rsync\gnudate >>c:\rsync\rsync.log
==============================================
The only catch is that the user will have to type in his/her windows password during the setup. schtasks requires it for some reason.

==============================================
To avoid pop-up DOS window:
Go to your scheduler:
Start - Setting - Control panel - Scheduled Tasks
Right-click on "backup" job, select properties.
Edit run line to look like:
start c:\rsync\backup.bat /B




Tue Aug 25 17:32:07 2009: Subject:   Igor


I learned the hard way that
"start c:\rsync\backup.bat /B" breaks the backup for some reason.

If you want to avoid that pop-up DOS window, you can use this little VBS script and call your batch file from it.

backup.vbs

Set WinScriptHost = CreateObject("WScript.Shell")
WinScriptHost.Run Chr(34) & "C:\rsync\backup.bat" & Chr(34), 0
Set WinScriptHost = Nothing


Ref: http://serverfault.com/questions/9038/run-a-bat-file-in-a-scheduled-task-without-a-window




Mon Sep 28 20:13:08 2009: Subject: Thank you!   AlexQuinn


Thank you so much for figuring this out and posting the details.

Many months ago, before you posted this, I several fruitless countless hours trying to do this without the daemon by setting the RSYNC_RSH variable to "plink...". I saw several other sites with people trying to do the same thing unsuccessfully.

Super-helpful. Thanks!

---

Macworld Mac Basics Superguide $12.95
Macworld Mac Basics Superguide, Leopard Edition $12.95
Macworld Mac Gems Superguide $12.95
Macworld Mac OS X Hints Superguide, Leopard Edition $12.95

---

Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments

If you want a picture to show with your comment, go get a Gravatar

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.