APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Downloading Fedora with BitTorrent

As I explained at http://aplawrence.com/Blog/B662.html, I had wanted to take a look at RedHat 9, but was stopped by some problem with CD 2. I then thought I'd look into Fedora, but quickly ran into download problems there also: I tried a number of servers, but kept running into timeouts and just couldn't get the CD's down. I don't seem to be having any problems with other downloads, so I suspect there are just too many people wanting to look at Fedora. As download is your only choice with Fedora (no boxed sets to buy), I was annoyingly stuck. I could have bought a set from Discount Linux CD's or one of the many other similar sites, but I'm impatient, and didn't want to wait for delivery.

The problem with timeouts and other aborts is that you have part of what you need, and really would like to just start there and try to get another chunk. Some http and ftp sites do support resuming downloads, but not all do, and it's not necessarily easy to find out whether or not they have the feature. However, rsync always can do that, so I decided to use that instead.

There are not many rsync servers listed at http://fedora.redhat.com/download/mirrors.html (link dead, sorry), but distro.ibiblio.org is one. The http://www.fedora.us/wiki/FedoraMirrorHOWTO(link dead, sorry) site tells you how to setup for rsync, but it is typical geek: because most of the people who would to do that are fellow geeks, it doesn't really tell you what you need to do. The asssumption is that illiterates will use ftp or http, and anyone else just needs to be faced in the right direction.

There's also an implicit assumption that you aren't going to experience any transmission problems. They suggest using "rsync auv", which is NOT what you want to do if your session is locking up or failing and you have not yet got any of what you want.

The other problem is that the instructions assume that you want to mirror; that is, provide an identical directory structure to what the rsync server has. In the case of just trying to get the darn cd images, that may not be what you want at all. For example, I don't want the Source RPMS's just now: I'll get them later, but I want to get going with the installation first. If I followed the suggested scripts, I'd have to wait for all the SRPMS to download first, and even with cable, that's not quick. But how to find out what I wanted? Well, rsync can help:

apl$ rsync rsync://distro.ibiblio.org/fedora-linux-core/* .

  Welcome to distro.ibiblio.org's rsync server!

  You may mirror any of our distributions by using the "distros"
  module.  Simply append the name of the distribution you would
  like to use.  For example:

  rsync -avr distro.ibiblio.org::distros/bluelinux/ /local/path/

  If you will, please subscribe to our mirror announcement list:
  http://lists.ibiblio.org/mailman/listinfo/ibiblio-mirrors

  If you have problems or questions,
  feel free to mail [email protected]

  Thanks!

skipping directory /1
skipping directory /development
skipping directory /test
skipping directory /updates
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
 

Ahh. That showed me that the subdirecories were 1, development, test, and updates. I expanded a bit:


apl$ rsync rsync://distro.ibiblio.org/fedora-linux-core/1/* .

  Welcome to distro.ibiblio.org's rsync server!

  You may mirror any of our distributions by using the "distros"
  module.  Simply append the name of the distribution you would
  like to use.  For example:

  rsync -avr distro.ibiblio.org::distros/bluelinux/ /local/path/

  If you will, please subscribe to our mirror announcement list:
  http://lists.ibiblio.org/mailman/listinfo/ibiblio-mirrors

  If you have problems or questions,
  feel free to mail [email protected]

  Thanks!

skipping directory /1/SRPMS
skipping directory /1/i386
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
 

OK, now we are getting closer. I continued this until I knew what I wanted. It was then time to use a different invocation:

rsync -auv --progress --partial rsync://distro.ibiblio.org/fedora-linux-core/1/i386/iso/yarrow-i386-disc* .
 

The important additions are the --partial and --progress. The latter lets me know when the transfer has hung up, and the --partial lets me restart where I left off.

When rsync goes after a file, it first brings it to a temporary file:

apl $ ls -l .yarr*;sleep 10;ls -l .yarr*;sleep 10;ls -l .yarr*
-rw-------  1 apl  staff   1048576 10 Nov 08:36 .yarrow-i386-disc1.iso.UypdBA
-rw-------  1 apl  staff   1900544 10 Nov 08:36 .yarrow-i386-disc1.iso.UypdBA
-rw-------  1 apl  staff   2883584 10 Nov 08:36 .yarrow-i386-disc1.iso.UypdBA
 

If you haven't specified --partial, interrupting the rsync will cause that file to be deleted. However, if you do specify --partial, it will be copied to the final file. If you already had the final file, and were just looking for a newer version, you would NOT want to specify --partial, but as I did not have it yet, this was exactly what I wanted.

If you interrupt that rsync, the temporary version gets copied to yarrow-i386-disc1.iso. Start it up again, and yarrow-i386-disc1.iso is copied to a new temporary file and then the download resumes:

-rw-r--r--  1 apl  staff   3112960  5 Nov 17:39 yarrow-i386-disc1.iso
-rw-------  1 apl  staff   3506176 10 Nov 08:37 .yarrow-i386-disc1.iso.Yw3T5d

-rw-------  1 apl  staff   7733248 10 Nov 08:38 .yarrow-i386-disc1.iso.Yw3T5d
-rw-r--r--  1 apl  staff   3112960  5 Nov 17:39 yarrow-i386-disc1.iso
 
-rw-------  1 apl  staff   9011200 10 Nov 08:38 .yarrow-i386-disc1.iso.Yw3T5d
-rw-r--r--  1 apl  staff   3112960  5 Nov 17:39 yarrow-i386-disc1.iso
 

Had to interrupt here again. Notice how the yarrow-i386-disc1.iso is increased by what was downloaded.

-rw-r--r--  1 apl  staff   9863168  5 Nov 17:39 yarrow-i386-disc1.iso
 

Restart:

-rw-------  1 apl  staff   7143424 10 Nov 08:39 .yarrow-i386-disc1.iso.Aawu8W
-rw-r--r--  1 apl  staff   9863168  5 Nov 17:39 yarrow-i386-disc1.iso
 
-rw-------  1 apl  staff  10518528 10 Nov 08:39 .yarrow-i386-disc1.iso.Aawu8W
wrw-r--r--  1 apl  staff   9863168  5 Nov 17:39 yarrow-i386-disc1.iso
 

Well, I don't want to watch this thing for hangups all day, so one more change:

while true
do
rsync --timeout=60 -auv --progress --partial rsync://distro.ibiblio.org/fedora-linux-core/1/i386/iso/yarrow-i386-disc* .
sleep 1800
done
 

That let me leave this thing running. If it gets nothing in a three minute period, it sleeps for thirty minutes, and tries again. Unfortunately, this still didn't result in much success: after a full day of trying, I only had most of one CD pulled down.

Someone had suggested using BitTorrent, which is a distributed downloader - wonderful idea (see http://bitconjurer.org/BitTorrent/ for details), though I hadn't realized they had a Mac version - as the aborted RedHat 9 install killed my Linux box, I need to download these on my Mac. I had to open some ports on my firewall, and I hoped this would be the final solution. I started up BitTorrent, gave it the http://torrent.dulug.duke.edu/yarrow-binary-i386-iso.torrent url, and off it went. In a few minutes I had as much of the first CD as rsync had managed to get after trying all day, I certainly had good expectations. I did find it odd that BitTorrent takes a long time to settle down to a real representation of what the actual download time will be. I could tell almost immediately (based on the reported download rate) that this would take a little more than two hours, but the BitTorrent display took quite a while to realize that.

I went off and had supper and a small birthday celebration for my wife, and when I returned all the CD's were downloaded. The BitTorrent docs say there is no point in checksumming their downloads, but I did it anyway. I left BitTorrent running for a while after I got my images; that's how you pay back for using it.

Now all that is left is to burn the CD's and actually install Fedora, but that's another day.



Got something to add? Send me email.





(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> Downloading Fedora with BitTorrent




Increase ad revenue 50-250% with Ezoic


More Articles by

Find me on Google+

© Tony Lawrence






---October 16, 2004



---December 2, 2004



Kerio Samepage


Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us





The teaching of BASIC should be rated as a criminal offence: it mutilates the mind beyond recovery. (Edsger W. Dijkstra)

Anyone who slaps a 'this page is best viewed with Browser X' label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network. (Tim Berners-Lee)












This post tagged: