Downloading Fedora with BitTorrent
As I explained at http://aplawrence.com/Blog/B662.html, I had wanted to take a look at RedHat 9, but was stopped by some problem with CD 2. I then thought I'd look into Fedora, but quickly ran into download problems there also: I tried a number of servers, but kept running into timeouts and just couldn't get the CD's down. I don't seem to be having any problems with other downloads, so I suspect there are just too many people wanting to look at Fedora. As download is your only choice with Fedora (no boxed sets to buy), I was annoyingly stuck. I could have bought a set from Discount Linux CD's or one of the many other similar sites, but I'm impatient, and didn't want to wait for delivery.
The problem with timeouts and other aborts is that you have part of what you need, and really would like to just start there and try to get another chunk. Some http and ftp sites do support resuming downloads, but not all do, and it's not necessarily easy to find out whether or not they have the feature. However, rsync always can do that, so I decided to use that instead.
There are not many rsync servers listed at
http://fedora.redhat.com/download/mirrors.html (link dead, sorry),
but distro.ibiblio.org is one. The http://www.fedora.us/wiki/FedoraMirrorHOWTO(link dead, sorry)
site tells you how to setup for rsync, but it is typical geek:
because most of the people who would to do that are fellow geeks,
it doesn't really tell you what you need to do. The asssumption is
that illiterates will use ftp or http, and anyone else just needs
to be faced in the right direction.
There's also an implicit assumption that you aren't going to experience any transmission problems. They suggest using "rsync auv", which is NOT what you want to do if your session is locking up or failing and you have not yet got any of what you want.
The other problem is that the instructions assume that you want to mirror; that is, provide an identical directory structure to what the rsync server has. In the case of just trying to get the darn cd images, that may not be what you want at all. For example, I don't want the Source RPMS's just now: I'll get them later, but I want to get going with the installation first. If I followed the suggested scripts, I'd have to wait for all the SRPMS to download first, and even with cable, that's not quick. But how to find out what I wanted? Well, rsync can help:
apl$ rsync rsync://distro.ibiblio.org/fedora-linux-core/* . Welcome to distro.ibiblio.org's rsync server! You may mirror any of our distributions by using the "distros" module. Simply append the name of the distribution you would like to use. For example: rsync -avr distro.ibiblio.org::distros/bluelinux/ /local/path/ If you will, please subscribe to our mirror announcement list: http://lists.ibiblio.org/mailman/listinfo/ibiblio-mirrors If you have problems or questions, feel free to mail firstname.lastname@example.org Thanks! skipping directory /1 skipping directory /development skipping directory /test skipping directory /updates client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
Ahh. That showed me that the subdirecories were 1, development, test, and updates. I expanded a bit:
apl$ rsync rsync://distro.ibiblio.org/fedora-linux-core/1/* . Welcome to distro.ibiblio.org's rsync server! You may mirror any of our distributions by using the "distros" module. Simply append the name of the distribution you would like to use. For example: rsync -avr distro.ibiblio.org::distros/bluelinux/ /local/path/ If you will, please subscribe to our mirror announcement list: http://lists.ibiblio.org/mailman/listinfo/ibiblio-mirrors If you have problems or questions, feel free to mail email@example.com Thanks! skipping directory /1/SRPMS skipping directory /1/i386 client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
OK, now we are getting closer. I continued this until I knew what I wanted. It was then time to use a different invocation:
rsync -auv --progress --partial rsync://distro.ibiblio.org/fedora-linux-core/1/i386/iso/yarrow-i386-disc* .
The important additions are the --partial and --progress. The latter lets me know when the transfer has hung up, and the --partial lets me restart where I left off.
When rsync goes after a file, it first brings it to a temporary file:
apl $ ls -l .yarr*;sleep 10;ls -l .yarr*;sleep 10;ls -l .yarr* -rw------- 1 apl staff 1048576 10 Nov 08:36 .yarrow-i386-disc1.iso.UypdBA -rw------- 1 apl staff 1900544 10 Nov 08:36 .yarrow-i386-disc1.iso.UypdBA -rw------- 1 apl staff 2883584 10 Nov 08:36 .yarrow-i386-disc1.iso.UypdBA
If you haven't specified --partial, interrupting the rsync will cause that file to be deleted. However, if you do specify --partial, it will be copied to the final file. If you already had the final file, and were just looking for a newer version, you would NOT want to specify --partial, but as I did not have it yet, this was exactly what I wanted.
If you interrupt that rsync, the temporary version gets copied to yarrow-i386-disc1.iso. Start it up again, and yarrow-i386-disc1.iso is copied to a new temporary file and then the download resumes:
-rw-r--r-- 1 apl staff 3112960 5 Nov 17:39 yarrow-i386-disc1.iso -rw------- 1 apl staff 3506176 10 Nov 08:37 .yarrow-i386-disc1.iso.Yw3T5d -rw------- 1 apl staff 7733248 10 Nov 08:38 .yarrow-i386-disc1.iso.Yw3T5d -rw-r--r-- 1 apl staff 3112960 5 Nov 17:39 yarrow-i386-disc1.iso -rw------- 1 apl staff 9011200 10 Nov 08:38 .yarrow-i386-disc1.iso.Yw3T5d -rw-r--r-- 1 apl staff 3112960 5 Nov 17:39 yarrow-i386-disc1.iso
Had to interrupt here again. Notice how the yarrow-i386-disc1.iso is increased by what was downloaded.
-rw-r--r-- 1 apl staff 9863168 5 Nov 17:39 yarrow-i386-disc1.iso
-rw------- 1 apl staff 7143424 10 Nov 08:39 .yarrow-i386-disc1.iso.Aawu8W -rw-r--r-- 1 apl staff 9863168 5 Nov 17:39 yarrow-i386-disc1.iso -rw------- 1 apl staff 10518528 10 Nov 08:39 .yarrow-i386-disc1.iso.Aawu8W wrw-r--r-- 1 apl staff 9863168 5 Nov 17:39 yarrow-i386-disc1.iso
Well, I don't want to watch this thing for hangups all day, so one more change:
while true do rsync --timeout=60 -auv --progress --partial rsync://distro.ibiblio.org/fedora-linux-core/1/i386/iso/yarrow-i386-disc* . sleep 1800 done
That let me leave this thing running. If it gets nothing in a three minute period, it sleeps for thirty minutes, and tries again. Unfortunately, this still didn't result in much success: after a full day of trying, I only had most of one CD pulled down.
Someone had suggested using BitTorrent, which is a distributed downloader - wonderful idea (see http://bitconjurer.org/BitTorrent/ for details), though I hadn't realized they had a Mac version - as the aborted RedHat 9 install killed my Linux box, I need to download these on my Mac. I had to open some ports on my firewall, and I hoped this would be the final solution. I started up BitTorrent, gave it the http://torrent.dulug.duke.edu/yarrow-binary-i386-iso.torrent url, and off it went. In a few minutes I had as much of the first CD as rsync had managed to get after trying all day, I certainly had good expectations. I did find it odd that BitTorrent takes a long time to settle down to a real representation of what the actual download time will be. I could tell almost immediately (based on the reported download rate) that this would take a little more than two hours, but the BitTorrent display took quite a while to realize that.
I went off and had supper and a small birthday celebration for my wife, and when I returned all the CD's were downloaded. The BitTorrent docs say there is no point in checksumming their downloads, but I did it anyway. I left BitTorrent running for a while after I got my images; that's how you pay back for using it.
Now all that is left is to burn the CD's and actually install Fedora, but that's another day.
Got something to add? Send me email.
(OLDER) <- More Stuff -> (NEWER) (NEWEST)
Printer Friendly Version
Increase ad revenue 50-250% with Ezoic
Inexpensive and informative Apple related e-books:
Take Control of the Mac Command Line with Terminal, Second Edition
Take Control of Numbers
Take Control of OS X Server
iOS 10: A Take Control Crash Course
iOS 8: A Take Control Crash Course