Too much traffic

This website limps along serving up a little less than 200,000 pages a month. If you divide that up, it's a bit over 6,000 a day, which comes down to 4 or 5 pages per minute. That's hardly a lot to ask of any webserver.

Of course sometimes we get peaks. There's not much happening at 3:00 AM so obviously there's more happening at noon. Every now and then some article gets a little burst of popularity despite my best efforts to discourage such things. Those cause peaks too.

But lately we've been getting some plain old peaks. I mean that they don't come because a particular page has attracted some attention; they come just because a lot of people came at once. For example, we had one just a few minutes ago and it hit just about one page view per second. As each page also has to pull in other stuff, the load on the server was three times that and it started throwing errors:

"unable to include /cgi-bin/blah-blah.pl", "Premature end of script headers:", "Permission denied".. not good.

Well, you can sympathise. If that load kept up all day and all night, I'd be serving over 2 million pages per month and that might be a little too much to ask of a shared web server.

So what to do? Well, I could use mod_perl. That's annoying, but it's probably what I have to do. Unfortunately, my code is far from ready for mod_perl so that will take some time and effort.

I could create static pages. That would be even more annoying.

I could ask more of you to use RSS. Go ahead and do that. I don't think it will help that much.

I could switch to a more expensive server. I really don't want to do that either.

Any other suggestions?

Click here to add your comments




Thu Feb 12 21:40:58 2009: Subject:   NickBarron


How odd, I was just thinking about this yesterday.

How popular this site had become over the last year or so, even with your best efforts to discourage it ;)



Thu Feb 12 21:48:55 2009: Subject:   TonyLawrence


Funny, isn't it: everybody else is begging for more RSS subscribers, asking people to vote up their posts on Digg, sucking up to anyone who might give them a Tweet.. and I want exactly the opposite: keep the unwashed masses away and I'll be happy..

Really, traffic is fine and I will have to deal with it. Mod_perl is the simplest way I guess.. I just have to clean up my sloppy programming :-)





Fri Feb 13 13:52:17 2009: Subject:   MikeHostetler
http://mike.hostetlerhome.com

If you didn't have so many things to say . . . .

mod_perl would work, but if your traffic keeps going up, you may have to deal with a "caching" solution that periodically generates static pages. This has become the accepted WordPress solution:
http://wordpress.org/extend/plugins/wp-super-cache/






Fri Feb 13 13:57:24 2009: Subject:   TonyLawrence


Yeah, after sleeping on it I started thinking that rewriting to make static pages would be easier than rewriting everything for mod_perl. I'll start working on that today.



Fri Feb 13 17:10:12 2009: Subject:   TonyLawrence


OK, just finished up the code. I'm not going to convert everything at one - I'll test a few pages first to make sure I didn't bollix it.. so I'll test with few high volume pages and then roll it out..



Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments

If you want a picture to show with your comment, go get a Gravatar

Versatile Site Map Generator $59.00
Buy A1 Sitemap Generator

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.