APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Getting your pages indexed

I've written about this numerous times, but there have been some changes recently, so perhaps it is time to revisit this subject.

The first rule never changes: if you build it, they will come. However, you could wait a long, long time for the search engines to find you if you do nothing to help them.

The second rule hasn't changed either: the more often you update your site (add new pages), the more interesting you are to the engines and the more often they will return.

But after those two rules, things have changed. First, RSS has become very important. You need an RSS Feed, and you need to register with sites like Technorati, Syndic8 and so on - even though those sites may not do much for you.

I use Feedburner for my RSS feeds.

Versatile Site Map Generator $49.00
A1 Sitemap Generator

Secondly, both Google and Yahoo now let you help them with "sitemaps". For Yahoo, that's just a simple listing into a file they want you to call "urllist.txt" or "urllist.txt.gz" if it's big enough to compress. On a Unix or Linux box, you can easily create that with "find" and "sed":

find . -name "*.html" | sed 's?^.?http://yoursite.com?' > urllist.txt

You then visit Yahoo's free submit page and put "http://yoursit.com/urllist.tx" into the "Enter your URL" box. That's it - Yahoo takes it from there. All you need to do is update urllist.txt; you don't need to tell Yahoo about it again.

For Google, it's a little more complicated. Google does offer a free Sitemap Generator, but it's Python code. I have a Perl version at Perl code for Google sitemap generation. Like Yahoo, you visit Google Webmaster Tools once to tell Google about your sitemap, but unlike Yahoo, you'll probably want to go back now and then because Google gives great feedback on the activities of its crawlers, including errors and a tool to test your robots.txt file.

If possible, you'll also want to submit your site to DMOZ and to any other directories that make sense for your market. If there are relevant newgroups or forums that you post in, include a link in your signature. These are minor points, but they can help with overall popularity.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> Getting your pages indexed

Increase ad revenue 50-250% with Ezoic

More Articles by

Find me on Google+

© Anthony Lawrence

Kerio Samepage

Have you tried Searching this site?

Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

If we define Futurism as an exploration beyond accepted limits, then the nature of limiting systems becomes the first object of exploration. (Frank Herbert)

This post tagged: