APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Getting your pages indexed

© February 2006 Anthony Lawrence

I've written about this numerous times, but there have been some changes recently, so perhaps it is time to revisit this subject.

The first rule never changes: if you build it, they will come. However, you could wait a long, long time for the search engines to find you if you do nothing to help them.

The second rule hasn't changed either: the more often you update your site (add new pages), the more interesting you are to the engines and the more often they will return.

But after those two rules, things have changed. First, RSS has become very important. You need an RSS Feed, and you need to register with sites like Technorati, Syndic8 and so on - even though those sites may not do much for you.

I use Feedburner for my RSS feeds.

Versatile Site Map Generator $49.00
A1 Sitemap Generator

Secondly, both Google and Yahoo now let you help them with "sitemaps". For Yahoo, that's just a simple listing into a file they want you to call "urllist.txt" or "urllist.txt.gz" if it's big enough to compress. On a Unix or Linux box, you can easily create that with "find" and "sed":

find . -name "*.html" | sed 's?^.?https://yoursite.com?' > urllist.txt

You then visit Yahoo's free submit page and put "https://yoursit.com/urllist.tx" into the "Enter your URL" box. That's it - Yahoo takes it from there. All you need to do is update urllist.txt; you don't need to tell Yahoo about it again.

For Google, it's a little more complicated. Google does offer a free Sitemap Generator, but it's Python code. I have a Perl version at Perl code for Google sitemap generation. Like Yahoo, you visit Google Webmaster Tools once to tell Google about your sitemap, but unlike Yahoo, you'll probably want to go back now and then because Google gives great feedback on the activities of its crawlers, including errors and a tool to test your robots.txt file.

If possible, you'll also want to submit your site to DMOZ and to any other directories that make sense for your market. If there are relevant newgroups or forums that you post in, include a link in your signature. These are minor points, but they can help with overall popularity.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> Getting your pages indexed

Inexpensive and informative Apple related e-books:

Take Control of Apple Mail, Third Edition

Take Control of Pages

Take Control of Preview

Take Control of Upgrading to El Capitan

El Capitan: A Take Control Crash Course

More Articles by © Anthony Lawrence

Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

Printer Friendly Version

In C++ it's harder to shoot yourself in the foot, but when you do, you blow off your whole leg. (Bjarne Stroustrup)

Linux posts

Troubleshooting posts

This post tagged:



Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode