APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Analysing Web Logs for Search

People will often find you and your website through Internet searches. When they do, your log file will usually contain information on what they were searching for when they found you.

The following is a typical log entry showing a visit that came from a Google search. I bolded the "q=" section that shows the actual search string: - - [05/Oct/2009:02:23:50 +0000]
"GET /Forum/TonyLawrence13.html HTTP/1.1" 200
20191 "http://www.google.com.sg/search?hl=en
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv: Gecko/20090824 Firefox/3.5.3 (.NET
CLR 3.5.30729)"

Bing and many other search engines also use the "q=" format, so it's easy to mine this data from your logs.

This is obviously very useful information. Unfortunately, the search engine may not be sending your visitor to the page you'd really like them to hit. The page they got may indeed be related to their search, but when you as a human look at what search words they actually used, you would have sent them to a different page.

I look for that in my logs. When I spot something like that, I'll update the found page with a link and text that suggests the other page as possibly being something of interest also. Note that I don't want to just redirect to that page, because for other search keys the original page might be a perfect match.

Or I may find that all I need is to pull something from another page to update the page the search engines favor. The important thing is to help your visitors find what they need - if you aren't reading your logs, you don't know that they are or are not.

In the log entry shown above, the person was searching for "qmail queues stats". The page Google picked is appropriate in that case. However, I often find searches that aren't quite correct.

Analysing these logs can be a bit boring - often the search engines pick exactly what I would have picked. However, I find small problems often enough to make the effort worthwhile.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> -> Analysing Web Logs for Search

Increase ad revenue 50-250% with Ezoic

More Articles by

Find me on Google+

© Anthony Lawrence

Kerio Samepage

Have you tried Searching this site?

Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

I am not out to destroy Microsoft, that would be a completely unintended side effect. (Linus Torvalds)

This post tagged: