APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

Analysing Web Logs for Search


© October 2009 Anthony Lawrence

People will often find you and your website through Internet searches. When they do, your log file will usually contain information on what they were searching for when they found you.

The following is a typical log entry showing a visit that came from a Google search. I bolded the "q=" section that shows the actual search string:

119.75.43.62 - - [05/Oct/2009:02:23:50 +0000]
"GET /Forum/TonyLawrence13.html HTTP/1.1" 200
20191 "https://www.google.com.sg/search?hl=en
&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=agd
&q=qmail+queues+stats&btnG=Search&meta=&aq=f&oq="
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET
CLR 3.5.30729)"
 

Bing and many other search engines also use the "q=" format, so it's easy to mine this data from your logs.

This is obviously very useful information. Unfortunately, the search engine may not be sending your visitor to the page you'd really like them to hit. The page they got may indeed be related to their search, but when you as a human look at what search words they actually used, you would have sent them to a different page.

I look for that in my logs. When I spot something like that, I'll update the found page with a link and text that suggests the other page as possibly being something of interest also. Note that I don't want to just redirect to that page, because for other search keys the original page might be a perfect match.

Or I may find that all I need is to pull something from another page to update the page the search engines favor. The important thing is to help your visitors find what they need - if you aren't reading your logs, you don't know that they are or are not.

In the log entry shown above, the person was searching for "qmail queues stats". The page Google picked is appropriate in that case. However, I often find searches that aren't quite correct.

Analysing these logs can be a bit boring - often the search engines pick exactly what I would have picked. However, I find small problems often enough to make the effort worthwhile.


Got something to add? Send me email.





(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

->
-> Analysing Web Logs for Search


Inexpensive and informative Apple related e-books:

Take Control of iCloud, Fifth Edition

Digital Sharing Crash Course

Take Control of Upgrading to El Capitan

Are Your Bits Flipped?

Take Control of Preview




More Articles by © Anthony Lawrence




Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us


Printer Friendly Version





Zawinski's Law: Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. (Jamie Zawinski)




Linux posts

Troubleshooting posts


This post tagged:

Blogging

Web/HTML



Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode