I call that "scramble" and leave it running as "while :;do scramble mywords;done" (that way I can add words while it is running or have it use different lists).

Basic TrueCrypt Usage

I and other people here have mentioned TrueCrypt before. I thought (and perhaps you did too) that it was very simple and obvious to use but I've had several people write to me complaining that they downloaded and installed it, but have no idea what to do next.

OK, maybe the interface isn't all that user friendly. It really is simple, but after looking at it from an "ordinary person" perspective, I can agree that it could leave you staring at the screen saying "Huh?" So let's run through using this in plain English.

What you don't want to do

The most common fear I heard from people was that they were afraid TrueCrypt was going to encrypt their hard drive and that something would go wrong or they'd forget the password. Yes, TrueCrypt can encrypt entire hard drives, and yes, things could go horribly wrong or you could forget your password. So, yes, you have reason to be concerned. I definitely would NOT advise using TrueCrypt for that purpose unless you completely understand what you are doing, what the risks are, and (perhaps most importantly) WHY you are doing it.

Most of us need to protect individual files. Maybe you have a text file with all your passwords in it. Maybe you handle sensitive documents for your clients. Whatever it is, you usually don't need to encrypt a whole drive. You just need to lock up those particular files.

Protect a file or files

This is the simplest and safest TrueCrypt operation. Start up TrueCrypt. You've never used it before, so what you want to do is click on Create Volume. You want to create an "Encrypted File Container" (that's the default). Click Next, and then select "Standard True Crypt Volume" (again that's the default). What happens next seems to confuse people: a file dialog comes up, which perhaps makes you think that you need to select some file.

No, it's looking for you to give the name and location of a NEW file. This file will be the "container" for the files you actually want to hide. It's going to eventually end up as another disk drive on your system, which is perhaps another reason this can confuse folks: it's a "volume", it's a "container", it's a disk drive. No wonder people are hesitant to proceed!

So click on "Select File", navigate to where you want to keep this, and give it a name. Remember, this is the "container". It's the box your secret files will hide in. You might call it "Secrets", "My Secret Stuff" or "Fred" - choose something that makes sense to you. IF YOU CHOOSE AN EXISTING FILE, IT WILL BE DELETED.

So, after choosing a name, click Next and the following screen asks what kind of encryption you want to use. For most of us, the default AES is fine. The TrueCrypt help file suggests reasons why you might choose one of the others:

If you store the backup volume in any location where an adversary can make a copy of the volume, consider encrypting the volume with a cascade of ciphers (for example, with AES-Twofish- Serpent). Otherwise, if the volume is encrypted only with a single encryption algorithm and the algorithm is later broken (for example, due to advances in cryptanalysis), the attacker might be able to decrypt his copies of the volume. The probability that three distinct encryption algorithms will be broken is significantly lower than the probability that only one of them will be broken.

You can take the default choice for the hash algorithm and click Next.

Now you need to choose the size of your container. Obviously it needs to be large enough to store the files you want to hide, but you may want to think about making more than one smaller container. For example, if you are going to store backups of this container (a good idea!), you might want to do that on a CD or DVD - obviously the container size has to be small enough to fit on the storage media. Or perhaps you plan on using one of the many free Internet storage sites - your choice of size may be limited by what they will give you for free.

There's also a minimum size - not because TrueCrypt really cares, but because your operating system can't create a disk drive (which is what this container ultimately becomes) smaller than a minimum size. Once you've decided how big or small this wiill be, you click Next and it's time to choose a password.

Think of a sentence

TrueCrypt isn't looking for "joe123" or even :"P^%WErt45!@p.k" . It's looking for a long sequence - they recommend at least 20 characters and you can use up to 64.

You could make up a long string of nonsense, but how are you going to remember "Ht^%f2HH(hpo&mnE$%d";q\n*^$sdf"? I suggest using a phrase - a sentence - that you can remember. It might be words from a song: "Memories are all I have to cling to - cling to!" or a string of names: "Thomas, Jonathan, Sarah and THEN William!". If you always keep the books on your shelf in the same order, maybe you could use their titles: "Programming Perl, Perl Cookbook, Linux Firewalls and Linux Cookbook". It is best if you can include some random punctuation, but if this password is never going to be written down and will live only in your head, it's better to be a little more weak than risk forgetting it - once you've locked your files up with this, they are not coming back without that password!

A weaker password can be augmented with a "key" file or files. These are simply files that TrueCrypt takes 1024 bytes from and mixes into your encryption. You can use any file (or multiple files) on your disk as long as the first 1024 bytes of it will never change. You could use a file stored on a USB stick - if someone stole your computer but didn't get that USB drive, they can't open your TrueCrypt files even if they have the password. Of course, you can't either - you have to have the key file(s) available to get at your stuff.

Once you have decided on your password and any key files, it's time to actually create the container. You'll be asked to move your mouse randomly for a bit and then click Format. The purpose of the random mouse stuff is to generate better encryption, so just do it even though it sounds like someone might be pulling your leg. After you click Format and TrueCrypt says it is all done, you can exit back to the main screen.

Adding Files

You have now created a container. You haven't put anything in it yet and to do that, you need to mount it as a disk drive. You'd think you cold just click "Mount" and TrueCrypt would ask you what you want to mount, but no, you need to first click "Select File", find your container, point at your key file(s) if you used any), and then click "Mount". You can select what drive letter or (Mac) volume identifier to use and once it is mounted you can exit TrueCrypt - you can unmount the container using ordinary operating system methods if you wish.

While it is mounted, you can put files in it. I suggest keeping safe copies of your files until you feel completely comfortable with TrueCrypt - remember, if you can't recall the password or lose any required key files, you will have no access to your data.

That's it. After you have loaded the drive up with files, you unmount it and that's it - the encrypted container is protected by your password and any key files you specified. It was pretty simple, wasn't it?

Internet Scrabble

Our family has recently discovered online Scrabble (the name Scrabble is a trademark of Hasbro, Inc. in the United States and Canada and of Mattel elsewhere) through Facebook. We've been a Scrabble playing family for many decades; we all love the game, but we are seldom all together to play and even when we are, we usually have other things to do.

Internet Scrabble changes all that. You can make one move per day or even less. More importantly, you can play more than one game at once: I have one game going with my wife, one with each of our daughters, and one with the four of us. I also have more games going with friends and a few with strangers. I can spend as much or as little time playing as I like - and so can all my opponents.

Playing Scrabble on-line is different in other ways. There's no arguing about words: the game enforces acceptable words. On the other hand, words our family never would have allowed in home games can be used; in one of my first games with a stranger, I was shocked to see "NGWEE" appear. That word is the Nigerian unit of currency, but I'm sure you knew that. I'm sure you also know that "AA" is a dry from of lava and that "DUIT" is a Dutch coin - and that all of them are acceptable words in Facebook Scrabble.

Another issue, especially when playing with strangers, is the easy availability of on-line anagram solvers. In our family games, we allow checking the dictionary for spelling and minor word-hunting, but using an anagram program is a bit much. On the other hand, there is a fair amount of strategy and plain old luck to Scrabble, so the cheaters aren't guaranteed a win. Finally, if you really feel someone must be using an anagram program, you can fight back by doing the same yourself.

Speaking of anagram solvers, there are Scrabble robots at other sites. I haven't tried playing against one, so I can't say how good they are at strategy - I'm sure they are pretty good at finding the best scores, but that's not always the best play, so a human may still have a fighting chance.

So, if you are wondering what I'm doing in between working, eating, sleeping, breathing, and all that, well, I'm probably playing Scrabble. Maybe I'll see some of you on the other side of the board!

Another popular Scrabble site is Internet Scrabble Club. They have timed play there, but when I tried it out they were having server problems and that made playing very difficult. I don't know if that's usual or unusual.

Detecting Comment Spam, Part 3

This is a continuation of Detecting Comment Spam, Part 2

In the previous posts in this series, I've said that spammers habits allow us to detect their attempts to leave inappropriate comments. I use these techniques here and am able to send most spam comments directly to the bit-bucket without ever having to examine them myself. When I suspect spam but am not sure, I just send the comment to moderation. In practice, very few spam attempts get by the automatic filters.

The code I use is a hodgepodge I developed over many years of fighting spam comments. I need to rewrite it and my thought is to move toward a scoring system similar to that used by Spamassassin for mail spam: each "bad habit" increases an overall spam score and the final judgement is made based on the total score accumulated. To that end, I have been pulling out the various tests I use now and examining them. Some always cause the comment to be treated as spam; those would add enough points to ensure that would still happen. Other habits now cause moderation, but in my new design each of those will increase the spam score. Any spam score at all is cause for moderation, but if the post accumulates enough minor points, I can skip that and just throw it away.

By the way, there is an effort to create a BlogBlogSpamAssassin.

Seven (or more) habits of ineffective spammers

I'll be reviewing things covered in the previous two posts and will introduce some new ideas. Items covered in the previous posts are marked with a "*".

* Known spam links: Reusing links we already know are spam. This would carry enough points to always be spam.

* High link to text ratio: Sometimes nothing but links. This could be legitimate, but it's a strong indicator. In my current code, this always treats the comment as spam, but I think I will change it so that at least one other spam point is required.

* Nonsense words: The higher the ratio of nonsense to total word count, the more likely this is spam.

* Many quick posts: Legitimate commenters may post more than once per day, but there will be some time delay between comments as they read the articles they are commenting on. I simply enforce a "posting too frequently" policy.

* Direct posts: Some spammers bypass your forms and send direct POST requests. That's definite spam.

* Typed too fast: Legitimate commenters MAY use cut and paste, but moving from form load to POST too quickly needs to carry some weight.

Multiple posts to same article: Legitimate commenters almost never post more than one comment without some other comment intervening. Sure, someone may have an afterthought and add a second comment, but more usually this indicates a spammer, so we should add points.

Same comment at another article: As always, spammers are lazy. They sometimes post exactly the same text to different articles. The posts may not come from the same IP address, but the content is the same. This should probably carry enough points for flat rejection. It does mean that you need to maintain a database, but you only need to store 100 characters or so (spam comments are usually short) and you can clean it out after 24 or 48 hours.

Text and link have same word: This is a minor spam indicator, but if the word "Horrivea" appears as "Buy Horrivea: http://someaddress/horivea", it may be spam.

Failure to reload: I require clicking on a link to reload the original page after submitting a comment. A legitimate poster will almost always click, a spammer almost never will. Again, this is not an absolute indicator, but is worth a point or two.

External tests

Akismet is a popular comment checker. I have found it to be less effective than my own tests, but it can't hurt to have another opinion, can it?

Captcha tests: See Creating a blog... with anti-spam measures for an example of incorporating Captcha. As this technology is annoying, you might consider adding it as a confirmation only when some other conditions have raised suspicion. Similar tests involve solving simple math or providing answers to obvious questions. Spammers are in a hurry; they are usually using bots and if not they just don't want to spend time thinking, however small that time is.

Legitimate Commenters Habits

There are a few things that spammers don't usually do. Legitimate comments almost always use punctuation, so the presence of punctuation could decrease a spam count. Legitimate posters sometimes quote text from the article or from a previous comment; spammers almost never do that, so again we might decrease a spam score if we see this.

Conclusion

We can control spam comments. Not all of it, but we can limit the amount that has to be checked manually and we don't need to annoy our visitors with required registration or automatic "human reader" testing.

I'd love to hear your thoughts, comments and ideas.

Detecting Comment Spam, Part 2

This is a continuation of Detecting Comment Spam, Part 1

In part 1, I talked about code to read a list of spammish words from a file and look for those words in comment posts. Commenters pointed out that spammers will obfuscate words with dashes, spaces, bizarre spellings and so on, making it very difficult to catch these programmatically. That's true, but there's more to the story.

The spam list I use here has some of those common spam words, but most of it is taken up with web addresses. Links are much more difficult for spammers to mangle - they can use redirection at the destination site, but the site itself is static: if a spammer wants you to visit some page at Iamaspammer.com, either that name or its IP must be in a link. Most of my spam list is websites that have been the destination for spammers links. Once the site is in my list, the spammer can never post anything with that link in it - no matter how they mung other words, they will never be allowed to post. A comment here that contains one of those links doesn't even go to moderation - it just gets flatly rejected.

Spammers do move on - jklljas.blogspot.com may be a spam link now, but it will get abandoned eventually. I trim the list every month to remove old entries.

IP Blocking

Should't I just block the spammer's IP from leaving comments? Yes, but spammer's IP's change over time - their IP gets blocked everywhere so they move on. If you block a particular IP forever, it may end up being the IP of a legitimate user, so you probably don't want to do that. There's is also the issue that your list of banned IP's could get very large over time.

For some websites, it makes sense to block by country of origin. I don't do that here, but if you only want U.S. visitors, you could certainly do that. See Blocking Unwanted Visitors.

You can do the blocking inside your script or use .htaccess (or your Apache configuration files). I prefer to block inside my comment posting script because if I'm wrong or if the IP has transferred to a non-spammer, I'm only blocking them from commenting, not from any site access. In extreme cases (such as a spammer attempting to use any cgi script it can find or guess) I will add them to the .htaccess file. However, whether in scripts or .htaccess, I don't keep the ip blocked for more than a week at a time.

For me, the need to block by IP is infrequent enough that I don't need to automate the removal of bans, but it isn't difficult to write such code if needed.

Moderation

Let's talk about moderation for a moment. Some sites moderate all comments, but that's annoying for both the moderator and the people leaving comments. Regular posters shouldn't have to wait for their comments to appear and most web site owners have better things to do than moderate comments all day long.

One solution is to require registration. If a user can provide a login and password that has been previously approved, their comment can be posted immediately. Many sites use that scheme, but there is still a degree of annoyance: the registration process is an extra step that annoys some people.

I do something similar here, but there's no registration process per se. If you have posted here legitimately in the past and are posting again from the same IP address and with the same username (actual username, not "anonymous") your post will not need to be moderated - assuming it passes all the other tests I'll talk about below!

But as alluded to when talking about destination links above, there's no need to moderate posts that are definitely spam - we just throw those away.

Other spam control

I'm going to talk frankly about all the things I do here to limit spam and what I plan to do in my next version of the commenting software. I suppose there is some small risk to this; I've been reluctant to discuss all of it before because I don't want to help spammers learn better ways to bypass systems like this, but I don't think spammers are likely to read this and if they do, oh well: the war goes on.

Lazy Spammers

Spammers aren't generally going to spend a lot of time and effort on posts. The pros are using scripts, the inept are probably at least automated enough to use cut and paste. Even if they are typing comments in, they probably tend to reuse the same words and links.

Not much text

Some habits of lazy spammers are fairly easy to block. One is a habit of only leaving links with little or any text. That's easy to detect: just count the total words in the posts and divide by the number of http links: if the result is too small, the post is probably spam. I do that here now - if you leave a comment that is only a link, it won't get posted and it won't even go to moderation - whether the link is legitimate or not. That might be too draconian, but I think you should at least flag such posts for moderation. I do need to fix my present code to allow known user/ip posters to leave such short comments.

Nonsense words

When lazy spammers do pad their word count, they often use nonsense words. Again, that's relatively easy to thwart if you have the luxury of time and disk space- look up each word in a dictionary and compare found words to unfound - if the ratio is too high, this is probably spam. But that requires a fair amount of work. I take a simpler approach:

If $in is the text to be checked, the Perl expression

$consonants++ while $in =~ /[qwrtpssdfghjklzxcvbnm]{4,}/ig;

counts the number of times four or more consonants appear in a row. That is, it counts garbage like "fghk" or "hdfr" - the kind of junk you'll get from random banging on the keyboard. If that count is high when compared to the number of words in the input, you probably have spam. My code says if the ratio of nonsense to words is over 15% it's spam. This is much faster than checking input against a dictionary and is very effective.

Remember that "jklljas.blogspot.com"? If all he posts is something like "Xanax: http://klljas.blogspot.com", he's got a 50% nonsense count ("http" and "jklljas" against 4 words) - that's enough to count as spam right there. Not all spammers use nonsense words in their sites, but quite a few do, and many are too lazy to type much more than that.

This won't catch all nonsense: we've all seen random phrases from books used as a preface to a link. Nonetheless, this stops SOME spam, and every post we stop is one that doesn't annoy us or our readers.

Direct POSTS

Another thing spammers do is send direct POSTS. That is, their automated software examines your comment form once, picks out the fields it needs to supply, and then submits multiple POST requests. I'll don't allow that: when the comment form is first loaded, the commenter's IP address is stored in a database. When they actually POST, the IP is checked and immediately removed. If the IP doesn't exist (which it will not after the first POST), the comment is thrown away. The spammer can defeat this by requesting the form before doing each POST, but many of these folks are too lazy to bother.

Typed too fast

After reading a link suggested in the comments, I realized that there out to be a minimum thinking/typing time also. So along with the IP, I store the time the form was loaded. When the POST is made, I count the words and divide by 10 - if at least that many seconds haven't elapsed, I won't allow the post. Only a cut and paste spammer can type more than 10 words per second, so that's a fair limit - maybe even less is fair, but a legitimate poster might cut and paste some text..

Excessive posting

Some spammers are greedy. They aren't content with putting one piece of graffiti on your site; they want to leave many spam posts. I use a timing algorithm to control that. It's simple enough: for each post from your IP, a counter is incremented. I use that counter to determine how long before you are allowed to post again. You can't make your second post until 15 seconds after your first, your third until 120 seconds after that, your fourth until 405 seconds and so on - it's the number of posts cubed times 15 seconds. This very effectively stops greedy spammers - they don't hang around. In the current version, these limits affect everyone (even me!) but I want to make them less restrictive (maybe posts cubed times 5 seconds) for known users in the next version.

The war is never over

So that's how I control spam comments here. Most obvious spam goes right to a black hole, users I know and trust get posted immediately, and everything else goes to moderation. I'm going to add an Akismet check in the next version - it never hurts to have a second opinion!

Instead of moderation, you could also throw up Captcha or arithmetic challenges, or require a random series of other answers or actions when a post is suspect. A spammer (especially an automated spammer) probably won't respond to even a simple "Click here to confirm your post" - I may add something like that to my next version. That's a minor annoyance for a first time poster or someone who insists upon using "anonymous", but it will stop most spammers dead.

Of course it is impossible to stop all spam. No matter what we do, we'll at least have to moderate a post now and then. However, with the spam controls I have here, I very seldom see spam - and if I do see it, I'm unlikely to see it again, because I'll adjust my code as necessary. I do have to moderate new visitors and frequent posters with new IP addresses, but that's not very onerous.

The war on spam never ends, but we can win most of the battles.

See Detecting Comment Spam, Part 3 for the continuation of this series.

/Web/detecting-comment-spam2.html copyright and reprint notice

Comments /Web/detecting-comment-spam2.html

Wed Dec 23 20:37:03 2009 MikeHostetler

http://squarepegsystems.com

I think you have a good plan here -- it's one thing for the spammers to know what you are doing to stop them, it's another for them to find a hole in your system. One thing about a good plan is that it has to be adaptable -- you may lose a battle here and there, but if you learn from your mistakes and tweak here and there, you will win many more than you will lose.

It would be interesting if you had a "minority report" bucket for comments -- ones that your system and Akismet have different opinions on. Maybe you can review them manually for a while -- perhaps you will learn a few things. Or maybe you will find out that you have a better system than they do!

Another interesting idea is to see if you can hook up SpamAssassin also help. I can't imagine the rules for email spam to be tons different than blog spam. Or maybe they are. Some notes have been done on it, but I think that's about it:
http://wiki.apache.org/spamassassin/BlogSpamAssassin

Wed Dec 23 20:41:46 2009 TonyLawrence

You'll be amused by this, Mike: Akismet says your comment is spam :-)

Obviously I disagree. I'll take a look at that Spamasassin link, thanks.

Wed Dec 23 20:51:38 2009 TonyLawrence

A comment at that Spammassin link gave me another idea:

When the form is loaded, I store the time against the users IP. When they hit POST, it's allowed if the time lapse is less than 2 hours (that's a pretty generous time limit to compose your post). The time is reset to zero upon posting, so a new POST without a form load won't work.

The Spamassasin link suggested that a minimum time makes sense too - a real user presumably spends at least a few seconds composing a message. That makes sense - I'll be adding something to do that.


Wed Dec 23 20:57:51 2009 TonyLawrence

Yeah, that makes sense: count the words, divide by 10 (pretty fast typing). If it hasn't been that many seconds between loading and posting, disallow.

Wed Dec 23 21:49:22 2009 MikeHostetler

http://squarepegsystems.com

That's hilarious that they marked me as spam. Probably because I used a link. See -- evidence that you need a multi-prong attack!

Thu Dec 24 01:51:57 2009 TonyLawrence

Subject:

Website: I don't think it was the link - I tried running it again to see why, and it didn't hit that time. We'll see what happens over time.

Thu Dec 24 02:04:17 2009 TonyLawrence

Subject:

Website: Ooops - I see part of the problem. I introduced a bug in my code so that the akismet got called twice. It was the first one that failed - but why would two calls with the same data return different results?

Thu Dec 24 02:06:22 2009 pcunixtony

Subject:

Website: Just testing with a different name to see the bug.

Thu Dec 24 03:12:01 2009 TonyLawrence

I found the problem. As I mentioned before, the old server sent cgi href's as GET's - this one always uses POST. My comment code was still expecting a GET on the first load - which led me to check Akismet twice, the first time without any data.

Fixed now (I hope).

Thu Dec 24 13:27:41 2009 TonyLawrence

Just as a general point of interest:

Last night this code caught 4 comments, three of which were pure spam and thrown away and one which I had to moderate (and subsequently toss because it was spam). Akismet didn't think that one was spam amd I don't send the pure junk to them for checking.

I don't track attempted comments thrown away because of posting too quickly but I can see in the logs that spammers do get caught by that.


Thu Dec 24 14:02:00 2009 TonyLawrence

Akismet just caught one. My consonant filter also caught it and it was tagged again for links to words ratio, so there wasn't much hope for that one to get through :-)

Fri Dec 25 02:58:24 2009 Michiel

Interesting, I'm going to follow these articles. It gives me some good ideas. In the meantime I have some stuff done, and reCAPTCHA is next. See http://www.michielovertoom.com/simpleblog/log.html for a short story on my proceedings.

Tue Dec 29 02:54:38 2009 Michiel

Yesterday I implemented reCAPTCHA in my prototype blog software. It was easier than expected: all you have to do is to sign up and receive some keys in the mail, then include a little bit of code in your script. Basically two functions do all the work, and they are provided in a ready-to-include script file.

I followed the suggestion done earlier to remember both the username and IP adress as a combination, and only require captcha validation once.

I also updated the webpage in which I describe the steps I took. See http://michielovertoom.com/simpleblog/log.html .

I'm looking forward to implement some more antispam ideas!


Fri Feb 5 14:00:49 2010 TonyLawrence

I tried Akismet for a month or so. I found that it seldom helped me - that is, if it said something was spam, my code usually already knew that was true and my code caught spam that it didn't know.

There were a few instances where it saw spam that I didn't. These were so few that I didn't feel it was worth paying the monthly fee (I have far too much traffic to qualify for the free version).

It's a good tool - if you can't write your own code, I would definitely recommend it.

Add your comments

---

Detecting Comment Spam, Part 1

2009/12/21

Suppose you were writing a commenting system for a website and you wanted to check user input against a list of words that might indicate spam. You'd want the list of suspicious words in a file and you'd run through that list. An easy way to do that in Perl is to use Perl's "grep" command.

We'll start with a program that won't work. This will show a side effect of grep you need to be aware of:

#!/usr/bin/perl # Build the "comment" while (<>) { push @TST, $_; } # Now test against our list open(SPAM,"spamlist"); while (<SPAM>) { chomp; if (grep /$_/, @TST ) { print "found $_\n"; } }

You need this code and a "spamlist" file. You'd put the words you want to match in that file. For the purposes of this article, I'll assume that "fribble" is NOT in the list. Therefore, if you run this little script and type "fribble" and press Enter and then Cntrl-D, you'd expect no response - "fribble" isn't in the list of spam words.

But that's not what happens. When you run the program, type "fribble", it seems like "fribble" (or any other input) matches every word in the list. That can't be right, can it?

The problem is that "grep" modifies $_ in your loop. That's simple to fix; we set a temporary variable:

#!/usr/bin/perl # Build the "comment" while (<>) { push @TST, $_; } # Now test against our list open(SPAM,"spamlist"); while (<SPAM>) { chomp; $testing=$_; if (grep /$testing/, @TST ) { print "found $testing\n"; } }

That's a bit better. Running it with "fribble" produces no output, but if you give it something in your list, it finds it. Great!

Not quite. Let's say you had "ambien" in your list because you want to stop common pharmaceutical spam. If you type "ambien" when running the program, yes, it finds it, but it will also find "ambient". That's not good - how do we fix that?

Well, we want "ambien" only when it's a word by itself. Your first thought might be to use a space or "\s":

if (grep / $testing /, @TST ) ... if (grep /\s$testing\s/, @TST )

But that fails if "ambien" is at the beginning of a line in @TST. It works if "ambien" is at the end because \s matches end of line as "space" in addition to real spaces, tabs and formfeeds.

OK, we could do this:

if (grep /\s$testing\s/, @TST or grep /\s$testing$/, @TST or igrep /^$testing\s/, @TST )

Fortunately, we don't need to. Perl has a better way:

if (grep /\b$testing\b/, @TST)

That "\b" is for "word boundary" and it does exactly what we want: it matches "ambien" wherever it is in a line by itself. Note that this same syntax works with command line grep, but "grep "\<word\>" files" only works with command line grep, not Perl.

So, finding "spam" words isn't too hard. The next question is what to do about them when you find them. A few thoughts come to mind:

• Refuse the comment flatly.
• Refuse the comment but tell the user what word(s) triggered the rejection.
• Remove the offending words from the comment
• Increment a counter for each spam indicator found; refuse if the count exceeds some value (this is how SpamAssassin works).
• Set the comment to require administrative approval before posting

None of these are ideal. In our next post, I'll dig into that a bit more deeply.

/Web/detecting-comment-spam.html copyright and reprint notice

Comments /Web/detecting-comment-spam.html

Mon Dec 21 21:18:25 2009 MikeHostetler

http://squarepegsystems.com

I've actually used Akismet in the past -- if you are a small user, it's free and it works well. It uses the same ideas that Tony is going to talk about, but it uses it for a much wider audience. More people to count means (ideally) better patter detection.

http://akismet.com/

Mon Dec 21 21:31:23 2009 TonyLawrence

I see a lot of people use Askimet. I still prefer to roll my own - for many reasons.

Mon Dec 21 21:44:45 2009 TonyLawrence

I do notice that Askimet has a Perl module on CPAN :

http://search.cpan.org/~nikolay/Net-Akismet/lib/Net/Akismet.pm

With that, you could "roll your own" while still taking advantage of Askimet.

I need to rewrite my comments code; I might just do that... no harm in having another opinion, after all.

Tue Dec 22 11:33:08 2009 Michiel

I'm experimenting with anti-spam measures too, and I think this is what I will use: When my blog detects that a contribution is made from an unknown IP address, it'll present a 'ReCAPTCHA' (See http://recaptcha.net/ ) for verification. I'm not sure scanning for spam words will work as well, since spammers always find new ways to hide their message from automatic scanners.

Tue Dec 22 11:45:02 2009 TonyLawrence

I'm not sure scanning for spam words will work as well, since spammers always find new ways to hide their message from automatic scanners.

I'll be talking about this more in the next part, but "words" aren't just things like pharmaceutical names. One thing spammers can't obfuscate is their link destination - the whole point is to post a link. The destination is part of the spam list.

Tue Dec 22 12:55:34 2009 Ralph

http://linuxcoaching.eu

In recent times there have been blog comments that only consist of legitimate words, so that the spam checker would not find any cause for rejection. But I regard these comments as spam too, because they are totally unrelated to the blog posting and their only purpose is to place a link to a suspect page. These comments tend to be overwhelmingly "positive" (and slimey) but they're nevertheless junk and clutter the blog with nonsensical praise.

I fear, we cannot fight spammers with software.

Tue Dec 22 13:08:19 2009 TonyLawrence

their only purpose is to place a link to a suspect page

Exactly. That's why the page(s) go into my spam list.

we cannot fight spammers with software

We can. But like any war, we'll lose a few battles. We can't stop EVERY spam post with software, but we can stop most of them.


Tue Dec 22 13:43:58 2009 MikeHostetler

http://squarepegsystems.com

Michial said, "When my blog detects that a contribution is made from an unknown IP address, it'll present a 'ReCAPTCHA' (See http://recaptcha.net/ ) for verification"

The problem with ReCAPTCHA is that it annoys legitimate users. But your use of only doing it for unknown IP addresses will help a little bit. But what if a legitimate user and a spammer are both behind the same firewall? Not to mention that sometimes figuring out what the Captcha is can be difficult (though ReCAPTCHA is better than most).

Tony said, "I need to rewrite my comments code; I might just do that... no harm in having another opinion, after all. "

I used Python's Askimet library to do some integration and it was very easy. You're right -- it's never bad to another opinion.

Tue Dec 22 14:23:41 2009 TonyLawrence

But what if a legitimate user and a spammer are both behind the same firewall?

Perhaps similar to what I do to decide if the comment can be published immediately or needs moderation. I key on IP plus username (anonymous is always moderated). A spammer could only guess at usernames that would match a legitimate users IP.

So Michiel could check both the IP and the username before throwing up a captcha - if the user has had that ip before, no captcha.



Tue Dec 22 14:40:43 2009 TonyLawrence

By the way, I want to wish all who are reading today a happy holiday.

We're pretty busy this week getting ready to have family in over the weekend, so I don't plan on Part 2 of this until next week. In that post, I want to discuss more about some of the ideas raised here in the comments: the ideals and realities of controlling spam comments, possible methods, pros and cons and so on.



Tue Dec 22 15:52:00 2009 anonymous

Subject:

Website: Tony replied:

their only purpose is to place a link to a suspect page
Exactly. That's why the page(s) go into my spam list.

But if you want to build such a list in these cases you have to examine the link "by hand" to find out if it is suspect or not, I cannot imagine any software to do that. I've decided to delete an unrelated comment based on the lack of information in it without looking at the link.

Tue Dec 22 15:52:19 2009 Ralph

linuxcoaching.eu

Tony replied:

their only purpose is to place a link to a suspect page
Exactly. That's why the page(s) go into my spam list.

But if you want to build such a list in these cases you have to examine the link "by hand" to find out if it is suspect or not, I cannot imagine any software to do that. I've decided to delete an unrelated comment based on the lack of information in it without looking at the link.

Tue Dec 22 16:09:25 2009 TonyLawrence

But if you want to build such a list in these cases you have to examine the link "by hand" to find out if it is suspect or not

Not entirely. I'll be talking more about that in the next post.

The war on spam is never a matter of using one tool. It's a combination of approaches that will give you good control.

Tue Dec 22 17:25:00 2009 BigDumbDInosaur

http://bcstechnology.net

If qualifying words in text against a list of verboten words (e.g., Viagra) is to be a primary anti-spam tool, I'd be inclined to keep the verboten word list sorted and use a binary search on it. A binary search for a single object in an ordered list executes in, worst-case, O(logN) iterations (where N is the number of words in the list), whereas grep's linear search will always require N iterations to determine that a suspect word is not verboten. This aspect of grep could represent a significant amount of processing time on a busy site, especially with long-winded posts.

Also, it might be useful to develop some sort of mechanism that could automate the adding of bad words to the verboten list. Phrases might be good as well, as oftentimes phrases are spam whereas individual words used in a phrase may be benign.

Tue Dec 22 21:55:35 2009 TonyLawrence

If you are searching a large list, certainly. Here, it's less than 200 lines and 2K. . But more importantly - we're not searching the list, we are searching the post for words in the list. You'd need to invert the search (take each word in the post and search in the list). I'm doubt this would be faster for the typical data sets, but it would be interesting to try some tests.

Add your comments

---

Unix and Linux startup scripts, Part 3

2009/12/14

This is a continuation of Unix and Linux startup scripts, Part 2

So, after being rudely interrupted by a server crash and a few days of website migration, we're ready to continue exploring Unix and Linux startup scripts. We looked at both System V and BSD methods; until fairly recently that would have been the end of the discussion: if you were running Unix/Linux, your system used one or the other of these. Not everyone was satisfied, though

You can see the hints of unhappiness in the script directives that crept in to BSD startup scripts. More fine grained control was needed and neither System V inittab nor the BSD rc scripts provided enough.

Why replace init?

One reason is boot speed. Even when scripts can be run in parallel as in SCO's prc_sync "P" scripts , running mostly serial shell scripts takes time; you and I and everyone else want our systems up NOW. But it's not just that.

if you step back and look at init objectively, it is just a program starter. That was certainly true with the original BSD /etc/rc and inittab only added a bit more complexity. Init is a "starter", but so are inetd /xinetd, at and cron. Why shouldn't init be able to take on some of that work?

There are "event driven" tasks. Today, there are many kinds of hardware we expect to be able to just plug into a running system and be recognized. Usually some program needs to be started up when that happens. Sometimes you want something special to happen when a new file appears in a certain directory, or when some other task finishes or fails - life can be complicated and init, (x)inetd and cron/at don't always match our needs.

Think of some of the things you might want to control for any given script or process:

• Persistency - should this process be restarted when it ends? Does it matter whether it ends normally or abnormally?
  
  
• Runlevel - should this run in single user mode? In the default runlevel?
  
  
• Privilege - what id should this run in?
  
  
• Dependencies - what other tasks must be running or not running?
  
  
• Security - should this be run in a chroot jail? Should it be sandboxed in other ways (no network access, for example)? Should only certain users, remote or local, be allowed to use this service? Start it? Stop it?
  
  
• Resource limits - how many concurrent instances of this process should run? How many times can it fork? How many clients can it have? How much disk space, ram, and so on?
  
  
• Timing - should this be running after working hours?
  
  
• Logging - did the task run and if not, why not?
  
  
• Grouping - there may be several tasks to which we would like to apply identical constraints or privileges
  
  

It's not that you can't have all of that; it's that some of it can be done by inittab, some by (x)inetd, some by cron/at, some by device drivers and some you have to do inside your script or program. It's a hodgepodge of stuff, and that's why there are attempts to replace it all with something more powerful and complete. The the man pages for inittab, cron, at, xinetd, udev and maybe a few others and glom them all together: THAT'S the ideal.

Implementing all that is something else entirely.

Aside from very real dependencies from programs that expect certain directories, certain processes and certain system calls, you also have inertia, developer resistance to seeing their projects replaced, user/administrator/programmer resistance to learning anything new and of course inevitable arguments about how, how much, when - given all that it's surprising that much progress has been made at all. But it has. Fedora and Ubuntu now use Upstart. Mac OS X has Launchd. The path to get there hasn't always been smooth - launchd was implemented in stages, first replacing cron, then xinetd and the rc scripts have only disappeared in Snow Leopard. Ubuntu 9.10 has replaced init with upstart but rc scripts still exist. xinetd is replaced, but not cron.

Be aware of this when reading Internet articles. For example, Mac OS X posts on launchd may not have been updated to reflect its current state on Snow Leopard or beyond. Even the Ubuntu Upstart FAQ has a section stating that Upstart probably won't replace xinetd - but it did.

You also need to be careful about the things you think you know. You might add something to /etc/rc.common on Snow Leopard, but it won't be run. On Ubuntu 9.10, scripts are still in /etc/init.d, but they get run by Upstart (see /etc/init for the scripts that start them). Fedora still has /etc/inittab, but it is only used to set the default run level - anything else added there will be ignored. There is plenty enough confusion to be had, especially for those of us who work on different systems and release levels.

References and further reading

• Getting Started with launchd
  
  
• Ubuntu Upstart FAQ
  
  
• How to (and why) supervise forking processes
  
  
• Replacing init with Upstart
  
  
• runit - a UNIX init scheme with service supervision
  
  
• ReplacementInit
  
  
• initng- another init replacement designed to speed up boot by starting processes asynchronously.
  
  
• Service Management Utility- an init augmenter from Solaris 10
  
  

/Basics/unix-startup-scripts-3.html copyright and reprint notice

Comments /Basics/unix-startup-scripts-3.html

Add your comments

---