Skip to content

Category: Uncategorized

DnsblAccuracy082005 – Spamassassin Wiki

Do you use anti-spam DNS blocklists? If so, you should probably go take a look at DnsblAccuracy082005 on the SpamAssassin wiki; I’ve collated the results from our recent mass-check rescoring runs for 3.1.0, to produce have up-to-date measurements of the accuracy and hit-rates for most of the big DNS blocklists.

A few highlights:

We don’t have accurate figures for the new URIBL.COM lists, btw — only the rulesets that are distributed with SpamAssassin were measured.

Bogus Challenge-Response Bounces: I’ve Had Enough

I get quite a lot of spam. For one random day last month (Aug 21st), I got 48 low-scoring spam mails (between 5 and 10 points according to SpamAssassin), and 955 high-scorers (anything over 10). I don’t know how much malware I get, since my virus filter blocks them outright, instead of delivering to a folder.

That’s all well and good, because spam and viruses are now relatively easy to filter — and if I recall correctly, they were all correctly filed, no FPs or FNs (well, I’m not sure about the malware, but fingers crossed ;).

The hard part is now ‘bogus bounces’ — the bounces from ‘good’ mail systems, responding to the forged use of my addresses as the sender of malware/spam mails. There were 306 of those, that day.

Bogus bounces are hard to filter as spam, because they’re not spam — they’re ‘bad’ traffic originating from ‘good’, but misguided, email systems. They’re not malware, either. They’re a whole new category of abusive mail traffic.

I say ‘misguided’, because a well-designed mail system shouldn’t produce these. By only performing bounce rejection with a 4xx or 5xx response as part of the SMTP transaction, when the TCP/IP connection is open between the originator and the receiving MX MTA, you avoid most of the danger of ‘spamming’ a forged sender address. However, many mail systems were designed before spammers and malware writers started forging on a massive scale, and therefore haven’t fixed this yet.

I’ve been filtering these for a while using this SpamAssassin ruleset; it works reasonably well at filtering bounces in general, catching almost all of the bounces. (There is a downside, though, which is that it catches more than just bogus bounces — it also catches real bounces, those in response to mails I sent. At this stage, though, I consider that to be functionality I’m willing to lose.)

The big remaining problem is challenge-response messages.

C-R is initially attractive. If you install it, your spam load will dwindle to zero (or virtually zero) immediately — it’ll appear to be working great. What you won’t see, however, is what’s happening behind the scenes:

  • your legitimate correspondents are getting challenges, will become annoyed (or confused), and may be unwilling or unable to get themselves whitelisted;

  • spam that fakes other, innocent third party addresses as the sender, will be causing C-R challenges to be sent to innocent, uninvolved parties.

The latter is the killer. In effect, you’re creating spam, as part of your attempts to reduce your own spam load. C-R shifts the cost of spam-filtering from the recipient and their systems, to pretty much everyone else, and generates spam in the process. I’m not alone in this opinion.

That’s all just background — just establishing that we already know that C-R is abusive. But now, it’s time for the next step for me — I’ve had enough.

I initially didn’t mind the bogus-bounce C-R challenges too much, but the levels have increased. Each day, I’m now getting a good 10 or so C-R challenges in response to mails I didn’t send. Worse, these are the ones that get past the SpamAssassin ruleset I’ve written to block them, since they don’t include an easy-to-filter signature signifying that they’re C-R messages, such as Earthlink’s ‘spamblocker-challenge’ SMTP sender address or UOL‘s ‘AntiSpam UOL’ From address. There seems to be hundreds of half-assed homegrown C-R filters out there!

So now, when I get challenge-response messages in response to spam which forges one of my addresses as the ‘From’ address, and it doesn’t get blocked by the ruleset, I’m going to jump through their hoops so the spam is delivered to the C-R-protected recipient. Consider it a form of protest; creating spam, in order to keep youself spam-free, is simply not acceptable, and I’ve had enough.

And if you’re using one of these C-R filters — get a real spam filter. Sure they cost a bit of CPU time — but they work, without pestering innocent third parties in the process.

Beardy Justin

Yes, I’ve been growing a beard. Strangely, it seems to be going quite well! Here’s a good pic of beardy Justin, standing on a bridge over the Merced river in Yosemite:

Lots more pics from the holiday should be appearing here shortly, if you’re curious.

Mosquitos, Snakes and a Bear

Well, I’m back… it appears that Google Maps link I posted wasn’t too much use in deciphering where I was going; sorry about that. Myself and C spent a fun week and a bit, driving up to Kings Canyon and Yosemite, backpacking around for a few days, then driving back down via the 395 via Bishop, Mammoth Lakes, Lone Pine and so on.

Kings Canyon: Unfortunately, not so much fun; we had the bad luck of encountering what must be the tail end of the mosquito season, and spent most of our 2 days there running up and down the Woods Creek trail without a break, entirely surrounded by clouds of mozzies. Possibly this headlong dashing explains how we ran into so much other wildlife — including a (harmless) California Mountain King Snake and, less enjoyably — and despite wearing bear bells on our packs to avoid this — a black bear…

We rounded a corner on the trail, and there it was, munching on elderberries. Once we all spotted each other, there were some audible sounds of surprise from both bear and humans, and the bear ran off in the opposite direction; the humans, however, did not. We were about 500 feet from our camp for the night, so we needed to get past where the bear had been, or face a long walk back.

Despite some fear (hey, this was our first bear encounter!), we stuck around, shouted, waved things, and took the various actions you take. It all went smoothly, the bear had probably long since departed, but we took it slow regardless, and had a very jittery night in our tent afterwards. After that, and the unceasing mozzie onslaught, we were in little hurry to carry on around the planned loop, so we cut short our Kings Canyon trip by a day and just returned down the trail to its base.

Yosemite: a much more successful trip. There were many reasons, primarily that the mosquito population was much, much lower, and discovering that the Tuolumne Meadows Lodge — comfortable tent cabins, excellent food, and fantastic company — provided a truly excellent base camp.

But I’d have to say that the incredible beauty of Tuolumne Meadows and the Vogelsang Pass really blew me away. I don’t think I’ve seen any landscape quite like that, since trekking to Annapurna Base Camp in Nepal. I’m with John Muir — Yosemite and its surrounds are a wonder of the world.

Lee Vining: had to pick up a sarnie at the world-famous Whoa Nellie Deli. Yum! After all the camping, we stayed in a hotel with TV, got some washing done, and watched scenes from a J.G. Ballard novel play out on NBC and CNN. Mind-boggling.

Mammoth Lakes: A quick kvetch. Mammoth is possibly the most pedestrian-hostile town I’ve ever visited. They have a hilarious section of 100 feet of sidewalk, where I encountered a fellow pedestrian using those ski-pole-style hiking walking sticks, and entirely in seriousness. Was the concept of walking so foreign in that town that long-distance walking accessories were required? I don’t know, but it didn’t make up for the other 90% of the streets where peds were shoved off onto the shoulder, in full-on ‘sidewalk users aren’t welcome here’ Orange County style.

On top of that, the single pedestrian crossing in the main street spans five lanes of traffic, with no lighting, warning signs, or indeed any effective way for drivers to know whether peds were crossing or not. Unsurprisingly we nearly got run over when we tried using the damn thing. Best avoided.

I’m amazed — it’s like they designed the town to be ped-hostile. Surely allowing peds to get around your town is a bonus when you’re a ski resort for half of the year? Meh.

Anyway, back again, a little refreshed. Once more into the fray…

Faster string search alternative to Boyer-Moore: BloomAV

An interesting technique, from the ClamAV development list — using Bloom filters to speed up string searching. This kind of thing works well when you’ve got 1 input stream, and a multitude of simple patterns that you want to match against the stream. Bloom filters are a hashing-based technique to perform extremely fast and memory-efficient, but false-positive-prone, binary lookups.

The mailing list posting (‘Faster string search alternative to Boyer-Moore‘) gives some benchmarks from the developers’ testing, along with the core (GPL-licensed) code:

Regular signatures (28,326) :

  • Extended Boyer-Moore: 11 MB/s

  • BloomAV 1-byte: 89 MB/s

  • BloomAV 4-bytes: 122 MB/s

Some implementation details:

the (implementation) we chose is a simple bit array of (256 K * 8) bits. The filter is at first initialized to all zeros. Then, for every virus signature we load, we take the first 7 bytes, and hash it with four very fast hash functions. The corresponding four bits in the bloom filter are set to 1s.

Our intuition is that if the filter is small enough to fit in the CPU cache, we should be able to avoid memory accesses that cost around 200 CPU cycles each.

Also, in followup discussion, the following paper was mentioned: A paper describing hardware-level Bloom filters in the Snort IDS — S. Dharmapurikar, P. Krishnamurthy, T. Sproull, and J. W. Lockwood, "Deep packet inspection using parallel Bloom filters," in Hot Interconnects, (Stanford, CA), pp. 44–51, Aug. 2003.

This system is dubbed ‘BloomAV’. Pretty cool. It’s unclear if the ClamAV developers were keen to incorporate it, though, but it does point at interesting new techniques for spam signatures.

Tech Camp Ireland

Irish techies, mark your calendars! Various Irish bloggers are proposing a Tech Camp Ireland geek get-together, similar to Bar Camp in approach, for Saturday October 15th.

Ed Byrne and James Corbett are both blogging up a storm already. I’d go, but it’d be a hell of a trip ;)

I would say it needs a little less blog, a little more code, and a little more open source, but it does look very exciting, and it’s great to see the Bar Camp spirit hitting Ireland.

More on ‘Bluetooth As a Laptop Sensor’

Bluetooth As a Laptop Sensor in Cambridge, England.

I link-blogged this yesterday, where it got picked up by Waxy, and thence to Boing Boing — where some readers are reportedly considering it doubtful. Craig also expressed some skepticism. However, I think it’s for real.

Check out the comments section of Schneier’s post — there’s a few notable points:

  • Some Bluetooth-equipped laptops will indeed wake from suspend to respond to BT signals.

  • Davi Ottenheimer reports that the current Bluetooth spec offers "always-on discoverability" as a feature. (Obviously the protocol designers let usability triumph over security on that count.)

  • Many cellphones are equipped with Bluetooth, and can therefore be used to detect other ‘discoverable’ BT devices in range.

  • Walking around a UK hotel car park, while pressing buttons on a mobile phone, would be likely to appear innocuous — I know I’ve done it myself on several occasions. ;)

Finally — this isn’t the first time the problem has been noted. The same problem was reported at Disney World, in the US:

Here’s the interesting part: every break-in in the past month (in the Disney parking lots) had involved a laptop with internal bluetooth. Apparently if you just suspend the laptop the bluetooth device will still acknowledge certain requests, allowing the thief to target only cars containing these laptops.

Mind you, perhaps this is a ‘chinese whispers’ case of the Disney World thefts being amplified. Perhaps it was noted as happening in Disney World, reported in an ’emerging threats’ forum where the Cambridgeshire cop heard it, and he then picked it up as something worth warning the public about, without knowing for sure that it was happening locally.

Update: aha. An observant commenter on Bruce Schneier’s post has hit on a possibly good reason why laptops implement wake-on-Bluetooth:

On my PowerBook, the default Bluetooth settings were "Discoverable" and "Wake-on-Bluetooth" — the latter so that a Bluetooth keyboard or mouse can wake the computer up after it has gone to sleep.

Emergent Chaos: I’m a Spamateur

Emergent Chaos: I’m a Spamateur:

In private email to Justin “SpamAssassin” Mason, I commented about blog spam and “how to fix it,” then realized that my comments were really dumb. In realizing my stupidity, I termed the word “spamateur,” which is henceforth defined as someone inexperienced enough to think that any simple solution has a hope of fixing the problem.

I think this is my new favourite spam neologism ;)

How convenient does the ‘right thing’ have to be?

Environment: Kung Fu Monkey: Hybrids and Hypotheses. A great discussion of the Toyota Prius:

Kevin Drum recently quoted a study which re-iterated that there’s no "real" advantage to buying a hybrid. It’s only just as convenient — so if you’re driving a hybrid, you’re doing it for some other reason than financial incentive.

That made me think: what a perfect example of just how fucking useless as a society we’ve become. We can’t even bring ourselves to do the right thing when it’s only JUST as convenient as doing the wrong thing. And that’s not even considered odd. Even sadder.

Box Office Patents

Forbes: Box Office Patents.

It’s the kind of plot twist that will send some critics screaming into the aisles: Why not let writers patent their screenplay ideas? The U.S. Patent and Trademark Office already approves patents for software, business methods — remember Amazon.com’s patent on ‘one-click’ Internet orders? — even role-playing games. So why not let writers patent the intricate plot of the next cyberthriller?

So in other words, a law grad called Andrew Knight actually wants to see the world RMS described in his ‘Patent Absurdity’ article for the Guardian, where Les Miserables was unpublishable due to patent infringement. Incredible.

He himself plays the classic lines, familiar to those who followed the EU software patenting debate:

Knight agrees, up to a point. He won’t reveal the exact details of the plots he’s submitted to the Patent Office, other than to say they involve cyberspace. And he says patents would apply only to ideas that are unique and complex. But he worries that without patent protection, some Hollywood sharpies could change ideas like his around and pass them off as their own.

”I’m trying to address a person who comes up with a brand-new form of entertainment who may not be a Poe, may not be a Shakespeare, but still deserves to be paid for his work,” Knight says. ”Otherwise, who will create anything?”

A perfect pro-patent hat trick!

Running on WordPress!

I’ve decided to try out the real deal — a ‘proper’ weblogging platform, namely WordPress. Be sure to comment if you spot problems…

Grumpiness and Cigarettes

Meta: My apologies if you wound up running into me online at some stage this week — I’ve been in a lousy mood.

Where the ‘cursor’ came from

Stuff: So C is a massive antiques nut, and got tickets for the Antiques Roadshow next month in LA. As a result, we’ve been shopping around for interesting stuff for her to bring along.

The Life of a SpamAssassin Rule

Spam: during a recent discussion on the SpamAssassin dev list, the question came up as to how long a rule could expect to maintain its effectiveness once it was public — the rule secrecy issue.

‘Irish EFF’

Ireland: There’s been some discussion about ‘an Irish EFF’ recently, reminding me of the old days of Electronic Frontier Ireland in the 1990s.

Happy Birthday to the RISKS Forum!

Tech: One of the first online periodicals I started reading regularly, when I first got access to USENET back in 1989 or so, was comp.risks — Peter G. Neumann’s RISKS Forum. Since then, I’ve been reading it religiously, in various formats over the years.

Patents and Laches

Patents: This has come up twice recently in discussions of software patenting, so it’s worth posting a blog entry as a note.

looking at the new DKIM draft

The combined DKIM standard, mixing Yahoo!’s DomainKeys and Cisco’s IIM, has been submitted to the IETF as a candidate spec by the MASS ‘pre-working group effort’. I like the idea behind both (a few years back, I, a few other SpamAssassin developers, and several others came up with the roots of a message-signature anti-forgery scheme we called ‘porkhash’, but never really went anywhere with it), so I’m glad to see this one progressing nicely.

McCreevy seeing anti-globalisation protesters everywhere

Patents: I’m just back from a fantastic holiday weekend, totally offline, hiking through Catalina Island. I’m a little bit sunburnt, my nose is peeling, but it was great fun. I got a fantastic picture of the sun setting over hundreds of boats bobbing at their moorings in Two Harbors, which I must upload at some stage.

The meaning of the term ‘technical’ in software patenting

Patents: One of the key arguments in favour of the new EU software patenting directive as it’s currently worded, from the ‘pro’ side, is that it doesn’t ‘allow software patents as such’, since it requires a ‘technical’ inventive step for a patent to be considered valid.

Latest Script Hack: utf8lint

Perl: double-encoding is a frequent problem when dealing with UTF-8 text, where a UTF-8 string is treated as (typically) ISO Latin-1, and is re-encoded.

Amazing article series on Climate Change

Science: in April and May, the New Yorker printed an amazing series of articles on climate change by Elizabeth Kolbert, full of outstanding research and interviews with the key players.