Skip to content

Author: Justin

Justin Mason, the author of this weblog.

IIA’s nasty infection

Published April 9, 2008

The Irish Internet Association have a weblog at blog.iia.ie. Back on January 30, this had a Technorati rank of 587893, with 21 inbound links from 14 blogs. That’s about what you’d expect — comparable with Chris Horn’s blog, for instance.

However, fast forward to today, and in the intervening 3 months, it seems to have suddenly shot up to 23,322 inbound links from 550 blogs, giving it a Technorati rank of 6,870.

To put that in perspective, that puts it comfortably in the top 3 in the Irish Blogs Technorati Top 100 — beating Damien Mulley‘s 7,859, but just short of Donncha O’Caoimh‘s stellar 3,434 — and ahead of these other gods of the Irish blogosphere:

Pretty impressive ;)

I was curious, so I went investigating. Of those thousands of inbound links, here’s some samples of the most recent, pasted from the Technorati inbound links page:

barkingmoose

Atacand Free instant online credit report Application credit card Cheap Paxil Does your credit score Household bank credit card application Apr for credit cards Buy Cephalexin? Aciphex Cheap Feldene Zovirax Risperdal Buy Naprosyn, Propecia Credit score codes Poor credit score, Propecia Uk Canada credit card online application Motrin Business credit score Cheap Cialis Jelly 50 Cent Free Ringtones Celexa How to improve my credit score Buy Inderal

4 days ago in barkingmoose by barkingmoose · Authority: 3

The Peninsula’s Edge

Jc penny credit card application Credit cards 1.99 apr ny Affect credit score For credit score American express credit card application Freee credit report Instant fleet 0 apr credit card application? Hydrocodone For low credit scores, No credit instant approval credit cards Annual creditreport.com, Tramadol Credit reporting service Configuration VPN Cheap credit reports. Buy Premarin Carisoprodol Soma Propecia Generic

6 days ago in The Peninsula’s Edge by ricsmith510 · Authority: 9

The Incredible Blog

Prepaid credit card uk Phentrimine Cheap Zovirax: Calan Highest credit score Ambien Valtrex: Ultram 3 credit reporting agencies Credit cards online application Instant approval student credit cards, Apr balance transfer credit cards Free government credit report Transunion free credit report Credit card debt bankruptcy? Propecia Propecia Uk! Correcting credit reports Cialis Uk Credit rating report Buy Synthroid Instant capital one 0 interest credit card application

7 days ago in The Incredible Blog · Authority: 1

Quilters’ Blogs

Annualcreditreport Instantly instant free online credit report Credit cards instant approval Guaranteed instant approval credit cards Lexapro Get my credit score, Card consolidation credit debt financial internet Chevron credit card services. Risperdal Lower credit card debt VPN connection One credit card application Xanax Viagra! Vasotec Diazepam Fix my credit report Credit report bureau. Cialis Soft Tabs! Ativan? Secured loans to increase credit score Cheap Amaryl Cheap Prednisone Alprazolam! Cheap 7 days ago in Quilters’ Blogs · Authority: 5

TPN :: Martial Arts Explorer

Luvox Credit score of Plavix 50 Cent Free Ringtones, Cheap Elavil? Free consumer credit report: Famvir Improve credit score fast Phentermine Online Zovirax Cialis Soft Tabs Apr for credit cards! Ultram Zoloft Credit card deal 0 Deltasone! VPN: Cheap Cardura Credit score rankings! Annual credit report .com Interest rate credit score: Carisoprodol Flagyl ER Online Cialis Soft Tabs Enable VPN 0 apr credit card application Free business credit report Ambien Low 7 days ago in TPN :: Martial Arts Explorer · Authority: 56

Take a look at the ‘inbound links’ list — thousands more just like that.

All of the affected blogs have been hacked to deliver these spam links. They run unpatched versions of WordPress vulnerable to a major security hole. On a casual visit, their pages seem fine — but “View Source”, scroll to the bottom, and there are thousands of spam links for drugs, ringtones, cheap credit, etc. on each one, exactly as above, and as described by Kevin Burton in his description of the current epidemic of blog spam.

How did links to the IIA’s blog wind up in this collection?

It’s worth noting that the IIA’s blog does not display the same symptoms — the links aren’t present on their pages.

However, this post provided a good tip as to what has happened. Those infected blog pages point, in turn, to other infected blogs. Somewhere within the IIA’s blog setup, there’s a page inserted by a bad guy, collecting thousands of illicit links from thousands of other infected sites — and sure enough, Irish Web Watcher found it on the IIA’s site — here it is.

Looks like the IIA have a pretty major disinfection job on their hands, and urgently — there’s already a lot of spammy results appearing in the Google index from that site, and the next step after that is usually removal from the index once Google notice it.

Google now include Code Search in normal results

Published April 7, 2008

Latest Google curiosity… I hadn’t spotted this before: it appears Google is now including ‘Code Snippet’ results in the results for its normal search. For example, a search for XSLoader gives this result:

xsloader

The results highlighted on the page are for a local variable in a Java module, rather than the much more common XSLoader perl module. I guess ‘Code Snippet’ search is case-sensitive.

RAII in perl

Published April 2, 2008

Suppose you have matching start() and end() functions. You want to ensure that each start() is always matched with its corresponding end(), without having to explicitly pepper your code with calls to that function. Here’s a good way to do it in perl — create a guard object:

package Scoper;
sub new {
  my $class = shift; bless({ func => shift },$class);
}
sub DESTROY {
  my $self = shift; $self->{func}->();
}

Here’s an example of its use:

{
  start();
  my $s = Scoper->new(sub { end(); });
  [... do something...]
}
[at this point, end() has been called, even if a die() occurred]

The idea is simply to use DESTROY to perform whatever the cleanup operation is. Once the $s object goes out of scope, it’ll be deleted by perl’s GC, in the process of which, calling $s->DESTROY(). In other words, it’s using the GC for its own ends.

Unlike an eval { } block to catch die()s, this will even be called if exit() or POSIX::exit() is called. (POSIX::_exit(), however, skips DESTROY.)

This is a pretty old C++ pattern — Resource Acquisition Is Initialization. C++’s auto_ptr template class is the best-known example in that language. Here’s a perl.com article on its use in perl, from last year, mostly regarding the CPAN module Object::Destroyer. To be honest, though, it’s 6 lines of code — not sure if that warrants a CPAN module! ;)

RAII is used in SpamAssassin, in the Mail::SpamAssassin::Util::ScopedTimer class.

“What’s New” archaeology

Published March 31, 2008

jwz has, incredibly, resurrected home.mcom.com, the WWW site of the Mosaic Communications Corporation, as it was circa Oct 1994.

Edmund Roche-Kelly was kind enough to get in touch and note this link — http://home.mcom.com/home/whatsnew/whats_new_0993.html:

September 3, 1993

IONA Technologies (whose product, Orbix, is the first full and complete implementation of the Object Management Group’s Common Object Request Broker Architecture, or CORBA) is now running a Web server.

An online pamphlet on the Church of the SubGenius is now available.

Guess who was responsible for those two ;)

I was, indeed, running the IONA web server — it was set up in June 1993, and ran Plexus, a HTTP server written in Perl. IONA’s server was somewhere around public web server number 70, world-wide.

The SubGenius pamphlet is still intact, btw, although at a more modern, “hyplan”-less URL these days. It’ll be 15 years old in 6 months… how time flies!

Sharing, not consuming, news

Published March 28, 2008

The New York Times yesterday had a great article about modern news consumption:

According to interviews and recent surveys, younger voters tend to be not just consumers of news and current events but conduits as well — sending out e-mailed links and videos to friends and their social networks. And in turn, they rely on friends and online connections for news to come to them. In essence, they are replacing the professional filter — reading The Washington Post, clicking on CNN.com — with a social one.

“There are lots of times where I’ll read an interesting story online and send the URL to 10 friends,” said Lauren Wolfe, 25, the president of College Democrats of America. “I’d rather read an e-mail from a friend with an attached story than search through a newspaper to find the story.”

[Jane Buckingham, the founder of the Intelligence Group, a market research company] recalled conducting a focus group where one of her subjects, a college student, said, “If the news is that important, it will find me.”

In other words, as Techdirt put it, this generation of news readers now focuses on sharing the news, rather than just consuming it — and if you want to share a news story, there’s no point passing on a subscription-only URL that your friends and contacts cannot read.

What newspapers need to do to remain relevant for this generation of news consumers is not to hide their content behind paywalls and registration-required screens. The Guardian got their heads around this a few years back, and have come along in leaps and bounds since then. I wonder if the Irish Times is listening?

converting TAP output to JUnit-style XML

Published March 26, 2008

Here’s a perl script that may prove useful: tap-to-junit-xml…

NAME

tap-to-junit-xml – convert perl-style TAP test output to JUnit-style XML

SYNOPSIS

tap-to-junit-xml "test suite name" [ outputprefix ] < tap_output.log

DESCRIPTION

Parse test suite output in TAP (Test Anything Protocol) format, and produce XML output in a similar format to that produced by the <junit> ant task. This is useful for consumption by continuous-integration systems like Hudson.

Written in perl, requires TAP::Parser and XML::Generator. It's based on junit_xml.pl by Matisse Enzer, although pretty much entirely rewritten.

Pulseaudio ate my wifi

Published March 21, 2008

I’ve just spent a rather frustrating morning attempting to debug major performance problems with my home wireless network; one of my machines couldn’t associate with the AP at all anymore, and the laptop (which was upstairs in the home office, for a change) was getting horrific, sub-dialup speeds.

I did lots of moving of Linksys APs and tweaking of “txpower” settings, without much in the way of results. Cue tearing hair out etc.

Eventually, I logged into the OpenWRT AP over SSH, ran iftop to see what clients were using the wifi, and saw that right at the top, chewing up all the available bandwidth, was a multicast group called 224.0.0.56. The culprit! There was nothing wrong with the wifi setup after all — the problem was massive bandwidth consumption, crowding out all other traffic.

You see, “pulseaudio”, the new Linux sound server, has a very nifty feature — streaming of music to any number of listeners, over RTP. This is great. What’s not so great is that this seems to have magically turned itself on, and was broadcasting UDP traffic over multicast on my wifi network, which didn’t have enough bandwidth to host it.

Here’s how to turn this off without killing “pulseaudio”. Start “paman”, the PulseAudio Manager, and open the “Devices” tab:

(click on the image to view separately, if it’s partly obscured.)

Select the “RTP Monitor Stream” in the “Sources” list, and open “Properties”:

Hit the “Kill” button, and your network is back to normal. Phew.

Another (quicker) way to do this, is using the command-line “pacmd” tool:

echo kill-source-output 0 | pacmd

It’s a mystery where this is coming from, btw. Here’s what “paman” says it came from:

But I don’t seem to have an active ‘module-rtp-send’ line in my configuration:

: jm 98...; grep module-rtp-send /etc/pulse/* /home/jm/.pulse*
/etc/pulse/default.pa:#load-module module-rtp-send source=rtp.monitor

Curious. And irritating.

Update: it turns out there’s another source of configuration — GConf. “paprefs” can be used to examine that, and that’s where the setting had been set, undoubtedly by me hacking about at some stage. :(

more crap from St. Petersburg

Published March 19, 2008

Noted with alarm in this comment regarding the horrific privacy-invading adware that is Phorm:

Their programmers are mostly Saint Petersburg-based, home to the Russian Business Network. Their servers are kept only in Saint Petersburg and China, so no ISP customer data is ever stored in the UK. Any personally identifying information they obtain about UK citizens can never be seen or purged using existing UK Data Protection Laws.

St. Petersburg is turning out to be quite a source of online nastiness — the new Boca Raton.

Evading Audible Magic’s Copysense filtering

Published March 12, 2008

As I noted on Monday, the Irish branches of several major record companies have brought a case against Eircom, demanding in part that the ISP install Audible Magic’s Copysense anti-filesharing appliances on their network infrastructure.

I thought I’d do a quick bit of research online into how they do their filtering. Here’s what the EFF had to say:

Audible Magic’s technology can easily be defeated by using one-time session key encryption (e.g., SSL) or by modifying the behavior of the network stack to ignore RST packets.

It’s interesting to see that they used RST packets — this is the same mechanism used by the “Great Firewall of China” to censor the internet:

the keyword detection is not actually being done in large routers on the borders of the Chinese networks, but in nearby subsidiary machines. When these machines detect the keyword, they do not actually prevent the packet containing the keyword from passing through the main router (this would be horribly complicated to achieve and still allow the router to run at the necessary speed). Instead, these subsiduary machines generate a series of TCP reset packets, which are sent to each end of the connection. When the resets arrive, the end-points assume they are genuine requests from the other end to close the connection — and obey. Hence the censorship occurs.

But there’s a very easy way to avoid this, according to that blog post:

However, because the original packets are passed through the firewall unscathed, if both of the endpoints were to completely ignore the firewall’s reset packets, then the connection will proceed unhindered! We’ve done some real experiments on this — and it works just fine!! Think of it as the Harry Potter approach to the Great Firewall — just shut your eyes and walk onto Platform 9¾.

Clayton, Murdoch, and Watson’s paper on this technique provides the Linux and FreeBSD firewall commands they used to do this. Here’s Linux:

   iptables -A INPUT -p tcp --tcp-flags RST RST -j DROP

For FreeBSD, the command is:

   ipfw add 1000 drop tcp from any to me tcpflags rst in

So assuming Copysense haven’t changed their approach yet, it’s trivial to block Copysense’s filtering, if both ends are running Linux or BSD. I predict if Copysense becomes widespread, someone will patch Windows TCP to do the same.

I love Audible Magic’s response:

The current appliance happens to use the TCP Reset to accomplish this today. There are many other technical methods of blocking transfers. Again, we have strategies to deal with them should they ever prove necessary. This is why we recommend our customers purchase a software support agreement which provides for these enhancements that keep their purchase up-to-date and protect their investment.

in other words, “hey customers! if you don’t have a support contract, you’re shit out of luck when the p2p guys get around our filters!” Nice. ;)

Vim hanging while running VMWare: fixed

Published March 12, 2008

I’ve just fixed a bug on my linux desktop which had been annoying me for a while. Since there seems to be little online written about it, here’s a blog post to help future Googlers.

Here’s the symptoms: while you’re running VMWare, your Vim editing sessions freeze up for 20 seconds or so, roughly every 5 minutes. The editor is entirely hung.

If you strace -p the process ID before the hang occurs, you’ll see something like this:

select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
_llseek(7, 4096, [4096], SEEK_SET)      = 0
write(7, "tp\21\0\377\0\0\0\2\0\0\0|\0\0\0\1\0\0\0\1\0\0\0\6\0\0"..., 4096) = 4096
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
_llseek(7, 20480, [20480], SEEK_SET)    = 0
write(7, "ad\0\0\245\4\0\0\341\5\0\0\0\20\0\0J\0\0\0\250\17\0\0\247"..., 4096) = 4096
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
fsync(

In other words, the hung process is sitting in an fsync() call, attempting to flush changed data for the current file to disk.

Investigation threw up the following: a kerneltrap thread about disk activity, poor responsiveness with Firefox 3.0b3 on linux, and a VIM bug report regarding this feature interfering with laptop-mode and spun-down hard disks.

VMWare must be issuing lots of unsynced I/O, so when Vim issues its fsync() or sync() call, it needs to wait for the VMWare I/O to complete before it can return — even though the machine is otherwise idle. A bit of a Linux kernel (or specifically, ext3) misfeature, it seems.

Synthesising details from those threads comes up with this fix: edit your ~/.vimrc and add the following lines —

set swapsync=
set nofsync

This will inhibit use of both fsync() and sync() by Vim, and the problem is avoided nicely.

Update: one of the Firefox developers discusses how this affects FF 3.0.

Irish ISPs in record company crosshairs

Published March 11, 2008

RTE reports that 4 record companies, EMI, Sony BMG, Universal Music and Warner Music, have brought a High Court action to compel Eircom — Ireland’s largest ISP — to prevent its networks being used for the illegal downloading of music:

Willie Kavanagh, Managing Director of EMI Ireland and chairman of IRMA, said because of illegal downloading and other factors, the Irish music industry was experiencing a “dramatic and accelerating decline” in income. He said sales in the Irish market dropped 30% in the six years up to 2007.

EMI and the other companies are challenging Eircom’s refusal to use filtering technology or other measures to voluntarily block or filter illegally downloaded material. Last October Eircom told the companies it was not in a position to use the filtering software.

(I wonder if those dropping sales in the Irish market comprise only CDs sold by Irish shops? 2001 to 2007 is also the time period when physical sales have given way to online shopping on a gigantic scale, especially for music.)

The Irish Times coverage includes another interesting factoid, which appears in a lot of press regarding this case:

Latest figures available, for 2006, indicate that 20 billion music files were illegally downloaded worldwide that year. The music industry estimates that for every single legal download, there are 20 illegal ones.

A little research reveals that that figure comes from the IFPI Digital Music Report 2008. I’d have a totally different take on it, however. In my opinion, the figure is probably correct, but not for the reasons the IFPI want them to be. There are a number of factors:

There’s more commentary on the 20-to-1 figure here.

The IFPI Digital Music Report 2008 also notes:

“2007 was the year ISP responsibility started to become an accepted principle. 2008 must be the year it becomes reality”

Governments are starting to accept that Internet Service Providers (ISPs) should take a far bigger role in protecting music on the internet, but urgent action is needed to translate this into reality, a new report from the international music industry says today.

ISP cooperation, via systematic disconnection of infringers and the use of filtering technologies, is the most effective way copyright theft can be controlled. Independent estimates say up to 80 per cent of ISP traffic comprises distribution of copyright-infringing files.

The IFPI Digital Music Report 2008 points to French President Sarkozy’s November 2007 plan for ISP cooperation in fighting piracy as a groundbreaking example internationally. Momentum is also gathering in the UK, Sweden and Belgium. The report calls for legislative action by the European Union and other governments where existing discussions between the music industry and record companies fail to progress.

So it seems Ireland is the vanguard of an international effort by IFPI members to force ISPs to install filtering, worldwide. It seems the same happened in Belgium last year — and I reckon there’ll be similar cases elsewhere soon.

Either way, I doubt this will be good for Irish internet users.

(PS: while I’m talking about buying MP3s online — a quick plug for 7digital. Last time I used them, I had a pretty crappy experience, but the situation is a lot better nowadays. They now have a great website that works perfectly in Firefox on Linux; they sell brand new releases like the Hercules and Love Affair album as 320kbps DRM-free MP3s; they support PayPal payments; and downloads are fast and simple — right click, “Save As”. hooray!)

Some other blog coverage: Lex Ferenda with some details about the legal situation, and Jim Carroll.

Update: EMI Ireland seem to be singing from a different hymn-sheet than their head office… interesting.

Update 2: I’ve taken a look at the Copysense filtering technology, and how it can be evaded.

Announcing IrishPulse

Published March 6, 2008

As I previously threatened, I’ve gone ahead and created a “Microplanet” for Irish twitterers, similar to Portland’s Pulse of PDX — an aggregator of the “stream of consciousness” that comes out of our local Twitter community: IrishPulse.

Here’s what you can do:

Add yourself: if you’re an Irish Twitter user, follow the user ‘irishpulse’. This will add you to the sources list.

Publicise it: feel free to pass on the URL to other Irish Twitter users, and blog about it.

Read it: bookmark and take a look now and again!

In terms of implementation, it’s just a (slightly patched) copy of Venus and a perl script using Net::Twitter to generate an OPML file of the Twitter followers. Here’s the source. I’d love to see more “Pulse” sites using this…

Google’s CAPTCHA – not entirely broken after all?

Published March 5, 2008

A couple of weeks ago, WebSense posted this article with details of a spammer’s attack on Google’s CAPTCHA puzzle, using web services running on two centralized servers:

[…] It is observed that two separate hosts active on same domain are contacted during the entire process. These two hosts work collaboratively during the CAPTCHA break process. […]

Why [use 2 hosts]? Because of variations included in the Google CAPTCHA image, chances are that host 1 may fail breaking the code. Hence, the spammers have a backup or second CAPTCHA-learning host 2 that tries to learn and break the CAPTCHA code. However, it is possible that spammers also use these two hosts to check the efficiency and accuracy of both hosts involved in breaking one CAPTCHA code at a time, with the ultimate goal of having a successful CAPTCHA breaking process.

To be specific, host 1 has a similar concept that was used to attack Live mail CAPTCHA. This involved extracting an image from a victim’s machine in the form of a bitmap file, bearing BM.. file headers and breaking the code. Host 2 uses an entirely different concept wherein the CAPTCHA image is broken into segments and then sent as a portable image / graphic file bearing PV..X file headers as requests. […]

While it doesn’t say as such, some have read the post to mean that Google’s CAPTCHA has been solved algorithmically. I’m pretty sure this isn’t the case. Here’s why.

Firstly, the FAQ text that appears on “host 1” (thanks Alex for the improved translation!):

img

FAQ

If you cannot recognize the image or if it doesn’t load (a black or empty image gets displayed), just press Enter.

Whatever happens, do not enter random characters!!!

If there is a delay in loading images, exit from your account, refresh the page, and log in again.

The system was tested in the following browsers: Internet Explorer Mozilla Firefox

Before each payment, recognized images are checked by the admin. We pay only for correctly recognized images!!!

Payment is made once per 24 hours. The minimum payment amount is $3. To request payment, send your request to the admin by ICQ. If the admin is free, your request will be processed within 10-15 minutes, and if he is busy, it will be processed as soon as possible.

If you have any problems (questions), ICQ the admin.

That reads to me a lot like instructions to human “CAPTCHA farmers”, working as a distributed team via a web interface.

Secondly, take a look at the timestamps in this packet trace:

img2

The interesting point is that there’s a 40-second gap between the invocation on “Captcha breaking host 1” and the invocation on “Captcha breaking host 2”. There is then a short gap of 5 seconds before the invocations occur on the Gmail websites.

Here’s my theory: “host 1” is a web service gateway, proxying for a farm of human CAPTCHA solvers. “host 2”, however, is an algorithm-driven server, with no humans involved. A human may take 40 seconds to solve a CAPTCHA, but pure code should be a lot speedier.

Interesting to note that they’re running both systems in parallel, on the same data. By doing this, the attackers can

  1. collect training data for a machine-learning algorithm (this is implied by the ‘do not enter random characters!’ warning from the FAQ — they don’t want useless training data)

  2. collect test cases for test-driven development of improvements to the algorithm

  3. measure success/failure rates of their algorithms, “live”, as the attack progresses

Worth noting this, too:

Observation*: On average, only 1 in every 5 CAPTCHA breaking requests are successfully including both algorithms used by the bot, approximating a success rate of 20%. The second algorithm (segmentation) has very poor performance that sometimes totally fails and returns garbage or incorrect answers.

So their algorithm is unreliable, and hasn’t yet caught up with the human farmers. Good news for Google — and for the CAPTCHA farmers of Romania ;)

Update: here’s the NYTimes’ take, with broadly agreeing comments from Brad Taylor of Google. (The Register coverage is off-base, however.)

On the effects of lowering your SpamAssassin threshold

Published February 29, 2008

So I was chatting to Danny O’Brien a few days ago. He noted that he’d reduced his Spamassassin “this is spam” threshold from the default 5.0 points to 3.7, and was wondering what that meant:

I know what it means in raw technical terms — spamassassin now marks anything >3.7 as spam, as opposed to the default of five. But given the genetic algorithm way that SA calculates the rule scoring, what does lowering the score mean? That I’m more confident that stuff marked ham is stuffed marked ham than the average person? That my bayesian scoring is now really good?

Do people usually do this without harmful side-effects? What does it mean about them if they do it?

Does it make me a good person? Will I smell of ham? These are the things that keep me awake at night.

It’s a good question! Here’s what I responded with — it occurs to me that this is probably quite widely speculated about, so let’s blog it here, too.

As you tweak the threshold, it gets more or less aggressive.

By default, we target a false positive rate of less than 0.1% — that means 1 FP, a ham marked as spam incorrectly, per 1000 ham messages. Last time the scores were generated, we ran our usual accuracy estimation tests, and got a false positive rate of 0.06% (1 in 1667 hams) and a false negative rate of 1.49% (1 in 67 spams) for the default threshold of 5.0 points. That’s assuming you’re using network tests (you should be) and have Bayes training (this is generally the case after running for a few weeks with autolearning on).

If you lower the threshold, then, that trades off the false negatives (reducing them — less spam getting past) in exchange for more false positives (hams getting caught). In those tests, here’s some figures for other thresholds:

SUMMARY for threshold 3.0: False positives: 290 0.43% False negatives: 313 0.26%

SUMMARY for threshold 4.0: False positives: 104 0.15% False negatives: 1084 0.91%

SUMMARY for threshold 4.5: False positives: 68 0.10% False negatives: 1345 1.13%

so you can see FPs rise quite quickly as the threshold drops. At 4.0 points, the nearest to 3.7, 1 in 666 ham messages (0.15%) will be marked incorrectly as spam. That’s nearly 3 times as many FPs as the default setting’s value (0.06%). On the other hand, only 1 in 109 spams will be mis-filed.

Here’s the reports from the last release, with all those figures for different thresholds — should be useful for figuring out the likelihoods!

In fact, let’s get some graphs from that report. Here is a graph of false positives (in orange) vs false negatives (in blue) as the threshold changes…

and, to illustrate the details a little better, zoom in to the area between 0% and 1%…

You can see that the default threshold of 5.0 isn’t where the FP% and FN% rates meet; instead, it’s got a much lower FP% rate than FN%. This is because we consider FPs to be much more dangerous than missed spams, so we try to avoid them to a higher degree.

An alternative, more standardized way to display this info is as a Receiver Operating Characteristic curve, which is basically a plot of the true positive rate vs false positives, on a scale from 0 to 1.

Here’s the SpamAssassin ROC curve:

More usefully, here’s the ROC curve zoomed in nearer the “perfect accuracy” top-left corner:

Unfortunately, this type of graph isn’t much use for picking a SpamAssassin threshold. GNUplot doesn’t allow individual points to be marked with the value from a certain column, otherwise this would be much more useful, since we’d be able to tell which threshold value corresponds to each point. C’est la vie!

Update:: this is possible with GNUplot 4.2 onwards, it seems. great news! Hat tip to Philipp K Janert for the advice. here are updated graphs using this feature:

(GNUplot commands to render these graphs are here.)

Update again: much better interactive Flash graphs here.

Microplanets

Published February 26, 2008

Intriguing! Via Glynn Moody comes an interesting new site, Pulse of Open Source:

To highlight open source activity on Twitter, I have launched a new web application today called The Pulse of Open Source. This is the stream of collective consciousness from the open source community on Twitter. You can follow this stream by simply bookmarking the site and visiting regularly or by adding the RSS feed to your feed reader. You can also create a Twitter account and add the individuals you’d like to follow to your own Twitter friends list if you’d prefer. There is also a mobile version of the site for on-the-go viewing.

I’m not entirely convinced it makes sense — the “open source community” is a pretty wide and amorphous concept, covering “enterprisey” types like Iona, to conference organisers, to web standards guys to GNOME developers. That’s a wide range.

However, that site links to the original, and a version which resonates better: PulseOfPDX.com, ‘the stream of Portland’s collective consciousness‘. Basically, this is a local syndication site, with microblogging from a community of local Twitterers. Similar to the “Planet” concept, which aggregates posts from multiple weblogs into a new ‘river of news’ combined feed, as seen on Planet Antispam, Planet Perl, Planet.journals.ie, but for off-the-cuff Twitter microblog comments. It’s a microplanet, to coin a phrase.

I think I might set up one of these for Ireland… what a great idea!

Update: Ted Leung posted about this today as well, I see, linking to this call for an “out-of-the-box” Twitter aggregator:

In theory, this whole pulse idea could be packaged up to be as easily deployable as “planet” sites. Here, “pulse” is the operational brand-name of aggregating Twitter accounts, where as “planet” is the tried and true operational brand-name of aggregating blogs.

I think I still prefer “microplanet” ;)

Update 2: check out IrishPulse!

Plug plug

Published February 22, 2008

It’s been a while since I’ve posted about good shopping experiences I’ve had. Here’s a couple:

SoleTrader.co.uk: I’m a terrible shopper. I hate shops, I always wind up having to visit them at their busiest times on the weekend, and the last time I tried to go shopping for a new pair of shoes, I got caught in torrential rain, fell over and broke my thumb instead. seriously. So feck that.

Instead, I resolved to buy them online, and that I did — from SoleTrader. They had a great range of trainers, I found what I was after, the price was grand, and delivery on time. Shoes are always the same size — their sizes are standardised, after all — so naturally they fit fine. All in all, it worked out great.

Be Organic: these guys operate in North Dublin, delivering bags of organic fruit and vegetables to your door, weekly. We get the Essential Fruit Bag and the Mini Box, with a bi-weekly bag of spuds on top, for EUR 32 per week. The quality of the food is absolutely fantastic, there’s never any spoilage or wilting, and it’s always fresh and delicious. Compared to supermarket fare, it’s leagues ahead. They’ve also been grand and flexible when we need to tweak the order slightly — for example we have a veto on celery, and that’s not an issue at all. The only problem would be that they’ve recently increased their prices… but unfortunately that seems to be a general problem in Ireland these days!

vote for Dustin on Saturday

Published February 20, 2008

A friend of a friend writes:

Unless you are pretty good at avoiding the media, you will be aware that Dustin the Turkey has been chosen as one of six finalists for RTE’s Eurosong, the winner of which will go on to represent Ireland in the Eurovision Song Contest in Serbia in May.

What you may not be aware of is that I wrote and recorded the song with him and need your votes to help get me to Serbia!!!

The TV show will be broadcast live on RTE this Saturday Feb 23rd, at 7pm. It is a televote (a la X-factor format), so get your mobile phones ready. The results are at 9:45pm.

The song, Irlande Douze Points, is a parody on the current types of songs, acts and block-voting in the Eurovision. It may make your ears bleed a bit, you may ask yourself why, but what the hell, send someone you know to the final!!!

Apparently, Dustin urges the contest judges to “give douze points to Ireland, for its lowlands and its highlands, for Terry Wogan’s wig and Bono’s leather pants. We brought you Guinness and Westlife, 800-years of war and strife, but we all apologise for Riverdance.”

Check out the outraged reactions from Ireland’s past Eurovision “winners”:

Frank McNamara, who wrote two of the Irish Eurovision winners, asked whether RTE, the state broadcaster that selected the six acts, was “giving two fingers” to Irish ‘song’writers. “I think it is absolutely disgraceful.”

Shay Healy, who wrote Johnny Logan’s Eurovision hit What’s Another Year?, wondered “how any bunch of grown-ups could come up with this as a solution”

Phil Coulter thought that Eurovision was going “down the tubes”.

The choice on Saturday is between a turkey puppet taking the piss in a Northside accent, and such po-faced “serious pop” mawkfests as ‘“Double Cross My Heart” performed by Donal Skehan’ and ‘“Time to Rise” performed by Maya’. snore. You know it’s got to be the turkey.

Here’s the official Bebo page, and the Facebook group — and here’s the song itself:

Update: actually, here’s another, higher quality clip — with an entirely different song! Let’s hope this is the one…

Update 2: he won. Dana and the other professional Eurovision types have been chewing wasps, it’s hilarious!

A historical DailyWTF moment

Published February 14, 2008

Today, in work, we wound up discussing this classic DailyWTF.com article — “Remember, the enterprisocity of an application is directly proportionate to the number of constants defined”:

public class SqlWords
{
  public const string SELECT = " SELECT ";
  public const string TOP = " TOP ";
  public const string DISTINCT = " DISTINCT ";
  /* etc. */
}

public class SqlQueries
{
  public const string SELECT_ACTIVE_PRODCUTS =
    SqlWords.SELECT +
    SqlWords.STAR +
    SqlWords.FROM +
    SqlTables.PRODUCTS +
    SqlWords.WHERE +
    SqlColumns.PRODUCTS_ISACTIVE +
    SqlWords.EQUALS +
    SqlMisc.NUMBERS_ONE;
  /* etc. */
}

This made me recall the legendary source code for the original Bourne shell, in Version 7 Unix. As this article notes:

Steve Bourne, at Bell Labs, worked on his version of shell starting from 1974 and this shell was released in 1978 as Bourne shell. Steve previously was involved with the development of Algol-68 compiler and he transferred general approach and some syntax sugar to his new project.

“Some syntax sugar” is an understatement. Here’s an example, from cmd.c:

LOCAL REGPTR    syncase(esym)
        REG INT esym;
{
        skipnl();
        IF wdval==esym
        THEN    return(0);
        ELSE    REG REGPTR      r=getstak(REGTYPE);
                r->regptr=0;
                LOOP wdarg->argnxt=r->regptr;
                     r->regptr=wdarg;
                     IF wdval ORF ( word()!=')' ANDF wdval!='|' )
                     THEN synbad();
                     FI
                     IF wdval=='|'
                     THEN word();
                     ELSE break;
                     FI
                POOL
                r->regcom=cmd(0,NLFLG|MTFLG);
                IF wdval==ECSYM
                THEN    r->regnxt=syncase(esym);
                ELSE    chksym(esym);
                        r->regnxt=0;
                FI
                return(r);
        FI
}

Here are the #define macros Bourne used to “Algolify” the C compiler, in mac.h:

/*
 *      UNIX shell
 *
 *      S. R. Bourne
 *      Bell Telephone Laboratories
 *
 */

#define LOCAL   static
#define PROC    extern
#define TYPE    typedef
#define STRUCT  TYPE struct
#define UNION   TYPE union
#define REG     register

#define IF      if(
#define THEN    ){
#define ELSE    } else {
#define ELIF    } else if (
#define FI      ;}

#define BEGIN   {
#define END     }
#define SWITCH  switch(
#define IN      ){
#define ENDSW   }
#define FOR     for(
#define WHILE   while(
#define DO      ){
#define OD      ;}
#define REP     do{
#define PER     }while(
#define DONE    );
#define LOOP    for(;;){
#define POOL    }


#define SKIP    ;
#define DIV     /
#define REM     %
#define NEQ     ^
#define ANDF    &&
#define ORF     ||

#define TRUE    (-1)
#define FALSE   0
#define LOBYTE  0377
#define STRIP   0177
#define QUOTE   0200

#define EOF     0
#define NL      '\n'
#define SP      ' '
#define LQ      '`'
#define RQ      '\''
#define MINUS   '-'
#define COLON   ':'

#define MAX(a,b)        ((a)>(b)?(a):(b))

Having said all that, the Bourne shell was an awesome achievement; many of the coding constructs we still use in modern Bash scripts, 30 years later, are identical to the original design.

Technorati bloginfo API wierdness

Published February 13, 2008

For the benefit of other Technorati API users…

In a comment on this entry, Padraig Brady mentioned that his blog had mysteriously disappeared from the Irish Blogs Top 100 list.

I investigated, and found something odd — it seems Technorati has made a change to their bloginfo API, now listing weblogs with their ‘rank’, but without some of the important metadata, like ‘inboundblogs’, ‘inboundlinks’, and with a ‘lastupdate’ time set to the epoch (1970-01-01 00:00:00 GMT), in the API. Here’s an example:

<!-- generator="Technorati API version 1.0" -->
<!DOCTYPE tapi PUBLIC "-//Technorati, Inc.//DTD TAPI 0.02//EN"
                 "http://api.technorati.com/dtd/tapi-002.xml">
<tapi version="1.0">
<document>
    <result>
        <url>http://www.pixelbeat.org</url>
                    <weblog>
                <name>Pádraig Brady</name>
                <url>http://www.pixelbeat.org</url>
                <rssurl></rssurl>
                <atomurl></atomurl>
                <inboundblogs></inboundblogs>
                <inboundlinks></inboundlinks>
                <lastupdate>1970-01-01 00:00:00 GMT</lastupdate>
                <rank>74830</rank>
            </weblog>
                            </result>
</document>
</tapi>

Compare that with this lookup result, on my own blog:

<?xml version="1.0" encoding="utf-8"?>
<!-- generator="Technorati API version 1.0" -->
<!DOCTYPE tapi PUBLIC "-//Technorati, Inc.//DTD TAPI 0.02//EN"
                 "http://api.technorati.com/dtd/tapi-002.xml">
<tapi version="1.0">
<document>
    <result>
        <url>http://taint.org</url>
                    <weblog>
                <name>taint.org: Justin Mason’s Weblog</name>
                <url>http://taint.org</url>
                <rssurl>http://taint.org/feed</rssurl>
                <atomurl>http://taint.org/feed/atom</atomurl>
                <inboundblogs>143</inboundblogs>
                <inboundlinks>227</inboundlinks>
                <lastupdate>2008-02-12 11:48:10 GMT</lastupdate>
                <rank>43404</rank>
            </weblog>
                            <inboundblogs>143</inboundblogs>
                            <inboundlinks>227</inboundlinks>
            </result>
</document>
</tapi>

This bug had caused a number of blogs to be dropped from the list, since I was using “inboundblogs and inboundlinks == 0” as an indication that a blog was not registered with Technorati.

It’s now worked around in my code, although a side-effect is that blogs which have this set will appear with question-marks in the ‘inboundblogs’ and ‘inboundlinks’ columns, and will perform poorly in the ‘ranked by inbound link count’ table (unsurprisingly).

I’ve posted a query to the support forum — let’s see what the story is.

Interesting Irish Blog Awards shortlistee

Published February 12, 2008

This year’s Irish Blog Awards shortlists were posted yesterday. I maintain the Irish Blogs Technorati Top 100 list, so good sources of Irish blog URLs are always welcome; I took the shortlisted blogs and added them all.

Interestingly, straight in at number 2 went towleroad.com (warning: not worksafe!). It has a staggering Technorati rank of 1074 — way ahead of Donncha’s 5831 or Mulley’s 8678. I was pretty curious as to how an Irish blog could hit those heights without me having heard of it, so I took a look.

Let’s just say the content isn’t quite what you’d expect to find in a blog shortlisted for ‘Best News/Current Affairs Blog’ — a little bit short on Irish news, but heavy on pictures of naked guys getting off with each other. ;)

I took a quick glance, and I couldn’t spot any Irish content. WHOIS says the the publisher is LA-based. so I’m curious as to what qualified it as an “Irish blog”…

(by the way, I tried to leave a comment on the blog entry, but it appears Akismet is marking my comments as spam on a number of Wordpress-based blogs at the moment. Yes, I am aware of the irony. No, if SpamAssassin was a blog-spam filter, it wouldn’t do that ;)

Update: it’s sorted — they’re now gone. Also, it appears I’ve been removed from the Akismet blacklist, yay.

More on the Trend Micro patent

Published February 12, 2008

Dutch free knowledge and culture advocacy group ScriptumLibre.org has called for a worldwide boycott of Trend Micro products. Their chairman, Wiebe van der Worp, claims Trend Micro’s aggressive use of litigation is “well beyond the borders of decency”.

Also, this Linux.com feature has a great quote from Jim Zemlin, the executive director of the Linux Foundation:

“A company that files a patent claim against code coming from a widely adopted open source project vastly underestimates the self-inflicted damage to its customer and community relationships. In today’s world, all of our customers in the software industry are enjoying the benefits of a wide variety of open source projects that provide stability and vendor-neutral solutions to the most basic of their computing needs. I talk to those customers every day. They consider these claims short-sighted and those that assert them to be fearful of their ability to compete in today’s economy.”

Well said.

Plug: Lenovo service still rocks

Published February 8, 2008

I needed to buy a new laptop for work a few months back, and after a little agonizing between the MacBook Pro and a Thinkpad T61p, I plumped for the latter. As I noted at the time, one of the major selling points was the quality of IBM/Lenovo’s after-sales warranty service, compared to the atrocious stories I’d heard about AppleCare in Europe. I was, however, taking a leap of faith — I had used IBM service to great effect in the US, but had never actually tried it out in Ireland.

Sadly, I had to put this to the test today, after the hard disk started producing these warnings:

/var/log/messages:Feb  7 11:21:13 wall kernel: 
[2075890.116000] end_request: I/O error, dev sda, sector 116189461
/var/log/messages:Feb  7 11:21:38 wall kernel: 
[2075914.824000] end_request: I/O error, dev sda, sector 116189460
/var/log/messages:Feb  7 11:24:18 wall kernel: 
[2076075.072000] end_request: I/O error, dev sda, sector 116189462
/var/log/messages:Feb  7 11:25:05 wall kernel: 
[2076121.932000] end_request: I/O error, dev sda, sector 116189463

It’s a brand new machine, and a Hitachi TravelStar 7K100 drive, with a good reputation for reliability — but these things do happen. :(

Interestingly, I thought this was a case of the Bathtub curve in action — but this comprehensive CMU study of hard drive reliability notes that the ‘infant mortality’ concept doesn’t seem to apply to current hard-drive technology:

Replacement rates [of hard drives in a cluster] are rising significantly over the years, even during early years in the lifecycle. Replacement rates in HPC1 nearly double from year 1 to 2, or from year 2 to 3. This ob- servation suggests that wear-out may start much earlier than expected, leading to steadily increasing replacement rates during most of a system’s useful life. This is an in- teresting observation because it does not agree with the common assumption that after the first year of operation, failure rates reach a steady state for a few years, forming the “bottom of the bathtub”.

Anyway, I digress.

I ran the BIOS hard disk self-test, got the expected failure, then rang up Lenovo’s International Warranty line for Ireland. I got through immediately to a helpful guy in India, and gave him my details and the BIOS error message; he had no tricky questions, no guff about me using Linux rather than Windows, and there were no attempts to sting me for shipping.

There’s now a replacement HD (and a set of spare recovery disks, bonus!) winging their way via 2-day shipping, expected on Tuesday; I’m to hand over the broken HD to the courier once it arrives. Fantastic stuff!

Assuming the courier doesn’t screw up, this is yet another major win for IBM/Lenovo support, and I feel vindicated. ;)

Update: the HD arrived this morning at 10am — a day early. Very impressive!

CEAS needs your ham

Published February 5, 2008

CEAS 2008 is doing another Spam Challenge test of various spam-filters, and as part of this, they need samples of ham mail messages.

As part of the data collection effort, we have set up a website through which it is possible to donate non-sensitive legitimate email, to be used in the evaluation. Any kind of email that the recipient considers legitimate is welcome, including computer generated (non-spam) messages.

After the CEAS evaluation, the benchmark data will be made publicly available to facilitate future reasearch and development in the field of spam prevention.

Here is the collection site; they accept UNIX mbox format, and tar.gz or zip files of same, with an 8MB upload limit.

Remote sound playback through a Nokia 770

Published February 3, 2008

For a while now, I’ve been using various hacks to play music from my Linux laptop, holding my main music collection, to client systems which drive the speakers.

Previously, I used this setup to play via my MythTV box. Nowadays, however, my TV isn’t in the room where I want to listen to music. Instead, I have my Nokia 770 hooked up to the speakers; this plays the BBC Radio 4 RealAudio streams nicely, and also the laptop’s MP3 collection using a uPnP AV MediaServer.

I specifically use TwonkyMedia right now, playing back via the N770’s Media Streamer app. (That works pretty well — uPnP AV is one of those standards plagued with incompatibilities, but TwonkyMedia and Media Streamer seem to be a reliable combination.)

However, TwonkyMedia sometimes fails to notice updates of the library, and nothing has quite as good a music-player user interface as JuK, the KDE music player and organiser app, so a way to play directly from the laptop instead of via uPnP would be nice…

A weekend’s hacking reveals that this is pretty easily done nowadays, thanks to some cool features in pulseaudio, the current standard sound server on Ubuntu gutsy, and the Esound server running on the N770.

Unfortunately, the N770 doesn’t (yet) support pulseaudio directly, otherwise we could use its seriously cool support for RTP multicast streams. Still, we can hack something up using the venerable “esd” protocol (again!) Here’s how to set it up…

On the N770:

You need to fix the N770’s “esd” sound server to allow public connections. Set up your wifi network’s DHCP server to give the N770 a static IP address. Log in over SSH, or fire up an xterm. Run the following:

mv /usr/bin/esd /usr/bin/esd.real

cat > /usr/bin/esd <<EOM
#!/bin/sh
exec /usr/bin/esd.real -tcp -public -promiscuous -port 5678 $*
EOM

chmod 755 /usr/bin/esd
/etc/init.d/esd restart

On the server:

Download this file, and save it as n770.pa. Edit it, and change server=n770:5678 on the fourth line to use the IP address or hostname of your Nokia 770 instead of n770. Then run:

cp n770.pa ~/.n770.pa

cat > ~/bin/sound_n770 <<EOM
#!/bin/sh
pulseaudio -k; pulseaudio -nF $HOME/.n770.pa &
EOM

cat > ~/bin/sound_here <<EOM
#!/bin/sh
pulseaudio -k; pulseaudio &
EOM

chmod 755 ~/bin/sound_here ~/bin/sound_n770

Now you just need to run ‘~/bin/sound_n770’ to redirect sound playback to the N770, and ‘~/bin/sound_here’ to reset back to laptop speaker output, for the entire desktop environment. Nifty!

Update: it appears that things may work more reliably if you add “rate=22050” at the end of the “load-module module-esound-sink” line — this halves the bitrate of the network stream, which copes better with harsh wifi network conditions. The n770.pa file above now includes this.

Irish crumblies don’t trust blogs

Published January 31, 2008

It appears a public relations firm, Edelman’s, recently performed a phone survey which concluded that bloggers are the “least trusted” group of authority figures source of information in Ireland. This has been widely reported:

on Edelman Dublin’s blog:

When we consider who we trust the most as a spokesperson in Ireland, the most trusted sources of information include, financial or industry analysts at 62%, followed by a doctor or healthcare specialist at 57%, an NGO representative at 57% and academics at 53%. Bloggers are the least trusted at 7%.

at Silicon Republic:

Bloggers have emerged as the “least trusted” group in the country.

and on ElectricNews.net:

“What has been interesting to note in this year’s findings is the apparent low standings of bloggers and social media in general,” said [Mark Cahalane, managing director of Edelman Dublin]. “One interpretation of the survey would be that bloggers have now entered the mainstream and people no longer distinguish between blogs and ordinary websites. This is also reflected by the fact that numerous high profile bloggers are widely quoted in the media.”

However, as Damien noted, Piaras Kelly raised a very significant point about this — ‘the people surveyed for the research had to fit a certain demographic, including having to be aged between 35-64.’ […] ‘A Generational gap is evident.’ This press release corroborates that. Sure enough, most blog readers (and writers) would tend to be of the younger generation — a pretty key point, one would assume, but one that most of the non-blogger coverage has omitted ;)

(Update: the term “authority figure” wasn’t quite correct; replaced with what Edelman themselves use, “source of information”.)

Trend Micro’s attack on open source

Published January 29, 2008

Trend Micro are demanding that Barracuda Networks pay licensing fees, alleging that they infringe U.S. Patent No. 5,623,600 with their use of the open-source anti-virus tool ClamAV. Here’s a Barracuda press release, and here’s some details from Barracuda:

Trend Micro alleges that Barracuda Networks and ClamAV infringe on Trend Micro’s U.S. Patent No. 5,623,600. Barracuda Networks believes that the patent is invalid due to prior art and further believes that neither its products nor the ClamAV software infringe the patent.

On Sept. 21, 2006, Trend Micro sent Barracuda Networks a letter regarding a license to Trend Micro’s ‘600 patent. After several discussions on paying a license for the patent, Trend Micro demanded Barracuda Networks either remove ClamAV from its products or pay a patent license fee. Barracuda Networks felt it had no choice other than to file for a declaratory judgment in early 2007 in U.S. Federal Court to invalidate Trend Micro’s ‘600 patent and end continued legal threats against Barracuda Networks for use of the free and open source ClamAV software.

Trend Micro subsequently responded to that declaratory action and more recently, Trend Micro filed a claim with the International Trade Commission (ITC). The ITC voted to investigate the claim in December 2007. Trend Micro’s ITC claim alleges that Barracuda Networks infringes on Trend Micro’s ‘600 patent, but effectively implies that anyone using the free and open source ClamAV software at the gateway infringes the patent.

The interesting aspects of this case, from my point of view, are twofold — the patent is a classic bad software patent, very broad and totally obvious both now and at the time it was issued; and it hinges on Barracuda’s use of the free software antivirus product, ClamAV. Given Apache SpamAssassin‘s prevalence in many anti-spam mail filtering appliances (including Barracuda!), this is a very worrying precedent for us — our product could be next, for some other patent troll company’s extortion scheme.

For what it’s worth, it appears this patent has long been a licensing moneyspinner for Trend. In 1997, once the patent was issued, Trend went on a spree; McAfee, Symantec and Integralis were sued, eventually buying licenses, as did Electric Mail Company. 2 years ago, Fortinet were sued and settled in their case.

I happily gave Barracuda a quote for their press release on this:

“Trend Micro’s actions are clearly an attack on free and open source software and its users, as well as on Barracuda Networks. The ‘600 patent covers a trivial method, one which was obvious to anyone skilled in the art at the time the patent was written, and should be rendered invalid as soon as possible. I hope that Barracuda Networks is successful in its attempts to defend all users from this patent shakedown.”

If you know of prior art for this patent, please head over to Barracuda’s site and provide details — helping to fend off this protection racket would be good for all of us. Barracuda say:

People should look for art dated prior to Trend Micro’s filing date of September 26, 1995. The ‘600 patent is entitled “Virus Detection And Removal Apparatus For Computer Networks.” We are interested in all material, including software, code, publications or papers, patents, communications, other media or Web sites that relate to the technology described prior to the filing date.

In particular, this prior art should show antivirus scanning on a firewall or gateway. However, many of the claims do not require virus detection at a gateway. So any material that illustrates virus scanning on a file server is also of interest.

We also believe that a product called MIMESweeper 1.0 from a company called Clearswift, Authentium, or Integralis anticipates several claims of the ‘600 patent. We have yet to locate a copy of this product and would appreciate anyone who has a copy sending it our way.

Some more coverage:

  • Don Marti at LinuxWorld: ‘Regardless of the decision in this case, software patent trolls will continue to be a problem for all software companies, Eben Moglen says. “Getting them to [not operate] in your neighborhood is the best you can do.”‘

  • Matt Asay at C|Net: ‘Antivirus and antispam innovation has tended to come from open source, not the large proprietary vendors. Trend Micro’s lawsuit is designed to put cash in its pocket but will end up hurting the consumer.’ (Matt led with my quote ;)

  • GrokLaw: ‘Anyone using ClamAV, should Trend Micro be successful, is potentially a target.’

  • Ars Technica: ‘The patent is very clearly without merit, but that hasn’t stopped Trend Micro from using it to threaten ClamAV and extort money from several companies. Situations like this demonstrate a very urgent need for patent reform and illuminate the risks posed by broad software patents, particularly in the area of security.’

Interview with two phish-scene infiltrators

Published January 28, 2008

/. posted a link to this interview with Nitesh Dhanjani and Billy Rios, two guys who have infiltrated the “phishing underground”.

It’s a good article — lots of detail on the current toolset of a typical phisher, and some details on the community itself:

I had always thought that most phishers were clever hackers evading authorities using the latest evasion techniques and tools. The reality of the matter is most of the phishers we tracked were sloppy and unsophisticated. The tools they used were rarely created by the phisher deploying the actual scam, and for the most part it seemed the phisher merely downloaded kits and tools from some place and reused over and over and over again. It also seemed that many phishers don’t even really understand how the phishing kits they’ve deployed work! We also came across many phishing kits and tools that had simple backdoors written into the source code (essentially, phishers phishing phishers). These backdoors are easily spotted by anyone who has even a basic idea of how the source code flow worked, yet was undetected by many phishers. Maybe a few phishers out there are skilled, but the majority are clueless.

Here’s something I’ve noted about spammers, too — there’s no honour among thieves:

The number of backdoors we saw was staggering. The servers serving the phishing sites had backdoors, the code used in the phishing kits had backdoors, the tools used by phishers had backdoors. Phishers aren’t afraid to steal from regulars people and they are also not afraid to steal from other phishers. Some of the backdoors were meant to keep control over a compromised server, while other simply stole information that had been stolen by other phishers! We came across several forums where phishers, scammers, and carders basically identified other phishers, scammers, and carders that had scammed them. These shady characters may work with each other but they sure don’t trust each other, that’s for sure.

And this is a very important point about blacklists:

Phishers are likely to abuse the blacklists published for [anti-phishing] plugins for their own benefit. The blacklists are a list of known phishing sites that the plugins consume in order to identify what websites are fraudulent. These blacklists therefore contain IP addresses and host names of servers hosting phishing sites. Since phishing sites are commonly installed on servers that have been compromised, and phishers don’t bother to patch systems they have installed their kits on, this list translates to a ‘list of easily compromisable hosts’ for other phishers.

On the latter point, this is one of the key benefits of DNS blocklists, compared to the downloaded, text-based style that Google initially used for its anti-phishing toolbar. To query a DNSBL, you need to know the address you’re looking for first of all; but with a text file, you can read the lists in their entirety, without knowing the address in advance. (Google is now apparently tending to use the enchash format, which fixes this.)

And a final word:

For the next few years, we are going to continue to apply band-aids around the problem of data leakage, and continue to play whack-a-mole with the phishers without solving the actual problem at hand. In order to make any significant progress, we must come up with a brand new system that does away with depending on static identifiers. We will know weâ??ve accomplished this when we will be able to publish our credit reports publicly without fearing for our identities.

(I’d place more importance on the liability of the financial institutions, myself — I think they get away with placing too much blame on the victims of fraud and identity theft.)

Good interview — worth reading.

Insane Dell.ie markup

Published January 21, 2008

A good deal came up on a mailing list I’m on: SAMSUNG 245BW Black High Glossy 24″ 5ms DVI Widescreen LCD Monitor for $459.99, or $409.99 after rebate, via Newegg.

A follow-up from a German poster: he’d just picked up a Dell 2407WFP-HC ‘for the low, low price of 659 EUR’.

We marvelled at the price difference — then I looked up Dell.ie forcomparison. I thought 659 EUR was bad, but Dell.ie is asking for 1,117.74 Euros inc VAT for the same product — insane!!

What possible excuse could there be for that? EUR 458.74 worth of shipping maybe? Do they encase it in platinum? That’s nearly three times the price of the Newegg monitor.

Update: Duh. I’m an idiot. That’s a 2707WFP, not a 2407WFP; it’s 3″ bigger and quite a bit fancier. It appears Dell.ie is no longer selling the 2407WFP.

Bad law in North Dakota

Published January 17, 2008

This is very bad news for North Dakota-based anti-spammers — a guy called David Ritz is being sued there by alleged porn spammer Jerry Reynolds, for performing DNS lookups, a DNS zone transfer and a Whois lookup. It appears the judge has found Ritz guilty.

This is astonishingly bad lawmaking by the judge. These are entirely innocuous tools, part of every network administrator’s toolkit for debugging and examining internet traffic legitimately. There’s nothing remotely criminal or malicious in their use, and the judge has allowed himself to be misled.

North Dakota Judge Gets it Wrong:

‘Ritz’s behavior in conducting a zone transfer was unauthorized within the meaning of the North Dakota Computer Crime Law. A zone transfer is simply asking a DNS server for all the particular public info it provides about a given domain. This is a common task performed by system administrators for many purposes. The judge is saying that DNS zone transfers are now illegal in North Dakota.’

More details from Ed Falk

David’s legal defense fund

My Commodore 64 demos

Published January 10, 2008

I recently came across my record at the Commodore Scene Database, and was happy to find that someone had found and uploaded two demos I had written, back in my days as a member of the C=64 demo scene between 1988 and 1990:

(I was a member of the groups ‘Excess’ and ‘Thundertronix’ / ‘TNT’, going by the handle of ‘Mantis’.)

With the help of CBA, I was overjoyed to track down another long-lost demo, my crowning achievement on the platform:

If you’re curious, feel free to go read those wiki pages or download the .d64’s — they run fine in VICE, the Commodore emulator (amazingly). If you’ve only got time to check one, check Rhaphanadosis; it’s much better than the others.

I’m very impressed with VICE. As far as I can tell, it’s perfectly bug-for-bug compatible with the real hardware, playing all of the demos perfectly (apart from a little additional speed due to differing hardware performance). If you haven’t already got VICE set up, bear in mind that after installing it, you’ll need a copy of the C=64’s ROM images; here’s a local set.

Also, the Commodore Scene Database is pretty awesome — it’s a full-scale IMDB-style setup, tracking the history of the Commodore demo scene in massive detail. Nice work guys!

The demos were written 100% in 6502/6510 assembly. I developed them using an Action Replay cartridge’s built-in monitor; it had an assembler, but one which didn’t support symbolic addressing. In other words, every piece of assembly used hand-computed branch offsets, and every variable and subroutine was tracked — on paper — by memory location, rather than using symbolic labels. If you want to know what the monitor was like, the VICE built-in monitor is almost identical!

I wrote these when I was 16; part 4 of Rhaphandosis notes the date as being 20 May 1989.

It’s interesting reading the scrollers, and doing web and CSDB searches in follow-up to see what happened next — one of the other Excess members, Raistlin is now Robert Troughton, a successful game developer in the UK with several major titles under his belt.

A Google search for Thundertronix finds a copy of “sex’n’crime” zine, issue 17, July 1990, which notes:

one of the new groups formed in 1990 (jm: slightly off, I think) is THUNDERTRONIX, better known as TNT. they are based in ireland and are doing very well for themselves. they have, in my mind, one of the best coders in the uk, namely MANTIS. he is currently coding a game with many new routines, etc… hopefully he should get some demos out soon!

woo! Er, unfortunately that game never went anywhere. ah well. ;)

BTW, it’s funny reading my scrollers in those demos. At the time, I was convinced that the c=64 was a dead platform — yet here we are in 2008, and there’s still a thriving demo scene on the Commodore. Incredible!

Vincent Browne on RTE’s coke habit

Published January 4, 2008

Before Christmas, it seemed you could hardly read a newspaper, listen to the radio or watch TV in Ireland without being bombarded with stories about how the country was awash in cocaine.

It’s an attractive story, tying in nicely with the death of lingerie model Katy French, hand-wringing over Ireland’s recent ‘celtic tiger’ wealth, a supposed loss of our traditions, etc. etc. RTE, our national broadcaster, made a tabloid series called ‘High Society’, which cashed in on the issue in a particularly crass way — crappy “reconstructions” of actors chopping lines with voiceovers, dodgy-looking men handing over money to ominous music, that kind of thing.

Well, just before Christmas, Vincent Browne wrote a fantastic op-ed in the Irish Times regarding this. I have to quote this particularly perceptive passage:

Cocaine abuse is a social problem, but the thrust of much of RTE’s coverage of the phenomenon is to suggest that it is a widespread, pervasive problem. There are no recent statistics available on the prevalence of cocaine consumption in Ireland – the last survey was done four years ago. The National Advisory Committee on Drugs (NACD) will be publishing a prevalence report next month and we will know then the size of the phenomenon.

But we have some indicators about the scale of cocaine use. The European drug agency EMCDDA estimates that 3 per cent of all adults in Europe aged between 15 and 64 have used cocaine at least once in their lives.

A third of these took cocaine during the previous year and half of these took cocaine during the previous month. This means that about 0.5 per cent of the adult population took cocaine over the previous month. And the data suggests that, for at least two-thirds of those who have ever taken cocaine, the drug is not a problem for them.

In the US the statistics are higher. Almost 15 per cent of the population aged between 12 and 64 have taken cocaine in their lives and 2.5 per cent took cocaine over the previous year. Again, this is suggestive that cocaine use for most people is not a problem, otherwise the number of people who took cocaine during the previous year as a proportion of the number of people who ever took cocaine would be far higher.

The figures for Ireland are likely to be that about 4 per cent of the adult population have taken cocaine in their lifetime, with about 1 per cent having taken cocaine in the previous year and 0.5 per cent having taken cocaine in the previous month.

It would be better if people did not take cocaine, but the prevalent contention that the consumption of cocaine at all is necessarily harmful and addictive is obviously false.

It would also be better if people did not drink here, for the problems related to the consumption of alcohol are far, far greater than in the case of cocaine.

Instead of presenting a balanced picture of the cocaine phenomenon, RTE has greatly exaggerated the issue, in a way more typically associated with tabloid journalism.

Well said!

Spambots stealing GMail and Hotmail passwords?

Published December 21, 2007

I just received this mail from a friend:

Dear friend

Welcome to stwoxy.com ! We are one of the largest electronic distributors and wholesalers in Beijing China. We offer qualified digital products: Motorcycles?TVs, Notebooks, phones. PSP, projectors, GPS, DVD, DV, DC, MP3/4 and so on, which are of world famous brands, such as Sony, IBM, PHILIPS, NOKIA, DELL and so on. All our items are brand new from the manufactures and they come with 1-3 years’ after service. These days we are expanding our overseas market, and every item is sold in extremely low price. Such chances should never be missed, ladies and gentlemen, do come to stwoxy.com! you will surely have a big surprise! We are looking forward to hearing from you!

It was sent from a HTTP connection into GMail, and was delivered from there using valid DKIM, Domain Keys and SPF signatures. In addition, it was sent to all the addresses in his address book. In other words, this was no run-of-the-mill impersonation spam — for this one, the spammer obtained my friend’s username and password somehow, logged into GMail, scraped the address book, and then sent spam via GMail that way.

My friend says he didn’t access GMail using a desktop mail client, but did have his Google password saved in his web browser (a pretty typical configuration). My theory is that some virus/malware has infected his desktop machine, captured the saved-passwords file from the web browser configuration, and used that to log into GMail. Alternatively, it could also be a guessable username and password which was picked up via dictionary attack, I guess…

This is the first case I’ve heard of where spammers are actively stealing user account authentication tokens, in order to take over the accounts for spamming. (We’d long predicted it, of course, since it’s a natural response to “pay for mail” schemes… but since there’s no widely-used pay-for-mail system available yet, it’s premature!)

It seems this is not just a GMail thing, btw. Here’s a report of the same thing happening to some French guy via HotMail last month (or in english). I don’t speak Dutch, but this forum post looks like it might be the same situation.

If you’re curious, here’s a copy of the spam, delivered to a Yahoo! group; it appears these spammers aren’t too sophisticated in terms of the text they’re sending, since they haven’t morphed that text, HTML, or even the domain in the link yet. It’s just the malware that’s sophisticated, at this stage.

GNOME, Google and the UNIX user interface

Published December 20, 2007

Recently, after a flurry of annoying user interface issues, I’ve switched my RSS reader from Liferea to Google Reader. Interestingly, it turns out that Google Reader actually fits better with the traditional UNIX user interface concept, I’ve found.

What triggered this was an upgrade from Liferea 1.0.x to 1.4.4 as part of Ubuntu Gutsy; this brought with it a lot of changed behaviours, such as ‘drag-and-drop of feed URL to HTML view no longer subscribes’, and one crucial UI issue, ‘”Skim through articles” only works with ctrl+space’.

I’ve been a long-time UNIX user, dating back to the days where curses-based interfaces were the norm. As such, I tend to drive commonly-used applications using keyboard commands where possible. (This isn’t a purely UNIX thing; Windows has the phenomenon of the keyboard-wielding “power user”, too.)

Liferea was attractive, since it offered the ability to skim through articles quickly by just pressing the “Space” key; simply press space to page down, or to skip to the next unread article if at the end of the current one. Unfortunately, Liferea 1.4.x breaks this, and it wasn’t going to be fixed, since apparently a GNOME app shouldn’t behave this way:

GTK explicitely does implement as a key binding for several of it’s widgets. Rebinding means to break the default behaviour for such widgets (tree views, buttons, input fields). [….] Liferea as a web-browsing application should behave like any other web browser and like every other GNOME/GTK application as much as possible.

Now, I don’t know if it’s GNOME’s fault, or what, but for a UNIX desktop app to break with UNIX UI conventions, that’s a bad move in my opinion. I gave it a bit of argument in the bug tracker, but eventually gave up as I clearly wasn’t getting anywhere. :(

Instead, based on recommendation from friends, I gave Google Reader a try, and quickly figured out its extensive collection of keyboard shortcuts. Now, I’m skimming through my feeds in even less time than it took with Liferea, simply by hitting “ga” to go to my “all unread items” list, then “j”, “j”, “j” to skip through the postings one by one. Sweet!

It’s interesting to note that other Google web apps use the same concepts; Gmail also has a hefty set, and can be driven using them in a manner very reminiscent of the classic UNIX mailreader, Mutt. So, despite being designed with end-users in mind by extremely clever professional user experience designers, these apps still find space for power-user keyboard operation. Take note, GNOME.

Anyway, I’m not too bothered. Google Reader brings other benefits, such as fixing this bug: ‘please add ability to go to previous entry in Unread feed’, avoiding ‘constant memory leak requires daily restarts’, and, of course, the utility of being able to track the same set of feeds and keep track of which items I’ve read in two places (work and home).

If only it was open source ;)

Planet Antispam update

Published December 7, 2007

A brief update on Planet Antispam…

I’ve just added MailChannels’ Anti-Spam Blog. Now — in the interests of disclosure — I’m a member of MailChannels’ Technical Advisory Board. However, that didn’t affect this — their blog has had consistently good, interesting posts dealing with anti-spam-related topics, and without too much plugging of their own products. ;)

Also added recently:

If you know of any other good email anti-spam-related blogs, drop a line in the comments here. (Note that I’m trying to keep it email-related, however, so we’re not covering web-spam.)

Spammers “giving up” according to Google

Published November 30, 2007

According to this Wired story, Google reckons spammers are giving up on spam:

a remarkable trend is underfoot, according to Brad Taylor, a staff software engineer at Google: The number of spam attempts — that is, the number of junk messages sent out by spammers — is flat, and may even be declining for the first time in years.

Actually, this is a wilful misunderstanding of what the Googler in question really said, which was that ‘attempts to spam Gmail users have been leveling off over the last year and more recently, even declining slightly’. In other words, they didn’t make an observation about the state of the spam problem on an internet-wide basis — just about the “local” situation as it pertains to Gmail. Bad reporting there, Wired.

But, in passing…

David Berlind at ZDNet recently blogged a rather grumpy response to InfoWorld coverage of CEAS 2007. He raised a very important point:

If I could say something to the author of that story, it would be that so long as any anti-spam solution is not deployed universally throughout the Internet’s e-mail system (in other words, so long as some anti-spam tech is not a standard), that anti-spam solution actually makes the spam problem worse. You read that right. Worse. Proprietary anti-spam solutions make the global spam problem worse. They are digging us deeper into the hole that the Internet is already in because everyone who makes those solutions is under the false belief that “s/he who is finally successful at filtering out all spam while allowing the legitimate mail in wins.”

Google’s blog post is a case in point: ‘we’re keeping more spam out of your inbox than ever before, so more and more, you can use Gmail for things you enjoy without even realizing that the spam filter is there most of the time.’

That’s great — but it doesn’t help anyone except Gmail. It’s a myopic view of the spam problem, and David’s point stands.

(I disagree with his later conclusion that the only way forward is for Google, MS, AOL and Yahoo! to get together and ‘commit to jointly supporting the same technical solutions’ — when the usual BigCos get together, they tend to focus on their own priorities. Take what happened back in 2005 with nofollow for blog-spam — while it helped the search giants with their own overriding priority, which was to tweak their algorithms to filter out the spam on the search results page, it did nothing to slow the spam flood itself, which has continued unabated.)

We need more open-source, and open-data, anti-spam work.

Informed

Published November 30, 2007

This should be in the running for “least informative dialog ever”.

(The information in question was that Firefox had been upgraded by the Ubuntu Gutsy Update Manager app, if you’re curious…)

Working around O2 Ireland

Published November 28, 2007

I’m pretty conservative with my mobile phones — until recently, my mobiles were all cheap, low-end, super-lightweight Nokias with long battery life and low “worry factor” (ie. not a big deal if they were lost or stolen). Very sensible.

I’ve finally started catching up with the gadgetorati, though — my current phone is now a Sony Ericsson K550i, which is still small and light, but has nice features like a 2 megapixel camera, a decent amount of onboard flash space, and a good implementation of Java, hence support for GMail and Google Maps. (Thanks to Joe for the recommendation!)

The only downside is that it came from my operator, O2 Ireland, with some broken configuration settings. (This shouldn’t be surprising, of course — I don’t think I’ve ever heard of a phone arriving with working data connectivity, from any operator, anywhere in the world.)

Anyway, here’s what I’ve done so far to fix it. Hopefully this might be helpful for random google searchers.

1. “Failed to resolve hostname” when publishing photos:

Generally, when I’d try to publish a photo using its Blogger support, I’d get a “failed to resolve hostname” error message. Investigating further, I found that the “O2 WAP” service used a proxy server — turning that off fixed the problem nicely. Nice reliable proxy you’ve got there, O2 ;)

Here’s how to do that. Open the menu, then select Settings -> Connectivity -> Internet settings -> Internet Profiles. Select O2 WAP and hit More -> Settings. Select Use proxy and change it to No, then hit Save. Problem solved.

2. Cannot send email from the device:

O2’s default mail server has a tendency to refuse to accept outbound mail from the phone. Switching to GMail for outbound SMTP works fine. Notice a trend here?

Open the menu, Messaging -> Email -> Settings -> New account. Set the Account name to “gmail”. Scroll down to Email address, set it to “yourname@gmail.com”. Connection type is “POP3”, Username and Password are whatever your GMail account uses. Outgoing server is “smtp.gmail.com”. Enter Advanced settings, and set Encryption to “TLS/SSL”. Set Outgoing port to “25”. Press the back button, then select the “gmail” account’s tickbox to make it active, before pressing back again to exit the configuration screen.

3. The “side” buttons go online:

By default, if you hit the “globe” button or the “open window” button on the side of the phone, to the left and right of the main joystick, it’s set to open various URLs at www.o2.ie. These buttons are prime UI real estate, and easily accidentally hit; I don’t want to go online (and possibly incur a charge) if they’re pressed.

Easily fixed. Open the menu, then select Settings -> Connectivity -> Internet settings -> Internet Profiles. Select O2 WAP and hit More -> Advanced, then Change homepage and enter “file:///” under Address and hit Save. It’ll now issue an ugly warning if you press those buttons, but at least it won’t go online. (It’d be nice to get a nicer fix for this.)

I’m sure there’s plenty more; if you’ve got this phone and have any tips to share, feel free to drop a comment below.

In particular, I’d love to know how to further “de-O2ify” the UI; the top 3 buttons on the menu screen are taken up with worthless operator spam (“O2 Music Store”, “O2 Menu” and “Entertainment”, all of which go to various URLs at www.o2.ie), while the useful Applications and Alarm screens, which I use all the time, are hidden in a submenu. ugh.

Investing in real estate

Published November 21, 2007

Screen real estate, that is — 3600×1050 pixels of it:

(That’s a Samsung SyncMaster 225bw226bw connected to a Thinkpad T61p running Ubuntu Gutsy, if you’re curious.)