Skip to content

Category: Uncategorized

Links for 2010-01-14

Published January 14, 2010

Links for 2010-01-12

Published January 12, 2010

Links for 2010-01-11

Published January 11, 2010

Links for 2010-01-06

Published January 6, 2010

  • Una "UnaRocks" Mullally on the state of Irish blogs : 'I think that ‘first wave’ of Irish blogging was over a long time ago, probably around the time Blogorrah hit the dirt, but in spite of time and an increase of participants and bigger audience there seems to be no real drive to improve content. People will always read something good – online or offline – and until that something good (hopefully in plural) starts to emerge and while good bloggers log off indefinitely, Irish blogging, for what it’s worth, is in a state of disarray.'
    (tags: irish irishblogs ireland writing blogosphere blogging unarocks)

SAY2K10 Doh

Published January 4, 2010

Happy new year! Or maybe not. Doh.

Over a year ago, Lee Maguire noticed that a contributed SpamAssassin rule, FH_DATE_PAST_20XX, was naively written -- simply to match any date in the year 2010 or later -- and would start to false-positive on all mail in 14 months. We made the trivial fix to avoid this (for at least 10 years, by which point the rule would have obsoleted itself through normal means), and I committed it to SVN.

Problem solved, right? Nope. I'd committed to trunk, but in a moment of inattention had forgotten to backport the fix to the stable release branch, 3.2.x, as well. Nobody else noticed the mistake, and several months later, boom:

Bugger.

Annoyingly, the GA had assigned this rule 3.5 points in the 3.2.0 rescoring run. This meant that the effective default threshold had been lowered from 5.0 points to 1.5, which produced a 2% false positive rate during the first 13 hours of the new year.

After that point, the fix was pushed to the sa-update channel, and anyone who runs sa-update regularly (as they should!) was brought back to normal filtering behaviour.

The rule is superfluous anyway, since it overlaps with a better-written "eval" rule, DATE_IN_FUTURE_96_XX. Accordingly, most likely scenario is that it'll be removed.

Personally, I see a few lessons from this:

  • Obviously, I need to pay more attention. This is easier said than done though, since SpamAssassin has nothing to do with my day job anymore; it's a spare-time thing nowadays, and that's a rare resource, unfortunately. :( But still, a chastening result, and I'm very sorry for my part in this screwup.

  • We need more active committers on Apache SpamAssassin. If we'd had more eyes, the fact that I'd forgotten to backport the fix might have been spotted. we're definitely in a better situation now in this regard than we were 6 months ago, so that's good.

  • IMO, this is a good demonstration of how too many simple rules are risky; without careful vetting and moderation, it's easy for a bad one to slip past. Perhaps we need to move more towards a DNSBL/network-rule driven approach, although this has its downsides too. Still thinking about this.

  • It'd be good to fix the GA so that it wouldn't assign such high points to simple rules like this, without some indication that a human has vetted them and believes them trustworthy.

Daryl posted a good comment on /.:

Clearly we dropped the ball on this one. As far as I know it's our first big rule screw up in the project's 10 years. If you're going to screw up you might as well do it well.

+1 to that!

And to everyone who had to clean up the fallout and spend a holiday recovering lost mails from spam folders... sorry :(

Sup Rocks

Published December 18, 2009

For the past 2 years or so, I've been using GMail to handle my main mail feed for jmason.org. I'm an absolute convert to its "river of threads"/search-based workflow.

Since starting at Amazon, I've had to start dealing with a heavy volume of work mail. Previously jobs have either had low mail volumes, or used Google Apps hosting for their mail, but Amazon's volumes are high and -- obviously -- they're not using Google. ;) For a while, I tried using Thunderbird, but it just didn't really cut it; I could never keep track of mails I wanted archived, or remember which folder they were in, etc. -- the same old problems that GMail solved.

Enter Sup. It's a console-based *nix email client, with a Mutt-like curses interface, which offers something closely approximating the GMail experience:


Sup is a console-based email client for people with a lot of email. It supports tagging, very fast full-text search, automatic contact-list management, custom code insertion via a hook system, and more. If you're the type of person who treats email as an extension of your long-term memory, Sup is for you.

Inbox Zero is a daily occurrence for my work email now; I can simply archive pretty much everything, and reliably know the excellent full-text search support will allow me to find it again in an instant when I need it. The new-user guide is well worth a read to get an idea of its featureset and UI.

Setting it up

The process of getting it set up is quite hairy; here are some instructions for Ubuntu, which thoroughly failed to work for me on 9.04. I had a similarly tricky time using some Ruby packages on the Red Hat work desktop, but eventually avoided it by just building vanilla Ruby from source, then using that to install "gem" and from that, "sudo gem install sup". Much easier...

Next step is to get the mail. From some reading, it appears the most reliable way to deal with a MS Exchange 2007 server is to use offlineimap to sync it to a local set of maildirs, then add those as Sup "sources" using sup-add, one by one. This is very well supported in Sup, and works well. Offlineimap is very easy to install on Ubuntu, and can easily be built from source if that's not an option. My config is pretty much a vanilla copy of the minimal config.

There's a good Sup hook to run "offlineimap" every poll interval, and rescan synced sources that contain new mail. It works well.

Sup has an interesting approach to mail storage -- it doesn't. Instead, it stores pointers to the messages' locations in their source storage. This is a great idea, since bugs in Sup therefore cannot lose your mail -- just your metadata about your mail. However, it means that if the source changes in a way which moves or removes messages, you need to tell Sup to rescan (using "sup-sync"), but that's no big deal in practice; in the more usual case, if new mail arrives, it's automatically rescanned.

I have just under 7000 mail messages in my Sup index, and rescans are speedy and searches super-fast. It's very nicely done.

Outbound mail is delivered using /usr/sbin/sendmail by default, which should be working on any decent *nix desktop anyway ;)

Recommended Hooks

The Hooks wiki page has a few good hooks that you should install:

  • ~/.sup/hooks/before-poll.rb: the above-mentioned offlineimap poll hook
  • ~/.sup/hooks/mime-decode.rb: 'uses w3m to translate all HTML attachments that don't have a text/html alternative.' Well worth installing.
  • ~/.sup/hooks/before-add-message.rb: essential to filter out cron noise and the like so it doesn't hit the inbox; unfortunately Sup doesn't (yet) support GMail's "filter messages like this" UI.

Bad Points

  • Long URIs: unfortunately, very long URIs are broken by Sup's renderer, and it doesn't offer a native way to "activate" URIs and have them displayed in the browser; instead one has to cut and paste them. This is pretty lame. I've hacked up a perl script that will reconstruct the full URLs from the broken rendering, when the text is piped to it, but that's a horrible hack.

  • Index Corruption: I've had the misfortune (once, in the month since I started) of corrupting my search index, causing Ruby exception stack traces when I attempted to run "sup-sync" to scan new mail. The only fix appeared to be to restore my index from a "sup-dump" backup. Thankfully all seems fine now, but it was a definite reminder of the product's beta status.

  • Calendaring: still as painful as it's ever been with UNIX command line email.

  • HTML: A good-quality, email-oriented, native HTML renderer would be awesome.

  • MIME: Sup again takes the traditional approach from UNIX command line clients of delegating to the mailcap file and its rules; unfortunately my RHEL5 desktop is too crappy to have a good mailcap setup. So I've had to write this from scratch to deal with the usual .docs and .xls's etc., flying about.

  • Inconsistent Key Mapping: Given that it shares so much UI with GMail in other respects, it's a little annoying that Sup doesn't have the same key mapping. Not a big deal, as it took only a couple of hours to get the hang of Sup's, though.

Overall

If you're happy enough to spend a day or two getting the damn thing installed, and aren't afraid of a little dalliance with the bleeding edge, I strongly recommend it. It's definitely the best *NIX mail reader at the moment.

Links for 2009-12-15

Published December 15, 2009

Links for 2009-12-09

Published December 9, 2009

Links for 2009-11-25

Published November 25, 2009

Links for 2009-11-17

Published November 17, 2009

Links for 2009-11-14

Published November 14, 2009

Met iPhone

Published November 12, 2009

Irish iPhone users -- you may find this useful. I've written a web scraper which takes a couple of the more useful pages on Met Eireann's website -- the regional forecast and the rainfall radar page -- and reformats them in an iPhone-optimised style. Enjoy:

(updated: supports all the provincial forecasts now)

Links for 2009-11-08

Published November 8, 2009

Links for 2009-11-03

Published November 3, 2009

Links for 2009-10-30

Published October 30, 2009

Links for 2009-10-23

Published October 23, 2009

Links for 2009-10-22

Published October 22, 2009

Links for 2009-10-21

Published October 21, 2009

Links for 2009-10-19

Published October 19, 2009

Links for 2009-10-02

Published October 2, 2009

Lest we forget

Published October 2, 2009

Regarding Google Wave's similarity to Lotus Notes, which is a meme I've heard from several angles -- David Jones hits the nail on the head:

Well, I used Notes from 1994 to 1999. It did have a database backend for e-mail and a rich collaborative editing model. But it didn't have realtime shared editing, or instant annotation.

And it was shit. No-one in their right minds would have wanted the future of the web to have been Notes. Even though, and I completely agree, it did things that the web is now only just getting round to.

+1 to that!

Links for 2009-09-30

Published September 30, 2009

Links for 2009-09-29

Published September 29, 2009

Links for 2009-09-11

Published September 11, 2009

n+30 Days

Published September 2, 2009

Colm's "n+1" post reminded me that I'd forgotten to write about this.

On July 27th, I started at Amazon, in a new Dublin-based software dev team working on infrastructure automation. It's now (just over) a month later, and I'm enjoying it immensely.

Needless to say, this company does some very interesting web-scale technology, and getting to look inside the AWS sausage factory is really enjoyable, believe it or not ;)

(I should also post a pic of my glorious screen real-estate. The hardware is a massive improvement over the previous gig, thankfully.)

Unfortunately, however, this has coincided with a lack of free time to blog and keep up with interweb-based leisure pursuits, including SpamAssassin. Really though, this is more due to looking after two wonderful little girls under 2 years of age, rather than the job -- but still, I need to remedy my neglect of this site...

In SpamAssassin news: we've been putting out some alpha releases of 3.3.0, and are planning to do a mass-check for score-generation in the next couple of days. Hopefully we can drive 3.3.0 to a GA release in a few weeks.

Also -- we're still looking for more people in the Amazon team, and hiring aggressively. If you're looking for an interesting software dev role in Dublin, get in touch!

PS: it was Bea's second birthday last weekend. Check out the awesome Very Hungry Caterpillar cupcake cake made by the missus for the occasion:

Links for 2009-09-02

Published September 2, 2009

Links for 2009-09-01

Published September 1, 2009

Links for 2009-08-27

Published August 27, 2009

Links for 2009-08-25

Published August 25, 2009

Links for 2009-08-21

Published August 21, 2009

Links for 2009-08-13

Published August 13, 2009

Links for 2009-08-11

Published August 11, 2009

Links for 2009-08-09

Published August 9, 2009

Links for 2009-08-06

Published August 6, 2009

Links for 2009-08-01

Published August 1, 2009

Links for 2009-07-31

Published July 31, 2009

Links for 2009-07-23

Published July 23, 2009

Embedded software development

Published July 23, 2009

Found in an Ivan Krstic post about Sugar and the OLPC:

In truth, the XO ships a pretty shitty operating system, and this fact has very little to do with Sugar the GUI. It has a lot to do with the choice of incompetent hardware vendors that provided half-assedly built, unsupported and unsupportable components with broken closed-source firmware blobs that OLPC could neither examine nor fix. [...]

We had an embedded controller that blocks keyboard events and stops machine suspend, and to which we -- after a long battle -- received the source, under strict NDA, only to find a jungle of nested if statements, twelve levels deep, and no code history. (The company that wrote the code doesn't use version control, see. They put dates into code comments when they make changes, and the developers mail each other zip files with new versions.)

Haha. Been there, done that. Sometimes it's great not to have to work with custom hardware anymore...

Links for 2009-07-22

Published July 22, 2009

Links for 2009-07-21

Published July 21, 2009

YA link-blog aggregator

Published July 20, 2009

Alex Payne writing about "Fever", a new link-blog aggregator app:

Fever's proposition is straightforward: supply it with the feeds you always want to read, and supplement those with feeds that you only want to read the juicy bits of. Fever will then show you a sort of personal Techmeme or Google News, pulling together stories that reference common URLs.

Fever is commercial software, costing $30. Alternatively, I've been doing something very similar for the past few years using SpicyLinks, which is free (if a great deal less pretty on the UI end).

It's nice to see the idea getting some polish, though. ;)

Alex does raise an interesting point towards the end:

Fever is just fine for floating good techie content to the top, but poor for most any other subject. I'd love it if Fever could find me good posts from the set of minimal techno or cocktail blogs I subscribe to, but link blogs -- and, indeed, linking outside one's own site -- just aren't as prevalent in those communities.

True.

Links for 2009-07-17

Published July 17, 2009

Links for 2009-07-16

Published July 16, 2009

Links for 2009-07-15

Published July 15, 2009

Eircom’s “DDOS”, or not

Published July 15, 2009

I woke up this morning to hear speculation on RTE Radio as to how Eircom's DDOS woes were possibly being caused by the Russian mob, of all things. This absurd speculation is not helped by lines in statements like this:

'The company blamed the problems on "an unusual and irregular volume of internet traffic" directed at its website, which affected the systems and servers that provide access to the internet for its customers.'

I'm speculating, too, but it seems a lot more likely to me that this isn't just a DDOS, and someone -- possibly just a lone Irish teenager -- is running an attempted DNS cache-poisoning attack. Here's why.

Last week, there were two features of the attack in reports: DDOS levels of traffic and incorrect pages coming up for some popular websites. To operate a Kaminsky DNS cache-poisoning attack requires buckets of packets -- easily perceivable as DDOS levels. This level of traffic would be the first noticeable symptom on Eircom's network management consoles, so it'd be easy to jump to the conclusion that a simple DDOS attack was the root cause.

This week, there's just the DDOS levels of traffic. No cache poisoning effects have been reported. This would be consistent with Eircom's engineers getting the finger out over the weekend, and upgrading the NSes to a non-vulnerable version. ;)

Once the attacker(s) realise this, they'll probably stop the attack.

It's not even a good attack for a bad guy to make, by the way. Given the timing, right after major press about a North Korean DDOS on US servers. it's extremely high-profile, and made the news in several national newspapers (albeit in rather inept fashion). If someone wanted to make money from an attack, a massive-scale packet flood indistinguishable from a DDOS against the nation's largest ISP is not exactly a subtle way to do it.

In the meantime, apparently OpenDNS have really seen the effects, with mass switchover of Eircom's customers to the OpenDNS resolvers. Probably just as well...

Links for 2009-07-14

Published July 14, 2009

Links for 2009-07-12

Published July 12, 2009

I’m a Dermotologist!

Published July 11, 2009

Found here:

On Wednesday 20 May 2009, speaking at a parliamentary Justice Committee debating his new blasphemy law, Dermot Ahern joked that people were making blasphemous comments about him, and he compared his own purity to that of the baby Jesus.

So we have a Justice Minister joking about himself being blasphemed, at a parliamentary Justice Committee discussing his own blasphemy law, that could make his own jokes illegal.

In honour of this Ministerial revelation, we have founded the Church of Dermotology. We believe God sent Dermot Ahern to save Ireland from rational thinking. Our sacred symbol is the Star of Dermot.

Our sacred beliefs are quite similar to those of other religions.

  • We believe ice cream wafers are literally the body of Dermot Ahern.
  • We believe Dermot Ahern created the universe on Wed 20 may 2009.
  • We’re sometimes not sure whether Dermot Ahern really exists.
  • We believe it is blasphemous to publish an image of Dermot Ahern.
  • We refuse to gather sticks on the Sabbath, which is Wednesday.
  • We wear magic underpants that protect us from fire and bullets.
  • We are outraged whenever anybody insults our sacred beliefs.
  • We fervently support Dermot Ahern’s proposed blasphemy law.
  • If it is passed, we will be regularly outraged, and will take test cases.

Like Scientologists, Dermotologists offer a free personality test. Question one: are you vulnerable? Question two: have you money? If you answer yes to either of these questions, you’re in.

After you join, check out the campaign against the Irish blasphemy law at blasphemy.ie.

Health and Safety

Published July 10, 2009

A while back a friend of mine mailed us all with this classic of overweening health-and-safety bureaucrats gone wild:

The company are now installing wallpaper on our PCs with their 5 golden safety rules:

  1. Always hold the handrail

  2. Always reverse park

  3. Assess Risks

  4. Accept Challenges

  5. Wear PPE [Personal Protective Equipment] gear

We also have to drink from metal cups with plastic lids on them.

The thing that really got me was #2 -- 'always reverse park'. Apparently, someone decided that reversing into the parking space was safer than going in head-first, and to such a significant degree that it was worth mandating it across a medium-sized company. On the other hand, another friend noted:

The college i went to [in the US] would ticket you if you backed into a parking space -- they said it was a "fire hazard".

so we've got "fire hazard" in one direction and "unsafe" in the other. Parse that.

Another friend was told that she couldn't bring her folding bike in the lift because "what would happen if the president was in the lift going to the board room?". She says "I could not work out the health and safety implications."

What health and safety insanity have you encountered recently?