Author: Justin

I've just been setting up a new Macbook, running MacOS Sequoia, and my previous trick I'd used to handle an ANSI keyboard in an Irish/UK English locale, specifically to remap shift-3 from "£" to "#", no longer works. So here's a replacement approach, using a Karabiner-Elements "Complex Modification" rule:

{
    "description": "Change right-shift-3 to hash",
    "manipulators": [
        {
            "from": {
                "key_code": "3",
                "modifiers": { "mandatory": ["right_shift"] }
            },
            "to": [
                {
                    "key_code": "3",
                    "modifiers": ["right_option"]
                }
            ],
            "type": "basic"
        }
    ]
}

My Self-Hosted GMail Backup

For the past few months, I’ve had a bit of a background project going to ensure that my cloud-hosted personal data is safely archived on my own, self-hosted hardware, just in case. Google services are nice 'n' all, but I’m not 100% happy trusting them with everything in the long run.

Part of this project has been to archive my old email collection from GMail, which dates back to the initial public beta in 2004(ish?) -- and make it searchable, because what’s the point in having all that email if you can’t find the needle in the 20-year haystack when you need it?

Enter “notmuch” -- a “fast, global-search and tag-based email system”, which runs as a set of UNIX CLI commands, and is inspired by Sup, a mailreader I used previously. I have a self-hosted home server running Ubuntu 20.04 with a chunky SATA disk, so that's where I'll run it.

Here’s the process I followed:

Order a Google Takeout of your GMail account. This takes a couple of days to prepare. Request the 50GB tgz files.

When you get the email telling you it’s ready, download the files (this is awkward as you can only download one at a time, and only via your web browser, not fun). scp them to your server, and to a disk with lots of free space (/x/4 in my case).

Extract each one:

cd /x/4/tmp
tar xvfz takeout-20250322T145242Z-001.tgz
tar xvfz takeout-20250322T145242Z-002.tgz
...
rm takeout-20250322T145242Z-00*tgz

You will wind up with a few bits of uninteresting metadata, and one gigantic mbox file: Takeout/Mail/All\ mail\ Including\ Spam\ and\ Trash.mbox . In order to make this useful, it needs to be converted into Maildir format, so install “mb2md”:

sudo apt install mb2md

Now run it, creating a GMailTakeout directory for the result:

mkdir -p /x/4/GMailTakeout
mb2md -s /x/4/tmp/Takeout/Mail/All\ mail\ Including\ Spam\ and\ Trash.mbox -d /x/4/GMailTakeout

This takes quite a while for 20 years of email! Unfortunately, the resulting single directory is still unusably huge, so split it into 100 new Maildir folders:

cd /x/4/GMailTakeout/cur
find . -type f -print > /tmp/dirlisting
perl -ne '
  $dir = sprintf("dir_%03d", ($. % 100));
  (-d $dir) or mkdir($dir);
  chop; rename($_, "$dir/$_") or die "cannot rename $_";
' /tmp/dirlisting

cd /x/4/GMailTakeout
mv cur/* .
for f in dir_* ; do mkdir mail$f mail$f/{new,tmp} ; mv $f mail$f/cur ; done

The result of this is 100 Maildirs, /x/4/GMailTakeout/maildir_000 to /x/4/GMailTakeout/maildir_099, each containing about 300MB of email, in my case.

There really isn't much need to keep the mails labelled as spam, so let's just nuke them in advance:

grep -r 'X-Gmail-Labels: Spam' . | perl -pe 's/:.*$//' | xargs -n 100 rm -f

Next step is to install “notmuch” and create a “notmuch” configuration. I used the Debian packaged “notmuch”, version 0.29.3. Install using apt-get, and then run “notmuch”. Accept the defaults for the config, and don’t add any mail folders yet.

My initial attempt was simply to import the lot in one go: this went badly, throwing up a multi-day progress indicator, and with no safe way to checkpoint partial progress, and it quickly started consuming lots of RAM, causing me to suspect some leaking.

I aborted it and tried this instead to index each dir one-by-one:

for f in /x/4/GMailTakeout/maildir_* ; do ln -s $f ~/mail/ && nice notmuch new ; done

Unfortunately, this also turned out badly. The import of each maildir gradually slowed as data built up in notmuch’s Xapian indexes. After processing about 60 maildirs, memory consumption during the import became a problem, and the “notmuch” processes started being killed by the Linux OOM killer. In a couple of cases this resulted in corrupt index files and data loss. Ouch.

So I started again, with a new approach:

#!/bin/sh
set -exu
mkdir -p /x/4/GMailTakeout/notmuchbackup/xapian/
for f in /x/4/GMailTakeout/maildir_0*
do
    ln -s $f ~/mail/ && nice notmuch new
    nice notmuch compact
    cp /home/jm/mail/.notmuch/xapian/* /x/4/GMailTakeout/notmuchbackup/xapian/
done

Calling “notmuch compact” does seem to help, trimming the size of the indexes as it goes; taking a copy of the Xapian indexes in a backup dir is for extra safety. Since the “-e” shell flag is in place, any OOMs or other random failures will crash the entire script and ensure the last backup is still safe to use for recovery.

Unfortunately this still got bogged down and started OOMing fairly reliably after about maildir_065, 2 days into the process; at this point, I decided to keep that set of dirs as “notmuch config 1” and start a separate import process, into another index, as “notmuch config 2”. Accordingly, I moved ~/mail to ~/mail1 , ~/.notmuch-config to ~/.notmuch-config1, created a ~/mail2 , and started a new notmuch config file pointing at that instead. Ideally I’ll be able to merge the indexes at some point, but it’s no biggie.

With these two aliases, it’s pretty painless:

alias notmuch1='notmuch --config=$HOME/.notmuch-config1'
alias notmuch2='notmuch --config=$HOME/.notmuch-config2'

After another day or so of indexing, this is the result --

du -sh /home/jm/mail?/.notmuch/xapian
19G     /home/jm/mail1/.notmuch/xapian
4.2G    /home/jm/mail2/.notmuch/xapian

Notmuch supports pretty much all the nice email search features that GMail does, but seemingly more reliably, and faster; I’ve already been able to use this new mail index to find a mail that (worryingly!) GMail's own search can’t seem to locate -- my license for the Moom OSX window manager tool purchased over a decade ago:

time notmuch1 search moom "Many Tricks"
thread:00000000000034fe   2013-10-15 [1/1] Many Tricks; Your Many Tricks purchase (inbox unread)
thread:00000000000c267b   2013-10-15 [1/1] sales@manytricks.com; Your Moom License (attachment inbox unread)

real    0m0.068s
user    0m0.048s
sys     0m0.016s

And it’s just nice to have 20 years of email archived safely, off the cloud, and indexed.

Next steps? Maybe lieer would be good to try, to download incremental updates as we go forward. Let's see.

My Solar PV Output For 2024

A couple of years ago, I had 5.8kW of solar panels and a 5kW battery installed on my (fairly typical) Dublin house.

The tricky part with solar PV is that, while you may have a set of solar panels, these may not be generating electricity at the time you want to use it. Even with a battery, your available stored power may wind up fully discharged by 7pm, leaving you running from expensive, non-renewable grid power for the rest of the evening. And up here in the high latitudes of northern Europe, you just don't get a whole lot of solar energy in December and January.

2024 was the first year in which (a) my panels and battery were fully up and running, (b) we were using a day/peak/night rate for grid electricity, and (c) for much of the year I had load-shifting in place; in other words, charging the battery from cheap night-rate electricity, then discharging it gradually over the course of the day, topping up with solar power once the sun gets high enough. As such, it’s worth doing the sums for the entire year to see how effective it’s been in real-world usage terms.

The total solar power generated across the year was reported from my Solis inverter as 4119 kWh.

Over the course of 2024, the entire household consumption comes to 8628 kWh. This was comprised of a fairly constant 800ish kWh per month, across the year; we still have gas-fired heating, so the winter months generally use gas energy instead of scaling up our electricity consumption.

Of that, the power consumed from solar PV was 2653 kWh (reported from the Solis web app as “annual PV to consumption”), and that from the grid was 5975 kWh (reported by the ESB Networks data feed).

So the correct figure is that 30% of our household consumption was driven from solar. This is a big difference from the naive figure of 4119/8628 = 47%; you can see that a big chunk of that power is being “lost”, due to happening at the wrong time to provide household power.

Of course, that power isn’t really “lost” -- it was exported to the grid instead. This export comprised 1403 kWh; this occurred when the battery was full, the household power usage was low, but there was still plenty of solar power being generated. (Arguably a bigger battery would be worthwhile to capture this, but at least we get paid for this export.)

There was a 2%-4% discrepancy between the Solis data and that from ESB Networks; Solis reported higher consumption (6102 kWh vs 5975) and higher export (1465 kWh vs 1403). I’m liable to believe ESB Networks more though.

In monetary terms:

The household consumption was 8628 kWh. Had we consumed this with the normal 24-hour rate tariff, we'd have paid (€236.62 standing charge per year) + (8628 at 23.61 cents per kWh) = (236.62 + 8628 * 0.2361) = €2273.69.

Checking the bills received over the year, taking into account load-shifting to take advantage of day/night variable rates and the power generated by the panels, and discounting the one-off government bill credits -- we spent €1325.97 -- 58.2% of the non-solar price.

Here! Have some graphs:

Ridding My Home Network of IP Addresses

(Republishing this one on the blog, instead of just as a gist)

Recent changes in the tech scene have made it clear that relying on commercial companies to provide services I rely on isn't a good strategy in the long term, and given that Tailscale is so effective these days as a remote-access system, I've gradually been expanding a small collection of self-hosted web apps and services running on my home network.

Until now they've mainly been addressed using their IP addresses and random high ports on the internal LAN, for example:

1. Pihole: http://10.19.72.7/admin
2. Home Assistant: http://10.19.72.11:8123/
3. Linkding: http://10.19.72.6:9092/
4. Grafana: http://10.19.72.6:3000/
5. (plus a good few others)

Needless to say this is a bit messy and inelegant, so I've been planning to sort it out for a while. My requirements:

1. no more ugly bare IP addresses!
2. a DNS domain;
3. with HTTPS URLs;
4. one per service;
5. no visible port numbers;
6. fully valid TLS certs, no having to click through warnings or install funny CA certs;
7. accessible regardless of which DNS server is in use -- ie. using public DNS records. This may seem slightly unusual, but it's useful so that the internal services can still be accessed when I'm using my work VPN (which forces its own DNS servers);
8. accessible internally;
9. accessible externally, over Tailscale;
10. not accessible externally without Tailscale.

After a few false starts, I'm pretty happy with the current setup, which uses Caddy.

Hosting The Domain At Cloudflare

First off, since the service URLs are not to be accessible externally without Tailscale active, the HTTP challenge approach to provision Let's Encrypt certs cannot be used. That would require an open-to-the-internet publicly-accessible HTTP server on my home network, which I absolutely want to avoid.

In order to use the ACME DNS challenge instead, I set up my public domain "taint.org" to use Cloudflare as the authoritative DNS server (in Cloudflare terms, "full setup"). This lets Caddy edit the DNS records via the Cloudflare API to handle the ACME challenge process.

One of the internal hosts is needed to run the Caddy server's reverse proxies; I picked "hass", 10.19.72.11, the Home Assistant host, which didn't have anything already running on port 80 or port 443. (All of my internal hosts are running on a private /24 IP range, at 10.19.72.0/24.)

The dedicated DNS domain I'm using for my home services is "home.taint.org". In order to use this, I clicked through to the Cloudflare admin panel and created a DNS record as follows:

Type   Name      Content             Proxy Status               TTL
A      *.home    10.19.72.11         DNS only - reserved IP     Auto

Now, any hostnames under "home.taint.org" will return the IP 10.19.72.11 (where Caddy will run).

I don't particularly care about exposing my internal home network IPs to the world, as a trade-off to allow the URLs to work even if an internal host is using the work VPN, or resolving with 8.8.8.8, or whatever. That's worth missing out on a little bit of paranoia, since the IPs won't be accessible from outside without Tailscale anyway.

It is worth noting that the Cloudflare-hosted domain doesn't have to be the same one used for URLs in the home network; using dns_challenge_override_domain you can delegate the ACME challenge from any "home" domain to one which is hosted in Cloudflare.

The Caddy Setup

One wrinkle is that I had to generate a custom Caddy build in order to get the "dns.providers.cloudflare" non-standard module, from https://caddyserver.com/download . This is a click-and-download page which generates a custom Caddy binary on the fly. It would have been nicer if the Cloudflare module was standard, but hey.

Once that's installed, I can get this output:

$ /usr/local/bin/caddy list-modules
[long list of standard modules omitted]

dns.providers.cloudflare
dns.providers.route53

  Non-standard modules: 2

  Unknown modules: 0

(Yes, I have Caddy running as a normal service, not as a Docker container. No particular reason; I think Docker should work fine.)

Go to the Cloudflare account dashboard, and create a user API token as described at https://developers.cloudflare.com/fundamentals/api/get-started/create-token/ . In my case, it has Zone / DNS / Edit permission, on the specific zone taint.org.

Copy that token as it's needed in the "Caddyfile", which now looks like the following:

hass.home.taint.org {
        tls {
                dns cloudflare cloudflare_api_token_goes_here
        }
        reverse_proxy /* 10.19.72.11:8123
}

links.home.taint.org {
        tls {
                dns cloudflare cloudflare_api_token_goes_here
        }
        reverse_proxy /* 10.19.72.6:9092
}

pi.home.taint.org {
        tls {
                dns cloudflare cloudflare_api_token_goes_here
        }
        redir / /admin/
        reverse_proxy /admin/* 10.19.72.7:80
}

grafana.home.taint.org {
        tls {
                dns cloudflare cloudflare_api_token_goes_here
        }
        reverse_proxy /* 10.19.72.6:3000
}

[many other services omitted]

Running sudo caddy run in the same dir will start up and verbosely log what it's doing. (Once you're happy enough, you can get Caddy running in the normal systemd service way.)

After setting those up, I now have my services accessible locally as:

1. Home Assistant: https://hass.home.taint.org/
2. Pihole: https://pi.home.taint.org/
3. Grafana: https://grafana.home.taint.org/
4. Linkding: https://links.home.taint.org/

Caddy seamlessly goes off and configures fully valid TLS certs with no fuss. I found it much tidier than Certbot, or Nginx Proxy Manager.

The Tailscale Setup

So this has now sorted out all of the requirements bar one:

9. accessible externally, over Tailscale.

To do this I had to log into Tailscale's admin console and go to https://login.tailscale.com/admin/machines , pick a host on the 10.19.72/24 internal LAN, click it's dropdown menu and "Edit Route Settings...", and enable a Subnet Route for 10.19.72/24. By doing this, all of the service.home.taint.org DNS records are now accessible, remotely, once Tailscale is enabled; I don't even need to use ts.net names to access them! Perfect.

Anyway, that's the setup -- hopefully this writeup will help others. And kudos to Caddy, Let's Encrypt and Tailscale for making this relatively easy.

For the past several years, since the demise of Google Reader, I’ve been augmenting the RSS/Atom syndication of this linkblog with posts to various social media platforms using bot accounts. This is kind of a form of POSSE -- “Publish (on your) Own Site, Syndicate Elsewhere” (ideally I’d be self-hosting Pinboard to qualify for that I guess).

The destination for cross-posts were first to Twitter (RIP), and more recently to Mastodon via botsin.space. With the shutdown of that instance, I’ve had to make a few changes to my syndication script which gateways the contents to Mastodon, and I also took the opportunity to set up a BlueSky gateway at the same time. On the prompting of @kellan, here’s a quick write-up of where it all currently stands…

Primary Source: Pinboard

The primary source for the blog’s contents is my long-suffering account at https://pinboard.in/u:jm/, where I have been collecting links since 2009 (and before that, del.icio.us since I think 2004?, so that’s 20 years of links by now).

Pinboard has a pretty simple UI for link collection using a bookmarklet, which I’ve improved a tiny bit to open a large editor textbox instead of the default tiny one.

The resulting posts generally tend to include a blockquote, a short lede, and a few tags in the normal Pinboard/Del.icio.us style.

I find editing text posts in the Pinboard bare-bones UI to be easier and more pleasant than WordPress, so I generally use that as the primary source. Based on the POSSE principle, I should really figure out a way to get this onto something self-hosted, but Pinboard works for me (at the moment at least).

Publish from Pinboard to Blog

I use a Python script run from cron, to gateway new bookmarks from https://pinboard.in/u:jm/ as individual posts, formatted with Markdown, to this blog using the WordPress posting API: Github repo

Publish from Pinboard to Mastodon

This reads the Pinboard RSS feed for https://pinboard.in/u:jm/ and posts any new URLs (and the first 500 chars of its description) to the “jmason_links” account at mstdn.social: Github repo

Migration from the old Mastodon account at botsin.space to mstdn.social was really quite easy; after manually setting up the new account at mstdn.social and copying over the bio text, I hit the "Move from a different account" page, and entered @jm_links@botsin.space for the handle of the old account to migrate from.

I then logged in to the old account on botsin.space and hit the "Move to a different account" page, entering @jmason_links@mstdn.social for the handle to migrate to. This triggered copying of the followers from one account to the other, and left the old account dormant with a link to the new location instead.

(One thing to watch out for is that once the move is triggered, the profile for the old account becomes read-only; I've since had to temporarily undo the "moved" status in order to update the profile text, which was a bit messy.)

Publish from Pinboard to BlueSky

This reads the same Pinboard RSS feed as the Mastodon gateway, and gateways new posts from there to the “jmason.ie” account at BlueSky. This is slightly more involved than the Mastodon script, as it attempts to generate an embed card and mark up any links in the post appropriately: Github repo

I have a cron on my home server which runs those Mastodon and BlueSky gateway scripts every 15 minutes, and that seems to be a reasonable cadence without hammering the various APIs too much.

Over the past decade or so, I've been suffering with chronic migraine, sometimes with multiple attacks per week. It's been a curse -- not only do you have to suffer the periodic migraine attacks, but also the "prodrome", where unpleasant symptoms like brain fog and an inability to concentrate can impact you.

After a long process of getting a referral to the appropriate headache clinic, and eliminating other possible medications, I finally got approved to receive Ajovy (fremanezumab), one of the new generation of CGRP inhibitor monoclonals -- these work by blocking the action of a peptide on receptors in your brain. I started the course of these a month ago.

The results have, frankly, been amazing. As I hoped, the migraine episodes have reduced in frequency, and in impact; they are now milder. But on top of that, I hadn't realised just how much impact the migraine "prodrome" had been having on my day-to-day life. I now have more ability to concentrate, without it causing a headache or brain fog; I have more energy and am less exhausted on a day-to-day basis; judging by my CPAP metrics, I'm even sleeping better. It is a radical improvement. After 10 years I'd forgotten what it was like to be able to concentrate for prolonged periods!

They are so effective that the American Headache Society is now recommending them as a first-line option for migraine prevention, ahead of almost all other treatments.

If you're a migraine sufferer, this is a game changer. I'm delighted. It seems there may even be further options of concomitant treatment with other CGRP-targeting medications in the future, to improve matters further.

More papers on the topic: a real-world study on CGRP inhibitor effectiveness after 6 months; no "wearing-off" effect is expected.

About a year ago, I installed a solar PV system at my home. I wound up with a set of 14 panels on my roof, which can produce a max of 5.6 kilowatts output, and a 4.8 kW Dyness battery to store any excess power.

Since my car is an EV, I already had a home car charger installed, but chose to upgrade this to a MyEnergi Zappi at the same time, as the Zappi has some good features to charge from solar power only -- and part of that feature set involved adding a Harvi power monitor.

With HomeAssistant, I’ve been able to extract metrics from both the MyEnergi components and the Solis inverter for the solar PV system, and can publish those from HomeAssistant to my Graphite store, where my home Grafana can access them -- and I can thoroughly nerd out on building an optimal dashboard.

I’ve gone through a couple of iterations, and here’s the current top-line dashboard graph which I’m quite happy with...

Let’s go through the components to explain it. First off, the grid power:

Grid Import sans Charging

This is power drawn from the grid, instead of from the solar PV system. Ideally, this is minimised, but generally after about 8pm at night the battery is exhausted, and the inverter switches to run the house’s power needs from the grid.

In this case, there are notable spikes just after midnight, where the EV charge is topped up by a scheduled charge on the Zappi, and then a couple of short duration load spikes of 2kW from some appliance or another over the course of the night.

(What isn’t visible on this graph is a longer spike of 2kW charging from 07:00 until about 08:40, when a scheduled charge on the Solis inverter charges the house batteries to 100%, in order to load shift -- I’m on the Energia Smart Data contract, which gives cheap power between 23:00 and 08:00. Since this is just a scheduled load shift, I’ve found it clearer to leave it off, hence “sans charging”.)

Solar Generation

This is the power generated by the panels; on this day, it peaked at 4kW (which isn’t bad for an Irish slightly sunny day in April).

To Battery From Solar

Power charged from the panels to the Dyness battery. As can be seen here, during the period from 06:50 to 09:10, the battery charged using virtually all of the panels’ power output. From then on, it periodically applied short spikes of up to 1kW, presumably to maintain optimal battery operation.

From Battery

Pretty much any time the batteries are not charging, they are discharging at a low rate. So even during the day time with high solar output, there’s a little bit of battery drain going on -- until 20:00 when the solar output has tailed off and the battery starts getting used up.

<

p>

Grid Export

This covers excess power, beyond what can be used directly by the house, or charged to the battery; the excess is exported back to the power grid, at the (currently) quite generous rate of 24 cents per kilowatt-hour.

Rendering

All usages of solar power (either from battery or directly from PV) are rendered as positive values, above the 0 axis line; usage of (expensive) grid power is represented as negative, below the line.

For clarity, a number of lines are stacked:

From Battery (orange) and Solar Generation (green) are stacked together, since those are two separate complementary power sources in the PV system.

Grid Export (blue) and To Battery From Solar (yellow) are also stacked together, since those are subsets of the (green) Solar Generation block.

The grafana dashboard JSON export is available here, if you're curious.

Quick plug for a good tool for self-hosting -- Cronitor.io. I have been using this for the past year or so as I migrate more of my personal stuff off cloud and back onto self-hosted setups, and it's been a really nice way to monitor simple cron-driven home workloads, and (together with graphite/grafana alerts) has saved my bacon many times. Integrates nicely with Slack, or even PagerDuty (although that would be overkill for my setup for sure).

I'm happy to announce that I'm now listed on TechArchives.Irish as one of the pioneers of the Irish web!

After extensive interviewing and collaboration with John Sterne, my testimony and timeline of those early days of the Irish web is now up at TechArchives.

It's been a good opportunity to reflect on the differences between the tech scene, then and now. I was very idealistic 30 years ago at the possibilities that the web and internet technologies had to offer; nowadays, I'm a bit more grizzled and pragmatic. But I still have hope -- particularly if we can apply this tech in a way that helps address climate change, in particular.... here's to the next 30 years!

Anyway, I hope writing this down helps record the history of those great early years of the web. Please take a look.

DynamoDB Local is one of the best features of AWS DynamoDB. It allows you to run a local instance of the data store, and is perfect for use in unit tests to validate correctness of your DynamoDB client code without calling out to the real service "in the cloud" and involving all sorts of authentication trickiness.

Unfortunately, if you're using one of the new MacBooks with M1 Apple silicon, you may run into trouble:

11:08:56.893 [DEBUG] [TestEventLogger]          DynamoDB > Feb 04, 2022 11:08:56 AM com.almworks.sqlite4java.Internal log
11:08:56.893 [DEBUG] [TestEventLogger]          DynamoDB > SEVERE: [sqlite] SQLiteQueue[]: error running job queue
11:08:56.893 [DEBUG] [TestEventLogger]          DynamoDB > com.almworks.sqlite4java.SQLiteException: [-91] cannot load library: java.lang.UnsatisfiedLinkError: /.../DynamoDBLocal_lib/libsqlite4java-osx.dylib: dlopen(/.../DynamoDBLocal_lib/libsqlite4java-osx.dylib, 0x0001): tried: '/.../DynamoDBLocal_lib/libsqlite4java-osx.dylib' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e')), '/usr/lib/libsqlite4java-osx.dylib' (no such file)
11:08:56.893 [DEBUG] [TestEventLogger]          DynamoDB >      at com.almworks.sqlite4java.SQLite.loadLibrary(SQLite.java:97)
11:08:56.893 [DEBUG] [TestEventLogger]          DynamoDB >      at com.almworks.sqlite4java.SQLiteConnection.open0(SQLiteConnection.java:1441)
11:08:56.893 [DEBUG] [TestEventLogger]          DynamoDB >      at com.almworks.sqlite4java.SQLiteConnection.open(SQLiteConnection.java:282)
11:08:56.894 [DEBUG] [TestEventLogger]          DynamoDB >      at com.almworks.sqlite4java.SQLiteConnection.open(SQLiteConnection.java:293)

It's possible to invoke it via Rosetta, Apple's qemu-based x86 emulation layer, like so:

arch -x86_64 /path/to/openjdk/bin/java dynamodb-local.jar

But if you don't have control over the invocation of the Java command, or just don't want to involve emulation, this is a bit hacky. Here's a better way to make it work.

First, download dynamodb_local_latest.tar.gz from the DynamoDB downloads page, and extract it.

The DynamoDBLocal_lib/libsqlite4java-osx.dylib file in this tarball is the problem. It's OSX x86 only, and will not run with an ARM64 JVM. However, the same lib is available for ARM64 in the libsqlite4java artifacts list, so this will work:

wget -O libsqlite4java-osx.dylib.arm64 'https://search.maven.org/remotecontent?filepath=io/github/ganadist/sqlite4java/libsqlite4java-osx-arm64/1.0.392/libsqlite4java-osx-arm64-1.0.392.dylib'
mv DynamoDBLocal_lib/libsqlite4java-osx.dylib libsqlite4java-osx.dylib.x86_64
lipo -create -output libsqlite4java-osx.dylib.fat libsqlite4java-osx.dylib.x86_64 libsqlite4java-osx.dylib.arm64
mv libsqlite4java-osx.dylib.fat DynamoDBLocal_lib/libsqlite4java-osx.dylib

This is now a "fat" lib which supports both ARM64 and x86 hardware. Hey presto, you can now invoke DynamoDBLocal in the normal Rosetta-free manner, and it'll all work -- on both hardware platforms.

(This post is correct as of version 2022-1-10 (1.18.0) of DynamoDB-Local -- let me know by mail, or at @jmason on Twitter, if things break in future, and I'll update it.)

This is new to me -- Thanks to David Mee for the pointer.

'During WWII, one of Nazi Germany’s most notorious communication codes was broken by a mild mannered librarian and family man from West Limerick, Richard Hayes. His day-job was as Director of the National Library of Ireland - but during wartime, he secretly led a team of cryptanalysts as they worked feverishly on the infamous "Görtz Cipher" - a fiendish Nazi code that had stumped some of the greatest code breaking minds at Bletchley Park, the centre of British wartime cryptography.

But who was Richard Hayes? He was a man of many lives. An academic, an aesthete, a loving father and one of World War Two’s most prolific Nazi Codebreakers.

At the outbreak of WWII, Hayes, being highly regarded for his mathematical and linguistic expertise, was approached by the head of Irish Military Intelligence (G2), Colonel Dan Bryan, with a Top Secret mission. At the behest of Taoiseach Éamon de Valera, Hayes was given an office and three lieutenants to decode wireless messages being covertly transmitted via Morse code from a house in north Dublin owned by the German Embassy. The coded messages posed a huge threat to Irish national security and the wider war effort. As Hayes team worked to break the code, it was all academic until he met his greatest challenge yet. The man who was to be his nemesis, Dr. Herman Görtz, a German agent who parachuted into Ireland in 1940 in full Luftwaffe uniform in an attempt to spy and transmit his own coded messages back to Berlin. [...] The events that transpired were a battle of wits between the mild mannered genius librarian and his nemesis, the flamboyant Nazi spy.

Hayes has been referred to by MI5 as Irelands "greatest unsung hero" and the American Office of Strategic Services as "a colossus of a man" yet due to the secret nature of his work he is virtually unheard of in his own country.'

Hayes was our lead code-breaker, director of the National Library of Ireland, and then director of the Chester Beatty Museum; he was the first to discover the German use of microdots to hide secret messages; and MI5 credited him with a "whole series of ciphers that couldn't have been solved without [his] input". Quite the polymath!

The book is apparently well worth a read: Code Breaker, by Marc McMenamin, and I can strongly recommend this RTE radio documentary. It's full of amazing details, such as the process of feeding Hermann Görtz false information while he was in prison, in order to mislead the Nazis.

After the war, he fruitlessly warned the Irish government not to use a "Swedish cipher machine", presumably one made by Boris Hagelin, who went on to found Crypto AG, which later proved to be providing backdoors in its machines to the CIA and BND.

Quite a towering figure in the history of Irish cryptography and cryptanalysis!

As a lockdown project, I decided to build a home dashboard to display a few useful details for ambient home use. I've written up some details here.

Maciej Ceglowski asks for a massive surveillance program to defeat COVID-19.

However, as I mentioned on twitter -- there IS an alternative, privacy-preserving approach, which is what is being done in Singapore with their TraceTogether app.

In summary, everyone carries a phone running an app which has an anonymized a random ID, scans local Bluetooth periodically for other people's apps with their random IDs, and records them locally (not uploading to a server). If you find out you have COVID-19 you then trigger an upload of your contact history to a central server. That server then broadcasts out the list of IDs, and everyone you've been in contact with will then get a ping on their app to get tested, self-isolate, etc.

No central surveillance, no creepy big brother watching your location.

My pinboard has a few more write-ups on basically the same idea from various other places, including MIT. This is similar to what China's app does, but (as far as I can tell) with more privacy.

It looks like the Singaporean government digital services team behind TraceTogether is putting together an open source version, at Bluetrace.io.

IMO we have to do this or we will never get out of COVID-19 lockdown before 2021. I am massively in favour of adopting this approach in Ireland and across the world.

Here's a quick tip for people using Huawei or Honor phones.

Huawei recently released EMUI version 9.1.0.326 as an OTA update, which I applied once it was offered as an upgrade option.

Once I installed that OS upgrade, however, I noticed that whenever I listened to music or podcasts using a Bluetooth headset or stereo speakers, there was a new and very noticeable 'echoing' effect on the audio.

It appears this was due to the addition of Huawei Histen, a 3D audio/equaliser feature, which apparently will add 3D audio effects when listening on wired headphones of various varieties -- however this is supposed to be disabled on Bluetooth devices.

I spent several days fruitlessly googling how to disable Histen, but with no luck. Eventually, through trial and error, I discovered a workaround -- simply plug in a pair of wired headphones, go into Settings -> Sounds -> Huawei Histen sound effects, and choose "Natural sound". Hey presto, next time you use Bluetooth headphones, it should no longer have the echo.

I came across this cocktail in Pals, in Catalonia, in 30 degree heat, a few weeks back -- I saw it on the menu at the cafe in the square of the old town, and had to give it a go. It's incredible. Basically, it's lager mixed with a lemon granita -- like a beer slushy. Nothing is better at thirst quenching on a hot day, and best of all it's quite low in alcohol so no worries about lorrying into it during the daytime :)

This year at Groovefest, our yearly get together/mini-festival, I got to serve up a few, with great results -- they were quite popular. So here's the recipe!

First off, a day or two in advance, make a batch of lemon granita. I based mine on this recipe which I'll copy here just in case the original goes away:

Lemon Granita

Serves: about 8

Ingredients:

• 3-4 lemons
• 1L water
• 150g of sugar

Method:

• Zest the lemons and set the zest aside. Juice the lemons until you have 150ml juice (you may not need all of them).

• Add the water and sugar to a large pan and bring to the boil. Reduce to a simmer and cook for 2 minutes, stirring to dissolve the sugar.

• Add the lemon juice and zest, remove from the heat and cover. Set aside to cool for 20 minutes.

• Strain the mixture into 2 containers that will fit in your freezer and leave to cool to room temperature.

• Freeze until the mixture is partially frozen, which should take several hours. (I just left them overnight)

• Remove the granita from the freezer and leave at room temperature until you can break it into chunks with a large spoon or fork.

• Either transfer to a blender or food processor and blitz, or break it up with a fork. It doesn't need to be perfectly smooth and snowy -- a slushy texture is just right for this drink.

• Store in the freezer. Take out 30 minutes before serving and break it up again with a fork.

Clara Con Limón Granizado

To serve: half-fill a half-pint glass with the lemon granita. Pour the beer on top to fill the glass. Stir once or twice to mix. Enjoy!

PS: I think -- not sure as my Catalan is pretty terrible -- it may be a clara granitzada in Catalonia...

It's been a while since I wrote a long-form blog post here, but this post on the Swrve Engineering blog is worth a read; it describes how we use SSD caching on our EC2 instances to greatly improve EBS throughput.

Man sues RMV after driver's license mistakenly revoked by automated anti-terror false positive:

John H. Gass hadn’t had a traffic ticket in years, so the Natick resident was surprised this spring when he received a letter from the Massachusetts Registry of Motor Vehicles informing him to cease driving because his license had been revoked. [...] After frantic calls and a hearing with Registry officials, Gass learned the problem: An antiterrorism computerized facial recognition system that scans a database of millions of state driver’s license images had picked his as a possible fraud. “We send out 1,500 suspension letters every day," said Registrar Rachel Kaprielian. [...] “There are mistakes that can be made."

See also this New Scientist story. This story notes that the system's pretty widespread:

Massachusetts bought the system with a $1.5 million grant from the Department of Homeland Security. At least 34 states use such systems, which law enforcement officials say help prevent identity theft and ID fraud.

In my opinion, this kind of thing -- trial by inaccurate, false-positive-prone algorithm, is one of the most worrying things about the post-PRISM world.

When we created SpamAssassin, we were well aware of the risk of automated misclassification. Any machine-learning classifier will always make mistakes. The key is to carefully calibrate the expected false-positive/false-negative ratio so that the negative side-effects of a misclassification corresponds to the expected rate.

These anti-terrorism machine learning systems are calibrated to catch as many potential cases as possible, but by aiming to reduce false negatives to this degree, they become wildly prone to false positives. And when they're applied as a dragnet across all citizens' interactions with the state -- or even in the case of PRISM, all citizens' interactions that can be surveilled en masse -- it's going to create buckets of bureaucratic false-positive horror stories, as random innocent citizens are incorrectly tagged as criminals due to software bugs and poor calibration.

So, after just over 3 and a half years, I'm leaving Amazon.

It's been great fun -- I can honestly say, even with my code being used by hundreds of millions of users in SpamAssassin and elsewhere, I hadn't really had to come to grips with the distributed systems problems that an Amazon-scale service involves.

During my time at Amazon, I've had the pleasure of building out a brand-new, groundbreaking innovative internal service, from scratch to its current status where it's deployed in production datacenters worldwide. It's a low-latency service, used to monitor Amazon's internal networks using massive quantities of measurement data and machine learning algorithms. It's really very nifty, and I'm quite proud of what we've achieved. I was lucky to work closely with some very smart people during this, too -- Amazon has some top-notch engineers.

But time to move on! In a week's time, I'll be joining Swrve to work on the server-side architecture of their system. Swrve have a very interesting product, extending the A/B-testing model into gaming, and a great team; and it'll be nice to get back into startup-land once again, for a welcome change. (It's not all roses working for a big company. ;) I'm looking forward to it. Who knows, I may even start blogging here again...

Pity about losing those 12 phone tool icons though!

Reminder to Dublin-based readers -- next week, Amazon (my employers) will be putting on Under the Hood at Amazon, billed as 'A night of Beer, Pizza and Cloud Computing for Software Developers'. I'll be speaking at it.

It's partially a recruiting event, but even if you're not looking for a new job, please come along. It's also useful for us to talk about some details of what we've been doing in Dublin, since we've been operating to date with a pretty low profile, and in reality there's some very interesting stuff going on here... particularly the product I'll be talking about, naturally.

Also, there'll be free beer and some Kindles to be won ;)

It's next Thursday night, in our offices in Kilmainham. More info on this Facebook page.

I'm uncomfortable voting for David Norris for President. Here's why.

In November last year, he was a key voice in a Senate debate on the topic of "Protection of Intellectual Property Rights", where he quoted heavily from the flawed judgement by Mr. Justice Peter Charleton in the Warner, Universal, Sony BMG and EMI vs UPC case. (There are allegations that he called the debate after speaking to Paul McGuinness (U2's manager) and Niall Stokes (of Hot Press).)

In the debate, Norris quotes Mr Justice Charleton, saying:

'In failing to provide legislative provision for blocking, diverting and interrupting internet copyright theft, Ireland is not yet fully in compliance with its obligations under European law.' Norris then says: 'Irish law could be brought into alignment with the intention of the European directive through a simple statutory instrument.' [1]

Now, let me clarify my position -- I'm in favour of some means of resolving the level of piracy of music and movies which is widespread nowadays, and I believe there's a mutually agreeable way to do this. But what Norris and Mr Justice Charleton propose is not it. Here are the problems as I see them.

It Lets The Internet Filtering Genie Out Of The Bottle

The big one.

The problem is that any infrastructure for 'blocking, diverting and interrupting internet copyright theft' is effectively infrastructure for 'blocking, diverting and interrupting' any communication on the net. We have to be very careful about how this is permitted, as it'll very quickly suffer "feature creep" and become a general-purpose censorship system -- the Great Firewall Of Ireland. As Damien Mulley put it:

'first they’ll start with the Pirate Bay. Then comes Mininova, IsoHunt, then comes YouTube (they have dodgy stuff, right?), how long before we have Boards.ie because someone quoted a newspaper article or a section of a book? And don’t think they’ll stop there too, any site that links to The Pirate Bay and the others on the hate list will probably be added to the list too...'

In Australia, the anti-child-porn filtering system was quickly used to block gambling websites, gay and straight porn sites, political parties, Wikipedia entries, Christian sites, Wikileaks, and a dentist; in Thailand, a similar system was used to block criticism of the royal family.

Will It Help? I Don't Think So

Norris:

'As long as Irish law is deficient, Mr. Justice Charleton has found that all creative Irish industries are losing money.'

This is quite a hilariously overblown and sweeping statement. ALL creative Irish industries? What qualifies as a 'creative' industry? I suspect some in this country have been involved in industrial acts of creation that made money. ;)

While they're not Irish, the well-known indie label Beggar's Banquet has gone on the record as stating the opposite where the current music situation is concerned --

"There's fewer gatekeepers now. We don't have to knock on a TV station's door or a radio station's door and it's made us far more competitive. [...] There's a wide highway in front of us we can go speeding down, and it wasn't there even two years ago. It means the majors are looking at a world where only 35 Gold Albums a year are certified compared to ten times that recently. But going above Gold in the US is not a problem for us."

So it appears a 'creative' industry (albeit in the UK) is finding things not quite so bad.

Norris again:

'the facts were established in the judgment of Mr. Justice Charleton in which he stated: “Between 2005 and 2009 the recording companies experienced a reduction of 40% in the Irish market for the legal sale of recorded music.” That is a devastating blow. [...] He went on to state: “Some 675,000 people are likely to be engaged in some form of illegal downloading from time to time.”'

Without quite lining up one statement with the other, this reinforces the impression that the only reason the recording companies have seen these drops in revenues is due to internet-borne piracy. However, quoting the brilliant Mumblin' Deaf Ro on the topic of lies, damn lies, and music biz statistics:

'The drop in the value of Irish retail music sales was 11.7% between 2008 and 2009, which is significantly less than the 18% overall drop in retail sales for the economy that year. Digital album sales have increased by 30% since 2007 both in terms of volume and market value.'

So in other words, between 2008 and 2009, Irish retail music sales outperformed the retail sales economy as a whole!

In addition, Ro provides the following BPI figures for UK market volumes over the 2005-2009 period:

    Year  Albums  Singles
    2005  159.0m   47.9m
    2006  154.7m   66.9m
    2007  138.1m   86.6m
    2008  133.6m  115.1m
    2009  128.9m  152.7m

It's clear that singles sales went through the roof, more than tripling. Album sales did drop however, but nowhere near by 40% -- and this coincided with the general drop in the prevailing global economy around that time. He also notes that digital sales in the UK went through the roof globally on a number of metrics in 2009.

While this does not provide figures for the Irish market, I'm at a loss as to how it could be radically different -- Irish and UK consumers have pretty similar musical tastes and consumption habits, I would guess.

Here's a theory: perhaps the issue could be that "Irish" music sales are associated with bricks-and-mortar music shops selling the physical product, whereas digital music sales are associated with online services based outside Ireland, and an Irish buyer buying an album at 7digital.co.uk, or on iTunes, isn't counted as an "Irish retail sale"? Could the problem be that we don't have any significant Irish shops selling music online, I wonder?

Bricks-and-mortar music shops, such as ex-Senator Donie Cassidy's "Celtic Note" (who coincidentally was quite vociferous in that Seanad debate), are indeed hurting in this new model of music consumption -- and that's a problem. But given that good, working digital music sales systems are in operation, it doesn't necessarily appear to be due to massive volumes of internet-borne piracy, going by these figures.

Essentially, internet piracy is a convenient bogeyman, especially for the technophobic old guard, but may have little bearing on the current woes of the Irish record industry and bricks-and-mortar music shops.

(Update: a couple of days after this was posted, a pair of economists at the LSE have said basically the same thing.)

Audible Magic Won't Work For Long Anyway

Audible Magic, which Norris suggests is IRMA's favoured filtering system, received the following verdict from the EFF back in 2004:

'Should Audible Magic's technology be widely adopted, it is likely that P2P file-sharing applications would be revised to implement encryption. Accordingly, network administrators will want to ask Audible Magic tough questions before investing in the company's technology, lest the investment be rendered worthless by the next P2P "upgrade."'

Naturally, encryption is widespread nowadays, so this may already be the case.

Internet Censorship Harms Our Global Image

As Adrian Weckler points out:

'do we really want to send out the message that, digitally, we're the new France? Come to think of it, do we want to tell Google, Facebook, Apple and Twitter that, digitally, we're the new Britain?'

Right now, more than ever, we need to put out an image that we're ready to do business on our end of the internet. Mandatory censorship systems don't exactly support this.

In Summary

So in summary, I would hope to see a more balanced approach to the issue from Norris. Most of the problematic statements in his speech were directly sourced from Mr. Justice Charleton's flawed judgement, but some critical thinking would be vital, I would have thought. The fact that this was lacking, particularly given the allegations of heavy music-biz lobbying beforehand, leaves me feeling less inclined to vote for him than I would have been before, particularly since I haven't heard any clarification on these issues.

([1]: Funnily enough, an SI similar to this was nearly sneaked through a couple of weeks ago, according to reports.)

It's pretty common for apps to require "configuration" -- external files which can contain settings to customise their behaviour. Ideally, apps shouldn't require configuration, and this is always a good aim. But in some situations, it's unavoidable.

In the abstract, it may seem attractive to use a fully-fledged programming language as the language to express configuration in. However, I think this is not a good idea. Here are some reasons why configuration files should not be expressed in a programming language (and yes, I include "Ruby without parentheses" in that bucket):

Provability

If a configuration language is Turing-incomplete, configuration files written in it can be validated "offline", ie. without executing the program it configures. All programming languages are, by definition, Turing-complete, meaning that the program must be executed in full before its configuration can be considered valid.

Offline validation is a useful feature for operational usability, as we've found with "spamassassin --lint".

Security

Some configuration settings may be insecure in certain circumstances; for example, in SpamAssassin, we allow certain classes of settings like whitelist/blacklists to be set in a users ~/.spamassassin/user_prefs file, while disallowing rule definitions (which can cause poor performance if poorly written).

If your configuration file is simply an evaluated chunk of code, it becomes more difficult to protect against an attacker introspecting the interpreter and overriding the security limitations. It's not impossible, since you can, for instance, use a sandboxed interpreter, but this is typically not particularly easy to implement.

Usability

Here's a rather hairy configuration file I've concocted.

    #! /usr/bin/somelanguage
    !$ app.status load html
    !c = []
    ;c['sources'] = < >
    ;c['sources'].append(
        NewConfigurationThingy("foo_bar",
            baz="flargle"))
    ;c['builders'] = < >
    ;c['bots'] = < >
    !$ app.steps load source, shell
    ;bf_mc_generic = factory.SomethingFactory( <
        woo(source.SVN, svnurl="http://example.com/foo/bar"),
        woo(shell.Configure, command="/bar/baz start"),
        woo(shell.Test, command="/bar/baz test"),
        woo(shell.Configure, command="/bar/baz stop")
        > );
    ;b1 = < "name": "mc-fast", "slavename": "mc-fast",
                 "builddir": "mc-fast", "factory": ;bf_mc_generic >
    ;c['builders'].append(;b1)
    ;SomethingOrOther = ;c

This isn't actually entirely concocted from thin air -- it's actually bits of our BuildBot configuration file, from before we switched to using Hudson. I've replaced the familiar Python syntax with deliberately-unfamiliar made-up syntax, to emulate the user experience I had attempting to configure BuildBot with no pre-existing Python knowledge. ;)

Compare with this re-stating of the same configuration data in a simplified, "configuration-oriented" imaginary DSL:

add_source NewConfigurationThingy foo_bar baz=flargle

buildfactory bf_mc_generic source.SVN http://example.com/foo/bar
buildfactory bf_mc_generic shell.Configure /bar/baz start
buildfactory bf_mc_generic shell.Test /bar/baz test
buildfactory bf_mc_generic shell.Configure /bar/baz stop

add_builder name=mc-fast slavename=mc-fast
     builddir=mc-fast factory=bf_mc_generic

Essentially, I've extracted the useful configuration data from the hairy example, discarded the symbology used to indicate types, function calls, data structure construction, and let the configuration domain knowledge imply what's necessary. Not only is this easier to comprehend for the casual reader, it also reduces the risk of syntax errors, by simply minimising the number of syntactical components.

See Also

The Wikipedia page on DSLs is quite good on the topic, with a succinct list of pros and cons.

This StackOverflow thread has some good comments -- I particularly like this point:

When you need your application to be very "configurable" in ways that you cannot imagine today, then what you really need is a plugins system. You need to develop your application in a way that someone else can code a new plugin and hook it into your application in the future.

+1.

This seems to be a controversial topic -- as you can see, that page has people on both sides of the issue. Maybe it fundamentally comes down to a matter of taste. Anyway -- my $.02.

Update: discussions elsewhere: HackerNews

Another Update, 2012-04-06: Robey Pointer wrote a post called Why Config?, in which he describes a Scala-based configuration language in use at Twitter, which uses Scala's runtime code evaluation, and a Scala trait, to express configuration succinctly in a Scala source file and load it at runtime. The downside? It's a Scala source file, executed at runtime, containing configuration. :(

However, this comment in the comments section is worth a read:

At Netli (now part of Akamai) we had a configuration framework very similar in spirit and appearance to Configgy. It was in early 2000-s, we open sourced it since. (http://ncnf.sourceforge.net/). It would provide on-the-fly reload for the C-based programs (the ncnf if a C library). It also had some perks like attribute inheritance and a concept of block references. Most importantly though, it contained a separate schema language and a validator to allow configuration be checked before pushing in production. At Netli we used it to configure 1200 services on over 400 hardware boxes, the configuration becoming about 20+mb in length (assembled from several pieces by the CPP, then M4 templating library).

Naturally, it wasn't Netli's first attempt at doing configuration. One of the first attempts failed since it was Turing-complete. That approach was to specify the configuration as a Perl data specification. In a very short time the lure of unused expressiveness of such Turing-complete environment prevailed and people started to write for-loops around data pieces and doing other tricks to remove redundancy from the configuration. It turned out to be a disaster in the end, with configuration becoming unmaintainable and flaky.

One principle I got out out of that exercise is that configuration shall not be Turing-complete. We've got burned specifically by that property far too many times. Yet I do agree with you that a validation facility is a must-have, which is something not usually part of the simple text-based frameworks. C-based NCNF had it almost from the very beginning though, and it proved to be a very useful harness.

+1. There's lots more info on that system at this post at lionet.livejournal.com.

Another Update, 2017-05-09: casio_juarez on Twitter:

Dev: I'll use a declarative language for config this time.
6 months later: Let's add variables.
12 mos: And conditionals.
18 mos: Fuck.

— 0x0DEADA55 (@casio_juarez) May 8, 2017

Also related: The Configuration Complexity Clock.

(Image credit: Turn The Dial by VERY URGENT Photography)

Happy new year! Or maybe not. Doh.

Over a year ago, Lee Maguire noticed that a contributed SpamAssassin rule, FH_DATE_PAST_20XX, was naively written -- simply to match any date in the year 2010 or later -- and would start to false-positive on all mail in 14 months. We made the trivial fix to avoid this (for at least 10 years, by which point the rule would have obsoleted itself through normal means), and I committed it to SVN.

Problem solved, right? Nope. I'd committed to trunk, but in a moment of inattention had forgotten to backport the fix to the stable release branch, 3.2.x, as well. Nobody else noticed the mistake, and several months later, boom:

• LWN
• Slashdot
• Heise
• jwz
• Spam Resource
• Mike Cardwell

Bugger.

Annoyingly, the GA had assigned this rule 3.5 points in the 3.2.0 rescoring run. This meant that the effective default threshold had been lowered from 5.0 points to 1.5, which produced a 2% false positive rate during the first 13 hours of the new year.

After that point, the fix was pushed to the sa-update channel, and anyone who runs sa-update regularly (as they should!) was brought back to normal filtering behaviour.

The rule is superfluous anyway, since it overlaps with a better-written "eval" rule, DATE_IN_FUTURE_96_XX. Accordingly, most likely scenario is that it'll be removed.

Personally, I see a few lessons from this:

• Obviously, I need to pay more attention. This is easier said than done though, since SpamAssassin has nothing to do with my day job anymore; it's a spare-time thing nowadays, and that's a rare resource, unfortunately. :( But still, a chastening result, and I'm very sorry for my part in this screwup.

• We need more active committers on Apache SpamAssassin. If we'd had more eyes, the fact that I'd forgotten to backport the fix might have been spotted. we're definitely in a better situation now in this regard than we were 6 months ago, so that's good.

• IMO, this is a good demonstration of how too many simple rules are risky; without careful vetting and moderation, it's easy for a bad one to slip past. Perhaps we need to move more towards a DNSBL/network-rule driven approach, although this has its downsides too. Still thinking about this.

• It'd be good to fix the GA so that it wouldn't assign such high points to simple rules like this, without some indication that a human has vetted them and believes them trustworthy.

Daryl posted a good comment on /.:

Clearly we dropped the ball on this one. As far as I know it's our first big rule screw up in the project's 10 years. If you're going to screw up you might as well do it well.

+1 to that!

And to everyone who had to clean up the fallout and spend a holiday recovering lost mails from spam folders... sorry :(

Found here:

On Wednesday 20 May 2009, speaking at a parliamentary Justice Committee debating his new blasphemy law, Dermot Ahern joked that people were making blasphemous comments about him, and he compared his own purity to that of the baby Jesus.

So we have a Justice Minister joking about himself being blasphemed, at a parliamentary Justice Committee discussing his own blasphemy law, that could make his own jokes illegal.

In honour of this Ministerial revelation, we have founded the Church of Dermotology. We believe God sent Dermot Ahern to save Ireland from rational thinking. Our sacred symbol is the Star of Dermot.

Our sacred beliefs are quite similar to those of other religions.

• We believe ice cream wafers are literally the body of Dermot Ahern.
• We believe Dermot Ahern created the universe on Wed 20 may 2009.
• We’re sometimes not sure whether Dermot Ahern really exists.
• We believe it is blasphemous to publish an image of Dermot Ahern.
• We refuse to gather sticks on the Sabbath, which is Wednesday.
• We wear magic underpants that protect us from fire and bullets.
• We are outraged whenever anybody insults our sacred beliefs.
• We fervently support Dermot Ahern’s proposed blasphemy law.
• If it is passed, we will be regularly outraged, and will take test cases.

Like Scientologists, Dermotologists offer a free personality test. Question one: are you vulnerable? Question two: have you money? If you answer yes to either of these questions, you’re in.

After you join, check out the campaign against the Irish blasphemy law at blasphemy.ie.