Author: Justin

Justin Mason, the author of this weblog.

looking at the new DKIM draft

Published July 11, 2005

The combined DKIM standard, mixing Yahoo!'s DomainKeys and Cisco's IIM, has been submitted to the IETF as a candidate spec by the MASS 'pre-working group effort'. I like the idea behind both (a few years back, I, a few other SpamAssassin developers, and several others came up with the roots of a message-signature anti-forgery scheme we called 'porkhash', but never really went anywhere with it), so I'm glad to see this one progressing nicely.

Seeing as I never seem to write much about anti-spam here any more, I might as well remedy that now with some comments on the new DKIM draft. ;)

It's a very good synthesis of the two previous drafts, DomainKeys and IIM, more DK-ish, but taking the nice features from IIM.

The 'h=' tag is now listed as REQUIRED. This specifies the list of headers that are to be signed. If I recall correctly, this was added in IIM, modifies the behaviour of DK, and is a good feature -- it protects against in-transit corruption by, (a) specifying an order of the headers, to protect against MTAs that reorder them; and (b) allowing sites to protect the 'important' headers (From, To, Subject etc.) and ignore possible additions by MTAs down the line (scanner additions, mailing list munging and additions, and so on).

A list of recommended headers to sign is included, with From as a MUST and Subject, Date, Content-Type and Content-Transfer-Encoding as a SHOULD.

Forwarding is, of course, just fine. This one doesn't suffer from the SPF failure mode, whereby a forwarder will break a signature if it doesn't rewrite the SMTP MAIL FROM sender address. (Of course, it now has its own new failure modes -- the message must be forwarded in a nearly-pristine state.)

The message length to sign can be specified with 'l='. This may be useful to protect against the issue where mailing list managers add a footer to a signed message. It recommends that verifiers remove text after the 'l' length, if it appears, since that offers a way for spammers to reuse existing signatures. I still have to think about this, but I suspect SpamAssassin could give points for additional text beyond the 'l=' point that doesn't match mailing list footer profiles.

The IIM HTTP-based public-key infrastructure is gone; it's all DNS, as it was in DK.

The 'z=' field, which contains copies of the original headers, is a great feature for filters -- we can now pragmatically detect 'acceptable' header rewriting if necessary, and handle recovery at the receiver end.

Multiple signatures, unfortunately, couldn't be supported. I can see why, though, it's a very hard problem.

The 'Security Considerations' section is excellent -- 9.1.2 uses a very clever HTML attack.

Looks like development of DKIM-Milter, and an associated library, libdkim, are underway.

Given all that, it looks good. It's not clear how much we can do with DK, and now DKIM, in SpamAssassin, however -- it's very important in these schemes that the message be entirely unmunged, and in most SpamAssassin installs, the filter doesn't get to see the message until after the delivering MTA, or the MDA (Message Delivery Agent), has performed some rewriting. This would cause FPs if we're not very, very careful.

I hope though, that we can find a useful way to trust DKIM results. It appears likely that they'd make an excellent way to provide trustworthy whitelisting -- 'whitelist_from_dkim' rules, similarly to our new whitelist_from_spf support. (In fact, we could probably just merge both into some new 'whitelist_from_authenticated' setting.)

OpenWRT vs Netgear MR814: no contest

Published July 8, 2005

Hardware: After a few weeks running OpenWRT on a Linksys WRT54G, here's a status report.

Things that the new WRT54G running OpenWRT does a whole lot better than the Netgear MR814:

Baseline: obviously it doesn't DDoS the University of Wisconsin, and it doesn't lose the internet connection regularly, as noted in that prior post. I knew that, so those are not really new wins, though.
It's quite noticeably faster. I've seen it spike to double the old throughput rates, and it's solid, too; less deviation in those rates.
It doesn't break my work VPN. I wasn't sure if it was the MR814 that was doing this, requiring an average of about 20 reconnects per day -- now, I know it for a fact. I've had to reconnect my VPN connection about 4 times over the past week.
It doesn't break the Gigafast UIC-741 USB wifi dongle I'm using on the MythTV box. Previously that would periodically disappear from the HAN. Again, I had this pegged as an issue with the driver for that; removing the MR814 from the equation has solved it, too, and it's now running with 100% uptime so far.
It does traffic shaping with Wondershaper, so I can use interactive SSH, VNC, or remote desktop while downloading, even if it's another machine on the HAN doing the download.
It's running linux -- ssh'ing in, using ifconfig, and vi'ing shell scripts on my router is very, very nice.

Man, that MR814 was a piece of crud. ;) I can't recommend OpenWRT enough...

EU software patents directive is history

Published July 7, 2005

Patents: A great outcome! The proposed Directive has been dropped, in the face of massive opposition. Coverage: /., FFII, FT.com, VNUnet, FSFE.

Unfortunately, Rocard's proposed amendments which would have turned this directive into a major win for us, didn't pass -- but it's still a good win. Software patents are not explicitly legal throughout Europe; although some jurisdictions do permit them, they're in a legal grey area, and prosecution is therefore hard (and very expensive for patent holders). This is a much better situation than if the directive as proposed by the Council had passed, since that would have explicitly legalised them throughout the EU.

On top of this win, what I find significant is that we've now brought the issue from where it was a few years ago, as a minor concern known only to a few uber-geeks, to a major political issue that made headlines around the globe. Even my local NPR affiliate reported on this decision! That's a far cry from the mid-90's, when I had a hard time explaining the point of theLeague for Programming Freedom to my hacker friends in the TCD Maths Department.

A great quote from the VNUnet article:

'This represents a clear victory for open source,' said Simon Phipps, chief open source officer at Sun Microsystems. 'It expresses Parliament's clear desire to provide a balanced competitive market for software.'

Yes, that's Sun saying that less software patenting is a good thing. Believe me, that's a great leap forward. Or check out Irish MEP Kathy Sinnott's amazing comments. She hits the nail right on the head; I'm very impressed by that speech.

McCreevy seeing anti-globalisation protesters everywhere

Published July 5, 2005

Patents: I'm just back from a fantastic holiday weekend, totally offline, hiking through Catalina Island. I'm a little bit sunburnt, my nose is peeling, but it was great fun. I got a fantastic picture of the sun setting over hundreds of boats bobbing at their moorings in Two Harbors, which I must upload at some stage.

Anyway, it seems that over the weekend, the EU software-patents debate has swung back heavily towards the anti-swpat side. Fingers crossed -- the vote is this week.

Also, today, EUpolitix.com has an interview with Charlie McCreevy, quoting him as saying:

'The theme, or the background music, to both of these particular directives (the CII and Services Directives) you could see as part of, anti-globalisation, anti-Americanism, anti-big business protests -- in lots of senses, anti-the opening up of markets'

This is standard practice for the Irish government -- they did exactly the same thing with the e-voting issue, painting the ICTE as 'linked to the anti-globalisation movement'. (I have a feeling they think that any group organised online must be 'anti-globalisation', at this stage.)

Of course, with these accusations of being anti-free-market, it's important to remember that a patent is a government-issued monopoly on an invention (or in the software field, on an idea), in a particular local jurisdiction. If anything, being against software patenting is a pro-free-market position, one shared by prominent US libertarians; and nothing gets more pro-free-market than those guys. ;)

CEAS coming up soon…

Published July 2, 2005

Spam: if you work in anti-spam, especially in filtering, or even just in working with email in general, it's well worth going to CEAS 2005, the Conference on Email and Anti-Spam, on Thursday July 21st and Friday 22nd in Stanford:

The organizers of the Conference on Email and Anti-Spam invite you to participate in its second annual meeting. This forum brings together academic and industrial researchers to present new work in all aspects of email, messaging and spam -- with papers this year covering fields as diverse as text classification, clustering and visualization of email, social network analysis applied to both email and spam, spam filtering methods including text classification and systems approaches, game theory, data analysis, Human Interactive Proofs, and legal studies, among others. The conference will feature 26 paper presentations, a banquet, and two invited speakers. See http://www.ceas.cc for details of the current program, as well as on-line registration.

Registration runs out on July 10th.

I went last year, and it was excellent -- several very interesting papers were presented. I'm going this year, too, along with quite a few SpamAssassin committers, and I'm looking forward to it.

Hackability as a selling point

Published June 29, 2005

Hardware: On my home network, I recently replaced my NetGear MR814 with a brand new Linksys WRT54G.

My top criteria for what hardware to buy for this job weren't price, form factor, how pretty the hardware is, or even what features it had -- instead, I bought it because it's an extremely hackable router/NAT/AP platform. Thanks to a few dedicated reverse engineers, the WRT hardware can now be easily reflashed with a wide variety of alternative firmware distributions, including OpenWRT, a fully open-source distro that offers no UI beyond a command-line.

Initially, I considered a few prettier UIs -- HyperWRT, for example -- since I didn't want to have to spend days hacking on my router, of all things, looking stuff up in manuals, HOWTOs and in Google. Finally I decided to give OpenWRT a spin first. I'm glad I did -- it turned out to be a great decision.

(There was one setup glitch btw -- by default, OpenWRT defaults to setting up WPA, but the documentation claims that the default is still no crypto, as it was previously.)

The flexibility is amazing; I can log in over SSH and run the iftop tool to see what's going on on the network, which internal IPs are using how much bandwidth, how much bandwidth I'm really seeing going out the pipe, and get all sorts of low-level facts out of the device that I'd never see otherwise. I could even run a range of small servers directly on the router, if I wanted.

Bonus: it's rock solid. My NetGear device had a tendency to hang frequently, requiring a power cycle to fix; this bug has been going on for nearly a year and a half without a fix from NetGear, who had long since moved on to the next rev of cheapo home equipment and weren't really bothering to support the MR814. I know this is cheap home equipment -- which is why I was still muddling along with it -- but that's just ridiculous. None of that crap with the (similarly low-cost) WRT. OpenWRT also doesn't contain code to DDOS NTP servers at the University of Wisconsin, which is a bonus, too. ;)

Sadly, I don't think Cisco/Linksys realise how this hackability is making their market for them. They've been plugging the security holes used to gain access to reflash the firmware in recent revisions of the product (amazingly, you have to launch a remote command execution attack through an insecure CGI script!), turning off the ability to boot via TFTP, and gradually removing the ways to reflash the hardware. If they succeed, it appears the hackability market will have to find another low-cost router manufacturer to give our money to. (update, June 2006: they since split the product line into a reflashable Linux-based "L" model and a less hackable "S" model, so it appears they get this 100%. great!)

Given that, it's interesting to read this interview with Jack Kelliher of pcHDTV, a company making HDTV video capture cards:

Our market isn't really the mass market. We were always targeting early adopters: videophiles, hobbyists, and students. Those groups already use Linux, and those are our customers.
Matthew Gast: The sort of people who buy Linksys APs to hack on the firmware?
Jack Kelliher: Exactly. The funny thing is that we completely underestimated the size of the market. When we were starting up the company, we went to the local Linux LUG and found out how many people were interested in video capture. Only about 2 percent were interested in video on Linux, so we thought we could sell 2,000 cards. (Laughs.) We've moved way beyond that!

Well worth a read. There's some good stuff about ulterior motives for video card manufacturers to build MPEG decoding into their hardware, too:

The broadcast flag rules are conceptually simple. After the digital signal is demodulated, the video stream must be encrypted before it goes across a user accessible bus. User accessible is defined in an interesting way. Essentially, it's any bus that a competent user with a soldering iron can get the data from. Video streams can only be decrypted right before the MPEG decode and playback to the monitor.
To support the broadcast flag, the video capture must have an encryptor, and the display card must have a decryptor. Because you can't send the video stream across a user accessible bus, the display card needs to be a full MPEG decoder as well, so that unencrypted video never has to leave the card.
Matthew Gast: So the MPEG acceleration in most new video cards really isn't really for my benefit? Is it to help the vendors comply with the broadcast flag?
Jack Kelliher: Not quite yet. Most video cards don't have a full decoder, so they can't really implement the broadcast flag. ATI and nVidia don't have full decoders yet. They depend on some software support from the operating system, so they can't really implement the broadcast flag. Via has a chipset with a full decoder, so it would be relatively easy for them to build the broadcast flag into that chipset.

Aha.

Project management, deadlines etc.

Published June 28, 2005

Work: I took a look over at Edd Dumbill's weblog recently, and came across this posting on planning programming projects. He links to another article and mentions:

My recent return to managing a team of people has highlighted for me the difficulties of the arbitrary deadline approach to project management. Unfortunately, it's also the default management approach applied by a lot of people, because the concept is easy to grasp.
The arbitrary deadline method is troublesome because of the difficulty of estimation. As John's post elaborates, you can never foresee all of the problems you'll meet along the way. The distressing inevitability of 90% of the effort being required by 2% of the deliverable is frequently inexplicable to developers themselves. Never mind the managers remote from the development!

I've been considering why my experience of working with open source seems generally preferable to commercial work, and this may be one of the key elements. Commercial software development is deadline-driven, whereas most open source development has not been, in my experience; 'it's ready when it's ready'.

Edd suggests that using a trouble-ticket-based system for progress tracking and management is superior. I'm inclined to agree.

Irish SME associations quiet on patenting

Published June 23, 2005

Patents: yes, I keep rattling on about this -- the vote is coming up on July 6th. I promise I'll shut up after that ;)

UEAPME has issued a statement regarding the directive which is strongly critical of its current wording (UEAPME is the European small and medium-sized business trade association, comprising 11 million SMEs). Quote:

'The failure to clearly remove software from the scope of the directive is a setback for small businesses throughout Europe. UEAPME is now calling on the European Parliament to reverse yesterday's decision at plenary session next month and send a strong message that an EU software patent is not an option,' Hans-Werner Müller, UEAPME Secretary General, stated.
'There is growing agreement among all actors that software should not be patented, so providing an unequivocal definition in the directive that guarantees this is clearly in the general interest. We are calling on the Parliament to support the amendments that would ensure this,' said Mr Müller.
'The cacophony of misinformation and misleading spin from the large industry lobby in the run up to this vote has obscured the general consensus on preventing the patenting of pure software.'

That's all well and good. So presumably the Irish members of UEAPME, ISME and the SFA, are agreeing, right? Sadly, neither of these have issued any press releases on the subject, as far as I can see, and approaches by members of IFSO have been totally fruitless.

Since both have made recent press noting that Irish small businesses face difficulties with the rising costs of doing business, this would seem to be a no-brainer -- legalising software patents would immediately open Irish SMEs up to the costs associated with them: licensing fees, fighting spurious infringement litigation from 'patent troll' companies, the 'chilling effects' on investors noted by Laura Creighton, and of course the high price of retaining patent lawyers to file patents on your own innovations. One wonders why they aren't concerned about these costs...

Happy Midwinter’s Day!

Published June 22, 2005

Antarctic: Happy Midwinter's Day!

I've just finished reading Big Dead Place , Nicholas Johnson's book about life at McMurdo Base and the US South Pole Station, with anecdotes from his time there in the early years of this decade.

It's a fantastic book -- very illustrative of how life really goes on on a distant research base, once you get beyond romantic notions of exploration of the wild frontiers. (Like many geek kids, I spent my childhood dreaming of space exploration, and Antarctica is the nearest thing you can get to that right now.) A bonus: it's hilarious, too.

Unfortunately it's far from all good -- as one review notes, it's like 'M*A*S*H on ice, a bleak, black comedy.' There's story after story of moronic bureaucratic edicts emailed from comparatively-sub-tropical Denver, Colorado, ass-covering emails from management on a massive scale, and injuries and asbestos exposures covered up to avoid spoiling 'metrics'.

Here's a sample of such absurdity, from an interview with Norwegian world-record breaking Antarctic explorer, Eirik Sønneland:

BDP: I was working at McMurdo when you arrived in 2001. I remember it well because we were commanded by NSF not to accommodate you in any way, and were forbidden to invite you to our rooms or into any buildings. We were told not to send mail for you, nor to send email messages for you. While you were in the area, NSF was keeping a close eye on you. What did the managers say to you when you arrived?
They asked us what plans we had for getting home. The manager at Scott Base (jm: the New Zealand base) was calm and listened to what we had to say. I must be honest and say that this was not the way we were treated by the U.S. manager. It was like an interrogation. Very unpleasant. He acted arrogant. However, it seemed like he started to realize after a couple of days that we didn't try to fool anybody. He probably got his orders from people that were not in Antarctica at the time. And, to be honest, today I don't have bad feelings toward anyone in McMurdo. Bottom line, what did hurt us was that people could not think without using bureaucracy. If people could only try to listen to what we said and stop looking up paragraphs in some kind of standard operating procedures for a short while, a lot could have been solved in a shorter time.
One example: our home office, together with Steven McLachlan and Klaus Pettersen in New Zealand, got a green light from the captain of the cargo ship that would deliver cargo (beer, etc.) to McMurdo, who said he would let us travel for free back to New Zealand if it was okay with his company. At first the company was agreeable, but then NSF told them that the ship would be under their rent until it left McMurdo and was 27 km away. Reason for the 27 km? The cargo ship needed support from the Coast Guard icebreaker to get through the ice. Since, technically, the contract with NSF did not cease until the ship left the ice, NSF could stop us from going on the ship. At which point NSF offered to fly us from McMurdo for US$50,000 each.

He also maintains an excellent website at BigDeadPlace.com, so go there for an idea of the writing. BTW, it appears the UK also maintains an Antarctic base. Here's hoping they keep the bureaucracy at a saner level over there.

The meaning of the term ‘technical’ in software patenting

Published June 22, 2005

Patents: One of the key arguments in favour of the new EU software patenting directive as it's currently worded, from the 'pro' side, is that it doesn't 'allow software patents as such', since it requires a 'technical' inventive step for a patent to be considered valid.

Various MEPs have tried to clarify the meaning of this vague phrase, but without luck so far.

Coverage has mostly noted this as meaning that 'pure software' patents are not permissible, for example this Washington Post article, FT.com,and InformationWeek.

But is this really the case, in pragmatic terms? What does a 'technical inventive step' mean to the European Patent Office?

Well, it doesn't look at all promising, according to this report from the Boards of Appeal of the European Patent Office from 21 April 2004, dealing with a Hitachi business method patent on an 'automatic auction method'. The claims of that patent application (97 306 722.6) covered the algorithm of performing an auction over a computer network using client-server technology. The actual nature of this patent isn't important, anyway -- but what is important is how the Boards of Appeal judge its 'technical' characteristics.

The key section is 3.7, where the Board writes:

For these reasons the Board holds that, contrary to the examining division's assessment, the apparatus of claim 3 is an invention within the meaning of Article 52(1) EPC since it comprises clearly technical features such as a "server computer", "client computers" and a "network".

So in other words, if the idea of a computer network is involved in the claims of a patent, it 'includes technical aspects'. It then goes on to discuss other technical characteristics that may appear in patents:

The Board is aware that its comparatively broad interpretation of the term "invention" in Article 52(1) EPC will include activities which are so familiar that their technical character tends to be overlooked, such as the act of writing using pen and paper.

So even writing with a pen and paper has technical character!

It's a cop-out, designed to fool MEPs and citizens into thinking that a reasonable limitation is being placed on what can be patented, when in reality there's effectively no limits, if there's any kind of equipment involved beyond counting on your fingers.

The only way to be sure is to ensure the directive as it eventually passes is crystal clear on this point, with the help of the amendments that the pro-patent side are so keen to throw out.

(BTW, I found this link via RMS' great article in the Guardian where he discusses software patenting using literature as an analogy. recommended reading!)

Latest Script Hack: utf8lint

Published June 18, 2005

Perl: double-encoding is a frequent problem when dealing with UTF-8 text, where a UTF-8 string is treated as (typically) ISO Latin-1, and is re-encoded.

utf8lint is a quick hack script which uses perl's Encode module to detect this. Feed it your data on STDIN, and it'll flag lines that contain text which may be doubly-encoded UTF-8, in a lintish way.

BSA Spams Patent Holders

Published June 17, 2005

Patents: An anonymous contributor writes:

'I just received this letter and these pre-addressed postcards in the post this morning. I was surprised when I saw the envelope, because I'd never received anything from the BSA before. It turned out that they had extracted my name and address from the European Patents database, because I registered a software patent once. So a lot of these letters have been probably been sent out.

According to the letter, from Francisco Mingorance, the draft directive is being turned around to 'rob small businesses of their intellectual property assets'.

I find it hard to see how that could be true. However the BSA's letter has an important message you should heed - it is critical to contact your European representatives (your MEP and your country's Commissioner) within the next two weeks. Let them know that the European Union should curtail software patents for once and for all.

Get out your best stationery and write to your MEP at the address given on this page.

Make sure your message is short and clear. SME's don't benefit from patents. Few patents are held by SME's and the cost of applying for, maintaining and defending them is crippling.'

jm: I would suggest noting that you support the position of rapporteur
Michel Rocard MEP, and/or the FFII -- details here. Please do write!

BTW, the contributor also offers: 'if anyone is interested in doctoring up the BSA postcards, I can provide the hi-res scans.' ;)

Amazing article series on Climate Change

Published June 16, 2005

Science: in April and May, the New Yorker printed an amazing series of articles on climate change by Elizabeth Kolbert, full of outstanding research and interviews with the key players.

Unlike much coverage, it includes the expected results of climate change in the US:

Different climate models offer very different predictions about future water availability; in the paper, Rind applied the criteria used in the Palmer index to GISS's model and also to a model operated by NOAA's Geophysical Fluid Dynamics Laboratory. He found that as carbon-dioxide levels rose the world began to experience more and more serious water shortages, starting near the equator and then spreading toward the poles. When he applied the index to the giss model for doubled CO2, it showed most of the continental United States to be suffering under severe drought conditions. When he applied the index to the G.F.D.L. model, the results were even more dire. Rind created two maps to illustrate these findings. Yellow represented a forty-to-sixty-per-cent chance of summertime drought, ochre a sixty-to-eighty-per-cent chance, and brown an eighty-to-a-hundred-per-cent chance. In the first map, showing the GISS results, the Northeast was yellow, the Midwest was ochre, and the Rocky Mountain states and California were brown. In the second, showing the G.F.D.L. results, brown covered practically the entire country.
'I gave a talk based on these drought indices out in California to water-resource managers,' Rind told me. 'And they said, 'Well, if that happens, forget it.' There's just no way they could deal with that.'
He went on, 'Obviously, if you get drought indices like these, there's no adaptation that's possible. But let's say it's not that severe. What adaptation are we talking about? Adaptation in 2020? Adaptation in 2040? Adaptation in 2060? Because the way the models project this, as global warming gets going, once you've adapted to one decade you're going to have to change everything the next decade.

And how the anti-climate-change side are attempting to control US public opinion:

The pollster Frank Luntz prepared a strategy memo for Republican members of Congress, coaching them on how to deal with a variety of environmental issues. (Luntz, who first made a name for himself by helping to craft Newt Gingrich's 'Contract with America,' has been described as 'a political consultant viewed by Republicans as King Arthur viewed Merlin.') Under the heading 'Winning the Global Warming Debate,' Luntz wrote, 'The scientific debate is closing (against us) but not yet closed. There is still a window of opportunity to challenge the science.' He warned, 'Voters believe that there is no consensus about global warming in the scientific community. Should the public come to believe that the scientific issues are settled, their views about global warming will change accordingly.'

They're a great synthesis. Go read the articles -- part 1 ('Disappearing islands, thawing permafrost, melting polar ice. How the earth is changing'), part 2 ('The curse of Akkad'), and part 3 ('What can be done?'). They're long, but if you're still on the fence about this one, they'll wake you up.

Bayesian learning animation

Published June 16, 2005

Spam: via John Graham-Cumming's excellent anti-spam newsletter this month, comes a very cool animation of the dbacl Bayesian anti-spam filter being trained to classify a mail corpus. Here's the animation:

And Laird's explanation:

dbacl computes two scores for each document, a ham score and a spam score. Technically, each score is a kind of distance, and the best category for a document is the lowest scoring one. One way to define the spamminess is to take the numerical difference of these scores.
Each point in the picture is one document, with the ham score on the x-axis and the spam score on the y-axis. If a point falls on the diagonal y=x, then its scores are identical and both categories are equally likely. If the point is below the diagonal, then the classifier must mark it as spam, and above the diagonal it marks it as ham.
The points are colour coded. When a document is learned we draw a square (blue for ham, red for spam). The picture shows the current scores of both the training documents, and the as yet unknown documents in the SA corpus. The unknown documents are either cyan (we know it's ham but the classifier doesn't), magenta (spam), or black. Black means that at the current state of learning, the document would be misclassified, because it falls on the wrong side of the diagonal. We don't distinguish the types of errors. Only we know the point is black, the classifier doesn't.
At time zero, when nothing has been learned, all the points are on the diagonal, because the two categories are symmetric.
Over time, the points move because the classifier's probabilities change a little every time training occurs, and the clouds of points give an overall picture of what dbacl thinks of the unknown points. Of course, the more documents are learned, the fewer unknown points are left.

This is an excellent visualisation of the process, and demonstrates nicely what happens when you train a Bayesian spam-filter. You can clearly see the 'unsure' classifications becoming more reliable as the training corpus size increases. Very nice work!

It's interesting to note the effects of an unbalanced corpus early on; a lot of spam training and little ham training results in a noticeable bias towards the classifier returning a spam classification.

Flickr as a ‘TypePad service for groups’

Published June 13, 2005

Web: a while back, I posted some musings about a web service to help authenticate users as members of a private group, similarly to how TypeKey authenticates users in general.

Well, Flickr have just posted this draft authentication API which does this very nicely -- it now allows third-party web apps to authenticate against Flickr, TypeKey-style, and perform a limited subset of actions on the user's behalf.

This means that using Flickr as a group authentication web service is now doable, as far as I can see...

DVD annoyances

Published June 13, 2005

Hardware: I've been needing a decent backup solution, since I've got 60GB of crud on my hard disk that isn't being rsynced offsite yet. So I bought myself a nifty DVD writer from woot.com a week ago, supporting DVD+RW, DVD+R, DVD-RW, and DVD-R, and a spindle of 20 DVD+Rs from Target. Little did I realise the world of pain I was entering.

Did you know there are no less than 6 barely-compatible DVD formats? Prerecorded DVD, DVD-RAM, DVD-R, and DVD-RW, from the DVD Forum, and DVD+RW and DVD+R, from the 'DVD+RW Alliance'. Interoperability is, needless to say, a total mess, even with the Sony 4-format drive I picked up.

I eventually managed to burn myself a DVD+R backup of bits of my home dir, making several coasters in the process (DVD+Rs apparently do not support simulated-write dry-runs, at least not with growfs). So, great!

Next thing to do was try it out on my laptop's internal CD/DVD drive to make sure it worked. Needless to say, it didn't.

Apparently, single-session, single-track DVD+Rs are virtually identical to DVD-ROMs, which most generic DVD-reader drives support. However, Sony drives do not support setting the 'book type' bits, which is the trick that turns a DVD+R 'into' a DVD-ROM-compatible disc. Guess why (hint: it's Sony). Yep, that's right, paranoia about piracy. Well, thanks a bunch, Sony -- my backups are now of decidedly limited usefulness, since I don't know if I'm ever going to be able to read them again! (more info from the OSTA.) I think I now see why Woot were flogging them cheap.

I'm not sure where to go with this -- do I have a spindle of 17 shiny frisbees? I have a very nasty feeling I'm heading into dead media territory here. What a mess...

Aaaanyway. Here's some possibly-useful bookmarks.

OTOH, I got to watch the BBC's new documentary, The Power of Nightmares, a fantastic history of the two parallel ideological worlds of al-Qaeda and the US neo-conservatives. Mind-boggling, but highly recommended.

European swpat update letter

Published June 10, 2005

Patents: Ian Clarke copied the FSFE-IE mailing list with a good mail he sent to Mairead McGuinness MEP, detailing the current state of proposed fixes to the European software patenting directive. He discusses a comment from an Ericsson employee asking for software patentability:

It may be the case that this employee was concerned about Ericsson's ability to compete against smaller competitors if Ericsson cannot use software patents against them. I would argue that it is not the responsibility of any EU institution to protect Ericsson against legitimate competition from other companies, indeed competition must be encouraged. Software patents will have a stifling effect on competition in Europe, and this is why some large companies like Ericsson are strong advocates for this directive.

And a brief overview of the amendments we want:

The Foundation for a Free Information Infrastructure, an organisation whose line we endorse, has prepared an analysis of the amendments, indicating which will help to ensure that software patents do not become patentable, and which will not. This document may be downloaded here.
In particular, we support the position and amendments of Piia Noora Kauppi MEP, who has taken a strong position against the introduction of software patents within the EPP group, and also the position of Michel Rocard MEP who is the rapporteur for this Directive.

The only other thing it misses, in my opinion, is a paragraph discussing the 'as such' loophole that has been heavily relied upon by most pro-swpat politicians recently -- the trick of saying 'this directive does not permit software patenting, as such'.

Indeed, it does not permit patenting of all software techniques, but instead permits the patenting of software techniques as long as it is of 'a technical nature' -- without defining what that means. Given that it's clearly arguable that all software is technical, and since patent offices earn money based on the patents they accept, rather than those they reject, this is a loophole the size of a bus. Many of the desired amendments concern cleaning up this obvious omission.

Anyway, here's the full text of Ian's mail from the list archive.

Remote playback from laptop to MythTV box

Published June 10, 2005

Linux: the MythTV hacking continues (infrequently). Here's the latest -- a way to play music from my laptop, with sound output via the Mythbox.

Dot-coms and geographical insularity

Published June 8, 2005

Web: i caught sight of (8 June 2005, Interconnected), on the geographical insularity of the dot-com boom. A good read:

The huge influx of cash at the turn of the millennium led to the whole Web being built in the image of the Bay area. The website patterns that started there and - just by coincidence - happened to scale to other environments, those were the ones that survived.

Lots to think about. He's spot on, of course -- many of the web's big commercial success stories are almost shamelessly US-oriented, and if they work outside that, it's purely by accident.

I'd love to see more web businesses that work well for other parts of the world, but that'll take money -- and from what I saw in Dublin, the money either (a) just isn't there, or (b) frequently goes to the companies that talk the talk, but then piddle it away on ludicrous 'e-business architectures' and get nothing useful out the other end.

On both counts, Silicon Valley has an ace up its sleeve. The VCs are smart and well-funded, and the developers have experience, and know which tools are right for the job.

I'd be curious to hear how other high-tech hotspots in the US (Boston, for example) find this.

IBM patents web transcoding proxies

Published June 2, 2005

Web: I link-blogged this, but it's generated some email already, so it deserves a proper posting.

One thing you quickly learn about IBM where software patents are concerned, is that if IBM Research is making noise about a new software technique, they've probably patented it already. A few years ago, IBM was keen on HTTP transcoding -- rewriting web content in a proxy, to be more suitable for display and access from less-capable devices, like PDAs and mobile phones.

So I probably should not have been surprised today when I came across USPTO patent 6,886,013, which is an IBM patent on a 'HTTP caching proxy to filter and control display of data in a web browser'. It was applied for on Sep 11 1997, and finally granted on Apr 26 of this year.

The first claim covers:

A method of controlling presentation on a client of a Web document formatted according to a markup language and supported on a server, the client including a browser and connectable to the server via a computer network, the method comprising the steps of:
as the Web document is received on the client, parsing the Web document to identify formatting information;
altering the formatting information to modify at least one display characteristic of the Web document; and
passing the Web document to the browser for display.

Notice that there's actually no mention of a HTTP proxy there -- in other words, an in-browser rewriting element, such as Greasemonkey or Trixie may be covered by that claim. However, the claim does indicate that the document is passed from the 'client' to the 'browser', so perhaps having the 'client' inside the 'browser' evades that.

It appears this really wasn't original research even when the patent was applied for -- there's probable prior art, even if the patent itself doesn't cite it. For example, WWW4 in 1995 included Application-Specific Proxy Servers as HTTP Stream Transducers, which discusses 'transduction' of the HTTP traffic and gives an example of 'A ``rewriting'' OreO (transducer element) that encapsulates each anchor inside the Netscape Blink extension, making anchors easier to spot on monochrome displays'. On top of that, Craig Hughes notes that his 'senior project at Stanford in 1992 was an implementation of a content-modifying HTTP proxy. It re-worked HTML in http streams to add some markup to enable full navigability through touch screen or voice control, for screen-only kiosks.'

Add this to the ever-growing list of over-broad software patents.

Getting JuK to output sound via ALSA

Published June 2, 2005

Linux: Linux sound is still a mess. Due to the ever-changing 'sound server of the week' system used to decide how an app should output sound, it's perfectly possible to have 3 apps on your desktop happily making noise at the same time, while another app complains about requiring exclusive access to /dev/dsp -- or worse, hangs silently while it attempts to grab an exclusive lock on the device.

This page gives a reasonably good guide to getting software mixing working across (virtually) all apps, using ALSA software mixing and esd.

However, some cases are still very kludgy -- in particular, JuK, the excellent KDE mp3 jukebox app, has a tendency to play poorly with others, requiring playback via no less than two sound servers -- artsd and esd -- to work correctly in the above setup. In addition, the support for mp3 files in artsd is buggy -- it's frequently unable to open certain mp3s, depending on how they were encoded.

Well, good news -- the current release of JuK now supports direct playback from GStreamer via ALSA. Here's how. By adding these lines:

[GStreamerPlayer]
SinkName=alsasink

to ~/.kde/share/config/jukrc, you can skip sending JuK mp3 playback via 2 sound servers, and just play directly to the hardware from the mp3 player. An improvement! Not quite optimal, and certainly not user-friendly -- but getting there...

Patents come to computer gaming

Published June 2, 2005

Patents: in a recent discussion about games and patents, it emerged that these common elements are patented:

streamed loading
minigames during a loading screen
ghost cars in racing games

Looks like software patenting is coming to computer games in a big way. I'm not sure how any game on a modern platform can avoid the 'streamed loading' patent.

Naturally, I can remember playing games on the Commodore 64 in the 1980s that included these...

Yet another non-smoking weblog

Published June 1, 2005

Life: seeing as yesterday was World No Tobacco Day, it's worth noting that I gave up smoking last Thursday.

This is the first time I've taken the step of quitting with any seriousness. I've been smoking since I was 18 or 19, without any real attempts to quit before now. It was a gradual process, but imagining a smoker's future, with the diseases and reduced life expectancy it involves, makes it quite sensible in the end. So far, it's going pretty well -- lots of occasional pangs, but nothing I can't say no to... especially with the aid of Liquorice Altoids. wish me luck!

Irish Oireachtas take care of their own

Published May 27, 2005

Net: Fergus Cassidy reports that 'bandwidth-starved TDs and Senators' in the Oireachtas will be taking a shortcut around Ireland's woeful consumer broadband situation, especially in terms of deployment outside of the main urban areas.

There's a tender up to implement 'an enhanced remote access system, which will improve access from Members' homes or constituency Offices to data and services on servers in Leinster House'.

No similar luck for their constituents, of course. That really takes the biscuit...

Backscatter X-ray ‘naked scanners’ in the news

Published May 27, 2005

Security: the use of backscatter x-ray scanners has hit the US press now that the TSA are taking an interest.

These are interesting devices; unlike normal X-rays, they effectively render clothes invisible. That's obviously got big privacy implications.

Quite a few of the press stories include images that have been blurred or obscured, presumably to render them printable. However, this image seems closer to the real results (not work-safe).

They were trialled in Heathrow's Terminal 4 last year. One slashdotter's experience:

Every Nth person in the line had to go through. They take you to a seperate are which is blocked off, make you lift up your arms and then move, facing three different directions. There was one operator and the screen was blocked off. The operator is always the gender of the person being scanned. Still I felt very offended for two reasons. First, even though it was enclosed it still made me feel exposed and my personal space violated, second, any questions I asked the operator with regards to their data storage, or if I could see the images that had been made were met with ignorance and my questions were ignored. However, turning down a scan you would probably get a strip search which would be even worse. I disliked airplane security checks before, but now it is incredibly annoying.

The Times has some passenger's reactions to images from their scans:

'I was quite shocked by what I saw,' said Gary Cook, 40, a graphic designer from Shaftesbury, Dorset. 'I felt a bit embarrassed looking at the image.'
A female passenger, who did not want to be named, said: 'It was really horrible. It doesn't leave much to the imagination because you're virtually naked, but I guess it's less intrusive than being hand searched.'

If these are installed more widely, I wonder how long it'll take before we start seeing backscatter images of supermodels being saved to floppy by unscrupulous staff, and leaked?

Also, SpyBlog notes that images of children scanned with this device would constitute 'making, distributing or possessing child pornography' in the UK, presuming the machine stores them internally in electronic form. oops!

I’m a new ASF Member!

Published May 27, 2005

Apache: It seems I've been elected as a member of the Apache Software Foundation! That's a nice surprise ;)

Massive US bank breaches, and Europe

Published May 25, 2005

Security: Adam Shostack has been tracking the immense volume of recent bank disclosures of compromised customer data. Bruce Schneier has also commented, and an interesting question arose in his posting's comments -- why are there seemingly no similar problems with European banks?

One responder points to a WSJ article which broadly misses the point. It discusses the additional layers of security imposed by European banks above the usual username/password combo. This is true -- Eurobanks generally have higher security at the 'front gate'; for example, I recall Bank of Ireland even issued SecurID-type tokens in its earliest online banking system. However, that misses the 'insider' attack, as in the most recent case of these 676,000 accounts, so I think it misses the point.

Bruce Schneier's take:

Personal data is 1) not collected as widely, and 2) much less valuable as a tool to commit fraud. The second reason is far more important.

I think he's partially right. Access to new and existing accounts in the US often requires little more than an SSN or similar trivial, easily-discoverable, data which is used in common across multiple institutions, and can be performed online; whereas in Europe, one requires documentary proof of address, ID, and the act must be performed in person at a bank branch. (This is often exceedingly annoying, of course. ;) In general, identity theft seems to be at a greater level in the US, and this is one reason why, I'd guess.

Adam Shostack has another take: these disclosures have all arrived on the heels of California's SB 1386. It's very unlikely that these kind of breaches never occurred before this, and suddenly began recently -- it's more likely that they've always gone on, but are unreported in Europe (and of course were unreported in the US, pre-SB 1386).

I'd add another point -- the US has a large population of targets, with banks sharing financial systems across the entire country. Europe, by contrast, has many individual countries which each have their own set of banks and banking systems, and less interoperability and cross-state data flow. The potential return from ID theft fraud is increased by the larger pool of candidate victims in the US, compared to what an attacker could achieve in each individual European country. This means both that (a) an attack will affect a smaller number of victims in Europe than the US, and (b) widening the scale of an attack becomes significantly harder when the attacker must deal with new systems. It's the 'security monoculture' issue again, applied to banking instead of operating systems.

The Nokia 770 Internet Tablet

Published May 25, 2005

Hardware: Slashdot: Nokia's Linux Handheld. It's to be called the Nokia 770 Internet Tablet, and runs on an open source development platform called Maemo.

This looks really nifty. ARM processor, 800x480 pixel resolution, GTK+, 2.6 kernel, wifi, 3 hours of active battery life, and a clever panning system to get around the clunkiness of scrollbars on a touchscreen.

I note particularly that they seem to have planned to include an RSS reader based on Liferea.

The Maemo site looks interesting, in that it's clearly a bunch of switched-on, open-source-comprehending developers who set it up; it's built using Apache Forrest, they use Bugzilla for issue tracking, Mailman for lists, the terms of use for user contributions explicitly call out OSI-approved licenses as a requirement, there's plentiful references to Debian's apt as the preferred means of installing developer platform software, and Maemo apps are distributed as Debian packages.

There's clearly been quite a lot of work going on behind the scenes. There's already some third-party apps out there, such as those on INdT's Maemo apps page, and the the SDK tutorial contains copious detail, suggesting it's been seeing some use.

That SDK tutorial is full of tantalizing glimpses into Maemo's operation.

It all looks very promising, and nicely hackable! I'm looking forward to a closer look at one of these. It's especially good to see such a solid comprehension of the open source model by such a major company. (If only they could have a word with their patents department ;)

Update: They've ported WebCore to GTK+. Mobile Gazette has more info, too, including this worrying line:

And although Nokia hold several patents for (the Maemo development platform), they intent to open up access to their intellectual property to aid development.

(My emphasis.) That line is not encouraging, seeing as it seems to be a pretty typical cross-compilation platform as seen in embedded systems development. But hey, let's see the patents first.

Threadless RSS

Published May 25, 2005

Clothing: I love Threadless. Unfortunately, they don't have an RSS feed for new T-shirts. So I wrote a quick scraper:

Threadless New Tees (RSS)

with pictures, naturally. This is not going to help my Threadless habit. ;)

Here's a preview of what the feed looks like:

Del.icio.us ranking systems

Published May 25, 2005

Weblogs: there's been a few attempts to mine 'trend' data from del.icio.us:

trendalicious, a 'near real-time view of website popularity trends as reflected by the del.icio.us social bookmarking service'
grafolicious, which is similar
hublog: gatherers of the month lists the 'trendsetters' who link to popular sites first

However, none consider how many links a user generates. A user who links to every single page on the web would quickly gain a good 'trendsetting' rating, and would also skew the website trends upwards, without actually providing useful data to others.

A look at the hublog top posters does seem to indicate they're linking prolifically to any old crap that looks likely to be popular, which is a more humanly-possible way to do that. ;)

However, populicious new links is quite cool -- popular sites that are new in the last 24 hours. Especially handy to find out where one could download Daily Show torrents these days. ;)

There's also the venerable Hot Links, which unfortunately tracks a very small population, but still gets interesting stuff.

Lexis-Nexis hacked through spam

Published May 20, 2005

Spam: WashPost: Computers Seized in Data-Theft Probe:

According to an account provided by the teenaged member of the hacker group -- and confirmed by the law enforcement source who insisted on anonymity -- the LexisNexis break-in was set in motion by a blast of junk e-mail. Sometime in February a small group of hackers ... sent out hundreds of e-mails with a message urging recipients to open an attached file to view pornographic child images. The attachments had nothing to do with child porn; rather, the files harbored a virus (sic) that allowed the group's members to record anything a recipient typed on his or her computer keyboard.
According to the teenage source, a police officer in Florida was among those who opened the infected e-mail message. Not long after his computer was infected with the keystroke-capturing virus, the officer logged on to his police department's account at Accurint, a LexisNexis service provided by Florida-based subsidiary Seisint Inc. ...
The young hacker said the group members then created a series of sub-accounts using the police department's name and billing information. Over several days, the hacker said the group looked up thousands of names in the database, including friends and celebrities. The law enforcement source said the group eventually began selling Social Security numbers and other sensitive consumer information to a ring of identity thieves in California.

Justice Bradley on patent law

Published May 17, 2005

Mr. Justice Bradley, discussing US patent law in 1882:

The design of the patent laws is to reward those who make some substantial discovery or invention, which adds to our knowledge and makes a step in advance in the useful arts. Such inventors are worthy of all favor. It was never the object of those laws to grant a monopoly for every trifling device, every shadow of a shade of an idea, which would naturally and spontaneously occur to any skilled mechanic or operator in the ordinary progress of manufactures.
Such an indiscriminate creation of exclusive privileges tends rather to obstruct than to stimulate invention. It creates a class of speculative schemers who make it their business to watch the advancing wave of improvement, and gather its foam in the form of patented monopolies, which enable them to lay a heavy tax upon the industry of the country, without contributing anything to the real advancement of the arts. It embarrasses the honest pursuit of business with fears and apprehensions of concealed liens and unknown liabilities to lawsuits and vexatious accountings for profits made in good faith.

Well said that man! (via)

Virtualisation is good for the environment

Published May 16, 2005

Computing: mentioned in a Slashdot thread about green server farms -- a page extolling the OpenVPS virtual-server software's environmental benefits:

OpenVPS is good for the environment: a low-end server these days consumes no less than 200W. Given that typical servers run 24/7/365 this amounts (to) 1752 KWh per year. And because every joule of energy consumed by a server is transformed to heat, you need to at least double this to consider the air conditioning costs, which brings us to 3504 KWh per year. ...
At some point this becomes an ethical question: If my CPU is 99.9% idle, is it environmentally (not to mention fiscally!) responsible of me to keep this server running?
Virtualization technologies such Linux VServer used by OpenVPS offer a very viable alternative. If the server acts and feels like a dedicated server, what difference does it really make if it's actually virtual? Yet consolidating 30 physical servers into 30 OpenVPS accounts running on one (albeit power hungry) server would save over 100000 kWh per year. That's as much energy as is consumed on average by 10 houses!

What an excellent point! The OpenVPS dev's slashdot commment reveals another good demo of this --

  # cat /proc/uptime
  16000520.62 9482790.31

The first number is seconds of uptime, the second number is seconds spent in a CPU-idle state. So the server for taint.org, going by those numbers, has spent 59% of its time in a CPU-idle state -- and converting fossil fuels to waste heat in the process...

UBE, not UCE

Published May 16, 2005

Spam: About this time last year, German neo-nazis launched a massive worldwide spam run with the aid of the Sober.H worm.

Well, it looks like they're planning to make this a regular occurrence, because it's on again, spamming nazi opinions linking to stories on reputable news sites, as well as pages on less reputable right-wing sites, Joe Wein has posted some samples. I've already received nearly a thousand since last night.

The good news -- here's a SpamAssassin ruleset that catches these nicely. thanks Raymond!

Using sound as a dead man’s switch

Published May 13, 2005

Software: a nifty trick in this Slashdot comment:

... This reminds me of an old trick we developed to use on the Amiga on a public-access cable channel. The software was under development and crashed occasionally, so rather than having a flashing guru meditation up on a local TV channel until it was rebooted the next day, we came up with a plan, that would probably work on a Windows machine as well (or just about any other system)
The idea was that while the software application was running, it drove a continuous 1khz tone out the audio port that kept a relay energized (that kept the signal on-air). When the system crashed, the audio output stopped, which meant the relay was no longer energized = video signal switched back to a stock SMPTE bars signal from a test generator.

Nowadays, I'd probably pay the money for a hardware watchdog timer. But this is a good, cheap way to implement a dead man's switch. Very clever!

The Stag’s new owner: Louis Fitzgerald

Published May 12, 2005

Dublin: Sorry to the non-Dublin readership, I'm sure you all are getting quite bored of this by now. But anyway...

According to jd on the discussion page, the new owner of the Stag's Head is Louis Fitzgerald, who picked it up for EUR 5.8 million.

Reportedly, he's 'the biggest publican in Dublin' (sic), and owns The Quays in Temple Bar, The Palmerstown House in Palmerstown, The Big Tree on Dorset Street and The Poitin Stil in Rathcoole -- and Kehoe's on South Anne Street. Quite an empire.

I'll have to leave the speculation on Fitzgerald's pros and cons to more recent residents of Dublin, but I agree with jd's comment: 'hope he does half a good as job as the Shaffrys, and the bicycles are left outside rather than on the ceiling,' Amen to that.

The Bayh-Dole Act and publicly-funded research

Published May 11, 2005

Science: in passing -- this came up elsewhere, and it's worth copying here, too (for reference).

The question was: how much should publicly-funded researchers be required to disclose - should they be allowed to generate 'closed-source' solutions at the taxpayers' expense?

In the US and world-wide, there used to be a tradition that government-funded research should be made open to all, since if it was funded from public taxation, the fruits of that taxation should go back to the public. However, 25 years ago, the US enacted the Bayh-Dole Act, in which:

Universities were encouraged to collaborate commercial concerns to promote the utilization of inventions arising from federal funding.
It was clearly stated that universities may elect to retain title to inventions developed through government funding.
Universities must file patents on inventions they elect to own.

So in other words, the government has dictated since 1980 that government-funded research should not produce open-source or public-domain solutions, necessarily, as the results of research are to be considered private-sector profit-generating centers for the host universities. Naturally, cash-strapped universities have imposed internal regulations to maximise revenue from their research staff.

The implications for whatever 'the next BSD TCP/IP stack' may be are obvious.

Stag’s on the block today

Published May 11, 2005

Dublin: Lean forwards on this story from today's Irish Times. Sadly, it's behind their subscription firewall, so I'll just snip out a few choice quotes from Philip Shaffry, the current owner:

'(The Stag's Head) has been part of my life for three decades and I've been running it for 10 years,' he says. 'I've two small children and I'm living 10 miles out of town, so I'm hoping to find a pub a bit out of the city centre. But of course I'll miss this place. I have got really attached to the clientele and the crowd that comes in.'
Looking around at the Victorian bar, opulently decorated with mahogany panelling and a red Connemara marble bar counter, Shaffry is confident there will be no changes to the building.
'They won't be able to touch it. This is the crème de la crème, the jewel in the crown, of Dublin pubs. It has been here since 1760, although it was completely refurbished in 1895. This is a grade-one listed building.'

But the bad news?

There are no State laws regulating some aspects of the pub, namely his family's refusal to allow music - live or otherwise - or television in the bar. Any new owner could change this tradition, says Shaffry, which is a source of concern for some regulars. (....)
A spokesman for CBRE Gunne, which will auction the pub this afternoon, says there had been 'enormous interest' in the premises from Irish and international buyers.

Eeek! The guide price is 5 million Euros, if you fancy a shot.

Thanks for Philip for his excellent stewardship -- here's hoping any new buyer will keep his approach. That approach made the Stag's what it is today -- the best pub in Dublin. (In my opinion, at least ;)

PVR Build Log

Published May 11, 2005

TV: I've taken a little time to throw up my PVR build log.

If you're hacking on one yourself, or curious about what it takes, or just like reading cut-and-pasted UNIX command lines -- go take a look!

Tip: secure SSH tunneling for cron jobs

Published May 11, 2005

UNIX: a quick recap of a good tip combo picked up from ILUG recently. To paraphrase Conor Wynne's original question:

What's the best way to set up a secure connection between two hosts, possibly over the internet, using SSH, suitable for use from cron so that it can run via crontab without entering authentication manually?

Barry O'Donovan replied:

I suggested ssh keys without passphrases ... in
http://www.barryodonovan.com/publications/lg/104/ and it includes instructions. ... You can invoke rsync over ssh and specify a specific key with:
rsync -a -e 'ssh -i /home/username/.ssh/id_rsa-serverbackup'

Colm MacCárthaigh followed up with:

You can restrict what commands an ssh account can run in the ssh public key. This is how some of our more important projects (like Debian, FreshRPMS, and a few more) push us updates. The key looks like (jm: all on one line, no space between 'no-pty,' and 'command'):
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty, command="/home/ximian/rsync-ximian-nolog &"
ssh-dss keydata username@blah
So, create a passwordless public key like so, and just change the command to whatver rsync runs.

Combined, that's a useful tip -- I knew about the ssh command restriction technique, but being able to use a specific single-purpose key from the ssh client is very useful.

(updated: mbp mailed to note some missing quotes in Barry's command above; they'd been eaten by WebMake. drat.)

Tip: expand a bash commandline as you type it

Published May 4, 2005

UNIX: another useful tip. Bash supports a wide variety of command line editing tricks; you have the usual GUIish editing (backspace, insert new characters, delete, blah blah) through the GNU Readline library, and in addition to that you have the traditional csh-style history expansion (like '!!' to refer to the previous command typed).

The latter are great, but they won't actually be expanded until you hit Enter and run the command line. That can be inconvenient, resulting in the user being forced to reach for the rodent for some cut'n'paste instead.

Here's a handy trick -- add this line to ~/.inputrc (creating the file if necessary):

Control-x: shell-expand-line

Start a new bash shell. Now, if you type CTRL-X during command line entry, any shell metacharacters will be expanded on the current command line. For example:

% echo Hello world
Hello world

% echo Hi !$       (press CTRL-X)
           (current command line expands to:)
% echo Hi world

There's a few more commands supported, but none of them are really quite as useful as shell-expand-line.

Update: 'Smylers' wrote to point me at this UKUUG talk from 2003 which discusses .inputrc expansions, and provides some insanely useful tips.

In particular, Magic Space clearly knocks this tip into a cocked hat, by performing the expansion on the fly as you type the command, with no additional keypresses -- amazing! Bonus: it works if you use Emacs-mode line editing as well as Vi-mode.

I strongly recommend reading that paper -- lots of other good tips there.

Sony coins new name for vapour

Published May 2, 2005

Patents: New Scientist: Sony patent takes first step towards real-life Matrix:

IMAGINE movies and computer games in which you get to smell, taste and perhaps even feel things. That's the tantalising prospect raised by a patent on a device for transmitting sensory data directly into the human brain - granted to none other than the entertainment giant Sony.

It's a very lame 'first step' though -- Sony has done no research and development on this invention whatsoever, it's just a patent form of the old 'in the future, we'll wear tinfoil suits! And here's how they'll probably work!' speculation. Sony's comment:

Elizabeth Boukis, spokeswoman for Sony Electronics, says the work is speculative. 'There were not any experiments done,' she says. 'This particular patent was a prophetic invention. It was based on an inspiration that this may someday be the direction that technology will take us.'

That's nice; I'm sure they have some in the pipeline for flying cars, too.

It's good to know that if an inventor does eventually come up with an ultrasound-based human-computer brain interface, they'll have to pay license fees to Sony so they can use their 'prophecy' in their invention. The USPTO's high standards are being maintained, as usual...

Forfás Intellectual Property Lecture Series

Published April 27, 2005

Ireland: Worth watching for european software-patent watchers, Forfás, Ireland's 'national policy advisory board on enterprise, trade, science, technology and innovation' are running a series of monthly seminars on 'Intellectual Property' in association with Licensing Executives Society Britain and Ireland.

This one looks quite interesting -- 10 June: 'Patenting Software - The Current State of Play', Author Barry Moore, of Hanna Moore & Curley, patent attorneys.

Interested parties can attend with pre-registration, or wait to download the mp3 at Forfás' website, apparently, along with the rest of the lecture series. (No sign what the license is on those files, though ;)

the ISA has a new chair

Published April 21, 2005

Patents: It seems the Irish Software Association has a new chairperson, namely Bernadette Cullinane. Whether this has anything to do with Cathal Friel's 'out of line' statements, who knows...

John McCormac passed on some interesting quotes from an Irish Times interview, which were also syndicated here:

'The incoming chairwoman of the Irish Software Association (ISA), Bernie Cullinane, has pledged to support the introduction of a proposed European Union directive on software patents.
She also warned members of the European parliament against blocking the controversial new directive or weakening it by proposing a host of amendments. ...
Ms. Cullinane, a former chief operating officer of the Irish company Performix said European firms needed to protect their intellectual property in a similar manner to the way US firms can.
'We don't want any further dilution of the current situation on patents,' she said in an interview with The Irish Times following her ratification as chairwoman of the ISA last night.'

My emphasis -- given that the current situation is that they are unenforceable in Europe, that's good, because we on the other side don't want a dilution either!

'We do need to look at how the US is developing its software industry and a removal of the patent (sic) could weaken venture capitalists' appetites for investing in new innovative companies.'

The whole 'venture capital requires patents' line is easily debunked. I'm sure the VC companies are telling Ms. Cullinane that they want patents, of course; it's just that they're wrong. ;) Laura Creighton, a European investor, gave a fantastic speech in Brussels in 2003 about investment and patents:

Software Patents (in the US in the 1990s) encouraged venture capitalists to make foolish investments, because they believed the patents were worth something. Venture capitalists often do not mind if the companies where they have invested go bankrupt -- as long as they hold title to the patents. They can start over again with a different team.
Sadly, when the bubble burst, the venture capitalists discovered that their patents were only good for a trip to court -- or at least some legal wrangling with a bunch of lawyers. A software patent is not like a hardware patent, where typically one, or at most a few covers the whole invention. Dozens, sometimes hundreds of patents, are relevant to any piece of software. So an investor, who now owns the assets of a defunct company -- cannot take its patents and hand them to a new development team and say 'build this'. It is impossible to develop software today without infringing somebody's American patent.
The venture capitalists, having lost fortunes backing companies which had no real product, are now uninterested in investing in any software companies whatsoever. Right now the American economy could benefit from more investment -- but the capital is not going into software companies. Again, part of the problem is software patents. The venture capitalists have learned that all software is in violation of somebody's patent. So they do not want to touch the stuff. Thus on the up side, and the down side, the existence of software patents have contributed to creating the stock bubble, and making the recovery slower and harder than it needed to be. So #4 is right out -- the existence of software patents are inhibiting investment right now, and for very good reason.

In other words, the presence of software patents has 'weakened venture capitalists' appetites for investing in new innovative companies', as Ms. Cullinane put it.

Anyway -- to keep the VCs happy, small companies can still obtain software patents in the US, and spend the tens of thousands of dollars required to register and enforce them in court, if they so desire. They can bring the US software industry to a legal standstill if they like, as they seem to have done, as long as European software developers can quietly carry on developing software for use outside the US ;)

But at least things aren't as bad as the situation with my neighbours -- I live a few miles from the offices of Acacia Research, the notorious patent trolls, who've just initiated a new lawsuit against Intel and TI.

Reportedly however, they're planning to open a European office this quarter...

The Stag’s Head days may be numbered

Published April 21, 2005

Dublin: This is it -- it could be the end of an era. CB Richard Ellis auctioneers have a page up noting a new property to be auctioned on Wednesday 11th May 2005 -- The Stag's Head, 1 Dame Court, Dublin 2:

The Stag's Head is one of Dublin's most famous and finest landmark licensed premises, with many outstanding Victorian features.
The bar is lavishly appointed with many fine Victorian features from the beautiful mahogany panelling through to the red Connemara marble counter and the ornate stained glass windows.
Accommodation briefly comprises ground floor traditional style bar with feature mahogany and marble topped bar counter and terrazzo flooring with a snug area to the rear with ornate stained glass skylight. On the first floor there is a further lounge bar area with feature bay window. On the second floor there is a large catering kitchen, dry goods store and office. In the basement there is a further lounge bar area, cold room and toilets.

Many nights were spent in the Stag's Head partaking of their excellent Guinness. It used to be my local, at one stage, and I still drop back in for a night when I get the chance. Save the Stag's!

As my mate Ben put it --

The new owners will doubtless get rid of the (moth-eaten, stuffed) fox, put in recorded music and big-screen televisions, hire bouncers, open on Sundays, extinguish the distinctive odour of damp, replace the marble with formica, and dig up the Dame St mosaic and trade it to the Russian Mafia for heroin and trafficked women. Evil bastards.

Anonymous blogging made simple

Published April 20, 2005

Privacy: after reading Adam Shostack's weblog posting about private/anonymous blogging, I've been driven to think about that, and would up writing up a case study of Cogair, which was an influential anonymously-published proto-weblog in Ireland in the '90s.

Now, quinn at ambiguous.org quotes a review of EFF's recent 'anonymous blogging' guidelines, which largely comes up with one conclusion: it's a usability nightmare. The problem is, the EFF report recommends using invisiblog.com, which in turns uses the Mixmaster remailers. Those things are awful, and I doubt anyone but their authors could possibly know how to use them ;)

Here's an easier way to blog anonymously. I haven't tried it (honest ;) but from keeping up on this stuff, it should work...

Firefox

First off, install Firefox. No point giving your identity away through an MSIE security hole. Clear out all cookies in Preferences:Privacy:Cookies (or better still -- start a new Firefox profile from scratch).
Visit IPID and note down the IP address noted (this is your own, traceable, IP address).

Tor

Next, install Tor, EFF's 'Onion routing' anonymizer system. This also means installing privoxy as directed in the Tor install guide.
Set up Tor on your machine, so that Firefox will browse via that software.
Using Tor, visit IPID and make sure it doesn't give you the same traceable IP address. This is to make sure you're browsing securely.

Hushmail

visit Hushmail and create a new free email account. Obviously, don't use usernames and passwords that map in any way to your existing ones, and avoid words that may show up under your interests (especially if they're googleable)...

Blogger

Using that Hushmail account as the email address, go to Blogger.com and create yourself a blog, then get publishing.
Hey presto -- anonymous blogging the easy way!
For safety, don't use the Firefox anonymous-blogging profile for any sites other than Hushmail and Blogger.com's publishing end. (A future Firefox vulnerability could expose personal info directly from Firefox itself.)

This is essentially the 'TOR to blog server' method described at the privateblogging wiki.

Now, note that along that chain we have 3 levels of identity -- the IP address (hidden by Tor), the email address (traceable to Hushmail, who could conceivably give up the Tor router's IP), and the Blogger.com weblog site (traceable to Blogger, who could give up the Hushmail address and the Tor router's IP).

As long as you don't give it away in your writings on that weblog -- and as long as Tor remains safe -- your own identity in turn is safe, too; and Tor has proved safe, so far.

There are still problems:

The weblog site itself could still get taken down, e.g. via a DMCA takedown notice. This could be an issue, depending on what's being published.
Tor traffic is identifiable as such as it traverses the internet. For bloggers in countries with a pervasive internet surveillance regime at the local ISP end, the watchers will be able to tell that Tor is in use, and tell who is the person using Tor. (They won't be able to tell what it's being used for, just that it's being used.)

PS, for the future: the guys behind Tor are working on a replacement for Mixmaster anonymous remailer software, called Mixminion. There's also a wiki for discussion of 'private blogging' here.

MythTV and KnoppMyth progress

Published April 19, 2005

TV: here's a quick update on my PVR box progress. I have a very extensive /etc/LOG which I should probably just publish as-is, really, rather than trying to make it legible ;)

Anyway, the hardware arrived last month, but the main VIA EPIA ME6000 board was non-functional -- it could never get as far as powering up the CRT for the BIOS self-test. So it was RMA'd back to http://www.mini-box.com, and they sent out a replacement, which arrived a couple of weeks ago.

I finally got to checking this out the weekend before last, and hey presto, it powered up nicely. There followed a whole week of busy nights doing a load of cautious hardware hooking-up, not-so-cautious KnoppMyth installation, and thoroughly non-cautious hacking crazily at the desired enclosure with a hacksaw (because I was too cheap to buy a Dremel).

Things got a little hairy with respect to CPU temperatures, but some looking at specs (the VIA Eden CPU can deal with up to 90 degrees C!), and repurposing of a bin-bound case fan together with some soldering and snipping, has that under control.

Eventually, we're now at the stage where it can:

watch live TV in perfect realtime, pause, rewind, timeshift, ffwd, etc. (the PVR-350 output is good)
record our desired shows (bloody Antiques Roadshow! argh), according to the TV schedule
play mp3s
be ssh'able and sftp'able via a wifi USB dongle
expose its schedule and allow recording via MythWeb
expose its full desktop UI via x11vnc

and it looks good doing it, too. Credit goes to the MythTV guys for a fantastic job on their project, especially with its well-polished UI.

In addition, I have to plug KnoppMyth heavily. They're dealing with an awful situation with hardware compatibility where bleeding edge features like MPEG2 decoding and TV out are concerned, and doing a great job -- there's been several occasions where I've been staring down the barrel of a daunting patch/rebuild/test cycle, and then find out that KnoppMyth includes that component built-in for free.

But -- on the other hand -- no credit to the hardware vendors. As I link-blogged yesterday, VIA is doing the classic 'throw it over the wall' trick with respect to their linux support -- video drivers are written and deposited on their website, with scant documentation and virtually no support.

That's bad enough, but even worse is the situation with Hauppauge's PVR-250 and PVR-350 TV encoder/decoder cards. I realised soon into the setup process that other options for these should have been considered -- Hauppauge have done a great job at confusing the issue for driver developers, as far as I can see. Here's an example. When you buy a 'WinTV PVR-350' card, you may get the same box with the same manuals etc., but including these bonuses under the covers:

one of seemingly about 5 different tuner chips, which you'll need to edit /etc/modules.conf for;
one of about 3 different remote controls with differing output codes;
a good chance you'll have to enter two mysterious ioctls to fix the colour registers, because recent PVR-350 models have changed these somehow and everything shows up as purple-on-green through its TV-Out.

It's absurd. The results are threads like this and a truly daunting setup procedure, which (of course) everyone blames on the software (and Linux itself).

Anyway -- how am I doing vs. Brendan's progress? ;)

pro: my X display sizes are good
pro: no need to switch audio outputs
pro: I'm not using a separate cable box, so no need to hack up something IR to switch channels for me
con: I can't yet watch AVIs or other video files, which I think he has working.

More on the latter when I eventually solve it. (it's tricky. I suspect I'll need to run two X servers with two TV-Outs to do this acceptably, and that's uncharted waters.)

More ways malware damages internet infrastructure: DNS servers

Published April 15, 2005

Malware: spotted on NANOG -- Six PCs caused BigPond problems:

Disconnecting six compromised personal computers on Tuesday evening eased the difficulties caused by bogus requests which clogged BigPond's domain name servers (DNS), slowing customer e-mail and Web site access, Telstra said.
A Telstra spokesperson said the carrier had narrowed the list of malware that could have infected the computers to three, adding the problem could have been caused by a combination of those viruses or Trojans. He declined to name the suspects.
He said the PCs generated 95 percent of the bogus requests which caused the problems that evening.

The 'problems' in question are described here :

One forum participant (on Aussie forum Whirlpool), who claimed to be a BigPond customer, said on Monday: 'I'm in Canberra and it's been almost unusable all afternoon. I'm snowed under at the moment and it is really driving me crazy. Three out of four links fail to load first time and sometimes take eight or nine tries before it does.'
Another said: 'I am having problems loading Web pages, I get the 404 error. I have to retry five to 10 times to get some places.'

Petri Helenius, in a post to NANOG, notes:

Consumer ISP's who don't proactively take care of security/abuse usually end up with harvesting-bots which consume significant amount of DNS resources, typically doing anything from a few dozen to a thousand queries a second. A few hundred of these will seriously hamper an usually provisioned recursive server.

Interesting. It's been a long time since I've relied on an ISP's recursive DNS servers; in my recent experience (Comcast, Cox.net) they've always been overloaded, and take aaaages to give me answers. Maybe this is why.

It makes sense; most Windows machines will indeed use the ISP's NSes, because that's what DHCP tells you to do; and setting up a BIND or djbdns instance locally to query the roots directly is still a UNIX-only trick, as far as I know.

The upshot?

1. Yet another good reason why ISPs should proactively disconnect infected customers, as they deny service to other users of the ISP.
2. A good demonstration of yet another way the techie community's experience of web surfing and internet use differs from that of the unwashed masses in the hinternet -- that 'shanty-town of pop-ups and porn adware', as Danny O'Brien puts it.
3. Sometime soon, if it hasn't happened already, someone's going to bundle up an 'Internet Accelerator' lump of shareware that sets up a local recursive NS on Windows which queries the roots, and it'll become the latest popular Windows download. Then the load on the root servers will really start rising.

(PS: top tip -- ever wanted a publically-queriable recursive nameserver, or a good IP address for pinging, that's easy to remember? 4.2.2.1 is what you're after.)

Congressional Open URL Redirectors

Published April 14, 2005

Spam: Matthew Wilson at Boomer Consulting has been having a field day -- it looks like some smart google hacking has thrown up some doozies of places that should have fixed this by now:

and my favourites:

Of course, all of these are immaterial to SpamAssassin -- we catch spammers using them anyway. But still, a surprising number of these out there.

pick a ‘flu, any ‘flu — well, maybe not that one

Published April 13, 2005

Health: Meridian Bioscience Inc. of Cincinnati, Ohio mails lethal pandemic strain of 'flu to nearly 5000 labs in 18 countries:

The firm was told to pick an influenza A sample and chose from its stockpile the deadly 1957 H2N2 strain.

Check out how it was spotted:

On March 26, National Microbial Laboratory Canada detected the 1957 pandemic strain in a sample not connected with the test kit. After informing WHO and the CDC of the strange finding, the lab investigated. It informed the U.N. health agency on Friday that it had traced the virus to the test kit.

My emphasis. omgwtfbbq!

(WHO's influenza chief) Klaus Stohr said the test kits are not the only supplies of the 1957 pandemic strain sitting in laboratories around the world. 'The world really has to think what routine labs should be doing with these samples they have kept in the back of their fridges,' Stohr said.

True: the lovely C has a story from her TCD days of a vial of smallpox
found buried deep in the ice in the back of a long-forgotten freezer, apparently rediscovered by someone during a routine spring cleaning. This was in the early '90s, when smallpox was supposedly down to samples in just two high-security labs, in Russia and America.

Interesting fall-out from the Irish Times Microsoft supplement

Published April 12, 2005

Open Source: on the 18th March the Irish Times published a commercial supplement for Microsoft. Naturally, given that it was paid advertising, there were lots of MS plugs -- but in the mix there was also a couple of more worrying articles: one by Tom Kitt, government 'Minister for the Information Society', noting

Microsoft has been one of the most innovative companies in the world and has a long track record over several decades of creating new product markets. The EU has to be open to allowing such innovation in Europe. Ireland will continue to argue at EU level, based on the solid evidence of our successful economy, that the Community must look at its rules on innovation and intellectual property rights to ensure they encourage risk taking in Europe and growth in the IT industry in the EU and around the globe.

And another with Cathal Friel, credited as 'chairman of the Irish Software Association'. Quoting the article text:

(Friel) also noted that Open Source software - which is developed by large communities of programmers and distributed for free or at low cost - is also going to have an effect on the software market. While Friel believes Open Source itself has a limited business model - 'at the end of the day, there's nothing but services to sell' - it is nonetheless becoming more pervasive and is 'a fact of life' for more traditional software companies. He believes the Open Source movement is actually stifling innovation, because fewer programmers will develop software without the financial incentive of success.

MS observers will note that both Kitt and Friel's statements mirror the MS 'party line' -- either the lads were well-briefed, or they just put their names to a story written by MS PR.

Well, there's been an interesting follow-up. Éibhear Ó hAnluain put pen to paper about Cathal Friel's statements, and received an interesting reply:

I received a 'phone call from Kathryn Raleigh, Director of the ISA, in reponse to my letter. As I was unable to take notes at the time, what follows is a memory of the conversation. She told me that the ISA would like to apologise to me for any offense that I took from the comments. She said that the first the ISA heard of the comments was after the piece was published and the Mr. Friel was not speaking with the ISA's authority. She told me that the ISA had indeed conducted some sort of analysis of the market regarding licensing and the 'proprietary' versus Free Software competition, and that the ISA's position on the matter is not to have a position. She gave me the impression that Mr. Friel has been told that he was out of line. She asked me to convey the ISA's regrets to my colleagues.

Well now, that's interesting!

I find it very encouraging to see that the ISA don't take the position noted in Friel's article, anyway. In my opinion, this is wise -- alienating free software and open-source-using companies doesn't seem likely to be a good idea, given that many of today's SMEs use open source extensively 'behind the scenes' in production, if not directly in the products they sell.

There's also the matter of Google's recent major entry into the Irish software industry, with its new offices in Barrow St. in Dublin. MS are no longer the only major multinational player on the Irish scene to whom open source's success, or failure, is a key factor in their business plans. Google use free software extremely extensively internally, are members of several major free software bodies including the FSF, and have released quite a few interesting pieces of open source software themselves.

Spam and Broken Windows, and wecanstopspam.org

Published April 11, 2005

Spam: Spam Chongqing: Spamming Experiment:

Kasia at unix-girl.com decided to run a spamming experiment on her blog. She posted a couple spams to her own blog and waited to see what would happen. In less than 24 hours she received 356 more spams.

The chongqing guys confirm this, and I've noticed this as well (although just in passing, I've never tried testing it).

Interestingly, I'm pretty sure the same thing can happen with mailing lists, if the mailing list archives are allowed to contain the mailing list's posting address, and the list allows open posting. It works like this:

spammer A posts a spam to the list
spam is archived
google finds archived spam
list-builders B, C, D google for search terms, find archive page for that mail message
B, C, D scrape the addresses from that page and pick up the list posting address
they then either sell on to spammers E, F, and G, who spam that address, or they spam the address themselves
and redo loop from the start.

One key factor is the search terms B, C, and D use. My theory is that they are intending to generate 'targeted' lists, and in spamming, most targeted lists are simply lists of addresses scraped from pages that show up in a google search for a specific keyword -- 'meds', 'viagra', 'degree', etc.

Joe at chonqing surmises that it may be through the Broken Windows Theory -- that spam appearing in a weblog's comments, or in a wiki page, indicates that the administrator is asleep at the wheel and more spam can be posted with impunity. in my opinion, that's probably more likely for google-spam and wiki-spam than for email spam, but undoubtedly is a factor.

PS: href="https://web.archive.org/web/20050529025334/https%3A//chongq.blogspot.com/2005/04/another-spammer-owned-antispam-site.html"> wecanstopspam.org has been allowed to lapse and has been stolen by a spammer. Oh dear.

Nose Leeches

Published April 11, 2005

Health: On a lighter note, I've been getting through my last two weeks mail and RSS data, and came across this beauty.

It's a truly venerable internet urban legend -- the Nepalese Nose Leech story. Even given that I assumed it was more than likely a UL, I still took care not to drink from streams when I visited leech-infested areas, especially in Nepal!

Well, it appears it may not be a UL after all --

Doctors have removed a leech from the nose of a 55-year-old Hong Kong woman after she swam and washed her face in a stream, a medical journal reported.
The woman went to her doctor complaining of nose bleeds and an occasional sensation that something was blocking her left nostril, the Hong Kong Medical Journal said in its April issue. Her family doctor noticed a brownish mass in her nostril but couldn't remove it because of heavy bleeding, the journal said.
The patient was taken to the emergency room, where doctors identified the problem as a bloodsucking leech. They had trouble pulling it out because the 2 inch invertebrate retracted into the nostril and disappeared, the journal said.
Part of the slimy leech was in a passage of her nasal cavity and a larger segment was in her sinus cavity, the article said.
Doctors used a nasal spray to anesthetize the dark brown leech that had a sucker on the front part of its body. After two minutes, the leech moved slowly out of the antrum (sinus) and was retrieved with forceps, the journal said.
The woman said that one month before her symptoms developed, she swam and washed her face in a stream while hiking. Doctors checked other members of her hiking group and found another leech in the nose of a man who washed his face in the stream, the journal said.

Link via jwz, AP wire story, abstract at Hong Kong Medical Journal site, MEDLINE abstract, including a line noting 'this form of leech infestation has not been previously reported' -- except on teh internets!

Where I’d gotten to

Published April 11, 2005

Meta: You might have noticed things being a bit quite around here recently. Unfortunately, it wasn't for good reasons.

A close family member in Ireland died suddenly on Good Friday. Once we found out, being in Death Valley (of all places) that weekend, we made a mad dash back home for the removal, funeral, and so on. The past two weeks have been not so much fun, all in all.

I'm torn between eulogising here, and keeping it offline. All in all, I think it'd be better to not use this weblog for that; I don't think it'd be appropriate. But he'll be greatly missed.

Reorganisation, and ancient history

Published March 23, 2005

Life: Alec Muffett quotes an Economist opening line:

We tend to meet any new situation in life by reorganising, Petronius Arbiter, a 1st-century Roman satirist, is supposed to have remarked. And what a wonderful method it can be for creating the illusion of progress while producing confusion, inefficiency and demoralisation.

As apt today as it was then.

(I was recently talking to a mate who's a post-grad in the classics. She noted that classicists aren't the fastest-moving academicians around, speculating that maybe it was because, in studying the classics, you realise the same problems and the same solutions have been around for over two thousand years regardless of change in other aspects of life.)

Bad Blogger.com Security Model

Published March 22, 2005

Security: Hey user auth systems! If you're going to require me to sign in, and publish my login as a signature to prove that I'm 'me', please do me a favour -- don't delete the account if it's been 'inactive', and allow anyone to re-register that name without my knowledge!

I just tried to leave a comment on a Blogger.com weblog, to find that my user account at Blogger had been deleted. Re-creating a new account with the same name wasn't a problem -- the previous account data had been simply deleted outright. (Presumably they don't do this to people with a Blogger.com weblog -- I hope.)

The risks of this are pretty clear; given that I'd already established an identity (at least in comments on certain Blogger weblogs) as 'justinmason23', if an attacker were to have re-registered that identity before I did, they could impersonate me.

Massive topographical map of Ireland

Published March 21, 2005

Mapping: NASA's Earth Observatory has put up a 4 MB high-res topographical image of Ireland. A rough calculation indicates that each pixel is under 0.1 of a mile on a side. It's fantastic. ;)

Best of all, since NASA operate under the US' enlightened copyright and licensing policies for government-funded data, it's free -- the masthead notes 'Any and all materials published on the Earth Observatory are freely available for re-publication or re-use, except where copyright is indicated. We ask that NASA's Earth Observatory be given credit for its original materials.' Copyright is not indicated on this image as far as I can see. So go ahead and save a copy for future use, too.

(via EirePreneur in turn via Irish Typepad)

Open API for online group-based services maintainance

Published March 19, 2005

Web: I've been doing a little thinking about group-based networking and services.

Here's the situation. Let's say you have a small group of people, and want to offer some kind of online service to them (like a private chat area, mailing list, etc. etc.) That's all well and good, but maintainance of 'who's in the group' is hard. You need:

the ability to let other 'admins' add/remove people
a nice UI for doing so
a nice UI for people to request to sign up
possibly, multiple groups
privacy for group members
possibly, some public groups
decent authentication, username/password
the usual stuff that goes with that -- 'I've forgotten my password, please email it to my listed address'
did I mention a nice UI?

The traditional approach is to code all that up myself, in my copious free time presumably. Urgh, talk about wheel reinvention on a massive scale.

I'd prefer to use something like TypeKey, a web service that exposes an API I can use to offload all this hard work to. Initially, I was in the 'ugh, Typekey 0wnz my auth data' camp, but I've eventually realised that (a) they're not quite as evil as MS, (b) they're not quite as stupid as MS (deleting Passport accounts if you don't log in to Hotmail, which is only one of the supposedly many services, including third party services? hello?!), and (c) it's actually really convenient having a single-sign-on for weblog commenting after all.

Having said all that -- TypeKey's out. Unfortunately, it only does authentication, without dealing with group maintainance.

However, social networking services are all about groups and group maintainance.

Running through the options -- LinkedIn, Friendster and Orkut are all grabby and gropy and 'my data! mine!', so they're out immediately.

The next step was to take a look at Tribe.net, which seems kind of nice and had a good rep for open APIs -- but as far as I can see, all they've got really in that department is FOAF output, and a simple server-side-include thing called TribeCast. I could list all the group members in a FOAF file, but without authentication, that's pretty useless since anyone could claim to be one of the FOAFs.

That leaves Flickr, which has a great set of APIs. Using that is looking quite promising. If you're curious, I've gone into detail on this at the taint.org wiki.

DCC no longer open source

Published March 19, 2005

Patents: DCC (Distributed Checksum Clearinghouse) is a venerable, and widely-used anti-spam system created by Vernon Schryver; we've supported it in SpamAssassin for yonks.

It now appears that DCC is now no longer open source software; it's still free for personal and noncommercial use, but this clause has been added to the new license text:

This agreement is not applicable to any entity which sells anti-spam solutions to others or provides an anti-spam solution as part of a security solution sold to other entities, or to a private network which employes DCC or uses data provided by operation of DCC but does not provide corresponding data to other users.

So there's talk that those commercial users should now license it -- interestingly, from another company called Commtouch, not Vernon's Rhyolite Software. (More info).

It appears that the license change is part of an agreement with Commtouch, owner of US Patent 6,330,590, a patent on the idea of hash-sharing antispam techniques. (I haven't read the patent due to ASF and other policies so I can't tell you what it really covers.)

It looks like we'll be disabling DCC's use in SpamAssassin by default, as we did with Razor, as a result. (Our policy is that the default ruleset used in SpamAssassin be usable by anyone who can use our software, so that the normal usage is open source by default, rather than subsets of the overall functionality.)

Greasemonkey: transcoding extension for Firefox

Published March 16, 2005

Web: Now this is very cool stuff: 'Greasemonkey is a Firefox extension which lets you to add bits of DHTML ("user scripts") to any webpage to change it's behavior.'

In other words, you can rewrite any page viewed in Firefox, as it transits between the server and your client's display; a form of transcoding.

Traditionally, transcoding is performed using a HTTP proxy which applies the transformation, or a specialised HTTP user agent which transcodes and outputs a whole new set of documents with the results.

That was all a little hacky for full-scale integration into your web browser, though, so Greasemonkey is a big improvement for that use-case.

Some good links:

And some demos:

Remember, these are single, sub-100-line JS scripts, running entirely locally in the user's web browser. The last one gives you an idea of what coolness is possible...

My contribution: an ad-removal script for Metafilter. It took some 30 seconds of hacking to produce this -- soooo easy. It's a whole new world of site customisation and hackable filtering. You thought AdBlock was good, this is ever niftier ;)

Taxation: a Happy Ending

Published March 16, 2005

Tax: Following on from the previous entry, I've had a stroke of luck. It turns out that I did indeed quality as a US resident for tax purposes, and therefore could use Turbotax.

20 minutes later, both state and federal forms were e-filed with the very minimum of fuss -- computers and the net illustrating their worth as labour-saving devices quite nicely. ;)

(Oh -- also -- a PSA for Google's benefit: I'm pretty sure that form 6251 had incorrect instructions. It claims that the items it refers to in form 1040, can also be referred to in form 1040NR by the same numbers. In fact, parts of 1040NR are radically different in numbering than 1040. It's a bug in the form!)

Taxation Ventage

Published March 14, 2005

Taxes: it's that time of year again, when every inhabitant of the US, resident and 'non-resident', gets The Fear. Yep, it's tax time. (Warning: this is a long and protracted vent.)

In the US, every worker is required to prepare and file their own taxes, in detail. Nowhere outside of India can do bureaucracy quite like the US, as far as I can tell -- even the brits have embraced simplicity to a greater degree -- so this is no trivial undertaking; however, they do have a few outs, if you're eligible.

Naturally, given my luck, I'm not. ;)

Now, I'm no slouch when it comes to form-filling; I've had to deal with messy forms many times before. But these are masterpieces. Check out this gem:

The ATNOLD is the sum of the alternative tax net operating loss (ATNOL) carryovers and carrybacks to the tax year, subject to the limitation explained below. Figure your ATNOLD as follows.
Your ATNOL for a loss year is the excess of the deductions allowed for figuring AMTI (excluding the ATNOLD) over the income included in AMTI. Figure this excess with the modifications in section 172(d), taking into account the adjustments in sections 56 and 58 and preferences in section 57 (that is, the section 172(d) modifications must be separately figured for the ATNOL). For example, the limitation of nonbusiness deductions to the amount of nonbusiness income must be separately figured for the ATNOL, using only nonbusiness income and deductions that are included in AMTI.
Your ATNOLD may be limited. To figure the ATNOLD limitation, you must first figure your AMTI without regard to the ATNOLD. To do this, first figure a tentative amount for line 9 by treating line 27 as if it were zero. Next, figure a tentative total of lines 1 through 26 using the tentative line 9 amount and treating line 27 as if it were zero. Your ATNOLD limitation is 90% of this tentative total.
Enter on line 27 the smaller of the ATNOLD or the ATNOLD limitation.
Any ATNOL not used may be carried back 2 years or forward up to 20 years (15 years for loss years beginning before 1998). In some cases, the carryback period is longer than 2 years; see section 172(b) for details. The treatment of ATNOLs does not affect your regular tax NOL.

That pretty much appears as-is; there's no additional explanation of those acronyms elsewhere, it's just a big block of jargon. Obviously not intended for human consumption. There's also this:

Medical and Dental. Enter the smaller of Schedule A (Form 1040), line 4, or 2.5 % of Form 1040, line 37.

That seems well and good, and according to the instructions, the 1040NR is 100% compatible with the 1040. Except Schedule (Form 1040NR), line 4 is:

Gifts to U.S. Charities. Gifts by cash or check.

What do charity donations have to do with medical and dental expenses? WTF? (I suspect the compatibility claim is incorrect.)

Last year, I hit up H&R Block for their help; it saved a lot of hassle, but was pretty expensive, costing over $200. Overblown TV advertising alert: of course there was no great refund, despite what their ads claim. However they did recommend that I donate old clothes to thrift stores, keep the receipts, and claim that back as a tax contribution. I'm serious. Given my wardrobe, that should net about $10.

This year should be a lot simpler, since I'm just a US nonresident working visa holder doing nothing but paying plain old income tax -- so I was intending to just fill out the forms myself.

I think I'll tick that idea off my list and check out the online options.

All I can say is, no wonder quite a few US citizens seem to think that government involvement is something to be minimized if at all possible. There are alternatives though -- I'd happily take an Ireland-style 'nanny state' which will compute my tax liabilities for me if I so choose. It's not like I'd be in a position to argue with them anyway, aside from the common case of hiring a tax attorney, if we disagree; so why not let the government do the heavy lifting? ;)

(PS: the good news is it now appears I may qualify as a resident. This means Turbotax.com is a viable option... yay!)

Back in the US, and Daniel’s interview

Published March 8, 2005

Misc: So I was travelling last week -- a very productive trip to the UK visiting the main work dev office, and getting a little socialising in too while I was at it. A pretty good trip overall, especially since I seem to have figured out how to use my frequent flyer miles effectively to get great seats! ;)

Here's a good interview with SpamAssassin PMC chair, Daniel; well worth a read if you want to see what we in SpamAssassin think about the state of the onion in spam-filtering.

In not-so-good news, it seems Charlie McCreevy has managed to push the software patent directive through, despite massive EU Parliament unhappiness. Third time around at the Fisheries meeting, naturally; and there's some serious questions about the legitimacy of the procedural rules invoked by the Commission in refusing to take the directive off the A-item menu. Now that's what I call democracy...

It can still be defeated, but it's an uphill battle now -- for it to be thrown out in the second reading at the European Parliament, it'll need a two-thirds majority of all MEPs (not just the MEPs present), reportedly.

In the meantime, thanks to the FF and PDs' bullying tactics, Ireland's small but growing pool of homegrown software developers are being ignored, and the Irish software industry looks more like a lame import operation for the likes of Microsoft. Our reputation is dragged through the mud for a few multinationals, and the rest of Europe resents us for it. Wonderful.

BTW, even if it does pass, there are ways to fix it -- directives must be implemented into national law in each country. This means that Ireland could still write their implementation of the directive to exclude software inventions (even the ones where it's supposedly a patent on hardware like 'a CPU connected to a hard disk, with such-and-such software running on the CPU'). However, given McCreevy's obvious bias in favour of getting this specific text into place, how likely is that going to be?

RFID Scan Detector

Published February 23, 2005

RFID: Over on Adam Shostack's weblog, in a comment on an entry regarding the plans to mandate remotely-readable RFID passports, Martin Forssen brings up a great idea:

What I want is a device which beeps every time somebody scans me for RFID-tags. I assume this would be fairly easy to construct since the scanner must send a signal of some strength to activate the chip.

I wonder if that'd work? A keyfob, for example, something similar in size to the dinky Chrysalis Wifi Seeker I have on my keyring, would be perfect. It'd be probably pretty cheap to make, would make a great geek toy, and be quite educational too. ;)

Amazing quotes from Michel Rocard

Published February 18, 2005

Patents: So the Conference of Presidents has ratified the JURI decision to throw out the flawed software patents directive text. Phew! That's a lot more pressure on the European Commission. Charlie McCreevy could still carry on his attempt to steamroller European democracy on this one, but it looks likely that he wouldn't get away with it now -- possibly facing sanctions as a result.

Found in a Slashdot comment -- an amazing quote from Michel Rocard (former French Prime Minister, now European Deputy), recounting a meeting with Microsoft representatives on the software-patent issue:

"We never could (speak) a common language with the companies representatives we met - in particular those from Microsoft. Speaking about (the free circulation of ideas), free access to knowledge, was like speaking chinese to them. In their way of thinking, everything that is not usable for immediate profit ceases to be an engine of growth. They don't seem to be able to understand that an invention which is a pure spirit creation (sic) can't be patented. It's simply terrifying. Many of us, at the Parliament, agree to say that they never have know such a pressure and such a verbal violence during their parliamentary work. It is a huge case."

In addition, he takes aim at the Irish Presidency's tactics:

"To adopt it formally, there is an expeditious procedure -- the (A-item) at the Council of Ministers, where the it is adopted without discussion. The Irish and Dutch presidencies attempted this tactic three times, twice at meetings of the (Fisheries Council)! This is simply scandalous."

Blimey, he's really pissed off. Great! Go Rocard! ;)

See here for the original interview (in French), and here for a bad Babelfish translation.

In happier news -- take a look at some pictures from the presentation of 30,000 verified signatures (and flowers!) from people around the world, thanking the Polish Government for their repeated stands against the flawed directive in December.

Continuations in perl

Published February 17, 2005

Code: Ugo Cei: Building Interactive Web Programs with Continuations quoting Phil Windley:

This leads to the question: what if I could write programs for the Web that were 'structured' in the programming sense of that word? The result would be Web programs that were more natural to write and easy to read. You'd no longer have to maintain the state of your program outside the language and the data could be kept in variables, where it belongs. The answer is: you can.

I hate the 'save all state' model imposed by developing for the web, and have been hoping for a way to do this for a while -- and now I know what it's called ;)

It seems Seaside is the leading continuations-based web-app framework, using Smalltalk, and (as Ugo noted) Apache Cocoon has it too, but there's a whole load more. Can you tell I haven't been following web-app development techniques much recently?

Never mind those other languages, though -- Continuity looks promising as a Perl framework based around continuations. Perl 6 will reportedly have native continuation support, and Dan Sugalski gives a good write-up of how they're implemented and their ramifications there.

BillG threatens to shut down Denmark’s tech sector if he doesn’t get his way

Published February 15, 2005

Patents: Børsen: Bill Gates threatened to kill 800 Danish jobs if Denmark opposed software patent directive:

Danish financial newspaper Børsen reports that Microsoft founder Bill Gates threatened the Danish government in connection with software patents. According to the article, Gates told Rasmussen and two Danish ministers in November that he would kill all 800 jobs in Navision, a Danish company acquired by Microsoft in 2002, unless the EU were to quickly decide to legalize software patents through a directive. Denmark is a country with only 5 million inhabitants and a relatively small high-tech sector to which the loss of 800 jobs would have significant implications.

Lovely -- a blunt blackmail attempt. The article goes on:

It would not be the first threat of its kind. A group of large corporations including Philips is reported to have previously threatened European governments to outsource all of their European software development jobs to low-wage countries unless the EU were to allow patents on software through the directive that is currently being worked on.
In January, leading Polish daily Gazeta Wyborcza reported on a letter addressed by the Polish subsidiaries of Siemens, Nokia, Philips, Ericsson and Alcatel to Poland's prime minister Marek Belka ... it is said to have indicated that the respective companies would reconsider making investments in Poland if the Polish government upheld its resistance to the legalization of software patents in the EU.

Again, note the FUD-busting on this point. I notice that Florian Mueller of NoSoftwarePatents.comhas a a good one-liner response along the same lines -- 'The country in which you develop a technology has nothing to do with where you can take out patents.' He goes on:

If they move jobs to Asia, they won't get a single additional patent, neither in Asia nor in Europe. If you warn politicians of consequences that are directly related to a legislative issue, that's acceptable. If you threaten with causing damage that has no factual connection whatsoever, then it's blackmail. Plain and simple.

Software Patent Legalisation And Its Effects On Research And Development

Published February 11, 2005

Patents: an interesting FUD-busting point from the FSFE-IE mailing list today. Malcolm Tyrrell wrote:

Why does the following point keep coming up? Do I misunderstand the issue, or is this just plain nonsense: (quoting this ENN article)
'Indeed, the big businesses that backed the directive -- such as Philips, Nokia, Alcatel and Microsoft (...) also say, in somewhat ominous terms, that without patent protection, big companies will be less inclined to spend cash on European R&D projects, because the governments of Europe cannot offer any guarantees that commercially useful technology will be protected. In the US, those much-needed safeguards are in place, patent supporters note.'
I presume that these big companies will obtain patents in all territories where patents are available, regardless of where the R&D is performed. Unless they are threatening this merely as revenge (and I would think that there responsibility to their own shareholders precludes this), there would be no more or less reason to do R&D in Europe whether software is patentable there or not. Am I wrong?

He's right; in my experience, software patents are applied for world-wide, in as many regions as possible (and as funds and time permit) -- and there's very little barrier for an inventor in one country to obtain patents in other countries (apart from money to pay for all those billable hours).

However, Fergal Daly had a more interesting additional point:

'As far as I can see you're right and in fact this is a plus for Europe, as labs in Europe would be free to use other people's patents during their research, whereas in other regions they would have to license them before they could implement them, even for private use.'

He's right, too, as far as I can see. This would be quite a big win for European R&D, since it would also mean they could develop an algorithm similar to a patented algorithm, as long as the patented technique was only implemented in software inside their European labs. This would be illegal to do anywhere else in the world where software patents were legal, hence is a competitive advantage over their international competitors.

In addition, it would mean that in the scenario where a product is produced using a patented algorithm, but the algorithm doesn't appear in the final product, that would allow them to perform production in Europe without paying the license fees that would be payable elsewhere.

In summary -- the 'patents needed for R&D' line is FUD, and the reality is in fact the opposite!

Open APIs, Open Source, And Giving Away The Crown Jewels

Published February 11, 2005

Tech: Bit of a long essay, this one.

World+dog have been linking to this interview with Flickr's Stewart Butterfield on the O'Reilly Network, so I wasn't going to bother. But I came across a great illustration of what I think is a very important point:

Koman: In the write-up for your web services session at ETech, you say, Capturing the creative energy of the hive can be scary. It requires giving up some control, and eliminating lock-in as a strategy. Tell me some more about that.
Butterfield: Ofoto is a pretty good example. I don't want to pick on them too much, but they create a pretty artificial kind of lock-in. When you upload your pictures to them, you might upload a three- or four-megapixel image, but all you can get back from them is a 600-pixel image; if you want to get the original back, you have to buy it on a CD. There's no way to get it out because if you got it out, then your friends and family could get it out and print it out at home, and they're in competition with Lexmark and HP as well as the other online photo services. So that's one aspect of it.
There's also a tendency to want to capture all the value that's being generated or will potentially be generated by new business. What I mean by that is, we don't explicitly allow commercial uses of the API yet, but we definitely plan to. And we know that there are people working on products based on our API that we want to do, but outside developers will get to it first. What letting go in that context means is letting go of all the control you have over users by being the one who owns the database, because other developers can generate businesses and products that hook into you, and that takes some value away.

This is a point that still, to this day, most people miss.

The traditional viewpoint is that, if you've got something, you hoard it, and ensure you're the guy who makes the money from it. So you do what Ofoto do -- you keep the full-resolution images, and charge for access to them; or you don't publish APIs, and keep the data to yourself; or in the world of source code, you hold onto the source so no-one else can see it, because it's your 'crown jewels'. Then, the idea goes, you can ensure that you're the only one who can do prints, or add a feature to the source, or whatever.

But the problem is, you're not always the one with the idea; or alternatively, every feature request has to go through you, and be implemented by you, on your time. And in the meantime, your users are considering the big question -- 'do I want to get locked in, here? what if he goes out of business? am I a small customer who's going to be ignored?'

In fact, I've been guilty of this myself. When I started writing open-source software, I used the GPL as a license, which prohibits commercial use (mostly) -- except by myself or through my explicit permission. I had no intentions of making it available for commercial use, because I couldn't see the commercial uses.

But that was me being short-sighted -- soon, people starting asking if they could license the code for commercial use, or hire me. I realised that I didn't have the time, or inclination, to go the whole hog, and risk my livelihood on a piece of software -- especially risky since I didn't think that software could support me alone.

So when I wrote SpamAssassin, I picked the Perl dual license, a license that did permit commercial use, while still being an open-source license. By now, there are quite a few commercial versions of SpamAssassin, all making money (I hope!), I'm getting paid to work on SpamAssassin, and everyone's happy ;)

Perhaps I should have kept commercial rights to myself. But I have no doubt that doing so would have ensured SpamAssassin remained a small-time solution, and would not have received the number of contributors, committers, and patches it has by now. (for example, Matt Sergeant, who was an SpamAssassin committer, joined the project explicitly to use that code in MessageLabs' product.)

Plus, at the time, there were already quite a few commercial competitors -- and there's a lot more to being a commercial success than the simple things required to be an open-source success; I'd be dubious that SpamAssassin would have been able to compete as a purely-commercial play, and I'm not sure I'd have been keen to risk my livelihood to do so, anyway. (I'm not really dot-com CTO material, anyway. I like hacking code too much.)

I think things have worked out well: the software's better, I'm earning a livelihood from open-source software regardless, and the software's usable for more people. As usual, Larry Wall was right ;)

A highlight (or low-light) from the world of spam bounces

Published February 8, 2005

Spam: recently, I've been getting a lot of spam bounces; that is, messages sent by people's autoresponders, in response to forged spam claiming to come from my domain. (I have an SPF record, but these autoresponders naturally don't bother to check that before replying.)

I have a SpamAssassin ruleset which catches these, and it gets rid of the vast majority -- but the odd wierd one gets past. This one caught my eye before I deleted it:

On October 5, 2004, I will be going to the Illinois Department of Corrections for approximately 18 months. If you wish to contact me, please snail mail me at: (address deleted)
Your letters will be forwarded to me and I will reply as soon as I receive them! Thanks...and please do write! Mail is vitally important! :-)

... ouch. Good luck to this guy, whoever he is...

Spamhaus article on ISPs hosting spam gangs

Published February 4, 2005

Spam: Should ISPs Be Profiting From Knowingly Hosting Spam Gangs? -- a new article up on Spamhaus.org, well worth a read. Some snippets:

So where is this stealth proxy spamware sold and distributed from? For Send Safe the answer is, www.send-safe.com, hosted by MCI Worldcom.
... MCI executives have refused to stop providing service to these gangs, insisting that the sale and distribution of stealth spamming software is not against MCI's policy.
... It's no surprise therefore that MCI has consistently occupied first place in Spamhaus TOP 10 World Worst Spam Service ISPs chart, with over 200 spammers and spam gangs on the MCI network in full knowledge of the security managers and the General Counsel.
... MCI Worldcom's official position on the issue is that MCI can't stop their spam gangs selling proxy hijacking spamware from MCI's network as that would be 'censoring' the distribution and sale of illegal proxy hijacking software.

interesting Antarctic factoid

Published February 3, 2005

Antarctic: It seems that Ernest Shackleton, during his exploration of Antarctica, relied heavily on 'Forced March' tablets:

Reportedly 'sold over the counter at Harrod's until 1916', these were primarily cocaine-based.

EU Software Patents law back to square one

Published February 2, 2005

Patents: FFII are reporting that 'the Legal Affairs Committee of the European Parliament (JURI) has decided with a large majority to ask the Commission for a renewed referral of the software patents directive. With only two or three votes against and one abstention, the resolution had overwhelming support from the committee, and all-party backing.'

Michel Rocard MEP gave a very strong speech at the meeting with the Commissioner. Apart from noting several "inelegancies" by the Commission, such as not taking into account any of the Parliament's substantive amendments in its recommendation to the Council, he also took issue with the Dutch and German governments ignoring their respective parliaments, the Irish Presidency's sponsorship by Microsoft and the attempted ratifications of the political (dis)agreement at several fishery Council meetings.
He mentioned that at a meeting with the Polish government, the industry players confirmed that the Council text allowed pure software patents, and wondered how the Commission could continue claiming the reverse. He was also curious about how the Commission's perfectly tautological definition of the concept "technical" could help in any way to distinguish between what is patentable and what is not. Despite his own abstention when voting on the restart later that day, the fact that almost everyone else supported it is probably his personal achievement.
The Commissioner made clear that "any agreement will need to strike a fair balance between different interests", and that "a constructive dialogue between the Council and Parliament will be vital for an agreement". He does have the option to deny a new first reading. But given the strength of feeling in the Parliament and the concerns of so many member states in the Council, the Parliament request looks like the best way to achieve a clean way forward for this Directive that everyone has been looking for.

This is good news for the anti-swpat side. Nul points for the Irish Commissioner, Charlie McCreevy, who 'had in the morning assured the JURI Committee that the Council would finally adopt its beleaguered Common Position text. He announced that "the Luxembourg Presidency has now received written assurances concerning the re-instatement of this issue as an A point at a forthcoming Council". Given that A points are to be adopted without discussion, this left no possibilities for renewed negotiations in the Council'.

interesting sysadmin talk next week in Dublin

Published February 1, 2005

Networking: Donal Cunningham, president of SAGE-IE, mails to note an interesting talk on in Dublin next week:

The System Administrators' Guild of Ireland and Dublin University Internet Society present...
What : From the ground up; a greenfield deployment in Liberia
Who : Comdt. Kieran Motherway, Corps of Comms. and IS, Defence Forces
Where: Walton Lecture Theatre, Arts Building, TCD
When : Tuesday the 8th of February, 7 p.m.
Why : The Irish Defence Forces deployed to a greenfield site in Liberia in 2004, and had to build Comms/IT infrastructure from the ground up. Comdt. Motherway will talk about the Irish Army's experiences with this deployment, and just how far removed from an air-conditioned, climate-controlled comms room you can get...

Sounds like fun, and I know a few taint.org readers will be interested ;)

Building a Freevo

Published January 27, 2005

Freevo: so I'm planning to build myself a PVR, of the home-built, running Linux with mythTV or Freevo, mini-ITX variety.

So far I'm still at the hardware planning stages, but the price looks good -- around $455 (plus shipping) for a working, thoroughly hackable, silent, set-top PVR system.

(Silence is a key aim here -- last thing I want is something noisy taking over the room. But silence typically seems to cost the dollars, once you get into Shuttle gear and the like.)

If anyone wants to follow along, or provide some tips -- I'm going to track progress (very slowly) on this wiki page. Like all wiki pages, it's editable -- although you'll need to create an account to edit pages there (sorry, anti-spam measure).

BTW, lately, there's been a lot of talk about using a Mac mini as a media center. So I took a quick look -- but wow, it's pricey! $499 + $329 for an EyeTV 200 tuner? Dude, that's over 800 dollars, not include shipping or sales tax. Given whatever extras turn out to be appropriate, I wouldn't be surprised if it hits double the mini-ITX's price.

January 24th: a day of partition table misery

Published January 25, 2005

Tech: January 24th, besides being the date the first Apple Macintosh went on sale, is supposedly the day of maximal post-xmas misery. Well, it certainly was for me today.

I decided to power on my old desktop to set it up as a back-room fileserver, and twiddled the partition table accordingly to nuke a few unused Windows partitions and maximise usable space.

Somehow or other, some component of my system decided that it would henceforth be non-bootable. It seems some BIOSes don't like partition tables where a high-numbered logical partition have a lower starting sector than a boot logical partition, or something... GRUB just errored out with an obscure 'Error 17', which apparently means that it couldn't find its boot partition any more.

OK, so I needed a boot disk. But I had 1 laptop with a CD/DVD drive but no floppy drive, and a desktop with a floppy drive but no CD drive (due to hardware failure)... and the original linux boot floppy was long gone, seeing as I'd hardly booted this machine in the duration of two house moves. Argh.

A dinky little Cruzer mini 128MB USB flash drive saved the day. (R)ecovery (I)s (P)ossible is a tiny Linux distro that fits into 27MB, well inside the USB drive's limits; it has an exceptionally helpful and detailed README detailing exactly what needs to be done to create a bootable USB flash drive from its ISO image, using just the generic linux toolchain.

Together with fdisk and parted's 'rescue a lost partition' mode, I was able to get the mangled partition table back into shape, mount the boot disk, change the fstab and grub configuration file, and reboot into a working system. phew!

Many thanks to Kent Robotti, who's done a great job with RIP.

On the other hard -- no thanks to whoever came up with the arcane rules behind the IDE partition table... argh.

OpenStreetMap.org

Published January 21, 2005

Map: much interesting geowankery going on in London, where they suffer under the same Ordnance Survey monopoly as we do in Ireland.

This message to their mailing list notes a quote from IKONOS of $1,172.50 USD plus shipping for a 1m Color Geo referenced satellite image of central London, covering 67 square kilometers.

Given 'enough processing', data extracted from that map becomes a Derived Work, and have no copyright restrictions. 'Processing' includes 'vector extraction, classification, etc.'

Now, I worked it out -- central Dublin city centre covers about 3km x 4km. At the named rates for London, that works out at an inexpensive $210! Looks like it was imaged in September 2003.

There's something interesting for a local geohacker to add to their list of projects ;)

(There's also some old Landsat-7 data that may be usable.)

‘Spam Kings’ review

Published January 20, 2005

Spam: Before xmas, I received a copy of Brian McWilliams' new book, Spam Kings.

It's a great book -- full of behind-the-scenes details on how the spammers operate, how they get away with it on the sending end, how they try to evade filters on the receiving end, and how they're fundamentally running the usual simple scams that have been around since before email spam came into existence. Well worth reading.

In addition, Brian's continuing to write about spam and spammers at the Spam Kings weblog, and will be giving a talk at this year's MIT Spam Conference, tomorrow.

Anyway, pick up a copy if you're interested in the spam problem -- this is one of the best books I've read on the subject, and this kind of information is essential for an understanding of the people we're up against.

Echo chamber goes crazy about ‘nofollow’

Published January 19, 2005

Blogs: Just to expand on a linkblog posting I made yesterday, Google's search team have announced support for a new piece of Google functionality; they'll fix their crawlers to ignore links with a rel="nofollow" attribute, for PageRank calculations, the idea being that spammers will stop blog-spamming once they can't get PageRank out of it.

The blog world has been all aflutter:

Google Blog
MSN Search
SixApart
Dave Winer
Robert Scoble: 'now we can link to things without raising their search engine whuffie'
burningbird: 'I remember something like this being discussed before'

BurningBird is right, to a degree. In fact, it's been solved before.

Here's a taint.org posting from November 2003 where I point out that by using a trivial Javascript URL one can link to another page without conferring PageRank. The format is:

javascript:document.location=target

The result looks like this, and work in any browser with a basic JS engine, from IE 3.02 and Netscape Navigator 2 onwards. I've been using it for my referrer logs, among other things, for over a year. I wrote a patch that implemented it for external links in the Moin Moin wiki software.

Amazingly, despite my plugging this idea at virtually every opportunity, it seems nobody noticed! At least, nobody among the people who (it would seem) should be looking into comment spam, thinking about how to deal with it, etc.

Disappointing -- the echo chamber keeps talking to itself, once again. Maybe I'll stick with dealing with email spam instead ;)

Ah, whatever. Anyway, this is a nicer fix; relying on JS isn't a good thing. So nice work, Google.

(PS: worth noting that while this is a good plan, comment spam won't be going away any time soon, as Mark Pilgrim noted. Still, here's hoping it'll help in the long term...)

IPC::DirQueue 0.04 released

Published January 19, 2005

Perl: at last, a perl-related posting! I've released IPC::DirQueue 0.04; details of what's changed (summary, a couple of bugs fixed) are at that link.

BTW, thanks to Ask and Robert at perl.org, who are providing free SVN repository and list hosting for CPAN modules! And don't overlook the fact that the mailing list/newsgroups each have their own RSS feed, woot!)

Prescient tsunami spam

Published January 19, 2005

Spam: I was just looking back through the archives here on taint.org, and noticed this entry from December 2 last year:

A huge 300 ft. high ocean wave is moving towards your continent. Your and many other cities are in a real danger. Approximate wave moving speed is 700 km/h. cmoym eaaa yypbzz
Please read more about this catastrophe here: (link)
We are strongly urging you to evacuate yourself and your family as soon as possible, even though you may live far away from your city. The tsunami will reach the continent in approximately FOUR hours.

It appears that the spam was a phish attack -- the site in question is full of Internet Exploder exploits. It was 'targeted', at least as well as such things ever are, at Australian readers. AUSCERT issued a warning about it at the time.

But how's about that for timing? Spooky! What did those phishers know?

eWeek’s ‘Spammers Upending DNS’ article

Published January 13, 2005

Spam: eWeek recently published an article entitled 'Spammers' New Tactic Upends DNS' , which notes that:

One .. technique finding favor with spammers involves sending mass mailings in the middle of the night from a domain that has not yet been registered. After the mailings go out, the spammer registers the domain early the next morning.
By doing this, spammers hope to avoid stiff CAN-SPAM fines through minimal exposure and visibility with a given domain. The ruse, they hope, makes them more difficult to find and prosecute.
The scheme, however, has unintended consequences of its own. During the interval between mailing and registration, the SMTP servers on the recipients' networks attempt Domain Name System look-ups on the nonexistent domain, causing delays and timeouts on the DNS servers and backups in SMTP message queues.

This had me stumped when I read it, since an email from a nonexistent domain is a pretty reliable spamsign (it's used in the NO_DNS_FOR_FROM rule in SpamAssassin, for example, which hits about 2% of spam), has been a rule in the default ruleset for several years, and there's no sign of that behaviour in our spam traps.

After some discussion, Suresh Ramasubramanian came up with this explanation of what's really happening:

Verisign now allows immediate (well, within about 10 minutes) updates of .com/.net zones (also same for .biz) while whois data is still updated once or twice a day. That means if spammer registers (a) new domain he'll be able to use it immediatly (sic) and it'll not yet show up in whois (and so not be immediatly identifiable to spam reporting tools) - and spammers are in fact using this "feature" more and more!

That does sound a much more likely explanation, and matches what's been seen in the traps.

So: WHOIS, not DNS.

IBM Pledges 500 U.S. Patents to Open Source

Published January 11, 2005

Patents: wow, this is amazing news! 'IBM today pledged open access to key innovations covered by 500 IBM software patents to individuals and groups working on open source software. IBM believes this is the largest pledge ever of patents of any kind and represents a major shift in the way IBM manages and deploys its intellectual property (IP) portfolio.'

Even better, they are hoping to begin a 'patent commons' for other companies to join, and the OSI definitions of which licenses are judged 'open' apply.

More details:

Of course, it would be better if it were also safe for commercial software development. But this is a valuable bulwark against Microsoft-style patent tactics.

Web-browser style history for the command line

Published January 11, 2005

Code: Here's something I came up with recently -- it's actually an evolution of the idea of pushd and popd, as included in BASH. To quote the POD docs:

cdhistory is a perl script used to implement web-browser style "history" for UNIX shells; as you use the cd command to explore the filesystem, your moves are remembered, and you can go "back" through history, and "forward" again, as you like.

Download the perl script here.

Annoying anti-arab Republican talking points, pt. xxviii

Published January 6, 2005

Politics: This moronic comic from Pat Oliphant came up in my comics page the other day, and, after a few days of hearing this particular talking point through the usual propaganda channels, I just saw it again. It pissed me off enough that I took a look at the stats.

Naturally, it's bullshit. The top 50 governments pledging tsunami aid, per GDP:

Qatar (#2)
UAE (#5)
Kuwait (#9)
Bahrain (#10)
Saudi Arabia (#15)

Given that the USA's at #29, and the UK at #22, I think the arab states are coming up with a pretty good result there.

I guess it's hard to look beyond today's talking points when you're still drawing cartoons at the age of 70.

A Firefox Extension plug

Published January 5, 2005

Web: Urgh, I still have this damn cold I picked up in Ireland... sniffle cough etc. More vitamin C needed!

Anyway, just a quick plug for a very deserving Firefox extension, one I haven't seen mentioned widely. It's pretty common, when you wish to print out a web page, that you wish you could get rid of the obnoxious extra-wide sidebar tables, gigantic ads, or other extraneous parts of the page. Well, now you can:

Nuke Anything is a Mozilla/Firefox extension which offers two great features in the right-click context menu:

Remove this object: this will remove the object you've right-clicked on -- a table TD, paragraphs, images, IFRAMEs, etc.
Remove selection: more usefully, this allows you to select exactly what you want to remove with a left-button drag, then right-click to remove it.

It's really useful. I almost never print anything out these days without scrubbing off a few unwanted sidebars ;)

HOWTO: invalidate a patent application with prior art

Published January 4, 2005

Patents: here's an interesting technique I heard recently. (credit: I'm not sure who told me about it, but I think it may have come from or via John Levine.)

If you become aware of a patent application (note: not an issued patent!) for which you are aware of possible prior art, you may be able to help invalidate it, or at least ensure any resulting patent is narrow enough to be relatively sane. Here's how.

If you have knowledge of techniques that you believe may be prior art, you can send them on to the filers or the patent examiner. At this stage, the onus is on them to prove that the technique is not prior art for the application (once it's granted, the onus would be on you to prove that it is).
The filer also must indicate techniques that they are aware of, that may be prior art, during filing; so CC'ing a public forum with a copy of whatever you send to them, may at some point in the future help indicate that they did not do this.

Of course, you have to go find the patent application number, the contact addresses of the filers, and the contact address for the patent examiner to do this ;) But it beats posting a whinge to Slashdot.

An unnamed patent agent comments:

'I believe an examiner is not under obligation to review art sent directly to them, but certainly the applicant and his agents are required to report any art they come across. That means the inventor as well as the law firm representing them.

You should include a cover letter that you saw their application (give details), and that you believe that what you are sending them is prior art, and that now that they have it, they are obligated to report it to the PTO. The same can be done to their counsel.

Probably, anything sent should be sent with some sort of delivery confirmation, and to make sure that the sending of the prior art is of public record, create a Web site where all sent art is listed, along with destination and confirmation information. This would help show inequitable conduct should the patent later be asserted and the art you provided not be shown as of record in the examination.

Mind you - I have not heard of these being done before (bombarding listed inventors and their agents with prior art, forcing them to have to disclose it), but I think it's a great idea. One caution - if you send too much, you over inundate the examiner, and then really good art could get overlooked during examination.

Separately, please keep in mind that the claims in a published application have probably not yet even been seen by the examiner at the PTO. These are the claims that the applicant would love to have the examiner accept, but until prosecution of the application actually commences (and completes), there's no way to know what claims will ultimately result.'

Update: some good additional points:

'The prior art must have been published or been publicly available at least as early as the earliest priority date of the patent. The priority date is either the filing date, or the filing date of a parent application. This information can be found on the cover page of a patent.

A patent's scope is covered by the claims. The claims define what the invention is. All other material in the patent is supporting material, and usually non-binding. In order to be anticipatory (the best kind) prior art for a particular claim, the piece of art must contain or described every element of the claim you are seeking to invalidate. Note that dependent claims add additional elements that the prior art needs to contain if you want to invalidate the dependent claims as well.

Prior art which is not anticipatory may be used in combination with other art or knowledge at the time to show obviousness. This type of art may have some impact during prosecution of a patent, but if a patent has already been issued, obviousness is a real uphill battle to fight in the courts. Few patents have been invalidated because of obviousness in trials.'

Another attorney notes: 'You can actually send it anonymously if you want. Just keep the certified receipt to prove they got it. As long as they know it exists, the onus is on them to disclose it to the PTO.'

'It's best to send them something printed out or on tangible media, along with a brief note explaining what it is and most importantly, when it was first publicly available. Certified means using certified mail or FedEx or something where you have a valid receipt.

As far as (discovering) who the (filer's patent lawyers) are ... it's usually listed on the patent applications. you can search the USPTO website for them.'

And a report that this technique is now in use: 'some patent attorneys are reporting that this approach is a valid one that people have started using.'

Update 2: More assent from another unnamed patent lawyer:

'Anyone who wishes to do so can send a letter to the Patent Office letting them know of any prior art of which they are aware. The Patent Office will then place it in the application file. Anyone who cares about this patent will surely order up a copy of the application file from the Patent Office, and will come into possession of whatever you sent.

Later you can see whatever you sent them. Go to
http://portal.uspto.gov/external/portal/pair and plug in the serial number (for the desired patent). Click on "image file wrapper".'

It's the right thing to do for any patent or patent application.'

Verizon.net blocks the world

Published January 3, 2005

Spam: I'm still catching up, but this is just plain hilarious. Pure, solid-gold, insanity. Verizon.net, the ISP branch of the US telco, has decided that the easiest way to fix their spam problems (uh, spam-receiving problems, that is), is now blocking inbound email from non-U.S. IP ranges:

A little birdie with insider knowledge has confirmed that Verizon is blocking all international IP space from RIPE, APNIC, and more, and is only unblocking specific domains, based on their IP address, when complaints are made and escalated.
According to the source 'the security team management thinks this is going to stop their inbound spam problems.'
Well, it may stop their inbound spam problem, but it's also going to stop that pesky 'wanted email making it to their customers' problem.

A quick check from my Ireland-hosted colo box does indeed indicate that this is still the case, and I can't connect to relay.verizon.net (206.46.170.12):

  : jm ftp 1...; telnet 206.46.170.12 25
  Trying 206.46.170.12...
  telnet: Unable to connect to remote host: Connection timed out

Back, in the flurry of a mini-tornado

Published January 3, 2005

Meta: Back. Not even 'mini-tornados' at Dublin Airport can keep me away -- although it gave it a damn good try, with a 3 hour delay, a missed connection, and an overnight stay in Chicago. Arggh.

Mail: I generally leave the laptop at home when on vacation, to do some proper winding down. Not sure it was a great idea this time, since I was joe-jobbed by some pretty extensive spam runs recently, resulting in over 30,000 bounces sitting unread in my email when I got back.

Thankfully, Tim Jackson's bogus-virus-warnings.cf SpamAssassin ruleset (with a few updates) got most of them, with only a few hundred getting past. I should really hack on making those more complete, but some of the bounces are really obscure; along the lines of 'Hi from J Random Luser, Esq.! I no longer use this address because it gets too much spam! Please send to this new one instead: jrluser98@example.com!', generally without any obvious identifying headers that indicate it's an autoresponse.

Sigh -- each of those messages is just utterly random, and I can't see much recourse but to come up with some nasty phrase-based content filtering rules, which I was hoping to avoid. But 29,500 hits isn't bad ;)

I'm not sure they'd be suitable yet for use as default SpamAssassin rules, since they now generally just match any kind of bounce message, not specifically joe-job or virus-forgery blowback. But that suits me just fine -- I can live without bounces, as long as I don't have to suffer the bounce blow-back.

Science: Good news from New Scientist -- they're opening up their archives! NS has consistently the best science journalism around, and I've been a subscriber for years. But until recently, they had a lousy approach to their website -- most of the useful stuff, like the archives, were walled-off, subscriber-only features; a classic case of missing the Clue Train. Well, here's an archive search for 'spam' -- pretty impressive, and most of the short articles are available in full, with only the full text for features and opinion pieces requiring a login.

In addition, they've added a massive batch of RSS feeds. Sadly, no full article text excerpts, however. But still -- getting the clue, eventually -- this way they may actually get links on the web, in place of the mangled and chinese-whispered versions of their articles republished in the UK newspapers...

Ireland: Due to monopolistic pricing of Irish GIS data, consumer GPS maps of Ireland's road system are appalling, and this page collects a few great demos -- for example, MS Autoroute quintuples the distance from Galway to Roundstone! That's a major tourist route, BTW. I knew it was bad, but not that bad...

Anyway, I'm still waaay behind, but slowly catching up.

Xmas hols

Published December 22, 2004

Meta: I'm back in Dublin for a couple of weeks over xmas, so I won't be updating this weblog very much. See you in January!

BTW I flew back via Chicago, which is obviously the stopover of choice to Dublin from Silicon Valley -- surrounded by 1 iBook per every 8 passengers. ;)

PS: looks like they forgot Poland!

An Open Letter to Sound System Developers

Published December 16, 2004

Linux: after about 3 months of tweaking and twisting, performed by someone who's been using UNIX for over a decade, I've finally got sound working the way I want it on my Linux desktop. In other words, I can hear sounds made by Flash applets, and I don't have to shut down the best music player on the platform every time another app wants to make a sound.

This is pretty clearly absurd.

So here's my open letter to the developers of the various systems (GStreamer, aRts, ALSA, EsounD polypaudio, et al):

Please DO do some testing with crappy sound hardware. I don't care if your sound system works great with a SoundBlasterLive 2006 with the kryptonite connectors, I have a laptop, for god's sake. That means software mixing is essential, because cheapo hardware doesn't do hardware mixing.
As an extension: please DO include software mixing by default. ALSA's pretty good in general, but having to hack out 55 lines of hand-tweaked config file before software mixing works, is insane. (Especially when the Wiki documenting that is full of notes that some of the magic numbers may not work on your hardware.)
Please DO use existing APIs if possible. That means esd. I'm looking at you, aRts. At least the latest sound project, polypaudio, looks like it's getting this right.
I DON'T care about network transparency, realtime response, or having a wah-wah pedal effect built into my sound server. That's just silly. Use a modular architecture to allow that in future, but concentrate on getting the basic stuff working first!
Please DON'T hardcode output device or output 'sink' names into the source. Looking at the kgst component of KDE here.

Meh.

Anyway, here's the scoop on what I had to do to get software mixing working in both GNOME, KDE, and Firefox, on my Thinkpad T40 running Debian unstable. Once I figured out the magic incantations, it now seems to be working without stutters or hangs.

Sometime in the next few months, of course, I plan to upgrade to Ubuntu Linux, and all bets will once again be off ;)

BSA’s Spam Statistics

Published December 15, 2004

Spam: The Business Software Alliance, a UK anti-piracy body representing many of the major software vendors, recently issued a spam-related press release which got a lot of attention in the UK press (they have great press contacts!).

To quote John Graham-Cumming's newsletter on the subject:

1 in 5 British Consumers Buy Software from Spam: that's according to a survey by the Business Software Alliance. I find that a pretty surprisingly high number and considering it comes from an advocacy group that tries to get people to buy legitimate copies of software I expect it's not totally accurate. The one thing I find really surprising from the survey are these two statistics: 23% of spam is read by the person receiving it and 22% of people have bought software. Apparently, 11% of people surveyed like the idea of buying through spam because the software is cheaper.

It's still an interesting figure, but the BSA has come up with some pretty suspect statistics in the past, so pinch of salt applies. As jgc points out, the BSA have a vested interest in making the problem sound worse than it may be in reality.

Still, the survey PDF can be read here, and is worth a look.

EU Software Patent tricks — very fishy antics

Published December 13, 2004

Patents: This is really absurd -- according to this ZDNet UK article, it now looks like the EU Council is considering railroading the EU software patent directive through, by hiding it as an 'A-item' in a Fisheries Council Meeting the week before xmas:

Laura Creighton, the vice-president of the Foundation for a Free Information Infrastructure (FFII), is concerned that the EU Council could be contemplating passing the directive without discussion in an unrelated meeting.
'Before today it was possible for generous people to look charitably at this text (the proposed patent directive) as an example of a tragic mistake, not malice,' said Creighton in a statement on the FFII Web site. 'But not with this last-minute manoeuvring.'
'Only the most committed opponent to the democratic process would believe that the proper response to the widespread consensus that there is something profoundly wrong with the Council's text is to race it through with an A-item approval the week before Christmas in a Fisheries Council Meeting. The bad smell coming from Brussels has nothing to do with the fish.'

Reportedly, A-items are dealt with by asking the assembled councillors if they have any objections to any of the outstanding items. They're not listed in detail at the meeting, so this way the directive can be passed in what is effectively a submarine (boom boom!) manner.

In a Talkback to ZDNet UK's earlier story highlighting the issue, Cox wrote: 'I too was mysteriously overlooked despite having written to my MP and received an answer.' .... Cox, who has previously been invited to speak on software patents at the EU, said the Patent Office apparently fears 'every word I have to say about their plans'. He went on to add: 'Unfortunately with all the underhand game playing both in the EU council of ministers and in UK government and patent circles it isn't the slightest surprise.'

Here, the patents at issue were less valuable to companies that actually produce Web services products than they were to firms that produce nothing but lawsuits and licensing threats. In other words, patents like these have become worth more as weapons than as protections for companies competing in the marketplace.
Many have compared these new patent licensing firms to terrorists, and in some ways, the analogy is apt. When the Soviet Union collapsed, one of the biggest worries was that rogue military personnel might sell off one or more of the USSR's nuclear missiles to a terrorist group. Securing those weapons became a top priority. The reason was fear -- fear that the terrorists, who had little to nothing at stake in terms of world peace and national stability, would use the missiles to extort or manipulate the world political climate. Unlike the United States or China, which could be retaliated against and which had a stake in stability, terrorists were essentially immune from attack, and thrived on instability.
With the patents of bankrupt dot-coms, the dynamics are similar. Rogue licensing firms buy up these patents and then threaten legitimate innovators and producers. They have no products on which a countersuit can be based and no interest in stable marketplaces, competition or consumer benefit. Their only interest is in the bottom line.
While profit itself is often a worthy objective, it is not always synonymous with innovation. Every dollar a tech company pays to patent lawyers or licensing firms is one less dollar available for R&D or new hires. Thus, many companies that offer new products end up paying a 'tax' on innovation instead of receiving a reward. When this happens, it's a signal that the patent system is broken. Forcing companies to pay lawyers instead of creating jobs and new products is the wrong direction for our economy to be headed and not the result our patent system should be promoting.

playing around with Google Suggest

Published December 10, 2004

Web: Google Suggest, a drop-down list of suggestions -- with hitrates! The one letter hits are interesting, too.

"spam" hitrates, the top 3 (aside from "spam" itself):

"spam filter": 6,400,000 results
"spamcop": 1,570,000
"spamassassin": 1,350,000

in the top 3. getting there!

unfortunately, you have to get as far as "justin ma" before my name shows up, so not doing too great in that competition. ;)

too busy worrying about patents to care about copyrights

Published December 3, 2004

Patents: oh, this is painfully ironic.

patents4innovation.org is a PR site set up by EICTA, a consortium of several pro-software-patent multinational companies, to put some PR money into lobbying for the legalisation of swpats in the EU. I've mentioned it before in the context of another boo-boo. Well, here's the next one.

According to FFII, they recently took a Creative-Commons-licensed article from another website, and:

republished it without the required attribution to the author
translated it, creating a 'derived work', against the terms of the license
and then failed to notify readers of the licensing terms, as required

In other words, they managed to infringe the terms of its copyright-based licensing in multiple clauses.

No wonder they claim that patents are required to protect people's inventions. It seems they just don't understand how copyright-based licensing works ;)

(The article's been taken down from the p4i site, but not before the boo-boo was spotted by an eagle-eyed FFII'er.)

Interesting/bizarre recent spam

Published December 2, 2004

Spam: some good crazy spam recently -- firstly, some Seventh Day Adventist lunacy:

THE PAPACY IS THE ANTICHRIST THAT IS TRYING TO CHANGE THE LAW OF GOD. DANIEL 7:25
THIS IS THE LAST WARNING.
THE LAW OF GOD IS ETERNAL BECAUSE GOD IS ETERNAL 14:12. MT. 5:17 SATURDAY SEVENTH DAY IS THE TRUE LORD'S DAY. EXO. 20.8-11 SUNDAY IS A FALSE PAGAN DAY. IT IS NOT IN THE BIBLE. IT WAS USED TO WORSHIP SATAN

It runs on in that vein for quite a while. Interestingly, most of the text from there on in is 'gappy' -- in other words, the spammer has inserted spaces between each character of a word -- even inside link addresses. As a result, they no longer work. oops!

And a new one to me -- natural-disaster spam (via Mark Pilkington):

THIS IS AN OFFICIAL WARNING!
fngva uvtt chloez
A huge 300 ft. high ocean wave is moving towards your continent. Your and many other cities are in a real danger.
Approximate wave moving speed is 700 km/h.
cmoym eaaa yypbzz
Please read more about this catastrophe here: (link)
We are strongly urging you to evacuate yourself and your family as soon as possible,
even though you may live far away from your city. The tsunami will reach the continent in approximately FOUR hours.
venbz nwvw exepmi
YOU HAVE BEEN WARNED!

I've removed the link, btw -- the site it links to contains a bunch of nasty malware-installing IE-bug exploits. In case you were wondering: you can tell it's genuine because it says IT'S AN OFFICIAL WARNING at the top.

(ObSpamComment: note -- this here's a good example of why spam is unsolicited bulk email, not unsolicited commercial email; neither are selling anything. one's religious craziness, the other one's trying to r00t your machine.)

Moving House

Published November 27, 2004

Life: I'm moving house -- I've just filled about 20 boxes, now to get moving them! Sadly, there's no wifi in range of my new house, so the upshot is I may be offline for a few days. Boo.

Fun Times Ahead with Nathan Myhrvold

Published November 24, 2004

Patents: Newsweek: Factory of the Future?:

The dino's ferociously bared teeth hint at elements of Intellectual Ventures' bold business plan. Myhrvold and his partner, former Microsoft chief software architect Edward Jung, have created the quintessential company for the 21st century. It doesn't actually make anything ... Only patent attorneys populate the quiet hallways. ...
Sources familiar with Myhrvold's strategy say that he has raised $350 million from some of the largest companies in high tech: Microsoft, Intel, Sony, Nokia and Apple. Google and eBay also recently invested. With this large bankroll, the company is out buying existing patents in droves. (Myhrvold won't comment on these activities, but sources say he has already purchased about 1,000 patents.) The strategy is to set up a sort of patent marketplace. Patent owners get money upfront for the dusty ideas sitting on their shelves, the investors get the rights to use the ideas without being sued and Myhrvold gets to rent those same ideas to other companies that need them to continue creating products. ...
"We're concerned that these giant pools of patent rights are going to prevent entrepreneurs from entering markets, as opposed to being used to promote innovation," says one worried Silicon Valley venture capitalist.

Now that's scary...

JFK Reloaded

Published November 22, 2004

Games: OK, JFK Reloaded is very, very wierd.

Read the insanely detailed FAQ and boggle at the author's obsessive research and fetishistic recreation of the events at Dealey Plaza, November 22nd 1963.

Quite worrying, to be honest!

EFF’s clueless spam filtering white paper

Published November 22, 2004

Spam: The EFF are a great organisation -- damn, I even helped set up an organisation based on its goals in Ireland, back in the day! But this white paper is shockingly clueless.

(Note: this posting has been updated. Original left intact, but there's an update below worth noting.)

For example:

Spam Assassin, a popular program that does ad hoc pattern matching, assigns 'points' to various features of an email to determine whether it is spam. ... One of the major problems with this system is that messages from certain countries -- like China, for example -- can be blocked purely on the basis of where they come from and what language they're in. The implications for free speech here are very troubling indeed: ... thus anti-spam technology unintentionally works as a political censorship mechanism.

SpamAssassin does not give points for country of origin, or language the message arrives in, unless the user explicitly either (a) adds rules from an external source, or (b) modifies the 'ok_languages' setting in their configuration, from the default, to specify that they do not want to receive messages in particular languages. No country- or language-blocking happens by default. This is by design.

It's a shame that the authors felt the need to outright fabricate a danger, here.

The white paper features more broad generalisations about 'spam filters', mostly using unsubstantiated friend-of-a-friend stories, without detailed data. And I do know that there have been cases of MoveOn.org, at least, being a source of UBE, in the past -- so it's not valid to claim that this is all a 'free speech' issue; political UBE is still spam.

They need to realise there's a lot of very smart, very reasonable anti-spammers out there, and most of us agree with the rest of their goals, except for their spam position. This is hurting them.

Still, it appears they're finally getting a clue about requiring subscription requests be confirmed using closed-loop opt-in, so that's good. More political newsletters, and political campaigns, need to get this clue -- just because it's political speech does not mean it's not spam. (I have several thousand political spams in my spam folder -- most from that German anti-immigration virus from earlier this year.)

Note that Rod is unsure if they're practicing what they preach...

Update: Annalee Newitz has been in touch, and pointed out that the white paper in fact says 'mails ... can be blocked', rather than 'are blocked' based on country of origin. In other words, it's purely a matter of this being possible, rather than the default, and that administrators apply these customisations.

In addition, she notes that the conclusions recommend that ISPs and administrators of spam blocking systems allow end users to control their own filtering settings, saying 'If a user wants to block all mail from China, great. If a sysadmin does it for a bunch of users without permission, then that is a problem in our opinion.'

So I agree with that. Misdirected outrage hereby turned off ;)

(Mind you, I still think they need to work more with the reasonable anti-spammers... and fix that unconfirmed sign-up that Rod mentioned, if it's really still unconfirmed!)