TV: I've taken a little time to throw up my PVR build log.
If you're hacking on one yourself, or curious about what it takes, or just like reading cut-and-pasted UNIX command lines -- go take a look!
Category: Uncategorized
UNIX: a quick recap of a good tip combo picked up from ILUG recently. To paraphrase Conor Wynne's original question:
What's the best way to set up a secure connection between two hosts, possibly over the internet, using SSH, suitable for use from cron so that it can run via crontab without entering authentication manually?
Barry O'Donovan replied:
I suggested ssh keys without passphrases ... in
http://www.barryodonovan.com/publications/lg/104/ and it includes instructions. ... You can invoke rsync over ssh and specify a specific key with:rsync -a -e 'ssh -i /home/username/.ssh/id_rsa-serverbackup'
Colm MacCárthaigh followed up with:
You can restrict what commands an ssh account can run in the ssh public key. This is how some of our more important projects (like Debian, FreshRPMS, and a few more) push us updates. The key looks like (jm: all on one line, no space between 'no-pty,' and 'command'):
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty, command="/home/ximian/rsync-ximian-nolog &"
ssh-dss keydata username@blahSo, create a passwordless public key like so, and just change the command to whatver rsync runs.
Combined, that's a useful tip -- I knew about the ssh command restriction technique, but being able to use a specific single-purpose key from the ssh client is very useful.
(updated: mbp mailed to note some missing quotes in Barry's command above; they'd been eaten by WebMake. drat.)
UNIX: another useful tip. Bash supports a wide variety of command line editing tricks; you have the usual GUIish editing (backspace, insert new characters, delete, blah blah) through the GNU Readline library, and in addition to that you have the traditional csh-style history expansion (like '!!' to refer to the previous command typed).
The latter are great, but they won't actually be expanded until you hit Enter and run the command line. That can be inconvenient, resulting in the user being forced to reach for the rodent for some cut'n'paste instead.
Here's a handy trick -- add this line to ~/.inputrc (creating the
file if necessary):
Control-x: shell-expand-line
Start a new bash shell. Now, if you type CTRL-X during command line entry, any shell metacharacters will be expanded on the current command line. For example:
% echo Hello world
Hello world
% echo Hi !$ (press CTRL-X)
(current command line expands to:)
% echo Hi world
There's a
few more commands supported, but none of them are really quite
as useful as shell-expand-line.
Update: 'Smylers' wrote to point me at this UKUUG talk from 2003 which discusses .inputrc expansions, and provides some insanely useful tips.
In particular, Magic Space clearly knocks this tip into a cocked hat, by performing the expansion on the fly as you type the command, with no additional keypresses -- amazing! Bonus: it works if you use Emacs-mode line editing as well as Vi-mode.
I strongly recommend reading that paper -- lots of other good tips there.
Patents: New Scientist: Sony patent takes first step towards real-life Matrix:
IMAGINE movies and computer games in which you get to smell, taste and perhaps even feel things. That's the tantalising prospect raised by a patent on a device for transmitting sensory data directly into the human brain - granted to none other than the entertainment giant Sony.
It's a very lame 'first step' though -- Sony has done no research and development on this invention whatsoever, it's just a patent form of the old 'in the future, we'll wear tinfoil suits! And here's how they'll probably work!' speculation. Sony's comment:
Elizabeth Boukis, spokeswoman for Sony Electronics, says the work is speculative. 'There were not any experiments done,' she says. 'This particular patent was a prophetic invention. It was based on an inspiration that this may someday be the direction that technology will take us.'
That's nice; I'm sure they have some in the pipeline for flying cars, too.
It's good to know that if an inventor does eventually come up with an ultrasound-based human-computer brain interface, they'll have to pay license fees to Sony so they can use their 'prophecy' in their invention. The USPTO's high standards are being maintained, as usual...
Ireland: Worth watching for european software-patent watchers, Forfás, Ireland's 'national policy advisory board on enterprise, trade, science, technology and innovation' are running a series of monthly seminars on 'Intellectual Property' in association with Licensing Executives Society Britain and Ireland.
This one looks quite interesting -- 10 June: 'Patenting Software - The Current State of Play', Author Barry Moore, of Hanna Moore & Curley, patent attorneys.
Interested parties can attend with pre-registration, or wait to download the mp3 at Forfás' website, apparently, along with the rest of the lecture series. (No sign what the license is on those files, though ;)
Patents: It seems the Irish Software Association has a new chairperson, namely Bernadette Cullinane. Whether this has anything to do with Cathal Friel's 'out of line' statements, who knows...
John McCormac passed on some interesting quotes from an Irish Times interview, which were also syndicated here:
'The incoming chairwoman of the Irish Software Association (ISA), Bernie Cullinane, has pledged to support the introduction of a proposed European Union directive on software patents.
She also warned members of the European parliament against blocking the controversial new directive or weakening it by proposing a host of amendments. ...
Ms. Cullinane, a former chief operating officer of the Irish company Performix said European firms needed to protect their intellectual property in a similar manner to the way US firms can.
'We don't want any further dilution of the current situation on patents,' she said in an interview with The Irish Times following her ratification as chairwoman of the ISA last night.'
My emphasis -- given that the current situation is that they are unenforceable in Europe, that's good, because we on the other side don't want a dilution either!
'We do need to look at how the US is developing its software industry and a removal of the patent (sic) could weaken venture capitalists' appetites for investing in new innovative companies.'
The whole 'venture capital requires patents' line is easily debunked. I'm sure the VC companies are telling Ms. Cullinane that they want patents, of course; it's just that they're wrong. ;) Laura Creighton, a European investor, gave a fantastic speech in Brussels in 2003 about investment and patents:
Software Patents (in the US in the 1990s) encouraged venture capitalists to make foolish investments, because they believed the patents were worth something. Venture capitalists often do not mind if the companies where they have invested go bankrupt -- as long as they hold title to the patents. They can start over again with a different team.
Sadly, when the bubble burst, the venture capitalists discovered that their patents were only good for a trip to court -- or at least some legal wrangling with a bunch of lawyers. A software patent is not like a hardware patent, where typically one, or at most a few covers the whole invention. Dozens, sometimes hundreds of patents, are relevant to any piece of software. So an investor, who now owns the assets of a defunct company -- cannot take its patents and hand them to a new development team and say 'build this'. It is impossible to develop software today without infringing somebody's American patent.
The venture capitalists, having lost fortunes backing companies which had no real product, are now uninterested in investing in any software companies whatsoever. Right now the American economy could benefit from more investment -- but the capital is not going into software companies. Again, part of the problem is software patents. The venture capitalists have learned that all software is in violation of somebody's patent. So they do not want to touch the stuff. Thus on the up side, and the down side, the existence of software patents have contributed to creating the stock bubble, and making the recovery slower and harder than it needed to be. So #4 is right out -- the existence of software patents are inhibiting investment right now, and for very good reason.
In other words, the presence of software patents has 'weakened venture capitalists' appetites for investing in new innovative companies', as Ms. Cullinane put it.
Anyway -- to keep the VCs happy, small companies can still obtain software patents in the US, and spend the tens of thousands of dollars required to register and enforce them in court, if they so desire. They can bring the US software industry to a legal standstill if they like, as they seem to have done, as long as European software developers can quietly carry on developing software for use outside the US ;)
But at least things aren't as bad as the situation with my neighbours -- I live a few miles from the offices of Acacia Research, the notorious patent trolls, who've just initiated a new lawsuit against Intel and TI.
Reportedly however, they're planning to open a European office this quarter...
Dublin: This is it -- it could be the end of an era. CB Richard Ellis auctioneers have a page up noting a new property to be auctioned on Wednesday 11th May 2005 -- The Stag's Head, 1 Dame Court, Dublin 2:
The Stag's Head is one of Dublin's most famous and finest landmark licensed premises, with many outstanding Victorian features.
The bar is lavishly appointed with many fine Victorian features from the beautiful mahogany panelling through to the red Connemara marble counter and the ornate stained glass windows.
Accommodation briefly comprises ground floor traditional style bar with feature mahogany and marble topped bar counter and terrazzo flooring with a snug area to the rear with ornate stained glass skylight. On the first floor there is a further lounge bar area with feature bay window. On the second floor there is a large catering kitchen, dry goods store and office. In the basement there is a further lounge bar area, cold room and toilets.
Many nights were spent in the Stag's Head partaking of their excellent Guinness. It used to be my local, at one stage, and I still drop back in for a night when I get the chance. Save the Stag's!
As my mate Ben put it --
The new owners will doubtless get rid of the (moth-eaten, stuffed) fox, put in recorded music and big-screen televisions, hire bouncers, open on Sundays, extinguish the distinctive odour of damp, replace the marble with formica, and dig up the Dame St mosaic and trade it to the Russian Mafia for heroin and trafficked women. Evil bastards.
Privacy: after reading Adam Shostack's weblog posting about private/anonymous blogging, I've been driven to think about that, and would up writing up a case study of Cogair, which was an influential anonymously-published proto-weblog in Ireland in the '90s.
Now, quinn at ambiguous.org quotes a review of EFF's recent 'anonymous blogging' guidelines, which largely comes up with one conclusion: it's a usability nightmare. The problem is, the EFF report recommends using invisiblog.com, which in turns uses the Mixmaster remailers. Those things are awful, and I doubt anyone but their authors could possibly know how to use them ;)
Here's an easier way to blog anonymously. I haven't tried it (honest ;) but from keeping up on this stuff, it should work...
Firefox
- First off, install Firefox. No point giving your identity away through an MSIE security hole. Clear out all cookies in Preferences:Privacy:Cookies (or better still -- start a new Firefox profile from scratch).
- Visit IPID and note down the IP address noted (this is your own, traceable, IP address).
Tor
- Next, install Tor, EFF's 'Onion routing' anonymizer system. This also means installing privoxy as directed in the Tor install guide.
- Set up Tor on your machine, so that Firefox will browse via that software.
- Using Tor, visit IPID and make sure it doesn't give you the same traceable IP address. This is to make sure you're browsing securely.
Hushmail
- visit Hushmail and create a new free email account. Obviously, don't use usernames and passwords that map in any way to your existing ones, and avoid words that may show up under your interests (especially if they're googleable)...
Blogger
- Using that Hushmail account as the email address, go to Blogger.com and create yourself a blog, then get publishing.
- Hey presto -- anonymous blogging the easy way!
- For safety, don't use the Firefox anonymous-blogging profile for any sites other than Hushmail and Blogger.com's publishing end. (A future Firefox vulnerability could expose personal info directly from Firefox itself.)
This is essentially the 'TOR to blog server' method described at the privateblogging wiki.
Now, note that along that chain we have 3 levels of identity -- the IP address (hidden by Tor), the email address (traceable to Hushmail, who could conceivably give up the Tor router's IP), and the Blogger.com weblog site (traceable to Blogger, who could give up the Hushmail address and the Tor router's IP).
As long as you don't give it away in your writings on that weblog -- and as long as Tor remains safe -- your own identity in turn is safe, too; and Tor has proved safe, so far.
There are still problems:
- The weblog site itself could still get taken down, e.g. via a DMCA takedown notice. This could be an issue, depending on what's being published.
- Tor traffic is identifiable as such as it traverses the internet. For bloggers in countries with a pervasive internet surveillance regime at the local ISP end, the watchers will be able to tell that Tor is in use, and tell who is the person using Tor. (They won't be able to tell what it's being used for, just that it's being used.)
PS, for the future: the guys behind Tor are working on a replacement for Mixmaster anonymous remailer software, called Mixminion. There's also a wiki for discussion of 'private blogging' here.
TV: here's a quick update on my PVR box progress. I have a very extensive /etc/LOG which I should probably just publish as-is, really, rather than trying to make it legible ;)
Anyway, the hardware arrived last month, but the main VIA EPIA ME6000 board was non-functional -- it could never get as far as powering up the CRT for the BIOS self-test. So it was RMA'd back to http://www.mini-box.com, and they sent out a replacement, which arrived a couple of weeks ago.
I finally got to checking this out the weekend before last, and hey presto, it powered up nicely. There followed a whole week of busy nights doing a load of cautious hardware hooking-up, not-so-cautious KnoppMyth installation, and thoroughly non-cautious hacking crazily at the desired enclosure with a hacksaw (because I was too cheap to buy a Dremel).
Things got a little hairy with respect to CPU temperatures, but some looking at specs (the VIA Eden CPU can deal with up to 90 degrees C!), and repurposing of a bin-bound case fan together with some soldering and snipping, has that under control.
Eventually, we're now at the stage where it can:
- watch live TV in perfect realtime, pause, rewind, timeshift, ffwd, etc. (the PVR-350 output is good)
- record our desired shows (bloody Antiques Roadshow! argh), according to the TV schedule
- play mp3s
- be ssh'able and sftp'able via a wifi USB dongle
- expose its schedule and allow recording via MythWeb
- expose its full desktop UI via x11vnc
and it looks good doing it, too. Credit goes to the MythTV guys for a fantastic job on their project, especially with its well-polished UI.
In addition, I have to plug KnoppMyth heavily. They're dealing with an awful situation with hardware compatibility where bleeding edge features like MPEG2 decoding and TV out are concerned, and doing a great job -- there's been several occasions where I've been staring down the barrel of a daunting patch/rebuild/test cycle, and then find out that KnoppMyth includes that component built-in for free.
But -- on the other hand -- no credit to the hardware vendors. As I link-blogged yesterday, VIA is doing the classic 'throw it over the wall' trick with respect to their linux support -- video drivers are written and deposited on their website, with scant documentation and virtually no support.
That's bad enough, but even worse is the situation with Hauppauge's PVR-250 and PVR-350 TV encoder/decoder cards. I realised soon into the setup process that other options for these should have been considered -- Hauppauge have done a great job at confusing the issue for driver developers, as far as I can see. Here's an example. When you buy a 'WinTV PVR-350' card, you may get the same box with the same manuals etc., but including these bonuses under the covers:
- one of seemingly about 5 different tuner chips, which you'll need to edit
/etc/modules.conffor; - one of about 3 different remote controls with differing output codes;
- a good chance you'll have to enter two mysterious ioctls to fix the colour registers, because recent PVR-350 models have changed these somehow and everything shows up as purple-on-green through its TV-Out.
It's absurd. The results are threads like this and a truly daunting setup procedure, which (of course) everyone blames on the software (and Linux itself).
Anyway -- how am I doing vs. Brendan's progress? ;)
- pro: my X display sizes are good
- pro: no need to switch audio outputs
- pro: I'm not using a separate cable box, so no need to hack up something IR to switch channels for me
- con: I can't yet watch AVIs or other video files, which I think he has working.
More on the latter when I eventually solve it. (it's tricky. I suspect I'll need to run two X servers with two TV-Outs to do this acceptably, and that's uncharted waters.)
Malware: spotted on NANOG -- Six PCs caused BigPond problems:
Disconnecting six compromised personal computers on Tuesday evening eased the difficulties caused by bogus requests which clogged BigPond's domain name servers (DNS), slowing customer e-mail and Web site access, Telstra said.
A Telstra spokesperson said the carrier had narrowed the list of malware that could have infected the computers to three, adding the problem could have been caused by a combination of those viruses or Trojans. He declined to name the suspects.
He said the PCs generated 95 percent of the bogus requests which caused the problems that evening.
The 'problems' in question are described here :
One forum participant (on Aussie forum Whirlpool), who claimed to be a BigPond customer, said on Monday: 'I'm in Canberra and it's been almost unusable all afternoon. I'm snowed under at the moment and it is really driving me crazy. Three out of four links fail to load first time and sometimes take eight or nine tries before it does.'
Another said: 'I am having problems loading Web pages, I get the 404 error. I have to retry five to 10 times to get some places.'
Petri Helenius, in a post to NANOG, notes:
Consumer ISP's who don't proactively take care of security/abuse usually end up with harvesting-bots which consume significant amount of DNS resources, typically doing anything from a few dozen to a thousand queries a second. A few hundred of these will seriously hamper an usually provisioned recursive server.
Interesting. It's been a long time since I've relied on an ISP's recursive DNS servers; in my recent experience (Comcast, Cox.net) they've always been overloaded, and take aaaages to give me answers. Maybe this is why.
It makes sense; most Windows machines will indeed use the ISP's NSes, because that's what DHCP tells you to do; and setting up a BIND or djbdns instance locally to query the roots directly is still a UNIX-only trick, as far as I know.
The upshot?
- 1. Yet another good reason why ISPs should proactively disconnect infected customers, as they deny service to other users of the ISP.
- 2. A good demonstration of yet another way the techie community's experience of web surfing and internet use differs from that of the unwashed masses in the hinternet -- that 'shanty-town of pop-ups and porn adware', as Danny O'Brien puts it.
- 3. Sometime soon, if it hasn't happened already, someone's going to bundle up an 'Internet Accelerator' lump of shareware that sets up a local recursive NS on Windows which queries the roots, and it'll become the latest popular Windows download. Then the load on the root servers will really start rising.
(PS: top tip -- ever wanted a publically-queriable recursive nameserver,
or a good IP address for pinging, that's easy to remember? 4.2.2.1
is what you're after.)
Spam: Matthew Wilson at Boomer Consulting has been having a field day -- it looks like some smart google hacking has thrown up some doozies of places that should have fixed this by now:
and my favourites:
Of course, all of these are immaterial to SpamAssassin -- we catch spammers using them anyway. But still, a surprising number of these out there.
The firm was told to pick an influenza A sample and chose from its stockpile the deadly 1957 H2N2 strain.
Check out how it was spotted:
On March 26, National Microbial Laboratory Canada detected the 1957 pandemic strain in a sample not connected with the test kit. After informing WHO and the CDC of the strange finding, the lab investigated. It informed the U.N. health agency on Friday that it had traced the virus to the test kit.
My emphasis. omgwtfbbq!
(WHO's influenza chief) Klaus Stohr said the test kits are not the only supplies of the 1957 pandemic strain sitting in laboratories around the world. 'The world really has to think what routine labs should be doing with these samples they have kept in the back of their fridges,' Stohr said.
True: the lovely C has a story from her TCD days of a vial of smallpox
found buried deep in the ice in the back of a long-forgotten freezer,
apparently rediscovered by someone during a routine spring cleaning. This
was in the early '90s, when smallpox was supposedly down to samples in
just two high-security labs, in Russia and America.
Open Source: on the 18th March the Irish Times published a commercial supplement for Microsoft. Naturally, given that it was paid advertising, there were lots of MS plugs -- but in the mix there was also a couple of more worrying articles: one by Tom Kitt, government 'Minister for the Information Society', noting
Microsoft has been one of the most innovative companies in the world and has a long track record over several decades of creating new product markets. The EU has to be open to allowing such innovation in Europe. Ireland will continue to argue at EU level, based on the solid evidence of our successful economy, that the Community must look at its rules on innovation and intellectual property rights to ensure they encourage risk taking in Europe and growth in the IT industry in the EU and around the globe.
And another with Cathal Friel, credited as 'chairman of the Irish Software Association'. Quoting the article text:
(Friel) also noted that Open Source software - which is developed by large communities of programmers and distributed for free or at low cost - is also going to have an effect on the software market. While Friel believes Open Source itself has a limited business model - 'at the end of the day, there's nothing but services to sell' - it is nonetheless becoming more pervasive and is 'a fact of life' for more traditional software companies. He believes the Open Source movement is actually stifling innovation, because fewer programmers will develop software without the financial incentive of success.
MS observers will note that both Kitt and Friel's statements mirror the MS 'party line' -- either the lads were well-briefed, or they just put their names to a story written by MS PR.
Well, there's been an interesting follow-up. Éibhear Ó hAnluain put pen to paper about Cathal Friel's statements, and received an interesting reply:
I received a 'phone call from Kathryn Raleigh, Director of the ISA, in reponse to my letter. As I was unable to take notes at the time, what follows is a memory of the conversation. She told me that the ISA would like to apologise to me for any offense that I took from the comments. She said that the first the ISA heard of the comments was after the piece was published and the Mr. Friel was not speaking with the ISA's authority. She told me that the ISA had indeed conducted some sort of analysis of the market regarding licensing and the 'proprietary' versus Free Software competition, and that the ISA's position on the matter is not to have a position. She gave me the impression that Mr. Friel has been told that he was out of line. She asked me to convey the ISA's regrets to my colleagues.
Well now, that's interesting!
I find it very encouraging to see that the ISA don't take the position noted in Friel's article, anyway. In my opinion, this is wise -- alienating free software and open-source-using companies doesn't seem likely to be a good idea, given that many of today's SMEs use open source extensively 'behind the scenes' in production, if not directly in the products they sell.
There's also the matter of Google's recent major entry into the Irish software industry, with its new offices in Barrow St. in Dublin. MS are no longer the only major multinational player on the Irish scene to whom open source's success, or failure, is a key factor in their business plans. Google use free software extremely extensively internally, are members of several major free software bodies including the FSF, and have released quite a few interesting pieces of open source software themselves.
Health: On a lighter note, I've been getting through my last two weeks mail and RSS data, and came across this beauty.
It's a truly venerable internet urban legend -- the Nepalese Nose Leech story. Even given that I assumed it was more than likely a UL, I still took care not to drink from streams when I visited leech-infested areas, especially in Nepal!
Well, it appears it may not be a UL after all --
Doctors have removed a leech from the nose of a 55-year-old Hong Kong woman after she swam and washed her face in a stream, a medical journal reported.
The woman went to her doctor complaining of nose bleeds and an occasional sensation that something was blocking her left nostril, the Hong Kong Medical Journal said in its April issue. Her family doctor noticed a brownish mass in her nostril but couldn't remove it because of heavy bleeding, the journal said.
The patient was taken to the emergency room, where doctors identified the problem as a bloodsucking leech. They had trouble pulling it out because the 2 inch invertebrate retracted into the nostril and disappeared, the journal said.
Part of the slimy leech was in a passage of her nasal cavity and a larger segment was in her sinus cavity, the article said.
Doctors used a nasal spray to anesthetize the dark brown leech that had a sucker on the front part of its body. After two minutes, the leech moved slowly out of the antrum (sinus) and was retrieved with forceps, the journal said.
The woman said that one month before her symptoms developed, she swam and washed her face in a stream while hiking. Doctors checked other members of her hiking group and found another leech in the nose of a man who washed his face in the stream, the journal said.
Link via jwz, AP wire story, abstract at Hong Kong Medical Journal site, MEDLINE abstract, including a line noting 'this form of leech infestation has not been previously reported' -- except on teh internets!
Life: Alec Muffett quotes an Economist opening line:
We tend to meet any new situation in life by reorganising, Petronius Arbiter, a 1st-century Roman satirist, is supposed to have remarked. And what a wonderful method it can be for creating the illusion of progress while producing confusion, inefficiency and demoralisation.
As apt today as it was then.
(I was recently talking to a mate who's a post-grad in the classics. She noted that classicists aren't the fastest-moving academicians around, speculating that maybe it was because, in studying the classics, you realise the same problems and the same solutions have been around for over two thousand years regardless of change in other aspects of life.)
Security: Hey user auth systems! If you're going to require me to sign in, and publish my login as a signature to prove that I'm 'me', please do me a favour -- don't delete the account if it's been 'inactive', and allow anyone to re-register that name without my knowledge!
I just tried to leave a comment on a Blogger.com weblog, to find that my user account at Blogger had been deleted. Re-creating a new account with the same name wasn't a problem -- the previous account data had been simply deleted outright. (Presumably they don't do this to people with a Blogger.com weblog -- I hope.)
The risks of this are pretty clear; given that I'd already established an identity (at least in comments on certain Blogger weblogs) as 'justinmason23', if an attacker were to have re-registered that identity before I did, they could impersonate me.
Mapping: NASA's Earth Observatory has put up a 4 MB high-res topographical image of Ireland. A rough calculation indicates that each pixel is under 0.1 of a mile on a side. It's fantastic. ;)
Best of all, since NASA operate under the US' enlightened copyright and licensing policies for government-funded data, it's free -- the masthead notes 'Any and all materials published on the Earth Observatory are freely available for re-publication or re-use, except where copyright is indicated. We ask that NASA's Earth Observatory be given credit for its original materials.' Copyright is not indicated on this image as far as I can see. So go ahead and save a copy for future use, too.
(via EirePreneur in turn via Irish Typepad)
Web: I've been doing a little thinking about group-based networking and services.
Here's the situation. Let's say you have a small group of people, and want to offer some kind of online service to them (like a private chat area, mailing list, etc. etc.) That's all well and good, but maintainance of 'who's in the group' is hard. You need:
- the ability to let other 'admins' add/remove people
- a nice UI for doing so
- a nice UI for people to request to sign up
- possibly, multiple groups
- privacy for group members
- possibly, some public groups
- decent authentication, username/password
- the usual stuff that goes with that -- 'I've forgotten my password, please email it to my listed address'
- did I mention a nice UI?
The traditional approach is to code all that up myself, in my copious free time presumably. Urgh, talk about wheel reinvention on a massive scale.
I'd prefer to use something like TypeKey, a web service that exposes an API I can use to offload all this hard work to. Initially, I was in the 'ugh, Typekey 0wnz my auth data' camp, but I've eventually realised that (a) they're not quite as evil as MS, (b) they're not quite as stupid as MS (deleting Passport accounts if you don't log in to Hotmail, which is only one of the supposedly many services, including third party services? hello?!), and (c) it's actually really convenient having a single-sign-on for weblog commenting after all.
Having said all that -- TypeKey's out. Unfortunately, it only does authentication, without dealing with group maintainance.
However, social networking services are all about groups and group maintainance.
Running through the options -- LinkedIn, Friendster and Orkut are all grabby and gropy and 'my data! mine!', so they're out immediately.
The next step was to take a look at Tribe.net, which seems kind of nice and had a good rep for open APIs -- but as far as I can see, all they've got really in that department is FOAF output, and a simple server-side-include thing called TribeCast. I could list all the group members in a FOAF file, but without authentication, that's pretty useless since anyone could claim to be one of the FOAFs.
That leaves Flickr, which has a great set of APIs. Using that is looking quite promising. If you're curious, I've gone into detail on this at the taint.org wiki.
Patents: DCC (Distributed Checksum Clearinghouse) is a venerable, and widely-used anti-spam system created by Vernon Schryver; we've supported it in SpamAssassin for yonks.
It now appears that DCC is now no longer open source software; it's still free for personal and noncommercial use, but this clause has been added to the new license text:
This agreement is not applicable to any entity which sells anti-spam solutions to others or provides an anti-spam solution as part of a security solution sold to other entities, or to a private network which employes DCC or uses data provided by operation of DCC but does not provide corresponding data to other users.
So there's talk that those commercial users should now license it -- interestingly, from another company called Commtouch, not Vernon's Rhyolite Software. (More info).
It appears that the license change is part of an agreement with Commtouch, owner of US Patent 6,330,590, a patent on the idea of hash-sharing antispam techniques. (I haven't read the patent due to ASF and other policies so I can't tell you what it really covers.)
It looks like we'll be disabling DCC's use in SpamAssassin by default, as we did with Razor, as a result. (Our policy is that the default ruleset used in SpamAssassin be usable by anyone who can use our software, so that the normal usage is open source by default, rather than subsets of the overall functionality.)
Web: Now this is very cool stuff: 'Greasemonkey is a Firefox extension which lets you to add bits of DHTML ("user scripts") to any webpage to change it's behavior.'
In other words, you can rewrite any page viewed in Firefox, as it transits between the server and your client's display; a form of transcoding.
Traditionally, transcoding is performed using a HTTP proxy which applies the transformation, or a specialised HTTP user agent which transcodes and outputs a whole new set of documents with the results.
That was all a little hacky for full-scale integration into your web browser, though, so Greasemonkey is a big improvement for that use-case.
Some good links:
- The Greasemonkey homepage
- The Greasemonkey script repository (wiki)
- Mailing list archives for greasemonkey users
And some demos:
- a good demo of using it to fix a Bloglines bug, by Michael Moncur
- an MSDN bug-fixing script
- a Boing Boing ad-blocking script
- truly awesome: persistent searches (think vFolders) in GMail!
Remember, these are single, sub-100-line JS scripts, running entirely locally in the user's web browser. The last one gives you an idea of what coolness is possible...
My contribution: an ad-removal script for Metafilter. It took some 30 seconds of hacking to produce this -- soooo easy. It's a whole new world of site customisation and hackable filtering. You thought AdBlock was good, this is ever niftier ;)
Tax: Following on from the previous entry, I've had a stroke of luck. It turns out that I did indeed quality as a US resident for tax purposes, and therefore could use Turbotax.
20 minutes later, both state and federal forms were e-filed with the very minimum of fuss -- computers and the net illustrating their worth as labour-saving devices quite nicely. ;)
(Oh -- also -- a PSA for Google's benefit: I'm pretty sure that form 6251 had incorrect instructions. It claims that the items it refers to in form 1040, can also be referred to in form 1040NR by the same numbers. In fact, parts of 1040NR are radically different in numbering than 1040. It's a bug in the form!)
Taxes: it's that time of year again, when every inhabitant of the US, resident and 'non-resident', gets The Fear. Yep, it's tax time. (Warning: this is a long and protracted vent.)
In the US, every worker is required to prepare and file their own taxes, in detail. Nowhere outside of India can do bureaucracy quite like the US, as far as I can tell -- even the brits have embraced simplicity to a greater degree -- so this is no trivial undertaking; however, they do have a few outs, if you're eligible.
Naturally, given my luck, I'm not. ;)
Now, I'm no slouch when it comes to form-filling; I've had to deal with messy forms many times before. But these are masterpieces. Check out this gem:
The ATNOLD is the sum of the alternative tax net operating loss (ATNOL) carryovers and carrybacks to the tax year, subject to the limitation explained below. Figure your ATNOLD as follows.
Your ATNOL for a loss year is the excess of the deductions allowed for figuring AMTI (excluding the ATNOLD) over the income included in AMTI. Figure this excess with the modifications in section 172(d), taking into account the adjustments in sections 56 and 58 and preferences in section 57 (that is, the section 172(d) modifications must be separately figured for the ATNOL). For example, the limitation of nonbusiness deductions to the amount of nonbusiness income must be separately figured for the ATNOL, using only nonbusiness income and deductions that are included in AMTI.
Your ATNOLD may be limited. To figure the ATNOLD limitation, you must first figure your AMTI without regard to the ATNOLD. To do this, first figure a tentative amount for line 9 by treating line 27 as if it were zero. Next, figure a tentative total of lines 1 through 26 using the tentative line 9 amount and treating line 27 as if it were zero. Your ATNOLD limitation is 90% of this tentative total.
Enter on line 27 the smaller of the ATNOLD or the ATNOLD limitation.
Any ATNOL not used may be carried back 2 years or forward up to 20 years (15 years for loss years beginning before 1998). In some cases, the carryback period is longer than 2 years; see section 172(b) for details. The treatment of ATNOLs does not affect your regular tax NOL.
That pretty much appears as-is; there's no additional explanation of those acronyms elsewhere, it's just a big block of jargon. Obviously not intended for human consumption. There's also this:
Medical and Dental. Enter the smaller of Schedule A (Form 1040), line 4, or 2.5 % of Form 1040, line 37.
That seems well and good, and according to the instructions, the 1040NR is 100% compatible with the 1040. Except Schedule (Form 1040NR), line 4 is:
Gifts to U.S. Charities. Gifts by cash or check.
What do charity donations have to do with medical and dental expenses? WTF? (I suspect the compatibility claim is incorrect.)
Last year, I hit up H&R Block for their help; it saved a lot of hassle, but was pretty expensive, costing over $200. Overblown TV advertising alert: of course there was no great refund, despite what their ads claim. However they did recommend that I donate old clothes to thrift stores, keep the receipts, and claim that back as a tax contribution. I'm serious. Given my wardrobe, that should net about $10.
This year should be a lot simpler, since I'm just a US nonresident working visa holder doing nothing but paying plain old income tax -- so I was intending to just fill out the forms myself.
I think I'll tick that idea off my list and check out the online options.
All I can say is, no wonder quite a few US citizens seem to think that government involvement is something to be minimized if at all possible. There are alternatives though -- I'd happily take an Ireland-style 'nanny state' which will compute my tax liabilities for me if I so choose. It's not like I'd be in a position to argue with them anyway, aside from the common case of hiring a tax attorney, if we disagree; so why not let the government do the heavy lifting? ;)
(PS: the good news is it now appears I may qualify as a resident. This means Turbotax.com is a viable option... yay!)
Misc: So I was travelling last week -- a very productive trip to the UK visiting the main work dev office, and getting a little socialising in too while I was at it. A pretty good trip overall, especially since I seem to have figured out how to use my frequent flyer miles effectively to get great seats! ;)
Here's a good interview with SpamAssassin PMC chair, Daniel; well worth a read if you want to see what we in SpamAssassin think about the state of the onion in spam-filtering.
In not-so-good news, it seems Charlie McCreevy has managed to push the software patent directive through, despite massive EU Parliament unhappiness. Third time around at the Fisheries meeting, naturally; and there's some serious questions about the legitimacy of the procedural rules invoked by the Commission in refusing to take the directive off the A-item menu. Now that's what I call democracy...
It can still be defeated, but it's an uphill battle now -- for it to be thrown out in the second reading at the European Parliament, it'll need a two-thirds majority of all MEPs (not just the MEPs present), reportedly.
In the meantime, thanks to the FF and PDs' bullying tactics, Ireland's small but growing pool of homegrown software developers are being ignored, and the Irish software industry looks more like a lame import operation for the likes of Microsoft. Our reputation is dragged through the mud for a few multinationals, and the rest of Europe resents us for it. Wonderful.
BTW, even if it does pass, there are ways to fix it -- directives must be implemented into national law in each country. This means that Ireland could still write their implementation of the directive to exclude software inventions (even the ones where it's supposedly a patent on hardware like 'a CPU connected to a hard disk, with such-and-such software running on the CPU'). However, given McCreevy's obvious bias in favour of getting this specific text into place, how likely is that going to be?
Patents: So the Conference of Presidents has ratified the JURI decision to throw out the flawed software patents directive text. Phew! That's a lot more pressure on the European Commission. Charlie McCreevy could still carry on his attempt to steamroller European democracy on this one, but it looks likely that he wouldn't get away with it now -- possibly facing sanctions as a result.
Found in a Slashdot comment -- an amazing quote from Michel Rocard (former French Prime Minister, now European Deputy), recounting a meeting with Microsoft representatives on the software-patent issue:
"We never could (speak) a common language with the companies representatives we met - in particular those from Microsoft. Speaking about (the free circulation of ideas), free access to knowledge, was like speaking chinese to them. In their way of thinking, everything that is not usable for immediate profit ceases to be an engine of growth. They don't seem to be able to understand that an invention which is a pure spirit creation (sic) can't be patented. It's simply terrifying. Many of us, at the Parliament, agree to say that they never have know such a pressure and such a verbal violence during their parliamentary work. It is a huge case."
In addition, he takes aim at the Irish Presidency's tactics:
"To adopt it formally, there is an expeditious procedure -- the (A-item) at the Council of Ministers, where the it is adopted without discussion. The Irish and Dutch presidencies attempted this tactic three times, twice at meetings of the (Fisheries Council)! This is simply scandalous."
Blimey, he's really pissed off. Great! Go Rocard! ;)
See here for the original interview (in French), and here for a bad Babelfish translation.
In happier news -- take a look at some pictures from the presentation of 30,000 verified signatures (and flowers!) from people around the world, thanking the Polish Government for their repeated stands against the flawed directive in December.
Code: Ugo Cei: Building Interactive Web Programs with Continuations quoting Phil Windley:
This leads to the question: what if I could write programs for the Web that were 'structured' in the programming sense of that word? The result would be Web programs that were more natural to write and easy to read. You'd no longer have to maintain the state of your program outside the language and the data could be kept in variables, where it belongs. The answer is: you can.
I hate the 'save all state' model imposed by developing for the web, and have been hoping for a way to do this for a while -- and now I know what it's called ;)
It seems Seaside is the leading continuations-based web-app framework, using Smalltalk, and (as Ugo noted) Apache Cocoon has it too, but there's a whole load more. Can you tell I haven't been following web-app development techniques much recently?
Never mind those other languages, though -- Continuity looks promising as a Perl framework based around continuations. Perl 6 will reportedly have native continuation support, and Dan Sugalski gives a good write-up of how they're implemented and their ramifications there.
Patents: Børsen: Bill Gates threatened to kill 800 Danish jobs if Denmark opposed software patent directive:
Danish financial newspaper Børsen reports that Microsoft founder Bill Gates threatened the Danish government in connection with software patents. According to the article, Gates told Rasmussen and two Danish ministers in November that he would kill all 800 jobs in Navision, a Danish company acquired by Microsoft in 2002, unless the EU were to quickly decide to legalize software patents through a directive. Denmark is a country with only 5 million inhabitants and a relatively small high-tech sector to which the loss of 800 jobs would have significant implications.
Lovely -- a blunt blackmail attempt. The article goes on:
It would not be the first threat of its kind. A group of large corporations including Philips is reported to have previously threatened European governments to outsource all of their European software development jobs to low-wage countries unless the EU were to allow patents on software through the directive that is currently being worked on.
In January, leading Polish daily Gazeta Wyborcza reported on a letter addressed by the Polish subsidiaries of Siemens, Nokia, Philips, Ericsson and Alcatel to Poland's prime minister Marek Belka ... it is said to have indicated that the respective companies would reconsider making investments in Poland if the Polish government upheld its resistance to the legalization of software patents in the EU.
Again, note the FUD-busting on this point. I notice that Florian Mueller of NoSoftwarePatents.comhas a a good one-liner response along the same lines -- 'The country in which you develop a technology has nothing to do with where you can take out patents.' He goes on:
If they move jobs to Asia, they won't get a single additional patent, neither in Asia nor in Europe. If you warn politicians of consequences that are directly related to a legislative issue, that's acceptable. If you threaten with causing damage that has no factual connection whatsoever, then it's blackmail. Plain and simple.
Tech: Bit of a long essay, this one.
World+dog have been linking to this interview with Flickr's Stewart Butterfield on the O'Reilly Network, so I wasn't going to bother. But I came across a great illustration of what I think is a very important point:
Koman: In the write-up for your web services session at ETech, you say, Capturing the creative energy of the hive can be scary. It requires giving up some control, and eliminating lock-in as a strategy. Tell me some more about that.
Butterfield: Ofoto is a pretty good example. I don't want to pick on them too much, but they create a pretty artificial kind of lock-in. When you upload your pictures to them, you might upload a three- or four-megapixel image, but all you can get back from them is a 600-pixel image; if you want to get the original back, you have to buy it on a CD. There's no way to get it out because if you got it out, then your friends and family could get it out and print it out at home, and they're in competition with Lexmark and HP as well as the other online photo services. So that's one aspect of it.
There's also a tendency to want to capture all the value that's being generated or will potentially be generated by new business. What I mean by that is, we don't explicitly allow commercial uses of the API yet, but we definitely plan to. And we know that there are people working on products based on our API that we want to do, but outside developers will get to it first. What letting go in that context means is letting go of all the control you have over users by being the one who owns the database, because other developers can generate businesses and products that hook into you, and that takes some value away.
This is a point that still, to this day, most people miss.
The traditional viewpoint is that, if you've got something, you hoard it, and ensure you're the guy who makes the money from it. So you do what Ofoto do -- you keep the full-resolution images, and charge for access to them; or you don't publish APIs, and keep the data to yourself; or in the world of source code, you hold onto the source so no-one else can see it, because it's your 'crown jewels'. Then, the idea goes, you can ensure that you're the only one who can do prints, or add a feature to the source, or whatever.
But the problem is, you're not always the one with the idea; or alternatively, every feature request has to go through you, and be implemented by you, on your time. And in the meantime, your users are considering the big question -- 'do I want to get locked in, here? what if he goes out of business? am I a small customer who's going to be ignored?'
In fact, I've been guilty of this myself. When I started writing open-source software, I used the GPL as a license, which prohibits commercial use (mostly) -- except by myself or through my explicit permission. I had no intentions of making it available for commercial use, because I couldn't see the commercial uses.
But that was me being short-sighted -- soon, people starting asking if they could license the code for commercial use, or hire me. I realised that I didn't have the time, or inclination, to go the whole hog, and risk my livelihood on a piece of software -- especially risky since I didn't think that software could support me alone.
So when I wrote SpamAssassin, I picked the Perl dual license, a license that did permit commercial use, while still being an open-source license. By now, there are quite a few commercial versions of SpamAssassin, all making money (I hope!), I'm getting paid to work on SpamAssassin, and everyone's happy ;)
Perhaps I should have kept commercial rights to myself. But I have no doubt that doing so would have ensured SpamAssassin remained a small-time solution, and would not have received the number of contributors, committers, and patches it has by now. (for example, Matt Sergeant, who was an SpamAssassin committer, joined the project explicitly to use that code in MessageLabs' product.)
Plus, at the time, there were already quite a few commercial competitors -- and there's a lot more to being a commercial success than the simple things required to be an open-source success; I'd be dubious that SpamAssassin would have been able to compete as a purely-commercial play, and I'm not sure I'd have been keen to risk my livelihood to do so, anyway. (I'm not really dot-com CTO material, anyway. I like hacking code too much.)
I think things have worked out well: the software's better, I'm earning a livelihood from open-source software regardless, and the software's usable for more people. As usual, Larry Wall was right ;)
Spam: recently, I've been getting a lot of spam bounces; that is, messages sent by people's autoresponders, in response to forged spam claiming to come from my domain. (I have an SPF record, but these autoresponders naturally don't bother to check that before replying.)
I have a SpamAssassin ruleset which catches these, and it gets rid of the vast majority -- but the odd wierd one gets past. This one caught my eye before I deleted it:
On October 5, 2004, I will be going to the Illinois Department of Corrections for approximately 18 months. If you wish to contact me, please snail mail me at: (address deleted)
Your letters will be forwarded to me and I will reply as soon as I receive them! Thanks...and please do write! Mail is vitally important! :-)
... ouch. Good luck to this guy, whoever he is...
Spam: Should ISPs Be Profiting From Knowingly Hosting Spam Gangs? -- a new article up on Spamhaus.org, well worth a read. Some snippets:
So where is this stealth proxy spamware sold and distributed from? For Send Safe the answer is, www.send-safe.com, hosted by MCI Worldcom.
... MCI executives have refused to stop providing service to these gangs, insisting that the sale and distribution of stealth spamming software is not against MCI's policy.
... It's no surprise therefore that MCI has consistently occupied first place in Spamhaus TOP 10 World Worst Spam Service ISPs chart, with over 200 spammers and spam gangs on the MCI network in full knowledge of the security managers and the General Counsel.
... MCI Worldcom's official position on the issue is that MCI can't stop their spam gangs selling proxy hijacking spamware from MCI's network as that would be 'censoring' the distribution and sale of illegal proxy hijacking software.
Antarctic: It seems that Ernest Shackleton, during his exploration of Antarctica, relied heavily on 'Forced March' tablets:
Reportedly 'sold over the counter at Harrod's until 1916', these were primarily cocaine-based.
Patents: FFII are reporting that 'the Legal Affairs Committee of the European Parliament (JURI) has decided with a large majority to ask the Commission for a renewed referral of the software patents directive. With only two or three votes against and one abstention, the resolution had overwhelming support from the committee, and all-party backing.'
Michel Rocard MEP gave a very strong speech at the meeting with the Commissioner. Apart from noting several "inelegancies" by the Commission, such as not taking into account any of the Parliament's substantive amendments in its recommendation to the Council, he also took issue with the Dutch and German governments ignoring their respective parliaments, the Irish Presidency's sponsorship by Microsoft and the attempted ratifications of the political (dis)agreement at several fishery Council meetings.
He mentioned that at a meeting with the Polish government, the industry players confirmed that the Council text allowed pure software patents, and wondered how the Commission could continue claiming the reverse. He was also curious about how the Commission's perfectly tautological definition of the concept "technical" could help in any way to distinguish between what is patentable and what is not. Despite his own abstention when voting on the restart later that day, the fact that almost everyone else supported it is probably his personal achievement.
The Commissioner made clear that "any agreement will need to strike a fair balance between different interests", and that "a constructive dialogue between the Council and Parliament will be vital for an agreement". He does have the option to deny a new first reading. But given the strength of feeling in the Parliament and the concerns of so many member states in the Council, the Parliament request looks like the best way to achieve a clean way forward for this Directive that everyone has been looking for.
This is good news for the anti-swpat side. Nul points for the Irish Commissioner, Charlie McCreevy, who 'had in the morning assured the JURI Committee that the Council would finally adopt its beleaguered Common Position text. He announced that "the Luxembourg Presidency has now received written assurances concerning the re-instatement of this issue as an A point at a forthcoming Council". Given that A points are to be adopted without discussion, this left no possibilities for renewed negotiations in the Council'.
Networking: Donal Cunningham, president of SAGE-IE, mails to note an interesting talk on in Dublin next week:
The System Administrators' Guild of Ireland and Dublin University Internet Society present...
What : From the ground up; a greenfield deployment in Liberia
Who : Comdt. Kieran Motherway, Corps of Comms. and IS, Defence Forces
Where: Walton Lecture Theatre, Arts Building, TCD
When : Tuesday the 8th of February, 7 p.m.
Why : The Irish Defence Forces deployed to a greenfield site in Liberia in 2004, and had to build Comms/IT infrastructure from the ground up. Comdt. Motherway will talk about the Irish Army's experiences with this deployment, and just how far removed from an air-conditioned, climate-controlled comms room you can get...
Sounds like fun, and I know a few taint.org readers will be interested ;)
Freevo: so I'm planning to build myself a PVR, of the home-built, running Linux with mythTV or Freevo, mini-ITX variety.
So far I'm still at the hardware planning stages, but the price looks good -- around $455 (plus shipping) for a working, thoroughly hackable, silent, set-top PVR system.
(Silence is a key aim here -- last thing I want is something noisy taking over the room. But silence typically seems to cost the dollars, once you get into Shuttle gear and the like.)
If anyone wants to follow along, or provide some tips -- I'm going to track progress (very slowly) on this wiki page. Like all wiki pages, it's editable -- although you'll need to create an account to edit pages there (sorry, anti-spam measure).
BTW, lately, there's been a lot of talk about using a Mac mini as a media center. So I took a quick look -- but wow, it's pricey! $499 + $329 for an EyeTV 200 tuner? Dude, that's over 800 dollars, not include shipping or sales tax. Given whatever extras turn out to be appropriate, I wouldn't be surprised if it hits double the mini-ITX's price.
Tech: January 24th, besides being the date the first Apple Macintosh went on sale, is supposedly the day of maximal post-xmas misery. Well, it certainly was for me today.
I decided to power on my old desktop to set it up as a back-room fileserver, and twiddled the partition table accordingly to nuke a few unused Windows partitions and maximise usable space.
Somehow or other, some component of my system decided that it would henceforth be non-bootable. It seems some BIOSes don't like partition tables where a high-numbered logical partition have a lower starting sector than a boot logical partition, or something... GRUB just errored out with an obscure 'Error 17', which apparently means that it couldn't find its boot partition any more.
OK, so I needed a boot disk. But I had 1 laptop with a CD/DVD drive but no floppy drive, and a desktop with a floppy drive but no CD drive (due to hardware failure)... and the original linux boot floppy was long gone, seeing as I'd hardly booted this machine in the duration of two house moves. Argh.
A dinky little Cruzer mini 128MB USB flash drive saved the day. (R)ecovery (I)s (P)ossible is a tiny Linux distro that fits into 27MB, well inside the USB drive's limits; it has an exceptionally helpful and detailed README detailing exactly what needs to be done to create a bootable USB flash drive from its ISO image, using just the generic linux toolchain.
Together with fdisk and parted's 'rescue a lost partition' mode, I was able to get the mangled partition table back into shape, mount the boot disk, change the fstab and grub configuration file, and reboot into a working system. phew!
Many thanks to Kent Robotti, who's done a great job with RIP.
On the other hard -- no thanks to whoever came up with the arcane rules behind the IDE partition table... argh.
Spam: Before xmas, I received a copy of Brian McWilliams' new book, Spam Kings.
It's a great book -- full of behind-the-scenes
details on how the spammers operate, how they get away with it on the
sending end, how they try to evade filters on the receiving end, and how
they're fundamentally running the usual simple scams that have been around
since before email spam came into existence. Well worth reading.
In addition, Brian's continuing to write about spam and spammers at the Spam Kings weblog, and will be giving a talk at this year's MIT Spam Conference, tomorrow.
Anyway, pick up a copy if you're interested in the spam problem -- this is one of the best books I've read on the subject, and this kind of information is essential for an understanding of the people we're up against.
Blogs: Just to expand on a linkblog
posting I made yesterday, Google's search team have announced support
for a new piece of Google functionality; they'll fix their crawlers to
ignore links with a rel="nofollow" attribute, for PageRank
calculations, the idea being that spammers will stop blog-spamming
once they can't get PageRank out of it.
The blog world has been all aflutter:
- Google Blog
- MSN Search
- SixApart
- Dave Winer
- Robert Scoble: 'now we can link to things without raising their search engine whuffie'
- burningbird: 'I remember something like this being discussed before'
BurningBird is right, to a degree. In fact, it's been solved before.
Here's a taint.org posting from November 2003 where I point out that by using a trivial Javascript URL one can link to another page without conferring PageRank. The format is:
javascript:document.location=target
The result looks like this, and work in any browser with a basic JS engine, from IE 3.02 and Netscape Navigator 2 onwards. I've been using it for my referrer logs, among other things, for over a year. I wrote a patch that implemented it for external links in the Moin Moin wiki software.
Amazingly, despite my plugging this idea at virtually every opportunity, it seems nobody noticed! At least, nobody among the people who (it would seem) should be looking into comment spam, thinking about how to deal with it, etc.
Disappointing -- the echo chamber keeps talking to itself, once again. Maybe I'll stick with dealing with email spam instead ;)
Ah, whatever. Anyway, this is a nicer fix; relying on JS isn't a good thing. So nice work, Google.
(PS: worth noting that while this is a good plan, comment spam won't be going away any time soon, as Mark Pilgrim noted. Still, here's hoping it'll help in the long term...)
Spam: I was just looking back through the archives here on taint.org, and noticed this entry from December 2 last year:
A huge 300 ft. high ocean wave is moving towards your continent. Your and many other cities are in a real danger. Approximate wave moving speed is 700 km/h. cmoym eaaa yypbzz
Please read more about this catastrophe here: (link)
We are strongly urging you to evacuate yourself and your family as soon as possible, even though you may live far away from your city. The tsunami will reach the continent in approximately FOUR hours.
It appears that the spam was a phish attack -- the site in question is full of Internet Exploder exploits. It was 'targeted', at least as well as such things ever are, at Australian readers. AUSCERT issued a warning about it at the time.
But how's about that for timing? Spooky! What did those phishers know?
Spam: eWeek recently published an article entitled 'Spammers' New Tactic Upends DNS' , which notes that:
One .. technique finding favor with spammers involves sending mass mailings in the middle of the night from a domain that has not yet been registered. After the mailings go out, the spammer registers the domain early the next morning.
By doing this, spammers hope to avoid stiff CAN-SPAM fines through minimal exposure and visibility with a given domain. The ruse, they hope, makes them more difficult to find and prosecute.
The scheme, however, has unintended consequences of its own. During the interval between mailing and registration, the SMTP servers on the recipients' networks attempt Domain Name System look-ups on the nonexistent domain, causing delays and timeouts on the DNS servers and backups in SMTP message queues.
This had me stumped when I read it, since an email from a nonexistent domain is a pretty reliable spamsign (it's used in the NO_DNS_FOR_FROM rule in SpamAssassin, for example, which hits about 2% of spam), has been a rule in the default ruleset for several years, and there's no sign of that behaviour in our spam traps.
After some discussion, Suresh Ramasubramanian came up with this explanation of what's really happening:
Verisign now allows immediate (well, within about 10 minutes) updates of .com/.net zones (also same for .biz) while whois data is still updated once or twice a day. That means if spammer registers (a) new domain he'll be able to use it immediatly (sic) and it'll not yet show up in whois (and so not be immediatly identifiable to spam reporting tools) - and spammers are in fact using this "feature" more and more!
That does sound a much more likely explanation, and matches what's been seen in the traps.
So: WHOIS, not DNS.
Code: Here's something I came up with recently -- it's actually
an evolution of the idea of pushd and popd, as included
in BASH.
To quote the POD docs:
cdhistoryis a perl script used to implement web-browser style "history" for UNIX shells; as you use thecdcommand to explore the filesystem, your moves are remembered, and you can go "back" through history, and "forward" again, as you like.
Download the perl script here.
Politics: This moronic comic from Pat Oliphant came up in my comics page the other day, and, after a few days of hearing this particular talking point through the usual propaganda channels, I just saw it again. It pissed me off enough that I took a look at the stats.
Naturally, it's bullshit. The top 50 governments pledging tsunami aid, per GDP:
- Qatar (#2)
- UAE (#5)
- Kuwait (#9)
- Bahrain (#10)
- Saudi Arabia (#15)
Given that the USA's at #29, and the UK at #22, I think the arab states are coming up with a pretty good result there.
I guess it's hard to look beyond today's talking points when you're still drawing cartoons at the age of 70.
Web: Urgh, I still have this damn cold I picked up in Ireland... sniffle cough etc. More vitamin C needed!
Anyway, just a quick plug for a very deserving Firefox extension, one I haven't seen mentioned widely. It's pretty common, when you wish to print out a web page, that you wish you could get rid of the obnoxious extra-wide sidebar tables, gigantic ads, or other extraneous parts of the page. Well, now you can:
Nuke Anything is a Mozilla/Firefox extension which offers two great features in the right-click context menu:
- Remove this object: this will remove the object you've right-clicked on -- a table TD, paragraphs, images, IFRAMEs, etc.
- Remove selection: more usefully, this allows you to select exactly what you want to remove with a left-button drag, then right-click to remove it.
It's really useful. I almost never print anything out these days without scrubbing off a few unwanted sidebars ;)
Patents: here's an interesting technique I heard recently. (credit: I'm not sure who told me about it, but I think it may have come from or via John Levine.)
If you become aware of a patent application (note: not an issued patent!) for which you are aware of possible prior art, you may be able to help invalidate it, or at least ensure any resulting patent is narrow enough to be relatively sane. Here's how.
- If you have knowledge of techniques that you believe may be prior art, you can send them on to the filers or the patent examiner. At this stage, the onus is on them to prove that the technique is not prior art for the application (once it's granted, the onus would be on you to prove that it is).
- The filer also must indicate techniques that they are aware of, that may be prior art, during filing; so CC'ing a public forum with a copy of whatever you send to them, may at some point in the future help indicate that they did not do this.
Of course, you have to go find the patent application number, the contact addresses of the filers, and the contact address for the patent examiner to do this ;) But it beats posting a whinge to Slashdot.
An unnamed patent agent comments:
'I believe an examiner is not under obligation to review art sent directly to them, but certainly the applicant and his agents are required to report any art they come across. That means the inventor as well as the law firm representing them.
You should include a cover letter that you saw their application (give details), and that you believe that what you are sending them is prior art, and that now that they have it, they are obligated to report it to the PTO. The same can be done to their counsel.
Probably, anything sent should be sent with some sort of delivery confirmation, and to make sure that the sending of the prior art is of public record, create a Web site where all sent art is listed, along with destination and confirmation information. This would help show inequitable conduct should the patent later be asserted and the art you provided not be shown as of record in the examination.
Mind you - I have not heard of these being done before (bombarding listed inventors and their agents with prior art, forcing them to have to disclose it), but I think it's a great idea. One caution - if you send too much, you over inundate the examiner, and then really good art could get overlooked during examination.
Separately, please keep in mind that the claims in a published application have probably not yet even been seen by the examiner at the PTO. These are the claims that the applicant would love to have the examiner accept, but until prosecution of the application actually commences (and completes), there's no way to know what claims will ultimately result.'
Update: some good additional points:
'The prior art must have been published or been publicly available at least as early as the earliest priority date of the patent. The priority date is either the filing date, or the filing date of a parent application. This information can be found on the cover page of a patent.
A patent's scope is covered by the claims. The claims define what the invention is. All other material in the patent is supporting material, and usually non-binding. In order to be anticipatory (the best kind) prior art for a particular claim, the piece of art must contain or described every element of the claim you are seeking to invalidate. Note that dependent claims add additional elements that the prior art needs to contain if you want to invalidate the dependent claims as well.
Prior art which is not anticipatory may be used in combination with other art or knowledge at the time to show obviousness. This type of art may have some impact during prosecution of a patent, but if a patent has already been issued, obviousness is a real uphill battle to fight in the courts. Few patents have been invalidated because of obviousness in trials.'
Another attorney notes: 'You can actually send it anonymously if you want. Just keep the certified receipt to prove they got it. As long as they know it exists, the onus is on them to disclose it to the PTO.'
'It's best to send them something printed out or on tangible media, along with a brief note explaining what it is and most importantly, when it was first publicly available. Certified means using certified mail or FedEx or something where you have a valid receipt.
As far as (discovering) who the (filer's patent lawyers) are ... it's usually listed on the patent applications. you can search the USPTO website for them.'
And a report that this technique is now in use: 'some patent attorneys are reporting that this approach is a valid one that people have started using.'
Update 2: More assent from another unnamed patent lawyer:
'Anyone who wishes to do so can send a letter to the Patent Office letting them know of any prior art of which they are aware. The Patent Office will then place it in the application file. Anyone who cares about this patent will surely order up a copy of the application file from the Patent Office, and will come into possession of whatever you sent.
Later you can see whatever you sent them. Go to
http://portal.uspto.gov/external/portal/pair and plug in the serial number
(for the desired patent). Click on "image file wrapper".'
It's the right thing to do for any patent or patent application.'
Spam: I'm still catching up, but this is just plain hilarious. Pure, solid-gold, insanity. Verizon.net, the ISP branch of the US telco, has decided that the easiest way to fix their spam problems (uh, spam-receiving problems, that is), is now blocking inbound email from non-U.S. IP ranges:
A little birdie with insider knowledge has confirmed that Verizon is blocking all international IP space from RIPE, APNIC, and more, and is only unblocking specific domains, based on their IP address, when complaints are made and escalated.
According to the source 'the security team management thinks this is going to stop their inbound spam problems.'
Well, it may stop their inbound spam problem, but it's also going to stop that pesky 'wanted email making it to their customers' problem.
A quick check from my Ireland-hosted colo box does indeed indicate that this is still the case, and I can't connect to relay.verizon.net (206.46.170.12):
: jm ftp 1...; telnet 206.46.170.12 25 Trying 206.46.170.12... telnet: Unable to connect to remote host: Connection timed out
Linux: after about 3 months of tweaking and twisting, performed by someone who's been using UNIX for over a decade, I've finally got sound working the way I want it on my Linux desktop. In other words, I can hear sounds made by Flash applets, and I don't have to shut down the best music player on the platform every time another app wants to make a sound.
This is pretty clearly absurd.
So here's my open letter to the developers of the various systems (GStreamer, aRts, ALSA, EsounD polypaudio, et al):
- Please DO do some testing with crappy sound hardware. I don't care if your sound system works great with a SoundBlasterLive 2006 with the kryptonite connectors, I have a laptop, for god's sake. That means software mixing is essential, because cheapo hardware doesn't do hardware mixing.
- As an extension: please DO include software mixing by default. ALSA's pretty good in general, but having to hack out 55 lines of hand-tweaked config file before software mixing works, is insane. (Especially when the Wiki documenting that is full of notes that some of the magic numbers may not work on your hardware.)
- Please DO use existing APIs if possible. That means
esd. I'm looking at you, aRts. At least the latest sound project, polypaudio, looks like it's getting this right. - I DON'T care about network transparency, realtime response, or having a wah-wah pedal effect built into my sound server. That's just silly. Use a modular architecture to allow that in future, but concentrate on getting the basic stuff working first!
- Please DON'T hardcode output device or output 'sink' names into the source. Looking at the kgst component of KDE here.
Meh.
Anyway, here's the scoop on what I had to do to get software mixing working in both GNOME, KDE, and Firefox, on my Thinkpad T40 running Debian unstable. Once I figured out the magic incantations, it now seems to be working without stutters or hangs.
Sometime in the next few months, of course, I plan to upgrade to Ubuntu Linux, and all bets will once again be off ;)
Spam: The Business Software Alliance, a UK anti-piracy body representing many of the major software vendors, recently issued a spam-related press release which got a lot of attention in the UK press (they have great press contacts!).
To quote John Graham-Cumming's newsletter on the subject:
1 in 5 British Consumers Buy Software from Spam: that's according to a survey by the Business Software Alliance. I find that a pretty surprisingly high number and considering it comes from an advocacy group that tries to get people to buy legitimate copies of software I expect it's not totally accurate. The one thing I find really surprising from the survey are these two statistics: 23% of spam is read by the person receiving it and 22% of people have bought software. Apparently, 11% of people surveyed like the idea of buying through spam because the software is cheaper.
It's still an interesting figure, but the BSA has come up with some pretty suspect statistics in the past, so pinch of salt applies. As jgc points out, the BSA have a vested interest in making the problem sound worse than it may be in reality.
Still, the survey PDF can be read here, and is worth a look.
Patents: This is really absurd -- according to this ZDNet UK article, it now looks like the EU Council is considering railroading the EU software patent directive through, by hiding it as an 'A-item' in a Fisheries Council Meeting the week before xmas:
Laura Creighton, the vice-president of the Foundation for a Free Information Infrastructure (FFII), is concerned that the EU Council could be contemplating passing the directive without discussion in an unrelated meeting.
'Before today it was possible for generous people to look charitably at this text (the proposed patent directive) as an example of a tragic mistake, not malice,' said Creighton in a statement on the FFII Web site. 'But not with this last-minute manoeuvring.'
'Only the most committed opponent to the democratic process would believe that the proper response to the widespread consensus that there is something profoundly wrong with the Council's text is to race it through with an A-item approval the week before Christmas in a Fisheries Council Meeting. The bad smell coming from Brussels has nothing to do with the fish.'
Reportedly, A-items are dealt with by asking the assembled councillors if they have any objections to any of the outstanding items. They're not listed in detail at the meeting, so this way the directive can be passed in what is effectively a submarine (boom boom!) manner.
Related: Alan Cox has not been invited to the UK Patents office's public meeting on software patents tomorrow.
In a Talkback to ZDNet UK's earlier story highlighting the issue, Cox wrote: 'I too was mysteriously overlooked despite having written to my MP and received an answer.' .... Cox, who has previously been invited to speak on software patents at the EU, said the Patent Office apparently fears 'every word I have to say about their plans'. He went on to add: 'Unfortunately with all the underhand game playing both in the EU council of ministers and in UK government and patent circles it isn't the slightest surprise.'
Also related: Jason Schultz (EFF) on the Commerce One web-services patent auction last week:
Here, the patents at issue were less valuable to companies that actually produce Web services products than they were to firms that produce nothing but lawsuits and licensing threats. In other words, patents like these have become worth more as weapons than as protections for companies competing in the marketplace.
Many have compared these new patent licensing firms to terrorists, and in some ways, the analogy is apt. When the Soviet Union collapsed, one of the biggest worries was that rogue military personnel might sell off one or more of the USSR's nuclear missiles to a terrorist group. Securing those weapons became a top priority. The reason was fear -- fear that the terrorists, who had little to nothing at stake in terms of world peace and national stability, would use the missiles to extort or manipulate the world political climate. Unlike the United States or China, which could be retaliated against and which had a stake in stability, terrorists were essentially immune from attack, and thrived on instability.
With the patents of bankrupt dot-coms, the dynamics are similar. Rogue licensing firms buy up these patents and then threaten legitimate innovators and producers. They have no products on which a countersuit can be based and no interest in stable marketplaces, competition or consumer benefit. Their only interest is in the bottom line.
While profit itself is often a worthy objective, it is not always synonymous with innovation. Every dollar a tech company pays to patent lawyers or licensing firms is one less dollar available for R&D or new hires. Thus, many companies that offer new products end up paying a 'tax' on innovation instead of receiving a reward. When this happens, it's a signal that the patent system is broken. Forcing companies to pay lawyers instead of creating jobs and new products is the wrong direction for our economy to be headed and not the result our patent system should be promoting.
Patents: oh, this is painfully ironic.
patents4innovation.org is a PR site set up by EICTA, a consortium of several pro-software-patent multinational companies, to put some PR money into lobbying for the legalisation of swpats in the EU. I've mentioned it before in the context of another boo-boo. Well, here's the next one.
According to FFII, they recently took a Creative-Commons-licensed article from another website, and:
- republished it without the required attribution to the author
- translated it, creating a 'derived work', against the terms of the license
- and then failed to notify readers of the licensing terms, as required
In other words, they managed to infringe the terms of its copyright-based licensing in multiple clauses.
No wonder they claim that patents are required to protect people's inventions. It seems they just don't understand how copyright-based licensing works ;)
(The article's been taken down from the p4i site, but not before the boo-boo was spotted by an eagle-eyed FFII'er.)
Spam: some good crazy spam recently -- firstly, some Seventh Day Adventist lunacy:
THE PAPACY IS THE ANTICHRIST THAT IS TRYING TO CHANGE THE LAW OF GOD. DANIEL 7:25
THIS IS THE LAST WARNING.
THE LAW OF GOD IS ETERNAL BECAUSE GOD IS ETERNAL 14:12. MT. 5:17 SATURDAY SEVENTH DAY IS THE TRUE LORD'S DAY. EXO. 20.8-11 SUNDAY IS A FALSE PAGAN DAY. IT IS NOT IN THE BIBLE. IT WAS USED TO WORSHIP SATAN
It runs on in that vein for quite a while. Interestingly, most of the text from there on in is 'gappy' -- in other words, the spammer has inserted spaces between each character of a word -- even inside link addresses. As a result, they no longer work. oops!
And a new one to me -- natural-disaster spam (via Mark Pilkington):
THIS IS AN OFFICIAL WARNING!
fngva uvtt chloezA huge 300 ft. high ocean wave is moving towards your continent. Your and many other cities are in a real danger.
Approximate wave moving speed is 700 km/h.
cmoym eaaa yypbzzPlease read more about this catastrophe here: (link)
We are strongly urging you to evacuate yourself and your family as soon as possible,
even though you may live far away from your city. The tsunami will reach the continent in approximately FOUR hours.venbz nwvw exepmi
YOU HAVE BEEN WARNED!
I've removed the link, btw -- the site it links to contains a bunch of nasty malware-installing IE-bug exploits. In case you were wondering: you can tell it's genuine because it says IT'S AN OFFICIAL WARNING at the top.
(ObSpamComment: note -- this here's a good example of why spam is unsolicited bulk email, not unsolicited commercial email; neither are selling anything. one's religious craziness, the other one's trying to r00t your machine.)
Life: I'm moving house -- I've just filled about 20 boxes, now to get moving them! Sadly, there's no wifi in range of my new house, so the upshot is I may be offline for a few days. Boo.
Patents:
Newsweek: Factory of the Future?:
The dino's ferociously bared teeth hint at elements of Intellectual Ventures' bold business plan. Myhrvold and his partner, former Microsoft chief software architect Edward Jung, have created the quintessential company for the 21st century. It doesn't actually make anything ... Only patent attorneys populate the quiet hallways. ...
Sources familiar with Myhrvold's strategy say that he has raised $350 million from some of the largest companies in high tech: Microsoft, Intel, Sony, Nokia and Apple. Google and eBay also recently invested. With this large bankroll, the company is out buying existing patents in droves. (Myhrvold won't comment on these activities, but sources say he has already purchased about 1,000 patents.) The strategy is to set up a sort of patent marketplace. Patent owners get money upfront for the dusty ideas sitting on their shelves, the investors get the rights to use the ideas without being sued and Myhrvold gets to rent those same ideas to other companies that need them to continue creating products. ...
"We're concerned that these giant pools of patent rights are going to prevent entrepreneurs from entering markets, as opposed to being used to promote innovation," says one worried Silicon Valley venture capitalist.
Now that's scary...
Spam: The EFF are a great organisation -- damn, I even helped set up an organisation based on its goals in Ireland, back in the day! But this white paper is shockingly clueless.
(Note: this posting has been updated. Original left intact, but there's an update below worth noting.)
For example:
Spam Assassin, a popular program that does ad hoc pattern matching, assigns 'points' to various features of an email to determine whether it is spam. ... One of the major problems with this system is that messages from certain countries -- like China, for example -- can be blocked purely on the basis of where they come from and what language they're in. The implications for free speech here are very troubling indeed: ... thus anti-spam technology unintentionally works as a political censorship mechanism.
SpamAssassin does not give points for country of origin, or language the message arrives in, unless the user explicitly either (a) adds rules from an external source, or (b) modifies the 'ok_languages' setting in their configuration, from the default, to specify that they do not want to receive messages in particular languages. No country- or language-blocking happens by default. This is by design.
It's a shame that the authors felt the need to outright fabricate a danger, here.
The white paper features more broad generalisations about 'spam filters', mostly using unsubstantiated friend-of-a-friend stories, without detailed data. And I do know that there have been cases of MoveOn.org, at least, being a source of UBE, in the past -- so it's not valid to claim that this is all a 'free speech' issue; political UBE is still spam.
They need to realise there's a lot of very smart, very reasonable anti-spammers out there, and most of us agree with the rest of their goals, except for their spam position. This is hurting them.
Still, it appears they're finally getting a clue about requiring subscription requests be confirmed using closed-loop opt-in, so that's good. More political newsletters, and political campaigns, need to get this clue -- just because it's political speech does not mean it's not spam. (I have several thousand political spams in my spam folder -- most from that German anti-immigration virus from earlier this year.)
Note that Rod is unsure if they're practicing what they preach...
Update: Annalee Newitz has been in touch, and pointed out that the white paper in fact says 'mails ... can be blocked', rather than 'are blocked' based on country of origin. In other words, it's purely a matter of this being possible, rather than the default, and that administrators apply these customisations.
In addition, she notes that the conclusions recommend that ISPs and administrators of spam blocking systems allow end users to control their own filtering settings, saying 'If a user wants to block all mail from China, great. If a sysadmin does it for a bunch of users without permission, then that is a problem in our opinion.'
So I agree with that. Misdirected outrage hereby turned off ;)
(Mind you, I still think they need to work more with the reasonable anti-spammers... and fix that unconfirmed sign-up that Rod mentioned, if it's really still unconfirmed!)
Trips: So I'm just back from ApacheCon 2004, which took place in the lovely Alexis Park building site ;)
Good fun was had -- very interesting to meet all the faces behind the names from various mailing lists and blogs, and get the inside track on how the ASF really works... there's quite a lot you don't get to understand from the outside, or even from being a committer. So, a useful trip.
Most of the talks were, naturally, very web-oriented -- we'll have to see what we can do about that, next time around! One useful tidbit: I didn't realise, but found out at the conference, that the ASF ConCom are very generous with paying speakers' expenses. So maybe next time I'll join the speaker line-up, too.
A major goal, one we achieved, was an impromptu SpamAssassin developer summit, 5 days sitting down together hammering on bugs and plans, with 4 of the main developers present (myself, Daniel, Theo and Michael). Pretty much achieved, although there were some thorny bugs to deal with... one interesting factor is that we may now be moving towards emulating the Apache httpd's preforking model to deal with a memory/performance issue we're seeing in 3.0.x.
Finally -- this sequence of photos has been cropping up all over the internets. When I saw it, I immediately thought it looked a lot like Ireland -- and Roundstone, Co. Galway, in particular. Sure enough, it appears it is! I guess the Connemara landscape of Roundstone's bay is pretty memorable, after all...
Patents: Patents4Innovation is a new site set up by several European multinational companies to lobby for the legalisation of software patenting in Europe.
Their FAQ is good for a laugh, including an answer that basically says that the Patent Office needs to do a better job, and another which states that opponents of software patents have been unable to demonstrate any 'convincing' evidence of swpats causing economic harm (despite the FTC and PriceWaterhouseCoopers reports, which are pretty high-profile organisations).
But the best quote -- in fact, the only quote -- appears on their Testimonials and Quotes page. Here is in full:
"... successful and profitable high-tech enterprises seem to be characterised by holding patents and copyrights. These instruments might reduce the vulnerability of firms with respect to competition and equip them with a more favourable market position."
Source: the "Observatory of European SMEs 2002 / No 6: High-tech SMEs in Europe", commissioned by the European Commission
(My emphasis.) Of course, there's nothing worse than having to compete on a level playing field. ;)
(Thanks to Christian Beauprez for spotting that!)
Web: The lovely C sent me a link of note -- it's the eglu, 'the world's most stylish and innovative chicken house and is the perfect way to keep chickens as pets'. (She has a thing about keeping chickens.)
So I was all set to link to that on NoMoreSocks.newscientist.com, New Scientist's nifty new xmas-pressies site; but -- get this: it will not load in Firefox 1.0PR, 1.0, or Konqueror at all -- in fact, using telnet, the site doesn't actually respond to requests on port 80 from my linux desktop.
The only browser it seems to work with is MS Internet Explorer in VMWare, presumably using MSIE's psychic powers to contact it without going through TCP/IP.
Mysteriously, it can be lynxed from my server in Ireland, but similarly doesn't work for C's Firefox installation on her desktop. How wierd!
Usability: So, I've just found out about a useless feature of my microwave oven the hard way. The microwave's manual notes:
Demo Mode
The Demo Mode is ideal for learning how to use the microwave oven. When set, functions can be entered without actually turning on the magnetron. The microwave oven light will come on, the fan will run and, if on, the turntable will rotate.
To Turn On/Off: The microwave oven and Timer must be off. Touch and hold TIMER SET.OFF for 5 seconds until 2 tones sound and 'd' appears on the display. Repeat to turn off and remove 'd' from the display.
Great! How useful! What a friendly UI! A tiny 'd' that indicates that, although your microwave looks like it's cooking away, it's actually doing nothing.
If you ever want to prank a friend with a Whirlpool oven, I'd recommend this one ;)
Politics: Global Guerrillas: 'Networked organizations, infrastructure disruption, and the emerging marketplace of violence. An open notebook on the epochal war of the 21st Century.'
This stuff is scary -- and I think it's right; the author thinks Bin Laden sees the US as an over-stretched 1980's Russia, vulnerable to economic collapse; his last communique talks about what he calls the 'bleed-until-bankruptcy plan'.
Bruce Sterling says 'You ever read my 80s novel ISLANDS IN THE NET? The one with a sinister, imaginary book of military strategy in it called THE LAWRENCE DOCTRINE AND POSTINDUSTRIAL INSURGENCY? Well, this blog feels a lot like I imagined that book would have felt. Kinda floods the mind with creepily subversive 21st century insight.'
Definitely worth reading.
Spam: 2 found guilty in first felony spam conviction: 'LEESBURG, Va. - A brother and sister who sent unsolicited junk e-mail to millions of America Online customers were convicted Wednesday in the nation's first felony prosecution of distributors of spam.'
Jeremy D. Jaynes, 30, (aka. Gaven Stubberfield) and Jessica DeGroot, 28, convicted to nine years in prison and a $7,500 fine respectively.
Nine years -- wow, that's a serious conviction for spamming... Virginia clearly takes this very seriously, as the home of AOL. Let's see if this causes any of the remaining spammers to think twice.
Politics: Disaster. I can't believe it.
- CNN sums it up
- some cute kittens, take your mind off it
Patents: I haven't been blogging much, as I've got the damn flu. I thought I was clear after a minor bout, then it came back around again for another run... urgh. Now I'm all hot, confused, achy, and (due to it affecting my ears and therefore sense of balance) clumsy. Damn you, influenza virus! :(
But there's some good news -- NoSoftwarePatents.com has opened, with a nice, clean, clear website full of excellent content. Not only does it cover the usual list of basics, with good examples like FFII's web-shop example, it goes into more detail about parallels; here's a great one:
Patents on software are just as wrong as expanding the patent system to literature. With patents on story elements, no movie could be published without having to firstly check whether there is any general idea in the storyline that someone patented during the last 20 years. Here's an example: At first sight, Dirty Dancing and Titanic are two very distinct movies. However, if there were patents on story elements, then the makers of Dirty Dancing could have sued the studio of Titanic. Both movies have a scene in which a poor boy takes a rich girl from a party of her social peers to a dancing party of his group, and she enjoys it. Dirty Dancing came out only nine years before Titanic, so any patent would still have been in force. No one knows whether James Cameron had that Dirty Dancing scene in mind as he wrote the Titanic script. Maybe Cameron never saw Dirty Dancing but the patent (if it existed) could be used against him anyway.
That's exactly the right parallel to make! Software patents cover the tens to hundreds of little algorithms you need to put together to make a software product, and comparing an algorithm in a software product to a scene concept in a movie, illustrates that nicely.
Each page has a little quote from heavy-hitters like Bill Gates, Oracle, Deutsche Bank, PriceWaterhouseCoopers, etc., coming out against swpats. There's also a section dealing with the disinformation that the other side has been putting out.
Next time you need to send a URL over to educate someone about this issue, this is the one to reach for.
Some news on the EU Software Patents Directive -- IDA eGovernment News reports that 'the formal approval of the draft Directive on the patentability of computer-implemented inventions by the EU Competitiveness Council has been postponed due to translation delays', and notes that 'on 6/10/2004, the Parliament's Internal Market Committee selected former French Prime Minister Michel Rocard to draw up its response.' It goes on:
Unsurprisingly, the appointment of Mr Rocard -- an outspoken adversary of software patents -- was welcomed by opponents of the proposed directive and criticised by its supporters. That Michel Rocard is taking over the dossier reflects the fact that the wider economic, infrastructural and social implications for Europe are now seen more clearly. Also, in the Council a learning process has begun, and it will be supported by the Parliament's move, Mr Holger Blasum of the Foundation for a Free Information Infrastructure told journalists. On the oher hand, Francisco Mingorance of the Business Software Alliance pointed out that Mr Rocard hasn't shown any sympathy to the directive in the past. We can only hope he will be sufficiently open to the view of persons and groups which have a different opinion, he said.
Well, that sounds like good news to me ;)
Security: The RFID vendors are clearly on a roll, with all manner of uses being proposed. The most recent story is that VeriChip plans to implant them subdermally in hospital patients.
The company line is that it's privacy-safe, since it doesn't expose health records per se -- just the patient's ID number. However, that's missing the point, in my opinion.
RFID chips will broadcast their ID whenever they are within range of a compatible scanner, and the range (in this case) is several feet -- although the story notes that their readers used to track farmed salmon work from 10-12 feet, and the Schmoo Group guys I met last month had no doubts that a high-powered directional antenna like their wi-fi sniper rifle could extend that. There's no encryption, or handshaking, in these chips, it sounds like.
There's no mention if the chip is removed after you leave hospital; some comments about the idea behind this is that it may help if you're involved in an accident, and want your info available to healthcare users, in which case you'd have the chip implanted and broadcasting at other times, in other places, as well.
So, if you've got one of these implanted, it'll broadcast a unique code to readers in range at all times. If an attacker can scan while you're nearby, and picks up that code, they know that it's you, and you only. They only have to match that ID code to a visual identification once, and henceforth you can be tracked by that ID code.
There's a possibility that they'll fix this, by upping the CPU power and incorporating some decent public-key encryption -- but then you need a PKI big enough to track every implanted citizen in the entire country, and the costs will go up and up. I'd find that doubtful. (Mind you, they seem to assume that having a centralized secure database of medical records is a fait accompli in most of the articles anyway, so...)
Web: The slashdot story. The comments contain a massive amount of noise, but there are some highlights...
Some details of the backend; it appears Indymedia need more mirrors, and the imc-tech list and #tech channel are the best contact locations to get in touch. The comment also notes that the Mir CMS used by most IMCs generates static HTML -- which is a good thing! I hereby withdraw my kvetching about server-side dynamic scripting in that case ;)
The techie who 'had the contract with Rackspace' comments, and provides a link to his weblog, which contains copies of the trouble tickets.
He also notes that the possible illegal posting was a newswire submission -- therefore not 'published' per se, just uploaded in the same way an unmoderated-up slashdot comment is.
And finally -- he notes that the EFF are offering to represent himself and Indymedia pro bono. Yay EFF!
The Electronic Frontier Foundation (EFF) is currently assisting Indymedia investigate possible responses to the seizure of its information. More than 20 Indymedia-related websites, along with Indymedia's online radio, were hosted on the servers, which were dedicated machines provided by Rackspace.
'This seizure has grave implications for free speech and privacy. The Constitution does not permit the government unilaterally to cut off the speech of an independent media outlet, especially without providing a reason or even allowing Indymedia the information necessary to contest the seizure,' said EFF Staff Attorney Kurt Opsahl.
This is great news. Top-secret takedowns are not a good thing, especially when they span three national borders...
Law: t r u t h o u t quotes this press release from Rackspace:
In the present matter regarding Indymedia, Rackspace Managed Hosting, a U.S. based company with offices in London, is acting in compliance with a court order pursuant to a Mutual Legal Assistance Treaty (MLAT), which establishes procedures for countries to assist each other in investigations such as international terrorism, kidnapping and money laundering. Rackspace responded to a Commissioner's subpoena, duly issued under Title 28, United States Code, Section 1782 in an investigation that did not arise in the United States. Rackspace is acting as a good corporate citizen and is cooperating with international law enforcement authorities. The court prohibits Rackspace from commenting further on this matter.
(my emphasis.) I wonder which of those 3 Indymedia is supposed to have been infringing? It's pretty clear how Rackspace feel about this situation, I think.
It seems MLATs have been used before to shut down Indymedia sites in the US; this cryptome mirror of Montreal IMC pages documents one such case. Here's a summary from a quoted email there:
Heres a quite interesting story on the power of mlats and what we will have to look forward to with the COE treaty :
A cop car was broken into in Quebec and a security doc relating to measures for the Free Trade Area of the Americas summit protests was stolen and posted in the net in Seattle. At the behest of the RCMP, a magistrate judge issued an order to grab the records from a Seattle web site called the 'independent media center' using the US/CAN mlat. They were then visited by the FBI/Secret Service. They then had a gag order on this for several days before it was released today.
Great precedent. I wonder if when my car gets broken into again, I can use the cybercrime treaty to find my stereo again...
And snippets from the IMC press release of the time:
On the evening of Saturday, April 21, a day which saw tens of thousands demonstrate against the FTAA in the streets of Quebec City, the Independent Media Center in Seattle was served with a sealed court order by two FBI agents and an agent of the US Secret Service. The terms of the sealed order prevented IMC volunteers from publicizing its contents; volunteers immediately began discussions with legal counsel to amend the order. This morning, April 27, Magistrate Judge Monica Benton issued an amended order, freeing us to discuss the situation without the threat of being held in contempt.
The original order, also issued by Judge Benton, directed the IMC to supply the FBI with 'all user connection logs' for April 20 and 21st from a web server occupying an IP address which the Secret Service believed belonged to the IMC. The order stated that this was part of an 'ongoing criminal investigation' into acts that could constitute violations of Canadian law, specifically theft and mischief. IMC legal counsel David Sobel, of the Electronic Privacy Information Center, comments: 'As the U.S. Supreme Court has recognized, the First Amendment protects the right to communicate anonymously with the press and for political purposes. An order compelling the disclosure of information identifying an indiscriminately large number of users of a website devoted to political discourse raises very serious constitutional issues. To provide the same protection to the press and anonymous sources in the Internet world as with more traditional media, the Government must be severely limited in its ability to demand their Internet identity--their 'Internet Protocol addresses.' A federal statute already requires that such efforts against the press be approved by the Attorney General, and only where essential and after alternatives have been exhausted. There is no suggestion that these standards were met here.
The sealed court order also directed the IMC not to disclose 'the existence of this Application or Order, or the existence of this investigation, unless or until ordered by this court.' Such a prior restraint on a media organization goes to the heart of the First Amendment. Ironically, the Seattle Post-Intelligencer learned about the existence of the order from 'federal sources,' suggesting that the purpose of the gag order was simply to allow the government to spin the issue its way.
The order did not specify what acts were being investigated, and the Secret Service agent acknowledged that the IMC itself was not suspected of criminal activity. No violation of US law was alleged.
Of course, cryptome is still chugging away as it always has been; simple HTML and no server-side dynamic scripting, means easy offshore mirroring ;)
Politics: Indymedia's hard drives in Rackspace UK seized by FBI order, seemingly as a 'courtesy' to Swiss police. There's several morals to be learned:
- Rackspace UK are happy to roll over for the US feds;
- it appears the action was taken using powers granted under the USA-Patriot Act;
- hosting in Europe is not safe from bad US laws.
However, the UK site is back on the air, and reportedly they're recovering nicely; 'All this goes to prove that Indymedia is decentralised enough (but not perfectly) to survive an attack and that as a cooperative international network, we rock!'
Health: A few hours after ( ;) I link-blogged this New Scientist article about a case of the H5N1 avian flu transmitting itself between humans, Boing Boing put up this entry titled 'Bird Flu risk extremely low', which concludes that the risk is effectively not worth worrying about.
It's fundamentally wrong, and is well worth pointing out as a result. As Quinn at ambiguous.org says, it's not the danger now that's important here -- it's the potential.
I read New Scientist religiously, so I've been following it, and this search on H5N1 gives the perfect illustration of why this is well worth worrying about:
- supplies of human flu vaccine in Thailand are very low, due to rich countries in the North taking up the excess; one route that flu strains pick up human-to-human transmission is through 'co-infection', where multiple flu strains infect the same person and swap genes.
- H5N1 has developed the ability to infect mammals already, namely cats, and therefore has another vector.
- the Chinese scientists studying H5N1 warn that 'it is still evolving', and the current 'Z genotype has so far had difficulty infecting humans, but is lethal when it does. If it becomes more adept at this, it could cause a severe human pandemic.'
- the virus may have evolved human-to-human transmission. This is the biggie; a flu that transmits bird-to-human is inherently much more controllable, but once it gains the ability to jump directly to other humans, the pandemic danger becomes much higher. If it gets better at this, we could be in big trouble.
(Now, while it's worth worrying about, it's not us end-users who should be doing the worrying. It's the politicians who need to ensure CDC and the WHO are funded well, the terrible state of vaccine development and production be sorted out, the lack of outbreak monitoring infrastructure be addressed, and research into these strains is funded and given a priority, in case things do go all pear-shaped influenza-wise.)
Patents: Sun files for patents on per-employee software pricing plans (/.). 'Method for licensing software to an entity, including determining a per-employee cost for the software, determining a number of employees of the entity, and determining a total licensing cost using the number of employees and the per-employee cost, wherein the total licensing cost comprises a software license for all employees of the entity and all customers of the entity.'
But, in my opinion, here's the good news -- this is a patent on a license agreement. In other words, this is a new front for patents -- the field of law.
Once the lawyers start running into situations where trivial concepts in their license agreements are patented, you can be sure the situation will start to turn around. ;)
Security: Given the current prevalence of phishing attacks and spyware infestations, designing a good user interface that protects naive users against malware is now more urgent than ever.
Firefox is, of course, widely touted as more secure than MSIE. This is by and large true, due partly to MS' emphasis in their UIs on one-step 'easy' installation and confirmation-dialog reduction (in my opinion) -- but also due to the fact that spyware companies don't yet see Firefox as a target to the same extent.
This changed recently -- spyware 'toolbars' started to appear for Firefox as well. It was quite a surprise to see a dialog pop up when accessing an otherwise normal-looking (though advertising-heavy) page, using my Linux desktop, prompting me to install some 'toolbar' .xpi file!
Firefox 1.0PR now includes code to deal with this. Here's how it works.
If a site I'm viewing attempts to install an XPI file, I get this prompt:
Note that it's NOT a dialog. This is pretty handy, because it means that I won't get annoying dialogs all the time if I do accidentally go to a unscrupulous site; it just appears like the part of the page. In the clueless user case, they may not even notice that they've been protected, which reduces the risk that they'll install the extension anyway.
(However, I would have extended it by using an icon or look-and-feel that indicated that this was a 'trustworthy' part of the UI, rather than possibly part of the page.)
If I hit the 'Edit Options...' button, I get this:
A simple-enough dialog containing the list of sites permitted to
install extensions. update.mozilla.org is in there by default,
and I've added texturizer.net so I can install from their more
extensive list of older extensions. The address of the current
site has been dropped in automatically.
To permit the site, I have to hit 'Allow', then 'OK'. So I do that, and hit the 'install' link on the webpage again:
And there's the Software Installation dialog. Note the red Unsigned warning, the proportion of text that is a warning about installing bad stuff (fully half!), and -- this is interesting -- a greyed-out 'Install' button.
The button is on a timer -- it becomes clickable after 2 seconds. This, presumably, is to ensure that people read the dialog! Reportedly, users no longer read dialogs, instead hitting OK on every dialog that appears. In my opinion, this is arguably due to 'the boy who cried wolf' syndrome: by default, MSIE and older Mozilla versions will ask all sorts of stupid questions about 'are you sure you want to send stuff on the intarweb?' whenever you use Google. If anything is guaranteed to induce dialog fatigue, it's that feature.
(Update: actually, that's not the reason. Reportedly, it's a workaround for a couple of social-engineering attacks, whereby an attacker could persuade the user to type a word ending in 'Y', and time the dialog to appear just before 'Y' is typed -- causing the keyboard shortcut for 'Yes' to take effect; or persuade the user to double-click in the right spot, and similarly time the dialog to appear in the right place, in time for the second click. Still, I maintain the measure is useful to deal with the 'dialog fatigue' issue too. ;) Thanks to Smyler and Rod for pointing this out.)
I would have gone further:
- the 'a software install was blocked' page element should have an indication that it's 'trustworthy content'
- both dialogs should default to 'Cancel', to avoid users deliberately pressing 'OK'
- I would possibly require a 'yes, I read this' tickbox to be ticked before the software is installed.
Interesting though. This is the way internet-facing UIs are going to have to develop, in my opinion.
Spam: my slides from the presentation I gave at Toorcon 2004, 'Spam Forensics: Reverse-Engineering Spammer Tactics', are now up. Hope they prove enlightening ;)
Travel: Toorcon was great fun! Lots of interesting conversations.
Unfortunately they had a cruddy internet connection, so I'm majorly backlogged, and can't write about any of it just yet ;)
Spam: SpamAssassin 3.0.0 is now released! w00t! Only 4 months late this time ;) Announcement, techie details, Slashdot. New logo too:
(Note: if you're running SpamAssassin 2.x and plan to upgrade, this is a new major release cycle -- so we've taken the chance to break some backwards compatibility. Be sure to read the UPGRADE doc!)
Politics: So I was listening to that this morning. Did I hear correctly? Did Bush really say that one of the good side-effects of Iraq's invasion, was that there were now hopefully less attacks inside other countries? sure looks like it:
'Coalition forces now serving in Iraq are confronting the terrorists and foreign fighters so peaceful nations around the world will never have to face them within our own borders.'
I'm sure the Iraqi civilians will love that. 'Hey guys, sorry about all the missing limbs, but you're doing a really good job of being flypaper so we don't get hurt. Cheers! Have a 15% corporate tax rate!'
Conferences: Hey -- I'm talking at ToorCon 2004 down in San Diego this weekend! Come along and check it out, if you can.
I'd better hurry up and file my presentation slides pronto ;) The topic is:
Spam Forensics: Reverse-Engineering Spammer Tactics
In this talk, I'll discuss how the SpamAssassin project has identified reliable signatures indicating that a message is spam, by reverse-engineering spammer tactics from the spam mails themselves. I'll also discuss several specific features that we have identified, how we found them, and why the spammers add them.
Patents: Now that the summer break is over, software patents are back on the EU's agenda. The FFII (via EDRI-gram) reports
On 24 September 2004, the European Council will probably meet to rubber-stamp the 'political agreement' achieved on 18 May 2004 on the highly controversial software patents directive (2002/0047 COM-COD).
According to the FFII the text was designed to mislead ministers about its real effects. 'It consists of many sentences of the form or 'software is ... unpatentable, unless ... [condition, which, upon closer scrutiny, turns to be always true]'.' And, states FFII, 'It can be said with certainty that only a minority of governments really agrees with what was negotiated, but several governments were misrepresented by their negotiators, who broke intra-ministerial agreements or even violated instructions from their superiors.'
More info:
Web: Flickr's latest trend -- using just an eye (or similar minimalist face part) as your avatar pic:
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Life: Luke writes:
Lean and I were joined by Lara at 6.10pm on Saturday 28th September. Lara is a little (8lb 7oz, so not _that_ little) girl. And she is gorgeous. Of course.
Congrats! I'll be dropping in on the three of them next week, looking forward to it...
Funny: The Daily Show's GWB reelection film: 'George W. Bush -- Because He Says So' (Quicktime MOV, 6MB). This is the funniest thing I've seen in ages.
Remember -- don't listen to the facts -- listen to the words!
(thanks to anaxamander for the file. This URL is cached through CoralCDN, so pass it on!)
Web: My Nearly-Live Planetary Desktop Backgrounds site is now using NYU's Coral Content Distribution Network instead of FreeCache.org. (FreeCache wasn't caching the files, because they were too small. drat.)
Coral is a 'decentralized, self-organizing, peer-to-peer web-content distribution network', using a distributed sloppy hash table and peer-to-peer DNS redirection infrastructure.
At least, apparently. ;) I haven't read the papers yet, but what I do know is that so far, it seems to be working perfectly -- each file is requested exactly once by the CDN servers:
193.10.133.129 - - [31/Aug/2004:16:50:31 +0100] "GET /xplanet/tmp/200408311455.399750/day_clouds_800x600.png HTTP/1.1" 200 706936 "-" "CoralWebPrx/0.1 (See http://www.scs.cs.nyu.edu/coral/)"
and never requested again. That's a big saving... nifty!
Linux: Everyone who's used a non-MS system will have learned -- typically the hard way -- that not all hardware is equal. Not just in terms of specs, flexibility and power, but also in terms of whether or not it can be used at all.
Most hardware vendors consider their specification and interface documentation to be their crown jewels; giving access to these without a signed NDA is impossible. On the other hand, for free software developers, signing an NDA makes life quite difficult -- it can be done, but nobody else can help you maintain it further without signing an NDA, the resulting code may 'disclose' too much of the 'IP', and so on. In a lot of cases, the vendor isn't interested in giving access to the specs, even with an NDA -- it's their IP and why isn't the customer just using Windows?
The end result: lots of hardware with crappy support on non-MS operating systems.
Things aren't as bad as they used to be, though -- since nowadays the high-end hardware is more likely to support standards, and Linux is a top choice on embedded hardware (set-top boxes for example), so it has a much higher profile. But cheap, end-user oriented PCs still wind up with components from vendors who couldn't be bothered with non-Windows customers, and that can mean using a hacked-up, reverse-engineered driver and hoping it works. (That's not to denigrate reverse-engineered drivers. some of them work great. But fundamentally, the vendors are making a mistake here.)
So it's pretty impressive to see that LaCie are now sponsoring development of k3b, the CD/DVD burning application for KDE!
Good timing too, I was about to buy a DVD burner ;)
Linux: Here's a patch that adds support for ALSA/Artsd/ESD output from kgst, the KDE gstreamer middleware used by JuK.
Background: JuK is a great music player app for KDE. However, it hogs
the sound device while running, which means that nothing else gets access
to play sound until the app is shut down. This is suboptimal.
The reason it does this is because it plays sound via this chain of
components: juk -> kgst -> gstreamer -> sink. Unfortunately, the
kgst component doesn't allow control over what output sink to use,
instead hard-coding the string 'osssink' -- the OSS drivers, for
traditional Linux /dev/dsp sound. My laptop doesn't support mixing in
the sound hardware, which means I need to use a software mixer.
'osssink' doesn't support software mixing, instead giving the caller
exclusive access to the sound card, and other apps will just have to wait
for it to finish.
(As to why JuK doesn't just play mp3s by running 'mpg321 name-of-file.mp3', and let us specify the '-o' switch to use, I wish I knew. (ObOldbieGripe: component-based architectures are full of this kind of needless over-complexity ;)
Anyway, the patch in the bug above lets the user provide an environment
variable for a string for kgst to use instead of 'osssink'.
Tech: This is the second entry talking about 'Life Hacks'. Possibly the best tip I came away from the talk with, is this one:
All geeks have a todo.txt file. They use texteditors (Word, BBEdit, Emacs, Notepad) not Outlook or whathaveyou.
What we keep in our todo is the stuff we want to forget. Geeks say they remember details well, but they forget their spouses' birthdays and the dry-cleaning. Because it's not interesting.
It's the 10-second rule: if you can't file something in 10 seconds, you won't do it. Todo.txt involves cut-and-paste, the simplest interface we can imagine.
It's also the simplest way to find intercomation. EMACS, Moz and Panther have incremental search: when you type a "t" it goes to the first mention of "t", add "to" and you jump to the first instance of "to", etc.
Power-users don't trust complicated apps. Every time power-geeks has had a crash, s/he moves away from it. You can't trust software unless you've written it -- and then you're just more forgiving. Text files are portable (except for CRLF issues) between mac and win and *nix. Geeks will try the Brain, etc, but they want to stay in text.
I was already doing this, having learned the latter lesson ;), but I was making one mistake -- I was trying to keep the TODO.txt file small by clearing out old stuff, done stuff, and cut-and-paste snippets of command lines, and by moving things into files in 'storage' directories.
That doesn't work. You think you'll be able to grep for it later, but you'll have forgotten what to grep for. You'll even have forgotten what storage directory you used. The solution is to keep it all in one big file, and use i-search. That really does work.
In fairness, I actually have two files of this type. One is the "real" TODO.txt. But the other is a GPG-encrypted file containing usernames, URLs, passwords, nameservers, VPN settings, etc. I have a feeling this is another common Life Hack idiom, too...
Another great tip in the same vein, from JWZ -- make an /etc/LOG:
Every machine I admin has a file called /etc/LOG where I keep a script of every system-level change I make (installing software, etc.) I rsync these LOG files around (keeping redundant copies of all of them in several places) so that if/when I need to re-build a server from scratch, it's just a matter of following the script.
This has been working out great (when I remember to do it. Discipline! ;)
Tech: So Danny O'Brien's 'Life Hacks' talk is one of the most worthwhile reflections on productivity (and productivity technology) I've heard. (Cory Doctorow's transcript from NotCon 2004, video from ETCon.)
There's a couple of things I wanted to write about it, so I'll do them in separate blog entries.
(First off, I'd love to see Ward Cunningham's 'cluster files by time' hack, it sounds very useful. But that's not what I wanted to write about ;)
People don't extract stuff from big complex apps using OLE and so on; it's brittle, and undocumented. Instead they write little command-line scriptlets. Sometimes they do little bits of 'open this URL in a new window' OLE-type stuff to use in a pipeline, but that's about it. And fundamentally, they pipe.
This ties into the post that reminded me to write about it -- Diego Doval's atomflow, which is essentially a small set of command-line apps for Atom storage. Diego notes:
Now, here's what's interesting. I have of course been using pipes for years. And yet the power and simplicity of this approach had simply not occurred to me at all. I have been so focused on end-user products for so long that my thoughts naturally move to complex uber-systems that do everything in an integrated way. But that is overkill in this case.
Exactly! He's not the only one to get that recently -- MS and Google are two very high-profile organisations that have picked up the insight; it's the Egypt way.
There's fundamentally a breakage point where shrink-wrapped GUI apps cannot do everything you want done, and you have to start developing code yourself -- and the best APIs for that, after 30 years, has been the command-line and pipe metaphor.
(Also, complex uber-apps are what people think is needed -- however, that's just a UI scheme that's prevailing at the moment. Bear in mind that anyone using the web today uses a command line every day. A command line will not necessarily confuse users.)
Tying back into the Life Hacks stuff -- one thing that hasn't yet been
done properly as a command-line-and-pipe tool, though, is web-scraping.
Right now, if you scrape, you've got to do either (a) lots of munging in a
single big fat script of your own devising, if you're lucky using
something like WWW::Mechanize (which
is excellent!); (b) use a scraping app like sitescooper; or (c) get hacky with a
shell script that runs wget and greps bits of output out in a really
brittle way.
I've been considering a 'next-generation sitescooper' a little bit occasionally over the past year, and I think the best way to do it is to split its functionality up into individual scripts/perl modules:
- one to download files, maintaining a cache, taking likely freshness into account, and dealing with crappy HTTP/HTTPS wierdness like cookies, logins and redirects;
- one to diff HTML;
- one to lobotomise (ie. simplify) HTML;
- one to scrape out the 'good bits' using sitescooper-style regions
Tie those into HTML Tidy and XMLStarlet, and you have an excellent command-line scraping framework.
Still haven't got any time to do all that though. :(
Spam: I'm quoted in
New
Scientist! w00t!
SlashDot picked it up pretty quickly. One comment there misses the point, though:
This is interesting and promising technology. But like all antispam techniques, spammers will find a way around it. Once spammers get a copy of the software, they can create and test countermeasures in the comfort of their own sleazy lairs.
It's worth talking about this. Newsflash: spammers have no difficulty testing their spam against closed-source spam filters, even when they can't 'get a copy' and test them in 'their sleazy lairs'.
How do they do it? Easy -- just set up an account at a site that uses that filter (AOL, Yahoo!, Hotmail, and GMail, it's pretty obvious how to do that; for other closed-source filters, find an ISP that uses it). Then send 'test mails' repeatedly to that account, and apply trial and error to see what gets past the filter and what doesn't. Eventually, they figure out what works for that filter, and what doesn't.
How did I figure this out? Well, I came across the manual for the Send-Safe ratware on-line. It noted that the 'hashbuster' randomisation technique, which we in the SpamAssassin team had long assumed was intended to block hash matches by DCC, Pyzor and Razor, was in fact intended to block AOL's implementation of that system. The open source ones weren't even mentioned.
Update: found it -- from their FAQ:
Mime Encoded content
If you want to get into AOL... use it.
MIME encoders allow you to send documents written within a specific application through email without causing readability or formatting problems. For example, you can send a letter created in MSWord with and be certain that it arrives at its destination in the same format by encoding it with MIME first. The recipient then decodes it back into the original MSWord format.
That isn't why we use it though.
We use it to cause 'uniqueness'.
When you put a rotate tag at the beginning of a MIME encoded email, it causes everything after that point (including checksums) to be 'different' in every message.
Why is that that important?
Because it throws off filters that look for many copies of the same message to nuke.
TV: A choice quote from NBC's Olympics coverage: 'This girl (one of the US beach volleyball team) reads a book a week!' (delivered in shocked tones.)
Web: 
My Dad runs a couple of
websites -- his architectural
photography business, and Andalucia Photo Gallery, a
side project selling
some lovely photos from the Andalusia region of Spain.
Needless to say, as the family geek, guess who coded all that up? Using WebMake, naturally ;) This was the main reason I wrote the 'thumbnail_tag' plugin.
You'll note, however, that the image to right is watermarked, quite small, and encoded with a low quality setting. It turned out after a couple of years of operation, that the images were being downloaded and used in print all over the place -- from both sites!
It seems photo piracy is rampant. Even with terms of use clearly linked on the sites, it's still commonplace for print publications to swipe the images -- and not just the little guys, either -- some big commercial names have apparently used the images without asking (or paying licensing fees).
The Andalucia gallery site was a favourite; being a good hit for 'travel photos spain' meant lots of images being used for holiday pages in magazines, newspapers, and so on.
Needless to say, digital watermarking software doesn't work -- it's trivial to load an image into Photoshop, resize or crop, and resave, apparently. Even if PS did respect the watermarks, netpbm doesn't, and a watermarked image isn't identifiable as such once it appears in print anyway! So we went for the blunt-tool approach, adding visible watermarks to the images.
It's pretty easy -- pamcomp allows you to overlay one image on top of another, using a third as an 'alpha mask' to control transparency. The results are pretty nice and not too intrusive.
It's a shame it has to be done, though... :(
RFID Scan Detector
RFID: Over on Adam Shostack's weblog, in a comment on an entry regarding the plans to mandate remotely-readable RFID passports, Martin Forssen brings up a great idea:
I wonder if that'd work? A keyfob, for example, something similar in size to the dinky Chrysalis Wifi Seeker I have on my keyring, would be perfect. It'd be probably pretty cheap to make, would make a great geek toy, and be quite educational too. ;)