(Things I found interesting recently.)
Back in 2006, I wrote a script I called "goog-love.pl"; it used Google’s now-dead SOAP search API (thanks, Nelson!) to figure out which Google queries your web site was "winning" on. Unfortunately, Google shut down new signups for the SOAP interface later that year.
I was just looking through Google’s Webmaster Tools page for taint.org, when I came across the Statistics / Top search queries page:
This is exactly what goog-love.pl produced. hooray!
Check out this logic from the Bank of Ireland, spotted by waider in today’s news:
Last week, the bank said that medical records, bank account details, names, addresses and dates of birth of 10,000 customers were on the laptops. […]
Bank of Ireland said an assessment had concluded that the risk of fraud arising from the thefts was ‘very low’, as the data on the laptops did not include bank account passwords, PINs or copies of signatures.
So a fraudster would have medical records, bank account details, names, addresses and dates of birth of 10,000 customers, but the risk of fraud is ‘very low’? Incredible.
Update: make that 30,000 customers.
Update 2: 31,500 customers, and a sample letter.
Peter G. Neumann at the RISKS Forum notes that Last Friday was the anniversary of the sending of the first e-mail spam:
[Thanks to Mike Hogsett for noting this event, and Brad Templeton for recording it.]
What is allegedly the very first spam message was sent roughly 30 years over the ARPANET.
In seeing this, Mike was amused because he works with some of the people it was addressed to, of whom a few are still at SRI:
NEUMANN@SRI-KA, GARVEY@SRI-KL, MABREY@SRI-KL, WALDINGER@SRI-KLand some of whom are retired:ENGELBART@SRI-KL, NIELSON@SRI-KL, GOLDBERG@SRI-KL(I am always amused when some of these old ARPANET addresses show up in today’s incarnations of spam.)Also somewhat before Mike’s time, Geoff Goodfellow, Eric Kunzelman, Dan Lynch, and many others at SRI were instrumental in the evolution of the ARPANET.
Also included in the enormous enumerated TO: list (historically interesting in itself by not having been suppressed!) are Bill English (who was the catalyst for much of Doug Engelbart’s innovations being transitioned from SRI to PARC), Dave Farber, Irv Jacobs, Bob Metcalfe, Jon Postel (who by then had moved from SRI to ISI), three Sutherlands, and Lauren Weinstein, to name just a few.
Happy Birthday, Spam! Sorry I cannot wish you many happy returns.
Bank of Ireland, one of Ireland’s biggest high-street banks, was the subject of a breach notification yesterday — 4 laptops, containing unencrypted "sensitive personal information" about up to 10,000 customers, were stolen between June and October 2007. It seems the Irish Data Protection Commissioner was not informed until last Friday. The Financial Regulator is also looking into the incidents.
According to the Independent, the laptops ‘were being used by staff working for Bank of Ireland’s life assurance division. They contained the information about medical history, life assurance details, bank account details, names and addresses.’
This breach has raised quite a few issues.
First off, I was watching Questions and Answers last night, and was shocked by the naivete of the assembled panel. One panelist, for example, reckoned that common criminals wouldn’t understand the value of this data — so it was probably nothing to worry about!
There was absolutely no concept of how widespread identity theft has become — using stolen identity information to apply for credit cards is part of Petty Theft 101 these days, since filling out forms is a lot easier than breaking and entering, obviously. There was also no appreciation of how little protection Irish consumers have in this regard with current Irish banking T&Cs.
According to previous research, about 2% of accounts compromised in data breaches become victim to identity theft.
Some comments from the bank from those articles:
‘The data was not encrypted, although it is understood there was software security installed on the stolen computers.’
Doubtless, "software security" refers to some kind of useless Maginot Line boondoggle like Norton Internet Security. This would have absolutely no useful effect in this case. The only useful way to protect customer data on a stolen laptop is to use encrypted storage.
‘In the interim the bank has monitored all of these customer accounts and can confirm that there has been no evidence of fraudulent or suspicious activity.’
This is a fallacy. This data provides plenty of information regarding the customer’s identity — information which is useful to receive loans and credit fraudulently, elsewhere. Monitoring the bank’s accounts is of no help in that case. On top of that, identity information like your date of birth, mother’s maiden name, health status, and so on doesn’t expire — that info will still be useful for identity theft, 10 years from now, or as a stepping-stone to further fraud.
As John O’Shea noted on Twitter earlier, there was nothing on their website about it this morning; there is now, however — a broken link on the front page. oops!
Figuring out the puzzle and fixing the URL’s errors gets you to this page, which notes:
The laptops contained information relating to some customers who either obtained a quote or took out a Life Assurance policy with Bank of Ireland Life from the following branches:
- Drogheda
- Dunleer
- Bagnelstown
- Court Place Carlow
- Stephens Green
- Tallaght
- Montrose
Anybody who is not a customer of these branches is not affected by this incident.
As far as I can make out, the bank didn’t issue this breach notification. It appears from the coverage that this information was first announced by Data Protection Commissioner Billy Hawkes to RTE yesterday, leaving the bank apparently scrambling to catch up:
"The thefts of the laptops were only brought to the attention of the appropriate authorities in the bank in the past number of weeks," Bank of Ireland said in a statement that offered no other explanation for the long delay.
It would have been so much better if BoI had been proactive with breach notification — examples from overseas have illustrated its value. As Adam Shostack has noted repeatedly over the past few years: the rules have changed.
As for repercussions for BoI, it’ll be interesting to see if anything happens. For "live" customer data on up to 10,000 customers to be stored, in unencrypted form, on a laptop is terrible security practice — but as far as I know, there are no laws or regulations requiring anything better in Ireland, unfortunately. :( However:
Consideration will be given as to what further action will be sought from Bank of Ireland to ensure that the obligations contained in the Data Protection Acts in this area are met.
On a broader level, this issue serves to highlight once again the absolute necessity for all organisations in the public and private sector to take their data protection responsibilities seriously. In particular, all organisations should be assessing immediately the necessity for storing personal data on laptops. If a need is found, appropriate security measures such as encryption should be put in place immediately.
Go Billy! ;)
Caerphilly is a small commuter town in South Wales, notable mainly for Caerphilly cheese and a castle.
Well, you can add one more thing to that list; its inhabitants also provided some key data in a major health study, from which emerged one great finding — it turns out that if you’re male, sex twice a week reduces the risk of death from heart disease by about half:
Men who said they had sex twice a week had a risk of dying half that of the less passionate participants who said they had sex once a month, Dr. Davey-Smith’s team said.
No other risk factor showed a statistically significant link to the frequency of orgasm.
The authors said that they had tried to adjust the study’s design to account for a factor that might explain the findings — that healthier, fitter men with more healthy life styles engaged in more sex. Even so, they could not explain the differences in risk. Hormonal effects on the body resulting from frequent sex could be among other possible explanations for the findings, Dr. Davey-Smith said.
Here’s the science bit, via the BMJ — a paper entitled ‘Sex and death: are they related? Findings from the Caerphilly cohort study’:
Result: Mortality risk was 50% lower in the group with high orgasmic frequency than in the group with low orgasmic frequency, with evidence of a dose-response relation across the groups. Age adjusted odds ratio for all cause mortality was 2.0 for the group with low frequency of orgasm (95% confidence interval 1.1 to 3.5, test for trend P=0.02). With adjustment for risk factors this became 1.9 (1.0 to 3.4, test for trend P=0.04). Death from coronary heart disease and from other causes showed similar associations with frequency of orgasm, although the gradient was most marked for deaths from coronary heart disease. Analysed in terms of actual frequency of orgasm, the odds ratio for total mortality associated with an increase in 100 orgasms per year was 0.64 (0.44 to 0.95).
Conclusion: Sexual activity seems to have a protective effect on men’s health.
The perfect excuse ;) Thanks, Caerphilly!
Last weekend, I picked up a super-cheap cycling computer in Aldi for 20 Euros. I cycle to work, and I thought it’d be fun to get some geeky number-crunching in on my daily commute.
Here are the figures for my trip into work:
Given that there are 46 kilocalories in a Jaffa Cake, 136 KCal means that every day, I can eat 3 Jaffa Cakes with impunity. Result! ;)
Google Calendar has a nifty feature, "Quick Add", where you can enter a natural-language string like "lunch with Justin, 1pm 20/4/08", it parses it, and adds an appointment to your calendar. However, the link in the Calendar UI can’t be bookmarked; you have to go to the Calendar page, wait for it to sloooowly load all its AJAX bits, hit the link, and only then type the appointment details, by which time I’ve forgotten it anyway ADD-style. ;)
Elias Torrez came up with a Firefox extension to use the Quick Add feature in one keypress, but in my opinion that’s overkill — I don’t want the overhead of another extension, the upgrade worries, and I don’t want it using up a keyboard shortcut either. I’d prefer to just have this as a Firefox Smart Keyword — and thankfully the trick is in the comments for his blog post, from someone called Bjorn. So here’s the deal:
Name: Google Calendar Quick Add
Location:
http://www.google.com/calendar/event?ctext=+%s+&action=TEMPLATE& pprop=HowCreated%3AQUICKADD Keyword: newcal
Description: add a new event in Google Calendar
enjoy!
So Mulley mentions that Moviestar.ie are planning to offer downloadable movies. Great concept, but I can guarantee the execution will be crap on a stick. :(
First off, the content available:
‘When the service goes live on 1 May, customers will be able to avail of content from several Irish producers including Network Ireland Television, as well as Video International’s film library which includes films like The Little Shop of Horrors. The company is also seeking content from both the History and Biography Channels, which would mean a substantial back catalogue of documentary shows.’
Sorry, but: snore.
Secondly, the technology used:
‘Moviestar.ie content must be downloaded onto a PC or laptop but can then be transferred over to digital media players like the iPod Touch for viewing on the go. This service will be compatible with Apple Macs but only if the user downloads Windows Media Player.’
So in other words, it’s Windows Media. That means it won’t play on my TV through my MythTV box, on a USB stick plugged into a Philips DVD player, on my Linux laptop, or even on a normal DVD player using a burned DVD.
Too little, too late. Plenty of Irish consumers are already consuming downloaded video — as the popularity of the Philips DVP5960 demonstrates. For legal video downloads to work, they need to be somewhere remotely near as convenient and usable as BitTorrent.
Using DRM is just falling down the same rabbit hole that swallowed up downloadable music for 5 years. Nobody used that either, until gradually the companies involved realised that opening up was the only way to get customers, bringing us to where we are today — legal downloads using the MP3 format.
BTW, I know that’s the same DRM technology used by Channel 4’s "4oD" download service. Big deal — I don’t bother trying to watch that stuff either, for the same reasons. If Channel 4 jumped off a cliff, would Moviestar.ie jump after them?
(By the way, that Philips DVD player is a total success story. That’s a name-brand hardware manufacturer, making a low-end, $60 DVD player, with support for viewing downloaded XviD AVI movies on a USB stick. Apparently it’ll also play off USB hard disks, too. It’s immensely popular; for example, here’s a customer review of 10/10: "Best thing ever". Several of my friends have them, and praise them highly. I’m coming up to DVD player replacement time, and I’m planning to get one too.)
Recently, more and more people have been complaining about backscatter; its levels seem to have increased over the past few weeks.
If you’re unfamiliar with the terminology — backscatter is mail you didn’t ask to receive, generated by legitimate, non-spam-sending systems in response to spam. Here are some examples, courtesy of Al Iverson:
It used to be OK to send some of these types of mail — but no longer. Nowadays, due to the rise in backscatter caused by spammer/malware abuse, it is no longer considered good practice to "accept then bounce" mail from an SMTP session, or in any other way respond by mail to an unauthorized address of the mail’s senders.
Backscatter as spam delivery mechanism
I would hazard a guess that this rise is due to one of the major spam-sending botnets adopting the use of "real" sender addresses rather than randomly-generated fake ones, probably in order to evade broken-by-design Sender-Address Verification filters.
There’s an alternate theory that spammers use backscatter as a means of spam delivery — intending for the mails to bounce, in effect using the bounce as the spam delivery mechanism. Symantec’s most recent "State of Spam" report in particular highlights this.
I don’t buy it, however. Compare their own example message — here’s what the mail originally sent by the spammer to the bouncer, rendered:
And here’s what it looks like once it passes through the bouncer’s mail system:
That’s simply unreadable. There’s absolutely no way for a targeted end user to read the "payload" there…
I haven’t run into this recent spike in backscatter at all, myself, since I have a working setup that deals with it. This blog post describes it. If you’re using Postfix and SpamAssassin, it would be well worth taking a look; if you’re just using SpamAssassin and not Postfix, you should still try using the Virus Bounce Ruleset to rid yourself of various forms of unwanted bounce message.
Note that you need to set the ‘whitelist_bounce_relays’ setting to use the ruleset, otherwise its rules will not fire.
There’s a theory that setting SPF records (or other sender-auth mechanisms like DomainKeys or DKIM) on your domains, will reduce the amount of backscatter sent to your domains. Again, I doubt it.
Backscatter is being sent by old, legacy mail systems. These systems aren’t configured to take SPF into account either. When they’re eventually updated, it’s likely they’ll be fixed to simply not send "accept then bounce" responses after the SMTP transaction has completed. It’s unlikely that a system will be fixed to take SPF into account, but not fixed to stop sending backscatter noise.
It’s good advice to use these records anyway, but don’t do it because you want to stop backscatter.
You might be worried that the SpamAssassin VBounce ruleset will block bounces sent in response to your own mail. As long as the error conditions are flagged during the SMTP transaction (as they should be nowadays), and you’ve specified your own mailserver(s) in ‘whitelist_bounce_relays’, you’re fine.
Steven Murdoch at Light Blue Touchpaper notes that the UK banking code now includes wording to make the customer liable for losses attributable to them "acting without reasonable care", where "reasonable care" bizarrely includes installing anti-virus software on their PCs.
The Register also picked up on this, as did Brian Krebs in the Washington Post, comparing it with the vastly superior customer protection offered by the US banks.
I was curious, so I went looking at the Irish situation. Needless to say, it’s not pretty.
I couldn’t find anything in the Irish Banking Federation’s Code Of Practice for Personal Customers, unfortunately. However, AIB’s terms and conditions for use of their Internet Banking product contain this:
5 Transactions on the Account:
5.1 The User authorises AIB to act upon any instruction to debit an Account received through AIB Phone & Internet Banking which has been transmitted using all or part of the Registration Number, PAC and/or any other authentication process which AIB may require to be used in connection with AIB Phone & Internet Banking (including but not limited to a Code Card) without requiring AIB to make any further authentication or enquiry, and all such debits shall constitute a liability of the User. Where the User’s Account is maintained in joint names the liability of the Account Holders shall be joint and several.
5.6 Entries in an Account in respect of Bill Payments, Fund Transfers and Top-Ups shall be prima facie evidence that the transfer or debit represented thereby has been duly authorised and shall be binding on AIB and the User unless and until proved to the contrary.
6 International Payments:
6.9 To the extent permitted by law, and notwithstanding anything to the contrary herein, AIB shall not be liable for, and shall be indemnified in full by the User against, any loss, damage or other liability that the User or AIB may suffer arising out of or in connection with the User’s use of the International Payment services (whether as the sender or receiver of an International Payment) unless such loss, damage or liability is caused by AIB’s fraud, wilful default or negligence. In no circumstances will AIB be liable for any increased costs or expenses, or for any loss of profit, business, contracts, revenues or anticipated savings or for any special, indirect or consequential damage of any nature whatever.
As far as I can tell, basically the AIB have no liability here at all — if a bad guy gets hold of your PIN code and account number, and empties your account, tough luck.
What about Bank of Ireland? It seems they agreed to refund phishing losses in an incident back in 2006. But their 365online Terms and Conditions now say this:
13 Indemnity
13.2 Without prejudice to the generality of Clause 13.1 above, the Bank shall have no liability whatsoever in respect of any loss suffered by the Customer as a result of their breach of Clause 4 [jm: Security/Authentication] by way of knowingly, negligently or recklessly disclosing the Security Devices or any of them.
So it’s all pretty bad news for Irish banking customers. This is pretty bad news — it’s only a matter of time before Irish banks are targeted by a new Banking Trojan, and given that antivirus software has an 80% miss rate these days, even having an up-to-date AV scanner isn’t going to be much help.
My answer? Don’t do internet banking on Windows machines. Simple as that.
The Irish Internet Association have a weblog at blog.iia.ie. Back on January 30, this had a Technorati rank of 587893, with 21 inbound links from 14 blogs. That’s about what you’d expect — comparable with Chris Horn’s blog, for instance.
However, fast forward to today, and in the intervening 3 months, it seems to have suddenly shot up to 23,322 inbound links from 550 blogs, giving it a Technorati rank of 6,870.
To put that in perspective, that puts it comfortably in the top 3 in the Irish Blogs Technorati Top 100 — beating Damien Mulley‘s 7,859, but just short of Donncha O’Caoimh‘s stellar 3,434 — and ahead of these other gods of the Irish blogosphere:
Pretty impressive ;)
I was curious, so I went investigating. Of those thousands of inbound links, here’s some samples of the most recent, pasted from the Technorati inbound links page:
Atacand Free instant online credit report Application credit card Cheap Paxil Does your credit score Household bank credit card application Apr for credit cards Buy Cephalexin? Aciphex Cheap Feldene Zovirax Risperdal Buy Naprosyn, Propecia Credit score codes Poor credit score, Propecia Uk Canada credit card online application Motrin Business credit score Cheap Cialis Jelly 50 Cent Free Ringtones Celexa How to improve my credit score Buy Inderal
4 days ago in barkingmoose by barkingmoose · Authority: 3
Jc penny credit card application Credit cards 1.99 apr ny Affect credit score For credit score American express credit card application Freee credit report Instant fleet 0 apr credit card application? Hydrocodone For low credit scores, No credit instant approval credit cards Annual creditreport.com, Tramadol Credit reporting service Configuration VPN Cheap credit reports. Buy Premarin Carisoprodol Soma Propecia Generic
6 days ago in The Peninsula’s Edge by ricsmith510 · Authority: 9
Prepaid credit card uk Phentrimine Cheap Zovirax: Calan Highest credit score Ambien Valtrex: Ultram 3 credit reporting agencies Credit cards online application Instant approval student credit cards, Apr balance transfer credit cards Free government credit report Transunion free credit report Credit card debt bankruptcy? Propecia Propecia Uk! Correcting credit reports Cialis Uk Credit rating report Buy Synthroid Instant capital one 0 interest credit card application
7 days ago in The Incredible Blog · Authority: 1
Annualcreditreport Instantly instant free online credit report Credit cards instant approval Guaranteed instant approval credit cards Lexapro Get my credit score, Card consolidation credit debt financial internet Chevron credit card services. Risperdal Lower credit card debt VPN connection One credit card application Xanax Viagra! Vasotec Diazepam Fix my credit report Credit report bureau. Cialis Soft Tabs! Ativan? Secured loans to increase credit score Cheap Amaryl Cheap Prednisone Alprazolam! Cheap 7 days ago in Quilters’ Blogs · Authority: 5
Luvox Credit score of Plavix 50 Cent Free Ringtones, Cheap Elavil? Free consumer credit report: Famvir Improve credit score fast Phentermine Online Zovirax Cialis Soft Tabs Apr for credit cards! Ultram Zoloft Credit card deal 0 Deltasone! VPN: Cheap Cardura Credit score rankings! Annual credit report .com Interest rate credit score: Carisoprodol Flagyl ER Online Cialis Soft Tabs Enable VPN 0 apr credit card application Free business credit report Ambien Low 7 days ago in TPN :: Martial Arts Explorer · Authority: 56
Take a look at the ‘inbound links’ list — thousands more just like that.
All of the affected blogs have been hacked to deliver these spam links. They run unpatched versions of WordPress vulnerable to a major security hole. On a casual visit, their pages seem fine — but "View Source", scroll to the bottom, and there are thousands of spam links for drugs, ringtones, cheap credit, etc. on each one, exactly as above, and as described by Kevin Burton in his description of the current epidemic of blog spam.
How did links to the IIA’s blog wind up in this collection?
It’s worth noting that the IIA’s blog does not display the same symptoms — the links aren’t present on their pages.
However, this post provided a good tip as to what has happened. Those infected blog pages point, in turn, to other infected blogs. Somewhere within the IIA’s blog setup, there’s a page inserted by a bad guy, collecting thousands of illicit links from thousands of other infected sites — and sure enough, Irish Web Watcher found it on the IIA’s site — here it is.
Looks like the IIA have a pretty major disinfection job on their hands, and urgently — there’s already a lot of spammy results appearing in the Google index from that site, and the next step after that is usually removal from the index once Google notice it.
Latest Google curiosity… I hadn’t spotted this before: it appears Google is now including ‘Code Snippet’ results in the results for its normal search. For example, a search for XSLoader gives this result:
The results highlighted on the page are for a local variable in a Java module, rather than the much more common XSLoader perl module. I guess ‘Code Snippet’ search is case-sensitive.
Suppose you have matching start() and end() functions. You want to ensure that each start() is always matched with its corresponding end(), without having to explicitly pepper your code with calls to that function. Here’s a good way to do it in perl — create a guard object:
package Scoper;
sub new {
my $class = shift; bless({ func => shift },$class);
}
sub DESTROY {
my $self = shift; $self->{func}->();
}
Here’s an example of its use:
{
start();
my $s = Scoper->new(sub { end(); });
[... do something...]
}
[at this point, end() has been called, even if a die() occurred]
The idea is simply to use DESTROY to perform whatever the cleanup operation is. Once the $s object goes out of scope, it’ll be deleted by perl’s GC, in the process of which, calling $s->DESTROY(). In other words, it’s using the GC for its own ends.
Unlike an eval { } block to catch die()s, this will even be called if exit() or POSIX::exit() is called. (POSIX::_exit(), however, skips DESTROY.)
This is a pretty old C++ pattern — Resource Acquisition Is Initialization. C++’s auto_ptr template class is the best-known example in that language. Here’s a perl.com article on its use in perl, from last year, mostly regarding the CPAN module Object::Destroyer. To be honest, though, it’s 6 lines of code — not sure if that warrants a CPAN module! ;)
RAII is used in SpamAssassin, in the Mail::SpamAssassin::Util::ScopedTimer class.
jwz has, incredibly, resurrected home.mcom.com, the WWW site of the Mosaic Communications Corporation, as it was circa Oct 1994.
Edmund Roche-Kelly was kind enough to get in touch and note this link — http://home.mcom.com/home/whatsnew/whats_new_0993.html:
September 3, 1993
IONA Technologies (whose product, Orbix, is the first full and complete implementation of the Object Management Group’s Common Object Request Broker Architecture, or CORBA) is now running a Web server.
An online pamphlet on the Church of the SubGenius is now available.
Guess who was responsible for those two ;)
I was, indeed, running the IONA web server — it was set up in June 1993, and ran Plexus, a HTTP server written in Perl. IONA’s server was somewhere around public web server number 70, world-wide.
The SubGenius pamphlet is still intact, btw, although at a more modern, "hyplan"-less URL these days. It’ll be 15 years old in 6 months… how time flies!
The New York Times yesterday had a great article about modern news consumption:
According to interviews and recent surveys, younger voters tend to be not just consumers of news and current events but conduits as well — sending out e-mailed links and videos to friends and their social networks. And in turn, they rely on friends and online connections for news to come to them. In essence, they are replacing the professional filter — reading The Washington Post, clicking on CNN.com — with a social one.
“There are lots of times where I’ll read an interesting story online and send the URL to 10 friends,” said Lauren Wolfe, 25, the president of College Democrats of America. “I’d rather read an e-mail from a friend with an attached story than search through a newspaper to find the story.”
[Jane Buckingham, the founder of the Intelligence Group, a market research company] recalled conducting a focus group where one of her subjects, a college student, said, “If the news is that important, it will find me.”
In other words, as Techdirt put it, this generation of news readers now focuses on sharing the news, rather than just consuming it — and if you want to share a news story, there’s no point passing on a subscription-only URL that your friends and contacts cannot read.
What newspapers need to do to remain relevant for this generation of news consumers is not to hide their content behind paywalls and registration-required screens. The Guardian got their heads around this a few years back, and have come along in leaps and bounds since then. I wonder if the Irish Times is listening?
Here’s a perl script that may prove useful: tap-to-junit-xml…
NAME
tap-to-junit-xml – convert perl-style TAP test output to JUnit-style XML
SYNOPSIS
tap-to-junit-xml "test suite name" [ outputprefix ] < tap_output.log
DESCRIPTION
Parse test suite output in TAP (Test Anything Protocol) format, and produce XML output in a similar format to that produced by the <junit> ant task. This is useful for consumption by continuous-integration systems like Hudson.
Written in perl, requires TAP::Parser and XML::Generator. It’s based on junit_xml.pl by Matisse Enzer, although pretty much entirely rewritten.
Following on from this post, I came across a great graphing Flash applet from amCharts which is ideal for putting a much improved UI on those SpamAssassin ‘required_score’ threshold line charts.
Here’s the result — nice, zoomable, dynamic charts, displaying the effectiveness of various ‘required_score’ thresholds depending on whether Bayes and network rules are active in your SpamAssassin configuration. Very cool!
I’ve just spent a rather frustrating morning attempting to debug major performance problems with my home wireless network; one of my machines couldn’t associate with the AP at all anymore, and the laptop (which was upstairs in the home office, for a change) was getting horrific, sub-dialup speeds.
I did lots of moving of Linksys APs and tweaking of "txpower" settings, without much in the way of results. Cue tearing hair out etc.
Eventually, I logged into the OpenWRT AP over SSH, ran iftop to see what clients were using the wifi, and saw that right at the top, chewing up all the available bandwidth, was a multicast group called 224.0.0.56. The culprit! There was nothing wrong with the wifi setup after all — the problem was massive bandwidth consumption, crowding out all other traffic.
You see, "pulseaudio", the new Linux sound server, has a very nifty feature — streaming of music to any number of listeners, over RTP. This is great. What’s not so great is that this seems to have magically turned itself on, and was broadcasting UDP traffic over multicast on my wifi network, which didn’t have enough bandwidth to host it.
Here’s how to turn this off without killing "pulseaudio". Start "paman", the PulseAudio Manager, and open the "Devices" tab:
(click on the image to view separately, if it’s partly obscured.)
Select the "RTP Monitor Stream" in the "Sources" list, and open "Properties":
Hit the "Kill" button, and your network is back to normal. Phew.
Another (quicker) way to do this, is using the command-line "pacmd" tool:
echo kill-source-output 0 | pacmd
It’s a mystery where this is coming from, btw. Here’s what "paman" says it came from:
But I don’t seem to have an active ‘module-rtp-send’ line in my configuration:
: jm 98...; grep module-rtp-send /etc/pulse/* /home/jm/.pulse* /etc/pulse/default.pa:#load-module module-rtp-send source=rtp.monitor
Curious. And irritating.
Update: it turns out there’s another source of configuration — GConf. "paprefs" can be used to examine that, and that’s where the setting had been set, undoubtedly by me hacking about at some stage. :(
Noted with alarm in this comment regarding the horrific privacy-invading adware that is Phorm:
Their programmers are mostly Saint Petersburg-based, home to the Russian Business Network. Their servers are kept only in Saint Petersburg and China, so no ISP customer data is ever stored in the UK. Any personally identifying information they obtain about UK citizens can never be seen or purged using existing UK Data Protection Laws.
St. Petersburg is turning out to be quite a source of online nastiness — the new Boca Raton.
As I noted on Monday, the Irish branches of several major record companies have brought a case against Eircom, demanding in part that the ISP install Audible Magic’s Copysense anti-filesharing appliances on their network infrastructure.
I thought I’d do a quick bit of research online into how they do their filtering. Here’s what the EFF had to say:
Audible Magic’s technology can easily be defeated by using one-time session key encryption (e.g., SSL) or by modifying the behavior of the network stack to ignore RST packets.
It’s interesting to see that they used RST packets — this is the same mechanism used by the "Great Firewall of China" to censor the internet:
the keyword detection is not actually being done in large routers on the borders of the Chinese networks, but in nearby subsidiary machines. When these machines detect the keyword, they do not actually prevent the packet containing the keyword from passing through the main router (this would be horribly complicated to achieve and still allow the router to run at the necessary speed). Instead, these subsiduary machines generate a series of TCP reset packets, which are sent to each end of the connection. When the resets arrive, the end-points assume they are genuine requests from the other end to close the connection — and obey. Hence the censorship occurs.
But there’s a very easy way to avoid this, according to that blog post:
However, because the original packets are passed through the firewall unscathed, if both of the endpoints were to completely ignore the firewall’s reset packets, then the connection will proceed unhindered! We’ve done some real experiments on this — and it works just fine!! Think of it as the Harry Potter approach to the Great Firewall — just shut your eyes and walk onto Platform 9¾.
Clayton, Murdoch, and Watson’s paper on this technique provides the Linux and FreeBSD firewall commands they used to do this. Here’s Linux:
iptables -A INPUT -p tcp --tcp-flags RST RST -j DROP
For FreeBSD, the command is:
ipfw add 1000 drop tcp from any to me tcpflags rst in
So assuming Copysense haven’t changed their approach yet, it’s trivial to block Copysense’s filtering, if both ends are running Linux or BSD. I predict if Copysense becomes widespread, someone will patch Windows TCP to do the same.
I love Audible Magic’s response:
The current appliance happens to use the TCP Reset to accomplish this today. There are many other technical methods of blocking transfers. Again, we have strategies to deal with them should they ever prove necessary. This is why we recommend our customers purchase a software support agreement which provides for these enhancements that keep their purchase up-to-date and protect their investment.
in other words, "hey customers! if you don’t have a support contract, you’re shit out of luck when the p2p guys get around our filters!" Nice. ;)
I’ve just fixed a bug on my linux desktop which had been annoying me for a while. Since there seems to be little online written about it, here’s a blog post to help future Googlers.
Here’s the symptoms: while you’re running VMWare, your Vim editing sessions freeze up for 20 seconds or so, roughly every 5 minutes. The editor is entirely hung.
If you strace -p the process ID before the hang occurs, you’ll see something like this:
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
_llseek(7, 4096, [4096], SEEK_SET) = 0
write(7, "tp\21\0\377\0\0\0\2\0\0\0|\0\0\0\1\0\0\0\1\0\0\0\6\0\0"..., 4096) = 4096
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
_llseek(7, 20480, [20480], SEEK_SET) = 0
write(7, "ad\0\0\245\4\0\0\341\5\0\0\0\20\0\0J\0\0\0\250\17\0\0\247"..., 4096) = 4096
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
fsync(
In other words, the hung process is sitting in an fsync() call, attempting to flush changed data for the current file to disk.
Investigation threw up the following: a kerneltrap thread about disk activity, poor responsiveness with Firefox 3.0b3 on linux, and a VIM bug report regarding this feature interfering with laptop-mode and spun-down hard disks.
VMWare must be issuing lots of unsynced I/O, so when Vim issues its fsync() or sync() call, it needs to wait for the VMWare I/O to complete before it can return — even though the machine is otherwise idle. A bit of a Linux kernel (or specifically, ext3) misfeature, it seems.
Synthesising details from those threads comes up with this fix: edit your ~/.vimrc and add the following lines —
set swapsync= set nofsync
This will inhibit use of both fsync() and sync() by Vim, and the problem is avoided nicely.
Update: one of the Firefox developers discusses how this affects FF 3.0.
RTE reports that 4 record companies, EMI, Sony BMG, Universal Music and Warner Music, have brought a High Court action to compel Eircom — Ireland’s largest ISP — to prevent its networks being used for the illegal downloading of music:
Willie Kavanagh, Managing Director of EMI Ireland and chairman of IRMA, said because of illegal downloading and other factors, the Irish music industry was experiencing a "dramatic and accelerating decline" in income. He said sales in the Irish market dropped 30% in the six years up to 2007.
EMI and the other companies are challenging Eircom’s refusal to use filtering technology or other measures to voluntarily block or filter illegally downloaded material. Last October Eircom told the companies it was not in a position to use the filtering software.
(I wonder if those dropping sales in the Irish market comprise only CDs sold by Irish shops? 2001 to 2007 is also the time period when physical sales have given way to online shopping on a gigantic scale, especially for music.)
The Irish Times coverage includes another interesting factoid, which appears in a lot of press regarding this case:
Latest figures available, for 2006, indicate that 20 billion music files were illegally downloaded worldwide that year. The music industry estimates that for every single legal download, there are 20 illegal ones.
A little research reveals that that figure comes from the IFPI Digital Music Report 2008. I’d have a totally different take on it, however. In my opinion, the figure is probably correct, but not for the reasons the IFPI want them to be. There are a number of factors:
it’s bloody hard to buy legit MP3s online. You can buy all sorts of shitty DRM-laden formats, or rent access to files that "expire", or buy MP3s in certain jurisdictions but not in others, or on certain platforms. So this is the main reason, in my opinion, why legit music downloads are a lot lower than they could be — because it’s tricky to get a legit download!
By contrast it’s a lot easier to log in to a filesharing service and download high-quality, DRM-free MP3s. Consumers have indicated that DRM frustrations drive them to filesharing sites, and no less than the Executive Vice President of the MPAA recognises this. (It looks like the IFPI dispute this, however.)
people, given the opportunity, tend to download free music, whether or not they plan to listen to it. It could be called the "radio effect"; they might listen to it, or they might not, there’s no pressure. If they have to pay for it, however, they’ll only download the stuff they really want. I definitely noticed this when the legal download site EMusic switched from an "all you can eat" model to a certain number of files per month; I immediately stopped downloading tunes speculatively, and restricted myself just to the ones I had already decided I wanted.
on BitTorrent filesharing sites, downloaders need to maintain high sharing ratios — so they download files just in order to re-upload them, regardless of whether they want them or not. (so I’m told. ;)
There’s more commentary on the 20-to-1 figure here.
The IFPI Digital Music Report 2008 also notes:
“2007 was the year ISP responsibility started to become an accepted principle. 2008 must be the year it becomes reality”
Governments are starting to accept that Internet Service Providers (ISPs) should take a far bigger role in protecting music on the internet, but urgent action is needed to translate this into reality, a new report from the international music industry says today.
ISP cooperation, via systematic disconnection of infringers and the use of filtering technologies, is the most effective way copyright theft can be controlled. Independent estimates say up to 80 per cent of ISP traffic comprises distribution of copyright-infringing files.
The IFPI Digital Music Report 2008 points to French President Sarkozy’s November 2007 plan for ISP cooperation in fighting piracy as a groundbreaking example internationally. Momentum is also gathering in the UK, Sweden and Belgium. The report calls for legislative action by the European Union and other governments where existing discussions between the music industry and record companies fail to progress.
So it seems Ireland is the vanguard of an international effort by IFPI members to force ISPs to install filtering, worldwide. It seems the same happened in Belgium last year — and I reckon there’ll be similar cases elsewhere soon.
Either way, I doubt this will be good for Irish internet users.
(PS: while I’m talking about buying MP3s online — a quick plug for 7digital. Last time I used them, I had a pretty crappy experience, but the situation is a lot better nowadays. They now have a great website that works perfectly in Firefox on Linux; they sell brand new releases like the Hercules and Love Affair album as 320kbps DRM-free MP3s; they support PayPal payments; and downloads are fast and simple — right click, "Save As". hooray!)
Some other blog coverage: Lex Ferenda with some details about the legal situation, and Jim Carroll.
Update: EMI Ireland seem to be singing from a different hymn-sheet than their head office… interesting.
Update 2: I’ve taken a look at the Copysense filtering technology, and how it can be evaded.
As I previously threatened, I’ve gone ahead and created a "Microplanet" for Irish twitterers, similar to Portland’s Pulse of PDX — an aggregator of the "stream of consciousness" that comes out of our local Twitter community: IrishPulse.
Here’s what you can do:
Add yourself: if you’re an Irish Twitter user, follow the user ‘irishpulse’. This will add you to the sources list.
Publicise it: feel free to pass on the URL to other Irish Twitter users, and blog about it.
Read it: bookmark and take a look now and again!
In terms of implementation, it’s just a (slightly patched) copy of Venus and a perl script using Net::Twitter to generate an OPML file of the Twitter followers. Here’s the source. I’d love to see more "Pulse" sites using this…
A couple of weeks ago, WebSense posted this article with details of a spammer’s attack on Google’s CAPTCHA puzzle, using web services running on two centralized servers:
[…] It is observed that two separate hosts active on same domain are contacted during the entire process. These two hosts work collaboratively during the CAPTCHA break process. […]
Why [use 2 hosts]? Because of variations included in the Google CAPTCHA image, chances are that host 1 may fail breaking the code. Hence, the spammers have a backup or second CAPTCHA-learning host 2 that tries to learn and break the CAPTCHA code. However, it is possible that spammers also use these two hosts to check the efficiency and accuracy of both hosts involved in breaking one CAPTCHA code at a time, with the ultimate goal of having a successful CAPTCHA breaking process.
To be specific, host 1 has a similar concept that was used to attack Live mail CAPTCHA. This involved extracting an image from a victim’s machine in the form of a bitmap file, bearing BM.. file headers and breaking the code. Host 2 uses an entirely different concept wherein the CAPTCHA image is broken into segments and then sent as a portable image / graphic file bearing PV..X file headers as requests. […]
While it doesn’t say as such, some have read the post to mean that Google’s CAPTCHA has been solved algorithmically. I’m pretty sure this isn’t the case. Here’s why.
Firstly, the FAQ text that appears on "host 1" (thanks Alex for the improved translation!):
FAQ
If you cannot recognize the image or if it doesn’t load (a black or empty image gets displayed), just press Enter.
Whatever happens, do not enter random characters!!!
If there is a delay in loading images, exit from your account, refresh the page, and log in again.
The system was tested in the following browsers: Internet Explorer Mozilla Firefox
Before each payment, recognized images are checked by the admin. We pay only for correctly recognized images!!!
Payment is made once per 24 hours. The minimum payment amount is $3. To request payment, send your request to the admin by ICQ. If the admin is free, your request will be processed within 10-15 minutes, and if he is busy, it will be processed as soon as possible.
If you have any problems (questions), ICQ the admin.
That reads to me a lot like instructions to human "CAPTCHA farmers", working as a distributed team via a web interface.
Secondly, take a look at the timestamps in this packet trace:
The interesting point is that there’s a 40-second gap between the invocation on "Captcha breaking host 1" and the invocation on "Captcha breaking host 2". There is then a short gap of 5 seconds before the invocations occur on the Gmail websites.
Here’s my theory: "host 1" is a web service gateway, proxying for a farm of human CAPTCHA solvers. "host 2", however, is an algorithm-driven server, with no humans involved. A human may take 40 seconds to solve a CAPTCHA, but pure code should be a lot speedier.
Interesting to note that they’re running both systems in parallel, on the same data. By doing this, the attackers can
collect training data for a machine-learning algorithm (this is implied by the ‘do not enter random characters!’ warning from the FAQ — they don’t want useless training data)
collect test cases for test-driven development of improvements to the algorithm
measure success/failure rates of their algorithms, "live", as the attack progresses
Worth noting this, too:
Observation*: On average, only 1 in every 5 CAPTCHA breaking requests are successfully including both algorithms used by the bot, approximating a success rate of 20%. The second algorithm (segmentation) has very poor performance that sometimes totally fails and returns garbage or incorrect answers.
So their algorithm is unreliable, and hasn’t yet caught up with the human farmers. Good news for Google — and for the CAPTCHA farmers of Romania ;)
Update: here’s the NYTimes’ take, with broadly agreeing comments from Brad Taylor of Google. (The Register coverage is off-base, however.)
So I was chatting to Danny O’Brien a few days ago. He noted that he’d reduced his Spamassassin "this is spam" threshold from the default 5.0 points to 3.7, and was wondering what that meant:
I know what it means in raw technical terms — spamassassin now marks anything >3.7 as spam, as opposed to the default of five. But given the genetic algorithm way that SA calculates the rule scoring, what does lowering the score mean? That I’m more confident that stuff marked ham is stuffed marked ham than the average person? That my bayesian scoring is now really good?
Do people usually do this without harmful side-effects? What does it mean about them if they do it?
Does it make me a good person? Will I smell of ham? These are the things that keep me awake at night.
It’s a good question! Here’s what I responded with — it occurs to me that this is probably quite widely speculated about, so let’s blog it here, too.
As you tweak the threshold, it gets more or less aggressive.
By default, we target a false positive rate of less than 0.1% — that means 1 FP, a ham marked as spam incorrectly, per 1000 ham messages. Last time the scores were generated, we ran our usual accuracy estimation tests, and got a false positive rate of 0.06% (1 in 1667 hams) and a false negative rate of 1.49% (1 in 67 spams) for the default threshold of 5.0 points. That’s assuming you’re using network tests (you should be) and have Bayes training (this is generally the case after running for a few weeks with autolearning on).
If you lower the threshold, then, that trades off the false negatives (reducing them — less spam getting past) in exchange for more false positives (hams getting caught). In those tests, here’s some figures for other thresholds:
SUMMARY for threshold 3.0: False positives: 290 0.43% False negatives: 313 0.26%
SUMMARY for threshold 4.0: False positives: 104 0.15% False negatives: 1084 0.91%
SUMMARY for threshold 4.5: False positives: 68 0.10% False negatives: 1345 1.13%
so you can see FPs rise quite quickly as the threshold drops. At 4.0 points, the nearest to 3.7, 1 in 666 ham messages (0.15%) will be marked incorrectly as spam. That’s nearly 3 times as many FPs as the default setting’s value (0.06%). On the other hand, only 1 in 109 spams will be mis-filed.
Here’s the reports from the last release, with all those figures for different thresholds — should be useful for figuring out the likelihoods!
In fact, let’s get some graphs from that report. Here is a graph of false positives (in orange) vs false negatives (in blue) as the threshold changes…
and, to illustrate the details a little better, zoom in to the area between 0% and 1%…
You can see that the default threshold of 5.0 isn’t where the FP% and FN% rates meet; instead, it’s got a much lower FP% rate than FN%. This is because we consider FPs to be much more dangerous than missed spams, so we try to avoid them to a higher degree.
An alternative, more standardized way to display this info is as a Receiver Operating Characteristic curve, which is basically a plot of the true positive rate vs false positives, on a scale from 0 to 1.
Here’s the SpamAssassin ROC curve:
More usefully, here’s the ROC curve zoomed in nearer the "perfect accuracy" top-left corner:
Unfortunately, this type of graph isn’t much use for picking a SpamAssassin threshold. GNUplot doesn’t allow individual points to be marked with the value from a certain column, otherwise this would be much more useful, since we’d be able to tell which threshold value corresponds to each point. C’est la vie!
Update:: this is possible with GNUplot 4.2 onwards, it seems. great news! Hat tip to Philipp K Janert for the advice. here are updated graphs using this feature:
(GNUplot commands to render these graphs are here.)
Update again: much better interactive Flash graphs here.
It’s been a while since I’ve posted about good shopping experiences I’ve had. Here’s a couple:
SoleTrader.co.uk: I’m a terrible shopper. I hate shops, I always wind up having to visit them at their busiest times on the weekend, and the last time I tried to go shopping for a new pair of shoes, I got caught in torrential rain, fell over and broke my thumb instead. seriously. So feck that.
Instead, I resolved to buy them online, and that I did — from SoleTrader. They had a great range of trainers, I found what I was after, the price was grand, and delivery on time. Shoes are always the same size — their sizes are standardised, after all — so naturally they fit fine. All in all, it worked out great.
Be Organic: these guys operate in North Dublin, delivering bags of organic fruit and vegetables to your door, weekly. We get the Essential Fruit Bag and the Mini Box, with a bi-weekly bag of spuds on top, for EUR 32 per week. The quality of the food is absolutely fantastic, there’s never any spoilage or wilting, and it’s always fresh and delicious. Compared to supermarket fare, it’s leagues ahead. They’ve also been grand and flexible when we need to tweak the order slightly — for example we have a veto on celery, and that’s not an issue at all. The only problem would be that they’ve recently increased their prices… but unfortunately that seems to be a general problem in Ireland these days!
A friend of a friend writes:
Unless you are pretty good at avoiding the media, you will be aware that Dustin the Turkey has been chosen as one of six finalists for RTE’s Eurosong, the winner of which will go on to represent Ireland in the Eurovision Song Contest in Serbia in May.
What you may not be aware of is that I wrote and recorded the song with him and need your votes to help get me to Serbia!!!
The TV show will be broadcast live on RTE this Saturday Feb 23rd, at 7pm. It is a televote (a la X-factor format), so get your mobile phones ready. The results are at 9:45pm.
The song, Irlande Douze Points, is a parody on the current types of songs, acts and block-voting in the Eurovision. It may make your ears bleed a bit, you may ask yourself why, but what the hell, send someone you know to the final!!!
Apparently, Dustin urges the contest judges to "give douze points to Ireland, for its lowlands and its highlands, for Terry Wogan’s wig and Bono’s leather pants. We brought you Guinness and Westlife, 800-years of war and strife, but we all apologise for Riverdance."
Check out the outraged reactions from Ireland’s past Eurovision "winners":
Frank McNamara, who wrote two of the Irish Eurovision winners, asked whether RTE, the state broadcaster that selected the six acts, was “giving two fingers” to Irish ‘song’writers. “I think it is absolutely disgraceful."
Shay Healy, who wrote Johnny Logan’s Eurovision hit What’s Another Year?, wondered “how any bunch of grown-ups could come up with this as a solution”
Phil Coulter thought that Eurovision was going “down the tubes”.
The choice on Saturday is between a turkey puppet taking the piss in a Northside accent, and such po-faced "serious pop" mawkfests as ‘"Double Cross My Heart" performed by Donal Skehan’ and ‘"Time to Rise" performed by Maya’. snore. You know it’s got to be the turkey.
Here’s the official Bebo page, and the Facebook group — and here’s the song itself:
Update: actually, here’s another, higher quality clip — with an entirely different song! Let’s hope this is the one…
Update 2: he won. Dana and the other professional Eurovision types have been chewing wasps, it’s hilarious!
Today, in work, we wound up discussing this classic DailyWTF.com article — "Remember, the enterprisocity of an application is directly proportionate to the number of constants defined":
public class SqlWords
{
public const string SELECT = " SELECT ";
public const string TOP = " TOP ";
public const string DISTINCT = " DISTINCT ";
/* etc. */
}
public class SqlQueries
{
public const string SELECT_ACTIVE_PRODCUTS =
SqlWords.SELECT +
SqlWords.STAR +
SqlWords.FROM +
SqlTables.PRODUCTS +
SqlWords.WHERE +
SqlColumns.PRODUCTS_ISACTIVE +
SqlWords.EQUALS +
SqlMisc.NUMBERS_ONE;
/* etc. */
}
This made me recall the legendary source code for the original Bourne shell, in Version 7 Unix. As this article notes:
Steve Bourne, at Bell Labs, worked on his version of shell starting from 1974 and this shell was released in 1978 as Bourne shell. Steve previously was involved with the development of Algol-68 compiler and he transferred general approach and some syntax sugar to his new project.
"Some syntax sugar" is an understatement. Here’s an example, from cmd.c:
LOCAL REGPTR syncase(esym)
REG INT esym;
{
skipnl();
IF wdval==esym
THEN return(0);
ELSE REG REGPTR r=getstak(REGTYPE);
r->regptr=0;
LOOP wdarg->argnxt=r->regptr;
r->regptr=wdarg;
IF wdval ORF ( word()!=')' ANDF wdval!='|' )
THEN synbad();
FI
IF wdval=='|'
THEN word();
ELSE break;
FI
POOL
r->regcom=cmd(0,NLFLG|MTFLG);
IF wdval==ECSYM
THEN r->regnxt=syncase(esym);
ELSE chksym(esym);
r->regnxt=0;
FI
return(r);
FI
}
Here are the #define macros Bourne used to "Algolify" the C compiler, in mac.h:
/*
* UNIX shell
*
* S. R. Bourne
* Bell Telephone Laboratories
*
*/
#define LOCAL static
#define PROC extern
#define TYPE typedef
#define STRUCT TYPE struct
#define UNION TYPE union
#define REG register
#define IF if(
#define THEN ){
#define ELSE } else {
#define ELIF } else if (
#define FI ;}
#define BEGIN {
#define END }
#define SWITCH switch(
#define IN ){
#define ENDSW }
#define FOR for(
#define WHILE while(
#define DO ){
#define OD ;}
#define REP do{
#define PER }while(
#define DONE );
#define LOOP for(;;){
#define POOL }
#define SKIP ;
#define DIV /
#define REM %
#define NEQ ^
#define ANDF &&
#define ORF ||
#define TRUE (-1)
#define FALSE 0
#define LOBYTE 0377
#define STRIP 0177
#define QUOTE 0200
#define EOF 0
#define NL '\n'
#define SP ' '
#define LQ '`'
#define RQ '\''
#define MINUS '-'
#define COLON ':'
#define MAX(a,b) ((a)>(b)?(a):(b))
Having said all that, the Bourne shell was an awesome achievement; many of the coding constructs we still use in modern Bash scripts, 30 years later, are identical to the original design.
For the benefit of other Technorati API users…
In a comment on this entry, Padraig Brady mentioned that his blog had mysteriously disappeared from the Irish Blogs Top 100 list.
I investigated, and found something odd — it seems Technorati has made a change to their bloginfo API, now listing weblogs with their ‘rank’, but without some of the important metadata, like ‘inboundblogs’, ‘inboundlinks’, and with a ‘lastupdate’ time set to the epoch (1970-01-01 00:00:00 GMT), in the API. Here’s an example:
<!-- generator="Technorati API version 1.0" -->
<!DOCTYPE tapi PUBLIC "-//Technorati, Inc.//DTD TAPI 0.02//EN"
"http://api.technorati.com/dtd/tapi-002.xml">
<tapi version="1.0">
<document>
<result>
<url>http://www.pixelbeat.org</url>
<weblog>
<name>Pádraig Brady</name>
<url>http://www.pixelbeat.org</url>
<rssurl></rssurl>
<atomurl></atomurl>
<inboundblogs></inboundblogs>
<inboundlinks></inboundlinks>
<lastupdate>1970-01-01 00:00:00 GMT</lastupdate>
<rank>74830</rank>
</weblog>
</result>
</document>
</tapi>
Compare that with this lookup result, on my own blog:
<?xml version="1.0" encoding="utf-8"?>
<!-- generator="Technorati API version 1.0" -->
<!DOCTYPE tapi PUBLIC "-//Technorati, Inc.//DTD TAPI 0.02//EN"
"http://api.technorati.com/dtd/tapi-002.xml">
<tapi version="1.0">
<document>
<result>
<url>http://taint.org</url>
<weblog>
<name>taint.org: Justin Mason’s Weblog</name>
<url>http://taint.org</url>
<rssurl>http://taint.org/feed</rssurl>
<atomurl>http://taint.org/feed/atom</atomurl>
<inboundblogs>143</inboundblogs>
<inboundlinks>227</inboundlinks>
<lastupdate>2008-02-12 11:48:10 GMT</lastupdate>
<rank>43404</rank>
</weblog>
<inboundblogs>143</inboundblogs>
<inboundlinks>227</inboundlinks>
</result>
</document>
</tapi>
This bug had caused a number of blogs to be dropped from the list, since I was using "inboundblogs and inboundlinks == 0" as an indication that a blog was not registered with Technorati.
It’s now worked around in my code, although a side-effect is that blogs which have this set will appear with question-marks in the ‘inboundblogs’ and ‘inboundlinks’ columns, and will perform poorly in the ‘ranked by inbound link count’ table (unsurprisingly).
I’ve posted a query to the support forum — let’s see what the story is.
This year’s Irish Blog Awards shortlists were posted yesterday. I maintain the Irish Blogs Technorati Top 100 list, so good sources of Irish blog URLs are always welcome; I took the shortlisted blogs and added them all.
Interestingly, straight in at number 2 went towleroad.com (warning: not worksafe!). It has a staggering Technorati rank of 1074 — way ahead of Donncha’s 5831 or Mulley’s 8678. I was pretty curious as to how an Irish blog could hit those heights without me having heard of it, so I took a look.
Let’s just say the content isn’t quite what you’d expect to find in a blog shortlisted for ‘Best News/Current Affairs Blog’ — a little bit short on Irish news, but heavy on pictures of naked guys getting off with each other. ;)
I took a quick glance, and I couldn’t spot any Irish content. WHOIS says the the publisher is LA-based. so I’m curious as to what qualified it as an "Irish blog"…
(by the way, I tried to leave a comment on the blog entry, but it appears Akismet is marking my comments as spam on a number of Wordpress-based blogs at the moment. Yes, I am aware of the irony. No, if SpamAssassin was a blog-spam filter, it wouldn’t do that ;)
Update: it’s sorted — they’re now gone. Also, it appears I’ve been removed from the Akismet blacklist, yay.
I needed to buy a new laptop for work a few months back, and after a little agonizing between the MacBook Pro and a Thinkpad T61p, I plumped for the latter. As I noted at the time, one of the major selling points was the quality of IBM/Lenovo’s after-sales warranty service, compared to the atrocious stories I’d heard about AppleCare in Europe. I was, however, taking a leap of faith — I had used IBM service to great effect in the US, but had never actually tried it out in Ireland.
Sadly, I had to put this to the test today, after the hard disk started producing these warnings:
/var/log/messages:Feb 7 11:21:13 wall kernel: [2075890.116000] end_request: I/O error, dev sda, sector 116189461 /var/log/messages:Feb 7 11:21:38 wall kernel: [2075914.824000] end_request: I/O error, dev sda, sector 116189460 /var/log/messages:Feb 7 11:24:18 wall kernel: [2076075.072000] end_request: I/O error, dev sda, sector 116189462 /var/log/messages:Feb 7 11:25:05 wall kernel: [2076121.932000] end_request: I/O error, dev sda, sector 116189463
It’s a brand new machine, and a Hitachi TravelStar 7K100 drive, with a good reputation for reliability — but these things do happen. :(
Interestingly, I thought this was a case of the Bathtub curve in action — but this comprehensive CMU study of hard drive reliability notes that the ‘infant mortality’ concept doesn’t seem to apply to current hard-drive technology:
Replacement rates [of hard drives in a cluster] are rising significantly over the years, even during early years in the lifecycle. Replacement rates in HPC1 nearly double from year 1 to 2, or from year 2 to 3. This ob- servation suggests that wear-out may start much earlier than expected, leading to steadily increasing replacement rates during most of a system’s useful life. This is an in- teresting observation because it does not agree with the common assumption that after the first year of operation, failure rates reach a steady state for a few years, forming the “bottom of the bathtub”.
Anyway, I digress.
I ran the BIOS hard disk self-test, got the expected failure, then rang up Lenovo’s International Warranty line for Ireland. I got through immediately to a helpful guy in India, and gave him my details and the BIOS error message; he had no tricky questions, no guff about me using Linux rather than Windows, and there were no attempts to sting me for shipping.
There’s now a replacement HD (and a set of spare recovery disks, bonus!) winging their way via 2-day shipping, expected on Tuesday; I’m to hand over the broken HD to the courier once it arrives. Fantastic stuff!
Assuming the courier doesn’t screw up, this is yet another major win for IBM/Lenovo support, and I feel vindicated. ;)
Update: the HD arrived this morning at 10am — a day early. Very impressive!
CEAS 2008 is doing another Spam Challenge test of various spam-filters, and as part of this, they need samples of ham mail messages.
As part of the data collection effort, we have set up a website through which it is possible to donate non-sensitive legitimate email, to be used in the evaluation. Any kind of email that the recipient considers legitimate is welcome, including computer generated (non-spam) messages.
After the CEAS evaluation, the benchmark data will be made publicly available to facilitate future reasearch and development in the field of spam prevention.
Here is the collection site; they accept UNIX mbox format, and tar.gz or zip files of same, with an 8MB upload limit.
My "Threadless New Tees" scraper feed is currently listing all items as being called ‘height="146"’. This is obviously not correct ;)
I’ll fix it ASAP…
Update: fixed!
For a while now, I’ve been using various hacks to play music from my Linux laptop, holding my main music collection, to client systems which drive the speakers.
Previously, I used this setup to play via my MythTV box. Nowadays, however, my TV isn’t in the room where I want to listen to music. Instead, I have my Nokia 770 hooked up to the speakers; this plays the BBC Radio 4 RealAudio streams nicely, and also the laptop’s MP3 collection using a uPnP AV MediaServer.
I specifically use TwonkyMedia right now, playing back via the N770’s Media Streamer app. (That works pretty well — uPnP AV is one of those standards plagued with incompatibilities, but TwonkyMedia and Media Streamer seem to be a reliable combination.)
However, TwonkyMedia sometimes fails to notice updates of the library, and nothing has quite as good a music-player user interface as JuK, the KDE music player and organiser app, so a way to play directly from the laptop instead of via uPnP would be nice…
A weekend’s hacking reveals that this is pretty easily done nowadays, thanks to some cool features in pulseaudio, the current standard sound server on Ubuntu gutsy, and the Esound server running on the N770.
Unfortunately, the N770 doesn’t (yet) support pulseaudio directly, otherwise we could use its seriously cool support for RTP multicast streams. Still, we can hack something up using the venerable "esd" protocol (again!) Here’s how to set it up…
On the N770:
You need to fix the N770’s "esd" sound server to allow public connections. Set up your wifi network’s DHCP server to give the N770 a static IP address. Log in over SSH, or fire up an xterm. Run the following:
mv /usr/bin/esd /usr/bin/esd.real
cat > /usr/bin/esd <<EOM
#!/bin/sh
exec /usr/bin/esd.real -tcp -public -promiscuous -port 5678 $*
EOM
chmod 755 /usr/bin/esd
/etc/init.d/esd restartOn the server:
Download this file, and save it as n770.pa. Edit it, and change server=n770:5678 on the fourth line to use the IP address or hostname of your Nokia 770 instead of n770. Then run:
cp n770.pa ~/.n770.pa
cat > ~/bin/sound_n770 <<EOM
#!/bin/sh
pulseaudio -k; pulseaudio -nF $HOME/.n770.pa &
EOM
cat > ~/bin/sound_here <<EOM
#!/bin/sh
pulseaudio -k; pulseaudio &
EOM
chmod 755 ~/bin/sound_here ~/bin/sound_n770Now you just need to run ‘~/bin/sound_n770’ to redirect sound playback to the N770, and ‘~/bin/sound_here’ to reset back to laptop speaker output, for the entire desktop environment. Nifty!
Update: it appears that things may work more reliably if you add "rate=22050" at the end of the "load-module module-esound-sink" line — this halves the bitrate of the network stream, which copes better with harsh wifi network conditions. The n770.pa file above now includes this.
It appears a public relations firm, Edelman’s, recently performed a phone survey which concluded that bloggers are the "least trusted" group of authority figures source of information in Ireland. This has been widely reported:
When we consider who we trust the most as a spokesperson in Ireland, the most trusted sources of information include, financial or industry analysts at 62%, followed by a doctor or healthcare specialist at 57%, an NGO representative at 57% and academics at 53%. Bloggers are the least trusted at 7%.
at Silicon Republic:
Bloggers have emerged as the “least trusted” group in the country.
and on ElectricNews.net:
"What has been interesting to note in this year’s findings is the apparent low standings of bloggers and social media in general," said [Mark Cahalane, managing director of Edelman Dublin]. "One interpretation of the survey would be that bloggers have now entered the mainstream and people no longer distinguish between blogs and ordinary websites. This is also reflected by the fact that numerous high profile bloggers are widely quoted in the media."
However, as Damien noted, Piaras Kelly raised a very significant point about this — ‘the people surveyed for the research had to fit a certain demographic, including having to be aged between 35-64.’ […] ‘A Generational gap is evident.’ This press release corroborates that. Sure enough, most blog readers (and writers) would tend to be of the younger generation — a pretty key point, one would assume, but one that most of the non-blogger coverage has omitted ;)
(Update: the term "authority figure" wasn’t quite correct; replaced with what Edelman themselves use, "source of information".)