Justin's Linklog Posts

NZ police affidavits show use of PRISM for surveillance of Kim “Megaupload” Dotcom

The discovery was made by blogger Keith Ng who wrote on his On Point blog (http://publicaddress.net/onpoint/ich-bin-ein-cyberpunk/) that the Organised and Financial Crime Agency New Zealand (OFCANZ) requested assistance from the Government Communications Security Bureau (GCSB), the country’s signals intelligence unit, which is charge of surveilling the Pacific region under the Five-Eyes agreement. A list of so-called selectors or search terms were provided to GCSB by the police [PDF, redacted] for the surveillance of emails and other data traffic generated by Dotcom and his Megaupload associates. ‘Selectors’ is the term used for the National Security Agency (NSA) XKEYSCORE categorisation system that Australia and New Zealand contribute to and which was leaked by Edward Snowden as part of his series of PRISM revelations. Some “selectors of interest” have been redacted out, but others such as Kim Dotcom’s email addresses, the mail proxy server used for some of the accounts and websites, remain in the documents.

(tags: surveillance prism nsa new-zealand xkeyscore gcsb kim-dotcom piracy privacy data-retention megaupload filesharing)

“Scalable Eventually Consistent Counters over Unreliable Networks” [paper, pdf]

Counters are an important abstraction in distributed computing, and play a central role in large scale geo-replicated systems, counting events such as web page impressions or social network “likes”. Classic distributed counters, strongly consistent, cannot be made both available and partition-tolerant, due to the CAP Theorem, being unsuitable to large scale scenarios. This paper defines Eventually Consistent Distributed Counters (ECDC) and presents an implementation of the concept, Handoff Counters, that is scalable and works over unreliable networks. By giving up the sequencer aspect of classic distributed counters, ECDC implementations can be made AP in the CAP design space, while retaining the essence of counting. Handoff Counters are the first CRDT (Conflict-free Replicated Data Type) based mechanism that overcomes the identity explosion problem in naive CRDTs, such as G-Counters (where state size is linear in the number of independent actors that ever incremented the counter), by managing identities towards avoiding global propagation, and garbage collecting temporary entries. The approach used in Handoff Counters is not restricted to counters, being more generally applicable to other data types with associative and commutative operations.

(tags: pdf papers eventual-consistency counters distributed-systems distcomp cap-theorem ecdc handoff-counters crdts data-structures g-counters)

LMDB response to a LevelDB-comparison blog post

This seems like a good point to note about LMDB in general:

We state quite clearly that LMDB is read-optimized, not write-optimized. I wrote this for the OpenLDAP Project; LDAP workloads are traditionally 80-90% reads. Write performance was not the goal of this design, read performance is. We make no claims that LMDB is a silver bullet, good for every situation. It’s not meant to be – but it is still far better at many things than all of the other DBs out there that *do* claim to be good for everything.

(tags: lmdb leveldb databases openldap storage persistent)

How to avoid crappy ISP caches when viewing YouTube video

Must give this a try when I get home — I frequently have latency problems watching YT on my UPC connection, and I bet they have a crappily-managed, overloaded cache box on their network.

(tags: streaming youtube caching isps caches firewalls iptables hacks video networking)

How to configure ntpd so it will not move time backwards

The “-x” switch will expand the step/slew boundary from 128ms to 600 seconds, ensuring the time is slewed (drifted slowly towards the correct time at a max of 5ms per second) rather than “stepped” (a sudden jump, potentially backwards). Since slewing has a max of 5ms per second, time can never “jump backwards”, which is important to avoid some major application bugs (particularly in Java timers).

(tags: ntpd time ntp ops sysadmin slew stepping time-synchronization linux unix java bugs)

Snowizard

‘a Java port of Twitter’s Snowflake thrift service presented as an HTTP-based Dropwizard service’.

an HTTP-based service for generating unique ID numbers at high scale with some simple guarantees. supports returning ID numbers as: JSON and JSONP; Google’s Protocol Buffers; Plain text. At GE, we were more interested in the uncoordinated aspects of Snowflake than its throughput requirements, so HTTP was fine for our needs. We also exposed the core of Snowflake as an embeddable module so it can be directly integrated into our applications. We don’t have the guarantees that the Snowflake-Zookeeper integration was providing, but that was also acceptable to us. In places where we really needed high throughput, we leveraged the snowizard-core embeddable module directly.

(tags: java open-source ids soa services snowflake http)

Containers and Docker: How Secure Are They?

pretty extensive article. (via Tony Finch)

(tags: via:fanf security containerization docker containers lxc linux ops)

Groklaw – Forced Exposure ~pj

I loved doing Groklaw, and I believe we really made a significant contribution. But even that turns out to be less than we thought, or less than I hoped for, anyway. My hope was always to show you that there is beauty and safety in the rule of law, that civilization actually depends on it. How quaint. If you have to stay on the Internet, my research indicates that the short term safety from surveillance, to the degree that is even possible, is to use a service like Kolab for email, which is located in Switzerland, and hence is under different laws than the US, laws which attempt to afford more privacy to citizens. I have now gotten for myself an email there, p.jones at mykolab.com in case anyone wishes to contact me over something really important and feels squeamish about writing to an email address on a server in the US. But both emails still work. It’s your choice. My personal decision is to get off of the Internet to the degree it’s possible. I’m just an ordinary person. But I really know, after all my research and some serious thinking things through, that I can’t stay online personally without losing my humanness, now that I know that ensuring privacy online is impossible. I find myself unable to write. I’ve always been a private person. That’s why I never wanted to be a celebrity and why I fought hard to maintain both my privacy and yours. Oddly, if everyone did that, leap off the Internet, the world’s economy would collapse, I suppose. I can’t really hope for that. But for me, the Internet is over. So this is the last Groklaw article. I won’t turn on comments. Thank you for all you’ve done. I will never forget you and our work together. I hope you’ll remember me too. I’m sorry I can’t overcome these feelings, but I yam what I yam, and I tried, but I can’t.

(tags: nsa surveillance privacy groklaw law us-politics data-protection snooping mail kolab)

Nelson’s Weblog: tech / bad / failure-of-encryption

One of the great failures of the Internet era has been giving up on end-to-end encryption. PGP dates back to 1991, 22 years ago. It gave us the technical means to have truly secure email between two people. But it was very difficult to use. And in 22 years no one has ever meaningfully made email encryption really usable. […] We do have SSL/HTTPS, the only real end-to-end encryption most of us use daily. But the key distribution is hopelessly centralized, authority rooted in 40+ certificates. At least 4 of those certs have been compromised by blackhat hackers in the past few years. How many more have been subverted by government agencies? I believe the SSL Observatory is the only way we’d know.

(tags: ssh ssl tls pki crypto end-to-end pgp security surveillance)

Recordinality

a new, and interesting, sketching algorithm, with a Java implementation:

Recordinality is unique in that it provides cardinality estimation like HLL, but also offers “distinct value sampling.” This means that Recordinality can allow us to fetch a random sample of distinct elements in a stream, invariant to cardinality. Put more succinctly, given a stream of elements containing 1,000,000 occurrences of ‘A’ and one occurrence each of ‘B’ – ‘Z’, the probability of any letter appearing in our sample is equal. Moreover, we can also efficiently store the number of times elements in our distinct sample have been observed. This can help us to understand the distribution of occurrences of elements in our stream. With it, we can answer questions like “do the elements we’ve sampled present in a power law-like pattern, or is the distribution of occurrences relatively even across the set?”

(tags: sketching coding algorithms recordinality cardinality estimation hll hashing murmurhash java)

You can’t “waste your vote”!

A fantastic infographic explaining Australia’s Preferential Voting system, featuring Dennis the Election Koala and Ken the Voting Dingo

(tags: infographics funny pr voting australia images via:fp)

David Miranda, schedule 7 and the danger that all reporters now face | Alan Rusbridger | Comment is free | The Guardian

The man was unmoved. And so one of the more bizarre moments in the Guardian’s long history occurred – with two GCHQ security experts overseeing the destruction of hard drives in the Guardian’s basement just to make sure there was nothing in the mangled bits of metal which could possibly be of any interest to passing Chinese agents. “We can call off the black helicopters,” joked one as we swept up the remains of a MacBook Pro. Whitehall was satisfied, but it felt like a peculiarly pointless piece of symbolism that understood nothing about the digital age. We will continue to do patient, painstaking reporting on the Snowden documents, we just won’t do it in London. The seizure of Miranda’s laptop, phones, hard drives and camera will similarly have no effect on Greenwald’s work. The state that is building such a formidable apparatus of surveillance will do its best to prevent journalists from reporting on it. Most journalists can see that. But I wonder how many have truly understood the absolute threat to journalism implicit in the idea of total surveillance, when or if it comes – and, increasingly, it looks like “when”. We are not there yet, but it may not be long before it will be impossible for journalists to have confidential sources. Most reporting – indeed, most human life in 2013 – leaves too much of a digital fingerprint. Those colleagues who denigrate Snowden or say reporters should trust the state to know best (many of them in the UK, oddly, on the right) may one day have a cruel awakening. One day it will be their reporting, their cause, under attack. But at least reporters now know to stay away from Heathrow transit lounges.

(tags: nsa gchq surveillance spying snooping guardian reporters journalism uk david-miranda glenn-greenwald edward-snowden)

al3x/sovereign

‘Sovereign is a set of Ansible playbooks that you can use to build and maintain’ your own GMail/Google calendar/etc. on a VPS. Some up-to-date hosting tips, basically

(tags: sovereign gmail google vps ansible al3x hosting)

New Tweets per second record, and how | Twitter Blog

How Twitter scaled up massively in 3 years — replacing Ruby with the JVM, adopting SOA and custom sharding. Good summary post, looking forward to more techie details soon

(tags: twitter performance scalability jvm ruby soa scaling)

Massive Overblocking Hits Hundreds Of UK Sites | Techdirt

Customers of UK ISPs Virgin Media and Be Broadband found they were unable to access hundreds of sites, including the Radio Times and Zooniverse, due to a secret website-blocking court order from the Premier League. PC Pro believe that 3 other ISPs’ customers were also affected. According to customers reverse-engineering, it looks like the court order incorrectly demanded the blocking of “http-redirection-a.dnsmadeeasy.com”, a HTTP redirector operated by the DNS operator DNSMadeEasy.

The fact that the court could issue an order which didn’t see this coming and that the ISPs would act on it without checking that what they were doing was sensible is, in my opinion, extremely worrying.

(tags: overblocking censorship org uk sky be-broadband virgin-media dnsmadeeasy filtering premier-league false-positives isps)

Beating the CAP Theorem Checklist

‘Your ( ) tweet ( ) blog post ( ) marketing material ( ) online comment advocates a way to beat the CAP theorem. Your idea will not work. Here is why it won’t work:’ lovely stuff, via Bill De hOra

(tags: via:dehora funny cap cs distributed-systems distcomp networking partitions state checklists)

‘Sparrow: Scalable Scheduling for Sub-Second Parallel Jobs’ [tech report]

(tags: scheduling sparrow load-balancing algorithms distributed-systems distcomp papers)

From derelict to delightful: Art Tunnel Smithfield

I do like the Art Tunnel. Smithfield is a great demo of reclaiming Dublin’s increasing dereliction and I hope the DCC allow this to continue

(tags: smithfield d7 dublin ireland art art-tunnel reclamation derelict economy dcc)

How A ‘Deviant’ Philosopher Built Palantir, A CIA-Funded Data-Mining Juggernaut – Forbes

Palantir — the free-market state-surveillance data-retention nightmare. At the end of this slightly overenthusiastic puff piece we get to:

Katz-Lacabe wasn’t impressed. Palantir’s software, he points out, has no default time limits — all information remains searchable for as long as it’s stored on the customer’s servers. And its auditing function? “I don’t think it means a damn thing,” he says. “Logs aren’t useful unless someone is looking at them.” […] What if Palantir’s audit logs — its central safeguard against abuse — are simply ignored? Karp responds that the logs are intended to be read by a third party. In the case of government agencies, he suggests an oversight body that reviews all surveillance — an institution that is purely theoretical at the moment. “Something like this will exist,” Karp insists. “Societies will build it, precisely because the alternative is letting terrorism happen or losing all our liberties.” Palantir’s critics, unsurprisingly, aren’t reassured by Karp’s hypothetical court. Electronic Privacy Information Center activist Amie Stepanovich calls Palantir “naive” to expect the government to start policing its own use of technology. The Electronic Frontier Foundation’s Lee Tien derides Karp’s argument that privacy safeguards can be added to surveillance systems after the fact. “You should think about what to do with the toxic waste while you’re building the nuclear power plant,” he argues, “not some day in the future.”

(tags: palantir data-retention privacy surveillance state cia forbes andy-greenberg eff epic snooping)

London orders rubbish bins to stop collecting smartphone data

Good call.

AUTHORITIES IN LONDON’S financial district have ordered a company using high-tech rubbish bins to collect smartphone data from passers-by to cease its activities, and referred the firm to the privacy watchdog. The City of London Corporation, which manages the so-called “Square Mile” around St Paul’s Cathedral, said such data collection “needs to stop” until there could be a public debate about it.

(tags: via:dobrien privacy phones wifi mac-address data-protection data-retention renew london bins snooping sniffing)

The Irish State wishes to uninvent computers with new FOI Bill

Mark Coughlan noticed this:

The FOI body shall take reasonable steps to search for and extract the records to which the request relates, having due regard to the steps that would be considered reasonable if the records were held in paper format.

(tags: funny irish ireland foi open-data freedom computerisation punch-cards paper databases)

Hamlet is Banned in the British Library

Pretty hilarious account of the usual, run-of-the-mill overblocking in the British Library from last weekend:

I asked [the information desk] if they saw the problem, perhaps just the symbolism, of Hamlet being banned in the British Library. They shrugged. The IT department said there was nothing to be done, as it was only the British Library’s wifi service that was blocking Hamlet, and the British Library’s wifi service, they seemed sure, had nothing to do with the British Library. They were merely ships that passed in the night. Children crying to each other from either bank of an uncrossable river.

(tags: censorship filters overblocking hamlet shakespeare literature funny sad british-library blocking)

The algorithm for a perfectly balanced photo gallery – Summit Stories from Crispy Mountain

Nice application of a partitioning exhaustive search algorithm using dynamic programming (via Tom)

(tags: algorithms javascript python dynamic-programming partitioning images gallery)

Soviets map America

An amazing Soviet map of the US economy from 1979. Wonderful piece of cold war memorabilia

(tags: cold-war ussr usa mapping maps soviet economy memorabilia)

Randomly Failed! The State of Randomness in Current Java Implementations

This would appear to be the paper which sparked off the drama around BitCoin thefts from wallets generated on Android devices:

The SecureRandom PRNG is the primary source of randomness for Java and is used e.g., by cryptographic operations. This underlines its importance regarding security. Some of fallback solutions of the investigated implementations [are] revealed to be weak and predictable or capable of being in?uenced. Very alarming are the defects found in Apache Harmony, since it is partly used by Android.

(tags: android java prng random security bugs apache-harmony apache crypto bitcoin papers)

The Getty Museum offers a huge chunk of their collection for free use

We’ve launched the Open Content Program to share, freely and without restriction, as many of the Getty’s digital resources as possible. The initial focus of the Open Content Program is to make available all images of public domain artworks in the Getty’s collections. Today we’ve taken a first step toward this goal by making roughly 4,600 high-resolution images of the Museum’s collection free to use, modify, and publish for any purpose. Why open content? Why now? The Getty was founded on the conviction that understanding art makes the world a better place, and sharing our digital resources is the natural extension of that belief. This move is also an educational imperative. Artists, students, teachers, writers, and countless others rely on artwork images to learn, tell stories, exchange ideas, and feed their own creativity. In its discussion of open content, the most recent Horizon Report, Museum Edition stated that “it is now the mark—and social responsibility—of world-class institutions to develop and share free cultural and educational resources.” I agree wholeheartedly.

(tags: getty art via:tupp_ed open-content free images pictures paintings museums)

The NSA Is Commandeering the Internet – Bruce Schneier

You, an executive in one of those companies, can fight. You’ll probably lose, but you need to take the stand. And you might win. It’s time we called the government’s actions what it really is: commandeering. Commandeering is a practice we’re used to in wartime, where commercial ships are taken for military use, or production lines are converted to military production. But now it’s happening in peacetime. Vast swaths of the Internet are being commandeered to support this surveillance state. If this is happening to your company, do what you can to isolate the actions. Do you have employees with security clearances who can’t tell you what they’re doing? Cut off all automatic lines of communication with them, and make sure that only specific, required, authorized acts are being taken on behalf of government. Only then can you look your customers and the public in the face and say that you don’t know what is going on — that your company has been commandeered.

(tags: nsa america politics privacy data-protection data-retention law google microsoft security bruce-schneier)

We are the Operations team at Etsy. Ask us anything! : IAmA

great AMA from Etsy ops staff (via Nelson)

(tags: etsy reddit devops ops architecture ama via:nelson)

Building a panopticon: The evolution of the NSA’s XKeyscore

This is an amazing behind-the-scenes look at the architecture of XKeyscore, and how it evolved from an earlier large-scale packet interception system, Narus’ Semantic Traffic Analyzer. XKeyscore is a federated, distributed system, with distributed packet-capture agents running on Linux, built with protocol-specific plugins, which write 3 days of raw packet data, and 30 days of intercept metadata, to local buffer stores. Central queries are then ‘distributed across all of the XKeyscore tap sites, and any results are returned and aggregated’. Dunno about you, but this is pretty much how I would have built something like this, IMO….

(tags: panopticon xkeyscore nsa architecture scalability packet-capture narus sniffing snooping interception lawful-interception li tapping)

Police may block recording with Apple patent

Creeptastic, Apple.

Apple has patented a piece of technology which would allow government and police to block transmission of information, including video and photographs, from any public gathering or venue they deem “sensitive”, and “protected from externalities.” In other words, these powers will have control over what can and cannot be documented on wireless devices during any public event. And while the company says the affected sites are to be mostly cinemas, theaters, concert grounds and similar locations, Apple Inc. also says “covert police or government operations may require complete ‘blackout’ conditions.”

(tags: apple iphone via:devore creepy police photos recording remote-control phones blackout)

Ivan Risti?: Defending against the BREACH attack

One interesting response to this HTTPS compression-based MITM attack:

The award for least-intrusive and entirely painless mitigation proposal goes to Paul Querna who, on the httpd-dev mailing list, proposed to use the HTTP chunked encoding to randomize response length. Chunked encoding is a HTTP feature that is typically used when the size of the response body is not known in advance; only the size of the next chunk is known. Because chunks carry some additional information, they affect the size of the response, but not the content. By forcing more chunks than necessary, for example, you can increase the length of the response. To the attacker, who can see only the size of the response body, but not anything else, the chunks are invisible. (Assuming they’re not sent in individual TCP packets or TLS records, of course.) This mitigation technique is very easy to implement at the web server level, which makes it the least expensive option. There is only a question about its effectiveness. No one has done the maths yet, but most seem to agree that response length randomization slows down the attacker, but does not prevent the attack entirely. But, if the attack can be slowed down significantly, perhaps it will be as good as prevented.

(tags: mitm attacks hacking security compression http https protocols tls ssl tcp chunked-encoding apache)

Totoro Isn’t All Cute. For Some, He’s the God of Death.

“Everyone, do not worry,” read the Studio Ghibli statement. “There’s absolutely no truth or configuration that Totoro is the God of Death or that Mei is dead in My Neighbor Totoro.”

(tags: totoro studio-ghibli death morbid japan film movies urban-legends alternate plot)

Hogan describes bin charge increases as ‘opportunistic’ – Environmental News | The Irish Times

LOL Greyhound.

Greyhound Recycling last month announced increases of 50 cents a month for customers on a flat monthly charge, 50 cents for each black bin collection for customers who pay by the lift and two cents a kilo for customers who pay by weight only. In a letter to customers, it described the levy as “tax imposed by the Government of Ireland on the people of Ireland”. However, following a complaint to the [National Consumer Agency] that the by-weight increase was 76 per cent more than the [government landfill levy] increase, Greyhound reduced the charge to an additional one cent a kilo.

(tags: greyhound ireland dublin rubbish recycling consumer ripoffs tax)

IrelandOffline broadband availability map

Marking the locations of broadband options in your area, along with VDSL cabinets, local exchanges, and wireless ISP coverage, and the landing sites of submarine cables (presumably from submarinecablemap.com data)

(tags: irelandoffline cables network internet ireland coverage wisps vdsl broadband)

Filters ‘not a silver bullet’ that will stop perverts, warns Interpol chief – Independent.ie

Sunday Independent interview with Interpol assistant director Mick Moran:

Moran spoke out after child welfare organisations here called on the Government to follow the UK’s example by placing anti-pornography filters on Irish home broadband connections. The Irish Society for the Prevention of Cruelty to Children argued that pornography was damaging to young children and should be removed from their line of sight. But Moran warned this would only lull parents into a false sense of security. “If we imagine the access people had to porn in the past – that access is now complete and total. They have access to the most horrific material out there. We now need to focus on parental responsibility about how kids are using the internet.”

(tags: mick-moran cam interpol policing ispcc filtering parenting children broadband)

Coordinated Omission

Gil Tene raises an extremely good point about load testing, high-percentile response-time measurement, and behaviour when testing a system under load:

I’ve been harping for a while now about a common measurement technique problem I call “Coordinated Omission” for a while, which can often render percentile data useless. […] I believe that this problem occurs extremely frequently in test results, but it’s usually hard to deduce it’s existence purely from the final data reported. But every once in a while, I see test results where the data provided is enough to demonstrate the huge percentile-misreporting effect of Coordinated Omission based purely on the summary report. I ran into just such a case in Attila’s cool posting about log4j2’s truly amazing performance, so I decided to avoid polluting his thread with an elongated discussion of how to compute 99.9%’ile data, and started this topic here. That thread should really be about how cool log4j2 is, and I’m certain that it really is cool, even after you correct the measurements. […] Basically, I think that the 99.99% observation computation is wrong, and demonstrably (using the data in the graph data posted) exhibits the classic “coordinated omission” measurement problem I’ve been preaching about. This test is not alone in exhibiting this, and there is nothing to be ashamed of when you find yourself making this mistake. I only figured it out after doing it myself many many times, and then I noticed that everyone else seems to also be doing it but most of them haven’t yet figured it out. In fact, I run into this issue so often in percentile reporting and load testing that I’m starting to wonder if coordinated omission is there in 99.9% of latency tests ;-)

(tags: measurement testing latency load-testing gil-tene coordinated-omission validity log4j percentiles)

Xerox scanners/photocopiers randomly alter numbers in scanned documents · D. Kriesel

Pretty major Xerox fail: photocopied/scanned docs are found to have replaced the digit ‘6’ with ‘8’, due to a poor choice of compression techniques:

Several mails I got suggest that the xerox machines use JBIG2 for compression. This algorithm creates a dictionary of image patches it finds “similar”. Those patches then get reused instead of the original image data, as long as the error generated by them is not “too high”. Makes sense. This also would explain, why the error occurs when scanning letters or numbers in low resolution (still readable, though). In this case, the letter size is close to the patch size of JBIG2, and whole “similar” letters or even letter blocks get replaced by each other.

(tags: jbig2 compression xerox photocopying scanning documents fonts arial image-compression images)

The 1940s origins of Whataboutery

The exchange is indicative of a rhetorical strategy known as ‘whataboutism’, which occurs when officials implicated in wrongdoing whip out a counter-example of a similar abuse from the accusing country, with the goal of undermining the legitimacy of the criticism itself. (In Latin, this rhetorical defense is called tu quoque, or “you, too.”)

(tags: history language whataboutism whataboutery politics 1940s russia ussr)

etcd

A highly-available key value store for shared configuration and service discovery. etcd is inspired by zookeeper and doozer, with a focus on: Simple: curl’able user facing API (HTTP+JSON); Secure: optional SSL client cert authentication; Fast: benchmarked 1000s of writes/s per instance; Reliable: Properly distributed using Raft; Etcd is written in go and uses the raft consensus algorithm to manage a highly availably replicated log.

(tags: configuration distributed raft ha doozer zookeeper go replication consensus-algorithm etcd coreos)

_In Search of an Understandable Consensus Algorithm_, Diego Ongaro and John Ousterhout, Stanford

Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundation for building practical systems. In order to enhance understandability, Raft separates the key elements of consensus, such as leader election and log replication, and it enforces a stronger degree of coherency to reduce the number of states that must be considered. Raft also includes a new mechanism for changing the cluster membership, which uses overlapping majorities to guarantee safety. Results from a user study demonstrate that Raft is easier for students to learn than Paxos.

(tags: distributed algorithms paxos raft consensus-algorithms distcomp leader-election replication clustering)

Extract from 1973 HM Treasury document concerning post-nuclear-attack responses

‘Extract from 1973 HM Treasury document concerning post-nuclear-attack monetary policy’ includes this amazing snippet:

[Contingency] …(d) a total nuclear attack employing high power missiles which would destroy all but a small percentage of the UK population and almost all physical assets or civilised life. […] As for (d), the money policy would of course be absurdly unrealistic for the few surviving administrators and politicians as they struggled to organise food and shelter for the tiny bands of surviving able-bodied and the probably larger number of sick and dying. Most of the other departments contingency planning might also be irrelevant in such a situation. Within a fairly short time the survivors would evacuate the UK and try to find some sort of life in less-effected countries (southern Ireland?).

(tags: nuclear attack contingency government monetary policy uk ireland history 1960s via:cstross insane fallout)

WhatClinic.com’s zombie recruitment video. We want your brains…

BRAAAAAAINS

(tags: whatclinic braaaaaains zombies funny video recruitment)

Guacamole Norteño

A very tasty-looking guac recipe, from h2g market veteran Lily Ramirez-Foran — her family’s traditional one. I like the addition of pomegranate seeds

(tags: guacamole avocados pomegranate recipes lily-ramirez-foran food h2g)

RA Forum: Button Factory – August 14th Simonetti (Goblin) Horror Project

LIVE – for the first time ever in Ireland, Claudio Simonetti (Goblin) & band will perform the classics of horror movie scores by seminal Italian progressive rock band Goblin, Simonetti himself and possibly one or two curve-balls ! Horror rock maestro Claudio Simonetti will fulfill fans’ dreams and nightmares as the band perform the notably eerie soundtracks from Suspriria, Tenebre, Dawn of the Dead, Creepers, Demons and more! This epic show will also feature an intense A/V screening element featuring the electric scenes from some of these revered classics of horror and giallo.

(tags: goblin bands music horror movies claudio-simonetti)

Python Infrastructure Status – SSL Verification Errors on PyPI

There appears to be a problem affecting a number of users where SSL verification errors will be shown saying “pypi.python.org” does not match “addvocate.com”. As Best we can tell this appears to be related to the ISP. It seems to be affecting folks using O2 or O2 related companies. We’ve also reports of it affecting people using Free. Cause appears to be one of the IP addresses returned in the Geo DNS for Europe returning a certificate for addvocate.com. It’s not clear at this time *why* that IP address is returning a certificate for addvocate.com.

(tags: via:micktwomey o2 censorship filtering internet ssl tls pypi python geodns pki)

“Toxic” behaviour in games is largely from “usually good” people

Only 5% of toxic behavior comes from toxic people; 77% of it comes from people who are usually good. That finding has all sorts of implications for how to stop toxic behavior in an online community. It’s not enough to just ban the jerks; good people have bad days too. Instead you have to teach the whole community what the community standards are. And quickly identify people who are having a bad day, intervene before their toxicity infects too many other people.

(tags: gaming toxic bad-behaviour trolls abuse online games league-of-legends)

Setting up FamilyShield

OpenDNS’s simple DNS-based blocking of dodgy content. Will need to set this up on the home router now that the kids are surfing…

(tags: opendns dns blocking filtering home porn familyshield)

Mail from the (Velvet) Cybercrime Underground

Brian Krebs manages to thwart an attempted framing for possession of Silk Road heroin. bloody hell

(tags: silk-road drugs bitcoin ecommerce brian-krebs crime framed cybercrime russia scary law-enforcement)

Clare dolphin attacks fourth swimmer in a month as Dusty protects her patch

Dusty the Dolphin has gone bad!

Locals say the three-metre long mammal has been responsible for injuring a number of people over the past two years, with several of those being hospitalised with significant injuries. She struck a 40-year-old woman in the abdomen earlier this month. In response, lifeguards now fly the red danger flag any time the dolphin enters the area. The Irish Whale and Dolphin Group has also erected warning posters at Doolin pier. IWDG coordinator Dr Simon Berrow said: “It is our policy to discourage people swimming with whales and dolphins in Ireland. “We’ve drafted a poster recommending people do not swim with Dusty, but if they must, then they should respect her as a wild dolphin and not grab, lunge or chase after her. If she shows aggressive behaviour or is boisterous they should leave the water.”

(tags: dusty dolphins wildlife nature fanore county-clare ireland swimming doolin animals)

Why YouTube buffers: The secret deals that make — and break — online video

Should ISPs be required to ensure they have sufficient upstream bandwidth to video sites like YouTube and Netflix?

“Verizon has chosen to sell its customers a product [Netflix] that they hope those customers don’t actually use,” Schaeffer said. “And when customers use it and request movies, they have not ensured there is adequate connectivity to get that video content back to their customers.”

(tags: netflix youtube streaming video isps net-neutrality peering comcast bandwidth upstream)

ISPAI Responds to Porn Filtering Debacle

Quite a strong statement:

The issue of access to age-inappropriate content is not a new matter and it is important not to have “knee-jerk” reactions which don’t solve the perceived problem and have major implications for the public’s right to access information in general. Notably the European Commission, as stated by vice-president Nellie Kroes [jm: sic], has come out strongly against blocking of the Internet, seeing it as an important platform for freedom of speech and she intends to “guarantee access without restriction.”  We in Ireland would do well to consider carefully the impact that any rash adoption or attempted copying of UK measures might have here in the light of current and future EU legislation and policy.

(tags: ispai filtering overblocking david-cameron porn internet ireland politics blocking web uk)

Forecast.io

Excellent weather site, displaying beautifully interpolated rainfall visualization, from the team behind the Dark Sky app

(tags: weather ireland dark-sky apps iphone ipad forecast rain dataviz mapping via:marcomorain)

Applied Cryptography, Cryptography Engineering, and how they need to be updated

Whoa, I had no idea my knowledge of crypto was so out of date! For example:

ECC is going to replace RSA within the next 10 years. New systems probably shouldn’t use RSA at all.

(tags: thomas-ptacek crypto cryptography coding design security aes cbc ctr ecb hmac side-channels rsa ecc)

When ‘Smart Homes’ Get Hacked: I Haunted A Complete Stranger’s House Via The Internet – Forbes

Hardware designers do their usual trick — omit the whole security part:

[Trustwave’s Crowley] found security flaws that would allow a digital intruder to take control of a number of sensitive devices beyond the Insteon systems, from the Belkin WeMo Switch to the Satis Smart Toilet. Yes, they found that a toilet was hackable. You only have to have the Android app for the $5,000 toilet on your phone and be close enough to the toilet to communicate with it. “It connects through Bluetooth, with no username or password using the pin ‘0000’,” said Crowley. “So anyone who has the application on their phone and was connected to the network could control anyone else’s toilet. You could turn the bidet on while someone’s in there.”

(tags: home automation insteon security hardware fail attacks bluetooth han trustwave belkin satis)

France Kills Three Strikes

Missed bookmarking this news —

After years of debate and controversy the French Government has finally backtracked on the law which allowed errant subscribers to be disconnected from the Internet. This morning a decree was published which removed the possibility for file-sharers to have their connections cut for copyright infringement. Instead, those caught by rightsholders will now be subjected to a system of automated fines.

(tags: france legal ip piracy filesharing three-strikes)

BBC News – Chinese firm Huawei controls net filter praised by PM

Talk Talk’s porn-filtering, system praised by David Cameron in the UK as a model for porn filtering for the country’s ISPs, is operated by Huawei. Of course, there’s no possible problems with allowing Huawei, with its alleged close ties to the Chinese government, operate a state-wide internet censorship system in the UK without any functioning oversight, right? ;) Also worth noting: all TalkTalk traffic passes through the Huawei filtering infrastructure, even when the customer has “opted in”.

(tags: huawei talk-talk oversight overblocking politics china uk david-cameron filtering censorship)

Branded to death | Features | Times Higher Education

The most abominable monster now threatening the intellectual health and the integrity of pure enquiry as well as conscientious teaching is the language of advertising, or better, the machinery of propaganda. Any number of critics from within university walls have warned the people at large and academics in particular of the way the helots of advertising and the state police of propaganda bloat and distort the language of thoughtful description, peddle with a confident air generalisations without substance, and serenely circulate orotund lies while ignoring their juniors’ rebuttals and abuse.

(tags: branding advertising newspeak universities third-level eolas higher-education education research university-of-california ucb patents ip swpats)

Twilio Billing Incident Post-Mortem

At 1:35 AM PDT on July 18, a loss of network connectivity caused all billing redis-slaves to simultaneously disconnect from the master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time. Receiving full sync requests from each redis-slave caused the master to suffer extreme load, resulting in performance degradation of the master and timeouts from redis-slaves to redis-master. By 2:39 AM PDT the host’s load became so extreme, services relying on redis-master began to fail. At 2:42 AM PDT, our monitoring system alerted our on-call engineering team of a failure in the Redis cluster. Observing extreme load on the host, the redis process on redis-master was misdiagnosed as requiring a restart to recover. This caused redis-master to read an incorrect configuration file, which in turn caused Redis to attempt to recover from a non-existent AOF file, instead of the binary snapshot. As a result of that failed recovery, redis-master dropped all balance data. In addition to forcing recovery from a non-existent AOF, an incorrect configuration also caused redis-master to boot as a slave of itself, putting it in read-only mode and preventing the billing system from updating account balances.

(tags: redis caching storage networking network-partitions twilio postmortems ops billing replication)

Tuning and benchmarking Java 7’s Garbage Collectors: Default, CMS and G1

Rudiger Moller runs through a typical GC-tuning session, in exhaustive detail

(tags: java gc tuning jvm cms g1 ops)

 Censum

[JVM] GC is a difficult, specialised area that can be very frustrating for busy developers or devops folks to deal with. The JVM has a number of Garbage Collectors and a bewildering array of switches that can alter the behaviour of each collector. Censum does all of the parsing, number crunching and statistical analysis for you, so you don’t have to go and get that PhD in Computer Science in order to solve your GC performance problem. Censum gives you straight answers as opposed to a ton of raw data. can eat any GC log you care to throw at it. is easy to install and use.

(tags: censum gc tuning ops java jvm commercial)

The Web’s longest nightmare ends: Eolas patents are dead on appeal | Ars Technica

Ding dong, the troll is dead! Ars Technica with a great description of the Eolas web patent fiasco, and the UC system’s sorry role. I blame Bayh-Dole for creating this insane mindset where places of learning are forced to “monetize” their research.

Under Doyle’s conception of his own invention, practically any modern website owed him royalties. Playing a video online or rotating an image on a shopping website were “interactive” features that infringed his patents. And unlike many “patent trolls” who simply settle for settlements just under the cost of litigation, Doyle’s company had the chops, the lawyers, and the early filing date needed to extract tens of millions of dollars from the accused companies. […] The role of the University of California is one of the most perplexing twists in the Eolas saga. The university kept a low profile during the lead-up to trial; but once in Texas, Eolas lawyers constantly reminded the jury they were asserting “these University of California patents.” A lawyer from UC’s patent-licensing division described support for Eolas at trial by simply saying that the university “stands by its licensees.” (Eolas was technically an exclusive licensee of the UC-owned patent, which also gives it the right to sue.) At the same time, the University of California, and the Berkeley campus in particular, was a key institution in creating early web technology. While UC lawyers cooperated with the plaintiffs, two UC Berkeley-trained computer scientists were key witnesses in the effort to demolish the Eolas patents. Pei-Yuan Wei created the pioneering Viola browser, a key piece of prior art, while he was a student at UC-Berkeley in the early 1990s. Scott Silvey, another UC-Berkeley student at that time, testified about a program he made called VPlot, which allowed users to rotate an image of an airplane using Wei’s browser. VPlot and Viola were demonstrated to Sun Microsystems in May 1993, months before Doyle claimed to have conceived of his invention.

(tags: patents swpats eolas web patent-trolls ucb universities research viola plugins berkeley)

Irish Comms Minister Pat Rabbitte ignores calls for State role in blocking online porn

Good call.

Mr Rabbitte says that legal concerns attached to mandatory filters, as well as a fear of imposing censorship, have persuaded him against trying to force ISPs to impose mandatory pornography-blocking internet filters. “I remain to be convinced that blanket censorship or a default-on blocker is the correct or workable response,” he said. “Even if it were possible to ensure that such measures were not easily circumvented or didn’t inadvertently block perfectly acceptable content, the principled question of whether the State should be encouraging service providers to filter or block content to all users, regardless of whether there are children resident, would still arise.”

(tags: pat-rabbitte internet filtering censorship blocking porn overblocking default-on isps ireland)

Grove

Hosted IRC, 20 users for $50/month. Useful now that Google have fecked up Chat entirely

(tags: irc chat collaboration groupware hosted-services)

UK Internet censorship plan no less stupid than it was last year – Boing Boing

Cory Doctorow’s long list of articles describing how the UK’s censorware-for-all plan is going to fail. I like this bit:

When we argued our case to the vendor’s representative, he was categorical: any nudity, anywhere on [Boing Boing], makes it into a “nudity site” for the purposes of blocking. The vendor went so far as to state that a single image of Michelangelo’s David, on one page among hundreds of thousands on a site, would be sufficient grounds for a nudity classification. I suspect that none of the censorship advocates in the Lords understand that the offshore commercial operators they’re proposing to put in charge of the nation’s information access apply this kind of homeopathic standard to objectionable material.

(tags: daily-mail fail censorship censorware boing-boing michelangelo sculpture nudity uk politics filtering overblocking web internet)

Content Aware Typography

Photoshop’s “Content Aware Fill” applied to text. some very cool results

(tags: images cool art typography algorithms via:pentadact photoshop)

A Tour Inside CloudFlare’s Latest Generation Servers

great transparency from CloudFront! Looking at their current 4th-gen rackmount server buildout — now with HP after Dell and ZT. Shitloads of SSDs for lower power and greater predictability in failure rates. 128GB RAM. consistent hashing to address stores instead of RAID. Sandybridge chipset. Solarflare SFC9020 10Gbps network cards. This is really impressive openness for a high-scale custom datacenter server platform…

(tags: datacenter cloudflare hardware rackmount ssds intel)

3D-Printer Manufacturer Creates Software Filter To Prevent Firearm Printing

‘[Create It REAL], which sells 3D printer component parts and software, recently announced that it has come up with a firearm component detection algorithm that will give 3D printers the option to block any gun parts. The software compares each component a user is trying to print with a database of potential firearms parts, and shuts down the modeling software if it senses the user is trying to make a gun.’

(tags: blocklists filtering guns weapons 3d-printing future firearms)

Fund it :: Upstart Granby Park

help fund Granby Park, a pop-up park to replace a vacant site on the corner of Dominick St and Parnell St in Dublin 1: http://upstart.ie/

(tags: fund-it granby-park dublin d1 parks pop-up city funding grassroots)

Rooting SIM cards

the details of Karsten Nohl’s attack against SIM cards, allowing remote-root malware via SMS.

Cracking SIM update keys: [Over The Air] commands, such as software updates, are cryptographically-secured SMS messages, which are delivered directly to the SIM. While the option exists to use state-of-the-art AES or the somewhat outdated 3DES algorithm for OTA, many (if not most) SIM cards still rely on the 70s-era DES cipher. […] To derive a DES OTA key, an attacker starts by sending a binary SMS to a target device. The SIM does not execute the improperly signed OTA command, but does in many cases respond to the attacker with an error code carrying a cryptographic signature, once again sent over binary SMS. A rainbow table resolves this plaintext-signature tuple to a 56-bit DES key within two minutes on a standard computer.

(tags: des 3des crypto security sms sim-cards smartcards java applets ota rainbow-tables cracking karsten-nohl)

Machine Learning Speeds TCP

Cool. A machine-learning-generated TCP congestion control algorithm which handily beats sfqCoDel, Vegas, Reno et al. But:

“Although the [computer-generated congestion control algorithms] appear to work well on networks whose parameters fall within or near the limits of what they were prepared for — even beating in-network schemes at their own game and even when the design range spans an order of magnitude variation in network parameters — we do not yet understand clearly why they work, other than the observation that they seem to optimize their intended objective well. We have attempted to make algorithms ourselves that surpass the generated RemyCCs, without success. That suggests to us that Remy may have accomplished something substantive. But digging through the dozens of rules in a RemyCC and ?guring out their purpose and function is a challenging job in reverse-engineering. RemyCCs designed for broader classes of networks will likely be even more complex, compounding the problem.” So are network engineers willing to trust an algorithm that seems to work but has no explanation as to why it works other than optimizing a specific objective function? As AI becomes increasingly successful the question could also be asked in a wider context.  

(tags: via-dehora machine-learning tcp networking hmm mit algorithms remycc congestion)

Street Cuffs: L.A. Sees Big Jump In Bike Thefts

Some [LA] bike messengers last month took justice into their own hands when they caught two suspected thieves, teenage boys who attended a local Catholic high school. According to police, the messengers stripped down the teens to their boxer shorts before taking their cellphones, backpacks and clothes. “They meted out street justice. We don’t condone street justice. They never threatened them. But they made it clear: don’t mess with another person’s property,” Los Angeles Police Lt. Paul Vernon said. “This incident and the arrests are the tip of the iceberg when comes to people stealing bicycles.” Vernon said the two boys told police they were robbed by about 20 men on bicycles at 6th Street and Grand Avenue about 3 p.m. on Jan. 12. Investigators said they cannot prove the boys were stealing bikes and continue to look for the assailants.

(tags: cycling theft robbery bike-theft la crime vigilantes cycle-couriers)

ICO’s Tame Investigation Of Google Street View Data Slurping

“People will yet again be asking whether Google has been let off without the kind of full and rigorous investigation that you would expect after this kind of incident,” Nick Pickles, director of the Big Brother Watch, told TechWeekEurope. “Let’s not forget that information was collected without permission from thousands of people’s Wi-Fi networks, in a way that if an individual had done so they would have almost certainly have been prosecuted. It seems strange that ICO [the UK’s Data Protection regulatory agency] did not want to inspect the [datacenter] cages housing the data, while it is also troubling that Google’s assurances were taken at face value, despite this not being the first incident where consumers have seen their privacy violated by the company.”

(tags: privacy google ico regulation data-protection snooping wifi sniffing network-traffic street-view)

Mexican Pickled Potatoes

‘My researches on the pickling matter had lead me to conclude that Mexico was, in fact, one of the few places where pickled potatoes were “a thing” and, in discussing same with Lily last month at her Mexican food stall in the Honest To Goodness market, I discovered that her soon-to-be-visiting Mexican mama was, in fact, a maker of such pickles. Not long afterward, I watched as Lily sat down with her mother, querying the ways of her pickled potatoes, translating and scribbling instructions for me as the details were recalled, not in an orderly series of steps, but in a series of asides and by-the-ways, by one for whom the practice of pickling potatoes was entirely second nature.’

(tags: pickling yum food mexico potatoes spuds recipes)

Porn to be Blocked in the UK – “What’s new?” Say Pirate Bay Users | TorrentFreak

It seems likely that the ISPs will implement a system similar to the one currently being used by TalkTalk, as the prime minister will specifically single the ISP out for praise in his speech. TalkTalk’s HomeSafe is a system which filters out URLs based on a remote blocklist provided and maintained by…. well, no one quite knows. This is worrying since when things don’t go quite to plan there’s no one to complain to. As previously reported, when TalkTalk customers are asked whether they want to block file-sharing sites, TorrentFreak.com is rendered inaccessible. Despite our pleas and complaints that we are a news resource, the company said it would not remove us from their blocklist. We doubt we’re the only ones being silenced.

(tags: talktalk blocking uk isps torrentfreak politics filtering david-cameron porn overblocking)

The Trello Tech Stack

Good description of how Fog Creek built out their Trello product; client-side JS rendering, model synced across the wire, HAProxy, Redis, and WebSockets. Bookmarked notably for this paragraph, which doesn’t ameliorate my fear of WebSockets as a tech:

The Socket.io server currently has some problems with scaling up to more than 10K [jm: oh dear] simultaneous client connections when using multiple processes and the Redis store, and the client has some issues that can cause it to open multiple connections to the same server, or not know that its connection has been severed.

(tags: websockets javascript architecture fog-creek trello ajax push)

Log4j 2: Performance close to insane

Nice writeup on Log4j 2’s new AsyncAppender implementation, based on the LMAX Disruptor. sounds pretty excellent:

“One nice little detail I should mention is that both Async Loggers and Async Appenders fix something that has always bothered me in Log4j-1.x, which is that they will flush the buffer after logging the last event in the queue . With Log4j-1.x, if you used buffered I/O, you often could not see the last few log events, as they were still stuck in the memory buffer. Your only option was setting immediateFlush to true, which forces disk I/O on every single log event and has a performance impact. With Async Loggers and Appenders in Log4j-2.0 your log statements are all flushed to disk, so they are always visible, but this happens in a very efficient manner.”

(tags: logging java performance async disruptor low-latency)

Chronicle

an ultra low latency, high throughput, persisted, messaging and event driven in memory database. The typical latency is as low as 80 nano-seconds and supports throughputs of 5-20 million messages/record updates per second. This library also supports distributed, durable, observable collections (Map, List, Set) The performance depends on the data structures used, but simple data structures can achieve throughputs of 5 million elements or key/value pairs in batches (eg addAll or putAll) and 500K elements or key/values per second when added/updated/removed individually. It uses almost no heap, trivial GC impact, can be much larger than your physical memory size (only limited by the size of your disk) and can be shared between processes with better than 1/10th latency of using Sockets over loopback. It can change the way you design your system because it allows you to have independent processes which can be running or not at the same time (as no messages are lost) This is useful for restarting services and testing your services from canned data. e.g. like sub-microsecond durable messaging. You can attach any number of readers, including tools to see the exact state of the data externally.

(tags: library messaging performance java chronicle disk mmap)

Stayhold

a completely new patent pending product designed in Ireland that is going to change the way people use their cars for carrying goods.  It is a solid plastic product that grips the carpet in your car and acts as a barrier to hold loose items securely against the side wall in your car trunk or boot.

(tags: stayhold transportation cars boot gadgets toget)

Docker

‘the Linux container engine’. I totally misunderstood what Docker was — this is cool.

Heterogeneous payloads: Any combination of binaries, libraries, configuration files, scripts, virtualenvs, jars, gems, tarballs, you name it. No more juggling between domain-specific tools. Docker can deploy and run them all. Any server: Docker can run on any x64 machine with a modern linux kernel – whether it’s a laptop, a bare metal server or a VM. This makes it perfect for multi-cloud deployments. Isolation: Docker isolates processes from each other and from the underlying host, using lightweight containers. Repeatability: Because each container is isolated in its own filesystem, they behave the same regardless of where, when, and alongside what they run.

(tags: lxc containers virtualization cloud ops linux docker deployment)

Next Generation Continuous Integration & Deployment with dotCloud’s Docker and Strider

Since Docker treats it’s images as a tree of derivations from a source image, you have the ability to store an image at each stage of a build. This means we can provide full binary images of the environment in which the tests failed. This allows you to run locally bit-for-bit the same container as the CI server ran. Due to the magic of Docker and AUFS Copy-On-Write filesystems, we can store this cheaply. Often tests pass when built in a CI environment, but when built in another (e.g. production) environment break due to subtle differences. Docker makes it trivial to take exactly the binary environment in which the tests pass, and ship that to production to run it.

(tags: docker strider continuous-integration continuous-deployment deployment devops ops dotcloud lxc virtualisation copy-on-write images)

Pinterest’s follower graph store, built on Redis

This is a good, high-availability Redis configuration; sharded by userid across 8192 shards, with a Redis master/slave pair of instances for each set of N shards. I like their use of two redundancy systems — hot slave and backup snapshots:

We run our cluster in a Redis master-slave configuration, and the slaves act as hot backups. Upon a master failure, we failover the slave as the new master and either bring up a new slave or reuse the old master as the new slave. We rely on ZooKeeper to make this as quick as possible. Each master Redis instance (and slave instance) is configured to write to AOF on Amazon EBS. This ensures that if the Redis instances terminate unexpectedly then the loss of data is limited to 1 second of updates. The slave Redis instances also perform BGsave hourly which is then loaded to a more permanent store (Amazon S3). This copy is also used by Map Reduce jobs for analytics. As a production system, we need many failure modes to guard ourselves. As mentioned, if the master host is down, we will manually failover to slave. If a single master Redis instance reboots, monit restart restores from AOF, implying a 1 second window of data loss on the shards on that instance. If the slave host goes down, we bring up a replacement. If a single slave Redis instance goes down, we rely on monit to restart using the AOF data. Because we may encounter AOF or BGsave file corruption, we BGSave and copy hourly backups to S3. Note that large file sizes can cause BGsave induced delays but in our cluster this is mitigated by smaller Redis data due to the sharding scheme.

(tags: graph redis architecture ha high-availability design redundancy sharding)

Flower Filter

‘A simple time-decaying approximate membership filter’ — like a Bloom filter with time decay. See also http://eng.42go.com/flower-filter-an-update/ for some notes on the non-independence of survival probabilities, and how that imposes negligible differences in practice.

(tags: bloom-filter algorithms coding probabilistic approximate time decay)

Spybike

This is brilliant. ‘covert bicycle GPS tracker; Notifies you by SMS if your bicycle moves; Online tracking’. ‘Spybike is a covert tracking device that is hidden inside your bicycle steerer tube. The device is disguised to look like a normal head set cap to avoid suspicion. If someone steals your bike, you can use SpyBike to track their movements online and on your mobile.’ More details: http://www.integratedtrackers.com/GPSTrack/pdf/Spybike_Instructions_2.pdf

(tags: spybike cycling theft gps tracking)

No Time To Spare [infographic]

‘On August 2, 2005, a fully-loaded Air France Airbus A340 arriving from Paris crash-landed at Toronto’s Pearson International Airport and caught fire. Only 4 of the 8 exits were usable, yet all 309 people on board made it off the aircraft in two minutes, before it was consumed by flames. Here, five of the passengers recount their escape.’

(tags: infographics travel air accidents fire airbus safety escape a340)

Merkel call for data protection rules puts Ireland in spotlight – Technology News

Irish Times on EU unhappiness with Ireland’s “light touch” data protection regime:

Hawkes’s appearance last month on RTÉ’s Morning Ireland regarding the US Prism surveillance programme, since posted to YouTube, reheated lingering resentment among many European data authorities. His admission that he “knew in a general way” about such programmes and didn’t “regard this particular revelation as particularly new” was a red rag to his European colleagues who fear Ireland is the transmission point of wholesale EU data to the US.

(tags: eu ireland data-protection privacy billy-hawkes regulation dpc)

Java Garbage Collection Distilled

a great summary of the state of JVM garbage collection from Martin Thompson

(tags: jvm java gc garbage-collection tuning memory performance martin-thompson)

Improved HTTPS Performance with Early SSL Termination

This is a neat hack. Since SSL/TLS connection establishment requires lots of consecutive round trips before the connection is ready, by performing that closer to the user and reusing an existing region-to-region connection behind the scenes, the overall latency is greatly improved. Works for HTTP as well

(tags: http https ssl architecture aws ec2 performance latency internet round-trip nginx tls)

How to secure your webapp

Locking down a webapp with current strict HTTPS policies.

It’s impossible to get to 100% security but there are steps you can take to secure your webapp for your users, to help mitigate against different types of attacks both against you, your webapp and your customers themselves. These are all things we’ve implemented with Server Density v2 to help harden the product as much as possible. These tips are in addition to security best practices such as protecting against SQL injection, filtering, session handling, and XSRF protection. Check out the OWASP cheat sheets and top 10 lists to ensure you’re covered for the basics before implementing the suggestions below.

(tags: https ssl security web webdev tls)

Breakthrough silicon scanning discovers backdoor in military chip [PDF]

Wow, I’d missed this:

This paper is a short summary of the ?rst real world detection of a backdoor in a military grade FPGA. Using an innovative patented technique we were able to detect and analyse in the ?rst documented case of its kind, a backdoor inserted into the Actel/Microsemi ProASIC3 chips for accessing FPGA con?guration. The backdoor was found amongst additional JTAG functionality and exists on the silicon itself, it was not present in any ?rmware loaded onto the chip. Using Pipeline Emission Analysis (PEA), our pioneered technique, we were able to extract the secret key to activate the backdoor, as well as other security keys such as the AES and the Passkey. This way an attacker can extract all the con?guration data from the chip, reprogram crypto and access keys, modify low-level silicon features, access unencrypted con?guration bitstream or permanently damage the device. Clearly this means the device is wide open to intellectual property (IP) theft, fraud, re-programming as well as reverse engineering of the design which allows the introduction of a new backdoor or Trojan. Most concerning, it is not possible to patch the backdoor in chips already deployed, meaning those using this family of chips have to accept the fact they can be easily compromised or will have to be physically replaced after a redesign of the silicon itself.

(tags: chips hardware backdoors security scanning pea jtag actel microsemi silicon fpga trojans)

small town council in Oz has been snooping on mobile phone records to catch litterbugs and owners of unregistered pets

Privacy advocates have slammed Wyndham council for spying on residents’ mobile phone data and email records almost 50 times in the past three years, “not to hunt down terrorists but to catch litterbugs and owners of unregistered pets”. Figures from the attorney-general’s department reveal Wyndham is the only Victorian council that has been snooping on personal data, seizing residents’ information 31 times during 2010-11 and 2011-12. Council’s acting chief executive Kelly Grigsby told the Weekly there had been another 18 authorisations in the past 12 months to chase people for unauthorised advertising, unregistered pets and illegal littering.

(tags: victoria australia oz privacy snooping data-retention metadata overreach)

Traditional AQM is not enough!

Jim Gettys on modern web design, HTTP, buffering, and FIFO queues in the network.

Web surfing is putting impulses of packets, without congestion avoidance, into FIFO queues where they do severe collateral damage to anything sharing the link (including itself!). So today’s web behavior incurs huge collateral damage on itself, data centers, the edge of the network, and in particular any application that hopes to have real time behavior. How do we solve this problem?

(tags: buffering networking internet web http protocols tcp bufferbloat jim-gettys codel fq_codel)

We interrupt this program to warn the Emergency Alert System is hackable | Ars Technica

Private SSH key included in a firmware update. Oh dear:

The US Emergency Alert System, which interrupts live TV and radio broadcasts with information about national emergencies in progress, is vulnerable to attacks that allow hackers to remotely disseminate bogus reports and tamper with gear, security researchers warned. The remote takeover vulnerability affects the DASDEC-I and DASDEC-II application servers made by a company called Digital Alert Systems. It stems from the a recent firmware update that mistakenly included the private secure shell (SSH) key, according to an advisory published Monday by researchers from security firm IOActive. Administrators use such keys to remotely log in to a server to gain unfettered “root” access. The publication of the key makes it trivial for hackers to gain unauthorized access on Digital Alert System appliances that run default settings on older firmware. “An attacker who gains control of one or more DASDEC systems can disrupt these stations’ ability to transmit and could disseminate false emergency information over a large geographic area,” the IOActive advisory warned. “In addition, depending on the configuration of this and other devices, these messages could be forwarded and mirrored by other DASDEC systems.”

(tags: ssh security fail emergency alert warning tv radio)

The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds

Good read.

Twitter is primarily a consumption mechanism, not a production mechanism. 300K QPS are spent reading timelines and only 6000 requests per second are spent on writes.

(tags: scale architecture scalability twitter high-scalability redis mysql)

Lightning Memory-Mapped Database

Sounds like a good potential replacement for Berkeley DB, at least for cases where LevelDB isn’t proving practical.

LMDB is a database storage engine similar to LevelDB or BDB which database authors often use as a base for building databases on top of. LMDB was designed as a replacement for BDB within the OpenLDAP project but it has been pretty useful to use with other databases as well. It’s API design is highly influenced by BDB so that replacing BDB is straightforward.

(tags: openldap lmdb databases bdb berkeley-db storage persistence oss open-source)

ssh – fabric appears to start apache2 but doesn’t – Stack Overflow

fabric fail. pty=False fixes the bug

(tags: fabric fail bugs pty ssh automation ops)

‘Copysets: Reducing the Frequency of Data Loss in Cloud Storage’ [paper]

An improved replica-selection algorithm for replicated storage systems.

We present Copyset Replication, a novel general purpose replication technique that signi?cantly reduces the frequency of data loss events. We implemented and evaluated Copyset Replication on two open source data center storage systems, HDFS and RAMCloud, and show it incurs a low overhead on all operations. Such systems require that each node’s data be scattered across several nodes for parallel data recovery and access. Copyset Replication presents a near optimal tradeoff between the number of nodes on which the data is scattered and the probability of data loss. For example, in a 5000-node RAMCloud cluster under a power outage, Copyset Replication reduces the probability of data loss from 99.99% to 0.15%. For Facebook’s HDFS cluster, it reduces the probability from 22.8% to 0.78%.

(tags: storage cloud-storage replication data reliability fault-tolerance copysets replicas data-loss)

Testing your database backups: the test environment database refresh pattern | #F80046

Great idea. I’m surprised I hadn’t come across this before

(tags: backups restoring ops sysadmin testing)

Clean Code Cheat Sheet [pdf]

‘principles, patterns, smells and guidelines for clean code, class and package design, TDD, Acceptance Test Driven Development, and CI’

(tags: clean-code code-smells coding tdd testing continous-integration patterns pdf)

Applegate’s Law

‘Over time, the probability of someone drawing a cock with your [user-generated content] app approaches one.’

(tags: cocks time-to-penis user-generated-content content ugc via:rob-manuel qwghlm funny applegates-law web b3ta lol)

Fat Tails

Nice d3.js demo of the fat-tailed distribution:

A fat-tailed distribution looks normal but the parts far away from the average are thicker, meaning a higher chance of huge deviations. […] Fat tails don’t mean more variance; just different variance. For a given variance, a higher chance of extreme deviations implies a lower chance of medium ones.

(tags: dataviz via:hn statistics visualization distributions fat-tailed kurtosis d3.js javascript variance deviation)

Google Cloud Messaging for Android

GCM is a service that allows you to send data from your server to your users’ Android-powered device, and also to receive messages from devices on the same connection. The GCM service handles all aspects of queueing of messages and delivery to the target Android application running on the target device. GCM is completely free no matter how big your messaging needs are, and there are no quotas.

(tags: gcm messaging android google push)

packetdrill – network stack testing tool

[Google’s] packetdrill scripting tool enables quick, precise tests for entire TCP/UDP/IPv4/IPv6 network stacks, from the system call layer down to the NIC hardware. packetdrill currently works on Linux, FreeBSD, OpenBSD, and NetBSD. It can test network stack behavior over physical NICs on a LAN, or on a single machine using a tun virtual network device.

(tags: testing networking tun google linux papers tcp ip udp freebsd openbsd netbsd)

the TCP bounded buffer deadlock problem

I’ve wound up mentioning this twice in the past week, so it’s worth digging up and bookmarking!

Under certain circumstances a TCP connection can end up in a “deadlock”, where neither the client nor the server is able to write data out or read data in. This is caused by two factors. First, a client or server cannot perform two transactions at once; a read cannot be performed if a write transaction is in progress, and vice versa. Second, the buffers that exist at either end of the TCP connection are of limited size. The deadlock occurs when both the client and server are trying to send an amount of data that is larger than the combined input and output buffer size.

(tags: tcp ip bounded-buffer deadlock bugs buffering connections distributed-systems)

An excellent writeup of the TCP bounded-buffer deadlock problem

on pages 146-149 of ‘TCP/IP Sockets in C: Practical Guide for Programmers’ by Michael J. Donahoo and Kenneth L. Calvert.

(tags: tcp ip bounded-buffer deadlock bugs buffering connections distributed-systems)

How The Copyright Industry Pushed For Internet Surveillance | TorrentFreak

Rick Falkvinge with a good point:

The reason for the copyright industry to push for surveillance is simple: any digital communications channel can be used for private conversation, but it can also be used to share culture and knowledge that is under copyright monopoly. In order to tell which communications is which, you must sort all of it – and to do that, you must look at all of it. In other words, if enforcing the copyright monopoly is your priority, you need to kill privacy, and specifically anonymity and secrecy of correspondence.

(tags: copyright privacy state data-protection rick-falkvinge copyfight internet filtering surveillance anonymity)

Aer Lingus set to resume flights to San Francisco from Dublin

Yay!

Google, Apple and Facebook have persuaded Aer Lingus to reopen the San Francisco to Dublin route, according to sources in the US. The technology giants have their European headquarters in Dublin and their American bases in San Francisco. According to insiders, Aer Lingus will make an announcement soon having received assurances that Silicon Valley companies will take up seats.

(tags: flights travel ireland san-francisco sf aer-lingus)

Comics For Children…. a visual list…. | The Forbidden Planet International Blog

some great recommendations here. Hildafolk has been popular with my 5-year-old, must pick up a few more

(tags: comics kids children books reading library toget toread)

_Measuring Mobile Web Performance_ [slides]

Notable slide is #13, displaying a graph of HSDPA packet RTTs measured from a train. Max RTT gets up to 20,266ms. ouch

(tags: rtt packets latency hsdpa mobile internet trains packet-loss)

Latest leak of EU Data Protection Regulation makes fines impossible

Well, isn’t this convenient. The leaked proposed regulation document from the Irish EU presidency contains the following changes from current law:

what is new is a set of prescriptive conditions which, if adopted, appears to make a Monetary Penalty Notice (MPN) almost impracticable to serve. This is because the [Data Protection] Commissioner would have consider a dozen factors (many of which will give no doubt rise to appeal). […] In addition, the fines in the Regulation require consideration of the actual damage caused; this compares unfavourably with the current MPN where large fines have been contingent on grave security errors on the part of the data controller (i.e. the MPN of the UK DPA does not need damage to data subjects – only the likelihood of substantial distress or damage which should have been preventable/foreseeable).

(tags: data-protection law eu ec ireland privacy fines regulation mpn)

Google Translate of “Lorem ipsum”

The perils of unsupervised machine learning… here’s what GTranslate reckons “lorem ipsum” translates to:

We will be sure to post a comment. Add tomato sauce, no tank or a traditional or online. Until outdoor environment, and not just any competition, reduce overall pain. Cisco Security, they set up in the throat develop the market beds of Cura; Employment silently churn-class by our union, very beginner himenaeos. Monday gate information. How long before any meaningful development. Until mandatory functional requirements to developers. But across the country in the spotlight in the notebook. The show was shot. Funny lion always feasible, innovative policies hatred assured. Information that is no corporate Japan

(tags: lorem-ipsum boilerplate machine-learning translation google translate probabilistic tomato-sauce cisco funny)

how RAID fits in with Riak

Write heavy, high performance applications should probably use RAID 0 or avoid RAID altogether and consider using a larger n_val and cluster size. Read heavy applications have more options, and generally demand more fault tolerance with the added benefit of easier hardware replacement procedures.

(tags: via:dehora riak cluster fault-tolerance raid ops)

Locally Repairable Codes

Facebook’s new erasure coding algorithm (via High Scalability).

Disk I/O and network traffic were reduced by half compared to RS codes. The LRC required 14% more storage than RS (ie. 60% of data size). Repair times were much lower thanks to the local repair codes. Much greater reliability thanks to fast repairs. Reduced network traffic makes them suitable for geographic distribution.

(tags: erasure-coding facebook redundancy repair algorithms papers via:highscalability data storage fault-tolerance)

Boundary’s Early Warnings alarm

Anomaly detection on network throughput metrics, alarming if throughputs on selected flows deviate by 1, 2, or 3 standard deviations from a historical baseline.

(tags: network-monitoring throughput boundary service-metrics alarming ops statistics)

My email to Irish Times Editor, sent 25th June

Daragh O’Brien noting 3 stories on 3 consecutive days voicing dangerously skewed misinformation about data protection and privacy law in Ireland:

There is a worrying pattern in these stories. The first two decry the Data Protection legislation (current and future) as being dangerous to children and damaging to the genealogy trade. The third sets up an industry “self-regulation” straw man and heralds it as progress (when it is decidedly not, serving only to further confuse consumers about their rights). If I was a cynical person I would find it hard not to draw the conclusion that the Irish Times, the “paper of record” has been stooged by organisations who are resistant to the defence of and validation of fundamental rights to privacy as enshrined in the Data Protection Acts and EU Treaties, and in the embryonic Data Protection Regulation. That these stories emerge hot on the heels of the pendulum swing towards privacy concerns that the NSA/Prism revelations have triggered is, I must assume, a co-incidence. It cannot be the case that the Irish Times blindly publishes press releases without conducting cursory fact checking on the stories contained therein? Three stories over three days is insufficient data to plot a definitive trend, but the emphasis is disconcerting. Is it the Irish Times’ editorial position that Data Protection legislation and the protection of fundamental rights is a bad thing and that industry self-regulation that operates in ignorance of legislation is the appropriate model for the future? It surely cannot be that press releases are regurgitated as balanced fact and news by the Irish Times without fact checking and verification? If I was to predict a “Data Protection killed my Puppy” type headline for tomorrow’s edition or another later this week would I be proved correct?

(tags: daragh-obrien irish-times iab bias advertising newspapers press-releases journalism data-protection privacy ireland)

_Bolt-On Causal Consistency_ [slides]

SIGMOD 2013 presentation from Peter Bailis, Ali Ghodsi, Joseph M. Hellerstein, Ion Stoica — adding consistency to an eventually-consistent store by tracking dependencies

(tags: eventual-consistency state cap-theorem storage peter-bailis)