Justin's Linklog Posts

How to make a security geek feel very old: #Factorisation, #DKIM and @DrZacharyHarris

“A 384-bit key I can factor on my laptop in 24 hours. The 512-bit keys I can factor in about 72 hours using Amazon Web Services for $75. And I did do a number of those. Then there are the 768-bit keys. Those are not factorable by a normal person like me with my resources alone. But the government of Iran probably could, or a large group with sufficient computing resources could pull it off.” Remember when we thought 512-bit keys would be enough? how time flies! Of course, John Aycock raised this problem back in 2007, although he assumed it’d take a 100,000-host botnet to crack them (in 153 minutes).

(tags: factorisation moores-law cpu speed dkim domain-keys 512-bit cracking security via:alec-muffet)

Data distribution in the cloud with Node.js

Very interesting presentation from ex-IONAian Darach Ennis of Push Technology on eep.js, embedded event processing in Javascript for node.js stream processing. Handles tumbling, monotonic, periodic and sliding windows at 8-40 million events per second; no multi-dimensional, infinite or predicate event-processing windows. (via Sergio Bossa)

(tags: via:sbtourist events event-processing streaming data ex-iona darach-ennis push-technology cep javascript node.js streams)

Raspberry Pi gets open-source video drivers

‘As of right now, all of the VideoCore driver code which runs on the ARM is available under a FOSS license (3-Clause BSD to be precise). If you’re not familiar with the status of open source drivers on ARM SoCs this announcement may not seem like such a big deal, but it does actually mean that the BCM2835 used in the Raspberry Pi is the first ARM-based multimedia SoC with fully-functional, vendor-provided (as opposed to partial, reverse engineered) fully open-source drivers, and that Broadcom is the first vendor to open their mobile GPU drivers up in this way.’ This is a great result — congrats to the Raspberry Pi team for getting this to happen.

(tags: raspberry-pi open-source hardware drivers gpu graphics embedded-linux linux broadcom bsd bcm2835)

experimental CPU-cache-aware hash table implementations in Cloudera’s Impala

via Todd Lipcon — https://twitter.com/tlipcon/status/261113382642532352 ‘another cool piece of cloudera impala source: cpu-cache-aware hash table implementations by @jackowayed’. ‘L1-sized hash table that hopes to use cache well. Each bucket is a chunk list of tuples. Each chunk is a cache line.’

(tags: hashing hash-tables data-structures performance c++ l1 cache cpu)

ElementCostInDataStructures

“The cost per element in major data structures offered by Java and Guava (r11)].” A very useful reference!

Ever wondered what’s the cost of adding each entry to a HashMap? Or one new element in a TreeSet? Here are the answers: the cost per-entry for each well-known structure in Java and Guava. You can use this to estimate the cost of a structure, like this: if the per-entry cost of a structure is 32 bytes, and your structure contains 1024 elements, the structure’s footprint will be around 32 kilobytes. Note that non-tree mutable structures are amortized (adding an element might trigger a resize, and be expensive, otherwise it would be cheap), making the measurement of the “average per element cost” measurement hard, but you can expect that the real answers are close to what is reported below.

(tags: java coding guava reference memory cost performance data-structures)

SmoothLifeL

‘A continuous version of Conway’s Life, using floating point values instead of integers’. ‘SmoothLifeL supports many interesting phenomena such as gliders that can travel in any direction, rotating pairs of gliders, ‘wickstretchers’ and the appearance of elastic tension in the ‘cords’ that join the blobs.’ paper: http://arxiv.org/abs/1111.1567 , and slides: http://www.youtube.com/watch?v=iyTIXRhjXII (via jwz)

(tags: life games emergent-behaviour algorithms graphics via:jwz cool eye-candy conways-life floating-point continuous gliders)

Trident: a high-level abstraction for realtime computation

built on Storm:

Trident is a new high-level abstraction for doing realtime computing on top of Twitter Storm, available in Storm 0.8.0. It allows you to seamlessly mix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you’re familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar – Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies.

(tags: distributed realtime twitter storm trident distcomp stream-processing low-latency nathan-marz)

Adam Ferriss

glitch art, colour separations, etc. (via mlkshk)

(tags: art glitch graphics colour)

Cliff Click’s 2008 JavaOne talk about the NonBlockingHashTable

I’m a bit late to this data structure — highly scalable, nearly lock-free, benchmarks very well (except with the G1 GC): http://edwwang.com/blog/2012/02/10/concurrent-hashmap-benchmark/ . Having said that, it doesn’t cope well with frequently-changing unique keys: http://sourceforge.net/tracker/?func=detail&aid=3563980&group_id=194172&atid=948362 . More background at: http://www.azulsystems.com/blog/cliff/2007-03-26-non-blocking-hashtable and http://www.azulsystems.com/blog/cliff/2007-04-01-non-blocking-hashtable-part-2 This was used in Cassandra for a while, although I think the above bug may have caused its removal?

(tags: nonblockinghashtable data-structures hashmap concurrency scaling java jvm)

Ingenious Dublin

Excellent stuff, by Mary Mulvihill:

Where in Dublin can you see a Victorian diving bell? What about the skeleton of Tommy, the prince’s elephant? The site of the world’s first earthquake experiment? Or the world’s sports pirate radio broadcast? Our new e-book Ingenious Dublin has all these fascinating stories and more. It is packed with information, places to visit, and lots of illustrations, and covers the city and county, from Skerries windmills to Ballybetagh’s fossil deer.’

(tags: kindle reading books mary-mulvihill science facts dublin ireland history)

Weathering the Unexpected – ACM Queue

Failures happen, and resilience drills help organizations prepare for them.

(tags: game-day tests disaster-recovery dirt exercises history amazon google etsy resilience acm)

Meeting A Troll…

This is a must-read. One journalist’s experience of constant online harassment by an antisemitic internet troll, and their eventual unmasking.

(tags: internet trolling harassment trolls antisemitism stories twitter)

Your Approach to Saving British Newspapers Will Not Work

the bad-anti-spam-idea checklist, repurposed

(tags: checklists via:trish-byrne funny media news newspapers uk ireland)

Facebook monitoring cache with Claspin

reasonably nice heatmap viz for large-scale instance monitoring. I like the “snake” pattern for racks

(tags: facebook monitoring dataviz heatmaps claspin cache memcached ui)

The Oireachtas great leap backwards: it’s not just about KildareStreet.com

‘it appears that the Oireachtas has decided to save time and money by eliminating entirely the stage in their workflow that parsed raw debates records into XML. This stage has been replaced with a (presumably automated) process that generates web pages from Lotus Notes. It’s easy to see how somebody with little appreciation of the value of providing open public data in a structured format could have viewed this stage as a costly luxury, and its elimination as a simple and obvious “efficiency”. It’s particularly disappointing, however, that nobody in the decision-making process seemed to be aware of how much of a backward step this “efficiency” would represent. As John Handelaar of KildareStreet.com told The Irish Times, “We are replacing 2012 with 1995 overnight”.’

(tags: kildare-street open-data opengov ireland data oireachtas)

PCRE Performance Project

Excellent stuff. Using “sljit”, a stackless platform-independent JIT compiler, this compiles Perl-compatible regular expressions to machine code on ARM, x86, MIPS and PowerPC platforms, resulting in ‘similar matching speed to DFA based engines (like re2) on common patterns’ with Perl compatibility. ‘This work has been released as part of PCRE 8.20 and above. Now (PCRE 8.31), nearly all PCRE features are supported including UTF-8/16 and partial matching.’

(tags: pcre regexps regex performance optimization jit compilation dfa re2 via:akohli)

Spanner: Google’s Globally-Distributed Database [PDF]

Abstract: Spanner is Google’s scalable, multi-version, globally-distributed, and synchronously-replicated database. It is the first system to distribute data at global scale and support externally-consistent distributed transactions. This paper describes how Spanner is structured, its feature set, the rationale underlying various design decisions, and a novel time API that exposes clock uncertainty. This API and its implementation are critical to supporting external consistency and a variety of powerful features: non-blocking reads in the past, lock-free read-only transactions, and atomic schema changes, across all of Spanner. To appear in: OSDI’12: Tenth Symposium on Operating System Design and Implementation, Hollywood, CA, October, 2012.

(tags: database distributed google papers toread pdf scalability distcomp transactions cap consistency)

NCBI ROFL: Probably the most horrifying scientific lecture ever

In 1983, at the Urodynamics Society meeting in Las Vegas, Professor G.S. Brindley first announced to the world his experiments on self-injection with papaverine to induce a penile erection. This was the first time that an effective medical therapy for erectile dysfunction (ED) was described, and was a historic development in the management of ED. The way in which this information was first reported was completely unique and memorable, and provides an interesting context for the development of therapies for ED. I was present at this extraordinary lecture, and the details are worth sharing. Although this lecture was given more than 20 years ago, the details have remained fresh in my mind, for reasons which will become obvious.

(tags: medicine science funny erectile-dysfunction omgwtf conferences)

Yuri Suzuki: London Underground circuit map radio

Japanese designer yuri suzuki has sent designboom images of his ‘london underground circuit maps’ project developed as part of the designers in residence program at the london design museum, on show until january 13th, 2013. responding to ‘thrift’ as a theme, suzuki’s work explores communication systems in consumer electronics. a printed circuit board (PCB) is used as a precedent for developing a electrical circuit influenced by harry beck’s iconic london underground map diagrams. by strategically positioning certain speaker, resistor and battery components throughout the map, users can visually understand the complex networks associated with electricity and how power is generated within a radio.

(tags: electronics london art design underground travel yuri-suzuki circuitry)

The meanings and origins of ‘feck’

It’s a “minced oath”, apparently:

‘Feck is a popular minced oath in Ireland, occupying ground between the ultra-mild expletive flip and the often taboo (but also popular) fuck. It’s strongly associated with Irish speech, and serves a broad range of linguistic purposes that I’ll address briefly in this post.’

It doesn’t derive from the obvious source:

So where does the curse, the not-quite-rude word, come from? It’s commonly assumed to stem from its coarser cousin fuck, the simple vowel change undercutting its power and making it more suitable for public expression. But Julian Walker, an educator at the British Library, offers a more roundabout route: “In faith” becomes the improbable “in faith’s kin” shortened to “i’fackins”, which gradually shrinks to “fac” and “feck”.

(tags: feck swearing ireland irish hiberno-english father-ted etymology cursing)

SnapTree benchmarks

nice concurrent Map data structure for the JVM; beats out ConcurrentHashMap, ConcurrentLinkedHashMap from guava, ConcurrentSkipListMap under both CMS and G1 garbage collectors.

(tags: concurrency benchmarks hashmap map data-structures java jvm snaptree)

Chip and Skim: cloning EMV cards with the pre-play attack

Worrying stuff from the LBT team. ATM RNGs are predictable, and can be spoofed by intermediate parties:

‘So far we have performed more than 1000 transactions at more than 20 ATMs and a number of POS terminals, and are collating a data set for statistical analysis. We have developed a passive transaction logger which can be integrated into the substrate of a real bank card, which records up to 100 unpredictable numbers in its EEPROM. Our analysis is ongoing but so far we have established non-uniformity of unpredictable numbers in half of the ATMs we have looked at. First, there is an easier attack than predicting the RNG. Since the unpredictable number is generated by the terminal but the relying party is the issuing bank, any intermediate party – from POS terminal software, to payment switches, or a middleman on the phone line – can intercept and superimpose their own choice of UN. Attacks such as those of Nohl and Roth, and MWR Labs show that POS terminals can be remotely hacked simply by inserting a sabotaged smartcard into the terminal.

(tags: atm banking security attack prngs spoofing banks chip-and-pin emv smartcards)

New UK Conservative Party Co-Chair Grant Shapps Founded Google Spamming Business

Wow. Scummy stuff.

Shapps founded HowToCorp in 2005, a site that, among other products, pitches the TrafficPaymaster software. The software apparently “scrapes” or copies content from all over the web, from RSS feeds to even sets of search results, to automatically generate pages that probably make little sense to the human visitor but which may pick up some traffic from Google and, in turn, generate clicks on Google AdSense or other ads.

(tags: grant-shapps uk politics tories spammers spamming spinning adsense google spam trafficpaymaster)

NunatsiaqOnline 2012-09-06: The First Non-Inuk on the Moon

No, I am not a conspiracy theorist who believes that Armstrong’s moon landing was faked at some mysterious location in the Nevada desert. Armstrong reached the moon. But his accolades are undeserved because he was not first. All right-thinking Nunavummiut know this, because we know that Inuit regularly visited the moon for centuries. David Iqaqrialu said as much in a heated exchange in the Nunavut legislature on May 6, 2002. We know it was heated because he prefaced his remarks by telling the Speaker, “I am starting to get hot under the collar…” He then went on to say, as reported in Hansard, “…it is not really related to the question that I posed, but this is background material. Inuit had reached the moon quite some time ago during the shamanistic ages, prior to the Americans reaching it with their machines and finding out it wasn’t what they thought it was.”

(tags: inuit via:daev shaman nunavut neil-armstrong moon space exploration)

Dublin City contact numbers for potholes, dangerous drivers, illegal parking etc.

I’m sure these are about as useful as a chocolate teapot, but what the hey

(tags: dublin parking cycling roads safety potholes reporting)

Knots on Mars! (and a few thoughts on NASA’s knots)

amazing post from the International Guild of Knot Tyers Forum:

While a few of the folks here are no doubt aware, it might surprise most people to learn that knots tied in cords and thin ribbons have probably traveled on every interplanetary mission ever flown. If human civilization ends tomorrow, interplanetary landers, orbiters, and deep space probes will preserve evidence of both the oldest and newest of human technologies for millions of years. Knots are still used in this high-tech arena because cable lacing has long been the preferred cable management technique in aerospace applications. That it remains so to this day is a testament to the effectiveness of properly chosen knots tied by skilled craftspeople. It also no doubt has a bit to do with the conservative nature of aerospace design and engineering practices. Proven technologies are rarely cast aside unless they no longer fulfill requirements or there is something substantially better available. While the knots used for cable lacing in general can be quite varied — in some cases even a bit idiosyncratic — NASA has in-house standards for the knots and methods used on their spacecraft. These are specified in NASA Technical Standard NASA-STD-8739.4 — Crimping, Interconnecting Cables, Harnesses, and Wiring. As far as I’ve been able to identify in the rover images below, all of the lacings shown are one of two of the several patterns specified in the standard. The above illustration shows the so-called “Spot Tie”. It is a clove hitch topped by two half-knots in the form of a reef (square) knot. In addition to its pure binding role, it is also used to affix cable bundles to tie-down point.

(tags: via:fanf mars nasa science knots tying rope cables cabling geek aerospace standards)

Estonia introduces coding classes to 8-year-olds

‘ProgreTiiger education will start with students in the first grade, which starts around the age of 7 or 8 for Estonians. The compsci education will continue through a student’s final years of public school, around age 16. Teachers are being trained on the new skills, and private sector IT companies are also getting involved, which makes sense, given that these entities will likely end up being the long-term beneficiaries of a technologically literate populace. The ProgreTiiger program is launching at a few pilot schools and will soon be rolling out to all general education schools in Estonia.’

(tags: estonia education coding programming kids children students learning school)

Avoiding Hash Lookups in a Ruby Implementation

‘If I were to sum up the past 6 years I’ve spent optimizing JRuby it would be with the following phrase: Get Rid Of Hash Lookups.’ This has been a particular theme of some recent optimization hacks I’ve been working on. Hashes may be O(1) to read, on average, but that doesn’t necessarily mean they’re the right tool for performance… (via Declan McGrath)

(tags: via:declanmcgrath hash optimization ruby performance jruby hashing data-structures big-o optimisation)

River Poddle underneath the city of Dublin’s streets

Rarely-seen pictures of Dublin’s underground river which runs beneath Dublin Castle. I wonder if these are what those blokes spotted entering the drains were up to

(tags: hidden-dublin ireland dublin history poddle rivers waterways subterrainean)

Striped (Guava: Google Core Libraries for Java 13.0.1 API)

Nice piece of Guava concurrency infrastructure in the latest release:

A striped Lock/Semaphore/ReadWriteLock. This offers the underlying lock striping similar to that of ConcurrentHashMap in a reusable form, and extends it for semaphores and read-write locks. Conceptually, lock striping is the technique of dividing a lock into many stripes, increasing the granularity of a single lock and allowing independent operations to lock different stripes and proceed concurrently, instead of creating contention for a single lock.
The guarantee provided by this class is that equal keys lead to the same lock (or semaphore), i.e. if (key1.equals(key2)) then striped.get(key1) == striped.get(key2) (assuming Object.hashCode() is correctly implemented for the keys). Note that if key1 is not equal to key2, it is not guaranteed that striped.get(key1) != striped.get(key2); the elements might nevertheless be mapped to the same lock. The lower the number of stripes, the higher the probability of this happening.
Prior to this class, one might be tempted to use Map, where K represents the task. This maximizes concurrency by having each unique key mapped to a unique lock, but also maximizes memory footprint. On the other extreme, one could use a single lock for all tasks, which minimizes memory footprint but also minimizes concurrency. Instead of choosing either of these extremes, Striped allows the user to trade between required concurrency and memory footprint. For example, if a set of tasks are CPU-bound, one could easily create a very compact Striped of availableProcessors() * 4 stripes, instead of possibly thousands of locks which could be created in a Map structure.

(tags: locking concurrency java guava semaphores coding via:twitter)

HotSpot JVM garbage collection options cheat sheet (v2)

‘In this article I have collected a list of options related to GC tuning in JVM. This is not a comprehensive list, I have only collected options which I use in practice (or at least understand why I may want to use them). Compared to previous version a few useful diagnostic options was added. Additionally section for G1 specific options was introduced.’

(tags: hotspot jvm coding gc java performance)

Martin “Disruptor” Thompson’s Single Writer Principle

Contains these millisecond estimates for highly-contended inter-thread signalling when incrementing a 64-bit counter in java:

One Thread300
One Thread with Memory Barrier4,700
One Thread with CAS5,700
Two Threads with CAS18,000
One Thread with Lock10,000
Two Threads with Lock118,000

(tags: java jvm performance coding concurrency threading cas locking)

Locks & Condition Variables – Latency Impact

Firstly, this is 3 orders of magnitude greater latency than what I illustrated in the previous article using just memory barriers to signal between threads. This cost comes about because the kernel needs to get involved to arbitrate between the threads for the lock, and then manage the scheduling for the threads to awaken when the condition is signalled. The one-way latency to signal a change is pretty much the same as what is considered current state of the art for network hops between nodes via a switch. It is possible to get ~1µs latency with InfiniBand and less than 5µs with 10GigE and user-space IP stacks. Secondly, the impact is clear when letting the OS choose what CPUs the threads get scheduled on rather than pinning them manually. I’ve observed this same issue across many use cases whereby Linux, in default configuration for its scheduler, will greatly impact the performance of a low-latency system by scheduling threads on different cores resulting in cache pollution. Windows by default seems to make a better job of this.

(tags: locking concurrency java jvm signalling locks linux threading)

Evolution of SoundCloud’s Architecture

nice write-up. nginx, Rails, RabbitMQ, MySQL, Cassandra, Elastic Search, HAProxy

(tags: soundcloud webdev architecture scaling scalability)

What Happens to Stolen Bicycles?

‘Bike thievery is essentially a risk-free crime. If you were a criminal, that might just strike your fancy. If Goldman Sachs didn’t have more profitable market inefficencies to exploit, they might be out there arbitraging stolen bikes.’ Good summary, and I suspect a lot applies in Dublin too — flea markets and vanloads of stolen bikes being sent to other cities for reselling.

(tags: via:hn economics crime bikes theft goldman-sachs)

The urban SUVs of bicycles

Great (Dublin-focused) writeup on cargo bikes

(tags: cargo-bikes cycling commute kids dutch-bikes bikes)

1024cores

Some good algorithms and notes by Dmitry Vyukov on ‘lockfree, waitfree, obstruction-free synchronization algorithms and data structures, scalability-oriented architecture, multicore/multiprocessor design patterns, high-performance computing, threading technologies and libraries (OpenMP, TBB, PPL), message-passing systems and related topics.’ The catalog of lock-free queue implementations is particularly extensive (via Sergio Bossa)

(tags: algorithms concurrency articles dmitry-vyukov go c++ coding via:sergio-bossa)

Sting op exposes Andrews over FF Twitter rants – National News – Independent.ie

Incredible sting op uncovers the real identity of an anonymous Twitter account posting Fianna Fail gossip:

He discovered that each tweet had originated from the Twitter web interface, meaning it had been posted from a web browser on a computer, rather than sent from a mobile phone or other portable device. Based on the times that tweets were posted by @brianformerff, he deduced that the Tweets were being posted while the user was on a work break, using a company computer or an internet cafe. The next stage in the hunt was uncovering the IP address of the computer where the tweets originated. “I created my own web redirection service which would allow me to take links to articles of interest, for example in the Irish Times, and then transform them into short links that would pass through a redirection server I controlled. In this way, if someone read the tweets and clicked on the link, I would be able to establish the IP address of the computer that was being used at the time.” The author created a new twitter account, @john_cant _type, based on the persona of a politics student based in Kildare. He started sending several messages and tweets to “brian” and other users to establish himself as a genuine twitter user. Eventually @brianformerff responded to a post from @john_cant_type to a link to an article at Silicon Republic. The bait was taken and the IP address was tracked to an internet cafe, Amazon cyber/net Rathmines which offers web access “at the very reasonable rate of €1/hour”. What happened next descended almost into the realms of farce. The author waited for tweets from @brianformerff and then rushed to the internet cafe to try and catch Chris Andrews. Eventually the plan worked and the author used photography and video surveillance, even taking covert photographs of tweets as they were being posted in the internet cafe by Chris Andrews and analysing if the word count and structure matched the tweets appearing in cyberspace under the tag @brianformerff.

(tags: chris-andrews twitter surveillance privacy anonymity politics ireland fianna-fail)

Rootbeer

The Rootbeer GPU Compiler makes it easy to use Graphics Processing Units from within Java. Rootbeer is more advanced that CUDA or OpenCL Java Language Bindings. With bindings the developer must serialize complex graphs of objects into arrays of primitive types. With Rootbeer this is done automatically. Also with language bindings, the developer must write the GPU kernel in CUDA or OpenCL. With Rootbeer a static analysis of the Java Bytecode is done (using Soot) and CUDA code is automatically generated. […] All of the familar Java code you have been writing can be executed on the GPU.

(tags: gpu java coding cuda compiler)

“In Which The Irish Invent Twitter in 1984”

A fascinating story of 1980s tech history — ‘The initial Text Tell PX-1000 was developed by Text Lite Ltd. in Ireland in the early 1980s, probably in 1983. It allowed people to create simple text messages and send them by phone anywhere in the world. It had a built-in memory that could hold up to 7400 characters. The firmware inside the PX-1000 was written by West-Tec Ltd. in Ireland, who were probably also the hardware manufacturers. [… A later version was] the Philips version of the PX-1000Cr, as it features advanced cryptographic capabilities. It was intended for small companies and journalists, and was also used by the Dutch Government. […] it played an important role in the fight for Nelson Mandela’s release from prison.’

(tags: nelson-mandela ireland history crypto texting text-lite 1980s philips)

French illegal downloads agency Hadopi may be abolished

According to recent statistics, Hadopi has sent 1 million warning emails, 99,000 “strike two” letters and identified 314 people for referral to the courts for possible disconnection. No one has actually been disconnected. According to Aurelie Filipetti, culture minister in the new French Government, Hadopi has been nothing but a waste of money. “€12 million per year and 60 officials; that’s an expensive way to send 1 million emails,” Filipetti said. “Hadopi has not fulfilled its mission of developing legal downloads. I prefer to reduce the funding of things that have not been proven to be useful.”

(tags: hadopi privacy law three-strikes france money)

NASA’s Mars Rover Crashed Into a DMCA Takedown

An hour or so after Curiosity’s 1.31 a.m. EST landing in Gale Crater, I noticed that the space agency’s main YouTube channel had posted a 13-minute excerpt of the stream. Its title was in an uncharacteristic but completely justified all caps: “NASA LANDS CAR-SIZE ROVER BESIDE MARTIAN MOUNTAIN.” When I returned to the page ten minutes later, […] the video was gone, replaced with an alien message: “This video contains content from Scripps Local News, who has blocked it on copyright grounds. Sorry about that.” That is to say, a NASA-made public domain video posted on NASA’s official YouTube channel, documenting the landing of a $2.5 billion Mars rover mission paid for with public taxpayer money, was blocked by YouTube because of a copyright claim by a private news service.

(tags: dmca google fail nasa copyright false-positives scripps youtube video mars)

High-frequency trading: The fast and the furious | The Economist

“The NYMEX panel found that Infinium had finished writing the algorithm only the day before it introduced it to the market, and had tested it for only a couple of hours in a simulated trading environment to see how it would perform. The firm’s normal testing processes take six to eight weeks. When the algorithm started its frenetic buying spree, the measures designed to shut it down automatically did not work. One was supposed to turn the system off if a maximum order size was breached, but because the machine was placing lots of small orders rather than a single big one the shutdown was not triggered. The other measure was meant to prevent Infinium from selling or buying more than a certain number of contracts, but because of an error in the way the rogue algorithm had been written, this, too, failed to spot a problem.”

(tags: hft automation trading markets stocks nymex bugs software)

Lessons in website security anti-patterns by Tesco : Troy Hunt, an Aussie software architect working on a .Net security product called ASafaWeb, does a great job extensively deconstructing Tesco’s appalling website security on their shopping site. In the process, he gets this wonderful tweet from their customer-care account: “@troyhunt Let me assure you that all customer passwords are stored securely & in line with industry standards across online retailers.” As he says, this is a clear demonstration that Tesco is in the first stage of the four stages of competence — “unconscious incompetence”: “The individual does not understand or know how to do something and does not necessarily recognise the deficit.” ( http://en.wikipedia.org/wiki/Four_stages_of_competence )
(tags: tesco security passwords web http https ssl funny dot-net shopping uk customer-care)

Accident: Ryanair B738 and American B763 at Barcelona on Apr 14th 2011 : An accident report concerning a Ryanair flight.

An American Airlines Boeing 767-300, registration N366AA performing flight AA-67 from Barcelona,SP (Spain) to New York JFK, NY (USA), had taxied to the holding point runway 25L and was holding short of the runway. A Ryanair Boeing 737-800, registration EI-EKB performing flight FR-8136 from Barcelona,SP (Spain) to Ibiza,SP (Spain) with 169 passengers and 6 crew, was taxiing along Barcelona’s taxiway K for departure from runway 25L and was maneouvering to pass behind the Boeing 767-300. A number of passengers on board of the Boeing 737-800 observed the right hand wing of the aircraft contact the tailplane of the Boeing 767-300 and rose out of their seats attracting the attention of a flight attendant. A passenger told the flight attendant, that their aircraft had hit the aircraft besides them. The flight attendant contacted the purser, who instructed her to contact the flight deck, she contacted the flight deck and informed the captain that passengers had seen their aircraft had hit another aircraft. The captain responded however everything was fine and she continued with the takeoff about 2 minutes after the Boeing 767. Immediately after departure the passengers insisted the flight was not safe and they had collided with another aircraft, one of the passengers identified himself as an engineer. The flight attendant told the engineer that the captain had been informed and had told everything was fine. No further information was forwarded to the flight deck. After landing in Ibiza, while disembarking, the passengers again spoke up claiming the flight had been unsafe. During the turnaround the flight attendant informed the purser that one of the passengers observing the collision was an engineer. Neither approached the flight crew however. Following the return flight FR-8137 the purser talked to the captain and informed her that one of the passengers observing the collision was an engineer. In the following it was identified that the right hand winglet of the Boeing 737-800 had received damage, the Boeing 767-300 was found with damage to the left hand stabilizer following landing in New York.

CIAIAC report : The official report on that Ryanair/AA collision in Barcelona in July 2011, on pages 211-255.
(tags: collisions safety travel air ryanair)

Practical machine learning tricks from the KDD 2011 best industry paper : Wow, this is a fantastic paper. It’s a Google paper on detecting scam/spam ads using machine learning — but not just that, it’s how to build out such a classifier to production scale, and make it operationally resilient, and, indeed, operable. I’ve come across a few of these ideas before, and I’m happy to say I might have reinvented a few (particularly around the feature space), but all of them together make extremely good sense. If I wind up working on large-scale classification again, this is the first paper I’ll go back to. Great info! (via Toby diPasquale.)
(tags: classification via:codeslinger training machine-learning google ops kdd best-practices anti-spam classifiers ensemble map-reduce)

The world’s first 3D-printed gun : I wasn’t expecting to see this for a few years. The future is ahead of schedule!

A .22-caliber pistol, formed from a 3D-printed AR-15 (M16) lower receiver, and a normal, commercial upper. In other words, the main body of the gun is plastic, while the chamber — where the bullets are actually struck — is solid metal. […] While this pistol obviously wasn’t created from scratch using a 3D printer, the interesting thing is that the lower receiver — in a legal sense at least — is what actually constitutes a firearm. Without a lower receiver, the gun would not work; thus, the receiver is the actual legally-controlled part. In short, this means that people without gun licenses — or people who have had their licenses revoked — could print their own lower receiver and build a complete, off-the-books gun. What a chilling thought.

“Are You Human?” urban intervention, 2009 : turn CAPTCHAs into cut-outs, mount them in the urban environment, and they blend into the tag landscape. This came up after contemplating “artisanal integers”, and the concept of taking something digital and ephemeral and making a hand-made, long-lived physical artifact from it. (via ted byfield)
(tags: art sculpture captchas physical artifacts tags graffiti human urban)

Brooklyn Integers | Integers as a service : Integers artisanally hand-crafted for you. See also the sister site, missionintegers.com: “Each of our bespoke numbers is created just for you in San Francisco’s historic Mission District. What will you use it for? A letter-pressed receipt. A special touch of latte art. A globally-unique user ID. A woolen hat. The possibilities are as infinite as the space of 64-bit unsigned ints.” (via John Allspaw)
(tags: via:allspaw humour funny integers artisan satire hand-crafted)

This park’s life – The Irish Times – Thu, Jul 26, 2012 : Great article about Dublin’s Phoenix Park, Europe’s largest enclosed urban park (more than twice the size of New York’s Central Park, in fact). Now that I have two little kids, I’ve been spending a good portion of my weekends there — it’s a wonderful thing to have on our doorstep. Also:

The park even breeds celebrities. “The lion that roars at the start of the MGM movies. He’s a Dub. He was born in Dublin Zoo.”

Universal properties of mythological networks – Abstract – EPL (Europhysics Letters) – IOPscience : Abstract:

As in statistical physics, the concept of universality plays an important, albeit qualitative, role in the field of comparative mythology. Here we apply statistical mechanical tools to analyse the networks underlying three iconic mythological narratives with a view to identifying common and distinguishing quantitative features. Of the three narratives, an Anglo-Saxon and a Greek text are mostly believed by antiquarians to be partly historically based while the third, an Irish epic [jm: “An Táin Bó Cúailnge”, The Tain, to be specific], is often considered to be fictional. Here we use network analysis in an attempt to discriminate real from imaginary social networks and place mythological narratives on the spectrum between them. This suggests that the perceived artificiality of the Irish narrative can be traced back to anomalous features associated with six characters. Speculating that these are amalgams of several entities or proxies, renders the plausibility of the Irish text comparable to the others from a network-theoretic point of view.

The society in the 1st century story of the Táin Bó Cúailnge looked artificial at first analysis of the networks between 404 characters in the story. However, the researchers found the society reflected real rather than fictional networks when the weakest links to six of the characters are removed. These six characters included Medb, Queen of Connacht; Conchobor, King of Ulster and Cúchulainn. They were “similar to superheroes of the Marvel universe” and are “too superhuman” or too well-connected to be real, researchers said. The researchers suggest that each of these superhuman characters may be an amalgam of many which became fused and exaggerated as the story was passed down orally through generations.

Irish campsite recommendations : the conclusion of a Twitter/Facebook recommendations-gathering exercise; winners seem to be Lough Key Forest Park, Renvyle Beach, Fintra, Eagle Point, and Hidden Valley
(tags: camping ireland tips recommendations caravan holidays vacation)

CloudBurst : ‘Highly Sensitive Short Read Mapping with MapReduce’. current state of the art in DNA sequence read-mapping algorithms.

CloudBurst uses well-known seed-and-extend algorithms to map reads to a reference genome. It can map reads with any number of differences or mismatches. [..] Given an exact seed, CloudBurst attempts to extend the alignment into an end-to-end alignment with at most k mismatches or differences by either counting mismatches of the two sequences, or with a dynamic programming algorithm to allow for gaps. CloudBurst uses [Hadoop] to catalog and extend the seeds. In the map phase, the map function emits all length-s k-mers from the reference sequences, and all non-overlapping length-s kmers from the reads. In the shuffle phase, read and reference kmers are brought together. In the reduce phase, the seeds are extended into end-to-end alignments. The power of MapReduce and CloudBurst is the map and reduce functions run in parallel over dozens or hundreds of processors.

Expensive lessons in Python performance tuning : some good advice for large-scale Python performance: prun and guppy for profiling, namedtuples for memory efficiency, and picloud for trivial EC2-based scale-out. (via Nelson)
(tags: picloud prun guppy namedtuples python optimization performance tuning profiling)

On Patents : Notch comes up with a perfect analogy for software patents.

I am mostly fine with the concept of “selling stuff you made”, so I’m also against copyright infringement. I don’t think it’s quite as bad as theft, and I’m not sure it’s good for society that some professions can get paid over and over long after they did the work (say, in the case of a game developer), whereas others need to perform the job over and over to get paid (say, in the case of a hairdresser or a lawyer). But yeah, “selling stuff you made” is good. But there is no way in hell you can convince me that it’s beneficial for society to not share ideas. Ideas are free. They improve on old things, make them better, and this results in all of society being better. Sharing ideas is how we improve. A common argument for patents is that inventors won’t invent unless they can protect their ideas. The problem with this argument is that patents apply even if the infringer came up with the idea independently. If the idea is that easy to think of, why do we need to reward the person who happened to be first?

Marsh’s Library : Dublin museum of antiquarian books, open to the public — well worth a visit, apparently (I will definitely be making my way there soon I suspect), to check out their new “Marvels of Science” exhibit. Not only that though, but they have a beautiful website with some great photos — exemplary
(tags: museum dublin ireland libraries books science)

‘Poisoning Attacks against Support Vector Machines’, Battista Biggio, Blaine Nelson, Pavel Laskov : The perils of auto-training SVMs on unvetted input.

We investigate a family of poisoning attacks against Support Vector Machines (SVM). Such attacks inject specially crafted training data that increases the SVM’s test error. Central to the motivation for these attacks is the fact that most learning algorithms assume that their training data comes from a natural or well-behaved distribution. However, this assumption does not generally hold in security-sensitive settings. As we demonstrate, an intelligent adversary can, to some extent, predict the change of the SVM’s decision function due to malicious input and use this ability to construct malicious data. The proposed attack uses a gradient ascent strategy in which the gradient is computed based on properties of the SVM’s optimal solution. This method can be kernelized and enables the attack to be constructed in the input space even for non-linear kernels. We experimentally demonstrate that our gradient ascent procedure reliably identifies good local maxima of the non-convex validation error surface, which significantly increases the classifier’s test error.

C500k in Action at Urban Airship : I missed this back in 2010; 500k active TCP connections to a single EC2 large instance using Java and NIO
(tags: c10k java linux ec2 scaling nio netty urban-airship)

GraphChi : “big data, small machine” — perform computation on very large graphs using an algorithm they’re calling Parallel Sliding Windows. similar to Google’s Pregel, apparently
(tags: graphs graphchi big-data algorithms parallel)

High performance network programming on the JVM, OSCON 2012 : by Erik Onnen of Urban Airship. very good presentation on the current state of the art in large-scale low-latency service operation using the JVM on Linux. Lots of good details on async vs sync, HTTPS/TLS/TCP tuning, etc.
(tags: http https scaling jvm async sync oscon presentations tcp)

udev – How do I change the keymap of a single device (Logitech presenter)? – Ask Ubuntu : nifty, udev lets you remap USB “keyboard” gadgets randomly. this may be handy for my new cheapo USB foot pedals
(tags: pedals usb hid interface gadgets remapping ubuntu linux udev)

Science funding doesn’t add up – The Irish Times : ‘[Science Foundation Ireland] said it was continuing to support basic research, but there are a number of leading scientists here who were refused funding despite having qualified for it in the past. Dr Mike Peardon of the School of Mathematics was recently been turned down, having been “administratively withdrawn”. This means the application for funding was rejected at the first post during initial consideration and before it had a chance to be assessed by external experts. Several others in his department suffered a similar fate. “The school of mathematics at Trinity is ranked the 15th best maths department in the world and now we are not fundable by Science Foundation Ireland,” he said. “The cases I heard of have all been in pure maths,” said Prof Lorraine Hanlon in UCD’s school of physics. “All reported that the people in pure maths were returned unreviewed.” She believes other areas may also come under pressure. “Pure maths is the thin end of the wedge. The Government says mathematics is fundamental, but on the other side says we dont really care enough to support it. That is a schizophrenic approach,” she said.’
(tags: mathematics ireland science research academia funding tcd ucd sfi)

Microsoft’s ill-chosen magic constants : ‘Paolo Bonzini noticed something a little awkward in the Linux kernel support code for Microsoft’s HyperV virtualisation environment – specifically, that the magic constant passed through to the hypervisor was “0xB16B00B5”, or, in English, “BIG BOOBS”. It turns out that this isn’t an exception – when the code was originally submitted it also contained “0x0B00B135”.’ me, I prefer my magic constants less offensive and more Subgenius-oriented: “0xB0BD0BB5”
(tags: constants via:kevin-lyda oh-dear microsoft fail magic-numbers boobs linux kernel)

Scaling lessons learned at Dropbox : website-scaling tips and suggestions, “particularly for a resource-constrained, fast-growing environment that can’t always afford to do things “the right way” (i.e., any real-world engineering project”. I really like the “run with fake load” trick; add additional queries/load which you can quickly turn off if the service starts browning out, giving you a few days breathing room to find a real fix before customers start being affected. Neat
(tags: dropbox scalability webdev load scaling-up)

Ansible : ‘SSH-Based Configuration Management & Deployment’. deploy via SSH; no target-side daemons required. GPLv3 licensed, unfortunately :(
(tags: ansible devops configuration deployment sysadmin python ssh)