Skip to content

Category: Uncategorized

Links for 2013-01-09

Published January 9, 2013

  • Requests: HTTP for Humans

    'an elegant and simple HTTP library for Python, built for human beings.' 'Requests is an Apache2 Licensed HTTP library, written in Python, for human beings. Python’s standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks. Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests.'

    (tags: python http urllib libraries requests via:mikeste)

  • Surprisingly Good Evidence That Real Name Policies Fail To Improve Comments

    'Enough theorizing, there’s actually good evidence to inform the debate. For 4 years, Koreans enacted increasingly stiff real-name commenting laws, first for political websites in 2003, then for all websites receiving more than 300,000 viewers in 2007, and was finally tightened to 100,000 viewers a year later after online slander was cited in the suicide of a national figure. The policy, however, was ditched shortly after a Korean Communications Commission study found that it only decreased malicious comments by 0.9%. Korean sites were also inundated by hackers, presumably after valuable identities. Further analysis by Carnegie Mellon’s Daegon Cho and Alessandro Acquisti, found that the policy actually increased the frequency of expletives in comments for some user demographics. While the policy reduced swearing and “anti-normative” behavior at the aggregate level by as much as 30%, individual users were not dismayed. “Light users”, who posted 1 or 2 comments, were most affected by the law, but “heavy” ones (11-16+ comments) didn’t seem to mind. Given that the Commission estimates that only 13% of comments are malicious, a mere 30% reduction only seems to clean up the muddied waters of comment systems a depressingly negligent amount. The finding isn’t surprising: social science researchers have long known that participants eventually begin to ignore cameras video taping their behavior. In other words, the presence of some phantom judgmental audience doesn’t seem to make us better versions of ourselves.' (via Ronan Lyons)

    (tags: anonymity identity policy comments privacy politics new-media via:ronanlyons)

Links for 2013-01-08

Published January 8, 2013

  • HAT-trie: A Cache-conscious Trie-based Data Structure for Strings [PDF]

    'Tries are the fastest tree-based data structures for managing strings in-memory, but are space-intensive. The burst-trie is almost as fast but reduces space by collapsing trie-chains into buckets. This is not however, a cache-conscious approach and can lead to poor performance on current processors. In this paper, we introduce the HAT-trie, a cache-conscious trie-based data structure that is formed by carefully combining existing components. We evaluate performance using several real-world datasets and against other highperformance data structures. We show strong improvements in both time and space; in most cases approaching that of the cache-conscious hash table. Our HAT-trie is shown to be the most e?cient trie-based data structure for managing variable-length strings in-memory while maintaining sort order.' (via Tony Finch)

    (tags: via:fanf data-structures tries cache-aware trees)

  • The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases [PDF]

    'Main memory capacities have grown up to a point where most databases ?t into RAM. For main-memory database systems, index structure performance is a critical bottleneck. Traditional in-memory data structures like balanced binary search trees are not ef?cient on modern hardware, because they do not optimally utilize on-CPU caches. Hash tables, also often used for main-memory indexes, are fast but only support point queries. To overcome these shortcomings, we present ART, an adaptive radix tree (trie) for ef?cient indexing in main memory. Its lookup performance surpasses highly tuned, read-only search trees, while supporting very ef?cient insertions and deletions as well. At the same time, ART is very space ef?cient and solves the problem of excessive worst-case space consumption, which plagues most radix trees, by adaptively choosing compact and ef?cient data structures for internal nodes. Even though ART’s performance is comparable to hash tables, it maintains the data in sorted order, which enables additional operations like range scan and pre?x lookup.' (via Tony Finch)

    (tags: via:fanf data-structures trees indexing cache-aware tries)

  • Ef?cient In-Memory Indexing with Generalized Pre?x Trees [PDF]

    'Ef?cient data structures for in-memory indexing gain in importance due to (1) the exponentially increasing amount of data, (2) the growing main-memory capacity, and (3) the gap between main-memory and CPU speed. In consequence, there are high performance demands for in-memory data structures. Such index structures are used—with minor changes—as primary or secondary indices in almost every DBMS. Typically, tree-based or hash-based structures are used, while structures based on prefix-trees (tries) are neglected in this context. For tree-based and hash-based structures, the major disadvantages are inherently caused by the need for reorganization and key comparisons. In contrast, the major disadvantage of trie-based structures in terms of high memory consumption (created and accessed nodes) could be improved. In this paper, we argue for reconsidering pre?x trees as in-memory index structures and we present the generalized trie, which is a pre?x tree with variable prefix length for indexing arbitrary data types of fixed or variable length. The variable prefix length enables the adjustment of the trie height and its memory consumption. Further, we introduce concepts for reducing the number of created and accessed trie levels. This trie is order-preserving and has deterministic trie paths for keys, and hence, it does not require any dynamic reorganization or key comparisons. Finally, the generalized trie yields improvements compared to existing in-memory index structures, especially for skewed data. In conclusion, the generalized trie is applicable as general-purpose in-memory index structure in many different OLTP or hybrid (OLTP and OLAP) data management systems that require balanced read/write performance.' (via Tony Finch)

    (tags: via:fanf prefix-trees tries data-structures)

  • A Non-Blocking HashTable by Dr. Cliff Click : programming

    Proggit discovers the NonBlockingHashMap. This comment from Boundary's cscotta is particularly interesting: "The code is intricate and curiously-formatted, but NBHM is quite excellent. The majority of our analytics platform is backed by NBHMs updated rapidly in parallel. Cliff's a great, friendly, approachable guy; if you have any specific questions about the approaches or implementation, he may be happy to answer."

    (tags: data-structures algorithms non-blocking concurrency threading multicore cliff-click azul maps java boundary)

Links for 2013-01-07

Published January 7, 2013

Links for 2013-01-04

Published January 4, 2013

  • Dan McKinley :: Effective Web Experimentation as a Homo Narrans

    Good demo from Etsy's A/B testing, of how the human brain can retrofit a story onto statistically-insignificant results. To fix: 'avoid building tooling that enables fishing expeditions; limit our post-hoc rationalization by explicitly constraining it before the experiment. Whenever we test a feature on Etsy, we begin the process by identifying metrics that we believe will change if we 1) understand what is happening and 2) get the effect we desire.'

    (tags: testing etsy statistics a-b-testing fishing ulysses-contract brain experiments)

  • Lesser known crimes: do you own that copyright?

    A very interesting crime on the Irish statute books:

    Section 141 of the Copyright and Related Rights Act 2000 provides: A person who, for financial gain, makes a claim to enjoy a right under this Part [ie. copyright] which is, and which he or she knows or has reason to believe is, false, shall be guilty of an offence and shall be liable on conviction on indictment to a fine not exceeding £100,000, or to imprisonment for a term not exceeding 5 years, or both.

    (tags: ireland copyright ip false-claims law)

Links for 2013-01-02

Published January 2, 2013

  • Patent trolls want $1,000 for using scanners

    We are truly living in the future -- a dystopian future, but one nonetheless. A patent troll manages to obtain "gobbledigook" patents on using a scanner to scan to PDF, then attempts to shake down a bunch of small companies before eventually running into resistance, at which point it "forks" into a bunch of algorithmically-named shell companies, spammer-style, sending the same demands. Those demands in turn contain this beauty of Stockholm-syndrome-inducing prose:

    'You should know also that we have had a positive response from the business community to our licensing program. As you can imagine, most businesses, upon being informed that they are infringing someone’s patent rights, are interested in operating lawfully and taking a license promptly. Many companies have responded to this licensing program in such a manner. Their doing so has allowed us to determine that a fair price for a license negotiated in good faith and without the need for court action is a payment of $900 per employee. We trust that your organization will agree to conform your behavior to respect our patent rights by negotiating a license rather than continuing to accept the benefits of our patented technology without a license. Assuming this is the case, we are prepared to make this pricing available to you.'
    And here's an interesting bottom line:
    The best strategy for target companies? It may be to ignore the letters, at least for now. “Ignorance, surprisingly, works,” noted Prof. Chien in an e-mail exchange with Ars. Her study of startups targeted by patent trolls found that when confronted with a patent demand, 22 percent ignored it entirely. Compare that with the 35 percent that decided to fight back and 18 percent that folded. Ignoring the demand was the cheapest option ($3,000 on average) versus fighting in court, which was the most expensive ($870,000 on average). Another tactic that clearly has an effect: speaking out, even when done anonymously. It hardly seems a coincidence that the Project Paperless patents were handed off to a web of generic-sounding LLCs, with demand letters signed only by “The Licensing Team,” shortly after the “Stop Project Paperless” website went up. It suggests those behind such low-level licensing campaigns aren’t proud of their behavior. And rightly so.

    (tags: patents via:fanf networks printing printers scanning patent-trolls project-paperless adzpro gosnel faslan)

  • Keep predicting and you’ll be right eventually?

    debunking Ken Ring, the kiwi “long term weather prediction” “scientist” who gets trundled out every year around this time

    (tags: ken-ring weather predictions ireland rain)

Links for 2013-01-01

Published January 1, 2013

Links for 2012-12-18

Published December 18, 2012

  • Baklava code

    'thin software layers don’t add much value, especially when you have many such layers piled on each other. Each layer has to be pushed onto your mental stack as you dive into the code. Furthermore, the layers of phyllo dough are permeable, allowing the honey to soak through. But software abstractions are best when they don’t leak. When you pile layer on top of layer in software, the layers are bound to leak.'

    (tags: code design terminology food antipatterns)

Links for 2012-12-17

Published December 17, 2012

Links for 2012-12-16

Published December 16, 2012

Links for 2012-12-14

Published December 14, 2012

  • Authentication is machine learning

    This may be the most insightful writing about authentication in years:

    From my brief time at Google, my internship at Yahoo!, and conversations with other companies doing web authentication at scale, I’ve observed that as authentication systems develop they gradually merge with other abuse-fighting systems dealing with various forms of spam (email, account creation, link, etc.) and phishing. Authentication eventually loses its binary nature and becomes a fuzzy classification problem.

    This is not a new observation. It’s generally accepted for banking authentication and some researchers like Dinei Florêncio and Cormac Herley have made it for web passwords. Still, much of the security research community thinks of password authentication in a binary way [..]. Spam and phishing provide insightful examples: technical solutions (like Hashcash, DKIM signing, or EV certificates), have generally failed but in practice machine learning has greatly reduced these problems. The theory has largely held up that with enough data we can train reasonably effective classifiers to solve seemingly intractable problems.

    (via Tony Finch.)

    (tags: passwords authentication big-data machine-learning google abuse antispam dkim via:fanf)

  • Hotels to pay royalties on music - The Irish Times - Fri, Dec 14, 2012

    'The operators of hotels, guesthouses and bed & breakfasts will have to pay royalties for any copyright music played in guest bedrooms [in Ireland]. [...] Under the agreement, the music charges will be set by Phonographic Performance Ireland Ltd (PPI). [...] When it initiated its case in 2010, the PPI said it was seeking payment of about €1 per bedroom per week or about 14 cent a night.' I don't understand this. Most hotels do not play music in the rooms themselves. Does this apply if there is no music playing in the bedroom? Does it apply if the customer brings their own music? Are Dublin Bus to be next?

    (tags: hotels ppi ireland music money royalties)

  • The Mathematical Hacker

    'The trouble with the Lisp-hacker tradition is that it is overly focused on the problem of programming -- compilers, abstraction, editors, and so forth -- rather than the problems outside the programmer's cubicle. I conjecture that the Lisp-school essayists -- Raymond, Graham, and Yegge -- have not “needed mathematics” because they spend their time worrying about how to make code more abstract. This kind of thinking may lead to compact, powerful code bases, but in the language of economics, there is an opportunity cost.'

    (tags: mathematics coding maths essay hackers lisp fortran)

  • The Aggregate Magic Algorithms

    Obscure, low-level bit-twiddling tricks -- specifically:

    Absolute Value of a Float, Alignment of Pointers, Average of Integers, Bit Reversal, Comparison of Float Values, Comparison to Mask Conversion, Divide Rounding, Dual-Linked List with One Pointer Field, GPU Any, GPU SyncBlocks, Gray Code Conversion, Integer Constant Multiply, Integer Minimum or Maximum, Integer Power, Integer Selection, Is Power of 2, Leading Zero Count, Least Significant 1 Bit, Log2 of an Integer, Next Largest Power of 2, Most Significant 1 Bit, Natural Data Type Precision Conversions, Polynomials, Population Count (Ones Count), Shift-and-Add Optimization, Sign Extension, Swap Values Without a Temporary, SIMD Within A Register (SWAR) Operations, Trailing Zero Count.
    Many of these would be insane to use in anything other than the hottest of hot-spots, but good to have on file. (via Toby diPasquale)

    (tags: hot-spots optimisation bit-twiddling algorithms via:codeslinger snippets)

  • Shell Scripts Are Like Gremlins

    Shell Scripts are like Gremlins. You start out with one adorably cute shell script. You commented it and it does one thing really well. It’s easy to read, everyone can use it. It’s awesome! Then you accidentally spill some water on it, or feed it late one night and omgwtf is happening!?
    +1. I have to wean myself off the habit of automating with shell scripts where a clean, well-unit-tested piece of code would work better.

    (tags: shell-scripts scripting coding automation sysadmin devops chef deployment)

Links for 2012-12-13

Published December 13, 2012

Links for 2012-12-12

Published December 12, 2012

Links for 2012-12-11

Published December 11, 2012

  • Damn Fine Print

    lovely signed and editioned prints by Dublin's best illustrators at good prices. Turns out this was in connection with a show a few days ago, so the best ones are now sold out -- I love the Chris Judge Liberty Hall print -- but there's still a few good ones left. Brian Gallagher's Georgian doorway is a beauty.

    (tags: illustration dublin prints art chris-judge)

Links for 2012-12-10

Published December 10, 2012

  • A map of Dublin from 1686

    via Come Here To Me -- 'The whole population of the county at the time was under 60,000. Ringsend, Merrion, Monkstown, Bullock and Dalkey on the Southside and Ballybough, Clontarf, Sutton and Hoath/Howth on the Northside are marked. Taken from the book Dublin: through space and time (2001).'

    Massive tracts of land were reclaimed since then, clearly -- the North bay comes all the way in to Ballybough!

    (tags: via:chtm maps dublin ireland history)

  • Back-up Tut and other decoy spatial antiquities

    I like this idea -- a complete facsimile of King Tut's burial chamber. Bldgblog comments:

    “On the 90th anniversary of the discovery of King Tut’s tomb, an “authorized facsimile of the burial chamber” has been created, complete “with sarcophagus, sarcophagus lid and the missing fragment from the south wall.” The resulting duplicate, created with the help of high-res cameras and lasers, is “an exact facsimile of the burial chamber,” one that is now “being sent to Cairo by The Ministry of Tourism of Egypt.” [...]

    'Interestingly, we read that this was "done under a licence to the University of Basel," which implies the very real possibility that unlicensed duplicate rooms might also someday be produced—that is, pirate interiors ripped or printed from the original data set, like building-scale "physibles," a kind of infringed architecture of object torrents taking shape as inhabitable rooms.' [...]

    'In their book Anachronic Renaissance, for instance, Alexander Nagel and Christopher Wood write of what they call a long "chain of effective substitutions" or "effective surrogates for lost originals" that nonetheless reached the value and status of an icon in medieval Europe. "[O]ne might know that [these objects] were fabricated in the present or in the recent past," Nagel and Wood write, "but at the same time value them and use them as if they were very old things." They call this seeing in "substitutional terms".'

    (tags: via:new-aesthetic bldgblog archaeology facsimiles copying king-tut egypt history 3d-printing physibles)

Links for 2012-12-06

Published December 6, 2012

  • low-gc-membuffers

    "This project aims at creating a simple efficient building block for "Big Data" libraries, applications and frameworks; thing that can be used as an in-memory, bounded queue with opaque values (sequence of JDK primitive values): insertions at tail, removal from head, single entry peeks), and that has minimal garbage collection overhead. Insertions and removals are as individual entries, which are sub-sequences of the full buffer. GC overhead minimization is achieved by use of direct ByteBuffers (memory allocated outside of GC-prone heap); and bounded nature by only supporting storage of simple primitive value (byte, `long') sequences where size is explicitly known. Conceptually memory buffers are just simple circular buffers (ring buffers) that hold a sequence of primitive values, bit like arrays, but in a way that allows dynamic automatic resizings of the underlying storage. Library supports efficient reusing and sharing of underlying segments for sets of buffers, although for many use cases a single buffer suffices."

    (tags: gc java jvm bytebuffer)

Links for 2012-12-03

Published December 3, 2012

  • Scoop! The inside story of the news website that saved the BBC

    The Register's take on the early days of www.bbc.co.uk. Lots of politics, unsurprisingly.

    Fifteen years ago this month the BBC launched its News Online website. Developed internally with a skeleton team, the web service rapidly became the face of the BBC on the internet, and its biggest success story – winning four successive BAFTA awards. Remarkably, it operated at a third of the cost of rival commercial online news operations – unheard of in public-sector IT projects. Devised before there were really any content management systems, the technical architecture became a template for all major news systems, and one that’s still in use today. The team endured some furious internal politicking and sabotage to survive.

    (tags: bbc news history web uk the-register)

  • Irish mobile phone companies: still spammy

    'Pro tip: if you're going to spam, try not to spam the DPC's Director of Investigations.' -- lolz

    (tags: funny oh-dear three hutchinson ireland mobile spam dpc law)

  • Hamming weight

    Wikipedia page.

    The Hamming weight of a string is the number of symbols that are different from the zero-symbol of the alphabet used. It is thus equivalent to the Hamming distance from the all-zero string of the same length. For the most typical case, a string of bits, this is the number of 1's in the string. In this binary case, it is also called the population count, popcount or sideways sum. It is the digit sum of the binary representation of a given number.
    Contains an efficient algorithm to compute this for a given long value, by 'adding counts in a tree pattern.'

    (tags: algorithms hamming-distance bits hamming weight binary)

  • Efficient concurrent long set and map

    An ordered set and map data structure and algorithm for long keys and values, supporting concurrent reads by multiple threads and updates by a single thread.
    Some good stuff in the linked blog posts about Clojure's PersistentHashMap and PersistentVector data structures, too.

    (tags: arrays java tries data-structures persistent clojure concurrent set map)

Links for 2012-11-28

Published November 28, 2012

  • The Rise And Fall Of The Obscure Music Download Blog: A Roundtable

    One internet music "sharing" trend largely unnoticed by the powers that sue was the niche explosion of obscure music download blogs, lasting roughly from 2004-2008. Using free filesharing services like Rapidshare and Mediafire, and setting up sites on Blogspot and similar providers, these internet hubs stayed hidden in the open by catering to more discerning kleptomaniac audiophiles. Their specialty: parceling out ripped recordings — many of them copyrighted — from the more collectible and unknown corners of music's oddball, anomalous past. While the RIAA was suing dead people for downloading Michael Jackson songs (and Madonna was using Soulseek to curse at teenagers), obscure music blogs racked up millions of hits, ripping and sharing 80s Japanese noise, 70s German prog, 60s San Francisco hippie freak-outs, 50s John Cage bootlegs, 30s gramophone oddities, Norwegian death metal, cold wave cassettes made by kids in their garages, and the like. It was the mid aughts, and the advent of digitization had inadvertently put the value of the music industry's "Top Ten" commercial product in peril. That same process transformed the value of old, collectible music as well. If one smart record collector was able to share the entire contents—music, artwork and all—of one vinyl LP on his blog, for free, and upload another item from his 1,000+ collection the next day, for weeks and years, and others like him did the same, competing with each other about who could upload the rarest and most sought-after record, and anyone who downloaded it could then share it again and again… Suddenly everyone in the world had the coolest record collection in the world; and soon, nobody in the world had the coolest record collection in the world. Obscure music download blogs weren't shut down like Napster or Megaupload were (though they were indirectly affected by that crackdown); they just, mysteriously, seemed to burn out on their own sometime around 2008. While some are still around, their number represents only a fraction of that mid-00s heyday. Was this because obscure music blogs had overshared the underexposed and blown the whole thing into oblivion? Is the fact that a guy in Japan will no longer pay $500 on eBay for a first pressing of the No New York compilation because he can find it for free on the internet good for the world? Was the commodity-lost but the knowledge-gained an even exchange? To explore what was going on then, I assembled this email roundtable discussion between creators of some of the most popular blogs of the time: Eric Lumbleau of Mutant Sounds, Liam Elms of 8 Days in April, Frank of Systems of Romance and Brian Turner, Music Director of WFMU.
    (via Loreana Rushe)

    (tags: music mp3 blogs obscure via-loreana-rushe history 2000s)

Links for 2012-11-26

Published November 26, 2012

  • Conor’s 2012 Raspberry Pi Christmas Gift Guide

    Ah, memories! Wish my kiddies were old enough for one of these...

    I really think this Christmas could be a lovely replay of 1982 for a lot of people, like me, who got their first home computer that year. You could have so much fun on Christmas Day messing with the RPi rather than falling asleep in front of the fire. Just don’t fight over who gets the telly when Doctor Who is on. Whilst the bare-bones nature of the Raspberry Pi is wonderful, it is unusable out of the box unless you are a house with smartphones, digital cameras and existing PCs already that you can raid for components. What you want to avoid is a repeat of me that December in 1982 with my brand-new 16K ZX Spectrum which didn’t work on our Nordmende TV until two weeks later when the RTV Rentals guy came and replaced the TV Tuner. Two weeks typing Beep 1,2 to make sure it wasn’t broken.

    (tags: raspberry-pi gifts computers kids hacking education gadgets christmas)

  • Nintendo's work on Miiverse Penis Drawing Detection

    'The unique feature of the Miiverse is being able to send drawings, not just text. But since the advent of the internet, there have always been those who have used it for unsavory purposes.'
    'Motoyama: we never had such a problem with our Hatena services. But, when we brought Hatena Flipnote to the West, we were caught off-guard by the amount of penises drawn by people.
    Kurisu: So the team and I had to come up with a way to create a system that auto-detects those types of pictures. [...]
    'Motoyama: After a week, we made very good progress on the system. Then we tested the system with Nintendo of America and told them to start drawing. It went horribly.
    Kurisu: What we learned is that people enjoy drawing penises. Multiple ones. (laughs) The system was not prepared to handle that.'
    See also the "time-to-penis" metric in MMO games: http://www.joystiq.com/2009/03/24/overheard-gdc09-ttp-time-to-penis/

    (tags: nintendo image-detection ttp metrics games gaming mmo miiverse drawing)

  • The trench talk that is now entrenched in the English language

    'From cushy to crummy and blind spot to binge drink, a new study reveals the impact the First World War had on the English language and the words it introduced.' Incredible comments, too...

    (tags: english etymology history wwi great-war via:sinead-gleeson words language)

  • Special encoding of small aggregate data types in Redis

    Nice performance trick in Redis on hash storage: 'In theory in order to guarantee that we perform lookups in constant time (also known as O(1) in big O notation) there is the need to use a data structure with a constant time complexity in the average case, like an hash table. But many times hashes contain just a few fields. When hashes are small we can instead just encode them in an O(N) data structure, like a linear array with length-prefixed key value pairs. Since we do this only when N is small, the amortized time for HGET and HSET commands is still O(1): the hash will be converted into a real hash table as soon as the number of elements it contains will grow too much (you can configure the limit in redis.conf). This does not work well just from the point of view of time complexity, but also from the point of view of constant times, since a linear array of key value pairs happens to play very well with the CPU cache (it has a better cache locality than an hash table).'

    (tags: memory redis performance big-o hash-tables storage coding cache arrays)

  • HTTP Error 403: The service you requested is restricted - Vodafone Community

    Looks like Vodafone Ireland are failing to scale their censorware; clients on their network reporting "HTTP Error 403: The service you requested is restricted". According to a third-party site, this error is produced by the censorship software they use when it's insufficiently scaled for demand:

    "When you try to use HTTP Vodafone route a request to their authentication server to see if your account is allow to connect to the site. By default they block a list of adult/premium web sites (this is service you have switched on or off with your account). The problem is at busy times this validation service is overloaded and so their systems get no response as to whether the site is allowed, so assume the site you asked for is restricted and gives the 403 error. Once this happens you seem to have to make new 3G data connection (reset the phone, move cell or let the connection time out) to get it to try again."
    Sample: http://pic.twitter.com/N1lAwBjW

    (tags: scaling ireland vodafone fail censorware scalability customer-service)

Links for 2012-11-24

Published November 24, 2012

Links for 2012-11-23

Published November 23, 2012

  • IBM insider: How I caught my wife while bug-hunting on OS/2 • The Register

    Wow, working for IBM in the 80's was truly shitty.

    'IBM HR came up with a plan that summed up the department's view of tech staff: a dinner dance. In Southsea. For our non-British readers this is not a glamorous location. As a scumbag contractor I wasn’t invited, but since I was dating one of the seven women on the project, I went anyway and was impressed by the way IBM had tried so very hard to make the inside of a municipal leisure centre look like Hawaii. This is so crap that the integrity checks I’ve installed to watch myself for incipient senility keep flagging it as a false memory. The only way I can force myself to believe the idea that the richest corporation on the planet behaved that way is that the girl who took me is now a reassuringly expensive lawyer who was kind enough to marry me and so we have photographic evidence. (I wish to make it clear that I’m not saying IBM had the worst HR of any firm in the world, merely that my 28 years in technology and banking have never exposed a worse one to me.)'
    And indeed, so were MS:
    'We, on the other hand, were regarded as hopelessly bureaucratic. After Microsoft lost the source code for the actual build of OS/2 we shipped, I reported a bug triggered when you double-clicked on Chkdsk twice: the program would fire up twice and both would try to fix the disk at the same time, causing corruption. I noted that this “may not be consistent with the user's goals as he sees them at this time”. This was labelled a user error, and some guy called Ballmer questioned why I had this “obsession” with perfect code.'
    (thanks, Conor!)

    (tags: via:conor-delaney os2 ibm microsoft work 1980s pc uk steve-ballmer)

Links for 2012-11-21

Published November 21, 2012

Links for 2012-11-19

Published November 19, 2012

  • drip

    Unlike other tools intended to solve the JVM startup problem (e.g. Nailgun, Cake), Drip does not use a persistent JVM. There are many pitfalls to using a persistent JVM, which we discovered while working on the Cake build tool for Clojure. The main problem is that the state of the persistent JVM gets dirty over time, producing strange errors and requiring liberal use of cake kill whenever any error is encountered, just in case dirty state is the cause. Instead of going down this road, Drip uses a different strategy. It keeps a fresh JVM spun up in reserve with the correct classpath and other JVM options so you can quickly connect and use it when needed, then throw it away. Drip hashes the JVM options and stores information about how to connect to the JVM in a directory with the hash value as its name.
    (via HN)

    (tags: java command-line tools startup speed)

Links for 2012-11-14

Published November 14, 2012

Links for 2012-11-08

Published November 8, 2012

Links for 2012-10-31

Published October 31, 2012

  • The Future of Markdown

    'I'd really prefer not to fork the language; I'd much rather collectively help carry the banner of Markdown forward into the future, with the blessing of John Gruber and in collaboration with other popular sites that use Markdown. So... who's with me?'

    (tags: markdown markup html web standards)

Links for 2012-10-28

Published October 28, 2012

  • SipHash: a fast short-input PRF

    a family of pseudorandom functions optimized for short inputs. Target applications include network traffic authentication and hash-table lookups protected against hash-flooding denials-of-service attacks. SipHash is simpler than MACs based on universal hashing, and faster on short inputs. Compared to dedicated designs for hash-table lookup, SipHash has well-defined security goals and competitive performance. For example, SipHash processes a 16-byte input with a fresh key in 140 cycles on an AMD FX-8150 processor, which is much faster than state-of-the-art MACs.

    (tags: hashing siphash djb security algorithms)

Links for 2012-10-27

Published October 27, 2012

Links for 2012-10-26

Published October 26, 2012

Flood of posts

Published October 26, 2012

Sorry for the flood of recent posts -- turns out my cron job to gateway from Pinboard had stopped running due to cron fail. (I should really set up some monitoring someday ;)

Links for 2012-10-25

Published October 26, 2012

Links for 2012-10-24

Published October 26, 2012

Links for 2012-10-12

Published October 26, 2012

  • ElementCostInDataStructures

    "The cost per element in major data structures offered by Java and Guava (r11)]." A very useful reference!

    Ever wondered what's the cost of adding each entry to a HashMap? Or one new element in a TreeSet? Here are the answers: the cost per-entry for each well-known structure in Java and Guava. You can use this to estimate the cost of a structure, like this: if the per-entry cost of a structure is 32 bytes, and your structure contains 1024 elements, the structure's footprint will be around 32 kilobytes. Note that non-tree mutable structures are amortized (adding an element might trigger a resize, and be expensive, otherwise it would be cheap), making the measurement of the "average per element cost" measurement hard, but you can expect that the real answers are close to what is reported below.

    (tags: java coding guava reference memory cost performance data-structures)

Links for 2012-10-11

Published October 26, 2012

Links for 2012-10-08

Published October 26, 2012

  • Trident: a high-level abstraction for realtime computation

    built on Storm:

    Trident is a new high-level abstraction for doing realtime computing on top of Twitter Storm, available in Storm 0.8.0. It allows you to seamlessly mix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar - Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies.

    (tags: distributed realtime twitter storm trident distcomp stream-processing low-latency nathan-marz)

Links for 2012-10-05

Published October 26, 2012

  • Cliff Click's 2008 JavaOne talk about the NonBlockingHashTable

    I'm a bit late to this data structure -- highly scalable, nearly lock-free, benchmarks very well (except with the G1 GC): http://edwwang.com/blog/2012/02/10/concurrent-hashmap-benchmark/ . Having said that, it doesn't cope well with frequently-changing unique keys: http://sourceforge.net/tracker/?func=detail&aid=3563980&group_id=194172&atid=948362 . More background at: http://www.azulsystems.com/blog/cliff/2007-03-26-non-blocking-hashtable and http://www.azulsystems.com/blog/cliff/2007-04-01-non-blocking-hashtable-part-2 This was used in Cassandra for a while, although I think the above bug may have caused its removal?

    (tags: nonblockinghashtable data-structures hashmap concurrency scaling java jvm)

Links for 2012-10-01

Published October 26, 2012

  • Ingenious Dublin

    Excellent stuff, by Mary Mulvihill:

    Where in Dublin can you see a Victorian diving bell? What about the skeleton of Tommy, the prince’s elephant? The site of the world’s first earthquake experiment? Or the world’s sports pirate radio broadcast? Our new e-book Ingenious Dublin has all these fascinating stories and more. It is packed with information, places to visit, and lots of illustrations, and covers the city and county, from Skerries windmills to Ballybetagh’s fossil deer.'
    EUR 4.99 for the Kindle e-book. I'll buy that!

    (tags: kindle reading books mary-mulvihill science facts dublin ireland history)

Links for 2012-09-20

Published October 26, 2012

  • Facebook monitoring cache with Claspin

    reasonably nice heatmap viz for large-scale instance monitoring. I like the "snake" pattern for racks

    (tags: facebook monitoring dataviz heatmaps claspin cache memcached ui)

  • The Oireachtas great leap backwards: it’s not just about KildareStreet.com

    'it appears that the Oireachtas has decided to save time and money by eliminating entirely the stage in their workflow that parsed raw debates records into XML. This stage has been replaced with a (presumably automated) process that generates web pages from Lotus Notes. It’s easy to see how somebody with little appreciation of the value of providing open public data in a structured format could have viewed this stage as a costly luxury, and its elimination as a simple and obvious “efficiency”. It’s particularly disappointing, however, that nobody in the decision-making process seemed to be aware of how much of a backward step this “efficiency” would represent. As John Handelaar of KildareStreet.com told The Irish Times, “We are replacing 2012 with 1995 overnight”.'

    (tags: kildare-street open-data opengov ireland data oireachtas)

Links for 2012-09-18

Published October 26, 2012

  • PCRE Performance Project

    Excellent stuff. Using "sljit", a stackless platform-independent JIT compiler, this compiles Perl-compatible regular expressions to machine code on ARM, x86, MIPS and PowerPC platforms, resulting in 'similar matching speed to DFA based engines (like re2) on common patterns' with Perl compatibility. 'This work has been released as part of PCRE 8.20 and above. Now (PCRE 8.31), nearly all PCRE features are supported including UTF-8/16 and partial matching.'

    (tags: pcre regexps regex performance optimization jit compilation dfa re2 via:akohli)

Links for 2012-09-15

Published September 15, 2012

  • Spanner: Google's Globally-Distributed Database [PDF]

    Abstract: Spanner is Google's scalable, multi-version, globally-distributed, and synchronously-replicated database. It is the first system to distribute data at global scale and support externally-consistent distributed transactions. This paper describes how Spanner is structured, its feature set, the rationale underlying various design decisions, and a novel time API that exposes clock uncertainty. This API and its implementation are critical to supporting external consistency and a variety of powerful features: non-blocking reads in the past, lock-free read-only transactions, and atomic schema changes, across all of Spanner. To appear in: OSDI'12: Tenth Symposium on Operating System Design and Implementation, Hollywood, CA, October, 2012.

    (tags: database distributed google papers toread pdf scalability distcomp transactions cap consistency)

  • NCBI ROFL: Probably the most horrifying scientific lecture ever

    In 1983, at the Urodynamics Society meeting in Las Vegas, Professor G.S. Brindley first announced to the world his experiments on self-injection with papaverine to induce a penile erection. This was the first time that an effective medical therapy for erectile dysfunction (ED) was described, and was a historic development in the management of ED. The way in which this information was first reported was completely unique and memorable, and provides an interesting context for the development of therapies for ED. I was present at this extraordinary lecture, and the details are worth sharing. Although this lecture was given more than 20 years ago, the details have remained fresh in my mind, for reasons which will become obvious.
    Go on, guess.

    (tags: medicine science funny erectile-dysfunction omgwtf conferences)

  • Yuri Suzuki: London Underground circuit map radio

    Japanese designer yuri suzuki has sent designboom images of his 'london underground circuit maps' project developed as part of the designers in residence program at the london design museum, on show until january 13th, 2013. responding to 'thrift' as a theme, suzuki's work explores communication systems in consumer electronics. a printed circuit board (PCB) is used as a precedent for developing a electrical circuit influenced by harry beck's iconic london underground map diagrams. by strategically positioning certain speaker, resistor and battery components throughout the map, users can visually understand the complex networks associated with electricity and how power is generated within a radio.
    Beautifully done (via jwz.)

    (tags: electronics london art design underground travel yuri-suzuki circuitry)

Links for 2012-09-14

Published September 14, 2012

  • The meanings and origins of ‘feck’

    It's a "minced oath", apparently:

    'Feck is a popular minced oath in Ireland, occupying ground between the ultra-mild expletive flip and the often taboo (but also popular) fuck. It’s strongly associated with Irish speech, and serves a broad range of linguistic purposes that I’ll address briefly in this post.'
    It doesn't derive from the obvious source:
    So where does the curse, the not-quite-rude word, come from? It’s commonly assumed to stem from its coarser cousin fuck, the simple vowel change undercutting its power and making it more suitable for public expression. But Julian Walker, an educator at the British Library, offers a more roundabout route: “In faith” becomes the improbable “in faith’s kin” shortened to “i’fackins”, which gradually shrinks to “fac” and “feck”.

    (tags: feck swearing ireland irish hiberno-english father-ted etymology cursing)

Links for 2012-09-11

Published September 11, 2012

  • Chip and Skim: cloning EMV cards with the pre-play attack

    Worrying stuff from the LBT team. ATM RNGs are predictable, and can be spoofed by intermediate parties:

    'So far we have performed more than 1000 transactions at more than 20 ATMs and a number of POS terminals, and are collating a data set for statistical analysis. We have developed a passive transaction logger which can be integrated into the substrate of a real bank card, which records up to 100 unpredictable numbers in its EEPROM. Our analysis is ongoing but so far we have established non-uniformity of unpredictable numbers in half of the ATMs we have looked at. First, there is an easier attack than predicting the RNG. Since the unpredictable number is generated by the terminal but the relying party is the issuing bank, any intermediate party – from POS terminal software, to payment switches, or a middleman on the phone line – can intercept and superimpose their own choice of UN. Attacks such as those of Nohl and Roth, and MWR Labs show that POS terminals can be remotely hacked simply by inserting a sabotaged smartcard into the terminal.

    (tags: atm banking security attack prngs spoofing banks chip-and-pin emv smartcards)

Links for 2012-09-07

Published September 7, 2012

  • New UK Conservative Party Co-Chair Grant Shapps Founded Google Spamming Business

    Wow. Scummy stuff.

    Shapps founded HowToCorp in 2005, a site that, among other products, pitches the TrafficPaymaster software. The software apparently “scrapes” or copies content from all over the web, from RSS feeds to even sets of search results, to automatically generate pages that probably make little sense to the human visitor but which may pick up some traffic from Google and, in turn, generate clicks on Google AdSense or other ads.
    Google are not happy: On Sunday sources at Google confirmed TrafficPaymaster was in “violation” of its policies and that its search engine’s algorithms had been equipped to drop the ranking of any webpages created using HowToCorp’s software. Officially, Google said it does not comment on individual cases. “We have strict policies in place to ensure web users are presented with useful ads when browsing sites in our content network and to ensure our advertisers reach an engaged audience. If we are alerted to a site which breaks our AdSense policies, we will review it and can remove it from our network.”

    (tags: grant-shapps uk politics tories spammers spamming spinning adsense google spam trafficpaymaster)

  • NunatsiaqOnline 2012-09-06: The First Non-Inuk on the Moon

    No, I am not a conspiracy theorist who believes that Armstrong’s moon landing was faked at some mysterious location in the Nevada desert. Armstrong reached the moon. But his accolades are undeserved because he was not first. All right-thinking Nunavummiut know this, because we know that Inuit regularly visited the moon for centuries. David Iqaqrialu said as much in a heated exchange in the Nunavut legislature on May 6, 2002. We know it was heated because he prefaced his remarks by telling the Speaker, “I am starting to get hot under the collar...” He then went on to say, as reported in Hansard, “...it is not really related to the question that I posed, but this is background material. Inuit had reached the moon quite some time ago during the shamanistic ages, prior to the Americans reaching it with their machines and finding out it wasn’t what they thought it was.”
    (via Dave Walsh)

    (tags: inuit via:daev shaman nunavut neil-armstrong moon space exploration)

Links for 2012-09-06

Published September 6, 2012

  • Dublin City contact numbers for potholes, dangerous drivers, illegal parking etc.

    I'm sure these are about as useful as a chocolate teapot, but what the hey

    (tags: dublin parking cycling roads safety potholes reporting)

  • Knots on Mars! (and a few thoughts on NASA's knots)

    amazing post from the International Guild of Knot Tyers Forum:

    While a few of the folks here are no doubt aware, it might surprise most people to learn that knots tied in cords and thin ribbons have probably traveled on every interplanetary mission ever flown. If human civilization ends tomorrow, interplanetary landers, orbiters, and deep space probes will preserve evidence of both the oldest and newest of human technologies for millions of years. Knots are still used in this high-tech arena because cable lacing has long been the preferred cable management technique in aerospace applications. That it remains so to this day is a testament to the effectiveness of properly chosen knots tied by skilled craftspeople. It also no doubt has a bit to do with the conservative nature of aerospace design and engineering practices. Proven technologies are rarely cast aside unless they no longer fulfill requirements or there is something substantially better available. While the knots used for cable lacing in general can be quite varied -- in some cases even a bit idiosyncratic -- NASA has in-house standards for the knots and methods used on their spacecraft. These are specified in NASA Technical Standard NASA-STD-8739.4 -- Crimping, Interconnecting Cables, Harnesses, and Wiring. As far as I've been able to identify in the rover images below, all of the lacings shown are one of two of the several patterns specified in the standard. The above illustration shows the so-called "Spot Tie". It is a clove hitch topped by two half-knots in the form of a reef (square) knot. In addition to its pure binding role, it is also used to affix cable bundles to tie-down point.
    Some amazing scholarship on knot technology in this post -- lots to learn! (via Tony Finch, iirc)

    (tags: via:fanf mars nasa science knots tying rope cables cabling geek aerospace standards)

Links for 2012-09-05

Published September 5, 2012

Links for 2012-09-01

Published September 1, 2012

  • Striped (Guava: Google Core Libraries for Java 13.0.1 API)

    Nice piece of Guava concurrency infrastructure in the latest release:

    A striped Lock/Semaphore/ReadWriteLock. This offers the underlying lock striping similar to that of ConcurrentHashMap in a reusable form, and extends it for semaphores and read-write locks. Conceptually, lock striping is the technique of dividing a lock into many stripes, increasing the granularity of a single lock and allowing independent operations to lock different stripes and proceed concurrently, instead of creating contention for a single lock.
    The guarantee provided by this class is that equal keys lead to the same lock (or semaphore), i.e. if (key1.equals(key2)) then striped.get(key1) == striped.get(key2) (assuming Object.hashCode() is correctly implemented for the keys). Note that if key1 is not equal to key2, it is not guaranteed that striped.get(key1) != striped.get(key2); the elements might nevertheless be mapped to the same lock. The lower the number of stripes, the higher the probability of this happening.
    Prior to this class, one might be tempted to use Map, where K represents the task. This maximizes concurrency by having each unique key mapped to a unique lock, but also maximizes memory footprint. On the other extreme, one could use a single lock for all tasks, which minimizes memory footprint but also minimizes concurrency. Instead of choosing either of these extremes, Striped allows the user to trade between required concurrency and memory footprint. For example, if a set of tasks are CPU-bound, one could easily create a very compact Striped of availableProcessors() * 4 stripes, instead of possibly thousands of locks which could be created in a Map structure.

    (tags: locking concurrency java guava semaphores coding via:twitter)

  • HotSpot JVM garbage collection options cheat sheet (v2)

    'In this article I have collected a list of options related to GC tuning in JVM. This is not a comprehensive list, I have only collected options which I use in practice (or at least understand why I may want to use them). Compared to previous version a few useful diagnostic options was added. Additionally section for G1 specific options was introduced.'

    (tags: hotspot jvm coding gc java performance)

  • Martin "Disruptor" Thompson's Single Writer Principle

    Contains these millisecond estimates for highly-contended inter-thread signalling when incrementing a 64-bit counter in java:

    One Thread300
    One Thread with Memory Barrier4,700
    One Thread with CAS5,700
    Two Threads with CAS18,000
    One Thread with Lock10,000
    Two Threads with Lock118,000
    Undoubtedly not realistic for a lot of cases, but it's still useful for order-of-magnitude estimates of locking cost. Bottom line: don't lock if you can avoid it, even with 'volatile' or AtomicFoo types.

    (tags: java jvm performance coding concurrency threading cas locking)

  • Locks & Condition Variables - Latency Impact

    Firstly, this is 3 orders of magnitude greater latency than what I illustrated in the previous article using just memory barriers to signal between threads. This cost comes about because the kernel needs to get involved to arbitrate between the threads for the lock, and then manage the scheduling for the threads to awaken when the condition is signalled. The one-way latency to signal a change is pretty much the same as what is considered current state of the art for network hops between nodes via a switch. It is possible to get ~1µs latency with InfiniBand and less than 5µs with 10GigE and user-space IP stacks. Secondly, the impact is clear when letting the OS choose what CPUs the threads get scheduled on rather than pinning them manually. I've observed this same issue across many use cases whereby Linux, in default configuration for its scheduler, will greatly impact the performance of a low-latency system by scheduling threads on different cores resulting in cache pollution. Windows by default seems to make a better job of this.

    (tags: locking concurrency java jvm signalling locks linux threading)

  • Evolution of SoundCloud's Architecture

    nice write-up. nginx, Rails, RabbitMQ, MySQL, Cassandra, Elastic Search, HAProxy

    (tags: soundcloud webdev architecture scaling scalability)

Links for 2012-08-31

Published August 31, 2012

  • What Happens to Stolen Bicycles?

    'Bike thievery is essentially a risk-free crime. If you were a criminal, that might just strike your fancy. If Goldman Sachs didn’t have more profitable market inefficencies to exploit, they might be out there arbitraging stolen bikes.' Good summary, and I suspect a lot applies in Dublin too -- flea markets and vanloads of stolen bikes being sent to other cities for reselling.

    (tags: via:hn economics crime bikes theft goldman-sachs)

Links for 2012-08-19

Published August 19, 2012

  • 1024cores

    Some good algorithms and notes by Dmitry Vyukov on 'lockfree, waitfree, obstruction-free synchronization algorithms and data structures, scalability-oriented architecture, multicore/multiprocessor design patterns, high-performance computing, threading technologies and libraries (OpenMP, TBB, PPL), message-passing systems and related topics.' The catalog of lock-free queue implementations is particularly extensive (via Sergio Bossa)

    (tags: algorithms concurrency articles dmitry-vyukov go c++ coding via:sergio-bossa)

Links for 2012-08-12

Published August 12, 2012

  • Sting op exposes Andrews over FF Twitter rants - National News - Independent.ie

    Incredible sting op uncovers the real identity of an anonymous Twitter account posting Fianna Fail gossip:

    He discovered that each tweet had originated from the Twitter web interface, meaning it had been posted from a web browser on a computer, rather than sent from a mobile phone or other portable device. Based on the times that tweets were posted by @brianformerff, he deduced that the Tweets were being posted while the user was on a work break, using a company computer or an internet cafe. The next stage in the hunt was uncovering the IP address of the computer where the tweets originated. "I created my own web redirection service which would allow me to take links to articles of interest, for example in the Irish Times, and then transform them into short links that would pass through a redirection server I controlled. In this way, if someone read the tweets and clicked on the link, I would be able to establish the IP address of the computer that was being used at the time." The author created a new twitter account, @john_cant _type, based on the persona of a politics student based in Kildare. He started sending several messages and tweets to "brian" and other users to establish himself as a genuine twitter user. Eventually @brianformerff responded to a post from @john_cant_type to a link to an article at Silicon Republic. The bait was taken and the IP address was tracked to an internet cafe, Amazon cyber/net Rathmines which offers web access "at the very reasonable rate of €1/hour". What happened next descended almost into the realms of farce. The author waited for tweets from @brianformerff and then rushed to the internet cafe to try and catch Chris Andrews. Eventually the plan worked and the author used photography and video surveillance, even taking covert photographs of tweets as they were being posted in the internet cafe by Chris Andrews and analysing if the word count and structure matched the tweets appearing in cyberspace under the tag @brianformerff.

    (tags: chris-andrews twitter surveillance privacy anonymity politics ireland fianna-fail)

  • Rootbeer

    The Rootbeer GPU Compiler makes it easy to use Graphics Processing Units from within Java. Rootbeer is more advanced that CUDA or OpenCL Java Language Bindings. With bindings the developer must serialize complex graphs of objects into arrays of primitive types. With Rootbeer this is done automatically. Also with language bindings, the developer must write the GPU kernel in CUDA or OpenCL. With Rootbeer a static analysis of the Java Bytecode is done (using Soot) and CUDA code is automatically generated. [...] All of the familar Java code you have been writing can be executed on the GPU.

    (tags: gpu java coding cuda compiler)

Links for 2012-08-09

Published August 9, 2012

  • "In Which The Irish Invent Twitter in 1984"

    A fascinating story of 1980s tech history -- 'The initial Text Tell PX-1000 was developed by Text Lite Ltd. in Ireland in the early 1980s, probably in 1983. It allowed people to create simple text messages and send them by phone anywhere in the world. It had a built-in memory that could hold up to 7400 characters. The firmware inside the PX-1000 was written by West-Tec Ltd. in Ireland, who were probably also the hardware manufacturers. [... A later version was] the Philips version of the PX-1000Cr, as it features advanced cryptographic capabilities. It was intended for small companies and journalists, and was also used by the Dutch Government. [...] it played an important role in the fight for Nelson Mandela's release from prison.'

    (tags: nelson-mandela ireland history crypto texting text-lite 1980s philips)

Links for 2012-08-06

Published August 6, 2012

  • French illegal downloads agency Hadopi may be abolished

    According to recent statistics, Hadopi has sent 1 million warning emails, 99,000 "strike two" letters and identified 314 people for referral to the courts for possible disconnection. No one has actually been disconnected. According to Aurelie Filipetti, culture minister in the new French Government, Hadopi has been nothing but a waste of money. "€12 million per year and 60 officials; that's an expensive way to send 1 million emails," Filipetti said. "Hadopi has not fulfilled its mission of developing legal downloads. I prefer to reduce the funding of things that have not been proven to be useful."
    0 disconnections. Not one.

    (tags: hadopi privacy law three-strikes france money)

  • NASA's Mars Rover Crashed Into a DMCA Takedown

    An hour or so after Curiosity’s 1.31 a.m. EST landing in Gale Crater, I noticed that the space agency’s main YouTube channel had posted a 13-minute excerpt of the stream. Its title was in an uncharacteristic but completely justified all caps: “NASA LANDS CAR-SIZE ROVER BESIDE MARTIAN MOUNTAIN.” When I returned to the page ten minutes later, [...] the video was gone, replaced with an alien message: “This video contains content from Scripps Local News, who has blocked it on copyright grounds. Sorry about that.” That is to say, a NASA-made public domain video posted on NASA’s official YouTube channel, documenting the landing of a $2.5 billion Mars rover mission paid for with public taxpayer money, was blocked by YouTube because of a copyright claim by a private news service.

    (tags: dmca google fail nasa copyright false-positives scripps youtube video mars)

Links for 2012-08-03

Published August 3, 2012

  • High-frequency trading: The fast and the furious | The Economist

    "The NYMEX panel found that Infinium had finished writing the algorithm only the day before it introduced it to the market, and had tested it for only a couple of hours in a simulated trading environment to see how it would perform. The firm's normal testing processes take six to eight weeks. When the algorithm started its frenetic buying spree, the measures designed to shut it down automatically did not work. One was supposed to turn the system off if a maximum order size was breached, but because the machine was placing lots of small orders rather than a single big one the shutdown was not triggered. The other measure was meant to prevent Infinium from selling or buying more than a certain number of contracts, but because of an error in the way the rogue algorithm had been written, this, too, failed to spot a problem."

    (tags: hft automation trading markets stocks nymex bugs software)

Links for 2012-07-30

Published July 30, 2012

  • Lessons in website security anti-patterns by Tesco : Troy Hunt, an Aussie software architect working on a .Net security product called ASafaWeb, does a great job extensively deconstructing Tesco's appalling website security on their shopping site. In the process, he gets this wonderful tweet from their customer-care account: "@troyhunt Let me assure you that all customer passwords are stored securely & in line with industry standards across online retailers." As he says, this is a clear demonstration that Tesco is in the first stage of the four stages of competence -- "unconscious incompetence": "The individual does not understand or know how to do something and does not necessarily recognise the deficit." ( http://en.wikipedia.org/wiki/Four_stages_of_competence )
    (tags: tesco security passwords web http https ssl funny dot-net shopping uk customer-care)

  • Accident: Ryanair B738 and American B763 at Barcelona on Apr 14th 2011 : An accident report concerning a Ryanair flight.

    An American Airlines Boeing 767-300, registration N366AA performing flight AA-67 from Barcelona,SP (Spain) to New York JFK, NY (USA), had taxied to the holding point runway 25L and was holding short of the runway. A Ryanair Boeing 737-800, registration EI-EKB performing flight FR-8136 from Barcelona,SP (Spain) to Ibiza,SP (Spain) with 169 passengers and 6 crew, was taxiing along Barcelona's taxiway K for departure from runway 25L and was maneouvering to pass behind the Boeing 767-300. A number of passengers on board of the Boeing 737-800 observed the right hand wing of the aircraft contact the tailplane of the Boeing 767-300 and rose out of their seats attracting the attention of a flight attendant. A passenger told the flight attendant, that their aircraft had hit the aircraft besides them. The flight attendant contacted the purser, who instructed her to contact the flight deck, she contacted the flight deck and informed the captain that passengers had seen their aircraft had hit another aircraft. The captain responded however everything was fine and she continued with the takeoff about 2 minutes after the Boeing 767. Immediately after departure the passengers insisted the flight was not safe and they had collided with another aircraft, one of the passengers identified himself as an engineer. The flight attendant told the engineer that the captain had been informed and had told everything was fine. No further information was forwarded to the flight deck. After landing in Ibiza, while disembarking, the passengers again spoke up claiming the flight had been unsafe. During the turnaround the flight attendant informed the purser that one of the passengers observing the collision was an engineer. Neither approached the flight crew however. Following the return flight FR-8137 the purser talked to the captain and informed her that one of the passengers observing the collision was an engineer. In the following it was identified that the right hand winglet of the Boeing 737-800 had received damage, the Boeing 767-300 was found with damage to the left hand stabilizer following landing in New York.
    According to the story, it appears the AA flight crew were not informed of the potential damage to their plane before or during their transatlantic flight to JFK. (via Juan Flynn)
    (tags: via:juanflynn flight travel safety ryanair collisions)

  • CIAIAC report : The official report on that Ryanair/AA collision in Barcelona in July 2011, on pages 211-255.
    (tags: collisions safety travel air ryanair)

  • Practical machine learning tricks from the KDD 2011 best industry paper : Wow, this is a fantastic paper. It's a Google paper on detecting scam/spam ads using machine learning -- but not just that, it's how to build out such a classifier to production scale, and make it operationally resilient, and, indeed, operable. I've come across a few of these ideas before, and I'm happy to say I might have reinvented a few (particularly around the feature space), but all of them together make extremely good sense. If I wind up working on large-scale classification again, this is the first paper I'll go back to. Great info! (via Toby diPasquale.)
    (tags: classification via:codeslinger training machine-learning google ops kdd best-practices anti-spam classifiers ensemble map-reduce)

Links for 2012-07-29

Published July 29, 2012

  • The world’s first 3D-printed gun : I wasn't expecting to see this for a few years. The future is ahead of schedule!

    A .22-caliber pistol, formed from a 3D-printed AR-15 (M16) lower receiver, and a normal, commercial upper. In other words, the main body of the gun is plastic, while the chamber — where the bullets are actually struck — is solid metal. [...] While this pistol obviously wasn’t created from scratch using a 3D printer, the interesting thing is that the lower receiver — in a legal sense at least — is what actually constitutes a firearm. Without a lower receiver, the gun would not work; thus, the receiver is the actual legally-controlled part. In short, this means that people without gun licenses — or people who have had their licenses revoked — could print their own lower receiver and build a complete, off-the-books gun. What a chilling thought.

    (tags: via:peakscale guns scary future grim-meathook-future 3d-printing thingiverse weapons)

Links for 2012-07-28

Published July 28, 2012

Links for 2012-07-27

Published July 27, 2012

  • This park's life - The Irish Times - Thu, Jul 26, 2012 : Great article about Dublin's Phoenix Park, Europe's largest enclosed urban park (more than twice the size of New York's Central Park, in fact). Now that I have two little kids, I've been spending a good portion of my weekends there -- it's a wonderful thing to have on our doorstep. Also:

    The park even breeds celebrities. “The lion that roars at the start of the MGM movies. He’s a Dub. He was born in Dublin Zoo.”

    (tags: phoenix-park dublin history parks deer lion kids)

Links for 2012-07-26

Published July 26, 2012

  • Universal properties of mythological networks - Abstract - EPL (Europhysics Letters) - IOPscience : Abstract:

    As in statistical physics, the concept of universality plays an important, albeit qualitative, role in the field of comparative mythology. Here we apply statistical mechanical tools to analyse the networks underlying three iconic mythological narratives with a view to identifying common and distinguishing quantitative features. Of the three narratives, an Anglo-Saxon and a Greek text are mostly believed by antiquarians to be partly historically based while the third, an Irish epic [jm: "An Táin Bó Cúailnge", The Tain, to be specific], is often considered to be fictional. Here we use network analysis in an attempt to discriminate real from imaginary social networks and place mythological narratives on the spectrum between them. This suggests that the perceived artificiality of the Irish narrative can be traced back to anomalous features associated with six characters. Speculating that these are amalgams of several entities or proxies, renders the plausibility of the Irish text comparable to the others from a network-theoretic point of view.
    Here's what the Irish Times said:
    The society in the 1st century story of the Táin Bó Cúailnge looked artificial at first analysis of the networks between 404 characters in the story. However, the researchers found the society reflected real rather than fictional networks when the weakest links to six of the characters are removed. These six characters included Medb, Queen of Connacht; Conchobor, King of Ulster and Cúchulainn. They were "similar to superheroes of the Marvel universe" and are "too superhuman" or too well-connected to be real, researchers said. The researchers suggest that each of these superhuman characters may be an amalgam of many which became fused and exaggerated as the story was passed down orally through generations.

    (tags: networks society the-tain epics history mythology ireland statistics network-analysis papers)

  • Irish campsite recommendations : the conclusion of a Twitter/Facebook recommendations-gathering exercise; winners seem to be Lough Key Forest Park, Renvyle Beach, Fintra, Eagle Point, and Hidden Valley
    (tags: camping ireland tips recommendations caravan holidays vacation)

Links for 2012-07-23

Published July 23, 2012

  • CloudBurst : 'Highly Sensitive Short Read Mapping with MapReduce'. current state of the art in DNA sequence read-mapping algorithms.

    CloudBurst uses well-known seed-and-extend algorithms to map reads to a reference genome. It can map reads with any number of differences or mismatches. [..] Given an exact seed, CloudBurst attempts to extend the alignment into an end-to-end alignment with at most k mismatches or differences by either counting mismatches of the two sequences, or with a dynamic programming algorithm to allow for gaps. CloudBurst uses [Hadoop] to catalog and extend the seeds. In the map phase, the map function emits all length-s k-mers from the reference sequences, and all non-overlapping length-s kmers from the reads. In the shuffle phase, read and reference kmers are brought together. In the reduce phase, the seeds are extended into end-to-end alignments. The power of MapReduce and CloudBurst is the map and reduce functions run in parallel over dozens or hundreds of processors.
    JM_SOUGHT -- the next generation ;)
    (tags: bioinformatics mapreduce hadoop read-alignment dna sequencing sought antispam algorithms)

  • Expensive lessons in Python performance tuning : some good advice for large-scale Python performance: prun and guppy for profiling, namedtuples for memory efficiency, and picloud for trivial EC2-based scale-out. (via Nelson)
    (tags: picloud prun guppy namedtuples python optimization performance tuning profiling)

  • On Patents : Notch comes up with a perfect analogy for software patents.

    I am mostly fine with the concept of “selling stuff you made”, so I’m also against copyright infringement. I don’t think it’s quite as bad as theft, and I’m not sure it’s good for society that some professions can get paid over and over long after they did the work (say, in the case of a game developer), whereas others need to perform the job over and over to get paid (say, in the case of a hairdresser or a lawyer). But yeah, “selling stuff you made” is good. But there is no way in hell you can convince me that it’s beneficial for society to not share ideas. Ideas are free. They improve on old things, make them better, and this results in all of society being better. Sharing ideas is how we improve. A common argument for patents is that inventors won’t invent unless they can protect their ideas. The problem with this argument is that patents apply even if the infringer came up with the idea independently. If the idea is that easy to think of, why do we need to reward the person who happened to be first?
    Of course, in reality it's even worse, since you don't actually have to be first to invent -- just first to file without sufficient people noticing, and people are actively dissuaded from noticing (since it makes their lives riskier if they know about the existence of patents)...
    (tags: business legal ip copyright patents notch minecraft patent-trolls)

  • Marsh's Library : Dublin museum of antiquarian books, open to the public -- well worth a visit, apparently (I will definitely be making my way there soon I suspect), to check out their new "Marvels of Science" exhibit. Not only that though, but they have a beautiful website with some great photos -- exemplary
    (tags: museum dublin ireland libraries books science)

  • 'Poisoning Attacks against Support Vector Machines', Battista Biggio, Blaine Nelson, Pavel Laskov : The perils of auto-training SVMs on unvetted input.

    We investigate a family of poisoning attacks against Support Vector Machines (SVM). Such attacks inject specially crafted training data that increases the SVM's test error. Central to the motivation for these attacks is the fact that most learning algorithms assume that their training data comes from a natural or well-behaved distribution. However, this assumption does not generally hold in security-sensitive settings. As we demonstrate, an intelligent adversary can, to some extent, predict the change of the SVM's decision function due to malicious input and use this ability to construct malicious data. The proposed attack uses a gradient ascent strategy in which the gradient is computed based on properties of the SVM's optimal solution. This method can be kernelized and enables the attack to be constructed in the input space even for non-linear kernels. We experimentally demonstrate that our gradient ascent procedure reliably identifies good local maxima of the non-convex validation error surface, which significantly increases the classifier's test error.
    Via Alexandre Dulaunoy
    (tags: papers svm machine-learning poisoning auto-learning security via:adulau)

Links for 2012-07-21

Published July 21, 2012

Links for 2012-07-16

Published July 16, 2012

  • Science funding doesn't add up - The Irish Times : '[Science Foundation Ireland] said it was continuing to support basic research, but there are a number of leading scientists here who were refused funding despite having qualified for it in the past. Dr Mike Peardon of the School of Mathematics was recently been turned down, having been “administratively withdrawn”. This means the application for funding was rejected at the first post during initial consideration and before it had a chance to be assessed by external experts. Several others in his department suffered a similar fate. “The school of mathematics at Trinity is ranked the 15th best maths department in the world and now we are not fundable by Science Foundation Ireland,” he said. “The cases I heard of have all been in pure maths,” said Prof Lorraine Hanlon in UCD’s school of physics. “All reported that the people in pure maths were returned unreviewed.” She believes other areas may also come under pressure. “Pure maths is the thin end of the wedge. The Government says mathematics is fundamental, but on the other side says we dont really care enough to support it. That is a schizophrenic approach,” she said.'
    (tags: mathematics ireland science research academia funding tcd ucd sfi)

  • Microsoft's ill-chosen magic constants : 'Paolo Bonzini noticed something a little awkward in the Linux kernel support code for Microsoft's HyperV virtualisation environment - specifically, that the magic constant passed through to the hypervisor was "0xB16B00B5", or, in English, "BIG BOOBS". It turns out that this isn't an exception - when the code was originally submitted it also contained "0x0B00B135".' me, I prefer my magic constants less offensive and more Subgenius-oriented: "0xB0BD0BB5"
    (tags: constants via:kevin-lyda oh-dear microsoft fail magic-numbers boobs linux kernel)

Links for 2012-07-14

Published July 14, 2012

  • Scaling lessons learned at Dropbox : website-scaling tips and suggestions, "particularly for a resource-constrained, fast-growing environment that can’t always afford to do things “the right way” (i.e., any real-world engineering project". I really like the "run with fake load" trick; add additional queries/load which you can quickly turn off if the service starts browning out, giving you a few days breathing room to find a real fix before customers start being affected. Neat
    (tags: dropbox scalability webdev load scaling-up)

Links for 2012-07-11

Published July 11, 2012

  • Don’t waste your time in crappy startup jobs : 7 reasons why working for a startup sucks. Been there, done that -- I wish I'd read this years ago. It should be permalinked at the top of Hacker News. "In 1995, a lot of talented young people went into large corporations because they saw no other option in the private sector– when, in fact, there were credible alternatives, startups being a great option. In 2012, a lot of young talent is going into startups for the same reason: a belief that it’s the only legitimate work opportunity for top talent, and that their careers are likely to stagnate if they work in more established businesses. They’re wrong, I think, and this mistaken belief allows them to be taken advantage of. The typical equity offer for a software engineer is dismally short of what he’s giving up in terms of reduced salary, and the career path offered by startups is not always what it’s made out to be. For all this, I don’t intend to argue that people shouldn’t join startups. If the offer’s good, and the job looks interesting, it’s worth trying out. I just don’t think that the current, unconditional “startups are awesome!” mentality serves us well. It’s not good for any of us, because there’s no tyrant worse than a peer selling himself short, and right now there are a lot of great people selling themselves very short for a shot at the “startup experience” -- whatever that is."
    (tags: startups work job life career tech vc companies pay stock share-options)

Links for 2012-07-10

Published July 10, 2012

Links for 2012-07-03

Published July 3, 2012

Links for 2012-06-29

Published June 29, 2012

  • Facts still sacred despite Ireland's spectrum of conflicting views on abortion - The Irish Times - Fri, Jun 29, 2012 : Very good data-driven analysis. "Pro-life” groups claim abortion is a serious mental health risk for women. Youth Defence claims women who opt for an abortion rather than carrying to term or giving the baby up for adoption suffer mental maladies such as depression, suicide and other problems. But this is at heart a scientific claim, and can thus be tested. [...] Psychologist Dr Brenda Majors studied this in depth and found no evidence that ["post-abortion syndrome"] exists. As long as a woman was not depressive before an abortion, “elective abortion of an unintended pregnancy does not pose a risk to mental health”. The same results were found in several other studies [...] Essentially these studies found there was no difference in mental health between those who opted for abortion and those who carried to term. Curiously, there was a markedly increased risk to mental health for women who gave a child up for adoption. A corollary of the research was that while women did not suffer long-term mental health effects due to abortion, short-term guilt and sadness was far more likely if the women had a background where abortion was viewed negatively or their decisions were decried -- the kind of attitude fostered by “pro-life” activists."
    (tags: pro-choice pro-life abortion data facts via:irish-times research science pregnancy depression pas)

Links for 2012-06-28

Published June 28, 2012

  • "Machine Learning That Matters" [paper, PDF] : Great paper. This point particularly resonates: "It is easy to sit in your o?ce and run a Weka algorithm on a data set you downloaded from the web. It is very hard to identify a problem for which machine learning may o?er a solution, determine what data should be collected, select or extract relevant features, choose an appropriate learning method, select an evaluation method, interpret the results, involve domain experts, publicize the results to the relevant scienti?c community, persuade users to adopt the technique, and (only then) to truly have made a di?erence (see Figure 1). An ML researcher might well feel fatigued or daunted just contemplating this list of activities. However, each one is a necessary component of any research program that seeks to have a real impact on the world outside of machine learning."
    (tags: machine-learning ml software data real-world algorithms)

  • Massive identity-theft breach in South Korea results in calls for national ID system to be abandoned : In South Korea, web users are required to provide their national ID number for "virtually every type of Internet activity, not only for encrypted communications like e-commerce, online banking and e-government services but also casual tasks like e-mail and blogging", apparently in an attempt to "curb cyber-bullying". The result is obvious -- those ID numbers being collected in giant databases at companies like "SK Communications, which runs top social networking service Cyworld and search site Nate", and those giant databases being tasty targets for black-hats. Now: "In Korea’s biggest-ever case of data theft the recent hacking attack at SK Communications, which runs top social networking service Cyworld and search site Nate, breached 35 million accounts, a mind-boggling total for a country that has about 50 million people and an economically-active population of 25 million. The compromised information includes names, passwords, phone numbers, e-mail addresses, and most alarmingly, resident registration numbers, the country’s equivalent to social security numbers." This is an identity-fraudster's dream: "In the hands of criminals, resident registration numbers could become master keys that open every door, allowing them to construct an entire identity based on the quality and breadth of data involved."
    (tags: south-korea identity fraud identity-theft web bullying authentication hacking)

Links for 2012-06-27

Published June 27, 2012

  • WeatherSpark : Beatiful dataviz of weather data from met.no, NOAA.gov, World Weather Online and Weather Central. The main graph includes: mean and percentiles of historical temperature data for time of year, the temperature and precipitation forecast over the chosen period, wind direction and speed, with hourly data. Very nicely done! (via Una Mullally)
    (tags: via:unamullally dataviz temperature forecasts weather graphing percentiles wind rain)

  • the recruiter honeypot : wow, I thought it was hard hiring in Dublin. Sounds like Silicon Valley is insane. "Unfortunately, it’s not all about the numbers. Though external recruiters perform well for start-ups, there’s another side to this story. It pains me to write this but I think it’s important to share. Meebo employed lots of external recruiters when we were getting off the ground. We had standard 18-month no-poach restrictions with all of our contractors that specified that those recruiters were not allowed to contact Meebo employees within 18 months of our contract expiring. Most of those contracts expired in 2008-2009. However, every recruiter and firm we’d worked with who was still in the recruiting business tried to poach [the 'honeypot' employee] Pete London." (Another lesson: don't build a product in javascript, since it's impossible to hire engineers ;)
    (tags: honeypots hiring silicon-valley recruiting coding experts meebo)

Links for 2012-06-26

Published June 26, 2012

  • CEO Of Internet Provider Sonic.net: We Delete User Logs After Two Weeks. Your Internet Provider Should, Too. - Forbes : "what we saw was a shift towards customers being made part of a business model that involved–I don’t know if extortion is the right word–but embarassment for gain. An individual would download a movie, using bittorrent, and infringe copyright. And that might be our customer, like Bob Smith who owns a Sonic.net account, or it might be their spouse, or it might be their child. Or it might be one of his three roommates in a loft in San Francisco, who Bob is not responsible for, and who rent out their loft on AirBnB and have couch surfers and buddies from college and so on and open Wifi. When lawyers asked us for these users’ information, some of our customers I spoke with said “Oh yeah, crap, they caught me,” and were willing to admit they engaged in piracy and pay a settlement. But in other cases, it turned out the roommate did it, or no one would admit to doing it. But they would pay the settlement anyway. Because no one wants to be named in the public record in a case from So-And-So Productions vs. 1,600 names including Bob Smith for downloading a film called “Don’t Tell My Wife I B—F—— The Babysitter.” AG: Is that a real title? DJ: Yes. I’ve read about cases where a lawyer was doing this for the movie “The Expendables,” and 5% of people settled. So then he switched to representing someone with an embarassing porn title, and like 30% of people paid. It seemed like half the time, the customer wasn’t the one right one, but they rolled over because it would be very embarassing. And I think that’s an abuse of process. I was unwilling to become part of that business model. In many cases the lawyers never pursued the case, and it was all bluster. But under that threat, you pay."
    (tags: interview isps freedom copyright internet shakedown lawyers sonic.net data-retention via:oisin)

  • an ex-RBSG engineer on the NatWest/RBS/UlsterBank IT fiasco : 'Turning over your systems support staff in a wave of redundancies is not the best way to manage the transfer of knowledge. Not everyone who worked the batch at [Royal Bank of Scotland Group] even knew what it is they knew; how, then, could they explain it to people who didn’t know there was knowledge to acquire? Outsourcing the work from Edinburgh to Aberdeen and sacking the staff would have exposed them to the same risks. [...] I Y2K tested one of the batch feeder systems at RBS from 1997 - 1998, and managed acceptance testing in payments processing systems from 1999 - 2001. I was one of the people who watched over the first batch of the millennium instead of going to a party. I was part of the project that moved the National Westminster batch onto the RBS software without a single failure. I haven’t worked for the bank for five years, and I am surprised at how personally affronted I am that they let that batch fail. But I shouldn’t be. Protectiveness of the batch was the defining characteristic of our community. We were proud of how well that complex structure of disparate components hummed along. It was a thing of beauty, of art and craft, and they dropped it all over the floor.'
    (tags: systems ops support maintainance legacy ca-7 banking rbs natwest ulster-bank fail outsourcing)

  • Some Facts & Insights Into The Whole Discussion Of 'Ethics' And Music Business Models | Techdirt : David "Camper Van Beethoven" Lowery's blogpost about music sales, ethics, piracy etc. looks like it was pretty much riddled with errors regarding the viability of the music business, then and now. Empirical figures from Jeff Price from Tunecore, and others, to debunk it: "'Well here’s some truth about the old industry that David somehow misses. Previously, artists were not rolling in money. Most were not allowed into the system by the gatekeepers. Of those that were allowed on the major labels, over 98% of them failed. Yes, 98%?. Of the 2% that succeeded, less than a half percent of those ever got paid a band royalty from the sale of recorded music. How in the world is an artist making at least something, no matter how small, worse than 99% of the world’s unsigned artists making nothing and of the 1% signed, less than a half a percent of them ever making a single band royalty ever?'" [...] "Another example of Lowery being wrong that Price responds to is the claim that recorded music revenue to artists has been going down. Price has data: 'This is empirically false. Revenue to labels has collapsed. Revenue to artists has gone up with more artists making more money now than at any time in history, off of the sale of pre-recorded music. Taken a step further, a $17.98 list price CD earned a band $1.40 as a band royalty that they only got if they were recouped (over 99% of bands never recouped). If an artist sells just two songs for $0.99 on iTunes via TuneCore, they gross $1.40. If they sell an album for $9.99 on iTunes via TuneCore, they gross $7.00. This is an INCREASE of over 700% in revenue to artists for recorded music sales.'"
    (tags: music mp3 music-business piracy techdirt david-lowery tunecore)

Links for 2012-06-25

Published June 25, 2012

  • Eight Real Tales of Learning Computer Science as a High School Girl : 'All students at Stuyvesant High School are required to take a year of computer science. As it turns out, the advanced computer science classes skew mostly male anyway. But for a year, boys and girls get exposed to computer programming together. We asked Mike Zamansky, the head of the computer science program, to share some stories from his female students. They did us one better. Eight students sent in first-hand accounts of what it’s like to learn computer programming as a teenage girl.' Some interesting comments here. This topic is weighing on my mind now that I have two girls...
    (tags: schools learning education computer-science technology nyc girls teenage)

  • RBS collapse details revealed - The Register : as noted in the gossip last week. 'The main batch scheduling software used by RBS is CA-7, said one source, a former RBS employee who left the company recently.' 'RBS do use CA-7 and do update all accounts overnight on a mainframe via thousands of batch jobs scheduled by CA-7 ... Backing out of a failed update to CA-7 really ought to have been a trivial matter for experienced operations and systems programming staff, especially if they knew that an update had been made. That this was not the case tends to imply that the criticisms of the policy to "offshore" also hold some water.'
    (tags: outsourcing failure software rbs natwest ulster-bank ulster-blank offshoring downsizing ca-7 upgrades)

Links for 2012-06-23

Published June 23, 2012

  • Natwest, RBS: When will bank glitch be fixed? Probably not today • The Register Forums : Some amazing insider-info posts on the Reg forum for the gigantic RBS/NatWest/Ulster Bank multi-day outage. Fingers pointing at their outsourcing/downsizing practices -- in a word, they've sacked the experienced staff, replaced them with noobs thousands of miles away, and not paid down any technical debt on the legacy code they're maintaining. Classic legacy IT fail. "I worked for RBS during and after the merger with Natwest, I left their Global Financial Markets Department in 2004 after a 5 year stint. They had already moved some IT functions to India at that point and have continued to do so year on year since. The numbers some people are quoting 1600/800 are possibly the more recent figures, the total is way way beyond this. The comments on documentation are comical, as if a document is the thing you turn to at a time of crisis. The fact is, when you work closely with systems and the business users, you understand not only the quirks of the systems, but the risks and consequences of failure. You work with those users on the work around solutions that will get the banking day complete. They haven't just outsourced the IT staff, but the very experienced and valuable back office / operations staff that would work with IT staff to solve the serious issues. I beleive these guys are mostly posted out in Singapore, who probably have never met the IT staff in India. The unseen cost of outsourcing is a compounding loss of shared experience and commitment, which becomes accutely apparent when the sh!t hits the ... cash machines The chaps I trained out in India were nice enough, but they simply lacked the knowledge and experience of Finacial Markets trading, trade and settlement processing, Swift messaging blah blah and the risks involved. I'll be drinking with a bunch of ex RBS/Natwesties soon enough, where we'll all be saying..... "WE TOLD YOU SO!!!!!!!" Another poster says: "I understand that your description of the RBS Mainframe based batch update process is fairly accurate. The source of the problem was a software update to Batch scheduling suite CA7. The upgrade when so well that now there is no schedule to run all of those thousands of batch jobs to receive and make BACS payments, update balance, schedule printouts, etc. I am sure the problem with the CA7 upgrade and the unfortunate misplacing of the Batch schedule has absolutely nothing to do the with the last UK based technicians leaving recently. The guys in India of course are perfectly able to cope and fix their mistake. I'm sure they understand how the thousands of jobs in the schedule need to ordered to make sure there is data corruption or loss. After all the problem happened on Tuesday and it's only Friday. I wonder how many ex-RBS staff have received very lucrative short term contracts in the last few days......"
    (tags: natwest it rbs the-register outsourcing fail organisations ulster-bank ulster-blank)

Links for 2012-06-22

Published June 22, 2012

Links for 2012-06-21

Published June 21, 2012

Links for 2012-06-20

Published June 20, 2012

  • The Hydra Bay : "How to set up a Pirate Bay proxy". Step-by-step instructions for MacOS and Linux on how to run a fully-functional reverse proxy for The Pirate Bay -- in other words, provide a duplicate URL for users to circumvent ISP blocks of TPB. http://about.piratereverse.info/proxy/list.html contains about a hundred others. See also http://unblockedpiratebay.com/ for a standalone PHP script which does the same (albeit a little less efficiently). A good demonstration of how futile filtering techniques like IP or domain name blocks are, when applied to a popular website like TPB.
    (tags: piratebay filtering censorship copyright php proxies reverse-proxies ip-blocking dns-blocking)

  • how to restore from iCloud backup : the trick: don't try and do it through iTunes, it won't give you the option, apparently. I have a carrier unlock, and apparently need to wipe the phone for it to take place; this scares the crap out of me
    (tags: backup iphone restore sysadmin phones icloud apple howto)

Links for 2012-06-19

Published June 19, 2012

Links for 2012-06-15

Published June 15, 2012

  • PGP founder, Navy SEALs uncloak encrypted comms biz • The Register : 'The company, called Silent Circle, will launch later this year, when $20 a month will buy you encrypted email, text messages, phone calls, and videoconferencing in a package that looks to be strong enough to have the NSA seriously worried. Zimmermann says that surveillance by the state and others has increased vastly over the last few years, and privacy improvement are again needed. "At the very least I want people, as part of their right in a free society to be able to communicate securely," he said in a promotional video. "I should be able to whisper in your ear, even if your ear is a thousand miles away." [...] While software can handle most of the work, there still needs to be a small backend of servers to handle traffic. The company surveyed the state of privacy laws around the world and found that the top three choices were Switzerland, Iceland, and Canada, so they went for the one within driving distance.'
    (tags: pgp phil-zimmermann privacy crypto silent-circle apps vc security)

Links for 2012-06-13

Published June 13, 2012

  • The Silencing of Maya : software patent shakedown threatens to remove a 4-year-old's only means of verbal expression: 'Maya can speak to us, clearly, for the first time in her life. We are hanging on her every word. We’ve learned that she loves talking about the days of the week, is weirdly interested in the weather, and likes to pretend that her toy princesses are driving the bus to school (sometimes) and to work (other times). This app has not only allowed her to communicate her needs, but her thoughts as well. It’s given us the gift of getting to know our child on a totally different level. I’ve been so busy embracing this new reality and celebrating, that I kind of forgot that there was an ongoing lawsuit, until last Monday. When Speak for Yourself was removed from the iTunes store.'
    (tags: speak-for-yourself children law swpats patenting stories ipad apps)

  • _Building High-level Features Using Large Scale Unsupervised Learning_ [paper, PDF] : "We consider the problem of building highlevel, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images using unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also ?nd that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 20,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art."
    (tags: algorithms machine-learning neural-networks sgd labelling training unlabelled-learning google research papers pdf)

Links for 2012-06-12

Published June 12, 2012

Links for 2012-06-11

Published June 11, 2012

Links for 2012-05-25

Published May 25, 2012

Telegraph spam in 1864

Published May 24, 2012

Here's a letter to the editor of The Times, dated 1st June 1864:

TO THE EDITOR OF THE TIMES.
Sir, --- On my arrival home late yesterday evening a "telegram," by "London District Telegraph," addressed in full to me, was put into my hands. It was as follows :--
"Messrs. Gabriel, dentists, 27, Harley-street, Cavendish-square. Until October Messrs. Gabriel's professional attendance at 27, Harley-street, will be 10 till 5."
I have never had any dealings with Messrs. Gabriel, and beg to ask by what right do they disturb me by a telegram which is evidently simply the medium of advertisement? A word from you would, I feel sure, put a stop to this intolerable nuisance. I enclose the telegram, and am,
Your faithful servant,
M.P.
Upper Grosvenor-street, May 30.

(thanks to Tony Finch for the forward)