Skip to content

Justin's Linklog Posts

Links for 2013-04-03

Published April 3, 2013

  • The Patent Protection Racket

    Joel On Software weighs in (via Tony Finch):

    The fastest growing industry in the US right now, even during this time of slow economic growth, is probably the patent troll protection racket industry.

    (tags: joel-on-software patents swpats shakedown extortion us-politics patent-trolls via:fanf)

  • Cap’n Proto

    Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Cap’n Proto is INFINITY TIMES faster than Protocol Buffers.
    Basically, marshalling like writing an aligned C struct to the wire, QNX messaging protocol-style. Wasteful on space, but responds to this by suggesting compression (which is a fair point tbh). C++-only for now. I’m not seeing the same kind of support for optional data that protobufs has though. Overall I’m worried there’s some useful features being omitted here…

    (tags: serialization formats protobufs capn-proto protocols coding c++ rpc qnx messaging compression compatibility interoperability i14y)

  • CRDTs – Commutative Replicated Data Types [pdf]

    Shared read-only data is easy to scale by using well-understood replication techniques. However, sharing mutable data at a large scale is a dicult problem, because of the CAP impossibility result [5]. Two approaches dominate in practice. One ensures scalability by giving up consistency guarantees, for instance using the Last-Writer-Wins (LWW) approach [7]. The alternative guarantees consistency by serialising all updates, which does not scale beyond a small cluster [12]. Optimistic replication allows replicas to diverge, eventually resolving conflicts either by LWW-like methods or by serialisation [11]. In some (limited) cases, a radical simplication is possible. If concurrent updates to some datum commute, and all of its replicas execute all updates in causal order, then the replicas converge.1 We call this a Commutative Replicated Data Type (CRDT). The CRDT approach ensures that there are no conflicts, hence, no need for consensus-based concurrency control. CRDTs are not a universal solution, but, perhaps surprisingly, we were able to design highly useful CRDTs. This new research direction is promising as it ensures consistency in the large scale at a low cost, at least for some applications.

    (tags: consistency algorithms concurrency crdts distcomp data)

  • CRDT toolbox

    ‘The CRDT toolbox provides a collection of basic Conflict-free replicated data types as well as a common interface for defining your own CRDTs’. – in Eric Moritz’ github. Also includes some more links to CRDT background reading.

    (tags: crdt github eric-moritz python algorithms)

  • Eventually-Consistent Data Structures [slides]

    implementing CRDTs in Riak and Voldemort

    (tags: crdt algorithms distcomp riak voldemort distributed)

Links for 2013-04-02

Published April 2, 2013

Links for 2013-03-29

Published March 29, 2013

  • East Texas Judge Says Mathematical Algorithms Can’t Be Patented, Dismisses Uniloc Claim Against Rackspace

    This seems pretty significant. Is the tide turning in the Texas Eastern District against patent trolls, at last? And does it establish sufficient precedent?

    A federal judge has thrown out a patent claim against Rackspace, ruling that mathematical algorithms can’t be patented. The ruling in the Eastern Disrict stemmed from a 2012 complaint filed by Uniloc USA asserting that processing of floating point numbers by the Linux operating system was a patent violation. Chief Judge Leonard Davis based the ruling on U.S. Supreme Court case law that prohibits the patenting of mathematical algorithms. According to Rackspace, this is the first reported instance in which the Eastern District of Texas has granted an early motion to dismiss finding a patent invalid because it claimed unpatentable subject matter. Red Hat, which supplies Linux to Rackspace, provided Rackspace’s defense. Red Hat has a policy of standing behind customers through its Open Source Assurance program.
    See https://news.ycombinator.com/item?id=5455869 for more discussion.

    (tags: east-texas patents swpats maths patenting law judges rackspace linux red-hat uniloc-usa floating-point)

  • Introducing Chronos: A Replacement for Cron

    A distributed, fault-tolerant “cron” is something which comes up frequently — it makes for a great fault-tolerance building block. This one sounds like it’s too closely tied into Mesos, though (IMO).

    Chronos is our replacement for cron. It is a distributed and fault-tolerant scheduler which runs on top of Mesos. It’s a framework and supports custom mesos executors as well as the default command executor. Thus by default, Chronos executes SH (on most systems BASH) scripts. Chronos can be used to interact with systems such as Hadoop (incl. EMR), even if the mesos slaves on which execution happens do not have Hadoop installed. Included wrapper scripts allow transfering files and executing them on a remote machine in the background and using asynchroneous callbacks to notify Chronos of job completion or failures.

    (tags: cron scheduling mesos stacks design airbnb chronos fault-tolerance distcomp distributed-computing scripts jobs)

Links for 2013-03-28

Published March 28, 2013

  • One of CloudFlare’s upstream providers on the “death of the internet” scare-mongering

    Having a bad day on the Internet is nothing new. These are the types of events we deal with on a regular basis, and most large network operators are very good at responding quickly to deal with situations like this. In our case, we worked with Cloudflare to quickly identify the attack profile, rolled out global filters on our network to limit the attack traffic without adversely impacting legitimate users, and worked with our other partner networks (like NTT) to do the same. If the attacks had stopped here, nobody in the “mainstream media” would have noticed, and it would have been just another fun day for a few geeks on the Internet. The next part is where things got interesting, and is the part that nobody outside of extremely technical circles has actually bothered to try and understand yet. After attacking Cloudflare and their upstream Internet providers directly stopped having the desired effect, the attackers turned to any other interconnection point they could find, and stumbled upon Internet Exchange Points like LINX (in London), AMS-IX (in Amsterdam), and DEC-IX (in Frankfurt), three of the largest IXPs in the world. An IXP is an “interconnection fabric”, or essentially just a large switched LAN, which acts as a common meeting point for different networks to connect and exchange traffic with each other. One downside to the way this architecture works is that there is a single big IP block used at each of these IXPs, where every network who interconnects is given 1 IP address, and this IP block CAN be globally routable. When the attackers stumbled upon this, probably by accident, it resulted in a lot of bogus traffic being injected into the IXP fabrics in an unusual way, until the IXP operators were able to work with everyone to make certain the IXP IP blocks weren’t being globally re-advertised. Note that the vast majority of global Internet traffic does NOT travel over IXPs, but rather goes via direct private interconnections between specific networks. The IXP traffic represents more of the “long tail” of Internet traffic exchange, a larger number of smaller networks, which collectively still adds up to be a pretty big chunk of traffic. So, what you actually saw in this attack was a larger number of smaller networks being affected by something which was an completely unrelated and unintended side-effect of the actual attacks, and thus *poof* you have the recipe for a lot of people talking about it. :) Hopefully that clears up a bit of the situation.

    (tags: bandwidth internet gizmodo traffic cloudflare ddos hacking)

Links for 2013-03-27

Published March 27, 2013

Links for 2013-03-25

Published March 25, 2013

  • The first pillar of agile sysadmin: We alert on what we draw

    ‘One of [the] purposes of monitoring systems was to provide data to allow us, as engineers, to detect patterns, and predict issues before they become production impacting. In order to do this, we need to be capturing data and storing it somewhere which allows us to analyse it. If we care about it – if the data could provide the kind of engineering insight which helps us to understand our systems and give early warning – we should be capturing it. ‘ …. ‘There are a couple of weaknesses in [Nagios’ design]. Assuming we’ve agreed that if we care about a metric enough to want to alert on it then we should be gathering that data for analysis, and graphing it, then we already have the data upon which to base our check. Furthermore, this data is not on the machine we’re monitoring, so our checks don’t in any way add further stress to that machine.’ I would add that if we are alerting on a different set of data from what we collect for graphing, then using the graphs to investigate an alarm may run into problems if they don’t sync up.

    (tags: devops monitoring deployment production sysadmin ops alerting metrics)

  • JPL Institutional Coding Standard for the Java Programming Language

    From JPL’s Laboratory for Reliable Software (LaRS). Great reference; there’s some really useful recommendations here, and good explanations of familiar ones like “prefer composition over inheritance”. Many are supported by FindBugs, too. Here’s the full list:

    compile with checks turned on; apply static analysis; document public elements; write unit tests; use the standard naming conventions; do not override field or class names; make imports explicit; do not have cyclic package and class dependencies; obey the contract for equals(); define both equals() and hashCode(); define equals when adding fields; define equals with parameter type Object; do not use finalizers; do not implement the Cloneable interface; do not call nonfinal methods in constructors; select composition over inheritance; make fields private; do not use static mutable fields; declare immutable fields final; initialize fields before use; use assertions; use annotations; restrict method overloading; do not assign to parameters; do not return null arrays or collections; do not call System.exit; have one concept per line; use braces in control structures; do not have empty blocks; use breaks in switch statements; end switch statements with default; terminate if-else-if with else; restrict side effects in expressions; use named constants for non-trivial literals; make operator precedence explicit; do not use reference equality; use only short-circuit logic operators; do not use octal values; do not use floating point equality; use one result type in conditional expressions; do not use string concatenation operator in loops; do not drop exceptions; do not abruptly exit a finally block; use generics; use interfaces as types when available; use primitive types; do not remove literals from collections; restrict numeric conversions; program against data races; program against deadlocks; do not rely on the scheduler for synchronization; wait and notify safely; reduce code complexity

    (tags: nasa java reference guidelines coding-standards jpl reliability software coding oo concurrency findbugs bugs)

Links for 2013-03-24

Published March 24, 2013

  • KDE’s brush with git repository corruption: post-mortem

    a barely-averted disaster… phew.

    while we planned for the case of the server losing a disk or entirely biting the dust, or the total loss of the VM’s filesystem, we didn’t plan for the case of filesystem corruption, and the way the corruption affected our mirroring system triggered some very unforeseen and pathological conditions. […] the corruption was perfectly mirrored… or rather, due to its nature, imperfectly mirrored. And all data on the anongit [mirrors] was lost.
    One risk demonstrated: by trusting in mirroring, rather than a schedule of snapshot backups covering a wide time range, they nearly had a major outage. Silent data corruption, and code bugs, happen — backups protect against this, but RAID, replication, and mirrors do not. Another risk: they didn’t have a rate limit on project-deletion, which resulted in the “anongit” mirrors deleting their (safe) data copies in response to the upstream corruption. Rate limiting to sanity-check automated changes is vital. What they should have had in place was described by the fix: ‘If a new projects file is generated and is more than 1% different than the previous file, the previous file is kept intact (at 1500 repositories, that means 15 repositories would have to be created or deleted in the span of three minutes, which is extremely unlikely).’

    (tags: rate-limiting case-studies post-mortems kde git data-corruption risks mirroring replication raid bugs backups snapshots sanity-checks automation ops)

  • SpaceX software dev practices

    Metrics rule the roost — I guess there’s been a long history of telemetry in space applications.

    To make software more visible, you need to know what it is doing, he said, which means creating “metrics on everything you can think of”…. Those metrics should cover areas like performance, network utilization, CPU load, and so on. The metrics gathered, whether from testing or real-world use, should be stored as it is “incredibly valuable” to be able to go back through them, he said. For his systems, telemetry data is stored with the program metrics, as is the version of all of the code running so that everything can be reproduced if needed. SpaceX has programs to parse the metrics data and raise an alarm when “something goes bad”. It is important to automate that, Rose said, because forcing a human to do it “would suck”. The same programs run on the data whether it is generated from a developer’s test, from a run on the spacecraft, or from a mission. Any failures should be seen as an opportunity to add new metrics. It takes a while to “get into the rhythm” of doing so, but it is “very useful”. He likes to “geek out on error reporting”, using tools like libSegFault and ftrace. Automation is important, and continuous integration is “very valuable”, Rose said. He suggested building for every platform all of the time, even for “things you don’t use any more”. SpaceX does that and has found interesting problems when building unused code. Unit tests are run from the continuous integration system any time the code changes. “Everyone here has 100% unit test coverage”, he joked, but running whatever tests are available, and creating new ones is useful. When he worked on video games, they had a test to just “warp” the character to random locations in a level and had it look in the four directions, which regularly found problems. “Automate process processes”, he said. Things like coding standards, static analysis, spaces vs. tabs, or detecting the use of Emacs should be done automatically. SpaceX has a complicated process where changes cannot be made without tickets, code review, signoffs, and so forth, but all of that is checked automatically. If static analysis is part of the workflow, make it such that the code will not build unless it passes that analysis step. When the build fails, it should “fail loudly” with a “monitor that starts flashing red” and email to everyone on the team. When that happens, you should “respond immediately” to fix the problem. In his team, they have a full-size Justin Bieber cutout that gets placed facing the team member who broke the build. They found that “100% of software engineers don’t like Justin Bieber”, and will work quickly to fix the build problem.

    (tags: spacex dev coding metrics deplyment production space justin-bieber)

  • on the etymology of “Ketchup”

    ‘the story of ketchup is a story of globalization and centuries of economic domination by a world superpower. But the superpower isn’t America, and the century isn’t ours. Ketchup’s origins in the fermented sauces of China and Southeast Asia mean that those little plastic packets under the seat of your car are a direct result of Chinese and Asian domination of a single global world economy for most of the last millenium.’

    (tags: ketchup china nam-pla food etymology condiments history trade)

Links for 2013-03-23

Published March 23, 2013

  • dumping a JVM heap using gdb

    now this is a neat trick — having been stuck having to flip to spares and do other antics while a long-running heap dump took place, this is a winner.

    Dumping a JVM’s heap is an extremely useful tool for debugging problems with a J2EE application. Unfortunately, when a JVM explodes, using the standard jmap tool can take an inordinate amount of time to execute for lots of different reasons. This leads to extended downtime when a heap dump is attempted and even then, jmap regularly fails. This blog post is intended to outline an alternate method using [gdb] to achieve a heap dump that only requires mere seconds of additional downtime allowing the slow jmap process to happen once the application is back in service.

    (tags: heap-dump gdb heap jvm java via:peakscale gcore core core-dump debugging)

  • Edition – Irish Design

    ‘Edition has a ‘design for life’ philosophy – we think that unique designer-made items can be a part of our everyday lives without costing the earth. We stock affordable, contemporary and functional products (mostly handmade), including jewellery, home-ware, accessories, art and toys. Every item has been carefully selected and are all designed here in Ireland.’

    (tags: edition design ireland art graphics jewellery toys)

Links for 2013-03-21

Published March 21, 2013

Links for 2013-03-20

Published March 20, 2013

Links for 2013-03-19

Published March 19, 2013

Links for 2013-03-18

Published March 18, 2013

Links for 2013-03-16

Published March 16, 2013

  • Roko’s basilisk – RationalWiki

    Wacky transhumanists.

    Roko’s basilisk is notable for being completely banned from discussion on LessWrong, where any mention of it is deleted. Eliezer Yudkowsky, founder of LessWrong, considers the basilisk would not work, but will not explain why because he does not consider open discussion of the notion of acausal trade with possible superintelligences to be provably safe. Silly over-extrapolations of local memes are posted to LessWrong quite a lot; almost all are just downvoted and ignored. But this one, Yudkowsky reacted to hugely, then doubled-down on his reaction. Thanks to the Streisand effect, discussion of the basilisk and the details of the affair soon spread outside of LessWrong. The entire affair is a worked example of spectacular failure at community management and at controlling purportedly dangerous information. Some people familiar with the LessWrong memeplex have suffered serious psychological distress after contemplating basilisk-like ideas — even when they’re fairly sure intellectually that it’s a silly problem.[5] The notion is taken sufficiently seriously by some LessWrong posters that they try to work out how to erase evidence of themselves so a future AI can’t reconstruct a copy of them to torture.[6]

    (tags: transhumanism funny insane stupid singularity ai rokos-basilisk via:maciej lesswrong rationalism superintelligences striesand-effect absurd)

  • How the America Invents Act Will Change Patenting Forever

    Bet you didn’t think the US software patents situation could get worse? wrong!

    “Now it’s really important to be the first to file, and it’s really important to file before somebody else puts a product out, or puts the invention in their product,” says Barr, adding that it will “create a new urgency on the part of everyone to file faster — and that’s going to be a problem for the small inventor.”

    (tags: first-to-file omnishambles uspto swpats patents software-patents law legal)

Links for 2013-03-14

Published March 14, 2013

Links for 2013-03-13

Published March 13, 2013

Links for 2013-03-12

Published March 12, 2013

Links for 2013-03-11

Published March 11, 2013

  • Bunnie Huang’s “Hacking the Xbox” now available as a free PDF

    ‘No Starch Press and I have decided to release this free ebook version of Hacking the Xbox in honor of Aaron Swartz. As you read this book, I hope that you’ll be reminded of how important freedom is to the hacking community and that you’ll be inclined to support the causes that Aaron believed in. I agreed to release this book for free in part because Aaron’s treatment by MIT is not unfamiliar to me. In this book, you will find the story of when I was an MIT graduate student, extracting security keys from the original Microsoft Xbox. You’ll also read about the crushing disappointment of receiving a letter from MIT legal repudiating any association with my work, effectively leaving me on my own to face Microsoft. The difference was that the faculty of my lab, the AI laboratory, were outraged by this treatment. They openly defied MIT legal and vowed to publish my work as an official “AI Lab Memo,” thereby granting me greater negotiating leverage with Microsoft. Microsoft, mindful of the potential backlash from the court of public opinion over suing a legitimate academic researcher, came to a civil understanding with me over the issue.’ This is a classic text on hardware reverse-engineering and the freedom to tinker — strongly recommended.

    (tags: hacking bunnie-huang xbox free hardware drm freedom-to-tinker books reading mit microsoft history)

Links for 2013-03-07

Published March 7, 2013

  • 4 Things Java Programmers Can Learn from Clojure (without learning Clojure)

    ‘1. Use immutable values; 2. Do no work in the constructor; 3. Program to small interfaces; 4. Represent computation, not the world’. Strongly agreed with #1, and the others look interesting too

    (tags: clojure lisp design programming coding java)

  • Tactical Chat: How the U.S. Military Uses IRC to Wage War

    Excellent stuff. Lessons to be learned from this: IRC has some key features that mean it can be useful in this case. 1. simple text, everything supports it, no fancy UI clients are necessary; 2. resilient against lossy/transient/low-bandwidth/high-latency networks; 3. standards-compliant and “battle-hardened” (so to speak); 4. open-source/non-proprietary.

    Despite the U.S. military’s massive spending each year on advanced communications technology, the use of simple text chat or tactical chat has outpaced other systems to become one of the most popular paths for communicating practical information on the battlefield.  Though the use of text chat by the U.S. military first began in the early 1990s, in recent years tactical chat has evolved into a “primary ‘comms’ path, having supplanted voice communications as the primary means of common operational picture (COP) updating in support of situational awareness.”  An article from January 2012 in the Air Land Sea Bulletin describes the value of tactical chat as an effective and immediate communications method that is highly effective in distributed, intermittent, low bandwidth environments which is particularly important with “large numbers of distributed warfighters” who must “frequently jump onto and off of a network” and coordinate with other coalition partners.  Text chat also provides “persistency in situational understanding between those leaving and those assuming command watch duties” enabling a persistent record of tactical decision making. A 2006 thesis from the Naval Postgraduate School states that internet relay chat (IRC) is one of the most widely used chat protocols for military command and control (C2).  Software such as mIRC, a Windows-based chat client, or integrated systems in C2 equipment are used primarily in tactical conditions though efforts are underway to upgrade systems to newer protocols. 
    (via JK)

    (tags: via:jk war irc chat mirc us-military tactical-chat distcomp networking)

  • “Whataboutery”

    Great neologism from Mick Fealty:

    Familiar to anyone who’s followed public debate on Northern Ireland. Some define it as the often multiple blaming and finger pointing that goes on between communities in conflict. Political differences are marked by powerful emotional (often tribal) reactions as opposed to creative conflict over policy and issues. It’s beginning to be known well beyond the bounds of Northern Ireland. […] Evasion may not be the intention but it is the obvious effect. It occurs when individuals are confronted with a difficult or uncomfortable question. The respondent retrenches his/her position and rejigs the question, being careful to pick open a sore point on the part of questioner’s ‘tribe’. He/she then fires the original query back at the inquirer.

    (tags: words etymology whataboutery argument debate northern-ireland mick-fealty slugger-otoole)

  • Dropbox Sync API

    Give your app its own private Dropbox client and leave the syncing to us.

    (tags: apps dropbox synchronization sync ios android api)

  • the real reason Marissa Mayer canned remote Y! employees (apparently)

    After spending months frustrated at how empty Yahoo parking lots were, Mayer consulted Yahoo’s VPN logs to see if remote employees were checking in enough. Mayer discovered they were not — and her decision was made. we’re hearing from people close to Yahoo executives and employees that she made the right decision banning work from home. “The employees at Yahoo are thrilled,” says one source close to the company. “There isn’t massive uprising. The truth is, they’ve all been pissed off that people haven’t been working.”

    (tags: yahoo work remote-work teleworking slacking marissa-mayer funny)

Links for 2013-03-06

Published March 6, 2013

  • Online Schema Change for MySQL

    A tool written by Facebook to ease the pain of online MySQL schema-change migrations.

    Some ALTER TABLE statements take too long form the perspective of some MySQL users. The fast index create feature for the InnoDB plugin in MySQL 5.1 makes this less of an issue but this can still take minutes to hours for a large table and for some MySQL deployments that is too long.   A workaround is to perform the change on a slave first and then promote the slave to be the new master. But this requires a slave located near the master. MySQL 5.0 added support for triggers and some replication systems have been built using triggers to capture row changes. Why not use triggers for this? The openarkkit toolkit did just that with oak-online-alter-table. We have published our version of an online schema change utility (OnlineSchemaChange.php aka OSC).

    (tags: facebook mysql sql schema database migrations ops alter-table)

  • Netflix Queue: Data migration for a high volume web application

    There will come a time in the life of most systems serving data, when there is a need to migrate data to [another] data store while maintaining or improving data consistency, latency and efficiency. This document explains the data migration technique we used at Netflix to migrate the user’s queue data between two different distributed NoSQL storage systems [SimpleDB to Cassandra].

    (tags: cassandra netflix migrations data schema simpledb storage)

  • Monitoring Apache Hadoop, Cassandra and Zookeeper using Graphite and JMXTrans

    nice enough, but a lot of moving parts. It would be nice to see a simpler ZK+Graphite setup using the ‘mntr’ verb

    (tags: graphite monitoring ops zookeeper cassandra hadoop jmx jmxtrans graphs)

  • RFC 6585 – Additional HTTP Status Codes

    includes “429 Too Many Requests”, for rate limits

    (tags: api rfc http reference standards web rest)

  • Curator Framework: Reducing the Complexity of Building Distributed Systems | Marketing Technology

    good +1 for using Netflix’ Curator ZK client library

    (tags: zookeeper curator netflix oss libraries distributed)

  • Netflix Curator

    a high-level API that greatly simplifies using ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations. Some of the features are: Automatic connection management: There are potential error cases that require ZooKeeper clients to recreate a connection and/or retry operations. Curator automatically and transparently (mostly) handles these cases. Cleaner API: simplifies the raw ZooKeeper methods, events, etc.; provides a modern, fluent interface Recipe implementations (see Recipes): Leader election, Shared lock, Path cache and watcher, Distributed Queue, Distributed Priority Queue

    (tags: zookeeper java netflix distcomp libraries oss open-source distributed)

  • OscarGodson.js | What I Learned At Yammer

    some pretty interesting lessons, it turns out: a ‘take what you need’ vacation policy means nobody takes vacations (unsurprising); Yammer actively work to avoid employee burnout (good idea); Yammer A/B test every feature; and Yammer mgmt try to let their devs work autonomously.

    (tags: yammer startups testing analytics culture work)

  • moreutils

    Some really cool-looking UNIX command line utils, packaged in Debian (and therefore in Ubuntu too). A few of these I’ve reimplemented separately, but it’s always good to replace a hack with a more widely available “official” tool. Thanks, Joey Hess!

    sponge: accept input, wait til EOF, then rewrite a file; chronic: runs a command quietly unless it fails; combine: combine the lines in two files using boolean operations; ifdata: get network interface info without parsing ifconfig output; ifne: run a program if the standard input is not empty; isutf8: check if a file or standard input is utf-8; lckdo: execute a program with a lock held; mispipe: pipe two commands, returning the exit status of the first; parallel: run multiple jobs at once; pee: tee standard input to pipes; sponge: soak up standard input and write to a file; ts: timestamp standard input; vidir: edit a directory in your text editor; vipe: insert a text editor into a pipe; zrun: automatically uncompress arguments to command

    (tags: bash shell cli unix scripting via:peakscale joey-hess debian ubuntu tools command-line commands)

  • Test-Driven Infrastructure with Chef

    Interesting idea.

    The book introduces “Infrastructure as Code,” test-driven development, Chef, and cucumber-chef, and then proceeds to a simple example using Chef to provision a shared Linux server. The recipes for the server are developed test-first, demonstrating both the technique and the workflow.

    (tags: tdd chef server provisioning build deploy linux coding ops sysadmin)

  • Peek and poke in the age of Linux

    Neat demo of using ptrace to inject into a running process, just like the good old days ;)

    Some time ago I ran into a production issue where the init process (upstart) stopped behaving properly. Specifically, instead of spawning new processes, it deadlocked in a transitional state. […] What’s worse, upstart doesn’t allow forcing a state transition and trying to manually create and send DBus events didn’t help either. That meant the sane options we were left with were: restart the host (not desirable at all in that scenario); start the process manually and hope auto-respawn will not be needed. Of course there are also some insane options. Why not cheat like in the old times and just PEEK and POKE the process in the right places? The solution used at the time involved a very ugly script driving gdb which probably summoned satan in some edge cases. But edge cases were not hit and majority of hosts recovered without issues.

    (tags: debugging memory linux upstart peek poke ptrace gdb processes hacks)

Links for 2013-03-05

Published March 5, 2013

  • The World Wide Web is Moving to AOL! | Brian Bailey

    brilliant parody of those “we’re so happy to be shutting down!” posts.

    Don’t worry, all of that hard work won’t be wasted. The World Wide Web will remain accessible for 30 days, which will give you plenty of time to update your readers and customers. Each of you will also receive a 30-day free trial for AOL. Look for your CD in the mail soon. Even better, we’ve created an import tool to make it easy to migrate everything you’ve put on the web to American Online! The address will change, of course, but now it will be available to every AOL member. You may find that you don’t need to bother, though. America Online already has groups and pages about almost every topic you can imagine. Take a look around first and you might save yourself a lot of time. There are only so many different ways to say that Citizen Kane was a good movie! We understand that not all of you will become AOL subscribers and not all web sites will move to the new platform. Just to be safe, be sure to print out all of your favorite pages before the end of the month.

    (tags: acquihired acquisitions aol www funny parody humour web)

Links for 2013-03-04

Published March 4, 2013

Links for 2013-03-01

Published March 1, 2013

  • Denominator: A Multi-Vendor Interface for DNS

    the latest good stuff from Netflix.

    Denominator is a portable Java library for manipulating DNS clouds. Denominator has pluggable back-ends, initially including AWS Route53, Neustar Ultra, DynECT, and a mock for testing. We also ship a command line version so it’s easy for anyone to try it out. The reason we built Denominator is that we are working on multi-region failover and traffic sharing patterns to provide higher availability for the streaming service during regional outages caused by our own bugs and AWS issues. To do this we need to directly control the DNS configuration that routes users to each region and each zone. When we looked at the features and vendors in this space we found that we were already using AWS Route53, which has a nice API but is missing some advanced features; Neustar UltraDNS, which has a SOAP based API; and DynECT, which has a REST API that uses a quite different pseudo-transactional model. We couldn’t find a Java based API that grouped together common set of capabilities that we are interested in, so we created one. The idea is that any feature that is supported by more than one vendor API is the highest common denominator, and that functionality can be switched between vendors as needed, or in the event of a DNS vendor outage.

    (tags: dns netflix java tools ops route53 aws ultradns dynect)

  • Making Really Executable Jars

    Who knew? you can make a runnable JAR file!

    There has long been a hack known in some circles, but not widely known, to make jars really executable, in the chmod +x sense. The hack takes advantage of the fact that jar files are zip files, and zip files allow arbitrary cruft to be prepended to the zip file itself (this is how self-extracting zip files work).

    (tags: jars via:netflix shell java executable chmod zip hacks command-line cli)

Links for 2013-02-28

Published February 28, 2013

  • Two surgeons debate the use of cycle helmets

    ‘I am a neurosurgeon and a cyclist, and I am also married to a dedicated cyclist. I wear a cycling helmet and encourage cyclists to wear one. I don’t find that wearing one impedes me in any way. I am under no illusion that it will save me in the event of a high speed collision with a car or lorry (nothing will), but most cycling accidents aren’t of the high-speed variety.’ versus: ‘I am a consultant Trauma orthopaedic surgeon working in Edinburgh and have many years of experience treating cyclists after serious road traffic, cycle sport and commuting cycle injuries. I believe there is no justification for helmet laws or promotional campaigns that portray cycling as a particularly ‘dangerous’ activity, or that make unfounded claims about the effectiveness of helmets. By reducing cycle use even slightly, helmet laws or promotion campaigns are likely to cause a significant net disbenefit to public health, regardless of the effectiveness or otherwise of helmets.’ Generally a lot of sense on either side.

    (tags: helmets cycling bicycles health safety surgeons doctors)

  • Storm and Hadoop: Convergence of Big-Data and Low-Latency Processing

    Yahoo! are going big with Storm for their next-generation internal cloud platform: ‘Yahoo! engineering teams are developing technologies to enable Storm applications and Hadoop applications to be hosted on a single cluster. • We have enhanced Storm to support Hadoop style security mechanism (including Kerberos authentication), and thus enable Storm applications authorized to access Hadoop datasets on HDFS and HBase. • Storm is being integrated into Hadoop YARN for resource management. Storm-on-YARN enables Storm applications to utilize the computation resources in our tens of thousands of Hadoop computation nodes. YARN is used to launch Storm application master (Nimbus) on demand, and enables Nimbus to request resources for Storm application slaves (Supervisors).’

    (tags: yahoo yarn cloud-computing private-clouds big-data latency storm hadoop elastic-computing hbase)

  • Trojan paralyses speed cameras in Moscow

    what a coincidence! (via Tony Finch)

    (tags: virus trojans malware via:fanf kaspersky)

  • IOS TCP wifi optimizer

    Basically, tweaking a few suboptimal sysctls to optimize for 802.11b/n; requires a Jailbroken IOS device. I’m surprised that Apple defaulted segment size to 512 to be honest, and disabling delayed ACKs sounds like it might be useful (see also http://www.stuartcheshire.org/papers/NagleDelayedAck/).

    TCP optimizer modifies a few settings inside iOS, including increasing the TCP receive buffer from 131072 to 292000, disabling TCP delayed ACK’s, allowing a maximum of 16 un-ACK’d packets instead of 8 and set the default packet size to 1460 instead of 512. These changes won’t only speed up your YouTube videos, they’ll also improve your internet connection’s performance overall, including Wi-Fi network connectivity.

    (tags: tcp performance tuning ios apple wifi wireless 802.11n sysctl ip)

  • It’s the Sugar, Folks

    A study published in the Feb. 27 issue of the journal PLoS One links increased consumption of sugar with increased rates of diabetes by examining the data on sugar availability and the rate of diabetes in 175 countries over the past decade. And after accounting for many other factors, the researchers found that increased sugar in a population’s food supply was linked to higher diabetes rates independent of rates of obesity. In other words, according to this study, obesity doesn’t cause diabetes: sugar does. The study demonstrates this with the same level of confidence that linked cigarettes and lung cancer in the 1960s. As Rob Lustig, one of the study’s authors and a pediatric endocrinologist at the University of California, San Francisco, said to me, “You could not enact a real-world study that would be more conclusive than this one.”

    (tags: nytimes health food via:fanf sugar eating diabetes papers medicine)

Links for 2013-02-26

Published February 26, 2013

Links for 2013-02-25

Published February 25, 2013

  • UnoDNS

    ‘Watch Netflix USA, Hulu, Pandora, BBC iPlayer, and more in [sic] anywhere you live!’ — seems to use similar techniques to tunlr.net, looks like it works for my Netflix

    (tags: netflix dns tv tunnelling drm networking spotify hulu)

  • Cassandra, Hive, and Hadoop: How We Picked Our Analytics Stack

    reasonably good whole-stack performance testing and analysis; HBase, Riak, MongoDB, and Cassandra compared. Riak did pretty badly :(

    (tags: riak mongodb cassandra hbase performance analytics hadoop hive big-data storage databases nosql)

  • Big Data Analytics at Netflix. Interview with Christos Kalantzis and Jason Brown.

    Good interview with the Cassandra guys at Netflix, and some top Mongo-bashing in the comments

    (tags: cassandra netflix user-stories testimonials nosql storage ec2 mongodb)

  • Werner Knaupp – Acrylbilder

    my favourite art of the moment. Thick, heavy layers of acrylic black and white paint, evoking the stormy Atlantic (brr). Gallery Bode, which showed this in Nuremberg in 2011, wrote the following at http://www.bode-galerie.de/en/exhibitions/schwarz_weiss :

    Gallery Bode is pleased to constitute the cooperation with Werner Knaupp with an exhibition of a new workseries. The exhibition showcases artworks out of the series “Westmen Isles”. […] The journeys to Iceland are a background to the development of this new workseries. These paintings are telling of a forbidding nature. The beholder can’t take a [safe] position but he is involved into the event which becomes comprehensible in a nearly physical way. These pictures of a overwhelming nature could be traced back to Knaupp’s confrontation with the force of nature while his journeys. The experience of this force pushes the limits of human being and evokes primal fear. With the abdication of colours the artworks reach dynamic. This foots on the consistency of colour and on the changing between reality and abstraction. In an art historical view the new black and white paintings detached themselves from traditional landscape painting. Werner Knaupp implements the pure force of nature into pure painting, to visualise the force fields of nature. The beholder experiences with these artworks a nature without human dimension. In Werner Knaupp’s Oeuvre the “Westmen Isles” paintings are a new expression of his examination with existential fundamental questions.

    (tags: germany art painting werner-knaupp paintings monochrome sea iceland)

Links for 2013-02-22

Published February 22, 2013

  • Indymedia: It’s time to move on

    Our decision to curtail publishing on the Nottingham Indymedia site and call a meeting is an attempt to create a space for new ideas. We are not interested in continuing along the slow but certain path to total irrelevance but want to draw in new people and start off in new directions whilst remaining faithful to the underlying principles of Indymedia.

    (tags: indymedia community communication web anonymity publishing left-wing)

  • How to revert a faulty merge in git

    omgwtf, this is pretty horrific.

    (tags: merging git merge omgwtf version-control branching)

  • #AltDevBlogADay » Latency Mitigation Strategies

    John Carmack on the low-latency coding techniques used to support head mounted display devices.

    Virtual reality (VR) is one of the most demanding human-in-the-loop applications from a latency standpoint. The latency between the physical movement of a user’s head and updated photons from a head mounted display reaching their eyes is one of the most critical factors in providing a high quality experience. Human sensory systems can detect very small relative delays in parts of the visual or, especially, audio fields, but when absolute delays are below approximately 20 milliseconds they are generally imperceptible. Interactive 3D systems today typically have latencies that are several times that figure, but alternate configurations of the same hardware components can allow that target to be reached. A discussion of the sources of latency throughout a system follows, along with techniques for reducing the latency in the processing done on the host system.

    (tags: head-mounted-display display ui latency vision coding john-carmack)

Links for 2013-02-21

Published February 21, 2013

  • Distributed Streams Algorithms for Sliding Windows [PDF]

    ‘Massive data sets often arise as physically distributed, parallel data streams, and it is important to estimate various aggregates and statistics on the union of these streams. This paper presents algorithms for estimating aggregate functions over a “sliding window” of the N most recent data items in one or more streams. […] Our results are obtained using a novel family of synopsis data structures called waves.’

    (tags: waves papers streaming algorithms percentiles histogram distcomp distributed aggregation statistics estimation streams)

  • good blog post on histogram-estimation stream processing algorithms

    After reviewing several dozen papers, a score or so in depth, I identified two data structures that appear to enable us to answer these recency and frequency queries: exponential histograms (from “Maintaining Stream Statistics Over Sliding Windows” by Datar et al.) and waves (from “Distributed Streams Algorithms for Sliding Windows” by Gibbons and Tirthapura). Both of these data structures are used to solve the so-called counting problem, the problem of determining, with a bound on the relative error, the number of 1s in the last N units of time. In other words, the data structures are able to answer the question: how many 1s appeared in the last n units of time within a factor of Error (e.g., 50%). The algorithms are neat, so I’ll present them briefly.

    (tags: streams streaming stream-processing histograms percentiles estimation waves statistics algorithms)

  • Timelike 2: everything fails all the time

    Fantastic post on large-scale distributed load balancing strategies from @aphyr. Random and least-conns routing comes out on top in his simulation (although he hasn’t yet tried Marc Brooker’s two-randoms routing strategy)

    (tags: via:hn routing distributed least-conns load-balancing round-robin distcomp networking scaling)

  • Marc Brooker’s “two-randoms” load balancing approach

    Marc Brooker on this interesting load-balancing algorithm, including simulation results:

    Using stale data for load balancing leads to a herd behavior, where requests will herd toward a previously quiet host for much longer than it takes to make that host very busy indeed. The next refresh of the cached load data will put the server high up the load list, and it will become quiet again. Then busy again as the next herd sees that it’s quiet. Busy. Quiet. Busy. Quiet. And so on. One possible solution would be to give up on load balancing entirely, and just pick a host at random. Depending on the load factor, that can be a good approach. With many typical loads, though, picking a random host degrades latency and reduces throughput by wasting resources on servers which end up unlucky and quiet. The approach taken by the studies surveyed by Mitzenmacher is to try two hosts, and pick the one with the least load. This can be done directly (by querying the hosts) but also works surprisingly well on cached load data. […] Best of 2 is good because it combines the best of both worlds: it uses real information about load to pick a host (unlike random), but rejects herd behavior much more strongly than the other two approaches.
    Having seen what Marc has worked on, and written, inside Amazon, I’d take this very seriously… cool to see he is blogging externally too.

    (tags: algorithm load-balancing distcomp distributed two-randoms marc-brooker least-conns)

  • Can regular expressions parse HTML?

    ‘a summary of the main points: The “regular expressions” used by programmers have very little in common with the original notion of regularity in the context of formal language theory. Regular expressions (at least PCRE) can match all context-free languages. As such they can also match well-formed HTML and pretty much all other programming languages. Regular expressions can match at least some context-sensitive languages. Matching of regular expressions is NP-complete. As such you can solve any other NP problem using regular expressions.’

    (tags: compsci regexps regular-expressions programming np-complete chomsky-grammar context-free languages)

Links for 2013-02-19

Published February 19, 2013

Links for 2013-02-18

Published February 18, 2013

  • Fatcache

    from Twitter — ‘a cache for your big data. Even though memory is thousand times faster than SSD, network connected SSD-backed memory makes sense, if we design the system in a way that network latencies dominate over the SSD latencies by a large factor. To understand why network connected SSD makes sense, it is important to understand the role distributed memory plays in large-scale web architecture. In recent years, terabyte-scale, distributed, in-memory caches have become a fundamental building block of any web architecture. In-memory indexes, hash tables, key-value stores and caches are increasingly incorporated for scaling throughput and reducing latency of persistent storage systems. However, power consumption, operational complexity and single node DRAM cost make horizontally scaling this architecture challenging. The current cost of DRAM per server increases dramatically beyond approximately 150 GB, and power cost scales similarly as DRAM density increases. Fatcache extends a volatile, in-memory cache by incorporating SSD-backed storage.’

    (tags: twitter ssd cache caching memcached memcache memory network storage)

  • Passively Monitoring Network Round-Trip Times – Boundary

    ‘how Boundary uses [TCP timestamps] to calculate round-trip times (RTTs) between any two hosts by passively monitoring TCP traffic flows, i.e., without actively launching ICMP echo requests (pings). The post is primarily an overview of this one aspect of TCP monitoring, it also outlines the mechanism we are using, and demonstrates its correctness.’

    (tags: tcp boundary monitoring network ip passive-monitoring rtt timestamping)

  • drug cartel-controlled mobile comms networks

    “The Mexican military has recently broken up several secret telecommunications networks that were built and controlled by drug cartels so they could coordinate drug shipments, monitor their rivals and orchestrate attacks on the security forces. A network that was dismantled just last week provided cartel members with cellphone and radio communications across four northeastern states. The network had coverage along almost 500 miles of the Texas border and extended nearly another 500 miles into Mexico’s interior. Soldiers seized 167 antennas, more than 150 repeaters and thousands of cellphones and radios that operated on the system. Some of the remote antennas and relay stations were powered with solar panels.”

    (tags: mexico drugs networks mobile-phones crime)

  • Heroku finds out that distributed queueing is hard

    Stage 3 of the Rap Genius/Heroku blog drama. Summary (as far as I can tell): Heroku gave up on a fully-synchronised load-balancing setup (“intelligent routing”), since it didn’t scale, in favour of randomised queue selection; they didn’t sufficiently inform their customers, and metrics and docs were not updated to make this change public; the pessimal case became pretty damn pessimal; a customer eventually noticed and complained publicly, creating a public shit-storm. Comments: 1. this is why you monitor real HTTP request latency (scroll down for crazy graphs!). 2. include 90/99 percentiles to catch the “tail” of poorly-performing requests. 3. Load balancers are hard. http://aphyr.com/posts/277-timelike-a-network-simulator has more info on the intricacies of distributed load balancing — worth a read.

    (tags: heroku rap-genius via:hn networking distcomp distributed load-balancing ip queueing percentiles monitoring)

  • Unhelpful Graphite Tips

    10 particularly good — actually helpful — tips on using the Graphite metric graphing system

    (tags: graphite ops metrics service-metrics graphing ui dataviz)

  • Literate Jenks Natural Breaks and How The Idea Of Code is Lost

    A crazy amount of code archaeology to discover exactly an algorithm — specifically ‘Jenks natural breaks”, works, after decades of cargo-cult copying (via Nelson): ‘I spent a day reading the original text and decoding as much as possible of the code’s intention, so that I could write a ‘literate’ implementation. My definition of literate is highly descriptive variable names, detailed and narrative comments, and straightforward code with no hijinks. So: yes, this isn’t the first implementation of Jenks in Javascript. And it took me several times longer to do things this way than to just get the code working. But the sad and foreboding state of this algorithm’s existing implementations said that to think critically about this code, its result, and possibilities for improvement, we need at least one version that’s clear about what it’s doing.’

    (tags: jenks-natural-breaks algorithms chloropleth javascript reverse-engineering history software copyright via:nelson)

  • don’t order a Raspberry Pi from RS

    I’ve been waiting 24 days for mine so far. Frankly amazing they are so apparently inept, particularly since it seems in breach of EU distance selling regulation if they go beyond 30 days without an update. They’ve just posted this:

    Quick update- we received our delivery of raspberry pi’s last week and as of Friday we had shipped up to order reference 1010239854. We will continue daily to get your orders shipped out as quickly as we possibly can; so that you will all receive your raspberry pi’s shortly. Many thanks everyone for your patience and again apologies for the delay in the dispatch update message on the Pi Store which I know has caused some confusion.

    (tags: rs raspberry-pi inept etailers uk e-commerce shopping hardware)

  • more details on the UK distance selling regulations governing Raspberry Pi RS orders

    ‘my understanding is that according to the Distance Selling Regulations […], unless you agreed otherwise with RS, then they were obligated to fulfill their side of the contract within thirty days from the day after you ordered, and if they were unable to do so they were also obligated to inform you that they could not and repay you within thirty days;ons (more info here in a nice, easy-to-read format), unless you agreed otherwise with RS, then they were obligated to fulfill their side of the contract within thirty days from the day after you ordered, and if they were unable to do so they were also obligated to inform you that they could not and repay you within thirty days’

    (tags: rs shopping etailers inept distance-selling uk law)

Links for 2013-02-12

Published February 12, 2013

Links for 2013-02-11

Published February 11, 2013

Links for 2013-02-09

Published February 9, 2013