Skip to content

Justin's Linklog Posts

Links for 2015-08-03

Published August 3, 2015

Links for 2015-07-30

Published July 30, 2015

  • danilop/yas3fs · GitHub

    YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.

    (tags: aws s3 s3fs yas3fs filesystems fuse sns)

  • danilop/runjop · GitHub

    RunJOP (Run Just Once Please) is a distributed execution framework to run a command (i.e. a job) only once in a group of servers [built using AWS DynamoDB and S3].
    nifty! Distributed cron is pretty easy when you’ve got Dynamo doing the heavy lifting.

    (tags: dynamodb cron distributed-cron scheduling runjop danilop hacks aws ops)

Links for 2015-07-29

Published July 29, 2015

Links for 2015-07-28

Published July 28, 2015

  • Taming Complexity with Reversibility

    This is a great post from Kent Beck, putting a lot of recent deployment/rollout patterns in a clear context — that of supporting “reversibility”:

    Development servers. Each engineer has their own copy of the entire site. Engineers can make a change, see the consequences, and reverse the change in seconds without affecting anyone else. Code review. Engineers can propose a change, get feedback, and improve or abandon it in minutes or hours, all before affecting any people using Facebook. Internal usage. Engineers can make a change, get feedback from thousands of employees using the change, and roll it back in an hour. Staged rollout. We can begin deploying a change to a billion people and, if the metrics tank, take it back before problems affect most people using Facebook. Dynamic configuration. If an engineer has planned for it in the code, we can turn off an offending feature in production in seconds. Alternatively, we can dial features up and down in tiny increments (i.e. only 0.1% of people see the feature) to discover and avoid non-linear effects. Correlation. Our correlation tools let us easily see the unexpected consequences of features so we know to turn them off even when those consequences aren’t obvious. IRC. We can roll out features potentially affecting our ability to communicate internally via Facebook because we have uncorrelated communication channels like IRC and phones. Right hand side units. We can add a little bit of functionality to the website and turn it on and off in seconds, all without interfering with people’s primary interaction with NewsFeed. Shadow production. We can experiment with new services under real load, from a tiny trickle to the whole flood, without affecting production. Frequent pushes. Reversing some changes require a code change. On the website we never more than eight hours from the next schedule code push (minutes if a fix is urgent and you are willing to compensate Release Engineering). The time frame for code reversibility on the mobile applications is longer, but the downward trend is clear from six weeks to four to (currently) two. Data-informed decisions. (Thanks to Dave Cleal) Data-informed decisions are inherently reversible (with the exceptions noted below). “We expect this feature to affect this metric. If it doesn’t, it’s gone.” Advance countries. We can roll a feature out to a whole country, generate accurate feedback, and roll it back without affecting most of the people using Facebook. Soft launches. When we roll out a feature or application with a minimum of fanfare it can be pulled back with a minimum of public attention. Double write/bulk migrate/double read. Even as fundamental a decision as storage format is reversible if we follow this format: start writing all new data to the new data store, migrate all the old data, then start reading from the new data store in parallel with the old.
    We do a bunch of these in work, and the rest are on the to-do list. +1 to these!

    (tags: software deployment complexity systems facebook reversibility dark-releases releases ops cd migration)

Links for 2015-07-27

Published July 27, 2015

  • Benchmarking GitHub Enterprise – GitHub Engineering

    Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)

    (tags: github listen-backlog tcp debugging timeouts load-testing benchmarking testing ops linux)

  • How .uk came to be (and why it’s not .gb)

    WB: By the late 80s the IANA [the Internet Assigned Numbers Authority, set up in 1988 to manage global IP address allocations] was trying to get all those countries that were trying to join the internet to use the ISO 3166 standard for country codes. It was used for all sorts of things?—?you see it on cars, “GB” for the UK. […] At that point, we’re faced with a problem that Jon Postel would like to have changed it to .gb to be consistent with the rest of the world. Whereas .uk had already been established, with a few tens of thousands of domain names with .uk on them. I remember chairing one of the JANET net workshops that were held every year, and the Northern Irish were adamant that they were part of the UK?—?so the consensus was, we’d try and keep .uk, we’d park .gb and not use it. PK: I didn’t particularly want to change to .gb because I was responsible for Northern Ireland as well. And what’s more, there was a certain question as to whether a research group in the US should be allowed to tell the British what to do. So this argy-bargy continued for a little while and, in the meantime, one of my clients was the Ministry of Defence, and they decided they couldn’t wait this long, and they decided I was going to lose the battle, and so bits of MOD went over to .gb?—?I didn’t care, as I was running .gb and .uk in any case.

    (tags: dot-uk history internet dot-gb britain uk northern-ireland ireland janet)

  • That time the Internet sent a SWAT team to my mom’s house – Boing Boing

    The solution is for social media sites and the police to take threats or jokes about swatting, doxxing, and organized crime seriously. Tweeting about buying a gun and shooting up a school would be taken seriously, and so should the threat of raping, doxxing, swatting or killing someone. Privacy issues and online harassment are directly linked, and online harassment isn’t going anywhere. My fear is that, in reaction to online harassment, laws will be passed that will break down our civil freedoms and rights online, and that more surveillance will be sold to users under the guise of safety. More surveillance, however, would not have helped me or my mother. A platform that takes harassment and threats seriously instead of treating them like jokes would have.

    (tags: twitter gamergate 4chan 8chan privacy doxxing swatting harrassment threats social-media facebook law feminism)

  • Why Google’s Deep Dream Is Future Kitsch

    Deep Dream estranges us from our fears, perhaps, but it doesn’t make them go away. It’s easy to discuss Deep Dream as an independent creature, a foreign intelligence that we interact with for fun. Yet like all kitsch, it comes straight back to its creators.

    (tags: kitsch deep-dream art graphics google inceptionism)

  • It’s Not Climate Change?—?It’s Everything Change

    now this is a Long Read. the inimitable Margaret Atwood on climate change, beautifully illustrated

    (tags: climate climate-change margaret-atwood long-reads change life earth green future)

  • In Praise of the AK-47 — Dear Design Student — Medium

    While someone can certainly make the case that an AK-47, or any other kind of gun or rifle is designed, nothing whose primary purpose is to take away life can be said to be designed well. And that attempting to separate an object from its function in order to appreciate it for purely aesthetic reasons, or to be impressed by its minimal elegance, is a coward’s way of justifying the death they’ve designed into the word, and the money with which they’re lining their pockets.

    (tags: design ux ak-47 kalashnikov guns function work)

Links for 2015-07-22

Published July 22, 2015

Links for 2015-07-21

Published July 21, 2015

  • Java lambdas and performance

    Lambdas in Java 8 introduce some unpredictable performance implications, due to reliance on escape analysis to eliminate object allocation on every lambda invocation. Peter Lawrey has some details

    (tags: lambdas java-8 java performance low-latency optimization peter-lawrey coding escape-analysis)

  • Mikhail Panchenko’s thoughts on the July 2015 CircleCI outage

    an excellent followup operational post on CircleCI’s “database is not a queue” outage

    (tags: database-is-not-a-queue mysql sql databases ops outages postmortems)

  • Men who harass women online are quite literally losers, new study finds

    (1) players are anonymous, and the possibility of “policing individual behavior is almost impossible”; (2) they only encounter each other a few times in passing — it’s very possible to hurl an expletive at another player, and never “see” him or her again; and (3) finally, and perhaps predictably, the sex-ratio of players is biased pretty heavily toward men. (A 2014 survey of gender ratios on Reddit found that r/halo was over 95 percent male.) [….] In each of these environments, Kasumovic suggests, a recent influx of female participants has disrupted a pre-existing social hierarchy. That’s okay for the guys at the top — but for the guys at the bottom, who stand to lose more status, that’s very threatening. (It’s also in keeping with the evolutionary framework on anti-lady hostility, which suggests sexism is a kind of Neanderthal defense mechanism for low-status, non-dominant men trying to maintain a shaky grip on their particular cave’s supply of women.) “As men often rely on aggression to maintain their dominant social status,” Kasumovic writes, “the increase in hostility towards a woman by lower-status males may be an attempt to disregard a female’s performance and suppress her disturbance on the hierarchy to retain their social rank.”

    (tags: losers sexism mysogyny women halo gaming gamergate 4chan abuse harrassment papers bullying social-status)

  • The old suburban office park is the new American ghost town – The Washington Post

    Most analyses of the market indicate that office parks simply aren’t as appealing or profitable as they were in the 20th century and that Americans just aren’t as keen to cloister themselves in workspaces that are reachable only by car.

    (tags: cbd cities work life office-parks commuting america history workplaces)

  • HACKERS REMOTELY KILL A JEEP ON THE HIGHWAY—WITH ME IN IT

    Jaysus, this is terrifying.

    Miller and Valasek’s full arsenal includes functions that at lower speeds fully kill the engine, abruptly engage the brakes, or disable them altogether. The most disturbing maneuver came when they cut the Jeep’s brakes, leaving me frantically pumping the pedal as the 2-ton SUV slid uncontrollably into a ditch.
    Avoid any car which supports this staggeringly-badly-conceived Uconnect feature:
    All of this is possible only because Chrysler, like practically all carmakers, is doing its best to turn the modern automobile into a smartphone. Uconnect, an Internet-connected computer feature in hundreds of thousands of Fiat Chrysler cars, SUVs, and trucks, controls the vehicle’s entertainment and navigation, enables phone calls, and even offers a Wi-Fi hot spot.
    :facepalm: Also, Chrysler’s response sucks: “Chrysler’s patch must be manually implemented via a USB stick or by a dealership mechanic.”

    (tags: hacking security cars driving safety brakes jeeps chrysler fiat uconnect can-bus can)

Links for 2015-07-20

Published July 20, 2015

Links for 2015-07-17

Published July 17, 2015

  • Angela Merkel told a sobbing girl she couldn’t save her from deportation. It was a lie. – Vox

    Argentina has, as a matter of constitutional law, effectively open borders. There are no caps or quotas or lottery systems. You can move there legally if you have an employer or family member to sponsor you. That’s all you need. If you don’t have a sponsor, and make your way in illegally, you’re recognized as an “irregular migrant.” Discrimination against irregular migrants in health care or education is illegal, and deportation in noncriminal cases is exceptionally rare. Large-scale amnesties are the norm. Obviously Argentina is not nearly as rich as Germany or the US or the UK. But it’s considerably richer than three of its neighbors (Bolivia, Paraguay, and Brazil). And yet it doesn’t try hard to keep their residents out. It welcomes them — as it should. “One could have expected catastrophe—an uncontrollable flow of poorer immigrants streaming into the country coupled with angry public backlash,” Elizabeth Slater writes in the World Policy Journal. “That hasn’t happened.” Angela Merkel clearly expects catastrophe if she lets people like this weeping young Palestinian girl stay in Germany. That catastrophe is simply a myth; it wouldn’t happen. What would happen is that Germany’s economy would grow, its culture would grow richer, and that girl and more like her could see their lives improve immeasurably.

    (tags: argentina immigration angela-merkel germany eu migrants deportation economics)

Links for 2015-07-16

Published July 16, 2015

Links for 2015-07-15

Published July 15, 2015

Links for 2015-07-14

Published July 14, 2015

Links for 2015-07-13

Published July 13, 2015

  • OkHttp

    A new HTTP client library for Android and Java, with a lot of nice features:

    HTTP/2 and SPDY support allows all requests to the same host to share a socket. Connection pooling reduces request latency (if SPDY isn’t available). Transparent GZIP shrinks download sizes. Response caching avoids the network completely for repeat requests. OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails. Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.

    (tags: android http java libraries okhttp http2 spdy microservices jdk)

  • Eircode tech specs

    via Ossian.

    (tags: via:smytho tech-specs specs eircode addresses geocoding ireland mapping)

  • AWS Best Practices for DDoS Resiliency [pdf]

    Reasonably solid white paper

    (tags: ddos amazon aws security dos whitepapers pdf)

Links for 2015-07-11

Published July 11, 2015

Links for 2015-06-25

Published June 25, 2015

Links for 2015-06-23

Published June 23, 2015

Links for 2015-06-22

Published June 22, 2015

Links for 2015-06-21

Published June 21, 2015

  • jwz on Inceptionism

    “Shoggoth ovipositors”:

    So then they reach inside to one of the layers and spin the knob randomly to fuck it up. Lower layers are edges and curves. Higher layers are faces, eyes and shoggoth ovipositors. [….] But the best part is not when they just glitch an image — which is a fun kind of embossing at one end, and the “extra eyes” filter at the other — but is when they take a net trained on some particular set of objects and feed it static, then zoom in, and feed the output back in repeatedly. That’s when you converge upon the platonic ideal of those objects, which — it turns out — tend to be Giger nightmare landscapes. Who knew. (I knew.)
    This stuff is still boggling my mind. All those doggy faces! That is one dog-obsessed ANN.

    (tags: neural-networks ai jwz funny shoggoths image-recognition hr-giger art inceptionism)

Links for 2015-06-19

Published June 19, 2015

Links for 2015-06-18

Published June 18, 2015

  • Inceptionism: Going Deeper into Neural Networks

    This is amazing, and a little scary.

    If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge. Again, we just start with an existing image and give it to our neural net. We ask the network: “Whatever you see there, I want more of it!” This creates a feedback loop: if a cloud looks a little bit like a bird, the network will make it look more like a bird. This in turn will make the network recognize the bird even more strongly on the next pass and so forth, until a highly detailed bird appears, seemingly out of nowhere.
    An enlightening comment from the G+ thread:
    This is the most fun we’ve had in the office in a while. We’ve even made some of those ‘Inceptionistic’ art pieces into giant posters. Beyond the eye candy, there is actually something deeply interesting in this line of work: neural networks have a bad reputation for being strange black boxes that that are opaque to inspection. I have never understood those charges: any other model (GMM, SVM, Random Forests) of any sufficient complexity for a real task is completely opaque for very fundamental reasons: their non-linear structure makes it hard to project back the function they represent into their input space and make sense of it. Not so with backprop, as this blog post shows eloquently: you can query the model and ask what it believes it is seeing or ‘wants’ to see simply by following gradients. This ‘guided hallucination’ technique is very powerful and the gorgeous visualizations it generates are very evocative of what’s really going on in the network.?

    (tags: art machine-learning algorithm inceptionism research google neural-networks learning dreams feedback graphics)

Links for 2015-06-17

Published June 17, 2015

Links for 2015-06-15

Published June 15, 2015

  • How We Moved Our API From Ruby to Go and Saved Our Sanity

    Parse on their ditching-Rails story. I haven’t heard a nice thing about Ruby or Rails as an operational, production-quality platform in a long time :(

    (tags: go ruby rails ops parse languages platforms)

  • VPC Flow Logs

    we are introducing Flow Logs for the Amazon Virtual Private Cloud.  Once enabled for a particular VPC, VPC subnet, or Elastic Network Interface (ENI), relevant network traffic will be logged to CloudWatch Logs for storage and analysis by your own applications or third-party tools. You can create alarms that will fire if certain types of traffic are detected; you can also create metrics to help you to identify trends and patterns. The information captured includes information about allowed and denied traffic (based on security group and network ACL rules). It also includes source and destination IP addresses, ports, the IANA protocol number, packet and byte counts, a time interval during which the flow was observed, and an action (ACCEPT or REJECT).

    (tags: ec2 aws vpc logging tracing ops flow-logs network tcpdump packets packet-capture)

  • Tim Hunt “jokes” about women scientists. Or not. (with image, tweets) · deborahblum · Storify

    ‘[Tim Hunt] said that while he meant to be ironic, he did think it was hard to collaborate with women because they are too emotional – that he was trying to be honest about the problems.’ So much for the “nasty twitter took my jokes seriously” claims then.

    (tags: twitter science misogyny women tim-hunt deborah-blum journalism)

  • Why I dislike systemd

    Good post, and hard to disagree.

    One of the “features” of systemd is that it allows you to boot a system without needing a shell at all. This seems like such a senseless manoeuvre that I can’t help but think of it as a knee-jerk reaction to the perception of Too Much Shell in sysv init scripts. In exactly which universe is it reasonable to assume that you have a running D-Bus service (or kdbus) and a filesystem containing unit files, all the binaries they refer to, all the libraries they link against, and all the configuration files any of them reference, but that you lack that most ubiquitous of UNIX binaries, /bin/sh?

    (tags: history linux unix systemd bsd system-v init ops dbus)

  • Adrian Colyer reviews the Twitter Heron paper

    ouch, really sounds like Storm didn’t cut the muster. ‘It’s hard to imagine something more damaging to Apache Storm than this. Having read it through, I’m left with the impression that the paper might as well have been titled “Why Storm Sucks”, which coming from Twitter themselves is quite a statement.’ If I was to summarise the lessons learned, it sounds like: backpressure is required; and multi-tenant architectures suck.

    (tags: storm twitter heron big-data streaming realtime backpressure)

Links for 2015-06-14

Published June 14, 2015

  • Security theatre at Allied Irish Banks

    Allied Irish Banks’s web and mobile banking portals are ludicrously insecure. Vast numbers of accounts have easily-guessable registration numbers and are thus ‘protected’ by a level of security that is twice as easy to crack as would be provided by a single password containing only two lowercase letters. A person of malicious intent could easily gain access to hundreds, possibly thousands, of accounts as well as completely overwhelm the branch network by locking an estimated several 100,000s of people out of their online banking. Both AIB and the Irish Financial Services Ombudsman have refused to respond meaningfully to multiple communications each in which these concerns were raised privately.

    (tags: aib banking security ireland hacking ifso online-banking)

  • Leveraging AWS to Build a Scalable Data Pipeline

    Nice detailed description of an auto-scaled SQS worker pool

    (tags: sqs aws ec2 auto-scaling asg worker-pools architecture scalability)

Links for 2015-06-13

Published June 13, 2015

  • China’s Spies Hit the Blackmail Jackpot With Data on 4 Million Federal Workers

    The Daily Beast is scathing re the OPM hack:

    Here’s where things start to get scary. Whoever has OPM’s records knows an astonishing amount about millions of federal workers, members of the military, and security clearance holders. They can now target those Americans for recruitment or influence. After all, they know their vices, every last one—the gambling habit, the inability to pay bills on time, the spats with former spouses, the taste for something sexual on the side—since all that is recorded in security clearance paperwork. (To get an idea of how detailed this gets, you can see the form, called an SF86, here.) Speaking as a former counterintelligence officer, it really doesn’t get much worse than this.

    (tags: daily-beast sf86 clearance us-government america china cyberwar hacking opm privacy)

Links for 2015-06-11

Published June 11, 2015

  • Facebook Infer

    New static analysis goodnews, freshly open-sourced by Facebook:

    Facebook Infer uses logic to do reasoning about a program’s execution, but reasoning at this scale — for large applications built from millions of lines of source code — is hard. Theoretically, the number of possibilities that need to be checked is more than the number of estimated atoms in the observable universe. Furthermore, at Facebook our code is not a fixed artifact but an evolving system, updated frequently and concurrently by many developers. It is not unusual to see more than a thousand modifications to our mobile code submitted for review in a given day. The requirements on the program analyzer then become even more challenging because we expect a tool to report quickly on these code modifications — in the region of 10 minutes — to fit in with developers’ workflow. Coping with this scale and velocity requires advanced mathematical techniques. Facebook Infer uses two such techniques: separation logic and bi-abduction. Separation logic is a theory that allows Facebook Infer’s analysis to reason about small, independent parts of the application storage, rather than having to consider the entirety of the memory potentially at every step. That would be a daunting task on modern processors with their large addressable virtual memories. Bi-abduction is a logical inference technique that allows Facebook Infer to discover properties about the behavior of independent parts of the application code. By storing these properties between runs, Facebook Infer needs to analyze only the parts of the software that have changed, reusing the results of its previous analysis where it can. By combining these approaches, our analyzer is able to find complex problems in modifications to an application built from millions of lines of code, in minutes.
    (via Bryan O’Sullivan)

    (tags: via:bos infer facebook static-analysis lint code java ios android coding bugs)

  • The Tamborzão Goes to Thailand

    This is great. the story of how cheesy funk carioca tune “A Minha Amiga Fran” managed to become “Kawo Kawo” and become a massive hit in Thailand

    (tags: thai brazil carioca music dance-music kawo-kawo)

Links for 2015-06-10

Published June 10, 2015

Links for 2015-06-08

Published June 8, 2015

Links for 2015-06-04

Published June 4, 2015

Links for 2015-06-03

Published June 3, 2015

  • Performance Testing at LMAX

    Good series of blog posts on the LMAX trading platform’s performance testing strategy — they capture live traffic off the wire, then build statistical models simulating its features. See also http://epickrram.blogspot.co.uk/2014/07/performance-testing-at-lmax-part-two.html and http://epickrram.blogspot.co.uk/2014/08/performance-testing-at-lmax-part-three.html .

    (tags: performance testing tests simulation latency lmax trading sniffing packet-capture)

  • The Violence of Algorithms: Why Big Data Is Only as Smart as Those Who Generate It

    The modern state system is built on a bargain between governments and citizens. States provide collective social goods, and in turn, via a system of norms, institutions, regulations, and ethics to hold this power accountable, citizens give states legitimacy. This bargain created order and stability out of what was an increasingly chaotic global system. If algorithms represent a new ungoverned space, a hidden and potentially ever-evolving unknowable public good, then they are an affront to our democratic system, one that requires transparency and accountability in order to function. A node of power that exists outside of these bounds is a threat to the notion of collective governance itself. This, at its core, is a profoundly undemocratic notion—one that states will have to engage with seriously if they are going to remain relevant and legitimate to their digital citizenry who give them their power.

    (tags: palantir algorithms big-data government democracy transparency accountability analytics surveillance war privacy protest rights)