Skip to content

Category: Uncategorized

Links for 2015-08-23

Published August 23, 2015

Links for 2015-08-22

Published August 22, 2015

Links for 2015-08-19

Published August 19, 2015

Links for 2015-08-18

Published August 18, 2015

Links for 2015-08-17

Published August 17, 2015

  • The world beyond batch: Streaming 101 - O'Reilly Media

    To summarize, in this post I’ve: Clarified terminology, specifically narrowing the definition of “streaming” to apply to execution engines only, while using more descriptive terms like unbounded data and approximate/speculative results for distinct concepts often categorized under the “streaming” umbrella. Assessed the relative capabilities of well-designed batch and streaming systems, positing that streaming is in fact a strict superset of batch, and that notions like the Lambda Architecture, which are predicated on streaming being inferior to batch, are destined for retirement as streaming systems mature. Proposed two high-level concepts necessary for streaming systems to both catch up to and ultimately surpass batch, those being correctness and tools for reasoning about time, respectively. Established the important differences between event time and processing time, characterized the difficulties those differences impose when analyzing data in the context of when they occurred, and proposed a shift in approach away from notions of completeness and toward simply adapting to changes in data over time. Looked at the major data processing approaches in common use today for bounded and unbounded data, via both batch and streaming engines, roughly categorizing the unbounded approaches into: time-agnostic, approximation, windowing by processing time, and windowing by event time.

    (tags: streaming batch big-data lambda-architecture dataflow event-processing cep millwheel data data-processing)

  • What the hell is going on with SoundCloud?

    tl;dr: major labels.

    Despite having revenue coming in from ads and subscriptions, SoundCloud still relies on outside investment. While the company received $150 million in a funding round at the end of last year, it pales next to the reported $526 million Spotify gained in June, and if one report is to be believed, SoundCloud is running very low on cash. Furthermore, sources suggest that potential investors are waiting to see what happens with Sony and Universal before ploughing in more money. With the high sums reported to be involved, it’s a stalemate that could potentially break the company whether it decides to pay or not.

    (tags: soundcloud music mp3 copyright sony universal spotify funding startups)

  • GSMem: Data Exfiltration from Air-Gapped Computers over GSM Frequencies

    Holy shit.

    Air-gapped networks are isolated, separated both logically and physically from public networks. Although the feasibility of invading such systems has been demonstrated in recent years, exfiltration of data from air-gapped networks is still a challenging task. In this paper we present GSMem, a malware that can exfiltrate data through an air-gap over cellular frequencies. Rogue software on an infected target computer modulates and transmits electromagnetic signals at cellular frequencies by invoking specific memory-related instructions and utilizing the multichannel memory architecture to amplify the transmission. Furthermore, we show that the transmitted signals can be received and demodulated by a rootkit placed in the baseband firmware of a nearby cellular phone.

    (tags: gsmem gsm exfiltration air-gaps memory radio mobile-phones security papers)

Links for 2015-08-16

Published August 16, 2015

Links for 2015-08-12

Published August 12, 2015

Links for 2015-08-11

Published August 11, 2015

  • Reddit comments from a nuclear-power expert

    Reddit user "Hiddencamper" is a senior nuclear reactor operator in the US, and regularly posts very knowledgeable comments about reactor operations, safety procedures, and other details. It's fascinating (via Maciej)

    (tags: via:maciej nuclear-power nuclear atomic power energy safety procedures operations history chernobyl scram)

  • Amazon EC2 2015 Benchmark: Testing Speeds Between AWS EC2 and S3 Regions

    Here we are again, a year later, and still no bloody percentiles! Just amateurish averaging. This is not how you measure anything, ffs. Still, better than nothing I suppose

    (tags: fail latency measurement aws ec2 percentiles s3)

  • background doc on the Jeep hack

    "Remote Exploitation of an Unaltered Passenger Vehicle", by Dr. Charlie Miller (cmiller@openrce.org) and Chris Valasek (cvalasek@gmail.com). QNX, unauthenticated D-Bus, etc.

    'Since a vehicle can scan for other vulnerable vehicles and the exploit doesn’t require any user interaction, it would be possible to write a worm. This worm would scan for vulnerable vehicles, exploit them with their payload which would scan for other vulnerable vehicles, etc. This is really interesting and scary. Please don’t do this. Please.'

    (tags: jeep hacks exploits d-bus qnx cars safety risks)

  • Care.data and access to UK health records: patient privacy and public trust

    'In 2013, the United Kingdom launched care.data, an NHS England initiative to combine patient records, stored in the machines of general practitioners (GPs), with information from social services and hospitals to make one centralized data archive. One aim of the initiative is to gain a picture of the care being delivered between different parts of the healthcare system and thus identify what is working in health care delivery, and what areas need greater attention and resources. This case study analyzes the complications around the launch of care.data. It explains the historical context of the program and the controversies that emerged in the course of the rollout. It explores problems in management and communications around the centralization effort, competing views on the safety of “anonymous” and “pseudonymous” health data, and the conflicting legal duties imposed on GPs with the introduction of the 2012 Health and Social Care Act. This paper also explores the power struggles in the battle over care.data and outlines the tensions among various stakeholders, including patients, GPs, the Health and Social Care Information Centre (HSCIC), the government, privacy experts and data purchasers. The predominant public policy question that emerges from this review centers on how best to utilize technological advances and simultaneously strike a balance between the many competing interests around health and personal privacy.'

    (tags: care.data privacy healthcare uk nhs trust anonymity anonymization gps medicine)

Links for 2015-08-10

Published August 10, 2015

Links for 2015-08-07

Published August 7, 2015

Links for 2015-08-06

Published August 6, 2015

Links for 2015-08-05

Published August 5, 2015

Links for 2015-08-03

Published August 3, 2015

Links for 2015-07-30

Published July 30, 2015

  • danilop/yas3fs · GitHub

    YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.

    (tags: aws s3 s3fs yas3fs filesystems fuse sns)

  • danilop/runjop · GitHub

    RunJOP (Run Just Once Please) is a distributed execution framework to run a command (i.e. a job) only once in a group of servers [built using AWS DynamoDB and S3].
    nifty! Distributed cron is pretty easy when you've got Dynamo doing the heavy lifting.

    (tags: dynamodb cron distributed-cron scheduling runjop danilop hacks aws ops)

Links for 2015-07-29

Published July 29, 2015

Links for 2015-07-28

Published July 28, 2015

  • Taming Complexity with Reversibility

    This is a great post from Kent Beck, putting a lot of recent deployment/rollout patterns in a clear context -- that of supporting "reversibility":

    Development servers. Each engineer has their own copy of the entire site. Engineers can make a change, see the consequences, and reverse the change in seconds without affecting anyone else. Code review. Engineers can propose a change, get feedback, and improve or abandon it in minutes or hours, all before affecting any people using Facebook. Internal usage. Engineers can make a change, get feedback from thousands of employees using the change, and roll it back in an hour. Staged rollout. We can begin deploying a change to a billion people and, if the metrics tank, take it back before problems affect most people using Facebook. Dynamic configuration. If an engineer has planned for it in the code, we can turn off an offending feature in production in seconds. Alternatively, we can dial features up and down in tiny increments (i.e. only 0.1% of people see the feature) to discover and avoid non-linear effects. Correlation. Our correlation tools let us easily see the unexpected consequences of features so we know to turn them off even when those consequences aren't obvious. IRC. We can roll out features potentially affecting our ability to communicate internally via Facebook because we have uncorrelated communication channels like IRC and phones. Right hand side units. We can add a little bit of functionality to the website and turn it on and off in seconds, all without interfering with people's primary interaction with NewsFeed. Shadow production. We can experiment with new services under real load, from a tiny trickle to the whole flood, without affecting production. Frequent pushes. Reversing some changes require a code change. On the website we never more than eight hours from the next schedule code push (minutes if a fix is urgent and you are willing to compensate Release Engineering). The time frame for code reversibility on the mobile applications is longer, but the downward trend is clear from six weeks to four to (currently) two. Data-informed decisions. (Thanks to Dave Cleal) Data-informed decisions are inherently reversible (with the exceptions noted below). "We expect this feature to affect this metric. If it doesn't, it's gone." Advance countries. We can roll a feature out to a whole country, generate accurate feedback, and roll it back without affecting most of the people using Facebook. Soft launches. When we roll out a feature or application with a minimum of fanfare it can be pulled back with a minimum of public attention. Double write/bulk migrate/double read. Even as fundamental a decision as storage format is reversible if we follow this format: start writing all new data to the new data store, migrate all the old data, then start reading from the new data store in parallel with the old.
    We do a bunch of these in work, and the rest are on the to-do list. +1 to these!

    (tags: software deployment complexity systems facebook reversibility dark-releases releases ops cd migration)

Links for 2015-07-27

Published July 27, 2015

  • Benchmarking GitHub Enterprise - GitHub Engineering

    Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)

    (tags: github listen-backlog tcp debugging timeouts load-testing benchmarking testing ops linux)

  • How .uk came to be (and why it's not .gb)

    WB: By the late 80s the IANA [the Internet Assigned Numbers Authority, set up in 1988 to manage global IP address allocations] was trying to get all those countries that were trying to join the internet to use the ISO 3166 standard for country codes. It was used for all sorts of things?—?you see it on cars, “GB” for the UK. [...] At that point, we’re faced with a problem that Jon Postel would like to have changed it to .gb to be consistent with the rest of the world. Whereas .uk had already been established, with a few tens of thousands of domain names with .uk on them. I remember chairing one of the JANET net workshops that were held every year, and the Northern Irish were adamant that they were part of the UK?—?so the consensus was, we’d try and keep .uk, we’d park .gb and not use it. PK: I didn’t particularly want to change to .gb because I was responsible for Northern Ireland as well. And what’s more, there was a certain question as to whether a research group in the US should be allowed to tell the British what to do. So this argy-bargy continued for a little while and, in the meantime, one of my clients was the Ministry of Defence, and they decided they couldn’t wait this long, and they decided I was going to lose the battle, and so bits of MOD went over to .gb?—?I didn’t care, as I was running .gb and .uk in any case.

    (tags: dot-uk history internet dot-gb britain uk northern-ireland ireland janet)

  • That time the Internet sent a SWAT team to my mom's house - Boing Boing

    The solution is for social media sites and the police to take threats or jokes about swatting, doxxing, and organized crime seriously. Tweeting about buying a gun and shooting up a school would be taken seriously, and so should the threat of raping, doxxing, swatting or killing someone. Privacy issues and online harassment are directly linked, and online harassment isn’t going anywhere. My fear is that, in reaction to online harassment, laws will be passed that will break down our civil freedoms and rights online, and that more surveillance will be sold to users under the guise of safety. More surveillance, however, would not have helped me or my mother. A platform that takes harassment and threats seriously instead of treating them like jokes would have.

    (tags: twitter gamergate 4chan 8chan privacy doxxing swatting harrassment threats social-media facebook law feminism)

  • Why Google's Deep Dream Is Future Kitsch

    Deep Dream estranges us from our fears, perhaps, but it doesn't make them go away. It's easy to discuss Deep Dream as an independent creature, a foreign intelligence that we interact with for fun. Yet like all kitsch, it comes straight back to its creators.

    (tags: kitsch deep-dream art graphics google inceptionism)

  • It’s Not Climate Change?—?It’s Everything Change

    now this is a Long Read. the inimitable Margaret Atwood on climate change, beautifully illustrated

    (tags: climate climate-change margaret-atwood long-reads change life earth green future)

  • In Praise of the AK-47 — Dear Design Student — Medium

    While someone can certainly make the case that an AK-47, or any other kind of gun or rifle is designed, nothing whose primary purpose is to take away life can be said to be designed well. And that attempting to separate an object from its function in order to appreciate it for purely aesthetic reasons, or to be impressed by its minimal elegance, is a coward’s way of justifying the death they’ve designed into the word, and the money with which they’re lining their pockets.

    (tags: design ux ak-47 kalashnikov guns function work)

Links for 2015-07-22

Published July 22, 2015

Links for 2015-07-21

Published July 21, 2015

  • Java lambdas and performance

    Lambdas in Java 8 introduce some unpredictable performance implications, due to reliance on escape analysis to eliminate object allocation on every lambda invocation. Peter Lawrey has some details

    (tags: lambdas java-8 java performance low-latency optimization peter-lawrey coding escape-analysis)

  • Mikhail Panchenko's thoughts on the July 2015 CircleCI outage

    an excellent followup operational post on CircleCI's "database is not a queue" outage

    (tags: database-is-not-a-queue mysql sql databases ops outages postmortems)

  • Men who harass women online are quite literally losers, new study finds

    (1) players are anonymous, and the possibility of “policing individual behavior is almost impossible”; (2) they only encounter each other a few times in passing — it’s very possible to hurl an expletive at another player, and never “see” him or her again; and (3) finally, and perhaps predictably, the sex-ratio of players is biased pretty heavily toward men. (A 2014 survey of gender ratios on Reddit found that r/halo was over 95 percent male.) [....] In each of these environments, Kasumovic suggests, a recent influx of female participants has disrupted a pre-existing social hierarchy. That’s okay for the guys at the top — but for the guys at the bottom, who stand to lose more status, that’s very threatening. (It’s also in keeping with the evolutionary framework on anti-lady hostility, which suggests sexism is a kind of Neanderthal defense mechanism for low-status, non-dominant men trying to maintain a shaky grip on their particular cave’s supply of women.) “As men often rely on aggression to maintain their dominant social status,” Kasumovic writes, “the increase in hostility towards a woman by lower-status males may be an attempt to disregard a female’s performance and suppress her disturbance on the hierarchy to retain their social rank.”

    (tags: losers sexism mysogyny women halo gaming gamergate 4chan abuse harrassment papers bullying social-status)

  • The old suburban office park is the new American ghost town - The Washington Post

    Most analyses of the market indicate that office parks simply aren’t as appealing or profitable as they were in the 20th century and that Americans just aren’t as keen to cloister themselves in workspaces that are reachable only by car.

    (tags: cbd cities work life office-parks commuting america history workplaces)

  • HACKERS REMOTELY KILL A JEEP ON THE HIGHWAY—WITH ME IN IT

    Jaysus, this is terrifying.

    Miller and Valasek’s full arsenal includes functions that at lower speeds fully kill the engine, abruptly engage the brakes, or disable them altogether. The most disturbing maneuver came when they cut the Jeep’s brakes, leaving me frantically pumping the pedal as the 2-ton SUV slid uncontrollably into a ditch.
    Avoid any car which supports this staggeringly-badly-conceived Uconnect feature:
    All of this is possible only because Chrysler, like practically all carmakers, is doing its best to turn the modern automobile into a smartphone. Uconnect, an Internet-connected computer feature in hundreds of thousands of Fiat Chrysler cars, SUVs, and trucks, controls the vehicle’s entertainment and navigation, enables phone calls, and even offers a Wi-Fi hot spot.
    :facepalm: Also, Chrysler's response sucks: "Chrysler’s patch must be manually implemented via a USB stick or by a dealership mechanic."

    (tags: hacking security cars driving safety brakes jeeps chrysler fiat uconnect can-bus can)

Links for 2015-07-20

Published July 20, 2015

Links for 2015-07-17

Published July 17, 2015

  • Angela Merkel told a sobbing girl she couldn't save her from deportation. It was a lie. - Vox

    Argentina has, as a matter of constitutional law, effectively open borders. There are no caps or quotas or lottery systems. You can move there legally if you have an employer or family member to sponsor you. That's all you need. If you don't have a sponsor, and make your way in illegally, you're recognized as an "irregular migrant." Discrimination against irregular migrants in health care or education is illegal, and deportation in noncriminal cases is exceptionally rare. Large-scale amnesties are the norm. Obviously Argentina is not nearly as rich as Germany or the US or the UK. But it's considerably richer than three of its neighbors (Bolivia, Paraguay, and Brazil). And yet it doesn't try hard to keep their residents out. It welcomes them — as it should. "One could have expected catastrophe—an uncontrollable flow of poorer immigrants streaming into the country coupled with angry public backlash," Elizabeth Slater writes in the World Policy Journal. "That hasn't happened." Angela Merkel clearly expects catastrophe if she lets people like this weeping young Palestinian girl stay in Germany. That catastrophe is simply a myth; it wouldn't happen. What would happen is that Germany's economy would grow, its culture would grow richer, and that girl and more like her could see their lives improve immeasurably.

    (tags: argentina immigration angela-merkel germany eu migrants deportation economics)

Links for 2015-07-16

Published July 16, 2015

Links for 2015-07-15

Published July 15, 2015

Links for 2015-07-14

Published July 14, 2015

Links for 2015-07-13

Published July 13, 2015

  • OkHttp

    A new HTTP client library for Android and Java, with a lot of nice features:

    HTTP/2 and SPDY support allows all requests to the same host to share a socket. Connection pooling reduces request latency (if SPDY isn’t available). Transparent GZIP shrinks download sizes. Response caching avoids the network completely for repeat requests. OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails. Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.

    (tags: android http java libraries okhttp http2 spdy microservices jdk)

  • Eircode tech specs

    via Ossian.

    (tags: via:smytho tech-specs specs eircode addresses geocoding ireland mapping)

  • AWS Best Practices for DDoS Resiliency [pdf]

    Reasonably solid white paper

    (tags: ddos amazon aws security dos whitepapers pdf)

Links for 2015-07-11

Published July 11, 2015

Links for 2015-06-25

Published June 25, 2015

Links for 2015-06-23

Published June 23, 2015

Links for 2015-06-22

Published June 22, 2015

Links for 2015-06-21

Published June 21, 2015

  • jwz on Inceptionism

    "Shoggoth ovipositors":

    So then they reach inside to one of the layers and spin the knob randomly to fuck it up. Lower layers are edges and curves. Higher layers are faces, eyes and shoggoth ovipositors. [....] But the best part is not when they just glitch an image -- which is a fun kind of embossing at one end, and the "extra eyes" filter at the other -- but is when they take a net trained on some particular set of objects and feed it static, then zoom in, and feed the output back in repeatedly. That's when you converge upon the platonic ideal of those objects, which -- it turns out -- tend to be Giger nightmare landscapes. Who knew. (I knew.)
    This stuff is still boggling my mind. All those doggy faces! That is one dog-obsessed ANN.

    (tags: neural-networks ai jwz funny shoggoths image-recognition hr-giger art inceptionism)

Links for 2015-06-19

Published June 19, 2015

Links for 2015-06-18

Published June 18, 2015

  • Inceptionism: Going Deeper into Neural Networks

    This is amazing, and a little scary.

    If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge. Again, we just start with an existing image and give it to our neural net. We ask the network: “Whatever you see there, I want more of it!” This creates a feedback loop: if a cloud looks a little bit like a bird, the network will make it look more like a bird. This in turn will make the network recognize the bird even more strongly on the next pass and so forth, until a highly detailed bird appears, seemingly out of nowhere.
    An enlightening comment from the G+ thread:
    This is the most fun we've had in the office in a while. We've even made some of those 'Inceptionistic' art pieces into giant posters. Beyond the eye candy, there is actually something deeply interesting in this line of work: neural networks have a bad reputation for being strange black boxes that that are opaque to inspection. I have never understood those charges: any other model (GMM, SVM, Random Forests) of any sufficient complexity for a real task is completely opaque for very fundamental reasons: their non-linear structure makes it hard to project back the function they represent into their input space and make sense of it. Not so with backprop, as this blog post shows eloquently: you can query the model and ask what it believes it is seeing or 'wants' to see simply by following gradients. This 'guided hallucination' technique is very powerful and the gorgeous visualizations it generates are very evocative of what's really going on in the network.?

    (tags: art machine-learning algorithm inceptionism research google neural-networks learning dreams feedback graphics)

Links for 2015-06-17

Published June 17, 2015

Links for 2015-06-15

Published June 15, 2015

  • How We Moved Our API From Ruby to Go and Saved Our Sanity

    Parse on their ditching-Rails story. I haven't heard a nice thing about Ruby or Rails as an operational, production-quality platform in a long time :(

    (tags: go ruby rails ops parse languages platforms)

  • VPC Flow Logs

    we are introducing Flow Logs for the Amazon Virtual Private Cloud.  Once enabled for a particular VPC, VPC subnet, or Elastic Network Interface (ENI), relevant network traffic will be logged to CloudWatch Logs for storage and analysis by your own applications or third-party tools. You can create alarms that will fire if certain types of traffic are detected; you can also create metrics to help you to identify trends and patterns. The information captured includes information about allowed and denied traffic (based on security group and network ACL rules). It also includes source and destination IP addresses, ports, the IANA protocol number, packet and byte counts, a time interval during which the flow was observed, and an action (ACCEPT or REJECT).

    (tags: ec2 aws vpc logging tracing ops flow-logs network tcpdump packets packet-capture)

  • Tim Hunt "jokes" about women scientists. Or not. (with image, tweets) · deborahblum · Storify

    '[Tim Hunt] said that while he meant to be ironic, he did think it was hard to collaborate with women because they are too emotional - that he was trying to be honest about the problems.' So much for the "nasty twitter took my jokes seriously" claims then.

    (tags: twitter science misogyny women tim-hunt deborah-blum journalism)

  • Why I dislike systemd

    Good post, and hard to disagree.

    One of the "features" of systemd is that it allows you to boot a system without needing a shell at all. This seems like such a senseless manoeuvre that I can't help but think of it as a knee-jerk reaction to the perception of Too Much Shell in sysv init scripts. In exactly which universe is it reasonable to assume that you have a running D-Bus service (or kdbus) and a filesystem containing unit files, all the binaries they refer to, all the libraries they link against, and all the configuration files any of them reference, but that you lack that most ubiquitous of UNIX binaries, /bin/sh?

    (tags: history linux unix systemd bsd system-v init ops dbus)

  • Adrian Colyer reviews the Twitter Heron paper

    ouch, really sounds like Storm didn't cut the muster. 'It’s hard to imagine something more damaging to Apache Storm than this. Having read it through, I’m left with the impression that the paper might as well have been titled “Why Storm Sucks”, which coming from Twitter themselves is quite a statement.' If I was to summarise the lessons learned, it sounds like: backpressure is required; and multi-tenant architectures suck.

    (tags: storm twitter heron big-data streaming realtime backpressure)

Links for 2015-06-14

Published June 14, 2015

  • Security theatre at Allied Irish Banks

    Allied Irish Banks's web and mobile banking portals are ludicrously insecure. Vast numbers of accounts have easily-guessable registration numbers and are thus 'protected' by a level of security that is twice as easy to crack as would be provided by a single password containing only two lowercase letters. A person of malicious intent could easily gain access to hundreds, possibly thousands, of accounts as well as completely overwhelm the branch network by locking an estimated several 100,000s of people out of their online banking. Both AIB and the Irish Financial Services Ombudsman have refused to respond meaningfully to multiple communications each in which these concerns were raised privately.

    (tags: aib banking security ireland hacking ifso online-banking)

  • Leveraging AWS to Build a Scalable Data Pipeline

    Nice detailed description of an auto-scaled SQS worker pool

    (tags: sqs aws ec2 auto-scaling asg worker-pools architecture scalability)

Links for 2015-06-13

Published June 13, 2015

  • China’s Spies Hit the Blackmail Jackpot With Data on 4 Million Federal Workers

    The Daily Beast is scathing re the OPM hack:

    Here’s where things start to get scary. Whoever has OPM’s records knows an astonishing amount about millions of federal workers, members of the military, and security clearance holders. They can now target those Americans for recruitment or influence. After all, they know their vices, every last one—the gambling habit, the inability to pay bills on time, the spats with former spouses, the taste for something sexual on the side—since all that is recorded in security clearance paperwork. (To get an idea of how detailed this gets, you can see the form, called an SF86, here.) Speaking as a former counterintelligence officer, it really doesn’t get much worse than this.

    (tags: daily-beast sf86 clearance us-government america china cyberwar hacking opm privacy)

Links for 2015-06-11

Published June 11, 2015

  • Facebook Infer

    New static analysis goodnews, freshly open-sourced by Facebook:

    Facebook Infer uses logic to do reasoning about a program's execution, but reasoning at this scale — for large applications built from millions of lines of source code — is hard. Theoretically, the number of possibilities that need to be checked is more than the number of estimated atoms in the observable universe. Furthermore, at Facebook our code is not a fixed artifact but an evolving system, updated frequently and concurrently by many developers. It is not unusual to see more than a thousand modifications to our mobile code submitted for review in a given day. The requirements on the program analyzer then become even more challenging because we expect a tool to report quickly on these code modifications — in the region of 10 minutes — to fit in with developers' workflow. Coping with this scale and velocity requires advanced mathematical techniques. Facebook Infer uses two such techniques: separation logic and bi-abduction. Separation logic is a theory that allows Facebook Infer's analysis to reason about small, independent parts of the application storage, rather than having to consider the entirety of the memory potentially at every step. That would be a daunting task on modern processors with their large addressable virtual memories. Bi-abduction is a logical inference technique that allows Facebook Infer to discover properties about the behavior of independent parts of the application code. By storing these properties between runs, Facebook Infer needs to analyze only the parts of the software that have changed, reusing the results of its previous analysis where it can. By combining these approaches, our analyzer is able to find complex problems in modifications to an application built from millions of lines of code, in minutes.
    (via Bryan O'Sullivan)

    (tags: via:bos infer facebook static-analysis lint code java ios android coding bugs)

  • The Tamborzão Goes to Thailand

    This is great. the story of how cheesy funk carioca tune “A Minha Amiga Fran” managed to become "Kawo Kawo" and become a massive hit in Thailand

    (tags: thai brazil carioca music dance-music kawo-kawo)

Links for 2015-06-10

Published June 10, 2015

Links for 2015-06-08

Published June 8, 2015

Links for 2015-06-04

Published June 4, 2015

Links for 2015-06-03

Published June 3, 2015

  • Performance Testing at LMAX

    Good series of blog posts on the LMAX trading platform's performance testing strategy -- they capture live traffic off the wire, then build statistical models simulating its features. See also http://epickrram.blogspot.co.uk/2014/07/performance-testing-at-lmax-part-two.html and http://epickrram.blogspot.co.uk/2014/08/performance-testing-at-lmax-part-three.html .

    (tags: performance testing tests simulation latency lmax trading sniffing packet-capture)

  • The Violence of Algorithms: Why Big Data Is Only as Smart as Those Who Generate It

    The modern state system is built on a bargain between governments and citizens. States provide collective social goods, and in turn, via a system of norms, institutions, regulations, and ethics to hold this power accountable, citizens give states legitimacy. This bargain created order and stability out of what was an increasingly chaotic global system. If algorithms represent a new ungoverned space, a hidden and potentially ever-evolving unknowable public good, then they are an affront to our democratic system, one that requires transparency and accountability in order to function. A node of power that exists outside of these bounds is a threat to the notion of collective governance itself. This, at its core, is a profoundly undemocratic notion—one that states will have to engage with seriously if they are going to remain relevant and legitimate to their digital citizenry who give them their power.

    (tags: palantir algorithms big-data government democracy transparency accountability analytics surveillance war privacy protest rights)

Links for 2015-06-02

Published June 2, 2015

Links for 2015-05-29

Published May 29, 2015

Links for 2015-05-28

Published May 28, 2015

  • I Fooled Millions Into Thinking Chocolate Helps Weight Loss

    “Slim by Chocolate!” the headlines blared. A team of German researchers had found that people on a low-carb diet lost weight 10 percent faster if they ate a chocolate bar every day. It made the front page of Bild, Europe’s largest daily newspaper, just beneath their update about the Germanwings crash. From there, it ricocheted around the internet and beyond, making news in more than 20 countries and half a dozen languages. It was discussed on television news shows. It appeared in glossy print, most recently in the June issue of Shape magazine (“Why You Must Eat Chocolate Daily”, page 128). Not only does chocolate accelerate weight loss, the study found, but it leads to healthier cholesterol levels and overall increased well-being. The Bild story quotes the study’s lead author, Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health: “The best part is you can buy chocolate everywhere.” I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I’m a journalist. I do have a Ph.D., but it’s in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That’s nothing more than a website. Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.
    Interesting bit: the online commenters commenting on the published stories quickly saw through the bullshit. Why can't the churnalising journos do that?

    (tags: chocolate journalism science diet food churnalism pr bild health clinical-trials papers peer-review research)

  • Snake-Oil Superfoods

    mainly interesting for the dataviz and the Google-Doc-driven backend. wish they published the script though

    (tags: google snake-oil superfoods food dataviz bubble-race-chart graphics infographics google-docs spreadsheets)

Links for 2015-05-27

Published May 27, 2015

  • Three Questions to Answer When Reporting an Error

    Very long, but tl;dr:

    the trick to creating an effective error message is to answer the 3 Questions within your message: What is the error? What was the probable cause of the error? What is the probable remedy?

    (tags: errors ui ux reporting logging coding)

  • Volvo says horrible 'self-parking car accident' happened because driver didn't have 'pedestrian detection'

    Grim meathook future, courtesy of Volvo:

    “The Volvo XC60 comes with City Safety as a standard feature however this does not include the Pedestrian detection functionality [...] The pedestrian detection feature [...] costs approximately $3,000.
    However, there's another lesson here, in crappy car UX and the risks thereof:
    But even if it did have the feature, Larsson says the driver would have interfered with it by the way they were driving and “accelerating heavily towards the people in the video.” “The pedestrian detection would likely have been inactivated due to the driver inactivating it by intentionally and actively accelerating,” said Larsson. “Hence, the auto braking function is overrided by the driver and deactivated.” Meanwhile, the people in the video seem to ignore their instincts and trust that the car assumed to be endowed with artificial intelligence knows not to hurt them. It is a sign of our incredible faith in the power of technology, but also, it’s a reminder that companies making AI-assisted vehicles need to make safety features standard and communicate clearly when they aren’t.

    (tags: self-driving-cars cars ai pedestrian computer-vision volvo fail accidents grim-meathook-future)

  • iPhone UTF-8 text vulnerability

    'Due to how the banner notifications process the Unicode text. The banner briefly attempts to present the incoming text and then "gives up" thus the crash'. Apparently the entire Springboard launcher crashes.

    (tags: apple vulnerability iphone utf-8 unicode fail bugs springboard ios via:abetson)

Links for 2015-05-26

Published May 26, 2015

  • Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC)

    The Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a “chime” message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses. You can use the chime attributes to run your code every fifteen minutes, or only run your code once an hour (e.g., when minute == "00") or once a day (e.g., when hour == "00" and minute == "00") or any other series of intervals. You can even subscribe a function you only want to run only once at a specific time in the future: Have the function ignore all invocations until it’s after the time it wants. When it is time, it can perform its job, then unsubscribe itself from the SNS Topic.

    (tags: alestic aws lambda cron time clock periodic-tasks recurrence hacks)

Links for 2015-05-25

Published May 25, 2015

Links for 2015-05-18

Published May 18, 2015

Links for 2015-05-17

Published May 17, 2015

  • 'Can People Distinguish Pâté from Dog Food?'

    Ugh.

    Considering the similarity of its ingredients, canned dog food could be a suitable and inexpensive substitute for pâté or processed blended meat products such as Spam or liverwurst. However, the social stigma associated with the human consumption of pet food makes an unbiased comparison challenging. To prevent bias, Newman's Own dog food was prepared with a food processor to have the texture and appearance of a liver mousse. In a double-blind test, subjects were presented with five unlabeled blended meat products, one of which was the prepared dog food. After ranking the samples on the basis of taste, subjects were challenged to identify which of the five was dog food. Although 72% of subjects ranked the dog food as the worst of the five samples in terms of taste (Newell and MacFarlane multiple comparison, P<0.05), subjects were not better than random at correctly identifying the dog food.

    (tags: pate food omgwtf science research dog-food meat economics taste flavour)

  • Redditor runs the secret Python code in Ex Machina

    and finds:

    when you run with python2.7 you get the following: ISBN = 9780199226559 Which is Embodiment and the inner life: Cognition and Consciousness in the Space of Possible Minds. and so now I have a lot more respect for the Director.

    (tags: python movies ex-machina cool books easter-eggs)

  • Metalwoman beer recipe

    via the Dublin Ladies Beer Society ;)

    (tags: metalman metalwoman recipes beer brewing hops dlbs)

Links for 2015-05-14

Published May 14, 2015

Links for 2015-05-12

Published May 12, 2015

  • Input: Fonts for Code

    Non-monospaced coding fonts! I'm all in favour...

    As writing and managing code becomes more complex, today’s sophisticated coding environments are evolving to include everything from breakpoint markers to code folding and syntax highlighting. The typography of code should evolve as well, to explore possibilities beyond one font style, one size, and one character width.

    (tags: input fonts via:its typography code coding font text ide monospace)

  • Apache HTrace

    a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator

    (tags: zipkin tracing trace apache incubator java debugging)

  • Intel speeds up etcd throughput using ADR Xeon-only hardware feature

    To reduce the latency impact of storing to disk, Weaver’s team looked to buffering as a means to absorb the writes and sync them to disk periodically, rather than for each entry. Tradeoffs? They knew memory buffers would help, but there would be potential difficulties with smaller clusters if they violated the stable storage requirement. Instead, they turned to Intel’s silicon architects about features available in the Xeon line. After describing the core problem, they found out this had been solved in other areas with ADR. After some work to prove out a Linux OS supported use for this, they were confident they had a best-of-both-worlds angle. And it worked. As Weaver detailed in his CoreOS Fest discussion, the response time proved stable. ADR can grab a section of memory, persist it to disk and power it back. It can return entries back to disk and restore back to the buffer. ADR provides the ability to make small (<100MB) segments of memory “stable” enough for Raft log entries. It means it does not need battery-backed memory. It can be orchestrated using Linux or Windows OS libraries. ADR allows the capability to define target memory and determine where to recover. It can also be exposed directly into libs for runtimes like Golang. And it uses silicon features that are accessible on current Intel servers.

    (tags: kubernetes coreos adr performance intel raft etcd hardware linux persistence disk storage xeon)

Links for 2015-05-11

Published May 11, 2015

  • streamtools: a graphical tool for working with streams of data | nytlabs

    Visual programming, Yahoo! Pipes style, back again:

    we have created streamtools – a new, open source project by The New York Times R&D Lab which provides a general purpose, graphical tool for dealing with streams of data. It provides a vocabulary of operations that can be connected together to create live data processing systems without the need for programming or complicated infrastructure. These systems are assembled using a visual interface that affords both immediate understanding and live manipulation of the system.
    via Aman

    (tags: via:akohli streaming data nytimes visual-programming coding)

  • MappedBus

    a Java based low latency, high throughput message bus, built on top of a memory mapped file; inspired by Java Chronicle with the main difference that it's designed to efficiently support multiple writers – enabling use cases where the order of messages produced by multiple processes are important. MappedBus can be also described as an efficient IPC mechanism which enable several Java programs to communicate by exchanging messages.

    (tags: ipc java jvm mappedbus low-latency mmap message-bus data-structures queue message-passing)

Links for 2015-05-10

Published May 10, 2015

  • Amazon's Drone Delivery Patent Just Feels Like Trolling At This Point

    Oh dear, Amazon.

    These aren’t actual technologies yet. [...] All of which underscores that Amazon might never ever ever ever actually implement delivery drones. The patent paperwork was filed nearly a year after Amazon’s splashy drone program reveal on 60 Minutes. At the time we called it revolutionary marketing because, you know, delivery drones are technical and logistical madness, not to mention that commercial drone use is illegal right now. Although, in fairness the FAA did just relax some rules so that Amazon could test drones. At this point it feels like Amazon is just trolling. It’s trolling us with public relations BS about its future drones, and it’s trolling future competitors -- Google is also apparently working on this -- so that if somebody ever somehow does anything relating to drone delivery, Amazon can sue them. If I’m wrong, I’ll deliver my apology via Airmail.

    (tags: amazon trolling patents uspto delivery drones uavs competition faa)

  • Red Hat on rkt vs Docker

    This is like watching a train-wreck in slow motion on Groundhog Day. We, in the broader Linux and open source community, have been down this path multiple times over the past fifteen years, specifically with package formats. While there needs to be room for experimentation, having two incompatible specs driven by two startups trying to differentiate and in direct competition is *not* a good thing. It would be better for the community and for everyone who depends on our collective efforts if CoreOS and Docker collaborated on a standardized common spec, image format, and distribution protocol. To this end, we at Red Hat will continue to contribute to both initiatives with the goal of driving convergence.

    (tags: rkt docker appc coreos red-hat dpkg rpm linux packaging collaboration open-source)

Links for 2015-05-09

Published May 9, 2015

Links for 2015-05-08

Published May 8, 2015

Links for 2015-05-07

Published May 7, 2015

  • KillBiller

    Excellent mobile-phone plan comparison site for the Irish market, using apps which you install and which analyse your call history, data usage, etc. over the past month to compute the optimal plan based on your usage. Pretty amazing results in my case! The only downside is the privacy policy, which allows the company to resell your usage data (anonymised, and in aggregate) -- I'd really prefer if this wasn't the case :(

    (tags: mobile-phones shopping tesco emobile 3g 4g ireland plans comparison-shopping killbiller via:its)

  • Family in No poster Says YES to Marriage Equality | Amnesty International

    Beyond the politics, the risks of stock photo usage are pretty evident too:

    "In 2014, as a young family, we did a photo shoot with a photographer friend to get some nice shots for the family album. No money was exchanged – we got nice photos for free, they got nice images for their portfolio. As part of this agreement, we agreed to let them upload them to a stock photo album. We knew that these were available for purchase and we gave permission. Perhaps, naïvely, we imagined that on the off chance that any was ever selected, it might be for a small magazine or website. To confirm, we have not received any money for the photo – then or now, and nor do we expect any. We were surprised and upset to see that the photo was being used as part of a campaign with which we do not agree. We completely support same-sex marriage, and we believe that same-sex couples’ should of course be able to adopt, as we believe that they are equally able to provide children with much-needed love and care. To suggest otherwise is offensive to us, and to many others."

    (tags: ssm ireland politics amnesty stock-photos ip rights photos campaigns ads)

  • Lambda: Bees with Frickin' Laser Beams

    a HTTP testing tool in AWS Lambda. nice enough, but still a toy...

    (tags: lambda aws node javascript hacks http load-testing)

Links for 2015-05-06

Published May 6, 2015

Links for 2015-05-05

Published May 5, 2015

  • Smarter testing Java code with Spock Framework

    hmm, looks quite nice as a potential next-gen JUnit replacement for unit tests

    (tags: java testing bdd tests junit unit-tests spock via:trishagee)

  • Tots To Travel

    'Baby Friendly Holidays | Child, Toddler & Family Villas | France | Spain | Portugal | Italy'. Joe swears by it, will give it a go next year

    (tags: holidays vacation travel europe kids children via:joe)

  • How the NSA Converts Spoken Words Into Searchable Text - The Intercept

    This hits the nail on the head, IMO:

    To Phillip Rogaway, a professor of computer science at the University of California, Davis, keyword-search is probably the “least of our problems.” In an email to The Intercept, Rogaway warned that “When the NSA identifies someone as ‘interesting’ based on contemporary NLP methods, it might be that there is no human-understandable explanation as to why beyond: ‘his corpus of discourse resembles those of others whom we thought interesting'; or the conceptual opposite: ‘his discourse looks or sounds different from most people’s.' If the algorithms NSA computers use to identify threats are too complex for humans to understand, it will be impossible to understand the contours of the surveillance apparatus by which one is judged.  All that people will be able to do is to try your best to behave just like everyone else.”

    (tags: privacy security gchq nsa surveillance machine-learning liberty future speech nlp pattern-analysis cs)

  • awslabs/aws-lambda-redshift-loader

    Load data into Redshift from S3 buckets using a pre-canned Lambda function. Looks like it may be a good example of production-quality Lambda

    (tags: lambda aws ec2 redshift s3 loaders etl pipeline)

  • Call me maybe: Aerospike

    'Aerospike offers phenomenal latencies and throughput -- but in terms of data safety, its strongest guarantees are similar to Cassandra or Riak in Last-Write-Wins mode. It may be a safe store for immutable data, but updates to a record can be silently discarded in the event of network disruption. Because Aerospike’s timeouts are so aggressive–on the order of milliseconds -- even small network hiccups are sufficient to trigger data loss. If you are an Aerospike user, you should not expect “immediate”, “read-committed”, or “ACID consistency”; their marketing material quietly assumes you have a magical network, and I assure you this is not the case. It’s certainly not true in cloud environments, and even well-managed physical datacenters can experience horrible network failures.'

    (tags: aerospike outages cap testing jepsen aphyr databases storage reliability)

Links for 2015-05-04

Published May 4, 2015

Links for 2015-05-02

Published May 2, 2015

  • In the privacy of your own home

    I didn't know about this:

    Last spring, as 41,000 runners made their way through the streets of Dublin in the city’s Women’s Mini Marathon, an unassuming redheaded man by the name of Candid Wueest stood on the sidelines with a scanner. He had built it in a couple of hours with $75 worth of parts, and he was using it to surreptitiously pick up data from activity trackers worn on the runners’ wrists. During the race, Wueest managed to collect personal info from 563 racers, including their names, addresses, and passwords, as well as the unique IDs of the devices they were carrying.

    (tags: dublin candid-wueest privacy data marathon running iot activity-trackers)

Links for 2015-04-30

Published April 30, 2015

Links for 2015-04-29

Published April 29, 2015

  • s3.amazonaws.com "certificate verification failed" errors due to crappy Verisign certs and overzealous curl policies

    Seth Vargo is correct. Its not the bit length of the key which is at issue, its the signature algorithm. The entire keychain for the s3.awsamazon.com key is signed with SHA1withRSA: https://www.ssllabs.com/ssltest/analyze.html?d=s3.amazonaws.com&s=54.231.244.0&hideResults=on At issue is that the root verisign key has been marked as weak because of SHA1 and taken out of the curl bundle which is widely popular, and this issue will continue to cause more and more issues going forwards as that bundle makes it way into shipping o/s distributions and aws certification verification breaks.
    'This is still happening and curl is now failing on my machine causing all sorts of fun issues (including breaking CocoaPods that are using S3 for storage).' -- @jmhodges This may be a contributory factor to the issue @nelson saw: https://nelsonslog.wordpress.com/2015/04/28/cyberduck-is-responsible-for-my-bad-ssl-certificate/ Curl's ca-certs bundle is also used by Node: https://github.com/joyent/node/issues/8894 and doubtless many other apps and packages. Here's a mailing list thread discussing the issue: http://curl.haxx.se/mail/archive-2014-10/0066.html -- looks like the curl team aren't too bothered about it.

    (tags: curl s3 amazon aws ssl tls certs sha1 rsa key-length security cacerts)

  • Cassandra moving to using G1 as the default recommended GC implementation

    This is a big indicator that G1 is ready for primetime. CMS has long been the go-to GC for production usage, but requires careful, complex hand-tuning -- if G1 is getting to a stage where it's just a case of giving it enough RAM, that'd be great. Also, looks like it'll be the JDK9 default: https://twitter.com/shipilev/status/593175793255219200

    (tags: cassandra tuning ops g1gc cms gc java jvm production performance memory)

  • The Colossal Shop

    ThisIsColossal now have a shop! bookmarking for some lovely gifts

    (tags: art design shop colossal shopping christmas gifts)

Links for 2015-04-28

Published April 28, 2015

Links for 2015-04-27

Published April 27, 2015

Links for 2015-04-26

Published April 26, 2015

  • StackShare

    'Discover and discuss the best dev tools and cloud infrastructure services' -- fun!

    (tags: stackshare architecture stack ops software ranking open-source)

  • OWASP KeyBox

    a web-based SSH console that centrally manages administrative access to systems. Web-based administration is combined with management and distribution of user's public SSH keys. Key management and administration is based on profiles assigned to defined users. Administrators can login using two-factor authentication with FreeOTP or Google Authenticator . From there they can create and manage public SSH keys or connect to their assigned systems through a web-shell. Commands can be shared across shells to make patching easier and eliminate redundant command execution.

    (tags: keybox owasp security ssh tls ssl ops)

  • 32-bit overflow in BitGo js code caused an accidental 85 BTC transaction fee

    Yes, this is a fucking 32-bit integer overflow. Whatever software was used, it calculated the sum of all inputs using 32-bit variables, which overflow at about 20 BTC if signed or 40 BTC if not. The fee was supposed to be 0xC350 = 50,000 satoshis, but it turned out to be 0x2,0000,C350 = 8,589,984,592 satoshis. Captains of the industry. If they were captains of any other industry, like say for example automotive, we'd have people dying in car crashes between two stationary vehicles.

    (tags: bitcoin fail bitgo javascript bugs 32-bit overflow btc)

  • Eight Docker Development Patterns

    good Docker tips

    (tags: tips docker ops deployment)

  • Google Online Security Blog: A Javascript-based DDoS Attack [the Greatfire DDoS] as seen by Safe Browsing

    We hope this report helps to round out the overall facts known about this attack. It also demonstrates that collectively there is a lot of visibility into what happens on the web. At the HTTP level seen by Safe Browsing, we cannot confidently attribute this attack to anyone. However, it makes it clear that hiding such attacks from detailed analysis after the fact is difficult. Had the entire web already moved to encrypted traffic via TLS, such an injection attack would not have been possible. This provides further motivation for transitioning the web to encrypted and integrity-protected communication. Unfortunately, defending against such an attack is not easy for website operators. In this case, the attack Javascript requests web resources sequentially and slowing down responses might have helped with reducing the overall attack traffic. Another hope is that the external visibility of this attack will serve as a deterrent in the future.
    Via Nelson.

    (tags: google security via:nelson ddos javascript tls ssl safe-browsing networking china greatfire)

Links for 2015-04-24

Published April 24, 2015

Links for 2015-04-23

Published April 23, 2015

  • attacks using U+202E - RIGHT-TO-LEFT OVERRIDE

    Security implications of in-band signalling strikes again, 43 years after the "Blue Box" hit the mainstream. Jamie McCarthy on Twitter: ".@cmdrtaco - Remember when we had to block the U+202E code point in Slashdot comments to stop siht ekil stnemmoc? https://t.co/TcHxKkx9Oo" See also http://krebsonsecurity.com/2011/09/right-to-left-override-aids-email-attacks/ -- GMail was vulnerable too; and http://en.wikipedia.org/wiki/Unicode_control_characters for more inline control chars. http://unicode.org/reports/tr36/#Bidirectional_Text_Spoofing has some official recommendations from the Unicode consortium on dealing with bidi override chars.

    (tags: security attacks rlo unicode control-characters codepoints bidi text gmail slashdot sanitization input)

  • Meet the e-voting machine so easy to hack, it will take your breath away | Ars Technica

    The AVS WinVote system -- mind-bogglingly shitty security.

    If an election was held using the AVS WinVote, and it wasn’t hacked, it was only because no one tried. The vulnerabilities were so severe, and so trivial to exploit, that anyone with even a modicum of training could have succeeded. They didn’t need to be in the polling place—within a few hundred feet (e.g., in the parking lot) is easy, and within a half mile with a rudimentary antenna built using a Pringles can. Further, there are no logs or other records that would indicate if such a thing ever happened, so if an election was hacked any time in the past, we will never know. I’ve been in the security field for 30 years, and it takes a lot to surprise me. But the VITA report really shocked me—as bad as I thought the problems were likely to be, VITA’s five-page report showed that they were far worse. And the WinVote system was so fragile that it hardly took any effort. While the report does not state how much effort went into the investigation, my estimation based on the description is that it was less than a person week.

    (tags: security voting via:johnke winvote avs shoup wep wifi windows)

Links for 2015-04-22

Published April 22, 2015

  • 'Continuous Deployment: The Dirty Details'

    Good slide deck from Etsy's Mike Brittain regarding their CD setup. Some interesting little-known details: Slide 41: database schema changes are not CD'd -- they go out on "Schema change Thursdays". Slide 44: only the webapp is CD'd -- PHP, Apache, memcache components (Etsy.com, support and back-office tools, developer API, gearman async worker queues). The external "services" are not -- databases, Solr/JVM search (rolling restarts), photo storage (filters, proxy cache, S3), payments (PCI-DSS, controlled access). They avoid schema changes and breaking changes using an approach they call "non-breaking expansions" -- expose new version in a service interface; support multiple versions in the consumer. Example from slides 50-63, based around a database schema migration. Slide 66: "dev flags" (rollout oriented) are promoted to "feature flags" (long lived degradation control). Slide 71: some architectural philosophies: deploying is cheap; releasing is cheap; gathering data should be cheap too; treat first iterations as experiments. Slide 102: "Canary pools". They have multiple pools of users for testing in production -- the staff pool, users who have opted in to see prototypes/beta stuff, 0-100% gradual phased rollout.

    (tags: cd deploy etsy slides migrations database schema ops ci version-control feature-flags)

  • Etsy's Release Management process

    Good info on how Etsy use their Deployinator tool, end-to-end. Slide 11: git SHA is visible for each env, allowing easy verification of what code is deployed. Slide 14: Code is deployed to "princess" staging env while CI tests are running; no need to wait for unit/CI tests to complete. Slide 23: smoke tests of pre-prod "princess" (complete after 8 mins elapsed). Slide 31: dashboard link for deployed code is posted during deploy; post-release prod smoke tests are run by Jenkins. (short ones! they complete in 42 seconds)

    (tags: deployment etsy deploy deployinator princess staging ops testing devops smoke-tests production jenkins)

  • Makerbot’s Saddest Hour | TechCrunch

    I’ve been speaking to a few people [at Makerbot] who prefer to remain anonymous and most of my contacts there are gone (the head of PR was apparently fired) and don’t want to talk. But the new from inside is troubling. The mass-layoffs are blamed on low revenue and one former employee wrote “Company was failing. Couldn’t pay vendors, had to downsize.” Do I think Makerbot will sink? At this point I don’t know.

    (tags: makerbot 3d-printing startups downsizing layoffs ouch)

  • credstash

    'CredStash is a very simple, easy to use credential management and distribution system that uses AWS Key Management System (KMS) for key wrapping and master-key storage, and DynamoDB for credential storage and sharing.'

    (tags: aws credstash python security keys key-management secrets kms)

  • ferd.ca -> Lessons Learned while Working on Large-Scale Server Software

    Good advice

    (tags: distributed scalability systems coding server-side erlang devops networking reliability)

Links for 2015-04-20

Published April 20, 2015

  • FBI admits flaws in hair analysis over decades

    Wow, this is staggering.

    The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000. [....] The review confirmed that FBI experts systematically testified to the near-certainty of “matches” of crime-scene hairs to defendants, backing their claims by citing incomplete or misleading statistics drawn from their case work. In reality, there is no accepted research on how often hair from different people may appear the same. Since 2000, the lab has used visual hair comparison to rule out someone as a possible source of hair or in combination with more accurate DNA testing. Warnings about the problem have been mounting. In 2002, the FBI reported that its own DNA testing found that examiners reported false hair matches more than 11 percent of the time.

    (tags: fbi false-positives hair dna biometrics trials justice experts crime forensics inaccuracy csi)

  • The missing MtGox bitcoins

    Most or all of the missing bitcoins were stolen straight out of the MtGox hot wallet over time, beginning in late 2011. As a result, MtGox operated at fractional reserve for years (knowingly or not), and was practically depleted of bitcoins by 2013. A significant number of stolen bitcoins were deposited onto various exchanges, including MtGox itself, and probably sold for cash (which at the bitcoin prices of the day would have been substantially less than the hundreds of millions of dollars they were worth at the time of MtGox's collapse). MtGox' bitcoins continuously went missing over time, but at a decreasing pace. Again by the middle of 2013, the curve goes more or less flat, matching the hypothesis that by that time there may not have been any more bitcoins left to lose. The rate of loss otherwise seems unusually smooth and at the same time not strictly relative to any readily available factors such as remaining BTC holdings, transaction volumes or the BTC price. Worth pointing out is that, thanks to having matched up most of the deposit/withdrawal log earlier, we can at this point at least rule out the possibility of any large-scale fake deposits — the bitcoins going into MtGox were real, meaning the discrepancy was likely rather caused by bitcoins leaving MtGox without going through valid withdrawals.

    (tags: mtgox bitcoin security fail currency theft crime btc)

  • Bank of the Underworld - The Atlantic

    Prosecutors analyzed approximately 500 of Liberty Reserve’s biggest accounts, which constituted 44 percent of its business. The government contends that 32 of these accounts were connected to the sale of stolen credit cards and 117 were used by Ponzi-scheme operators. All of this activity flourished, prosecutors said, because Liberty Reserve made no real effort to monitor its users for criminal behavior. What’s more, records showed that one of the company’s top tech experts, Mark Marmilev, who was also arrested, appeared to have promoted Liberty Reserve in chat rooms devoted to Ponzi schemes.
    (via Nelson)

    (tags: scams fraud crime currency the-atlantic liberty-reserve ponzi-schemes costa-rica arthur-budovsky banking anonymity cryptocurrency money-laundering carding)

  • I was a Lampedusa refugee. Here’s my story of fleeing Libya – and surviving

    'The boy next to me fell to the floor and for a moment I didn’t know if he had fainted or was dead – then I saw that he was covering his eyes so he didn’t have to see the waves any more. A pregnant woman vomited and started screaming. Below deck, people were shouting that they couldn’t breathe, so the men in charge of the boat went down and started beating them. By the time we saw a rescue helicopter, two days after our boat had left Libya with 250 passengers on board, some people were already dead – flung into the sea by the waves, or suffocated downstairs in the dark.'

    (tags: lampedusa migration asylum europe fortress-europe italy politics immigration libya refugees)

  • Run your own high-end cloud gaming service on EC2

    Using Steam streaming and EC2 g2.2xlarge spot instances -- 'comes out to around $0.52/hr'. That's pretty compelling IMO

    (tags: aws ec2 gaming games graphics spot-instances hacks windows steam)

  • Running Arbitrary Executables in AWS Lambda

    actually an officially-supported mode. huh

    (tags: lambda aws architecture ops node.js javascript unix linux)

Links for 2015-04-18

Published April 18, 2015

Links for 2015-04-17

Published April 17, 2015

Links for 2015-04-16

Published April 16, 2015

  • Extracting Structured Data From Recipes Using Conditional Random Fields

    nice probabilistic/ML approach to recipe parsing

    (tags: nytimes recipes parsing text nlp machine-learning probabilistic crf++ algorithms feature-extraction)

  • Large-scale cluster management at Google with Borg

    Google's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.
    (via Conall)

    (tags: via:conall clustering google papers scale to-read borg cluster-management deployment packing reliability redundancy)

  • Keeping Your Car Safe From Electronic Thieves - NYTimes.com

    In a normal scenario, when you walk up to a car with a keyless entry and try the door handle, the car wirelessly calls out for your key so you don’t have to press any buttons to get inside. If the key calls back, the door unlocks. But the keyless system is capable of searching for a key only within a couple of feet. Mr. Danev said that when the teenage girl turned on her device, it amplified the distance that the car can search, which then allowed my car to talk to my key, which happened to be sitting about 50 feet away, on the kitchen counter. And just like that, open sesame.
    What the hell -- who designed a system that would auto-unlock based on signal strength alone?!!

    (tags: security fail cars keys signal proximity keyless-entry prius toyota crime amplification power-amplifiers 3db keyless)

  • Closed access means people die

    'We've paid 100 BILLION USD over the last 10 years to "publish" science and medicine. Ebola is a massive systems failure.' See also https://www.techdirt.com/articles/20150409/17514230608/dont-think-open-access-is-important-it-might-have-prevented-much-ebola-outbreak.shtml : 'The conventional wisdom among public health authorities is that the Ebola virus, which killed at least 10,000 people in Liberia, Sierra Leone and Guinea, was a new phenomenon, not seen in West Africa before 2013. [...] But, as the team discovered, that "conventional wisdom" was wrong. In fact, they found a bunch of studies, buried behind research paywalls, that revealed that there was significant evidence of antibodies to the Ebola virus in Liberia and in other nearby nations. There was one from 1982 that noted: "medical personnel in Liberian health centers should be aware of the possibility that they may come across active cases and thus be prepared to avoid nosocomial epidemics."

    (tags: deaths liberia ebola open-access papers elsevier science medicine reprints)

  • Making Pinterest — Learn to stop using shiny new things and love MySQL

    'The third reason people go for shiny is because older tech isn’t advertised as aggressively as newer tech. The younger companies needs to differentiate from the old guard and be bolder, more passionate and promise to fulfill your wildest dreams. But most new tech sales pitches aren’t generally forthright about their many failure modes. In our early days, we fell into this third trap. We had a lot of growing pains as we scaled the architecture. The most vocal and excited database companies kept coming to us saying they’d solve all of our scalability problems. But nobody told us of the virtues of MySQL, probably because MySQL just works, and people know about it.' It's true! -- I'm still a happy MySQL user for some use cases, particularly read-mostly relational configuration data...

    (tags: mysql storage databases reliability pinterest architecture)

  • Microservices and elastic resource pools with Amazon EC2 Container Service

    interesting approach to working around ECS' shortcomings -- bit specific to Hailo's microservices arch and IPC mechanism though. aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!

    (tags: versioning microservices hailo aws ec2 ecs docker containers scheduling allocation deployment provisioning qos)

  • Please Kill Me (Eventually) | Motherboard

    There is much that the wise application of technology can do to help us ease off this mortal coil, instead of tormenting ourselves at the natural end of life in a futile, undignified and excruciating attempt to keep it somehow duct-taped on. Train more people in geriatrics, for example. Learn new ways to make life safe, healthy, fun and interesting for the old. Think like a community, a brotherhood, not like atomized competing individuals a few of whom can somehow "beat the system" of the universe. Maybe it is better to examine clearly what we are with a view to understanding and acceptance than it is to try to escape what perhaps should be our inevitable ending.

    (tags: death mortality cryogenics alcor geriatrics life singularity mind-uploading ray-kurzweil)

  • CGA in 1024 Colors - a New Mode: the Illustrated Guide

    awesome hackery. brings me back to my C=64 demo days

    (tags: pc cga graphics hacks art 1024-colours)

Links for 2015-04-15

Published April 15, 2015

  • Keywhiz

    'a secret management and distribution service [from Square] that is now available for everyone. Keywhiz helps us with infrastructure secrets, including TLS certificates and keys, GPG keyrings, symmetric keys, database credentials, API tokens, and SSH keys for external services — and even some non-secrets like TLS trust stores. Automation with Keywhiz allows us to seamlessly distribute and generate the necessary secrets for our services, which provides a consistent and secure environment, and ultimately helps us ship faster. [...] Keywhiz has been extremely useful to Square. It’s supported both widespread internal use of cryptography and a dynamic microservice architecture. Initially, Keywhiz use decoupled many amalgamations of configuration from secret content, which made secrets more secure and configuration more accessible. Over time, improvements have led to engineers not even realizing Keywhiz is there. It just works. Please check it out.'

    (tags: square security ops keys pki key-distribution key-rotation fuse linux deployment secrets keywhiz)

Links for 2015-04-14

Published April 14, 2015

Links for 2015-04-13

Published April 13, 2015

  • Amazon Machine Learning

    Upsides of this new AWS service: * great UI and visualisations. * solid choice of metric to evaluate the results. Maybe things moved on since I was working on it, but the use of AUC, false positives and false negatives was pretty new when I was working on it. (er, 10 years ago!) Downsides: * it could do with more support for unsupervised learning algorithms. Supervised learning means you need to provide training data, which in itself can be hard work. My experience with logistic regression in the past is that it requires very accurate training data, too -- its tolerance for misclassified training examples is poor. * Also, in my experience, 80% of the hard work of using ML algorithms is writing good tokenisation and feature extraction algorithms. I don't see any help for that here unfortunately. (probably not that surprising as it requires really detailed knowledge of the input data to know what classes can be abbreviated into a single class, etc.)

    (tags: amazon aws ml machine-learning auc data-science)

  • Rob Pike's 5 rules of optimization

    these are great. I've run into rule #3 ("fancy algorithms are slow when n is small, and n is usually small") several times...

    (tags: twitter rob-pike via:igrigorik coding rules laws optimization performance algorithms data-structures aphorisms)

  • AWS Lambda Event-Driven Architecture With Amazon SNS

    Any message posted to an SNS topic can trigger the execution of custom code you have written, but you don’t have to maintain any infrastructure to keep that code available to listen for those events and you don’t have to pay for any infrastructure when the code is not being run. This is, in my opinion, the first time that Amazon can truly say that AWS Lambda is event-driven, as we now have a central, independent, event management system (SNS) where any authorized entity can trigger the event (post a message to a topic) and any authorized AWS Lambda function can listen for the event, and neither has to know about the other.

    (tags: aws ec2 lambda sns events cep event-processing coding cloud hacks eric-hammond)

  • Texting at the wheel kills more US teenagers every year than drink-driving

    Texting while behind the wheel has overtaken drink driving as the biggest cause of death among teenagers in America. More than 3,000 teenagers are killed every year in car crashes caused by texting while driving compared to 2,700 from drink driving. The study by Cohen Children’s Medical Center also discovered that 50 per cent of students admit to texting while driving.

    (tags: texting sms us driving car-safety safety drink-driving)

  • China’s Great Cannon

    Conducting such a widespread attack clearly demonstrates the weaponization of the Chinese Internet to co-opt arbitrary computers across the web and outside of China to achieve China’s policy ends.  The repurposing of the devices of unwitting users in foreign jurisdictions for covert attacks in the interests of one country’s national priorities is a dangerous precedent — contrary to international norms and in violation of widespread domestic laws prohibiting the unauthorized use of computing and networked systems.

    (tags: censorship ddos internet security china great-cannon citizen-lab reports web)

  • Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers

    How to build an Intelligent Personal Assistant: 'Sirius is an open end-to-end standalone speech and vision based intelligent personal assistant (IPA) similar to Apple’s Siri, Google’s Google Now, Microsoft’s Cortana, and Amazon’s Echo. Sirius implements the core functionalities of an IPA including speech recognition, image matching, natural language processing and a question-and-answer system. Sirius is developed by Clarity Lab at the University of Michigan. Sirius is published at the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2015.'

    (tags: sirius siri cortana google-now echo ok-google ipa assistants search video audio speech papers clarity nlp wikipedia)

  • Why We Will Not Be Registering easyDNS.SUCKS - blog.easydns.org

    If you're not immersed in the naming business you may find the jargon in it hard to understand. The basic upshot is this: the IPC believes that the mechanisms that were enacted to protect trademark holders during the deluge of new TLD rollouts are being gamed by the .SUCKS TLD operator to extort inflated fees from trademark holders.
    (via Nelson)

    (tags: shakedown business internet domains dns easydns dot-sucks scams tlds trademarks ip)

Links for 2015-04-12

Published April 12, 2015

Links for 2015-04-11

Published April 11, 2015

Links for 2015-04-10

Published April 10, 2015

  • Hacked French network exposed its own passwords during TV interview

    lols

    (tags: passwords post-its fail tv5monde authentication security tv funny)

  • RADStack - an open source Lambda Architecture built on Druid, Kafka and Samza

    'In this paper we presented the RADStack, a collection of complementary technologies that can be used together to power interactive analytic applications. The key pieces of the stack are Kafka, Samza, Hadoop, and Druid. Druid is designed for exploratory analytics and is optimized for low latency data exploration, aggregation, and ingestion, and is well suited for OLAP workflows. Samza and Hadoop complement Druid and add data processing functionality, and Kafka enables high throughput event delivery.'

    (tags: druid samza kafka streaming cep lambda-architecture architecture hadoop big-data olap)

  • outbrain/gruffalo

    an asynchronous Netty based graphite proxy. It protects Graphite from the herds of clients by minimizing context switches and interrupts; by batching and aggregating metrics. Gruffalo also allows you to replicate metrics between Graphite installations for DR scenarios, for example. Gruffalo can easily handle a massive amount of traffic, and thus increase your metrics delivery system availability. At Outbrain, we currently handle over 1700 concurrent connections, and over 2M metrics per minute per instance.

    (tags: graphite backpressure metrics outbrain netty proxies gruffalo ops)

  • Privacy Security Talk in TOG – 22nd April @ 7pm – FREE

    Dublin is lucky enough to have great speakers pass through town on occasion and on Wednesday the 22nd April 2015, Runa A. Sandvik (@runasand) and Per Thorsheim (@thorsheim) have kindly offered to speak in TOG from 7pm. The format for the evening is a general meet and greet, but both speakers have offered to give a presentation on a topic of their choice. Anyone one interested in privacy, security, journalism, Tor and/or has previously attended a CryptoParty would be wise to attend. Doors are from 7pm and bring any projects with you you would like to share with other attendees. This is a free event, open to the public and no need to book. See you Wednesday. Runa A. Sandvik is an independent privacy and security researcher, working at the intersection of technology, law and policy. She contributes to The Tor Project, writes for Forbes, and is a technical advisor to both the Freedom of the Press Foundation and the TrueCrypt Audit project. Per Thorsheim as founder/organizer of PasswordsCon.org, his topic of choice is of course passwords, but in a much bigger context than most people imagine. Passwords, pins, biometrics, 2-factor authentication, security/usability and all the way into surveillance and protecting your health, kids and life itself.

    (tags: privacy security runa-sandvik per-thorsheim passwords tor truecrypt tog via:oisin events dublin)

Links for 2015-04-09

Published April 9, 2015

Links for 2015-04-08

Published April 8, 2015

Links for 2015-04-07

Published April 7, 2015