Skip to content

Category: Uncategorized

Links for 2015-11-12

Published November 12, 2015

Links for 2015-11-11

Published November 11, 2015

  • Dynalite

    Awesome new mock DynamoDB implementation:

    An implementation of Amazon's DynamoDB, focussed on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise). This project aims to match the live DynamoDB instances as closely as possible (and is tested against them in various regions), including all limits and error messages. Why not Amazon's DynamoDB Local? Because it's too buggy! And it differs too much from the live instances in a number of key areas.
    We use DynamoDBLocal in our tests -- the availability of that tool is one of the key reasons we have adopted Dynamo so heavily, since we can safely test our code properly with it. This looks even better.

    (tags: dynamodb testing unit-tests integration-testing tests ops dynalite aws leveldb)

  • Alarm design: From nuclear power to WebOps

    Imagine you are an operator in a nuclear power control room. An accident has started to unfold. During the first few minutes, more than 100 alarms go off, and there is no system for suppressing the unimportant signals so that you can concentrate on the significant alarms. Information is not presented clearly; for example, although the pressure and temperature within the reactor coolant system are shown, there is no direct indication that the combination of pressure and temperature mean that the cooling water is turning into steam. There are over 50 alarms lit in the control room, and the computer printer registering alarms is running more than 2 hours behind the events. This was the basic scenario facing the control room operators during the Three Mile Island (TMI) partial nuclear meltdown in 1979. The Report of the President’s Commission stated that, “Overall, little attention had been paid to the interaction between human beings and machines under the rapidly changing and confusing circumstances of an accident” (p. 11). The TMI control room operator on the day, Craig Faust, recalled for the Commission his reaction to the incessant alarms: “I would have liked to have thrown away the alarm panel. It wasn’t giving us any useful information”. It was the first major illustration of the alarm problem, and the accident triggered a flurry of human factors/ergonomics (HF/E) activity.
    A familiar topic for this ex-member of the Amazon network monitoring team...

    (tags: ergonomics human-factors ui ux alarms alerts alerting three-mile-island nuclear-power safety outages ops)

  • An Analysis of Reshipping Mule Scams

    We observed that the vast majority of the re-shipped packages end up in the Moscow, Russia area, and that the goods purchased with stolen credit cards span multiple categories, from expensive electronics such as Apple products, to designer clothes, to DSLR cameras and even weapon accessories. Given the amount of goods shipped by the reshipping mule sites that we analysed, the annual revenue generated from such operations can span between 1.8 and 7.3 million US dollars. The overall losses are much higher though: the online merchant loses an expensive item from its inventory and typically has to refund the owner of the stolen credit card. In addition, the rogue goods typically travel labeled as “second hand goods” and therefore custom taxes are also evaded. Once the items purchased with stolen credit cards reach their destination they will be sold on the black market by cybercriminals. [...] When applying for the job, people are usually required to send the operator copies of their ID cards and passport. After they are hired, mules are promised to be paid at the end of their first month of employment. However, from our data it is clear that mules are usually never paid. After their first month expires, they are never contacted back by the operator, who just moves on and hires new mules. In other words, the mules become victims of this scam themselves, by never seeing a penny. Moreover, because they sent copies of their documents to the criminals, mules can potentially become victims of identity theft.

    (tags: crime law cybercrime mules shipping-scams identity-theft russia moscow scams papers)

Links for 2015-11-10

Published November 10, 2015

  • No Harm, No Fowl: Chicken Farm Inappropriate Choice for Data Disposal

    That’s a lesson that Spruce Manor Special Care Home in Saskatchewan had to learn the hard way (as surprising as that might sound). As a trustee with custody of personal health information, Spruce Manor was required under section 17(2) of the Saskatchewan Health Information Protection Act to dispose of its patient records in a way that protected patient privacy. So, when Spruce Manor chose a chicken farm for the job, it found itself the subject of an investigation by the Saskatchewan Information and Privacy Commissioner.  In what is probably one of the least surprising findings ever, the commissioner wrote in his final report that “I recommend that Spruce Manor […] no longer use [a] chicken farm to destroy records”, and then for good measure added “I find using a chicken farm to destroy records unacceptable.”

    (tags: data law privacy funny chickens farming via:pinboard data-protection health medical-records)

Links for 2015-11-09

Published November 9, 2015

  • Caffeine cache adopts Window TinyLfu eviction policy

    'Caffeine is a Java 8 rewrite of Guava's cache. In this version we focused on improving the hit rate by evaluating alternatives to the classic least-recenty-used (LRU) eviction policy. In collaboration with researchers at Israel's Technion, we developed a new algorithm that matches or exceeds the hit rate of the best alternatives (ARC, LIRS). A paper of our work is being prepared for publication.' Specifically:

    W-TinyLfu uses a small admission LRU that evicts to a large Segmented LRU if accepted by the TinyLfu admission policy. TinyLfu relies on a frequency sketch to probabilistically estimate the historic usage of an entry. The window allows the policy to have a high hit rate when entries exhibit a high temporal / low frequency access pattern which would otherwise be rejected. The configuration enables the cache to estimate the frequency and recency of an entry with low overhead. This implementation uses a 4-bit CountMinSketch, growing at 8 bytes per cache entry to be accurate. Unlike ARC and LIRS, this policy does not retain non-resident keys.

    (tags: tinylfu caches caching cache-eviction java8 guava caffeine lru count-min sketching algorithms)

  • What Do WebLogic, WebSphere, JBoss, Jenkins, OpenNMS, and Your Application Have in Common? This Vulnerability.

    The ever-shitty Java serialization creates a security hole

    (tags: java serialization security exploits jenkins)

  • Gallery - Steffen Dam

    Danish glassware artist making wonderful Wunderkammers -- cabinets of curiosities --- entirely from glass. Seeing as one of his works sold for UKP50,000 last year, I suspect these are a bit out of my league, sadly

    (tags: art glassware steffen-dam wunderkammers museums)

  • London garden bridge users to have mobile phone signals tracked

    If it goes ahead, people’s progress across the structure would be tracked by monitors detecting the Wi-Fi signals from their phones, which show up the device’s Mac address, or unique identifying code. The Garden Bridge Trust says it will not store any of this data and is only tracking phones to count numbers and prevent overcrowding.

    (tags: london surveillance mobile-phones mac-trackers tracking)

  • Red lines and no-go zones - the coming surveillance debate

    The Anderson Report to the House of Lords in the UK on RIPA introduces a concept of a "red line":

    "Firm limits must also be written into the law: not merely safeguards, but red lines that may not be crossed." …    "Some might find comfort in a world in which our every interaction and movement could be recorded, viewed in real time and indefinitely retained for possible future use by the authorities. Crime fighting, security, safety or public health justifications are never hard to find." [13.19]  The Report then gives examples, such as a perpetual video feed from every room in every house, the police undertaking to view the record only on receipt of a complaint; blanket drone-based surveillance; licensed service providers, required as a condition of the licence to retain within the jurisdiction a complete plain-text version of every communication to be made available to the authorities on request; a constant data feed from vehicles, domestic appliances and health-monitoring personal devices; fitting of facial recognition software to every CCTV camera and the insertion of a location-tracking chip under every individual's skin. It goes on: "The impact of such powers on the innocent could be mitigated by the usual apparatus of safeguards, regulators and Codes of Practice. But a country constructed on such a basis would surely be intolerable to many of its inhabitants. A state that enjoyed all those powers would be truly totalitarian, even if the authorities had the best interests of its people at heart." [13.20] …   "The crucial objection is that of principle. Such a society would have gone beyond Bentham's Panopticon (whose inmates did not know they were being watched) into a world where constant surveillance was a certainty and quiescence the inevitable result. There must surely come a point (though it comes at different places for different people) where the escalation of intrusive powers becomes too high a price to pay for a safer and more law abiding environment." [13.21]

    (tags: panopticon jeremy-bentham law uk dripa ripa surveillance spying police drones facial-recognition future tracking cctv crime)

  • Dublin is a medium-density city

    Comparable to Copenhagen or Amsterdam, albeit without sufficient cycling/public-transport infrastructural investment

    (tags: infrastructure density housing dublin ireland cities travel commuting cycling)

Links for 2015-11-07

Published November 7, 2015

  • Ignoring ESR won't do anymore

    I'm tired of this shit. Full stop tired. It's 2015 and these turds who grope their way around conferences and the like can make allegations like this, get a hand wave and an, "Oh, that's just crazy Raymond!" Fuck that. Fuck it from here to hell and back. Here's a man who really hasn't done anything all that special, is a totally crazy gun-toting misogynist of the highest order and, yet, he remains mostly unchallenged after the tempest dies down, time after time. [...] I'm sure ESR will still be haunting conferences when your daughters reach their professional years unless you get serious about outing the assholes like him and making the community a lot less toxic than it is now.?
    Amen to that.

    (tags: esr toxic harassment conferences sexism misogyny culture)

Links for 2015-11-05

Published November 5, 2015

Links for 2015-11-04

Published November 4, 2015

  • PICO-8:

    PICO-8 is a fantasy console for making, sharing and playing tiny games and other computer programs. When you turn it on, the machine greets you with a shell for typing in Lua commands and provides simple built-in tools for creating your own cartridges.
    So cute! See also Voxatron, something similar for voxel-oriented 3D gaming

    (tags: consoles games gaming lua coding retro 2d pico-8)

  • Why Static Website Generators Are The Next Big Thing

    Now _this_ makes me feel old. Alternative title: "why static website generators have been a good idea since WebMake, 15 years ago". WebMake does pretty well on the checklist of "key features of the modern static website generator", which are: 1. Templating (check); 2. Markdown support (well, EtText, which predated Markdown by several years); 3. Metadata (check); and 4. Javascript asset pipeline (didn't support this one, since complex front-end DHTML JS wasn't really a thing at the turn of the century. But I would have if it had ;). So I guess I was on the right track!

    (tags: web html history webmake static-sites bake-dont-fry site-generators cms)

  • Food Trucks Are Great Incubators. Why Don't We Have More?

    So is that kind of thriving food-truck scene something the city should work to encourage? Theresa Hernandez, one of the owners of K Chido Mexico, thinks so. “There’s a whole market there for a new culture,” she says. “There’s no doubt about it, the appetite is there. It’s just a matter for somebody who is innovative enough in Dublin City Council to say: ‘Right, let’s do this.’”
    Amen to that.

    (tags: k-chido food-trucks dublin food ireland dcc)

  • wangle/Codel.h at master · facebook/wangle

    Facebook's open-source implementation of the CoDel queue management algorithm applied to server request-handling capacity in their C++ service bootstrap library, Wangle.

    (tags: wangle facebook codel services capacity reliability queueing)

Links for 2015-11-02

Published November 2, 2015

  • Structural and semantic deficiencies in the systemd architecture for real-world service management, a technical treatise

    Despite its overarching abstractions, it is semantically non-uniform and its complicated transaction and job scheduling heuristics ordered around a dependently networked object system create pathological failure cases with little debugging context that would otherwise not necessarily occur on systems with less layers of indirection. The use of bus APIs complicate communication with the service manager and lead to duplication of the object model for little gain. Further, the unit file options often carry implicit state or are not sufficiently expressive. There is an imbalance with regards to features of an eager service manager and that of a lazy loading service manager, having rusty edge cases of both with non-generic, manager-specific facilities. The approach to logging and the circularly dependent architecture seem to imply that lots of prior art has been ignored or understudied.

    (tags: analysis systemd linux unix ops init critiques software logging)

  • How Facebook avoids failures

    Great paper from Ben Maurer of Facebook in ACM Queue.

    A "move-fast" mentality does not have to be at odds with reliability. To make these philosophies compatible, Facebook's infrastructure provides safety valves.
    This is full of interesting techniques. * Rapidly deployed configuration changes: Make everybody use a common configuration system; Statically validate configuration changes; Run a canary; Hold on to good configurations; Make it easy to revert. * Hard dependencies on core services: Cache data from core services. Provide hardened APIs. Run fire drills. * Increased latency and resource exhaustion: Controlled Delay (based on the anti-bufferbloat CoDel algorithm -- this is really cool); Adaptive LIFO (last-in, first-out) for queue busting; Concurrency Control (essentially a form of circuit breaker). * Tools that Help Diagnose Failures: High-Density Dashboards with Cubism (horizon charts); What just changed? * Learning from Failure: the DERP (!) methodology,

    (tags: ben-maurer facebook reliability algorithms codel circuit-breakers derp failure ops cubism horizon-charts charts dependencies soa microservices uptime deployment configuration change-management)

Links for 2015-11-01

Published November 1, 2015

Links for 2015-10-30

Published October 30, 2015

Links for 2015-10-29

Published October 29, 2015

  • Google tears Symantec a new one on its CA failure

    Symantec are getting a crash course in how to conduct an incident post-mortem to boot:

    More immediately, we are requesting of Symantec that they further update their public incident report with: A post-mortem analysis that details why they did not detect the additional certificates that we found. Details of each of the failures to uphold the relevant Baseline Requirements and EV Guidelines and what they believe the individual root cause was for each failure. We are also requesting that Symantec provide us with a detailed set of steps they will take to correct and prevent each of the identified failures, as well as a timeline for when they expect to complete such work. Symantec may consider this latter information to be confidential and so we are not requesting that this be made public.

    (tags: google symantec ev ssl certificates ca security postmortems ops)

  • Google is Maven Central's New Best Friend

    google now mirroring Maven Central.

    (tags: google maven maven-central jars hosting java packages build)

  • Apache Kafka, Purgatory, and Hierarchical Timing Wheels

    In the new design, we use Hierarchical Timing Wheels for the timeout timer and DelayQueue of timer buckets to advance the clock on demand. Completed requests are removed from the timer queue immediately with O(1) cost. The buckets remain in the delay queue, however, the number of buckets is bounded. And, in a healthy system, most of the requests are satisfied before timeout, and many of the buckets become empty before pulled out of the delay queue. Thus, the timer should rarely have the buckets of the lower interval. The advantage of this design is that the number of requests in the timer queue is the number of pending requests exactly at any time. This allows us to estimate the number of requests need to be purged. We can avoid unnecessary purge operation of the watcher lists. As the result we achieve a higher scalability in terms of request rate with much better CPU usage.

    (tags: algorithms timers kafka scheduling timing-wheels delayqueue queueing)

Links for 2015-10-28

Published October 28, 2015

Links for 2015-10-27

Published October 27, 2015

Links for 2015-10-23

Published October 23, 2015

Links for 2015-10-22

Published October 22, 2015

Links for 2015-10-21

Published October 21, 2015

  • How a criminal ring defeated the secure chip-and-PIN credit cards | Ars Technica

    Ingenious --

    The stolen cards were still considered evidence, so the researchers couldn’t do a full tear-down or run any tests that would alter the data on the card, so they used X-ray scans to look at where the chip cards had been tampered with. They also analyzed the way the chips distributed electricity when in use and used read-only programs to see what information the cards sent to a Point of Sale (POS) terminal. According to the paper, the fraudsters were able to perform a man-in-the-middle attack by programming a second hobbyist chip called a FUN card to accept any PIN entry, and soldering that chip onto the card’s original chip. This increased the thickness of the chip from 0.4mm to 0.7mm, "making insertion into a PoS somewhat uneasy but perfectly feasible,” the researchers write. [....] The researchers explain that a typical EMV transaction involves three steps: card authentication, cardholder verification, and then transaction authorization. During a transaction using one of the altered cards, the original chip was allowed to respond with the card authentication as normal. Then, during card holder authentication, the POS system would ask for a user’s PIN, the thief would respond with any PIN, and the FUN card would step in and send the POS the code indicating that it was ok to proceed with the transaction because the PIN checked out. During the final transaction authentication phase, the FUN card would relay the transaction data between the POS and the original chip, sending the issuing bank an authorization request cryptogram which the card issuer uses to tell the POS system whether to accept the transaction or not.

    (tags: security chip-and-pin hacking pos emv transactions credit-cards debit-cards hardware chips pin fun-cards smartcards)

  • How-to: Index Scanned PDFs at Scale Using Fewer Than 50 Lines of Code

    using Spark, Tesseract, HBase, Solr and Leptonica. Actually pretty feasible

    (tags: spark tesseract hbase solr leptonica pdfs scanning cloudera hadoop architecture)

  • Existential Consistency: Measuring and Understanding Consistency at Facebook

    The metric is termed ?(P)-consistency, and is actually very simple. A read for the same data is sent to all replicas in P, and ?(P)-consistency is defined as the frequency with which that read returns the same result from all replicas. ?(G)-consistency applies this metric globally, and ?(R)-consistency applies it within a region (cluster). Facebook have been tracking this metric in production since 2012.

    (tags: facebook eventual-consistency consistency metrics papers cap distributed-computing)

  • Holistic Configuration Management at Facebook

    How FB push config changes from Git (where it is code reviewed, version controlled, and history tracked with strong auth) to Zeus (their Zookeeper fork) and from there to live production servers.

    (tags: facebook configuration zookeeper git ops architecture)

  • Hyperscan

    a high-performance multiple regex matching library. Hyperscan uses hybrid automata techniques to allow simultaneous matching of large numbers (up to tens of thousands) of regular expressions and for the matching of regular expressions across streams of data.
    Via Tony Finch

    (tags: via:fanf regexps regex dpi hyperscan dfa nfa hybrid-automata text-matching matching text strings streams)

Links for 2015-10-20

Published October 20, 2015

  • Hologram

    Hologram exposes an imitation of the EC2 instance metadata service on developer workstations that supports the [IAM Roles] temporary credentials workflow. It is accessible via the same HTTP endpoint to calling SDKs, so your code can use the same process in both development and production. The keys that Hologram provisions are temporary, so EC2 access can be centrally controlled without direct administrative access to developer workstations.

    (tags: iam roles ec2 authorization aws adroll open-source cli osx coding dev)

Links for 2015-10-18

Published October 18, 2015

Links for 2015-10-16

Published October 16, 2015

  • Your Relative's DNA Could Turn You Into A Suspect

    Familial DNA searching has massive false positives, but is being used to tag suspects:

    The bewildered Usry soon learned that he was a suspect in the 1996 murder of an Idaho Falls teenager named Angie Dodge. Though a man had been convicted of that crime after giving an iffy confession, his DNA didn’t match what was found at the crime scene. Detectives had focused on Usry after running a familial DNA search, a technique that allows investigators to identify suspects who don’t have DNA in a law enforcement database but whose close relatives have had their genetic profiles cataloged. In Usry’s case the crime scene DNA bore numerous similarities to that of Usry’s father, who years earlier had donated a DNA sample to a genealogy project through his Mormon church in Mississippi. That project’s database was later purchased by Ancestry, which made it publicly searchable—a decision that didn’t take into account the possibility that cops might someday use it to hunt for genetic leads. Usry, whose story was first reported in The New Orleans Advocate, was finally cleared after a nerve-racking 33-day wait — the DNA extracted from his cheek cells didn’t match that of Dodge’s killer, whom detectives still seek. But the fact that he fell under suspicion in the first place is the latest sign that it’s time to set ground rules for familial DNA searching, before misuse of the imperfect technology starts ruining lives.

    (tags: dna familial-dna false-positives law crime idaho murder mormon genealogy ancestry.com databases biometrics privacy genes)

Links for 2015-10-15

Published October 15, 2015

  • Cluster benchmark: Scylla vs Cassandra

    ScyllaDB (the C* clone in C++) is now actually looking promising -- still need more reassurance about its consistency/reliabilty side though

    (tags: scylla databases storage cassandra nosql)

  • _What We Know About Spreadsheet Errors_ [paper]

    As we will see below, there has long been ample evidence that errors in spreadsheets are pandemic. Spreadsheets, even after careful development, contain errors in one percent or more of all formula cells. In large spreadsheets with thousands of formulas, there will be dozens of undetected errors. Even significant errors may go undetected because formal testing in spreadsheet development is rare and because even serious errors may not be apparent.

    (tags: business coding maths excel spreadsheets errors formulas error-rate)

  • Defending Your Time

    great post from Ross Duggan on avoiding developer burnout

    (tags: coding burnout productivity work)

  • How is NSA breaking so much crypto?

    If a client and server are speaking Diffie-Hellman, they first need to agree on a large prime number with a particular form. There seemed to be no reason why everyone couldn’t just use the same prime, and, in fact, many applications tend to use standardized or hard-coded primes. But there was a very important detail that got lost in translation between the mathematicians and the practitioners: an adversary can perform a single enormous computation to “crack” a particular prime, then easily break any individual connection that uses that prime. How enormous a computation, you ask? Possibly a technical feat on a scale (relative to the state of computing at the time) not seen since the Enigma cryptanalysis during World War II. Even estimating the difficulty is tricky, due to the complexity of the algorithm involved, but our paper gives some conservative estimates. For the most common strength of Diffie-Hellman (1024 bits), it would cost a few hundred million dollars to build a machine, based on special purpose hardware, that would be able to crack one Diffie-Hellman prime every year. Would this be worth it for an intelligence agency? Since a handful of primes are so widely reused, the payoff, in terms of connections they could decrypt, would be enormous. Breaking a single, common 1024-bit prime would allow NSA to passively decrypt connections to two-thirds of VPNs and a quarter of all SSH servers globally. Breaking a second 1024-bit prime would allow passive eavesdropping on connections to nearly 20% of the top million HTTPS websites. In other words, a one-time investment in massive computation would make it possible to eavesdrop on trillions of encrypted connections.
    (via Eric)

    (tags: via:eric encryption privacy security nsa crypto)

Links for 2015-10-14

Published October 14, 2015

Links for 2015-10-13

Published October 13, 2015

  • Chromecast Speakers

    Supports Spotify -- totally getting one of these

    (tags: spotify speakers music home google gadgets toget)

  • Where do 'mama'/'papa' words come from?

    The sounds came first — as experiments in vocalization — and parents adopted them as pet names for themselves. If you open your mouth and make a sound, it will probably be an open vowel like /a/ unless you move your tongue or lips. The easiest consonants are perhaps the bilabials /m/, /p/, and /b/, requiring no movement of the tongue, followed by consonants made by raising the front of the tongue: /d/, /t/, and /n/. Add a dash of reduplication, and you get mama, papa, baba, dada, tata, nana. That such words refer to people (typically parents or other guardians) is something we have imposed on the sounds and incorporated into our languages and cultures; the meanings don’t inhere in the sounds as uttered by babies, which are more likely calls for food or attention.

    (tags: sounds voice speech babies kids phonetics linguist language)

  • remind101/conveyor

    'A fast build system for Docker images', open source, in Go, hooks into Github

    (tags: build ci docker github go)

  • England opens up 11TB of LiDAR data covering the entire country as open data

    All 11 terabytes of our LIDAR data (that’s roughly equivalent to 2,750,000 MP3 songs) will eventually be available through our new Open LIDAR portal under an Open Government Licence, allowing it to be used for any purpose. We hope that by giving free access to our data businesses and local communities will develop innovative solutions to benefit the environment, grow our thriving rural economy, and boost our world-leading food and farming industry. The possibilities are endless and we hope that making LIDAR data open will be a catalyst for new ideas and innovation.
    Are you reading, Ordnance Survey Ireland?

    (tags: data maps uk lidar mapping geodata open-data ogl)

Links for 2015-10-12

Published October 12, 2015

  • SuperChief: From Apache Storm to In-House Distributed Stream Processing

    Another sorry tale of Storm issues:

    Storm has been successful at Librato, but we experienced many of the limitations cited in the Twitter Heron: Stream Processing at Scale paper and outlined here by Adrian Colyer, including: Inability to isolate, reason about, or debug performance issues due to the worker/executor/task paradigm. This led to building and configuring clusters specifically designed to attempt to mitigate these problems (i.e., separate clusters per topology, only running a worker per server.), which added additional complexity to development and operations and also led to over-provisioning. Ability of tasks to move around led to difficult to trace performance problems. Storm’s work provisioning logic led to some tasks serving more Kafka partitions than others. This in turn created latency and performance issues that were difficult to reason about. The initial solution was to over-provision in an attempt to get a better hashing/balancing of work, but eventually we just replaced the work allocation logic. Due to Storm’s architecture, it was very difficult to get a stack trace or heap dump because the processes that managed workers (Storm supervisor) would often forcefully kill a Java process while it was being investigated in this way. The propensity for unexpected and subsequently unhandled exceptions to take down an entire worker led to additional defensive verbose error handling everywhere. This nasty bug STORM-404 coupled with the aforementioned fact that a single exception can take down a worker led to several cascading failures in production, taking down entire topologies until we upgraded to 0.9.4. Additionally, we found the performance we were getting from Storm for the amount of money we were spending on infrastructure was not in line with our expectations. Much of this is due to the fact that, depending upon how your topology is designed, a single tuple may make multiple hops across JVMs, and this is very expensive. For example, in our time series aggregation topologies a single tuple may be serialized/deserialized and shipped across the wire 3-4 times as it progresses through the processing pipeline.

    (tags: scalability storm kafka librato architecture heron ops)

  • librato/disco-java

    Librato's service discovery library using Zookeeper (so strongly consistent, but with the ZK downside that an AZ outage can stall service discovery updates region-wide)

    (tags: zookeeper service-discovery librato java open-source load-balancing)

  • Tech companies like Facebook not above the law, says Max Schrems

    “Big companies didn’t only rely on safe harbour: they also rely on binding corporate rules and standard contractual clauses. But it’s interesting that the court decided the case on fundamental rights grounds: so it doesn’t matter remotely what ground you transfer on, if that process is still illegal under 7 and 8 of charter, it can’t be done.”
    Also:
    “Ireland has no interest in doing its job, and will continue not to, forever. Clearly it’s an investment issue – but overall the policy is: we don’t regulate companies here. The cost of challenging any of this in the courts is prohibitive. And the people don’t seem to care.”
    :(

    (tags: ireland guardian max-schrems privacy surveillance safe-harbor eu us nsa dpc data-protection)

  • After Bara: All your (Data)base are belong to us

    Sounds like the CJEU's Bara decision may cause problems for the Irish government's wilful data-sharing:

    Articles 10, 11 and 13 of Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995, on the protection of individuals with regard to the processing of personal data and on the free movement of such data, must be interpreted as precluding national measures, such as those at issue in the main proceedings, which allow a public administrative body of a Member State to transfer personal data to another public administrative body and their subsequent processing, without the data subjects having been informed of that transfer or processing.

    (tags: data databases bara cjeu eu law privacy data-protection)

Links for 2015-10-10

Published October 10, 2015

  • Outage postmortem (2015-10-08 UTC) : Stripe: Help & Support

    There was a breakdown in communication between the developer who requested the index migration and the database operator who deleted the old index. Instead of working on the migration together, they communicated in an implicit way through flawed tooling. The dashboard that surfaced the migration request was missing important context: the reason for the requested deletion, the dependency on another index’s creation, and the criticality of the index for API traffic. Indeed, the database operator didn’t have a way to check whether the index had recently been used for a query.
    Good demo of how the Etsy-style chatops deployment approach would have helped avoid this risk.

    (tags: stripe postmortem outages databases indexes deployment chatops deploy ops)

  • net.wars: Unsafe harbor

    Wendy Grossman on where the Safe Harbor decision is leading.

    One clause would require European companies to tell their relevant data protection authorities if they are being compelled to turn over data - even if they have been forbidden to disclose this under US law. Sounds nice, but doesn't mobilize the rock or soften the hard place, since companies will still have to pick a law to violate. I imagine the internal discussions there revolving around two questions: which violation is less likely to land the CEO in jail and which set of fines can we afford?
    (via Simon McGarr)

    (tags: safe-harbor privacy law us eu surveillance wendy-grossman via:tupp_ed)

  • CHICKEN COOP & RUN

    bookmarking as a potential future addition to the back garden

    (tags: chickens pets food garden ebay)

Links for 2015-10-09

Published October 9, 2015

Links for 2015-10-08

Published October 8, 2015

  • Fuzzing Raft for Fun and Publication

    Good intro to fuzz-testing a distributed system; I've had great results using similar approaches in unit tests

    (tags: fuzzing fuzz-testing testing raft akka tests)

  • EC2 Spot Blocks for Defined-Duration Workloads

    you can now launch Spot instances that will run continuously for a finite duration (1 to 6 hours). Pricing is based on the requested duration and the available capacity, and is typically 30% to 45% less than On-Demand.

    (tags: ec2 aws spot-instances spot pricing time)

  • The Surveillance Elephant in the Room…

    Very perceptive post on the next steps for safe harbor, post-Schrems.

    And behind that elephant there are other elephants: if US surveillance and surveillance law is a problem, then what about UK surveillance? Is GCHQ any less intrusive than the NSA? It does not seem so – and this puts even more pressure on the current reviews of UK surveillance law taking place. If, as many predict, the forthcoming Investigatory Powers Bill will be even more intrusive and extensive than current UK surveillance laws this will put the UK in a position that could rapidly become untenable. If the UK decides to leave the EU, will that mean that the UK is not considered a safe place for European data? Right now that seems the only logical conclusion – but the ramifications for UK businesses could be huge. [....] What happens next, therefore, is hard to foresee. What cannot be done, however, is to ignore the elephant in the room. The issue of surveillance has to be taken on. The conflict between that surveillance and fundamental human rights is not a merely semantic one, or one for lawyers and academics, it’s a real one. In the words of historian and philosopher Quentin Skinner “the current situation seems to me untenable in a democratic society.” The conflict over Safe Harbor is in many ways just a symptom of that far bigger problem. The biggest elephant of all.

    (tags: ec cjeu surveillance safe-harbor schrems privacy europe us uk gchq nsa)

  • ECJ ruling on Irish privacy case has huge significance

    The only current way to comply with EU law, the judgment indicates, is to keep EU data within the EU. Whether those data can be safely managed within facilities run by US companies will not be determined until the US rules on an ongoing Microsoft case. Microsoft stands in contempt of court right now for refusing to hand over to US authorities, emails held in its Irish data centre. This case will surely go to the Supreme Court and will be an extremely important determination for the cloud business, and any company or individual using data centre storage. If Microsoft loses, US multinationals will be left scrambling to somehow, legally firewall off their EU-based data centres from US government reach.
    (cough, Amazon)

    (tags: aws hosting eu privacy surveillance gchq nsa microsoft ireland)

Links for 2015-10-07

Published October 7, 2015

Links for 2015-10-06

Published October 6, 2015

  • Marvin.ie: Order Takeaway Food Online

    new Dublin delivery service takes Bitcoin?!

    (tags: bitcoin food delivery takeaway payment ireland dublin wtf)

  • qp tries: smaller and faster than crit-bit tries

    interesting new data structure from Tony Finch. "Some simple benchmarks say qp tries have about 1/3 less memory overhead and are about 10% faster than crit-bit tries."

    (tags: crit-bit popcount bits bitmaps tries data-structures via:fanf qp-tries crit-bit-tries hacks memory)

  • Schneier on Automatic Face Recognition and Surveillance

    When we talk about surveillance, we tend to concentrate on the problems of data collection: CCTV cameras, tagged photos, purchasing habits, our writings on sites like Facebook and Twitter. We think much less about data analysis. But effective and pervasive surveillance is just as much about analysis. It's sustained by a combination of cheap and ubiquitous cameras, tagged photo databases, commercial databases of our actions that reveal our habits and personalities, and ­-- most of all ­-- fast and accurate face recognition software. Don't expect to have access to this technology for yourself anytime soon. This is not facial recognition for all. It's just for those who can either demand or pay for access to the required technologies ­-- most importantly, the tagged photo databases. And while we can easily imagine how this might be misused in a totalitarian country, there are dangers in free societies as well. Without meaningful regulation, we're moving into a world where governments and corporations will be able to identify people both in real time and backwards in time, remotely and in secret, without consent or recourse. Despite protests from industry, we need to regulate this budding industry. We need limitations on how our images can be collected without our knowledge or consent, and on how they can be used. The technologies aren't going away, and we can't uninvent these capabilities. But we can ensure that they're used ethically and responsibly, and not just as a mechanism to increase police and corporate power over us.

    (tags: privacy regulation surveillance bruce-schneier faces face-recognition machine-learning ai cctv photos)

Links for 2015-10-05

Published October 5, 2015

Links for 2015-10-02

Published October 2, 2015

Links for 2015-10-01

Published October 1, 2015

Links for 2015-09-30

Published September 30, 2015

Links for 2015-09-29

Published September 29, 2015

Links for 2015-09-28

Published September 28, 2015

Links for 2015-09-24

Published September 24, 2015

  • Byteman

    a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don't need 100% hindsight to be able to obtain the information you need.

    (tags: tracing java byteman injection jvm ops debugging testing)

  • Henry Robinson on testing and fault discovery in distributed systems

    'Let's talk about finding bugs in distributed systems for a bit. These chaos monkey-style fault testing systems are all well and good, but by being application independent they're a very blunt instrument. Particularly they make it hard to search the fault space for bugs in a directed manner, because they don't 'know' what the system is doing. Application-aware scripting of faults in a dist. systems seems to be rarely used, but allows you to directly stress problem areas. For example, if a bug manifests itself only when one RPC returns after some timeout, hard to narrow that down with iptables manipulation. But allow a script to hook into RPC invocations (and other trace points, like DTrace's probes), and you can script very specific faults. That way you can simulate cross-system integration failures, *and* write reproducible tests for the bugs they expose! Anyhow, I've been doing this in Impala, and it's been very helpful. Haven't seen much evidence elsewhere.'

    (tags: henry-robinson testing fault-discovery rpc dtrace tracing distributed-systems timeouts chaos-monkey impala)

  • The Best Bourbon Cocktail You’ve Never Heard Of

    The "Paper Plane", by Sam Ross of Chicago's "Violet Hour": .75 oz Bourbon .75 oz Aperol .75 oz Amaro Nonino .75 oz Fresh lemon juice ice-filled shaker, shake, strain.

    (tags: bourbon drinks cocktails recipes aperol amaro-nonino lemon)

  • Seastar

    C++ high-performance app framework; 'currently focused on high-throughput, low-latency I/O intensive applications.' Scylla (Cassandra-compatible NoSQL store) is written in this.

    (tags: c++ opensource performance framework scylla seastar latency linux shared-nothing multicore)

Links for 2015-09-23

Published September 23, 2015

Links for 2015-09-22

Published September 22, 2015

  • Brotli: a new compression algorithm for the internet from Google

    While Zopfli is Deflate-compatible, Brotli is a whole new data format. This new format allows us to get 20–26% higher compression ratios over Zopfli. In our study ‘Comparison of Brotli, Deflate, Zopfli, LZMA, LZHAM and Bzip2 Compression Algorithms’ we show that Brotli is roughly as fast as zlib’s Deflate implementation. At the same time, it compresses slightly more densely than LZMA and bzip2 on the Canterbury corpus. The higher data density is achieved by a 2nd order context modeling, re-use of entropy codes, larger memory window of past data and joint distribution codes. Just like Zopfli, the new algorithm is named after Swiss bakery products. Brötli means ‘small bread’ in Swiss German.

    (tags: brotli zopfli deflate gzip compression algorithms swiss google)

Links for 2015-09-21

Published September 21, 2015

  • Nelson recommends Ubiquiti

    'The key thing about Ubiquiti gear is the high quality radios and antennas. It just seems much more reliable than most consumer WiFi gear. Their airOS firmware is good too, it’s a bit complicated to set up but very capable and flexible. And in addition to normal 802.11n or 802.11ac they also have an optional proprietary TDMA protocol called airMax that’s designed for serving several long haul links from a single basestation. They’re mostly marketing to business customers but the equipment is sold retail and well documented for ordinary nerds to figure out.'

    (tags: ubiquiti wifi wireless 802.11 via:nelson ethernet networking prosumer hardware wan)

  • httpry

    a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.
    via Eoin Brazil

    (tags: via:eoinbrazil httpry http networking tools ops testing tcpdump tracing)

  • ustwo Reimagines the In-Car Cluster

    Designers behind the cult mobile game, Monument Valley, take on the legacy-bound in-car UI

    (tags: ux ui cars driving safety ustwo monument-valley speed)

  • Little Drummer Boy Challenge

    'It's very easy: So long as you don't hear "The Little Drummer Boy," you're a contender. As soon as you hear it on the radio, on TV, in a store, wherever, you're out.'

    (tags: ldbc games funny xmas christmas music songs cheese)

Links for 2015-09-18

Published September 18, 2015

Links for 2015-09-14

Published September 14, 2015

  • Kate Heddleston: How Our Engineering Environments Are Killing Diversity

    '[There are] several problem areas for [diversity in] engineering environments and ways to start fixing them. The problems we face aren't devoid of solutions; there are a lot of things that companies, teams, and individuals can do to fix problems in their work environment. For the month of March, I will be posting detailed articles about the problem areas I will cover in my talk: argument cultures, feedback, promotions, employee on-boarding, benefits, safety, engineering process, and environment adaptation.' via Baron Schwartz.

    (tags: via:xaprb culture tech diversity sexism feminism engineering work workplaces feedback)

  • Michael Kagan | Prints

    'Heavily tinted blue paintings form space stations, spacesuits, and rockets just after blast. Michael Kagan paints these large-scale works to celebrate the man-made object—machinery that both protects and holds the possibility of instantly killing those that operate the equipment from the inside. To paint the large works, Kagan utilizes an impasto technique with thick strokes that are deliberate and unique, showing an aggression in his application of oil paint on linen. The New York-based artist focuses on iconic images in his practice, switching back and forth between abstract and representational styles. “The painting is finished when it can fall apart and come back together depending on how it is read and the closeness to the work,” said Kagan about his work. “Each painting is an image, a snapshot, a flash moment, a quick read that is locked into memory by the iconic silhouettes.”' Via http://www.thisiscolossal.com/2015/08/michael-kagens-space-paintings/

    (tags: paintings prints art michael-kagan space abstract-art tobuy)

  • Dark corners of Unicode

    I’m assuming, if you are on the Internet and reading kind of a nerdy blog, that you know what Unicode is. At the very least, you have a very general understanding of it — maybe “it’s what gives us emoji”. That’s about as far as most people’s understanding extends, in my experience, even among programmers. And that’s a tragedy, because Unicode has a lot of… ah, depth to it. Not to say that Unicode is a terrible disaster — more that human language is a terrible disaster, and anything with the lofty goals of representing all of it is going to have some wrinkles. So here is a collection of curiosities I’ve encountered in dealing with Unicode that you generally only find out about through experience. Enjoy.

    (tags: unicode characters encoding emoji utf-8 utf-16 utf mysql text)

Links for 2015-09-08

Published September 8, 2015

Links for 2015-09-07

Published September 7, 2015

  • You're probably wrong about caching

    Excellent cut-out-and-keep guide to why you should add a caching layer. I've been following this practice for the past few years, after I realised that #6 (recovering from a failed cache is hard) is a killer -- I've seen a few large-scale outages where a production system had gained enough scale that it required a cache to operate, and once that cache was damaged, bringing the system back online required a painful rewarming protocol. Better to design for the non-cached case if possible.

    (tags: architecture caching coding design caches ops production scalability)

  • The Alternative Universe Of Soviet Arcade Games

    Unlike machines in the West, every single machine that was produced during Soviet-era Russia had to align with Marxist ideology. [...] The most popular games were created to teach hand-eye coordination, reaction speed, and logical, focused thinking. Not unlike many American games, these games were influenced by military training, crafted to teach and instill patriotism for the state by making the human body better, stronger, and more willful. It also means no high scores, no adrenaline rushes, or self-serving feather-fluffing as you add your hard-earned initials to the list of the best. In Communist Russia, there was no overt competition.

    (tags: high-scores communism russia cccp ussr arcade-games games history)

Links for 2015-09-03

Published September 3, 2015

Links for 2015-09-02

Published September 2, 2015

Links for 2015-09-01

Published September 1, 2015

  • What Are the Worst Airports in the World?

    this is a great resource when picking a stopover for a 2-stop flight. Pity "best kids play area" isn't a criterion

    (tags: airports comparison via:boingboing flying travel ranking world skytrax)

  • Using Samsung's Internet-Enabled Refrigerator for Man-in-the-Middle Attacks

    Whilst the fridge implements SSL, it FAILS to validate SSL certificates, thereby enabling man-in-the-middle attacks against most connections. This includes those made to Google's servers to download Gmail calendar information for the on-screen display. So, MITM the victim's fridge from next door, or on the road outside and you can potentially steal their Google credentials.
    The Internet of Insecure Things strikes again.

    (tags: iot security fridges samsung fail mitm ssl tls google papers defcon)

  • Malware infecting jailbroken iPhones stole 225,000 Apple account logins | Ars Technica

    KeyRaider, as the malware family has been dubbed, is distributed through a third-party repository of Cydia, which markets itself as an alternative to Apple's official App Store. Malicious code surreptitiously included with Cydia apps is creating problems for people in China and at least 17 other countries, including France, Russia, Japan, and the UK. Not only has it pilfered account data for 225,941 Apple accounts, it has also disabled some infected phones until users pay a ransom, and it has made unauthorized charges against some victims' accounts.
    Ouch. Not a good sign for Cydia

    (tags: cydia apple security exploits jailbreaking ios iphone malware keyraider china)

  • GoTTY

    'a simple command line tool that turns your CLI tools into web applications'

    (tags: cli terminal web tools unix)

  • S3QL

    a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X. S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival. S3QL is designed to favor simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection and error handling have been included from the very first line, and S3QL comes with extensive automated test cases for all its components.

    (tags: filesystems aws s3 storage unix google-storage openstack)

Links for 2015-08-31

Published August 31, 2015

Links for 2015-08-28

Published August 28, 2015

  • toxy

    toxy is a fully programmatic and hackable HTTP proxy to simulate server failure scenarios and unexpected network conditions. It was mainly designed for fuzzing/evil testing purposes, when toxy becomes particularly useful to cover fault tolerance and resiliency capabilities of a system, especially in service-oriented architectures, where toxy may act as intermediate proxy among services. toxy allows you to plug in poisons, optionally filtered by rules, which essentially can intercept and alter the HTTP flow as you need, performing multiple evil actions in the middle of that process, such as limiting the bandwidth, delaying TCP packets, injecting network jitter latency or replying with a custom error or status code.

    (tags: toxy proxies proxy http mitm node.js soa network failures latency slowdown jitter bandwidth tcp)

  • Drone Oversight Is Coming to Construction Sites

    Grim Meathook Future

    (tags: grim-meathook-future drones work panopticon future sacramento building-sites)

  • grsecurity

    Open source security team has had enough of embedded-systems vendors taking the piss with licensing:

    This announcement is our public statement that we've had enough. Companies in the embedded industry not playing by the same rules as every other company using our software violates users' rights, misleads users and developers, and harms our ability to continue our work. Though I've only gone into depth in this announcement on the latest trademark violation against us, our experience with two GPL violations over the previous year have caused an incredible amount of frustration. These concerns are echoed by the complaints of many others about the treatment of the GPL by the embedded Linux industry in particular over many years. With that in mind, today's announcement is concerned with the future availability of our stable series of patches. We decided that it is unfair to our sponsors that the above mentioned unlawful players can get away with their activity. Therefore, two weeks from now, we will cease the public dissemination of the stable series and will make it available to sponsors only. The test series, unfit in our view for production use, will however continue to be available to the public to avoid impact to the Gentoo Hardened and Arch Linux communities. If this does not resolve the issue, despite strong indications that it will have a large impact, we may need to resort to a policy similar to Red Hat's, described here or eventually stop the stable series entirely as it will be an unsustainable development model.

    (tags: culture gpl linux opensource security grsecurity via:nelson gentoo arch-linux gnu)

  • London Calling: Two-Factor Authentication Phishing From Iran

    some rather rudimentary anti-2FA attempts, presumably from Iranian security services

    (tags: authentication phishing security iran activism 2fa mfa)

  • Vegemite May Power The Electronics Of The Future

    Professor Marc in het Panhuis at the ARC Centre of Excellence for Electromaterials Science figured out that you can 3D print the paste and use it to carry current, effectively creating Vegemite bio-wires. What does this mean? Soon you can run electricity through your food. “The iconic Australian Vegemite is ideal for 3D printing edible electronics,” said the professor. “It contains water so it’s not a solid and can easily be extruded using a 3D printer. Also, it’s salty, so it conducts electricity.”
    I'm sure the same applies for Marmite...

    (tags: vegemite marmite 3d-printing electronics bread food silly)

  • Beoir.org Community - Recent Attack on McGargles

    bizarre conspiracy theory going around about McGargles microbrewery being owned by Molson in an "astroturf craft beer" operation -- they apparently were set up by a bunch of ex-Molson employees. Their beer is getting stickered in off-licenses. Mental!

    (tags: beer craft-beer ireland mcgargles conspiracy-theories bizarre beoir)

Links for 2015-08-27

Published August 27, 2015

  • Mining High-Speed Data Streams: The Hoeffding Tree Algorithm

    This paper proposes a decision tree learner for data streams, the Hoeffding Tree algorithm, which comes with the guarantee that the learned decision tree is asymptotically nearly identical to that of a non-incremental learner using infinitely many examples. This work constitutes a significant step in developing methodology suitable for modern ‘big data’ challenges and has initiated a lot of follow-up research. The Hoeffding Tree algorithm has been covered in various textbooks and is available in several public domain tools, including the WEKA Data Mining platform.

    (tags: hoeffding-tree algorithms data-structures streaming streams cep decision-trees ml learning papers)

  • Chinese scammers are now using Stingray tech to SMS-phish

    A Stingray-style false GSM base station, hidden in a backpack; presumably they detect numbers in the vicinity, and SMS-spam those numbers with phishing messages. Reportedly the scammers used this trick in "Guangzhou, Zhuhai, Shenzhen, Changsha, Wuhan, Zhengzhou and other densely populated cities". Dodgy machine translation:

    March 26, Zhengzhou police telecommunications fraud cases together, for the first time seized a small backpack can hide pseudo station equipment, and arrested two suspects. Yesterday, the police informed of this case, to remind the general public to pay attention to prevention. “I am the landlord, I changed number, please rent my wife hit the bank card, card number ×××, username ××.” Recently, Jiefang Road, Zhengzhou City Public Security Bureau police station received a number of cases for investigation brigade area of ??the masses police said, frequently received similar phone scam messages. Alarm, the police investigators to determine: the suspect may be in the vicinity of twenty-seven square, large-scale use of mobile pseudo-base release fraudulent information. [...] Yesterday afternoon, the Jiefang Road police station, the reporter saw the portable pseudo-base is made up of two batteries, a set-top box the size of the antenna box and a chassis, as well as a pocket computer composed together at most 5 kg.
    (via t byfield and Danny O'Brien)

    (tags: via:mala via:tbyfield privacy scams phishing sms gsm stingray base-stations mobile china)

Links for 2015-08-25

Published August 25, 2015

Links for 2015-08-23

Published August 23, 2015

Links for 2015-08-22

Published August 22, 2015

Links for 2015-08-19

Published August 19, 2015

Links for 2015-08-18

Published August 18, 2015

Links for 2015-08-17

Published August 17, 2015

  • The world beyond batch: Streaming 101 - O'Reilly Media

    To summarize, in this post I’ve: Clarified terminology, specifically narrowing the definition of “streaming” to apply to execution engines only, while using more descriptive terms like unbounded data and approximate/speculative results for distinct concepts often categorized under the “streaming” umbrella. Assessed the relative capabilities of well-designed batch and streaming systems, positing that streaming is in fact a strict superset of batch, and that notions like the Lambda Architecture, which are predicated on streaming being inferior to batch, are destined for retirement as streaming systems mature. Proposed two high-level concepts necessary for streaming systems to both catch up to and ultimately surpass batch, those being correctness and tools for reasoning about time, respectively. Established the important differences between event time and processing time, characterized the difficulties those differences impose when analyzing data in the context of when they occurred, and proposed a shift in approach away from notions of completeness and toward simply adapting to changes in data over time. Looked at the major data processing approaches in common use today for bounded and unbounded data, via both batch and streaming engines, roughly categorizing the unbounded approaches into: time-agnostic, approximation, windowing by processing time, and windowing by event time.

    (tags: streaming batch big-data lambda-architecture dataflow event-processing cep millwheel data data-processing)

  • What the hell is going on with SoundCloud?

    tl;dr: major labels.

    Despite having revenue coming in from ads and subscriptions, SoundCloud still relies on outside investment. While the company received $150 million in a funding round at the end of last year, it pales next to the reported $526 million Spotify gained in June, and if one report is to be believed, SoundCloud is running very low on cash. Furthermore, sources suggest that potential investors are waiting to see what happens with Sony and Universal before ploughing in more money. With the high sums reported to be involved, it’s a stalemate that could potentially break the company whether it decides to pay or not.

    (tags: soundcloud music mp3 copyright sony universal spotify funding startups)

  • GSMem: Data Exfiltration from Air-Gapped Computers over GSM Frequencies

    Holy shit.

    Air-gapped networks are isolated, separated both logically and physically from public networks. Although the feasibility of invading such systems has been demonstrated in recent years, exfiltration of data from air-gapped networks is still a challenging task. In this paper we present GSMem, a malware that can exfiltrate data through an air-gap over cellular frequencies. Rogue software on an infected target computer modulates and transmits electromagnetic signals at cellular frequencies by invoking specific memory-related instructions and utilizing the multichannel memory architecture to amplify the transmission. Furthermore, we show that the transmitted signals can be received and demodulated by a rootkit placed in the baseband firmware of a nearby cellular phone.

    (tags: gsmem gsm exfiltration air-gaps memory radio mobile-phones security papers)

Links for 2015-08-16

Published August 16, 2015

Links for 2015-08-12

Published August 12, 2015

Links for 2015-08-11

Published August 11, 2015

  • Reddit comments from a nuclear-power expert

    Reddit user "Hiddencamper" is a senior nuclear reactor operator in the US, and regularly posts very knowledgeable comments about reactor operations, safety procedures, and other details. It's fascinating (via Maciej)

    (tags: via:maciej nuclear-power nuclear atomic power energy safety procedures operations history chernobyl scram)

  • Amazon EC2 2015 Benchmark: Testing Speeds Between AWS EC2 and S3 Regions

    Here we are again, a year later, and still no bloody percentiles! Just amateurish averaging. This is not how you measure anything, ffs. Still, better than nothing I suppose

    (tags: fail latency measurement aws ec2 percentiles s3)

  • background doc on the Jeep hack

    "Remote Exploitation of an Unaltered Passenger Vehicle", by Dr. Charlie Miller (cmiller@openrce.org) and Chris Valasek (cvalasek@gmail.com). QNX, unauthenticated D-Bus, etc.

    'Since a vehicle can scan for other vulnerable vehicles and the exploit doesn’t require any user interaction, it would be possible to write a worm. This worm would scan for vulnerable vehicles, exploit them with their payload which would scan for other vulnerable vehicles, etc. This is really interesting and scary. Please don’t do this. Please.'

    (tags: jeep hacks exploits d-bus qnx cars safety risks)

  • Care.data and access to UK health records: patient privacy and public trust

    'In 2013, the United Kingdom launched care.data, an NHS England initiative to combine patient records, stored in the machines of general practitioners (GPs), with information from social services and hospitals to make one centralized data archive. One aim of the initiative is to gain a picture of the care being delivered between different parts of the healthcare system and thus identify what is working in health care delivery, and what areas need greater attention and resources. This case study analyzes the complications around the launch of care.data. It explains the historical context of the program and the controversies that emerged in the course of the rollout. It explores problems in management and communications around the centralization effort, competing views on the safety of “anonymous” and “pseudonymous” health data, and the conflicting legal duties imposed on GPs with the introduction of the 2012 Health and Social Care Act. This paper also explores the power struggles in the battle over care.data and outlines the tensions among various stakeholders, including patients, GPs, the Health and Social Care Information Centre (HSCIC), the government, privacy experts and data purchasers. The predominant public policy question that emerges from this review centers on how best to utilize technological advances and simultaneously strike a balance between the many competing interests around health and personal privacy.'

    (tags: care.data privacy healthcare uk nhs trust anonymity anonymization gps medicine)

Links for 2015-08-10

Published August 10, 2015

Links for 2015-08-07

Published August 7, 2015

Links for 2015-08-06

Published August 6, 2015

Links for 2015-08-05

Published August 5, 2015

Links for 2015-08-03

Published August 3, 2015

Links for 2015-07-30

Published July 30, 2015

  • danilop/yas3fs · GitHub

    YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.

    (tags: aws s3 s3fs yas3fs filesystems fuse sns)

  • danilop/runjop · GitHub

    RunJOP (Run Just Once Please) is a distributed execution framework to run a command (i.e. a job) only once in a group of servers [built using AWS DynamoDB and S3].
    nifty! Distributed cron is pretty easy when you've got Dynamo doing the heavy lifting.

    (tags: dynamodb cron distributed-cron scheduling runjop danilop hacks aws ops)

Links for 2015-07-29

Published July 29, 2015

Links for 2015-07-28

Published July 28, 2015

  • Taming Complexity with Reversibility

    This is a great post from Kent Beck, putting a lot of recent deployment/rollout patterns in a clear context -- that of supporting "reversibility":

    Development servers. Each engineer has their own copy of the entire site. Engineers can make a change, see the consequences, and reverse the change in seconds without affecting anyone else. Code review. Engineers can propose a change, get feedback, and improve or abandon it in minutes or hours, all before affecting any people using Facebook. Internal usage. Engineers can make a change, get feedback from thousands of employees using the change, and roll it back in an hour. Staged rollout. We can begin deploying a change to a billion people and, if the metrics tank, take it back before problems affect most people using Facebook. Dynamic configuration. If an engineer has planned for it in the code, we can turn off an offending feature in production in seconds. Alternatively, we can dial features up and down in tiny increments (i.e. only 0.1% of people see the feature) to discover and avoid non-linear effects. Correlation. Our correlation tools let us easily see the unexpected consequences of features so we know to turn them off even when those consequences aren't obvious. IRC. We can roll out features potentially affecting our ability to communicate internally via Facebook because we have uncorrelated communication channels like IRC and phones. Right hand side units. We can add a little bit of functionality to the website and turn it on and off in seconds, all without interfering with people's primary interaction with NewsFeed. Shadow production. We can experiment with new services under real load, from a tiny trickle to the whole flood, without affecting production. Frequent pushes. Reversing some changes require a code change. On the website we never more than eight hours from the next schedule code push (minutes if a fix is urgent and you are willing to compensate Release Engineering). The time frame for code reversibility on the mobile applications is longer, but the downward trend is clear from six weeks to four to (currently) two. Data-informed decisions. (Thanks to Dave Cleal) Data-informed decisions are inherently reversible (with the exceptions noted below). "We expect this feature to affect this metric. If it doesn't, it's gone." Advance countries. We can roll a feature out to a whole country, generate accurate feedback, and roll it back without affecting most of the people using Facebook. Soft launches. When we roll out a feature or application with a minimum of fanfare it can be pulled back with a minimum of public attention. Double write/bulk migrate/double read. Even as fundamental a decision as storage format is reversible if we follow this format: start writing all new data to the new data store, migrate all the old data, then start reading from the new data store in parallel with the old.
    We do a bunch of these in work, and the rest are on the to-do list. +1 to these!

    (tags: software deployment complexity systems facebook reversibility dark-releases releases ops cd migration)

Links for 2015-07-27

Published July 27, 2015

  • Benchmarking GitHub Enterprise - GitHub Engineering

    Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)

    (tags: github listen-backlog tcp debugging timeouts load-testing benchmarking testing ops linux)

  • How .uk came to be (and why it's not .gb)

    WB: By the late 80s the IANA [the Internet Assigned Numbers Authority, set up in 1988 to manage global IP address allocations] was trying to get all those countries that were trying to join the internet to use the ISO 3166 standard for country codes. It was used for all sorts of things?—?you see it on cars, “GB” for the UK. [...] At that point, we’re faced with a problem that Jon Postel would like to have changed it to .gb to be consistent with the rest of the world. Whereas .uk had already been established, with a few tens of thousands of domain names with .uk on them. I remember chairing one of the JANET net workshops that were held every year, and the Northern Irish were adamant that they were part of the UK?—?so the consensus was, we’d try and keep .uk, we’d park .gb and not use it. PK: I didn’t particularly want to change to .gb because I was responsible for Northern Ireland as well. And what’s more, there was a certain question as to whether a research group in the US should be allowed to tell the British what to do. So this argy-bargy continued for a little while and, in the meantime, one of my clients was the Ministry of Defence, and they decided they couldn’t wait this long, and they decided I was going to lose the battle, and so bits of MOD went over to .gb?—?I didn’t care, as I was running .gb and .uk in any case.

    (tags: dot-uk history internet dot-gb britain uk northern-ireland ireland janet)

  • That time the Internet sent a SWAT team to my mom's house - Boing Boing

    The solution is for social media sites and the police to take threats or jokes about swatting, doxxing, and organized crime seriously. Tweeting about buying a gun and shooting up a school would be taken seriously, and so should the threat of raping, doxxing, swatting or killing someone. Privacy issues and online harassment are directly linked, and online harassment isn’t going anywhere. My fear is that, in reaction to online harassment, laws will be passed that will break down our civil freedoms and rights online, and that more surveillance will be sold to users under the guise of safety. More surveillance, however, would not have helped me or my mother. A platform that takes harassment and threats seriously instead of treating them like jokes would have.

    (tags: twitter gamergate 4chan 8chan privacy doxxing swatting harrassment threats social-media facebook law feminism)

  • Why Google's Deep Dream Is Future Kitsch

    Deep Dream estranges us from our fears, perhaps, but it doesn't make them go away. It's easy to discuss Deep Dream as an independent creature, a foreign intelligence that we interact with for fun. Yet like all kitsch, it comes straight back to its creators.

    (tags: kitsch deep-dream art graphics google inceptionism)

  • It’s Not Climate Change?—?It’s Everything Change

    now this is a Long Read. the inimitable Margaret Atwood on climate change, beautifully illustrated

    (tags: climate climate-change margaret-atwood long-reads change life earth green future)

  • In Praise of the AK-47 — Dear Design Student — Medium

    While someone can certainly make the case that an AK-47, or any other kind of gun or rifle is designed, nothing whose primary purpose is to take away life can be said to be designed well. And that attempting to separate an object from its function in order to appreciate it for purely aesthetic reasons, or to be impressed by its minimal elegance, is a coward’s way of justifying the death they’ve designed into the word, and the money with which they’re lining their pockets.

    (tags: design ux ak-47 kalashnikov guns function work)

Links for 2015-07-22

Published July 22, 2015

Links for 2015-07-21

Published July 21, 2015

  • Java lambdas and performance

    Lambdas in Java 8 introduce some unpredictable performance implications, due to reliance on escape analysis to eliminate object allocation on every lambda invocation. Peter Lawrey has some details

    (tags: lambdas java-8 java performance low-latency optimization peter-lawrey coding escape-analysis)

  • Mikhail Panchenko's thoughts on the July 2015 CircleCI outage

    an excellent followup operational post on CircleCI's "database is not a queue" outage

    (tags: database-is-not-a-queue mysql sql databases ops outages postmortems)

  • Men who harass women online are quite literally losers, new study finds

    (1) players are anonymous, and the possibility of “policing individual behavior is almost impossible”; (2) they only encounter each other a few times in passing — it’s very possible to hurl an expletive at another player, and never “see” him or her again; and (3) finally, and perhaps predictably, the sex-ratio of players is biased pretty heavily toward men. (A 2014 survey of gender ratios on Reddit found that r/halo was over 95 percent male.) [....] In each of these environments, Kasumovic suggests, a recent influx of female participants has disrupted a pre-existing social hierarchy. That’s okay for the guys at the top — but for the guys at the bottom, who stand to lose more status, that’s very threatening. (It’s also in keeping with the evolutionary framework on anti-lady hostility, which suggests sexism is a kind of Neanderthal defense mechanism for low-status, non-dominant men trying to maintain a shaky grip on their particular cave’s supply of women.) “As men often rely on aggression to maintain their dominant social status,” Kasumovic writes, “the increase in hostility towards a woman by lower-status males may be an attempt to disregard a female’s performance and suppress her disturbance on the hierarchy to retain their social rank.”

    (tags: losers sexism mysogyny women halo gaming gamergate 4chan abuse harrassment papers bullying social-status)

  • The old suburban office park is the new American ghost town - The Washington Post

    Most analyses of the market indicate that office parks simply aren’t as appealing or profitable as they were in the 20th century and that Americans just aren’t as keen to cloister themselves in workspaces that are reachable only by car.

    (tags: cbd cities work life office-parks commuting america history workplaces)

  • HACKERS REMOTELY KILL A JEEP ON THE HIGHWAY—WITH ME IN IT

    Jaysus, this is terrifying.

    Miller and Valasek’s full arsenal includes functions that at lower speeds fully kill the engine, abruptly engage the brakes, or disable them altogether. The most disturbing maneuver came when they cut the Jeep’s brakes, leaving me frantically pumping the pedal as the 2-ton SUV slid uncontrollably into a ditch.
    Avoid any car which supports this staggeringly-badly-conceived Uconnect feature:
    All of this is possible only because Chrysler, like practically all carmakers, is doing its best to turn the modern automobile into a smartphone. Uconnect, an Internet-connected computer feature in hundreds of thousands of Fiat Chrysler cars, SUVs, and trucks, controls the vehicle’s entertainment and navigation, enables phone calls, and even offers a Wi-Fi hot spot.
    :facepalm: Also, Chrysler's response sucks: "Chrysler’s patch must be manually implemented via a USB stick or by a dealership mechanic."

    (tags: hacking security cars driving safety brakes jeeps chrysler fiat uconnect can-bus can)

Links for 2015-07-20

Published July 20, 2015

Links for 2015-07-17

Published July 17, 2015

  • Angela Merkel told a sobbing girl she couldn't save her from deportation. It was a lie. - Vox

    Argentina has, as a matter of constitutional law, effectively open borders. There are no caps or quotas or lottery systems. You can move there legally if you have an employer or family member to sponsor you. That's all you need. If you don't have a sponsor, and make your way in illegally, you're recognized as an "irregular migrant." Discrimination against irregular migrants in health care or education is illegal, and deportation in noncriminal cases is exceptionally rare. Large-scale amnesties are the norm. Obviously Argentina is not nearly as rich as Germany or the US or the UK. But it's considerably richer than three of its neighbors (Bolivia, Paraguay, and Brazil). And yet it doesn't try hard to keep their residents out. It welcomes them — as it should. "One could have expected catastrophe—an uncontrollable flow of poorer immigrants streaming into the country coupled with angry public backlash," Elizabeth Slater writes in the World Policy Journal. "That hasn't happened." Angela Merkel clearly expects catastrophe if she lets people like this weeping young Palestinian girl stay in Germany. That catastrophe is simply a myth; it wouldn't happen. What would happen is that Germany's economy would grow, its culture would grow richer, and that girl and more like her could see their lives improve immeasurably.

    (tags: argentina immigration angela-merkel germany eu migrants deportation economics)

Links for 2015-07-16

Published July 16, 2015

Links for 2015-07-15

Published July 15, 2015

Links for 2015-07-14

Published July 14, 2015

Links for 2015-07-13

Published July 13, 2015

  • OkHttp

    A new HTTP client library for Android and Java, with a lot of nice features:

    HTTP/2 and SPDY support allows all requests to the same host to share a socket. Connection pooling reduces request latency (if SPDY isn’t available). Transparent GZIP shrinks download sizes. Response caching avoids the network completely for repeat requests. OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails. Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.

    (tags: android http java libraries okhttp http2 spdy microservices jdk)

  • Eircode tech specs

    via Ossian.

    (tags: via:smytho tech-specs specs eircode addresses geocoding ireland mapping)

  • AWS Best Practices for DDoS Resiliency [pdf]

    Reasonably solid white paper

    (tags: ddos amazon aws security dos whitepapers pdf)

Links for 2015-07-11

Published July 11, 2015

Links for 2015-06-25

Published June 25, 2015

Links for 2015-06-23

Published June 23, 2015

Links for 2015-06-22

Published June 22, 2015

Links for 2015-06-21

Published June 21, 2015

  • jwz on Inceptionism

    "Shoggoth ovipositors":

    So then they reach inside to one of the layers and spin the knob randomly to fuck it up. Lower layers are edges and curves. Higher layers are faces, eyes and shoggoth ovipositors. [....] But the best part is not when they just glitch an image -- which is a fun kind of embossing at one end, and the "extra eyes" filter at the other -- but is when they take a net trained on some particular set of objects and feed it static, then zoom in, and feed the output back in repeatedly. That's when you converge upon the platonic ideal of those objects, which -- it turns out -- tend to be Giger nightmare landscapes. Who knew. (I knew.)
    This stuff is still boggling my mind. All those doggy faces! That is one dog-obsessed ANN.

    (tags: neural-networks ai jwz funny shoggoths image-recognition hr-giger art inceptionism)

Links for 2015-06-19

Published June 19, 2015

Links for 2015-06-18

Published June 18, 2015

  • Inceptionism: Going Deeper into Neural Networks

    This is amazing, and a little scary.

    If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge. Again, we just start with an existing image and give it to our neural net. We ask the network: “Whatever you see there, I want more of it!” This creates a feedback loop: if a cloud looks a little bit like a bird, the network will make it look more like a bird. This in turn will make the network recognize the bird even more strongly on the next pass and so forth, until a highly detailed bird appears, seemingly out of nowhere.
    An enlightening comment from the G+ thread:
    This is the most fun we've had in the office in a while. We've even made some of those 'Inceptionistic' art pieces into giant posters. Beyond the eye candy, there is actually something deeply interesting in this line of work: neural networks have a bad reputation for being strange black boxes that that are opaque to inspection. I have never understood those charges: any other model (GMM, SVM, Random Forests) of any sufficient complexity for a real task is completely opaque for very fundamental reasons: their non-linear structure makes it hard to project back the function they represent into their input space and make sense of it. Not so with backprop, as this blog post shows eloquently: you can query the model and ask what it believes it is seeing or 'wants' to see simply by following gradients. This 'guided hallucination' technique is very powerful and the gorgeous visualizations it generates are very evocative of what's really going on in the network.?

    (tags: art machine-learning algorithm inceptionism research google neural-networks learning dreams feedback graphics)

Links for 2015-06-17

Published June 17, 2015