Skip to content

Category: Uncategorized

Links for 2014-11-27

Published November 27, 2014

  • Consul case study from Hootsuite

    Hootsuite used Consul for distributed configuration, specifically dark-launch feature flags, with great results: 'Trying out bleeding edge software can be a risky proposition, but in the case of Consul, we’ve found it to be a solid system that works basically as described and was easy to get up and running. We managed to go from initial investigations to production within a month. The value was immediately obvious after looking into the key-value store combined with the events system and it’s DNS features and each of these has worked how we expected. Overall it has been fun to work with and has worked well and based on the initial work we have done with the Dark Launching system we’re feeling confident in Consul’s operation and are looking forward to expanding the scope of it’s use.'

    (tags: consul dark-launches feature-flags configuration distributed hootsuite notification)

Links for 2014-11-25

Published November 25, 2014

Links for 2014-11-24

Published November 24, 2014

Links for 2014-11-23

Published November 23, 2014

Links for 2014-11-21

Published November 21, 2014

  • Goat Simulator's fake MMO

    This is classic. I love the "Rouge":

    We also wanted the Rouge to actually look like a stealth-oriented make-up artist, but our 3D artist thought the goat looked ridiculous with a pink wig and a Gucci bag, so we remade the Rouge to actually look like a Rogue.

    (tags: rogue rouge goats goat-simulator funny satire mmos mmorpg games warcraft)

  • The boss has malware, again... : talesfromtechsupport

    Finally after all traditional means of infection were covered; IT started looking into other possibilities. They finally asked the Executive, “Have there been any changes in your life recently”? The executive answer “Well yes, I quit smoking two weeks ago and switched to e-cigarettes”. And that was the answer they were looking for, the made in china e-cigarette had malware hard coded into the charger and when plugged into a computer’s USB port the malware phoned home and infected the system. Moral of the story is have you ever question the legitimacy of the $5 dollar EBay made in China USB item that you just plugged into your computer? Because you should, you damn well should.
    (Via Elliot)

    (tags: via:elliot malware e-cigarettes cigarettes smoking china risks)

Links for 2014-11-20

Published November 20, 2014

  • Update on Azure Storage Service Interruption

    As part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this “flighting,” as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues.
    I'm really surprised MS deployment procedures allow a change to be rolled out globally across multiple regions on a single day. I suspect they soon won't.

    (tags: change-management cm microsoft outages postmortems azure deployment multi-region flighting azure-storage)

  • AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems - YouTube

    This is a really solid talk -- not surprising, alv@ is one of the speakers!

    "AWS and Amazon.com operate some of the world's largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives -- DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS. "
    Slides: http://www.slideshare.net/AmazonWebServices/spot302-under-the-covers-of-aws-core-distributed-systems-primitives-that-power-our-platform-aws-reinvent-2014

    (tags: scale scaling aws amazon dht logging data-structures distcomp via:marc-brooker dynamodb s3)

  • How Curiosity, Luck, and the Flip of a Switch Saved the Moon Program | Motherboard

    "SCE to off?" someone said. The switch was so obscure that neither of his bosses knew what he was talking about. "What the hell's that," blurted out Gerald Carr, who was in charge of communicating with the capsule. The rookie flight director, Gerry Griffin, didn't know either. Sixty seconds had passed since the initial lightning strike. No one else knew what to do. The call to abort was fast approaching.  Finally, Carr reluctantly gave the order in a voice far cooler than the moment. "Apollo 12, Houston, try SCE to Auxiliary, over."

    (tags: spaceflight stories apollo sce-to-aux power lightning weather outages simulation training nasa)

Links for 2014-11-19

Published November 19, 2014

  • Building a complete Tweet index

    Twitter's new massive-scale twitter search backend. Sharding galore

    (tags: architecture search twitter sharding earlybird)

  • The Oral History Of The Poop Emoji (Or, How Google Brought Poop To America)

    'I went over to Japan right around the time Takeshi was deciding which emoji were going to make it into the first cut of Gmail emoji. The [PILE_OF_POO emoji] was absolutely one of the necessary emoji that Takeshi said we have to have. There was actually conflict because there were people back at headquarters who had no idea what emoji were, and thought that having an animated [turd] in their Gmail was offensive.' '[The poop emoji] got very popular when a comic called "Dr. Slump" was broadcast in Japan back to the ‘90s. Such poop was not an object to be disliked, but it had a funny meaning. This was a very popular comedy animation where a girl played a trick on other people using the poop. The poop was this funny object to play with. It was never serious.' 'In Japanese that’s called “unchi.” It’s a child word with a benign meaning. '

    (tags: culture emoji google pile-of-poo turd poo japan gmail unchi dr-slump)

  • LUNAR MISSION ONE: A new lunar mission for everyone. by Lunar Missions Ltd — Kickstarter

    We plan to send an unmanned robotic landing module to the South Pole of the Moon – an area unexplored by previous missions. We’re going to use pioneering technology to drill down to a depth of at least 20m – 10 times deeper than has ever been drilled before – and potentially as deep as 100m. By doing this, we will access lunar rock dating back up to 4.5 billion years to discover the geological composition of the Moon, the ancient relationship it shares with our planet and the effects of asteroid bombardment. Ultimately, the project will improve scientific understanding of the early solar system, the formation of our planet and the Moon, and the conditions that initiated life on Earth.
    Kickstarter-funded -- UKP 600k goal. Just in time for xmas!

    (tags: kickstarter science moon lunar-mission-one exploration)

  • Flow, a new static type checker for JavaScript

    Unlike the (excellent) Typescript, it'll infer types:

    Flow’s type checking is opt-in — you do not need to type check all your code at once. However, underlying the design of Flow is the assumption that most JavaScript code is implicitly statically typed; even though types may not appear anywhere in the code, they are in the developer’s mind as a way to reason about the correctness of the code. Flow infers those types automatically wherever possible, which means that it can find type errors without needing any changes to the code at all. On the other hand, some JavaScript code, especially frameworks, make heavy use of reflection that is often hard to reason about statically. For such inherently dynamic code, type checking would be too imprecise, so Flow provides a simple way to explicitly trust such code and move on. This design is validated by our huge JavaScript codebase at Facebook: Most of our code falls in the implicitly statically typed category, where developers can check their code for type errors without having to explicitly annotate that code with types.

    (tags: facebook flow javascript coding types type-inference ocaml typescript)

  • Exactly-Once Delivery May Not Be What You Want

    An extremely good explanation from Marc Brooker that exactly-once delivery in a distributed system is very hard.

    And so on. There's always a place to slot in one more turtle. The bad news is that I'm not aware of a nice solution to the general problem for all side effects, and I suspect that no such solution exists. On the bright side, there are some very nice solutions that work really well in practice. The simplest is idempotence. This is a very simple idea: we make the tasks have the same effect no matter how many times they are executed.

    (tags: architecture messaging queues exactly-once-delivery reliability fault-tolerance distcomp marc-brooker)

  • GpsMapIreland

    This topographic map represents Ireland. It is designed for "hillwalking". The contour lines are extracted from SRTM public data provided by NASA. These files contain a digitized ground represented by points. The sample rate defines a grid resolution for Ireland around 90m in northing and 60m in easting. In major cases, digitized points do not correspond with summits. Carrauntoohil (1039m, the highest summit of Ireland) does not appear in SRTM data. The altitude reaches only 1018m. Data were obtain from space with a radar. Because of the relative position between the radar and the earth, a shadow appears in some conditions (along ridges, behind summits...). This shadow matches with a gap in data (Imagine you with a flashlight in a dark room. It is hard to see what is in shadow). To close these gaps, you need other data or you can do interpolation. The second solution is applied in our case. There is one square degree per SRTM file with a sample rate of 1200x1200 points/square degree at Ireland latitude. [...] All in all you obtain contour lines pretty sufficient for walking.

    (tags: hillwalking walking ireland gps garmin open-data srtm maps hiking via:alan)

  • castnow

    Marcus Ramberg says: 'If you have a chromecast and you’re not using castnow, I don’t know what is wrong with you.'

    (tags: chromecast castnow media movies tv video)

  • The Infinite Hows, instead of the Five Whys

    John Allspaw with an interesting assertion that we need to ask "how", not "why" in five-whys postmortems:

    “Why?” is the wrong question. In order to learn (which should be the goal of any retrospective or post-hoc investigation) you want multiple and diverse perspectives. You get these by asking people for their own narratives. Effectively, you’re asking “how?“ Asking “why?” too easily gets you to an answer to the question “who?” (which in almost every case is irrelevant) or “takes you to the ‘mysterious’ incentives and motivations people bring into the workplace.” Asking “how?” gets you to describe (at least some) of the conditions that allowed an event to take place, and provides rich operational data.

    (tags: ops five-whys john-allspaw questions postmortems analysis root-causes)

  • the JVM now supports globbing in classpath specifications

    hooray, no more uberjars or monster commandlines!

    (tags: java jvm globbing classpath uberjars jars deployment)

  • Microsoft Azure 9-hour outage

    'From 19 Nov, 2014 00:52 to 05:50 UTC a subset of customers using Storage, Virtual Machines, SQL Geo-Restore, SQL Import/export, Websites, Azure Search, Azure Cache, Management Portal, Service Bus, Event Hubs, Visual Studio, Machine Learning, HDInsights, Automation, Virtual Network, Stream Analytics, Active Directory, StorSimple and Azure Backup Services in West US and West Europe experienced connectivity issues. This incident has now been mitigated.' There was knock-on impact until 11:00 UTC (storage in N Europe), 11:45 UTC (websites, West Europe), and 09:15 UTC (storage, West Europe), from the looks of things. Should be an interesting postmortem.

    (tags: outages azure microsoft ops)

Links for 2014-11-18

Published November 18, 2014

Links for 2014-11-17

Published November 17, 2014

  • FBI's "Suicide Letter" to Dr. Martin Luther King, Jr., and the Dangers of Unchecked Surveillance

    The entire letter could have been taken from a page of GCHQ’s Joint Threat Research and Intelligence Group (JTRIG)—though perhaps as an email or series of tweets. The British spying agency GCHQ is one of the NSA’s closest partners. The mission of JTRIG, a unit within GCHQ, is to “destroy, deny, degrade [and] disrupt enemies by discrediting them.” And there’s little reason to believe the NSA and FBI aren’t using such tactics. The implications of these types of strategies in the digital age are chilling. Imagine Facebook chats, porn viewing history, emails, and more made public to discredit a leader who threatens the status quo, or used to blackmail a reluctant target into becoming an FBI informant. These are not far-fetched ideas. They are the reality of what happens when the surveillance state is allowed to grow out of control, and the full King letter, as well as current intelligence community practices illustrate that reality richly.

    (tags: fbi surveillance mlk history blackmail snooping gchq nsa)

  • WriterReaderPhaser

    A nice new concurrency primitive from Gil Tene:

    Have you ever had a need for logging or analyzing data that is actively being updated? Have you ever wanted to do that without stalling the writers (recorders) in any way? If so, then WriterReaderPhaser is for you.  I'm not talking about logging messages or text lines here.  I'm talking about data.  Data larger than one word of memory.  Data that holds actual interesting state. Data that keeps being updated, but needs to be viewed in a stable and coherent way for analysis or logging.  Data like frame buffers. Data like histograms.  Data like usage counts. Data that changes.
    see also Left-Right: http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.html

    (tags: phasers data-structures concurrency primitives algorithms performance wait-free)

  • 3D Secure and Verified By Visa to be canned

    Yay.

    Mastercard and Visa are removing the need for users to enter their passwords for identity confirmation as part of a revamp of the existing (oft-criticised) 3-D Secure scheme. The arrival of 3D Secure 2.0 next year will see the credit card giants moving away from the existing system of secondary static passwords to authorise online purchases, as applied by Verified by Visa and MasterCard SecureCode, towards a next-gen system based on more secure biometric and token-based prompts.
    (via Gordon)

    (tags: via:gsyme verified-by-visa 3d-secure mastercard visa credit-cards authentication authorization win passwords)

  • Aeron: Do we really need another messaging system? - High Scalability

    excellent writeup on Aeron

    (tags: aeron messing libraries java martin-thompson performance mechanical-sympathy queueing ipc tcp)

Links for 2014-11-15

Published November 15, 2014

  • IAB Statement on Internet Confidentiality

    Newly designed protocols should prefer encryption to cleartext operation. There may be exceptions to this default, but it is important to recognize that protocols do not operate in isolation.  Information leaked by one protocol can be made part of a more substantial body of information by cross-correlation of traffic observation.  There are protocols which may as a result require encryption on the Internet even when it would not be a requirement for that protocol operating in isolation. We recommend that encryption be deployed throughout the protocol stack since there is not a single place within the stack where all kinds of communication can be protected.
    Wow. so much for IPSec

    (tags: ipsec iab ietf snowden surveillance crypto protocols internet)

  • Facebook's datacenter fabric

    FB goes public with its take on the Clos network-based datacenter network architecture

    (tags: networking scaling facebook clos-networks fabrics datacenters network-architecture)

Links for 2014-11-14

Published November 14, 2014

Links for 2014-11-13

Published November 13, 2014

Links for 2014-11-12

Published November 12, 2014

Links for 2014-11-11

Published November 11, 2014

  • Life expectancy increases are due mainly to healthier children, not longer old age

    Interesting -- I hadn't expected this. 'Life expectancy at birth [in the US] in 1930 was indeed only 58 for men and 62 for women, and the retirement age was 65. But life expectancy at birth in the early decades of the 20th century was low due mainly to high infant mortality, and someone who died as a child would never have worked and paid into Social Security. A more appropriate measure is probably life expectancy after attainment of adulthood.' .... 'Men who attained age 65 could expect to collect Social Security benefits for almost 13 years (and the numbers are even higher for women).' In Ireland, life expectancy at birth has increased 18.4 years since 1926 -- but life expectancy for men aged 65 (the pension age) has only increased by 3.8 years. This means that increased life expectancy figures are not particularly relevant to the "pension crunch" story. Via Fred Logue: https://twitter.com/fplogue/status/532093184646873089

    (tags: via:fplogue statistics taxes life-expectancy pensions infant-mortality health 1930s)

  • DynamoDB Streams

    This is pretty awesome. All changes to a DynamoDB table can be streamed to a Kinesis stream, MySQL-replication-style. The nice bit is that it has a solid way to ensure readers won't get overwhelmed by the stream volume (since ddb tables are IOPS-rate-limited), and Kinesis has a solid way to read missed updates (since it's a Kafka-style windowed persistent stream). With this you have a pretty reliable way to ensure you're not going to suffer data loss.

    (tags: iops dynamodb aws kinesis reliability replication multi-az multi-region failover streaming kafka)

  • Help the GNOME Foundation defend the GNOME trademark

    Recently Groupon announced a product with the same product name as GNOME. Groupon’s product is a tablet based point of sale “operating system for merchants to run their entire operation." The GNOME community was shocked that Groupon would use our mark for a product so closely related to the GNOME desktop and technology. It was almost inconceivable to us that Groupon, with over $2.5 billion in annual revenue, a full legal team and a huge engineering staff would not have heard of the GNOME project, found our trademark registration using a casual search, or even found our website, but we nevertheless got in touch with them and asked them to pick another name. Not only did Groupon refuse, but it has now filed even more trademark applications (the full list of applications they filed can be found here, here and here). To use the GNOME name for a proprietary software product that is antithetical to the fundamental ideas of the GNOME community, the free software community and the GNU project is outrageous. Please help us fight this huge company as they try to trade on our goodwill and hard earned reputation.

    (tags: gnome groupon trademark infringement open-source operating-systems ip law floss)

  • StatusPage.io

    'Hosted Status Pages for Your Company'. We use these guys in $work, and their service is fantastic -- it's a line of javascript in the page template which will easily allow you to add a "service degraded" banner when things go pear-shaped, along with an external status site for when things get really messy. They've done a good clean job.

    (tags: monitoring server status outages uptime saas infrastructure)

Links for 2014-11-10

Published November 10, 2014

  • Eircom have run out of network capacity

    This is due in part to huge growth in the data volumes and data traffic that is transported over our network, which has exceeded our forecasted growth. We are making a number of improvements to our international connectivity which will add significant capacity and this work will be completed in the next two or three weeks.
    Guess this is what happens when Amazon poach your IP network engineers. doh! More seriously though, if you're marketing eFibre heavily, shouldn't you be investing in the upstream capacity to go with it?

    (tags: eircom fail internet capacity forecasting networking)

  • Apple site lets you deactivate iMessage and solve your missing text problem

    FINALLY.

    (tags: apple imessage android switch hardware lock-in)

  • Yes, Isis exploits technology. But that’s no reason to compromise our privacy | Technology | The Observer

    From the very beginning, Isis fanatics have been up to speed on [social media]. Which raises an interesting question: how come that GCHQ and the other intelligence agencies failed to notice the rise of the Isis menace until it was upon us? Were they so busy hoovering metadata and tapping submarine cables and “mastering the internet” (as the code name of one of their projects puts it) that they didn’t have time to see what every impressionable Muslim 14-year-old in the world with an internet connection could see?

    (tags: gchq guardian encryption nsa isis technology social-media snooping surveillance)

  • This Canadian Artist Halted Pipeline Development by Copyrighting His Land as a Work of Art

    One of the really important pieces on my land was this white-picket fence. The picket fence is probably 100 yards or less, within 100 yards of where they wanted to build this pipeline. I [plan to] extend it 8 feet every year for the rest of my life and I've been doing that for 25 years. It got me thinking, where does this piece end? Does it end at the actual structure of the fence or the things growing around it, growing through it, that are part of the photography, the documentation of it? I realized at that point that [the fence], and the other sculptures and pieces and incursions and conceptual works, were actually integral to that piece of land and to my practice. I had not intended for it to be a political piece, it was just a piece, an idea the follow-through of which at some point became poetic, you go, "Wait a minute the fence actually stopped them!" But the fence doesn't actually enclose anything. It's just a straight line. And it's marking something that's actually unmarkable, which is time. And one day it'll be gone, as will I. The land will be changed--but it was just this crazy irony that kicked into play when I was standing there with those oil negotiators.

    (tags: copyright art pipelines canada politics oil land conceptual-art ip)

  • wrk2

    'A constant throughput, correct latency-recording variant of wrk. This is a must-have when measuring network service latency -- corrects for Coordinated Omission error:

    wrk's model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received. While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.

    (tags: wrk latency measurement tools cli http load-testing testing load-generation coordinated-omission gil-tene)

  • The problem of managing schemas

    Good post on the pain of using CSV/JSON as a data interchange format:

    eventually, the schema changes. Someone refactors the code generating the JSON and moves fields around, perhaps renaming few fields. The DBA added new columns to a MySQL table and this reflects in the CSVs dumped from the table. Now all those applications and scripts must be modified to handle both file formats. And since schema changes happen frequently, and often without warning, this results in both ugly and unmaintainable code, and in grumpy developers who are tired of having to modify their scripts again and again.

    (tags: schema json avro protobuf csv data-formats interchange data hadoop files file-formats)

Links for 2014-11-07

Published November 7, 2014

Links for 2014-11-06

Published November 6, 2014

Links for 2014-11-05

Published November 5, 2014

Links for 2014-11-04

Published November 4, 2014

  • Zookeeper: not so great as a highly-available service registry

    Turns out ZK isn't a good choice as a service discovery system, if you want to be able to use that service discovery system while partitioned from the rest of the ZK cluster:

    I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances.  This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones.  What I saw was that the two other instances noticed the first server “going away”, but they continued to function as they still saw a majority (66%).  More interestingly the first instance noticed the other two servers “going away”, dropping the ensemble availability to 33%.  This caused the first server to stop serving requests to clients (not only writes, but also reads).
    So: within that offline AZ, service discovery *reads* (as well as writes) stopped working due to a lack of ZK quorum. This is quite a feasible outage scenario for EC2, by the way, since (at least when I was working there) the network links between AZs, and the links with the external internet, were not 100% overlapping. In other words, if you want a highly-available service discovery system in the fact of network partitions, you want an AP service discovery system, rather than a CP one -- and ZK is a CP system. Another risk, noted on the Netflix Eureka mailing list at https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/tA9UnerrBHUJ :
    ZooKeeper, while tolerant against single node failures, doesn't react well to long partitioning events. For us, it's vastly more important that we maintain an available registry than a necessarily consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that's OK with us.
    I guess this means that a long partition can trigger SESSION_EXPIRED state, resulting in ZK client libraries requiring a restart/reconnect to fix. I'm not entirely clear what happens to the ZK cluster itself in this scenario though. Finally, Pinterest ran into other issues relying on ZK for service discovery and registration, described at http://engineering.pinterest.com/post/77933733851/zookeeper-resilience-at-pinterest ; sounds like this was mainly around load and the "thundering herd" overload problem. Their workaround was to decouple ZK availability from their services' availability, by building a Smartstack-style sidecar daemon on each host which tracked/cached ZK data.

    (tags: zookeeper service-discovery ops ha cap ap cp service-registry availability ec2 aws network partitions eureka smartstack pinterest)

  • Why We Didn’t Use Kafka for a Very Kafka-Shaped Problem

    A good story of when Kafka _didn't_ fit the use case:

    We came up with a complicated process of app-level replication for our messages into two separate Kafka clusters. We would then do end-to-end checking of the two clusters, detecting dropped messages in each cluster based on messages that weren’t in both. It was ugly. It was clearly going to be fragile and error-prone. It was going to be a lot of app-level replication and horrible heuristics to see when we were losing messages and at least alert us, even if we couldn’t fix every failure case. Despite us building a Kafka prototype for our ETL — having an existing investment in it — it just wasn’t going to do what we wanted. And that meant we needed to leave it behind, rewriting the ETL prototype.

    (tags: cassandra java kafka scala network-partitions availability multi-region multi-az aws replication onlive)

  • Madhumita Venkataramanan: My identity for sale (Wired UK)

    If the data aggregators know everything about you -- including biometric data, healthcare history, where you live, where you work, what you do at the weekend, what medicines you take, etc. -- and can track you as an individual, does it really matter that they don't know your _name_? They legally track, and sell, everything else.

    As the data we generate about ourselves continues to grow exponentially, brokers and aggregators are moving on from real-time profiling -- they're cross-linking data sets to predict our future behaviour. Decisions about what we see and buy and sign up for aren't made by us any more; they were made long before. The aggregate of what's been collected about us previously -- which is near impossible for us to see in its entirety -- defines us to companies we've never met. What I am giving up without consent, then, is not just my anonymity, but also my right to self-determination and free choice. All I get to keep is my name.

    (tags: wired privacy data-aggregation identity-theft future grim biometrics opt-out healthcare data data-protection tracking)

  • Linux kernel's Transparent Huge Pages feature causing 300ms-800ms pauses

    bad news for low-latency apps. See also its impact on redis: http://antirez.com/news/84

    (tags: redis memory defrag huge-pages linux kernel ops latency performance transparent-huge-pages)

  • Please grow your buffers exponentially

    Although in some cases x1.5 is considered good practice. YMMV I guess

    (tags: malloc memory coding buffers exponential jemalloc firefox heap allocation)

  • How I created two images with the same MD5 hash

    I found that I was able to run the algorithm in about 10 hours on an AWS large GPU instance bringing it in at about $0.65 plus tax.
    Bottom line: MD5 is feasibly attackable by pretty much anyone now.

    (tags: crypto images md5 security hashing collisions ec2 via:hn)

Links for 2014-11-03

Published November 3, 2014

Links for 2014-10-31

Published October 31, 2014

  • Chip & PIN vs. Chip & Signature

    Trust US banks to fuck up their attempts at security :( US "chip-and-signature" cards are still entirely forgeable because the banks fear that consumers are too stupid to use a PIN, basically.

    BK: So, I guess we should all be grateful that banks and retailers in the United States are finally taking steps to move toward chip [and signature] cards, but it seems to me that as long as these chip cards still also store cardholder data on a magnetic stripe as a backup, that the thieves can still steal and counterfeit this card data — even from chip cards. Litan: Yes, that’s the key problem for the next few years. Once mag stripe goes away, chip-and-PIN will be a very strong solution. The estimates are now that by the end of 2015, 50 percent of the cards and terminals will be chip-enabled, but it’s going to be several years before we get closer to full compliance. So, we’re probably looking at about 2018 before we can start making plans to get rid of the magnetic stripe on these cards.

    (tags: magstripe banks banking chip-and-pin security brian-krebs chip-and-signature)

  • Elastic MapReduce vs S3

    Turns out there are a few bugs in EMR's S3 support, believe it or not. 1. 'Consider disabling Hadoop's speculative execution feature if your cluster is experiencing Amazon S3 concurrency issues. You do this through the mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution configuration settings. This is also useful when you are troubleshooting a slow cluster.' 2. Upgrade to AMI 3.1.0 or later, otherwise retries of S3 ops don't work.

    (tags: s3 emr hadoop aws bugs speculative-execution ops)

Links for 2014-10-30

Published October 30, 2014

  • IT Change Management

    Stephanie Dean on Amazon's approach to CMs. This is solid gold advice for any company planning to institute a sensible technical change management process

    (tags: ops tech process changes change-management bureaucracy amazon stephanie-dean infrastructure)

  • Stephanie Dean on event management and incident response

    I asked around my ex-Amazon mates on twitter about good docs on incident response practices outside the "iron curtain", and they pointed me at this blog (which I didn't realise existed). Stephanie Dean was the front-line ops manager for Amazon for many years, over the time where they basically *fixed* their availability problems. She since moved on to Facebook, Demonware, and Twitter. She really knows her stuff and this blog is FULL of great details of how they ran (and still run) front-line ops teams in Amazon.

    (tags: ops incident-response outages event-management amazon stephanie-dean techops tos sev1)

  • RICON 2014: CRDTs

    Carlos Baquero presents several operation, state-based CRDTs for use in AP systems like Voldemort and Riak

    (tags: ap cap-theorem crdts ricon carlos-baquero data-structures distcomp)

  • Brownout: building more robust cloud applications

    Applications can saturate – i.e. become unable to serve users in a timely manner. Some users may experience high latencies, while others may not receive any service at all. The authors argue that it is better to downgrade the user experience and continue serving a larger number of clients with reasonable latency. "We define a cloud application as brownout compliant if it can gradually downgrade user experience to avoid saturation." This is actually very reminiscent of circuit breakers, as described in Nygard’s ‘Release It!’ and popularized by Netflix. If you’re already designing with circuit breakers, you’ve probably got all the pieces you need to add brownout support to your application relatively easily. "Our work borrows from the concept of brownout in electrical grids. Brownouts are an intentional voltage drop often used to prevent blackouts through load reduction in case of emergency. In such a situation, incandescent light bulbs dim, hence originating the term." "To lower the maintenance effort, brownouts should be automatically triggered. This enables cloud applications to rapidly and robustly avoid saturation due to unexpected environmental changes, lowering the burden on human operators."
    This is really similar to the Circuit Breaker pattern -- in fact it feels to me like a variation on that, driven by measured latencies of operations/requests. See also http://blog.acolyer.org/2014/10/27/improving-cloud-service-resilience-using-brownout-aware-load-balancing/ .

    (tags: circuit-breaker patterns brownout robustness reliability load latencies degradation)

  • Photographs of Sellafield nuclear plant prompt fears over radioactive risk

    "Slow-motion Chernobyl", as Greenpeace are calling it. You thought legacy code was a problem? try legacy Magnox fuel rods.

    Previously unseen pictures of two storage ponds containing hundreds of highly radioactive fuel rods at the Sellafield nuclear plant show cracked concrete, seagulls bathing in the water and weeds growing around derelict machinery. But a spokesman for owners Sellafield Ltd said the 60-year-old ponds will not be cleaned up for decades, despite concern that they are in a dangerous state and could cause a large release of radioactive material if they are allowed to deteriorate further. “The concrete is in dreadful condition, degraded and fractured, and if the ponds drain, the Magnox fuel will ignite and that would lead to a massive release of radioactive material,” nuclear safety expert John Large told the Ecologist magazine. “I am very disturbed at the run-down condition of the structures and support services. In my opinion there is a significant risk that the system could fail.

    (tags: energy environment nuclear uk sellafield magnox seagulls time long-now)

  • The man who made a game to change the world

    An interview with Richard Bartle, the creator of MUD, back in 1978.

    Perceiving the different ways in which players approached the game led Bartle to consider whether MMO players could be classified according to type. "A group of admins was having an argument about what people wanted out of a MUD in about 1990," he recalls. "This began a 200-long email chain over a period of six months. Eventually I went through everybody's answers and categorised them. I discovered there were four types of MMO player. I published some short versions of them then, when the journal of MUD research came out I wrote it up as a paper." The so-called Bartle test, which classifies MMO players as Achievers, Explorers, Socialisers or Killers (or a mixture thereof) according to their play-style remains in widespread use today. Bartle believes that you need a healthy mix of all dominant types in order to maintain a successful MMO ecosystem. "If you have a game full of Achievers (players for whom advancement through a game is the primary goal) the people who arrive at the bottom level won't continue to play because everyone is better than them," he explains. "This removes the bottom tier and, over time, all of the bottom tiers leave through irritation. But if you have Socialisers in the mix they don't care about levelling up and all of that. So the lowest Achievers can look down on the Socialisers and the Socialisers don't care. If you're just making the game for Achievers it will corrode from the bottom. All MMOs have this insulating layer, even if the developers don't understand why it's there."

    (tags: mmo mud gaming history internet richard-bartle)

  • Testing fork time on AWS/Xen infrastructure

    Redis uses forking to perform persistence flushes, which means that once every 30 minutes it performs like crap (and kills the 99th percentile latency). Given this, various Redis people have been benchmarking fork() times on various Xen platforms, since Xen has a crappy fork() implementation

    (tags: fork xen redis bugs performance latency p99)

  • A Teenager Gets Grilled By Her Dad About Why She’s Not That Into Coding

    Jay Rosen interviews his 17-year-old daughter. it's pretty eye-opening. Got to start them early!

    (tags: culture tech coding girls women feminism teenagers school jay-rosen stem)

Links for 2014-10-29

Published October 29, 2014

Links for 2014-10-28

Published October 28, 2014

Links for 2014-10-27

Published October 27, 2014

  • PSA: don't run 'strings' on untrusted files (CVE-2014-8485)

    ffs.

    Perhaps simply by the virtue of being a part of that bundle, the strings utility tries to leverage the common libbfd infrastructure to detect supported executable formats and "optimize" the process by extracting text only from specific sections of the file. Unfortunately, the underlying library can be hardly described as safe: a quick pass with afl (and probably with any other competent fuzzer) quickly reveals a range of troubling and likely exploitable out-of-bounds crashes due to very limited range checking

    (tags: strings libbfd gnu security fuzzing buffer-overflows)

Links for 2014-10-22

Published October 22, 2014

  • Jauter

    This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.

    (tags: java jauter scala request-routing http netty open-source)

  • "Viewstamped Replication Revisited", Liskov and Cowling [pdf]

    classic replication paper, via aphyr: 'This paper presents an updated version of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group. The paper also describes a number of important optimizations and presents a protocol for handling reconfigurations that can change both the group membership and the number of failures the group is able to handle.'

Links for 2014-10-21

Published October 21, 2014

  • BioBrick

    Holy shit we are living in the future.

    BioBrick parts are DNA sequences which conform to a restriction-enzyme assembly standard.[1][2] These Lego-like building blocks are used to design and assemble synthetic biological circuits, which would then be incorporated into living cells such as Escherichia coli cells to construct new biological systems.[3] Examples of BioBrick parts include promoters, ribosomal binding sites (RBS), coding sequences and terminators.
    (via Soren)

    (tags: via:sorenrags biobricks fabrication organisms artificial-life biology e-coli genetic-engineering)

  • Is Docker ready for production? Feedbacks of a 2 weeks hands on

    I have to agree with this assessment -- there are a lot of loose ends still for production use of Docker in a SOA stack environment:

    From my point of view, Docker is probably the best thing I’ve seen in ages to automate a build. It allows to pre build and reuse shared dependencies, ensuring they’re up to date and reducing your build time. It avoids you to either pollute your Jenkins environment or boot a costly and slow Virtualbox virtual machine using Vagrant. But I don’t feel like it’s production ready in a complex environment, because it adds too much complexity. And I’m not even sure that’s what it was designed for.

    (tags: docker complexity devops ops production deployment soa web-services provisioning networking logging)

Links for 2014-10-20

Published October 20, 2014

  • Load testing Apache Kafka on AWS

    This is a very solid benchmarking post, examining Kafka in good detail. Nicely done. Bottom line:

    I basically spend 2/3 of my work time torture testing and operationalizing distributed systems in production. There's some that I'm not so pleased with (posts pending in draft forever) and some that have attributes that I really love. Kafka is one of those systems that I pretty much enjoy every bit of, and the fact that it performs predictably well is only a symptom of the reason and not the reason itself: the authors really know what they're doing. Nothing about this software is an accident. Performance, everything in this post, is only a fraction of what's important to me and what matters when you run these systems for real. Kafka represents everything I think good distributed systems are about: that thorough and explicit design decisions win.

    (tags: testing aws kafka ec2 load-testing benchmarks performance)

Links for 2014-10-17

Published October 17, 2014

Links for 2014-10-16

Published October 16, 2014

  • Landlords not liable for tenants’ water bills

    What an utter fuckup. Business as usual for Irish Water:

    However the spokeswoman said application packs for rented dwellings would be addressed to the landlord, at the landlord’s residence, and it would be the landlord’s responsibility to ensure the tenant received the application pack. Bills are to be issued quarterly, but as Irish Water will have the tenant’s PPS number, the utility firm will be able to pursue the tenant for any arrears and even apply any arrears to new accounts, when the tenant moves to a new address. Last week landlords had expressed concern over potential arrears, the liability for them and the possibility of being used as collection agents by Irish Water.

    (tags: landlords ireland irish-water tenancy rental ppsn)

  • Irish Water responds to landlords’ questions

    ugh, what a mess....

    * Every rental unit in the State is to get a pack addressed personally to the occupant. If Irish Water does not have details of a tenant, the pack will be addressed to ‘The Occupier’ * Packs will only be issued to individual rental properties in so far as Irish Water is aware of them * Landlords can contact Irish Water to advise they have let a property * Application Packs are issued relative to the information on the Irish Water mailing list. If this is incorrect or out of date, landlords can contact Irish Water to have the information adjusted *Irish Water will contact known landlords after the initial customer application campaign, to advise of properties for which no application has been received * Irish Water said that when a household is occupied the tenant is liable and when vacant the owner is liable. Both should advise Irish Water of change of status to the property - the tenant to cease liability, the landlord to take it up. Either party may take a reading and provide it to Irish Water, alternatively Irish Water will bill on average consumption, based on the date of change.

    (tags: irish-water water ireland liability bills landlords tenancy rental)

Links for 2014-10-15

Published October 15, 2014

Links for 2014-10-14

Published October 14, 2014

  • Dublin's Best-Kept Secret: Blas Cafe

    looks great, around the corner from Cineworld on King's Inn St, D1

    (tags: dublin cafes food blas-cafe eating northside)

  • "Meta-Perceptual Helmets For The Dead Zoo"

    with Neil McKenzie, Nov 9-16 2014, in the National History Museum in Dublin: 'These six helmets/viewing devices start off by exploring physical conditions of viewing: if we have two eyes, they why is our vision so limited? Why do we have so little perception of depth? Why don’t our two eyes offer us two different, complementary views of the world around us? Why can’t they extend from our body so we can see over or around things? Why don’t they allow us to look behind and in front at the same time, or sideways in both directions? Why can’t our two eyes simultaneously focus on two different tasks? Looking through Michael Land’s defining work Animal Eyes, we see that nature has indeed explored all of these possibilities: a Hammerhead Shark has hyper-stereo vision; a horse sees 350° around itself; a chameleon has separately rotatable eyes… The series of Meta-Perceptual Helmets do indeed explore these zoological typologies: proposing to humans the hyper-stereo vision of the hammerhead shark; or the wide peripheral vision of the horse; or the backward/forward vision of the chameleon… but they also take us into the unnatural world of mythology and literature: the Cheshire Cat Helmet is so called because of the strange lingering effect of dominating visual information such as a smile or the eyes; the Cyclops allows one large central eye to take in the world around while a second tiny hidden eye focuses on a close up task (why has the creature never evolved that can focus on denitting without constantly having to glance around?).' (via Emma)

    (tags: perception helmets dublin ireland museums dead-zoo sharks eyes vision art)

  • Grade inflation figures from Irish universities

    The figures show that, between 2004 and 2013, an average of 71.7 per cent of students at TCD graduated with either a 1st or a 2.1. DCU and UCC had the next highest rate of such awards (64.3 per cent and 64.2 per cent respectively), followed by UCD (55.8 per cent), NUI Galway (54.7 per cent), Maynooth University (53.7 per cent) and University of Limerick (50.2 per cent).

    (tags: tcd grades grade-inflation dcu ucc ucd ireland studies academia third-level)

  • webrtcH4cKS: ~ coTURN: the open-source multi-tenant TURN/STUN server you were looking for

    Last year we interviewed Oleg Moskalenko and presented the rfc5766-turn-server project, which is a free open source and extremely popular implementation of TURN and STURN server. A few months later we even discovered Amazon is using this project to power its Mayday service. Since then, a number of features beyond the original RFC 5766 have been defined at the IETF and a new open-source project was born: the coTURN project.

    (tags: webrtc turn sturn rfc-5766 push nat stun firewalls voip servers internet)

  • Google Online Security Blog: This POODLE bites: exploiting the SSL 3.0 fallback

    Today we are publishing details of a vulnerability in the design of SSL version 3.0. This vulnerability allows the plaintext of secure connections to be calculated by a network attacker.
    ouch.

    (tags: ssl3 ssl tls security exploits google crypto)

Links for 2014-10-13

Published October 13, 2014

Links for 2014-10-11

Published October 11, 2014

Links for 2014-10-10

Published October 10, 2014

  • How Videogames Like Minecraft Actually Help Kids Learn to Read | WIRED

    I analyzed several chunks of The Ultimate Player's Guide using the Flesch-Kincaid Reading Ease scale, and they scored from grade 8 to grade 11. Yet in my neighborhood they're being devoured by kids in the early phases of elementary school. Games, it seems, can motivate kids to read—and to read way above their level. This is what Constance Steinkuehler, a games researcher at the University of Wisconsin-Madison, discovered. She asked middle and high school students who were struggling readers (one 11th-grade student read at a 6th-grade level) to choose a game topic they were interested in, and then she picked texts from game sites for them to read—some as difficult as first-year-college language. The kids devoured them with no help and nearly perfect accuracy. How could they do this? “Because they're really, really motivated,” Steinkuehler tells me. It wasn't just that the students knew the domain well; there were plenty of unfamiliar words. But they persisted more because they cared about the task. “It's situated knowledge. They see a piece of language, a turn of phrase, and they figure it out.”
    When my kids are playing Minecraft, there's a constant stream of "how do you spell X?" as they craft nametags for their pets. It's great!

    (tags: minecraft gaming kids education spelling school reading literacy)

  • "Gold" 4-star review from the Irish Times

    Niall Heery belatedly follows up Small Engine Repair, his 2006 mumblecore critical hit, with a slightly less off-centre comedy that makes imaginative use of a smashing cast. The story skirts tragedy on its leisurely passage from mishap to misadventure, but Gold remains the sort of picture you want to hug indulgently to a welcoming bosom. It gives humanism a good name.
    Go Niall! it's a great movie, go see it

    (tags: gold movies friends reviews irish-times indie)

  • JCDecaux Developer API

    web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):

    Two kinds of data are delivered by the platform: Static data provides stable information like station position, number of bike stands, payment terminal availability, etc. Dynamic data provides station state, number of available bikes, number of free bike stands, etc. Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.
    Ruby API: https://github.com/oisin/bikes

    (tags: jcdecaux bikesharing dublin dublin-bikes api web-services http json open-data)

  • Best games I played in 2013

    my coworker JK's favourite games of 2013: Gone Home, Last Of Us, Proteus, Papers Please etc. I really want to play these, since they're all totally my bag too.

    (tags: games gaming todo want 2013)

  • Why Amazon Has No Profits (And Why It Works)

    Amazon has perhaps 1% of the US retail market by value. Should it stop entering new categories and markets and instead take profit, and by extension leave those segments and markets for other companies? Or should it keep investing to sweep them into the platform? Jeff Bezos’s view is pretty clear: keep investing, because to take profit out of the business would be to waste the opportunity. He seems very happy to keep seizing new opportunities, creating new businesses, and using every last penny to do it.

    (tags: amazon business strategy capex spending stocks investing retail)

  • Spark Breaks Previous Large-Scale Sort Record – Databricks

    Massive improvement over plain old Hadoop. This blog post goes into really solid techie reasons why, including:

    First and foremost, in Spark 1.1 we introduced a new shuffle implementation called sort-based shuffle (SPARK-2045). The previous Spark shuffle implementation was hash-based that required maintaining P (the number of reduce partitions) concurrent buffers in memory. In sort-based shuffle, at any given point only a single buffer is required. This has led to substantial memory overhead reduction during shuffle and can support workloads with hundreds of thousands of tasks in a single stage (our PB sort used 250,000 tasks).
    Also, use of Timsort, an external shuffle service to offload from the JVM, Netty, and EC2 SR-IOV.

    (tags: spark hadoop map-reduce batch parallel sr-iov benchmarks performance netty shuffle algorithms sort-based-shuffle timsort)

  • UK psyops created N. Irish Satanic Panic during the Troubles - Boing Boing

    During the 1970s, when Northern Ireland was gripped by near-civil-war, British military intelligence staged the evidence of "black masses" in order to create a Satanism panic among the "superstitious" Irish to discredit the paramilitaries. The secret history of imaginary Irish Satanism is documented in Black Magic and Bogeymen: Fear, Rumour and Popular Belief in the North of Ireland 1972-74, a new book from Sheffield University's Richard Jenkins, who interviewed Captain Colin Wallace, the former head of British Army "black operations" for Northern Ireland.

    (tags: northern-ireland 1970s the-troubles ireland uvf ira history black-magic satanism weird fear mi5)

  • Netflix release new code to production before completing tests

    Interesting -- I hadn't heard of this being an official practise anywhere before (although we actually did it ourselves this week)...

    If a build has made it [past the 'integration test' phase], it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments. 

    (tags: devops deployment feature-flags release testing integration-tests uat qa production ops gating netflix)

Links for 2014-10-09

Published October 9, 2014

Links for 2014-10-08

Published October 8, 2014

  • Trouble at the Koolaid Point — Serious Pony

    This is a harrowing post from Kathy Sierra, full of valid observations:

    You’re probably more likely to win the lottery than to get any law enforcement agency in the United States to take action when you are harassed online, no matter how visciously and explicitly. Local agencies lack the resources, federal agencies won’t bother.
    That to the power of ten in Ireland, too, I'd suspect. Fuck this. Troll culture is way out of control....

    (tags: twitter harassment feminism weev abuse trolls 4chan kathy-sierra)

  • Tehuti

    An embryonic metrics library for Java/Scala from Felix GV at LinkedIn, extracted from Kafka's metric implementation and in the new Voldemort release. It fixes the major known problems with the Meter/Timer implementations in Coda-Hale/Dropwizard/Yammer Metrics. 'Regarding Tehuti: it has been extracted from Kafka's metric implementation. The code was originally written by Jay Kreps, and then maintained improved by some Kafka and Voldemort devs, so it definitely is not the work of just one person. It is in my repo at the moment but I'd like to put it in a more generally available (git and maven) repo in the future. I just haven't had the time yet... As for comparing with CodaHale/Yammer, there were a few concerns with it, but the main one was that we didn't like the exponentially decaying histogram implementation. While that implementation is very appealing in terms of (low) memory usage, it has several misleading characteristics (a lack of incoming data points makes old measurements linger longer than they should, and there's also a fairly high possiblity of losing interesting outlier data points). This makes the exp decaying implementation robust in high throughput fairly constant workloads, but unreliable in sparse or spiky workloads. The Tehuti implementation provides semantics that we find easier to reason with and with a small code footprint (which we consider a plus in terms of maintainability). Of course, it is still a fairly young project, so it could be improved further.' More background at the kafka-dev thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E

    (tags: kafka metrics dropwizard java scala jvm timers ewma statistics measurement latency sampling tehuti voldemort linkedin jay-kreps)

  • "Quantiles on Streams" [paper, 2009]

    'Chiranjeeb Buragohain and Subhash Suri: "Quantiles on Streams" in Encyclopedia of Database Systems, Springer, pp 2235–2240, 2009. ISBN: 978-0-387-35544-3', cited by Martin Kleppman in http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E as a good, short literature survey re estimating percentiles with a small memory footprint.

    (tags: latency percentiles coding quantiles streams papers algorithms)

  • Belkin Router Apocalypse

    Many Belkin routers attempt to determine if they're connected to the internet by pinging 'heartbeat.belkin.com', in a classic amateur fail move. Good reason not to run Belkin firmware if that's the level of code quality to expect

    (tags: belkin fail ping icmp funny internet dailywtf broken)

  • Seven deadly sins of talking about “types”

    Good essay

    (tags: programming types rants coding languages fp)

  • Shellshock

    An _extremely_ detailed resource about the bash bug

    (tags: bash hacking security shell exploits reference shellshock)

Links for 2014-10-07

Published October 7, 2014

Links for 2014-10-06

Published October 6, 2014

  • Reddit’s crappy ultimatum to remote workers and offices

    Reddit forces all remote workers (about half the workforce, in SLC and NYC) to move to SF, provoking a shitstorm:

    In a tweet confirming the move, Reddit’s CEO justified his treatment of non-San Francisco workers with a push for Optimal Teamwork to drive the New And $50M Improved Reddit forward. I shit you not. That was the actual term! (I added the New & Improved fan fiction here). So let’s leave aside the debate over whether working remotely is as efficient as being in the same office all the time. Let’s just focus on the size of the middle finger given to the people who work at Reddit outside the Bay Area, given the choice of forced, express relocation or a pink slip. How optimal do you think these employees will feel about leadership and the rest of the team going forward? Do you think they’ll just show up at the new, apparently-not-even-in-San-Francisco-proper office with a smile from ear to ear, ready to begin in earnest on Optimal Teamwork, left-behind former colleagues be damned?

    (tags: telecommuting reddit working remote-working ceos optimal-teamwork teamwork relocation)

  • Space Jacket

    'I designed this jacket as a tribute to the continuing legacy of American spaceflight. I wanted it to embody everything I loved about the space program, and to eventually serve as an actual flight jacket for present-day astronauts on missions to the ISS (International Space Station). There are other “replica” flight jackets made for space enthusiasts, but I decided to come up with something boldly different, yet also completely wearable and well-suited for space.'

    (tags: space clothing fashion geekery jackets)

  • How did Twitter become the hate speech wing of the free speech party?

    Kevin Marks has a pretty good point here:

    Your tweet could win the fame lottery, and everyone on the Internet who thinks you are wrong could tell you about it. Or one of the "verified" could call you out to be the tribute for your community and fight in their Hunger Games. Say something about feminism, or race, or sea lions and you'd find yourself inundated by the same trite responses from multitudes. Complain about it, and they turn nasty, abusing you, calling in their friends to join in. Your phone becomes useless under the weight of notifications; you can't see your friends support amongst the flood. The limited tools available - blocking, muting, going private - do not match well with these floods. Twitter's abuse reporting form takes far longer than a tweet, and is explicitly ignored if friends try to help.

    (tags: harassment twitter 4chan abuse feminism hate-speech gamergate sea-lions filtering social-media kevin-marks)

  • Mnesia and CAP

    A common “trick” is to claim: 'We assume network partitions can’t happen. Therefore, our system is CA according to the CAP theorem.' This is a nice little twist. By asserting network partitions cannot happen, you just made your system into one which is not distributed. Hence the CAP theorem doesn’t even apply to your case and anything can happen. Your system may be linearizable. Your system might have good availability. But the CAP theorem doesn’t apply. [...] In fact, any well-behaved system will be “CA” as long as there are no partitions. This makes the statement of a system being “CA” very weak, because it doesn’t put honesty first. I tries to avoid the hard question, which is how the system operates under failure. By assuming no network partitions, you assume perfect information knowledge in a distributed system. This isn’t the physical reality.

    (tags: cap erlang mnesia databases storage distcomp reliability ca postgres partitions)

  • Integrating Kafka and Spark Streaming: Code Examples and State of the Game

    Spark Streaming has been getting some attention lately as a real-time data processing tool, often mentioned alongside Apache Storm. [...] I added an example Spark Streaming application to kafka-storm-starter that demonstrates how to read from Kafka and write to Kafka, using Avro as the data format and Twitter Bijection for handling the data serialization. In this post I will explain this Spark Streaming example in further detail and also shed some light on the current state of Kafka integration in Spark Streaming. All this with the disclaimer that this happens to be my first experiment with Spark Streaming.

    (tags: spark kafka realtime architecture queues avro bijection batch-processing)

  • Mandos

    'a system for allowing servers with encrypted root file systems to reboot unattended and/or remotely.' (via Tony Finch)

    (tags: via:fanf mandos encryption security server ops sysadmin linux)

  • Zonify

    'a set of command line tools for managing Route53 DNS for an AWS infrastructure. It intelligently uses tags and other metadata to automatically create the associated DNS records.'

    (tags: zonify aws dns ec2 route53 ops)

  • Mike Perham on Twitter: "Sweet, monit just sent a DMCA takedown notice to @github to remove Inspeqtor."

    'The work, Inspeqtor which is hosted at GitHub, is far from a “clean-room” implementation. This is basically a rewrite of Monit in Go, even using the same configuration language that is used in Monit, verbatim. a. [private] himself admits that Inspeqtor is "heavily influenced“ by Monit https://github.com/mperham/inspeqtor/wiki/Other-Solutions. b. This tweet by [private] demonstrate intent. https://twitter.com/mperham/status/452160352940064768 "OSS nerds: redesign and build monit in Go. Sell it commercially. Make $$$$. I will be your first customer.”' IANAL, but using the same config language does not demonstrate copyright infringement...

    (tags: copyright dmca tildeslash monit inspeqtor github ops oss agpl)

  • YOU AND YOUR DAMNED GAMES, JON STONE — Why bother with #gamergate?

    So what is #gamergate? #gamergate is a mob with torches aloft, hunting for any combustible dwelling and calling it a monster’s lair. #gamergate is a rage train, and everyone with an axe to grind wants a ride. Its fuel is a sour mash of entitlement, insecurity, arrogance and alienation. #gamergate is a vindication quest for political intolerance. #gamergate is revenge for every imagined slight. #gamergate is Viz’s Meddlesome Ratbag.

    (tags: gamergate culture gaming 4chan mobs feminism)

Links for 2014-10-05

Published October 5, 2014

  • 'In 1976 I discovered Ebola, now I fear an unimaginable tragedy' | World news | The Observer

    An interview with the scientist who was part of the team which discovered the Ebola virus in 1976:

    Other samples from the nun, who had since died, arrived from Kinshasa. When we were just about able to begin examining the virus under an electron microscope, the World Health Organisation instructed us to send all of our samples to a high-security lab in England. But my boss at the time wanted to bring our work to conclusion no matter what. He grabbed a vial containing virus material to examine it, but his hand was shaking and he dropped it on a colleague's foot. The vial shattered. My only thought was: "Oh, shit!" We immediately disinfected everything, and luckily our colleague was wearing thick leather shoes. Nothing happened to any of us.

    (tags: ebola epidemiology health africa labs history medicine)

Links for 2014-10-04

Published October 4, 2014

  • Ebola vaccine delayed by IP spat

    This is the downside of publicly-funded labs selling patent-licensing rights to private companies:

    Given the urgency, it's inexplicable that one of the candidate vaccines, developed at the Public Health Agency of Canada (PHAC) in Winnipeg, has yet to go in the first volunteer's arm, says virologist Heinz Feldmann, who helped develop the vaccine while at PHAC. "It’s a farce; these doses are lying around there while people are dying in Africa,” says Feldmann, who now works at the Rocky Mountain Laboratories of the U.S. National Institute of Allergy and Infectious Diseases (NIAID) in Hamilton, Montana. At the center of the controversy is NewLink Genetics, a small company in Ames, Iowa, that bought a license to the vaccine's commercialization from the Canadian government in 2010, and is now suddenly caught up in what WHO calls "the most severe acute public health emergency seen in modern times.” Becker and others say the company has been dragging its feet the past 2 months because it is worried about losing control over the development of the vaccine.

    (tags: ip patents drugs ebola canada phac newlink-genetics health epidemics vaccines)

  • sferik/t

    "A command-line power tool for Twitter." It really is -- much better timeline searchability than the "real" Twitter UI, for example

    (tags: twitter ruby github cli tools unix search)

  • Ebola: While Big Pharma Slept

    We’ve had almost 40 years to develop, test and stockpile an Ebola vaccine. That has not happened because big pharma has been entirely focused on shareholder value and profits over safety and survival from a deadly virus. For the better part of Ebola’s 38 years ? big pharma has been asleep. The question ahead is what virus or superbug will wake them up?

    (tags: pharma ebola ip patents health drugs africa research)

Links for 2014-10-02

Published October 2, 2014

Links for 2014-10-01

Published October 1, 2014

Links for 2014-09-29

Published September 29, 2014

  • Prototype

    Prototype is a brand new festival of play and interaction. This is your chance to experience the world from a new perspective with removable camera eyes, to jostle and joust to a Bach soundtrack whilst trying to disarm an opponent, to throw shapes as you figure out who got an invite to the silent disco, to duel with foam pool noodles, and play chase in the dark with flashlights. A unique festival that incites new types of social interaction, involving technology and the city, Prototype is a series of performances, workshops, talks, and games that spill across the city, alongside an adult playground in the heart of Temple Bar.
    Project Arts Centre, 17-18 October. looks nifty

    (tags: prototype festivals dublin technology make vr gaming)

  • Confessions of a former internet troll - Vox

    I want to tell you about when violent campaigns against harmless bloggers weren't any halfway decent troll's idea of a good time — even the then-malicious would've found it too easy to be fun. When the punches went up, not down. Before the best players quit or went criminal or were changed by too long a time being angry. When there was cruelty, yes, and palpable strains of sexism and racism and every kind of phobia, sure, but when these things had the character of adolescents pushing the boundaries of cheap shock, disagreeable like that but not criminal. Not because that time was defensible — it wasn't, not really — but because it was calmer and the rage wasn't there yet. Because trolling still meant getting a rise for a laugh, not making helpless people fear for their lives because they're threatening some Redditor's self-proclaimed monopoly on reason. I want to tell you about it because I want to make sense of how it is now and why it changed.

    (tags: vox trolls blogging gamergate 4chan weev history teenagers)

Links for 2014-09-27

Published September 27, 2014

  • La Maison des Amis

    Paul Hickey's gite near Toulouse, available for rent! 'a beautifully converted barn on 5 acres, wonderfully located in the French countryside. 4 Bedrooms, sleeps 2-10, Large Pool, Tennis Court, Large Trampoline, Broadband Internet, 30 Mins Toulouse/Albi, 65 Mins Carcassonne, 90 Mins Rodez'

    (tags: ex-iona gites france holidays vacation-rentals vacation hotels)

  • waxpancake on Ello

    The Ello founders are positioning it as an alternative to other social networks — they won't sell your data or show you ads. "You are not the product." If they were independently-funded and run as some sort of co-op, bootstrapped until profitable, maybe that's plausible. Hard, but possible. But VCs don't give money out of goodwill, and taking VC funding — even seed funding — creates outside pressures that shape the inevitable direction of a company.

    (tags: advertising money vc ello waxy funding series-a)

  • Inviso: Visualizing Hadoop Performance

    With the increasing size and complexity of Hadoop deployments, being able to locate and understand performance is key to running an efficient platform.  Inviso provides a convenient view of the inner workings of jobs and platform.  By simply overlaying a new view on existing infrastructure, Inviso can operate inside any Hadoop environment with a small footprint and provide easy access and insight.  
    This sounds pretty useful.

    (tags: inviso netflix hadoop emr performance ops tools)

  • The End of Linux

    'Linux is becoming the thing that we adopted Linux to get away from.' Great post on the horrible complexity of systemd. It reminds me of nothing more than mid-90s AIX, which I had the displeasure of opsing for a while -- the Linux distros have taken a very wrong turn here.

    (tags: linux unix complexity compatibility ops rant systemd bloat aix)

Links for 2014-09-24

Published September 24, 2014

Links for 2014-09-23

Published September 23, 2014

  • Avoiding Chef-Suck with Auto Scaling Groups - forty9ten

    Some common problems which arise using Chef with ASGs in EC2, and how these guys avoided it -- they stopped using Chef for service provisioning, and instead baked AMIs when a new version was released. ASGs using pre-baked AMIs definitely works well so this makes good sense IMO.

    (tags: infrastructure chef ops asg auto-scaling ec2 provisioning deployment)

  • Introducing Groups.io

    Mark "ONEList" Fletcher's back, and he's reinventing the email group! awesome.

    email groups (the modern version of mailing lists) have stagnated over the past decade. Yahoo Groups and Google Groups both exude the dank air of benign neglect. Google Groups hasn’t been updated in years, and some of Yahoo’s recent changes have actually made Yahoo Groups worse! And yet, millions of people put up with this uncertainty and neglect, because email groups are still one of the best ways to communicate with groups of people. And I have a plan to make them even better. So today I’m launching Groups.io in beta, to bring email groups into the 21st Century. At launch, we have many features that those other services don’t have, including: Integration with other services, including: Github, Google Hangouts, Dropbox, Instagram, Facebook Pages, and the ability to import Feeds into your groups. Businesses and organizations can have their own private groups on their own subdomain. Better archive organization, using hashtags. Many more email delivery options. The ability to mute threads or hashtags. Fully searchable archives, including searching within attachments. One other feature that Groups.io has that Yahoo and Google don’t, is a business model that’s not based on showing ads to you. Public groups are completely free on Groups.io. Private groups and organizations are very reasonably priced.

    (tags: email groups communication discussion mailing-lists groups.io yahoo google google-groups yahoo-groups)

Links for 2014-09-22

Published September 22, 2014

  • The Open Source Software Engagement Award

    SFU announces award for students who demonstrate excellence in contributing to an Open Source project

    (tags: sfu awards students open-source oss universities funding)

  • DublinDashboard

    'provides citizens, public sector workers and companies with real-time information, time-series indicator data, and interactive maps about all aspects of the city. It enables users to gain detailed, up to date intelligence about the city that aids everyday decision making and fosters evidence-informed analysis.'

    (tags: dublin dashboards maps geodata time-series open-data ireland)

  • mcrouter: A memcached protocol router for scaling memcached deployments

    New from Facebook engineering:

    Last year, at the Data@Scale event and at the USENIX Networked Systems Design and Implementation conference , we spoke about turning caches into distributed systems using software we developed called mcrouter (pronounced “mick-router”). Mcrouter is a memcached protocol router that is used at Facebook to handle all traffic to, from, and between thousands of cache servers across dozens of clusters distributed in our data centers around the world. It is proven at massive scale — at peak, mcrouter handles close to 5 billion requests per second. Mcrouter was also proven to work as a standalone binary in an Amazon Web Services setup when Instagram used it last year before fully transitioning to Facebook's infrastructure. Today, we are excited to announce that we are releasing mcrouter’s code under an open-source BSD license. We believe it will help many sites scale more easily by leveraging Facebook’s knowledge about large-scale systems in an easy-to-understand and easy-to-deploy package.
    This is pretty crazy -- basically turns a memcached cluster into a much more usable clustered-storage system, with features like shadowing production traffic, cold cache warmup, online reconfiguration, automatic failover, prefix-based routing, replicated pools, etc. Lots of good features.

    (tags: facebook scaling cache proxy memcache open-source clustering distcomp storage)

  • DIRECT MARKETING - A GENERAL GUIDE FOR DATA CONTROLLERS

    In particular:

    Where you have obtained contact details in the context of the sale of a product or service, you may only use these details for direct marketing by electronic mail if the following conditions are met: the product or service you are marketing is of a kind similar to that which you sold to the customer at the time you obtained their contact details At the time you collected the details, you gave the customer the opportunity to object, in an easy manner and without charge, to their use for marketing purposes Each time you send a marketing message, you give the customer the right to object to receipt of further messages The sale of the product or service occurred not more than twelve months prior to the sending of the electronic marketing communication or, where applicable, the contact details were used for the sending of an electronic marketing communication in that twelve month period.

    (tags: email spam regulations ireland law dpc marketing direct-marketing)

Links for 2014-09-21

Published September 21, 2014

Links for 2014-09-19

Published September 19, 2014

Links for 2014-09-18

Published September 18, 2014

  • 75% of domestic violence victims in US shelters were spied on by their abusers using spyware

    via Mikko

  • Alex Payne — Thoughts On Five Years of Emerging Languages

    One could read the success of Go as an indictment of contemporary PLT, but I prefer to see it as a reminder of just how much language tooling matters. Perhaps even more critical, Go’s lean syntax, selective semantics, and cautiously-chosen feature set demonstrate the importance of a strong editorial voice in a language’s design and evolution. Having co-authored a book on Scala, it’s been painful to see systems programmers in my community express frustration with the ambitious hybrid language. I’ve watched them abandon ship and swim back to the familiar shores of Java, or alternately into the uncharted waters of Clojure, Go, and Rust. A pity, but not entirely surprising if we’re being honest with ourselves. Unlike Go, Scala has struggled with tooling from its inception. More than that, Scala has had a growing editorial problem. Every shop I know that’s been successful with Scala has limited itself to some subset of the language. Meanwhile, in pursuit of enterprise developers, its surface area has expanded in seemingly every direction. The folks behind Scala have, thankfully, taken notice: upcoming releases are promised to focus on simplicity, clarity, and better tooling.

    (tags: scala go coding languages)

Links for 2014-09-17

Published September 17, 2014

  • Texas Judge References 'The Big Lebowski'

    "The First Amendment of the U.S. Constitution is similarly suspicious of prior restraints," wrote Justice Lehrmann in the decision highlighting a cornerstone that has "been reaffirmed time and again by the Supreme Court, this Court, Texas courts of appeals, legal treatises, and even popular culture." That last reference to popular culture contained an interesting footnote citing none other than Walter Sobchak, a character in ['The Big Lebowski'].

    (tags: lebowski movies coen-brothers prior-restraint law supreme-court walter-sobchak funny)

  • on using JSON as a config file format

    Ben Hughes on twitter: "JSON is fine for config files, if you don't want to comment your config file. Which is a way of saying, it isn't fine for config files."

    (tags: ben-hughes funny json file-formats config-files configuration software coding)

  • Understanding weak isolation is a serious problem

    Peter Bailis complaining about the horrors of modern transactional databases and their unserializability, which noone seems to be paying attention to: 'As you’re probably aware, there’s an ongoing and often lively debate between transactional adherents and more recent “NoSQL” upstarts about related issues of usability, data corruption, and performance. But, in contrast, many of these transactional inherents and the research community as a whole have effectively ignored weak isolation — even in a single server setting and despite the fact that literally millions of businesses today depend on weak isolation and that many of these isolation levels have been around for almost three decades.' 'Despite the ubiquity of weak isolation, I haven’t found a database architect, researcher, or user who’s been able to offer an explanation of when, and, probably more importantly, why isolation models such as Read Committed are sufficient for correct execution. It’s reasonably well known that these weak isolation models represent “ACID in practice,” but I don’t think we have any real understanding of how so many applications are seemingly (!?) okay running under them. (If you haven’t seen these models before, they’re a little weird. For example, Read Committed isolation generally prevents users from reading uncommitted or non-final writes but allows a number of bad things to happen, like lost updates during concurrent read-modify-write operations. Why is this apparently okay for many applications?)'

    (tags: acid consistency databases peter-bailis transactional corruption serializability isolation reliability)

  • "Left-Right: A Concurrency Control Technique with Wait-Free Population Oblivious Reads" [pdf]

    'In this paper, we describe a generic concurrency control technique with Blocking write operations and Wait-Free Population Oblivious read operations, which we named the Left-Right technique. It is of particular interest for real-time applications with dedicated Reader threads, due to its wait-free property that gives strong latency guarantees and, in addition, there is no need for automatic Garbage Collection. The Left-Right pattern can be applied to any data structure, allowing concurrent access to it similarly to a Reader-Writer lock, but in a non-blocking manner for reads. We present several variations of the Left-Right technique, with different versioning mechanisms and state machines. In addition, we constructed an optimistic approach that can reduce synchronization for reads.' See also http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.html for java implementation code.

    (tags: left-right concurrency multithreading wait-free blocking realtime gc latency reader-writer locking synchronization java)

  • Russell91/sshrc

    'bring your .bashrc, .vimrc, etc. with you when you ssh'. A really nice implementation of this idea (much nicer than my own version!)

    (tags: hacks productivity ssh remote shell sh bash via:johnke home-directory unix)

  • Troubleshooting Production JVMs with jcmd

    remotely trigger GCs, finalization, heap dumps etc. Handy

    (tags: jvm jcmd debugging ops java gc heap troubleshooting)

Links for 2014-09-16

Published September 16, 2014

Links for 2014-09-15

Published September 15, 2014

  • The State of ZFS on Linux

    Linux users familiar with other filesystems or ZFS users from other platforms will often ask whether ZFS on Linux (ZoL) is “stable”. The short answer is yes, depending on your definition of stable. The term stable itself is somewhat ambiguous.
    Oh dear. that's not a good start. Good reference page, though

    (tags: zfs linux filesystems ops solaris)

  • Screen time: Steve Jobs was a low tech parent

    “This is rule No. 1: There are no screens in the bedroom. Period. Ever.”

    (tags: screen-time kids children tv mobile technology life rules parenting)

  • CausalImpact: A new open-source package for estimating causal effects in time series

    How can we measure the number of additional clicks or sales that an AdWords campaign generated? How can we estimate the impact of a new feature on app downloads? How do we compare the effectiveness of publicity across countries? In principle, all of these questions can be answered through causal inference. In practice, estimating a causal effect accurately is hard, especially when a randomised experiment is not available. One approach we've been developing at Google is based on Bayesian structural time-series models. We use these models to construct a synthetic control — what would have happened to our outcome metric in the absence of the intervention. This approach makes it possible to estimate the causal effect that can be attributed to the intervention, as well as its evolution over time. We've been testing and applying structural time-series models for some time at Google. For example, we've used them to better understand the effectiveness of advertising campaigns and work out their return on investment. We've also applied the models to settings where a randomised experiment was available, to check how similar our effect estimates would have been without an experimental control. Today, we're excited to announce the release of CausalImpact, an open-source R package that makes causal analyses simple and fast. With its release, all of our advertisers and users will be able to use the same powerful methods for estimating causal effects that we've been using ourselves. Our main motivation behind creating the package has been to find a better way of measuring the impact of ad campaigns on outcomes. However, the CausalImpact package could be used for many other applications involving causal inference. Examples include problems found in economics, epidemiology, or the political and social sciences.

    (tags: causal-inference r google time-series models bayes adwords advertising statistics estimation metrics)

  • Top 10 Historic Sites in Ireland and Northern Ireland -- National Geographic

    Shamefully, I haven't visited most of these!

    (tags: history neolithic ireland northern-ireland national-geographic tourism places)

  • Software patents are crumbling, thanks to the Supreme Court

    Now a series of decisions from lower courts is starting to bring the ruling's practical consequences into focus. And the results have been ugly for fans of software patents. By my count there have been 11 court rulings on the patentability of software since the Supreme Court's decision — including six that were decided this month.  Every single one of them has led to the patent being invalidated. This doesn't necessarily mean that all software patents are in danger — these are mostly patents that are particularly vulnerable to challenge under the new Alice precedent. But it does mean that the pendulum of patent law is now clearly swinging in an anti-patent direction. Every time a patent gets invalidated, it strengthens the bargaining position of every defendant facing a lawsuit from a patent troll.

    (tags: patents law alice swpats software supreme-court patent-trolls)

  • Riding with the Stars: Passenger Privacy in the NYC Taxicab Dataset

    A practical demo of "differential privacy" -- allowing public data dumps to happen without leaking privacy, using Laplace noise addition

    (tags: differential-privacy privacy leaks public-data open-data data nyc taxis laplace noise randomness)

Links for 2014-09-14

Published September 14, 2014

  • Platform Game

    I'm ambivalent about Microsoft acquiring Mojang. Will they Embrace and Extend Minecraft as they've done with other categories? Let's hope not. On the other hand, some adult supervision and a Plugin API would be welcome. Mojang have the financial resources but lack the will and focus needed to publish and support a Plugin API. Perhaps Mojang themselves don't realise just how important their little game has become.

    (tags: minecraft platforms games plugins mojang microsoft)

Links for 2014-09-11

Published September 11, 2014

Links for 2014-09-10

Published September 10, 2014

  • Apple: Untrustable

    Today, Apple announced their “Most Personal Device Ever”. They also announced Apple Pay (the only mentions of “security” and “privacy” in today’s event), and are rolling out health tracking and home automation in iOS 8. Given their feckless track record [with cloud-service security], would you really trust Apple with (even more of) your digital life?

    (tags: icloud apple fail security hacks privacy)

  • Not Safe For Not Working On

    Excellent post from Dan Kaminsky on concrete actions that cloud service providers like Apple and Google need to start taking.

    *It's time to ban Password1*: [...] Defenders are using simple rules like “doesn’t have an uppercase letter” and “not enough punctuation” to block passwords while attackers are just straight up analyzing password dumps and figuring out the most likely passwords to attempt in any scenario.  Attackers are just way ahead.  That has to change.  Defenders have password dumps too now.  It’s time we start outright blocking passwords common enough that they can be online brute forced, and it’s time we admit we know what they are. [...] *People use communication technologies for sexy times. Deal with it*: Just like browsers have porn mode for the personal consumption of private imagery, cell phones have applications that are significantly less likely to lead to anyone else but your special friends seeing your special bits. I personally advise Wickr, an instant messaging firm that develops secure software for iPhone and Android. What’s important about Wickr here isn’t just the deep crypto they’ve implemented, though it’s useful too. What’s important in this context is that with this code there’s just a lot fewer places to steal your data from. Photos and other content sent in Wickr don’t get backed up to your desktop, don’t get saved in any cloud, and by default get removed from your friend’s phone after an amount of time you control. Wickr is of course not the only company supporting what’s called “ephemeral messaging”; SnapChat also dramatically reduces the exposure of your private imagery. [...]
    via Leonard.

    (tags: icloud apple privacy security via:lhl snapchat wickr dan-kaminsky cloud-services backup)

  • Inside Apple’s Live Event Stream Failure, And Why It Happened: It Wasn’t A Capacity Issue

    The bottom line with this event is that the encoding, translation, JavaScript code, the video player, the call to S3 single storage location and the millisecond refreshes all didn’t work properly together and was the root cause of Apple’s failed attempt to make the live stream work without any problems. So while it would be easy to say it was a CDN capacity issue, which was my initial thought considering how many events are taking place today and this week, it does not appear that a lack of capacity played any part in the event not working properly. Apple simply didn’t provision and plan for the event properly.

    (tags: cdn streaming apple fail scaling s3 akamai caching)

Links for 2014-09-09

Published September 9, 2014

Links for 2014-09-08

Published September 8, 2014

Links for 2014-09-06

Published September 6, 2014

  • 'The very first release of Gmail simply used spamassassin on the backend'

    Excellent. Confirming what I'd heard from a few other sources, too ;) This is a well-written history of the anti-spam war so far, from Mike Hearn, writing with the Google/Gmail point of view:

    Brief note about my background, to establish credentials: I worked at Google for about 7.5 years. For about 4.5 of those I worked on the Gmail abuse team, which is very tightly linked with the spam team (they use the same software, share the same on-call rotations etc).
    Reading this kind of stuff is awesome for me, since it's a nice picture of a fun problem to work on -- the Gmail team took the right ideas about how to fight spam, and scaled them up to the 10s-of-millions DAU mark. Nicely done. The second half is some interesting musings on end-to-end encrypted communications and how it would deal with spam. Worth a read...

    (tags: gmail google spam anti-spam filtering spamassassin history)

  • The FBI Finally Says How It ‘Legally’ Pinpointed Silk Road’s Server

    The answer, according to a new filing by the case’s prosecution, is far more mundane: The FBI claims to have found the server’s location without the NSA’s help, simply by fiddling with the Silk Road’s login page until it leaked its true location.

    (tags: fbi nsa silk-road tor opsec dread-pirate-roberts wired)

Links for 2014-09-05

Published September 5, 2014

Links for 2014-09-04

Published September 4, 2014

  • Visualizing Garbage Collection Algorithms

    Great dataviz with animated GIFs

    (tags: algorithms gc memory visualization garbage-collection dataviz refcounting mark-and-sweep)

  • Standard Markdown

    John Gruber’s canonical description of Markdown’s syntax does not specify the syntax unambiguously. In the absence of a spec, early implementers consulted the original Markdown.pl code to resolve these ambiguities. But Markdown.pl was quite buggy, and gave manifestly bad results in many cases, so it was not a satisfactory replacement for a spec. Because there is no unambiguous spec, implementations have diverged considerably. As a result, users are often surprised to find that a document that renders one way on one system (say, a GitHub wiki) renders differently on another (say, converting to docbook using Pandoc). To make matters worse, because nothing in Markdown counts as a “syntax error,” the divergence often isn't discovered right away. There's no standard test suite for Markdown; the unofficial MDTest is the closest thing we have. The only way to resolve Markdown ambiguities and inconsistencies is Babelmark, which compares the output of 20+ implementations of Markdown against each other to see if a consensus emerges. We propose a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations against this specification. We believe this is necessary, even essential, for the future of Markdown.

    (tags: writing markdown specs standards text formats html)

  • Postcodes at last but random numbers don’t address efficiency

    Karlin Lillington assembles a fine collection of quotes from various sources panning the new Eircode system:

    Critics say the opportunity has been missed to use Ireland’s clean-slate status to produce a technologically innovative postcode system that would be at the cutting edge globally; similar to the competitive leap that was provided when the State switched to a digital phone network in the 1980s, well ahead of most of the world. Instead, say organisations such as the Freight Transport Association of Ireland (FTAI), the proposed seven-digit format of scrambled letters and numbers is almost useless for a business sector that should most benefit from a proper postcode system: transport and delivery companies, from international giants like FedEx and UPS down to local courier, delivery and service supplier firms. Because each postcode will reveal the exact address of a home or business, privacy advocates are concerned that online use of postcodes could link many types of internet activity, including potentially sensitive online searches, to a specific household or business.

    (tags: eircode government fail ireland postcodes location ftai random)

Links for 2014-09-03

Published September 3, 2014

Links for 2014-09-02

Published September 2, 2014

  • Nix: The Purely Functional Package Manager

    'a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments. ' Basically, this is a third-party open source reimplementation of Amazon's (excellent) internal packaging system, using symlinks to versioned package directories to ensure atomicity and the ability to roll back. This is definitely the *right* way to build packages -- I know what tool I'll be pushing for, next time this question comes up. See also nixos.org for a Linux distro built on Nix.

    (tags: ops linux devops unix packaging distros nix nixos atomic upgrades rollback versioning)

Links for 2014-09-01

Published September 1, 2014

  • Facebook's drop-in replacement for std::vector

    Fixes some low-hanging fruit, performance-wise. 'Simply replacing std::vector with folly::fbvector (after having included the folly/FBVector.h header file) will improve the performance of your C++ code using vectors with common coding patterns. The improvements are always non-negative, almost always measurable, frequently significant, sometimes dramatic, and occasionally spectacular.' (via Tony Finch)

    (tags: c++ facebook performance algorithms vectors via:fanf optimization)

  • Applying cardiac alarm management techniques to your on-call

    An ops-focused take on a recent story about alarm fatigue, and how a Boston hospital dealt with it. When I was in Amazon, many of the teams in our division had a target to reduce false positive pages, with a definite monetary value attached to it, since many teams had "time off in lieu" payments for out-of-hours pages to the on-call staff. As a result, reducing false-positive pages was reasonably high priority and we dealt with this problem very proactively, with a well-developed sense of how to do so. It's interesting to see how the outside world is only just starting to look into its amelioration. (Another benefit of a TOIL policy ;)

    (tags: ops monitoring sysadmin alerts alarms nagios alarm-fatigue false-positives pages)

  • "Invertible Bloom Lookup Tables" [paper]

    'We present a version of the Bloom filter data structure that supports not only the insertion, deletion, and lookup of key-value pairs, but also allows a complete listing of the pairs it contains with high probability, as long the number of key- value pairs is below a designed threshold. Our structure allows the number of key-value pairs to greatly exceed this threshold during normal operation. Exceeding the threshold simply temporarily prevents content listing and reduces the probability of a successful lookup. If entries are later deleted to return the structure below the threshold, everything again functions appropriately. We also show that simple variations of our structure are robust to certain standard errors, such as the deletion of a key without a corresponding insertion or the insertion of two distinct values for a key. The properties of our structure make it suitable for several applications, including database and networking applications that we highlight.'

    (tags: iblt bloom-filters data-structures performance algorithms coding papers probabilistic)

  • Some UX Dark Patterns now illegal in the EU

    The EU’s new consumer rights law bans certain dark patterns related to e-commerce across Europe. The “sneak into basket” pattern is now illegal. Full stop, end of story. You cannot create a situation where additional items and services are added by default. [...] Hidden costs are now illegal, whether that’s an undeclared subscription, extra shipping charges, or extra items. [....] Forced continuity, when imposed on the user as a form of bait-and-switch, has been banned. Just the other day a web designer mentioned to me that he had only just discovered he had been charged for four years of annual membership dues in a “theme club”, having bought what he thought was a one-off theme. Since he lives in Europe, he may be able to claim all of this money back. All he needs to do is prove that the website did not inform him that the purchase included a membership with recurring payments.

    (tags: design europe law ecommerce ux dark-patterns scams ryanair selling online consumer consumer-rights bait-and-switch)

  • Girl Not Against Fluoride

    The CDC (Centre for Disease Control) lists water fluoridation as one of the ten great public health achievements of the 20th Century. Today, Dublin City Council will vote on whether to remove fluoride from our water supply, and when they do, it will not be because the CDC or the WHO have changed their mind about fluoridation, or because new and compelling information makes it the only choice. It will be because people who believe in angel healing, homeopathy, and chemtrails, have somehow gained the ability to influence public policy.

    (tags: dcc dublin law flouride science zenbuffy homeopathy woo health teeth)

  • Revisiting How We Put Together Linux Systems

    Building a running OS out of layered btrfs filesystems. This sounds awesome.

    Instantiating a new system or OS container (which is exactly the same in this scheme) just consists of creating a new appropriately named root sub-volume. Completely naturally you can share one vendor OS copy in one specific version with a multitude of container instances. Everything is double-buffered (or actually, n-fold-buffered), because usr, runtime, framework, app sub-volumes can exist in multiple versions. Of course, by default the execution logic should always pick the newest release of each sub-volume, but it is up to the user keep multiple versions around, and possibly execute older versions, if he desires to do so. In fact, like on ChromeOS this could even be handled automatically: if a system fails to boot with a newer snapshot, the boot loader can automatically revert back to an older version of the OS.
    (via Tony Finch)

    (tags: via:fanf linux docker btrfs filesystems unionfs copy-on-write os hacking unix)

Links for 2014-08-29

Published August 29, 2014

Links for 2014-08-28

Published August 28, 2014

Links for 2014-08-27

Published August 27, 2014

  • Apache Kafka 0.8 basic training

    This is a pretty voluminous and authoritative presentation about getting started with Kafka; wish this was around when we started using it for 0.7. (We use our own homegrown realtime system nowadays, due to better partitioning, monitoring and operability.)

    (tags: storm kafka presentations documentation ops)

  • Wiki Loves Monuments

    Wiki Loves Monuments is an international photo contest, organised by Wikimedia [...]. This year, the Wikimedia Ireland Community are running the competition for the very first time in Ireland. The contest is inspired by the successful 2010 pilot in the Netherlands which resulted in 12,500 freely licensed images uploaded to Wikimedia Commons. It has grown substantially since its inception; in 2013 369,589 photographs were submitted by 11,943 participants from over 50 countries. Cultural heritage is an important part of the knowledge that Wikipedia collects and disseminates. An image is worth a thousand words, in any language and local enthusiasts can (re)discover the cultural, historical, or scientific significance of their neighbourhood. The Irish contest, focussing on Ireland’s national monuments, runs from August 23 - September 30. Follow our step-by-step guide to find out how you can take part.

    (tags: wikipedia wikimedia images monuments history ireland contests creative-commons licensing)

  • "CryptoPhone" claims to detect IMSI catchers in operation

    To show what the CryptoPhone can do that less expensive competitors cannot, he points me to a map that he and his customers have created, indicating 17 different phony cell towers known as “interceptors,” detected by the CryptoPhone 500 around the United States during the month of July alone.  Interceptors look to a typical phone like an ordinary tower.  Once the phone connects with the interceptor, a variety of “over-the-air” attacks become possible, from eavesdropping on calls and texts to pushing spyware to the device. “Interceptor use in the U.S. is much higher than people had anticipated,” Goldsmith says.  “One of our customers took a road trip from Florida to North Carolina and he found 8 different interceptors on that trip.  We even found one at South Point Casino in Las Vegas.”

    (tags: imsi-catchers security cryptophone phones mobile 3g 4g eavesdropping surveillance)

Links for 2014-08-25

Published August 25, 2014

Links for 2014-08-22

Published August 22, 2014

Links for 2014-08-21

Published August 21, 2014

Links for 2014-08-19

Published August 19, 2014

  • Nyms Identity Directory

    The way that [problems with the PGP bootstrapping] are supposed to be resolved is with an authentication model called the Web of Trust where users sign keys of other users after verifying that they are who they say they are. In theory, if some due diligence is applied in signing other people’s keys and a sufficient number of people participate you’ll be able to follow a short chain of signatures from people you already know and trust to new untrusted keys you download from a key server. In practice this has never worked out very well as it burdens users with the task of manually finding people to sign their keys and even experts find the Web of Trust model difficult to reason about. This also reveals the social graph of certain communities which may place users at risk for their associations. Such signatures also reveal metadata about times and thus places for meetings for key signings. The Nyms Identity Directory is a replacement for all of this. Keyservers are replaced with an identity directory that gives users full control over publication of their key information and web of trust is replaced with a distributed network of trusted notaries which validate user keys with an email verification protocol.

    (tags: web-of-trust directories nyms privacy crypto identity trust pgp gpg security via:ioerror keyservers notaries)

  • Frogsort

    Frogsort as an exam question (via qwghlm)

    (tags: via:qwghlm frogsort sorting big-o algorithms funny comics smbc)

  • Punished for Being Poor: Big Data in the Justice System

    This is awful. Totally the wrong tool for the job -- a false positive rate which is miniscule for something like spam filtering, could translate to a really horrible outcome for a human life.

    Currently, over 20 states use data-crunching risk-assessment programs for sentencing decisions, usually consisting of proprietary software whose exact methods are unknown, to determine which individuals are most likely to re-offend. The Senate and House are also considering similar tools for federal sentencing. These data programs look at a variety of factors, many of them relatively static, like criminal and employment history, age, gender, education, finances, family background, and residence. Indiana, for example, uses the LSI-R, the legality of which was upheld by the state’s supreme court in 2010. Other states use a model called COMPAS, which uses many of the same variables as LSI-R and even includes high school grades. Others are currently considering the practice as a way to reduce the number of inmates and ensure public safety. (Many more states use or endorse similar assessments when sentencing sex offenders, and the programs have been used in parole hearings for years.) Even the American Law Institute has embraced the practice, adding it to the Model Penal Code, attesting to the tool’s legitimacy.
    (via stroan)

    (tags: via:stroan statistics false-positives big-data law law-enforcement penal-code risk sentencing)

Links for 2014-08-18

Published August 18, 2014

  • Microservices - Not a free lunch! - High Scalability

    Some good reasons not to adopt microservices blindly. Testability and distributed-systems complexity are my biggest fears

    (tags: microservices soa devops architecture testing distcomp)

  • Richard Clayton - Failing at Microservices

    Solid warts-and-all confessional blogpost about a team failing to implement a microservices architecture. I'd put most of the blame on insufficient infrastructure to support them (at a code level), inter-personal team problems, and inexperience with large-scale complex multi-service production deployment and the work it was going to require

    (tags: microservices devops collaboration architecture fail team deployment soa)

  • Box Tech Blog » A Tale of Postmortems

    How Box introduced COE-style dev/ops outage postmortems, and got them working. This PIE metric sounds really useful to head off the dreaded "it'll all have to come out missus" action item:

    The picture was getting clearer, and we decided to look into individual postmortems and action items and see what was missing. As it was, action items were wasting away with no owners. Digging deeper, we noticed that many action items entailed massive refactorings or vague requirements like “make system X better” (i.e. tasks that realistically were unlikely to be addressed). At a higher level, postmortem discussions often devolved into theoretical debates without a clear outcome. We needed a way to lower and focus the postmortem bar and a better way to categorize our action items and our technical debt. Out of this need, PIE (“Probability of recurrence * Impact of recurrence * Ease of addressing”) was born. By ranking each factor from 1 (“low”) to 5 (“high”), PIE provided us with two critical improvements: 1. A way to police our postmortems discussions. I.e. a low probability, low impact, hard to implement solution was unlikely to get prioritized and was better suited to a discussion outside the context of the postmortem. Using this ranking helped deflect almost all theoretical discussions. 2. A straightforward way to prioritize our action items. What’s better is that once we embraced PIE, we also applied it to existing tech debt work. This was critical because we could now prioritize postmortem action items alongside existing work. Postmortem action items became part of normal operations just like any other high-priority work.

    (tags: postmortems action-items outages ops devops pie metrics ranking refactoring prioritisation tech-debt)

  • NTP's days are numbered for consumer devices

    An accurate clock is required to negotiate SSL/TLS, so clock sync is important for internet-of-things usage. but:

    Unfortunately for us, the traditional and most widespread method for clock synchronisation (NTP) has been caught up in a DDoS issue which has recently caused some ISPs to start blocking all NTP communication. [....] Because the DDoS attacks are so widespread, and the lack of obvious commercial pressure to fix the issue, it’s possible that the days of using NTP as a mechanism for setting clocks may well be numbered. Luckily for us there is a small but growing project that replaces it. tlsdate was started by Jacob Appelbaum of the Tor project in 2012, making use of the SSL handshake in order to extract time from a remote server, and its usage is on the rise. [....] Since we started encountering these problems, we’ve incorporated tlsdate into an over-the-air update, and have successfully started using this in situations where NTP is blocked.

    (tags: tlsdate ntp clocks time sync iot via:gwire ddos isps internet protocols security)

  • Cloudwash – Creating the Technical Prototype

    This is a lovely demo of integrating modern IoT connectivity functionality (remote app control, etc.) with a washing machine using Bergcloud's hardware and backend, and a little logic-analyzer reverse engineering.

    (tags: arduino diy washing-machines iot bergcloud hacking reversing logic-analyzers hardware)

  • Systemd: Harbinger of the Linux apocalypse

    While there are many defensible aspects of Systemd, other aspects boggle the mind. Not the least of these was that, as of a few months ago, trying to debug the kernel from the boot line would cause the system to crash. This was because of Systemd's voracious logging and the fact that Systemd responds to the "debug" flag on the kernel boot line -- a flag meant for the kernel, not anything else. That, straight up, is a bug. However, the Systemd developers didn't see it that way and actively fought with those experiencing the problem. Add the fact that one of the Systemd developers was banned by Linus Torvalds for poor attitude and bad design and another was responsible for causing significant issues with Linux audio support, but blamed the problem on everything else but his software, and you have a bad situation on your hands. There's no shortage of egos in the open source development world. There's no shortage of new ideas and veteran developers and administrators pooh-poohing something new simply because it's new. But there are also 45 years of history behind Unix and extremely good reasons it's still flourishing. Tools designed like Systemd do not fit the Linux mold, to their own detriment. Systemd's design has more in common with Windows than with Unix -- down to the binary logging.
    The link re systemd consuming the "debug" kernel boot arg is a canonical example of inflexible coders refusing to fix their own bugs. (via Jason Dixon)

    (tags: systemd linux red-hat egos linus-torvalds unix init booting debugging logging design software via:obfuscurity)

  • Inside a Chinese Bitcoin Mine

    The mining operation resides on an old, repurposed factory floor, and contains 2500 machines hashing away at 230 Gh/s, each. (That’s 230 billion calculations per second, per unit). [...] The operators told me that the power bill of this specific operation is in excess of ¥400,000 per month [..] about $60,000 USD.

    (tags: currency china economics bitcoin power environment green mining datacenters)

  • Moving Big Data into the Cloud with Tsunami UDP - AWS Big Data Blog

    Pretty serious speedup. 81 MB/sec with Tsunami UDP, compared to 9 MB/sec with plain old scp. Probably kills internet performance for everyone else though!

    (tags: tsunami-udp udp scp copying transfers internet long-distance performance speed)

  • The "sidecar" pattern

    Ha, great name. We use this (in the form of Smartstack).

    For what it is worth, we faced a similar challenge in earlier services (mostly due to existing C/C++ applications) and we created what was called a "sidecar".  By sidecar, what I mean is a second process on each node/instance that did Cloud Service Fabric operations on behalf of the main process (the side-managed process).  Unfortunately those sidecars all went off and created one-offs for their particular service.  In this post, I'll describe a more general sidecar that doesn't force users to have these one-offs. Sidenote:  For those not familiar with sidecars, think of the motorcycle sidecar below.  Snoopy would be the main process with Woodstock being the sidecar process.  The main work on the instance would be the motorcycle (say serving your users' REST requests).  The operational control is the sidecar (say serving health checks and management plane requests of the operational platform).

    (tags: netflix sidecars architecture patterns smartstack netflixoss microservices soa)

Links for 2014-08-15

Published August 15, 2014

Links for 2014-08-14

Published August 14, 2014

Links for 2014-08-13

Published August 13, 2014

Links for 2014-08-12

Published August 12, 2014

Links for 2014-08-10

Published August 10, 2014

Links for 2014-08-09

Published August 9, 2014

Links for 2014-08-08

Published August 8, 2014

  • AWS Speed Test: What are the Fastest EC2 and S3 Regions?

    My god, this test is awful -- this is how NOT to test networked infrastructure. (1) testing from a single EC2 instance in each region; (2) uploading to a single test bucket for each test; (3) results don't include min/max or percentiles, just an averaged measurement for each test. FAIL

    (tags: fail testing networking performance ec2 aws s3 internet)

  • Hacker Redirects Traffic From 19 Internet Providers to Steal Bitcoins | Threat Level | WIRED

    'The attacker specifically targeted a collection of bitcoin mining “pools”–bitcoin-producing cooperatives in which users contribute their computers’ processing power and are rewarded with a cut of the resulting cryptocurrency the pool produces. The redirection technique tricked the pools’ participants into continuing to devote their processors to bitcoin mining while allowing the hacker to keep the proceeds. At its peak, according to the researchers’ measurements, the hacker’s scam was pocketing a flow of bitcoins and other digital currencies including dogecoin and worldcoin worth close to $9,000 a day. “With this kind of hijacking, you can quite easily grab a large collection of clients,” says Pat Litke, one of the Dell researchers. “It takes less than a minute, and you end up with a lot of mining traffic under your control.”' 'In total, Stewart and Litke were able to measure $83,000 worth of cryptocurrency stolen in the BGP attack [...] but the total haul could be larger'

    (tags: bitcoin mining fraud internet bgp routing security attacks hacking)

Links for 2014-08-07

Published August 7, 2014

Links for 2014-08-06

Published August 6, 2014

Links for 2014-08-05

Published August 5, 2014

Links for 2014-08-01

Published August 1, 2014

Links for 2014-07-31

Published July 31, 2014

  • UK private copying exception plans face possible legal action

    Under the proposed private copying exception, individuals in the UK would be given a new right to make a copy of copyrighted material they have lawfully and permanently acquired for their private use, provided it was not for commercial ends. Making a private copy of the material in these circumstances would not be an act of copyright infringement, although making a private copy of a computer program would still be prohibited under the plans. There is no mechanism envisaged in the draft legislation for rights holders to be specifically compensated for the act of private copying. This prompted the Joint Committee on Statutory Instruments (JCSI), tasked with scrutinising the proposals, to warn parliamentarians that the rules may be deemed to be in breach of EU copyright laws as a result of the lack of 'fair compensation' mechanism. [...] "We are disappointed that the private copying exception will be introduced without providing fair compensation for British songwriters, performers and other rights holders within the creative sector. A mechanism for fair compensation is a requirement of European law. In response we are considering our legal options," [UK Music] said.

    (tags: uk law copyright music copying private-copying personal infringement piracy transcoding backup)

  • Moominvalley Map Print | Magic Pony

    Lovely print! Shipping would be a bit crazy, though. There has to be an english-language print of one of Tove Jansson's maps on sale somewhere in Europe...

    (tags: prints moomins moominvalley maps hattifatteners magic-pony tove-jannson art)

Links for 2014-07-29

Published July 29, 2014

  • How to take over the computer of any JVM developer

    To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.

    (tags: jars dependencies java build clojure security mitm http proxies backdoors scala maven gradle)

  • Spain pushes for 'Google tax' to restrict linking

    The government wants to put a tax on linking on the internet. They say that if you want to link to some newspaper's content, you have to pay a tax. The primary targets of this law are Google News and other aggregators. It would be absurd enough just like that, but the law goes further: they declared it an "inalienable right" so even if I have a blog or a new small digital media publication and I want to let people freely link to my content, I can't opt-out--they are charging the levy, and giving it to the big press media. It was just the last and only way that the old traditional media companies can get some money from the government, and they strongly lobbied for it. The bill has passed in the Congress where the party in the government has majority (PP, Partido Popular) and it's headed to the Senate, where they have a majority also.

    (tags: spain stupidity law via:boingboing linking links web news google google-news newspapers old-media taxes)

  • Keyes New Starter Kit for Arduino Fans

    $53 for a reasonable-looking Arduino starter kit, from DealExtreme. cheap cheap! In the inimitable DX style:

    Keyes new beginner starter kit, pay more attention to beginners learning. Users can get rid of the difficult technological learning, from module used to quick start production.

    (tags: learning arduino hardware hacking robotics toys dealextreme tobuy)

Links for 2014-07-28

Published July 28, 2014

  • Check If A Hotel’s WiFi Sucks Before It’s Too Late

    http://www.hotelwifitest.com/ and http://speedspot.org/ .

    (tags: wifi hotels travel reviews techcrunch internet)

  • Collection Pipeline

    a nice summarisation of the state of pipe/stream-oriented collection operations in various languages, from Martin Fowler

    (tags: martin-fowler patterns coding ruby clojure streams pipelines pipes unix lambda fp java languages)

  • REST Commander: Scalable Web Server Management and Monitoring

    We dynamically monitor and manage a large and rapidly growing number of web servers deployed on our infrastructure and systems. However, existing tools present major challenges when making REST/SOAP calls with server-specific requests to a large number of web servers, and then performing aggregated analysis on the responses. We therefore developed REST Commander, a parallel asynchronous HTTP client as a service to monitor and manage web servers. REST Commander on a single server can send requests to thousands of servers with response aggregation in a matter of seconds. And yes, it is open-sourced at http://www.restcommander.com. Feature highlights: Click-to-run with zero installation; Generic HTTP request template supporting variable-based replacement for sending server-specific requests; Ability to send the same request to different servers, different requests to different servers, and different requests to the same server; Maximum concurrency control (throttling) to accommodate server capacity; Commander itself is also “as a service”: with its powerful REST API, you can define ad-hoc target servers, an HTTP request template, variable replacement, and a regular expression all in a single call. In addition, intuitive step-by-step wizards help you achieve the same functionality through a GUI.

    (tags: rest http clients load-testing ebay soap async testing monitoring)

  • South Downs litter picker has truck named after him - West Sussex County Times

    This is amazing. In http://www.newyorker.com/magazine/2014/06/30/stepping-out-3 , David Sedaris had written: 'in recognition of all the rubbish I’ve collected since getting my Fitbit, my local council is naming a garbage truck after me'; naturally, I assumed he was joking, but it looks like he wasn't:

    Horsham District Council has paid thanks to a volunteer who devotes a great deal of time and energy to walking many miles clearing litter from near where he lives as well as surrounding areas. David Sedaris litter picks in areas including Parham, Coldwaltham, Storrington and beyond. In recognition for all his fantastic work and dedication and as a token of Horsham District Council’s appreciation, the council has named one of their waste vehicles after him. The vehicle, bedecked with its bespoke ‘Pig Pen Sedaris’ sign was officially unveiled by the Lord-Lieutenant of West Sussex Mrs Susan Pyper at an outdoor ceremony on July 23.
    Best of all, the article utterly fails to mention who he is. Amazing. (via John Braine)

    (tags: via:john-braine funny david-sedaris litter uk horsham rubbish garbage cleaning volunteering walking)

Links for 2014-07-24

Published July 24, 2014

Links for 2014-07-23

Published July 23, 2014

  • This tree produces 40 different types of fruit

    An art professor from Syracuse University in the US, Van Aken grew up on a family farm before pursuing a career as an artist, and has combined his knowledge of the two to develop his incredible Tree of 40 Fruit.  In 2008, Van Aken learned that an orchard at the New York State Agricultural Experiment Station was about to be shut down due to a lack of funding. This single orchard grew a great number of heirloom, antique, and native varieties of stone fruit, and some of these were 150 to 200 years old. To lose this orchard would render many of these rare and old varieties of fruit extinct, so to preserve them, Van Aken bought the orchard, and spent the following years figuring out how to graft parts of the trees onto a single fruit tree. [...] Aken’s Tree of 40 Fruit looks like a normal tree for most of the year, but in spring it reveals a stunning patchwork of pink, white, red and purple blossoms, which turn into an array of plums, peaches, apricots, nectarines, cherries and almonds during the summer months, all of which are rare and unique varieties. 

    (tags: fruit art amazing food agriculture grafting orchards sam-van-aken farming)

Links for 2014-07-22

Published July 22, 2014

  • Metrics-Driven Development

    we believe MDD is equal parts engineering technique and cultural process. It separates the notion of monitoring from its traditional position of exclusivity as an operations thing and places it more appropriately next to its peers as an engineering process. Provided access to real-time production metrics relevant to them individually, both software engineers and operations engineers can validate hypotheses, assess problems, implement solutions, and improve future designs.
    Broken down into the following principles: 'Instrumentation-as-Code', 'Single Source of Truth', 'Developers Curate Visualizations and Alerts', 'Alert on What You See', 'Show me the Graph', 'Don’t Measure Everything (YAGNI)'. We do all of these at Swrve, naturally (a technique I happily stole from Amazon).

    (tags: metrics coding graphite mdd instrumentation yagni alerting monitoring graphs)

  • Auto Scale DynamoDB With Dynamic DynamoDB

    Nicely-packaged auto-scaler for DynamoDB

    (tags: dynamodb autoscaling scalability provisioning aws ec2 cloudformation)

Links for 2014-07-21

Published July 21, 2014