Skip to content

Category: Uncategorized

Links for 2015-02-11

Published February 11, 2015

  • Automating Tinder with Eigenfaces

    While my friends were getting sucked into "swiping" all day on their phones with Tinder, I eventually got fed up and designed a piece of software that automates everything on Tinder.
    This is awesome. (via waxy)

    (tags: via:waxy tinder eigenfaces machine-learning k-nearest-neighbour algorithms automation ai)

  • RateLimitedLogger

    Our latest open source release from Swrve Labs: an Apache-licensed, SLF4J-compatible, simple, fluent API for rate-limited logging in Java: 'A RateLimitedLog object tracks the rate of log message emission, imposes an internal rate limit, and will efficiently suppress logging if this is exceeded. When a log is suppressed, at the end of the limit period, another log message is output indicating how many log lines were suppressed. This style of rate limiting is the same as the one used by UNIX syslog; this means it should be comprehensible, easy to predict, and familiar to many users, unlike more complex adaptive rate limits.' We've been using this in production for months -- it's pretty nifty ;) Never fear your logs again!

    (tags: logs logging coding java open-source swrve slf4j rate-limiting libraries)

Links for 2015-02-10

Published February 10, 2015

Links for 2015-02-06

Published February 6, 2015

Links for 2015-02-05

Published February 5, 2015

Links for 2015-02-03

Published February 3, 2015

Links for 2015-02-02

Published February 2, 2015

  • A Quiet Defense of Patterns

    Marc Brooker: 'When it comes to building working software in the long term, the emotional pursuit of craft is not as important as the human pursuit of teamwork, or the intellectual pursuit of correctness. Patterns is one of the most powerful ideas we have. The critics may be right that it devalues the craft, but we would all do well to remember that the craft of software is a means, not an end.'

    (tags: marc-brooker design-patterns coding software teamwork)

  • One Hundred Miles of Solitude

    Via Walter, the best description of the appeal of Minecraft I've read:

    Minecraft is exceptionally good at intrinsic narrative. It recognises, preserves and rewards everything you do. It presses you to play frontiersman. A Minecraft world ends up dotted with torchlit paths, menhirs, landmarks, emergency caches. Here’s the hole where you dug stone for your first house. Here’s the causeway you built from your spawn point to a handy woodland. Here’s the crater in the landscape where the exploding monster took out you and your wheatfield at once. And, of course, here’s your enormous castle above a waterfall. There’s no utility in building anything bigger than a hut, but the temptations of architecture are irresistible. Minecraft isn’t so much a world generator as a screenshot-generator and a war-story generator. This is what will get the game the bulk of its critical attention, and deservedly so. That’s why I want to call attention to the extrinsic narrative. It’s minimal, implicit,  accidental and very powerful. It’s this: you wake alone beside an endless sea in a pristine, infinite wilderness. The world is yours. You can literally sculpt mountains, with time and effort. You’ll die and be reborn on the beach where you woke first. You’ll walk across the world forever and never see another face. You can build a whole empire of roads and palaces and beacon towers, and the population of that empire will only ever be you. When you leave, your towers will stand empty forever. I haven’t seen that surfaced in a game before. It’s strong wine.

    (tags: minecraft narrative gaming games story)

  • Backstage Blog - Prometheus: Monitoring at SoundCloud - SoundCloud Developers

    whoa, this is pretty excellent. The major improvement over a graphite-based system would be the multi-dimensional tagging of metrics, which we currently have to do by simply expanding the graphite metric's name to encompass all those dimensions and use searching at query time, inefficiently.

    (tags: monitoring soundcloud prometheus metrics service-metrics graphite alerting)

  • 'Prometheus instrumentation library for JVM applications'

    Good example of a clean java OSS release, from Soundcloud. will be copying bits of this myself soon...

    (tags: prometheus java libraries oss github sonatype maven releases)

  • Google Java Style

    A good set of basic, controversy-free guidelines for clean java code style

    (tags: style java google coding guidelines formatting coding-standards)

  • A Brief History of NSA Backdoors

    from 1946 to present

    (tags: nsa security backdoors sigint actel dual_ec_drbg crypto-ag crypto)

  • Study: You Can't Change an Anti-Vaxxer's Mind

    According to a major new study in the journal 'Pediatrics', trying to [persuade anti-vaxxers to vaccinate] may actually make the problem worse. The paper tested the effectiveness of four separate pro-vaccine messages, three of which were based very closely on how the Centers for Disease Control and Prevention (CDC) itself talks about vaccines. The results can only be called grim: Not a single one of the messages was successful when it came to increasing parents' professed intent to vaccinate their children. And in several cases the messages actually backfired, either increasing the ill-founded belief that vaccines cause autism or even, in one case, apparently reducing parents' intent to vaccinate.

    (tags: vaccination health measles mmr autism facts via:mrneutron stupidity cdc papers vaccines)

  • Coining "Dysguria"

    “dysaguria” is the perfect noun, and “dysagurian” is the perfect adjective, to describe the eponymous company in Dave Eggers’ The Circle. It’s not in the same league as Orwell, or Huxley, or Bradbury, or Burgess. But it does raise very important questions about what could possibly go wrong if one company controlled all the world’s information. In the novel, the company operates according to the motto “all that happens must be known”; and one of its bosses, Eamon Bailey, encourages everywoman employee Mae Holland to live an always-on (clear, transparent) life according the maxims “secrets are lies”, “sharing is caring”, and “privacy is theft”. Eggers’s debts to dystopian fiction are apparent. But, whereas writers like Orwell, Huxley, Bradbury, and Burgess were concerned with totalitarian states, Eggers is concerned with a totalitarian company. However, the noun “dystopia” and the adjective “dystopian” – perfect though they are for the terror of military/security authoritarianism in 1984, or Brave new World, or Farenheit 451, or A Clockwork Orange – do not to my mind encapsulate the nightmare of industrial/corporate tyranny in The Circle. On the other hand, “dysaguria” as a noun and “dysagurian” as an adjective, in my view really do capture the essence of that “frightening company”.

    (tags: dysaguria dystopia future sf authoritarianism surveillance the-circle google facebook)

  • A NetHack bot ascends!

    Via negatendo: 'I would like to share my excitement about the fact that after almost a year of development, an instance of my NetHack bot has finally managed to ascend a game for the first time without human interventions, wizard mode cheats or bones stuffing, and did so at the public server at acehack.de.' The bot is written in Clojure. Apparently 'pudding farming' did the trick...

    (tags: clojure via:negatendo pudding-farming games nethack bots)

Links for 2015-01-30

Published January 30, 2015

  • H.P. Lovecat

    Because there exists no method known to man, more terribly suited to expose the cosmic meaningless of existence than pairing the words of H.P. Lovecraft with seemingly delightful and charming pictures of adorable kittens.

    (tags: lovecraft cthulhu horror funny kittens cats images gif)

  • 8 gdb tricks you should know (Ksplice Blog)

    These are very good -- bookmarking for the next time I'm using gdb, probably about 3 years from now

    (tags: c debugging gdb c++ tips coding)

  • EFF’s Game Plan for Ending Global Mass Surveillance

    For years, we’ve been working on a strategy to end mass surveillance of digital communications of innocent people worldwide. Today we’re laying out the plan, so you can understand how all the pieces fit together—that is, how U.S. advocacy and policy efforts connect to the international fight and vice versa. Decide for yourself where you can get involved to make the biggest difference. This plan isn’t for the next two weeks or three months. It’s a multi-year battle that may need to be revised many times as we better understand the tools and authorities of entities engaged in mass surveillance and as more disclosures by whistleblowers help shine light on surveillance abuses.

    (tags: eff privacy nsa surveillance gchq law policy us-politics)

  • No POD

    This group aims to consolidate opposition, give clear information and support letter writing and information awareness against the Dept. of Education's Primary Online Database.

    (tags: pod ireland privacy data-protection children kids schools)

  • Apple Pay suffering fraud problems

    Fraud in Apple Pay will in time, come to be managed – but the fact that easily available PII can waylay best in class protection should give us all pause.

    (tags: fraud apple apple-pay pii identity-theft)

  • Excellent example of failed "anonymisation" of a dataset

    Fred Logue notes how this failed Mayo TD Michelle Mulherin:

    From recent reports it mow appears that the Department of Education is discussing anonymisation of the Primary Online Database with the Data Protection Commissioner. Well someone should ask Mayo TD Michelle Mulherin how anonymisation is working for her. The Sunday Times reports that Ms Mulherin was the only TD in the Irish parliament on the dates when expensive phone calls were made to a mobile number in Kenya. The details of the calls were released under the Freedom of Information Act in an “anonymised” database. While it must be said the fact that Ms Mulherin was the only TD present on those occasions does not prove she made the calls – the reporting in the press is now raising the possibility that it was her. From a data protection point of view this is a perfect example of the difficulty with anonymisation. Data protection rules apply to personal data which is defined as data relating to a living individual who is or can be identified from the data or from the data in conjunction with other information. Anonymisation is often cited as a means for processing data outside the scope of data protection law but as Ms Mulherin has discovered individuals can be identified using supposedly anonymised data when analysed in conjunction with other data. In the case of the mysterious calls to Kenya even though the released information was “anonymised” to protect the privacy of public representatives, the phone log used in combination with the attendance record of public representatives and information on social media was sufficient to identify individuals and at least raise evidence of association between individuals and certain phone calls. While this may be well and good in terms of accounting for abuses of the phone service it also has worrying implications for the ability of public representatives to conduct their business in private. The bottom line is that anonymisation is very difficult if not impossible as Ms Mulherin has learned to her cost. It certainly is a lot more complex than simply removing names and other identifying features from a single dataset. The more data that there is and the more diverse the sources the greater the risk that individuals can be identified from supposedly anonymised datasets.

    (tags: data anonymisation fred-logue ireland michelle-mulherin tds kenya data-protection privacy)

Links for 2015-01-26

Published January 26, 2015

Links for 2015-01-25

Published January 25, 2015

Links for 2015-01-24

Published January 24, 2015

  • How to Catch a Terrorist - The New Yorker

    This is spot on --

    By flooding the system with false positives, big-data approaches to counterterrorism might actually make it harder to identify real terrorists before they act. Two years before the Boston Marathon bombing, Tamerlan Tsarnaev, the older of the two brothers alleged to have committed the attack, was assessed by the city’s Joint Terrorism Task Force. They determined that he was not a threat. This was one of about a thousand assessments that the Boston J.T.T.F. conducted that year, a number that had nearly doubled in the previous two years, according to the Boston F.B.I. As of 2013, the Justice Department has trained nearly three hundred thousand law-enforcement officers in how to file “suspicious-activity reports.” In 2010, a central database held about three thousand of these reports; by 2012 it had grown to almost twenty-eight thousand. “The bigger haystack makes it harder to find the needle,” Sensenbrenner told me. Thomas Drake, a former N.S.A. executive and whistle-blower who has become one of the agency’s most vocal critics, told me, “If you target everything, there’s no target.”

    (tags: terrorism false-positives filtering detection jttf nsa fbi surveillance gchq)

  • Politwoops

    'All deleted tweets from politicians'. Great idea

    (tags: delete twitter politics politicians ireland social-media news)

  • Zoë Keating on getting a shitty deal from Google's new Music Key licensing

    The Youtube music service was introduced to me as a win win and they don’t understand why I don’t see it that way. “We are trying to create a new revenue stream on top of the platform that exists today.” A lot of people in the music industry talk about Google as evil. I don’t think they are evil. I think they, like other tech companies, are just idealistic in a way that works best for them. I think this because I used to be one of them. The people who work at Google, Facebook, etc can’t imagine how everything they make is not, like, totally awesome. If it’s not awesome for you it’s because you just don’t understand it yet and you’ll come around. They can’t imagine scenarios outside their reality and that is how they inadvertently unleash things like the algorithmic cruelty of Facebook’s yearly review (which showed me a picture I had posted after a doctor told me my husband had 6-8 weeks to live).

    (tags: google business music youtube zoe-keating music-key licensing tech)

  • Smash the Engine

    Jacobin Magazine on the revolutionary political allegory in "Snowpiercer": 'If Snowpiercer had merely told the tale of an oppressed working class rising up to seize power from an evil overlord, it would already have been an improvement over most of the political messages in mainstream cinema. There are all sorts of nice touches in its portrayal of a declining capitalism that can maintain its ideological legitimacy even when it literally has no more bullets in its guns. But the story Bong tells goes beyond that. It’s about the limitations of a revolution which merely takes over the existing social machinery rather than attempting to transcend it. '

    (tags: dystopia revolution snowpiercer movies marxism sf politics)

  • Debunking The Dangerous “If You Have Nothing To Hide, You Have Nothing To Fear”

    A great resource bookmark from Falkvinge.

    There are at least four good reasons to reject this argument solidly and uncompromisingly: The rules may change, it’s not you who determine if you’re guilty, laws must be broken for society to progress, and privacy is a basic human need.

    (tags: nsa politics privacy security surveillance gchq rick-falkvinge society)

Links for 2015-01-20

Published January 20, 2015

Links for 2015-01-19

Published January 19, 2015

  • carbon-c-relay

    A much better carbon-relay, written in C rather than Python. Linking as we've been using it in production for quite a while with no problems.

    The main reason to build a replacement is performance and configurability. Carbon is single threaded, and sending metrics to multiple consistent-hash clusters requires chaining of relays. This project provides a multithreaded relay which can address multiple targets and clusters for each and every metric based on pattern matches.

    (tags: graphite carbon c python ops metrics)

  • Surveillance of social media not way to fight terrorism – Minister

    Blanket surveillance of social media is not the solution to combating terrorism and the rights of the individual to privacy must be protected, Data Protection Minister Dara Murphy said on Monday. [He] said Ireland and the European Union must protect the privacy rights of individuals on social media. “Freedom of expression, freedom of movement, and the protection of privacy are core tenets of the European Union, which must be upheld.”

    (tags: dara-murphy data-protection privacy surveillance europe eu ireland social-media)

Links for 2015-01-18

Published January 18, 2015

  • Amazing comment from a random sysadmin who's been targeted by the NSA

    'Here's a story for you. I'm not a party to any of this. I've done nothing wrong, I've never been suspected of doing anything wrong, and I don't know anyone who has done anything wrong. I don't even mean that in the sense of "I pissed off the wrong people but technically haven't been charged." I mean that I am a vanilla, average, 9-5 working man of no interest to anybody. My geographical location is an accident of my birth. Even still, I wasn't accidentally born in a high-conflict area, and my government is not at war. I'm a sysadmin at a legitimate ISP and my job is to keep the internet up and running smoothly. This agency has stalked me in my personal life, undermined my ability to trust my friends attempting to connect with me on LinkedIn, and infected my family's computer. They did this because they wanted to bypass legal channels and spy on a customer who pays for services from my employer. Wait, no, they wanted the ability to potentially spy on future customers. Actually, that is still not accurate - they wanted to spy on everybody in case there was a potentially bad person interacting with a customer. After seeing their complete disregard for anybody else, their immense resources, and their extremely sophisticated exploits and backdoors - knowing they will stop at nothing, and knowing that I was personally targeted - I'll be damned if I can ever trust any electronic device I own ever again. You all rationalize this by telling me that it "isn't surprising", and that I don't live in the [USA,UK] and therefore I have no rights. I just have one question. Are you people even human?'

    (tags: nsa via:ioerror privacy spying surveillance linkedin sysadmins gchq security)

  • DRI’s Unchanged Position on Eircode

    'Broadly, they are satisfied with what we are doing' versus: 'We have deep concerns about the Eircode initiative… We want to state clearly that we are not at all ‘satisfied’ with the postcode that has been designed or the implementation proposals.'

    (tags: dri ireland eircode postcodes privacy data-protection quotes misrepresentation)

Links for 2015-01-17

Published January 17, 2015

  • Misogyny in the Valley

    The young women interns [in one story in this post] worked in a very different way. As I explored their notes, I noticed that ideas were expanded upon, not abandoned. Challenges were identified, but the male language so often heard in Silicon Valley conference rooms - “Well, let me tell you what the problem with that idea is….” - was not in the room.  These young women, without men to define the “appropriate business behavior,” used different behaviors and came up with a startling and valuable solution. They showed many of the values that exist outside of dominance-based leadership: strategic thinking, intuition, nurturing and relationship building, values-based decision-making and acceptance of other’s input. Women need space to be themselves at work. Until people who have created their success by worshipping at the temple of male behavior, like Sheryl Sandberg, learn to value alternate behaviors, the working world will remain a foreign and hostile culture to women. And if we do not continuously work to build corporate cultures where there is room for other behaviors, women will be cast from or abandoned in a world not of our making, where we continuously “just do not fit in,” but where we still must go to earn our livings.

    (tags: sexism misogyny silicon-valley tech work sheryl-sandberg business collaboration)

  • Are you better off running your big-data batch system off your laptop?

    Heh, nice trolling.

    Here are two helpful guidelines (for largely disjoint populations): If you are going to use a big data system for yourself, see if it is faster than your laptop. If you are going to build a big data system for others, see that it is faster than my laptop. [...] We think everyone should have to do this, because it leads to better systems and better research.

    (tags: graph coding hadoop spark giraph graph-processing hardware scalability big-data batch algorithms pagerank)

  • BBC uses RIPA terrorism laws to catch TV licence fee dodgers in Northern Ireland

    Give them the power, they'll use that power. 'A document obtained under Freedom of Information legislation confirms the BBC's use of RIPA in Northern Ireland. It states: "The BBC may, in certain circumstances, authorise under the Regulation of Investigatory Powers Act 2000 and Regulation of Investigatory Powers (British Broadcasting Corporation) Order 2001 the lawful use of detection equipment to detect unlicensed use of television receivers... the BBC has used detection authorised under this legislation in Northern Ireland."'

    (tags: ripa privacy bbc tv license-fee uk northern-ireland law scope-creep)

  • Australia tries to ban crypto research – by ACCIDENT • The Register

    Researchers are warned off [discussing] 512-bits-plus key lengths, systems “designed or modified to perform cryptanalytic functions, or “designed or modified to use 'quantum cryptography'”. [....] “an email to a fellow academic could land you a 10 year prison sentence”.
    https://twitter.com/_miw/status/556023024009224192 notes 'the DSGL 5A002 defines it as >512bit RSA, >512bit DH, >112 bit ECC and >56 bit symmetric ciphers; weak as fuck i say.'

    (tags: law australia crime crypto ecc rsa stupidity fail)

Links for 2015-01-16

Published January 16, 2015

  • A Case Study of Toyota Unintended Acceleration and Software Safety

    I drive a Toyota, and this is scary stuff. Critical software systems need to be coded with care, and this isn't it -- they don't even have a bug tracking system!

    Investigations into potential causes of Unintended Acceleration (UA) for Toyota vehicles have made news several times in the past few years. Some blame has been placed on floor mats and sticky throttle pedals. But, a jury trial verdict was based on expert opinions that defects in Toyota's Electronic Throttle Control System (ETCS) software and safety architecture caused a fatal mishap.  This talk will outline key events in the still-ongoing Toyota UA litigation process, and pull together the technical issues that were discovered by NASA and other experts. The results paint a picture that should inform future designers of safety critical software in automobiles and other systems.

    (tags: toyota safety realtime coding etcs throttle-control nasa code-review embedded)

Links for 2015-01-15

Published January 15, 2015

  • Group warns of postcode project dangers | Irish Examiner

    “We have spoken to the National Consumer Agency, logistics companies and Digital Rights Ireland, with which we have had an indepth conversation to see if there is anything in the proposal that might be considered to have an impact on anyone’s privacy. Broadly, they are satisfied with what we are doing,” [Patricia Cronin, head of the Department of Communications’ postcodes division] told the committee. However in his letter, [DRI's] O’Lachtnain said the group “want to state clearly that we are not at all ‘satisfied’ with the postcode that has been designed or the implementation proposals”.
    Some nerve!

    (tags: dri nca privacy patricia-cronin goverment postcodes eircode dpc ireland)

Links for 2015-01-14

Published January 14, 2015

  • Of Course 23andMe's Plan Has Been to Sell Your Genetic Data All Along

    Today, 23andMe announced what Forbes reports is only the first of ten deals with big biotech companies: Genentech will pay up to $60 million for access to 23andMe's data to study Parkinson's. You think 23andMe was about selling fun DNA spit tests for $99 a pop? Nope, it's been about selling your data all along.

    (tags: testing ethics dna genentech 23andme parkinsons diseases health privacy)

  • Facette

    Really nice time series dashboarding app. Might consider replacing graphitus with this...

    (tags: time-series data visualisation graphs ops dashboards facette)

  • Getting good cancer care through 3D printing

    This is pretty incredible.

    Balzer downloaded a free software program called InVesalius, developed by a research center in Brazil to convert MRI and CT scan data to 3D images. He used it to create a 3D volume rendering from Scott’s DICOM images, which allowed him to look at the tumor from any angle. Then he uploaded the files to Sketchfab and shared them with neurosurgeons around the country in the hope of finding one who was willing to try a new type of procedure. Perhaps unsurprisingly, he found the doctor he was looking for at UPMC, where Scott had her thyroid removed. A neurosurgeon there agreed to consider a minimally invasive operation in which he would access the tumor through Scott’s left eyelid and remove it using a micro drill. Balzer had adapted the volume renderings for 3D printing and produced a few full-size models of the front section of Scott’s skull on his MakerBot. To help the surgeon vet his micro drilling idea and plan the procedure, Balzer packed up one of the models and shipped it off to Pittsburgh.

    (tags: diy surgery health cancer tumours medicine 3d-printing 3d scanning mri dicom)

Links for 2015-01-13

Published January 13, 2015

Links for 2015-01-12

Published January 12, 2015

Links for 2015-01-11

Published January 11, 2015

Links for 2015-01-10

Published January 10, 2015

Links for 2015-01-09

Published January 9, 2015

  • A World Transfixed by Screens - The Atlantic

    Excellent "In Focus" this week -- 'The continued massive growth of connected mobile devices is shaping not only how we communicate with each other, but how we look, behave, and experience the world around us. Smartphones and other handheld devices have become indispensable tools, appendages held at arm's length to record a scene or to snap a selfie. Recent news photos show refugees fleeing war-torn regions holding up their phones as prized possessions to be saved, and relatives of victims lost to a disaster holding up their smartphones to show images of their loved ones to the press. Celebrity selfies, people alone in a crowd with their phones, events obscured by the very devices used to record that event, the brightly lit faces of those bent over their small screens, these are some of the scenes depicted below.'

    (tags: mobile photography in-focus alan-taylor the-atlantic phones selfies pictures)

  • "Incremental Stream Processing using Computational Conflict-free Replicated Data Types" [paper]

    'Unlike existing alternatives, such as stream processing, that favor the execution of arbitrary application code, we want to capture much of the processing logic as a set of known operations over specialized Computational CRDTs, with particular semantics and invariants, such as min/max/average/median registers, accumulators, top-N sets, sorted sets/maps, and so on. Keeping state also allows the system to decrease the amount of propagated information. Preliminary results obtained in a single example show that Titan has an higher throughput when compared with state of the art stream processing systems.'

    (tags: crdt distributed stream-processing replication titan papers)

Links for 2015-01-07

Published January 7, 2015

Links for 2015-01-06

Published January 6, 2015

  • Mantis: Netflix's Event Stream Processing System

    Rx/reactive in style, autoscaling, support for queue/broker-based strong consistency as well as TCP-based lossy delivery

    (tags: netflix rx reactive autoscaling mantis stream-processing)

  • Bad Kids Jokes

    'I now a man with a wooden leg named sea what was the name of the other leg SAND'

    (tags: funny humor kids jokes humour)

  • The Hit Team

    Fergal Crehan's new gig -- good idea!

    The Hit Team helps you fight back against leaked photos and videos, internet targeting and revenge porn.

    (tags: revenge-porn revenge law privacy porn leaks photos videos images selfies)

  • F1: A Distributed SQL Database That Scales

    Beyond the interesting-enough stuff about scalability in a distributed SQL store, there's this really nifty point about avoiding the horrors of the SQL/ORM impedance mismatch:

    At Google, Protocol Buffers are ubiquitous for data storage and interchange between applications. When we still had a MySQL schema, users often had to write tedious and error-prone transformations between database rows and in-memory data structures. Putting protocol buffers in the schema removes this impedance mismatch and gives users a universal data structure they can use both in the database and in application code…. Protocol Buffer columns are more natural and reduce semantic complexity for users, who can now read and write their logical business objects as atomic units, without having to think about materializing them using joins across several tables.
    This is something that pretty much any store can already adopt. Go protobufs. (or Avro, etc.) Also, I find this really neat, and I hope this idea is implemented elsewhere soon: asynchronous schema updates:
    Schema changes are applied asynchronously on multiple F1 servers. Anomalies are prevented by the use of a schema leasing mechanism with support for only current and next schema versions; and by subdividing schema changes into multiple phases where consecutive pairs of changes are mutually compatible and cannot cause anomalies.

    (tags: schema sql f1 google papers orm protobuf)

Links for 2015-01-05

Published January 5, 2015

  • Avleen Vig on distributed engineering teams

    This is a really excellent post on the topic, rebutting Paul Graham's Bay-Area-centric thoughts on the topic very effectively. I've worked in both distributed and non-distributed, as well as effective and ineffective teams ;), and Avleen's thoughts are very much on target.

    I've been involved in the New York start up scene since I joined Etsy in 2010. Since that time, I've seen more and more companies there embrace having distributed teams. Two companies I know which have risen to the top while doing this have been Etsy and DigitalOcean. Both have exceptional engineering teams working on high profile products used by many, many people around the world. There are certainly others outside New York, including Automattic, GitHub, Chef Inc, Puppet... the list goes on. So how did this happen? And why do people continue to insist that distributed teams lower performance, and are a bad idea? Partly because we've done a poor job of showing our industry how to be successful at it, and partly because it's hard. Having successful distributed teams requires special skills from management, which arent't easily learned until you have to manage a distributed team. Catch 22.

    (tags: business culture management communication work distributed-teams avleen-vig engineering)

  • Hack workaround to get JVM thread priorities working on Linux

    As used in Cassandra ( http://grokbase.com/t/hbase/dev/13bf9kezes/about-xx-threadprioritypolicy-42 )!

    if you just set the "ThreadPriorityPolicy" to something else than the legal values 0 or 1, [...] a slight logic bug in Sun's JVM code kicks in, and thus sets the policy to be as if running with root - thus you get exactly what one desire. The operating system, Linux, won't allow priorities to be heightened above "Normal" (negative nice value), and thus just ignores those requests (setting it to normal instead, nice value 0) - but it lets through the requests to set it lower (setting the nice value to some positive value).

    (tags: cassandra thread-priorities threads java jvm linux nice hacks)

Links for 2015-01-04

Published January 4, 2015

  • Amiko Alien2 / Enigma Discussion Thread - boards.ie

    Enigma is a Linux based alternative to the default Spark operating system on these boxes. Enigma is a more customisable OS and provides the ability to add plugins which can accomplish many tasks enabling users to have a box which might look and perform like a Sky box, giving a 7 day EPG and an alternative to series link.
    Looks like a pretty solid hacker community...

    (tags: alien2 tv enigma dvr freeview saorview pvr)

  • Hague reassures MPs on Office 365 data storage as Microsoft ordered to hand over email data

    William Hague, the leader of the House of Commons, has responded to concerns raised by an MP about the security of parliamentary data stored on Microsoft’s Cloud-based servers in Europe. “The relevant servers are situated in the Republic of Ireland and the Netherlands, both being territories covered by the EC Data Protection Directive," William Hague wrote in a letter to John Hemming, MP for Birmingham Yardley. "Any access by US authorities to such data would have to be by way of mutual legal assistance arrangements with those countries.” [...] John Hemming MP told Computer Weekly Hague’s reassurances carried little weight in the face of aggressive legal action by the US government.  “The Microsoft case makes it clear that, in the end, the fact that Microsoft is a US company legally trumps the European Data Protection Directive [...] and where [the letter says] the US authorities could not exercise a right of search and seizure on an extraterritorial basis, well, they are doing that, in America, today.”
    Sounds like they didn't think that through...

    (tags: mail privacy parliament office-365 microsoft mlat surveillance)

Links for 2015-01-03

Published January 3, 2015

Links for 2015-01-03

Published January 3, 2015

Links for 2015-01-02

Published January 2, 2015

  • The open-office trend is destroying the workplace

    Wow, where has this person been for the past 20 years that they haven't had to encounter this? I can only imagine having a private office, tbh.

    my personal performance at work has hit an all-time low. Each day, my associates and I are seated at a table staring at each other, having an ongoing 12-person conversation from 9 a.m. to 5 p.m.  It’s like being in middle school with a bunch of adults. Those who have worked in private offices for decades have proven to be the most vociferous and rowdy. They haven’t had to consider how their loud habits affect others, so they shout ideas at each other across the table and rehash jokes of yore. As a result, I can only work effectively during times when no one else is around, or if I isolate myself in one of the small, constantly sought-after, glass-windowed meeting rooms around the perimeter.

    (tags: business office productivity work desks open-plan)

Links for 2015-01-02

Published January 2, 2015

  • The open-office trend is destroying the workplace

    Wow, where has this person been for the past 20 years that they haven't had to encounter this? I can only imagine having a private office, tbh.

    my personal performance at work has hit an all-time low. Each day, my associates and I are seated at a table staring at each other, having an ongoing 12-person conversation from 9 a.m. to 5 p.m.  It’s like being in middle school with a bunch of adults. Those who have worked in private offices for decades have proven to be the most vociferous and rowdy. They haven’t had to consider how their loud habits affect others, so they shout ideas at each other across the table and rehash jokes of yore. As a result, I can only work effectively during times when no one else is around, or if I isolate myself in one of the small, constantly sought-after, glass-windowed meeting rooms around the perimeter.

    (tags: business office productivity work desks open-plan)

Links for 2014-12-28

Published December 28, 2014

  • 'Uncertain: A First-Order Type for Uncertain Data' [paper, PDF]

    'Emerging applications increasingly use estimates such as sensor data (GPS), probabilistic models, machine learning, big data, and human data. Unfortunately, representing this uncertain data with discrete types (floats, integers, and booleans) encourages developers to pretend it is not probabilistic, which causes three types of uncertainty bugs. (1) Using estimates as facts ignores random error in estimates. (2) Computation compounds that error. (3) Boolean questions on probabilistic data induce false positives and negatives. This paper introduces Uncertain, a new programming language abstraction for uncertain data. We implement a Bayesian network semantics for computation and conditionals that improves program correctness. The runtime uses sampling and hypothesis tests to evaluate computation and conditionals lazily and efficiently. We illustrate with sensor and machine learning applications that Uncertain improves expressiveness and accuracy.' (via Tony Finch)

    (tags: uncertainty estimation types strong-typing coding probability statistics machine-learning sampling via:fanf)

Links for 2014-12-27

Published December 27, 2014

  • Why Airlines Want to Make You Suffer

    'The fee [airline pricing] model comes with systematic costs that are not immediately obvious. Here’s the thing: in order for fees to work, there needs be something worth paying to avoid. That necessitates, at some level, a strategy that can be described as “calculated misery.” Basic service, without fees, must be sufficiently degraded in order to make people want to pay to escape it. And that’s where the suffering begins.'

    (tags: travel airlines pricing fees economy consumer jetblue)

  • A Virtual Machine in Excel

    'Ádám was trying his hand at a problem in Excel, but the official rules prohibit the use of Excel macros. In a daze, he came up with one of the most clever uses of Excel: building an assembly interpreter with the most popular spreadsheet program. This is a virtual Harvard architecture machine without writable RAM; the stack is only lots and lots of IFs.'

    (tags: vms excel hacks spreadsheets coding)

Links for 2014-12-22

Published December 22, 2014

  • coz

    A causal profiler for C++.

    Causal profiling is a novel technique to measure optimization potential. This measurement matches developers' assumptions about profilers: that optimizing highly-ranked code will have the greatest impact on performance. Causal profiling measures optimization potential for serial, parallel, and asynchronous programs without instrumentation of special handling for library calls and concurrency primitives. Instead, a causal profiler uses performance experiments to predict the effect of optimizations. This allows the profiler to establish causality: "optimizing function X will have effect Y," exactly the measurement developers had assumed they were getting all along.
    I can see this being a good technique to stochastically discover race conditions and concurrency bugs, too.

    (tags: optimization c++ performance coding profiling speed causal-profilers)

  • Spark 1.2 released

    This is the version with the superfast petabyte-sort record:

    Spark 1.2 includes several cross-cutting optimizations focused on performance for large scale workloads. Two new features Databricks developed for our world record petabyte sort with Spark are turned on by default in Spark 1.2. The first is a re-architected network transfer subsystem that exploits Netty 4’s zero-copy IO and off heap buffer management. The second is Spark’s sort based shuffle implementation, which we’ve now made the default after significant testing in Spark 1.1. Together, we’ve seen these features give as much as 5X performance improvement for workloads with very large shuffles.

    (tags: spark sorting hadoop map-reduce batch databricks apache netty)

  • The VATMOSS debacle: does the "manual email" loophole work?

    As the 1 January deadline gallops towards the EU, microbusinesses desperate to stay open without breaking the law try to find out, "Can I email stuff out instead?" Well... Yes. - No - It depends - and simultaneously yes AND no, according to Schrödinger’s VAT. So that's clear, then.

    (tags: vat vatmoss eu tax fiasco email microbusiness sme)

  • One artist closing up their Bandcamp site due to new VATMOSS laws

    Nice work, EU

    (tags: eu law tax vat vatmoss matt-stevens bandcamp music downloads)

Links for 2014-12-21

Published December 21, 2014

Links for 2014-12-19

Published December 19, 2014

  • AN OFFER TO SONY FROM 2600

    To demonstrate that hackers have no interest in suppressing speech, quashing controversy, or being intimidated by vague threats, we ask that Sony allow the hacker community to distribute "The Interview" for them on the 25th of December. Now, we're aware that Sony may refer to this distribution method as piracy, but in this particular case, it may well prove to be the salvation of the motion picture industry. By freely offering the film online, millions of people will get to see it and decide for themselves if it has any redeeming qualities whatsoever - as opposed to nobody seeing it and the studios writing it off as a total loss. Theaters would be free from panic as our servers would become the target of any future vague threats (and we believe Hollywood will be most impressed with how resilient peer-to-peer distribution can be in the face of attacks). Most importantly, we would be defying intimidation, something the motion picture industry doesn't quite have a handle on, which is surprising considering how much they've relied upon it in the past.

    (tags: 2600 funny hackers security sony north-korea the-interview movies piracy)

Links for 2014-12-18

Published December 18, 2014

Links for 2014-12-17

Published December 17, 2014

  • 'Machine Learning: The High-Interest Credit Card of Technical Debt' [PDF]

    Oh god yes. This is absolutely spot on, as you would expect from a Google paper -- at this stage they probably have accumulated more real-world ML-at-scale experience than anywhere else. 'Machine learning offers a fantastically powerful toolkit for building complex systems quickly. This paper argues that it is dangerous to think of these quick wins as coming for free. Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning. The goal of this paper is highlight several machine learning specific risk factors and design patterns to be avoided or refactored where possible. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, changes in the external world, and a variety of system-level anti-patterns. [....] 'In this paper, we focus on the system-level interaction between machine learning code and larger systems as an area where hidden technical debt may rapidly accumulate. At a system-level, a machine learning model may subtly erode abstraction boundaries. It may be tempting to re-use input signals in ways that create unintended tight coupling of otherwise disjoint systems. Machine learning packages may often be treated as black boxes, resulting in large masses of “glue code” or calibration layers that can lock in assumptions. Changes in the external world may make models or input signals change behavior in unintended ways, ratcheting up maintenance cost and the burden of any debt. Even monitoring that the system as a whole is operating as intended may be difficult without careful design. Indeed, a remarkable portion of real-world “machine learning” work is devoted to tackling issues of this form. Paying down technical debt may initially appear less glamorous than research results usually reported in academic ML conferences. But it is critical for long-term system health and enables algorithmic advances and other cutting-edge improvements.'

    (tags: machine-learning ml systems ops tech-debt maintainance google papers hidden-costs development)

  • The FBI Used the Web's Favorite Hacking Tool to Unmask Tor Users | WIRED

    Since Operation Torpedo [use of a Metasploit side project], there’s evidence the FBI’s anti-Tor capabilities have been rapidly advancing. Torpedo was in November 2012. In late July 2013, computer security experts detected a similar attack through Dark Net websites hosted by a shady ISP called Freedom Hosting—court records have since confirmed it was another FBI operation. For this one, the bureau used custom attack code that exploited a relatively fresh Firefox vulnerability—the hacking equivalent of moving from a bow-and-arrow to a 9-mm pistol. In addition to the IP address, which identifies a household, this code collected the MAC address of the particular computer that infected by the malware. “In the course of nine months they went from off the shelf Flash techniques that simply took advantage of the lack of proxy protection, to custom-built browser exploits,” says Soghoian. “That’s a pretty amazing growth … The arms race is going to get really nasty, really fast.”

    (tags: fbi tor police flash security privacy anonymity darknet wired via:bruces)

Links for 2014-12-16

Published December 16, 2014

  • Digital Rights Ireland files Amicus Brief in Microsoft v USA with Liberty and ORG

    Microsoft -v- USA is an important ongoing case, currently listed for hearing in 2015 before the US Federal Court of Appeal of the 2nd Circuit. However, as the case centres around the means by which NY law enforcement are seeking to access data of an email account which resides in Dublin, it is also crucially significant to Ireland and the rest of the EU. For that reason, Digital Rights Ireland instructed us to file an Amicus Brief in the US case, in conjunction with the global law firm of White & Case, who have acted pro bono in their representation. Given the significance of the case for the wider EU, both Liberty and the Open Rights Group in the UK have joined Digital Rights Ireland as amici on this brief. We hope it will be of aid to the US court in assessing the significance of the order being appealed by Microsoft for EU citizens and European states, in the light of the existing US and EU Mutual Legal Assistance Treaty.

    (tags: amicus-briefs law us dri microsoft mlats org liberty eu privacy)

Links for 2014-12-15

Published December 15, 2014

Links for 2014-12-13

Published December 13, 2014

  • littleBits Synth Kit

    Wow, this looks cool. $159

    littleBits and Korg have demystified a traditional analog synthesizer, making it super easy for novices and experts alike to create music. connects to speakers, computers and headphones. can be used to make your own instruments. fits into the littleBits modular system for infinite combos of audio, visual and sensory experiences

    (tags: diy hardware music littlebits gadgets make analog synths)

Links for 2014-12-12

Published December 12, 2014

Links for 2014-12-11

Published December 11, 2014

Links for 2014-12-10

Published December 10, 2014

  • AWS Key Management Service Cryptographic Details

    "AWS Key Management Service (AWS KMS) provides cryptographic keys and operations scaled for the cloud. AWS KMS keys and functionality are used by other AWS cloud services, and you can use them to protect user data in your applications that use AWS. This white paper provides details on the cryptographic operations that are executed within AWS when you use AWS KMS."

    (tags: white-papers aws amazon kms key-management crypto pdf)

Links for 2014-12-09

Published December 9, 2014

  • Aurora for MySQL is coming

    some good details of Aurora innards

    (tags: mysql databases aurora aws ec2 sql storage transactions replication)

  • If Eventual Consistency Seems Hard, Wait Till You Try MVCC

    ex-Percona MySQL wizard Baron Schwartz, noting that MVCC as implemented in common SQL databases is not all that simple or reliable compared to big bad NoSQL Eventual Consistency:

    Since I am not ready to assert that there’s a distributed system I know to be better and simpler than eventually consistent datastores, and since I certainly know that InnoDB’s MVCC implementation is full of complexities, for right now I am probably in the same position most of my readers are: the two viable choices seem to be single-node MVCC and multi-node eventual consistency. And I don’t think MVCC is the simpler paradigm of the two.

    (tags: nosql concurrency databases mysql riak voldemort eventual-consistency reliability storage baron-schwartz mvcc innodb postgresql)

  • Scaling email transparency

    This is quite interesting/weird -- Stripe's protocol for mass-CCing email as they scale up the company, based around http://en.wikipedia.org/wiki/Civil_inattention

    (tags: communication culture email management stripe cc transparency civil-inattention)

  • Shanley Kane of Model View Culture Challenges a “Corrupt” Silicon Valley | MIT Technology Review

    If their interests were better serving the world, using technology as a force for social justice, and equitably distributing technology wealth to enrich society … sure, they’d be acting against their interests. But the reality is that tech companies centralize power and wealth in a small group of privileged white men. When that’s the goal, then exploiting the labor of marginalized people and denying them access to power and wealth is 100 percent in line with the endgame. A more diverse tech industry would be better for its workers and everyone else, but it would be worse for the privileged white men at the top of it, because it would mean they would have to give up their monopoly on money and power. And they will fight that with everything they’ve got, which is why we see barriers to equality at every level of the industry.

    (tags: culture feminism tech mit-tech-review shanley-kane privilege vcs silicon-valley)

  • Announcing Snappy Ubuntu

    Awesome! I was completely unaware this was coming down the pipeline.

    A new, transactionally updated Ubuntu for the cloud. Ubuntu Core is a new rendition of Ubuntu for the cloud with transactional updates. Ubuntu Core is a minimal server image with the same libraries as today’s Ubuntu, but applications are provided through a simpler mechanism. The snappy approach is faster, more reliable, and lets us provide stronger security guarantees for apps and users — that’s why we call them “snappy” applications. Snappy apps and Ubuntu Core itself can be upgraded atomically and rolled back if needed — a bulletproof approach to systems management that is perfect for container deployments. It’s called “transactional” or “image-based” systems management, and we’re delighted to make it available on every Ubuntu certified cloud.

    (tags: ubuntu linux packaging snappy ubuntu-core transactional-updates apt docker ops)

  • Dan McKinley :: Thoughts on the Technical Track

    Ouch. I think Amazon did a better job of the Technical Track concept than this, at least

    (tags: engineering management technical-track principal-engineer career work)

  • OSTree

    "git for operating system binaries". OSTree is a tool for managing bootable, immutable, versioned filesystem trees. It is not a package system; nor is it a tool for managing full disk images. Instead, it sits between those levels, offering a blend of the advantages (and disadvantages) of both. You can use any build system you like to place content into it on a build server, then export an OSTree repository via static HTTP. On each client system, "ostree admin upgrade" can incrementally replicate that content, creating a new root for the next reboot. This provides fully atomic upgrades. Any changes made to /etc are propagated forwards, and all local state in /var is shared. A key goal of the project is to complement existing package systems like RPM and Debian packages, and help further their evolution. In particular for example, RPM-OSTree (linked below) has as a goal a hybrid tree/package model, where you replicate a base tree via OSTree, and then add packages on top.

    (tags: os gnome git linux immutable deployment packaging via:fanf)

Links for 2014-12-06

Published December 6, 2014

  • State sanctions foreign phone and email tapping

    Well, this stinks.

    Foreign law enforcement agencies will be allowed to tap Irish phone calls and intercept emails under a statutory instrument signed into law by Minister for Justice Frances Fitzgerald. Companies that object or refuse to comply with an intercept order could be brought before a private “in camera” court. The legislation, which took effect on Monday, was signed into law without fanfare on November 26th, the day after documents emerged in a German newspaper indicating the British spy agency General Communications Headquarters (GCHQ) had directly tapped undersea communications cables between Ireland and Britain for years.

    (tags: ireland law gchq surveillance mlats phone-tapping)

  • "Looks like Chicago PD had a stingray out at the Eric Garner protest last night"

    Your tax dollars at work: Spying on people just because they demand that the government's agents stop killing black people. [...] Anonymous has released a video featuring what appear to be Chicago police radio transmissions revealing police wiretapping of organizers' phones at the protests last night the day after Thanksgiving, perhaps using a stingray. The transmissions pointing to real-time wiretapping involve the local DHS-funded spy 'fusion' center.

    (tags: imsi-catcher stingray surveillance eric-garner protests privacy us-politics anonymous chicago police wiretapping dhs)

  • When data gets creepy: the secrets we don’t realise we’re giving away | Technology | The Guardian

    Very good article around the privacy implications of derived and inferred aggregate metadata from Ben Goldacre.

    We are entering an age – which we should welcome with open arms – when patients will finally have access to their own full medical records online. So suddenly we have a new problem. One day, you log in to your medical records, and there’s a new entry on your file: “Likely to die in the next year.” We spend a lot of time teaching medical students to be skilful around breaking bad news. A box ticked on your medical records is not empathic communication. Would we hide the box? Is that ethical? Or are “derived variables” such as these, on a medical record, something doctors should share like anything else?

    (tags: advertising ethics privacy security law data aggregation metadata ben-goldacre)

  • Stellar/Ripple suffer a failure of their consensus system, resulting in a split-brain failure

    Prof. Mazières’s research indicated some risk that consensus could fail, though we were nor certain if the required circumstances for such a failure were realistic. This week, we discovered the first instance of a consensus failure. On Tuesday night, the nodes on the network began to disagree and caused a fork of the ledger. The majority of the network was on ledger chain A. At some point, the network decided to switch to ledger chain B. This caused the roll back of a few hours of transactions that had only been recorded on chain A. We were able to replay most of these rolled back transactions on chain B to minimize the impact. However, in cases where an account had already sent a transaction on chain B the replay wasn’t possible.

    (tags: consensus distcomp stellar ripple split-brain postmortems outages ledger-fork payment)

  • the "Unknown Pleasures" cover, emulated in Mathematica

    In July 1967, astronomers at the Cavendish Laboratory in Cambridge, observed an unidentified radio signal from interstellar space, which flashed periodically every 1.33730 seconds. This object flashed with such regularity that it was accurate enough to be used as a clock and only be off by one part in a hundred million. It was eventually determined that this was the first discovery of a pulsar, CP-1919.  This is an object that has about the same mass as the Sun, but is the size of the San Francisco Bay at its widest (~20 kilometers) that is rotating so fast that its emitting a beam of light towards Earth like a strobing light house! Pulsars are neutron stars that are formed from the remnants of a massive star when it experiences stellar death. A hand drawn graph plotted in the style of a waterfall plot, in the Cambridge Encyclopedia of Astronomy, later became renown for its use on the cover of the album "Unknown Pleasures"  by 1970s English band Joy Division.
    The entire blog at http://intothecontinuum.tumblr.com/ is pretty great. Lots of nice mathematical animated GIFs, accompanied by Mathematica source and related ponderings.

    (tags: maths gifs animation art unknown-pleasures mathematica cp-1919 pulsars astronomy joy-division waterfall-plots cambridge blogs)

Links for 2014-12-05

Published December 5, 2014

  • Pubs With A Fire

    'Pubs & Bars With Raging Fires in Dublin'. This is important!

    (tags: pubs dublin fires fireplaces beer christmas)

  • Aurora for MySQL is coming

    'Anurag@AWS posts a quite interesting comment on Aurora failover: We asynchronously write to 6 copies and ack the write when we see four completions. So, traditional 4/6 quorums with synchrony as you surmised. Now, each log record can end up with a independent quorum from any other log record, which helps with jitter, but introduces some sophistication in recovery protocols. We peer to peer to fill in holes. We also will repair bad segments in the background, and downgrade to a 3/4 quorum if unable to place in an AZ for any extended period. You need a pretty bad failure to get a write outage.' (via High Scalability)

    (tags: via:highscalability mysql aurora failover fault-tolerance aws replication quorum)

Links for 2014-12-02

Published December 2, 2014

Links for 2014-12-01

Published December 1, 2014

  • Speeding up Rails 4.2

    Reading between the lines, it looks like Rails 4 is waaay slower than 3....

    (tags: rails ruby performance profiling discourse)

  • Day 1 - Docker in Production: Reality, Not Hype

    Good Docker info from Bridget Kromhout, on their production and dev usage of Docker at DramaFever. lots of good real-world tips

    (tags: docker ops boot2docker tips sysadvent)

  • Lost avant-garde painting found in Stuart Little’s living room

    Two years later, he heard from Lisa S., an assistant set designer on [the movie] Stuart Little. She had bought the painting for $500 from an antiques store in Pasadena specifically for the movie because she thought its cool elegance was perfectly suited for the Little’s New York City apartment. Lisa S. had tracked it down in another warehouse and purchased it from Sony just because she liked it so much. When she contacted Barki, she had no idea of the history of the painting hanging on her bedroom wall. After Barki visited the painting in person and confirmed its identity, Lisa sold it to a private collector. That collector has now been persuaded to sell it in Hungary. It will go up for auction at the Virag Judit Art Gallery in Budapest on December 13th with a starting price of 110,000 euros ($160,000). Gergely Barki won’t make a dime off of his discovery, but he will have a great story to tell in his biography of the artist.

    (tags: stuart-little art history hungary pasadena movies set-design antiques robert-bereny post-impressionism)

Links for 2014-11-30

Published November 30, 2014

  • rjbs's rubric: In Soviet Minecraft, server op you!

    wow, that is too much effort for a 7-year-old's Minecraft server ;) Very impressive

    (tags: minecraft game-servers kids teleport gaming rjbs perl)

  • Rust borrow and lifetimes

    How Rust avoids GC overhead using it's "borrow" system:

    Rust achieves memory safety without GC by using a sophiscated borrow system. For any resource (stack memory, heap memory, file handle and so on), there is exactly one owner which takes care of its resource deallocation, if needed. You may create new bindings to refer to the resource using & or &mut, which is called a borrow or mutable borrow. The compiler ensures all owners and borrowers behave correctly.

    (tags: languages rust gc borrow lifecycle stack heap allocation)

  • Java for Everything

    Actually, I'm really agreeing with a lot of this. Particularly this part:

    Programmers will cringe at writing some kind of command dispatch list: if command = "up": up() elif command = "status": status() elif command = "revert": revert() ... so they’ll go off and write some introspecting auto-dispatch cleverness, but that takes longer to write and will surely confuse future readers who’ll wonder how the heck revert() ever gets called. Yet the programmer will incorrectly feel as though he saved himself time. This is the trap of the dynamic language. It feels like you’re being more productive, but aside from the first 10 minutes of a new program, you’re not. Just write the stupid dispatch manually and get on with the real work.
    I've also gone right off dynamic languages for any kind of non-toy work. Mind you he needs to get around to ditching Vim for a proper IDE. That's the key thing that makes coding in a statically-typed language really pleasant -- when graphical refactoring becomes easy and usable, and errors are visible as you type them...

    (tags: java coding static-typing python unit-tests)

  • Facebook Fabric Networking Deconstructed

    whoa, this is incredibly in-depth

    (tags: facebook datacenter networking clos-networks infrastructure networks fat-tree)

Links for 2014-11-28

Published November 28, 2014

  • OS X doesn't support 'ndots' DNS resolution

    "ping foo.bar" will not append the "search" domains configured in /etc/resolv.conf. Apparently this has been broken since OS X Lion, no sign of a fix. Nice work Apple

    (tags: apple fail bugs resolv dns domains osx)

  • TCP incast

    a catastrophic TCP throughput collapse that occurs as the number of storage servers sending data to a client increases past the ability of an Ethernet switch to buffer packets. In a clustered file system, for example, a client application requests a data block striped across several storage servers, issuing the next data block request only when all servers have responded with their portion (Figure 1). This synchronized request workload can result in packets overfilling the buffers on the client's port on the switch, resulting in many losses. Under severe packet loss, TCP can experience a timeout that lasts a minimum of 200ms, determined by the TCP minimum retransmission timeout (RTOmin).

    (tags: incast networking performance tcp bandwidth buffering switch ethernet capacity)

  • Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale | Engineering Blog | Facebook Code

    Excellent real-world war story from Facebook -- a long-running mystery bug was eventually revealed to be a combination of edge-case behaviours across all the layers of the networking stack, from L2 link aggregation at the agg-router level, up to the L7 behaviour of the MySQL client connection pool.

    Facebook collocates many of a user’s nodes and edges in the social graph. That means that when somebody logs in after a while and their data isn’t in the cache, we might suddenly perform 50 or 100 database queries to a single database to load their data. This starts a race among those queries. The queries that go over a congested link will lose the race reliably, even if only by a few milliseconds. That loss makes them the most recently used when they are put back in the pool. The effect is that during a query burst we stack the deck against ourselves, putting all of the congested connections at the top of the deck.

    (tags: architecture debugging devops facebook layer-7 mysql connection-pooling aggregation networking tcp-stack)

  • "Macaroons" for fine-grained secure database access

    Macaroons are an excellent fit for NoSQL data storage for several reasons. First, they enable an application developer to enforce security policies at very fine granularity, per object. Gone are the clunky security policies based on the IP address of the client, or the per-table access controls of RDBMSs that force you to split up your data across many tables. Second, macaroons ensure that a client compromise does not lead to loss of the entire database. Third, macaroons are very flexible and expressive, able to incorporate information from external systems and third-party databases into authorization decisions. Finally, macaroons scale well and are incredibly efficient, because they avoid public-key cryptography and instead rely solely on fast hash functions.

    (tags: security macaroons cookies databases nosql case-studies storage authorization hyperdex)

  • Richard Tynan on Twitter: "GCHQ Tapping Eircom owned cable"

    Cable listed as owned by Eircom and Cable and Wireless (now Vodafone?)

    (tags: vodafone cables tapping surveillance eircom internet uk)

  • Hermitage: Testing the "I" in ACID

    [Hermitage is] a test suite for databases which probes for a variety of concurrency issues, and thus allows a fair and accurate comparison of isolation levels. Each test case simulates a particular kind of race condition that can happen when two or more transactions concurrently access the same data. Each test can pass (if the database’s implementation of isolation prevents the race condition from occurring) or fail (if the race condition does occur).

    (tags: acid architecture concurrency databases nosql)

Links for 2014-11-27

Published November 27, 2014

  • Consul case study from Hootsuite

    Hootsuite used Consul for distributed configuration, specifically dark-launch feature flags, with great results: 'Trying out bleeding edge software can be a risky proposition, but in the case of Consul, we’ve found it to be a solid system that works basically as described and was easy to get up and running. We managed to go from initial investigations to production within a month. The value was immediately obvious after looking into the key-value store combined with the events system and it’s DNS features and each of these has worked how we expected. Overall it has been fun to work with and has worked well and based on the initial work we have done with the Dark Launching system we’re feeling confident in Consul’s operation and are looking forward to expanding the scope of it’s use.'

    (tags: consul dark-launches feature-flags configuration distributed hootsuite notification)

Links for 2014-11-25

Published November 25, 2014

Links for 2014-11-24

Published November 24, 2014

Links for 2014-11-23

Published November 23, 2014

Links for 2014-11-21

Published November 21, 2014

  • Goat Simulator's fake MMO

    This is classic. I love the "Rouge":

    We also wanted the Rouge to actually look like a stealth-oriented make-up artist, but our 3D artist thought the goat looked ridiculous with a pink wig and a Gucci bag, so we remade the Rouge to actually look like a Rogue.

    (tags: rogue rouge goats goat-simulator funny satire mmos mmorpg games warcraft)

  • The boss has malware, again... : talesfromtechsupport

    Finally after all traditional means of infection were covered; IT started looking into other possibilities. They finally asked the Executive, “Have there been any changes in your life recently”? The executive answer “Well yes, I quit smoking two weeks ago and switched to e-cigarettes”. And that was the answer they were looking for, the made in china e-cigarette had malware hard coded into the charger and when plugged into a computer’s USB port the malware phoned home and infected the system. Moral of the story is have you ever question the legitimacy of the $5 dollar EBay made in China USB item that you just plugged into your computer? Because you should, you damn well should.
    (Via Elliot)

    (tags: via:elliot malware e-cigarettes cigarettes smoking china risks)

Links for 2014-11-20

Published November 20, 2014

  • Update on Azure Storage Service Interruption

    As part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this “flighting,” as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues.
    I'm really surprised MS deployment procedures allow a change to be rolled out globally across multiple regions on a single day. I suspect they soon won't.

    (tags: change-management cm microsoft outages postmortems azure deployment multi-region flighting azure-storage)

  • AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems - YouTube

    This is a really solid talk -- not surprising, alv@ is one of the speakers!

    "AWS and Amazon.com operate some of the world's largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives -- DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS. "
    Slides: http://www.slideshare.net/AmazonWebServices/spot302-under-the-covers-of-aws-core-distributed-systems-primitives-that-power-our-platform-aws-reinvent-2014

    (tags: scale scaling aws amazon dht logging data-structures distcomp via:marc-brooker dynamodb s3)

  • How Curiosity, Luck, and the Flip of a Switch Saved the Moon Program | Motherboard

    "SCE to off?" someone said. The switch was so obscure that neither of his bosses knew what he was talking about. "What the hell's that," blurted out Gerald Carr, who was in charge of communicating with the capsule. The rookie flight director, Gerry Griffin, didn't know either. Sixty seconds had passed since the initial lightning strike. No one else knew what to do. The call to abort was fast approaching.  Finally, Carr reluctantly gave the order in a voice far cooler than the moment. "Apollo 12, Houston, try SCE to Auxiliary, over."

    (tags: spaceflight stories apollo sce-to-aux power lightning weather outages simulation training nasa)

Links for 2014-11-19

Published November 19, 2014

  • Building a complete Tweet index

    Twitter's new massive-scale twitter search backend. Sharding galore

    (tags: architecture search twitter sharding earlybird)

  • The Oral History Of The Poop Emoji (Or, How Google Brought Poop To America)

    'I went over to Japan right around the time Takeshi was deciding which emoji were going to make it into the first cut of Gmail emoji. The [PILE_OF_POO emoji] was absolutely one of the necessary emoji that Takeshi said we have to have. There was actually conflict because there were people back at headquarters who had no idea what emoji were, and thought that having an animated [turd] in their Gmail was offensive.' '[The poop emoji] got very popular when a comic called "Dr. Slump" was broadcast in Japan back to the ‘90s. Such poop was not an object to be disliked, but it had a funny meaning. This was a very popular comedy animation where a girl played a trick on other people using the poop. The poop was this funny object to play with. It was never serious.' 'In Japanese that’s called “unchi.” It’s a child word with a benign meaning. '

    (tags: culture emoji google pile-of-poo turd poo japan gmail unchi dr-slump)

  • LUNAR MISSION ONE: A new lunar mission for everyone. by Lunar Missions Ltd — Kickstarter

    We plan to send an unmanned robotic landing module to the South Pole of the Moon – an area unexplored by previous missions. We’re going to use pioneering technology to drill down to a depth of at least 20m – 10 times deeper than has ever been drilled before – and potentially as deep as 100m. By doing this, we will access lunar rock dating back up to 4.5 billion years to discover the geological composition of the Moon, the ancient relationship it shares with our planet and the effects of asteroid bombardment. Ultimately, the project will improve scientific understanding of the early solar system, the formation of our planet and the Moon, and the conditions that initiated life on Earth.
    Kickstarter-funded -- UKP 600k goal. Just in time for xmas!

    (tags: kickstarter science moon lunar-mission-one exploration)

  • Flow, a new static type checker for JavaScript

    Unlike the (excellent) Typescript, it'll infer types:

    Flow’s type checking is opt-in — you do not need to type check all your code at once. However, underlying the design of Flow is the assumption that most JavaScript code is implicitly statically typed; even though types may not appear anywhere in the code, they are in the developer’s mind as a way to reason about the correctness of the code. Flow infers those types automatically wherever possible, which means that it can find type errors without needing any changes to the code at all. On the other hand, some JavaScript code, especially frameworks, make heavy use of reflection that is often hard to reason about statically. For such inherently dynamic code, type checking would be too imprecise, so Flow provides a simple way to explicitly trust such code and move on. This design is validated by our huge JavaScript codebase at Facebook: Most of our code falls in the implicitly statically typed category, where developers can check their code for type errors without having to explicitly annotate that code with types.

    (tags: facebook flow javascript coding types type-inference ocaml typescript)

  • Exactly-Once Delivery May Not Be What You Want

    An extremely good explanation from Marc Brooker that exactly-once delivery in a distributed system is very hard.

    And so on. There's always a place to slot in one more turtle. The bad news is that I'm not aware of a nice solution to the general problem for all side effects, and I suspect that no such solution exists. On the bright side, there are some very nice solutions that work really well in practice. The simplest is idempotence. This is a very simple idea: we make the tasks have the same effect no matter how many times they are executed.

    (tags: architecture messaging queues exactly-once-delivery reliability fault-tolerance distcomp marc-brooker)

  • GpsMapIreland

    This topographic map represents Ireland. It is designed for "hillwalking". The contour lines are extracted from SRTM public data provided by NASA. These files contain a digitized ground represented by points. The sample rate defines a grid resolution for Ireland around 90m in northing and 60m in easting. In major cases, digitized points do not correspond with summits. Carrauntoohil (1039m, the highest summit of Ireland) does not appear in SRTM data. The altitude reaches only 1018m. Data were obtain from space with a radar. Because of the relative position between the radar and the earth, a shadow appears in some conditions (along ridges, behind summits...). This shadow matches with a gap in data (Imagine you with a flashlight in a dark room. It is hard to see what is in shadow). To close these gaps, you need other data or you can do interpolation. The second solution is applied in our case. There is one square degree per SRTM file with a sample rate of 1200x1200 points/square degree at Ireland latitude. [...] All in all you obtain contour lines pretty sufficient for walking.

    (tags: hillwalking walking ireland gps garmin open-data srtm maps hiking via:alan)

  • castnow

    Marcus Ramberg says: 'If you have a chromecast and you’re not using castnow, I don’t know what is wrong with you.'

    (tags: chromecast castnow media movies tv video)

  • The Infinite Hows, instead of the Five Whys

    John Allspaw with an interesting assertion that we need to ask "how", not "why" in five-whys postmortems:

    “Why?” is the wrong question. In order to learn (which should be the goal of any retrospective or post-hoc investigation) you want multiple and diverse perspectives. You get these by asking people for their own narratives. Effectively, you’re asking “how?“ Asking “why?” too easily gets you to an answer to the question “who?” (which in almost every case is irrelevant) or “takes you to the ‘mysterious’ incentives and motivations people bring into the workplace.” Asking “how?” gets you to describe (at least some) of the conditions that allowed an event to take place, and provides rich operational data.

    (tags: ops five-whys john-allspaw questions postmortems analysis root-causes)

  • the JVM now supports globbing in classpath specifications

    hooray, no more uberjars or monster commandlines!

    (tags: java jvm globbing classpath uberjars jars deployment)

  • Microsoft Azure 9-hour outage

    'From 19 Nov, 2014 00:52 to 05:50 UTC a subset of customers using Storage, Virtual Machines, SQL Geo-Restore, SQL Import/export, Websites, Azure Search, Azure Cache, Management Portal, Service Bus, Event Hubs, Visual Studio, Machine Learning, HDInsights, Automation, Virtual Network, Stream Analytics, Active Directory, StorSimple and Azure Backup Services in West US and West Europe experienced connectivity issues. This incident has now been mitigated.' There was knock-on impact until 11:00 UTC (storage in N Europe), 11:45 UTC (websites, West Europe), and 09:15 UTC (storage, West Europe), from the looks of things. Should be an interesting postmortem.

    (tags: outages azure microsoft ops)

Links for 2014-11-18

Published November 18, 2014

Links for 2014-11-17

Published November 17, 2014

  • FBI's "Suicide Letter" to Dr. Martin Luther King, Jr., and the Dangers of Unchecked Surveillance

    The entire letter could have been taken from a page of GCHQ’s Joint Threat Research and Intelligence Group (JTRIG)—though perhaps as an email or series of tweets. The British spying agency GCHQ is one of the NSA’s closest partners. The mission of JTRIG, a unit within GCHQ, is to “destroy, deny, degrade [and] disrupt enemies by discrediting them.” And there’s little reason to believe the NSA and FBI aren’t using such tactics. The implications of these types of strategies in the digital age are chilling. Imagine Facebook chats, porn viewing history, emails, and more made public to discredit a leader who threatens the status quo, or used to blackmail a reluctant target into becoming an FBI informant. These are not far-fetched ideas. They are the reality of what happens when the surveillance state is allowed to grow out of control, and the full King letter, as well as current intelligence community practices illustrate that reality richly.

    (tags: fbi surveillance mlk history blackmail snooping gchq nsa)

  • WriterReaderPhaser

    A nice new concurrency primitive from Gil Tene:

    Have you ever had a need for logging or analyzing data that is actively being updated? Have you ever wanted to do that without stalling the writers (recorders) in any way? If so, then WriterReaderPhaser is for you.  I'm not talking about logging messages or text lines here.  I'm talking about data.  Data larger than one word of memory.  Data that holds actual interesting state. Data that keeps being updated, but needs to be viewed in a stable and coherent way for analysis or logging.  Data like frame buffers. Data like histograms.  Data like usage counts. Data that changes.
    see also Left-Right: http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.html

    (tags: phasers data-structures concurrency primitives algorithms performance wait-free)

  • 3D Secure and Verified By Visa to be canned

    Yay.

    Mastercard and Visa are removing the need for users to enter their passwords for identity confirmation as part of a revamp of the existing (oft-criticised) 3-D Secure scheme. The arrival of 3D Secure 2.0 next year will see the credit card giants moving away from the existing system of secondary static passwords to authorise online purchases, as applied by Verified by Visa and MasterCard SecureCode, towards a next-gen system based on more secure biometric and token-based prompts.
    (via Gordon)

    (tags: via:gsyme verified-by-visa 3d-secure mastercard visa credit-cards authentication authorization win passwords)

  • Aeron: Do we really need another messaging system? - High Scalability

    excellent writeup on Aeron

    (tags: aeron messing libraries java martin-thompson performance mechanical-sympathy queueing ipc tcp)

Links for 2014-11-15

Published November 15, 2014

  • IAB Statement on Internet Confidentiality

    Newly designed protocols should prefer encryption to cleartext operation. There may be exceptions to this default, but it is important to recognize that protocols do not operate in isolation.  Information leaked by one protocol can be made part of a more substantial body of information by cross-correlation of traffic observation.  There are protocols which may as a result require encryption on the Internet even when it would not be a requirement for that protocol operating in isolation. We recommend that encryption be deployed throughout the protocol stack since there is not a single place within the stack where all kinds of communication can be protected.
    Wow. so much for IPSec

    (tags: ipsec iab ietf snowden surveillance crypto protocols internet)

  • Facebook's datacenter fabric

    FB goes public with its take on the Clos network-based datacenter network architecture

    (tags: networking scaling facebook clos-networks fabrics datacenters network-architecture)

Links for 2014-11-14

Published November 14, 2014

Links for 2014-11-13

Published November 13, 2014

Links for 2014-11-12

Published November 12, 2014

Links for 2014-11-11

Published November 11, 2014

  • Life expectancy increases are due mainly to healthier children, not longer old age

    Interesting -- I hadn't expected this. 'Life expectancy at birth [in the US] in 1930 was indeed only 58 for men and 62 for women, and the retirement age was 65. But life expectancy at birth in the early decades of the 20th century was low due mainly to high infant mortality, and someone who died as a child would never have worked and paid into Social Security. A more appropriate measure is probably life expectancy after attainment of adulthood.' .... 'Men who attained age 65 could expect to collect Social Security benefits for almost 13 years (and the numbers are even higher for women).' In Ireland, life expectancy at birth has increased 18.4 years since 1926 -- but life expectancy for men aged 65 (the pension age) has only increased by 3.8 years. This means that increased life expectancy figures are not particularly relevant to the "pension crunch" story. Via Fred Logue: https://twitter.com/fplogue/status/532093184646873089

    (tags: via:fplogue statistics taxes life-expectancy pensions infant-mortality health 1930s)

  • DynamoDB Streams

    This is pretty awesome. All changes to a DynamoDB table can be streamed to a Kinesis stream, MySQL-replication-style. The nice bit is that it has a solid way to ensure readers won't get overwhelmed by the stream volume (since ddb tables are IOPS-rate-limited), and Kinesis has a solid way to read missed updates (since it's a Kafka-style windowed persistent stream). With this you have a pretty reliable way to ensure you're not going to suffer data loss.

    (tags: iops dynamodb aws kinesis reliability replication multi-az multi-region failover streaming kafka)

  • Help the GNOME Foundation defend the GNOME trademark

    Recently Groupon announced a product with the same product name as GNOME. Groupon’s product is a tablet based point of sale “operating system for merchants to run their entire operation." The GNOME community was shocked that Groupon would use our mark for a product so closely related to the GNOME desktop and technology. It was almost inconceivable to us that Groupon, with over $2.5 billion in annual revenue, a full legal team and a huge engineering staff would not have heard of the GNOME project, found our trademark registration using a casual search, or even found our website, but we nevertheless got in touch with them and asked them to pick another name. Not only did Groupon refuse, but it has now filed even more trademark applications (the full list of applications they filed can be found here, here and here). To use the GNOME name for a proprietary software product that is antithetical to the fundamental ideas of the GNOME community, the free software community and the GNU project is outrageous. Please help us fight this huge company as they try to trade on our goodwill and hard earned reputation.

    (tags: gnome groupon trademark infringement open-source operating-systems ip law floss)

  • StatusPage.io

    'Hosted Status Pages for Your Company'. We use these guys in $work, and their service is fantastic -- it's a line of javascript in the page template which will easily allow you to add a "service degraded" banner when things go pear-shaped, along with an external status site for when things get really messy. They've done a good clean job.

    (tags: monitoring server status outages uptime saas infrastructure)

Links for 2014-11-10

Published November 10, 2014

  • Eircom have run out of network capacity

    This is due in part to huge growth in the data volumes and data traffic that is transported over our network, which has exceeded our forecasted growth. We are making a number of improvements to our international connectivity which will add significant capacity and this work will be completed in the next two or three weeks.
    Guess this is what happens when Amazon poach your IP network engineers. doh! More seriously though, if you're marketing eFibre heavily, shouldn't you be investing in the upstream capacity to go with it?

    (tags: eircom fail internet capacity forecasting networking)

  • Apple site lets you deactivate iMessage and solve your missing text problem

    FINALLY.

    (tags: apple imessage android switch hardware lock-in)

  • Yes, Isis exploits technology. But that’s no reason to compromise our privacy | Technology | The Observer

    From the very beginning, Isis fanatics have been up to speed on [social media]. Which raises an interesting question: how come that GCHQ and the other intelligence agencies failed to notice the rise of the Isis menace until it was upon us? Were they so busy hoovering metadata and tapping submarine cables and “mastering the internet” (as the code name of one of their projects puts it) that they didn’t have time to see what every impressionable Muslim 14-year-old in the world with an internet connection could see?

    (tags: gchq guardian encryption nsa isis technology social-media snooping surveillance)

  • This Canadian Artist Halted Pipeline Development by Copyrighting His Land as a Work of Art

    One of the really important pieces on my land was this white-picket fence. The picket fence is probably 100 yards or less, within 100 yards of where they wanted to build this pipeline. I [plan to] extend it 8 feet every year for the rest of my life and I've been doing that for 25 years. It got me thinking, where does this piece end? Does it end at the actual structure of the fence or the things growing around it, growing through it, that are part of the photography, the documentation of it? I realized at that point that [the fence], and the other sculptures and pieces and incursions and conceptual works, were actually integral to that piece of land and to my practice. I had not intended for it to be a political piece, it was just a piece, an idea the follow-through of which at some point became poetic, you go, "Wait a minute the fence actually stopped them!" But the fence doesn't actually enclose anything. It's just a straight line. And it's marking something that's actually unmarkable, which is time. And one day it'll be gone, as will I. The land will be changed--but it was just this crazy irony that kicked into play when I was standing there with those oil negotiators.

    (tags: copyright art pipelines canada politics oil land conceptual-art ip)

  • wrk2

    'A constant throughput, correct latency-recording variant of wrk. This is a must-have when measuring network service latency -- corrects for Coordinated Omission error:

    wrk's model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received. While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.

    (tags: wrk latency measurement tools cli http load-testing testing load-generation coordinated-omission gil-tene)

  • The problem of managing schemas

    Good post on the pain of using CSV/JSON as a data interchange format:

    eventually, the schema changes. Someone refactors the code generating the JSON and moves fields around, perhaps renaming few fields. The DBA added new columns to a MySQL table and this reflects in the CSVs dumped from the table. Now all those applications and scripts must be modified to handle both file formats. And since schema changes happen frequently, and often without warning, this results in both ugly and unmaintainable code, and in grumpy developers who are tired of having to modify their scripts again and again.

    (tags: schema json avro protobuf csv data-formats interchange data hadoop files file-formats)

Links for 2014-11-07

Published November 7, 2014

Links for 2014-11-06

Published November 6, 2014

Links for 2014-11-05

Published November 5, 2014

Links for 2014-11-04

Published November 4, 2014

  • Zookeeper: not so great as a highly-available service registry

    Turns out ZK isn't a good choice as a service discovery system, if you want to be able to use that service discovery system while partitioned from the rest of the ZK cluster:

    I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances.  This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones.  What I saw was that the two other instances noticed the first server “going away”, but they continued to function as they still saw a majority (66%).  More interestingly the first instance noticed the other two servers “going away”, dropping the ensemble availability to 33%.  This caused the first server to stop serving requests to clients (not only writes, but also reads).
    So: within that offline AZ, service discovery *reads* (as well as writes) stopped working due to a lack of ZK quorum. This is quite a feasible outage scenario for EC2, by the way, since (at least when I was working there) the network links between AZs, and the links with the external internet, were not 100% overlapping. In other words, if you want a highly-available service discovery system in the fact of network partitions, you want an AP service discovery system, rather than a CP one -- and ZK is a CP system. Another risk, noted on the Netflix Eureka mailing list at https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/tA9UnerrBHUJ :
    ZooKeeper, while tolerant against single node failures, doesn't react well to long partitioning events. For us, it's vastly more important that we maintain an available registry than a necessarily consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that's OK with us.
    I guess this means that a long partition can trigger SESSION_EXPIRED state, resulting in ZK client libraries requiring a restart/reconnect to fix. I'm not entirely clear what happens to the ZK cluster itself in this scenario though. Finally, Pinterest ran into other issues relying on ZK for service discovery and registration, described at http://engineering.pinterest.com/post/77933733851/zookeeper-resilience-at-pinterest ; sounds like this was mainly around load and the "thundering herd" overload problem. Their workaround was to decouple ZK availability from their services' availability, by building a Smartstack-style sidecar daemon on each host which tracked/cached ZK data.

    (tags: zookeeper service-discovery ops ha cap ap cp service-registry availability ec2 aws network partitions eureka smartstack pinterest)

  • Why We Didn’t Use Kafka for a Very Kafka-Shaped Problem

    A good story of when Kafka _didn't_ fit the use case:

    We came up with a complicated process of app-level replication for our messages into two separate Kafka clusters. We would then do end-to-end checking of the two clusters, detecting dropped messages in each cluster based on messages that weren’t in both. It was ugly. It was clearly going to be fragile and error-prone. It was going to be a lot of app-level replication and horrible heuristics to see when we were losing messages and at least alert us, even if we couldn’t fix every failure case. Despite us building a Kafka prototype for our ETL — having an existing investment in it — it just wasn’t going to do what we wanted. And that meant we needed to leave it behind, rewriting the ETL prototype.

    (tags: cassandra java kafka scala network-partitions availability multi-region multi-az aws replication onlive)

  • Madhumita Venkataramanan: My identity for sale (Wired UK)

    If the data aggregators know everything about you -- including biometric data, healthcare history, where you live, where you work, what you do at the weekend, what medicines you take, etc. -- and can track you as an individual, does it really matter that they don't know your _name_? They legally track, and sell, everything else.

    As the data we generate about ourselves continues to grow exponentially, brokers and aggregators are moving on from real-time profiling -- they're cross-linking data sets to predict our future behaviour. Decisions about what we see and buy and sign up for aren't made by us any more; they were made long before. The aggregate of what's been collected about us previously -- which is near impossible for us to see in its entirety -- defines us to companies we've never met. What I am giving up without consent, then, is not just my anonymity, but also my right to self-determination and free choice. All I get to keep is my name.

    (tags: wired privacy data-aggregation identity-theft future grim biometrics opt-out healthcare data data-protection tracking)

  • Linux kernel's Transparent Huge Pages feature causing 300ms-800ms pauses

    bad news for low-latency apps. See also its impact on redis: http://antirez.com/news/84

    (tags: redis memory defrag huge-pages linux kernel ops latency performance transparent-huge-pages)

  • Please grow your buffers exponentially

    Although in some cases x1.5 is considered good practice. YMMV I guess

    (tags: malloc memory coding buffers exponential jemalloc firefox heap allocation)

  • How I created two images with the same MD5 hash

    I found that I was able to run the algorithm in about 10 hours on an AWS large GPU instance bringing it in at about $0.65 plus tax.
    Bottom line: MD5 is feasibly attackable by pretty much anyone now.

    (tags: crypto images md5 security hashing collisions ec2 via:hn)

Links for 2014-11-03

Published November 3, 2014

Links for 2014-10-31

Published October 31, 2014

  • Chip & PIN vs. Chip & Signature

    Trust US banks to fuck up their attempts at security :( US "chip-and-signature" cards are still entirely forgeable because the banks fear that consumers are too stupid to use a PIN, basically.

    BK: So, I guess we should all be grateful that banks and retailers in the United States are finally taking steps to move toward chip [and signature] cards, but it seems to me that as long as these chip cards still also store cardholder data on a magnetic stripe as a backup, that the thieves can still steal and counterfeit this card data — even from chip cards. Litan: Yes, that’s the key problem for the next few years. Once mag stripe goes away, chip-and-PIN will be a very strong solution. The estimates are now that by the end of 2015, 50 percent of the cards and terminals will be chip-enabled, but it’s going to be several years before we get closer to full compliance. So, we’re probably looking at about 2018 before we can start making plans to get rid of the magnetic stripe on these cards.

    (tags: magstripe banks banking chip-and-pin security brian-krebs chip-and-signature)

  • Elastic MapReduce vs S3

    Turns out there are a few bugs in EMR's S3 support, believe it or not. 1. 'Consider disabling Hadoop's speculative execution feature if your cluster is experiencing Amazon S3 concurrency issues. You do this through the mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution configuration settings. This is also useful when you are troubleshooting a slow cluster.' 2. Upgrade to AMI 3.1.0 or later, otherwise retries of S3 ops don't work.

    (tags: s3 emr hadoop aws bugs speculative-execution ops)

Links for 2014-10-30

Published October 30, 2014

  • IT Change Management

    Stephanie Dean on Amazon's approach to CMs. This is solid gold advice for any company planning to institute a sensible technical change management process

    (tags: ops tech process changes change-management bureaucracy amazon stephanie-dean infrastructure)

  • Stephanie Dean on event management and incident response

    I asked around my ex-Amazon mates on twitter about good docs on incident response practices outside the "iron curtain", and they pointed me at this blog (which I didn't realise existed). Stephanie Dean was the front-line ops manager for Amazon for many years, over the time where they basically *fixed* their availability problems. She since moved on to Facebook, Demonware, and Twitter. She really knows her stuff and this blog is FULL of great details of how they ran (and still run) front-line ops teams in Amazon.

    (tags: ops incident-response outages event-management amazon stephanie-dean techops tos sev1)

  • RICON 2014: CRDTs

    Carlos Baquero presents several operation, state-based CRDTs for use in AP systems like Voldemort and Riak

    (tags: ap cap-theorem crdts ricon carlos-baquero data-structures distcomp)

  • Brownout: building more robust cloud applications

    Applications can saturate – i.e. become unable to serve users in a timely manner. Some users may experience high latencies, while others may not receive any service at all. The authors argue that it is better to downgrade the user experience and continue serving a larger number of clients with reasonable latency. "We define a cloud application as brownout compliant if it can gradually downgrade user experience to avoid saturation." This is actually very reminiscent of circuit breakers, as described in Nygard’s ‘Release It!’ and popularized by Netflix. If you’re already designing with circuit breakers, you’ve probably got all the pieces you need to add brownout support to your application relatively easily. "Our work borrows from the concept of brownout in electrical grids. Brownouts are an intentional voltage drop often used to prevent blackouts through load reduction in case of emergency. In such a situation, incandescent light bulbs dim, hence originating the term." "To lower the maintenance effort, brownouts should be automatically triggered. This enables cloud applications to rapidly and robustly avoid saturation due to unexpected environmental changes, lowering the burden on human operators."
    This is really similar to the Circuit Breaker pattern -- in fact it feels to me like a variation on that, driven by measured latencies of operations/requests. See also http://blog.acolyer.org/2014/10/27/improving-cloud-service-resilience-using-brownout-aware-load-balancing/ .

    (tags: circuit-breaker patterns brownout robustness reliability load latencies degradation)

  • Photographs of Sellafield nuclear plant prompt fears over radioactive risk

    "Slow-motion Chernobyl", as Greenpeace are calling it. You thought legacy code was a problem? try legacy Magnox fuel rods.

    Previously unseen pictures of two storage ponds containing hundreds of highly radioactive fuel rods at the Sellafield nuclear plant show cracked concrete, seagulls bathing in the water and weeds growing around derelict machinery. But a spokesman for owners Sellafield Ltd said the 60-year-old ponds will not be cleaned up for decades, despite concern that they are in a dangerous state and could cause a large release of radioactive material if they are allowed to deteriorate further. “The concrete is in dreadful condition, degraded and fractured, and if the ponds drain, the Magnox fuel will ignite and that would lead to a massive release of radioactive material,” nuclear safety expert John Large told the Ecologist magazine. “I am very disturbed at the run-down condition of the structures and support services. In my opinion there is a significant risk that the system could fail.

    (tags: energy environment nuclear uk sellafield magnox seagulls time long-now)

  • The man who made a game to change the world

    An interview with Richard Bartle, the creator of MUD, back in 1978.

    Perceiving the different ways in which players approached the game led Bartle to consider whether MMO players could be classified according to type. "A group of admins was having an argument about what people wanted out of a MUD in about 1990," he recalls. "This began a 200-long email chain over a period of six months. Eventually I went through everybody's answers and categorised them. I discovered there were four types of MMO player. I published some short versions of them then, when the journal of MUD research came out I wrote it up as a paper." The so-called Bartle test, which classifies MMO players as Achievers, Explorers, Socialisers or Killers (or a mixture thereof) according to their play-style remains in widespread use today. Bartle believes that you need a healthy mix of all dominant types in order to maintain a successful MMO ecosystem. "If you have a game full of Achievers (players for whom advancement through a game is the primary goal) the people who arrive at the bottom level won't continue to play because everyone is better than them," he explains. "This removes the bottom tier and, over time, all of the bottom tiers leave through irritation. But if you have Socialisers in the mix they don't care about levelling up and all of that. So the lowest Achievers can look down on the Socialisers and the Socialisers don't care. If you're just making the game for Achievers it will corrode from the bottom. All MMOs have this insulating layer, even if the developers don't understand why it's there."

    (tags: mmo mud gaming history internet richard-bartle)

  • Testing fork time on AWS/Xen infrastructure

    Redis uses forking to perform persistence flushes, which means that once every 30 minutes it performs like crap (and kills the 99th percentile latency). Given this, various Redis people have been benchmarking fork() times on various Xen platforms, since Xen has a crappy fork() implementation

    (tags: fork xen redis bugs performance latency p99)

  • A Teenager Gets Grilled By Her Dad About Why She’s Not That Into Coding

    Jay Rosen interviews his 17-year-old daughter. it's pretty eye-opening. Got to start them early!

    (tags: culture tech coding girls women feminism teenagers school jay-rosen stem)

Links for 2014-10-29

Published October 29, 2014

Links for 2014-10-28

Published October 28, 2014

Links for 2014-10-27

Published October 27, 2014

  • PSA: don't run 'strings' on untrusted files (CVE-2014-8485)

    ffs.

    Perhaps simply by the virtue of being a part of that bundle, the strings utility tries to leverage the common libbfd infrastructure to detect supported executable formats and "optimize" the process by extracting text only from specific sections of the file. Unfortunately, the underlying library can be hardly described as safe: a quick pass with afl (and probably with any other competent fuzzer) quickly reveals a range of troubling and likely exploitable out-of-bounds crashes due to very limited range checking

    (tags: strings libbfd gnu security fuzzing buffer-overflows)

Links for 2014-10-22

Published October 22, 2014

  • Jauter

    This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.

    (tags: java jauter scala request-routing http netty open-source)

  • "Viewstamped Replication Revisited", Liskov and Cowling [pdf]

    classic replication paper, via aphyr: 'This paper presents an updated version of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group. The paper also describes a number of important optimizations and presents a protocol for handling reconfigurations that can change both the group membership and the number of failures the group is able to handle.'

Links for 2014-10-21

Published October 21, 2014

  • BioBrick

    Holy shit we are living in the future.

    BioBrick parts are DNA sequences which conform to a restriction-enzyme assembly standard.[1][2] These Lego-like building blocks are used to design and assemble synthetic biological circuits, which would then be incorporated into living cells such as Escherichia coli cells to construct new biological systems.[3] Examples of BioBrick parts include promoters, ribosomal binding sites (RBS), coding sequences and terminators.
    (via Soren)

    (tags: via:sorenrags biobricks fabrication organisms artificial-life biology e-coli genetic-engineering)

  • Is Docker ready for production? Feedbacks of a 2 weeks hands on

    I have to agree with this assessment -- there are a lot of loose ends still for production use of Docker in a SOA stack environment:

    From my point of view, Docker is probably the best thing I’ve seen in ages to automate a build. It allows to pre build and reuse shared dependencies, ensuring they’re up to date and reducing your build time. It avoids you to either pollute your Jenkins environment or boot a costly and slow Virtualbox virtual machine using Vagrant. But I don’t feel like it’s production ready in a complex environment, because it adds too much complexity. And I’m not even sure that’s what it was designed for.

    (tags: docker complexity devops ops production deployment soa web-services provisioning networking logging)

Links for 2014-10-20

Published October 20, 2014

  • Load testing Apache Kafka on AWS

    This is a very solid benchmarking post, examining Kafka in good detail. Nicely done. Bottom line:

    I basically spend 2/3 of my work time torture testing and operationalizing distributed systems in production. There's some that I'm not so pleased with (posts pending in draft forever) and some that have attributes that I really love. Kafka is one of those systems that I pretty much enjoy every bit of, and the fact that it performs predictably well is only a symptom of the reason and not the reason itself: the authors really know what they're doing. Nothing about this software is an accident. Performance, everything in this post, is only a fraction of what's important to me and what matters when you run these systems for real. Kafka represents everything I think good distributed systems are about: that thorough and explicit design decisions win.

    (tags: testing aws kafka ec2 load-testing benchmarks performance)

Links for 2014-10-17

Published October 17, 2014

Links for 2014-10-16

Published October 16, 2014

  • Landlords not liable for tenants’ water bills

    What an utter fuckup. Business as usual for Irish Water:

    However the spokeswoman said application packs for rented dwellings would be addressed to the landlord, at the landlord’s residence, and it would be the landlord’s responsibility to ensure the tenant received the application pack. Bills are to be issued quarterly, but as Irish Water will have the tenant’s PPS number, the utility firm will be able to pursue the tenant for any arrears and even apply any arrears to new accounts, when the tenant moves to a new address. Last week landlords had expressed concern over potential arrears, the liability for them and the possibility of being used as collection agents by Irish Water.

    (tags: landlords ireland irish-water tenancy rental ppsn)

  • Irish Water responds to landlords’ questions

    ugh, what a mess....

    * Every rental unit in the State is to get a pack addressed personally to the occupant. If Irish Water does not have details of a tenant, the pack will be addressed to ‘The Occupier’ * Packs will only be issued to individual rental properties in so far as Irish Water is aware of them * Landlords can contact Irish Water to advise they have let a property * Application Packs are issued relative to the information on the Irish Water mailing list. If this is incorrect or out of date, landlords can contact Irish Water to have the information adjusted *Irish Water will contact known landlords after the initial customer application campaign, to advise of properties for which no application has been received * Irish Water said that when a household is occupied the tenant is liable and when vacant the owner is liable. Both should advise Irish Water of change of status to the property - the tenant to cease liability, the landlord to take it up. Either party may take a reading and provide it to Irish Water, alternatively Irish Water will bill on average consumption, based on the date of change.

    (tags: irish-water water ireland liability bills landlords tenancy rental)

Links for 2014-10-15

Published October 15, 2014

Links for 2014-10-14

Published October 14, 2014

  • Dublin's Best-Kept Secret: Blas Cafe

    looks great, around the corner from Cineworld on King's Inn St, D1

    (tags: dublin cafes food blas-cafe eating northside)

  • "Meta-Perceptual Helmets For The Dead Zoo"

    with Neil McKenzie, Nov 9-16 2014, in the National History Museum in Dublin: 'These six helmets/viewing devices start off by exploring physical conditions of viewing: if we have two eyes, they why is our vision so limited? Why do we have so little perception of depth? Why don’t our two eyes offer us two different, complementary views of the world around us? Why can’t they extend from our body so we can see over or around things? Why don’t they allow us to look behind and in front at the same time, or sideways in both directions? Why can’t our two eyes simultaneously focus on two different tasks? Looking through Michael Land’s defining work Animal Eyes, we see that nature has indeed explored all of these possibilities: a Hammerhead Shark has hyper-stereo vision; a horse sees 350° around itself; a chameleon has separately rotatable eyes… The series of Meta-Perceptual Helmets do indeed explore these zoological typologies: proposing to humans the hyper-stereo vision of the hammerhead shark; or the wide peripheral vision of the horse; or the backward/forward vision of the chameleon… but they also take us into the unnatural world of mythology and literature: the Cheshire Cat Helmet is so called because of the strange lingering effect of dominating visual information such as a smile or the eyes; the Cyclops allows one large central eye to take in the world around while a second tiny hidden eye focuses on a close up task (why has the creature never evolved that can focus on denitting without constantly having to glance around?).' (via Emma)

    (tags: perception helmets dublin ireland museums dead-zoo sharks eyes vision art)

  • Grade inflation figures from Irish universities

    The figures show that, between 2004 and 2013, an average of 71.7 per cent of students at TCD graduated with either a 1st or a 2.1. DCU and UCC had the next highest rate of such awards (64.3 per cent and 64.2 per cent respectively), followed by UCD (55.8 per cent), NUI Galway (54.7 per cent), Maynooth University (53.7 per cent) and University of Limerick (50.2 per cent).

    (tags: tcd grades grade-inflation dcu ucc ucd ireland studies academia third-level)

  • webrtcH4cKS: ~ coTURN: the open-source multi-tenant TURN/STUN server you were looking for

    Last year we interviewed Oleg Moskalenko and presented the rfc5766-turn-server project, which is a free open source and extremely popular implementation of TURN and STURN server. A few months later we even discovered Amazon is using this project to power its Mayday service. Since then, a number of features beyond the original RFC 5766 have been defined at the IETF and a new open-source project was born: the coTURN project.

    (tags: webrtc turn sturn rfc-5766 push nat stun firewalls voip servers internet)

  • Google Online Security Blog: This POODLE bites: exploiting the SSL 3.0 fallback

    Today we are publishing details of a vulnerability in the design of SSL version 3.0. This vulnerability allows the plaintext of secure connections to be calculated by a network attacker.
    ouch.

    (tags: ssl3 ssl tls security exploits google crypto)

Links for 2014-10-13

Published October 13, 2014

Links for 2014-10-11

Published October 11, 2014

Links for 2014-10-10

Published October 10, 2014

  • How Videogames Like Minecraft Actually Help Kids Learn to Read | WIRED

    I analyzed several chunks of The Ultimate Player's Guide using the Flesch-Kincaid Reading Ease scale, and they scored from grade 8 to grade 11. Yet in my neighborhood they're being devoured by kids in the early phases of elementary school. Games, it seems, can motivate kids to read—and to read way above their level. This is what Constance Steinkuehler, a games researcher at the University of Wisconsin-Madison, discovered. She asked middle and high school students who were struggling readers (one 11th-grade student read at a 6th-grade level) to choose a game topic they were interested in, and then she picked texts from game sites for them to read—some as difficult as first-year-college language. The kids devoured them with no help and nearly perfect accuracy. How could they do this? “Because they're really, really motivated,” Steinkuehler tells me. It wasn't just that the students knew the domain well; there were plenty of unfamiliar words. But they persisted more because they cared about the task. “It's situated knowledge. They see a piece of language, a turn of phrase, and they figure it out.”
    When my kids are playing Minecraft, there's a constant stream of "how do you spell X?" as they craft nametags for their pets. It's great!

    (tags: minecraft gaming kids education spelling school reading literacy)

  • "Gold" 4-star review from the Irish Times

    Niall Heery belatedly follows up Small Engine Repair, his 2006 mumblecore critical hit, with a slightly less off-centre comedy that makes imaginative use of a smashing cast. The story skirts tragedy on its leisurely passage from mishap to misadventure, but Gold remains the sort of picture you want to hug indulgently to a welcoming bosom. It gives humanism a good name.
    Go Niall! it's a great movie, go see it

    (tags: gold movies friends reviews irish-times indie)

  • JCDecaux Developer API

    web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):

    Two kinds of data are delivered by the platform: Static data provides stable information like station position, number of bike stands, payment terminal availability, etc. Dynamic data provides station state, number of available bikes, number of free bike stands, etc. Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.
    Ruby API: https://github.com/oisin/bikes

    (tags: jcdecaux bikesharing dublin dublin-bikes api web-services http json open-data)

  • Best games I played in 2013

    my coworker JK's favourite games of 2013: Gone Home, Last Of Us, Proteus, Papers Please etc. I really want to play these, since they're all totally my bag too.

    (tags: games gaming todo want 2013)

  • Why Amazon Has No Profits (And Why It Works)

    Amazon has perhaps 1% of the US retail market by value. Should it stop entering new categories and markets and instead take profit, and by extension leave those segments and markets for other companies? Or should it keep investing to sweep them into the platform? Jeff Bezos’s view is pretty clear: keep investing, because to take profit out of the business would be to waste the opportunity. He seems very happy to keep seizing new opportunities, creating new businesses, and using every last penny to do it.

    (tags: amazon business strategy capex spending stocks investing retail)

  • Spark Breaks Previous Large-Scale Sort Record – Databricks

    Massive improvement over plain old Hadoop. This blog post goes into really solid techie reasons why, including:

    First and foremost, in Spark 1.1 we introduced a new shuffle implementation called sort-based shuffle (SPARK-2045). The previous Spark shuffle implementation was hash-based that required maintaining P (the number of reduce partitions) concurrent buffers in memory. In sort-based shuffle, at any given point only a single buffer is required. This has led to substantial memory overhead reduction during shuffle and can support workloads with hundreds of thousands of tasks in a single stage (our PB sort used 250,000 tasks).
    Also, use of Timsort, an external shuffle service to offload from the JVM, Netty, and EC2 SR-IOV.

    (tags: spark hadoop map-reduce batch parallel sr-iov benchmarks performance netty shuffle algorithms sort-based-shuffle timsort)

  • UK psyops created N. Irish Satanic Panic during the Troubles - Boing Boing

    During the 1970s, when Northern Ireland was gripped by near-civil-war, British military intelligence staged the evidence of "black masses" in order to create a Satanism panic among the "superstitious" Irish to discredit the paramilitaries. The secret history of imaginary Irish Satanism is documented in Black Magic and Bogeymen: Fear, Rumour and Popular Belief in the North of Ireland 1972-74, a new book from Sheffield University's Richard Jenkins, who interviewed Captain Colin Wallace, the former head of British Army "black operations" for Northern Ireland.

    (tags: northern-ireland 1970s the-troubles ireland uvf ira history black-magic satanism weird fear mi5)

  • Netflix release new code to production before completing tests

    Interesting -- I hadn't heard of this being an official practise anywhere before (although we actually did it ourselves this week)...

    If a build has made it [past the 'integration test' phase], it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments. 

    (tags: devops deployment feature-flags release testing integration-tests uat qa production ops gating netflix)

Links for 2014-10-09

Published October 9, 2014

Links for 2014-10-08

Published October 8, 2014

  • Trouble at the Koolaid Point — Serious Pony

    This is a harrowing post from Kathy Sierra, full of valid observations:

    You’re probably more likely to win the lottery than to get any law enforcement agency in the United States to take action when you are harassed online, no matter how visciously and explicitly. Local agencies lack the resources, federal agencies won’t bother.
    That to the power of ten in Ireland, too, I'd suspect. Fuck this. Troll culture is way out of control....

    (tags: twitter harassment feminism weev abuse trolls 4chan kathy-sierra)

  • Tehuti

    An embryonic metrics library for Java/Scala from Felix GV at LinkedIn, extracted from Kafka's metric implementation and in the new Voldemort release. It fixes the major known problems with the Meter/Timer implementations in Coda-Hale/Dropwizard/Yammer Metrics. 'Regarding Tehuti: it has been extracted from Kafka's metric implementation. The code was originally written by Jay Kreps, and then maintained improved by some Kafka and Voldemort devs, so it definitely is not the work of just one person. It is in my repo at the moment but I'd like to put it in a more generally available (git and maven) repo in the future. I just haven't had the time yet... As for comparing with CodaHale/Yammer, there were a few concerns with it, but the main one was that we didn't like the exponentially decaying histogram implementation. While that implementation is very appealing in terms of (low) memory usage, it has several misleading characteristics (a lack of incoming data points makes old measurements linger longer than they should, and there's also a fairly high possiblity of losing interesting outlier data points). This makes the exp decaying implementation robust in high throughput fairly constant workloads, but unreliable in sparse or spiky workloads. The Tehuti implementation provides semantics that we find easier to reason with and with a small code footprint (which we consider a plus in terms of maintainability). Of course, it is still a fairly young project, so it could be improved further.' More background at the kafka-dev thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E

    (tags: kafka metrics dropwizard java scala jvm timers ewma statistics measurement latency sampling tehuti voldemort linkedin jay-kreps)

  • "Quantiles on Streams" [paper, 2009]

    'Chiranjeeb Buragohain and Subhash Suri: "Quantiles on Streams" in Encyclopedia of Database Systems, Springer, pp 2235–2240, 2009. ISBN: 978-0-387-35544-3', cited by Martin Kleppman in http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E as a good, short literature survey re estimating percentiles with a small memory footprint.

    (tags: latency percentiles coding quantiles streams papers algorithms)

  • Belkin Router Apocalypse

    Many Belkin routers attempt to determine if they're connected to the internet by pinging 'heartbeat.belkin.com', in a classic amateur fail move. Good reason not to run Belkin firmware if that's the level of code quality to expect

    (tags: belkin fail ping icmp funny internet dailywtf broken)

  • Seven deadly sins of talking about “types”

    Good essay

    (tags: programming types rants coding languages fp)

  • Shellshock

    An _extremely_ detailed resource about the bash bug

    (tags: bash hacking security shell exploits reference shellshock)