Category: Uncategorized

• Dublin Cycle Planner needs a health warning - Irish Cycle

  An extensive catalogue of shitty routing. Poor...

  It’s expected that any new mapping and routing systems will have errors which will need to be ironed out but the level of issues with the NTA Cycle Planner is far beyond what you’d expect in a light and quiet beta launch. It’s beyond acceptable for a public PR launch directing people to a route planner with no clear warnings. It looks like a rush job which allows junior minister Alan Kelly to get his name in another press release before the end of the year.

  (tags: cycling dublin commute mapping nta ireland maps)

• Reflected hidden faces in photographs revealed in pupil

  The pupil of the eye in a photograph of a face can be mined for hidden information, such as reflected faces of the photographer and bystanders, according to research led by Dr. Rob Jenkins, of the Department of Psychology at the University of York and published in PLOS ONE (open access).

  (via Waxy)

  (tags: via:waxy future zoom-and-enhance privacy photography eyes photos)

• Jesse Willms, the Dark Lord of the Internet - Taylor Clark - The Atlantic

  “It was an out-and-out hijacking,” LeFevre told me. “They counterfeited our product, they pirated our Web site, and they basically directed all of their customer service to us.” At the peak of Willms’s sales, LeFevre says, dazzlesmile was receiving 1,000 calls a day from customers trying to cancel orders for a product it didn’t even sell. When irate consumers made the name dazzlesmile synonymous with online scamming, LeFevre’s sales effectively dropped to zero. Dazzlesmile sued Willms in November 2009; he later paid a settlement.

  (tags: scams hijacking ads affiliate one-wierd-trick health dieting crime)

• Content filtering by UK ISPs

  An exhaustive list from the UK's Open Rights Group

  (tags: filtering uk isps org porn blocklists internet)

• Netflix: Your Linux AMI: optimization and performance [slides]

  a fantastic bunch of low-level kernel tweaks and tunables which Netflix have found useful in production to maximise productivity of their fleet. Interesting use of SCHED_BATCH process scheduler class for batch processes, in particular. Also, great docs on their experience with perf and SystemTap. Perf really looks like a tool I need to get to grips with...

  (tags: netflix aws tuning ami perf systemtap tunables sched_batch batch hadoop optimization performance)

• creepypasta, Slenderman, and Lovecraft

  our use of networked computers is daily coloured by fear of infection and corruption, of predators and those who would assume our identity, of viruses and data-sucking catastrophes. What if something dark is able to breach that all-important final firewall, the gap between the central processing unit and the person sitting at the keyboard? What if it already has? That would be ‘a malign and particular suspension or defeat of those fixed laws of Nature which are our only safeguard’, without a doubt — but the unplumbed space haunted by demons and chaos is the network, not the cosmos. In using the internet to creep ourselves out recreationally, we begin to understand the real ways in which it haunts our fears.

  (via etienneshrdlu)

  (tags: via:etienneshrdlu literature stories horror slenderman something-awful creepypasta copypasta lovecraft)

• BitCoin exchange CoinBase uses MongoDB as their 'primary datastore'

  'Coinbase uses MongoDB for their primary datastore for their web app, api requests, etc.'

  (tags: coinbase mongodb reliability hn via:aphyr ops banking bitcoin)

• Alex Payne — Bitcoin, Magical Thinking, and Political Ideology

  Working in technology has an element of pioneering, and with new frontiers come those would prefer to leave civilization behind. But in a time of growing inequality, we need technology that preserves and renews the civilization we already have. The first step in this direction is for technologists to engage with the experiences and struggles of those outside their industry and community. There’s a big, wide, increasingly poor world out there, and it doesn’t need 99% of what Silicon Valley is selling. I’ve enjoyed the thought experiment of Bitcoin as much as the next nerd, but it’s time to dispense with the opportunism and adolescent fantasies of a crypto-powered stateless future and return to the work of building technology and social services that meaningfully and accountably improve our collective quality of life.

  (tags: bitcoin business economics silicon-valley tech alex-payne writing libertarianism futurism crypto civilization frontier community)

• MP Claire Perry tells UK that worrying about filter overblocking is a "load of cock"

  the bottom line appears to be "think of the children" -- in other words, any degree of overblocking is acceptable as long as children cannot access porn:

  The debate and letter confuse legal, illegal and potentially harmful content, all of which require very different tactics to deal with. Without a greater commitment to evidence and rational debate, poor policy outcomes will be the likely result. There's a pattern, much the same as the Digital Economy Act, or the Snooper's Charter. Start with moral panic; dismiss evidence; legislate; and finally, watch the policy unravel, either delivering unintended harms, even to children in this case, or simply failing altogether.

  See https://www.openrightsgroup.org/blog/2013/talktalk-wordpress for a well-written exploration of a case of overblocking and its fallout. Talk Talk, one UK ISP, has filters which incorrectly dealt with IWF data and blocked WordPress.com's admin interface, resulting in all blogs there become unusable for their owners for over a week, with seemingly nobody able to diagnose and fix the problem competently.

  (tags: filtering overblocking uk politics think-of-the-children porn cam claire-perry open-rights-group false-positives talk-talk networking internet wordpress)

• stereopsis : graphics : radix tricks

  some nice super-optimized Radix Sort code which handles floating point values. See also http://codercorner.com/RadixSortRevisited.htm for more info on the histogramming/counter concept

  (tags: sorting programming coding algorithms radix-sort optimization floating-point)

• Numeronyms

  ie. "i18n", "a11y" etc.

  According to Tex Texin, the first numeronym [..] was "S12n", the electronic mail account name given to Digital Equipment Corporation (DEC) employee Jan Scherpenhuizen by a system administrator because his surname was too long to be an account name. By 1985, colleagues who found Jan's name unpronounceable often referred to him verbally as "S12n". The use of such numeronyms became part of DEC corporate culture.[1]

  (tags: numbers names etymology numeronyms history dec i18n a11y l10n s12n)

• On undoing, fixing, or removing commits in git

  Choose-your-own-adventure style. "Oh dear. This is going to get complicated." (via Tom)

  (tags: via:tom cyoa git fixing revert source-control coding)

• British Library uploads one million public domain images to the net for remix and reuse - Boing Boing

  this is excellent!

  The British Library has uploaded one million public domain scans from 17th-19th century books to Flickr! They're embarking on an ambitious programme to crowdsource novel uses and navigation tools for the huge corpus. Already, the manifest of image descriptions is available through Github. This is a remarkable, public spirited, archival project, and the British Library is to be loudly applauded for it!

  (tags: british-library libraries public-domain art graphics images history 19th-century 17th-century 18th-century books crowdsourcing via:boingboing github)

• The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering

  Fantastic long-form blog post by Jay Kreps on this key concept. great stuff

  (tags: coding databases log network kafka jay-kreps linkedin architecture storage)

• Difference Engine: Obituary for software patents

  The Economist reckons we're finally seeing the light at the end of the tunnel where the patent troll shakedown is concerned:

  If the use of state consumer-protection laws to ward off frivolous patent suits were to catch on, it could give the trolls serious pause for thought—especially if their mass mailings of threatening letters to businesses were met by dozens of law suits from attorneys general demanding their presence in state courts across the land. One way or another, things are beginning to look ominous for those who would exploit the inadequacies of America’s patent system.

  (tags: the-economist patents swpats trolls us east-texas law)

• Load Balancer Testing with a Honeypot Daemon

  nice post on writing BDD unit tests for infrastructure, in this case specifically a load balancer (via Devops Weekly)

  (tags: load-balancers ops devops sysadmin testing unit-tests networking honeypot infrastructure bdd)

• Karlin Lillington on DRI's looming victory in the European Court of Justice

  If the full European Court of Justice (ECJ) accepts the opinion of its advocate general in a final ruling due early next year – and it almost always does – it will prove a huge vindication of Ireland’s small privacy advocacy group, Digital Rights Ireland (DRI). Its case against Irish retention laws, which began in 2006, forms the basis of this broader David v Goliath challenge and initial opinion. The advocate general’s advice largely upholds the key concerns put forward by DRI against Ireland’s laws. Withholding so much data about every citizen, including children, in case someone commits a future crime, is too intrusive into private life, and could allow authorities to create a “faithful and exhaustive map of a large portion of a person’s [private] conduct”. Retained data is so comprehensive that they could easily reveal private identities, which are supposed to remain anonymous. And the data, entrusted to third parties, is at too much risk of fraudulent or malicious use. Cruz Villalón argues that there must be far greater oversight to the retention process, and controls on access to data, and that citizens should have the right to be notified after the fact if their data has been scrutinised. The Irish Government had repeatedly waved off such concerns from Digital Rights Ireland in the past.

  (tags: dri rights ireland internet surveillance data-retention privacy eu ecj law)

• Meet the Robot Telemarketer Who Denies She's A Robot

  Florida's spammers strike again - pushing the boundaries of intrusive direct sales and marketing

  (tags: florida ai spam direct-marketing bots sales health-insurance)

• DigitalOcean's guide to using Docker on their hosts

  must give this a spin

  (tags: lxc docker digital-ocean hosting ops)

• Reinforcing gender stereotypes: how our schools narrow children's choices | Athene Donald | Science | theguardian.com

  Our children should be free to choose to study what really excites them, not subtly steered away from certain subjects because teachers believe in and propagate the stereotypes. Last year the IOP published a report "It's Different for Girls" which demonstrated that essentially half of state coeducational schools did not see a single girl progress to A-level physics. By contrast, the likelihood of girls progressing from single sex schools were two and a half times greater.

  Amen to this.

  (tags: sexism schools teaching uk phyics girls children bias stereotypes)

• Simple Binary Encoding

  'SBE is an OSI layer 6 representation for encoding and decoding application messages in binary format for low-latency applications.' Licensed under ASL2, C++ and Java supported.

  (tags: sbe encoding codecs persistence binary low-latency open-source java c++ serialization)

• websocketd

  'like inetd, but for WebSockets' -- 'a small command line tool that will wrap an existing command line interface program, and allow it to be accessed via a WebSocket. It provides a quick mechanism for allowing web-applications to interact with existing command line tools.' Awesome idea. BSD-licensed. (Via Mike Loukides)

  (tags: websockets cli server tools unix inetd web http open-source)

• Cyanite

  a metric storage daemon, exposing both a carbon listener and a simple web service. Its aim is to become a simple, scalable and drop-in replacement for graphite's backend.

  Pretty alpha for now, but definitely worth keeping an eye on to potentially replace our burgeoning Carbon fleet...

  (tags: graphite carbon cassandra storage metrics ops graphs service-metrics)

• Twitter tech talk video: "Profiling Java In Production"

  In this talk Kaushik Srenevasan describes a new, low overhead, full-stack tool (based on the Linux perf profiler and infrastructure built into the Hotspot JVM) we've built at Twitter to solve the problem of dynamically profiling and tracing the behavior of applications (including managed runtimes) in production.

  Looks very interesting. Haven't watched it yet though

  (tags: twitter tech-talks video presentations java jvm profiling testing monitoring service-metrics performance production hotspot perf)

• Spy agencies in covert push to infiltrate virtual world of online gaming

  [MMOGs], the [NSA] analyst wrote, "are an opportunity!". According to the briefing notes, so many different US intelligence agents were conducting operations inside games that a "deconfliction" group was required to ensure they weren't spying on, or interfering with, each other.

  (tags: spies spying games mmog online surveillance absurd east-germany funny warcraft)

• Ryan Lizza: Why Won’t Obama Rein in the N.S.A.? : The New Yorker

  Fantastic wrap-up of the story so far on the pervasive global surveillance story.

  The history of the intelligence community, though, reveals a willingness to violate the spirit and the letter of the law, even with oversight. What’s more, the benefits of the domestic-surveillance programs remain unclear. Wyden contends that the N.S.A. could find other ways to get the information it says it needs. Even Olsen, when pressed, suggested that the N.S.A. could make do without the bulk-collection program. “In some cases, it’s a bit of an insurance policy,” he told me. “It’s a way to do what we otherwise could do, but do it a little bit more quickly.” In recent years, Americans have become accustomed to the idea of advertisers gathering wide swaths of information about their private transactions. The N.S.A.’s collecting of data looks a lot like what Facebook does, but it is fundamentally different. It inverts the crucial legal principle of probable cause: the government may not seize or inspect private property or information without evidence of a crime. The N.S.A. contends that it needs haystacks in order to find the terrorist needle. Its definition of a haystack is expanding; there are indications that, under the auspices of the “business records” provision of the Patriot Act, the intelligence community is now trying to assemble databases of financial transactions and cell-phone location information. Feinstein maintains that data collection is not surveillance. But it is no longer clear if there is a distinction.

  (tags: nsa gchq surveillance spying privacy dianne-feinstein new-yorker journalism long-reads us-politics probable-cause)

• Same Old Stories From Sean Sherlock

  Sherlock’s record is spotty at best when it comes to engagement. Setting aside the 80,680 people who were ignored by the minister, he was hostile and counter productive to debate from the beginning, going so far as to threaten to pull out of a public debate because a campaigner against the ['Irish SOPA'] SI would be in attendance. His habit of blocking people online who publicly ask him tough yet legitimate questions has earned him the nickname “Sherblock”.

  (tags: sean-sherlock sherblock labour ireland politics blocking filtering internet freedom copyright emi music law piracy debate twitter)

• Smart Metering in the UK is FCUKED

  Most utilities don’t want smart metering.  In fact they seem to have used the wrong dictionary.  It is difficult to find anything smart about the UK deployment, until you realise that the utilities use smart in the sense of “it hurts”.  They consider they have a perfectly adequate business model which has no need for new technology.  In many Government meetings, their reluctant support seems to be a veneer for the hope that it will all end in disaster, letting them go back to the world they know, of inflated bills and demands for money with menaces. [...] Even when smart meters are deployed, there is no evidence that any utility will use the resulting data to transform their business, rather than persecute the consumer.  At a recent US conference a senior executive for a US utility which had deployed smart meters, stated that their main benefit was “to give them more evidence to blame the customer”.  That’s a good description of the attitude displayed by our utilities.

  (tags: smart-metering energy utilities uk services metering consumer)

• Kelly "kellabyte" Sommers on Redis' "relaxed CP" approach to the CAP theorem

  Similar to ACID properties, if you partially provide properties it means the user has to _still_ consider in their application that the property doesn't exist, because sometimes it doesn't. In you're fsync example, if fsync is relaxed and there are no replicas, you cannot consider the database durable, just like you can't consider Redis a CP system. It can't be counted on for guarantees to be delivered. This is why I say these systems are hard for users to reason about. Systems that partially offer guarantees require in-depth knowledge of the nuances to properly use the tool. Systems that explicitly make the trade-offs in the designs are easier to reason about because it is more obvious and _predictable_.

  (tags: kellabyte redis cp ap cap-theorem consistency outages reliability ops database storage distcomp)

• Building a Balanced Universe - EVE Community

  Good blog post about EVE's algorithm to load-balance a 3D map of star systems

  (tags: eve eve-online algorithms 3d space load-balancing sharding games)

• Virtual Clock - Testing Patterns Encyclopedia

  a nice pattern for unit tests which need deterministic time behaviour. Trying to think up a really nice API for this....

  (tags: testing unit-tests time virtual-clock real-time coding)

• We're sending out the wrong signals in bid to lure the big data bucks - Independent.ie

  Simon McGarr on Ireland's looming data-protection train-crash.

  Last week, during the debate of his proposals to increase fees for making a Freedom of Information request, Brendan Howlin was asked how one of his amendments would affect citizens looking for data from the State's electronic databases. His reply was to cheerfully admit he didn't even understand the question. "I have no idea what an SQL code is. Does anyone know what an SQL code is?" Unlike the minister, it probably isn't your job to know that SQL is the computer language that underpins the data industry. The amendment he had originally proposed would have effectively allowed civil servants to pretend that their computer files were made of paper when deciding whether a request was reasonable. His answer showed how the Government could have proposed such an absurd idea in the first place. Like it or not – fair or not – these are not the signals a country that wanted to build a long-term data industry would choose to send out. They are the sort of signals that Ireland used to send out about Financial Regulation. I think it's agreed, that approach didn't work out so well.

  (tags: foi ireland brendan-howlin technology illiteracy sql civil-service government data-protection privacy regulation dpa)

• Chef Testing at PagerDuty

  Good article on how PagerDuty test their chef changes -- lint, unit tests using ChefSpec, integ tests and their "Failure Friday" game days

  (tags: testing chef ops devops chefspec game-days pagerduty)

• Flock for Cron jobs

  good blog post writing up the 'flock -n -c' trick to ensure single-concurrent-process locking for cron jobs

  (tags: cron concurrency unix linux flock locking ops)

• What an RAF pilot can teach us about being safe on the road

  Good article on road safety and visual perception, for both cyclists and drivers.

  (tags: vision driving cycling tips cognitive-psychology safety hi-viz)

• wrk

  a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.  An optional LuaJIT script can perform HTTP request generation, response processing, and custom reporting.

  Written in C, ASL2 licensed.

  (tags: wrk benchmarking http performance testing lua load-testing load-generation)

• Removing DRM Boosts Music Sales by 10%

  Based on a working paper from University of Toronto researcher Laurina Zhang

  Comparing album sales of four major labels before and after the removal of DRM reveals that digital music revenue increases by 10% when restrictions are removed. The effect goes up to 30% for long tail content, while top-selling albums show no significant jump. The findings suggest that dropping technical restrictions can benefit both artists and the major labels.

  more details: http://inside.rotman.utoronto.ca/laurinazhang/files/2013/11/laurina_zhang_jmp_nov4.pdf , "Intellectual Property Strategy and the Long Tail: Evidence from the Recorded Music Industry", Laurina Zhang, November 4, 2013

  (tags: ip copyright drm mp3 music laurina-zhang research long-tail albums rights-management piracy)

• 100 Years of Breed “Improvement” | Science of Dogs

  The English bulldog has come to symbolize all that is wrong with the dog fancy and not without good reason; they suffer from almost every possible disease. A 2004 survey by the Kennel Club found that they die at the median age of 6.25 years (n=180). There really is no such thing as a healthy bulldog. The bulldog’s monstrous proportions makes them virtually incapable of mating or birthing without medical intervention.

  (via Bryan)

  (tags: dogs eugenics breeding horror science genetics traits animals pets bulldog pedigree)

• SkyJack - autonomous drone hacking

  Samy Kamkar strikes again. 'Using a Parrot AR.Drone 2, a Raspberry Pi, a USB battery, an Alfa AWUS036H wireless transmitter, aircrack-ng, node-ar-drone, node.js, and my SkyJack software, I developed a drone that flies around, seeks the wireless signal of any other drone in the area, forcefully disconnects the wireless connection of the true owner of the target drone, then authenticates with the target drone pretending to be its owner, then feeds commands to it and all other possessed zombie drones at my will.'

  (tags: drones amazon hacking security samy-kamkar aircrack node raspberry-pi airborne-zombies)

• Why Did 9,000 Porny Spambots Descend on This San Diego High Schooler? - Alexis C. Madrigal - The Atlantic

  Good article about emergent behaviour from networked malware: 'The metabot, therefore, is viral. You get followed because of who follows you. This tendency explains the strange geographical cluster among San Diego high school students. Perhaps one of those kids was being followed by a really popular account (like @Interscope records, perhaps, which follows hundreds of thousands of people), and through that link, the bot stumbled into this little circle of San Diego teens. All of this activity would have remained under the radar, of course, all part of the silent non-human web. Except something went awry. For some reason, Olivia got stuck in a weird loop, and the metabot kept spawning spambots that chose to follow her over and over, relentlessly. Maybe once the metabot reached the San Diego kids, a bug kicked in. Instead of negative feedback keeping her (and everyone else) from being followed too often, we got runaway positive feedback. The bots followed her because other bots followed her. And on and on. Which is, perhaps a kind of reasoning that we can understand: It's the core logic of fame and celebrity itself. Attention flows to Snooki because attention flowed to Snooki. Attention flows to Olivia because attention flowed to Olivia. Olivia and her friends weren't wrong when they thought she'd become suddenly famous. Her audience just wasn't human.'

  (tags: socialnetworking spam twitter bots fame alexis-madrigal)

• AMAZORK - by Zachary Mason

  > reorg Ok, you reorganize all zero of your direct reports. Way to stay out of trouble, Hoss. Perhaps you'd like to coin an acronym?

  (tags: amazon amazork via:jrauser sev2s reorgs work zachary-mason games interactive-fiction zork text-adventures)

• Bunny Rabbit helmet cover

  y'know, for kids. now that would improve the slightly boring, functional helmet my middle kid wears...

  (tags: helmets helmet-covers tail-wags safety cycling skating kids)

• Make The Web Fast - The HAR Show: Capturing and Analyzing performance data with HTTP Archive format — Google Developers

  Wow, I didn't know about this. Great idea.

  Need a flexible format to record, export, and analyze network performance data? Well, that's exactly what the HTTP Archive format (HAR) is designed to do! Even better, did you know that Chrome DevTools supports it? In this episode we'll take a deep dive into the format (as you'll see, its very simple), and explore the many different ways it can help you capture and analyze your sites performance. Join Ilya Grigorik and Peter Lubbers to find out how to capture HAR network traces in Chrome, visualize the data via an online tool, share the reports with your clients and coworkers, automate the logging and capture of HAR data for your build scripts, and even adapt it to server-side analysis use cases

  (tags: capturing logging performance http debugging trace capture har archives protocols recording)

• flood.io » Convert HAR to a JMeter JMX plan file

  this is absolutely fantastic. Thanks flood.io!

  (tags: har http archive jmeter jmx recording testing debugging captures conversion)

• Who Is Watching the Watch Lists? - NYTimes.com

  it might seem that current efforts to identify and track potential terrorists would be approached with caution. Yet the federal government’s main terrorist watch list has grown to at least 700,000 people, with little scrutiny over how the determinations are made or the impact on those marked with the terrorist label. “If you’ve done the paperwork correctly, then you can effectively enter someone onto the watch list,” said Anya Bernstein, an associate professor at the SUNY Buffalo Law School and author of “The Hidden Costs of Terrorist Watch Lists,” published by the Buffalo Law Review in May. “There’s no indication that agencies undertake any kind of regular retrospective review to assess how good they are at predicting the conduct they’re targeting.”

  (tags: terrorism watchlists blacklists filtering safety air-travel government security dhs travel)

• [JavaSpecialists 215] - StampedLock Idioms

  a demo of Doug Lea's latest concurrent data structure in Java 8

  (tags: doug-lea concurrency coding java-8 java threads)

• DROP TABLE CONSULTANTS; LTD

  lulz. (via John Handelaar)

  (tags: funny little-johnny-tables companies registry uk plc)

• Docker all the things at Atlassian: automation and wiring

  A nice worked-through Docker example

  (tags: docker infrastructure devops ops deployment lxc containers linux)

• This Flaw In Facebook Lets You Create As Many Fake Likes As You Want - Business Insider

  Really stupid -- Facebook infers a "like" for a site when you send a reference to a URL on that site. Obviously broken behaviour. (via http://www.forbes.com/sites/anthonykosner/2013/01/21/facebook-is-recycling-your-likes-to-promote-stories-youve-never-seen-to-all-your-friends/ )

  (tags: facebook advertising bad-data social-graph duh)

• ThinkGeek :: Sriracha Rooster Tee

  WANT

  (tags: tees apparel t-shirts sriracha food)

• Jury: Newegg infringes Spangenberg patent, must pay $2.3 million | Ars Technica

  Newegg, an online retailer that has made a name for itself fighting the non-practicing patent holders sometimes called "patent trolls," sits on the losing end of a lawsuit tonight. An eight-person jury came back shortly after 7:00pm and found that the company infringed all four asserted claims of a patent owned by TQP Development, a company owned by patent enforcement expert Erich Spangenberg.

  "patent enforcement expert". That's one way to put it. This is insanity.

  (tags: tech swpats patents newegg tqp crypto whitfield-diffie)

• Shared Libraries are Obsolete

  pretty strong argument. However, I think shlibs still have an advantage in that their pages are easier to share...

  (tags: shared-libraries unix linux linker deployment)

• Newegg trial: Crypto legend takes the stand, goes for knockout patent punch | Ars Technica

  "We've heard a good bit in this courtroom about public key encryption," said Albright. "Are you familiar with that? "Yes, I am," said Diffie, in what surely qualified as the biggest understatement of the trial. "And how is it that you're familiar with public key encryption?" "I invented it."

  (via burritojustice)

  (tags: crypto tech security patents swpats pki whitfield-diffie history east-texas newegg patent-trolls)

• SAMOA, an open source platform for mining big data streams

  Yahoo!'s streaming machine learning platform, built on Storm, implementing:

  As a library, SAMOA contains state-of-the-art implementations of algorithms for distributed machine learning on streams. The first alpha release allows classification and clustering. For classification, we implemented a Vertical Hoeffding Tree (VHT), a distributed streaming version of decision trees tailored for sparse data (e.g., text). For clustering, we included a distributed algorithm based on CluStream. The library also includes meta-algorithms such as bagging.

  (tags: storm streaming big-data realtime samoa yahoo machine-learning ml decision-trees clustering bagging classification)

• Spam-Friendly Registrar ‘Dynamic Dolphin’ Shuttered

  yay (via Tony Finch)

  (tags: dynamic-dolphin registrars dns spam scott-richter anti-spam brian-krebs)

• Photographer wins $1.2 million from companies that took pictures off Twitter | Reuters

  The jury found that Agence France-Presse and Getty Images willfully violated the Copyright Act when they used photos Daniel Morel took in his native Haiti after the 2010 earthquake that killed more than 250,000 people, Morel's lawyer, Joseph Baio, said

  (tags: copyright twitter facebook social-media via:niall-harbison law getty-images afp daniel-morel haiti photography)

• Failure Friday: How We Ensure PagerDuty is Always Reliable

  Basically, they run the kind of exercise which Jesse Robbins invented at Amazon -- "Game Days". Scarily, they do these on a Friday -- living dangerously!

  (tags: game-days testing failure devops chaos-monkey ops exercises)

• Shadows in the Woods

  beautiful German boardgame, suitable for playing with kids -- an adult moves a tealight candle around the board, while kids take turns moving gnomes around in the shadows behind tall "trees". recommended by JK

  (tags: games boardgames german kids candles light)

• 'No basis in law' : Gardai probe Ballyphehane group after raid

  Freemen wackiness in Cork.

  The house of one member of the group was raided by gardaí last week, but it is not thought that any arrests were made, according to an eyewitness. Gardaí broke down the front door of the house. The group, which appears to be part of the Freemen of the Land movement, which does not recognise the State, has attempted to hold 'trials' in Ballyphehane Community Centre. It attempted to summon HSE staff, gardaí, social workers, solicitors and others to appear to be tried by a self-selected jury earlier this month. The group handed out documents purporting to be a summons to HSE staff and garda stations, demanding that named people attend a trial by 'éire court' on Tuesday 5 November at 9am “to stand trial for their acts of terrorism against mothers, their offspring and others in our community”, according to the group's literature. This week the group has begun posting about UCC, saying the college is “a private for profit corporation, and a business partner of and partly owned by Pfizers and Bank of Ireland”. The group suggest that UCC bases its “authority” on Maritime Law. UCC has yet to respond to the group's allegations.

  (tags: freemen crazy cork politics ireland hse gardai ucc law)

• Rasmus' home NAS design

  I'm trying to avoid doing this in order to avoid more power consumption and unpopular hardware in the house -- but if necessary, this is a good up-to-date homebuild design

  (tags: nas hardware home storage ops disks)

• Asynchronous logging versus Memory Mapped Files

  Interesting article around using mmap'd files from Java using RandomAccessFile.getChannel().map(), which allows them to be accessed directly as a ByteBuffer. together with Atomic variable lazySet() operations, this provides pretty excellent performance results on low-latency writes to disk. See also: http://psy-lob-saw.blogspot.ie/2012/12/atomiclazyset-is-performance-win-for.html

  (tags: atomic lazyset putordered jmm java synchronization randomaccessfile bytebuffers performance optimization memory disk queues)

• Grape

  a realtime processing engine, built on a persistent queue and a set of workers. 'The main goal is data availability and persistency. We created grape for those who cannot afford losing data'. It does this by allowing infinite expansion of the pending queue in Elliptics, their Dynamo-like horizontally-scaled storage backend.

  (tags: kafka queue queueing storage realtime fault-tolerance grape cep event-processing)

• How To Run a 5 Whys (With Humans, Not Robots)

  'remember, there is no axe murderer. probably'

  (tags: process management howto post-mortems five-whys 5-whys investigation)

• The New Threat: Targeted Internet Traffic Misdirection

  MITM attacks via BGP route hijacking now relatively commonplace on the internet, with 60 cases observed so far this year by Renesys

  (tags: bgp mitm internet security routing attacks hijacking)

• Software Detection of Currency

  Steven J. Murdoch presents some interesting results indicating that the EURion constellation may have been obsoleted:

  Recent printers, scanners and image manipulation software identify images of currency, will not process the image and display an error message linking to www.rulesforuse.org. The detection algorithm is not disclosed, however it is possible to test sample images as to whether they are identified as currency. This webpage shows an initial analysis of the algorithm's properties, based on results from the automated generation and testing of images. [...] Initially it was thought that the "Eurion constellation" was used to identify banknotes in the newly deployed software based system, since this has been confirmed to be the technique used by colour photocopiers, and was both necessary and sufficient to prevent an item being duplicated using the photocopier tested. However further investigation showed that the detection performed by software is different from the system used in colour photocopiers, and the Eurion constellation is neither necessary nor sufficent, and in fact it probably is not even a factor.

  (tags: eurion algorithms photoshop security currency money euro copying obscurity reversing)

• Factual/drake

  a simple-to-use, extensible, text-based data workflow tool that organizes command execution around data and its dependencies. Data processing steps are defined along with their inputs and outputs and Drake automatically resolves their dependencies. [...] Drake is similar to GNU Make, but designed especially for data workflow management. It has HDFS [and S3] support, allows multiple inputs and outputs, and includes a host of features designed to help you bring sanity to your otherwise chaotic data processing workflows.

  Via Nelson. Looks interesting, although I'd like to see more features around retries, single-executor locking, parallelism, alerting/metrics, and unattended cron-like operation -- those are always the hard part when I wind up coding up a data pump.

  (tags: make data data-pump drake via:nelson pipelines workflow)

• AK at re:Invent 2013: Getting Maximum Performance from Redshift

  good Redshift tips

  (tags: redshift aws amazon performance scaling s3 rdbms sql ops analytics)

• Tintin And The Copyright Sharks - Falkvinge on Infopolicy

  A rather sordid tale of IP acquisition and exploitation, from the sounds of it

  (tags: tintin moulinsart belgium history herge ip copyright royalties rick-falkvinge)

• IPSO representative trivialising impact of the Loyaltybuild data breach

  A very worrying quote from Una Dillon of the Irish Payment Services Organisation in regard to the Loyaltybuild incident:

  “I wouldn’t be overly concerned if one of my cards was caught up in this,” Dillon says. “Even in the worst-case scenario – one in which my card was used fraudulently – my card provider will refund me everything that is taken”.

  This reflects a deep lack of understanding of (a) how identity fraud works, and (b) how card-fraud refunds in Ireland appear to work. (a): Direct misuse of credit card data is not always the result. Fraudsters may prefer to instead obtain separate credit through identity theft, ie. using other personal identifying data. (b): Visa debit cards have no credit limit -- your bank account can be cleared out in its entirety, and refunds can take a long time. For instance, http://www.askaboutmoney.com/showthread.php?t=174482 describes several cases, including one customer who waited 21 days for a refund. All in all it's trivialising a major risk for consumers. As I understand it, a separate statement from IPSO recommended that all customers of Loyaltybuild schemes need to monitor their bank accounts daily to keep an eye out for fraud, which is pretty absurd. Not impressive at all.

  (tags: loyaltybuild ipso money cards credit-cards visa debit-cards payment fraud identity-theft ireland)

• Why GitHub is not your CV

  There is really astonishingly little value in looking at someone’s GitHub projects out of context. For a start, GitHub has no way of customising your profile page, and what is shown by default is the projects with the most stars, and the projects you’ve recently pushed to. That is, GitHub picks your most popular repos and puts those at the top. You have no say about what you consider important, or worthwhile, or interesting, or well-engineered, or valuable. You just get what other people think is useful. Aside from which, GitHub displays a lot of useless stats about how many followers you have, and some completely psychologically manipulative stats about how often you commit and how many days it is since you had a day off. So really, your GitHub profile displays two things: how ‘influential’ you are, and how easily you can be coerced into constantly working. It’s honestly about as relevant to a decent hiring decision as your Klout score.

  (tags: cv github open-source hiring career meritocracy work via:apyhr)

• An Empirical Evaluation of TCP Performance in Online Games

  In this paper, we have analyzed the performance of TCP in of ShenZhou Online, a commercial, mid-sized MMORPG. Our study indicates that, though TCP is full-fledged and robust, simply transmitting game data over TCP could cause unexpected performance problems. This is due to the following distinctive characteristics of game traffic: 1) tiny packets, 2) low packet rate, 3) application-limited traffic generation, and 4) bi-directional traffic. We have shown that because TCP was originally designed for unidirectional and network-limited bulk data transfers, it cannot adapt well to MMORPG traffic. In particular, the window-based congestion control mechanism and the fast retransmit algorithm for loss recovery are ineffective. This suggests that the selective acknowledgement option should be enabled whenever TCP is used, as it significantly enhances the loss recovery process. Furthermore, TCP is overkill, as not every game packet needs to be transmitted reliably and processed in an orderly manner. We have also shown that the degraded network performance did impact users' willingness to continue a game. Finally, a number of design guidelines have been proposed by exploiting the unique characteristics of game traffic.

  via Nelson

  (tags: tcp games udp protocols networking internet mmos retransmit mmorpgs)

• Column: The Loyaltybuild breach shows it’s time to take data protection seriously

  What is afoot here is a rerun of the Celtic Tiger era “light touch regulation” of financial services. Ireland has again made a Faustian pact whereby we lure employers here on the understanding that they will not subject to too-stringent a regulatory system. As the Loyaltybuild breach has shown, this is a bargain that will probably end badly. And as with the financial services boom, it is making the Germans nervous. Perhaps we will listen to them this time.

  (tags: fergal-crehan loyaltybuild celtic-tiger ireland dpa regulation data-protection privacy credit-cards)

• mgodave/barge

  Looks very alpha, but one to watch.

  A JVM Implementation of the Raft Consensus Protocol

  (tags: via:sbtourist raft jvm java consensus distributed-computing)

• RocksDB

  ' A persistent key-value store for fast storage environments', ie. BerkeleyDB/LevelDB competitor, from Facebook.

  RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support IO-bound, in-memory and write-once workloads, and to be flexible to allow for innovation. We benchmarked LevelDB and found that it was unsuitable for our server workloads. Thebenchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine - where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal. By contrast, we've published the RocksDB benchmark results for server side workloads on Flash. We also measured the performance of LevelDB on these server-workload benchmarks and found that RocksDB solidly outperforms LevelDB for these IO bound workloads. We found that LevelDB's single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.

  Lots of good discussion at https://news.ycombinator.com/item?id=6736900 too.

  (tags: flash ssd rocksdb databases storage nosql facebook bdb disk key-value-stores lsm leveldb)

• Amazon Route 53 Infima

  Colm McCarthaigh has open sourced Infima, 'a library for managing service-level fault isolation using Amazon Route 53'.

  Infima provides a Lattice container framework that allows you to categorize each endpoint along one or more fault-isolation dimensions such as availability-zone, software implementation, underlying datastore or any other common point of dependency endpoints may share. Infima also introduces a new ShuffleShard sharding type that can exponentially increase the endpoint-level isolation between customer/object access patterns or any other identifier you choose to shard on. Both Infima Lattices and ShuffleShards can also be automatically expressed in Route 53 DNS failover configurations using AnswerSet and RubberTree.

  (tags: infima colmmacc dns route-53 fault-tolerance failover multi-az sharding service-discovery)

• LatencyUtils by giltene

  The LatencyUtils package includes useful utilities for tracking latencies. Especially in common in-process recording scenarios, which can exhibit significant coordinated omission sensitivity without proper handling.

  (tags: gil-tene metrics java measurement coordinated-omission latency speed service-metrics open-source)

• High Performance Browser Networking

  slides from Ilya Grigorik's tutorial on the topic at O'Reilly's Velocity conference. lots of good data and tips for internet protocol optimization

  (tags: slides presentations ilya-grigorik performance http https tcp tutorials networking internet)

• Alan Toner on the CRC report

  tl;dr: 'a lot to like'.

  The grand design and originality thus of ‘Modernising Copyright’ thus is the injection of targeted flexibility into the legal framework – this is no mere echo of the Hargreaves Report in the UK, which backed away from Fair Use out of fear at the uncertainty it would necessarily entail. If the Report’s authors have their way,  contested uses in Ireland will first be examined to see if they fit the exceptions spelled out in the EUCD, or checked against the innovation exception if they are derivative works/adaptations. Only if they have fallen at those two fences, will the fair use test be their last chance saloon.

  (tags: alan-toner copyright reform crc ireland fair-use)

• Reactor hits GA

  'It can't just be Big Data, it has to be Fast Data: Reactor 1.0 goes GA':

  Reactor provides the necessary abstractions to build high-throughput, low-latency--what we now call "fast data"--applications that absolutely must work with thousands, tens of thousands, or even millions of concurrent requests per second. Modern JVM applications must be built on a solid foundation of asynchronous and reactive components that efficiently manage the execution of a very large number of tasks on a very small number of system threads. Reactor is specifically designed to help you build these kinds of applications without getting in your way or forcing you to work within an opinionated pattern.

  Featuring the LMAX Disruptor ringbuffer, the JavaChronicle fast persistent message-passing queue, Groovy closures, and Netty 4.0. This looks very handy indeed....

  (tags: disruptor reactive-programming reactor async libraries java jvm frameworks spring netty fast-data)

• Backblaze Blog » How long do disk drives last?

  According to Backblaze's data, 80% of drives last 4 years, and the median lifespan is projected to be 6 years

  (tags: backblaze storage disk ops mtbf hardware failure lifespan)

• Heirloom Chemistry Set by John Farrell Kuhns — Kickstarter

  This is a beauty. I wonder if they can ship to Ireland?

  To tell our story for this Kickstarter project, we really have to start in Christmas of 1959. Like many young scientists of the time, I received a Gilbert Chemistry set. This chemistry set provided me hours of great fun and learning as well as laying the foundation for my future as a research chemist. As I became an adult I wanted to share these types of experiences with my daughter, my nephews and nieces, and friends. But soon I became aware real chemistry sets were no longer available. Without real chemistry sets and opportunities for students to learn and explore, where would our future chemists come from? So .... I set out on a mission.

  (tags: chemistry science chemistry-sets education play kickstarter)

• Philippe Flajolet’s contribution to streaming algorithms [preso]

  Nice deck covering HyperLogLog and its origins, plus a slide at the end covering the Flajolet/Wegman Adaptive Sampling algorithm ("how do you count the number of elements which appear only once in stream using constant size memory?")

  (tags: algorithms sketching hyperloglog flajolet wegman adaptive-sampling sampling presentations slides)

• 3 Tacos or 4 Flautas Per Order Make a Healthy Diet in Greatest Scientific Study Ever

  "In reality, [tacos and flautas] aren't bad meals," the report argues. "The error that many of us Mexicans [Gustavo note: and gabachos] commit is including these types of dishes in our regular diet without an appropriate balance of them and falling into excessively eating them; accompanied by a lack of physical activity, it creates bad eating habits." The good docs go on to note that people can eat tacos and flautas without negatively affecting their health, but "the key resides in controlling the quantity and frequency of eating these types of meals." They also make the point that overall, tacos and flautas have less grease than doughnuts, french fries and even some health bars, although they didn't specify which brands in the latter. In a subsequent blog post, the scientists go on to describe flautas as an "energy food" due to their composition, and conclude by recommending that a healthy diet can include three tacos al pastor or four flautas per order, "controlling the frequency of intake." So have at it, boyos, but in moderation. And I can already hear the skeptics: What about tacos de chicharrones? Why not focus on carne asada? Did they take into consideration chiles de mordida? Did they factor in horchata? And whither the burrito variable?

  (tags: science tacos flautas mexican-food food eating yay)

• Jeff Dean - Taming Latency Variability and Scaling Deep Learning [talk]

  'what Jeff Dean and team have been up to at Google'. Reducing request latency in a network SOA architecture using backup requests, etc., via Ilya Grigorik

  (tags: youtube talks google low-latency soa architecture distcomp jeff-dean networking)

• error-prone - Catch common Java mistakes as compile-time errors

  It's common for even the best programmers to make simple mistakes. And commonly, a refactoring which seems safe can leave behind code which will never do what's intended. We're used to getting help from the compiler, but it doesn't do much beyond static type checking. Using error-prone to augment the compiler's static analysis, you can catch more mistakes before they cost you time, or end up as bugs in production. We use error-prone in Google's Java build system to eliminate classes of serious bugs from entering our code, and we've open-sourced it, so you can too!

  (tags: analysis java static-analysis code errors bugs)

• HotSpot JVM garbage collection options cheat sheet (v3)

  authoritative

  (tags: jvm hotspot java tuning ops gc)

• Where your "full Irish" really comes from

  This is really disappointing; many meats labelled as "Irish" are anything but. The only trustworthy mark is the Bord Bia "Origin Ireland" stamp -- I'll be avoiding any products without this in future.

  Under European labelling law, country of origin is mandatory for beef, fish, olive oil, honey and fresh fruit and vegetables. Next month the EU will make it law to specify country of origin for the meat of pigs, chicken, sheep and goats, with a lead-in time of anywhere up to three years for food companies to comply. The pork rule, however, will only apply to fresh pork and not to processed meat, so consumers still won’t get a country-of-origin label on rashers, sausages or ham. In the meantime, the Bord Bia Origin-Ireland stamp is a guarantee that your Irish breakfast ingredients are indeed Irish.

  (tags: bord-bia labelling eu country-of-origin meat pork food quality)

• Killing Freedom of Information in Ireland

  TheStory.ie will, in all likelihood, cease all FOI requests. And we will not seek funding from the public to support an immoral, cynical, unjustified and probably illegal FOI fee regime. We will not pay for information that the public already pays for. We will not support a system that perpetuates an outrageous infringement of citizen rights. The legislation was gutted in 2003 and it is being gutted again. More generally the number of requests from journalists from all news organisations in Ireland will fall as a result of these amendments, and the resulting efforts to shine a light on the administration of the State will certainly deteriorate. And secrecy will prevail.

  (tags: ireland politics foi information secrecy law)

• 10 Things You Should Know About AWS

  Some decent tips in here, mainly EC2-focussed

  (tags: amazon ec2 aws ops rds)

• Tracing Brazil’s Guy Fawkes Masks

  really fascinating, from Ethan Zuckerman:

  The photo of workers making Guy Fawkes masks is something of a Rorschach test. If you’re primed to see the exploitative nature of global capitalism when you see people making a plastic mask, it’s there in the image. if you’re looking for the global spread of a protest movement, it’s there too, with a Brazilian factory making a local knock-off of a global icon to cash in on a national protest. Because the internet is a copying machine, it’s very bad at context. It’s easier to encounter the image of masks being manufactured devoid of accompanying details than it is to find the story behind the images. And given our tendency to ignore information in languages we don’t read, it’s easy to see how the masks come detached from their accompanying story. For me, the image is more powerful with context behind it. It’s possible to reflect on the irony of a Hollywood prop becoming an activist trope, the tensions between mass-production and anonymity and the individuality of one’s identity and grievance, the tensions between local and global, Warner Bros and Condal, intellectual property and piracy, all in the same image.

  (tags: anonymous globalization manufacturing piracy knock-offs brazil ethan-zuckerman global local hollywood capitalism)

• ReCreate Ireland - Creativity through Reuse

  Great idea.

  For creative groups, we aim to offer easy access to a rich and varied selection of textures, colours and shapes. Members are also be able to participate in creativity workshops facilitated by fully trained professional artists either in-house or on your own premises. We intend to be the first choice of teachers, early childhood educators and arts animators in the community. For businesses, ReCreate reduces the costs of moving on end-of-line materials. We are a professional, credible and reliable partner organisation and our aim is to divert approximately 115 metric tonnes of clean materials from landfill annually. All collections are free of charge.

  (tags: recreate diy make-and-do recycling landfill art play scrap)

• 3D-Print Your Own 20-Million-Year-Old Fossils

  When I get my hands on a 3-D printer, this will be high up my list of things to fabricate: a replica of a 20-million year old hominid skull.

  With over 40 digitized fossils in their collection, you can explore 3D renders of fossils representing prehistoric animals, human ancestors, and even ancient tools. Captured using Autodesk software, an SLR camera, and often the original specimen (rather than a cast replica), these renderings bring us closer than most will ever get to holding ancient artifacts. And if you've got an additive manufacturing device at your disposal, you can even download Sketchfab plans to generate your own.

  (tags: 3d-printing fossils africa history hominids replication fabrication sketchfab)

• Makers & Brothers & Others

  'A Tiny Seasonal Department Store', featuring the amazing cakes of Wildflour Bakery among others, at 5 Dame Lane, D2.

  The tiny department store will be a wonderful seasonal gathering of Makers & Brothers favourite local and international brands. The Others in this project are a carefully considered bunch of partners from the worlds of flowers, food, fashion, beauty, homeware, gifts and more.  Makers & Brothers & Others, the tiny department store, promises to be a unique, exciting and engaging retail environment. A place to explore, a seasonal store alive with wonder and served by experts. Kindly hosted by the Fumbally Exchange.

  (tags: dublin shopping food cakes wildflour-bakery makers-and-brothers xmas)

• Modernising (Irish) Copyright Katseries #2: linking & marshalling as exceptions

  Good commentary on the recent CRC report's recommendations. See also http://ipkitten.blogspot.ie/2013/10/modernising-irish-copyright-katseries-1.html

  (tags: ipkat ip copyright crc law ireland commentary)

• "The Top 6 Reasons This Infographic Is Just Wrong Enough To Sound Convincing"

  +1 to all of this, but especially #5 (polar area diagrams).

  (tags: diagrams infographics infoviz visualisation data fail statistics)

• Presto: Interacting with petabytes of data at Facebook

  Presto has become a major interactive system for the company’s data warehouse. It is deployed in multiple geographical regions and we have successfully scaled a single cluster to 1,000 nodes. The system is actively used by over a thousand employees,who run more than 30,000 queries processing one petabyte daily. Presto is 10x better than Hive/MapReduce in terms of CPU efficiency and latency for most queries at Facebook. It currently supports a large subset of ANSI SQL, including joins, left/right outer joins, subqueries,and most of the common aggregate and scalar functions, including approximate distinct counts (using HyperLogLog) and approximate percentiles (based on quantile digest). The main restrictions at this stage are a size limitation on the join tables and cardinality of unique keys/groups. The system also lacks the ability to write output data back to tables (currently query results are streamed to the client).

  (tags: facebook hadoop hdfs open-source java sql hive map-reduce querying olap)

• Herbal supplements are often 'rice and weeds'

  DNA tests show that many pills labeled as healing herbs are little more than powdered rice and weeds. [...] Among their findings were bottles of echinacea supplements, used by millions of Americans to prevent and treat colds, that contained ground up bitter weed, Parthenium hysterophorus, an invasive plant found in India and Australia that has been linked to rashes, nausea and flatulence.

  (tags: herbal-remedies scams quality medicine dna testing fillers allergies st-johns-wort echinacea)

• Scryer: Netflix’s Predictive Auto Scaling Engine

  Scryer is a new system that allows us to provision the right number of AWS instances needed to handle the traffic of our customers. But Scryer is different from Amazon Auto Scaling (AAS), which reacts to real-time metrics and adjusts instance counts accordingly. Rather, Scryer predicts what the needs will be prior to the time of need and provisions the instances based on those predictions.

  (tags: scaling infrastructure aws ec2 netflix scryer auto-scaling aas metrics prediction spikes)

• Your Assignment for Today: Chew Gum

  We have known about [the dental health benefits of xylitol in chewing gum] for a surprisingly long time. In the 1980s, a high-quality, randomized trial in Finland found that children who chewed xylitol-sweetened gum had as much as 60 percent fewer cavities compared with children who didn’t. A 1989-93 randomized study of children around age 10 in Belize showed an even greater benefit; chewing xylitol-sweetened gum decreased the risk of cavities by up to 70 percent, and a follow-up study showed that the benefit lasted for up to five years.

  (tags: xylitol via:eoin health dentist teeth chewing-gum snacks medicine)

• Mike Hearn - Google+ - The packet capture shown in these new NSA slides shows…

  The packet capture shown in these new NSA slides shows internal database replication traffic for the anti-hacking system I worked on for over two years. Specifically, it shows a database recording a user login.

  This kind of confirms my theory that the majority of interesting traffic for the NSA/GCHQ MUSCULAR sniffing system would have been inter-DC replication. Was, since it sounds like that stuff's all changing now to use end-to-end crypto...

  (tags: google crypto security muscular nsa gchq mike-hearn replication sniffing spying surveillance)

• NettoSphere demo

  'This article will use NettoSphere, a framework build on top of the popular Netty Framework and Atmosphere with support of WebSockets, Server Side Events and Long-Polling. NettoSphere allows [async JVM framework] Atmosphere's applications to run on top of the Netty Framework.'

  (tags: atmosphere netty async java scala websockets sse long-polling http demos games)

• Pushing to 100,000 API clients simultaneously

  This looks really nice -- it's quite similar to something I was hacking on a while back. Only problem is that it's AGPL-licensed... 'Pushpin makes it easy to create HTTP long-polling and streaming services using any web stack as the backend. It’s compatible with any framework, whether Django, Rails, ASP, or even PHP. Pushpin works as a reverse proxy, sitting in front of your server application and managing all of the open client connections.'

  (tags: pushpin opensource agpl http long-polling reverse-proxy architecture callbacks)

• European ruling raises questions over liability and online comment

  'A recent ruling by the European Court of Human Rights (ECHR) has called into question [...] the liability of media organisations for online comment.' Delfi, a news website in Estonia, found liable for a user's comments by the ECHR

  (tags: echr comments news web law regulation estonia delfi liability slander defamation)

• Why Every Company Needs A DevOps Team Now - Feld Thoughts

  Bookmarking particularly for the 3 "favourite DevOps patterns":

  "Make sure we have environments available early in the Development process"; enforce a policy that the code and environment are tested together, even at the earliest stages of the project; “Wake up developers up at 2 a.m. when they break things"; and "Create reusable deployment procedures".

  (tags: devops work ops deployment testing pager-duty)

• There is NO spare capacity for Dublin's water supply

  The problem in a nutshell is that for an uncomfortable amount of the year the demand outstrips what the system can comfortably supply. In the graph below you’ll see the red line (demand for water) matches and regularly exceeds the blue line (what’s produced).

  (tags: drought water dublin mismanagement capacity dcc dublin-council graphs)

• Old Fashioned 101

  Circa 1800, the Cocktail was a “hair of the dog” morning drink that tamed spirits with water, sugar and bitters (patent medicine). The late 19th Century expanded the use of the word “cocktail” to encompass just about any mixed drink. Since then, the Old Fashioned—literally, the old-fashioned way of making a cocktail—has been our contemporary expression of the original drink. During the 20th Century, various bad ideas encrusted the Old Fashioned. Here we will strip off those barnacles to expose the amazingly simple and sublime drink beneath.

  thanks to Ben for this one...

  (tags: recipe alcohol drinks cocktails old-fashioned bourbon bitters)

• Metropolitan police detained David Miranda for promoting 'political' causes | World news | The Observer

  "We assess that Miranda is knowingly carrying material [...] the disclosure or threat of disclosure is designed to influence a government, and is made for the purpose of promoting a political or ideological cause. This therefore falls within the definition of terrorism."

  (tags: security david-miranda journalism censorship terrorism the-guardian)

• A Brief Tour of FLP Impossibility

  One of the most important results in distributed systems theory was published in April 1985 by Fischer, Lynch and Patterson. Their short paper ‘Impossibility of Distributed Consensus with One Faulty Process’, which eventually won the Dijkstra award given to the most influential papers in distributed computing, definitively placed an upper bound on what it is possible to achieve with distributed processes in an asynchronous environment. This particular result, known as the ‘FLP result’, settled a dispute that had been ongoing in distributed systems for the previous five to ten years. The problem of consensus – that is, getting a distributed network of processors to agree on a common value – was known to be solvable in a synchronous setting, where processes could proceed in simultaneous steps. In particular, the synchronous solution was resilient to faults, where processors crash and take no further part in the computation. Informally, synchronous models allow failures to be detected by waiting one entire step length for a reply from a processor, and presuming that it has crashed if no reply is received. This kind of failure detection is impossible in an asynchronous setting, where there are no bounds on the amount of time a processor might take to complete its work and then respond with a message. Therefore it’s not possible to say whether a processor has crashed or is simply taking a long time to respond. The FLP result shows that in an asynchronous setting, where only one processor might crash, there is no distributed algorithm that solves the consensus problem.

  (tags: distributed-systems flp consensus-algorithms algorithms distcomp papers proofs)

• Find a separating hyperplane with this One Weird Kernel Trick

  Terrible internet ad-spam recast as machine-learning spam

  '37-year-old patriot discovers "weird" trick to end slavery to the Bayesian monopoly. Discover the underground trick she used to slash her empirical risk by 75% in less than 30 days... before they shut her down. Click here to watch the shocking video! Get the Shocking Free Report!'

  (tags: funny via:hmason machine-learning spam wtf svms bayesian)

• It’s time for Silicon Valley to ask: Is it worth it?

  These companies and their technologies are built on data, and the data is us. If we are to have any faith in the Internet, we have to trust them to protect it. That’s a relationship dynamic that will become only more intertwined as the Internet finds its way into more aspects of our daily existences, from phones that talk to us to cars that drive themselves. The US’s surveillance programs threaten to destroy that trust permanently. America’s tech companies must stand up to this pervasive and corrosive surveillance system. They must ask that difficult question: “Is it worth it?”

  (tags: silicon-valley tech nsa gchq spying surveillance internet privacy data-protection)

• Serf

  'a service discovery and orchestration tool that is decentralized, highly available, and fault tolerant. Serf runs on every major platform: Linux, Mac OS X, and Windows. It is extremely lightweight: it uses 5 to 10 MB of resident memory and primarily communicates using infrequent UDP messages [and an] efficient gossip protocol.'

  (tags: clustering service-discovery ops linux gossip broadcast clusters)

• "Effective Computation of Biased Quantiles over Data Streams" [paper]

  Skew is prevalent in many data sources such as IP traffic streams.To continually summarize the distribution of such data, a high-biased set of quantiles (e.g., 50th, 90th and 99th percentiles) with finer error guarantees at higher ranks (e.g., errors of 5, 1 and 0.1 percent, respectively) is more useful than uniformly distributed quantiles (e.g., 25th, 50th and 75th percentiles) with uniform error guarantees. In this paper, we address the following two prob-lems. First, can we compute quantiles with finer error guarantees for the higher ranks of the data distribution effectively, using less space and computation time than computing all quantiles uniformly at the finest error? Second, if specific quantiles and their error bounds are requested a priori, can the necessary space usage and computation time be reduced? We answer both questions in the affirmative by formalizing them as the “high-biased” quantiles and the “targeted” quantiles problems, respectively, and presenting algorithms with provable guarantees, that perform significantly better than previously known solutions for these problems. We implemented our algorithms in the Gigascope data stream management system, and evaluated alternate approaches for maintaining the relevant summary structures.Our experimental results on real and synthetic IP data streams complement our theoretical analyses, and highlight the importance of lightweight, non-blocking implementations when maintaining summary structures over high-speed data streams.

  Implemented as a timer-histogram storage system in http://armon.github.io/statsite/ .

  (tags: statistics quantiles percentiles stream-processing skew papers histograms latency algorithms)

• Statsite

  A C reimplementation of Etsy's statsd, with some interesting memory optimizations.

  Statsite is designed to be both highly performant, and very flexible. To achieve this, it implements the stats collection and aggregation in pure C, using libev to be extremely fast. This allows it to handle hundreds of connections, and millions of metrics. After each flush interval expires, statsite performs a fork/exec to start a new stream handler invoking a specified application. Statsite then streams the aggregated metrics over stdin to the application, which is free to handle the metrics as it sees fit. This allows statsite to aggregate metrics and then ship metrics to any number of sinks (Graphite, SQL databases, etc). There is an included Python script that ships metrics to graphite.

  (tags: statsd graphite statsite performance statistics service-metrics metrics ops)

• 34 Irish pubs listed in Michelin good food guide

  if Linnane's and Cronin's are anything to go by, these will be worth a visit

  (tags: pubs ireland tourism food holidays michelin)

• Fax vs Twilio

  A fax machine called my #twilio voice number, this is how @twilio transcribed it.... http://pic.twitter.com/RYh19Pg2pG

  This is amazing. Machine talking to machine, with hilarious results

  (tags: twilio transcription machine audio fax hey-hey-hey you-know-its-hey funny)

• Dark Mail Alliance

  Founded by Silent Circle and Lavabit. this is promising....

  To bring the world our unique end-to-end encrypted protocol and architecture that is the 'next-generation' of private and secure email. As founding partners of The Dark Mail Alliance, both Silent Circle and Lavabit will work to bring other members into the alliance, assist them in implementing the new protocol and jointly work to proliferate the worlds first end-to-end encrypted 'Email 3.0' throughout the world's email providers. Our goal is to open source the protocol and architecture and help others implement this new technology to address privacy concerns against surveillance and back door threats of any kind.

  (tags: privacy surveillance email smtp silent-circle lavabit dark-mail open-source standards crypto)

• Ponies by Kij Johnson | Tor.com

  A rather dark short story about little girls, peer pressure, and childhood. no fun for this dad of 3 girls :( (via Tatu Saloranta)

  (tags: via:cowtowncoder writing fiction sf childhood peer-pressure tor ponies)

• HdrHistogram by giltene

  A Histogram that supports recording and analyzing sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording, and provides control over value quantization behavior across the value range and the subsequent value resolution at any given level.

  (tags: hdr histogram data-structures coding gil-tene sampling measuring)

• Counterfactual Thinking, Rules, and The Knight Capital Accident

  John Allspaw with an interesting post on the Knight Capital disaster

  (tags: john-allspaw ops safety post-mortems engineering procedures)

• Toyota's killer firmware: Bad design and its consequences

  This is exactly what you do NOT want to read about embedded systems controlling acceleration in your car:

  The Camry electronic throttle control system code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable). Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them. Barr also discovered inadequate and untracked peer code reviews and the absence of any bug-tracking system at Toyota.

  On top of this, there was no error-correcting RAM in use; stack-killing recursive code; a quoted 94% stack usage; risks of unintentional RTOS task shutdown; buffer overflows; unsafe casting; race conditions; unchecked error code return values; and a trivial watchdog timer check. Crappy, unsafe coding.

  (tags: firmware horror embedded-systems toyota camry safety acceleration misra-c coding code-verification spaghetti-code cyclomatic-complexity realtime rtos c code-reviews bug-tracking quality)

• Forensic Topology

  The sounds were not, however, caused by ghosts but by a group of three or four men at least to some degree professionally trained, the FBI now believes, in tunneling: a close-knit and highly disciplined team, perhaps from the construction industry, perhaps even a disgruntled public works crew who decided to put their knowledge of the city’s underside to more lucrative work. After all, Rehder explained, their route into the bank was as much brute-force excavation as it was a retracing of the region’s buried waterways, accessing the neighborhood by way of the city’s complicated storm-sewer network, itself built along old creek beds that no longer appear on city maps. As LAPD lieutenant Doug Collisson, one of the men present on the day of the tunnel’s discovery, explained to the Los Angeles Times back in 1987, the crew behind the burglary “would have had to require some knowledge of soil composition and technical engineering. … The way the shaft itself was constructed, it was obviously well-researched and extremely sophisticated.” Rehder actually goes further, remarking that when Detective Dennis Pagenkopp “showed crime scene photos of the core bit holes” produced by the burglars’ boring upward into the vault “to guys who were in the concrete-coring business, they whistled with professional admiration.”

  (tags: cities crime architecture digging tunnels subterranean la lapd banks via:bldgblog sewers)

• Link without fear – Copyright in Ireland in a Digital Age

  The Copyright Review Committee report has been published. Headline recommendations:

  Ensure the right of free speech is a central element of the new copyright regime, including in the areas of parody and satire; Legalise legitimate forms of copying by introducing an explicit and broadly defined “Fair Use” policy. Ensure the extent of copyright ownership is balanced against the public good; Design a system which is clear to all parties, including end users; Design an enforcement mechanism which is easy to understand, transparent and accessible to all parties; Target penalties at those who infringe on copyright rather than on third parties such as intermediaries; Future-proof the new regime by basing it on applicable principles rather than rules relevant to today’s technology only; Make it easy for end-users to identify and engage with owners of copyright material.

  Here's hoping Sean Sherlock now does what he said he'd do, and acts on these recommendations.

  (tags: copyright law ireland reports fair-use free-speech satire parody copying copyfight ownership ip drm linking)

• Storm at spider.io - London Storm Meetup 2013-06-18

  Not just a Storm success story. Interesting slides indicating where a startup *stopped* using Storm as realtime wasn't useful to their customers

  (tags: storm realtime hadoop cascading python cep spider.io anti-spam events architecture distcomp low-latency slides rabbitmq)

• 14 Apple hacks from sugru

  I like the impromptu docking station hack

  (tags: apple sugru hacks hardware fixing repair diy)

• Bruce Schneier On The Feudal Internet And How To Fight It

  This is very well-put.

  In its early days, there was a lot of talk about the "natural laws of the Internet" and how it would empower the masses, upend traditional power blocks, and spread freedom throughout the world. The international nature of the Internet made a mockery of national laws. Anonymity was easy. Censorship was impossible. Police were clueless about cybercrime. And bigger changes were inevitable. Digital cash would undermine national sovereignty. Citizen journalism would undermine the media, corporate PR, and political parties. Easy copying would destroy the traditional movie and music industries. Web marketing would allow even the smallest companies to compete against corporate giants. It really would be a new world order. Unfortunately, as we know, that's not how it worked out. Instead, we have seen the rise of the feudal Internet: Feudal security consolidates power in the hands of the few. These companies [like Google, Apple, Microsoft, Facebook etc.] act in their own self-interest. They use their relationship with us to increase their profits, sometimes at our expense. They act arbitrarily. They make mistakes. They're deliberately changing social norms. Medieval feudalism gave the lords vast powers over the landless peasants; we’re seeing the same thing on the Internet.

  (tags: bruce-schneier politics internet feudal-internet google apple microsoft facebook government)

• Russia: Hidden chips 'launch malware attacks from irons'

  Cyber criminals are planting chips in electric irons and kettles to launch spam [jm: actually, malware] attacks, reports in Russia suggest. State-owned channel Rossiya 24 even showed footage of a technician opening up an iron included in a batch of Chinese imports to find a "spy chip" with what he called "a little microphone". Its correspondent said the hidden devices were mostly being used to spread viruses, by connecting to any computer within a 200m (656ft) radius which were using unprotected Wi-Fi networks. Other products found to have rogue components reportedly included mobile phones and car dashboard cameras.

  (tags: wifi viruses spam malware security russia china toasters kettles appliances)

• Asteroid "mining" with Linux and FOSS

  Planetary Resources is a company with a sky-high (some might claim "pie in the sky") goal: to find and mine asteroids for useful minerals and other compounds. It is also a company that uses Linux and lots of free software. So two of the engineers from Planetary Resources, Ray Ramadorai and Marc Allen, gave a presentation at LinuxCon North America to describe how and why the company uses FOSS—along with a bit about what it is trying to do overall.

  (tags: lwn mining planets asteroids space linux foss open-source)

• Mac OS 10.9 – Infinity times your spam

  a pretty stupid Mail.app IMAP bug hoses Fastmail:

  Yes you read that right. It’s copying all the email from the Junk Folder back into the Junk Folder again!. This is legal IMAP, so our server proceeds to create a new copy of each message in the folder. It then expunges the old copies of the messages, but it’s happening so often that the current UID on that folder is up to over 3 million. It was just over 2 million a few days ago when I first emailed the user to alert them to the situation, so it’s grown by another million since. The only way I can think this escaped QA was that they used a server which (like gmail) automatically suppresses duplicates for all their testing, because this is a massively bad problem.

  (tags: osx bugs mail.app mail imap fastmail fail)

• Google: Our Robot Cars Are Better Drivers Than Puny Humans | MIT Technology Review

  One of those analyses showed that when a human was behind the wheel, Google’s cars accelerated and braked significantly more sharply than they did when piloting themselves. Another showed that the cars’ software was much better at maintaining a safe distance from the vehicle ahead than the human drivers were. “We’re spending less time in near-collision states,” said Urmson. “Our car is driving more smoothly and more safely than our trained professional drivers.”

  (tags: google cars driving safety roads humans robots automation)

• java.util.stream.SpinedBuffer

  interesting new data structure, pending addition in Java 8. Basically an array of arrays which presents the API of a single List.

  An ordered collection of elements. Elements can be added, but not removed. Goes through a building phase, during which elements can be added, and a traversal phase, during which elements can be traversed in order but no further modifications are possible.

  (tags: spinedbuffer data-structures algorithms java jdk jvm java-8 arrays lists)

• New political ideals ravaged by ... politics

  Direct Democracy Ireland, the party linked to Freemen-on-the-land and the Christian Solidarity Party, is having a bit of a bumpy ride with party governance it sounds like

  (tags: ddi politics freemen csp)

• Tables Turned On Former NSA Boss Michael Hayden, As 'Off-The-Record' Call Is Live Tweeted By Train Passenger

  Ho ho.

  Michael Hayden, former NSA and CIA boss, who famously argued that the only people complaining about NSA surveillance were internet shut-ins who couldn't get laid, apparently never learned that when you're in a public place, someone might overhear your phone calls. Entrepreneur and former MoveOn.org director Tom Matzzie just so happened to be on the Acela express train from DC to NY when he (1) spotted Hayden sitting behind him and (2) started overhearing a series of "off the record" phone calls with press about the story of the week: the revelations of the NSA spying on foreign leaders. Matzzie did what any self-respecting American would do: live-tweet the calls.

  (tags: nsa michael-hayden twitter tom-matzzie funny irony trains interviewing public surveillance)

• Dockerize

  A tool to manage inter-container dependencies so that continuous delivery with Jenkins and Docker is feasible. Looks very helpful

  (tags: docker provisioning vms containers dockerize jenkins continuous-delivery continuous-integration)

• Is Google building a hulking floating data center in SF Bay?

  Looks pretty persuasive, especially considering they hold a patent on the design

  (tags: google data-centers bay-area ships containers shipping sea wave-power treasure-island)

• Roma, Racism And Tabloid Policing: Interview With Gary Younge : rabble

  [This case] shows the link between the popular and the state. This is tabloid journalism followed by tabloid policing. It’s also completely ignorant. I wrote my article on the Roma after covering the community for a week. I thought, “that’s interesting – there’s a range of phenotypes, ways of looking, that include Roma.” I mentioned two blonde kids by chance. I mentioned that Roma are more likely to speak the language of the country they’re in than Romani, more likely to have the religion of the country they’re in. But they have the basic aspect that is true for all identities – they know each other and other people know them. It’s not like I’m an expert on the Roma. I was covering them for a week and after the second day I knew Roma children had blonde hair and blue eyes. These people who took that kid away knew nothing. And on that basis they abducted a child.

  (tags: roma racism ireland gary-younge tabloid journalist children hse gardai)

• Experian Sold Consumer Data to ID Theft Service

  This is what happens when you don't have strong controls on data protection/data privacy -- the US experience.

  While [posing as a US-based private investigator] may have gotten the [Vietnam-based gang operating the massive identity fraud site Superget.info] past Experian and/or CourtVentures’ screening process, according to Martin there were other signs that should have alerted Experian to potential fraud associated with the account. For example, Martin said the Secret Service told him that the alleged proprietor of Superget.info had paid Experian for his monthly data access charges using wire transfers sent from Singapore. “The issue in my mind was the fact that this went on for almost a year after Experian did their due diligence and purchased” Court Ventures, Martin said. “Why didn’t they question cash wires coming in every month? Experian portrays themselves as the data-breach experts, and they sell identity theft protection services. How this could go on without them detecting it I don’t know. Our agreement with them was that our information was to be used for fraud prevention and ID verification, and was only to be sold to licensed and credentialed U.S. businesses, not to someone overseas.”

  via Simon McGarr

  (tags: via:tupp_ed privacy security crime data-protection data-privacy experian data-breaches courtventures superget scams fraud identity identity-theft)

• European Parliament passes a vote calling for the EU/US SWIFT agreement to be suspended

  "the European Parliament has today sent a clear message that enough is enough. The revelations about NSA interception of SWIFT data make a mockery of the EU's agreement with the US, through which the bank data of European citizens is delivered to the US anti-terror system (TFTP). What is the purpose of an agreement like this, which was concluded in good faith, if the US authorities are going to circumvent its provisions? "The EU cannot continue to remain silent in the face of these ongoing revelations: it gives the impression we are little more than a lap dog of the US. If we are to have a healthy relationship with the US, based on mutual respect and benefit, EU governments must not be afraid of defending core EU values when they are infringed. EU leaders must finally take a clear and unambiguous stance on the NSA violations at this week's summit."

  (tags: swift banking data eu us nsa interception surveillance snooping diplomacy)

• Response to "Optimizing Linux Memory Management..."

  A follow up to the LinkedIn VM-tuning blog post at http://engineering.linkedin.com/performance/optimizing-linux-memory-management-low-latency-high-throughput-databases --

  Do not read in to this article too much, especially for trying to understand how the Linux VM or the kernel works.  The authors misread the "global spinlock on the zone" source code and the interpretation in the article is dead wrong.

  (tags: linux tuning vm kernel linkedin memory numa)

• Making Storm fly with Netty | Yahoo Engineering

  Y! engineer doubles the speed of Storm's messaging layer by replacing the zeromq implementation with Netty

  (tags: netty async zeromq storm messaging tcp benchmarks yahoo clusters)

• Airbnb's Smartstack

  Service discovery a la Airbnb -- Nerve and Synapse: two external daemons that run on each node, Nerve to manage registration in Zookeeper, and Synapse to generate a haproxy configuration file from that, running on each host, allowing connections to all other hosts.

  (tags: haproxy services ops load-balancing service-discovery nerve synapse airbnb)

• The New York Review of Bots - @TwoHeadlines: Comedy, Tragedy, Chicago Bears

  What is near-future late-capitalist dystopian fiction but a world where there is no discernible difference between corporations, nations, sports teams, brands, and celebrities? Adam was partly right in our original email thread. @TwoHeadlines is not generating jokes about current events. It is generating jokes about the future: a very specific future dictated by what a Google algorithm believes is important about humans and our affairs.

  (tags: google-news google algorithms word-frequency twitter twoheadlines bots news emergent jokes)

• The New York Review of Bots

  'Welcome to the New York Review of Bots, a professional journal of automated-agent studies. We aspire to the highest standards of rigorous analysis, but will often just post things we liked that a computer made.'

  (tags: robots bots tumblr ai word-frequency markov-chain random twitter)

• How to lose $172,222 a second for 45 minutes

  Major outage and $465m of trading loss, caused by staggeringly inept software management: 8 years of incremental bitrot, technical debt, and failure to have correct processes to engage an ops team in incident response. Hopefully this will serve as a lesson that software is more than just coding, at least to one industry

  (tags: trading programming coding software inept fail bitrot tech-debt ops incident-response)

• Basho and Seagate partner to deliver scale-out cloud storage breakthrough

  Ha, cool. Skip the OS, write the Riak store natively to the drive. This sounds frankly terrifying ;)

  The Seagate Kinetic Open Storage platform eliminates the storage server tier of traditional data center architectures by enabling applications to speak directly to the storage system, thereby reducing expenses associated with the acquisition, deployment, and support of hyperscale storage infrastructures. The platform leverages Seagate’s expertise in hardware and software storage systems integrating an open source API and Ethernet connectivity with Seagate hard drive technology.

  (tags: seagate basho riak storage hardware drivers os ops)

• Sorry, lobbyists! Europe’s post-Snowden privacy reform gets a major boost

  Following months of revelations, and on the same day that France heard its citizens’ phone calls were being reportedly recorded en masse by the Americans, the Parliament’s committee gave a resounding thumbs-up to every single amendment proposed by industrious German Green MEP Jan Phillip Albrecht (pictured above).

  lolz.

  (tags: lobbying tech surveillance privacy eu jan-phillip-albrecht ep spying)

• NCCA Junior Cycle - Programming and Coding Consultation Page

  the National Council for Curriculum and Assessment are looking for feedback on adding programming to the junior cycle (ie., early secondary school) in Ireland. Add your EUR.02!

  (tags: ireland programming coding education schools)

• Everything You Always Wanted to Know About Synchronization but Were Afraid to Ask

  'the most exhaustive study of [multi-core] synchronization to date'

  (tags: synchronization scalability cpus hardware papers via:fanf multicore cas)

• WISH: A Monumental 11-Acre Portrait in Belfast by Jorge Rodríguez-Gerada

  Must go up and visit this.

  Unveiled several days ago in Belfast, Northern Ireland as part of the Belfast Festival, WISH is the latest public art project by Cuban-American artist Jorge Rodriguez-Gerada. The image depicted is of an anonymous Belfast girl and is so large it can only be viewed from the highest points in Belfast or an airplane. Several years in the making, WISH was first plotted on a grid using state-of-the-art Topcon GPS technology and 30,000 manually placed wooden stakes in Belfast’s Titanic Quarter. The portrait was then “drawn” with aid of volunteers who helped place nearly 8 million pounds of natural materials including soil, sand, and rock over a period of four weeks.

  (tags: belfast ireland art portraits jorge-rodriguez-gerada land soil)

• Introducing Chaos to C*

  Autoremediation, ie. auto-replacement, of Cassandra nodes in production at Netflix

  (tags: ops autoremediation outages remediation cassandra storage netflix chaos-monkey)

• Barbarians at the Gateways - ACM Queue

  I am a former high-frequency trader. For a few wonderful years I led a group of brilliant engineers and mathematicians, and together we traded in the electronic marketplaces and pushed systems to the edge of their capability.

  Insane stuff -- FPGAs embedded in the network switches to shave off nanoseconds of latency.

  (tags: low-latency hft via:nelson markets stock-trading latency fpgas networking)

• Online Algorithms in High-frequency Trading - ACM Queue

  one-pass algorithms for computing mean, variance, and linear regression, from the HFT world.

  (tags: linear-regression variance mean variability volatility stream-processing online algorithms hft trading)

• "Toy Story 2" was almost entirely deleted by accident at one point

  A stray "rm -rf" on the main network share managed to wipe out 90% of the movie's assets, and the backups were corrupt. Horrific backups war story

  (tags: movies ops backups pixar recovery accidents rm-rf delete)

• The Impossible Music of Black MIDI

  excellently bananas. 8.49 million separate musical notes in a single 4-minute-long composition (via Paddy Benson)

  (tags: music hardcore black-midi midi composition halp digital via:pbenson)

• Bitcoin Mining Operating Margin

  "The graph showing miners' revenue minus estimated electricity and bandwidth costs." -- down to -694% right now, oh dear

  (tags: bitcoin via:peakscale economics mining profit revenue charts electricity bubble)

• How to Read a Scientific Paper (About That Researcher With a Nematode in His Mouth) - Wired Science

  Let’s rewind to September 2012. It was about then- according to this recently published report (paywall) in The American Journal of Tropical Medicine – that an “otherwise healthy, 36-year-old man” felt a rough patch in his mouth, a scaly little area his right cheek. It didn’t hurt. But then it didn’t stay there either. He started testing for it with his tongue. It traveled. It moved to the back of his mouth, then forward, coiled backwards again. In the language of science: “These rough patches would appear and disappear on a daily basis, giving the patient the indirect sense that there was an organism moving within the oral cavity.”

  (tags: nematodes parasites biology medicine paper gross funny wired mouth)

• "High Performance Browser Networking", by Ilya Grigorik, read online for free

  Wow, this looks excellent. A must-read for people working on systems with high-volume, low-latency phone-to-server communications -- and free!

  How prepared are you to build fast and efficient web applications? This eloquent book provides what every web developer should know about the network, from fundamental limitations that affect performance to major innovations for building even more powerful browser applications—including HTTP 2.0 and XHR improvements, Server-Sent Events (SSE), WebSocket, and WebRTC. Author Ilya Grigorik, a web performance engineer at Google, demonstrates performance optimization best practices for TCP, UDP, and TLS protocols, and explains unique wireless and mobile network optimization requirements. You’ll then dive into performance characteristics of technologies such as HTTP 2.0, client-side network scripting with XHR, real-time streaming with SSE and WebSocket, and P2P communication with WebRTC. Deliver optimal TCP, UDP, and TLS performance; Optimize network performance over 3G/4G mobile networks; Develop fast and energy-efficient mobile applications; Address bottlenecks in HTTP 1.x and other browser protocols; Plan for and deliver the best HTTP 2.0 performance; Enable efficient real-time streaming in the browser; Create efficient peer-to-peer videoconferencing and low-latency applications with real-time WebRTC transports

  Via Eoin Brazil.

  (tags: book browser networking performance phones mobile 3g 4g hsdpa http udp tls ssl latency webrtc websockets ebooks via:eoin-brazil google http2 sse xhr ilya-grigorik)

• Even the NSA is finding it hard to cope with spam

  3 new Snowden leaks, covering acquisition of Yahoo address books, buddy lists, and email account activity, and how spammer activity required intervention to avoid losing useful data in the noise

  (tags: spam spammers nsa snowden leaks anti-spam yahoo im mail)

• "What Should I Monitor?"

  slides (lots of slides) from Baron Schwartz' talk at Velocity in NYC.

  (tags: slides monitoring metrics ops devops baron-schwartz pdf capacity)

• The trouble with timestamps

  Timestamps, as implemented in Riak, Cassandra, et al, are fundamentally unsafe ordering constructs. In order to guarantee consistency you, the user, must ensure locally monotonic and, to some extent, globally monotonic clocks. This is a hard problem, and NTP does not solve it for you. When wall clocks are not properly coupled to the operations in the system, causal constraints can be violated. To ensure safety properties hold all the time, rather than probabilistically, you need logical clocks.

  (tags: clocks time distributed databases distcomp ntp via:fanf aphyr vector-clocks last-write-wins lww cassandra riak)

• Reverse Engineering a D-Link Backdoor

  Using the correct User-Agent: string, all auth is bypassed on several released models of D-Link and Planex routers. Horrific fail by D-Link

  (tags: d-link security backdoors authorization reversing planex networking routers)

• The US fears back-door routes into the net because it's building them too | Technology | The Observer

  one of the most obvious inferences from the Snowden revelations published by the Guardian, New York Times and ProPublica recently is that the NSA has indeed been up to the business of inserting covert back doors in networking and other computing kit. The reports say that, in addition to undermining all of the mainstream cryptographic software used to protect online commerce, the NSA has been "collaborating with technology companies in the United States and abroad to build entry points into their products". These reports have, needless to say, been strenuously denied by the companies, such as Cisco, that make this networking kit. Perhaps the NSA omitted to tell DARPA what it was up to? In the meantime, I hear that some governments have decided that their embassies should no longer use electronic communications at all, and are returning to employing couriers who travel the world handcuffed to locked dispatch cases. We're back to the future, again.

  (tags: politics backdoors snowden snooping networking cisco nsa gchq)

• Azerbaijan accidentally publishes the results of its election before the polls open

  The mistake came when an electoral commission accidentally published results showing a victory for Ilham Aliyev, the country’s long-standing President, a day before voting. Meydan TV, an online channel critical of the government, released a screenshot from a mobile app for the Azerbaijan Central Election Commission which showed that Mr Aliyev had received 72.76 per cent of the vote compared with 7.4 per cent for the opposition candidate, Jamil Hasanli. The screenshot also indicates that the app displayed information about how many people voted at various times during the day. Polls opened at 8am.

  (tags: azerbaijan corruption fix elections voting voter-fraud)

• DNS registrars that complied with "shakedown" anti-piracy requests may now be in violation of ICANN Transfers Policy

  According to EasyDNS:

  Any registrar that has taken one of these sites offline that now impedes the registrants of those domains from simply getting their domain names out of there and back online somewhere else will then be subject to the TDRP – Transfer Dispute Resolution Policy and if they lose (which they will) they will be subject to TDRP fees assesed by the registry operator, and to quote the TDRP itself "Transfer dispute resolution fees can be substantial". This is why it is never a good idea to just react to pressure in the face of obnoxious bluster – in the very act of trying to diffuse any perceived culpability you end up opening yourself to real liability.

  (tags: tdrp easydns dns registrars domains piracy law due-process)

• Schneier on Security: Air Gaps

  interesting discussion in the comments. "Patricia"'s process is particularly hair-raisingly complex, involving 3 separate machines and a multitude of VMs

  (tags: air-gaps security networking bruce-schneier via:adulau)

• New faculty positions versus new PhDs

  The ever-plummeting chances of a PhD finding a faculty job:

  Since 1982, almost 800,000 PhDs were awarded in science and engineering fields, whereas only about 100,000 academic faculty positions were created in those fields within the same time frame. The number of S&E PhDs awarded annually has also increased over this time frame, from ~19,000 in 1982 to ~36,000 in 2011. The number of faculty positions created each year, however, has not changed, with roughly 3,000 new positions created annually.

  (via Javier Omar Garcia)

  (tags: via:javier career academia phd science work study research)

• The Ethics of Autonomous Cars

  Sometimes good judgment can compel us to act illegally. Should a self-driving vehicle get to make that same decision?

  (tags: ethics stories via:chris-horn the-atlantic driving cars law robots self-driving-vehicles)

• Timecop

  'A Ruby gem providing "time travel" and "time freezing" capabilities, making it dead simple to test time-dependent code. It provides a unified method to mock Time.now, Date.today, and DateTime.now in a single call.' This is about the nicest mock-time library I've found so far. (via Ben)

  (tags: time ruby testing coding unit-tests mocking timecop via:ben)

• The 29 Stages Of A Twitterstorm

  this is brilliant

  (tags: uk twitter media funny pricehound racism outrage pitchforks rage social-media)

• 'Experience of software engineers using TLA+, PlusCal and TLC' [slides] [pdf]

  by Chris Newcombe, an AWS principal engineer. Several Amazonians sharing their results in simulating tricky distributed-systems problems using formal methods

  (tags: tla+ pluscal tlc formal-methods simulation proving aws amazon architecture design)

• LinkBench: A database benchmark for the social graph

  However, the gold standard for database benchmarking is to test the performance of a system on the real production workload, since synthetic benchmarks often don't exercise systems in the same way. When making decisions about a significant component of Facebook's infrastructure, we need to understand how a database system will really perform in Facebook's production workload. [....] LinkBench addresses these needs by replicating the data model, graph structure, and request mix of our MySQL social graph workload.

  Mentioned in a presentation from Peter Bailis, http://www.hpts.ws/papers/2013/bailis-hpts-2013.pdf

  (tags: graph databases mysql facebook performance testing benchmarks workloads)

• pt-summary

  from the Percona toolkit. 'Conveniently summarizes the status and configuration of a server. It is not a tuning tool or diagnosis tool. It produces a report that is easy to diff and can be pasted into emails without losing the formatting. This tool works well on many types of Unix systems.' --- summarises OOM history, top, netstat connection table, interface stats, network config, RAID, LVM, disks, inodes, disk scheduling, mounts, memory, processors, and CPU.

  (tags: percona tools cli unix ops linux diagnosis raid netstat oom)

• How much can an extra hour's sleep change you?

  What they discovered is that when the volunteers cut back from seven-and-a-half to six-and-a-half hours' sleep a night, genes that are associated with processes like inflammation, immune response and response to stress became more active. The team also saw increases in the activity of genes associated with diabetes and risk of cancer. The reverse happened when the volunteers added an hour of sleep.

  (tags: sleep health rest cancer bbc science)

• Kovet

  some great phone cases from an Irish company, with nifty art by Irish illustrators and artists including Fatti Burke and Chris Judge

  (tags: chris-judge fatti-burke illustrators art ireland iphone cases)

• What drives JVM full GC duration

  Interesting empirical results using JDK 7u21:

  Full GC duration depends on the number of objects allocated and the locality of their references. It does not depend that much on actual heap size.

  Reference locality has a surprisingly high effect.

  (tags: java jvm data gc tuning performance cms g1)

• Rhizome | Occupy.here: A tiny, self-contained darknet

  Occupy.here began two years ago as an experiment for the encampment at Zuccotti Park. It was a wifi router hacked to run OpenWrt Linux (an operating system mostly used for computer networking) and a small "captive portal" website. When users joined the wifi network and attempted to load any URL, they were redirected to http://occupy.here. The web software offered up a simple BBS-style message board providing its users with a space to share messages and files.

  Nifty project from Dan Phiffer.

  (tags: occupy.here openwrt hacking wifi network community)

• Whatever Happened to "Due Process" ?

  Mark Jeftovic is on fire after receiving yet another "take down this domain or else" mail from the City of London police:

  We have an obligation to our customers and we are bound by our Registrar Accreditation Agreements not to make arbitrary changes to our customers settings without a valid FOA (Form of Authorization). To supersede that we need a legal basis. To get a legal basis something has to happen in court. [...] What gets me about all of this is that the largest, most egregious perpetrators of online criminal activity right now are our own governments, spying on their own citizens, illegally wiretapping our own private communications and nobody cares, nobody will answer for it, it's just an out-of-scope conversation that is expected to blend into the overall background malaise of our ever increasing serfdom. If I can't make various governments and law enforcement agencies get warrants or court orders before they crack my private communications then I can at least require a court order before I takedown my own customer.

  (tags: city-of-london police takedowns politics mark-jeftovic easydns registrars dns via:tjmcintyre)

• Intellectual Ventures' Evil Knows No Bounds: Buys Patent AmEx Donated For Public Good... And Starts Suing

  The problem with software patents, part XVII.

  So you have a situation where even when the original patent holder donated the patent for "the public good," sooner or later, an obnoxious patent troll like IV comes along and turns it into a weapon. Again: AmEx patented those little numbers on your credit card, and then for the good of the industry and consumer protection donated the patent to a non-profit, who promised not to enforce the patent against banks... and then proceeded to sell the patent to Intellectual Ventures who is now suing banks over it.

  (tags: intellectual-ventures scams patents swpats shakedown banking cvv american-express banks amex cmaf)

• SPSC revisited part III - FastFlow + Sparse Data

  holy moly. This is some heavily-optimized mechanical-sympathy Java code. By using a sparse data structure, cache-aligned fields, and wait-free low-level CAS concurrency primitives via sun.misc.Unsafe, a single-producer/single-consumer queue implementation goes pretty damn fast compared to the current state of the art

  (tags: nitsanw optimization concurrency java jvm cas spsc queues data-structures algorithms)

• Non-blocking transactional atomicity

  interesting new distributed atomic transaction algorithm from Peter Bailis

  (tags: algorithms database distributed scalability storage peter-bailis distcomp)

• ZeroMQ: Helping us Block Malicious Domains in Real Time - Umbrella Security Labs

  nice writeup of a ZeroMQ/Hadoop event processing pipeline architecture

  (tags: zeromq hadoop event-processing architecture dns backend reputation)

Man sues RMV after driver's license mistakenly revoked by automated anti-terror false positive:

John H. Gass hadn’t had a traffic ticket in years, so the Natick resident was surprised this spring when he received a letter from the Massachusetts Registry of Motor Vehicles informing him to cease driving because his license had been revoked. [...] After frantic calls and a hearing with Registry officials, Gass learned the problem: An antiterrorism computerized facial recognition system that scans a database of millions of state driver’s license images had picked his as a possible fraud. “We send out 1,500 suspension letters every day," said Registrar Rachel Kaprielian. [...] “There are mistakes that can be made."

See also this New Scientist story. This story notes that the system's pretty widespread:

Massachusetts bought the system with a $1.5 million grant from the Department of Homeland Security. At least 34 states use such systems, which law enforcement officials say help prevent identity theft and ID fraud.

In my opinion, this kind of thing -- trial by inaccurate, false-positive-prone algorithm, is one of the most worrying things about the post-PRISM world.

When we created SpamAssassin, we were well aware of the risk of automated misclassification. Any machine-learning classifier will always make mistakes. The key is to carefully calibrate the expected false-positive/false-negative ratio so that the negative side-effects of a misclassification corresponds to the expected rate.

These anti-terrorism machine learning systems are calibrated to catch as many potential cases as possible, but by aiming to reduce false negatives to this degree, they become wildly prone to false positives. And when they're applied as a dragnet across all citizens' interactions with the state -- or even in the case of PRISM, all citizens' interactions that can be surveilled en masse -- it's going to create buckets of bureaucratic false-positive horror stories, as random innocent citizens are incorrectly tagged as criminals due to software bugs and poor calibration.

• Rapid read protection in Cassandra 2.0.2

  Nifty new feature -- if a request takes over the 99th percentile for requests to that server, it'll be repeated against another replica. Unnecessary for Voldemort, of course, which queries all replicas anyway!

  (tags: cassandra nosql replication distcomp latency storage)

• Attacking Tor: how the NSA targets users' online anonymity

  As part of the Turmoil system, the NSA places secret servers, codenamed Quantum, at key places on the internet backbone. This placement ensures that they can react faster than other websites can. By exploiting that speed difference, these servers can impersonate a visited website to the target before the legitimate website can respond, thereby tricking the target's browser to visit a Foxacid server.

  whoa, I missed this before.

  (tags: nsa gchq packet-injection attacks security backbone http latency)

• GCHQ report on 'MULLENIZE' program to 'stain' anonymous electronic traffic

  By modifying the User-Agent: header string, each HTTP transaction is "stained" to allow tracking. huh

  (tags: gchq nsa snooping sniffing surveillance user-agent http browsers leaks)

• Giving Docker/LXC containers a routable IP address

  ugh, this is a mess. Docker, automate this crap

  (tags: docker routing linux ops networking containers virtualization)

• How the feds took down the Dread Pirate Roberts | Ars Technica

  Well-written, comprehensive writeup of the Silk Road takedown, and the libertarian craziness of Ross William Ulbricht, it's alleged owner and operator

  (tags: silk-road drugs crazy ross-william-ulbricht fbi libertarian murder tor)

• Patent troll Lodsys chickens out, folds case rather than face Eugene Kaspersky

  In Kaspersky's view, patent trolls are no better than the extortionists who cropped up in Russia after the fall of the Soviet Union, when crime ran rampant. Kaspersky saw more and more people becoming victims of various extortion schemes. US patent trolls seemed very similar. "Kaspersky's view was that paying patent trolls was like paying a protection racket," said Kniser. He wasn't going to do it.

  yay! pity it didn't manage to establish precedent, though. But go Kaspersky!

  (tags: eugene-kaspersky shakedowns law east-texas swpats patents patent-trolls)

• Sergio Bossa's thoughts about Datomic

  good comments from Sergio, particularly about the scalability of the single transactor in the Datomic architecture. I agree it's a worrying design flaw

  (tags: clojure nosql datomic sergio-bossa transactor spof architecture storage)

• Codex Seraphinianus: A new edition of the strangest book in the world

  Excited! one commenter claims a paperback of the new edition of Luigi Serafini's masterwork should cost about $75 when it comes out in a couple of months. sign me up, this is an amazing work

  (tags: codex-seraphinianus art weird strange books luigi-serafini)

• The Snowden files: why the British public should be worried about GCHQ

  When the Guardian offered John Lanchester access to the GCHQ files, the journalist and novelist was initially unconvinced. But what the papers told him was alarming: that Britain is sliding towards an entirely new kind of surveillance society

  (tags: john-lanchester gchq guardian surveillance snooping police-state nsa privacy government)

• Groundbreaking Results for High Performance Trading with FPGA and x86 Technologies

  The enhancement in performance was achieved by providing a fast-path where trades are executed directly by the FPGA under the control of trigger rules processed by the x86 based functions. The latency is reduced further by two additional techniques in the FPGA – inline parsing and pre-emption. As market data enters the switch, the Ethernet frame is parsed serially as bits arrive, allowing partial information to be extracted and matched before the whole frame has been received. Then, instead of waiting until the end of a potential triggering input packet, pre-emption is used to start sending the overhead part of a response which contains the Ethernet, IP, TCP and FIX headers. This allows completion of an outgoing order almost immediately after the end of the triggering market feed packet.

  Insane stuff. (Via Martin Thompson)

  (tags: via:martin-thompson insane speed low-latency fpga fast-path trading stock-markets performance optimization ethernet)

• Why Tellybug moved from Cassandra to Amazon DynamoDB

  Summary: poor reliability, better latencies, and cheaper (!)

  (tags: aws dynamodb cassandra nosql storage tellybug counters scalability reliability latency)

• The Best Bike Lock

  Interviews with 2 New York bike thieves (one bottom feeder, one professional), reviewing the current batch of bicycle locks. Summary: U-locks are good, when used correctly, particularly the Kryptonite New York Lock ($80). On the other hand, Dublin's recent spate of thefts are largely driven by wide availability of battery-powered angle grinders (thanks Lidl!), which, according to this article, are relatively quiet and extremely fast. :(

  (tags: bike review locks cycling u-locks theft security)

• Fingerprints are Usernames, not Passwords

  I could see some value, perhaps, in a tablet that I share with my wife, where each of us have our own accounts, with independent configurations, apps, and settings.  We could each conveniently identify ourselves by our fingerprint.  But biometrics cannot, and absolutely must not, be used to authenticate an identity.  For authentication, you need a password or passphrase.  Something that can be independently chosen, changed, and rotated. [...] Once your fingerprint is compromised (and, yes, it almost certainly already is, if you've crossed an international border or registered for a driver's license in most US states), how do you change it?  Are you starting to see why this is a really bad idea?

  (tags: biometrics apple security fingerprints passwords authentication authorization identity)

• Silk Road busted

  This is a pretty good summary of the salient points from the criminal complaint against Ross William Ulbricht -- I'd say it's pretty bad news for any users of the dodgy site, particularly given this:

  "During the 60-day period from May 24, 2013 to July 23, 2013, there were approximately 1,217,218 communications sent between Silk Road users through Silk Road's private-message system."

  According to the complaint, those are now in the FBI's hands -- likely unencrypted.

  (tags: crime silk-road drugs busts tor ross-william-ulbricht fbi)

• Vitamin T: Hold the Salsa, New York Times! We've Got Something to Taco ‘Bout - Digest - Los Angeles magazine

  ouch. some serious slagging here, along with taco science. (BTW we have the same problem with carne asada in Ireland, our taquerias use the cheater method too, sadly)

  (tags: la tacos mexican food new-york slagging burritos taquerias carne-asada)

• Edward Snowden's E-Mail Provider Defied FBI Demands to Turn Over SSL Keys, Documents Show

  Levison lost [in secret court against the government's order]. In a work-around, Levison complied the next day by turning over the private SSL keys as an 11 page printout in 4-point type. The government called the printout “illegible” and the court ordered Levison to provide a more useful electronic copy.

  Nice try though! Bottom line is they demanded the SSL private key. (via Waxy)

  (tags: government privacy security ssl tls crypto fbi via:waxy secrecy snooping)

• Poisson Rouge: Crowdfunding Red Fish style

  the fantastic French kids' site is now crowdfunding new work -- first off being a German Alphabet part of the site. My kids love their stuff, so -- bonne chance!

  (tags: french poisson-rouge flash web kids children education)

• How an Engineer Earned 1.25 Million Air Miles By Buying Pudding

  An amazing hack. 'Air Miles are awesome, they can be used to score free flights, hotel stays and if you’re really lucky, the scorn and hatred of everyone you come in contact with who has to pay full price when they travel. The king of all virtually free travelers is one David Phillips, a civil engineer who teaches at the University of California, Davis. David came to the attention of the wider media when he managed to convert about 12,150 cups of Healthy Choice chocolate pudding [costing $3000] into over a million Air Miles. Ever since, David and his entire family have been travelling the world for next to nothing.' (via al3xandru)

  (tags: via:al3xandru hacks cool pudding small-print air-miles free)

• Down the Rabbit Hole

  An adventure that takes you through several popular Java language features and shows how they compile to bytecode and eventually JIT to assembly code.

  (tags: charles-nutter java jvm compilation reversing talks slides)

• Model checking for highly concurrent code

  Applied formal methods in order to test distributed systems -- specifically GlusterFS:

  I'll use an example from my own recent experience. I'm developing a new kind of replication for GlusterFS. To make sure the protocol behaves correctly even across multiple failures, I developed a Murphi model for it. [...] I added a third failure [to the simulated model]. I didn't expect a three-node system to continue working if more than one of those were concurrent (the model allows the failures to be any mix of sequential and concurrent), but I expected it to fail cleanly without reaching an invalid state. Surprise! It managed to produce a case where a reader can observe values that go back in time. This might not make much sense without knowing the protocol involved, but it might give some idea of the crazy conditions a model checker will find that you couldn't possibly have considered. [...] So now I have a bug to fix, and that's a good thing. Clearly, it involves a very specific set of ill-timed reads, writes, and failures. Could I have found it by inspection or ad-hoc analysis? Hell, no. Could I have found it by testing on live systems? Maybe, eventually, but it probably would have taken months for this particular combination to occur on its own. Forcing it to occur would require a lot of extra code, plus an exerciser that would amount to a model checker running 100x slower across machines than Murphi does. With enough real deployments over enough time it would have happened, but the only feasible way to prevent that was with model checking. These are exactly the kinds of bugs that are hardest to fix in the field, and that make users distrust distributed systems, so those of us who build such systems should use every tool at our disposal to avoid them.

  (tags: model-checking formal-methods modelling murphi distcomp distributed-systems glusterfs testing protocols)

• Is Trypophobia a Real Phobia? | Popular Science

  ie. "fear of small, clustered holes". Sounds like it's not so much a "phobia" as some kind of innate, visceral disgust response; I get it. 'As for who actually made the word up, that distinction probably belongs to a blogger in Ireland named Louise, Andrews says. According to an archived Geocities page, Louise settled on "trypophobia" (Greek for "boring holes" + "fear") after corresponding with a representative at the Oxford English Dictionary. Louise, Andrews and trypophobia Facebook group members have petitioned the dictionary to include the word. The term will need to be used for years and have multiple petitions and scholarly references before the dictionary accepts it, Andrews says. I, for one, would prefer to forget about it forever.'

  (tags: disgusting revulsion fear phobias trypophobia holes ugh innate)

• Common phobia you have never heard of: Fear of holes may stem from evolutionary survival response

  "We think that everyone has trypophobic tendencies even though they may not be aware of it," said Dr Cole. "We found that people who don't have the phobia still rate trypophobic images as less comfortable to look at than other images. It backs up the theory that we are set-up to be fearful of things which hurt us in our evolutionary past. We have an innate predisposition to be wary of things that can harm us."

  (tags: trypophobia holes fear aversion disgust ugh evolution innate)

• Mesosphere · Docker on Mesos

  This is cool. Deploy Docker container images onto a Mesos cluster: key point, in the description of the Redis example: 'there’s no need to install Redis or its supporting libraries on your Mesos hosts.'

  (tags: mesos docker deployment ops images virtualization containers linux)

• Call me maybe: Kafka

  Aphyr takes a look at Kafka 0.8's replication with the Jepsen test suite. It doesn't go great. Jay Kreps responds here: http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen

  (tags: jay-kreps kafka replication distributed-systems distcomp networking reliability fault-tolerance jepsen)

• The Hole in Our Collective Memory: How Copyright Made Mid-Century Books Vanish - Rebecca J. Rosen - The Atlantic

  A book published during the presidency of Chester A. Arthur has a greater chance of being in print today than one published during the time of Reagan.

  This is not a gently sloping downward curve. Publishers seem unwilling to sell their books on Amazon for more than a few years after their initial publication. The data suggest that publishing business models make books disappear fairly shortly after their publication and long before they are scheduled to fall into the public domain. Copyright law then deters their reappearance as long as they are owned. On the left side of the graph before 1920, the decline presents a more gentle time-sensitive downward sloping curve.

  (tags: business books legal copyright law public-domain reading history publishers amazon papers)

• Horse_ebooks is human after all

  Curated dissociated text. That's great

  (tags: ebooks art horse_ebooks internet twitter markov-chains)

• The Slow Winter

  (tags: coding funny processors multicore multiprocessing branch-prediction hardware)

• To my daughter's high school programming teacher

  During the first semester of my daughter's junior/senior year, she took her first programming class. She knew I'd be thrilled, but she did it anyway. When my daughter got home from the first day of the semester, I asked her about the class. "Well, I'm the only girl in class," she said. Fortunately, that didn't bother her, and she even liked joking around with the guys in class. My daughter said that you noticed and apologized to her because she was the only girl in class. And when the lessons started (Visual Basic? Seriously??), my daughter flew through the assigments. After she finished, she'd help classmates who were behind or struggling in class. Over the next few weeks, things went downhill. While I was attending SC '12 in Salt Lake City last November, my daughter emailed to tell me that the boys in her class were harassing her. "They told me to get in the kitchen and make them sandwiches," she said. I was painfully reminded of the anonymous men boys who left comments on a Linux Pro Magazine blog post I wrote a few years ago, saying the exact same thing.

  I am sick to death of this 'brogrammer' bullshit.

  (tags: brogrammers sexism culture tech teaching coding software education)

• "The cricket bat that died for Ireland"

  The bat had the misfortune of being on display in the shop front of Elvery’s store on O’Connell Street, then Sackville Street, during the Easter Rising. J.W. Elvery & Co. was Ireland’s oldest sports store, specialising in sporting goods and waterproofed wear, with branches in Dublin, Cork (Patrick Street) and London (Conduit Street). [...] Its location, about one block from the GPO, meant it was in the middle of the cross-fire and general destruction of the main street.

  (tags: ireland cricket 1916 history easter-rising crossfire sports elverys)

• _Availability in Globally Distributed Storage Systems_ [pdf]

  empirical BigTable and GFS failure numbers from Google are orders of magnitude higher than naïve independent-failure models. (via kragen)

  (tags: via:kragen failure bigtable gfs statistics outages reliability)

• Why We Hate Infographics (And Why You Should)

  YES. (via Des Traynor)

  (tags: via:destraynor infographics visualization dataviz graphics fail)

• Apple iOS 7 surprises as first with new multipath TCP connections - Network World

  iOS 7 includes -- and uses -- multipath TCP, right now for device-to-Siri communications.

  MPTCP is a TCP extension that enables the simultaneous use of several IP addresses or interfaces. Existing applications – completely unmodified -- see what appears to be a standard TCP interface. But under the covers, MPTCP is spreading the connection’s data across several subflows, sending it over the least congested paths.

  (tags: ios7 ios networking apple mptcp tcp protocols fault-tolerance)

• _How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP_ [pdf]

  (tags: mptcp tcp protocols networking ip)

• DynamoDB Local

  'a client-side database that supports the complete DynamoDB API, but doesn't manipulate any tables or data in DynamoDB itself. You can write code while sitting in a tree, on the beach, or in the desert. When you are ready to deploy your application, you simply instruct it to connect to the actual DynamoDB endpoint. No other modifications will be needed.' This is good -- an in-memory data store for integration testing is absolutely vital for production usage. (Voldemort does this well, for example.)

  (tags: dynamodb aws ec2 testing integration-testing unit-tests)

• Excellent Rob Pike quote about algorithmic complexity

  'Fancy algorithms are slow when n is small, and n is usually small.' -- Rob Pike

  Been there, bought the t-shirt ;)

  (tags: rob-pike quotes algorithms big-o complexity coding)

• Raft: The Understandable Distributed Consensus Protocol

  good slides explaining the Raft protocol

  (tags: raft slides presentation distcomp algorithms)

• RSA warns developers not to use RSA products

  In case you're missing the story here, Dual_EC_DRBG (which I wrote about yesterday) is the random number generator voted most likely to be backdoored by the NSA. The story here is that -- despite many valid concerns about this generator -- RSA went ahead and made it the default generator used for all cryptography in its flagship cryptography library. The implications for RSA and RSA-based products are staggering. In a modestly bad but by no means worst case, the NSA may be able to intercept SSL/TLS connections made by products implemented with BSafe.

  (tags: bsafe rsa crypto backdoors nsa security dual_ec_drbg rngs randomness)

• A Case Against Cucumber

  This is exactly my problem with Cucumber and similar BDD test frameworks.

  When I write a Cucumber feature, I have to write the Gherkin that describes the acceptance criteria, and the Ruby code that implements the step definitions. Since the code to implement the step definitions is just normal RSpec (or whichever testing library you use), if someone else is writing the Gherkin, the amount of setup to create a working test should be about the same. So you’re only breaking even! However, I don’t believe that it would really be breaking even. Cucumber adds another layer of indirection on top of your tests. When I’m trying to see why a specific scenario is failing, first I need to find the step that is failing. Since these steps are defined with regular expressions, I have to grep for the step definition.

  (tags: ruby testing bdd cucumber rspec coding)

• Gamasutra - Opinion: The tragedy of Grand Theft Auto V

  This is watching your sharp, witty father start telling old fart jokes as his mind slows down. And as much as the internet is habituated to defending GTA as "satire," what is it satirizing, if everything is either sad or awful? Where is the "satire" when the awful parts no longer seem edgy or provocative, just attempts at catch-all "offense" that aren't honed enough to even connect? Here's a series that has been creating real, meaningful friction with conventional entertainment for as long as I can remember, and rather than push the envelope by creating new kinds of monsters, it's reciting the same old gangland fantasies, like a college boy who can't stop staring at the Godfather II poster on his wall, talking about how he's gonna be a big Hollywood director in between bong rips. You call the trading index BAWSAQ? Oh, bro, you're so funny, you're gonna be huge.

  (tags: gamasutra games gaming gta gta-v via:skamille)

• CCC | Chaos Computer Club breaks Apple TouchID

  "We hope that this finally puts to rest the illusions people have about fingerprint biometrics. It is plain stupid to use something that you can´t change and that you leave everywhere every day as a security token", said Frank Rieger, spokesperson of the CCC. "The public should no longer be fooled by the biometrics industry with false security claims. Biometrics is fundamentally a technology designed for oppression and control, not for securing everyday device access." iPhone users should avoid protecting sensitive data with their precious biometric fingerprint not only because it can be easily faked, as demonstrated by the CCC team. Also, you can easily be forced to unlock your phone against your will when being arrested. Forcing you to give up your (hopefully long) passcode is much harder under most jurisdictions than just casually swiping your phone over your handcuffed hands.

• Piracy is a 'minority activity', pirates spend more on content, and piracy rates dropped in the UK during 2012

  OfCom has published a report on online piracy, which found that the practice is becoming less common and that pirates tend to spend more on legitimate content than non-pirates. The research, which was not funded by the entertainment industry, was conducted by Kantar Media among 21,474 participants and took place in 2012 across four separate stages. Over that time, the ratio of legal to illegal content fell -- confirming a suspected trend as legal streaming options became more available. It also confirmed another suspicion -- that a relatively small number of web users are responsible for most piracy. In OfCom's data, just two percent of users conducted three quarters of all piracy. Ofcom described piracy as "a minority activity". Of those surveyed, 58 percent accessed music, movie or TV content online, while 17 percent accessed illegal content sources. Those who admitted pirating content spent on average £26 every three months on legitimate content, set against an average spend of £16 among non-pirates.

  (tags: wired piracy studies ofcom streaming)

• Want to back an Irish Microbrewery?

  The excellent Trouble Brewing are looking for investors

  (tags: trouble-brewing ireland brewing beer business investment crowdfunding microbreweries)

• Tips for Tuning the Garbage First Garbage Collector

  (tags: g1gc gc java jvm tuning ops optimization)

• _An Improved Construction For Counting Bloom Filters_

  'A counting Bloom filter (CBF) generalizes a Bloom filter data structure so as to allow membership queries on a set that can be changing dynamically via insertions and deletions. As with a Bloom filter, a CBF obtains space savings by allowing false positives. We provide a simple hashing-based alternative based on d-left hashing called a d-left CBF (dlCBF). The dlCBF offers the same functionality as a CBF, but uses less space, generally saving a factor of two or more. We describe the construction of dlCBFs, provide an analysis, and demonstrate their effectiveness experimentally'

  (tags: bloom-filter data-structures algorithms counting cbf storage false-positives d-left-hashing hashing)

• To solve hard problems, you need to use bricolage

  In a talk about a neat software component he designed, Bruce Haddon observed that there is no way that the final structure and algorithmic behavior of this component could have been predicted, designed, or otherwise anticipated. Haddon observed that computer science serves as a source of core ideas: it provides the data structures and algorithms that are the building blocks. Meanwhile, he views software engineering as a useful set of methods to help design reliable software without losing your mind. Yet he points out that neither captures the whole experience. That’s because much of the work is what Haddon calls hacking, but what others would call bricolage. Simply put, there is much trial and error: we put ideas to together and see where it goes.

  This is a great post, and I agree (broadly). IMO, most software engineering requires little CS, but there are occasional moments where a single significant aspect of a project requires a particular algorithm, and would be kludgy, hacky, or over-complex to solve without it.

  (tags: bricolage hacking cs computer-science work algorithms)

• Getting Real About Distributed System Reliability

  I have come around to the view that the real core difficulty of [distributed] systems is operations, not architecture or design. Both are important but good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations. This is quite different from the view of unbreakable, self-healing, self-operating systems that I see being pitched by the more enthusiastic NoSQL hypesters. Worse yet, you can’t easily buy good operations in the same way you can buy good software—you might be able to hire good people (if you can find them) but this is more than just people; it is practices, monitoring systems, configuration management, etc.

  (tags: reliability nosql distributed-systems jay-kreps ops)

• Don't use Hadoop - your data isn't that big

  see also HN comments: https://news.ycombinator.com/item?id=6398650 , particularly davidmr's great one:

  I suppose all of this is to say that the amount of required parallelization of a problem isn't necessarily related to the size of the problem set as is mentioned most in the article, but also the inherent CPU and IO characteristics of the problem. Some small problems are great for large-scale map-reduce clusters, some huge problems are horrible for even bigger-scale map-reduce clusters (think fluid dynamics or something that requires each subdivision of the problem space to communicate with its neighbors). I've had a quote printed on my door for years: Supercomputers are an expensive tool for turning CPU-bound problems into IO-bound problems.

  I love that quote!

  (tags: hadoop big-data scaling map-reduce)

• Gilt Tech

  Gilt ran a stress-test of Riak to replace Voldemort (I think) in a shadow stack, with good results:

  Riak’s strong performance suggests that, should we pursue implementation, it will withstand our unique traffic needs and prove reliable. As for the Gilt-Basho team’s strong performance: It was amazing that we were able to accomplish so much in just a week’s time! Thanks again to Seth and Steve for making this possible.

  (tags: riak testing shadow-stack voldemort storage gilt)

• THE LONG DARK, a first-person post-disaster survival sim by Hinterland — Kickstarter

  wow this looks great.

  The Long Dark is a thoughtful, first-person survival simulation that emphasizes quiet exploration in a stark, yet hauntingly beautiful, post-disaster setting. The breathtakingly picturesque Pacific Northwest frames the backdrop for the drama of The Long Dark.

  (tags: games survival via:fp eclaire the-long-dark kickstarter)

• The Rational Choices of Crack Addicts - NYTimes.com

  “The key factor is the environment, whether you’re talking about humans or rats,” Dr. Hart said. “The rats that keep pressing the lever for cocaine are the ones who are stressed out because they’ve been raised in solitary conditions and have no other options. But when you enrich their environment, and give them access to sweets and let them play with other rats, they stop pressing the lever.”

  (tags: crack drugs policy science addiction society)

• Inside the mind of NSA chief Gen Keith Alexander | Glenn Greenwald

  featuring some mental pics of the "Information Dominance Center", the Star Trek bridge which NSA chief Keith Alexander built with taxpayer money

  (tags: big-brother nsa politics keith-alexander star-trek funny bizarre)

• Schneier on Security: Reforming the NSA

  Regardless of how we got here, the NSA can't reform itself. Change cannot come from within; it has to come from above. It's the job of government: of Congress, of the courts, and of the president. These are the people who have the ability to investigate how things became so bad, rein in the rogue agency, and establish new systems of transparency, oversight, and accountability. Any solution we devise will make the NSA less efficient at its eavesdropping job. That's a trade-off we should be willing to make, just as we accept reduced police efficiency caused by requiring warrants for searches and warning suspects that they have the right to an attorney before answering police questions. We do this because we realize that a too-powerful police force is itself a danger, and we need to balance our need for public safety with our aversion of a police state.

  (tags: nsa politics us-politics surveillance snooping society government police public-safety police-state)

• Biometric authentication failing in Mysore

  Biometrics was rolled out for food distribution in order to cut down on fraud, but it's now resulting in a subset of users being unable to authenticate:

  The biometric authentication system installed at the PDS outlets fails to establish the identity of many genuine beneficiaries, mostly workers, as their daily grind in the agricultural fields, construction sites or as domestic help have eroded the lines on their thumb resulting in distorted impressions.

  (tags: fail risks biometrics authentication mysore security india fingerprinting)

• Sketch of the Day – Frugal Streaming

  ha, this is very clever! If you have enough volume, this is a nice estimation algorithm to compute stream quantiles in very little RAM

  (tags: memory streaming stream-processing clever algorithms hacks streams)

• How not to stop spammers

  Spam Arrest is a company that sells an anti-spam service. They attempted to sue some spammers and, as has been widely reported, lost badly. This case emphasizes three points that litigious antispammers seem not to grasp: Under CAN SPAM, a lot of spam is legal. Judges hate plaintiffs who try to be too clever, and hate sloppy preparation even more. Never, ever, file a spam suit in Seattle.

  (tags: anti-spam spam law seattle us can-spam spamarrest sentient-jets)

• Benchmarking Redis on AWS ElastiCache

  good data points, but could do with latency percentiles

  (tags: latency redis measurement benchmarks ec2 elasticache aws storage tests)

• Being poor changes your thinking about everything

  Very interesting research into poverty and scarcity, in the Washington Post:

  The scarcity trap captures this notion we see again and again in many domains. When people have very little, they undertake behaviors that maintain or reinforce their future disadvantage. If you have very little, you often behave in such a way so that you'll have little in the future. In economics, people talk about the poverty trap. We're generalizing that, saying this happens a lot, and we've experienced it.

  (tags: poor poverty society economics scarcity washington-post)

• Good SSL for your website is absurdly difficult in practice

  Yet again, security software fails on packaging and UI. via Tony Finch

  (tags: security ssl tls packaging via:fanf)

• Former NSA and CIA director says terrorists love using Gmail

  At one point, Hayden expressed a distaste for online anonymity, saying "The problem I have with the Internet is that it's anonymous." But he noted, there is a struggle over that issue even inside government. The issue came to a head during the Arab Spring movement when the State Department was funding technology [presumably Tor?] to protect the anonymity of activists so governments could not track down or repress their voices. "We have a very difficult time with this," Hayden said. He then asked, "is our vision of the World Wide Web the global digital commons -- at this point you should see butterflies flying here and soft background meadow-like music -- or a global free fire zone?" Given that Hayden also compared the Internet to the wild west and Somalia, Hayden clearly leans toward the "global free fire zone" vision of the Internet.

  well, that's a good analogy for where we're going -- a global free-fire zone.

  (tags: gmail cia nsa surveillance michael-hayden security snooping law tor arab-spring)

• Google swaps out MySQL, moves to MariaDB

  When we asked Sallner to quantify the scale of the migration he said, "They're moving it all. Everything they have. All of the MySQL servers are moving to MariaDB, as far as I understand." By moving to MariaDB, Google can free itself of any dependence on technology dictated by Oracle – a company whose motivations are unclear, and whose track record for working with the wider technology community is dicey, to say the least. Oracle has controlled MySQL since its acquisition of Sun in 2010, and the key InnoDB storage engine since it got ahold of Innobase in 2005. [...] We asked Cole why Google would shift from MySQL to MariaDB, and what the key technical differences between the systems were. "From my perspective, they're more or less equivalent other than if you look at specific features and how they implement them," Cole said, speaking in a personal capacity and not on behalf of Google. "Ideologically there are lots of differences."

  So -- AWS, when will RDS offer MariaDB as an option?

  (tags: google mysql mariadb sql open-source licensing databases storage innodb oracle)

• FBI Admits It Controlled Tor Servers Behind Mass Malware Attack

  The code’s behavior, and the command-and-control server’s Virginia placement, is also consistent with what’s known about the FBI’s “computer and internet protocol address verifier,” or CIPAV, the law enforcement spyware first reported by WIRED in 2007. Court documents and FBI files released under the FOIA have described the CIPAV as software the FBI can deliver through a browser exploit to gather information from the target’s machine and send it to an FBI server in Virginia. The FBI has been using the CIPAV since 2002 against hackers, online sexual predators, extortionists, and others, primarily to identify suspects who are disguising their location using proxy servers or anonymity services, like Tor. Prior to the Freedom Hosting attack, the code had been used sparingly, which kept it from leaking out and being analyzed.

  (tags: cipav fbi tor malware spyware security wired)

• Creating Flight Recordings

  lots more detail on the new "Java Mission Control" feature in Hotspot 7u40 JVMs, and how to use it to start and stop profiling in a live, production JVM from a separate "jcmd" command-line client. If the overhead is small, this could be really neat -- turn on profiling for 1 minute every hour on a single instance, and collect realtime production profile data on an automated basis for post-facto analysis if required

  (tags: instrumentation logging profiling java jvm ops)

• Necessary and Proportionate -- In Which Civil Society is Caught Between a Cop and a Spy

  Modern telecommunications technology implied the development of modern telecommunications surveillance, because it moved the scope of action from the physical world (where intelligence, generally seen as part of the military mission, had acted) to the virtual world—including the scope of those actions that could threaten state power. While the public line may have been, as US Secretary of State Henry Stimson said in 1929, “gentlemen do not open each other’s mail”, you can bet that they always did keep a keen eye on the comings and goings of each other’s shipping traffic. The real reason that surveillance in the context of state intelligence was limited until recently was because it was too expensive, and it was too expensive for everyone. The Westphalian compromise demands equality of agency as tied to territory. As soon as one side gains a significant advantage, the structure of sovereignty itself is threatened at a conceptual level?—?hence Oppenheimer as the death of any hope of international rule of law. Once surveillance became cheap enough, all states were (and will increasingly be) forced to attempt it at scale, as a reaction to this pernicious efficiency. The US may be ahead of the game now, but Moore’s law and productization will work their magic here.

  (tags: government telecoms snooping gchq nsa surveillance law politics intelligence spying internet)

• Observability at Twitter

  Bit of detail into Twitter's TSD metric store.

  There are separate online clusters for different data sets: application and operating system metrics, performance critical write-time aggregates, long term archives, and temporal indexes. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired. Aggregation is generally performed at write-time to avoid extra storage operations for metrics that are expected to be immediately consumed. Indexing occurs along several dimensions–service, source, and metric names–to give users some flexibility in finding relevant data.

  (tags: twitter monitoring metrics service-metrics tsd time-series storage architecture cassandra)

• NSA: Possibly breaking US laws, but still bound by laws of computational complexity

  I didn’t clearly explain that there’s an enormous continuum between, on the one hand, a full break of RSA or Diffie-Hellman (which still seems extremely unlikely to me), and on the other, “pure side-channel attacks” involving no new cryptanalytic ideas.  Along that continuum, there are many plausible places where the NSA might be.  For example, imagine that they had a combination of side-channel attacks, novel algorithmic advances, and sheer computing power that enabled them to factor, let’s say, ten 2048-bit RSA keys every year.  In such a case, it would still make perfect sense that they’d want to insert backdoors into software, sneak vulnerabilities into the standards, and do whatever else it took to minimize their need to resort to such expensive attacks.  But the possibility of number-theoretic advances well beyond what the open world knows certainly wouldn’t be ruled out.  Also, as Schneier has emphasized, the fact that NSA has been aggressively pushing elliptic-curve cryptography in recent years invites the obvious speculation that they know something about ECC that the rest of us don’t.

  (tags: ecc rsa crypto security nsa gchq snooping sniffing diffie-hellman pki key-length)

• Low Overhead Method Profiling with Java Mission Control now enabled in the most recent HotSpot JVM release

  Built into the HotSpot JVM [in JDK version 7u40] is something called the Java Flight Recorder. It records a lot of information about/from the JVM runtime, and can be thought of as similar to the Data Flight Recorders you find in modern airplanes. You normally use the Flight Recorder to find out what was happening in your JVM when something went wrong, but it is also a pretty awesome tool for production time profiling. Since Mission Control (using the default templates) normally don’t cause more than a per cent overhead, you can use it on your production server.

  I'm intrigued by the idea of always-on profiling in production. This could be cool.

  (tags: performance java measurement profiling jvm jdk hotspot mission-control instrumentation telemetry metrics)

• How the NSA Spies on Smartphones

  One of the US agents' tools is the use of backup files established by smartphones. According to one NSA document, these files contain the kind of information that is of particular interest to analysts, such as lists of contacts, call logs and drafts of text messages. To sort out such data, the analysts don't even require access to the iPhone itself, the document indicates. The department merely needs to infiltrate the target's computer, with which the smartphone is synchronized, in advance. Under the heading "iPhone capability," the NSA specialists list the kinds of data they can analyze in these cases. The document notes that there are small NSA programs, known as "scripts," that can perform surveillance on 38 different features of the iPhone 3 and 4 operating systems. They include the mapping feature, voicemail and photos, as well as the Google Earth, Facebook and Yahoo Messenger applications.

  and, of course, the alternative means of backup is iCloud.... wonder how secure those backups are.

  (tags: nsa surveillance gchq iphone smartphones backups icloud security)

• Behind the Screens at Loggly

  Boost ASIO at the front end (!), Kafka 0.8, Storm, and ElasticSearch

  (tags: boost scalability loggly logging ingestion cep stream-processing kafka storm architecture elasticsearch)

• Schneier on Security: Excess Automobile Deaths as a Result of 9/11

  The inconvenience of extra passenger screening and added costs at airports after 9/11 cause many short-haul passengers to drive to their destination instead, and, since airline travel is far safer than car travel, this has led to an increase of 500 U.S. traffic fatalities per year. Using DHS-mandated value of statistical life at $6.5 million, this equates to a loss of $3.2 billion per year, or $32 billion over the period 2002 to 2011 (Blalock et al. 2007).

  (tags: risk security death 9-11 politics screening dhs air-travel driving road-safety)

• Perhaps I'm out of step and Britons just don't think privacy is important | Henry Porter | Comment is free | The Observer

  The debate has been stifled in Britain more successfully than anywhere else in the free world and, astonishingly, this has been with the compliance of a media and public that regard their attachment to liberty to be a matter of genetic inheritance. So maybe it is best for me to accept that the BBC, together with most of the newspapers, has moved with society, leaving me behind with a few old privacy-loving codgers, wondering about the cause of this shift in attitudes. Is it simply the fear of terror and paedophiles? Are we so overwhelmed by the power of the surveillance agencies that we feel we can't do anything? Or is it that we have forgotten how precious and rare truly free societies are in history?

  (tags: privacy uk politics snooping spies gchq society nsa henry-porter)

• Big data is watching you

  Some great street art from Brighton, via Darach Ennis

  (tags: via:darachennis street-art graffiti big-data snooping spies gchq nsa art)

• Blocking The Pirate Bay appears to have 'no lasting net impact' on illegal downloading

  In the fight against the unauthorised sharing of copyright protected material, aka piracy, Dutch Internet Service Providers have been summoned by courts to block their subscribers’ access to The Pirate Bay (TPB) and related sites. This paper studies the effectiveness of this approach towards online copyright enforcement, using both a consumer survey and a newly developed non-infringing technology for BitTorrent monitoring. While a small group of respondents download less from illegal sources or claim to have stopped, and a small but significant effect is found on the distribution of Dutch peers, no lasting net impact is found on the percentage of the Dutch population downloading from illegal sources.

  (tags: fail blocking holland pirate-bay tpb papers via:tjmcintyre internet isps)

• How Advanced Is the NSA's Cryptanalysis — And Can We Resist It?

  Bruce Schneier's suggestions:

  Assuming the hypothetical NSA breakthroughs don’t totally break public-cryptography — and that’s a very reasonable assumption — it’s pretty easy to stay a few steps ahead of the NSA by using ever-longer keys. We’re already trying to phase out 1024-bit RSA keys in favor of 2048-bit keys. Perhaps we need to jump even further ahead and consider 3072-bit keys. And maybe we should be even more paranoid about elliptic curves and use key lengths above 500 bits. One last blue-sky possibility: a quantum computer. Quantum computers are still toys in the academic world, but have the theoretical ability to quickly break common public-key algorithms — regardless of key length — and to effectively halve the key length of any symmetric algorithm. I think it extraordinarily unlikely that the NSA has built a quantum computer capable of performing the magnitude of calculation necessary to do this, but it’s possible. The defense is easy, if annoying: stick with symmetric cryptography based on shared secrets, and use 256-bit keys.

  (tags: bruce-schneier cryptography wired nsa surveillance snooping gchq cryptanalysis crypto future key-lengths)

• DevOps Eye for the Coding Guy: Metrics

  a pretty good description of the process of adding service metrics to a Django webapp using graphite and statsd. Bookmarking mainly for the great real-time graphing hack at the end...

  (tags: statsd django monitoring metrics python graphite)

• Probabalistic Scraping of Plain Text Tables

  a nifty hack.

  Recently I have been banging my head trying to import a ton of OCR acquired data expressed in tabular form. I think I have come up with a neat approach using probabilistic reasoning combined with mixed integer programming. The method is pretty robust to all sorts of real world issues. In particular, the method leverages topological understanding of tables, encodes it declaratively into a mixed integer/linear program, and integrates weak probabilistic signals to classify the whole table in one go (at sub second speeds). This method can be used for any kind of classification where you have strong logical constraints but noisy data.

  (via proggit)

  (tags: scraping tables ocr probabilistic linear-programming optimization machine-learning via:proggit)

• vimeo/timeserieswidget

  'Plugin to make highly interactive graphite graph objects ((i.e. graphs where you can interactively toggle on/off individual series, inspect datapoints, zoom in realtime, etc) Supports Flot (canvas), Rickshaw (svg) and standard graphite png images (in case you're nostalgic and don't like interactivity).'

  (tags: graphs graphing graphite dataviz flot rickshaw svg canvas javascript)

• modern JVM concurrency primitives are broken if the system clock steps backwards

  'The implementation of the concurrency primitive LockSupport.parkNanos(), the function that controls *every* concurrency primitive on the JVM, is flawed, and any NTP sync, or system time change, can potentially break it with unexpected results across the board when running a 64bit JVM on Linux 64bit.' Basically, LockSupport.parkNanos() calls pthread_cond_timedwait() using a CLOCK_REALTIME instead of CLOCK_MONOTONIC. 'tinker step 0' in ntp.conf may be a viable workaround.

  (tags: clocks timing ntp slew sync step pthreads java jvm timers clock_realtime clock_monotonic)

• Schneier on Security: The NSA Is Breaking Most Encryption on the Internet

  The new Snowden revelations are explosive. Basically, the NSA is able to decrypt most of the Internet. They're doing it primarily by cheating, not by mathematics. It's joint reporting between the Guardian, the New York Times, and ProPublica. I have been working with Glenn Greenwald on the Snowden documents, and I have seen a lot of them. These are my two essays on today's revelations. Remember this: The math is good, but math has no agency. Code has agency, and the code has been subverted.

  (tags: encryption communication government nsa security bruce-schneier crypto politics snooping gchq guardian journalism)

• How To Buffer Full YouTube Videos Before Playing

  summary - turn off DASH (Dynamic adaptive streaming) using a userscript.

  (tags: chrome youtube google video dash mpeg streaming)

• Voldemort on Solid State Drives [paper]

  'This paper and talk was given by the LinkedIn Voldemort Team at the Workshop on Big Data Benchmarking (WBDB May 2012).'

  With SSD, we find that garbage collection will become a very significant bottleneck, especially for systems which have little control over the storage layer and rely on Java memory management. Big heapsizes make the cost of garbage collection expensive, especially the single threaded CMS Initial mark. We believe that data systems must revisit their caching strategies with SSDs. In this regard, SSD has provided an efficient solution for handling fragmentation and moving towards predictable multitenancy.

  (tags: voldemort storage ssd disk linkedin big-data jvm tuning ops gc)

• Streaming MapReduce with Summingbird

  Before Summingbird at Twitter, users that wanted to write production streaming aggregations would typically write their logic using a Hadoop DSL like Pig or Scalding. These tools offered nice distributed system abstractions: Pig resembled familiar SQL, while Scalding, like Summingbird, mimics the Scala collections API. By running these jobs on some regular schedule (typically hourly or daily), users could build time series dashboards with very reliable error bounds at the unfortunate cost of high latency. While using Hadoop for these types of loads is effective, Twitter is about real-time and we needed a general system to deliver data in seconds, not hours. Twitter’s release of Storm made it easy to process data with very low latencies by sacrificing Hadoop’s fault tolerant guarantees. However, we soon realized that running a fully real-time system on Storm was quite difficult for two main reasons: Recomputation over months of historical logs must be coordinated with Hadoop or streamed through Storm with a custom log loading mechanism; Storm is focused on message passing and random-write databases are harder to maintain. The types of aggregations one can perform in Storm are very similar to what’s possible in Hadoop, but the system issues are very different. Summingbird began as an investigation into a hybrid system that could run a streaming aggregation in both Hadoop and Storm, as well as merge automatically without special consideration of the job author. The hybrid model allows most data to be processed by Hadoop and served out of a read-only store. Only data that Hadoop hasn’t yet been able to process (data that falls within the latency window) would be served out of a datastore populated in real-time by Storm. But the error of the real-time layer is bounded, as Hadoop will eventually get around to processing the same data and will smooth out any error introduced. This hybrid model is appealing because you get well understood, transactional behavior from Hadoop, and up to the second additions from Storm. Despite the appeal, the hybrid approach has the following practical problems: Two sets of aggregation logic have to be kept in sync in two different systems; Keys and values must be serialized consistently between each system and the client. The client is responsible for reading from both datastores, performing a final aggregation and serving the combined results Summingbird was developed to provide a general solution to these problems.

  Very interesting stuff. I'm particularly interested in the design constraints they've chosen to impose to achieve this -- data formats which require associative merging in particular.

  (tags: mapreduce streaming big-data twitter storm summingbird scala pig hadoop aggregation merging)

• Thoughts on Granby Park, the recent pop-up park off Parnell St

  We mentioned above that pop-up spaces have become popular across Europe because they allow developers and city councils to harness urban creativity in order to drive up real estate prices without ceding control of a given site. Those who produce the space through hard work, collaboration and passion move on, making way for property development and speculation. The international research in this area is very clear on this point and it has been documented in places from Lower-East Side Manhattan to Berlin’s Kreuzberg. Most perversely, increased property prices make it even more difficult for creativity to flourish in a given area and end up driving out long-term working class communities, migrants and young people. But what can we do? If every attempt we make to make our city a better place simply ends up being captured in the calculations of real estate players, surely the situation is hopeless? Is it better, then, to do nothing? We don’t think it is better to do nothing and, like Upstart, we still believe we can find a way together through experimentation and collaboration. However, this means questioning, reflecting on and publicly discussing the relationship between our efforts to make a city more after our hearts desire and the process of gentrification. As noted above, this is especially the case with pop-up spaces given their temporary nature. It is really necessary that we think about how to make sure our activities don’t contribute to gentrification in the long term, but instead benefit the city as a whole. We certainly don’t have the solutions, but if we sweep these awkward questions under the carpet we risk contributing to the very forces we want to challenge and alienating those who will perceive us as the ‘front-line’ of gentrification.

  (tags: gentrification pop-up parks dublin ireland cities upstart spaces urban-planning)

• [#CASSANDRA-5582] Replace CustomHsHaServer with better optimized solution based on LMAX Disruptor

  Disruptor: decimating P99s since 2011

  (tags: disruptor cassandra java p99 latency speed performance concurrency via:kellabyte)

• Time is a Dimension

  I love these.

  Photographic prints are great because they don’t need power to be displayed. They are more or less permanent. Videos are great because they record a sequence of time which shows reality almost like how we experience. Is it possible to combine the two? And not via long exposure photography where often details are lost from motion. So I played around with the tools of digital photography and post processing to give you this series: Time is a dimension. This series of images are mostly landscapes, seascapes and cityscapes, and they are a single composite made from sequences that span 2-4 hours, mostly of sunrises and sunsets. The basic structure of a landscape is present in every piece. But each panel or concentric layer shows a different slice of time, which is related to the adjacent panel/layer. The transition from daytime to night is gradual and noticeable in every piece, but would not be something you expect to see in a still image.

  (tags: photography beautiful photos art time dimensions prints via:matthaughey)

• WTF Visualizations

  'Visualizations that make no sense.' Some of these are unintentional comedy gold -- pie charts feature heavily, of course. (via Des Traynor)

  (tags: via:destraynor infographics wtf visualization dataviz data fail funny graphics pie-charts)

• Non-blocking transactional atomicity

  Peter Bailis with an interesting distributed-storage atomicity algorithm for performing multi-record transactional updates

  (tags: algorithms nbta transactions databases storage distcomp distributed atomic coding eventual-consistency crdts)

• Interview with the Github Elasticsearch Team

  good background on Github's Elasticsearch scaling efforts. Some rather horrific split-brain problems under load, and crashes due to OpenJDK bugs (sounds like OpenJDK *still* isn't ready for production). painful

  (tags: elasticsearch github search ops scaling split-brain outages openjdk java jdk jvm)

• The Irish Times, terminations and Holles Street: The story that wasn’t there.

  Summarising a very shoddy tale from our paper of record.

  I don’t know what happened here. I don’t know whether there ever was a woman who met the description given by the Irish Times who suffered a medical crisis during pregnancy. I don’t know why a group of men in positions of authority in the Irish Times decided that, if there was such a woman, they had any right to tell the rest of the country about her experiences. I don’t know why, when they discovered that a mistake had been made in the one legal fact used to justify that decision they didn’t immediately apologise. And I don’t know what happened between the 23rd August 2013 and 31st August 2013 to prompt them to print a shoulder shrugging ‘acceptance’ that the case ‘hadn’t happened’ and limit the paper’s apology to an institution, as opposed to its readers. But, from what I’ve seen this week, I do know one thing. Whatever questions readers might have, The Irish Times isn’t interested in giving them any answers.

  (tags: irish-times fail shoddy abortion health public-interest journalism pregnancy corrections)

• Blueflood by rackerlabs

  Rackspace's large-scale TSD storage system, built on Cassandra, Java, ASL2

  (tags: cassandra tsd storage time-series data open-source java rackspace)

• Reversing Sinclair's amazing 1974 calculator hack - half the ROM of the HP-35

  Amazing reverse engineering.

  In a hotel room in Texas, Clive Sinclair had a big problem. He wanted to sell a cheap scientific calculator that would grab the market from expensive calculators such as the popular HP-35. Hewlett-Packard had taken two years, 20 engineers, and a million dollars to design the HP-35, which used 5 complex chips and sold for $395. Sinclair's partnership with calculator manufacturer Bowmar had gone nowhere. Now Texas Instruments offered him an inexpensive calculator chip that could barely do four-function math. Could he use this chip to build a $100 scientific calculator? Texas Instruments' engineers said this was impossible - their chip only had 3 storage registers, no subroutine calls, and no storage for constants such as ?. The ROM storage in the calculator held only 320 instructions, just enough for basic arithmetic. How could they possibly squeeze any scientific functions into this chip? Fortunately Clive Sinclair, head of Sinclair Radionics, had a secret weapon - programming whiz and math PhD Nigel Searle. In a few days in Texas, they came up with new algorithms and wrote the code for the world's first single-chip scientific calculator, somehow programming sine, cosine, tangent, arcsine, arccos, arctan, log, and exponentiation into the chip. The engineers at Texas Instruments were amazed. How did they do it? Up until now it's been a mystery. But through reverse engineering, I've determined the exact algorithms and implemented a simulator that runs the calculator's actual code. The reverse-engineered code along with my detailed comments is in the window below.

  (tags: reversing reverse-engineering history calculators sinclair ti hp chips silicon hacks)

• Microsoft CEO Steve Ballmer retires: A firsthand account of the company’s employee-ranking system

  LOL MS. Sadly, this talk of "core competencies" and "visibility" is pretty reminiscent of Amazon's review season, too:

  This illustrated another problem with [stack ranking]: It destroyed trust between individual contributors and management, because the stack rank required that all lower-level managers systematically lie to their reports. Why? Because for years Microsoft did not admit the existence of the stack rank to nonmanagers. Knowledge of the process gradually leaked out, becoming a recurrent complaint on the much-loathed (by Microsoft) Mini-Microsoft blog, where a high-up Microsoft manager bitterly complained about organizational dysfunction and was joined in by a chorus of hundreds of employees. The stack rank finally made it into a Vanity Fair article in 2012, but for many years it was not common knowledge, inside or outside Microsoft. It was presented to the individual contributors as a system of objective assessment of “core competencies,” with each person being judged in isolation. When review time came, and programmers would fill out a short self-assessment talking about their achievements, strengths, and weaknesses, only some of them knew that their ratings had been more or less already foreordained at the stack rank. [...] If you did know about the stack rank, you weren’t supposed to admit it. So you went through the pageantry of the performance review anyway, arguing with your manager in the rhetoric of “core competencies.” The managers would respond in kind. Since the managers had little control over the actual score and attendant bonus and raise (if any), their job was to write a review to justify the stack rank in the language of absolute merit. (“Higher visibility” was always a good catch-all: Sure, you may be a great coder and work 80 hours a week, but not enough people have heard of you!)

  (tags: amazon stack-ranking employees ranking work microsoft core-competencies)

• BBC News - How one man turns annoying cold calls into cash

  This is hilarious. Quid pro quo!

  Once he had set up the 0871 line, every time a bank, gas or electricity supplier asked him for his details online, he submitted it as his contact number. He added he was "very honest" and the companies did ask why he had a premium number. He told the programme he replied: "Because I'm getting annoyed with PPI phone calls when I'm trying to watch Coronation Street so I'd rather make 10p a minute." He said almost all of the companies he dealt with were happy to use it and if they refused he asked them to email.

  (tags: spam cold-calls phone ads uk funny 0871 premium-rate ppi)

• The Edge Minecraft cover

  This is brilliant. Half of the office now wants prints.

  Massive congratulations to Edge magazine. The stellar publication has been around for 20 years! To celebrate, their 258th issue comes in 20 different flavours, and one of those flavours includes the earthly overtones of both Minecraft and Dungeons & Dragons. Junkboy drew it, and I [Owen] worded it a few weeks ago.

  (tags: covers images edge minecraft gaming funny dungeons-and-dragons retro dnd)

• Forecast Blog

  Forecast.io are doing such a great job of applying modern machine-learning to traditional weather data. "Quicksilver" is their neural-net-adjusted global temperature geodata, and here's how it's built

  (tags: quicksilver forecast forecast.io neural-networks ai machine-learning algorithms weather geodata earth temperature)

• _MillWheel: Fault-Tolerant Stream Processing at Internet Scale_ [paper, pdf]

  from VLDB 2013:

  MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework’s fault-tolerance guarantees. This paper describes MillWheel’s programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel’s features are used. MillWheel’s programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel’s unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.

  (tags: millwheel google data-processing cep low-latency fault-tolerance scalability papers event-processing stream-processing)

• GCHQ tapping at least 14 EU fiber-optic cables

  Süddeutsche Zeitung (SZ) had already revealed in late June that the British had access to the cable TAT-14, which connects Germany with the USA, UK, Denmark, France and the Netherlands. In addition to TAT-14, the other cables that GCHQ has access to include Atlantic Crossing 1, Circe North, Circe South, Flag Atlantic-1, Flag Europa-Asia, SeaMeWe-3 and SeaMeWe-4, Solas, UK France 3, UK Netherlands-14, Ulysses, Yellow and the Pan European Crossing.

  (tags: sz germany cables fiber-optic tapping snooping tat-14 eu politics gchq)

• In historic vote, New Zealand bans software patents | Ars Technica

  This is amazing news. Paying attention, Sean Sherlock?

  A major new patent bill, passed in a 117-4 vote by New Zealand's Parliament after five years of debate, has banned software patents. The relevant clause of the patent bill actually states that a computer program is "not an invention." Some have suggested that was a way to get around the wording of the TRIPS intellectual property treaty, which requires patents to be "available for any inventions, whether products or processes, in all fields of technology." [...] One Member of Parliament who was deeply involved in the debate, Clare Curran, quoted several heads of software firms complaining about how the patenting process allowed "obvious things" to get patented and that "in general software patents are counter-productive." Curran quoted one developer as saying, "It's near impossible for software to be developed without breaching some of the hundreds of thousands of patents granted around the world for obvious work." "These are the heavyweights of the new economy in software development," said Curran. "These are the people that needed to be listened to, and thankfully, they were."

  (tags: new-zealand nz patents swpats law trips ip software-patents yay)

• Docker: Git for deployment

  Docker is to deployment as Git is to development. Developers are able to leverage Git's performance and flexibility when building applications. Git encourages experiments and doesn't punish you when things go wrong: start your experiments in a branch, if things fall down, just git rebase or git reset. It's easy to start a branch and fast to push it. Docker encourages experimentation for operations. Containers start quickly. Building images is a snap. Using another images as a base image is easy. Deploying whole images is fast, and last but not least, it's not painful to rollback. Fast + flexible = deployments are about to become a lot more enjoyable.

  (tags: docker deployment sysadmin ops devops vms vagrant virtualization containers linux git)

• Using set cover algorithm to optimize query latency for a large scale distributed graph | LinkedIn Engineering

  how LI solved a tricky graph-database-query latency problem with a set-cover algorithm

  (tags: linkedin algorithms coding distributed-systems graph databases querying set-cover set replication)

• How might the feds have snooped on Lavabit?

  "I have been told that they cannot change your fundamental business practices," said Callas, who unlike Levison was able to say SilentCircle has received no NSLs or court orders of any kind. "I presume that would mean things like getting SSL keys because that would mean they could impersonate your servers. That would be like setting up a store front that says your business name and putting [government agents] in your company uniforms." Similarly, he added: "They cannot make changes to existing operating systems. They can't make you change source code." To which [Lavabit's] Levison replied: "That was always my understanding, too. That's why this is so important. Like [Callas] at SilentCircle said, the assumption has been that the government can't force us to change our business practices like that and compromise that information. Like I said, I don't hold those beliefs anymore."

  (tags: ars-technica security privacy nsls ssl silentcircle jon-callas crypto)

• Lock-Based vs Lock-Free Concurrent Algorithms

  An excellent post from Martin Thompson showing a new JSR166 concurrency primitive, StampedLock, compared against a number of alternatives in a simple microbenchmark. The most interesting thing for me is how much the lock-free, AtomicReference.compareAndSet()-based approach blows away all the lock-based approaches -- even in the 1-reader-1-writer case. Its code is extremely simple, too: https://github.com/mjpt777/rw-concurrency/blob/master/src/LockFreeSpaceship.java

  (tags: concurrency java threads lock-free locking compare-and-set cas atomic jsr166 microbenchmarks performance)

• Juniper Adds Puppet support

  This is super-cool. 'Network engineering no longer should be mundane tasks like conf, set interfaces fe-0/0/0 unit o family inet address 10.1.1.1/24. How does mindless CLI work translate to efficiently spent time ? What if you need to change 300 devices? What if you are writing it by hand? An error-prone waste of time. Juniper today announced Puppet support for their 12.2R3,5 JUNOS code. This is compatible with EX4200, EX4550, and QFX3500 switches. These are top end switches, but this start is directly aimed at their DC and enterprise devices. Initially, the manifest interactions offered are interface, layer 2 interface, vlan, port aggregation groups, and device names.' Based on what I saw in the Network Automation team in Amazon, this is an amazing leap forward; it'd instantly render obsolete a bunch of horrific SSH-CLI automation cruft.

  (tags: ssh cli automation networking networks puppet ops juniper cisco)

• awscli

  The future of the AWS command line tools is awscli, a single, unified, consistent command line tool that works with almost all of the AWS services. Here is a quick list of the services that awscli currently supports: Auto Scaling, CloudFormation, CloudSearch, CloudWatch, Data Pipeline, Direct Connect, DynamoDB, EC2, ElastiCache, Elastic Beanstalk, Elastic Transcoder, ELB, EMR, Identity and Access Management, Import/Export, OpsWorks, RDS, Redshift, Route 53, S3, SES, SNS, SQS, Storage Gateway, Security Token Service, Support API, SWF, VPC. Support for the following appears to be planned: CloudFront, Glacier, SimpleDB. The awscli software is being actively developed as an open source project on Github, with a lot of support from Amazon. You’ll note that the biggest contributors to awscli are Amazon employees with Mitch Garnaat leading. Mitch is also the author of boto, the amazing Python library for AWS.

  (tags: aws awscli cli tools command-line ec2 s3 amazon api)

• Let Me Explain Why Miley Cyrus’ VMA Performance Was Our Top Story This Morning | The Onion - America's Finest News Source

  Absolute genius from The Onion.

  Those of us watching on Google Analytics saw the number of homepage visits skyrocket the second we put up that salacious image of Miley Cyrus dancing half nude on the VMA stage. But here’s where it gets great: We don’t just do a top story on the VMA performance and call it a day. No, no. We also throw in a slideshow called “Evolution of Miley,” which, for those of you who don’t know, is just a way for you to mindlessly click through 13 more photos of Miley Cyrus. And if we get 500,000 of you to do that, well, 500,000 multiplied by 13 means we can get 6.5 million page views on that slideshow alone. Throw in another slideshow titled “6 ‘don’t miss’ VMA moments,” and it’s starting to look like a pretty goddamned good Monday, numbers-wise. Also, there are two videos -- one of the event and then some bullshit two-minute clip featuring our “entertainment experts” talking about the performance. Side note: Advertisers, along with you idiots, love videos. Another side note: The Miley Cyrus story was in the same top spot we used for our 9/11 coverage.

  (tags: humor journalism cnn miley-cyrus vma news funny advertising ads)

• Why wireless mesh networks won't save us from censorship

  I'm not saying mesh networks don't work ever; the people in the wireless mesh community I've met are all great people doing fantastic work. What I am saying is that unplanned wireless mesh networks never work at scale. I think it's a great problem to think about, but in terms of actual allocation of time and resources I think there are other, more fruitful avenues of action to fight Internet censorship.

  (via Kragen)

  (tags: wireless censorship internet networking mesh mesh-networks organisation scaling wifi)

• Information on Google App Engine's recent US datacenter relocations - Google Groups

  or, really, 'why we had some glitches and outages recently'. A few interesting tidbits about GAE innards though (via Bill De hOra)

  (tags: gae google app-engine outages ops paxos eventual-consistency replication storage hrd)

• Newest YouTube user to fight a takedown is copyright guru Lawrence Lessig

  This is lovely. Here's hoping it provides a solid precedent.

  Illegitimate or simply unnecessary copyright claims are, unfortunately, commonplace in the Internet era. But if there's one person who's probably not going to back down from a claim of copyright infringement, it's Larry Lessig, one of the foremost writers and thinkers on digital-age copyright. [..] If Liberation Music was thinking they'd have an easy go of it when they demanded that YouTube take down a 2010 lecture of Lessig's entitled "Open," they were mistaken. Lessig has teamed up with the Electronic Frontier Foundation to sue Liberation, claiming that its overly aggressive takedown violates the DMCA and that it should be made to pay damages.

  (tags: liberation-music eff copyright law larry-lessig fair-use)

• TCP is UNreliable

  Great account from Cliff Click describing an interest edge-case risk of using TCP without application-level acking, and how it caused a messy intermittent bug in production.

  In all these failures the common theme is that the receiver is very heavily loaded, with many hundreds of short-lived TCP connections being opened/read/closed every second from many other machines.  The sender sends a ‘SYN’ packet, requesting a connection. The sender (optimistically) sends 1 data packet; optimistic because the receiver has yet to acknowledge the SYN packet.  The receiver, being much overloaded, is very slow.  Eventually the receiver returns a ‘SYN-ACK’ packet, acknowledging both the open and the data packet.  At this point the receiver’s JVM has not been told about the open connection; this work is all opening at the OS layer alone.  The sender, being done, sends a ‘FIN’ which it does NOT wait for acknowledgement (all data has already been acknowledged).  The receiver, being heavily overloaded, eventually times-out internally (probably waiting for the JVM to accept the open-call, and the JVM being overloaded is too slow to get around to it) – and sends a RST (reset) packet back…. wiping out the connection and the data.  The sender, however, has moved on – it already sent a FIN & closed the socket, so the RST is for a closed connection.  Net result: sender sent, but the receiver reset the connection without informing either the JVM process or the sender.

  (tags: tcp protocols SO_LINGER FIN RST connections cliff-click ip)

• The ultimate SO_LINGER page, or: why is my tcp not reliable

  If we look at the HTTP protocol, there data is usually sent with length information included, either at the beginning of an HTTP response, or in the course of transmitting information (so called ‘chunked’ mode). And they do this for a reason. Only in this way can the receiving end be sure it received all information that it was sent. Using the shutdown() technique above really only tells us that the remote closed the connection. It does not actually guarantee that all data was received correctly by program B. The best advice is to send length information, and to have the remote program actively acknowledge that all data was received.

  (tags: SO_LINGER sockets tcp ip networking linux protocols shutdown FIN RST)

• NZ police affidavits show use of PRISM for surveillance of Kim "Megaupload" Dotcom

  The discovery was made by blogger Keith Ng who wrote on his On Point blog (http://publicaddress.net/onpoint/ich-bin-ein-cyberpunk/) that the Organised and Financial Crime Agency New Zealand (OFCANZ) requested assistance from the Government Communications Security Bureau (GCSB), the country's signals intelligence unit, which is charge of surveilling the Pacific region under the Five-Eyes agreement. A list of so-called selectors or search terms were provided to GCSB by the police [PDF, redacted] for the surveillance of emails and other data traffic generated by Dotcom and his Megaupload associates. 'Selectors' is the term used for the National Security Agency (NSA) XKEYSCORE categorisation system that Australia and New Zealand contribute to and which was leaked by Edward Snowden as part of his series of PRISM revelations. Some "selectors of interest" have been redacted out, but others such as Kim Dotcom's email addresses, the mail proxy server used for some of the accounts and websites, remain in the documents.

  So to recap; police investigating an entirely non-terrorism-related criminal case in NZ was given access to live surveillance traffic for surveillance of an NZ citizen. Scary stuff

  (tags: surveillance prism nsa new-zealand xkeyscore gcsb kim-dotcom piracy privacy data-retention megaupload filesharing)

• "Scalable Eventually Consistent Counters over Unreliable Networks" [paper, pdf]

  Counters are an important abstraction in distributed computing, and play a central role in large scale geo-replicated systems, counting events such as web page impressions or social network "likes". Classic distributed counters, strongly consistent, cannot be made both available and partition-tolerant, due to the CAP Theorem, being unsuitable to large scale scenarios. This paper defines Eventually Consistent Distributed Counters (ECDC) and presents an implementation of the concept, Handoff Counters, that is scalable and works over unreliable networks. By giving up the sequencer aspect of classic distributed counters, ECDC implementations can be made AP in the CAP design space, while retaining the essence of counting. Handoff Counters are the first CRDT (Conflict-free Replicated Data Type) based mechanism that overcomes the identity explosion problem in naive CRDTs, such as G-Counters (where state size is linear in the number of independent actors that ever incremented the counter), by managing identities towards avoiding global propagation, and garbage collecting temporary entries. The approach used in Handoff Counters is not restricted to counters, being more generally applicable to other data types with associative and commutative operations.

  (tags: pdf papers eventual-consistency counters distributed-systems distcomp cap-theorem ecdc handoff-counters crdts data-structures g-counters)

• LMDB response to a LevelDB-comparison blog post

  This seems like a good point to note about LMDB in general:

  We state quite clearly that LMDB is read-optimized, not write-optimized. I wrote this for the OpenLDAP Project; LDAP workloads are traditionally 80-90% reads. Write performance was not the goal of this design, read performance is. We make no claims that LMDB is a silver bullet, good for every situation. It’s not meant to be – but it is still far better at many things than all of the other DBs out there that *do* claim to be good for everything.

  (tags: lmdb leveldb databases openldap storage persistent)

• How to avoid crappy ISP caches when viewing YouTube video

  Must give this a try when I get home -- I frequently have latency problems watching YT on my UPC connection, and I bet they have a crappily-managed, overloaded cache box on their network.

  (tags: streaming youtube caching isps caches firewalls iptables hacks video networking)

• How to configure ntpd so it will not move time backwards

  The "-x" switch will expand the step/slew boundary from 128ms to 600 seconds, ensuring the time is slewed (drifted slowly towards the correct time at a max of 5ms per second) rather than "stepped" (a sudden jump, potentially backwards). Since slewing has a max of 5ms per second, time can never "jump backwards", which is important to avoid some major application bugs (particularly in Java timers).

  (tags: ntpd time ntp ops sysadmin slew stepping time-synchronization linux unix java bugs)

• Snowizard

  'a Java port of Twitter's Snowflake thrift service presented as an HTTP-based Dropwizard service'.

  an HTTP-based service for generating unique ID numbers at high scale with some simple guarantees. supports returning ID numbers as: JSON and JSONP; Google's Protocol Buffers; Plain text. At GE, we were more interested in the uncoordinated aspects of Snowflake than its throughput requirements, so HTTP was fine for our needs. We also exposed the core of Snowflake as an embeddable module so it can be directly integrated into our applications. We don't have the guarantees that the Snowflake-Zookeeper integration was providing, but that was also acceptable to us. In places where we really needed high throughput, we leveraged the snowizard-core embeddable module directly.

  Odd OSS license, though -- BSDish?

  (tags: java open-source ids soa services snowflake http)

• Containers and Docker: How Secure Are They?

  pretty extensive article. (via Tony Finch)

  (tags: via:fanf security containerization docker containers lxc linux ops)

• Groklaw - Forced Exposure ~pj

  I loved doing Groklaw, and I believe we really made a significant contribution. But even that turns out to be less than we thought, or less than I hoped for, anyway. My hope was always to show you that there is beauty and safety in the rule of law, that civilization actually depends on it. How quaint. If you have to stay on the Internet, my research indicates that the short term safety from surveillance, to the degree that is even possible, is to use a service like Kolab for email, which is located in Switzerland, and hence is under different laws than the US, laws which attempt to afford more privacy to citizens. I have now gotten for myself an email there, p.jones at mykolab.com in case anyone wishes to contact me over something really important and feels squeamish about writing to an email address on a server in the US. But both emails still work. It's your choice. My personal decision is to get off of the Internet to the degree it's possible. I'm just an ordinary person. But I really know, after all my research and some serious thinking things through, that I can't stay online personally without losing my humanness, now that I know that ensuring privacy online is impossible. I find myself unable to write. I've always been a private person. That's why I never wanted to be a celebrity and why I fought hard to maintain both my privacy and yours. Oddly, if everyone did that, leap off the Internet, the world's economy would collapse, I suppose. I can't really hope for that. But for me, the Internet is over. So this is the last Groklaw article. I won't turn on comments. Thank you for all you've done. I will never forget you and our work together. I hope you'll remember me too. I'm sorry I can't overcome these feelings, but I yam what I yam, and I tried, but I can't.

  (tags: nsa surveillance privacy groklaw law us-politics data-protection snooping mail kolab)

• Nelson's Weblog: tech / bad / failure-of-encryption

  One of the great failures of the Internet era has been giving up on end-to-end encryption. PGP dates back to 1991, 22 years ago. It gave us the technical means to have truly secure email between two people. But it was very difficult to use. And in 22 years no one has ever meaningfully made email encryption really usable. [...] We do have SSL/HTTPS, the only real end-to-end encryption most of us use daily. But the key distribution is hopelessly centralized, authority rooted in 40+ certificates. At least 4 of those certs have been compromised by blackhat hackers in the past few years. How many more have been subverted by government agencies? I believe the SSL Observatory is the only way we’d know.

  We do also have SSH. Maybe more services need to adopt that model?

  (tags: ssh ssl tls pki crypto end-to-end pgp security surveillance)

• Recordinality

  a new, and interesting, sketching algorithm, with a Java implementation:

  Recordinality is unique in that it provides cardinality estimation like HLL, but also offers "distinct value sampling." This means that Recordinality can allow us to fetch a random sample of distinct elements in a stream, invariant to cardinality. Put more succinctly, given a stream of elements containing 1,000,000 occurrences of 'A' and one occurrence each of 'B' - 'Z', the probability of any letter appearing in our sample is equal. Moreover, we can also efficiently store the number of times elements in our distinct sample have been observed. This can help us to understand the distribution of occurrences of elements in our stream. With it, we can answer questions like "do the elements we've sampled present in a power law-like pattern, or is the distribution of occurrences relatively even across the set?"

  (tags: sketching coding algorithms recordinality cardinality estimation hll hashing murmurhash java)

• You can't "waste your vote"!

  A fantastic infographic explaining Australia's Preferential Voting system, featuring Dennis the Election Koala and Ken the Voting Dingo

  (tags: infographics funny pr voting australia images via:fp)

• David Miranda, schedule 7 and the danger that all reporters now face | Alan Rusbridger | Comment is free | The Guardian

  The man was unmoved. And so one of the more bizarre moments in the Guardian's long history occurred – with two GCHQ security experts overseeing the destruction of hard drives in the Guardian's basement just to make sure there was nothing in the mangled bits of metal which could possibly be of any interest to passing Chinese agents. "We can call off the black helicopters," joked one as we swept up the remains of a MacBook Pro. Whitehall was satisfied, but it felt like a peculiarly pointless piece of symbolism that understood nothing about the digital age. We will continue to do patient, painstaking reporting on the Snowden documents, we just won't do it in London. The seizure of Miranda's laptop, phones, hard drives and camera will similarly have no effect on Greenwald's work. The state that is building such a formidable apparatus of surveillance will do its best to prevent journalists from reporting on it. Most journalists can see that. But I wonder how many have truly understood the absolute threat to journalism implicit in the idea of total surveillance, when or if it comes – and, increasingly, it looks like "when". We are not there yet, but it may not be long before it will be impossible for journalists to have confidential sources. Most reporting – indeed, most human life in 2013 – leaves too much of a digital fingerprint. Those colleagues who denigrate Snowden or say reporters should trust the state to know best (many of them in the UK, oddly, on the right) may one day have a cruel awakening. One day it will be their reporting, their cause, under attack. But at least reporters now know to stay away from Heathrow transit lounges.

  (tags: nsa gchq surveillance spying snooping guardian reporters journalism uk david-miranda glenn-greenwald edward-snowden)

• al3x/sovereign

  'Sovereign is a set of Ansible playbooks that you can use to build and maintain' your own GMail/Google calendar/etc. on a VPS. Some up-to-date hosting tips, basically

  (tags: sovereign gmail google vps ansible al3x hosting)

• New Tweets per second record, and how | Twitter Blog

  How Twitter scaled up massively in 3 years -- replacing Ruby with the JVM, adopting SOA and custom sharding. Good summary post, looking forward to more techie details soon

  (tags: twitter performance scalability jvm ruby soa scaling)

• Massive Overblocking Hits Hundreds Of UK Sites | Techdirt

  Customers of UK ISPs Virgin Media and Be Broadband found they were unable to access hundreds of sites, including the Radio Times and Zooniverse, due to a secret website-blocking court order from the Premier League. PC Pro believe that 3 other ISPs' customers were also affected. According to customers reverse-engineering, it looks like the court order incorrectly demanded the blocking of "http-redirection-a.dnsmadeeasy.com", a HTTP redirector operated by the DNS operator DNSMadeEasy.

  The fact that the court could issue an order which didn’t see this coming and that the ISPs would act on it without checking that what they were doing was sensible is, in my opinion, extremely worrying.

  (tags: overblocking censorship org uk sky be-broadband virgin-media dnsmadeeasy filtering premier-league false-positives isps)