2013
Replicant: Replicated State Machines Made Easy
The next time you reach for ZooKeeper, ask yourself whether it provides the primitive you really need. If ZooKeeper's filesystem and znode abstractions truly meet your needs, great. But the odds are, you'll be better off writing your application as a replicated state machine.
zookeeper paxos replicant replication consensus state-machines distcomp
Dublin Cycle Planner needs a health warning - Irish Cycle
An extensive catalogue of shitty routing. Poor...
It's expected that any new mapping and routing systems will have errors which will need to be ironed out but the level of issues with the NTA Cycle Planner is far beyond what you'd expect in a light and quiet beta launch. It's beyond acceptable for a public PR launch directing people to a route planner with no clear warnings. It looks like a rush job which allows junior minister Alan Kelly to get his name in another press release before the end of the year.
Reflected hidden faces in photographs revealed in pupil
The pupil of the eye in a photograph of a face can be mined for hidden information, such as reflected faces of the photographer and bystanders, according to research led by Dr. Rob Jenkins, of the Department of Psychology at the University of York and published in PLOS ONE (open access).
(via Waxy)via:waxy future zoom-and-enhance privacy photography eyes photos
Jesse Willms, the Dark Lord of the Internet - Taylor Clark - The Atlantic
"It was an out-and-out hijacking," LeFevre told me. "They counterfeited our product, they pirated our Web site, and they basically directed all of their customer service to us." At the peak of Willms's sales, LeFevre says, dazzlesmile was receiving 1,000 calls a day from customers trying to cancel orders for a product it didn't even sell. When irate consumers made the name dazzlesmile synonymous with online scamming, LeFevre's sales effectively dropped to zero. Dazzlesmile sued Willms in November 2009; he later paid a settlement.
scams hijacking ads affiliate one-wierd-trick health dieting crime
-
An exhaustive list from the UK's Open Rights Group
Netflix: Your Linux AMI: optimization and performance [slides]
a fantastic bunch of low-level kernel tweaks and tunables which Netflix have found useful in production to maximise productivity of their fleet. Interesting use of SCHED_BATCH process scheduler class for batch processes, in particular. Also, great docs on their experience with perf and SystemTap. Perf really looks like a tool I need to get to grips with...
netflix aws tuning ami perf systemtap tunables sched_batch batch hadoop optimization performance
creepypasta, Slenderman, and Lovecraft
our use of networked computers is daily coloured by fear of infection and corruption, of predators and those who would assume our identity, of viruses and data-sucking catastrophes. What if something dark is able to breach that all-important final firewall, the gap between the central processing unit and the person sitting at the keyboard? What if it already has? That would be 'a malign and particular suspension or defeat of those fixed laws of Nature which are our only safeguard', without a doubt — but the unplumbed space haunted by demons and chaos is the network, not the cosmos. In using the internet to creep ourselves out recreationally, we begin to understand the real ways in which it haunts our fears.
(via etienneshrdlu)via:etienneshrdlu literature stories horror slenderman something-awful creepypasta copypasta lovecraft
BitCoin exchange CoinBase uses MongoDB as their 'primary datastore'
'Coinbase uses MongoDB for their primary datastore for their web app, api requests, etc.'
coinbase mongodb reliability hn via:aphyr ops banking bitcoin
Alex Payne — Bitcoin, Magical Thinking, and Political Ideology
Working in technology has an element of pioneering, and with new frontiers come those would prefer to leave civilization behind. But in a time of growing inequality, we need technology that preserves and renews the civilization we already have. The first step in this direction is for technologists to engage with the experiences and struggles of those outside their industry and community. There's a big, wide, increasingly poor world out there, and it doesn't need 99% of what Silicon Valley is selling. I've enjoyed the thought experiment of Bitcoin as much as the next nerd, but it's time to dispense with the opportunism and adolescent fantasies of a crypto-powered stateless future and return to the work of building technology and social services that meaningfully and accountably improve our collective quality of life.
bitcoin business economics silicon-valley tech alex-payne writing libertarianism futurism crypto civilization frontier community
MP Claire Perry tells UK that worrying about filter overblocking is a "load of cock"
the bottom line appears to be "think of the children" -- in other words, any degree of overblocking is acceptable as long as children cannot access porn:
The debate and letter confuse legal, illegal and potentially harmful content, all of which require very different tactics to deal with. Without a greater commitment to evidence and rational debate, poor policy outcomes will be the likely result. There's a pattern, much the same as the Digital Economy Act, or the Snooper's Charter. Start with moral panic; dismiss evidence; legislate; and finally, watch the policy unravel, either delivering unintended harms, even to children in this case, or simply failing altogether.
See https://www.openrightsgroup.org/blog/2013/talktalk-wordpress for a well-written exploration of a case of overblocking and its fallout. Talk Talk, one UK ISP, has filters which incorrectly dealt with IWF data and blocked WordPress.com's admin interface, resulting in all blogs there become unusable for their owners for over a week, with seemingly nobody able to diagnose and fix the problem competently.filtering overblocking uk politics think-of-the-children porn cam claire-perry open-rights-group false-positives talk-talk networking internet wordpress
stereopsis : graphics : radix tricks
some nice super-optimized Radix Sort code which handles floating point values. See also http://codercorner.com/RadixSortRevisited.htm for more info on the histogramming/counter concept
sorting programming coding algorithms radix-sort optimization floating-point
-
ie. "i18n", "a11y" etc.
According to Tex Texin, the first numeronym [..] was "S12n", the electronic mail account name given to Digital Equipment Corporation (DEC) employee Jan Scherpenhuizen by a system administrator because his surname was too long to be an account name. By 1985, colleagues who found Jan's name unpronounceable often referred to him verbally as "S12n". The use of such numeronyms became part of DEC corporate culture.[1]
numbers names etymology numeronyms history dec i18n a11y l10n s12n
On undoing, fixing, or removing commits in git
Choose-your-own-adventure style. "Oh dear. This is going to get complicated." (via Tom)
-
this is excellent!
The British Library has uploaded one million public domain scans from 17th-19th century books to Flickr! They're embarking on an ambitious programme to crowdsource novel uses and navigation tools for the huge corpus. Already, the manifest of image descriptions is available through Github. This is a remarkable, public spirited, archival project, and the British Library is to be loudly applauded for it!
british-library libraries public-domain art graphics images history 19th-century 17th-century 18th-century books crowdsourcing via:boingboing github
-
Fantastic long-form blog post by Jay Kreps on this key concept. great stuff
coding databases log network kafka jay-kreps linkedin architecture storage
Difference Engine: Obituary for software patents
The Economist reckons we're finally seeing the light at the end of the tunnel where the patent troll shakedown is concerned:
If the use of state consumer-protection laws to ward off frivolous patent suits were to catch on, it could give the trolls serious pause for thought—especially if their mass mailings of threatening letters to businesses were met by dozens of law suits from attorneys general demanding their presence in state courts across the land. One way or another, things are beginning to look ominous for those who would exploit the inadequacies of America's patent system.
Load Balancer Testing with a Honeypot Daemon
nice post on writing BDD unit tests for infrastructure, in this case specifically a load balancer (via Devops Weekly)
load-balancers ops devops sysadmin testing unit-tests networking honeypot infrastructure bdd
Karlin Lillington on DRI's looming victory in the European Court of Justice
If the full European Court of Justice (ECJ) accepts the opinion of its advocate general in a final ruling due early next year – and it almost always does – it will prove a huge vindication of Ireland's small privacy advocacy group, Digital Rights Ireland (DRI). Its case against Irish retention laws, which began in 2006, forms the basis of this broader David v Goliath challenge and initial opinion. The advocate general's advice largely upholds the key concerns put forward by DRI against Ireland's laws. Withholding so much data about every citizen, including children, in case someone commits a future crime, is too intrusive into private life, and could allow authorities to create a "faithful and exhaustive map of a large portion of a person's [private] conduct". Retained data is so comprehensive that they could easily reveal private identities, which are supposed to remain anonymous. And the data, entrusted to third parties, is at too much risk of fraudulent or malicious use. Cruz Villalón argues that there must be far greater oversight to the retention process, and controls on access to data, and that citizens should have the right to be notified after the fact if their data has been scrutinised. The Irish Government had repeatedly waved off such concerns from Digital Rights Ireland in the past.
dri rights ireland internet surveillance data-retention privacy eu ecj law
Meet the Robot Telemarketer Who Denies She's A Robot
Florida's spammers strike again - pushing the boundaries of intrusive direct sales and marketing
florida ai spam direct-marketing bots sales health-insurance
DigitalOcean's guide to using Docker on their hosts
must give this a spin
-
Our children should be free to choose to study what really excites them, not subtly steered away from certain subjects because teachers believe in and propagate the stereotypes. Last year the IOP published a report "It's Different for Girls" which demonstrated that essentially half of state coeducational schools did not see a single girl progress to A-level physics. By contrast, the likelihood of girls progressing from single sex schools were two and a half times greater.
Amen to this.sexism schools teaching uk phyics girls children bias stereotypes
-
'SBE is an OSI layer 6 representation for encoding and decoding application messages in binary format for low-latency applications.' Licensed under ASL2, C++ and Java supported.
sbe encoding codecs persistence binary low-latency open-source java c++ serialization
-
'like inetd, but for WebSockets' -- 'a small command line tool that will wrap an existing command line interface program, and allow it to be accessed via a WebSocket. It provides a quick mechanism for allowing web-applications to interact with existing command line tools.' Awesome idea. BSD-licensed. (Via Mike Loukides)
-
a metric storage daemon, exposing both a carbon listener and a simple web service. Its aim is to become a simple, scalable and drop-in replacement for graphite's backend.
Pretty alpha for now, but definitely worth keeping an eye on to potentially replace our burgeoning Carbon fleet...graphite carbon cassandra storage metrics ops graphs service-metrics
Twitter tech talk video: "Profiling Java In Production"
In this talk Kaushik Srenevasan describes a new, low overhead, full-stack tool (based on the Linux perf profiler and infrastructure built into the Hotspot JVM) we've built at Twitter to solve the problem of dynamically profiling and tracing the behavior of applications (including managed runtimes) in production.
Looks very interesting. Haven't watched it yet thoughtwitter tech-talks video presentations java jvm profiling testing monitoring service-metrics performance production hotspot perf
Spy agencies in covert push to infiltrate virtual world of online gaming
[MMOGs], the [NSA] analyst wrote, "are an opportunity!". According to the briefing notes, so many different US intelligence agents were conducting operations inside games that a "deconfliction" group was required to ensure they weren't spying on, or interfering with, each other.
spies spying games mmog online surveillance absurd east-germany funny warcraft
Ryan Lizza: Why Won't Obama Rein in the N.S.A.? : The New Yorker
Fantastic wrap-up of the story so far on the pervasive global surveillance story.
The history of the intelligence community, though, reveals a willingness to violate the spirit and the letter of the law, even with oversight. What's more, the benefits of the domestic-surveillance programs remain unclear. Wyden contends that the N.S.A. could find other ways to get the information it says it needs. Even Olsen, when pressed, suggested that the N.S.A. could make do without the bulk-collection program. "In some cases, it's a bit of an insurance policy," he told me. "It's a way to do what we otherwise could do, but do it a little bit more quickly." In recent years, Americans have become accustomed to the idea of advertisers gathering wide swaths of information about their private transactions. The N.S.A.'s collecting of data looks a lot like what Facebook does, but it is fundamentally different. It inverts the crucial legal principle of probable cause: the government may not seize or inspect private property or information without evidence of a crime. The N.S.A. contends that it needs haystacks in order to find the terrorist needle. Its definition of a haystack is expanding; there are indications that, under the auspices of the "business records" provision of the Patriot Act, the intelligence community is now trying to assemble databases of financial transactions and cell-phone location information. Feinstein maintains that data collection is not surveillance. But it is no longer clear if there is a distinction.
nsa gchq surveillance spying privacy dianne-feinstein new-yorker journalism long-reads us-politics probable-cause
Same Old Stories From Sean Sherlock
Sherlock's record is spotty at best when it comes to engagement. Setting aside the 80,680 people who were ignored by the minister, he was hostile and counter productive to debate from the beginning, going so far as to threaten to pull out of a public debate because a campaigner against the ['Irish SOPA'] SI would be in attendance. His habit of blocking people online who publicly ask him tough yet legitimate questions has earned him the nickname "Sherblock".
sean-sherlock sherblock labour ireland politics blocking filtering internet freedom copyright emi music law piracy debate twitter
Smart Metering in the UK is FCUKED
Most utilities don't want smart metering. In fact they seem to have used the wrong dictionary. It is difficult to find anything smart about the UK deployment, until you realise that the utilities use smart in the sense of "it hurts". They consider they have a perfectly adequate business model which has no need for new technology. In many Government meetings, their reluctant support seems to be a veneer for the hope that it will all end in disaster, letting them go back to the world they know, of inflated bills and demands for money with menaces. [...] Even when smart meters are deployed, there is no evidence that any utility will use the resulting data to transform their business, rather than persecute the consumer. At a recent US conference a senior executive for a US utility which had deployed smart meters, stated that their main benefit was "to give them more evidence to blame the customer". That's a good description of the attitude displayed by our utilities.
smart-metering energy utilities uk services metering consumer
Kelly "kellabyte" Sommers on Redis' "relaxed CP" approach to the CAP theorem
Similar to ACID properties, if you partially provide properties it means the user has to _still_ consider in their application that the property doesn't exist, because sometimes it doesn't. In you're fsync example, if fsync is relaxed and there are no replicas, you cannot consider the database durable, just like you can't consider Redis a CP system. It can't be counted on for guarantees to be delivered. This is why I say these systems are hard for users to reason about. Systems that partially offer guarantees require in-depth knowledge of the nuances to properly use the tool. Systems that explicitly make the trade-offs in the designs are easier to reason about because it is more obvious and _predictable_.
kellabyte redis cp ap cap-theorem consistency outages reliability ops database storage distcomp
Building a Balanced Universe - EVE Community
Good blog post about EVE's algorithm to load-balance a 3D map of star systems
eve eve-online algorithms 3d space load-balancing sharding games
Virtual Clock - Testing Patterns Encyclopedia
a nice pattern for unit tests which need deterministic time behaviour. Trying to think up a really nice API for this....
We're sending out the wrong signals in bid to lure the big data bucks - Independent.ie
Simon McGarr on Ireland's looming data-protection train-crash.
Last week, during the debate of his proposals to increase fees for making a Freedom of Information request, Brendan Howlin was asked how one of his amendments would affect citizens looking for data from the State's electronic databases. His reply was to cheerfully admit he didn't even understand the question. "I have no idea what an SQL code is. Does anyone know what an SQL code is?" Unlike the minister, it probably isn't your job to know that SQL is the computer language that underpins the data industry. The amendment he had originally proposed would have effectively allowed civil servants to pretend that their computer files were made of paper when deciding whether a request was reasonable. His answer showed how the Government could have proposed such an absurd idea in the first place. Like it or not – fair or not – these are not the signals a country that wanted to build a long-term data industry would choose to send out. They are the sort of signals that Ireland used to send out about Financial Regulation. I think it's agreed, that approach didn't work out so well.
foi ireland brendan-howlin technology illiteracy sql civil-service government data-protection privacy regulation dpa
-
good blog post writing up the 'flock -n -c' trick to ensure single-concurrent-process locking for cron jobs
What an RAF pilot can teach us about being safe on the road
Good article on road safety and visual perception, for both cyclists and drivers.
vision driving cycling tips cognitive-psychology safety hi-viz
-
a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue. An optional LuaJIT script can perform HTTP request generation, response processing, and custom reporting.
Written in C, ASL2 licensed.wrk benchmarking http performance testing lua load-testing load-generation
Removing DRM Boosts Music Sales by 10%
Based on a working paper from University of Toronto researcher Laurina Zhang
Comparing album sales of four major labels before and after the removal of DRM reveals that digital music revenue increases by 10% when restrictions are removed. The effect goes up to 30% for long tail content, while top-selling albums show no significant jump. The findings suggest that dropping technical restrictions can benefit both artists and the major labels.
more details: http://inside.rotman.utoronto.ca/laurinazhang/files/2013/11/laurina_zhang_jmp_nov4.pdf , "Intellectual Property Strategy and the Long Tail: Evidence from the Recorded Music Industry", Laurina Zhang, November 4, 2013ip copyright drm mp3 music laurina-zhang research long-tail albums rights-management piracy
100 Years of Breed "Improvement" | Science of Dogs
The English bulldog has come to symbolize all that is wrong with the dog fancy and not without good reason; they suffer from almost every possible disease. A 2004 survey by the Kennel Club found that they die at the median age of 6.25 years (n=180). There really is no such thing as a healthy bulldog. The bulldog's monstrous proportions makes them virtually incapable of mating or birthing without medical intervention.
(via Bryan)dogs eugenics breeding horror science genetics traits animals pets bulldog pedigree
SkyJack - autonomous drone hacking
Samy Kamkar strikes again. 'Using a Parrot AR.Drone 2, a Raspberry Pi, a USB battery, an Alfa AWUS036H wireless transmitter, aircrack-ng, node-ar-drone, node.js, and my SkyJack software, I developed a drone that flies around, seeks the wireless signal of any other drone in the area, forcefully disconnects the wireless connection of the true owner of the target drone, then authenticates with the target drone pretending to be its owner, then feeds commands to it and all other possessed zombie drones at my will.'
drones amazon hacking security samy-kamkar aircrack node raspberry-pi airborne-zombies
-
Good article about emergent behaviour from networked malware: 'The metabot, therefore, is viral. You get followed because of who follows you. This tendency explains the strange geographical cluster among San Diego high school students. Perhaps one of those kids was being followed by a really popular account (like @Interscope records, perhaps, which follows hundreds of thousands of people), and through that link, the bot stumbled into this little circle of San Diego teens. All of this activity would have remained under the radar, of course, all part of the silent non-human web. Except something went awry. For some reason, Olivia got stuck in a weird loop, and the metabot kept spawning spambots that chose to follow her over and over, relentlessly. Maybe once the metabot reached the San Diego kids, a bug kicked in. Instead of negative feedback keeping her (and everyone else) from being followed too often, we got runaway positive feedback. The bots followed her because other bots followed her. And on and on. Which is, perhaps a kind of reasoning that we can understand: It's the core logic of fame and celebrity itself. Attention flows to Snooki because attention flowed to Snooki. Attention flows to Olivia because attention flowed to Olivia. Olivia and her friends weren't wrong when they thought she'd become suddenly famous. Her audience just wasn't human.'
-
> reorg Ok, you reorganize all zero of your direct reports. Way to stay out of trouble, Hoss. Perhaps you'd like to coin an acronym?
amazon amazork via:jrauser sev2s reorgs work zachary-mason games interactive-fiction zork text-adventures
-
y'know, for kids. now that would improve the slightly boring, functional helmet my middle kid wears...
-
Wow, I didn't know about this. Great idea.
Need a flexible format to record, export, and analyze network performance data? Well, that's exactly what the HTTP Archive format (HAR) is designed to do! Even better, did you know that Chrome DevTools supports it? In this episode we'll take a deep dive into the format (as you'll see, its very simple), and explore the many different ways it can help you capture and analyze your sites performance. Join Ilya Grigorik and Peter Lubbers to find out how to capture HAR network traces in Chrome, visualize the data via an online tool, share the reports with your clients and coworkers, automate the logging and capture of HAR data for your build scripts, and even adapt it to server-side analysis use cases
capturing logging performance http debugging trace capture har archives protocols recording
flood.io » Convert HAR to a JMeter JMX plan file
this is absolutely fantastic. Thanks flood.io!
har http archive jmeter jmx recording testing debugging captures conversion
Who Is Watching the Watch Lists? - NYTimes.com
it might seem that current efforts to identify and track potential terrorists would be approached with caution. Yet the federal government's main terrorist watch list has grown to at least 700,000 people, with little scrutiny over how the determinations are made or the impact on those marked with the terrorist label. "If you've done the paperwork correctly, then you can effectively enter someone onto the watch list," said Anya Bernstein, an associate professor at the SUNY Buffalo Law School and author of "The Hidden Costs of Terrorist Watch Lists," published by the Buffalo Law Review in May. "There's no indication that agencies undertake any kind of regular retrospective review to assess how good they are at predicting the conduct they're targeting."
terrorism watchlists blacklists filtering safety air-travel government security dhs travel
[JavaSpecialists 215] - StampedLock Idioms
a demo of Doug Lea's latest concurrent data structure in Java 8
-
lulz. (via John Handelaar)
Docker all the things at Atlassian: automation and wiring
A nice worked-through Docker example
docker infrastructure devops ops deployment lxc containers linux
This Flaw In Facebook Lets You Create As Many Fake Likes As You Want - Business Insider
Really stupid -- Facebook infers a "like" for a site when you send a reference to a URL on that site. Obviously broken behaviour. (via http://www.forbes.com/sites/anthonykosner/2013/01/21/facebook-is-recycling-your-likes-to-promote-stories-youve-never-seen-to-all-your-friends/ )
Jury: Newegg infringes Spangenberg patent, must pay $2.3 million | Ars Technica
Newegg, an online retailer that has made a name for itself fighting the non-practicing patent holders sometimes called "patent trolls," sits on the losing end of a lawsuit tonight. An eight-person jury came back shortly after 7:00pm and found that the company infringed all four asserted claims of a patent owned by TQP Development, a company owned by patent enforcement expert Erich Spangenberg.
"patent enforcement expert". That's one way to put it. This is insanity.-
pretty strong argument. However, I think shlibs still have an advantage in that their pages are easier to share...
Newegg trial: Crypto legend takes the stand, goes for knockout patent punch | Ars Technica
"We've heard a good bit in this courtroom about public key encryption," said Albright. "Are you familiar with that? "Yes, I am," said Diffie, in what surely qualified as the biggest understatement of the trial. "And how is it that you're familiar with public key encryption?" "I invented it."
(via burritojustice)crypto tech security patents swpats pki whitfield-diffie history east-texas newegg patent-trolls
SAMOA, an open source platform for mining big data streams
Yahoo!'s streaming machine learning platform, built on Storm, implementing:
As a library, SAMOA contains state-of-the-art implementations of algorithms for distributed machine learning on streams. The first alpha release allows classification and clustering. For classification, we implemented a Vertical Hoeffding Tree (VHT), a distributed streaming version of decision trees tailored for sparse data (e.g., text). For clustering, we included a distributed algorithm based on CluStream. The library also includes meta-algorithms such as bagging.
storm streaming big-data realtime samoa yahoo machine-learning ml decision-trees clustering bagging classification
Spam-Friendly Registrar 'Dynamic Dolphin' Shuttered
yay (via Tony Finch)
dynamic-dolphin registrars dns spam scott-richter anti-spam brian-krebs
Photographer wins $1.2 million from companies that took pictures off Twitter | Reuters
The jury found that Agence France-Presse and Getty Images willfully violated the Copyright Act when they used photos Daniel Morel took in his native Haiti after the 2010 earthquake that killed more than 250,000 people, Morel's lawyer, Joseph Baio, said
copyright twitter facebook social-media via:niall-harbison law getty-images afp daniel-morel haiti photography
Failure Friday: How We Ensure PagerDuty is Always Reliable
Basically, they run the kind of exercise which Jesse Robbins invented at Amazon -- "Game Days". Scarily, they do these on a Friday -- living dangerously!
-
beautiful German boardgame, suitable for playing with kids -- an adult moves a tealight candle around the board, while kids take turns moving gnomes around in the shadows behind tall "trees". recommended by JK
'No basis in law' : Gardai probe Ballyphehane group after raid
Freemen wackiness in Cork.
The house of one member of the group was raided by gardaí last week, but it is not thought that any arrests were made, according to an eyewitness. Gardaí broke down the front door of the house. The group, which appears to be part of the Freemen of the Land movement, which does not recognise the State, has attempted to hold 'trials' in Ballyphehane Community Centre. It attempted to summon HSE staff, gardaí, social workers, solicitors and others to appear to be tried by a self-selected jury earlier this month. The group handed out documents purporting to be a summons to HSE staff and garda stations, demanding that named people attend a trial by 'éire court' on Tuesday 5 November at 9am "to stand trial for their acts of terrorism against mothers, their offspring and others in our community", according to the group's literature. This week the group has begun posting about UCC, saying the college is "a private for profit corporation, and a business partner of and partly owned by Pfizers and Bank of Ireland". The group suggest that UCC bases its "authority" on Maritime Law. UCC has yet to respond to the group's allegations.
-
I'm trying to avoid doing this in order to avoid more power consumption and unpopular hardware in the house -- but if necessary, this is a good up-to-date homebuild design
Asynchronous logging versus Memory Mapped Files
Interesting article around using mmap'd files from Java using RandomAccessFile.getChannel().map(), which allows them to be accessed directly as a ByteBuffer. together with Atomic variable lazySet() operations, this provides pretty excellent performance results on low-latency writes to disk. See also: http://psy-lob-saw.blogspot.ie/2012/12/atomiclazyset-is-performance-win-for.html
atomic lazyset putordered jmm java synchronization randomaccessfile bytebuffers performance optimization memory disk queues
-
a realtime processing engine, built on a persistent queue and a set of workers. 'The main goal is data availability and persistency. We created grape for those who cannot afford losing data'. It does this by allowing infinite expansion of the pending queue in Elliptics, their Dynamo-like horizontally-scaled storage backend.
kafka queue queueing storage realtime fault-tolerance grape cep event-processing
How To Run a 5 Whys (With Humans, Not Robots)
'remember, there is no axe murderer. probably'
process management howto post-mortems five-whys 5-whys investigation
The New Threat: Targeted Internet Traffic Misdirection
MITM attacks via BGP route hijacking now relatively commonplace on the internet, with 60 cases observed so far this year by Renesys
Software Detection of Currency
Steven J. Murdoch presents some interesting results indicating that the EURion constellation may have been obsoleted:
Recent printers, scanners and image manipulation software identify images of currency, will not process the image and display an error message linking to www.rulesforuse.org. The detection algorithm is not disclosed, however it is possible to test sample images as to whether they are identified as currency. This webpage shows an initial analysis of the algorithm's properties, based on results from the automated generation and testing of images. [...] Initially it was thought that the "Eurion constellation" was used to identify banknotes in the newly deployed software based system, since this has been confirmed to be the technique used by colour photocopiers, and was both necessary and sufficient to prevent an item being duplicated using the photocopier tested. However further investigation showed that the detection performed by software is different from the system used in colour photocopiers, and the Eurion constellation is neither necessary nor sufficent, and in fact it probably is not even a factor.
eurion algorithms photoshop security currency money euro copying obscurity reversing
-
a simple-to-use, extensible, text-based data workflow tool that organizes command execution around data and its dependencies. Data processing steps are defined along with their inputs and outputs and Drake automatically resolves their dependencies. [...] Drake is similar to GNU Make, but designed especially for data workflow management. It has HDFS [and S3] support, allows multiple inputs and outputs, and includes a host of features designed to help you bring sanity to your otherwise chaotic data processing workflows.
Via Nelson. Looks interesting, although I'd like to see more features around retries, single-executor locking, parallelism, alerting/metrics, and unattended cron-like operation -- those are always the hard part when I wind up coding up a data pump. AK at re:Invent 2013: Getting Maximum Performance from Redshift
good Redshift tips
redshift aws amazon performance scaling s3 rdbms sql ops analytics
Tintin And The Copyright Sharks - Falkvinge on Infopolicy
A rather sordid tale of IP acquisition and exploitation, from the sounds of it
tintin moulinsart belgium history herge ip copyright royalties rick-falkvinge
IPSO representative trivialising impact of the Loyaltybuild data breach
A very worrying quote from Una Dillon of the Irish Payment Services Organisation in regard to the Loyaltybuild incident:
"I wouldn't be overly concerned if one of my cards was caught up in this," Dillon says. "Even in the worst-case scenario – one in which my card was used fraudulently – my card provider will refund me everything that is taken".
This reflects a deep lack of understanding of (a) how identity fraud works, and (b) how card-fraud refunds in Ireland appear to work. (a): Direct misuse of credit card data is not always the result. Fraudsters may prefer to instead obtain separate credit through identity theft, ie. using other personal identifying data. (b): Visa debit cards have no credit limit -- your bank account can be cleared out in its entirety, and refunds can take a long time. For instance, http://www.askaboutmoney.com/showthread.php?t=174482 describes several cases, including one customer who waited 21 days for a refund. All in all it's trivialising a major risk for consumers. As I understand it, a separate statement from IPSO recommended that all customers of Loyaltybuild schemes need to monitor their bank accounts daily to keep an eye out for fraud, which is pretty absurd. Not impressive at all.loyaltybuild ipso money cards credit-cards visa debit-cards payment fraud identity-theft ireland
-
There is really astonishingly little value in looking at someone's GitHub projects out of context. For a start, GitHub has no way of customising your profile page, and what is shown by default is the projects with the most stars, and the projects you've recently pushed to. That is, GitHub picks your most popular repos and puts those at the top. You have no say about what you consider important, or worthwhile, or interesting, or well-engineered, or valuable. You just get what other people think is useful. Aside from which, GitHub displays a lot of useless stats about how many followers you have, and some completely psychologically manipulative stats about how often you commit and how many days it is since you had a day off. So really, your GitHub profile displays two things: how 'influential' you are, and how easily you can be coerced into constantly working. It's honestly about as relevant to a decent hiring decision as your Klout score.
cv github open-source hiring career meritocracy work via:apyhr
An Empirical Evaluation of TCP Performance in Online Games
In this paper, we have analyzed the performance of TCP in of ShenZhou Online, a commercial, mid-sized MMORPG. Our study indicates that, though TCP is full-fledged and robust, simply transmitting game data over TCP could cause unexpected performance problems. This is due to the following distinctive characteristics of game traffic: 1) tiny packets, 2) low packet rate, 3) application-limited traffic generation, and 4) bi-directional traffic. We have shown that because TCP was originally designed for unidirectional and network-limited bulk data transfers, it cannot adapt well to MMORPG traffic. In particular, the window-based congestion control mechanism and the fast retransmit algorithm for loss recovery are ineffective. This suggests that the selective acknowledgement option should be enabled whenever TCP is used, as it significantly enhances the loss recovery process. Furthermore, TCP is overkill, as not every game packet needs to be transmitted reliably and processed in an orderly manner. We have also shown that the degraded network performance did impact users' willingness to continue a game. Finally, a number of design guidelines have been proposed by exploiting the unique characteristics of game traffic.
via Nelsontcp games udp protocols networking internet mmos retransmit mmorpgs
Column: The Loyaltybuild breach shows it's time to take data protection seriously
What is afoot here is a rerun of the Celtic Tiger era "light touch regulation" of financial services. Ireland has again made a Faustian pact whereby we lure employers here on the understanding that they will not subject to too-stringent a regulatory system. As the Loyaltybuild breach has shown, this is a bargain that will probably end badly. And as with the financial services boom, it is making the Germans nervous. Perhaps we will listen to them this time.
fergal-crehan loyaltybuild celtic-tiger ireland dpa regulation data-protection privacy credit-cards
-
Looks very alpha, but one to watch.
A JVM Implementation of the Raft Consensus Protocol
-
' A persistent key-value store for fast storage environments', ie. BerkeleyDB/LevelDB competitor, from Facebook.
RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support IO-bound, in-memory and write-once workloads, and to be flexible to allow for innovation. We benchmarked LevelDB and found that it was unsuitable for our server workloads. Thebenchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine - where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal. By contrast, we've published the RocksDB benchmark results for server side workloads on Flash. We also measured the performance of LevelDB on these server-workload benchmarks and found that RocksDB solidly outperforms LevelDB for these IO bound workloads. We found that LevelDB's single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.
Lots of good discussion at https://news.ycombinator.com/item?id=6736900 too.flash ssd rocksdb databases storage nosql facebook bdb disk key-value-stores lsm leveldb
-
Colm McCarthaigh has open sourced Infima, 'a library for managing service-level fault isolation using Amazon Route 53'.
Infima provides a Lattice container framework that allows you to categorize each endpoint along one or more fault-isolation dimensions such as availability-zone, software implementation, underlying datastore or any other common point of dependency endpoints may share. Infima also introduces a new ShuffleShard sharding type that can exponentially increase the endpoint-level isolation between customer/object access patterns or any other identifier you choose to shard on. Both Infima Lattices and ShuffleShards can also be automatically expressed in Route 53 DNS failover configurations using AnswerSet and RubberTree.
infima colmmacc dns route-53 fault-tolerance failover multi-az sharding service-discovery
-
The LatencyUtils package includes useful utilities for tracking latencies. Especially in common in-process recording scenarios, which can exhibit significant coordinated omission sensitivity without proper handling.
gil-tene metrics java measurement coordinated-omission latency speed service-metrics open-source
High Performance Browser Networking
slides from Ilya Grigorik's tutorial on the topic at O'Reilly's Velocity conference. lots of good data and tips for internet protocol optimization
slides presentations ilya-grigorik performance http https tcp tutorials networking internet
-
tl;dr: 'a lot to like'.
The grand design and originality thus of 'Modernising Copyright' thus is the injection of targeted flexibility into the legal framework – this is no mere echo of the Hargreaves Report in the UK, which backed away from Fair Use out of fear at the uncertainty it would necessarily entail. If the Report's authors have their way, contested uses in Ireland will first be examined to see if they fit the exceptions spelled out in the EUCD, or checked against the innovation exception if they are derivative works/adaptations. Only if they have fallen at those two fences, will the fair use test be their last chance saloon.
-
'It can't just be Big Data, it has to be Fast Data: Reactor 1.0 goes GA':
Reactor provides the necessary abstractions to build high-throughput, low-latency--what we now call "fast data"--applications that absolutely must work with thousands, tens of thousands, or even millions of concurrent requests per second. Modern JVM applications must be built on a solid foundation of asynchronous and reactive components that efficiently manage the execution of a very large number of tasks on a very small number of system threads. Reactor is specifically designed to help you build these kinds of applications without getting in your way or forcing you to work within an opinionated pattern.
Featuring the LMAX Disruptor ringbuffer, the JavaChronicle fast persistent message-passing queue, Groovy closures, and Netty 4.0. This looks very handy indeed....disruptor reactive-programming reactor async libraries java jvm frameworks spring netty fast-data
Backblaze Blog » How long do disk drives last?
According to Backblaze's data, 80% of drives last 4 years, and the median lifespan is projected to be 6 years
Heirloom Chemistry Set by John Farrell Kuhns — Kickstarter
This is a beauty. I wonder if they can ship to Ireland?
To tell our story for this Kickstarter project, we really have to start in Christmas of 1959. Like many young scientists of the time, I received a Gilbert Chemistry set. This chemistry set provided me hours of great fun and learning as well as laying the foundation for my future as a research chemist. As I became an adult I wanted to share these types of experiences with my daughter, my nephews and nieces, and friends. But soon I became aware real chemistry sets were no longer available. Without real chemistry sets and opportunities for students to learn and explore, where would our future chemists come from? So .... I set out on a mission.
Philippe Flajolet's contribution to streaming algorithms [preso]
Nice deck covering HyperLogLog and its origins, plus a slide at the end covering the Flajolet/Wegman Adaptive Sampling algorithm ("how do you count the number of elements which appear only once in stream using constant size memory?")
algorithms sketching hyperloglog flajolet wegman adaptive-sampling sampling presentations slides
3 Tacos or 4 Flautas Per Order Make a Healthy Diet in Greatest Scientific Study Ever
"In reality, [tacos and flautas] aren't bad meals," the report argues. "The error that many of us Mexicans [Gustavo note: and gabachos] commit is including these types of dishes in our regular diet without an appropriate balance of them and falling into excessively eating them; accompanied by a lack of physical activity, it creates bad eating habits." The good docs go on to note that people can eat tacos and flautas without negatively affecting their health, but "the key resides in controlling the quantity and frequency of eating these types of meals." They also make the point that overall, tacos and flautas have less grease than doughnuts, french fries and even some health bars, although they didn't specify which brands in the latter. In a subsequent blog post, the scientists go on to describe flautas as an "energy food" due to their composition, and conclude by recommending that a healthy diet can include three tacos al pastor or four flautas per order, "controlling the frequency of intake." So have at it, boyos, but in moderation. And I can already hear the skeptics: What about tacos de chicharrones? Why not focus on carne asada? Did they take into consideration chiles de mordida? Did they factor in horchata? And whither the burrito variable?
Jeff Dean - Taming Latency Variability and Scaling Deep Learning [talk]
'what Jeff Dean and team have been up to at Google'. Reducing request latency in a network SOA architecture using backup requests, etc., via Ilya Grigorik
youtube talks google low-latency soa architecture distcomp jeff-dean networking
error-prone - Catch common Java mistakes as compile-time errors
It's common for even the best programmers to make simple mistakes. And commonly, a refactoring which seems safe can leave behind code which will never do what's intended. We're used to getting help from the compiler, but it doesn't do much beyond static type checking. Using error-prone to augment the compiler's static analysis, you can catch more mistakes before they cost you time, or end up as bugs in production. We use error-prone in Google's Java build system to eliminate classes of serious bugs from entering our code, and we've open-sourced it, so you can too!
Where your "full Irish" really comes from
This is really disappointing; many meats labelled as "Irish" are anything but. The only trustworthy mark is the Bord Bia "Origin Ireland" stamp -- I'll be avoiding any products without this in future.
Under European labelling law, country of origin is mandatory for beef, fish, olive oil, honey and fresh fruit and vegetables. Next month the EU will make it law to specify country of origin for the meat of pigs, chicken, sheep and goats, with a lead-in time of anywhere up to three years for food companies to comply. The pork rule, however, will only apply to fresh pork and not to processed meat, so consumers still won't get a country-of-origin label on rashers, sausages or ham. In the meantime, the Bord Bia Origin-Ireland stamp is a guarantee that your Irish breakfast ingredients are indeed Irish.
bord-bia labelling eu country-of-origin meat pork food quality
Killing Freedom of Information in Ireland
TheStory.ie will, in all likelihood, cease all FOI requests. And we will not seek funding from the public to support an immoral, cynical, unjustified and probably illegal FOI fee regime. We will not pay for information that the public already pays for. We will not support a system that perpetuates an outrageous infringement of citizen rights. The legislation was gutted in 2003 and it is being gutted again. More generally the number of requests from journalists from all news organisations in Ireland will fall as a result of these amendments, and the resulting efforts to shine a light on the administration of the State will certainly deteriorate. And secrecy will prevail.
10 Things You Should Know About AWS
Some decent tips in here, mainly EC2-focussed
Tracing Brazil's Guy Fawkes Masks
really fascinating, from Ethan Zuckerman:
The photo of workers making Guy Fawkes masks is something of a Rorschach test. If you're primed to see the exploitative nature of global capitalism when you see people making a plastic mask, it's there in the image. if you're looking for the global spread of a protest movement, it's there too, with a Brazilian factory making a local knock-off of a global icon to cash in on a national protest. Because the internet is a copying machine, it's very bad at context. It's easier to encounter the image of masks being manufactured devoid of accompanying details than it is to find the story behind the images. And given our tendency to ignore information in languages we don't read, it's easy to see how the masks come detached from their accompanying story. For me, the image is more powerful with context behind it. It's possible to reflect on the irony of a Hollywood prop becoming an activist trope, the tensions between mass-production and anonymity and the individuality of one's identity and grievance, the tensions between local and global, Warner Bros and Condal, intellectual property and piracy, all in the same image.
anonymous globalization manufacturing piracy knock-offs brazil ethan-zuckerman global local hollywood capitalism
ReCreate Ireland - Creativity through Reuse
Great idea.
For creative groups, we aim to offer easy access to a rich and varied selection of textures, colours and shapes. Members are also be able to participate in creativity workshops facilitated by fully trained professional artists either in-house or on your own premises. We intend to be the first choice of teachers, early childhood educators and arts animators in the community. For businesses, ReCreate reduces the costs of moving on end-of-line materials. We are a professional, credible and reliable partner organisation and our aim is to divert approximately 115 metric tonnes of clean materials from landfill annually. All collections are free of charge.
3D-Print Your Own 20-Million-Year-Old Fossils
When I get my hands on a 3-D printer, this will be high up my list of things to fabricate: a replica of a 20-million year old hominid skull.
With over 40 digitized fossils in their collection, you can explore 3D renders of fossils representing prehistoric animals, human ancestors, and even ancient tools. Captured using Autodesk software, an SLR camera, and often the original specimen (rather than a cast replica), these renderings bring us closer than most will ever get to holding ancient artifacts. And if you've got an additive manufacturing device at your disposal, you can even download Sketchfab plans to generate your own.
3d-printing fossils africa history hominids replication fabrication sketchfab
-
'A Tiny Seasonal Department Store', featuring the amazing cakes of Wildflour Bakery among others, at 5 Dame Lane, D2.
The tiny department store will be a wonderful seasonal gathering of Makers & Brothers favourite local and international brands. The Others in this project are a carefully considered bunch of partners from the worlds of flowers, food, fashion, beauty, homeware, gifts and more. Makers & Brothers & Others, the tiny department store, promises to be a unique, exciting and engaging retail environment. A place to explore, a seasonal store alive with wonder and served by experts. Kindly hosted by the Fumbally Exchange.
dublin shopping food cakes wildflour-bakery makers-and-brothers xmas
Modernising (Irish) Copyright Katseries #2: linking & marshalling as exceptions
Good commentary on the recent CRC report's recommendations. See also http://ipkitten.blogspot.ie/2013/10/modernising-irish-copyright-katseries-1.html
"The Top 6 Reasons This Infographic Is Just Wrong Enough To Sound Convincing"
+1 to all of this, but especially #5 (polar area diagrams).
diagrams infographics infoviz visualisation data fail statistics
Presto: Interacting with petabytes of data at Facebook
Presto has become a major interactive system for the company's data warehouse. It is deployed in multiple geographical regions and we have successfully scaled a single cluster to 1,000 nodes. The system is actively used by over a thousand employees,who run more than 30,000 queries processing one petabyte daily. Presto is 10x better than Hive/MapReduce in terms of CPU efficiency and latency for most queries at Facebook. It currently supports a large subset of ANSI SQL, including joins, left/right outer joins, subqueries,and most of the common aggregate and scalar functions, including approximate distinct counts (using HyperLogLog) and approximate percentiles (based on quantile digest). The main restrictions at this stage are a size limitation on the join tables and cardinality of unique keys/groups. The system also lacks the ability to write output data back to tables (currently query results are streamed to the client).
facebook hadoop hdfs open-source java sql hive map-reduce querying olap
Herbal supplements are often 'rice and weeds'
DNA tests show that many pills labeled as healing herbs are little more than powdered rice and weeds. [...] Among their findings were bottles of echinacea supplements, used by millions of Americans to prevent and treat colds, that contained ground up bitter weed, Parthenium hysterophorus, an invasive plant found in India and Australia that has been linked to rashes, nausea and flatulence.
herbal-remedies scams quality medicine dna testing fillers allergies st-johns-wort echinacea
Scryer: Netflix's Predictive Auto Scaling Engine
Scryer is a new system that allows us to provision the right number of AWS instances needed to handle the traffic of our customers. But Scryer is different from Amazon Auto Scaling (AAS), which reacts to real-time metrics and adjusts instance counts accordingly. Rather, Scryer predicts what the needs will be prior to the time of need and provisions the instances based on those predictions.
scaling infrastructure aws ec2 netflix scryer auto-scaling aas metrics prediction spikes
Your Assignment for Today: Chew Gum
We have known about [the dental health benefits of xylitol in chewing gum] for a surprisingly long time. In the 1980s, a high-quality, randomized trial in Finland found that children who chewed xylitol-sweetened gum had as much as 60 percent fewer cavities compared with children who didn't. A 1989-93 randomized study of children around age 10 in Belize showed an even greater benefit; chewing xylitol-sweetened gum decreased the risk of cavities by up to 70 percent, and a follow-up study showed that the benefit lasted for up to five years.
xylitol via:eoin health dentist teeth chewing-gum snacks medicine
Mike Hearn - Google+ - The packet capture shown in these new NSA slides shows…
The packet capture shown in these new NSA slides shows internal database replication traffic for the anti-hacking system I worked on for over two years. Specifically, it shows a database recording a user login.
This kind of confirms my theory that the majority of interesting traffic for the NSA/GCHQ MUSCULAR sniffing system would have been inter-DC replication. Was, since it sounds like that stuff's all changing now to use end-to-end crypto...google crypto security muscular nsa gchq mike-hearn replication sniffing spying surveillance
-
'This article will use NettoSphere, a framework build on top of the popular Netty Framework and Atmosphere with support of WebSockets, Server Side Events and Long-Polling. NettoSphere allows [async JVM framework] Atmosphere's applications to run on top of the Netty Framework.'
atmosphere netty async java scala websockets sse long-polling http demos games
Pushing to 100,000 API clients simultaneously
This looks really nice -- it's quite similar to something I was hacking on a while back. Only problem is that it's AGPL-licensed... 'Pushpin makes it easy to create HTTP long-polling and streaming services using any web stack as the backend. It's compatible with any framework, whether Django, Rails, ASP, or even PHP. Pushpin works as a reverse proxy, sitting in front of your server application and managing all of the open client connections.'
pushpin opensource agpl http long-polling reverse-proxy architecture callbacks
European ruling raises questions over liability and online comment
'A recent ruling by the European Court of Human Rights (ECHR) has called into question [...] the liability of media organisations for online comment.' Delfi, a news website in Estonia, found liable for a user's comments by the ECHR
echr comments news web law regulation estonia delfi liability slander defamation
Why Every Company Needs A DevOps Team Now - Feld Thoughts
Bookmarking particularly for the 3 "favourite DevOps patterns":
"Make sure we have environments available early in the Development process"; enforce a policy that the code and environment are tested together, even at the earliest stages of the project; "Wake up developers up at 2 a.m. when they break things"; and "Create reusable deployment procedures".
There is NO spare capacity for Dublin's water supply
The problem in a nutshell is that for an uncomfortable amount of the year the demand outstrips what the system can comfortably supply. In the graph below you'll see the red line (demand for water) matches and regularly exceeds the blue line (what's produced).
drought water dublin mismanagement capacity dcc dublin-council graphs
-
Circa 1800, the Cocktail was a "hair of the dog" morning drink that tamed spirits with water, sugar and bitters (patent medicine). The late 19th Century expanded the use of the word "cocktail" to encompass just about any mixed drink. Since then, the Old Fashioned—literally, the old-fashioned way of making a cocktail—has been our contemporary expression of the original drink. During the 20th Century, various bad ideas encrusted the Old Fashioned. Here we will strip off those barnacles to expose the amazingly simple and sublime drink beneath.
thanks to Ben for this one...recipe alcohol drinks cocktails old-fashioned bourbon bitters
-
"We assess that Miranda is knowingly carrying material [...] the disclosure or threat of disclosure is designed to influence a government, and is made for the purpose of promoting a political or ideological cause. This therefore falls within the definition of terrorism."
security david-miranda journalism censorship terrorism the-guardian
A Brief Tour of FLP Impossibility
One of the most important results in distributed systems theory was published in April 1985 by Fischer, Lynch and Patterson. Their short paper 'Impossibility of Distributed Consensus with One Faulty Process', which eventually won the Dijkstra award given to the most influential papers in distributed computing, definitively placed an upper bound on what it is possible to achieve with distributed processes in an asynchronous environment. This particular result, known as the 'FLP result', settled a dispute that had been ongoing in distributed systems for the previous five to ten years. The problem of consensus – that is, getting a distributed network of processors to agree on a common value – was known to be solvable in a synchronous setting, where processes could proceed in simultaneous steps. In particular, the synchronous solution was resilient to faults, where processors crash and take no further part in the computation. Informally, synchronous models allow failures to be detected by waiting one entire step length for a reply from a processor, and presuming that it has crashed if no reply is received. This kind of failure detection is impossible in an asynchronous setting, where there are no bounds on the amount of time a processor might take to complete its work and then respond with a message. Therefore it's not possible to say whether a processor has crashed or is simply taking a long time to respond. The FLP result shows that in an asynchronous setting, where only one processor might crash, there is no distributed algorithm that solves the consensus problem.
distributed-systems flp consensus-algorithms algorithms distcomp papers proofs
Find a separating hyperplane with this One Weird Kernel Trick
Terrible internet ad-spam recast as machine-learning spam
'37-year-old patriot discovers "weird" trick to end slavery to the Bayesian monopoly. Discover the underground trick she used to slash her empirical risk by 75% in less than 30 days... before they shut her down. Click here to watch the shocking video! Get the Shocking Free Report!'
It's time for Silicon Valley to ask: Is it worth it?
These companies and their technologies are built on data, and the data is us. If we are to have any faith in the Internet, we have to trust them to protect it. That's a relationship dynamic that will become only more intertwined as the Internet finds its way into more aspects of our daily existences, from phones that talk to us to cars that drive themselves. The US's surveillance programs threaten to destroy that trust permanently. America's tech companies must stand up to this pervasive and corrosive surveillance system. They must ask that difficult question: "Is it worth it?"
silicon-valley tech nsa gchq spying surveillance internet privacy data-protection
-
'a service discovery and orchestration tool that is decentralized, highly available, and fault tolerant. Serf runs on every major platform: Linux, Mac OS X, and Windows. It is extremely lightweight: it uses 5 to 10 MB of resident memory and primarily communicates using infrequent UDP messages [and an] efficient gossip protocol.'
clustering service-discovery ops linux gossip broadcast clusters
"Effective Computation of Biased Quantiles over Data Streams" [paper]
Skew is prevalent in many data sources such as IP traffic streams.To continually summarize the distribution of such data, a high-biased set of quantiles (e.g., 50th, 90th and 99th percentiles) with finer error guarantees at higher ranks (e.g., errors of 5, 1 and 0.1 percent, respectively) is more useful than uniformly distributed quantiles (e.g., 25th, 50th and 75th percentiles) with uniform error guarantees. In this paper, we address the following two prob-lems. First, can we compute quantiles with finer error guarantees for the higher ranks of the data distribution effectively, using less space and computation time than computing all quantiles uniformly at the finest error? Second, if specific quantiles and their error bounds are requested a priori, can the necessary space usage and computation time be reduced? We answer both questions in the affirmative by formalizing them as the "high-biased" quantiles and the "targeted" quantiles problems, respectively, and presenting algorithms with provable guarantees, that perform significantly better than previously known solutions for these problems. We implemented our algorithms in the Gigascope data stream management system, and evaluated alternate approaches for maintaining the relevant summary structures.Our experimental results on real and synthetic IP data streams complement our theoretical analyses, and highlight the importance of lightweight, non-blocking implementations when maintaining summary structures over high-speed data streams.
Implemented as a timer-histogram storage system in http://armon.github.io/statsite/ .
statistics quantiles percentiles stream-processing skew papers histograms latency algorithms
-
A C reimplementation of Etsy's statsd, with some interesting memory optimizations.
Statsite is designed to be both highly performant, and very flexible. To achieve this, it implements the stats collection and aggregation in pure C, using libev to be extremely fast. This allows it to handle hundreds of connections, and millions of metrics. After each flush interval expires, statsite performs a fork/exec to start a new stream handler invoking a specified application. Statsite then streams the aggregated metrics over stdin to the application, which is free to handle the metrics as it sees fit. This allows statsite to aggregate metrics and then ship metrics to any number of sinks (Graphite, SQL databases, etc). There is an included Python script that ships metrics to graphite.
statsd graphite statsite performance statistics service-metrics metrics ops
34 Irish pubs listed in Michelin good food guide
if Linnane's and Cronin's are anything to go by, these will be worth a visit
-
A fax machine called my #twilio voice number, this is how @twilio transcribed it.... http://pic.twitter.com/RYh19Pg2pG
This is amazing. Machine talking to machine, with hilarious resultstwilio transcription machine audio fax hey-hey-hey you-know-its-hey funny
-
Founded by Silent Circle and Lavabit. this is promising....
To bring the world our unique end-to-end encrypted protocol and architecture that is the 'next-generation' of private and secure email. As founding partners of The Dark Mail Alliance, both Silent Circle and Lavabit will work to bring other members into the alliance, assist them in implementing the new protocol and jointly work to proliferate the worlds first end-to-end encrypted 'Email 3.0' throughout the world's email providers. Our goal is to open source the protocol and architecture and help others implement this new technology to address privacy concerns against surveillance and back door threats of any kind.
privacy surveillance email smtp silent-circle lavabit dark-mail open-source standards crypto
Ponies by Kij Johnson | Tor.com
A rather dark short story about little girls, peer pressure, and childhood. no fun for this dad of 3 girls :( (via Tatu Saloranta)
via:cowtowncoder writing fiction sf childhood peer-pressure tor ponies
-
A Histogram that supports recording and analyzing sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording, and provides control over value quantization behavior across the value range and the subsequent value resolution at any given level.
hdr histogram data-structures coding gil-tene sampling measuring
Counterfactual Thinking, Rules, and The Knight Capital Accident
John Allspaw with an interesting post on the Knight Capital disaster
Toyota's killer firmware: Bad design and its consequences
This is exactly what you do NOT want to read about embedded systems controlling acceleration in your car:
The Camry electronic throttle control system code was found to have 11,000 global variables. Barr described the code as "spaghetti." Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable). Toyota loosely followed the widely adopted MISRA-C coding rules but Barr's group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them. Barr also discovered inadequate and untracked peer code reviews and the absence of any bug-tracking system at Toyota.
On top of this, there was no error-correcting RAM in use; stack-killing recursive code; a quoted 94% stack usage; risks of unintentional RTOS task shutdown; buffer overflows; unsafe casting; race conditions; unchecked error code return values; and a trivial watchdog timer check. Crappy, unsafe coding.firmware horror embedded-systems toyota camry safety acceleration misra-c coding code-verification spaghetti-code cyclomatic-complexity realtime rtos c code-reviews bug-tracking quality
-
The sounds were not, however, caused by ghosts but by a group of three or four men at least to some degree professionally trained, the FBI now believes, in tunneling: a close-knit and highly disciplined team, perhaps from the construction industry, perhaps even a disgruntled public works crew who decided to put their knowledge of the city's underside to more lucrative work. After all, Rehder explained, their route into the bank was as much brute-force excavation as it was a retracing of the region's buried waterways, accessing the neighborhood by way of the city's complicated storm-sewer network, itself built along old creek beds that no longer appear on city maps. As LAPD lieutenant Doug Collisson, one of the men present on the day of the tunnel's discovery, explained to the Los Angeles Times back in 1987, the crew behind the burglary "would have had to require some knowledge of soil composition and technical engineering. … The way the shaft itself was constructed, it was obviously well-researched and extremely sophisticated." Rehder actually goes further, remarking that when Detective Dennis Pagenkopp "showed crime scene photos of the core bit holes" produced by the burglars' boring upward into the vault "to guys who were in the concrete-coring business, they whistled with professional admiration."
cities crime architecture digging tunnels subterranean la lapd banks via:bldgblog sewers
Link without fear – Copyright in Ireland in a Digital Age
The Copyright Review Committee report has been published. Headline recommendations:
Ensure the right of free speech is a central element of the new copyright regime, including in the areas of parody and satire; Legalise legitimate forms of copying by introducing an explicit and broadly defined "Fair Use" policy. Ensure the extent of copyright ownership is balanced against the public good; Design a system which is clear to all parties, including end users; Design an enforcement mechanism which is easy to understand, transparent and accessible to all parties; Target penalties at those who infringe on copyright rather than on third parties such as intermediaries; Future-proof the new regime by basing it on applicable principles rather than rules relevant to today's technology only; Make it easy for end-users to identify and engage with owners of copyright material.
Here's hoping Sean Sherlock now does what he said he'd do, and acts on these recommendations.copyright law ireland reports fair-use free-speech satire parody copying copyfight ownership ip drm linking
Storm at spider.io - London Storm Meetup 2013-06-18
Not just a Storm success story. Interesting slides indicating where a startup *stopped* using Storm as realtime wasn't useful to their customers
storm realtime hadoop cascading python cep spider.io anti-spam events architecture distcomp low-latency slides rabbitmq
-
I like the impromptu docking station hack
Bruce Schneier On The Feudal Internet And How To Fight It
This is very well-put.
In its early days, there was a lot of talk about the "natural laws of the Internet" and how it would empower the masses, upend traditional power blocks, and spread freedom throughout the world. The international nature of the Internet made a mockery of national laws. Anonymity was easy. Censorship was impossible. Police were clueless about cybercrime. And bigger changes were inevitable. Digital cash would undermine national sovereignty. Citizen journalism would undermine the media, corporate PR, and political parties. Easy copying would destroy the traditional movie and music industries. Web marketing would allow even the smallest companies to compete against corporate giants. It really would be a new world order. Unfortunately, as we know, that's not how it worked out. Instead, we have seen the rise of the feudal Internet: Feudal security consolidates power in the hands of the few. These companies [like Google, Apple, Microsoft, Facebook etc.] act in their own self-interest. They use their relationship with us to increase their profits, sometimes at our expense. They act arbitrarily. They make mistakes. They're deliberately changing social norms. Medieval feudalism gave the lords vast powers over the landless peasants; we're seeing the same thing on the Internet.
bruce-schneier politics internet feudal-internet google apple microsoft facebook government
Russia: Hidden chips 'launch malware attacks from irons'
Cyber criminals are planting chips in electric irons and kettles to launch spam [jm: actually, malware] attacks, reports in Russia suggest. State-owned channel Rossiya 24 even showed footage of a technician opening up an iron included in a batch of Chinese imports to find a "spy chip" with what he called "a little microphone". Its correspondent said the hidden devices were mostly being used to spread viruses, by connecting to any computer within a 200m (656ft) radius which were using unprotected Wi-Fi networks. Other products found to have rogue components reportedly included mobile phones and car dashboard cameras.
wifi viruses spam malware security russia china toasters kettles appliances
Asteroid "mining" with Linux and FOSS
Planetary Resources is a company with a sky-high (some might claim "pie in the sky") goal: to find and mine asteroids for useful minerals and other compounds. It is also a company that uses Linux and lots of free software. So two of the engineers from Planetary Resources, Ray Ramadorai and Marc Allen, gave a presentation at LinuxCon North America to describe how and why the company uses FOSS—along with a bit about what it is trying to do overall.
Mac OS 10.9 – Infinity times your spam
a pretty stupid Mail.app IMAP bug hoses Fastmail:
Yes you read that right. It's copying all the email from the Junk Folder back into the Junk Folder again!. This is legal IMAP, so our server proceeds to create a new copy of each message in the folder. It then expunges the old copies of the messages, but it's happening so often that the current UID on that folder is up to over 3 million. It was just over 2 million a few days ago when I first emailed the user to alert them to the situation, so it's grown by another million since. The only way I can think this escaped QA was that they used a server which (like gmail) automatically suppresses duplicates for all their testing, because this is a massively bad problem.
Google: Our Robot Cars Are Better Drivers Than Puny Humans | MIT Technology Review
One of those analyses showed that when a human was behind the wheel, Google's cars accelerated and braked significantly more sharply than they did when piloting themselves. Another showed that the cars' software was much better at maintaining a safe distance from the vehicle ahead than the human drivers were. "We're spending less time in near-collision states," said Urmson. "Our car is driving more smoothly and more safely than our trained professional drivers."
-
interesting new data structure, pending addition in Java 8. Basically an array of arrays which presents the API of a single List.
An ordered collection of elements. Elements can be added, but not removed. Goes through a building phase, during which elements can be added, and a traversal phase, during which elements can be traversed in order but no further modifications are possible.
spinedbuffer data-structures algorithms java jdk jvm java-8 arrays lists
New political ideals ravaged by ... politics
Direct Democracy Ireland, the party linked to Freemen-on-the-land and the Christian Solidarity Party, is having a bit of a bumpy ride with party governance it sounds like
-
Ho ho.
Michael Hayden, former NSA and CIA boss, who famously argued that the only people complaining about NSA surveillance were internet shut-ins who couldn't get laid, apparently never learned that when you're in a public place, someone might overhear your phone calls. Entrepreneur and former MoveOn.org director Tom Matzzie just so happened to be on the Acela express train from DC to NY when he (1) spotted Hayden sitting behind him and (2) started overhearing a series of "off the record" phone calls with press about the story of the week: the revelations of the NSA spying on foreign leaders. Matzzie did what any self-respecting American would do: live-tweet the calls.
nsa michael-hayden twitter tom-matzzie funny irony trains interviewing public surveillance
-
A tool to manage inter-container dependencies so that continuous delivery with Jenkins and Docker is feasible. Looks very helpful
docker provisioning vms containers dockerize jenkins continuous-delivery continuous-integration
Is Google building a hulking floating data center in SF Bay?
Looks pretty persuasive, especially considering they hold a patent on the design
google data-centers bay-area ships containers shipping sea wave-power treasure-island
Roma, Racism And Tabloid Policing: Interview With Gary Younge : rabble
[This case] shows the link between the popular and the state. This is tabloid journalism followed by tabloid policing. It's also completely ignorant. I wrote my article on the Roma after covering the community for a week. I thought, "that's interesting – there's a range of phenotypes, ways of looking, that include Roma." I mentioned two blonde kids by chance. I mentioned that Roma are more likely to speak the language of the country they're in than Romani, more likely to have the religion of the country they're in. But they have the basic aspect that is true for all identities – they know each other and other people know them. It's not like I'm an expert on the Roma. I was covering them for a week and after the second day I knew Roma children had blonde hair and blue eyes. These people who took that kid away knew nothing. And on that basis they abducted a child.
roma racism ireland gary-younge tabloid journalist children hse gardai
Experian Sold Consumer Data to ID Theft Service
This is what happens when you don't have strong controls on data protection/data privacy -- the US experience.
While [posing as a US-based private investigator] may have gotten the [Vietnam-based gang operating the massive identity fraud site Superget.info] past Experian and/or CourtVentures' screening process, according to Martin there were other signs that should have alerted Experian to potential fraud associated with the account. For example, Martin said the Secret Service told him that the alleged proprietor of Superget.info had paid Experian for his monthly data access charges using wire transfers sent from Singapore. "The issue in my mind was the fact that this went on for almost a year after Experian did their due diligence and purchased" Court Ventures, Martin said. "Why didn't they question cash wires coming in every month? Experian portrays themselves as the data-breach experts, and they sell identity theft protection services. How this could go on without them detecting it I don't know. Our agreement with them was that our information was to be used for fraud prevention and ID verification, and was only to be sold to licensed and credentialed U.S. businesses, not to someone overseas."
via Simon McGarrvia:tupp_ed privacy security crime data-protection data-privacy experian data-breaches courtventures superget scams fraud identity identity-theft
European Parliament passes a vote calling for the EU/US SWIFT agreement to be suspended
"the European Parliament has today sent a clear message that enough is enough. The revelations about NSA interception of SWIFT data make a mockery of the EU's agreement with the US, through which the bank data of European citizens is delivered to the US anti-terror system (TFTP). What is the purpose of an agreement like this, which was concluded in good faith, if the US authorities are going to circumvent its provisions? "The EU cannot continue to remain silent in the face of these ongoing revelations: it gives the impression we are little more than a lap dog of the US. If we are to have a healthy relationship with the US, based on mutual respect and benefit, EU governments must not be afraid of defending core EU values when they are infringed. EU leaders must finally take a clear and unambiguous stance on the NSA violations at this week's summit."
swift banking data eu us nsa interception surveillance snooping diplomacy
Response to "Optimizing Linux Memory Management..."
A follow up to the LinkedIn VM-tuning blog post at http://engineering.linkedin.com/performance/optimizing-linux-memory-management-low-latency-high-throughput-databases --
Do not read in to this article too much, especially for trying to understand how the Linux VM or the kernel works. The authors misread the "global spinlock on the zone" source code and the interpretation in the article is dead wrong.
Making Storm fly with Netty | Yahoo Engineering
Y! engineer doubles the speed of Storm's messaging layer by replacing the zeromq implementation with Netty
netty async zeromq storm messaging tcp benchmarks yahoo clusters
-
Service discovery a la Airbnb -- Nerve and Synapse: two external daemons that run on each node, Nerve to manage registration in Zookeeper, and Synapse to generate a haproxy configuration file from that, running on each host, allowing connections to all other hosts.
haproxy services ops load-balancing service-discovery nerve synapse airbnb
The New York Review of Bots - @TwoHeadlines: Comedy, Tragedy, Chicago Bears
What is near-future late-capitalist dystopian fiction but a world where there is no discernible difference between corporations, nations, sports teams, brands, and celebrities? Adam was partly right in our original email thread. @TwoHeadlines is not generating jokes about current events. It is generating jokes about the future: a very specific future dictated by what a Google algorithm believes is important about humans and our affairs.
google-news google algorithms word-frequency twitter twoheadlines bots news emergent jokes
-
'Welcome to the New York Review of Bots, a professional journal of automated-agent studies. We aspire to the highest standards of rigorous analysis, but will often just post things we liked that a computer made.'
robots bots tumblr ai word-frequency markov-chain random twitter
How to lose $172,222 a second for 45 minutes
Major outage and $465m of trading loss, caused by staggeringly inept software management: 8 years of incremental bitrot, technical debt, and failure to have correct processes to engage an ops team in incident response. Hopefully this will serve as a lesson that software is more than just coding, at least to one industry
trading programming coding software inept fail bitrot tech-debt ops incident-response
Basho and Seagate partner to deliver scale-out cloud storage breakthrough
Ha, cool. Skip the OS, write the Riak store natively to the drive. This sounds frankly terrifying ;)
The Seagate Kinetic Open Storage platform eliminates the storage server tier of traditional data center architectures by enabling applications to speak directly to the storage system, thereby reducing expenses associated with the acquisition, deployment, and support of hyperscale storage infrastructures. The platform leverages Seagate's expertise in hardware and software storage systems integrating an open source API and Ethernet connectivity with Seagate hard drive technology.
Sorry, lobbyists! Europe's post-Snowden privacy reform gets a major boost
Following months of revelations, and on the same day that France heard its citizens' phone calls were being reportedly recorded en masse by the Americans, the Parliament's committee gave a resounding thumbs-up to every single amendment proposed by industrious German Green MEP Jan Phillip Albrecht (pictured above).
lolz.lobbying tech surveillance privacy eu jan-phillip-albrecht ep spying
NCCA Junior Cycle - Programming and Coding Consultation Page
the National Council for Curriculum and Assessment are looking for feedback on adding programming to the junior cycle (ie., early secondary school) in Ireland. Add your EUR.02!
Everything You Always Wanted to Know About Synchronization but Were Afraid to Ask
'the most exhaustive study of [multi-core] synchronization to date'
synchronization scalability cpus hardware papers via:fanf multicore cas
WISH: A Monumental 11-Acre Portrait in Belfast by Jorge Rodríguez-Gerada
Must go up and visit this.
Unveiled several days ago in Belfast, Northern Ireland as part of the Belfast Festival, WISH is the latest public art project by Cuban-American artist Jorge Rodriguez-Gerada. The image depicted is of an anonymous Belfast girl and is so large it can only be viewed from the highest points in Belfast or an airplane. Several years in the making, WISH was first plotted on a grid using state-of-the-art Topcon GPS technology and 30,000 manually placed wooden stakes in Belfast's Titanic Quarter. The portrait was then "drawn" with aid of volunteers who helped place nearly 8 million pounds of natural materials including soil, sand, and rock over a period of four weeks.
belfast ireland art portraits jorge-rodriguez-gerada land soil
-
Autoremediation, ie. auto-replacement, of Cassandra nodes in production at Netflix
ops autoremediation outages remediation cassandra storage netflix chaos-monkey
Barbarians at the Gateways - ACM Queue
I am a former high-frequency trader. For a few wonderful years I led a group of brilliant engineers and mathematicians, and together we traded in the electronic marketplaces and pushed systems to the edge of their capability.
Insane stuff -- FPGAs embedded in the network switches to shave off nanoseconds of latency.low-latency hft via:nelson markets stock-trading latency fpgas networking
Online Algorithms in High-frequency Trading - ACM Queue
one-pass algorithms for computing mean, variance, and linear regression, from the HFT world.
linear-regression variance mean variability volatility stream-processing online algorithms hft trading
"Toy Story 2" was almost entirely deleted by accident at one point
A stray "rm -rf" on the main network share managed to wipe out 90% of the movie's assets, and the backups were corrupt. Horrific backups war story
The Impossible Music of Black MIDI
excellently bananas. 8.49 million separate musical notes in a single 4-minute-long composition (via Paddy Benson)
music hardcore black-midi midi composition halp digital via:pbenson
Bitcoin Mining Operating Margin
"The graph showing miners' revenue minus estimated electricity and bandwidth costs." -- down to -694% right now, oh dear
bitcoin via:peakscale economics mining profit revenue charts electricity bubble
How to Read a Scientific Paper (About That Researcher With a Nematode in His Mouth) - Wired Science
Let's rewind to September 2012. It was about then- according to this recently published report (paywall) in The American Journal of Tropical Medicine – that an "otherwise healthy, 36-year-old man" felt a rough patch in his mouth, a scaly little area his right cheek. It didn't hurt. But then it didn't stay there either. He started testing for it with his tongue. It traveled. It moved to the back of his mouth, then forward, coiled backwards again. In the language of science: "These rough patches would appear and disappear on a daily basis, giving the patient the indirect sense that there was an organism moving within the oral cavity."
nematodes parasites biology medicine paper gross funny wired mouth
"High Performance Browser Networking", by Ilya Grigorik, read online for free
Wow, this looks excellent. A must-read for people working on systems with high-volume, low-latency phone-to-server communications -- and free!
How prepared are you to build fast and efficient web applications? This eloquent book provides what every web developer should know about the network, from fundamental limitations that affect performance to major innovations for building even more powerful browser applications—including HTTP 2.0 and XHR improvements, Server-Sent Events (SSE), WebSocket, and WebRTC. Author Ilya Grigorik, a web performance engineer at Google, demonstrates performance optimization best practices for TCP, UDP, and TLS protocols, and explains unique wireless and mobile network optimization requirements. You'll then dive into performance characteristics of technologies such as HTTP 2.0, client-side network scripting with XHR, real-time streaming with SSE and WebSocket, and P2P communication with WebRTC. Deliver optimal TCP, UDP, and TLS performance; Optimize network performance over 3G/4G mobile networks; Develop fast and energy-efficient mobile applications; Address bottlenecks in HTTP 1.x and other browser protocols; Plan for and deliver the best HTTP 2.0 performance; Enable efficient real-time streaming in the browser; Create efficient peer-to-peer videoconferencing and low-latency applications with real-time WebRTC transports
Via Eoin Brazil.book browser networking performance phones mobile 3g 4g hsdpa http udp tls ssl latency webrtc websockets ebooks via:eoin-brazil google http2 sse xhr ilya-grigorik
Even the NSA is finding it hard to cope with spam
3 new Snowden leaks, covering acquisition of Yahoo address books, buddy lists, and email account activity, and how spammer activity required intervention to avoid losing useful data in the noise
-
slides (lots of slides) from Baron Schwartz' talk at Velocity in NYC.
slides monitoring metrics ops devops baron-schwartz pdf capacity
-
Timestamps, as implemented in Riak, Cassandra, et al, are fundamentally unsafe ordering constructs. In order to guarantee consistency you, the user, must ensure locally monotonic and, to some extent, globally monotonic clocks. This is a hard problem, and NTP does not solve it for you. When wall clocks are not properly coupled to the operations in the system, causal constraints can be violated. To ensure safety properties hold all the time, rather than probabilistically, you need logical clocks.
clocks time distributed databases distcomp ntp via:fanf aphyr vector-clocks last-write-wins lww cassandra riak
Reverse Engineering a D-Link Backdoor
Using the correct User-Agent: string, all auth is bypassed on several released models of D-Link and Planex routers. Horrific fail by D-Link
d-link security backdoors authorization reversing planex networking routers
-
one of the most obvious inferences from the Snowden revelations published by the Guardian, New York Times and ProPublica recently is that the NSA has indeed been up to the business of inserting covert back doors in networking and other computing kit. The reports say that, in addition to undermining all of the mainstream cryptographic software used to protect online commerce, the NSA has been "collaborating with technology companies in the United States and abroad to build entry points into their products". These reports have, needless to say, been strenuously denied by the companies, such as Cisco, that make this networking kit. Perhaps the NSA omitted to tell DARPA what it was up to? In the meantime, I hear that some governments have decided that their embassies should no longer use electronic communications at all, and are returning to employing couriers who travel the world handcuffed to locked dispatch cases. We're back to the future, again.
politics backdoors snowden snooping networking cisco nsa gchq
Azerbaijan accidentally publishes the results of its election before the polls open
The mistake came when an electoral commission accidentally published results showing a victory for Ilham Aliyev, the country's long-standing President, a day before voting. Meydan TV, an online channel critical of the government, released a screenshot from a mobile app for the Azerbaijan Central Election Commission which showed that Mr Aliyev had received 72.76 per cent of the vote compared with 7.4 per cent for the opposition candidate, Jamil Hasanli. The screenshot also indicates that the app displayed information about how many people voted at various times during the day. Polls opened at 8am.
-
According to EasyDNS:
Any registrar that has taken one of these sites offline that now impedes the registrants of those domains from simply getting their domain names out of there and back online somewhere else will then be subject to the TDRP – Transfer Dispute Resolution Policy and if they lose (which they will) they will be subject to TDRP fees assesed by the registry operator, and to quote the TDRP itself "Transfer dispute resolution fees can be substantial". This is why it is never a good idea to just react to pressure in the face of obnoxious bluster – in the very act of trying to diffuse any perceived culpability you end up opening yourself to real liability.
Schneier on Security: Air Gaps
interesting discussion in the comments. "Patricia"'s process is particularly hair-raisingly complex, involving 3 separate machines and a multitude of VMs
New faculty positions versus new PhDs
The ever-plummeting chances of a PhD finding a faculty job:
Since 1982, almost 800,000 PhDs were awarded in science and engineering fields, whereas only about 100,000 academic faculty positions were created in those fields within the same time frame. The number of S&E PhDs awarded annually has also increased over this time frame, from ~19,000 in 1982 to ~36,000 in 2011. The number of faculty positions created each year, however, has not changed, with roughly 3,000 new positions created annually.
(via Javier Omar Garcia)-
Sometimes good judgment can compel us to act illegally. Should a self-driving vehicle get to make that same decision?
ethics stories via:chris-horn the-atlantic driving cars law robots self-driving-vehicles
-
'A Ruby gem providing "time travel" and "time freezing" capabilities, making it dead simple to test time-dependent code. It provides a unified method to mock Time.now, Date.today, and DateTime.now in a single call.' This is about the nicest mock-time library I've found so far. (via Ben)
The 29 Stages Of A Twitterstorm
this is brilliant
uk twitter media funny pricehound racism outrage pitchforks rage social-media
'Experience of software engineers using TLA+, PlusCal and TLC' [slides] [pdf]
by Chris Newcombe, an AWS principal engineer. Several Amazonians sharing their results in simulating tricky distributed-systems problems using formal methods
tla+ pluscal tlc formal-methods simulation proving aws amazon architecture design
LinkBench: A database benchmark for the social graph
However, the gold standard for database benchmarking is to test the performance of a system on the real production workload, since synthetic benchmarks often don't exercise systems in the same way. When making decisions about a significant component of Facebook's infrastructure, we need to understand how a database system will really perform in Facebook's production workload. [....] LinkBench addresses these needs by replicating the data model, graph structure, and request mix of our MySQL social graph workload.
Mentioned in a presentation from Peter Bailis, http://www.hpts.ws/papers/2013/bailis-hpts-2013.pdfgraph databases mysql facebook performance testing benchmarks workloads
-
from the Percona toolkit. 'Conveniently summarizes the status and configuration of a server. It is not a tuning tool or diagnosis tool. It produces a report that is easy to diff and can be pasted into emails without losing the formatting. This tool works well on many types of Unix systems.' --- summarises OOM history, top, netstat connection table, interface stats, network config, RAID, LVM, disks, inodes, disk scheduling, mounts, memory, processors, and CPU.
How much can an extra hour's sleep change you?
What they discovered is that when the volunteers cut back from seven-and-a-half to six-and-a-half hours' sleep a night, genes that are associated with processes like inflammation, immune response and response to stress became more active. The team also saw increases in the activity of genes associated with diabetes and risk of cancer. The reverse happened when the volunteers added an hour of sleep.
-
some great phone cases from an Irish company, with nifty art by Irish illustrators and artists including Fatti Burke and Chris Judge
chris-judge fatti-burke illustrators art ireland iphone cases
What drives JVM full GC duration
Interesting empirical results using JDK 7u21:
Full GC duration depends on the number of objects allocated and the locality of their references. It does not depend that much on actual heap size.
Reference locality has a surprisingly high effect.Rhizome | Occupy.here: A tiny, self-contained darknet
Occupy.here began two years ago as an experiment for the encampment at Zuccotti Park. It was a wifi router hacked to run OpenWrt Linux (an operating system mostly used for computer networking) and a small "captive portal" website. When users joined the wifi network and attempted to load any URL, they were redirected to http://occupy.here. The web software offered up a simple BBS-style message board providing its users with a space to share messages and files.
Nifty project from Dan Phiffer.Whatever Happened to "Due Process" ?
Mark Jeftovic is on fire after receiving yet another "take down this domain or else" mail from the City of London police:
We have an obligation to our customers and we are bound by our Registrar Accreditation Agreements not to make arbitrary changes to our customers settings without a valid FOA (Form of Authorization). To supersede that we need a legal basis. To get a legal basis something has to happen in court. [...] What gets me about all of this is that the largest, most egregious perpetrators of online criminal activity right now are our own governments, spying on their own citizens, illegally wiretapping our own private communications and nobody cares, nobody will answer for it, it's just an out-of-scope conversation that is expected to blend into the overall background malaise of our ever increasing serfdom. If I can't make various governments and law enforcement agencies get warrants or court orders before they crack my private communications then I can at least require a court order before I takedown my own customer.
city-of-london police takedowns politics mark-jeftovic easydns registrars dns via:tjmcintyre
-
The problem with software patents, part XVII.
So you have a situation where even when the original patent holder donated the patent for "the public good," sooner or later, an obnoxious patent troll like IV comes along and turns it into a weapon. Again: AmEx patented those little numbers on your credit card, and then for the good of the industry and consumer protection donated the patent to a non-profit, who promised not to enforce the patent against banks... and then proceeded to sell the patent to Intellectual Ventures who is now suing banks over it.
intellectual-ventures scams patents swpats shakedown banking cvv american-express banks amex cmaf
SPSC revisited part III - FastFlow + Sparse Data
holy moly. This is some heavily-optimized mechanical-sympathy Java code. By using a sparse data structure, cache-aligned fields, and wait-free low-level CAS concurrency primitives via sun.misc.Unsafe, a single-producer/single-consumer queue implementation goes pretty damn fast compared to the current state of the art
nitsanw optimization concurrency java jvm cas spsc queues data-structures algorithms
Non-blocking transactional atomicity
interesting new distributed atomic transaction algorithm from Peter Bailis
algorithms database distributed scalability storage peter-bailis distcomp
ZeroMQ: Helping us Block Malicious Domains in Real Time - Umbrella Security Labs
nice writeup of a ZeroMQ/Hadoop event processing pipeline architecture
zeromq hadoop event-processing architecture dns backend reputation
Man sues RMV after driver's license mistakenly revoked by automated anti-terror false positive:
John H. Gass hadn't had a traffic ticket in years, so the Natick resident was surprised this spring when he received a letter from the Massachusetts Registry of Motor Vehicles informing him to cease driving because his license had been revoked. [...] After frantic calls and a hearing with Registry officials, Gass learned the problem: An antiterrorism computerized facial recognition system that scans a database of millions of state driver's license images had picked his as a possible fraud. "We send out 1,500 suspension letters every day," said Registrar Rachel Kaprielian. [...] "There are mistakes that can be made."
See also this New Scientist story. This story notes that the system's pretty widespread:
Massachusetts bought the system with a $1.5 million grant from the Department of Homeland Security. At least 34 states use such systems, which law enforcement officials say help prevent identity theft and ID fraud.
In my opinion, this kind of thing -- trial by inaccurate, false-positive-prone algorithm, is one of the most worrying things about the post-PRISM world.
When we created SpamAssassin, we were well aware of the risk of automated misclassification. Any machine-learning classifier will always make mistakes. The key is to carefully calibrate the expected false-positive/false-negative ratio so that the negative side-effects of a misclassification corresponds to the expected rate.
These anti-terrorism machine learning systems are calibrated to catch as many potential cases as possible, but by aiming to reduce false negatives to this degree, they become wildly prone to false positives. And when they're applied as a dragnet across all citizens' interactions with the state -- or even in the case of PRISM, all citizens' interactions that can be surveilled en masse -- it's going to create buckets of bureaucratic false-positive horror stories, as random innocent citizens are incorrectly tagged as criminals due to software bugs and poor calibration.
privacy algorithms us-politics statistics false-positives ai
Rapid read protection in Cassandra 2.0.2
Nifty new feature -- if a request takes over the 99th percentile for requests to that server, it'll be repeated against another replica. Unnecessary for Voldemort, of course, which queries all replicas anyway!
Attacking Tor: how the NSA targets users' online anonymity
As part of the Turmoil system, the NSA places secret servers, codenamed Quantum, at key places on the internet backbone. This placement ensures that they can react faster than other websites can. By exploiting that speed difference, these servers can impersonate a visited website to the target before the legitimate website can respond, thereby tricking the target's browser to visit a Foxacid server.
whoa, I missed this before.nsa gchq packet-injection attacks security backbone http latency
GCHQ report on 'MULLENIZE' program to 'stain' anonymous electronic traffic
By modifying the User-Agent: header string, each HTTP transaction is "stained" to allow tracking. huh
gchq nsa snooping sniffing surveillance user-agent http browsers leaks
Giving Docker/LXC containers a routable IP address
ugh, this is a mess. Docker, automate this crap
docker routing linux ops networking containers virtualization
How the feds took down the Dread Pirate Roberts | Ars Technica
Well-written, comprehensive writeup of the Silk Road takedown, and the libertarian craziness of Ross William Ulbricht, it's alleged owner and operator
silk-road drugs crazy ross-william-ulbricht fbi libertarian murder tor
Patent troll Lodsys chickens out, folds case rather than face Eugene Kaspersky
In Kaspersky's view, patent trolls are no better than the extortionists who cropped up in Russia after the fall of the Soviet Union, when crime ran rampant. Kaspersky saw more and more people becoming victims of various extortion schemes. US patent trolls seemed very similar. "Kaspersky's view was that paying patent trolls was like paying a protection racket," said Kniser. He wasn't going to do it.
yay! pity it didn't manage to establish precedent, though. But go Kaspersky!eugene-kaspersky shakedowns law east-texas swpats patents patent-trolls
Sergio Bossa's thoughts about Datomic
good comments from Sergio, particularly about the scalability of the single transactor in the Datomic architecture. I agree it's a worrying design flaw
clojure nosql datomic sergio-bossa transactor spof architecture storage
Codex Seraphinianus: A new edition of the strangest book in the world
Excited! one commenter claims a paperback of the new edition of Luigi Serafini's masterwork should cost about $75 when it comes out in a couple of months. sign me up, this is an amazing work
The Snowden files: why the British public should be worried about GCHQ
When the Guardian offered John Lanchester access to the GCHQ files, the journalist and novelist was initially unconvinced. But what the papers told him was alarming: that Britain is sliding towards an entirely new kind of surveillance society
john-lanchester gchq guardian surveillance snooping police-state nsa privacy government
Groundbreaking Results for High Performance Trading with FPGA and x86 Technologies
The enhancement in performance was achieved by providing a fast-path where trades are executed directly by the FPGA under the control of trigger rules processed by the x86 based functions. The latency is reduced further by two additional techniques in the FPGA – inline parsing and pre-emption. As market data enters the switch, the Ethernet frame is parsed serially as bits arrive, allowing partial information to be extracted and matched before the whole frame has been received. Then, instead of waiting until the end of a potential triggering input packet, pre-emption is used to start sending the overhead part of a response which contains the Ethernet, IP, TCP and FIX headers. This allows completion of an outgoing order almost immediately after the end of the triggering market feed packet.
Insane stuff. (Via Martin Thompson)via:martin-thompson insane speed low-latency fpga fast-path trading stock-markets performance optimization ethernet
Why Tellybug moved from Cassandra to Amazon DynamoDB
Summary: poor reliability, better latencies, and cheaper (!)
aws dynamodb cassandra nosql storage tellybug counters scalability reliability latency
-
Interviews with 2 New York bike thieves (one bottom feeder, one professional), reviewing the current batch of bicycle locks. Summary: U-locks are good, when used correctly, particularly the Kryptonite New York Lock ($80). On the other hand, Dublin's recent spate of thefts are largely driven by wide availability of battery-powered angle grinders (thanks Lidl!), which, according to this article, are relatively quiet and extremely fast. :(
Fingerprints are Usernames, not Passwords
I could see some value, perhaps, in a tablet that I share with my wife, where each of us have our own accounts, with independent configurations, apps, and settings. We could each conveniently identify ourselves by our fingerprint. But biometrics cannot, and absolutely must not, be used to authenticate an identity. For authentication, you need a password or passphrase. Something that can be independently chosen, changed, and rotated. [...] Once your fingerprint is compromised (and, yes, it almost certainly already is, if you've crossed an international border or registered for a driver's license in most US states), how do you change it? Are you starting to see why this is a really bad idea?
biometrics apple security fingerprints passwords authentication authorization identity
-
This is a pretty good summary of the salient points from the criminal complaint against Ross William Ulbricht -- I'd say it's pretty bad news for any users of the dodgy site, particularly given this:
"During the 60-day period from May 24, 2013 to July 23, 2013, there were approximately 1,217,218 communications sent between Silk Road users through Silk Road's private-message system."
According to the complaint, those are now in the FBI's hands -- likely unencrypted. -
ouch. some serious slagging here, along with taco science. (BTW we have the same problem with carne asada in Ireland, our taquerias use the cheater method too, sadly)
la tacos mexican food new-york slagging burritos taquerias carne-asada
Edward Snowden's E-Mail Provider Defied FBI Demands to Turn Over SSL Keys, Documents Show
Levison lost [in secret court against the government's order]. In a work-around, Levison complied the next day by turning over the private SSL keys as an 11 page printout in 4-point type. The government called the printout "illegible" and the court ordered Levison to provide a more useful electronic copy.
Nice try though! Bottom line is they demanded the SSL private key. (via Waxy)government privacy security ssl tls crypto fbi via:waxy secrecy snooping
Poisson Rouge: Crowdfunding Red Fish style
the fantastic French kids' site is now crowdfunding new work -- first off being a German Alphabet part of the site. My kids love their stuff, so -- bonne chance!
How an Engineer Earned 1.25 Million Air Miles By Buying Pudding
An amazing hack. 'Air Miles are awesome, they can be used to score free flights, hotel stays and if you're really lucky, the scorn and hatred of everyone you come in contact with who has to pay full price when they travel. The king of all virtually free travelers is one David Phillips, a civil engineer who teaches at the University of California, Davis. David came to the attention of the wider media when he managed to convert about 12,150 cups of Healthy Choice chocolate pudding [costing $3000] into over a million Air Miles. Ever since, David and his entire family have been travelling the world for next to nothing.' (via al3xandru)
-
An adventure that takes you through several popular Java language features and shows how they compile to bytecode and eventually JIT to assembly code.
Model checking for highly concurrent code
Applied formal methods in order to test distributed systems -- specifically GlusterFS:
I'll use an example from my own recent experience. I'm developing a new kind of replication for GlusterFS. To make sure the protocol behaves correctly even across multiple failures, I developed a Murphi model for it. [...] I added a third failure [to the simulated model]. I didn't expect a three-node system to continue working if more than one of those were concurrent (the model allows the failures to be any mix of sequential and concurrent), but I expected it to fail cleanly without reaching an invalid state. Surprise! It managed to produce a case where a reader can observe values that go back in time. This might not make much sense without knowing the protocol involved, but it might give some idea of the crazy conditions a model checker will find that you couldn't possibly have considered. [...] So now I have a bug to fix, and that's a good thing. Clearly, it involves a very specific set of ill-timed reads, writes, and failures. Could I have found it by inspection or ad-hoc analysis? Hell, no. Could I have found it by testing on live systems? Maybe, eventually, but it probably would have taken months for this particular combination to occur on its own. Forcing it to occur would require a lot of extra code, plus an exerciser that would amount to a model checker running 100x slower across machines than Murphi does. With enough real deployments over enough time it would have happened, but the only feasible way to prevent that was with model checking. These are exactly the kinds of bugs that are hardest to fix in the field, and that make users distrust distributed systems, so those of us who build such systems should use every tool at our disposal to avoid them.
model-checking formal-methods modelling murphi distcomp distributed-systems glusterfs testing protocols
Is Trypophobia a Real Phobia? | Popular Science
ie. "fear of small, clustered holes". Sounds like it's not so much a "phobia" as some kind of innate, visceral disgust response; I get it. 'As for who actually made the word up, that distinction probably belongs to a blogger in Ireland named Louise, Andrews says. According to an archived Geocities page, Louise settled on "trypophobia" (Greek for "boring holes" + "fear") after corresponding with a representative at the Oxford English Dictionary. Louise, Andrews and trypophobia Facebook group members have petitioned the dictionary to include the word. The term will need to be used for years and have multiple petitions and scholarly references before the dictionary accepts it, Andrews says. I, for one, would prefer to forget about it forever.'
disgusting revulsion fear phobias trypophobia holes ugh innate
Common phobia you have never heard of: Fear of holes may stem from evolutionary survival response
"We think that everyone has trypophobic tendencies even though they may not be aware of it," said Dr Cole. "We found that people who don't have the phobia still rate trypophobic images as less comfortable to look at than other images. It backs up the theory that we are set-up to be fearful of things which hurt us in our evolutionary past. We have an innate predisposition to be wary of things that can harm us."
trypophobia holes fear aversion disgust ugh evolution innate
-
This is cool. Deploy Docker container images onto a Mesos cluster: key point, in the description of the Redis example: 'there's no need to install Redis or its supporting libraries on your Mesos hosts.'
mesos docker deployment ops images virtualization containers linux
-
Aphyr takes a look at Kafka 0.8's replication with the Jepsen test suite. It doesn't go great. Jay Kreps responds here: http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen
jay-kreps kafka replication distributed-systems distcomp networking reliability fault-tolerance jepsen
-
A book published during the presidency of Chester A. Arthur has a greater chance of being in print today than one published during the time of Reagan.
This is not a gently sloping downward curve. Publishers seem unwilling to sell their books on Amazon for more than a few years after their initial publication. The data suggest that publishing business models make books disappear fairly shortly after their publication and long before they are scheduled to fall into the public domain. Copyright law then deters their reappearance as long as they are owned. On the left side of the graph before 1920, the decline presents a more gentle time-sensitive downward sloping curve.
business books legal copyright law public-domain reading history publishers amazon papers
Horse_ebooks is human after all
Curated dissociated text. That's great
-
coding funny processors multicore multiprocessing branch-prediction hardware
To my daughter's high school programming teacher
During the first semester of my daughter's junior/senior year, she took her first programming class. She knew I'd be thrilled, but she did it anyway. When my daughter got home from the first day of the semester, I asked her about the class. "Well, I'm the only girl in class," she said. Fortunately, that didn't bother her, and she even liked joking around with the guys in class. My daughter said that you noticed and apologized to her because she was the only girl in class. And when the lessons started (Visual Basic? Seriously??), my daughter flew through the assigments. After she finished, she'd help classmates who were behind or struggling in class. Over the next few weeks, things went downhill. While I was attending SC '12 in Salt Lake City last November, my daughter emailed to tell me that the boys in her class were harassing her. "They told me to get in the kitchen and make them sandwiches," she said. I was painfully reminded of the anonymous men boys who left comments on a Linux Pro Magazine blog post I wrote a few years ago, saying the exact same thing.
I am sick to death of this 'brogrammer' bullshit.brogrammers sexism culture tech teaching coding software education
"The cricket bat that died for Ireland"
The bat had the misfortune of being on display in the shop front of Elvery's store on O'Connell Street, then Sackville Street, during the Easter Rising. J.W. Elvery & Co. was Ireland's oldest sports store, specialising in sporting goods and waterproofed wear, with branches in Dublin, Cork (Patrick Street) and London (Conduit Street). [...] Its location, about one block from the GPO, meant it was in the middle of the cross-fire and general destruction of the main street.
ireland cricket 1916 history easter-rising crossfire sports elverys
_Availability in Globally Distributed Storage Systems_ [pdf]
empirical BigTable and GFS failure numbers from Google are orders of magnitude higher than naïve independent-failure models. (via kragen)
via:kragen failure bigtable gfs statistics outages reliability
Why We Hate Infographics (And Why You Should)
YES. (via Des Traynor)
via:destraynor infographics visualization dataviz graphics fail
Apple iOS 7 surprises as first with new multipath TCP connections - Network World
iOS 7 includes -- and uses -- multipath TCP, right now for device-to-Siri communications.
MPTCP is a TCP extension that enables the simultaneous use of several IP addresses or interfaces. Existing applications – completely unmodified -- see what appears to be a standard TCP interface. But under the covers, MPTCP is spreading the connection's data across several subflows, sending it over the least congested paths.
ios7 ios networking apple mptcp tcp protocols fault-tolerance
_How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP_ [pdf]
-
'a client-side database that supports the complete DynamoDB API, but doesn't manipulate any tables or data in DynamoDB itself. You can write code while sitting in a tree, on the beach, or in the desert. When you are ready to deploy your application, you simply instruct it to connect to the actual DynamoDB endpoint. No other modifications will be needed.' This is good -- an in-memory data store for integration testing is absolutely vital for production usage. (Voldemort does this well, for example.)
Excellent Rob Pike quote about algorithmic complexity
'Fancy algorithms are slow when n is small, and n is usually small.' -- Rob Pike
Been there, bought the t-shirt ;)Raft: The Understandable Distributed Consensus Protocol
good slides explaining the Raft protocol
RSA warns developers not to use RSA products
In case you're missing the story here, Dual_EC_DRBG (which I wrote about yesterday) is the random number generator voted most likely to be backdoored by the NSA. The story here is that -- despite many valid concerns about this generator -- RSA went ahead and made it the default generator used for all cryptography in its flagship cryptography library. The implications for RSA and RSA-based products are staggering. In a modestly bad but by no means worst case, the NSA may be able to intercept SSL/TLS connections made by products implemented with BSafe.
bsafe rsa crypto backdoors nsa security dual_ec_drbg rngs randomness
-
This is exactly my problem with Cucumber and similar BDD test frameworks.
When I write a Cucumber feature, I have to write the Gherkin that describes the acceptance criteria, and the Ruby code that implements the step definitions. Since the code to implement the step definitions is just normal RSpec (or whichever testing library you use), if someone else is writing the Gherkin, the amount of setup to create a working test should be about the same. So you're only breaking even! However, I don't believe that it would really be breaking even. Cucumber adds another layer of indirection on top of your tests. When I'm trying to see why a specific scenario is failing, first I need to find the step that is failing. Since these steps are defined with regular expressions, I have to grep for the step definition.
Gamasutra - Opinion: The tragedy of Grand Theft Auto V
This is watching your sharp, witty father start telling old fart jokes as his mind slows down. And as much as the internet is habituated to defending GTA as "satire," what is it satirizing, if everything is either sad or awful? Where is the "satire" when the awful parts no longer seem edgy or provocative, just attempts at catch-all "offense" that aren't honed enough to even connect? Here's a series that has been creating real, meaningful friction with conventional entertainment for as long as I can remember, and rather than push the envelope by creating new kinds of monsters, it's reciting the same old gangland fantasies, like a college boy who can't stop staring at the Godfather II poster on his wall, talking about how he's gonna be a big Hollywood director in between bong rips. You call the trading index BAWSAQ? Oh, bro, you're so funny, you're gonna be huge.
CCC | Chaos Computer Club breaks Apple TouchID
"We hope that this finally puts to rest the illusions people have about fingerprint biometrics. It is plain stupid to use something that you can´t change and that you leave everywhere every day as a security token", said Frank Rieger, spokesperson of the CCC. "The public should no longer be fooled by the biometrics industry with false security claims. Biometrics is fundamentally a technology designed for oppression and control, not for securing everyday device access." iPhone users should avoid protecting sensitive data with their precious biometric fingerprint not only because it can be easily faked, as demonstrated by the CCC team. Also, you can easily be forced to unlock your phone against your will when being arrested. Forcing you to give up your (hopefully long) passcode is much harder under most jurisdictions than just casually swiping your phone over your handcuffed hands.
-
OfCom has published a report on online piracy, which found that the practice is becoming less common and that pirates tend to spend more on legitimate content than non-pirates. The research, which was not funded by the entertainment industry, was conducted by Kantar Media among 21,474 participants and took place in 2012 across four separate stages. Over that time, the ratio of legal to illegal content fell -- confirming a suspected trend as legal streaming options became more available. It also confirmed another suspicion -- that a relatively small number of web users are responsible for most piracy. In OfCom's data, just two percent of users conducted three quarters of all piracy. Ofcom described piracy as "a minority activity". Of those surveyed, 58 percent accessed music, movie or TV content online, while 17 percent accessed illegal content sources. Those who admitted pirating content spent on average £26 every three months on legitimate content, set against an average spend of £16 among non-pirates.
Want to back an Irish Microbrewery?
The excellent Trouble Brewing are looking for investors
trouble-brewing ireland brewing beer business investment crowdfunding microbreweries
_An Improved Construction For Counting Bloom Filters_
'A counting Bloom filter (CBF) generalizes a Bloom filter data structure so as to allow membership queries on a set that can be changing dynamically via insertions and deletions. As with a Bloom filter, a CBF obtains space savings by allowing false positives. We provide a simple hashing-based alternative based on d-left hashing called a d-left CBF (dlCBF). The dlCBF offers the same functionality as a CBF, but uses less space, generally saving a factor of two or more. We describe the construction of dlCBFs, provide an analysis, and demonstrate their effectiveness experimentally'
bloom-filter data-structures algorithms counting cbf storage false-positives d-left-hashing hashing
To solve hard problems, you need to use bricolage
In a talk about a neat software component he designed, Bruce Haddon observed that there is no way that the final structure and algorithmic behavior of this component could have been predicted, designed, or otherwise anticipated. Haddon observed that computer science serves as a source of core ideas: it provides the data structures and algorithms that are the building blocks. Meanwhile, he views software engineering as a useful set of methods to help design reliable software without losing your mind. Yet he points out that neither captures the whole experience. That's because much of the work is what Haddon calls hacking, but what others would call bricolage. Simply put, there is much trial and error: we put ideas to together and see where it goes.
This is a great post, and I agree (broadly). IMO, most software engineering requires little CS, but there are occasional moments where a single significant aspect of a project requires a particular algorithm, and would be kludgy, hacky, or over-complex to solve without it.Getting Real About Distributed System Reliability
I have come around to the view that the real core difficulty of [distributed] systems is operations, not architecture or design. Both are important but good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations. This is quite different from the view of unbreakable, self-healing, self-operating systems that I see being pitched by the more enthusiastic NoSQL hypesters. Worse yet, you can't easily buy good operations in the same way you can buy good software—you might be able to hire good people (if you can find them) but this is more than just people; it is practices, monitoring systems, configuration management, etc.
Don't use Hadoop - your data isn't that big
see also HN comments: https://news.ycombinator.com/item?id=6398650 , particularly davidmr's great one:
I suppose all of this is to say that the amount of required parallelization of a problem isn't necessarily related to the size of the problem set as is mentioned most in the article, but also the inherent CPU and IO characteristics of the problem. Some small problems are great for large-scale map-reduce clusters, some huge problems are horrible for even bigger-scale map-reduce clusters (think fluid dynamics or something that requires each subdivision of the problem space to communicate with its neighbors). I've had a quote printed on my door for years: Supercomputers are an expensive tool for turning CPU-bound problems into IO-bound problems.
I love that quote!-
Gilt ran a stress-test of Riak to replace Voldemort (I think) in a shadow stack, with good results:
Riak's strong performance suggests that, should we pursue implementation, it will withstand our unique traffic needs and prove reliable. As for the Gilt-Basho team's strong performance: It was amazing that we were able to accomplish so much in just a week's time! Thanks again to Seth and Steve for making this possible.
THE LONG DARK, a first-person post-disaster survival sim by Hinterland — Kickstarter
wow this looks great.
The Long Dark is a thoughtful, first-person survival simulation that emphasizes quiet exploration in a stark, yet hauntingly beautiful, post-disaster setting. The breathtakingly picturesque Pacific Northwest frames the backdrop for the drama of The Long Dark.
The Rational Choices of Crack Addicts - NYTimes.com
"The key factor is the environment, whether you're talking about humans or rats," Dr. Hart said. "The rats that keep pressing the lever for cocaine are the ones who are stressed out because they've been raised in solitary conditions and have no other options. But when you enrich their environment, and give them access to sweets and let them play with other rats, they stop pressing the lever."
Inside the mind of NSA chief Gen Keith Alexander | Glenn Greenwald
featuring some mental pics of the "Information Dominance Center", the Star Trek bridge which NSA chief Keith Alexander built with taxpayer money
big-brother nsa politics keith-alexander star-trek funny bizarre
Schneier on Security: Reforming the NSA
Regardless of how we got here, the NSA can't reform itself. Change cannot come from within; it has to come from above. It's the job of government: of Congress, of the courts, and of the president. These are the people who have the ability to investigate how things became so bad, rein in the rogue agency, and establish new systems of transparency, oversight, and accountability. Any solution we devise will make the NSA less efficient at its eavesdropping job. That's a trade-off we should be willing to make, just as we accept reduced police efficiency caused by requiring warrants for searches and warning suspects that they have the right to an attorney before answering police questions. We do this because we realize that a too-powerful police force is itself a danger, and we need to balance our need for public safety with our aversion of a police state.
nsa politics us-politics surveillance snooping society government police public-safety police-state
Biometric authentication failing in Mysore
Biometrics was rolled out for food distribution in order to cut down on fraud, but it's now resulting in a subset of users being unable to authenticate:
The biometric authentication system installed at the PDS outlets fails to establish the identity of many genuine beneficiaries, mostly workers, as their daily grind in the agricultural fields, construction sites or as domestic help have eroded the lines on their thumb resulting in distorted impressions.
fail risks biometrics authentication mysore security india fingerprinting
Sketch of the Day – Frugal Streaming
ha, this is very clever! If you have enough volume, this is a nice estimation algorithm to compute stream quantiles in very little RAM
memory streaming stream-processing clever algorithms hacks streams
-
Spam Arrest is a company that sells an anti-spam service. They attempted to sue some spammers and, as has been widely reported, lost badly. This case emphasizes three points that litigious antispammers seem not to grasp: Under CAN SPAM, a lot of spam is legal. Judges hate plaintiffs who try to be too clever, and hate sloppy preparation even more. Never, ever, file a spam suit in Seattle.
anti-spam spam law seattle us can-spam spamarrest sentient-jets
Benchmarking Redis on AWS ElastiCache
good data points, but could do with latency percentiles
latency redis measurement benchmarks ec2 elasticache aws storage tests
Being poor changes your thinking about everything
Very interesting research into poverty and scarcity, in the Washington Post:
The scarcity trap captures this notion we see again and again in many domains. When people have very little, they undertake behaviors that maintain or reinforce their future disadvantage. If you have very little, you often behave in such a way so that you'll have little in the future. In economics, people talk about the poverty trap. We're generalizing that, saying this happens a lot, and we've experienced it.
Good SSL for your website is absurdly difficult in practice
Yet again, security software fails on packaging and UI. via Tony Finch
Former NSA and CIA director says terrorists love using Gmail
At one point, Hayden expressed a distaste for online anonymity, saying "The problem I have with the Internet is that it's anonymous." But he noted, there is a struggle over that issue even inside government. The issue came to a head during the Arab Spring movement when the State Department was funding technology [presumably Tor?] to protect the anonymity of activists so governments could not track down or repress their voices. "We have a very difficult time with this," Hayden said. He then asked, "is our vision of the World Wide Web the global digital commons -- at this point you should see butterflies flying here and soft background meadow-like music -- or a global free fire zone?" Given that Hayden also compared the Internet to the wild west and Somalia, Hayden clearly leans toward the "global free fire zone" vision of the Internet.
well, that's a good analogy for where we're going -- a global free-fire zone.gmail cia nsa surveillance michael-hayden security snooping law tor arab-spring
Google swaps out MySQL, moves to MariaDB
When we asked Sallner to quantify the scale of the migration he said, "They're moving it all. Everything they have. All of the MySQL servers are moving to MariaDB, as far as I understand." By moving to MariaDB, Google can free itself of any dependence on technology dictated by Oracle – a company whose motivations are unclear, and whose track record for working with the wider technology community is dicey, to say the least. Oracle has controlled MySQL since its acquisition of Sun in 2010, and the key InnoDB storage engine since it got ahold of Innobase in 2005. [...] We asked Cole why Google would shift from MySQL to MariaDB, and what the key technical differences between the systems were. "From my perspective, they're more or less equivalent other than if you look at specific features and how they implement them," Cole said, speaking in a personal capacity and not on behalf of Google. "Ideologically there are lots of differences."
So -- AWS, when will RDS offer MariaDB as an option?google mysql mariadb sql open-source licensing databases storage innodb oracle
FBI Admits It Controlled Tor Servers Behind Mass Malware Attack
The code's behavior, and the command-and-control server's Virginia placement, is also consistent with what's known about the FBI's "computer and internet protocol address verifier," or CIPAV, the law enforcement spyware first reported by WIRED in 2007. Court documents and FBI files released under the FOIA have described the CIPAV as software the FBI can deliver through a browser exploit to gather information from the target's machine and send it to an FBI server in Virginia. The FBI has been using the CIPAV since 2002 against hackers, online sexual predators, extortionists, and others, primarily to identify suspects who are disguising their location using proxy servers or anonymity services, like Tor. Prior to the Freedom Hosting attack, the code had been used sparingly, which kept it from leaking out and being analyzed.
-
lots more detail on the new "Java Mission Control" feature in Hotspot 7u40 JVMs, and how to use it to start and stop profiling in a live, production JVM from a separate "jcmd" command-line client. If the overhead is small, this could be really neat -- turn on profiling for 1 minute every hour on a single instance, and collect realtime production profile data on an automated basis for post-facto analysis if required
Necessary and Proportionate -- In Which Civil Society is Caught Between a Cop and a Spy
Modern telecommunications technology implied the development of modern telecommunications surveillance, because it moved the scope of action from the physical world (where intelligence, generally seen as part of the military mission, had acted) to the virtual world—including the scope of those actions that could threaten state power. While the public line may have been, as US Secretary of State Henry Stimson said in 1929, "gentlemen do not open each other's mail", you can bet that they always did keep a keen eye on the comings and goings of each other's shipping traffic. The real reason that surveillance in the context of state intelligence was limited until recently was because it was too expensive, and it was too expensive for everyone. The Westphalian compromise demands equality of agency as tied to territory. As soon as one side gains a significant advantage, the structure of sovereignty itself is threatened at a conceptual level?—?hence Oppenheimer as the death of any hope of international rule of law. Once surveillance became cheap enough, all states were (and will increasingly be) forced to attempt it at scale, as a reaction to this pernicious efficiency. The US may be ahead of the game now, but Moore's law and productization will work their magic here.
government telecoms snooping gchq nsa surveillance law politics intelligence spying internet
-
Bit of detail into Twitter's TSD metric store.
There are separate online clusters for different data sets: application and operating system metrics, performance critical write-time aggregates, long term archives, and temporal indexes. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired. Aggregation is generally performed at write-time to avoid extra storage operations for metrics that are expected to be immediately consumed. Indexing occurs along several dimensions–service, source, and metric names–to give users some flexibility in finding relevant data.
twitter monitoring metrics service-metrics tsd time-series storage architecture cassandra
NSA: Possibly breaking US laws, but still bound by laws of computational complexity
I didn't clearly explain that there's an enormous continuum between, on the one hand, a full break of RSA or Diffie-Hellman (which still seems extremely unlikely to me), and on the other, "pure side-channel attacks" involving no new cryptanalytic ideas. Along that continuum, there are many plausible places where the NSA might be. For example, imagine that they had a combination of side-channel attacks, novel algorithmic advances, and sheer computing power that enabled them to factor, let's say, ten 2048-bit RSA keys every year. In such a case, it would still make perfect sense that they'd want to insert backdoors into software, sneak vulnerabilities into the standards, and do whatever else it took to minimize their need to resort to such expensive attacks. But the possibility of number-theoretic advances well beyond what the open world knows certainly wouldn't be ruled out. Also, as Schneier has emphasized, the fact that NSA has been aggressively pushing elliptic-curve cryptography in recent years invites the obvious speculation that they know something about ECC that the rest of us don't.
ecc rsa crypto security nsa gchq snooping sniffing diffie-hellman pki key-length
-
Built into the HotSpot JVM [in JDK version 7u40] is something called the Java Flight Recorder. It records a lot of information about/from the JVM runtime, and can be thought of as similar to the Data Flight Recorders you find in modern airplanes. You normally use the Flight Recorder to find out what was happening in your JVM when something went wrong, but it is also a pretty awesome tool for production time profiling. Since Mission Control (using the default templates) normally don't cause more than a per cent overhead, you can use it on your production server.
I'm intrigued by the idea of always-on profiling in production. This could be cool.performance java measurement profiling jvm jdk hotspot mission-control instrumentation telemetry metrics
How the NSA Spies on Smartphones
One of the US agents' tools is the use of backup files established by smartphones. According to one NSA document, these files contain the kind of information that is of particular interest to analysts, such as lists of contacts, call logs and drafts of text messages. To sort out such data, the analysts don't even require access to the iPhone itself, the document indicates. The department merely needs to infiltrate the target's computer, with which the smartphone is synchronized, in advance. Under the heading "iPhone capability," the NSA specialists list the kinds of data they can analyze in these cases. The document notes that there are small NSA programs, known as "scripts," that can perform surveillance on 38 different features of the iPhone 3 and 4 operating systems. They include the mapping feature, voicemail and photos, as well as the Google Earth, Facebook and Yahoo Messenger applications.
and, of course, the alternative means of backup is iCloud.... wonder how secure those backups are.nsa surveillance gchq iphone smartphones backups icloud security
-
Boost ASIO at the front end (!), Kafka 0.8, Storm, and ElasticSearch
boost scalability loggly logging ingestion cep stream-processing kafka storm architecture elasticsearch
Schneier on Security: Excess Automobile Deaths as a Result of 9/11
The inconvenience of extra passenger screening and added costs at airports after 9/11 cause many short-haul passengers to drive to their destination instead, and, since airline travel is far safer than car travel, this has led to an increase of 500 U.S. traffic fatalities per year. Using DHS-mandated value of statistical life at $6.5 million, this equates to a loss of $3.2 billion per year, or $32 billion over the period 2002 to 2011 (Blalock et al. 2007).
risk security death 9-11 politics screening dhs air-travel driving road-safety
-
The debate has been stifled in Britain more successfully than anywhere else in the free world and, astonishingly, this has been with the compliance of a media and public that regard their attachment to liberty to be a matter of genetic inheritance. So maybe it is best for me to accept that the BBC, together with most of the newspapers, has moved with society, leaving me behind with a few old privacy-loving codgers, wondering about the cause of this shift in attitudes. Is it simply the fear of terror and paedophiles? Are we so overwhelmed by the power of the surveillance agencies that we feel we can't do anything? Or is it that we have forgotten how precious and rare truly free societies are in history?
privacy uk politics snooping spies gchq society nsa henry-porter
-
Some great street art from Brighton, via Darach Ennis
via:darachennis street-art graffiti big-data snooping spies gchq nsa art
Blocking The Pirate Bay appears to have 'no lasting net impact' on illegal downloading
In the fight against the unauthorised sharing of copyright protected material, aka piracy, Dutch Internet Service Providers have been summoned by courts to block their subscribers' access to The Pirate Bay (TPB) and related sites. This paper studies the effectiveness of this approach towards online copyright enforcement, using both a consumer survey and a newly developed non-infringing technology for BitTorrent monitoring. While a small group of respondents download less from illegal sources or claim to have stopped, and a small but significant effect is found on the distribution of Dutch peers, no lasting net impact is found on the percentage of the Dutch population downloading from illegal sources.
fail blocking holland pirate-bay tpb papers via:tjmcintyre internet isps
How Advanced Is the NSA's Cryptanalysis — And Can We Resist It?
Bruce Schneier's suggestions:
Assuming the hypothetical NSA breakthroughs don't totally break public-cryptography — and that's a very reasonable assumption — it's pretty easy to stay a few steps ahead of the NSA by using ever-longer keys. We're already trying to phase out 1024-bit RSA keys in favor of 2048-bit keys. Perhaps we need to jump even further ahead and consider 3072-bit keys. And maybe we should be even more paranoid about elliptic curves and use key lengths above 500 bits. One last blue-sky possibility: a quantum computer. Quantum computers are still toys in the academic world, but have the theoretical ability to quickly break common public-key algorithms — regardless of key length — and to effectively halve the key length of any symmetric algorithm. I think it extraordinarily unlikely that the NSA has built a quantum computer capable of performing the magnitude of calculation necessary to do this, but it's possible. The defense is easy, if annoying: stick with symmetric cryptography based on shared secrets, and use 256-bit keys.
bruce-schneier cryptography wired nsa surveillance snooping gchq cryptanalysis crypto future key-lengths
DevOps Eye for the Coding Guy: Metrics
a pretty good description of the process of adding service metrics to a Django webapp using graphite and statsd. Bookmarking mainly for the great real-time graphing hack at the end...
Probabalistic Scraping of Plain Text Tables
a nifty hack.
Recently I have been banging my head trying to import a ton of OCR acquired data expressed in tabular form. I think I have come up with a neat approach using probabilistic reasoning combined with mixed integer programming. The method is pretty robust to all sorts of real world issues. In particular, the method leverages topological understanding of tables, encodes it declaratively into a mixed integer/linear program, and integrates weak probabilistic signals to classify the whole table in one go (at sub second speeds). This method can be used for any kind of classification where you have strong logical constraints but noisy data.
(via proggit)scraping tables ocr probabilistic linear-programming optimization machine-learning via:proggit
-
'Plugin to make highly interactive graphite graph objects ((i.e. graphs where you can interactively toggle on/off individual series, inspect datapoints, zoom in realtime, etc) Supports Flot (canvas), Rickshaw (svg) and standard graphite png images (in case you're nostalgic and don't like interactivity).'
graphs graphing graphite dataviz flot rickshaw svg canvas javascript
modern JVM concurrency primitives are broken if the system clock steps backwards
'The implementation of the concurrency primitive LockSupport.parkNanos(), the function that controls *every* concurrency primitive on the JVM, is flawed, and any NTP sync, or system time change, can potentially break it with unexpected results across the board when running a 64bit JVM on Linux 64bit.' Basically, LockSupport.parkNanos() calls pthread_cond_timedwait() using a CLOCK_REALTIME instead of CLOCK_MONOTONIC. 'tinker step 0' in ntp.conf may be a viable workaround.
clocks timing ntp slew sync step pthreads java jvm timers clock_realtime clock_monotonic
Schneier on Security: The NSA Is Breaking Most Encryption on the Internet
The new Snowden revelations are explosive. Basically, the NSA is able to decrypt most of the Internet. They're doing it primarily by cheating, not by mathematics. It's joint reporting between the Guardian, the New York Times, and ProPublica. I have been working with Glenn Greenwald on the Snowden documents, and I have seen a lot of them. These are my two essays on today's revelations. Remember this: The math is good, but math has no agency. Code has agency, and the code has been subverted.
encryption communication government nsa security bruce-schneier crypto politics snooping gchq guardian journalism
How To Buffer Full YouTube Videos Before Playing
summary - turn off DASH (Dynamic adaptive streaming) using a userscript.
Voldemort on Solid State Drives [paper]
'This paper and talk was given by the LinkedIn Voldemort Team at the Workshop on Big Data Benchmarking (WBDB May 2012).'
With SSD, we find that garbage collection will become a very significant bottleneck, especially for systems which have little control over the storage layer and rely on Java memory management. Big heapsizes make the cost of garbage collection expensive, especially the single threaded CMS Initial mark. We believe that data systems must revisit their caching strategies with SSDs. In this regard, SSD has provided an efficient solution for handling fragmentation and moving towards predictable multitenancy.
voldemort storage ssd disk linkedin big-data jvm tuning ops gc
Streaming MapReduce with Summingbird
Before Summingbird at Twitter, users that wanted to write production streaming aggregations would typically write their logic using a Hadoop DSL like Pig or Scalding. These tools offered nice distributed system abstractions: Pig resembled familiar SQL, while Scalding, like Summingbird, mimics the Scala collections API. By running these jobs on some regular schedule (typically hourly or daily), users could build time series dashboards with very reliable error bounds at the unfortunate cost of high latency. While using Hadoop for these types of loads is effective, Twitter is about real-time and we needed a general system to deliver data in seconds, not hours. Twitter's release of Storm made it easy to process data with very low latencies by sacrificing Hadoop's fault tolerant guarantees. However, we soon realized that running a fully real-time system on Storm was quite difficult for two main reasons: Recomputation over months of historical logs must be coordinated with Hadoop or streamed through Storm with a custom log loading mechanism; Storm is focused on message passing and random-write databases are harder to maintain. The types of aggregations one can perform in Storm are very similar to what's possible in Hadoop, but the system issues are very different. Summingbird began as an investigation into a hybrid system that could run a streaming aggregation in both Hadoop and Storm, as well as merge automatically without special consideration of the job author. The hybrid model allows most data to be processed by Hadoop and served out of a read-only store. Only data that Hadoop hasn't yet been able to process (data that falls within the latency window) would be served out of a datastore populated in real-time by Storm. But the error of the real-time layer is bounded, as Hadoop will eventually get around to processing the same data and will smooth out any error introduced. This hybrid model is appealing because you get well understood, transactional behavior from Hadoop, and up to the second additions from Storm. Despite the appeal, the hybrid approach has the following practical problems: Two sets of aggregation logic have to be kept in sync in two different systems; Keys and values must be serialized consistently between each system and the client. The client is responsible for reading from both datastores, performing a final aggregation and serving the combined results Summingbird was developed to provide a general solution to these problems.
Very interesting stuff. I'm particularly interested in the design constraints they've chosen to impose to achieve this -- data formats which require associative merging in particular.mapreduce streaming big-data twitter storm summingbird scala pig hadoop aggregation merging
Thoughts on Granby Park, the recent pop-up park off Parnell St
We mentioned above that pop-up spaces have become popular across Europe because they allow developers and city councils to harness urban creativity in order to drive up real estate prices without ceding control of a given site. Those who produce the space through hard work, collaboration and passion move on, making way for property development and speculation. The international research in this area is very clear on this point and it has been documented in places from Lower-East Side Manhattan to Berlin's Kreuzberg. Most perversely, increased property prices make it even more difficult for creativity to flourish in a given area and end up driving out long-term working class communities, migrants and young people. But what can we do? If every attempt we make to make our city a better place simply ends up being captured in the calculations of real estate players, surely the situation is hopeless? Is it better, then, to do nothing? We don't think it is better to do nothing and, like Upstart, we still believe we can find a way together through experimentation and collaboration. However, this means questioning, reflecting on and publicly discussing the relationship between our efforts to make a city more after our hearts desire and the process of gentrification. As noted above, this is especially the case with pop-up spaces given their temporary nature. It is really necessary that we think about how to make sure our activities don't contribute to gentrification in the long term, but instead benefit the city as a whole. We certainly don't have the solutions, but if we sweep these awkward questions under the carpet we risk contributing to the very forces we want to challenge and alienating those who will perceive us as the 'front-line' of gentrification.
gentrification pop-up parks dublin ireland cities upstart spaces urban-planning
[#CASSANDRA-5582] Replace CustomHsHaServer with better optimized solution based on LMAX Disruptor
Disruptor: decimating P99s since 2011
disruptor cassandra java p99 latency speed performance concurrency via:kellabyte
-
I love these.
Photographic prints are great because they don't need power to be displayed. They are more or less permanent. Videos are great because they record a sequence of time which shows reality almost like how we experience. Is it possible to combine the two? And not via long exposure photography where often details are lost from motion. So I played around with the tools of digital photography and post processing to give you this series: Time is a dimension. This series of images are mostly landscapes, seascapes and cityscapes, and they are a single composite made from sequences that span 2-4 hours, mostly of sunrises and sunsets. The basic structure of a landscape is present in every piece. But each panel or concentric layer shows a different slice of time, which is related to the adjacent panel/layer. The transition from daytime to night is gradual and noticeable in every piece, but would not be something you expect to see in a still image.
photography beautiful photos art time dimensions prints via:matthaughey
-
'Visualizations that make no sense.' Some of these are unintentional comedy gold -- pie charts feature heavily, of course. (via Des Traynor)
via:destraynor infographics wtf visualization dataviz data fail funny graphics pie-charts
Non-blocking transactional atomicity
Peter Bailis with an interesting distributed-storage atomicity algorithm for performing multi-record transactional updates
algorithms nbta transactions databases storage distcomp distributed atomic coding eventual-consistency crdts
Interview with the Github Elasticsearch Team
good background on Github's Elasticsearch scaling efforts. Some rather horrific split-brain problems under load, and crashes due to OpenJDK bugs (sounds like OpenJDK *still* isn't ready for production). painful
elasticsearch github search ops scaling split-brain outages openjdk java jdk jvm
The Irish Times, terminations and Holles Street: The story that wasn't there.
Summarising a very shoddy tale from our paper of record.
I don't know what happened here. I don't know whether there ever was a woman who met the description given by the Irish Times who suffered a medical crisis during pregnancy. I don't know why a group of men in positions of authority in the Irish Times decided that, if there was such a woman, they had any right to tell the rest of the country about her experiences. I don't know why, when they discovered that a mistake had been made in the one legal fact used to justify that decision they didn't immediately apologise. And I don't know what happened between the 23rd August 2013 and 31st August 2013 to prompt them to print a shoulder shrugging 'acceptance' that the case 'hadn't happened' and limit the paper's apology to an institution, as opposed to its readers. But, from what I've seen this week, I do know one thing. Whatever questions readers might have, The Irish Times isn't interested in giving them any answers.
irish-times fail shoddy abortion health public-interest journalism pregnancy corrections
-
Rackspace's large-scale TSD storage system, built on Cassandra, Java, ASL2
cassandra tsd storage time-series data open-source java rackspace
Reversing Sinclair's amazing 1974 calculator hack - half the ROM of the HP-35
Amazing reverse engineering.
In a hotel room in Texas, Clive Sinclair had a big problem. He wanted to sell a cheap scientific calculator that would grab the market from expensive calculators such as the popular HP-35. Hewlett-Packard had taken two years, 20 engineers, and a million dollars to design the HP-35, which used 5 complex chips and sold for $395. Sinclair's partnership with calculator manufacturer Bowmar had gone nowhere. Now Texas Instruments offered him an inexpensive calculator chip that could barely do four-function math. Could he use this chip to build a $100 scientific calculator? Texas Instruments' engineers said this was impossible - their chip only had 3 storage registers, no subroutine calls, and no storage for constants such as ?. The ROM storage in the calculator held only 320 instructions, just enough for basic arithmetic. How could they possibly squeeze any scientific functions into this chip? Fortunately Clive Sinclair, head of Sinclair Radionics, had a secret weapon - programming whiz and math PhD Nigel Searle. In a few days in Texas, they came up with new algorithms and wrote the code for the world's first single-chip scientific calculator, somehow programming sine, cosine, tangent, arcsine, arccos, arctan, log, and exponentiation into the chip. The engineers at Texas Instruments were amazed. How did they do it? Up until now it's been a mystery. But through reverse engineering, I've determined the exact algorithms and implemented a simulator that runs the calculator's actual code. The reverse-engineered code along with my detailed comments is in the window below.
reversing reverse-engineering history calculators sinclair ti hp chips silicon hacks
Microsoft CEO Steve Ballmer retires: A firsthand account of the company's employee-ranking system
LOL MS. Sadly, this talk of "core competencies" and "visibility" is pretty reminiscent of Amazon's review season, too:
This illustrated another problem with [stack ranking]: It destroyed trust between individual contributors and management, because the stack rank required that all lower-level managers systematically lie to their reports. Why? Because for years Microsoft did not admit the existence of the stack rank to nonmanagers. Knowledge of the process gradually leaked out, becoming a recurrent complaint on the much-loathed (by Microsoft) Mini-Microsoft blog, where a high-up Microsoft manager bitterly complained about organizational dysfunction and was joined in by a chorus of hundreds of employees. The stack rank finally made it into a Vanity Fair article in 2012, but for many years it was not common knowledge, inside or outside Microsoft. It was presented to the individual contributors as a system of objective assessment of "core competencies," with each person being judged in isolation. When review time came, and programmers would fill out a short self-assessment talking about their achievements, strengths, and weaknesses, only some of them knew that their ratings had been more or less already foreordained at the stack rank. [...] If you did know about the stack rank, you weren't supposed to admit it. So you went through the pageantry of the performance review anyway, arguing with your manager in the rhetoric of "core competencies." The managers would respond in kind. Since the managers had little control over the actual score and attendant bonus and raise (if any), their job was to write a review to justify the stack rank in the language of absolute merit. ("Higher visibility" was always a good catch-all: Sure, you may be a great coder and work 80 hours a week, but not enough people have heard of you!)
amazon stack-ranking employees ranking work microsoft core-competencies
BBC News - How one man turns annoying cold calls into cash
This is hilarious. Quid pro quo!
Once he had set up the 0871 line, every time a bank, gas or electricity supplier asked him for his details online, he submitted it as his contact number. He added he was "very honest" and the companies did ask why he had a premium number. He told the programme he replied: "Because I'm getting annoyed with PPI phone calls when I'm trying to watch Coronation Street so I'd rather make 10p a minute." He said almost all of the companies he dealt with were happy to use it and if they refused he asked them to email.
-
This is brilliant. Half of the office now wants prints.
Massive congratulations to Edge magazine. The stellar publication has been around for 20 years! To celebrate, their 258th issue comes in 20 different flavours, and one of those flavours includes the earthly overtones of both Minecraft and Dungeons & Dragons. Junkboy drew it, and I [Owen] worded it a few weeks ago.
covers images edge minecraft gaming funny dungeons-and-dragons retro dnd
-
Forecast.io are doing such a great job of applying modern machine-learning to traditional weather data. "Quicksilver" is their neural-net-adjusted global temperature geodata, and here's how it's built
quicksilver forecast forecast.io neural-networks ai machine-learning algorithms weather geodata earth temperature
_MillWheel: Fault-Tolerant Stream Processing at Internet Scale_ [paper, pdf]
from VLDB 2013:
MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework's fault-tolerance guarantees. This paper describes MillWheel's programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel's features are used. MillWheel's programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel's unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.
millwheel google data-processing cep low-latency fault-tolerance scalability papers event-processing stream-processing
GCHQ tapping at least 14 EU fiber-optic cables
Süddeutsche Zeitung (SZ) had already revealed in late June that the British had access to the cable TAT-14, which connects Germany with the USA, UK, Denmark, France and the Netherlands. In addition to TAT-14, the other cables that GCHQ has access to include Atlantic Crossing 1, Circe North, Circe South, Flag Atlantic-1, Flag Europa-Asia, SeaMeWe-3 and SeaMeWe-4, Solas, UK France 3, UK Netherlands-14, Ulysses, Yellow and the Pan European Crossing.
sz germany cables fiber-optic tapping snooping tat-14 eu politics gchq
In historic vote, New Zealand bans software patents | Ars Technica
This is amazing news. Paying attention, Sean Sherlock?
A major new patent bill, passed in a 117-4 vote by New Zealand's Parliament after five years of debate, has banned software patents. The relevant clause of the patent bill actually states that a computer program is "not an invention." Some have suggested that was a way to get around the wording of the TRIPS intellectual property treaty, which requires patents to be "available for any inventions, whether products or processes, in all fields of technology." [...] One Member of Parliament who was deeply involved in the debate, Clare Curran, quoted several heads of software firms complaining about how the patenting process allowed "obvious things" to get patented and that "in general software patents are counter-productive." Curran quoted one developer as saying, "It's near impossible for software to be developed without breaching some of the hundreds of thousands of patents granted around the world for obvious work." "These are the heavyweights of the new economy in software development," said Curran. "These are the people that needed to be listened to, and thankfully, they were."
new-zealand nz patents swpats law trips ip software-patents yay
-
Docker is to deployment as Git is to development. Developers are able to leverage Git's performance and flexibility when building applications. Git encourages experiments and doesn't punish you when things go wrong: start your experiments in a branch, if things fall down, just git rebase or git reset. It's easy to start a branch and fast to push it. Docker encourages experimentation for operations. Containers start quickly. Building images is a snap. Using another images as a base image is easy. Deploying whole images is fast, and last but not least, it's not painful to rollback. Fast + flexible = deployments are about to become a lot more enjoyable.
docker deployment sysadmin ops devops vms vagrant virtualization containers linux git
-
how LI solved a tricky graph-database-query latency problem with a set-cover algorithm
linkedin algorithms coding distributed-systems graph databases querying set-cover set replication
How might the feds have snooped on Lavabit?
"I have been told that they cannot change your fundamental business practices," said Callas, who unlike Levison was able to say SilentCircle has received no NSLs or court orders of any kind. "I presume that would mean things like getting SSL keys because that would mean they could impersonate your servers. That would be like setting up a store front that says your business name and putting [government agents] in your company uniforms." Similarly, he added: "They cannot make changes to existing operating systems. They can't make you change source code." To which [Lavabit's] Levison replied: "That was always my understanding, too. That's why this is so important. Like [Callas] at SilentCircle said, the assumption has been that the government can't force us to change our business practices like that and compromise that information. Like I said, I don't hold those beliefs anymore."
ars-technica security privacy nsls ssl silentcircle jon-callas crypto
Lock-Based vs Lock-Free Concurrent Algorithms
An excellent post from Martin Thompson showing a new JSR166 concurrency primitive, StampedLock, compared against a number of alternatives in a simple microbenchmark. The most interesting thing for me is how much the lock-free, AtomicReference.compareAndSet()-based approach blows away all the lock-based approaches -- even in the 1-reader-1-writer case. Its code is extremely simple, too: https://github.com/mjpt777/rw-concurrency/blob/master/src/LockFreeSpaceship.java
concurrency java threads lock-free locking compare-and-set cas atomic jsr166 microbenchmarks performance
-
This is super-cool. 'Network engineering no longer should be mundane tasks like conf, set interfaces fe-0/0/0 unit o family inet address 10.1.1.1/24. How does mindless CLI work translate to efficiently spent time ? What if you need to change 300 devices? What if you are writing it by hand? An error-prone waste of time. Juniper today announced Puppet support for their 12.2R3,5 JUNOS code. This is compatible with EX4200, EX4550, and QFX3500 switches. These are top end switches, but this start is directly aimed at their DC and enterprise devices. Initially, the manifest interactions offered are interface, layer 2 interface, vlan, port aggregation groups, and device names.' Based on what I saw in the Network Automation team in Amazon, this is an amazing leap forward; it'd instantly render obsolete a bunch of horrific SSH-CLI automation cruft.
ssh cli automation networking networks puppet ops juniper cisco
-
The future of the AWS command line tools is awscli, a single, unified, consistent command line tool that works with almost all of the AWS services. Here is a quick list of the services that awscli currently supports: Auto Scaling, CloudFormation, CloudSearch, CloudWatch, Data Pipeline, Direct Connect, DynamoDB, EC2, ElastiCache, Elastic Beanstalk, Elastic Transcoder, ELB, EMR, Identity and Access Management, Import/Export, OpsWorks, RDS, Redshift, Route 53, S3, SES, SNS, SQS, Storage Gateway, Security Token Service, Support API, SWF, VPC. Support for the following appears to be planned: CloudFront, Glacier, SimpleDB. The awscli software is being actively developed as an open source project on Github, with a lot of support from Amazon. You'll note that the biggest contributors to awscli are Amazon employees with Mitch Garnaat leading. Mitch is also the author of boto, the amazing Python library for AWS.
-
Absolute genius from The Onion.
Those of us watching on Google Analytics saw the number of homepage visits skyrocket the second we put up that salacious image of Miley Cyrus dancing half nude on the VMA stage. But here's where it gets great: We don't just do a top story on the VMA performance and call it a day. No, no. We also throw in a slideshow called "Evolution of Miley," which, for those of you who don't know, is just a way for you to mindlessly click through 13 more photos of Miley Cyrus. And if we get 500,000 of you to do that, well, 500,000 multiplied by 13 means we can get 6.5 million page views on that slideshow alone. Throw in another slideshow titled "6 'don't miss' VMA moments," and it's starting to look like a pretty goddamned good Monday, numbers-wise. Also, there are two videos -- one of the event and then some bullshit two-minute clip featuring our "entertainment experts" talking about the performance. Side note: Advertisers, along with you idiots, love videos. Another side note: The Miley Cyrus story was in the same top spot we used for our 9/11 coverage.
humor journalism cnn miley-cyrus vma news funny advertising ads
Why wireless mesh networks won't save us from censorship
I'm not saying mesh networks don't work ever; the people in the wireless mesh community I've met are all great people doing fantastic work. What I am saying is that unplanned wireless mesh networks never work at scale. I think it's a great problem to think about, but in terms of actual allocation of time and resources I think there are other, more fruitful avenues of action to fight Internet censorship.
(via Kragen)wireless censorship internet networking mesh mesh-networks organisation scaling wifi
Information on Google App Engine's recent US datacenter relocations - Google Groups
or, really, 'why we had some glitches and outages recently'. A few interesting tidbits about GAE innards though (via Bill De hOra)
gae google app-engine outages ops paxos eventual-consistency replication storage hrd
Newest YouTube user to fight a takedown is copyright guru Lawrence Lessig
This is lovely. Here's hoping it provides a solid precedent.
Illegitimate or simply unnecessary copyright claims are, unfortunately, commonplace in the Internet era. But if there's one person who's probably not going to back down from a claim of copyright infringement, it's Larry Lessig, one of the foremost writers and thinkers on digital-age copyright. [..] If Liberation Music was thinking they'd have an easy go of it when they demanded that YouTube take down a 2010 lecture of Lessig's entitled "Open," they were mistaken. Lessig has teamed up with the Electronic Frontier Foundation to sue Liberation, claiming that its overly aggressive takedown violates the DMCA and that it should be made to pay damages.
-
Great account from Cliff Click describing an interest edge-case risk of using TCP without application-level acking, and how it caused a messy intermittent bug in production.
In all these failures the common theme is that the receiver is very heavily loaded, with many hundreds of short-lived TCP connections being opened/read/closed every second from many other machines. The sender sends a 'SYN' packet, requesting a connection. The sender (optimistically) sends 1 data packet; optimistic because the receiver has yet to acknowledge the SYN packet. The receiver, being much overloaded, is very slow. Eventually the receiver returns a 'SYN-ACK' packet, acknowledging both the open and the data packet. At this point the receiver's JVM has not been told about the open connection; this work is all opening at the OS layer alone. The sender, being done, sends a 'FIN' which it does NOT wait for acknowledgement (all data has already been acknowledged). The receiver, being heavily overloaded, eventually times-out internally (probably waiting for the JVM to accept the open-call, and the JVM being overloaded is too slow to get around to it) – and sends a RST (reset) packet back…. wiping out the connection and the data. The sender, however, has moved on – it already sent a FIN & closed the socket, so the RST is for a closed connection. Net result: sender sent, but the receiver reset the connection without informing either the JVM process or the sender.
The ultimate SO_LINGER page, or: why is my tcp not reliable
If we look at the HTTP protocol, there data is usually sent with length information included, either at the beginning of an HTTP response, or in the course of transmitting information (so called 'chunked' mode). And they do this for a reason. Only in this way can the receiving end be sure it received all information that it was sent. Using the shutdown() technique above really only tells us that the remote closed the connection. It does not actually guarantee that all data was received correctly by program B. The best advice is to send length information, and to have the remote program actively acknowledge that all data was received.
SO_LINGER sockets tcp ip networking linux protocols shutdown FIN RST
NZ police affidavits show use of PRISM for surveillance of Kim "Megaupload" Dotcom
The discovery was made by blogger Keith Ng who wrote on his On Point blog (http://publicaddress.net/onpoint/ich-bin-ein-cyberpunk/) that the Organised and Financial Crime Agency New Zealand (OFCANZ) requested assistance from the Government Communications Security Bureau (GCSB), the country's signals intelligence unit, which is charge of surveilling the Pacific region under the Five-Eyes agreement. A list of so-called selectors or search terms were provided to GCSB by the police [PDF, redacted] for the surveillance of emails and other data traffic generated by Dotcom and his Megaupload associates. 'Selectors' is the term used for the National Security Agency (NSA) XKEYSCORE categorisation system that Australia and New Zealand contribute to and which was leaked by Edward Snowden as part of his series of PRISM revelations. Some "selectors of interest" have been redacted out, but others such as Kim Dotcom's email addresses, the mail proxy server used for some of the accounts and websites, remain in the documents.
So to recap; police investigating an entirely non-terrorism-related criminal case in NZ was given access to live surveillance traffic for surveillance of an NZ citizen. Scary stuffsurveillance prism nsa new-zealand xkeyscore gcsb kim-dotcom piracy privacy data-retention megaupload filesharing
"Scalable Eventually Consistent Counters over Unreliable Networks" [paper, pdf]
Counters are an important abstraction in distributed computing, and play a central role in large scale geo-replicated systems, counting events such as web page impressions or social network "likes". Classic distributed counters, strongly consistent, cannot be made both available and partition-tolerant, due to the CAP Theorem, being unsuitable to large scale scenarios. This paper defines Eventually Consistent Distributed Counters (ECDC) and presents an implementation of the concept, Handoff Counters, that is scalable and works over unreliable networks. By giving up the sequencer aspect of classic distributed counters, ECDC implementations can be made AP in the CAP design space, while retaining the essence of counting. Handoff Counters are the first CRDT (Conflict-free Replicated Data Type) based mechanism that overcomes the identity explosion problem in naive CRDTs, such as G-Counters (where state size is linear in the number of independent actors that ever incremented the counter), by managing identities towards avoiding global propagation, and garbage collecting temporary entries. The approach used in Handoff Counters is not restricted to counters, being more generally applicable to other data types with associative and commutative operations.
pdf papers eventual-consistency counters distributed-systems distcomp cap-theorem ecdc handoff-counters crdts data-structures g-counters
LMDB response to a LevelDB-comparison blog post
This seems like a good point to note about LMDB in general:
We state quite clearly that LMDB is read-optimized, not write-optimized. I wrote this for the OpenLDAP Project; LDAP workloads are traditionally 80-90% reads. Write performance was not the goal of this design, read performance is. We make no claims that LMDB is a silver bullet, good for every situation. It's not meant to be – but it is still far better at many things than all of the other DBs out there that *do* claim to be good for everything.
How to avoid crappy ISP caches when viewing YouTube video
Must give this a try when I get home -- I frequently have latency problems watching YT on my UPC connection, and I bet they have a crappily-managed, overloaded cache box on their network.
streaming youtube caching isps caches firewalls iptables hacks video networking
How to configure ntpd so it will not move time backwards
The "-x" switch will expand the step/slew boundary from 128ms to 600 seconds, ensuring the time is slewed (drifted slowly towards the correct time at a max of 5ms per second) rather than "stepped" (a sudden jump, potentially backwards). Since slewing has a max of 5ms per second, time can never "jump backwards", which is important to avoid some major application bugs (particularly in Java timers).
ntpd time ntp ops sysadmin slew stepping time-synchronization linux unix java bugs
-
'a Java port of Twitter's Snowflake thrift service presented as an HTTP-based Dropwizard service'.
an HTTP-based service for generating unique ID numbers at high scale with some simple guarantees. supports returning ID numbers as: JSON and JSONP; Google's Protocol Buffers; Plain text. At GE, we were more interested in the uncoordinated aspects of Snowflake than its throughput requirements, so HTTP was fine for our needs. We also exposed the core of Snowflake as an embeddable module so it can be directly integrated into our applications. We don't have the guarantees that the Snowflake-Zookeeper integration was providing, but that was also acceptable to us. In places where we really needed high throughput, we leveraged the snowizard-core embeddable module directly.
Odd OSS license, though -- BSDish? Containers and Docker: How Secure Are They?
pretty extensive article. (via Tony Finch)
via:fanf security containerization docker containers lxc linux ops
-
I loved doing Groklaw, and I believe we really made a significant contribution. But even that turns out to be less than we thought, or less than I hoped for, anyway. My hope was always to show you that there is beauty and safety in the rule of law, that civilization actually depends on it. How quaint. If you have to stay on the Internet, my research indicates that the short term safety from surveillance, to the degree that is even possible, is to use a service like Kolab for email, which is located in Switzerland, and hence is under different laws than the US, laws which attempt to afford more privacy to citizens. I have now gotten for myself an email there, p.jones at mykolab.com in case anyone wishes to contact me over something really important and feels squeamish about writing to an email address on a server in the US. But both emails still work. It's your choice. My personal decision is to get off of the Internet to the degree it's possible. I'm just an ordinary person. But I really know, after all my research and some serious thinking things through, that I can't stay online personally without losing my humanness, now that I know that ensuring privacy online is impossible. I find myself unable to write. I've always been a private person. That's why I never wanted to be a celebrity and why I fought hard to maintain both my privacy and yours. Oddly, if everyone did that, leap off the Internet, the world's economy would collapse, I suppose. I can't really hope for that. But for me, the Internet is over. So this is the last Groklaw article. I won't turn on comments. Thank you for all you've done. I will never forget you and our work together. I hope you'll remember me too. I'm sorry I can't overcome these feelings, but I yam what I yam, and I tried, but I can't.
nsa surveillance privacy groklaw law us-politics data-protection snooping mail kolab
Nelson's Weblog: tech / bad / failure-of-encryption
One of the great failures of the Internet era has been giving up on end-to-end encryption. PGP dates back to 1991, 22 years ago. It gave us the technical means to have truly secure email between two people. But it was very difficult to use. And in 22 years no one has ever meaningfully made email encryption really usable. [...] We do have SSL/HTTPS, the only real end-to-end encryption most of us use daily. But the key distribution is hopelessly centralized, authority rooted in 40+ certificates. At least 4 of those certs have been compromised by blackhat hackers in the past few years. How many more have been subverted by government agencies? I believe the SSL Observatory is the only way we'd know.
We do also have SSH. Maybe more services need to adopt that model?-
a new, and interesting, sketching algorithm, with a Java implementation:
Recordinality is unique in that it provides cardinality estimation like HLL, but also offers "distinct value sampling." This means that Recordinality can allow us to fetch a random sample of distinct elements in a stream, invariant to cardinality. Put more succinctly, given a stream of elements containing 1,000,000 occurrences of 'A' and one occurrence each of 'B' - 'Z', the probability of any letter appearing in our sample is equal. Moreover, we can also efficiently store the number of times elements in our distinct sample have been observed. This can help us to understand the distribution of occurrences of elements in our stream. With it, we can answer questions like "do the elements we've sampled present in a power law-like pattern, or is the distribution of occurrences relatively even across the set?"
sketching coding algorithms recordinality cardinality estimation hll hashing murmurhash java
-
A fantastic infographic explaining Australia's Preferential Voting system, featuring Dennis the Election Koala and Ken the Voting Dingo
-
The man was unmoved. And so one of the more bizarre moments in the Guardian's long history occurred – with two GCHQ security experts overseeing the destruction of hard drives in the Guardian's basement just to make sure there was nothing in the mangled bits of metal which could possibly be of any interest to passing Chinese agents. "We can call off the black helicopters," joked one as we swept up the remains of a MacBook Pro. Whitehall was satisfied, but it felt like a peculiarly pointless piece of symbolism that understood nothing about the digital age. We will continue to do patient, painstaking reporting on the Snowden documents, we just won't do it in London. The seizure of Miranda's laptop, phones, hard drives and camera will similarly have no effect on Greenwald's work. The state that is building such a formidable apparatus of surveillance will do its best to prevent journalists from reporting on it. Most journalists can see that. But I wonder how many have truly understood the absolute threat to journalism implicit in the idea of total surveillance, when or if it comes – and, increasingly, it looks like "when". We are not there yet, but it may not be long before it will be impossible for journalists to have confidential sources. Most reporting – indeed, most human life in 2013 – leaves too much of a digital fingerprint. Those colleagues who denigrate Snowden or say reporters should trust the state to know best (many of them in the UK, oddly, on the right) may one day have a cruel awakening. One day it will be their reporting, their cause, under attack. But at least reporters now know to stay away from Heathrow transit lounges.
nsa gchq surveillance spying snooping guardian reporters journalism uk david-miranda glenn-greenwald edward-snowden
-
'Sovereign is a set of Ansible playbooks that you can use to build and maintain' your own GMail/Google calendar/etc. on a VPS. Some up-to-date hosting tips, basically
New Tweets per second record, and how | Twitter Blog
How Twitter scaled up massively in 3 years -- replacing Ruby with the JVM, adopting SOA and custom sharding. Good summary post, looking forward to more techie details soon
Massive Overblocking Hits Hundreds Of UK Sites | Techdirt
Customers of UK ISPs Virgin Media and Be Broadband found they were unable to access hundreds of sites, including the Radio Times and Zooniverse, due to a secret website-blocking court order from the Premier League. PC Pro believe that 3 other ISPs' customers were also affected. According to customers reverse-engineering, it looks like the court order incorrectly demanded the blocking of "http-redirection-a.dnsmadeeasy.com", a HTTP redirector operated by the DNS operator DNSMadeEasy.
The fact that the court could issue an order which didn't see this coming and that the ISPs would act on it without checking that what they were doing was sensible is, in my opinion, extremely worrying.
overblocking censorship org uk sky be-broadband virgin-media dnsmadeeasy filtering premier-league false-positives isps
Beating the CAP Theorem Checklist
'Your ( ) tweet ( ) blog post ( ) marketing material ( ) online comment advocates a way to beat the CAP theorem. Your idea will not work. Here is why it won't work:' lovely stuff, via Bill De hOra
via:dehora funny cap cs distributed-systems distcomp networking partitions state checklists
'Sparrow: Scalable Scheduling for Sub-Second Parallel Jobs' [tech report]
scheduling sparrow load-balancing algorithms distributed-systems distcomp papers
From derelict to delightful: Art Tunnel Smithfield
I do like the Art Tunnel. Smithfield is a great demo of reclaiming Dublin's increasing dereliction and I hope the DCC allow this to continue
smithfield d7 dublin ireland art art-tunnel reclamation derelict economy dcc
How A 'Deviant' Philosopher Built Palantir, A CIA-Funded Data-Mining Juggernaut - Forbes
Palantir -- the free-market state-surveillance data-retention nightmare. At the end of this slightly overenthusiastic puff piece we get to:
Katz-Lacabe wasn't impressed. Palantir's software, he points out, has no default time limits -- all information remains searchable for as long as it's stored on the customer's servers. And its auditing function? "I don't think it means a damn thing," he says. "Logs aren't useful unless someone is looking at them." [...] What if Palantir's audit logs -- its central safeguard against abuse -- are simply ignored? Karp responds that the logs are intended to be read by a third party. In the case of government agencies, he suggests an oversight body that reviews all surveillance -- an institution that is purely theoretical at the moment. "Something like this will exist," Karp insists. "Societies will build it, precisely because the alternative is letting terrorism happen or losing all our liberties." Palantir's critics, unsurprisingly, aren't reassured by Karp's hypothetical court. Electronic Privacy Information Center activist Amie Stepanovich calls Palantir "naive" to expect the government to start policing its own use of technology. The Electronic Frontier Foundation's Lee Tien derides Karp's argument that privacy safeguards can be added to surveillance systems after the fact. "You should think about what to do with the toxic waste while you're building the nuclear power plant," he argues, "not some day in the future."
palantir data-retention privacy surveillance state cia forbes andy-greenberg eff epic snooping
London orders rubbish bins to stop collecting smartphone data
Good call.
AUTHORITIES IN LONDON'S financial district have ordered a company using high-tech rubbish bins to collect smartphone data from passers-by to cease its activities, and referred the firm to the privacy watchdog. The City of London Corporation, which manages the so-called "Square Mile" around St Paul's Cathedral, said such data collection "needs to stop" until there could be a public debate about it.
(via Daragh O'Brien)via:dobrien privacy phones wifi mac-address data-protection data-retention renew london bins snooping sniffing
The Irish State wishes to uninvent computers with new FOI Bill
Mark Coughlan noticed this:
The FOI body shall take reasonable steps to search for and extract the records to which the request relates, having due regard to the steps that would be considered reasonable if the records were held in paper format.
In other words, pretend that computerised database technology, extant since the 1960s, does not exist. Genius (via Simon McGarr)funny irish ireland foi open-data freedom computerisation punch-cards paper databases
Hamlet is Banned in the British Library
Pretty hilarious account of the usual, run-of-the-mill overblocking in the British Library from last weekend:
I asked [the information desk] if they saw the problem, perhaps just the symbolism, of Hamlet being banned in the British Library. They shrugged. The IT department said there was nothing to be done, as it was only the British Library's wifi service that was blocking Hamlet, and the British Library's wifi service, they seemed sure, had nothing to do with the British Library. They were merely ships that passed in the night. Children crying to each other from either bank of an uncrossable river.
censorship filters overblocking hamlet shakespeare literature funny sad british-library blocking
The algorithm for a perfectly balanced photo gallery – Summit Stories from Crispy Mountain
Nice application of a partitioning exhaustive search algorithm using dynamic programming (via Tom)
algorithms javascript python dynamic-programming partitioning images gallery
-
An amazing Soviet map of the US economy from 1979. Wonderful piece of cold war memorabilia
Randomly Failed! The State of Randomness in Current Java Implementations
This would appear to be the paper which sparked off the drama around BitCoin thefts from wallets generated on Android devices:
The SecureRandom PRNG is the primary source of randomness for Java and is used e.g., by cryptographic operations. This underlines its importance regarding security. Some of fallback solutions of the investigated implementations [are] revealed to be weak and predictable or capable of being in?uenced. Very alarming are the defects found in Apache Harmony, since it is partly used by Android.
More on the BitCoin drama: https://bitcointalk.org/index.php?topic=271486.40 , http://bitcoin.org/en/alert/2013-08-11-androidandroid java prng random security bugs apache-harmony apache crypto bitcoin papers
The Getty Museum offers a huge chunk of their collection for free use
We've launched the Open Content Program to share, freely and without restriction, as many of the Getty's digital resources as possible. The initial focus of the Open Content Program is to make available all images of public domain artworks in the Getty's collections. Today we've taken a first step toward this goal by making roughly 4,600 high-resolution images of the Museum's collection free to use, modify, and publish for any purpose. Why open content? Why now? The Getty was founded on the conviction that understanding art makes the world a better place, and sharing our digital resources is the natural extension of that belief. This move is also an educational imperative. Artists, students, teachers, writers, and countless others rely on artwork images to learn, tell stories, exchange ideas, and feed their own creativity. In its discussion of open content, the most recent Horizon Report, Museum Edition stated that "it is now the mark—and social responsibility—of world-class institutions to develop and share free cultural and educational resources." I agree wholeheartedly.
getty art via:tupp_ed open-content free images pictures paintings museums
The NSA Is Commandeering the Internet - Bruce Schneier
You, an executive in one of those companies, can fight. You'll probably lose, but you need to take the stand. And you might win. It's time we called the government's actions what it really is: commandeering. Commandeering is a practice we're used to in wartime, where commercial ships are taken for military use, or production lines are converted to military production. But now it's happening in peacetime. Vast swaths of the Internet are being commandeered to support this surveillance state. If this is happening to your company, do what you can to isolate the actions. Do you have employees with security clearances who can't tell you what they're doing? Cut off all automatic lines of communication with them, and make sure that only specific, required, authorized acts are being taken on behalf of government. Only then can you look your customers and the public in the face and say that you don't know what is going on -- that your company has been commandeered.
nsa america politics privacy data-protection data-retention law google microsoft security bruce-schneier
We are the Operations team at Etsy. Ask us anything! : IAmA
great AMA from Etsy ops staff (via Nelson)
Building a panopticon: The evolution of the NSA's XKeyscore
This is an amazing behind-the-scenes look at the architecture of XKeyscore, and how it evolved from an earlier large-scale packet interception system, Narus' Semantic Traffic Analyzer. XKeyscore is a federated, distributed system, with distributed packet-capture agents running on Linux, built with protocol-specific plugins, which write 3 days of raw packet data, and 30 days of intercept metadata, to local buffer stores. Central queries are then 'distributed across all of the XKeyscore tap sites, and any results are returned and aggregated'. Dunno about you, but this is pretty much how I would have built something like this, IMO....
panopticon xkeyscore nsa architecture scalability packet-capture narus sniffing snooping interception lawful-interception li tapping
Police may block recording with Apple patent
Creeptastic, Apple.
Apple has patented a piece of technology which would allow government and police to block transmission of information, including video and photographs, from any public gathering or venue they deem "sensitive", and "protected from externalities." In other words, these powers will have control over what can and cannot be documented on wireless devices during any public event. And while the company says the affected sites are to be mostly cinemas, theaters, concert grounds and similar locations, Apple Inc. also says "covert police or government operations may require complete 'blackout' conditions."
apple iphone via:devore creepy police photos recording remote-control phones blackout
Ivan Risti?: Defending against the BREACH attack
One interesting response to this HTTPS compression-based MITM attack:
The award for least-intrusive and entirely painless mitigation proposal goes to Paul Querna who, on the httpd-dev mailing list, proposed to use the HTTP chunked encoding to randomize response length. Chunked encoding is a HTTP feature that is typically used when the size of the response body is not known in advance; only the size of the next chunk is known. Because chunks carry some additional information, they affect the size of the response, but not the content. By forcing more chunks than necessary, for example, you can increase the length of the response. To the attacker, who can see only the size of the response body, but not anything else, the chunks are invisible. (Assuming they're not sent in individual TCP packets or TLS records, of course.) This mitigation technique is very easy to implement at the web server level, which makes it the least expensive option. There is only a question about its effectiveness. No one has done the maths yet, but most seem to agree that response length randomization slows down the attacker, but does not prevent the attack entirely. But, if the attack can be slowed down significantly, perhaps it will be as good as prevented.
mitm attacks hacking security compression http https protocols tls ssl tcp chunked-encoding apache
Totoro Isn't All Cute. For Some, He's the God of Death.
"Everyone, do not worry," read the Studio Ghibli statement. "There's absolutely no truth or configuration that Totoro is the God of Death or that Mei is dead in My Neighbor Totoro."
totoro studio-ghibli death morbid japan film movies urban-legends alternate plot
Hogan describes bin charge increases as 'opportunistic' - Environmental News | The Irish Times
LOL Greyhound.
Greyhound Recycling last month announced increases of 50 cents a month for customers on a flat monthly charge, 50 cents for each black bin collection for customers who pay by the lift and two cents a kilo for customers who pay by weight only. In a letter to customers, it described the levy as "tax imposed by the Government of Ireland on the people of Ireland". However, following a complaint to the [National Consumer Agency] that the by-weight increase was 76 per cent more than the [government landfill levy] increase, Greyhound reduced the charge to an additional one cent a kilo.
greyhound ireland dublin rubbish recycling consumer ripoffs tax
IrelandOffline broadband availability map
Marking the locations of broadband options in your area, along with VDSL cabinets, local exchanges, and wireless ISP coverage, and the landing sites of submarine cables (presumably from submarinecablemap.com data)
irelandoffline cables network internet ireland coverage wisps vdsl broadband
Filters 'not a silver bullet' that will stop perverts, warns Interpol chief - Independent.ie
Sunday Independent interview with Interpol assistant director Mick Moran:
Moran spoke out after child welfare organisations here called on the Government to follow the UK's example by placing anti-pornography filters on Irish home broadband connections. The Irish Society for the Prevention of Cruelty to Children argued that pornography was damaging to young children and should be removed from their line of sight. But Moran warned this would only lull parents into a false sense of security. "If we imagine the access people had to porn in the past – that access is now complete and total. They have access to the most horrific material out there. We now need to focus on parental responsibility about how kids are using the internet."
mick-moran cam interpol policing ispcc filtering parenting children broadband
-
Gil Tene raises an extremely good point about load testing, high-percentile response-time measurement, and behaviour when testing a system under load:
I've been harping for a while now about a common measurement technique problem I call "Coordinated Omission" for a while, which can often render percentile data useless. [...] I believe that this problem occurs extremely frequently in test results, but it's usually hard to deduce it's existence purely from the final data reported. But every once in a while, I see test results where the data provided is enough to demonstrate the huge percentile-misreporting effect of Coordinated Omission based purely on the summary report. I ran into just such a case in Attila's cool posting about log4j2's truly amazing performance, so I decided to avoid polluting his thread with an elongated discussion of how to compute 99.9%'ile data, and started this topic here. That thread should really be about how cool log4j2 is, and I'm certain that it really is cool, even after you correct the measurements. [...] Basically, I think that the 99.99% observation computation is wrong, and demonstrably (using the data in the graph data posted) exhibits the classic "coordinated omission" measurement problem I've been preaching about. This test is not alone in exhibiting this, and there is nothing to be ashamed of when you find yourself making this mistake. I only figured it out after doing it myself many many times, and then I noticed that everyone else seems to also be doing it but most of them haven't yet figured it out. In fact, I run into this issue so often in percentile reporting and load testing that I'm starting to wonder if coordinated omission is there in 99.9% of latency tests ;-)
measurement testing latency load-testing gil-tene coordinated-omission validity log4j percentiles
Xerox scanners/photocopiers randomly alter numbers in scanned documents · D. Kriesel
Pretty major Xerox fail: photocopied/scanned docs are found to have replaced the digit '6' with '8', due to a poor choice of compression techniques:
Several mails I got suggest that the xerox machines use JBIG2 for compression. This algorithm creates a dictionary of image patches it finds "similar". Those patches then get reused instead of the original image data, as long as the error generated by them is not "too high". Makes sense. This also would explain, why the error occurs when scanning letters or numbers in low resolution (still readable, though). In this case, the letter size is close to the patch size of JBIG2, and whole "similar" letters or even letter blocks get replaced by each other.
jbig2 compression xerox photocopying scanning documents fonts arial image-compression images
The 1940s origins of Whataboutery
The exchange is indicative of a rhetorical strategy known as 'whataboutism', which occurs when officials implicated in wrongdoing whip out a counter-example of a similar abuse from the accusing country, with the goal of undermining the legitimacy of the criticism itself. (In Latin, this rhetorical defense is called tu quoque, or "you, too.")
history language whataboutism whataboutery politics 1940s russia ussr
-
A highly-available key value store for shared configuration and service discovery. etcd is inspired by zookeeper and doozer, with a focus on: Simple: curl'able user facing API (HTTP+JSON); Secure: optional SSL client cert authentication; Fast: benchmarked 1000s of writes/s per instance; Reliable: Properly distributed using Raft; Etcd is written in go and uses the raft consensus algorithm to manage a highly availably replicated log.
One of the core components of CoreOS -- http://coreos.com/ .configuration distributed raft ha doozer zookeeper go replication consensus-algorithm etcd coreos
_In Search of an Understandable Consensus Algorithm_, Diego Ongaro and John Ousterhout, Stanford
Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundation for building practical systems. In order to enhance understandability, Raft separates the key elements of consensus, such as leader election and log replication, and it enforces a stronger degree of coherency to reduce the number of states that must be considered. Raft also includes a new mechanism for changing the cluster membership, which uses overlapping majorities to guarantee safety. Results from a user study demonstrate that Raft is easier for students to learn than Paxos.
distributed algorithms paxos raft consensus-algorithms distcomp leader-election replication clustering
Extract from 1973 HM Treasury document concerning post-nuclear-attack responses
'Extract from 1973 HM Treasury document concerning post-nuclear-attack monetary policy' includes this amazing snippet:
[Contingency] ...(d) a total nuclear attack employing high power missiles which would destroy all but a small percentage of the UK population and almost all physical assets or civilised life. [...] As for (d), the money policy would of course be absurdly unrealistic for the few surviving administrators and politicians as they struggled to organise food and shelter for the tiny bands of surviving able-bodied and the probably larger number of sick and dying. Most of the other departments contingency planning might also be irrelevant in such a situation. Within a fairly short time the survivors would evacuate the UK and try to find some sort of life in less-effected countries (southern Ireland?).
Hey, at least they were considering these scenarios. (via Charlie Stross)nuclear attack contingency government monetary policy uk ireland history 1960s via:cstross insane fallout
WhatClinic.com's zombie recruitment video. We want your brains...
BRAAAAAAINS
-
A very tasty-looking guac recipe, from h2g market veteran Lily Ramirez-Foran -- her family's traditional one. I like the addition of pomegranate seeds
guacamole avocados pomegranate recipes lily-ramirez-foran food h2g
RA Forum: Button Factory - August 14th Simonetti (Goblin) Horror Project
LIVE - for the first time ever in Ireland, Claudio Simonetti (Goblin) & band will perform the classics of horror movie scores by seminal Italian progressive rock band Goblin, Simonetti himself and possibly one or two curve-balls ! Horror rock maestro Claudio Simonetti will fulfill fans' dreams and nightmares as the band perform the notably eerie soundtracks from Suspriria, Tenebre, Dawn of the Dead, Creepers, Demons and more! This epic show will also feature an intense A/V screening element featuring the electric scenes from some of these revered classics of horror and giallo.
Python Infrastructure Status - SSL Verification Errors on PyPI
There appears to be a problem affecting a number of users where SSL verification errors will be shown saying "pypi.python.org" does not match "addvocate.com". As Best we can tell this appears to be related to the ISP. It seems to be affecting folks using O2 or O2 related companies. We've also reports of it affecting people using Free. Cause appears to be one of the IP addresses returned in the Geo DNS for Europe returning a certificate for addvocate.com. It's not clear at this time *why* that IP address is returning a certificate for addvocate.com.
Turned out to be a routing loop in the fast.ly London POP (via Mick Twomey)via:micktwomey o2 censorship filtering internet ssl tls pypi python geodns pki
"Toxic" behaviour in games is largely from "usually good" people
Only 5% of toxic behavior comes from toxic people; 77% of it comes from people who are usually good. That finding has all sorts of implications for how to stop toxic behavior in an online community. It's not enough to just ban the jerks; good people have bad days too. Instead you have to teach the whole community what the community standards are. And quickly identify people who are having a bad day, intervene before their toxicity infects too many other people.
Great post by Nelson.gaming toxic bad-behaviour trolls abuse online games league-of-legends
-
OpenDNS's simple DNS-based blocking of dodgy content. Will need to set this up on the home router now that the kids are surfing...
Mail from the (Velvet) Cybercrime Underground
Brian Krebs manages to thwart an attempted framing for possession of Silk Road heroin. bloody hell
silk-road drugs bitcoin ecommerce brian-krebs crime framed cybercrime russia scary law-enforcement
Clare dolphin attacks fourth swimmer in a month as Dusty protects her patch
Dusty the Dolphin has gone bad!
Locals say the three-metre long mammal has been responsible for injuring a number of people over the past two years, with several of those being hospitalised with significant injuries. She struck a 40-year-old woman in the abdomen earlier this month. In response, lifeguards now fly the red danger flag any time the dolphin enters the area. The Irish Whale and Dolphin Group has also erected warning posters at Doolin pier. IWDG coordinator Dr Simon Berrow said: "It is our policy to discourage people swimming with whales and dolphins in Ireland. "We've drafted a poster recommending people do not swim with Dusty, but if they must, then they should respect her as a wild dolphin and not grab, lunge or chase after her. If she shows aggressive behaviour or is boisterous they should leave the water."
dusty dolphins wildlife nature fanore county-clare ireland swimming doolin animals
Why YouTube buffers: The secret deals that make -- and break -- online video
Should ISPs be required to ensure they have sufficient upstream bandwidth to video sites like YouTube and Netflix?
"Verizon has chosen to sell its customers a product [Netflix] that they hope those customers don't actually use," Schaeffer said. "And when customers use it and request movies, they have not ensured there is adequate connectivity to get that video content back to their customers."
netflix youtube streaming video isps net-neutrality peering comcast bandwidth upstream
ISPAI Responds to Porn Filtering Debacle
Quite a strong statement:
The issue of access to age-inappropriate content is not a new matter and it is important not to have "knee-jerk" reactions which don't solve the perceived problem and have major implications for the public's right to access information in general. Notably the European Commission, as stated by vice-president Nellie Kroes [jm: sic], has come out strongly against blocking of the Internet, seeing it as an important platform for freedom of speech and she intends to "guarantee access without restriction." We in Ireland would do well to consider carefully the impact that any rash adoption or attempted copying of UK measures might have here in the light of current and future EU legislation and policy.
ispai filtering overblocking david-cameron porn internet ireland politics blocking web uk
-
Excellent weather site, displaying beautifully interpolated rainfall visualization, from the team behind the Dark Sky app
weather ireland dark-sky apps iphone ipad forecast rain dataviz mapping via:marcomorain
Applied Cryptography, Cryptography Engineering, and how they need to be updated
Whoa, I had no idea my knowledge of crypto was so out of date! For example:
ECC is going to replace RSA within the next 10 years. New systems probably shouldn't use RSA at all.
This blogpost is full of similar useful guidelines and rules of thumb. Here's hoping I don't need to work on a low-level cryptosystem any time soon, as the risk of screwing it up is always high, but if I do this is a good reference for how it needs to be done nowadays.thomas-ptacek crypto cryptography coding design security aes cbc ctr ecb hmac side-channels rsa ecc
When 'Smart Homes' Get Hacked: I Haunted A Complete Stranger's House Via The Internet - Forbes
Hardware designers do their usual trick -- omit the whole security part:
[Trustwave's Crowley] found security flaws that would allow a digital intruder to take control of a number of sensitive devices beyond the Insteon systems, from the Belkin WeMo Switch to the Satis Smart Toilet. Yes, they found that a toilet was hackable. You only have to have the Android app for the $5,000 toilet on your phone and be close enough to the toilet to communicate with it. "It connects through Bluetooth, with no username or password using the pin '0000'," said Crowley. "So anyone who has the application on their phone and was connected to the network could control anyone else's toilet. You could turn the bidet on while someone's in there."
home automation insteon security hardware fail attacks bluetooth han trustwave belkin satis
-
Missed bookmarking this news --
After years of debate and controversy the French Government has finally backtracked on the law which allowed errant subscribers to be disconnected from the Internet. This morning a decree was published which removed the possibility for file-sharers to have their connections cut for copyright infringement. Instead, those caught by rightsholders will now be subjected to a system of automated fines.
BBC News - Chinese firm Huawei controls net filter praised by PM
Talk Talk's porn-filtering, system praised by David Cameron in the UK as a model for porn filtering for the country's ISPs, is operated by Huawei. Of course, there's no possible problems with allowing Huawei, with its alleged close ties to the Chinese government, operate a state-wide internet censorship system in the UK without any functioning oversight, right? ;) Also worth noting: all TalkTalk traffic passes through the Huawei filtering infrastructure, even when the customer has "opted in".
huawei talk-talk oversight overblocking politics china uk david-cameron filtering censorship
Branded to death | Features | Times Higher Education
The most abominable monster now threatening the intellectual health and the integrity of pure enquiry as well as conscientious teaching is the language of advertising, or better, the machinery of propaganda. Any number of critics from within university walls have warned the people at large and academics in particular of the way the helots of advertising and the state police of propaganda bloat and distort the language of thoughtful description, peddle with a confident air generalisations without substance, and serenely circulate orotund lies while ignoring their juniors' rebuttals and abuse.
Relevant to this argument -- http://arstechnica.com/tech-policy/2013/07/the-webs-longest-nightmare-ends-eolas-patents-are-dead-on-appeal/ notes that 'the role of the University of California [was] one of the most perplexing twists in the Eolas saga. The university kept a low profile during the lead-up to trial; but once in Texas, Eolas' lawyers constantly reminded the jury they were asserting "these University of California patents." A lawyer from UC's patent-licensing division described support for Eolas at trial by simply saying that the university "stands by its licensees."'branding advertising newspeak universities third-level eolas higher-education education research university-of-california ucb patents ip swpats
Twilio Billing Incident Post-Mortem
At 1:35 AM PDT on July 18, a loss of network connectivity caused all billing redis-slaves to simultaneously disconnect from the master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time. Receiving full sync requests from each redis-slave caused the master to suffer extreme load, resulting in performance degradation of the master and timeouts from redis-slaves to redis-master. By 2:39 AM PDT the host's load became so extreme, services relying on redis-master began to fail. At 2:42 AM PDT, our monitoring system alerted our on-call engineering team of a failure in the Redis cluster. Observing extreme load on the host, the redis process on redis-master was misdiagnosed as requiring a restart to recover. This caused redis-master to read an incorrect configuration file, which in turn caused Redis to attempt to recover from a non-existent AOF file, instead of the binary snapshot. As a result of that failed recovery, redis-master dropped all balance data. In addition to forcing recovery from a non-existent AOF, an incorrect configuration also caused redis-master to boot as a slave of itself, putting it in read-only mode and preventing the billing system from updating account balances.
See also http://antirez.com/news/60 for antirez' response. Here's the takeaways I'm getting from it: 1. network partitions happen in production, and cause cascading failures. this is a great demo of that. 2. don't store critical data in Redis. this was the case for Twilio -- as far as I can tell they were using Redis as a front-line cache for billing data -- but it's worth saying anyway. ;) 3. Twilio were just using Redis as a cache, but a bug in their code meant that the writes to the backing SQL store were not being *read*, resulting in repeated billing and customer impact. In other words, it turned a (fragile) cache into the authoritative store. 4. they should probably have designed their code so that write failures would not result in repeated billing for customers -- that's a bad failure path. Good post-mortem anyway, and I'd say their customers are a good deal happier to see this published, even if it contains details of the mistakes they made along the way.redis caching storage networking network-partitions twilio postmortems ops billing replication
Tuning and benchmarking Java 7's Garbage Collectors: Default, CMS and G1
Rudiger Moller runs through a typical GC-tuning session, in exhaustive detail
-
[JVM] GC is a difficult, specialised area that can be very frustrating for busy developers or devops folks to deal with. The JVM has a number of Garbage Collectors and a bewildering array of switches that can alter the behaviour of each collector. Censum does all of the parsing, number crunching and statistical analysis for you, so you don't have to go and get that PhD in Computer Science in order to solve your GC performance problem. Censum gives you straight answers as opposed to a ton of raw data. can eat any GC log you care to throw at it. is easy to install and use.
Commercial software, UKP 495 per license.
The Web's longest nightmare ends: Eolas patents are dead on appeal | Ars Technica
Ding dong, the troll is dead! Ars Technica with a great description of the Eolas web patent fiasco, and the UC system's sorry role. I blame Bayh-Dole for creating this insane mindset where places of learning are forced to "monetize" their research.
Under Doyle's conception of his own invention, practically any modern website owed him royalties. Playing a video online or rotating an image on a shopping website were "interactive" features that infringed his patents. And unlike many "patent trolls" who simply settle for settlements just under the cost of litigation, Doyle's company had the chops, the lawyers, and the early filing date needed to extract tens of millions of dollars from the accused companies. [...] The role of the University of California is one of the most perplexing twists in the Eolas saga. The university kept a low profile during the lead-up to trial; but once in Texas, Eolas lawyers constantly reminded the jury they were asserting "these University of California patents." A lawyer from UC's patent-licensing division described support for Eolas at trial by simply saying that the university "stands by its licensees." (Eolas was technically an exclusive licensee of the UC-owned patent, which also gives it the right to sue.) At the same time, the University of California, and the Berkeley campus in particular, was a key institution in creating early web technology. While UC lawyers cooperated with the plaintiffs, two UC Berkeley-trained computer scientists were key witnesses in the effort to demolish the Eolas patents. Pei-Yuan Wei created the pioneering Viola browser, a key piece of prior art, while he was a student at UC-Berkeley in the early 1990s. Scott Silvey, another UC-Berkeley student at that time, testified about a program he made called VPlot, which allowed users to rotate an image of an airplane using Wei's browser. VPlot and Viola were demonstrated to Sun Microsystems in May 1993, months before Doyle claimed to have conceived of his invention.
patents swpats eolas web patent-trolls ucb universities research viola plugins berkeley
Irish Comms Minister Pat Rabbitte ignores calls for State role in blocking online porn
Good call.
Mr Rabbitte says that legal concerns attached to mandatory filters, as well as a fear of imposing censorship, have persuaded him against trying to force ISPs to impose mandatory pornography-blocking internet filters. "I remain to be convinced that blanket censorship or a default-on blocker is the correct or workable response," he said. "Even if it were possible to ensure that such measures were not easily circumvented or didn't inadvertently block perfectly acceptable content, the principled question of whether the State should be encouraging service providers to filter or block content to all users, regardless of whether there are children resident, would still arise."
pat-rabbitte internet filtering censorship blocking porn overblocking default-on isps ireland
-
Hosted IRC, 20 users for $50/month. Useful now that Google have fecked up Chat entirely
UK Internet censorship plan no less stupid than it was last year - Boing Boing
Cory Doctorow's long list of articles describing how the UK's censorware-for-all plan is going to fail. I like this bit:
When we argued our case to the vendor's representative, he was categorical: any nudity, anywhere on [Boing Boing], makes it into a "nudity site" for the purposes of blocking. The vendor went so far as to state that a single image of Michelangelo's David, on one page among hundreds of thousands on a site, would be sufficient grounds for a nudity classification. I suspect that none of the censorship advocates in the Lords understand that the offshore commercial operators they're proposing to put in charge of the nation's information access apply this kind of homeopathic standard to objectionable material.
I guess this means the Daily Mail will be similarly classified as containing "nudity" and blocked, given their smut column on every page?daily-mail fail censorship censorware boing-boing michelangelo sculpture nudity uk politics filtering overblocking web internet
-
Photoshop's "Content Aware Fill" applied to text. some very cool results
images cool art typography algorithms via:pentadact photoshop
A Tour Inside CloudFlare's Latest Generation Servers
great transparency from CloudFront! Looking at their current 4th-gen rackmount server buildout -- now with HP after Dell and ZT. Shitloads of SSDs for lower power and greater predictability in failure rates. 128GB RAM. consistent hashing to address stores instead of RAID. Sandybridge chipset. Solarflare SFC9020 10Gbps network cards. This is really impressive openness for a high-scale custom datacenter server platform...
3D-Printer Manufacturer Creates Software Filter To Prevent Firearm Printing
'[Create It REAL], which sells 3D printer component parts and software, recently announced that it has come up with a firearm component detection algorithm that will give 3D printers the option to block any gun parts. The software compares each component a user is trying to print with a database of potential firearms parts, and shuts down the modeling software if it senses the user is trying to make a gun.'
blocklists filtering guns weapons 3d-printing future firearms
Fund it :: Upstart Granby Park
help fund Granby Park, a pop-up park to replace a vacant site on the corner of Dominick St and Parnell St in Dublin 1: http://upstart.ie/
fund-it granby-park dublin d1 parks pop-up city funding grassroots
-
the details of Karsten Nohl's attack against SIM cards, allowing remote-root malware via SMS.
Cracking SIM update keys: [Over The Air] commands, such as software updates, are cryptographically-secured SMS messages, which are delivered directly to the SIM. While the option exists to use state-of-the-art AES or the somewhat outdated 3DES algorithm for OTA, many (if not most) SIM cards still rely on the 70s-era DES cipher. [...] To derive a DES OTA key, an attacker starts by sending a binary SMS to a target device. The SIM does not execute the improperly signed OTA command, but does in many cases respond to the attacker with an error code carrying a cryptographic signature, once again sent over binary SMS. A rainbow table resolves this plaintext-signature tuple to a 56-bit DES key within two minutes on a standard computer.
2 minutes. Sic transit gloria DES. The next step after that is to send a signed request to run a Java applet, then exploit a hole in the JVM sandbox, and the SIM card is rooted. Looking forward to the full paper on July 31st...des 3des crypto security sms sim-cards smartcards java applets ota rainbow-tables cracking karsten-nohl
-
Cool. A machine-learning-generated TCP congestion control algorithm which handily beats sfqCoDel, Vegas, Reno et al. But:
"Although the [computer-generated congestion control algorithms] appear to work well on networks whose parameters fall within or near the limits of what they were prepared for -- even beating in-network schemes at their own game and even when the design range spans an order of magnitude variation in network parameters -- we do not yet understand clearly why they work, other than the observation that they seem to optimize their intended objective well. We have attempted to make algorithms ourselves that surpass the generated RemyCCs, without success. That suggests to us that Remy may have accomplished something substantive. But digging through the dozens of rules in a RemyCC and ?guring out their purpose and function is a challenging job in reverse-engineering. RemyCCs designed for broader classes of networks will likely be even more complex, compounding the problem." So are network engineers willing to trust an algorithm that seems to work but has no explanation as to why it works other than optimizing a specific objective function? As AI becomes increasingly successful the question could also be asked in a wider context.
(via Bill de hOra)via-dehora machine-learning tcp networking hmm mit algorithms remycc congestion
Street Cuffs: L.A. Sees Big Jump In Bike Thefts
Some [LA] bike messengers last month took justice into their own hands when they caught two suspected thieves, teenage boys who attended a local Catholic high school. According to police, the messengers stripped down the teens to their boxer shorts before taking their cellphones, backpacks and clothes. "They meted out street justice. We don't condone street justice. They never threatened them. But they made it clear: don't mess with another person's property," Los Angeles Police Lt. Paul Vernon said. "This incident and the arrests are the tip of the iceberg when comes to people stealing bicycles." Vernon said the two boys told police they were robbed by about 20 men on bicycles at 6th Street and Grand Avenue about 3 p.m. on Jan. 12. Investigators said they cannot prove the boys were stealing bikes and continue to look for the assailants.
cycling theft robbery bike-theft la crime vigilantes cycle-couriers
ICO's Tame Investigation Of Google Street View Data Slurping
"People will yet again be asking whether Google has been let off without the kind of full and rigorous investigation that you would expect after this kind of incident," Nick Pickles, director of the Big Brother Watch, told TechWeekEurope. "Let's not forget that information was collected without permission from thousands of people's Wi-Fi networks, in a way that if an individual had done so they would have almost certainly have been prosecuted. It seems strange that ICO [the UK's Data Protection regulatory agency] did not want to inspect the [datacenter] cages housing the data, while it is also troubling that Google's assurances were taken at face value, despite this not being the first incident where consumers have seen their privacy violated by the company."
privacy google ico regulation data-protection snooping wifi sniffing network-traffic street-view
-
'My researches on the pickling matter had lead me to conclude that Mexico was, in fact, one of the few places where pickled potatoes were "a thing" and, in discussing same with Lily last month at her Mexican food stall in the Honest To Goodness market, I discovered that her soon-to-be-visiting Mexican mama was, in fact, a maker of such pickles. Not long afterward, I watched as Lily sat down with her mother, querying the ways of her pickled potatoes, translating and scribbling instructions for me as the details were recalled, not in an orderly series of steps, but in a series of asides and by-the-ways, by one for whom the practice of pickling potatoes was entirely second nature.'
Porn to be Blocked in the UK – "What's new?" Say Pirate Bay Users | TorrentFreak
It seems likely that the ISPs will implement a system similar to the one currently being used by TalkTalk, as the prime minister will specifically single the ISP out for praise in his speech. TalkTalk's HomeSafe is a system which filters out URLs based on a remote blocklist provided and maintained by…. well, no one quite knows. This is worrying since when things don't go quite to plan there's no one to complain to. As previously reported, when TalkTalk customers are asked whether they want to block file-sharing sites, TorrentFreak.com is rendered inaccessible. Despite our pleas and complaints that we are a news resource, the company said it would not remove us from their blocklist. We doubt we're the only ones being silenced.
talktalk blocking uk isps torrentfreak politics filtering david-cameron porn overblocking
-
Good description of how Fog Creek built out their Trello product; client-side JS rendering, model synced across the wire, HAProxy, Redis, and WebSockets. Bookmarked notably for this paragraph, which doesn't ameliorate my fear of WebSockets as a tech:
The Socket.io server currently has some problems with scaling up to more than 10K [jm: oh dear] simultaneous client connections when using multiple processes and the Redis store, and the client has some issues that can cause it to open multiple connections to the same server, or not know that its connection has been severed.
websockets javascript architecture fog-creek trello ajax push
Log4j 2: Performance close to insane
Nice writeup on Log4j 2's new AsyncAppender implementation, based on the LMAX Disruptor. sounds pretty excellent:
"One nice little detail I should mention is that both Async Loggers and Async Appenders fix something that has always bothered me in Log4j-1.x, which is that they will flush the buffer after logging the last event in the queue . With Log4j-1.x, if you used buffered I/O, you often could not see the last few log events, as they were still stuck in the memory buffer. Your only option was setting immediateFlush to true, which forces disk I/O on every single log event and has a performance impact. With Async Loggers and Appenders in Log4j-2.0 your log statements are all flushed to disk, so they are always visible, but this happens in a very efficient manner."
-
an ultra low latency, high throughput, persisted, messaging and event driven in memory database. The typical latency is as low as 80 nano-seconds and supports throughputs of 5-20 million messages/record updates per second. This library also supports distributed, durable, observable collections (Map, List, Set) The performance depends on the data structures used, but simple data structures can achieve throughputs of 5 million elements or key/value pairs in batches (eg addAll or putAll) and 500K elements or key/values per second when added/updated/removed individually. It uses almost no heap, trivial GC impact, can be much larger than your physical memory size (only limited by the size of your disk) and can be shared between processes with better than 1/10th latency of using Sockets over loopback. It can change the way you design your system because it allows you to have independent processes which can be running or not at the same time (as no messages are lost) This is useful for restarting services and testing your services from canned data. e.g. like sub-microsecond durable messaging. You can attach any number of readers, including tools to see the exact state of the data externally.
-
a completely new patent pending product designed in Ireland that is going to change the way people use their cars for carrying goods. It is a solid plastic product that grips the carpet in your car and acts as a barrier to hold loose items securely against the side wall in your car trunk or boot.
Found out about this online -- a US-based acquaintance raving about them being worth the shipping from Ireland. nice work!
-
'the Linux container engine'. I totally misunderstood what Docker was -- this is cool.
Heterogeneous payloads: Any combination of binaries, libraries, configuration files, scripts, virtualenvs, jars, gems, tarballs, you name it. No more juggling between domain-specific tools. Docker can deploy and run them all. Any server: Docker can run on any x64 machine with a modern linux kernel - whether it's a laptop, a bare metal server or a VM. This makes it perfect for multi-cloud deployments. Isolation: Docker isolates processes from each other and from the underlying host, using lightweight containers. Repeatability: Because each container is isolated in its own filesystem, they behave the same regardless of where, when, and alongside what they run.
lxc containers virtualization cloud ops linux docker deployment
Next Generation Continuous Integration & Deployment with dotCloud's Docker and Strider
Since Docker treats it's images as a tree of derivations from a source image, you have the ability to store an image at each stage of a build. This means we can provide full binary images of the environment in which the tests failed. This allows you to run locally bit-for-bit the same container as the CI server ran. Due to the magic of Docker and AUFS Copy-On-Write filesystems, we can store this cheaply. Often tests pass when built in a CI environment, but when built in another (e.g. production) environment break due to subtle differences. Docker makes it trivial to take exactly the binary environment in which the tests pass, and ship that to production to run it.
docker strider continuous-integration continuous-deployment deployment devops ops dotcloud lxc virtualisation copy-on-write images
Pinterest's follower graph store, built on Redis
This is a good, high-availability Redis configuration; sharded by userid across 8192 shards, with a Redis master/slave pair of instances for each set of N shards. I like their use of two redundancy systems -- hot slave and backup snapshots:
We run our cluster in a Redis master-slave configuration, and the slaves act as hot backups. Upon a master failure, we failover the slave as the new master and either bring up a new slave or reuse the old master as the new slave. We rely on ZooKeeper to make this as quick as possible. Each master Redis instance (and slave instance) is configured to write to AOF on Amazon EBS. This ensures that if the Redis instances terminate unexpectedly then the loss of data is limited to 1 second of updates. The slave Redis instances also perform BGsave hourly which is then loaded to a more permanent store (Amazon S3). This copy is also used by Map Reduce jobs for analytics. As a production system, we need many failure modes to guard ourselves. As mentioned, if the master host is down, we will manually failover to slave. If a single master Redis instance reboots, monit restart restores from AOF, implying a 1 second window of data loss on the shards on that instance. If the slave host goes down, we bring up a replacement. If a single slave Redis instance goes down, we rely on monit to restart using the AOF data. Because we may encounter AOF or BGsave file corruption, we BGSave and copy hourly backups to S3. Note that large file sizes can cause BGsave induced delays but in our cluster this is mitigated by smaller Redis data due to the sharding scheme.
graph redis architecture ha high-availability design redundancy sharding
-
'A simple time-decaying approximate membership filter' -- like a Bloom filter with time decay. See also http://eng.42go.com/flower-filter-an-update/ for some notes on the non-independence of survival probabilities, and how that imposes negligible differences in practice.
bloom-filter algorithms coding probabilistic approximate time decay
-
This is brilliant. 'covert bicycle GPS tracker; Notifies you by SMS if your bicycle moves; Online tracking'. 'Spybike is a covert tracking device that is hidden inside your bicycle steerer tube. The device is disguised to look like a normal head set cap to avoid suspicion. If someone steals your bike, you can use SpyBike to track their movements online and on your mobile.' More details: http://www.integratedtrackers.com/GPSTrack/pdf/Spybike_Instructions_2.pdf
No Time To Spare [infographic]
'On August 2, 2005, a fully-loaded Air France Airbus A340 arriving from Paris crash-landed at Toronto's Pearson International Airport and caught fire. Only 4 of the 8 exits were usable, yet all 309 people on board made it off the aircraft in two minutes, before it was consumed by flames. Here, five of the passengers recount their escape.'
infographics travel air accidents fire airbus safety escape a340
Merkel call for data protection rules puts Ireland in spotlight - Technology News
Irish Times on EU unhappiness with Ireland's "light touch" data protection regime:
Hawkes's appearance last month on RTɒs Morning Ireland regarding the US Prism surveillance programme, since posted to YouTube, reheated lingering resentment among many European data authorities. His admission that he "knew in a general way" about such programmes and didn't "regard this particular revelation as particularly new" was a red rag to his European colleagues who fear Ireland is the transmission point of wholesale EU data to the US.
eu ireland data-protection privacy billy-hawkes regulation dpc
Java Garbage Collection Distilled
a great summary of the state of JVM garbage collection from Martin Thompson
jvm java gc garbage-collection tuning memory performance martin-thompson
Improved HTTPS Performance with Early SSL Termination
This is a neat hack. Since SSL/TLS connection establishment requires lots of consecutive round trips before the connection is ready, by performing that closer to the user and reusing an existing region-to-region connection behind the scenes, the overall latency is greatly improved. Works for HTTP as well
http https ssl architecture aws ec2 performance latency internet round-trip nginx tls
-
Locking down a webapp with current strict HTTPS policies.
It's impossible to get to 100% security but there are steps you can take to secure your webapp for your users, to help mitigate against different types of attacks both against you, your webapp and your customers themselves. These are all things we've implemented with Server Density v2 to help harden the product as much as possible. These tips are in addition to security best practices such as protecting against SQL injection, filtering, session handling, and XSRF protection. Check out the OWASP cheat sheets and top 10 lists to ensure you're covered for the basics before implementing the suggestions below.
Breakthrough silicon scanning discovers backdoor in military chip [PDF]
Wow, I'd missed this:
This paper is a short summary of the ?rst real world detection of a backdoor in a military grade FPGA. Using an innovative patented technique we were able to detect and analyse in the ?rst documented case of its kind, a backdoor inserted into the Actel/Microsemi ProASIC3 chips for accessing FPGA con?guration. The backdoor was found amongst additional JTAG functionality and exists on the silicon itself, it was not present in any ?rmware loaded onto the chip. Using Pipeline Emission Analysis (PEA), our pioneered technique, we were able to extract the secret key to activate the backdoor, as well as other security keys such as the AES and the Passkey. This way an attacker can extract all the con?guration data from the chip, reprogram crypto and access keys, modify low-level silicon features, access unencrypted con?guration bitstream or permanently damage the device. Clearly this means the device is wide open to intellectual property (IP) theft, fraud, re-programming as well as reverse engineering of the design which allows the introduction of a new backdoor or Trojan. Most concerning, it is not possible to patch the backdoor in chips already deployed, meaning those using this family of chips have to accept the fact they can be easily compromised or will have to be physically replaced after a redesign of the silicon itself.
chips hardware backdoors security scanning pea jtag actel microsemi silicon fpga trojans
-
Privacy advocates have slammed Wyndham council for spying on residents' mobile phone data and email records almost 50 times in the past three years, "not to hunt down terrorists but to catch litterbugs and owners of unregistered pets". Figures from the attorney-general's department reveal Wyndham is the only Victorian council that has been snooping on personal data, seizing residents' information 31 times during 2010-11 and 2011-12. Council's acting chief executive Kelly Grigsby told the Weekly there had been another 18 authorisations in the past 12 months to chase people for unauthorised advertising, unregistered pets and illegal littering.
victoria australia oz privacy snooping data-retention metadata overreach
Traditional AQM is not enough!
Jim Gettys on modern web design, HTTP, buffering, and FIFO queues in the network.
Web surfing is putting impulses of packets, without congestion avoidance, into FIFO queues where they do severe collateral damage to anything sharing the link (including itself!). So today's web behavior incurs huge collateral damage on itself, data centers, the edge of the network, and in particular any application that hopes to have real time behavior. How do we solve this problem?
tl;dr: fq_codel. Now I want it!buffering networking internet web http protocols tcp bufferbloat jim-gettys codel fq_codel
We interrupt this program to warn the Emergency Alert System is hackable | Ars Technica
Private SSH key included in a firmware update. Oh dear:
The US Emergency Alert System, which interrupts live TV and radio broadcasts with information about national emergencies in progress, is vulnerable to attacks that allow hackers to remotely disseminate bogus reports and tamper with gear, security researchers warned. The remote takeover vulnerability affects the DASDEC-I and DASDEC-II application servers made by a company called Digital Alert Systems. It stems from the a recent firmware update that mistakenly included the private secure shell (SSH) key, according to an advisory published Monday by researchers from security firm IOActive. Administrators use such keys to remotely log in to a server to gain unfettered "root" access. The publication of the key makes it trivial for hackers to gain unauthorized access on Digital Alert System appliances that run default settings on older firmware. "An attacker who gains control of one or more DASDEC systems can disrupt these stations' ability to transmit and could disseminate false emergency information over a large geographic area," the IOActive advisory warned. "In addition, depending on the configuration of this and other devices, these messages could be forwarded and mirrored by other DASDEC systems."
-
Good read.
Twitter is primarily a consumption mechanism, not a production mechanism. 300K QPS are spent reading timelines and only 6000 requests per second are spent on writes.
* their approach of precomputing the timeline for the non-search case is a good example of optimizing for the more frequently-exercised path. * MySQL and Redis are the underlying stores. Redis is acting as a front-line in-RAM cache. they're pretty happy with it: https://news.ycombinator.com/item?id=6011254 * these further talks go into more detail, apparently (haven't watched them yet): http://www.infoq.com/presentations/Real-Time-Delivery-Twitter http://www.infoq.com/presentations/Twitter-Timeline-Scalability http://www.infoq.com/presentations/Timelines-Twitter * funny thread of comments on HN, from a big-iron fan: https://news.ycombinator.com/item?id=6008228scale architecture scalability twitter high-scalability redis mysql
Lightning Memory-Mapped Database
Sounds like a good potential replacement for Berkeley DB, at least for cases where LevelDB isn't proving practical.
LMDB is a database storage engine similar to LevelDB or BDB which database authors often use as a base for building databases on top of. LMDB was designed as a replacement for BDB within the OpenLDAP project but it has been pretty useful to use with other databases as well. It's API design is highly influenced by BDB so that replacing BDB is straightforward.
Licensed under the OpenLDAP Public License (is that BSDish?)openldap lmdb databases bdb berkeley-db storage persistence oss open-source
ssh - fabric appears to start apache2 but doesn't - Stack Overflow
fabric fail. pty=False fixes the bug
'Copysets: Reducing the Frequency of Data Loss in Cloud Storage' [paper]
An improved replica-selection algorithm for replicated storage systems.
We present Copyset Replication, a novel general purpose replication technique that signi?cantly reduces the frequency of data loss events. We implemented and evaluated Copyset Replication on two open source data center storage systems, HDFS and RAMCloud, and show it incurs a low overhead on all operations. Such systems require that each node's data be scattered across several nodes for parallel data recovery and access. Copyset Replication presents a near optimal tradeoff between the number of nodes on which the data is scattered and the probability of data loss. For example, in a 5000-node RAMCloud cluster under a power outage, Copyset Replication reduces the probability of data loss from 99.99% to 0.15%. For Facebook's HDFS cluster, it reduces the probability from 22.8% to 0.78%.
storage cloud-storage replication data reliability fault-tolerance copysets replicas data-loss
-
'principles, patterns, smells and guidelines for clean code, class and package design, TDD, Acceptance Test Driven Development, and CI'
clean-code code-smells coding tdd testing continous-integration patterns pdf
-
'Over time, the probability of someone drawing a cock with your [user-generated content] app approaches one.'
cocks time-to-penis user-generated-content content ugc via:rob-manuel qwghlm funny applegates-law web b3ta lol
-
Nice d3.js demo of the fat-tailed distribution:
A fat-tailed distribution looks normal but the parts far away from the average are thicker, meaning a higher chance of huge deviations. [...] Fat tails don't mean more variance; just different variance. For a given variance, a higher chance of extreme deviations implies a lower chance of medium ones.
dataviz via:hn statistics visualization distributions fat-tailed kurtosis d3.js javascript variance deviation
Google Cloud Messaging for Android
GCM is a service that allows you to send data from your server to your users' Android-powered device, and also to receive messages from devices on the same connection. The GCM service handles all aspects of queueing of messages and delivery to the target Android application running on the target device. GCM is completely free no matter how big your messaging needs are, and there are no quotas.
packetdrill - network stack testing tool
[Google's] packetdrill scripting tool enables quick, precise tests for entire TCP/UDP/IPv4/IPv6 network stacks, from the system call layer down to the NIC hardware. packetdrill currently works on Linux, FreeBSD, OpenBSD, and NetBSD. It can test network stack behavior over physical NICs on a LAN, or on a single machine using a tun virtual network device.
testing networking tun google linux papers tcp ip udp freebsd openbsd netbsd
the TCP bounded buffer deadlock problem
I've wound up mentioning this twice in the past week, so it's worth digging up and bookmarking!
Under certain circumstances a TCP connection can end up in a "deadlock", where neither the client nor the server is able to write data out or read data in. This is caused by two factors. First, a client or server cannot perform two transactions at once; a read cannot be performed if a write transaction is in progress, and vice versa. Second, the buffers that exist at either end of the TCP connection are of limited size. The deadlock occurs when both the client and server are trying to send an amount of data that is larger than the combined input and output buffer size.
tcp ip bounded-buffer deadlock bugs buffering connections distributed-systems
An excellent writeup of the TCP bounded-buffer deadlock problem
on pages 146-149 of 'TCP/IP Sockets in C: Practical Guide for Programmers' by Michael J. Donahoo and Kenneth L. Calvert.
tcp ip bounded-buffer deadlock bugs buffering connections distributed-systems
How The Copyright Industry Pushed For Internet Surveillance | TorrentFreak
Rick Falkvinge with a good point:
The reason for the copyright industry to push for surveillance is simple: any digital communications channel can be used for private conversation, but it can also be used to share culture and knowledge that is under copyright monopoly. In order to tell which communications is which, you must sort all of it – and to do that, you must look at all of it. In other words, if enforcing the copyright monopoly is your priority, you need to kill privacy, and specifically anonymity and secrecy of correspondence.
This was exactly my biggest worry -- a side-effect of effective copyright filtering is the creation of infrastructure for online oppression by the state.copyright privacy state data-protection rick-falkvinge copyfight internet filtering surveillance anonymity
Aer Lingus set to resume flights to San Francisco from Dublin
Yay!
Google, Apple and Facebook have persuaded Aer Lingus to reopen the San Francisco to Dublin route, according to sources in the US. The technology giants have their European headquarters in Dublin and their American bases in San Francisco. According to insiders, Aer Lingus will make an announcement soon having received assurances that Silicon Valley companies will take up seats.
Comics For Children…. a visual list…. | The Forbidden Planet International Blog
some great recommendations here. Hildafolk has been popular with my 5-year-old, must pick up a few more
_Measuring Mobile Web Performance_ [slides]
Notable slide is #13, displaying a graph of HSDPA packet RTTs measured from a train. Max RTT gets up to 20,266ms. ouch
rtt packets latency hsdpa mobile internet trains packet-loss
Latest leak of EU Data Protection Regulation makes fines impossible
Well, isn't this convenient. The leaked proposed regulation document from the Irish EU presidency contains the following changes from current law:
what is new is a set of prescriptive conditions which, if adopted, appears to make a Monetary Penalty Notice (MPN) almost impracticable to serve. This is because the [Data Protection] Commissioner would have consider a dozen factors (many of which will give no doubt rise to appeal). [...] In addition, the fines in the Regulation require consideration of the actual damage caused; this compares unfavourably with the current MPN where large fines have been contingent on grave security errors on the part of the data controller (i.e. the MPN of the UK DPA does not need damage to data subjects – only the likelihood of substantial distress or damage which should have been preventable/foreseeable).
data-protection law eu ec ireland privacy fines regulation mpn
Google Translate of "Lorem ipsum"
The perils of unsupervised machine learning... here's what GTranslate reckons "lorem ipsum" translates to:
We will be sure to post a comment. Add tomato sauce, no tank or a traditional or online. Until outdoor environment, and not just any competition, reduce overall pain. Cisco Security, they set up in the throat develop the market beds of Cura; Employment silently churn-class by our union, very beginner himenaeos. Monday gate information. How long before any meaningful development. Until mandatory functional requirements to developers. But across the country in the spotlight in the notebook. The show was shot. Funny lion always feasible, innovative policies hatred assured. Information that is no corporate Japan
lorem-ipsum boilerplate machine-learning translation google translate probabilistic tomato-sauce cisco funny
-
Write heavy, high performance applications should probably use RAID 0 or avoid RAID altogether and consider using a larger n_val and cluster size. Read heavy applications have more options, and generally demand more fault tolerance with the added benefit of easier hardware replacement procedures.
Good to see official guidance on this (via Bill de hOra) -
Facebook's new erasure coding algorithm (via High Scalability).
Disk I/O and network traffic were reduced by half compared to RS codes. The LRC required 14% more storage than RS (ie. 60% of data size). Repair times were much lower thanks to the local repair codes. Much greater reliability thanks to fast repairs. Reduced network traffic makes them suitable for geographic distribution.
erasure-coding facebook redundancy repair algorithms papers via:highscalability data storage fault-tolerance
Boundary's Early Warnings alarm
Anomaly detection on network throughput metrics, alarming if throughputs on selected flows deviate by 1, 2, or 3 standard deviations from a historical baseline.
network-monitoring throughput boundary service-metrics alarming ops statistics
My email to Irish Times Editor, sent 25th June
Daragh O'Brien noting 3 stories on 3 consecutive days voicing dangerously skewed misinformation about data protection and privacy law in Ireland:
There is a worrying pattern in these stories. The first two decry the Data Protection legislation (current and future) as being dangerous to children and damaging to the genealogy trade. The third sets up an industry "self-regulation" straw man and heralds it as progress (when it is decidedly not, serving only to further confuse consumers about their rights). If I was a cynical person I would find it hard not to draw the conclusion that the Irish Times, the "paper of record" has been stooged by organisations who are resistant to the defence of and validation of fundamental rights to privacy as enshrined in the Data Protection Acts and EU Treaties, and in the embryonic Data Protection Regulation. That these stories emerge hot on the heels of the pendulum swing towards privacy concerns that the NSA/Prism revelations have triggered is, I must assume, a co-incidence. It cannot be the case that the Irish Times blindly publishes press releases without conducting cursory fact checking on the stories contained therein? Three stories over three days is insufficient data to plot a definitive trend, but the emphasis is disconcerting. Is it the Irish Times' editorial position that Data Protection legislation and the protection of fundamental rights is a bad thing and that industry self-regulation that operates in ignorance of legislation is the appropriate model for the future? It surely cannot be that press releases are regurgitated as balanced fact and news by the Irish Times without fact checking and verification? If I was to predict a "Data Protection killed my Puppy" type headline for tomorrow's edition or another later this week would I be proved correct?
daragh-obrien irish-times iab bias advertising newspapers press-releases journalism data-protection privacy ireland
_Bolt-On Causal Consistency_ [slides]
SIGMOD 2013 presentation from Peter Bailis, Ali Ghodsi, Joseph M. Hellerstein, Ion Stoica -- adding consistency to an eventually-consistent store by tracking dependencies
-
Over the last couple of years, we have built and deployed a reliable publish-subscribe system called Wormhole. Wormhole has become a critical part of Facebook's software infrastructure. At a high level, Wormhole propagates changes issued in one system to all systems that need to reflect those changes – within and across data centers.
Facebook's Kafka-alike, basically, although with some additional low-latency guarantees. FB appear to be using it for multi-region and multi-AZ replication. Proprietary.pub-sub scalability facebook realtime low-latency multi-region replication multi-az wormhole
-
Turns out gnuplot has a pretty readable ASCII terminal rendering mode; combined with 'watch' it makes for a nifty graphing one-liner
gnuplot plotting charts graphs cli command-line unix gnu hacks dataviz visualization ascii
(oh look, a proper blog post!)
JMX is the de-facto standard in the Java and JVM-based world for exposing service metrics, and feeds nicely to tools like Graphite using JMXTrans and others. However, it's pretty obtuse and over-complex, and it can be hard to figure out what path the JMX metrics will show up under once deployed.
Unfortunately, once a JVM-based service is deployed to EC2, it becomes very difficult to use jconsole to connect to it, due to deficiencies and crappy design in the JMX RMI protocol (I love the way they reinvented the broken parts of IIOP in that respect). Don't even bother; instead, use jmxsh: https://code.google.com/p/jmxsh/ .
To use this, you need to modify the service process' command line to include the following JVM args, so that the remote JMX API is exposed:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16660 -Dcom.sun.management.jmxremote.local.only=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Change the port number if there is already a process running on that port. Ensure the port isn't accessible from off-host; in EC2, this should be safe enough to use once that port number is not in the EC2 security group.
Go to https://code.google.com/p/jmxsh/downloads/list and download the latest jmxsh-FOO.jar; e.g. 'wget https://jmxsh.googlecode.com/files/jmxsh-R5.jar'. Then on the host, as the UID the service is running under, run: 'java -jar jmxsh-R5.jar -h 127.0.0.1 -p 16660'. You can then hit "Enter" to go into "Browse Mode", and you'll get text menus like this:
====================================================
Attribute List:
1. -r- long MaxFileDescriptorCount
2. -r- long OpenFileDescriptorCount
3. -r- long CommittedVirtualMemorySize
4. -r- long FreePhysicalMemorySize
5. -r- long FreeSwapSpaceSize
6. -r- long ProcessCpuTime
7. -r- long TotalPhysicalMemorySize
8. -r- long TotalSwapSpaceSize
9. -r- String Name
10. -r- int AvailableProcessors
11. -r- String Arch
12. -r- double SystemLoadAverage
13. -r- String Version
SERVER: service:jmx:rmi:///jndi/rmi://127.0.0.1:16660/jmxrmi
DOMAIN: java.lang
MBEAN: java.lang:type=OperatingSystem
====================================================
Navigate through the MBean tree looking for good Attributes which would make good metrics (5 in the list above, for example). Note the MBean and the Attribute names.
Liberty issues claim against British Intelligence Services over PRISM and Tempora privacy scandal
James Welch, Legal Director for Liberty, said: "Those demanding the Snoopers' Charter seem to have been indulging in out-of-control snooping even without it – exploiting legal loopholes and help from Uncle Sam. "No-one suggests a completely unpoliced internet but those in power cannot swap targeted investigations for endless monitoring of the entire globe."
Go Liberty! Take note, ICCL, this is how a civil liberties group engages with internet issues.prism nsa gchq surveillance liberty civil-liberties internet snooping
-
A command-line utility in Ruby to perform (a) OLAP cubing and (b) histogramming, given whitespace-delimited line data
'If I was your cloud provider, I'd never let you down'
This is the thing that's put me off Joyent. They make claims like this one from October 2012:
We've given our other partners 99.9999% uptime.
This despite a 10-day outage of their BingoDisk and Strongspace storage services in January 2008, 1734 days previously (http://www.datacenterknowledge.com/archives/2008/01/21/joyent-services-back-after-8-day-outage/). If you assume that is the only outage they've had since then, that works out as 99.4% uptime. Quite a few less nines...-
Good UI for exploration of HyperLogLog set intersections and unions.
One of the first things that we wanted to do with HyperLogLog when we first started playing with it was to support and expose it natively in the browser. The thought of allowing users to directly interact with these structures -- perform arbitrary unions and intersections on effectively unbounded sets all on the client -- was exhilarating to us. [...] we are pleased to announce the open-source release of AK's HyperLogLog implementation for JavaScript, js-hll. We are releasing this code under the Apache License, Version 2.0. We knew that we couldn't just release a bunch of JavaScript code without allowing you to see it in action — that would be a crime. We passed a few ideas around and the one that kept bubbling to the top was a way to kill two birds with one stone. We wanted something that would showcase what you can do with HLL in the browser and give us a tool for explaining HLLs. It is typical for us to explain how HLL intersections work using a Venn diagram. You draw some overlapping circles with a border that represents the error and you talk about how if that border is close to or larger than the intersection then you can't say much about the size of that intersection. This works just ok on a whiteboard but what you really want is to just build a visualization that allows you to select from some sets and see the overlap. Maybe even play with the precision a little bit to see how that changes the result. Well, we did just that!
javascript ui hll hyperloglog algorithms sketching js sets intersection union apache open-source
Sketch of the Day: K-Minimum Values
Another sketching algorithm -- this one supports set union and intersection operations more easily than HyperLogLog when there are more than 2 sets
algorithms coding space-saving cardinality streams stream-processing estimation sets sketching
Skype's principal architect explains why they no longer have end-to-end crypto
Mobile devices can't handle the CPU and constantly-online requirements, and an increased reliance on dedicated routing supernodes to avoid Windows-client monoculture and p2p network fragility (via the IP list, via kragen)
skype p2p mobile architecture networking internet snooping crypto via:ip via:kragen phones windows
Accuweather long-range forecast accuracy questionable
"questionable" is putting it mildly:
Now to to the point: Are the 25-day forecasts any good? In a word, no. Specifically, after running this data, I would not trust a forecast high temperature more than a week out. I'd rather look at the normal (historical average) temperature for that day than the forecast. Similarly, I would not even look at a precipitation forecast more than 6 days in advance, and I wouldn't start to trust it for anything important until about 3 days ahead of time.
accuweather accuracy fail graphs data weather forecasting philadelphia
Setting up Perfect Forward Secrecy for nginx or stud
Matt Sergeant writes up a pretty solid HOWTO:
There has been a lot of discussion recently about Perfect Forward Secrecy (PFS) and the benefits it can bring you, especially in terms of any kind of traffic sniffing attack. Unfortunately setting this up I found very few guides telling you exactly what you need to do. The downside to PFS [via ECDHE] is that it uses more CPU power than other ciphers. This is a trade-off between security and cost.
Java Concurrent Counters By Numbers
threadsafe counters in the JVM compared. AtomicLong, Doug Lea's LongAdder, a ThreadLocal counter, and a field-on-the-Thread-object counter int (via Darach Ennis). Nitsan's posts on concurrency are fantastic
-
Tic-Tac-Toe Inception. whoa
-
a high-performance C server which is used to expose HyperLogLog sets and operations over them to networked clients. It uses a simple ASCII protocol which is human readable, and similar to memcached. HyperLogLog's are a relatively new sketching data structure. They are used to estimate cardinality, i.e. the unique number of items in a set. They are based on the observation that any bit in a "good" hash function is indepedenent of any other bit and that the probability of getting a string of N bits all set to the same value is 1/(2^N). There is a lot more in the math, but that is the basic intuition. What is even more incredible is that the storage required to do the counting is log(log(N)). So with a 6 bit register, we can count well into the trillions. For more information, its best to read the papers referenced at the end. TL;DR: HyperLogLogs enable you to have a set with about 1.6% variance, using 3280 bytes, and estimate sizes in the trillions.
(via:cscotta)hyper-log-log hlld hll data-structures memcached daemons sketching estimation big-data cardinality algorithms via:cscotta
-
'The TLS handshake has multiple variations, but let's pick the most common one – anonymous client and authenticated server (the connections browsers use most of the time).' Works out to 4 packets, in addition to the TCP handshake's 3, and about 6.5k bytes on average.
network tls ssl performance latency speed networking internet security packets tcp handshake
McLibel leaflet was co-written by undercover police officer Bob Lambert | UK news | guardian.co.uk
The true identity of one of the authors of the "McLibel leaflet" is Bob Lambert, a police officer who used the alias Bob Robinson in his five years infiltrating the London Greenpeace group. [...] McDonald's famously sued green campaigners over the roughly typed leaflet, in a landmark three-year high court case, that was widely believed to have been a public relations disaster for the corporation. Ultimately the company won a libel battle in which it spent millions on lawyers. Lambert was deployed by the special demonstration squad (SDS) – a top-secret Metropolitan police unit that targeted political activists between 1968 until 2008, when it was disbanded. He co-wrote the defamatory six-page leaflet in 1986 – and his role in its production has been the subject of an internal Scotland Yard investigation for several months. At no stage during the civil legal proceedings brought by McDonald's in the 1990s was it disclosed that a police infiltrator helped author the leaflet.
infiltration police mcdonalds libel greenpeace bob-lambert undercover 1980s uk-politics
Project Voldemort: measuring BDB space consumption
HOWTO measure this using the BDB-JE command line tools. this is exposed through JMX as the CleanerBacklog metric, too, I think, but good to bookmark just in case
rendering pcm with simulated phosphor persistence
This is something readily applicable to display of sampled time-series metric data -- it really makes regular patterns visible (and is nicely retro to boot).
When PCM waveforms and similar function plots are displayed on screen, computational speed is often preferred over beauty and information content. For example, Audacity only draws the local maximum envelope amplitude and (what appears to be) RMS power when zoomed out, and when zoomed in, displays a very straightforward linear interpolation between samples. Analogue oscilloscopes, on the other hand, do things differently. An electron beam scans a phosphor screen at a constant X velocity, lighting a dot everywhere it hits. The dot brightness is proportional to the time the electron beam was directed at it. Because the X speed of the beam is constant and the Y position is modulated by the waveform, brightness gives information about the local derivative of the function. Now how cool is that? It looks like an X-ray of the signal. We can see right away that the beep is roughly a square wave, because there's light on top and bottom of the oscillation envelope but mostly darkness in between. Minute changes in the harmonic content are also visible as interesting banding and ribbons.
(via an _amazing_ kragen post on ghetto electronics)via:kragen pcm waveforms oscilloscopes analog analogue dataviz time-series waves ui phosphor retro
stuff Google has learned from their hiring data
A. On the hiring side, we found that [interview] brainteasers are a complete waste of time. How many golf balls can you fit into an airplane? How many gas stations in Manhattan? A complete waste of time. They don't predict anything. They serve primarily to make the interviewer feel smart. Instead, what works well are structured behavioral interviews, where you have a consistent rubric for how you assess people, rather than having each interviewer just make stuff up. Behavioral interviewing also works — where you're not giving someone a hypothetical, but you're starting with a question like, "Give me an example of a time when you solved an analytically difficult problem." The interesting thing about the behavioral interview is that when you ask somebody to speak to their own experience, and you drill into that, you get two kinds of information. One is you get to see how they actually interacted in a real-world situation, and the valuable "meta" information you get about the candidate is a sense of what they consider to be difficult.
This makes sense, and matches what I learned in Amazon. Bad news for Microsoft though! (Correction: Adam Shostack got in touch to note that MS haven't done this for 10+ years either.)Also, I like this:
A. One of the things we've seen from all our data crunching is that G.P.A.'s are worthless as a criteria for hiring, and test scores are worthless — no correlation at all except for brand-new college grads, where there's a slight correlation. Google famously used to ask everyone for a transcript and G.P.A.'s and test scores, but we don't anymore, unless you're just a few years out of school. We found that they don't predict anything. What's interesting is the proportion of people without any college education at Google has increased over time as well. So we have teams where you have 14 percent of the team made up of people who've never gone to college.
google hiring interviewing interviews brainteasers gpa microsoft star amazon
Java Garbage Collection Distilled
Martin Thompson lays it out:
Serial, Parallel, Concurrent, CMS, G1, Young Gen, New Gen, Old Gen, Perm Gen, Eden, Tenured, Survivor Spaces, Safepoints, and the hundreds of JVM start-up flags. Does this all baffle you when trying to tune the garbage collector while trying to get the required throughput and latency from your Java application? If it does then don't worry, you are not alone. Documentation describing garbage collection feels like man pages for an aircraft. Every knob and dial is detailed and explained but nowhere can you find a guide on how to fly. This article will attempt to explain the tradeoffs when choosing and tuning garbage collection algorithms for a particular workload.
-
Appalled by mass surveillance scandals? So are we. We're doing something about it – and you can too. In 2006 we started a case challenging Irish and European laws that require your mobile phone company and ISP to monitor your location, your calls, your texts and your emails and to store that information for up to two years. That case has now made it to the European Court of Justice and will be heard on July 9th. If we are successful, it will strike down these laws for all of Europe and will declare illegal this type of mass surveillance of the entire population. Here's where you come in. You can take part by: making a donation to help us pay for the expenses we incur; following our updates and keeping abreast of the issues; spreading the word on social media. With your help, we can strike a blow for the privacy of all citizens.
activism privacy politics ireland dri digital-rights data-protection data-retention
3-D Printer Brings Dexterity To Children With No Fingers
'A South African man who lost part of his hand in a home carpentry accident and an American puppeteer he met via YouTube have teamed up to make 3D-printable hands for children who have no fingers. So far, over 100 children have been given "robohands" for free, and a simplified version released just yesterday snaps together like LEGO bricks and costs just $5 in materials.' This is incredible. Check out the video of Liam and his robohand in action: http://www.youtube.com/watch?v=kB53-D_N8Uc
3d-printing 3d makers robohands hands prosthetics future youtube via:gruverja
Open Rights Group - EU Commission caved to US demands to drop anti-PRISM privacy clause
Reports this week revealed that the US successfully pressed the European Commission to drop sections of the Data Protection Regulation that would, as the Financial Times explains, "have nullified any US request for technology and telecoms companies to hand over data on EU citizens. The article [...] would have prohibited transfers of personal information to a third country under a legal request, for example the one used by the NSA for their PRISM programme, unless "expressly authorized by an international agreement or provided for by mutual legal assistance treaties or approved by a supervisory authority." The Article was deleted from the draft Regulation proper, which was published shortly afterwards in January 2012. The reports suggest this was due to intense pressure from the US. Commission Vice-President Viviane Reding favoured keeping the the clause, but other Commissioners seemingly did not grasp the significance of the article.
org privacy us surveillance fisaaa viviane-reding prism nsa ec eu data-protection
Verified by Visa and MasterCard SecureCode kill 10-12% of your business
As Chris Shiflett noted: not only are they bad for security, they're bad for business too.
12 percent of users consider abandoning [an online shopping transaction] when they see either the Verified by Visa or the American Express SafeKey logos, while 10 percent will consider abandoning when the see the MasterCard Secure card logo.
ecommerce vbv online-shopping mastercard visa securecode security fail
The Cold Hard Facts of Freezing to Death
an amazing account of near-death from hypothermia (via Dor)
Atelier olschinsky - "Cities III 05"
Fine Art Print on Hahnemuehle Photo Rag Bright White, 310g: 40x50cm up to 70x100cm. Some great art based on decayed urban landscape shots, from a Vienna-based design studio. See also http://english.mashkulture.net/2011/10/17/atelier-olschinsky-cities-iii/ , http://www.mascontext.com/tag/atelier-olschinsky/
Possible ban on 'factory food' in French restaurants
I am very much in favour of this in Ireland, too. The pre-prepared food thing makes for crappy food:
In an attempt to crack down on the proliferation of restaurants serving boil-in-a-bag or microwave-ready meals, which could harm France's reputation for good food, MP Daniel Fasquelle is putting a new law to parliament this month. [...] The proposed law would limit the right to use the word "restaurant" to eateries where food is prepared on site using raw ingredients, either fresh or frozen. Exceptions would be made for some prepared products, such as bread, charcuterie and ice cream.
restaurants food france cuisine boil-in-the-bag microwave cooking daniel-fasquelle
-
great, comprehensive review of the language, its pros and misfeatures, from Bill de hOra
Introducing Kale « Code as Craft
Etsy have implemented a tool to perform auto-correlation of service metrics, and detection of deviation from historic norms:
at Etsy, we really love to make graphs. We graph everything! Anywhere we can slap a StatsD call, we do. As a result, we've found ourselves with over a quarter million distinct metrics. That's far too many graphs for a team of 150 engineers to watch all day long! And even if you group metrics into dashboards, that's still an awful lot of dashboards if you want complete coverage. Of course, if a graph isn't being watched, it might misbehave and no one would know about it. And even if someone caught it, lots of other graphs might be misbehaving in similar ways, and chances are low that folks would make the connection. We'd like to introduce you to the Kale stack, which is our attempt to fix both of these problems. It consists of two parts: Skyline and Oculus. We first use Skyline to detect anomalous metrics. Then, we search for that metric in Oculus, to see if any other metrics look similar. At that point, we can make an informed diagnosis and hopefully fix the problem.
It'll be interesting to see if they can get this working well. I've found it can be tricky to get working with low false positives, without massive volume to "smooth out" spikes caused by normal activity. Amazon had one particularly successful version driving severity-1 order drop alarms, but it used massive event volumes and still had periodic false positives. Skyline looks like it will alarm on a single anomalous data point, and in the comments Abe notes "our algorithms err on the side of noise and so alerting would be very noisy."etsy monitoring service-metrics alarming deviation correlation data search graphs oculus skyline kale false-positives
Paper: "Root Cause Detection in a Service-Oriented Architecture" [pdf]
LinkedIn have implemented an automated root-cause detection system:
This paper introduces MonitorRank, an algorithm that can reduce the time, domain knowledge, and human effort required to ?nd the root causes of anomalies in such service-oriented architectures. In the event of an anomaly, MonitorRank provides a ranked order list of possible root causes for monitoring teams to investigate. MonitorRank uses the historical and current time-series metrics of each sensor as its input, along with the call graph generated between sensors to build an unsupervised model for ranking. Experiments on real production outage data from LinkedIn, one of the largest online social networks, shows a 26% to 51% improvement in mean average precision in ?nding root causes compared to baseline and current state-of-the-art methods.
This is a topic close to my heart after working on something similar for 3 years in Amazon! Looks interesting, although (a) I would have liked to see more case studies and examples of "real world" outages it helped with; and (b) it's very much a machine-learning paper rather than a systems one, and there is no discussion of fault tolerance in the design of the detection system, which would leave me worried that in the case of a large-scale outage event, the system itself will disappear when its help is most vital. (This was a major design influence on our team's work.) Overall, particularly given those 2 issues, I suspect it's not in production yet. Ours certainly was ;)linkedin soa root-cause alarming correlation service-metrics machine-learning graphs monitoring
Announcing Zuul: Edge Service in the Cloud
Netflix' library to implement "edge services" -- ie. a front end to their API, web servers, and streaming servers. Some interesting features: dynamic filtering using Groovy scripts; Hystrix for software load balancing, fault tolerance, and error handling for originated HTTP requests; fine-grained service metrics; Archaius for configuration; and canary requests to detect overload risks. Pretty complex though
edge-services api netflix zuul archaius canary-requests http groovy hystrix load-balancing fault-tolerance error-handling configuration
CloudFlare, PRISM, and Securing SSL Ciphers
Matthew Prince of CloudFlare has an interesting theory on the NSA's capabilities:
It is not inconceivable that the NSA has data centers full of specialized hardware optimized for SSL key breaking. According to data shared with us from a survey of SSL keys used by various websites, the majority of web companies were using 1024-bit SSL ciphers and RSA-based encryption through 2012. Given enough specialized hardware, it is within the realm of possibility that the NSA could within a reasonable period of time reverse engineer 1024-bit SSL keys for certain web companies. If they'd been recording the traffic to these web companies, they could then use the broken key to go back and decrypt all the transactions. While this seems like a compelling theory, ultimately, we remain skeptical this is how the PRISM program described in the slides actually works. Cracking 1024-bit keys would be a big deal and likely involve some cutting-edge cryptography and computational power, even for the NSA. The largest SSL key that is known to have been broken to date is 768 bits long. While that was 4 years ago, and the NSA undoubtedly has some of the best cryptographers in the world, it's still a considerable distance from 768 bits to 1024 bits -- especially given the slide suggests Microsoft's key would have to had been broken back in 2007. Moreover, the slide showing the dates on which "collection began" for various companies also puts the cost of the program at $20M/year. That may sound like a lot of money, but it is not for an undertaking like this. Just the power necessary to run the server farm needed to break a 1024-bit key would likely cost in excess of $20M/year. While the NSA may have broken 1024-bit SSL keys as part of some other program, if the slide is accurate and complete, we think it's highly unlikely they did so as part of the PRISM program. A not particularly glamorous alternative theory is that the NSA didn't break the SSL key but instead just cajoled rogue employees at firms with access to the private keys -- whether the companies themselves, partners they'd shared the keys with, or the certificate authorities who issued the keys in the first place -- to turn them over. That very well may be possible on a budget of $20M/year. [....] Google is a notable anomaly. The company uses a 1024-bit key, but, unlike all the other companies listed above, rather than using a default cipher suite based on the RSA encryption algorithm, they instead prefer the Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) cipher suites. Without going into the technical details, a key difference of ECDHE is that they use a different private key for each user's session. This means that if the NSA, or anyone else, is recording encrypted traffic, they cannot break one private key and read all historical transactions with Google. The NSA would have to break the private key generated for each session, which, in Google's case, is unique to each user and regenerated for each user at least every 28-hours. While ECDHE arguably already puts Google at the head of the pack for web transaction security, to further augment security Google has publicly announced that they will be increasing their key length to 2048-bit by the end of 2013. Assuming the company continues to prefer the ECDHE cipher suites, this will put Google at the cutting edge of web transaction security.
2048-bit ECDHE sounds like the way to go, and CloudFlare now support that too.prism security nsa cloudflare ssl tls ecdhe elliptic-curve crypto rsa key-lengths
Record companies to target 20 more pirate sites after court ruling - Independent.ie
Looks like IRMA are following the lead of the UK's BPI, by chasing the proxy sites next:
Up to 20 internet sites are to be targeted by an organisation representing record companies in a move to stamp out the illegal pirating of music and other copyright material. The Irish Recorded Music Association (IRMA) said it would be immediately moving against the 20 "worst offenders" to "take out" internet sites involved in the illegal downloading of copyright work.
However, looks like this will involve more court time:Last night IRMA director general, Dick Doyle said the High Court ruling was only the first step in "taking out many internet sites involved in illegally downloading music. "We will be back in court very shortly to take out five to 10 other sites. We have already selected a total of 20 of the worst offender sites and we will go after the next five in the very near future," he said.
That's not going to be cheap!courts ireland law irma piracy pirate-bay bpi proxies filesharing copyright
Building a Modern Website for Scale (QCon NY 2013) [slides]
some great scalability ideas from LinkedIn. Particularly interesting are the best practices suggested for scaling web services: 1. store client-call timeouts and SLAs in Zookeeper for each REST endpoint; 2. isolate backend calls using async/threadpools; 3. cancel work on failures; 4. avoid sending requests to GC'ing hosts; 5. rate limits on the server. #4 is particularly cool. They do this using a "GC scout" request before every "real" request; a cheap TCP request to a dedicated "scout" Netty port, which replies near-instantly. If it comes back with a 1-packet response within 1 millisecond, send the real request, else fail over immediately to the next host in the failover set. There's still a potential race condition where the "GC scout" can be achieved quickly, then a GC starts just before the "real" request is issued. But the incidence of GC-blocking-request is probably massively reduced. It also helps against packet loss on the rack or server host, since packet loss will cause the drop of one of the TCP packets, and the TCP retransmit timeout will certainly be higher than 1ms, causing the deadline to be missed. (UDP would probably work just as well, for this reason.) However, in the case of packet loss in the client's network vicinity, it will be vital to still attempt to send the request to the final host in the failover set regardless of a GC-scout failure, otherwise all requests may be skipped. The GC-scout system also helps balance request load off heavily-loaded hosts, or hosts with poor performance for other reasons; they'll fail to achieve their 1 msec deadline and the request will be shunted off elsewhere. For service APIs with real low-latency requirements, this is a great idea.
gc-scout gc java scaling scalability linkedin qcon async threadpools rest slas timeouts networking distcomp netty tcp udp failover fault-tolerance packet-loss
Why I won't give the European Parliament the data protection analysis it wanted
Holy crap. Simon Davies rips into the EU data-protection reform disaster with gusto:
The situation was an utter disgrace. The advertising industry even gave an award to an Irish Minister for destroying some of the rights in the regulation while the UK managed to force a provision that would make the direct marketing industry a "legitimate" processing operation in its own right, putting it on the same level of lawful processing as fraud prevention. Things got to the point where even the most senior data protection officials in Europe stopped trying to influence events and had told me "let the chips fall as they may". [...] But let's take a step back for a moment from this travesty. Out on the streets – while most may not know what data protection is – people certainly know what it is supposed to protect. People value their privacy and they will be vocal about attempts to destroy it. I had said as much to the joint parliamentary meeting, observing "the one element that has been left out of all these efforts is the public". However, as the months rolled on, the only message being sent to the public was that data protection is an anachronism stitched together with self interest and impracticality. [...] I wasn't aware at the time that there was a vast stitch-up to kill the reforms. I cannot bring myself to present a temperate report with measured wording that pretends this is all just normal business. It isn't normal business, and it should never be normal business in any civilized society. How does one talk in measured tones about such endemic hypocrisy and deception? If you want to know who the real enemy of privacy is, don't just look to the American agencies. The real enemy is right here in the European Parliament in the guise of MEPs who have knowingly sold our rights away to maintain powerful relationships. I'd like to say they were merely hoodwinked into supporting the vandalism, but many are smart people who knew exactly what they were doing.
Nice work, Irish presidency! His bottom line:Is there a way forward? I believe so. First, governments should yield to common decency and scrap the illegitimate and poisoned Irish Council draft and hand the task to the Lithuanian Presidency that commences next month. Second, the Irish and British governments should be infinitely more transparent about their cooperation with intrusive interests that fuelled the deception.
ireland eu europe reform law data-protection privacy simon-davies meps iab
Persuading David Simon (Pinboard Blog)
Maciej Ceglowski with a strongly-argued rebuttal of David Simon's post about the NSA's PRISM. This point in particular is key:
The point is, you don't need human investigators to find leads, you can have the algorithms do it [based on the call graph or network of who-calls-who]. They will find people of interest, assemble the watch lists, and flag whomever you like for further tracking. And since the number of actual terrorists is very, very, very small, the output of these algorithms will consist overwhelmingly of false positives.
false-positives maciej privacy security nsa prism david-simon accuracy big-data filtering anti-spam
Schneier on Security: Blowback from the NSA Surveillance
Unintended consequences on US-focused governance of the internet and cloud computing:
Writing about the new Internet nationalism, I talked about the ITU meeting in Dubai last fall, and the attempt of some countries to wrest control of the Internet from the US. That movement just got a huge PR boost. Now, when countries like Russia and Iran say the US is simply too untrustworthy to manage the Internet, no one will be able to argue. We can't fight for Internet freedom around the world, then turn around and destroy it back home. Even if we don't see the contradiction, the rest of the world does.
internet freedom cloud-computing amazon google hosting usa us-politics prism nsa surveillance
EU unlocks a great new source of online innovation
Today the European Parliament voted to formally agree new rules on open data – effectively making a reality of the proposal which I first put forward just over 18 months ago, and making it easier to open up huge amounts of public sector data.
Great news -- wonder how it'll affect the Ordnance Survey of Ireland?UK ISPs Secretly Start Blocking Torrent Site Proxies | TorrentFreak
The next step of cat-and-mouse. Let's see what the pirate sites do next...
The blocking orders are intended to deter online piracy and were requested by the music industry group BPI on behalf of a variety of major labels. Thus far they've managed to block access to The Pirate Bay, Kat.ph, H33T and Fenopy, and preparations are being made to add many others. The effectiveness of these initial measures has been called into doubt, as they are relatively easy to bypass. For example, in response to the blockades hundreds of proxy sites popped up, allowing subscribers to reach the prohibited sites via a detour. However, as of this week these proxies are also covered by the same blocklist they aim to circumvent, without a new court ruling. The High Court orders give music industry group BPI the authority to add sites to the blocklist without oversight. Until now some small changes have been made, mostly in response to The Pirate Bay's domain hopping endeavors, but with the latest blocklist update a whole new range of websites is being targeted.
bittorrent blocking filesharing copyright bpi piracy pirate-bay proxies fenopy kat.ph h33t filtering uk
-
'Not long ago, we began rendering 3D models on GitHub. Today we're excited to announce the latest addition to the visualization family - geographic data. Any .geojson file in a GitHub repository will now be automatically rendered as an interactive, browsable map, annotated with your geodata.' As this HN comment notes, https://news.ycombinator.com/item?id=5875693 -- 'I'd much rather Github cleaned up the UI for existing features than added these little flourishes that I can't imagine even 1% of users use.' Something is seriously wrong in how GitHub decides product direction if this kind of wankology (and that Judy-array crap) is what gets prioritised. :( (via Marc O'Morain)
via:marc github mapping maps geojson hacking product-management ui pull-requests
-
The issue [...] is that it's just not cost effective for anyone to actually stand up and challenge Warner Music, who has strong financial incentive to pretend the copyright is still valid. Well, apparently, someone is pissed off enough to try. The creatively named Good Morning to You Productions, a documentary film company planning a film about the song Happy Birthday, has now filed a lawsuit concerning the copyright of Happy Birthday and are seeking to force Warner/Chappell to return the millions of dollars it has collected over the years. That's going to make this an interesting case.
music copyright law via:bwalsh public-domain happy-birthday songs warner-music lawsuits
-
metric collectors for various stuff not (or poorly) handled by other monitoring daemons Core of the project is a simple daemon (harvestd), which collects metric values and sends them to graphite carbon daemon (and/or other configured destinations) once per interval. Includes separate data collection components ("collectors") for processing of: /proc/slabinfo for useful-to-watch values, not everything (configurable). /proc/vmstat and /proc/meminfo in a consistent way. /proc/stat for irq, softirq, forks. /proc/buddyinfo and /proc/pagetypeinfo (memory fragmentation). /proc/interrupts and /proc/softirqs. Cron log to produce start/finish events and duration for each job into a separate metrics, adapts jobs to metric names with regexes. Per-system-service accounting using systemd and it's cgroups. sysstat data from sadc logs (use something like sadc -F -L -S DISK -S XDISK -S POWER 60 to have more stuff logged there) via sadf binary and it's json export (sadf -j, supported since sysstat-10.0.something, iirc). iptables rule "hits" packet and byte counters, taken from ip{,6}tables-save, mapped via separate "table chain_name rule_no metric_name" file, which should be generated along with firewall rules (I use this script to do that).
Pretty exhaustive list of system metrics -- could have some interesting ideas for Linux OS-level metrics to monitor in future.graphite monitoring metrics unix linux ops vm iptables sysadmin
Former NSA Boss: We Don't Data Mine Our Giant Data Collection, We Just Ask It Questions
'Well, that's - no, we're going to use it. But we're not going to use it in the way that some people fear. You put these records, you store them, you have them. It's kind of like, I've got the haystack now. And now let's try to find the needle. And you find the needle by asking that data a question. I'm sorry to put it that way, but that's fundamentally what happens. All right. You don't troll through the data looking for patterns or anything like that. The data is set aside. And now I go into that data with a question that - a question that is based on articulable(ph), arguable, predicate to a terrorist nexus.'
Yep, that's data mining.data-mining questions haystack needle nsa usa politics privacy data-protection michael-hayden
-
fastutil extends the Java™ Collections Framework by providing type-specific maps, sets, lists and queues with a small memory footprint and fast access and insertion; provides also big (64-bit) arrays, sets and lists, and fast, practical I/O classes for binary and text files. It is free software distributed under the Apache License 2.0. It requires Java 6 or newer.
used by Facebook (along with Apache Giraph, Netty, Unsafe) to speed up "weekend Hive jobs" to "coffee breaks". http://www.slideshare.net/nitayj/2013-0603-berlin-buzzwordsvia:highscalability facebook giraph optimization java speed fastutil collections data-structures
-
good microbenchmarking of a bunch of Java collections; Trove, fastutil, PCJ, mahout-collections, hppc
java collections benchmarks performance speed coding data-structures optimization
Spamalot reigns: the spoils of Ireland's EU kingship | The Irish Times - Thu, Jun 13, 2013
The spam presidency. As European citizens are made the miserable targets of unimpeded "direct marketing", that may be how Ireland's stint in the EU presidency seat is recalled for years to come. Under the guiding hand of Minister for Justice Alan Shatter, the Council of the European Union has submitted proposals for amendments to a proposed new data protection regulation, all of which overwhelmingly favour business and big organisations, not citizens. The most obviously repugnant and surprising element in the amendments is a watering down of existing protections for EU citizens against the willy-nilly marketing Americans are forced to endure. In the US there are few meaningful restrictions on what businesses can do with people's personal information when pitching products and services at them. In the EU, this has always been strictly controlled; information gathered for one purpose cannot be used by a business to sell whatever it wants – unless you have opted in to receive such solicitations. This means you are not constantly bombarded by emails and junk mail, nor do you get non-stop phone calls from telemarketers. Under the proposed amendments to the draft data protection regulation, direct marketing would become a legal form of data processing. In effect, this would legitimise spam email, junk print mail and marketing calls. This unexpected provision signals just how successful powerful corporate lobbyists have been in convincing ministers that business matters more than privacy or giving citizens reasonable control over their personal information. Far worse is contained in other amendments, which in effect turn the original draft of the regulation upside down.
Fantastic article from Karlin Lillington in today's Times on the terrible amendments proposed for the EU's data protection law.eu law prism data-protection privacy ireland ec marketing spam anti-spam email
Vagrant and Chef to provision dev test environments
We have recently switched from a manually configured development environment to a nearly fully automated one using Vagrant, Chef, and a few other tools. With this transition, we've moved to an environment where data on the dev boxes is considered disposable and only what's checked into the SCM is "real". This is where we've always wanted to be, but without the ability to easily rebuild the dev environment from scratch, it's hard to internalize this behavior pattern.
Rapid Response: The NSA Prism Leak
'The biggest leak in the history of US security or nothing to worry about? A breach of trust and a data protection issue or a necessary secret project to protect American interests? [Tomorrow] lunchtime Science Gallery Rapid Response event [sic] will pick through the jargon, examine the minutiae of the National Security Agency's PRISM project and the whistle blower Edward Snowden's revelations, and discuss what it means for you and everyone. And we'll look at the bigger picture too. Journalist Una Mullally will chair a panel of guests on the story that everyone is talking about. '
-
Four major music companies have secured court orders requiring six internet service providers to block access by subscribers to various Pirate Bay websites within some 30 days in a bid to prevent illegal downloading of copyright music and other material. [...] Today, Mr Justice Brian McGovern said he was satisfied to make the order in circumstances including that new copyright laws here and in the EU permitted such orders to be made. He said he fully agreed with a previous High Court judge who had said he would make such blocking orders if the law permitted and noted the law now allowed for such orders. The form of the orders means the music companies will not have to make fresh applications to court if Pirate Bay changes its location on the internet.
pirate-bay blocking filtering internet ireland upc eircom vodafone digiweb three imagine o2 copyright
Labour TD ignores tough questions on web case
I [Tom Murphy] have asked [Sean Sherlock] a question: Does he have any comment about the lawsuit between EMI and UPC (and a raft of other ISPs too btw) which is using his SI to attempt to block PirateBay? A court case he said would not happen. Now, I am blocked from following him on Twitter. This is not how a proper political system works.
politics ireland twitter sean-sherlock tom-murphy boards devore copyright
PRISM explains the wider lobbying issues surrounding EU data protection reform | EDRI
The US has very successfully and expertly lobbied against the [EU] data protection package directly, it has mobilised and supported US industry lobbying. US industry has lobbied in its own name and mobilised malleable European trade associations to lobby on their behalf to amplify their message, "independent" "think tanks" have been created to amplify their message again. The result is not just the biggest lobbying effort that Brussels has ever seen, but also the broadest. Compliant Members of the European Parliament (MEPs) and EU Member States [...] have been imposing a "death by a thousand cuts" on the Regulation. Where previously there was a clear obligation to collect the "minimum necessary" data for any given service, the vague requirement to retain "not excessive" data is now preferred. Where previously companies could only use data for purposes that were "compatible" with the original reason for collecting the data, the Irish EU Presidency (pdf) has proposed a comical definition of "compatible" based on five elements, only one of which is related to the dictionary definition of the word. Members of the European Parliament and EU Member States are falling over themselves to ensure that the EU does not maintain its strategic advantage over the US. In addition to dismantling the proposed Regulation, countries like the UK desperately seek to delay the whole process and subsume it into the EU-US free trade agreement (the so-called "investment partnership" TTIP/TAFTA), which would subordinate a fundamental rights discussion in a trade negotiation. The UK government is even prepared to humiliate itself by arguing in favour of the US position on the basis that two and a half years (see Communication from 2010, pdf) of discussion is too fast!
edri data-protection eu ec ireland politics usa meps privacy uk free-trade
Microsoft admits US government can access EU-based cloud data
interesting point from an MS Q&A back in 2011, quite relevant nowadays:
Q: Can Microsoft guarantee that EU-stored data, held in EU based datacenters, will not leave the European Economic Area under any circumstances — even under a request by the Patriot Act? A: Frazer explained that, as Microsoft is a U.S.-headquartered company, it has to comply with local laws (the United States, as well as any other location where one of its subsidiary companies is based). Though he said that "customers would be informed wherever possible," he could not provide a guarantee that they would be informed — if a gagging order, injunction or U.S. National Security Letter permits it. He said: "Microsoft cannot provide those guarantees. Neither can any other company." While it has been suspected for some time, this is the first time Microsoft, or any other company, has given this answer. Any data which is housed, stored or processed by a company, which is a U.S. based company or is wholly owned by a U.S. parent company, is vulnerable to interception and inspection by U.S. authorities.
microsoft privacy cloud-computing eu data-centers data-protection nsa fisa usa
-
Irish MEP serving as a rapporteur on reform of the EU data protection regime, was given an award by an advertising trade group last month:
Sean Kelly, Fine Gael MEP for Ireland South [who serves as the EU's Industry Committee Rapporteur for the General Data Protection Regulation], has been selected to receive the prestigious IAB Europe Award for Leadership and Excellence for his approach to dealing with privacy concerns over shortcomings in the European Commission's data protection proposal. IAB Europe represents more than 5,500 online advertising media, research and analytics organisations.
iab-europe awards spam sean-kelly ireland meps politics eu data-protection privacy ec
-
No subject appears to be more controversial to distributed systems engineers than the oft-quoted, oft-misunderstood CAP theorem. The purpose of this FAQ is to explain what is known about CAP, so as to help those new to the theorem get up to speed quickly, and to settle some common misconceptions or points of disagreement.
seeing into the UV spectrum after Cataract Surgery with Crystalens
I've been very happy so far with the Crystalens implant for Cataract Surgery [...] one unexpected/interesting aspect is I see a violet glow that others do not - perhaps I'm more sensitive to the low end of the visible light spectrum.
(via Tony Finch)via:fanf science perception augmentation uv light sight cool cataracts surgery lens eyes
Instagram: Making the Switch to Cassandra from Redis, a 75% 'Insta' Savings
shifting data out of RAM and onto SSDs -- unsurprisingly, big savings.
a 12 node cluster of EC2 hi1.4xlarge instances; we store around 1.2TB of data across this cluster. At peak, we're doing around 20,000 writes per second to that specific cluster and around 15,000 reads per second. We've been really impressed with how well Cassandra has been able to drop into that role.
ram ssd cassandra databases nosql redis instagram storage ec2
-
Oh god. this sounds like an impending privacy and anti-spam disaster. "business-focussed":
Overall, the [Irish EC Presidency's] draft compromise text can be seen as a more business-focused, pragmatic approach. For example, the Presidency has drafted an additional recital (Recital 3a), clarifying the right to data protection as a qualified right, highlighting the principle of proportionality and importance of other competing fundamental rights, including the freedom to conduct a business.
and some pretty serious relaxation of how consent for use of personal data is measured:The criterion for valid consent is amended from "explicit" to "unambiguous," except in the case of processing special categories of data (i.e., sensitive personal data) (Recital 25 and Article 9(2)). This reverts to the current position under the Data Protection Directive and is a concession to the practical difficulty of obtaining explicit consent in all cases. The criteria for valid consent are further relaxed by the ability to obtain consent in writing, orally or in an electronic manner, and where technically feasible and effective, valid consent can be given using browser settings and other technical solutions. Further, the requirement that the controller bear the burden of proof that valid consent was obtained is limited to a requirement that the controller be able to "demonstrate" that consent was obtained (Recital 32 and Article 7(1)). The need for "informed" consent is also relaxed from the requirement to provide the full information requirements laid out in Article 14 to the minimal requirements that the data subject "at least" be made aware of: (1) the identity of the data controller, and (2) the purpose(s) of the processing of their personal data (Recitals 33 and 48).
anti-spam privacy data-protection spam ireland eu ec regulation
-
wow, great view of which MEPs are eviscerating the EU's data protection regime:
Currently the EU is negotiating about new data privacy laws. This new EU Regulation will replace all existing national laws on data privacy. Here you can see a general overview which Members of the European Parliament (MEPs) are pushing for more or less data privacy. Choose a country, a political group or a MEP from the "Top 10" list to find out more.
EDRI's comments on EU proposals to reform privacy law
Amendments 762, 764 and 765 in particular seem to move portions of the law from "confirmed opt-in required" to "opt-out is ok" -- which sounds like a risk where spam and unsolicited actions on a person's data are concerned
-
'Easy Amazon EC2 Instance Comparison'. a nice UI on the various EC2 instance types on offer with their key attributes. Misses out availability of EBS-optimized instances though
HyperLevelDB: A High-Performance LevelDB Fork
'HyperLevelDB improves on LevelDB in two key ways: Improved parallelism: HyperLevelDB uses more fine-grained locking internally to provide higher throughput for multiple writer threads. Improved compaction: HyperLevelDB uses a different method of compaction that achieves higher throughput for write-heavy workloads, even as the database grows.'
leveldb storage key-value-stores persistence unix libraries open-source
EU Council deals killer blow to privacy reforms
'In an extraordinary result for corporate lobbying, direct marketing would by default be considered a legitimate data process and would therefore – by default – be lawful.'
eu politics data-protection privacy anti-spam spam eu-council direct-marketing
Care and Feeding of Large Scale Graphite Installations [slides]
good docs for large-scale graphite use: 'Tip and tricks of using and scaling graphite. First presented at DevOpsDays Austin Texas 2013-05-01'
Low-latency stock trading "jumps the gun" due to default NTP configuration settings
On June 3, 2013, trading in SPY exploded at 09:59:59.985, which is 15 milliseconds before the ISM's Manufacturing number released at 10:00:00. Activity in the eMini (traded in Chicago), exploded at 09:59:59.992, which is 8 milliseconds before the news release, but 7 milliseconds after SPY. Note how SPY and the eMini traded within a millisecond for the Consumer Confidence release last week, but the eMini lagged SPY by about 7 milliseconds for the ISM Manufacturing release. The simultaneous trading on Consumer Confidence is because that number is released at the same time in both NYC and Chicago. The ISM Manufacturing number is probably released on a low latency feed in NYC, and then takes 5-7 milliseconds, due to the speed of light, to reach Chicago. Either the clock used to release the ISM number was 15 milliseconds fast, or someone (correctly) jumped the gun. Update: [...] The clock used to release the ISM was indeed, 15 milliseconds fast. This could be from using the default setting of many NTP clients, which allows the clock to drift up to about 16 milliseconds before adjusting time.
ntp time synchronization spy trading stocks low-latency clocks internet
the infamous 2008 S3 single-bit-corruption outage
Neat, I didn't realise this was publicly visible. A single corrupted bit infected the S3 gossip network, taking down the whole S3 service in (iirc) one region:
We've now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect. We use MD5 checksums throughout the system, for example, to prevent, detect, and recover from corruption that can occur during receipt, storage, and retrieval of customers' objects. However, we didn't have the same protection in place to detect whether [gossip state] had been corrupted. As a result, when the corruption occurred, we didn't detect it and it spread throughout the system causing the symptoms described above. We hadn't encountered server-to-server communication issues of this scale before and, as a result, it took some time during the event to diagnose and recover from it. During our post-mortem analysis we've spent quite a bit of time evaluating what happened, how quickly we were able to respond and recover, and what we could do to prevent other unusual circumstances like this from having system-wide impacts. Here are the actions that we're taking: (a) we've deployed several changes to Amazon S3 that significantly reduce the amount of time required to completely restore system-wide state and restart customer request processing; (b) we've deployed a change to how Amazon S3 gossips about failed servers that reduces the amount of gossip and helps prevent the behavior we experienced on Sunday; (c) we've added additional monitoring and alarming of gossip rates and failures; and, (d) we're adding checksums to proactively detect corruption of system state messages so we can log any such messages and then reject them.
This is why you checksum all the things ;)s3 aws post-mortems network outages failures corruption grey-failures amazon gossip
-
Aphyr and Peter Bailis collect an authoritative list of known network partition and outage cases from published post-mortem data:
This post is meant as a reference point -- to illustrate that, according to a wide range of accounts, partitions occur in many real-world environments. Processes, servers, NICs, switches, local and wide area networks can all fail, and the resulting economic consequences are real. Network outages can suddenly arise in systems that are stable for months at a time, during routine upgrades, or as a result of emergency maintenance. The consequences of these outages range from increased latency and temporary unavailability to inconsistency, corruption, and data loss. Split-brain is not an academic concern: it happens to all kinds of systems -- sometimes for days on end. Partitions deserve serious consideration.
I honestly cannot understand people who didn't think this was the case. 3 years reading (and occasionally auto-cutting) Amazon's network-outage tickets as part of AWS network monitoring will do that to you I guess ;) -
from Atelier Olschinsky. 'Fine Art Print on Hahnemuehle Photo Rag Bright White 310g; Limited Edition / Numbered and signed by the artist'
incompetent error-handling code in the mongo-java-driver project
an unexplained invocation of Math.random() in the exception handling block of this MongoDB java driver class causes roflscale lols in the github commit notes. http://stackoverflow.com/a/16833798 has more explanation.
github commits mongodb webscale roflscale random daily-wtf wtf
-
'What is a Hermetic Server? The short definition would be a "server in a box". If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! This is a special case of the more general "hermetic" concept which applies to an isolated system not necessarily on a single machine. Why is it useful to have a hermetic server? Because if your entire [system under test] is composed of hermetic servers, it could all be started on a single machine for testing; no network connection necessary! The single machine could be a physical or virtual machine.' These also qualify as "fakes", using the terminology Martin Fowler suggests at http://martinfowler.com/bliki/TestDouble.html , I think
google testing hermetic-servers test test-doubles unit-testing
-
hooray, sanity from the Google Testing blog. this has been a major cause of pain in the past, dealing with tricky rewrites of mock-heavy unit test code
Casalattico - Wikipedia, the free encyclopedia
How wierd. Many of the well-known chippers in Ireland are run by families from the same comune in Italy.
In the late 19th and early 20th century a significant number of young people left Casalattico to work in Ireland, with many founding chip shops there. Most second, third and fourth generation Irish-Italians can trace their lineage back to the municipality, with names such as Magliocco, Fusco, Marconi, Borza, Macari, Rosato and Forte being the most common. Although the Forte family actually originates from the village of Mortale, renamed Mon Forte due to the achievements of the Forte family. It is believed that up to 8,000 Irish-Italians have ancestors from Casalattico. The village is home to an Irish festival every summer to celebrate the many families that moved from there to Ireland.
(via JK)rome lazio italy ireland chip-shops chippers history emigration casalattico work irish-italians via:jk
Videos from the Continuous Delivery track at QCon SF 2012
Think we'll be watching some of these in work soon -- Jez Humble's talk (the last one) in particular looks good:
Amazon, Etsy, Google and Facebook are all primarily software development shops which command enormous amounts of resources. They are, to use Christopher Little's metaphor, unicorns. How can the rest of us adopt continuous delivery? That's the subject of my talk, which describes four case studies of organizations that adopted continuous delivery, with varying degrees of success. One of my favourites – partly because it's embedded software, not a website – is the story of HP's LaserJet Firmware team, who re-architected their software around the principles of continuous delivery. People always want to know the business case for continuous delivery: the FutureSmart team provide one in the book they wrote that discusses how they did it.
continuous-integration continuous-delivery build release process dev deployment videos qcon towatch hp
_Dynamic Histograms: Capturing Evolving Data Sets_ [pdf]
Currently, histograms are static structures: they are created from scratch periodically and their creation is based on looking at the entire data distribution as it exists each time. This creates problems, however, as data stored in DBMSs usually varies with time. If new data arrives at a high rate and old data is likewise deleted, a histogram's accuracy may deteriorate fast as the histogram becomes older, and the optimizer's effectiveness may be lost. Hence, how often a histogram is reconstructed becomes very critical, but choosing the right period is a hard problem, as the following trade-off exists: If the period is too long, histograms may become outdated. If the period is too short, updates of the histogram may incur a high overhead. In this paper, we propose what we believe is the most elegant solution to the problem, i.e., maintaining dynamic histograms within given limits of memory space. Dynamic histograms are continuously updateable, closely tracking changes to the actual data. We consider two of the best static histograms proposed in the literature [9], namely V-Optimal and Compressed, and modify them. The new histograms are naturally called Dynamic V-Optimal (DVO) and Dynamic Compressed (DC). In addition, we modified V-Optimal's partition constraint to create the Static Average-Deviation Optimal (SADO) and Dynamic Average-Deviation Optimal (DADO) histograms.
(via d2fn)via:d2fn histograms streaming big-data data dvo dc sado dado dynamic-histograms papers toread
How I decoded the human genome - Salon.com
classic long-read article from John Sundman: 'We are becoming the masters of our own DNA. But does that give us the right to decide that my children should never have been born?' part two at http://www.salon.com/2003/10/22/genome_two/
human genome genomics eugenics politics life john-sundman disability health dna medicine salon long-reads children
The "Meme Hustler" hustler: Evgeny Morozov's Stupid Talk about Tim O'Reilly
great long-read blog post from John Sundman debunking Evgeny Morozov's takedown of Tim O'Reilly
debunking john-sundman evgeny-morozov tim-oreilly tech technological-solutionism futurism writing silicon-valley utopianism open-source oss
Strange Passion Presents Chant Chant Chant, Choice & SM Corporation live
'We are delighted to announce, for one night only, 3 legendary Irish Post Punk bands performing live in Dublin after a 30 year hiatus. This follows on from the critically acclaimed release of the Strange Passion Irish Post Punk compilation in 2012. Post punk legends Chant Chant Chant will perform along with electronic music pioneers Choice and SM Corporation. '
choice music ireland post-punk electronic dublin strange-passion gigs
'Mythbusting Modern Hardware to gain "Mechanical Sympathy"' [slides]
Martin Thompson's latest talk -- taking a few common concepts about modern hardware performance and debunking/confirming them, mythbusters-style
mythbusters hardware mechanical-sympathy martin-thompson java performance cpu disks ssd
High home ownership can seriously damage labor market, new study suggests
Interesting -- a healthy rental market is needed to allow sufficient labour mobility. This matches what I heard and saw from friends and coworkers in the US, anecdotally
Concert Industry Struggles With 'Bots' That Siphon Off Tickets - NYTimes.com
Bots now buying more than 60% of tickets, one group requesting up to 200,000 per day; bot writers now charging $14 per 10k captchas (via Shane Naughton)
ticketmaster scalping tickets via:shane-naughton bots captchas abuse
Instant artist statement: Arty Bollocks Generator
'My work explores the relationship between the body and vegetarian ethics. With influences as diverse as Munch and Francis Bacon, new synergies are created from both orderly and random narratives. Ever since I was a postgraduate I have been fascinated by the essential unreality of the moment. What starts out as undefined soon becomes corroded into a hegemony of greed, leaving only a sense of failing and the chance of a new order. As temporal replicas become transformed through diligent and undefined practice, the viewer is left with an impression of the darkness of our culture.'
funny humor art arty bollocks generator hacks via:leroideplywood
Communication costs in real-world networks
Peter Bailis has generated some good real-world data about network performance and latency, measured using EC2 instances, between ec2 regions, between zones, and between hosts in a single AZ. good data (particularly as I was looking for this data in a public source not too long ago).
I wasn't aware of any datasets describing network behavior both within and across datacenters, so we launched m1.small Amazon EC2 instances in each of the eight geo-distributed "Regions," across the three us-east "Availability Zones" (three co-located datacenters in Virginia), and within one datacenter (us-east-b). We measured RTTs between hosts for a week at a granularity of one ping per second.
Some of the high-percentile measurements are undoubtedly impact of host and VM behaviour, but that is still good data for a typical service built in EC2.networks performance measurements benchmarks ops ec2 networking internet az latency
Reducing MongoDB traffic by 78% with Redis | Crashlytics Blog
One for @roflscaletips. Crashlytics reduce MongoDB load by hacking in some hand-coded caching into their Rails app, instead of just using a front-line HTTP cache to reduce Rails *and* db load. duh. (via Oisin)
crashlytics fail roflscale rails caching redis ruby via:oisin
Display Hidden Files in OS X Open and Save Dialog Boxes
yet another laughable UI kludge in OS X. ridiculous
usability osx apple ui kludges hidden-files dot-files command-shift-option-elbow magic
-
"Dear Mr Tilman, this is the only way I can help you. saluti, Giorgio Moroder". I love it -- someone call Tufte
graphics giorgio-moroder history music ilx basslines donna-summer synths
Hollywood Studios [attempt to censor] Pirate Bay Documentary
Probably not deliberate, but pretty damn inept.
Over the past weeks several movie studios have been trying to suppress the availability of TPB-AFK [the Pirate Bay documentary] by asking Google to remove links to the documentary from its search engine. The links are carefully hidden in standard DMCA takedown notices for popular movies and TV-shows. The silent attacks come from multiple Hollywood sources including Viacom, Paramount, Fox and Lionsgate and are being sent out by multiple anti-piracy outfits. Fox, with help from six-strikes monitoring company Dtecnet, asked Google to remove a link to TPB-AFK on Mechodownload. Paramount did the same with a link on the Warez.ag forums. Viacom sent at least two takedown requests targeting links to the Pirate Bay documentary on Mrworldpremiere and Rapidmoviez. Finally, Lionsgate jumped in by asking Google to remove a copy of TPB-AFK from a popular Pirate Bay proxy.
funny inept hollywood lionsgate fox viacom paramount dtecnet tpb-afk piratebay piracy copyright movies google
Flashback: How Yahoo Killed Flickr and Lost the Internet
This is about the best tech journalism I've ever read on Flickr. nice one Mat Honan
gizmodo flickr acquisition mergers yahoo corporate-culture mat-honan tech journalism
Resisting the lure of the Freeman movement | Workers Solidarity Movement
An anarchist critique of the Freeman movement from the WSM:
This has been a very brief overview of the Freeman movement that has tried to capture with broad strokes its nature and possible responses. There is room for much more work, including a more in-depth analysis of the various flaws in the approach to the law. The greatest danger however is allowing a movement to develop within anarchist circles that ignores the principle of mutual aid and implicitly promotes private ownership of resources, that by granting absolute right to individuals gives them the ability to ignore their responsibilities to the wider community and ecology that sustains them. In more traditional terms, the movement is one all about negative freedoms, ignoring positive freedom as a concept.
The Reactionary 'Freeman-?on-?the-?land' and a Political Fracture
Another leftie view on the Freeman movement
freeman-on-the-land politics ireland left-wing anarchism law
-
Well, apparently tomorrow, but close enough. Happy birthday to bradfitz' greatest creation and its wonderful slab allocator!
birthdays code via:alex-popescu open-source history malloc memory caching memcached
Newegg nukes "corporate troll" Alcatel in third patent appeal win this year
I am loving this. Particularly this:
At trial in East Texas Cheng took the stand to tell Newegg's story. Alcatel-Lucent's corporate representative, at the heart of its massive licensing campaign, couldn't even name the technology or the patents it was suing Newegg over. "Successful defendants have their litigation managed by people who care," said Cheng. "For me, it's easy. I believe in Newegg, I care about Newegg. Alcatel Lucent, meanwhile, they drag out some random VP—who happens to be a decorated Navy veteran, who happens to be handsome and has a beautiful wife and kids—but the guy didn't know what patents were being asserted. What a joke." "Shareholders of public companies that engage in patent trolling should ask themselves if they're really well-served by their management teams," Cheng added. "Are they properly monetizing their R&D? Surely there are better ways to make money than to just rely on litigating patents. If I was a shareholder, I would take a hard look as to whether their management was competent."
patents ip swpats alcatel bell-labs newegg east-texas litigation lucent
Call me maybe: Carly Rae Jepsen and the perils of network partitions
Kyle "aphyr" Kingsbury expands on his slides demonstrating the real-world failure scenarios that arise during some kinds of partitions (specifically, the TCP-hang, no clear routing failure, network partition scenario). Great set of blog posts clarifying CAP
distributed network databases cap nosql redis mongodb postgresql riak crdt aphyr
-
Welcome to the Galapagos of Chinese "open" source. I call it "gongkai" (??). Gongkai is the transliteration of "open" as applied to "open source". I feel it deserves a term of its own, as the phenomenon has grown beyond the so-called "shanzhai" (??) and is becoming a self-sustaining innovation ecosystem of its own. Just as the Galapagos Islands is a unique biological ecosystem evolved in the absence of continental species, gongkai is a unique innovation ecosystem evolved with little western influence, thanks to political, language, and cultural isolation. Of course, just as the Galapagos was seeded by hardy species that found their way to the islands, gongkai was also seeded by hardy ideas that came from the west. These ideas fell on the fertile minds of the Pearl River delta, took root, and are evolving. Significantly, gongkai isn't a totally lawless free-for-all. It's a network of ideas, spread peer-to-peer, with certain rules to enforce sharing and to prevent leeching. It's very different from Western IP concepts, but I'm trying to have an open mind about it.
gongkai bunnie-huang china phone mobile hardware devices open-source
Stability Patterns and Antipatterns [slides]
Michael "Release It!" Nygard's slides from a recent O'Reilly event, discussing large-scale service reliability design patterns
michael-nygard design-patterns architecture systems networking reliability soa slides pdf
Deep In The Game: Not The RTE Guide
Good interview with Alan Maguire, the satirist behind the very funny @NotTheRTEGuide on Twitter:
I've always been a huge fan of TV Go Home and Charlie Brooker in general and it seemed like Irish TV and culture was a good target for the kind of barbed surrealism that he does. (I'm not claiming I'm in his league or anything but he's the main influence). I was really surprised that there hadn't been a parody RTÉ Guide already. TV listings are 140-ish characters already and the RTÉ Guide has a kind of weird place in Irish culture where everybody knows it but nobody our age really has any idea of what's in it anymore. We associate it with a small-c conservatism, or I did at least and I play that up occasionally with the account.
nottherteguide rte rte-guide ireland funny satire interviews
-
'based on my observations while I was a Site Reliability Engineer at Google.' - by Rob Ewaschuk; very good, and matching the similar recommendations and best practices at Amazon for that matter
Monitoring the Status of Your EBS Volumes
Page in the AWS docs which describes their derived metrics and how they are computed -- these are visible in the AWS Management Console, and alarmable, but not viewable in the Cloudwatch UI. grr. (page-joshea!)
Interpol filter scope creep: ASIC ordering unilateral website blocks
Bloody hell. This is stupidity of the highest order, and a canonical example of "filter creep" by a government -- secret state censorship of 1200 websites due to a single investment scam site.
The Federal Government has confirmed its financial regulator has started requiring Australian Internet service providers to block websites suspected of providing fraudulent financial opportunities, in a move which appears to also open the door for other government agencies to unilaterally block sites they deem questionable in their own portfolios. The instrument through which the ISPs are blocking the Interpol list of sites is Section 313 of the Telecommunications Act. Under the Act, the Australian Federal Police is allowed to issue notices to telcos asking for reasonable assistance in upholding the law. [...] Tonight Senator Conroy's office revealed that the incident that resulted in Melbourne Free University and more than a thousand other sites being blocked originated from a different source — financial regulator the Australian Securities and Investment Commission. On 22 March this year, ASIC issued a media release warning consumers about the activities of a cold-calling investment scam using the name 'Global Capital Wealth', which ASIC said was operating several fraudulent websites — www.globalcapitalwealth.com and www.globalcapitalaustralia.com. In its release on that date, ASIC stated: "ASIC has already blocked access to these websites."
scams australia filtering filter-creep false-positives isps asic fraud secrecy
Obfuscatory pie-chart from Garda penalty-points corruption report
"Twitter / gavinsblog: For sake of clarity here is helpful pie chart of the 95.4% of fixed charge notices not terminated #missingthepoint" Paging Edward Tufte: classic example of an obfuscatory pie-chart, diagramming the wrong thing misleadingly. By presenting it like this, it appears that the 95.4% of cases where fixed charge notices were issued by the guards are relevant to the discussion of the other classes; in reality, that means that 4.6% of cases, 37,000 cases, were terminated, some for good reasons, others for not, and it's the difference between those two classes that are relevant. In my opinion, 2 separate pie charts would be better; one to show the dismissed-versus-undismissed count (which IMO could have been omitted entirely), and one to show the good-vs-not-so-good termination reason counts (which is the meat of the issue).
dataviz visualisation data obfuscation gardai police corruption penalty-points
Berkeley DB Java Edition Architecture [PDF]
background white paper on the BDB-JE innards and design, from 2006. Still pretty accurate and good info
bdb-je java berkeley-db bdb design databases pdf white-papers trees
-
This Court has developed a new awareness and understanding of a category of vexatious litigant. As we shall see, while there is often a lack of homogeneity, and some individuals or groups have no name or special identity, they (by their own admission or by descriptions given by others) often fall into the following descriptions: Detaxers; Freemen or Freemen-on-the-Land; Sovereign Men or Sovereign Citizens; Church of the Ecumenical Redemption International (CERI); Moorish Law; and other labels - there is no closed list. In the absence of a better moniker, I have collectively labelled them as Organized Pseudolegal Commercial Argument litigants ["OPCA litigants"], to functionally define them collectively for what they literally are. These persons employ a collection of techniques and arguments promoted and sold by 'gurus' (as hereafter defined) to disrupt court operations and to attempt to frustrate the legal rights of governments, corporations, and individuals. Over a decade of reported cases have proven that the individual concepts advanced by OPCA litigants are invalid. What remains is to categorize these schemes and concepts, identify global defects to simplify future response to variations of identified and invalid OPCA themes, and develop court procedures and sanctions for persons who adopt and advance these vexatious litigation strategies. One participant in this matter [...] appears to be a sophisticated and educated person, but is also an OPCA litigant. One of the purposes of these Reasons is, through this litigant, to uncover, expose, collate, and publish the tactics employed by the OPCA community, as a part of a process to eradicate the growing abuse that these litigants direct towards the justice and legal system we otherwise enjoy in Alberta and across Canada. I will respond on a point-by-point basis to the broad spectrum of OPCA schemes, concepts, and arguments advanced in this action by [him].
Via Ronan Luptonvia:ronanlupton law canada legal freeman opca court tax judgements
-
This classic came up in discussions yesterday...
In the Linux Kernel community Rusty Russell came up with a API rating scheme to help us determine if our API is sensible, or not. It's a rating from -10 to 10, where 10 is perfect is -10 is hell. Unfortunately there are too many examples at the wrong end of the scale.
rusty-russell quality coding kernel linux apis design code-reviews code
-
hooray! Command-line gmailish goodness returns. And with a signed gem, to boot
Martin Thompson, Luke "Snabb Switch" Gorrie etc. review the C10M presentation from Schmoocon
on the mechanical-sympathy mailing list. Some really interesting discussion on handling insane quantities of TCP connections using low volumes of hardware:
This talk has some good points and I think the subject is really interesting. I would take the suggested approach with serious caution. For starters the Linux kernel is nowhere near as bad as it made out. Last year I worked with a client and we scaled a single server to 1 million concurrent connections with async programming in Java and some sensible kernel tuning. I've heard they have since taken this to over 5 million concurrent connections. BTW Open Onload is an open source implementation. Writing a network stack is a serious undertaking. In a previous life I wrote a network probe and had to reassemble TCP streams and kept getting tripped up by edge cases. It is a great exercise in data structures and lock-free programming. If you need very high-end performance I'd talk to the Solarflare or Mellanox guys before writing my own. There are some errors and omissions in this talk. For example, his range of ephemeral ports is not quite right, and atomic operations are only 15 cycles on Sandy Bridge when hitting local cache. A big issue for me is when he defined C10M he did not mention the TIME_WAIT issue with closing connections. Creating and destroying 1 million connections per second is a major issue. A protocol like HTTP is very broken in that the server closes the socket and therefore has to retain the TCB until the specified timeout occurs to ensure no older packet is delivered to a new socket connection.
mechanical-sympathy hardware scaling c10m tcp http scalability snabb-switch martin-thompson
-
This program creates an EBS snapshot for an Amazon EC2 EBS volume. To help ensure consistent data in the snapshot, it tries to flush and freeze the filesystem(s) first as well as flushing and locking the database, if applicable. Filesystems can be frozen during the snapshot. Prior to Linux kernel 2.6.29, XFS must be used for freezing support. While frozen, a filesystem will be consistent on disk and all writes will block. There are a number of timeouts to reduce the risk of interfering with the normal database operation while improving the chances of getting a consistent snapshot. If you have multiple EBS volumes in a RAID configuration, you can specify all of the volume ids on the command line and it will create snapshots for each while the filesystem and database are locked. Note that it is your responsibility to keep track of the resulting snapshot ids and to figure out how to put these back together when you need to restore the RAID setup.
Handy! Measuring & Optimizing I/O Performance
Another good writeup on iostat and EBS, from Ilya Grigorik
AWS forum post on interpreting iostat output for EBS
Great post from AndrewC@EBS on interpreting iostat output on EBS volumes -- from 2009, but still looks reasonable enough
Operations is Dead, but Please Don't Replace it with DevOps
This is so damn spot on.
Functional silos (and a standalone DevOps team is a great example of one) decouple actions from responsibility. Functional silos allow people to ignore, or at least feel disconnected from, the consequences of their actions. DevOps is a cultural change that encourages, rewards and exposes people taking responsibility for what they do, and what is expected from them. As Werner Vogels from Amazon Web Services says, "you build it, you run it". So a "DevOps team" is a risky and ultimately doomed strategy. Sure there are some technical roles, specifically related to the enablement of DevOps as an approach and these roles and tools need to be filled and built. Self service platforms, collaboration and communication systems, tool chains for testing, deployment and operations are all necessary. Sure someone needs to deliver on that stuff. But those are specific technical deliverables and not DevOps. DevOps is about people, communication and collaboration. Organizations ignore that at their peril.
Universal Music Group adding audible "watermarks"
including on paid-for, losslessly-compressed digital audio music files:
Why isn't UMG's watermark talked about more? Maybe people think the audio quality problems are due to some kind of lossy compression, as I did, and ignore it completely, or blame the streaming service/distributor. The problem here is that the UMG watermark degrades the audio to about the equivalent of a 96 kbit MP3. My guess is that if consumers were informed about what is going on, they would care. Especially those who pay full retail price for digital downloads advertised as lossless audio.
lame audio drm media music umg universal watermarks noise consumer mp3
"Call Me Maybe: Carly Rae Jepsen and the Perils of Network Partitions"
Aphyr's epic RICON talk, exploring distributed-database failure modes through music. and what a lot of fail there is! Bottom line: CRDTs win
crdts data-structures storage ricon apyhr failures network partitions puns slides
Cloudera Impala 1.0: It's Here, It's Real, It's Already the Standard for SQL on Hadoop
we are proud to announce the first production drop of Impala, which reflects feedback from across the user community based on multiple types of real-world workloads. Just as a refresher, the main design principle behind Impala is complete integration with the Hadoop platform (jointly utilizing a single pool of storage, metadata model, security framework, and set of system resources). This integration allows Impala users to take advantage of the time-tested cost, flexibility, and scale advantages of Hadoop for interactive SQL queries, and makes SQL a first-class Hadoop citizen alongside MapReduce and other frameworks. The net result is that all your data becomes available for interactive analysis simultaneously with all other types of processing, with no ETL delays needed.
Along with some great benchmark numbers against Hive. nifty stuffcloudera impala sql querying etl olap hadoop analytics business-intelligence reports
Alex Feinberg's response to Damien Katz' anti-Dynamoish/pro-Couchbase blog post
Insightful response, worth bookmarking. (the original post is at http://damienkatz.net/2013/05/dynamo_sure_works_hard.html ).
while you are saving on read traffic (online reads only go to the master), you are now decreasing availability (contrary to your stated goal), and increasing system complexity. You also do hurt performance by requiring all writes and reads to be serialized through a single node: unless you plan to have a leader election whenever the node fails to meet a read SLA (which is going to result a disaster -- I am speaking from personal experience), you will have to accept that you're bottlenecked by a single node. With a Dynamo-style quorum (for either reads or writes), a single straggler will not reduce whole-cluster latency. The core point of Dynamo is low latency, availability and handling of all kinds of partitions: whether clean partitions (long term single node failures), transient failures (garbage collection pauses, slow disks, network blips, etc...), or even more complex dependent failures. The reality, of course, is that availability is neither the sole, nor the principal concern of every system. It's perfect fine to trade off availability for other goals -- you just need to be aware of that trade off.
cap distributed-databases databases quorum availability scalability damien-katz alex-feinberg partitions network dynamo riak voldemort couchbase
CAP Confusion: Problems with 'partition tolerance'
Another good clarification about CAP which resurfaced during last week's discussion:
So what causes partitions? Two things, really. The first is obvious – a network failure, for example due to a faulty switch, can cause the network to partition. The other is less obvious, but fits with the definition [...]: machine failures, either hard or soft. In an asynchronous network, i.e. one where processing a message could take unbounded time, it is impossible to distinguish between machine failures and lost messages. Therefore a single machine failure partitions it from the rest of the network. A correlated failure of several machines partitions them all from the network. Not being able to receive a message is the same as the network not delivering it. In the face of sufficiently many machine failures, it is still impossible to maintain availability and consistency, not because two writes may go to separate partitions, but because the failure of an entire 'quorum' of servers may render some recent writes unreadable.
(sorry, catching up on old interesting things posted last week...)failure scalability network partitions cap quorum distributed-databases fault-tolerance
Big-O Algorithm Complexity Cheat Sheet
nicely done, very readable
algorithms reference cheat-sheet big-o complexity estimation coding
Did Conroy's AFP filter wrongly block 1,200 sites?
Looks like many Aussie network operators were legally required to block 1,200 websites (presumably, one target and 1199 false positives), in secret. Quoting http://lists.ausnog.net/pipermail/ausnog/2013-April/017993.html : "You get a notice to block. You block or either get fined, go to jail or lose your carrier licence. It is a blunt instrument and it is a condition of being at 'the big boys table' i.e. you're a carrier or a carriage service provider."
australia law afp filtering internet blocking censorship secret eff
Making sense out of BDB-JE fast stats
good info on the system metrics recorded by BDB-JE's EnvironmentStats code, particularly where cache and cleaner activity are concerned. Particularly useful for Voldemort
voldemort caching bdb bdb-je storage tuning ops metrics reference
Approximate Heavy Hitters -The SpaceSaving Algorithm
nice, readable intro to SpaceSaving (which I've linked to before) -- a simple stream-processing cardinality top-K estimation algorithm with bounded error.
algorithms coding space-saving cardinality streams stream-processing estimation
Darach Ennis on CEP, Stream Processing, Messaging, OOP vs Functional Architecture
good interview -- lots of food for thought!
darach-ennis stream-processing messaging architecture qcon interviews erlang cep realtime rx comet events
One Year Later, the Results of Tor Books UK Going DRM-Free
As it is, we've seen no discernible increase in piracy on any of our titles, despite them being DRM-free for nearly a year.
Understanding Elastic Block Store Availability and Performance [slides]
fantastic in-depth presentation on EBS usage; lots of good advice here if you're using EBS volumes with/without PIOPS
piops ebs performance aws ec2 ops storage amazon presentations
-
Github get good results using Judy arrays to replace a Ruby hash. However: the whole blog post is a bit dodgy to me. It feels like there are much better ways to fix the problem: 1. the big one: don't do GC-heavy activity in the front-end web servers. Split that language-classification code into a separate service. Write its results to a cache and don't re-query needlessly. 2. why isn't this benchmarked against a C/C++ hash? it's only 36000 entries, loaded once at startup. lookups against that should be blisteringly fast even with the basic data structures, and that would also be outside the Ruby heap so avoid the GC overhead. Feels like the use of a Judy array was a "because I want to" decision. 3. personally, I'd have preferred they spend time fixing their uptime problems.... See also https://news.ycombinator.com/item?id=5639013 for more kvetching.
-
Mozilla's experience with Kanban. We've had good results in Amazon, too. good intro links in this post -- might start talking about it in Swrve...
Secret Bitcoin mining code added to game sparks outrage
Thunberg's admission that [the E-Sports Entertainment Association client software] ran Bitcoin-mining software without explicit user consent is startling. Aside from potentially opening the company up to huge legal liability, the move is likely to engender distrust among some of the company's most loyal fans. The nonchalance of some of Thunberg's comments may only add insult to the betrayal many users are likely to feel. "But for the record, I told jag he shouldn't be lazy and run the miner in a separate process," he wrote in a post, referring to one of his software engineers with the screen name Jaguar, who didn't take steps to conceal the Bitcoin miner. "Rookie move." In the later post he wrote: "100% of the funds are going into the s14 prize pot, so at the very least your melted gpus contributed to a good cause."
Gap's application of Knockout.js and the MVVM model
Interesting, first time I'd heard of it; the Model-View-View Model pattern.
mvvm architecture javascript web ui knockout-js martin-fowler json
-
very nice single-purpose site -- figure out who represents any given Irish postal address
Lectures in Advanced Data Structures (6.851)
Good lecture notes on the current state of the art in data structure research.
Data structures play a central role in modern computer science. You interact with data structures even more often than with algorithms (think Google, your mail server, and even your network routers). In addition, data structures are essential building blocks in obtaining efficient algorithms. This course covers major results and current directions of research in data structures: TIME TRAVEL We can remember the past efficiently (a technique called persistence), but in general it's difficult to change the past and see the outcomes on the present (retroactivity). So alas, Back To The Future isn't really possible. GEOMETRY When data has more than one dimension (e.g. maps, database tables). DYNAMIC OPTIMALITY Is there one binary search tree that's as good as all others? We still don't know, but we're close. MEMORY HIERARCHY Real computers have multiple levels of caches. We can optimize the number of cache misses, often without even knowing the size of the cache. HASHING Hashing is the most used data structure in computer science. And it's still an active area of research. INTEGERS Logarithmic time is too easy. By careful analysis of the information you're dealing with, you can often reduce the operation times substantially, sometimes even to constant. We will also cover lower bounds that illustrate when this is not possible. DYNAMIC GRAPHS A network link went down, or you just added or deleted a friend in a social network. We can still maintain essential information about the connectivity as it changes. STRINGS Searching for phrases in giant text (think Google or DNA). SUCCINCT Most "linear size" data structures you know are much larger than they need to be, often by an order of magnitude. Some data structures require almost no space beyond the raw data but are still fast (think heaps, but much cooler).
(via Tim Freeman)data-structures lectures mit video data algorithms coding csail strings integers hashing sorting bst memory
Older Is Wiser: Study Shows Software Developers' Skills Improve Over Time
At least in terms of StackOverflow rep:
For the first part of the study, the researchers compared the age of users with their reputation scores. They found that an individual's reputation increases with age, at least into a user's 40s. There wasn't enough data to draw meaningful conclusions for older programmers. The researchers then looked at the number of different subjects that users asked and answered questions about, which reflects the breadth of their programming interests. The researchers found that there is a sharp decline in the number of subjects users weighed in on between the ages of 15 and 30 – but that the range of subjects increased steadily through the programmers' 30s and into their early 50s. Finally, the researchers evaluated the knowledge of older programmers (ages 37 and older) compared to younger programmers (younger than 37) in regard to relatively recent technologies – meaning technologies that have been around for less than 10 years. For two smartphone operating systems, iOS and Windows Phone 7, the veteran programmers had a significant edge in knowledge over their younger counterparts. For every other technology, from Django to Silverlight, there was no statistically significant difference between older and younger programmers. "The data doesn't support the bias against older programmers – if anything, just the opposite," Murphy-Hill says.
Damn right ;)coding age studies software work stack-overflow ncsu knowledge skills life
-
Test Double is a generic term for any case where you replace a production object for testing purposes. There are various kinds of double that Gerard lists: Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists. Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an InMemoryTestDatabase is a good example). Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent. Mocks are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don't expect and are checked during verification to ensure they got all the calls they were expecting.
test-doubles naming patterns tdd testing mocking tests martin-fowler
Limerick-Tralee walking/cycling route blocked by farmers
Oh for god's sake. I know a few people who've made a trip to Mayo explicitly because the Greenway was there to visit. This is shocking, backwards stuff:
The success of [Mayo's] Great Western Greenway [trail] has overtaken that of others, such as the Great Southern Trail group, which has been working hard to install a walking and cycling route on sections of the former Limerick-Tralee railway line. On February 2nd, to mark the 50th anniversary of its closure, about 150 members and supporters of the Great Southern Trail set out from the old railway station at Abbeyfeale, Co Limerick, along the most recently developed section to cross the Kerry county boundary. The trailers were greeted by a barricade on the border, manned by more than 30 farmers, including the Listowel Fine Gael town councillor Denis Stack. A stand-off continued for three hours, with the Garda mediating in vain. The farmers were trying to lay claim to the land occupied by the disused railway line, even though Minister for Transport Leo Varadkar had made it clear that CIÉ "is the owner of the property [and] will object to any application by others to register these lands".
(via Rossa McMahon)via:rossamcmahon cycling walking hiking trails ireland kerry limerick listowel denis-stack cie
-
like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. [it] is written in portable C, and it has zero runtime dependencies. You can download a single binary, scp it to a far away machine, and expect it to work.
Nice tool. Needs to get into the Debian/Ubuntu apt repos pronto ;)
"Clickwrap" licensing established as legal in Irish court
"The evidence does establish that there is a practice in the airline and online travel agency sectors of contractually binding web users by click wrapping or browse wrapping, which practice is generally and regularly followed by the operators in those sectors. In reality, it is difficult to see how online trade could be carried on in the absence of those devices. As regards the third question which arises from the MSG decision, in this case it is whether the defendant was aware or is presumed to have been aware of the practice. The evidence before the Court, in my view, clearly demonstrates that the defendant was aware of the practice, it being a practice which is generally and regularly followed when making bookings with online travel agents and with airlines and which, in the words of the Court in the MSG case, may be regarded as being a consolidated practice. Accordingly, in my view, by application of Article 23(1)(c), the defendant is bound by the jurisdiction clause in the Terms of Use on the plaintiff's website by its use, either through the medium of an automaton or a manual operator or a third party data provider, of the website."
(via Rossa McMahon)
Functional Reactive Programming in the Netflix API with RxJava
Hmm, this seems nifty as a compositional building block for Java code to enable concurrency without thread-safety and sync problems.
Functional reactive programming offers efficient execution and composition by providing a collection of operators capable of filtering, selecting, transforming, combining and composing Observable's. The Observable data type can be thought of as a "push" equivalent to Iterable which is "pull". With an Iterable, the consumer pulls values from the producer and the thread blocks until those values arrive. By contrast with the Observable type, the producer pushes values to the consumer whenever values are available. This approach is more flexible, because values can arrive synchronously or asynchronously.
concurrency java jvm threads thread-safety coding rx frp fp functional-programming reactive functional async observable
You probably shouldn't use a spreadsheet for important work
Daniel Lemire comments on the recent cases of bugs in spreadsheets causing major impact:
There are several critical problems with a tool like Excel that need to be widely known: * Spreadsheets do not support testing. For anything that matters, you should validate and test your code automatically and systematically; * Spreadsheets make code reviews impractical. To visually inspect the code, you need to click and each and every cell. In practice, this means that you cannot reasonably ask someone to read over your formulas to make sure that there is no mistake; * Spreadsheets encourage redundancies. Spreadsheets encourage copy-and-paste. Though copying and pasting is sometimes the right tool, it also creates redundancies. These redundancies make it very difficult to update a spreadsheet: are you absolutely sure that you have changed the formula throughout?
Agreed on all three, particularly on the impossibility of testing. IMO, everyone who may be in a job where automation via spreadsheet is likely, needs training in SDE fundamentals: unit testing, the important of open source and open data for reproducibility, version control, and code review. We are all computer scientists now.spreadsheets excel coding errors bugs testability unit-testing testing quality sde sde-fundamentals dry
Log4j2 Asynchronous Loggers for Low-Latency Logging - Apache Log4j 2
implemented using the LMAX Disruptor library -- very impressive performance figures. I presume in real-world usage, these latencies are dwarfed by hardware costs, though
-
Google Drive and GMail have a built-in scripting engine. I had no idea
-
How the Irish media are partly to blame for the catastrophic property bubble, from a paper entitled _The Role Of The Media In Propping Up Ireland's Housing Bubble_, by Dr Julien Mercille, in the _Social Europe Journal_:
"The overall argument is that the Irish media are part and parcel of the political and corporate establishment, and as such the news they convey tend to reflect those sectors' interests and views. In particular, the Celtic Tiger years involved the financialisation of the economy and a large property bubble, all of it wrapped in an implicit neoliberal ideology. The media, embedded within this particular political economy and itself a constitutive element of it, thus mostly presented stories sustaining it. In particular, news organisations acquired direct stakes in an inflated real estate market by purchasing property websites and receiving vital advertising revenue from the real estate sector. Moreover, a number of their board members were current or former high officials in the finance industry and government, including banks deeply involved in the bubble's expansion."
economics irish-times ireland newspapers media elite insiders bubble property-bubble property celtic-tiger papers news bias
-
Ugh. low-end ISPs MITM'ing DNS queries:
Some ISP's are now using a technology called 'Transparent DNS proxy'. Using this technology, they will intercept all DNS lookup requests (TCP/UDP port 53) and transparently proxy the results. This effectively forces you to use their DNS service for all DNS lookups. If you have changed your DNS settings to an open DNS service such as Google, Comodo or OpenDNS expecting that your DNS traffic is no longer being sent to your ISP's DNS server, you may be surprised to find out that they are using transparent DNS proxying.
(via Nelson) BitTorrent's Secure Dropbox Alternative Goes Public
As kragen says, 'a decentralized way to sync a folder of large files, using BitTorrent instead of an untrustworthy central server'. Windows, OSX, and Linux supported
bittorrent dropbox cloud storage filesharing sharing sync synchronization
DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second
250 million tweets per day, 30-node HBase cluster, 400TB of storage, Kafka and 0mq. This is from 2011, hence this dated line: 'for a distributed application they thought AWS was too limited, especially in the network. AWS doesn't do well when nodes are connected together and they need to talk to each other. Not low enough latency network. Their customers care about latency.' (Nowadays, it would be damn hard to build a lower-latency network than that attached to a cc2.8xlarge instance.)
datasift architecture scalability data twitter firehose hbase kafka zeromq
Breaking the 1000 ms Time to Glass Mobile Barrier [slides]
Great presentation from Google on HTML5 CSS+JS render speed, 3G/4G network latency, etc. (via John G)
google slides 3g 4g lte networking telcos telecom css js html5 web via:jg
Lucene 4 - Revisiting Problems For Speed [slides]
a Presentation from Simon Willnauer on optimization work performed on Lucene in 2011. The most interesting stuff here is the work done to replace an O(n^2) FuzzyQuery fuzzy-match algorithm with a FSM trie is extremely cool -- benchmarked at 214 times faster!
benchmarks slides lucene search fuzzy-matching text-matching strings algorithms coding fsm tries
Microsoft Code Digger extension
Miguel de Icaza says it's witchcraft -- I'm inclined to agree:
Code Digger analyzes possible execution paths through your .NET code. The result is a table where each row shows a unique behavior of your code. The table helps you understand the behavior of the code, and it may also uncover hidden bugs. Through the new context menu item "Generate Inputs / Outputs Table" in the Visual Studio editor, you can invoke Code Digger to analyze your code. Code Digger computes and displays input-output pairs. Code Digger systematically hunts for bugs, exceptions, and assertion failures.
testing constraint-solving solver witchcraft magic dot-net coding tests code-digger microsoft
Swansea measles outbreak: was an MMR scare in the local press to blame?
Sixteen years ago, journalists had a much easier job assembling "balanced" stories about MMR in south Wales. When I wrote about the measles outbreak last week, I suggested that it was related to Andrew Wakefield's discredited 1998 Lancet research, but the Swansea contagion seems more likely to be the result of a separate scare a year earlier in the South Wales Evening Post. Before 1997, uptake of MMR in the distribution area of the Post was 91%, and 87.2% in the rest of Wales. After the Post's campaign, uptake in the distribution area fell to 77.4% (it was 86.8% in the rest of Wales). That's almost a 14% drop where the Post had influence, compared with less than 3% elsewhere. In the dry wording of the BMJ, "the [South West Evening Post] campaign is the most likely explanation". In other words, what we can see in Swansea is the local effect of local reporting' in all probability, just a taster of what happens when the news irresponsibly creates unfounded terror. [...] The 1997 coverage focused on a group of families who blamed MMR for various ailments in their children, including learning difficulties, digestive problems and autism' none of which have been found to have any connection with the vaccine. The Post's coverage was at the time deemed a success, and in 1998 it won a prize for investigative reporting in the BT Wales Press Awards. That year, the SWEP ran at least 39 stories related to the alleged dangers of MMR. And yes, it's true that the paper never directly endorsed non-vaccination. What it did do was publicise the idea of "vaccine damage" as a risk, one that parents would then likely weigh up against the risk of contracting measles, mumps or rubella. And this went beyond the reporting of parental anxieties' it was part of the Post's editorial line. One article is entitled "Young bodies cannot take it". The all-important "journalistic balance" was constantly available, thanks to campaigning parents and their solicitor Richard Barr. (It was Barr who engaged Wakefield for a lawsuit, leading to the "fishing expedition" research that became the Lancet paper.) They were happy to provide a quote on the dangers of the "triple jab", which health authorities were then obliged to rebut politely. The Post also seemed to downplay the risk of measles, reporting on 6 July 1998 that "not a single child has been hit by the illness' despite a 13% drop in take-up levels". It's not parents who should feel embarrassed by the Swansea measles outbreak: some may have acted from overt dread at the prospect of harming their child, and some simply from omission, but all were encouraged by a press that focused on non-existent risks and downplayed the genuine horror of the diseases MMR prevents. The shame belongs to journalists: those of the South West Evening Post who allowed themselves to be recruited in the service of a speculative lawsuit, and any who let a specious devotion to "balance" overrule a duty to tell the truth.
south-wales wales mmr health vaccination scares journalism ethics disease measles south-wales-evening-post
-
mostly a DynamoDB puff-piece from last week's Amazon Cloud Connect, but contains some good real-world figures for a 20-billion-GUID deduping table use-case at end. ($4,150 per month, to cut to the chase)
dynamodb aws figures costs architecture ec2 dedupe cloud-connect slides
Excel, untestability, and the reliability of quants
Wow, this is a great software-quality story -- I knew Excel was the most widely used programming environment out there, but this is a factor I'd overlooked:
In his remarks on the final panel, Frank Partnoy mentioned something I missed when it came out a few weeks ago: the role of Microsoft Excel in the "London Whale" trading debacle. [..] To summarize: JPMorgan's Chief Investment Office needed a new value-at-risk (VaR) model for the synthetic credit portfolio (the one that blew up) and assigned a quantitative whiz [...] to create it. The new model "operated through a series of Excel spreadsheets, which had to be completed manually, by a process of copying and pasting data from one spreadsheet to another." The internal Model Review Group identified this problem as well as a few others, but approved the model, while saying that it should be automated and another significant flaw should be fixed. After the London Whale trade blew up, the Model Review Group discovered that the model had not been automated and found several other errors. Most spectacularly, "After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR ..." I write periodically about the perils of bad software in the business world in general and the financial industry in particular, by which I usually mean back-end enterprise software that is poorly designed, insufficiently tested, and dangerously error-prone. But this is something different. [...] While Excel the program is reasonably robust, the spreadsheets that people create with Excel are incredibly fragile. There is no way to trace where your data come from, there's no audit trail (so you can overtype numbers and not know it), and there's no easy way to test spreadsheets, for starters. The biggest problem is that anyone can create Excel spreadsheets -- badly. Because it's so easy to use, the creation of even important spreadsheets is not restricted to people who understand programming and do it in a methodical, well-documented way. This is why the JPMorgan VaR model is the rule, not the exception: manual data entry, manual copy-and-paste, and formula errors. This is another important reason why you should pause whenever you hear that banks' quantitative experts are smarter than Einstein, or that sophisticated risk management technology can protect banks from blowing up. At the end of the day, it's all software. While all software breaks occasionally, Excel spreadsheets break all the time. But they don't tell you when they break: they just give you the wrong number.
excel reliability software coding ides jpmorgan value-at-risk finance london-whale quants spreadsheets unit-tests testability testing
Riak, CAP, and eventual consistency
Good (albeit draft) write-up of the implications of CAP, allow_mult, and last_write_wins conflict-resolution policies in Riak:
As Brewer's CAP theorem established, distributed systems have to make hard choices. Network partition is inevitable. Hardware failure is inevitable. When a partition occurs, a well-behaved system must choose its behavior from a spectrum of options ranging from "stop accepting any writes until the outage is resolved" (thus maintaining absolute consistency) to "allow any writes and worry about consistency later" (to maximize availability). Riak leans toward the availability end of the spectrum, but allows the operator and even the developer to tune read and write requests to better meet the business needs for any given set of data.
riak cap eventual-consistency distcomp distributed-systems partition last-write-wins voldemort allow_mult
How You Can Help Save Upcoming.org, Posterous, and More
Yahoo! sucks. shutting down in days? ArchiveTeam Warrior to the rescue; install the VM!
The Excel Depression - NYTimes.com
Krugman on the Reinhart-Rogoff Excel-bug fiasco.
What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But "economic research" showed no such thing; a couple of economists made that assertion, while many others disagreed. Policy makers abandoned the unemployed and turned to austerity because they wanted to, not because they had to. So will toppling Reinhart-Rogoff from its pedestal change anything? I'd like to think so. But I predict that the usual suspects will just find another dubious piece of economic analysis to canonize, and the depression will go on and on.
paul-krugman economics excel coding bugs software austerity debt
Vaccination 'herd immunity' demonstration
'Stochastic monte-carlo epidemic SIR model to reveal herd immunity'. Fantastic demo of this important medical concept (via Colin Whittaker)
via:colinwh stochastic herd-immunity random sir epidemics health immunity vaccination measles medicine monte-carlo-simulations simulations
Fred's ImageMagick Scripts: SIMILAR
compute an image-similarity metric, to discover mostly-identical-but-slightly-tweaked images:
SIMILAR computes the normalized cross correlation similarity metric between two equal dimensioned images. The normalized cross correlation metric measures how similar two images are, not how different they are. The range of ncc metric values is between 0 (dissimilar) and 1 (similar). If mode=g, then the two images will be converted to grayscale. If mode=rgb, then the two images first will be converted to colorspace=rgb. Next, the ncc similarity metric will be computed for each channel. Finally, they will be combined into an rms value.
(via Dan O'Neill)image photos pictures similar imagemagick via:dano metrics similarity
-
a first-person game prototype in which players navigate a 3D space while picking up orbs that reduce the speed of light in increments. Custom-built, open-source relativistic graphics code allows the speed of light in the game to approach the player's own maximum walking speed. Visual effects of special relativity gradually become apparent to the player, increasing the challenge of gameplay. These effects, rendered in realtime to vertex accuracy, include the Doppler effect (red- and blue-shifting of visible light, and the shifting of infrared and ultraviolet light into the visible spectrum); the searchlight effect (increased brightness in the direction of travel); time dilation (differences in the perceived passage of time from the player and the outside world); Lorentz transformation (warping of space at near-light speeds); and the runtime effect (the ability to see objects as they were in the past, due to the travel time of light). Players can choose to share their mastery and experience of the game through Twitter. A Slower Speed of Light combines accessible gameplay and a fantasy setting with theoretical and computational physics research to deliver an engaging and pedagogically rich experience.
Eventual Consistency Today: Limitations, Extensions, and Beyond - ACM Queue
Good overview of the current state of eventually-consistent data store research, covering CALM and CRDTs, from Peter Bailis and Ali Ghodsi
eventual-consistency data storage horizontal-scaling research distcomp distributed-systems via:martin-thompson crdts calm acid cap
Latency's Worst Nightmare: Performance Tuning Tips and Tricks [slides]
the basics of running a service stack (web, app servers, data stores) on AWS. some good benchmark figures in the final slides
benchmarks aws ec2 ebs piops services scaling scalability presentations
Rob "b3ta" Manuel in Dublin next week
The Bottom Half Of The Internet -- "Racism; typos; filth; spam; ignorance; rage – that's all the bottom half of the internet is good for, right? Rob Manuel wants you to question the internet dictum, most beloved of high-profile columnists, that you should ignore all of the comments all of the time. The 'war on comments', he reckons, might just be an echo of a fourth estate that's having trouble adjusting to the idea of an unwashed public disagreeing with their sacred opinions. Sous les pavés, la plage." On Tuesday, le cool Dublin & Pilcrow present SPIEL. Rob Manuel is the flashy animator behind B3ta and he's joined by Ed Melvin, who wants to educate you on 'The Unreal Engines' of virtual currencies and economies.
rob-manuel b3ta dublin comments internet meetings talks lecool
Reality, Reactivity, Relevance and Repeatability in Java Application Profiling
this product from JInspired appears to support runtime profiling of java apps with < 5% performance impact
You Lookin' At Me? Reflections on Google Glass
ex-Nokia product design guru Jan Chipchase on Google Glass
google privacy technology google-glass pervasive-computing life future
Not the 'best in the world' - The Medical Independent
Debunking this prolife talking point:
'Our maternity services are amongst the best in the world'. This phrase has been much hackneyed since the heartbreaking death of Savita Halappanavar was revealed in mid October. James Reilly and other senior politicians are particularly guilty of citing this inaccurate position. So what is the state of Irish maternity services and how do our figures compare with other comparable countries? Let's start with the statistics.
The bottom line:Eight deaths per 100,000 is not bad, but it ranks our maternity services far from the best in world and below countries such as Slovakia and Poland.
pro-choice ireland savita medicine health maternity morbidity statistics
How Kaggle Is Changing How We Work - Thomas Goetz - The Atlantic
Founded in 2010, Kaggle is an online platform for data-mining and predictive-modeling competitions. A company arranges with Kaggle to post a dump of data with a proposed problem, and the site's community of computer scientists and mathematicians -- known these days as data scientists -- take on the task, posting proposed solutions. [...] On one level, of course, Kaggle is just another spin on crowdsourcing, tapping the global brain to solve a big problem. That stuff has been around for a decade or more, at least back to Wikipedia (or farther back, Linux, etc). And companies like TaskRabbit and oDesk have thrown jobs to the crowd for several years. But I think Kaggle, and other online labor markets, represent more than that, and I'll offer two arguments. First, Kaggle doesn't incorporate work from all levels of proficiency, professionals to amateurs. Participants are experts, and they aren't working for benevolent reasons alone: they want to win, and they want to get better to improve their chances of winning next time. Second, Kaggle doesn't just create the incidental work product, it creates a new marketplace for work, a deeper disruption in a professional field. Unlike traditional temp labor, these aren't bottom of the totem pole jobs. Kagglers are on top. And that disruption is what will kill Joy's Law. Because here's the thing: the Kaggle ranking has become an essential metric in the world of data science. Employers like American Express and the New York Times have begun listing a Kaggle rank as an essential qualification in their help wanted ads for data scientists. It's not just a merit badge for the coders; it's a more significant, more valuable, indicator of capability than our traditional benchmarks for proficiency or expertise. In other words, your Ivy League diploma and IBM resume don't matter so much as my Kaggle score. It's flipping the resume, where your work is measurable and metricized and your value in the marketplace is more valuable than the place you work.
academia datamining economics data kaggle data-science ranking work competition crowdsourcing contracting
-
a good reference, with lots of sample output. Not clear if it takes 1.6/1.7 differences into account, though
Austerity policies founded on Excel typo
You've probably heard that countries with a high debt:GDP ratio suffer from slow economic growth. The specific number 90 percent has been invoked frequently. That's all thanks to a study conducted by Carmen Reinhardt and Kenneth Rogoff for their book This Time It's Different. But the results have been difficult for other researchers to replicate. Now three scholars at the University of Massachusetts have done so in "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff" and they find that the Reinhart/Rogoff result is based on opportunistic exclusion of Commonwealth data in the late-1940s, a debatable premise about how to weight the data, and most of all a sloppy Excel coding error. Read Mike Konczal for the whole rundown, but I'll just focus on the spreadsheet part. At one point they set cell L51 equal to AVERAGE(L30:L44) when the correct procuedure was AVERAGE(L30:L49). By typing wrong, they accidentally left Denmark, Canada, Belgium, Austria, and Australia out of the average. When you run the math correctly "the average real GDP growth rate for countries carrying a public debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0.1 percent."
austerity politics excel coding errors bugs spreadsheets economics economy
Is Your MySQL Buffer Pool Warm? Make It Sweat!
How GroupOn are warming up a failover warm MySQL spare, using Percona stuff and a "tee" of the live in-flight queries. (via Dave Doran)
via:dave-doran mysql databases warm-spares spares failover groupon percona replication
So now you know who gets some of those excessive Ticketmaster fees….
Interesting evidence; it appears Irish music promoters are getting "rebates" from the massive TicketMaster "booking fee", on each ticket sold. This sounds like a cartel to me, and we need to regulate this. Where is the National Consumer Agency and Competition Authority?
The matter is something which should be of concern to every gig-going music fan, regardless of whether they go to Stradbally or not. For years, many have asked about TicketMaster's quasi-monopoly position in the marketplace and why this is so. We've always been told that promoters preferred to deal with one company rather than several and that TM's systems and nationwide reach yadda yadda yadda was the bees' knees etc. Other companies have tried to compete but no-one has been able to beat TM at this game. But why would promoters go elsewhere when they're getting a slice of the TM fees back as rebates? Those past off-the-record attempts by and briefings from promoters blaming TM for those fees can now be seen as hypocritical. They're sticking with TM because they're receiving a take of the fees paid by punters who have no other choice in service provider if they want to get their hands on tickets. You wonder what the acts make of this cash-grab – perhaps some whip-smart agent is already making a claim for a percentage of the rebates because there would be no rebates in the first place without the act. Surely this is an issue for the Competition Authority and National Consumers Association too, given the manner in which the rebates are made and TM's deals with the promoters? While promoters under TM deals are free to sell a certain proportion of their tickets with another provider, it's usually only a very small percentage of the total and unlikely to trouble TM's bottom line. Also, given that the rebates are volume-driven, it's better for the promoters to keep the largest possible chunk of their business with TM. It seems that we have a new suspect in the blame game about why ticket prices are so high.
regulation ireland cartels competition ticketing tickets ticketmaster music gigs consumer
Blog shines spotlight on Dublin city's illegal dumping problem
Hooray, Eoin's activism gets some coverage!
THE SCALE OF Dublin's dumping problem is laid bare in a blog that has seen contributors send in photos of chairs, fridges and heaps of rubbish strewn on city streets. Eoin Parker, one of organisers behind DublinLitterBlog.com, spoke to TheJournal.ie about the problem, saying that the blog was set up following the privatisation of waste management by Dublin City Council in 2012.
dumping dublin litter rubbish blogs dcc d1 activism community
-
To our knowledge, Ked is the first scripting language to emerge from The People's Republic of Cork. Below is an account of what we know so far about the mysterious Corkonian language. Any suggested updates or contributions are encouraged.
Genius. Just how bad are RTE's finances?
A sobering examination by NAMAwinelake into the quagmire of Ireland's publicly-funded national broadcaster:
It seems that RTE has become a disaster zone, with libels and incompetence overseen by incapable management, and this is reflected in that organisation's financial results. RTE still employs nearly 2,000 people and supports jobs and industry across independent producers and suppliers; it is a major business. But the time has come to call a halt to delusional management that is sinking the organization deeper into a quagmire which will ultimately need to be bailed out by the State. And Noel Curran is fobbing us off with flying a kite about a reduction in 65-year old Pat Kenny's salary from €630,000 to €570,000?!
rte namawinelake public funding finances money mismanagement ireland incompetence tv news
High Scalability - Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
wow, Pinterest have a pretty hardcore architecture. Sharding to the max. This is scary stuff for me:
a [Cassandra-style] Cluster Management Algorithm is a SPOF. If there's a bug it impacts every node. This took them down 4 times.
yeah, so, eek ;)clustering sharding architecture aws scalability scaling pinterest via:matt-sergeant redis mysql memcached
Expert in Savita inquiry confirms Irish women get lower standard of care with chorioamnionitis
Dr. Jen Gunter again:
Dr. Knowles' testimony confirms for me that the law played a role, because her statements indicate the standard of care for treatment of chorioamnionitis is less aggressive in Ireland. This can only be because of the law as there is no medical evidence to support delaying delivery when chorioamnionitis is diagnosed. Standard of care is not to wait until a woman is sick enough to need a termination, the idea is to treat her, you know, before she gets sick enough. An elevated white count and ruptured membranes at 17 weeks is typically enough to make the diagnosis, so Dr. Knowles needs to testify as to what in Savita's medical record made it safe to not recommend a delivery. By the way, I also disagree with Dr. Knowles about her interpretation of Savita's medical record, the chart doesn't have "subtle indicators" of infection, it screams chorioamnionitis long before Wednesday morning. In North America the standard of care with chorioamnionitis is to recommend delivery as soon as the diagnosis is made, not wait until women enter the antechamber of death in the hopes that we can somehow snatch them back from the brink. If Irish law, or the interpretation thereof, had nothing to do with Savita's death no expert would be mentioning sick enough at all.
jen-gunter ob-gyn medicine savita law ireland abortion tragedy galway hospital
Boundary Product Update: Trends Dashboard Now Available
Boundary implement week-on-week trend display. Pity they use silly "giant number" dashboard boxes showing comparisons of the current datapoint with the previous week's datapoint; there's no indication of smoothing being applied, and "giant number" dashboards are basically useless anyway compared to a time-series graph, for unsmoothed time-series data. Also, no prediction bands. :(
boundary time-series tsd prediction metrics smoothing dataviz dashboards
ESB Networks | Power Check | Service Interruptions Map
real-time service outage information on a map, from Ireland's power network
Project Voldemort at Gilt Groupe: When Failure Isn't an Option [slides]
Geir Magnusson explains how Gilt Groupe is using Project Voldemort to scale out their e-commerce transactional system. The initial SQL solution had to be replaced because it could not handle the transactional spikes the site is experiencing daily due to its particular way of selling their inventory: each day at noon. Magnusson explains why they chose Voldemort and talks about the architecture.
via Filippovia:filippo database architecture nosql data voldemort gilt-groupe ops storage presentations
The full timeline of Savita Halappanavar's mistreatment
a comment on Dr. Jen Gunter's blog puts it all together
-
No holds barred:
Speaking today, spokesman Charles Stanley-Smith said; "This idea is insane. This area has suffered from dumping due to a lack of enforcement - yet the council now propose to effectively withdraw services altogether. As numerous studies such as 'the broken window hypothesis' indicate, where a small problem is left un-tackled it is likely to become far worse rather than better. In other words, rather than increase enforcement to solve the problem, Dublin City Council is going to remove enforcement. How will this deal with the problem? Imagine if that logic were applied to crime; would the removal of police services in an area help resolve criminal behaviour - or increase it? The answer is obvious."
an-taisce environment cleaning dublin ireland dcc rubbish trash society d1
-
Written by Google, this library is a flexible, efficient, and powerful Java client library for accessing any resource on the web via HTTP. It features a pluggable HTTP transport abstraction that allows any low-level library to be used, such as java.net.HttpURLConnection, Apache HTTP Client, or URL Fetch on Google App Engine. It also features efficient JSON and XML data models for parsing and serialization of HTTP response and request content. The JSON and XML libraries are also fully pluggable, including support for Jackson and Android's GSON libraries for JSON.
Not quite as simple an API as Python's requests, sadly, but still an improvement on the verbose Apache HttpComponent API. Good support for unit testing via a built-in mock-response class. Still in betagoogle beta software http libraries json xml transports protocols
Former IMF chief of mission to Ireland says not burning the bondholders was "a mistake"
Former IMF chief of mission to Ireland, Ashoka Mody, above left with Ajai Chopra in 2010. Melancholy of eye and large of loafer, Ashoka was involved in negotiating Ireland's EU/IMF bailout. [...] This morning Ashok gave an interview to Gavin Jennings on Morning Ireland, in which he admitted Ireland's bailout was riddled with mistakes, namely the non-burning of the senior bondholders and the program of austerity. Jennings: "So, if imposing austerity on Ireland was wrong, or a mistake; if not allowing any burning of bondholders, whether official, sovereign or private was a mistake; you were centrally involved in that program. I know Ajai Chopra was very much the public face of the IMF mission to Ireland. But you were centrally involved in constructing this bailout. How much responsibility do you take for those errors." Mody: "Yes, so, obviously, I have to take the responsibility in…but I'm in very good company in taking responsibility in this. There were many parties involved. And my role really was to bring such matters to the attention of people who finally made these decisions."
Great.bondholders imf ireland economy default ajai-chopra ashoka-mody
Savita Halappanavar's inquest: the three questions that must be answered | Dr. Jen Gunter
A professional OB/GYN analyses the horrors coming to light in the Savita inquest. Here's one particular gem:
Fetal survival with ruptured membranes at 17 weeks is 0%, this is from prospective study. [...but] "real and substantial risk" to the woman's life is what is required by the Irish constitution to terminate a pregnancy, *whether or not the foetus is viable*.
So the foetus had 0% chance of survival -- but still termination was not considered an option. Bloody hell.religion ireland savita horrors malpractice galway guh hospitals hse health inquest abortion pro-choice pregnancy
Minister Rabbitte welcomes EU agreement on re-use of Public Sector Information
Lots of talk about "charging regimes", "income-generating public sector bodies" etc., but not a single mention of open data or free access. Terrible stuff. :( (via conoro)
via:conoro open-access government public-sector ireland eu open-data public free
Compression in Kafka: GZIP or Snappy ?
With Ack: in this mode, as far as compression is concerned, the data gets compressed at the producer, decompressed and compressed on the broker before it sends the ack to the producer. The producer throughput with Snappy compression was roughly 22.3MB/s as compared to 8.9MB/s of the GZIP producer. Producer throughput is 150% higher with Snappy as compared to GZIP. No ack, similar to Kafka 0.7 behavior: In this mode, the data gets compressed at the producer and it doesn't wait for the ack from the broker. The producer throughput with Snappy compression was roughly 60.8MB/s as compared to 18.5MB/s of the GZIP producer. Producer throughput is 228% higher with Snappy as compared to GZIP. The higher compression savings in this test are due to the fact that the producer does not wait for the leader to re-compress and append the data; it simply compresses messages and fires away. Since Snappy has very high compression speed and low CPU usage, a single producer is able to compress the same amount of messages much faster as compared to GZIP.
The Bw-Tree: A B-tree for New Hardware - Microsoft Research
The emergence of new hardware and platforms has led to reconsideration of how data management systems are designed. However, certain basic functions such as key indexed access to records remain essential. While we exploit the common architectural layering of prior systems, we make radically new design decisions about each layer. Our new form of B tree, called the Bw-tree achieves its very high performance via a latch-free approach that effectively exploits the processor caches of modern multi-core chips. Our storage manager uses a unique form of log structuring that blurs the distinction between a page and a record store and works well with flash storage. This paper describes the architecture and algorithms for the Bw-tree, focusing on the main memory aspects. The paper includes results of our experiments that demonstrate that this fresh approach produces outstanding performance.
bw-trees database paper toread research algorithms microsoft sql sql-server b-trees data-structures storage cache-friendly mechanical-sympathy
Boundary Techtalk - Large-scale OLAP with Kobayashi
Boundary on their TSD-on-Riak store.
Dietrich Featherston, Engineer at Boundary, walks through the process of designing Kobayashi, the time-series analytics database behind our network metrics. He goes through the false-starts and lessons learned in effectively using Riak as the storage layer for a large-scale OLAP database. The system is ultimately capable of answering complex, ad-hoc queries at interactive latencies.
video boundary tsd riak eventual-consistency storage kobayashi olap time-series
-
A few days old, but already an instant Streisand-Effect classic:
Sometimes people borrow [Colin Purrington's free guide about making scientific posters] without giving him credit. This happens fairly regularly, and when he finds out about it, he sends an e-mail asking them to take it down. Usually they do. But when he sent an e-mail to the Consortium for Plant Biotechnology Research, asking that a roughly 1,200-word, near-verbatim, uncredited chunk from his guide be removed from the consortium's materials, the response was unexpected. Rather than apologise, a lawyer sent him a cease-and-desist letter accusing him of plagiarizing the consortium's materials and demanding that he take down his guide or face a lawsuit seeking damages up to $150,000.
streisand-effect lawsuits law infringement copyright cpbr bullying science posters
Kafka 0.8 Producer Performance
Great benchmarking from Piotr Kozikowski at the LiveRamp team, into performance of the upcoming Kafka 0.8 release
Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node
an excellent writeup on Kafka 0.8's use and operation, including details of the new replication features
-
'A €10 silver coin being offered for sale to the public in honour of James Joyce by the Central Bank tomorrow contains a misquote from the author. The line used on the coin from Chapter 3 of Ulysses includes a superfluous conjunction – a rogue 'that'.' [..] The coin reads:
"Ineluctable modality of the visible: at least that if no more, thought through my eyes. Signatures of all things *that* I am here to read."
(Incorrect 'that' emphasised)for:robotwisdom james-joyce typos funny fail central-bank ireland coins minting errors ulysses
Netflix ISP Speed Index for Ireland
Via Mulley. Magnet doing well, with UPC coming second; UPC have dropped a fair bit in the past month. Would love to see it broken down by region...
upc ireland isps speed bandwidth netflix broadband magnet eircom
Why I'm Walking Away From CouchDB
In practice there are two gotchas that are so painful I am looking for a replacement with a different featureset than couchdb provides. The location tracking project icecondor.com uses couchdb to store 20,000 new records per day. It has more write traffic than read traffic and runs on modest hardware. Those two gotchas are: 1. View Index updates. While I have a vague understanding of why view index updates are slow and bulky and important, in practice it is unworkable. Every write sets up a trap for the first reader to come along after the write. The more writes there are, the bigger the trap for the first reader which has to wait on the couchdb process that refreshes the view index on an as-needed basis. I believe this trade-off was made to keep writes fast. No need to update the view index until all writes are actually complete, right? Write traffic is heavier than read traffic and the time needed for that index refresh causes the webapp to crash because its not setup to handle timeouts from a database query. The workaround is as hackish as one can imagine - cron jobs to hit every map/reduce query to keep indexes fresh. 2. Append only database file Append only is in theory a great way to ensure on-disk reliability. A system crash during an append should only affect that append. Its a crash during an update to existing parts of the file that risks the integrity of more than whats being updated. With so many layers of caching and optimizations in the kernel and the filesystem and now in the workings of SSD drives, I'm not sure append-only gives extra protection anymore. What it does do is a create a huge operational headache. The on-disk file can never grow beyond half the available storage space. Record deletion uses new disk space and if the half-full mark approaches, vacuuming must be done. The entire database is rewritten to the filesystem, leaving out no longer needed records. If the data file should happen to grow beyond half the partition, the system has esentially crashed because there is no way to compact the file and soon the partition will be full. This is a likely scenario when there is a lot of record deletion activity. The system in question does a lot of writes of temporary data that is followed up by deletes a few days later. There is also a lot of permanent storage that hardly gets used. Rewriting every byte of the records that are long-lived due to compaction is an enormous amount of wasted I/O - doubly so given SSD drives have a short write-cycle lifespan.
nosql couchdb consistency checkpointing databases data-stores indexing
CouchDB: not drinking the kool-aid
Jonathan Ellis on some CouchDB negatives:
Here are some reasons you should think twice and do careful testing before using CouchDB in a non-toy project: Writes are serialized. Not serialized as in the isolation level, serialized as in there can only be one write active at a time. Want to spread writes across multiple disks? Sorry. CouchDB uses a MVCC model, which means that updates and deletes need to be compacted for the space to be made available to new writes. Just like PostgreSQL, only without the man-years of effort to make vacuum hurt less. CouchDB is simple. Gloriously simple. Why is that a negative? It's competing with systems (in the popular imagination, if not in its author's mind) that have been maturing for years. The reason PostgreSQL et al have those features is because people want them. And if you don't, you should at least ask a DBA with a few years of non-MySQL experience what you'll be missing. The majority of CouchDB fans don't appear to really understand what a good relational database gives them, just as a lot of PHP programmers don't get what the big deal is with namespaces. A special case of simplicity deserves mention: nontrivial queries must be created as a view with mapreduce. MapReduce is a great approach to trivially parallelizing certain classes of problem. The problem is, it's tedious and error-prone to write raw MapReduce code. This is why Google and Yahoo have both created high-level languages on top of it (Sawzall and Pig, respectively). Poor SQL; even with DSLs being the new hotness, people forget that SQL is one of the original domain-specific languages. It's a little verbose, and you might be bored with it, but it's much better than writing low-level mapreduce code.
cassandra couch nosql storage distributed databases consistency
What is the CouchDB replication protocol? Is it like Git? - Stack Overflow
Good write up of CouchDB replication
protocols couchdb sync replication git mvcc databases merging timelines
TouchDB's reverse-engineered write-up of the Couch replication protocol
There really isn't a separate "protocol" per se for replication. Instead, replication uses CouchDB's REST API and data model. It's therefore a bit difficult to talk about replication independently of the rest of CouchDB. In this document I'll focus on the algorithm used, and link to documentation of the APIs it invokes. The "protocol" is simply the set of those APIs operating over HTTP.
couchdb protocols touchdb nosql replication sync mvcc revisions rest
-
A good writeup of how to detect cases of copyright infringement for photography, art and other visual media.
Von Glitschka, Modern Dog and myriad others make clear that the support of the creative community is absolutely vital in raising awareness of copyright infringements. Sites like www.youthoughtwewouldntnotice.com name and shame clear breaches of copyright, while the Modern Dog case shows that there is no better IP tracing system than the eyes and ears of the design community itself. "It's the industry at large that has kept me aware of infringements," states Von. "Without that I would miss most of them because I don't go looking – they find me via the eyes of others."
photography art visual-media copyright infringement piracy ripping
FastBit: An Efficient Compressed Bitmap Index Technology
an [LGPL] open-source data processing library following the spirit of NoSQL movement. It offers a set of searching functions supported by compressed bitmap indexes. It treats user data in the column-oriented manner similar to well-known database management systems such as Sybase IQ, MonetDB, and Vertica. It is designed to accelerate user's data selection tasks without imposing undue requirements. In particular, the user data is NOT required to be under the control of FastBit software, which allows the user to continue to use their existing data analysis tools. The key technology underlying the FastBit software is a set of compressed bitmap indexes. In database systems, an index is a data structure to accelerate data accesses and reduce the query response time. Most of the commonly used indexes are variants of the B-tree, such as B+-tree and B*-tree. FastBit implements a set of alternative indexes called compressed bitmap indexes. Compared with B-tree variants, these indexes provide very efficient searching and retrieval operations, but are somewhat slower to update after a modification of an individual record. A key innovation in FastBit is the Word-Aligned Hybrid compression (WAH) for the bitmaps.[...] Another innovation in FastBit is the multi-level bitmap encoding methods.
fastbit nosql algorithms indexing search compressed-bitmaps indexes wah bitmaps compression
-
The bit array data structure is implemented in Java as the BitSet class. Unfortunately, this fails to scale without compression. JavaEWAH is a word-aligned compressed variant of the Java bitset class. It uses a 64-bit run-length encoding (RLE) compression scheme. We trade-off some compression for better processing speed. We also have a 32-bit version which compresses better, but is not as fast. In general, the goal of word-aligned compression is not to achieve the best compression, but rather to improve query processing time. Hence, we try to save CPU cycles, maybe at the expense of storage. However, the EWAH scheme we implemented is always more efficient storage-wise than an uncompressed bitmap (as implemented in the BitSet class). Unlike some alternatives, javaewah does not rely on a patented scheme.
javaewah wah rle compression bitmaps bitmap-indexes bitset algorithms data-structures
Measure Anything, Measure Everything « Code as Craft
the classic Etsy pro-metrics "measure everything" post. Some good basic rules and mindset
Testing Your Automation [slides]
Test-driven infrastructure, using Chef -- slides from Big Ruby 2013. Tools used: foodcritic (lol), Chefspec, minitest-chef-handler, fauxhai, cucumber chef. This is really good to see -- TDD applied to ops. Video at: http://confreaks.com/videos/2309-bigruby2013-testing-your-automation-ttd-for-chef-cookbooks
devops ops chef automation testing tdd infrastructure provisioning deployment
Meet the nice-guy lawyers who want $1,000 per worker for using scanners | Ars Technica
Great investigative journalism, interviewing the legal team behind the current big patent-troll shakedown; that on scanning documents with a button press, using a scanner attached to a network. They express whole-hearted belief in the legality of their actions, unsurprisingly -- they're exactly what you think they'd be like (via Nelson)
via:nelson ethics business legal patents swpats patent-trolls texas shakedown
[#HADOOP-9448] Reimplement things - ASF JIRA
Pretty good April Fools from this year -- a patch to delete the entirety of Hadoop's codebase:
To avoid any bias to the existing code and make the same mistakes we should just delete trunk completely. Attached it is a script that deletes everything.
Lucas Nussbaum's Blog » Blog Archive » RVM: seriously?
+1. RVM is atrocious code -- some of the worst bash script I've seen. And it's not just installing as a command, it requires that it be sourced and hooks into your login shell. If you then use "set -e", it crashes; "set -u", it crashes; reset $HOME, crash. It's dire.
-
Next April 11th, at the IIEA in North Gt Georges St:
Rick Falkvinge, founder of the Swedish Pirate Party, will examine the case for reform of copyright and patent law in the EU. Legalised file sharing, free sampling and shortened copyright protection times are the main elements of a proposal co-authored by Mr. Falkvinge which was submitted to the European Parliament in 2012. He will question whether, in the context of ever-increasing online activity, existing legal frameworks pose a threat to users' civil liberties.
rick-falkvinge pirate-party ireland iiea dublin copyright patents filesharing
High Performance MongoDB Clusters with Amazon EBS Provisioned IOPS
yeah yeah, Mongo. bookmarking for the good data on EBS+PIOPS
ebs piops aws performance tips ops ec2 mongodb presentations
-
These notes are intended to help users and system administrators maximize TCP/IP performance on their computer systems. They summarize all of the end-system (computer system) network tuning issues including a tutorial on TCP tuning, easy configuration checks for non-experts, and a repository of operating system specific instructions for getting the best possible network performance on these platforms.
Some tips for maximizing HPC network performance for the intra-DC case; recommended by the LinkedIn Kafka operations page. Increasing EBS Performance - Amazon Elastic Compute Cloud
good docs from EC2
-
an open source virtualized Ethernet networking stack. I am developing Snabb Switch in response to several exciting trends: x86 has risen to be a powerful networking platform. Virtualization and SDN are pulling more networking into servers. Optimized user-space software is out-performing kernel-space software. Snabb Switch's simple and fast software-only data plane makes developing networking software easier than ever before.
Written in LuaJIT but aiming to be very fast. cool stuff, worth watchingsdn software networking emulation snabb-switch luajit lua virtualization
Abusing hash kernels for wildly unprincipled machine learning
what, is this the first time our spam filtering approach of hashing a giant feature space is hitting mainstream machine learning? that can't be right!
ai machine-learning python data hashing features feature-selection anti-spam spamassassin
-
Joel On Software weighs in (via Tony Finch):
The fastest growing industry in the US right now, even during this time of slow economic growth, is probably the patent troll protection racket industry.
joel-on-software patents swpats shakedown extortion us-politics patent-trolls via:fanf
-
Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Cap'n Proto is INFINITY TIMES faster than Protocol Buffers.
Basically, marshalling like writing an aligned C struct to the wire, QNX messaging protocol-style. Wasteful on space, but responds to this by suggesting compression (which is a fair point tbh). C++-only for now. I'm not seeing the same kind of support for optional data that protobufs has though. Overall I'm worried there's some useful features being omitted here...serialization formats protobufs capn-proto protocols coding c++ rpc qnx messaging compression compatibility interoperability i14y
CRDTs - Commutative Replicated Data Types [pdf]
Shared read-only data is easy to scale by using well-understood replication techniques. However, sharing mutable data at a large scale is a dicult problem, because of the CAP impossibility result [5]. Two approaches dominate in practice. One ensures scalability by giving up consistency guarantees, for instance using the Last-Writer-Wins (LWW) approach [7]. The alternative guarantees consistency by serialising all updates, which does not scale beyond a small cluster [12]. Optimistic replication allows replicas to diverge, eventually resolving conflicts either by LWW-like methods or by serialisation [11]. In some (limited) cases, a radical simplication is possible. If concurrent updates to some datum commute, and all of its replicas execute all updates in causal order, then the replicas converge.1 We call this a Commutative Replicated Data Type (CRDT). The CRDT approach ensures that there are no conflicts, hence, no need for consensus-based concurrency control. CRDTs are not a universal solution, but, perhaps surprisingly, we were able to design highly useful CRDTs. This new research direction is promising as it ensures consistency in the large scale at a low cost, at least for some applications.
-
'The CRDT toolbox provides a collection of basic Conflict-free replicated data types as well as a common interface for defining your own CRDTs'. - in Eric Moritz' github. Also includes some more links to CRDT background reading.
Eventually-Consistent Data Structures [slides]
implementing CRDTs in Riak and Voldemort
-
What do you get if you take one accountant with "a fondness for spreadsheets, finance and business" and mix with "a life-long passion for video games"? Well it's obvious isn't it? A turn-based RPG made and played entirely in Microsoft Excel.
(via Paul Moloney)via:oceanclub arena.xlsm excel spreadsheets games gaming rpg
serverspec - unit tests for servers
With serverspec, you can write RSpec tests for checking your servers are provisioned correctly. Serverspec tests your servers' actual state through SSH access, so you don't need to install any agent softwares on your servers and can use any provisioning tools, Puppet, Chef, CFEngine and so on.
(via Dave Doran)via:dave-doran puppet testing chef cfengine unit-testing ops provisioning serverspec rspec ruby
joshua's blog: overclocking the lecture
Joshua's old tip on watching videos at 2x speed using Perian
quicktime video hacks mac speed lectures presentations learning
-
This seems pretty significant. Is the tide turning in the Texas Eastern District against patent trolls, at last? And does it establish sufficient precedent?
A federal judge has thrown out a patent claim against Rackspace, ruling that mathematical algorithms can't be patented. The ruling in the Eastern Disrict stemmed from a 2012 complaint filed by Uniloc USA asserting that processing of floating point numbers by the Linux operating system was a patent violation. Chief Judge Leonard Davis based the ruling on U.S. Supreme Court case law that prohibits the patenting of mathematical algorithms. According to Rackspace, this is the first reported instance in which the Eastern District of Texas has granted an early motion to dismiss finding a patent invalid because it claimed unpatentable subject matter. Red Hat, which supplies Linux to Rackspace, provided Rackspace's defense. Red Hat has a policy of standing behind customers through its Open Source Assurance program.
See https://news.ycombinator.com/item?id=5455869 for more discussion.east-texas patents swpats maths patenting law judges rackspace linux red-hat uniloc-usa floating-point
Introducing Chronos: A Replacement for Cron
A distributed, fault-tolerant "cron" is something which comes up frequently -- it makes for a great fault-tolerance building block. This one sounds like it's too closely tied into Mesos, though (IMO).
Chronos is our replacement for cron. It is a distributed and fault-tolerant scheduler which runs on top of Mesos. It's a framework and supports custom mesos executors as well as the default command executor. Thus by default, Chronos executes SH (on most systems BASH) scripts. Chronos can be used to interact with systems such as Hadoop (incl. EMR), even if the mesos slaves on which execution happens do not have Hadoop installed. Included wrapper scripts allow transfering files and executing them on a remote machine in the background and using asynchroneous callbacks to notify Chronos of job completion or failures.
cron scheduling mesos stacks design airbnb chronos fault-tolerance distcomp distributed-computing scripts jobs
One of CloudFlare's upstream providers on the "death of the internet" scare-mongering
Having a bad day on the Internet is nothing new. These are the types of events we deal with on a regular basis, and most large network operators are very good at responding quickly to deal with situations like this. In our case, we worked with Cloudflare to quickly identify the attack profile, rolled out global filters on our network to limit the attack traffic without adversely impacting legitimate users, and worked with our other partner networks (like NTT) to do the same. If the attacks had stopped here, nobody in the "mainstream media" would have noticed, and it would have been just another fun day for a few geeks on the Internet. The next part is where things got interesting, and is the part that nobody outside of extremely technical circles has actually bothered to try and understand yet. After attacking Cloudflare and their upstream Internet providers directly stopped having the desired effect, the attackers turned to any other interconnection point they could find, and stumbled upon Internet Exchange Points like LINX (in London), AMS-IX (in Amsterdam), and DEC-IX (in Frankfurt), three of the largest IXPs in the world. An IXP is an "interconnection fabric", or essentially just a large switched LAN, which acts as a common meeting point for different networks to connect and exchange traffic with each other. One downside to the way this architecture works is that there is a single big IP block used at each of these IXPs, where every network who interconnects is given 1 IP address, and this IP block CAN be globally routable. When the attackers stumbled upon this, probably by accident, it resulted in a lot of bogus traffic being injected into the IXP fabrics in an unusual way, until the IXP operators were able to work with everyone to make certain the IXP IP blocks weren't being globally re-advertised. Note that the vast majority of global Internet traffic does NOT travel over IXPs, but rather goes via direct private interconnections between specific networks. The IXP traffic represents more of the "long tail" of Internet traffic exchange, a larger number of smaller networks, which collectively still adds up to be a pretty big chunk of traffic. So, what you actually saw in this attack was a larger number of smaller networks being affected by something which was an completely unrelated and unintended side-effect of the actual attacks, and thus *poof* you have the recipe for a lot of people talking about it. :) Hopefully that clears up a bit of the situation.
21 graphs that show America's health-care prices are ludicrous
Excellent data, this. I'd heard a few of these prices, but these graphs really hit home. $26k for a caesarean section at the 95th percentile!? talk about out of control price gouging.
healthcare costs economics us-politics world comparison graphs charts data via:hn america
Design for developers [presentation]
A nice set of practical web/UI/tpyography design guidelines, naming specific sources (via Rob C)
-
'13 Security Gotchas You Should Know About'
Film4 Presents A Season Of Studio Ghibli Classics
hooray! Plenty of dubs, too, which is handy when you have little kids like mine ;)
The first pillar of agile sysadmin: We alert on what we draw
'One of [the] purposes of monitoring systems was to provide data to allow us, as engineers, to detect patterns, and predict issues before they become production impacting. In order to do this, we need to be capturing data and storing it somewhere which allows us to analyse it. If we care about it - if the data could provide the kind of engineering insight which helps us to understand our systems and give early warning - we should be capturing it. ' .... 'There are a couple of weaknesses in [Nagios' design]. Assuming we've agreed that if we care about a metric enough to want to alert on it then we should be gathering that data for analysis, and graphing it, then we already have the data upon which to base our check. Furthermore, this data is not on the machine we're monitoring, so our checks don't in any way add further stress to that machine.' I would add that if we are alerting on a different set of data from what we collect for graphing, then using the graphs to investigate an alarm may run into problems if they don't sync up.
devops monitoring deployment production sysadmin ops alerting metrics
JPL Institutional Coding Standard for the Java Programming Language
From JPL's Laboratory for Reliable Software (LaRS). Great reference; there's some really useful recommendations here, and good explanations of familiar ones like "prefer composition over inheritance". Many are supported by FindBugs, too. Here's the full list:
compile with checks turned on; apply static analysis; document public elements; write unit tests; use the standard naming conventions; do not override field or class names; make imports explicit; do not have cyclic package and class dependencies; obey the contract for equals(); define both equals() and hashCode(); define equals when adding fields; define equals with parameter type Object; do not use finalizers; do not implement the Cloneable interface; do not call nonfinal methods in constructors; select composition over inheritance; make fields private; do not use static mutable fields; declare immutable fields final; initialize fields before use; use assertions; use annotations; restrict method overloading; do not assign to parameters; do not return null arrays or collections; do not call System.exit; have one concept per line; use braces in control structures; do not have empty blocks; use breaks in switch statements; end switch statements with default; terminate if-else-if with else; restrict side effects in expressions; use named constants for non-trivial literals; make operator precedence explicit; do not use reference equality; use only short-circuit logic operators; do not use octal values; do not use floating point equality; use one result type in conditional expressions; do not use string concatenation operator in loops; do not drop exceptions; do not abruptly exit a finally block; use generics; use interfaces as types when available; use primitive types; do not remove literals from collections; restrict numeric conversions; program against data races; program against deadlocks; do not rely on the scheduler for synchronization; wait and notify safely; reduce code complexity
nasa java reference guidelines coding-standards jpl reliability software coding oo concurrency findbugs bugs
KDE's brush with git repository corruption: post-mortem
a barely-averted disaster... phew.
while we planned for the case of the server losing a disk or entirely biting the dust, or the total loss of the VM's filesystem, we didn't plan for the case of filesystem corruption, and the way the corruption affected our mirroring system triggered some very unforeseen and pathological conditions. [...] the corruption was perfectly mirrored... or rather, due to its nature, imperfectly mirrored. And all data on the anongit [mirrors] was lost.
One risk demonstrated: by trusting in mirroring, rather than a schedule of snapshot backups covering a wide time range, they nearly had a major outage. Silent data corruption, and code bugs, happen -- backups protect against this, but RAID, replication, and mirrors do not. Another risk: they didn't have a rate limit on project-deletion, which resulted in the "anongit" mirrors deleting their (safe) data copies in response to the upstream corruption. Rate limiting to sanity-check automated changes is vital. What they should have had in place was described by the fix: 'If a new projects file is generated and is more than 1% different than the previous file, the previous file is kept intact (at 1500 repositories, that means 15 repositories would have to be created or deleted in the span of three minutes, which is extremely unlikely).'rate-limiting case-studies post-mortems kde git data-corruption risks mirroring replication raid bugs backups snapshots sanity-checks automation ops
-
Metrics rule the roost -- I guess there's been a long history of telemetry in space applications.
To make software more visible, you need to know what it is doing, he said, which means creating "metrics on everything you can think of".... Those metrics should cover areas like performance, network utilization, CPU load, and so on. The metrics gathered, whether from testing or real-world use, should be stored as it is "incredibly valuable" to be able to go back through them, he said. For his systems, telemetry data is stored with the program metrics, as is the version of all of the code running so that everything can be reproduced if needed. SpaceX has programs to parse the metrics data and raise an alarm when "something goes bad". It is important to automate that, Rose said, because forcing a human to do it "would suck". The same programs run on the data whether it is generated from a developer's test, from a run on the spacecraft, or from a mission. Any failures should be seen as an opportunity to add new metrics. It takes a while to "get into the rhythm" of doing so, but it is "very useful". He likes to "geek out on error reporting", using tools like libSegFault and ftrace. Automation is important, and continuous integration is "very valuable", Rose said. He suggested building for every platform all of the time, even for "things you don't use any more". SpaceX does that and has found interesting problems when building unused code. Unit tests are run from the continuous integration system any time the code changes. "Everyone here has 100% unit test coverage", he joked, but running whatever tests are available, and creating new ones is useful. When he worked on video games, they had a test to just "warp" the character to random locations in a level and had it look in the four directions, which regularly found problems. "Automate process processes", he said. Things like coding standards, static analysis, spaces vs. tabs, or detecting the use of Emacs should be done automatically. SpaceX has a complicated process where changes cannot be made without tickets, code review, signoffs, and so forth, but all of that is checked automatically. If static analysis is part of the workflow, make it such that the code will not build unless it passes that analysis step. When the build fails, it should "fail loudly" with a "monitor that starts flashing red" and email to everyone on the team. When that happens, you should "respond immediately" to fix the problem. In his team, they have a full-size Justin Bieber cutout that gets placed facing the team member who broke the build. They found that "100% of software engineers don't like Justin Bieber", and will work quickly to fix the build problem.
spacex dev coding metrics deplyment production space justin-bieber
-
'the story of ketchup is a story of globalization and centuries of economic domination by a world superpower. But the superpower isn't America, and the century isn't ours. Ketchup's origins in the fermented sauces of China and Southeast Asia mean that those little plastic packets under the seat of your car are a direct result of Chinese and Asian domination of a single global world economy for most of the last millenium.'
ketchup china nam-pla food etymology condiments history trade
-
now this is a neat trick -- having been stuck having to flip to spares and do other antics while a long-running heap dump took place, this is a winner.
Dumping a JVM's heap is an extremely useful tool for debugging problems with a J2EE application. Unfortunately, when a JVM explodes, using the standard jmap tool can take an inordinate amount of time to execute for lots of different reasons. This leads to extended downtime when a heap dump is attempted and even then, jmap regularly fails. This blog post is intended to outline an alternate method using [gdb] to achieve a heap dump that only requires mere seconds of additional downtime allowing the slow jmap process to happen once the application is back in service.
heap-dump gdb heap jvm java via:peakscale gcore core core-dump debugging
-
'Edition has a 'design for life' philosophy - we think that unique designer-made items can be a part of our everyday lives without costing the earth. We stock affordable, contemporary and functional products (mostly handmade), including jewellery, home-ware, accessories, art and toys. Every item has been carefully selected and are all designed here in Ireland.'
BBC Test Card image (1080p HD version)
via colinwh. The de-facto standard HTPC desktop background
-
Neil Fraser visits a school in Vietnam, and investigates their computer science curriculum. They are doing an incredible job, it looks like -- very impressive!
vietnam programming education cs computer-science schools coding children
TOSEC: Commodore C64 (2012-04-23) : Free Download & Streaming : Internet Archive
A massive, 6.5GB collection of C64 history.
There are an astounding 134,000+ disk, cassette and documentation items in this Commodore 64 collection, including games, demos, cractros, and compilations.
By the numbers: How Google Compute Engine stacks up to Amazon EC2
Scalr's thoughts on Google's EC2 competitor.
with Google Compute Engine, AWS has a formidable new competitor in the public cloud space, and we'll likely be moving some of Scalr's production workloads from our hybrid aws-rackspace-softlayer setup to it when it leaves beta. There's a strong technical case for migrating heavy workloads to GCE, and I'll be grabbing popcorn to eagerly watch as the battle unfolds between the giants.
-
realtime collaboration API. nifty! but can it collaborate on a per-app shared doc, or does it require that the app user auth to Google and access their own docs?
Percona Playback's tcpdump plugin
Capture MySQL traffic via tcpdump, tee it over the network to replay against a second database. Even supports query execution times and pauses between queries to playback the same load level
tcpdump production load-testing testing staging tee networking netcat percona replay mysql
Riak CS is now ASL2 open source
'Organizations and users can now access the source code on Github and download the latest packages from the downloads page. Also, today, we announced that Riak CS Enterprise is now available as commercial licensed software, featuring multi-datacenter replication technology and 24×7 Basho customer support.'
riak riak-cs nosql storage basho open-source github apache asl2
Hadoop Operations at LinkedIn [slides]
another good Hadoop-at-scale presentation, from LI this time
Sift Science says it can sniff out cyber fraud — before it gets expensive
Great idea for a startup. This stuff is complex, right in the heart of every company's ordering pipeline, and I can see a lot of customers for this
sift-science anti-fraud fraud b2b b2c ecommerce startups aws
What would you do: Part 2, the Island of Surpyc
Amazing. 'Cyprus Bailout Choose Your Own Adventure', basically
cyoa adventure dice games cyprus politics eu bailouts ecb banking troika
Running the Largest Hadoop DFS Cluster
Facebook's 1PB Hadoop cluster. features improved NameNode availability work and 4 levels of data aging, with reduced replication and Reed-Solomon RAID encoding for colder data ages
aging data facebook hadoop hdfs reed-solomon error-correction replication erasure-coding
The America Invents Act: Fighting Patent Trolls With "Prior Art"
Don Marti makes some suggestions regarding the America Invents Act: record your work's timeline; use the new Post-Grant Challenging process; and use the new "prior user" defence, which lets you rely on your own non-public uses.
many of the best practices for tracking new versions of software and other digital assets can also help protect you against patent trolls. It's a good time to talk to your lawyer about a defensive strategy, and to connect that strategy to your version control and deployment systems to make sure you're collecting and retaining all of the information that could help you under this new law.
Announcing the Voldemort 1.3 Open Source Release
new release from LinkedIn -- better p90/p99 PUT performance, improvements to the BDB-JE storage layer, massively-improved rebalance performance
Data Corruption To Go: The Perils Of sql_mode = NULL « Code as Craft
bloody hell. A load of cases where MySQL will happily accommodate all sorts of malformed and invalid input -- thankfully with fixes
-
a high-performance C server which is used to expose bloom filters and operations over them to networked clients. It uses a simple ASCII protocol which is human readable, and similar to memcached.
(via Tony Finch) Thoughts on configuration file complexity
some interesting thoughts on the old "Turing complete configuration language" question
From a monolithic Ruby on Rails app to the JVM
How Soundcloud have ditched the monolithic Rails for nimbler, small-scale distributed polyglot services running on the JVM
soundcloud rails slides jvm scalability ruby scala clojure coding
Opinion: The Internet is a surveillance state
Bruce Schneier op-ed on CNN.com.
So, we're done. Welcome to a world where Google knows exactly what sort of porn you all like, and more about your interests than your spouse does. Welcome to a world where your cell phone company knows exactly where you are all the time. Welcome to the end of private conversations, because increasingly your conversations are conducted by e-mail, text, or social networking sites. And welcome to a world where all of this, and everything else that you do or is done on a computer, is saved, correlated, studied, passed around from company to company without your knowledge or consent; and where the government accesses it at will without a warrant. Welcome to an Internet without privacy, and we've ended up here with hardly a fight.
freedom surveillance legal privacy internet bruce-schneier web google facebook
Single Producer/Consumer lock free Queue step by step
great dissection of Martin "Disruptor" Thompson's lock-free single-producer/single-consumer queue data structure, with benchmark results showing crazy speedups. This is particularly useful since it's a data structure that can be used to provide good lock-free speedups without adopting the entire Disruptor design pattern.
disruptor coding java jvm martin-thompson lock-free volatile atomic queue data-structures
Roko's basilisk - RationalWiki
Wacky transhumanists.
Roko's basilisk is notable for being completely banned from discussion on LessWrong, where any mention of it is deleted. Eliezer Yudkowsky, founder of LessWrong, considers the basilisk would not work, but will not explain why because he does not consider open discussion of the notion of acausal trade with possible superintelligences to be provably safe. Silly over-extrapolations of local memes are posted to LessWrong quite a lot; almost all are just downvoted and ignored. But this one, Yudkowsky reacted to hugely, then doubled-down on his reaction. Thanks to the Streisand effect, discussion of the basilisk and the details of the affair soon spread outside of LessWrong. The entire affair is a worked example of spectacular failure at community management and at controlling purportedly dangerous information. Some people familiar with the LessWrong memeplex have suffered serious psychological distress after contemplating basilisk-like ideas — even when they're fairly sure intellectually that it's a silly problem.[5] The notion is taken sufficiently seriously by some LessWrong posters that they try to work out how to erase evidence of themselves so a future AI can't reconstruct a copy of them to torture.[6]
transhumanism funny insane stupid singularity ai rokos-basilisk via:maciej lesswrong rationalism superintelligences striesand-effect absurd
How the America Invents Act Will Change Patenting Forever
Bet you didn't think the US software patents situation could get worse? wrong!
"Now it's really important to be the first to file, and it's really important to file before somebody else puts a product out, or puts the invention in their product," says Barr, adding that it will "create a new urgency on the part of everyone to file faster -- and that's going to be a problem for the small inventor."
first-to-file omnishambles uspto swpats patents software-patents law legal
Distributed Systems Tracing with Zipkin
Twitter's version of the "canary"/"tracer" request concept
twitter zipkin tracing tracer-requests canary-requests http debugging production live distributed-systems distcomp stack infrastructure ops
Transitioning from Google Reader to feedly
xpecting for some time: We have been working on a project called Normandy which is a feedly clone of the Google Reader API – running on Google App Engine. When Google Reader shuts down, feedly will seamlessly transition to the Normandy back end.
Excellent stuff -- I've just tried feedly and it's looking good -- in fact it may be a better UI overall anyway.Double vision: seeing both sides of Syria's war
A skirmish is filmed, using HD video cameras, by both sides. Storyful pinpoint the location. War as panopticon
storyful war syria future tanks battle video youtube hd panopticon
Using DiffMerge as your Git visual merge and diff tool
A decent 3-way-diff GUI merge tool which works with git on OSX. "git config" command-lines included in this blog post
git merge osx mac macosx diff mergetool merging cli diffmerge
-
A bunch of magic command lines to set useful OS X prefs without pointy-clicky. at least some also seem to work on Mountain Lion
-
'bootstrap an OSX development machine with a one-liner'.
Many teams use chef to manage their production machines, but developers often build their development boxes by hand. SoloWizard makes it painless to create a configurable chef solo script to get your development machine humming: mysql, sublime text, .bash_profile tweaks to OS-X settings - it's all there!
osx chef mac build-out ops macosx deployment developers desktops laptops mysql rabbitmq activemq nginx
-
'Our results suggest that the Cablevision decision, [which was widely seen as easing certain ambiguities surrounding intellectual property], led to additional incremental investment in U.S. cloud computing firms that ranged from $728 million to approximately $1.3 billion over the two-and-a-half years after the decision. When paired with the findings of the enhanced effects of VC investment relative to corporate investment, this may be the equivalent of $2 to $5 billion in traditional R&D investment.' via Fred Logue.
via:fplogue law ip copyright policy cablevision funding vc cloud-computing investment legal buffering
A History Of Ireland In 100 Objects
Now free!
The Royal Irish Academy, the National Museum of Ireland, and The Irish Times are collaborating with the EU Presidency, the Department of Foreign Affairs and Trade and Adobe to bring you a gift of A History of Ireland in 100 objects 'from the people of Ireland to the people of the world' for St Patrick's Day. It is available as an interactive app for Apple iPhone and iPad, for most Android tablets and on the Kindle Fire, from our website, as well as associated app stores. You can also experience the book on your computer, smartphone or eReader by clicking on the 'eBook' button below. The gift is free to download until the end of March.
free st-patricks-day museum ireland history objects eu apps iphone ipad android books ebooks
First 5 Minutes Troubleshooting A Server
quite a good checklist of first steps for troubleshooting. Worth bookmarking for "dstat --top-io --top-bio" alone, which is an absolutely excellent tool and new to me
dstat server io disks hardware performance linux sysadmin ops troubleshooting checklists root-cause
-
you really know you've made it as an inept Irish politician when Panti Bliss gets dressed up in her most senatorial wig to take the mickey out of you
funny comedy fidelma-healy-eames politics ireland social-media inept youtube video
Confusion reigns over three "hijacked" ccTLDs
This kind of silliness is only likely to increase as the number of TLDs increases (and they become more trivial).
What seems to be happening here is that [two companies involved] have had some kind of dispute, and that as a result the registrants and the reputation of three countries' ccTLDs have been harmed. Very amateurish.
tlds domains via:fanf amateur-hour dns cctlds registrars adamsnames
-
interesting details about Riak's support for secondary indexes. Not quite SQL, but still more powerful than plain old K/V storage (via dehora)
via:dehora riak indexes storage nosql key-value-stores 2i range-queries
Metric Collection and Storage with Cassandra | DataStax
DataStax' documentation on how they store TSD data in Cass. Pretty generic
datastax nosql metrics analytics cassandra tsd time-series storage
Jeff Dean's list of "Numbers Everyone Should Know"
from a 2007 Google all-hands, the list of typical latency timings from ranging from an L1 cache reference (0.5 nanoseconds) to a CA->NL->CA IP round trip (150 milliseconds).
performance latencies google jeff-dean timing caches speed network zippy disks via:kellabyte
-
'a columnar storage format that supports nested data', from Twitter and Cloudera, encoded using Apache Thrift in a Dremel-based record shredding and assembly algorithm. Pretty crazy stuff:
We created Parquet to make the advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem. Parquet is built from the ground up with complex nested data structures in mind, and uses the record shredding and assembly algorithm described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces. Parquet is built to support very efficient compression and encoding schemes. Multiple projects have demonstrated the performance impact of applying the right compression and encoding scheme to the data. Parquet allows compression schemes to be specified on a per-column level, and is future-proofed to allow adding more encodings as they are invented and implemented. Parquet is built to be used by anyone. The Hadoop ecosystem is rich with data processing frameworks, and we are not interested in playing favorites. We believe that an efficient, well-implemented columnar storage substrate should be useful to all frameworks without the cost of extensive and difficult to set up dependencies.
twitter cloudera storage parquet dremel columns record-shredding hadoop marshalling columnar-storage compression data
Bunnie Huang's "Hacking the Xbox" now available as a free PDF
'No Starch Press and I have decided to release this free ebook version of Hacking the Xbox in honor of Aaron Swartz. As you read this book, I hope that you'll be reminded of how important freedom is to the hacking community and that you'll be inclined to support the causes that Aaron believed in. I agreed to release this book for free in part because Aaron's treatment by MIT is not unfamiliar to me. In this book, you will find the story of when I was an MIT graduate student, extracting security keys from the original Microsoft Xbox. You'll also read about the crushing disappointment of receiving a letter from MIT legal repudiating any association with my work, effectively leaving me on my own to face Microsoft. The difference was that the faculty of my lab, the AI laboratory, were outraged by this treatment. They openly defied MIT legal and vowed to publish my work as an official "AI Lab Memo," thereby granting me greater negotiating leverage with Microsoft. Microsoft, mindful of the potential backlash from the court of public opinion over suing a legitimate academic researcher, came to a civil understanding with me over the issue.' This is a classic text on hardware reverse-engineering and the freedom to tinker -- strongly recommended.
hacking bunnie-huang xbox free hardware drm freedom-to-tinker books reading mit microsoft history
Daemon Showdown: Upstart vs. Runit vs. Systemd vs. Circus vs. God
strangely, no mention of runit being total shite though
daemons runit upstart systemd supervisord circus god nannies processes unix crash-only-software linux ops
-
Clojure-style lazy functional collections (via QCon via Caro)
via:caro collections java functional lazy-loading lazy-computation lazy clojure
4 Things Java Programmers Can Learn from Clojure (without learning Clojure)
'1. Use immutable values; 2. Do no work in the constructor; 3. Program to small interfaces; 4. Represent computation, not the world'. Strongly agreed with #1, and the others look interesting too
Tactical Chat: How the U.S. Military Uses IRC to Wage War
Excellent stuff. Lessons to be learned from this: IRC has some key features that mean it can be useful in this case. 1. simple text, everything supports it, no fancy UI clients are necessary; 2. resilient against lossy/transient/low-bandwidth/high-latency networks; 3. standards-compliant and "battle-hardened" (so to speak); 4. open-source/non-proprietary.
Despite the U.S. military's massive spending each year on advanced communications technology, the use of simple text chat or tactical chat has outpaced other systems to become one of the most popular paths for communicating practical information on the battlefield. Though the use of text chat by the U.S. military first began in the early 1990s, in recent years tactical chat has evolved into a "primary 'comms' path, having supplanted voice communications as the primary means of common operational picture (COP) updating in support of situational awareness." An article from January 2012 in the Air Land Sea Bulletin describes the value of tactical chat as an effective and immediate communications method that is highly effective in distributed, intermittent, low bandwidth environments which is particularly important with "large numbers of distributed warfighters" who must "frequently jump onto and off of a network" and coordinate with other coalition partners. Text chat also provides "persistency in situational understanding between those leaving and those assuming command watch duties" enabling a persistent record of tactical decision making. A 2006 thesis from the Naval Postgraduate School states that internet relay chat (IRC) is one of the most widely used chat protocols for military command and control (C2). Software such as mIRC, a Windows-based chat client, or integrated systems in C2 equipment are used primarily in tactical conditions though efforts are underway to upgrade systems to newer protocols.
(via JK)via:jk war irc chat mirc us-military tactical-chat distcomp networking
-
Great neologism from Mick Fealty:
Familiar to anyone who's followed public debate on Northern Ireland. Some define it as the often multiple blaming and finger pointing that goes on between communities in conflict. Political differences are marked by powerful emotional (often tribal) reactions as opposed to creative conflict over policy and issues. It's beginning to be known well beyond the bounds of Northern Ireland. [...] Evasion may not be the intention but it is the obvious effect. It occurs when individuals are confronted with a difficult or uncomfortable question. The respondent retrenches his/her position and rejigs the question, being careful to pick open a sore point on the part of questioner's 'tribe'. He/she then fires the original query back at the inquirer.
words etymology whataboutery argument debate northern-ireland mick-fealty slugger-otoole
-
Give your app its own private Dropbox client and leave the syncing to us.
the real reason Marissa Mayer canned remote Y! employees (apparently)
After spending months frustrated at how empty Yahoo parking lots were, Mayer consulted Yahoo's VPN logs to see if remote employees were checking in enough. Mayer discovered they were not — and her decision was made. we're hearing from people close to Yahoo executives and employees that she made the right decision banning work from home. "The employees at Yahoo are thrilled," says one source close to the company. "There isn't massive uprising. The truth is, they've all been pissed off that people haven't been working."
yahoo work remote-work teleworking slacking marissa-mayer funny
Online Schema Change for MySQL
A tool written by Facebook to ease the pain of online MySQL schema-change migrations.
Some ALTER TABLE statements take too long form the perspective of some MySQL users. The fast index create feature for the InnoDB plugin in MySQL 5.1 makes this less of an issue but this can still take minutes to hours for a large table and for some MySQL deployments that is too long. A workaround is to perform the change on a slave first and then promote the slave to be the new master. But this requires a slave located near the master. MySQL 5.0 added support for triggers and some replication systems have been built using triggers to capture row changes. Why not use triggers for this? The openarkkit toolkit did just that with oak-online-alter-table. We have published our version of an online schema change utility (OnlineSchemaChange.php aka OSC).
facebook mysql sql schema database migrations ops alter-table
Netflix Queue: Data migration for a high volume web application
There will come a time in the life of most systems serving data, when there is a need to migrate data to [another] data store while maintaining or improving data consistency, latency and efficiency. This document explains the data migration technique we used at Netflix to migrate the user's queue data between two different distributed NoSQL storage systems [SimpleDB to Cassandra].
Monitoring Apache Hadoop, Cassandra and Zookeeper using Graphite and JMXTrans
nice enough, but a lot of moving parts. It would be nice to see a simpler ZK+Graphite setup using the 'mntr' verb
graphite monitoring ops zookeeper cassandra hadoop jmx jmxtrans graphs
RFC 6585 - Additional HTTP Status Codes
includes "429 Too Many Requests", for rate limits
Curator Framework: Reducing the Complexity of Building Distributed Systems | Marketing Technology
good +1 for using Netflix' Curator ZK client library
-
a high-level API that greatly simplifies using ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations. Some of the features are: Automatic connection management: There are potential error cases that require ZooKeeper clients to recreate a connection and/or retry operations. Curator automatically and transparently (mostly) handles these cases. Cleaner API: simplifies the raw ZooKeeper methods, events, etc.; provides a modern, fluent interface Recipe implementations (see Recipes): Leader election, Shared lock, Path cache and watcher, Distributed Queue, Distributed Priority Queue
zookeeper java netflix distcomp libraries oss open-source distributed
OscarGodson.js | What I Learned At Yammer
some pretty interesting lessons, it turns out: a 'take what you need' vacation policy means nobody takes vacations (unsurprising); Yammer actively work to avoid employee burnout (good idea); Yammer A/B test every feature; and Yammer mgmt try to let their devs work autonomously.
-
Some really cool-looking UNIX command line utils, packaged in Debian (and therefore in Ubuntu too). A few of these I've reimplemented separately, but it's always good to replace a hack with a more widely available "official" tool. Thanks, Joey Hess!
sponge: accept input, wait til EOF, then rewrite a file; chronic: runs a command quietly unless it fails; combine: combine the lines in two files using boolean operations; ifdata: get network interface info without parsing ifconfig output; ifne: run a program if the standard input is not empty; isutf8: check if a file or standard input is utf-8; lckdo: execute a program with a lock held; mispipe: pipe two commands, returning the exit status of the first; parallel: run multiple jobs at once; pee: tee standard input to pipes; sponge: soak up standard input and write to a file; ts: timestamp standard input; vidir: edit a directory in your text editor; vipe: insert a text editor into a pipe; zrun: automatically uncompress arguments to command
bash shell cli unix scripting via:peakscale joey-hess debian ubuntu tools command-line commands
Test-Driven Infrastructure with Chef
Interesting idea.
The book introduces "Infrastructure as Code," test-driven development, Chef, and cucumber-chef, and then proceeds to a simple example using Chef to provision a shared Linux server. The recipes for the server are developed test-first, demonstrating both the technique and the workflow.
tdd chef server provisioning build deploy linux coding ops sysadmin
Peek and poke in the age of Linux
Neat demo of using ptrace to inject into a running process, just like the good old days ;)
Some time ago I ran into a production issue where the init process (upstart) stopped behaving properly. Specifically, instead of spawning new processes, it deadlocked in a transitional state. [...] What's worse, upstart doesn't allow forcing a state transition and trying to manually create and send DBus events didn't help either. That meant the sane options we were left with were: restart the host (not desirable at all in that scenario); start the process manually and hope auto-respawn will not be needed. Of course there are also some insane options. Why not cheat like in the old times and just PEEK and POKE the process in the right places? The solution used at the time involved a very ugly script driving gdb which probably summoned satan in some edge cases. But edge cases were not hit and majority of hosts recovered without issues.
debugging memory linux upstart peek poke ptrace gdb processes hacks
The World Wide Web is Moving to AOL! | Brian Bailey
brilliant parody of those "we're so happy to be shutting down!" posts.
Don't worry, all of that hard work won't be wasted. The World Wide Web will remain accessible for 30 days, which will give you plenty of time to update your readers and customers. Each of you will also receive a 30-day free trial for AOL. Look for your CD in the mail soon. Even better, we've created an import tool to make it easy to migrate everything you've put on the web to American Online! The address will change, of course, but now it will be available to every AOL member. You may find that you don't need to bother, though. America Online already has groups and pages about almost every topic you can imagine. Take a look around first and you might save yourself a lot of time. There are only so many different ways to say that Citizen Kane was a good movie! We understand that not all of you will become AOL subscribers and not all web sites will move to the new platform. Just to be safe, be sure to print out all of your favorite pages before the end of the month.
Irish government attacked using 'MiniDuke' PDF malware
although I haven't seen a word of it in the Irish media yet -- wonder if the government have noticed?
Cyber criminals have targeted government officials in more than 20 countries, including Ireland and Romania, in a complex online assault seen rarely since the turn of the millennium. The attack, dubbed "MiniDuke" by researchers, has infected government computers as recently as this week in an attempt to steal geopolitical intelligence, according to security experts.
ireland malware attacks pdf security espionage romania miniduke
The MiniDuke Mystery: PDF 0-day Government Spy Assembler 0x29A Micro Backdoor - Securelist
By analysing the logs from the command servers, we have observed 59 unique victims in 23 countries: Belgium, Brazil, Bulgaria, Czech Republic, Georgia, Germany, Hungary, Ireland, Israel, Japan, Latvia, Lebanon, Lithuania, Montenegro, Portugal, Romania, Russian Federation, Slovenia, Spain, Turkey, Ukraine, United Kingdom and United States.
Romania believes rival nation behind MiniDuke cyber attack | Reuters
"It is a cyber attack ... pursued by an entity that has the characteristics of a state actor," [Romanian secret service] SRI spokesman Sorin Sava told Reuters [...]. "Our estimations show the attack is certainly relevant to Romania's national security taking into account the profile of the compromised entities." [...] In this case, computer experts say an attacker from the former Soviet Union could be more likely. "MiniDuke" in some ways resembles a banking fraud Trojan dubbed "TinBa" believed to have been created by Russian criminal hackers.
ireland malware attacks pdf security espionage romania miniduke
Compress data more densely with Zopfli - Google Developers Blog
New compressor from Google, gzip/zip-compatible, slower but slightly smaller results
Denominator: A Multi-Vendor Interface for DNS
the latest good stuff from Netflix.
Denominator is a portable Java library for manipulating DNS clouds. Denominator has pluggable back-ends, initially including AWS Route53, Neustar Ultra, DynECT, and a mock for testing. We also ship a command line version so it's easy for anyone to try it out. The reason we built Denominator is that we are working on multi-region failover and traffic sharing patterns to provide higher availability for the streaming service during regional outages caused by our own bugs and AWS issues. To do this we need to directly control the DNS configuration that routes users to each region and each zone. When we looked at the features and vendors in this space we found that we were already using AWS Route53, which has a nice API but is missing some advanced features; Neustar UltraDNS, which has a SOAP based API; and DynECT, which has a REST API that uses a quite different pseudo-transactional model. We couldn't find a Java based API that grouped together common set of capabilities that we are interested in, so we created one. The idea is that any feature that is supported by more than one vendor API is the highest common denominator, and that functionality can be switched between vendors as needed, or in the event of a DNS vendor outage.
-
Who knew? you can make a runnable JAR file!
There has long been a hack known in some circles, but not widely known, to make jars really executable, in the chmod +x sense. The hack takes advantage of the fact that jar files are zip files, and zip files allow arbitrary cruft to be prepended to the zip file itself (this is how self-extracting zip files work).
jars via:netflix shell java executable chmod zip hacks command-line cli
Two surgeons debate the use of cycle helmets
'I am a neurosurgeon and a cyclist, and I am also married to a dedicated cyclist. I wear a cycling helmet and encourage cyclists to wear one. I don't find that wearing one impedes me in any way. I am under no illusion that it will save me in the event of a high speed collision with a car or lorry (nothing will), but most cycling accidents aren't of the high-speed variety.' versus: 'I am a consultant Trauma orthopaedic surgeon working in Edinburgh and have many years of experience treating cyclists after serious road traffic, cycle sport and commuting cycle injuries. I believe there is no justification for helmet laws or promotional campaigns that portray cycling as a particularly 'dangerous' activity, or that make unfounded claims about the effectiveness of helmets. By reducing cycle use even slightly, helmet laws or promotion campaigns are likely to cause a significant net disbenefit to public health, regardless of the effectiveness or otherwise of helmets.' Generally a lot of sense on either side.
Storm and Hadoop: Convergence of Big-Data and Low-Latency Processing
Yahoo! are going big with Storm for their next-generation internal cloud platform: 'Yahoo! engineering teams are developing technologies to enable Storm applications and Hadoop applications to be hosted on a single cluster. • We have enhanced Storm to support Hadoop style security mechanism (including Kerberos authentication), and thus enable Storm applications authorized to access Hadoop datasets on HDFS and HBase. • Storm is being integrated into Hadoop YARN for resource management. Storm-on-YARN enables Storm applications to utilize the computation resources in our tens of thousands of Hadoop computation nodes. YARN is used to launch Storm application master (Nimbus) on demand, and enables Nimbus to request resources for Storm application slaves (Supervisors).'
yahoo yarn cloud-computing private-clouds big-data latency storm hadoop elastic-computing hbase
Trojan paralyses speed cameras in Moscow
what a coincidence! (via Tony Finch)
-
Basically, tweaking a few suboptimal sysctls to optimize for 802.11b/n; requires a Jailbroken IOS device. I'm surprised that Apple defaulted segment size to 512 to be honest, and disabling delayed ACKs sounds like it might be useful (see also http://www.stuartcheshire.org/papers/NagleDelayedAck/).
TCP optimizer modifies a few settings inside iOS, including increasing the TCP receive buffer from 131072 to 292000, disabling TCP delayed ACK's, allowing a maximum of 16 un-ACK'd packets instead of 8 and set the default packet size to 1460 instead of 512. These changes won't only speed up your YouTube videos, they'll also improve your internet connection's performance overall, including Wi-Fi network connectivity.
tcp performance tuning ios apple wifi wireless 802.11n sysctl ip
-
A study published in the Feb. 27 issue of the journal PLoS One links increased consumption of sugar with increased rates of diabetes by examining the data on sugar availability and the rate of diabetes in 175 countries over the past decade. And after accounting for many other factors, the researchers found that increased sugar in a population's food supply was linked to higher diabetes rates independent of rates of obesity. In other words, according to this study, obesity doesn't cause diabetes: sugar does. The study demonstrates this with the same level of confidence that linked cigarettes and lung cancer in the 1960s. As Rob Lustig, one of the study's authors and a pediatric endocrinologist at the University of California, San Francisco, said to me, "You could not enact a real-world study that would be more conclusive than this one."
nytimes health food via:fanf sugar eating diabetes papers medicine
-
Stoneybatter's not-for-profit art space needs contributions
art stoneybatter dublin d7 ireland fundit fundraising the-joinery
Are volatile reads really free?
Marc Brooker with some good test data:
It appears as though reads to volatile variables are not free in Java on x86, or at least on the tested setup. It's true that the difference isn't so huge (especially for the read-only case) that it'll make a difference in any but the more performance sensitive case, but that's a different statement from free.
-
'Watch Netflix USA, Hulu, Pandora, BBC iPlayer, and more in [sic] anywhere you live!' -- seems to use similar techniques to tunlr.net, looks like it works for my Netflix
Cassandra, Hive, and Hadoop: How We Picked Our Analytics Stack
reasonably good whole-stack performance testing and analysis; HBase, Riak, MongoDB, and Cassandra compared. Riak did pretty badly :(
riak mongodb cassandra hbase performance analytics hadoop hive big-data storage databases nosql
Big Data Analytics at Netflix. Interview with Christos Kalantzis and Jason Brown.
Good interview with the Cassandra guys at Netflix, and some top Mongo-bashing in the comments
cassandra netflix user-stories testimonials nosql storage ec2 mongodb
-
my favourite art of the moment. Thick, heavy layers of acrylic black and white paint, evoking the stormy Atlantic (brr). Gallery Bode, which showed this in Nuremberg in 2011, wrote the following at http://www.bode-galerie.de/en/exhibitions/schwarz_weiss :
Gallery Bode is pleased to constitute the cooperation with Werner Knaupp with an exhibition of a new workseries. The exhibition showcases artworks out of the series "Westmen Isles". [...] The journeys to Iceland are a background to the development of this new workseries. These paintings are telling of a forbidding nature. The beholder can't take a [safe] position but he is involved into the event which becomes comprehensible in a nearly physical way. These pictures of a overwhelming nature could be traced back to Knaupp's confrontation with the force of nature while his journeys. The experience of this force pushes the limits of human being and evokes primal fear. With the abdication of colours the artworks reach dynamic. This foots on the consistency of colour and on the changing between reality and abstraction. In an art historical view the new black and white paintings detached themselves from traditional landscape painting. Werner Knaupp implements the pure force of nature into pure painting, to visualise the force fields of nature. The beholder experiences with these artworks a nature without human dimension. In Werner Knaupp's Oeuvre the "Westmen Isles" paintings are a new expression of his examination with existential fundamental questions.
germany art painting werner-knaupp paintings monochrome sea iceland
Indymedia: It's time to move on
Our decision to curtail publishing on the Nottingham Indymedia site and call a meeting is an attempt to create a space for new ideas. We are not interested in continuing along the slow but certain path to total irrelevance but want to draw in new people and start off in new directions whilst remaining faithful to the underlying principles of Indymedia.
indymedia community communication web anonymity publishing left-wing
How to revert a faulty merge in git
omgwtf, this is pretty horrific.
#AltDevBlogADay » Latency Mitigation Strategies
John Carmack on the low-latency coding techniques used to support head mounted display devices.
Virtual reality (VR) is one of the most demanding human-in-the-loop applications from a latency standpoint. The latency between the physical movement of a user's head and updated photons from a head mounted display reaching their eyes is one of the most critical factors in providing a high quality experience. Human sensory systems can detect very small relative delays in parts of the visual or, especially, audio fields, but when absolute delays are below approximately 20 milliseconds they are generally imperceptible. Interactive 3D systems today typically have latencies that are several times that figure, but alternate configurations of the same hardware components can allow that target to be reached. A discussion of the sources of latency throughout a system follows, along with techniques for reducing the latency in the processing done on the host system.
head-mounted-display display ui latency vision coding john-carmack
Distributed Streams Algorithms for Sliding Windows [PDF]
'Massive data sets often arise as physically distributed, parallel data streams, and it is important to estimate various aggregates and statistics on the union of these streams. This paper presents algorithms for estimating aggregate functions over a "sliding window" of the N most recent data items in one or more streams. [...] Our results are obtained using a novel family of synopsis data structures called waves.'
waves papers streaming algorithms percentiles histogram distcomp distributed aggregation statistics estimation streams
good blog post on histogram-estimation stream processing algorithms
After reviewing several dozen papers, a score or so in depth, I identified two data structures that appear to enable us to answer these recency and frequency queries: exponential histograms (from "Maintaining Stream Statistics Over Sliding Windows" by Datar et al.) and waves (from "Distributed Streams Algorithms for Sliding Windows" by Gibbons and Tirthapura). Both of these data structures are used to solve the so-called counting problem, the problem of determining, with a bound on the relative error, the number of 1s in the last N units of time. In other words, the data structures are able to answer the question: how many 1s appeared in the last n units of time within a factor of Error (e.g., 50%). The algorithms are neat, so I'll present them briefly.
streams streaming stream-processing histograms percentiles estimation waves statistics algorithms
Timelike 2: everything fails all the time
Fantastic post on large-scale distributed load balancing strategies from @aphyr. Random and least-conns routing comes out on top in his simulation (although he hasn't yet tried Marc Brooker's two-randoms routing strategy)
via:hn routing distributed least-conns load-balancing round-robin distcomp networking scaling
Marc Brooker's "two-randoms" load balancing approach
Marc Brooker on this interesting load-balancing algorithm, including simulation results:
Using stale data for load balancing leads to a herd behavior, where requests will herd toward a previously quiet host for much longer than it takes to make that host very busy indeed. The next refresh of the cached load data will put the server high up the load list, and it will become quiet again. Then busy again as the next herd sees that it's quiet. Busy. Quiet. Busy. Quiet. And so on. One possible solution would be to give up on load balancing entirely, and just pick a host at random. Depending on the load factor, that can be a good approach. With many typical loads, though, picking a random host degrades latency and reduces throughput by wasting resources on servers which end up unlucky and quiet. The approach taken by the studies surveyed by Mitzenmacher is to try two hosts, and pick the one with the least load. This can be done directly (by querying the hosts) but also works surprisingly well on cached load data. [...] Best of 2 is good because it combines the best of both worlds: it uses real information about load to pick a host (unlike random), but rejects herd behavior much more strongly than the other two approaches.
Having seen what Marc has worked on, and written, inside Amazon, I'd take this very seriously... cool to see he is blogging externally too.algorithm load-balancing distcomp distributed two-randoms marc-brooker least-conns
Can regular expressions parse HTML?
'a summary of the main points: The "regular expressions" used by programmers have very little in common with the original notion of regularity in the context of formal language theory. Regular expressions (at least PCRE) can match all context-free languages. As such they can also match well-formed HTML and pretty much all other programming languages. Regular expressions can match at least some context-sensitive languages. Matching of regular expressions is NP-complete. As such you can solve any other NP problem using regular expressions.'
compsci regexps regular-expressions programming np-complete chomsky-grammar context-free languages
How to Create Application Shortcuts in Google Chrome for Mac
a rather hacky script is required. Ugh
hacks osx google chrome mac application-shortcuts site-specific-browsers
-
I couldn't remember the name for this design principle, so it's worth a bookmark to remind me in future... 'This refers to computer programs that handle failures by simply restarting, without attempting any sophisticated recovery. Correctly written components of crash-only software can microreboot to a known-good state without the help of a user. Since failure-handling and normal startup use the same methods, this can increase the chance that bugs in failure-handling code will be noticed.'
crashing crash-only-software design architecture coding software fault-tolerance erlang let-it-fail microreboot recovery autosave
Europe Is Warmer Than Canada Because of the Gulf Stream, Right? Not So Fast
The common tale—the one bandied around for more than a hundred years—goes something like this: Warm water flowing to the northeast out of the Gulf of Mexico—the Gulf Stream—cuts across the North Atlantic ocean, bringing extra energy to the Isles and driving up temperatures relative to the comparatively-frigid North Americas. The only problem with this simple explanation, say Stephen Riser and Susan Lozier in Scientific American, is that it doesn't actually account for the difference.
gulf-stream myths ireland europe science currents ocean temperature climate
Dear Prudence: My wife and I came from the same sperm donor
yes, really. Bloody hell
sperm-donor birth dear-prudence omgwtfbbq via:davewiner reproduction
-
from Twitter -- 'a cache for your big data. Even though memory is thousand times faster than SSD, network connected SSD-backed memory makes sense, if we design the system in a way that network latencies dominate over the SSD latencies by a large factor. To understand why network connected SSD makes sense, it is important to understand the role distributed memory plays in large-scale web architecture. In recent years, terabyte-scale, distributed, in-memory caches have become a fundamental building block of any web architecture. In-memory indexes, hash tables, key-value stores and caches are increasingly incorporated for scaling throughput and reducing latency of persistent storage systems. However, power consumption, operational complexity and single node DRAM cost make horizontally scaling this architecture challenging. The current cost of DRAM per server increases dramatically beyond approximately 150 GB, and power cost scales similarly as DRAM density increases. Fatcache extends a volatile, in-memory cache by incorporating SSD-backed storage.'
twitter ssd cache caching memcached memcache memory network storage
Passively Monitoring Network Round-Trip Times - Boundary
'how Boundary uses [TCP timestamps] to calculate round-trip times (RTTs) between any two hosts by passively monitoring TCP traffic flows, i.e., without actively launching ICMP echo requests (pings). The post is primarily an overview of this one aspect of TCP monitoring, it also outlines the mechanism we are using, and demonstrates its correctness.'
tcp boundary monitoring network ip passive-monitoring rtt timestamping
drug cartel-controlled mobile comms networks
"The Mexican military has recently broken up several secret telecommunications networks that were built and controlled by drug cartels so they could coordinate drug shipments, monitor their rivals and orchestrate attacks on the security forces. A network that was dismantled just last week provided cartel members with cellphone and radio communications across four northeastern states. The network had coverage along almost 500 miles of the Texas border and extended nearly another 500 miles into Mexico's interior. Soldiers seized 167 antennas, more than 150 repeaters and thousands of cellphones and radios that operated on the system. Some of the remote antennas and relay stations were powered with solar panels."
Heroku finds out that distributed queueing is hard
Stage 3 of the Rap Genius/Heroku blog drama. Summary (as far as I can tell): Heroku gave up on a fully-synchronised load-balancing setup ("intelligent routing"), since it didn't scale, in favour of randomised queue selection; they didn't sufficiently inform their customers, and metrics and docs were not updated to make this change public; the pessimal case became pretty damn pessimal; a customer eventually noticed and complained publicly, creating a public shit-storm. Comments: 1. this is why you monitor real HTTP request latency (scroll down for crazy graphs!). 2. include 90/99 percentiles to catch the "tail" of poorly-performing requests. 3. Load balancers are hard. http://aphyr.com/posts/277-timelike-a-network-simulator has more info on the intricacies of distributed load balancing -- worth a read.
heroku rap-genius via:hn networking distcomp distributed load-balancing ip queueing percentiles monitoring
-
10 particularly good -- actually helpful -- tips on using the Graphite metric graphing system
Literate Jenks Natural Breaks and How The Idea Of Code is Lost
A crazy amount of code archaeology to discover exactly an algorithm -- specifically 'Jenks natural breaks", works, after decades of cargo-cult copying (via Nelson): 'I spent a day reading the original text and decoding as much as possible of the code's intention, so that I could write a 'literate' implementation. My definition of literate is highly descriptive variable names, detailed and narrative comments, and straightforward code with no hijinks. So: yes, this isn't the first implementation of Jenks in Javascript. And it took me several times longer to do things this way than to just get the code working. But the sad and foreboding state of this algorithm's existing implementations said that to think critically about this code, its result, and possibilities for improvement, we need at least one version that's clear about what it's doing.'
jenks-natural-breaks algorithms chloropleth javascript reverse-engineering history software copyright via:nelson
don't order a Raspberry Pi from RS
I've been waiting 24 days for mine so far. Frankly amazing they are so apparently inept, particularly since it seems in breach of EU distance selling regulation if they go beyond 30 days without an update. They've just posted this:
Quick update- we received our delivery of raspberry pi's last week and as of Friday we had shipped up to order reference 1010239854. We will continue daily to get your orders shipped out as quickly as we possibly can; so that you will all receive your raspberry pi's shortly. Many thanks everyone for your patience and again apologies for the delay in the dispatch update message on the Pi Store which I know has caused some confusion.
rs raspberry-pi inept etailers uk e-commerce shopping hardware
more details on the UK distance selling regulations governing Raspberry Pi RS orders
'my understanding is that according to the Distance Selling Regulations [...], unless you agreed otherwise with RS, then they were obligated to fulfill their side of the contract within thirty days from the day after you ordered, and if they were unable to do so they were also obligated to inform you that they could not and repay you within thirty days;ons (more info here in a nice, easy-to-read format), unless you agreed otherwise with RS, then they were obligated to fulfill their side of the contract within thirty days from the day after you ordered, and if they were unable to do so they were also obligated to inform you that they could not and repay you within thirty days'
Sketch of the Day: HyperLogLog — Cornerstone of a Big Data Infrastructure
includes a nice javascript demo of HLL
hyperloglog loglog algorithms stream-processing streams estimation demos javascript
-
'log scale for lists; Decaying lists allow to manage large range of values. A decaying list grows logarithmically with the number of items. It follows that some items are dropped when other are inserted.' (via Tony Finch)
via:fanf clojure algorithms decay backoff half-life data-structures
Cycling in Dublin City: the numbers
7.6% of the Dublin commuter population "mainly cycle". some interesting stats here
-
Apache-licensed open source java lib to implement retrying behaviour cleanly.
a general purpose method for retrying arbitrary Java code with specific stop, retry, and exception handling capabilities that are enhanced by Guava's predicate matching. It also includes an exponential backoff WaitStrategy that might be useful for situations where more well-behaved service polling is preferred.
retries retrying resiliency fault-tolerance java open-source guava
-
Utilizing an iPhone/Android App known as "Talking Tom Cat", the tool has been transformed into a new media mouthpiece, addressing very specific particulars of the conflict that are glossed over by international media: alliances between MNLA and Ansar Dine, critiques of hypocrisy of the MUJAO factions, and ousting of corrupt politicians.
apps wtf politics talking-tom-cat bizarre tuareg africa via:neilmajor
-
Black hats steal code-signing keys from software whitelisting anti-malware firm. Pretty audacious
How did I do the Starwars Traceroute?
It is accomplished using many vrfs on 2 Cisco 1841s. For those less technical, VRFs are essentially private routing tables similar to a VPN. When a packet destined to 216.81.59.173 (AKA obiwan.scrye.net) hits my main gateway, I forward it onto the first VRF on the "ASIDE" router on 206.214.254.1. That router then has a specific route for 216.81.59.173 to 206.214.254.6, which resides on a different VRF on the "BSIDE" router. It then has a similar set up which points it at 206.214.254.9 which lives in another VPN on "ASIDE" router. All packets are returned using a default route pointing at the global routing table. This was by design so the packets TTL expiration did not have to return fully through the VRF Maze. I am a consultant to Epik Networks who let me use the Reverse DNS for an unused /24, and I used PowerDNS to update all of the entries through mysql. This took about 30 minutes to figure out how to do it, and about 90 minutes to implement.
Real-time Analytics in Scala [slides, PDF]
some good approximation/streaming algorithms and tips on Scala implementation
'E?cient Computation of Frequent and Top-k Elements in Data Streams' [paper, PDF]
The Space-Saving algorithm to compute top-k in a stream. I've been asking a variation of this problem as an interview question for a while now, pretty cool to find such a neat solution. Pity neither myself nor anyone I've interviewed has come up with it ;)
space-saving approximation streams stream-processing cep papers pdf algorithms
-
ASL-licensed open source library of stream-processing/approximation algorithms: count-min sketch, space-saving top-k, cardinality estimation, LogLog, HyperLogLog, MurmurHash, lookup3 hash, Bloom filters, q-digest, stochastic top-k
algorithms coding streams cep stream-processing approximation probabilistic space-saving top-k cardinality estimation bloom-filters q-digest loglog hyperloglog murmurhash lookup3
'Medians and Beyond: New Aggregation Techniques for Sensor Networks' [paper, PDF]
'We introduce Quantile Digest or q-digest, a novel data structure which provides provable guarantees on approximation error and maximum resource consumption. In more concrete terms, if the values returned by the sensors are integers in the range [1;n], then using q-digest we can answer quantile queries using message size m within an error of O(log(n)/m). We also outline how we can use q-digest to answer other queries such as range queries, most frequent items and histograms. Another notable property of q-digest is that in addition to the theoretical worst case bound error, the structure carries with itself an estimate of error for this particular query.'
q-digest algorithms streams approximation histograms median percentiles quantiles
Russia's anti-child-porn internet blocklist allegedly being used for general censorship
Allegedly being used to censor political and anti-corruption journalism, and a Russian wikipedia-like site for hosting an article about suicide
HyperLogLog++: Google's Take On Engineering HLL
Google and AggregateKnowledge's improvements to the HyperLogLog cardinality estimation algorithm
hyperloglog cardinality estimation streaming stream-processing cep
osx - Remap "Home" and "End" to beginning and end of line
in summary: ~/Library/KeyBindings/DefaultKeyBinding.dict. Thanks, Apple, this is stupid
mac keyboard bindings it-just-works compatibility ui rebinding
-
Hadoop, a batch-generated read-only Voldemort cluster, and an intriguing optimal-storage histogram bucketing algorithm:
The optimal histogram is computed using a random-restart hill climbing approximated algorithm. The algorithm has been shown very fast and accurate: we achieved 99% accuracy compared to an exact dynamic algorithm, with a speed increase of one factor. [...] The amount of information to serve in Voldemort for one year of BBVA's credit card transactions on Spain is 270 GB. The whole processing flow would run in 11 hours on a cluster of 24 "m1.large" instances. The whole infrastructure, including the EC2 instances needed to serve the resulting data would cost approximately $3500/month.
scalability scaling voldemort hadoop batch algorithms histograms statistics bucketing percentiles
-
'Splout is a scalable, open-source, easy-to-manage SQL big data view. Splout is to Hadoop + SQL what Voldemort or Elephant DB are to Hadoop + Key/Value. Splout serves a read-only, partitioned SQL view which is generated and indexed by Hadoop.' Some FAQs: 'What's the difference between Splout SQL and Dremel-like solutions such as BigQuery, Impala or Apache Drill? Splout SQL is not a "fast analytics" Dremel-like engine. It is more thought to be used for serving datasets under web / mobile high-throughput, many lookups, low-latency applications. Splout SQL is more like a NoSQL database in the sense that it has been thought for answering queries under sub-second latencies. It has been thought for performing queries that impact a very small subset of the data, not queries that analyze the whole dataset at once.'
splout sql big-data hadoop read-only scaling queries analytics
Goonwaffe Stories: A Guide For Newbies [PDF]
impressively high-quality newbie's guide from the Goonswarm Federation -- as themittani.com describes it, 'frankly a work of art: a 1950's Pulp Scifi magazine full of internet spaceships and sociopathy.'
eve-online space goonswarm gaming mmo pdf pulp science-fiction
Evasi0n Jailbreak's Userland Component
Good writeup of the exploit techniques used in the new iOS jailbreak.
Evasi0n is interesting because it escalates privileges and has full access to the system partition all without any memory corruption. It does this by exploiting the /var/db/timezone vulnerability to gain access to the root user's launchd socket. It then abuses launchd to load MobileFileIntegrity with an inserted codeless library, which is overriding MISValidateSignature to always return 0.
Programming Language Checklist
'You appear to be advocating a new: [ ] functional [ ] imperative [ ] object-oriented [ ] procedural [ ] stack-based [ ] "multi-paradigm" [ ] lazy [ ] eager [ ] statically-typed [ ] dynamically-typed [ ] pure [ ] impure [ ] non-hygienic [ ] visual [ ] beginner-friendly [ ] non-programmer-friendly [ ] completely incomprehensible programming language. Your language will not work. Here is why it will not work.'
Jetty-9 goes fast with Mechanical Sympathy
This is very cool! Applying Mechanical Sympathy optimization techniques to Jetty, specifically: "False sharing" on the BlockingArrayQueue data structure resolved; a new ArrayTernaryTrie data structure to improve header field storage, making it faster to build. look up, efficient on RAM, cheap to GC, and more cache-friendly than a traditional trie; and a branchless hex-to-byte conversion statement. The results are a 30%-faster microbenchmark on amd64, with 50% less Young Gen garbage collections. Lovely to see low-level infrastructure libs like Jetty getting this kind of optimization.
-
craft beer kegs for hire in Dublin, Sligo, Limerick and Galway. Needs more Metalman, of course ;)
A Continuous Packaging Pipeline
presentation describing some nice automation tools for packaging vendor code for deployment
deployment fosdem presentations slides debian deb fpm apt-get
-
a new C++ template library from Google which implements an in-memory B-Tree container type, suitable for use as a drop-in replacement for std::map, set, multimap and multiset. Lower memory use, and reportedly faster due to better cache-friendliness
c++ google data-structures containers b-trees stl map set open-source
Clairvoyant Squirrel: Large Scale Malicious Domain Classification
Storm-based service to detect malicious DNS domain usage from streaming pcap data in near-real-time. Uses string features in the DNS domain, along with randomness metrics using Markov analysis, combined with a Random Forest classifier, to achieve 98% precision at 10,000 matches/sec
storm distributed distcomp random-forest classifiers machine-learning anti-spam slides
"Security Engineering" now online in full
Ross Anderson says: 'I'm delighted to announce that my book Security Engineering – A Guide to Building Dependable Distributed Systems is now available free online in its entirety. You may download any or all of the chapters from the book's web page.'
security books reference coding software encryption ross-anderson
Slide Rule Calculations By Example
Harder than using a calculator, that's for sure
A Continuous Packaging Pipeline
presentation describing some nice automation tools for packaging vendor code for deployment
deployment fosdem presentations slides debian deb fpm apt-get
-
a new C++ template library from Google which implements an in-memory B-Tree container type, suitable for use as a drop-in replacement for std::map, set, multimap and multiset. Lower memory use, and reportedly faster due to better cache-friendliness
c++ google data-structures containers b-trees stl map set open-source
Clairvoyant Squirrel: Large Scale Malicious Domain Classification
Storm-based service to detect malicious DNS domain usage from streaming pcap data in near-real-time. Uses string features in the DNS domain, along with randomness metrics using Markov analysis, combined with a Random Forest classifier, to achieve 98% precision at 10,000 matches/sec
storm distributed distcomp random-forest classifiers machine-learning anti-spam slides
-
'Intel's Intelligent Platform Management Interface (IPMI), which is implemented and added onto by all server vendors, grant system administrators with a means to manage their hardware in an Out of Band (OOB) or Lights Out Management (LOM) fashion. However there are a series of design, utilization, and vendor issues that cause complex, pervasive, and serious security infrastructure problems. The BMC is an embedded computer on the motherboard that implements IPMI; it enjoys an asymmetrical relationship with its host, with the BMC able to gain full control of memory and I/O, while the server is both blind and impotent against the BMC. Compromised servers have full access to the private IPMI network The BMC uses reusable passwords that are infrequently changed, widely shared among servers, and stored in clear text in its storage. The passwords may be disclosed with an attack on the server, over the network network against the BMC, or with a physical attack against the motherboard (including after the server has been decommissioned.) IT's reliance on IPMI to reduce costs, the near-complete lack of research, 3rd party products, or vendor documentation on IPMI and the BMC security, and the permanent nature of the BMC on the motherboard make it currently very difficult to defend, fix or remediate against these issues.' (via Tony Finch)
via:fanf security ipmi power-management hardware intel passwords bios
-
Massive Java concurrency fail in recent 1.6 and 1.7 JDK releases -- the java.util.HashMap type now spin-locks on an AtomicLong in its constructor. Here's the response from the author: 'I'll acknowledge right up front that the initialization of hashSeed is a bottleneck but it is not one we expected to be a problem since it only happens once per Hash Map instance. For this code to be a bottleneck you would have to be creating hundreds or thousands of hash maps per second. This is certainly not typical. Is there really a valid reason for your application to be doing this? How long do these hash maps live?' Oh dear. Assumptions of "typical" like this are not how you design a fundamental data structure. fail. For now there is a hacky reflection-based workaround, but this is lame and needs to be fixed as soon as possible. (Via cscotta)
java hashmap concurrency bugs fail security hashing jdk via:cscotta
High Scalability - geo-aware traffic load balancing and caching at CNBC.com
Dyn's anycast DNS service, as used by CNBC.com
anycast dns scalability dyn failover geographical load-balancing
Using Statsd and Graphite From a Rails App
Reasonable simple, from the looks of it
The colour of London's commute
Nice visualisation. 'What the map shows is the mix of transport to work of residents living in each part of London*, using ONS data at Middle Super Output Area (MSOA) level. Each MSOA is given an RGB colour determined by the modal share, with red colours representing travel by car, taxi or motorbike, blue travel by public transport and green cycling or walking. The result is a fairly simple pattern, with motor vehicles predominating on London's fringes, public transport in the inner suburbs and cycling and walking in the very centre. Those tendrils of blue reaching out presumably represent major public transport links.'
Where are the free WiFi spots in Dublin City Centre?
hooray, free wifi! beautiful Invader-style pixel-art mosaics to highlight them, too. nice one Joe
-
some lovely pixel art to advertise the free wifi areas, by Craig Robinson. I see a girl in pyjamas, a Dub hurler, a viking, Molly Malone, Phil Lynott, Oscar Wilde, a Moore St market trader, a busker, and the Spire...
DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
thumbs-up for DNSMadeEasy's Global Traffic Director anycast-based geographically-segmented DNS service, in particular
dns architecture scalability search duckduckgo geoip anycast
Announcing Ribbon: Tying the Netflix Mid-Tier Services Together
Netflix' load balancing client-side library, open source
netflix load-balancing coding libraries client-side open-source
-
'an expressive toolset for constructing scalable, resilient [service] architectures. It works in the cloud, in the data center, and on your laptop, and it makes your system diagram visible and inevitable. Inevitable systems coordinate automatically to interconnect, removing the hassle of manual configuration of connection points (and the associated danger of human error).' Looks like a pretty neat cluster deployment tool; driven from a single configuration file, using Chef, integrating closely with AWS and providing many useful additional features
chef deployment clusters knife services aws ec2 ops ironfan demo
Fox DMCA Takedowns Order Google to Remove Fox DMCA Takedowns
Chilling Effects is setup to stop the 'chilling effects' of Internet censorship. Google sees this as a good thing and sends takedown requests it receives to be added to the database. Fox sends takedown requests to Google for pages which the company says contain links to material it holds the copyright to. Those pages include those on Chilling Effects which show which links Fox wants taken down. Google delists the Chilling Effects pages from its search engine, thus completing the circle and defeating the very reason Chilling Effects was set up for in the first place.
-
At Railscamp X it became clear there is a gap in the current HTTP specification. There are many ways for a developer to screw up their implementation, but no code to share the nature of the error with the end user. We humbly suggest the following status codes are included in the HTTP spec in the 7XX range.
Includes such useful status codes as "724 - This line should be unreachable". How Newegg crushed the "shopping cart" patent and saved online retail
Very cool account of Newegg's battle against a ludicrous patent-troll shakedown. Great quote from their Chief Legal Officer, Lee Cheng:
Patent trolling is based upon deficiencies in a critical, but underdeveloped, area of the law. The faster we drive these cases to verdict, and through appeal, and also get legislative reform on track, the faster our economy will be competitive in this critical area. We're competing with other economies that are not burdened with this type of litigation. China doesn't have this, South Korea doesn't have this, Europe doesn't have this. [...] It's actually surprising how quickly people forget what Lemelson did. [referring to Jerome Lemelson, an infamous patent troll who used so-called "submarine patents" to make billions in licensing fees.] This activity is very similar. Trolls right now "submarine" as well. They use timing, like he used timing. Then they pop up and say "Hello, surprise! Give us your money or we will shut you down!" Screw them. Seriously, screw them. You can quote me on that.
patent-trolls east-texas newegg shopping-cart swpat software-patents patents ecommerce soverain
Implementing strcmp, strlen, and strstr using SSE 4.2 instructions - strchr.com
Using new Intel Core i7 instructions to speed up string manipulation.
Fascinating stuff. SSE ftwsse optimization simd assembly intel i7 intel-core strstr strings string-matching strchr strlen coding
All polar bears descended from one Irish grizzly
'THE ARCTIC'S DWINDLING POPULATION of polar bears all descend from a single mamma brown bear which lived 20,000 to 50,000 years ago in present-day Ireland, new research suggests. DNA samples from the great white carnivores - taken from across their entire range in Russia, Canada, Greenland, Norway and Alaska - revealed that every individual's lineage could be traced back to this Irish forebear.' More than the average bear, I guess
animals biology science dna history ireland bears polar-bears grizzly-bears via:ben
Basho | Alert Logic Relies on Riak to Support Rapid Growth
'The new [Riak-based] analytics infrastructure performs statistical and correlation processing on all data [...] approximately 5 TB/day. All of this data is processed in real-time as it streams in. [...] Alert Logic's analytics infrastructure, powered by Riak, achieves performance results of up to 35k operations/second across each node in the cluster – performance that eclipses the existing MySQL deployment by a large margin on single node performance. In real business terms, the initial deployment of the combination of Riak and the analytic infrastructure has allowed Alert Logic to process in real-time 7,500 reports, which previously took 12 hours of dedicated processing every night.' Twitter discussion here: https://twitter.com/fisherpk/status/294984960849367040 , which notes 'heavily cached SAN storage, 12 core blades and 90% get to put ops', and '3 riak nodes, 12-cores, 30k get heavy riak ops/sec. 8 nodes driving ops to that cluster'. Apparently the use of SAN storage on all nodes is historic, but certainly seems to have produced good iops numbers as an (expensive) side-effect...
iops riak basho ops systems alert-logic storage nosql databases
Turn a Raspberry Pi Into an AirPlay Receiver for Streaming Music in Your Living Room
hooray, a viable domestic Raspberry Pi use case at last ;)
Antigua Government Set to Launch "Pirate" Website To Punish United States
oh the lulz.
The Government of Antigua is planning to launch a website selling movies, music and software, without paying U.S. copyright holders. The Caribbean island is taking the unprecedented step because the United States refuses to lift a trade "blockade" preventing the island from offering Internet gambling services, despite several WTO decisions in Antigua's favor. The country now hopes to recoup some of the lost income through a WTO approved "warez" site.
us-politics antigua piracy filesharing pirate gambling wto ip blockades
-
An article by Nathan "Storm" Marz describing the system architecture he's been talking about for a while; Hadoop-driven batch view, Storm-driven "speed view", and a merging API
storm systems architecture lambda-architecture design Hadoop
Network graph viz of Irish politicians and organisations on Twitter
generated by the Clique Research Cluster at UCD and DERI. 'a visualization of the unified graph representation for the users in the data, produced using Gephi and sigma.js. Users are coloured according to their community (i.e. political affiliation). The size of each node is proportional to its in-degree (i.e. number of incoming links).' sigma.js provides a really user-friendly UI to the graphs, although -- as with most current graph visualisations -- it'd be particularly nice if it was possible to 'tease out' and focus on interesting nodes, and get a pasteable URL of the result, in context. Still, the most usable graph viz I've seen in a while...
graphs dataviz ucd research ireland twitter networks community sigma.js javascript canvas gephi
-
Incredible blog of book covers and illustrations, much from the 1970s
Namazu-e: Earthquake catfish prints
'In November 1855, the Great Ansei Earthquake struck the city of Edo (now Tokyo), claiming 7,000 lives and inflicting widespread damage. Within days, a new type of color woodblock print known as namazu-e (lit. "catfish pictures") became popular among the residents of the shaken city. These prints featured depictions of mythical giant catfish (namazu) who, according to popular legend, caused earthquakes by thrashing about in their underground lairs. In addition to providing humor and social commentary, many prints claimed to offer protection from future earthquakes.'
Implementing Real-Time Trending Topics With a Distributed Rolling Count Algorithm in Storm
Storm demo with a reasonably complex topology. 'how to implement a distributed, real-time trending topics algorithm in Storm. It uses the latest features available in Storm 0.8 (namely tick tuples) and should be a good starting point for anyone trying to implement such an algorithm for their own application. The new code is now available in the official storm-starter repository, so feel free to take a deeper look.'
-
'a UNIX init scheme with service supervision' - philosophically similar to daemontools, widely packaged, LSB init.d-script-compliant, BSD-licensed
'The Uni?ed Logging Infrastructure for Data Analytics at Twitter' [PDF]
A picture of how Twitter standardized their internal service event logging formats to allow batch analysis and analytics. They surface service metrics to dashboards from Pig jobs on a daily basis, which frankly doesn't sound too great...
Ivan Beshoff, Last Survivor Of Mutiny on the Potemkin, founded Beshoffs
wow. there's a factoid! the "Beshoffs" chain of chippers in Dublin were founded by this historic figure, who died in 1987
factoids beshoffs chips dublin history small-world battleship-potemkin russia
-
Excellent demo of how use of a block cipher with a known secret key makes an insecure MAC. "In short, CBC-MAC is a Message Authentication Code, not a strong hash function. While MACs can be built out of hash functions (e.g. HMAC), and hash functions can be built out of block ciphers like AES, not all MACs are also hash functions. CBC-MAC in particular is completely unsuitable for use as a hash function, because it only allows two parties with knowledge of a particular secret key to securely transmit messages between each other. Anyone with knowledge of that key can forge the messages in a way that keeps the MAC ("hash value") the same. All you have to do is run the forged message through CBC-MAC as usual, then use the AES decryption operation on the original hash value to find the last intermediate state. XORing this state with the CBC-MAC for the forged message yields a new block of data which, when appended to the forged message, will cause it to have the original hash value. Because the input is taken backwards, you can either modify the first block of the file, or just run the hash function backwards until you reach the block that you want to modify. You can make a forged file pass the hash check as long as you can modify an arbitrary aligned 16-byte block in it."
Scala 2.8 Collections API -- Performance Characteristics
wow. Every library vending a set of collection types should have a page like this
collections scala performance reference complexity big-o coding
So, after just over 3 and a half years, I'm leaving Amazon.
It's been great fun -- I can honestly say, even with my code being used by hundreds of millions of users in SpamAssassin and elsewhere, I hadn't really had to come to grips with the distributed systems problems that an Amazon-scale service involves.
During my time at Amazon, I've had the pleasure of building out a brand-new, groundbreaking innovative internal service, from scratch to its current status where it's deployed in production datacenters worldwide. It's a low-latency service, used to monitor Amazon's internal networks using massive quantities of measurement data and machine learning algorithms. It's really very nifty, and I'm quite proud of what we've achieved. I was lucky to work closely with some very smart people during this, too -- Amazon has some top-notch engineers.
But time to move on! In a week's time, I'll be joining Swrve to work on the server-side architecture of their system. Swrve have a very interesting product, extending the A/B-testing model into gaming, and a great team; and it'll be nice to get back into startup-land once again, for a welcome change. (It's not all roses working for a big company. ;) I'm looking forward to it. Who knows, I may even start blogging here again...
Pity about losing those 12 phone tool icons though!
CES: Worse Products Through Software
'The companies out there that know how to make decent software have been steadily eating their way into and through markets previously dominated by the hardware guys. Apple with music players, TiVo with video recording, even Microsoft with its decade-old Xbox Live service, which continues to embarrass the far weaker offerings from Sony and Nintendo. (And, yes, iOS is embarrassing all three console makers.)' See also Mat Honan's article at http://www.wired.com/gadgetlab/2012/12/internet-tv-sucks/ : 'Smart TVs are just too complicated. They have terrible user interfaces that differ wildly from device to device. It's not always clear what content is even available — for example, after more than two years on the market, you still can't watch Hulu Plus on your Google TV. [...] They give us too many options for apps most people will never use, and they do so at the expense of making it simple to find the shows and movies we want to watch, no matter where they are, be it online or on the air. As NPD puts it in the conclusion to its report, "OEMs and retailers need to focus less on new innovation in this space and more on simplification of the user experience and messaging if they want to drive additional, and new, behaviors on the TV." Which is a more polite way of saying, clean up your horrible interface, Samsung.' (via Craig)
via:craig design ui tv hardware television sony ces software
Fast Packed String Matching for Short Patterns [paper, PDF]
'Searching for all occurrences of a pattern in a text is a fundamental problem in computer science with applications in many other fields, like NLP, information retrieval and computational biology. In the last two decades a general trend has appeared trying to exploit the power of the word RAM model to speed-up the performances of classical string matching algorithms. [...] In this paper we use specialized word-size packed string matching instructions, based on the Intel streaming SIMD extensions (SSE) technology, to design very fast string matching algorithms in the case of short patterns.' Reminds me of http://en.wikipedia.org/wiki/Rabin%E2%80%93Karp_algorithm , but taking advantage of SIMD extensions, which should make things nice and speedy, at the cost of tying it to specific hardware platforms. (via Tony Finch)
rabin-karp algorithms strings string-matching papers via:fanf
Irish EU Council Presidency proposes destruction of right to privacy | EDRI
'For example, based on the current situation in Ireland, the idea is that all companies can do whatever they want with personal data, without fear of sanction. Sanctions, such as fines, "should be optional or at least conditional upon a prior warning or reprimand". In other words, do what you want, the worst that can happen is that you will receive a warning.' Shame! Daragh O'Brien's comment: 'utter idiocy'. ( at https://twitter.com/daraghobrien/status/292041500873850880 )
privacy ireland eu fail data-protection data-privacy politics
Belgium plans artificial island to store wind power
' Belgium is planning to build a doughnut-shaped island in the North Sea that will store wind energy by pumping water out of a hollow in the middle, as it looks for ways to lessen its reliance on nuclear power. One of the biggest problems with electricity is that it is difficult to store and the issue is exaggerated in the case of renewable energy from wind or sun because it is intermittent depending on the weather.' 'The island is still in the planning stages, but will be built out of sand 3 km off the Belgian coast near the town of Wenduine if it gets the final go-ahead. The island, which would also work as an offshore substation to transform the voltage of the electricity generated by wind turbines, could take five or more years to plan and build.'
power via:daev belgium wind-power hydro sea islands manmade storage
-
so Reddit uses the Wilson score confidence interval approach, it turns out; more details here (via Toby diPasquale)
ranking rating algorithms popularity python wilson-score-interval sorting statistics confidence-sort
The Neurocritic: Fisher-Price Synesthesia
'Synesthesia [jm: sic] is a rare perceptual phenomenon in which the stimulation of one sensory modality, or exposure to one type of stimulus, leads to a sensory (or cognitive) experience in a different, non-stimulated modality. For instance, some synesthetes have colored hearing while others might taste shapes. GRAPHEME-COLOR SYNESTHESIA is the condition in which individual printed letters are perceived in a specific, constant color. This occurs involuntarily and in the absence of colored font. [...] A new study has identified 11 synesthetes whose grapheme-color mappings appear to be based on the Fisher Price plastic letter set made between 1972-1990.' (via Dave Green)
fisher-price synesthesia synaesthesia colors colours sight neuroscience brain via-dave-green toys
Extreme Performance with Java - Charlie Hunt [slides, PDF]
presentation slides for Charlie Hunt's 2012 QCon presentation, where he discusses 'what you need to know about a modern JVM in order to be effective at writing a low latency Java application'. The talk video is at http://www.infoq.com/presentations/Extreme-Performance-Java
low-latency charlie-hunt performance java jvm presentations qcon slides pdf
-
'Bloomsday Map Of Dublin Based On Ulysses'. Beautiful! 'The Leopold's Day map is a stunning marriage of typography and cartography plotting all the streets alluded to by Joyce in Ulysses which were in existence on June 16th 1904. It is accompanied by a comprehensive and beautifully typeset directory with over 400 entries noting the landmarks, business and people of Dublin that were referenced in the text. The Leopold's Day map is an exquisitely detailed, limited edition piece. It has an impressive dimension of 1000mm x 700mm which means it can also fit into a ready made frame. Price: €125.00'
bloomsday ulysses dublin ireland maps james-joyce art prints
aaw/hyperloglog-redis - GitHub
'This gem is a pure Ruby implementation of the HyperLogLog algorithm for estimating cardinalities of sets observed via a stream of events. A Redis instance is used for storing the counters.'
-
'uses DNS witchcraft to allow you to access US/UK-only audio and video services like Hulu.com, BBC iPlayer, etc. without using a VPN or Web proxy.' According to http://superuser.com/questions/461316/how-does-tunlr-work , it proxies the initial connection setup and geo-auth, then mangles the stream address to stream directly, not via proxy. Sounds pretty useful
proxy network vpn dns tunnel content video audio iplayer bbc hulu streaming geo-restriction
OmniTI's Experiences Adopting Chef
A good, in-depth writeup of OmniTI's best practices with respect to build-out of multiple customer deployments, using multi-tenant Chef from a version-controlled repo. Good suggestions, and I am really looking forward to this bit: 'Chef tries to turn your system configuration into code. That means you now inherit all the woes of software engineering: making changes in a coordinated manner and ensuring that changes integrate well are now an even greater concern. In part three of this series, we'll look at applying software quality assurance and release management practices to Chef cookbooks and roles.'
-
Twitter's Scala style guide. 'While highly effective, Scala is also a large language, and our experiences have taught us to practice great care in its application. What are its pitfalls? Which features do we embrace, which do we eschew? When do we employ "purely functional style", and when do we avoid it? In other words: what have we found to be an effective use of the language? This guide attempts to distill our experience into short essays, providing a set of best practices. Our use of Scala is mainly for creating high volume services that form distributed systems — and our advice is thus biased — but most of the advice herein should translate naturally to other domains.'
Notes on Distributed Systems for Young Bloods -- Something Similar
'Below is a list of some lessons I've learned as a distributed systems engineer that are worth being told to a new engineer. Some are subtle, and some are surprising, but none are controversial. This list is for the new distributed systems engineer to guide their thinking about the field they are taking on. It's not comprehensive, but it's a good beginning.' This is a pretty nice list, a little over-stated, but that's the format. I particularly like the following: 'Exploit data-locality'; 'Learn to estimate your capacity'; 'Metrics are the only way to get your job done'; 'Use percentiles, not averages'; 'Extract services'.
-
'a Nagios plugin to poll Graphite'. Necessary, since service metrics are the true source of service health information
paperplanes. The Virtues of Monitoring, Redux
A rather vague and touchy-feely "state of the union" post on monitoring. Good set of links at the end, though; I like the look of Sensu and Tasseo, but am still unconvinced about the value of Boundary's offering
What happened to KHTML after Apple announced Safari
'There was a huge amount of excitement at the announcement that Safari would be using KHTML. At that time, it was almost a given that the OSS rendering engine was Gecko. KHTML was KDE's little engine that could. But nobody ever expected it to be picked up by other folks. One of the original parts of the KHTML-to-OS X port was KWQ (pronounced, "quack") that abstracted out the KDE API portions that were used in KHTML. Folks were pretty ecstatic at first. It seemed very validating. But that changed quickly. As Zack's post indicates, WebKit became a thing of unmergable code-drops. Even inside of the KDE community there became a split between the KHTML purists and the WebKit faction. They'd previously more or less all been KHTML developers, but post-WebKit there was something of a pragmatists vs. idealists split. Zack fell on the latter side of that (for understandable reasons: there was an existing community project, with its own set of values, and that was hijacked to a large extent by WebKit). A few years later WebKit transformed itself into a more or less valid open source project (see webkit.org), but that didn't close the rift in the KDE community between the two, at that point rather divergent, rendering engines. There's still some remaining melancholy that stems from that initial hope and what could have potentially been, but wasn't.'
history safari open-source code-drops over-the-wall webkit khtml kde oss apple
-
whoa. (via Dave O'Riordan)
Dan McKinley :: Whom the Gods Would Destroy, They First Give Real-time Analytics
'It's important to divorce the concepts of operational metrics and product analytics. [..] Funny business with timeframes can coerce most A/B tests into statistical significance.' 'The truth is that there are very few product decisions that can be made in real time.' HN discussion: http://news.ycombinator.com/item?id=5032588
Greyhound agrees to change consumer contracts and make refunds - National Consumer Agency
Take note, switchers: 'The National Consumer Agency (NCA) has received a commitment from Greyhound that it will amend certain terms in its standard consumer contract, which the NCA thinks are unfair to consumers. This will be done by January 18 2013. Among the terms considered unfair by the NCA are that consumers must forfeit their credit balance and pay a €45 administration fee, if they cancel their contract with Greyhound within 12 months. If you were charged money in these circumstances, Greyhound has agreed to refund you. Greyhound will communicate these changes to all of its consumers by 18 January 2013. If you have any questions about the changes or getting a refund, you should contact Greyhound directly.'
Pushover: Simple Mobile Notifications for Android and iOS
'Pushover makes it easy to send real-time notifications to your Android and iOS devices.' extremely simple HTTPS API; 'Pushover has no monthly subscription fees and users will always be able to receive unlimited messages for free. Most applications can send messages for free, subject to monthly limits.' Also supported by ifttt.com
-
'an elegant and simple HTTP library for Python, built for human beings.' 'Requests is an Apache2 Licensed HTTP library, written in Python, for human beings. Python's standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks. Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There's no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests.'
Surprisingly Good Evidence That Real Name Policies Fail To Improve Comments
'Enough theorizing, there's actually good evidence to inform the debate. For 4 years, Koreans enacted increasingly stiff real-name commenting laws, first for political websites in 2003, then for all websites receiving more than 300,000 viewers in 2007, and was finally tightened to 100,000 viewers a year later after online slander was cited in the suicide of a national figure. The policy, however, was ditched shortly after a Korean Communications Commission study found that it only decreased malicious comments by 0.9%. Korean sites were also inundated by hackers, presumably after valuable identities. Further analysis by Carnegie Mellon's Daegon Cho and Alessandro Acquisti, found that the policy actually increased the frequency of expletives in comments for some user demographics. While the policy reduced swearing and "anti-normative" behavior at the aggregate level by as much as 30%, individual users were not dismayed. "Light users", who posted 1 or 2 comments, were most affected by the law, but "heavy" ones (11-16+ comments) didn't seem to mind. Given that the Commission estimates that only 13% of comments are malicious, a mere 30% reduction only seems to clean up the muddied waters of comment systems a depressingly negligent amount. The finding isn't surprising: social science researchers have long known that participants eventually begin to ignore cameras video taping their behavior. In other words, the presence of some phantom judgmental audience doesn't seem to make us better versions of ourselves.' (via Ronan Lyons)
anonymity identity policy comments privacy politics new-media via:ronanlyons
HAT-trie: A Cache-conscious Trie-based Data Structure for Strings [PDF]
'Tries are the fastest tree-based data structures for managing strings in-memory, but are space-intensive. The burst-trie is almost as fast but reduces space by collapsing trie-chains into buckets. This is not however, a cache-conscious approach and can lead to poor performance on current processors. In this paper, we introduce the HAT-trie, a cache-conscious trie-based data structure that is formed by carefully combining existing components. We evaluate performance using several real-world datasets and against other highperformance data structures. We show strong improvements in both time and space; in most cases approaching that of the cache-conscious hash table. Our HAT-trie is shown to be the most e?cient trie-based data structure for managing variable-length strings in-memory while maintaining sort order.' (via Tony Finch)
The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases [PDF]
'Main memory capacities have grown up to a point where most databases ?t into RAM. For main-memory database systems, index structure performance is a critical bottleneck. Traditional in-memory data structures like balanced binary search trees are not ef?cient on modern hardware, because they do not optimally utilize on-CPU caches. Hash tables, also often used for main-memory indexes, are fast but only support point queries. To overcome these shortcomings, we present ART, an adaptive radix tree (trie) for ef?cient indexing in main memory. Its lookup performance surpasses highly tuned, read-only search trees, while supporting very ef?cient insertions and deletions as well. At the same time, ART is very space ef?cient and solves the problem of excessive worst-case space consumption, which plagues most radix trees, by adaptively choosing compact and ef?cient data structures for internal nodes. Even though ART's performance is comparable to hash tables, it maintains the data in sorted order, which enables additional operations like range scan and pre?x lookup.' (via Tony Finch)
Ef?cient In-Memory Indexing with Generalized Pre?x Trees [PDF]
'Ef?cient data structures for in-memory indexing gain in importance due to (1) the exponentially increasing amount of data, (2) the growing main-memory capacity, and (3) the gap between main-memory and CPU speed. In consequence, there are high performance demands for in-memory data structures. Such index structures are used—with minor changes—as primary or secondary indices in almost every DBMS. Typically, tree-based or hash-based structures are used, while structures based on prefix-trees (tries) are neglected in this context. For tree-based and hash-based structures, the major disadvantages are inherently caused by the need for reorganization and key comparisons. In contrast, the major disadvantage of trie-based structures in terms of high memory consumption (created and accessed nodes) could be improved. In this paper, we argue for reconsidering pre?x trees as in-memory index structures and we present the generalized trie, which is a pre?x tree with variable prefix length for indexing arbitrary data types of fixed or variable length. The variable prefix length enables the adjustment of the trie height and its memory consumption. Further, we introduce concepts for reducing the number of created and accessed trie levels. This trie is order-preserving and has deterministic trie paths for keys, and hence, it does not require any dynamic reorganization or key comparisons. Finally, the generalized trie yields improvements compared to existing in-memory index structures, especially for skewed data. In conclusion, the generalized trie is applicable as general-purpose in-memory index structure in many different OLTP or hybrid (OLTP and OLAP) data management systems that require balanced read/write performance.' (via Tony Finch)
A Non-Blocking HashTable by Dr. Cliff Click : programming
Proggit discovers the NonBlockingHashMap. This comment from Boundary's cscotta is particularly interesting: "The code is intricate and curiously-formatted, but NBHM is quite excellent. The majority of our analytics platform is backed by NBHMs updated rapidly in parallel. Cliff's a great, friendly, approachable guy; if you have any specific questions about the approaches or implementation, he may be happy to answer."
data-structures algorithms non-blocking concurrency threading multicore cliff-click azul maps java boundary
English Letter Frequency Counts: Mayzner Revisited or ETAOIN SRHLDCU
Amazing how consistent the n-gram counts are between Peter Norvig's analysis (here) against the 20120701 Google Books corpus, and Mark Mayzner's 20,000-word corpus from the early 1960s
english statistics n-grams words etaoin-shrdlu peter-norvig mark-mayzner
Solving monitoring state storage problems using Redis
a nice basic Redis-in-practice post
-
'a HTTP client mock library for Python, 100% inspired on ruby's FakeWeb [ https://github.com/chrisk/fakeweb ].' 'HTTPretty monkey patches Python's socket core module, reimplementing the HTTP protocol by mocking requests and responses.'
mocking testing http python ruby unit-tests tests monkey-patching
Why did infinite scroll fail at Etsy?
'A/B testing must be done in a modularized fashion. The "fail" case he gave was when Etsy spent months developing and testing infinite scroll to their search listings, only to find that it had a negative impact on engagement.' [...] 'instead of having the goal of "test infinite scroll," Etsy realized it needed to test each assumption separately, and this going forward is their game plan.'
usability testing design etsy ab-testing test modularization via:hn
-
Annotations-based git-like CLI helper for Java
"Matters Computational - Ideas, Algorithms, Source Code"
A hefty tome (in PDF format) containing lots of interesting algorithms and computational tricks; code is GPLv3 licensed
Dan McKinley :: Effective Web Experimentation as a Homo Narrans
Good demo from Etsy's A/B testing, of how the human brain can retrofit a story onto statistically-insignificant results. To fix: 'avoid building tooling that enables fishing expeditions; limit our post-hoc rationalization by explicitly constraining it before the experiment. Whenever we test a feature on Etsy, we begin the process by identifying metrics that we believe will change if we 1) understand what is happening and 2) get the effect we desire.'
testing etsy statistics a-b-testing fishing ulysses-contract brain experiments
Lesser known crimes: do you own that copyright?
A very interesting crime on the Irish statute books:
Section 141 of the Copyright and Related Rights Act 2000 provides: A person who, for financial gain, makes a claim to enjoy a right under this Part [ie. copyright] which is, and which he or she knows or has reason to believe is, false, shall be guilty of an offence and shall be liable on conviction on indictment to a fine not exceeding £100,000, or to imprisonment for a term not exceeding 5 years, or both.
-
Makers of "Feck It Sure It's Grand" merchandise are flogging stuff for 50 cent (+ shipping) in their "out with the old" sale
Patent trolls want $1,000 for using scanners
We are truly living in the future -- a dystopian future, but one nonetheless. A patent troll manages to obtain "gobbledigook" patents on using a scanner to scan to PDF, then attempts to shake down a bunch of small companies before eventually running into resistance, at which point it "forks" into a bunch of algorithmically-named shell companies, spammer-style, sending the same demands. Those demands in turn contain this beauty of Stockholm-syndrome-inducing prose:
'You should know also that we have had a positive response from the business community to our licensing program. As you can imagine, most businesses, upon being informed that they are infringing someone's patent rights, are interested in operating lawfully and taking a license promptly. Many companies have responded to this licensing program in such a manner. Their doing so has allowed us to determine that a fair price for a license negotiated in good faith and without the need for court action is a payment of $900 per employee. We trust that your organization will agree to conform your behavior to respect our patent rights by negotiating a license rather than continuing to accept the benefits of our patented technology without a license. Assuming this is the case, we are prepared to make this pricing available to you.'
And here's an interesting bottom line:The best strategy for target companies? It may be to ignore the letters, at least for now. "Ignorance, surprisingly, works," noted Prof. Chien in an e-mail exchange with Ars. Her study of startups targeted by patent trolls found that when confronted with a patent demand, 22 percent ignored it entirely. Compare that with the 35 percent that decided to fight back and 18 percent that folded. Ignoring the demand was the cheapest option ($3,000 on average) versus fighting in court, which was the most expensive ($870,000 on average). Another tactic that clearly has an effect: speaking out, even when done anonymously. It hardly seems a coincidence that the Project Paperless patents were handed off to a web of generic-sounding LLCs, with demand letters signed only by "The Licensing Team," shortly after the "Stop Project Paperless" website went up. It suggests those behind such low-level licensing campaigns aren't proud of their behavior. And rightly so.
patents via:fanf networks printing printers scanning patent-trolls project-paperless adzpro gosnel faslan
Keep predicting and you'll be right eventually?
debunking Ken Ring, the kiwi "long term weather prediction" "scientist" who gets trundled out every year around this time
29c3 HashDOS presentation slides (PDF)
Summary: MurmurHash still vulnerable, likewise Cityhash and Python's hash -- use SipHash
via:fanf cityhash siphash hash dos security hashdos murmurhash
Scaling Crashlytics: Building Analytics on Redis 2.6
How one analytics/metrics co is using Redis on the backend
Systemd, systemd-nspawn, and namespaces for Linux service compartmentalization
"Using ReadOnlyDirectories= andInaccessibleDirectories= you may setup a file system namespace jail for your service. Initially, it will be identical to your host OS' file system namespace. By listing directories in these directives you may then mark certain directories or mount points of the host OS as read-only or even completely inaccessible to the daemon."
security systemd jails namespaces linux compartmentalisation