'Uncertain<T>: A First-Order Type for Uncertain Data' [paper, PDF]
'Emerging applications increasingly use estimates such as sensor data (GPS), probabilistic models, machine learning, big data, and human data. Unfortunately, representing this uncertain data with discrete types (floats, integers, and booleans) encourages developers to pretend it is not probabilistic, which causes three types of uncertainty bugs. (1) Using estimates as facts ignores random error in estimates. (2) Computation compounds that error. (3) Boolean questions on probabilistic data induce false positives and negatives. This paper introduces Uncertain<T>, a new programming language abstraction for uncertain data. We implement a Bayesian network semantics for computation and conditionals that improves program correctness. The runtime uses sampling and hypothesis tests to evaluate computation and conditionals lazily and efficiently. We illustrate with sensor and machine learning applications that Uncertain<T> improves expressiveness and accuracy.' (via Tony Finch)
uncertainty estimation types strong-typing coding probability statistics machine-learning sampling via:fanf
2014
Why Airlines Want to Make You Suffer
'The fee [airline pricing] model comes with systematic costs that are not immediately obvious. Here's the thing: in order for fees to work, there needs be something worth paying to avoid. That necessitates, at some level, a strategy that can be described as "calculated misery." Basic service, without fees, must be sufficiently degraded in order to make people want to pay to escape it. And that's where the suffering begins.'
-
'Ádám was trying his hand at a problem in Excel, but the official rules prohibit the use of Excel macros. In a daze, he came up with one of the most clever uses of Excel: building an assembly interpreter with the most popular spreadsheet program. This is a virtual Harvard architecture machine without writable RAM; the stack is only lots and lots of IFs.'
-
A causal profiler for C++.
Causal profiling is a novel technique to measure optimization potential. This measurement matches developers' assumptions about profilers: that optimizing highly-ranked code will have the greatest impact on performance. Causal profiling measures optimization potential for serial, parallel, and asynchronous programs without instrumentation of special handling for library calls and concurrency primitives. Instead, a causal profiler uses performance experiments to predict the effect of optimizations. This allows the profiler to establish causality: "optimizing function X will have effect Y," exactly the measurement developers had assumed they were getting all along.
I can see this being a good technique to stochastically discover race conditions and concurrency bugs, too.optimization c++ performance coding profiling speed causal-profilers
-
This is the version with the superfast petabyte-sort record:
Spark 1.2 includes several cross-cutting optimizations focused on performance for large scale workloads. Two new features Databricks developed for our world record petabyte sort with Spark are turned on by default in Spark 1.2. The first is a re-architected network transfer subsystem that exploits Netty 4's zero-copy IO and off heap buffer management. The second is Spark's sort based shuffle implementation, which we've now made the default after significant testing in Spark 1.1. Together, we've seen these features give as much as 5X performance improvement for workloads with very large shuffles.
spark sorting hadoop map-reduce batch databricks apache netty
The VATMOSS debacle: does the "manual email" loophole work?
As the 1 January deadline gallops towards the EU, microbusinesses desperate to stay open without breaking the law try to find out, "Can I email stuff out instead?" Well... Yes. - No - It depends - and simultaneously yes AND no, according to Schrödinger's VAT. So that's clear, then.
One artist closing up their Bandcamp site due to new VATMOSS laws
Nice work, EU
eu law tax vat vatmoss matt-stevens bandcamp music downloads
Use sshuttle to Keep Safe on Insecure Wi-Fi
I keep forgetting about sshuttle. It's by far the easiest way to get a cheapo IP-over-SSH VPN working with an OSX client, particularly since it's in homebrew
ssh vpn sshuttle tunnelling security ip wifi networking osx homebrew
BoardGameGeek's best board games for kids
great list
Warning: Do NOT use my mirrors services until I have reviewed the situation
Things hotting up in TOR-land.
Until I have had the time and information available to review the situation, I am strongly recommending my mirrors are not used under any circumstances. If they come back online without a PGP signed message from myself to further explain the situation, exercise extreme caution and treat even any items delivered over TLS to be potentially hostile.
Working Effectively with Unit Tests
$14.99 ebook, recommended by Steve Vinoski, looks good
unit-testing testing ebooks jay-fields tests steve-vinoski coding
Fixing tethering on Android KitKat
Google made a change in Android 4.4 which allows operators to know when users are using tethering and conveniently block tethered devices from accessing internet. This can be fixed permanently using the following procedure.
Well this is stupid. (via Tony Finch)Why Ireland must protect privacy of Irish emails and internet usage from surveillance
It's now over a year since Edward Snowden went public with evidence of mass surveillance and extensive abuses by the NSA, GCHQ and other intelligence agencies. In other countries these revelations prompted parliamentary inquiries, diplomatic representations and legislation. In Ireland the only response was a promise [..] to help extradite Mr Snowden should he land here.
ireland politics edward-snowden extradition privacy nsa gchq spying surveillance tj-mcintyre
Final Root Cause Analysis and Improvement Areas: Nov 18 Azure Storage Service Interruption
For the record
root-cause azure outages postmortem cloud microsoft deployment
-
To demonstrate that hackers have no interest in suppressing speech, quashing controversy, or being intimidated by vague threats, we ask that Sony allow the hacker community to distribute "The Interview" for them on the 25th of December. Now, we're aware that Sony may refer to this distribution method as piracy, but in this particular case, it may well prove to be the salvation of the motion picture industry. By freely offering the film online, millions of people will get to see it and decide for themselves if it has any redeeming qualities whatsoever - as opposed to nobody seeing it and the studios writing it off as a total loss. Theaters would be free from panic as our servers would become the target of any future vague threats (and we believe Hollywood will be most impressed with how resilient peer-to-peer distribution can be in the face of attacks). Most importantly, we would be defying intimidation, something the motion picture industry doesn't quite have a handle on, which is surprising considering how much they've relied upon it in the past.
2600 funny hackers security sony north-korea the-interview movies piracy
Top 20 Bourbons Around $30 | The Bourbon Review
Need to keep an eye out for a few of these -- will probably be a little more than $30 given the whole import/export carry-on of course
Why You Shouldn't Use ZooKeeper for Service Discovery
In CAP terms, ZooKeeper is CP, meaning that it's consistent in the face of partitions, not available. For many things that ZooKeeper does, this is a necessary trade-off. Since ZooKeeper is first and foremost a coordination service, having an eventually consistent design (being AP) would be a horrible design decision. Its core consensus algorithm, Zab, is therefore all about consistency. For coordination, that's great. But for service discovery it's better to have information that may contain falsehoods than to have no information at all. It is much better to know what servers were available for a given service five minutes ago than to have no idea what things looked like due to a transient network partition. The guarantees that ZooKeeper makes for coordination are the wrong ones for service discovery, and it hurts you to have them.
Yes! I've been saying this for months -- good to see others concurring.architecture zookeeper eureka outages network-partitions service-discovery cap partitions
-
omg, Die Gute Fabrik's game collection featuring the AMAZING Johann Sebastian Joust -- now available on Mac, Linux and (missing JSJ) Windows. Time to buy an assload of Move controllers!
jsj johann-sebastian-joust games fun die-gute-fabrik sportsfriends gaming linux mac
Lambda, Javascript Micro-Services on AWS
Nice worked-through Lambda example
'Machine Learning: The High-Interest Credit Card of Technical Debt' [PDF]
Oh god yes. This is absolutely spot on, as you would expect from a Google paper -- at this stage they probably have accumulated more real-world ML-at-scale experience than anywhere else. 'Machine learning offers a fantastically powerful toolkit for building complex systems quickly. This paper argues that it is dangerous to think of these quick wins as coming for free. Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning. The goal of this paper is highlight several machine learning specific risk factors and design patterns to be avoided or refactored where possible. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, changes in the external world, and a variety of system-level anti-patterns. [....] 'In this paper, we focus on the system-level interaction between machine learning code and larger systems as an area where hidden technical debt may rapidly accumulate. At a system-level, a machine learning model may subtly erode abstraction boundaries. It may be tempting to re-use input signals in ways that create unintended tight coupling of otherwise disjoint systems. Machine learning packages may often be treated as black boxes, resulting in large masses of "glue code" or calibration layers that can lock in assumptions. Changes in the external world may make models or input signals change behavior in unintended ways, ratcheting up maintenance cost and the burden of any debt. Even monitoring that the system as a whole is operating as intended may be difficult without careful design. Indeed, a remarkable portion of real-world "machine learning" work is devoted to tackling issues of this form. Paying down technical debt may initially appear less glamorous than research results usually reported in academic ML conferences. But it is critical for long-term system health and enables algorithmic advances and other cutting-edge improvements.'
machine-learning ml systems ops tech-debt maintainance google papers hidden-costs development
The FBI Used the Web's Favorite Hacking Tool to Unmask Tor Users | WIRED
Since Operation Torpedo [use of a Metasploit side project], there's evidence the FBI's anti-Tor capabilities have been rapidly advancing. Torpedo was in November 2012. In late July 2013, computer security experts detected a similar attack through Dark Net websites hosted by a shady ISP called Freedom Hosting—court records have since confirmed it was another FBI operation. For this one, the bureau used custom attack code that exploited a relatively fresh Firefox vulnerability—the hacking equivalent of moving from a bow-and-arrow to a 9-mm pistol. In addition to the IP address, which identifies a household, this code collected the MAC address of the particular computer that infected by the malware. "In the course of nine months they went from off the shelf Flash techniques that simply took advantage of the lack of proxy protection, to custom-built browser exploits," says Soghoian. "That's a pretty amazing growth … The arms race is going to get really nasty, really fast."
fbi tor police flash security privacy anonymity darknet wired via:bruces
Digital Rights Ireland files Amicus Brief in Microsoft v USA with Liberty and ORG
Microsoft -v- USA is an important ongoing case, currently listed for hearing in 2015 before the US Federal Court of Appeal of the 2nd Circuit. However, as the case centres around the means by which NY law enforcement are seeking to access data of an email account which resides in Dublin, it is also crucially significant to Ireland and the rest of the EU. For that reason, Digital Rights Ireland instructed us to file an Amicus Brief in the US case, in conjunction with the global law firm of White & Case, who have acted pro bono in their representation. Given the significance of the case for the wider EU, both Liberty and the Open Rights Group in the UK have joined Digital Rights Ireland as amici on this brief. We hope it will be of aid to the US court in assessing the significance of the order being appealed by Microsoft for EU citizens and European states, in the light of the existing US and EU Mutual Legal Assistance Treaty.
amicus-briefs law us dri microsoft mlats org liberty eu privacy
-
Hey look, PID 1 segfaulting! I haven't seen that happen since we managed to corrupt /bin/sh on Ultrix in 1992. Nice work Fedora
Operation Socialist: How GCHQ Spies Hacked Belgium's Largest Telco
Chilling.
GCHQ maintains a huge repository named MUTANT BROTH that stores billions of these intercepted cookies, which it uses to correlate with IP addresses to determine the identity of a person. GCHQ refers to cookies internally as "target detection identifiers."
privacy gchq surveillance belgacom regin uk spying belgium isps cookies malware
-
Generate graphs/flowcharts from text a la Markdown. Pretty much identical to graphviz surely?
How Etsy Does Continuous Integration for Mobile Apps
Very impressive. I particularly like the use of Tester Dojos to get through a backlog of unwritten tests -- we had a similar problem recently...
dojos testing ci cd builds etsy mobile ios shenzen trylib jenkins tester-dojos
-
From 7-8pm on Friday, [RepricerExpress] software, used by third-party sellers to ensure their products are the cheapest on the market, went a bit haywire and reduced prices to as little as 1p.
1p amazon resellers repricer-express fail price-cutting automation risks undercutting
-
Wow, this looks cool. $159
littleBits and Korg have demystified a traditional analog synthesizer, making it super easy for novices and experts alike to create music. connects to speakers, computers and headphones. can be used to make your own instruments. fits into the littleBits modular system for infinite combos of audio, visual and sensory experiences
Cold, Cough, Flu: Best Medicines For Your Symptoms - Iodine
'Find the best cold & flu meds for your symptoms' -- actually pretty useful, although of course the US-only brandnames aren't available over here...
Introducing Atlas: Netflix's Primary Telemetry Platform
This sounds really excellent -- the dimensionality problem it deals with is a familiar one, particularly with red/black deployments, autoscaling, and so on creating trees of metrics when new transient servers appear and disappear. Looking forward to Netflix open sourcing enough to make it usable for outsiders
-
Hello! I love satellite imagery and topographic maps so I made several wallpapers with those gorgeous pictures. All wallpapers are iPhone 6 Plus optimized with 1242×2208 pixels, but you can resize for any device you have. The original sources are listed below.
(Via This Is Colossal)
Good advice on running large-scale database stress tests
I've been bitten by poor key distribution in tests in the past, so this is spot on: 'I'd run it with Zipfian, Pareto, and Dirac delta distributions, and I'd choose read-modify-write transactions.' And of course, a dataset bigger than all combined RAM. Also: http://smalldatum.blogspot.ie/2014/04/biebermarks.html -- the "Biebermark", where just a single row out of the entire db is contended on in a read/modify/write transaction: "the inspiration for this is maintaining counts for [highly contended] popular entities like Justin Bieber and One Direction."
biebermark benchmarks testing performance stress-tests databases storage mongodb innodb foundationdb aphyr measurement distributions keys zipfian
AWS Key Management Service Cryptographic Details
"AWS Key Management Service (AWS KMS) provides cryptographic keys and operations scaled for the cloud. AWS KMS keys and functionality are used by other AWS cloud services, and you can use them to protect user data in your applications that use AWS. This white paper provides details on the cryptographic operations that are executed within AWS when you use AWS KMS."
-
some good details of Aurora innards
mysql databases aurora aws ec2 sql storage transactions replication
If Eventual Consistency Seems Hard, Wait Till You Try MVCC
ex-Percona MySQL wizard Baron Schwartz, noting that MVCC as implemented in common SQL databases is not all that simple or reliable compared to big bad NoSQL Eventual Consistency:
Since I am not ready to assert that there's a distributed system I know to be better and simpler than eventually consistent datastores, and since I certainly know that InnoDB's MVCC implementation is full of complexities, for right now I am probably in the same position most of my readers are: the two viable choices seem to be single-node MVCC and multi-node eventual consistency. And I don't think MVCC is the simpler paradigm of the two.
nosql concurrency databases mysql riak voldemort eventual-consistency reliability storage baron-schwartz mvcc innodb postgresql
-
This is quite interesting/weird -- Stripe's protocol for mass-CCing email as they scale up the company, based around http://en.wikipedia.org/wiki/Civil_inattention
communication culture email management stripe cc transparency civil-inattention
Shanley Kane of Model View Culture Challenges a "Corrupt" Silicon Valley | MIT Technology Review
If their interests were better serving the world, using technology as a force for social justice, and equitably distributing technology wealth to enrich society … sure, they'd be acting against their interests. But the reality is that tech companies centralize power and wealth in a small group of privileged white men. When that's the goal, then exploiting the labor of marginalized people and denying them access to power and wealth is 100 percent in line with the endgame. A more diverse tech industry would be better for its workers and everyone else, but it would be worse for the privileged white men at the top of it, because it would mean they would have to give up their monopoly on money and power. And they will fight that with everything they've got, which is why we see barriers to equality at every level of the industry.
culture feminism tech mit-tech-review shanley-kane privilege vcs silicon-valley
-
Awesome! I was completely unaware this was coming down the pipeline.
A new, transactionally updated Ubuntu for the cloud. Ubuntu Core is a new rendition of Ubuntu for the cloud with transactional updates. Ubuntu Core is a minimal server image with the same libraries as today's Ubuntu, but applications are provided through a simpler mechanism. The snappy approach is faster, more reliable, and lets us provide stronger security guarantees for apps and users — that's why we call them "snappy" applications. Snappy apps and Ubuntu Core itself can be upgraded atomically and rolled back if needed — a bulletproof approach to systems management that is perfect for container deployments. It's called "transactional" or "image-based" systems management, and we're delighted to make it available on every Ubuntu certified cloud.
ubuntu linux packaging snappy ubuntu-core transactional-updates apt docker ops
Dan McKinley :: Thoughts on the Technical Track
Ouch. I think Amazon did a better job of the Technical Track concept than this, at least
engineering management technical-track principal-engineer career work
-
"git for operating system binaries". OSTree is a tool for managing bootable, immutable, versioned filesystem trees. It is not a package system; nor is it a tool for managing full disk images. Instead, it sits between those levels, offering a blend of the advantages (and disadvantages) of both. You can use any build system you like to place content into it on a build server, then export an OSTree repository via static HTTP. On each client system, "ostree admin upgrade" can incrementally replicate that content, creating a new root for the next reboot. This provides fully atomic upgrades. Any changes made to /etc are propagated forwards, and all local state in /var is shared. A key goal of the project is to complement existing package systems like RPM and Debian packages, and help further their evolution. In particular for example, RPM-OSTree (linked below) has as a goal a hybrid tree/package model, where you replicate a base tree via OSTree, and then add packages on top.
State sanctions foreign phone and email tapping
Well, this stinks.
Foreign law enforcement agencies will be allowed to tap Irish phone calls and intercept emails under a statutory instrument signed into law by Minister for Justice Frances Fitzgerald. Companies that object or refuse to comply with an intercept order could be brought before a private "in camera" court. The legislation, which took effect on Monday, was signed into law without fanfare on November 26th, the day after documents emerged in a German newspaper indicating the British spy agency General Communications Headquarters (GCHQ) had directly tapped undersea communications cables between Ireland and Britain for years.
"Looks like Chicago PD had a stingray out at the Eric Garner protest last night"
Your tax dollars at work: Spying on people just because they demand that the government's agents stop killing black people. [...] Anonymous has released a video featuring what appear to be Chicago police radio transmissions revealing police wiretapping of organizers' phones at the protests last night the day after Thanksgiving, perhaps using a stingray. The transmissions pointing to real-time wiretapping involve the local DHS-funded spy 'fusion' center.
imsi-catcher stingray surveillance eric-garner protests privacy us-politics anonymous chicago police wiretapping dhs
When data gets creepy: the secrets we don't realise we're giving away | Technology | The Guardian
Very good article around the privacy implications of derived and inferred aggregate metadata from Ben Goldacre.
We are entering an age – which we should welcome with open arms – when patients will finally have access to their own full medical records online. So suddenly we have a new problem. One day, you log in to your medical records, and there's a new entry on your file: "Likely to die in the next year." We spend a lot of time teaching medical students to be skilful around breaking bad news. A box ticked on your medical records is not empathic communication. Would we hide the box? Is that ethical? Or are "derived variables" such as these, on a medical record, something doctors should share like anything else?
advertising ethics privacy security law data aggregation metadata ben-goldacre
Stellar/Ripple suffer a failure of their consensus system, resulting in a split-brain failure
Prof. Mazières's research indicated some risk that consensus could fail, though we were nor certain if the required circumstances for such a failure were realistic. This week, we discovered the first instance of a consensus failure. On Tuesday night, the nodes on the network began to disagree and caused a fork of the ledger. The majority of the network was on ledger chain A. At some point, the network decided to switch to ledger chain B. This caused the roll back of a few hours of transactions that had only been recorded on chain A. We were able to replay most of these rolled back transactions on chain B to minimize the impact. However, in cases where an account had already sent a transaction on chain B the replay wasn't possible.
consensus distcomp stellar ripple split-brain postmortems outages ledger-fork payment
the "Unknown Pleasures" cover, emulated in Mathematica
In July 1967, astronomers at the Cavendish Laboratory in Cambridge, observed an unidentified radio signal from interstellar space, which flashed periodically every 1.33730 seconds. This object flashed with such regularity that it was accurate enough to be used as a clock and only be off by one part in a hundred million. It was eventually determined that this was the first discovery of a pulsar, CP-1919. This is an object that has about the same mass as the Sun, but is the size of the San Francisco Bay at its widest (~20 kilometers) that is rotating so fast that its emitting a beam of light towards Earth like a strobing light house! Pulsars are neutron stars that are formed from the remnants of a massive star when it experiences stellar death. A hand drawn graph plotted in the style of a waterfall plot, in the Cambridge Encyclopedia of Astronomy, later became renown for its use on the cover of the album "Unknown Pleasures" by 1970s English band Joy Division.
The entire blog at http://intothecontinuum.tumblr.com/ is pretty great. Lots of nice mathematical animated GIFs, accompanied by Mathematica source and related ponderings.maths gifs animation art unknown-pleasures mathematica cp-1919 pulsars astronomy joy-division waterfall-plots cambridge blogs
-
'Pubs & Bars With Raging Fires in Dublin'. This is important!
-
'Anurag@AWS posts a quite interesting comment on Aurora failover: We asynchronously write to 6 copies and ack the write when we see four completions. So, traditional 4/6 quorums with synchrony as you surmised. Now, each log record can end up with a independent quorum from any other log record, which helps with jitter, but introduces some sophistication in recovery protocols. We peer to peer to fill in holes. We also will repair bad segments in the background, and downgrade to a 3/4 quorum if unable to place in an AZ for any extended period. You need a pretty bad failure to get a write outage.' (via High Scalability)
via:highscalability mysql aurora failover fault-tolerance aws replication quorum
-
Nice list -- lots of random toy services
-
actual stats and data on how programming languages affect coding work
CoreOS is building a container runtime, Rocket
Whoa, trouble at mill in Dockerland!
When Docker was first introduced to us in early 2013, the idea of a "standard container" was striking and immediately attractive: a simple component, a composable unit, that could be used in a variety of systems. The Docker repository included a manifesto of what a standard container should be. This was a rally cry to the industry, and we quickly followed. Brandon Philips, co-founder/CTO of CoreOS, became a top Docker contributor, and now serves on the Docker governance board. CoreOS is one of the most widely used platforms for Docker containers, and ships releases to the community hours after they happen upstream. We thought Docker would become a simple unit that we can all agree on. Unfortunately, a simple re-usable component is not how things are playing out. Docker now is building tools for launching cloud servers, systems for clustering, and a wide range of functions: building images, running images, uploading, downloading, and eventually even overlay networking, all compiled into one monolithic binary running primarily as root on your server. The standard container manifesto was removed. We should stop talking about Docker containers, and start talking about the Docker Platform. It is not becoming the simple composable building block we had envisioned.
The Wirecutter's Best Everyday Things for $50, $100, $200
excellent guide (via JK)
-
isn't that curious.
-
Reading between the lines, it looks like Rails 4 is waaay slower than 3....
Day 1 - Docker in Production: Reality, Not Hype
Good Docker info from Bridget Kromhout, on their production and dev usage of Docker at DramaFever. lots of good real-world tips
Lost avant-garde painting found in Stuart Little's living room
Two years later, he heard from Lisa S., an assistant set designer on [the movie] Stuart Little. She had bought the painting for $500 from an antiques store in Pasadena specifically for the movie because she thought its cool elegance was perfectly suited for the Little's New York City apartment. Lisa S. had tracked it down in another warehouse and purchased it from Sony just because she liked it so much. When she contacted Barki, she had no idea of the history of the painting hanging on her bedroom wall. After Barki visited the painting in person and confirmed its identity, Lisa sold it to a private collector. That collector has now been persuaded to sell it in Hungary. It will go up for auction at the Virag Judit Art Gallery in Budapest on December 13th with a starting price of 110,000 euros ($160,000). Gergely Barki won't make a dime off of his discovery, but he will have a great story to tell in his biography of the artist.
stuart-little art history hungary pasadena movies set-design antiques robert-bereny post-impressionism
rjbs's rubric: In Soviet Minecraft, server op you!
wow, that is too much effort for a 7-year-old's Minecraft server ;) Very impressive
-
How Rust avoids GC overhead using it's "borrow" system:
Rust achieves memory safety without GC by using a sophiscated borrow system. For any resource (stack memory, heap memory, file handle and so on), there is exactly one owner which takes care of its resource deallocation, if needed. You may create new bindings to refer to the resource using & or &mut, which is called a borrow or mutable borrow. The compiler ensures all owners and borrowers behave correctly.
-
Actually, I'm really agreeing with a lot of this. Particularly this part:
Programmers will cringe at writing some kind of command dispatch list: if command = "up": up() elif command = "status": status() elif command = "revert": revert() ... so they'll go off and write some introspecting auto-dispatch cleverness, but that takes longer to write and will surely confuse future readers who'll wonder how the heck revert() ever gets called. Yet the programmer will incorrectly feel as though he saved himself time. This is the trap of the dynamic language. It feels like you're being more productive, but aside from the first 10 minutes of a new program, you're not. Just write the stupid dispatch manually and get on with the real work.
I've also gone right off dynamic languages for any kind of non-toy work. Mind you he needs to get around to ditching Vim for a proper IDE. That's the key thing that makes coding in a statically-typed language really pleasant -- when graphical refactoring becomes easy and usable, and errors are visible as you type them... Facebook Fabric Networking Deconstructed
whoa, this is incredibly in-depth
facebook datacenter networking clos-networks infrastructure networks fat-tree
OS X doesn't support 'ndots' DNS resolution
"ping foo.bar" will not append the "search" domains configured in /etc/resolv.conf. Apparently this has been broken since OS X Lion, no sign of a fix. Nice work Apple
-
a catastrophic TCP throughput collapse that occurs as the number of storage servers sending data to a client increases past the ability of an Ethernet switch to buffer packets. In a clustered file system, for example, a client application requests a data block striped across several storage servers, issuing the next data block request only when all servers have responded with their portion (Figure 1). This synchronized request workload can result in packets overfilling the buffers on the client's port on the switch, resulting in many losses. Under severe packet loss, TCP can experience a timeout that lasts a minimum of 200ms, determined by the TCP minimum retransmission timeout (RTOmin).
incast networking performance tcp bandwidth buffering switch ethernet capacity
-
Excellent real-world war story from Facebook -- a long-running mystery bug was eventually revealed to be a combination of edge-case behaviours across all the layers of the networking stack, from L2 link aggregation at the agg-router level, up to the L7 behaviour of the MySQL client connection pool.
Facebook collocates many of a user's nodes and edges in the social graph. That means that when somebody logs in after a while and their data isn't in the cache, we might suddenly perform 50 or 100 database queries to a single database to load their data. This starts a race among those queries. The queries that go over a congested link will lose the race reliably, even if only by a few milliseconds. That loss makes them the most recently used when they are put back in the pool. The effect is that during a query burst we stack the deck against ourselves, putting all of the congested connections at the top of the deck.
architecture debugging devops facebook layer-7 mysql connection-pooling aggregation networking tcp-stack
"Macaroons" for fine-grained secure database access
Macaroons are an excellent fit for NoSQL data storage for several reasons. First, they enable an application developer to enforce security policies at very fine granularity, per object. Gone are the clunky security policies based on the IP address of the client, or the per-table access controls of RDBMSs that force you to split up your data across many tables. Second, macaroons ensure that a client compromise does not lead to loss of the entire database. Third, macaroons are very flexible and expressive, able to incorporate information from external systems and third-party databases into authorization decisions. Finally, macaroons scale well and are incredibly efficient, because they avoid public-key cryptography and instead rely solely on fast hash functions.
security macaroons cookies databases nosql case-studies storage authorization hyperdex
Richard Tynan on Twitter: "GCHQ Tapping Eircom owned cable"
Cable listed as owned by Eircom and Cable and Wireless (now Vodafone?)
Hermitage: Testing the "I" in ACID
[Hermitage is] a test suite for databases which probes for a variety of concurrency issues, and thus allows a fair and accurate comparison of isolation levels. Each test case simulates a particular kind of race condition that can happen when two or more transactions concurrently access the same data. Each test can pass (if the database's implementation of isolation prevents the race condition from occurring) or fail (if the race condition does occur).
Consul case study from Hootsuite
Hootsuite used Consul for distributed configuration, specifically dark-launch feature flags, with great results: 'Trying out bleeding edge software can be a risky proposition, but in the case of Consul, we've found it to be a solid system that works basically as described and was easy to get up and running. We managed to go from initial investigations to production within a month. The value was immediately obvious after looking into the key-value store combined with the events system and it's DNS features and each of these has worked how we expected. Overall it has been fun to work with and has worked well and based on the initial work we have done with the Dark Launching system we're feeling confident in Consul's operation and are looking forward to expanding the scope of it's use.'
consul dark-launches feature-flags configuration distributed hootsuite notification
Docker at Shopify: How we built containers that power over 100,000 onl
excellent case study of production-scale usage of Docker
-
UNIX system service [jmason: ie a sidecar] that collects events and reliably delivers them to kafka, relieving other services on the same system from having to do so. Journals events through bolt-db so that in the event of an kafka outage, events can still be accepted, and will be delivered when kafka becomes available.
ExecutorService - 10 tips and tricks
Excellent advice from Tomasz Nurkiewicz' blog for anyone using java.util.concurrent.ExecutorService regularly. The whole blog is full of great posts btw
7 Tips for Improving MapReduce Performance | Cloudera Engineering Blog
Good advice for Hadoop optimization
AWS re:Invent 2014 Video & Slide Presentation Links
Nice work by Andrew Spyker -- this should be an official feature of the re:Invent website, really
reinvent aws conferences talks slides ec2 s3 ops presentations
(SDD416) Amazon EBS Deep Dive | AWS re:Invent 2014
Excellent data on current EBS performance characteristics
-
A desktop app for finding and inserting GIFs into any conversation
Oh yes. -
The researchers have no doubt that Regin is a nation-state tool and are calling it the most sophisticated espionage machine uncovered to date—more complex even than the massive Flame platform, uncovered by Kaspersky and Symantec in 2012 and crafted by the same team who created Stuxnet. "In the world of malware threats, only a few rare examples can truly be considered groundbreaking and almost peerless," writes Symantec in its report about Regin. Though no one is willing to speculate on the record about Regin's source, news reports about the Belgacom and Quisquater hacks pointed a finger at GCHQ and the NSA. Kaspersky confirms that Quisqater was infected with Regin, and other researchers familiar with the Belgacom attack have told WIRED that the description of Regin fits the malware that targeted the telecom, though the malicious files used in that attack were given a different name, based on something investigators found inside the platform's main file.
regin malware security hacking exploits nsa gchq symantec espionage
-
Charted is a tool for automatically visualizing data, created by the Product Science team at Medium. Give it the link to a data file and Charted returns a beautiful, shareable chart of the data.
Nice, but it's no graphite -- pretty basic.
Why Canada should de-activate Uber
The Uber controversy is not just—or even mainly—a technology story, it's fundamentally a deregulation story; the story of a uniquely American fundamentalist free-market worldview being sold to us in the name of "car-sharing" and innovation.
-
'Design your own beer labels'
-
This is classic. I love the "Rouge":
We also wanted the Rouge to actually look like a stealth-oriented make-up artist, but our 3D artist thought the goat looked ridiculous with a pink wig and a Gucci bag, so we remade the Rouge to actually look like a Rogue.
rogue rouge goats goat-simulator funny satire mmos mmorpg games warcraft
The boss has malware, again... : talesfromtechsupport
Finally after all traditional means of infection were covered; IT started looking into other possibilities. They finally asked the Executive, "Have there been any changes in your life recently"? The executive answer "Well yes, I quit smoking two weeks ago and switched to e-cigarettes". And that was the answer they were looking for, the made in china e-cigarette had malware hard coded into the charger and when plugged into a computer's USB port the malware phoned home and infected the system. Moral of the story is have you ever question the legitimacy of the $5 dollar EBay made in China USB item that you just plugged into your computer? Because you should, you damn well should.
(Via Elliot)via:elliot malware e-cigarettes cigarettes smoking china risks
Update on Azure Storage Service Interruption
As part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this "flighting," as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues.
I'm really surprised MS deployment procedures allow a change to be rolled out globally across multiple regions on a single day. I suspect they soon won't.change-management cm microsoft outages postmortems azure deployment multi-region flighting azure-storage
AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems - YouTube
This is a really solid talk -- not surprising, alv@ is one of the speakers!
"AWS and Amazon.com operate some of the world's largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives -- DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS. "
Slides: http://www.slideshare.net/AmazonWebServices/spot302-under-the-covers-of-aws-core-distributed-systems-primitives-that-power-our-platform-aws-reinvent-2014scale scaling aws amazon dht logging data-structures distcomp via:marc-brooker dynamodb s3
How Curiosity, Luck, and the Flip of a Switch Saved the Moon Program | Motherboard
"SCE to off?" someone said. The switch was so obscure that neither of his bosses knew what he was talking about. "What the hell's that," blurted out Gerald Carr, who was in charge of communicating with the capsule. The rookie flight director, Gerry Griffin, didn't know either. Sixty seconds had passed since the initial lightning strike. No one else knew what to do. The call to abort was fast approaching. Finally, Carr reluctantly gave the order in a voice far cooler than the moment. "Apollo 12, Houston, try SCE to Auxiliary, over."
spaceflight stories apollo sce-to-aux power lightning weather outages simulation training nasa
Building a complete Tweet index
Twitter's new massive-scale twitter search backend. Sharding galore
The Oral History Of The Poop Emoji (Or, How Google Brought Poop To America)
'I went over to Japan right around the time Takeshi was deciding which emoji were going to make it into the first cut of Gmail emoji. The [PILE_OF_POO emoji] was absolutely one of the necessary emoji that Takeshi said we have to have. There was actually conflict because there were people back at headquarters who had no idea what emoji were, and thought that having an animated [turd] in their Gmail was offensive.' '[The poop emoji] got very popular when a comic called "Dr. Slump" was broadcast in Japan back to the '90s. Such poop was not an object to be disliked, but it had a funny meaning. This was a very popular comedy animation where a girl played a trick on other people using the poop. The poop was this funny object to play with. It was never serious.' 'In Japanese that's called "unchi." It's a child word with a benign meaning. '
culture emoji google pile-of-poo turd poo japan gmail unchi dr-slump
LUNAR MISSION ONE: A new lunar mission for everyone. by Lunar Missions Ltd — Kickstarter
We plan to send an unmanned robotic landing module to the South Pole of the Moon – an area unexplored by previous missions. We're going to use pioneering technology to drill down to a depth of at least 20m – 10 times deeper than has ever been drilled before – and potentially as deep as 100m. By doing this, we will access lunar rock dating back up to 4.5 billion years to discover the geological composition of the Moon, the ancient relationship it shares with our planet and the effects of asteroid bombardment. Ultimately, the project will improve scientific understanding of the early solar system, the formation of our planet and the Moon, and the conditions that initiated life on Earth.
Kickstarter-funded -- UKP 600k goal. Just in time for xmas!Flow, a new static type checker for JavaScript
Unlike the (excellent) Typescript, it'll infer types:
Flow's type checking is opt-in — you do not need to type check all your code at once. However, underlying the design of Flow is the assumption that most JavaScript code is implicitly statically typed; even though types may not appear anywhere in the code, they are in the developer's mind as a way to reason about the correctness of the code. Flow infers those types automatically wherever possible, which means that it can find type errors without needing any changes to the code at all. On the other hand, some JavaScript code, especially frameworks, make heavy use of reflection that is often hard to reason about statically. For such inherently dynamic code, type checking would be too imprecise, so Flow provides a simple way to explicitly trust such code and move on. This design is validated by our huge JavaScript codebase at Facebook: Most of our code falls in the implicitly statically typed category, where developers can check their code for type errors without having to explicitly annotate that code with types.
facebook flow javascript coding types type-inference ocaml typescript
Exactly-Once Delivery May Not Be What You Want
An extremely good explanation from Marc Brooker that exactly-once delivery in a distributed system is very hard.
And so on. There's always a place to slot in one more turtle. The bad news is that I'm not aware of a nice solution to the general problem for all side effects, and I suspect that no such solution exists. On the bright side, there are some very nice solutions that work really well in practice. The simplest is idempotence. This is a very simple idea: we make the tasks have the same effect no matter how many times they are executed.
architecture messaging queues exactly-once-delivery reliability fault-tolerance distcomp marc-brooker
-
This topographic map represents Ireland. It is designed for "hillwalking". The contour lines are extracted from SRTM public data provided by NASA. These files contain a digitized ground represented by points. The sample rate defines a grid resolution for Ireland around 90m in northing and 60m in easting. In major cases, digitized points do not correspond with summits. Carrauntoohil (1039m, the highest summit of Ireland) does not appear in SRTM data. The altitude reaches only 1018m. Data were obtain from space with a radar. Because of the relative position between the radar and the earth, a shadow appears in some conditions (along ridges, behind summits...). This shadow matches with a gap in data (Imagine you with a flashlight in a dark room. It is hard to see what is in shadow). To close these gaps, you need other data or you can do interpolation. The second solution is applied in our case. There is one square degree per SRTM file with a sample rate of 1200x1200 points/square degree at Ireland latitude. [...] All in all you obtain contour lines pretty sufficient for walking.
hillwalking walking ireland gps garmin open-data srtm maps hiking via:alan
-
Marcus Ramberg says: 'If you have a chromecast and you're not using castnow, I don't know what is wrong with you.'
The Infinite Hows, instead of the Five Whys
John Allspaw with an interesting assertion that we need to ask "how", not "why" in five-whys postmortems:
"Why?" is the wrong question. In order to learn (which should be the goal of any retrospective or post-hoc investigation) you want multiple and diverse perspectives. You get these by asking people for their own narratives. Effectively, you're asking "how?" Asking "why?" too easily gets you to an answer to the question "who?" (which in almost every case is irrelevant) or "takes you to the 'mysterious' incentives and motivations people bring into the workplace." Asking "how?" gets you to describe (at least some) of the conditions that allowed an event to take place, and provides rich operational data.
ops five-whys john-allspaw questions postmortems analysis root-causes
the JVM now supports globbing in classpath specifications
hooray, no more uberjars or monster commandlines!
-
'From 19 Nov, 2014 00:52 to 05:50 UTC a subset of customers using Storage, Virtual Machines, SQL Geo-Restore, SQL Import/export, Websites, Azure Search, Azure Cache, Management Portal, Service Bus, Event Hubs, Visual Studio, Machine Learning, HDInsights, Automation, Virtual Network, Stream Analytics, Active Directory, StorSimple and Azure Backup Services in West US and West Europe experienced connectivity issues. This incident has now been mitigated.' There was knock-on impact until 11:00 UTC (storage in N Europe), 11:45 UTC (websites, West Europe), and 09:15 UTC (storage, West Europe), from the looks of things. Should be an interesting postmortem.
What's the probability of a hash collision?
Handy calculator
probability hashing hashes collision risk md5 sha sha1 calculators
-
Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps. Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as WAH, EWAH or Concise. In some instances, they can be hundreds of times faster and they often offer significantly better compression. Roaring bitmaps are used in Apache Lucene (as of version 5.0 using an independent implementation) and Apache Spark (as of version 1.2).
bitmaps bitsets sets data-structures bits compression lucene spark daniel-lemire algorithms
'Histogram-based Outlier Score (HBOS): A fast Unsupervised Anomaly Detection Algorithm' [PDF]
'Unsupervised anomaly detection is the process of finding outliers in data sets without prior training. In this paper, a histogram-based outlier detection (HBOS) algorithm is presented, which scores records in linear time. It assumes independence of the features making it much faster than multivariate approaches at the cost of less precision. A comparative evaluation on three UCI data sets and 10 standard algorithms show, that it can detect global outliers as reliable as state-of-the-art algorithms, but it performs poor on local outlier problems. HBOS is in our experiments up to 5 times faster than clustering based algorithms and up to 7 times faster than nearest-neighbor based methods.'
histograms anomaly-detection anomalies machine-learning algorithms via:paperswelove outliers unsupervised-learning hbos
Stupid Projects From The Stupid Hackathon
Amazing.
iPad On A Face by Cheryl Wu is a telepresence robot, except it's a human with an iPad on his or her face.
funny hacking stupid hackathons ipad-on-a-face telepresence hacks via:hn
FBI's "Suicide Letter" to Dr. Martin Luther King, Jr., and the Dangers of Unchecked Surveillance
The entire letter could have been taken from a page of GCHQ's Joint Threat Research and Intelligence Group (JTRIG)—though perhaps as an email or series of tweets. The British spying agency GCHQ is one of the NSA's closest partners. The mission of JTRIG, a unit within GCHQ, is to "destroy, deny, degrade [and] disrupt enemies by discrediting them." And there's little reason to believe the NSA and FBI aren't using such tactics. The implications of these types of strategies in the digital age are chilling. Imagine Facebook chats, porn viewing history, emails, and more made public to discredit a leader who threatens the status quo, or used to blackmail a reluctant target into becoming an FBI informant. These are not far-fetched ideas. They are the reality of what happens when the surveillance state is allowed to grow out of control, and the full King letter, as well as current intelligence community practices illustrate that reality richly.
-
A nice new concurrency primitive from Gil Tene:
Have you ever had a need for logging or analyzing data that is actively being updated? Have you ever wanted to do that without stalling the writers (recorders) in any way? If so, then WriterReaderPhaser is for you. I'm not talking about logging messages or text lines here. I'm talking about data. Data larger than one word of memory. Data that holds actual interesting state. Data that keeps being updated, but needs to be viewed in a stable and coherent way for analysis or logging. Data like frame buffers. Data like histograms. Data like usage counts. Data that changes.
see also Left-Right: http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.htmlphasers data-structures concurrency primitives algorithms performance wait-free
3D Secure and Verified By Visa to be canned
Yay.
Mastercard and Visa are removing the need for users to enter their passwords for identity confirmation as part of a revamp of the existing (oft-criticised) 3-D Secure scheme. The arrival of 3D Secure 2.0 next year will see the credit card giants moving away from the existing system of secondary static passwords to authorise online purchases, as applied by Verified by Visa and MasterCard SecureCode, towards a next-gen system based on more secure biometric and token-based prompts.
(via Gordon)via:gsyme verified-by-visa 3d-secure mastercard visa credit-cards authentication authorization win passwords
Aeron: Do we really need another messaging system? - High Scalability
excellent writeup on Aeron
aeron messing libraries java martin-thompson performance mechanical-sympathy queueing ipc tcp
IAB Statement on Internet Confidentiality
Newly designed protocols should prefer encryption to cleartext operation. There may be exceptions to this default, but it is important to recognize that protocols do not operate in isolation. Information leaked by one protocol can be made part of a more substantial body of information by cross-correlation of traffic observation. There are protocols which may as a result require encryption on the Internet even when it would not be a requirement for that protocol operating in isolation. We recommend that encryption be deployed throughout the protocol stack since there is not a single place within the stack where all kinds of communication can be protected.
Wow. so much for IPSecipsec iab ietf snowden surveillance crypto protocols internet
-
FB goes public with its take on the Clos network-based datacenter network architecture
networking scaling facebook clos-networks fabrics datacenters network-architecture
/dev/full - Wikipedia, the free encyclopedia
This is handy! 'In Linux, /dev/full or the always full device[1][2] is a special file that always returns the error code ENOSPC (meaning "No space left on device") on writing, and provides an infinite number of null characters to any process that reads from it (similar to /dev/zero). This device is usually used when testing the behaviour of a program when it encounters a "disk full" error.'
dev /dev/full filesystems devices linux testing enospc error-handling
Netty: Using as a generic library
Some cool stuff that comes along with Netty: an improved ByteBuffer, a thread-local object pool, a hashed-wheel Timer, and some nice mechanical-sympathy utils.
mechanical-sympathy netty java bytebuffer object-pools data-structures hashed-wheel-timer algorithms timers
Should Airplanes Be Flying Themselves?
Excellent Vanity Fair article on the AF447 disaster, covering pilots' team-leadership skills, Clipper Skippers, Alternate Law, and autopilot design: 'There is an old truth in aviation that the reasons you get into trouble become the reasons you don't get out of it.' Also interesting: 'The best pilots discard the [autopilot] automation naturally when it becomes unhelpful, and again there appear to be some cultural traits involved. Simulator studies have shown that Irish pilots, for instance, will gleefully throw away their crutches, while Asian pilots will hang on tightly. It's obvious that the Irish are right, but in the real world Sarter's advice is hard to sell. The automation is simply too compelling. The operational benefits outweigh the costs. The trend is toward more of it, not less. And after throwing away their crutches, many pilots today would lack the wherewithal to walk.' (via Gavin Sheridan)
airlines automation flight flying accidents post-mortems af447 air-france autopilot alerts pilots team-leaders clipper-skippers alternate-law
-
MetricsGraphics.js is a library built on top of D3 that is optimized for visualizing and laying out time-series data. It provides a simple way to produce common types of graphics in a principled, consistent and responsive way. The library currently supports line charts, scatterplots and histograms as well as features like rug plots and basic linear regression.
Only 10% of serious cycling injuries in Ireland were recorded by Gardai
The Bedford Report for the HSE in 2011 showed that only approximately 10% of serious injuries (with hospital admission to a bed) incurred by cyclists in road traffic collisions were recorded by Gardai. If a cyclist is knocked off his/her bike from impact with a motorised vehicle that is a potential criminal offence if serious injury results. Cyclists expect all such RTCs to be properly and fully investigated and recorded with appropriate follow-up. That clearly is not happening at present. Acute hospitals need to document all admission cases arising from cyclist RTCs and inform the Gardai of them.
garda police ireland cycling injuries accidents reporting data bedford-report hse hospital
-
an [XPath-style] query language for JSON. You can extract and transform elements from a JSON document.
Supported by the "aws" CLI tool, and in boto.
"Aeron: High-Performance Open Source Message Transport" [slides, PDF]
a new networked pub/sub library from Martin "Disruptor" Thompson, based around a replicated, persistent log of messages, with exceptionally low latency. Apache-licensed. Very similar to the realtime messaging stack we've built in Swrve. ;) https://github.com/real-logic/Aeron
realtime messaging pub-sub ipc queues transports martin-thompson slides latencies open-source java libraries
How "Computer Geeks" replaced "Computer Girls"
As historian Nathan Ensmenger explained to a Stanford audience, as late as the 1960s many people perceived computer programming as a natural career choice for savvy young women. Even the trend-spotters at Cosmopolitan Magazine urged their fashionable female readership to consider careers in programming. In an article titled "The Computer Girls," the magazine described the field as offering better job opportunities for women than many other professional careers. As computer scientist Dr. Grace Hopper told a reporter, programming was "just like planning a dinner. You have to plan ahead and schedule everything so that it's ready when you need it…. Women are 'naturals' at computer programming." James Adams, the director of education for the Association for Computing Machinery, agreed: "I don't know of any other field, outside of teaching, where there's as much opportunity for a woman."
Microsoft Open Sources .NET and Mono - Miguel de Icaza
Whoa. Pity they didn't do this earlier! Patent license, and all
mono dot-net languages microsoft open-source mit-license swpats patents
Life expectancy increases are due mainly to healthier children, not longer old age
Interesting -- I hadn't expected this. 'Life expectancy at birth [in the US] in 1930 was indeed only 58 for men and 62 for women, and the retirement age was 65. But life expectancy at birth in the early decades of the 20th century was low due mainly to high infant mortality, and someone who died as a child would never have worked and paid into Social Security. A more appropriate measure is probably life expectancy after attainment of adulthood.' .... 'Men who attained age 65 could expect to collect Social Security benefits for almost 13 years (and the numbers are even higher for women).' In Ireland, life expectancy at birth has increased 18.4 years since 1926 -- but life expectancy for men aged 65 (the pension age) has only increased by 3.8 years. This means that increased life expectancy figures are not particularly relevant to the "pension crunch" story. Via Fred Logue: https://twitter.com/fplogue/status/532093184646873089
via:fplogue statistics taxes life-expectancy pensions infant-mortality health 1930s
-
This is pretty awesome. All changes to a DynamoDB table can be streamed to a Kinesis stream, MySQL-replication-style. The nice bit is that it has a solid way to ensure readers won't get overwhelmed by the stream volume (since ddb tables are IOPS-rate-limited), and Kinesis has a solid way to read missed updates (since it's a Kafka-style windowed persistent stream). With this you have a pretty reliable way to ensure you're not going to suffer data loss.
iops dynamodb aws kinesis reliability replication multi-az multi-region failover streaming kafka
Help the GNOME Foundation defend the GNOME trademark
Recently Groupon announced a product with the same product name as GNOME. Groupon's product is a tablet based point of sale "operating system for merchants to run their entire operation." The GNOME community was shocked that Groupon would use our mark for a product so closely related to the GNOME desktop and technology. It was almost inconceivable to us that Groupon, with over $2.5 billion in annual revenue, a full legal team and a huge engineering staff would not have heard of the GNOME project, found our trademark registration using a casual search, or even found our website, but we nevertheless got in touch with them and asked them to pick another name. Not only did Groupon refuse, but it has now filed even more trademark applications (the full list of applications they filed can be found here, here and here). To use the GNOME name for a proprietary software product that is antithetical to the fundamental ideas of the GNOME community, the free software community and the GNU project is outrageous. Please help us fight this huge company as they try to trade on our goodwill and hard earned reputation.
gnome groupon trademark infringement open-source operating-systems ip law floss
-
'Hosted Status Pages for Your Company'. We use these guys in $work, and their service is fantastic -- it's a line of javascript in the page template which will easily allow you to add a "service degraded" banner when things go pear-shaped, along with an external status site for when things get really messy. They've done a good clean job.
Eircom have run out of network capacity
This is due in part to huge growth in the data volumes and data traffic that is transported over our network, which has exceeded our forecasted growth. We are making a number of improvements to our international connectivity which will add significant capacity and this work will be completed in the next two or three weeks.
Guess this is what happens when Amazon poach your IP network engineers. doh! More seriously though, if you're marketing eFibre heavily, shouldn't you be investing in the upstream capacity to go with it?Apple site lets you deactivate iMessage and solve your missing text problem
FINALLY.
-
From the very beginning, Isis fanatics have been up to speed on [social media]. Which raises an interesting question: how come that GCHQ and the other intelligence agencies failed to notice the rise of the Isis menace until it was upon us? Were they so busy hoovering metadata and tapping submarine cables and "mastering the internet" (as the code name of one of their projects puts it) that they didn't have time to see what every impressionable Muslim 14-year-old in the world with an internet connection could see?
gchq guardian encryption nsa isis technology social-media snooping surveillance
This Canadian Artist Halted Pipeline Development by Copyrighting His Land as a Work of Art
One of the really important pieces on my land was this white-picket fence. The picket fence is probably 100 yards or less, within 100 yards of where they wanted to build this pipeline. I [plan to] extend it 8 feet every year for the rest of my life and I've been doing that for 25 years. It got me thinking, where does this piece end? Does it end at the actual structure of the fence or the things growing around it, growing through it, that are part of the photography, the documentation of it? I realized at that point that [the fence], and the other sculptures and pieces and incursions and conceptual works, were actually integral to that piece of land and to my practice. I had not intended for it to be a political piece, it was just a piece, an idea the follow-through of which at some point became poetic, you go, "Wait a minute the fence actually stopped them!" But the fence doesn't actually enclose anything. It's just a straight line. And it's marking something that's actually unmarkable, which is time. And one day it'll be gone, as will I. The land will be changed--but it was just this crazy irony that kicked into play when I was standing there with those oil negotiators.
copyright art pipelines canada politics oil land conceptual-art ip
-
'A constant throughput, correct latency-recording variant of wrk. This is a must-have when measuring network service latency -- corrects for Coordinated Omission error:
wrk's model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received. While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.
wrk latency measurement tools cli http load-testing testing load-generation coordinated-omission gil-tene
The problem of managing schemas
Good post on the pain of using CSV/JSON as a data interchange format:
eventually, the schema changes. Someone refactors the code generating the JSON and moves fields around, perhaps renaming few fields. The DBA added new columns to a MySQL table and this reflects in the CSVs dumped from the table. Now all those applications and scripts must be modified to handle both file formats. And since schema changes happen frequently, and often without warning, this results in both ugly and unmaintainable code, and in grumpy developers who are tired of having to modify their scripts again and again.
schema json avro protobuf csv data-formats interchange data hadoop files file-formats
-
15 years ago today -- Sitescooper appeared in NTK!
Official NTK policy is that if you're not reading this in its definitive, non-proportional e-mail form, you're a fricking girl. And all the best fricking girls these days have a Palm, so JUSTIN MASON has been kindly running the Web page through his brilliant sitescooper (aka snarfnews) program, and dumping the results for download at his site. NTK is available in DOC and iSilo formats, as are all kinds of other girlish, lavender-smelling Websites you may want to read, like The Register and the Linux Weekly News. And "Dr Koop's Health News".
-
Excellent write-up of this little-known undocumented GDocs behaviour, an artifact of its operational-transformation sync mechanism
operational-transformation ot google gdocs coding docs sync undocumented reversing
Introducing Proxygen, Facebook's C++ HTTP framework
Facebook's take on libevent, I guess:
We are excited to announce the release of Proxygen, a collection of C++ HTTP libraries, including an easy-to-use HTTP server. In addition to HTTP/1.1, Proxygen (rhymes with "oxygen") supports SPDY/3 and SPDY/3.1. We are also iterating and developing support for HTTP/2. Proxygen is not designed to replace Apache or nginx — those projects focus on building extremely flexible HTTP servers written in C that offer good performance but almost overwhelming amounts of configurability. Instead, we focused on building a high performance C++ HTTP framework with sensible defaults that includes both server and client code and that's easy to integrate into existing applications. We want to help more people build and deploy high performance C++ HTTP services, and we believe that Proxygen is a great framework to do so.
c++ facebook http servers libevent https spdy proxygen libraries
Doing Constant Work to Avoid Failures
A good example of a design pattern -- by performing a relatively constant amount of work regardless of the input, we can predict scalability and reduce the risk of overload when something unexpected changes in that input
scalability scaling architecture aws route53 via:brianscanlan overload constant-load loading
Call for co-ordinated plan to combat soaring bike theft | Dublin Cycling Campaign
Bicycle theft in Ireland has doubled in Ireland since the introduction of the Bike to Work scheme in 2009. Almost 4,500 bicycle thefts[1] were reported in Dublin in 2013, but the actual number of bike thefts is likely to be in the region of 20,000 in 2013 according to Irish household surveys[2] and international experience[3,4]. The chances of a bike thief being caught is low, with a conviction rate of only 2%[5] or reported thefts. Approximately 230,000 bicycles are imported into Ireland each year[6]. "Bike theft is a low-risk, high-reward crime. If cars were being stolen at this rate there would be uproar." Says Keith Byrne, Chairperson of the Dublin Cycling Campaign. Fear of bicycle theft may discourage bicycle use and many bicycle theft victims do not buy a replacement [7,8]. "Many people give up on cycling after their bicycle is stolen and it discourages others from taking up cycling as the word about the high risk of theft spreads. We need a co-ordinated multi-agency plan to tackle bicycle theft if we are to reach the Government target of 10% of journeys by bicycle by 2020" says Keith Byrne.
Amen to that.how to run a datacenter at the South Pole
it's not easy, basically (via Tony Finch)
via:fanf antarctica datacenters icecube wipac south-pole cold ops
Amazon announces 300 jobs at Dublin base - RTÉ News
DUB6 is expanding (or is it DUB14 now? can't keep up)
The jobs will be across a variety of positions, including software engineers, technical engineers, technical managers, customer support and IT security.
-
A curated list of Docker resources.
testing latency measurements using CTRL-Z
An excellent tip from Gil "HDRHistogram" Tene:
Good example of why I always "calibrate" latency tools with ^Z tests. If ^Z results don't make sense, don't use [the] tool. ^Z test math examples: If you ^Z for half the time, Max is obvious. [90th percentile] should be 80% of the ^Z stall time.
control-z suspend unix testing latencies latency measurement percentiles tips
Announcing Confluent, A Company for Apache Kafka And Realtime Data
Jay Kreps, Neha Narkhede, and Jun Rao are leaving LinkedIn to form a Kafka-oriented realtime event processing company
realtime event-processing logs kafka streaming open-source jay-kreps jun-rao confluent
Great quote from Voldemort author Jay Kreps
"Reading papers: essential. Slavishly implementing ideas you read: not necessarily a good idea. Trust me, I wrote an Amazon Dynamo clone." Later in the discussion, on complex conflict resolution logic (as used in Dynamo, Voldemort, and Riak): "I reviewed 200 Voldemort stores, 190 used default lww conflict resolution. 10 had custom logic, all 10 of which had bugs." -- https://twitter.com/jaykreps/statuses/528292617784537088 (although IMO I'd prefer complex resolution to non-availability, when AP is required)
voldemort jay-kreps dynamo cap-theorem ap riak papers lww conflict-resolution distcomp
Rails migrations with no downtime
Ugh, Rails fail. It is impossible to drop a column from a Rails-managed table without downtime, even if nothing in the code accesses it (!!), without ugly hacks that don't even seem to work on recent versions of ActiveRecord.
Belgian and French copyright laws ban photos of EP buildings
An obscure clause in EU copyright rules means no one can publish photos of public buildings in Belgium, like the Atomium, or France's Eiffel tower at night without first asking permission from the rights owners.
Ah, copyright.
Zookeeper: not so great as a highly-available service registry
Turns out ZK isn't a good choice as a service discovery system, if you want to be able to use that service discovery system while partitioned from the rest of the ZK cluster:
I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances. This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones. What I saw was that the two other instances noticed the first server "going away", but they continued to function as they still saw a majority (66%). More interestingly the first instance noticed the other two servers "going away", dropping the ensemble availability to 33%. This caused the first server to stop serving requests to clients (not only writes, but also reads).
So: within that offline AZ, service discovery *reads* (as well as writes) stopped working due to a lack of ZK quorum. This is quite a feasible outage scenario for EC2, by the way, since (at least when I was working there) the network links between AZs, and the links with the external internet, were not 100% overlapping. In other words, if you want a highly-available service discovery system in the fact of network partitions, you want an AP service discovery system, rather than a CP one -- and ZK is a CP system. Another risk, noted on the Netflix Eureka mailing list at https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/tA9UnerrBHUJ :ZooKeeper, while tolerant against single node failures, doesn't react well to long partitioning events. For us, it's vastly more important that we maintain an available registry than a necessarily consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that's OK with us.
I guess this means that a long partition can trigger SESSION_EXPIRED state, resulting in ZK client libraries requiring a restart/reconnect to fix. I'm not entirely clear what happens to the ZK cluster itself in this scenario though. Finally, Pinterest ran into other issues relying on ZK for service discovery and registration, described at http://engineering.pinterest.com/post/77933733851/zookeeper-resilience-at-pinterest ; sounds like this was mainly around load and the "thundering herd" overload problem. Their workaround was to decouple ZK availability from their services' availability, by building a Smartstack-style sidecar daemon on each host which tracked/cached ZK data.zookeeper service-discovery ops ha cap ap cp service-registry availability ec2 aws network partitions eureka smartstack pinterest
Why We Didn't Use Kafka for a Very Kafka-Shaped Problem
A good story of when Kafka _didn't_ fit the use case:
We came up with a complicated process of app-level replication for our messages into two separate Kafka clusters. We would then do end-to-end checking of the two clusters, detecting dropped messages in each cluster based on messages that weren't in both. It was ugly. It was clearly going to be fragile and error-prone. It was going to be a lot of app-level replication and horrible heuristics to see when we were losing messages and at least alert us, even if we couldn't fix every failure case. Despite us building a Kafka prototype for our ETL — having an existing investment in it — it just wasn't going to do what we wanted. And that meant we needed to leave it behind, rewriting the ETL prototype.
cassandra java kafka scala network-partitions availability multi-region multi-az aws replication onlive
Madhumita Venkataramanan: My identity for sale (Wired UK)
If the data aggregators know everything about you -- including biometric data, healthcare history, where you live, where you work, what you do at the weekend, what medicines you take, etc. -- and can track you as an individual, does it really matter that they don't know your _name_? They legally track, and sell, everything else.
As the data we generate about ourselves continues to grow exponentially, brokers and aggregators are moving on from real-time profiling -- they're cross-linking data sets to predict our future behaviour. Decisions about what we see and buy and sign up for aren't made by us any more; they were made long before. The aggregate of what's been collected about us previously -- which is near impossible for us to see in its entirety -- defines us to companies we've never met. What I am giving up without consent, then, is not just my anonymity, but also my right to self-determination and free choice. All I get to keep is my name.
wired privacy data-aggregation identity-theft future grim biometrics opt-out healthcare data data-protection tracking
Linux kernel's Transparent Huge Pages feature causing 300ms-800ms pauses
bad news for low-latency apps. See also its impact on redis: http://antirez.com/news/84
redis memory defrag huge-pages linux kernel ops latency performance transparent-huge-pages
Please grow your buffers exponentially
Although in some cases x1.5 is considered good practice. YMMV I guess
malloc memory coding buffers exponential jemalloc firefox heap allocation
How I created two images with the same MD5 hash
I found that I was able to run the algorithm in about 10 hours on an AWS large GPU instance bringing it in at about $0.65 plus tax.
Bottom line: MD5 is feasibly attackable by pretty much anyone now.
UK museums lobbying for copyright reform with empty display cases
Great to see museums campaigning for copyright reform -- this makes perfect sense.
Display cases in the Imperial War Museum, National Library of Scotland and University of Leeds sit empty. They should contain letters from the First World War; from a young girl to her father serving as a soldier and from soldiers to their families back home. Because of current UK copyright laws the original letters cannot be displayed. At the moment the duration of copyright in certain unpublished works is to the end of the year 2039, regardless how old the work is. The Free Our History campaign wants the term of copyright protection in unpublished texts to be reduced to the author's lifetime plus 70 years.
The Roman typefaces used in Chinese and Japanese text
I am obsessed with these.
From this Ask MetaFilter post, it seems that the Roman letters are tacked onto [Chinese and Japanese] fonts almost as an afterthought, for when you need to interject a few words of English into your Chinese website, for example. Hence, they aren't really optimised for the eye of a non-Chinese writer, or perhaps aren't optimised much at all, and usually look like this. It's not one specific font as I thought.
Informed! (via Elliot)Asus trackpad driver sets the CPU speed to maximum during scrolling
LOL, hardware people writing drivers. Good reason not to buy Asus, I guess
asus fail hardware drivers throttling cpu touchpad trackpad scrolling laptops
-
'People telling people to execute arbitrary code over the network. Run code from our servers as root. But HTTPS, so it's no biggie.' YES.
Why Gandhi Is Such An Asshole In Civilization
When a player adopted democracy in Civilization, their aggression would be automatically reduced by 2. Code being code, if Gandhi went democratic his aggression wouldn't go to -1, it looped back around to the ludicrously high figure of 255, making him as aggressive as a civilization could possibly be.
civ civilization funny videogames bugs gandhi nuclear-war integers overflow
Scaling Micro-Services in Go - HighLoad++ 2014 // Speaker Deck
good talk from Hailo's Matt Heath, doing nice stuff with Go and a well-supported microservices architecture
microservices presentation go architecture hailo presentations
Chip & PIN vs. Chip & Signature
Trust US banks to fuck up their attempts at security :( US "chip-and-signature" cards are still entirely forgeable because the banks fear that consumers are too stupid to use a PIN, basically.
BK: So, I guess we should all be grateful that banks and retailers in the United States are finally taking steps to move toward chip [and signature] cards, but it seems to me that as long as these chip cards still also store cardholder data on a magnetic stripe as a backup, that the thieves can still steal and counterfeit this card data — even from chip cards. Litan: Yes, that's the key problem for the next few years. Once mag stripe goes away, chip-and-PIN will be a very strong solution. The estimates are now that by the end of 2015, 50 percent of the cards and terminals will be chip-enabled, but it's going to be several years before we get closer to full compliance. So, we're probably looking at about 2018 before we can start making plans to get rid of the magnetic stripe on these cards.
magstripe banks banking chip-and-pin security brian-krebs chip-and-signature
-
Turns out there are a few bugs in EMR's S3 support, believe it or not. 1. 'Consider disabling Hadoop's speculative execution feature if your cluster is experiencing Amazon S3 concurrency issues. You do this through the mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution configuration settings. This is also useful when you are troubleshooting a slow cluster.' 2. Upgrade to AMI 3.1.0 or later, otherwise retries of S3 ops don't work.
-
Stephanie Dean on Amazon's approach to CMs. This is solid gold advice for any company planning to institute a sensible technical change management process
ops tech process changes change-management bureaucracy amazon stephanie-dean infrastructure
Stephanie Dean on event management and incident response
I asked around my ex-Amazon mates on twitter about good docs on incident response practices outside the "iron curtain", and they pointed me at this blog (which I didn't realise existed). Stephanie Dean was the front-line ops manager for Amazon for many years, over the time where they basically *fixed* their availability problems. She since moved on to Facebook, Demonware, and Twitter. She really knows her stuff and this blog is FULL of great details of how they ran (and still run) front-line ops teams in Amazon.
ops incident-response outages event-management amazon stephanie-dean techops tos sev1
-
Carlos Baquero presents several operation, state-based CRDTs for use in AP systems like Voldemort and Riak
ap cap-theorem crdts ricon carlos-baquero data-structures distcomp
Brownout: building more robust cloud applications
Applications can saturate – i.e. become unable to serve users in a timely manner. Some users may experience high latencies, while others may not receive any service at all. The authors argue that it is better to downgrade the user experience and continue serving a larger number of clients with reasonable latency. "We define a cloud application as brownout compliant if it can gradually downgrade user experience to avoid saturation." This is actually very reminiscent of circuit breakers, as described in Nygard's 'Release It!' and popularized by Netflix. If you're already designing with circuit breakers, you've probably got all the pieces you need to add brownout support to your application relatively easily. "Our work borrows from the concept of brownout in electrical grids. Brownouts are an intentional voltage drop often used to prevent blackouts through load reduction in case of emergency. In such a situation, incandescent light bulbs dim, hence originating the term." "To lower the maintenance effort, brownouts should be automatically triggered. This enables cloud applications to rapidly and robustly avoid saturation due to unexpected environmental changes, lowering the burden on human operators."
This is really similar to the Circuit Breaker pattern -- in fact it feels to me like a variation on that, driven by measured latencies of operations/requests. See also http://blog.acolyer.org/2014/10/27/improving-cloud-service-resilience-using-brownout-aware-load-balancing/ .circuit-breaker patterns brownout robustness reliability load latencies degradation
Photographs of Sellafield nuclear plant prompt fears over radioactive risk
"Slow-motion Chernobyl", as Greenpeace are calling it. You thought legacy code was a problem? try legacy Magnox fuel rods.
Previously unseen pictures of two storage ponds containing hundreds of highly radioactive fuel rods at the Sellafield nuclear plant show cracked concrete, seagulls bathing in the water and weeds growing around derelict machinery. But a spokesman for owners Sellafield Ltd said the 60-year-old ponds will not be cleaned up for decades, despite concern that they are in a dangerous state and could cause a large release of radioactive material if they are allowed to deteriorate further. "The concrete is in dreadful condition, degraded and fractured, and if the ponds drain, the Magnox fuel will ignite and that would lead to a massive release of radioactive material," nuclear safety expert John Large told the Ecologist magazine. "I am very disturbed at the run-down condition of the structures and support services. In my opinion there is a significant risk that the system could fail.
energy environment nuclear uk sellafield magnox seagulls time long-now
The man who made a game to change the world
An interview with Richard Bartle, the creator of MUD, back in 1978.
Perceiving the different ways in which players approached the game led Bartle to consider whether MMO players could be classified according to type. "A group of admins was having an argument about what people wanted out of a MUD in about 1990," he recalls. "This began a 200-long email chain over a period of six months. Eventually I went through everybody's answers and categorised them. I discovered there were four types of MMO player. I published some short versions of them then, when the journal of MUD research came out I wrote it up as a paper." The so-called Bartle test, which classifies MMO players as Achievers, Explorers, Socialisers or Killers (or a mixture thereof) according to their play-style remains in widespread use today. Bartle believes that you need a healthy mix of all dominant types in order to maintain a successful MMO ecosystem. "If you have a game full of Achievers (players for whom advancement through a game is the primary goal) the people who arrive at the bottom level won't continue to play because everyone is better than them," he explains. "This removes the bottom tier and, over time, all of the bottom tiers leave through irritation. But if you have Socialisers in the mix they don't care about levelling up and all of that. So the lowest Achievers can look down on the Socialisers and the Socialisers don't care. If you're just making the game for Achievers it will corrode from the bottom. All MMOs have this insulating layer, even if the developers don't understand why it's there."
Testing fork time on AWS/Xen infrastructure
Redis uses forking to perform persistence flushes, which means that once every 30 minutes it performs like crap (and kills the 99th percentile latency). Given this, various Redis people have been benchmarking fork() times on various Xen platforms, since Xen has a crappy fork() implementation
A Teenager Gets Grilled By Her Dad About Why She's Not That Into Coding
Jay Rosen interviews his 17-year-old daughter. it's pretty eye-opening. Got to start them early!
culture tech coding girls women feminism teenagers school jay-rosen stem
-
a new "types for Javascript" framework, from the team behind Angular.js -- they plan to "harmonize" it with TypeScript and pitch it for standardization, which would be awesome. (via Rob Clancy)
via:robc atscript javascript typescript types languages coding google angular
Carbon vs Megacarbon and Roadmap ? · Issue #235 · graphite-project/carbon
Carbon is a great idea, but fundamentally, twisted doesn't do what carbon-relay or carbon-aggregator were built to do when hit with sustained and heavy throughput. Much to my chagrin, concurrency isn't one of python's core competencies.
+1, sadly. We are patching around the edges with half-released third-party C rewrites in our graphite setup, as we exceed the scale Carbon can support.Most page loads will experience the 99th percentile response latency
MOST of the page view attempts will experience the 99%'lie server response time in modern web applications. You didn't read that wrong.
-
The next Pub Standards, on Thursday 13th November, will be the last one. When I started Pub Standards in August 2010, there wasn't very many meetups for people who build apps, interfaces and businesses. These days, there are loads! I don't feel that Pub Standards is needed anymore. It served it's purpose -- other meetups were formed, startups were founded, projects were created and people got hired. We had a good run :)
Bay Point print by Grant Haffner
$50 print (plus shipping of course), 16" x 16"
Smart Clients, haproxy, and Riak
Good, thought-provoking post on good client library approaches for complex client-server systems, particularly distributed stores like Voldemort or Riak. I'm of the opinion that a smart client lib is unavoidable, and in fact essential, since the clients are part of the distributed system, personally.
clients libraries riak voldemort distsys haproxy client-server storage
David Malone planning a commemoration of Dublin Mean Time next year
Dublin had its own time zone, 25 minutes off what would become GMT, until 1916
1916 dublin rising time dublin-mean-time dmt gmt perfidious-albion dunsink
-
a Riak-based clone of Roshi, the CRDT server built on top of Redis. some day I'll write up the CRDT we use on top of Voldemort in $work. Comments: https://lobste.rs/s/tim5xc
Vodafone UK, Verizon add mandatory device-tracking token on all web requests
'Verizon Wireless is monitoring users' mobile internet traffic, using a token slapped onto web requests, to facilitate targeted advertising even if a user has opted out. The unique identifier token header (UIDH) was launched two years ago, and has caused an uproar in tech circles after it was re-discovered Thursday by Electronic Frontier Foundation staffer Jacob Hoffman-Andrews. The Relevant Mobile Advertising program, under which the UIDH was used, allowed a restaurant to advertised to locals only or for retail websites to promote to previous visitors, according to Verizon Wireless.'
uidh verizon vodafone privacy tracking http cookies advertising
-
'In many networking systems, Bloom filters are used for high-speed set membership tests. They permit a small fraction of false positive answers with very good space efficiency. However, they do not permit deletion of items from the set, and previous attempts to extend "standard" Bloom filters to support deletion all degrade either space or performance. We propose a new data structure called the cuckoo filter that can replace Bloom filters for approximate set member- ship tests. Cuckoo filters support adding and removing items dynamically while achieving even higher performance than Bloom filters. For applications that store many items and target moderately low false positive rates, cuckoo filters have lower space overhead than space-optimized Bloom filters. Our experimental results also show that cuckoo filters out-perform previous data structures that extend Bloom filters to support deletions substantially in both time and space.'
algorithms cs coding cuckoo-filters bloom-filters sets data-structures
Irish government in favour of ISDS court-evasion for multinationals
This has _already_ been used to trump national law. As Simon McGarr noted at https://twitter.com/Tupp_Ed/statuses/526103760041680898 : 'Philip Morris initiated a dispute under the Australia-Hong Kong Bilateral Investment Treaty to force #plainpacks repeal and compensation'. "Plain packs" anti-smoking is being bitterly fought at the moment here in Ireland. More from the US point of view: http://www.washingtonpost.com/opinions/harold-meyerson-allowing-foreign-firms-to-sue-nations-hurts-trade-deals/2014/10/01/4b3725b0-4964-11e4-891d-713f052086a0_story.html : 'The Obama administration's insistence on ISDS may please Wall Street, but it threatens to undermine some of the president's landmark achievements in curbing pollution and fighting global warming, not to mention his commitment to a single standard of justice. It's not worthy of the president, and he should join Europe in scrapping it.'
isds national-law law ireland sovereignty multinationals philip-morris us-politics eu free-trade
Jonathan Bergknoff: Building good docker images
Good advice
Game Day Exercises at Stripe: Learning from `kill -9`
We've started running game day exercises at Stripe. During a recent game day, we tested failing over a Redis cluster by running kill -9 on its primary node, and ended up losing all data in the cluster. We were very surprised by this, but grateful to have found the problem in testing. This result and others from this exercise convinced us that game days like these are quite valuable, and we would highly recommend them for others.
Excellent post. Game days are a great idea. Also: massive Redis clustering failThe Laborers Who Keep Dick Pics and Beheadings Out of Your Facebook Feed | WIRED
"Everybody hits the wall, generally between three and five months," says a former YouTube content moderator I'll call Rob. "You just think, 'Holy shit, what am I spending my day doing? This is awful.'"
facebook wired beheadings moderation nsfw google youtube social-media filtering porn abuse
PSA: don't run 'strings' on untrusted files (CVE-2014-8485)
ffs.
Perhaps simply by the virtue of being a part of that bundle, the strings utility tries to leverage the common libbfd infrastructure to detect supported executable formats and "optimize" the process by extracting text only from specific sections of the file. Unfortunately, the underlying library can be hardly described as safe: a quick pass with afl (and probably with any other competent fuzzer) quickly reveals a range of troubling and likely exploitable out-of-bounds crashes due to very limited range checking
Hungary plans new tax on Internet traffic, public calls for rally
37p (EUR0.46) per GB -- that's a lot of money! bloody hell
-
This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.
"Viewstamped Replication Revisited", Liskov and Cowling [pdf]
classic replication paper, via aphyr: 'This paper presents an updated version of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group. The paper also describes a number of important optimizations and presents a protocol for handling reconfigurations that can change both the group membership and the number of failures the group is able to handle.'
-
Holy shit we are living in the future.
BioBrick parts are DNA sequences which conform to a restriction-enzyme assembly standard.[1][2] These Lego-like building blocks are used to design and assemble synthetic biological circuits, which would then be incorporated into living cells such as Escherichia coli cells to construct new biological systems.[3] Examples of BioBrick parts include promoters, ribosomal binding sites (RBS), coding sequences and terminators.
(via Soren)via:sorenrags biobricks fabrication organisms artificial-life biology e-coli genetic-engineering
Is Docker ready for production? Feedbacks of a 2 weeks hands on
I have to agree with this assessment -- there are a lot of loose ends still for production use of Docker in a SOA stack environment:
From my point of view, Docker is probably the best thing I've seen in ages to automate a build. It allows to pre build and reuse shared dependencies, ensuring they're up to date and reducing your build time. It avoids you to either pollute your Jenkins environment or boot a costly and slow Virtualbox virtual machine using Vagrant. But I don't feel like it's production ready in a complex environment, because it adds too much complexity. And I'm not even sure that's what it was designed for.
docker complexity devops ops production deployment soa web-services provisioning networking logging
Load testing Apache Kafka on AWS
This is a very solid benchmarking post, examining Kafka in good detail. Nicely done. Bottom line:
I basically spend 2/3 of my work time torture testing and operationalizing distributed systems in production. There's some that I'm not so pleased with (posts pending in draft forever) and some that have attributes that I really love. Kafka is one of those systems that I pretty much enjoy every bit of, and the fact that it performs predictably well is only a symptom of the reason and not the reason itself: the authors really know what they're doing. Nothing about this software is an accident. Performance, everything in this post, is only a fraction of what's important to me and what matters when you run these systems for real. Kafka represents everything I think good distributed systems are about: that thorough and explicit design decisions win.
Tesco Hudl 2 review: a lot of tablet for the money
wow, an actually quite-good cheapo Android tablet from Tesco for UKP65 of Clubcard vouchers, recommended by conoro. Good for the kids
[KAFKA-1555] provide strong consistency with reasonable availability
Major improvements for Kafka consistency coming in 0.8.2; replication to multiple in-sync replicas, controlled by a new "min.isr" setting
Falsehoods programmers believe about time
I have repeatedly been confounded to discover just how many mistakes in both test and application code stem from misunderstandings or misconceptions about time. By this I mean both the interesting way in which computers handle time, and the fundamental gotchas inherent in how we humans have constructed our calendar — daylight savings being just the tip of the iceberg. In fact I have seen so many of these misconceptions crop up in other people's (and my own) programs that I thought it would be worthwhile to collect a list of the more common problems here.
See also the follow-up: http://infiniteundo.com/post/25509354022/more-falsehoods-programmers-believe-about-time-wisdom (via Marc)via:marcomorain time dates timezones coding gotchas calendar bugs
Landlords not liable for tenants' water bills
What an utter fuckup. Business as usual for Irish Water:
However the spokeswoman said application packs for rented dwellings would be addressed to the landlord, at the landlord's residence, and it would be the landlord's responsibility to ensure the tenant received the application pack. Bills are to be issued quarterly, but as Irish Water will have the tenant's PPS number, the utility firm will be able to pursue the tenant for any arrears and even apply any arrears to new accounts, when the tenant moves to a new address. Last week landlords had expressed concern over potential arrears, the liability for them and the possibility of being used as collection agents by Irish Water.
Irish Water responds to landlords' questions
ugh, what a mess....
* Every rental unit in the State is to get a pack addressed personally to the occupant. If Irish Water does not have details of a tenant, the pack will be addressed to 'The Occupier' * Packs will only be issued to individual rental properties in so far as Irish Water is aware of them * Landlords can contact Irish Water to advise they have let a property * Application Packs are issued relative to the information on the Irish Water mailing list. If this is incorrect or out of date, landlords can contact Irish Water to have the information adjusted *Irish Water will contact known landlords after the initial customer application campaign, to advise of properties for which no application has been received * Irish Water said that when a household is occupied the tenant is liable and when vacant the owner is liable. Both should advise Irish Water of change of status to the property - the tenant to cease liability, the landlord to take it up. Either party may take a reading and provide it to Irish Water, alternatively Irish Water will bill on average consumption, based on the date of change.
irish-water water ireland liability bills landlords tenancy rental
The Future Of The Culture Wars Is Here, And It's Gamergate
Like, say, the Christian right, which came together through the social media of its day — little-watched television broadcasts, church bulletins, newsletters—or the Tea Party, which found its way through self-selection on social media and through back channels, Gamergate, in the main, comprises an assortment of agitators who sense which way the winds are blowing and feel left out. It has found a mobilizing event, elicited response from the established press, and run a successful enough public relations campaign that it's begun attracting visible advocates who agree with the broad talking points and respectful-enough coverage from the mainstream press. If there is a ground war being waged, as the movement's increasingly militaristic rhetoric suggests, Gamergate is fighting largely unopposed. A more important resemblance to the Tea Party, though, is in the way in which it's focused the anger of people who realize the world is changing, and not necessarily to their benefit.
culture gaming journalism gamergate tea-party grim-meathook-future culture-wars misogyny
Facebook and Apple Offer Egg-Freezing Perk So Women Never Stop Working
Grim.
grim-meathook-future egg-freezing perks apple facebook work life children work-life-balance
BBC News - South Korean ID system to be rebuilt from scratch
There are several reasons that the ID cards have proved so easy to steal: Identity numbers started to be issued in the 1960s and still follow the same pattern. The first few digits are the user's birth date, followed by either a one for male or two for female; Their usage across different sectors makes them master keys for hackers, say experts; If details are leaked, citizens are unable to change them
via Tony Finch.
Dublin's Best-Kept Secret: Blas Cafe
looks great, around the corner from Cineworld on King's Inn St, D1
"Meta-Perceptual Helmets For The Dead Zoo"
with Neil McKenzie, Nov 9-16 2014, in the National History Museum in Dublin: 'These six helmets/viewing devices start off by exploring physical conditions of viewing: if we have two eyes, they why is our vision so limited? Why do we have so little perception of depth? Why don't our two eyes offer us two different, complementary views of the world around us? Why can't they extend from our body so we can see over or around things? Why don't they allow us to look behind and in front at the same time, or sideways in both directions? Why can't our two eyes simultaneously focus on two different tasks? Looking through Michael Land's defining work Animal Eyes, we see that nature has indeed explored all of these possibilities: a Hammerhead Shark has hyper-stereo vision; a horse sees 350° around itself; a chameleon has separately rotatable eyes… The series of Meta-Perceptual Helmets do indeed explore these zoological typologies: proposing to humans the hyper-stereo vision of the hammerhead shark; or the wide peripheral vision of the horse; or the backward/forward vision of the chameleon… but they also take us into the unnatural world of mythology and literature: the Cheshire Cat Helmet is so called because of the strange lingering effect of dominating visual information such as a smile or the eyes; the Cyclops allows one large central eye to take in the world around while a second tiny hidden eye focuses on a close up task (why has the creature never evolved that can focus on denitting without constantly having to glance around?).' (via Emma)
perception helmets dublin ireland museums dead-zoo sharks eyes vision art
Grade inflation figures from Irish universities
The figures show that, between 2004 and 2013, an average of 71.7 per cent of students at TCD graduated with either a 1st or a 2.1. DCU and UCC had the next highest rate of such awards (64.3 per cent and 64.2 per cent respectively), followed by UCD (55.8 per cent), NUI Galway (54.7 per cent), Maynooth University (53.7 per cent) and University of Limerick (50.2 per cent).
tcd grades grade-inflation dcu ucc ucd ireland studies academia third-level
webrtcH4cKS: ~ coTURN: the open-source multi-tenant TURN/STUN server you were looking for
Last year we interviewed Oleg Moskalenko and presented the rfc5766-turn-server project, which is a free open source and extremely popular implementation of TURN and STURN server. A few months later we even discovered Amazon is using this project to power its Mayday service. Since then, a number of features beyond the original RFC 5766 have been defined at the IETF and a new open-source project was born: the coTURN project.
webrtc turn sturn rfc-5766 push nat stun firewalls voip servers internet
Google Online Security Blog: This POODLE bites: exploiting the SSL 3.0 fallback
Today we are publishing details of a vulnerability in the design of SSL version 3.0. This vulnerability allows the plaintext of secure connections to be calculated by a network attacker.
ouch.
how King Cormac predicted Arguing On The Internet
From The Wisdom of King Cormac:
"O Cormac, grandson of Conn", said Carbery, "What is the worst pleading and arguing?" "Not hard to tell", said Cormac. "Contending against knowledge, contending without proofs, taking refuge in bad language, a stiff delivery, a muttering speech, hair-splitting, uncertain proofs, despising books, turning against custom, shifting one's pleading, inciting the mob, blowing one's own trumpet, shouting at the top of one's voice."
internet arguing history ireland king-cormac hair-splitting shouting reddit
-
a simple, lightweight HTTP server for storing and distributing custom Debian packages around your organisation. It is designed to make it as easy as possible to use Debian packages for code deployments and to ease other system administration tasks.
Linus Torvalds and others on Linux's systemd
ZDNet's Steven J. Vaughan-Nichols on the systemd mess (via Kragen)
UK police to investigate alleged Bahraini hacking of exiles' computers
Criminal complaints have been filed in the UK against Gamma "acting as an accessory to Bahrain's illegal targeting of activists" using the FinFisher spyware
finfisher spyware malware gamma bahrain law surveillance privacy germany hacking
Tech's Meritocracy Problem — Medium
Meritocracy is a myth. And our belief in it is holding back the tech industry from getting better.
culture hiring diversity meritocracy tech software jobs work misogyny
GamerGate Death Threats - Business Insider
"It's completely insane. It's insane that you even have to say out loud that sending death threats to people who disagree with your opinion of video games is wrong. Yet here we are: Apparently, it needs to be said."
#Gamergate Trolls Aren't Ethics Crusaders; They're a Hate Group
#Gamergate, as they have treated myself and peers in our industry, is a hate group. This word, again, should not lend them any mystique or credence. Rather it should illuminate the fact that even the most nebulous and inconsistent ideas can proliferate wildly if strung onto the organizational framework of the hate group, which additionally gains a startling amount of power online. #Gamergate is a hate group, and they are all the more dismissible for it. And the longer we treat them otherwise, the longer I fear for our industry's growth.
harassment gamergate abuse twitter hate-groups gaming misogyny
Eircode postcodes will cost lives, warn emergency workers
A group representing frontline emergency staff has warned lives will be lost unless the Government reverses its decision on a new national postcode system due to be rolled out next spring. John Kidd, chairman of the Irish Fire and Emergency Services Association, said the "mainly random nature" of the Eircode system would mean errors by users would go unnoticed, as well as cause confusion and may be "catastrophic" in terms of sending services to the wrong location. [....] Neil McDonnell, general manager of the Freight Transport Association Ireland, said he understood Mr Kidd's concerns. "Take, for example, two adjacent houses in Glasnevin, Dublin," said Mr McDonnell. "One could be D11 ZXQ8, the other one D11 67TR. The four-character unique identifier is completely random, with no sequence or algorithm linking one house to the other."
The Problem Isn't Vancouver's Astronomical Housing Costs— It's the People Who Buy
Two types of people own homes in Vancouver?—?wealthy foreigners who are looking for a place to park their money, and long-time Vancouver residents who have benefited from skyrocketing equity, through no actual effort of their own. There is a simple problem with these people being the primary homeowners in any city?—?they don't actually create much value for the place they live in. A very large percentage of wealthy foreigners who "park" their money here don't actually live in Vancouver. Take a drive around most expensive areas and you'll realize the homes are empty. At most, they send their kids to live in Vancouver, learn english/go to school, and then return to their country (usually to Hong Kong). For some reason this is okay with people who live here. The amount of value added to a city from this sort of activity approaches zero. In fact, I'd argue that these people actually leech off of the system more than anything else.
Tonx's Fuss Proof Cold Brew Coffee Guide
via potentato
Yet another woman in gaming has been driven from her home by death threats
Fuck gamergate
games harassment feminism misogyny 4chan 8chan mra trolls gamergate
Game Devs on Gamergate (with images, tweets)
Welp, that's the end of my reading The Escapist. this is fucked up. 'these people say that this is a hate movement, but let's see what these white supremacists and serial harassers have to say'
ethics gaming journalism the-escapist gamergate misogyny sexism
To "patch" software comes from a physical patch applied to paper tape
hmason: TIL that the phrase software "patch" is from a physical patch applied to Mark 1 paper tape to modify the program.
It's amazing how a term like that can become so divorced from its original meaning so effectively. History!
How Videogames Like Minecraft Actually Help Kids Learn to Read | WIRED
I analyzed several chunks of The Ultimate Player's Guide using the Flesch-Kincaid Reading Ease scale, and they scored from grade 8 to grade 11. Yet in my neighborhood they're being devoured by kids in the early phases of elementary school. Games, it seems, can motivate kids to read—and to read way above their level. This is what Constance Steinkuehler, a games researcher at the University of Wisconsin-Madison, discovered. She asked middle and high school students who were struggling readers (one 11th-grade student read at a 6th-grade level) to choose a game topic they were interested in, and then she picked texts from game sites for them to read—some as difficult as first-year-college language. The kids devoured them with no help and nearly perfect accuracy. How could they do this? "Because they're really, really motivated," Steinkuehler tells me. It wasn't just that the students knew the domain well; there were plenty of unfamiliar words. But they persisted more because they cared about the task. "It's situated knowledge. They see a piece of language, a turn of phrase, and they figure it out."
When my kids are playing Minecraft, there's a constant stream of "how do you spell X?" as they craft nametags for their pets. It's great!minecraft gaming kids education spelling school reading literacy
"Gold" 4-star review from the Irish Times
Niall Heery belatedly follows up Small Engine Repair, his 2006 mumblecore critical hit, with a slightly less off-centre comedy that makes imaginative use of a smashing cast. The story skirts tragedy on its leisurely passage from mishap to misadventure, but Gold remains the sort of picture you want to hug indulgently to a welcoming bosom. It gives humanism a good name.
Go Niall! it's a great movie, go see it-
web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):
Two kinds of data are delivered by the platform: Static data provides stable information like station position, number of bike stands, payment terminal availability, etc. Dynamic data provides station state, number of available bikes, number of free bike stands, etc. Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.
Ruby API: https://github.com/oisin/bikesjcdecaux bikesharing dublin dublin-bikes api web-services http json open-data
-
my coworker JK's favourite games of 2013: Gone Home, Last Of Us, Proteus, Papers Please etc. I really want to play these, since they're all totally my bag too.
Why Amazon Has No Profits (And Why It Works)
Amazon has perhaps 1% of the US retail market by value. Should it stop entering new categories and markets and instead take profit, and by extension leave those segments and markets for other companies? Or should it keep investing to sweep them into the platform? Jeff Bezos's view is pretty clear: keep investing, because to take profit out of the business would be to waste the opportunity. He seems very happy to keep seizing new opportunities, creating new businesses, and using every last penny to do it.
amazon business strategy capex spending stocks investing retail
Spark Breaks Previous Large-Scale Sort Record – Databricks
Massive improvement over plain old Hadoop. This blog post goes into really solid techie reasons why, including:
First and foremost, in Spark 1.1 we introduced a new shuffle implementation called sort-based shuffle (SPARK-2045). The previous Spark shuffle implementation was hash-based that required maintaining P (the number of reduce partitions) concurrent buffers in memory. In sort-based shuffle, at any given point only a single buffer is required. This has led to substantial memory overhead reduction during shuffle and can support workloads with hundreds of thousands of tasks in a single stage (our PB sort used 250,000 tasks).
Also, use of Timsort, an external shuffle service to offload from the JVM, Netty, and EC2 SR-IOV.spark hadoop map-reduce batch parallel sr-iov benchmarks performance netty shuffle algorithms sort-based-shuffle timsort
UK psyops created N. Irish Satanic Panic during the Troubles - Boing Boing
During the 1970s, when Northern Ireland was gripped by near-civil-war, British military intelligence staged the evidence of "black masses" in order to create a Satanism panic among the "superstitious" Irish to discredit the paramilitaries. The secret history of imaginary Irish Satanism is documented in Black Magic and Bogeymen: Fear, Rumour and Popular Belief in the North of Ireland 1972-74, a new book from Sheffield University's Richard Jenkins, who interviewed Captain Colin Wallace, the former head of British Army "black operations" for Northern Ireland.
northern-ireland 1970s the-troubles ireland uvf ira history black-magic satanism weird fear mi5
Netflix release new code to production before completing tests
Interesting -- I hadn't heard of this being an official practise anywhere before (although we actually did it ourselves this week)...
If a build has made it [past the 'integration test' phase], it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments.
devops deployment feature-flags release testing integration-tests uat qa production ops gating netflix
-
Felix says: 'Like I said, I'd like to move it to a more general / non-personal repo in the future, but haven't had the time yet. Anyway, you can still browse the code there for now. It is not a big code base so not that hard to wrap one's mind around it. It is Apache licensed and both Kafka and Voldemort are using it so I would say it is pretty self-contained (although Kafka has not moved to Tehuti proper, it is essentially the same code they're using, minus a few small fixes missing that we added). Tehuti is a bit lower level than CodaHale (i.e.: you need to choose exactly which stats you want to measure and the boundaries of your histograms), but this is the type of stuff you would build a wrapper for and then re-use within your code base. For example: the Voldemort RequestCounter class.'
asl2 apache open-source tehuti metrics percentiles quantiles statistics measurement latency kafka voldemort linkedin
-
Great presentation about Github dev culture and building software without breakage, but still with real progress.
github programming communication process coding teams management dev-culture breakage
Unity, one gaming development platform to unite them all, up for sale
gulp
-
Syncthing is becoming Ind.ie Pulse. Pulse replaces proprietary sync and cloud services with something open, trustworthy and decentralised. Your data is your data alone and you deserve to choose where it is stored, if it is shared with some third party, and how it's transmitted over the Internet.
syncing storage cloud dropbox utilities gpl decentralization
Trouble at the Koolaid Point — Serious Pony
This is a harrowing post from Kathy Sierra, full of valid observations:
You're probably more likely to win the lottery than to get any law enforcement agency in the United States to take action when you are harassed online, no matter how visciously and explicitly. Local agencies lack the resources, federal agencies won't bother.
That to the power of ten in Ireland, too, I'd suspect. Fuck this. Troll culture is way out of control....twitter harassment feminism weev abuse trolls 4chan kathy-sierra
-
An embryonic metrics library for Java/Scala from Felix GV at LinkedIn, extracted from Kafka's metric implementation and in the new Voldemort release. It fixes the major known problems with the Meter/Timer implementations in Coda-Hale/Dropwizard/Yammer Metrics. 'Regarding Tehuti: it has been extracted from Kafka's metric implementation. The code was originally written by Jay Kreps, and then maintained improved by some Kafka and Voldemort devs, so it definitely is not the work of just one person. It is in my repo at the moment but I'd like to put it in a more generally available (git and maven) repo in the future. I just haven't had the time yet... As for comparing with CodaHale/Yammer, there were a few concerns with it, but the main one was that we didn't like the exponentially decaying histogram implementation. While that implementation is very appealing in terms of (low) memory usage, it has several misleading characteristics (a lack of incoming data points makes old measurements linger longer than they should, and there's also a fairly high possiblity of losing interesting outlier data points). This makes the exp decaying implementation robust in high throughput fairly constant workloads, but unreliable in sparse or spiky workloads. The Tehuti implementation provides semantics that we find easier to reason with and with a small code footprint (which we consider a plus in terms of maintainability). Of course, it is still a fairly young project, so it could be improved further.' More background at the kafka-dev thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E
kafka metrics dropwizard java scala jvm timers ewma statistics measurement latency sampling tehuti voldemort linkedin jay-kreps
"Quantiles on Streams" [paper, 2009]
'Chiranjeeb Buragohain and Subhash Suri: "Quantiles on Streams" in Encyclopedia of Database Systems, Springer, pp 2235–2240, 2009. ISBN: 978-0-387-35544-3', cited by Martin Kleppman in http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E as a good, short literature survey re estimating percentiles with a small memory footprint.
latency percentiles coding quantiles streams papers algorithms
-
Many Belkin routers attempt to determine if they're connected to the internet by pinging 'heartbeat.belkin.com', in a classic amateur fail move. Good reason not to run Belkin firmware if that's the level of code quality to expect
Seven deadly sins of talking about "types"
Good essay
-
An _extremely_ detailed resource about the bash bug
-
Brilliant. Nice use of an anime avatar, to boot.... 'Consulting for men who have better things to do than educate themselves about feminism. Got a question for a feminist? I would be happy to educate you! Below are my rates.'
feminism gamergate funny mansplaining misandry misogyny twitter lmgtfy
"Linux Containers And The Future Cloud" [slides]
by Rami Rosen -- extremely detailed presentation into the state of Linux containers, LXC, Docker, namespaces, cgroups, and checkpoint/restore in userspace (via lusis)
lsx docker criu namespaces cgroups linux via:lusis ops containers rami-rosen presentations
Reddit's crappy ultimatum to remote workers and offices
Reddit forces all remote workers (about half the workforce, in SLC and NYC) to move to SF, provoking a shitstorm:
In a tweet confirming the move, Reddit's CEO justified his treatment of non-San Francisco workers with a push for Optimal Teamwork to drive the New And $50M Improved Reddit forward. I shit you not. That was the actual term! (I added the New & Improved fan fiction here). So let's leave aside the debate over whether working remotely is as efficient as being in the same office all the time. Let's just focus on the size of the middle finger given to the people who work at Reddit outside the Bay Area, given the choice of forced, express relocation or a pink slip. How optimal do you think these employees will feel about leadership and the rest of the team going forward? Do you think they'll just show up at the new, apparently-not-even-in-San-Francisco-proper office with a smile from ear to ear, ready to begin in earnest on Optimal Teamwork, left-behind former colleagues be damned?
telecommuting reddit working remote-working ceos optimal-teamwork teamwork relocation
-
'I designed this jacket as a tribute to the continuing legacy of American spaceflight. I wanted it to embody everything I loved about the space program, and to eventually serve as an actual flight jacket for present-day astronauts on missions to the ISS (International Space Station). There are other "replica" flight jackets made for space enthusiasts, but I decided to come up with something boldly different, yet also completely wearable and well-suited for space.'
How did Twitter become the hate speech wing of the free speech party?
Kevin Marks has a pretty good point here:
Your tweet could win the fame lottery, and everyone on the Internet who thinks you are wrong could tell you about it. Or one of the "verified" could call you out to be the tribute for your community and fight in their Hunger Games. Say something about feminism, or race, or sea lions and you'd find yourself inundated by the same trite responses from multitudes. Complain about it, and they turn nasty, abusing you, calling in their friends to join in. Your phone becomes useless under the weight of notifications; you can't see your friends support amongst the flood. The limited tools available - blocking, muting, going private - do not match well with these floods. Twitter's abuse reporting form takes far longer than a tweet, and is explicitly ignored if friends try to help.
harassment twitter 4chan abuse feminism hate-speech gamergate sea-lions filtering social-media kevin-marks
-
A common "trick" is to claim: 'We assume network partitions can't happen. Therefore, our system is CA according to the CAP theorem.' This is a nice little twist. By asserting network partitions cannot happen, you just made your system into one which is not distributed. Hence the CAP theorem doesn't even apply to your case and anything can happen. Your system may be linearizable. Your system might have good availability. But the CAP theorem doesn't apply. [...] In fact, any well-behaved system will be "CA" as long as there are no partitions. This makes the statement of a system being "CA" very weak, because it doesn't put honesty first. I tries to avoid the hard question, which is how the system operates under failure. By assuming no network partitions, you assume perfect information knowledge in a distributed system. This isn't the physical reality.
cap erlang mnesia databases storage distcomp reliability ca postgres partitions
Integrating Kafka and Spark Streaming: Code Examples and State of the Game
Spark Streaming has been getting some attention lately as a real-time data processing tool, often mentioned alongside Apache Storm. [...] I added an example Spark Streaming application to kafka-storm-starter that demonstrates how to read from Kafka and write to Kafka, using Avro as the data format and Twitter Bijection for handling the data serialization. In this post I will explain this Spark Streaming example in further detail and also shed some light on the current state of Kafka integration in Spark Streaming. All this with the disclaimer that this happens to be my first experiment with Spark Streaming.
spark kafka realtime architecture queues avro bijection batch-processing
-
'a system for allowing servers with encrypted root file systems to reboot unattended and/or remotely.' (via Tony Finch)
via:fanf mandos encryption security server ops sysadmin linux
-
'a set of command line tools for managing Route53 DNS for an AWS infrastructure. It intelligently uses tags and other metadata to automatically create the associated DNS records.'
-
'The work, Inspeqtor which is hosted at GitHub, is far from a "clean-room" implementation. This is basically a rewrite of Monit in Go, even using the same configuration language that is used in Monit, verbatim. a. [private] himself admits that Inspeqtor is "heavily influenced" by Monit https://github.com/mperham/inspeqtor/wiki/Other-Solutions. b. This tweet by [private] demonstrate intent. https://twitter.com/mperham/status/452160352940064768 "OSS nerds: redesign and build monit in Go. Sell it commercially. Make $$$$. I will be your first customer."' IANAL, but using the same config language does not demonstrate copyright infringement...
copyright dmca tildeslash monit inspeqtor github ops oss agpl
YOU AND YOUR DAMNED GAMES, JON STONE — Why bother with #gamergate?
So what is #gamergate? #gamergate is a mob with torches aloft, hunting for any combustible dwelling and calling it a monster's lair. #gamergate is a rage train, and everyone with an axe to grind wants a ride. Its fuel is a sour mash of entitlement, insecurity, arrogance and alienation. #gamergate is a vindication quest for political intolerance. #gamergate is revenge for every imagined slight. #gamergate is Viz's Meddlesome Ratbag.
'In 1976 I discovered Ebola, now I fear an unimaginable tragedy' | World news | The Observer
An interview with the scientist who was part of the team which discovered the Ebola virus in 1976:
Other samples from the nun, who had since died, arrived from Kinshasa. When we were just about able to begin examining the virus under an electron microscope, the World Health Organisation instructed us to send all of our samples to a high-security lab in England. But my boss at the time wanted to bring our work to conclusion no matter what. He grabbed a vial containing virus material to examine it, but his hand was shaking and he dropped it on a colleague's foot. The vial shattered. My only thought was: "Oh, shit!" We immediately disinfected everything, and luckily our colleague was wearing thick leather shoes. Nothing happened to any of us.
Ebola vaccine delayed by IP spat
This is the downside of publicly-funded labs selling patent-licensing rights to private companies:
Given the urgency, it's inexplicable that one of the candidate vaccines, developed at the Public Health Agency of Canada (PHAC) in Winnipeg, has yet to go in the first volunteer's arm, says virologist Heinz Feldmann, who helped develop the vaccine while at PHAC. "It's a farce; these doses are lying around there while people are dying in Africa," says Feldmann, who now works at the Rocky Mountain Laboratories of the U.S. National Institute of Allergy and Infectious Diseases (NIAID) in Hamilton, Montana. At the center of the controversy is NewLink Genetics, a small company in Ames, Iowa, that bought a license to the vaccine's commercialization from the Canadian government in 2010, and is now suddenly caught up in what WHO calls "the most severe acute public health emergency seen in modern times." Becker and others say the company has been dragging its feet the past 2 months because it is worried about losing control over the development of the vaccine.
ip patents drugs ebola canada phac newlink-genetics health epidemics vaccines
-
"A command-line power tool for Twitter." It really is -- much better timeline searchability than the "real" Twitter UI, for example
-
We've had almost 40 years to develop, test and stockpile an Ebola vaccine. That has not happened because big pharma has been entirely focused on shareholder value and profits over safety and survival from a deadly virus. For the better part of Ebola's 38 years ? big pharma has been asleep. The question ahead is what virus or superbug will wake them up?
Ex-Apple managers reveal Cupertino's killer workload
a "firehose of emails that are just going out at 2:45 in the morning" and "if you forwarded something to one of your people at 1 o'clock in the morning and they didn't reply promptly, you got a little annoyed at them."
Fuck. That.apple workplaces work time-life-balance downtime insane sick 1am management corporate-culture
-
"Snopes for Twitter". great idea
aggregator facebook twitter snopes urban-legends news rumours
More on Facebook's "Cold Storage"
FB are using a Blu-Ray robot library
-
That the company's consistent, nearly frozen posture of disingenuous smirking means that the most perceptible "Uber problem" is almost always how it frames things, rather than how it actually operates, whether it's systematically sabotaging of competitors or using its quarter-billion-dollar war chest to relentlessly cut fares and driver pay to unsustainable levels in order to undercut existing transit systems, is remarkable in its way, though. If your company's trying to conquer the world, in the end, being a dick might be the best PR strategy of all.
uber dicks dystopia grim-meathook-future teachers california free-markets optics pr economy america
Validate SQL queries at compile-time in Rust
The sql! macro will validate that its string literal argument parses as a valid Postgres query.
Based on https://pganalyze.com/blog/parse-postgresql-queries-in-ruby.html , which links the PostgreSQL server code directly into a C extension. Mad stuff, Ted! (via Rob Clancy)The Mottly Brew - Dublin's Only Home Brew Shop
right down the road from my house! how convenient
-
'a distribution of long-living [distributed] transactions where steps may interleave, each with associated compensating transactions providing a compensation path across databases in the occurrence of a fault that may or may not compensate the entire chain back to the originator.'
distributed messaging saga patterns architecture transactions distributed-transactions distcomp
-
this is nuts. 99 cents per month for a super-cheap host -- I'm sure there's a use case for this (via Elliot)
-
Prototype is a brand new festival of play and interaction. This is your chance to experience the world from a new perspective with removable camera eyes, to jostle and joust to a Bach soundtrack whilst trying to disarm an opponent, to throw shapes as you figure out who got an invite to the silent disco, to duel with foam pool noodles, and play chase in the dark with flashlights. A unique festival that incites new types of social interaction, involving technology and the city, Prototype is a series of performances, workshops, talks, and games that spill across the city, alongside an adult playground in the heart of Temple Bar.
Project Arts Centre, 17-18 October. looks nifty Confessions of a former internet troll - Vox
I want to tell you about when violent campaigns against harmless bloggers weren't any halfway decent troll's idea of a good time — even the then-malicious would've found it too easy to be fun. When the punches went up, not down. Before the best players quit or went criminal or were changed by too long a time being angry. When there was cruelty, yes, and palpable strains of sexism and racism and every kind of phobia, sure, but when these things had the character of adolescents pushing the boundaries of cheap shock, disagreeable like that but not criminal. Not because that time was defensible — it wasn't, not really — but because it was calmer and the rage wasn't there yet. Because trolling still meant getting a rise for a laugh, not making helpless people fear for their lives because they're threatening some Redditor's self-proclaimed monopoly on reason. I want to tell you about it because I want to make sense of how it is now and why it changed.
-
Paul Hickey's gite near Toulouse, available for rent! 'a beautifully converted barn on 5 acres, wonderfully located in the French countryside. 4 Bedrooms, sleeps 2-10, Large Pool, Tennis Court, Large Trampoline, Broadband Internet, 30 Mins Toulouse/Albi, 65 Mins Carcassonne, 90 Mins Rodez'
ex-iona gites france holidays vacation-rentals vacation hotels
-
The Ello founders are positioning it as an alternative to other social networks — they won't sell your data or show you ads. "You are not the product." If they were independently-funded and run as some sort of co-op, bootstrapped until profitable, maybe that's plausible. Hard, but possible. But VCs don't give money out of goodwill, and taking VC funding — even seed funding — creates outside pressures that shape the inevitable direction of a company.
Inviso: Visualizing Hadoop Performance
With the increasing size and complexity of Hadoop deployments, being able to locate and understand performance is key to running an efficient platform. Inviso provides a convenient view of the inner workings of jobs and platform. By simply overlaying a new view on existing infrastructure, Inviso can operate inside any Hadoop environment with a small footprint and provide easy access and insight.
This sounds pretty useful.-
'Linux is becoming the thing that we adopted Linux to get away from.' Great post on the horrible complexity of systemd. It reminds me of nothing more than mid-90s AIX, which I had the displeasure of opsing for a while -- the Linux distros have taken a very wrong turn here.
linux unix complexity compatibility ops rant systemd bloat aix
Amazon's CloudSearch is now powered by Solr
good testimonial
oss-sec: Re: CVE-2014-6271: remote code execution through bash
this is truly heinous. Given that any CGI which invokes popen()/system() on a Linux system where /bin/sh is a link to bash is vulnerable, there will be a lot of vulnerable services out there (via Elliot)
via:elliottucker cgi security bash sh exploits linux popen unix
Avoiding Chef-Suck with Auto Scaling Groups - forty9ten
Some common problems which arise using Chef with ASGs in EC2, and how these guys avoided it -- they stopped using Chef for service provisioning, and instead baked AMIs when a new version was released. ASGs using pre-baked AMIs definitely works well so this makes good sense IMO.
infrastructure chef ops asg auto-scaling ec2 provisioning deployment
-
Mark "ONEList" Fletcher's back, and he's reinventing the email group! awesome.
email groups (the modern version of mailing lists) have stagnated over the past decade. Yahoo Groups and Google Groups both exude the dank air of benign neglect. Google Groups hasn't been updated in years, and some of Yahoo's recent changes have actually made Yahoo Groups worse! And yet, millions of people put up with this uncertainty and neglect, because email groups are still one of the best ways to communicate with groups of people. And I have a plan to make them even better. So today I'm launching Groups.io in beta, to bring email groups into the 21st Century. At launch, we have many features that those other services don't have, including: Integration with other services, including: Github, Google Hangouts, Dropbox, Instagram, Facebook Pages, and the ability to import Feeds into your groups. Businesses and organizations can have their own private groups on their own subdomain. Better archive organization, using hashtags. Many more email delivery options. The ability to mute threads or hashtags. Fully searchable archives, including searching within attachments. One other feature that Groups.io has that Yahoo and Google don't, is a business model that's not based on showing ads to you. Public groups are completely free on Groups.io. Private groups and organizations are very reasonably priced.
email groups communication discussion mailing-lists groups.io yahoo google google-groups yahoo-groups
The Open Source Software Engagement Award
SFU announces award for students who demonstrate excellence in contributing to an Open Source project
-
'provides citizens, public sector workers and companies with real-time information, time-series indicator data, and interactive maps about all aspects of the city. It enables users to gain detailed, up to date intelligence about the city that aids everyday decision making and fosters evidence-informed analysis.'
dublin dashboards maps geodata time-series open-data ireland
mcrouter: A memcached protocol router for scaling memcached deployments
New from Facebook engineering:
Last year, at the Data@Scale event and at the USENIX Networked Systems Design and Implementation conference , we spoke about turning caches into distributed systems using software we developed called mcrouter (pronounced "mick-router"). Mcrouter is a memcached protocol router that is used at Facebook to handle all traffic to, from, and between thousands of cache servers across dozens of clusters distributed in our data centers around the world. It is proven at massive scale — at peak, mcrouter handles close to 5 billion requests per second. Mcrouter was also proven to work as a standalone binary in an Amazon Web Services setup when Instagram used it last year before fully transitioning to Facebook's infrastructure. Today, we are excited to announce that we are releasing mcrouter's code under an open-source BSD license. We believe it will help many sites scale more easily by leveraging Facebook's knowledge about large-scale systems in an easy-to-understand and easy-to-deploy package.
This is pretty crazy -- basically turns a memcached cluster into a much more usable clustered-storage system, with features like shadowing production traffic, cold cache warmup, online reconfiguration, automatic failover, prefix-based routing, replicated pools, etc. Lots of good features.facebook scaling cache proxy memcache open-source clustering distcomp storage
DIRECT MARKETING - A GENERAL GUIDE FOR DATA CONTROLLERS
In particular:
Where you have obtained contact details in the context of the sale of a product or service, you may only use these details for direct marketing by electronic mail if the following conditions are met: the product or service you are marketing is of a kind similar to that which you sold to the customer at the time you obtained their contact details At the time you collected the details, you gave the customer the opportunity to object, in an easy manner and without charge, to their use for marketing purposes Each time you send a marketing message, you give the customer the right to object to receipt of further messages The sale of the product or service occurred not more than twelve months prior to the sending of the electronic marketing communication or, where applicable, the contact details were used for the sending of an electronic marketing communication in that twelve month period.
email spam regulations ireland law dpc marketing direct-marketing
A Linear-Time, One-Pass Majority Vote Algorithm
This algorithm, which Bob Boyer and I invented in 1980, decides which element of a sequence is in the majority, provided there is such an element.
-
tinystat is used to compare two or more sets of measurements (e.g., runs of a multiple runs of benchmarks of two possible implementations) and determine if they are statistically different, using Student's t-test. It's inspired largely by FreeBSD's ministat (written by Poul-Henning Kamp).
t-test student statistics go coda-hale tinystat stats tools command-line unix
Internet Trolls Are Narcissists, Psychopaths, and Sadists | Psychology Today
The relationship between this Dark Tetrad [of narcissism, Machiavellianism, psychopathy, and sadism] and trolling is so significant, that the authors write the following in their paper: "... the associations between sadism and GAIT (Global Assessment of Internet Trolling) scores were so strong that it might be said that online trolls are prototypical everyday sadists." [emphasis added] Trolls truly enjoy making you feel bad. To quote the authors once more (because this is a truly quotable article): "Both trolls and sadists feel sadistic glee at the distress of others. Sadists just want to have fun ... and the Internet is their playground!"
Bloody hell.trolls sadism narcissism psychopaths online trolling psychology papers
Cassandra Summit Recap: Diagnosing Problems in Production
Great runbook for C* ops
-
get page cache statistics for files.
A common question when tuning databases and other IO-intensive applications is, "is Linux caching my data or not?" pcstat gets that information for you using the mincore(2) syscall. I wrote this is so that Apache Cassandra users can see if ssTables are being cached.
linux page-cache caching go performance cassandra ops mincore fincore
75% of domestic violence victims in US shelters were spied on by their abusers using spyware
via Mikko
Alex Payne — Thoughts On Five Years of Emerging Languages
One could read the success of Go as an indictment of contemporary PLT, but I prefer to see it as a reminder of just how much language tooling matters. Perhaps even more critical, Go's lean syntax, selective semantics, and cautiously-chosen feature set demonstrate the importance of a strong editorial voice in a language's design and evolution. Having co-authored a book on Scala, it's been painful to see systems programmers in my community express frustration with the ambitious hybrid language. I've watched them abandon ship and swim back to the familiar shores of Java, or alternately into the uncharted waters of Clojure, Go, and Rust. A pity, but not entirely surprising if we're being honest with ourselves. Unlike Go, Scala has struggled with tooling from its inception. More than that, Scala has had a growing editorial problem. Every shop I know that's been successful with Scala has limited itself to some subset of the language. Meanwhile, in pursuit of enterprise developers, its surface area has expanded in seemingly every direction. The folks behind Scala have, thankfully, taken notice: upcoming releases are promised to focus on simplicity, clarity, and better tooling.
Texas Judge References 'The Big Lebowski'
"The First Amendment of the U.S. Constitution is similarly suspicious of prior restraints," wrote Justice Lehrmann in the decision highlighting a cornerstone that has "been reaffirmed time and again by the Supreme Court, this Court, Texas courts of appeals, legal treatises, and even popular culture." That last reference to popular culture contained an interesting footnote citing none other than Walter Sobchak, a character in ['The Big Lebowski'].
lebowski movies coen-brothers prior-restraint law supreme-court walter-sobchak funny
on using JSON as a config file format
Ben Hughes on twitter: "JSON is fine for config files, if you don't want to comment your config file. Which is a way of saying, it isn't fine for config files."
ben-hughes funny json file-formats config-files configuration software coding
Understanding weak isolation is a serious problem
Peter Bailis complaining about the horrors of modern transactional databases and their unserializability, which noone seems to be paying attention to: 'As you're probably aware, there's an ongoing and often lively debate between transactional adherents and more recent "NoSQL" upstarts about related issues of usability, data corruption, and performance. But, in contrast, many of these transactional inherents and the research community as a whole have effectively ignored weak isolation — even in a single server setting and despite the fact that literally millions of businesses today depend on weak isolation and that many of these isolation levels have been around for almost three decades.' 'Despite the ubiquity of weak isolation, I haven't found a database architect, researcher, or user who's been able to offer an explanation of when, and, probably more importantly, why isolation models such as Read Committed are sufficient for correct execution. It's reasonably well known that these weak isolation models represent "ACID in practice," but I don't think we have any real understanding of how so many applications are seemingly (!?) okay running under them. (If you haven't seen these models before, they're a little weird. For example, Read Committed isolation generally prevents users from reading uncommitted or non-final writes but allows a number of bad things to happen, like lost updates during concurrent read-modify-write operations. Why is this apparently okay for many applications?)'
acid consistency databases peter-bailis transactional corruption serializability isolation reliability
"Left-Right: A Concurrency Control Technique with Wait-Free Population Oblivious Reads" [pdf]
'In this paper, we describe a generic concurrency control technique with Blocking write operations and Wait-Free Population Oblivious read operations, which we named the Left-Right technique. It is of particular interest for real-time applications with dedicated Reader threads, due to its wait-free property that gives strong latency guarantees and, in addition, there is no need for automatic Garbage Collection. The Left-Right pattern can be applied to any data structure, allowing concurrent access to it similarly to a Reader-Writer lock, but in a non-blocking manner for reads. We present several variations of the Left-Right technique, with different versioning mechanisms and state machines. In addition, we constructed an optimistic approach that can reduce synchronization for reads.' See also http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.html for java implementation code.
left-right concurrency multithreading wait-free blocking realtime gc latency reader-writer locking synchronization java
-
'bring your .bashrc, .vimrc, etc. with you when you ssh'. A really nice implementation of this idea (much nicer than my own version!)
hacks productivity ssh remote shell sh bash via:johnke home-directory unix
Troubleshooting Production JVMs with jcmd
remotely trigger GCs, finalization, heap dumps etc. Handy
UK's ICO spam regulator even more toothless now
We appealed this decision, but on June 2014 the Upper Tribunal agreed with the First-tier Tribunal, cancelling our monetary penalty notice against Niebel and McNeish, and largely rendering our power to issue fines for breaches of PECR involving spam texts redundant.
This is pretty terrible. The UK appears to have the weakest anti-spam regime in Europe due to the lack of powers given to ICO.-
A nice curl/wget replacement which supports multi-TCP-connection downloads of HTTP/FTP resources. packaged for most Linux variants and OSX via brew
axel curl wget via:johnke downloading tcp http ftp ubuntu debian unix linux
-
Linux users familiar with other filesystems or ZFS users from other platforms will often ask whether ZFS on Linux (ZoL) is "stable". The short answer is yes, depending on your definition of stable. The term stable itself is somewhat ambiguous.
Oh dear. that's not a good start. Good reference page, though Screen time: Steve Jobs was a low tech parent
"This is rule No. 1: There are no screens in the bedroom. Period. Ever."
screen-time kids children tv mobile technology life rules parenting
CausalImpact: A new open-source package for estimating causal effects in time series
How can we measure the number of additional clicks or sales that an AdWords campaign generated? How can we estimate the impact of a new feature on app downloads? How do we compare the effectiveness of publicity across countries? In principle, all of these questions can be answered through causal inference. In practice, estimating a causal effect accurately is hard, especially when a randomised experiment is not available. One approach we've been developing at Google is based on Bayesian structural time-series models. We use these models to construct a synthetic control — what would have happened to our outcome metric in the absence of the intervention. This approach makes it possible to estimate the causal effect that can be attributed to the intervention, as well as its evolution over time. We've been testing and applying structural time-series models for some time at Google. For example, we've used them to better understand the effectiveness of advertising campaigns and work out their return on investment. We've also applied the models to settings where a randomised experiment was available, to check how similar our effect estimates would have been without an experimental control. Today, we're excited to announce the release of CausalImpact, an open-source R package that makes causal analyses simple and fast. With its release, all of our advertisers and users will be able to use the same powerful methods for estimating causal effects that we've been using ourselves. Our main motivation behind creating the package has been to find a better way of measuring the impact of ad campaigns on outcomes. However, the CausalImpact package could be used for many other applications involving causal inference. Examples include problems found in economics, epidemiology, or the political and social sciences.
causal-inference r google time-series models bayes adwords advertising statistics estimation metrics
Top 10 Historic Sites in Ireland and Northern Ireland -- National Geographic
Shamefully, I haven't visited most of these!
history neolithic ireland northern-ireland national-geographic tourism places
Software patents are crumbling, thanks to the Supreme Court
Now a series of decisions from lower courts is starting to bring the ruling's practical consequences into focus. And the results have been ugly for fans of software patents. By my count there have been 11 court rulings on the patentability of software since the Supreme Court's decision — including six that were decided this month. Every single one of them has led to the patent being invalidated. This doesn't necessarily mean that all software patents are in danger — these are mostly patents that are particularly vulnerable to challenge under the new Alice precedent. But it does mean that the pendulum of patent law is now clearly swinging in an anti-patent direction. Every time a patent gets invalidated, it strengthens the bargaining position of every defendant facing a lawsuit from a patent troll.
patents law alice swpats software supreme-court patent-trolls
Riding with the Stars: Passenger Privacy in the NYC Taxicab Dataset
A practical demo of "differential privacy" -- allowing public data dumps to happen without leaking privacy, using Laplace noise addition
differential-privacy privacy leaks public-data open-data data nyc taxis laplace noise randomness
-
I'm ambivalent about Microsoft acquiring Mojang. Will they Embrace and Extend Minecraft as they've done with other categories? Let's hope not. On the other hand, some adult supervision and a Plugin API would be welcome. Mojang have the financial resources but lack the will and focus needed to publish and support a Plugin API. Perhaps Mojang themselves don't realise just how important their little game has become.
Open Invention Network Symposium on Open Source Software and Patents in Context
Dublin, 24th September 2014, hosted by Enterprise Ireland. Hosted by former Ubuntu counsel (via gcarr)
via:gcarr ubuntu law legal open-source floss oss oin inventions patents swpat software ireland ei events
Chris Baus: TCP_CORK: More than you ever wanted to know
Even with buffered streams the application must be able to instruct the OS to forward all pending data when the stream has been flushed for optimal performance. The application does not know where packet boundaries reside, hence buffer flushes might not align on packet boundaries. TCP_CORK can pack data more effectively, because it has direct access to the TCP/IP layer. [..] If you do use an application buffering and streaming mechanism (as does Apache), I highly recommend applying the TCP_NODELAY socket option which disables Nagle's algorithm. All calls to write() will then result in immediate transfer of data.
networking tcp via:nmaurer performance ip tcp_cork linux syscalls writev tcp_nodelay nagle packets
-
relatively-new Japanese place in the North Strand -- delivers, too. Comes recommended by JK. Must try it out soon!
takeaways delivery food restaurants japanese north-strand dublin
A gut microbe that stops food allergies
Actual scientific research showing that antibiotic use may be implicated in allergies: 'Nagler's team first confirmed that mice given antibiotics early in life were far more susceptible to peanut sensitization, a model of human peanut allergy. Then, they introduced a solution containing Clostridia, a common class of bacteria that's naturally found in the mammalian gut, into the rodents' mouths and stomachs. The animals' food allergen sensitization disappeared, the team reports online today in the Proceedings of the National Academy of Sciences. When the scientists instead introduced another common kind of healthy bacteria, called Bacteroides, into similarly allergy-prone mice, they didn't see the same effect. Studying the rodents more carefully, the researchers determined that Clostridia were having a surprising effect on the mouse gut: Acting through certain immune cells, the bacteria helped keep peanut proteins that can cause allergic reactions out of the bloodstream. "The bacteria are maintaining the integrity of the [intestinal] barrier," Nagler says.'
allergies health food peanuts science research clostridium bacteria gut intestines immune-system mice papers pnas
-
ah, memories. This is the bug that caused me to have to run a fleet-wide upgrade across the EC2 substrate. Thanks, boost::asio!
-
Today, Apple announced their "Most Personal Device Ever". They also announced Apple Pay (the only mentions of "security" and "privacy" in today's event), and are rolling out health tracking and home automation in iOS 8. Given their feckless track record [with cloud-service security], would you really trust Apple with (even more of) your digital life?
-
Excellent post from Dan Kaminsky on concrete actions that cloud service providers like Apple and Google need to start taking.
*It's time to ban Password1*: [...] Defenders are using simple rules like "doesn't have an uppercase letter" and "not enough punctuation" to block passwords while attackers are just straight up analyzing password dumps and figuring out the most likely passwords to attempt in any scenario. Attackers are just way ahead. That has to change. Defenders have password dumps too now. It's time we start outright blocking passwords common enough that they can be online brute forced, and it's time we admit we know what they are. [...] *People use communication technologies for sexy times. Deal with it*: Just like browsers have porn mode for the personal consumption of private imagery, cell phones have applications that are significantly less likely to lead to anyone else but your special friends seeing your special bits. I personally advise Wickr, an instant messaging firm that develops secure software for iPhone and Android. What's important about Wickr here isn't just the deep crypto they've implemented, though it's useful too. What's important in this context is that with this code there's just a lot fewer places to steal your data from. Photos and other content sent in Wickr don't get backed up to your desktop, don't get saved in any cloud, and by default get removed from your friend's phone after an amount of time you control. Wickr is of course not the only company supporting what's called "ephemeral messaging"; SnapChat also dramatically reduces the exposure of your private imagery. [...]
via Leonard.icloud apple privacy security via:lhl snapchat wickr dan-kaminsky cloud-services backup
Inside Apple's Live Event Stream Failure, And Why It Happened: It Wasn't A Capacity Issue
The bottom line with this event is that the encoding, translation, JavaScript code, the video player, the call to S3 single storage location and the millisecond refreshes all didn't work properly together and was the root cause of Apple's failed attempt to make the live stream work without any problems. So while it would be easy to say it was a CDN capacity issue, which was my initial thought considering how many events are taking place today and this week, it does not appear that a lack of capacity played any part in the event not working properly. Apple simply didn't provision and plan for the event properly.
BLDGBLOG: Procedural Brutalism
a few GIFs of procedurally generated architecture by a game developer named Cedric, built using Unity. Cedric describes himself as an "indie game dev focused on social AI, emergent narrative and procedural worlds." Imagine whole game worlds powered by real-time computation at the building level, constantly and parametrically fizzing with architectural forms, barely predictable new Woolworth Buildings and Barbicans sprouting on-demand from the ground whenever needed.
brutalism architecture games graphics design procedural generation gifs animation
Comcast Wi-Fi serving self-promotional ads via JavaScript injection | Ars Technica
Comcast is adding data into the broadband packet stream. In 2007, it was packets serving up disconnection commands. Today, Comcast is inserting JavaScript that is serving up advertisements, according to [Robb] Topolski, who reviewed Singel's data. "It's the duty of the service provider to pull packets without treating them or modifying them or injecting stuff or forging packets. None of that should be in the province of the service provider," he said. "Imagine every Web page with a Comcast bug in the lower righthand corner. It's the antithesis of what a service provider is supposed to do. We want Internet access, not another version of cable TV."
The company appears to be called Front Porch: http://arstechnica.com/tech-policy/2014/09/meet-the-tech-company-performing-ad-injections-for-big-cable/comcast ads injection security javascript http network-neutrality isps
"Perspectives On The CAP Theorem" [pdf]
"We cannot achieve [CAP theorem] consistency and availability in a partition-prone network."
papers cap distcomp cap-theorem consistency availability partitions network reliability
Aerospike's CA boast gets a thumbs-down from @aphyr
Specifically, @aerospikedb cannot offer cursor stability, repeatable read, snapshot isolation, or any flavor of serializability. @nasav @aerospikedb At *best* you can offer Read Committed, which is not, I assert, what most people would expect from an "ACID" database.
aphyr aerospike availability consistency acid transactions distcomp databases storage
How Twitter Uses Redis to Scale
'105TB RAM, 39MM QPS, 10,000+ instances.' Notes from a talk given by Yao Yu of Twitter's Cache team, where she's worked for 4 years. Lots of interesting insights into large-scale Redis caching usage -- as in, large enough to max out the cluster hosts' network bandwidth.
CLion – Brand New IDE for C and C++ Developers
JetBrains (makers of the excellent Intelli/J) have come out with a C/C++ refactoring IDE which looks utterly fantastic. If I wind up hacking on C/C++ again in future, I'll be using this one
c c++ refactoring ide intelli-j clion jetbrains editors coding
-
'turns a fresh cloud computer into a working mail server. You get contact synchronization, spam filtering, and so on. On your phone, you can use apps like K-9 Mail and CardDAV-Sync free beta to sync your email and contacts between your phone and your box.' (via Tony Finch)
Irish Water Data Protection Notice: A review…
Tried and came up wanting. Particularly notable for its illegal "Marketing" section, which attempts to evade opt-in-required anti-spam law with a "consent landgrab" on SMS and email
irish-water law dpc data-protection privacy spam opt-in si336 sms email ireland
SI336 - current Irish anti-spam law
"European Communities (Electronic Communications Networks and Services) (Privacy and Electronic Communications) Regulations 2011". Spam is covered under 13.1, "Unsolicited communications", on page 16 of this PDF
spam anti-spam law ireland eu ec sms email si336 privacy regulation
-
Lots of good advice for parents here
kids online social-media internet web facebook privacy security
'The very first release of Gmail simply used spamassassin on the backend'
Excellent. Confirming what I'd heard from a few other sources, too ;) This is a well-written history of the anti-spam war so far, from Mike Hearn, writing with the Google/Gmail point of view:
Brief note about my background, to establish credentials: I worked at Google for about 7.5 years. For about 4.5 of those I worked on the Gmail abuse team, which is very tightly linked with the spam team (they use the same software, share the same on-call rotations etc).
Reading this kind of stuff is awesome for me, since it's a nice picture of a fun problem to work on -- the Gmail team took the right ideas about how to fight spam, and scaled them up to the 10s-of-millions DAU mark. Nicely done. The second half is some interesting musings on end-to-end encrypted communications and how it would deal with spam. Worth a read...The FBI Finally Says How It 'Legally' Pinpointed Silk Road's Server
The answer, according to a new filing by the case's prosecution, is far more mundane: The FBI claims to have found the server's location without the NSA's help, simply by fiddling with the Silk Road's login page until it leaked its true location.
The Ramifications of Alice: A Conversation with Mark Lemley - IPWatchdog.com
I think you need to review what is actually happening at the USPTO in terms of rejections and how the Federal Circuit is applying Alice to find software patent claims patent ineligible. We are not crying wolf. It is really, factually, truthfully happening.
On the face of it, this sounds like great news ;)Why A Dead Alkaline Battery Bounces [video]
Nice bit of science
Visualizing Garbage Collection Algorithms
Great dataviz with animated GIFs
algorithms gc memory visualization garbage-collection dataviz refcounting mark-and-sweep
-
John Gruber's canonical description of Markdown's syntax does not specify the syntax unambiguously. In the absence of a spec, early implementers consulted the original Markdown.pl code to resolve these ambiguities. But Markdown.pl was quite buggy, and gave manifestly bad results in many cases, so it was not a satisfactory replacement for a spec. Because there is no unambiguous spec, implementations have diverged considerably. As a result, users are often surprised to find that a document that renders one way on one system (say, a GitHub wiki) renders differently on another (say, converting to docbook using Pandoc). To make matters worse, because nothing in Markdown counts as a "syntax error," the divergence often isn't discovered right away. There's no standard test suite for Markdown; the unofficial MDTest is the closest thing we have. The only way to resolve Markdown ambiguities and inconsistencies is Babelmark, which compares the output of 20+ implementations of Markdown against each other to see if a consensus emerges. We propose a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations against this specification. We believe this is necessary, even essential, for the future of Markdown.
Postcodes at last but random numbers don't address efficiency
Karlin Lillington assembles a fine collection of quotes from various sources panning the new Eircode system:
Critics say the opportunity has been missed to use Ireland's clean-slate status to produce a technologically innovative postcode system that would be at the cutting edge globally; similar to the competitive leap that was provided when the State switched to a digital phone network in the 1980s, well ahead of most of the world. Instead, say organisations such as the Freight Transport Association of Ireland (FTAI), the proposed seven-digit format of scrambled letters and numbers is almost useless for a business sector that should most benefit from a proper postcode system: transport and delivery companies, from international giants like FedEx and UPS down to local courier, delivery and service supplier firms. Because each postcode will reveal the exact address of a home or business, privacy advocates are concerned that online use of postcodes could link many types of internet activity, including potentially sensitive online searches, to a specific household or business.
eircode government fail ireland postcodes location ftai random
-
List of websites and whether or not they support 2FA. Also see the list of 2FA providers and the platforms they support.
-
Excellent post on all of the ins and outs of EC2 spot instance usage
Nik Cubrilovic - Notes on the Celebrity Data Theft
tl;dr: a lot of people are spending a lot of time stealing nudie pics from celebrities. See also http://www.zdziarski.com/blog/?p=3783 for more details on the probable approaches used. Grim.
apple privacy security celebrities pics hacking iphone ipad ios exploits brute-force passwords 2fa mfa find-my-iphone icloud backups
How To Remove a Stripped Screw Without an Extractor
one for future reference. Hate when this happens
Nix: The Purely Functional Package Manager
'a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments. ' Basically, this is a third-party open source reimplementation of Amazon's (excellent) internal packaging system, using symlinks to versioned package directories to ensure atomicity and the ability to roll back. This is definitely the *right* way to build packages -- I know what tool I'll be pushing for, next time this question comes up. See also nixos.org for a Linux distro built on Nix.
ops linux devops unix packaging distros nix nixos atomic upgrades rollback versioning
Facebook's drop-in replacement for std::vector
Fixes some low-hanging fruit, performance-wise. 'Simply replacing std::vector with folly::fbvector (after having included the folly/FBVector.h header file) will improve the performance of your C++ code using vectors with common coding patterns. The improvements are always non-negative, almost always measurable, frequently significant, sometimes dramatic, and occasionally spectacular.' (via Tony Finch)
c++ facebook performance algorithms vectors via:fanf optimization
Applying cardiac alarm management techniques to your on-call
An ops-focused take on a recent story about alarm fatigue, and how a Boston hospital dealt with it. When I was in Amazon, many of the teams in our division had a target to reduce false positive pages, with a definite monetary value attached to it, since many teams had "time off in lieu" payments for out-of-hours pages to the on-call staff. As a result, reducing false-positive pages was reasonably high priority and we dealt with this problem very proactively, with a well-developed sense of how to do so. It's interesting to see how the outside world is only just starting to look into its amelioration. (Another benefit of a TOIL policy ;)
ops monitoring sysadmin alerts alarms nagios alarm-fatigue false-positives pages
"Invertible Bloom Lookup Tables" [paper]
'We present a version of the Bloom filter data structure that supports not only the insertion, deletion, and lookup of key-value pairs, but also allows a complete listing of the pairs it contains with high probability, as long the number of key- value pairs is below a designed threshold. Our structure allows the number of key-value pairs to greatly exceed this threshold during normal operation. Exceeding the threshold simply temporarily prevents content listing and reduces the probability of a successful lookup. If entries are later deleted to return the structure below the threshold, everything again functions appropriately. We also show that simple variations of our structure are robust to certain standard errors, such as the deletion of a key without a corresponding insertion or the insertion of two distinct values for a key. The properties of our structure make it suitable for several applications, including database and networking applications that we highlight.'
iblt bloom-filters data-structures performance algorithms coding papers probabilistic
Some UX Dark Patterns now illegal in the EU
The EU's new consumer rights law bans certain dark patterns related to e-commerce across Europe. The "sneak into basket" pattern is now illegal. Full stop, end of story. You cannot create a situation where additional items and services are added by default. [...] Hidden costs are now illegal, whether that's an undeclared subscription, extra shipping charges, or extra items. [....] Forced continuity, when imposed on the user as a form of bait-and-switch, has been banned. Just the other day a web designer mentioned to me that he had only just discovered he had been charged for four years of annual membership dues in a "theme club", having bought what he thought was a one-off theme. Since he lives in Europe, he may be able to claim all of this money back. All he needs to do is prove that the website did not inform him that the purchase included a membership with recurring payments.
design europe law ecommerce ux dark-patterns scams ryanair selling online consumer consumer-rights bait-and-switch
-
The CDC (Centre for Disease Control) lists water fluoridation as one of the ten great public health achievements of the 20th Century. Today, Dublin City Council will vote on whether to remove fluoride from our water supply, and when they do, it will not be because the CDC or the WHO have changed their mind about fluoridation, or because new and compelling information makes it the only choice. It will be because people who believe in angel healing, homeopathy, and chemtrails, have somehow gained the ability to influence public policy.
dcc dublin law flouride science zenbuffy homeopathy woo health teeth
Revisiting How We Put Together Linux Systems
Building a running OS out of layered btrfs filesystems. This sounds awesome.
Instantiating a new system or OS container (which is exactly the same in this scheme) just consists of creating a new appropriately named root sub-volume. Completely naturally you can share one vendor OS copy in one specific version with a multitude of container instances. Everything is double-buffered (or actually, n-fold-buffered), because usr, runtime, framework, app sub-volumes can exist in multiple versions. Of course, by default the execution logic should always pick the newest release of each sub-volume, but it is up to the user keep multiple versions around, and possibly execute older versions, if he desires to do so. In fact, like on ChromeOS this could even be handled automatically: if a system fails to boot with a newer snapshot, the boot loader can automatically revert back to an older version of the OS.
(via Tony Finch)via:fanf linux docker btrfs filesystems unionfs copy-on-write os hacking unix
-
A nice Lua/C++ implementation of Aho-Corasick for fast string matching against multiple patterns (via JGC). This uses an interesting technique to get better performance by compacting the data structure into a single buffer, to avoid following pointers all over RAM and busting the cache.
optimization speed performance aho-corasick tries string-matching strings algorithms lua c++ via:jgc
On-Demand Jenkins Slaves With Amazon EC2
This is very likely where we'll be going for our acceptance tests in Swrve
testing jenkins ec2 spot-instances scalability auto-scaling ops build
Google's new end-to-end key distribution proposal
'For End-To-End, our current approach to key distribution, is to use a model similar to Certificate Transparency, and use the email messages themselves as a gossip protocol, which allow the users themselves to keep the centralized authorities honest. This approach allows users to not have to know about keys, but at the same time, be able to make sure that the servers involved aren't doing anything malicious behind the users' back.'
end-to-end encryption google security email crypto key-distribution
-
'The Irish Times podcast ends with both the NUJ's Seamus Dooley and Prof Kenny agreeing that somebody must regulate the internet so that it can be brought into line.'
regulation ireland law dangerous nuj bai journalism censorship
Apache Kafka 0.8 basic training
This is a pretty voluminous and authoritative presentation about getting started with Kafka; wish this was around when we started using it for 0.7. (We use our own homegrown realtime system nowadays, due to better partitioning, monitoring and operability.)
-
Wiki Loves Monuments is an international photo contest, organised by Wikimedia [...]. This year, the Wikimedia Ireland Community are running the competition for the very first time in Ireland. The contest is inspired by the successful 2010 pilot in the Netherlands which resulted in 12,500 freely licensed images uploaded to Wikimedia Commons. It has grown substantially since its inception; in 2013 369,589 photographs were submitted by 11,943 participants from over 50 countries. Cultural heritage is an important part of the knowledge that Wikipedia collects and disseminates. An image is worth a thousand words, in any language and local enthusiasts can (re)discover the cultural, historical, or scientific significance of their neighbourhood. The Irish contest, focussing on Ireland's national monuments, runs from August 23 - September 30. Follow our step-by-step guide to find out how you can take part.
wikipedia wikimedia images monuments history ireland contests creative-commons licensing
"CryptoPhone" claims to detect IMSI catchers in operation
To show what the CryptoPhone can do that less expensive competitors cannot, he points me to a map that he and his customers have created, indicating 17 different phony cell towers known as "interceptors," detected by the CryptoPhone 500 around the United States during the month of July alone. Interceptors look to a typical phone like an ordinary tower. Once the phone connects with the interceptor, a variety of "over-the-air" attacks become possible, from eavesdropping on calls and texts to pushing spyware to the device. "Interceptor use in the U.S. is much higher than people had anticipated," Goldsmith says. "One of our customers took a road trip from Florida to North Carolina and he found 8 different interceptors on that trip. We even found one at South Point Casino in Las Vegas."
imsi-catchers security cryptophone phones mobile 3g 4g eavesdropping surveillance
The poisoned NUL byte, 2014 edition
A successful exploit of Fedora glibc via a single NUL overflow (via Tony Finch)
via:fanf buffer-overflows security nul byte exploits google project-zero
'Addressing the rebalancing problem in bike-sharing systems' [paper]
Many of the bike-sharing systems introduced around the world in the past 15 years have the same problem: Riders tend to take some routes and not others. As a result, the bikes tend to collect in a few places, which is a drag for users and a costly problem for the operators, who "rebalance" the system using trucks that take bikes from full stations to empty ones. Now, scientists are coming up with special algorithms to improve this process. One of them, developed by scientists at the Vienna University of Technology and the Austrian Institute of Technology, is now being tested in Vienna's bike-sharing system; another, developed at Cornell University, is already in use in New York City.
Timely -- here's what Dublin Bikes looked like this morning: https://twitter.com/jmason/status/503828246086295552 (via Andrew Caines)cycling bike-sharing borisbikes dublinbikes rebalancing fleet availability optimization maths papers toread algorithms
'Join-Idle-Queue: A Novel Load Balancing Algorithm for Dynamically Scalable Web Services' [paper]
We proposed the JIQ algorithms for web server farms that are dynamically scalable. The JIQ algorithms significantly outperform the state-of-the-art SQ(d) algorithm in terms of response time at the servers, while incurring no communication overhead on the critical path. The overall complexity of JIQ is no greater than that of SQ(d). The extension of the JIQ algorithms proves to be useful at very high load. It will be interesting to acquire a better understanding of the algorithm with a varying reporting threshold. We would also like to understand better the relationship of the reporting frequency to response times, as well as an algorithm to further reduce the complexity of the JIQ-SQ(2) algorithm while maintaining its superior performance.
join-idle-queue algorithms scheduling load-balancing via:norman-maurer jiq microsoft load-balancers performance
3 Rules of thumb for Bloom Filters
I often need to do rough back-of-the-envelope reasoning about things, and I find that doing a bit of work to develop an intuition for how a new technique performs is usually worthwhile. So, here are three broad rules of thumb to remember when discussing Bloom filters down the pub: One byte per item in the input set gives about a 2% false positive rate. The optimal number of hash functions is about 0.7 times the number of bits per item. 3 - The number of hashes dominates performance.
bloom-filters algorithm probabilistic rules reasoning via:norman-maurer false-positives hashing coding
Logentries Announces Machine Learning Analytics for IT Ops Monitoring and Real-time Alerting
This sounds pretty neat:
With Logentries Anomaly Detection, users can: Set-up real-time alerting based on deviations from important patterns and log events. Easily customize Anomaly thresholds and compare different time periods. With Logentries Inactivity Alerting, users can: Monitor standard, incoming events such as an application heart beat. Receive real-time alerts based on log inactivity (i.e. receive alerts when something does not occur).
logging syslog logentries anomaly-detection ops machine-learning inactivity alarms alerting heartbeats
A beginner's guide to drills and bits - Boing Boing
This is actually quite educational
-
Some vague details of the antispam system in use at Twitter.
The main challenges in supporting this type of system are evaluating rules with low enough latency that they can run on the write path for Twitter's main features (i.e., Tweets, Retweets, favorites, follows and messages), supporting computationally intense machine learning based rules, and providing Twitter engineers with the ability to modify and create new rules instantaneously.
EcoJel jellyfish identification card
To identify the jellyfish found in Irish waters -- good, recognisable photos
jellyfish identification ecojel ireland sea swimming safety id-cards
DealExtreme are now selling a Google Cardboard kit
$10 with free shipping. You can't go wrong!
The Double Identity of an "Anti-Semitic" Commenter
Hasbara out of control. This is utterly nuts.
His intricate campaign, which he has admitted to Common Dreams, included posting comments by a screen name, "JewishProgressive," whose purpose was to draw attention to and denounce the anti-Semitic comments that he had written under many other screen names. The deception was many-layered. At one point he had one of his characters charge that the anti-Semitic comments and the criticism of the anti-Semitic comments must be written by "internet trolls who have been known to impersonate anti-Semites in order to then double-back and accuse others of supporting anti-Semitism"--exactly what he was doing.
hasbara israel trolls propaganda web racism comments anonymity commondreams
WWN'S Guide To Abortion In Ireland
"Why are you still reading this? Go to England!" funny because it's (horribly) true.
abortion ireland politics women rights wwn england ovaries rosaries religion
Java tip: optimizing memory consumption
Good tips on how to tell if object allocation rate is a bottleneck in your JVM-based code
yourkit memory java jvm allocation gc bottlenecks performance
-
The way that [problems with the PGP bootstrapping] are supposed to be resolved is with an authentication model called the Web of Trust where users sign keys of other users after verifying that they are who they say they are. In theory, if some due diligence is applied in signing other people's keys and a sufficient number of people participate you'll be able to follow a short chain of signatures from people you already know and trust to new untrusted keys you download from a key server. In practice this has never worked out very well as it burdens users with the task of manually finding people to sign their keys and even experts find the Web of Trust model difficult to reason about. This also reveals the social graph of certain communities which may place users at risk for their associations. Such signatures also reveal metadata about times and thus places for meetings for key signings. The Nyms Identity Directory is a replacement for all of this. Keyservers are replaced with an identity directory that gives users full control over publication of their key information and web of trust is replaced with a distributed network of trusted notaries which validate user keys with an email verification protocol.
web-of-trust directories nyms privacy crypto identity trust pgp gpg security via:ioerror keyservers notaries
-
Frogsort as an exam question (via qwghlm)
via:qwghlm frogsort sorting big-o algorithms funny comics smbc
Punished for Being Poor: Big Data in the Justice System
This is awful. Totally the wrong tool for the job -- a false positive rate which is miniscule for something like spam filtering, could translate to a really horrible outcome for a human life.
Currently, over 20 states use data-crunching risk-assessment programs for sentencing decisions, usually consisting of proprietary software whose exact methods are unknown, to determine which individuals are most likely to re-offend. The Senate and House are also considering similar tools for federal sentencing. These data programs look at a variety of factors, many of them relatively static, like criminal and employment history, age, gender, education, finances, family background, and residence. Indiana, for example, uses the LSI-R, the legality of which was upheld by the state's supreme court in 2010. Other states use a model called COMPAS, which uses many of the same variables as LSI-R and even includes high school grades. Others are currently considering the practice as a way to reduce the number of inmates and ensure public safety. (Many more states use or endorse similar assessments when sentencing sex offenders, and the programs have been used in parole hearings for years.) Even the American Law Institute has embraced the practice, adding it to the Model Penal Code, attesting to the tool's legitimacy.
(via stroan)via:stroan statistics false-positives big-data law law-enforcement penal-code risk sentencing
Microservices - Not a free lunch! - High Scalability
Some good reasons not to adopt microservices blindly. Testability and distributed-systems complexity are my biggest fears
Richard Clayton - Failing at Microservices
Solid warts-and-all confessional blogpost about a team failing to implement a microservices architecture. I'd put most of the blame on insufficient infrastructure to support them (at a code level), inter-personal team problems, and inexperience with large-scale complex multi-service production deployment and the work it was going to require
microservices devops collaboration architecture fail team deployment soa
Box Tech Blog » A Tale of Postmortems
How Box introduced COE-style dev/ops outage postmortems, and got them working. This PIE metric sounds really useful to head off the dreaded "it'll all have to come out missus" action item:
The picture was getting clearer, and we decided to look into individual postmortems and action items and see what was missing. As it was, action items were wasting away with no owners. Digging deeper, we noticed that many action items entailed massive refactorings or vague requirements like "make system X better" (i.e. tasks that realistically were unlikely to be addressed). At a higher level, postmortem discussions often devolved into theoretical debates without a clear outcome. We needed a way to lower and focus the postmortem bar and a better way to categorize our action items and our technical debt. Out of this need, PIE ("Probability of recurrence * Impact of recurrence * Ease of addressing") was born. By ranking each factor from 1 ("low") to 5 ("high"), PIE provided us with two critical improvements: 1. A way to police our postmortems discussions. I.e. a low probability, low impact, hard to implement solution was unlikely to get prioritized and was better suited to a discussion outside the context of the postmortem. Using this ranking helped deflect almost all theoretical discussions. 2. A straightforward way to prioritize our action items. What's better is that once we embraced PIE, we also applied it to existing tech debt work. This was critical because we could now prioritize postmortem action items alongside existing work. Postmortem action items became part of normal operations just like any other high-priority work.
postmortems action-items outages ops devops pie metrics ranking refactoring prioritisation tech-debt
NTP's days are numbered for consumer devices
An accurate clock is required to negotiate SSL/TLS, so clock sync is important for internet-of-things usage. but:
Unfortunately for us, the traditional and most widespread method for clock synchronisation (NTP) has been caught up in a DDoS issue which has recently caused some ISPs to start blocking all NTP communication. [....] Because the DDoS attacks are so widespread, and the lack of obvious commercial pressure to fix the issue, it's possible that the days of using NTP as a mechanism for setting clocks may well be numbered. Luckily for us there is a small but growing project that replaces it. tlsdate was started by Jacob Appelbaum of the Tor project in 2012, making use of the SSL handshake in order to extract time from a remote server, and its usage is on the rise. [....] Since we started encountering these problems, we've incorporated tlsdate into an over-the-air update, and have successfully started using this in situations where NTP is blocked.
tlsdate ntp clocks time sync iot via:gwire ddos isps internet protocols security
Cloudwash – Creating the Technical Prototype
This is a lovely demo of integrating modern IoT connectivity functionality (remote app control, etc.) with a washing machine using Bergcloud's hardware and backend, and a little logic-analyzer reverse engineering.
arduino diy washing-machines iot bergcloud hacking reversing logic-analyzers hardware
Systemd: Harbinger of the Linux apocalypse
While there are many defensible aspects of Systemd, other aspects boggle the mind. Not the least of these was that, as of a few months ago, trying to debug the kernel from the boot line would cause the system to crash. This was because of Systemd's voracious logging and the fact that Systemd responds to the "debug" flag on the kernel boot line -- a flag meant for the kernel, not anything else. That, straight up, is a bug. However, the Systemd developers didn't see it that way and actively fought with those experiencing the problem. Add the fact that one of the Systemd developers was banned by Linus Torvalds for poor attitude and bad design and another was responsible for causing significant issues with Linux audio support, but blamed the problem on everything else but his software, and you have a bad situation on your hands. There's no shortage of egos in the open source development world. There's no shortage of new ideas and veteran developers and administrators pooh-poohing something new simply because it's new. But there are also 45 years of history behind Unix and extremely good reasons it's still flourishing. Tools designed like Systemd do not fit the Linux mold, to their own detriment. Systemd's design has more in common with Windows than with Unix -- down to the binary logging.
The link re systemd consuming the "debug" kernel boot arg is a canonical example of inflexible coders refusing to fix their own bugs. (via Jason Dixon)systemd linux red-hat egos linus-torvalds unix init booting debugging logging design software via:obfuscurity
-
The mining operation resides on an old, repurposed factory floor, and contains 2500 machines hashing away at 230 Gh/s, each. (That's 230 billion calculations per second, per unit). [...] The operators told me that the power bill of this specific operation is in excess of ¥400,000 per month [..] about $60,000 USD.
currency china economics bitcoin power environment green mining datacenters
Moving Big Data into the Cloud with Tsunami UDP - AWS Big Data Blog
Pretty serious speedup. 81 MB/sec with Tsunami UDP, compared to 9 MB/sec with plain old scp. Probably kills internet performance for everyone else though!
tsunami-udp udp scp copying transfers internet long-distance performance speed
-
Ha, great name. We use this (in the form of Smartstack).
For what it is worth, we faced a similar challenge in earlier services (mostly due to existing C/C++ applications) and we created what was called a "sidecar". By sidecar, what I mean is a second process on each node/instance that did Cloud Service Fabric operations on behalf of the main process (the side-managed process). Unfortunately those sidecars all went off and created one-offs for their particular service. In this post, I'll describe a more general sidecar that doesn't force users to have these one-offs. Sidenote: For those not familiar with sidecars, think of the motorcycle sidecar below. Snoopy would be the main process with Woodstock being the sidecar process. The main work on the instance would be the motorcycle (say serving your users' REST requests). The operational control is the sidecar (say serving health checks and management plane requests of the operational platform).
netflix sidecars architecture patterns smartstack netflixoss microservices soa
Six things we know from the latest FinFisher documents | Privacy International
The publishing of materials from a support server belonging to surveillance-industry giant Gamma International has provided a trove of information for technologists, security researchers and activists. This has given the world a direct insight into a tight-knit industry, which demands secrecy for themselves and their clients, but ultimately assists in the violation human rights of ordinary people without care or reproach. Now for the first time, there is solid confirmation of Gamma's activities from inside the company's own files, despite their denials, on their clients and support provided to a range of governments.
finfisher gamma-international privacy surveillance iphone android rootkits wiretapping germany privacy-international spying bahrain turkmenistan arab-spring egypt phones mobile
BAI says Mooney Show was wrong to broadcast programme supporting same-sex marriage
This is a terrible decision. As Fintan O'Toole wrote afterwards: [The] 'BAI decision actually makes the point: a gay couple is a political "issue"; a straight couple is just a couple'
ireland law bai radio derek-mooney same-sex-marriage gay equal-rights
The Internet's Original Sin - The Atlantic
Ethan Zuckerberg: 'It's not too late to ditch the ad-based business model and build a better web.'
advertising business internet ads business-models the-atlantic ethan-zuckerberg via:anildash web privacy surveillance google
Comment #28 : Bug #255161 : Bugs : "cupsys" package : Ubuntu
file(1) bug causes the input Postscript file to be misidentified as an Erlang JAM file if it contains the string 'Tue' starting at byte 4.
via:hackernews file unix cups printing funny bugs fail ubuntu linux
Syria's 2012 internet disconnection wasn't on purpose
According to Edward Snowden, it was a side-effect of the NSA attempting to install an exploit in one of the core routers at a major Syrian ISP, and accidentally bricking the router
routers exploits hacking software tao nsa edward-snowden syria internet privacy
Edward Snowden: The Untold Story | Threat Level | WIRED
Snowden interviewed by James "The Puzzle Palace" Bamford, no less
james-bamford nsa edward-snowden wired interviews toread leaks whistleblowers us-politics
Profiling Hadoop jobs with Riemann
I've built a very simple distributed profiler for soft-real-time telemetry from hundreds to thousands of JVMs concurrently. It's nowhere near as comprehensive in its analysis as, say, Yourkit, but it can tell you, across a distributed system, which functions are taking the most time, and what their dominant callers are.
Potentially useful.
-
the world's largest permanent scale model of the Solar System. The Sun is represented by the Ericsson Globe in Stockholm, the largest hemispherical building in the world. The inner planets can also be found in Stockholm but the outer planets are situated northward in other cities along the Baltic Sea. The system was started by Nils Brenning and Gösta Gahm and is on the scale of 1:20 million.
(via JK)scale models solar-system astronomy sun sweden science cool via:jk
All Data Are Belong to AWS: Streaming upload via Fluentd
Fluentd looks like a decent foundation for tailing/streaming event processing in Ruby, supporting batched output to S3 and a bunch of other AWS services, Kafka, and RabbitMQ for output. Claims to have ok performance, despite its Rubbitude. However, its high-availability story is shite, so not to be used where availability is important
ruby rabbitmq kafka tail event-streaming cep event-processing s3 aws sqs fluentd
Twitter / mzmyslowski: Why Nigerian scam emails are so poorly written
Great explanation from MS Research's Corman Herley
corman-herley microsoft research spam nigerian-scam 419 scams conversion targeting mugus twitter
-
install inotify-tools, then: 'while true do inotifywait -r -e modify -e create -e close . ./run.sh done' #opscookie
How Stewart "Whole Earth Catalog" Brand helped killed off the metric system in the US
In May of 1981, party people gathered for one of the nerdiest soirees ever to grace lower Manhattan. Billed as the "Foot Ball," the event was an anti-metric shindig. Its revelers—including author Tom Wolfe and Whole Earth Catalog founder Stewart Brand—had joined to protest the encroachment of the metric system into modern American life. They threw shade on the meter and kilogram, and toasted the simple beauty of old classics like the yard and the pound.
Crazy. (via _stunned)via:_stunned us-politics tom-wolfe stewart-brand luddism metric imperial feet path-dependence
-
Facebook's Autoscale service, which scales up/down the fleet in order to optimize power consumption; see also Google's Pegasus (http://csl.stanford.edu/~christos/publications/2014.pegasus.isca.pdf)
scaling via:eoinbrazil facebook autoscaling power optimization
A tick bite can make you allergic to red meat
The bugs harbor a sugar that humans don't have, called alpha-gal. The sugar is also is found in red meat — beef, pork, venison, rabbit — and even some dairy products. It's usually fine when people encounter it through food that gets digested. But a tick bite triggers an immune system response, and in that high-alert state, the body perceives the sugar the tick transmitted to the victim's bloodstream and skin as a foreign substance, and makes antibodies to it. That sets the stage for an allergic reaction the next time the person eats red meat and encounters the sugar.
Via Shane Naughtonticks meat food allergies immune-system health via:inundata sugar alpha-gal red-meat
Real time analytics with Netty, Storm, Kafka
Arch of a fairly typical Kafka/Storm realtime ad-tracking setup, from eClick/mc2ads, via Trustin Lee
via:trustinlee kafka storm netty architecture ad-tracking ads realtime
AWS Speed Test: What are the Fastest EC2 and S3 Regions?
My god, this test is awful -- this is how NOT to test networked infrastructure. (1) testing from a single EC2 instance in each region; (2) uploading to a single test bucket for each test; (3) results don't include min/max or percentiles, just an averaged measurement for each test. FAIL
Hacker Redirects Traffic From 19 Internet Providers to Steal Bitcoins | Threat Level | WIRED
'The attacker specifically targeted a collection of bitcoin mining "pools"–bitcoin-producing cooperatives in which users contribute their computers' processing power and are rewarded with a cut of the resulting cryptocurrency the pool produces. The redirection technique tricked the pools' participants into continuing to devote their processors to bitcoin mining while allowing the hacker to keep the proceeds. At its peak, according to the researchers' measurements, the hacker's scam was pocketing a flow of bitcoins and other digital currencies including dogecoin and worldcoin worth close to $9,000 a day. "With this kind of hijacking, you can quite easily grab a large collection of clients," says Pat Litke, one of the Dell researchers. "It takes less than a minute, and you end up with a lot of mining traffic under your control."' 'In total, Stewart and Litke were able to measure $83,000 worth of cryptocurrency stolen in the BGP attack [...] but the total haul could be larger'
bitcoin mining fraud internet bgp routing security attacks hacking
UK piracy police arrest man suspected of running proxy server (Wired UK)
The site, Immunicity.org, offers a proxy server and a proxy autoconfiguration file (PAC) to tell browsers to access various blocked sites (PirateBay, KickassTorrents et al) via the proxy.
The Police Intellectual Property Crime Unit has arrested a 20-year-old man in Nottingham on suspicion of copyright infringement for running a proxy server providing access to other sites subject to legal blocking orders.
Is operating a proxy server illegal? Interesting. Seems unlikely that this will go to court though. (Via TJ McIntyre)immunicity via:tjmcintyre police uk piracy proxies http pac pipcu copyright
-
brilliant. a great threadless sub from Threadless user NickOG back in 2012
-
Excellent: 'a Twitter-fueled link aggregator that favors new projects/sites over news/articles' from Andy Baio.
Announcing UberPool, Carpooling with Uber
Ah, I was waiting for this; rest-of-world-style carpooling on demand, in an app. Great stuff
via:belong.io carpooling uber ride-sharing apps taxi travel uberpool
Painless, effective peer reviews
This sounds like a nice way to do effective peer-driven team reviews without herculean effort, which were one of the most effective reviewing techniques (along with upwards reviewing of management) I encountered at Amazon. (Yes, the Amazon approach was very time-consuming and universally loathed.) The potential downside I can see is that it doesn't give the reviewer enough time to revise any review comments they have second thoughts about, whereas written reviews do, but that would be an easy fix at the end of the process. Also, it's worth noting that in most cases, a good review requires a bit of time to marshal thoughts and come up with a coherent review of a peer, so this doesn't completely avoid the impact on effort. Still, a definite improvement I would say.
hr management reviews performance peer-driven-review 360-reviews staff peers work teams amazon
The problem with OKCupid is the problem with the social web
This is why it really stings whenever somebody turns around and says, "well actually, the terms you've signed give us permission to do whatever we want. Not just the thing you were afraid of, but a huge range of things you never thought of." You can't on one hand tell us to pay no attention when you change these things on us, and with the other insist that this is what we've really wanted to do all along. I mean, fuck me over, but don't tell me that I really wanted you to fuck me over all along. Because ultimately, the reason you needed me to agree in the first place isn't just because I'm using your software, but because you're using my stuff. And the reason I'm letting you use my stuff, and spending all this time working on it, is so that you can show it to people. I'm not just a user of your service, somebody who reads the things that you show it to me: I'm one of the reasons you have anything that you can show to anyone at all.
users web facebook okcupid terms-of-service jason-kottke privacy a-b-testing experiments ethics
-
A Java-oriented practical intro to the MinHash duplicate-detection shingling algo
shingling algorithms minhash hashing duplicates duplicate-detection fuzzy-matching java
-
The two charts indicate that current EU copyright is very unbalanced. When one side is completely satisfied with the status quo and the other is very unhappy then this is not a balanced situation. Given that a good compromise should leave everybody equally unhappy, the results of the consultation also show the direction for copyright reform efforts of the new EU Commission: re-balancing copyright requires at least some reform as demanded by end users and institutional users, most importantly a more harmonized and flexible system of exceptions and limitations.
'TCP And The Lower Bound of Web Performance' [pdf, slides]
John Rauser, Velocity, June 2010. Good data on real-world web perf based on the limitations which TCP and the speed of light impose
-
This is a yet another Java collections library of primitive specializations. Java 6+. Apache 2.0 license. Currently only hash sets and hash maps are implemented.
openhft performance java jvm collections asl hashsets hashmaps data-structures
China detains 1,530 in telecom spam crackdown
via Christopher Soghoian: 'IMSI catchers/fake base stations are out of control in China. The gov shut down 24 IMSI catcher factories, 1500+ people were arrested.'
Does This Soldier's Instagram Account Prove Russia Is Covertly Operating In Ukraine?
"sitting around, working on a buk, listening to music, basically a good sunday"
ukraine buzzfeed politics sam missiles mh-17 war-crimes russia facebook instagram social-media whoops
UK private copying exception plans face possible legal action
Under the proposed private copying exception, individuals in the UK would be given a new right to make a copy of copyrighted material they have lawfully and permanently acquired for their private use, provided it was not for commercial ends. Making a private copy of the material in these circumstances would not be an act of copyright infringement, although making a private copy of a computer program would still be prohibited under the plans. There is no mechanism envisaged in the draft legislation for rights holders to be specifically compensated for the act of private copying. This prompted the Joint Committee on Statutory Instruments (JCSI), tasked with scrutinising the proposals, to warn parliamentarians that the rules may be deemed to be in breach of EU copyright laws as a result of the lack of 'fair compensation' mechanism. [...] "We are disappointed that the private copying exception will be introduced without providing fair compensation for British songwriters, performers and other rights holders within the creative sector. A mechanism for fair compensation is a requirement of European law. In response we are considering our legal options," [UK Music] said.
uk law copyright music copying private-copying personal infringement piracy transcoding backup
Moominvalley Map Print | Magic Pony
Lovely print! Shipping would be a bit crazy, though. There has to be an english-language print of one of Tove Jansson's maps on sale somewhere in Europe...
prints moomins moominvalley maps hattifatteners magic-pony tove-jannson art
-
Ladyada's intro to electronics and microcontrollers using Arduino. Some day I'll get around to refreshing my memory, it's been years since I fiddled with a resistor ;)
electronics arduino hardware gadgets learning tutorial microcontrollers embedded-systems ladyada
How to take over the computer of any JVM developer
To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.
jars dependencies java build clojure security mitm http proxies backdoors scala maven gradle
Spain pushes for 'Google tax' to restrict linking
The government wants to put a tax on linking on the internet. They say that if you want to link to some newspaper's content, you have to pay a tax. The primary targets of this law are Google News and other aggregators. It would be absurd enough just like that, but the law goes further: they declared it an "inalienable right" so even if I have a blog or a new small digital media publication and I want to let people freely link to my content, I can't opt-out--they are charging the levy, and giving it to the big press media. It was just the last and only way that the old traditional media companies can get some money from the government, and they strongly lobbied for it. The bill has passed in the Congress where the party in the government has majority (PP, Partido Popular) and it's headed to the Senate, where they have a majority also.
spain stupidity law via:boingboing linking links web news google google-news newspapers old-media taxes
Keyes New Starter Kit for Arduino Fans
$53 for a reasonable-looking Arduino starter kit, from DealExtreme. cheap cheap! In the inimitable DX style:
Keyes new beginner starter kit, pay more attention to beginners learning. Users can get rid of the difficult technological learning, from module used to quick start production.
learning arduino hardware hacking robotics toys dealextreme tobuy
Check If A Hotel's WiFi Sucks Before It's Too Late
http://www.hotelwifitest.com/ and http://speedspot.org/ .
-
a nice summarisation of the state of pipe/stream-oriented collection operations in various languages, from Martin Fowler
martin-fowler patterns coding ruby clojure streams pipelines pipes unix lambda fp java languages
REST Commander: Scalable Web Server Management and Monitoring
We dynamically monitor and manage a large and rapidly growing number of web servers deployed on our infrastructure and systems. However, existing tools present major challenges when making REST/SOAP calls with server-specific requests to a large number of web servers, and then performing aggregated analysis on the responses. We therefore developed REST Commander, a parallel asynchronous HTTP client as a service to monitor and manage web servers. REST Commander on a single server can send requests to thousands of servers with response aggregation in a matter of seconds. And yes, it is open-sourced at http://www.restcommander.com. Feature highlights: Click-to-run with zero installation; Generic HTTP request template supporting variable-based replacement for sending server-specific requests; Ability to send the same request to different servers, different requests to different servers, and different requests to the same server; Maximum concurrency control (throttling) to accommodate server capacity; Commander itself is also "as a service": with its powerful REST API, you can define ad-hoc target servers, an HTTP request template, variable replacement, and a regular expression all in a single call. In addition, intuitive step-by-step wizards help you achieve the same functionality through a GUI.
rest http clients load-testing ebay soap async testing monitoring
South Downs litter picker has truck named after him - West Sussex County Times
This is amazing. In http://www.newyorker.com/magazine/2014/06/30/stepping-out-3 , David Sedaris had written: 'in recognition of all the rubbish I've collected since getting my Fitbit, my local council is naming a garbage truck after me'; naturally, I assumed he was joking, but it looks like he wasn't:
Horsham District Council has paid thanks to a volunteer who devotes a great deal of time and energy to walking many miles clearing litter from near where he lives as well as surrounding areas. David Sedaris litter picks in areas including Parham, Coldwaltham, Storrington and beyond. In recognition for all his fantastic work and dedication and as a token of Horsham District Council's appreciation, the council has named one of their waste vehicles after him. The vehicle, bedecked with its bespoke 'Pig Pen Sedaris' sign was officially unveiled by the Lord-Lieutenant of West Sussex Mrs Susan Pyper at an outdoor ceremony on July 23.
Best of all, the article utterly fails to mention who he is. Amazing. (via John Braine)via:john-braine funny david-sedaris litter uk horsham rubbish garbage cleaning volunteering walking
-
Heapster provides an agent library to do heap profiling for JVM processes with output compatible with Google perftools. The goal of Heapster is to be able to do meaningful (sampled) heap profiling in a production setting.
Used by Twitter in production, apparently. The Network is Reliable - ACM Queue
Peter Bailis and Kyle Kingsbury accumulate a comprehensive, informal survey of real-world network failures observed in production. I remember that April 2011 EBS outage...
ec2 aws networking outages partitions jepsen pbailis aphyr acm-queue acm survey ops
This tree produces 40 different types of fruit
An art professor from Syracuse University in the US, Van Aken grew up on a family farm before pursuing a career as an artist, and has combined his knowledge of the two to develop his incredible Tree of 40 Fruit. In 2008, Van Aken learned that an orchard at the New York State Agricultural Experiment Station was about to be shut down due to a lack of funding. This single orchard grew a great number of heirloom, antique, and native varieties of stone fruit, and some of these were 150 to 200 years old. To lose this orchard would render many of these rare and old varieties of fruit extinct, so to preserve them, Van Aken bought the orchard, and spent the following years figuring out how to graft parts of the trees onto a single fruit tree. [...] Aken's Tree of 40 Fruit looks like a normal tree for most of the year, but in spring it reveals a stunning patchwork of pink, white, red and purple blossoms, which turn into an array of plums, peaches, apricots, nectarines, cherries and almonds during the summer months, all of which are rare and unique varieties.
fruit art amazing food agriculture grafting orchards sam-van-aken farming
-
we believe MDD is equal parts engineering technique and cultural process. It separates the notion of monitoring from its traditional position of exclusivity as an operations thing and places it more appropriately next to its peers as an engineering process. Provided access to real-time production metrics relevant to them individually, both software engineers and operations engineers can validate hypotheses, assess problems, implement solutions, and improve future designs.
Broken down into the following principles: 'Instrumentation-as-Code', 'Single Source of Truth', 'Developers Curate Visualizations and Alerts', 'Alert on What You See', 'Show me the Graph', 'Don't Measure Everything (YAGNI)'. We do all of these at Swrve, naturally (a technique I happily stole from Amazon).metrics coding graphite mdd instrumentation yagni alerting monitoring graphs
Auto Scale DynamoDB With Dynamic DynamoDB
Nicely-packaged auto-scaler for DynamoDB
dynamodb autoscaling scalability provisioning aws ec2 cloudformation
Google's mighty mess-up on 'right to be forgotten' - Independent.ie
In this context, the search giant says that it has "a team of people reviewing each application individually". Really? Did this team of people decide that redacting links to an article reporting a criminal conviction was consistent with an individual's right to privacy and 'right to be forgotten'? Either Google is deliberately letting egregious errors through to try and bait journalists and freedom of expression activists into protesting or its system at vetting 'right to be forgotten' applications is awfully flawed.
google right-to-be-forgotten privacy law ireland adrian-weckler journalism freedom-of-expression censorship redaction
"Ark: A Real-World Consensus Implementation" [paper]
"an implementation of a consensus algorithm similar to Paxos and Raft, designed as an improvement over the existing consensus algorithm used by MongoDB and TokuMX." It'll be interesting to see how this gets on in review from the distributed-systems community. The phrase "similar to Paxos and Raft" is both worrying and promising ;)
paxos raft consensus algorithms distsys distributed leader-election mongodb tokumx
A Japanese Artist Launches Plants Into Space
This is amazing.
though the vessel was found on the ground, the flowers were not.
japan art bonsai flowers space nevada black-rock-desert exobiotanica
'Identifying Back Doors, Attack Points and Surveillance Mechanisms in iOS Devices'
lots of scary stuff in this presentation from this year's Hackers On Planet Earth conf. I'm mainly interested to find out that Jonathan "D-Spam" Zdziarski was also a jailbreak dev-team member until around iOS 4 ;)
d-spam jonathan-zdziarski security apple ios iphone surveillance bugging
-
a Chrome extension to aid working with REST APIs. Formats XML and JSON responses, supports file uploads, key/value editors, autocomplete, open source under ASL2
open-source chrome extensions browser postman rest hateoas api xml json web-services via:eonnen
-
A Go implementation of Greenwald-Khanna streaming quantiles: http://infolab.stanford.edu/~datar/courses/cs361a/papers/quantiles.pdf - 'a new online algorithm for computing approximate quantile summaries of very large data sequences with a worst-case space requirement of O(1/e log eN))'
quantiles go algorithms greenwald-khanna percentiles streaming cep space-efficient
-
Some great tips on managing a busy calendar, from Etsy's managers. Block out time; refuse double-booked meetings by default; rely on apps; office hours. Thankfully I have a pretty slim calendar these days, but bookmarking for future use...
calendar etsy via:kellan google google-calendar office-hours life-hacks hacks tips managing managers scheduling
Nanex: "The stock market is rigged" [by HFTs]
All this evidence points to one inescapable conclusion: the order cancellations and trade executions just before, and during the trader's order were not a coincidence. This is premeditated, programmed theft, plain and simple. Michael Lewis probably said it best when he told 60 Minutes that the stock market is rigged.
Nanex have had enough, basically. Mad stuff.hft stocks finance market trading nanex 60-minutes michael-lewis scams sec regulation low-latency exploits hacks
Boundary's new server monitoring free offering
'High resolution, 1 second intervals for all metrics; Fluid analytics, drag any graph to any point in time; Smart alarms to cut down on false positives; Embedded graphs and customizable dashboards; Up to 10 servers for free' Pre-registration is open now. Could be interesting, although the limit of 10 machines is pretty small for any production usage
boundary monitoring network ops metrics alarms tcp ip netstat
-
A really excellent-looking workflow/orchestration engine for Hadoop, Pig, Hive, Redshift and other ETL jobs, featuring inter-job dependencies, cron-like scheduling, and failure handling. Open source, from Spotify
workflow orchestration scheduling cron spotify open-source luigi redshift pig hive hadoop emr jobs make dependencies
Obama administration says the world's servers are ours | Ars Technica
In its briefs filed last week, the US government said that content stored online doesn't enjoy the same type of Fourth Amendment protections as data stored in the physical world. The government cited (PDF) the Stored Communications Act (SCA), a President Ronald Reagan-era regulation.
Michael McDowell has filed a declaration in support of MS' position (attached to that article a couple of paras down) suggesting that the MLAT between the US and Ireland is the correct avenue.privacy eu us-politics microsoft michael-mcdowell law surveillance servers sca internet
-
'This tool can be described as a Tiny Dirty Linux Only C command that looks for coreutils basic commands (cp, mv, dd, tar, gzip/gunzip, cat, ...) currently running on your system and displays the percentage of copied data. It can now also display an estimated throughput (using -w flag).'
"In Search of an Understandable Consensus Algorithm"
Diego Ongaro and John Ousterhout, USENIX ATC 2014 -- won best paper for this paper on the Raft algorithm. (via Eoin Brazil)
raft consensus algorithms distcomp john-ousterhout via:eoinbrazil usenix atc papers paxos
-
Great map-comparison tool from Jef Poskanzer
jef-poskanzer mapping maps comparison visualization geo world cities
Divinity: Original Sin review | PC Gamer
I've become accustomed to RPGs that lock away combat and magic within their own part of the game. I'm used to the idea that a fireball won't work unless it's aimed at an enemy, or that every environmental hazard will be placed such that I'm guaranteed to be able to get past it. I'm used to the idea that some characters can be killed and some can't, that some obstacles are destructible and others are 'just furniture'. Divinity shrugs off those assumptions. Combat might be turn-based when you're fighting an enemy, but there's nothing stopping you from waving your sword around in the middle of town. Fling a fireball at some innocent barrels and you'll start a fresh fire of your own, and this time the locals won't be applauding when you rush to put it out.
wow, this sounds great. (via Paul Moloney)
-
a client side IPC library that is battle-tested in cloud. It provides the following features: Load balancing; Fault tolerance; Multiple protocol (HTTP, TCP, UDP) support in an asynchronous and reactive model; Caching and batching.
I like the integration of Eureka and Hystrix in particular, although I would really like to read more about Eureka's approach to availability during network partitions and CAP. https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/-5nElGl1OQ0J has some interesting discussion on the topic. It actually sounds like the Eureka approach is more correct than using ZK: 'Eureka is available. ZooKeeper, while tolerant against single node failures, doesn't react well to long partitioning events. For us, it's vastly more important that we maintain an available registry than a necessary consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that's OK with us.' See also http://ispyker.blogspot.ie/2013/12/zookeeper-as-cloud-native-service.html which corroborates this:I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances. This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones. What I saw was that the two other instances noticed that the first server "going away", but they continued to function as they still saw a majority (66%). More interestingly the first instance noticed the other two servers "going away" dropping the ensemble availability to 33%. This caused the first server to stop serving requests to clients (not only writes, but also reads). [...] To me this seems like a concern, as network partitions should be considered an event that should be survived. In this case (with this specific configuration of zookeeper) no new clients in that availability zone would be able to register themselves with consumers within the same availability zone. Adding more zookeeper instances to the ensemble wouldn't help considering a balanced deployment as in this case the availability would always be majority (66%) and non-majority (33%).
netflix ribbon availability libraries java hystrix eureka aws ec2 load-balancing networking http tcp architecture clients ipc
The Myth of Schema-less [NoSQL]
We don't seem to gain much in terms of database flexibility. Is our application more flexible? I don't think so. Even without our schema explicitly defined in our database, it's there... somewhere. You simply have to search through hundreds of thousands of lines to find all the little bits of it. It has the potential to be in several places, making it harder to properly identify. The reality of these codebases is that they are error prone and rarely lack the necessary documentation. This problem is magnified when there are multiple codebases talking to the same database. This is not an uncommon practice for reporting or analytical purposes. Finally, all this "flexibility" rears its head in the same way that PHP and Javascript's "neat" weak typing stabs you right in the face. There are some somethings you can be cavalier about, and some things you should be strict about. Your data model is one you absolutely need to be strict on. If a field should store an int, it should store nothing else. Not a string, not a picture of a horse, but an integer. It's nice to know that I have my database doing type checking for me and I can expect a field to be the same type across all records. All this leads us to an undeniable fact: There is always a schema. Wearing "I don't do schema" as a badge of honor is a complete joke and encourages a terrible development practice.
-
from yesterday's AWS Summit in NYC:
Cheat sheet of EBS-optimized instances. http://t.co/vmTlhUtpWk Optimize your queue depth to achieve lower latency & highest IOPS. http://t.co/EO48oa0D6X When configuring your RAID, use a stripe size of 128KB or 256KB. http://t.co/N0ldtFJ4t6 Use larger block size to speed up the pre-warming process. http://t.co/8UoIeWE2px
173 million 2013 NYC taxi rides shared on BigQuery : bigquery
Interesting! (a) there's a subreddit for Google BigQuery, with links to interesting data sets, like this one; (b) the entire 173-million-row dataset for NYC taxi rides in 2013 is available for querying; and (c) the tip percentage histogram is cool.
datasets bigquery sql google nyc new-york taxis data big-data histograms tipping
"Pitfalls of Object Oriented Programming", SCEE R&D
Good presentation discussing "data-oriented programming" -- the concept of optimizing memory access speed by laying out large data in a columnar format in RAM, rather than naively in the default layout that OOP design suggests
columnar ram memory optimization coding c++ oop data-oriented-programming data cache performance
Google's Influential Papers for 2013
Googlers across the company actively engage with the scientific community by publishing technical papers, contributing open-source packages, working on standards, introducing new APIs and tools, giving talks and presentations, participating in ongoing technical debates, and much more. Our publications offer technical and algorithmic advances, feature aspects we learn as we develop novel products and services, and shed light on some of the technical challenges we face at Google. Below are some of the especially influential papers co-authored by Googlers in 2013.
google papers toread reading 2013 scalability machine-learning algorithms
-
'Leak of the secret German Internet Censorship URL blacklist BPjM-Modul'. Turns out there's a blocklist of adult-only or prohibited domains issued by a German government department, The Federal Department for Media Harmful to Young Persons (German: "Bundesprüfstelle für jugendgefährdende Medien" or BPjM), issued in the form of a list of hashes of those domains. These were extracted from an AVM router, then the hashes were brute forced using several other plaintext URL blocklists and domain lists. Needless to say, there's an assortment of silly false positives, such as the listing of the website for the 1997 3D Realms game "Shadow Warrior": http://en.wikipedia.org/wiki/Shadow_Warrior
hashes reversing reverse-engineering germany german bpjm filtering blocklists blacklists avm domains censorship fps
Brave Men Take Paternity Leave - Gretchen Gavett - Harvard Business Review
The use of paternity leave has a "snowball effect":
In the end, Dahl says, "coworkers and brothers who were linked to a father who had his child immediately after the [Norwegian paid paternity leave] reform — versus immediately before the reform — were 3.5% and 4.7% more likely, respectively, to take parental leave." But when a coworker actually takes parental leave, "the next coworker to have a child at his workplace is 11% more likely to take paternity leave." Slightly more pronounced, the next brother to have a child is 15% more likely to take time off. And while any male coworker taking leave can reduce stigma, the effect of a manager doing so is more profound. Specifically, "the estimated peer effect is over two and a half times larger if the peer father is predicted to be a manager in the firm as opposed to a regular coworker."
-
by Jeffrey Dean and Luiz Andre Barroso, Google. A selection of Google's architectural mechanisms used to defeat 99th-percentile latency spikes: hedged requests, tied requests, micro-partitioning, selective replication, latency-induced probation, canary requests.
google architecture distcomp soa http partitioning replication latency 99th-percentile canary-requests hedged-requests
Breaking Spotify DRM with PANDA
Reverse engineering a DRM implementation, by instrumenting a VM and performing entropy/compressability analysis on function call inputs and outputs. Impressive
reversing spotify drm panda vm compression entropy compressability qemu via:hn
-
Book a domestic cleaner online in 60 seconds; "like Hailo for cleaners" apparently. Live in Dublin, London, Manchester, Birmingham and Leeds. Use code HASSLEDUBLIN for 15% off
Layered Glass Table Concept Creates a Cross-Section of the Ocean
beautiful stuff -- and a snip at only UKP 5,800 ex VAT. it'd make a good DIY project though ;)
art tables glass layering 3d cross-sections water ocean sea mapping cartography layers this-is-colossal design furniture
Two traps in iostat: %util and svctm
Marc Brooker:
As a measure of general IO busyness %util is fairly handy, but as an indication of how much the system is doing compared to what it can do, it's terrible. Iostat's svctm has even fewer redeeming strengths. It's just extremely misleading for most modern storage systems and workloads. Both of these fields are likely to mislead more than inform on modern SSD-based storage systems, and their use should be treated with extreme care.
ioutil iostat svctm ops ssd disks hardware metrics stats linux
New AWS Web Services region: eu-central-1 (soon)
Iiiinteresting. Sounds like new anti-NSA-snooping privacy laws will be driving a lot of new mini-regions in AWS. Hope Amazon have their new-region-standup process a little more streamlined by now than when I was there ;)
How A Spam Newsletter Caused a Bank Run in Bulgaria
According to the Bulgarian National Security Agency (see here, for a reporting in English), an investment company that "built a network of associated companies for marketing services" that was used to diffuse panic by means of an alert, uncomfortably titled "Information Bulletin of on the Risk of Deposits in Bulgarian Banks". The "bulletin" claimed – Bloomberg reports – KTB was undergoing a liquidity shortage. The message apparently also said that the government deposit guarantee fund was under-capitalised to meet possible repayments, that banks could go bankrupt and that the peg of the currency with the euro could be broken. Allegedly, the alert was diffused by text, email and even Facebook messages, thus ensuring a very widespread outreach. In a country that in 1997 underwent a very serious banking crisis featuring all these characteristics – whose memory is still fresh – this was enough to spur panic.
spam banking bulgaria banks euro panic facebook social-media
New Russian Law To Forbid Storing Russians' Data Outside the Country - Slashdot
On Friday Russia's parliament passed a law "which bans online businesses from storing personal data of Russian citizens on servers located abroad[.] ... According to ITAR-TASS, the changes to existing legislation will come into effect in September 2016, and apply to email services, social networks and search engines, including the likes of Facebook and Google. Domain names or net addresses not complying with regulations will be put on a blacklist maintained by Roskomnadzor (the Federal Supervision Agency for Information Technologies and Communications), the organisation which already has the powers to take down websites suspected of copyright infringement without a court order. In the case of non-compliance, Roskomnadzor will be able to impose 'sanctions,' and even instruct local Internet Service Providers (ISPs) to cut off access to the offending resource."
Irish parliament pressing ahead with increased access to retained telecoms data
While much of the new bill is concerned with the dissolution of the Competition Authority and the National Consumer Agency and the formation of a new merged Competition and Consumer Protection Commission (CCPC) the new bill also proposed to extend the powers of the new CCPC to help it investigate serious anticompetitive behaviour. Strikingly the new bill proposes to give members of the CCPC the power to access data retained under the Communications (Retention of Data) Act 2011. As readers will recall this act implements Directive 2006/24/EC which obliges telecommunications companies to archive traffic and location data for a period of up to two years to facilitate the investigation of serious crime. Ireland chose to implement the maximum two year retention period and provided access to An Garda Siochana, The Defence Forces and the Revenue Commissioners. The current reform of Irish competition law now proposes to extend data access powers to the members of the CCPC for the purposes of investigating cartel offences.
data-retention privacy surveillance competition ccpc ireland law dri
NSA: Linux Journal is an "extremist forum" and its readers get flagged for extra surveillance
DasErste.de has published the relevant XKEYSCORE source code, and if you look closely at the rule definitions, you will see linuxjournal.com/content/linux* listed alongside Tails and Tor. According to an article on DasErste.de, the NSA considers Linux Journal an "extremist forum". This means that merely looking for any Linux content on Linux Journal, not just content about anonymizing software or encryption, is considered suspicious and means your Internet traffic may be stored indefinitely.
This is, sadly, entirely predictable -- that's what happens when you optimize the system for over-sampling, with poor oversight.false-positives linuxjournal linux terrorism tor tails nsa surveillance snooping xkeyscore selectors oversight
-
a C++ library adding some modern language features like Option, Try, Stopwatch, and other Guava-ish things (via @cscotta)
Tor exit node operator prosecuted in Austria
'The operator of an exit node is guilty of complicity, because he enabled others to transmit content of an illegal nature through the service.' Via Tony Finch.
austria tor security law liability internet tunnelling eu via:fanf
IRS says free software projects can't be nonprofits - Boing Boing
In a disturbing precedent, the Yorba Foundation, which makes apps for [GNOME], has had its nonprofit status application rejected by the IRS because some of [its] projects may benefit for-profit entities.
law us gnome yorba-foundation linux gpl free-software oss nonprofits 501c3 tax
How to perform a load/latency test, correcting for coordinated-omission error
p-code from Gil Tene
gil-tene coordinated-omission measurement jmh latency testing errors code
Questioning the Lambda Architecture
Jay Kreps (Kafka, Samza) with a thought-provoking post on the batch/stream-processing dichotomy
jay-kreps toread architecture data stream-processing batch hadoop storm lambda-architecture
-
Urban Airship with a new open-source Graphite front-end UI; similar enough to Grafana at a glance, no releases yet, ASL2-licensed
This is the actual hack that saved the astronauts of the Apollo XIII
Duct tape ftw
New Low Cost EC2 Instances with Burstable Performance
Oh, very neat. New micro, small, and medium-class instances with burstable CPU scaling:
The T2 instances are built around a processing allocation model that provides you a generous, assured baseline amount of processing power coupled with the ability to automatically and transparently scale up to a full core when you need more compute power. Your ability to burst is based on the concept of "CPU Credits" that you accumulate during quiet periods and spend when things get busy. You can provision an instance of modest size and cost and still have more than adequate compute power in reserve to handle peak demands for compute power.
Facebook Doesn't Understand The Fuss About Its Emotion Manipulation Study
This is quite unethical, and I'm amazed it was published at all. Kashmir Hill at Forbes nails it:
While many users may already expect and be willing to have their behavior studied — and while that may be warranted with "research" being one of the 9,045 words in the data use policy — they don't expect that Facebook will actively manipulate their environment in order to see how they react. That's a new level of experimentation, turning Facebook from a fishbowl into a petri dish, and it's why people are flipping out about this.
Shocking stuff. We need a new social publishing platform, built on ethical, open systems.ethics facebook privacy academia depression feelings emotion social-publishing social experimentation papers
Building a Smarter Application Stack - DevOps Ireland
This sounds like a very interesting Dublin meetup -- Engine Yard on thursday night:
This month, we'll have Tomas Doran from Yelp talking about Docker, service discovery, and deployments. 'There are many advantages to a container based, microservices architecture - however, as always, there is no silver bullet. Any serious deployment will involve multiple host machines, and will have a pressing need to migrate containers between hosts at some point. In such a dynamic world hard coding IP addresses, or even host names is not a viable solution. This talk will take a journey through how Yelp has solved the discovery problems using Airbnb's SmartStack to dynamically discover service dependencies, and how this is helping unify our architecture, from traditional metal to EC2 'immutable' SOA images, to Docker containers.'
meetups talks dublin deployment smartstack ec2 docker yelp service-discovery
Smart Integration Testing with Dropwizard, Flyway and Retrofit
Retrofit in particular looks neat. Mind you having worked with in-memory SQL databases before for integration testing, I'd never do that again -- too many interop glitches compared to "real world" MySQL/Postgres
testing integration-testing retrofit flyway dropwizard logentries
-
TSAR = "Time Series AggregatoR". Twitter's new event processor-style architecture for internal metrics. It's notable that now Twitter and Google are both apparently moving towards this idea of a model of code which is designed to run equally in realtime streaming and batch modes (Summingbird, Millwheel, Flume).
analytics architecture twitter tsar aggregation event-processing metrics streaming hadoop batch
'Robust De-anonymization of Large Sparse Datasets' [pdf]
paper by Arvind Narayanan and Vitaly Shmatikov, 2008. 'We present a new class of statistical de- anonymization attacks against high-dimensional micro-data, such as individual preferences, recommendations, transaction records and so on. Our techniques are robust to perturbation in the data and tolerate some mistakes in the adversary's background knowledge. We apply our de-anonymization methodology to the Netflix Prize dataset, which contains anonymous movie ratings of 500,000 subscribers of Netflix, the world's largest online movie rental service. We demonstrate that an adversary who knows only a little bit about an individual subscriber can easily identify this subscriber's record in the dataset. Using the Internet Movie Database as the source of background knowledge, we successfully identified the Netflix records of known users, uncovering their apparent political preferences and other potentially sensitive information.'
anonymisation anonymization sanitisation databases data-dumps privacy security papers
HSE data releases may be de-anonymisable
Although the data has been kept anonymous, the increasing sophistication of computer-driven data-mining techniques has led to fears patients could be identified. A HSE spokesman confirmed yesterday that the office responded to requests for data from a variety of sources, including researchers, the universities, GPs, the media, health insurers and pharmaceutical companies. An average of about two requests a week was received. [...] The information provided by the HPO has significant patient identifiers removed, such as name and date of birth. According to the HSE spokesman, individual patient information is not provided and, where information is sought for a small group of patients, this is not provided where the number involved is under five. "In such circumstances, it is highly unlikely that anyone could be identified. Nevertheless, we will have another look at data releases from the office," he said.
I'd say this could be readily reversible, from the sounds of it.anonymisation sanitisation data-dumps hse health privacy via:tjmcintyre
Beautiful algorithm visualisations from Mike Bostock
This is a few days old, but unmissable. I swear, the 'Wilson's algorithm transformed into a tidy tree layout' viz brought tears to my eyes ;)
dataviz algorithms visualization visualisation mazes trees sorting animation mike-bostock
ByteArrayOutputStream is really, really slow sometimes in JDK6
This leads us to the bug. The size of the array is determined by Math.max(buf.length << 1, newcount). Ordinarily, buf.length << 1 returns double buf.length, which would always be much larger than newcount for a 2 byte write. Why was it not? The problem is that for all integers larger than Integer.MAX_INTEGER / 2, shifting left by one place causes overflow, setting the sign bit. The result is a negative integer, which is always less than newcount. So for all byte arrays larger than 1073741824 bytes (i.e. one GB), any write will cause the array to resize, and only to exactly the size required.
Ouch.bugs java jdk6 bytearrayoutputstream impala performance overflow
Cory Doctorow on Thomas Piketty's 'Capital in the 21st Century'
quite a leftie analysis
history capitalism economics piketty capital finance taxation growth money cory-doctorow thomas-piketty
-
Google's purify/valgrind-like concurrency checking tool: 'As a bonus, ThreadSanitizer finds some other types of bugs: thread leaks, deadlocks, incorrect uses of mutexes, malloc calls in signal handlers, and more. It also natively understands atomic operations and thus can find bugs in lock-free algorithms. [...] The tool is supported by both Clang and GCC compilers (only on Linux/Intel64). Using it is very simple: you just need to add a -fsanitize=thread flag during compilation and linking. For Go programs, you simply need to add a -race flag to the go tool (supported on Linux, Mac and Windows).'
concurrency bugs valgrind threadsanitizer threading deadlocks mutexes locking synchronization coding testing
-
'A repair café brings together people with things that need fixin' with people who have the skills to fix them in a social cafe style environment. It is an effort to move away from the throwaway culture that prevailed at the end of the twentieth century and move towards a more sustainable and enlightened approach to our relationship with consumer goods. Repair cafes are self organising events at a community level run by local volunteers with the support of local community groups, local agencies and other interested organisations. They are not-for-profit but not anti-profit and an important part of their goal is to promote local repair businesses and initiatives. www.repaircafe.ie is the online hub of a network of repair cafés across Ireland.' Sounds interesting: https://twitter.com/DubCityCouncil/status/481777655445204992 says they'll be doing it tomorrow from 2-5pm in Sandymount in Dublin.
dublin sandymount repair fixing diy frugality repaircafe hardware
-
A way to securely store secrets (auth details, API keys, etc.) in Chef
chef storage knife authorisation api-keys security encryption
Amazon EC2 Service Limits Report Now Available
'designed to make it easier for you to view and manage your limits for Amazon EC2 by providing the latest information on service limits and links to quickly request limit increases. EC2 Service Limits Report displays all your service limit information in one place to help you avoid encountering limits on future EC2, EBS, Auto Scaling, and VPC usage.'
Delivery Notifications for Simple Email Service
Today we are enhancing SES with the addition of delivery notifications. You can now elect to receive an Amazon SNS notification each time SES successfully delivers a message to a recipient's email server. These notifications give you increased visibility into the mail delivery process. With today's release, you can now track deliveries, bounces, and complaints, all via notification to the SNS topic or topics of your choice.
How Emoji Get Lost In Translation
I recently texted a friend to say how I was excited to meet her new boyfriend, and, because "excited" doesn't look so exciting on an iPhone screen, I editorialized with what seemed then like an innocent "[dancer]". (Translation: Can't wait for the fun night out!) On an Android phone, I realized later, that panache would have been a put-down: The dancers become "[playboy bunny]." (Translation: You're a Playboy bunny who gets around!)
Hailo pulling in EUR1M per month in Dublin alone
based on these (pretty rough) estimates. Good going, I'm a massive fan
Google Replaces MapReduce With New Hyper-Scale Cloud Analytics System
MR no more:
"We don't really use MapReduce anymore," [Urs] Hölzle said in his keynote presentation at the Google I/O conference in San Francisco Wednesday. The company stopped using the system "years ago." Cloud Dataflow, which Google will also offer as a service for developers using its cloud platform, does not have the scaling restrictions of MapReduce. "Cloud Dataflow is the result of over a decade of experience in analytics," Hölzle said. "It will run faster and scale better than pretty much any other system out there."
map-reduce google hadoop cloud-dataflow scalability big-data urs-holzle google-io
This Internet Millionaire Has a New Deal For You - D Magazine
Good interview with Dave "Woot" Rutledge, who's now well out of Amazon and plans to get back into the crap-clearing business at Meh.com: 'Amazon's fundamental misunderstanding of what made Woot great can be seen today on the site. It sells many items simultaneously. It's a marketplace, not an event. The write-ups are cute, not subversively funny. Woot is no longer a bug-eyed beast with eight tentacles. It's a pancake with two smaller pancakes for Mickey Mouse ears and a smile made of whipped cream. In 2012, two years into his three-year deal with Amazon, Rutledge walked. He won't say how many millions his early departure cost him, but his contract with Amazon included a three-year non-compete clause from the date of sale, and he was watching the clock.'
amazon ecommerce business b2c woot.com meh.com dave-rutledge selling acquisitions
NYC generates hash-anonymised data dump, which gets reversed
There are about 1000*26**3 = 21952000 or 22M possible medallion numbers. So, by calculating the md5 hashes of all these numbers (only 24M!), one can completely deanonymise the entire data. Modern computers are fast: so fast that computing the 24M hashes took less than 2 minutes.
(via Bruce Schneier) The better fix is a HMAC (see http://benlog.com/2008/06/19/dont-hash-secrets/ ), or just to assign opaque IDs instead of hashing.hashing sha1 md5 bruce-schneier anonymization deanonymization security new-york nyc taxis data big-data hmac keyed-hashing salting
Older programmers aren't gone, they're just outnumbered
So says "Uncle Bob" Martin
culture coding software age career reputation stack-overflow staffing
Benchmarking LevelDB vs. RocksDB vs. HyperLevelDB vs. LMDB Performance for InfluxDB
A few interesting things come out of these results. LevelDB is the winner on disk space utilization, RocksDB is the winner on reads and deletes, and HyperLevelDB is the winner on writes. On smaller runs (30M or less), LMDB came out on top on most of the metrics except for disk size. This is actually what we'd expect for B-trees: they're faster the fewer keys you have in them.
Mind you, I'd prefer if this had tunable read/write/delete ratios, as YCSB does. Take with a pinch of salt, as with all benchmarks!benchmarks leveldb datastores storage hyperleveldb rocksdb ycsb lmdb influxdb
How to make breaking changes and not break all the things
Well-written description of the "several backward-compatible changes" approach to breaking-change schema migration (via Marc)
databases coding compatibility migration schemas sql continuous-deployment
Minnesota Measles Outbreak Traced Back To A Single Unvaccinated Child
A single child caught measles while visiting Kenya, returned to Minnesota, infected 4 others, who in turn exposed others, with an ultimate count of 3000 exposed and 21 confirmed cases. (16 of the 21 were unvaccinated; 46% of the Somali children in this community were unvaccinated in a 2010 survey.)
Report of the Internet Content Governance Advisory Group
looking at the summary, looks broadly sensible; no government-mandated filtering/blocking I can spot quickly
internet filtering safety kids porn blocking ireland pegi ratings reports pdf
-
'Two months ago, an early Uber employee thought that he had found a buyer for his vested stock, at $200 per share. But when his agent tried to seal the deal, Uber refused to sign off on the transfer. Instead, it offered to buy back the shares for around $135 a piece, which is within the same price range that Google Ventures and TPG Capital had paid to invest in Uber the previous July. Take it or hold it.' As rbranson on Twitter put it: 'reminder that startup equity is basically worthless unless you're a founder or investor, OR the company goes public.'
startups uber stock stock-options shares share-option equity via:rbranson work
-
Wow, these are terrible results. From the sounds of it, ES just cannot deal with realistic outage scenarios and is liable to suffer catastrophic damage in reasonably-common partitions.
If you are an Elasticsearch user (as I am): good luck. Some people actually advocate using Elasticsearch as a primary data store; I think this is somewhat less than advisable at present. If you can, store your data in a safer database, and feed it into Elasticsearch gradually. Have processes in place that continually traverse the system of record, so you can recover from ES data loss automatically.
elasticsearch ops storage databases jepsen partition network outages reliability
Code Spaces data and backups deleted by hackers
Rather scary story of an extortionist wiping out a company's AWS-based infrastructure. Turns out S3 supports MFA-required deletion as a feature, though, which would help against that.
ops security extortion aws ec2 s3 code-spaces delete mfa two-factor-authentication authentication infrastructure
Google forced to e-forget a company worldwide
Here we go.... Canadian company wins case to censor search results for its competitors.
When Google argued that Canadian law couldn't be applied to the entire world, the court responded by citing British Columbia's Law and Equity Act, which grants broad power for a court to issue injunctions when it's "just or convenient that the order should be made." Google also tried to argue against the injunction on the basis of it amounting to censorship. The court responded that there are already entire categories of content that get censored, such as child abuse imagery. Will this be the first of a new wave of requests for company website take-downs?
Via stx.canada via:stx censorship google search takedowns datalink equustek gw1000 hardware
The dark truth about modern Ireland its media don't talk about
Sinead O'Shea writing for the Guardian:
The economy has been built on cronyism, group-think, the double talk of absurdly low corporate tax rates and light touch regulation, the cult of the leader, an over reliance on "strong" international forces. These were the factors that caused the Celtic Tiger to collapse. This has had consequences for all. It's the same for the system of shame and sexual repression. The impact has not been restricted to its most obvious victims. Ireland is not just a bad place to be a woman or an immigrant, it's a bad place to be in any way "different." As a result, sadly, it's a bad place to be anyone at all.
ireland history women celtic-tiger industrial-schools immigration sinead-o-shea tuam abortion pregnancy
Data sharing deal with U.S. referred to EU's top court | Reuters
High Court Justice Gerard Hogan said that given the Safe Harbour agreement, which says that U.S. has sufficient data safeguards in place, the Irish regulator did not have the authority to investigate. If Safe Harbour stands, the student group's application must fail, he said. "The critical issue which arises is whether the proper interpretation of the 1995 [EU data protection] directive and the 2000 Commission decision [on the Safe Harbour principles] should be re-evaluated in the light of the subsequent entry into force of article 8 of the EU charter," on the right to the protection of personal data, Hogan said.
eu safe-harbor privacy high-court ireland law data-protection
-
A good example of "raw" BDD, without using a framework like Cucumber, Steak etc.
-
a minimal extension of RSpec-Rails that adds several conveniences to do acceptance testing of Rails applications using Capybara. It's an alternative to Cucumber in plain Ruby.
Good approach here to copy, but very tied to Rails.rails ruby testing acceptance-testing steak bdd rspec coding
-
Scala's BDD approach -- very similar to Steak in Rubyland I think
quotly/test/acceptance/adding_quotes_spec.rb at master · cavalle/quotly · GitHub
Decent demo of acceptance testing using rspec (and some syntactic sugar to make it read like Steak code, I think)
Facebook introduce "Wedge" and "FBOSS"
a new top-of-rack network switch, code-named "Wedge," and a new Linux-based operating system for that switch, code-named "FBOSS." These projects break down the hardware and software components of the network stack even further, to provide a new level of visibility, automation, and control in the operation of the network. By combining the hardware and software modules together in new ways, "Wedge" and "FBOSS" depart from current networking design paradigms to leverage our experience in operating hundreds of thousands of servers in our data centers. In other words, our goal with these projects was to make our network look, feel, and operate more like the OCP servers we've already deployed, both in terms of hardware and software.
Sayonara, Cisco, and good riddance.cisco juniper wedge fboss facebook tor switches racks networking datacenter routers
Cap'n Proto, FlatBuffers, and SBE
a feature comparison of these new serialization formats from Kenton, the capnp dude
serialization protobuf capnproto sbe flatbuffers google coding storage
Concurrency Improvements in HyperLevelDB
Good-looking benchmark results here from HyperDex
hyperdex hyperleveldb leveldb rocksdb concurrency lock-free storage persistence
-
"Sell products directly to your audience" -- Ben says it doesn't break the flow, doesn't take you to another site, no complicated registration forms -- the customer just enters CC details and that's it.
A dive into a UTF-8 validation regexp
Once again, I find myself checking over the UTF-8 validation code in websocket-driver, and once again I find I cannot ever remember how to make sense of this regex that performs the validation. I just copied it off a webpage once and it took a while (and reimplementing UTF-8 myself) to fully understand what it does. If you write software that processes text, you'll probably need to understand this too.
utf-8 unicode utf8 javascript node encoding text strings validation websockets regular-expressions regexps
-
This [shell one-liner] will take a picture of a whiteboard and use parts of the ImageMagick library with sane defaults to clean it up tremendously.: convert "$1" -morphology Convolve DoG:15,100,0 -negate -normalize -blur 0x1 -channel RBG -level 60%,91%,0.1 "$2"
Some kind soul has put up a quickie web UI here: http://api.o2b.ru/whiteboardcleanergraphics tools whiteboard imagemagick text images cleanup gimp photoshop via:fanf
A Rare Peek Inside Amazon's Massive Wish-Fulfilling Machine
Wired get a tour of PHX6, one of Amazon's FCs
Paleo is the Scientology of Diet
Being paleo is like paying a stupidity tax. Again, it's not you who is stupid, but the diet sure is, because it lets you drink paleo coffee while putting paleo butter and paleo syrup on your paleo waffles before you drive your paleo minivan to the paleo office to sit in your paleo cube and do spreadsheets on your paleo computer. See, the paleo diet made up a bunch of silly rules on how we allegedly ate, and then goes and twists them all to hell in the name of selling you a crappy, overpriced product. That is scientology-level stupid.
scientology paleo rants funny food diet health bulletproof-coffee stupid
-
a power-management subsystem for warehouse-scale computing farms. "It adjusts the power-performance settings of servers so that the overall workload barely meets its latency constraints for user queries."
pegasus power-management power via:fanf google latency scaling
-
A new serialization format from Google's Android gaming team, supporting C++ and Java, open source under the ASL v2. Reasons to use it:
Access to serialized data without parsing/unpacking - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility). Memory efficiency and speed - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations. FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details. Flexible - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures. Tiny code footprint - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details. Strongly typed - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you. Convenient to use - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).
Looks nice, but it misses the language coverage of protobuf. Definitely more practical than capnproto.c++ google java serialization json formats protobuf capnproto storage flatbuffers
AWS SDK for Java Client Configuration
turns out the AWS SDK has lots of tuning knobs: region selection, socket buffer sizes, and debug logging (including wire logging).
-
The simple woven multicoloured bracelet has made Cheong Choon Ng, a Malaysian immigrant to the US, a dollar millionaire. He invented the "Rainbow Loom" after watching his daughters making bracelets with rubber bands.
So, really, it's his daughters that invented it. ;) My kids are massive fans. This is a 100% legit, Rubik's-Cube-style craze. (via Conor O'Neill) -
BorderPatrol is an nginx module to perform authentication and session management at the border of your network. BorderPatrol makes the assumption that you have some set of services that require authentication and a service that hands out tokens to clients to access that service. You may not want those tokens to be sent across the internet, even over SSL, for a variety of reasons. To this end, BorderPatrol maintains a lookup table of session-id to auth token in memcached.
borderpatrol nginx modules authentication session-management web-services http web authorization
Use of Formal Methods at Amazon Web Services
Chris Newcombe, Marc Brooker, et al. writing about their experience using formal specification and model-checking languages (TLA+) in production in AWS:
The success with DynamoDB gave us enough evidence to present TLA+ to the broader engineering community at Amazon. This raised a challenge; how to convey the purpose and benefits of formal methods to an audience of software engineers? Engineers think in terms of debugging rather than 'verification', so we called the presentation "Debugging Designs". Continuing that metaphor, we have found that software engineers more readily grasp the concept and practical value of TLA+ if we dub it 'Exhaustively-testable pseudo-code'. We initially avoid the words 'formal', 'verification', and 'proof', due to the widespread view that formal methods are impractical. We also initially avoid mentioning what the acronym 'TLA' stands for, as doing so would give an incorrect impression of complexity.
More slides at http://tla2012.loria.fr/contributed/newcombe-slides.pdf ; proggit discussion at http://www.reddit.com/r/programming/comments/277fbh/use_of_formal_methods_at_amazon_web_services/formal-methods model-checking tla tla+ programming distsys distcomp ebs s3 dynamodb aws ec2 marc-brooker chris-newcombe
-
We used Knossos and Jepsen to prove the obvious: RabbitMQ is not a lock service. That investigation led to a discovery hinted at by the documentation: in the presence of partitions, RabbitMQ clustering will not only deliver duplicate messages, but will also drop huge volumes of acknowledged messages on the floor. This is not a new result, but it may be surprising if you haven't read the docs closely–especially if you interpreted the phrase "chooses Consistency and Partition Tolerance" to mean, well, either of those things.
rabbitmq network partitions failure cap-theorem consistency ops reliability distcomp jepsen
Jump Consistent Hash: A Fast, Minimal Memory, Consistent Hash Algorithm
'a fast, minimal memory, consistent hash algorithm that can be expressed in about 5 lines of code. In comparison to the algorithm of Karger et al., jump consistent hash requires no storage, is faster, and does a better job of evenly dividing the key space among the buckets and of evenly dividing the workload when the number of buckets changes. Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching.' Implemented in Guava. This is also noteworthy: 'Google has not applied for patent protection for this algorithm, and, as of this writing, has no plans to. Rather, it wishes to contribute this algorithm to the community.'
hashing consistent-hashing google guava memory algorithms sharding
Bike Wheel Spoke ABS Safety Reflective Tube Reflector
Available in blue, orange, and grey for $2.84 from the insanely-cheap China-based DealExtreme.com. Also available: rim-based reflective stickers
#AltDevBlog » Parallel Implementations
John Carmack describes this code-evolution approach to adding new code:
The last two times I did this, I got the software rendering code running on the new platform first, so everything could be tested out at low frame rates, then implemented the hardware accelerated version in parallel, setting things up so you could instantly switch between the two at any time. For a mobile OpenGL ES application being developed on a windows simulator, I opened a completely separate window for the accelerated view, letting me see it simultaneously with the original software implementation. This was a very significant development win. If the task you are working on can be expressed as a pure function that simply processes input parameters into a return structure, it is easy to switch it out for different implementations. If it is a system that maintains internal state or has multiple entry points, you have to be a bit more careful about switching it in and out. If it is a gnarly mess with lots of internal callouts to other systems to maintain parallel state changes, then you have some cleanup to do before trying a parallel implementation. There are two general classes of parallel implementations I work with: The reference implementation, which is much smaller and simpler, but will be maintained continuously, and the experimental implementation, where you expect one version to "win" and consign the other implementation to source control in a couple weeks after you have some confidence that it is both fully functional and a real improvement. It is completely reasonable to violate some generally good coding rules while building an experimental implementation – copy, paste, and find-replace rename is actually a good way to start. Code fearlessly on the copy, while the original remains fully functional and unmolested. It is often tempting to shortcut this by passing in some kind of option flag to existing code, rather than enabling a full parallel implementation. It is a grey area, but I have been tending to find the extra path complexity with the flag approach often leads to messing up both versions as you work, and you usually compromise both implementations to some degree.
(via Marc)via:marc coding john-carmack parallel development evolution lifecycle project-management
5 Reasons to Use Protocol Buffers Instead of JSON For Your Next Service
A good writeup of the case for protobuf > JSON (via Marc)
via:marc api soa web-services protobuf json interop protocols marshalling
Plumbr.eu's reference page for java.lang.OutOfMemoryError
With examples of each possible cause of a Java OOM, and suggested workarounds. succinct
-
Imagine buying a high-end Core i7 or AMD CPU, opening the box, and finding a midrange part sitting there with an asterisk and the label "Performs Just Like Our High End CPU In Single-Threaded SuperPi!"
ssd storage hardware sketchy kingston pny bait-and-switch components vendors via:hn
-
Manages migrations for your Cassandra data stores. Pillar grew from a desire to automatically manage Cassandra schema as code. Managing schema as code enables automated build and deployment, a foundational practice for an organization striving to achieve Continuous Delivery. Pillar is to Cassandra what Rails ActiveRecord migrations or Play Evolutions are to relational databases with one key difference: Pillar is completely independent from any application development framework.
migrations database ops pillar cassandra activerecord scala continuous-delivery automation build
How to use TuneIn's Record Timer feature
handy
Continuous Deployment for Mobile Apps with Jenkins: iOS Builds
the CloudBees-std way
-
'A tribute to the addictive Ktarian Game featured in Season 5, Episode 6 of Star Trek, The Next Generation.'
-
a single application IP packet sniffer that captures all TCP and UDP packets of a single Linux process. It consists of the following elements: * ptrace monitor - tracks bind(), connect() and sendto() syscalls and extracts local port numbers that the traced application uses; * pcap sniffer - using information from the previous module, it captures IP packets on an AF_PACKET socket (with an appropriate BPF filter attached); * garbage collector - periodically reads /proc/net/{tcp,udp} files in order to detect the sockets that the application no longer uses. As the output, tracedump generates a PCAP file with SLL-encapsulated IP packets - readable by eg. Wireshark. This file can be later used for detailed analysis of the networking operations made by the application. For instance, it might be useful for IP traffic classification systems.
debugging networking linux strace ptrace tracedump tracing tcp udp sniffer ip tcpdump
You Are Not a Digital Native: Privacy in the Age of the Internet
an open letter from Cory Doctorow to teen readers re privacy. 'The problem with being a "digital native" is that it transforms all of your screw-ups into revealed deep truths about how humans are supposed to use the Internet. So if you make mistakes with your Internet privacy, not only do the companies who set the stage for those mistakes (and profited from them) get off Scot-free, but everyone else who raises privacy concerns is dismissed out of hand. After all, if the "digital natives" supposedly don't care about their privacy, then anyone who does is a laughable, dinosauric idiot, who isn't Down With the Kids.'
children privacy kids teens digital-natives surveillance cory-doctorow danah-boyd
Shutterbits replacing hardware load balancers with local BGP daemons and anycast
Interesting approach. Potentially risky, though -- heavy use of anycast on a large-scale datacenter network could increase the scale of the OSPF graph, which scales exponentially. This can have major side effects on OSPF reconvergence time, which creates an interesting class of network outage in the event of OSPF flapping. Having said that, an active/passive failover LB pair will already announce a single anycast virtual IP anyway, so, assuming there are a similar number of anycast IPs in the end, it may not have any negative side effects. There's also the inherent limitation noted in the second-to-last paragraph; 'It comes down to what your hardware router can handle for ECMP. I know a Juniper MX240 can handle 16 next-hops, and have heard rumors that a software update will bump this to 64, but again this is something to keep in mind'. Taking a leaf from the LB design, and using BGP to load-balance across a smaller set of haproxy instances, would seem like a good approach to scale up.
scalability networking performance load-balancing bgp exabgp ospf anycast routing datacenters scaling vips juniper haproxy shutterstock
Tron: Legacy Encom Boardroom Visualization
this is great. lovely, silly, HTML5 dataviz, with lots of spinning globes and wobbling sines on a black background
demo github wikipedia dataviz visualisation mapping globes rob-scanlan graphics html5 animation tron-legacy tron movies
-
a distributed key/value datastore which supports ACID transactional semantics and versioned values as first-class features. The primary design goal is global consistency and survivability, hence the name. Cockroach aims to tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention. Cockroach nodes are symmetric; a design goal is one binary with minimal configuration and no required auxiliary services. Cockroach implements a single, monolithic sorted map from key to value where both keys and values are byte strings (not unicode). Cockroach scales linearly (theoretically up to 4 exabytes (4E) of logical data). The map is composed of one or more ranges and each range is backed by data stored in RocksDB (a variant of LevelDB), and is replicated to a total of three or more cockroach servers. Ranges are defined by start and end keys. Ranges are merged and split to maintain total byte size within a globally configurable min/max size interval. Range sizes default to target 64M in order to facilitate quick splits and merges and to distribute load at hotspots within a key range. Range replicas are intended to be located in disparate datacenters for survivability (e.g. { US-East, US-West, Japan }, { Ireland, US-East, US-West}, { Ireland, US-East, US-West, Japan, Australia }). Single mutations to ranges are mediated via an instance of a distributed consensus algorithm to ensure consistency. We've chosen to use the Raft consensus algorithm. All consensus state is stored in RocksDB. A single logical mutation may affect multiple key/value pairs. Logical mutations have ACID transactional semantics. If all keys affected by a logical mutation fall within the same range, atomicity and consistency are guaranteed by Raft; this is the fast commit path. Otherwise, a non-locking distributed commit protocol is employed between affected ranges. Cockroach provides snapshot isolation (SI) and serializable snapshot isolation (SSI) semantics, allowing externally consistent, lock-free reads and writes--both from an historical snapshot timestamp and from the current wall clock time. SI provides lock-free reads and writes but still allows write skew. SSI eliminates write skew, but introduces a performance hit in the case of a contentious system. SSI is the default isolation; clients must consciously decide to trade correctness for performance. Cockroach implements a limited form of linearalizability, providing ordering for any observer or chain of observers.
This looks nifty. One to watch.cockroachdb databases storage georeplication raft consensus acid go key-value-stores rocksdb
-
good docs from Riak
-
method for bootstrapping one cryptocurrency off of another. The idea is that miners should show proof that they burned some coins - that is, sent them to a verifiably unspendable address. This is expensive from their individual point of view, just like proof of work; but it consumes no resources other than the burned underlying asset. To date, all proof of burn cryptocurrencies work by burning proof-of-work-mined cryptocurrencies, so the ultimate source of scarcity remains the proof-of-work-mined "fuel".
The programming error that cost Mt Gox 2609 bitcoins
Digging into broken Bitcoin scripts in the blockchain. Fascinating:
While analyzing coinbase transactions, I came across another interesting bug that lost bitcoins. Some transactions have the meaningless and unredeemable script: OP_IFDUP OP_IF OP_2SWAP OP_VERIFY OP_2OVER OP_DEPTH That script turns out to be the ASCII text script. Instead of putting the redemption script into the transaction, the P2Pool miners accidentally put in the literal word "script". The associated bitcoins are lost forever due to this error.
(via Nelson)programming script coding bitcoin mtgox via:nelson scripting dsls
-
a Java implementation of an MQTT 3.1 broker. Its code base is small. At its core, Moquette is an events processor; this lets the code base be simple, avoiding thread sharing issues. The Moquette broker is lightweight and easy to understand so it could be embedded in other projects.
mqtt moquette netty messaging queueing push-notifications iot internet push eclipse
-
aka. lock acquisition. ex-Amazon-Dublin lingo, observed in the wild ;)
language hotdog archie-mcphee amazon dublin intercom coding locks synchronization
Organic Cat Litter Chief Suspect In Nuclear Waste Accident
What a headline. interesting story to boot (via Eoin)
environment energy chemistry cat-litter waste-disposal nuclear-waste accidents new-mexico
Friends don't let friends use mmap(2)
Rather horrific update from the trenches of Mozilla
67 Books Every Geek Should Read to Their Kids Before Age 10 | GeekDad | Wired.com
Lots and lots of good book recommendations, a little US-centric though
How the patent trolls won in Congress: Ars Technica
"We felt really good the last couple of days," said the tech lobbyist. "It was a good deal—one we could live with. Then the trial lawyers and pharma went to Senator Reid late this morning and said that's it. Enough with the children playing in the playground—go kill it."
ars-technica patents swpats patent-trolls pharma tech us-politics congress lawyers
Dublin City North Inner City count results, animated
A nice visualisation of Single-Transferable-Vote proportional representation in action
New Statesman: Let's call the Isla Vista killings what they were: misogynist extremism
We have been told for a long time that the best way to deal with this sort of harrassment and violence is to laugh it off. Women and girls and queer people have been told that online misogynists pose no real threat, even when they're sharing intimate guides to how to destroy a woman's self-esteem and force her into sexual submission. Well, now we have seen what the new ideology of misogyny looks like at its most extreme. We have seen incontrovertible evidence of real people being shot and killed in the name of that ideology, by a young man barely out of childhood himself who had been seduced into a disturbing cult of woman-hatred. Elliot Rodger was a victim - but not for the reasons he believed.
elliot-rodger extremism feminism isla-vista mass-killings pua mens-rights harrassment misogyny penny-red
-
'In essence Tcpdump asks the kernel to execute a BPF program within the kernel context. This might sound risky, but actually isn't. Before executing the BPF bytecode kernel ensures that it's safe: * All the jumps are only forward, which guarantees that there aren't any loops in the BPF program. Therefore it must terminate. * All instructions, especially memory reads are valid and within range. * The single BPF program has less than 4096 instructions. All this guarantees that the BPF programs executed within kernel context will run fast and will never infinitely loop. That means the BPF programs are not Turing complete, but in practice they are expressive enough for the job and deal with packet filtering very well.' Good example of a carefully-designed DSL allowing safe "programs" to be written and executed in a privileged context without security risk, or risk of running out of control.
coding dsl security via:oisin linux tcpdump bpf bsd kernel turing-complete configuration languages
Handmade Kitchen Goods from Makers & Brothers - Cool Hunting
lovely kitchen-gear design from local-boys-made-good Makers & Brothers
makers-and-brothers design crafts kitchen nyc terrazo chopping-boards
'Monitoring and detecting causes of failures of network paths', US patent 8,661,295 (B1)
The first software patent in my name -- couldn't avoid it forever :(
Systems and methods are provided for monitoring and detecting causes of failures of network paths. The system collects performance information from a plurality of nodes and links in a network, aggregates the collected performance information across paths in the network, processes the aggregated performance information for detecting failures on the paths, analyzes each of the detected failures to determine at least one root cause, and initiates a remedial workflow for the at least one root cause determined. In some aspects, processing the aggregated information may include performing a statistical regression analysis or otherwise solving a set of equations for the performance indications on each of a plurality of paths. In another aspect, the system may also include an interface which makes available for display one or more of the network topology, the collected and aggregated performance information, and indications of the detected failures in the topology.
The patent describes an early version of Pimms, the network failure detection and remediation system we built for Amazon.amazon pimms swpats patents networking ospf autoremediation outage-detection
Monitoring Reactive Applications with Kamon
"quality monitoring tools for apps built in Akka, Spray and Play!". Uses Gil Tene's HDRHistogram and dropwizard Metrics under the hood.
metrics dropwizard hdrhistogram gil-tene kamon akka spray play reactive statistics java scala percentiles latency
-
storage of structured data in a continuous block of memory. The memory can be allocated on the heap using a byte[] array or can be allocated off the java heap in native memory. [...] Use cases: store/cache huge amounts of data records without impact on GC duration; high performance data transfer in a cluster or in between processes
handy OSS from Ruediger Moeller Dynamic Tuple Performance On the JVM
More JVM off-heap storage from Boundary:
generates heterogeneous collections of primitive values and ensures as best it can that they will be laid out adjacently in memory. The individual values in the tuple can either be accessed from a statically bound interface, via an indexed accessor, or via reflective or other dynamic invocation techniques. FastTuple is designed to deal with a large number of tuples therefore it will also attempt to pool tuples such that they do not add significantly to the GC load of a system. FastTuple is also capable of allocating the tuple value storage entirely off-heap, using Java's direct memory capabilities.
-
Teaches the basics of computer science - K-8 Intro to CS, 15-25 hours. Introduces core CS and programming concepts, with lots of nice graphics, scenarios and characters from games to get the kids hooked ;) Recommended by Tom Raftery; his youngest (7yo) is having great fun with it.
education programming learning coding kids k-8 code.org games
-
one of the world's leading news organizations giving itself a rigorous self-examination. I've spoken with multiple digital-savvy Times staffers in recent days who described the report with words like "transformative" and "incredibly important" and "a big big moment for the future of the Times." One admitted crying while reading it because it surfaced so many issues about Times culture that digital types have been struggling to overcome for years.
via Antoin. This is pretty insightful -- the death of the homepage is notablenytimes publishing media journalism tech internet web news leaks via:antoin
Microsoft Security Essentials reporting false positives on the Bitcoin blockchain
Earlier today, a virus signature from the virus "DOS/STONED" was uploaded into the Bitcoin blockchain, which allows small snippets of text to accompany user transactions with bitcoin. Since this is only the virus signature and not the virus itself, there apparently is no danger to users in any way. However, MSE recognizes the signature for the virus and continuously reports it as a threat, and every time it deletes the file, the bitcoin client will simply re-download the missing blockchain.
What a heinous prank! Hilarity ensues (via gwire)via:gwire av antivirus false-positives fp blockchain microsoft bitcoin pranks viruses
Stuck in the iMessage abyss? Here's how to get your texts back
some potential (apocryphal) workarounds for this extremely annoying Apple bug
-
an extension of the core Spark API that allows enables high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets and be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark's in-built machine learning algorithms, and graph processing algorithms on data streams.
spark streams stream-processing cep scalability apache machine-learning graphs
"Crypto Won't Save You Either"
fantastic slides from Peter Gutmann
crypto cryptography security exploits nsa gchq dual_ec_drbg rsa
In the Zone: How Gamers Experience The Real Chernobyl
Great article. I've experienced this in LA, particularly, after playing the GTA series
pripyat gaming tourism reality virtual-reality via:oceanclub games
Shapecatcher: Draw the Unicode character you want!
'This is a tool to help you find Unicode characters. Finding a specific character whose name you don't know is cumbersome. On shapecatcher.com, all you need to know is the shape of the character!' Handy.
Unchi-kun Candy - Japanese Lucky Poop Candy
What doesn't look like Christmas more than a smiling piece of poop, called unchi in Japanese? Because the shape of unchi looks similar to that of mochi used for shrine offerings, and because the sound "unchi" like the Japanese word for luck, this treat is actually a lucky gift -- at least that is how you can explain yourself when you give it as a gift. Each Unchi-kun comes packed with poop candy, taken out from the bottom. Once finished eating, you can open the slot in the back with a box-cutter and turn it into a bank.
Want one!unchi-kun unchi pile-of-poo emoji unicode cute funny japan j-list sweets food gross candy
Bletchley Park Trust erects "Berlin Wall" to cut off on-site computer history museum - Boing Boing
The Bletchley Park trust have erected a fence, nicknamed "The Berlin Wall," between their well-funded museum and its poorer on-site neighbour, the UK National Museum of Computing, which houses the hand-built replica of the codebreaking Colossus computer. The trust received an £8m lottery-funded grant and set about shitcanning long-serving volunteers, cutting off the computer history museum, and generally behaving like greedy jerks, systematically alienating long-term supporters. Oh, and there was that Snowden business.
WTF. Stupid antics.-
Good benchmark data on the performance of JVM exceptions
java jvm exceptions benchmarking performance optimization coding
-
Oh Apple, you asshats. This is some seriously shitty programming. iMessage on iOS devices caches the "iMessage-capable" flag for all numbers, indefinitely, so if you switch from iPhone to Android, messages from your friends' iPhones won't get delivered to you henceforth -- and to add insult to injury, it claims they do with a "Delivered." status appearing under the message. This is happening to me right now...
apple sms messaging phones mobile imessage android fail bad-programming bugs
-
an Android-based stingray (IMSI catcher) detector that uses machine learning to detect the presence of stingray devices which can be used to eavesdrop on cellular communication.
In pre-launch right now. Via EthanZ via Antoinimsi-catcher stingray surveillance via:ethanz snooping spying privacy mobile
-
+1. What has happened at Google? Did they fire ever employee in the UX department?
Spark - A small web framework for Java
A Sinatra-like minimal web framework built on Java 8 lambdas:
public class HelloWorld { public static void main(String[] args) { get("/hello", (request, response) -> { return "Hello World!"; }); } }via:sampullara web java sinatra lambdas closures java8 spark
Building a Global, Highly Available Service Discovery Infrastructure with ZooKeeper
This is the written version of a presentation [Camille Fournier] made at the ZooKeeper Users Meetup at Strata/Hadoop World in October, 2012 (slides available here). This writeup expects some knowledge of ZooKeeper.
good advice from one of the ZK committers.zookeeper service-discovery architecture distcomp camille-fournier availability wan network
"Replicated abstract data types: Building blocks for collaborative applications"
cited at https://news.ycombinator.com/item?id=7737423 as 'one of my favorite papers on CRDTs and provides practical pseudocode for learning how to implement CRDTs yourself', in a discussion on cemerick's "Distributed Systems and the End of the API": http://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/
distcomp networking distributed crdts algorithms text data-structures cap
Android 4.4 KitKat Problems: HTC UK Speaks Out About HTC One Update Issues
Good advice on improving battery life with the KitKat 4.4.2 point release on a HTC One. I had serious battery problems, but they seem to have been resolved by following this advice
wifi kitkat android 4.4.2 htc-one htc battery blinkfeed mobile phones
Transform any text into a patent application
'An apparatus and device for staring into vacancy. The devices comprises a good cage, a narrow gangway, an electric pocket, a flower-bedecked cage, an insensitive felt.' (The Hunger Artist by Kafka)
The Big List of Local Multiplayer Games
authoritative!
-
Mock Boto: 'a library that allows your python tests to easily mock out the boto library.' Supports S3, Autoscaling, EC2, DynamoDB, ELB, Route53, SES, SQS, and STS currently, and even supports a standalone server mode, to act as a mock service for non-Python clients. Excellent! (via Conor McDermottroe)
python aws testing mocks mocking system-tests unit-tests coding ec2 s3
Why Disqus made the Python->Go switchover
for their realtime component, from the horse's mouth:
at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.
python languages concurrency go threading gevent scalability disqus realtime hn
Database Migrations Done Right
The rule is simple. You should never tie database migrations to application deploys or vice versa. By minimising dependencies you enable faster, easier and cleaner deployments.
A solid description of why this is a good idea, from an ex-Guardian dev.migrations database sql mysql postgres deployment ops dependencies loose-coupling
-
some cute brooches/jewellery here, for the next time I need to pick a nice gift
Building a large scale CDN with Apache Traffic Server
via Ilya Grigorik: 'Great under-the-hood look at how Comcast built and operates their internal CDN for delivering video (on-demand + live). Some highlights: switched to own (open-source) stack; ~250 servers pushing ~1.5Pb of data/day with ~5Pb of storage capacity.'
An analysis of Facebook photo caching
excellent analysis of caching behaviour at scale, from the FB engineering blog (via Tony Finch)
via:fanf caching facebook architecture photos images cache fifo lru scalability
-
good advice. next time I go over, I'll have to get a Clipper card. Also: 'Brunch is its own section because I have never encountered a place that takes brunch so seriously.'
Alexey Shipilev on Java's System.nanoTime()
System.nanoTime is as bad as String.intern now: you can use it, but use it wisely. The latency, granularity, and scalability effects introduced by timers may and will affect your measurements if done without proper rigor. This is one of the many reasons why System.nanoTime should be abstracted from the users by benchmarking frameworks, monitoring tools, profilers, and other tools written by people who have time to track if the underlying platform is capable of doing what we want it to do. In some cases, there is no good solution to the problem at hand. Some things are not directly measurable. Some things are measurable with unpractical overheads. Internalize that fact, weep a little, and move on to building the indirect experiments. This is not the Wonderland, Alice. Understanding how the Universe works often needs side routes to explore. In all seriousness, we should be happy our $1000 hardware can measure 30 nanosecond intervals pretty reliably. This is roughly the time needed for the Internet packets originating from my home router to leave my apartment. What else do you want, you spoiled brats?
benchmarking jdk java measurement nanoseconds nsecs nanotime jvm alexey-shipilev jmh
-
aka. "zero-shot learning". ok starting point
-
Ilya Grigorik describes the design of the Bitcoin/altcoin block chain algorithm. Illuminating writeup
-
The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave. Optionally, the container can be committed, so that (for example) manual QA could be performed by the container being imported into a local docker provider, and run from there.
The holy grail of Jenkins/Docker integration. How cool is that...jenkins docker ops testing ec2 hosting scaling elastic-scaling system-testing
-
an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. [...] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message. The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer. The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations. The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again. The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.
sbe encoding protobuf protocol-buffers json messages messaging binary formats low-latency martin-thompson xml
Observations of an Internet Middleman
That leaves the remaining six [consumer ISPs peering with Level3] with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content. Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.
Amazing that L3 are happy to publish this -- that's where big monopoly ISPs have led their industry.net-neutrality networking internet level3 congestion isps us-politics
interview with Google VP of SRE Ben Treynor
interviewed by Niall Murphy, no less ;). Some good info on what Google deems important from an ops/SRE perspective
Faster BAM Sorting with SAMtools and RocksDB
Now this is really really clever. Heap-merging a heavyweight genomics format, using RocksDB to speed it up.
There's a problem with the single-pass merge described above when the number of intermediate files, N/R, is large. Merging the sorted intermediate files in limited memory requires constantly reading little bits from all those files, incurring a lot of disk seeks on rotating drives. In fact, at some point, samtools sort performance becomes effectively bound to disk seeking. [...] In this scenario, samtools rocksort can sort the same data in much less time, using no more memory, by invoking RocksDB's background compaction capabilities. With a few extra lines of code we configure RocksDB so that, while we're still in the process of loading the BAM data, it runs additional background threads to merge batches of existing sorted temporary files into fewer, larger, sorted files. Just like the final merge, each background compaction requires only a modest amount of working memory.
(via the RocksDB facebook group)rocksdb algorithms sorting leveldb bam samtools merging heaps compaction
Coding For Life (Battery Life, That Is)
great presentation on Android mobile battery life, and what to avoid
presentations via:sergio android mobile battery battery-life 3g wifi gprs hardware
Oisin's mobile app release checklist
'This form is to document the testing that has been done on each app version before submitting to the App Store. For each item, indicate Yes if the testing has been done, Not Applicable if the testing does not apply (eg testing audio for an app that doesn't play any), or No if the testing has not been done for another reason.'
"A New Data Structure For Cumulative Frequency Tables"
paper by Peter M Fenwick, 1993. 'A new method (the 'binary indexed tree') is presented for maintaining the cumulative frequencies which are needed to support dynamic arithmetic data compression. It is based on a decomposition of the cumulative frequencies into portions which parallel the binary representation of the index of the table element (or symbol). The operations to traverse the data structure are based on the binary coding of the index. In comparison with previous methods, the binary indexed tree is faster, using more compact data and simpler code. The access time for all operations is either constant or proportional to the logarithm of the table size. In conjunction with the compact data structure, this makes the new method particularly suitable for large symbol alphabets.' via Jakob Buchgraber, who's implementing it right now in Netty ;)
netty frequency-tables data-structures algorithms coding binary-tree indexing compression symbol-alphabets
-
Patent trolls have sued or threatened to sue tens of thousands of end-users. For example, Innovatio attacked cafes, bakeries, and even a funeral parlor for using off-the-shelf Wi-Fi routers. And the notorious scanner troll, MPHJ, targeted small businesses and nonprofits around the country for using ordinary office equipment. As a recent paper explained: "Mass suits against technology customers have become too common, involving building block technologies like wi-fi, scanning, email and website technologies." The growth in patent suits against customers reveals the importance of the Limelight case. A ruling that made it even easier to sue customers (by allowing suits against someone who performs just some steps of a patent) would encourage patent trolls to launch more abusive litigation campaigns. We hope the Supreme Court will restore the sensible rule that only a single entity (or its agents) can infringe a patent.
patents uspto swpats eff consumer law legal patent-infringement scanners wifi printers
Hanging on the telephone – has anyone got it right on the new ban on text driving?
Some good legal commentary on this new Irish law.
There has been much hand-wringing and concern about whether or not the 2014 Regulations prohibit the use of Google Maps or Hailo, for example. They don't, but this does not mean that drivers should feel free to use non-texting functions of their phones while driving – holding a mobile phone (which could include a tablet) while driving remains prohibited, whatever the use it is being put to. Moreover, offences of dangerous and careless driving and driving without due care and attention could cover a wide range of bad driving, and could include, for example, driving while zooming in and out of maps on your phone or sending stickers on WhatsApp.
ireland law driving safety mobile-phones texting google-maps satnav
-
'better dates and times for Python', to fix the absurd proliferation of slightly-incompatible Python date/time types and APIs. unfortunately, http://imgs.xkcd.com/comics/standards.png applies....
python libraries time dates timestamps timezones apis proliferation iso-8601
Holdings: Guinness's Brewery Dublin
'Guinness's Brewery Dublin. Malt House, malt on floor; sign' - One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection
nli ireland photos t-h-mason history dublin guinness maltings beer
Published image: 'An Irish Village'.
'Cart, man/woman; 2 men and boy serving beer outside, + sign 'Rich King Spirits'. Ragged attire' - One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection
-
One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection
-
One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection
-
One of the SmartStack developers at AirBNB responds to Consul.io's comments. FWIW, we use SmartStack in Swrve and it works pretty well...
smartstack airbnb ops consul serf load-balancing availability resiliency network-partitions outages
A Closer Look At OC's Anti-Vaccination Cluster
In communities such as San Clemente, Laguna Beach, Laguna Niguel, Aliso Viejo, Mission Viejo and Capistrano Beach, where Dr. Bob Sears practices, there are clusters of unvaccinated children. Last year, at 15 of the 40 elementary schools in the Capistrano Unified School District, more than 10 percent of kindergartners had [Personal Belief exemptions], according to data from the California Department of Public Health. At one public charter school, Journey, 56 percent of kindergartners were unvaccinated, at least partially, due to their parents' beliefs.
This is going to end horribly. Typical OCorange-county health vaccination laguna-beach oc dr-bob-sears kindergarten measles mumps rubella pertussis epidemiology
-
Today we're open sourcing Secor, a zero data loss log persistence service whose initial use case was to save logs produced by our monetization pipeline. Secor persists Kafka logs to long-term storage such as Amazon S3. It's not affected by S3's weak eventual consistency model, incurs no data loss, scales horizontally, and optionally partitions data based on date.
pinterest hadoop secor storm kafka architecture s3 logs archival
Coping with the TCP TIME-WAIT state on busy Linux servers
extensive blog post
Flood IO Offering Network Emulation
Performance-testing-as-a-service company Flood.IO now offering emulation of various crappy end-user networks: GSM, DSL, GPRS, 3G, 4G etc. Great idea.
flood.io performance networking internet load-testing testing jmeter gatling tests gsm 3g mobile simulation
-
Disqus' realtime architecture -- nginx PushStream module doing the heavy lifting, basically. See https://gist.github.com/dctrwatson/0b3b52050254e273ff11 for the production nginx configs they use. I am very impressed that push-stream has grown to be so solid; it's a great way to deal with push from the sounds of it. http://blog.disqus.com/post/51155103801/trying-out-this-go-thing now notes that some of the realtime backends are in Go. https://speakerdeck.com/dctrwatson/c1m-and-nginx ("C1M and Nginx") is a more up to date presentation. It notes that PushStream supports "EventSource, WebSocket, Long Polling, and forever iframe". More sysctls and nginx tuning in that prez.
sysctl nginx tuning go disqus realtime push eventsource websockets long-polling iframe python
'The Design And Implementation Of Modern Column-Oriented Database Systems'
paper, PDF; Daniel Abadi et al.
papers pdf columnar-stores column-oriented databases storage architecture algorithms
'Pickles & Spores: Improving Support for Distributed Programming in Scala
'Spores are "small units of possibly mobile functional behavior". They're a closure-like abstraction meant for use in distributed or concurrent environments. Spores provide a guarantee that the environment is effectively immutable, and safe to ship over the wire. Spores aim to give library authors some confidence in exposing functions (or, rather, spores) in public APIs for safe consumption in a distributed or concurrent environment. The first part of the talk covers a simpler variant of spores as they are proposed for inclusion in Scala 2.11. The second part of the talk briefly introduces a current research project ongoing at EPFL which leverages Scala's type system to provide type constraints that give authors finer-grained control over spore capturing semantics. What's more, these type constraints can be composed during spore composition, so library authors are effectively able to propagate expert knowledge via these composable constraints. The last part of the talk briefly covers Scala/Pickling, a fast new, open serialization framework.'
pickling scala presentations spores closures fp immutability coding distributed distcomp serialization formats network
BBC News - Microsoft 'must release' data held on Dublin server
Messy. I can't see this lasting beyond an appeal.
Law enforcement efforts would be seriously impeded and the burden on the government would be substantial if they had to co-ordinate with foreign governments to obtain this sort of information from internet service providers such as Microsoft and Google, Judge Francis said. In a blog post, Microsoft's deputy general counsel, David Howard, said: "A US prosecutor cannot obtain a US warrant to search someone's home located in another country, just as another country's prosecutor cannot obtain a court order in her home country to conduct a search in the United States. "We think the same rules should apply in the online world, but the government disagrees."
microsoft regions law us-law privacy google cloud international-law surveillance
Russia passes bill requiring bloggers to register with government
A bill passed by the Russian parliament on Tuesday says that any blogger read by at least 3,000 people a day has to register with the government telecom watchdog and follow the same rules as those imposed by Russian law on mass media. These include privacy safeguards, the obligation to check all facts, silent days before elections and loose but threatening injunctions against "abetting terrorism" and "extremism."
Russian blogging platforms have responded by changing view-counter tickers to display "2500+" as a max.russia blogs blogging terrorism extremism internet regulation chilling-effects censorship
-
as used in the Apollo guidance computer systems -- hand-woven by "little old ladies". Amazing
core-memory memory rope-core guidance apollo space nasa history 1960s via:hn
How I used Heartbleed to steal a site's private crypto key
good writeup from Robin Xu
robin-xu heartbleed rsa private-keys openssl hacking security tls ssl
Making runbooks more useful by exposing them through monitoring
Nice example of an ops runbook wiki for a service
-
Ubuntu, C*, HAProxy, MySQL, RDS, multiple AWS regions.
hailo cassandra ubuntu mysql rds aws ec2 haproxy architecture
They called it "big iron" for a reason: the Cray Motor-Generator Unit
I think the deal with the Motor-Generator Unit was that the Cray 1 needed not just enormous amounts of power (over a hundred kilowatts!), but also very stable power. So it ran from a huge electric generator connected directly to a huge electric motor, the motor running from dirty grid power and the generator, in turn, feeding the computer's own multi-voltage PSU. The Cray 1 itself weighed a mere 2.4 tonnes, but all this support stuff added several more tonnes.
via RobS.via:rob-synnott cray history big-iron motors power electricity generators
Go: Best Practices for Production Environments
how Soundcloud deploy their Go services, after 2.5 years of Go in production
-
'Location Codes for Irish Addresses'. Looks like, as expected, this will not have no-cost licensing terms; companies and non-profit orgs will all have to pay Capita Business Support Services Ireland for access. boo.
eircode mapping addressing geocoding ireland open-source licensing postcodes
Why your company shouldn't use Git submodules
'It is not uncommon at all when working on any kind of larger-scale project with Git to find yourself wanting to share code between multiple different repositories – whether it be some core system among multiple different products built on top of that system, or perhaps a shared utility library between projects. At first glance, Git submodules seem to be the perfect answer for this: they come built-in with Git, they act like miniature repositories (so people are already familiar with how to change them), et cetera. They even support pointing at specific versions of the shared code, so if one project doesn't want to deal with integrating the "latest and greatest" version, it doesn't have to. It's after you've actually worked with submodules for a while that you start to notice just how half-baked Git's submodules system really is.'
Eyes Over Compton: How Police Spied on a Whole City
The law-enforcement pervasive-surveillance CCTV PVR.
In a secret test of mass surveillance technology, the Los Angeles County Sheriff's Department sent a civilian aircraft* over Compton, California, capturing high-resolution video of everything that happened inside that 10-square-mile municipality. Compton residents weren't told about the spying, which happened in 2012. "We literally watched all of Compton during the times that we were flying, so we could zoom in anywhere within the city of Compton and follow cars and see people," Ross McNutt of Persistence Surveillance Systems told the Center for Investigative Reporting, which unearthed and did the first reporting on this important story. The technology he's trying to sell to police departments all over America can stay aloft for up to six hours. Like Google Earth, it enables police to zoom in on certain areas. And like TiVo, it permits them to rewind, so that they can look back and see what happened anywhere they weren't watching in real time.
(via New Aesthetic)pvr cctv law-enforcement police compton los-angeles law surveillance future
-
Blog post from the ES team. They use "evil tests" -- basically unit/system tests, particularly using randomized error-injecting mock infrastructure. Good practices; I've done the same myself quite recently for Swrve's realtime infrastructure
elasticsearch resiliency network-partitions reliability testing mocking error-injection
Meet Ireland's first bitcoin politician
Ossian Smyth -- Green Party internet spokesman and representative for communications, energy, and natural resources, with a top wheeze: "I think it is one of the most transparent ways of receiving donations. No one would know how much money can be donated into a bank account, but with bitcoin anyone can go to the block chain and look at the wallet." excellent ;)
ossian-smyth bitcoin fundraising greens politics ireland dublin green-party internet
OpenPostCode demolishes the planned "Eircode" postcode system
Comprehensively ripped to shreds. Bottom line: 'Postcodes will be largely meaningless to anyone without access to the pay-walled database. It is another tax on business.'
postcodes ireland eircode addressing geocoding mapping maps open-data
Netflix comes out strongly against Comcast
In sum, Comcast is not charging Netflix for transit service. It is charging Netflix for access to its subscribers. Comcast also charges its subscribers for access to Internet content providers like Netflix. In this way, Comcast is double dipping by getting both its subscribers and Internet content providers to pay for access to each other.
FIGHT!netflix comcast network-neutrality cartels competition us-politics business isps
co-founder of the Boston Beer Company swears by active dry yeast as a hangover-avoidance remedy
what [Joe] Owades knew was that active dry yeast has an enzyme in it called alcohol dehydrogenases (ADH). Roughly put, ADH is able to break alcohol molecules down into their constituent parts of carbon, hydrogen, and oxygen. Which is the same thing that happens when your body metabolizes alcohol in its liver. Owades realized if you also have that enzyme in your stomach when the alcohol first hits it, the ADH will begin breaking it down before it gets into your bloodstream and, thus, your brain.
Plausible!beer science health yeast alcohol adh medicine enzymes stomach food
-
At Comcast, our applications need convenient, low-latency access to important reference datasets. For example, our XfinityTV websites and apps need to use entertainment-related data to serve almost every API or web request to our datacenters: information like what year Casablanca was released, or how many episodes were in Season 7 of Seinfeld, or when the next episode of the Voice will be airing (and on which channel!). We traditionally managed this information with a combination of relational databases and RESTful web services but yearned for something simpler than the ORM, HTTP client, and cache management code our developers dealt with on a daily basis. As main memory sizes on commodity servers continued to grow, however, we asked ourselves: How can we keep this reference data entirely in RAM, while ensuring it gets updated as needed and is easily accessible to application developers? The Sirius distributed system library is our answer to that question, and we're happy to announce that we've made it available as an open source project. Sirius is written in Scala and uses the Akka actor system under the covers, but is easily usable by any JVM-based language.
Also includes a Paxos implementation with "fast follower" read-only slave replication. ASL2-licensed open source. The only thing I can spot to be worried about is speed of startup; they note that apps need to replay a log at startup to rebuild state, which can be slow if unoptimized in my experience. Update: in a twitter conversation at https://twitter.com/jon_moore/status/459363751893139456 , Jon Moore indicated they haven't had problems with this even with 'datasets consuming 10-20GB of heap', and have 'benchmarked a 5-node Sirius ingest cluster up to 1k updates/sec write throughput.' That's pretty solid!open-source comcast paxos replication read-only datastores storage memory memcached redis sirius scala akka jvm libraries
AWS Elastic Beanstalk for Docker
This is pretty amazing. nice work, Beanstalk team. not sure how well it integrates with the rest of AWS though
TDD is dead. Long live testing
Oh god. I agree with DHH. shoot me now.
Test-first units leads to an overly complex web of intermediary objects and indirection in order to avoid doing anything that's "slow". Like hitting the database. Or file IO. Or going through the browser to test the whole system. It's given birth to some truly horrendous monstrosities of architecture. A dense jungle of service objects, command patterns, and worse. I rarely unit test in the traditional sense of the word, where all dependencies are mocked out, and thousands of tests can close in seconds. It just hasn't been a useful way of dealing with the testing of Rails applications. I test active record models directly, letting them hit the database, and through the use of fixtures. Then layered on top is currently a set of controller tests, but I'd much rather replace those with even higher level system tests through Capybara or similar. I think that's the direction we're heading. Less emphasis on unit tests, because we're no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests.
tdd rails testing unit-tests system-tests integration-testing ruby dhh mocks
All at sea: global shipping fleet exposed to hacking threat | Reuters
Hackers recently shut down a floating oil rig by tilting it, while another rig was so riddled with computer malware that it took 19 days to make it seaworthy again; Somali pirates help choose their targets by viewing navigational data online, prompting ships to either turn off their navigational devices, or fake the data so it looks like they're somewhere else; and hackers infiltrated computers connected to the Belgian port of Antwerp, located specific containers, made off with their smuggled drugs and deleted the records.
(via Mikko Hypponen)via:mikko security hacking oilrigs shipping ships maritime antwerp piracy malware
Search Results - (Author:Thomas H Mason)
Photographs taken by my great-grandfather, Thomas H. Mason, in the National Library of Ireland's newly-digitized online collection
family thomas-h-mason history ireland photography archive nli
Syria's lethal Facebook checkpoints
An anonymous tip from a highly reliable source: "There are checkpoints in Syria where your Facebook is checked for affiliation with the rebellious groups or individuals aligned with the rebellion. People are then disappeared or killed if they are found to be connected. Drivers are literally forced to load their Facebook/Twitter accounts and then they are riffled through. It's happening daily, and has been for a year at least."
boing-boing war facebook social-media twitter internet checkpoints syria
The ancient Egyptian word for 'cat' was pronounced 'miaow'
and many other cool-but-true factoids
fun quora interesting via:dorothy factoids true urban-legends facts
-
kellabyte's hack in progress -- 'an asynchronous HTTP server framework written in C. The goal of Haywire is to learn how to create a server with a minimal feature set that can handle a high rate of requests and connections with as low of latency and resource usage as possible. Haywire uses the event loop based libuv platform layer that node.js is built on top of (also written in C). libuv abstracts IOCP on Windows and epoll/kqueue/event ports/etc. on Unix systems to provide efficient asynchronous I/O on all supported platforms.' Outperforms libevent handily, it seems. Apache-licensed.
server http asynchronous libuv haywire kellabyte c events open-source asl2
spoofing the samsung smart tv internet check
If this kind of bullshit -- a HTTP GET of an XML file from www.samsung.com -- is how the Samsung Smart TV firmware decides if the internet is working or not, I dread to think how crappy the rest of the code is. (At least in Netnote we performed a bunch of bigco-domain DNS lookups before giving up...)
smart-tv samsung fail xml http internet embedded-software firmware crap-code
ImperialViolet - No, don't enable revocation checking
...because it doesn't stop attacks. Turning it on does nothing but slow things down. You can tell when something is security theater because you need some absurdly specific situation in order for it to be useful.
cryptography crypto heartbleed ssl security tls https internet revocation crls
-
*Really* intriguing slide deck on how Asia and Africa have invented new ways of operating a business via the internet, and are turning globalisation upside down (via Yoz)
via:yoz africa asia globalisation internet web mobile payment business ecommerce global
Using AWS in the context of Australian Privacy Considerations
interesting new white paper from Amazon regarding recent strengthening of the Aussie privacy laws, particularly w.r.t. geographic location of data and access by overseas law enforcement agencies...
amazon aws security law privacy data-protection ec2 s3 nsa gchq five-eyes
For world's biggest troll, first patent case ends up in tatters
Love it. Intellectual Ventures suffers a major bloody nose in IV/Capital One patent-trolling litigation
trolls patent-trolls patents swpats capital-one intellectual-ventures
Notes On Concurrent Ring Buffer Queue Mechanics
great notes from Nitsan Wakart, who's been hacking on ringbuffers a lot in JAQ
jaq nitsanw atomic concurrency data-structures ring-buffers queueing queues algorithms
Uplink Latency of WiFi and 4G Networks
It's high. Wifi in particular shows high variability and long latency tails
-
vim-flake8 is a Vim plugin that runs the currently open file through Flake8, a static syntax and style checker for Python source code. It supersedes both vim-pyflakes and vim-pep8. Flake8 is a wrapper around PyFlakes (static syntax checker), PEP8 (style checker) and Ned's MacCabe script (complexity checker).
Recommended by several pythonistas of my acquaintance!vim python syntax error-checking errors flake8 editors ides coding
-
Nice-looking new tool from Hashicorp; service discovery and configuration service, built on Raft for leader election, Serf for gossip-based messaging, and Go. Some features: * Gossip is performed over both TCP and UDP; * gossip messages are encrypted symmetrically and therefore secure from eavesdropping, tampering, spoofing and packet corruption (like the incident which brought down S3 for days: http://status.aws.amazon.com/s3-20080720.html ); * exposes both a HTTP interface and (even better) DNS; * includes explicit support for long-distance WAN operation as well as on LANs. It all looks very practical and usable. MPL-licensed. The only potential risk I can see is that expecting to receive config updates from a blocking poll of the HTTP interface needs some good "best practice" docs, to ensure that people don't mishandle the scenario where there is a network partition between your calling code and the Consul server/agent. Without any heartbeating protocol behind the scenes, HTTP is vulnerable to "hung connections" which would result in a config change being silently missed by the client until the connection eventually is timed out, either by the calling code or the client-side kernel. This could potentially take minutes to occur, which in some usage scenarios could be a big, unforeseen problem.
configuration service-discovery distcomp raft consensus-algorithms go mpl open-source dns http gossip-protocol hashicorp
Druid | How We Scaled HyperLogLog: Three Real-World Optimizations
3 optimizations Druid.io have made to the HLL algorithm to scale it up for production use in Metamarkets: compacting registers (fixes a bug with unions of multiple HLLs); a sparse storage format (to optimize space); faster lookups using a lookup table.
druid.io metamarkets scaling hyperloglog hll algorithms performance optimization counting estimation
HyperLogLog - Intersection Arithmetic
'In general HLL intersection in StreamLib works. |A INTERSECT B| = |A| + |B| - |A UNION B|. Timon's article on intersection is important to read though. The usefulness of HLL intersection depends on the features of the HLLs you are intersecting.'
hyperloglog hll hyperloglogplus streamlib intersections sets estimation algorithms
Structural Integrity | 99% Invisible
'The student (who has since been lost to history) was studying Citicorp Center as part of his thesis and had found that the building was particularly vulnerable to quartering winds (winds that strike the building at its corners). Normally, buildings are strongest at their corners, and it's the perpendicular winds (winds that strike the building at its face) that cause the greatest strain. But this was not a normal building. LeMessurier had accounted for the perpendicular winds, but not the quartering winds. He checked the math, and found that the student was right. He compared what velocity winds the building could withstand with weather data, and found that a storm strong enough to topple Citicorp Center hits New York City every 55 years. But that's only if the tuned mass damper, which keeps the building stable, is running. LeMessurier realized that a major storm could cause a blackout and render the tuned mass damper inoperable. Without the tuned mass damper, LeMessurier calculated that a storm powerful enough to take out the building his New York every sixteen years.'
william-lemessurier architecture danger risk buildings nyc citicorp-center wind mass-dampers physics
Linode announces new instance specs
'TL;DR: SSDs + Insane network + Faster processors + Double the RAM + Hourly Billing'
-
Fcron is a scheduler. It aims at replacing Vixie Cron, so it implements most of its functionalities. But contrary to Vixie Cron, fcron does not need your system to be up 7 days a week, 24 hours a day : it also works well with systems which are running only occasionnally (contrary to anacrontab). In other words, fcron does both the job of Vixie Cron and anacron, but does even more and better :)) ...
Thanks Craig!via:chughes cron fcron unix linux ops scheduler automation scripts
-
They've done the classic website-redesign screwup -- omitted redirects from the old URLs.
Sam Silverwood-Cope, director of Intelligent Positioning, said: "They've ignored the legacy of the old Ryanair.com. It's quite startling. They are doing it just before their busiest time of the year." A change in [URLs] without proper redirects means many results found by Google now simply return error pages, he added. "Unless redirects get put in pretty soon, the position is going to get worse and worse."
ryanair inept fail funny via:christinebohan web google search redirects
-
Scarfolk is a town in North West England that did not progress beyond 1979. Instead, the entire decade of the 1970s loops ad infinitum. Here in Scarfolk, pagan rituals blend seamlessly with science; hauntology is a compulsory subject at school, and everyone must be in bed by 8pm because they are perpetually running a slight fever. "Visit Scarfolk today. Our number one priority is keeping rabies at bay." For more information please reread.
scarfolk 1970s england history funny humour public-information pagan morbid
-
OpenBSD are going wild ripping out "arcane VMS hacks" in an attempt to render OpenSSL's source code comprehensible, and finding amazing horrors like this: 'Well, even if time() isn't random, your RSA private key is probably pretty random. Do not feed RSA private key information to the random subsystem as entropy. It might be fed to a pluggable random subsystem…. What were they thinking?!'
random security openssl openbsd coding horror rsa private-keys entropy
-
This is something Jenkins have come up to randomize and distribute load, in order to avoid the "thundering-herd" bug. Good call
jenkins randomization load-balancing load thundering-herd ops capacity sleep
Shared Space and other bad junction designs lead to crashes and injuries
Just because something is "Dutch", that doesn't mean it's good. The Netherlands has many excellent examples, but you have to be very selective about what serves as a model. Cyclists fare best where their interactions with motor vehicles are limited and controlled. They fare best where infrastructure ensures that minor mistakes do not result in injuries. Anywhere that we rely upon everyone behaving perfectly but where we do not protect the most vulnerable, there will be injuries. Good design takes human nature into account and removes the causes of danger from those who are most vulnerable.
via Tony Finchcycling design junctions shared-space dutch holland roads safety crashes
-
A sane Google Protocol Buffers library for Ruby. It's all about being Buf; ProtoBuf.
protobuf google protocol-buffers ruby coding libraries gems open-source
-
When I said that we expected better of OpenSSL, it's not merely that there's some sense that security-driven code should be of higher quality. (OpenSSL is legendary for being considered a mess, internally.) It's that the number of systems that depend on it, and then expose that dependency to the outside world, are considerable. This is security's largest contributed dependency, but it's not necessarily the software ecosystem's largest dependency. Many, maybe even more systems depend on web servers like Apache, nginx, and IIS. We fear vulnerabilities significantly more in libz than libbz2 than libxz, because more servers will decompress untrusted gzip over bzip2 over xz. Vulnerabilities are not always in obvious places – people underestimate just how exposed things like libxml and libcurl and libjpeg are. And as HD Moore showed me some time ago, the embedded space is its own universe of pain, with 90's bugs covering entire countries. If we accept that a software dependency becomes Critical Infrastructure at some level of economic dependency, the game becomes identifying those dependencies, and delivering direct technical and even financial support. What are the one million most important lines of code that are reachable by attackers, and least covered by defenders? (The browsers, for example, are very reachable by attackers but actually defended pretty zealously – FFMPEG public is not FFMPEG in Chrome.) Note that not all code, even in the same project, is equally exposed. It's tempting to say it's a needle in a haystack. But I promise you this: Anybody patches Linux/net/ipv4/tcp_input.c (which handles inbound network for Linux), a hundred alerts are fired and many of them are not to individuals anyone would call friendly. One guy, one night, patched OpenSSL. Not enough defenders noticed, and it took Neel Mehta to do something.
development openssl heartbleed ssl security dan-kaminsky infrastructure libraries open-source dependencies
-
'a command line tool for Amazon's Simple Storage Service (S3). Written in Python, easy_install the package to install as an egg. Supports multithreaded operations for large volumes. Put, get, or delete many items concurrently, using a fixed-size pool of threads. Built on workerpool for multithreading and boto for access to the Amazon S3 API. Unix-friendly input and output. Pipe things in, out, and all around.' MIT-licensed open source. (via Paul Dolan)
-
The intuition behind Hydra is something like this, "I have a lot of data, and there are a lot of things I could try to learn about it -- so many that I'm not even sure what I want to know." It's about the curse of dimensionality -- more dimensions means exponentially more cost for exhaustive analysis. Hydra tries to make it easy to reduce the number of dimensions, or the cost of watching them (via probabilistic data structures), to just the right point where everything runs quickly but can still answer almost any question you think you might care about.
Code: https://github.com/addthis/hydra Getting Started blog post: https://www.addthis.com/blog/2014/02/18/getting-started-with-hydra/hyrda hadoop data-processing big-data trees clusters analysis
Stalled SCP and Hanging TCP Connections
a Cisco fail.
It looks like there's a firewall in the middle that's doing additional TCP sequence randomisation which was a good thing, but has been fixed in all current operating systems. Unfortunately, it seems that firewall doesn't understand TCP SACK, which when coupled with a small amount of packet loss and a stateful host firewall that blocks invalid packets results in TCP connections that stall randomly. A little digging revealed that firewall to be the Cisco Firewall Services Module on our Canterbury network border.
(via Tony Finch)via:fanf cisco networking firewalls scp tcp hangs sack tcpdump
Akamai's "Secure Heap" patch wasn't good enough
'Having the private keys inaccessible is a good defense in depth move. For this patch to work you have to make sure all sensitive values are stored in the secure area, not just check that the area looks inaccessible. You can't do that by keeping the private key in the same process. A review by a security engineer would have prevented a false sense of security. A version where the private key and the calculations are in a separate process would be more secure. If you decide to write that version, I'll gladly see if I can break that too.' Akamai's response: https://blogs.akamai.com/2014/04/heartbleed-update-v3.html -- to their credit, they recognise that they need to take further action. (via Tony Finch)
via:fanf cryptography openssl heartbleed akamai security ssl tls
-
Colm MacCarthaigh writes about a simple sharding/load-balancing algorithm which uses randomized instance selection and optional additional compartmentalization. See also: continuous hashing, and http://aphyr.com/posts/278-timelike-2-everything-fails-all-the-time
hashing load-balancing sharding partitions dist-sys distcomp architecture coding
Open Crypto Audit Project: TrueCrypt
phase I, a source code audit by iSEC Partners, is now complete. Bruce Schneier says: "I'm still using it".
encryption security crypto truecrypt audits source-code isec matthew-green
-
In the PNAS paper, Brad Bushman and colleagues looked at 107 couples over 21 days and found that people experiencing uncharacteristically low blood sugar were more likely to display anger toward their spouse. (The researchers measured this by having subjects stick needles into voodoo dolls representing their significant others.)
hangry hunger food eating science health blood-sugar voodoo-dolls glucose
insane ESB health and safety policy
Where it is not possible to avoid reversing, it is ESB policy that staff driving on behalf of the company or anybody on company premises should reverse into car spaces/bays, allowing them to drive out subsequently.
BUT WHYYYYYYYYYYesb health-n-safety policies crazy funny driving reversing lol safety
Cloudflare demonstrate Heartbleed key extraction
from nginx. 'Based on the findings, we recommend everyone reissue + revoke their private keys.'
When two-factor authentication is not enough
Fastmail.FM nearly had their domain stolen through an attack exploiting missing 2FA authentication in Gandi.
An important lesson learned is that just because a provider has a checkbox labelled "2 factor authentication" in their feature list, the two factors may not be protecting everything – and they may not even realise that fact themselves. Security risks always come on the unexpected paths – the "off label" uses that you didn't think about, and the subtle interaction of multiple features which are useful and correct in isolation.
gandi 2fa fastmail authentication security mfa two-factor-authentication mail
Of Money, Responsibility, and Pride
Steve Marquess of the OpenSSL Foundation on their funding, and lack thereof:
I stand in awe of their talent and dedication, that of Stephen Henson in particular. It takes nerves of steel to work for many years on hundreds of thousands of lines of very complex code, with every line of code you touch visible to the world, knowing that code is used by banks, firewalls, weapons systems, web sites, smart phones, industry, government, everywhere. Knowing that you'll be ignored and unappreciated until something goes wrong. The combination of the personality to handle that kind of pressure with the relevant technical skills and experience to effectively work on such software is a rare commodity, and those who have it are likely to already be a valued, well-rewarded, and jealously guarded resource of some company or worthy cause. For those reasons OpenSSL will always be undermanned, but the present situation can and should be improved. There should be at least a half dozen full time OpenSSL team members, not just one, able to concentrate on the care and feeding of OpenSSL without having to hustle commercial work. If you're a corporate or government decision maker in a position to do something about it, give it some thought. Please. I'm getting old and weary and I'd like to retire someday.
funding open-source openssl heartbleed internet security money
-
a system for building agents that perform automated tasks for you online. They can read the web, watch for events, and take actions on your behalf. Huginn's Agents create and consume events, propagating them along a directed event flow graph. Think of it as Yahoo! Pipes plus IFTTT on your own server. You always know who has your data. You do.
MIT-licensed open source, built on Rails.
Why no SSL ? — Varnish version 4.0.0 documentation
Poul-Henning Kemp details why Varnish doesn't do SSL -- basically due to the quality and complexity of open-source SSL implementations:
There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump.
Now looking pretty smart, post-Heartbleed.ssl tls varnish open-source poul-henning-kemp https http proxies security coding
Basho LevelDB supports tiered storage
Tiered storage is turning out to be a pretty practical trick to take advantage of SSDs:
The justification for two types/speeds of storage arrays is simple. leveldb is extremely write intensive in its lower levels. The write intensity drops off as the level number increases. Similarly, current and frequently updated data tends to be in lower levels while archival data tends to be in higher levels. These leveldb characteristics create a desire to have faster, more expensive storage arrays for the high intensity lower levels. This branch allows the high intensity lower levels to be on expensive storage arrays while slower, less expensive storage arrays to hold the higher level data to reduce costs.
caching tiered-storage storage ssds ebs leveldb basho patches riak iops
Forbes on the skeleton crew nature of OpenSSL
This is a great point:
Obviously, those tending to the security protocols that support the rest of the Web need better infrastructure and more funding. "Large portions of the software infrastructure of the Internet are built and maintained by volunteers, who get little reward when their code works well but are blamed, and sometimes savagely derided, when it fails," writes Foster in the New Yorker. [...] "money and support still tend to flow to the newest and sexiest projects, while boring but essential elements like OpenSSL limp along as volunteer efforts," he writes. "It's easy to take open-source software for granted, and to forget that the Internet we use every day depends in part on the freely donated work of thousands of programmers." We need to find ways to pay for work that is currently essentially donated freely. One promising project is Bithub, from Whisper Systems, where people who make valuable contributions to open source projects are rewarded (with Bitcoin of course). But the pool of Bitcoin is still donation based. The Internet has helped create a culture of free, but what we may need to recognize is that we get what we pay for. Well-funded companies pulling critical code from open source projects for their sites should have formal fee arrangements, rather than the volunteer group simply hoping these users will pony up some Benjamins for "prominent logo placement" on a website most people had never heard of before Heartbleed.
The Heartbleed Hit List: The Passwords You Need to Change Right Now
Good list
-
An excellent list of aspects of the Heartbleed OpenSSL bug which need to be thought about/talked about/considered
Does the heartbleed vulnerability affect clients as severely?
'Yes, clients are vulnerable to attack. A malicious server can use the Heartbleed vulnerability to compromise an affected client.' Ouch.
R3: Announcing the next generation of Amazon EC2 Memory-optimized instances
pricing announced, and available in most regions
Scalable Atomic Visibility with RAMP Transactions
Great new distcomp protocol work from Peter Bailis et al:
We've developed three new algorithms—called Read Atomic Multi-Partition (RAMP) Transactions—for ensuring atomic visibility in partitioned (sharded) databases: either all of a transaction's updates are observed, or none are. [...] How they work: RAMP transactions allow readers and writers to proceed concurrently. Operations race, but readers autonomously detect the races and repair any non-atomic reads. The write protocol ensures readers never stall waiting for writes to arrive. Why they scale: Clients can't cause other clients to stall (via synchronization independence) and clients only have to contact the servers responsible for items in their transactions (via partition independence). As a consequence, there's no mutual exclusion or synchronous coordination across servers. The end result: RAMP transactions outperform existing approaches across a variety of workloads, and, for a workload of 95% reads, RAMP transactions scale to over 7 million ops/second on 100 servers at less than 5% overhead.
scale synchronization databases distcomp distributed ramp transactions scalability peter-bailis protocols sharding concurrency atomic partitions
-
wow, WFMU's "Beware of the Blog" did a really good job of assembling these classic 1960s Cambodian psych mp3s, featuring Sinn Sisamouth and Ros Sereysothea among others. I still love the start of Ma Pi Naok....
mp3 download music cambodia psychedelia wfmu 1960s sinn-sisamouth ros-sereysothea
MICA: A Holistic Approach To Fast In-Memory Key-Value Storage [paper]
Very interesting new approach to building a scalable in-memory K/V store. As Rajiv Kurian notes on the mechanical-sympathy list: 'The basic idea is that each core is responsible for a portion of the key-space and requests are forwarded to the right core, avoiding multiple-writer scenarios. This is opposed to designs like memcache which uses locks and shared memory. Some of the things I found interesting: The single writer design is taken to an extreme. Clients assist the partitioning of requests, by calculating hashes before submitting GET requests. It uses Intel DPDK instead of sockets to forward packets to the right core, without processing the packet on any core. Each core is paired with a dedicated RX/TX queue. The design for a lossy cache is simple but interesting. It does things like replacing a hash slot (instead of chaining) etc. to take advantage of the lossy nature of caches. There is a lossless design too. A bunch of tricks to optimize for memory performance. This includes pre-allocation, design of the hash indexes, prefetching tricks etc. There are some other concurrency tricks that were interesting. Handling dangling pointers was one of them.' Source code here: https://github.com/efficient/mica
mica in-memory memory ram key-value-stores storage smp dpdk multicore memcached concurrency
Google's Open Bidder stack moving from Jetty to Netty
Open Bidder traditionally used Jetty as an embedded webserver, for the critical tasks of accepting connections, processing HTTP requests, managing service threads, etc. Jetty is a robust, but traditional stack that carries the weight and tradeoffs of Servlet's 15 years old design. For a maximum performance RTB agent that must combine very large request concurrency with very low latencies, and often benefit also from low-level control over the transport, memory management and other issue, a different webserver stack was required. Open Bidder now supports Netty, an asynchronous, event-driven, high-performance webserver stack. For existing code, the most important impact is that Netty is not compatible with the Servlet API. Its own internal APIs are often too low-level, not to mention proprietary to Netty; so Open Bidder v0.5 introduces some new, stack-neutral APIs for things like HTTP requests and responses, cookies, request handlers, and even simple HTML templating based on Mustache. These APIs will work with both Netty and Jetty. This means you don't need to change any code to switch between Jetty and Netty; on the other hand, it also means that existing code written for Open Bidder 0.4 may need some changes even if you plan to keep using Jetty. [....] Netty's superior efficiency is very significant; it supports 50% more traffic in the same hardware, and it maintains a perfect latency distribution even at the peak of its supported load.
This doc is noteworthy on a couple of grounds: 1. the use of Netty in a public API/library, and the additional layer in place to add a friendlier API on top of that. I hope they might consider releasing that part as OSS at some point. 2. I also find it interesting that their API uses protobufs to marshal the message, and they plan in a future release to serialize those to JSON documents -- that makes a lot of sense.apis google protobufs json documents interoperability netty jetty servlets performance java
The University Times: TCD Provost Under Pressure To "Re-think" Identity Initiative
Students, staff and alumni put pressure on Provost to reconsider changes to Trinity College Dublin's name and coat of arms.
alumni scholars from 2004 and 1994 who had been invited back for the dinner shouted 'Dublin' after the Provost welcomed them back to "Trinity College".
Daring Fireball: Rethinking What We Mean by 'Mobile Web'
We shouldn't think of "the web" as only what renders in web browsers. We should think of the web as anything transmitted using HTTP and HTTPS. Apps and websites are peers, not competitors. They're all just clients to the same services.
+1. Finally, a Daring Fireball post I agree with! ;)daring-fireball apps web http https mobile apple android browsers
The little ssh that (sometimes) couldn't - Mina Naguib
A good demonstration of what it looks like when network-level packet corruption occurs on a TCP connection
ssh sysadmin networking tcp bugs bit-flips cosmic-rays corruption packet
DRI wins their case at the ECJ!
Great stuff!
The Court has found that data retention "entails a wide-ranging and particularly serious interference with the fundamental rights to respect for private life and to the protection of personal data" and that it "entails an interference with the fundamental rights of practically the entire European population". TJ McIntyre, Chairman of Digital Rights Ireland, said that "This is the first assessment of mass surveillance by a supreme court since the Snowden revelations. The ECJ's judgement finds that untargeted monitoring of the entire population is unacceptable in a democratic society." [...] Though the Directive has now been struck down, the issue will remain live in all the countries who have passed domestic law to implement the data retention mass surveillance regime. Digital Rights Ireland's challenge to the Irish data retention system will return to the High Court in Dublin for the next phase of litigation.
dri digital-rights ireland eu ecj surveillance snooping law data-retention
-
a cron job monitoring tool that keeps an eye on your periodic processes and notifies you when something doesn't happen. Daily backups, monthly emails, or cron jobs you need to monitor? Dead Man's Snitch has you covered. Know immediately when one of these processes doesn't work.
via Marc. VERY high resolution scans of original Apollo 11 and Apollo 14 charts
the Apollo 11 ALO and LM Descent Monitoring charts are tidied up and downloadable
Garbage Collection Optimization for High-Throughput and Low-Latency Java Applications
LinkedIn talk about the GC opts they used to optimize the Feed. good detail
Great comment from the "banking uses eventual consistency" thread...
'In the grand scheme of things, insurance trumps consistency.' Love it.
banking banks insurance eventual-consistency databases consistency transactions nosql
S3 as a single-web-page application engine
neat hack. Pity it returns a 403 error code due to the misuse of the ErrorDocument feature though
-
Tracking Irish News Stories Over Time; Irish NewsDiffs archives changes in articles after publication. Currently, we track rte.ie and irishtimes.com.
rte irish-times diffing diffs changes tracking newspapers news ireland history
Here's Why You're Not Hiring the Best and the Brightest
Jeff Atwood's persuasive argument that remote working needs to be the norm in tech work:
There's an elephant in the room in the form of an implied clause: Always hire the best people… who are willing to live in San Francisco. Substitute Mountain View, New York, Boston, Chicago, or any other city. The problem is the same. We pay lip service to the idea of hiring the best people in the world — but in reality, we're only hiring the best people who happen to be close by.
recruiting remote hiring business coding work remote-work telecommuting jobs silicon-valley jeff-atwood
The First Few Milliseconds of an HTTPS Connection
in excruciating detail
-
For EVERY youtube video, I always open the video and then immediately punch the slider bar to about 30 percent. For example, in this video, it should have just started at :40. Everything before :40 was a waste. This holds true for nearly every video in the universe.
Via Joe Drumgoole -- it's true!via:joedrumgoole videos youtube skip wadsworth-constant fast-forward funny
-
Advice from a developer who helped rebuild Walmart.ca with Scala and Play
This is really good advice.
-
open source, system-level exploration: capture system state and activity from a running Linux instance, then save, filter and analyze. Think of it as strace + tcpdump + lsof + awesome sauce. With a little Lua cherry on top.
This sounds excellent. Linux-based, GPLv2.debugging tools linux ops tracing strace open-source sysdig cli tcpdump lsof
Manhattan, our real-time, multi-tenant distributed database for Twitter scale | Twitter Blogs
Impressive, but a fierce whiff of "NIH" off of this
manhattan consistency database twitter eventual-consistency nosql voldemort cassandra riak time-series
European Parliament passes strong net neutrality law, along with major roaming reforms
European fans of the open internet can breathe a sigh of relief: the European parliament has passed a major package of telecoms law reform, complete with amendments that properly define and protect net neutrality. The amendments were introduced by the Socialist, Liberal, Green and Left blocs in the European Parliament after the final committee to tweak the package – the industry committee – left in a bunch of loopholes that would have allowed telcos to start classifying web services of their choice as "specialized services" that they can treat differently. [...] Now the whole package gets passed through to the next Parliament (elections are coming up in May), then the representatives of European countries for final approval.
netneutrality eu ep europe neelie-kroes freedom isps telecom
-
Finally got around to migrating this old CPAN module to github
-
an easily embeddable, decentralized, k-ordered unique ID generator. It can use the same encoded ID format as Twitter's Snowflake or Boundary's Flake implementations as well as any other customized encoding without too much effort. The fauxflake-core module has no external dependencies and is meant to be about as light as possible while still delivering useful functionality. Essentially, if you want to be able to generate a unique identifier across your infrastructure with reasonable assurances about collisions, then you might find this useful.
From the same guy as the excellent Guava Retrier library; java, ASL2-licensed open source.open-source java asl2 fauxflake tools libraries unique-ids ids unique snowflake distsys
-
Another cool library from Roy Holder: 'an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.' Similar to his Guava-Retrier java lib, but using a decorator.
retrying python libraries tools backoff retry error-handling
Redis adds support for HyperLogLog
good comment thread on HN, discussing hlld and bloomd as well
hll bloom-filters hyperloglog redis data-structures estimation cardinality probabilistic probability hashing random
LastPass Sentry Warns You When Your Online Accounts Have Been Breached
This is a brilliant feature. It just sent a warning to a friend about an old account he was no longer using
7 Hidden Dublin Secrets You Probably Never Knew About
from Pól Ó Conghaile's "Secret Dublin". great stuff
-
must give this a try. performance will be the issue I suspect
How Gmail Happened: The Inside Story of Its Launch 10 Years Ago Today
the inside story of the great work done by Paul Buchheit, Kevin Fox, and Sanjeev Singh to reinvent email in 2004
history gmail email smtp mua paul-buchheit kevin-fox launches google web
-
this HN thread on the age-old UDP vs TCP question is way better than the original post -- lots of salient comments
udp tcp games protocols networking latency internet gaming hackernews
-
'a.k.a. Faster == Better'. Slides from a talk given at Facebook on 19th March 2014 by Norman Maurer
netty java performance optimization facebook slides presentations
Daylight saving time linked to heart attacks, study finds
Switching over to daylight saving time, and losing one hour of sleep, raised the risk of having a heart attack the following Monday by 25 per cent, compared to other Mondays during the year, according to a new US study released today. [...] The study found that heart attack risk fell 21 per cent later in the year, on the Tuesday after the clock was returned to standard time, and people got an extra hour's sleep.
One clear answer: we need 25-hour days. More details: http://www.sciencedaily.com/releases/2014/03/140329175108.htm --Researchers used Michigan's BMC2 database, which collects data from all non-federal hospitals across the state, to identify admissions for heart attacks requiring percutaneous coronary intervention from Jan. 1, 2010 through Sept. 15, 2013. A total of 42,060 hospital admissions occurring over 1,354 days were included in the analysis. Total daily admissions were adjusted for seasonal and weekday variation, as the rate of heart attacks peaks in the winter and is lowest in the summer and is also greater on Mondays and lower over the weekend. The hospitals included in this study admit an average of 32 patients having a heart attack on any given Monday. But on the Monday immediately after springing ahead there were on average an additional eight heart attacks. There was no difference in the total weekly number of percutaneous coronary interventions performed for either the fall or spring time changes compared to the weeks before and after the time change.
daylight dst daylight-savings time dates calendar science health heart-attacks michigan hospitals statistics
Steve Jobs on the disease of believing that 90% of the work is having a great idea
Good quote
steve-jobs quotes entrepreneurs startups making products building ideas concepts apple history
DNS results now being manipulated in Turkey
Deep-packet inspection and rewriting on DNS packets for Google and OpenDNS servers. VPNs and DNSSEC up next!
turkey twitter dpi dns opendns google networking filtering surveillance proxying packets udp
Phusion Passenger now supports the new Ruby 2.1 Out-Of-Band GC
a reasonable workaround for Ruby's GC problems in web apps
ruby gc ops performance phusion passenger rails unicorn out-of-band web-services
Duplicity + S3: easy, cheap, encrypted, automated full-disk backups for your servers
actually sounds quite nice
-
This is a couple of years old, but I like this:
Turbo Boyer-Moore is disappointing, its name doesn't do it justice. In academia constant overhead doesn't matter, but here we see that it matters a lot in practice. Turbo Boyer-Moore's inner loop is so complex that we think we're better off using the original Boyer-Moore.
A good demo of how large values of O(n) can be slower than small values of O(mn).algorithms search strings coding big-o string-search searching
-
in a field as critical and competitive as smartphones, Google's R&D strategy was being dictated, not by the company's board, or by its shareholders, but by a desire not to anger the CEO of a rival company.
This is utterly bananas and anti-competitive. (via Des Traynor)via:destraynor wage-fixing apple google tech paris r-and-d steve-jobs jean-marie-hullot france competition poaching assholes
[#1259] Add optimized queue for SCMP pattern and use it in NIO and nativ... · 6efac61 · netty/netty
Interesting -- Netty has imported an optimized ASL2-licensed MPSC queue implementation from Akka (presumably for performance raisins)
performance optimization open-source mpsc queues data-structures netty akka java
Ruby Garbage Collection: Still Not Ready for Production
disastrous GC bugs in Ruby, requiring horrible kludgy workarounds
-
[via Boing Boing:] A new, exhaustive report from Human Rights Watch details the way the young state of modern Ethiopia has become a kind of pilot program for the abuse of "off-the-shelf" surveillance, availing itself of commercial products from the US, the UK, France, Italy and China in order to establish an abusive surveillance regime that violates human rights and suppresses legitimate political opposition under the guise of a anti-terrorism law that's so broadly interpreted as to be meaningless. The 137 page report [from Human Rights Watch] details the technologies the Ethiopian government has acquired from several countries and uses to facilitate surveillance of perceived political opponents inside the country and among the diaspora. The government's surveillance practices violate the rights to freedom of expression, association, and access to information. The government's monopoly over all mobile and Internet services through its sole, state-owned telecom operator, Ethio Telecom, facilitates abuse of surveillance powers.
human-rights surveillance ethiopia spying off-the-shelf spyware big-brother hrw human-rights-watch
Chinese cops cuff 1,500 in fake base station spam raid
The street finds its own uses for things, in this case Stinger/IMSI-catcher-type fake mobile-phone base stations:
Fake base stations are becoming a particularly popular modus operandi. Often concealed in a van or car, they are driven through city streets to spread their messages. The professional spammer in question charged 1,000 yuan (£100) to spam thousands of users in a radius of a few hundred metres. The pseudo-base station used could send out around 6,000 messages in just half an hour, the report said. Often such spammers are hired by local businessmen to promote their wares.
(via Bernard Tyers)stingers imsi-catcher mobile-phones mobile cellphones china spam via:bernard-tyers
-
The most grave issue is that each recording likely amounted to a serious criminal offence. Under Irish law, the recording of a telephone conversation on a public network without the consent of at least one party to the call amounts to an "interception", a criminal offence carrying a possible term of imprisonment of up to five years. [...] Consequently, unless gardai were notified that their calls might be recorded then a large number of criminal offences are likely to have been committed by and within the Garda Siochana itself.
gubu surveillance gardai ags tjmcintyre bugging tapping phones ireland politics
-
A cool-looking new debugging tool for C/C++ from Mozilla.
Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.
Low runtime overhead; easy deployability; targeted at 32-bit (?!) Linux; OSS. (via Bryan O'Sullivan)via:bos mozilla debugging coding firefox rr record replay gdb c++ linux
-
AIB now have a dedicated customer-support forum on Boards.ie. That is a *great* idea
Microservices and nanoservices
A great reaction to Martin Fowler's "microservices" coinage, from Arnon Rotem-Gal-Oz: 'I guess it is easier to use a new name (Microservices) rather than say that this is what SOA actually meant'; 'these are the very principles of SOA before vendors does pushed the [ESB] in the middle.' Others have also chosen to define microservices slightly differently, as a service written in 10-100 LOC. Arnon's reaction: "Nanoservice is an antipattern where a service is too fine-grained. A nanoservice is a service whose overhead (communications, maintenance, and so on) outweighs its utility." Having dealt with maintaining an over-fine-grained SOA stack in Amazon, I can only agree with this definition; it's easy to make things too fine-grained and create a raft of distributed-computing bugs and deployment/management complexity where there is no need to do so.
architecture antipatterns nanoservices microservices soa services design esb
-
slightly ruined by the inclusion of some "deliberately Turing-complete" systems
turing computation software via:jwz turing-complete accidents automatons
-
The next step in the Turkish twitter-block arms race.
Bridge relays (or "bridges" for short) are Tor relays that aren't listed in the main Tor directory. Since there is no complete public list of them, even if your ISP is filtering connections to all the known Tor relays, they probably won't be able to block all the bridges. If you suspect your access to the Tor network is being blocked, you may want to use the bridge feature of Tor. The addition of bridges to Tor is a step forward in the blocking resistance race. It is perfectly possible that even if your ISP filters the Internet, you do not require a bridge to use Tor. So you should try to use Tor without bridges first, since it might work.
tor privacy turkey bridging networking tor-bridges twitter filtering blocking censorship
Adrian Cockroft's Cloud Outage Reports Collection
The detailed summaries of outages from cloud vendors are comprehensive and the response to each highlights many lessons in how to build robust distributed systems. For outages that significantly affected Netflix, the Netflix techblog report gives insight into how to effectively build reliable services on top of AWS. [....] I plan to collect reports here over time, and welcome links to other write-ups of outages and how to survive them.
outages post-mortems documentation ops aws ec2 amazon google dropbox microsoft azure incident-response
-
This looks like an excellent new feature for parents:
A supervised user is a special type of Chrome user who can browse the web with guidance. Under the supervision of the manager, a supervised user can browse the web and sign in to websites. Supervised users don't need a Google Account or an email address because the manager creates a profile for the supervised user through the manager's Google Account. As a manager of a supervised user, you can see the user's browsing history, block specific sites, and approve which sites the user can see, all from the supervised users dashboard that is accessible from any browser.
users chrome supervision parental-control parents safety web browsing kids
The Stony Brook Algorithm Repository
This WWW page is intended to serve as a comprehensive collection of algorithm implementations for over seventy of the most fundamental problems in combinatorial algorithms. The problem taxonomy, implementations, and supporting material are all drawn from my [ie. Steven Skiena's] book 'The Algorithm Design Manual'. Since the practical person is more often looking for a program than an algorithm, we provide pointers to solid implementations of useful algorithms, when they are available.
The Overprotected Kid - The Atlantic
Great article.
There is a big difference between avoiding major hazards and making every decision with the primary goal of optimizing child safety (or enrichment, or happiness). We can no more create the perfect environment for our children than we can create perfect children. To believe otherwise is a delusion, and a harmful one; remind yourself of that every time the panic rises.
child-safety parenting safety kids education risk danger playgrounds the-land
Issue 122 - android-query - HTTP 204 Response results in Network Error (-101)
an empty 204 response to a HTTP PUT will trigger this. See also https://code.google.com/p/android/issues/detail?id=24672, '"java.io.IOException: unexpected end of stream" on HttpURLConnection HEAD call'.
http urlconnection httpurlconnection java android dalvik bugs 204 head get exceptions
-
'The European election will take place between 22 and 25 May 2014. Citizens, promise to vote for candidates that have signed a 10-point charter of digital rights! Show candidates that they need to earn your vote by signing our charter!'
europarl ep digital-rights rights ireland eu data-privacy data-protection privacy
-
amazing 80's-VHS-feel animated GIFs from Gustavo Torres, a video artist from Argentina
anigifs animated gif retro cyberpunk gustavo-torres kidmograph fx video-art via:mlkshk
-
'Microsoft went through a blogger's private Hotmail account in order to trace the identity of a source who allegedly leaked trade secrets.' Bear in mind that the alleged violation which MS allege allows them to read their email was a breach of the terms of service, which also include distribution of content which 'incites, advocates, or expresses pornography, obscenity, vulgarity, [or] profanity'. So no dirty jokes on Hotmail!
hotmail fail scroogled microsoft stupid tos law privacy data-protection trade-secrets ip
Theresa May warns Yahoo that its move to Dublin is a security worry
Y! is moving to Dublin to evade GCHQ spying on its users. And what is the UK response?
"There are concerns in the Home Office about how Ripa will apply to Yahoo once it has moved its headquarters to Dublin," said a Whitehall source. "The home secretary asked to see officials from Yahoo because in Dublin they don't have equivalent laws to Ripa. This could particularly affect investigations led by Scotland Yard and the national crime agency. They regard this as a very serious issue."
There's priorities for you!ripa gchq guardian uk privacy data-protection ireland dublin london spying surveillance yahoo
A Look At Airbnb's Irish Pub-Inspired Office In Dublin - DesignTAXI.com
Very nice, Airbnb!
Internet Tolls And The Case For Strong Net Neutrality
Netflix CEO Reed Hastings blogs about the need for Net Neutrality:
Interestingly, there is one special case where no-fee interconnection is embraced by the big ISPs -- when they are connecting among themselves. They argue this is because roughly the same amount of data comes and goes between their networks. But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download** -- thus filling their upstream networks and nearly doubling our total traffic -- there is an uncomfortable silence. That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs. ISPs around the world are investing in high-speed Internet and most already practice strong net neutrality. With strong net neutrality, new services requiring high-speed Internet can emerge and become popular, spurring even more demand for the lucrative high-speed packages ISPs offer. With strong net neutrality, everyone avoids the kind of brinkmanship over blackouts that plague the cable industry and harms consumers. As the Wall Street Journal chart shows, we're already getting to the brownout stage. Consumers deserve better.
consumer net-neutrality comcast netflix protectionism cartels isps us congestion capacity
Micro jitter, busy waiting and binding CPUs
pinning threads to CPUs to reduce jitter and latency. Lots of graphs and measurements from Peter Lawrey
The Day Today - Pool Supervisor - YouTube
"in 1979, no-one died. in 1980, some one died. in 1981, no-one died. in 1982, no-one died. ... I could go on"
the-day-today no-one-died safety pool supervisor tricky-word-puzzles funny humour classic video
The colossal arrogance of Newsweek's Bitcoin "scoop" | Ars Technica
Many aspects of the story already look like a caricature of journalism gone awry. The man Goodman fingered as being worth $400 million or more is just as modest as his house suggests. He's had a stroke and struggles with other health issues. Unemployed since 2001, he strives to take care of basic needs for himself and his 93-year-old mother, according to a reddit post by his brother Arthur Nakamoto (whom Goodman quoted as calling his brother an "asshole"). If Goodman has mystery evidence supporting the Dorian Nakamoto theory, it should have been revealed days ago. Otherwise, Newsweek and Goodman are delaying an inevitable comeuppance and doubling down on past mistakes. Nakamoto's multiple denials on the record have changed the dynamic of the story. Standing by the story, at this point, is an attack on him and his credibility. The Dorian Nakamoto story is a "Dewey beats Truman" moment for the Internet age, with all of the hubris and none of the humor. It shouldn't be allowed to end in the mists of "he said, she said." Whether or not a lawsuit gets filed, Nakamoto v. Newsweek faces an imminent verdict in the court of public opinion: either the man is lying or the magazine is wrong.
dorian-nakamoto newsweek journalism bitcoin privacy satoshi-nakamoto
-
While going through her papa's old belongings, a young girl discovered something incredible - a mind-bogglingly intricate maze that her father had drawn by hand 30 years ago. While working as a school janitor it had taken him 7 years to produce the piece, only for it to be forgotten about... until now.
34" x 24" print, $40 Continuous Delivery with ETL Systems [video]
Lonely Planet and Dr Foster Intelligence both make heavy use of ETL in their products, and both organisations have applied the principles of Continuous Delivery to their delivery process. Some of the Continuous Delivery norms need to be adapted in the context of ETL, and some interesting patterns emerge, such as running Continuous Integration against data, as well as code.
etl video presentations lonely-planet dr-foster-intelligence continuous-delivery deployment pipelines
-
'On March 9th a group posted a data leak, which included the trading history of all MtGox users from April 2011 to November 2013. The graphs below explore the trade behaviors of the 500 highest volume MtGox users from the leaked data set. These are the Bitcoin barons, wealthy speculators, dueling algorithms, greater fools, and many more who took bitcoin to the moon.'
What We Know 2/5/14: The Mt. Chiliad Mystery
hats off to Rockstar -- GTA V has a great mystery mural with clues dotted throughout the game, and it's as-yet unsolved
mysteries gaming via:hilary_w games gta gta-v rockstar mount-chiliad ufos
Make Your Own 3-D Printer Filament From Old Milk Jugs
Creating your own 3-D printer filament from old used milk jugs is exponentially cheaper, and uses considerably less energy, than buying new filament, according to new research from Michigan Technological University. [...] The savings are really quite impressive — 99 cents on the dollar, in addition to the reduced use of energy. Interestingly (but again not surprisingly), the amount of energy used to 'recycle' the old milk jugs yourself is considerably less than that used in recycling such jugs conventionally.
-
This is a really good post on governmental computing, open data, and so on:
The fact that I can go months hearing about "open data" without a single mention of ETL is a problem. ETL is the pipes of your house: it's how you open data.
civic open-data government etl data-pipeline tech via:timoreilly
-
as TJ McIntyre noted: '€100 fine for a repeat spammer. Data Protection Commissioner calls this "strong protection". With a straight face.' Next will doubtless fork over the 100 Euros out of the petty cash drawer, then carry on regardless. This isn't a useful fine. What a farce...
cheap farce dpc data-protection privacy anti-spam next spam convictions fines ireland
-
The mass surveillance methods employed in [the UK, USA, and India], many of them exposed by NSA whistleblower Edward Snowden, are all the more intolerable because they will be used and indeed are already being used by authoritarians countries such as Iran, China, Turkmenistan, Saudi Arabia and Bahrain to justify their own violations of freedom of information. How will so-called democratic countries will able to press for the protection of journalists if they adopt the very practices they are criticizing authoritarian regimes for?
This is utterly jaw-dropping -- throughout the world, real-time mass-monitoring infrastructure is silently being dropped into place. France and India are particularly pervasivejournalism censorship internet france india privacy data-protection surveillance spying law snowden authoritarianism
The Microservice Declaration of Independence
"Microservices" seems to be yet another term for SOA; small, decoupled, independently-deployed services, with well-defined public HTTP APIs. Pretty much all the services I've worked on over the past few years have been built in this style. Still, let's keep an eye on this concept anyway. Another definition seems to be a more FP-style one: http://www.slideshare.net/michaelneale/microservices-and-functional-programming -- where the "microservice" does one narrowly-defined thing, and that alone.
microservices soa architecture handwaving http services web deployment
No, Nate, brogrammers may not be macho, but that's not all there is to it
Great essay on sexism in tech, "brogrammer" culture, "clubhouse chemistry", outsiders, wierd nerds and exclusion:
Every group, including the excluded and disadvantaged, create cultural capital and behave in ways that simultaneously create a sense of belonging for them in their existing social circle while also potentially denying them entry into another one, often at the expense of economic capital. It's easy to see that wearing baggy, sagging pants to a job interview, or having large and visible tattoos in a corporate setting, might limit someone's access. These are some of the markers of belonging used in social groups that are often denied opportunities. By embracing these markers, members of the group create real barriers to acceptance outside their circle even as they deepen their peer relationships. The group chooses to adopt values that are rejected by the society that's rejecting them. And that's what happens to "weird nerd" men as well—they create ways of being that allow for internal bonding against a largely exclusionary backdrop.
(via Bryan O'Sullivan)nerds outsiders exclusion society nate-silver brogrammers sexism racism tech culture silicon-valley essays via:bos31337
Impact of large primitive arrays (BLOBS) on JVM Garbage Collection
some nice graphs and data on CMS performance, with/without -XX:ParGCCardsPerStrideChunk
cms java jvm performance optimization tuning off-heap-storage memory
Anatomical Collages by Travis Bedel
these are fantastic
-
a utility to perform parallel, pipelined execution of a single HTTP GET. htcat is intended for the purpose of incantations like: htcat https://host.net/file.tar.gz | tar -zx It is tuned (and only really useful) for faster interconnects: [....] 109MB/s on a gigabit network, between an AWS EC2 instance and S3. This represents 91% use of the theoretical maximum of gigabit (119.2 MiB/s).
-
Abe Stanway crunches the stats on Citibike usage in NYC, compared to the weather data from Wunderground.
data correlation statistics citibike cycling nyc data-science weather
NSA surveillance recording every single voice call in at least 1 country
Storing them in a 30-day rolling buffer, allowing retrospective targeting weeks after the call. 100% of all voice calls in that country, although it's unclear which country that is
-
a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X. S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.
-
good explanation of all the new features -- I'm really looking forward to fixing up all the crappy over-verbose interface-as-lambdas we have scattered throughout our code
java java8 lambdas fp functional-programming currying joda-time
-
a compressed full-text substring index based on the Burrows-Wheeler transform, with some similarities to the suffix array. It was created by Paolo Ferragina and Giovanni Manzini,[1] who describe it as an opportunistic data structure as it allows compression of the input text while still permitting fast substring queries. The name stands for 'Full-text index in Minute space'. It can be used to efficiently find the number of occurrences of a pattern within the compressed text, as well as locate the position of each occurrence. Both the query time and storage space requirements are sublinear with respect to the size of the input data.
kragen notes 'gene sequencing is using [them] in production'.sequencing bioinformatics algorithms bowtie fm-index indexing compression search burrows-wheeler bwt full-text-search
How to turn your smartphone photos from good to great
some good tips
How the Irish helped weave the web
Nice Irish Times article on the first 3 web servers in Ireland -- including the one I set up at Iona Technologies. 21 years ago!
history ireland tech web internet www james-casey peter-flynn irish-times iona-technologies
Health privacy: formal complaint to ICO
'Light Blue Touchpaper' notes:
Three NGOs have lodged a formal complaint to the Information Commissioner about the fact that PA Consulting uploaded over a decade of UK hospital records to a US-based cloud service. This appears to have involved serious breaches of the UK Data Protection Act 1998 and of multiple NHS regulations about the security of personal health information.
Let's see if ICO can ever do anything useful.... not holding my breathico privacy data-protection dpa nhs health data ross-anderson
Why Google Flu Trends Can't Track the Flu (Yet)
It's admittedly hard for outsiders to analyze Google Flu Trends, because the company doesn't make public the specific search terms it uses as raw data, or the particular algorithm it uses to convert the frequency of these terms into flu assessments. But the researchers did their best to infer the terms by using Google Correlate, a service that allows you to look at the rates of particular search terms over time. When the researchers did this for a variety of flu-related queries over the past few years, they found that a couple key searches (those for flu treatments, and those asking how to differentiate the flu from the cold) tracked more closely with Google Flu Trends' estimates than with actual flu rates, especially when Google overestimated the prevalence of the ailment. These particular searches, it seems, could be a huge part of the inaccuracy problem. There's another good reason to suspect this might be the case. In 2011, as part of one of its regular search algorithm tweaks, Google began recommending related search terms for many queries (including listing a search for flu treatments after someone Googled many flu-related terms) and in 2012, the company began providing potential diagnoses in response to symptoms in searches (including listing both "flu" and "cold" after a search that included the phrase "sore throat," for instance, perhaps prompting a user to search for how to distinguish between the two). These tweaks, the researchers argue, likely artificially drove up the rates of the searches they identified as responsible for Google's overestimates.
via Boing Boinggoogle flu trends feedback side-effects colds health google-flu-trends
Implementing a web server in a single printf() call
clever hack -- shellcode in a format string
Ucas sells access to student data for phone and drinks firms' marketing | Technology | The Guardian
The UK government's failure to deal with spam law in a consumer-friendly way escalates further: UCAS, the university admissions service, is operating as a mass-mailer of direct marketing on behalf of Vodafone, O2, Microsoft, Red Bull and others, without even a way to later opt out from that spam without missing important admissions-related mail as a side effect. 'Teenagers using Ucas Progress must explicitly opt in to mailings from the organisation and advertisers, though the organisation's privacy statement says: "We do encourage you to tick the box as it helps us to help you."' Their website also carries advertising, and the details of parents are sold on to advertisers as well. Needless to say, the toothless ICO say they 'did not appear to breach marketing rules under the privacy and electronic communications regulations', as usual. Typical ICO fail.
ucas advertising privacy data-protection opt-in opt-out spam direct-marketing vodafone o2 microsoft red-bull uk universities grim-meathook-future ico
Good explanation of exponential backoff
I've often had to explain this key feature verbosely, and it's hard to do without handwaving. Great to have a solid, well-explained URL to point to
exponential-backoff backoff retries reliability web-services http networking internet coding design
Sacked Google worker says staff ratings fixed to fit template
Allegations of fixing to fit the stack-ranking curve: 'someone at Google always had to get a low score "of 2.9", so the unit could match the bell curve. She said senior staff "calibrated" the ratings supplied by line managers to ensure conformity with the template and these calibrations could reduce a line manager's assessment of an employee, in effect giving them the poisoned score of less than three.'
stack-ranking google ireland employment work bell-curve statistics eric-schmidt
Corporate Tax 2014: Irish Government's "flawed premise" on Apple's avoidance
According to our calculation about €40bn or over 40% of Irish services exports of €90bn in 2012 and related national output, resulted from global tax avoidance schemes. It is true that Ireland gains little from tax cheating but at some point, the US tax system will be reformed and a territorial system where companies are only liable in the US on US profits, would only be viable if there was a disincentive to shift profits to non-tax or low tax countries. The risk for Ireland is that a minimum foreign tax would be introduced that would be greater than the Irish headline rate of 12.5%. It's also likely that US investment in Ireland would not have been jeopardized if Irish politicians had not been so eager as supplicants to doff the cap. Nevertheless today it would be taboo to admit the reality of participation in massive tax avoidance and the Captain Renaults of Merrion Street will continue with their version of the Dance of the Seven Veils.
apple tax double-irish tax-avoidance google investment itax tax-evasion ireland
An online Magna Carta: Berners-Lee calls for bill of rights for web
TimBL backing the "web we want" campaign -- https://webwewant.org/
freedom gchq nsa censorship internet privacy web-we-want human-rights timbl tim-berners-lee
How the search for flight AF447 used Bayesian inference
Via jgc, the search for the downed Air France flight was optimized using this technique: 'Metron's approach to this search planning problem is rooted in classical Bayesian inference, which allows organization of available data with associated uncertainties and computation of the Probability Distribution Function (PDF) for target location given these data. In following this approach, the first step was to gather the available information about the location of the impact site of the aircraft. This information was sometimes contradictory and filled with ambiguities and uncertainties. Using a Bayesian approach we organized this material into consistent scenarios, quantified the uncertainties with probability distributions, weighted the relative likelihood of each scenario, and performed a simulation to produce a prior PDF for the location of the wreck.'
metron bayes bayesian-inference machine-learning statistics via:jgc air-france disasters probability inference searching
How the NSA Plans to Infect 'Millions' of Computers with Malware - The Intercept
The implants being deployed were once reserved for a few hundred hard-to-reach targets, whose communications could not be monitored through traditional wiretaps. But the documents analyzed by The Intercept show how the NSA has aggressively accelerated its hacking initiatives in the past decade by computerizing some processes previously handled by humans. The automated system – codenamed TURBINE – is designed to "allow the current implant network to scale to large size (millions of implants) by creating a system that does automated control implants by groups instead of individually." In a top-secret presentation, dated August 2009, the NSA describes a pre-programmed part of the covert infrastructure called the "Expert System," which is designed to operate "like the brain."
Great. Automated malware deployment to millions of random victims. See also the "I hunt sysadmins" section further down...malware gchq nsa oversight infection expert-systems turbine false-positives the-intercept surveillance
-
Burrito Justice nerds out on 'Goodnight Moon'. 'Maybe the bunny and the old lady are actually in a space elevator, getting closer to the moon as he gets into bed? Or as suggested by @transitmaps, the bunny can bend space and time? I do not have a good answer to this conundrum, but that is what the comments are for.'
goodnight-moon moon space time space-elevators childrens-books books physics
Inside the Mind of an anti-fluoridationist
An exceptionally well-researched and thorough disassembly of 'Public Health Investigation of Epidemiological data on Disease and Mortality in Ireland related to Water Fluoridation and Fluoride Exposure' by Declan Waugh, which appears to be going around currently
declan-waugh debunking flouride flouridation science mortality health ireland water
David Robert Grimes on the flouride kerfuffle
Hilariously, "The Girl Against Flouride" and other antiflouridation campaigners now allege he's a undercover agent of Alcoa and/or Glaxo Smith Kline, rather than dealing with any awkwardly hostile realities
flouride flouridation david-robert-grimes conspiracy funny science ireland alcoa glaxo-smith-kline
-
fantastic piece of C=64 history -- the "Ocean fast loader" by Paul Hughes, which allowed Commodore 64 games to load from tape at 4000 baud, far faster than the built-in system implementation, and with graphics and music at the same time
ocean-loader tapes c=64 commodore-64 history 1980s freeload paul-hughes
IntelliJ IDEA 13.1 will support Chronon Debugger
This, IMO, would be a really good reason to upgrade to the payware version of IDEA - Chronon looks cool.
Chronon is a new revolutionary tool keeping track of running Java programs and recording their execution process for later analysis, which can be helpful when you need to thoroughly retrace your steps when dealing with complicated bugs.
chronon debugging java intellij idea ides coding time-warp time
"Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" [PDF]
Google paper describing the infrastructure they've built for cross-service request tracing (ie. "tracer requests"). Features: low code changes required (since they've built it into the internal protobuf libs), low performance impact, sampling, deployment across the ~entire production fleet, output visibility in minutes, and has been live in production for over 2 years. Excellent read
dapper tracing http services soa google papers request-tracing tracers protobuf devops
-
'a Japanese term that means "mistake-proofing". A poka-yoke is any mechanism in a lean manufacturing process that helps an equipment operator avoid (yokeru) mistakes (poka). Its purpose is to eliminate product defects by preventing, correcting, or drawing attention to human errors as they occur.'
human-error errors mistakes poka-yoke failures prevention bugproofing manufacturing japan
The Hands That Made The Moomins
lovely New Yorker writeup on Tove Jansson, author of those beautiful children's books
tove-jansson moomins books childrens-books reading literature via:etienneshrdlu
James Casey writes about working at CERN
I am very heartened by Minister of State for Research and Innovation Sean Sherlock's recent announcement of a review of the costs and benefits of Ireland's membership of international research organisations including CERN. I disagreed with the conclusion of the last review which suggested that costs outweighed the benefits to Ireland. I think it was an extreme oversight not to be a part of the engineering phase of the Collider during the period 1998-2008 – but it's not too late. CERN will celebrate its 60th anniversary in 2014. There is no public scientific institution its equal in terms of the scale and complexity of problems being analysed and solved. No longer excluding young Irish people from being a part of this, from learning and growing from it, can only help Ireland.
Also, spot my name in lights ;)ireland cern science europe eu sean-sherlock james-casey www web history
Digging for cryptocurrency: The newbie's guide to mining altcoins
Mining Arscoins, dogecoins and litecoins -- CPU/GPU mining apps and how to run 'em
currency bitcoin altcoins dogecoin crypto mining ars-technica
A cautionary tale about building large-scale polyglot systems
'a fucking nightmare':
Cascading requires a compilation step, yet since you're writing Ruby code, you get get none of the benefits of static type checking. It was standard to discover a type issue only after kicking off a job on, oh, 10 EC2 machines, only to have it fail because of a type mismatch. And user code embedded in strings would regularly fail to compile – which you again wouldn't discover until after your job was running. Each of these were bad individually, together, they were a fucking nightmare. The interaction between the code in strings and the type system was the worst of all possible worlds. No type checking, yet incredibly brittle, finicky and incomprehensible type errors at run time. I will never forget when one of my friends at Etsy was learning Cascading.JRuby and he couldn't get a type cast to work. I happened to know what would work: a triple cast. You had to cast the value to the type you wanted, not once, not twice, but THREE times.
etsy scalding cascading adtuitive war-stories languages polyglot ruby java strong-typing jruby types hadoop
-
Attempting to cash out of Bitcoins turns out to be absurdly difficult:
Trying to sell the coins in person, and basically saying he ether wants Cash, or a Cashiers check (since it can be handed over right then and there), has apparently been a hilarious clusterfuck. Today he met some guy infront of his bank, and apparently as soon as he mentioned that he needs to get the cash checked to make sure it is not counterfeit, the guy freaked out and basically walked away. Stuff like this has been happening all week, and he apparently so far has only sold a single coin of several hundred.
bitcoin fail funny mtgox fraud cash fiat-currency via:rsynnott buttcoin
Florida cops used IMSI catchers over 200 times without a warrant
Harris is the leading maker of [IMSI catchers aka "stingrays"] in the U.S., and the ACLU has long suspected that the company has been loaning the devices to police departments throughout the state for product testing and promotional purposes. As the court document notes in the 2008 case, "the Tallahassee Police Department is not the owner of the equipment." The ACLU now suspects these police departments may have all signed non-disclosure agreements with the vendor and used the agreement to avoid disclosing their use of the equipment to courts. "The police seem to have interpreted the agreement to bar them even from revealing their use of Stingrays to judges, who we usually rely on to provide oversight of police investigations," the ACLU writes.
aclu police stingrays imsi-catchers privacy cellphones mobile-phones security wired
The Netflix Dynamic Scripting Platform
At the core of the redesign is a Dynamic Scripting Platform which provides us the ability to inject code into a running Java application at any time. This means we can alter the behavior of the application without a full scale deployment. As you can imagine, this powerful capability is useful in many scenarios. The API Server is one use case, and in this post, we describe how we use this platform to support a distributed development model at Netflix.
Holy crap.scripting dynamic-languages groovy java server-side architecture netflix
ZooKeeper Resilience at Pinterest
essentially decoupling the client services from ZK using a local daemon on each client host; very similar to Airbnb's Smartstack. This is a bit of an indictment of ZK's usability though
ops architecture clustering network partitions cap reliability smartstack airbnb pinterest zookeeper
FOI is better than tea and biscuits
Good post on the 'FOI costs too much' talking point.
I realise if you're a councillor, tea and biscuits sounds much more appealing than transparency and being held accountable and actually having to answer to voters, but those things are what you signed up to when you stood for election.
Answer to How many topics (queues) can be created in Apache Kafka? - Quora
Good to know:
'As far as I understand (this was true as of 2013, when I last looked into this issue) there's at least one Apache ZooKeeper znode per topic in Kafka. While there is no hard limitation in Kafka itself (Kafka is linearly scalable), it does mean that the maximum number of znodes comfortable supported by ZooKeeper (on the order of about ten thousand) is the upper limit of Kafka's scalability as far as the number of topics goes.'
Care.data is in chaos. It breaks my heart | Ben Goldacre
There are people in my profession who think they can ignore this problem. Some are murmuring that this mess is like MMR, a public misunderstanding to be corrected with better PR. They are wrong: it's like nuclear power. Medical data, rarefied and condensed, presents huge power to do good, but it also presents huge risks. When leaked, it cannot be unleaked; when lost, public trust will take decades to regain. This breaks my heart. I love big medical datasets, I work on them in my day job, and I can think of a hundred life-saving uses for better ones. But patients' medical records contain secrets, and we owe them our highest protection. Where we use them – and we have used them, as researchers, for decades without a leak – this must be done safely, accountably, and transparently. New primary legislation, governing who has access to what, must be written: but that's not enough. We also need vicious penalties for anyone leaking medical records; and HSCIC needs to regain trust, by releasing all documentation on all past releases, urgently. Care.data needs to work: in medicine, data saves lives.
hscic nhs care.data data privacy data-protection medicine hospitals pr
-
bookmarking as a future reference
Only 0.15 percent of mobile gamers account for 50 percent of all in-game revenue
Nice bit of marketing from the day job:
The group of gamers responsible for half of all in-game revenue in mobile titles is frightening because it is so narrow, according to a survey by Swrve, an established analytics and app marketing firm. About 0.15 percent of mobile gamers contribute 50 percent of all of the in-app purchases generated in free-to-play games. This means it may even more important than game companies realized in the past to find and retain the users that fall into the category of big spenders, or "whales." The vast majority of users never spend any money, despite the clever tactics that game publishers have developed to incentivize people to spend money in their favorite games.
-
'EAT CELEBRITY MEAT! BiteLabs grows meat from celebrity tissue samples and uses it to make artisanal salami.' Genius. (via John Looney)
via:john-looney meat startups food funny salami tissue-samples celebrity jennifer-lawrence
'Bobtail: Avoiding Long Tails in the Cloud' [pdf]
'A system that proactively detects and avoids bad neighbouring VMs without significantly penalizing node instantiation [in EC2]. With Bobtail, common [datacenter] communication patterns benefit from reductions of up to 40% in 99.9th percentile response times.' Excellent stuff -- another conclusion they come to is that it's not the network's fault, it's the Xen hosts themselves. The EC2 networking team will be happy about that ;)
-
Charlie Stross on GCHQ's 1984-esque webcam spying
webcams porn charlie-stross funny 1984 dystopian masturbation surveillance spying
Big doubts on big data: Why I won't be sharing my medical data with anyone - yet
These problems can be circumvented, but they must be dealt with, publically and soberly, if the NHS really does want to win public confidence. The NHS should approach selling the scheme to the public as if was opt-in, not opt-out, then work to convince us to join it. Tell us how sharing our data can help, but tell us what risk too. Let us decide if that balance is worth it. If it's found wanting, the NHS must go back to the drawing board and retool the scheme until it is. It's just too important to get wrong.
nhs uk privacy data-protection data-privacy via:mynosql big-data healthcare insurance
Welcome to Algorithmic Prison - Bill Davidow - The Atlantic
"Computer says no", taken to the next level.
Even if an algorithmic prisoner knows he is in a prison, he may not know who his jailer is. Is he unable to get a loan because of a corrupted file at Experian or Equifax? Or could it be TransUnion? His bank could even have its own algorithms to determine a consumer's creditworthiness. Just think of the needle-in-a-haystack effort consumers must undertake if they are forced to investigate dozens of consumer-reporting companies, looking for the one that threw them behind algorithmic bars. Now imagine a future that contains hundreds of such companies. A prisoner might not have any idea as to what type of behavior got him sentenced to a jail term. Is he on an enhanced screening list at an airport because of a trip he made to an unstable country, a post on his Facebook page, or a phone call to a friend who has a suspected terrorist friend?
privacy data big-data algorithms machine-learning equifax experian consumer society bill-davidow
RTE star Sharon Ni Bheolain stalked for six months - Independent.ie
as @Fergal says: '[this] case shows (a) the internet isn't anonymous, (b) we [ie. Ireland -jm] have laws to deal with threats and harassment'
law ireland harassment internet twitter email abuse cyberstalking
ImperialViolet - Apple's SSL/TLS bug
as we all know by now, a misplaced "goto fail" caused a critical, huge security flaw in versions of IOS and OSX SSL, since late 2012. Lessons: 1. unit test the failure cases, particularly for critical security code! 2. use braces. 3. dead-code analysis would have caught this. I'm not buying the "goto considered harmful" line, though, since any kind of control flow structure would have had the same problem.
coding apple osx ios crypto ssl security goto-fail goto fail unit-testing coding-standards
Comcast's deal with Netflix makes network neutrality obsolete
in a world where Netflix and Yahoo connect directly to residential ISPs, every Internet company will have its own separate pipe. And policing whether different pipes are equally good is a much harder problem than requiring that all of the traffic in a single pipe be treated the same. If it wanted to ensure a level playing field, the FCC would be forced to become intimately involved in interconnection disputes, overseeing who Verizon interconnects with, how fast the connections are and how much they can charge to do it.
verizon comcast internet peering networking netflix network-neutrality
Data visualization: breaking down The Economist's classic chart style
nice piece of classic graph design
Netflix packets being dropped every day because Verizon wants more money | Ars Technica
With Cogent and Verizon fighting, [peering capacity] upgrades are happening at a glacial pace, according to Schaeffer. "Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material." The congestion isn't only happening at peak times, he said. "These ports are so over-congested that they're running in this packet dropping state 22, 24 hours a day. Maybe at four in the morning on Tuesday or something there might be a little bit of headroom," he said.
packet-loss networking internet cogent netflix verizon peering
Hospital records of all NHS patients sold to insurers - Telegraph
The 274-page report describes the NHS Hospital Episode Statistics as a "valuable data source in developing pricing assumptions for 'critical illness' cover." It says that by combining hospital data with socio-economic profiles, experts were able to better calculate the likelihood of conditions, with "amazingly" clear forecasts possible for certain diseases, in particular lung cancer. Phil Booth, from privacy campaign group medConfidential, said: "The language in the document is extraordinary; this isn't about patients, this is about exploiting a market. Of course any commercial organisation will focus on making a profit – the question is why is the NHS prepared to hand this data over?"
-
'A Monumental Land Art Installation in the Sahara Desert', by the D.A.S.T. Arteam in 1997. More correctly, near the Red Sea resort of El Gouna -- so possible to visit!
el-gouna sahara deserts land-art art via:colossal desert-breath spirals
Harvard Research Computing Resources Misused for 'Dogecoin' Mining Operation
A member of the Harvard community was stripped of his or her access to the University's research computing facilities last week after setting up a "dogecoin" mining operation using a Harvard research network, according to an internal email circulated by Faculty of Arts and Sciences Research Computing officials.
-
turn Youtube videos into animated GIFs (via Waxy)
'Scaling to Millions of Simultaneous Connections' [pdf]
Presentation by Rick Reed of WhatsApp on the large-scale Erlang cluster backing the WhatsApp API, delivered at Erlang Factory SF, March 30 2012. lots of juicy innards here
erlang scaling scalability performance whatsapp freebsd presentations
Traffic Graph – Google Transparency Report
this is cool. Google are exposing an aggregated 'all services' hit count time-series graph, broken down by country, as part of their Transparency Report pages
transparency filtering web google http graphs monitoring syria
-
I want to emphasize that if you use redis as intended (as a slightly-persistent, not-HA cache), it's great. Unfortunately, more and more shops seem to be thinking that Redis is a full-service database and, as someone who's had to spend an inordinate amount of time maintaining such a setup, it's not. If you're writing software and you're thinking "hey, it would be easy to just put a SET key value in this code and be done," please reconsider. There are lots of great products out there that are better for the overwhelming majority of use cases.
Ouch. (via Aphyr) -
I'm going to need this pretty soon -- lots of white spots showing up with the current BenQ :(
Belkin managed to put their firmware update private key in the distribution
'The firmware updates are encrypted using GPG, which is intended to prevent this issue. Unfortunately, Belkin misuses the GPG asymmetric encryption functionality, forcing it to distribute the firmware-signing key within the WeMo firmware image. Most likely, Belkin intended to use the symmetric encryption with a signature and a shared public key ring. Attackers could leverage the current implementation to easily sign firmware images.' Using GPG to sign your firmware updates: yay. Accidentally leaving the private key in the distribution: sad trombone.
fail wemo belkin firmware embedded-systems security updates distribution gpg crypto public-key pki home-automation ioactive
-
On-the-fly video transcoding during live streaming. They've done a great job of this!
At the beginning of the development of this feature, we entertained the idea to simply pre-transcode all the videos in Dropbox to all possible target devices. Soon enough we realized that this simple approach would be too expensive at our scale, so we decided to build a system that allows us to trigger a transcoding process only upon user request and cache the results for subsequent fetches. This on-demand approach: adapts to heterogeneous devices and network conditions, is relatively cheap (everything is relative at our scale), guarantees low latency startup time.
ffmpeg dropbox streaming video cdn ec2 hls http mp4 nginx haproxy aws h264
GPLv2 being tested in US court
The case is still ongoing, so one to watch.
Plaintiff wrote an XML parser and made it available as open source software under the GPLv2. Defendant acquired from another vendor software that included the code, and allegedly distributed that software to parties outside the organization. According to plaintiff, defendant did not comply with the conditions of the GPL, so plaintiff sued for copyright infringement. Defendants moved to dismiss for failure to state a claim. The court denied the motion.
Latest Snowden leak: GCHQ spying on Wikileaks users
"How could targeting an entire website's user base be necessary or proportionate?" says Gus Hosein, executive director of the London-based human rights group Privacy International. "These are innocent people who are turned into suspects based on their reading habits. Surely becoming a target of a state's intelligence and security apparatus should require more than a mere click on a link." The agency's covert targeting of WikiLeaks, Hosein adds, call into question the entire legal rationale underpinning the state's system of surveillance. "We may be tempted to see GCHQ as a rogue agency, ungoverned in its use of unprecedented powers generated by new technologies," he says. "But GCHQ's actions are authorized by [government] ministers. The fact that ministers are ordering the monitoring of political interests of Internet users shows a systemic failure in the rule of law."
"Hackers" unsubscribed a former Mayor from concerned citizen's emails
"The dog ate my homework, er, I mean, hackers hacked my account."
Former Mayor of Kildare, Cllr. Michael Nolan, has denied a claim he asked a local campaigner to stop e-mailing him. Cllr. Michael Nolan from Newbridge said his site was hacked and wrong e-mails were sent out to a number of people, including Leixlip based campaigner, John Weigel. Mr. Weigel has been campaigning, along with others, about the danger of electromagnetic radiation to humans and the proximity of communications masts to homes and, in particular schools. He regularly updates local politicians on news items relating to the issue. Recently, he said that he had received an e-mail from Cllr. Nolan asking to be removed from Mr. Weigel's e-mail list. The Leader asked Cllr. Nolan why he had done this. But the Fine Gael councillors said that "his e-mail account was hacked and on one particular day a number of mails a were sent from my account pertaining to be from me."
dog-ate-my-homework hackers funny kildare newbridge fine-gael michael-nolan email politics ireland excuses
-
very good, workable tips on how to remote-work effectively (both in the comments of this thread and the original article)
tips productivity collaboration hn via:lhl remote-working telecommuting work
Disgraced Scientist Granted U.S. Patent for Work Found to be Fraudulent - NYTimes.com
Korean researcher Hwang Woo-suk electrified the science world 10 years ago with his claim that he had created the world's first cloned human embryos and had extracted stem cells from them. But the work was later found to be fraudulent, and Dr. Hwang was fired from his university and convicted of crimes. Despite all that, Dr. Hwang has just been awarded an American patent covering the disputed work, leaving some scientists dumbfounded and providing fodder to critics who say the Patent Office is too lax. "Shocked, that's all I can say," said Shoukhrat Mitalipov, a professor at Oregon Health and Science University who appears to have actually accomplished what Dr. Hwang claims to have done. "I thought somebody was kidding, but I guess they were not." Jeanne F. Loring, a stem cell scientist at the Scripps Research Institute in San Diego, said her first reaction was "You can't patent something that doesn't exist." But, she said, she later realized that "you can."
patents absurd hwang-woo-suk cloning stem-cells science biology uspto
-
'Testing applications under slow or flaky network conditions can be difficult and time consuming. Blockade aims to make that easier. A config file defines a number of docker containers and a command line tool makes introducing controlled network problems simple.' Open-source release from Dell's Cloud Manager team (ex-Enstratius), inspired by aphyr's Jepsen. Simulates packet loss using "tc netem", so no ability to e.g. drop packets on certain flows or certain ports. Still, looks very usable -- great stuff.
testing docker networking distributed distcomp enstratius jepsen network outages partitions cap via:lusis
-
what US airports are causing the most misery? Looks like that old favourite, storms in ORD, right now.... (via Theo Schlossnagle)
via:postwait misery air-travel travel flying ord weather maps
-
This sounds amazing. I hope it makes it to some kind of "semi-finished".
A semi-roguelike game inspired by Jorge Borges, Umberto Eco, Neal Stephenson, Shadow of the Colossus, Europa Universalis and Civilization. Although currently in its early stages, URR aims to explore several philosophical and sociological issues that both arose during the sixteenth and seventeenth century (when the game is approximately set), and in the present day, whilst almost being a deep, complex and highly challenging roguelike. To do this the game seeks to generate realistic world histories, though ones containing a few unusual happenings and anomalous experiences. The traditional roguelike staple of combat will be rare and deadly – whilst these mechanics will be modeled in detail, exploration, trade and diplomacy factors will have just as much effort put into them.
games ultima-ratio-regum roguelikes borges umberto-eco worlds ascii-art
-
It is interesting to note that the fake UK network was the only one detected by Verrimus. However, given that IMSI Catchers operate multiple fake towers simultaneously, it is highly likely that one or more Irish networks were also being intercepted. Very often a misconfiguration, such as an incorrect country code, is the only evidence available of an IMSI Catcher being deployed when forensic tools are not being used to look for one.
privacy imsi-catchers surveillance bugging spying gsocgate gsoc ireland mobile-phones
-
An extremely congested local network segment causes the "TCP incast" throughput collapse problem -- packet loss occurs, and TCP throughput collapses as a side effect. So far, this is pretty unsurprising, and anyone designing a service needs to keep bandwidth requirements in mind. However it gets worse with Riak. Due to a bug, this becomes a serious issue for all clients: the Erlang network distribution port buffers fill up in turn, and the Riak KV vnode process (in its entirety) will be descheduled and 'cannot answer any more queries until the A-to-B network link becomes uncongested.' This is where EC2's fully-uncontended-1:1-network compute cluster instances come in handy, btw. ;)
incast tcp networking bandwidth riak architecture erlang buffering queueing
Irish Law Society takes a stand for "brand owners IP rights"
The Law Society will attend a meeting of the Oireachtas Health Committee today to outline its strong opposition to the Government proposals to introduce legislation that will require tobacco products to use plain packaging. The society's director general Ken Murphy will be its principal representative at the meeting today to discuss its submission on the legislation, and to discuss its concerns that a plain packaging regime will undermine registered trade mark, and design, systems and will amount to an "expropriation of brand owners intellectual property rights'. Speaking ahead of the meeting, Mr Murphy told The Irish Times the views contained in it represent those of the Law Society as a whole, and its 10,000 members, and have been endorsed by the society as a whole, rather than the committee. Mr Murphy also said the purpose of the Law Society submission was not to protect the tobacco industry, rather the wider effect and impact such a law would have on intellectual property rights, trade marks, in other areas. "There is a real concern also that plain packaging in the tobacco industry is just the beginning of a trend that will severely undermine intellectual property owners' rights in other sectors such as alcohol, soft drinks and fast foods."
Judging by some reactions on Twitter, "endorsed by the society as a whole" may be over-egging it a little.law-society gubu law ireland ip packaging branding trademarks cigarettes health tobacco
British American Tobacco - Plain packaging of tobacco products
Compare and contrast with the Law Society's comments:
We believe we are entitled to use our packs to distinguish our products from those of our competitors. Our brands are our intellectual property which we have created and invested in. Plain packaging would deny us the right to use brands. But also, a brand is also an important tool for consumers. As the British Brands Group has stated , plain packaging legislation "ignores the crucial role that branding plays in providing consumers with high quality, consistent products they can trust". The restriction of valuable corporate brands by any government would risk placing it in breach of legal obligations relating to intellectual property rights and, in most cases, international trade.
law-society branding ip ireland tobacco cigarettes law trademarks
Why dispute resolution is hard
Good stuff (as usual) from Ross Anderson and Stephen Murdoch. 'Today we release a paper on security protocols and evidence which analyses why dispute resolution mechanisms in electronic systems often don't work very well. On this blog we've noted many many problems with EMV (Chip and PIN), as well as other systems from curfew tags to digital tachographs. Time and again we find that electronic systems are truly awful for courts to deal with. Why? The main reason, we observed, is that their dispute resolution aspects were never properly designed, built and tested. The firms that delivered the main production systems assumed, or hoped, that because some audit data were available, lawyers would be able to use them somehow. As you'd expect, all sorts of things go wrong. We derive some principles, and show how these are also violated by new systems ranging from phone banking through overlay payments to Bitcoin. We also propose some enhancements to the EMV protocol which would make it easier to resolve disputes over Chip and PIN transactions.'
finance security ross-anderson emv bitcoin chip-and-pin banking architecture verification vvat logging
CJEU in #Svensson says that in general it is OK to hyperlink to protected works without permission
IPKat says 'this morning the Court of Justice of the European Union issued its keenly awaited decision in Case C-466/12 Svensson [...]: The owner of a website may, without the authorisation of the copyright holders, redirect internet users, via hyperlinks, to protected works available on a freely accessible basis on another site. This is so even if the internet users who click on the link have the impression that the work is appearing on the site that contains the link.' This is potentially big news. Not so much for the torrent-site scenario, but for the NNI/NLI linking-to-newspaper-stories scenario.
ip svensson cjeu eu law pirate-bay internet web links http copyright linking hyperlinks
Migrating from MongoDB to Cassandra
Interesting side-effect of using LUKS for full-disk encryption: 'For every disk read, we were pulling in 3MB of data (RA is sectors, SSZ is sector size, 6144*512=3145728 bytes) into cache. Oops. Not only were we doing tons of extra work, but we were trashing our page cache too. The default for the device-mapper used by LUKS under Ubuntu 12.04LTS is incredibly sub-optimal for database usage, especially our usage of Cassandra (more small random reads vs. large rows). We turned this down to 128 sectors — 64KB.'
-
Good to see the guys cracking on without me ;) '2014-02-11: SpamAssassin 3.4.0 has been released adding native support for IPv6, improved DNS Blocklist technology and support for massively-scalable Bayesian filtering using the Redis backend.'
193_Cellxion_Brochure_UGX Series 330
The Cellxion UGX Series 330 is a 'transportable Dual GSM/Triple UMTS Firewall and Analysis Tool' -- ie. an IMSI catcher in a briefcase, capable of catching IMSI/IMEIs in 3G. It even supports configurable signal strength. Made in the UK
cellxion imsi-catchers imei surveillance gsocgate gsm 3g mobile-phones security spying
-
'an interesting approach to a common problem, that of securely passing secrets around an infrastructure. It uses GPG signed files under the hood and nicely integrates with both version control systems and S3.' I like this as an approach to securely distributing secrets across a stack of services during deployment. Check in the file of keys, gpg keygen on the server, and add it to the keyfile's ACL during deployment. To simplify, shared or pre-generated GPG keys could also be used. (via the Devops Weekly newsletter)
gpg encryption crypto secrets key-distribution pki devops deployment
java - Why not use Double or Float to represent currency?
A good canonical URL for this piece of coding guidance.
For example, suppose you have $1.03 and you spend 42c. How much money do you have left? System.out.println(1.03 - .42); => prints out 0.6100000000000001.
"I'm Sorry for what I said when I was Hungry" tee-shirt
I can relate to this
-
'One case involved Julian Assange's current home at the Ecuadorian Embassy in London, where visitors were surprised to receive welcome messages from a Ugandan telephone company. It turned out the messages were coming from a foreign base station device installed on the roof, masquerading as a cell tower for surveillance purposes. Appelbaum suspects the GCHQ simply forgot to reformat the device from an earlier Ugandan operation.'
via T.J. McIntyre.surveillance nsa privacy imsi-catchers gchq london uganda mobile-phones julian-assange ecuador embassies
The Spyware That Enables Mobile-Phone Snooping - Bloomberg
More background on IMSI catchers -- looking likely to have been the "government-level technology" used to snoop on the Garda Ombudsman's offices, particularly given the 'detection of an unexpected UK 3G network near the GSOC offices':
The technology involved is called cellular interception. The active variety of this, the "IMSI catcher," is a portable device that masquerades as a mobile phone tower. Any phone within range (a mile for a low-grade IMSI catcher; as much as 100 miles for a passive interception device with a very large antenna, such as those used in India) automatically checks to see if the device is a tower operated by its carrier, and the false "tower" indicates that it is. It then logs the phone's International Mobile Subscriber Identity number -- and begins listening in on its calls, texts and data communications. No assistance from any wireless carrier is needed; the phone has been tricked. [...] "network extender" devices -- personal mobile-phone towers -- sold by the carriers themselves, often called femtocells, can be turned into IMSI catchers.
Via T.J. McIntyrevia:tjmcintyre imsi-catchers surveillance privacy gsocgate mobile-phones spying imsi
Git is not scalable with too many refs/*
Mailing list thread from 2011; git starts to keel over if you tag too much
Survey results of EU teens using the internet
A lot of unsupervised use:
Just under half of children said they access the internet from their own bedroom on a daily basis with 22pc saying they do so several times a day.
surveys eu ireland politics filtering internet social-media facebook children teens cyber-bullying
-
a pretty thought-provoking article from Linux Journal on women in computing, and how we're doing it all wrong
feminism community programming coding women computing software society work linux-journal children teaching
-
leading Bitcoin exchange "Magic The Gatherine Online Exchange" turns out to suffer from crappy code, surprise:
why does Mt. Gox experience this issue? They run a custom Bitcoin daemon, with a custom implementation of the Bitcoin protocol. Their implementation, against all advice, does rely on the transaction ID, which makes this attack possible. They have actually been warned about it months ago by gmaxwell, and have apparently decided to ignore this warning. In other words, this is not a vulnerability in the Bitcoin protocol, but an implementation error in Mt. Gox' custom Bitcoin software.
The rest of the article is eyeopening, including the MySQL injection vulnerabilities and failure to correctly secure a Prolexic-defended server. https://news.ycombinator.com/item?id=7211286 has some other shocking reports of Bitcoin operators being incompetent, including 'Bitomat, the incompetent exchange that deleted their own [sole] amazon instance accidentally which contained all their keys, and thus customer funds'. wtfbbq -
The side-effects of algorithmic false-positives get worse and worse.
What's more, he adds, the NSA often locates drone targets by analyzing the activity of a SIM card, rather than the actual content of the calls. Based on his experience, he has come to believe that the drone program amounts to little more than death by unreliable metadata. "People get hung up that there's a targeted list of people," he says. "It's really like we're targeting a cell phone. We're not going after people – we're going after their phones, in the hopes that the person on the other end of that missile is the bad guy."
false-positives glenn-greenwald drones nsa death-by-metadata us-politics terrorism sim-cards phones mobile-phones
IBM's creepy AI cyberstalking plans
'let's say that you tweet that you've gotten a job offer to move to San Francisco. Using IBM's linguistic analysis technologies, your bank would analyze your Twitter feed and not only tailor services it could offer you ahead of the move--for example, helping you move your account to another branch, or offering you a loan for a new house -- but also judge your psychological profile based upon the tone of your messages about the move, giving advice to your bank's representatives about the best way to contact you.'
Ugh. Here's hoping they've patented this shit so we don't actually have to suffer through it. Creeeepy. (via Adam Shostack)datamining ai ibm stupid-ideas creepy stalking twitter via:adamshostack
-
This is bananas. Confirmation bias running amok.
Brandon Mayfield was a US Army veteran and an attorney in Portland, OR. After the 2004 Madrid train bombing, his fingerprint was partially matched to one belonging to one of the suspected bombers, but the match was a poor one. But by this point, the FBI was already convinced they had their man, so they rationalized away the non-matching elements of the print, and set in motion a train of events that led to Mayfield being jailed without charge; his home and office burgled by the FBI; his client-attorney privilege violated; his life upended.
confirmation-bias bias law brandon-mayfield terrorism fingerprints false-positives fbi scary
A patent on 'Birth of a Child By Centrifugal Force'
On November 9 1965, the Blonskys were granted US Patent 3,216,423, for an Apparatus for Facilitating the Birth of a Child by Centrifugal Force. The drawings, as well as the text, are a revelation. The Patent Office has them online at http://tinyurl.com/jd4ra and I urge you - if you have any shred of curiosity in your body - to look them up. For conceiving what appears to be the greatest labour-saving device ever invented, George and Charlotte Blonsky won the 1999 Ig Nobel Prize in the field of Managed Health Care.
This is utterly bananas. (via christ)via:christ crazy patents 1960s centrifuge birth medicine ignobels
A Linguist Explains the Grammar of Doge. Wow.
In this sense, doge really is the next generation of LOLcat, in terms of a pet-based snapshot of a certain era in internet language. We've kept the idea that animals speak like an exaggerated version of an internet-savvy human, but as our definitions of what it means to be a human on the internet have changed, so too have the voices that we give our animals. Wow.
Big, Small, Hot or Cold - Your Data Needs a Robust Pipeline
'(Examples [of big-data B-I crunching pipelines] from Stripe, Tapad, Etsy & Square)'
stripe tapad etsy square big-data analytics kafka impala hadoop hdfs parquet thrift
Realtime water level data across Ireland
Some very nice Dygraph-based time-series graphs in here, along with open CSV data. Good job!
The Gardai haven't requested info on any Twitter accounts in the past 6 months
This seems to imply they haven't been investigating any allegations of cyber-bullying/harassment from "anonymous" Twitter handles, despite having the legal standing to do so. Enforcement is needed, not new laws
cyber-bullying twitter social-media enforcement gardai policing harassment online society law government
QuakeNet IRC Network- Article - PRESS RELEASE: IRC NETWORKS UNDER SYSTEMATIC ATTACK FROM GOVERNMENTS
QuakeNet are not happy about GCHQ's DDoS attacks against them.
Yesterday we learned ... that GCHQ, the British intelligence agency, are performing persistent social and technological attacks against IRC networks. These attacks are performed without informing the networks and are targeted at users associated with politically motivated movements such as "Anonymous". While QuakeNet does not condone or endorse and actively forbids any illegal activity on its servers we encourage discussion on all topics including political and social commentary. It is apparent now that engaging in such topics with an opinion contrary to that of the intelligence agencies is sufficient to make people a target for monitoring, coercion and denial of access to communications platforms. The ... documents depict GCHQ operatives engaging in social engineering of IRC users to entrap themselves by encouraging the target to leak details about their location as well as wholesale attacks on the IRC servers hosting the network. These attacks bring down the IRC network entirely affecting every user on the network as well as the company hosting the server. The collateral damage and numbers of innocent people and companies affected by these forms of attack can be huge and it is highly illegal in many jurisdictions including the UK under the Computer Misuse Act.
-
Good to know; this generic anti-flap damping algorithm has a name.
A proportional-integral-derivative controller (PID controller) is a generic control loop feedback mechanism (controller) widely used in industrial control systems. A PID controller calculates an "error" value as the difference between a measured process variable and a desired setpoint. The controller attempts to minimize the error by adjusting the process control outputs.
control damping flapping pid-controller industrial error algorithms
German IT Industry Looks for Boom from Snowden Revelations - SPIEGEL ONLINE
This is a great idea -- Neelie Kroes suggesting that there be a certification mark for EU companies who have top-of-the-line data protection practices.
GCHQ slide claiming that they DDoS'd anonymous' IRC servers
Mikko Hypponen: "This makes British Government the only Western government known to have launched DDoS attacks."
RTE internal memo to unhappy staff re Pantigate
'I want to reassure you that RTÉ explored every option available to it, including right of reply. Legal advice was sought and all avenues were explored, including an offer to make a donation to a neutral charity.' And they folded. Notable lack of testicular fortitude by our national broadcaster.
fail rte leaks memos pantigate panti-bliss homophobia libel defamation ireland
A looming breakthrough in indistinguishability obfuscation
'The team's obfuscator works by transforming a computer program into what Sahai calls a "multilinear jigsaw puzzle." Each piece of the program gets obfuscated by mixing in random elements that are carefully chosen so that if you run the garbled program in the intended way, the randomness cancels out and the pieces fit together to compute the correct output. But if you try to do anything else with the program, the randomness makes each individual puzzle piece look meaningless. This obfuscation scheme is unbreakable, the team showed, provided that a certain newfangled problem about lattices is as hard to solve as the team thinks it is. Time will tell if this assumption is warranted, but the scheme has already resisted several attempts to crack it, and Sahai, Barak and Garg, together with Yael Tauman Kalai of Microsoft Research New England and Omer Paneth of Boston University, have proved that the most natural types of attacks on the system are guaranteed to fail. And the hard lattice problem, though new, is closely related to a family of hard problems that have stood up to testing and are used in practical encryption schemes.' (via Tony Finch)
obfuscation cryptography via:fanf security hard-lattice-problem crypto science
Little's Law, Scalability and Fault Tolerance: The OS is your bottleneck. What you can do?
good blog post on Little's Law, plugging quasar, pulsar, and comsat, 3 new open-source libs offering Erlang-like lightweight threads on the JVM
jvm java quasar pulsar comsat littles-law scalability async erlang
Target Hackers Broke in Via HVAC Company
Avivah Litan, a fraud analyst with Gartner Inc., said that although the current PCI standard does not require organizations to maintain separate networks for payment and non-payment operations (page 7), it does require merchants to incorporate two-factor authentication for remote network access originating from outside the network by personnel and all third parties.
Target shared the same network for outside contractor access and the critical POS devices. fail. (via Joe Feise)via:joe-feise hvac contractors fraud malware 2fa security networking payment pci
Yahoo! moving EMEA operations to Dublin
Like many companies, the structure of Yahoo's business is driven by the needs of the business. There are a number of factors which influence decisions about the locations in which the business operates. To encourage more collaboration and innovation, we're increasing our headcount in Dublin, thus continuing to bring more Yahoos together in fewer locations. Dublin is already the European home to many of the world's leading global technology brands and has been a home for Yahoo for over a decade already.
Via Conor O'Neill
-
zero-install, one-click video chat, using WebRTC. nifty
conference webrtc chat collaboration video google-chrome conferencing
Opinion: How can we get over 'Pantigate'?
The fact that RTÉ had agreed to pay damages (€80,000 in total, according to reports yesterday) to the 'injured parties', only came to light in an email from the [far-right Catholic lobby group Iona Institute] to its members last Tuesday. Given the ramifications of the decision to make any kind of payment – regardless of the amount – both for the TV licence payer and those who voice contrarian opinions, the lack of coverage in print media as soon as the Iona email came to light marked a low point for print journalism in Ireland. Aside from a lead story on the damages printed in this paper last Wednesday and ongoing debate online, the media has been glacially slow with commentary and even reportage of the affair. The debacle has untold ramifications for public life in this country. That many liberal commentators may now baulk at the opportunity to speak and write openly and honestly about homophobia is the most obvious issue here. Most worrying of all, however, is the question that with a referendum on the introduction of gay marriage on the horizon, how can we expect the national broadcaster to facilitate even-handed debate on the subject when they've already found themselves cowed before reaching the first hurdle?
homophobia politics ireland libel dissent lobbying defamation law gay-marriage iona-institute journalism newspapers
-
Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs. Rest.li fills a niche for building RESTful service architectures at scale, offering a developer workflow for defining data and REST APIs that promotes uniform interfaces, consistent data modeling, type-safety, and compatibility checked API evolution.
The new underlying comms layer for Voldemort, it seems.voldemort d2 rest.li linkedin json rest http api frameworks java
Hardened SSL Ciphers Using ELB and HAProxy
ELBs support the PROXY protocol
elb security proxying ssl tls https haproxy perfect-forward-secrecy aws ec2
-
"A data scientist is a statistician who lives in San Francisco" - slide from Monkigras this year. lols
data-scientist statistics statistician funny jokes san-francisco tech monkigras
The Million Dollar Deal - YouTube
My mate Luke's doc on the World Series of Poker -- now online in full. it's great.
A documentary about the World Series Of Poker in Las Vegas. Featuring Andrew Black, Donnacha O'Dea, Mike Magee, "Mad" Martyn Wilson, Mark Napolitano, Amarillo Slim, Scotty Nguyen, Dave "Devilfish" Ulliott & Matt Damon. Narrated by John Hurt. Directed by John Butler, Produced by Luke McManus
documentaries film poker world-series-of-poker mike-magee andrew-black donnacha-odea matt-damon
How to invoke section 4 of the Data Protection Acts in Ireland
One wierd trick to get your personal data (in any format) from any random organisation, for only EUR6.35 and up to 40 days wait! Good to know.
Hospitals and doctors' offices in Ireland will give a person their medical records if they ask for them. Mostly. Eventually. When they get to it. And, sometimes, if you pay them over €100 (for a large file). But, like so much else in the legal world, there is a set of magic words you can incant to place a 40 day deadline on the delivery of your papers and limit the cost to €6.35 -- you invoke the Data Protection Acts data access request procedure.
data-protection privacy data-retention dpa-section-4 data ireland medical law dpa
Save 10% on rymdkapsel on Steam
rymdkapsel is a game where you take command of a space station and its minions. You will have to plan your expansion and manage your resources to explore the galaxy.
recommended by JK.
Yammer Engineering - Resiliency at Yammer
Not content with adding Hystrix (circuit breakers, threadpooling, request time limiting, metrics, etc.) to their entire SOA stack, they've made it incredibly configurable by hooking in a web-based configuration UI, allowing dynamic on-the-fly reconfiguration by their ops guys of the circuit breakers and threadpools in production. Mad stuff
hystrix circuit-breakers resiliency yammer ops threadpools soa dynamic-configuration archaius netflix
A network of 'homes', where children's happiness was relentlessly destroyed
Stories of this sort will tumble out to the inquiry over the next 18 months, making it plain that the network of "homes" where children's happiness had relentlessly, deliberately, systematically been destroyed, this archipelago of Catholic evil, had covered the entire island. These things should be kept in mind when next we hear it said that the social ills of today can be explained by reference to loss of faith in the traditional institutions of moral authority. This is the reverse of the truth and an insult to the victims of an unforgiveable sin.
horror care-homes politics catholicism religion ireland derry church abuse children
Ukrainian police use cellphones to track protestors, court order shows
Protesters for weeks had suspected that the government was using location data from cellphones near the demonstration to pinpoint people for political profiling, and they received alarming confirmation when a court formally ordered a telephone company to hand over such data. [...] Three cellphone companies — Kyivstar, MTS and Life — denied that they had provided the location data to the government or had sent the text messages. Kyivstar suggested that it was instead the work of a "pirate" cellphone tower set up in the area. In a ruling made public on Wednesday, a city court ordered Kyivstar to disclose to the police which cellphones were turned on during an antigovernment protest outside the courthouse on Jan. 10.
tech location-tracking tracking privacy ukraine cellphones mobile-phones civil-liberties
-
Netflix open-source library to make using ZooKeeper from Java less of a PITA. I really wish I'd used this now, having reimplemented some key parts of it after failures in prod ;)
10 Things We Forgot to Monitor
a list of not-so-common outage causes which are easy to overlook; swap rate, NTP drift, SSL expiration, fork rate, etc.
Irish Company Locates Office in Ireland
Hot on the heels of Dropbox, AirBnB, Twitter, Facebook and many others, Irish online ticket sales company Tito are amongst the latest in a long series of companies choosing to locate their offices in Ireland. "It just seemed to make sense," said founder Paul Campbell, talking about the decision making process that led him to set up shop in the capital, Dublin. "Dublin is great. There's something really familiar about it that I can't quite put my finger on."
Har har!
-
Sugru + neodymium magnets = WANT
Capabilities of Movements and Affordances of Digital Media: Paradoxes of Empowerment | DMLcentral
Paradoxically, it's possible that the widespread use of digital tools facilitates capabilities in some domains, such as organization, logistics, and publicity, while simultaneously engendering hindrances to [political] movement impacts on other domains, including those related to policy and electoral spheres.
society politics activism tech internet gezi-park tahrir-square euromaidan occupy
-
Good description of the "hero coder" organisational antipattern.
Now imagine that most of the team is involved in fire-fighting. New recruits see the older recruits getting praised for their brave work in the line-of-fire and they want that kind of praise and reward too. Before long everyone is focused on putting out fires and it is no ones interest to step back and take on the risks that long-term DevOps-focused goals entail.
coding ops admin hero-coder hero-culture firefighting organisations teams culture
Open-Sourcing Ssync: An Out-of-the-Box Distributed Rsync
a script to perform divide-and-conquer recursive rsync over SSH
Improving compaction in Cassandra with cardinality estimation
nice use of HyperLogLog
hyperloglog hll algorithms cassandra bloom-filters sstables cardinality
-
Ad company InMobi are using graphite heavily (albeit not as heavily as $work are), ran into the usual scaling issues, and chose to fix it in code by switching from a filesystem full of whisper files to a LevelDB per carbon-cache:
The carbon server is now able to run without breaking a sweat even when 500K metrics per minute is being pumped into it. This has been in production since late August 2013 in every datacenter that we operate from.
Very nice. I hope this gets merged/supported.graphite scalability metrics leveldb storage inmobi whisper carbon open-source
BBC News - Pair jailed over abusive tweets to feminist campaigner
When a producer from BBC Two's Newsnight programme tracked Nimmo down after he had sent the abuse, the former call centre worker told him: "The police will do nothing, it's only Twitter."
bbc bullying social-media twitter society uk trolls trolling abuse feminism cyberbullying
If You Used This Secure Webmail Site, the FBI Has Your Inbox
TorMail was a Tor-based webmail system, and apparently its drives have been imaged and seized by the FBI. More info on the Freedom Hosting seizure:
The connection, if any, between the FBI obtaining Freedom Hosting's data and apparently launching the malware campaign through TorMail and the other sites isn't spelled out in the new document. The bureau could have had the cooperation of the French hosting company that Marques leased his servers from. Or it might have set up its own Tor hidden services using the private keys obtained from the seizure, which would allow it to adopt the same .onion addresses used by the original sites. The French company also hasn't been identified. But France's largest hosting company, OVH, announced on July 29, in the middle of the FBI's then-secret Freedom Hosting seizure, that it would no longer allow Tor software on its servers. A spokesman for the company says he can't comment on specific cases, and declined to say whether Freedom Hosting was a customer. "Wherever the data center is located, we conduct our activities in conformity with applicable laws, and as a hosting company, we obey search warrants or disclosure orders," OVH spokesman Benjamin Bongoat told WIRED. "This is all we can say as we usually don't make any comments on hot topics."
fbi freedom-hosting hosting tor tormail seizures ovh colo servers
Sky parental controls break many JQuery-using websites
An 11 hour outage caused by a false positive in Sky's anti-phishing filter; all sites using the code.jquery.com CDN for JQuery would have seen errors.
Sky still appears to be blocking code.jquery.com and all files served via the site, and more worryingly is that if you try to report the incorrect category, once signing in on the Sky website you an error page. We suspect the site was blocked due to being linked to by a properly malicious website, i.e. code.jquery.com and some javascript files were being used on a dodgy website and every domain mentioned was subsequently added to a block list.
(via Tony Finch)via:fanf sky filtering internet uk anti-phishing phish jquery javascript http web fps false-positives
Coders performing code reviews of scientific projects: pilot study
'PLOS and Mozilla conducted a month-long pilot study in which professional developers performed code reviews on software associated with papers published in PLOS Computational Biology. While the developers felt the reviews were limited by (a) lack of familiarity with the domain and (b) lack of two-way contact with authors, the scientists appreciated the reviews, and both sides were enthusiastic about repeating the experiment. ' Actually sounds like it was more successful than this summary implies.
plos mozilla code-reviews coding science computational-biology biology studies
-
The views expressed by [the Iona Institute] – especially in relation to gay people – are very much at odds with the liberal secular society that Ireland has become. Indeed, Rory O'Neill suggested that the only time he experiences homophobia is online or at the hands of Iona and Waters. When they're done with that, they can ask why Iona is given so much room in the media. In any other country in the world, an organisation as litigious as Iona would never be asked to participate in anything.
homophobia ireland john-waters iona-institute politics catholicism religion libel defamation rte the-irish-times
-
mine's a Smoky/Spicy/Medicinal, thanks
Cassandra: tuning the JVM for read heavy workloads
The cluster we tuned is hosted on AWS and is comprised of 6 hi1.4xlarge EC2 instances, with 2 1TB SSDs raided together in a raid 0 configuration. The cluster's dataset is growing steadily. At the time of this writing, our dataset is 341GB, up from less than 200GB a few months ago, and is growing by 2-3GB per day. The workload on this cluster is very read heavy, with quorum reads making up 99% of all operations.
Some careful GC tuning here. Probably not applicable to anyone else, but good approach in general.Terms of Reference for the DCENR Internet Content Advisory Group
this is definitely one to send a consultation document response to
internet policing cyberbullying bullying antisocial free-speech governance children blocking filtering consultations dcenr
Stupid Simple Things SF Techies Could Do To Stop Being Hated - Anil Dash
I've seen a lot of hand-wringing from techies in San Francisco and Silicon Valley saying "Why are we so hated?" now that there's been a more vocal contingent of people being critical of their lack of civic responsibility. Is it true that corruption and NIMBYism have kept affordable housing from being built? Sure. Is it true that members of the tech industry do contribute tax dollars to the city? Absolutely. But does that mean techies have done enough? Nope.
anil-dash politics society san-francisco gentrification helping tech community housing
-
Some basic succinct data structures. [...] The main highlights are: a novel, broadword-based implementation of rank/select queries for up to 264 bits that is highly competitive with known 32-bit implementations on 64-bit architectures (additional space required is 25% for ranking and 12.5%-37.5% for selection); several Java structures using the Elias–Fano representation of monotone sequences for storing pointers, variable-length bit arrays, etc. Java code implementing minimal perfect hashing using around 2.68 bits per element (also using some broadword ideas); a few Java implementations of monotone minimal perfect hashing. Sux is free software distributed under the GNU Lesser General Public License.
Why sugar helped remove Victoria Line concrete flood
Sugar blocks concrete from setting. This I did not know
Ukrainian government targeting protesters using threatening SMS messages
The government's opponents said three recent actions had been intended to incite the more radical protesters and sow doubt in the minds of moderates: the passing of laws last week circumscribing the right of public assembly, the blocking of a protest march past the Parliament building on Sunday, and the sending of cellphone messages on Tuesday to people standing in the vicinity of the fighting that said, "Dear subscriber, you are registered as a participant in a mass disturbance." [....] The phrasing of the message, about participating in a "mass disturbance," echoed language in a new law making it a crime to participate in a protest deemed violent. The law took effect on Tuesday. And protesters were concerned that the government seemed to be using cutting-edge technology from the advertising industry to pinpoint people for political profiling. Three cellphone companies in Ukraine — Kyivstar, MTS and Life — denied that they had provided the location data to the government or had sent the text messages, the newspaper Ukrainskaya Pravda reported. Kyivstar suggested that it was instead the work of a "pirate" cellphone tower set up in the area.
targeting mobile-phones sms text-messaging via:tjmcintyre geotargeting protest ukraine privacy surveillance tech 1984
UK porn filter blocks game update that contained 'sex' in URL
Staggeringly inept. The UK national porn filter blocks based on a regexp match of the URL against /.*sex.*/i -- the good old "Scunthorpe problem". Better, it returns a 404 response. This is also a good demonstration of how web filtering has unintended side effects, breaking third-party software updates with its false positives.
The update to online strategy game League of Legends was disrupted by the internet filter because the software attempted to access files that accidentally include the word "sex" in the middle of their file names. The block resulted in the update failing with "file not found" errors, which are usually created by missing files or broken updates on the part of the developers.
uk porn filtering guardian regular-expressions false-positives scunthorpe http web league-of-legends sex
Register article on Amazon's attitude to open source
This article is frequently on target; this secrecy (both around open source and publishing papers) was one of the reasons I left Amazon.
Of the sources with whom we spoke, many indicated that Amazon's lack of participation was a key reason for why people left the company – or never joined at all. This is why Amazon's strategy of maintaining secrecy may derail the e-retailer's future if it struggles to hire the best talent. [...] "In many cases in the big companies and all the small startups, your Github profile is your resume," explained another former Amazonian. "When I look at developers that's what I'm looking for, [but] they go to Amazon and that resume stops ... It absolutely affects the quality of their hires." "You had no portfolio you could share with the world," said another insider on life after working at Amazon. "The argument this was necessary to attract talent and to retain talent completely fell on deaf ears."
amazon recruitment secrecy open-source hiring work research conferences
Chinese Internet Traffic Redirected to Small Wyoming House
'That address — which is home to some 2,000 companies on paper — was the subject of a lengthy 2011 Reuters investigation that found that among the entities registered to the address were a shell company controlled by a jailed former Ukraine prime minister; the owner of a company charged with helping online poker operators evade an Internet gambling ban; and one entity that was banned from government contracts after selling counterfeit truck parts to the Pentagon.'
china internet great-firewall dns wyoming attacks security not-the-onion
James Friend | PCE.js - Classic Mac OS in the Browser
This is a demo of PCE's classic Macintosh emulation, running System 7.0.1 with MacPaint, MacDraw, and Kid Pix. If you want to try out more apps and games see this demo.
Incredible. I remember using this version of MacPaint!javascript browser emulation mac macos macpaint macdraw claris kid-pix history desktop pce
-
'Lightweight performance tools'.
Likwid stands for 'Like I knew what I am doing'. This project contributes easy to use command line tools for Linux to support programmers in developing high performance multi-threaded programs. It contains the following tools: likwid-topology: Show the thread and cache topology likwid-perfctr: Measure hardware performance counters on Intel and AMD processors likwid-features: Show and Toggle hardware prefetch control bits on Intel Core 2 processors likwid-pin: Pin your threaded application without touching your code (supports pthreads, Intel OpenMP and gcc OpenMP) likwid-bench: Benchmarking framework allowing rapid prototyping of threaded assembly kernels likwid-mpirun: Script enabling simple and flexible pinning of MPI and MPI/threaded hybrid applications likwid-perfscope: Frontend for likwid-perfctr timeline mode. Allows live plotting of performance metrics. likwid-powermeter: Tool for accessing RAPL counters and query Turbo mode steps on Intel processor. likwid-memsweeper: Tool to cleanup ccNUMA memory domains.
No kernel patching required. (via kellabyte)via:kellabyte linux performance testing perf likwid threading multithreading multicore mpi numa
Backblaze Blog » What Hard Drive Should I Buy?
Because Backblaze has a history of openness, many readers expected more details in my previous posts. They asked what drive models work best and which last the longest. Given our experience with over 25,000 drives, they asked which ones are good enough that we would buy them again. In this post, I'll answer those questions.
Safe cross-thread publication of a non-final variable in the JVM
Scary, but potentially useful in future, so worth bookmarking. By carefully orchestrating memory accesses using volatile and non-volatile fields, one can ensure that a non-volatile, non-synchronized field's value is safely visible to all threads after that point due to JMM barrier semantics.
What you are looking to do is enforce a barrier between your initializing stores and your publishing store, without that publishing store being made to a volatile field. This can be done by using volatile access to other fields in the publication path, without using those variables in the later access paths to the published object.
volatile atomic java jvm gil-tene synchronization performance threading jmm memory-barriers
Irish quango allegedly buys fake twitter followers
The Consumers Association of Ireland had a sudden jump from 300 to 3000 Twitter followers, mostly from Latin and South America -- with more followers in Brazil than Ireland. They are now blaming "hacking": http://www.independent.ie/irish-news/consumers-body-denies-buying-3000-twitter-fans-29931196.html
consumers quangos ireland politics twitter funny fake-followers latin-america south-america brazil social-media tech
Big Red Kitchen on buying Irish honey
1. There is NO SUCH THING as "Organic Irish Honey" (due to EU directives making it impossible to certify); 2. In the absence of Organic the best thing you can look for is "Raw Irish honey" (which is of Irish origin, and not heated to very high temperatures, so it retains its antibacterial properties); 3. Blended honeys, or honeys which say EEC/Non EEC are NOT Irish, however they may be packed in Ireland; 4. Look for the NIHBS "Produced by Native Irish Honey Bees" or similar, for confirmation that the honey you are buying is indeed of Irish origin.
irish ireland honey buy-irish big-red-kitchen food organic-food
More than 50% of Irish companies have "suffered a data breach" in 2013
The research, conducted among hundreds of Irish companies' IT managers by the Irish Computer Society, reveals that 51 per cent of Irish firms have suffered a data breach over the last year, a jump on 43 per cent recorded in 2012.
Wow, that's high.Irish Internet Providers Roll Out KickassTorrents Blockade
The lucrative whack-a-mole business continues -- mostly in response to High Court actions, although Eircom are just helping out. I bet a google for "kickass proxy" doesn't return anything useful at all, of course....
kat kickasstorrents bittorrent piracy copyright high-court ireland eircom filtering blocking
Internet Censors Came For TorrentFreak & Now I'm Really Mad
TF are not happy about Sky blocking their blog.
There can be little doubt that little by little, piece by piece, big corporations and governments are taking chunks out of the free Internet. Today they pretend that the control is in the hands of the people, but along the way they are prepared to mislead and misdirect, even when their errors are pointed out to them. I'm calling on Sky, Symantec, McAfee and other ISPs about to employ filtering to categorize this site correctly as a news site or blog and to please start listening to people's legitimate complaints about other innocent sites. It serves nobody's interests to wrongfully block legitimate information.
censorship isps uk sky torrentfreak piracy copyright filtering blocking symantec filesharing
Harry - A Tool for Measuring String Similarity
a small tool for comparing strings and measuring their similarity. The tool supports several common distance and kernel functions for strings as well as some exotic similarity measures. The focus of Harry lies on implicit similarity measures, that is, comparison functions that do not give rise to an explicit vector space. Examples of such similarity measures are the Levenshtein distance and the Jaro-Winkler distance. For comparison Harry loads a set of strings from input, computes the specified similarity measure and writes a matrix of similarity values to output. The similarity measure can be computed based on the granularity of characters as well as words contained in the strings. The configuration of this process, such as the input format, the similarity measure and the output format, are specified in a configuration file and can be additionally refined using command-line options. Harry is implemented using OpenMP, such that the computation time for a set of strings scales linear with the number of available CPU cores. Moreover, efficient implementations of several similarity measures, effective caching of similarity values and low-overhead locking further speedup the computation.
via kragen.via:kragen strings similarity levenshtein-distance algorithms openmp jaro-winkler edit-distance cli commandline hamming-distance compression
-
A nice node.js app to perform continuous deployment from a GitHub repo via its webhook support, from Matt Sergeant
github node.js runit deployment git continuous-deployment devops ops
-
yummy-looking recipe from Lily at amexicancook.ie
Succinct Data Structures: Cramming 80,000 words into a Javascript file
a succinctly-encoded trie -- slow to encode, super-compact, but fast to look up
succinct-encoding tries coding performance compression data-structures algorithms
Transport Minister planning to make hi-vis jackets mandatory for cyclists
The minister also spoke of a number of new transport initiatives, such as mandatory use of high visibility jackets by cyclists.
The Malware That Duped Target Has Been Found
a Windows 'RAM scraper' trojan known as Trojan.POSRAM, which was used to attack the Windows-based point-of-sales systems which the POS terminals are connected to. part of an operation called Kaptoxa. 'The code is based on a previous malicious tool known as BlackPOS that is believed to have been developed in 2013 in Russia, though the new variant was highly customized to prevent antivirus programs from detecting it' ... 'The tool monitors memory address spaces used by specific programs, such as payment application programs like pos.exe and PosW32.exe that process the data embossed in the magnetic strip of credit and debit cards data. The tool grabs the data from memory.' ... 'The siphoned data is stored on the system, and then every seven hours the malware checks the local time on the compromised system to see if it's between the hours of 10 a.m. and 5 p.m. If so, it attempts to send the data over a temporary NetBIOS share to an internal host inside the compromised network so the attackers can then extract the data over an FTP ... connection.' http://www.pcworld.com/article/2088920/target-credit-card-data-was-sent-to-server-in-russia.html says the data was then transmitted to another US-based server, and from there relayed to Russia, and notes: 'At the time of its discovery, Trojan.POSRAM "had a zero percent antivirus detection rate, which means that fully updated antivirus engines on fully patched computers could not identify the software as malicious," iSight said.' Massive AV fail.
kaptoxa trojans ram-scrapers trojan.posram posram point-of-sale security hacks target credit-cards pin ftp netbios smb
Full iSight report on the Kaptoxa attack on Target
'POS malware is becoming increasingly available to cyber criminals' ... 'there is growing demand for [this kind of malware]'. Watch your credit cards...
debit-cards credit-cards security card-present attacks kaptoxa ram-scrapers trojans point-of-sale pos malware target
-
Both Heartland Payment Systems and Hannaford Bros. were in fact certified PCI-compliant while the hackers were in their system. In August 2006, Wal-Mart was also certified PCI-compliant while unknown attackers were lurking on its network. [...] "This PCI standard just ain't working," says Litan, the Gartner analyst. "I wouldn't say it's completely pointless. Because you can't say security is a bad thing. But they're trying to patch a really weak [and] insecure payment system [with it]."
Basically, RAM scrapers have been in use in live attacks, sniffing credentials in the clear, since 2007. Ouch.ram-scrapers trojans pins pci-dss compliance security gartner walmart target
ISPAI responds to TD Patrick O'Donovan's bizarre comments regarding "open source browsers"
ISPAI is rather dismayed and somewhat confused by the recent press release issued by Deputy Patrick O'Donovan (FG). He appears to be asking the Oireachtas Communications Committee (of which he is a member) to investigate: "the matter of tougher controls on the use of open source internet browsers and payment systems" which he claims "allow users to remain anonymous for illegal trade of drugs weapons and pornography." Deputy O'Donovan would do well to ask the advice of industry experts on these matters given that legislating to curtail the use of such legitimate software or services, which may be misused by some, is neither practical nor logical. Whether or not a browser is open source bears no relevance to its ability to be the subject of anonymous use. Indeed, Deputy O'Donovan must surely be confusing and conflating different technical concepts? In tracing illegal activities, Law Enforcement Agencies and co-operating parties will use IP addresses – users' choice of browser has little relevance to an investigation of criminal activity. Equally, it may be that the Deputy is uncomfortable with the concept of electronic payment systems but these underpin the digital economy which is bringing enormous benefit to Ireland. Yes, these may be misused by criminals but so are cash and traditional banking services. Restricting the growth of innovative financial services is not the solution to tackling cyber criminals who might be operating what he describes as "online supermarkets for illegal goods." Tackling international cybercrime requires more specialist Law Enforcement resources at national level and improved international police cooperation supported by revision of EU legislation relating to obtaining server log evidence existing in other jurisdictions.
ispai open-source patrick-o-donovan fine-gael press-releases tor darknet crime
-
I use it to modify Time Machine's backup behavior using weighted reservoir sampling. I built Time Warp to preserve important backup snapshots and prevent Time Machine from deleting them.
via Aman. Nifty!backup python time-machine decay exponential-decay weighting algorithms snapshots ops
Nominet now filtering .uk domain registrations for 'sex-crime content'
Amazing. Massive nanny-stateism of the 'something must be done' variety, with a 100% false-alarm hit rate, and it's now policy.
'Nominet have made a decision, based on a report by Lord Macdonald QC, that recommends that they check any domain registration that signals sex crime content or is in itself a sex crime. This is screening of domains within 48 hours of registration, and de-registration. The report says that such domains should be reported to the police.' [....] 'The report itself states [...] that in 2013 Nominet checked domains for key words used by the IWF, and as a result reported tens of thousands of domains to IWF for checking, all of which were false positives. Not one was, in fact, related to child sex abuse.'
filtering nominet false-positives nanny-state uk sex-crimes false-alarms domains iwf
Tuning advice for HTTPS for nginx and HAProxy
from Ilya Grigorik. nginx version here: http://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/
A common error when using the Metrics library is to record Timer metrics on things like API calls, using the default settings, then to publish those to a time-series store like Graphite. Here's why this is a problem.
By default, a Timer uses an Exponentially Decaying Reservoir. The docs say:
'A histogram with an exponentially decaying reservoir produces quantiles which are representative of (roughly) the last five minutes of data. It does so by using a forward-decaying priority reservoir with an exponential weighting towards newer data. Unlike the uniform reservoir, an exponentially decaying reservoir represents recent data, allowing you to know very quickly if the distribution of the data has changed.'
This is more-or-less correct -- but the key phrase is 'roughly'. In reality, if the frequency of updates to such a timer drops off, it could take a lot longer, and if you stop updating a timer which uses this reservoir type, it'll never decay at all. The GraphiteReporter will dutifully capture the percentiles, min, max, etc. from that timer's reservoir every minute thereafter, and record those to Graphite using the current timestamp -- even though the data it was derived from is becoming more and more ancient.
Here's a demo. Note the long stretch of 800ms 99th-percentile latencies on the green line in the middle of this chart:

However, the blue line displays the number of events. As you can see, there were no calls to this API for that 8-hour period -- this one was a test system, and the user population was safely at home, in bed. So while Graphite is claiming that there's an 800ms latency at 7am, in reality the 800ms-latency event occurred 8 hours previously.
I observed the same thing in our production systems for various APIs which suffered variable invocation rates; if rates dropped off during normal operation, the high-percentile latencies hung around for far longer than they should have. This is quite misleading when you're looking at a graph for 10pm and seeing a high 99th-percentile latency, when the actual high-latency event occurred hours earlier. On several occasions, this caused lots of user confusion and FUD with our production monitoring, so we needed to fix it.
Here are some potential fixes.
-
Modify ExponentiallyDecayingReservoir to also call rescaleIfNeeded() inside getSnapshot() -- but based on this discussion, it appears the current behaviour is intended (at least for the mean measurement), so that may not be acceptable. Another risk of this is that it leaves us in a position where the percentiles displayed for time T may actually have occurred several minutes prior to that, which is still misleading (albeit less so).
-
Switch to sliding time window reservoirs, but those are unbounded in size -- so a timer on an unexpectedly-popular API could create GC pressure and out-of-memory scenarios. It's also the slowest reservoir type, according to the docs. That made it too risky for us to adopt in our production code as a general-purpose Timer implementation.
-
Update, Dec 2017: as of version 3.2.3 of Dropwizard Metrics, there is a new SlidingTimeWindowArrayReservoir reservoir implementation, which is a drop-in replacement for SlidingTimeWindowReservoir, with much more acceptable memory footprint and GC impact. It costs roughly 128 bits per stored measurement, and is therefore judged to be 'comparable with ExponentiallyDecayingReservoir in terms of GC overhead and performance'. (thanks to Bogdan Storozhuk for the tip)
-
What we eventually did in our code was to use this Reporter class instead of GraphiteReporter; it clears all Timer metrics' reservoirs after each write to Graphite. This is dumb and dirty, reaching across logical class boundaries, but at the same time it's simple and comprehensible behaviour: with this, we can guarantee that the percentile/min/max data recorded at timestamp T is measuring events in that timestamp's 1-minute window -- not any time before that. This is exactly what you want to see in a time-series graph like those in Graphite, so is a very valuable feature for our metrics, and one that others have noted to be important in comparable scenarios elsewhere.
Here's an example of what a graph like the above should look like (captured from our current staging stack):

Note that when there are no invocations, the reported 99th-percentile latency is 0, and each measurement doesn't stick around after its 1-minute slot.
Another potential bug fix for a related issue, would be to add support to Metrics so that it can use Gil Tene's LatencyUtils package, and its HdrHistogram class, as a reservoir. (Update: however, I don't think this would address the "old data leaking into newer datapoints" problem as fully.) This would address some other bugs in the Exponentially Decaying Reservoir, as Gil describes:
'In your example of a system logging 10K operations/sec with the histogram being sampled every second, you'll be missing 9 out of each 10 actual outliers. You can have an outlier every second and think you have one roughly every 10. You can have a huge business affecting outlier happening every hour, and think that they are only occurring once a day.'
Eek.
graphite hdrhistogram instrumentation latency metrics percentiles reservoirs statistics tsd
Branchless hex-to-decimal conversion hack
via @simonebordet, on the mechanical-sympathy list: ((c & 0x1F) + ((c >> 6) * 0x19) – 0x10)
hacks one-liners coding performance optimization hex conversion numbers ascii
A sampling profiler for your daily browsing - Google Groups
via Ilya Grigorik: Chrome Canary now has a built-in, always-on, zero-overhead code profiler. I want this in my server-side JVMs!
chrome tracing debugging performance profiling google sampling-profiler javascript blink v8
-
from tonx. Good advice
-
'The web's only open collection of legal contracts and the best way to negotiate and sign documents online'. (via Kowalshki)
How an emulator-fueled robot reprogrammed Super Mario World on the fly
Suffice it to say that the first minute-and-a-half or so of this [speedrun] is merely an effort to spawn a specific set of sprites into the game's Object Attribute Memory (OAM) buffer in a specific order. The TAS runner then uses a stun glitch to spawn an unused sprite into the game, which in turn causes the system to treat the sprites in that OAM buffer as raw executable code. In this case, that code has been arranged to jump to the memory location for controller data, in essence letting the user insert whatever executable program he or she wants into memory by converting the binary data for precisely ordered button presses into assembly code (interestingly, this data is entered more quickly by simulating the inputs of eight controllers plugged in through simulated multitaps on each controller port).
oh. my. god. This is utterly bananas.games hacking omgwtfbbq hacks buffer-overrun super-mario snes security
Nassim Taleb: retire Standard Deviation
Use the mean absolute deviation [...] it corresponds to "real life" much better than the first—and to reality. In fact, whenever people make decisions after being supplied with the standard deviation number, they act as if it were the expected mean deviation.' Graydon Hoare in turn recommends the median absolute deviation. I prefer percentiles, anyway ;)
statistics standard-deviation stddev maths nassim-taleb deviation volatility rmse distributions
Mathematical Purity in Distributed Systems: CRDTs Without Fear
Via Tony Finch. Funnily enough, the example describes Swrve: mobile game analytics, backed by a CRDT-based eventually consistent data store ;)
storage crdts semilattice idempotency commutativity data-structures distcomp eventual-consistency
-
some good data (and graphs) on baby names (via Ruth)
-
Crowdsourcing transcription of some WWI artifacts: 'The story of the British Army on the Western Front during the First World War is waiting to be discovered in 1.5 million pages of unit war diaries. We need your help to reveal the stories of those who fought in the global conflict that shaped the world we live in today.' (via Luke)
Map of Steamship Routes of the World, 1914
massive image. very cool (via burritojustice)
maps desktop images steamships shipping history 1914 travel world
Google Fonts recently switched to using Zopfli
Google Fonts recently switched to using new Zopfli compression algorithm: the fonts are ~6% smaller on average, and in some cases up to 15% smaller! [...] What's Zopfli? It's an algorithm that was developed by the compression team at Google that delivers ~3~8% bytesize improvement when compared to gzip with maximum compression. This byte savings comes at a cost of much higher encoding cost, but the good news is, fonts are static files and decompression speed is exactly the same. Google Fonts pays the compression cost once and every clients gets the benefit of smaller download. If you're curious to learn more about Zopfli: http://bit.ly/Y8DEL4
"Understanding the Robustness of SSDs under Power Fault", FAST '13 [paper]
Horrific. SSDs (including "enterprise-class storage") storing sync'd writes in volatile RAM while claiming they were synced; one device losing 72.6GB, 30% of its data, after 8 injected power faults; and all SSDs tested displayed serious errors including random bit errors, metadata corruption, serialization errors and shorn writes. Don't trust lone unreplicated, unbacked-up SSDs!
pdf papers ssd storage reliability safety hardware ops usenix serialization shorn-writes bit-errors corruption fsync
Irish politician calls for ban on "open source browsers"
'Fine Gael TD for Limerick, Patrick O'Donovan has called for tougher controls on the use of open source internet browsers and payment systems which allow users to remain anonymous in the illegal trade of drugs, weapons and pornography.' Amazing. Yes, this is real.
open-source clueless omgwtfbbq fine-gael ireland fail funny tor inept
Little-known Apollo 10 incident
'Apollo 10 had a little known incident in flight as evidenced by this transcript.' http://pic.twitter.com/NCZy7OdxDU
poo turds space spaceflight funny history apollo-10 apollo accidents
-
As can be guessed, the higher the compression ratio, the more efficient FSE becomes compared to Huffman, since Huffman can't break the "1 bit per symbol" limit. FSE speed is also very stable, under all probabilities. I'm quite please with the result, especially considering that, since the invention of arithmetic coding in the 70's, nothing really new has been brought to this field. This is still beta stuff, so please consider this first release for testing purposes mostly.
Looking forward to this making it into a production release of some form.compression algorithms via:kragen fse finite-state-entropy-coding huffman arithmetic-coding
-
A bug in a scheduled OS upgrade script caused live production DB servers to be upgraded while live. Fixes include fixing that script by verifying non-liveness on the host itself, and a faster parallel MySQL binary-log recovery command.
Creative Commons event in Dublin this Friday
'Maximising Digital Creativity, Sharing and Innovation', Event organised by Creative Commons Ireland and Faculty of Law, University College Cork, Lecture Theatre, National Gallery of Ireland, Clare Street entrance, Dublin 2, Friday 17 January 2014, 9.45 a.m. to 1 p.m. (via Darius Whelan)
Growing up unvaccinated: A healthy lifestyle couldn't prevent many childhood illnesses.
I understand, to a point, where the anti-vaccine parents are coming from. Back in the '90s, when I was a concerned, 19-year-old mother, frightened by the world I was bringing my child into, I was studying homeopathy, herbalism, and aromatherapy; I believed in angels, witchcraft, clairvoyants, crop circles, aliens at Nazca, giant ginger mariners spreading their knowledge to the Aztecs, the Incas, and the Egyptians, and that I was somehow personally blessed by the Holy Spirit with healing abilities. I was having my aura read at a hefty price and filtering the fluoride out of my water. I was choosing to have past life regressions instead of taking antidepressants. I was taking my daily advice from tarot cards. I grew all my own veg and made my own herbal remedies. I was so freaking crunchy that I literally crumbled. It was only when I took control of those paranoid thoughts and fears about the world around me and became an objective critical thinker that I got well. It was when I stopped taking sugar pills for everything and started seeing medical professionals that I began to thrive physically and mentally.
Life on Mars: Irish man signs up for colony mission
Last week, a private space exploration company called Mars One announced that it has shortlisted 1,058 people from 200,000 applicants who wanted to travel to Mars. Roche is the only Irishman on the list. The catch? If he goes, he can never come back.
Mad stuff. Works at the Science Gallery, so a co-worker of a friend, to bootscience-gallery dublin ireland mars-one mars one-way-trips exploration future space science joseph-roche
UK NHS will soon require GPs pass confidential medical data to third parties
Specifically, unanonymised, confidential, patient-identifying data, for purposes of "admin, healthcare planning, and research", to be held indefinitely, via the HSCIC. Opt-outs may be requested, however
opt-out privacy medical data healthcare nhs uk data-privacy data-protection
-
'why the fuck does my fridge need Twitter?'
twitter funny tech home fridges internet web appliances consume
Visualisation of the Raft distributed consensus protocol
Very pretty
consensus raft visualization distributed distcomp algorithms
Directv DCA2SR0 01 Deca II Connected Home Adapter
a John-Looney-recommended MoCA adapter, allowing legacy coax home wiring to be used to transmit ethernet
Bruce Schneier and Matt Blaze on TAO's Methods
An important point:
As scarily impressive as [NSA's TAO] implant catalog is, it's targeted. We can argue about how it should be targeted -- who counts as a "bad guy" and who doesn't -- but it's much better than the NSA's collecting cell phone location data on everyone on the planet. The more we can deny the NSA the ability to do broad wholesale surveillance on everyone, and force them to do targeted surveillance in individuals and organizations, the safer we all are.
nsa tao security matt-blaze bruce-schneier surveillance tempest
How the NSA (may have) put a backdoor in RSA's cryptography: A technical primer
An excellent description of how the Dual_EC_DRBG backdoor works
surveillance tech dual_ec_drbg nsa rsa security backdoors via:jgc elliptic-curves
Who Made That Nigerian Scam? - NYTimes.com
The history behind the 419 advance-fee fraud scam.
According to Robert Whitaker, a historian at the University of Texas, an earlier version of the con, known as the Spanish Swindle or the Spanish Prisoner trick, plagued Britain throughout the 19th century.
True facts about Ocean Radiation and the Fukushima Disaster
solid science
fukushima japan radiation risk ocean disasters sieverts contamination sea fish science
Packet Flight: Facebook News Feed @8X
good dataviz of a HTTP page load: 'this is a visualization of a Facebook News Feed load from the perspective of the client, over a 3G wireless connection. Different packet types have different shapes and colors.' (via John Harrington)
via:johnharrington visualization facebook dataviz networking tcp 3g
URGENT: Input needed on EU copyright consultation - Boing Boing
The EC is looking for feedback -- but not much, and pretty sharpish.
Go to www.copywrongs.eu and answer the questions which are important to you. You do not have to answer all the questions, only the ones that matter to you. [...] The deadline is 5 February 2014. Until then, we should provide the European Commission with as many responses as possible!
Peter Norvig writes a program to play regex golf with arbitrary lists
In response to XKCD 1313. This is excellent. It's reminiscent of my SpamAssassin SOUGHT-ruleset regexp-discovery algorithm, described in http://taint.org/2007/03/05/134447a.html , albeit without the BLAST step intended to maximise pattern length and minimise false positives
python regex xkcd blast rule-discovery spamassassin rules regexps regular-expressions algorithms peter-norvig
-
Good description of Etsy's take on continuous deployment, committing directly to trunk, hidden with feature-flags, from Rafe Colburn
continuous-deployment coding agile deployment devops etsy rafe-colburn
Dogs like to excrete in alignment with the Earth's magnetic field
Dogs preferred to excrete with the body being aligned along the North-south axis under calm magnetic field conditions.
Paul Graham and the Manic Pixie Dream Hacker
Under Graham's influence, Mark [Zuckerberg], like many in Silicon Valley, subscribes to the Manic Pixie Dream Hacker ideal, making self-started teenage hackers Facebook's most desired recruiting targets, not even so much for their coding ability as their ability to serve as the faces of hacking culture. "Culture fit", in this sense, is one's ability to conform to the Valley's boyish hacker fantasy, which is easier, obviously, the closer you are to a teenage boy. Like the Manic Pixie Dream Girl's role of existing to serve the male film protagonist's personal growth, the Manic Pixie Dream Hacker's job is to embody the dream hacker role while growing the VC's portfolio. This is why the dream hacker never ages, never visibly develops interests beyond hardware and code, and doesn't question why nearly all the other people receiving funding look like him. Like the actress playing the pixie dream girl, the pixie dream boy isn't being paid to question the role for which he has been cast. In this way, for all his supposed "disruptiveness", the hacker pixie actually does exactly what he is told: to embody, while he can, the ideal hacker, until he is no longer young, mono-focused, and boyish-seeming enough to qualify for the role (at that point, vested equity may allow him to retire). And like in Hollywood, VCs will have already recruited newer, younger ones to play him.
hackers manic-pixie-dream-girl culture-fit silicon-valley mark-zuckerberg paul-graham y-combinator vc work investment technology recruitment facebook ageism equality sexism
-
Flapjack aims to be a flexible notification system that handles: Alert routing (determining who should receive alerts based on interest, time of day, scheduled maintenance, etc); Alert summarisation (with per-user, per media summary thresholds); Your standard operational tasks (setting scheduled maintenance, acknowledgements, etc). Flapjack sits downstream of your check execution engine (like Nagios, Sensu, Icinga, or cron), processing events to determine if a problem has been detected, who should know about the problem, and how they should be told.
We need your help to keep working for European digital rights in 2014
Grim. DRI are facing a 5-figure legal bill from the music industry - they need your donations to avoid shutdown
donations dri funding amicus-curiae law ireland digital-rights-ireland emi irma