Justin's Linklog Posts

Sandymount Repair Cafe

‘A repair café brings together people with things that need fixin’ with people who have the skills to fix them in a social cafe style environment. It is an effort to move away from the throwaway culture that prevailed at the end of the twentieth century and move towards a more sustainable and enlightened approach to our relationship with consumer goods. Repair cafes are self organising events at a community level run by local volunteers with the support of local community groups, local agencies and other interested organisations. They are not-for-profit but not anti-profit and an important part of their goal is to promote local repair businesses and initiatives. www.repaircafe.ie is the online hub of a network of repair cafés across Ireland.’ Sounds interesting: https://twitter.com/DubCityCouncil/status/481777655445204992 says they’ll be doing it tomorrow from 2-5pm in Sandymount in Dublin.

(tags: dublin sandymount repair fixing diy frugality repaircafe hardware)

Chef Vault

A way to securely store secrets (auth details, API keys, etc.) in Chef

(tags: chef storage knife authorisation api-keys security encryption)

Amazon EC2 Service Limits Report Now Available

‘designed to make it easier for you to view and manage your limits for Amazon EC2 by providing the latest information on service limits and links to quickly request limit increases. EC2 Service Limits Report displays all your service limit information in one place to help you avoid encountering limits on future EC2, EBS, Auto Scaling, and VPC usage.’

(tags: aws ec2 vpc ebs autoscaling limits ops)

Delivery Notifications for Simple Email Service

Today we are enhancing SES with the addition of delivery notifications. You can now elect to receive an Amazon SNS notification each time SES successfully delivers a message to a recipient’s email server. These notifications give you increased visibility into the mail delivery process. With today’s release, you can now track deliveries, bounces, and complaints, all via notification to the SNS topic or topics of your choice.

(tags: delivery email smtp ses aws sns notifications ops)

How Emoji Get Lost In Translation

I recently texted a friend to say how I was excited to meet her new boyfriend, and, because “excited” doesn’t look so exciting on an iPhone screen, I editorialized with what seemed then like an innocent “[dancer]”. (Translation: Can’t wait for the fun night out!) On an Android phone, I realized later, that panache would have been a put-down: The dancers become “[playboy bunny].” (Translation: You’re a Playboy bunny who gets around!)

(tags: emoji icons graphics text speech phones)

Hailo pulling in EUR1M per month in Dublin alone

based on these (pretty rough) estimates. Good going, I’m a massive fan

(tags: hailo taxis driving cars public-transport dublin b2c b2b)

Google Replaces MapReduce With New Hyper-Scale Cloud Analytics System

MR no more:

“We don’t really use MapReduce anymore,” [Urs] Hölzle said in his keynote presentation at the Google I/O conference in San Francisco Wednesday. The company stopped using the system “years ago.” Cloud Dataflow, which Google will also offer as a service for developers using its cloud platform, does not have the scaling restrictions of MapReduce. “Cloud Dataflow is the result of over a decade of experience in analytics,” Hölzle said. “It will run faster and scale better than pretty much any other system out there.”

(tags: map-reduce google hadoop cloud-dataflow scalability big-data urs-holzle google-io)

This Internet Millionaire Has a New Deal For You – D Magazine

Good interview with Dave “Woot” Rutledge, who’s now well out of Amazon and plans to get back into the crap-clearing business at Meh.com: ‘Amazon’s fundamental misunderstanding of what made Woot great can be seen today on the site. It sells many items simultaneously. It’s a marketplace, not an event. The write-ups are cute, not subversively funny. Woot is no longer a bug-eyed beast with eight tentacles. It’s a pancake with two smaller pancakes for Mickey Mouse ears and a smile made of whipped cream. In 2012, two years into his three-year deal with Amazon, Rutledge walked. He won’t say how many millions his early departure cost him, but his contract with Amazon included a three-year non-compete clause from the date of sale, and he was watching the clock.’

(tags: amazon ecommerce business b2c woot.com meh.com dave-rutledge selling acquisitions)

NYC generates hash-anonymised data dump, which gets reversed

There are about 1000*26**3 = 21952000 or 22M possible medallion numbers. So, by calculating the md5 hashes of all these numbers (only 24M!), one can completely deanonymise the entire data. Modern computers are fast: so fast that computing the 24M hashes took less than 2 minutes.

(tags: hashing sha1 md5 bruce-schneier anonymization deanonymization security new-york nyc taxis data big-data hmac keyed-hashing salting)

Older programmers aren’t gone, they’re just outnumbered

So says “Uncle Bob” Martin

(tags: culture coding software age career reputation stack-overflow staffing)

Benchmarking LevelDB vs. RocksDB vs. HyperLevelDB vs. LMDB Performance for InfluxDB

A few interesting things come out of these results. LevelDB is the winner on disk space utilization, RocksDB is the winner on reads and deletes, and HyperLevelDB is the winner on writes. On smaller runs (30M or less), LMDB came out on top on most of the metrics except for disk size. This is actually what we’d expect for B-trees: they’re faster the fewer keys you have in them.

(tags: benchmarks leveldb datastores storage hyperleveldb rocksdb ycsb lmdb influxdb)

How to make breaking changes and not break all the things

Well-written description of the “several backward-compatible changes” approach to breaking-change schema migration (via Marc)

(tags: databases coding compatibility migration schemas sql continuous-deployment)

Minnesota Measles Outbreak Traced Back To A Single Unvaccinated Child

A single child caught measles while visiting Kenya, returned to Minnesota, infected 4 others, who in turn exposed others, with an ultimate count of 3000 exposed and 21 confirmed cases. (16 of the 21 were unvaccinated; 46% of the Somali children in this community were unvaccinated in a 2010 survey.)

(tags: minnesota safety measles health vaccination kenya somali)

Report of the Internet Content Governance Advisory Group

looking at the summary, looks broadly sensible; no government-mandated filtering/blocking I can spot quickly

(tags: internet filtering safety kids porn blocking ireland pegi ratings reports pdf)

Startup equity gotcha

‘Two months ago, an early Uber employee thought that he had found a buyer for his vested stock, at $200 per share. But when his agent tried to seal the deal, Uber refused to sign off on the transfer. Instead, it offered to buy back the shares for around $135 a piece, which is within the same price range that Google Ventures and TPG Capital had paid to invest in Uber the previous July. Take it or hold it.’ As rbranson on Twitter put it: ‘reminder that startup equity is basically worthless unless you’re a founder or investor, OR the company goes public.’

(tags: startups uber stock stock-options shares share-option equity via:rbranson work)

Call me maybe: Elasticsearch

Wow, these are terrible results. From the sounds of it, ES just cannot deal with realistic outage scenarios and is liable to suffer catastrophic damage in reasonably-common partitions.

If you are an Elasticsearch user (as I am): good luck. Some people actually advocate using Elasticsearch as a primary data store; I think this is somewhat less than advisable at present. If you can, store your data in a safer database, and feed it into Elasticsearch gradually. Have processes in place that continually traverse the system of record, so you can recover from ES data loss automatically.

(tags: elasticsearch ops storage databases jepsen partition network outages reliability)

Code Spaces data and backups deleted by hackers

Rather scary story of an extortionist wiping out a company’s AWS-based infrastructure. Turns out S3 supports MFA-required deletion as a feature, though, which would help against that.

(tags: ops security extortion aws ec2 s3 code-spaces delete mfa two-factor-authentication authentication infrastructure)

Google forced to e-forget a company worldwide

Here we go…. Canadian company wins case to censor search results for its competitors.

When Google argued that Canadian law couldn’t be applied to the entire world, the court responded by citing British Columbia’s Law and Equity Act, which grants broad power for a court to issue injunctions when it’s “just or convenient that the order should be made.” Google also tried to argue against the injunction on the basis of it amounting to censorship. The court responded that there are already entire categories of content that get censored, such as child abuse imagery. Will this be the first of a new wave of requests for company website take-downs?

(tags: canada via:stx censorship google search takedowns datalink equustek gw1000 hardware)

The dark truth about modern Ireland its media don’t talk about

Sinead O’Shea writing for the Guardian:

The economy has been built on cronyism, group-think, the double talk of absurdly low corporate tax rates and light touch regulation, the cult of the leader, an over reliance on “strong” international forces. These were the factors that caused the Celtic Tiger to collapse. This has had consequences for all. It’s the same for the system of shame and sexual repression. The impact has not been restricted to its most obvious victims. Ireland is not just a bad place to be a woman or an immigrant, it’s a bad place to be in any way “different.” As a result, sadly, it’s a bad place to be anyone at all.

(tags: ireland history women celtic-tiger industrial-schools immigration sinead-o-shea tuam abortion pregnancy)

Data sharing deal with U.S. referred to EU’s top court | Reuters

High Court Justice Gerard Hogan said that given the Safe Harbour agreement, which says that U.S. has sufficient data safeguards in place, the Irish regulator did not have the authority to investigate. If Safe Harbour stands, the student group’s application must fail, he said. “The critical issue which arises is whether the proper interpretation of the 1995 [EU data protection] directive and the 2000 Commission decision [on the Safe Harbour principles] should be re-evaluated in the light of the subsequent entry into force of article 8 of the EU charter,” on the right to the protection of personal data, Hogan said.

(tags: eu safe-harbor privacy high-court ireland law data-protection)

PetRegistrationAndPurchase.cs

A good example of “raw” BDD, without using a framework like Cucumber, Steak etc.

(tags: bdd testing csharp acceptance-tests coding)

cavalle/steak · GitHub

a minimal extension of RSpec-Rails that adds several conveniences to do acceptance testing of Rails applications using Capybara. It’s an alternative to Cucumber in plain Ruby.

(tags: rails ruby testing acceptance-testing steak bdd rspec coding)

ScalaTest

Scala’s BDD approach — very similar to Steak in Rubyland I think

(tags: scala testing bdd acceptance-testing steak coding scalatest)

quotly/test/acceptance/adding_quotes_spec.rb at master · cavalle/quotly · GitHub

Decent demo of acceptance testing using rspec (and some syntactic sugar to make it read like Steak code, I think)

(tags: rspec acceptance-testing bdd testing ruby coding)

Facebook introduce “Wedge” and “FBOSS”

a new top-of-rack network switch, code-named “Wedge,” and a new Linux-based operating system for that switch, code-named “FBOSS.” These projects break down the hardware and software components of the network stack even further, to provide a new level of visibility, automation, and control in the operation of the network. By combining the hardware and software modules together in new ways, “Wedge” and “FBOSS” depart from current networking design paradigms to leverage our experience in operating hundreds of thousands of servers in our data centers. In other words, our goal with these projects was to make our network look, feel, and operate more like the OCP servers we’ve already deployed, both in terms of hardware and software.

(tags: cisco juniper wedge fboss facebook tor switches racks networking datacenter routers)

Cap’n Proto, FlatBuffers, and SBE

a feature comparison of these new serialization formats from Kenton, the capnp dude

(tags: serialization protobuf capnproto sbe flatbuffers google coding storage)

Concurrency Improvements in HyperLevelDB

Good-looking benchmark results here from HyperDex

(tags: hyperdex hyperleveldb leveldb rocksdb concurrency lock-free storage persistence)

Gumroad

“Sell products directly to your audience” — Ben says it doesn’t break the flow, doesn’t take you to another site, no complicated registration forms — the customer just enters CC details and that’s it.

(tags: payments ecommerce selling ebooks music ux)

A dive into a UTF-8 validation regexp

Once again, I find myself checking over the UTF-8 validation code in websocket-driver, and once again I find I cannot ever remember how to make sense of this regex that performs the validation. I just copied it off a webpage once and it took a while (and reimplementing UTF-8 myself) to fully understand what it does. If you write software that processes text, you’ll probably need to understand this too.

(tags: utf-8 unicode utf8 javascript node encoding text strings validation websockets regular-expressions regexps)

Whiteboard Picture Cleaner

This [shell one-liner] will take a picture of a whiteboard and use parts of the ImageMagick library with sane defaults to clean it up tremendously.: convert “$1” -morphology Convolve DoG:15,100,0 -negate -normalize -blur 0x1 -channel RBG -level 60%,91%,0.1 “$2”

(tags: graphics tools whiteboard imagemagick text images cleanup gimp photoshop via:fanf)

A Rare Peek Inside Amazon’s Massive Wish-Fulfilling Machine

Wired get a tour of PHX6, one of Amazon’s FCs

(tags: amazon wired fcs warehouses ecommerce)

Paleo is the Scientology of Diet

Being paleo is like paying a stupidity tax. Again, it’s not you who is stupid, but the diet sure is, because it lets you drink paleo coffee while putting paleo butter and paleo syrup on your paleo waffles before you drive your paleo minivan to the paleo office to sit in your paleo cube and do spreadsheets on your paleo computer. See, the paleo diet made up a bunch of silly rules on how we allegedly ate, and then goes and twists them all to hell in the name of selling you a crappy, overpriced product. That is scientology-level stupid.

(tags: scientology paleo rants funny food diet health bulletproof-coffee stupid)

Google’s Pegasus

a power-management subsystem for warehouse-scale computing farms. “It adjusts the power-performance settings of servers so that the overall workload barely meets its latency constraints for user queries.”

(tags: pegasus power-management power via:fanf google latency scaling)

FlatBuffers: Main Page

A new serialization format from Google’s Android gaming team, supporting C++ and Java, open source under the ASL v2. Reasons to use it:

Access to serialized data without parsing/unpacking – What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility). Memory efficiency and speed – The only memory needed to access your data is that of the buffer. It requires 0 additional allocations. FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details. Flexible – Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don’t have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don’t, and how you design data structures. Tiny code footprint – Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details. Strongly typed – Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you. Convenient to use – Generated C++ code allows for terse access & construction code. Then there’s optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).

(tags: c++ google java serialization json formats protobuf capnproto storage flatbuffers)

AWS SDK for Java Client Configuration

turns out the AWS SDK has lots of tuning knobs: region selection, socket buffer sizes, and debug logging (including wire logging).

(tags: aws sdk java logging ec2 s3 dynamodb sockets tuning)

Behind the loom band

The simple woven multicoloured bracelet has made Cheong Choon Ng, a Malaysian immigrant to the US, a dollar millionaire. He invented the “Rainbow Loom” after watching his daughters making bracelets with rubber bands.

(tags: via:conoro loom-bands rubber-bands toys crazes)

lookout/ngx_borderpatrol

BorderPatrol is an nginx module to perform authentication and session management at the border of your network. BorderPatrol makes the assumption that you have some set of services that require authentication and a service that hands out tokens to clients to access that service. You may not want those tokens to be sent across the internet, even over SSL, for a variety of reasons. To this end, BorderPatrol maintains a lookup table of session-id to auth token in memcached.

(tags: borderpatrol nginx modules authentication session-management web-services http web authorization)

Use of Formal Methods at Amazon Web Services

Chris Newcombe, Marc Brooker, et al. writing about their experience using formal specification and model-checking languages (TLA+) in production in AWS:

The success with DynamoDB gave us enough evidence to present TLA+ to the broader engineering community at Amazon. This raised a challenge; how to convey the purpose and benefits of formal methods to an audience of software engineers? Engineers think in terms of debugging rather than ‘verification’, so we called the presentation “Debugging Designs”. Continuing that metaphor, we have found that software engineers more readily grasp the concept and practical value of TLA+ if we dub it ‘Exhaustively-testable pseudo-code’. We initially avoid the words ‘formal’, ‘verification’, and ‘proof’, due to the widespread view that formal methods are impractical. We also initially avoid mentioning what the acronym ‘TLA’ stands for, as doing so would give an incorrect impression of complexity.

(tags: formal-methods model-checking tla tla+ programming distsys distcomp ebs s3 dynamodb aws ec2 marc-brooker chris-newcombe)

Call me maybe: RabbitMQ

We used Knossos and Jepsen to prove the obvious: RabbitMQ is not a lock service. That investigation led to a discovery hinted at by the documentation: in the presence of partitions, RabbitMQ clustering will not only deliver duplicate messages, but will also drop huge volumes of acknowledged messages on the floor. This is not a new result, but it may be surprising if you haven’t read the docs closely–especially if you interpreted the phrase “chooses Consistency and Partition Tolerance” to mean, well, either of those things.

(tags: rabbitmq network partitions failure cap-theorem consistency ops reliability distcomp jepsen)

Jump Consistent Hash: A Fast, Minimal Memory, Consistent Hash Algorithm

‘a fast, minimal memory, consistent hash algorithm that can be expressed in about 5 lines of code. In comparison to the algorithm of Karger et al., jump consistent hash requires no storage, is faster, and does a better job of evenly dividing the key space among the buckets and of evenly dividing the workload when the number of buckets changes. Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching.’ Implemented in Guava. This is also noteworthy: ‘Google has not applied for patent protection for this algorithm, and, as of this writing, has no plans to. Rather, it wishes to contribute this algorithm to the community.’

(tags: hashing consistent-hashing google guava memory algorithms sharding)

Bike Wheel Spoke ABS Safety Reflective Tube Reflector

Available in blue, orange, and grey for $2.84 from the insanely-cheap China-based DealExtreme.com. Also available: rim-based reflective stickers

(tags: bikes cycling reflective safety dealextreme tat)

#AltDevBlog » Parallel Implementations

John Carmack describes this code-evolution approach to adding new code:

The last two times I did this, I got the software rendering code running on the new platform first, so everything could be tested out at low frame rates, then implemented the hardware accelerated version in parallel, setting things up so you could instantly switch between the two at any time.  For a mobile OpenGL ES application being developed on a windows simulator, I opened a completely separate window for the accelerated view, letting me see it simultaneously with the original software implementation.  This was a very significant development win. If the task you are working on can be expressed as a pure function that simply processes input parameters into a return structure, it is easy to switch it out for different implementations.  If it is a system that maintains internal state or has multiple entry points, you have to be a bit more careful about switching it in and out.  If it is a gnarly mess with lots of internal callouts to other systems to maintain parallel state changes, then you have some cleanup to do before trying a parallel implementation. There are two general classes of parallel implementations I work with:  The reference implementation, which is much smaller and simpler, but will be maintained continuously, and the experimental implementation, where you expect one version to “win” and consign the other implementation to source control in a couple weeks after you have some confidence that it is both fully functional and a real improvement. It is completely reasonable to violate some generally good coding rules while building an experimental implementation – copy, paste, and find-replace rename is actually a good way to start.  Code fearlessly on the copy, while the original remains fully functional and unmolested.  It is often tempting to shortcut this by passing in some kind of option flag to existing code, rather than enabling a full parallel implementation.  It is a  grey area, but I have been tending to find the extra path complexity with the flag approach often leads to messing up both versions as you work, and you usually compromise both implementations to some degree.

(tags: via:marc coding john-carmack parallel development evolution lifecycle project-management)

5 Reasons to Use Protocol Buffers Instead of JSON For Your Next Service

A good writeup of the case for protobuf > JSON (via Marc)

(tags: via:marc api soa web-services protobuf json interop protocols marshalling)

Plumbr.eu’s reference page for java.lang.OutOfMemoryError

With examples of each possible cause of a Java OOM, and suggested workarounds. succinct

(tags: reference plumbr.eu oom java memory heap ops)

SSD shadiness: Kingston and PNY caught bait-and-switching cheaper components after good reviews | ExtremeTech

Imagine buying a high-end Core i7 or AMD CPU, opening the box, and finding a midrange part sitting there with an asterisk and the label “Performs Just Like Our High End CPU In Single-Threaded SuperPi!”

(tags: ssd storage hardware sketchy kingston pny bait-and-switch components vendors via:hn)

Pillar

Manages migrations for your Cassandra data stores. Pillar grew from a desire to automatically manage Cassandra schema as code. Managing schema as code enables automated build and deployment, a foundational practice for an organization striving to achieve Continuous Delivery. Pillar is to Cassandra what Rails ActiveRecord migrations or Play Evolutions are to relational databases with one key difference: Pillar is completely independent from any application development framework.

(tags: migrations database ops pillar cassandra activerecord scala continuous-delivery automation build)

How to use TuneIn’s Record Timer feature

handy

(tags: tunein radio recording podcasts mp3 apps)

Continuous Deployment for Mobile Apps with Jenkins: iOS Builds

the CloudBees-std way

(tags: build deployment ios jenkins iphone continuous-deployment)

The Game on Google Glass

‘A tribute to the addictive Ktarian Game featured in Season 5, Episode 6 of Star Trek, The Next Generation.’

(tags: star-trek google-glass the-game games glass google tv)

Tracedump

a single application IP packet sniffer that captures all TCP and UDP packets of a single Linux process. It consists of the following elements: * ptrace monitor – tracks bind(), connect() and sendto() syscalls and extracts local port numbers that the traced application uses; * pcap sniffer – using information from the previous module, it captures IP packets on an AF_PACKET socket (with an appropriate BPF filter attached); * garbage collector – periodically reads /proc/net/{tcp,udp} files in order to detect the sockets that the application no longer uses. As the output, tracedump generates a PCAP file with SLL-encapsulated IP packets – readable by eg. Wireshark. This file can be later used for detailed analysis of the networking operations made by the application. For instance, it might be useful for IP traffic classification systems.

(tags: debugging networking linux strace ptrace tracedump tracing tcp udp sniffer ip tcpdump)

You Are Not a Digital Native: Privacy in the Age of the Internet

an open letter from Cory Doctorow to teen readers re privacy. ‘The problem with being a “digital native” is that it transforms all of your screw-ups into revealed deep truths about how humans are supposed to use the Internet. So if you make mistakes with your Internet privacy, not only do the companies who set the stage for those mistakes (and profited from them) get off Scot-free, but everyone else who raises privacy concerns is dismissed out of hand. After all, if the “digital natives” supposedly don’t care about their privacy, then anyone who does is a laughable, dinosauric idiot, who isn’t Down With the Kids.’

(tags: children privacy kids teens digital-natives surveillance cory-doctorow danah-boyd)

Shutterbits replacing hardware load balancers with local BGP daemons and anycast

Interesting approach. Potentially risky, though — heavy use of anycast on a large-scale datacenter network could increase the scale of the OSPF graph, which scales exponentially. This can have major side effects on OSPF reconvergence time, which creates an interesting class of network outage in the event of OSPF flapping. Having said that, an active/passive failover LB pair will already announce a single anycast virtual IP anyway, so, assuming there are a similar number of anycast IPs in the end, it may not have any negative side effects. There’s also the inherent limitation noted in the second-to-last paragraph; ‘It comes down to what your hardware router can handle for ECMP. I know a Juniper MX240 can handle 16 next-hops, and have heard rumors that a software update will bump this to 64, but again this is something to keep in mind’. Taking a leaf from the LB design, and using BGP to load-balance across a smaller set of haproxy instances, would seem like a good approach to scale up.

(tags: scalability networking performance load-balancing bgp exabgp ospf anycast routing datacenters scaling vips juniper haproxy shutterstock)

Tron: Legacy Encom Boardroom Visualization

this is great. lovely, silly, HTML5 dataviz, with lots of spinning globes and wobbling sines on a black background

(tags: demo github wikipedia dataviz visualisation mapping globes rob-scanlan graphics html5 animation tron-legacy tron movies)

CockroachDB

a distributed key/value datastore which supports ACID transactional semantics and versioned values as first-class features. The primary design goal is global consistency and survivability, hence the name. Cockroach aims to tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention. Cockroach nodes are symmetric; a design goal is one binary with minimal configuration and no required auxiliary services. Cockroach implements a single, monolithic sorted map from key to value where both keys and values are byte strings (not unicode). Cockroach scales linearly (theoretically up to 4 exabytes (4E) of logical data). The map is composed of one or more ranges and each range is backed by data stored in RocksDB (a variant of LevelDB), and is replicated to a total of three or more cockroach servers. Ranges are defined by start and end keys. Ranges are merged and split to maintain total byte size within a globally configurable min/max size interval. Range sizes default to target 64M in order to facilitate quick splits and merges and to distribute load at hotspots within a key range. Range replicas are intended to be located in disparate datacenters for survivability (e.g. { US-East, US-West, Japan }, { Ireland, US-East, US-West}, { Ireland, US-East, US-West, Japan, Australia }). Single mutations to ranges are mediated via an instance of a distributed consensus algorithm to ensure consistency. We’ve chosen to use the Raft consensus algorithm. All consensus state is stored in RocksDB. A single logical mutation may affect multiple key/value pairs. Logical mutations have ACID transactional semantics. If all keys affected by a logical mutation fall within the same range, atomicity and consistency are guaranteed by Raft; this is the fast commit path. Otherwise, a non-locking distributed commit protocol is employed between affected ranges. Cockroach provides snapshot isolation (SI) and serializable snapshot isolation (SSI) semantics, allowing externally consistent, lock-free reads and writes–both from an historical snapshot timestamp and from the current wall clock time. SI provides lock-free reads and writes but still allows write skew. SSI eliminates write skew, but introduces a performance hit in the case of a contentious system. SSI is the default isolation; clients must consciously decide to trade correctness for performance. Cockroach implements a limited form of linearalizability, providing ordering for any observer or chain of observers.

(tags: cockroachdb databases storage georeplication raft consensus acid go key-value-stores rocksdb)

Tuning LevelDB

good docs from Riak

(tags: leveldb tuning performance ops riak)

Proof of burn – Bitcoin

method for bootstrapping one cryptocurrency off of another. The idea is that miners should show proof that they burned some coins – that is, sent them to a verifiably unspendable address. This is expensive from their individual point of view, just like proof of work; but it consumes no resources other than the burned underlying asset. To date, all proof of burn cryptocurrencies work by burning proof-of-work-mined cryptocurrencies, so the ultimate source of scarcity remains the proof-of-work-mined “fuel”.

(tags: bitcoin proof money mining cryptocurrency)

The programming error that cost Mt Gox 2609 bitcoins

Digging into broken Bitcoin scripts in the blockchain. Fascinating:

While analyzing coinbase transactions, I came across another interesting bug that lost bitcoins. Some transactions have the meaningless and unredeemable script: OP_IFDUP OP_IF OP_2SWAP OP_VERIFY OP_2OVER OP_DEPTH That script turns out to be the ASCII text script. Instead of putting the redemption script into the transaction, the P2Pool miners accidentally put in the literal word “script”. The associated bitcoins are lost forever due to this error.

(tags: programming script coding bitcoin mtgox via:nelson scripting dsls)

Moquette MQTT

a Java implementation of an MQTT 3.1 broker. Its code base is small. At its core, Moquette is an events processor; this lets the code base be simple, avoiding thread sharing issues. The Moquette broker is lightweight and easy to understand so it could be embedded in other projects.

(tags: mqtt moquette netty messaging queueing push-notifications iot internet push eclipse)

“Taking the hotdog”

aka. lock acquisition. ex-Amazon-Dublin lingo, observed in the wild ;)

(tags: language hotdog archie-mcphee amazon dublin intercom coding locks synchronization)

Organic Cat Litter Chief Suspect In Nuclear Waste Accident

What a headline. interesting story to boot (via Eoin)

(tags: environment energy chemistry cat-litter waste-disposal nuclear-waste accidents new-mexico)

Friends don’t let friends use mmap(2)

Rather horrific update from the trenches of Mozilla

(tags: mozilla mmap performance linux io files memory unix windows)

67 Books Every Geek Should Read to Their Kids Before Age 10 | GeekDad | Wired.com

Lots and lots of good book recommendations, a little US-centric though

(tags: reading books kids children education fiction development)

How the patent trolls won in Congress: Ars Technica

“We felt really good the last couple of days,” said the tech lobbyist. “It was a good deal—one we could live with. Then the trial lawyers and pharma went to Senator Reid late this morning and said that’s it. Enough with the children playing in the playground—go kill it.”

(tags: ars-technica patents swpats patent-trolls pharma tech us-politics congress lawyers)

Dublin City North Inner City count results, animated

A nice visualisation of Single-Transferable-Vote proportional representation in action

(tags: pr-stv voting dataviz visualisation dublin elections pr)

New Statesman: Let’s call the Isla Vista killings what they were: misogynist extremism

We have been told for a long time that the best way to deal with this sort of harrassment and violence is to laugh it off. Women and girls and queer people have been told that online misogynists pose no real threat, even when they’re sharing intimate guides to how to destroy a woman’s self-esteem and force her into sexual submission. Well, now we have seen what the new ideology of misogyny looks like at its most extreme. We have seen incontrovertible evidence of real people being shot and killed in the name of that ideology, by a young man barely out of childhood himself who had been seduced into a disturbing cult of woman-hatred. Elliot Rodger was a victim – but not for the reasons he believed.

(tags: elliot-rodger extremism feminism isla-vista mass-killings pua mens-rights harrassment misogyny penny-red)

BPF – the forgotten bytecode

‘In essence Tcpdump asks the kernel to execute a BPF program within the kernel context. This might sound risky, but actually isn’t. Before executing the BPF bytecode kernel ensures that it’s safe: * All the jumps are only forward, which guarantees that there aren’t any loops in the BPF program. Therefore it must terminate. * All instructions, especially memory reads are valid and within range. * The single BPF program has less than 4096 instructions. All this guarantees that the BPF programs executed within kernel context will run fast and will never infinitely loop. That means the BPF programs are not Turing complete, but in practice they are expressive enough for the job and deal with packet filtering very well.’ Good example of a carefully-designed DSL allowing safe “programs” to be written and executed in a privileged context without security risk, or risk of running out of control.

(tags: coding dsl security via:oisin linux tcpdump bpf bsd kernel turing-complete configuration languages)

Handmade Kitchen Goods from Makers & Brothers – Cool Hunting

lovely kitchen-gear design from local-boys-made-good Makers & Brothers

(tags: makers-and-brothers design crafts kitchen nyc terrazo chopping-boards)

‘Monitoring and detecting causes of failures of network paths’, US patent 8,661,295 (B1)

The first software patent in my name — couldn’t avoid it forever :(

Systems and methods are provided for monitoring and detecting causes of failures of network paths. The system collects performance information from a plurality of nodes and links in a network, aggregates the collected performance information across paths in the network, processes the aggregated performance information for detecting failures on the paths, analyzes each of the detected failures to determine at least one root cause, and initiates a remedial workflow for the at least one root cause determined. In some aspects, processing the aggregated information may include performing a statistical regression analysis or otherwise solving a set of equations for the performance indications on each of a plurality of paths. In another aspect, the system may also include an interface which makes available for display one or more of the network topology, the collected and aggregated performance information, and indications of the detected failures in the topology.

(tags: amazon pimms swpats patents networking ospf autoremediation outage-detection)

Dublin City Council rows back on speed bumps for cyclists

“bicycle-calming measures”. FFS, DCC

(tags: idiots dublin dcc council cycling fail holland funny bicycle-calming)

Monitoring Reactive Applications with Kamon

“quality monitoring tools for apps built in Akka, Spray and Play!”. Uses Gil Tene’s HDRHistogram and dropwizard Metrics under the hood.

(tags: metrics dropwizard hdrhistogram gil-tene kamon akka spray play reactive statistics java scala percentiles latency)

Structs

storage of structured data in a continuous block of memory. The memory can be allocated on the heap using a byte[] array or can be allocated off the java heap in native memory. […] Use cases: store/cache huge amounts of data records without impact on GC duration; high performance data transfer in a cluster or in between processes

(tags: structs java jvm memory off-heap storage reference)

Dynamic Tuple Performance On the JVM

More JVM off-heap storage from Boundary:

generates heterogeneous collections of primitive values and ensures as best it can that they will be laid out adjacently in memory. The individual values in the tuple can either be accessed from a statically bound interface, via an indexed accessor, or via reflective or other dynamic invocation techniques. FastTuple is designed to deal with a large number of tuples therefore it will also attempt to pool tuples such that they do not add significantly to the GC load of a system. FastTuple is also capable of allocating the tuple value storage entirely off-heap, using Java’s direct memory capabilities.

(tags: jvm java gc off-heap storage boundary memory)

Learn.code.org

Teaches the basics of computer science – K-8 Intro to CS, 15-25 hours. Introduces core CS and programming concepts, with lots of nice graphics, scenarios and characters from games to get the kids hooked ;) Recommended by Tom Raftery; his youngest (7yo) is having great fun with it.

(tags: education programming learning coding kids k-8 code.org games)

The leaked New York Times innovation report is one of the key documents of this media age » Nieman Journalism Lab

one of the world’s leading news organizations giving itself a rigorous self-examination. I’ve spoken with multiple digital-savvy Times staffers in recent days who described the report with words like “transformative” and “incredibly important” and “a big big moment for the future of the Times.” One admitted crying while reading it because it surfaced so many issues about Times culture that digital types have been struggling to overcome for years.

(tags: nytimes publishing media journalism tech internet web news leaks via:antoin)

Microsoft Security Essentials reporting false positives on the Bitcoin blockchain

Earlier today, a virus signature from the virus “DOS/STONED” was uploaded into the Bitcoin blockchain, which allows small snippets of text to accompany user transactions with bitcoin.  Since this is only the virus signature and not the virus itself, there apparently is no danger to users in any way.  However, MSE recognizes the signature for the virus and continuously reports it as a threat, and every time it deletes the file, the bitcoin client will simply re-download the missing blockchain.

(tags: via:gwire av antivirus false-positives fp blockchain microsoft bitcoin pranks viruses)

Stuck in the iMessage abyss? Here’s how to get your texts back

some potential (apocryphal) workarounds for this extremely annoying Apple bug

(tags: apple bugs imessage sms phones mobile android hacks)

Spark Streaming

an extension of the core Spark API that allows enables high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets and be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s in-built machine learning algorithms, and graph processing algorithms on data streams.

(tags: spark streams stream-processing cep scalability apache machine-learning graphs)

“Crypto Won’t Save You Either”

fantastic slides from Peter Gutmann

(tags: crypto cryptography security exploits nsa gchq dual_ec_drbg rsa)

In the Zone: How Gamers Experience The Real Chernobyl

Great article. I’ve experienced this in LA, particularly, after playing the GTA series

(tags: pripyat gaming tourism reality virtual-reality via:oceanclub games)

Shapecatcher: Draw the Unicode character you want!

‘This is a tool to help you find Unicode characters. Finding a specific character whose name you don’t know is cumbersome. On shapecatcher.com, all you need to know is the shape of the character!’ Handy.

(tags: shapes drawing unicode characters language recognition web)

Unchi-kun Candy – Japanese Lucky Poop Candy

What doesn’t look like Christmas more than a smiling piece of poop, called unchi in Japanese? Because the shape of unchi looks similar to that of mochi used for shrine offerings, and because the sound “unchi” like the Japanese word for luck, this treat is actually a lucky gift — at least that is how you can explain yourself when you give it as a gift. Each Unchi-kun comes packed with poop candy, taken out from the bottom. Once finished eating, you can open the slot in the back with a box-cutter and turn it into a bank.

(tags: unchi-kun unchi pile-of-poo emoji unicode cute funny japan j-list sweets food gross candy)

Bletchley Park Trust erects “Berlin Wall” to cut off on-site computer history museum – Boing Boing

The Bletchley Park trust have erected a fence, nicknamed “The Berlin Wall,” between their well-funded museum and its poorer on-site neighbour, the UK National Museum of Computing, which houses the hand-built replica of the codebreaking Colossus computer. The trust received an £8m lottery-funded grant and set about shitcanning long-serving volunteers, cutting off the computer history museum, and generally behaving like greedy jerks, systematically alienating long-term supporters. Oh, and there was that Snowden business.

(tags: bletchley-park history wankers uk museums computing)

Exceptional Performance

Good benchmark data on the performance of JVM exceptions

(tags: java jvm exceptions benchmarking performance optimization coding)

iMessage purgatory

Oh Apple, you asshats. This is some seriously shitty programming. iMessage on iOS devices caches the “iMessage-capable” flag for all numbers, indefinitely, so if you switch from iPhone to Android, messages from your friends’ iPhones won’t get delivered to you henceforth — and to add insult to injury, it claims they do with a “Delivered.” status appearing under the message. This is happening to me right now…

(tags: apple sms messaging phones mobile imessage android fail bad-programming bugs)

SpideyApp

an Android-based stingray (IMSI catcher) detector that uses machine learning to detect the presence of stingray devices which can be used to eavesdrop on cellular communication.

(tags: imsi-catcher stingray surveillance via:ethanz snooping spying privacy mobile)

Google Maps Has Forsaken Us

+1. What has happened at Google? Did they fire ever employee in the UX department?

(tags: ux ui google gmaps mapping web techcrunch)

Spark – A small web framework for Java

A Sinatra-like minimal web framework built on Java 8 lambdas: public class HelloWorld { public static void main(String[] args) { get("/hello", (request, response) -> { return "Hello World!"; }); } }

(tags: via:sampullara web java sinatra lambdas closures java8 spark)

Building a Global, Highly Available Service Discovery Infrastructure with ZooKeeper

This is the written version of a presentation [Camille Fournier] made at the ZooKeeper Users Meetup at Strata/Hadoop World in October, 2012 (slides available here). This writeup expects some knowledge of ZooKeeper.

(tags: zookeeper service-discovery architecture distcomp camille-fournier availability wan network)

“Replicated abstract data types: Building blocks for collaborative applications”

cited at https://news.ycombinator.com/item?id=7737423 as ‘one of my favorite papers on CRDTs and provides practical pseudocode for learning how to implement CRDTs yourself’, in a discussion on cemerick’s “Distributed Systems and the End of the API”: http://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/

(tags: distcomp networking distributed crdts algorithms text data-structures cap)

Android 4.4 KitKat Problems: HTC UK Speaks Out About HTC One Update Issues

Good advice on improving battery life with the KitKat 4.4.2 point release on a HTC One. I had serious battery problems, but they seem to have been resolved by following this advice

(tags: wifi kitkat android 4.4.2 htc-one htc battery blinkfeed mobile phones)

Transform any text into a patent application

‘An apparatus and device for staring into vacancy. The devices comprises a good cage, a narrow gangway, an electric pocket, a flower-bedecked cage, an insensitive felt.’ (The Hunger Artist by Kafka)

(tags: python patents text language generator)

What have you made with Llama? : Android

nice tips. this is a great feature!

(tags: llama locale location android phones mobile hacking)

The Big List of Local Multiplayer Games

authoritative!

(tags: couch games gaming local-multiplayer multiplayer)

moto

Mock Boto: ‘a library that allows your python tests to easily mock out the boto library.’ Supports S3, Autoscaling, EC2, DynamoDB, ELB, Route53, SES, SQS, and STS currently, and even supports a standalone server mode, to act as a mock service for non-Python clients. Excellent! (via Conor McDermottroe)

(tags: python aws testing mocks mocking system-tests unit-tests coding ec2 s3)

Why Disqus made the Python->Go switchover

for their realtime component, from the horse’s mouth:

at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.

(tags: python languages concurrency go threading gevent scalability disqus realtime hn)

Database Migrations Done Right

The rule is simple. You should never tie database migrations to application deploys or vice versa. By minimising dependencies you enable faster, easier and cleaner deployments.

(tags: migrations database sql mysql postgres deployment ops dependencies loose-coupling)

Julie Moon | Magic Pony

some cute brooches/jewellery here, for the next time I need to pick a nice gift

(tags: julie-moon art magic-pony jewellery brooches gifts)

Building a large scale CDN with Apache Traffic Server

via Ilya Grigorik: ‘Great under-the-hood look at how Comcast built and operates their internal CDN for delivering video (on-demand + live). Some highlights: switched to own (open-source) stack; ~250 servers pushing ~1.5Pb of data/day with ~5Pb of storage capacity.’

(tags: cdn comcast video presentations apache traffic-server vod)

An analysis of Facebook photo caching

excellent analysis of caching behaviour at scale, from the FB engineering blog (via Tony Finch)

(tags: via:fanf caching facebook architecture photos images cache fifo lru scalability)

SF for Londoners

good advice. next time I go over, I’ll have to get a Clipper card. Also: ‘Brunch is its own section because I have never encountered a place that takes brunch so seriously.’

(tags: brunch sf travel california tips san-francisco clipper-card)

Alexey Shipilev on Java’s System.nanoTime()

System.nanoTime is as bad as String.intern now: you can use it, but use it wisely. The latency, granularity, and scalability effects introduced by timers may and will affect your measurements if done without proper rigor. This is one of the many reasons why System.nanoTime should be abstracted from the users by benchmarking frameworks, monitoring tools, profilers, and other tools written by people who have time to track if the underlying platform is capable of doing what we want it to do. In some cases, there is no good solution to the problem at hand. Some things are not directly measurable. Some things are measurable with unpractical overheads. Internalize that fact, weep a little, and move on to building the indirect experiments. This is not the Wonderland, Alice. Understanding how the Universe works often needs side routes to explore. In all seriousness, we should be happy our $1000 hardware can measure 30 nanosecond intervals pretty reliably. This is roughly the time needed for the Internet packets originating from my home router to leave my apartment. What else do you want, you spoiled brats?

(tags: benchmarking jdk java measurement nanoseconds nsecs nanotime jvm alexey-shipilev jmh)

Unsupervised machine learning

aka. “zero-shot learning”. ok starting point

(tags: machine-learning zero-shot unsupervised algorithms ml)

Minimum Viable Block Chain

Ilya Grigorik describes the design of the Bitcoin/altcoin block chain algorithm. Illuminating writeup

(tags: algorithms bitcoin security crypto blockchain ilya-grigorik)

Docker Plugin for Jenkins

The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave. Optionally, the container can be committed, so that (for example) manual QA could be performed by the container being imported into a local docker provider, and run from there.

(tags: jenkins docker ops testing ec2 hosting scaling elastic-scaling system-testing)

Simple Binary Encoding

an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. […] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message. The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer. The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations. The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again. The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.

(tags: sbe encoding protobuf protocol-buffers json messages messaging binary formats low-latency martin-thompson xml)

Observations of an Internet Middleman

That leaves the remaining six [consumer ISPs peering with Level3] with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content. Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.

(tags: net-neutrality networking internet level3 congestion isps us-politics)

interview with Google VP of SRE Ben Treynor

interviewed by Niall Murphy, no less ;). Some good info on what Google deems important from an ops/SRE perspective

(tags: sre ops devops google monitoring interviews ben-treynor)

Faster BAM Sorting with SAMtools and RocksDB

Now this is really really clever. Heap-merging a heavyweight genomics format, using RocksDB to speed it up.

There’s a problem with the single-pass merge described above when the number of intermediate files, N/R, is large. Merging the sorted intermediate files in limited memory requires constantly reading little bits from all those files, incurring a lot of disk seeks on rotating drives. In fact, at some point, samtools sort performance becomes effectively bound to disk seeking. […] In this scenario, samtools rocksort can sort the same data in much less time, using no more memory, by invoking RocksDB’s background compaction capabilities. With a few extra lines of code we configure RocksDB so that, while we’re still in the process of loading the BAM data, it runs additional background threads to merge batches of existing sorted temporary files into fewer, larger, sorted files. Just like the final merge, each background compaction requires only a modest amount of working memory.

(tags: rocksdb algorithms sorting leveldb bam samtools merging heaps compaction)

Coding For Life (Battery Life, That Is)

great presentation on Android mobile battery life, and what to avoid

(tags: presentations via:sergio android mobile battery battery-life 3g wifi gprs hardware)

Oisin’s mobile app release checklist

‘This form is to document the testing that has been done on each app version before submitting to the App Store. For each item, indicate Yes if the testing has been done, Not Applicable if the testing does not apply (eg testing audio for an app that doesn’t play any), or No if the testing has not been done for another reason.’

(tags: apps checklists release coding ios android mobile ohurley)

“A New Data Structure For Cumulative Frequency Tables”

paper by Peter M Fenwick, 1993. ‘A new method (the ‘binary indexed tree’) is presented for maintaining the cumulative frequencies which are needed to support dynamic arithmetic data compression. It is based on a decomposition of the cumulative frequencies into portions which parallel the binary representation of the index of the table element (or symbol). The operations to traverse the data structure are based on the binary coding of the index. In comparison with previous methods, the binary indexed tree is faster, using more compact data and simpler code. The access time for all operations is either constant or proportional to the logarithm of the table size. In conjunction with the compact data structure, this makes the new method particularly suitable for large symbol alphabets.’ via Jakob Buchgraber, who’s implementing it right now in Netty ;)

(tags: netty frequency-tables data-structures algorithms coding binary-tree indexing compression symbol-alphabets)

Actually, Mr. Waxman, Consumers Are Sued For Patent Infringement All the Time | Electronic Frontier Foundation

Patent trolls have sued or threatened to sue tens of thousands of end-users. For example, Innovatio attacked cafes, bakeries, and even a funeral parlor for using off-the-shelf Wi-Fi routers. And the notorious scanner troll, MPHJ, targeted small businesses and nonprofits around the country for using ordinary office equipment. As a recent paper explained: “Mass suits against technology customers have become too common, involving building block technologies like wi-fi, scanning, email and website technologies.” The growth in patent suits against customers reveals the importance of the Limelight case. A ruling that made it even easier to sue customers (by allowing suits against someone who performs just some steps of a patent) would encourage patent trolls to launch more abusive litigation campaigns. We hope the Supreme Court will restore the sensible rule that only a single entity (or its agents) can infringe a patent.

(tags: patents uspto swpats eff consumer law legal patent-infringement scanners wifi printers)

Hanging on the telephone – has anyone got it right on the new ban on text driving?

Some good legal commentary on this new Irish law.

There has been much hand-wringing and concern about whether or not the 2014 Regulations prohibit the use of Google Maps or Hailo, for example. They don’t, but this does not mean that drivers should feel free to use non-texting functions of their phones while driving – holding a mobile phone (which could include a tablet) while driving remains prohibited, whatever the use it is being put to. Moreover, offences of dangerous and careless driving and driving without due care and attention could cover a wide range of bad driving, and could include, for example, driving while zooming in and out of maps on your phone or sending stickers on WhatsApp.

(tags: ireland law driving safety mobile-phones texting google-maps satnav)

Arrow

‘better dates and times for Python’, to fix the absurd proliferation of slightly-incompatible Python date/time types and APIs. unfortunately, http://imgs.xkcd.com/comics/standards.png applies….

(tags: python libraries time dates timestamps timezones apis proliferation iso-8601)

Holdings: Guinness’s Brewery Dublin

‘Guinness’s Brewery Dublin. Malt House, malt on floor; sign’ – One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection

(tags: nli ireland photos t-h-mason history dublin guinness maltings beer)

Published image: ‘An Irish Village’.

‘Cart, man/woman; 2 men and boy serving beer outside, + sign ‘Rich King Spirits’. Ragged attire’ – One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection

(tags: ireland history poverty t-h-mason photos)

Cell Development

One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection

(tags: ireland history science biology t-h-mason photos)

Composition of crystals

One of the photos taken by my great-grandfather, Thomas H. Mason, around the turn of the century from the NLI collection

(tags: ireland history science chemistry crystals t-h-mason photos)

SmartStack vs. Consul

One of the SmartStack developers at AirBNB responds to Consul.io’s comments. FWIW, we use SmartStack in Swrve and it works pretty well…

(tags: smartstack airbnb ops consul serf load-balancing availability resiliency network-partitions outages)

A Closer Look At OC’s Anti-Vaccination Cluster

In communities such as San Clemente, Laguna Beach, Laguna Niguel, Aliso Viejo, Mission Viejo and Capistrano Beach, where Dr. Bob Sears practices, there are clusters of unvaccinated children. Last year, at 15 of the 40 elementary schools in the Capistrano Unified School District, more than 10 percent of kindergartners had [Personal Belief exemptions], according to data from the California Department of Public Health. At one public charter school, Journey, 56 percent of kindergartners were unvaccinated, at least partially, due to their parents’ beliefs.

(tags: orange-county health vaccination laguna-beach oc dr-bob-sears kindergarten measles mumps rubella pertussis epidemiology)

Pinterest Secor

Today we’re open sourcing Secor, a zero data loss log persistence service whose initial use case was to save logs produced by our monetization pipeline. Secor persists Kafka logs to long-term storage such as Amazon S3. It’s not affected by S3’s weak eventual consistency model, incurs no data loss, scales horizontally, and optionally partitions data based on date.

(tags: pinterest hadoop secor storm kafka architecture s3 logs archival)

Coping with the TCP TIME-WAIT state on busy Linux servers

extensive blog post

(tags: networking linux tcp performance time-wait sysctls tuning)

Flood IO Offering Network Emulation

Performance-testing-as-a-service company Flood.IO now offering emulation of various crappy end-user networks: GSM, DSL, GPRS, 3G, 4G etc. Great idea.

(tags: flood.io performance networking internet load-testing testing jmeter gatling tests gsm 3g mobile simulation)

Scaling Realtime at DISQUS

Disqus’ realtime architecture — nginx PushStream module doing the heavy lifting, basically. See https://gist.github.com/dctrwatson/0b3b52050254e273ff11 for the production nginx configs they use. I am very impressed that push-stream has grown to be so solid; it’s a great way to deal with push from the sounds of it. http://blog.disqus.com/post/51155103801/trying-out-this-go-thing now notes that some of the realtime backends are in Go. https://speakerdeck.com/dctrwatson/c1m-and-nginx (“C1M and Nginx”) is a more up to date presentation. It notes that PushStream supports “EventSource, WebSocket, Long Polling, and forever iframe”. More sysctls and nginx tuning in that prez.

(tags: sysctl nginx tuning go disqus realtime push eventsource websockets long-polling iframe python)

‘The Design And Implementation Of Modern Column-Oriented Database Systems’

paper, PDF; Daniel Abadi et al.

(tags: papers pdf columnar-stores column-oriented databases storage architecture algorithms)

‘Pickles & Spores: Improving Support for Distributed Programming in Scala

‘Spores are “small units of possibly mobile functional behavior”. They’re a closure-like abstraction meant for use in distributed or concurrent environments. Spores provide a guarantee that the environment is effectively immutable, and safe to ship over the wire. Spores aim to give library authors some confidence in exposing functions (or, rather, spores) in public APIs for safe consumption in a distributed or concurrent environment. The first part of the talk covers a simpler variant of spores as they are proposed for inclusion in Scala 2.11. The second part of the talk briefly introduces a current research project ongoing at EPFL which leverages Scala’s type system to provide type constraints that give authors finer-grained control over spore capturing semantics. What’s more, these type constraints can be composed during spore composition, so library authors are effectively able to propagate expert knowledge via these composable constraints. The last part of the talk briefly covers Scala/Pickling, a fast new, open serialization framework.’

(tags: pickling scala presentations spores closures fp immutability coding distributed distcomp serialization formats network)

BBC News – Microsoft ‘must release’ data held on Dublin server

Messy. I can’t see this lasting beyond an appeal.

Law enforcement efforts would be seriously impeded and the burden on the government would be substantial if they had to co-ordinate with foreign governments to obtain this sort of information from internet service providers such as Microsoft and Google, Judge Francis said. In a blog post, Microsoft’s deputy general counsel, David Howard, said: “A US prosecutor cannot obtain a US warrant to search someone’s home located in another country, just as another country’s prosecutor cannot obtain a court order in her home country to conduct a search in the United States. “We think the same rules should apply in the online world, but the government disagrees.”

(tags: microsoft regions law us-law privacy google cloud international-law surveillance)

Russia passes bill requiring bloggers to register with government

A bill passed by the Russian parliament on Tuesday says that any blogger read by at least 3,000 people a day has to register with the government telecom watchdog and follow the same rules as those imposed by Russian law on mass media. These include privacy safeguards, the obligation to check all facts, silent days before elections and loose but threatening injunctions against “abetting terrorism” and “extremism.”

(tags: russia blogs blogging terrorism extremism internet regulation chilling-effects censorship)

Rope-core memory

as used in the Apollo guidance computer systems — hand-woven by “little old ladies”. Amazing

(tags: core-memory memory rope-core guidance apollo space nasa history 1960s via:hn)

How I used Heartbleed to steal a site’s private crypto key

good writeup from Robin Xu

(tags: robin-xu heartbleed rsa private-keys openssl hacking security tls ssl)

Making runbooks more useful by exposing them through monitoring

Nice example of an ops runbook wiki for a service

(tags: runbooks ops devops monitoring sysadmin documentation wiki)

AWS Case Study: Hailo

Ubuntu, C*, HAProxy, MySQL, RDS, multiple AWS regions.

(tags: hailo cassandra ubuntu mysql rds aws ec2 haproxy architecture)

They called it “big iron” for a reason: the Cray Motor-Generator Unit

I think the deal with the Motor-Generator Unit was that the Cray 1 needed not just enormous amounts of power (over a hundred kilowatts!), but also very stable power. So it ran from a huge electric generator connected directly to a huge electric motor, the motor running from dirty grid power and the generator, in turn, feeding the computer’s own multi-voltage PSU. The Cray 1 itself weighed a mere 2.4 tonnes, but all this support stuff added several more tonnes.

(tags: via:rob-synnott cray history big-iron motors power electricity generators)

Go: Best Practices for Production Environments

how Soundcloud deploy their Go services, after 2.5 years of Go in production

(tags: go tips deployment best-practices soundcloud ops)

Eircode

‘Location Codes for Irish Addresses’. Looks like, as expected, this will not have no-cost licensing terms; companies and non-profit orgs will all have to pay Capita Business Support Services Ireland for access. boo.

(tags: eircode mapping addressing geocoding ireland open-source licensing postcodes)

Why your company shouldn’t use Git submodules

‘It is not uncommon at all when working on any kind of larger-scale project with Git to find yourself wanting to share code between multiple different repositories – whether it be some core system among multiple different products built on top of that system, or perhaps a shared utility library between projects. At first glance, Git submodules seem to be the perfect answer for this: they come built-in with Git, they act like miniature repositories (so people are already familiar with how to change them), et cetera. They even support pointing at specific versions of the shared code, so if one project doesn’t want to deal with integrating the “latest and greatest” version, it doesn’t have to. It’s after you’ve actually worked with submodules for a while that you start to notice just how half-baked Git’s submodules system really is.’

(tags: git source-control revision-control submodules storage)

Eyes Over Compton: How Police Spied on a Whole City

The law-enforcement pervasive-surveillance CCTV PVR.

In a secret test of mass surveillance technology, the Los Angeles County Sheriff’s Department sent a civilian aircraft* over Compton, California, capturing high-resolution video of everything that happened inside that 10-square-mile municipality. Compton residents weren’t told about the spying, which happened in 2012. “We literally watched all of Compton during the times that we were flying, so we could zoom in anywhere within the city of Compton and follow cars and see people,” Ross McNutt of Persistence Surveillance Systems told the Center for Investigative Reporting, which unearthed and did the first reporting on this important story. The technology he’s trying to sell to police departments all over America can stay aloft for up to six hours. Like Google Earth, it enables police to zoom in on certain areas. And like TiVo, it permits them to rewind, so that they can look back and see what happened anywhere they weren’t watching in real time. 

(tags: pvr cctv law-enforcement police compton los-angeles law surveillance future)

Resiliency And Elasticsearch

Blog post from the ES team. They use “evil tests” — basically unit/system tests, particularly using randomized error-injecting mock infrastructure. Good practices; I’ve done the same myself quite recently for Swrve’s realtime infrastructure

(tags: elasticsearch resiliency network-partitions reliability testing mocking error-injection)

Meet Ireland’s first bitcoin politician

Ossian Smyth — Green Party internet spokesman and representative for communications, energy, and natural resources, with a top wheeze: “I think it is one of the most transparent ways of receiving donations. No one would know how much money can be donated into a bank account, but with bitcoin anyone can go to the block chain and look at the wallet.” excellent ;)

(tags: ossian-smyth bitcoin fundraising greens politics ireland dublin green-party internet)

OpenPostCode demolishes the planned “Eircode” postcode system

Comprehensively ripped to shreds. Bottom line: ‘Postcodes will be largely meaningless to anyone without access to the pay-walled database. It is another tax on business.’

(tags: postcodes ireland eircode addressing geocoding mapping maps open-data)

Netflix comes out strongly against Comcast

In sum, Comcast is not charging Netflix for transit service. It is charging Netflix for access to its subscribers. Comcast also charges its subscribers for access to Internet content providers like Netflix. In this way, Comcast is double dipping by getting both its subscribers and Internet content providers to pay for access to each other.

(tags: netflix comcast network-neutrality cartels competition us-politics business isps)

co-founder of the Boston Beer Company swears by active dry yeast as a hangover-avoidance remedy

what [Joe] Owades knew was that active dry yeast has an enzyme in it called alcohol dehydrogenases (ADH). Roughly put, ADH is able to break alcohol molecules down into their constituent parts of carbon, hydrogen, and oxygen. Which is the same thing that happens when your body metabolizes alcohol in its liver. Owades realized if you also have that enzyme in your stomach when the alcohol first hits it, the ADH will begin breaking it down before it gets into your bloodstream and, thus, your brain.

(tags: beer science health yeast alcohol adh medicine enzymes stomach food)

Sirius by Comcast

At Comcast, our applications need convenient, low-latency access to important reference datasets. For example, our XfinityTV websites and apps need to use entertainment-related data to serve almost every API or web request to our datacenters: information like what year Casablanca was released, or how many episodes were in Season 7 of Seinfeld, or when the next episode of the Voice will be airing (and on which channel!). We traditionally managed this information with a combination of relational databases and RESTful web services but yearned for something simpler than the ORM, HTTP client, and cache management code our developers dealt with on a daily basis. As main memory sizes on commodity servers continued to grow, however, we asked ourselves: How can we keep this reference data entirely in RAM, while ensuring it gets updated as needed and is easily accessible to application developers? The Sirius distributed system library is our answer to that question, and we’re happy to announce that we’ve made it available as an open source project. Sirius is written in Scala and uses the Akka actor system under the covers, but is easily usable by any JVM-based language.

(tags: open-source comcast paxos replication read-only datastores storage memory memcached redis sirius scala akka jvm libraries)

AWS Elastic Beanstalk for Docker

This is pretty amazing. nice work, Beanstalk team. not sure how well it integrates with the rest of AWS though

(tags: aws amazon docker ec2 beanstalk ops containers linux)

TDD is dead. Long live testing

Oh god. I agree with DHH. shoot me now.

Test-first units leads to an overly complex web of intermediary objects and indirection in order to avoid doing anything that’s “slow”. Like hitting the database. Or file IO. Or going through the browser to test the whole system. It’s given birth to some truly horrendous monstrosities of architecture. A dense jungle of service objects, command patterns, and worse. I rarely unit test in the traditional sense of the word, where all dependencies are mocked out, and thousands of tests can close in seconds. It just hasn’t been a useful way of dealing with the testing of Rails applications. I test active record models directly, letting them hit the database, and through the use of fixtures. Then layered on top is currently a set of controller tests, but I’d much rather replace those with even higher level system tests through Capybara or similar. I think that’s the direction we’re heading. Less emphasis on unit tests, because we’re no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests.

(tags: tdd rails testing unit-tests system-tests integration-testing ruby dhh mocks)

All at sea: global shipping fleet exposed to hacking threat | Reuters

Hackers recently shut down a floating oil rig by tilting it, while another rig was so riddled with computer malware that it took 19 days to make it seaworthy again; Somali pirates help choose their targets by viewing navigational data online, prompting ships to either turn off their navigational devices, or fake the data so it looks like they’re somewhere else; and hackers infiltrated computers connected to the Belgian port of Antwerp, located specific containers, made off with their smuggled drugs and deleted the records.

(tags: via:mikko security hacking oilrigs shipping ships maritime antwerp piracy malware)

Search Results – (Author:Thomas H Mason)

Photographs taken by my great-grandfather, Thomas H. Mason, in the National Library of Ireland’s newly-digitized online collection

(tags: family thomas-h-mason history ireland photography archive nli)

Syria’s lethal Facebook checkpoints

An anonymous tip from a highly reliable source: “There are checkpoints in Syria where your Facebook is checked for affiliation with the rebellious groups or individuals aligned with the rebellion. People are then disappeared or killed if they are found to be connected. Drivers are literally forced to load their Facebook/Twitter accounts and then they are riffled through. It’s happening daily, and has been for a year at least.”

(tags: boing-boing war facebook social-media twitter internet checkpoints syria)

The ancient Egyptian word for ‘cat’ was pronounced ‘miaow’

and many other cool-but-true factoids

(tags: fun quora interesting via:dorothy factoids true urban-legends facts)

Haywire

kellabyte’s hack in progress — ‘an asynchronous HTTP server framework written in C. The goal of Haywire is to learn how to create a server with a minimal feature set that can handle a high rate of requests and connections with as low of latency and resource usage as possible. Haywire uses the event loop based libuv platform layer that node.js is built on top of (also written in C). libuv abstracts IOCP on Windows and epoll/kqueue/event ports/etc. on Unix systems to provide efficient asynchronous I/O on all supported platforms.’ Outperforms libevent handily, it seems. Apache-licensed.

(tags: server http asynchronous libuv haywire kellabyte c events open-source asl2)

spoofing the samsung smart tv internet check

If this kind of bullshit — a HTTP GET of an XML file from www.samsung.com — is how the Samsung Smart TV firmware decides if the internet is working or not, I dread to think how crappy the rest of the code is. (At least in Netnote we performed a bunch of bigco-domain DNS lookups before giving up…)

(tags: smart-tv samsung fail xml http internet embedded-software firmware crap-code)

ImperialViolet – No, don’t enable revocation checking

…because it doesn’t stop attacks. Turning it on does nothing but slow things down. You can tell when something is security theater because you need some absurdly specific situation in order for it to be useful.

(tags: cryptography crypto heartbleed ssl security tls https internet revocation crls)

The Emerging Global Web

*Really* intriguing slide deck on how Asia and Africa have invented new ways of operating a business via the internet, and are turning globalisation upside down (via Yoz)

(tags: via:yoz africa asia globalisation internet web mobile payment business ecommerce global)

Using AWS in the context of Australian Privacy Considerations

interesting new white paper from Amazon regarding recent strengthening of the Aussie privacy laws, particularly w.r.t. geographic location of data and access by overseas law enforcement agencies…

(tags: amazon aws security law privacy data-protection ec2 s3 nsa gchq five-eyes)

For world’s biggest troll, first patent case ends up in tatters

Love it. Intellectual Ventures suffers a major bloody nose in IV/Capital One patent-trolling litigation

(tags: trolls patent-trolls patents swpats capital-one intellectual-ventures)

Notes On Concurrent Ring Buffer Queue Mechanics

great notes from Nitsan Wakart, who’s been hacking on ringbuffers a lot in JAQ

(tags: jaq nitsanw atomic concurrency data-structures ring-buffers queueing queues algorithms)

Uplink Latency of WiFi and 4G Networks

It’s high. Wifi in particular shows high variability and long latency tails

(tags: wifi 3g 4g mobile networking internet latency tcp)

vim-flake8

vim-flake8 is a Vim plugin that runs the currently open file through Flake8, a static syntax and style checker for Python source code. It supersedes both vim-pyflakes and vim-pep8. Flake8 is a wrapper around PyFlakes (static syntax checker), PEP8 (style checker) and Ned’s MacCabe script (complexity checker).

(tags: vim python syntax error-checking errors flake8 editors ides coding)

Consul

Nice-looking new tool from Hashicorp; service discovery and configuration service, built on Raft for leader election, Serf for gossip-based messaging, and Go. Some features: * Gossip is performed over both TCP and UDP; * gossip messages are encrypted symmetrically and therefore secure from eavesdropping, tampering, spoofing and packet corruption (like the incident which brought down S3 for days: http://status.aws.amazon.com/s3-20080720.html ); * exposes both a HTTP interface and (even better) DNS; * includes explicit support for long-distance WAN operation as well as on LANs. It all looks very practical and usable. MPL-licensed. The only potential risk I can see is that expecting to receive config updates from a blocking poll of the HTTP interface needs some good “best practice” docs, to ensure that people don’t mishandle the scenario where there is a network partition between your calling code and the Consul server/agent. Without any heartbeating protocol behind the scenes, HTTP is vulnerable to “hung connections” which would result in a config change being silently missed by the client until the connection eventually is timed out, either by the calling code or the client-side kernel. This could potentially take minutes to occur, which in some usage scenarios could be a big, unforeseen problem.

(tags: configuration service-discovery distcomp raft consensus-algorithms go mpl open-source dns http gossip-protocol hashicorp)

Druid | How We Scaled HyperLogLog: Three Real-World Optimizations

3 optimizations Druid.io have made to the HLL algorithm to scale it up for production use in Metamarkets: compacting registers (fixes a bug with unions of multiple HLLs); a sparse storage format (to optimize space); faster lookups using a lookup table.

(tags: druid.io metamarkets scaling hyperloglog hll algorithms performance optimization counting estimation)

HyperLogLog – Intersection Arithmetic

‘In general HLL intersection in StreamLib works.  |A INTERSECT B| = |A| + |B| – |A UNION B|.  Timon’s article on intersection is important to read though.  The usefulness of HLL intersection depends on the features of the HLLs you are intersecting.’

(tags: hyperloglog hll hyperloglogplus streamlib intersections sets estimation algorithms)

Structural Integrity | 99% Invisible

‘The student (who has since been lost to history) was studying Citicorp Center as part of his thesis and had found that the building was particularly vulnerable to quartering winds (winds that strike the building at its corners). Normally, buildings are strongest at their corners, and it’s the perpendicular winds (winds that strike the building at its face) that cause the greatest strain. But this was not a normal building. LeMessurier had accounted for the perpendicular winds, but not the quartering winds. He checked the math, and found that the student was right. He compared what velocity winds the building could withstand with weather data, and found that a storm strong enough to topple Citicorp Center hits New York City every 55 years. But that’s only if the tuned mass damper, which keeps the building stable, is running. LeMessurier realized that a major storm could cause a blackout and render the tuned mass damper inoperable. Without the tuned mass damper, LeMessurier calculated that a storm powerful enough to take out the building his New York every sixteen years.’

(tags: william-lemessurier architecture danger risk buildings nyc citicorp-center wind mass-dampers physics)

Linode announces new instance specs

‘TL;DR: SSDs + Insane network + Faster processors + Double the RAM + Hourly Billing’

(tags: hosting linode ssd performance linux ops datacenters)

fcron

Fcron is a scheduler. It aims at replacing Vixie Cron, so it implements most of its functionalities. But contrary to Vixie Cron, fcron does not need your system to be up 7 days a week, 24 hours a day : it also works well with systems which are running only occasionnally (contrary to anacrontab). In other words, fcron does both the job of Vixie Cron and anacron, but does even more and better :)) …

(tags: via:chughes cron fcron unix linux ops scheduler automation scripts)

Ryanair drops out of top Google flight search results after website overhaul | Business | theguardian.com

They’ve done the classic website-redesign screwup — omitted redirects from the old URLs.

Sam Silverwood-Cope, director of Intelligent Positioning, said: “They’ve ignored the legacy of the old Ryanair.com. It’s quite startling. They are doing it just before their busiest time of the year.” A change in [URLs] without proper redirects means many results found by Google now simply return error pages, he added. “Unless redirects get put in pretty soon, the position is going to get worse and worse.”

(tags: ryanair inept fail funny via:christinebohan web google search redirects)

Scarfolk Council

Scarfolk is a town in North West England that did not progress beyond 1979. Instead, the entire decade of the 1970s loops ad infinitum. Here in Scarfolk, pagan rituals blend seamlessly with science; hauntology is a compulsory subject at school, and everyone must be in bed by 8pm because they are perpetually running a slight fever. “Visit Scarfolk today. Our number one priority is keeping rabies at bay.” For more information please reread.

(tags: scarfolk 1970s england history funny humour public-information pagan morbid)

OpenSSL Valhalla Rampage

OpenBSD are going wild ripping out “arcane VMS hacks” in an attempt to render OpenSSL’s source code comprehensible, and finding amazing horrors like this: ‘Well, even if time() isn’t random, your RSA private key is probably pretty random. Do not feed RSA private key information to the random subsystem as entropy. It might be fed to a pluggable random subsystem…. What were they thinking?!’

(tags: random security openssl openbsd coding horror rsa private-keys entropy)

“H” in cron syntax

This is something Jenkins have come up to randomize and distribute load, in order to avoid the “thundering-herd” bug. Good call

(tags: jenkins randomization load-balancing load thundering-herd ops capacity sleep)

Shared Space and other bad junction designs lead to crashes and injuries

Just because something is “Dutch”, that doesn’t mean it’s good. The Netherlands has many excellent examples, but you have to be very selective about what serves as a model. Cyclists fare best where their interactions with motor vehicles are limited and controlled. They fare best where infrastructure ensures that minor mistakes do not result in injuries. Anywhere that we rely upon everyone behaving perfectly but where we do not protect the most vulnerable, there will be injuries. Good design takes human nature into account and removes the causes of danger from those who are most vulnerable.

(tags: cycling design junctions shared-space dutch holland roads safety crashes)

Beefcake

A sane Google Protocol Buffers library for Ruby. It’s all about being Buf; ProtoBuf.

(tags: protobuf google protocol-buffers ruby coding libraries gems open-source)

Dan Kaminsky on Heartbleed

When I said that we expected better of OpenSSL, it’s not merely that there’s some sense that security-driven code should be of higher quality.  (OpenSSL is legendary for being considered a mess, internally.)  It’s that the number of systems that depend on it, and then expose that dependency to the outside world, are considerable.  This is security’s largest contributed dependency, but it’s not necessarily the software ecosystem’s largest dependency.  Many, maybe even more systems depend on web servers like Apache, nginx, and IIS.  We fear vulnerabilities significantly more in libz than libbz2 than libxz, because more servers will decompress untrusted gzip over bzip2 over xz.  Vulnerabilities are not always in obvious places – people underestimate just how exposed things like libxml and libcurl and libjpeg are.  And as HD Moore showed me some time ago, the embedded space is its own universe of pain, with 90’s bugs covering entire countries. If we accept that a software dependency becomes Critical Infrastructure at some level of economic dependency, the game becomes identifying those dependencies, and delivering direct technical and even financial support.  What are the one million most important lines of code that are reachable by attackers, and least covered by defenders?  (The browsers, for example, are very reachable by attackers but actually defended pretty zealously – FFMPEG public is not FFMPEG in Chrome.) Note that not all code, even in the same project, is equally exposed.    It’s tempting to say it’s a needle in a haystack.  But I promise you this:  Anybody patches Linux/net/ipv4/tcp_input.c (which handles inbound network for Linux), a hundred alerts are fired and many of them are not to individuals anyone would call friendly.  One guy, one night, patched OpenSSL.  Not enough defenders noticed, and it took Neel Mehta to do something.

(tags: development openssl heartbleed ssl security dan-kaminsky infrastructure libraries open-source dependencies)

s3funnel

‘a command line tool for Amazon’s Simple Storage Service (S3). Written in Python, easy_install the package to install as an egg. Supports multithreaded operations for large volumes. Put, get, or delete many items concurrently, using a fixed-size pool of threads. Built on workerpool for multithreading and boto for access to the Amazon S3 API. Unix-friendly input and output. Pipe things in, out, and all around.’ MIT-licensed open source. (via Paul Dolan)

(tags: via:pdolan s3 s3funnel tools ops aws python mit open-source)