Skip to content

Justin's Linklog Posts

Links for 2014-05-14

Published May 14, 2014

Links for 2014-05-13

Published May 13, 2014

Links for 2014-05-12

Published May 12, 2014

Links for 2014-05-09

Published May 9, 2014

Links for 2014-05-07

Published May 7, 2014

Links for 2014-05-06

Published May 6, 2014

  • Minimum Viable Block Chain

    Ilya Grigorik describes the design of the Bitcoin/altcoin block chain algorithm. Illuminating writeup

    (tags: algorithms bitcoin security crypto blockchain ilya-grigorik)

  • Docker Plugin for Jenkins

    The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave. Optionally, the container can be committed, so that (for example) manual QA could be performed by the container being imported into a local docker provider, and run from there.
    The holy grail of Jenkins/Docker integration. How cool is that…

    (tags: jenkins docker ops testing ec2 hosting scaling elastic-scaling system-testing)

  • Simple Binary Encoding

    an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. […] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message. The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer. The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations. The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again. The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.

    (tags: sbe encoding protobuf protocol-buffers json messages messaging binary formats low-latency martin-thompson xml)

  • Observations of an Internet Middleman

    That leaves the remaining six [consumer ISPs peering with Level3] with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content. Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.
    Amazing that L3 are happy to publish this — that’s where big monopoly ISPs have led their industry.

    (tags: net-neutrality networking internet level3 congestion isps us-politics)

  • interview with Google VP of SRE Ben Treynor

    interviewed by Niall Murphy, no less ;). Some good info on what Google deems important from an ops/SRE perspective

    (tags: sre ops devops google monitoring interviews ben-treynor)

Links for 2014-05-02

Published May 2, 2014

  • Faster BAM Sorting with SAMtools and RocksDB

    Now this is really really clever. Heap-merging a heavyweight genomics format, using RocksDB to speed it up.

    There’s a problem with the single-pass merge described above when the number of intermediate files, N/R, is large. Merging the sorted intermediate files in limited memory requires constantly reading little bits from all those files, incurring a lot of disk seeks on rotating drives. In fact, at some point, samtools sort performance becomes effectively bound to disk seeking. […] In this scenario, samtools rocksort can sort the same data in much less time, using no more memory, by invoking RocksDB’s background compaction capabilities. With a few extra lines of code we configure RocksDB so that, while we’re still in the process of loading the BAM data, it runs additional background threads to merge batches of existing sorted temporary files into fewer, larger, sorted files. Just like the final merge, each background compaction requires only a modest amount of working memory.
    (via the RocksDB facebook group)

    (tags: rocksdb algorithms sorting leveldb bam samtools merging heaps compaction)

  • Coding For Life (Battery Life, That Is)

    great presentation on Android mobile battery life, and what to avoid

    (tags: presentations via:sergio android mobile battery battery-life 3g wifi gprs hardware)

  • Oisin’s mobile app release checklist

    ‘This form is to document the testing that has been done on each app version before submitting to the App Store. For each item, indicate Yes if the testing has been done, Not Applicable if the testing does not apply (eg testing audio for an app that doesn’t play any), or No if the testing has not been done for another reason.’

    (tags: apps checklists release coding ios android mobile ohurley)

  • “A New Data Structure For Cumulative Frequency Tables”

    paper by Peter M Fenwick, 1993. ‘A new method (the ‘binary indexed tree’) is presented for maintaining the cumulative frequencies which are needed to support dynamic arithmetic data compression. It is based on a decomposition of the cumulative frequencies into portions which parallel the binary representation of the index of the table element (or symbol). The operations to traverse the data structure are based on the binary coding of the index. In comparison with previous methods, the binary indexed tree is faster, using more compact data and simpler code. The access time for all operations is either constant or proportional to the logarithm of the table size. In conjunction with the compact data structure, this makes the new method particularly suitable for large symbol alphabets.’ via Jakob Buchgraber, who’s implementing it right now in Netty ;)

    (tags: netty frequency-tables data-structures algorithms coding binary-tree indexing compression symbol-alphabets)

Links for 2014-05-01

Published May 1, 2014

Links for 2014-04-30

Published April 30, 2014

Links for 2014-04-29

Published April 29, 2014

  • ‘Pickles & Spores: Improving Support for Distributed Programming in Scala

    ‘Spores are “small units of possibly mobile functional behavior”. They’re a closure-like abstraction meant for use in distributed or concurrent environments. Spores provide a guarantee that the environment is effectively immutable, and safe to ship over the wire. Spores aim to give library authors some confidence in exposing functions (or, rather, spores) in public APIs for safe consumption in a distributed or concurrent environment. The first part of the talk covers a simpler variant of spores as they are proposed for inclusion in Scala 2.11. The second part of the talk briefly introduces a current research project ongoing at EPFL which leverages Scala’s type system to provide type constraints that give authors finer-grained control over spore capturing semantics. What’s more, these type constraints can be composed during spore composition, so library authors are effectively able to propagate expert knowledge via these composable constraints. The last part of the talk briefly covers Scala/Pickling, a fast new, open serialization framework.’

    (tags: pickling scala presentations spores closures fp immutability coding distributed distcomp serialization formats network)

  • BBC News – Microsoft ‘must release’ data held on Dublin server

    Messy. I can’t see this lasting beyond an appeal.

    Law enforcement efforts would be seriously impeded and the burden on the government would be substantial if they had to co-ordinate with foreign governments to obtain this sort of information from internet service providers such as Microsoft and Google, Judge Francis said. In a blog post, Microsoft’s deputy general counsel, David Howard, said: “A US prosecutor cannot obtain a US warrant to search someone’s home located in another country, just as another country’s prosecutor cannot obtain a court order in her home country to conduct a search in the United States. “We think the same rules should apply in the online world, but the government disagrees.”

    (tags: microsoft regions law us-law privacy google cloud international-law surveillance)

  • Russia passes bill requiring bloggers to register with government

    A bill passed by the Russian parliament on Tuesday says that any blogger read by at least 3,000 people a day has to register with the government telecom watchdog and follow the same rules as those imposed by Russian law on mass media. These include privacy safeguards, the obligation to check all facts, silent days before elections and loose but threatening injunctions against “abetting terrorism” and “extremism.”
    Russian blogging platforms have responded by changing view-counter tickers to display “2500+” as a max.

    (tags: russia blogs blogging terrorism extremism internet regulation chilling-effects censorship)

Links for 2014-04-28

Published April 28, 2014

Links for 2014-04-25

Published April 25, 2014

Links for 2014-04-24

Published April 24, 2014

  • Sirius by Comcast

    At Comcast, our applications need convenient, low-latency access to important reference datasets. For example, our XfinityTV websites and apps need to use entertainment-related data to serve almost every API or web request to our datacenters: information like what year Casablanca was released, or how many episodes were in Season 7 of Seinfeld, or when the next episode of the Voice will be airing (and on which channel!). We traditionally managed this information with a combination of relational databases and RESTful web services but yearned for something simpler than the ORM, HTTP client, and cache management code our developers dealt with on a daily basis. As main memory sizes on commodity servers continued to grow, however, we asked ourselves: How can we keep this reference data entirely in RAM, while ensuring it gets updated as needed and is easily accessible to application developers? The Sirius distributed system library is our answer to that question, and we’re happy to announce that we’ve made it available as an open source project. Sirius is written in Scala and uses the Akka actor system under the covers, but is easily usable by any JVM-based language.
    Also includes a Paxos implementation with “fast follower” read-only slave replication. ASL2-licensed open source. The only thing I can spot to be worried about is speed of startup; they note that apps need to replay a log at startup to rebuild state, which can be slow if unoptimized in my experience. Update: in a twitter conversation at https://twitter.com/jon_moore/status/459363751893139456 , Jon Moore indicated they haven’t had problems with this even with ‘datasets consuming 10-20GB of heap’, and have ‘benchmarked a 5-node Sirius ingest cluster up to 1k updates/sec write throughput.’ That’s pretty solid!

    (tags: open-source comcast paxos replication read-only datastores storage memory memcached redis sirius scala akka jvm libraries)

  • AWS Elastic Beanstalk for Docker

    This is pretty amazing. nice work, Beanstalk team. not sure how well it integrates with the rest of AWS though

    (tags: aws amazon docker ec2 beanstalk ops containers linux)

  • TDD is dead. Long live testing

    Oh god. I agree with DHH. shoot me now.

    Test-first units leads to an overly complex web of intermediary objects and indirection in order to avoid doing anything that’s “slow”. Like hitting the database. Or file IO. Or going through the browser to test the whole system. It’s given birth to some truly horrendous monstrosities of architecture. A dense jungle of service objects, command patterns, and worse. I rarely unit test in the traditional sense of the word, where all dependencies are mocked out, and thousands of tests can close in seconds. It just hasn’t been a useful way of dealing with the testing of Rails applications. I test active record models directly, letting them hit the database, and through the use of fixtures. Then layered on top is currently a set of controller tests, but I’d much rather replace those with even higher level system tests through Capybara or similar. I think that’s the direction we’re heading. Less emphasis on unit tests, because we’re no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests.

    (tags: tdd rails testing unit-tests system-tests integration-testing ruby dhh mocks)

  • All at sea: global shipping fleet exposed to hacking threat | Reuters

    Hackers recently shut down a floating oil rig by tilting it, while another rig was so riddled with computer malware that it took 19 days to make it seaworthy again; Somali pirates help choose their targets by viewing navigational data online, prompting ships to either turn off their navigational devices, or fake the data so it looks like they’re somewhere else; and hackers infiltrated computers connected to the Belgian port of Antwerp, located specific containers, made off with their smuggled drugs and deleted the records.
    (via Mikko Hypponen)

    (tags: via:mikko security hacking oilrigs shipping ships maritime antwerp piracy malware)

  • Search Results – (Author:Thomas H Mason)

    Photographs taken by my great-grandfather, Thomas H. Mason, in the National Library of Ireland’s newly-digitized online collection

    (tags: family thomas-h-mason history ireland photography archive nli)

  • Syria’s lethal Facebook checkpoints

    An anonymous tip from a highly reliable source: “There are checkpoints in Syria where your Facebook is checked for affiliation with the rebellious groups or individuals aligned with the rebellion. People are then disappeared or killed if they are found to be connected. Drivers are literally forced to load their Facebook/Twitter accounts and then they are riffled through. It’s happening daily, and has been for a year at least.”

    (tags: boing-boing war facebook social-media twitter internet checkpoints syria)

Links for 2014-04-22

Published April 22, 2014

Links for 2014-04-18

Published April 18, 2014

  • Consul

    Nice-looking new tool from Hashicorp; service discovery and configuration service, built on Raft for leader election, Serf for gossip-based messaging, and Go. Some features: * Gossip is performed over both TCP and UDP; * gossip messages are encrypted symmetrically and therefore secure from eavesdropping, tampering, spoofing and packet corruption (like the incident which brought down S3 for days: http://status.aws.amazon.com/s3-20080720.html ); * exposes both a HTTP interface and (even better) DNS; * includes explicit support for long-distance WAN operation as well as on LANs. It all looks very practical and usable. MPL-licensed. The only potential risk I can see is that expecting to receive config updates from a blocking poll of the HTTP interface needs some good “best practice” docs, to ensure that people don’t mishandle the scenario where there is a network partition between your calling code and the Consul server/agent. Without any heartbeating protocol behind the scenes, HTTP is vulnerable to “hung connections” which would result in a config change being silently missed by the client until the connection eventually is timed out, either by the calling code or the client-side kernel. This could potentially take minutes to occur, which in some usage scenarios could be a big, unforeseen problem.

    (tags: configuration service-discovery distcomp raft consensus-algorithms go mpl open-source dns http gossip-protocol hashicorp)

Links for 2014-04-17

Published April 17, 2014

  • Druid | How We Scaled HyperLogLog: Three Real-World Optimizations

    3 optimizations Druid.io have made to the HLL algorithm to scale it up for production use in Metamarkets: compacting registers (fixes a bug with unions of multiple HLLs); a sparse storage format (to optimize space); faster lookups using a lookup table.

    (tags: druid.io metamarkets scaling hyperloglog hll algorithms performance optimization counting estimation)

  • HyperLogLog – Intersection Arithmetic

    ‘In general HLL intersection in StreamLib works.  |A INTERSECT B| = |A| + |B| – |A UNION B|.  Timon’s article on intersection is important to read though.  The usefulness of HLL intersection depends on the features of the HLLs you are intersecting.’

    (tags: hyperloglog hll hyperloglogplus streamlib intersections sets estimation algorithms)

  • Structural Integrity | 99% Invisible

    ‘The student (who has since been lost to history) was studying Citicorp Center as part of his thesis and had found that the building was particularly vulnerable to quartering winds (winds that strike the building at its corners). Normally, buildings are strongest at their corners, and it’s the perpendicular winds (winds that strike the building at its face) that cause the greatest strain. But this was not a normal building. LeMessurier had accounted for the perpendicular winds, but not the quartering winds. He checked the math, and found that the student was right. He compared what velocity winds the building could withstand with weather data, and found that a storm strong enough to topple Citicorp Center hits New York City every 55 years. But that’s only if the tuned mass damper, which keeps the building stable, is running. LeMessurier realized that a major storm could cause a blackout and render the tuned mass damper inoperable. Without the tuned mass damper, LeMessurier calculated that a storm powerful enough to take out the building his New York every sixteen years.’

    (tags: william-lemessurier architecture danger risk buildings nyc citicorp-center wind mass-dampers physics)

  • Linode announces new instance specs

    ‘TL;DR: SSDs + Insane network + Faster processors + Double the RAM + Hourly Billing’

    (tags: hosting linode ssd performance linux ops datacenters)

  • fcron

    Fcron is a scheduler. It aims at replacing Vixie Cron, so it implements most of its functionalities. But contrary to Vixie Cron, fcron does not need your system to be up 7 days a week, 24 hours a day : it also works well with systems which are running only occasionnally (contrary to anacrontab). In other words, fcron does both the job of Vixie Cron and anacron, but does even more and better :)) …
    Thanks Craig!

    (tags: via:chughes cron fcron unix linux ops scheduler automation scripts)

  • Ryanair drops out of top Google flight search results after website overhaul | Business | theguardian.com

    They’ve done the classic website-redesign screwup — omitted redirects from the old URLs.

    Sam Silverwood-Cope, director of Intelligent Positioning, said: “They’ve ignored the legacy of the old Ryanair.com. It’s quite startling. They are doing it just before their busiest time of the year.” A change in [URLs] without proper redirects means many results found by Google now simply return error pages, he added. “Unless redirects get put in pretty soon, the position is going to get worse and worse.”

    (tags: ryanair inept fail funny via:christinebohan web google search redirects)

  • Scarfolk Council

    Scarfolk is a town in North West England that did not progress beyond 1979. Instead, the entire decade of the 1970s loops ad infinitum. Here in Scarfolk, pagan rituals blend seamlessly with science; hauntology is a compulsory subject at school, and everyone must be in bed by 8pm because they are perpetually running a slight fever. “Visit Scarfolk today. Our number one priority is keeping rabies at bay.” For more information please reread.

    (tags: scarfolk 1970s england history funny humour public-information pagan morbid)

  • OpenSSL Valhalla Rampage

    OpenBSD are going wild ripping out “arcane VMS hacks” in an attempt to render OpenSSL’s source code comprehensible, and finding amazing horrors like this: ‘Well, even if time() isn’t random, your RSA private key is probably pretty random. Do not feed RSA private key information to the random subsystem as entropy. It might be fed to a pluggable random subsystem…. What were they thinking?!’

    (tags: random security openssl openbsd coding horror rsa private-keys entropy)

Links for 2014-04-16

Published April 16, 2014

  • “H” in cron syntax

    This is something Jenkins have come up to randomize and distribute load, in order to avoid the “thundering-herd” bug. Good call

    (tags: jenkins randomization load-balancing load thundering-herd ops capacity sleep)

  • Shared Space and other bad junction designs lead to crashes and injuries

    Just because something is “Dutch”, that doesn’t mean it’s good. The Netherlands has many excellent examples, but you have to be very selective about what serves as a model. Cyclists fare best where their interactions with motor vehicles are limited and controlled. They fare best where infrastructure ensures that minor mistakes do not result in injuries. Anywhere that we rely upon everyone behaving perfectly but where we do not protect the most vulnerable, there will be injuries. Good design takes human nature into account and removes the causes of danger from those who are most vulnerable.
    via Tony Finch

    (tags: cycling design junctions shared-space dutch holland roads safety crashes)

  • Beefcake

    A sane Google Protocol Buffers library for Ruby. It’s all about being Buf; ProtoBuf.

    (tags: protobuf google protocol-buffers ruby coding libraries gems open-source)

  • Dan Kaminsky on Heartbleed

    When I said that we expected better of OpenSSL, it’s not merely that there’s some sense that security-driven code should be of higher quality.  (OpenSSL is legendary for being considered a mess, internally.)  It’s that the number of systems that depend on it, and then expose that dependency to the outside world, are considerable.  This is security’s largest contributed dependency, but it’s not necessarily the software ecosystem’s largest dependency.  Many, maybe even more systems depend on web servers like Apache, nginx, and IIS.  We fear vulnerabilities significantly more in libz than libbz2 than libxz, because more servers will decompress untrusted gzip over bzip2 over xz.  Vulnerabilities are not always in obvious places – people underestimate just how exposed things like libxml and libcurl and libjpeg are.  And as HD Moore showed me some time ago, the embedded space is its own universe of pain, with 90’s bugs covering entire countries. If we accept that a software dependency becomes Critical Infrastructure at some level of economic dependency, the game becomes identifying those dependencies, and delivering direct technical and even financial support.  What are the one million most important lines of code that are reachable by attackers, and least covered by defenders?  (The browsers, for example, are very reachable by attackers but actually defended pretty zealously – FFMPEG public is not FFMPEG in Chrome.) Note that not all code, even in the same project, is equally exposed.    It’s tempting to say it’s a needle in a haystack.  But I promise you this:  Anybody patches Linux/net/ipv4/tcp_input.c (which handles inbound network for Linux), a hundred alerts are fired and many of them are not to individuals anyone would call friendly.  One guy, one night, patched OpenSSL.  Not enough defenders noticed, and it took Neel Mehta to do something.

    (tags: development openssl heartbleed ssl security dan-kaminsky infrastructure libraries open-source dependencies)

  • s3funnel

    ‘a command line tool for Amazon’s Simple Storage Service (S3). Written in Python, easy_install the package to install as an egg. Supports multithreaded operations for large volumes. Put, get, or delete many items concurrently, using a fixed-size pool of threads. Built on workerpool for multithreading and boto for access to the Amazon S3 API. Unix-friendly input and output. Pipe things in, out, and all around.’ MIT-licensed open source. (via Paul Dolan)

    (tags: via:pdolan s3 s3funnel tools ops aws python mit open-source)

Links for 2014-04-15

Published April 15, 2014

  • Hydra Takes On Hadoop

    The intuition behind Hydra is something like this, “I have a lot of data, and there are a lot of things I could try to learn about it — so many that I’m not even sure what I want to know.” It’s about the curse of dimensionality — more dimensions means exponentially more cost for exhaustive analysis. Hydra tries to make it easy to reduce the number of dimensions, or the cost of watching them (via probabilistic data structures), to just the right point where everything runs quickly but can still answer almost any question you think you might care about.
    Code: https://github.com/addthis/hydra Getting Started blog post: https://www.addthis.com/blog/2014/02/18/getting-started-with-hydra/

    (tags: hyrda hadoop data-processing big-data trees clusters analysis)

  • Stalled SCP and Hanging TCP Connections

    a Cisco fail.

    It looks like there’s a firewall in the middle that’s doing additional TCP sequence randomisation which was a good thing, but has been fixed in all current operating systems. Unfortunately, it seems that firewall doesn’t understand TCP SACK, which when coupled with a small amount of packet loss and a stateful host firewall that blocks invalid packets results in TCP connections that stall randomly. A little digging revealed that firewall to be the Cisco Firewall Services Module on our Canterbury network border.
    (via Tony Finch)

    (tags: via:fanf cisco networking firewalls scp tcp hangs sack tcpdump)

  • Akamai’s “Secure Heap” patch wasn’t good enough

    ‘Having the private keys inaccessible is a good defense in depth move. For this patch to work you have to make sure all sensitive values are stored in the secure area, not just check that the area looks inaccessible. You can’t do that by keeping the private key in the same process. A review by a security engineer would have prevented a false sense of security. A version where the private key and the calculations are in a separate process would be more secure. If you decide to write that version, I’ll gladly see if I can break that too.’ Akamai’s response: https://blogs.akamai.com/2014/04/heartbleed-update-v3.html — to their credit, they recognise that they need to take further action. (via Tony Finch)

    (tags: via:fanf cryptography openssl heartbleed akamai security ssl tls)

  • Shuffle Sharding

    Colm MacCarthaigh writes about a simple sharding/load-balancing algorithm which uses randomized instance selection and optional additional compartmentalization. See also: continuous hashing, and http://aphyr.com/posts/278-timelike-2-everything-fails-all-the-time

    (tags: hashing load-balancing sharding partitions dist-sys distcomp architecture coding)

  • Open Crypto Audit Project: TrueCrypt

    phase I, a source code audit by iSEC Partners, is now complete. Bruce Schneier says: “I’m still using it”.

    (tags: encryption security crypto truecrypt audits source-code isec matthew-green)

  • The science of ‘hangry’

    In the PNAS paper, Brad Bushman and colleagues looked at 107 couples over 21 days and found that people experiencing uncharacteristically low blood sugar were more likely to display anger toward their spouse. (The researchers measured this by having subjects stick needles into voodoo dolls representing their significant others.)

    (tags: hangry hunger food eating science health blood-sugar voodoo-dolls glucose)

  • insane ESB health and safety policy

    Where it is not possible to avoid reversing, it is ESB policy that staff driving on behalf of the company or anybody on company premises should reverse into car spaces/bays, allowing them to drive out subsequently.
    BUT WHYYYYYYYYYY

    (tags: esb health-n-safety policies crazy funny driving reversing lol safety)

Links for 2014-04-14

Published April 14, 2014

  • Cloudflare demonstrate Heartbleed key extraction

    from nginx. ‘Based on the findings, we recommend everyone reissue + revoke their private keys.’

    (tags: security nginx heartbleed ssl tls exploits private-keys)

  • When two-factor authentication is not enough

    Fastmail.FM nearly had their domain stolen through an attack exploiting missing 2FA authentication in Gandi.

    An important lesson learned is that just because a provider has a checkbox labelled “2 factor authentication” in their feature list, the two factors may not be protecting everything – and they may not even realise that fact themselves. Security risks always come on the unexpected paths – the “off label” uses that you didn’t think about, and the subtle interaction of multiple features which are useful and correct in isolation.

    (tags: gandi 2fa fastmail authentication security mfa two-factor-authentication mail)

  • Of Money, Responsibility, and Pride

    Steve Marquess of the OpenSSL Foundation on their funding, and lack thereof:

    I stand in awe of their talent and dedication, that of Stephen Henson in particular. It takes nerves of steel to work for many years on hundreds of thousands of lines of very complex code, with every line of code you touch visible to the world, knowing that code is used by banks, firewalls, weapons systems, web sites, smart phones, industry, government, everywhere. Knowing that you’ll be ignored and unappreciated until something goes wrong. The combination of the personality to handle that kind of pressure with the relevant technical skills and experience to effectively work on such software is a rare commodity, and those who have it are likely to already be a valued, well-rewarded, and jealously guarded resource of some company or worthy cause. For those reasons OpenSSL will always be undermanned, but the present situation can and should be improved. There should be at least a half dozen full time OpenSSL team members, not just one, able to concentrate on the care and feeding of OpenSSL without having to hustle commercial work. If you’re a corporate or government decision maker in a position to do something about it, give it some thought. Please. I’m getting old and weary and I’d like to retire someday.

    (tags: funding open-source openssl heartbleed internet security money)

  • Huginn

    a system for building agents that perform automated tasks for you online. They can read the web, watch for events, and take actions on your behalf. Huginn’s Agents create and consume events, propagating them along a directed event flow graph. Think of it as Yahoo! Pipes plus IFTTT on your own server. You always know who has your data. You do.
    MIT-licensed open source, built on Rails.

    (tags: ifttt automation huginn ruby rails open-source agents)

Links for 2014-04-13

Published April 13, 2014

Links for 2014-04-11

Published April 11, 2014

  • Basho LevelDB supports tiered storage

    Tiered storage is turning out to be a pretty practical trick to take advantage of SSDs:

    The justification for two types/speeds of storage arrays is simple. leveldb is extremely write intensive in its lower levels. The write intensity drops off as the level number increases. Similarly, current and frequently updated data tends to be in lower levels while archival data tends to be in higher levels. These leveldb characteristics create a desire to have faster, more expensive storage arrays for the high intensity lower levels. This branch allows the high intensity lower levels to be on expensive storage arrays while slower, less expensive storage arrays to hold the higher level data to reduce costs.

    (tags: caching tiered-storage storage ssds ebs leveldb basho patches riak iops)

  • Forbes on the skeleton crew nature of OpenSSL

    This is a great point:

    Obviously, those tending to the security protocols that support the rest of the Web need better infrastructure and more funding. “Large portions of the software infrastructure of the Internet are built and maintained by volunteers, who get little reward when their code works well but are blamed, and sometimes savagely derided, when it fails,” writes Foster in the New Yorker. […] “money and support still tend to flow to the newest and sexiest projects, while boring but essential elements like OpenSSL limp along as volunteer efforts,” he writes. “It’s easy to take open-source software for granted, and to forget that the Internet we use every day depends in part on the freely donated work of thousands of programmers.” We need to find ways to pay for work that is currently essentially donated freely. One promising project is Bithub, from Whisper Systems, where people who make valuable contributions to open source projects are rewarded (with Bitcoin of course). But the pool of Bitcoin is still donation based. The Internet has helped create a culture of free, but what we may need to recognize is that we get what we pay for. Well-funded companies pulling critical code from open source projects for their sites should have formal fee arrangements, rather than the volunteer group simply hoping these users will pony up some Benjamins for “prominent logo placement” on a website most people had never heard of before Heartbleed.

    (tags: open-source openssl free sponsorship forbes via:karl-whelan)

Links for 2014-04-10

Published April 10, 2014

Links for 2014-04-09

Published April 9, 2014

  • MICA: A Holistic Approach To Fast In-Memory Key-Value Storage [paper]

    Very interesting new approach to building a scalable in-memory K/V store. As Rajiv Kurian notes on the mechanical-sympathy list: ‘The basic idea is that each core is responsible for a portion of the key-space and requests are forwarded to the right core, avoiding multiple-writer scenarios. This is opposed to designs like memcache which uses locks and shared memory. Some of the things I found interesting: The single writer design is taken to an extreme. Clients assist the partitioning of requests, by calculating hashes before submitting GET requests. It uses Intel DPDK instead of sockets to forward packets to the right core, without processing the packet on any core. Each core is paired with a dedicated RX/TX queue. The design for a lossy cache is simple but interesting. It does things like replacing a hash slot (instead of chaining) etc. to take advantage of the lossy nature of caches. There is a lossless design too. A bunch of tricks to optimize for memory performance. This includes pre-allocation, design of the hash indexes, prefetching tricks etc. There are some other concurrency tricks that were interesting. Handling dangling pointers was one of them.’ Source code here: https://github.com/efficient/mica

    (tags: mica in-memory memory ram key-value-stores storage smp dpdk multicore memcached concurrency)

  • Google’s Open Bidder stack moving from Jetty to Netty

    Open Bidder traditionally used Jetty as an embedded webserver, for the critical tasks of accepting connections, processing HTTP requests, managing service threads, etc. Jetty is a robust, but traditional stack that carries the weight and tradeoffs of Servlet’s 15 years old design. For a maximum performance RTB agent that must combine very large request concurrency with very low latencies, and often benefit also from low-level control over the transport, memory management and other issue, a different webserver stack was required. Open Bidder now supports Netty, an asynchronous, event-driven, high-performance webserver stack. For existing code, the most important impact is that Netty is not compatible with the Servlet API. Its own internal APIs are often too low-level, not to mention proprietary to Netty; so Open Bidder v0.5 introduces some new, stack-neutral APIs for things like HTTP requests and responses, cookies, request handlers, and even simple HTML templating based on Mustache. These APIs will work with both Netty and Jetty. This means you don’t need to change any code to switch between Jetty and Netty; on the other hand, it also means that existing code written for Open Bidder 0.4 may need some changes even if you plan to keep using Jetty. [….] Netty’s superior efficiency is very significant; it supports 50% more traffic in the same hardware, and it maintains a perfect latency distribution even at the peak of its supported load.
    This doc is noteworthy on a couple of grounds: 1. the use of Netty in a public API/library, and the additional layer in place to add a friendlier API on top of that. I hope they might consider releasing that part as OSS at some point. 2. I also find it interesting that their API uses protobufs to marshal the message, and they plan in a future release to serialize those to JSON documents — that makes a lot of sense.

    (tags: apis google protobufs json documents interoperability netty jetty servlets performance java)

  • The University Times: TCD Provost Under Pressure To “Re-think” Identity Initiative

    Students, staff and alumni put pressure on Provost to reconsider changes to Trinity College Dublin’s name and coat of arms.

    alumni scholars from 2004 and 1994 who had been invited back for the dinner shouted ‘Dublin’ after the Provost welcomed them back to “Trinity College”.

    (tags: tcd tcuod rebranding fail identity dublin)

  • Daring Fireball: Rethinking What We Mean by ‘Mobile Web’

    We shouldn’t think of “the web” as only what renders in web browsers. We should think of the web as anything transmitted using HTTP and HTTPS. Apps and websites are peers, not competitors. They’re all just clients to the same services.
    +1. Finally, a Daring Fireball post I agree with! ;)

    (tags: daring-fireball apps web http https mobile apple android browsers)

Links for 2014-04-08

Published April 8, 2014

Links for 2014-04-07

Published April 7, 2014

Links for 2014-04-05

Published April 5, 2014

Links for 2014-04-03

Published April 3, 2014

Links for 2014-04-02

Published April 2, 2014

Links for 2014-04-01

Published April 1, 2014

Links for 2014-03-31

Published March 31, 2014

Links for 2014-03-28

Published March 28, 2014

  • “They Know Everything We Do”

    [via Boing Boing:] A new, exhaustive report from Human Rights Watch details the way the young state of modern Ethiopia has become a kind of pilot program for the abuse of “off-the-shelf” surveillance, availing itself of commercial products from the US, the UK, France, Italy and China in order to establish an abusive surveillance regime that violates human rights and suppresses legitimate political opposition under the guise of a anti-terrorism law that’s so broadly interpreted as to be meaningless. The 137 page report [from Human Rights Watch] details the technologies the Ethiopian government has acquired from several countries and uses to facilitate surveillance of perceived political opponents inside the country and among the diaspora. The government’s surveillance practices violate the rights to freedom of expression, association, and access to information. The government’s monopoly over all mobile and Internet services through its sole, state-owned telecom operator, Ethio Telecom, facilitates abuse of surveillance powers.

    (tags: human-rights surveillance ethiopia spying off-the-shelf spyware big-brother hrw human-rights-watch)

Links for 2014-03-26

Published March 26, 2014

  • Chinese cops cuff 1,500 in fake base station spam raid

    The street finds its own uses for things, in this case Stinger/IMSI-catcher-type fake mobile-phone base stations:

    Fake base stations are becoming a particularly popular modus operandi. Often concealed in a van or car, they are driven through city streets to spread their messages. The professional spammer in question charged 1,000 yuan (£100) to spam thousands of users in a radius of a few hundred metres. The pseudo-base station used could send out around 6,000 messages in just half an hour, the report said. Often such spammers are hired by local businessmen to promote their wares.
    (via Bernard Tyers)

    (tags: stingers imsi-catcher mobile-phones mobile cellphones china spam via:bernard-tyers)

  • TJ McIntyre on the incredible surveillance of telephone traffic at various Garda stations around the country

    The most grave issue is that each recording likely amounted to a serious criminal offence. Under Irish law, the recording of a telephone conversation on a public network without the consent of at least one party to the call amounts to an “interception”, a criminal offence carrying a possible term of imprisonment of up to five years. […] Consequently, unless gardai were notified that their calls might be recorded then a large number of criminal offences are likely to have been committed by and within the Garda Siochana itself.

    (tags: gubu surveillance gardai ags tjmcintyre bugging tapping phones ireland politics)

  • rr

    A cool-looking new debugging tool for C/C++ from Mozilla.

    Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.
    Low runtime overhead; easy deployability; targeted at 32-bit (?!) Linux; OSS. (via Bryan O’Sullivan)

    (tags: via:bos mozilla debugging coding firefox rr record replay gdb c++ linux)

  • Ask AIB – Boards.ie

    AIB now have a dedicated customer-support forum on Boards.ie. That is a *great* idea

    (tags: aib banking support forums boards.ie banks)

Links for 2014-03-25

Published March 25, 2014

  • Microservices and nanoservices

    A great reaction to Martin Fowler’s “microservices” coinage, from Arnon Rotem-Gal-Oz: ‘I guess it is easier to use a new name (Microservices) rather than say that this is what SOA actually meant’; ‘these are the very principles of SOA before vendors does pushed the [ESB] in the middle.’ Others have also chosen to define microservices slightly differently, as a service written in 10-100 LOC. Arnon’s reaction: “Nanoservice is an antipattern where a service is too fine-grained. A nanoservice is a service whose overhead (communications, maintenance, and so on) outweighs its utility.” Having dealt with maintaining an over-fine-grained SOA stack in Amazon, I can only agree with this definition; it’s easy to make things too fine-grained and create a raft of distributed-computing bugs and deployment/management complexity where there is no need to do so.

    (tags: architecture antipatterns nanoservices microservices soa services design esb)

  • Accidentally Turing-Complete

    slightly ruined by the inclusion of some “deliberately Turing-complete” systems

    (tags: turing computation software via:jwz turing-complete accidents automatons)