Skip to content

Justin's Linklog Posts

Links for 2014-04-17

Published April 17, 2014

  • Druid | How We Scaled HyperLogLog: Three Real-World Optimizations

    3 optimizations Druid.io have made to the HLL algorithm to scale it up for production use in Metamarkets: compacting registers (fixes a bug with unions of multiple HLLs); a sparse storage format (to optimize space); faster lookups using a lookup table.

    (tags: druid.io metamarkets scaling hyperloglog hll algorithms performance optimization counting estimation)

  • HyperLogLog – Intersection Arithmetic

    ‘In general HLL intersection in StreamLib works.  |A INTERSECT B| = |A| + |B| – |A UNION B|.  Timon’s article on intersection is important to read though.  The usefulness of HLL intersection depends on the features of the HLLs you are intersecting.’

    (tags: hyperloglog hll hyperloglogplus streamlib intersections sets estimation algorithms)

  • Structural Integrity | 99% Invisible

    ‘The student (who has since been lost to history) was studying Citicorp Center as part of his thesis and had found that the building was particularly vulnerable to quartering winds (winds that strike the building at its corners). Normally, buildings are strongest at their corners, and it’s the perpendicular winds (winds that strike the building at its face) that cause the greatest strain. But this was not a normal building. LeMessurier had accounted for the perpendicular winds, but not the quartering winds. He checked the math, and found that the student was right. He compared what velocity winds the building could withstand with weather data, and found that a storm strong enough to topple Citicorp Center hits New York City every 55 years. But that’s only if the tuned mass damper, which keeps the building stable, is running. LeMessurier realized that a major storm could cause a blackout and render the tuned mass damper inoperable. Without the tuned mass damper, LeMessurier calculated that a storm powerful enough to take out the building his New York every sixteen years.’

    (tags: william-lemessurier architecture danger risk buildings nyc citicorp-center wind mass-dampers physics)

  • Linode announces new instance specs

    ‘TL;DR: SSDs + Insane network + Faster processors + Double the RAM + Hourly Billing’

    (tags: hosting linode ssd performance linux ops datacenters)

  • fcron

    Fcron is a scheduler. It aims at replacing Vixie Cron, so it implements most of its functionalities. But contrary to Vixie Cron, fcron does not need your system to be up 7 days a week, 24 hours a day : it also works well with systems which are running only occasionnally (contrary to anacrontab). In other words, fcron does both the job of Vixie Cron and anacron, but does even more and better :)) …
    Thanks Craig!

    (tags: via:chughes cron fcron unix linux ops scheduler automation scripts)

  • Ryanair drops out of top Google flight search results after website overhaul | Business | theguardian.com

    They’ve done the classic website-redesign screwup — omitted redirects from the old URLs.

    Sam Silverwood-Cope, director of Intelligent Positioning, said: “They’ve ignored the legacy of the old Ryanair.com. It’s quite startling. They are doing it just before their busiest time of the year.” A change in [URLs] without proper redirects means many results found by Google now simply return error pages, he added. “Unless redirects get put in pretty soon, the position is going to get worse and worse.”

    (tags: ryanair inept fail funny via:christinebohan web google search redirects)

  • Scarfolk Council

    Scarfolk is a town in North West England that did not progress beyond 1979. Instead, the entire decade of the 1970s loops ad infinitum. Here in Scarfolk, pagan rituals blend seamlessly with science; hauntology is a compulsory subject at school, and everyone must be in bed by 8pm because they are perpetually running a slight fever. “Visit Scarfolk today. Our number one priority is keeping rabies at bay.” For more information please reread.

    (tags: scarfolk 1970s england history funny humour public-information pagan morbid)

  • OpenSSL Valhalla Rampage

    OpenBSD are going wild ripping out “arcane VMS hacks” in an attempt to render OpenSSL’s source code comprehensible, and finding amazing horrors like this: ‘Well, even if time() isn’t random, your RSA private key is probably pretty random. Do not feed RSA private key information to the random subsystem as entropy. It might be fed to a pluggable random subsystem…. What were they thinking?!’

    (tags: random security openssl openbsd coding horror rsa private-keys entropy)

Links for 2014-04-16

Published April 16, 2014

  • “H” in cron syntax

    This is something Jenkins have come up to randomize and distribute load, in order to avoid the “thundering-herd” bug. Good call

    (tags: jenkins randomization load-balancing load thundering-herd ops capacity sleep)

  • Shared Space and other bad junction designs lead to crashes and injuries

    Just because something is “Dutch”, that doesn’t mean it’s good. The Netherlands has many excellent examples, but you have to be very selective about what serves as a model. Cyclists fare best where their interactions with motor vehicles are limited and controlled. They fare best where infrastructure ensures that minor mistakes do not result in injuries. Anywhere that we rely upon everyone behaving perfectly but where we do not protect the most vulnerable, there will be injuries. Good design takes human nature into account and removes the causes of danger from those who are most vulnerable.
    via Tony Finch

    (tags: cycling design junctions shared-space dutch holland roads safety crashes)

  • Beefcake

    A sane Google Protocol Buffers library for Ruby. It’s all about being Buf; ProtoBuf.

    (tags: protobuf google protocol-buffers ruby coding libraries gems open-source)

  • Dan Kaminsky on Heartbleed

    When I said that we expected better of OpenSSL, it’s not merely that there’s some sense that security-driven code should be of higher quality.  (OpenSSL is legendary for being considered a mess, internally.)  It’s that the number of systems that depend on it, and then expose that dependency to the outside world, are considerable.  This is security’s largest contributed dependency, but it’s not necessarily the software ecosystem’s largest dependency.  Many, maybe even more systems depend on web servers like Apache, nginx, and IIS.  We fear vulnerabilities significantly more in libz than libbz2 than libxz, because more servers will decompress untrusted gzip over bzip2 over xz.  Vulnerabilities are not always in obvious places – people underestimate just how exposed things like libxml and libcurl and libjpeg are.  And as HD Moore showed me some time ago, the embedded space is its own universe of pain, with 90’s bugs covering entire countries. If we accept that a software dependency becomes Critical Infrastructure at some level of economic dependency, the game becomes identifying those dependencies, and delivering direct technical and even financial support.  What are the one million most important lines of code that are reachable by attackers, and least covered by defenders?  (The browsers, for example, are very reachable by attackers but actually defended pretty zealously – FFMPEG public is not FFMPEG in Chrome.) Note that not all code, even in the same project, is equally exposed.    It’s tempting to say it’s a needle in a haystack.  But I promise you this:  Anybody patches Linux/net/ipv4/tcp_input.c (which handles inbound network for Linux), a hundred alerts are fired and many of them are not to individuals anyone would call friendly.  One guy, one night, patched OpenSSL.  Not enough defenders noticed, and it took Neel Mehta to do something.

    (tags: development openssl heartbleed ssl security dan-kaminsky infrastructure libraries open-source dependencies)

  • s3funnel

    ‘a command line tool for Amazon’s Simple Storage Service (S3). Written in Python, easy_install the package to install as an egg. Supports multithreaded operations for large volumes. Put, get, or delete many items concurrently, using a fixed-size pool of threads. Built on workerpool for multithreading and boto for access to the Amazon S3 API. Unix-friendly input and output. Pipe things in, out, and all around.’ MIT-licensed open source. (via Paul Dolan)

    (tags: via:pdolan s3 s3funnel tools ops aws python mit open-source)

Links for 2014-04-15

Published April 15, 2014

  • Hydra Takes On Hadoop

    The intuition behind Hydra is something like this, “I have a lot of data, and there are a lot of things I could try to learn about it — so many that I’m not even sure what I want to know.” It’s about the curse of dimensionality — more dimensions means exponentially more cost for exhaustive analysis. Hydra tries to make it easy to reduce the number of dimensions, or the cost of watching them (via probabilistic data structures), to just the right point where everything runs quickly but can still answer almost any question you think you might care about.
    Code: https://github.com/addthis/hydra Getting Started blog post: https://www.addthis.com/blog/2014/02/18/getting-started-with-hydra/

    (tags: hyrda hadoop data-processing big-data trees clusters analysis)

  • Stalled SCP and Hanging TCP Connections

    a Cisco fail.

    It looks like there’s a firewall in the middle that’s doing additional TCP sequence randomisation which was a good thing, but has been fixed in all current operating systems. Unfortunately, it seems that firewall doesn’t understand TCP SACK, which when coupled with a small amount of packet loss and a stateful host firewall that blocks invalid packets results in TCP connections that stall randomly. A little digging revealed that firewall to be the Cisco Firewall Services Module on our Canterbury network border.
    (via Tony Finch)

    (tags: via:fanf cisco networking firewalls scp tcp hangs sack tcpdump)

  • Akamai’s “Secure Heap” patch wasn’t good enough

    ‘Having the private keys inaccessible is a good defense in depth move. For this patch to work you have to make sure all sensitive values are stored in the secure area, not just check that the area looks inaccessible. You can’t do that by keeping the private key in the same process. A review by a security engineer would have prevented a false sense of security. A version where the private key and the calculations are in a separate process would be more secure. If you decide to write that version, I’ll gladly see if I can break that too.’ Akamai’s response: https://blogs.akamai.com/2014/04/heartbleed-update-v3.html — to their credit, they recognise that they need to take further action. (via Tony Finch)

    (tags: via:fanf cryptography openssl heartbleed akamai security ssl tls)

  • Shuffle Sharding

    Colm MacCarthaigh writes about a simple sharding/load-balancing algorithm which uses randomized instance selection and optional additional compartmentalization. See also: continuous hashing, and http://aphyr.com/posts/278-timelike-2-everything-fails-all-the-time

    (tags: hashing load-balancing sharding partitions dist-sys distcomp architecture coding)

  • Open Crypto Audit Project: TrueCrypt

    phase I, a source code audit by iSEC Partners, is now complete. Bruce Schneier says: “I’m still using it”.

    (tags: encryption security crypto truecrypt audits source-code isec matthew-green)

  • The science of ‘hangry’

    In the PNAS paper, Brad Bushman and colleagues looked at 107 couples over 21 days and found that people experiencing uncharacteristically low blood sugar were more likely to display anger toward their spouse. (The researchers measured this by having subjects stick needles into voodoo dolls representing their significant others.)

    (tags: hangry hunger food eating science health blood-sugar voodoo-dolls glucose)

  • insane ESB health and safety policy

    Where it is not possible to avoid reversing, it is ESB policy that staff driving on behalf of the company or anybody on company premises should reverse into car spaces/bays, allowing them to drive out subsequently.
    BUT WHYYYYYYYYYY

    (tags: esb health-n-safety policies crazy funny driving reversing lol safety)

Links for 2014-04-14

Published April 14, 2014

  • Cloudflare demonstrate Heartbleed key extraction

    from nginx. ‘Based on the findings, we recommend everyone reissue + revoke their private keys.’

    (tags: security nginx heartbleed ssl tls exploits private-keys)

  • When two-factor authentication is not enough

    Fastmail.FM nearly had their domain stolen through an attack exploiting missing 2FA authentication in Gandi.

    An important lesson learned is that just because a provider has a checkbox labelled “2 factor authentication” in their feature list, the two factors may not be protecting everything – and they may not even realise that fact themselves. Security risks always come on the unexpected paths – the “off label” uses that you didn’t think about, and the subtle interaction of multiple features which are useful and correct in isolation.

    (tags: gandi 2fa fastmail authentication security mfa two-factor-authentication mail)

  • Of Money, Responsibility, and Pride

    Steve Marquess of the OpenSSL Foundation on their funding, and lack thereof:

    I stand in awe of their talent and dedication, that of Stephen Henson in particular. It takes nerves of steel to work for many years on hundreds of thousands of lines of very complex code, with every line of code you touch visible to the world, knowing that code is used by banks, firewalls, weapons systems, web sites, smart phones, industry, government, everywhere. Knowing that you’ll be ignored and unappreciated until something goes wrong. The combination of the personality to handle that kind of pressure with the relevant technical skills and experience to effectively work on such software is a rare commodity, and those who have it are likely to already be a valued, well-rewarded, and jealously guarded resource of some company or worthy cause. For those reasons OpenSSL will always be undermanned, but the present situation can and should be improved. There should be at least a half dozen full time OpenSSL team members, not just one, able to concentrate on the care and feeding of OpenSSL without having to hustle commercial work. If you’re a corporate or government decision maker in a position to do something about it, give it some thought. Please. I’m getting old and weary and I’d like to retire someday.

    (tags: funding open-source openssl heartbleed internet security money)

  • Huginn

    a system for building agents that perform automated tasks for you online. They can read the web, watch for events, and take actions on your behalf. Huginn’s Agents create and consume events, propagating them along a directed event flow graph. Think of it as Yahoo! Pipes plus IFTTT on your own server. You always know who has your data. You do.
    MIT-licensed open source, built on Rails.

    (tags: ifttt automation huginn ruby rails open-source agents)

Links for 2014-04-13

Published April 13, 2014

Links for 2014-04-11

Published April 11, 2014

  • Basho LevelDB supports tiered storage

    Tiered storage is turning out to be a pretty practical trick to take advantage of SSDs:

    The justification for two types/speeds of storage arrays is simple. leveldb is extremely write intensive in its lower levels. The write intensity drops off as the level number increases. Similarly, current and frequently updated data tends to be in lower levels while archival data tends to be in higher levels. These leveldb characteristics create a desire to have faster, more expensive storage arrays for the high intensity lower levels. This branch allows the high intensity lower levels to be on expensive storage arrays while slower, less expensive storage arrays to hold the higher level data to reduce costs.

    (tags: caching tiered-storage storage ssds ebs leveldb basho patches riak iops)

  • Forbes on the skeleton crew nature of OpenSSL

    This is a great point:

    Obviously, those tending to the security protocols that support the rest of the Web need better infrastructure and more funding. “Large portions of the software infrastructure of the Internet are built and maintained by volunteers, who get little reward when their code works well but are blamed, and sometimes savagely derided, when it fails,” writes Foster in the New Yorker. […] “money and support still tend to flow to the newest and sexiest projects, while boring but essential elements like OpenSSL limp along as volunteer efforts,” he writes. “It’s easy to take open-source software for granted, and to forget that the Internet we use every day depends in part on the freely donated work of thousands of programmers.” We need to find ways to pay for work that is currently essentially donated freely. One promising project is Bithub, from Whisper Systems, where people who make valuable contributions to open source projects are rewarded (with Bitcoin of course). But the pool of Bitcoin is still donation based. The Internet has helped create a culture of free, but what we may need to recognize is that we get what we pay for. Well-funded companies pulling critical code from open source projects for their sites should have formal fee arrangements, rather than the volunteer group simply hoping these users will pony up some Benjamins for “prominent logo placement” on a website most people had never heard of before Heartbleed.

    (tags: open-source openssl free sponsorship forbes via:karl-whelan)

Links for 2014-04-10

Published April 10, 2014

Links for 2014-04-09

Published April 9, 2014

  • MICA: A Holistic Approach To Fast In-Memory Key-Value Storage [paper]

    Very interesting new approach to building a scalable in-memory K/V store. As Rajiv Kurian notes on the mechanical-sympathy list: ‘The basic idea is that each core is responsible for a portion of the key-space and requests are forwarded to the right core, avoiding multiple-writer scenarios. This is opposed to designs like memcache which uses locks and shared memory. Some of the things I found interesting: The single writer design is taken to an extreme. Clients assist the partitioning of requests, by calculating hashes before submitting GET requests. It uses Intel DPDK instead of sockets to forward packets to the right core, without processing the packet on any core. Each core is paired with a dedicated RX/TX queue. The design for a lossy cache is simple but interesting. It does things like replacing a hash slot (instead of chaining) etc. to take advantage of the lossy nature of caches. There is a lossless design too. A bunch of tricks to optimize for memory performance. This includes pre-allocation, design of the hash indexes, prefetching tricks etc. There are some other concurrency tricks that were interesting. Handling dangling pointers was one of them.’ Source code here: https://github.com/efficient/mica

    (tags: mica in-memory memory ram key-value-stores storage smp dpdk multicore memcached concurrency)

  • Google’s Open Bidder stack moving from Jetty to Netty

    Open Bidder traditionally used Jetty as an embedded webserver, for the critical tasks of accepting connections, processing HTTP requests, managing service threads, etc. Jetty is a robust, but traditional stack that carries the weight and tradeoffs of Servlet’s 15 years old design. For a maximum performance RTB agent that must combine very large request concurrency with very low latencies, and often benefit also from low-level control over the transport, memory management and other issue, a different webserver stack was required. Open Bidder now supports Netty, an asynchronous, event-driven, high-performance webserver stack. For existing code, the most important impact is that Netty is not compatible with the Servlet API. Its own internal APIs are often too low-level, not to mention proprietary to Netty; so Open Bidder v0.5 introduces some new, stack-neutral APIs for things like HTTP requests and responses, cookies, request handlers, and even simple HTML templating based on Mustache. These APIs will work with both Netty and Jetty. This means you don’t need to change any code to switch between Jetty and Netty; on the other hand, it also means that existing code written for Open Bidder 0.4 may need some changes even if you plan to keep using Jetty. [….] Netty’s superior efficiency is very significant; it supports 50% more traffic in the same hardware, and it maintains a perfect latency distribution even at the peak of its supported load.
    This doc is noteworthy on a couple of grounds: 1. the use of Netty in a public API/library, and the additional layer in place to add a friendlier API on top of that. I hope they might consider releasing that part as OSS at some point. 2. I also find it interesting that their API uses protobufs to marshal the message, and they plan in a future release to serialize those to JSON documents — that makes a lot of sense.

    (tags: apis google protobufs json documents interoperability netty jetty servlets performance java)

  • The University Times: TCD Provost Under Pressure To “Re-think” Identity Initiative

    Students, staff and alumni put pressure on Provost to reconsider changes to Trinity College Dublin’s name and coat of arms.

    alumni scholars from 2004 and 1994 who had been invited back for the dinner shouted ‘Dublin’ after the Provost welcomed them back to “Trinity College”.

    (tags: tcd tcuod rebranding fail identity dublin)

  • Daring Fireball: Rethinking What We Mean by ‘Mobile Web’

    We shouldn’t think of “the web” as only what renders in web browsers. We should think of the web as anything transmitted using HTTP and HTTPS. Apps and websites are peers, not competitors. They’re all just clients to the same services.
    +1. Finally, a Daring Fireball post I agree with! ;)

    (tags: daring-fireball apps web http https mobile apple android browsers)

Links for 2014-04-08

Published April 8, 2014

Links for 2014-04-07

Published April 7, 2014

Links for 2014-04-05

Published April 5, 2014

Links for 2014-04-03

Published April 3, 2014

Links for 2014-04-02

Published April 2, 2014

Links for 2014-04-01

Published April 1, 2014

Links for 2014-03-31

Published March 31, 2014

Links for 2014-03-28

Published March 28, 2014

  • “They Know Everything We Do”

    [via Boing Boing:] A new, exhaustive report from Human Rights Watch details the way the young state of modern Ethiopia has become a kind of pilot program for the abuse of “off-the-shelf” surveillance, availing itself of commercial products from the US, the UK, France, Italy and China in order to establish an abusive surveillance regime that violates human rights and suppresses legitimate political opposition under the guise of a anti-terrorism law that’s so broadly interpreted as to be meaningless. The 137 page report [from Human Rights Watch] details the technologies the Ethiopian government has acquired from several countries and uses to facilitate surveillance of perceived political opponents inside the country and among the diaspora. The government’s surveillance practices violate the rights to freedom of expression, association, and access to information. The government’s monopoly over all mobile and Internet services through its sole, state-owned telecom operator, Ethio Telecom, facilitates abuse of surveillance powers.

    (tags: human-rights surveillance ethiopia spying off-the-shelf spyware big-brother hrw human-rights-watch)

Links for 2014-03-26

Published March 26, 2014

  • Chinese cops cuff 1,500 in fake base station spam raid

    The street finds its own uses for things, in this case Stinger/IMSI-catcher-type fake mobile-phone base stations:

    Fake base stations are becoming a particularly popular modus operandi. Often concealed in a van or car, they are driven through city streets to spread their messages. The professional spammer in question charged 1,000 yuan (£100) to spam thousands of users in a radius of a few hundred metres. The pseudo-base station used could send out around 6,000 messages in just half an hour, the report said. Often such spammers are hired by local businessmen to promote their wares.
    (via Bernard Tyers)

    (tags: stingers imsi-catcher mobile-phones mobile cellphones china spam via:bernard-tyers)

  • TJ McIntyre on the incredible surveillance of telephone traffic at various Garda stations around the country

    The most grave issue is that each recording likely amounted to a serious criminal offence. Under Irish law, the recording of a telephone conversation on a public network without the consent of at least one party to the call amounts to an “interception”, a criminal offence carrying a possible term of imprisonment of up to five years. […] Consequently, unless gardai were notified that their calls might be recorded then a large number of criminal offences are likely to have been committed by and within the Garda Siochana itself.

    (tags: gubu surveillance gardai ags tjmcintyre bugging tapping phones ireland politics)

  • rr

    A cool-looking new debugging tool for C/C++ from Mozilla.

    Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.
    Low runtime overhead; easy deployability; targeted at 32-bit (?!) Linux; OSS. (via Bryan O’Sullivan)

    (tags: via:bos mozilla debugging coding firefox rr record replay gdb c++ linux)

  • Ask AIB – Boards.ie

    AIB now have a dedicated customer-support forum on Boards.ie. That is a *great* idea

    (tags: aib banking support forums boards.ie banks)

Links for 2014-03-25

Published March 25, 2014

  • Microservices and nanoservices

    A great reaction to Martin Fowler’s “microservices” coinage, from Arnon Rotem-Gal-Oz: ‘I guess it is easier to use a new name (Microservices) rather than say that this is what SOA actually meant’; ‘these are the very principles of SOA before vendors does pushed the [ESB] in the middle.’ Others have also chosen to define microservices slightly differently, as a service written in 10-100 LOC. Arnon’s reaction: “Nanoservice is an antipattern where a service is too fine-grained. A nanoservice is a service whose overhead (communications, maintenance, and so on) outweighs its utility.” Having dealt with maintaining an over-fine-grained SOA stack in Amazon, I can only agree with this definition; it’s easy to make things too fine-grained and create a raft of distributed-computing bugs and deployment/management complexity where there is no need to do so.

    (tags: architecture antipatterns nanoservices microservices soa services design esb)

  • Accidentally Turing-Complete

    slightly ruined by the inclusion of some “deliberately Turing-complete” systems

    (tags: turing computation software via:jwz turing-complete accidents automatons)

Links for 2014-03-24

Published March 24, 2014

Links for 2014-03-21

Published March 21, 2014

  • Microsoft “Scroogles” Itself

    ‘Microsoft went through a blogger’s private Hotmail account in order to trace the identity of a source who allegedly leaked trade secrets.’ Bear in mind that the alleged violation which MS allege allows them to read their email was a breach of the terms of service, which also include distribution of content which ‘incites, advocates, or expresses pornography, obscenity, vulgarity, [or] profanity’. So no dirty jokes on Hotmail!

    (tags: hotmail fail scroogled microsoft stupid tos law privacy data-protection trade-secrets ip)

  • Theresa May warns Yahoo that its move to Dublin is a security worry

    Y! is moving to Dublin to evade GCHQ spying on its users. And what is the UK response?

    “There are concerns in the Home Office about how Ripa will apply to Yahoo once it has moved its headquarters to Dublin,” said a Whitehall source. “The home secretary asked to see officials from Yahoo because in Dublin they don’t have equivalent laws to Ripa. This could particularly affect investigations led by Scotland Yard and the national crime agency. They regard this as a very serious issue.”
    There’s priorities for you!

    (tags: ripa gchq guardian uk privacy data-protection ireland dublin london spying surveillance yahoo)

  • A Look At Airbnb’s Irish Pub-Inspired Office In Dublin – DesignTAXI.com

    Very nice, Airbnb!

    (tags: airbnb design offices work pubs ireland dublin)

  • Internet Tolls And The Case For Strong Net Neutrality

    Netflix CEO Reed Hastings blogs about the need for Net Neutrality:

    Interestingly, there is one special case where no-fee interconnection is embraced by the big ISPs — when they are connecting among themselves. They argue this is because roughly the same amount of data comes and goes between their networks. But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download** — thus filling their upstream networks and nearly doubling our total traffic — there is an uncomfortable silence. That’s because the ISP argument isn’t sensible. Big ISPs aren’t paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs. ISPs around the world are investing in high-speed Internet and most already practice strong net neutrality. With strong net neutrality, new services requiring high-speed Internet can emerge and become popular, spurring even more demand for the lucrative high-speed packages ISPs offer. With strong net neutrality, everyone avoids the kind of brinkmanship over blackouts that plague the cable industry and harms consumers. As the Wall Street Journal chart shows, we’re already getting to the brownout stage. Consumers deserve better.

    (tags: consumer net-neutrality comcast netflix protectionism cartels isps us congestion capacity)

  • Micro jitter, busy waiting and binding CPUs

    pinning threads to CPUs to reduce jitter and latency. Lots of graphs and measurements from Peter Lawrey

    (tags: pinning threads performance latency jitter tuning)

  • The Day Today – Pool Supervisor – YouTube

    “in 1979, no-one died. in 1980, some one died. in 1981, no-one died. in 1982, no-one died. … I could go on”

    (tags: the-day-today no-one-died safety pool supervisor tricky-word-puzzles funny humour classic video)

  • The colossal arrogance of Newsweek’s Bitcoin “scoop” | Ars Technica

    Many aspects of the story already look like a caricature of journalism gone awry. The man Goodman fingered as being worth $400 million or more is just as modest as his house suggests. He’s had a stroke and struggles with other health issues. Unemployed since 2001, he strives to take care of basic needs for himself and his 93-year-old mother, according to a reddit post by his brother Arthur Nakamoto (whom Goodman quoted as calling his brother an “asshole”). If Goodman has mystery evidence supporting the Dorian Nakamoto theory, it should have been revealed days ago. Otherwise, Newsweek and Goodman are delaying an inevitable comeuppance and doubling down on past mistakes. Nakamoto’s multiple denials on the record have changed the dynamic of the story. Standing by the story, at this point, is an attack on him and his credibility. The Dorian Nakamoto story is a “Dewey beats Truman” moment for the Internet age, with all of the hubris and none of the humor. It shouldn’t be allowed to end in the mists of “he said, she said.” Whether or not a lawsuit gets filed, Nakamoto v. Newsweek faces an imminent verdict in the court of public opinion: either the man is lying or the magazine is wrong.

    (tags: dorian-nakamoto newsweek journalism bitcoin privacy satoshi-nakamoto)

  • Papa’s Maze | spoon & tamago

    While going through her papa’s old belongings, a young girl discovered something incredible – a mind-bogglingly intricate maze that her father had drawn by hand 30 years ago. While working as a school janitor it had taken him 7 years to produce the piece, only for it to be forgotten about… until now.
    34″ x 24″ print, $40

    (tags: mazes art prints weird papas-maze japan)

  • Continuous Delivery with ETL Systems

    Lonely Planet and Dr Foster Intelligence both make heavy use of ETL in their products, and both organisations have applied the principles of Continuous Delivery to their delivery process. Some of the Continuous Delivery norms need to be adapted in the context of ETL, and some interesting patterns emerge, such as running Continuous Integration against data, as well as code.

    (tags: etl video presentations lonely-planet dr-foster-intelligence continuous-delivery deployment pipelines)

  • The MtGox 500

    ‘On March 9th a group posted a data leak, which included the trading history of all MtGox users from April 2011 to November 2013. The graphs below explore the trade behaviors of the 500 highest volume MtGox users from the leaked data set. These are the Bitcoin barons, wealthy speculators, dueling algorithms, greater fools, and many more who took bitcoin to the moon.’

    (tags: dataviz stamen bitcoin data leaks mtgox greater-fools)

  • What We Know 2/5/14: The Mt. Chiliad Mystery

    hats off to Rockstar — GTA V has a great mystery mural with clues dotted throughout the game, and it’s as-yet unsolved

    (tags: mysteries gaming via:hilary_w games gta gta-v rockstar mount-chiliad ufos)

  • Make Your Own 3-D Printer Filament From Old Milk Jugs

    Creating your own 3-D printer filament from old used milk jugs is exponentially cheaper, and uses considerably less energy, than buying new filament, according to new research from Michigan Technological University. […] The savings are really quite impressive — 99 cents on the dollar, in addition to the reduced use of energy. Interestingly (but again not surprisingly), the amount of energy used to ‘recycle’ the old milk jugs yourself is considerably less than that used in recycling such jugs conventionally.

    (tags: recycling 3d-printers printing tech plastic milk)

Links for 2014-03-20

Published March 20, 2014

Links for 2014-03-19

Published March 19, 2014

  • No, Nate, brogrammers may not be macho, but that’s not all there is to it

    Great essay on sexism in tech, “brogrammer” culture, “clubhouse chemistry”, outsiders, wierd nerds and exclusion:

    Every group, including the excluded and disadvantaged, create cultural capital and behave in ways that simultaneously create a sense of belonging for them in their existing social circle while also potentially denying them entry into another one, often at the expense of economic capital. It’s easy to see that wearing baggy, sagging pants to a job interview, or having large and visible tattoos in a corporate setting, might limit someone’s access. These are some of the markers of belonging used in social groups that are often denied opportunities. By embracing these markers, members of the group create real barriers to acceptance outside their circle even as they deepen their peer relationships. The group chooses to adopt values that are rejected by the society that’s rejecting them. And that’s what happens to “weird nerd” men as well—they create ways of being that allow for internal bonding against a largely exclusionary backdrop.
    (via Bryan O’Sullivan)

    (tags: nerds outsiders exclusion society nate-silver brogrammers sexism racism tech culture silicon-valley essays via:bos31337)

  • Impact of large primitive arrays (BLOBS) on JVM Garbage Collection

    some nice graphs and data on CMS performance, with/without -XX:ParGCCardsPerStrideChunk

    (tags: cms java jvm performance optimization tuning off-heap-storage memory)

  • Anatomical Collages by Travis Bedel

    these are fantastic

    (tags: collage anatomy art prints)

  • htcat/htcat

    a utility to perform parallel, pipelined execution of a single HTTP GET. htcat is intended for the purpose of incantations like: htcat https://host.net/file.tar.gz | tar -zx It is tuned (and only really useful) for faster interconnects: [….] 109MB/s on a gigabit network, between an AWS EC2 instance and S3. This represents 91% use of the theoretical maximum of gigabit (119.2 MiB/s).

    (tags: go cli http file-transfer ops tools)

Links for 2014-03-18

Published March 18, 2014

  • Analyzing Citibike Usage

    Abe Stanway crunches the stats on Citibike usage in NYC, compared to the weather data from Wunderground.

    (tags: data correlation statistics citibike cycling nyc data-science weather)

  • NSA surveillance recording every single voice call in at least 1 country

    Storing them in a 30-day rolling buffer, allowing retrospective targeting weeks after the call. 100% of all voice calls in that country, although it’s unclear which country that is

    (tags: nsa surveillance gchq telephones phone bugging)

  • S3QL

    a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X. S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.

    (tags: s3 s3ql backup aws filesystems linux freebsd osx ops)

  • What’s New in Java 8

    good explanation of all the new features — I’m really looking forward to fixing up all the crappy over-verbose interface-as-lambdas we have scattered throughout our code

    (tags: java java8 lambdas fp functional-programming currying joda-time)

  • FM-index

    a compressed full-text substring index based on the Burrows-Wheeler transform, with some similarities to the suffix array. It was created by Paolo Ferragina and Giovanni Manzini,[1] who describe it as an opportunistic data structure as it allows compression of the input text while still permitting fast substring queries. The name stands for ‘Full-text index in Minute space’. It can be used to efficiently find the number of occurrences of a pattern within the compressed text, as well as locate the position of each occurrence. Both the query time and storage space requirements are sublinear with respect to the size of the input data.
    kragen notes ‘gene sequencing is using [them] in production’.

    (tags: sequencing bioinformatics algorithms bowtie fm-index indexing compression search burrows-wheeler bwt full-text-search)

Links for 2014-03-14

Published March 14, 2014

  • Health privacy: formal complaint to ICO

    ‘Light Blue Touchpaper’ notes:

    Three NGOs have lodged a formal complaint to the Information Commissioner about the fact that PA Consulting uploaded over a decade of UK hospital records to a US-based cloud service. This appears to have involved serious breaches of the UK Data Protection Act 1998 and of multiple NHS regulations about the security of personal health information.
    Let’s see if ICO can ever do anything useful…. not holding my breath

    (tags: ico privacy data-protection dpa nhs health data ross-anderson)

  • Why Google Flu Trends Can’t Track the Flu (Yet)

    It’s admittedly hard for outsiders to analyze Google Flu Trends, because the company doesn’t make public the specific search terms it uses as raw data, or the particular algorithm it uses to convert the frequency of these terms into flu assessments. But the researchers did their best to infer the terms by using Google Correlate, a service that allows you to look at the rates of particular search terms over time. When the researchers did this for a variety of flu-related queries over the past few years, they found that a couple key searches (those for flu treatments, and those asking how to differentiate the flu from the cold) tracked more closely with Google Flu Trends’ estimates than with actual flu rates, especially when Google overestimated the prevalence of the ailment. These particular searches, it seems, could be a huge part of the inaccuracy problem. There’s another good reason to suspect this might be the case. In 2011, as part of one of its regular search algorithm tweaks, Google began recommending related search terms for many queries (including listing a search for flu treatments after someone Googled many flu-related terms) and in 2012, the company began providing potential diagnoses in response to symptoms in searches (including listing both “flu” and “cold” after a search that included the phrase “sore throat,” for instance, perhaps prompting a user to search for how to distinguish between the two). These tweaks, the researchers argue, likely artificially drove up the rates of the searches they identified as responsible for Google’s overestimates.
    via Boing Boing

    (tags: google flu trends feedback side-effects colds health google-flu-trends)

Links for 2014-03-13

Published March 13, 2014

Links for 2014-03-12

Published March 12, 2014

  • Sacked Google worker says staff ratings fixed to fit template

    Allegations of fixing to fit the stack-ranking curve: ‘someone at Google always had to get a low score “of 2.9”, so the unit could match the bell curve. She said senior staff “calibrated” the ratings supplied by line managers to ensure conformity with the template and these calibrations could reduce a line manager’s assessment of an employee, in effect giving them the poisoned score of less than three.’

    (tags: stack-ranking google ireland employment work bell-curve statistics eric-schmidt)

  • Corporate Tax 2014: Irish Government’s “flawed premise” on Apple’s avoidance

    According to our calculation about €40bn or over 40% of Irish services exports of €90bn in 2012 and related national output, resulted from global tax avoidance schemes. It is true that Ireland gains little from tax cheating but at some point, the US tax system will be reformed and a territorial system where companies are only liable in the US on US profits, would only be viable if there was a disincentive to shift profits to non-tax or low tax countries. The risk for Ireland is that a minimum foreign tax would be introduced that would be greater than the Irish headline rate of 12.5%. It’s also likely that US investment in Ireland would not have been jeopardized if Irish politicians had not been so eager as supplicants to doff the cap. Nevertheless today it would be taboo to admit the reality of participation in massive tax avoidance and the Captain Renaults of Merrion Street will continue with their version of the Dance of the Seven Veils.

    (tags: apple tax double-irish tax-avoidance google investment itax tax-evasion ireland)

  • An online Magna Carta: Berners-Lee calls for bill of rights for web

    TimBL backing the “web we want” campaign — https://webwewant.org/

    (tags: freedom gchq nsa censorship internet privacy web-we-want human-rights timbl tim-berners-lee)

  • How the search for flight AF447 used Bayesian inference

    Via jgc, the search for the downed Air France flight was optimized using this technique: ‘Metron’s approach to this search planning problem is rooted in classical Bayesian inference, which allows organization of available data with associated uncertainties and computation of the Probability Distribution Function (PDF) for target location given these data. In following this approach, the first step was to gather the available information about the location of the impact site of the aircraft. This information was sometimes contradictory and filled with ambiguities and uncertainties. Using a Bayesian approach we organized this material into consistent scenarios, quantified the uncertainties with probability distributions, weighted the relative likelihood of each scenario, and performed a simulation to produce a prior PDF for the location of the wreck.’

    (tags: metron bayes bayesian-inference machine-learning statistics via:jgc air-france disasters probability inference searching)

  • How the NSA Plans to Infect ‘Millions’ of Computers with Malware – The Intercept

    The implants being deployed were once reserved for a few hundred hard-to-reach targets, whose communications could not be monitored through traditional wiretaps. But the documents analyzed by The Intercept show how the NSA has aggressively accelerated its hacking initiatives in the past decade by computerizing some processes previously handled by humans. The automated system – codenamed TURBINE – is designed to “allow the current implant network to scale to large size (millions of implants) by creating a system that does automated control implants by groups instead of individually.” In a top-secret presentation, dated August 2009, the NSA describes a pre-programmed part of the covert infrastructure called the “Expert System,” which is designed to operate “like the brain.”
    Great. Automated malware deployment to millions of random victims. See also the “I hunt sysadmins” section further down…

    (tags: malware gchq nsa oversight infection expert-systems turbine false-positives the-intercept surveillance)

Links for 2014-03-11

Published March 11, 2014

Links for 2014-03-10

Published March 10, 2014

Links for 2014-03-06

Published March 6, 2014

Links for 2014-03-05

Published March 5, 2014

  • A cautionary tale about building large-scale polyglot systems

    ‘a fucking nightmare’:

    Cascading requires a compilation step, yet since you’re writing Ruby code, you get get none of the benefits of static type checking. It was standard to discover a type issue only after kicking off a job on, oh, 10 EC2 machines, only to have it fail because of a type mismatch. And user code embedded in strings would regularly fail to compile – which you again wouldn’t discover until after your job was running. Each of these were bad individually, together, they were a fucking nightmare. The interaction between the code in strings and the type system was the worst of all possible worlds. No type checking, yet incredibly brittle, finicky and incomprehensible type errors at run time. I will never forget when one of my friends at Etsy was learning Cascading.JRuby and he couldn’t get a type cast to work. I happened to know what would work: a triple cast. You had to cast the value to the type you wanted, not once, not twice, but THREE times.

    (tags: etsy scalding cascading adtuitive war-stories languages polyglot ruby java strong-typing jruby types hadoop)

  • It’s So Easy

    Attempting to cash out of Bitcoins turns out to be absurdly difficult:

    Trying to sell the coins in person, and basically saying he ether wants Cash, or a Cashiers check (since it can be handed over right then and there), has apparently been a hilarious clusterfuck. Today he met some guy infront of his bank, and apparently as soon as he mentioned that he needs to get the cash checked to make sure it is not counterfeit, the guy freaked out and basically walked away. Stuff like this has been happening all week, and he apparently so far has only sold a single coin of several hundred.

    (tags: bitcoin fail funny mtgox fraud cash fiat-currency via:rsynnott buttcoin)

  • Florida cops used IMSI catchers over 200 times without a warrant

    Harris is the leading maker of [IMSI catchers aka “stingrays”] in the U.S., and the ACLU has long suspected that the company has been loaning the devices to police departments throughout the state for product testing and promotional purposes. As the court document notes in the 2008 case, “the Tallahassee Police Department is not the owner of the equipment.” The ACLU now suspects these police departments may have all signed non-disclosure agreements with the vendor and used the agreement to avoid disclosing their use of the equipment to courts. “The police seem to have interpreted the agreement to bar them even from revealing their use of Stingrays to judges, who we usually rely on to provide oversight of police investigations,” the ACLU writes.

    (tags: aclu police stingrays imsi-catchers privacy cellphones mobile-phones security wired)

Links for 2014-03-04

Published March 4, 2014

Links for 2014-03-02

Published March 2, 2014

  • Answer to How many topics (queues) can be created in Apache Kafka? – Quora

    Good to know:

    ‘As far as I understand (this was true as of 2013, when I last looked into this issue) there’s at least one Apache ZooKeeper znode per topic in Kafka. While there is no hard limitation in Kafka itself (Kafka is linearly scalable), it does mean that the maximum number of znodes comfortable supported by ZooKeeper (on the order of about ten thousand) is the upper limit of Kafka’s scalability as far as the number of topics goes.’

    (tags: kafka queues zookeeper znodes architecture)

Links for 2014-03-01

Published March 1, 2014

  • Care.data is in chaos. It breaks my heart | Ben Goldacre

    There are people in my profession who think they can ignore this problem. Some are murmuring that this mess is like MMR, a public misunderstanding to be corrected with better PR. They are wrong: it’s like nuclear power. Medical data, rarefied and condensed, presents huge power to do good, but it also presents huge risks. When leaked, it cannot be unleaked; when lost, public trust will take decades to regain. This breaks my heart. I love big medical datasets, I work on them in my day job, and I can think of a hundred life-saving uses for better ones. But patients’ medical records contain secrets, and we owe them our highest protection. Where we use them – and we have used them, as researchers, for decades without a leak – this must be done safely, accountably, and transparently. New primary legislation, governing who has access to what, must be written: but that’s not enough. We also need vicious penalties for anyone leaking medical records; and HSCIC needs to regain trust, by releasing all documentation on all past releases, urgently. Care.data needs to work: in medicine, data saves lives.

    (tags: hscic nhs care.data data privacy data-protection medicine hospitals pr)

Links for 2014-02-27

Published February 27, 2014