Skip to content

Author: dailylinks

Links for 2014-01-15

Published January 15, 2014

Links for 2014-01-14

Published January 14, 2014

Links for 2014-01-13

Published January 13, 2014

Links for 2014-01-11

Published January 11, 2014

  • Growing up unvaccinated: A healthy lifestyle couldn’t prevent many childhood illnesses.

    I understand, to a point, where the anti-vaccine parents are coming from. Back in the ’90s, when I was a concerned, 19-year-old mother, frightened by the world I was bringing my child into, I was studying homeopathy, herbalism, and aromatherapy; I believed in angels, witchcraft, clairvoyants, crop circles, aliens at Nazca, giant ginger mariners spreading their knowledge to the Aztecs, the Incas, and the Egyptians, and that I was somehow personally blessed by the Holy Spirit with healing abilities. I was having my aura read at a hefty price and filtering the fluoride out of my water. I was choosing to have past life regressions instead of taking antidepressants. I was taking my daily advice from tarot cards. I grew all my own veg and made my own herbal remedies. I was so freaking crunchy that I literally crumbled. It was only when I took control of those paranoid thoughts and fears about the world around me and became an objective critical thinker that I got well. It was when I stopped taking sugar pills for everything and started seeing medical professionals that I began to thrive physically and mentally.

    (tags: health medicine science vaccination disease slate)

Links for 2014-01-09

Published January 9, 2014

Links for 2014-01-07

Published January 7, 2014

Links for 2014-01-06

Published January 6, 2014

Links for 2014-01-02

Published January 2, 2014

  • Dogs like to excrete in alignment with the Earth's magnetic field

    Dogs preferred to excrete with the body being aligned along the North-south axis under calm magnetic field conditions.

    (tags: dogs poo excrement shit magnetic-field earth zoology papers)

  • Paul Graham and the Manic Pixie Dream Hacker

    Under Graham’s influence, Mark [Zuckerberg], like many in Silicon Valley, subscribes to the Manic Pixie Dream Hacker ideal, making self-started teenage hackers Facebook’s most desired recruiting targets, not even so much for their coding ability as their ability to serve as the faces of hacking culture. “Culture fit”, in this sense, is one’s ability to conform to the Valley’s boyish hacker fantasy, which is easier, obviously, the closer you are to a teenage boy. Like the Manic Pixie Dream Girl’s role of existing to serve the male film protagonist’s personal growth, the Manic Pixie Dream Hacker’s job is to embody the dream hacker role while growing the VC’s portfolio. This is why the dream hacker never ages, never visibly develops interests beyond hardware and code, and doesn’t question why nearly all the other people receiving funding look like him. Like the actress playing the pixie dream girl, the pixie dream boy isn’t being paid to question the role for which he has been cast. In this way, for all his supposed “disruptiveness”, the hacker pixie actually does exactly what he is told: to embody, while he can, the ideal hacker, until he is no longer young, mono-focused, and boyish-seeming enough to qualify for the role (at that point, vested equity may allow him to retire). And like in Hollywood, VCs will have already recruited newer, younger ones to play him.

    (tags: hackers manic-pixie-dream-girl culture-fit silicon-valley mark-zuckerberg paul-graham y-combinator vc work investment technology recruitment facebook ageism equality sexism)

  • The How and Why of Flapjack

    Flapjack aims to be a flexible notification system that handles: Alert routing (determining who should receive alerts based on interest, time of day, scheduled maintenance, etc); Alert summarisation (with per-user, per media summary thresholds); Your standard operational tasks (setting scheduled maintenance, acknowledgements, etc). Flapjack sits downstream of your check execution engine (like Nagios, Sensu, Icinga, or cron), processing events to determine if a problem has been detected, who should know about the problem, and how they should be told.

    (tags: flapjack notification alerts ops nagios paging sensu)

Links for 2013-12-27

Published December 27, 2013

  • Dublin Cycle Planner needs a health warning - Irish Cycle

    An extensive catalogue of shitty routing. Poor...

    It’s expected that any new mapping and routing systems will have errors which will need to be ironed out but the level of issues with the NTA Cycle Planner is far beyond what you’d expect in a light and quiet beta launch. It’s beyond acceptable for a public PR launch directing people to a route planner with no clear warnings. It looks like a rush job which allows junior minister Alan Kelly to get his name in another press release before the end of the year.

    (tags: cycling dublin commute mapping nta ireland maps)

  • Reflected hidden faces in photographs revealed in pupil

    The pupil of the eye in a photograph of a face can be mined for hidden information, such as reflected faces of the photographer and bystanders, according to research led by Dr. Rob Jenkins, of the Department of Psychology at the University of York and published in PLOS ONE (open access).
    (via Waxy)

    (tags: via:waxy future zoom-and-enhance privacy photography eyes photos)

Links for 2013-12-23

Published December 23, 2013

  • Jesse Willms, the Dark Lord of the Internet - Taylor Clark - The Atlantic

    “It was an out-and-out hijacking,” LeFevre told me. “They counterfeited our product, they pirated our Web site, and they basically directed all of their customer service to us.” At the peak of Willms’s sales, LeFevre says, dazzlesmile was receiving 1,000 calls a day from customers trying to cancel orders for a product it didn’t even sell. When irate consumers made the name dazzlesmile synonymous with online scamming, LeFevre’s sales effectively dropped to zero. Dazzlesmile sued Willms in November 2009; he later paid a settlement.

    (tags: scams hijacking ads affiliate one-wierd-trick health dieting crime)

Links for 2013-12-21

Published December 21, 2013

Links for 2013-12-19

Published December 19, 2013

Links for 2013-12-16

Published December 16, 2013

Links for 2013-12-13

Published December 13, 2013

  • Karlin Lillington on DRI's looming victory in the European Court of Justice

    If the full European Court of Justice (ECJ) accepts the opinion of its advocate general in a final ruling due early next year – and it almost always does – it will prove a huge vindication of Ireland’s small privacy advocacy group, Digital Rights Ireland (DRI). Its case against Irish retention laws, which began in 2006, forms the basis of this broader David v Goliath challenge and initial opinion. The advocate general’s advice largely upholds the key concerns put forward by DRI against Ireland’s laws. Withholding so much data about every citizen, including children, in case someone commits a future crime, is too intrusive into private life, and could allow authorities to create a “faithful and exhaustive map of a large portion of a person’s [private] conduct”. Retained data is so comprehensive that they could easily reveal private identities, which are supposed to remain anonymous. And the data, entrusted to third parties, is at too much risk of fraudulent or malicious use. Cruz Villalón argues that there must be far greater oversight to the retention process, and controls on access to data, and that citizens should have the right to be notified after the fact if their data has been scrutinised. The Irish Government had repeatedly waved off such concerns from Digital Rights Ireland in the past.

    (tags: dri rights ireland internet surveillance data-retention privacy eu ecj law)

Links for 2013-12-11

Published December 11, 2013

Links for 2013-12-10

Published December 10, 2013

Links for 2013-12-09

Published December 9, 2013

  • Cyanite

    a metric storage daemon, exposing both a carbon listener and a simple web service. Its aim is to become a simple, scalable and drop-in replacement for graphite's backend.
    Pretty alpha for now, but definitely worth keeping an eye on to potentially replace our burgeoning Carbon fleet...

    (tags: graphite carbon cassandra storage metrics ops graphs service-metrics)

  • Twitter tech talk video: "Profiling Java In Production"

    In this talk Kaushik Srenevasan describes a new, low overhead, full-stack tool (based on the Linux perf profiler and infrastructure built into the Hotspot JVM) we've built at Twitter to solve the problem of dynamically profiling and tracing the behavior of applications (including managed runtimes) in production.
    Looks very interesting. Haven't watched it yet though

    (tags: twitter tech-talks video presentations java jvm profiling testing monitoring service-metrics performance production hotspot perf)

  • Spy agencies in covert push to infiltrate virtual world of online gaming

    [MMOGs], the [NSA] analyst wrote, "are an opportunity!". According to the briefing notes, so many different US intelligence agents were conducting operations inside games that a "deconfliction" group was required to ensure they weren't spying on, or interfering with, each other.

    (tags: spies spying games mmog online surveillance absurd east-germany funny warcraft)

  • Ryan Lizza: Why Won’t Obama Rein in the N.S.A.? : The New Yorker

    Fantastic wrap-up of the story so far on the pervasive global surveillance story.

    The history of the intelligence community, though, reveals a willingness to violate the spirit and the letter of the law, even with oversight. What’s more, the benefits of the domestic-surveillance programs remain unclear. Wyden contends that the N.S.A. could find other ways to get the information it says it needs. Even Olsen, when pressed, suggested that the N.S.A. could make do without the bulk-collection program. “In some cases, it’s a bit of an insurance policy,” he told me. “It’s a way to do what we otherwise could do, but do it a little bit more quickly.” In recent years, Americans have become accustomed to the idea of advertisers gathering wide swaths of information about their private transactions. The N.S.A.’s collecting of data looks a lot like what Facebook does, but it is fundamentally different. It inverts the crucial legal principle of probable cause: the government may not seize or inspect private property or information without evidence of a crime. The N.S.A. contends that it needs haystacks in order to find the terrorist needle. Its definition of a haystack is expanding; there are indications that, under the auspices of the “business records” provision of the Patriot Act, the intelligence community is now trying to assemble databases of financial transactions and cell-phone location information. Feinstein maintains that data collection is not surveillance. But it is no longer clear if there is a distinction.

    (tags: nsa gchq surveillance spying privacy dianne-feinstein new-yorker journalism long-reads us-politics probable-cause)

Links for 2013-12-07

Published December 7, 2013

  • Same Old Stories From Sean Sherlock

    Sherlock’s record is spotty at best when it comes to engagement. Setting aside the 80,680 people who were ignored by the minister, he was hostile and counter productive to debate from the beginning, going so far as to threaten to pull out of a public debate because a campaigner against the ['Irish SOPA'] SI would be in attendance. His habit of blocking people online who publicly ask him tough yet legitimate questions has earned him the nickname “Sherblock”.

    (tags: sean-sherlock sherblock labour ireland politics blocking filtering internet freedom copyright emi music law piracy debate twitter)

  • Smart Metering in the UK is FCUKED

    Most utilities don’t want smart metering.  In fact they seem to have used the wrong dictionary.  It is difficult to find anything smart about the UK deployment, until you realise that the utilities use smart in the sense of “it hurts”.  They consider they have a perfectly adequate business model which has no need for new technology.  In many Government meetings, their reluctant support seems to be a veneer for the hope that it will all end in disaster, letting them go back to the world they know, of inflated bills and demands for money with menaces. [...] Even when smart meters are deployed, there is no evidence that any utility will use the resulting data to transform their business, rather than persecute the consumer.  At a recent US conference a senior executive for a US utility which had deployed smart meters, stated that their main benefit was “to give them more evidence to blame the customer”.  That’s a good description of the attitude displayed by our utilities.

    (tags: smart-metering energy utilities uk services metering consumer)

  • Kelly "kellabyte" Sommers on Redis' "relaxed CP" approach to the CAP theorem

    Similar to ACID properties, if you partially provide properties it means the user has to _still_ consider in their application that the property doesn't exist, because sometimes it doesn't. In you're fsync example, if fsync is relaxed and there are no replicas, you cannot consider the database durable, just like you can't consider Redis a CP system. It can't be counted on for guarantees to be delivered. This is why I say these systems are hard for users to reason about. Systems that partially offer guarantees require in-depth knowledge of the nuances to properly use the tool. Systems that explicitly make the trade-offs in the designs are easier to reason about because it is more obvious and _predictable_.

    (tags: kellabyte redis cp ap cap-theorem consistency outages reliability ops database storage distcomp)

  • Building a Balanced Universe - EVE Community

    Good blog post about EVE's algorithm to load-balance a 3D map of star systems

    (tags: eve eve-online algorithms 3d space load-balancing sharding games)

  • Virtual Clock - Testing Patterns Encyclopedia

    a nice pattern for unit tests which need deterministic time behaviour. Trying to think up a really nice API for this....

    (tags: testing unit-tests time virtual-clock real-time coding)

  • We're sending out the wrong signals in bid to lure the big data bucks - Independent.ie

    Simon McGarr on Ireland's looming data-protection train-crash.

    Last week, during the debate of his proposals to increase fees for making a Freedom of Information request, Brendan Howlin was asked how one of his amendments would affect citizens looking for data from the State's electronic databases. His reply was to cheerfully admit he didn't even understand the question. "I have no idea what an SQL code is. Does anyone know what an SQL code is?" Unlike the minister, it probably isn't your job to know that SQL is the computer language that underpins the data industry. The amendment he had originally proposed would have effectively allowed civil servants to pretend that their computer files were made of paper when deciding whether a request was reasonable. His answer showed how the Government could have proposed such an absurd idea in the first place. Like it or not – fair or not – these are not the signals a country that wanted to build a long-term data industry would choose to send out. They are the sort of signals that Ireland used to send out about Financial Regulation. I think it's agreed, that approach didn't work out so well.

    (tags: foi ireland brendan-howlin technology illiteracy sql civil-service government data-protection privacy regulation dpa)

Links for 2013-12-04

Published December 4, 2013

  • wrk

    a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.  An optional LuaJIT script can perform HTTP request generation, response processing, and custom reporting.
    Written in C, ASL2 licensed.

    (tags: wrk benchmarking http performance testing lua load-testing load-generation)

  • Removing DRM Boosts Music Sales by 10%

    Based on a working paper from University of Toronto researcher Laurina Zhang

    Comparing album sales of four major labels before and after the removal of DRM reveals that digital music revenue increases by 10% when restrictions are removed. The effect goes up to 30% for long tail content, while top-selling albums show no significant jump. The findings suggest that dropping technical restrictions can benefit both artists and the major labels.
    more details: http://inside.rotman.utoronto.ca/laurinazhang/files/2013/11/laurina_zhang_jmp_nov4.pdf , "Intellectual Property Strategy and the Long Tail: Evidence from the Recorded Music Industry", Laurina Zhang, November 4, 2013

    (tags: ip copyright drm mp3 music laurina-zhang research long-tail albums rights-management piracy)

  • 100 Years of Breed “Improvement” | Science of Dogs

    The English bulldog has come to symbolize all that is wrong with the dog fancy and not without good reason; they suffer from almost every possible disease. A 2004 survey by the Kennel Club found that they die at the median age of 6.25 years (n=180). There really is no such thing as a healthy bulldog. The bulldog’s monstrous proportions makes them virtually incapable of mating or birthing without medical intervention.
    (via Bryan)

    (tags: dogs eugenics breeding horror science genetics traits animals pets bulldog pedigree)

  • SkyJack - autonomous drone hacking

    Samy Kamkar strikes again. 'Using a Parrot AR.Drone 2, a Raspberry Pi, a USB battery, an Alfa AWUS036H wireless transmitter, aircrack-ng, node-ar-drone, node.js, and my SkyJack software, I developed a drone that flies around, seeks the wireless signal of any other drone in the area, forcefully disconnects the wireless connection of the true owner of the target drone, then authenticates with the target drone pretending to be its owner, then feeds commands to it and all other possessed zombie drones at my will.'

    (tags: drones amazon hacking security samy-kamkar aircrack node raspberry-pi airborne-zombies)

  • Why Did 9,000 Porny Spambots Descend on This San Diego High Schooler? - Alexis C. Madrigal - The Atlantic

    Good article about emergent behaviour from networked malware: 'The metabot, therefore, is viral. You get followed because of who follows you. This tendency explains the strange geographical cluster among San Diego high school students. Perhaps one of those kids was being followed by a really popular account (like @Interscope records, perhaps, which follows hundreds of thousands of people), and through that link, the bot stumbled into this little circle of San Diego teens. All of this activity would have remained under the radar, of course, all part of the silent non-human web. Except something went awry. For some reason, Olivia got stuck in a weird loop, and the metabot kept spawning spambots that chose to follow her over and over, relentlessly. Maybe once the metabot reached the San Diego kids, a bug kicked in. Instead of negative feedback keeping her (and everyone else) from being followed too often, we got runaway positive feedback. The bots followed her because other bots followed her. And on and on. Which is, perhaps a kind of reasoning that we can understand: It's the core logic of fame and celebrity itself. Attention flows to Snooki because attention flowed to Snooki. Attention flows to Olivia because attention flowed to Olivia. Olivia and her friends weren't wrong when they thought she'd become suddenly famous. Her audience just wasn't human.'

    (tags: socialnetworking spam twitter bots fame alexis-madrigal)

Links for 2013-12-03

Published December 3, 2013

Links for 2013-12-02

Published December 2, 2013

Links for 2013-11-28

Published November 28, 2013

Links for 2013-11-26

Published November 26, 2013

Links for 2013-11-25

Published November 25, 2013

Links for 2013-11-21

Published November 21, 2013

  • Shadows in the Woods

    beautiful German boardgame, suitable for playing with kids -- an adult moves a tealight candle around the board, while kids take turns moving gnomes around in the shadows behind tall "trees". recommended by JK

    (tags: games boardgames german kids candles light)

  • 'No basis in law' : Gardai probe Ballyphehane group after raid

    Freemen wackiness in Cork.

    The house of one member of the group was raided by gardaí last week, but it is not thought that any arrests were made, according to an eyewitness. Gardaí broke down the front door of the house. The group, which appears to be part of the Freemen of the Land movement, which does not recognise the State, has attempted to hold 'trials' in Ballyphehane Community Centre. It attempted to summon HSE staff, gardaí, social workers, solicitors and others to appear to be tried by a self-selected jury earlier this month. The group handed out documents purporting to be a summons to HSE staff and garda stations, demanding that named people attend a trial by 'éire court' on Tuesday 5 November at 9am “to stand trial for their acts of terrorism against mothers, their offspring and others in our community”, according to the group's literature. This week the group has begun posting about UCC, saying the college is “a private for profit corporation, and a business partner of and partly owned by Pfizers and Bank of Ireland”. The group suggest that UCC bases its “authority” on Maritime Law. UCC has yet to respond to the group's allegations.

    (tags: freemen crazy cork politics ireland hse gardai ucc law)

Links for 2013-11-20

Published November 20, 2013

Links for 2013-11-19

Published November 19, 2013

  • Software Detection of Currency

    Steven J. Murdoch presents some interesting results indicating that the EURion constellation may have been obsoleted:

    Recent printers, scanners and image manipulation software identify images of currency, will not process the image and display an error message linking to www.rulesforuse.org. The detection algorithm is not disclosed, however it is possible to test sample images as to whether they are identified as currency. This webpage shows an initial analysis of the algorithm's properties, based on results from the automated generation and testing of images. [...] Initially it was thought that the "Eurion constellation" was used to identify banknotes in the newly deployed software based system, since this has been confirmed to be the technique used by colour photocopiers, and was both necessary and sufficient to prevent an item being duplicated using the photocopier tested. However further investigation showed that the detection performed by software is different from the system used in colour photocopiers, and the Eurion constellation is neither necessary nor sufficent, and in fact it probably is not even a factor.

    (tags: eurion algorithms photoshop security currency money euro copying obscurity reversing)

  • Factual/drake

    a simple-to-use, extensible, text-based data workflow tool that organizes command execution around data and its dependencies. Data processing steps are defined along with their inputs and outputs and Drake automatically resolves their dependencies. [...] Drake is similar to GNU Make, but designed especially for data workflow management. It has HDFS [and S3] support, allows multiple inputs and outputs, and includes a host of features designed to help you bring sanity to your otherwise chaotic data processing workflows.
    Via Nelson. Looks interesting, although I'd like to see more features around retries, single-executor locking, parallelism, alerting/metrics, and unattended cron-like operation -- those are always the hard part when I wind up coding up a data pump.

    (tags: make data data-pump drake via:nelson pipelines workflow)

  • AK at re:Invent 2013: Getting Maximum Performance from Redshift

    good Redshift tips

    (tags: redshift aws amazon performance scaling s3 rdbms sql ops analytics)

  • Tintin And The Copyright Sharks - Falkvinge on Infopolicy

    A rather sordid tale of IP acquisition and exploitation, from the sounds of it

    (tags: tintin moulinsart belgium history herge ip copyright royalties rick-falkvinge)

Links for 2013-11-16

Published November 16, 2013

  • IPSO representative trivialising impact of the Loyaltybuild data breach

    A very worrying quote from Una Dillon of the Irish Payment Services Organisation in regard to the Loyaltybuild incident:

    “I wouldn’t be overly concerned if one of my cards was caught up in this,” Dillon says. “Even in the worst-case scenario – one in which my card was used fraudulently – my card provider will refund me everything that is taken”.
    This reflects a deep lack of understanding of (a) how identity fraud works, and (b) how card-fraud refunds in Ireland appear to work. (a): Direct misuse of credit card data is not always the result. Fraudsters may prefer to instead obtain separate credit through identity theft, ie. using other personal identifying data. (b): Visa debit cards have no credit limit -- your bank account can be cleared out in its entirety, and refunds can take a long time. For instance, http://www.askaboutmoney.com/showthread.php?t=174482 describes several cases, including one customer who waited 21 days for a refund. All in all it's trivialising a major risk for consumers. As I understand it, a separate statement from IPSO recommended that all customers of Loyaltybuild schemes need to monitor their bank accounts daily to keep an eye out for fraud, which is pretty absurd. Not impressive at all.

    (tags: loyaltybuild ipso money cards credit-cards visa debit-cards payment fraud identity-theft ireland)

  • Why GitHub is not your CV

    There is really astonishingly little value in looking at someone’s GitHub projects out of context. For a start, GitHub has no way of customising your profile page, and what is shown by default is the projects with the most stars, and the projects you’ve recently pushed to. That is, GitHub picks your most popular repos and puts those at the top. You have no say about what you consider important, or worthwhile, or interesting, or well-engineered, or valuable. You just get what other people think is useful. Aside from which, GitHub displays a lot of useless stats about how many followers you have, and some completely psychologically manipulative stats about how often you commit and how many days it is since you had a day off. So really, your GitHub profile displays two things: how ‘influential’ you are, and how easily you can be coerced into constantly working. It’s honestly about as relevant to a decent hiring decision as your Klout score.

    (tags: cv github open-source hiring career meritocracy work via:apyhr)

  • An Empirical Evaluation of TCP Performance in Online Games

    In this paper, we have analyzed the performance of TCP in of ShenZhou Online, a commercial, mid-sized MMORPG. Our study indicates that, though TCP is full-fledged and robust, simply transmitting game data over TCP could cause unexpected performance problems. This is due to the following distinctive characteristics of game traffic: 1) tiny packets, 2) low packet rate, 3) application-limited traffic generation, and 4) bi-directional traffic. We have shown that because TCP was originally designed for unidirectional and network-limited bulk data transfers, it cannot adapt well to MMORPG traffic. In particular, the window-based congestion control mechanism and the fast retransmit algorithm for loss recovery are ineffective. This suggests that the selective acknowledgement option should be enabled whenever TCP is used, as it significantly enhances the loss recovery process. Furthermore, TCP is overkill, as not every game packet needs to be transmitted reliably and processed in an orderly manner. We have also shown that the degraded network performance did impact users' willingness to continue a game. Finally, a number of design guidelines have been proposed by exploiting the unique characteristics of game traffic.
    via Nelson

    (tags: tcp games udp protocols networking internet mmos retransmit mmorpgs)

  • Column: The Loyaltybuild breach shows it’s time to take data protection seriously

    What is afoot here is a rerun of the Celtic Tiger era “light touch regulation” of financial services. Ireland has again made a Faustian pact whereby we lure employers here on the understanding that they will not subject to too-stringent a regulatory system. As the Loyaltybuild breach has shown, this is a bargain that will probably end badly. And as with the financial services boom, it is making the Germans nervous. Perhaps we will listen to them this time.

    (tags: fergal-crehan loyaltybuild celtic-tiger ireland dpa regulation data-protection privacy credit-cards)

  • mgodave/barge

    Looks very alpha, but one to watch.

    A JVM Implementation of the Raft Consensus Protocol

    (tags: via:sbtourist raft jvm java consensus distributed-computing)

Links for 2013-11-15

Published November 15, 2013

  • RocksDB

    ' A persistent key-value store for fast storage environments', ie. BerkeleyDB/LevelDB competitor, from Facebook.

    RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support IO-bound, in-memory and write-once workloads, and to be flexible to allow for innovation. We benchmarked LevelDB and found that it was unsuitable for our server workloads. Thebenchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine - where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal. By contrast, we've published the RocksDB benchmark results for server side workloads on Flash. We also measured the performance of LevelDB on these server-workload benchmarks and found that RocksDB solidly outperforms LevelDB for these IO bound workloads. We found that LevelDB's single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.
    Lots of good discussion at https://news.ycombinator.com/item?id=6736900 too.

    (tags: flash ssd rocksdb databases storage nosql facebook bdb disk key-value-stores lsm leveldb)

  • Amazon Route 53 Infima

    Colm McCarthaigh has open sourced Infima, 'a library for managing service-level fault isolation using Amazon Route 53'.

    Infima provides a Lattice container framework that allows you to categorize each endpoint along one or more fault-isolation dimensions such as availability-zone, software implementation, underlying datastore or any other common point of dependency endpoints may share. Infima also introduces a new ShuffleShard sharding type that can exponentially increase the endpoint-level isolation between customer/object access patterns or any other identifier you choose to shard on. Both Infima Lattices and ShuffleShards can also be automatically expressed in Route 53 DNS failover configurations using AnswerSet and RubberTree.

    (tags: infima colmmacc dns route-53 fault-tolerance failover multi-az sharding service-discovery)

Links for 2013-11-13

Published November 13, 2013

Links for 2013-11-12

Published November 12, 2013

  • Reactor hits GA

    'It can't just be Big Data, it has to be Fast Data: Reactor 1.0 goes GA':

    Reactor provides the necessary abstractions to build high-throughput, low-latency--what we now call "fast data"--applications that absolutely must work with thousands, tens of thousands, or even millions of concurrent requests per second. Modern JVM applications must be built on a solid foundation of asynchronous and reactive components that efficiently manage the execution of a very large number of tasks on a very small number of system threads. Reactor is specifically designed to help you build these kinds of applications without getting in your way or forcing you to work within an opinionated pattern.
    Featuring the LMAX Disruptor ringbuffer, the JavaChronicle fast persistent message-passing queue, Groovy closures, and Netty 4.0. This looks very handy indeed....

    (tags: disruptor reactive-programming reactor async libraries java jvm frameworks spring netty fast-data)

  • Backblaze Blog » How long do disk drives last?

    According to Backblaze's data, 80% of drives last 4 years, and the median lifespan is projected to be 6 years

    (tags: backblaze storage disk ops mtbf hardware failure lifespan)

  • Heirloom Chemistry Set by John Farrell Kuhns — Kickstarter

    This is a beauty. I wonder if they can ship to Ireland?

    To tell our story for this Kickstarter project, we really have to start in Christmas of 1959. Like many young scientists of the time, I received a Gilbert Chemistry set. This chemistry set provided me hours of great fun and learning as well as laying the foundation for my future as a research chemist. As I became an adult I wanted to share these types of experiences with my daughter, my nephews and nieces, and friends. But soon I became aware real chemistry sets were no longer available. Without real chemistry sets and opportunities for students to learn and explore, where would our future chemists come from? So .... I set out on a mission.

    (tags: chemistry science chemistry-sets education play kickstarter)

  • Philippe Flajolet’s contribution to streaming algorithms [preso]

    Nice deck covering HyperLogLog and its origins, plus a slide at the end covering the Flajolet/Wegman Adaptive Sampling algorithm ("how do you count the number of elements which appear only once in stream using constant size memory?")

    (tags: algorithms sketching hyperloglog flajolet wegman adaptive-sampling sampling presentations slides)

  • 3 Tacos or 4 Flautas Per Order Make a Healthy Diet in Greatest Scientific Study Ever

    "In reality, [tacos and flautas] aren't bad meals," the report argues. "The error that many of us Mexicans [Gustavo note: and gabachos] commit is including these types of dishes in our regular diet without an appropriate balance of them and falling into excessively eating them; accompanied by a lack of physical activity, it creates bad eating habits." The good docs go on to note that people can eat tacos and flautas without negatively affecting their health, but "the key resides in controlling the quantity and frequency of eating these types of meals." They also make the point that overall, tacos and flautas have less grease than doughnuts, french fries and even some health bars, although they didn't specify which brands in the latter. In a subsequent blog post, the scientists go on to describe flautas as an "energy food" due to their composition, and conclude by recommending that a healthy diet can include three tacos al pastor or four flautas per order, "controlling the frequency of intake." So have at it, boyos, but in moderation. And I can already hear the skeptics: What about tacos de chicharrones? Why not focus on carne asada? Did they take into consideration chiles de mordida? Did they factor in horchata? And whither the burrito variable?

    (tags: science tacos flautas mexican-food food eating yay)

Links for 2013-11-11

Published November 11, 2013

Links for 2013-11-08

Published November 8, 2013

  • Where your "full Irish" really comes from

    This is really disappointing; many meats labelled as "Irish" are anything but. The only trustworthy mark is the Bord Bia "Origin Ireland" stamp -- I'll be avoiding any products without this in future.

    Under European labelling law, country of origin is mandatory for beef, fish, olive oil, honey and fresh fruit and vegetables. Next month the EU will make it law to specify country of origin for the meat of pigs, chicken, sheep and goats, with a lead-in time of anywhere up to three years for food companies to comply. The pork rule, however, will only apply to fresh pork and not to processed meat, so consumers still won’t get a country-of-origin label on rashers, sausages or ham. In the meantime, the Bord Bia Origin-Ireland stamp is a guarantee that your Irish breakfast ingredients are indeed Irish.

    (tags: bord-bia labelling eu country-of-origin meat pork food quality)

  • Killing Freedom of Information in Ireland

    TheStory.ie will, in all likelihood, cease all FOI requests. And we will not seek funding from the public to support an immoral, cynical, unjustified and probably illegal FOI fee regime. We will not pay for information that the public already pays for. We will not support a system that perpetuates an outrageous infringement of citizen rights. The legislation was gutted in 2003 and it is being gutted again. More generally the number of requests from journalists from all news organisations in Ireland will fall as a result of these amendments, and the resulting efforts to shine a light on the administration of the State will certainly deteriorate. And secrecy will prevail.

    (tags: ireland politics foi information secrecy law)

Links for 2013-11-07

Published November 7, 2013

  • 10 Things You Should Know About AWS

    Some decent tips in here, mainly EC2-focussed

    (tags: amazon ec2 aws ops rds)

  • Tracing Brazil’s Guy Fawkes Masks

    really fascinating, from Ethan Zuckerman:

    The photo of workers making Guy Fawkes masks is something of a Rorschach test. If you’re primed to see the exploitative nature of global capitalism when you see people making a plastic mask, it’s there in the image. if you’re looking for the global spread of a protest movement, it’s there too, with a Brazilian factory making a local knock-off of a global icon to cash in on a national protest. Because the internet is a copying machine, it’s very bad at context. It’s easier to encounter the image of masks being manufactured devoid of accompanying details than it is to find the story behind the images. And given our tendency to ignore information in languages we don’t read, it’s easy to see how the masks come detached from their accompanying story. For me, the image is more powerful with context behind it. It’s possible to reflect on the irony of a Hollywood prop becoming an activist trope, the tensions between mass-production and anonymity and the individuality of one’s identity and grievance, the tensions between local and global, Warner Bros and Condal, intellectual property and piracy, all in the same image.

    (tags: anonymous globalization manufacturing piracy knock-offs brazil ethan-zuckerman global local hollywood capitalism)

  • ReCreate Ireland - Creativity through Reuse

    Great idea.

    For creative groups, we aim to offer easy access to a rich and varied selection of textures, colours and shapes. Members are also be able to participate in creativity workshops facilitated by fully trained professional artists either in-house or on your own premises. We intend to be the first choice of teachers, early childhood educators and arts animators in the community. For businesses, ReCreate reduces the costs of moving on end-of-line materials. We are a professional, credible and reliable partner organisation and our aim is to divert approximately 115 metric tonnes of clean materials from landfill annually. All collections are free of charge.

    (tags: recreate diy make-and-do recycling landfill art play scrap)

  • 3D-Print Your Own 20-Million-Year-Old Fossils

    When I get my hands on a 3-D printer, this will be high up my list of things to fabricate: a replica of a 20-million year old hominid skull.

    With over 40 digitized fossils in their collection, you can explore 3D renders of fossils representing prehistoric animals, human ancestors, and even ancient tools. Captured using Autodesk software, an SLR camera, and often the original specimen (rather than a cast replica), these renderings bring us closer than most will ever get to holding ancient artifacts. And if you've got an additive manufacturing device at your disposal, you can even download Sketchfab plans to generate your own.

    (tags: 3d-printing fossils africa history hominids replication fabrication sketchfab)

  • Makers & Brothers & Others

    'A Tiny Seasonal Department Store', featuring the amazing cakes of Wildflour Bakery among others, at 5 Dame Lane, D2.

    The tiny department store will be a wonderful seasonal gathering of Makers & Brothers favourite local and international brands. The Others in this project are a carefully considered bunch of partners from the worlds of flowers, food, fashion, beauty, homeware, gifts and more.  Makers & Brothers & Others, the tiny department store, promises to be a unique, exciting and engaging retail environment. A place to explore, a seasonal store alive with wonder and served by experts. Kindly hosted by the Fumbally Exchange.

    (tags: dublin shopping food cakes wildflour-bakery makers-and-brothers xmas)

Links for 2013-11-06

Published November 6, 2013

Links for 2013-11-05

Published November 5, 2013

Links for 2013-11-04

Published November 4, 2013

Links for 2013-11-03

Published November 3, 2013

  • There is NO spare capacity for Dublin's water supply

    The problem in a nutshell is that for an uncomfortable amount of the year the demand outstrips what the system can comfortably supply. In the graph below you’ll see the red line (demand for water) matches and regularly exceeds the blue line (what’s produced).

    (tags: drought water dublin mismanagement capacity dcc dublin-council graphs)

  • Old Fashioned 101

    Circa 1800, the Cocktail was a “hair of the dog” morning drink that tamed spirits with water, sugar and bitters (patent medicine). The late 19th Century expanded the use of the word “cocktail” to encompass just about any mixed drink. Since then, the Old Fashioned—literally, the old-fashioned way of making a cocktail—has been our contemporary expression of the original drink. During the 20th Century, various bad ideas encrusted the Old Fashioned. Here we will strip off those barnacles to expose the amazingly simple and sublime drink beneath.
    thanks to Ben for this one...

    (tags: recipe alcohol drinks cocktails old-fashioned bourbon bitters)

  • Metropolitan police detained David Miranda for promoting 'political' causes | World news | The Observer

    "We assess that Miranda is knowingly carrying material [...] the disclosure or threat of disclosure is designed to influence a government, and is made for the purpose of promoting a political or ideological cause. This therefore falls within the definition of terrorism."

    (tags: security david-miranda journalism censorship terrorism the-guardian)

  • A Brief Tour of FLP Impossibility

    One of the most important results in distributed systems theory was published in April 1985 by Fischer, Lynch and Patterson. Their short paper ‘Impossibility of Distributed Consensus with One Faulty Process’, which eventually won the Dijkstra award given to the most influential papers in distributed computing, definitively placed an upper bound on what it is possible to achieve with distributed processes in an asynchronous environment. This particular result, known as the ‘FLP result’, settled a dispute that had been ongoing in distributed systems for the previous five to ten years. The problem of consensus – that is, getting a distributed network of processors to agree on a common value – was known to be solvable in a synchronous setting, where processes could proceed in simultaneous steps. In particular, the synchronous solution was resilient to faults, where processors crash and take no further part in the computation. Informally, synchronous models allow failures to be detected by waiting one entire step length for a reply from a processor, and presuming that it has crashed if no reply is received. This kind of failure detection is impossible in an asynchronous setting, where there are no bounds on the amount of time a processor might take to complete its work and then respond with a message. Therefore it’s not possible to say whether a processor has crashed or is simply taking a long time to respond. The FLP result shows that in an asynchronous setting, where only one processor might crash, there is no distributed algorithm that solves the consensus problem.

    (tags: distributed-systems flp consensus-algorithms algorithms distcomp papers proofs)

  • Find a separating hyperplane with this One Weird Kernel Trick

    Terrible internet ad-spam recast as machine-learning spam

    '37-year-old patriot discovers "weird" trick to end slavery to the Bayesian monopoly. Discover the underground trick she used to slash her empirical risk by 75% in less than 30 days... before they shut her down. Click here to watch the shocking video! Get the Shocking Free Report!'

    (tags: funny via:hmason machine-learning spam wtf svms bayesian)

Links for 2013-11-01

Published November 1, 2013

  • It’s time for Silicon Valley to ask: Is it worth it?

    These companies and their technologies are built on data, and the data is us. If we are to have any faith in the Internet, we have to trust them to protect it. That’s a relationship dynamic that will become only more intertwined as the Internet finds its way into more aspects of our daily existences, from phones that talk to us to cars that drive themselves. The US’s surveillance programs threaten to destroy that trust permanently. America’s tech companies must stand up to this pervasive and corrosive surveillance system. They must ask that difficult question: “Is it worth it?”

    (tags: silicon-valley tech nsa gchq spying surveillance internet privacy data-protection)

  • Serf

    'a service discovery and orchestration tool that is decentralized, highly available, and fault tolerant. Serf runs on every major platform: Linux, Mac OS X, and Windows. It is extremely lightweight: it uses 5 to 10 MB of resident memory and primarily communicates using infrequent UDP messages [and an] efficient gossip protocol.'

    (tags: clustering service-discovery ops linux gossip broadcast clusters)

  • "Effective Computation of Biased Quantiles over Data Streams" [paper]

    Skew is prevalent in many data sources such as IP traffic streams.To continually summarize the distribution of such data, a high-biased set of quantiles (e.g., 50th, 90th and 99th percentiles) with finer error guarantees at higher ranks (e.g., errors of 5, 1 and 0.1 percent, respectively) is more useful than uniformly distributed quantiles (e.g., 25th, 50th and 75th percentiles) with uniform error guarantees. In this paper, we address the following two prob-lems. First, can we compute quantiles with finer error guarantees for the higher ranks of the data distribution effectively, using less space and computation time than computing all quantiles uniformly at the finest error? Second, if specific quantiles and their error bounds are requested a priori, can the necessary space usage and computation time be reduced? We answer both questions in the affirmative by formalizing them as the “high-biased” quantiles and the “targeted” quantiles problems, respectively, and presenting algorithms with provable guarantees, that perform significantly better than previously known solutions for these problems. We implemented our algorithms in the Gigascope data stream management system, and evaluated alternate approaches for maintaining the relevant summary structures.Our experimental results on real and synthetic IP data streams complement our theoretical analyses, and highlight the importance of lightweight, non-blocking implementations when maintaining summary structures over high-speed data streams.
    Implemented as a timer-histogram storage system in http://armon.github.io/statsite/ .

    (tags: statistics quantiles percentiles stream-processing skew papers histograms latency algorithms)

  • Statsite

    A C reimplementation of Etsy's statsd, with some interesting memory optimizations.

    Statsite is designed to be both highly performant, and very flexible. To achieve this, it implements the stats collection and aggregation in pure C, using libev to be extremely fast. This allows it to handle hundreds of connections, and millions of metrics. After each flush interval expires, statsite performs a fork/exec to start a new stream handler invoking a specified application. Statsite then streams the aggregated metrics over stdin to the application, which is free to handle the metrics as it sees fit. This allows statsite to aggregate metrics and then ship metrics to any number of sinks (Graphite, SQL databases, etc). There is an included Python script that ships metrics to graphite.

    (tags: statsd graphite statsite performance statistics service-metrics metrics ops)

  • 34 Irish pubs listed in Michelin good food guide

    if Linnane's and Cronin's are anything to go by, these will be worth a visit

    (tags: pubs ireland tourism food holidays michelin)

  • Fax vs Twilio

    A fax machine called my #twilio voice number, this is how @twilio transcribed it.... http://pic.twitter.com/RYh19Pg2pG
    This is amazing. Machine talking to machine, with hilarious results

    (tags: twilio transcription machine audio fax hey-hey-hey you-know-its-hey funny)

Links for 2013-10-31

Published October 31, 2013

  • Dark Mail Alliance

    Founded by Silent Circle and Lavabit. this is promising....

    To bring the world our unique end-to-end encrypted protocol and architecture that is the 'next-generation' of private and secure email. As founding partners of The Dark Mail Alliance, both Silent Circle and Lavabit will work to bring other members into the alliance, assist them in implementing the new protocol and jointly work to proliferate the worlds first end-to-end encrypted 'Email 3.0' throughout the world's email providers. Our goal is to open source the protocol and architecture and help others implement this new technology to address privacy concerns against surveillance and back door threats of any kind.

    (tags: privacy surveillance email smtp silent-circle lavabit dark-mail open-source standards crypto)

  • Ponies by Kij Johnson | Tor.com

    A rather dark short story about little girls, peer pressure, and childhood. no fun for this dad of 3 girls :( (via Tatu Saloranta)

    (tags: via:cowtowncoder writing fiction sf childhood peer-pressure tor ponies)

  • HdrHistogram by giltene

    A Histogram that supports recording and analyzing sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording, and provides control over value quantization behavior across the value range and the subsequent value resolution at any given level.

    (tags: hdr histogram data-structures coding gil-tene sampling measuring)

Links for 2013-10-30

Published October 30, 2013

  • Counterfactual Thinking, Rules, and The Knight Capital Accident

    John Allspaw with an interesting post on the Knight Capital disaster

    (tags: john-allspaw ops safety post-mortems engineering procedures)

  • Toyota's killer firmware: Bad design and its consequences

    This is exactly what you do NOT want to read about embedded systems controlling acceleration in your car:

    The Camry electronic throttle control system code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable). Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them. Barr also discovered inadequate and untracked peer code reviews and the absence of any bug-tracking system at Toyota.
    On top of this, there was no error-correcting RAM in use; stack-killing recursive code; a quoted 94% stack usage; risks of unintentional RTOS task shutdown; buffer overflows; unsafe casting; race conditions; unchecked error code return values; and a trivial watchdog timer check. Crappy, unsafe coding.

    (tags: firmware horror embedded-systems toyota camry safety acceleration misra-c coding code-verification spaghetti-code cyclomatic-complexity realtime rtos c code-reviews bug-tracking quality)

  • Forensic Topology

    The sounds were not, however, caused by ghosts but by a group of three or four men at least to some degree professionally trained, the FBI now believes, in tunneling: a close-knit and highly disciplined team, perhaps from the construction industry, perhaps even a disgruntled public works crew who decided to put their knowledge of the city’s underside to more lucrative work. After all, Rehder explained, their route into the bank was as much brute-force excavation as it was a retracing of the region’s buried waterways, accessing the neighborhood by way of the city’s complicated storm-sewer network, itself built along old creek beds that no longer appear on city maps. As LAPD lieutenant Doug Collisson, one of the men present on the day of the tunnel’s discovery, explained to the Los Angeles Times back in 1987, the crew behind the burglary “would have had to require some knowledge of soil composition and technical engineering. … The way the shaft itself was constructed, it was obviously well-researched and extremely sophisticated.” Rehder actually goes further, remarking that when Detective Dennis Pagenkopp “showed crime scene photos of the core bit holes” produced by the burglars’ boring upward into the vault “to guys who were in the concrete-coring business, they whistled with professional admiration.”

    (tags: cities crime architecture digging tunnels subterranean la lapd banks via:bldgblog sewers)

Links for 2013-10-29

Published October 29, 2013

Links for 2013-10-28

Published October 28, 2013

  • 14 Apple hacks from sugru

    I like the impromptu docking station hack

    (tags: apple sugru hacks hardware fixing repair diy)

  • Bruce Schneier On The Feudal Internet And How To Fight It

    This is very well-put.

    In its early days, there was a lot of talk about the "natural laws of the Internet" and how it would empower the masses, upend traditional power blocks, and spread freedom throughout the world. The international nature of the Internet made a mockery of national laws. Anonymity was easy. Censorship was impossible. Police were clueless about cybercrime. And bigger changes were inevitable. Digital cash would undermine national sovereignty. Citizen journalism would undermine the media, corporate PR, and political parties. Easy copying would destroy the traditional movie and music industries. Web marketing would allow even the smallest companies to compete against corporate giants. It really would be a new world order. Unfortunately, as we know, that's not how it worked out. Instead, we have seen the rise of the feudal Internet: Feudal security consolidates power in the hands of the few. These companies [like Google, Apple, Microsoft, Facebook etc.] act in their own self-interest. They use their relationship with us to increase their profits, sometimes at our expense. They act arbitrarily. They make mistakes. They're deliberately changing social norms. Medieval feudalism gave the lords vast powers over the landless peasants; we’re seeing the same thing on the Internet.

    (tags: bruce-schneier politics internet feudal-internet google apple microsoft facebook government)

  • Russia: Hidden chips 'launch malware attacks from irons'

    Cyber criminals are planting chips in electric irons and kettles to launch spam [jm: actually, malware] attacks, reports in Russia suggest. State-owned channel Rossiya 24 even showed footage of a technician opening up an iron included in a batch of Chinese imports to find a "spy chip" with what he called "a little microphone". Its correspondent said the hidden devices were mostly being used to spread viruses, by connecting to any computer within a 200m (656ft) radius which were using unprotected Wi-Fi networks. Other products found to have rogue components reportedly included mobile phones and car dashboard cameras.

    (tags: wifi viruses spam malware security russia china toasters kettles appliances)

  • Asteroid "mining" with Linux and FOSS

    Planetary Resources is a company with a sky-high (some might claim "pie in the sky") goal: to find and mine asteroids for useful minerals and other compounds. It is also a company that uses Linux and lots of free software. So two of the engineers from Planetary Resources, Ray Ramadorai and Marc Allen, gave a presentation at LinuxCon North America to describe how and why the company uses FOSS—along with a bit about what it is trying to do overall.

    (tags: lwn mining planets asteroids space linux foss open-source)

Links for 2013-10-26

Published October 26, 2013

  • Mac OS 10.9 – Infinity times your spam

    a pretty stupid Mail.app IMAP bug hoses Fastmail:

    Yes you read that right. It’s copying all the email from the Junk Folder back into the Junk Folder again!. This is legal IMAP, so our server proceeds to create a new copy of each message in the folder. It then expunges the old copies of the messages, but it’s happening so often that the current UID on that folder is up to over 3 million. It was just over 2 million a few days ago when I first emailed the user to alert them to the situation, so it’s grown by another million since. The only way I can think this escaped QA was that they used a server which (like gmail) automatically suppresses duplicates for all their testing, because this is a massively bad problem.

    (tags: osx bugs mail.app mail imap fastmail fail)

  • Google: Our Robot Cars Are Better Drivers Than Puny Humans | MIT Technology Review

    One of those analyses showed that when a human was behind the wheel, Google’s cars accelerated and braked significantly more sharply than they did when piloting themselves. Another showed that the cars’ software was much better at maintaining a safe distance from the vehicle ahead than the human drivers were. “We’re spending less time in near-collision states,” said Urmson. “Our car is driving more smoothly and more safely than our trained professional drivers.”

    (tags: google cars driving safety roads humans robots automation)

Links for 2013-10-25

Published October 25, 2013

Links for 2013-10-24

Published October 24, 2013

  • Roma, Racism And Tabloid Policing: Interview With Gary Younge : rabble

    [This case] shows the link between the popular and the state. This is tabloid journalism followed by tabloid policing. It’s also completely ignorant. I wrote my article on the Roma after covering the community for a week. I thought, “that’s interesting – there’s a range of phenotypes, ways of looking, that include Roma.” I mentioned two blonde kids by chance. I mentioned that Roma are more likely to speak the language of the country they’re in than Romani, more likely to have the religion of the country they’re in. But they have the basic aspect that is true for all identities – they know each other and other people know them. It’s not like I’m an expert on the Roma. I was covering them for a week and after the second day I knew Roma children had blonde hair and blue eyes. These people who took that kid away knew nothing. And on that basis they abducted a child.

    (tags: roma racism ireland gary-younge tabloid journalist children hse gardai)

  • Experian Sold Consumer Data to ID Theft Service

    This is what happens when you don't have strong controls on data protection/data privacy -- the US experience.

    While [posing as a US-based private investigator] may have gotten the [Vietnam-based gang operating the massive identity fraud site Superget.info] past Experian and/or CourtVentures’ screening process, according to Martin there were other signs that should have alerted Experian to potential fraud associated with the account. For example, Martin said the Secret Service told him that the alleged proprietor of Superget.info had paid Experian for his monthly data access charges using wire transfers sent from Singapore. “The issue in my mind was the fact that this went on for almost a year after Experian did their due diligence and purchased” Court Ventures, Martin said. “Why didn’t they question cash wires coming in every month? Experian portrays themselves as the data-breach experts, and they sell identity theft protection services. How this could go on without them detecting it I don’t know. Our agreement with them was that our information was to be used for fraud prevention and ID verification, and was only to be sold to licensed and credentialed U.S. businesses, not to someone overseas.”
    via Simon McGarr

    (tags: via:tupp_ed privacy security crime data-protection data-privacy experian data-breaches courtventures superget scams fraud identity identity-theft)

Links for 2013-10-23

Published October 23, 2013

Links for 2013-10-22

Published October 22, 2013

Links for 2013-10-21

Published October 21, 2013

Links for 2013-10-20

Published October 20, 2013

  • WISH: A Monumental 11-Acre Portrait in Belfast by Jorge Rodríguez-Gerada

    Must go up and visit this.

    Unveiled several days ago in Belfast, Northern Ireland as part of the Belfast Festival, WISH is the latest public art project by Cuban-American artist Jorge Rodriguez-Gerada. The image depicted is of an anonymous Belfast girl and is so large it can only be viewed from the highest points in Belfast or an airplane. Several years in the making, WISH was first plotted on a grid using state-of-the-art Topcon GPS technology and 30,000 manually placed wooden stakes in Belfast’s Titanic Quarter. The portrait was then “drawn” with aid of volunteers who helped place nearly 8 million pounds of natural materials including soil, sand, and rock over a period of four weeks.

    (tags: belfast ireland art portraits jorge-rodriguez-gerada land soil)

Links for 2013-10-19

Published October 19, 2013

Links for 2013-10-15

Published October 15, 2013

  • How to Read a Scientific Paper (About That Researcher With a Nematode in His Mouth) - Wired Science

    Let’s rewind to September 2012. It was about then- according to this recently published report (paywall) in The American Journal of Tropical Medicine – that an “otherwise healthy, 36-year-old man” felt a rough patch in his mouth, a scaly little area his right cheek. It didn’t hurt. But then it didn’t stay there either. He started testing for it with his tongue. It traveled. It moved to the back of his mouth, then forward, coiled backwards again. In the language of science: “These rough patches would appear and disappear on a daily basis, giving the patient the indirect sense that there was an organism moving within the oral cavity.”

    (tags: nematodes parasites biology medicine paper gross funny wired mouth)

  • "High Performance Browser Networking", by Ilya Grigorik, read online for free

    Wow, this looks excellent. A must-read for people working on systems with high-volume, low-latency phone-to-server communications -- and free!

    How prepared are you to build fast and efficient web applications? This eloquent book provides what every web developer should know about the network, from fundamental limitations that affect performance to major innovations for building even more powerful browser applications—including HTTP 2.0 and XHR improvements, Server-Sent Events (SSE), WebSocket, and WebRTC. Author Ilya Grigorik, a web performance engineer at Google, demonstrates performance optimization best practices for TCP, UDP, and TLS protocols, and explains unique wireless and mobile network optimization requirements. You’ll then dive into performance characteristics of technologies such as HTTP 2.0, client-side network scripting with XHR, real-time streaming with SSE and WebSocket, and P2P communication with WebRTC. Deliver optimal TCP, UDP, and TLS performance; Optimize network performance over 3G/4G mobile networks; Develop fast and energy-efficient mobile applications; Address bottlenecks in HTTP 1.x and other browser protocols; Plan for and deliver the best HTTP 2.0 performance; Enable efficient real-time streaming in the browser; Create efficient peer-to-peer videoconferencing and low-latency applications with real-time WebRTC transports
    Via Eoin Brazil.

    (tags: book browser networking performance phones mobile 3g 4g hsdpa http udp tls ssl latency webrtc websockets ebooks via:eoin-brazil google http2 sse xhr ilya-grigorik)

  • Even the NSA is finding it hard to cope with spam

    3 new Snowden leaks, covering acquisition of Yahoo address books, buddy lists, and email account activity, and how spammer activity required intervention to avoid losing useful data in the noise

    (tags: spam spammers nsa snowden leaks anti-spam yahoo im mail)

  • "What Should I Monitor?"

    slides (lots of slides) from Baron Schwartz' talk at Velocity in NYC.

    (tags: slides monitoring metrics ops devops baron-schwartz pdf capacity)

Links for 2013-10-14

Published October 14, 2013

  • The trouble with timestamps

    Timestamps, as implemented in Riak, Cassandra, et al, are fundamentally unsafe ordering constructs. In order to guarantee consistency you, the user, must ensure locally monotonic and, to some extent, globally monotonic clocks. This is a hard problem, and NTP does not solve it for you. When wall clocks are not properly coupled to the operations in the system, causal constraints can be violated. To ensure safety properties hold all the time, rather than probabilistically, you need logical clocks.

    (tags: clocks time distributed databases distcomp ntp via:fanf aphyr vector-clocks last-write-wins lww cassandra riak)

Links for 2013-10-13

Published October 13, 2013

  • Reverse Engineering a D-Link Backdoor

    Using the correct User-Agent: string, all auth is bypassed on several released models of D-Link and Planex routers. Horrific fail by D-Link

    (tags: d-link security backdoors authorization reversing planex networking routers)

  • The US fears back-door routes into the net because it's building them too | Technology | The Observer

    one of the most obvious inferences from the Snowden revelations published by the Guardian, New York Times and ProPublica recently is that the NSA has indeed been up to the business of inserting covert back doors in networking and other computing kit. The reports say that, in addition to undermining all of the mainstream cryptographic software used to protect online commerce, the NSA has been "collaborating with technology companies in the United States and abroad to build entry points into their products". These reports have, needless to say, been strenuously denied by the companies, such as Cisco, that make this networking kit. Perhaps the NSA omitted to tell DARPA what it was up to? In the meantime, I hear that some governments have decided that their embassies should no longer use electronic communications at all, and are returning to employing couriers who travel the world handcuffed to locked dispatch cases. We're back to the future, again.

    (tags: politics backdoors snowden snooping networking cisco nsa gchq)

  • Azerbaijan accidentally publishes the results of its election before the polls open

    The mistake came when an electoral commission accidentally published results showing a victory for Ilham Aliyev, the country’s long-standing President, a day before voting. Meydan TV, an online channel critical of the government, released a screenshot from a mobile app for the Azerbaijan Central Election Commission which showed that Mr Aliyev had received 72.76 per cent of the vote compared with 7.4 per cent for the opposition candidate, Jamil Hasanli. The screenshot also indicates that the app displayed information about how many people voted at various times during the day. Polls opened at 8am.

    (tags: azerbaijan corruption fix elections voting voter-fraud)

  • DNS registrars that complied with "shakedown" anti-piracy requests may now be in violation of ICANN Transfers Policy

    According to EasyDNS:

    Any registrar that has taken one of these sites offline that now impedes the registrants of those domains from simply getting their domain names out of there and back online somewhere else will then be subject to the TDRP – Transfer Dispute Resolution Policy and if they lose (which they will) they will be subject to TDRP fees assesed by the registry operator, and to quote the TDRP itself "Transfer dispute resolution fees can be substantial". This is why it is never a good idea to just react to pressure in the face of obnoxious bluster – in the very act of trying to diffuse any perceived culpability you end up opening yourself to real liability.

    (tags: tdrp easydns dns registrars domains piracy law due-process)

  • Schneier on Security: Air Gaps

    interesting discussion in the comments. "Patricia"'s process is particularly hair-raisingly complex, involving 3 separate machines and a multitude of VMs

    (tags: air-gaps security networking bruce-schneier via:adulau)

Links for 2013-10-11

Published October 11, 2013

Links for 2013-10-10

Published October 10, 2013

Links for 2013-10-09

Published October 9, 2013

  • pt-summary

    from the Percona toolkit. 'Conveniently summarizes the status and configuration of a server. It is not a tuning tool or diagnosis tool. It produces a report that is easy to diff and can be pasted into emails without losing the formatting. This tool works well on many types of Unix systems.' --- summarises OOM history, top, netstat connection table, interface stats, network config, RAID, LVM, disks, inodes, disk scheduling, mounts, memory, processors, and CPU.

    (tags: percona tools cli unix ops linux diagnosis raid netstat oom)

  • How much can an extra hour's sleep change you?

    What they discovered is that when the volunteers cut back from seven-and-a-half to six-and-a-half hours' sleep a night, genes that are associated with processes like inflammation, immune response and response to stress became more active. The team also saw increases in the activity of genes associated with diabetes and risk of cancer. The reverse happened when the volunteers added an hour of sleep.

    (tags: sleep health rest cancer bbc science)

  • Kovet

    some great phone cases from an Irish company, with nifty art by Irish illustrators and artists including Fatti Burke and Chris Judge

    (tags: chris-judge fatti-burke illustrators art ireland iphone cases)

Links for 2013-10-08

Published October 8, 2013

  • What drives JVM full GC duration

    Interesting empirical results using JDK 7u21:

    Full GC duration depends on the number of objects allocated and the locality of their references. It does not depend that much on actual heap size.
    Reference locality has a surprisingly high effect.

    (tags: java jvm data gc tuning performance cms g1)

  • Rhizome | Occupy.here: A tiny, self-contained darknet

    Occupy.here began two years ago as an experiment for the encampment at Zuccotti Park. It was a wifi router hacked to run OpenWrt Linux (an operating system mostly used for computer networking) and a small "captive portal" website. When users joined the wifi network and attempted to load any URL, they were redirected to http://occupy.here. The web software offered up a simple BBS-style message board providing its users with a space to share messages and files.
    Nifty project from Dan Phiffer.

    (tags: occupy.here openwrt hacking wifi network community)

  • Whatever Happened to "Due Process" ?

    Mark Jeftovic is on fire after receiving yet another "take down this domain or else" mail from the City of London police:

    We have an obligation to our customers and we are bound by our Registrar Accreditation Agreements not to make arbitrary changes to our customers settings without a valid FOA (Form of Authorization). To supersede that we need a legal basis. To get a legal basis something has to happen in court. [...] What gets me about all of this is that the largest, most egregious perpetrators of online criminal activity right now are our own governments, spying on their own citizens, illegally wiretapping our own private communications and nobody cares, nobody will answer for it, it's just an out-of-scope conversation that is expected to blend into the overall background malaise of our ever increasing serfdom. If I can't make various governments and law enforcement agencies get warrants or court orders before they crack my private communications then I can at least require a court order before I takedown my own customer.

    (tags: city-of-london police takedowns politics mark-jeftovic easydns registrars dns via:tjmcintyre)

  • Intellectual Ventures' Evil Knows No Bounds: Buys Patent AmEx Donated For Public Good... And Starts Suing

    The problem with software patents, part XVII.

    So you have a situation where even when the original patent holder donated the patent for "the public good," sooner or later, an obnoxious patent troll like IV comes along and turns it into a weapon. Again: AmEx patented those little numbers on your credit card, and then for the good of the industry and consumer protection donated the patent to a non-profit, who promised not to enforce the patent against banks... and then proceeded to sell the patent to Intellectual Ventures who is now suing banks over it.

    (tags: intellectual-ventures scams patents swpats shakedown banking cvv american-express banks amex cmaf)

Links for 2013-10-07

Published October 7, 2013

Links for 2013-10-04

Published October 4, 2013

Links for 2013-10-03

Published October 3, 2013

Links for 2013-10-02

Published October 2, 2013

Links for 2013-10-01

Published October 1, 2013

  • How an Engineer Earned 1.25 Million Air Miles By Buying Pudding

    An amazing hack. 'Air Miles are awesome, they can be used to score free flights, hotel stays and if you’re really lucky, the scorn and hatred of everyone you come in contact with who has to pay full price when they travel. The king of all virtually free travelers is one David Phillips, a civil engineer who teaches at the University of California, Davis. David came to the attention of the wider media when he managed to convert about 12,150 cups of Healthy Choice chocolate pudding [costing $3000] into over a million Air Miles. Ever since, David and his entire family have been travelling the world for next to nothing.' (via al3xandru)

    (tags: via:al3xandru hacks cool pudding small-print air-miles free)

  • Down the Rabbit Hole

    An adventure that takes you through several popular Java language features and shows how they compile to bytecode and eventually JIT to assembly code.

    (tags: charles-nutter java jvm compilation reversing talks slides)

Links for 2013-09-30

Published September 30, 2013

  • Model checking for highly concurrent code

    Applied formal methods in order to test distributed systems -- specifically GlusterFS:

    I'll use an example from my own recent experience. I'm developing a new kind of replication for GlusterFS. To make sure the protocol behaves correctly even across multiple failures, I developed a Murphi model for it. [...] I added a third failure [to the simulated model]. I didn't expect a three-node system to continue working if more than one of those were concurrent (the model allows the failures to be any mix of sequential and concurrent), but I expected it to fail cleanly without reaching an invalid state. Surprise! It managed to produce a case where a reader can observe values that go back in time. This might not make much sense without knowing the protocol involved, but it might give some idea of the crazy conditions a model checker will find that you couldn't possibly have considered. [...] So now I have a bug to fix, and that's a good thing. Clearly, it involves a very specific set of ill-timed reads, writes, and failures. Could I have found it by inspection or ad-hoc analysis? Hell, no. Could I have found it by testing on live systems? Maybe, eventually, but it probably would have taken months for this particular combination to occur on its own. Forcing it to occur would require a lot of extra code, plus an exerciser that would amount to a model checker running 100x slower across machines than Murphi does. With enough real deployments over enough time it would have happened, but the only feasible way to prevent that was with model checking. These are exactly the kinds of bugs that are hardest to fix in the field, and that make users distrust distributed systems, so those of us who build such systems should use every tool at our disposal to avoid them.

    (tags: model-checking formal-methods modelling murphi distcomp distributed-systems glusterfs testing protocols)

  • Is Trypophobia a Real Phobia? | Popular Science

    ie. "fear of small, clustered holes". Sounds like it's not so much a "phobia" as some kind of innate, visceral disgust response; I get it. 'As for who actually made the word up, that distinction probably belongs to a blogger in Ireland named Louise, Andrews says. According to an archived Geocities page, Louise settled on "trypophobia" (Greek for "boring holes" + "fear") after corresponding with a representative at the Oxford English Dictionary. Louise, Andrews and trypophobia Facebook group members have petitioned the dictionary to include the word. The term will need to be used for years and have multiple petitions and scholarly references before the dictionary accepts it, Andrews says. I, for one, would prefer to forget about it forever.'

    (tags: disgusting revulsion fear phobias trypophobia holes ugh innate)

  • Common phobia you have never heard of: Fear of holes may stem from evolutionary survival response

    "We think that everyone has trypophobic tendencies even though they may not be aware of it," said Dr Cole. "We found that people who don't have the phobia still rate trypophobic images as less comfortable to look at than other images. It backs up the theory that we are set-up to be fearful of things which hurt us in our evolutionary past. We have an innate predisposition to be wary of things that can harm us."

    (tags: trypophobia holes fear aversion disgust ugh evolution innate)

Links for 2013-09-26

Published September 26, 2013

Links for 2013-09-25

Published September 25, 2013

Links for 2013-09-24

Published September 24, 2013

Links for 2013-09-23

Published September 23, 2013

Links for 2013-09-22

Published September 22, 2013

  • RSA warns developers not to use RSA products

    In case you're missing the story here, Dual_EC_DRBG (which I wrote about yesterday) is the random number generator voted most likely to be backdoored by the NSA. The story here is that -- despite many valid concerns about this generator -- RSA went ahead and made it the default generator used for all cryptography in its flagship cryptography library. The implications for RSA and RSA-based products are staggering. In a modestly bad but by no means worst case, the NSA may be able to intercept SSL/TLS connections made by products implemented with BSafe.

    (tags: bsafe rsa crypto backdoors nsa security dual_ec_drbg rngs randomness)

  • A Case Against Cucumber

    This is exactly my problem with Cucumber and similar BDD test frameworks.

    When I write a Cucumber feature, I have to write the Gherkin that describes the acceptance criteria, and the Ruby code that implements the step definitions. Since the code to implement the step definitions is just normal RSpec (or whichever testing library you use), if someone else is writing the Gherkin, the amount of setup to create a working test should be about the same. So you’re only breaking even! However, I don’t believe that it would really be breaking even. Cucumber adds another layer of indirection on top of your tests. When I’m trying to see why a specific scenario is failing, first I need to find the step that is failing. Since these steps are defined with regular expressions, I have to grep for the step definition.

    (tags: ruby testing bdd cucumber rspec coding)

  • Gamasutra - Opinion: The tragedy of Grand Theft Auto V

    This is watching your sharp, witty father start telling old fart jokes as his mind slows down. And as much as the internet is habituated to defending GTA as "satire," what is it satirizing, if everything is either sad or awful? Where is the "satire" when the awful parts no longer seem edgy or provocative, just attempts at catch-all "offense" that aren't honed enough to even connect? Here's a series that has been creating real, meaningful friction with conventional entertainment for as long as I can remember, and rather than push the envelope by creating new kinds of monsters, it's reciting the same old gangland fantasies, like a college boy who can't stop staring at the Godfather II poster on his wall, talking about how he's gonna be a big Hollywood director in between bong rips. You call the trading index BAWSAQ? Oh, bro, you're so funny, you're gonna be huge.

    (tags: gamasutra games gaming gta gta-v via:skamille)

  • CCC | Chaos Computer Club breaks Apple TouchID

    "We hope that this finally puts to rest the illusions people have about fingerprint biometrics. It is plain stupid to use something that you can´t change and that you leave everywhere every day as a security token", said Frank Rieger, spokesperson of the CCC. "The public should no longer be fooled by the biometrics industry with false security claims. Biometrics is fundamentally a technology designed for oppression and control, not for securing everyday device access." iPhone users should avoid protecting sensitive data with their precious biometric fingerprint not only because it can be easily faked, as demonstrated by the CCC team. Also, you can easily be forced to unlock your phone against your will when being arrested. Forcing you to give up your (hopefully long) passcode is much harder under most jurisdictions than just casually swiping your phone over your handcuffed hands.

Links for 2013-09-18

Published September 18, 2013

Links for 2013-09-17

Published September 17, 2013

  • To solve hard problems, you need to use bricolage

    In a talk about a neat software component he designed, Bruce Haddon observed that there is no way that the final structure and algorithmic behavior of this component could have been predicted, designed, or otherwise anticipated. Haddon observed that computer science serves as a source of core ideas: it provides the data structures and algorithms that are the building blocks. Meanwhile, he views software engineering as a useful set of methods to help design reliable software without losing your mind. Yet he points out that neither captures the whole experience. That’s because much of the work is what Haddon calls hacking, but what others would call bricolage. Simply put, there is much trial and error: we put ideas to together and see where it goes.
    This is a great post, and I agree (broadly). IMO, most software engineering requires little CS, but there are occasional moments where a single significant aspect of a project requires a particular algorithm, and would be kludgy, hacky, or over-complex to solve without it.

    (tags: bricolage hacking cs computer-science work algorithms)

  • Getting Real About Distributed System Reliability

    I have come around to the view that the real core difficulty of [distributed] systems is operations, not architecture or design. Both are important but good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations. This is quite different from the view of unbreakable, self-healing, self-operating systems that I see being pitched by the more enthusiastic NoSQL hypesters. Worse yet, you can’t easily buy good operations in the same way you can buy good software—you might be able to hire good people (if you can find them) but this is more than just people; it is practices, monitoring systems, configuration management, etc.

    (tags: reliability nosql distributed-systems jay-kreps ops)

  • Don't use Hadoop - your data isn't that big

    see also HN comments: https://news.ycombinator.com/item?id=6398650 , particularly davidmr's great one:

    I suppose all of this is to say that the amount of required parallelization of a problem isn't necessarily related to the size of the problem set as is mentioned most in the article, but also the inherent CPU and IO characteristics of the problem. Some small problems are great for large-scale map-reduce clusters, some huge problems are horrible for even bigger-scale map-reduce clusters (think fluid dynamics or something that requires each subdivision of the problem space to communicate with its neighbors). I've had a quote printed on my door for years: Supercomputers are an expensive tool for turning CPU-bound problems into IO-bound problems.
    I love that quote!

    (tags: hadoop big-data scaling map-reduce)

  • Gilt Tech

    Gilt ran a stress-test of Riak to replace Voldemort (I think) in a shadow stack, with good results:

    Riak’s strong performance suggests that, should we pursue implementation, it will withstand our unique traffic needs and prove reliable. As for the Gilt-Basho team’s strong performance: It was amazing that we were able to accomplish so much in just a week’s time! Thanks again to Seth and Steve for making this possible.

    (tags: riak testing shadow-stack voldemort storage gilt)

  • THE LONG DARK, a first-person post-disaster survival sim by Hinterland — Kickstarter

    wow this looks great.

    The Long Dark is a thoughtful, first-person survival simulation that emphasizes quiet exploration in a stark, yet hauntingly beautiful, post-disaster setting. The breathtakingly picturesque Pacific Northwest frames the backdrop for the drama of The Long Dark.

    (tags: games survival via:fp eclaire the-long-dark kickstarter)

  • The Rational Choices of Crack Addicts - NYTimes.com

    “The key factor is the environment, whether you’re talking about humans or rats,” Dr. Hart said. “The rats that keep pressing the lever for cocaine are the ones who are stressed out because they’ve been raised in solitary conditions and have no other options. But when you enrich their environment, and give them access to sweets and let them play with other rats, they stop pressing the lever.”

    (tags: crack drugs policy science addiction society)

Links for 2013-09-16

Published September 16, 2013

Links for 2013-09-15

Published September 15, 2013

  • Being poor changes your thinking about everything

    Very interesting research into poverty and scarcity, in the Washington Post:

    The scarcity trap captures this notion we see again and again in many domains. When people have very little, they undertake behaviors that maintain or reinforce their future disadvantage. If you have very little, you often behave in such a way so that you'll have little in the future. In economics, people talk about the poverty trap. We're generalizing that, saying this happens a lot, and we've experienced it.

    (tags: poor poverty society economics scarcity washington-post)

  • Good SSL for your website is absurdly difficult in practice

    Yet again, security software fails on packaging and UI. via Tony Finch

    (tags: security ssl tls packaging via:fanf)

  • Former NSA and CIA director says terrorists love using Gmail

    At one point, Hayden expressed a distaste for online anonymity, saying "The problem I have with the Internet is that it's anonymous." But he noted, there is a struggle over that issue even inside government. The issue came to a head during the Arab Spring movement when the State Department was funding technology [presumably Tor?] to protect the anonymity of activists so governments could not track down or repress their voices. "We have a very difficult time with this," Hayden said. He then asked, "is our vision of the World Wide Web the global digital commons -- at this point you should see butterflies flying here and soft background meadow-like music -- or a global free fire zone?" Given that Hayden also compared the Internet to the wild west and Somalia, Hayden clearly leans toward the "global free fire zone" vision of the Internet.
    well, that's a good analogy for where we're going -- a global free-fire zone.

    (tags: gmail cia nsa surveillance michael-hayden security snooping law tor arab-spring)

Links for 2013-09-14

Published September 14, 2013

  • Google swaps out MySQL, moves to MariaDB

    When we asked Sallner to quantify the scale of the migration he said, "They're moving it all. Everything they have. All of the MySQL servers are moving to MariaDB, as far as I understand." By moving to MariaDB, Google can free itself of any dependence on technology dictated by Oracle – a company whose motivations are unclear, and whose track record for working with the wider technology community is dicey, to say the least. Oracle has controlled MySQL since its acquisition of Sun in 2010, and the key InnoDB storage engine since it got ahold of Innobase in 2005. [...] We asked Cole why Google would shift from MySQL to MariaDB, and what the key technical differences between the systems were. "From my perspective, they're more or less equivalent other than if you look at specific features and how they implement them," Cole said, speaking in a personal capacity and not on behalf of Google. "Ideologically there are lots of differences."
    So -- AWS, when will RDS offer MariaDB as an option?

    (tags: google mysql mariadb sql open-source licensing databases storage innodb oracle)

  • FBI Admits It Controlled Tor Servers Behind Mass Malware Attack

    The code’s behavior, and the command-and-control server’s Virginia placement, is also consistent with what’s known about the FBI’s “computer and internet protocol address verifier,” or CIPAV, the law enforcement spyware first reported by WIRED in 2007. Court documents and FBI files released under the FOIA have described the CIPAV as software the FBI can deliver through a browser exploit to gather information from the target’s machine and send it to an FBI server in Virginia. The FBI has been using the CIPAV since 2002 against hackers, online sexual predators, extortionists, and others, primarily to identify suspects who are disguising their location using proxy servers or anonymity services, like Tor. Prior to the Freedom Hosting attack, the code had been used sparingly, which kept it from leaking out and being analyzed.

    (tags: cipav fbi tor malware spyware security wired)

  • Creating Flight Recordings

    lots more detail on the new "Java Mission Control" feature in Hotspot 7u40 JVMs, and how to use it to start and stop profiling in a live, production JVM from a separate "jcmd" command-line client. If the overhead is small, this could be really neat -- turn on profiling for 1 minute every hour on a single instance, and collect realtime production profile data on an automated basis for post-facto analysis if required

    (tags: instrumentation logging profiling java jvm ops)

Links for 2013-09-12

Published September 12, 2013

  • Necessary and Proportionate -- In Which Civil Society is Caught Between a Cop and a Spy

    Modern telecommunications technology implied the development of modern telecommunications surveillance, because it moved the scope of action from the physical world (where intelligence, generally seen as part of the military mission, had acted) to the virtual world—including the scope of those actions that could threaten state power. While the public line may have been, as US Secretary of State Henry Stimson said in 1929, “gentlemen do not open each other’s mail”, you can bet that they always did keep a keen eye on the comings and goings of each other’s shipping traffic. The real reason that surveillance in the context of state intelligence was limited until recently was because it was too expensive, and it was too expensive for everyone. The Westphalian compromise demands equality of agency as tied to territory. As soon as one side gains a significant advantage, the structure of sovereignty itself is threatened at a conceptual level?—?hence Oppenheimer as the death of any hope of international rule of law. Once surveillance became cheap enough, all states were (and will increasingly be) forced to attempt it at scale, as a reaction to this pernicious efficiency. The US may be ahead of the game now, but Moore’s law and productization will work their magic here.

    (tags: government telecoms snooping gchq nsa surveillance law politics intelligence spying internet)

Links for 2013-09-11

Published September 11, 2013

  • Observability at Twitter

    Bit of detail into Twitter's TSD metric store.

    There are separate online clusters for different data sets: application and operating system metrics, performance critical write-time aggregates, long term archives, and temporal indexes. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired. Aggregation is generally performed at write-time to avoid extra storage operations for metrics that are expected to be immediately consumed. Indexing occurs along several dimensions–service, source, and metric names–to give users some flexibility in finding relevant data.

    (tags: twitter monitoring metrics service-metrics tsd time-series storage architecture cassandra)

  • NSA: Possibly breaking US laws, but still bound by laws of computational complexity

    I didn’t clearly explain that there’s an enormous continuum between, on the one hand, a full break of RSA or Diffie-Hellman (which still seems extremely unlikely to me), and on the other, “pure side-channel attacks” involving no new cryptanalytic ideas.  Along that continuum, there are many plausible places where the NSA might be.  For example, imagine that they had a combination of side-channel attacks, novel algorithmic advances, and sheer computing power that enabled them to factor, let’s say, ten 2048-bit RSA keys every year.  In such a case, it would still make perfect sense that they’d want to insert backdoors into software, sneak vulnerabilities into the standards, and do whatever else it took to minimize their need to resort to such expensive attacks.  But the possibility of number-theoretic advances well beyond what the open world knows certainly wouldn’t be ruled out.  Also, as Schneier has emphasized, the fact that NSA has been aggressively pushing elliptic-curve cryptography in recent years invites the obvious speculation that they know something about ECC that the rest of us don’t.

    (tags: ecc rsa crypto security nsa gchq snooping sniffing diffie-hellman pki key-length)

  • Low Overhead Method Profiling with Java Mission Control now enabled in the most recent HotSpot JVM release

    Built into the HotSpot JVM [in JDK version 7u40] is something called the Java Flight Recorder. It records a lot of information about/from the JVM runtime, and can be thought of as similar to the Data Flight Recorders you find in modern airplanes. You normally use the Flight Recorder to find out what was happening in your JVM when something went wrong, but it is also a pretty awesome tool for production time profiling. Since Mission Control (using the default templates) normally don’t cause more than a per cent overhead, you can use it on your production server.
    I'm intrigued by the idea of always-on profiling in production. This could be cool.

    (tags: performance java measurement profiling jvm jdk hotspot mission-control instrumentation telemetry metrics)

Links for 2013-09-09

Published September 9, 2013

  • How the NSA Spies on Smartphones

    One of the US agents' tools is the use of backup files established by smartphones. According to one NSA document, these files contain the kind of information that is of particular interest to analysts, such as lists of contacts, call logs and drafts of text messages. To sort out such data, the analysts don't even require access to the iPhone itself, the document indicates. The department merely needs to infiltrate the target's computer, with which the smartphone is synchronized, in advance. Under the heading "iPhone capability," the NSA specialists list the kinds of data they can analyze in these cases. The document notes that there are small NSA programs, known as "scripts," that can perform surveillance on 38 different features of the iPhone 3 and 4 operating systems. They include the mapping feature, voicemail and photos, as well as the Google Earth, Facebook and Yahoo Messenger applications.
    and, of course, the alternative means of backup is iCloud.... wonder how secure those backups are.

    (tags: nsa surveillance gchq iphone smartphones backups icloud security)

  • Behind the Screens at Loggly

    Boost ASIO at the front end (!), Kafka 0.8, Storm, and ElasticSearch

    (tags: boost scalability loggly logging ingestion cep stream-processing kafka storm architecture elasticsearch)

  • Schneier on Security: Excess Automobile Deaths as a Result of 9/11

    The inconvenience of extra passenger screening and added costs at airports after 9/11 cause many short-haul passengers to drive to their destination instead, and, since airline travel is far safer than car travel, this has led to an increase of 500 U.S. traffic fatalities per year. Using DHS-mandated value of statistical life at $6.5 million, this equates to a loss of $3.2 billion per year, or $32 billion over the period 2002 to 2011 (Blalock et al. 2007).

    (tags: risk security death 9-11 politics screening dhs air-travel driving road-safety)

Links for 2013-09-08

Published September 8, 2013

  • Perhaps I'm out of step and Britons just don't think privacy is important | Henry Porter | Comment is free | The Observer

    The debate has been stifled in Britain more successfully than anywhere else in the free world and, astonishingly, this has been with the compliance of a media and public that regard their attachment to liberty to be a matter of genetic inheritance. So maybe it is best for me to accept that the BBC, together with most of the newspapers, has moved with society, leaving me behind with a few old privacy-loving codgers, wondering about the cause of this shift in attitudes. Is it simply the fear of terror and paedophiles? Are we so overwhelmed by the power of the surveillance agencies that we feel we can't do anything? Or is it that we have forgotten how precious and rare truly free societies are in history?

    (tags: privacy uk politics snooping spies gchq society nsa henry-porter)

  • Big data is watching you

    Some great street art from Brighton, via Darach Ennis

    (tags: via:darachennis street-art graffiti big-data snooping spies gchq nsa art)

  • Blocking The Pirate Bay appears to have 'no lasting net impact' on illegal downloading

    In the fight against the unauthorised sharing of copyright protected material, aka piracy, Dutch Internet Service Providers have been summoned by courts to block their subscribers’ access to The Pirate Bay (TPB) and related sites. This paper studies the effectiveness of this approach towards online copyright enforcement, using both a consumer survey and a newly developed non-infringing technology for BitTorrent monitoring. While a small group of respondents download less from illegal sources or claim to have stopped, and a small but significant effect is found on the distribution of Dutch peers, no lasting net impact is found on the percentage of the Dutch population downloading from illegal sources.

    (tags: fail blocking holland pirate-bay tpb papers via:tjmcintyre internet isps)

  • How Advanced Is the NSA's Cryptanalysis — And Can We Resist It?

    Bruce Schneier's suggestions:

    Assuming the hypothetical NSA breakthroughs don’t totally break public-cryptography — and that’s a very reasonable assumption — it’s pretty easy to stay a few steps ahead of the NSA by using ever-longer keys. We’re already trying to phase out 1024-bit RSA keys in favor of 2048-bit keys. Perhaps we need to jump even further ahead and consider 3072-bit keys. And maybe we should be even more paranoid about elliptic curves and use key lengths above 500 bits. One last blue-sky possibility: a quantum computer. Quantum computers are still toys in the academic world, but have the theoretical ability to quickly break common public-key algorithms — regardless of key length — and to effectively halve the key length of any symmetric algorithm. I think it extraordinarily unlikely that the NSA has built a quantum computer capable of performing the magnitude of calculation necessary to do this, but it’s possible. The defense is easy, if annoying: stick with symmetric cryptography based on shared secrets, and use 256-bit keys.

    (tags: bruce-schneier cryptography wired nsa surveillance snooping gchq cryptanalysis crypto future key-lengths)

  • DevOps Eye for the Coding Guy: Metrics

    a pretty good description of the process of adding service metrics to a Django webapp using graphite and statsd. Bookmarking mainly for the great real-time graphing hack at the end...

    (tags: statsd django monitoring metrics python graphite)

  • Probabalistic Scraping of Plain Text Tables

    a nifty hack.

    Recently I have been banging my head trying to import a ton of OCR acquired data expressed in tabular form. I think I have come up with a neat approach using probabilistic reasoning combined with mixed integer programming. The method is pretty robust to all sorts of real world issues. In particular, the method leverages topological understanding of tables, encodes it declaratively into a mixed integer/linear program, and integrates weak probabilistic signals to classify the whole table in one go (at sub second speeds). This method can be used for any kind of classification where you have strong logical constraints but noisy data.
    (via proggit)

    (tags: scraping tables ocr probabilistic linear-programming optimization machine-learning via:proggit)

  • vimeo/timeserieswidget

    'Plugin to make highly interactive graphite graph objects ((i.e. graphs where you can interactively toggle on/off individual series, inspect datapoints, zoom in realtime, etc) Supports Flot (canvas), Rickshaw (svg) and standard graphite png images (in case you're nostalgic and don't like interactivity).'

    (tags: graphs graphing graphite dataviz flot rickshaw svg canvas javascript)

Links for 2013-09-05

Published September 5, 2013

Links for 2013-09-04

Published September 4, 2013

  • How To Buffer Full YouTube Videos Before Playing

    summary - turn off DASH (Dynamic adaptive streaming) using a userscript.

    (tags: chrome youtube google video dash mpeg streaming)

  • Voldemort on Solid State Drives [paper]

    'This paper and talk was given by the LinkedIn Voldemort Team at the Workshop on Big Data Benchmarking (WBDB May 2012).'

    With SSD, we find that garbage collection will become a very significant bottleneck, especially for systems which have little control over the storage layer and rely on Java memory management. Big heapsizes make the cost of garbage collection expensive, especially the single threaded CMS Initial mark. We believe that data systems must revisit their caching strategies with SSDs. In this regard, SSD has provided an efficient solution for handling fragmentation and moving towards predictable multitenancy.

    (tags: voldemort storage ssd disk linkedin big-data jvm tuning ops gc)

Links for 2013-09-03

Published September 3, 2013

  • Streaming MapReduce with Summingbird

    Before Summingbird at Twitter, users that wanted to write production streaming aggregations would typically write their logic using a Hadoop DSL like Pig or Scalding. These tools offered nice distributed system abstractions: Pig resembled familiar SQL, while Scalding, like Summingbird, mimics the Scala collections API. By running these jobs on some regular schedule (typically hourly or daily), users could build time series dashboards with very reliable error bounds at the unfortunate cost of high latency. While using Hadoop for these types of loads is effective, Twitter is about real-time and we needed a general system to deliver data in seconds, not hours. Twitter’s release of Storm made it easy to process data with very low latencies by sacrificing Hadoop’s fault tolerant guarantees. However, we soon realized that running a fully real-time system on Storm was quite difficult for two main reasons: Recomputation over months of historical logs must be coordinated with Hadoop or streamed through Storm with a custom log loading mechanism; Storm is focused on message passing and random-write databases are harder to maintain. The types of aggregations one can perform in Storm are very similar to what’s possible in Hadoop, but the system issues are very different. Summingbird began as an investigation into a hybrid system that could run a streaming aggregation in both Hadoop and Storm, as well as merge automatically without special consideration of the job author. The hybrid model allows most data to be processed by Hadoop and served out of a read-only store. Only data that Hadoop hasn’t yet been able to process (data that falls within the latency window) would be served out of a datastore populated in real-time by Storm. But the error of the real-time layer is bounded, as Hadoop will eventually get around to processing the same data and will smooth out any error introduced. This hybrid model is appealing because you get well understood, transactional behavior from Hadoop, and up to the second additions from Storm. Despite the appeal, the hybrid approach has the following practical problems: Two sets of aggregation logic have to be kept in sync in two different systems; Keys and values must be serialized consistently between each system and the client. The client is responsible for reading from both datastores, performing a final aggregation and serving the combined results Summingbird was developed to provide a general solution to these problems.
    Very interesting stuff. I'm particularly interested in the design constraints they've chosen to impose to achieve this -- data formats which require associative merging in particular.

    (tags: mapreduce streaming big-data twitter storm summingbird scala pig hadoop aggregation merging)

  • Thoughts on Granby Park, the recent pop-up park off Parnell St

    We mentioned above that pop-up spaces have become popular across Europe because they allow developers and city councils to harness urban creativity in order to drive up real estate prices without ceding control of a given site. Those who produce the space through hard work, collaboration and passion move on, making way for property development and speculation. The international research in this area is very clear on this point and it has been documented in places from Lower-East Side Manhattan to Berlin’s Kreuzberg. Most perversely, increased property prices make it even more difficult for creativity to flourish in a given area and end up driving out long-term working class communities, migrants and young people. But what can we do? If every attempt we make to make our city a better place simply ends up being captured in the calculations of real estate players, surely the situation is hopeless? Is it better, then, to do nothing? We don’t think it is better to do nothing and, like Upstart, we still believe we can find a way together through experimentation and collaboration. However, this means questioning, reflecting on and publicly discussing the relationship between our efforts to make a city more after our hearts desire and the process of gentrification. As noted above, this is especially the case with pop-up spaces given their temporary nature. It is really necessary that we think about how to make sure our activities don’t contribute to gentrification in the long term, but instead benefit the city as a whole. We certainly don’t have the solutions, but if we sweep these awkward questions under the carpet we risk contributing to the very forces we want to challenge and alienating those who will perceive us as the ‘front-line’ of gentrification.

    (tags: gentrification pop-up parks dublin ireland cities upstart spaces urban-planning)

  • [#CASSANDRA-5582] Replace CustomHsHaServer with better optimized solution based on LMAX Disruptor

    Disruptor: decimating P99s since 2011

    (tags: disruptor cassandra java p99 latency speed performance concurrency via:kellabyte)

  • Time is a Dimension

    I love these.

    Photographic prints are great because they don’t need power to be displayed. They are more or less permanent. Videos are great because they record a sequence of time which shows reality almost like how we experience. Is it possible to combine the two? And not via long exposure photography where often details are lost from motion. So I played around with the tools of digital photography and post processing to give you this series: Time is a dimension. This series of images are mostly landscapes, seascapes and cityscapes, and they are a single composite made from sequences that span 2-4 hours, mostly of sunrises and sunsets. The basic structure of a landscape is present in every piece. But each panel or concentric layer shows a different slice of time, which is related to the adjacent panel/layer. The transition from daytime to night is gradual and noticeable in every piece, but would not be something you expect to see in a still image.

    (tags: photography beautiful photos art time dimensions prints via:matthaughey)

Links for 2013-09-02

Published September 2, 2013