Skip to content

Category: Uncategorized

Links for 2016-04-12

Published April 12, 2016

  • AWSume

    'AWS Assume Made Awesome' -- 'Here are Trek10, we work with many clients, and thus work with multiple AWS accounts on a regular (daily) basis. We needed a way to make managing all our different accounts easier. We create a standard Trek10 administrator role in our clients’ accounts that we can assume. For security we require that the role assumer have multifactor authentication enabled.'

    (tags: mfa aws awsume credentials accounts ops)

  • Gil Tene on benchmarking

    'I would strongly encourage you to avoid repeating the mistakes of testing methodologies that focus entirely on max achievable throughput and then report some (usually bogus) latency stats at those max throughout modes. The techempower numbers are a classic example of this in play, and while they do provide some basis for comparing a small aspect of behavior (what I call the "how fast can this thing drive off a cliff" comparison, or "pedal to the metal" testing), those results are not very useful for comparing load carrying capacities for anything that actually needs to maintain some form of responsiveness SLA or latency spectrum requirements.' Some excellent advice here on how to measure and represent stack performance. Also: 'DON'T use or report standard deviation for latency. Ever. Except if you mean it as a joke.'

    (tags: performance benchmarking testing speed gil-tene latency measurement hdrhistogram load-testing load)

  • Data Protection Mishap Leaves 55M Philippine Voters at Risk

    Every registered voter in the Philippines is now susceptible to fraud and other risks after a massive data breach leaked the entire database of the Philippines’ Commission on Elections (COMELEC). While initial reports have downplayed the impact of the leak, our investigations showed a huge number of sensitive personally identifiable information (PII)–including passport information and fingerprint data–were included in the data dump. [....] Based on our investigation, the data dumps include 1.3 million records of overseas Filipino voters, which included passport numbers and expiry dates. What is alarming is that this crucial data is just in plain text and accessible to everyone. Interestingly, we also found a whopping 15.8 million record of fingerprints and a list of people running for office since the 2010 elections. In addition, among the data leaked were files on all candidates running on the election with the filename VOTESOBTAINED. Based on the filename, it reflects the number of votes obtained by the candidate. Currently, all VOTESOBTAINED file are set to have NULL as figure.

    (tags: fingerprints biometrics philippines authentication data-dumps security hacks comelec e-voting pii passports voting)

  • The dark side of Guardian comments | Technology | The Guardian

    Excellent data on abusive commenters

    (tags: comments data guardian journalism abuse twitter racism sexism)

  • Emmet Connolly on Twitter: "Stick around after the end credits for a preview of Batman vs Pintman https://t.co/xgWWcvfpc9"

    The saddest superhero ever

    (tags: funny pintman dublin ireland alcoholics twitter history pubs)

Links for 2016-04-11

Published April 11, 2016

Links for 2016-04-10

Published April 10, 2016

Links for 2016-04-08

Published April 8, 2016

  • GCHQ intervenes to prevent catastrophically insecure UK smart meter plan - The Inquirer

    GCHQ barged in after spooks cast their eyes over the plans and realised that power companies were proposing to use a single decryption key for communications from the 53 million smart meters that will eventually be installed in the UK.
    holy crap.

    (tags: gchq security smart-meters power uk electricity gas infrastructure)

  • AWS Billing And Cost Control [slides]

    (tags: aws ec2 costs money hosting ops)

  • Irish drone register allowed access to personal details of 2,000 members

    The breach, which allowed registered users to view names, addresses, email addresses and phone numbers of other people registered on the site, was brought to the attention of the authority on Sunday night. In a statement to TheJournal.ie, the IAA revealed it was aware of four users who downloaded the file.

    (tags: fail drones ireland iaa security)

  • Running Docker on AWS from the ground up

    Advantages/disavantages section right at the bottom is good.

    ECS, believe it or not, is one of the simplest Schedulers out there. Most of the other alternatives I’ve tried offer all sorts of fancy bells & whistles, but they are either significantly more complicated to understand (lots of new concepts), take too much effort to set up (lots of new technologies to install and run), are too magical (and therefore impossible to debug), or some combination of all three. That said, ECS also leaves a lot to be desired.

    (tags: aws docker ecs ec2 schedulers)

  • Hungary proposes anti-crypto law

    up to 2 years imprisonment for use of apps for encrypted communication

    (tags: crypto hungary laws internet crackdown encryption)

  • good example of Application-Level Keepalive beating SO_KEEPALIVE

    we have now about 100 salt-minions which are installed in remote areas with 3G and satellite connections. We loose connectivity with all of those minions in about 1-2 days after installation, with test.ping reporting "minion did not return". The state was each time that the minions saw an ESTABLISHED TCP connection, while on the salt-master there were no connection listed at all. (Yes that is correct). Tighter keepalive settings were tried with no result. (OS is linux) Each time, restarting the salt-minion fixes the problem immediately. Obviously the connections are transparently proxied someplace, (who knows what happens with those SAT networks) so the whole tcp-keepalive mechanism of 0mq fails.
    Also notes in the thread that the default TCP timeout for Azure Load Balancer is 4 minutes: https://azure.microsoft.com/en-us/blog/new-configurable-idle-timeout-for-azure-load-balancer/ . The default Linux TCP keepalive doesn't send until 2 hours after last connection use, and it's a system-wide sysctl (/proc/sys/net/ipv4/tcp_keepalive_time). Further, http://networkengineering.stackexchange.com/questions/7207/why-bgp-implements-its-own-keepalive-instead-of-using-tcp-keepalive notes "some firewalls filter TCP keepalives".

    (tags: tcp keep-alive keepalive protocol timeouts zeromq salt firewalls nat)

Links for 2016-04-07

Published April 7, 2016

  • “Racist algorithms” and learned helplessness

    Whenever I’ve had to talk about bias in algorithms, I’ve tried be  careful to emphasize that it’s not that we shouldn’t use algorithms in search, recommendation and decision making. It’s that we often just don’t know how they’re making their decisions to present answers, make recommendations or arrive at conclusions, and it’s this lack of transparency that’s worrisome. Remember, algorithms aren’t just code. What’s also worrisome is the amplifier effect. Even if “all an algorithm is doing” is reflecting and transmitting biases inherent in society, it’s also amplifying and perpetuating them on a much larger scale than your friendly neighborhood racist. And that’s the bigger issue. [...] even if the algorithm isn’t creating bias, it’s creating a feedback loop that has powerful perception effects.

    (tags: feedback bias racism algorithms software systems society)

  • The revenge of the listening sockets

    More adventures in debugging the Linux kernel:

    You can't have a very large number of bound TCP sockets and we learned that the hard way. We learned a bit about the Linux networking stack: the fact that LHTABLE is fixed size and is hashed by destination port only. Once again we showed a couple of powerful of System Tap scripts.

    (tags: ops linux networking tcp network lhtable kernel)

  • s3git

    git for Cloud Storage. Create distributed, decentralized and versioned repositories that scale infinitely to 100s of millions of files and PBs of storage. Huge repos can be cloned on your local SSD for making changes, committing and pushing back. Oh yeah, and it dedupes too due to BLAKE2 Tree hashing. http://s3git.org

    (tags: git ops storage cloud s3 disk aws version-control blake2)

  • BLAKE2: simpler, smaller, fast as MD5

    'We present the cryptographic hash function BLAKE2, an improved version of the SHA-3 finalist BLAKE optimized for speed in software. Target applications include cloud storage, intrusion detection, or version control systems. BLAKE2 comes in two main flavors: BLAKE2b is optimized for 64-bit platforms, and BLAKE2s for smaller architectures. On 64-bit platforms, BLAKE2 is often faster than MD5, yet provides security similar to that of SHA-3. We specify parallel versions BLAKE2bp and BLAKE2sp that are up to 4 and 8 times faster, by taking advantage of SIMD and/or multiple cores. BLAKE2 has more benefits than just speed: BLAKE2 uses up to 32% less RAM than BLAKE, and comes with a comprehensive tree-hashing mode as well as an efficient MAC mode.'

    (tags: crypto hash blake2 hashing blake algorithms sha1 sha3 simd performance mac)

Links for 2016-04-06

Published April 6, 2016

  • When It Comes to Age Bias, Tech Companies Don’t Even Bother to Lie

    HubSpot’s CEO and co-founder, Brian Halligan, explained to the New York Times that this age imbalance was not something he wanted to remedy, but in fact something he had actively cultivated. HubSpot was “trying to build a culture specifically to attract and retain Gen Y’ers,” because, “in the tech world, gray hair and experience are really overrated,” Halligan said.  I gasped when I read that. Could anyone really believe this? Even if you did believe this, what CEO would be foolish enough to say it out loud? It was akin to claiming that you prefer to hire Christians, or heterosexuals, or white people. I assumed an uproar would follow. As it turned out, nobody at HubSpot saw this as a problem. Halligan didn’t apologize for his comments or try to walk them back. The lesson I learned is that when it comes to race and gender bias, the people running Silicon Valley at least pay lip service to wanting to do better — but with age discrimination they don’t even bother to lie. 

    (tags: hiring startups tech ageism age hubspot gen-y discrimination)

  • Gaeltacht development company defends sale of State seaweed company to Canadian multinational

    FFS. Fine Gael government sells off more of our national assets for cheap:

    Mr John O’Sullivan, chief executive of Bioatlantis Ltd in Co Kerry called on the Oireachtas environment committee to investigate the sale, or ask the Oireachtas public accounts committee to do so. Mr O’Sullivan said that his company had made a bid of €5.7 million for Arramara, comprising €1.5 million initially and €4.2 million in the post-investment phase, and had been given just 12 days to prepare the bid. He understood that two foreign companies – the Canadian Acadian Seaplants and French company Setalg – had been given over a year to prepare their bids. He said that Acadian’s bid was €1.8 million, and the French bid was €2 million, for initial purchase, and that the rating was “changed” when the final bids were in. No details had been released and the lack of transparency was “frightening” in relation to the final sale, he said.

    (tags: seaweed acadian setalg arramara bioatlantis government ireland selloff gaeltacht unag)

  • Not 'Going Dark': 15 Out Of 15 Most Recent EU Terrorists Were Known To The Authorities In Multiple Ways | Techdirt

    Comprehensive surveillance appears as seemingly inexpensive because it is a solution that scales thanks to technology: troubleshooting at the press of a button. Directly linked with the aim of saving more and more, just as with the State in general. But classic investigative work, which is proven to work, is expensive and labor intensive. This leads to a failure by the authorities because of a faith in technology that is driven by economics.

    (tags: tech surveillance techdirt terrorism brussels crypto going-dark)

  • So you're thinking of coming to Dublin...

    A really excellent list of stuff to do/see/eat/drink in Ireland, from Colin @ 3FE. top notch recommendations! (also, god I need to get out more)

    (tags: dublin travel food drink ireland tourism 3fe)

  • Nest Reminds Customers That Ownership Isn't What It Used to Be

    EFF weigh in on the internet of shit:

    Customers likely didn't expect that, 18 months after the last Revolv Hubs were sold, instead of getting more upgrades, the device would be intentionally, permanently, and completely disabled. .... Nest Labs and Google are both subsidiaries of Alphabet, Inc., and bricking the Hub sets a terrible precedent for a company with ambitions to sell self-driving cars, medical devices, and other high-end gadgets that may be essential to a person’s livelihood or physical safety.

    (tags: nest legal tech google alphabet internetofshit iot law)

  • Primary Online Database: POD now (mostly) not compulsory (for now)

    Ever since the introduction of the Primary Online Database of schoolchildren by the Department of Education, the Department and its Minister have been eager to point out that any parent who refused to allow a child’s data to be transferred would see that child’s education defunded. Well, for all children other than this week’s crop of new Junior Infants, that threat has now collapsed. This is despite the Minister and her department having claimed that the drastic threat of defunding was because it simply wasn’t possible to give grants without a child’s full data being transferred. [...] Oddly, as the prospect of defunding the education of 30% of the nation’s children in the run up to an election loomed large, the Department discovered it could, after all, pay for a child’s education without all its POD data.

    (tags: pod law ireland data-protection privacy children school)

  • Wired on the new O'Reilly SRE book

    "Site Reliability Engineering: How Google Runs Production Systems", by Chris Jones, Betsy Beyer, Niall Richard Murphy, Jennifer Petoff. Go Niall!

    (tags: google sre niall-murphy ops devops oreilly books toread reviews)

Links for 2016-04-05

Published April 5, 2016

  • Google's Nest killing off old devices

    Google is making customers' existing devices useless, less than 2 years after the devices were available for sale, with only 2 months warning. This is one of the reasons I won't spend money on the Internet Of Things shitshow. '"Which hardware will Google choose to intentionally brick next?" asks Arlo Gilbert. "If they stop supporting Android will they decide that the day after warranty expires that your phone will go dark? Is your Nexus device safe? What about your Nest fire alarm? What about your Dropcam? What about your Chromecast device?"'

    (tags: iot fail google alphabet nest revolv home shutdown)

  • 'Devastating' bug pops secure doors at airports, hospitals

    "A command injection vulnerability exists in this function due to a lack of any sanitisation on the user-supplied input that is fed to the system() call," Lawshae says.
    :facepalm:

    (tags: security iot funny fail linux unix backticks system udp hid vertx edge)

  • Counting with domain specific databases — The Smyte Blog — Medium

    whoa, pretty heavily engineered scalable counting system with Kafka, RocksDB and Kubernetes

    (tags: kafka rocksdb kubernetes counting databases storage ops)

  • Is anyone concerned about the future of Nest?

    wow, looks like Nest is fucked:

    As a Nest engineer, I won't say any numbers that aren't public, but this company is already on deathwatch. Once that happens, most people will quickly have shiny paperweights because it's a constant firefight keeping these systems up. We have $340M in revenue, not profit, against a ~$500M budget. No new products since the purchase, and sales/growth numbers are dire. Our budget deal expires soon, and all the good engineers on my teams have discreetly indicated they are going to flee once their golden handcuffs unlock (many have already left despite sacrificing a lot of money to do so). Tony and his goons demand crazy timelines so much that "crunch time" has basically lost meaning. Just when your labor bears fruit, they swoop in, 180 the specs you just delivered on, then have the gall to call your team "incompetent" for not reading their mind and delivering on these brand-new specs. I waste most of my time in pointless meetings, or defending my teams so they don't flip their desks and walk out. People fall asleep in corners and cry in the bathrooms, health and marriages are suffering. Already the churn is insane, close to half the company if not more. Skilled engineers can tell the environment is toxic, so we're filling vacancies with mostly sub-par talent.

    (tags: nest google business dotcoms churn iot)

Links for 2016-04-04

Published April 4, 2016

  • JitPack

    Publish JVM and Android libraries direct from github -- it'll build and package a lib on the fly, caching them via CDN

    (tags: build github java maven gradle dependencies packaging libraries)

  • Illicit trade in prescription drugs a growing problem for Dublin’s north inner city

    ughh. The latest scourge is Zopiclone, "zimmos", which are being dealt openly due to a bureaucratic loophole in enforcement.

    (tags: zopiclone zimmos drugs dublin northside drug-abuse)

  • A programming language for E. coli

    Mind = blown.

    MIT biological engineers have created a programming language that allows them to rapidly design complex, DNA-encoded circuits that give new functions to living cells. Using this language, anyone can write a program for the function they want, such as detecting and responding to certain environmental conditions. They can then generate a DNA sequence that will achieve it. "It is literally a programming language for bacteria," says Christopher Voigt, an MIT professor of biological engineering. "You use a text-based language, just like you're programming a computer. Then you take that text and you compile it and it turns it into a DNA sequence that you put into the cell, and the circuit runs inside the cell."

    (tags: dna mit e-coli bacteria verilog programming coding biohacking science)

  • GitHub now supports "squash on merge"

    yay. On the other hand -- http://www.thecaucus.net/#/content/caucus/tech_blog/516 is a good explanation of why not to adopt it. Pity GitHub haven't made it a per-review option...

    (tags: github code-reviews squashing merges git coding)

  • bcc

    Dynamic tracing tools for Linux, a la dtrace, ktrace, etc. Built using BPF, using kernel features in the 4.x kernel series, requiring at least version 4.1 of the kernel

    (tags: linux tracing bpf dynamic ops)

  • US government commits to publish publicly financed software under Free Software licenses

    Wow, this is significant:

    At the end of last week, the White House published a draft for a Source Code Policy. The policy requires every public agency to publish their custom-build software as Free Software for other public agencies as well as the general public to use, study, share and improve the software. At the Free Software Foundation Europe (FSFE) we believe that the European Union, and European member states should implement similar policies. Therefore we are interested in your feedback to the US draft.

    (tags: government open-source coding licenses fsf free-software source-code us-politics usa)

  • Elias gamma coding

    'used most commonly when coding integers whose upper-bound cannot be determined beforehand.'

    (tags: data-structures algorithms elias-gamma-coding encoding coding numbers integers)

  • A Decade Of Container Control At Google

    The big thing that can be gleaned from the latest paper out of Google on its container controllers is that the shift from bare metal to containers is a profound one – something that may not be obvious to everyone seeking containers as a better way – and we think cheaper way – of doing server virtualization and driving up server utilization higher. Everything becomes application-centric rather than machine-centric, which is the nirvana that IT shops have been searching for. The workload schedulers, cluster managers, and container controllers work together to get the right capacity to the application when it needs it, whether it is a latency-sensitive job or a batch job that has some slack in it, and all that the site recovery engineers and developers care about is how the application is performing and they can easily see that because all of the APIs and metrics coming out of them collect data at the application level, not on a per-machine basis. To do this means adopting containers, period. There is no bare metal at Google, and let that be a lesson to HPC shops or other hyperscalers or cloud builders that think they need to run in bare metal mode.

    (tags: google containers kubernetes borg bare-metal ops)

  • How wet is a cycling commute in Ireland?

    It turns out that you’ll get wet 3 times more often if you’re a Galway cyclist when compared to a Dubliner. Dublin is Ireland’s driest cycling city.
    Some good data and visualization on this extremely important issue

    (tags: rain rainfall-radar ireland climate weather dublin galway cycling)

Links for 2016-03-31

Published March 31, 2016

Links for 2016-03-30

Published March 30, 2016

Links for 2016-03-29

Published March 29, 2016

  • Mass surveillance silences minority opinions, according to study - The Washington Post

    This is excellent research, spot on.

    Elizabeth Stoycheff, lead researcher of the study and assistant professor at Wayne State University, is disturbed by her findings. “So many people I've talked with say they don't care about online surveillance because they don't break any laws and don't have anything to hide. And I find these rationales deeply troubling,” she said. She said that participants who shared the “nothing to hide” belief, those who tended to support mass surveillance as necessary for national security, were the most likely to silence their minority opinions. “The fact that the 'nothing to hide' individuals experience a significant chilling effect speaks to how online privacy is much bigger than the mere lawfulness of one's actions. It's about a fundamental human right to have control over one's self-presentation and image, in private, and now, in search histories and metadata,” she said.

    (tags: culture privacy psychology surveillance mass-surveillance via:snowden nothing-to-hide spiral-of-silence fear)

  • Qualys SSL Server Test

    pretty sure I had this bookmarked previously, but this is the current URL -- SSL/TLS quality report

    (tags: ssl tls security tests ops tools testing)

  • Observability at Twitter: technical overview, part II

    Interesting to me mainly for this tidbit which makes my own prejudices:

    “Pull” vs “push” in metrics collection: At the time of our previous blog post, all our metrics were collected by “pulling” from our collection agents. We discovered two main issues: * There is no easy way to differentiate service failures from collection agent failures. Service response time out and missed collection request are both manifested as empty time series. * There is a lack of service quality insulation in our collection pipeline. It is very difficult to set an optimal collection time out for various services. A long collection time from one single service can cause a delay for other services that share the same collection agent. In light of these issues, we switched our collection model from “pull” to “push” and increased our service isolation. Our collection agent on each host only collects metrics from services running on that specific host. Additionally, each collection agent sends separate collection status tracking metrics in addition to the metrics emitted by the services. We have seen a significant improvement in collection reliability with these changes. However, as we moved to self service push model, it becomes harder to project the request growth. In order to solve this problem, we plan to implement service quota to address unpredictable/unbounded growth.

    (tags: pull push metrics tcp stacks monitoring agents twitter fault-tolerance)

  • These unlucky people have names that break computers

    Pat McKenzie's name is too long to fit in Japanese database schemas; Janice Keihanaikukauakahihulihe'ekahaunaele's name was too long for US schemas; and Jennifer Null suffers from the obvious problem

    (tags: databases design programming names coding japan schemas)

  • How we implemented the video player in Mail.Ru Cloud

    We’ve recently added video streaming service to Mail.Ru Cloud. Development started with contemplating the new feature as an all-purpose “Swiss Army knife” that would both play files of any format and work on any device with the Cloud available. Video content uploaded to the Cloud mostly falls into one of the two categories: “movies/series” and “users’ videos”. The latter are the videos that users shoot with their phones and cameras, and these videos are most versatile in terms of formats and codecs. For many reasons, it is often a problem to watch these videos on other end-user devices without prior normalization: a required codec is missing, or the file size is too big to download, or whatever.
    Mainly around using HLS (HTTP Live Streaming).

    (tags: hls http streaming video audio mail.ru players codecs)

  • A shot that rang round the world

    The international impact of the Easter Rising has rarely been acknowledged. This rebellion did not only rattle British rule in Ireland — it inspired radical movements in Britain itself and across the globe, and it shook colonial rulers and states worldwide.

    (tags: history easter-rising 1916 ireland revolution colonialism)

  • Hashed Wheel Timer

    nice java impl of this efficient data structure, broken out from Project Reactor

    (tags: scalability java timers hashed-wheel-timers algorithms data-structures)

Links for 2016-03-27

Published March 27, 2016

  • Jenkins 2.0

    built-in support for CI/CD deployment pipelines, driven from a checked-in DSL file. great stuff, very glad to see them going this direction. (via Eric)

    (tags: via:eric jenkins ci cd deployment pipelines testing automation build)

  • Hey Microsoft, the Internet Made My Bot Racist, Too

    All machine learning algorithms strive to exaggerate and perpetuate the past. That is, after all, what they are learning from. The fundamental assumption of every machine learning algorithm is that the past is correct, and anything coming in the future will be, and should be, like the past. This is a fine assumption to make when you are Netflix trying to predict what movie you’ll like, but is immoral when applied to many other situations. For bots like mine and Microsoft’s, built for entertainment purposes, it can lead to embarrassment. But AI has started to be used in much more meaningful ways: predictive policing in Chicago, for example, has already led to widespread accusations of racial profiling. This isn’t a little problem. This is a huge problem, and it demands a lot more attention then it’s getting now, particularly in the community of scientists and engineers who design and apply these algorithms. It’s one thing to get cursed out by an AI, but wholly another when one puts you in jail, denies you a mortgage, or decides to audit you.

    (tags: machine-learning ml algorithms future society microsoft)

Links for 2016-03-25

Published March 25, 2016

  • Tahoe LAFS accidentally lose Bitcoin wallet with loads of donations in it, get it back

    But ECDSA private keys don't trigger the same protective instincts that we'd apply to, say, a bar of gold. One sequence of 256 random bits looks just as worthless as any other. And the cold hard unforgeability of these keys means we can't rely upon other humans to get our money back when we lose them. Plus, we have no experience at all with things that grow in value by four orders of magnitude, without any attention, in just three years. So we have a cryptocurrency-tool UX task in front of us: to avoid mistakes like the one we made, we must to either move these digital assets into solid-feeling physical containers, or retrain our perceptions to attach value to the key strings themselves.

    (tags: backups cryptography bitcoin cryptocurrency ecdsa private-keys ux money)

  • Visual Representation of SQL Joins

    useful bookmark to have (via Nelson)

    (tags: sql joins mysql reference database)

  • Interesting Lottery Terminal Hack - Schneier on Security

    Neat manual timing attack.

    An investigator for the Connecticut Lottery determined that terminal operators could slow down their lottery machines by requesting a number of database reports or by entering several requests for lottery game tickets. While those reports were being processed, the operator could enter sales for 5 Card Cash tickets. Before the tickets would print, however, the operator could see on a screen if the tickets were instant winners. If tickets were not winners, the operator could cancel the sale before the tickets printed.

    (tags: attacks security lottery connecticut kiosks)

Links for 2016-03-24

Published March 24, 2016

Links for 2016-03-23

Published March 23, 2016

Links for 2016-03-20

Published March 20, 2016

  • Modern Irish genome closely matches pre-Celt DNA, not Celtic

    Radiocarbon dating shows that the bones discovered at McCuaig's go back to about 2000 B.C. That makes them hundreds of years older than the oldest artifacts generally considered to be Celtic — relics unearthed from Celt homelands of continental Europe, most notably around Switzerland, Austria and Germany. For a group of scholars who in recent years have alleged that the Celts, beginning from the middle of Europe, may never have reached Ireland, the arrival of the DNA evidence provides the biological certitude that the science has sometimes brought to criminal trials. “With the genetic evidence, the old model [of Celtic colonisation of Ireland] is completely shot,” John Koch, a linguist at the Center for Advanced Welsh and Celtic Studies at the University of Wales.

    (tags: celts ireland history dna genetics genome carbon-dating bronze-age europe colonisation)

Links for 2016-03-18

Published March 18, 2016

  • GCM XMPP delivery receipt not always received - Google Groups

    Good to know:

    'GCM delivery receipts don't have an SLA at this time. Having your connection open longer will increase the odds that delivery receipts will arrive. 10 seconds seems a bit short. I'm glad it works. I would recommend longer like 10 min or an hour. The real design of this system is for persistent connections, hence connections that setup and tear down frequently will have difficulty receiving delivery receipts.'

    (tags: gcm xmpp receipts messaging push-notifications google)

Links for 2016-03-16

Published March 16, 2016

  • The disturbingly simple way dozens of celebrities had their nude photos stolen

    Basic phishing: 'Collins hacked over 100 people by sending emails that looked like they came from Apple and Google, such as “e-mail.protection318@icloud.com,” “noreply_helpdesk0118@outlook.com,” and “secure.helpdesk0019@gmail.com.” According to the government, Collins asked for his victims’ iCloud or Gmail usernames and passwords and “because of the victims’ belief that the email had come from their [Internet Service Providers], numerous victims responded by giving [them].”'

    (tags: security phishing nudes fappening celebs gmail icloud apple)

  • RFC 7754 - Technical Considerations for Internet Service Blocking and Filtering

    The Internet is structured to be an open communications medium. This openness is one of the key underpinnings of Internet innovation, but it can also allow communications that may be viewed as undesirable by certain parties. Thus, as the Internet has grown, so have mechanisms to limit the extent and impact of abusive or objectionable communications. Recently, there has been an increasing emphasis on "blocking" and "filtering", the active prevention of such communications. This document examines several technical approaches to Internet blocking and filtering in terms of their alignment with the overall Internet architecture. When it is possible to do so, the approach to blocking and filtering that is most coherent with the Internet architecture is to inform endpoints about potentially undesirable services, so that the communicants can avoid engaging in abusive or objectionable communications. We observe that certain filtering and blocking approaches can cause unintended consequences to third parties, and we discuss the limits of efficacy of various approaches.
    (via Tony Finch)

    (tags: via:fanf blocking censorship filtering internet rfcs rfc isps)

  • The Three Go Landmines

    'There are three easy to make mistakes in go. I present them here in the way they are often found in the wild, not in the way that is easiest to understand. All three of these mistakes have been made in Kubernetes code, getting past code review at least once each that I know of.'

    (tags: k8s go golang errors coding bugs)

  • Health of purebred vs mixed breed dogs: the actual data - The Institute of Canine Biology

    This study found that purebred dogs have a significantly greater risk of developing many of the hereditary disorders examined in this study. No, mixed breed dogs are not ALWAYS healthier than purebreds; and also, purebreds are not "as healthy" as mixed breed dogs. The results of this study will surprise nobody who understands the basics of Mendelian inheritance. Breeding related animals increases the expression of genetic disorders caused by recessive mutations, and it also increases the probability of producing offspring that will inherit the assortment of genes responsible for a polygenic disorder. 
    In conclusion, go mutts.

    (tags: dogs breeding genetics hereditary-disorders science inheritance recessive-mutation data)

Links for 2016-03-15

Published March 15, 2016

  • DeepMind founder Demis Hassabis on how AI will shape the future | The Verge

    Good interview with Demis Hassabis on DeepMind, AlphaGo and AI:

    I’d like to see AI-assisted science where you have effectively AI research assistants that do a lot of the drudgery work and surface interesting articles, find structure in vast amounts of data, and then surface that to the human experts and scientists who can make quicker breakthroughs. I was giving a talk at CERN a few months ago; obviously they create more data than pretty much anyone on the planet, and for all we know there could be new particles sitting on their massive hard drives somewhere and no-one’s got around to analyzing that because there’s just so much data. So I think it’d be cool if one day an AI was involved in finding a new particle.

    (tags: ai deepmind google alphago demis-hassabis cern future machine-learning)

  • Before the Split

    Good post on Dublin City Council's atrociously revisionist 1916-commemoration banner, celebrating Henry Grattan, Daniel O'Connell, Charles Stewart Parnell and John Redmond:

    The banner is not showing parliamentary nationalists who might be included in a history of 1916 (Redmond might have been joined by John Dillon and Tom Kettle, for instance), but displaying the parliamentarian tradition in Irish political history. The people chosen all worked for change via political means, whether obtaining an independent Irish parliament from 1782-1801 (Grattan), working for Catholic Emancipation (Grattan and O’Connell), land reform (Parnell), or trying to repeal the Act of Union and obtain Home Rule (O’Connell, Parnell, Redmond). All were MPs in Westminster at some point. None openly espoused physical force. None aimed at establishing an independent Irish Republic. Putting the history of parliamentarianism on a banner labelled 1916 suggests that 1916 was in the parliamentarian tradition. That suggestion is very far from the truth.

    (tags: parliamentarianism 1916 history revisionism dcc dublin politics)

  • Flow

    a static type checker for Javascript, from Facebook

    (tags: javascript code-analysis coding facebook types strong-types)

Links for 2016-03-08

Published March 8, 2016

  • lbzip2

    a free, multi-threaded compression utility with support for bzip2 compressed file format. lbzip2 can process standard bz2 files in parallel. It uses POSIX threading model (pthreads), which allows it to take full advantage of symmetric multiprocessing (SMP) systems. It has been proven to scale linearly, even to over one hundred processor cores. lbzip2 is fully compatible with bzip2 – both at file format and command line level. Files created by lbzip2 can be decompressed by all versions of bzip2 and other software supporting bz2 format. lbzip2 can decompress any bz2 files in parallel. All bzip2 command-line options are also accepted by lbzip2. This makes lbzip2 a drop-in replacement for bzip2.

    (tags: bzip2 gzip compression lbzip2 parallel cli tools)

Links for 2016-03-07

Published March 7, 2016

Links for 2016-03-04

Published March 4, 2016

Links for 2016-03-03

Published March 3, 2016

  • Protect me, I am the Donnybrook laundry

    Mannix Flynn makes a persuasive case to preserve the last remaining Magdalene Laundry still standing:

    Memory is something that fights an eternal battle with the passage of time and forgetfulness.  Time is a great healer for those who can heal and those who are offered healing.  There is no healing here. Time stands still like a festering wound in a well-to-do suburb as somebody attempts to erase a grave and mortal wrong. The McAleese report, the Justice for the Magdalenes, the hundreds of women still alive and their families should know of this place.  Should be present here to witness what can only be witnessed by them.  So that they can understand what’s lost, what cannot be given.  What was taken from them for generations.

    (tags: magdalenes injustice ireland history catholic-church abuse mannix-flynn)

Links for 2016-03-01

Published March 1, 2016

Links for 2016-02-29

Published February 29, 2016

Links for 2016-02-26

Published February 26, 2016

  • Proportional Representation in Ireland: How it Works

    Excellent explanation of PR-STV and the Irish voting system. Don't be a Plumper! (via John O'Shea)

    (tags: plumpers pr-stv pr voting ireland politics via:joshea)

  • Microsoft warns of risks to Irish operation in US search warrant case

    “Our concern is that if we lose the case more countries across Europe or elsewhere are going to be concerned about having their data in Ireland, ” Mr Smith said, after testifying before the House judiciary committee. Asked what would happen to its Irish unit if the company loses the case or doesn’t convince Congress to pass updated legislation governing cross-border data held by American companies, the Microsoft executive said: “We’ll certainly face a new set of risks that we don’t face today.” He added that the issue could be resolved by an executive order by the White House or through international negotiations between the Irish Government or the European Union and the US.

    (tags: microsoft data privacy us-politics surveillance usa)

  • How To Implement Secure Bitcoin Vaults

    At the Bitcoin workshop in Barbados, Malte Möser will present our solution to the Bitcoin private key management problem. Specifically, our paper describes a way to create vaults, special accounts whose keys can be neutralized if they fall into the hands of attackers. Vaults are Bitcoin’s decentralized version of you calling your bank to report a stolen credit card -- it renders the attacker’s transactions null and void. And here’s the interesting part: in so doing, vaults demotivate key theft in the first place. An attacker who knows that he will not be able to get away with theft is less likely to attack in the first place, compared to current Bitcoin attackers who are guaranteed that their hacking efforts will be handsomely rewarded.

    (tags: private-keys vaults bitcoin security crypto theft)

Links for 2016-02-25

Published February 25, 2016

  • Maglev: A Fast and Reliable Software Network Load Balancer

    Maglev is Google’s network load balancer. It is a large distributed software system that runs on commodity Linux servers. Unlike traditional hardware network load balancers, it does not require a specialized physical rack deployment, and its capacity can be easily adjusted by adding or removing servers. Network routers distribute packets evenly to the Maglev machines via Equal Cost Multipath (ECMP); each Maglev machine then matches the packets to their corresponding services and spreads them evenly to the service endpoints. To accommodate high and ever-increasing traffic, Maglev is specifically optimized for packet processing performance. A single Maglev machine is able to saturate a 10Gbps link with small packets. Maglev is also equipped with consistent hashing and connection tracking features, to minimize the negative impact of unexpected faults and failures on connection-oriented protocols. Maglev has been serving Google's traffic since 2008. It has sustained the rapid global growth of Google services, and it also provides network load balancing for Google Cloud Platform.
    Something we argued for quite a lot in Amazon, back in the day....

    (tags: google paper scale ecmp load-balancing via:conall maglev lbs)

  • DIY DOG

    BrewDog releases their beer recipes for free. so cool! 'So here it is. The keys to our kingdom. Every single BrewDog recipe, ever. So copy them, tear them to pieces, bastardise them, adapt them, but most of all, enjoy them. They are well travelled but with plenty of miles still left on the clock. Just remember to share your brews, and share your results. Sharing is caring.'

    (tags: brewing homebrew beer brewdog open-source free sharing)

  • National Children’s Science Centre due to open in 2018

    Good for science fans, not so hot for real tennis fans.

    The former real tennis court building close to the concert hall’s north wing would be used for temporary and visiting exhibitors, with a tunnel connecting it to the science centre. The National Children’s Science Centre is due to open in late 2018 and will also be known as the Exploration Station, said Dr Danny O’Hare, founding president of Dublin City University and chairman of the Exploration Station board since 2006.

    (tags: real-tennis tennis nch dublin science kids planetarium)

Links for 2016-02-18

Published February 18, 2016

  • Neutrino Software Load Balancer

    eBay's software LB, supporting URL matching, comparable to haproxy, built using Netty and Scala. Used in their QA infrastructure it seems

    (tags: netty scala ebay load-balancing load-balancers url http architecture)

  • This is Why People Fear the ‘Internet of Things’

    Ugh. This is a security nightmare. Nice work Foscam...

    Imagine buying an internet-enabled surveillance camera, network attached storage device, or home automation gizmo, only to find that it secretly and constantly phones home to a vast peer-to-peer (P2P) network run by the Chinese manufacturer of the hardware. Now imagine that the geek gear you bought doesn’t actually let you block this P2P communication without some serious networking expertise or hardware surgery that few users would attempt. This is the nightmare “Internet of Things” (IoT) scenario for any system administrator: The IP cameras that you bought to secure your physical space suddenly turn into a vast cloud network designed to share your pictures and videos far and wide. The best part? It’s all plug-and-play, no configuration necessary!

    (tags: foscam cameras iot security networking p2p)

Links for 2016-02-16

Published February 16, 2016

  • The NSA’s SKYNET program may be killing thousands of innocent people

    Death by Random Forest: this project is a horrible misapplication of machine learning. Truly appalling, when a false positive means death:

    The NSA evaluates the SKYNET program using a subset of 100,000 randomly selected people (identified by their MSIDN/MSI pairs of their mobile phones), and a a known group of seven terrorists. The NSA then trained the learning algorithm by feeding it six of the terrorists and tasking SKYNET to find the seventh. This data provides the percentages for false positives in the slide above. "First, there are very few 'known terrorists' to use to train and test the model," Ball said. "If they are using the same records to train the model as they are using to test the model, their assessment of the fit is completely bullshit. The usual practice is to hold some of the data out of the training process so that the test includes records the model has never seen before. Without this step, their classification fit assessment is ridiculously optimistic." The reason is that the 100,000 citizens were selected at random, while the seven terrorists are from a known cluster. Under the random selection of a tiny subset of less than 0.1 percent of the total population, the density of the social graph of the citizens is massively reduced, while the "terrorist" cluster remains strongly interconnected. Scientifically-sound statistical analysis would have required the NSA to mix the terrorists into the population set before random selection of a subset—but this is not practical due to their tiny number. This may sound like a mere academic problem, but, Ball said, is in fact highly damaging to the quality of the results, and thus ultimately to the accuracy of the classification and assassination of people as "terrorists." A quality evaluation is especially important in this case, as the random forest method is known to overfit its training sets, producing results that are overly optimistic. The NSA's analysis thus does not provide a good indicator of the quality of the method.

    (tags: terrorism surveillance nsa security ai machine-learning random-forests horror false-positives classification statistics)

Links for 2016-02-15

Published February 15, 2016

  • Lasers reveal 'lost' Roman roads

    UK open data success story, via Tony Finch:

    This LIDAR data bonanza has proved particularly helpful to archaeologists seeking to map Roman roads that have been ‘lost’, some for thousands of years. Their discoveries are giving clues to a neglected chapter in the history of Roman Britain: the roads built to help Rome’s legions conquer and control northern England.

    (tags: uk government lidar open-data data roman history mapping geodata)

Links for 2016-02-13

Published February 13, 2016

Links for 2016-02-10

Published February 10, 2016

Links for 2016-02-09

Published February 9, 2016

Links for 2016-02-08

Published February 8, 2016

Links for 2016-02-05

Published February 5, 2016

  • The science behind "don't drink when pregnant" is rubbish

    As the economist Emily Oster pointed out in her 2013 book Expecting Better, there is also no “proven safe” level of Tylenol or caffeine, and yet both are fine in moderation during pregnancy. Oster pored through reams of research on alcohol and pregnancy for her book and concluded that there is simply no scientific evidence that light drinking during pregnancy impacts a baby’s health. (In one frequently cited 2001 study that suggested light drinking in pregnancy increases the chances of a child displaying aggressive behaviors, the drinkers were also significantly likelier to have taken cocaine during pregnancy.)
    My wife also followed the paper trail on this issue in the past. In the papers from which these recommendations were derived, the level of drinking at which any effects were observed in babies was when women consumed at least *9 units every day* for the entire pregnancy. That's an entire bottle of wine, daily!

    (tags: booze alcohol science facts papers medicine emily-oster babies pregnancy pre-pregnant research)

  • GCHQ's Spam Problem

    '“Spam emails are a large proportion of emails seen in SIGINT [signals intelligence],” reads part of a dense document from the Snowden archive, published by Boing Boing on Tuesday. “GCHQ would like to reduce the impact of spam emails on data storage, processing and analysis.”' (circa 2011). Steganography, anyone? (via Tony Finch)

    (tags: spam anti-spam gchq funny boing-boing sigint snowden surveillance)

  • ECHR: Websites not liable for readers' comments

    'Lawyers for [a Hungarian news] site said the comments concerned had been taken down as soon as they were flagged. They said making their clients liable for everything readers posted "would have serious adverse repercussions for freedom of expression and democratic openness in the age of Internet". The ECHR agreed. "Although offensive and vulgar, the incriminated comments did not constitute clearly unlawful speech; and they certainly did not amount to hate speech or incitement to violence," the judges wrote.'

    (tags: echr law eu legal comments index-hu hungary)

  • research!rsc: Zip Files All The Way Down

    quine.zip, quine.gz, and quine.tar.gz. Here's what happens when you mail it through bad AV software: https://twitter.com/FioraAeterna/status/694655296707297281

    (tags: zip algorithms compression quines fun hacks gzip)

  • The Nuclear Missile Sites of Los Angeles

    Great article by Geoff "bldgblog" Manaugh on the ruins of the Nike air-to-air missile emplacements dotted around California. I had absolutely no idea that these -- the 1958-era Nike-Hercules missiles, at least -- carried 30-kiloton nuclear warheads, intended to be detonated at 50,000 feet *above* the cities they were defending, in order to destroy in-flight bomber formations. Nuclear war was truly bananas.

    (tags: war history la sf california nike-missiles missiles nuclear-war nike-hercules cold-war 1950s)

Links for 2016-02-03

Published February 3, 2016

  • Exclusive: Snowden intelligence docs reveal UK spooks' malware checklist / Boing Boing

    This is an excellent essay from Cory Doctorow on mass surveillance in the post-Snowden era, and the difference between HUMINT and SIGINT. So much good stuff, including this (new to me) cite for, "Goodhart's law", on secrecy as it affects adversarial classification:

    The problem with this is that once you accept this framing, and note the happy coincidence that your paymasters just happen to have found a way to spy on everyone, the conclusion is obvious: just mine all of the data, from everyone to everyone, and use an algorithm to figure out who’s guilty. The bad guys have a Modus Operandi, as anyone who’s watched a cop show knows. Find the MO, turn it into a data fingerprint, and you can just sort the firehose’s output into ”terrorist-ish” and ”unterrorist-ish.” Once you accept this premise, then it’s equally obvious that the whole methodology has to be kept from scrutiny. If you’re depending on three ”tells” as indicators of terrorist planning, the terrorists will figure out how to plan their attacks without doing those three things. This even has a name: Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." Google started out by gauging a web page’s importance by counting the number of links they could find to it. This worked well before they told people what they were doing. Once getting a page ranked by Google became important, unscrupulous people set up dummy sites (“link-farms”) with lots of links pointing at their pages.

    (tags: adversarial-classification classification surveillance nsa gchq cory-doctorow privacy snooping goodharts-law google anti-spam filtering spying snowden)

Links for 2016-02-02

Published February 2, 2016

Links for 2016-01-30

Published January 30, 2016

  • Seesaw: scalable and robust load balancing from Google

    After evaluating a number of platforms, including existing open source projects, we were unable to find one that met all of our needs and decided to set about developing a robust and scalable load balancing platform. The requirements were not exactly complex - we needed the ability to handle traffic for unicast and anycast VIPs, perform load balancing with NAT and DSR (also known as DR), and perform adequate health checks against the backends. Above all we wanted a platform that allowed for ease of management, including automated deployment of configuration changes. One of the two existing platforms was built upon Linux LVS, which provided the necessary load balancing at the network level. This was known to work successfully and we opted to retain this for the new platform. Several design decisions were made early on in the project — the first of these was to use the Go programming language, since it provided an incredibly powerful way to implement concurrency (goroutines and channels), along with easy interprocess communication (net/rpc). The second was to implement a modular multi-process architecture. The third was to simply abort and terminate a process if we ended up in an unknown state, which would ideally allow for failover and/or self-recovery.

    (tags: seesaw load-balancers google load-balancing vips anycast nat lbs go ops networking)

Links for 2016-01-29

Published January 29, 2016

Links for 2016-01-28

Published January 28, 2016

Links for 2016-01-27

Published January 27, 2016

Links for 2016-01-26

Published January 26, 2016

Links for 2016-01-25

Published January 25, 2016

  • Netflix Global Search

    handy -- search Netflix in all regions, then show where the show/movie is available. Probably going to be less handy from now on now that Netflix is blocking region-spoofing

    (tags: movies video netflix films tv world)

  • Why Eircode is a shambles, by someone who works in the transport industry

    This is full of good points.

    Without having a distinct SORT KEY for a geographically distinct area, a postcode is of no real benefit to any type of transport firm or agency.  To take one example, Eircode have used the same sort key, F92, for Arranmore (Donegal’s largest inhabited island) and the north western Donegal mainland.  Cill Rónáin, Inis Mór, the largest of the Aran Islands, has the same sort key H91, as Connemara and Galway City.  Galway city and the Aran Islands may be in a relatively small geographical area, but keen eyes may have noticed that the Aran Islands are separated from the mainland by a small section of the Atlantic Ocean.  Sort codes which ignore clear and obvious boundaries, like seas or oceans, need to be redesigned. In two seconds a [UK] website could tell a Hebridean that his delivery will take 4 days at a cost of fifty quid by using the first three characters of the postcode.  The Eircode-using Irish equivalent website would need to lookup a large database to tell an Arranmore resident the cost and time for delivery – and they’d need the full exact code.  Any mistake made here, and your estimated delivery time, and cost for delivery will be wrong.

    (tags: postcodes eircode loc8code fail couriers delivery geodata geocoding galway aran-islands)

Links for 2016-01-22

Published January 22, 2016

Links for 2016-01-21

Published January 21, 2016

Links for 2016-01-20

Published January 20, 2016

Links for 2016-01-16

Published January 16, 2016

  • Yosemite agrees to change the names of its significant locations to appease trademark troll / Boing Boing

    This is absolutely appalling. IP law gone mad:

    DNC Parks & Resorts at Yosemite, Inc (a division of one of the largest privately owned companies in the world) used to have the concessions to operate various businesses around Yosemite National Park. Now that they've been fired, they're using some decidedly dubious trademark to force the Park Service to change the names of buildings and locations that have stood for as much as a century, including some that have been designated national landmarks. The Parks Service has caved to these requests as it readies the park for its centennial celebration. It will not only change the names of publicly owned landmarks -- such as the Ahwahnee hotel, Yosemite Lodge, the Wawona Hotel, Curry Village, and Badger Pass ski area -- it will also have to change all its signs, maps and guidebooks.

    (tags: yosemite ip trademarks law fiasco national-parks usa)

Links for 2016-01-12

Published January 12, 2016

Links for 2016-01-08

Published January 8, 2016

Links for 2016-01-06

Published January 6, 2016

Links for 2016-01-04

Published January 4, 2016

Links for 2015-12-31

Published December 31, 2015

Links for 2015-12-22

Published December 22, 2015

  • Amazon EC2 Container Registry

    hooray, Docker registry here at last

    (tags: ecs docker registry ops containers aws)

  • How to inspect SSL/TLS traffic with Wireshark 2

    turns out it's easy enough -- Mozilla standardised a debugging SSL session-key logging file format which Wireshark and Chrome support

    (tags: chrome ssl browser firefox wireshark debugging tls)

  • ImperialViolet - Juniper: recording some Twitter conversations

    Adam Langley on the Juniper VPN-snooping security hole:

    ... if it wasn't the NSA who did this, we have a case where a US gov­ern­ment back­door ef­fort (Dual-EC) laid the ground­work for some­one else to at­tack US in­ter­ests. Cer­tainly this at­tack would be a lot eas­ier given the pres­ence of a back­door-friendly RNG al­ready in place. And I've not even dis­cussed the SSH back­door. [...]

    (tags: primes ecc security juniper holes exploits dual-ec-drbg vpn networking crypto prngs)

  • Excellent post from Matthew Green on the Juniper backdoor

    For the past several years, it appears that Juniper NetScreen devices have incorporated a potentially backdoored random number generator, based on the NSA's Dual_EC_DRBG algorithm. At some point in 2012, the NetScreen code was further subverted by some unknown party, so that the very same backdoor could be used to eavesdrop on NetScreen connections. While this alteration was not authorized by Juniper, it's important to note that the attacker made no major code changes to the encryption mechanism -- they only changed parameters. This means that the systems were potentially vulnerable to other parties, even beforehand. Worse, the nature of this vulnerability is particularly insidious and generally messed up. [....] The end result was a period in which someone -- maybe a foreign government -- was able to decrypt Juniper traffic in the U.S. and around the world. And all because Juniper had already paved the road. One of the most serious concerns we raise during [anti-law-enforcement-backdoor] meetings is the possibility that encryption backdoors could be subverted. Specifically, that a back door intended for law enforcement could somehow become a backdoor for people who we don't trust to read our messages. Normally when we talk about this, we're concerned about failures in storage of things like escrow keys. What this Juniper vulnerability illustrates is that the danger is much broader and more serious than that. The problem with cryptographic backdoors is not that they're the only way that an attacker can break intro our cryptographic systems. It's merely that they're one of the best. They take care of the hard work, the laying of plumbing and electrical wiring, so attackers can simply walk in and change the drapes.
    (via Tony Finch)

    (tags: via:fanf crypto backdoors politics juniper dual-ec-drbg netscreen vpn)

  • 2016 Wish List for AWS?

    good thread of AWS' shortcomings -- so many services still don't handle VPC for instance

    (tags: vpc aws ec2 ops wishlist)

Links for 2015-12-18

Published December 18, 2015

Links for 2015-12-16

Published December 16, 2015

Links for 2015-12-15

Published December 15, 2015

Links for 2015-12-14

Published December 14, 2015

  • Files Are Hard

    This is basically terrifying. A catalog of race conditions and reliability horrors around the POSIX filesystem abstraction in Linux -- it's a wonder anything works. 'Where’s this documented? Oh, in some mailing list post 6-8 years ago (which makes it 12-14 years from today). The fs devs whose posts I’ve read are quite polite compared to LKML’s reputation, and they generously spend a lot of time responding to basic questions, but it’s hard for outsiders to troll [sic] through a decade and a half of mailing list postings to figure out which ones are still valid and which ones have been obsoleted! I don’t mean to pick on filesystem devs. In their OSDI 2014 talk, the authors of the paper we’re discussing noted that when they reported bugs they’d found, developers would often respond “POSIX doesn’t let filesystems do that”, without being able to point to any specific POSIX documentation to support their statement. If you’ve followed Kyle Kingsbury’s Jepsen work, this may sound familiar, except devs respond with “filesystems don’t do that” instead of “networks don’t do that”.I think this is understandable, given how much misinformation is out there. Not being a filesystem dev myself, I’d be a bit surprised if I don’t have at least one bug in this post.'

    (tags: filesystems linux unix files operating-systems posix fsync osdi papers reliability)

  • [LUCENE-6917] Deprecate and rename NumericField/RangeQuery to LegacyNumeric - ASF JIRA

    Interesting performance-related tweak going into Lucene -- based on the Bkd-Tree I think: https://users.cs.duke.edu/~pankaj/publications/papers/bkd-sstd.pdf . Being used for all numeric index types, not just multidimensional ones?

    (tags: lucene performance algorithms patches bkd-trees geodata numeric indexing)

  • Kevin Lyda's mega pension post

    Cutting and pasting from Facebook for posterity... there are some really solid tips in here. 'Some people plan their lives out and then there are people like me who randomly do things and suddenly, in retrospect, it looks like a grand plan has come together. In reality it's more like my subconscious pulls in useful info and pokes me to go learn things as required. If you live/work in Ireland, the following "grand plan" might be useful. This year has apparently been "figure out how to retire" year. It started late last year with finally organising all my private Irish pensions (2 from employers, 1 personal). In the process I learned the following: * Many Irish pension plans allow you to start drawing down from them at age 50. There are downsides to this, but if you have several of them it allows you more room to avoid stock market downturns when you purchase annuities. * You can get 25% of each pension as a tax-free lump sum. I also learned a few property things. The key thing is that if you have a buy-to-let property you should *not* pay off its mortgage early. You can deduct 75% of the interest you pay against the taxes you'd owe for rental income. That means the interest you pay will essentially be close to or even under the rate of inflation. A residential mortgage might have a lower interest rate nominally, but the effective interest rate is higher. The Irish state pension is changing. If you are 68 after 2020 the rules have changed - and they're now much simpler. Work for 10 years and you get the minimum state pension (1/3 of a full pension). Work for 20, you get 2/3 of of a state pension. Work for 30, you get a full pension. But you can't collect it till you're 68 and remember that Irish employers can apparently force you to "retire" at 65 (ageism is legal). So you need to bridge those 3 years (or hope they change the law to stop employers from doing that). When I "retired" I kept a part time job for a number of reasons, but one was because I suspected I needed more PRSI credits for a pension. And it turns out this was correct. Part-time work counts as long as you make more than €38/week. And self-employment counts as long as you make more than €5,000/year. You can also make voluntary PRSI contributions (around €500/year but very situation dependent). If you've worked in Europe or the US or Canada or a few other countries, you can get credits for social welfare payments in those countries. But if you have enough here and you have enough for some pension in the other country, you can draw a pension from both. Lastly most people I've talked to about retirement this year have used the analogy of legs on a stool. Every source of post-retirement income is a leg on the stool - the more legs, the more secure your retirement. There are lots of options for legs: * Rental income. This is a little wobbly as legs go at least for me. But if you have more than one rental property - and better yet some commercial rental property - this leg firms up a bit. Still, it's a bit more work than most. * Savings. This isn't very tax-efficient, but it can help fill in blank spots some legs have (like rental income or age restrictions) or maximise another legs value (weathering downturns for stock-based legs). And in retirement you can even build savings up. Sell a house, the private pension lump sum, etc. But remember you're retired, go have fun. Savings won't do you much good when you're dead. * Stocks. I've cashed all mine in, but some friends have been more restrained in cashing in stocks they might have gotten from employers. This is a volatile leg, but it can pay off rather well if you know what you're doing. But be honest with yourself. I know I absolutely don't know what I'm doing on this so stayed away. * Government pension. This is generally a reliable source of income in retirement. It's usually not a lot, but it does tend to last from retirement to death and it shows up every month. You apply once and then it just shows up each month. If you've worked in multiple countries, you can hedge some bets by taking a pension in each country you qualify from. You did pay into them after all. * Private pension. This can also give you a solid source of income but you need to pay into it. And paying in during your 20s and 30s really pays off later. But you need to make your investments less risky as you get into your late 50s - so make sure to start looking at them then. And you need to provide yourself some flexibility for starting to draw it down in order to survive market drops. The crash in 2007 didn't fully recover until 2012 - that's 5 years. * Your home. Pay off your mortgage and your home can be a leg. Not having to pay rent/mortgage is a large expense removed and makes the other legs more effective. You can also "sell down" or look into things like reverse mortgages, but the former can take time and has costs while the latter usually seems to have a lot of fine print you should read up on. * Part-time work. I know a number of people who took part-time jobs when they retired. If you can find something that doesn't take a huge amount of time that you'd enjoy doing and that people will pay you for, fantastic! Do that. And it gets you out of the house and keeping active. For friends who are geeks and in my age cohort, I note that it will be 2037 around the time we hit 65. If you know why that matters, ka-ching!' Another particularly useful page about the state pension: "Six things every woman needs to know about the State pension", Irish Times, Dec 1 2015, https://www.irishtimes.com/business/personal-finance/six-things-every-woman-needs-to-know-about-the-state-pension-1.2448981 , which links to this page to get your state pension contribution record: http://www.welfare.ie/en/pages/secure/ RequestSIContributionRecord.aspx

    (tags: pensions money life via:klyda stocks savings shares property ireland old-age retirement)

  • Big Brother Watch on Twitter: "Anyone can legally have their phone or computer hacked by the police, intelligence agencies, HMRC and others #IPBill https://t.co/3ZS610srCJ"

    As Glynn Moody noted, if UK police, intelligence agencies, HMRC and others call all legally hack phones and computers, that also means that digital evidence can be easily and invisibly planted. This will undermine future court cases in the UK, which seems like a significant own goal...

    (tags: hmrc police gchq uk hacking security law-enforcement evidence law)

  • Why We Chose Kubernetes Over ECS

    3 months ago when we, at nanit.com, came to evaluate which Docker orchestration framework to use, we gave ECS the first priority. We were already familiar with AWS services, and since we already had our whole infrastructure there, it was the default choice. After testing the service for a while we had the feeling it was not mature enough and missing some key features we needed (more on that later), so we went to test another orchestration framework: Kubernetes. We were glad to discover that Kubernetes is far more comprehensive and had almost all the features we required. For us, Kubernetes won ECS on ECS’s home court, which is AWS.

    (tags: kubernetes ecs docker containers aws ec2 ops)

Links for 2015-12-11

Published December 11, 2015

Links for 2015-12-10

Published December 10, 2015

Links for 2015-12-09

Published December 9, 2015

Links for 2015-12-07

Published December 7, 2015

Links for 2015-12-03

Published December 3, 2015

  • Introducing Netty-HTTP from Cask

    netty-http library solves [Netty usability issues] by using JAX-RS annotations to build a HTTP path routing layer on top of netty. In addition, the library implements a guava service to manage the HTTP service. netty-http allows users of the library to just focus on writing the business logic in HTTP handlers without having to worry about the complexities of path routing or learning netty pipeline internals to build the HTTP service.
    We've written something very similar, although I didn't even bother supporting JAX-RS annotations -- just a simple code-level DSL.

    (tags: jax-rs netty http cask java services coding)

  • The Locals Xmas Gift Guide 2015

    some nice local gift suggestions from small businesses around Dublin. I'd love to get some of these, but I guess I'll have to settle for giving them instead ;)

    (tags: gifts dublin ireland shopping xmas christmas the-locals)

Links for 2015-12-02

Published December 2, 2015

  • Topics in High-Performance Messaging

    'We have worked together in the field of high-performance messaging for many years, and in that time, have seen some messaging systems that worked well and some that didn't. Successful deployment of a messaging system requires background information that is not easily available; most of what we know, we had to learn in the school of hard knocks. To save others a knock or two, we have collected here the essential background information and commentary on some of the issues involved in successful deployments. This information is organized as a series of topics around which there seems to be confusion or uncertainty. Please contact us if you have questions or comments.'

    (tags: messaging scalability scaling performance udp tcp protocols multicast latency)

Links for 2015-11-30

Published November 30, 2015

  • Control theory meets machine learning

    'DB: Is there a difference between how control theorists and machine learning researchers think about robustness and error? BR: In machine learning, we almost always model our errors as being random rather than worst-case. In some sense, random errors are actually much more benign than worst-case errors. [...] In machine learning, by assuming average-case performance, rather than worst-case, we can design predictive algorithms by averaging out the errors over large data sets. We want to be robust to fluctuations in the data, but only on average. This is much less restrictive than the worst-case restrictions in controls. DB: So control theory is model-based and concerned with worst case. Machine learning is data based and concerned with average case. Is there a middle ground? BR: I think there is! And I think there's an exciting opportunity here to understand how to combine robust control and reinforcement learning. Being able to build systems from data alone simplifies the engineering process, and has had several recent promising results. Guaranteeing that these systems won't behave catastrophically will enable us to actually deploy machine learning systems in a variety of applications with major impacts on our lives. It might enable safe autonomous vehicles that can navigate complex terrains. Or could assist us in diagnostics and treatments in health care. There are a lot of exciting possibilities, and that's why I'm excited about how to find a bridge between these two viewpoints.'

    (tags: control-theory interviews machine-learning ml worst-case self-driving-cars cs)

  • The End of Dynamic Languages

    This is my bet: the age of dynamic languages is over. There will be no new successful ones. Indeed we have learned a lot from them. We’ve learned that library code should be extendable by the programmer (mixins and meta-programming), that we want to control the structure (macros), that we disdain verbosity. And above all, we’ve learned that we want our languages to be enjoyable. But it’s time to move on. We will see a flourishing of languages that feel like you’re writing in a Clojure, but typed. Included will be a suite of powerful tools that we’ve never seen before, tools so convincing that only ascetics will ignore.

    (tags: programming scala clojure coding types strong-types dynamic-languages languages)

  • RobustIRC

    'IRC without netsplits' using Raft consensus

    (tags: raft irc netsplits resilience fault-tolerance)

  • Inside China's Memefacturing Factories, Where The Hottest New Gadgets Are Made - BuzzFeed News

    On a humid afternoon, Zhou went shopping for some of those very parts at a Bao An market. As he pulled his maroon minivan into a crowded parking lot, the full scale of Depu Electronics came into view: a three-story concrete behemoth roughly bigger than a Costco and roughly smaller than the Pentagon. Inside, it looked like the world’s largest Radio Shack going out of business sale: an endless series of booths with cables and circuit boards and plugs and ports and buttons and machines piled so high on tables that the faces of the clerks who were selling them were hidden from view. Each booth seemed to argue: We have exactly what you want and we have enough of it for all of your customers. Short of motorized wheels and molding, the market offered nearly everything an ambitious factory owner would need to build a hoverboard, just waiting to be bought, assembled, and shipped.

    (tags: hoverboards memes china manufacturing future gadgets tat bao-an electronics)

Links for 2015-11-28

Published November 28, 2015

Links for 2015-11-26

Published November 26, 2015

  • re:Work - The five keys to a successful Google team

    We learned that there are five key dynamics that set successful teams apart from other teams at Google: Psychological safety: Can we take risks on this team without feeling insecure or embarrassed? Dependability: Can we count on each other to do high quality work on time? Structure & clarity: Are goals, roles, and execution plans on our team clear? Meaning of work: Are we working on something that is personally important for each of us? Impact of work: Do we fundamentally believe that the work we’re doing matters?

    (tags: teams google culture work management productivity hr)

Links for 2015-11-25

Published November 25, 2015

Links for 2015-11-22

Published November 22, 2015

Links for 2015-11-20

Published November 20, 2015

Links for 2015-11-18

Published November 18, 2015

Links for 2015-11-17

Published November 17, 2015

Links for 2015-11-16

Published November 16, 2015

Links for 2015-11-13

Published November 13, 2015

  • The impact of Docker containers on the performance of genomic pipelines [PeerJ]

    In this paper, we have assessed the impact of Docker containers technology on the performance of genomic pipelines, showing that container “virtualization” has a negligible overhead on pipeline performance when it is composed of medium/long running tasks, which is the most common scenario in computational genomic pipelines. Interestingly for these tasks the observed standard deviation is smaller when running with Docker. This suggests that the execution with containers is more “homogeneous,” presumably due to the isolation provided by the container environment. The performance degradation is more significant for pipelines where most of the tasks have a fine or very fine granularity (a few seconds or milliseconds). In this case, the container instantiation time, though small, cannot be ignored and produces a perceptible loss of performance.

    (tags: performance docker ops genomics papers)

Links for 2015-11-12

Published November 12, 2015

Links for 2015-11-11

Published November 11, 2015

  • Dynalite

    Awesome new mock DynamoDB implementation:

    An implementation of Amazon's DynamoDB, focussed on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise). This project aims to match the live DynamoDB instances as closely as possible (and is tested against them in various regions), including all limits and error messages. Why not Amazon's DynamoDB Local? Because it's too buggy! And it differs too much from the live instances in a number of key areas.
    We use DynamoDBLocal in our tests -- the availability of that tool is one of the key reasons we have adopted Dynamo so heavily, since we can safely test our code properly with it. This looks even better.

    (tags: dynamodb testing unit-tests integration-testing tests ops dynalite aws leveldb)

  • Alarm design: From nuclear power to WebOps

    Imagine you are an operator in a nuclear power control room. An accident has started to unfold. During the first few minutes, more than 100 alarms go off, and there is no system for suppressing the unimportant signals so that you can concentrate on the significant alarms. Information is not presented clearly; for example, although the pressure and temperature within the reactor coolant system are shown, there is no direct indication that the combination of pressure and temperature mean that the cooling water is turning into steam. There are over 50 alarms lit in the control room, and the computer printer registering alarms is running more than 2 hours behind the events. This was the basic scenario facing the control room operators during the Three Mile Island (TMI) partial nuclear meltdown in 1979. The Report of the President’s Commission stated that, “Overall, little attention had been paid to the interaction between human beings and machines under the rapidly changing and confusing circumstances of an accident” (p. 11). The TMI control room operator on the day, Craig Faust, recalled for the Commission his reaction to the incessant alarms: “I would have liked to have thrown away the alarm panel. It wasn’t giving us any useful information”. It was the first major illustration of the alarm problem, and the accident triggered a flurry of human factors/ergonomics (HF/E) activity.
    A familiar topic for this ex-member of the Amazon network monitoring team...

    (tags: ergonomics human-factors ui ux alarms alerts alerting three-mile-island nuclear-power safety outages ops)

  • An Analysis of Reshipping Mule Scams

    We observed that the vast majority of the re-shipped packages end up in the Moscow, Russia area, and that the goods purchased with stolen credit cards span multiple categories, from expensive electronics such as Apple products, to designer clothes, to DSLR cameras and even weapon accessories. Given the amount of goods shipped by the reshipping mule sites that we analysed, the annual revenue generated from such operations can span between 1.8 and 7.3 million US dollars. The overall losses are much higher though: the online merchant loses an expensive item from its inventory and typically has to refund the owner of the stolen credit card. In addition, the rogue goods typically travel labeled as “second hand goods” and therefore custom taxes are also evaded. Once the items purchased with stolen credit cards reach their destination they will be sold on the black market by cybercriminals. [...] When applying for the job, people are usually required to send the operator copies of their ID cards and passport. After they are hired, mules are promised to be paid at the end of their first month of employment. However, from our data it is clear that mules are usually never paid. After their first month expires, they are never contacted back by the operator, who just moves on and hires new mules. In other words, the mules become victims of this scam themselves, by never seeing a penny. Moreover, because they sent copies of their documents to the criminals, mules can potentially become victims of identity theft.

    (tags: crime law cybercrime mules shipping-scams identity-theft russia moscow scams papers)