Eircode cost the Irish government EUR38m
The C&AG has said it is not clear that the €38m scheme will achieve the data-matching benefits the Government had hoped.
Well, that’s putting it mildly.(tags: eircode fail ireland costs money geo mapping geocoding)
Let a 1,000 flowers bloom. Then rip 999 of them out by the roots
The Twitter tech-debt story.
Somewhere along the way someone decided that it would be easier to convert the Birdcage to use Pants which had since learned how to build Scala and to deal with a maven-style layout. However at some point prior Pants been open sourced in throw it over the wall fashion and picked up by a few engineers at other companies, such as Square and Foursquare and moved forward. In the meantime, again because there weren’t enough people who’s job it was to take care of these things, Science was still on the original internally developed version and had in fact evolved independently of the open source version. However by the time we wanted to move Birdcage onto Pants, the open source version had moved ahead so that’s the one the Birdcage folks chose.
(cries)(tags: tech-debt management twitter productivity engineering monorepo build-systems war-stories dev)
-
Amazing. This is what happens when embedded software engineers make a UI, in my experience
(tags: embedded-software ui ux design graphics windows the-horror omgwtf atms)
Justin's Linklog Posts
EPA opposed rules that would have exposed VW’s cheating
[…] Two months ago, the EPA opposed some proposed measures that would help potentially expose subversive code like the so-called “defeat device” software VW allegedly used by allowing consumers and researchers to legally reverse-engineer the code used in vehicles. EPA opposed this, ironically, because the agency felt that allowing people to examine the software code in vehicles would potentially allow car owners to alter the software in ways that would produce more emissions in violation of the Clean Air Act. The issue involves the 1998 Digital Millennium Copyright Act (DCMA), which prohibits anyone from working around “technological protection measures” that limit access to copyrighted works. The Library of Congress, which oversees copyrights, can issue exemptions to those prohibitions that would make it legal, for example, for researchers to examine the code to uncover security vulnerabilities.
(tags: dmca volkswagen vw law code open-source air-quality diesel cheating regulation us-politics)
From Radio to Porn, British Spies Track Web Users’ Online Identities
Inside KARMA POLICE, GCHQ’s mass-surveillance operation aimed to record the browsing habits of “every visible user on the internet”, including UK-to-UK internal traffic. more details on the other GCHQ mass surveillance projects at https://theintercept.com/gchq-appendix/
(tags: surveillance gchq security privacy law uk ireland karma-police snooping)
Streaming will soon pass traditional TV – Tech Insider
the percentage of people who say they stream video from services like Netflix, YouTube, and Hulu each day has increased dramatically over the last five years, from about 30% in 2010 to more than 50% this year. During the same period, the percentage of people who say they watch traditional TV […] has dropped by about 10%. When the beige line surpasses the purple line [looks like 2016], it will mean that more people are streaming each day than are watching traditional TV.
Is there a CAP theorem for Durability?
Marc Brooker with another thought-provoking blogpost
(tags: databases storage marc-brooker cap-theorem cap durability pacelc nosql)
-
(via Aman)
(tags: via:akohli graphics ascii-art ascii visualization text boxes diagrams)
Scale it to Billions — What They Don’t Tell you in the Cassandra README
large-scale C* tips
(tags: cassandra configuration tuning scale ops)
Introduction to HDFS Erasure Coding in Apache Hadoop
How Hadoop did EC. Erasure Coding support (“HDFS-EC”) is set to be released in Hadoop 3.0 apparently
(tags: erasure-coding reed-solomon algorithms hadoop hdfs cloudera raid storage)
-
some details on Netflix’s Chaos Monkey, Chaos Kong and other aspects of their availability/failover testing
(tags: architecture aws netflix ops chaos-monkey chaos-kong testing availability failover ha)
-
Træf?k is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends (Docker , Mesos/Marathon, Consul, Etcd, Rest API, file…) to manage its configuration automatically and dynamically.
Hot-reloading is notably much easier than with nginx/haproxy. -
a proxy that mucks with your system and application context, operating at Layers 4 and 7, allowing you to simulate common failure scenarios from the perspective of an application under test; such as an API or a web application. If you are building a distributed system, Muxy can help you test your resilience and fault tolerance patterns.
(tags: proxy distributed testing web http fault-tolerance failure injection tcp delay resilience error-handling)
Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances — AdRoll
nice approach
(tags: data-pipelines docker luigi containers workflow)
-
a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don’t need 100% hindsight to be able to obtain the information you need.
(tags: tracing java byteman injection jvm ops debugging testing)
Henry Robinson on testing and fault discovery in distributed systems
‘Let’s talk about finding bugs in distributed systems for a bit. These chaos monkey-style fault testing systems are all well and good, but by being application independent they’re a very blunt instrument. Particularly they make it hard to search the fault space for bugs in a directed manner, because they don’t ‘know’ what the system is doing. Application-aware scripting of faults in a dist. systems seems to be rarely used, but allows you to directly stress problem areas. For example, if a bug manifests itself only when one RPC returns after some timeout, hard to narrow that down with iptables manipulation. But allow a script to hook into RPC invocations (and other trace points, like DTrace’s probes), and you can script very specific faults. That way you can simulate cross-system integration failures, *and* write reproducible tests for the bugs they expose! Anyhow, I’ve been doing this in Impala, and it’s been very helpful. Haven’t seen much evidence elsewhere.’
(tags: henry-robinson testing fault-discovery rpc dtrace tracing distributed-systems timeouts chaos-monkey impala)
The Best Bourbon Cocktail You’ve Never Heard Of
The “Paper Plane”, by Sam Ross of Chicago’s “Violet Hour”: .75 oz Bourbon .75 oz Aperol .75 oz Amaro Nonino .75 oz Fresh lemon juice ice-filled shaker, shake, strain.
(tags: bourbon drinks cocktails recipes aperol amaro-nonino lemon)
-
C++ high-performance app framework; ‘currently focused on high-throughput, low-latency I/O intensive applications.’ Scylla (Cassandra-compatible NoSQL store) is written in this.
(tags: c++ opensource performance framework scylla seastar latency linux shared-nothing multicore)
How VW tricked the EPA’s emissions testing system
In July 2015, CARB did some follow up testing and again the cars failed—the scrubber technology was present, but off most of the time. How this happened is pretty neat. Michigan’s Stefanopolou says computer sensors monitored the steering column. Under normal driving conditions, the column oscillates as the driver negotiates turns. But during emissions testing, the wheels of the car move, but the steering wheel doesn’t. That seems to have have been the signal for the “defeat device” to turn the catalytic scrubber up to full power, allowing the car to pass the test. Stefanopolou believes the emissions testing trick that VW used probably isn’t widespread in the automotive industry. Carmakers just don’t have many diesels on the road. And now that number may go down even more.
Depressing stuff — but at least they think VW’s fraud wasn’t widespread.(tags: fraud volkswagen vw diesel emissions air-quality epa carb catalytic-converters testing)
EU court adviser: data-share deal with U.S. is invalid | Reuters
The Safe Harbor agreement does not do enough to protect EU citizen’s private information when it reached the United States, Yves Bot, Advocate General at the European Court of Justice (ECJ), said. While his opinions are not binding, they tend to be followed by the court’s judges, who are currently considering a complaint about the system in the wake of revelations from ex-National Security Agency contractor Edward Snowden of mass U.S. government surveillance.
(tags: safe-harbor law eu ec ecj snowden surveillance privacy us data max-schrems)
Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region
Painful to read, but: tl;dr: monitoring oversight, followed by a transient network glitch triggering IPC timeouts, which increased load due to lack of circuit breakers, creating a cascading failure
(tags: aws postmortem outages dynamodb ec2 post-mortems circuit-breakers monitoring)
What Happens Next Will Amaze You
Maciej Ceglowski’s latest talk, on ads, the web, Silicon Valley and government:
‘I went to school with Bill. He’s a nice guy. But making him immortal is not going to make life better for anyone in my city. It will just exacerbate the rent crisis.’
(tags: talks slides funny ads advertising internet web privacy surveillance maciej silicon-valley)
Frame of Reference and Roaring Bitmaps
interesting performance-oriented algorithm tweak from Elastic/Lucene
(tags: lucene elasticsearch performance optimization roaring-bitmaps bitmaps frame-of-reference integers algorithms)
Uber Goes Unconventional: Using Driver Phones as a Backup Datacenter – High Scalability
Initially I thought they were just tracking client state on the phone, but it actually sounds like they’re replicating other users’ state, too. Mad stuff! Must cost a fortune in additional data transfer costs…
(tags: scalability failover multi-dc uber replication state crdts)
Brotli: a new compression algorithm for the internet from Google
While Zopfli is Deflate-compatible, Brotli is a whole new data format. This new format allows us to get 20–26% higher compression ratios over Zopfli. In our study ‘Comparison of Brotli, Deflate, Zopfli, LZMA, LZHAM and Bzip2 Compression Algorithms’ we show that Brotli is roughly as fast as zlib’s Deflate implementation. At the same time, it compresses slightly more densely than LZMA and bzip2 on the Canterbury corpus. The higher data density is achieved by a 2nd order context modeling, re-use of entropy codes, larger memory window of past data and joint distribution codes. Just like Zopfli, the new algorithm is named after Swiss bakery products. Brötli means ‘small bread’ in Swiss German.
(tags: brotli zopfli deflate gzip compression algorithms swiss google)
-
‘The key thing about Ubiquiti gear is the high quality radios and antennas. It just seems much more reliable than most consumer WiFi gear. Their airOS firmware is good too, it’s a bit complicated to set up but very capable and flexible. And in addition to normal 802.11n or 802.11ac they also have an optional proprietary TDMA protocol called airMax that’s designed for serving several long haul links from a single basestation. They’re mostly marketing to business customers but the equipment is sold retail and well documented for ordinary nerds to figure out.’
(tags: ubiquiti wifi wireless 802.11 via:nelson ethernet networking prosumer hardware wan)
-
a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.
via Eoin Brazil(tags: via:eoinbrazil httpry http networking tools ops testing tcpdump tracing)
ustwo Reimagines the In-Car Cluster
Designers behind the cult mobile game, Monument Valley, take on the legacy-bound in-car UI
(tags: ux ui cars driving safety ustwo monument-valley speed)
-
‘It’s very easy: So long as you don’t hear “The Little Drummer Boy,” you’re a contender. As soon as you hear it on the radio, on TV, in a store, wherever, you’re out.’
Geographically-accurate version of the London underground map
as Boing Boing says: ‘London’s subway system switched early to an abstract map (PDF), and it became a legendary work of design. It just published an internally-used geographic version of map (PDF), however, for the first time in a century—and it’s awesome.’
(tags: london maps mapping geography accuracy pdf subway underground)
Critiki’s top 10 tiki bars in the world
not a one in Europe, of course! I need to hit up one of these sometime
(tags: tiki bars drinks polynesian midcentury trader-vic critiki)
What is the fastest way to clone a git repository over a fast network connection? – Stack Overflow
“git bundle create” — neat trick
(tags: git distribution copying git-bundle cli)
-
a regex-based, Turing-complete programming language. It’s main feature is taking some text via standard input and repeatedly applying regex operations to it (e.g. matching, splitting, and most of all replacing). Under the hood, it uses .NET’s regex engine, which means that both the .NET flavour and the ECMAScript flavour are available.
Reminscent of sed(1); see http://codegolf.stackexchange.com/a/58166 for an example Retina program(tags: retina regexps regexes regular-expressions coding hacks dot-net languages)
Time on multi-core, multi-socket servers
Nice update on the state of System.currentTimeMillis() and System.nanoTime() in javaland. Bottom line: both are non-monotonic nowadays:
The conclusion I’ve reached is that except for the special case of using nanoTime() in micro benchmarks, you may as well stick to currentTimeMillis() —knowing that it may sporadically jump forwards or backwards. Because if you switched to nanoTime(), you don’t get any monotonicity guarantees, it doesn’t relate to human time any more —and may be more likely to lead you into writing code which assumes a fast call with consistent, monotonic results.
(tags: java time monotonic sequencing nanotime timers jvm multicore distributed-computing)
Anatomy of a Modern Production Stack
Interesting post, but I think it falls into a common trap for the xoogler or ex-Amazonian — assuming that all the BigCo mod cons are required to operate, when some are luxuries than can be skipped for a few years to get some real products built
(tags: architecture ops stack docker containerization deployment containers rkt coreos prod monitoring xooglers)
How We Use AWS Lambda for Rapidly Intensifying Workloads · CloudSploit
impressive — pretty much the entire workload is run from Lambda here
(tags: lambda aws ec2 autoscaling cloudsploit)
Introducing the Software Testing Cupcake (Anti-Pattern)
good post on the risks of overweighting towards manual testing rather than low-level automated tests (via Tony Byrne)
(tags: qa testing via:tonyjbyrne tests antipatterns dev)
Kate Heddleston: How Our Engineering Environments Are Killing Diversity
‘[There are] several problem areas for [diversity in] engineering environments and ways to start fixing them. The problems we face aren’t devoid of solutions; there are a lot of things that companies, teams, and individuals can do to fix problems in their work environment. For the month of March, I will be posting detailed articles about the problem areas I will cover in my talk: argument cultures, feedback, promotions, employee on-boarding, benefits, safety, engineering process, and environment adaptation.’ via Baron Schwartz.
(tags: via:xaprb culture tech diversity sexism feminism engineering work workplaces feedback)
-
‘Heavily tinted blue paintings form space stations, spacesuits, and rockets just after blast. Michael Kagan paints these large-scale works to celebrate the man-made object—machinery that both protects and holds the possibility of instantly killing those that operate the equipment from the inside. To paint the large works, Kagan utilizes an impasto technique with thick strokes that are deliberate and unique, showing an aggression in his application of oil paint on linen. The New York-based artist focuses on iconic images in his practice, switching back and forth between abstract and representational styles. “The painting is finished when it can fall apart and come back together depending on how it is read and the closeness to the work,” said Kagan about his work. “Each painting is an image, a snapshot, a flash moment, a quick read that is locked into memory by the iconic silhouettes.”’ Via http://www.thisiscolossal.com/2015/08/michael-kagens-space-paintings/
(tags: paintings prints art michael-kagan space abstract-art tobuy)
-
I’m assuming, if you are on the Internet and reading kind of a nerdy blog, that you know what Unicode is. At the very least, you have a very general understanding of it — maybe “it’s what gives us emoji”. That’s about as far as most people’s understanding extends, in my experience, even among programmers. And that’s a tragedy, because Unicode has a lot of… ah, depth to it. Not to say that Unicode is a terrible disaster — more that human language is a terrible disaster, and anything with the lofty goals of representing all of it is going to have some wrinkles. So here is a collection of curiosities I’ve encountered in dealing with Unicode that you generally only find out about through experience. Enjoy.
(tags: unicode characters encoding emoji utf-8 utf-16 utf mysql text)
httpbin(1): HTTP Client Testing Service
Testing an HTTP Library can become difficult sometimes. RequestBin is fantastic for testing POST requests, but doesn’t let you control the response. This exists to cover all kinds of HTTP scenarios. Additional endpoints are being considered.
-
amazing slideshow/WebGL demo talking about graphics programming, its maths, and GPUs
(tags: maths graphics webgl demos coding algorithms slides tflops gpus)
‘I wish to register a complaint’: know your consumer rights before the fight
Conor Pope on the basics of consumer law — and how to complain — in Ireland
(tags: consumer ireland irish-times articles law)
-
an object pooling library for Java. Use it to recycle objects that are expensive to create. The library will take care of creating and destroying your objects in the background. Stormpot is very mature, is used in production, and has done over a trillion claim-release cycles in testing. It is faster and scales better than any competing pool.
Apache-licensed, and extremely fast: https://medium.com/@chrisvest/released-stormpot-2-4-eeab4aec86d0(tags: java stormpot object-pooling object-pools pools allocation gc open-source apache performance)
Evolution of Babbel’s data pipeline on AWS: from SQS to Kinesis
Good “here’s how we found it” blog post:
Our new data pipeline with Kinesis in place allows us to plug new consumers without causing any damage to the current system, so it’s possible to rewrite all Queue Workers one by one and replace them with Kinesis Workers. In general, the transition to Kinesis was smooth and there were not so tricky parts. Another outcome was significantly reduced costs – handling almost the same amount of data as SQS, Kinesis appeared to be many times cheaper than SQS.
(tags: aws kinesis kafka streaming data-pipelines streams sqs queues architecture kcl)
You’re probably wrong about caching
Excellent cut-out-and-keep guide to why you should add a caching layer. I’ve been following this practice for the past few years, after I realised that #6 (recovering from a failed cache is hard) is a killer — I’ve seen a few large-scale outages where a production system had gained enough scale that it required a cache to operate, and once that cache was damaged, bringing the system back online required a painful rewarming protocol. Better to design for the non-cached case if possible.
(tags: architecture caching coding design caches ops production scalability)
The Alternative Universe Of Soviet Arcade Games
Unlike machines in the West, every single machine that was produced during Soviet-era Russia had to align with Marxist ideology. […] The most popular games were created to teach hand-eye coordination, reaction speed, and logical, focused thinking. Not unlike many American games, these games were influenced by military training, crafted to teach and instill patriotism for the state by making the human body better, stronger, and more willful. It also means no high scores, no adrenaline rushes, or self-serving feather-fluffing as you add your hard-earned initials to the list of the best. In Communist Russia, there was no overt competition.
(tags: high-scores communism russia cccp ussr arcade-games games history)
Large Java HashMap performance overview
Large HashMap overview: JDK, FastUtil, Goldman Sachs, HPPC, Koloboke, Trove – January 2015 version
(tags: java performance hashmap hashmaps optimization fastutil hppc jdk koloboke trove data-structures)
-
Is it too late to replace Eircode?
Addresses are hard. Who can remember street addresses or latitude/longitude pairs? You could do much better with three totally random English words, but then there’s that pesky language barrier. No system is perfect, except for emoji.
(tags: eircode maps parody via:nelson location geocoding mapping pile-of-poo)
Real Time Analytics With Spark Streaming and Cassandra
…and Kafka
(tags: spark-streaming kafka analytics cassandra architecture data batch)
Improvements to Kafka integration of Spark Streaming
looks decent as an approach
(tags: kafka spark spark-streaming data)
Diffy: Testing services without writing tests
Play requests against 2 versions of a service. A fair bit more complex than simply replaying logged requests, which took 10 lines of a shell script last time I did it
(tags: http testing thrift automation twitter diffy diff soa tests)
Gmail supports animated emoji in e-mail subjects
Currently only used in spam, naturally. (via Hilary Mason)
-
The Algorithmist is a resource dedicated to anything algorithms – from the practical realm, to the theoretical realm. There are also links and explanation to problemsets.
A wiki for algorithms. Not sure if this is likely to improve on Wikipedia, which of course covers the same subject matter quite well, though(tags: algorithms reference wikis coding data-structures)
-
analyzes Spot price history to help you determine a bid price that suits your needs.
(tags: ec2 aws spot spot-instances history)
What Are the Worst Airports in the World?
this is a great resource when picking a stopover for a 2-stop flight. Pity “best kids play area” isn’t a criterion
(tags: airports comparison via:boingboing flying travel ranking world skytrax)
Using Samsung’s Internet-Enabled Refrigerator for Man-in-the-Middle Attacks
Whilst the fridge implements SSL, it FAILS to validate SSL certificates, thereby enabling man-in-the-middle attacks against most connections. This includes those made to Google’s servers to download Gmail calendar information for the on-screen display. So, MITM the victim’s fridge from next door, or on the road outside and you can potentially steal their Google credentials.
The Internet of Insecure Things strikes again.(tags: iot security fridges samsung fail mitm ssl tls google papers defcon)
Malware infecting jailbroken iPhones stole 225,000 Apple account logins | Ars Technica
KeyRaider, as the malware family has been dubbed, is distributed through a third-party repository of Cydia, which markets itself as an alternative to Apple’s official App Store. Malicious code surreptitiously included with Cydia apps is creating problems for people in China and at least 17 other countries, including France, Russia, Japan, and the UK. Not only has it pilfered account data for 225,941 Apple accounts, it has also disabled some infected phones until users pay a ransom, and it has made unauthorized charges against some victims’ accounts.
Ouch. Not a good sign for Cydia(tags: cydia apple security exploits jailbreaking ios iphone malware keyraider china)
-
‘a simple command line tool that turns your CLI tools into web applications’
-
a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X. S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival. S3QL is designed to favor simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection and error handling have been included from the very first line, and S3QL comes with extensive automated test cases for all its components.
(tags: filesystems aws s3 storage unix google-storage openstack)
3 Lessons From The Amazon Takedown – Fortune
They are: The leaders we admire aren’t always that admirable; Economic performance and costs trump employee well-being; and people participate in and rationalize their own subjugation. ‘In the end, “Amazonians” are not that different from other people in their psychological dynamics. Their company is just a more extreme case of what many other organizations regularly do. And most importantly, let’s locate the problem, if there is one, and its solution where it most appropriately belongs—not with a CEO who is greatly admired (and wealthy beyond measure) running a highly admired company, but with a society where money trumps human well-being and where any price, maybe even lives, is paid for status and success.’ (via Lean)
(tags: amazon work work-life-balance life us fortune via:ldoody ceos employment happiness)
What does it take to make Google work at scale? [slides]
50-slide summary of Google’s stack, compared vs Facebook, Yahoo!, and open-source-land, with the odd interesting architectural insight
(tags: google architecture slides scalability bigtable spanner facebook gfs storage)
Scaling Analytics at Amplitude
Good blog post on Amplitude’s lambda architecture setup, based on S3 and a custom “real-time set database” they wrote themselves. antirez’ comment from a Redis angle on the set database: http://antirez.com/news/92 HN thread: https://news.ycombinator.com/item?id=10118413
(tags: lambda-architecture analytics via:hn redis set-storage storage databases architecture s3 realtime)
-
toxy is a fully programmatic and hackable HTTP proxy to simulate server failure scenarios and unexpected network conditions. It was mainly designed for fuzzing/evil testing purposes, when toxy becomes particularly useful to cover fault tolerance and resiliency capabilities of a system, especially in service-oriented architectures, where toxy may act as intermediate proxy among services. toxy allows you to plug in poisons, optionally filtered by rules, which essentially can intercept and alter the HTTP flow as you need, performing multiple evil actions in the middle of that process, such as limiting the bandwidth, delaying TCP packets, injecting network jitter latency or replying with a custom error or status code.
(tags: toxy proxies proxy http mitm node.js soa network failures latency slowdown jitter bandwidth tcp)
Drone Oversight Is Coming to Construction Sites
Grim Meathook Future
(tags: grim-meathook-future drones work panopticon future sacramento building-sites)
-
Open source security team has had enough of embedded-systems vendors taking the piss with licensing:
This announcement is our public statement that we’ve had enough. Companies in the embedded industry not playing by the same rules as every other company using our software violates users’ rights, misleads users and developers, and harms our ability to continue our work. Though I’ve only gone into depth in this announcement on the latest trademark violation against us, our experience with two GPL violations over the previous year have caused an incredible amount of frustration. These concerns are echoed by the complaints of many others about the treatment of the GPL by the embedded Linux industry in particular over many years. With that in mind, today’s announcement is concerned with the future availability of our stable series of patches. We decided that it is unfair to our sponsors that the above mentioned unlawful players can get away with their activity. Therefore, two weeks from now, we will cease the public dissemination of the stable series and will make it available to sponsors only. The test series, unfit in our view for production use, will however continue to be available to the public to avoid impact to the Gentoo Hardened and Arch Linux communities. If this does not resolve the issue, despite strong indications that it will have a large impact, we may need to resort to a policy similar to Red Hat’s, described here or eventually stop the stable series entirely as it will be an unsustainable development model.
(tags: culture gpl linux opensource security grsecurity via:nelson gentoo arch-linux gnu)
London Calling: Two-Factor Authentication Phishing From Iran
some rather rudimentary anti-2FA attempts, presumably from Iranian security services
(tags: authentication phishing security iran activism 2fa mfa)
Vegemite May Power The Electronics Of The Future
Professor Marc in het Panhuis at the ARC Centre of Excellence for Electromaterials Science figured out that you can 3D print the paste and use it to carry current, effectively creating Vegemite bio-wires. What does this mean? Soon you can run electricity through your food. “The iconic Australian Vegemite is ideal for 3D printing edible electronics,” said the professor. “It contains water so it’s not a solid and can easily be extruded using a 3D printer. Also, it’s salty, so it conducts electricity.”
I’m sure the same applies for Marmite…(tags: vegemite marmite 3d-printing electronics bread food silly)
Beoir.org Community – Recent Attack on McGargles
bizarre conspiracy theory going around about McGargles microbrewery being owned by Molson in an “astroturf craft beer” operation — they apparently were set up by a bunch of ex-Molson employees. Their beer is getting stickered in off-licenses. Mental!
(tags: beer craft-beer ireland mcgargles conspiracy-theories bizarre beoir)
Mining High-Speed Data Streams: The Hoeffding Tree Algorithm
This paper proposes a decision tree learner for data streams, the Hoeffding Tree algorithm, which comes with the guarantee that the learned decision tree is asymptotically nearly identical to that of a non-incremental learner using infinitely many examples. This work constitutes a significant step in developing methodology suitable for modern ‘big data’ challenges and has initiated a lot of follow-up research. The Hoeffding Tree algorithm has been covered in various textbooks and is available in several public domain tools, including the WEKA Data Mining platform.
(tags: hoeffding-tree algorithms data-structures streaming streams cep decision-trees ml learning papers)
Chinese scammers are now using Stingray tech to SMS-phish
A Stingray-style false GSM base station, hidden in a backpack; presumably they detect numbers in the vicinity, and SMS-spam those numbers with phishing messages. Reportedly the scammers used this trick in “Guangzhou, Zhuhai, Shenzhen, Changsha, Wuhan, Zhengzhou and other densely populated cities”. Dodgy machine translation:
March 26, Zhengzhou police telecommunications fraud cases together, for the first time seized a small backpack can hide pseudo station equipment, and arrested two suspects. Yesterday, the police informed of this case, to remind the general public to pay attention to prevention. “I am the landlord, I changed number, please rent my wife hit the bank card, card number ×××, username ××.” Recently, Jiefang Road, Zhengzhou City Public Security Bureau police station received a number of cases for investigation brigade area of ??the masses police said, frequently received similar phone scam messages. Alarm, the police investigators to determine: the suspect may be in the vicinity of twenty-seven square, large-scale use of mobile pseudo-base release fraudulent information. […] Yesterday afternoon, the Jiefang Road police station, the reporter saw the portable pseudo-base is made up of two batteries, a set-top box the size of the antenna box and a chassis, as well as a pocket computer composed together at most 5 kg.
(via t byfield and Danny O’Brien)(tags: via:mala via:tbyfield privacy scams phishing sms gsm stingray base-stations mobile china)
In search of performance – how we shaved 200ms off every POST request — GoCardless Blog
tl;dr: don’t use Ruby’s Net::HTTP and/or HAProxy prior to 1.4.19
(tags: http ruby tcp nagle performance rtt networking haproxy ack curl)
Non-Celiac Gluten Sensitivity May Not Exist
The data clearly indicated that a nocebo effect, the same reaction that prompts some people to get sick from wind turbines and wireless internet, was at work here. Patients reported gastrointestinal distress without any apparent physical cause. Gluten wasn’t the culprit; the cause was likely psychological. Participants expected the diets to make them sick, and so they did.
(tags: gluten placebo nocebo food science health diet gluten-free fodmaps)
-
Some nice real-world experimentation around large-scale data processing in differential dataflow:
If you wanted to do an iterative graph computation like PageRank, it would literally be faster to sort the edges from scratch each and every iteration, than to use unsorted edges. If you want to do graph computation, please sort your edges. Actually, you know what: if you want to do any big data computation, please sort your records. Stop talking sass about how Hadoop sorts things it doesn’t need to, read some papers, run some tests, and then sort your damned data. Or at least run faster than me when I sort your data for you.
(tags: algorithms graphs coding data-processing big-data differential-dataflow radix-sort sorting x-stream counting-sort pagerank)
Docker image creation, tagging and traceability in Shippable
this is starting to look quite impressive as a well-integrated Docker-meets-CI model; Shippable is basing its builds off Docker baselines and is automatically cutting Docker images of the post-CI stage. Must take another look
Analysis of PS4’s security and the state of hacking
FreeBSD jails and Return-Oriented Programming:
Think of [Return-Oriented Programming] as writing a new chapter to a book, using only words that have appeared at the end of sentences in the previous chapters.
(tags: ps4 freebsd jails security exploits hacking sony rop return-oriented-programming)
10 Lesser-Known Cocktails You Should Be Drinking
like the sound of some of these
My wife found my email in the Ashley Madison database
On misdirected emails and the potential side-effects:
The reasons why these people give out my email instead of one that they can access have always been a bit mysterious to me. It’s one thing to save yourself some spam by using a throwaway address. But why use someone else’s for correspondence you actually want to receive? The closest I’ve come to a working theory is that a lot of them, having been slow off the mark to obtain their own gmail, have addresses like eratliff75@gmail.com. Either they believe they can leave off the numbers and receive the messages anyway, or they often simply forget. That or the E. Ratliffs of the world just view eratliff@gmail.com as some kind of shared resource.
(tags: email mail ashley-madison gmail mistakes misdirected-email)
How to Make Raspberry-Thyme Shrub
looks tasty/non-tricky
How gaming terminology is part of modern mainstream Chinese slang
A few years ago, my mom called to ask for my advice on webcams. She explained (in the English-peppered Chinese that’s the official language of our Chinese-American household) that some of her friends had started sharing videos of themselves singing karaoke. She thought she could do better. “?????PK??,” she remarked: “I want to PK them a little.”
-
Powerline networking is vulnerable to sniffing and brute-force attacks. See also http://www.nosuchcon.org/talks/2014/D1_03_Sebastien_Dudek_HomePlugAV_PLC.pdf
(tags: powerline-networking power networking han home exploits security qualcomm homeplug plcs)
buildfarm_deployment/cleanup_docker_images.py
Cleanup old/obsolete Docker images in a repo.
(tags: disk-space ops docker cleanup cron)
-
Chronos (the Mesos distributed scheduler) comes out looking pretty crappy here
(tags: aphyr mesos chronos cron scheduling outages ops jepsen testing partitions cap)
Kubernetes and AWS VPC Peering – Ben Straub
the perils of overloading 10/8
(tags: 10/8 ec2 aws vpc kubernetes ops internet ip-addresses)
How your entire financial life will be stored in a new ‘digital vault’ – Telegraph
In a move to make it easier to open bank accounts and Isas, people will be asked to share all of their accounts, tax records and personal details with a central service. To check someone’s identity, a company would then ask potential customers a series of questions and check the answers against the information in the vault. The checks would replace the current system in which new customers must send by post copies of their passports, cross-signed by a friend, along with bank statements and utility bills.
hahahaha NO FUCKING WAY.(tags: bills banking uk tax privacy digital-vault accounts authentication identity-theft bad-ideas)
-
_FBGraphQLConnectionStorePersistentPageLoaderOperationDelegate-Protocol.h _FBReactionAcornSportsContentSettingsSetShouldNotPushNotificationsMutationCall.h FBBoostedComponentCreateInputDataCreativeObjectStorySpecLinkDataCallToActionValue.h FBEventUpdateNotificationSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h
I just threw up a little. See also https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920 , in which the FB Android devs happily reveal that they hot-patch the Dalvik VM at runtime to work around a limit — rather than refactoring their app.(tags: facebook horrors coding ios android dalvik hot-patching apps)
-
I can’t believe this is the state of food blogging in the UK and Ireland. full-on payola for reviews. See also @damienmulley’s excellent rant on the subject in this country: https://twitter.com/damienmulley/status/633353368757497858 — there’s even rate cards for positive review tweets/posts/facebook updates etc.
(tags: food blogging restaurants uk bakeries reviews payola blogger-blackmail pr)
The reusable holdout: Preserving validity in adaptive data analysis
Useful stats hack from Google: “We show how to safely reuse a holdout data set many times to validate the results of adaptively chosen analyses.”
(tags: statistics google reusable-holdout training ml machine-learning data-analysis holdout corpus sampling)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Extremely authoritative slide deck on building a recommendation system, from Xavier Amatriain, Research/Engineering Manager at Netflix
(tags: netflix recommendations recommenders ml machine-learning cmu clustering algorithms)
-
our full-featured, high performance, scalable web server designed to compete with the likes of nginx. It has been built from the ground-up with no external library dependencies entirely in x86_64 assembly language, and is the result of many years’ experience with high volume web environments. In addition to all of the common things you’d expect a modern web server to do, we also include assembly language function hooks ready-made to facilitate Rapid Web Application Server (in Assembler) development.
(tags: assembly http performance https ssl x86_64 web ops rwasa tls)
The world beyond batch: Streaming 101 – O’Reilly Media
To summarize, in this post I’ve: Clarified terminology, specifically narrowing the definition of “streaming” to apply to execution engines only, while using more descriptive terms like unbounded data and approximate/speculative results for distinct concepts often categorized under the “streaming” umbrella. Assessed the relative capabilities of well-designed batch and streaming systems, positing that streaming is in fact a strict superset of batch, and that notions like the Lambda Architecture, which are predicated on streaming being inferior to batch, are destined for retirement as streaming systems mature. Proposed two high-level concepts necessary for streaming systems to both catch up to and ultimately surpass batch, those being correctness and tools for reasoning about time, respectively. Established the important differences between event time and processing time, characterized the difficulties those differences impose when analyzing data in the context of when they occurred, and proposed a shift in approach away from notions of completeness and toward simply adapting to changes in data over time. Looked at the major data processing approaches in common use today for bounded and unbounded data, via both batch and streaming engines, roughly categorizing the unbounded approaches into: time-agnostic, approximation, windowing by processing time, and windowing by event time.
(tags: streaming batch big-data lambda-architecture dataflow event-processing cep millwheel data data-processing)
What the hell is going on with SoundCloud?
tl;dr: major labels.
Despite having revenue coming in from ads and subscriptions, SoundCloud still relies on outside investment. While the company received $150 million in a funding round at the end of last year, it pales next to the reported $526 million Spotify gained in June, and if one report is to be believed, SoundCloud is running very low on cash. Furthermore, sources suggest that potential investors are waiting to see what happens with Sony and Universal before ploughing in more money. With the high sums reported to be involved, it’s a stalemate that could potentially break the company whether it decides to pay or not.
(tags: soundcloud music mp3 copyright sony universal spotify funding startups)
GSMem: Data Exfiltration from Air-Gapped Computers over GSM Frequencies
Holy shit.
Air-gapped networks are isolated, separated both logically and physically from public networks. Although the feasibility of invading such systems has been demonstrated in recent years, exfiltration of data from air-gapped networks is still a challenging task. In this paper we present GSMem, a malware that can exfiltrate data through an air-gap over cellular frequencies. Rogue software on an infected target computer modulates and transmits electromagnetic signals at cellular frequencies by invoking specific memory-related instructions and utilizing the multichannel memory architecture to amplify the transmission. Furthermore, we show that the transmitted signals can be received and demodulated by a rootkit placed in the baseband firmware of a nearby cellular phone.
(tags: gsmem gsm exfiltration air-gaps memory radio mobile-phones security papers)
An Amazonian’s response to “Inside Amazon: Wrestling Big Ideas in a Bruising Workplace” — Medium
excellent response to the NYT hatchet job
-
This is great. Featuring Mount Buggery:
There were no tracks of any sort until they reached Mt Howitt and Stewart, perhaps not quite as fit as he could have been, was finding the going tough after the descent from Mt Speculation. Faced with the prospect of yet another laborious climb he exploded with the words ‘What another bugger! I’ll call this mountain Mt Buggery.’
and Mount Arsehole:“We always called it Mt Arsehole… Then they came along with all their fancy bloody maps and ideas. Changed it to Mt Arthur. Christ knows why. Bastard of a place anyway!”
(tags: swearing australia mount-buggery mount-arsehole nsw victoria places history names mountains)
minimaxir/big-list-of-naughty-strings
Late to this one — a nice list of bad input (Unicode zero-width spaces, etc) for testing
(tags: testing strings text data unicode utf-8 tests input corrupt)
Preventing Dependency Chain Attacks in Maven
using a whitelist of allowed dependency JARs and their SHAs
(tags: security whitelisting dependencies coding jar maven java jvm)
-
This is excellent — I wish more companies took this attitude. Applause for Travis CI.
after a couple of weeks of research, we made a decision to offer our expectant mothers AND fathers: 2 weeks before the due date paid at 100% (optional, but recommended); 20 weeks for normal births paid at 100%; 24 weeks for births with complications paid at 100%; Flexible working hours after the 20/24 weeks are complete (part-time arrangements can be made); Your job will be here for you when you return. When we relayed this information to the two US employees, one became a little teary because her last employer (a much bigger and older company), didn’t offer anything. This being her second child, it was a huge relief to know she was going to have paid time off with flexibility upon return. While it was a great reaction, it shouldn’t happen this way. If you value your employees, you should value their need for time away. At the same time, if you want to hire someone, whether or not they are already pregnant should be irrelevant.
Well exceeding even the Irish maternity leave entitlements, since it covers fathers too. And this is a startup!(tags: travisci startups work life family kids paternity-leave maternity-leave)
Improving The Weather On Twitter
lovely open-source dataviz improvement for near-term historical rainfall-radar images
(tags: dataviz weather rain rainfall radar nws twitter bots graphics ui)
Somewhere Over the Rainbow: How to Make Effective Use of Colors in Meteorological Visualizations
Linked from the “Improving the Weather On Twitter” post — choosing the “best” colour scheme for meteorological visualization. Great dataviz resource post
(tags: dataviz colour color meteorological weather nws papers rgb hcl)
Reddit comments from a nuclear-power expert
Reddit user “Hiddencamper” is a senior nuclear reactor operator in the US, and regularly posts very knowledgeable comments about reactor operations, safety procedures, and other details. It’s fascinating (via Maciej)
(tags: via:maciej nuclear-power nuclear atomic power energy safety procedures operations history chernobyl scram)
Amazon EC2 2015 Benchmark: Testing Speeds Between AWS EC2 and S3 Regions
Here we are again, a year later, and still no bloody percentiles! Just amateurish averaging. This is not how you measure anything, ffs. Still, better than nothing I suppose
(tags: fail latency measurement aws ec2 percentiles s3)
background doc on the Jeep hack
“Remote Exploitation of an Unaltered Passenger Vehicle”, by Dr. Charlie Miller (cmiller@openrce.org) and Chris Valasek (cvalasek@gmail.com). QNX, unauthenticated D-Bus, etc.
‘Since a vehicle can scan for other vulnerable vehicles and the exploit doesn’t require any user interaction, it would be possible to write a worm. This worm would scan for vulnerable vehicles, exploit them with their payload which would scan for other vulnerable vehicles, etc. This is really interesting and scary. Please don’t do this. Please.’
Care.data and access to UK health records: patient privacy and public trust
‘In 2013, the United Kingdom launched care.data, an NHS England initiative to combine patient records, stored in the machines of general practitioners (GPs), with information from social services and hospitals to make one centralized data archive. One aim of the initiative is to gain a picture of the care being delivered between different parts of the healthcare system and thus identify what is working in health care delivery, and what areas need greater attention and resources. This case study analyzes the complications around the launch of care.data. It explains the historical context of the program and the controversies that emerged in the course of the rollout. It explores problems in management and communications around the centralization effort, competing views on the safety of “anonymous” and “pseudonymous” health data, and the conflicting legal duties imposed on GPs with the introduction of the 2012 Health and Social Care Act. This paper also explores the power struggles in the battle over care.data and outlines the tensions among various stakeholders, including patients, GPs, the Health and Social Care Information Centre (HSCIC), the government, privacy experts and data purchasers. The predominant public policy question that emerges from this review centers on how best to utilize technological advances and simultaneously strike a balance between the many competing interests around health and personal privacy.’
(tags: care.data privacy healthcare uk nhs trust anonymity anonymization gps medicine)
-
coining a term for the awful buyer’s experience on sites like car-hire or air-travel websites
(tags: hate-selling conversion marking upselling travel web consumer)
How Irish Navy’s expertise saved 367 from 30-second sinking in Mediterranean
War-game exercises saved the day:
As the Ribs made their assessment of the situation and began reassuring those on board that help was at hand, the hopelessly overloaded vessel suddenly listed and sank. The sinking took just over 30 seconds. In those 30 seconds, the Captain of the LE Niamh took a number of instant command decisions that saved hundreds of lives. Most of the refugees cannot swim. Their life expectancy in the water would be measured in seconds. The crew of the Ribs immediately began throwing orange lifejackets into the water – encouraging the now frenzied and milling survivors to cling to them. Individuals, then groups clung to the lifejackets – and one another – as the Ribs rallied around trying to keep the floating human mass from dispersal into wider waters and almost certain death. In the meantime, the commander of the LE Niamh managed to manoeuvre close in to the survivors where spare life-rafts were launched into the water. These 25-man inflatable life-rafts were specifically ordered and kept on board the LE Niamh following a “war-gaming” exercise, where the officers and crew envisaged such a nightmare scenario. Had this forward planning not taken place – there would have been no such extra inflatable lifeboats on board.
(tags: war-gaming planning navy ireland mediterranean sea boats refugees migration drowning liferafts)
-
A well-maintained list with a potted description of each one (via HN)
(tags: postmortems ops uptime reliability)
Advantages of Monolithic Version Control
another Dan Luu post — good summary of the monorepo’s upside
(tags: monorepo git mercurial versioning source-control coding dependencies)
“A Review Of Criticality Accidents, 2000 Revision”
Authoritative report from LANL on accidents involving runaway nuclear reactions over the years from 1945 to 1999, around the world. Illuminating example of how incident post-mortems are handled in other industries, and (of course) fascinating in its own right
(tags: criticality nuclear safety atomic lanl post-mortems postmortems fission)
-
Netflix’ official test video — contains various scenarios which exercise frequent tricky edge cases in video compression and playback; A/V sync, shades of black, running water, etc.
(tags: networking netflix streaming video compression tests)
How to get your water tested for lead in Dublin
Ossian has written up this very informative post:
Irish Water is writing to thousands of people living in Dublin this week to warn them that their water is supplied through lead pipes. Irish Water says that most people receiving these letters have a level of lead in their water which is above safe limits. So, if you get one of these letters how do you get your water tested? Irish Water is refusing to supply test kits or to test everyone’s water who asks. However the HSE’s Public Analyst Lab has told me that they will test water for lead for a fee of €10.
(tags: ossian-smyth dun-laoghaire dublin drinking-water water lead green hse irish-water health)
-
Good advice on production-quality, decent-scale usage of Kinesis in Java with the official library: batching, retries, partial failures, backoff, and monitoring. (Also, jaysus, the AWS Cloudwatch API is awful, looking at this!)
(tags: kpl aws kinesis tips java batching streaming production cloudwatch monitoring coding)
IrishCycle.com on the Irish Times’ terrible victim-blaming anti-cycling op-ed
Even if The Irish Times wants to deny that it has engaged in victim blaming at a high level, it has also clearly errored in fact in a very significant way. It would be more forgiving if this was an isolated editorial. But it’s after two days of wrong or misleading coverage, which now seems to be a trend with the newspaper with unbalanced articles or headlines negatively focusing on cycle routes.
(tags: irish-times newspapers op-eds cycling dublin ireland safety)
-
plenty of stuff out of bounds in Dublin tomoz
India lifts porn ban after widespread outrage – BBC News
After a brief couple of days.
News of the ban caused a furore on Indian social media, with several senior politicians and members of civil society expressing their opposition to the move. The Indian government said that it was merely complying with the Supreme Court order and was committed to the freedom of communication on the Internet. “I reject with contempt the charge that it is a Talibani government, as being said by some of the critics. Our government supports free media, respects communication on social media and has respected freedom of communication always,” Mr Prasad told PTI.
(tags: india porn filtering isps internet web child-porn censorship)
17 of the most important things to ever happen to Irish Twitter
definitive. The David O’Doherty / Not The RTE Guide “your ma” battle is legendary (http://thedailyedge.thejournal.ie/your-ma-david-odoherty-1290482-Jan2014/)
(tags: ireland twitter funny social-media)
Amazon S3 Introduces New Usability Enhancements
bucket limit increase, and read-after-write consistency in US Standard. About time too! ;)
(tags: aws s3 storage consistency)
New study shows Spain’s “Google tax” has been a disaster for publishers
A study commissioned by Spanish publishers has found that a new intellectual property law passed in Spain last year, which charges news aggregators like Google for showing snippets and linking to news stories, has done substantial damage to the Spanish news industry. In the short-term, the study found, the law will cost publishers €10 million, or about $10.9 million, which would fall disproportionately on smaller publishers. Consumers would experience a smaller variety of content, and the law “impedes the ability of innovation to enter the market.” The study concludes that there’s no “theoretical or empirical justification” for the fee.
(tags: google news publishing google-tax spain law aggregation snippets economics)
Inside the sad, expensive failure of Google+
“It was clear if you looked at the per user metrics, people weren’t posting, weren’t returning and weren’t really engaging with the product,” says one former employee. “Six months in, there started to be a feeling that this isn’t really working.” Some lay the blame on the top-down structure of the Google+ department and a leadership team that viewed success as the only option for the social network. Failures and disappointing data were not widely discussed. “The belief was that we were always just one weird feature away from the thing taking off,” says the same employee.
(tags: google google+ failures post-mortems business facebook social-media fail bureaucracy vic-gundotra)
8,000 sq ft start-up meeting space revealed for Dublin
Neat. this is a good location for post-work user-group meetups and the like (via Oisin)
(tags: via:oisin meetups meetings ulster-bank dublin startups chq)