Why Uber Engineering Switched from Postgres to MySQL
Uber bringing the smackdown for the HN postgres fanclub, with some juicy technical details of issues that caused them pain. FWIW, I was bitten by crappy postgres behaviour in the past (specifically around vacuuming and pgbouncer), so I’ve long been a MySQL fan ;)
(tags: database mysql postgres postgresql uber architecture storage sql)
Justin's Linklog Posts
Noirmoutier Indigo Campsite, France
As recommended by J & F: ‘Most of the campsites we’ve stayed in have had great facilities for kids – pools, activities, entertainment etc – but the problem with that is you spend your day being dragged from one to the other. There’s none of that at Camping Indigo in Noirmoutier apart from a playground, some kayaks and some music in the bar at night but it is on the beach so the kids either run wild around the campsite or play on the beach – it was the best and most relaxing holiday we ever had and we definitely met the coolest people there. There’s a really nice town in the centre of the island and great beaches all around it so hire bikes and roam free.’ Bookmarking for next year’s holiday planning!
(tags: holidays fun france camping noirmoutier chaize-wood loire nantes recommendations)
-
his Monitorama 2016 talk, talking about the “deep health checks” concept (which I implemented at Swrve earlier this year ;)
(tags: monitorama health deep-health-checks healthz testing availability reliability)
-
I never knew we had a native take on the sauna, the “teach alluis”:
Sweathouses were used for the treatment for a wide range of ailments up to the late 19th and early 20th centuries, primarily rheumatism but also including sciatica, lameness, sore eyes, gout, skin disorders, psychiatric disorders, impotence and infertility. Surviving records indicate that treatment was often a group activity for 4-8 persons. The sweathouse was heated by filling the interior with fuel (turf, heather, wood etc. as available), and firing the structure for a period of up to two days to heat the stone structure, the hot ashes were then raked out and the interior floor lined with bracken, grass or straw. The bathers entered and blocked the entrance with turves, clothes or some other means. The sweating period could last a number of hours while the structure retained heat. Some authors note that water was thrown on hot stones to create steam. Afterwards, the “patients” would either take a cold plunge in the nearby water source, or go home and rest for a few hours, or simply return to their normal daily activities.
(via Aileen)(tags: via:aileen sweating sweat-houses irish history saunas heat)
-
excellent recipe for this classic cocktail
(tags: cocktails recipes old-fashioned booze)
-
meditations on this classic cocktail (with solid recipes)
-
Course notes from Gerald Jay Sussman’s “Adventures in Advanced Symbolic Programming” class at MIT. Hard to argue with this:
The syntax of the regular-expression language is awful. There are various incompatable forms of the language and the quotation conventions are baroquen [sic]. Nevertheless, there is a great deal of useful software, for example grep, that uses regular expressions to specify the desired behavior. Although regular-expression systems are derived from a perfectly good mathematical formalism, the particular choices made by implementers to expand the formalism into useful software systems are often disastrous: the quotation conventions adopted are highly irregular; the egregious misuse of parentheses, both for grouping and for backward reference, is a miracle to behold. In addition, attempts to increase the expressive power and address shortcomings of earlier designs have led to a proliferation of incompatible derivative languages.
(via Rob Pike’s twitter: https://twitter.com/rob_pike/status/755856685923639296)(tags: regex regexps regular-expressions functional combinators gjs rob-pike coding languages)
A Cute Internet Star Flirts. All He Wants Is Your Password. – The New York Times
whoa.
Mr. Johnson’s fans are not naïve. Handing over their passwords to some strange, cute boy actually constitutes a minor act of youthful rebellion. The whole encounter delivers a heady mix of intimacy and transgression — the closest digital simulation yet to a teenage crush.
(via Adam Shostack)(tags: via:adam-shostack passwords authentication security teens rebellion)
Just As We Warned: A Chinese Tech Giant Goes On The Patent Attack — In East Texas | Techdirt
Techdirt has been warning for years that the West’s repeated demands for China to “respect” patents could backfire badly. […] And guess what? That is exactly what has just happened, as The Wall Street Journal reports: ‘Huawei Technologies Co. said it has filed a lawsuit against T-Mobile US Inc., alleging the U.S. telecommunications carrier violated the Chinese company’s patents related to wireless networks. In its complaint filed this week in the U.S. District Court for the Eastern District of Texas, Huawei said T-Mobile is using its patented technology without signing a licensing agreement.’
At least this is the most likely scenario to result in patent reform, finally.A New Wrinkle in the Gig Economy: Workers Get Most of the Money – The New York Times
So using money from the sale of iStock to Getty, she and Mr. Livingstone set out to create Stocksy, paying photographers 50 to 75 percent of sales. That is well above the going rate of 15 to 45 percent that is typical in the stock photography field. The company also distributes 90 percent of its profit at the end of each year among its photographers. Stocksy is part of a new wave of start-ups that are borrowing the tools of Silicon Valley to create a more genuine “sharing” economy that rewards the individuals generating the value.
(tags: stocksy stock-photos photos fair sharing photography work)
-
(a) 83.5% uptime over 24 hours. GOOD JOB (b) excellent marketing by Datadog!
(tags: datadog games monitoring pokemon-go pokemon uptime)
-
eye-poppingly bizarre half-assed safety features of the 1950s — a megaton nuclear weapon rendered safe from accidental criticality accidents only by a plastic bag full of ball bearings
(tags: nuclear-weapons nukes safety 1950s uk funny bizarre violet-club ball-bearings via:cstross)
Frankly Useless Crank “Knowledge,” Only For Fools
A wonderfully-sweary post on the etymology of swear words, and how they’re not derived from acronyms, really.
shit? Also from an old Germanic root, descended equally to modern German Scheiss (which sounds closer to Scots shite). It shows up in Old English, fully inflected: “Wiþ þon þe men mete untela melte & gecirre on yfele wætan & scittan” (that scittan is an infinitive form of ‘shit’ and was said like “shit-tan”). I can assure you that an acronym Ship High In Transit – supposedly meaning that manure was to be loaded in the upper parts of ships – was not possible in the language in the Old English period, not just because transit was not borrowed from Latin until half a millennium later, or because they didn’t use acronyms like that then, but because what the fuck are you even thinking. They didn’t need to ship manure. Animals produce it on the spot everywhere. Holy shit, fucking seriously.
QA Instability Implies Production Instability
Invariably, when I see a lot of developer effort in production support I also find an unreliable QA environment. It is both unreliable in that it is frequently not available for testing, and unreliable in the sense that the system’s behavior in QA is not a good predictor of its behavior in production.
(tags: qa testing architecture patterns systems production)
-
Doorman is a solution for Global Distributed Client Side Rate Limiting. Clients that talk to a shared resource (such as a database, a gRPC service, a RESTful API, or whatever) can use Doorman to voluntarily limit their use (usually in requests per second) of the resource. Doorman is written in Go and uses gRPC as its communication protocol. For some high-availability features it needs a distributed lock manager. We currently support etcd, but it should be relatively simple to make it use Zookeeper instead.
From google — very interesting to see they’re releasing this as open source, and it doesn’t rely on G-internal services(tags: distributed distcomp locking youtube golang doorman rate-limiting rate-limits limits grpc etcd)
-
‘based my observations while I was a Site Reliability Engineer at Google’, courtesy of Rob Ewaschuk
. Seem pretty reasonable (tags: monitoring sysadmin alerting alerts nagios pager ops sre rob-ewaschuk)
-
‘Best Plex Media Server’ — this looks pretty superb for EUR240 or thereabouts
(tags: media-servers plex video home tv toget nvidia shield android)
-
‘a small library to manage encrypted secrets using asymmetric encryption.’
The main benefits provided by ejson are: Secrets can be safely stored in a git repo. Changes to secrets are auditable on a line-by-line basis with git blame. Anyone with git commit access has access to write new secrets. Decryption access can easily be locked down to production servers only. Secrets change synchronously with application source (as opposed to secrets provisioned by Configuration Management). Simple, well-tested, easily-auditable source.
(tags: crypto security credentials encryption ejson json configuration config)
The mysterious syndrome impairing astronauts’ sight – The Washington Post
Visual impairment intracranial pressure syndrome (VIIP) is named for the leading theory to explain it. On Earth, gravity pulls bodily fluids down toward the feet. That doesn’t happen in space, and it is thought that extra fluid in the skull increases pressure on the brain and the back of the eye.
Designing the Perfect Anti-Object
This pale, amorphous lump of sculpted concrete is designed to resist almost everything in a city that it might come into contact with. Named for the London authority that commissioned it, the Camden Bench has a special coating which makes it impervious to graffiti and vandalism. The squat, featureless surface gives drug dealers nowhere to hide their secret caches. The angled sides repel skateboarders and flyposters, litter and rain. The cambered top throws off rough sleepers. In fact, it is specially crafted to make sure that it is not used as anything except a bench. This makes it a strange artifact, defined far more by what it is not than what it is. The Camden Bench is a concerted effort to create a non-object.
(tags: non-objects objects city camden benches vandalism skating london)
The Apollo 11 AGC source code was uploaded to Github, and someone opened an issue
For the famous Apollo 13 near-fatal failure scenario:
‘A customer has had a fairly serious problem with stirring the cryogenic tanks with a circuit fault present. To reproduce: Build CSM; Perform mission up to translunar coast; During translunar coast, attempt to stir cryo tanks If a wiring fault exists, the issue may be replicated. Be aware that this may be hazardous to the tester attempting it.’ Sample response: ‘Does it happens only with translunar coast (sol-3-a), or any moon coasting? It may be a problem with the moon. Just trying to narrow down the issue.’
(tags: lol funny apollo apollo-11 apollo-13 agc history space github)
Law to allow snooping on social media defies European court ruling
Karlin on fire:
But there’s lots in this legislation that should scare the public far more. For example, the proposal that the legislation should allow the retention of “superfluous data” gathered in the course of an investigation, which is a direct contravention of the ECJ’s demand that surveillance must be targeted and data held must be specifically relevant, not a trawl to be stored for later perusal “just in case”. Or the claim that interception and retention of data, and access to it, will only be in cases of the most serious crime or terrorism threats. Oh, please. This was, and remains, the supposed basis for our existing, ECJ-invalidated legislation. Yet, as last year’s Gsoc investigation into Garda leaks revealed, it turns out a number of interconnected pieces of national legislation allow at least 10 different agencies access to retained data, including Gsoc, the Competition Authority, local authorities and the Irish Medicines Board.
(tags: surveillance ireland whatsapp viber snowden snooping karlin-lillington facebook internet data-retention)
Raintank investing in Graphite
paying Jason Dixon to work on it, improving the backend, possibly replacing the creaky Whisper format. great news!
(tags: graphite metrics monitoring ops open-source grafana raintank)
conventional-changelog-atom 502 Bad Gateway · Issue #13284 · npm/npm
npm down for most of the (EU) day. What a shitshow
(tags: npm fail javascript dependencies coding)
Camille Fournier’s excellent rant on microservices
I haven’t even gotten into the fact that your microservices are an inter-dependent environment, as much as you may wish otherwise, and one service acting up can cause operational problems for the whole team. Maybe if you have Netflix-scale operational hardening that’s not a problem. Do you? Really? Is that the best place to spend your focus and money right now, all so teams can throw shit against the wall to see if it sticks? Don’t sell people fantasies. This is not the reality for a mid-sized tech team working in microservices. There are enough valuable components to building out such a system without the fantastical claims of self-organizing teams who build cool hack projects in 2 week sprints that change the business. Microservices don’t make organizational problems disappear due to self-organization. They allow for some additional degrees of team and process independence and force very explicit decoupling, in exchange, there is overall system complexity and overall system coordination overhead. I personally think that’s enough value, especially when you are coming from a monolith that is failing to scale, but this model is not a panacea.
(tags: microservices rants camille-fournier architecture decoupling dependencies)
-
quotable: “I spend a lot of time on this task. I should write a program automating it!”
-
Quotable: “how long can work on making a routine task more efficient before you’re spending more time than you save?”
(tags: quotes time automation hacks life imdb productivity efficiency)
Why Did Yankee Doodle Call a Feather ‘Macaroni’? | Mental Floss
history!
(tags: usa history macaroni yankee-doodle language dandies 18th-century)
-
John Rauser on this oft-cited dictum of percentile usage in monitoring, and when it’s wrong and it’s actually possible to reason with averaged percentiles, and when it breaks down.
(tags: statistics percentiles quantiles john-rauser histograms averaging mean p99)
MRI software bugs could upend years of research – The Register
In their paper at PNAS, they write: “the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.” For example, a bug that’s been sitting in a package called 3dClustSim for 15 years, fixed in May 2015, produced bad results (3dClustSim is part of the AFNI suite; the others are SPM and FSL). That’s not a gentle nudge that some results might be overstated: it’s more like making a bonfire of thousands of scientific papers. Further: “Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape”. The researchers used published fMRI results, and along the way they swipe the fMRI community for their “lamentable archiving and data-sharing practices” that prevent most of the discipline’s body of work being re-analysed. ®
(tags: fmri science mri statistics cluster-inference autocorrelation data papers medicine false-positives fps neuroimaging)
-
‘a Ruby regular expression editor and tester’. Great for prototyping regexps with a little set of test data, providing a neat permalink for the results
Stick Insect Eggs – Live Bug Kits
going to do this with the kids next!
(tags: stick-insects pets animals insects)
-
by avoiding division
(tags: coding algorithms performance optimization shuffle shuffling)
Self-driving cars: overlooking data privacy is a car crash waiting to happen
Interesting point — self-driving cars are likely to be awash in telemetry data, “phoned home”
(tags: self-driving cars vehicles law data privacy data-privacy surveillance)
Push notifications delayed, Hearbeat Interval not reliable – Google Product Forums
Good thread on GCM notifications and their interactions with NAT — they are delivered over a single TCP connection to port 5228 to the google servers, kept alive, and NAT timeouts can hang the conn resulting in delayed notifications. Particularly useful is the *#*#426#*#* dial code, which displays a log screen on Android devices with GCM debugging info.
USE Method: Linux Performance Checklist
Really late in bookmarking this, but has some up-to-date sample commandlines for sar, mpstat and iostat on linux
(tags: linux sar iostat mpstat cli ops sysadmin performance tuning use metrics)
My kids don’t have a YouTube channel — but they pretend they do
“Dad is making a right turn now,” my 5-year-old son Jack will say as he newscasts the ride to school to a fictional audience. “Don’t forget to subscribe,” his sister Ella, 6, will often interject — again, to no one in particular. When I was their age, I’d pretend to be a soldier or a baseball player. Today, kids apparently aspire to be vloggers. It’s not enough for them to watch their favorite shows. They want to broadcast their lives, banter with commenters and keep their make-believe view counts high.
(tags: youtube kids wtf video broadcasting)
Cops Use Stingray To Almost Track Down Suspected Fast Food Thief
Law enforcement spokespeople will often point to the handful of homicide or kidnapping investigations successfully closed with the assistance of cell site simulators, but they’ll gloss over the hundreds of mundane deployments performed by officers who will use anything that makes their job easier — even if it’s a tool that’s Constitutionally dubious. Don’t forget, when a cell site simulator is deployed, it gathers cell phone info from everyone in the surrounding area, including those whose chicken wings have been lawfully purchased. And all of this data goes… somewhere and is held onto for as long as the agency feels like it, because most agencies don’t seem to have Stingray data retention policies in place until after they’ve been FOIA’ed/questioned by curious legislators. Regular policework — which seemed to function just fine without cell tracking devices — now apparently can’t be done without thousands of dollars of military equipment. And it’s not just about the chicken wing thieves law enforcement can’t locate. It’s about the murder suspects who are caught but who walk away when the surveillance device wipes its feet on the Fourth Amendment as it serves up questionable, post-facto search warrants and pen register orders.
(tags: stingrays mobile surveillance imsi-catchers data-retention privacy chicken-wings fast-food)
A fast alternative to the modulo reduction
(x * N) div 2^32 is an equally fair map reduction, but faster on modern 64-bit CPUs
(tags: fairness modulo arithmetic algorithms fair-mapping reduce daniel-lemire)
There are liars and then there’s Boris Johnson and Michael Gove
Post-brexit post-mortem from Nicholas Cohen in the grauniad:
The Vote Leave campaign followed the tactics of the sleazy columnist to the letter. First, it came out with the big, bold solution: leave. Then it dismissed all who raised well-founded worries with “the country is sick of experts”. Then, like Johnson the journalist, it lied.
(tags: eu politics uk brexit boris-johnson michael-gove)
-
The prime minister evidently thought that the whole debate could be cleanly started and finished in a matter of months. His Eton contemporary Boris Johnson – and, really, can you believe that the political story of the last four months has effectively been a catastrophic contest between two people who went to the same exclusive school? – opportunistically embraced the cause of Brexit in much the same spirit. What they had not figured out was that a diffuse, scattershot popular anger had not yet decisively found a powerful enough outlet, but that the staging of a referendum and the cohering of the leave cause would deliver exactly that. Ukip were held back by both the first-past-the-post electoral system, and the polarising qualities of Farage, but the coalition for Brexit effectively neutralised both. And so it came to pass: the cause of leaving the EU, for so long the preserve of cranks and chancers, attracted a share of the popular vote for which any modern political party would give its eye teeth.
In Wisconsin, a Backlash Against Using Data to Foretell Defendants’ Futures – The New York Times
More trial-by-algorithm horrors:
Company officials say the algorithm’s results are backed by research, but they are tight-lipped about its details. They do acknowledge that men and women receive different assessments, as do juveniles, but the factors considered and the weight given to each are kept secret. “The key to our product is the algorithms, and they’re proprietary,” said Jeffrey Harmon, Northpointe’s general manager. “We’ve created them, and we don’t release them because it’s certainly a core piece of our business. It’s not about looking at the algorithms. It’s about looking at the outcomes.” That secrecy is at the heart of Mr. Loomis’s lawsuit. His lawyer, Michael D. Rosenberg, who declined to be interviewed because of the pending appeal, argued that Mr. Loomis should be able to review the algorithm and make arguments about its validity as part of his defense. He also challenges the use of different scales for each sex. The Compas system, Mr. Rosenberg wrote in his brief, “is full of holes and violates the requirement that a sentence be individualized.”
(tags: ethics compas sentencing wisconsin northpointe law trial-by-algorithm algorithms)
-
high bike tire pressures are not faster, counterintuitively. I never knew! (via Tony Finch)
Holloways: Roads Tunneled into the Earth by Time
Appearing like trenches dragged into the earth, sunken lanes, also called hollow-ways or holloways, are centuries-old thoroughfares worn down by the traffic of time. They’re one of the few examples of human-made infrastructure still serving its original purpose, although many who walk through holloways don’t realize they’re retracing ancient steps.
(tags: cool hiking geography roads paths holloways psychogeography lanes)
Terrorism and internet blocking – is this the most ridiculous amendment ever? – EDRi
So, there you have it: Blocking is necessary, except it is not. Safeguards need to be implemented, except they don’t need to be. This approach is legal, except it isn’t. The text is based on the Child Exploitation Directive, except it isn’t. Is this really how we are going to create credible legislation on terrorism?
E-Voting in Estonia needs to be discontinued
After studying other e-voting systems around the world, the team was particularly alarmed by the Estonian I-voting system. It has serious design weaknesses that are exacerbated by weak operational management. It has been built on assumptions which are outdated and do not reflect the contemporary reality of state-level attacks and sophisticated cybercrime. These problems stem from fundamental architectural problems that cannot be resolved with quick fixes or interim steps. While we believe e-government has many promising uses, the Estonian I-voting system carries grave risks — elections could be stolen, disrupted, or cast into disrepute. In light of these problems, our urgent recommendation is that to maintain the integrity of the Estonian electoral process, use of the Estonian I-voting system should be immediately discontinued.
(tags: internet technology e-voting voting security via:mattblaze estonia i-voting russia cybercrime)
Squeezing blood from a stone: small-memory JVM techniques for microservice sidecars
Reducing service memory usage from 500MB to 105MB:
We found two specific techniques to be the most beneficial: turning off one of the two JIT compilers enabled by default (the “C2” compiler), and using a 32-bit, rather than a 64-bit, JVM.
The Irish Internet in the 1980s
from Dr Mark Humphrys in DCU:
A collection of bits and pieces of Internet history. Focusing somewhat (but not exclusively) on: (a) the 1980s, when I first started using the Internet, and: (b) Ireland.
(tags: mark-humphrys dcu history tcd bitnet ireland internet web www 1980s)
-
I need to get in touch about the early days of the Irish web!
an online home for stories from Ireland – stories about the country’s long and convoluted relationship with information technology. It aims to gather information on the most significant aspects of this relationship, to compile archives on the selected themes, and to store the assembled records for the benefit of future generations.
The History of the Irish Internet
This site is a companion effort to the techarchives website, except it is less well-researched, and is primarily a personal view of the development of the Internet in Ireland by your humble author, Niall Murphy.
(tags: niallm internet ireland history networking heanet ieunet)
What’s Actually Wrong with Yahoo’s Purchase of Summly
An old post about Y!’s acquisition of Summly, an iPhone app which uses NLP to summarise news stories. This is an excellent point about modern tech startups:
[Summly] licensed the core engine from another company. They are the quintessential bolt-on engineers, taking a Japanese bike engine, slapping together a badly constructed frame aligned solely by eyeballs, and laying down a marketing blitz. That’s why the story sells. “You, too, can do it.” But do you want to? […] it’s critical to keep tabs on the ratio known as “glue versus thought.” Sure, both imply progress and both are necessary. But the former is eminently mundane, replaceable, and outsource-able. The latter is typically what gives a company its edge, what is generally regarded as a competitive advantage. So, what is Yahoo signaling to the world? “We value glue more than thought.”
(tags: glue thought glue-vs-thought summly yahoo acquisitions licensing tech startups outsourcing open-source)
What the Irish Ate Before Potatoes – Bon Appétit
on the history of Irish cuisine — mostly milk and butter, and notably “bog butter”:
And the Irish didn’t like their butter just one way: from the 12th century on, there are records of butter flavored with onion and garlic, and local traditions of burying butter in bogs. Originally, it’s thought that bog butter began as a good storage system, but after a time, buried bog butter came to be valued for its uniquely boggy flavor.
(tags: bog-butter bogs ireland food eating milk curds whey banbidh dairy)
The tyranny of the algorithm yet again…
Paypal will no longer handle payments if the user’s address includes the word “Isis”:
That these place names exist won’t be a surprise to anyone familiar with English limnology – the study of rivers and inland waters. As Wikipedia helpfully tells us, “The Isis is the name given to the part of the River Thames above Iffley Lock which flows through the university city of Oxford”. In at least one local primary school I’m familiar with, the classes are called Windrush, Cherwell, Isis and Thames. […] Now PayPal has decided that they are not prepared to facilitate payments for goods to be delivered to an address which includes the word “Isis”. An Isis street resident ran into some unexpected difficulties when attempting to purchase a small quantity of haberdashery on the internet with the aid of a PayPal account. The transaction would not process. In puzzlement she eventually got irritated enough to brave the 24/7 customer support telephone tag labyrinth. The short version of the response from the eventual real person she managed to get through to was that PayPal have blacklisted addresses which include the name “Isis”. They will not process payments for goods to be delivered to an Isis related address, whatever state of privileged respectability the residents of such properties may have earned or inherited in their lifetimes to this point.
One has to wonder if this also brings the risk of adding the user to a secret list, somewhere. Trial by algorithm.(tags: isis algorithms automation fail law-enforcement paypal uk rivers)
Can the United Kingdom government legally disregard a vote for Brexit?
Oh thank god, there’s a “get out of jail” card before they destroy the global economy to appease the eurosceptics.
On the day after a vote for Brexit, the UK will still be a member state of the EU. All the legislation which gives effect to EU law will still be in place. Nothing as a matter of law changes in any way just because of a vote to Leave. What will make all the legal difference is not a decision to leave by UK voters in a non-binding advisory vote, but the decision of the prime minister on how to react before making any Article 50 notification. And what the prime minister will do politically after a referendum vote for Brexit is, at the moment, as unknown as the result of the result of the referendum itself.
(tags: brexit law uk government referenda eurosceptics eu)
-
comparison-shopping site for Irish car insurance. recommended by some random Broadsheet commenter, worth a try next time this comes up
(tags: comparison shopping ireland car-insurance insurance)
-
Apple have announced they plan to use it; Google use a DP algorithm called RAPPOR in Chrome usage statistics. In summary: “novel privacy technology that allows inferring statistics about populations while preserving the privacy of individual users”.
(tags: apple privacy anonymization google rappor algorithms sampling populations statistics differential-privacy)
-
The Department of Education has issued a new circular accepting it cannot defund the education of children whose parents do not want their kid’s data to be in POD [the privacy-infringing database of all Irish primary-school children]. They’ll only accept a written request as the basis of that refusal, however. So, here’s one you can use that meets the requirements. Send or give it to your school.
Three starts network-level ad blocking trial
Three, the mobile carrier, has begun warming up for a network-level ad blocking trial. It will become one of the first mobile carriers worldwide—and certainly in the UK—to try blocking ads before they are squirted over the network to the consumer, rather than attempting to hide or block ads locally on the device, which can cost both bandwidth and battery life. The ad blocking trial, which will affect both mobile websites and apps, will take place during a 24-hour period sometime between June 13 and 20. Three says it will contact customers and ask them to sign up for the trial, presumably via the online customer portal. It isn’t clear how large the trial will be. Technologically, the network-level ad blocking will be powered by Shine. Due to the nature of the beast—the constant tussle between ad publishers and ad blockers—Shine doesn’t like to talk about its tech in much detail. It sounds like Shine uses deep packet inspection and machine learning to find packets that contain ads, and then replaces or removes them in such a way that it doesn’t break the layout of the website or app.
Some thoughts on operating containers
R.I.Pienaar talks about the conventions he uses when containerising; looks like a decent approach.
(tags: ops containers docker ripienaar packaging)
ClickHouse — open-source distributed column-oriented DBMS
‘ClickHouse manages extremely large volumes of data in a stable and sustainable manner. It currently powers Yandex.Metrica, world’s second largest web analytics platform, with over 13 trillion database records and over 20 billion events a day, generating customized reports on-the-fly, directly from non-aggregated data. This system was successfully implemented at CERN’s LHCb experiment to store and process metadata on 10bn events with over 1000 attributes per event registered in 2011.’ Yandex-tastic, but still looks really interesting
Cross-Region Read Replicas for Amazon Aurora
Creating a read replica in another region also creates an Aurora cluster in the region. This cluster can contain up to 15 more read replicas, with very low replication lag (typically less than 20 ms) within the region (between regions, latency will vary based on the distance between the source and target). You can use this model to duplicate your cluster and read replica setup across regions for disaster recovery. In the event of a regional disruption, you can promote the cross-region replica to be the master. This will allow you to minimize downtime for your cross-region application. This feature applies to unencrypted Aurora clusters.
(tags: aws mysql databases storage replication cross-region failover reliability aurora)
Advanced Airflow (Lesson 1) : TriggerDagRunOperator
good intro to some Airflow concepts
Finding pearls; fuzzing ClamAV
great how-to for practical scanner fuzz testing
(tags: fuzz-testing clamav scanners security vulnerabilities testing)
Ireland goes Big Brother as police upgrade snooping abilities – The Register
The Garda Síochána has proposed to expand its surveillance on Irish citizens by swelling the amount of data it collects on them through an increase in its CCTV and ANPR set-ups, and will also introduce facial and body-in-a-crowd biometrics technologies. […] The use of Automated Facial Recognition (AFR) technology is fairly troubled in the UK, with the independent biometrics commissioner warning the government that it was risking inviting a legal challenge back in March. It is no less of an issue in Ireland, where the Data Protection Commissioner (DPC) audited Facebook in 2011 and 2012, and scolded the Zuckerborg over its use of facial recognition technology.
(tags: afr facial-recognition minority-report surveillance ireland gardai cctv anpr biometrics privacy)
-
“Here,” by Richard McGuire. Amazing piece of comic art from 1989
(tags: richard-mcguire art comics graphic-novels history time)
Stop it with short PGP key IDs!
What happened today? We still don’t really know, but it seems we found a first potentially malicious collision — that is, the first “nonacademic” case. Enrico found two keys sharing the 9F6C6333 short ID, apparently belonging to the same person (as would be the case of Asheesh, mentioned above). After contacting Gustavo, though, he does not know about the second — That is, it can be clearly regarded as an impersonation attempt. Besides, what gave away this attempt are the signatures it has: Both keys are signed by what appears to be the same three keys: B29B232A, F2C850CA and 789038F2. Those three keys are not (yet?) uploaded to the keyservers, though… But we can expect them to appear at any point in the future. We don’t know who is behind this, or what his purpose is. We just know this looks very evil. Now, don’t panic: Gustavo’s key is safe. Same for his certifiers, Marga, Agustín and Maxy. It’s just a 32-bit collision. So, in principle, the only parties that could be cheated to trust the attacker are humans, right? Nope. Enrico tested on the PGP pathfinder & key statistics service, a keyserver that finds trust paths between any two arbitrary keys in the strong set. Surprise: The pathfinder works on the short key IDs, even when supplied full fingerprints. So, it turns out I have three faked trust paths into our impostor.
UK at serious risk of over-blocking content online, human rights watchdog warns | Ars Technica UK
The IWF in the spotlight…
The blacklist operated by the IWF effectively amounts to censorship. Not only are the blacklist and notices sent to members of the IWF kept secret, but there is no requirement to notify website owners when their site has been added to the blacklist. Even where statutory rules do exist with respect to notice and take-down procedures (namely, the Terrorism Act 2006 and the Defamation (Operators of Websites) Regulations 2013), the provisions are not so concerned with safeguards for the protection of freedom of expression, as with offering an exemption from liability for ISPs.
Collecting my thoughts about Torus
Worryingly-optimistic communications about CoreOS’ recently-announced distributed storage system. I had similar thoughts, but Jeff Darcy is actually an expert on this stuff so he’s way more worth listening to on the topic ;)
(tags: jeff-darcy distcomp filesystems coreos torus storage)
German Privacy Regulators Fined Adobe, Others Over U.S. Data Transfers
Adobe was fined 8,000 euros, Punica 9,000 euros and Unilever 11,000 euros. The regulator said they had put in place alternative legal mechanisms for transferring data to the United States following the fine. “The fact that the companies have eventually implemented a legal basis for the transfer had to be taken into account in a favorable way for the calculation of the fines,” said Johannes Caspar, the Hamburg Commissioner for Data Protection. “For future infringements, stricter measures have to be applied.”
(tags: data-protection eu fines us privacy safe-harbor)
[dns-operations] Sad news today: systemd-resolved to be deployed in Ubuntu 16.10
systemd needs to stop breaking shit
The Mitsubishi Outlander vulnerability allows trivial remote car alarm unlocking.
Nearly-open wifi (easily-cracked weak WPA PSK), and a 6-byte string to disable the car alarm, discovered via replay attack. Massive fail
(tags: internetofshit mitsubishi fail outlander wpa alarms security replay-attack)
FullPageOS Automatically Boots Your Raspberry Pi Into a Full Page Web Kiosk Mode
set up to boot into a full-screen Chromium window on boot. This means if you’re using your Pi to power an information display, you won’t need to go through the process of disabling screen savers, editing display size, and forcing full-screen mode on your own. All you need to do is install FullPageOS on an SD card, then edit a TXT file to include your Wi-Fi network info and the URL you want it to load up.
(tags: kiosks raspberry-pi fullpageos chrome chromium web appliances hacks)
_Could a Neuroscientist Understand a Microprocessor?_
‘There is a popular belief in neuroscience that we are primarily data limited, that producing large, multimodal, and complex datasets will, enabled by data analysis algorithms, lead to fundamental insights into the way the brain processes information. Microprocessors are among those artificial information processing systems that are both complex and that we understand at all levels, from the overall logical flow, via logical gates, to the dynamics of transistors. Here we take a simulated classical microprocessor as a model organism, and use our ability to perform arbitrary experiments on it to see if popular data analysis methods from neuroscience can elucidate the way it processes information. We show that the approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the processor. This suggests that current approaches in neuroscience may fall short of producing meaningful models of the brain.’ via Bryan O’Sullivan.
(tags: via:bos neuroscience microprocessors 6502 computers hardware wetware brain biology neural-systems)
MPs’ private emails are routinely accessed by GCHQ
65% of parliamentary emails are routed via Dublin or the Netherlands, so liable to access via Tempora; NSA’s Prism program gives access to all Microsoft Office 365 docs; and MessageLabs, the anti-spam scanning system in use, has a GCHQ backdoor program called Haruspex, allegedly.
(tags: snowden privacy mps uk politics gchq nsa haruspex messagelabs symantec microsoft parliament)
-
‘centrally-planned object and thread pools’ for java. ‘In the default JVM thread pools, once a thread is created it will only be retired when it hasn’t performed a task in the last minute. In practice, this means that there are as many threads as the peak historical number of concurrent tasks handled by the pool, forever. These thread pools are also poorly instrumented, making it difficult to tune their latency or throughput. Dirigiste provides a fast, richly instrumented version of a java.util.concurrent.ExecutorService, and provides a means to feed that instrumentation into a control mechanism that can grow or shrink the pool as needed. Default implementations that optimize the pool size for thread utilization are provided. It also provides an object pool mechanism that uses a similar feedback mechanism to resize itself, and is significantly simpler than the Apache Commons object pool implementation.’ Great metric support, too.
(tags: async jvm dirigiste java threadpools concurrency utilization capacity executors object-pools object-pooling latency)
-
new Jenkins UX. looks great
Here comes the summer: how Daylight Saving Time came to Ireland
good history on the crapfest that is DST
‘Virtual nose’ may reduce simulator sickness
Anecdotal evidence has suggested simulator sickness is less intense when games contain fixed visual reference objects – such as a racecar’s dashboard or an airplane’s cockpit – located within the user’s field of view. “But you can’t have a cockpit in every VR simulation,” Whittinghill said. His research team was studying the problem when undergraduate student Bradley Ziegler suggested inserting the image of a virtual human nose in the center of the video display. “It was a stroke of genius,” said Whittinghill, who teaches video game design. “You are constantly seeing your own nose. You tune it out, but it’s still there, perhaps giving you a frame of reference to help ground you.” The researchers have discovered that the virtual nose, or “nasum virtualis,” reduces simulator sickness when inserted into popular games.
(tags: virtual-nose nose vr simulator-sickness nausea vr-sickness games)
-
Interesting new collections lib for Java 6+; generates Map-like and Set-like collections at runtime based on the contract annotations you desire. Fat (20MB) library-based implementation also available
(tags: collections java koloboke performance coding)
Symantec Issues Intermediate CA Certificate for Blue Coat Public Services
ugh, so dodgy
Green/Blue Deployments with AWS Lambda and CloudFormation – done right
Basically, use a Lambda to put all instances from an ASG into the ELB, then remove the old ASG
(tags: asg elb aws lambda deployment ops blue-green-deploys)
Six Years of Hacker News Comments about Twilio
love it.
(tags: twilio hn hackernews funny tech)
-
‘Tired of copy/pasting Dockerfiles around? Not sure about best practices for Dockerfiles or Docker entry points? This tool lets you Dockerize your applications using best practices to define your Dockerfile and Docker entry point files.’ The best practices in question are defined here: https://github.com/docker-library/official-images#review-guidelines
(tags: docker dockerfile images build best-practices alpine containers)
-
backward compatible replacement for Dockerfile. Yes, you can take any Dockerfile, rename it to Rockerfile and use rocker build instead of docker build. … Rocker aims to solve the following use cases, which are painful with plain Docker: Mount reusable volumes on build stage, so dependency management tools may use cache between builds. Share ssh keys with build (for pulling private repos, etc.), while not leaving them in the resulting image. Build and run application in different images, be able to easily pass an artifact from one image to another, ideally have this logic in a single Dockerfile. Tag/Push images right from Dockerfiles. Pass variables from shell build command so they can be substituted to a Dockerfile. And more. These are the most critical issues that were blocking our adoption of Docker at Grammarly. The most challenging part is caching. While implementing those features seems to be not a big deal, it’s not trivial to do that just by utilising Docker’s image cache (the one that docker build does). Actually, it is the main reason why those features are still not in Docker. With Rocker we achieve this by introducing a set of trade-offs. Search this page for “trade-off” to find out more details.
(tags: docker rocker build containers dockerfiles)
How big an issue is the nausea problem for Virtual Reality products? – Quora
Sadly (because I want a “holodeck” as much as the next red-blooded geek) – I don’t think it’s possible to make a VR system that both delivers the experience that everyone wants – and doesn’t make a sizeable proportion of the population so sick that they’ll never want to do it again. For the people who can stomach the display – my major concern is that the US Navy studies show that there is some disorientation that might persist long after finishing your game…so driving a car while “under the influence” of post-VR disorientation is probably as dangerous as drunk-driving. If these devices are in pretty much every home – then there are huge problems in store for the industry in terms of product liability. There have been plenty of warnings from the flight simulation industry – there are no excuses for not reading the Wikipedia article on the subject. If people are driving “under the influence” and the VR companies didn’t warn them about that – then they’re in deep trouble. IMHO, these consumer-grade VR devices should be carefully studied and if they do cause possible driving impairment, they should be banned until such time as the problems can be fixed…which may very well be “never”. Sorry to be the bearer of bad news.
(via Tony Finch)(tags: holodeck vr oculus-rift hmds nausea head-mounted-displays biology brain flight-simulation)
Why do Selenium-style record/replay tests of web applications break?
good data! Mostly because of element locations it seems….
LinkedIn called me a white supremacist
Wow. Massive, massive algorithm fail.
n the morning of May 12, LinkedIn, the networking site devoted to making professionals “more productive and successful,” emailed scores of my contacts and told them I’m a professional racist. It was one of those updates that LinkedIn regularly sends its users, algorithmically assembled missives about their connections’ appearances in the media. This one had the innocent-sounding subject, “News About William Johnson,” but once my connections clicked in, they saw a small photo of my grinning face, right above the headline “Trump put white nationalist on list of delegates.” […..] It turns out that when LinkedIn sends these update emails, people actually read them. So I was getting upset. Not only am I not a Nazi, I’m a Jewish socialist with family members who were imprisoned in concentration camps during World War II. Why was LinkedIn trolling me?
(tags: ethics fail algorithm linkedin big-data racism libel)
[RFE] add a way to run in a new systemd scope automatically · Issue #428 · tmux/tmux
omgwtfbbq. 1: User reports that their gnome session leaks processes; 2: systemd modifies default session behaviour to kill all processes, including screen/tmux; 3: _everyone_ complains because they break 30 years of UNIX process semantics, then 4: they request that tmux/screen hack their shit to workaround their brokenness. Get fucked, systemd. This is the kind of shit that would finally drive me to BSDland
(tags: systemd horror linux fail unix gnome tmux bugs omgwtfbbq)
The Dordogne Valley: What to Expect
French Foodie in Dublin writes and vlogs about the Dordogne Valley, good foodie tips
The Dutch word for “nitpicker” is significantly more sweary
via James Kelleher on Twitter: “‘Mierenneuker’ — Dutch slang for someone who pays (too much) attention to detail, literally ‘ant-fucker’.”; and in German, ‘Korinthenkacker’, “raisin-shitter”.
(tags: raisins funny words ants dutch german language nit-picking perfectionism)
#825394 – systemd kill background processes after user logs out – Debian Bug report logs
Systemd breaks UNIX behaviour which has been standard practice for 30 years:
It is now indeed the case that any background processes that were still running are killed automatically when the user logs out of a session, whether it was a desktop session, a VT session, or when you SSHed into a machine. Now you can no longer expect a long running background processes to continue after logging out. I believe this breaks the expectations of many users. For example, you can no longer start a screen or tmux session, log out, and expect to come back to it.
(tags: systemd ops debian linux fail background cli commandline)
-
Geofencing used for evil:
What Flynn realized is that he could use [ad targeting] to infer that a woman might be seeking an abortion, and to target her for ads from anti-choice groups [using geofenced advertising]. “We can reach every Planned Parenthood in the U.S.,” he wrote in a PowerPoint display sent to potential clients in February. The Powerpoint included a slide titled “Targets for Pro-Life,” in which Flynn said he could also reach abortion clinics, hospitals, doctors’ offices, colleges, and high schools in the United States and Canada, and then “[d]rill down to age and sex.” “We can gather a tremendous amount of information from the [smartphone] ID,” he wrote. “Some of the break outs include: Gender, age, race, pet owners, Honda owners, online purchases and much more.” Flynn explained that he would then use that data to send anti-choice ads to women “while they’re at the clinic.”
(tags: geofencing grim-meathook-future abortion phones smartphones pro-choice ads)
-
Rapid Fire is a special event we started hosting at our own in-person CTFs in 2014. The idea is pretty simple: Create several CTF challenges that can be solved in a few minutes each. Set up the challenges on 4 identical computers with some basic tools. Mirror the player’s screens so the audience can watch their actions. Whoever solves the most challenges the fastest wins. This event is interesting for a number of reasons: the players are under intense pressure, as everything they do is being watched by several people; the audience can watch several different approaches to the same problems; and people can follow along fairly easily with what is going on with the challenges.
With e-sports-style video!(tags: gaming hacking security e-sports streaming twitch ctf)
-
Twitter are open sourcing their Storm replacement, and moving it to an independent open source foundation
(tags: open-source twitter heron storm streaming architecture lambda-architecture)
Why the Very Silly Oracle v. Google Trial Actually Matters
If it’s illegal to write clean room implementations of APIs, then no one has clean hands. The now-shelved open source project Apache Harmony, like Android, reimplemented Java SE, and tech giant IBM contributed code to that project. Oracle itself built its business off a proprietary implementation of SQL, which was created by IBM. The proposition “Reimplementations of APIs are infringements” creates a recursive rabbit hole of liability that spans across the industry. Even the very 37 Java APIs at issue in this trial contain reimplementations of other APIs. Google witness Joshua Bloch—who, while at Sun Microsystems, wrote many of the Java APIs—testified that specific Java APIs are reimplementations of other APIs from Perl 5 and the C programming language.
(tags: apis fair-use copyright ip android java google oracle law)
-
GitLab continue to out-innovate Github, which is just wanking around with breaking the UI these days
(tags: gitlab github git ci cd containers docker deployment coding)
-
‘The algorithm of Lamport timestamps is a simple algorithm used to determine the order of events in a distributed computer system. As different nodes or processes will typically not be perfectly synchronized, this algorithm is used to provide a partial ordering of events with minimal overhead, and conceptually provide a starting point for the more advanced vector clock method. They are named after their creator, Leslie Lamport.’ See also vector clocks (which I think would be generally preferable nowadays).
(tags: vector-clocks distributed programming algorithm clocks time leslie-lamport coding distcomp)