Skip to content

Justin's Linklog Posts

Links for 2021-12-20

Links for 2021-12-19

  • The CFS quota container throttling problem

    Well, this is quite a messy one:

    Almost all services at Twitter run on Linux with the CFS scheduler, using CFS bandwidth control quota for isolation, with default parameters. The intention is to allow different services to be colocated on the same boxes without having one service’s runaway CPU usage impact other services and to prevent services on empty boxes from taking all of the CPU on the box, resulting in unpredictable performance, which service owners found difficult to reason about before we enabled quotas. The quota mechanism limits the amortized CPU usage of each container, but it doesn’t limit how many cores the job can use at any given moment. Instead, if a job “wants to” use more than that many cores over a quota timeslice, it will use more cores than its quota for a short period of time and then get throttled, i.e., basically get put to sleep, in order to keep its amortized core usage below the quota, which is disastrous for tail latency1. Since the vast majority of services at Twitter use thread pools that are much larger than their mesos core reservation, when jobs have heavy load, they end up requesting and then using more cores than their reservation and then throttling. This causes services that are provisioned based on load test numbers or observed latency under load to over provision CPU to avoid violating their SLOs. They either have to ask for more CPUs per shard than they actually need or they have to increase the number of shards they use.
    Note that Kubernetes uses CFS to implement CPU quotas by default, too. In the twitter thread about this post, a commenter noted: “‘By shrinking the CFS period, the worst case time between quota exhaustion causing throttling and the process group being able to run again is reduced proportionately’. Our P99s at previous gig reduced in line after I petitioned cloud provider to adjust setting.” — this at least seems like a relatively easy setting to tune.

    (tags: cgroups kubernetes linux k8s cfs scheduling containers quotas)

Links for 2021-12-17

  • The Web3 Fraud | USENIX

    By now effectively all ;login:’s readers have heard the term “web3” and “dapps” bandied about as if they are some great revolution.  They are not.  The technical underpinnings are so terrible that it is clear they exist only to hype the underlying cryptocurrencies.  The actual utility of these “decentralized” systems is already available in modern distributed systems in ways that are several orders of magnitude more efficient and more capable.

    (tags: bitcoin criticism cryptocurrency web3 crypto ethereum hype scams dapps)

Links for 2021-12-16

  • Consumer warranties and statutory rights

    wow, I didn’t realise we had statutory right to redress for faulty goods for 6 years:

    Statutory rights are provided for by legislation (Irish law and EU law as transposed in Ireland). These act as a kind of “legal guarantee”, entitling consumers to seek redress where an item is faulty. Consumers may rely on their statutory rights regardless of whether an item has a warranty or not. Under Irish law, consumers have up to six years to seek redress for faulty or defective items (both new and second-hand). If the product is defective, the seller is generally responsible for providing redress. If a fault arises within six months of purchase, it is presumed to have existed at the time of purchase. For this reason, the consumer should not have to provide proof of the defect. If the fault arises more than six months after purchase, the seller may request that the consumer prove the fault did not arise as a result of misuse – for instance, by obtaining a report from an independent expert. Where an item is faulty, the seller may first offer a repair or replacement item. If this is not possible or fails to correct the problem, a refund may then be provided. Remedies for faulty goods must be provided free of charge.

    (tags: rights repair support defective-goods guarantee warranty defects ireland eu)

Links for 2021-12-15

  • Web3 is going just great

    the ongoing shitfest that is crypto/NFTs

    (tags: blockchain crypto cryptocurrency nfts shitfest web3)

  • rTMS RCT produces excellent results

    This is pretty amazing:

    A recent randomized control trial, published in The American Journal of Psychiatry, shows astounding results are possible in five days or less. Almost 80% of patients crossed into remission — meaning they were symptom-free within days. This is compared to about 13% of people who received the placebo treatment. Patients did not report any serious side effects. The most common complaint was a light headache. […] “This study not only showed some of the best remission rates we’ve ever seen in depression,” said Shan Siddiqi, a Harvard psychiatrist not connected to the study, “but also managed to do that in people who had already failed multiple other treatments.” Siddiqi also said the study’s small sample size, which is only 29 patients, is not cause for concern. “Often, a clinical trial will be terminated early [according to pre-specified criteria] because the treatment is so effective that it would be unethical to continue giving people placebo,” said Siddiqi. “That’s what happened here. They’d originally planned to recruit a much larger sample, but the interim analysis was definitive.”

    (tags: depression fmri health neuroscience medicine rtms brain rcts)

Links for 2021-12-10

  • Bug #1624320 “systemd-resolved appends 127.0.0.53 to resolv.conf…” : Bugs : systemd package : Ubuntu

    Wow; recent Ubuntu versions force name resolution to operate via the systemd-resolved DNS resolver, which has some pretty major bugs and omissions:

    This bug just compromised every ubuntu machine on my network. It falsely says that DNSSEC is not supported by the nameserver and resorts to non-DNSSEC resolution. So every machine on my network just accepted bogus DNS replies from a MITM. Thanks.
    Is there anything systemd can’t break :(

    (tags: systemd fail dns dnssec mitm security resolvers ubuntu bugs linux)

  • Birds Aren’t Real, or Are They? Inside a Gen Z Conspiracy Theory

    This is glorious. Well done, this chap… very reminiscent of the Subgenii

    (tags: birds conspiracies qanon funny birds-arent-real us-politics)

  • The Stockdale Paradox

    On the one hand, they stoically accepted the brutal facts of reality. On the other hand, they maintained an unwavering faith in the endgame, and a commitment to prevail as a great company despite the brutal facts. [..] “I never lost faith in the end of the story,” [Stockdale] said, when I asked him. “I never doubted not only that I would get out, but also that I would prevail in the end and turn the experience into the defining event of my life, which, in retrospect, I would not trade.” I didn’t say anything for many minutes, and we continued the slow walk toward the faculty club, Stockdale limping and arc-swinging his stiff leg that had never fully recovered from repeated torture. Finally, after about a hundred meters of silence, I asked, “Who didn’t make it out?” “Oh, that’s easy,” he said. “The optimists.” “The optimists? I don’t understand,” I said, now completely confused, given what he’d said a hundred meters earlier. “The optimists. Oh, they were the ones who said, ‘We’re going to be out by Christmas.’ And Christmas would come, and Christmas would go. Then they’d say, ‘We’re going to be out by Easter.’ And Easter would come, and Easter would go. And then Thanksgiving, and then it would be Christmas again. And they died of a broken heart.” Another long pause, and more walking. Then he turned to me and said, “This is a very important lesson. You must never confuse faith that you will prevail in the end—which you can never afford to lose—with the discipline to confront the most brutal facts of your current reality, whatever they might be.” To this day, I carry a mental image of Stockdale admonishing the optimists: “We’re not getting out by Christmas; deal with it!”

    (tags: paradoxes jim-stockdale stoicism philosophy optimism pessimism)

Links for 2021-12-09

  • The Irish family who added to the Australian vernacular

    Great bit of Aussie/Irish etymology:

    When the first World War broke out in 1914, Furphy water carts were used to bring water to Australian troops in Australia, Europe and the Middle East.. Soldiers would gather round the Furphy to get a drink and to have a chat, telling jokes and tall stories. That gave rise to the use of the word Furphy as a rumour or a false report which continues to the present day. The two companies, Furphy Foundry and J Furphy and Sons remain after five generations in family ownership and continue to produce many products including watercarts, all of which proudly bear the name “Furphy” in prominent lettering.

    (tags: furphy ireland australia etymology words history)

  • Bros., Lecce: We Eat at The Worst Michelin Starred Restaurant, Ever

    This is hilarious: ‘Recommendation: Do not eat here. I cannot express this enough. This was single-handedly one of the worse wastes of money in my entire food and travel writing career bwah ha ha ha ha ha ha oh my god’ Top comment: ‘I’ve eaten there! It was, hands down, the WORST dining experience I’ve ever had — and I’ve eaten at a place where the food was so disgusting, I ended up vomiting on the table. It was worse than that.’

    (tags: dining food funny lecce sicily holidays michelin)

  • Discovering related sites by tracing shared ad accounts

    Nice process using https://well-known.dev/ . Very handy for tracing undisclosed links between astroturf political pressure groups, in particular

    (tags: web politics astroturf investigation ads google-ads advertising)

Links for 2021-12-07

  • Life360 sells kids’ location data to “approximately a dozen data brokers”

    This is shocking: Wolfie Christl says “Life360, a popular family safety app used by 33 million people worldwide, has been marketed as a great way for parents to track their children’s movements.” Also, it sells “data on kids’ and families’ whereabouts to approximately a dozen data brokers”. Former employees of data brokers “described Life360 as one of the largest sources of data for the industry” — “A former X-Mode engineer said the raw location data the company received from Life360 was among X-Mode’s most valuable offerings”. X-Mode sold data to the US military. An app that claims to be a family safety service selling exact location data to several other companies, this is a total disaster. It would be a problem if it’s any other app, and it’s even more a problem when it’s an app that claims to be a family safety service. Selling data on children to companies who sell to the military is probably the most extreme form of decontextualizing sensitive data for profit.” Life360 are now planning to buy Tile.

    (tags: refractive-surveillance surveillance children privacy data-privacy location gps life360 tile data-brokers)

Links for 2021-12-06

  • An upper bound on one-to-one exposure to infectious human respiratory particles | PNAS

    Masks just work:

    Our results show that face masks significantly reduce the risk of SARS-CoV-2 infection compared to social distancing. We find a very low risk of infection when everyone wears a face mask, even if it doesn’t fit perfectly on the face.

    (tags: masks covid-19 papers face-masks infection)

  • Ikea Vindriktning Air Quality Sensor Review and Accuracy

    ‘Ikea recently came out with a range of air purifiers and also an air quality sensor. The Vindriktning does not have a display but shows the air quality data in the form of a traffic light with red, yellow and green LEDs. One of the most striking features is actually the price as it costs only around EUR 10 depending where you live. It looks very nice and the build quality is quite good but this article will look beyond the looks and see how good it is at actually measuring the air quality.’ The results are mixed: ‘I really want to like the Vindriktning! It has a great built quality and price and is very simple to use. The addition of a small fan to improve the air flow through the sensor is a good upgrade and shows that Ikea wants to provide accurate measurements — even with a cheap sensor. However, the defined cut off values for the air quality and its description as “Good”, “OK”, and “Not Good” are not based on science or international recommendations and create the false understanding that the air is good, when in fact it is not good at all. I do hope that in one of the next upgrades of the Vindriktning, Ikea will bring its traffic light indicators more in line with WHO recommendations on healthy air quality.’ Personally, this sounds useful — as long as one remembers that the “OK” air quality level is in fact well into the “unhealthy” zone. Bit mysterious as to why IKEA made this choice though…

    (tags: ikea air-quality pm2.5 pm10 particulates home devices gadgets)

Links for 2021-12-03

  • The Labyrinth Stone

    In medieval times the labyrinth underwent a revival and became primarily a symbol of pilgrimage, and in particular pilgrimage to the holy shrine of Jerusalem (Coleman & Elsner 1995, 112). Shortly after the loss of Jerusalem to the Muslims in the twelfth century, large labyrinths of mosaic or paving stones were incorporated into the western nave bays of a number of European cathedrals in the twelfth and thirteenth centuries (Connolly 2005, 286). [….] By walking, or in some cases crawling on their knees, along the labyrinth, pilgrims could perform an imagined pilgrimage to Jerusalem (Westbury 2001, 51-52).

    (tags: pilgrimage history labyrinths mazes via:neil-jackman)

Links for 2021-11-24

  • FIND

    “Framework for Internal Navigation and Discovery” — track device locations using active or passive (wifi-based) scan methods within a house or office, then trigger Home Assistant automation based on device locations — e.g. turning on or off heating in specific rooms, etc.

    (tags: location home-assistant home automation tracking devices)

Links for 2021-11-23

  • Google Cloud incident was caused by a race condition which triggered 30 minutes before the bugfix deployment was due to complete

    Wow, this was tragic! “A Google engineer discovered this bug on 12 November, which caused us to declare an internal high-priority incident because of the latent risk to production systems. After analyzing the bug, we froze a part of our configuration system to make the likelihood of the race condition even lower. Since the race condition had existed in the fleet for several months already, the team believed that this extra step made the risk even lower. Thus the team believed the lowest-risk path […] was to roll out fixes in a controlled manner as opposed to a same-day emergency patch. […] Gradual rollouts of both patches started on Monday, 15 November, and patch B completed rollout by that evening. On Tuesday, 16 November, as the patch A rollout was within 30 minutes of completing, the race condition did manifest in an unpatched cluster, and the outage started.”

    (tags: cloud outages tragic google race-conditions gclb patching deployment ops)

  • “Risk compensation” is garbage

    Risk compensation does occur in very narrow and specific circumstances, but all the studies purporting to show that it is a widespread, predictable outcome of any safety regulation have failed to replicate. […] Risk compensation and health-and-safety panic are both part of a safety nihilism campaign that serves big business’s deregulatory agenda, and the cruel moralizing of right wing religious maniacs, the traditional turkeys-voting-for-Christmas coalition. But risk compensation is especially salient in these covid days, where it’s being used to fight rapid testing (“encourages risky behavior”).

    (tags: risk-compensation risks safety)

Links for 2021-11-09

Links for 2021-11-08

Links for 2021-11-02

Links for 2021-10-26

Links for 2021-10-22

  • GNI report on biomethane generation in Ireland

    Good news for biogas feasability here: ‘This report seeks to provide scientific analysis and real-world data on the key questions and knowledge gaps concerning the sustainability of an Irish agricultural-led biomethane industry. The core aim of this report is to assess whether Ireland can develop an environmentally sustainable biomethane industry without creating unintended negative consequences. […] This report provides evidence that the development of a sustainable biomethane industry in Ireland is technically feasible and so long as it is developed in a co-ordinated manner, can avoid any negative unintended consequences. As such, a number of proven methodologies have been provided to drive the rollout of a biomethane industry whilst ensuring continued agricultural productivity and improved environmental sustainability.’

    (tags: biogas biomethane methane fuel ireland sustainability climate-change gni farming)

Links for 2021-10-20

  • Datamuse API

    This is extremely cool! “a word-finding query engine for developers. You can use it in your apps to find words that match a given set of constraints and that are likely in a given context. You can specify a wide variety of constraints on meaning, spelling, sound, and vocabulary in your queries, in any combination. Applications use the API for a wide range of features, including autocomplete on text input fields, search relevancy ranking, assistive writing apps, word games, and more.” (via Rob Manuel)

    (tags: dictionary nlp words writing apis rhymes sounds-like adjectives nouns suggestions)

Links for 2021-10-18

  • Transmission of SARS-CoV-2 After COVID-19 Screening and Mitigation Measures for Primary School Children Attending School in Liège, Belgium

    This paper from a school in Belgium is really worrying, given Ireland’s approach to schools and COVID-19. “Despite the implementation of several mitigation measures, the incidence of COVID-19 among children attending primary school in this study was comparable to that observed among teachers and parents. Transmission tree reconstruction suggests that most transmission events originated from within the school.”

    Question: What is the possible role of children in SARS-CoV-2 transmission? Findings: This cohort study including 63 children and 118 adults found no significant difference between the number of children and the number of adults testing positive for SARS-CoV-2 infection during the study period; children were asymptomatic significantly more often compared with adults (46% vs 13%). In addition, a reconstruction of the outbreak showed that most transmission events originated from within the school. Meaning: These results suggest that children may play a larger role in the transmission of SARS-CoV-2 than previously assumed.

    (tags: transmission schools education covid-19 sars-cov-2 papers belgium infection)

Links for 2021-10-17

  • “Grievance Mining”

    What a great term for what the Tories are up to in the UK: “Facing chaos and needing a scapegoat, the Tories seek an endless fight with Europe” —

    Frost is well aware of the futility of his demands – indeed, it is the whole point of his Lisbon performance. Instead of declaring victory, accepting the EU’s munificent offers and turning down the heat in Northern Ireland, he and Johnson prefer to make an impossible demand so that they can blame the EU for rejecting it. They are, as the South Belfast MP, Claire Hanna, has put it, “mining for grievance”.

    (tags: grievances neologisms phrases boris-johnson uk brexit politics northern-ireland eu)

Links for 2021-10-14

  • Fruit fly brains include a variant of a Bloom filter data structure

    Wow, this is incredible!

    We found that the fruit fly olfactory circuit evolved a variant of a Bloom filter to assess the novelty of odors. Compared with a traditional Bloom filter, the fly adjusts novelty responses based on two additional features: the similarity of an odor to previously experienced odors and the time elapsed since the odor was last experienced. We elaborate and validate a framework to predict novelty responses of fruit flies to given pairs of odors. We also translate insights from the fly circuit to develop a class of distance- and time-sensitive Bloom filters that outperform prior filters when evaluated on several biological and computational datasets. Overall, our work illuminates the algorithmic basis of an important neurobiological problem and offers strategies for novelty detection in computational systems.

    (tags: fruit-flies data-structures algorithms brains neuroscience smell)

Links for 2021-10-08

  • Let’s Encrypt Root Expiration – Post-Mortem

    Overall, I think the expiration of the Let’s Encrypt CA certificates went really quite well, largely due to the work Let’s Encrypt did around arranging for a new cross-signed chain to be available beyond the expiration of the IdenTrust root. That said, there were far more issues in areas we didn’t anticipate. Modern devices, all the way through to latest versions of iOS and macOS hit issues when connecting to servers that had a misconfigured certificate chain and quite serious issues from huge companies like Google and Microsoft in their cloud products that could no longer validate certificate chains was surprising to say the least. In all, I think this just highlights something that many of us that work in this space have known for some time, that TLS/PKI are complex and fragile systems that often go overlooked for long periods of time because they ‘just work’ most of the time. [….] One thing that’s certain is that this event is coming again. Over the next few years we’re going to see a wide selection of Root Certificates expiring for all of the major CAs and we’re likely to keep experiencing the exact same issues unless something changes in the wider ecosystem.

    (tags: postmortem ssl tls pki fail post-mortems lets-encrypt cas)

Links for 2021-10-06

  • Frances Haugen says Facebook’s algorithms are dangerous. Here’s why. | MIT Technology Review

    This is a good article on FB’s disastrous situation, which would be bad enough were it not endangering our societies. Despite warnings from Google and others, they switched their engagement optimization tactics to rely heavily on machine learning, which (as noted elsewhere) devolves into a situation where it’s thoroughly inscrutable:

    It developed an internal tool known as FBLearner Flow that made it easy for engineers without machine learning experience to develop whatever models they needed at their disposal. By one data point, it was already in use by more than a quarter of Facebook’s engineering team in 2016. Many of the current and former Facebook employees I’ve spoken to say that this is part of why Facebook can’t seem to get a handle on what it serves up to users in the news feed. Different teams can have competing objectives, and the system has grown so complex and unwieldy that no one can keep track anymore of all of its different components. […] “64% of all extremist group joins are due to our recommendation tools,” the presentation said, predominantly thanks to the models behind the “Groups You Should Join” and “Discover” features. […] These phenomena are far worse in regions that don’t speak English because of Facebook’s uneven coverage of different languages. […] When the war in Tigray[, Ethiopia] first broke out in November, [AI ethics researcher Timnit] Gebru saw the platform flounder to get a handle on the flurry of misinformation. […] When fake news, hate speech, and even death threats aren’t moderated out, they are then scraped as training data to build the next generation of [language models]. And those models, parroting back what they’re trained on, end up regurgitating these toxic linguistic patterns on the internet.”
    What. A. Mess.

    (tags: machine-learning social-networking facebook the-algorithm llms models frances-haughen)

  • The Verica Open Incident Database (VOID)

    ‘A community-contributed collection of software-related incident reports’ — this looks like it’ll be a great resource.

    (tags: resilience engineering ops outages post-mortems rcas five-whys incidents)

  • debunking “it takes 48,000 miles for an EV to be greener than an ICE vehicle”

    Looks like this is disinformation produced by an Aston-Martin-affiliated lobbyist/PR company — the true figure is 18,000 miles

    (tags: debunking pr lobbying cars evs aston-martin spin greenwashing)

Links for 2021-10-05

  • _Endgame: A zero-carbon electricity plan for Ireland_

    A report commissioned by Wind Energy Ireland in June 2021 — key findings:

    Reducing power sector CO2 emissions in Ireland from around 9 million tonnes today to a target of less than 2 million tonnes of CO2 per year is very achievable by 2030, using the approach currently underway to achieve the ‘70 by 30’ target, and implementing more of existing and proven technologies; The current Programme for Government renewable capacity targets of 8.2 GW of onshore wind and 5 GW of offshore wind by 2030 should be maintained, with an additional target of 5 GW of solar PV. This target can be achieved at a lower cost to the end consumer in Ireland, compared to delivery of the less ambitious ‘70 by 30’ target. A zero-carbon power system is possible by 2030 and represents an achievable target in the 2030s.

    (tags: wind-energy energy ireland future 2030 climate-change co2 solar-power carbon)