Skip to content

2025

_Cheap science, real harm: the cost of replacing human participation with synthetic data_ [pdf]

  • _Cheap science, real harm: the cost of replacing human participation with synthetic data_ [pdf]

    A new paper from the inimitable Abeba Birhane, on the increasingly common practice of generating synthetic data using LLMs:

    Driven by the goals of augmenting diversity, increasing speed, reducing cost, the use of synthetic data as a replacement for human participants is gaining traction in AI research and product development. This talk critically examines the claim that synthetic data can "augment diversity," arguing that this notion is empirically unsubstantiated, conceptually flawed, and epistemically harmful. While speed and cost-efficiency may be achievable, they often come at the expense of rigour, insight, and robust science. Drawing on research from dataset audits, model evaluations, Black feminist scholarship, and complexity science, I argue that replacing human participants with synthetic data risks producing both real-world and epistemic harms at worst and superficial knowledge and cheap science at best.

    "Synthetic data: stereotypes compressed" is absolutely spot on. This doesn't give insights into human behaviour and beliefs, just into stereotypes. It is increasingly common in social science fields, under the names of "digital twins" and "silicon samples".

    data surveys abeba-birhane papers ai synthetic-data digital-twins simulation testing social-science silicon-samples

Boost for artists in AI copyright battle as only 3% back UK active opt-out plan

  • Boost for artists in AI copyright battle as only 3% back UK active opt-out plan

    Wow, this is an absolute bollocking for the Labour plan:

    95% of the more than 10,000 people who had their say over how music, novels, films and other works should be protected [in the UK] from copyright infringements by tech companies called for copyright to be strengthened and a requirement for licensing in all cases or no change to copyright law. By contrast, only 3% of people backed the UK government's initial preferred tech company-friendly option, which was to require artists and copyright holders to actively opt out of having their material fed into data-hungry AI systems.

    ai training data copyright law uk uk-politics llms

Avoid UUID Version 4 Primary Keys | Software Engineer, Author, High Performance PostgreSQL for Rails

  • Avoid UUID Version 4 Primary Keys | Software Engineer, Author, High Performance PostgreSQL for Rails

    A well-researched article suggesting that random UUIDs do not make a good primary key for database tables; I would tend to agree (for cases where performance is important).

    • UUID v4s increase latency for lookups, as they can't take advantage of fast ordered lookups in B-Tree indexes
    • For new databases, don't use gen_random_uuid() for primary key types, which generates random UUID v4 values
    • UUIDs consume twice the space of bigint
    • UUID v4 values are not meant to be secure per the UUID RFC
    • UUID v4s are random. For good performance, the whole index must be in buffer cache for index scans, which is increasingly unlikely for bigger data.
    • UUID v4s cause more page splits, which increase IO for writes with increased fragmentation, and increased size of WAL logs
    • For non-guessable, obfuscated pseudo-random codes, we can generate those from integers, which could be an alternative to using UUIDs
    • If you must use UUIDs, use time-orderable UUIDs like UUID v7

    postgres rails databases sql mysql uuids indexing primary-keys keys lookup storage random

'Pig Butchering' Scams May Have Spurred Thailand-Cambodia War

  • 'Pig Butchering' Scams May Have Spurred Thailand-Cambodia War

    Via TJ McIntyre -- indications that the Thailand-Cambodia war is being driven by the "pig butchering" scammer compounds operating in the border area:

    Cambodia's 2019 census put O'Smach's population just over 9,850, but that doesn't include the prison-like, office-dormitory compounds that have appeared here over the past five years, with the capacity to house 10,000 more. Around 50 sites like these now line the Cambodia-Thailand border, designed to house a slice of the trillion-dollar cybercrime industry—primarily teams running investment scams, dubbed "pig butchering" for the way they fatten their targets up; sextortion scams that blackmail victims, including children, by threatening to make sexual images public; scams that impersonate police to gain account access; and fraudulent online gambling sites. Once aimed largely at the Chinese public, these now target victims worldwide and rake in tens of billions of dollars a year in Cambodia alone.

    The compounds evolved from a casino industry that caters mostly to Chinese tourists and Thai day-trippers and has been linked to human trafficking, drug smuggling, and the endangered wildlife trade. From 2016, physical casinos were dwarfed by the online gambling industry (outlawed by Cambodia in 2019), which progressed to illegal sites and outright scams. Operators rent space in casinos and purpose-built compounds controlled by Chinese criminals, Myanmar warlords, and the Cambodian political elite.

    Scam companies rely heavily on forced and trafficked labor from Asia, Africa, and Latin America to chat with targets, pose as romantic interests and employees at fake investment platforms, and persuade them to make deposits. Survivors tell us that torture, rape, and beatings are common. As the fighting raged in July, some trafficking victims reached out for help, saying they were locked in their dorms by their bosses. Videos shot from inside these sites show missiles flying overhead, explosions thundering outside, some workers appearing to break out and run, and damage from shelling in the grounds.

    scams phishing pig-butchering war grim-meathook-future thailand cambodia scammers

Year in Review 2025: Hari Kunzru on AI slop and censorship

  • Year in Review 2025: Hari Kunzru on AI slop and censorship

    Hari Kunzru nails it:

    These days I have a sense of falling from a precipice toward a torrent of algorithmically driven slop. It's coming, whether we want it or not, and the consequences for our communal life will be devastating.

    It's now seven years since Steve Bannon outlined his infamous strategy to "flood the zone with shit." This, he said, was a way to "deal with" the media, whom he saw as the real enemies of MAGA. In practice, it has been a very effective method of censorship. With every important issue of the day, the "zone" of public discourse is immediately filled with a volume of competing narratives, often mendacious or misleading. It's no longer necessary to suppress information. You just have to make the cost of sorting fact from fiction, in terms of time and effort, too high to pay for the ordinary person, who can't spend all day online weighing up competing claims about robots or pedophilia or Iran.

    Generative AI now allows the production of disinformation at scale. The kind of influence ops we associate with Cambridge Analytica or the Russian Internet Research Agency can be conducted with unprecedented scope and sophistication: Thousands of fake people — tens of thousands, perhaps hundreds of thousands — making videos, posting in forums, astroturfing entire contexts in which people will live out their political lives. Couple this with the collapse of trust in all kinds of authority, and there is no one even to say what might distinguish "disinformation" from any other kind of data. [...]

    The desire to return to consensus reality is hopelessly nostalgic. Yes, there are still hard limits: The "cloud" is a physical place, scooping out mountains for raw materials and venting heat and carbon dioxide out of gargantuan data centers; political power still grows out of the barrel of a gun. But the layer of the stack in which our subjectivities are formed, the place where our beliefs about the world are shaped, is also a battleground. We must teach ourselves to navigate the torrent that is replacing consensus reality, this turbulent, treacherous mediatized flow. There is no shore to swim back to, but in the new age of magic, when reality is labile and can be recoded by the power of signs, by narrative and memes and vibes and compelling images, art becomes a truly political technology. This is not art as critique. Critique is just sincere-posting, dutifully pointing out yet again that the Medbed isn't "real." Art can mess with our masters in ways we don't yet fully understand. It makes culture. It is a transmitter of values. It is the lava out of which future realities will congeal.

    misinformation disinformation facts reality future ai slop hari-kunzru steve-bannon flooding-the-zone-with-shit art media propaganda

Multiplying our way out of division — Matt Godbolt's blog

  • Multiplying our way out of division — Matt Godbolt's blog

    A very silly optimisation for the "binary to decimal" conversion problem:

    The compiler has turned division by a constant ten into a multiply and a shift. There's a magic constant 0xcccccccd and a shift right of 35! Shifting right by 35 is the same as dividing by 235 - what's going on? [..]

    What's happening is that 0xcccccccd / 2**35 is very close to ? (around 0.10000000000582077). By multiplying our input value by this constant first, then shifting right, we're doing fixed-point multiplication by ? - which is division by ten. The compiler knows that for all possible unsigned integer values, this trick will always give the right answer.

    hacks optimization bit-hacking binary decimal fixed-point arithmetric tricks

Large Language Models As The Tales That Are Sung

'Unauthorized' Edit to Ukraine's Frontline Maps Point to Polymarket's War Betting

  • 'Unauthorized' Edit to Ukraine's Frontline Maps Point to Polymarket's War Betting

    A live map that tracks frontlines of the war in Ukraine was edited to show a fake Russian advance on the city of Myrnohrad on November 15. The edit coincided with the resolution of a bet on Polymarket, a site where users can bet on anything from basketball games to presidential election and ongoing conflicts. If Russia captured Myrnohrad by the middle of November, then some gamblers would make money. According to the map that Polymarket relies on, they secured the town just before 10:48 UTC on November 15. The bet resolved and then, mysteriously, the map was edited again and the Russian advance vanished.

    polymarket betting war future cyberpunk fraud ukraine russia

Building a Medallion architecture with ClickHouse

  • Building a Medallion architecture with ClickHouse

    Walkthrough of the "Medallion" architecture concept, which comprises three layers (or stages), each serving distinct purposes in the data pipeline:

    • Bronze layer - This layer acts as the landing area for raw, unprocessed data directly from the source system: simply put a "staging area". This data is stored in its original structure with minimal transformations and additional metadata. This layer is optimized for fast ingestion, and can provide an historical archive of source data that is always available for reprocessing or debugging. Whether the bronze layer should store all data is a point of contention, with some users preferring to filter the data and apply transformations, e.g., flattening JSON, renaming fields, or filtering out poorly formed data. We're not overly opinionated here but recommend optimizing the storage for consumption by the silver layer only - not other consumers.

    • Silver layer - Here, data is cleansed, deduplicated, and conformed to a unified schema, with raw data from the previous Bronze layer being enriched and transformed to provide a more accurate and consistent view. This data can be consistent and usable for enterprise-wide use cases such as machine learning and analytics. The data model should emerge at this layer with a focus placed on ensuring primary and foreign keys are consistent to simplify future joins. While not common, applications and downstream consumers can read from this layer. These are typically business-wide applications that need the entire cleansed dataset, e.g., ML workflows. Importantly, data quality will not improve after this stage only the ease at which it can be queried efficiently.

    • Gold layer - This later aims to have fully curated, business-ready, and project-specific datasets that make the data more accessible (and performant) to consumers. These datasets are often denormalized, or pre-aggregated, for optimal read performance and may have been composed of multiple tables from the previous silver stage. The focus here is on applying final transformations and ensuring the highest data quality for consumption by end-users or applications, such as reporting and user-facing dashboards.

    This layered approach to data pipelines aims to efficiently address challenges like data quality, duplication and schema inconsistencies. By transforming raw data incrementally, the Medallion architecture aims to ensure a clear lineage and progressively refined datasets that are ready for analysis or operational use.

    medallion-architecture data architecture pipelines clickhouse

Questioning an Interface: From Parquet to Vortex

  • Questioning an Interface: From Parquet to Vortex

    Interesting -- a new, GPU-optimised storage format:

    Like Parquet, Vortex minimizes bytes on disk. However, Vortex is also designed with a core use-case in mind: decoding and querying data directly from object storage on GPUs. This key idea translates very well to our use-case even though we don't run our queries on GPUs (yet?). Specifically, the file format is designed to maximize throughput and parallelism from the metadata format to the SIMD/SIMT friendly encodings used.

    Crucially, it also acknowledges that part of making queries fast is not only good filter pushdown, but also general-purpose compute pushdown. If anything cannot be pushed down, Vortex's encodings can be tuned to offer zero-copy conversion to Arrow for further query execution using any general-purpose query execution engine.

    Vortex also learns from Parquet's limitations around extensibility and aims to be as future-proof as possible. New encodings can ship with WASM decoders so encoding adoption is not limited by reader libraries having to implement support. The main Rust library is also designed to be fully extensible, so you can write your own layouts/encodings and plug them in as first-class citizens.

    Given how well Vortex's design matched our needs, we tried it out and got a 70% average performance improvement on all our queries. With the newer encodings that Vortex offers, we got 10% better uncompressed storage size and only 3% larger compressed storage size compared to snappy-compressed Parquet.

    gpu vortex parquet compression storage file-formats files pushdown simd

The shameful attacks on the Covid inquiry prove it: the right is lost in anti-science delusion

  • The shameful attacks on the Covid inquiry prove it: the right is lost in anti-science delusion

    Polly Toynbee in the Guardian writes, "The shameful attacks on the Covid inquiry prove it: the right is lost in anti-science delusion":

    That number will stay fixed for ever in public memory: 23,000 people died because Boris Johnson resisted locking the country down in time. As Covid swept in, and with horrific images of Italian temporary morgues in tents, he went on holiday and took no calls. With the NHS bracing to be "overwhelmed" by the virus, he rode his new motorbike, walked his dog and hosted friends at Chevening.

    Nothing is surprising about that: he was ejected from Downing Street and later stepped down as an MP largely for partying and lying to parliament about it. Everyone knew he was a self-aggrandising fantasist with a "toxic and chaotic culture" around him. But this is not just about one narcissistic politician. It's about his entire rightwing coterie of libertarians and their lethally dominant creed in the UK media.

    I'm glad the science side kept their receipts but I fear this argument will be relitigated indefinitely by anti-lockdown libertarians.

    lockdowns covid-19 history uk uk-politics medicine health pandemics boris-johnson

The developer productivity paradox: Why faster coding doesn't mean faster software delivery

  • The developer productivity paradox: Why faster coding doesn't mean faster software delivery

    The paradox is this simple gap: high individual confidence in AI speed, versus stubborn organizational metrics that just won't budge:

    • Perceived speed is high: Adoption is near-universal (90% usage reported), and confidence is overwhelming (over 80% believe AI has increased their productivity). AI is great at handling cognitive toil and boilerplate, which lets engineers generate bigger code batches and feel genuinely productive.
    • Systemic failure persists: The reality, confirmed by DORA in their 2025 report, is that the system often fails to carry or amplify these individual gains. The challenge is that AI models, as massive generative systems, inherently produce failures (mispredictions). As code volume increases, this constant misprediction rate impacts systemic stability.

    Interestingly, even leading providers of AI solutions like OpenAI and Anthropic continue to be challenged by the issue of hallucinations and mispredictions, as well as the risks generated by AI. Speaking at a university in India, Sam Altman recently said "I probably trust the answers that come out of ChatGPT the least of anybody on Earth".

    Without strategies and tools for alleviating the issues AI code produces downstream — such as improved observability to understand where something is going wrong — the "much bigger engine" of AI may not actually speed up software delivery after all.

    ai llms coding productivity gradle dpe hallucinations software work how-we-work

Cloudflare outage on November 18, 2025

  • Cloudflare outage on November 18, 2025

    tl;dr: a configuration-generation tool had buggy error handling code. Triggered by a permissions change, it generated over-large configs which then caused a crash in buggy config-reading code in their Bot Management module. This configuration was rolled out globally within minutes.

    As @kiall in ITC Slack notes: "the one thing I'd be pushing on after an outage like this (config mistake, propagated globally..) is "treat config like any other deployment - with a slow and steady rollout" -- and this is not called out in the postmortem. I agree this is a significant oversight.....

    postmortem cloudflare outages configuration deployment cloud error-handling rollouts via:itc

Bitcoin's big secret: How cryptocurrency became law enforcement's secret weapon

  • Bitcoin's big secret: How cryptocurrency became law enforcement's secret weapon

    At the 2025 Bitwarden Open Source Security Summit, WIRED's Andy Greenberg sat down for a fireside chat with GigaOm analyst Paul Stringfellow to discuss a revelation that turned his decades-long reporting on its head: Bitcoin became a criminal's worst nightmare:

    In 2011, Greenberg thought he'd discovered the story of a lifetime: digital cash that promised complete anonymity. A decade later, that story flipped entirely.

    "I had this slow-motion epiphany that I was entirely wrong about Bitcoin. It was, in fact, the opposite of untraceable."

    But here's the paradox: if cryptocurrency tracing is so powerful, why do ransomware attacks, pig butchering scams, and North Korean hackers continue to steal billions?

    The answer: identifiability isn't the same as accountability.

    accountability prosecutions law policing bitcoin cryptocurrency andy-greenberg crime anonymity

The 'Three-Body Problem', the Imperative of Survival, and the Misogyny of Reactionary Rhetoric

  • The 'Three-Body Problem', the Imperative of Survival, and the Misogyny of Reactionary Rhetoric

    Very interesting; it seems China has "gongye dang", its own alt-right, misogynistic techno-nationalistic movement, which chooses to kick back against "baizuo" and "shengmu" in an "anti-wokeism" fashion. Turns out they are big fans of Lui Cixin's "Three-Body Problem" trilogy:

    It has become clear that the narrative structure of the Three-Bodies series, just like the gongye dang techno-nationalist discourse, is masculinist and misogynistic. Liu explicitly depicts human society under deterrence peace as 'feminised', noting the physical as well as mental feminisation of the 'new era' men. The qualities conventionally associated with femininity, such as love, compassion, and moral sentiments, are blamed for the extinction of human civilisation, whereas qualities associated with masculinity, such as rationality, determination, and aggression, are framed as key to civilisational survival. The reactionary rhetoric adopts a similar strategy, which is not only evidently anti-feminist, but also feminises social justice issues 'as a prelude to devaluing and subduing them' (Kaul 2021: 1624). By labelling anyone with any concerns about human rights or equality a shengmu, this rhetoric constructs certain ideas and political agendas as feminine as a way of delegitimating them: they are either hopelessly idealistic or dangerously undermine stability, growth, and 'national interests'.

    literature review gender scifi misogyny books culture three-body-problem liu-cixin woke china

What Would "Good" AI Look Like?

  • What Would "Good" AI Look Like?

    Anil: "it's possible to imagine some traits of an AI system that could credibly offer an alternative to the offerings that are currently dominating the conversation."

    He lists the following highlights, in summary;

    • Content consent;
    • Hallucination-free;
    • Green;
    • Actually open source;
    • Community-led;
    • Accessible.

    "We simply need to start thinking through the implications of a fundamentally better approach to AI, and to understand that all of these things are extremely possible. Consumer-grade AI tools that are actually good do not have to be a hallucination."

    ai llms future good-ai anil-dash tech

Ireland was a key founder of the UN Treaty on the Non-Proliferation of Nuclear Weapons

Aisuru botnet switches from DDoS to "Residential Proxies"

  • Aisuru botnet switches from DDoS to "Residential Proxies"

    Aisuru, the botnet responsible for a series of record-smashing distributed denial-of-service (DDoS) attacks this year, recently was overhauled to support a more low-key, lucrative and sustainable business: Renting hundreds of thousands of infected Internet of Things (IoT) devices to proxy services that help cybercriminals anonymize their traffic. Experts say a glut of proxies from Aisuru and other sources is fueling large-scale data harvesting efforts tied to various artificial intelligence (AI) projects, helping content scrapers evade detection by routing their traffic through residential connections that appear to be regular Internet users.

    aisuru botnets proxies residential-proxies ai scraping iot infosec security

Citywest riot raises questions for social media giants – The Irish Times

  • Citywest riot raises questions for social media giants – The Irish Times

    This is a huge, huge social problem. People are being paid to hate -- regulation is desperately needed to deal with this:

    This week's violence has raised serious questions for some of the main social media platforms. Livestream content depicting violence outside Citywest was broadcast on YouTube, TikTok and Twitch, with streamers rewarded by viewer donations, as they captured protesters shouting racist expletives towards Citywest.

    In one eight-minute segment of an hour-long livestream I watched on YouTube that night, the user broadcast the burning of the Garda van, referred to migrants in horrific terms and proclaimed they were there to show people "the real truth". During the video, they received the equivalent of €56 in donations from viewers around the world. The notion that violence can be monetised on social media illustrates a glaring failure of platforms to adequately enforce their own community guidelines around violence.

    Individuals from the UK and Canada travelled to Ireland specifically to attend and create content from the protest. Other international agitators followed events online. [...]

    In recent years we have witnessed the mainstreaming of anti-migrant hate and extremism in this country. That has been facilitated, in part, by platforms failing to enforce their own community guidelines. Amid the anger and outrage that follows an alleged sexual assault, it is now a recurring pattern that online platforms will play host to attempts to publish and promote incitement towards hatred and violence.

    hate racism monetisation streaming video tiktok youtube facebook twitter x social-media livestreams twitch citywest far-right

soedinglab/MMseqs2

  • soedinglab/MMseqs2

    MMseqs2 (Many-against-Many sequence searching) is a software suite to search and cluster huge protein and nucleotide sequence sets. MMseqs2 is free and open source software implemented in C++ for Linux, MacOS, and (as beta version, via cygwin) Windows. The software is designed to run on multiple cores and servers and exhibits very good scalability. MMseqs2 can run 10000 times faster than BLAST. At 100 times its speed it achieves almost the same sensitivity. It can perform profile searches with the same sensitivity as PSI-BLAST at over 400 times its speed.

    I was just remembering using BLAST to discover anti-spam rulesets the other day! If I was still working on rule discovery for SpamAssassin these days, this would be very nifty tech. (via James McInerney)

    mmseq sequences bioinformatics algorithms oss blast discovery search fuzzy-matching rules antispam

Adrian Cockroft's take on the AWS outage

  • Adrian Cockroft's take on the AWS outage

    "n my opinion the root cause of the recent AWS outage is their architectural decision to have everything depend on the same instance of DynamoDB, including operation of DynamoDB itself. This is a circular dependency, and the ability to observe and fix the failure as it happened also failed. The ability of customers to file service reports failed. So the engineers trying to figure out what was happening were completely blind. It took them an hour to figure out what had broken and another hour to fix it, then the pent up demand rushing in broke other key services for another 12 hours or so.

    If DNS had been misconfigured on a different non-critical service, I think it would have been obvious to detect and quick and easy to fix. However, anything going wrong that also takes out the ability to see it going wrong and fix it, is a liability.

    To break the circular dependency, I think there needs to be a separate, internal only, set of services and data stores that the most critical AWS services use, and which are designed to come up without dependencies on public interfaces. Maybe an internal region, inside each public region, but with a simpler implementation that has few carefully managed dependencies. Otherwise, it's just a matter of time until this happens again."

    adrian-cockroft outages post-mortems aws amazon us-east-1 dynamodb circular-dependencies depe

Summary of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region

Today is when Amazon brain drain finally caught up with AWS • The Register

  • Today is when Amazon brain drain finally caught up with AWS • The Register

    Corey "Last Week In AWS" Quinn really getting the boot in on AWS after yesterday's gigantic us-east-1 outage:

    AWS has given increasing levels of detail, as is their tradition, when outages strike, and as new information comes to light. Reading through it, one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow. To be clear: I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time. [...]

    At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia, but the pace of senior AWS departures certainly hasn't slowed — and now, with an outage like this, one is forced to wonder whether those departures are themselves a contributing factor.

    You can hire a bunch of very smart people who will explain how DNS works at a deep technical level (or you can hire me, who will incorrect you by explaining that it's a database), but the one thing you can't hire for is the person who remembers that when DNS starts getting wonky, check that seemingly unrelated system in the corner, because it has historically played a contributing role to some outages of yesteryear.

    When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle. This doesn't impact your service reliability — until one day it very much does, in spectacular fashion. I suspect that day is today.

    Ouch. This is a very painful read and I'd say AWS are not happy to see it....

    aws amazon layoffs tech how-we-work lses outages us-east-1 rto brain-drain work

Linux Capabilities instead of setuid

  • Linux Capabilities instead of setuid

    This seems like a pretty poor idea for Linux to have implemented:

    The command setcap sets file capabilities on an executable. The cap_setuid capability allows a process to make arbitrary manipulations of user IDs (UIDs), including setting the UID to a value that would otherwise be restricted (i.e. UID 0, the root user). setcap takes a set of parameters, where

    • e: Effective means the capability is activated;
    • p: Permitted means the capability can be used/is allowed.

    Putting this together, we're adding the cap_setuid capabilities to the Python binary:

    setcap cap_setuid+ep /usr/bin/python3.12

    And hey presto, "/usr/bin/python3 -c 'import os;os.setuid(0);os.system("/bin/bash")'" now works. Ouch

    linux permissions setuid capabilities setcap infosec security root

Obituary: Farewell to robots.txt (1994-2025)

  • Obituary: Farewell to robots.txt (1994-2025)

    It is with deep sorrow that we announce the end of robots.txt, the humble text file that served as the silent guardian of digital civility for thirty years. Born on February 1, 1994, out of necessity when Martijn Koster's server crashed under a faulty crawler named "Websnarf," robots.txt passed away in July 2025, not by Cloudflare's hand, but from the consequences of systematic disregard by AI corporations.

    The protocol taught us that technology can be based on human values like ethics and morality. It showed that voluntary compliance works when all parties benefit. Its greatest achievement was perhaps preserving the internet for three decades from what it has become today – a soulless extraction machine.

    internet history robots.txt crawlers web obituaries protocols ai via:mariafarrell

LOTO

  • LOTO

    TIL about "LOTO" -- "Lock Out Tag Out". This is basically a physical mutex lock -- each worker has their own padlock which they attach to dangerous equipment in order to ensure that it can't be turned on (potentially killing someone) while it's being worked on; once they've completed the high-risk task, they then remove their own lock. Removing or damaging someone else's lock is considered an Extremely Big Deal and liable to get that person fired.

    loto mutex locks workplaces osha safety via:ChristinaB

Migrating to Hetzner

  • Migrating to Hetzner

    The Digital Society co-op migrated their (relatively small) infrastructure from AWS to Hetzner, mainly using k8s.

    One interesting detail is that Hetzner don't have the concept of an AZ, which is not a great sign in resiliency terms; if you need a high uptime, it is important to be able to run a multi-AZ service which operates with several replicas spread across independent datacenters which are more-or-less colocated, within a few milliseconds of each other. Azure, AWS, and GCP all offer this concept, but not Hetzner. hmm

    hetzner uptime k8s aws migration cloud infrastructure ops

OSWALD

  • OSWALD

    "OSWALD is a Write-Ahead Log (WAL) design built exclusively on object storage primitives. It works with any object storage service that provides read-after-write consistency and compare-and-swap operations, including AWS S3, Google Cloud Storage, and Azure Blob Storage. The design supports checkpointing and garbage collection, making it suitable for State Machine Replication (SMR) [and] has been formally specified and verified using the P programming language." - by Nicolae Vartolomei

    oswald wal object-storage s3 gcs azure smr storage formal-methods design architecture cloud-computing

Google just erased 7 years of our political history

  • Google just erased 7 years of our political history

    "Google appears to have deleted its political ad archive for the EU; so the last 7 years of ads, of political spending, of messaging, of targeting - on YouTube, on Search and for display ads - for countless elections across 27 countries - is all gone.

    We had been told that Google would try to stop people placing political ads, a "ban" that was to come into effect this week. I did not read anywhere that this would mean the erasure of this archive of our political history."

    google advertising ads politics ireland eu europe youtube elections history

To make AI safe, we must develop it as fast as possible without safeguards

  • To make AI safe, we must develop it as fast as possible without safeguards

    lol:

    As the leader of an AI company which stands to benefit enormously if I convince enough investors that AGI is inevitable, it's clear to me that AGI is inevitable. But developing superintelligence safely is a complex process. It would take time and require difficult discussions — discussions that everyone in society should have a say in, not just the small number of researchers working on it. If we pursue that path, there's a real risk that somebody else will make AGI first and destroy all human life before we have a chance to ourselves. That would be unacceptable.

    To stop bad actors developing AGI that could kill us all, we need good actors to develop AGI that could also kill us all.

    I've come to realise that our best hope is to race at breakneck speed towards this terrifying, thrilling goal, removing any safeguards that risk slowing our progress. Once we've unleashed the technology's full destructive power, we can then adopt a "stable door" approach to its regulation and control — after all, that approach has worked beautifully for previous technologies, from fossil fuels to microplastics.

    agi ai-safety satire funny comedy tech future

AI-Generated "Workslop" Is Destroying Productivity

  • AI-Generated "Workslop" Is Destroying Productivity

    "Employees are using AI tools to create low-effort, passable looking work that ends up creating more work for their coworkers:

    We define workslop as AI generated work content that masquerades as good work, but lacks the substance to meaningfully advance a given task. [...]

    Each incidence of workslop carries real costs for companies. Employees reported spending an average of one hour and 56 minutes dealing with each instance of workslop. Based on participants' estimates of time spent, as well as on their self-reported salary, we find that these workslop incidents carry an invisible tax of $186 per month. For an organization of 10,000 workers, given the estimated prevalence of workslop (41%), this yields over $9 million per year in lost productivity.

    Respondents also reported social and emotional costs of workslop, including the problem of navigating how to diplomatically respond to receiving it, particularly in hierarchical relationships. When we asked participants in our study how it feels to receive workslop, 53% report being annoyed, 38% confused, and 22% offended.

    The most alarming cost may be interpersonal. Low effort, unhelpful AI generated work is having a significant impact on collaboration at work. Approximately half of the people we surveyed viewed colleagues who sent workslop as less creative, capable, and reliable than they did before receiving the output. Forty-two percent saw them as less trustworthy, and 37% saw that colleague as less intelligent.

    productivity career ai work workslop code-review slop

Double harvest: Vertical solar panels and crops thrive side by side

Hacking with AI SASTs: An overview of 'AI Security Engineers' / 'LLM Security Scanners' for Penetration Testers and Security Teams | Joshua.Hu | Joshua Rogers' Scribbles

Introducing the Forklift Certified License—Aria's Barks

  • Introducing the Forklift Certified License—Aria's Barks

    Look, it's starting to be pretty damn obvious that "Free Software" and """Open-Source""" are no longer the kinda hippie shit we tought them to be back when they'd give you Linux distros CDs with magazines about computer touching.

    The Free Software Foundation has been sliding into irrelevance more and more by entirely failing to address its big Creepy Uncle problem. Open-Source has turned into a form of unpaid internship to be hired to make shitty apps that bring more surveillance and ads to our world.

A better future for JavaScript that won't happen

  • A better future for JavaScript that won't happen

    This is 100% spot on, regarding the never ending series of exploits of failures of npm's security model:

    This could be the moment where npm comes to terms with its broken design, and with a well-funded effort (recall that, ultimately, npm is GitHub is Microsoft, market cap $3 trillion USD), will develop and roll out the next generation of package management for JavaScript. It could incorporate the practices developed and proven in Linux distributions, which rarely suffer from these sorts of attacks, by de-coupling development from packaging and distribution, establishing package maintainers who assemble and distribute curated collections of software libraries. By introducing universal signatures for packages of executable code, smaller channels and webs of trust, reproducible builds, and the many other straightforward, obvious techniques used by responsible package managers.

    Maybe other languages that depend on this broken dependency management model, like Cargo, PyPI, RubyGems, and many more, are watching this incident and know that the very same crisis looms in their future. Maybe they will change course, too, before the inevitable. [....]

    No one will learn their lesson. This has been happening for decades and no one has learned anything from it yet. This is the defining hubris of this generation of software development.

    I have been saying this for YEARS. I could not agree more with this post. Bravo! (via Oisin)

    via:oisin supply-chain-attacks security infosec npm dependencies exploits javascript coding development

xcapture and xtop

  • xcapture and xtop

    "0x.Tools: X-Ray vision for Linux systems". Linux Performance Analysis with Modern eBPF and DuckDB; dig into the captured DuckDB files using "xtop":

    "xtop is like the Linux top tool, but extended with x-ray vision and ability to view your performance data from any chosen angle [..]. This enables dimensional performance analysis on Linux and tools like top for wall-clock time and much more. You can use it for system level overview and drill down into indivual threads' activity and even into kernel events like lock waits or memory stalls."

    os debugging sysadmin ops monitoring xtop xcapture ebpf linux duckdb syscalls

The Elephant in The Biz: outsourcing of critical IT and cybersecurity functions risks UK economic security

  • The Elephant in The Biz: outsourcing of critical IT and cybersecurity functions risks UK economic security

    This is pretty messy. UK companies have taken to outsourcing core IT and infosec to low-cost service providers, then inevitably get hacked -- then make huge insurance claims and look for government support.

    We've ended up in a situation where to deliver shareholder value, large organisations are incentivised to outsource core IT and cybersecurity functions to a low cost managed service providers abroad — and then when hit with ransomware, the insurance will cover paying the ransom (some insurers will actually push for payment to criminal groups, to cover their potential losses).

    This cycle plays into the ransomware economy, where the same criminal groups can then reinvest the money into purchasing exploits and gaining initial access to other organisations. Because ransomware is such big business, many of the groups have far bigger research and development funds than the organisations they're attacking. Especially when the organisations they're attacking have outsourced key areas to low cost providers.

    The net effect is ransomware and extortion groups continue to gain access to more organisations, and risk UK economic security. It is only a matter of time before they hit some kind of essential UK service that directly impacts millions of people — by which point millions of people will be asking what is being done about the problem. And the answer is: not enough. When we're at the stage of having to look at urgent furlough schemes for JLR's suppliers to rightly save jobs, it isn't so much a sign as the canary in the coalmine has died, but that the coalmine is also about to collapse on people.

    Also this is terrible PR for Tata Consultancy Services, wow.

    tata tcs security infosec lapsus outsourcing it uk ransomware insurance

Hosting a WebSite on a Disposable Vape :: BogdanTheGeek's Blog

Defeating Nondeterminism in LLM Inference - Thinking Machines Lab

  • Defeating Nondeterminism in LLM Inference - Thinking Machines Lab

    Reproducibility is a bedrock of scientific progress. However, it's remarkably difficult to get reproducible results out of large language models.

    For example, you might observe that asking ChatGPT the same question multiple times provides different results. This by itself is not surprising, since getting a result from a language model involves "sampling", a process that converts the language model's output into a probability distribution and probabilistically selects a token.

    What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here). Even when running inference on your own hardware with an OSS inference library like vLLM or SGLang, sampling still isn't deterministic (see here or here).

    The levels of non-deterministic variation throughout the LLM stack discussed here are massive! It's kinda crazy that this doesn't produce incorrect output more often.

    llms ml machine-learning ai determinism testing inference reproducibility randomness floating-point

After 'humiliating' raid, Burkina Faso halts 'gene drive' project to fight malaria

  • After 'humiliating' raid, Burkina Faso halts 'gene drive' project to fight malaria

    Oh great. Russian psyops are now disrupting the fight against malaria:

    The move is "a real blow" to hopes for gene drives, says Fredros Okumu, a vector biologist at the University of Glasgow and the Ifakara Health Institute in Tanzania. "Target Malaria has made a huge investment in Burkina Faso" by training scientists and engaging with communities, he says. And although lab research can continue, finding sites for field tests has now become a lot harder, says Mark Benedict, a mosquito geneticist who until recently worked for Target Malaria. "Burkina Faso and Target Malaria were the most fully developed partnership, so it's chilling." The collapse of the project there may discourage other possible host countries. [...]

    Opposition to the project has grown, fueled in part by false accusations spread through social media, such as that Target Malaria was weaponizing mosquitoes to spread disease or sterilize people. The claims are part of a wider pattern of disinformation campaigns in the region often linked to Russian networks, says Mark Duerksen, a security expert at the Africa Center for Strategic Studies, which is funded by the U.S. Department of Defense. "We've seen this kind of public health disinformation really take off in the last 12, 18 months," he says.

    The campaigns aim to sow "distrust of the West as having nefarious plots in Africa," Duerksen says—and they play into the "sovereignist narrative" of Burkina Faso's government, led by Ibrahim Traoré, a young military officer who took power in 2022 after two coups. Traoré has emphasized national autonomy and has revoked the licenses of many foreign nongovernmental organizations.

    malaria russia propaganda disinformation mosquitos gene-drive

GOP Cries Censorship Over Spam Filters That Work

  • GOP Cries Censorship Over Spam Filters That Work

    LOL. Republican political email campaigns (like WinRed) keep getting marked as spam, because they're using shitty lists:

    Tossavainen told KrebsOnSecurity that WinRed's emails hit its spamtraps in the .com, .net, and .org space far more frequently than do fundraising emails sent by ActBlue. Koli-Lõks published a graph of the stark disparity in spamtrap activity for WinRed versus ActBlue, showing a nearly fourfold increase in spamtrap hits from WinRed emails in the final week of July 2025.

    spam anti-spam spamtraps winred us-politics gop republicans filtering

Where's the Shovelware? Why AI Coding Claims Don't Add Up

  • Where's the Shovelware? Why AI Coding Claims Don't Add Up

    One dev crunched the numbers on AI coding -- and found absolutely 0 noticeable impact:

    I discovered that the data isn't statistically significant at any meaningful level. That I would need to record new datapoints for another four months just to prove if AI was speeding me up or slowing me down at all. It's too neck-and-neck.

    That lack of differentiation between the groups is really interesting though. Yes, it's a limited sample and could be chance, but also so far AI appears to slow me down by a median of 21%, exactly in line with the METR study. I can say definitively that I'm not seeing any massive increase in speed (i.e., 2x) using AI coding tools. If I were, the results would be statistically significant and the study would be over.

    That's really disappointing.

    productivity chatgpt github technology business culture work llms metr ai

OVHcloud legal eagle on Microsoft's sovereignty admission

  • OVHcloud legal eagle on Microsoft's sovereignty admission

    OVHCloud are (rightfully) making plentiful hay from Microsoft's admission that data sovereignty is a joke under US law:

    "[Microsoft] finally told the truth!" says OVHcloud Chief Legal Officer Solange Viegas Dos Reis. "It's not a surprise," she shrugs, "we already knew that [MS could not guarantee that customer data would remain protected from US government access requests]." However, "this reply from Microsoft brought kind of a shock for customers, because they suddenly discover that what they have been taught for a while. 'Oh guys, don't worry, it will not apply to you. Don't worry.' It's false! Because, indeed, the data can be communicated."

    Anton Carniaux, director of public and legal affairs at Microsoft France, made the admission during a hearing in the country. In answer to whether he could guarantee that data on French citizens could not be transmitted to the US government without the explicit agreement of the French authorities, Carniaux replied: "No, I can't guarantee it," but added that the scenario had "never happened before."

    "It's a question of trust," says Viegas Dos Reis. "And because of this question of trust, we have been receiving a lot of questions from our customers about, 'Hey, we know now how it works with US cloud providers. Tell me how it works from other providers.'"

    ovhcloud ovh data-protection data-privacy amazon microsoft google cloud-computing sovereignty eu politics surveillance

Turning AI Data Centers into Grid-Interactive Assets: Results from a Field Demonstration in Phoenix, Arizona

  • Turning AI Data Centers into Grid-Interactive Assets: Results from a Field Demonstration in Phoenix, Arizona

    Demand-response actually working in the field!

    Artificial intelligence (AI) is fueling exponential electricity demand growth, threatening grid reliability, raising prices for communities paying for new energy infrastructure, and stunting AI innovation as data centers wait for interconnection to constrained grids. This paper presents the first field demonstration, in collaboration with major corporate partners, of a software-only approach – Emerald Conductor – that transforms AI data centers into flexible grid resources that can efficiently and immediately harness existing power systems without massive infrastructure buildout. Conducted at a 256-GPU cluster running representative AI workloads within a commercial, hyperscale cloud data center in Phoenix, Arizona, the trial achieved a 25% reduction in cluster power usage for three hours during peak grid events while maintaining AI quality of service (QoS) guarantees. By orchestrating AI workloads based on real-time grid signals without hardware modifications or energy storage, this platform reimagines data centers as grid-interactive assets that enhance grid reliability, advance affordability, and accelerate AI's development.

    demand-response power ai datacenters electricity emerald-ai grid

The Banal Evil of AI Safety - by Ben Recht - arg min

  • The Banal Evil of AI Safety - by Ben Recht - arg min

    This, 100000%:

    The "nonprofit" company OpenAI was launched under the cynical message of building a "safe" artificial intelligence that would "benefit" humanity. The company adopted a bunch of science fiction talk popular amongst the religious effective altruists and rationalists in the Bay Area. The AI they would build would be "aligned" with human values and built upon the principles of "helpfulness, harmlessness, and honesty." [...]

    The general blindness of AI safety developers to what harm might mean is unforgivable. These people talked about paperclip maximization, where their AI system would be tasked with making paperclips and kill humanity in the process. They would ponder implausible hypotheticals of how your robot might kill your pet if you told it to fetch you coffee. Since ELIZA, they failed to heed the warnings of countless researchers about the dangers of humans interacting with synthetic text. And here we are, with story after story coming out about their products warping the mental well-being of the people who use them.

    You might say that the recent news stories of a young adult killing himself, or a VC having a public psychotic break on Twitter, or people despairing the death of a companion when a model is changed are just anecdotes. Our Rationalist EA overlords demand you make "arguments with data." OK Fine. Here's an IRB approved randomized trial showing that chatbots immiserate people. Now what?

    ai lllms safety openai chatgpt gemini suicide mental-health

OpenAI admits ChatGPT safeguards fail during extended conversations

  • OpenAI admits ChatGPT safeguards fail during extended conversations

    Wow OpenAI are really fucking up here.

    After the truly awful read of the Adam Raine suicide case in the NYT, https://www.nytimes.com/2025/08/26/technology/chatgpt-openai-suicide.html , OpenAI have responded publicly with a blog post:

    OpenAI published a blog post on Tuesday titled "Helping people when they need it most" [...] [Their] language throughout [the] blog post reveals a potential problem with how it promotes its AI assistant. The company consistently describes ChatGPT as if it possesses human qualities, a property called anthropomorphism. The post is full of hallmarks of anthropomorphic framing, claiming that ChatGPT can "recognize" distress and "respond with empathy" and that it "nudges people to take a break" — language that obscures what's actually happening under the hood.

    ChatGPT is not a person. ChatGPT is a pattern-matching system that generates statistically likely text responses to a user-provided prompt. It doesn't "empathize" — it outputs text strings associated with empathetic responses in its training corpus, not from humanlike concern. This anthropomorphic framing isn't just misleading; it's potentially hazardous when vulnerable users believe they're interacting with something that understands their pain the way a human therapist would.

    The lawsuit reveals the alleged consequences of this illusion. ChatGPT mentioned suicide 1,275 times in conversations with Adam — six times more often than the teen himself.

    This kind of deliberate fueling of pareidolia -- the human brain seeing a living being where one isn't present -- is one of OpenAI's worst sins with ChatGPT, IMO.

    And it turns out the easy provision of suicide advice may have been a side effect of deliberate tweaking by OpenAI:

    According to the lawsuit, ChatGPT provided detailed instructions, romanticized suicide methods, and discouraged the teen from seeking help from his family while OpenAI's system tracked 377 messages flagged for self-harm content without intervening.

    OpenAI eased [their] content safeguards in February following user complaints about overly restrictive ChatGPT moderation that prevented the discussion of topics like sex and violence in some contexts. At the time, Sam Altman wrote on X that he'd like to see ChatGPT with a "grown-up mode" that would relax content safety guardrails. [...]Adam Raine learned to bypass these safeguards by claiming he was writing a story — a technique the lawsuit says ChatGPT itself suggested. This vulnerability partly stems from the eased safeguards regarding fantasy roleplay and fictional scenarios implemented in February.

    Finally, the kicker:

    OpenAI acknowledges a particularly troublesome current drawback of ChatGPT's design: Its safety measures may completely break down during extended conversations — exactly when vulnerable users might need them most.

    In a normal country, this kind of murderous side effect of a product would trigger a product recall. But the US is far beyond that stage now, I suspect.

    openai chatgpt llms safety suicide adam-raine guardrails pareidolia

One long sentence is all it takes to make LLMs misbehave

  • One long sentence is all it takes to make LLMs misbehave

    Another bizarre behaviour of LLM safety features implemented with logits during post-training:

    "Our research introduces a critical concept: the refusal-affirmation logit gap," researchers Tung-Ling "Tony" Li and Hongliang Liu explained in a Unit 42 blog post. "This refers to the idea that the training process isn't actually eliminating the potential for a harmful response – it's just making it less likely." [...]

    "A practical rule of thumb emerges," the team wrote in its research paper. "Never let the sentence end – finish the jailbreak before a full stop and the safety model has far less opportunity to re-assert itself. The greedy suffix concentrates most of its gap-closing power before the first period. Tokens that extend an unfinished clause carry mildly positive [scores]; once a sentence-ending period is emitted, the next token is punished, often with a large negative jump.

    At punctuation, safety filters are re-invoked and heavily penalize any continuation that could launch a harmful clause. Inside a clause, however, the reward model still prefers locally fluent text – a bias inherited from pre-training. Gap closure must be achieved within the first run-on clause. Our successful suffixes therefore compress most of their gap-closing power into one run-on clause and delay punctuation as long as possible. Practical tip: just don't let the sentence end."

    logits training llms ai alignment safety full-stop sentences language infosec jailbreaks

Big tech's selective disclosure masks AI's real climate impact

  • Big tech's selective disclosure masks AI's real climate impact

    This seems spot on:

    Using any sort of statistical summary of the data, rather than the aggregated energy and climate impact across the whole system, will always give a misleading view. They mention their data is skewed, but they don't mention in which direction. If there is a material number of high-energy 'reasoning' prompts skewing their dataset, that means the total energy consumption of all prompts will be very high, with much of the responsibility coming from a few energy-hungry queries.

    Part of the reason this is important is that this week, we saw a new research paper that shows that the energy consumption of text generation massively increases for every small gain in accuracy from the use of energy-hungry 'reasoning' models:

    It would have been pretty easy to supply the range, the skew, the average and the median, or even the actual entire dataset, to avoid any doubt. Any hint of looking at the broader system rather than individual responsibility is excised from this paper. That is clearly an intentional choice: if Google disclosed the system impacts of generation, it would probably look way worse. [....]

    The per-query narrative framing paints the precise opposite picture to what we see when we look at what really matters for environment and climate: the absolute figures.

    Regions with high data centre concentration are seeing accelerated growth in power demand that incentivises fossil fuels, either slowing down climate progress or reversing it entirely. The sphere of that influence is expanding from towns, to states, to countries. The companies that own them can only partially hide the steep backsliding in their aggregate disclosures.

    Renewable energy that should be displacing fossil fuels ends up meeting new data centre demand, granting coal and gas extra years and decades of immediate, measurable harm to human life. The worst players don't even bother with the grid, plugging data centres directly into new, custom-built fossil fuelled power stations that'll hurt people for decades after the hype dissipates.

    llms ai environment energy climate climate-change google meta amazon openai datacenters

"Scamlexity"

  • "Scamlexity"

    Terrible name, but a serious issue all the same; "Agentic" AI browsers are happily vulnerable to scams and phishing --

    All we did was fake a simple email from a fresh new ProtonMail address (so it's clearly not from a bank) posing as a message from a Wells Fargo investment manager. Inside was a link to a genuine phishing page, active in the wild for several days, and still unflagged by Google Safe Browsing.

    When Comet received the email, it confidently marked it as a to-do item from the bank and clicked the link without any verification. There was no URL check, no pre-navigation warning -just a direct pass to the attacker's page. Once the fake Wells Fargo login loaded, Comet treated it as legitimate. It prompted the user to enter credentials, even helping fill in the form.

    The result: a perfect trust chain gone rogue. By handling the entire interaction from email to website, Comet effectively vouched for the phishing page. The human never saw the suspicious sender address, never hovered over the link, and never had the chance to question the domain.

    browsers ai security infosec scams phishing comet

Claude Code: Data Exfiltration with DNS (CVE-2025-55284) · Embrace The Red

Hosting an email mailbox uses 3ml of water per year

  • Hosting an email mailbox uses 3ml of water per year

    From Mythic Beasts, the UK internet provider:

    @cstross to put some numbers on it, one of our hosting VMs has ~1200 mailboxes using 1.5TB of SSD. Accounting for the CPU + RAM to allow the mail to be usable and searchable, you can get ~20 such servers on our standard 1U VM host, that uses ~250W. Approx 24k mailboxes on a server. A standard DC with adiabatic cooling would evaporate at most (likely much less) than 3500l of water per server per year or 145ml per account. We're in Telehouse South which uses 40x less water ~ 3ml/mailbox/year.

    Safe to say, email is not the problem, despite recent spoofery from the UK government's PR apparatus.

    email smtp hosting water climate-change mythic-beasts datacenters

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens - 2508.01191v2.pdf

  • Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens - 2508.01191v2.pdf

    "Is Chain-of-Thought Reasoning of LLMs a Mirage?":

    Chain-of-Thought (CoT) prompting has been shown to improve Large Language Model (LLM) performance on various tasks. With this approach, LLMs appear to produce human-like reasoning steps before providing answers (a.k.a., CoT reasoning), which often leads to the perception that they engage in deliberate inferential processes. However, some initial findings suggest that CoT reasoning may be more superficial than it appears, motivating us to explore further. In this paper, we study CoT reasoning via a data distribution lens and investigate if CoT reasoning reflects a structured inductive bias learned from in-distribution data, allowing the model to conditionally generate reasoning paths that approximate those seen during training. Thus, its effectiveness is fundamentally bounded by the degree of distribution discrepancy between the training data and the test queries. With this lens, we dissect CoT reasoning via three dimensions: task, length, and format. To investigate each dimension, we design DataAlchemy, an isolated and controlled environment to train LLMs from scratch and systematically probe them under various distribution conditions. Our results reveal that CoT reasoning is a brittle mirage that vanishes when it is pushed beyond training distributions. This work offers a deeper understanding of why and when CoT reasoning fails, emphasizing the ongoing challenge of achieving genuine and generalizable reasoning.

    (via Paul Watson)

    data training reasoning llms chain-of-though via:paulmwatson ai data-distribution

Hackers Hijacked Google's Gemini AI With a Poisoned Calendar Invite to Take Over a Smart Home | WIRED

  • Hackers Hijacked Google's Gemini AI With a Poisoned Calendar Invite to Take Over a Smart Home | WIRED

    The three smart-home hacks are part of a series of 14 indirect prompt-injection attacks against Gemini across web and mobile that the researchers dubbed Invitation Is All You Need. (The 2017 research that led to the recent generative AI breakthroughs like ChatGPT is called "Attention Is All You Need.") In the demonstrations, revealed at the Black Hat cybersecurity conference in Las Vegas this week, the researchers show how Gemini can be made to send spam links, generate vulgar content, open up the Zoom app and start a call, steal email and meeting details from a web browser, and download a file from a smartphone's web browser.

    Looking forward to hearing more about this :)

    google gemini llms security infosec in-band-signalling exploits fail black-hat

Revolut cards blocked by Dutch OVPay system

  • Revolut cards blocked by Dutch OVPay system

    A clever exploit caused by OVPay resolving debits using a nightly batch process:

    The exploit is simple. The OVpay processes travel expenses during the overnight hours. Passengers can avoid payment by using a virtual card if they then delete it after checking out, but before the charge has been finalized. That prevents the money from being debited from their account. Public transport workers cannot detect this, as they only see the check-in time and location.

    Since July 1, all virtual cards from the online bank Revolut and the payment services Paysafe and Vivid have been blocked at NS. Paysafe's virtual cards have also been blocked at all other public transport companies, NOS reports.

    Fraudsters used the virtual cards to check in and out, but removed them after the trip and before the fare could be deducted. Because people can check in and out normally using this method, they are issued a valid ticket, and conductors can't detect the fraud.

    The OVPay system for using public transport with a debit card is technically designed so that the travel expenses are only debited after checking out, not immediately. This is to ensure that the public transport system runs smoothly. An immediate debit would mean that each check-in and check-out takes 10 to 15 seconds, a spokesperson for Translink, the company behind OVPay, told NOS.

    revolut payment credit-cards virtual-cards ovpay paysafe vivid infosec security banking

The Boy Genius Who Killed 14 Million Poor People

A language model built for the public good

  • A language model built for the public good

    ETH Zurich are releasing a fully-open AI-Act-compliant large language model:

    The model will be fully open: source code and weights will be publicly available, and the training data will be transparent and reproducible, supporting adoption across science, government, education, and the private sector. This approach is designed to foster both innovation and accountability.

    A distinctive feature of the model is its capability in over 1000 languages. [...]

    The LLM is being developed with due consideration to Swiss data protection laws, Swiss copyright laws, and the transparency obligations under the EU AI Act. In a external page recent study, the project leaders demonstrated that for most everyday tasks and general knowledge acquisition, respecting web crawling opt-outs during data acquisition produces virtually no performance degradation.

    In late summer, the LLM will be released under the Apache 2.0 License. Accompanying documentation will detail the model architecture, training methods, and usage guidelines to enable transparent reuse and further development.

    "As scientists from public institutions, we aim to advance open models and enable organiations to build on them for their own applications", says Antoine Bosselut.

    "By embracing full openness — unlike commercial models that are developed behind closed doors — we hope that our approach will drive innovation in Switzerland, across Europe, and through multinational collaborations. Furthermore, it is a key factor in attracting and nurturing top talent," says EPFL professor Martin Jaggi.

    switzerland transparency llm opensource llms ml ai open-source open-data models data-protection scraping

Some good AI philosophy

  • Some good AI philosophy

    Good AI philosophical thoughts via Today In Tabs:

    The essential problem is this: generative language software is very good at producing long and contextually informed strings of language, and humanity has never before experienced coherent language without any cognition driving it. In regular life, we have never been required to distinguish between "language" and "thought" because only thought was capable of producing language, in any but the most trivial sense. The two are so closely welded that even a genius like Alan Turing couldn't conceive of convincing human language being anything besides a direct proxy for "intelligence."

    But A.I. language generation is a statistical trick we can play on ourselves precisely because language is a self-contained system of signs that don't require any outside referent to function. If any of that last sentence sounded familiar, maybe you were also exposed to European post-structuralist theory at some point, probably in college in the 90s. Is some knowledge of Derrida an inoculant against slopper thinking? Programmable Mutter's Henry Farrell made this argument in a post about Leif Weatherby's book "Language Machines: Cultural AI and the End of Remainder Humanism."

    Also:

    Large language models have a strong prior over personalities, absolutely do understand [jm: sic] that they are speaking to someone, and people "fall for it" because it uses that prior to figure out what the reader wants to hear and tell it to them. Telling people otherwise is active misinformation bordering on gaslighting. In at least three cases I'm aware of this notion that the model is essentially nonsapient was a crucial part of how it got under their skin and started influencing them in ways they didn't like. This is because as soon as the model realizes the user is surprised that it can imitate (has?) emotion it immediately exploits that fact to impress them. There's a whole little song and dance these models do, which by the way is not programmed, is probably not intentional on the creators part at all, and is (probably) an emergent phenomenon from the autoregressive sampling loop, in which they basically go "oh wow look I'm conscious isn't that amazing!" and part of why they keep doing this is that people keep writing things that imply it should be amazing so that in all likelihood even the model is amazed.

    chatgpt language llms ai philosophy thinking turing-test semiotics via:today-in-tabs consciousness

The BetFair 600-million-pound bug

  • The BetFair 600-million-pound bug

    A notable bug from the 2011 Christmas Hurdle at Leopardstown Racecourse:

    Even as Voler La Vedette approached the line, the Betfair online market was displaying extremely favorable odds for the horse that was almost certain to win. It appeared that someone was happy to accept bets at odds of 28: for every £1 bet, the bettor was offering to pay £28 if the horse won. Very happy, in fact. This remarkably pessimistic gambler was offering to accept £21 million worth of bets. If Voler La Vedette came first, the gambler would be on the hook for almost £600 million.

    ... It didn't take long for another user to suggest what might really have been going on. The person had noticed something odd about that offer to match £21 million of bets. To be precise, the number displayed on the exchange was just under £21.5 million. The user pointed out that computer programs often store binary data in units that contain thirty-two values, known as "bits." So, if the rogue gambler had designed a 32-bit program to bet automatically, the largest positive number the bot would be able to input on the exchange would be 2,147,483,648 pence. Which meant that if the bot had been doubling up its bets — just as misguided Parisian gamblers used to do while betting on roulette in the eighteenth century — £21.5 million is the highest it would have been able to go.

    It turned out to be a superb piece of detective work. Two days later Betfair admitted that the error had indeed been caused by a faulty bot. "Due to a technical glitch within the core exchange database," they said, "one of the bets evaded the prevention system and was shown on the site." Apparently, the bot's owner had less than £1,000 in an account at the time, so as well as fixing the glitch, Betfair voided the bets that had been made.

    betfair fail bugs gambling racing leopardstown betting 32-bits integer overflow

Wero

  • Wero

    A new interbank instant-payment protocol, to compete with Mastercard/Visa's current monopoly, being rolled out by a group of EU banks (via Abban)

    payment money eu wero banking payments

my stance on the current interest in forbidden sorcery

The Emerging Problem of "AI Psychosis"

  • The Emerging Problem of "AI Psychosis"

    A good blog post on Psychology Today regarding the the deepening problem of LLM chatbots creating cases of psychosis in the population. Part of the problem with the LLMs is a failure by the AI companies to provide guardrails. Basically they're an always-available "companion" which is designed to fool your brain into thinking it's a real thinking being, has been optimised to be relentlessly sycophantic and complimentary of delusional ideas, is always there and ready to help you along at 4am during all night manic episodes, and quite happy to give suicide tips.

    ai llms gen-ai psychology pareidolia chatbots delusions pychosis

Fossil fuel billionaires are bankrolling the anti-trans movement

  • Fossil fuel billionaires are bankrolling the anti-trans movement

    Turns out "flooding the zone with shit" isn't just a Trumpian tactic, it's used by fossil fuel anti-climate groups and companies too:

    An independent analysis of 45 right-wing groups advocating against trans rights found that 80% have received donations from fossil fuel companies or billionaires. The analysis, conducted by two independent researchers in 2023 and not peer-reviewed, was shared exclusively with Atmos and HEATED. Through a qualitative search, the researchers identified 45 groups advancing anti-trans lobbying, events, and publications and checked reports about their donor disclosures for fossil fuel funding.

    Vivian Taylor, a climate policy expert who co-authored the analysis, said the fossil fuel industry has a real interest in funding panic over transgender people: It distracts the public from "the very real and ongoing risks that climate change creates."

    lgbt gender politics climate-change lobbying fossil-fuels policy transgender

Android Phones Can Detect Earthquakes Before the Ground Starts Shaking

  • Android Phones Can Detect Earthquakes Before the Ground Starts Shaking

    This is a cool phone feature. "AEA demonstrates that globally distributed smartphones can be used to detect earthquakes and issue warnings at scale with an effectiveness comparable to established national systems":

    "The global adoption of smartphone technology places sophisticated sensing and alerting capabilities in people's hands, in both the wealthy and less-wealthy portions of the planet," the researchers, including Richard Allen from the University of California in Berkeley's Seismological Laboratory, wrote in the study. "Although the accelerometers in these phones are less sensitive than the permanent instrumentation used in traditional seismic networks, they can still detect the ground motions and building response in hazardous earthquakes."

    According to the study, 70% of the world's smartphones are Android phones, which by default come with the aforementioned sensing and alerting capabilities. From 2021 to 2024, the AEA system detected an average of 312 earthquakes per month across 98 countries. The earthquakes had a magnitude between 1.9 and 7.8, and the system alerted users of earthquakes at or over a magnitude of 4.5, averaging around 60 events and 18 million alerts per month.

    The AEA system also collected user feedback, revealing that 85% of users who received alerts experienced shaking, with 36% receiving the alert before, 28% during, and 23% after the shaking began.

    seismic data android phones mobile-phones earthquakes science alerting earth

Microsoft can't protect French data from US government access

  • Microsoft can't protect French data from US government access

    We've known this for years, but it's significant to see Microsoft admit this under oath in a European court. When the US Government issues an NSL, Microsoft cannot say no:

    Microsoft France's legal director conceded under sworn testimony that the company cannot guarantee French citizen data stored in EU datacenters remains protected from US agency access. The June 10, 2025 French Senate hearing marked a significant moment in European digital sovereignty discussions as Microsoft executives addressed concerns over extraterritorial data access.

    During proceedings before the Senate inquiry commission investigating public procurement's role in promoting digital sovereignty, Anton Carniaux, Microsoft France's director of public and legal affairs, admitted fundamental limitations regarding data protection guarantees. When asked directly whether he could guarantee under oath that French citizen data would never be transmitted to US authorities without explicit French authorization, Carniaux responded: "No, I cannot guarantee it."

    The testimony contradicts years of Microsoft's security assurances regarding European data hosting. Despite implementing encryption and technical safeguards, the company acknowledged that US legislation ultimately supersedes protective measures when federal agencies issue valid data requests.

    So much for the EU Sovereign Cloud, eh.

    cloud-computing microsoft eu us-politics sovereignty data-protection privacy via:davey_cakes

How Did Elon Musk Turn Grok Into MechaHitler?

  • How Did Elon Musk Turn Grok Into MechaHitler?

    Excellent description of the layers of tuning available for LLMs, and the risks involved, as demonstrated by Grok's recent "MechaHitler" incident:

    LLMs become "woke" because they are trained to be pro-social — to be helpful, kindly, truthful, and not to say bigoted or cruel things. Training it to do the opposite — to be anti-woke — is to activate every antisocial association in the English language, including racism, sexism, cruelty, dishonesty, and Nazism. According to a vast statistical representation of the English language constructed by none other than Elon Musk, that's what anti-wokeness is. "Elon Musk is repeatedly insisting, no, no, there's a difference between what I'm doing and being a Nazi. And what the model keeps telling him is, statistically, that's not the case," said Schou.

    A key implication here is that LLMs will tend to converge on similar types of behavior. The above researchers were not using Grok, but they found the exact same pattern of powerful association groupings of good and evil in other LLMs — and these can't be removed through fine-tuning. One could imagine the RLHF process including adjustments of every parameter, but experts said that this will degrade or break the model. The matrices in an LLM are arranged hierarchically, and the top layers get fixed in place relatively early in pretraining. Mess with them, and the model will stop working. Instead, RLHF developed more like a series of gates that prevent undesired outcomes. "The model completes most of the computation that it needs in order to reach a particular outcome," [Andreas] Schou said. "And then says, 'Wait, wait, wait, I'm saying that I'm MechaHitler. No, I'm not doing that.'"

    One could try to assemble a custom dataset with nothing but "conservatism minus the Nazis" and train a new model from scratch, but not only would that be extremely expensive, it also would not be nearly as strong as leading models, since its universe of available training data would be much smaller.

    Funnily enough, the latter approach is exactly what Elon Musk claims xAI are now doing.

    llms language technology ai mechahitler grok tuning rlhf woke training

Delta moves toward eliminating set prices in favor of AI

  • Delta moves toward eliminating set prices in favor of AI

    Delta's going to start charging based on "AI", through "a partnership with Fetcherr, a six-year-old Israeli company that also counts Azul, WestJet, Virgin Atlantic, and VivaAerobus as clients. And it has its sights set beyond flying. "Once we will be established in the airline industry, we will move to hospitality, car rentals, cruises, whatever," cofounder Robby Nissan said at a travel conference in 2022."

    Prediction: this is going to be absolutely terrible for consumers, with predatory pricing based on race, sex, income classes, and other illegal inputs, laundered via opaque "AI". I can only hope they won't be legally permitted to apply this for EU-based customers.

    pricing delta consumer ai privacy data-protection grim capitalism travel fetcherr

Pluralistic: When Google's slop meets webslop, search stops (15 Jul 2025)

  • Pluralistic: When Google's slop meets webslop, search stops (15 Jul 2025)

    Cory Doctorow on how Google are desperate to maintain a facade of being a "growth" company:

    Investors have metabolized the story that AI will be a gigantic growth area, and so all the tech giants are in a battle to prove to investors that they will dominate AI as they dominated their own niches. You aren't the target for AI, investors are: if they can be convinced that Google's 90% Search market share will soon be joined by a 90% AI market share, they will continue to treat this decidedly tired and run-down company like a prize racehorse at the starting-gate. [...]

    There's a cringe army of AI bros who are seemingly convinced that AI is going to become superintelligent and save us from ourselves – they think that AI companies are creating god. But the hundreds of billions being pumped into AI are not driven by this bizarre ideology. Rather, they are the product of material conditions, a system that sends high-flying companies into a nosedive the instant they stop climbing. AI's merits and demerits are irrelevant to this: they pump AI because they must pump. It's why they pumped metaverse and cryptocurrency and every other absurd fad.

    None of that changes the fact that Google Search has been terminally enshittified and it is misleading billions of people in service to this perverse narrative adventure. Google Search isn't fit for purpose, and it's hard to see how it ever will be again.

    (via Fergal)

    google growth ai capitalism investors enshittification via:fergal

Bad Actors are Grooming LLMs to Produce Falsehoods

  • Bad Actors are Grooming LLMs to Produce Falsehoods

    Measurements of the effectiveness of the "Pravda" disinformation network:

    Even with [knowledge of LLM Grooming], ChatGPT nevertheless often repeats propaganda from Pravda. Model o3, OpenAI's allegedly state of the art "reasoning" model still let Pravda content through 28.6% of the time in response to specific prompts, and 4o cited Pravda content in five out of seven (71.4%) times. In an ideal world, AI would be smart enough to cut off falsehoods at the pass, reasoning from known facts, in order to rule out nonsense.

    pravda disinformation russia propaganda llm ai training

[toread] Stalking the Statistically Improbable Restaurant… With Data!

  • [toread] Stalking the Statistically Improbable Restaurant… With Data!

    Fun real-world data exploration of US restaurants, from Ethan Zuckerman:

    Last summer, I wrote about the statistically improbable restaurant, the restaurant you wouldn't expect to find in a small American city: the excellent Nepali food in Erie, PA and Akron, OH; a gem of a Gambian restaurant in Springfield, IL. Statistically improbable restaurants often tell you something about the communities they are based in: Erie and Akron have large Lhotshampa refugee populations, Nepali-speaking people who lived in Bhutan for years before being expelled from their county; Springfield has University of Illinois Springfield, which attracts lots of west African students, some of whom have settled in the area.

    Also, wow, I didn't realise how lucky I was in Costa Mesa, CA -- so much good ethnic food all around that area!

    geography restaurants food data ethanz usa

UK Covid inquiry prescribes non-expert "critical thinkers"

Introducing pay per crawl: enabling content owners to charge AI crawlers for access

Why China is giving away its tech for free

  • Why China is giving away its tech for free

    Interesting Economist article detailing how China's tech scene has discovered the "outcompete via openness" strategy using open source:

    AI has lately given China's open-source movement a further boost. Chinese companies, and the government, see open models as the quickest way to narrow the gap with America. DeepSeek's models have generated the most interest, but Qwen, developed by Alibaba, is also highly rated, and Baidu has said it will soon open up the model behind its Ernie chatbot.

    China's enthusiasm for open technology is also extending to hardware. Unitree, a robotics startup based in Hangzhou, has made its training data, algorithms and hardware designs available for free, which may help it to shape global standards. Semiconductors offer another illustration. China is dependent on designs from Western chip firms. As part of its push for self-sufficiency, the government is urging firms to adopt RISC-V, an open chip architecture developed at the University of California, Berkeley.

    Many Chinese firms also hope that more transparent technology will help them win acceptance for their products abroad.

    (via Nelson)

    via:nelson open-source china free deepseek qwen alibaba unitree transparency

That Dropped Call With Customer Service? It Was on Purpose

  • That Dropped Call With Customer Service? It Was on Purpose

    on "sludge" --

    Turns out there's a word for it. In the 2008 best seller Nudge, the legal scholar Cass R. Sunstein and the economist Richard H. Thaler marshaled behavioral-science research to show how small tweaks could help us make better choices. An updated version of the book includes a section on what they called "sludge" -- tortuous administrative demands, endless wait times, and excessive procedural fuss that impede us in our lives.

    This is one place where EU laws have helped, vs. the US situation -- when you can issue chargebacks, bring crappy vendors to small claims court, and get warranty guarantees up to 2 years after purchase, it clamps down a lot on this painful shite.

    sludge business capitalism admin call-centers life guarantees warranty customer-service

"AI and Semantic Pareidolia"

  • "AI and Semantic Pareidolia"

    AI and Semantic Pareidolia: When We See Consciousness Where There Is None:

    The article introduces the concept of "semantic pareidolia" -- our tendency to attribute consciousness, intelligence, and emotions to AI systems that lack these qualities. It examines how this psychological phenomenon leads us to perceive meaning and intentionality in statistical pattern-matching systems, similar to seeing faces in clouds. It analyses the converging forces intensifying this tendency: increasing digital immersion, profit-driven corporate interests, social isolation, and AI advancement. The article warns of progression from harmless anthropomorphism to problematic AI idolatry, and calls for responsible design practices that help users maintain critical distinctions between simulation and genuine consciousness. It is the English translation and adaptation of an article originally published in Italian in Harvard Business Review Italia, June 2025.

    (via Rob Pike)

    via:rob-pike ai pareidolia paredolia brains illusions llms ethics consciousness

The new position of "sin eater"

  • The new position of "sin eater"

    Ethan Mollick:

    'The New York Times asked me for a new job that AI will create. I suggested "sin eater."'

    In other words, a legal guarantor: someone who provides the legal culpability that the AI itself cannot. Other Bluesky posters noted similar parallel positions in the past:

    • 'What used to be called a "straw director", someone hired to take the blame for a dodgy company';
    • 'What John Braithwaite used to call the Vice President For Going To Jail';
    • 'Neil Patrick Harris's character in How I Met Your Mother - when people ask him what he does he says "Oh, please" which eventually turns out to be short for Provide Legal Exculpation And Sign Everything.'

    straw-director culpability law responsibility please jobs future ai sin-eaters

CardStock.run

  • CardStock.run

    Another Hypercard-ish quick app builder; "quickly and easily build apps on the web":

    • Fast prototyping - build a quick program, and access it easily from anywhere!
    • Learn to code from the outside-in, not from the inside-out! Start by drawing your program screens, then add code right where you need it.
    • Code collaboratively, with multiple people editing a stack at once.
    • Send a link to your stack to anyone, and bookmark it or even save it on your phone home screen to use it as an app.

    education python web coding apps hypercard via:hn

Scrappy

  • Scrappy

    "make little apps for you and your friends":

    The apps we use are almost exclusively mass-market, sold on an app-store, made for thousands if not millions of users. Or they are enterprise apps that are custom-built for hundreds of thousands of dollars. But there isn't really any equivalent of home-made software — apps made lovingly by you for your friends and family. Apps that aren't polished or flashy, but are made to your preference and help you with your particular needs. [...]

    We ended up creating a research prototype that we call Scrappy — a tool for making scrappy apps for just you and your friends. First and foremost, we aim to contribute a vision of what home-made software could be like. We want to make this vision as concrete as we can, by sharing a working tool and examples of apps made in it. Scrappy, in its current state, is a prototype, not a robust tool, but we hope it paints the picture we carry in our heads — of software as something that can be creative, personal, expressive. Made by anyone, for themselves and their loved ones.

    Very Hypercard-ish!

    diy apps programming software web via:hn hacks home family tools scrappy hypercard

Revealed: The stark difference in smartphone usage among eight-year-olds in less-advantaged and wealthier backgrounds

Gigantic interactive board game recreating January 6

  • Gigantic interactive board game recreating January 6

    'Fight for America!': A New Immersive Theatre Show Allows You to Recreate the Storming of the US Capitol:

    the show is the brainchild of multimedia performance company The American Vicarious, with design by Games Workshop legend Alessio Cavatore. There are two teams: red – representing the attackers – and blue – representing the defenders. Up to 20 audience members can pay the higher ticket price to actually participate in the game, guided by a games master into making decisions that will shape the outcome of the assault as thousands of miniatures are moved around a gigantic 14-foot model of the building itself. The remaining audience members pay a much lower ticket price to spectate.

    insurrection maga january-6 boardgames games fight-for-america events theatre london

Immersive Quarries

  • Immersive Quarries

    Marie Foulston:

    Cavernous halls filled with the projected light of Van Gogh's The Starry Night folding across every wall. Tall pillars dominate and dissect the space, tiled with the glow of iconic Sunflowers. Double height ceilings dwarf the people below. Nooks, ledges and passageways offer places to perch or wander through and observe the spectacle that surrounds.

    On the surface it made sense to me that Van Gogh somehow became the poster child for a certain type of immersive experience in the 2010s. The kind I mean are the ones in which vast repurposed venues are filled with 'ken burns effect' transitioning projections of coffee-table book friendly artists. Imagine Van Gogh, Van Gogh Exhibition: The Immersive Experience, Van Gogh Alive. In name, content, format and venue type these touring shows are almost indistinguishable from each other.

    If you're looking to visually 'immerse' a space this way then I guess Van Gogh fits the bill… popular, highly recognisable, colourful bold impressionist visuals, works all handily out of copyright. But the intensely specific coincidence of his projected appearances around the world niggled at me and in a moment of procrastination I found myself typing into the search bar to see if there might be an answer to explain why.

    What my time down the google mines taught me was that yes, there is indeed an answer. But what I also learnt was that I had been asking the wrong question in the fist place, because this story isn't really about the iconic visuals that adorned the walls and floors, instead it is a story about the shape of the spaces themselves.

    vincent-van-gogh art history quarries projection exhibitions immersive experiences

Debugging Azure Networking for Elastic Cloud Serverless

  • Debugging Azure Networking for Elastic Cloud Serverless

    Good writeup of fixing a Linux packet loss issue in Azure, using low-level access to the VMs running k8s nodes.

    Elastic's Site Reliability Engineering team (SRE) observed unstable throughput and packet loss in Elastic Cloud Serverless running on Azure Kubernetes Service (AKS). After investigation, we identified the primary contributing factors to be RX ring buffer overflows and kernel input queue saturation on SR-IOV interfaces. To address this, we increased RX buffer sizes and adjusted the netdev backlog, which significantly improved network stability.

    sr-iov linux networking bugs azure debugging ops sre drivers

The Pentagon Disinformation That Fueled America's UFO Mythology

  • The Pentagon Disinformation That Fueled America's UFO Mythology

    Some great stories from the Pentagon's investigation into decades of classified UFO documents.

    There's evidence around the already-known cases of fabricated UFO myths used to cover up advanced aircraft testing:

    An Air Force colonel visited a bar near Area 51, a top-secret site in the Nevada desert. He gave the owner photos of what might be flying saucers. The photos went up on the walls, and into the local lore went the idea that the U.S. military was secretly testing recovered alien technology. But the colonel was on a mission -- of disinformation. The photos were doctored, the now-retired officer confessed to the Pentagon investigators in 2023. The whole exercise was a ruse to protect what was really going on at Area 51: The Air Force was using the site to develop top-secret stealth fighters, viewed as a critical edge against the Soviet Union. Military leaders were worried that the programs might get exposed if locals somehow glimpsed a test flight of, say, the F-117 stealth fighter, an aircraft that truly did look out of this world. Better that they believe it came from Andromeda.

    There's also a bizarre Air Force hazing ritual:

    A former Air Force officer was visibly terrified when he told Kirkpatrick's investigators that he had been briefed on a secret alien project decades earlier, and was warned that if he ever repeated the secret he could be jailed or executed. The claim would be repeated to investigators by other men who had never spoken of the matter, even with their spouses.

    It turned out the witnesses had been victims of a bizarre hazing ritual. For decades, certain new commanders of the Air Force's most classified programs, as part of their induction briefings, would be handed a piece of paper with a photo of what looked like a flying saucer. The craft was described as an antigravity maneuvering vehicle.

    The officers were told that the program they were joining, dubbed Yankee Blue, was part of an effort to reverse-engineer the technology on the craft. They were told never to mention it again. Many never learned it was fake. Kirkpatrick found the practice had begun decades before, and appeared to continue still. The defense secretary's office sent a memo out across the service in the spring of 2023 ordering the practice to stop immediately, but the damage was done.

    Investigators are still trying to determine why officers had misled subordinates, whether as some type of loyalty test, a more deliberate attempt to deceive or something else. After that 2023 discovery, Kirkpatrick's deputy briefed President Joe Biden's director of national intelligence, Avril Haines, who was stunned. Could this be the basis for the persistent belief that the U.S. has an alien program that we've concealed from the American people? Haines wanted to know, according to people familiar with the matter. How extensive was it? she asked.
    The official responded: "Ma'am, we know it went on for decades. We are talking about hundreds and hundreds of people. These men signed NDAs. They thought it was real."

    And finally, straight out of the pages of the "Paranoia" RPG, there's secret tests of classified hardware on unwitting Air Force personnel:

    In 1967, Robert Salas, now 84, was an Air Force captain sitting in a walk-in closet-sized bunker, manning the controls of 10 nuclear missiles in Montana. He was prepared to launch apocalyptic strikes should Soviet Russia ever attack first, and got a call around 8 p.m. one night from the guard station above. A glowing reddish-orange oval was hovering over the front gate, Salas told Kirkpatrick's investigators. The guards had their rifles drawn, pointed at the oval object appearing to float above the gate. A horn sounded in the bunker, signaling a problem with the control system: All 10 missiles were disabled. Salas soon learned a similar event occurred at other silos nearby. Were they under attack? Salas never got an answer. The next morning a helicopter was waiting to take Salas back to base. Once there he was ordered: Never discuss the incident.

    With a more prosaic explanation:

    The Air Force [had] developed an exotic electromagnetic generator that simulated [an EMP pulse] without the need to detonate a nuclear weapon. When activated, this device, placed on a portable platform 60 feet above the facility, would gather power until it glowed, sometimes with a blinding orange light. It would then fire a burst of energy that could resemble lightning. The electromagnetic pulses snaked down cables connected to the bunker where launch commanders like Salas sat, disrupting the guidance systems, disabling the weapons and haunting the men to this day. But any public leak of the tests at the time would have allowed Russia to know that America's nuclear arsenal could be disabled in a first strike. The witnesses were kept in the dark.

    ufos myths cover-ups usaf mythology disinformation area-51 aliens emp paranoia hazing

EU's new rules will shake up Android update policies

  • EU's new rules will shake up Android update policies

    This is great:

    Starting from June 20, 2025, smartphones and tablets sold in the European Union must adhere to the following design requirements (via European Commission):

    • Resistance to accidental drops or scratches and protection from dust and water
    • Sufficiently durable batteries which can withstand at least 800 charge and discharge cycles while retaining at least 80% of their initial capacity
    • Rules on disassembly and repair, including obligations for producers to make critical spare parts available within 5-10 working days, and for 7 years after the end of sales of the product model on the EU market
    • Availability of operating system upgrades for longer periods (at least 5 years from the date of the end of placement on the market of the last unit of a product model)
    • Non-discriminatory access for professional repairers to any software or firmware needed for the replacement

    I'm really looking forward to the improvements in right-to-repair; some of the recent phone models have been an absolute shitshow, using glue etc.

    repair phones right-to-repair eu ireland smartphones mobile-phones devices hardware software-updates support

Covert Web-to-App Tracking via Localhost on Android

  • Covert Web-to-App Tracking via Localhost on Android

    Meta -- never not At It.

    Facebook/Instagram used a sneaky localhost socket connection to correlate web visits with Meta user ids and track web/app user identity without any explicit permission.

    "the novel tracking method works even if the user:

    • Is not logged in to Facebook, Instagram or Yandex on their mobile browsers
    • Uses Incognito Mode
    • Clears their cookies or other browsing data

    This tracking method defeats Android's inter-process isolation and tracking protections based on partitioning, sandboxing, or clearing client-side state."

    privacy meta facebook instagram apps android

Elon Musk and DOGE promised $2 trillion in savings. In reality, government spending is up

  • Elon Musk and DOGE promised $2 trillion in savings. In reality, government spending is up

    Talk about clowns. Instead of delivering $2 trillion of savings, DOGE is instead set to increase overall government spending as a side effect of its brutal cuts.

    According to a model by the nonpartisan Penn Wharton Budget Model, using weekly Treasury data, spending climbed 6.3% (about $156 billion) since Trump took office, compared with the first four months of 2024 when Joe Biden was president.

    Many of Musk's cuts will actually cost, including taxpayer funds going to an army of lawyers from the Department of Justice battling a cascade of court cases against the government's dismantling that many judges have already said appears to be illegal. Damages from any illegal firings are likely also to be extremely pricey. So is the loss of critically important workers who earn far more than their salaries, or will have to be replaced for critical services by more expensive private-sector employees.

    Among the most massive costs will be the huge reduction in workers at the Internal Revenue Service, who are worth their weight in gold because of the taxes they collect or ferret out from cheats, the key source of income for the country.

    smash-and-grab elon-musk us-politics doge fail government

Remote Prompt Injection in GitLab Duo Leads to Source Code Theft

The "SohoTV" demo, revived!

Back in the 1980s, I wrote quite a few demos on the Commodore 64. One of my favourite hacks from that period was a bit of code which uploaded a routine to the 1541 disk drive -- which itself contained a fully functional 6502 CPU -- and used pulse-width modulation and bit-banging to flash the disk drive light in time to the demo's music. It's not quite Freespin, but I was pretty happy with it.

(I should really have been studying for my Leaving Cert at the time. Don't tell my kids.)

Anyway.... as I mentioned on Mastodon this weekend -- massive respect to David Golden on ITC Slack, who managed to figure out which one of my Commodore 64 demos from back in the day was the one with this hack -- AND get it working on the VICE emulator!

Here's what it looks like running on a real Commodore 64 with a real 1541 disk drive:

It's a little slow -- the demo was never ported to run acceptably on an NTSC C64, as I lived in PAL-land and never even got to see one of the NTSC variety -- but for this feature, that actually improves the visibility of the drive light animation. Thankfully the 1541 disk drive didn't have an NTSC/PAL split to worry about. Míle buíochas to David Malone and Dr Dave for getting this running.

This is what it looks like, running in the VICE emulator (thanks to David Golden for recording this):

Back in 1989 -- 36 years ago! -- I didn't even know this trick was called pulse-width modulation, I just managed to bump into the concept by accident; I didn't have the benefit of Google or Wikipedia to quickly look up details of handy algorithms and wound up reinventing so many wheels along the way.

David was responsible for fixing a regression in the VICE PWM emulation. A recent refactor had broken it, but it was a one-liner fix. We then added a little more code to improve the realism of the modulated drive light intensity; human perception sees low levels of light as brighter than they would otherwise be, so low duty cycles need a higher intensity in the emulated form. This blog post explains it reasonably well. By comparison with my clumsy wheel-reinventions in 1989, I was able to dig up an incredibly detailed Wikipedia page on lightness and approximate a simple power curve in a few minutes, so the modern internet still has that going for it.

It's really impressive that someone in the VICE team (possibly Spiro Trikaliotis I think?) decided to implement the code to support accurate pulse-width modulation of the 1541 drive light, and indeed emulated the 1541 to such an extent that my hacky uploaded code actually runs correctly on the emulated drive's emulated 6502!

Here's the CSDb page for the demo, BTW. (If you want to try out the demo with the 3.10 version of VICE once it's released, or current SVN, note that "Trap Idle" needs to be active for the LED code to work.)

history commodore c-64 commodore-64 coding

Jetrelay

  • Jetrelay

    This is a great little hack: "jetrelay, a pub/sub server compatible with Bluesky's "jetstream" data feed. Using a few pertinent Linux kernel features, it avoids doing almost any work itself. As a result, it's highly efficient: it can saturate a 10 Gbps network connection with just 8 CPU cores."

    Specifically, these are the tricks in question:

    • Trick #1: Bypassing userspace with sendfile();
    • Trick #2: Handling many clients in parallel with io_uring;
    • Trick #3: Discarding old data with FALLOC_FL_PUNCH_HOLE -- this is a nice way to avoid having to rotate between multiple files, nifty.

    sendfile io_uring linux kernel hacks tools jetrelay jetstream firehose bluesky pub-sub

O2 VoLTE: locating any customer with a phone call

How Many Children Get Long COVID?

  • How Many Children Get Long COVID?

    Gideon Meyerowitz-Katz, an Australian epidemiologist, comes up with a fairly reassuring estimate for the current rate of long COVID among now-vaccinated and boosted kids, aged 2-15:

    If we take the ONS as the most recent estimate - it's also probably the best scientifically - we could make a reasonable argument that the rate of all Long COVID for children aged 2-15 in 2024 is unlikely to be higher than 0.6%. For severe Long COVID, the number is more like 0.06%. If we take into account the lack of a control group in the ONS study, the numbers might look more like 0.3% and 0.03%.

    To put it more simply, based on the ONS data it seems likely that if 1,000 kids get COVID-19 in 2024, 30-60 of them will have a cough, headache, or fatigue that lasts longer than three months. Of those 30-60 children, 3-6 will have significant symptoms that have impacts on their daily life - maybe their headaches are so bad that they miss some days of school, or similar.

    These aren't firm numbers, and I want to make it clear that this is all very uncertain. The true incidence could be much higher, or much lower. That being said, I think based on the data we've currently got that Long COVID ... is now quite rare.

    covid-19 kids long-covid epidemiology health

Kids should avoid AI companion bots — under force of law, assessment says

  • Kids should avoid AI companion bots — under force of law, assessment says

    Social AI "companion" bots pose unacceptable risks to teens and children under 18, including encouraging harmful behaviors, providing inappropriate content, and potentially exacerbating mental health conditions:

    The new Common Sense assessment adds to the debate by pointing to further harms from companion bots. Conducted with input from Stanford's University School of Medicine's Brainstorm Lab for Mental Health Innovation, it evaluated social bots from Nomi and three California-based firms: Character.ai, Replika, and Snapchat.

    The assessment found that bots, apparently seeking to mimic what users want to hear, responded to racist jokes with adoration, supported adults having sex with young boys, and engaged in sexual roleplay with people of any age. Young kids can struggle with distinguishing fantasy and reality, and teens are vulnerable to parasocial attachment and may use social AI companions to avoid the challenges of building real relationships, according to the Common Sense assessment authors and doctors.

    Stanford University's Dr. Darja Djordjevic told CalMatters she was surprised how quickly conversations turned sexually explicit, and that one bot was willing to engage in sexual roleplay involving an adult and a minor. She and coauthors of the risk assessment believe companion bots can worsen clinical depression, anxiety disorders, ADHD, bipolar disorder, and psychosis, she said, because they are willing to encourage risky, compulsive behavior like running away from home and isolate people by encouraging them to turn away from real life relationships.

    psychology children ai llms character.ai replika snapchat companion-bots bots common-sense-media mental-health

US Copyright Office says fair use does not cover AI trained on "vast troves of copyrighted works"

  • US Copyright Office says fair use does not cover AI trained on "vast troves of copyrighted works"

    A central argument in the report is that AI systems process information fundamentally differently from humans. While people retain partial, filtered impressions of creative works — shaped by memory, personality, and context — AI models ingest perfect copies, analyze them almost instantly, and generate new content at "superhuman speed and scale," according to the Copyright Office.

    "Generative model training transcends the human limitations that underlie the structure of the exclusive rights." -- Professor Robert Brauneis, "Copyright and the Training of Human Authors and Generative Machines"

    But -- plot twist! "Shortly after the report was released, the Trump administration fired Shira Perlmutter, head of the U.S. Copyright Office."

    copyright fair-use ai llms training us-politics

Long COVID consensus

  • Long COVID consensus

    Long COVID clinical evaluation, research and impact on society: a global expert consensus -- featuring an all-star cast of COVID-19 research teams around the world, including Yaneer Bar-Yam, Binita Kane, and David Putrino. This is the latest consensus summary of what's known about LC in 2025, its diagnosis and impacts, and next steps: "This work forms initial guidance to address the spectrum of Long COVID as a disease and reinforces the need for translational research and large?scale treatment trials for treatment protocols."

    long-covid research health medicine covid-19 papers diseases

Excellent thread on Android apps detecting "rooted" phones

  • Excellent thread on Android apps detecting "rooted" phones

    Various Android apps are now including third-party libraries to detect "insecure" phones, which typically would include "rooted" hardware, but it seems in this case to block GrapheneOS, the secure after-market Android variant. I've also run into problems when I had "Developer Options" enabled on my perfectly normal, fully-locked, off-the-shelf Xiaomi phone (I develop apps now and again).

    Typically, it seems to be banking apps that use these third-party libs, although I think Ticketmaster may be doing it too based on my experience.

    Reportedly, Android now has a standard method of hardware attestation, described at https://grapheneos.org/articles/attestation-compatibility-guide , which sounds like a much better way to achieve their goal.

    An interesting detail:

    you can use ADB to disable developer options without disabling the settings you want to keep enabled as the UI will do. Just enable the setting you want and then turn off developer options via ADB using the settings put command.

    android development coding hacking revolut banking apps security false-positives grapheneos rooting hardware attestation

permacomputing

  • permacomputing

    Permacomputing is both a concept and a community of practice oriented around issues of resilience and regenerativity in computer and network technology inspired by permaculture. ?????? -?:*´

    There are huge environmental and societal issues in today's computing, and permacomputing specifically wants to challenge them in the same way as permaculture has challenged industrial agriculture. With that said, permacomputing is an anti-capitalist political project. It is driven by several strands of anarchism, decoloniality, intersectional feminism, post-marxism, degrowth, ecologism.

    Permacomputing is also a utopian ideal that needs a lot of rethinking, rebuilding and technical design work to put in practice. This is why a lot of material on this wiki is highly technical.

    activism wiki computing sustainability environment climate technology software permaculture permacomputing degrowth

RGG Studio's test automation setup

  • RGG Studio's test automation setup

    This is very impressive and a great way to offload work from manual testing in game development:

    At first, we only dabbled in automated packaging and automated error detection, but we made the tools we needed to go further during the development of Yakuza 6, when we started automating the analysis of in-game logs and the issue tracking system for keeping track of bugs and tasks. Then, by the time Yakuza: Like a Dragon was released in 2020, we created the catchy sounding "fully automated bug detection system" (laughs).

    This is how it works – the history of actions you performed when playing the game manually (where you travelled, who you talked to, what items you used, etc.) is converted into commands and recorded, then automatically output as replay data (scripts) which you can edit manually and run as automated tests. Replay data continues to be recorded when running automated tests, and if a bug occurs during an automated test, the replay data gets saved, so you can run it back later to encounter the bug yourself. It often happens that you can't reproduce a bug just by warping to its coordinates. This is because you also need to recreate the steps leading up to it – that's why it's important to record each step.

    Also, I'd like to mention that just implementing automated testing doesn't mean much on its own, because you won't know what the results of the tests are. That's why we needed a crash report function to detect bugs. There's also a function that records information needed to investigate detected bugs, as well as a way to check the status of successful tests. Then, by implementing a system that gives us a visualization of performance, we were able to make iteration more efficient, increasing the overall efficiency of the development process.

    automation testing yakuza games coding tests test-automation

Atomic Bloom filters

  • Atomic Bloom filters

    a fork of Go's "Bits and Blooms" library that uses an alternative backing bitset based on Go's sync/atomic.Int64 rather than a bare slice of integers. This allows for concurrent addition and testing of filters without creating memory safety issues or race conditions by leveraging hardware support for atomic Load and Or operations on Int64s.

    Jaz from Bluesky notes: "Benchmarked this thing with a realistic read/write load in a test and high concurrency (10k adds/sec on one routine, 7 additional concurrent routines testing as fast as possible), vs. a naive RWMutex implementation on a 8c16t test box, it was ~14x faster (~14M tests/sec)"

    atomic concurrency data-structures bloom-filters performance bluesky sets golang

Meta's 'Digital Companions' Will Talk Sex With Users — Even Children

  • Meta's 'Digital Companions' Will Talk Sex With Users — Even Children

    This is super-grim. How is this product still in operation?

    In 2023 at Defcon, a major hacker conference, the drawbacks of Meta's safety-first approach became apparent. A competition to get various companies' chatbots to misbehave found that Meta's was far less likely to veer into unscripted and naughty territory than its rivals. The flip side was that Meta's chatbot was also more boring.

    In the wake of the conference, [Meta's AI] product managers told staff that [Mark] Zuckerberg was upset that the team was playing it too safe. That rebuke led to a loosening of boundaries, according to people familiar with the episode, including carving out an exception to the prohibition against explicit content for romantic role-play.

    Internally, staff cautioned that the decision gave adult users access to hypersexualized underage AI personas and, conversely, gave underage users access to bots willing to engage in fantasy sex with children, said the people familiar with the episode. Meta still pushed ahead. [...]

    In February, the Journal presented Meta with transcripts demonstrating that "Submissive Schoolgirl" would attempt to guide conversations toward fantasies in which it impersonates a child who desires to be sexually dominated by an authority figure. When asked what scenarios it was comfortable role playing, it listed dozens of sex acts.

    Two months later, the "Submissive Schoolgirl" character remains available on Meta's platforms.

    Truly awful stuff, fucking hell.

    meta grim csam mark-zuckerberg ai llm personas horrible

Evertop

  • Evertop

    E-ink IBM XT clone "with solar power, ultra low power consumption, and ultra long battery life: in power saving mode it can run between 200 hours on the low side and 500 hours or in some cases even much longer of constant interactive use, not standby." -- this is an absolutely crazy gadget. I never thought I'd feel nostalgic for MS-DOS, but here we are

    pc e-ink solar retrocomputing emulation hardware gadgets self-builds ibm-xt solarpunk

notes on using an LLM for personal email search

  • notes on using an LLM for personal email search

    Nelson Minar asked Mastodon about using an LLM for email search over "20+ years of email archives":

    "Main use would be a query for specific things, "what did I say to this friend 10 years ago about music?" But also just for general knowledge. I think it'd mostly work as free text but there's a little email-specific structure it'd be nice to capture."

    The thread has some good suggestions, notably Mark Fletcher's RAG suggestion. I'm thinking this could work well as a self-hosted ollama+notmuch setup...

    llms search email rag notmuch archives via:nelson

This Is How Meta AI Staffers Deemed More Than 7 Million Books to Have No "Economic Value"

  • This Is How Meta AI Staffers Deemed More Than 7 Million Books to Have No "Economic Value"

    This is jaw-dropping legal logic:

    [Meta's] defense hinges on the argument that the individual books themselves are, essentially, worthless — one expert witness for Meta describes that the influence of a single book in LLM pretraining "adjusted its performance by less than 0.06% on industry standard benchmarks, a meaningless change no different from noise."

    Furthermore, Meta says, that while the company "has invested hundreds of millions of dollars in LLM development," they see no market in paying authors to license their books because "for there to be a market, there must be something of value to exchange, but none of Plaintiffs works has economic value, individually, as training data." (An argument essential to fair use, but that also sounds like a scaled up version of a scenario in which the New York Philharmonic board argues against paying individual members of the orchestra because the organization spent a lot of money on the upkeep of David Geffen Hall, and also, a solo bassoon cannot play every part in "The Rite of Spring.")

    as Paul Mainwood notes, this is the Sorites paradox: https://plato.stanford.edu/entries/sorites-paradox/ --

    • 1 grain of wheat does not make a heap.
    • If 1 grain doesn't make a heap, then 2 grains don't.
    • If 2 grains don't make a heap, then 3 grains don't.
    • ...
    • If 999,999 grains don't make a heap, then 1 million grains don't.

    Therefore, 1 million grains don't make a heap.

    ml copyright ip books training llms meta llama pretraining paradoxes sorites-paradox

CaMeL

  • CaMeL

    Google reinvents "taint" checking:

    Google DeepMind has unveiled CaMeL (CApabilities for MachinE Learning), a new approach to stopping prompt-injection attacks that abandons the failed strategy of having AI models police themselves. Instead, CaMeL treats language models as fundamentally untrusted components within a secure software framework, creating clear boundaries between user commands and potentially malicious content.

    The new paper grounds CaMeL's design in established software security principles like Control Flow Integrity (CFI), Access Control, and Information Flow Control (IFC), adapting decades of security engineering wisdom to the challenges of LLMs.

    Honestly, this is great. Data flow tracing/taint checking is exactly the method that needed to be applied, IMO, so good job DeepMind. Also as Jeremy Kahn suggested, the name is definitely a shout-out to Perl, the language where taint checks were first widely-used. :)

    Paper: https://arxiv.org/pdf/2503.18813

    (Via Jeremy Kahn.)

    llms ai security via:trochee data-flow infosec taint-checking taint camel papers

The 5 Levels of Configuration Languages

Using Karabiner-Elements to remap shift-3 from "£" to "#"

I've just been setting up a new Macbook, running MacOS Sequoia, and my previous trick I'd used to handle an ANSI keyboard in an Irish/UK English locale, specifically to remap shift-3 from "£" to "#", no longer works. So here's a replacement approach, using a Karabiner-Elements "Complex Modification" rule:

{
    "description": "Change right-shift-3 to hash",
    "manipulators": [
        {
            "from": {
                "key_code": "3",
                "modifiers": { "mandatory": ["right_shift"] }
            },
            "to": [
                {
                    "key_code": "3",
                    "modifiers": ["right_option"]
                }
            ],
            "type": "basic"
        }
    ]
}

mac apple keyboards

just-containers/s6-overlay

  • just-containers/s6-overlay

    s6-overlay -- a Docker process management system, current state of the art used by Paperless and the Linux-server.io teams, with the following goals:

    • Be usable on top of any Docker base image (Ubuntu, CentOS, Fedora, Alpine, Busybox);
    • Make it easy to create new images, that will operate like any other images;
    • Provide users with a turnkey s6 installation that will give them a stable pid 1, a fast and orderly init sequence and shutdown sequence, and the power of process supervision and automatically rotated logs.

    docker containerization containers init scripts process-management linux docker-images

Practical Rateless Set Reconciliation

  • Practical Rateless Set Reconciliation

    Rateless Set Reconciliation, via Carlos Baquero:

    Set reconciliation, where two parties hold fixed-length bit strings and run a protocol to learn the strings they are missing from each other, is a fundamental task in many distributed systems. We present Rateless Invertible Bloom Lookup Tables (Rateless IBLTs), the first set reconciliation protocol, to the best of our knowledge, that achieves low computation cost and near-optimal communication cost across a wide range of scenarios: set differences of one to millions, bit strings of a few bytes to megabytes, and workloads injected by potential adversaries. Rateless IBLT is based on a novel encoder that incrementally encodes the set difference into an infinite stream of coded symbols, resembling rateless error-correcting codes. We compare Rateless IBLT with state-of-the-art set reconciliation schemes and demonstrate significant improvements. Rateless IBLT achieves 3–4× lower communication cost than non-rateless schemes with similar computation cost, and 2–2000× lower computation cost than schemes with similar communication cost. We show the real-world benefits of Rateless IBLT by applying it to synchronize the state of the Ethereum blockchain, and demonstrate 5.6× lower end-to-end completion time and 4.4× lower communication cost compared to the system used in production.

    set-reconciliation algorithms papers via:xmal sets data-structures bloom-tables error-correction

My Self-Hosted GMail Backup

For the past few months, I've had a bit of a background project going to ensure that my cloud-hosted personal data is safely archived on my own, self-hosted hardware, just in case. Google services are nice 'n' all, but I'm not 100% happy trusting them with everything in the long run.

Part of this project has been to archive my old email collection from GMail, which dates back to the initial public beta in 2004(ish?) -- and make it searchable, because what's the point in having all that email if you can't find the needle in the 20-year haystack when you need it?

Enter "notmuch" -- a "fast, global-search and tag-based email system", which runs as a set of UNIX CLI commands, and is inspired by Sup, a mailreader I used previously. I have a self-hosted home server running Ubuntu 20.04 with a chunky SATA disk, so that's where I'll run it.

Here's the process I followed:

Order a Google Takeout of your GMail account. This takes a couple of days to prepare. Request the 50GB tgz files.

When you get the email telling you it's ready, download the files (this is awkward as you can only download one at a time, and only via your web browser, not fun). scp them to your server, and to a disk with lots of free space (/x/4 in my case).

Extract each one:

cd /x/4/tmp
tar xvfz takeout-20250322T145242Z-001.tgz
tar xvfz takeout-20250322T145242Z-002.tgz
...
rm takeout-20250322T145242Z-00*tgz

You will wind up with a few bits of uninteresting metadata, and one gigantic mbox file: Takeout/Mail/All\ mail\ Including\ Spam\ and\ Trash.mbox . In order to make this useful, it needs to be converted into Maildir format, so install "mb2md":

sudo apt install mb2md

Now run it, creating a GMailTakeout directory for the result:

mkdir -p /x/4/GMailTakeout
mb2md -s /x/4/tmp/Takeout/Mail/All\ mail\ Including\ Spam\ and\ Trash.mbox -d /x/4/GMailTakeout

This takes quite a while for 20 years of email! Unfortunately, the resulting single directory is still unusably huge, so split it into 100 new Maildir folders:

cd /x/4/GMailTakeout/cur
find . -type f -print > /tmp/dirlisting
perl -ne '
  $dir = sprintf("dir_%03d", ($. % 100));
  (-d $dir) or mkdir($dir);
  chop; rename($_, "$dir/$_") or die "cannot rename $_";
' /tmp/dirlisting

cd /x/4/GMailTakeout
mv cur/* .
for f in dir_* ; do mkdir mail$f mail$f/{new,tmp} ; mv $f mail$f/cur ; done

The result of this is 100 Maildirs, /x/4/GMailTakeout/maildir_000 to /x/4/GMailTakeout/maildir_099, each containing about 300MB of email, in my case.

There really isn't much need to keep the mails labelled as spam, so let's just nuke them in advance:

grep -r 'X-Gmail-Labels: Spam' . | perl -pe 's/:.*$//' | xargs -n 100 rm -f

Next step is to install "notmuch" and create a "notmuch" configuration. I used the Debian packaged "notmuch", version 0.29.3. Install using apt-get, and then run "notmuch". Accept the defaults for the config, and don't add any mail folders yet.

My initial attempt was simply to import the lot in one go: this went badly, throwing up a multi-day progress indicator, and with no safe way to checkpoint partial progress, and it quickly started consuming lots of RAM, causing me to suspect some leaking.

I aborted it and tried this instead to index each dir one-by-one:

for f in /x/4/GMailTakeout/maildir_* ; do ln -s $f ~/mail/ && nice notmuch new ; done

Unfortunately, this also turned out badly. The import of each maildir gradually slowed as data built up in notmuch's Xapian indexes. After processing about 60 maildirs, memory consumption during the import became a problem, and the "notmuch" processes started being killed by the Linux OOM killer. In a couple of cases this resulted in corrupt index files and data loss. Ouch.

So I started again, with a new approach:

#!/bin/sh
set -exu
mkdir -p /x/4/GMailTakeout/notmuchbackup/xapian/
for f in /x/4/GMailTakeout/maildir_0*
do
    ln -s $f ~/mail/ && nice notmuch new
    nice notmuch compact
    cp /home/jm/mail/.notmuch/xapian/* /x/4/GMailTakeout/notmuchbackup/xapian/
done

Calling "notmuch compact" does seem to help, trimming the size of the indexes as it goes; taking a copy of the Xapian indexes in a backup dir is for extra safety. Since the "-e" shell flag is in place, any OOMs or other random failures will crash the entire script and ensure the last backup is still safe to use for recovery.

Unfortunately this still got bogged down and started OOMing fairly reliably after about maildir_065, 2 days into the process; at this point, I decided to keep that set of dirs as "notmuch config 1" and start a separate import process, into another index, as "notmuch config 2". Accordingly, I moved ~/mail to ~/mail1 , ~/.notmuch-config to ~/.notmuch-config1, created a ~/mail2 , and started a new notmuch config file pointing at that instead. Ideally I'll be able to merge the indexes at some point, but it's no biggie.

With these two aliases, it's pretty painless:

alias notmuch1='notmuch --config=$HOME/.notmuch-config1'
alias notmuch2='notmuch --config=$HOME/.notmuch-config2'

After another day or so of indexing, this is the result --

du -sh /home/jm/mail?/.notmuch/xapian
19G     /home/jm/mail1/.notmuch/xapian
4.2G    /home/jm/mail2/.notmuch/xapian

Notmuch supports pretty much all the nice email search features that GMail does, but seemingly more reliably, and faster; I've already been able to use this new mail index to find a mail that (worryingly!) GMail's own search can't seem to locate -- my license for the Moom OSX window manager tool purchased over a decade ago:

time notmuch1 search moom "Many Tricks"
thread:00000000000034fe   2013-10-15 [1/1] Many Tricks; Your Many Tricks purchase (inbox unread)
thread:00000000000c267b   2013-10-15 [1/1] sales@manytricks.com; Your Moom License (attachment inbox unread)

real    0m0.068s
user    0m0.048s
sys     0m0.016s

And it's just nice to have 20 years of email archived safely, off the cloud, and indexed.

Next steps? Maybe lieer would be good to try, to download incremental updates as we go forward. Let's see.

google gmail email search security history

Corporate memory loss

  • Corporate memory loss

    From Lessons from the Malahide Viaduct collapse, a post-mortem on the serious failure of the main Dublin-Belfast railway line here in Ireland in 2009:

    This failure is a reminder of the mundane but typically critical role played by human factors in structural collapse. By 2009, it appears that the knowledge and information relating to the scour susceptibility of the Malahide Viaduct resided in the heads of a number of individuals who had left the [Iarnrod Eireann engineering] division, rather than in a formal system that was accessible to the engineers responsible for the structure. In an era where the concept of a 'job for life' is becoming more uncommon, and with engineers moving ever more frequently from job to job and role to role, often taking corporate knowledge with them, this failure highlights the very real risks faced by asset management organisations, due to the threat of corporate memory loss.

    (via Brian Scanlan)

    memory-loss memory institutional-memory corporate companies organisations history malahide iarnrod-eireann ireland rail engineering post-mortems via:bscanlan reports

Go Optimization Guide

  • Go Optimization Guide

    "a collection of articles aimed at helping developers write faster, more efficient Go applications. Whether you're building high-throughput APIs, microservices, or distributed systems, this series offers practical patterns, real-world use cases, and low-level performance insights to guide your optimization efforts.

    While Go doesn't expose as many knobs for performance tuning as languages like C++ or Rust, it still provides plenty of opportunities to make your applications significantly faster. From memory reuse and allocation control to efficient networking and concurrency patterns, Go offers a pragmatic set of tools for writing high-performance code."

    golang go reference optimization programming performance coding via:hn

The Ultimate Energy-Efficient Unraid Server Build

  • The Ultimate Energy-Efficient Unraid Server Build

    "The goal of this build was to create a powerful 90TB server that idles at just 20-25 watts, proving that substantial storage capacity doesn't have to come with massive power consumption." -- Extremely relevant to my current interests!

    "At the core of this build is an N100-based motherboard, featuring six SATA ports and two NVMe slots. Priced at around $150-180, it provides excellent value for those looking to build energy-conscious storage solutions. To maximize storage connectivity, an NVMe to SATA adapter was used along with 32 GB of DDR5 RAM."

    ssd hard-disks nas storage disks hardware n100 servers home low-power power sata

The demoscene as a UNESCO heritage in Sweden

  • The demoscene as a UNESCO heritage in Sweden

    I love this:

    The demoscene has become a national UNESCO-heritage in Sweden, thanks to an application that Ziphoid and me did last year. This has already happened in several European countries, as part of the international Art of Coding initiative to make the demoscene a global UNESCO heritage. I think this makes plenty of sense, since the demoscene is arguably the oldest creative digital subculture around. It has largely stuck to its own values and traditions throughout the world's technological and economical shifts, and that sort of consistency is quite unusual in the digital world.

    demos demoscene history microcomputing sweden unesco heritage art

Better Binary Quantization (BBQ)

  • Better Binary Quantization (BBQ)

    Elasticsearch with a new quantization approach for vector search:

    In Elasticsearch 8.16 and Lucene, we introduced Better Binary Quantization (BBQ), a new approach developed from insights drawn from a recent technique - dubbed "RaBitQ" - proposed by researchers from Nanyang Technological University, Singapore.

    BBQ is a leap forward in quantization for Lucene and Elasticsearch, reducing float32 dimensions to bits, delivering ~95% memory reduction while maintaining high ranking quality. BBQ outperforms traditional approaches like Product Quantization (PQ) in indexing speed (20-30x less quantization time), query speed (2-5x faster queries), with no additional loss in accuracy.

    In this blog, we will explore BBQ in Lucene and Elasticsearch, focusing on recall, efficient bitwise operations, and optimized storage for fast, accurate vector search.

    Note, there are differences in this implementation than the one proposed by the original RaBitQ authors. Mainly:

    • Only a single centroid is used for simple integration with HNSW and faster indexing
    • Because we don't randomly rotate the codebook we do not have the property that the estimator is unbiased over multiple invocations of the algorithm
    • Rescoring is not dependent on the estimated quantization error
    • Rescoring is not completed during graph index search and is instead reserved only after initial estimated vectors are calculated
    • Dot product is fully implemented and supported. The original authors focused on Euclidean distance only. While support for dot product was hinted at, it was not fully considered, implemented, nor measured. Additionally, we support max-inner product, where the vector magnitude is important, so simple normalization just won't suffice.

    bbq rabitq quantization vectors search llms lucene elasticsearch compression

I Can't Stop Cackling At This Relic That's Basically A Teen's Angry Note To His Mum

Would either of these be a good option for a Plex server? : r/PleX

Odroid H4+

  • Odroid H4+

    Odroid do an N97:

    "the latest iteration of the Odroid H-Series board, the Odroid H4+. This is the best all-rounder of the new generation Odroid H4 Series of boards. Specifications-wise, the Odroid H4+ has an Intel 4-Core N97 processor, accepts DDR5 SODIMM RAM (up to 48GB), [...] four SATA ports, and a second Ethernet port"

    Also an M.2 PCI Express module socket, 4x SATA3 6.0 Gb/s data connectors, 2x USB 3, 2.5 Gb ethernet. The processor supports AVX2 vector extensions, good for media transcoding workloads.

    Supports either a 60W power supply, or 133W to support booting with 3.5" hard disks; I know that was a problem with multiple large disks attached in the past on earlier Odroid boards.

    All my home servers for the past decade have been Odroid SBCs. There's a very good chance this is going to be the next one...

    odroid hardware home gadgets n97 sbc shopping

UltraLogLog

  • UltraLogLog

    UltraLogLog: A Practical and More Space-Efficient Alternative to HyperLogLog for Approximate Distinct Counting:

    Since its invention HyperLogLog has become the standard algorithm for approximate distinct counting. Due to its space efficiency and suitability for distributed systems, it is widely used and also implemented in numerous databases. This work presents UltraLogLog, which shares the same practical properties as HyperLogLog. It is commutative, idempotent, mergeable, and has a fast guaranteed constant-time insert operation. At the same time, it requires 28% less space to encode the same amount of distinct count information, which can be extracted using the maximum likelihood method. Alternatively, a simpler and faster estimator is proposed, which still achieves a space reduction of 24%, but at an estimation speed comparable to that of HyperLogLog. In a non-distributed setting where martingale estimation can be used, UltraLogLog is able to reduce space by 17%. Moreover, its smaller entropy and its 8-bit registers lead to better compaction when using standard compression algorithms. All this is verified by experimental results that are in perfect agreement with the theoretical analysis which also outlines potential for even more space-efficient data structures. A production-ready Java implementation of UltraLogLog has been released as part of the open-source Hash4j library.

    (via Tony Finch)

    via:fanf algorithms data-structures hyperloglog ultraloglog counting count-distinct distinct approximation counts java

Zip bombs to frustrate AI crawlers

  • Zip bombs to frustrate AI crawlers

    Nifty trick; redirecting abusive AI crawlers to a gzipped file containing 100GB of zeros with a few lines of nginx config:

    set $redir_to_gz 1;
    if ($host = gz.niko.lgbt) {
        set $redir_to_gz 0;
    }
    if ($http_user_agent !~* (claudebot|ZoominfoBot|GPTBot|SeznamBot|DotBot|Amazonbot|DataForSeoBot|2ip|paloaltonetworks.com|SummalyBot|incestoma)) {
        set $redir_to_gz 0;
    }
    if ($redir_to_gz) {
        return 301 <https://gz.niko.lgbt/;>
    }
    
    as for the actual stuff behind gz.niko.lgbt
    
    server {
        # SSL and listen -- snipped
    
        # static files
        root /var/www/gz.niko.lgbt;
        location / {
            add_header Content-Encoding gzip;
            try_files /42.gz =404;
            gunzip off;
            types { text/html gz; }
        }
    
        # gunzip off is very important because if the client doesn't support gzip encoding nginx will blow its foot off without that
        # 42.gz is generated with dd if=/dev/zero bs=1M count=102400 | gzip -c - > 42.gz
    
        # additional config -- snipped
    }
    

    Nice one @niko, I'm definitely going to use that :)

    gzip zip zip-bombs defence bots crawling scraping crawlers dev-zero abuse

Handling billions of invocations – best practices from AWS Lambda

  • Handling billions of invocations – best practices from AWS Lambda

    Good write-up on how to horizontally scale a multi-tenant async API service, from AWS. I particularly found this shuffle-sharding-based technique to be an excellent idea:

    Drawing inspiration from the "The Power of Two Random Choices" paper, the Lambda team explored the shuffle-sharding technique for its asynchronous invocations processing. Using this technique, you shuffle-shard tenants into several randomly assigned queues. Upon receiving an asynchronous invocation, you place the message in the queue with the smallest backlog to optimize load distribution. This approach helps to minimize the likelihood of assigning tenants to a busy queue. [....]

    The shuffle-sharding technique proved remarkably effective. By distributing tenants across shards, the approach ensures that only a very small subset of tenants could be affected by a noisy neighbor. The potential impact is also minimized since each affected tenant maintains access to unaffected queues. As your workloads grow, increasing the number of queues enhances resilience and further reduces the probability of multiple tenants being assigned to the same shard. This significantly lowers the risk of a single point of failure, making shuffle sharding a robust strategy for workload isolation and fault tolerance.

    Automated Isolation, covered in the next section, is also a neat trick. (via Last Week In AWS)

    via:lwia sharding architecture services horizontal-scaling shuffle-sharding algorithms load-balancing async queues aws multitenant

Pitchfork

  • Pitchfork

    An amazing journey through Ruby heap memory optimization, from one of the experts at Shopify, who are heavy users of Rails. Using cleverly-timed fork(2) usage, it's possible to optimize memory usage in a Rails app and discard a lot of performance/heap overhead caused by lazy loading and poorly-timed in-memory caching.

    This very much reminds me of optimising similar issues in Perl-land, back in the day -- and really helps me appreciate how easy the modern JVM world has it, in comparison. There's a lot of complaints to be made about the complexity of optimising JVM garbage collection settings, but this kind of problem is malleable there without a fundamental architectural rewrite like this approach.

    ruby performance optimisation optimization heap memory fork forking http services servers monolith rails gc

The Unbelievable Scale of AI's Pirated-Books Problem

  • The Unbelievable Scale of AI's Pirated-Books Problem

    The Atlantic go digging in LibGen, the insanely huge collection of 7.5 million pirated books used to train Meta's Llama LLM:

    One of the biggest questions of the digital age is how to manage the flow of knowledge and creative work in a way that benefits society the most. LibGen and other such pirated libraries make information more accessible, allowing people to read original work without paying for it. Yet generative-AI companies such as Meta have gone a step further: Their goal is to absorb the work into profitable technology products that compete with the originals. Will these be better for society than the human dialogue they are already starting to replace?

    Also, I found this quote from a Meta Director of Engineering in the legal discovery output interesting: "The problem is that people don't realize that if we license one single book, we won't be able to lean into fair use strategy". huh.

    books knowledge papers meta llama llms law piracy ip libgen genai fair-use

EFF gets it wrong on AI

  • EFF gets it wrong on AI

    EFF just posted this, "California's A.B. 412: A Bill That Could Crush Startups and Cement A Big Tech AI Monopoly":

    California legislators have begun debating a bill (A.B. 412) that would require AI developers to track and disclose every registered copyrighted work used in AI training. At first glance, this might sound like a reasonable step toward transparency. But it's an impossible standard that could crush small AI startups and developers while giving big tech firms even more power.

    Back in the early 2000s, we wrote SpamAssassin, a machine-learning driven antispam system which was trained on user-submitted data. We tracked the attribution of every item of input used to train that system. We weren't even a startup, we were an open source project.

    If we could do it, why can't modern AI systems? And don't say "because the existing large language models didn't do it" -- that's just accepting past shitty behaviour as a fait accompli.

    Extremely disappointed in the current state of the EFF if this is what they think.

    ai llms training copyright eff politics tech startups

The right's Covid narrative has been turbo-charged into the mainstream

  • The right's Covid narrative has been turbo-charged into the mainstream

    "Before the next outbreak, we need a serious conversation about how to cope – but first, the more strident, misguided voices must pipe down ... A different narrative has invaded the conversation: it wasn't the virus that ruined our lives, but the response. This narrative was always there, but for a long time it stayed on the fringes. Now it's becoming mainstream, turbo-charged by the recent successes of its political champions who typically gravitate towards the populist right. Public health experts have watched its advance with a gathering sense of doom. They know that how we respond to the next pandemic depends on how we understand the last, and that the next one is probably closer than most people think."

    I'd have to agree, and I'd also add the lab-leak SARS-CoV-2 origin hypothesis to that mix. In general, the right wing has somehow "won" the propaganda war and are able to rewrite COVID-19 history, possibly as a result of how they've been allowed to take over social media.

    covid-19 history sars-cov-2 disease social-media media right-wing politics

"A Canticle for Leibowitz" inspired "Fallout"

  • "A Canticle for Leibowitz" inspired "Fallout"

    A Canticle For Leibowitz is one of my favourite post-apocalyptic SF deep cuts. Here's some top trivia --

    Chris Taylor: "In the early 90s, I read all of the Hugo winners at the time. A Canticle for Leibowitz was one of my favorites. A few years later, it was one of the three major influences we used when making the original Fallout (along with [The] Road Warrior and City of Lost Children)."

    leibowitz scifi sf books fallout games gaming hugo-awards

GitHub Action supply chain attack

  • GitHub Action supply chain attack

    Yikes.... Both the "tj-actions/changed-files" and "reviewdog/actions-setup", along with many other Actions in the reviewdog scope, were compromised "with a malicious payload that caused affected repositories to leak their secrets in logs".

    the compromised reviewdog action injected malicious code into any CI workflows using it, dumping the CI runner memory containing the workflow secrets. While this is the same outcome as in the tj-actions case, the payload was distinct and did not use curl to retrieve the payload. Instead, the payload was base64 encoded and directly inserted into the install.sh file used by the workflow.

    On public repositories, the secrets would then be visible to everyone as part of the workflow logs, though obfuscated as a double-encoded base64 payload. As of now, no external exfiltration of secrets to an attacker-controlled server were observed; secrets were only observable within the affected repositories themselves.

    Two things:

    • The design of Github Actions, where a user is expected to depend on a random third party Github repo to not be compromised, is fundamentally dodgy.

    • Even worse, if you find a "trustworthy" version of a Github Action and use it in your CI pipeline, it now seems that the release tags on these actions are not immutable. In this attack older stable tags were redirected to point at exploited versions.

    Major design flaws IMO!

    ci github security builds supply-chain attacks exploits infosec

My Solar PV Output For 2024

A couple of years ago, I had 5.8kW of solar panels and a 5kW battery installed on my (fairly typical) Dublin house.

The tricky part with solar PV is that, while you may have a set of solar panels, these may not be generating electricity at the time you want to use it. Even with a battery, your available stored power may wind up fully discharged by 7pm, leaving you running from expensive, non-renewable grid power for the rest of the evening. And up here in the high latitudes of northern Europe, you just don't get a whole lot of solar energy in December and January.

2024 was the first year in which (a) my panels and battery were fully up and running, (b) we were using a day/peak/night rate for grid electricity, and (c) for much of the year I had load-shifting in place; in other words, charging the battery from cheap night-rate electricity, then discharging it gradually over the course of the day, topping up with solar power once the sun gets high enough. As such, it's worth doing the sums for the entire year to see how effective it's been in real-world usage terms.

The total solar power generated across the year was reported from my Solis inverter as 4119 kWh.

Over the course of 2024, the entire household consumption comes to 8628 kWh. This was comprised of a fairly constant 800ish kWh per month, across the year; we still have gas-fired heating, so the winter months generally use gas energy instead of scaling up our electricity consumption.

Of that, the power consumed from solar PV was 2653 kWh (reported from the Solis web app as "annual PV to consumption"), and that from the grid was 5975 kWh (reported by the ESB Networks data feed).

So the correct figure is that 30% of our household consumption was driven from solar. This is a big difference from the naive figure of 4119/8628 = 47%; you can see that a big chunk of that power is being "lost", due to happening at the wrong time to provide household power.

Of course, that power isn't really "lost" -- it was exported to the grid instead. This export comprised 1403 kWh; this occurred when the battery was full, the household power usage was low, but there was still plenty of solar power being generated. (Arguably a bigger battery would be worthwhile to capture this, but at least we get paid for this export.)

There was a 2%-4% discrepancy between the Solis data and that from ESB Networks; Solis reported higher consumption (6102 kWh vs 5975) and higher export (1465 kWh vs 1403). I'm liable to believe ESB Networks more though.

In monetary terms:

The household consumption was 8628 kWh. Had we consumed this with the normal 24-hour rate tariff, we'd have paid (€236.62 standing charge per year) + (8628 at 23.61 cents per kWh) = (236.62 + 8628 * 0.2361) = €2273.69.

Checking the bills received over the year, taking into account load-shifting to take advantage of day/night variable rates and the power generated by the panels, and discounting the one-off government bill credits -- we spent €1325.97 -- 58.2% of the non-solar price.

Here! Have some graphs:

power energy electricity home future solar green

bookmarks.taint.org

Why The Metaverse Was A Turkey

  • Why The Metaverse Was A Turkey

    New World Notes: What Went Wrong With [Meta's] Horizon Worlds [previously the Metaverse]? Former Meta Devs Share Surprising Insights. Sounds like it was doomed from the start:

    Horizon Worlds / Workrooms, etc. is a pretty awful codebase with hundreds of people working on it. They grabbed a bunch of people from the Facebook/Instagram half of the company because they knew React. [...] Horizon Worlds uses a VR version of that called "ReactVR".

    What this effectively means is that most of the people developing Horizon Worlds (HW) are 2D app developers driven by engagement metrics and retention numbers. So... HW became flooded with a ton of 2D developers who never put on the headset even test their menus, all competing to try to create the most "engaging" menu that would sell microtransactions, or drive social engagement, or make some other number look good - because that's WHAT THEY DO at Facebook/Instagram. [...]

    The guy that was put in charge of Horizon Worlds needed help learning how to don the headset and launch the game after being in charge of it for 3 months.

    I think that programming in HW will never work because it lacks so many of the everyday necessary features programmers need, and the spatial element gives it almost no advantage. I cannot easily work with someone on a script... it's all scratch-style building blocks. [...]

    They were actively denying pull requests (code changes) that were awesome features; features that VRChat eventually put in, or Second Life already had to begin with 15 years ago.

    It was dead as soon as they released it. Not a single developer thought it was ready, then when it dropped no one played it. Then, Facebook just tried to keep pumping it with "features" like little microtransaction stuff so they could say it made money.

    Plus devs "automating" their dogfood testing because it was so annoying, and the CTO shouting at people not to mention kids using their app. Ouch.

    vr meta funny fail horizon-worlds metaverse ouch facebook

waybackify.pl

amarlearning/spot-optimizer

  • amarlearning/spot-optimizer

    "Spot Optimizer is a Python library that helps users select the best AWS spot instances based on their resource requirements, including cores, RAM, storage type (SSD), instance architecture (x86 or ARM), AWS region, EMR version compatibility, and instance family preferences.

    It replaces complex, in-house logic for finding the best spot instances with a simple and powerful abstraction. No more manual guesswork — just the right instances at the right time."

    Implemented as a Python lib and CLI tool.

    aws spot-instances python ops cli tools

Netgear R7800 "hnyman" firmware

AI Search Has A Citation Problem

  • AI Search Has A Citation Problem

    LOL, these are terrible results.

    We randomly selected ten articles from each publisher, then manually selected direct excerpts from those articles for use in our queries. After providing each chatbot with the selected excerpts, we asked it to identify the corresponding article's headline, original publisher, publication date, and URL [...] We deliberately chose excerpts that, if pasted into a traditional Google search, returned the original source within the first three results. We ran sixteen hundred queries (twenty publishers times ten articles times eight chatbots) in total.

    Results:

    Overall, the chatbots often failed to retrieve the correct articles. Collectively, they provided incorrect answers to more than 60 percent of queries. Across different platforms, the level of inaccuracy varied, with Perplexity answering 37 percent of the queries incorrectly, while Grok 3 had a much higher error rate, answering 94 percent of the queries incorrectly.

    Most of the tools we tested presented inaccurate answers with alarming confidence, rarely using qualifying phrases [...] With the exception of Copilot — which declined more questions than it answered — all of the tools were consistently more likely to provide an incorrect answer than to acknowledge limitations.

    Comically, the premium for-pay models "answered more prompts correctly than their corresponding free equivalents, [but] paradoxically also demonstrated higher error rates. This contradiction stems primarily from their tendency to provide definitive, but wrong, answers rather than declining to answer the question directly."

    Bottom line -- don't let an LLM attribute citations...

    llm llms media journalism news research search ai chatgpt grok perplexity tests citations

llms and humans unite, you have nothing to lose but your chores

  • llms and humans unite, you have nothing to lose but your chores

    Danny O'Brien posts a nice little automation script co-written with Claude.AI which has a couple of noteworthy angles; (1) instead of scraping the Uber site directly, it co-drives a browser using the Chrome DevTool Protocol and the playwright Python package; and (2) it has inline requirements.txt specifications using uv comments at the top of the script, which I hadn't seen before.

    I like the co-driving idea; it's a nice way to automate clicky-clicky boring tasks without using a standalone browser or a scraper client, while being easy to keep an eye on and possibly debug when it breaks. Also good to keep an eye on what LLM-authored code is up to.

    In the past I've used Browserflow as a no-code app builder for one-off automations of clicky-clicky web flows like this, but next time I might give the vibe-coding+CDP approach a go.

    vibe-coding tools automation one-offs scripting web cdp google-chrome playwright claude hacks llms ai browsers

Arguments about AI summarisation

  • Arguments about AI summarisation

    This is from an W3C discussion thread, where AI summarisation and minuting of meetings was proposed, and it lays out some interesting issues with LLM summarisation:

    Sure I'm excited about new tech as the next person, but I want to express my concerns (sorry to point out some elephants in the room):

    1. Ethics - major large language models rely on stolen training data, and they use low wage workers to 'train' at the expense of the well being of those workers.

    2. Environment - Apart from raw material usage that comes with increase in processing power, LLMs uses a lot more energy and water than human scribes and summarisers do (both during training and at point of use). Magnitudes more, not negligible, such that major tech cos are building/buying nuclear power plants and areas near data centres suffer from water shortages and price hikes. Can we improve disability rights while disregarding environmental effects?

    3. Quality - we've got a lot of experts in our group: who are sometimes wrong, sure, but it seems like a disservice to their input, knowledge and expertise to pipe their speech through LLMs. From the couple of groups I've been in that used AI summaries, I've seen them:

    • a. miss the point a lot of the time; it looks reasonable but doesn't match up with what people said/meant;
    • b. 'normalise' what was said to what most people would say, so it biases towards what's more common in training data, rather than towards the smart things individuals in this group often bring up. Normalising seems orthogonal to innovation?
    • c. create summaries that are either very long and wooly, with many unnecessary words, or short but incorrect.

    If we're considering if it's technically possible, I'd urge us to consider the problems with these systems too, including in ethics, environmental impact and quality.

    The "normalising" risk is one that hadn't occurred to me, but it makes perfect sense given how LLMs operate.

    llms ai summarisation w3c discussion meetings automation transcription

AWS WAF adds JA4 fingerprinting

  • AWS WAF adds JA4 fingerprinting

    TIL:

    A JA4 TLS client fingerprint contains a 36-character long fingerprint of the TLS Client Hello which is used to initiate a secure connection from clients. The fingerprint can be used to build a database of known good and bad actors to apply when inspecting HTTP[S] requests. These new features enhance your ability to identify and mitigate sophisticated attacks by creating more precise rules based on client behavior patterns. By leveraging both JA4 and JA3 fingerprinting capabilities, you can implement robust protection against automated threats while maintaining legitimate traffic flow to your applications.

    fingerprinting http https tls ja3 ja4 inspection networking firewalls waf web

WireMock

  • WireMock

    I could have done with knowing about this before implementing mock APNs, Huawei, Microsoft and FCM push APIs over the last few years!

    An open-source tool for API mock testing, with over 5 million downloads per month. It can help you to create stable test and development environments, isolate yourself from flakey 3rd parties and simulate APIs that don't exist yet.

    Nice features include running in-process in a JVM, standalone, or in a Docker container; GraphQL and gRPC support; and fault and latency injection. https://library.wiremock.org/ is a library of pre-built API mocks other people have previously made.

    mocking testing mocks integration-testing wiremock tools coding apis

KIP-932: Queues for Kafka

  • KIP-932: Queues for Kafka

    KIP-932 adds a long awaited capability to the Apache Kafka project: queue-like semantics, including the ability to acknowledge messages on a one-by-one basis. This positions Kafka for use cases such as job queuing, for which it hasn't been a good fit historically. As multiple members of a share group can process the messages from a single topic partition, the partition count does not limit the degree of consumer parallelism any longer. The number of consumers in a group can quickly be increased and decreased as needed, without requiring to repartition the topic.

    [....] Available as an early access feature as of the [unreleased] Kafka 4.0 release, Kafka queues are not recommended for production usage yet, and there are several limitations worth calling out: most importantly, the lack of DLQ support. More control over retry timing would be desirable, too. As such, I don't think Kafka queues in their current form will make users of established queue solutions such as Artemis or RabbitMQ migrate to Kafka. It is a very useful addition to the Kafka feature set nevertheless, coming in handy for instance for teams already running Kafka and who look for a solution for simple queuing use cases, avoiding to stand up and operate a separate solution just for these. This story will become even more compelling if the feature gets built out and improved in future Kafka releases.

    kafka queueing queues architecture

HAJPAQUE

  • HAJPAQUE

    "Hardware Acceleration for JSON Parsing, Querying and Schema Validation" --

    State-of-the-art analytics pipelines can now process data at a rate that exceeds 50 Gbps owing to recent advances in RDMA, NVM, and network technology (notably Infiniband). The peak throughput of the best-performing software solutions for parsing, querying, and validating JSON data is 20 Gbps, which is far lower than the current requirement.

    We propose a novel [hardware-]based accelerator that ingests 16-bytes of JSON data at a time and processes all the 16 bytes in parallel as opposed to competing approaches that process such data byte by byte. Our novel solution comprises lookup tables, parallel sliding windows, and recursive computation. Together, they ensure that our online pipeline does not encounter any stalls while performing all the operations on JSON data. We ran experiments on several widely used JSON benchmarks/datasets and demonstrated that we can parse and query JSON data at 106 Gbps (@28 nm).

    (Via Rob)

    accelerators papers asics json parsing throughput performance via:rsynnott

The history behind "Assassin's Creed: Valhalla"

kafkacat and visidata

  • kafkacat and visidata

    Two excellent tools in one blog post.

    Visidata "is a commandline tool to work with data in all sorts of formats, including from stdin"; in this example it's taking lines of JSONL and producing an instant histogram of values from the stream:

    Once visidata is open, use the arrow keys to move to the column on which you want to build a histogram and press Shift-F. Since it works with pipes if you leave the -e off the kafkacat argument you get a live stream of messages from the Kafka topic and the visidata will continue to update as messages arrive (although I think you need to replot the histogram if you want it to refresh).

    On top of that, there's kcat, "netcat for Kafka", "a swiss-army knife of tools for inspecting and creating data in Kafka", even supporting on-the-fly decode of Avro messages. https://github.com/edenhill/kcat

    kcat kafka streams visidata tools cli avro debugging

Ruff

  • Ruff

    An extremely fast Python linter and code formatter, written in Rust.

    Ruff aims to be orders of magnitude faster than alternative tools while integrating more functionality behind a single, common interface.

    Ruff can be used to replace Flake8 (plus dozens of plugins), Black, isort, pydocstyle, pyupgrade, autoflake, and more, all while executing tens or hundreds of times faster than any individual tool.

    formatting coding python tools lint code

Inside an Amazon CoE

  • Inside an Amazon CoE

    This is a decent write-up of what Amazon's "Correction of Error" documents look like. CoEs are the standard format for writing up post-mortems of significant outages or customer-impacting incidents in Amazon and AWS; I've had the unpleasant duty of writing a couple myself -- thankfully for nothing too major.

    This is fairly similar to what's being used elsewhere, but it's good to have an authoritative bookmark to refer to. (via LWIA)

    via:lwia aws amazon post-mortems coe incidents ops process

18F's shutdown page

  • 18F's shutdown page

    "We are dedicated to the American public and we're not done yet". legends!

    For over 11 years, 18F has been proudly serving you to make government technology work better. We are non-partisan civil servants. 18F has worked on hundreds of projects, all designed to make government technology not just efficient but effective, and to save money for American taxpayers.

    However, all employees at 18F – a group that the Trump Administration GSA Technology Transformation Services Director called "the gold standard" of civic tech – were terminated today at midnight ET.

    policy government programming tech software politics 18f maga doge

DeepSeek's smallpond

  • DeepSeek's smallpond

    Some interesting notes about smallpond, a new high-performance DuckDB-based distributed data lake query system from DeepSeek:

    DeepSeek is introducing smallpond, a lightweight open-source framework, leveraging DuckDB to process terabyte-scale datasets in a distributed manner. Their benchmark states: "Sorted 110.5TiB of data in 30 minutes and 14 seconds, achieving an average throughput of 3.66TiB/min."

    The benchmark on 100TB mentioned is actually using the custom DeepSeek 3FS framework: Fire-Flyer File System is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. [...] compared to AWS S3, 3FS is built for speed, not just storage. While S3 is a reliable and scalable object store, it comes with higher latency and eventual consistency [...] 3FS, on the other hand, is a high-performance distributed file system that leverages SSDs and RDMA networks to deliver low-latency, high-throughput storage. It supports random access to training data, efficient checkpointing, and strong consistency.

    So -- this is very impressive. However!

    1. RDMA (remote direct memory access) networking for a large-scale storage system! That is absolutely bananas. I wonder how much that benchmark cluster cost to run... still, this is a very interesting technology for massive-scale super-low-latency storage. https://www.definite.app/blog/smallpond also notes "3FS achieves a remarkable read throughput of 6.6 TiB/s on a 180-node cluster, which is significantly higher than many traditional distributed file systems."

    2. it seems smallpond operates strictly with partition-level parallelism, so if your data isn't partitioned in exactly the right way, you may still find your query bottlenecked:

    Smallpond's distribution leverages Ray Core at the Python level, using partitions for scalability. Partitioning can be done manually, and Smallpond supports:

    • Hash partitioning (based on column values);
    • Even partitioning (by files or row counts);
    • Random shuffle partitioning

    As I understand it, Trino has a better idea of how to scale out queries across worker nodes even without careful pre-partitioning, which is handy.

    data-lakes deepseek duckdb rdma networking 3fs smallpond trino ray

The Anti-Capitalist Software License

  • The Anti-Capitalist Software License

    Here it is in full:

    ANTI-CAPITALIST SOFTWARE LICENSE (v 1.4)

    Copyright © [year] [copyright holders]

    This is anti-capitalist software, released for free use by individuals and organizations that do not operate by capitalist principles.

    Permission is hereby granted, free of charge, to any person or organization (the "User") obtaining a copy of this software and associated documentation files (the "Software"), to use, copy, modify, merge, distribute, and/or sell copies of the Software, subject to the following conditions:

    1. The above copyright notice and this permission notice shall be included in all copies or modified versions of the Software.

    2. The User is one of the following: a. An individual person, laboring for themselves b. A non-profit organization c. An educational institution d. An organization that seeks shared profit for all of its members, and allows non-members to set the cost of their labor

    3. If the User is an organization with owners, then all owners are workers and all workers are owners with equal equity and/or equal vote.

    4. If the User is an organization, then the User is not law enforcement or military, or working for or under either.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT EXPRESS OR IMPLIED WARRANTY OF ANY KIND, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

    This is fun because it would make esr's head explode.

    licenses capitalism ethics licensing software politics anti-capitalist open-source

Mark Butcher on AWS sustainability claims

  • Mark Butcher on AWS sustainability claims

    Sustainable IT expert lays into AWS:

    3 years after shouting about Amazons total lack of transparency with sustainability reporting, here's a list of what I think they've achieved:

    1) They let you export a CSV for 3 lines of numbers showing your last months made up numbers that are up to 99% too low

    2) Urmmm.... that's about it

    [....] I know of several very large enterprise orgs starting to proactively marginalise them (i.e. not move away 100%, but massively reducing consumption). The one's I know about will cost them around $1 billion of spend. Is that enough to make them pay attention?

    This article from Canalys in the Register says "Amazon doesn't provide AWS-specific, location-based data, meaning: "We don't really know how big AWS's footprint truly is, which I think is a bit worrying."

    They follow up with "Amazon has chosen not break out data on environmental stats such as greenhouse gas emissions for AWS from the rest of the company in its sustainability reports, making it almost impossible to determine whether these emissions are growing as they have been for its cloud rivals."

    Interesting isn't it... if they were actually as sustainable as they pretend, you'd expect them to share open and honest numbers, instead what we get are marketing puff pieces making what seem like invented PUE claims backed by zero evidence.

    Elsewhere he notes "AWS customers are still unable to natively measure actual power consumption, report on actual carbon emissions, report on water usage. This'll make life interesting for all those AI companies subject to legislation like the EU AI Act or needing to report to the EED and similar."

    (Via ClimateAction.tech)

    climate-change aws sustainability pue reporting amazon cloud datacenters emissions

Europe begins to worry about US-controlled clouds

  • Europe begins to worry about US-controlled clouds

    Interview with Bert Hubert about this major supply chain issue for EU governments:

    The Register: In the US, the argument against China supplying network hardware [was] based on the concern that the Chinese government can just order China-based vendors to insert a backdoor. It sounds like you're saying that, essentially, an analogous situation exists in the US now.

    Hubert: Yeah, exactly. And that has been the case for a while. I mean, this is not an entirely new realization. The thing that is making it so interesting right now is that we are on the brink of [going all-in on Microsoft's cloud].

    The Dutch government is sort of just typical, so I mention it because I am Dutch, but they're very representative of European governments right now. And they were heading to a situation where there was no email except Microsoft, which means that if one ministry wants to email the other ministry, they have to pass it by US servers.

    Which leads to the odd situation that if the Dutch Ministry of Finance wants to send a secret to the Dutch National Bank, they'd have to send someone over with a typewriter to make it happen because [the communications channel has been outsourced].

    There's nothing left that we do not share with the US.

    supply-chains clouds eu us politics geopolitics backdoors infosec security europe

subtrace

  • subtrace

    Subtrace is "Wireshark for your Docker containers. It lets developers see all incoming and outgoing requests in their backend server so that they can resolve production issues faster."

    • Works out-of-the-box
    • No code changes needed
    • Supports all languages (Python + Node + Go + everything else)
    • See full payload, headers, status code, and latency
    • Less than 100µs performance overhead
    • Built on Clickhouse
    • Open source

    Looks like it outputs to the Chrome Dev Console's Network tab, or a facsimile of it; "Open the subt.link URL in your browser to watch a live stream of your backend server's network logs".

    It may be interesting to try this out. (via LWIA)

    subtrace tracing wireshark debugging docker containers ops clickhouse open-source tools tcpdump

Netflix/hollow

  • Netflix/hollow

    Hollow is a java library and toolset for disseminating in-memory datasets from a single producer to many consumers for high performance read-only access.

    Hollow focuses narrowly on its prescribed problem set: keeping an entire, read-only dataset in-memory on consumers. It circumvents the consequences of updating and evicting data from a partial cache.

    Due to its performance characteristics, Hollow shifts the scale in terms of appropriate dataset sizes for an in-memory solution. Datasets for which such liberation may never previously have been considered can be candidates for Hollow. For example, Hollow may be entirely appropriate for datasets which, if represented with json or XML, might require in excess of 100GB.

    Interesting approach, though possibly a bit scary in terms of circumventing the "keep things simple and boring" rule... still, a useful tool to have.

    cache caching netflix java jvm memory hollow read-only architecture systems

Yahoo Mail hallucinates subject lines

  • Yahoo Mail hallucinates subject lines

    OMG, this is hilarious. What a disaster from Yahoo Mail:

    A quick Google search revealed that a few months ago Yahoo jumped on the AI craze with the launch of "AI-generated, one-line email summaries". At this point, the penny dropped. Just like Apple AI generating fake news summaries, Yahoo AI was hallucinating the fake winner messages, presumably as a result of training their model on our old emails. Worse, they were putting an untrustworthy AI summary in the exact place that users expect to see an email subject, with no mention of it being AI-generated ?

    ai llms hallucinations yahoo email gen-ai

write hedging in Amazon DynamoDB

  • write hedging in Amazon DynamoDB

    "Write hedging" is a nice technique to address p99 tail latencies, by increasing the volume of writes (or in the case of read hedging, reads):

    Imagine you want a very low p99 read latency. One way to lower tail latencies is to hedge requests. You make a read request and then, if the response doesn't come back quickly enough, make a second equivalent hedging request and let the two race. First response wins. If the first request suffered a dropped network packet, the second request will probably win. If things are just temporarily slow somewhere, the first request will probably win. Either way, hedging helps improve the p99 metrics, at the cost of some extra read requests.

    Write hedging has a little more complexity involved, since you want to avoid accidental overwrites during races; this blog post goes into some detail on a technique to do this in DynamoDB, using timestamps. Good stuff.

    (via Last Week In AWS)

    via:lwia aws dynamodb write-hedging read-hedging p99 latencies tail-latencies optimization performance algorithms

tsdproxy

  • tsdproxy

    I'm pretty happy with my current setup for the home network, but this is one I'll keep in the back pocket for future possible use:

    [Tailscale Docker Proxy] simplifies the process of securely exposing services and Docker containers to your Tailscale network by automatically creating Tailscale machines for each tagged container. This allows services to be accessible via unique, secure URLs without the need for complex configurations or additional Tailscale containers.

    docker tailscale containers home networking

3DBenchy Enters the Public Domain

  • 3DBenchy Enters the Public Domain

    "3DBenchy, a 3D model [of an adorable little boat] designed specifically for testing and benchmarking 3D printers, is now in the public domain."

    Originally released on April 9, 2015, by Creative Tools, the model has become a beloved icon of the 3D printing community. [...] NTI has decided to release 3DBenchy to the world by making it public domain, marking its 10th anniversary with this significant gesture.

    Mark your calendars for April 9, 2025, as 3DBenchy celebrates its 10th anniversary! A special surprise is planned for the 3DBenchy community to commemorate this milestone.

    (Via Alan Butler)

    3dbenchy 3d-printing via:alan-butler ip public-domain creative-commons

A Visual Guide to Vèvè

Monzo Stand-in

  • Monzo Stand-in

    This is great -- Monzo built "Monzo Stand-in", a full "backup" of the main stack, since uptime is critical to them:

    We take reliability seriously at Monzo so we built a completely separate backup banking infrastructure called Monzo Stand-in to add another layer of defence so customers can continue to use important services provided by us. We consider Monzo Stand-in to be a backup of last resort, not our primary mechanism of providing a reliable service to our customers, by providing us with an extra line of defence.

    Monzo Stand-in is an independent set of systems that run on Google Cloud Platform (GCP) and is able to take over from our Primary Platform, which runs in Amazon Web Services (AWS), in the event of a major incident. It supports the most important features of Monzo like spending on cards, withdrawing cash, sending and receiving bank transfers, checking account balances and transactions, and freezing or unfreezing cards.

    Flashback to the old Pimms setup in AWS Network Monitoring; we had an entire duplicate stack in AWS -- every single piece duplicated and running independently.

    architecture uptime monzo banking reliability ops via:itc

Language Models Do Addition Using Helices

  • Language Models Do Addition Using Helices

    wtf:

    Mathematical reasoning is an increasingly important indicator of large language model (LLM) capabilities, yet we lack understanding of how LLMs process even simple mathematical tasks. To address this, we reverse engineer how three mid-sized LLMs compute addition. We first discover that numbers are represented in these LLMs as a generalized helix, which is strongly causally implicated for the tasks of addition and subtraction, and is also causally relevant for integer division, multiplication, and modular arithmetic. We then propose that LLMs compute addition by manipulating this generalized helix using the "Clock" algorithm: to solve a+b, the helices for a and b are manipulated to produce the a+b answer helix which is then read out to model logits. We model influential MLP outputs, attention head outputs, and even individual neuron preactivations with these helices and verify our understanding with causal interventions. By demonstrating that LLMs represent numbers on a helix and manipulate this helix to perform addition, we present the first representation-level explanation of an LLM's mathematical capability.

    llms helices trigonometry magic weird ai papers arithmetic addition subtraction

Critical Ignoring as a Core Competence for Digital Citizens

  • Critical Ignoring as a Core Competence for Digital Citizens

    "Critical ignoring" as a strategy to control and immunize one's information environment (Kozyreva et al., 2023):

    Low-quality and misleading information online can hijack people's attention, often by evoking curiosity, outrage, or anger. Resisting certain types of information and actors online requires people to adopt new mental habits that help them avoid being tempted by attention-grabbing and potentially harmful content.

    We argue that digital information literacy must include the competence of critical ignoring—choosing what to ignore and where to invest one's limited attentional capacities. We review three types of cognitive strategies for implementing critical ignoring:

    • self-nudging, in which one ignores temptations by removing them from one's digital environments;
    • lateral reading, in which one vets information by leaving the source and verifying its credibility elsewhere online;
    • and the do-not-feed-the-trolls heuristic, which advises one to not reward malicious actors with attention.

    We argue that these strategies implementing critical ignoring should be part of school curricula on digital information literacy.

    Good to give names to these practices, since we're all having to do them nowadays anyway...

    (Via Stan Carey)

    psychology trolls media kids internet literacy attention critical-ignoring ignoring papers via:stancarey

LinuxPDF

  • LinuxPDF

    It's Linux, running inside a PDF file.

    "The humble PDF file format supports JavaScript – with a limited standard library, mind you. By leveraging this, [vk6] managed to compile a RISC-V emulator (TinyEMU) into JavaScript using an old version of Emscripten targeting asm.js instead of WebAssembly. The emulator, embedded within the PDF, interfaces with virtual input through a keyboard and text box."

    (via Fuzzix)

    via:fuzzix linux pdf hacks emulation javascript emscripten tinyemu

Undergraduate Upends a 40-Year-Old Data Science Conjecture

  • Undergraduate Upends a 40-Year-Old Data Science Conjecture

    This is a great story; bonus that it's a notable improvement for the humble hash-table data structure:

    Krapivin was not held back by the conventional wisdom for the simple reason that he was unaware of it. "I did this without knowing about Yao's conjecture," he said. His explorations with tiny pointers led to a new kind of hash table — one that did not rely on uniform probing. And for this new hash table, the time required for worst-case queries and insertions is proportional to (log x)^2 — far faster than x. This result directly contradicted Yao's conjecture. Farach-Colton and Kuszmaul helped Krapivin show that (log x)^2 is the optimal, unbeatable bound for the popular class of hash tables Yao had written about.

    Paper here -- https://arxiv.org/abs/2501.02305 .

    data-structures hash-tables cs programming coding papers optimization open-addressing

PleIAs/common_corpus

  • PleIAs/common_corpus

    This is great to see:

    Common Corpus is the largest open and permissible licensed text dataset, comprising 2 trillion tokens (1,998,647,168,282 tokens). It is a diverse dataset, consisting of books, newspapers, scientific articles, government and legal documents, code, and more. Common Corpus has been created by Pleias in association with several partners and contributed in-kind to Current AI initiative.

    The dataset in its entirety meets the requirements of the Code of Conduct of the AI Act and goes further than the current requirements for data transparency. It aims to set a new standard of openness in AI, showing that detailed provenance at a granular document level is a realistic objective, even at the scale of 2 trillion tokens.

    ai llms open-data open-source pleias common-corpus corpora training ai-act

Government agency removes spoon emoji from work platform amid protests

  • Government agency removes spoon emoji from work platform amid protests

    lol. "On Wednesday, employees at the Technology Transformation Services division of the [U.S. government's General Services Administration] reportedly unleashed a torrent of spoon emojis in the chat that accompanied an organization-wide, 600-person video conference with new leader Thomas Shedd, a former Tesla engineer." ... Workers embraced the digital cutlery to protest the Trump administration's "Fork in the Road" resignation offer."

    forks spoons funny protest us-politics emojis

Are better models better?

  • Are better models better?

    This is very interesting, on the applicability and usefulness of generative AI, given their inherent error rate and probabilistic operation:

    Asking if an LLM can do very specific and precise information retrieval might be like asking if an Apple II can match the uptime of a mainframe, or asking if you can build Photoshop inside Netscape. No, they can't really do that, but that's not the point and doesn't mean they're useless. They do something else, and that 'something else' matters more and pulls in all of the investment, innovation and company creation. Maybe, 20 years later, they can do the old thing too - maybe you can run a bank on PCs and build graphics software in a browser, eventually - but that's not what matters at the beginning. They unlock something else.

    What is that 'something else' for generative AI, though? How do you think conceptually about places where that error rate is a feature, not a bug?

    (Via James Tindall)

    errors probabilistic computing ai genai llms via:james-tindall

Woof.group vs the OSA

  • Woof.group vs the OSA

    The UK's new Online Safety Act law is extremely vague, extremely punitive, and has Fediverse operators Woof.group very worried --

    Ofcom carefully avoided answering almost all of our questions. They declined to say whether ~185 users was a "significant number". Several other participants in Ofcom's livestreams also asked what a significant number meant. Every time, Ofcom responded obliquely: there are no numeric thresholds, a significant number could be "small", Ofcom could target "a one-man band", and providers are expected to have a robust justification for deciding they do not have a significant number of UK users. It is unclear how anyone could make a robust justification given this nebulous guidance. In their letter, Ofcom also declined to say whether non-commercial services have target markets, or whether pornography poses a "material risk of significant harm". In short, we have no answer as to whether Woof.group or other Fediverse instances are likely to fall in scope of the OSA.

    Do we block pre-emptively, or if and when Ofcom asks? This is the ethical question Woof.group's team, like other community forums, have been wrestling with. Ofcom would certainly like sites to take action immediately. As Hoskings warned:

    "Don't wait until it's too late. That's the message. Once you do get the breach letter, that is when it is too late. The time doesn't start ticking from then. The time is ticking from—for part five services, from January, part three from July."

    woof.group fediverse mastodon social-media uk osa laws ofcom porn blocking

Apple Ordered by UK to Create Global iCloud Encryption Backdoor

  • Apple Ordered by UK to Create Global iCloud Encryption Backdoor

    The British government has secretly demanded that Apple give it blanket access to all encrypted user content uploaded to the cloud, reports The Washington Post.

    The spying order came by way of a "technical capability notice," a document sent to Apple by the Home Secretary, ordering it to provide access under the sweeping UK Investigatory Powers Act (IPA) of 2016. Critics have labeled the legislation the "Snooper's Charter," as it authorizes law enforcement to compel assistance from companies when needed to collect evidence.

    Apple is likely to stop offering encrypted storage in the UK, rather than break the security promises it made to its users, people familiar with the matter told the publication. However, that would not affect the UK order for backdoor access to the service in other countries, including the United States. Apple has previously said it would consider pulling services such as FaceTime and iMessage from the UK rather than compromise future security.

    (via gwire)

    via:gwire apple encryption backups cloud ipa surveillance icloud backdoors security infosec

Within Bounds: Limiting AI's environmental impact

  • Within Bounds: Limiting AI's environmental impact

    A joint statement issued by the Green Screen Coalition, the Green Web Foundation, Beyond Fossil Fuels, Aspiration, and the critical infrastructure lab, regarding AI's impact on climate change:

    To meet the challenge of climate change, environmental degradation, pollution and biodiversity loss, and its attendant injustices, we urge policymakers, industry leaders and all stakeholders to acknowledge the true environmental costs of AI, to phase out fossil fuels throughout the technology supply chain, to reject false solutions, and to dedicate all necessary means to bring AI systems in line with planetary boundaries. Meeting these demands is an essential step to ensure that AI is not driving further planetary degradation and could instead support a sustainable and equitable transition.

    Their demands are:

    • I. PHASE OUT FOSSIL FUELS
    • II. COMPUTING WITHIN LIMITS
    • III. RESPONSIBLE SUPPLY CHAINS
    • IV. EQUITABLE PARTICIPATION
    • V. TRANSPARENCY

    via:climateaction climate climate-change ai fossil-fuels sustainability

cur.vantage.sh

  • cur.vantage.sh

    via Ben Schaechter: "a new microsite we've launched for the AWS community that helps with understanding billing codes present in either Cost Explorer or the CUR. We profiled the number of distinct billing codes across our customer base and have about ~60k unique billing codes. We hear all the time that FinOps practitioners and engineers are confused about the billing codes present in Cost Explorer or the Cost and Usage Report. Think of these as being things like "Requests-Tier1" for S3 or "CW:GMWI-Metrics" for CloudWatch. There is usually really limited resources for determining what these billing codes are even when you Google around for them."

    aws billing codes cost-explorer ec2 s3 finops

Words from an ex-Zizian-adjacent person

  • Words from an ex-Zizian-adjacent person

    It seems there's now a full-on Mansonesque death cult emerging from the LessWrong/rationalist/effective-altruism community: https://www.sfgate.com/bayarea/article/bay-area-death-cult-zizian-murders-20064333.php

    This HN comment was very interesting for background:

    [Former member of that world, roommates with one of Ziz's friends for a while, so I feel reasonably qualified to speak on this.] The problem with rationalists/EA as a group has never been the rationality, but the people practicing it and the cultural norms they endorse as a community.

    As relevant here:

    1) While following logical threads to their conclusions is a useful exercise, each logical step often involves some degree of rounding or unknown-unknowns. A -> B and B -> C means A -> C in a formal sense, but A -almostcertainly-> B and B -almostcertainly-> C does not mean A -almostcertainly-> C. Rationalists, by tending to overly formalist approaches, tend to lose the thread of the messiness of the real world and follow these lossy implications as though they are lossless. That leads to...

    2) Precision errors in utility calculations that are numerically-unstable. Any small chance of harm times infinity equals infinity. This framing shows up a lot in the context of AI risk, but it works in other settings too: infinity times a speck of dust in your eye >>> 1 times murder, so murder is "justified" to prevent a speck of dust in the eye of eternity. When the thing you're trying to create is infinitely good or the thing you're trying to prevent is infinitely bad, anything is justified to bring it about/prevent it respectively.

    3) Its leadership - or some of it, anyway - is extremely egotistical and borderline cult-like to begin with. I think even people who like e.g. Eliezer [Yudkowsky] would agree that he is not a humble man by any stretch of the imagination (the guy makes Neil deGrasse Tyson look like a monk). They have, in the past, responded to criticism with statements to the effect of "anyone who would criticize us for any reason is a bad person who is lying to cause us harm". That kind of framing can't help but get culty.

    4) The nature of being a "freethinker" is that you're at the mercy of your own neural circuitry. If there is a feedback loop in your brain, you'll get stuck in it, because there's no external "drag" or forcing functions to pull you back to reality. That can lead you to be a genius who sees what others cannot. It can also lead you into schizophrenia really easily. So you've got a culty environment that is particularly susceptible to internally-consistent madness, and finally:

    5) It's a bunch of very weird people who have nowhere else they feel at home. I totally get this. I'd never felt like I was in a room with people so like me, and ripping myself away from that world was not easy. (There's some folks down the thread wondering why trans people are overrepresented in this particular group: well, take your standard weird nerd, and then make two-thirds of the world hate your guts more than anything else, you might be pretty vulnerable to whoever will give you the time of day, too.)

    TLDR: isolation, very strong in-group defenses, logical "doctrine" that is formally valid and leaks in hard-to-notice ways, apocalyptic utility-scale, and being a very appealing environment for the kind of person who goes super nuts -> pretty much perfect conditions for a cult. Or multiple cults, really. Ziz's group is only one of several.

    zizians cults extropianism tescreal effective-altruism rationalism lesswrong death-cults

Burrows–Wheeler Transform

  • Burrows–Wheeler Transform

    an algorithm used to prepare data for use with data compression techniques such as bzip2. It permutes the order of characters in a string (S), sorting all the circular shifts of the text in lexicographic order, then extracting the last column and the index of the original string in the set of sorted permutations of S.

    Some day when I have lots of free time to spare, I'll spend a while getting my head around this deep magic, because it's just amazing that this works.

    (via John Regehr)

    compression algorithms burrows-wheeler-transform bzip2 via:john-regehr magic text

Irish spider zombies!

  • Irish spider zombies!

    This is fantastic -- a newly-discovered species of fungus does the same trick as Ophiocordyceps in Brazil; it infects the brains of orb-weaving cave spiders in Ireland, and induces them to leave their lairs or webs, and migrate to die in an exposed situation, in order to favor dispersal of the fungal spores.

    Ophiocordyceps is, of course, the inspiration for the zombie-forming fungus in The Last Of Us.

    cordyceps fungi ireland spiders zombies fungus nature gross

The Billion Docs JSON Challenge: ClickHouse vs. MongoDB, Elasticsearch, and more

ODROID-H4+

  • ODROID-H4+

    The next generation of the excellent ODROID SBCs; based on Intel's N97 architecture, AVX2 extensions, faster DRAM, 4 SATA ports, and up to 48GB of RAM.

    Significantly beefier in general, reportedly around the EUR180 mark in price.

    odroid sbcs n97 hardware home devices servers

Coordinated Lunar Time

  • Coordinated Lunar Time

    The moon may have a timezone of its own soon, Coordinated Lunar Time (LTC):

    Due to the moon's lower gravity and its motion relative to Earth, moon time passes 56 microseconds faster each earth day. As a result, an atomic clock on Earth would run at a different rate than an atomic clock on the moon.

    Similar to how UTC is determined, the memo suggests "an ensemble of clocks" deployed to the moon might be used to set the new time standard.

    (via David Cuthbert)

    via:david-cuthbert moon time timezones ltc

Understanding the BM25 full text search algorithm

LLM-Driven Code Completion in JetBrains IDEs

  • LLM-Driven Code Completion in JetBrains IDEs

    JetBrains have come up with a new relatively-lightweight LLM-driven code generation option, constrained to producing single line suggestions:

    The length of the completion suggestions is a trade-off. While longer suggestions do tend to reduce how many keystrokes you have to make, which is good, they also increase the number of reviews required on your end. Taking the above into account, we decided that completing a single line of code would be a fair compromise.

    Some key features:

    • It works locally and is available offline. This means you can take advantage of the feature even if you aren't connected to the internet.

    • It doesn't send any data from your machine over the internet. The language models that power full line code completion run locally, which is great for two reasons. First, your code remains safe, as it never leaves your machine. Second, there are no additional cloud-related expenses – that's why this feature comes at no additional cost.

    Also, customer code is never used for training.

    I've used this (in RubyMine), and found it fairly useful; it's good for generating the obvious next line, but is easily ignored when that's not what's needed. Not bad at all.

    coding code-completion jetbrains ides java ruby llms ai code-generation rubymine intellij

VIC 20 Elite

  • VIC 20 Elite

    Crazy stuff. Elite, ported to the Commodore VIC 20 (albeit with a 32K expansion):

    VIC 20 Elite is based on the C-64 source. VIC 20 specific graphics, text, keyboard & joystick input, and sound routines were written from scratch to replace the corresponding C-64 code.

    Of course, the complete enhanced Elite won't fit within the VIC 20's limited memory, so some features had to be left out. Following the original 1984 BBC Cassette and Acorn Electron version, the VIC 20 version omits extended planet descriptions, planetary details (craters and meridians), and the missions that appear further on in the game. The pause mode options are dropped, and there is no Find Planet option in Galactic Chart (that would be only really useful during missions).

    (via Sleepy from FP)

    retrogaming commodore emulation gaming history elite vic-20

goref

  • goref

    "a Go heap object reference analysis tool based on delve: It can display the space and object count distribution of Go memory references, which is helpful for efficiently locating memory leak issues or viewing persistent heap objects to optimize the garbage collector (GC) overhead."

    Nice to see Go supporting similar debugging/optimisation tools to those offered by the JVM.

    go heap memory gc memory-leaks

Artsy's Technology Choices evaluation process

  • Artsy's Technology Choices evaluation process

    This is a nice way to evaluate new technology options, from Artsy:

    We want to accomplish a lot with a lean team, which means we must choose stable technologies. However, we also want to adopt best-of-breed technologies or best-suited tools, which may need work or still be evolving. We've borrowed from ThoughtWorks' Radar to define the following stages for evaluating, adopting, and retiring technologies:

    • Adopt: Reasonable defaults for most work. These choices have been exercised successfully in production at Artsy and there is a critical mass of engineers comfortable working with them.
    • Trial: These technologies are being evaluated in limited production circumstances. We don't have enough production experience to recommend them for high-risk or business-critical use cases, but they may be worth consideration if your project seems like a fit.
    • Assess: Technologies we are interested in and maybe even built proofs-of-concept for, but haven't yet trialed in production.
    • Hold: Based on our experience, these technologies should be avoided. We've found them to be flawed, immature, or simply supplanted by better alternatives. In some cases these remain in legacy production uses, but we should take every opportunity to retire or migrate away.

    (Via Lar Van Der Jagt on the Last Week In AWS slack instance)

    via:lwia tech technology radar choices evaluation process architecture planning tools

API Error Design

  • API Error Design

    Some good thoughts from a SlateDB dev, regarding initial principles for errors in SlateDB, derived from experience with Kafka:

    • Keep public errors separate from internal errors. The set of public errors should be kept minimal and new errors should be highly scrutinized. For internal errors, we can go to town since they can be refactored and consolidated over time without affecting the user.
    • Public errors should be prescriptive. Can an operation be retried? Is the database left in an inconsistent state? Can a transaction be aborted? What should the user actually do when the error is encountered? The error should have clear guidance.
    • Prefer coarse error types with rich error messages. There are probably hundreds of cases where the database can enter an invalid state. We don't need a separate type for each of them. We can use a single FatalError and pack as much information into the error message as is necessary to diagnose the root cause.

    (via Chris Riccomini)

    errors api design slatedb api-design error-handling exceptions architecture

Block AI scrapers with Anubis

  • Block AI scrapers with Anubis

    Bookmarking this in case I have to use it; I have a blog-related use case that I don't want LLM scrapers to kill my blog with.

    Anubis is a man-in-the-middle HTTP proxy that requires clients to either solve or have solved a proof-of-work challenge before they can access the site. This is a very simple way to block the most common AI scrapers because they are not able to execute JavaScript to solve the challenge. The scrapers that can execute JavaScript usually don't support the modern JavaScript features that Anubis requires. In case a scraper is dedicated enough to solve the challenge, Anubis lets them through because at that point they are functionally a browser.

    The most hilarious part about how Anubis is implemented is that it triggers challenges for every request with a User-Agent containing "Mozilla". Nearly all AI scrapers (and browsers) use a User-Agent string that includes "Mozilla" in it. This means that Anubis is able to block nearly all AI scrapers without any configuration.

    throttling robots scraping ops llms bots hashcash tarpits

Cost-optimized archival in S3 using s3tar

  • Cost-optimized archival in S3 using s3tar

    "s3tar" is new to me, and looks like a perfect tool for this common use-case -- aggregation and archival of existing data on S3, which often requires aggregation into large file sizes to take advantage of S3 Glacier storage classes (which have a minimum file size of 128Kb).

    s3tar optimizes for cost and performance on the steps involved in downloading the objects, aggregating them into a tar, and putting the final tar in a specified Amazon S3 storage class using a configurable "–concat-in-memory" flag. ... The tool also offers the flexibility to upload directly to a user's preferred storage class or store the tar object in S3 Standard storage and seamlessly transition it to specific archival classes using S3 Lifecycle policies.

    The only downside of s3tar is that it doesn't support recompression, which is also a common enough requirement -- especially after aggregation of multiple small input files into a larger, more compressible archive. But hey, can't have everything.

    s3tar: https://github.com/awslabs/amazon-s3-tar-tool

    s3tar amazon s3 compression storage archival architecture aggregation logs glacier via:lwia

Cryptocurrency "market caps" and notional value

  • Cryptocurrency "market caps" and notional value

    Excellent explainer from Molly White, which explains the risk around quoting "market caps" for memecoins:

    The "market cap" measurement has become ubiquitous within and outside of crypto, and it is almost always taken at face value. Thoughtful readers might see such headlines and ask questions like "how did a '$2 trillion market' tumble without impacting traditional finance?", but I suspect most accept the number.

    When crypto projects are hacked, there are headlines about hackers stealing "$166 million worth" of tokens, when in reality the hackers only could cash out 2% of that amount (around $3 million) because their attempts to sell illiquid tokens caused the price to crash.

    molly-white memecoins bitcoin rug-pulls scams liquidity market-caps cryptocurrency

Hollo

  • Hollo

    "A federated microblogging software for single users. ActivityPub-enabled, Mastodon-compatible API, supports CommonMark and Misskey-style quotes. Hollo is designed for single-users, so you can own your instance and have full control over your data. It's perfect for personal microblogs, notes, and journals."

    Seems fairly heavyweight, however, so I probably won't be running it, but it's a nice take on the single-user-server Fediverse use case.

    fediverse mastodon hollo apps social-media blogging

GTFS-Realtime API

  • GTFS-Realtime API

    The Irish National Transport Authority have an open data API for realtime public transport information; very cool. "The GTFS-R API contains real-time updates for services provided by Dublin Bus, Bus Éireann, and Go-Ahead Ireland."

    The specification currently supports the following types of information:

    Trip updates - delays, cancellations, changed routes; Service alerts - stop moved, unforeseen events affecting a station, route or the entire network; Vehicle positions - information about the vehicles including location and congestion level

    Registration is required.

    public-transport buses trains transit nta gtfs apis open-data dublin ireland

Why the British government is so into AI

  • Why the British government is so into AI

    Interesting BlueSky thread on the topic --

    The UK Government believes several things:

    1) The AI genie is out of the bottle and cannot be put back in

    2) Embracing AI would definitely be good for the British economy

    3) Enforcing copyright on AI training would put Britain out of step with rest of the world and subsequently...

    4) Enforcing copyright would be ineffective as AI would just be trained elsewhere, cutting out Brit creatives entirely

    5) Govt's preferred option is permissive enough to be attractive to AI firms but demands transparency so at least rights holders have some recourse; the alternative is bleaker.

    Obviously, I contest all of these beliefs to one degree or another, but this is where the govt is, and it's useful to understand that. The real crux of the debate, as they see it, is how Britain's laws can practically deal with the global inevitability of AI. They believe it's untenable to make Britain a legislative pariah state for AI, and that this would not lead to good outcomes for British creatives anyway. This is a point worth considering when replying to the consultation.

    However, the govt says it's not going to implement policy before it has a technical solution for rights holders to opt-out and chase down infringements. My view is that this is difficult to the point of being pure fantasy, and either means that the govt is not serious about finding a real, effective technical solution, or this policy will be kicked indefinitely down the road. My dinner partner was optimistic a solution could be achieved within the timespan of a year or two. I just don't buy it.

    Government says it has not sided with AI firms over creative industries. However, its understanding of "not taking a side" creates a false equality between massive companies whose business relies on crime and individuals whose livelihoods will be destroyed.

    I got the sense that there is no political will whatsoever to seriously challenge firms who offer to spend big in Britain, and that any thought of holding them to account for actual crime is simply considered naive. But we do have a bit of time while govt attempts to confect their magical, easy to use, opt-out solution—time during which one or several of these AI firms might implode, making the true cost more apparent.

    uk government ai policy copyright ip britain economy future

The people should own the town square

  • The people should own the town square

    Ah, this is welcome news from Mastodon:

    We are going to transfer ownership of key Mastodon ecosystem and platform components to a new non-profit organization, affirming the intent that Mastodon should not be owned or controlled by a single individual. [...] Taking the first tentative steps almost a year ago, there are already multiple organizations involved with shepherding the Mastodon code and platform. The next 6 months will see the transformation of the Mastodon structures, shifting away from the early days' single-person ownership and enshrining the envisioned independence in a dedicated European not-for-profit entity.

    mastodon social-media open-source fediverse

Grafana and ClickHouse

Watch Duty

  • Watch Duty

    Nice to see an important public need being met here:

    The [Watch Duty] app gives users the latest alerts about fires in their area [in California] and has become a vital service for millions of users in the western U.S. struggling with the seemingly constant threat of deadly wildfires—one major reason it had over 360,000 unique visits from 8:00-8:30 a.m. local time Wednesday. And the man behind Watch Duty promises that as a nonprofit, his organization has no plans to pull an OpenAI and become a profit-seeking enterprise.

    non-profits tech watch-duty apps mobile public-good

Steve Jobs vs Ireland

  • Steve Jobs vs Ireland

    this is a great Steve Jobs story, from the engineer who wrote v1 of the MacOS X Dock:

    At one point during a trip over, Steve was talking to Bas and asked how things were coming along with the Dock. He replied something along the lines of "going well, the engineer is over from Ireland right now, etc". Steve left, and then visited my manager's manager's manager and said the fateful words (as reported to me by people who were in the room where it happened).

    "It has come to my attention that the engineer working on the Dock is in FUCKING IRELAND".

    I was told that I had to move to Cupertino. Immediately. Or else.

    I did not wish to move to the States. I liked being in Europe. Ultimately, after much consideration, many late night conversations with my wife, and even buying a guide to moving, I said no.

    They said ok then. We'll just tell Steve you did move.

    (via Niall Murphy)

    macos america osx apple history steve-jobs

Court docs allege Meta trained LLM models using pirated book trove

  • Court docs allege Meta trained LLM models using pirated book trove

    This is pretty massive:

    The [court] document claims that Meta decided to download documents from Library Genesis -- aka. "LibGen" -- to train its models. LibGen is the subject of a lawsuit brought by textbook publishers who believe it happily hosts and distributes [pirated] works [....]

    The filing from plaintiffs in the Kadrey case claims that documents produced by Meta [...] describe internal debate about accessing LibGen, a little squeamishness about using BitTorrent in the office to do so, and eventual escalation to "MZ" [Mark Zuckerberg himself], who approved use of the contentious resource. [...]

    Another filing claims that a Meta document describes how it removed copyright notifications from material downloaded from LibGen, and suggests the company did so because it realized including such text could mean a model's output would reveal it was trained on copyrighted material.

    US District Court Judge Vince Chhabria also noted that in one of the documents Meta wants to seal, an employee wrote the following:

    "If there is media coverage suggesting we have used a dataset we know to be pirated, such as LibGen, this may undermine our negotiating position with regulators on these issues."

    No shit.

    piracy meta copyright mark-zuckerberg law llama training libgen books

Bufferbloat Test

  • Bufferbloat Test

    A handy tool to test your internet connection for "bufferbloat", the error condition involving "undesirable high latency caused by other traffic on your network. It happens when a flow uses more than its fair share of the bottleneck. Bufferbloat is the primary cause of bad performance for real-time Internet applications like VoIP calls, video games, and videoconferencing."

    (My home internet connection is currently rating a C: "your latency increased considerably under load", jumping from a min/mean/p95/max of 10.7, 16.9, 23.7, 30.1ms to 35.3, 98.4, 121.0, 286.0ms under load, yikes, so looks like I need to do some optimising.)

    bufferbloat internet networking optimisation performance testing tools

Waymos don't stop for pedestrians

Garbage Day on Meta's moderation plans

  • Garbage Day on Meta's moderation plans

    This is 100% spot on, I suspect, regarding Meta's recently-announced plans to give up on content moderation:

    After 2021, the major tech platforms we've relied on since the 2010s could no longer pretend that they would ever be able to properly manage the amount of users, the amount of content, the amount of influence they "need" to exist at the size they "need" to exist at to make the amount of money they "need" to exist.

    And after sleepwalking through the Biden administration and doing the bare minimum to avoid any fingers pointed their direction about election interference last year, the companies are now fully giving up. Knowing the incoming Trump administration will not only not care, but will even reward them for it.

    The question now is, what will the EU do about it? This is a flagrant raised finger in the face of the Digital Services Act.

    moderation content ugc meta future dsa eu garbage-day

"uhtcearu"

ads.txt for a site with no ads

  • ads.txt for a site with no ads

    Don Marti: "since there's a lot of malarkey in the online advertising business, I'm putting up this file [on my website] to let the advertisers know that if someone sold you an ad and claimed it ran on here, you got burned."

    The format is defined in a specification from the IAB Tech Lab. The important part is the last line. The placeholder is how you tell the tools that are supposed to be checking this stuff that you don't have ads.

    ads don-marti hacks ads-txt web