Obituary: Farewell to robots.txt (1994-2025)
It is with deep sorrow that we announce the end of robots.txt, the humble text file that served as the silent guardian of digital civility for thirty years. Born on February 1, 1994, out of necessity when Martijn Koster’s server crashed under a faulty crawler named “Websnarf,” robots.txt passed away in July 2025, not by Cloudflare’s hand, but from the consequences of systematic disregard by AI corporations.
The protocol taught us that technology can be based on human values like ethics and morality. It showed that voluntary compliance works when all parties benefit. Its greatest achievement was perhaps preserving the internet for three decades from what it has become today – a soulless extraction machine.
Tags: internet history robots.txt crawlers web obituaries protocols ai via:mariafarrell