-
This is a great little hack: "jetrelay, a pub/sub server compatible with Bluesky’s “jetstream” data feed. Using a few pertinent Linux kernel features, it avoids doing almost any work itself. As a result, it’s highly efficient: it can saturate a 10 Gbps network connection with just 8 CPU cores."
Specifically, these are the tricks in question:
- Trick #1: Bypassing userspace with sendfile();
- Trick #2: Handling many clients in parallel with io_uring;
- Trick #3: Discarding old data with FALLOC_FL_PUNCH_HOLE -- this is a nice way to avoid having to rotate between multiple files, nifty.
Tags: sendfile io_uring linux kernel hacks tools jetrelay jetstream firehose bluesky pub-sub