Skip to content

Archives

Jetrelay

  • Jetrelay

    This is a great little hack: "jetrelay, a pub/sub server compatible with Bluesky’s “jetstream” data feed. Using a few pertinent Linux kernel features, it avoids doing almost any work itself. As a result, it’s highly efficient: it can saturate a 10 Gbps network connection with just 8 CPU cores."

    Specifically, these are the tricks in question:

    • Trick #1: Bypassing userspace with sendfile();
    • Trick #2: Handling many clients in parallel with io_uring;
    • Trick #3: Discarding old data with FALLOC_FL_PUNCH_HOLE -- this is a nice way to avoid having to rotate between multiple files, nifty.

    Tags: sendfile io_uring linux kernel hacks tools jetrelay jetstream firehose bluesky pub-sub