SQLite has Write-Ahead Logging
TIL! Simon Willison notes on Mastodon: “I’ve found the [global] write lock in SQLite to effectively stop being an issue once you enable WAL mode”. I did not know that SQLite had a write-ahead log mode. Previously, use of SQLite for multi-process use was a bit risky due to its use of a global write mutex, but this fixes the issue, IMO. Simon’s benchmarking tests with Django: https://simonwillison.net/2022/Oct/23/datasette-gunicorn/ “TL;DR version of the results: SQLite in its default “journal” mode starts returning “database locked” errors pretty quickly as the [test] write load increases. But if you switch to “wal” mode those errors straight up vanish! I was expecting WAL mode to improve things, but I thought I’d still be able to hit errors even with it enabled. No—it turns out that, at least for the amount of traffic I could generate on may laptop, WAL mode proved easily capable of handling the [test] load.” ‘WAL journal mode supports one writer and many readers at the same time. A second writer will have to wait until the first write transaction is committed or rolled back.’ Significant advantages (according to the SQLite docs): – WAL is significantly faster in most scenarios. – WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently. – Disk I/O operations tends to be more sequential using WAL. – WAL uses many fewer fsync() operations and is thus less vulnerable to problems on systems where the fsync() system call is broken. The WAL is easy to enable: simply run `sqlite-utils enable-wal db.sqlite3` on an existing SQLite database file with no running users.
(tags: databases performance unix sqlite wordpress django wal concurrency)
-
Tony Finch on the PCG64 DXSM random number generator:
It is a relatively new flavour of PCG, which addresses a minor shortcoming of the original pcg64 that arose in the discussion when NumPy originally adopted PCG. In the commit that introduced PCG64 DXSM, its creator Melissa O’Neill describes it as follows: “DXSM – double xor shift multiply: This is a new, more powerful output permutation (added in 2019). It’s a more comprehensive scrambling than RXS M, but runs faster on 128-bit types. Although primarily intended for use at large sizes, also works at smaller sizes as well.” As well as the DXSM output permutation, pcg64_dxsm() uses a “cheap multiplier”, i.e. a 64-bit value half the width of the state, instead of a 128-bit value the same width as the state. The same multiplier is used for the LCG and the output permutation. The cheap multiplier improves performance: pcg64_dxsm() has fewer full-size 128 bit calculations.
(tags: pcg pcg64-dxsm rngs randomness algorithms performance random-numbers cryptography)
-
A thoughtful post from Bert Hubert, who is doing a good job on this side of things!
I and many of my friends are struggling to be, or at least feel, useful. Most of our professional opportunities are not particularly useful. If you are a ‘project lifecycle manager’ at a bland corporation, it can be hard to convince yourself you are achieving anything good for the world. […] Although there are many corporate jobs furthering inclusivity, sustainability and other worthy things, the work there largely consists of getting certifications or having people do the right kind of training. Often very little actual sustainability or inclusion is going on, and even if there is, your role in such a department is pretty far away from the action. But, unlike the project lifecycle manager, you can at least tell yourself your efforts are intended towards creating a better world. But, back to our challenge: how can we be useful, how can we try to contribute to at least trying to make things better? Because things aren’t looking that great for climate, societies, peace and democracies worldwide.
(tags: being-useful usefulness jobs work life career bert-hubert society)