sqlite resolves lock contention between processes with exponential backoff. When the WAL reaches 4MB it stops all writes while it gets compacted into the database. Once the compaction is over all the waiting processes probably have retry intervals in the hundred millisecond range, and as they exit they are immediately replaced with new processes with shorter initial retry intervals. I don't know enough queuing theory to state this nicely or prove it, but I imagine the tail latency for the existing processes goes up quickly as the throughput of new processes approaches the limit of the database.
That is interesting, I’ll have to look into that further. I would expect Go to have similar issues because the RPS isn’t that much less. But maybe there is some knife edge here.
OP posited SQLite database contention. I don't know enough about this space to agree or disagree. It would be interesting, and perhaps illuminating, to perform a similar experiment with Postgres.
