Root cause of #6987: StaticPool (in-memory SQLite default) hands one connection to concurrent sessions, which interleave transactions — commits silently lost #13428
Replies: 4 comments 6 replies
|
Hi, I guess the docs could to a better job here, but ultimately I'm not sure this can behave any differently unless we were to enforce a single checked out connection at at a time, using an AssertionPool or similar |
|
it's not possible to run concurrent tasks against a plain SQLite "memory" database without additional options; a "memory" connection supports only one database connection at a time. for a shared memory database you need to use regular |
the section at https://docs.sqlalchemy.org/en/20/dialects/sqlite.html#using-a-memory-database-in-multiple-threads was written over a decade ago and talks about how "memory" is not allowed by the driver unless check_same_thread=False. URI connections didnt exist at that time. im having claude rewrite it now |
|
Thanks both. Since the docs section is being rewritten right now, here's validated data Your
So the data loss follows the pool in every configuration; shared cache itself is fine. Two The memdb VFS also checks out as a modern alternative (SQLite's own docs discourage Already classifies as file-db on main, sync and aiosqlite. Same no-silent-loss behavior. I have a docs draft covering the above plus tests (pool-class selection for memdb URLs, a @CaselIT on detection: I prototyped a warning in StaticPool (fires when the sole connection |
Uh oh!
There was an error while loading. Please reload this page.
Prior report
This appears to be the same failure reported in #6987 (2021, unresolved): in-memory
sqlite+aiosqlite+ concurrent workers "sometimes doesn't find rows just added," while file-backed SQLite is unaffected. @CaselIT suspected "something funny going on with the StaticPool" there; below is the mechanism, a deterministic minimal repro (including a sync/pysqlite variant showing it's not aiosqlite-specific), and a docs/guard suggestion.Describe the behavior
create_async_engine("sqlite+aiosqlite://")(or.../:memory:) silently selectsStaticPool— one shared DBAPI connection. When two separateAsyncSessions are used concurrently (one session per asyncio task, per the documented best practice), their transaction demarcations interleave on that single connection: aROLLBACKissued when one session closes discards another session's flushed-but-uncommitted INSERT; that session's subsequentCOMMITthen commits nothing. No error, warning, or log is emitted at any layer — the write is simply gone.The same mechanism reproduces with the synchronous pysqlite dialect and an explicit
StaticPool, so this is pool-architecture-level, not an aiosqlite issue.Notably, SQLAlchemy 2.0 added proactive detection of concurrent method calls on a single Session/AsyncSession — but this failure mode uses one session per task (the recommended pattern) and escapes that detection entirely, while producing the same class of corruption the detection exists to prevent.
The SQLite dialect docs recommend/auto-select StaticPool for in-memory databases and contain no warning that concurrently-open sessions on such an engine can silently lose committed data.
Expected
Either (a) concurrent checkouts of a StaticPool connection raise or warn (matching the spirit of the 2.0 concurrent-session-use detection), or (b) the SQLite/aiosqlite docs prominently warn that an in-memory (auto-StaticPool) engine must not be used by concurrently-open sessions and that violations produce silent lost writes, not errors.
To Reproduce (minimal, both variants)
Versions
Additional context
Found via a real-world test suite: a pytest conftest engine fixture using the in-memory URL (never explicitly asking for StaticPool) fed tests that ran
asyncio.gatherover sessions; a committed audit row vanished deterministically with no signal.cache=sharednamed-memory URIs also lose the row; a file-backed database behaves correctly.All reactions