gh-135386: Fix "unable to open database file" errors on readonly DB #135566
Conversation
…M errors on readonly DB
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
There was a problem hiding this comment.
Please add a test case. You can refer to the devguide for how to do that.
Misc/NEWS.d/next/Library/2025-06-16-15-00-13.gh-issue-135386.lNrxLc.rst
Outdated
Show resolved
Hide resolved
707f4a3 to
779faf2
Compare
779faf2 to
3e3a3ba
Compare
|
No need to force-push, we squash at the end. |
Got it, thanks for the reminder! |
|
FYI, tests are failing on Windows. |
Thanks for the suggestion. I've skipped the test on Windows using |
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Sure, that makes sense — I'd be happy to look into making the test work on Windows as well. Also, thanks for the suggestions — I've adopted the URI change and simplified the flag check as suggested. : ) |
I've updated the test so it no longer skips on Windows — instead, it now explicitly closes the database before setting read-only permissions, and restores the original permissions after the test. |
Lib/test/test_dbm_sqlite3.py
Outdated
| def test_readonly_open_without_wal_shm(self): | ||
| wal_path = self.filename + "-wal" | ||
| shm_path = self.filename + "-shm" | ||
|
|
||
| for suffix in wal_path, shm_path: | ||
| os_helper.unlink(suffix) | ||
|
|
||
| try: | ||
| self.db.close() | ||
| except Exception: | ||
| pass | ||
|
|
||
| os.chmod(self.filename, stat.S_IREAD) | ||
|
|
||
| db = dbm_sqlite3.open(self.filename, "r") | ||
| try: | ||
| self.assertEqual(db[b"key1"], b"value1") | ||
| self.assertEqual(db[b"key2"], b"value2") | ||
| finally: | ||
| db.close() | ||
|
|
||
| os.chmod(self.filename, stat.S_IWRITE) |
There was a problem hiding this comment.
IIUC correctly, this change makes it so that the WAL/SHM files are no longer created in read-only mode? In that case, we could simplify the test to simply open the database in read-only mode and check that the auxiliary files are not written. I'm thinking something like this:
class Immutable(unittest.TestCase):
def setUp(self):
self.filename = os_helper.TESTFN
self.db = dbm_sqlite3.open(self.filename, "r")
def tearDown(self):
self.db.close()
def test_readonly_open_without_wal_shm(self):
wal_path = self.filename + "-wal"
shm_path = self.filename + "-shm"
self.assertFalse(os.path.exists(wal_path))
self.assertFalse(os.path.exists(shm_path))Or does this also affect the database file itself? i.e. does passing immutable=1 allow opening read-only files whereas otherwise it would be impossible?
There was a problem hiding this comment.
Thanks, I’ve replaced the original test with your suggested version in a new Immutable test class. It’s much cleaner and focused.
There was a problem hiding this comment.
Or does this also affect the database file itself? i.e. does passing
immutable=1allow opening read-only files whereas otherwise it would be impossible?
Yes, exactly — without immutable=1, even in mode=ro, SQLite may still attempt operations that require write access (like writing lock files or checking for journal/WAL state), and will fail if the database file is read-only or in a read-only directory.
Passing immutable=1 tells SQLite to treat the database as entirely static and read-only, which avoids those operations and allows the file to be opened successfully even with 0444 permissions or from a read-only mount.
|
In order to use https://www.sqlite.org/uri.html#recognized_query_parameters says
|
That's a good point — I agree That said, in the context of The use of If needed, we could consider adding a separate flag like 'ri' in the future for stricter use cases, but I think using |
There was a problem hiding this comment.
Thank you for your PR @GeneralK1ng. I have some notes about the test.
It does not actually test the database file is in a read-only directory. I suggest to create a directory, then create a database in that directory, then make both the directory readonly, then try to open and use it. You can also add a test that tries to open it in read-write mode to check how this fails. You can also add tests that make the database file itself readonly instead of the directory. Use os_helper.rmtree() for cleanup.
Thanks a lot for the detailed and thoughtful feedback, your suggestions make perfect sense and I really appreciate the guidance on improving the test coverage. I'm currently a bit tied up and might not be able to get to it in the next couple of days, but I’ll follow up soon and revise the tests as suggested. |
Thanks for the detailed suggestions! I’ve replaced the previous test with new ones that explicitly test:
Let me know if there’s anything else I can adjust. |
There was a problem hiding this comment.
Thank you for your update @GeneralK1ng. Here is a new batch of comments.
Isn't it a restriction of |
You're right — the restriction on concurrent access is explicitly stated in My use of The patch doesn’t try to enforce concurrency restrictions at the |
|
I skip the tests that check for 'w' mode failure in read-only directories without WAL/SHM. On Windows and Linux, SQLite appears to fall back gracefully and allows writing to the database even without those files. @serhiy-storchaka |
|
One thing puzzles me -- why these changes affect the result of the last two tests? They only affect the reading mode, right? But on Linux, attempt to change the DB in a read-only directory raises "dbm.sqlite3.error: attempt to write a readonly database" in main, and don't raise any exception on this PR code. I don't understand this. |
|
https://www.sqlite.org/wal.html#persistence_of_wal_mode
https://www.sqlite.org/wal.html#readonly
|
|
It seems that I missed something, but is not the SQLite database only opened in read-write mode in these tests? And is not this PR only affects opening in read-only mode? |
What
This PR adds the SQLite URI parameter
immutable=1when opening a database in read-only mode inLib/dbm/sqlite3.py. This explicitly informs SQLite that the database is read-only and avoids attempts to create or write to auxiliary WAL/SHM files.Why
Without this flag, opening a read-only SQLite database may fail with errors such as "unable to open database file" when the WAL or SHM files cannot be created due to filesystem permissions or other restrictions. This is especially common when the database file is read-only and the environment prevents creation of additional files.
Adding
immutable=1allows SQLite to operate correctly without needing to create WAL/SHM files, thus preventing these errors.Where
The change is made in the
_Database.__init__constructor inLib/dbm/sqlite3.py. The URI used to open the database connection is appended with&immutable=1only if theflagis"ro"(read-only).Related issue
#135386
This is my first contribution to CPython. I appreciate the opportunity and look forward to feedback from the community. Thank you!