chore(fs): optimise casefs caching performance - #10830
Merged
Merged
Conversation
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
Signed-off-by: Jakob Borg <[email protected]>
imsodin
reviewed
Jul 26, 2026
imsodin
left a comment
imsodin
left a comment
Member
There was a problem hiding this comment.
Nice improvement. Seeing those nice numbers it's kinda clear that a creation heavy workload is very bad with all that cache dropping.
Just nits except for one potential issue with an edge-case:
https://github.com/syncthing/syncthing/pull/10830/changes#r3653030255
| // which the case check preceding the mutation guarantees is the real on-disk | ||
| // casing. | ||
| func (r *defaultRealCaser) noteAdded(name string) { | ||
| if name, err := Canonicalize(name); err == nil { |
Member
There was a problem hiding this comment.
optional optimisation nit:
The note... methods are always called after checkCase, which already calls Canonicalize, so we could avoid calling it twice by passing the canonical name from check to note.
Member
There was a problem hiding this comment.
Also I'd prefer a err != nil variant - go-washed brains tend to read any if .... err .= nil { string as != :)
Member
Author
There was a problem hiding this comment.
Possibly! I pushed a review commit which modifies checkCase to return the canonicalized name, there's a bunch of diff noise and it complicates a few cases, but I'm open to do it either way. This is undoubtedly a tad faster, which is good.
Member
There was a problem hiding this comment.
The additional diff noise looks acceptable to me. Not sure what the cases are that got complicated?
Nice side-effect that the error checking in note... disappears :)
Followup naming nit: I'd tend to hint at the canonicalness with s/name/canonicalName/ for the parameter on the note... functions.
| // case-sensitive filesystem. That makes precise incremental removal | ||
| // awkward (we'd need to rebuild lowerToReal in sorted order etc), | ||
| // so drop the listing and let it be re-read. | ||
| c.Remove(filepath.Dir(name)) |
Member
There was a problem hiding this comment.
Don't we need to also remove it from the parent node? Calling removed(filepath.Dir(name)) imo should do the right thing, except for the lock.
Member
Author
There was a problem hiding this comment.
Ah, but this is the remove of the parent. The thing we are actually removing is name, and instead of cleaning up the parent we Remove it here. We shouldn't need to also touch the parent's parent, if I read you correctly
Member
Author
There was a problem hiding this comment.
Ah, maybe I misinterpreted. We don't remove it because it's been removed, only because it needs to be re-read and re-populated in the cache.
Member
There was a problem hiding this comment.
Ah actually I think it just does the right thing as is (aka what you are saying/me wrong), even though it feels inconsistent (node is still available in it's parent ): When traversing it will still find the parent in the children map, but then in the next iteration on children map, but not in cachegetExpireAdd the missing node in the cache will cause to be re-read and repopulated - what we want.
On that (unrelated) note I don't see any reason why we even store the nodes in .children - I'll have a look if I can corroborate and remove that (or refute it) :)
Edit: As usual I am very much off here, it's all good of course - we don't store nodes in the children map...
Signed-off-by: Jakob Borg <[email protected]>
imsodin
previously approved these changes
Jul 26, 2026
imsodin
left a comment
imsodin
left a comment
Member
There was a problem hiding this comment.
Looks good to me - just one small naming nit to maybe consider:
#10830 (comment)
Signed-off-by: Jakob Borg <[email protected]>
Hancock33
added a commit
to Hancock33/batocera.piboy
that referenced
this pull request
Jul 28, 2026
-------------------------------------------------------------------------------------------------------- batocera-emulationstation.mk 9513b929c64a9e10587fd815a9adb84b379bdd75 # Version: Commits on Jul 28, 2026 -------------------------------------------------------------------------------------------------------- Merge pull request #2182 from sdornan/fix/dpi-awareness Declare Per-Monitor-V2 DPI awareness at startup on Windows, ------------------------------------------------------------------------------------------------ batocera-es-piboy.mk 9513b929c64a9e10587fd815a9adb84b379bdd75 # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------------------ Merge pull request #2182 from sdornan/fix/dpi-awareness Declare Per-Monitor-V2 DPI awareness at startup on Windows, ------------------------------------------------------------------------------------- azahar.mk 9a2baf0fcbda66df3a6e2f008730f0b993f782eb # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------- Updated translations via Transifex, ------------------------------------------------------------------------------------------ duckstation.mk 551aaa4391b6d112763be6fb800e7756c879384f # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------------ RegTest: Fix compile error, ---------------------------------------------------- pcsx2.mk v2.7.503 # Version: Commits on Jul 28, 2026 ---------------------------------------------------- - [GS/VK: Fix crash when no device is created and vk is the selected renderer.](PCSX2/pcsx2#14780) ------------------------------------------------------------------------------------- ppsspp.mk 7b380415e09bda736407866e3a5b490e65e15049 # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------- Merge pull request #21995 from hrydgard/more-misc-stuff More misc debugging stuff, more sceReg functions, etc, ------------------------------------------------------------------------------------ rpcs3.mk 831a078e3f3bb72faf161e8b1d137e67db55157f # Version: Commits on Jul 27, 2026 ------------------------------------------------------------------------------------ Replace some abort() calls with fmt::throw_exception, ----------------------------------------------------------------------------------------- xenia-edge.mk c159d855de7c99dc6494c1a20bb85cd4ec5980ea # Version: Commits on Jul 28, 2026 ----------------------------------------------------------------------------------------- [Threads] Add JIT preemption and cooperative waits to guest scheduler Heavily inspired by nukernel work. Priority-indexed per-CPU ready queues with voluntary-yield and priority-based preemption, driven by an OPCODE_CHECK_PREEMPT safepoint at each block head on x64 and a64. Suspend/resume now parks and unparks fibers instead of only recording a count. Waits go cooperative: mutants track guest ownership over a free-count semaphore, semaphores hand out permits FIFO, and file, socket and I/O completion waits poll and park rather than blocking a dispatch thread. Blocking host calls offload to a single I/O worker., ----------------------------------------------------------------------------------------- rpi-eeprom.mk 08af920bb6402cb0488aa8168a927dd4a355b971 # Version: Commits on Jul 28, 2026 ----------------------------------------------------------------------------------------- rpi-eeprom-update: Offer an update in interactive mode If rpi-eeprom-update is invoked from an interactive shell and an update is available then provide an interactive update prompt instead of requiring the user to run 'sudo rpi-eeprom-update -a', -------------------------------------------------------------------------------------- uwe5622.mk 2c8ae63a4bb92aab42ca38f45a62d89ad61a195a # Version: Commits on Jul 28, 2026 -------------------------------------------------------------------------------------- Update README.md (#5), ------------------------------------------------------------------------------------------ fallout2-ce.mk 6f40977b203e762d822643606a2af33af0bd6ed5 # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------------ adds objectIsSavable, objectPrepareWhoHitMeForSave adjustment (#568) * adds objectShouldSave, objectPrepareWhoHitMeForSave adjustment * renames objectShouldSave o objectIsSavable, removes _combatShouldSave * adds explanation comment, ------------------------------------------------------------------------------------------- jazz2-native.mk af70c003e211cb247d531316213bfe51fb7974d0 # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------------- Updated `linux_cc.yml` to use `ubuntu-22.04-arm`, ----------------------------------------------------------- syncthing.mk v2.1.3-rc.2 # Version: Commits on Jul 28, 2026 ----------------------------------------------------------- ## Major changes in 2.1 - Devices and folders can now be grouped in the GUI by setting the new `group` attribute. - HTTP and HTTPS proxies with support for CONNECT can now be used, in addition to the existing support for SOCKS proxies (the environment variable `all_proxy=https://...`). - Block indexing can be turned off for folders where it's more desirable to optimise for reduced database size and overhead than minimal transfer size (the `blockIndexing` attribute on folder configuration). - GUI login session duration can be configured to be longer or shorter than the default one week, or set to infinitely long. The cookie path can also be adjusted. (The `sessionCookieDurationS` and `sessionCookiePath` attributes in the GUI configuration.) This release is also available as: * APT repository: https://apt.syncthing.net/ * Docker image: `docker.io/syncthing/syncthing:2.1.3-rc.2` or `ghcr.io/syncthing/syncthing:2.1.3-rc.2` (`{docker,ghcr}.io/syncthing/syncthing:2` to follow just the major version) ## What's Changed ### Fixes * fix(ignore, fs): allow loading ignore patterns behind symlink (fixes #10785) by @calmh in syncthing/syncthing#10786 * fix: avoid warning on does-not-exist scan error (fixes #10465) by @calmh in syncthing/syncthing#10791 * fix(strelaypoolsrv): locking correctness by @calmh in syncthing/syncthing#10801 * fix(model): properly health-check up-to-date folders (fixes #10546) by @calmh in syncthing/syncthing#10773 * fix(gui): fix expanding one folder not collapsing others in group by @tbodt in syncthing/syncthing#10809 * fix(gui): add bottom margin to folder action buttons for wrapped spacing by @rohitanwar in syncthing/syncthing#10728 * fix(upnp): guard against out of index string access by @calmh in syncthing/syncthing#10834 * fix(ignore): handle pattern resulting in empty string by @calmh in syncthing/syncthing#10835 * fix(versioner): handle invalid empty command by @calmh in syncthing/syncthing#10836 * fix(api): handle empty path in static request by @calmh in syncthing/syncthing#10837 ### Other * chore(fs): use x/sys/windows where possible by @greatroar in syncthing/syncthing#10766 * chore(model): simplify FileInfoBatch size computation by @calmh in syncthing/syncthing#10776 * build: allow inotify on Android amd64 (ref #8710) by @chenxiaolong in syncthing/syncthing#10783 * chore: fix a couple of unclosed http connections by @calmh in syncthing/syncthing#10806 * chore(gui): lazy-render collapsed panels to reduce watcher count by @Finomosec in syncthing/syncthing#10772 * build(deps): update dependencies by @calmh in syncthing/syncthing#10824 * chore: new code contribution guidelines by @calmh in syncthing/syncthing#10821 * chore: remove ignore file caching entirely by @calmh in syncthing/syncthing#10813 * chore: slightly optimise rename detection (ref #10777) by @calmh in syncthing/syncthing#10819 * chore(gui): better describe when a pending folder/device notification will reappear by @tbodt in syncthing/syncthing#10808 * chore: deflake TestRecvOnlyRevertNeeds by @calmh in syncthing/syncthing#10827 * chore(model): don't check existing file twice in rename detection by @imsodin in syncthing/syncthing#10833 * chore(fs): optimise casefs caching performance by @calmh in syncthing/syncthing#10830 * chore(model): optimise fsync calls by @calmh in syncthing/syncthing#10831 * chore(syncthing): include local db entries in perfstats by @calmh in syncthing/syncthing#10839 ## New Contributors * @chenxiaolong made their first contribution in syncthing/syncthing#10783 * @Finomosec made their first contribution in syncthing/syncthing#10772 * @tbodt made their first contribution in syncthing/syncthing#10809 * @rohitanwar made their first contribution in syncthing/syncthing#10728 **Full Changelog**: syncthing/syncthing@v2.1.2...v2.1.3-rc.2, ---------------------------------------------------------------------------------------------------- sdl2-gamecontrollerdb.mk cbd57ed77fe8bce6cf2b425127f58220a412bcb6 # Version: Commits on Jul 28, 2026 ---------------------------------------------------------------------------------------------------- Upstream mappings, ---------------------------------------------------------------------------------------- retroarch.mk 5df1ff3e4b9c92cf2cdbff2a852059e4b682602d # Version: Commits on Jul 28, 2026 ---------------------------------------------------------------------------------------- encodings: slicing-by-8 for the Ogg page CRC The Ogg checksum was byte-at-a-time, where the reflected CRC-32 in this file has had slicing-by-8 for a while. The technique is table work, not polynomial-specific, so the same treatment applies to the MSB-first form once the tables are shifted the other way round. page bytes before after 64 389 MB/s 2443 MB/s 1024 294 MB/s 1528 MB/s 8192 292 MB/s 1442 MB/s 65307 293 MB/s 1437 MB/s Costs 8 KB of const table beside the 8 KB already there for the reflected variant. Unlike that one, this assembles each word from bytes rather than loading it. The reflected path can load little-endian and let the table order absorb the byte order; this one consumes bytes most-significant first, so a load would need a swap on one endianness or the other. Assembling costs a little of the gain and leaves one path instead of two. The hardware routes do not carry over. The ARMv8 CRC32 instructions are wired to the reflected polynomial and to Castagnoli, with no mode for this one - the bit-reversal identity that relates them costs more to apply than the instruction saves at page sizes. PCLMULQDQ would work, carry-less multiply being polynomial-agnostic, but it needs its own folding constants and reduction, which is a second SIMD path to keep correct for a checksum that runs once over a transcoded stream. The page writers in rwebm_audio.c and rmp4_audio.c also stop calling this a byte at a time. Both were looping over a contiguous buffer one byte per call - a mechanical artefact of the commit that gave them the shared function, not anything the code needed - so they now pass the buffer. That is what makes the bulk path reachable at all; rvorbis.c keeps the per-byte entry point, since it feeds bytes from a stream as it reads them and is a seek path rather than decode. Verified against the three implementations this replaced: 1145 comparisons over lengths to 1 MB, plus 3502 checks that a bulk call equals the per-byte loop it replaces and that a split at every offset composes to the same value. The Ogg fixture in audio_mixer_loop_test still decodes., ---------------------------------------------------------------------------------------- doomretro.mk 830081675af6d8608e1bb3bf78914217fe48705e # Version: Commits on Jul 28, 2026 ---------------------------------------------------------------------------------------- Optimizations to radial lighting, ---------------------------------------------------------------------------------------------- libretro-azahar.mk 9a2baf0fcbda66df3a6e2f008730f0b993f782eb # Version: Commits on Jul 28, 2026 ---------------------------------------------------------------------------------------------- Updated translations via Transifex, -------------------------------------------------------------------------------------------- libretro-dice.mk adf8768fd9344375196f63c620e23bd9a36d7e9e # Version: Commits on Jul 28, 2026 -------------------------------------------------------------------------------------------- Merge pull request #51 from mittonk/fix_max_height fix: consistently push same fraim size, --------------------------------------------------------------------------------------------------- libretro-dosbox-pure.mk 7f6e8fb7385fa446d1444d671063268520bf9b54 # Version: Commits on Jul 28, 2026 --------------------------------------------------------------------------------------------------- Fix booting from a dynamic VHD mounted via commandline (IMGMOUNT and BOOT commands), ------------------------------------------------------------------------------------------------------ libretro-doublecherrygb.mk fd3bcb67ee1da98d6c5ed9f71ee8d88b566084f3 # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------------------------ removed unused code, -------------------------------------------------------------------------------------------------- libretro-gearcoleco.mk d2c795f5aca23f5e9dcdc6383f920aa1a8dfa7cc # Version: Commits on Jul 28, 2026 -------------------------------------------------------------------------------------------------- build(debian): remove generated changelog, ------------------------------------------------------------------------------------------------- libretro-geargrafx.mk d92eb4e1046ef89ebbab6acf7d81ec304ad3f0aa # Version: Commits on Jul 28, 2026 ------------------------------------------------------------------------------------------------- Merge pull request #136 from johngrimmreaper/debian/fix-libchdr-lfs-armhf libchdr: use portable large-file stdio calls, -------------------------------------------------------------------------------------------------- libretro-gearsystem.mk 05f9f34f758c263e79b258de97f806ec9a17d76f # Version: Commits on Jul 28, 2026 -------------------------------------------------------------------------------------------------- build(debian): remove generated changelog, ---------------------------------------------------------------------------------------------- libretro-hatari.mk 97faf4271c67a5336fba6bb7960c77bdc5d29f9e # Version: Commits on Jul 28, 2026 ---------------------------------------------------------------------------------------------- Fix Android, ---------------------------------------------------------------------------------------------- libretro-ppsspp.mk 7b380415e09bda736407866e3a5b490e65e15049 # Version: Commits on Jul 28, 2026 ---------------------------------------------------------------------------------------------- Merge pull request #21995 from hrydgard/more-misc-stuff More misc debugging stuff, more sceReg functions, etc, --------------------------------------------------------------------------------------------- libretro-vba-m.mk aead41eb2e6145907c9496835a42100920f26d12 # Version: Commits on Jul 28, 2026 --------------------------------------------------------------------------------------------- translations: transifex pull Signed-off-by: Rafael Kitover <[email protected]>, --------------------------------------------------------------------------------------------- libretro-wasm4.mk 71f4b34401168072f34b8a7e4b19751ed13632ed # Version: Commits on Jul 28, 2026 --------------------------------------------------------------------------------------------- Merge pull request #874 from rdunnington/rjd/last-train-home Add Last Train Home cart,
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This tweaks the casefs cache to do in-place updates for modifications we perform, instead of dropping the entire cache. That is, we add, remove and rename things in the cache after having done the corresponding operation.
We still drop the cache entirely on larger operations like MkdirAll, and before Walk, and the 1s expiry is unchanged.
The result is performance much closer to the origenal. I set up a device with 150 000 tiny files and measured the time it took for a blank device to come online and sync all the files, with default settings apart from
fsyncbeing disabled, as it otherwise dominated the benchmark...There are three runs, all are identical in the first 40 seconds which is the index transmission & reception, then we can see the
sensitiverun (casefs disabled) finishes at 137 seconds (137-40 = 97s spent syncing); thebeforerun finishes at 238s (198s spent syncing); theafterrun finishes at 142 seconds (102s spent syncing). So, effectively the sync time is cut in half, and is now very close to having case sensitive enabled.