plumbing: Optimise memory consumption for filesystem storage #799

pjbgf · 2023-06-27T22:24:23Z

Previously, as part of building the index representation, the resolveObject
func would create an interim plumbing.MemoryObject, which would then be
saved into storage via storage.SetEncodedObject. This meant that objects
would be unnecessarily loaded into memory, to then be saved into disk.

The changes streamlines this process by:

Introducing the LazyObjectWriter interface which enables the write
operation to take place directly against the filesystem-based storage.
Leverage multi-writers to process the input data once, while targeting
multiple writers (e.g. hasher and storage).

An additional change relates to the caching of object info children within
Parser.get. The cache is now skipped when a seekable filesystem is being
used.

The impact of the changes can be observed when using seekable filesystem
storages, especially when cloning large repositories.

The stats below were captured by adapting the BenchmarkPlainClone test
to clone https://github.com/torvalds/linux.git:

pkg: github.com/go-git/go-git/v5
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
              │  /tmp/old   │             /tmp/new              │
              │   sec/op    │   sec/op    vs base               │
PlainClone-16   41.68 ± 17%   48.04 ± 9%  +15.27% (p=0.015 n=6)

              │   /tmp/old    │               /tmp/new               │
              │     B/op      │     B/op       vs base               │
PlainClone-16   1127.8Mi ± 7%   256.7Mi ± 50%  -77.23% (p=0.002 n=6)

              │  /tmp/old   │              /tmp/new              │
              │  allocs/op  │  allocs/op   vs base               │
PlainClone-16   3.125M ± 0%   3.800M ± 0%  +21.60% (p=0.002 n=6)

Notice that on average the memory consumption per operation is over 75%
smaller. The time per operation was increased by 15%, which may actual be less
on long running applications, due to the decreased GC pressure and the
garbage collection costs.

Relates to: #251 #315

pprof ref using previous stats as base:

pjbgf · 2023-07-01T00:25:50Z

CI is currently broken, once #802 is merged this needs rebasing.

pjbgf · 2023-07-08T11:43:32Z

Rebased and fixed issue with ParserSuite.TestThinPack.

plumbing/format/packfile/parser.go

pjbgf · 2023-07-20T20:29:43Z

CI is broken, fix will be handled on a separate PR.

matankila · 2023-09-20T19:53:49Z

@pjbgf please fix your branch so we all enjoy your performance boost :)

The changes aim to make that specific benchmark more reliable for setting a baseline which can later be use to compare against future changes on the most basic feature of go-git: plain cloning repositories.

Previously, as part of building the index representation, the resolveObject func would create an interim plumbing.MemoryObject, which would then be saved into storage via storage.SetEncodedObject. This meant that objects would be unnecessarily loaded into memory, to then be saved into disk. The changes streamlines this process by: - Introducing the LazyObjectWriter interface which enables the write operation to take places directly against the filesystem-based storage. - Leverage multi-writers to process the input data once, while targeting multiple writers (e.g. hasher and storage). An additional change relates to the caching of object info children within Parser.get. The cache is now skipped when a seekable filesystem is being used. The impact of the changes can be observed when using seekable filesystem storages, especially when cloning large repositories. The stats below were captured by adapting the BenchmarkPlainClone test to clone https://github.com/torvalds/linux.git: pkg: github.com/go-git/go-git/v5 cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz │ /tmp/old │ /tmp/new │ │ sec/op │ sec/op vs base │ PlainClone-16 41.68 ± 17% 48.04 ± 9% +15.27% (p=0.015 n=6) │ /tmp/old │ /tmp/new │ │ B/op │ B/op vs base │ PlainClone-16 1127.8Mi ± 7% 256.7Mi ± 50% -77.23% (p=0.002 n=6) │ /tmp/old │ /tmp/new │ │ allocs/op │ allocs/op vs base │ PlainClone-16 3.125M ± 0% 3.800M ± 0% +21.60% (p=0.002 n=6) Notice that on average the memory consumption per operation is over 75% smaller. The time per operation increased by 15%, which may actual be less on long running applications, due to the decreased GC pressure and the garbage collection costs. Signed-off-by: Paulo Gomes <pjbgf@linux.com>

pjbgf · 2023-10-28T03:08:31Z

PR is now rebased.

hiddeco · 2023-11-06T10:38:18Z

plumbing/format/packfile/patch_delta.go

+			return 0, plumbing.ZeroHash, err
+		}
+
+		if isCopyFromSrc(cmd) {


Seems like a switch may be more fitting here as there is non-conditional else behavior which could be captured with a default.

Good shout. I will add this change on a follow-up PR. 👍

hiddeco

Besides the tiny comment, this looks good to me. Great work @pjbgf 🙇 🥇

plumbing: Optimise memory consumption for filesystem storage

pjbgf added this to the v5.8.0 milestone Jun 27, 2023

pjbgf force-pushed the perf2 branch 3 times, most recently from 47cf139 to 60ef5df Compare July 8, 2023 11:36

pjbgf requested review from mcuadros and hiddeco July 8, 2023 11:43

pjbgf mentioned this pull request Jul 8, 2023

clone large project , fatal error:out of memory #251

Closed

hiddeco reviewed Jul 11, 2023

View reviewed changes

plumbing/format/packfile/parser.go Outdated Show resolved Hide resolved

pjbgf removed this from the v5.8.0 milestone Jul 19, 2023

pjbgf mentioned this pull request Jul 20, 2023

PackWriter.buildIndex is too slow #813

Closed

pjbgf force-pushed the perf2 branch from 60ef5df to 4efd2fb Compare July 20, 2023 20:20

pjbgf force-pushed the perf2 branch from 4efd2fb to 7d60837 Compare July 21, 2023 06:50

pjbgf mentioned this pull request Aug 5, 2023

Repository.ResolveRevision is very slow #824

Closed

pjbgf force-pushed the perf2 branch from 7d60837 to 1c361ad Compare October 28, 2023 02:51

pjbgf added 2 commits October 28, 2023 03:51

*: Improve BenchmarkPlainClone

814abc0

The changes aim to make that specific benchmark more reliable for setting a baseline which can later be use to compare against future changes on the most basic feature of go-git: plain cloning repositories.

pjbgf added this to the v5.11.0 milestone Oct 28, 2023

pjbgf added the performance label Nov 3, 2023

pjbgf mentioned this pull request Nov 6, 2023

Getting object file Reader will load content of even large blobs into memory #104

Closed

pjbgf requested a review from hiddeco November 6, 2023 10:34

hiddeco reviewed Nov 6, 2023

View reviewed changes

hiddeco approved these changes Nov 6, 2023

View reviewed changes

pjbgf merged commit 48bec23 into go-git:master Nov 6, 2023
17 checks passed

pjbgf deleted the perf2 branch November 6, 2023 17:18

traidare pushed a commit to traidare/go-git that referenced this pull request Oct 26, 2024

Merge pull request go-git#799 from pjbgf/perf2

3db9012

plumbing: Optimise memory consumption for filesystem storage

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plumbing: Optimise memory consumption for filesystem storage #799

plumbing: Optimise memory consumption for filesystem storage #799

pjbgf commented Jun 27, 2023 •

edited

Loading

pjbgf commented Jul 1, 2023

pjbgf commented Jul 8, 2023

pjbgf commented Jul 20, 2023

matankila commented Sep 20, 2023

pjbgf commented Oct 28, 2023

hiddeco Nov 6, 2023

pjbgf Nov 6, 2023

hiddeco left a comment

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

plumbing: Optimise memory consumption for filesystem storage #799

plumbing: Optimise memory consumption for filesystem storage #799

Conversation

pjbgf commented Jun 27, 2023 • edited Loading

pjbgf commented Jul 1, 2023

pjbgf commented Jul 8, 2023

pjbgf commented Jul 20, 2023

matankila commented Sep 20, 2023

pjbgf commented Oct 28, 2023

hiddeco Nov 6, 2023

Choose a reason for hiding this comment

pjbgf Nov 6, 2023

Choose a reason for hiding this comment

hiddeco left a comment

Choose a reason for hiding this comment

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

pjbgf commented Jun 27, 2023 •

edited

Loading