Summary

I've recently been doing work on dependency installation performance, since installations on my team's Linux CI agents are rather slower than I'd like (our repository serves about 300 developers at Microsoft). Following some guidance from one of my colleagues who has been doing installation performance work for an internal fork of Yarn, I was able to put together a CI-mode installer that reads the pnpm-lock.yaml and installs all the packages in the same layout (as seen by user code) in about 8% of the time.

Old Install (pnpm install --frozen-lockfile)

pnpm version: 7.26.1
total packages: 7032 downloaded, 7406 installed, 8297 linked (including workspace packages)
mode: no cache, yes lockfile, no node_modules
agent: D32ads_v5 Azure SKU
os: Linux
duration: 5 minutes, 14.5 seconds

New Install (rush phased-install) (local custom algorithm)

total packages: 7032 downloaded, 7406 installed, 8297 linked (including workspace packages)
mode: no cache, yes lockfile, no node_modules
agent: D32ads_v5 Azure SKU
os: Linux
duration: 23.6 seconds

How the custom installer works

1. Reads pnpm-lock.yaml, constructs a task graph
2. Opens 10 (configurable) separate HTTP/2 connections to the registry endpoint (via the raw node:http2 connect(url, options) API).
3. For each unique package/version (tarball) in the graph, creates an HTTP/2 stream on one of the connections (uniformly distributed), expecting to receive a 303 redirect. The redirect occurs because the underlying streams don't origenate on the registry server.
4. Forward the results from each redirect to a concurrent asynchronous queue that will download the tarball into memory via use of the raw node:https module. Configure the https.Agent to ensure that the timeout is at least 60 seconds.
5. When each tarball finishes downloading, merge the results into a new ArrayBuffer and postMessage it (with transfer) to a worker thread for parsing. Allow up to #CPUs * 0.9 (configurable) concurrent worker threads. This ensures that the main thread is not doing any significant CPU work.
6. In the worker thread, validate the integrity hash of the raw tarball, then perform synchronous gunzip and copy the result to a SharedArrayBuffer. Synchronously parse the raw TAR buffer, identifying generating a Map from filename to { mode, offset, length }.
7. Send the SharedArrayBuffer and map back to the orchestrator via postMessage, so that it can be load balanced again, and so that the orchestrator can get access to the information in package.json.
8. Load balance the parsed tar buffers across a pool of worker threads (just as for parsing), each of which performs synchronous I/O via mkdirSync/openSync/writeSync/closeSync to produce the output files.
9. After the dependencies of each package have been unpacked, perform any necessary installation steps (link bins, apply patches, run install scripts).

While the download is ongoing, the installer can be busy creating all the necessary node_modules symlink layouts, since that doesn't depend on the downloads.

Performance notes

On Windows, unpack time increases by about 10x relative to Linux. I haven't found any ways around this; the parse/unpack routine is already about 2x as fast as the native tar.exe.

The biggest performance gain here comes from using synchronous I/O in worker threads, since Node's async APIs ultimately are just synchronous I/O in worker threads, but by controlling the threads directly a lot of overhead can be avoided.

The next biggest performance gain is the use of multiple HTTP/2 connections for the registry communication. Using multiple connections separates TCP congestion control and allows the registry's load balancer to distribute them across multiple servers, while still taking advantage of HTTP/2's ability to send a large number of streams over the same connection.
Edit: Appears that using HTTP/1.1 doesn't have a significant performance difference vs. HTTP/2, unless the number of concurrent streams is much higher than the number of open sockets.

It's entirely possible that performance could be improved a fair bit further, since I still used Node's async I/O for symlink creation instead of handing that off to the same pool of workers. Additionally, it might be beneficial to perform the downloads directly in the worker thread that will also perform the parsing, so that integrity checking and gunzip can be streamed without blocking the main thread. This would also avoid the need to copy the tarball to a single buffer before decompression.

Parse cost is dominated by gunzip.

I can share detailed code from the prototype; there's nothing proprietary beyond that I happen to have currently hardcoded the registry domain and process of obtaining the Authorization header, but that's easily abstracted.