What's the issue?

Running a parallel backfill of a partitioned asset under dagster dev can cause DbtProject.prepare_if_dev() to fail intermittently with:

dagster_dbt.errors.DagsterDbtCliRuntimeError:
The dbt CLI process with command `dbt parse --quiet` failed.

Compilation Error
  dbt found 4 package(s) specified in packages.yml, but only 0 package(s)
  installed in dbt_packages. Run "dbt deps" to install package dependencies.

When a partition backfill launches (running dg dev), every worker calls prepare_if_dev() while reconstructing the dbt project.

The partitioned asset does not necessarily need to invoke dbt itself. It is enough for its definitions to import a module containing the DbtProject and prepare_if_dev() call—for example, to declare a dependency on a dbt asset.

With multiple partition runs starting concurrently, multiple processes therefore prepare the same dbt project directory.

What did you expect to happen?

No race condition should happen.

How to reproduce?

1. Create a dbt project with a packages.yml containing at least one package.
2. Install the packages once so dbt_packages/ already exists.
3. Define a DbtProject and call prepare_if_dev() at module import time.
4. Add a partitioned Dagster asset to the same code location:

import dagster as dg

partitions = dg.DailyPartitionsDefinition(start_date="2026-01-01")

@dg.asset(partitions_def=partitions)
def partitioned_asset(context: dg.AssetExecutionContext) -> None:
    context.log.info("Processing %s", context.partition_key)

5. Start the code location using dagster dev.
6. Launch a backfill covering many partitions, configured as multiple runs with parallel run execution.
7. Observe multiple run workers reconstructing the repository and calling prepare_if_dev() concurrently.
8. After enough concurrent starts, one worker can run dbt parse while another runs dbt deps, causing the missing-packages compilation error.

Using several dbt packages or increasing the backfill run concurrency makes the race easier to reproduce.

Dagster version

1.13.8

Deployment type

Local

Deployment details

It only happens when using dagster locally with dg dev.

I am using dbt 1.11.11 with Python 3.12 on Linux.

Additional information

This appears to be a regression introduced in:

• Dagster 1.13.8
• dagster-dbt 0.29.8

The same partitioned backfill is stable with:

• Dagster 1.13.7
• dagster-dbt 0.29.7

This appears related to the following change authored by @smackesey:
055e40f

Before 0.29.8, DagsterDbtProjectPreparer.prepare() only ran dbt deps when project.has_uninstalled_deps was true. Starting with 0.29.8, it runs dbt deps whenever a dependency file exists. Consequently, every run worker can execute dbt deps, even when dbt_packages/ is already populated and the dependencies have not changed.

The preparation implementation uses separate locks:

• dbt deps is guarded through package-lock.concurrent-update-lock.
• dbt parse is guarded through target/manifest.concurrent-update-lock.

This serializes concurrent deps processes with each other and concurrent parse processes with each other, but it does not prevent dbt deps in one process from overlapping dbt parse in another process.

The dbt log shows this sequence with separate process IDs:

21:18:34.363 PID 171548 starts: dbt parse --quiet
21:18:34.942 PID 171549 starts: dbt deps --quiet
21:18:35.351 PID 171548 fails:
  dbt found 4 package(s) specified in packages.yml,
  but only 0 package(s) installed in dbt_packages
21:18:37.986 PID 171549 completes dbt deps successfully

It appears that dbt parse observes dbt_packages/ while the concurrent dbt deps process is replacing or repopulating that directory.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.