Skip to content

fix: handle potential DB conflict due to concurrent upload requests in postFile #19005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: handle potential DB conflict due to concurrent upload requests in postFile #19005

wants to merge 3 commits into from

Conversation

cstyan
Copy link
Contributor

@cstyan cstyan commented Jul 22, 2025

This issue manifests when users have multiple templates which rely on the same files, for example see: #17442

In the files table we have a constraint to enforce that there can only be one entry per hash, created_by combo. When running terraform apply to update templates, this can mean that the file upload can fail for some of the templates as they hit a race condition all trying to insert at the same time.

The fix here is to detect presence of the conflict error and run another GetFileByHashAndCreator. This should happen infrequently enough to not cause significant extra load on the DB. If in the future we notice that it does, we could change the underlying SQL for file insertion to run a CAS like call via the ON CONFLICT syntax.

Note that the test added here will fail without the change from the first commit. I also tested this manually via deploy.sh in my own workspace.

Apply diff example without code changes 
terraform apply --parallelism=10
var.coder_url
  Coder deployment URL

  Enter a value: ***

coderd_template.test_templates["template-a"]: Refreshing state... [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-b"]: Refreshing state... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-c"]: Refreshing state... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # coderd_template.test_templates["template-a"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-a" -> (known after apply)
        id                                = "5d457fb1-4697-4567-97e9-856627f12b1a"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-a"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "0de194f2590374f4d9f579712ff6eed00a71bf60a37c5df40d03a9640ecc5a00" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "7161d56a-1de3-43f1-afe0-f559e70712aa" -> (known after apply)
              ~ name           = "1.0.5" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-b"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-b" -> (known after apply)
        id                                = "898f2d5f-f22b-4741-abbb-ac5c3e36eb5f"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-b"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "c431e295-1026-4ca4-a467-0d875eb89f4f" -> (known after apply)
              ~ name           = "1.0.6" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-c"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-c" -> (known after apply)
        id                                = "acd53214-8bbd-402d-ab71-83fc9b5c0822"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-c"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "304215dc-bbde-4d41-8a5a-c9dd0926a96b" -> (known after apply)
              ~ name           = "1.0.6" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.
Error from apply 
Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

coderd_template.test_templates["template-b"]: Modifying... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-a"]: Modifying... [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-a"]: Modifications complete after 4s [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-b"]: Modifications complete after 6s [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
╷
│ Error: Provisioner Error
│ 
│   with coderd_template.test_templates["template-c"],
│   on main.tf line 19, in resource "coderd_template" "test_templates":
│   19: resource "coderd_template" "test_templates" {
│ 
│ failed to upload directory: POST ***/api/v2/files: unexpected
│ status code 500: Internal error saving file.
│       Error: pq: duplicate key value violates unique constraint "files_hash_created_by_key"
│
Running the apply again succeeds 
Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # coderd_template.test_templates["template-c"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-c" -> (known after apply)
        id                                = "acd53214-8bbd-402d-ab71-83fc9b5c0822"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-c"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "304215dc-bbde-4d41-8a5a-c9dd0926a96b" -> (known after apply)
              ~ name           = "1.0.6" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-c"]: Modifications complete after 3s [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Apply diff example with code changes 
coderd_template.test_templates["template-c"]: Refreshing state... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-b"]: Refreshing state... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-a"]: Refreshing state... [id=5d457fb1-4697-4567-97e9-856627f12b1a]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # coderd_template.test_templates["template-a"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-a" -> (known after apply)
        id                                = "5d457fb1-4697-4567-97e9-856627f12b1a"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-a"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54"
              ~ id             = "e130f32f-2c3e-4332-818f-fd6de980128f" -> (known after apply)
              ~ name           = "1.0.7" -> "1.0.8"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-b"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-b" -> (known after apply)
        id                                = "898f2d5f-f22b-4741-abbb-ac5c3e36eb5f"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-b"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54"
              ~ id             = "67207027-a3a0-4679-834e-1f8ac69ad31e" -> (known after apply)
              ~ name           = "1.0.7" -> "1.0.8"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-c"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-c" -> (known after apply)
        id                                = "acd53214-8bbd-402d-ab71-83fc9b5c0822"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-c"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54"
              ~ id             = "8771b486-16a2-47d8-bcc0-08d7aeae5e15" -> (known after apply)
              ~ name           = "1.0.7" -> "1.0.8"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
apply works on the first try 
Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

coderd_template.test_templates["template-a"]: Modifying... [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-b"]: Modifying... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-a"]: Modifications complete after 2s [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-c"]: Modifications complete after 5s [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-b"]: Modifications complete after 8s [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]

The template files are as follows:

Top level main.tf 
terraform {
  required_providers {
    coderd = {
      source = "coder/coderd"
    }
  }
}

provider "coderd" {
  url = var.coder_url
}

variable "coder_url" {
  description = "Coder deployment URL"
  type        = string
}

# Create 3 templates that use identical files
resource "coderd_template" "test_templates" {
  for_each = {
    "template-a" = "Template A"
    "template-b" = "Template B" 
    "template-c" = "Template C"
  }
  
  name        = each.key
  description = each.value
  
  versions = [{
    name      = "1.0.0"
    directory = "./template"  # Same directory = same files = same hash
    active    = true
    tf_vars = [
      {
        name  = "template_name"
        value = each.key  # Only difference between templates
      }
    ]
  }]
}
template/main.tf 
terraform {
  required_providers {
    coder = {
      source = "coder/coder"
    }
  }
}

variable "template_name" {
  description = "Name of the template"
  type        = string
  default     = "test"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  arch = "amd64"
  os   = "linux"
}

resource "null_resource" "example" {
  provisioner "local-exec" {
    command = "echo 'Template name is: ${var.template_name}'"
  }
}
template/README.md 
# Workspace Template

This is a test template for reproducing the file upload race condition.

## Features
- Basic workspace setup
- Agent configuration
- Multiple files to increase upload size

To test, simply modify the template files (adding a comment in each is all you need, plus updating the version #) and then terraform apply with the --parallelism flag.

cstyan added 2 commits July 22, 2025 18:48
@cstyan
handle conflict in postFile
a009ba7 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
@cstyan
add test to ensure that concurrent file uploads don't result in an error
df947ef 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
@cstyan
fix linting
f6ae297 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
brettkolodny
brettkolodny approved these changes Jul 24, 2025
View reviewed changes
Copy link
Contributor

@brettkolodny brettkolodny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thought abut logging but otherwise lgtm 👍

coderd/files.go
Comment on lines +121 to +127
if database.IsUniqueViolation(err, database.UniqueFilesHashCreatedByKey) {
// The file was uploaded by some concurrent process since the last time we checked for it, fetch it again.
file, err = api.Database.GetFileByHashAndCreator(ctx, database.GetFileByHashAndCreatorParams{
Hash: hash,
CreatedBy: apiKey.UserID,
})
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though we suspect that this won't happen often enough to be an issue, would it be worth adding a log here so that we can verify how often this happens later, or would that be needless clutter to the logs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@brettkolodny brettkolodny brettkolodny approved these changes

Assignees

@cstyan cstyan

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cstyan @brettkolodny
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy