feat: validate presets on template import #18844
Conversation
| @@ -1582,10 +1583,63 @@ func (api *API) postTemplateVersionsByOrganization(rw http.ResponseWriter, r *ht | |||
| } | |||
| } | |||
|
|
|||
| var files fs.FS | |||
There was a problem hiding this comment.
This was previously only necessary for dynamic parameters, but we now need it in both dynamic and classic templates because we validate presets in both cases. It feels a tad wasteful if considered in isolation, but I'd like to update the classic template case that uses tfparse to use this fs.FS as well. That would be a small first step in getting rid of tfparse.
There was a problem hiding this comment.
tfparse will eventually be removed, likely in a release or two.
Maybe we should only do preset validation if using dynamic parameters. It is overloading the feature a little bit, as calling it terraform-preview or something would be more accurate.
There was a problem hiding this comment.
I'd like to update the classic template case that uses tfparse to use this fs.FS as well.
There is a tfconfig.LoadModuleFromFilesystem but IIRC I tried using this before and ran into issues.
Maybe we should only do preset validation if using dynamic parameters.
@Emyrk what potential issues do we avoid by doing this? Tying a behaviour relating to a 'GA' feature to a per-template 'Beta' feature feels like it would be unexpected.
There was a problem hiding this comment.
@johnstcn preview is a large part of the dynamic parameters feature. It just feels premature to push preview to GA for this parsing, while the rest of its use is still getting tested in Beta.
If there is a bug in preview, we will know when Beta is in the wild
| owner := coderdtest.CreateFirstUser(t, client) | ||
| templateAdmin, _ := coderdtest.CreateAnotherUser(t, client, owner.OrganizationID, rbac.RoleTemplateAdmin()) | ||
|
|
||
| for _, tt := range []struct { |
There was a problem hiding this comment.
The different failure cases are more exhaustively tested in coder/preview#149, so I'm only doing a happy and sad path here.
| @@ -483,7 +483,7 @@ require ( | |||
| require ( | |||
| github.com/coder/agentapi-sdk-go v0.0.0-20250505131810-560d1d88d225 | |||
| github.com/coder/aisdk-go v0.0.9 | |||
| github.com/coder/preview v1.0.3-0.20250701142654-c3d6e86b9393 | |||
| github.com/coder/preview v1.0.3-0.20250713201143-17616ecf763a | |||
There was a problem hiding this comment.
NB: We must not merge this PR until we've merged coder/preview#149 and updated this dependency in kind.
| @@ -1582,10 +1583,63 @@ func (api *API) postTemplateVersionsByOrganization(rw http.ResponseWriter, r *ht | |||
| } | |||
| } | |||
|
|
|||
| var files fs.FS | |||
There was a problem hiding this comment.
tfparse will eventually be removed, likely in a release or two.
Maybe we should only do preset validation if using dynamic parameters. It is overloading the feature a little bit, as calling it terraform-preview or something would be more accurate.
| switch file.Mimetype { | ||
| case "application/x-tar": | ||
| files = archivefs.FromTarReader(bytes.NewBuffer(file.Data)) | ||
| case "application/zip": | ||
| files, err = archivefs.FromZipReader(bytes.NewReader(file.Data), int64(len(file.Data))) | ||
| if err != nil { | ||
| httpapi.Write(ctx, rw, http.StatusInternalServerError, codersdk.Response{ | ||
| Message: "Internal error reading file", | ||
| Detail: "extract zip archive: " + err.Error(), | ||
| }) | ||
| return | ||
| } | ||
| default: | ||
| httpapi.Write(ctx, rw, http.StatusBadRequest, codersdk.Response{ | ||
| Message: "Unsupported file type", | ||
| Detail: fmt.Sprintf("Mimetype %q is not supported", file.Mimetype), | ||
| }) | ||
| return | ||
| } | ||
| ownerData, err := dynamicparameters.WorkspaceOwner(ctx, api.Database, organization.ID, apiKey.UserID) | ||
| if err != nil { | ||
| if httpapi.Is404Error(err) { | ||
| httpapi.Write(ctx, rw, http.StatusBadRequest, codersdk.Response{ | ||
| Message: "Internal error checking workspace tags", | ||
| Detail: fmt.Sprintf("Owner not found, uuid=%s", apiKey.UserID.String()), | ||
| }) | ||
| return | ||
| } | ||
| httpapi.Write(ctx, rw, http.StatusInternalServerError, codersdk.Response{ | ||
| Message: "Internal error checking workspace tags", | ||
| Detail: "fetch owner data: " + err.Error(), | ||
| }) | ||
| return | ||
| } | ||
|
|
||
| previewInput := preview.Input{ | ||
| PlanJSON: nil, // Template versions are before `terraform plan` | ||
| ParameterValues: nil, // No user-specified parameters | ||
| Owner: *ownerData, | ||
| Logger: stdslog.New(stdslog.DiscardHandler), | ||
| } | ||
| previewOutput, previewDiags := preview.Preview(ctx, previewInput, files) | ||
|
|
||
| // Validate presets on template version import to avoid errors that would | ||
| // have caused workspace creation to fail: | ||
| presetErr := dynamicparameters.CheckPresets(previewOutput, nil) | ||
| if presetErr != nil { | ||
| code, resp := presetErr.Response() | ||
| httpapi.Write(ctx, rw, code, resp) | ||
| return | ||
| } | ||
|
|
||
| var parsedTags map[string]string | ||
| var ok bool | ||
| if dynamicTemplate { | ||
| parsedTags, ok = api.dynamicTemplateVersionTags(ctx, rw, organization.ID, apiKey.UserID, file) | ||
| parsedTags, ok = api.dynamicTemplateVersionTags(ctx, rw, previewOutput, previewDiags) |
There was a problem hiding this comment.
Could we leave this extracted to its own function and just rename it with different outputs?
I ask because this postTemplateVersionsByOrganization is kind of long already. And ideally, we only use preview if DynamicParameters is enabled on the template. Since the usage of preview is still in Beta.
func (api *API) templatePreview(ctx context.Context, rw http.ResponseWriter, orgID uuid.UUID, owner uuid.UUID, file database.File) (ParsedOutput, bool)There was a problem hiding this comment.
do you feel strongly that the only public interface of preview should be preview.Preview()? Some of this discomfort that you're describing here is imo a result of the preview API being too course. If we split Params, Tags and Presets into three separate functions in the preview project then we have the granular control we need to be able to decouple things and leave most of this logic down where it was.
There was a problem hiding this comment.
@SasSwart We might want to split it. If we can keep it as a single preview.Preview, that is ideal. But I understand what you are saying.
A few customers and users are already running into preset validation issues and it breaks their workspaces. I'd prefer to get preset validation in for classic templates if we can. |
| @@ -26,6 +26,14 @@ func tagValidationError(diags hcl.Diagnostics) *DiagnosticError { | |||
| } | |||
| } | |||
|
|
|||
| func presetValidationError(diags hcl.Diagnostics) *DiagnosticError { | |||
| return &DiagnosticError{ | |||
| Message: "Unable to parse presets", | |||
There was a problem hiding this comment.
nit:
| Message: "Unable to parse presets", | |
| Message: "Unable to validate presets", |
| // Validate presets on template version import to avoid errors that would | ||
| // have caused workspace creation to fail: |
There was a problem hiding this comment.
small nit:
| // Validate presets on template version import to avoid errors that would | |
| // have caused workspace creation to fail: | |
| // Fails early if presets are invalid to prevent downstream workspace creation errors |
| require.Zero(t, tv.MatchedProvisioners.Available) | ||
| require.Zero(t, tv.MatchedProvisioners.MostRecentlySeen.Time) | ||
| } else { | ||
| require.ErrorContains(t, err, tt.expectError) |
There was a problem hiding this comment.
Should we check that no provisioner job was created?
|
|
||
| // Validate presets on template version import to avoid errors that would | ||
| // have caused workspace creation to fail: | ||
| presetErr := dynamicparameters.CheckPresets(previewOutput, nil) |
There was a problem hiding this comment.
Is there any case where we call CheckPresets with a hcl.Diagnostics? Should we maybe remove this parameter?
@SasSwart I just fear a bug in preview could then break existing templates. We had some parsing bugs that prevented some templates from working. |
| rejectMu sync.RWMutex | ||
| reject bool | ||
| } | ||
|
|
||
| // SetReject toggles whether GetGitSSHKey should return an error or passthrough to the underlying store. | ||
| func (d *dbRejectGitSSHKey) SetReject(reject bool) { | ||
| d.rejectMu.Lock() | ||
| defer d.rejectMu.Unlock() | ||
| d.reject = reject |
There was a problem hiding this comment.
nit: Is this a fix for a separate flake? If so, might be no harm to separate in its own PR.
| @@ -1582,10 +1583,63 @@ func (api *API) postTemplateVersionsByOrganization(rw http.ResponseWriter, r *ht | |||
| } | |||
| } | |||
|
|
|||
| var files fs.FS | |||
There was a problem hiding this comment.
I'd like to update the classic template case that uses tfparse to use this fs.FS as well.
There is a tfconfig.LoadModuleFromFilesystem but IIRC I tried using this before and ran into issues.
Maybe we should only do preset validation if using dynamic parameters.
@Emyrk what potential issues do we avoid by doing this? Tying a behaviour relating to a 'GA' feature to a per-template 'Beta' feature feels like it would be unexpected.
WalkthroughThis change introduces backend validation and error handling for dynamic template version presets. It adds new functions for preset diagnostics, updates the control flow for template version creation to validate presets, and extends tests to cover valid and invalid preset scenarios. Dependency updates and minor test refactoring are also included. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant API
participant Preview
participant DB
Client->>API: POST /templateversions (with archive)
API->>API: Extract files from archive
API->>DB: Fetch workspace owner data
API->>Preview: Run preview on extracted files
Preview-->>API: Return preview output, diagnostics
API->>API: CheckPresets(preview output, diagnostics)
alt Preset validation error
API-->>Client: Return error response
else Preset validation ok
API->>API: dynamicTemplateVersionTags(preview output, diagnostics)
API-->>Client: Return tags or success response
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15–20 minutes Assessment against linked issues
Assessment against linked issues: Out-of-scope changes
Possibly related PRs
Suggested reviewers
Poem
Warning Review ran into problems🔥 Problemsresponse.data.map is not a function ✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
coderd/dynamicparameters/error.go (1)
31-31: Consider updating the error message as suggested in the previous review.The message "Unable to parse presets" should be "Unable to validate presets" to better reflect the function's purpose and align with the validation context.
go.mod (1)
486-486: Verify that the prerequisite PR has been merged.Based on the previous comment, this PR should not be merged until coder/preview#149 has been merged and the dependency updated. Please confirm that this prerequisite has been satisfied.
🧹 Nitpick comments (1)
coderd/templateversions_test.go (1)
645-755: Well-structured preset validation test.The test correctly follows the parallel testing pattern, uses unique identifiers, and covers both valid and invalid preset scenarios appropriately. The structure aligns well with other tests in the file.
[past_review_comments]
Regarding the past comment about checking if no provisioner job was created for the invalid case - this would be a good addition to ensure the validation fails before job creation rather than during job execution.Consider adding a check that no provisioner job is created for the invalid preset case:
} else { require.ErrorContains(t, err, tt.expectError) + // Verify no provisioner job was created for invalid preset + _, err := store.GetProvisionerJobByID(ctx, tv.Job.ID) + require.Error(t, err, "no job should be created for invalid preset") }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (6)
coderd/dynamicparameters/error.go(1 hunks)coderd/dynamicparameters/presets.go(1 hunks)coderd/parameters_test.go(3 hunks)coderd/templateversions.go(3 hunks)coderd/templateversions_test.go(1 hunks)go.mod(1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.go
📄 CodeRabbit Inference Engine (.cursorrules)
**/*.go: The codebase is rigorously linted with golangci-lint to maintain consistent code quality.
Coder emphasizes clear error handling, with specific patterns required: Concise error messages that avoid phrases like "failed to"; Wrapping errors with%wto maintain error chains; Using sentinel errors with the "err" prefix (e.g.,errNotFound).
**/*.go: NEVER use time.Sleep to mitigate timing issues. If an issue seems like it should use time.Sleep, read through https://github.com/coder/quartz and specifically the README to better understand how to handle timing issues.
Return OAuth2-compliant error responses using writeOAuth2Error(ctx, rw, http.StatusBadRequest, "invalid_grant", "description") for OAuth2 errors.
Follow Uber Go Style Guide for Go code style.
Use dbauthz.AsSystemRestricted(ctx) for public endpoints needing system access and plain ctx for authenticated endpoints with user context.
Files:
coderd/dynamicparameters/error.gocoderd/dynamicparameters/presets.gocoderd/parameters_test.gocoderd/templateversions_test.gocoderd/templateversions.go
**/*.{go,sql,ts,tsx,js,jsx,md}
📄 CodeRabbit Inference Engine (CLAUDE.md)
Ensure files end with a newline to avoid missing newlines.
Files:
coderd/dynamicparameters/error.gocoderd/dynamicparameters/presets.gocoderd/parameters_test.gocoderd/templateversions_test.gocoderd/templateversions.go
**/*_test.go
📄 CodeRabbit Inference Engine (.cursorrules)
**/*_test.go: All tests must uset.Parallel()to run concurrently, which improves test suite performance and helps identify race conditions.
All tests should run in parallel usingt.Parallel()to ensure efficient testing and expose potential race conditions.Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Files:
coderd/parameters_test.gocoderd/templateversions_test.go
🧠 Learnings (4)
go.mod (2)
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*.go : Coder emphasizes clear error handling, with specific patterns required: Concise error messages that avoid phrases like "failed to"; Wrapping errors with %w to maintain error chains; Using sentinel errors with the "err" prefix (e.g., errNotFound).
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*.go : The codebase is rigorously linted with golangci-lint to maintain consistent code quality.
coderd/parameters_test.go (5)
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:07.777Z
Learning: Applies to **/*_test.go : Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:10.063Z
Learning: Applies to **/*_test.go : Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:07.155Z
Learning: Applies to **/*_test.go : Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*_test.go : All tests must use t.Parallel() to run concurrently, which improves test suite performance and helps identify race conditions.
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*_test.go : All tests should run in parallel using t.Parallel() to ensure efficient testing and expose potential race conditions.
coderd/templateversions_test.go (7)
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*_test.go : All tests must use t.Parallel() to run concurrently, which improves test suite performance and helps identify race conditions.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:07.777Z
Learning: Applies to **/*_test.go : Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:10.063Z
Learning: Applies to **/*_test.go : Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:07.155Z
Learning: Applies to **/*_test.go : Use unique identifiers (e.g., fmt.Sprintf("test-client-%s-%d", t.Name(), time.Now().UnixNano())) and never use hardcoded names in concurrent tests to prevent race conditions.
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to coderd/coderdtest/**/* : The coderdtest package in coderd/coderdtest/ provides utilities for creating test instances of the Coder server, setting up test users and workspaces, and mocking external components.
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*_test.go : All tests should run in parallel using t.Parallel() to ensure efficient testing and expose potential race conditions.
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to coderdenttest/**/* : Enterprise features have dedicated test utilities in the coderdenttest package.
coderd/templateversions.go (4)
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to **/*.go : Coder emphasizes clear error handling, with specific patterns required: Concise error messages that avoid phrases like "failed to"; Wrapping errors with %w to maintain error chains; Using sentinel errors with the "err" prefix (e.g., errNotFound).
Learnt from: CR
PR: coder/coder#0
File: .cursorrules:0-0
Timestamp: 2025-07-21T14:32:43.064Z
Learning: Applies to coderd/coderd.go : The REST API is defined in coderd/coderd.go and uses Chi for HTTP routing.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:07.777Z
Learning: Applies to **/*.go : Return OAuth2-compliant error responses using writeOAuth2Error(ctx, rw, http.StatusBadRequest, "invalid_grant", "description") for OAuth2 errors.
Learnt from: CR
PR: coder/coder#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-28T00:33:10.063Z
Learning: Applies to **/*.go : Return OAuth2-compliant error responses using writeOAuth2Error(ctx, rw, http.StatusBadRequest, "invalid_grant", "description") for OAuth2 errors.
🔇 Additional comments (9)
coderd/dynamicparameters/presets.go (1)
13-24: LGTM! Clean and well-structured preset validation logic.The function correctly aggregates preset-specific diagnostics and follows the established error handling patterns in the codebase. The logic is clear and handles the error aggregation appropriately.
coderd/parameters_test.go (4)
6-6: LGTM: Required import for mutex usage.The
syncimport is needed for the new mutex-based error injection control.
419-428: Excellent improvement to test error injection.The addition of mutex-protected error control significantly improves test flexibility by allowing dynamic toggling of error behavior instead of always failing. The thread-safety implementation is correct using RWMutex for efficient concurrent access.
[past_review_comments]
Regarding the past comment about this being a separate flake fix - this change does improve test determinism and could help with flakiness by providing more controlled error injection.
430-440: Clean implementation of conditional error injection.The method correctly uses RLock for reading the shared state and properly delegates to the underlying store when not rejecting. The mutex usage pattern is optimal - releasing the lock before the potentially slow database call.
197-197: Proper usage of the improved error injection mechanism.The test correctly creates the wrapper, passes it to setup, and enables error injection at the appropriate time. This allows initial setup to succeed while still testing error handling in the target scenario.
Also applies to: 205-205, 212-213
coderd/templateversions.go (4)
19-19: LGTM! Import addition is necessary.The
hcl/v2import is required for thehcl.Diagnosticstype used in the refactored function signature.
1586-1637: LGTM! Well-structured file processing and preset validation.The refactoring centralizes file extraction and preview logic effectively. Key strengths:
- Comprehensive error handling for unsupported file types and extraction failures
- Proper preset validation that addresses the PR objective of preventing broken workspace builds
- Clean separation of concerns by moving shared logic to the main handler
- Follows coding guidelines with concise error messages and proper error handling
The preset validation using
dynamicparameters.CheckPresetswill catch invalid presets early and provide clear error messages, which aligns perfectly with the PR objectives.
1642-1642: Function call updated correctly for refactored signature.The call to
dynamicTemplateVersionTagsnow passes the processed preview results instead of raw file data, which is consistent with the refactoring that moved file processing logic to the main handler.
1819-1819: Excellent refactoring that improves separation of concerns.The function signature change eliminates duplicate file processing logic and creates a cleaner separation where the main handler manages file extraction while this function focuses solely on tag validation from preview results. This addresses the concerns raised in past review comments about extracting this logic.
| func CheckPresets(output *preview.Output, diags hcl.Diagnostics) *DiagnosticError { | ||
| de := presetValidationError(diags) | ||
| presets := output.Presets |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add nil check for the output parameter.
The function should guard against nil output parameter to prevent potential panic on line 12.
func CheckPresets(output *preview.Output, diags hcl.Diagnostics) *DiagnosticError {
+ if output == nil {
+ return nil
+ }
de := presetValidationError(diags)
presets := output.Presets🤖 Prompt for AI Agents
In coderd/dynamicparameters/presets.go around lines 10 to 12, add a nil check
for the output parameter at the start of the CheckPresets function to prevent a
panic when accessing output.Presets. If output is nil, return an appropriate
error or handle the case gracefully before proceeding.
Typos and other errors often result in invalid presets in a template. Coder would import these broken templates and present them to users when they create workspaces. An unsuspecting user who chooses a broken preset would then experience a failed workspace build with no obvious error message.
This PR adds additional validation beyond what is possible in the Terraform provider schema. Coder will now present a more helpful error message to template authors when they upload a new template version:
The frontend warning is less helpful right now, but I'd like to address that in a follow-up since I need frontend help:
closes #17333
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Chores