Add structural OpenAPI schema to Tekton CRDs #8490

burigolucas · 2025-01-15T16:09:17Z

Changes

Address Issue 1461 by implementing a script to update the OpenAPI schema to Tekton CRDs using controller-gen and adding the updated CRDs with OpenAPI schema.

Revise unit tests and example YAML files to address issues caused by the now enabled API server schema validation.

Note that the current script hack/update-schemas.sh return several errors/warnings and status code 1 from execution of the controller-gen CLI due to issues when parsing the schema from the source code. Besides, processing multiple files at once is causing segmentation fault. Therefore, in the script, each file is processed separately. In addition, FIXME notes are added to be addressed in the future once new features become available in controller-gen.

Closes #1461

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
Has Tests included if any functionality added or changed
pre-commit Passed
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

action required: The structural OpenAPI schema to Tekton CRDs are added enabling API server schema validation and supporting `kubectl explain` to describe fields and structure of Tekton CRDs. Due to the API server schema validation, users should make sure Tekton CRs have a valid schema when creating or updating CRs.

linux-foundation-easycla · 2025-01-15T16:09:22Z

The committers listed above are authorized under a signed CLA.

✅ login: burigolucas / name: Lucas Burigo (129612b, bc42ec6, a9dac85)

tekton-robot · 2025-01-15T16:09:28Z

Hi @burigolucas. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

burigolucas · 2025-01-15T16:10:35Z

/kind documentation

vdemeester · 2025-01-15T17:49:20Z

/ok-to-test

burigolucas · 2025-01-16T11:13:44Z

Hi @vdemeester, I have identified issues with the CRDs. A few of them were invalid due to recursive fields. Following the fix proposed in this issue Generating CRD manifest for struct with recursive field, I have added the markers

	// +kubebuilder:pruning:PreserveUnknownFields
	// +kubebuilder:validation:Schemaless

in the affected fields and regenerated the CRDs. I have also disabled the embedded object meta as it does not provide useful schema in my opinion and only makes the files more bloated.

AFAIK, there is still one issue to be solved, namely, the CRD for the PipelineRun is too large. We reach the file limit with kubectl create. Any ideas how best to address that?

vdemeester · 2025-01-16T12:31:58Z

AFAIK, there is still one issue to be solved, namely, the CRD for the PipelineRun is too large. We reach the file limit with kubectl create. Any ideas how best to address that?

Yeah, that's was the inital problem 😅

vdemeester · 2025-01-16T12:33:01Z

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "pipelines.tekton.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (RequestEntityTooLarge): error when creating "STDIN": Request entity too large: limit is 3145728
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "tasks.tekton.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "taskruns.tekton.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

vdemeester · 2025-01-16T12:36:14Z

@burigolucas could we refer to another schema ? PipelineRun gets too big because of pipelineSpec that is, well the Pipeline.Spec field (and that one also contains Task.Spec with taskSpec). Maybe we should "skip" those "embedded" type in the schema and instead "document" or "refer" to the other schema ?

burigolucas · 2025-01-16T12:38:06Z

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "pipelines.tekton.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (RequestEntityTooLarge): error when creating "STDIN": Request entity too large: limit is 3145728
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "tasks.tekton.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "taskruns.tekton.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Are you trying kubectl apply? In that case it will fail as it cannot store the changes in the annotations. It should work with kubectl create except for the PipelineRun due to the size. If you remove the older version of the PipelineRun in the CRD, then you can also verify that the CRD is valid. Perhaps we could omit parts of the schema of the PipelineRun for the older CRD version?

vdemeester · 2025-01-16T12:44:23Z

Are you trying kubectl apply? In that case it will fail as it cannot store the changes in the annotations. It should work with kubectl create except for the PipelineRun due to the size.

Yeah it's what (kubectl apply) the CI (github workflow) does (and most our users as well 😅 )

If you remove the older version of the PipelineRun in the CRD, then you can also verify that the CRD is valid. Perhaps we could omit parts of the schema of the PipelineRun for the older CRD version?

You mean the v1beta1 part right ?

vdemeester · 2025-01-16T12:46:19Z

I would be fine only having the shema for the v1 (or the highest) version available in the CRD to be honest.
@afrittoli @tektoncd/core-maintainers what do you think ?

burigolucas · 2025-01-16T12:58:33Z

I have just tried to remove the pipelineSpec from the schema of the pipelineRun and it is now small enough, even with both versions of the CRD. But we might need to reduce other CRDs further if we need to support kubectl apply due to the limits of the annotation field.

burigolucas · 2025-01-16T13:32:13Z

Removed also the schema of TaskSpec in TaskRun and added a comment for the user to refer to the schema of Task. This also reduced size but currently Task, TaskRun, Pipeline and PipelineRun are all too large to be stored in the annotations field when using kubectl apply. Any further suggestions what else to remove from the schema to address the length of the files?

vdemeester · 2025-01-16T14:30:23Z

Removed also the schema of TaskSpec in TaskRun and added a comment for the user to refer to the schema of Task. This also reduced size but currently Task, TaskRun, Pipeline and PipelineRun are all too large to be stored in the annotations field when using kubectl apply. Any further suggestions what else to remove from the schema to address the length of the files?

I am "fine" with requiring things to be done using kubectl create and document this I guess ?
cc @afrittoli

afrittoli · 2025-01-20T14:56:54Z

Removed also the schema of TaskSpec in TaskRun and added a comment for the user to refer to the schema of Task. This also reduced size but currently Task, TaskRun, Pipeline and PipelineRun are all too large to be stored in the annotations field when using kubectl apply. Any further suggestions what else to remove from the schema to address the length of the files?

I am "fine" with requiring things to be done using kubectl create and document this I guess ? cc @afrittoli

This would be for the CRDs only, right?
Existing installation scripts and tools may break if we make this change. To provide for a smoother transition, we could add an extra release file to our releases:

release.yaml: includes sha and tag for images, includes CRDs with schema
release.notag.yaml: includes CRDs with schema
release.oldcrds.yaml: includes sha and tag for images, includes CRDs without schema

This way users could point to release.oldcrds.yaml until they migrate to installing via kubectl create.

vdemeester · 2025-01-20T15:16:16Z

@afrittoli should it be the other way around for a while. Like next release we have a release-with-schemas.yaml and we announce a given timeline for changing the default and then we do the other way around ?

afrittoli · 2025-01-20T18:13:25Z

@afrittoli should it be the other way around for a while. Like next release we have a release-with-schemas.yaml and we announce a given timeline for changing the default and then we do the other way around ?

Sure, we can do that, I was trying to avoid too many changes :D but it's safer the way you proposed

burigolucas · 2025-01-21T05:28:04Z

Should we then keep the current CRDs untouched and store the new generated CRDs with schemas in a different path?

afrittoli · 2025-01-21T11:58:42Z

Should we then keep the current CRDs untouched and store the new generated CRDs with schemas in a different path?

If possible, I would prefer to have one version of the CRDs in the repos and, at release time:

copy release.yaml to release.schemas-in-crds.yaml
strip the schemas from release.yaml

@vdemeester @burigolucas WDYT?

burigolucas · 2025-03-03T08:45:31Z

@vdemeester, @afrittoli I have revised all the previously falling e2e tests and made the required changes to fix the issues. I have also rebased the branch to a address a merge conflict and squashed all my commits to a single one as suggested in the documentation for contributors. I do not know to to fix the failing job pull-tekton-pipeline-go-coverage-df though. Could one of you please approve the workflows to check if all tests are now passing? Thanks in advance!

afrittoli · 2025-03-03T09:25:38Z

/test pull-tekton-pipeline-go-coverage-df

tekton-robot · 2025-03-03T09:25:41Z

@afrittoli: No presubmit jobs available for tektoncd/pipeline@main

In response to this:

/test pull-tekton-pipeline-go-coverage-df

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hack/update-schemas.sh

vdemeester · 2025-03-03T09:57:00Z

@burigolucas ah there is one lint issue it seems.

afrittoli · 2025-03-03T10:03:59Z

examples/v1/taskruns/beta/stepaction-passing-results.yaml

@@ -60,7 +60,7 @@ kind: TaskRun
 metadata:
  name: step-action-run
 spec:
-  TaskSpec:
+  taskSpec:


Is this example not executed in CI, or is it executed but the capital letter version was accepted somehow by Tekton?

I thought only noci wasn't executed, but maybe I am wrong and those are not either ? 😓
It sounds weird that it would have accepted the capital letter.

Without the OpenAPI schema, the capitalization was not being validated and Tekton was accepting that. That is why I added a release note that users may be affected if they were not using capitalization correctly

afrittoli · 2025-03-03T10:11:43Z

hack/update-codegen.sh

@@ -109,3 +109,6 @@ ${REPO_ROOT_DIR}/hack/update-openapigen.sh

 # Make sure the generated API reference docs are up-to-date
 ${REPO_ROOT_DIR}/hack/update-reference-docs.sh
+
+# Add structural OpenAPI schema to Tekton CRDs
+${REPO_ROOT_DIR}/hack/update-schemas.sh


After running this script, we need to verify that the generated CRDs have not been modified.
The verify-codegen.sh script achieves this for the other code gens, but it does not include the config folder, so it needs to be extended, otherwise we risk missing schema updates.

@afrittoli Fixed on last commit, see hack/verify-codegen.sh

afrittoli · 2025-03-03T10:16:37Z

@burigolucas ah there is one lint issue it seems.

It's not in a file changed by this PR though, which should not happen... @vdemeester any idea why this happened?

vdemeester · 2025-03-03T10:17:39Z

@burigolucas ah there is one lint issue it seems.

It's not in a file changed by this PR though, which should not happen... @vdemeester any idea why this happened?

I need to check, but I think the github action checks all files by default (not just the changed one).

vdemeester · 2025-03-03T10:22:22Z

@afrittoli See #8604 🙃

- Update year in hack/update-schemas.sh - Remote unused directive `//nolint:musttag`

burigolucas · 2025-03-03T11:10:32Z

@burigolucas ah there is one lint issue it seems.

It's not in a file changed by this PR though, which should not happen... @vdemeester any idea why this happened?

I need to check, but I think the github action checks all files by default (not just the changed one).

@vdemeester I removed the unused directive in the latest commit. Should I rather revert it back then?

112358fn · 2025-03-06T14:05:32Z

This looks amazing!

Now, I am sorry for the scope-creep but would it be possible to include the x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key extensions for the same objects that we have x-kubernetes-list-type. This would help enable strategic merge patch using the openapi feature of kustomize.

(It is just a suggestion and I understand that you might prefer to keep it for a separate PR)

vdemeester · 2025-03-06T16:47:09Z

This looks amazing!

Now, I am sorry for the scope-creep but would it be possible to include the x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key extensions for the same objects that we have x-kubernetes-list-type. This would help enable strategic merge patch using the openapi feature of kustomize.

(It is just a suggestion and I understand that you might prefer to keep it for a separate PR)

Given the back and forth on the PR, I would suggest we do this in a follow-up PR, but it's up to you @burigolucas.

@vdemeester I removed the unused directive in the latest commit. Should I rather revert it back then?

If the CI is green, I think it's fine to keep as is 😉

burigolucas · 2025-03-06T17:00:17Z

This looks amazing!

Now, I am sorry for the scope-creep but would it be possible to include the x-kubernetes-patch-strategy and x-kubernetes-patch-merge-key extensions for the same objects that we have x-kubernetes-list-type. This would help enable strategic merge patch using the openapi feature of kustomize.

(It is just a suggestion and I understand that you might prefer to keep it for a separate PR)

Thanks for the great suggestion. I will be implementing this in a separate PR as soon we merge this one.

@112358fn, would you mind creating a dedicated issue for this request with the explanation why this should be implemented? Perhaps not all lists should support strategic merge patch. The dedicated issue would help to keep all the related discussions in a dedicated thread.

burigolucas · 2025-03-07T00:39:57Z

@vdemeester it looks like we are having rate limiting issues with DockerHub (See the extracts of the logs below for each of the 4 failed tests). Is this something new? Could we simply retry the failed set of tests to check for flaky results?

TestExamples/v1/taskruns/dind-sidecar

message: 'the sidecar "server" in TaskRun "dind-sidecar-882ss" failed to pull
              the image "". The pod errored with the message: "Back-off pulling image "docker@sha256:74e78208fc18da48ddf8b569abe21563730845c312130bd0f0b059746a7e10f5"."'

TestExamples/v1/pipelineruns/pipelinerun

2025-03-06T16:45:05.2945141Z         >>> Pod demo-pipeline-run-1-build-image-pod Container step-build-and-push:
2025-03-06T16:45:05.2945967Z         �[36mINFO�[0m[0000] Resolved base name golang:1.18 to builder    
2025-03-06T16:45:05.2946570Z         �[36mINFO�[0m[0000] Retrieving image manifest golang:1.18        
2025-03-06T16:45:05.2947300Z         �[36mINFO�[0m[0000] Retrieving image golang:1.18 from registry index.docker.io 
2025-03-06T16:45:05.2949243Z         error building image: GET https://index.docker.io/v2/library/golang/manifests/1.18: TOOMANYREQUESTS: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit

TestExamples/v1/taskruns/step-by-digest

-03-06T16:45:14.6943112Z           - lastTransitionTime: "2025-03-06T16:45:14Z"
2025-03-06T16:45:14.6944275Z             message: 'failed to create task run pod "step-by-digest-2c4dw": translating TaskSpec
2025-03-06T16:45:14.6947703Z               to Pod: GET https://index.docker.io/v2/library/busybox/manifests/sha256:1303dbf110c57f3edf68d9f5a16c082ec06c4cf7604831669faf2c712260b5a0:
2025-03-06T16:45:14.6950438Z               TOOMANYREQUESTS: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit.
2025-03-06T16:45:14.6952083Z               Maybe invalid TaskSpec'
2025-03-06T16:45:14.6952909Z             reason: PodCreationFailed
2025-03-06T16:45:14.6953425Z             status: "False"
2025-03-06T16:45:14.6953886Z             type: Succeeded

TestExamples/v1/taskruns/entrypoint-resolution

2025-03-06T16:46:58.1804050Z           - lastTransitionTime: "2025-03-06T16:46:57Z"
2025-03-06T16:46:58.1805171Z             message: 'failed to create task run pod "entrypoint-resolution": translating TaskSpec
2025-03-06T16:46:58.1806162Z               to Pod: GET https://index.docker.io/v2/amd64/ubuntu/manifests/latest: TOOMANYREQUESTS:
2025-03-06T16:46:58.1807221Z               You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit.
2025-03-06T16:46:58.1808371Z               Maybe invalid TaskSpec'
2025-03-06T16:46:58.1808760Z             reason: PodCreationFailed
2025-03-06T16:46:58.1809364Z             status: "False"
2025-03-06T16:46:58.1809680Z             type: Succeeded

burigolucas · 2025-03-07T00:51:21Z

/retest

112358fn · 2025-03-07T12:59:06Z

@burigolucas does this issue description looks ok?

vdemeester · 2025-03-11T15:18:29Z

/retest

tekton-robot · 2025-03-11T15:34:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abayer · 2025-03-11T15:43:37Z

/lgtm

tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 15, 2025

tekton-robot requested review from afrittoli and vdemeester January 15, 2025 16:09

tekton-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 15, 2025

tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 15, 2025

tekton-robot added the kind/documentation Categorizes issue or PR as related to documentation. label Jan 15, 2025

burigolucas mentioned this pull request Jan 15, 2025

Add structural OpenAPI schema to Tekton CRDs #1461

Closed

tekton-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 15, 2025

burigolucas force-pushed the issues/1461 branch from daba633 to f1f45fb Compare January 16, 2025 13:44

vdemeester mentioned this pull request Jan 20, 2025

Support kubectl explain #3416

Closed

afrittoli reviewed Mar 3, 2025

View reviewed changes

hack/update-schemas.sh Outdated Show resolved Hide resolved

afrittoli reviewed Mar 3, 2025

View reviewed changes

Minor fixes

129612b

- Update year in hack/update-schemas.sh - Remote unused directive `//nolint:musttag`

Make sure generated CRDs are up-to-date

a9dac85

112358fn mentioned this pull request Mar 7, 2025

Add support for strategic merge patch style #8628

Open

vdemeester approved these changes Mar 11, 2025

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 11, 2025

tekton-robot assigned abayer Mar 11, 2025

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 11, 2025

tekton-robot merged commit 8b8a098 into tektoncd:main Mar 11, 2025
19 checks passed

github-project-automation bot moved this from Todo to Done in Tekton Community Roadmap Mar 11, 2025

tricktron mentioned this pull request Apr 27, 2025

Fix CRD Generation Errors #8726

Open

8 tasks

Add structural OpenAPI schema to Tekton CRDs #8490

Add structural OpenAPI schema to Tekton CRDs #8490

Conversation

burigolucas commented Jan 15, 2025 • edited by vdemeester Loading

Changes

Submitter Checklist

Release Notes

linux-foundation-easycla bot commented Jan 15, 2025 • edited Loading

tekton-robot commented Jan 15, 2025

burigolucas commented Jan 15, 2025

vdemeester commented Jan 15, 2025

burigolucas commented Jan 16, 2025

vdemeester commented Jan 16, 2025

vdemeester commented Jan 16, 2025

vdemeester commented Jan 16, 2025

burigolucas commented Jan 16, 2025

vdemeester commented Jan 16, 2025

vdemeester commented Jan 16, 2025

burigolucas commented Jan 16, 2025

burigolucas commented Jan 16, 2025 • edited Loading

vdemeester commented Jan 16, 2025

afrittoli commented Jan 20, 2025

vdemeester commented Jan 20, 2025

afrittoli commented Jan 20, 2025

burigolucas commented Jan 21, 2025

afrittoli commented Jan 21, 2025

burigolucas commented Mar 3, 2025

afrittoli commented Mar 3, 2025

tekton-robot commented Mar 3, 2025

vdemeester commented Mar 3, 2025

afrittoli Mar 3, 2025

Choose a reason for hiding this comment

vdemeester Mar 3, 2025

Choose a reason for hiding this comment

burigolucas Mar 3, 2025

Choose a reason for hiding this comment

afrittoli Mar 3, 2025

Choose a reason for hiding this comment

burigolucas Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

afrittoli commented Mar 3, 2025

vdemeester commented Mar 3, 2025

vdemeester commented Mar 3, 2025

burigolucas commented Mar 3, 2025 • edited Loading

112358fn commented Mar 6, 2025

vdemeester commented Mar 6, 2025

burigolucas commented Mar 6, 2025 • edited Loading

burigolucas commented Mar 7, 2025

burigolucas commented Mar 7, 2025

112358fn commented Mar 7, 2025

vdemeester commented Mar 11, 2025

tekton-robot commented Mar 11, 2025

abayer commented Mar 11, 2025

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

burigolucas commented Jan 15, 2025 •

edited by vdemeester

Loading

linux-foundation-easycla bot commented Jan 15, 2025 •

edited

Loading

burigolucas commented Jan 16, 2025 •

edited

Loading

burigolucas Mar 3, 2025 •

edited

Loading

burigolucas commented Mar 3, 2025 •

edited

Loading

burigolucas commented Mar 6, 2025 •

edited

Loading