/sig api-machinery

/triage accepted
Add some background info: we currently only limit the access to apiVersion, kind, metadata.name and metadata.generateName as the current documentation states:

The apiVersion, kind, metadata.name and metadata.generateName are always accessible from the root of the object and from any x-kubernetes-embedded-resource annotated objects. No other metadata properties are accessible.

The consideration behind it is mainly that we wanna metadata to be homogeneous across types.

The alternative solution to this would be ValidatingAdmissionPolicy which will give you access to namespace.

Yes, I could just use ValidatingAdmissionPolicy, but tbh if that's the solution then what's the point of this restriction on CRD level, if I can just validate whatever I want using VAP?

Preferably if possible I'd like to have the source of those validation policies near the Go structs in my codebase (it would be the case if I had an access to namespace), and not "far" away in e.g helm chart in VAP, which might get de-synchronised with the fields in the CRD by accident.

Proposal: Allow CEL Validation for metadata.namespace in CRD Embedded Resources

Overview:
Currently, CRD embedded resources lack CEL validation for the metadata.namespace field. This hinders users seeking stable identities for embedded resources across different CR instances.

Proposed Solution:
Extend CEL validation to cover metadata.namespace in embedded resources. This aligns with existing practices for other metadata fields.

Benefits:

Stable Identities: Ensure stable identities for embedded resources, crucial for scenarios like multi-tenancy or cross-namespace resource management.

Operational Efficiency: Empower users to maintain predictability and reliability in applications relying on stable identity configurations.

Feasibility and Considerations:

Implementation: Discuss the feasibility, considering impacts on workflows, backward compatibility, and implementation effort.

Challenges: Acknowledge challenges and potential trade-offs, such as added complexity or dependencies.

Addressing Concerns:

Alternative Solutions: Discuss alternatives, including ValidatingAdmissionPolicy (VAP), highlighting pros and cons.

Community Input: Encourage community feedback and diverse perspectives on the proposal.

This does feel a bit quirky, given that x-kubernetes-embedded-resource doesn't preclude a CRD author from placing further constraints on the same subschema. I tried to write a schema that would enforce immutability on metadata.namespace in an embedded resource and came up with:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: foos.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                embedme:
                  type: object
                  x-kubernetes-embedded-resource: true
                  x-kubernetes-preserve-unknown-fields: true
                  required:
                  - metadata
                  properties:
                    apiVersion:
                      type: string
                      x-kubernetes-validations:
                      - rule: "self == oldSelf"
                        message: apiVersion is immutable
                    kind:
                      type: string
                      x-kubernetes-validations:
                      - rule: "self == oldSelf"
                        message: kind is immutable
                    metadata:
                      type: object
                      required:
                      - namespace
                      - name
                      properties:
                        name:
                          type: string
                          x-kubernetes-validations:
                          - rule: "self == oldSelf"
                            message: name is immutable
                        generateName:
                          type: string
                        namespace:
                          type: string
                          x-kubernetes-validations:
                          - rule: "self == oldSelf"
                            message: namespace is immutable
  scope: Cluster
  names:
    plural: foos
    singular: foo
    kind: Foo

This seems to work for embedded resources that must be namespaced. It's awkward for a couple reasons:

1. Unless metadata.name, metadata.generateName, metadata.namespace, apiVersion, and kind are all specified in the schema with type string, the CEL rule attached to metadata.namespace is not enforced. This is apparently because WithTypeAndObjectMeta stomps the metadata subschema entirely before CEL validation in those cases. That feels like a bug to me, since other OpenAPI validators are applied without explicitly including all of those fields in the schema (either the CEL rules should be enforced, or a CRD is invalid if one of its schemas attaches x-kubernetes-validations to a metadata property of an embedded resource other than name or generateName).
2. The fields of metadata aren't visible to any CEL rules on metadata itself, preventing expressions like has(self.namespace) == has(oldSelf.namespace). If the embedded resource must always be namespaced, then namespace can be made required without the CEL extension.
3. The fields apiVersion and kind are implicitly required, but metadata, metadata.name, and metadata.namespace are not.
4. If the field containing the embedded resource is optional, clients can effectively change the immutable fields through a series of updates that first omits the embedded resource field, then replaces it outright.
5. I don't expect that controller-gen can be made to generate a schema like this.

This seems to work for embedded resources that must be namespaced. It's awkward for a couple reasons:

If I'm following, (1), (2) and (3) all result in sub-par validation compared with the root resource. I'd be very interested in seeing that improved.

my notes:

(1) - This seems like a bug. We wouldn't see it on the root because we have built-in immutability validation. Is this also a problem with OpenAPI value validations? (e.g. a regex rule or length rule)?
(2) - The ability to specify if namespace is required is clearly useful. We get that for free on root resource validation.
(3) - The correct behavior would be that name or generateName is required.. I think?
(4) - I'm less surprised by this one. If someone doesn't want an embedded resource to be entirely replaced, they need a transition rule on it matching what they expect..
(5) - ack

@aerfio A workaround is to declare all the CEL rules "above" the embedded resource:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: withembeddeds.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required:
                - embedme
              x-kubernetes-validations:
                - rule: 'oldSelf.embedme.kind == self.embedme.kind'
                - rule: 'oldSelf.embedme.apiVersion == self.embedme.apiVersion'
                - rule: 'oldSelf.embedme.metadata.name == self.embedme.metadata.name'
                - rule: 'has(self.embedme.metadata.__namespace__)'
                - rule: 'oldSelf.embedme.metadata.__namespace__ == self.embedme.metadata.__namespace__'
              properties:
                embedme:
                  type: object
                  x-kubernetes-embedded-resource: true
                  x-kubernetes-preserve-unknown-fields: true
                  required:
                  - apiVersion
                  - kind
                  - metadata
                  properties:
                    apiVersion:
                      type: string
                    kind:
                      type: string
                    metadata:
                      type: object
                      required:
                      - namespace
                      - name
                      properties:
                        name:
                          type: string
                        generateName:
                          type: string
                        namespace:
                          type: string
  scope: Namespaced
  names:
    plural: withembeddeds
    singular: withembedded
    kind: WithEmbedded

Note that all the metadata fields must be explicitly declared.. I don't know if you get this with kubebuilder.

Also, the exact validation rules will depend on:

• if you want namespaced resources, globally scoped resources, or both.
• if you want to support generateName, name, or both.
• how you want to handle the embedded resource being removed entirely from the resource.

My example assumes the embedded resource is required, namespace scoped, and uses name (not generateName).

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

/triage accepted

This is also an issue I have been facing when building validations into the CRD, being unable to validate annotations, labels, and namespace.

The use cases we see in multiple operators:

• Often annotations are used to control operand behaviour, in lieu of content in the spec. However we cant have CEL validations on annotations.
• Some objects rely on a name+namespace combination, like Routes and Services. For this an other use cases we need to limit the name+namespace character limit, which we cant do without being able to access the namespace.

Created ValidationAdminissionPolicies also requires a level of permissions which we cannot give our users/operators, without concerns for escalating privileges for other resource types on the cluster. Controlling my own CRD allows us to restrict the variation rules to only the resources I own.

I would echo @aerfio comment that it would be ideal to have full access to the metadata section in CRD validations.

This is by design. There's a long history to this topic (#74620, #77653, #80493). This led to decision for metadata to have "generic semantics independently from the object at hand" which resulted in us limiting CRD metadata validation to name (and generateName), both for OpenAPI and CEL.

@sttts @deads2k

This is by design. There's a long history to this topic (#74620, #77653, #80493). This led to decision for metadata to have "generic semantics independently from the object at hand" which resulted in us limiting CRD metadata validation to name (and generateName), both for OpenAPI and CEL.

@sttts @deads2k

But namespace is a special case. It's like a part of the name. The user can't change it anyway, and there are many use-cases for limiting this field.

For example, my use- case is a CRD with a pointer field, so my CR could point to another CR of the same type (and I need to point by name. I can't use object reference, for multiple reasons), e.g.

metadata:
  name: a-name
  namespace: a-namespace
...
spec:
  ...
  pointer:
    name: another-cr
    namespace: some-namespace
...

I want to prevent the option to point to self CR.

I was trying to add this CEL rule, and got the same issue:

'!has(self.spec.pointer) || self.spec.pointer.name != self.metadata.name || self.spec.pointer.namespace != self.metadata.namespace'

This is tricky. I can see the argument for being able to use namespace as an input for validation.

But namespace is a special case. It's like a part of the name.

What I want to prevent is CRD authors declaring validation rules about what namespaces a custom resource can be created in. This should be done using ValidatingAdmissionPolicy. But once I provide namespace as and input to CRD validation rules, it becomes possible to do what I'm trying to prevent.

I think it's best to ask that users that need to inspect namespace to write a ValidatingAdmissionPolicy to enforce that rule.