Allow generating `nullable.Nullable` for nullable properties #1404

sonasingh46 · 2023-12-21T22:39:21Z

This PR adds capability to use Nullable types via opt in flag for Nullable types

Ref Issue: #1039

Maintainer edit: Release notes

examples/go.mod

pkg/codegen/configuration.go

pkg/codegen/schema.go

jamietanna · 2024-01-10T11:12:52Z

internal/test/go.mod

@@ -13,6 +13,7 @@ require (
 	github.com/gorilla/mux v1.8.0
 	github.com/kataras/iris/v12 v12.2.6-0.20230908161203-24ba4e8933b9
 	github.com/labstack/echo/v4 v4.11.3
+	github.com/oapi-codegen/nullable v1.0.0


I've pushed a new release (docs changes only) so worth updating too:

Suggested change

github.com/oapi-codegen/nullable v1.0.0

github.com/oapi-codegen/nullable v1.0.1

jamietanna · 2024-01-10T11:17:15Z

pkg/codegen/schema.go

 	if !p.Schema.SkipOptionalPointer &&
 		(!p.Required || p.Nullable ||
 			(p.ReadOnly && (!p.Required || !globalState.options.Compatibility.DisableRequiredReadOnlyAsPointer)) ||
 			p.WriteOnly) {

 		typeDef = "*" + typeDef
 	}
+


No. Wondering if this gets caught by the linter? I remember running make lint.
I will remove it anyways.

pkg/codegen/schema_test.go

jamietanna · 2024-01-10T16:26:09Z

internal/test/issues/issue-1039/issue_test.go

+
+	gotJSON, err := json.Marshal(updateReq)
+	require.NoError(t, err)
+	fmt.Println(string(gotJSON))


Suggested change

fmt.Println(string(gotJSON))

jamietanna · 2024-01-10T16:26:20Z

internal/test/issues/issue-1039/issue_test.go

+
+	gotJSON2, err := json.Marshal(updateReq2)
+	require.NoError(t, err)
+	fmt.Println(string(gotJSON2))


Suggested change

fmt.Println(string(gotJSON2))

jamietanna · 2024-01-10T16:26:58Z

pkg/codegen/schema.go

@@ -681,10 +685,15 @@ func GenFieldsFromProperties(props []Property) []string {

 		field += fmt.Sprintf("    %s %s", goFieldName, p.GoTypeDef())

-		omitEmpty := !p.Nullable &&
-			(!p.Required || p.ReadOnly || p.WriteOnly) &&
+		commonConditions := (!p.Required || p.ReadOnly || p.WriteOnly) &&


Not sure I like this name - maybe shouldOmitEmpty?

jamietanna · 2024-01-10T16:27:44Z

internal/test/issues/issue-1039/issue_test.go

+func TestIssue(t *testing.T) {
+	swagger, err := openapi3.NewLoader().LoadFromData(spec)
+	require.NoError(t, err)
+
+	opts := codegen.Configuration{
+		PackageName: "issue1039",
+		Generate: codegen.GenerateOptions{
+			ChiServer:    true,
+			Client:       true,
+			Models:       true,
+			EmbeddedSpec: true,
+			NullableType: true,
+		},
+	}
+
+	_, err = codegen.Generate(swagger, opts)
+	require.NoError(t, err)
+}


Was this just to check that generating the code works? If so, this shouldn't be needed - by having the go:generates in this package, that's good enough, unless there's something else you're trying to test?

Yes. Removed it.

jamietanna · 2024-01-10T16:28:33Z

internal/test/issues/issue-1039/spec.yaml

Is this the minimal spec we could use? If we can trim it down it'd be appreciated, so it's clear for long-term maintenance what's actually important about this

Sure. Let me look into this.

Update the spec.

jamietanna · 2024-01-11T09:07:16Z

pkg/codegen/configuration.go

@@ -35,6 +35,7 @@ type GenerateOptions struct {
 	Client        bool `yaml:"client,omitempty"`         // Client specifies whether to generate client boilerplate
 	Models        bool `yaml:"models,omitempty"`         // Models specifies whether to generate type definitions
 	EmbeddedSpec  bool `yaml:"embedded-spec,omitempty"`  // Whether to embed the swagger spec in the generated code
+	NullableType  bool `yaml:"nullable-type,omitempty"`  // Whether to generate nullable type for nullable fields


Sorry, would this be able to move to the OutputOptions? That's a better place for it

jamietanna · 2024-01-11T09:09:25Z

internal/test/issues/issue-1039/issue_test.go

+func ToStringPointer(str string) *string {
+	return &str
+}


nit: but doesn't need to be exported:

Suggested change

func ToStringPointer(str string) *string {

return &str

}

func toStringPointer(str string) *string {

return &str

}

jamietanna · 2024-01-11T09:09:57Z

internal/test/issues/issue-1039/issue_test.go

+func ToStringPointer(str string) *string {
+	return &str
+}


alternatively with generics:

Suggested change

func ToStringPointer(str string) *string {

return &str

}

func ptr[T any](v T) *T {

return &v

}

jamietanna · 2024-01-11T09:11:31Z

pkg/codegen/schema_test.go

@@ -213,3 +213,243 @@ func TestProperty_GoTypeDef(t *testing.T) {
 		})
 	}
 }
+
+func TestPropertyNullable_GoTypeDef(t *testing.T) {


This test should be collapsed in with the above, no? At the very least:

Suggested change

func TestPropertyNullable_GoTypeDef(t *testing.T) {

func TestProperty_GoTypeDef_nullable(t *testing.T) {

To indicate what it's testing

jamietanna

Couple of comments, and just want to make sure that we've made sure this is the minimal set of specs + tests needed to validate the functionality?

jamietanna · 2024-01-11T09:22:18Z

Draft release notes:

Notable features

`Nullable` types,

It's possible that you want to be able to determine whether a field isn't sent, is sent as null or has a value.

For instance, if you had the following OpenAPI property:

##### TODO indentation
S:
    type: object
    properties:
        Field:
          type: string
          nullable: true
    required: []

The current behaviour in oapi-codegen is to generate:

type S struct {
	Field *string `json:"field,omitempty"`
}

However, you lose the ability to understand the three cases, as there's no way to distinguish two of the types from each other:

is this field not sent? (Can be checked with S.Field == nil)
is this field null? (Can be checked with S.Field == nil)
does this field have a value? (S.Field != nil && *S.Field == "123")

Therefore, as requested in #1039, this is now possible to represent with the nullable.Nullable type from our new library, oapi-codegen/nullable.

If you configure your generator's Output Options as so:

output-options:
  nullable-type: true

You will now receive the following output:

type S struct {
    Field nullable.Nullable[string] `json:"field,omitempty"`
}

Note that this is opt-in only, due to it being a break in existing signatures and behaviour.

You can find out more about how this works in a blog post with further details.

jamietanna · 2024-01-11T11:53:47Z

internal/test/issues/issue-1039/issue_test.go

+	// exclude some fields in patch request
+	patchReq2 := issue1039nullable.PatchRequest{
+		BreakoutGroup: &issue1039nullable.BreakoutGroup{
+			Name: ToStringPointer("test-bk"),
+		},
+		Organization: &issue1039nullable.Organization{
+			AliasName: ToStringPointer("org-alias-name"),
+			Name:      ToStringPointer("org-name"),
+		},
+		Popularity: &popularity,
+	}


Suggested change

// exclude some fields in patch request

patchReq2 := issue1039nullable.PatchRequest{

BreakoutGroup: &issue1039nullable.BreakoutGroup{

Name: ToStringPointer("test-bk"),

},

Organization: &issue1039nullable.Organization{

AliasName: ToStringPointer("org-alias-name"),

Name: ToStringPointer("org-name"),

},

Popularity: &popularity,

}

// omit some fields

patchReq2 := issue1039nullable.PatchRequest{

BreakoutGroup: &issue1039nullable.BreakoutGroup{

Name: ToStringPointer("test-bk"),

},

// HomeCountry is omitted

Organization: &issue1039nullable.Organization{

AliasName: ToStringPointer("org-alias-name"),

Name: ToStringPointer("org-name"),

},

Popularity: &popularity,

// SearchBox is omitted

}

jamietanna · 2024-01-11T11:56:16Z

internal/test/issues/issue-1039/issue_test.go

+				assert.Falsef(t, obj.BreakoutGroup.IsNull(), "breakout group should not be null")
+
+				// check for non-nullable field
+				assert.Truef(t, obj.HomeCountry == nil, "home country should be nil")


Suggested change

assert.Truef(t, obj.HomeCountry == nil, "home country should be nil")

assert.Nilf(t, obj.HomeCountry, "home country should be nil")

jamietanna · 2024-01-11T11:58:03Z

internal/test/issues/issue-1039/type-config-nullabletype-disabled.yaml

+  embedded-spec: true
+output-options:
+  skip-prune: true
+output: nullable-disabled/types.gen.go


I think this would be clearer if this and the other package were named differently - nullabledisabled is a bit unclear at a first glance.

Maybe defaultbehaviour?

jamietanna · 2024-01-11T11:59:12Z

internal/test/issues/issue-1039/issue_test.go

+	return &str
+}
+
+func TestNullableDisabled(t *testing.T) {


I'd also recommend moving this into the package with the default behaviour, or making it clearer what this is - it was a little confusing reading this one first, then seeing the new behaviour - maybe we should keep this next to the existing behaviour's generated code?

jamietanna · 2024-01-11T12:01:32Z

internal/test/issues/issue-1039/types.gen.go

+	BreakoutGroup nullable.Nullable[BreakoutGroup] `json:"breakout_group"`
+
+	// HomeCountry Home country of the user.
+	HomeCountry *HomeCountry `json:"home_country,omitempty"`


At first glance of the tests, I wasn't sure why this was nil

It may be worth us tweaking the spec to make the property names clearer what they're testing - i.e. this could be OptionalButNotNullableProperty then it's super clear what's being tested (and why it's being checked for nil instead of expecting a nullable.Nullable)

This a very good point. Let me do that

jamietanna · 2024-01-11T12:02:33Z

internal/test/issues/issue-1039/issue_test.go

+		},
+
+		{
+			name: "when search box is null and organization has non zero value",


If this is the key part of what we're testing, I'm not sure we should also be checking i.e. the Popularity, etc?

I have checking that to ensure that because the other fields are not provided -- it should be basically not set and not null. But I think now when I am reading your comment, probably, we do not need to duplicate this in every test case.

The empty test case should already cover that and I will remove this

jamietanna · 2024-01-11T12:02:46Z

internal/test/issues/issue-1039/issue_test.go

+				// check for nullable field
+				// check for nullable fields


internal/test/issues/issue-1039/defaultbehaviour/defaultbehaviour_test.go

jamietanna · 2024-01-11T15:18:54Z

internal/test/issues/issue-1039/type-config-defaultbehaviour.yaml

+  old-aliasing: true
+  old-merge-schemas: true
+  disable-flatten-additional-properties: true


are these definitely required to make this a minimal reproduction case?

jamietanna · 2024-01-11T15:19:03Z

internal/test/issues/issue-1039/type-config-defaultbehaviour.yaml

+package: defaultbehaviour
+generate:
+  models: true
+  embedded-spec: true


not needed any more?

jamietanna · 2024-01-11T15:19:13Z

internal/test/issues/issue-1039/types-config.yaml

+package: issue1039
+generate:
+  models: true
+  embedded-spec: true


jamietanna · 2024-01-11T15:19:20Z

internal/test/issues/issue-1039/types-config.yaml

+compatibility:
+  old-aliasing: true
+  old-merge-schemas: true
+  disable-flatten-additional-properties: true


jamietanna · 2024-01-11T17:20:25Z

internal/test/issues/issue-1039/types-config.yaml

+  old-aliasing: true
+  old-merge-schemas: true


jamietanna · 2024-01-11T17:20:32Z

internal/test/issues/issue-1039/type-config-defaultbehaviour.yaml

+compatibility:
+  old-aliasing: true
+  old-merge-schemas: true


jamietanna · 2024-01-11T17:23:54Z

internal/test/issues/issue-1039/issue_test.go

+				AliasName: nullable.Nullable[string]{
+					true: "foo-alias",
+				},


I'd prefer to avoid the implementation details leaking - please use NewNullableWithValue

jamietanna · 2024-01-11T17:24:40Z

internal/test/issues/issue-1039/issue_test.go

+func TestNullableTypesMarshal(t *testing.T) {
+	// include all fields in patch request
+	patchReq := PatchRequest{
+		ComplexOptionalNullable: nullable.Nullable[ComplexOptionalNullable]{


can the order of fields in this function mirror what's in the other file? Will make it easier to diff

As part of oapi-codegen#1039, we've created a new library `oapi-codegen/nullable`, which allows tracking whether: - a field is not sent - a field is sent with an explicit `null` - a field is sent with an explicit value This introduces an opt-in `output-options` flag, `nullable-type`, which can generate the `nullable.Nullable` types. This is opt-in, as existing code will break due to the signature change, as well as a behaviour change. Closes oapi-codegen#1039. Co-authored-by: Ashutosh Kumar <ashutosh.kumar@elastic.co>

jamietanna reviewed Dec 22, 2023

View reviewed changes

examples/go.mod Outdated Show resolved Hide resolved

jamietanna reviewed Dec 22, 2023

View reviewed changes

pkg/codegen/configuration.go Outdated Show resolved Hide resolved

jamietanna reviewed Dec 22, 2023

View reviewed changes

pkg/codegen/schema.go Outdated Show resolved Hide resolved

sonasingh46 force-pushed the implement_nullable branch 3 times, most recently from 79d1b49 to 2c4fc12 Compare December 22, 2023 15:24

sonasingh46 changed the title ~~(wip)add optional type code generation~~ add optional type code generation Jan 2, 2024

sonasingh46 changed the title ~~add optional type code generation~~ add nullable type code generation Jan 3, 2024

sonasingh46 marked this pull request as ready for review January 9, 2024 16:46

sonasingh46 force-pushed the implement_nullable branch 2 times, most recently from 952ab96 to 60ceb79 Compare January 9, 2024 17:53

jamietanna reviewed Jan 10, 2024

View reviewed changes

jamietanna mentioned this pull request Jan 10, 2024

Enable forcing non-pointer types using Nullable #823

Open

jamietanna reviewed Jan 10, 2024

View reviewed changes

pkg/codegen/schema_test.go Show resolved Hide resolved

jamietanna reviewed Jan 10, 2024

View reviewed changes

sonasingh46 force-pushed the implement_nullable branch from 750f993 to ec6ff58 Compare January 10, 2024 17:26

jamietanna reviewed Jan 11, 2024

View reviewed changes

internal/test/issues/issue-1039/defaultbehaviour/defaultbehaviour_test.go Show resolved Hide resolved

jamietanna reviewed Jan 11, 2024

View reviewed changes

jamietanna force-pushed the implement_nullable branch from 2f4e90b to 224d94f Compare January 11, 2024 21:10

jamietanna added the enhancement New feature or request label Jan 11, 2024

jamietanna force-pushed the implement_nullable branch from 224d94f to bc97778 Compare January 11, 2024 21:40

jamietanna changed the title ~~add nullable type code generation~~ Allow generating nullable.Nullable for nullable properties Jan 11, 2024

jamietanna merged commit 66f9bb8 into oapi-codegen:master Jan 11, 2024

jamietanna mentioned this pull request Jan 25, 2024

Add OperationID in client request header #1390

Open

alehed mentioned this pull request Jan 26, 2024

Unmarshal into a *map instead of a map #478

Closed

	github.com/oapi-codegen/nullable v1.0.0
	github.com/oapi-codegen/nullable v1.0.1

	func TestPropertyNullable_GoTypeDef(t *testing.T) {
	func TestProperty_GoTypeDef_nullable(t *testing.T) {

	assert.Truef(t, obj.HomeCountry == nil, "home country should be nil")
	assert.Nilf(t, obj.HomeCountry, "home country should be nil")

Uh oh!

Allow generating nullable.Nullable for nullable properties #1404

Allow generating nullable.Nullable for nullable properties #1404

Uh oh!

Conversation

sonasingh46 commented Dec 21, 2023 • edited by jamietanna Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jamietanna left a comment

Choose a reason for hiding this comment

Uh oh!

jamietanna commented Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notable features

Nullable types,

Uh oh!

jamietanna Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Allow generating `nullable.Nullable` for nullable properties #1404

Allow generating `nullable.Nullable` for nullable properties #1404

sonasingh46 commented Dec 21, 2023 •

edited by jamietanna

Loading

jamietanna commented Jan 11, 2024 •

edited

Loading

`Nullable` types,

jamietanna Jan 11, 2024 •

edited

Loading