Fix nondeterministic type checking caused by nonassociativity of joins #19147
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hauntsaninja
changed the title
This comment has been minimized.
This comment has been minimized.
|
Primer looks pretty good, seems like we will usually get better results by ensuring Union gets joined against, arguably more consistent too |
This was referenced May 26, 2025
|
Diff from mypy_primer, showing the effect of this PR on open source code: colour (https://github.com/colour-science/colour)
- colour/utilities/array.py:270: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/utilities/array.py:626: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | type[unsignedinteger[_16Bit]] | type[unsignedinteger[_32Bit]] | type[unsignedinteger[_64Bit]] | None") [assignment]
- colour/utilities/array.py:633: error: "None" not callable [misc]
- colour/utilities/array.py:680: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:696: error: "None" not callable [misc]
- colour/utilities/array.py:724: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | type[unsignedinteger[_16Bit]] | type[unsignedinteger[_32Bit]] | type[unsignedinteger[_64Bit]] | None") [assignment]
- colour/utilities/array.py:759: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1158: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1222: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1287: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1351: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1422: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1492: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1560: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1629: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1697: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:1772: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/utilities/array.py:2197: error: Incompatible types in assignment (expression has type "type[generic[float]]", variable has type "type[numpy.bool[builtins.bool]] | type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | <7 more items> | None") [assignment]
- colour/utilities/array.py:2259: error: Incompatible types in assignment (expression has type "type[generic[float]]", variable has type "type[numpy.bool[builtins.bool]] | type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | <7 more items> | None") [assignment]
- colour/utilities/array.py:2580: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/utilities/array.py:2618: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/utilities/array.py:2658: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/algebra/interpolation.py:414: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/algebra/interpolation.py:428: error: Incompatible types in assignment (expression has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items>") [assignment]
- colour/algebra/interpolation.py:428: note: Item in the first union not in the second: "None"
- colour/algebra/interpolation.py:820: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/algebra/interpolation.py:824: error: Incompatible types in assignment (expression has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items>") [assignment]
- colour/algebra/interpolation.py:824: note: Item in the first union not in the second: "None"
- colour/algebra/interpolation.py:1046: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/algebra/interpolation.py:1053: error: Incompatible types in assignment (expression has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items>") [assignment]
- colour/algebra/interpolation.py:1053: note: Item in the first union not in the second: "None"
- colour/algebra/interpolation.py:1387: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/algebra/interpolation.py:1394: error: Incompatible types in assignment (expression has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items>") [assignment]
- colour/algebra/interpolation.py:1394: note: Item in the first union not in the second: "None"
- colour/algebra/extrapolation.py:151: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None") [assignment]
- colour/algebra/extrapolation.py:164: error: Incompatible types in assignment (expression has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items>") [assignment]
- colour/algebra/extrapolation.py:164: note: Item in the first union not in the second: "None"
- colour/geometry/primitives.py:144: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/geometry/primitives.py:145: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | type[unsignedinteger[_16Bit]] | type[unsignedinteger[_32Bit]] | type[unsignedinteger[_64Bit]] | None") [assignment]
- colour/geometry/primitives.py:218: error: Argument 2 to "zeros" has incompatible type "list[tuple[str, type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None, int]]"; expected "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None" [arg-type]
+ colour/geometry/primitives.py:218: error: Argument 2 to "zeros" has incompatible type "list[tuple[str, type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64], int]]"; expected "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None" [arg-type]
- colour/geometry/primitives.py:369: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/geometry/primitives.py:370: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | type[unsignedinteger[_16Bit]] | type[unsignedinteger[_32Bit]] | type[unsignedinteger[_64Bit]] | None") [assignment]
- colour/geometry/primitives.py:418: error: Argument 2 to "zeros" has incompatible type "list[tuple[str, type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None, int]]"; expected "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None" [arg-type]
+ colour/geometry/primitives.py:418: error: Argument 2 to "zeros" has incompatible type "list[tuple[str, type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64], int]]"; expected "type[signedinteger[_8Bit]] | type[signedinteger[_16Bit]] | type[signedinteger[_32Bit]] | type[signedinteger[_64Bit]] | type[unsignedinteger[_8Bit]] | <6 more items> | None" [arg-type]
- colour/continuous/signal.py:1165: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/continuous/multi_signals.py:1429: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
+ colour/colorimetry/spectrum.py:540: error: Incompatible return value type (got "ndarray[tuple[int, ...], dtype[generic[Any]]]", expected "ndarray[tuple[int, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]") [return-value]
- colour/colorimetry/spectrum.py:513: error: Incompatible types in assignment (expression has type "type[object]", variable has type "type[floating[_16Bit]] | type[floating[_32Bit]] | type[float64] | None") [assignment]
- colour/colorimetry/spectrum.py:521: error: "None" not callable [misc]
- colour/colorimetry/spectrum.py:522: error: "None" not callable [misc]
- colour/colorimetry/spectrum.py:523: error: "None" not callable [misc]
- colour/models/rgb/rgb_colourspace.py:1050: error: Argument 1 to "xy_to_xyY" has incompatible type "object | Any"; expected "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]" [arg-type]
- colour/models/rgb/rgb_colourspace.py:1191: error: Argument 1 to "xy_to_xyY" has incompatible type "object | Any"; expected "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]" [arg-type]
ibis (https://github.com/ibis-project/ibis)
- ibis/backends/sql/compilers/risingwave.py:30: error: Incompatible types in assignment (expression has type "tuple[AnnotableMeta, ...]", base class "PostgresCompiler" defined the type as "tuple[type[RowID], type[TimeDelta], type[ArrayFlatten]]") [assignment]
+ ibis/backends/sql/compilers/risingwave.py:30: error: Incompatible types in assignment (expression has type "tuple[type[Sample] | type[ApproxMultiQuantile] | type[MultiQuantile] | type[RandomUUID] | type[RandomScalar] | type[Mode] | type[Arbitrary] | type[GeoSpatialUnOp] | type[GeoSpatialBinOp] | type[GeoUnaryUnion], ...]", base class "PostgresCompiler" defined the type as "tuple[type[RowID], type[TimeDelta], type[ArrayFlatten]]") [assignment]
jax (https://github.com/google/jax)
- jax/experimental/pallas/ops/gpu/attention_mgpu.py:589: error: Unused "type: ignore" comment [unused-ignore]
+ jax/experimental/pallas/ops/gpu/attention_mgpu.py:590: error: Unused "type: ignore" comment [unused-ignore]
- jax/experimental/pallas/ops/gpu/attention_mgpu.py:610: error: Unused "type: ignore" comment [unused-ignore]
+ jax/experimental/pallas/ops/gpu/attention_mgpu.py:611: error: Unused "type: ignore" comment [unused-ignore]
cwltool (https://github.com/common-workflow-language/cwltool)
+ cwltool/command_line_tool.py:1312:33: error: Argument 1 to "extend" of "list" has incompatible type "list[dict[str, str]]"; expected "Iterable[CWLOutputType]" [arg-type]
|
JukkaL
approved these changes
May 27, 2025
| # join(int, join(str, int | str)) == join(int, int | str) == int | str | ||
| # Note that joins in theory should be commutative, but in practice some bugs mean this is | ||
| # also a source of non-deterministic type checking results. | ||
| sorted_lowers = sorted(lowers, key=_join_sorted_key) |
There was a problem hiding this comment.
Though this is probably fine, right now sorted with a key= argument is a bit slow when compiled using mypyc, so using a for loop here instead that moves union types to the front would likely be faster. It's also fine to merge this as is and only replace this with a for loop if there appears to be a measurable performance regression.
|
Thanks for the review! I will merge as is and watch the benchmarks, since there is another case of nondeterminism from nonassociativity, so the sort logic will get a little more complex |
JukkaL
pushed a commit
that referenced
this pull request
May 28, 2025
…oins (#19158) Fixes #19121 (xarray case) See #19147 for context The ordering of the union is still nondeterministic. We could solve this by change the solver to use `dict[Type, None` instead of `set[Type]` since dicts are ordered. But doing so could paper over further bad solving from nonassociativity or noncommutativity
hauntsaninja
added a commit
that referenced
this pull request
Jun 1, 2025
…col and type promotion commute (#18402) Fixes #16979 (bzoracler case only, OP case fixed by #19147) See #16979 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I thought about doing this in
join_type_list, but most callers look like they do have some deterministic order.Fixes #19121 (torchvision case only, haven't looked at xarray)
Fixes #16979 (OP case only, bzoracler case fixed by #18402)