[Variant] Support appending complex variants in `VariantBuilder` #7914

friendlymatthew · 2025-07-11T22:50:41Z

Which issue does this PR close?

Fixes [Variant] Panic when appending nested objects to VariantBuilder #7907

Rationale for this change

When trying to append VariantObject or VariantLists directly on the VariantBuilder, it will panic.

Changes to the public API

VariantBuilder now has these additional methods:

append_object, will panic if shallow validation fails or the object has duplicate field names
try_append_object, will perform full validation on the object before appending
append_list, will panic if shallow validation fails
try_append_list, will perform full validation on the list before appending

parquet-variant/src/builder.rs

abacef · 2025-07-11T23:56:49Z

parquet-variant/src/builder.rs

+        let variant = value.into();
+
+        match variant {
+            Variant::Object(obj) => self.append_object(obj),


Should we create a clone of this append_value function using the try_append functions

Yes, I agree -- I thik it would make a lot of sense to have a try_append_value that returns an error (rather than panic'ing) if an invalid variant is passed in.
Then append_value would just call self try_append_value(v).unwrap() 🤔

alamb

Thank you @friendlymatthew and @abacef -- I think this is pretty close

I think @abacef 's suggestions are spot on and I left some similar ones

Let me know what you think

alamb · 2025-07-12T18:57:18Z

parquet-variant/src/builder.rs

-                    "Nested values are handled specially by ObjectBuilder and ListBuilder"
-                );
-            }
+            _ => unreachable!("Objects and lists must be appended using VariantBuilder::append_object and VariantBuilder::append_list"),


WHy not call self.append_variant_object and self.append_variant_list here (to avoid the panic?)

parquet-variant/src/builder.rs

alamb · 2025-07-12T19:02:53Z

parquet-variant/src/builder.rs

+        let variant = value.into();
+
+        match variant {
+            Variant::Object(obj) => self.append_object(obj),


Yes, I agree -- I thik it would make a lot of sense to have a try_append_value that returns an error (rather than panic'ing) if an invalid variant is passed in.
Then append_value would just call self try_append_value(v).unwrap() 🤔

parquet-variant/src/builder.rs

scovich

Interesting PR -- it certainly uncovered some questions in my mind!

Out of curiosity, why only support this for the top-level builder and not the list/object builders?

scovich · 2025-07-12T17:00:57Z

parquet-variant/src/builder.rs

-                    "Nested values are handled specially by ObjectBuilder and ListBuilder"
-                );
-            }
+            _ => unreachable!("Objects and lists must be appended using VariantBuilder::append_object and VariantBuilder::append_list"),


This is an awkward division of responsibilities. What if we got rid of ValueBuffer::append_variant entirely, and just had the builder invoke the primitive append_xxx methods directly?

I guess the method is currently important because all three builders need to call it. I wonder more and more whether there might be some way to harmonize the implementations more, leveraging a combination of ParentState and VariantBuilderExt?

scovich · 2025-07-12T22:23:01Z

parquet-variant/src/builder.rs

+    /// Appends a [`VariantObject`] to the builder with full validation during iteration.
+    ///
+    /// Recursively validates all nested variants in the object during iteration.
+    pub fn try_append_object<'m, 'v>(


I wish there were an efficient way to reuse code for fallible vs. infallible versions of this method...

In parquet-variant: - Add a new function `Variant::get_path`: this traverses the path to create a new Variant (does not cast any of it). - Add a new module `parquet_variant::path`: adds structs/enums to define a path to access a variant value deeply. In parquet-variant-compute: - Add a new compute kernel `variant_get`: does the path traversal over a `VariantArray`. In the future, this would also cast the values to a specified type. - Includes some basic unit tests. Not comprehensive. - Includes a simple micro-benchmark for reference. Current limitations: - It can only return another VariantArray. Casts are not implemented yet. - Only top-level object/list access is supported. It panics on finding a nested object/list. Needs apache#7914 to fix this. - Perf is a TODO.

alamb · 2025-07-14T16:35:36Z

I think this PR unlocks several other APIs so I hope to spend some time helping it along later today

friendlymatthew · 2025-07-14T16:38:57Z

I think this PR unlocks several other APIs so I hope to spend some time helping it along later today

Hi, I am traveling today so I've been a bit preoccupied. I'll try to get this over the line during my flight.

In parquet-variant: - Add a new function `Variant::get_path`: this traverses the path to create a new Variant (does not cast any of it). - Add a new module `parquet_variant::path`: adds structs/enums to define a path to access a variant value deeply. In parquet-variant-compute: - Add a new compute kernel `variant_get`: does the path traversal over a `VariantArray`. In the future, this would also cast the values to a specified type. - Includes some basic unit tests. - Includes a simple micro-benchmark for reference. Current limitations: - It can only return another VariantArray. Casts are not implemented yet. - Only top-level object/list access is supported. It panics on finding a nested object/list. Needs apache#7914 to fix this. - Perf is a TODO.

alamb · 2025-07-14T20:29:02Z

I am starting to hack in this a bit

alamb

I think we can signifcantly simplify this PR in this way

pydantic#5

It isn't quite ready to go -- I need to run now -- but I got enough of the sketch done to show how it would look like.

I'll mess around with it a bit more tomorrow too

Unify nested variant handling

alamb · 2025-07-15T18:18:53Z

I think this is the most important Variant PR at the moment from my perspective as we will likely need this for many of the other things we are discussing.

Thus I plan to help with this one first and then move on to more shredding related ones

@friendlymatthew please let me know if you get stuck or need some help. Otherwise I'll wait for a ping from you

friendlymatthew · 2025-07-16T11:55:11Z

Hi, round tripping variant objects would fail because we were writing out object field names by the sort order. This is fine for variants with a sorted dictionary, but everything else will produce inconsistent results

What we want instead is to write out the object fields by insertion order, we can do this by passing in the VariantMetadata ahead of time. This way we know the insertion order of field names

I'm sure we can improve the implementation, but the feature is working

cc @alamb

friendlymatthew · 2025-07-16T17:51:30Z

I think this is the most important Variant PR at the moment from my perspective as we will likely need this for many of the other things we are discussing.

Thus I plan to help with this one first and then move on to more shredding related ones

@friendlymatthew please let me know if you get stuck or need some help. Otherwise I'll wait for a ping from you

Hi, I'm curious to get your thoughts.

Sorry for the awolness, I am fighting jet lag/conferences

Cc @scovich

alamb · 2025-07-16T18:42:28Z

Sorry for the awolness, I am fighting jet lag/conferences

No worries at all -- I think we are all struggling to find the time while juggling many other things. It is totally fine -- and the nature of open source work

alamb

Thank you @friendlymatthew -- I think there are several things to improve in this PR (such as dbg! and the need to sort the fields for appending) -- however in order to keep things moving I will merge this one as is and then make a follow on PR or ticket to improve

alamb · 2025-07-16T18:46:58Z

parquet-variant/src/builder.rs

@@ -310,6 +376,8 @@ impl MetadataBuilder {
    fn upsert_field_name(&mut self, field_name: &str) -> u32 {
        let (id, new_entry) = self.field_names.insert_full(field_name.to_string());

+        dbg!(new_entry);


I think this is left over

alamb · 2025-07-16T18:52:05Z

parquet-variant/src/builder.rs

+
+                let mut object_builder = self.new_object(metadata_builder);
+
+                // first add all object fields that exist in metadata builder


I see the code here is basically trying to preserve the order so that the variants compare deeply equal

Rather than trying to preserve the byte-for-byte equality here, I think a better idea could be to implement a Variant::eq_value type method that compares the Variants for logical equality, rather than byte-for-byte equality.

I am thinking we can merge this PR and then I will make a follow on PR for that idea

+1 for the logical equality function on variants. This will be needed in tests where the result and expected variants come from different sources (and thus might have a mismatch in metadata or ordering).

alamb · 2025-07-16T18:53:26Z

Let's get this in and keep iterating

alamb · 2025-07-16T18:54:13Z

(BTW @friendlymatthew if you are willing, if you make PRs from your own personal fork, rather than the pydantic one, I can push commits directly to the PR, which might speed up development)

friendlymatthew · 2025-07-16T18:54:39Z

Thank you @friendlymatthew -- I think there are several things to improve in this PR (such as dbg! and the need to sort the fields for appending) -- however in order to keep things moving I will merge this one as is and then make a follow on PR or ticket to improve

I will have time for the next hour so I would like to clean up

alamb · 2025-07-16T18:55:39Z

I will have time for the next hour so I would like to clean up

Awesome -- maybe a follow on PR!

alamb · 2025-07-16T18:59:09Z

What I suggest doing is looking into changing the PartialEq implementaion for Variant to implement "logical: equality rather than physical equality

So rather than the #derived implementaiton of PartialEQ, which will check equality of underyling bytes, recursively check the variant and its children

I am not 100% sure what the semantics should be for field name order

alamb · 2025-07-16T19:00:28Z

I think accoding to my reading of https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#value-data-for-object-basic_type2, the field order can vary from objet to object so when comparing two variants for equality the field order should not be taken into account

alamb · 2025-07-16T19:00:31Z

Does that make sense?

friendlymatthew · 2025-07-16T19:08:19Z

(BTW @friendlymatthew if you are willing, if you make PRs from your own personal fork, rather than the pydantic one, I can push commits directly to the PR, which might speed up development)

Ah sounds good. I haven't checked recently, but I noticed pushing up PRs from my fork requires approval to run CI which was kind of a pain

friendlymatthew · 2025-07-16T19:19:18Z

the field order can vary from objet to object so when comparing two variants for equality the field order should not be taken into account

Hm, I assume you are talking about

The field ids and field offsets must be in lexicographical order of the corresponding field names in the metadata dictionary.
However, the actual value entries do not need to be in any particular order.
This implies that the field_offset values may not be monotonically increasing.
For example, for the following object:
{
  "c": 3,
  "b": 2,
  "a": 1
}
The field_id list must be [<id for key "a">, <id for key "b">, <id for key "c">], in lexicographical order.
The field_offset list must be [<offset for value 1>, <offset for value 2>, <offset for value 3>, <last offset>].
The value list can be in any order.

# Which issue does this PR close? - Closes #7893 # What changes are included in this PR? In parquet-variant: - Add a new function `Variant::get_path`: this traverses the path to create a new Variant (does not cast any of it). - Add a new module `parquet_variant::path`: adds structs/enums to define a path to access a variant value deeply. In parquet-variant-compute: - Add a new compute kernel `variant_get`: does the path traversal over a `VariantArray`. In the future, this would also cast the values to a specified type. - Includes some basic unit tests. Not comprehensive. - Includes a simple micro-benchmark for reference. Current limitations: - It can only return another VariantArray. Casts are not implemented yet. - Only top-level object/list access is supported. It panics on finding a nested object/list. Needs #7914 to fix this. - Perf is a TODO. # Are these changes tested? Some basic unit tests are added. # Are there any user-facing changes? Yes --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2025-07-16T19:54:45Z

Ah sounds good. I haven't checked recently, but I noticed pushing up PRs from my fork requires approval to run CI which was kind of a pain

This happens for first time contributors. Once we have merged a PR from your CI will run automatically

I think since we have now merged several your CI runs should go automatically

scovich · 2025-07-17T17:30:51Z

I think accoding to my reading of https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#value-data-for-object-basic_type2, the field order can vary from objet to object so when comparing two variants for equality the field order should not be taken into account

I thought the fields (as in, the field id array) did need to be in (lexical) order? The corresponding value bytes can be out of order but hopefully their physical placement doesn't change the logical equality test.

One observation tho -- an unordered metadata dictionary can have duplicate field ids. Two fields have the same name if their field ids are the same, but field ids differing does NOT prove that the fields have different names.

alamb · 2025-07-17T18:37:04Z

One observation tho -- an unordered metadata dictionary can have duplicate field ids. Two fields have the same name if their field ids are the same, but field ids differing does NOT prove that the fields have different names.

I think @friendlymatthew correctly implemented these semantics in #7943

friendlymatthew changed the title ~~Append complex variants~~ [Variant] Append complex variants Jul 11, 2025

friendlymatthew commented Jul 11, 2025

View reviewed changes

parquet-variant/src/builder.rs Outdated Show resolved Hide resolved

friendlymatthew force-pushed the friendlymatthew/append-complex branch from b054960 to 9ba2156 Compare July 11, 2025 22:53

github-actions bot added the parquet Changes to the parquet crate label Jul 11, 2025

friendlymatthew force-pushed the friendlymatthew/append-complex branch from 9ba2156 to 9bb416b Compare July 11, 2025 23:00

Append complex variants

4dc7f30

friendlymatthew force-pushed the friendlymatthew/append-complex branch from 9bb416b to 4dc7f30 Compare July 11, 2025 23:03

friendlymatthew mentioned this pull request Jul 11, 2025

[Variant] Add VariantBuilder::new_with_buffers to write to existing buffers #7912

Merged

abacef reviewed Jul 11, 2025

View reviewed changes

parquet-variant/src/builder.rs Outdated Show resolved Hide resolved

abacef reviewed Jul 11, 2025

View reviewed changes

alamb reviewed Jul 12, 2025

View reviewed changes

alamb mentioned this pull request Jul 12, 2025

[Variant] VariantBuilder with VariantMetadata instead of MetadataBuilder #7915

Draft

alamb changed the title ~~[Variant] Append complex variants~~ [Variant] Support appending complex variants in VariantBuilder Jul 12, 2025

friendlymatthew marked this pull request as draft July 12, 2025 19:58

alamb mentioned this pull request Jul 12, 2025

[Variant] Add variant_get compute kernel #7919

Merged

scovich reviewed Jul 12, 2025

View reviewed changes

Unify nested variant handling

78dc655

alamb mentioned this pull request Jul 14, 2025

Unify nested variant handling pydantic/arrow-rs#5

Merged

comments

28832d5

alamb reviewed Jul 14, 2025

View reviewed changes

Merge pull request #5 from alamb/alamb/suggestions_complex

db32eef

Unify nested variant handling

friendlymatthew force-pushed the friendlymatthew/append-complex branch from 014ffad to db32eef Compare July 14, 2025 22:06

Propogate error

a2d9683

friendlymatthew marked this pull request as ready for review July 14, 2025 22:13

friendlymatthew added 4 commits July 16, 2025 11:00

Fix bug

f79ea2a

More edge cases

336a4be

Merge branch 'main' into friendlymatthew/append-complex

935565a

Fmt

668449b

friendlymatthew force-pushed the friendlymatthew/append-complex branch 3 times, most recently from 5113739 to 850dd15 Compare July 16, 2025 10:23

friendlymatthew force-pushed the friendlymatthew/append-complex branch from 850dd15 to e77576c Compare July 16, 2025 14:54

Comment

1314f5c

friendlymatthew force-pushed the friendlymatthew/append-complex branch from e77576c to 1314f5c Compare July 16, 2025 14:54

alamb approved these changes Jul 16, 2025

View reviewed changes

alamb merged commit 7af62d5 into apache:main Jul 16, 2025
12 checks passed

alamb mentioned this pull request Jul 18, 2025

[Variant] Support variant_get kernel for shredded variants #7941

Open


		let mut object_builder = self.new_object(metadata_builder);

		// first add all object fields that exist in metadata builder

[Variant] Support appending complex variants in VariantBuilder #7914

[Variant] Support appending complex variants in VariantBuilder #7914

Uh oh!

Conversation

friendlymatthew commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

Changes to the public API

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 14, 2025

Uh oh!

friendlymatthew commented Jul 14, 2025

Uh oh!

alamb commented Jul 14, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 15, 2025

Uh oh!

friendlymatthew commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

friendlymatthew commented Jul 16, 2025

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

Uh oh!

friendlymatthew commented Jul 16, 2025

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

alamb commented Jul 16, 2025

Uh oh!

friendlymatthew commented Jul 16, 2025

Uh oh!

friendlymatthew commented Jul 16, 2025

Uh oh!

[Variant] Support appending complex variants in `VariantBuilder` #7914

[Variant] Support appending complex variants in `VariantBuilder` #7914

friendlymatthew commented Jul 11, 2025 •

edited

Loading

friendlymatthew commented Jul 16, 2025 •

edited

Loading