GH-46905: [C++][Parquet] Expose Statistics.is_{min/max}_value_exact and default set to true if min/max are set #46992
+189
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
|
|
|
@wgtmac this is the PR I am working on that pairs with the new Parquet file on parquet-testing, will update parquet-testing submodule once the parquet file is merged to have tests passing. I would appreciate if you could give early feedback in case I am missing something. |
wgtmac
reviewed
Jul 8, 2025
…o std::optional<bool>
…ue) instead of (max, min, is_min_exact_value, is_max_exact_value)
|
General LGTM, also, currently C++ code doesn't generate "truncated" min-max, I wonder if parquet-cpp file can regard as "not-truncated"? |
Yes, currently C++ doesn't generate truncated min-max statistics, that's why I am defaulting
I am not sure I entirely understand what you mean on this part, do you mean to update |
pitrou
reviewed
Jul 8, 2025
cpp/src/parquet/reader_test.cc
Outdated
| @@ -1023,7 +1023,7 @@ Column 0 | |||
| Uncompressed Size: 103, Compressed Size: 104 | |||
| Column 1 | |||
| Values: 3, Null Values: 0, Distinct Values: 0 | |||
| Max: 1, Min: 1 | |||
| Max: 1, Min: 1, Is Max Value Exact: N/A, Is Min Value Exact: N/A | |||
There was a problem hiding this comment.
Perhaps we want to make the display terser? For example (but feel free to do better :)):
Suggested change
| Max: 1, Min: 1, Is Max Value Exact: N/A, Is Min Value Exact: N/A | |
| Max (exact: unknown): 1, Min (exact: unknown): 1 |
There was a problem hiding this comment.
I've tested the parquet-reader tool with the new file:
--- Rows: 12 ---
Column 0
Values: 12, Null Values: 0, Distinct Values: 0
Max (exact: false): Kf, Min (exact: false): Al
Compression: UNCOMPRESSED, Encodings: PLAIN RLE
Uncompressed Size: 250, Compressed Size: 250
Column 1
Values: 12, Null Values: 0, Distinct Values: 0
Max (exact: false): Kf, Min (exact: false): Al
Compression: UNCOMPRESSED, Encodings: PLAIN RLE
Uncompressed Size: 250, Compressed Size: 250
Column 2
Values: 12, Null Values: 0, Distinct Values: 0
Max (exact: true): 🚀Kevin Bacon, Min (exact: false): Al
Compression: UNCOMPRESSED, Encodings: PLAIN RLE
Uncompressed Size: 258, Compressed Size: 258
Column 3
Values: 12, Null Values: 0, Distinct Values: 0
Max (exact: true): ��, Min (exact: false): Al
Compression: UNCOMPRESSED, Encodings: PLAIN RLE
Uncompressed Size: 236, Compressed Size: 236
Column 4
Values: 12, Null Values: 0, Distinct Values: 0
Max (exact: true): Ke, Min (exact: true): Al
Compression: UNCOMPRESSED, Encodings: PLAIN RLE
Uncompressed Size: 210, Compressed Size: 210
Column 5
Values: 12, Null Values: 0, Distinct Values: 0
Max (exact: true): Ke, Min (exact: true): Al
Compression: UNCOMPRESSED, Encodings: PLAIN RLE
Uncompressed Size: 210, Compressed Size: 210
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
pitrou
reviewed
Jul 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The
is_{min/max}_value_exactfields exist on the thrift definition and some implementations are already using them and truncating min and max values. This PR aims to expose those values and to default to true when writing files on C++ as no truncation is happening at the moment. If min/max statistics are generated we can setis_{min/max}_value_exactto true.Truncation for string and binary min/max is out of scope for this PR, we can do this on a following one.
What changes are included in this PR?
ParquetFilePrinterAre these changes tested?
Yes on CI.
Are there any user-facing changes?
Yes, the new fields will be available for the users on the API when reading Parquet files.