Backport #109768 to 25.8: Mask session_token and Google ADC secrets in the explicit-url S3 form - #112333
Merged
Conversation
…n the explicit-url S3 form
Contributor
|
Workflow [BackportPR], commit [c11e745] |
QueryAnalyzer.cpp lost its transitive include of <boost/algorithm/string/predicate.hpp> when the include of the old, inline-bodied Parsers/FunctionSecretArgumentsFinder.h was swapped for the new declarations-only Analyzer/FunctionSecretArgumentsFinderTreeNode.h, breaking the pre-existing, unrelated boost::iequals ORDER BY ALL check. Added the include explicitly, matching master's QueryAnalyzer.cpp. The new 04510_s3_explicit_url_named_secret_mask test also exercised use_environment_credentials on the S3 database engine, which this branch's DatabaseS3 does not support; dropped that assertion. #112333
…tion
A duplicated key-value argument such as
s3('http://...', format = 'CSV', format = 'TSV') is user-controlled input,
but parseKeyValueArguments threw LOGICAL_ERROR on the duplicate-key emplace.
LOGICAL_ERROR aborts debug builds and is treated as a critical failure by
stress and fuzzer checks, so a user typo must not produce it. On this branch
the helper still lives in Storages/ObjectStorage/S3/Configuration.cpp
(pre-dates the shared ObjectStorage/Utils.cpp refactor).
Backport of 4939661 (already applied to the
26.4 private backport), needed here too since the new secret-masking tests
from #109768 exercise this exact duplicate-key path.
Closes: #104267
findS3FunctionSecretArguments only special-cased the named-collection
form for non-cluster functions (isNamedCollectionName(0)), so
s3Cluster('cluster', named_collection, ...) fell through to the
positional url/secret classification meant for the literal-url form,
leaving the named collection's secret_access_key (and session_token,
Google ADC secrets) unmasked. Ported master's findS3NamedCollectionSecretArguments
helper and its use in findS3FunctionSecretArguments, findS3TableEngineSecretArguments
and findS3DatabaseSecretArguments, matching the fully merged state of #109768.
This is what test_mask_sensitive_info/test.py::test_table_functions_object_storage_cluster
caught on this backport (amd_asan and amd_tsan integration runs).
#112333
BACKUP TABLE ... TO S3('https://user:pass@localhost:11111/x?...', 'ak', 'sak') relied on the one-character bucket name being rejected by S3 URI validation before any network access, matching master's comment for this case. On this branch BACKUP's S3 destination locator parses the url without going through S3::URI: the userinfo confuses its ad-hoc host:port parsing into a bogus endpoint ("user:0"), and the resulting request is misclassified as retryable, so BackupsWorker retries it (s3_retry_attempts = 1000) instead of failing fast. The client hung for the full 600s test timeout, and the background backup task then kept the server from shutting down cleanly. The masking behavior itself already works (the query is logged with the userinfo and signature correctly hidden before the hang); only the "fails before any network access" assumption doesn't hold here. #112333
findTableEngineSecretArguments's S3-family dispatch (S3, COSN, OSS, DeltaLake, Hudi, Iceberg, IcebergS3, S3Queue) was missing GCS and DeltaLakeS3, even though both exist as table engines on this branch (StorageObjectStorageDefinitions.h) and are included in master's merged state of #109768. Engines that fell through this dispatch got no masking at all: SHOW CREATE TABLE and the query log showed their secret_access_key in plaintext, as caught by 04510_s3_explicit_url_named_secret_mask's ENGINE = GCS(...) case. #112333
The `Backup` database engine (`ENGINE = Backup(database_name, S3(...))`) reconstructs its nested `S3(...)` destination for logging, since that destination isn't itself recognized as an S3 engine when the formatter recurses into an argument rather than `function`. This whole function, its dispatch entry, the `quote_replacement` field it needs on `Result` (to emit the reconstructed call as raw SQL text instead of a quoted string), and the matching ASTFunction.cpp formatting change all exist in #109768's own commits (42bda2e, ced7e82) and in master's merged state, but were dropped entirely in this cherry-pick: on this branch, `CREATE DATABASE ... ENGINE = Backup('db', S3('url','ak','sk'))` showed the secret access key in plaintext in SHOW CREATE and the query log, and nothing in the trimmed 25.8 test file caught it. Ported the function, wiring, and the corresponding SQL/reference test cases from 04510_s3_explicit_url_named_secret_mask (skipping the two cases with a userinfo-bearing localhost:11111 URL, which hit the same BACKUP-destination-parsing hang already worked around for the plain BACKUP TO S3 case), plus the integration test case in test_mask_sensitive_info::test_create_database. #112333
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original pull-request #109768
Cherry-pick pull-request #112289
This pull-request is a last step of an automated backporting.
Treat it as a standard pull-request: look at the checks and resolve conflicts.
Merge it only if you intend to backport changes to the target branch, otherwise just close it.
The PR source
The PR is created in the CI job
Version info
25.8.29.47