Skip to content

Low compilation cache hit rate leads to long ci compilation times #17805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
forsaken628 opened this issue Apr 17, 2025 · 3 comments
Open

Low compilation cache hit rate leads to long ci compilation times #17805

forsaken628 opened this issue Apr 17, 2025 · 3 comments

Comments

@forsaken628
Copy link
Collaborator

forsaken628 commented Apr 17, 2025 •
edited
Loading

Summary

Build scripts can make a crate difficult to cache compilation artefacts, resulting in long ci compilation times.
So we need to rationalise crates so that more code is not affected by build scripts,
which #17801 proved to be effective, but still has huge potential for optimisation.
The two crates that have the most impact at the moment are databend_common_proto_conv and databend_common_config,
both of which are difficult to cache and are also widely imported.

See sccache.log for the compile cache detail.
@forsaken628
Copy link
Collaborator Author

forsaken628 commented Apr 17, 2025 •
edited
Loading

cache miss list

[openssl_sys]: Compiled in 0.631 s
[databend_common_version]: Compiled in 0.057 s
[databend_common_meta_raft_store]: Compiled in 3.559 s
[databend_common_expression]: Compiled in 24.535 s
[databend_common_pipeline_core]: Compiled in 0.860 s
[databend_common_native]: Compiled in 1.328 s
[databend_functions_scalar_decimal]: Compiled in 1.531 s
[databend_common_pipeline_sinks]: Compiled in 0.609 s
[databend_common_meta_app]: Compiled in 4.132 s
[databend_functions_scalar_geo]: Compiled in 4.599 s
[databend_enterprise_fail_safe]: Compiled in 0.286 s
[databend_common_proto_conv]: Compiled in 1.119 s
[databend_common_pipeline_transforms]: Compiled in 4.932 s
[databend_functions_scalar_math]: Compiled in 4.672 s
[databend_functions_scalar_datetime]: Compiled in 7.596 s
[databend_functions_scalar_numeric_basic_arithmetic]: Compiled in 10.246 s
[databend_common_meta_api]: Compiled in 6.065 s
[databend_common_meta_embedded]: Compiled in 0.088 s
[databend_common_meta_client]: Compiled in 3.531 s
[databend_common_meta_semaphore]: Compiled in 0.784 s
[databend_common_meta_store]: Compiled in 0.533 s
[databend_enterprise_data_mask_feature]: Compiled in 0.254 s
[databend_common_storage]: Compiled in 18.728 s
[databend_meta]: Compiled in 9.025 s
[databend_functions_scalar_arithmetic]: Compiled in 26.220 s
[databend_common_meta_control]: Compiled in 1.066 s
[databend_functions_scalar_integer_basic_arithmetic]: Compiled in 18.592 s
[databend_common_config]: Compiled in 6.412 s
[databend_common_functions]: Compiled in 72.212 s
[databend_common_management]: Compiled in 5.526 s
[databend_enterprise_resources_management]: Compiled in 0.555 s
[databend_storages_common_table_meta]: Compiled in 7.672 s
[databend_storages_common_blocks]: Compiled in 0.585 s
[databend_common_users]: Compiled in 2.505 s
[databend_storages_common_session]: Compiled in 1.118 s
[databend_storages_common_index]: Compiled in 2.366 s
[databend_common_settings]: Compiled in 1.506 s
[databend_storages_common_pruner]: Compiled in 1.161 s
[databend_common_formats]: Compiled in 2.310 s
[databend_common_catalog]: Compiled in 4.531 s
[databend_enterprise_ngram_index]: Compiled in 0.599 s
[databend_enterprise_vacuum_handler]: Compiled in 0.677 s
[databend_enterprise_aggregating_index]: Compiled in 0.637 s
[databend_enterprise_inverted_index]: Compiled in 0.659 s
[databend_enterprise_hilbert_clustering]: Compiled in 0.648 s
[databend_storages_common_stage]: Compiled in 0.921 s
[databend_common_storages_view]: Compiled in 0.945 s
[databend_common_pipeline_sources]: Compiled in 0.889 s
[databend_storages_common_io]: Compiled in 0.881 s
[databend_common_storages_information_schema]: Compiled in 0.568 s
[databend_storages_common_cache]: Compiled in 1.921 s
[databend_common_storages_null]: Compiled in 1.118 s
[databend_common_storages_random]: Compiled in 1.331 s
[databend_common_storages_memory]: Compiled in 1.501 s
[databend_common_storages_orc]: Compiled in 2.900 s
[databend_common_storages_parquet]: Compiled in 6.062 s
[databend_common_storages_result_cache]: Compiled in 2.277 s
[databend_common_storages_delta]: Compiled in 3.590 s
[databend_common_storages_stage]: Compiled in 5.481 s
[databend_common_storages_iceberg]: Compiled in 9.740 s
[databend_common_sql]: Compiled in 70.502 s
[databend_enterprise_stream_handler]: Compiled in 0.424 s
[databend_enterprise_attach_table]: Compiled in 0.481 s
[databend_common_storages_hive]: Compiled in 17.815 s
[databend_common_storages_fuse]: Compiled in 28.498 s
[databend_enterprise_virtual_column]: Compiled in 0.469 s
[databend_common_storages_stream]: Compiled in 2.163 s
[databend_common_storages_factory]: Compiled in 0.775 s
[databend_common_storages_system]: Compiled in 14.723 s
[databend_query]: Compiled in 306.614 s
[databend_enterprise_query]: Compiled in 10.340 s
[databend_bendsave]: Compiled in 1.242 s

@forsaken628 forsaken628 mentioned this issue Apr 18, 2025
Merged
11 tasks
@forsaken628
Copy link
Collaborator Author

forsaken628 commented Apr 18, 2025 •
edited
Loading

after #17806

[openssl_sys]: Compiled in 1.259 s
[databend_common_version]: Compiled in 0.115 s
[databend_common_meta_raft_store]: Compiled in 7.632 s
[databend_common_meta_embedded]: Compiled in 0.316 s
[databend_common_meta_client]: Compiled in 8.431 s
[databend_common_meta_semaphore]: Compiled in 2.379 s
[databend_common_meta_store]: Compiled in 1.552 s
[databend_enterprise_data_mask_feature]: Compiled in 0.731 s
[databend_common_config]: Compiled in 17.543 s
[databend_common_management]: Compiled in 8.769 s
[databend_enterprise_resources_management]: Compiled in 0.727 s
[databend_common_users]: Compiled in 3.217 s
[databend_common_settings]: Compiled in 1.734 s
[databend_meta]: Compiled in 20.090 s
[databend_common_formats]: Compiled in 2.751 s
[databend_common_meta_control]: Compiled in 1.725 s
[databend_common_catalog]: Compiled in 5.348 s
[databend_enterprise_hilbert_clustering]: Compiled in 0.766 s
[databend_enterprise_inverted_index]: Compiled in 0.768 s
[databend_enterprise_aggregating_index]: Compiled in 0.790 s
[databend_enterprise_vacuum_handler]: Compiled in 0.887 s
[databend_enterprise_ngram_index]: Compiled in 0.975 s
[databend_common_storages_view]: Compiled in 1.214 s
[databend_storages_common_io]: Compiled in 1.156 s
[databend_common_pipeline_sources]: Compiled in 1.190 s
[databend_storages_common_stage]: Compiled in 1.415 s
[databend_common_storages_information_schema]: Compiled in 0.826 s
[databend_storages_common_cache]: Compiled in 2.782 s
[databend_common_storages_null]: Compiled in 1.643 s
[databend_common_storages_random]: Compiled in 1.950 s
[databend_common_storages_memory]: Compiled in 2.233 s
[databend_common_storages_orc]: Compiled in 4.672 s
[databend_common_storages_parquet]: Compiled in 13.667 s
[databend_common_storages_result_cache]: Compiled in 6.119 s
[databend_common_storages_delta]: Compiled in 14.581 s
[databend_common_storages_stage]: Compiled in 19.516 s
[databend_common_storages_iceberg]: Compiled in 40.583 s
[databend_common_sql]: Compiled in 138.542 s
[databend_enterprise_attach_table]: Compiled in 0.723 s
[databend_enterprise_stream_handler]: Compiled in 0.740 s
[databend_common_storages_hive]: Compiled in 29.617 s
[databend_common_storages_fuse]: Compiled in 46.077 s
[databend_enterprise_virtual_column]: Compiled in 0.822 s
[databend_common_storages_stream]: Compiled in 3.280 s
[databend_common_storages_factory]: Compiled in 1.252 s
[databend_common_storages_system]: Compiled in 24.719 s
[databend_query]: Compiled in 535.622 s
[databend_enterprise_query]: Compiled in 17.850 s
[databend_bendsave]: Compiled in 2.113 s

@forsaken628
Copy link
Collaborator Author

local debug tutorial

cargo install sccache --locked
export SCCACHE_ERROR_LOG=sccache_log.txt
export RUSTC_WRAPPER=sccache
SCCACHE_LOG=debug sccache --start-server
cargo build --profile ci --bin databend-sqllogictests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@forsaken628
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy