-
Notifications
You must be signed in to change notification settings - Fork 146
feat(fill): enable shared pre-allocation groups and add BlockchainEngineReorgFixture
#1706
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
danceratopz
wants to merge
17
commits into
main
Choose a base branch
from
feat/fill-shared-alloc
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add `SharedPreStateGroup` and `SharedPreState` models. - Support grouping tests by (fork, environment) hash. - Enable serialization for phase 1/2 coordination.
- Add `BlockchainEngineFixtureCommon` base class for shared functionality. - Add `BlockchainEngineReorgFixture` with `post_state_diff` field. - Remove unnecessary backwards compatibility validator from common class. - Update `test_collect_only` framework test for new fixture.
- Export `BlockchainEngineReorgFixture` and `BlockchainEngineFixtureCommon`. - Export `SharedPreState` and `SharedPreStateGroup` models. - Enable imports for shared pre-allocation functionality.
- Add `update_shared_pre_state()` and `compute_shared_pre_alloc_hash()`. - Add `get_genesis_environment()` for polymorphic environment access. - Support both `StateTest` and `BlockchainTest` formats.
- Add `pytest_sessionstart` and `pytest_sessionfinish` for phase coordination. - Add `calculate_post_state_diff()` for memory-efficient storage. - Add custom terminal summary with group statistics. - Load/save shared pre-allocation state.
… path - Add `shared_prealloc_path` property to `FixtureOutput`. - Extend global address iterators to all test formats. - Make the fixture scope of address iterators dynamic: session-scoped for shared pre-alloc generation, function-scoped for regular pre-alloc. - Update and improve `pre_alloc` tests.
- Support generating reorg fixtures from blockchain tests. - Enable shared pre-allocation integration with blockchain specs.
- Support pre-allocation hash tracking in index files. - Enable fixture consumption with shared pre-allocation metadata.
…uring two-phase execution The beacon root tests were failing when run with two-phase shared pre-allocation because they used `itertools.count()` objects directly in pytest parametrization. These iterator objects maintain internal state that persists across test phases when run in the same Python process. Changes: - Replace direct `count()` objects with factory functions that return fresh iterators. - Update `beacon_roots` fixture to create new iterator instances on each invocation. - Modify test parametrization to use `timestamps_factory` instead of `timestamps`. This bug only manifested in two-phase execution because: 1. In regular single-phase filling, each test gets its own fresh Python process. 2. In two-phase execution within one process, the same parametrized iterator. objects are reused, causing timestamps to continue from where phase 1 left off. 3. The second phase would see timestamps like 21000 instead of starting at 1000. The fix ensures each test invocation gets fresh iterators with reset state, preventing cross-phase contamination while maintaining the same test behavior.
…groups Add a new CLI tool that provides comprehensive analysis of shared pre-allocation groups generated by the test framework. The tool helps identify optimization opportunities for client teams by analyzing group distributions, test coverage, and identifying problematic test functions that create multiple size-1 groups. Key features: - Display overall statistics (groups, tests, accounts) - Show distribution by fork with average tests per group - Analyze group size frequency distribution - Calculate test coverage impact by group size - List test modules by execution complexity (group count) - Identify split test functions with Groups/Fork ratio analysis - Progressive disclosure with -v and -vv verbosity levels The Groups/Fork ratio metric distinguishes between necessary multi-fork coverage and problematic parameter-heavy tests, enabling targeted optimization strategies for CI workflows.
- CLI flags: `--generate-shared-alloc` → `--generate-shared-pre` - CLI flags: `--use-shared-alloc` → `--use-shared-pre` - Property: `shared_prealloc_path` → `shared_pre_alloc_path` - Pytest marker: `prealloc_group` → `pre_alloc_group` - Entry point: `show_pre_alloc_groups` → `groupstats`
05dd35c
to
969de94
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗒️ Description
This PR introduces:
hash(env, fork)
into a filepre_alloc.json
.BlockchainEngineReorgFixture
that omits thepre
field but instead uses the shared pre-allocs frompre_alloc.json
.The "engine reorg" fixtures will enable a new
consume engine
simulator variant (follow-up PR) that, instead of starting a new client for every test case, allows the re-use of a client instance across all tests within a pre-alloc group. As client start-up dominates test-run times, this will greatly decrease test execution time for EL client consensus testing.fill
's behavior for regular fixture formats remains unchanged. Unique EOA and contract addresses are generated by (dynamically) changing the scope of the address iterators in thepre_alloc
plugin fromfunction
tosession
if shared-pre mode is enabled on the command-line.Other Additions
filler
plugin CLI Options--generate-shared-pre
(to activate phase 1;pre_alloc.json
generation) and--use-shared-pre
(to activate phase 2; generateblockchain_test_engine_reorg
fixtures).fill
click
-CLI entrypoint. Phase 1 runs with-qq
:uv run fill --generate-shared-pre
will run both phases in two separate pytest sessions.pytest.mark.pre_alloc_group("separate|<group-id>", reason="This test deploys a system contract.")
: A new markpre_alloc_group
that allows test implementers to manually define pre-alloc groups to avoid address collisions/pre-state modifications that can't be detected viahash(env,fork)
alone.groupstats
: a new entry point script that analyzes apre_alloc.json
file and reports pre-alloc group information to gain understanding of the groups and optimize simulator execution by omitting groups that target less relevant EVM behavior, see output in "Pre-Alloc Group Analysis" below.docs/consuming_tests/blockchain_test_engine_reorg.md
Test Fixture Change
The test functions
tests/cancun/eip4788_beacon_root/test_beacon_root_contract.py::test_beacon_root_transition
andtests/cancun/eip4788_beacon_root/test_beacon_root_contract.py::test_multi_block_beacon_root_timestamp_calls
required a small change to ensure that the iterators they used for timestamp and beacon root generation were reset in between phases. This was required as the tests now get executed twice within a single Python process (due to the two-phase wrapping in thefill
click
CLI).All other fixture hashes maintain parity with the changes in this PR.
Future Work
This is an initial, fully-functional implementation, it is, however, limited to sequential test execution.
It currently takes approx 3 hours to fill only engine-reorg fixtures on our shared server using EELS with `-m "not zkevm and not slow".
Adding compatibility with
xdist
is necessary to align with other fixture formats and be usable in release generation. For more details and an implementation work, see:xdist
use with two-phase shared pre-alloc generation #1705Example Fixtures
Find an example tarball here: https://github.com/danceratopz/execution-spec-tests/releases/tag/v1.0.0b1
You can grab it with:
Which was generated with:
Pre-Alloc Group Analysis
These screenshots are from the output of the new
groupstats
entrypoint (using the pre_alloc.json for all tests--until Prague
from the tar ball above):🔗 Related Issues
The following issues were pre-requisite changes to allow shared pre-alloc group generation for all tests:
BLOBHASH
context tests to use thepre_alloc
plugin #1637pre
fixture inSELFDESTRUCT
collision tests #1643secret_key
fromtest_access_list
's transaction #1674This refactor was helpful to enable the wrapped two-phase execution in the
fill
click
CLI:click
interfaces for EEST pytest-based commands #1654✅ Checklist