Skip to content

feat(fill): enable shared pre-allocation groups and add BlockchainEngineReorgFixture #1706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

danceratopz
Copy link
Member

🗒️ Description

This PR introduces:

  1. A two-phase filling method:
    • Phase 1: Group test pre-allocs that have the same hash(env, fork) into a file pre_alloc.json.
    • Phase 2: Fill test cases using these shared pre-alloc groups.
  2. A new fixture format BlockchainEngineReorgFixture that omits the pre field but instead uses the shared pre-allocs from pre_alloc.json.

The "engine reorg" fixtures will enable a new consume engine simulator variant (follow-up PR) that, instead of starting a new client for every test case, allows the re-use of a client instance across all tests within a pre-alloc group. As client start-up dominates test-run times, this will greatly decrease test execution time for EL client consensus testing.

fill's behavior for regular fixture formats remains unchanged. Unique EOA and contract addresses are generated by (dynamically) changing the scope of the address iterators in the pre_alloc plugin from function to session if shared-pre mode is enabled on the command-line.

Other Additions

  1. filler plugin CLI Options --generate-shared-pre (to activate phase 1; pre_alloc.json generation) and --use-shared-pre (to activate phase 2; generate blockchain_test_engine_reorg fixtures).
  2. The two-phase execution is wrapped into a single command for better UX via the fill click-CLI entrypoint. Phase 1 runs with -qq: uv run fill --generate-shared-pre will run both phases in two separate pytest sessions.
  3. pytest.mark.pre_alloc_group("separate|<group-id>", reason="This test deploys a system contract."): A new mark pre_alloc_group that allows test implementers to manually define pre-alloc groups to avoid address collisions/pre-state modifications that can't be detected via hash(env,fork) alone.
  4. groupstats: a new entry point script that analyzes a pre_alloc.json file and reports pre-alloc group information to gain understanding of the groups and optimize simulator execution by omitting groups that target less relevant EVM behavior, see output in "Pre-Alloc Group Analysis" below.
  5. Docs for the new fixture format at docs/consuming_tests/blockchain_test_engine_reorg.md

Test Fixture Change

The test functions tests/cancun/eip4788_beacon_root/test_beacon_root_contract.py::test_beacon_root_transition and tests/cancun/eip4788_beacon_root/test_beacon_root_contract.py::test_multi_block_beacon_root_timestamp_calls required a small change to ensure that the iterators they used for timestamp and beacon root generation were reset in between phases. This was required as the tests now get executed twice within a single Python process (due to the two-phase wrapping in the fill click CLI).

All other fixture hashes maintain parity with the changes in this PR.

Future Work

This is an initial, fully-functional implementation, it is, however, limited to sequential test execution.

It currently takes approx 3 hours to fill only engine-reorg fixtures on our shared server using EELS with `-m "not zkevm and not slow".

Adding compatibility with xdist is necessary to align with other fixture formats and be usable in release generation. For more details and an implementation work, see:

Example Fixtures

Find an example tarball here: https://github.com/danceratopz/execution-spec-tests/releases/tag/v1.0.0b1

You can grab it with:

consume cache --input=https://github.com/danceratopz/execution-spec-tests/releases/download/v1.0.0b1/fixtures-reorg.tar.gz

Which was generated with:

uv run fill --generate-shared-pre --output=fixtures-reorg -m "not slow and not zkevm" --until Prague --clean -v -k "not test_delegation_replacement_call_previous_contract"

Pre-Alloc Group Analysis

These screenshots are from the output of the new groupstats entrypoint (using the pre_alloc.json for all tests --until Prague from the tar ball above):

uv run groupstats fixtures-reorg/blockchain_tests_engine_reorg/pre_alloc.json -v

image
image

🔗 Related Issues

The following issues were pre-requisite changes to allow shared pre-alloc group generation for all tests:

This refactor was helpful to enable the wrapped two-phase execution in the fill click CLI:

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.

@danceratopz danceratopz requested a review from marioevz June 3, 2025 09:54
@danceratopz danceratopz added type:feat type: Feature scope:fill Scope: fill command labels Jun 3, 2025
@danceratopz danceratopz marked this pull request as ready for review June 3, 2025 09:54
@danceratopz danceratopz requested a review from spencer-tb June 3, 2025 09:55
@danceratopz danceratopz marked this pull request as draft June 4, 2025 18:57
- Add `SharedPreStateGroup` and `SharedPreState` models.
- Support grouping tests by (fork, environment) hash.
- Enable serialization for phase 1/2 coordination.
- Add `BlockchainEngineFixtureCommon` base class for shared functionality.
- Add `BlockchainEngineReorgFixture` with `post_state_diff` field.
- Remove unnecessary backwards compatibility validator from common class.
- Update `test_collect_only` framework test for new fixture.
- Export `BlockchainEngineReorgFixture` and `BlockchainEngineFixtureCommon`.
- Export `SharedPreState` and `SharedPreStateGroup` models.
- Enable imports for shared pre-allocation functionality.
- Add `update_shared_pre_state()` and `compute_shared_pre_alloc_hash()`.
- Add `get_genesis_environment()` for polymorphic environment access.
- Support both `StateTest` and `BlockchainTest` formats.
- Add `pytest_sessionstart` and `pytest_sessionfinish` for phase coordination.
- Add `calculate_post_state_diff()` for memory-efficient storage.
- Add custom terminal summary with group statistics.
- Load/save shared pre-allocation state.
… path

- Add `shared_prealloc_path` property to `FixtureOutput`.
- Extend global address iterators to all test formats.
- Make the fixture scope of address iterators dynamic: session-scoped for shared pre-alloc generation, function-scoped for regular pre-alloc.
- Update and improve `pre_alloc` tests.
- Support generating reorg fixtures from blockchain tests.
- Enable shared pre-allocation integration with blockchain specs.
- Support pre-allocation hash tracking in index files.
- Enable fixture consumption with shared pre-allocation metadata.
…uring two-phase execution

The beacon root tests were failing when run with two-phase shared pre-allocation because they used `itertools.count()` objects directly in pytest parametrization. These iterator objects maintain internal state that persists across test phases when run in the same Python process.

Changes:
- Replace direct `count()` objects with factory functions that return fresh iterators.
- Update `beacon_roots` fixture to create new iterator instances on each invocation.
- Modify test parametrization to use `timestamps_factory` instead of `timestamps`.

This bug only manifested in two-phase execution because:
1. In regular single-phase filling, each test gets its own fresh Python process.
2. In two-phase execution within one process, the same parametrized iterator.
   objects are reused, causing timestamps to continue from where phase 1 left off.
3. The second phase would see timestamps like 21000 instead of starting at 1000.

The fix ensures each test invocation gets fresh iterators with reset state, preventing cross-phase contamination while maintaining the same test behavior.
…groups

Add a new CLI tool that provides comprehensive analysis of shared pre-allocation groups generated by the test framework. The tool helps identify optimization opportunities for client teams by analyzing group distributions, test coverage, and identifying problematic test functions that create multiple size-1 groups.

Key features:
- Display overall statistics (groups, tests, accounts)
- Show distribution by fork with average tests per group
- Analyze group size frequency distribution
- Calculate test coverage impact by group size
- List test modules by execution complexity (group count)
- Identify split test functions with Groups/Fork ratio analysis
- Progressive disclosure with -v and -vv verbosity levels

The Groups/Fork ratio metric distinguishes between necessary multi-fork coverage and problematic parameter-heavy tests, enabling targeted optimization strategies for CI workflows.
- CLI flags: `--generate-shared-alloc` → `--generate-shared-pre`
- CLI flags: `--use-shared-alloc` → `--use-shared-pre`
- Property: `shared_prealloc_path` → `shared_pre_alloc_path`
- Pytest marker: `prealloc_group` → `pre_alloc_group`
- Entry point: `show_pre_alloc_groups` → `groupstats`
@danceratopz danceratopz force-pushed the feat/fill-shared-alloc branch from 05dd35c to 969de94 Compare June 4, 2025 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:fill Scope: fill command type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy