Skip to content

feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

wants to merge 1 commit into from

Conversation

chfast
Copy link
Member

@chfast chfast commented May 23, 2025 •
edited
Loading

🗒️ Description

Add a benchmark where we place a very long initcode in the memory and then invoke CREATE instructions with this initcode up to the block gas limit. The initcode itself have minimal execution time but forces the EVM to perform the full jumpdest analysis on the parametrized byte pattern. The initicode is slightly modified between CREATE invocations to prevent caching.

🔗 Related Issues

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
  • Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.

@chfast chfast force-pushed the bench/initcode_jumpdest_analysis branch 2 times, most recently from 6b017c7 to f9b75b4 Compare May 23, 2025 14:37
@chfast
Copy link
Member Author

chfast commented May 23, 2025 •
edited
Loading

Benchmarking EIP-7907: Meter Contract Code Size And Increase Limit

This test can be used to benchmark EIP-7907. The implementation must allow initcodes longer than the Prague limit.

Generated state test: worst_initcode_jumpdest_analysis.json

Results

evmone

It looks like increasing the limit causes the test to be executed ~6x longer, but this is still ~192Mgas/s in the worst case.
Main reason should be the change of the CREATE cost - jumpdest analysis cost ratio. For the current limit of 48k the initcode jumpdest analysis cost is only ~3k, for the 512k max size the cost is 33k.

Osaka/0/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-00-initcode_size_49152]/1: 35 ms
Osaka/1/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-00-initcode_size_524288]/1: 197 ms
Osaka/2/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-5b-initcode_size_49152]/1: 127 ms
Osaka/3/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-5b-initcode_size_524288]/1: 740 ms
Osaka/4/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b-initcode_size_49152]/1: 66 ms
Osaka/5/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b-initcode_size_524288]/1: 374 ms
Osaka/6/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b5b-initcode_size_49152]/1: 53 ms
Osaka/7/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b5b-initcode_size_524288]/1: 293 ms
Osaka/8/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b-initcode_size_49152]/1: 46 ms
Osaka/9/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b-initcode_size_524288]/1: 260 ms
Osaka/10/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b5b-initcode_size_49152]/1: 38 ms
Osaka/11/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b5b-initcode_size_524288]/1: 214 ms

Geth

[
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-605b5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 102958028,
      "allocs": 110530,
      "bytesAllocated": 123316072,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-615b5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 97807776,
      "allocs": 110570,
      "bytesAllocated": 123322833,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-615b5b5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 89937716,
      "allocs": 110541,
      "bytesAllocated": 123318193,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-00-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 71727318,
      "allocs": 110613,
      "bytesAllocated": 123327866,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 69766863,
      "allocs": 110608,
      "bytesAllocated": 123328256,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-605b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 102013648,
      "allocs": 110550,
      "bytesAllocated": 123313963,
      "gasUsed": 72000000
    }
  }
]

@jochem-brouwer jochem-brouwer mentioned this pull request May 25, 2025
Merged
9 tasks
@chfast chfast force-pushed the bench/initcode_jumpdest_analysis branch from f9b75b4 to 5f1d2b2 Compare May 25, 2025 19:45
@chfast chfast added scope:tests Scope: Changes EL client test cases in `./tests` type:feat type: Feature feature:zkevm labels May 25, 2025
marioevz
marioevz requested changes May 25, 2025
View reviewed changes
@jochem-brouwer
Copy link
Member

jochem-brouwer commented May 25, 2025 •
edited
Loading

@chfast if there is a 6x performance degradation does this mean the initcode pricing from 3860 is too cheap? One would assume this extra initcode analysis is paid for? (2 gas per 32 bytes is thus too cheap for initcode analysis?)

jsign
jsign reviewed May 25, 2025
View reviewed changes
Copy link
Collaborator

@jsign jsign left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the #1649 discussion, LGTM!

Cycles:

tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-615b5b-initcode_size_49152]-1        255986171
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-605b-initcode_size_49152]-1  330872313
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-615b5b5b-initcode_size_49152]-1      381150402
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-00-initcode_size_49152]-1    406269727
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-605b5b-initcode_size_49152]-1        473544495
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-5b-initcode_size_49152]-1    758025210

chfast
chfast commented May 26, 2025
View reviewed changes
jochem-brouwer
jochem-brouwer reviewed May 26, 2025
View reviewed changes
@chfast chfast force-pushed the bench/initcode_jumpdest_analysis branch 2 times, most recently from 2303aa9 to 4fc2d6b Compare June 4, 2025 11:05
jsign
jsign approved these changes Jun 4, 2025
View reviewed changes
Copy link
Collaborator

@jsign jsign left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double checked the cycles with the last time I reported and looks good.

@chfast chfast force-pushed the bench/initcode_jumpdest_analysis branch from 4fc2d6b to e4eba6b Compare June 4, 2025 15:50
@jochem-brouwer jochem-brouwer requested a review from marioevz June 6, 2025 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jochem-brouwer jochem-brouwer jochem-brouwer left review comments

@jsign jsign jsign approved these changes

@marioevz marioevz Awaiting requested review from marioevz

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
feature:zkevm scope:tests Scope: Changes EL client test cases in `./tests` type:feat type: Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@chfast @jochem-brouwer @jsign @marioevz
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy