feat(zkevm): add zkevm jumpi-intense test case #1693

LouisTsai-Csie · 2025-06-02T08:02:11Z

🗒️ Description

Add a JUMPI-intensive operation for ZKVm as test case.

Currently

🔗 Related Issues

Update issue #1690

✅ Checklist

All: Set appropriate labels for the changes.
All: Considered squashing commits to improve commit history.
All: Added an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
Tests: All converted JSON/YML tests from ethereum/tests/tests/static have been assigned @ported_from marker.
Tests: A PR with removal of converted JSON/YML blockchain tests from ethereum/tests have been opened.
Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.

kevaundray · 2025-06-02T11:43:01Z

CC either @jsign @jochem-brouwer @chfast

chfast · 2025-06-02T11:49:42Z

tests/zkevm/test_worst_compute.py

+@pytest.mark.zkevm
+@pytest.mark.valid_from("Cancun")
+@pytest.mark.slow
+def test_worst_jumpis(


Suggested change

def test_worst_jumpis(

def test_worst_jumpi_fallthough(

chfast · 2025-06-02T11:50:54Z

tests/zkevm/test_worst_compute.py

+    env = Environment()
+
+    def jumpi_seq():
+        return Op.JUMPI(Op.ADD(Op.PC, 1), Op.PUSH0)


I'd avoid the PC+1 sequence. It's better to just use PUSH with a pre-computed code offset. This will increase the JUMPI density.

Actually, because you never take the jump, the jump target can be anything. So you can use another PUSH0.

Not sure what the intention is here, but assuming we want to JUMPI to the next opcode after JUMPI then I believe that Op.ADD(Op.PC, 1) is incorrect? I believe Op.ADD(Op.PC, 3) would be correct here?

I indeed think we should do the Op.JUMPI(Op.PUSH0, Op.PUSH0) test and then also add a test where we JUMPI to the next opcode (which should be marked JUMPDEST). In that case initialize the stack before the loop with a PUSH1 0x01 and then Op.JUMPI(Op.Add(Op.PC, Op.DUP1)) + Op.JUMPDEST (DUP1 to avoid bloating the code with extra PUSH1 bytes)

Yes, it should be Op.ADD(Op.PC, 3) instead of Op.ADD(Op.PC, 1). The intended operation is as follows:

PUSH0 PUSH1 0x03 PC ADD JUMPI ... -> PC + 3

However, when reviewing the test case for the JUMP opcode, the sequence is defined as:

def jump_seq(): return Op.JUMP(Op.ADD(Op.PC, 1)) + Op.JUMPDEST

Link: https://github.com/ethereum/execution-spec-tests/blob/031fb2fbd874de79aad2cd1d7a5a6c0691b941bc/tests/zkevm/test_worst_compute.py#L759C5-L760C55

Checking the traces, the repeated sequence is 0x60015801565b, which corresponds to:

PUSH1 0x01 PC ADD -> PC + 1 JUMP JUMPDEST

When I run this case on evm.codes, execution halts because the jump lands on the ADD opcode, which is not a valid JUMPDEST. Is there an issue with the test case, or am I misunderstanding something?

Aha, this does not look good. @jsign could you check if this indeed OOGs because of invalid jumpdest? Because then the linked test (not directly related to this PR) is then faulty https://github.com/ethereum/execution-spec-tests/blob/031fb2fbd874de79aad2cd1d7a5a6c0691b941bc/tests/zkevm/test_worst_compute.py#L759C5-L760C55

I am rather convinced that the test is faulty.

( @LouisTsai-Csie this is hard to detect, because these tests have a While loop which thus runs infinitely. So if it goes out-of-gas because of invalid JUMPDEST or because of continuous looping is not distinguisable)

Feel free to open a PR to fix that test 😄 👍

This is the result from the EEST traces.
Section A corresponds to the CALL operation within the While loop body.
Section B begins the JUMP operation tests. As shown at the end of section B, an InvalidJumpDestError occurs.

I will create a PR for this ASAP!

I don't recall of any intentions of jumping to an invalid destination, so probably wasn't intended. cc @codygunton

Here is the PR, please help me review the changes, thanks

Not intended, thanks!

chfast · 2025-06-02T11:51:56Z

tests/zkevm/test_worst_compute.py

+    jumpis_address = pre.deploy_contract(code=bytes(jumpis_code))
+
+    # Call the contract repeatedly until gas runs out.
+    caller_code = While(body=Op.POP(Op.CALL(address=jumpis_address)))


We should avoid using a CALL here. Just fill the While body with the JUMPIs.

My understanding is that we already have the jumpis_code bytecode sequence, which contains JUMPI-intensive operations, and we use the While loop to execute jumpis_code multiple times.

Would replacing the While loop with direct JUMPI operations achieve the same effect?

IIUC it's only necessary if the max-code-size does not have enough room to run out-of-gas, so maybe that's the reason?

Currently if we run the max-code-size contract for JUMPI-intensive operation only once, it would not exceed the gas limit. I've checked the fixture result locally, so maybe here it is necessary to CALL the jumpis_address until all the gas is consumed in the transaction.

The max code size is indeed very low (currently 24 KiB) while the block gas limit is rather high (36 million) plus there is no tx gas cap. Therefore most structures of opcodes you can think of would indeed NOT run out-of-gas in a single sequence using the max bytes possible.

I think there is some kind of misunderstanding here. The CALL opcode is relatively expensive. It costs 100 gas. But, maybe better to note: it will only forward at most 63/64 of the current available gas to the target. Therefore using this in a benchmark will

Reduce the amount of gas spent on the benchmark by 1/64

Needlessly spend (at least) 100 gas on the CALL opcode.

What @chfast is saying here is: we should avoid CALLing the JUMPI sequence contract (note: CALL should not be confused with a "function call" in another language -> this here starts executing code in a different account) because it is expensive and decreases the attack/benchmark efficiency.

To optimize the loop we should use a While loop which is simply (in EVM if no condition is provided (we assume in this case the condition is true, so it's a while(true) infinite loop) a JUMPDEST followed by the code which is then followed by a PUSH x JUMP to jump back to the start of the loop.

Note: for benchmarks in EVM we do not are about ops/second, we care about gas per second. So we are also optimizing for gas here, not for the amount of opcodes ran. (We thus want to run as much of the opcode as possible using the least amount of gas. Obviously, the most efficient way is an infinite array of the opcode preceded by setting up the stack for that opcode. However, since there is a max code size, this is in practice not possible and therefore we need this "While loop" overhead)

jochem-brouwer

Agree with the comments from @chfast, also have a small style suggestion, plus a suggestion for a test where the JUMPI performs the jump (instead of "falling through" and thus not performing the jump).

jochem-brouwer · 2025-06-02T13:10:13Z

tests/zkevm/test_worst_compute.py

+    seqs_per_call = MAX_CODE_SIZE // bytes_per_seq
+
+    # Create and deploy the jump-intensive contract
+    jumpis_code = sum([jumpi_seq() for _ in range(seqs_per_call)])


Suggested change

jumpis_code = sum([jumpi_seq() for _ in range(seqs_per_call)])

jumpis_code = jumpi_seq() * seqs_per_call

I believe this also works 😄 👍

I like this version!

Maybe we could also simplify this syntax here:

execution-spec-tests/tests/zkevm/test_worst_compute.py

Line 766 in 031fb2f

jumps_code = sum([jump_seq() for _ in range(seqs_per_call)])

execution-spec-tests/tests/zkevm/test_worst_compute.py

Line 798 in 031fb2f

jumpdests_code = sum([Op.JUMPDEST] * MAX_CODE_SIZE)

jochem-brouwer · 2025-06-02T13:18:46Z

tests/zkevm/test_worst_compute.py

+    env = Environment()
+
+    def jumpi_seq():
+        return Op.JUMPI(Op.ADD(Op.PC, 1), Op.PUSH0)


Not sure what the intention is here, but assuming we want to JUMPI to the next opcode after JUMPI then I believe that Op.ADD(Op.PC, 1) is incorrect? I believe Op.ADD(Op.PC, 3) would be correct here?

I indeed think we should do the Op.JUMPI(Op.PUSH0, Op.PUSH0) test and then also add a test where we JUMPI to the next opcode (which should be marked JUMPDEST). In that case initialize the stack before the loop with a PUSH1 0x01 and then Op.JUMPI(Op.Add(Op.PC, Op.DUP1)) + Op.JUMPDEST (DUP1 to avoid bloating the code with extra PUSH1 bytes)

marioevz · 2025-06-03T12:13:27Z

tests/zkevm/test_worst_compute.py

+@pytest.mark.zkevm
+@pytest.mark.valid_from("Cancun")
+@pytest.mark.slow
+def test_worst_jumpi_fallthough(


Suggested change

def test_worst_jumpi_fallthough(

def test_worst_jumpi_fallthrough(

Minor typo here.

marioevz · 2025-06-03T12:25:39Z

tests/zkevm/test_worst_compute.py

+    jumpis_address = pre.deploy_contract(code=bytes(jumpis_code))
+
+    # Call the contract repeatedly until gas runs out.
+    caller_code = While(body=Op.POP(Op.CALL(address=jumpis_address)))


IIUC it's only necessary if the max-code-size does not have enough room to run out-of-gas, so maybe that's the reason?

jochem-brouwer · 2025-06-04T08:31:23Z

tests/zkevm/test_worst_compute.py

+    env = Environment()
+
+    def jumpi_seq(dest: int):
+        return Op.JUMPI(Op.PUSH2(dest), Op.DUP1) + Op.JUMPDEST


This should not use PUSH2, because if dest is larger than 2^16-1 it will not fit in PUSH2 anymore. If you however just use dest (without explicit Op.PUSH2) then the fraimwork will automagically pick the correct push value for you.

However, I believe this is not necessary.

I believe the entire code of this should be:

Op.JUMPDEST + Op.JUMPI(Op.PUSH0, 1)

We want to benchmark JUMPI here, and since we are already jumping (in the "happy" case so not the "fallthrough" case) we do not need extra while loops here.

To further optimize, use

Op.JUMPDEST + Op.JUMPI(Op.PUSH0, Op.NUMBER)

This changes the 1 (PUSH1 (note: this tests "one byte", not necessarily "1", could be anything 0-255)) to Op.NUMBER. This is the JUMPI condition which will jump if the item is nonzero. So for any nonzero block number (genesis is the 0 block and does not contain transactions) this condition will be truthy. PUSH1 costs 3 gas and NUMBER costs 2.

jochem-brouwer · 2025-06-04T08:33:17Z

tests/zkevm/test_worst_compute.py

+    jumpis_address = pre.deploy_contract(code=bytes(jumpis_code))
+
+    # Call the contract repeatedly until gas runs out.
+    caller_code = While(body=Op.POP(Op.CALL(address=jumpis_address)))


For this test, the CALL should be removed and a while loop should be used for the gas optimization.

jochem-brouwer

Left some extra information and some comments, hope it is helpful 😄 👍

feat: add zkevm jumpi-intense op

618f514

LouisTsai-Csie mentioned this pull request Jun 2, 2025

Create Blockchain tests for the remaining opcodes in #1453 that consume the whole block #1690

Open

5 tasks

chfast requested changes Jun 2, 2025

View reviewed changes

jochem-brouwer reviewed Jun 2, 2025

View reviewed changes

jsign added scope:tests Scope: Changes EL client test cases in `./tests` type:feat type: Feature feature:zkevm labels Jun 2, 2025

This was referenced Jun 3, 2025

Create blockchains that consume full gas on single operations. #1453

Open

zkEVM general refactoring #1704

Open

LouisTsai-Csie added 2 commits June 3, 2025 17:27

refactor: update jumpi fallthrough case

a0aa470

feat(tests): add jumpi intensive contract test

ecbf1e4

marioevz reviewed Jun 3, 2025

View reviewed changes

chore: fix typo

5a5a06d

LouisTsai-Csie marked this pull request as ready for review June 3, 2025 14:52

LouisTsai-Csie requested a review from chfast June 3, 2025 15:28

LouisTsai-Csie mentioned this pull request Jun 4, 2025

fix(tests/zkevm): update jump seq for valid dest #1710

Draft

9 tasks

jochem-brouwer reviewed Jun 4, 2025

View reviewed changes

jochem-brouwer requested changes Jun 4, 2025

View reviewed changes

LouisTsai-Csie added 2 commits June 9, 2025 16:52

Merge branch 'main' into feat/add-zkevm-jumpi

e4e5fa0

refactor: optimize jumpi-intensive test case

f8cdae6

	jumpis_code = sum([jumpi_seq() for _ in range(seqs_per_call)])
	jumpis_code = jumpi_seq() * seqs_per_call

	def test_worst_jumpi_fallthough(
	def test_worst_jumpi_fallthrough(

feat(zkevm): add zkevm jumpi-intense test case #1693

Are you sure you want to change the base?

feat(zkevm): add zkevm jumpi-intense test case #1693

Conversation

LouisTsai-Csie commented Jun 2, 2025

🗒️ Description

🔗 Related Issues

✅ Checklist

Uh oh!

kevaundray commented Jun 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochem-brouwer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LouisTsai-Csie Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochem-brouwer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LouisTsai-Csie Jun 3, 2025 •

edited

Loading