Skip to content

feat: collect instrumentation metadata and node:test coverage data #6652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 58 commits into from
May 22, 2025
Merged

feat: collect instrumentation metadata and node:test coverage data #6652

merged 58 commits into from
May 22, 2025
+1,730 −106

Conversation

galargh
Copy link
Member

@galargh galargh commented Apr 30, 2025 •
edited
Loading

  • Because this PR includes a bug fix, relevant tests have been included.
  • Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
  • I didn't do anything of this.

This PR implements https://www.notion.so/nomicfoundation/Instrumentation-metadata-and-node-test-coverage-data-collection-1df578cdeaf580eb8847dd730a496df5

Example pnpm hardhat test node --coverage output as executed in example-project:

Compiling your Solidity contracts
2025-05-08T14:11:43.021Z hardhat:core:coverage:coverage-manager Added metadata []
2025-05-08T14:11:43.021Z hardhat:core:coverage:coverage-manager Added metadata []
2025-05-08T14:11:43.021Z hardhat:core:coverage:coverage-manager Added metadata []
2025-05-08T14:11:43.022Z hardhat:core:coverage:coverage-manager Added metadata [
  {
    "tag": "04c0713a6595166ac1183e644f25df4b9876781375b4e5924a37f8ce0302d6f6",
    "kind": "statement",
    "startUtf16": 120,
    "endUtf16": 129,
    "sourceName": "contracts/Counter.sol"
  }
]
2025-05-08T14:11:43.022Z hardhat:core:coverage:coverage-manager Added metadata []
2025-05-08T14:11:43.022Z hardhat:core:coverage:coverage-manager Added metadata [
  {
    "tag": "b1131cf32b2f152299c5e6aa9f8f3ef147f576fc44ac09a1c13ef2859d81280a",
    "kind": "statement",
    "startUtf16": 277,
    "endUtf16": 288,
    "sourceName": "contracts/Revert.sol"
  },
  {
    "tag": "31d20cf5600804b3368ab79267019eb33b587defac416789bcc23c5447283125",
    "kind": "statement",
    "startUtf16": 326,
    "endUtf16": 346,
    "sourceName": "contracts/Revert.sol"
  }
]
2025-05-08T14:11:43.022Z hardhat:core:coverage:coverage-manager Added metadata [
  {
    "tag": "dfea0d24aa2bea223a0fe6b8b3c100b64f85c314af2f4453c788a7c0865a406f",
    "kind": "statement",
    "startUtf16": 166,
    "endUtf16": 184,
    "sourceName": "contracts/Rocket.sol"
  },
  {
    "tag": "26b38a9ff93a7c0530e6b62fcecb64bfe0e428ae54bac43dc522f9b136f454e2",
    "kind": "statement",
    "startUtf16": 184,
    "endUtf16": 209,
    "sourceName": "contracts/Rocket.sol"
  },
  {
    "tag": "b1458c7409894c1aeead5f507e8be8599180130e3ff8c86191a1ce437d44139e",
    "kind": "statement",
    "startUtf16": 243,
    "endUtf16": 268,
    "sourceName": "contracts/Rocket.sol"
  }
]
2025-05-08T14:11:43.022Z hardhat:core:coverage:coverage-manager Added metadata []
2025-05-08T14:11:43.023Z hardhat:core:coverage:coverage-manager Added metadata []
Compiled 1 Solidity file with solc 0.7.1 (evm target: istanbul)
Compiled 10 Solidity files with solc 0.8.22 (evm target: shanghai)

2025-05-08T14:11:43.980Z hardhat:core:coverage:coverage-manager Cleared data
Running node:test tests

2025-05-08T14:11:44.798Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.799Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.799Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.799Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.799Z hardhat:core:coverage:coverage-manager Added data []
  Example EDR based test
    ✔ should work get the block number from the EDR Network (633ms)
2025-05-08T14:11:44.898Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.898Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.900Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.901Z hardhat:core:coverage:coverage-manager Added data []
2025-05-08T14:11:44.904Z hardhat:core:coverage:coverage-manager Added data [
  "b1131cf32b2f152299c5e6aa9f8f3ef147f576fc44ac09a1c13ef2859d81280a",
  "31d20cf5600804b3368ab79267019eb33b587defac416789bcc23c5447283125"
]
    ✔ should show stack traces when a transaction reverts (109ms)

2025-05-08T14:11:44.911Z hardhat:core:coverage:coverage-manager Saved data to ./cache/coverage/data/6b71fb4e-69b6-4b57-ba6f-2fe658b6d738.json [
  "b1131cf32b2f152299c5e6aa9f8f3ef147f576fc44ac09a1c13ef2859d81280a",
  "31d20cf5600804b3368ab79267019eb33b587defac416789bcc23c5447283125"
]
  Other example test
    ✔ should have the example task

2025-05-08T14:11:45.118Z hardhat:core:coverage:coverage-manager Saved data to ./cache/coverage/data/6924f8d3-e262-458a-bec0-37e5cfdc5530.json []

3 passing (1144ms)

2025-05-08T14:11:45.127Z hardhat:core:coverage:coverage-manager Loaded data from ./cache/coverage/data/6924f8d3-e262-458a-bec0-37e5cfdc5530.json []
2025-05-08T14:11:45.127Z hardhat:core:coverage:coverage-manager Loaded data from ./cache/coverage/data/6b71fb4e-69b6-4b57-ba6f-2fe658b6d738.json [
  "b1131cf32b2f152299c5e6aa9f8f3ef147f576fc44ac09a1c13ef2859d81280a",
  "31d20cf5600804b3368ab79267019eb33b587defac416789bcc23c5447283125"
]
2025-05-08T14:11:45.127Z hardhat:core:coverage:coverage-manager Loaded data [
  "b1131cf32b2f152299c5e6aa9f8f3ef147f576fc44ac09a1c13ef2859d81280a",
  "31d20cf5600804b3368ab79267019eb33b587defac416789bcc23c5447283125"
]

galargh added 20 commits April 30, 2025 10:47
@galargh
feat: add preprocessProjectFileBeforeBuilding hook
651eda4
@galargh
feat: add onCoverageData hook
22ef7bc
@galargh
feat: implement solidity hook handlers in the coverage plugin
69bf940
@galargh
feat: implement hre hook handlers in the coverage plugin
5c994ff
@galargh
feat: implement hasHandler on the hook manager
ec6e87f
@galargh
feat: enable code coverage when onCoverageData hook handlers exist
bc5fd48
@galargh
feat: run preprocessProjectFileBeforeBuilding hook handler chain from…
0fcb515 
… compilation job implementation
@galargh
feat: remove getContentHash from ResolvedFile interfaces
556094e
@galargh
feat: implement coverage manager
2f58ef3
@galargh
feat: add coverage manager to hre internally
f231673
@galargh
feat: implement and export coverage internal api
2efbd52
@galargh
chore: lint
0d7fb44
@galargh
feat: clear coverage data before node tests start and load it afterwards
279555c
@galargh
feat: save coverage after node test worker is done
ca41e9e
@galargh
chore: lint
1f28624
@galargh
chore: upgrade @ignored/edr-optimism to 0.13.0-alpha.1
42f0b5c
@galargh
feat: hook up to the edr coverage implementation
a35686d
@galargh
chore: add logging to coverage manager implementation
55d63a6
@galargh
fix: the source instrumentation for coverage
744e00e
@galargh
docs: changeset
6b84f1a
@changeset-bot changeset-bot
Copy link

changeset-bot bot commented Apr 30, 2025 •
edited
Loading

🦋 Changeset detected

Latest commit: 2bfc953

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 19 packages
Name Type
@nomicfoundation/hardhat-node-test-runner Patch
@nomicfoundation/hardhat-errors Patch
hardhat Patch
@nomicfoundation/hardhat-mocha Patch
@nomicfoundation/hardhat-ethers-chai-matchers Patch
@nomicfoundation/hardhat-ignition Patch
@nomicfoundation/ignition-core Patch
@nomicfoundation/hardhat-ignition-ethers Patch
@nomicfoundation/ignition-ui Patch
@nomicfoundation/hardhat-ignition-viem Patch
@nomicfoundation/hardhat-keystore Patch
@nomicfoundation/hardhat-network-helpers Patch
@nomicfoundation/hardhat-node-test-reporter Patch
@nomicfoundation/hardhat-test-utils Patch
@nomicfoundation/hardhat-typechain Patch
@nomicfoundation/hardhat-utils Patch
@nomicfoundation/hardhat-toolbox-mocha-ethers Patch
@nomicfoundation/hardhat-viem Patch
@nomicfoundation/hardhat-zod-utils Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions bot added the status:ready This issue is ready to be worked on label Apr 30, 2025
@github-project-automation github-project-automation bot moved this to Backlog in Hardhat Apr 30, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 30, 2025 •
edited
Loading

hardhat

Total size of the bundle: 222M
Total number of dependencies (including transitive): 111

List of dependencies (sorted by size) 
216M	total
24M	@ignored/edr-optimism-linux-x64-musl
24M	@ignored/edr-optimism-linux-x64-gnu
22M	@ignored/edr-optimism-win32-x64-msvc
21M	@ignored/edr-optimism-linux-arm64-musl
21M	@ignored/edr-optimism-linux-arm64-gnu
20M	esbuild
18M	@ignored/edr-optimism-darwin-x64
17M	@ignored/edr-optimism-darwin-arm64
7.7M	@opentelemetry/semantic-conventions
6.2M	@sentry/core
4.2M	@sentry/node
3.1M	@opentelemetry/api
2.5M	micro-eth-signer
2.3M	@opentelemetry/core
1.9M	@opentelemetry/resources
1.9M	@noble/curves
1.7M	undici
1.6M	@opentelemetry/sdk-trace-base
1.2M	@noble/hashes
956K	@opentelemetry/instrumentation
864K	@streamparser/json
864K	@sentry/opentelemetry
860K	@nomicfoundation/hardhat-utils
856K	zod
760K	@opentelemetry/api-logs
676K	import-in-the-middle
624K	resolve
624K	micro-packed
584K	tsx
572K	acorn
544K	fast-equals
492K	@scure/bip39
468K	@nomicfoundation/hardhat-errors
408K	json-stream-stringify
368K	ethereum-cryptography
352K	@ignored/edr
332K	@streamparser/json-node
320K	pg-protocol
320K	enquirer
300K	@opentelemetry/instrumentation-http
300K	@ignored/edr-optimism
288K	semver
224K	@opentelemetry/instrumentation-pg
212K	@opentelemetry/instrumentation-graphql
200K	undici-types
196K	ws
188K	@opentelemetry/instrumentation-mongodb
180K	@opentelemetry/instrumentation-undici
176K	@opentelemetry/instrumentation-express
172K	@opentelemetry/instrumentation-amqplib
168K	@scure/base
164K	cjs-module-lexer
156K	@opentelemetry/instrumentation-hapi
148K	@opentelemetry/instrumentation-redis-4
148K	@opentelemetry/instrumentation-mysql
144K	@opentelemetry/instrumentation-fs
136K	get-tsconfig
136K	adm-zip
136K	@opentelemetry/instrumentation-koa
132K	@opentelemetry/instrumentation-knex
124K	@opentelemetry/instrumentation-mongoose
124K	@opentelemetry/instrumentation-ioredis
120K	@opentelemetry/instrumentation-mysql2
120K	@opentelemetry/instrumentation-connect
120K	@fastify/otel
116K	@opentelemetry/instrumentation-kafkajs
112K	@opentelemetry/instrumentation-tedious
112K	@opentelemetry/context-async-hooks
96K	@scure/bip32
96K	@opentelemetry/instrumentation-dataloader
92K	chalk
80K	@opentelemetry/instrumentation-generic-pool
76K	pg-types
76K	@prisma/instrumentation
72K	@nomicfoundation/solidity-analyzer
68K	debug
68K	@opentelemetry/sql-common
68K	@opentelemetry/instrumentation-lru-memoizer
64K	@nomicfoundation/hardhat-zod-utils
60K	acorn-import-attributes
60K	@opentelemetry/redis-common
56K	rfdc
56K	is-core-module
56K	function-bind
52K	@types/tedious
48K	shimmer
48K	ansi-colors
44K	resolve.exports
40K	resolve-pkg-maps
40K	require-in-the-middle
40K	forwarded-parse
40K	@types/mysql
36K	supports-preserve-symlinks-flag
36K	p-map
36K	@types/pg
32K	hasown
28K	xtend
24K	strip-ansi
24K	postgres-interval
24K	postgres-array
24K	env-paths
24K	ansi-regex
20K	postgres-date
20K	postgres-bytea
20K	pg-int8
20K	path-parse
20K	ms
20K	module-details-from-path
20K	@types/shimmer
20K	@types/pg-pool
20K	@types/connect

@galargh galargh changed the title [WIP - not ready for review] Collect instrumentation metadata and node:test coverage data feat: collect instrumentation metadata and node:test coverage data May 8, 2025
@galargh galargh marked this pull request as ready for review May 8, 2025 14:24
@galargh galargh requested a review from alcuadrado May 8, 2025 14:24
This was linked to issues May 14, 2025
Reviewing serializable data formats shared with EDR #6556
Closed
Small set of examples for end to end #6557
Closed
Add global --coverage option #6579
Closed
Instrument the code on compilation job creation when coverage is enabled #6580
Closed
Collect coverage data in test node task #6581
Closed
Collect coverage in test mocha task #6582
Closed
Collect coverage in test solidity task #6583
Open
Implement coverage data merge function #6584
Closed
alcuadrado
alcuadrado reviewed May 21, 2025
View reviewed changes
alcuadrado
alcuadrado reviewed May 21, 2025
View reviewed changes
alcuadrado
alcuadrado approved these changes May 21, 2025
View reviewed changes
Copy link
Member

@alcuadrado alcuadrado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments, but it looks good in general

galargh added 5 commits May 22, 2025 11:42
@galargh
feat: report coverage from test, test mocha and test node tasks (#6687)
d485fd3 
* feat: generate lcov coverage report

* feat: infer line coverage from statement coverage

* fix: line number reporting

* wip

* feat: display coverage report on the terminal too

* chore: refactor coverage manager

* feat: use ids in coverage manager handlers

* feat: implement coverage for mocha

* feat: do not instrument test sources

* feat: display coverage report on hardhat test

* chore: remove empty file

* test: fix coverage manager implementation tests

* chore: show - for empty sets in the coverage report

* feat: pass coverage library path to edr

* chore: add comments

* feat: incldue branch coverage in the lcov report

* fix: the lcov file

* chore: update coverage manager interface

* chore: move coverage option check to helpers

* chore: format the markdown report better

* test: implement more coverage manager tests

* feat: pass fs path via the project file preprocessing hook

* chore: include fs path in the coverage metadata

* feat: replace source name with relative file path in coverage metadata

* fix: fix the coverage report formatting rules

* chore: use helpers to format total coverage

* chore: fix coverge related tests

* docs: added changesets

* test: fix the path formatting test on windows
@galargh
chore: remove the unsafe hre casting function
354b78a
@galargh
Merge remote-tracking branch 'origin/v-next' into coverage-v2
5535de9
@galargh
chore: remove coverage.sol from hardhat exports
2bfc953
@galargh
chore: update coverage data type
3c8a3c5
@galargh galargh added this pull request to the merge queue May 22, 2025
Merged via the queue into v-next with commit 8557f97 May 22, 2025
114 checks passed
@galargh galargh deleted the coverage-v2 branch May 22, 2025 12:05
@github-project-automation github-project-automation bot moved this from Backlog to Done in Hardhat May 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alcuadrado alcuadrado alcuadrado approved these changes

Assignees
No one assigned
Labels
status:ready This issue is ready to be worked on
Projects
Hardhat
Status: Done
Milestone
No milestone
2 participants
@galargh @alcuadrado
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy