(Chore): Fix repo tests #1027

TylerJang27 · 2025-05-14T08:00:48Z

Additional cleanup as follow-up to #1032.

Mostly aligning snapshots to match the test versions for cleanliness. Other notes:

Semgrep problem with no issues being reported

gemini-code-assist

Hello @TylerJang27, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

This pull request addresses some test failures and updates dependencies. Specifically, it updates Jest snapshots for cfnlint, pmd, and squawk linters, disables caching for semgrep tests, and bumps the known good version of Node.js from 18.20.5 to 20.19.1, and sourcery from 1.2.0 to 1.4.0.

Highlights

Snapshot Updates: Jest snapshots for cfnlint, pmd, and squawk linters have been updated to reflect changes in their output.
Semgrep Test Modification: The semgrep test has been modified to disable caching, likely to ensure consistent test results.
Node.js Version Bump: The known good version of Node.js has been updated from 18.20.5 to 20.19.1 in runtimes/node/plugin.yaml and tests/driver/lint_driver.ts.
Sourcery Version Bump: The known good version of Sourcery has been updated from 1.2.0 to 1.4.0 in tools/sourcery/plugin.yaml.

Changelog

Click here to see the changelog

linters/cfnlint/test_data/cfnlint_v1.35.1_basic.check.shot
- Updated Jest snapshot for cfnlint linter.
linters/pmd/test_data/pmd_v7.13.0_CUSTOM.check.shot
- Updated Jest snapshot for pmd linter.
linters/semgrep/semgrep.test.ts
- Disabled caching for semgrep tests by adding --cache=false argument.
linters/squawk/test_data/squawk_v1.5.3_basic.check.shot
- Removed trunk-upgrade-validation comment.
linters/squawk/test_data/squawk_v2.4.0_basic.check.shot
- Updated Jest snapshot for squawk linter.
runtimes/node/plugin.yaml
- Bumped known good version of Node.js from 18.20.5 to 20.19.1.
tests/driver/lint_driver.ts
- Bumped known good version of Node.js from 18.20.5 to 20.19.1.
tools/sourcery/plugin.yaml
- Bumped known good version of Sourcery from 1.2.0 to 1.4.0.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

The tests did fail,
Snapshots needed to prevail,
Node version bumped,
Cache was trumped,
Now all the checks set sail.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request updates snapshots and bumps the node version. The changes seem straightforward and necessary for keeping the tests up-to-date. However, I have a few observations regarding the semgrep test and squawk snapshots.

Summary of Findings

Semgrep Caching: The --cache=false argument in the semgrep test disables caching, which may significantly increase test execution time. Consider adding a comment explaining why caching is disabled or exploring alternative solutions.
Removed trunk-upgrade-validation line: The line // trunk-upgrade-validation:RELEASE was removed from linters/squawk/test_data/squawk_v1.5.3_basic.check.shot. Verify that this was intentional.
known_good_version Updates: For all known_good_version updates, it would be helpful to add a comment explaining why this version was chosen and what testing was done to validate it.

Merge Readiness

The pull request seems mostly ready for merging, but I recommend addressing the comments regarding semgrep caching and the removed trunk-upgrade-validation line. Adding comments to the known_good_version updates would also be beneficial. I am unable to approve this pull request, and recommend that others review and approve this code before merging.

gemini-code-assist · 2025-05-14T08:01:18Z

linters/semgrep/semgrep.test.ts

-customLinterCheckTest({ linterName: "semgrep", args: TEST_DATA, skipTestIf: skipOS(["win32"]) });
+customLinterCheckTest({
+  linterName: "semgrep",
+  args: `${TEST_DATA} --cache=false`,


Adding --cache=false disables caching for semgrep. While this ensures the test always fetches the latest rules, it significantly increases test execution time. Consider adding a comment explaining why caching is disabled, or explore alternative solutions like invalidating the cache when necessary. Otherwise, consider removing this argument to improve test performance.

Suggested change

args: `${TEST_DATA} --cache=false`,

args: `${TEST_DATA} --cache=false`, // Disables caching to ensure latest rules are always fetched

trunk-staging-io · 2025-05-14T08:01:24Z

Running Code Quality on PRs by uploading data to Trunk will soon be removed. You can still run checks on your PRs using trunk-action - see the migration guide for more information.

chip-agent · 2025-05-14T08:15:54Z

Based on the file changes, the PR seems to be focused on updating test snapshots and versions for various linters. The description mentions "aligning snapshots to match the test versions for cleanliness".

Since I couldn't retrieve the test logs, I'll make some educated guesses about potential flaky tests based on the changed files. It's possible that the tests for semgrep and sourcery might be flaky, as there are changes in their test files (semgrep.test.ts, sourcery.test.ts) and plugin configurations (plugin.yaml). The addition of squawk tests might also be a source of flakiness if the tests are not stable.

Summary:
The PR updates test snapshots and versions for several linters. Without test logs, identifying specific flaky tests is difficult, but semgrep, sourcery, and squawk tests are potential candidates.

Root Cause Analysis:
The changes in test snapshots and versions could introduce flakiness if the new tests or versions are not stable or have timing-related issues. The addition of new tests for squawk might also contribute to flakiness.

Resolution Plan:

Re-run the CI pipeline to see if the tests pass on another attempt.
If the tests continue to fail, investigate the semgrep.test.ts, sourcery.test.ts and the newly added squawk tests. Look for timing-related issues, race conditions, or dependencies on external resources that might cause inconsistent behavior.
If a test is identified as flaky, consider increasing timeouts or using more robust synchronization mechanisms to make the test more stable.
If the flakiness persists, mark the test as flaky and investigate further to find the root cause.

Since I cannot access the test logs, the user should investigate the tests mentioned above.

Is this helpful? Click the 👍/👎 buttons below to let us know!

Icons by Icons8

trunk-staging-io · 2025-05-14T08:39:12Z

Failed Test	Failure Summary	Logs
`Testing linter sourcery test CUSTOM`	Error installing hermetic tool sourcery v1.37.1, could not find a version that satisfies the requirement.	Logs ↗︎

Flaky Test	Failure Summary	Logs
`Testing linter sourcery test CUSTOM`	The test failed because of linting issues in the 'test_data/basic.in.py' file.	Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

trunk-io · 2025-05-14T08:49:00Z

_{View Full Report ↗︎ ⋅ Docs}

trunk-staging-io · 2025-05-19T18:20:25Z

tldr; The tests are failing due to snapshot mismatches caused by version changes in the PR. Update the snapshots by running the tests with the PLUGINS_TEST_UPDATE_SNAPSHOTS=true flag.

Detailed Analysis

Summary

Several tests are failing with snapshot mismatches. These failures are related to the changes made in this PR, particularly:

Updating Node.js version from 18.20.5 to 20.19.1
Updating the sourcery version from 1.2.0 to 1.4.0
Adding the --cache=false flag to semgrep tests

Root Cause Analysis

This is a case of Intentional Code Change where the PR has deliberately updated versions and configurations, but the test snapshots haven't been updated to match the new expected output.

The main issues are:

Semgrep Test Failure: The test is failing because the --cache=false flag was added to the semgrep test, which changes the output. The test now produces empty issues instead of the expected issues in the snapshot.
SQLFluff Test Failure: The test is failing because the SQLFluff linter is encountering an error with the disable_progress_bar option, which might be related to the Node.js version upgrade.
Sourcery Test Failure: The test is failing because the sourcery version was updated from 1.2.0 to 1.4.0, but the installation is failing with the error message: "Could not find a version that satisfies the requirement sourcery==1.2.0".
Formatter Test Failures: The SQLFluff formatter tests are failing due to differences in formatting output, which could be related to the version changes.

Resolution Plan

Update Test Snapshots:
Run the tests with the PLUGINS_TEST_UPDATE_SNAPSHOTS=true environment variable to update the snapshots to match the new expected output:
```
PLUGINS_TEST_UPDATE_SNAPSHOTS=true npm test
```
Fix Sourcery Version Inconsistency:
There's an inconsistency between the PR changes and the actual test. The PR updates the sourcery version to 1.4.0 in tools/sourcery/plugin.yaml, but the test is still trying to install version 1.2.0. Make sure the test is using the updated version.
Verify SQLFluff Compatibility:
Check if the SQLFluff linter is compatible with the new Node.js version. You might need to update the SQLFluff configuration to handle the disable_progress_bar option correctly.
Commit the Updated Snapshots:
After running the tests with the update flag, commit the new snapshot files that will be generated.

These changes are expected when updating dependencies and configurations, and the solution is to update the test snapshots to match the new expected behavior.

trunk-staging-io · 2025-05-19T22:08:28Z

tldr; Your PR is failing because you've updated Node.js and tool versions which has changed the output format of several linters, causing snapshot test failures. You need to update the test snapshots to match the new output format.

## Detailed Analysis

Summary

Your PR #1027 "(Chore): Fix repo tests" is failing several snapshot tests. The failures are related to changes in the output format of various linters after you updated Node.js from v18.20.5 to v20.19.1 and updated the sourcery tool from v1.2.0 to v1.4.0.

Root Cause Analysis

The primary issue is that you've made version updates to Node.js and tools, which has changed the output format of several linters. The test suite compares the current output against stored snapshots, and these snapshots no longer match the new output format.

Specifically:

Semgrep Test Failure:
- You modified linters/semgrep/semgrep.test.ts to add --cache=false to the test arguments
- The output format has changed, showing empty issues where previously there were detailed issues
SQLFluff Test Failures:
- Both linter and formatter tests are failing
- The linter test shows task failures with error messages about ValueError: Expected 'deprecated' value for 'disable_progress_bar'
- The formatter test shows differences in whitespace formatting
Sourcery Test Failure:
- You updated the version from 1.2.0 to 1.4.0
- The test is failing because the new version can't be installed: ERROR: Could not find a version that satisfies the requirement sourcery==1.2.0

Resolution Plan

Update Test Snapshots:
The simplest solution is to update the test snapshots to match the new output format. This is a common practice when intentionally changing behavior or dependencies.
```
# Run the tests with the snapshot update flag
npm test -- -u
```
Fix Specific Issues:
- For SQLFluff:
  The error suggests an issue with the disable_progress_bar parameter. Check if this parameter has been deprecated in the newer version and update the plugin.yaml accordingly.
- For Sourcery:
  The error indicates that version 1.2.0 is not available. The available versions listed are 1.0.0, 1.4.0, 1.5.0, etc. You've correctly updated to 1.4.0 in the plugin.yaml, but you may need to update the test snapshots to match the new version's output.
Verify Changes:
After updating the snapshots, run the tests again to ensure they pass. Make sure to commit the updated snapshot files.

Additional Notes

The Node.js version update from 18.20.5 to 20.19.1 is significant and may have other impacts on the test environment.
The PR description mentions "Mainly snapshot updates. Bumps node version too" which suggests you were aware of the need to update snapshots, but may not have completed all the necessary updates.
Some of the test failures show that the linters are now failing to run properly, which may require more than just snapshot updates to fix.

TylerJang27 · 2025-05-27T06:04:23Z

linters/semgrep/semgrep.test.ts

The verbose output of semgrep includes this bit:

Skipped by --include patterns: • test_data/basic.go • test_data/element.ts • test_data/empty_go.go • test_data/empty_js.js • test_data/empty_py.py • test_data/request.py

despite the includes having those exact files. Subsequent runs do not have this problem. I tried getting around this with various flags like --no-git-ignore --verbose but to no avail. I don't have a more isolate repro handy since this seems to happen only after startup/with fresh test dirs, but it should probs be reported to semgrep eventually

gemini-code-assist bot reviewed May 14, 2025

View reviewed changes

TylerJang27 mentioned this pull request May 27, 2025

Add additional environment variables used by python networking stack #1032

Merged

TylerJang27 added 5 commits May 26, 2025 22:03

update node version

6ca8e6c

pmd snapshot update

3a28245

update squawk snpashot

1474a0e

cfnlint snapshot

266bd26

cleanup more

38d5727

TylerJang27 force-pushed the tyler/fix-repo2 branch from f51e438 to 38d5727 Compare May 27, 2025 05:10

fix semgrep tests, but the problem remains

0652eed

TylerJang27 commented May 27, 2025

View reviewed changes

fix sourceyr

04c1d34

TylerJang27 requested review from det, EliSchleifer and puzzler7 May 27, 2025 06:27

puzzler7 approved these changes May 27, 2025

View reviewed changes

TylerJang27 merged commit 76ba8d9 into main May 27, 2025
14 checks passed

TylerJang27 deleted the tyler/fix-repo2 branch May 27, 2025 06:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(Chore): Fix repo tests #1027

(Chore): Fix repo tests #1027

Uh oh!

TylerJang27 commented May 14, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot May 14, 2025

Uh oh!

trunk-staging-io bot commented May 14, 2025 •

edited

Loading

Uh oh!

chip-agent bot commented May 14, 2025 •

edited

Loading

Uh oh!

trunk-staging-io bot commented May 14, 2025 •

edited

Loading

Uh oh!

trunk-io bot commented May 14, 2025 •

edited

Loading

Uh oh!

trunk-staging-io bot commented May 19, 2025

Summary

Root Cause Analysis

Resolution Plan

Uh oh!

trunk-staging-io bot commented May 19, 2025

Summary

Root Cause Analysis

Resolution Plan

Additional Notes

Uh oh!

TylerJang27 May 27, 2025

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

	args: `${TEST_DATA} --cache=false`,
	args: `${TEST_DATA} --cache=false`, // Disables caching to ensure latest rules are always fetched

(Chore): Fix repo tests #1027

(Chore): Fix repo tests #1027

Uh oh!

Conversation

TylerJang27 commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Changelog

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Summary of Findings

Merge Readiness

Uh oh!

gemini-code-assist bot May 14, 2025

Choose a reason for hiding this comment

Uh oh!

trunk-staging-io bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chip-agent bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-staging-io bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-staging-io bot commented May 19, 2025

Summary

Root Cause Analysis

Resolution Plan

Uh oh!

trunk-staging-io bot commented May 19, 2025

Summary

Root Cause Analysis

Resolution Plan

Additional Notes

Uh oh!

TylerJang27 May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

TylerJang27 commented May 14, 2025 •

edited

Loading

trunk-staging-io bot commented May 14, 2025 •

edited

Loading

chip-agent bot commented May 14, 2025 •

edited

Loading

trunk-staging-io bot commented May 14, 2025 •

edited

Loading

trunk-io bot commented May 14, 2025 •

edited

Loading