Resurrecting #56
At Canva they use a (home grown) package eslint-snapshot-test to snapshot the lint rule errors.

I found it's a really great way to work because it allows you to visualise the error ranges with the code instead of abstractly referencing the line/cols.

However the ergonomics of the package are off, IMO, because (a) it uses its own custom class to do testing which doesn't have any of the features of ESLint's RuleTester and (b) it prefers you organise rule tests as if they were bog standard tests (i.e. within describe/it) which sucks because there's so much boilerplate to reuse.

I propose we do two things:

(1) fork the base ESLint rule tester properly. Right now we are subclassing it so that we can add our customisations and it makes the code pretty complicated and convoluted. As I established in #6456 - I think we're getting to the scale with a lot of things where it's just plain better for us to fork instead of extending.

(2) add a new option to the RuleTester constructor like snapshot?: boolean = false which will automatically compute a snapshot for each invalid test.

For example

const tester = new RuleTester({ snapshot: true });

tester.run('no-methods', rule, {
  valid: [ ... ],
  invalid: [
    {
      description: 'Some meaningful label',
      code: `
class Foo {
  myMethodName() {
    // ...
  }
}
      `,
      errors: [{ messageId: 'noMethod' }],
    },
  ],
});

Would create a snapshot file like:

exports[`no-methods invalid Some meaningful label`] = `
class Foo {
  myMethodName() {
  ~~~~~~~~~~~~~~~~
    // ...
~~~~~~~~~~
  }
~~~ myMethodName is declared as a method, we prefer always using arrow functions because reasons
}
`;
exports[`no-methods invalid Some meaningful label`] = `
class Foo {
  myMethodName() {
  ~~~~~~~~~~~~~~~~
    // ...
~~~~~~~~~~
  }
~~~ myMethodName is declared as a method, we prefer always using arrow functions because reasons
}
`;

Which is much, much more readable than the current alternative test:

  invalid: [
    {
      description: 'Some meaningful label',
      code: `
class Foo {
  myMethodName() {
    // ...
  }
}
      `,
      errors: [
        {
          messageId: 'noMethod',
          data: { name: 'myMethodName' },
          line: 3,
          endLine: 5,
          column: 3,
          endColumn: 4
        },
      ],
    },
  ],

Pros:

• easier to understand the message users will actually see and validate it composes as expected
• easier to validate that the rule actually reports where you want it to
• easier for contributors to understand what the rule should do
• changes to report locations require "zero" maintenance - just snapshot updates

Cons:

• changes to message format require snapshot updates
• requires a separate snapshot file which adds some mental mapping to map a specific test object to a specific snapshot for review purposes.
  • to make this easier I propose we enforce that when snapshots are on, invalid tests must include the description attribute to help with identification and that each description is unique.

This system could also be extended to snapshot fixers.
For example you could envision a separate snapshot emitted for each fixer which is just the raw source output. We could do something similar to what we do with our AST snapshots and emit line-by-line diff of the code and output to make it clearer what the changes were.

In this case we'd emit 3..n snapshots per invalid case - one showing the error report location, (or clearly stating there was no fix performed), one showing the fix diff, and two for each suggestion fixer.

Problem to solve - sane organisation of snapshots.

We might have to think through how this should be structured though because emitting 3 snapshots per invalid test could be REALLY difficult to understand and mentally match up - especially if there are a lot of test cases. TBH just emitting 1 per test could be hard enough given the scale of invalid tests for some rules.

As mentioned before - enforcing the use of description could be pretty maintenance-heavy and cumbersome as well. Imagine trying to come up with 100 unique descriptions for tests that are slight syntactic variations. It could easily devolve into users just doing "Test Case 1", "Test Case 2", ... - which isn't really great (even though it would uniquely identify each test case, it would turn into a mess as you reorder the tests).

Even though our AST fixture system emits 6 snapshots per fixture (IMO) it is really easy to understand things because each snapshot which is a separate file on disk and is clearly named.

Spitballing possible solutions:
(1) just emit one monolithic snapshot file and trust that users will get used to it and "figure it out"
Pros:

• no work for us to do
• can leverage standard jest tooling to manage snapshots
• would probably work really well for incremental changes that add a few new test cases because the PR diff will be small and clearly show the new/changed cases

Cons:

• huge files are painful for the user to navigate.
• it will suck for brand new rules that introduce dozens or hundreds of test cases.

(2) create a unique, filename-safe name per test and emit one snapshot file for each test case.
Pros:

• shrink the size of each snapshot file and make it easy to marry things up because the user just needs to look for a specific file on disk, rather than a set of snapshots in a monolithic file.

Cons:

• creating a unique name that's easy to marry up with the original test would be difficult.
  • Perhaps this can be a requirement for the test case config instead of a description? I.e. push that burden onto the user?
• we would have to build logic to automatically manage these files as well as jest/test frameworks wouldn't do it out-of-the-box
  • Creating is easy (eg we can just leverage jest-specific-snapshot like we currently do)
  • Finding cases that no longer exist is harder and requires us to maintain a mapping of files and track which ones were/weren't tested
  • Would need to figure out a way to not falsely report skipped tests as not emitted.

(3) use the description to create folders on disk and emit individual snapshots.
Pros:

• This would be much the same as our AST fixture system, so we know what we're doing to some degree

Cons:

• Same problems as (2) though - would need a unique folder name per test and would ALSO need a unique file name per suggestion fixer.

(4) break compatibility entirely and instead use a similar structure to our AST fixture system - one file per fixture.
Pros:

• This would save us doing anything uniquely (aside from suggestion fixer filenames) as we'd place the naming burden directly on the user.
• Really clear system because the snapshots can live in a subfolder along-side the fixture

Cons:

• Might be difficult for the user to manage though - lots of files that they need to create and manage.
• Makes "grepping" for cases pretty hard as you need to grep across multiple files.
• We'd need to create a system for adding config to the test cases
  • Perhaps could just go the "dumb" route here and use eslint config comments in the fixture file like:

    /* eslint @plugin/no-methods: ['error', { config: value, ... }] */
    class Foo { ... }

  • Could also do a "fourslash" style system to add config directly in the fixture?
  • Could also just accept a separate config.ts file which exports a config object and provide some utils to easily type the object.

(5) do a similar system to eslint-snapshot-test in which the user decides how to name tests and how to properly handle the snapshot side of things.

For example they could use inline snapshots if they wanted:

const tester = new SnapshotRuleTester();

tester.run('rule-name', rule, {
  valid: [ ... ],
  invalid: [
    tester.invalid('foos-the-bar', (test) => {
      const result = test({
        code: ' ... ',
        options: [ ... ],
        errors: [ { messageId: 'foo' } ],
      });

      expect(result).toMatchInlineSnapshot();
    });
  ],
});

Pros:

• flexible
• allows the user to use any test framework
• saves us from the complexity of managing things
• opens the door to custom test logic if the user wants more power

Cons:

• de-standardises the ecosystem
• adds a lot of boilerplate
• adds a learning curve for users to figure out how to do things
• would require creating of lint rules to help set some best-practices.