PYTHON-5389: Add Tags to each buildvariant and augment test skipping/tracking policy. #82
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds tagging support to build variants and updates the test notification policy in the documentation.
- Updated README.md to document the new tags field and its effect on Slack notifications.
- Modified .evergreen/config.yml to add language tags for each build variant.
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| README.md | Updated documentation to include explanation for the new "tags" field. |
| .evergreen/config.yml | Added a "tags" property for each build variant to support language tagging. |
Comments suppressed due to low confidence (1)
README.md:190
- There is an inconsistency in the naming of the notification channel: 'dbx-ai-ml-testing-pipline-notifications' appears to be a typo compared to 'dbx-ai-ml-testing-pipeline-notifications-{language}'. Consider standardizing the channel names.
Tests are run periodically (nightly) and any failures will propagate into both the `dbx-ai-ml-testing-pipline-notifications` and `dbx-ai-ml-testing-pipeline-notifications-{language}` channel. Repo owners of this `ai-ml-testing-pipeline` library are required to join the `dbx-ai-ml-testing-pipeline-notifications`. Pipeline specific implementers must **at least** join `dbx-ai-ml-testing-pipline-notifications-{language}` (e.g. whomever implemented `langchain-js` must at least be a member of `dbx-ai-ml-testing-pipeline-notifications-js`).
| If tests are found to be failing, and cannot be addressed quickly, the responsible team MUST create a JIRA ticket, and disable the relevant tests | ||
| in the `config.yml` file, with a comment about the JIRA ticket that will address it. | ||
|
|
||
| This policy will help ensure that a single failing integration does not cause noise in the `dbx-ai-ml-testing-pipeline-notifications` that would mask other | ||
| This policy will help ensure that a single failing integration does not cause noise in the `dbx-ai-ml-testing-pipeline-notifications` or `dbx-ai-ml-testing-pipeline-notifications-{language}` that would mask other |
There was a problem hiding this comment.
Tests are run periodically (nightly) and any failures will propagate into both the
dbx-ai-ml-testing-pipline-notificationsanddbx-ai-ml-testing-pipeline-notifications-{language}channel.
Won't there still be noise in the shared channel because failures are propagated to both slack channels?
There was a problem hiding this comment.
Yes, the shared channel then becomes an opt-in. If there are several failures in one day across multiple drivers, it's difficult to parse. With individual channels it's easier to pay attention to only the relevant driver team notifications. The only users required to stay within that larger channel are those with a vested interest across driver implementations or the repo owners.
…heck in lint_config.py
Summary
I've added in a new stipulation on each
buildvariantto require atagspecifying a language. As well, I've added linters to abide by the naming conventions onbuildvariant.names as well.Changes in this PR
lint_config.pywhich checks each buildvariant has a tag of a defined languagelint_config.pyspecifically on theconfig.ymlfile.README.mdto highlight the need for atagfield with a language value on every buildvariant.Screenshot(s)
Below shows the code catching a missing tag on a buildvariant in the
config.yaml