Resolved -
On January 2, 2025 between 16:00:00 and 22:27:30 UTC, a bug in feature-flagged code that cleans up Pull Requests after they are closed or merged incorrectly cleared the merge commit SHA for ~139,000 pull requests. During the incident, Actions workflows triggered by the on: pull_request trigger for the closed type were not queued successfully because of these missing merge commit SHAs. Approximately 45,000 repositories experienced these missing workflow triggers in either of two possible scenarios: pull requests which were closed, but not merged; and pull requests which were merged. Impact was mitigated after rolling back the aforementioned feature flag.
Merged pull requests that were affected have had their merge commit SHAs restored. Closed pull requests have not had their merge commit SHA restored; however, customers can re-open and close them again to recalculate this SHA. We are investigating methods to improve detection of these kinds of errors in the future.
Jan 3, 00:19 UTC
Update -
All systems are operational, and we have a plan to backfill the missing metadata. In total, 139,000 PRs were impacted across 45,000 repositories. The backfilled metadata will be available in a few days.
Until the backfill is complete, there are several actions you can take to ensure an Action runs:
- Any Actions that should have run on closed but not merged PRs can be triggered by re-opening and re-closing the PR.
- Actions that should have run on PR merge can be re-run from the main branch of your repository.
The only Actions that cannot be re-run at this time are ones that specifically use the merge commit.
Additionally, the `merge_commit_sha` field on an impacted Pull Request will be `null` when queried via our API until the backfill completes.
We appreciate the error reports we received, and thank you for your patience. We mitigated the initial impact quickly by rolling back a feature flag. We will be improving the monitoring of our feature flag rollouts in the future to better catch these scenarios.
Jan 3, 00:19 UTC
Update -
We have remediated the issue impacting Actions workflows. During investigation and remediation, we realized there were also issues with recording metadata around merge commits. No git data or code has been lost. PRs merged today between 20:06 UTC and 22:15 UTC are impacted. We are working on a plan to regenerate the missing metadata and will provide an update once we have one in place.
Jan 2, 23:11 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Jan 2, 23:05 UTC
Update -
We have identified and begun to remediate the issue preventing Actions from triggering on closed pull requests. We are beginning to see recovery.
Jan 2, 22:30 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Jan 2, 22:09 UTC
