[CI] Need better way to detect OOMs especially on pet instances #157379
Description
🐛 Describe the bug
See https://hud.pytorch.org/hud/pytorch/pytorch/720c2c46b181ad446bf970d70a27fd0ce149114d/1?per_page=50&name_filter=macos-m2-15
It was unstable like that last Tue-Wed, but was table since Thu, see https://hud.pytorch.org/hud/pytorch/pytorch/720c2c46b181ad446bf970d70a27fd0ce149114d/4?per_page=50&name_filter=macos-m2-15
Versions
CI
cc @seemethere @pytorch/pytorch-dev-infra @clee2000