Setup TorchBench in Docker #158613

huydhn · 2025-07-18T00:21:08Z

This reduces the time spending to setup TorchBench in A100/H100 by another half an hour

Testing

H100 benchmark https://github.com/pytorch/pytorch/actions/runs/16396172453. Once this done, I will review the results on HUD to confirm that all models are there
A100 benchmark https://github.com/pytorch/pytorch/actions/runs/16396173932

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @Lucaskabela

[ghstack-poisoned]

huydhn · 2025-07-18T00:21:09Z

Stack from ghstack (oldest at bottom):

-> Setup TorchBench in Docker #158613

pytorch-bot · 2025-07-18T00:21:12Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158613

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

linux.aws.h100.8 instance is down, potentially longer queue on linux.aws.h100

❌ 8 New Failures, 1 Unrelated Failure

As of commit 9867f10 with merge base d36afac ():

NEW FAILURES - The following jobs have failed:

inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
fastNLP_Bert
inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
pytorch_CycleGAN_and_pix2pix
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.8xlarge.amx) (gh)
fastNLP_Bert
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.8xlarge.amx) (gh)
pytorch_CycleGAN_and_pix2pix
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 1, 2, linux.8xlarge.amx) (gh)
fastNLP_Bert
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.8xlarge.amx) (gh)
pytorch_CycleGAN_and_pix2pix
trunk / linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4) (gh)
distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context
trunk / verify-cachebench-cpu-test / test (verify_cachebench, 1, 1, lf.linux.2xlarge) (gh)
ModuleNotFoundError: No module named 'torchbenchmark'

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, linux.2xlarge) (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This reduces the time spending to setup TorchBench in A100/H100 by half an hour Signed-off-by: Huy Do <huydhn@gmail.com> ghstack-source-id: 7425aa2 ghstack-comment-id: 3086043340 Pull-Request: #158613

[ghstack-poisoned]

This reduces the time spending to setup TorchBench in A100/H100 by half an hour Signed-off-by: Huy Do <huydhn@gmail.com> ghstack-source-id: 9eaa733 ghstack-comment-id: 3086043340 Pull-Request: #158613 Signed-off-by: Huy Do <huydhn@gmail.com>

[ghstack-poisoned]

This reduces the time spending to setup TorchBench in A100/H100 by half an hour Signed-off-by: Huy Do <huydhn@gmail.com> ghstack-source-id: 536ab13 ghstack-comment-id: 3086043340 Pull-Request: #158613 Signed-off-by: Huy Do <huydhn@gmail.com>

[ghstack-poisoned]

This reduces the time spending to setup TorchBench in A100/H100 by another half an hour Signed-off-by: Huy Do <huydhn@gmail.com> ghstack-source-id: c3c6d5b ghstack-comment-id: 3086043340 Pull-Request: #158613 Signed-off-by: Huy Do <huydhn@gmail.com>

[ghstack-poisoned]

janeyx99

This only affects the docker images used in inductor benchmarks right? Or will this make pulling docker images across all of CI slightly slower?

janeyx99

Approving cuz this seems ok, my only question is that it does not affect anything else negatively

huydhn · 2025-07-21T16:46:17Z

This only affects the docker images used in inductor benchmarks right? Or will this make pulling docker images across all of CI slightly slower?

Yup, the size of that benchmark image increases, but it's cached on the runner. I'm trying to confirm at the moment that the cache works correctly by re-running the benchmark to warm it up. Let me confirm that before landing this change.

huydhn · 2025-07-21T17:18:52Z

Confirmed:

1st run, cold Docker cache took half an hour in pull Docker image step https://github.com/pytorch/pytorch/actions/runs/16396172453/job/46329724191
2nd run, warm Docker cache, same step took 8 minutes https://github.com/pytorch/pytorch/actions/runs/16396172453/job/46405894814

[ghstack-poisoned]

This reduces the time spending to setup TorchBench in A100/H100 by another half an hour Signed-off-by: Huy Do <huydhn@gmail.com> ghstack-source-id: ff22872 ghstack-comment-id: 3086043340 Pull-Request: #158613 Signed-off-by: Huy Do <huydhn@gmail.com>

huydhn · 2025-07-21T19:42:14Z

@pytorchbot merge

pytorch-bot · 2025-07-21T19:42:19Z

This PR has pending changes requested. Please address the comments and update the PR before merging.

Ready to land

huydhn · 2025-07-21T19:48:03Z

@pytorchbot merge

pytorchmergebot · 2025-07-21T19:50:11Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

ZainRizvi · 2025-07-22T00:10:59Z

@pytorchbot revert -c nosignal -m "Seems to have broken trunk. See GH job link HUD commit link"

pytorchmergebot · 2025-07-22T00:12:37Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-07-22T00:12:52Z

@huydhn your PR has been successfully reverted.

This reverts commit cab2833. Reverted #158613 on behalf of https://github.com/ZainRizvi due to Seems to have broken trunk. See [GH job link](https://github.com/pytorch/pytorch/actions/runs/16429779764/job/46430634676) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/b3c868d603e8f7b6661c93cd3d50c9a7b213ad6c) ([comment](#158613 (comment)))

huydhn added 8 commits July 17, 2025 15:51

Update

0e37fc8

[ghstack-poisoned]

Update

ec8fa3c

[ghstack-poisoned]

Update

3aec24e

[ghstack-poisoned]

Update

30c21cb

[ghstack-poisoned]

Update

444df0f

[ghstack-poisoned]

Update

55316ba

[ghstack-poisoned]

Update

a151d80

[ghstack-poisoned]

Update

0ce8369

[ghstack-poisoned]

huydhn requested review from a team and jeffdaily as code owners July 18, 2025 00:21

pytorch-bot bot added ciflow/inductor release notes: releng release notes category labels Jul 18, 2025

This was referenced Jul 18, 2025

Use linux.12xlarge.memory to build for H100/sm_90 #158598

Closed

Clean up some unused build env variables #158599

Closed

Build domain libraries on the build job #158600

Closed

Build domain libraries for all workflows with TorchBench config #158601

Closed

Update

9fe4955

[ghstack-poisoned]

huydhn added the no-runner-experiments Bypass Meta/LF runner determinator label Jul 18, 2025

huydhn added the test-config/default label Jul 18, 2025

huydhn added 3 commits July 17, 2025 18:14

Update

9ff8af5

[ghstack-poisoned]

Update

753fe8c

[ghstack-poisoned]

Update

2435a78

[ghstack-poisoned]

Update

2bc4a7d

[ghstack-poisoned]

Update

14a38c7

[ghstack-poisoned]

huydhn added test-config/default and removed ciflow/inductor labels Jul 19, 2025

huydhn requested a review from yangw-dev July 20, 2025 04:43

pytorch-bot bot added the ciflow/inductor label Jul 20, 2025

janeyx99 reviewed Jul 21, 2025

View reviewed changes

janeyx99 approved these changes Jul 21, 2025

View reviewed changes

Update

9867f10

[ghstack-poisoned]

huydhn mentioned this pull request Jul 21, 2025

[vllm] add vllm test base docker image #158755

Open

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 21, 2025

pytorchmergebot added the merging label Jul 21, 2025

pytorchmergebot closed this in cab2833 Jul 21, 2025

pytorchmergebot added Merged and removed merging labels Jul 21, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Jul 22, 2025

pytorchmergebot reopened this Jul 22, 2025

huydhn removed the test-config/default label Jul 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Setup TorchBench in Docker #158613

Setup TorchBench in Docker #158613

huydhn commented Jul 18, 2025 •

edited

Loading

Uh oh!

huydhn commented Jul 18, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

janeyx99 left a comment

Uh oh!

janeyx99 left a comment

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

pytorch-bot bot commented Jul 21, 2025

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Uh oh!

ZainRizvi commented Jul 22, 2025

Uh oh!

pytorchmergebot commented Jul 22, 2025

Uh oh!

pytorchmergebot commented Jul 22, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Setup TorchBench in Docker #158613

Are you sure you want to change the base?

Setup TorchBench in Docker #158613

Conversation

huydhn commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

huydhn commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158613

❗ 1 Active SEVs

❌ 8 New Failures, 1 Unrelated Failure

Uh oh!

janeyx99 left a comment

Choose a reason for hiding this comment

Uh oh!

janeyx99 left a comment

Choose a reason for hiding this comment

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

pytorch-bot bot commented Jul 21, 2025

Uh oh!

huydhn commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Merge started

Uh oh!

ZainRizvi commented Jul 22, 2025

Uh oh!

pytorchmergebot commented Jul 22, 2025

Uh oh!

pytorchmergebot commented Jul 22, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

huydhn commented Jul 18, 2025 •

edited

Loading

huydhn commented Jul 18, 2025 •

edited

Loading

pytorch-bot bot commented Jul 18, 2025 •

edited

Loading