Skip to content

Fixes and cleanup #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 190 commits into
base: modular-fused-experts
Choose a base branch
from
Open

Fixes and cleanup #82

wants to merge 190 commits into from

Conversation

varun-sundar-rabindranath
Copy link

@varun-sundar-rabindranath varun-sundar-rabindranath commented May 9, 2025 •
edited by github-actions bot
Loading

Fixes and Cleanups

  • Removed batched_silu_mul kernels (the existing kernels work as it is albeit a bit expensive)
  • Destroy a2a objects in parallel_state.py as part of pplx_finalize
  • Cleanup DPMetadata in forward_context
  • Fix other models that use shared experts (force shared_experts output reduce)

bnellnm added 30 commits May 7, 2025 15:35
@bnellnm
moe refactoring
1f86a2d 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
module deepgemm moe working
6b6630b 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
working deep gemm, wip cutlass
33c61d6 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
working cutlass
e1ab18a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
deepgemm working again
4965854 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cutlass working again
a6564c1 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cutlass working again
1968c4a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix inplace, format and name cleanups
be10ed4 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix inplace, format + name cleanups
f8b64f5 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
test improvements
e94d6c1 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
make modular triton classes, fix edge cases
d675138 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix outplace bug
a665564 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
refactor dispatch/combine stuff
a6459aa 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
initial pplx dispatch/combine class
da3fe2b 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
merge triton dispatch into standard, add some comments
f2fe65a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
format
caf9805 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
comments
fab51b8 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix linter
5694036 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix more linter stuff
124f0ba 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cleanup for review
d98a3c3 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
review comments
a00c12c 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
forgot return
27cc6d1 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
add dp_rank_num_tokens to DPMetadata
1db9fcf 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
better check for fp8 in _fp8_permute
374c55c 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
updates
fbf9370 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix merge issues
be5c8d8 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix lint
bdcaae2 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
add pplx tests
e3ba64f 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
lint
897f3a1 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
undo random lint changes
3feab72 
Signed-off-by: Bill Nell <bnell@redhat.com>
bnellnm and others added 15 commits May 7, 2025 15:37
@bnellnm
wip ref impl
0f2e37a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
improve ref impl
a003bd8 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
wip
2bafbe0 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix merge
ca763c3 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix merge
054c10a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
wip
0851b31 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
zero out attn outputs during profile run
3e2cf4b 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
lint
d2862c0 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
lint
e478c1a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
revert lint changes to requirements/test.txt
90e9d05 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
revert lint changes to compiler_interface.py
be1a8e5 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix merge
916f902 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix more lint errors
c04cb12 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix lint
f5bcc22 
Signed-off-by: Bill Nell <bnell@redhat.com>
Fixes and cleanup
f7b7070 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

varun-sundar-rabindranath
varun-sundar-rabindranath commented May 9, 2025
View reviewed changes
vllm/distributed/parallel_state.py

from vllm.model_executor.layers.fused_moe.layer import (
_all_to_all_cache)
_all_to_all_cache.destroy()
Copy link
Author

@varun-sundar-rabindranath varun-sundar-rabindranath May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this pollutes parallel_state.py with layer level details . Should we move the entire pplx_finalize thing elsewhere ? / is there a good place to put the _all_to_all_cache.destroy()

varun-sundar-rabindranath
varun-sundar-rabindranath commented May 9, 2025
View reviewed changes
vllm/distributed/utils.py
_shutdown_backend(pg)
else:
pg.shutdown()

Copy link
Author

@varun-sundar-rabindranath varun-sundar-rabindranath May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all the changes to this file, shutdown is still not smooth. Should revisit this - need help with this.

Varun Sundar Rabindranath added 2 commits May 9, 2025 09:18
import sort
af02167 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
fix shutdown
081f11f 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
@bnellnm bnellnm force-pushed the modular-fused-experts branch from f5bcc22 to 5ba84d2 Compare May 9, 2025 16:46
@bnellnm bnellnm requested a review from mgoin as a code owner May 9, 2025 16:46
@bnellnm bnellnm force-pushed the modular-fused-experts branch 4 times, most recently from 705da89 to 1f91cfd Compare May 14, 2025 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@varun-sundar-rabindranath @bnellnm @tlrmchlsmth @ilmarkov @yma11
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy