Expose vLLM Metrics to serve.llm API #52719

eicherseiji · 2025-05-01T16:32:04Z

Why are these changes needed?

This change provides visibility into Ray Serve LLM deployments, including vLLM-specific statistics.

Dashboard panels:

Docs:

Related issue number

JR-1864

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Tested following steps on https://docs.ray.io/en/latest/cluster/metrics.html

kouroshHakha

Just some V0 vs. V1 stuff. Could you also ask from observability team to review as well??

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py

python/ray/llm/_internal/serve/configs/server_models.py

python/ray/dashboard/modules/metrics/dashboards/common.py

python/ray/dashboard/modules/metrics/dashboards/serve_dashboard_panels.py

python/ray/dashboard/modules/metrics/dashboards/serve_llm_dashboard_panels.py

kouroshHakha

The changes to server_models and vllm_engine looks good to me. Thanks a ton.

kouroshHakha

Could You create docs for logging?

Basically you want to cover:

How to enable logging?
What does logging give you: i.e engine emitted metrics like vllm metrics about cache hit rate, spec decoding hit rate, etc + service level metrics like number of input tokens served, output tokens, etc
Maybe with some nice screenshots.

You don't need to create an extensive list of all metrics.

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_loggers.py

kouroshHakha

some minor change requests:

doc/source/serve/llm/serving-llms.rst

kouroshHakha

LGTM

doc/source/serve/llm/serving-llms.rst

dstrodtman

Some suggestions, mostly for clarity and to improve readability and SEO.

doc/source/serve/llm/serving-llms.rst

eicherseiji · 2025-05-13T14:40:48Z

Thanks @dstrodtman for comments!

angelinalg

Just some nits. Thanks for doing the tech writer review, Douglas and the quick resolutions, @eicherseiji!

doc/source/serve/llm/serving-llms.rst

eicherseiji · 2025-05-13T16:54:18Z

Thanks @angelinalg!

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

…1 only Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>

Signed-off-by: Seiji Eicher <seiji@anyscale.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: zhaoch23 <c233zhao@uwaterloo.ca>

Signed-off-by: Seiji Eicher <seiji@anyscale.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: iamjustinhsu <jhsu@anyscale.com>

Signed-off-by: Seiji Eicher <seiji@anyscale.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>

Adding back some default panel configurations that were accidentally removed in a prior PR #52719 Signed-off-by: Alan Guo <aguo@anyscale.com>

Adding back some default panel configurations that were accidentally removed in a prior PR ray-project#52719 Signed-off-by: Alan Guo <aguo@anyscale.com>

eicherseiji self-assigned this May 1, 2025

eicherseiji force-pushed the JR_1864 branch 2 times, most recently from 33cca0b to 56e7858 Compare May 7, 2025 02:52

hainesmichaelc added the community-contribution Contributed by the community label May 7, 2025

eicherseiji requested review from GeneDer and kouroshHakha May 8, 2025 00:05

kouroshHakha reviewed May 8, 2025

View reviewed changes

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py Outdated Show resolved Hide resolved

python/ray/llm/_internal/serve/configs/server_models.py Outdated Show resolved Hide resolved

kouroshHakha requested a review from alanwguo May 8, 2025 16:58

kouroshHakha removed the community-contribution Contributed by the community label May 8, 2025

hainesmichaelc added the community-contribution Contributed by the community label May 8, 2025

alanwguo reviewed May 8, 2025

View reviewed changes

eicherseiji marked this pull request as ready for review May 9, 2025 00:46

eicherseiji requested a review from a team as a code owner May 9, 2025 00:46

kouroshHakha approved these changes May 9, 2025

View reviewed changes

alanwguo approved these changes May 9, 2025

View reviewed changes

eicherseiji force-pushed the JR_1864 branch from e370115 to 92e324e Compare May 9, 2025 20:55

kouroshHakha approved these changes May 9, 2025

View reviewed changes

eicherseiji force-pushed the JR_1864 branch from 92e324e to e4aee09 Compare May 9, 2025 23:59

eicherseiji added the go add ONLY when ready to merge, run all tests label May 10, 2025

eicherseiji mentioned this pull request May 12, 2025

[Misc] Add Ray Prometheus logger to V1 vllm-project/vllm#17925

Merged

kouroshHakha reviewed May 12, 2025

View reviewed changes

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_loggers.py Show resolved Hide resolved

eicherseiji requested review from edoakes, zcin, akshay-anyscale and a team as code owners May 12, 2025 20:48

kouroshHakha reviewed May 12, 2025

View reviewed changes

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

kouroshHakha approved these changes May 12, 2025

View reviewed changes

kouroshHakha enabled auto-merge (squash) May 12, 2025 22:16

masoudcharkhabi added serve Ray Serve Related Issue usability labels May 12, 2025

eicherseiji force-pushed the JR_1864 branch from efdc9f0 to 89f9e89 Compare May 12, 2025 23:15

github-actions bot disabled auto-merge May 12, 2025 23:15

dstrodtman reviewed May 13, 2025

View reviewed changes

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

dstrodtman reviewed May 13, 2025

View reviewed changes

angelinalg approved these changes May 13, 2025

View reviewed changes

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

eicherseiji and others added 12 commits May 13, 2025 18:17

Expose vLLM Metrics to serve.llm API

cba9ec9

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Add configuration to LLMConfig

85e934f

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Add stat logger to MQLLMEngine

8caaed1

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Initial LLM Serve panel

24d61cf

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Initial Grafana panels

6386887

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Progress porting metric dashboards

655bb6d

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Finalized dashboards, respond to comments, support metrics for vLLM V…

586ec34

…1 only Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Bring RayPrometheusStatLogger to vLLM V1

6e0fd17

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Add docs

dd2d730

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Change rayllm -> serve_llm

dd45b4b

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Docs revisions

a204da5

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Apply suggestions from code review

3e0a60d

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>

eicherseiji force-pushed the JR_1864 branch from 1cb2343 to 3e0a60d Compare May 13, 2025 18:17

kouroshHakha merged commit 881cd91 into ray-project:master May 13, 2025
5 checks passed

alanwguo mentioned this pull request May 20, 2025

fix MAX chart not being a dotted line #53184

Merged

8 tasks

matthewdeng pushed a commit that referenced this pull request May 20, 2025

fix MAX chart not being a dotted line (#53184)

bc87b73

Adding back some default panel configurations that were accidentally removed in a prior PR #52719 Signed-off-by: Alan Guo <aguo@anyscale.com>

hainesmichaelc added the community-backlog label May 22, 2025

Expose vLLM Metrics to serve.llm API #52719

Expose vLLM Metrics to serve.llm API #52719

Uh oh!

Conversation

eicherseiji commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dstrodtman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eicherseiji commented May 13, 2025

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eicherseiji commented May 13, 2025

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

eicherseiji commented May 1, 2025 •

edited

Loading