RayCluster Headless Worker Service Should PublishNotReadyAddresses #2375
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
reviewed
Sep 12, 2024
| @@ -307,9 +307,10 @@ func BuildHeadlessServiceForRayCluster(rayCluster rayv1.RayCluster) (*corev1.Ser | |||
| Labels: labels, | |||
| }, | |||
| Spec: corev1.ServiceSpec{ | |||
There was a problem hiding this comment.
For another PR, add unit tests for BuildHeadlessServiceForRayCluster
There was a problem hiding this comment.
I went ahead and added a test case in this PR (c92042b) since it's a fairly small change.
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
reviewed
Sep 12, 2024
|
|
||
| actualSelector := svc.Spec.Selector[utils.RayClusterLabelKey] | ||
| expectedSelector := instanceForSvc.Name | ||
| if !reflect.DeepEqual(expectedSelector, actualSelector) { |
There was a problem hiding this comment.
Might be easier to just construct the expected Service object and do a reflect.DeepEqual check against that and the returned object from BuildHeadlessServiceForRayCluster.
Maybe for a separate PR because this is just following the pattern of the other tests.
|
@kevin85421 PTAL |
kevin85421
approved these changes
Sep 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR adds
PublishNotReadyAddresses: trueto the headless service created by KubeRay when a RayCluster requests multi-host TPU nodes. When creating a RayService for multi-host TPU inference, TPU initialization currently times out because 1 or more of the workers might be unreachable until a proxy actor is running on that node. This PR unblocks multi-host inference on TPUs with a RayService, since TPU initialization requires worker-to-worker communication even if all proxy actors haven't started yet.Related issue number
Checks
PublishNotReadyAddresses: trueand verified RayService deployment succeeded