Skip to content

DAOS-17712 cart: race in test_multisend_server #16599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

DAOS-17712 cart: race in test_multisend_server #16599

wants to merge 5 commits into from
+36 −22

Conversation

frostedcmos
Copy link
Contributor

When multisend servers start, rank=0 saves group config file, which lets clients know they can start sending RPCs.

Since multisend test servers dont communicate with each other there is a small race where rank=0 might save group config file before all other test servers finished registering for RPCs.

Further, previously even rank=0 was saving group config file before it itself was ready to accept RPCs.

The workaround moves RPC registration before group_config_save and adds 5 second delay on rank=0 before group info is saved.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@frostedcmos
DAOS-17712 cart: race in test_multisend_server
be3db5c 
When multisend servers start, rank=0 saves group config file,
which lets clients know they can start sending RPCs.

Since multisend test servers dont communicate with each other there
is a small race where rank=0 might save group config file before
all other test servers finished registering for RPCs.

Further, previously even rank=0 was saving group config file before
it itself was ready to accept RPCs.

The workaround moves RPC registration before group_config_save and
adds 5 second delay on rank=0 before group info is saved.

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 14, 2025 •
edited
Loading

Ticket title is 'cart/multisend_one_node.py:CartMultisendOneNodeTest.test_cart_multisend - stack trace for test_multisend_client'
Status is 'In Review'
Labels: 'ci_master_daily,pr_test,scrubbed_2.8'
https://daosio.atlassian.net/browse/DAOS-17712

@frostedcmos
DAOS-17712 cart: race in multisend server
3354605 
- crtu_start_basic_server helper function now takes optional protocol
  which is registered before progress threads are started

- multisend server modified to pass its rpc protocol.

Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>
@frostedcmos
- clang
c933b32 
Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>
@frostedcmos frostedcmos marked this pull request as ready for review July 14, 2025 20:31
@frostedcmos frostedcmos requested review from a team as code owners July 14, 2025 20:31
daltonbohning
daltonbohning reviewed Jul 14, 2025
View reviewed changes
Copy link
Contributor

@daltonbohning daltonbohning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ftest LGTM (just C code actually)

@frostedcmos
retrigger
07feba5 
Signed-off-by: Alexander A Oganezov <alexander.oganezov@hpe.com>
@frostedcmos frostedcmos requested a review from jolivier23 July 16, 2025 18:04
@daosbuild3
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16599/6/execution/node/340/log

@daosbuild3
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16599/6/execution/node/337/log

@daosbuild3
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16599/6/execution/node/353/log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daltonbohning daltonbohning daltonbohning left review comments

@jolivier23 jolivier23 jolivier23 approved these changes

@jgmoore-or jgmoore-or jgmoore-or approved these changes

@soumagne soumagne Awaiting requested review from soumagne

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@frostedcmos @daosbuild3 @jolivier23 @daltonbohning @jgmoore-or
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy