Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UCT/GGA: Support is_connected API #10403

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

shasson5
Copy link
Contributor

@shasson5 shasson5 commented Jan 6, 2025

What?

Support is_connected API

Why?

Allow GGA to work properly during EP reconfiguration

Comment on lines 632 to 641
uint32_t addr_qp = 0;
uct_gga_mlx5_ep_address_t *ep_addr;

if (params->field_mask & UCT_EP_IS_CONNECTED_FIELD_EP_ADDR) {
ep_addr = (uct_gga_mlx5_ep_address_t*)params->ep_addr;
addr_qp = uct_ib_unpack_uint24(ep_addr->qp_num);
}

return uct_rc_mlx5_ep_is_connected_to_addr(tl_ep, params, addr_qp) &&
uct_base_ep_is_connected(tl_ep, params);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's align with other transports which use uct_base_ep_is_connected:

Suggested change
uint32_t addr_qp = 0;
uct_gga_mlx5_ep_address_t *ep_addr;
if (params->field_mask & UCT_EP_IS_CONNECTED_FIELD_EP_ADDR) {
ep_addr = (uct_gga_mlx5_ep_address_t*)params->ep_addr;
addr_qp = uct_ib_unpack_uint24(ep_addr->qp_num);
}
return uct_rc_mlx5_ep_is_connected_to_addr(tl_ep, params, addr_qp) &&
uct_base_ep_is_connected(tl_ep, params);
uct_gga_mlx5_ep_address_t *ep_addr;
uint32_t addr_qp;
if (!(params->field_mask & UCT_EP_IS_CONNECTED_FIELD_EP_ADDR)) {
return 0;
}
if (!uct_base_ep_is_connected(tl_ep, params)) {
return 0;
}
ep_addr = (uct_gga_mlx5_ep_address_t*)params->ep_addr;
addr_qp = uct_ib_unpack_uint24(ep_addr->qp_num);
return uct_rc_mlx5_ep_is_connected_to_addr(tl_ep, params, addr_qp);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ep_addr is optional, so cannot use:

    if (!(params->field_mask & UCT_EP_IS_CONNECTED_FIELD_EP_ADDR)) {
        return 0;
    }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is not addr_qp == 0 means false for is_connected state?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it means we skip the comparison between addr_qp and qp_num (and only compare device address)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment on lines 635 to 638
if (params->field_mask & UCT_EP_IS_CONNECTED_FIELD_EP_ADDR) {
ep_addr = (uct_gga_mlx5_ep_address_t*)params->ep_addr;
addr_qp = uct_ib_unpack_uint24(ep_addr->qp_num);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is some code duplication with uct_rc_mlx5_base_ep_is_connected
to eliminate it we can create new parent struct for uct_gga_mlx5_ep_address_t and uct_rc_mlx5_ep_address_t - that will contain only qp_num. This struct can be called uct_rc_mlx5_base_ep_address_t.
uct_rc_mlx5_base_ep_is_connected will cast params->ep_addr to uct_rc_mlx5_base_ep_address_t*, if the flag UCT_EP_IS_CONNECTED_FIELD_EP_ADDR is specified.

@shasson5 shasson5 requested a review from yosefe January 7, 2025 10:12
Comment on lines +52 to +60
uct_rc_mlx5_base_ep_address_t super;
uint8_t flags;
uct_ib_uint24_t flush_rkey;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like braking of backward compatibility...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there currently a scenario where gga transport is selected by default by UCP?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it can be selected for rma_bw lane

const uct_ep_is_connected_params_t *params)
{
return uct_rc_mlx5_base_ep_is_connected(tl_ep, params) &&
uct_base_ep_is_connected(tl_ep, params);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need uct_base_ep_is_connected? seems rc_mlx5 does not need it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to compare iface address (only required for GGA)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both will compare IB device address which is redundant
can we reuse common code with uct_gga_mlx5_iface_is_reachable_v2 and compare iface address directly?
BTW @evgeny-leksikov @amaslenn the gga transport does not currently implement returning info string from uct_gga_mlx5_iface_is_reachable_v2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW @evgeny-leksikov @amaslenn the gga transport does not currently implement returning info string from uct_gga_mlx5_iface_is_reachable_v2

thanks, I'll add

return uct_rc_mlx5_base_ep_is_connected(tl_ep, params) &&
uct_base_ep_is_connected(tl_ep, params);
uct_gga_mlx5_iface_is_same_device(tl_ep->iface, iface_addr);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: swap order (call uct_gga_mlx5_iface_is_same_device first)

const uct_iface_is_reachable_params_t *params)
uct_gga_mlx5_iface_is_same_device(const uct_iface_h tl_iface,
const uct_iface_addr_t *iface_addr,
const uct_iface_is_reachable_params_t *params)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_iface_is_reachable_params_t looks redundant here, maybe just const char* str, size_t length and use ucs_snprintf_safe?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we drop uct_iface_is_reachable_params_t then there will be code duplication:

    char *info_str_buf      = UCS_PARAM_VALUE(UCT_IFACE_IS_REACHABLE_FIELD,
                                              params, info_string, INFO_STRING,
                                              NULL);
    size_t info_str_buf_len = UCS_PARAM_VALUE(UCT_IFACE_IS_REACHABLE_FIELD,
                                              params, info_string_length,
                                              INFO_STRING_LENGTH, 0);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can to move most of logic of this function up, to uct_gga_mlx5_iface_is_reachable_v2().
So uct_gga_mlx5_iface_is_same_device() will get only tl_iface and iface_addr, and return 0/1. It will contain only the logic in lines 627-629. Then no need to define dummy params inside uct_gga_mlx5_ep_is_connected()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there will still be code duplication:

    if (gga_addr == NULL) {
        uct_iface_fill_info_str_buf(params, "iface address is empty");
        return 0;
    }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_iface_fill_info_str_buf(params, "iface address is empty"); should be only in uct_gga_mlx5_iface_is_reachable_v2; uct_gga_mlx5_ep_is_connected can just check if (gga_addr == NULL)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (gga_addr == NULL) will be the duplication

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it's small enough (1 line) duplication and it can avoid extra initialization of dummy struct so it's worth it

Comment on lines 627 to 629
if (gga_addr->be_sys_image_guid !=
device->dev_attr.orig_attr.sys_image_guid) {
local_guid = device->dev_attr.orig_attr.sys_image_guid;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (gga_addr->be_sys_image_guid !=
device->dev_attr.orig_attr.sys_image_guid) {
local_guid = device->dev_attr.orig_attr.sys_image_guid;
local_guid = device->dev_attr.orig_attr.sys_image_guid;
if (gga_addr->be_sys_image_guid != local_guid) {

const uct_iface_is_reachable_params_t *params)
uct_gga_mlx5_iface_is_same_device(const uct_iface_h tl_iface,
const uct_iface_addr_t *iface_addr,
const uct_iface_is_reachable_params_t *params)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can to move most of logic of this function up, to uct_gga_mlx5_iface_is_reachable_v2().
So uct_gga_mlx5_iface_is_same_device() will get only tl_iface and iface_addr, and return 0/1. It will contain only the logic in lines 627-629. Then no need to define dummy params inside uct_gga_mlx5_ep_is_connected()

@yosefe
Copy link
Contributor

yosefe commented Jan 12, 2025

@shasson5 pls also note the merge conflicts

@shasson5 shasson5 requested a review from yosefe January 12, 2025 10:51
@shasson5
Copy link
Contributor Author

@yosefe please review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy