Skip to content

storage: log more info when requesting eviction #26298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 2, 2025
Merged

storage: log more info when requesting eviction #26298

merged 1 commit into from
Jun 2, 2025

Conversation

dotnwat
Copy link
Member

@dotnwat dotnwat commented May 30, 2025

Callers that request log eviction log the request, but there are still several conditions that must pass before the request is actually made to the eviction stm. This PR logs additional information to help debug if the request is successfully made.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.1.x
  • v24.3.x
  • v24.2.x

Release Notes

  • none

@dotnwat dotnwat requested review from andrwng, WillemKauf, nvartolomei, bharathv, Lazin and oleiman and removed request for andrwng and WillemKauf May 30, 2025 19:12
rockwotj
rockwotj previously approved these changes May 30, 2025
View reviewed changes
Copy link
Contributor

@rockwotj rockwotj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a question if this should really be info.

src/v/storage/disk_log_impl.cc Outdated
vlog(
gclog.debug,
"[{}] no segments to evict up to {} offset; skipping eviction",
gclog.info,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for this upgrade to info other than this is top of mind?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it's purely because we see systems out there where we have a mystery and one explanation would be explained by the eviction request not happening.

the caller of this method already logs at info level. lemme see if i can just merge all these together so there is no net-new log lines.

Copy link
Member Author

@dotnwat dotnwat May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah you're right. these messages can both be at debug level. i switched them back to debug level. in all but some extreme cases they are retriggered frequently so we don't need to worry about catching all instances of these logs.

@dotnwat dotnwat force-pushed the request-eviction-logging branch from 3b3299c to 5b67530 Compare May 30, 2025 19:17
@dotnwat
storage: log more info when requesting eviction
1561401 
Callers that request log eviction log the request, but there are still
several conditions that must pass before the request is actually made to
the eviction stm. This PR logs additional information to help debug if
the request is successfully made.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
@dotnwat dotnwat force-pushed the request-eviction-logging branch from 5b67530 to 1561401 Compare May 30, 2025 20:02
@dotnwat dotnwat requested a review from rockwotj May 30, 2025 20:03
@vbotbuildovich
Copy link
Collaborator

CI test results

test results on build#66644
test_class test_method test_arguments test_kind job_url test_status passed reason
TopicDeleteCloudStorageTest drop_lifecycle_marker_test {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66644#01972302-a5ae-482c-8469-efc77b5bdb90 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS

@dotnwat dotnwat merged commit df8111f into redpanda-data:dev Jun 2, 2025
17 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v25.1.x

@vbotbuildovich
Copy link
Collaborator

/backport v24.3.x

@vbotbuildovich
Copy link
Collaborator

/backport v24.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v24.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-26298-v24.3.x-980 remotes/upstream/v24.3.x
git cherry-pick -x 1561401605

Workflow run logs.

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v24.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-26298-v24.2.x-950 remotes/upstream/v24.2.x
git cherry-pick -x 1561401605

Workflow run logs.

This was referenced Jun 2, 2025
Open
Open
Merged
andrwng
andrwng reviewed Jun 2, 2025
View reviewed changes
Copy link
Contributor

@andrwng andrwng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a couple naming suggestions for clarity

src/v/storage/disk_log_impl.cc
auto have_segments_to_evict
= (_segs.size() > 1)
&& _segs.front()->offsets().get_committed_offset() <= max_offset;
const auto min_offset = [this]() -> std::optional<model::offset> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: IMO min_offset is kind of confusing, given it isn't clear it's the minimum of what set this is referring to. Same with max_offset for that matter

Maybe call it lowest_removable_offset, and call max_offset target_offset_to_remove or something? If you do make this change, would be helpful to also update the log message

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrwng andrwng andrwng left review comments

@WillemKauf WillemKauf WillemKauf approved these changes

@rockwotj rockwotj rockwotj approved these changes

@nvartolomei nvartolomei Awaiting requested review from nvartolomei

@bharathv bharathv Awaiting requested review from bharathv

@Lazin Lazin Awaiting requested review from Lazin

@oleiman oleiman Awaiting requested review from oleiman

Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dotnwat @vbotbuildovich @WillemKauf @rockwotj @andrwng
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy