Skip to content

APIServer APF estimates cost for LIST not work #130139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
AllenXu93 opened this issue Feb 13, 2025 · 4 comments
Open

APIServer APF estimates cost for LIST not work #130139

AllenXu93 opened this issue Feb 13, 2025 · 4 comments
Assignees
@MikeSpreitzer
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@AllenXu93
Copy link

What happened?

According to the documentation https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#seats-occupied-by-a-request , the APIServer's APF (API Priority and Fairness) calculates seats for each request as a reference to measure the consumption of each API. Specifically, List requests determine the number of seats based on the number of objects returned.

However, in actual usage, I have observed that all List requests (regardless of the number of objects) are assigned only 1 seat by APF.
For example, I have 1w+ pods in cluster, seat expected to be 10.

apiserver_flowcontrol_work_estimated_seats_bucket{flow_schema="list-pods",priority_level="list-pods",le="1"} 84
apiserver_flowcontrol_work_estimated_seats_bucket{flow_schema="list-pods",priority_level="list-pods",le="2"} 84
apiserver_flowcontrol_work_estimated_seats_bucket{flow_schema="list-pods",priority_level="list-pods",le="4"} 84
apiserver_flowcontrol_work_estimated_seats_bucket{flow_schema="list-pods",priority_level="list-pods",le="10"} 84
apiserver_flowcontrol_work_estimated_seats_bucket{flow_schema="list-pods",priority_level="list-pods",le="+Inf"} 84

I found that the seat calculation for List requests happens at

kubernetes/staging/src/k8s.io/apiserver/pkg/util/flowcontrol/request/list_work_estimator.go

Line 50 in 2642d82

minSeats := e.config.MinimumSeats

and every request hit the error:

kubernetes/staging/src/k8s.io/apiserver/pkg/util/flowcontrol/request/list_work_estimator.go

Line 100 in 2642d82

case err == ObjectCountNotFoundErr:

.

So I add some debug log in objectCountTracker set, found that objectCountTracker is nil, so it will never update the object count, which will cause ObjectCountNotFoundErr in APF list estimator.

kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go

Lines 1673 to 1675 in 2642d82

if objectCountTracker != nil {
objectCountTracker.Set(resourceName, count)
}

What did you expect to happen?

In APF (API Priority and Fairness), the seats for a List request should increase with the number of objects returned, rather than always being 1.

How can we reproduce it (as minimally and precisely as possible)?

There is my apf config.

apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: list-pods
spec:
  limited:
    nominalConcurrencyShares: 1
    borrowingLimitPercent: 0
    limitResponse:
      queuing:
        handSize: 5
        queueLengthLimit: 0
        queues: 5
      type: Queue
  type: Limited
---
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: FlowSchema
metadata:
  name: list-pods
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 100
  priorityLevelConfiguration:
    name: list-pods
  rules:
  - resourceRules:
    - apiGroups:
      - '*'
      clusterScope: true
      namespaces:
      resources:
      - pods
      verbs:
      - list
    subjects:
    - group:
        name: system:authenticated
      kind: Group

Anything else we need to know?

No response

Kubernetes version

Server Version: v1.32.2

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)
@AllenXu93 AllenXu93 added the kind/bug Categorizes issue or PR as related to a bug. label Feb 13, 2025
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 13, 2025
@AllenXu93
Copy link
Author

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 13, 2025
@yue9944882
Copy link
Member 

nominalConcurrencyShares: 1

low shares can cap the calculated seat i think

@MikeSpreitzer
Copy link
Member

/assign

@jpbetz
Copy link
Contributor

jpbetz commented May 15, 2025

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MikeSpreitzer MikeSpreitzer

Labels
kind/bug Categorizes issue or PR as related to a bug. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jpbetz @yue9944882 @MikeSpreitzer @k8s-ci-robot @AllenXu93
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy