/sig autoscaling

cc sig autoscaling maintainers: @gjtempleton @MaciekPytel

is this a bug or a known issue?

refer to #51650? (I will take a look tomorrow)

The logic were added in #60886.

Due to technical constraints, the HorizontalPodAutoscaler controller cannot exactly determine the first time a pod becomes ready when determining whether to set aside certain CPU metrics. Instead, it considers a Pod "not yet ready" if it's unready and transitioned to ready within a short, configurable window of time since it started. This value is configured with the --horizontal-pod-autoscaler-initial-readiness-delay flag, and its default is 30 seconds. Once a pod has become ready, it considers any transition to ready to be the first if it occurred within a longer, configurable time since it started. This value is configured with the --horizontal-pod-autoscaler-cpu-initialization-period flag, and its default is 5 minutes.

Meanwhile, https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details(quoted content) and #67252, there is --horizontal-pod-autoscaler-cpu-initialization-period. This may help in your case as well.

Hi @pacoxu, thanks for your explanation. However, we are using external metrics to drive our scaling so I think --horizontal-pod-autoscaler-cpu-initialization-period would not be applicable to our use case.

Hi @pacoxu Could you please help go on with this issue?

I'd argue that this is a known issue/design choice rather than a bug, as you pointed out @pacoxu the behaviour for object and external metrics was set to be ~the same as with resource metrics by #60886 (#33593 was the original source of a lot of these choices).

Definitely a valid use case that we don't currently support nicely.

/triage accepted

/remove-triage accepted

@gjtempleton Thanks for your explaining. Does your team have any plans to solve this problem?

@zheyli you can answer that question self-service: the backlog of issues is public, so you can search through them yourself.
Asking the maintainers to do that work for you is possible, but they may well decline.

Given #130130 (comment)
/remove-kind bug
/kind feature
/retitle Allow HPA to scale out when no matched Pods are ready

@gjtempleton
Just an idea: what if we use current replicas as fallback when there are no ready pods?

So instead of always using ready_pods (which becomes 0 in our problem case), we could do:

• Normal case: use ready_pods (keep current behavior)
• When no ready pods: use current_replicas

This way we don't get stuck at 0 when pods are unhealthy, and it should be pretty simple to implement.

@gjtempleton Just an idea: what if we use current replicas as fallback when there are no ready pods?

So instead of always using ready_pods (which becomes 0 in our problem case), we could do:

• Normal case: use ready_pods (keep current behavior)
• When no ready pods: use current_replicas

This way we don't get stuck at 0 when pods are unhealthy, and it should be pretty simple to implement.

Having a quick look at the code, this seems sane to me. I'm just not 100% sure if there are edge cases we need to be aware of, or if this will change the behaviour in a way that's not expected for the user.

Ill try to write something and see where it goes.
/assign

Hi @omerap12, thank you for taking this on. Using current replicas as a fallback during an extreme case of all pods becoming unready can prevent the emergency situation of being stuck at zero pods, but I think users utilizing external metrics would benefit from having the option to use replica count instead of ready pods from the start. In the scenario where pods start becoming unready due to an increase in external metric utilization, since HPA does not account for unready pods, we do not see the scaling behavior we would expect. HPA does slowly scale up, but not nearly as aggressively as it should according to the scale up policy. Instead utilization continues to cause pods to become unready until there is potentially service degradation or interruption.

Hi @omerap12, thank you for taking this on. Using current replicas as a fallback during an extreme case of all pods becoming unready can prevent the emergency situation of being stuck at zero pods, but I think users utilizing external metrics would benefit from having the option to use replica count instead of ready pods from the start. In the scenario where pods start becoming unready due to an increase in external metric utilization, since HPA does not account for unready pods, we do not see the scaling behavior we would expect. HPA does slowly scale up, but not nearly as aggressively as it should according to the scale up policy. Instead utilization continues to cause pods to become unready until there is potentially service degradation or interruption.

No problem at all :)
EDIT: After reading #60886 again, I think letting the HPA count unready pods when deciding how many replicas to run could be a problem. If all pods are unready, how can the HPA know if it's because of high traffic or something else? The HPA doesn’t know why the pods are unready. If we scale based on unready pods, we might just create more "broken" pods and expand the "blast radius." For example, if the pods can’t connect to the database, adding more won’t fix the issue - it might even make it worse by putting more load on the database.

In this case, I think it’s better to trigger an alert for the team, rather than expecting the HPA to handle it on its own.
So, I agree with the idea in #60886 to ignore unready pods when scaling.

Hi @omerap12, thank you for taking this on. Using current replicas as a fallback during an extreme case of all pods becoming unready can prevent the emergency situation of being stuck at zero pods, but I think users utilizing external metrics would benefit from having the option to use replica count instead of ready pods from the start. In the scenario where pods start becoming unready due to an increase in external metric utilization, since HPA does not account for unready pods, we do not see the scaling behavior we would expect. HPA does slowly scale up, but not nearly as aggressively as it should according to the scale up policy. Instead utilization continues to cause pods to become unready until there is potentially service degradation or interruption.

No problem at all :) EDIT: After reading #60886 again, I think letting the HPA count unready pods when deciding how many replicas to run could be a problem. If all pods are unready, how can the HPA know if it's because of high traffic or something else? The HPA doesn’t know why the pods are unready. If we scale based on unready pods, we might just create more "broken" pods and expand the "blast radius." For example, if the pods can’t connect to the database, adding more won’t fix the issue - it might even make it worse by putting more load on the database.

In this case, I think it’s better to trigger an alert for the team, rather than expecting the HPA to handle it on its own. So, I agree with the idea in #60886 to ignore unready pods when scaling.

I agree with everything you're saying. This seems to be a major downside of using external metrics for HPA. The most common scenario we have seen for pods becoming unready is that our Puma workers are exhausted on several pods. The average of our external metric, Puma worker utilization, reaches the the scaling target but since there are unready pods HPA does not scale as quickly as it should. We have also had some Puma worker latency due to slow DB queries which has increased load on the DB, but in the end the interim solution was to scale up the DB until optimization could be performed, and HPA saved us from an actual outage. In the event pods can't connect to the DB, I would rather set alerting for that type of event than not have external metric based HPA not work properly.

Hi @omerap12, thank you for taking this on. Using current replicas as a fallback during an extreme case of all pods becoming unready can prevent the emergency situation of being stuck at zero pods, but I think users utilizing external metrics would benefit from having the option to use replica count instead of ready pods from the start. In the scenario where pods start becoming unready due to an increase in external metric utilization, since HPA does not account for unready pods, we do not see the scaling behavior we would expect. HPA does slowly scale up, but not nearly as aggressively as it should according to the scale up policy. Instead utilization continues to cause pods to become unready until there is potentially service degradation or interruption.

No problem at all :) EDIT: After reading #60886 again, I think letting the HPA count unready pods when deciding how many replicas to run could be a problem. If all pods are unready, how can the HPA know if it's because of high traffic or something else? The HPA doesn’t know why the pods are unready. If we scale based on unready pods, we might just create more "broken" pods and expand the "blast radius." For example, if the pods can’t connect to the database, adding more won’t fix the issue - it might even make it worse by putting more load on the database.

In this case, I think it’s better to trigger an alert for the team, rather than expecting the HPA to handle it on its own. So, I agree with the idea in #60886 to ignore unready pods when scaling.

I agree with everything you're saying. This seems to be a major downside of using external metrics for HPA. The most common scenario we have seen for pods becoming unready is that our Puma workers are exhausted on several pods. The average of our external metric, Puma worker utilization, reaches the the scaling target but since there are unready pods HPA does not scale as quickly as it should. We have also had some Puma worker latency due to slow DB queries which has increased load on the DB, but in the end the interim solution was to scale up the DB until optimization could be performed, and HPA saved us from an actual outage. In the event pods can't connect to the DB, I would rather set alerting for that type of event than not have external metric based HPA not work properly.

Yeah, exactly — since the HPA doesn’t really know why pods are unready, scaling based on them could cause more problems, especially if the issue isn’t because of high load. I think it’s safer not to scale unready pods.