Skip to content

Failure cluster [e2e90f0c...] Connectivity Pod Lifecycle should be able to have zero downtime on a Blue Green deployment using Services and Readiness Gates #131707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
BenTheElder opened this issue May 9, 2025 · 6 comments · Fixed by #131780
Closed

Failure cluster [e2e90f0c...] Connectivity Pod Lifecycle should be able to have zero downtime on a Blue Green deployment using Services and Readiness Gates #131707

BenTheElder opened this issue May 9, 2025 · 6 comments · Fixed by #131780
Assignees
@aojea
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/network Categorizes an issue or PR as relevant to SIG Network.

Comments

@BenTheElder
Copy link
Member

Failure cluster e2e90f0c9cc473dc1dbf

Error text:
[FAILED] can not connect to pod green-pod on address 10.97.105.79:80 : expecting hostname green-pod got blue-pod
In [It] at: k8s.io/kubernetes/test/e2e/network/pod_lifecycle.go:327 @ 05/03/25 07:52:21.779

Recent failures:

5/9/2025, 9:35:32 AM e2e-kops-aws-k8s-latest
5/9/2025, 9:10:25 AM pr:pull-kubernetes-e2e-ec2
5/9/2025, 3:58:53 AM pr:pull-kubernetes-e2e-gce
5/9/2025, 3:20:17 AM ci-kubernetes-e2e-gci-gce-containerd
5/9/2025, 3:06:45 AM pr:pull-kubernetes-e2e-kind-beta-features

/kind failing-test

/sig network

spotted in https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/131702/pull-kubernetes-e2e-kind/1920954886661345280
@k8s-ci-robot k8s-ci-robot added kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. sig/network Categorizes an issue or PR as relevant to SIG Network. labels May 9, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label May 9, 2025
@BenTheElder BenTheElder mentioned this issue May 9, 2025
Merged
@pacoxu
Copy link
Member

pacoxu commented May 12, 2025

flake a lot in https://testgrid.k8s.io/sig-release-master-blocking#kind-ipv6-master-parallel master blocking board.
/cc @kubernetes/release-team-release-signal

@pacoxu
Copy link
Member

pacoxu commented May 12, 2025 •
edited
Loading

The test was added in #131538 to flake since 05/01)

cc @aojea

This was referenced May 12, 2025
Merged
Merged
This was referenced May 15, 2025
Merged
Merged
@aojea
Copy link
Member

aojea commented May 15, 2025

/assign

Still flakes after https://github.com/kubernetes/kubernetes/pull/131762/commits

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1922854539090726912

@aojea
Copy link
Member

aojea commented May 15, 2025

ok, it fails now because the kube-proxy in the GCE jobs has a sync interval of 10 seconds to progam the dataplane

I0515 03:48:15.004462       7 proxier.go:821] "SyncProxyRules complete" ipFamily="IPv4" elapsed="243.96395ms"
I0515 03:48:25.004971       7 proxier.go:827] "Syncing iptables rules" ipFamily="IPv4" fullSync=false
I0515 03:48:25.005075       7 topology.go:183] "Ignoring same-node topology hints for service since no hints were provided for node" service="10.0.212.104:80/TCP" node="bootstrap-e2e-minion-group-vsl1"
I0515 03:48:25.005092       7 topology.go:195] "Ignoring same-zone topology hints for service since no hints were provided for zone" service="10.0.212.104:80/TCP" zone="us-central1-b"
I0515 03:48:25.005171       7 topology.go:183] "Ignoring same-node topology hints for service since no hints were provided for node" service="10.0.63.125:90/UDP" node="bootstrap-e2e-minion-group-vsl1"
I0515 03:48:25.005199       7 topology.go:195] "Ignoring same-zone topology hints for service since no hints were provided for zone" service="10.0.63.125:90/UDP" zone="us-central1-b"
I0515 03:48:25.005242       7 topology.go:183] "Ignoring same-node topology hints for service since no hints were provided for node" service="10.0.212.104:90/UDP" node="bootstrap-e2e-minion-group-vsl1"
I0515 03:48:25.005250       7 topology.go:195] "Ignoring same-zone topology hints for service since no hints were provided for zone" service="10.0.212.104:90/UDP" zone="us-central1-b"
I0515 03:48:25.005261       7 topology.go:183] "Ignoring same-node topology hints for service since no hints were provided for node" service="10.0.63.125:80/TCP" node="bootstrap-e2e-minion-group-vsl1"
I0515 03:48:25.005269       7 topology.go:195] "Ignoring same-zone topology hints for service since no hints were provided for zone" service="10.0.63.125:80/TCP" zone="us-central1-b"
I0515 03:48:25.010060       7 proxier.go:1548] "Reloading service iptables data" ipFamily="IPv4" numServices=16 numEndpoints=26 numFilterChains=6 numFilterRules=12 numNATChains=27 numNATRules=52
I0515 03:48:25.051355       7 proxier.go:821] "SyncProxyRules complete" ipFamily="IPv4" elapsed="46.572488ms"

It means, that is not reliable to sync on the number of responses to validate the dataplane is programmed

cc: @aroradaman

@aojea aojea mentioned this issue May 15, 2025
Merged
@pacoxu
Copy link
Member

pacoxu commented May 16, 2025

No Flake in https://testgrid.k8s.io/sig-release-master-blocking#kind-ipv6-master-parallel after #131780 was merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aojea aojea

Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/network Categorizes an issue or PR as relevant to SIG Network.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix e2e network test flake because of network programming latency aojea/kubernetes
4 participants
@BenTheElder @pacoxu @aojea @k8s-ci-robot
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy