Description
What happened (please include outputs or screenshots):
ubuntu@xxxx-control-plane-1:~ kubectl logs -n foo xxx-controller-bc4db5d46-h594h
2022-03-10 23:43:49.065 INFO main: Check Apiserver connection
2022-03-10 23:43:51.946 WARNING urllib3.connectionpool: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4d5a8d3630>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
ubuntu@xxxx-control-plane-1:~
What you expected to happen:
ubuntu@xxxx-control-plane: kubectl logs xxx-controller-5696b686fb-c5tb5 --previous
2022-02-27 19:37:30.000 INFO main: Check Apiserver connection
2022-02-27 19:37:32.251 WARNING urllib3.connectionpool: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b6d8>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
2022-02-27 19:37:35.323 WARNING urllib3.connectionpool: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b7f0>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
2022-02-27 19:37:38.395 WARNING urllib3.connectionpool: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b898>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
2022-02-27 19:37:41.467 ERROR main: Exception when calling ApiextensionsV1Api->get_api_resources: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /apis/apiextensions.k8s.io/v1/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b978>: Failed to establish a new connection: [Errno 113] No route to host',))
How to reproduce it (as minimally and precisely as possible):
Happened intermittently, Cannot reproduce it manually. Only happens in automation testing in a multi-nodes k8s cluster
Anything else we need to know?:
#!/usr/bin/env python3
try:
from kubernetes import client, config, watch
from kubernetes.client.configuration import Configuration
from kubernetes.config import kube_config
import os
import time
import sys
except ImportError as e:
raise ImportError(str(e)
class xxx_controller:
def __init__(self):
self.api_client = client.ApiClient()
self.v1 = client.ApiextensionsV1Api(self.api_client)
self.api_instance = client.CoreV1Api(self.api_client)
self.crds = client.CustomObjectsApi(self.api_client)
def check_apiserver_conn(self):
try:
logger.info("Check Apiserver connection")
api_response = self.v1.get_api_resources()
except Exception as e:
logger.error("Exception when calling ApiextensionsV1Api->get_api_resources: %s\n" % e)
def main():
config.load_incluster_config()
cObj = xxx_controller()
cObj.check_apiserver_conn()
if __name__ == "__main__":
main()
Environment:
- Kubernetes version (
kubectl version
): 1.21 - OS (e.g., MacOS 10.13.6): Ubuntu 18.04
- Python version (
python --version
) 3.6 - Python client version (
pip list | grep kubernetes
)
root@xxx-bc4db5d46-h594h:/opt/run/server# pip list | grep kube
kubernetes 21.7.0
WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.