Thanks to visit codestin.com
Credit goes to github.com

Skip to content

api request hang and stuck #1740

Closed
Closed
@geotransformer

Description

@geotransformer

What happened (please include outputs or screenshots):
ubuntu@xxxx-control-plane-1:~ kubectl logs -n foo xxx-controller-bc4db5d46-h594h
2022-03-10 23:43:49.065 INFO main: Check Apiserver connection
2022-03-10 23:43:51.946 WARNING urllib3.connectionpool: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4d5a8d3630>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
ubuntu@xxxx-control-plane-1:~

What you expected to happen:
ubuntu@xxxx-control-plane: kubectl logs xxx-controller-5696b686fb-c5tb5 --previous
2022-02-27 19:37:30.000 INFO main: Check Apiserver connection
2022-02-27 19:37:32.251 WARNING urllib3.connectionpool: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b6d8>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
2022-02-27 19:37:35.323 WARNING urllib3.connectionpool: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b7f0>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
2022-02-27 19:37:38.395 WARNING urllib3.connectionpool: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b898>: Failed to establish a new connection: [Errno 113] No route to host',)': /apis/apiextensions.k8s.io/v1/
2022-02-27 19:37:41.467 ERROR main: Exception when calling ApiextensionsV1Api->get_api_resources: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /apis/apiextensions.k8s.io/v1/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7faacf65b978>: Failed to establish a new connection: [Errno 113] No route to host',))

How to reproduce it (as minimally and precisely as possible):
Happened intermittently, Cannot reproduce it manually. Only happens in automation testing in a multi-nodes k8s cluster

Anything else we need to know?:

#!/usr/bin/env python3

try:
    from kubernetes import client, config, watch
    from kubernetes.client.configuration import Configuration
    from kubernetes.config import kube_config
    import os
    import time
    import sys
except ImportError as e:
    raise ImportError(str(e)

class xxx_controller:
    def __init__(self):
        self.api_client = client.ApiClient()
        self.v1 = client.ApiextensionsV1Api(self.api_client)
        self.api_instance = client.CoreV1Api(self.api_client)
        self.crds = client.CustomObjectsApi(self.api_client)

    def check_apiserver_conn(self):
        try:
            logger.info("Check Apiserver connection")
            api_response = self.v1.get_api_resources()
        except Exception as e:
            logger.error("Exception when calling ApiextensionsV1Api->get_api_resources: %s\n" % e)

def main():
    config.load_incluster_config()
    cObj = xxx_controller()
    cObj.check_apiserver_conn()

if __name__ == "__main__":
    main()

Environment:

  • Kubernetes version (kubectl version): 1.21
  • OS (e.g., MacOS 10.13.6): Ubuntu 18.04
  • Python version (python --version) 3.6
  • Python client version (pip list | grep kubernetes)
    root@xxx-bc4db5d46-h594h:/opt/run/server# pip list | grep kube
    kubernetes 21.7.0
    WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available.
    You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions