Thanks to visit codestin.com
Credit goes to github.com

Skip to content

certificate-authority-data from kubeconfig fails for long-running process due to tmpfs cleanup #1782

Closed
@PaulFurtado

Description

@PaulFurtado

What happened (please include outputs or screenshots):
Long-running applications using a kubeconfig with certificate-authority-data encounter errors like:

Max retries exceeded with url: /api/v1/pods?fieldSelector=spec.nodeName%3Dip-172-18-66-203.ec2.internal (Caused by SSLError(FileNotFoundError(2, 'No such file or directory'),))

What you expected to happen:
Client should not expect tempfiles to live indefinitely. It is extremely common for servers to reap tempfiles.

Anything else we need to know?:
If you create a client you can easily see that it is using a temp file that cannot go away for the duration of the client:

>>> import kubernetes.config
>>> kubernetes.config.load_kube_config()
>>> api_client = kubernetes.client.ApiClient()
>>> api_client.rest_client.pool_manager.connection_pool_kw["ca_certs"]
'/tmp/tmpqkht2v2g'

You can reproduce the issue by deleting that temp file and attempting to make a request.

Code is here:

def _create_temp_file_with_content(content, temp_file_path=None):
if len(_temp_files) == 0:
atexit.register(_cleanup_temp_files)
# Because we may change context several times, try to remember files we
# created and reuse them at a small memory cost.
content_key = str(content)
if content_key in _temp_files:
return _temp_files[content_key]
if temp_file_path and not os.path.isdir(temp_file_path):
os.makedirs(name=temp_file_path)
fd, name = tempfile.mkstemp(dir=temp_file_path)
os.close(fd)
_temp_files[content_key] = name
with open(name, 'wb') as fd:
fd.write(content.encode() if isinstance(content, str) else content)
return name

In order for this to work reliably for long-running processes on standard linux systems, the temp file really needs to be created for each request rather than a single time at startup.

That said, on linux systems, a potential hack would be to use /proc/self/fd/<fileno> instead of the temfile path since that would share the lifecycle of the process.

Environment:

  • Python version (python --version): 3.6
  • Python client version (pip list | grep kubernetes): 21.7.0

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions