Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Stream Kubernetes event logs to Coder build/startup logs #7576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bpmct opened this issue May 16, 2023 · 9 comments
Closed

Stream Kubernetes event logs to Coder build/startup logs #7576

bpmct opened this issue May 16, 2023 · 9 comments
Assignees
@kylecarbs

Comments

@bpmct
Copy link
Member

bpmct commented May 16, 2023 •
edited
Loading

Sometimes, a Kubernetes pod scheduling event may take quite an unexpected time or even fail. Here are some cases when it would happen:

  • A node is full, triggering an autoscaling event. A workspace start may take 2-3 minutes until the new node is ready.
  • A user specifies an invalid image via a parameter, or the image no longer exists in the registry. This will cause the workspace to keep "building" until Terraform times out/fails.
  • A volume takes a very long time to mount/unmount. These show up in pod events but do not

I have been on several support calls for each of these cases.

However, these errors are not visible to the user or admin from the Coder dashboard/UI. The best way to see this is if a Kubernetes admin runs "kubectl describe pod ". The error will always be in the event logs there.

Ideally, these logs are streamed to the Coder workspace build. Since Coder aims to be platform agnostic, this will likely not be a core feature, but some extra sidecar or DaemonSet on the cluster that streams the logs to the appropriate workspace using our generic logs endpoint. The user could optionally deploy this, or perhaps it is enabled/disabled via the helm chart.

The tricky part is that many of these errors occur during the "provisioning" phase, not the "startup" phase when we typically show startup logs.
@kylecarbs kylecarbs self-assigned this May 17, 2023
@kylecarbs
Copy link
Member

@dcarrion87 this might be interesting to you!

I'm working on a small Kubernetes chart that automatically sends Kubernetes Pod events as startup logs. It should be a simple deployment with no configuration needed.

@kylecarbs
Copy link
Member

@bpmct wanna keep this open until we have some integrated docs?

@kylecarbs
Copy link
Member

Actually, we probably should. For those following, you can now use: https://github.com/coder/coder-logstream-kube

@bpmct
Copy link
Member Author

bpmct commented Jun 14, 2023

Yeah, we'll still need to do the following

  • Change our Kubernetes example to support logstream-kube (we need to use a deployment instead of pod)
  • Add this to our docs under "kubernetes"

Maybe:

  • Add support to helm chart to easily deploy this (perhaps we can do it by default)

@bpmct bpmct added this to the ❓Sprint 2 milestone Jun 14, 2023
@matifali
Copy link
Member

As per my experience, https://github.com/coder/coder-logstream-kube works with pod also. Haven't tried with deployment.

@bpmct
Copy link
Member Author

bpmct commented Jun 15, 2023

works with pod also

Will it show logs if an invalid image is referenced? I think we recommend using deployment because it does not wait for the image to pull to show the logs, covering a wider variety of error cases. In the pod case, I believe it will keep "building" for 5 minutes until it times out and show no extra logs.

@bpmct bpmct mentioned this issue Jun 21, 2023
Closed
@bpmct
Copy link
Member Author

bpmct commented Jun 27, 2023

This is almost complete, thanks to the coder-logstream-kube! We'll just need to document this and change our example templates.

@dcarrion87
Copy link
Contributor

dcarrion87 commented Jun 27, 2023 •
edited
Loading

Thanks team. I've added this in to try in an upcoming sprint. Should reduce some noise as to "why" things happen.

@ammario ammario removed this from the ❓Sprint 2 milestone Jun 29, 2023
@bpmct bpmct mentioned this issue Jul 17, 2023
Closed
9 tasks
@bpmct bpmct mentioned this issue Sep 3, 2023
Closed
@kylecarbs
Copy link
Member

kylecarbs commented Dec 15, 2023 •
edited by matifali
Loading

This has been done with logstream kube.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kylecarbs kylecarbs

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dcarrion87 @kylecarbs @ammario @matifali @bpmct