Thanks to visit codestin.com
Credit goes to github.com

Skip to content

cmd/contour: add a Kubernetes debug logging flag #2838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jpeach merged 1 commit into from
Sep 8, 2020
Merged

cmd/contour: add a Kubernetes debug logging flag #2838

jpeach merged 1 commit into from
Sep 8, 2020

Conversation

@jpeach
Copy link
Contributor

@jpeach jpeach commented Aug 27, 2020

Add a --kubernetes-debug flag that enables verbose client-go logging
in the klog package. The logs are piped through logrus so that the
output format is consistent.

Signed-off-by: James Peach [email protected]

@codecov
Copy link

codecov bot commented Aug 27, 2020 •
edited
Loading

Codecov Report

Merging #2838 into main will decrease coverage by 0.77%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2838      +/-   ##
==========================================
- Coverage   76.57%   75.80%   -0.78%     
==========================================
  Files          79       80       +1     
  Lines        5866     5923      +57     
==========================================
- Hits         4492     4490       -2     
- Misses       1283     1342      +59     
  Partials       91       91
Impacted Files Coverage Δ
cmd/contour/contour.go 1.35% <0.00%> (-2.88%) ⬇️
cmd/contour/serve.go 2.36% <0.00%> (-0.01%) ⬇️
cmd/contour/servecontext.go 93.49% <ø> (ø)
internal/k8s/log.go 0.00% <0.00%> (ø)

@stevesloka stevesloka changed the title cmd/contour: add a Kubernete debug logging flag cmd/contour: add a Kubernetes debug logging flag Aug 27, 2020
stevesloka
stevesloka suggested changes Aug 27, 2020
View reviewed changes
Copy link
Member

@stevesloka stevesloka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got some errors testing through this:

ERRO[0000] Error while reading from Writer: bufio.Scanner: token too long  context=kubernetes

@jpeach
Copy link
Contributor Author

jpeach commented Aug 28, 2020

I got some errors testing through this:

ERRO[0000] Error while reading from Writer: bufio.Scanner: token too long  context=kubernetes

Good find. This looks like logrus is using bufio.Scanner and hitting MaxScanTokenSize. I'll post a patch to switch it to bufio.Reader.

@jpeach
Copy link
Contributor Author

jpeach commented Aug 28, 2020

I got some errors testing through this:

ERRO[0000] Error while reading from Writer: bufio.Scanner: token too long  context=kubernetes

Good find. This looks like logrus is using bufio.Scanner and hitting MaxScanTokenSize. I'll post a patch to switch it to bufio.Reader.

I rewrote this, so the logs now look like this:

INFO[0017] elected leader                                context=loadBalancerStatusWriter
INFO[0017] received a new address for status.loadBalancer  context=loadBalancerStatusWriter loadbalancer-address=
INFO[0017] started informer                              context=StatusAddressUpdater
INFO[0017] Reflector from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 configured with expectedType of *unstructured.Unstructured with empty GroupVersionKind.  context=kubernetes location="reflector.go:159"
INFO[0017] Reflector from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 configured with expectedType of *unstructured.Unstructured with empty GroupVersionKind.  context=kubernetes location="reflector.go:159"
INFO[0017] Starting reflector *unstructured.Unstructured (0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125  context=kubernetes location="reflector.go:175"

@jpeach jpeach force-pushed the k-log-level branch 2 times, most recently from edc9633 to 5476fc4 Compare August 30, 2020 22:12
youngnick
youngnick approved these changes Aug 31, 2020
View reviewed changes
Copy link
Member

@youngnick youngnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with one nit.

@jpeach
Copy link
Contributor Author

jpeach commented Sep 2, 2020

@stevesloka Can you please re-review?

@stevesloka
Copy link
Member

stevesloka commented Sep 3, 2020

I don't have that error any more which is great @jpeach! Do you know if there's a way to keep each message together? What I see is each line of the log is a different log message. It would be nice to have each log from k8s be a single log in the output, but could always improve this later on.

Example:

INFO[0227] GET https://127.0.0.1:34987/api/v1/namespaces/projectcontour/configmaps/leader-elect 200 OK in 4 milliseconds  context=kubernetes location="round_trippers.go:443"
INFO[0227] Response Headers:                             context=kubernetes location="round_trippers.go:449"
INFO[0227]     Content-Type: application/json            context=kubernetes location="round_trippers.go:452"
INFO[0227]     Content-Length: 778                       context=kubernetes location="round_trippers.go:452"
INFO[0227]     Date: Thu, 03 Sep 2020 15:30:21 GMT       context=kubernetes location="round_trippers.go:452"
INFO[0227] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"leader-elect","namespace":"projectcontour","selfLink":"/api/v1/namespaces/projectcontour/configmaps/leader-elect","uid":"d3798361-6ec7-45bc-865a-6b1eaa20f58c","resourceVersion":"236573","creationTimestamp":"2020-07-17T21:27:16Z","annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"1f189465-876c-49b0-a6fd-4e70f2667834\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-09-03T15:26:49Z\",\"renewTime\":\"2020-09-03T15:30:19Z\",\"leaderTransitions\":3}"},"managedFields":[{"manager":"contour","operation":"Update","apiVersion":"v1","time":"2020-09-03T15:30:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:control-plane.alpha.kubernetes.io/leader":{}}}}}]}}  context=kubernetes location="request.go:1068"

@jpeach
Copy link
Contributor Author

jpeach commented Sep 3, 2020

I don't have that error any more which is great @jpeach! Do you know if there's a way to keep each message together? What I see is each line of the log is a different log message. It would be nice to have each log from k8s be a single log in the output, but could always improve this later on.

There's no way to do that, since you can only get an untyped stream of log messages. The klog format is google log format which is line-oriented and each line has a header.

youngnick
youngnick approved these changes Sep 7, 2020
View reviewed changes
Copy link
Member

@youngnick youngnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, seems like the best we can get, and clearly better than now, where kubernetes client lines are not formatted at all.

@jpeach
cmd/contour: add a Kubernetes debug logging flag
1735cda 
Add a `--kubernetes-debug` flag that enables verbose client-go logging
in the `klog` package. The logs are piped through `logrus` so that the
output format is consistent.

Signed-off-by: James Peach <[email protected]>
@jpeach jpeach merged commit 837a7d7 into projectcontour:main Sep 8, 2020
@jpeach jpeach deleted the k-log-level branch September 8, 2020 23:49
jpeach pushed a commit that referenced this pull request Sep 9, 2020
@ShaileshSurya @jpeach
cmd/contour: Replace check function by log.Fatal() (#2838)
1b21483 
There are quite a few places in Contour that call a local check helper
that prints to standard error and exits. This approach loses error context
and makes it difficult for aggregated log collectors to recognize that
this is a fatal error message.

Fixes #2811, #2838.

Signed-off-by: Shailesh Suryawanshi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@skriss skriss Awaiting requested review from skriss

2 more reviewers

@stevesloka stevesloka stevesloka approved these changes

@youngnick youngnick youngnick approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jpeach @stevesloka @youngnick