conmon: implement logging #162

cyphar · 2016-10-18T16:16:55Z

This adds a very simple implementation of logging within conmon, where
every buffer read from the masterfd of the container is also written to
the log file (with errors during writing to the log file ignored). It also
includes changes to how we compute the logPath for a container.

The WIP part of this PR is that conmon currently doesn't really handle
containers where terminal: false. This is going to be not-very-nice to
fix because the current runC API is broken (which is why we need to
make sure that the current state of opencontainers/runtime-spec#513
is fixed). However, while that's being sorted (and I work on the
relevant console changes in runC) we can solve it with a fork+execve
though no doubt there will be some fun issues with the exit code.

Closes #24.

Signed-off-by: Aleksa Sarai [email protected]

cyphar · 2016-10-18T16:17:24Z

Yeah, the CLA still isn't signed but I though I'd open a PR so nobody else re-does work I've already done. It also opens the door for testing.

mrunalp · 2016-10-18T16:31:06Z

Sweet! I'll test this out.

cyphar · 2016-10-19T10:48:22Z

I've just pushed a version where terminal=false logging actually works. It's all a little bit dodgy at the moment, but now (for the first time) we actually have full logging to a file. :D

mrunalp · 2016-10-19T17:55:50Z

@cyphar Fails tests. I will try manual testing next.

1..10
not ok 1 ctr remove
# (in test file ./ctr.bats, line 35)
#   `[ "$status" -eq 0 ]' failed
# time="2016-10-19T10:54:45-07:00" level=info msg="Starting reaper" 
# E1019 10:54:45.599650    7480 ocicni.go:136] error updating cni config: No networks found in /etc/cni/net.d
# time="2016-10-19T10:54:45-07:00" level=debug msg="sandboxes: map[]" 
# time="2016-10-19T10:54:45-07:00" level=debug msg="containers: &{map[] {{0 0} 0 0 0 0}}" 
# time="2016-10-19T10:54:45-07:00" level=info msg="Signal received: child exited" 
# time="2016-10-19T10:54:45-07:00" level=info msg="Signal received: child exited" 
# time="2016-10-19T10:54:45-07:00" level=debug msg="setting sandbox infra container's log_path" ctr.logfile=d35e7bc3a1bd8111f6c904dd72a5c65c6ffbde65ab170f08118391b2a7d8c46d log_path="/var/log/ocid/pods/5322d4b3450a79798e1a34f9c676fc2005bac828d09fbf7ba382
1b5cac6fc075/d35e7bc3a1bd8111f6c904dd72a5c65c6ffbde65ab170f08118391b2a7d8c46d.log" sbox.logdir="/var/log/ocid/pods/5322d4b3450a79798e1a34f9c676fc2005bac828d09fbf7ba3821b5cac6fc075" 
# time="2016-10-19T10:54:45-07:00" level=debug msg="copying infra rootfs binary: /root/gosrc/src/github.com/kubernetes-incubator/cri-o/pause/pause -> /tmp/tmp.3Qj9yykBiS/ocid/graph/vfs/pause/rootfs/pause" 
# time="2016-10-19T10:54:45-07:00" level=debug msg="running conmon: /root/gosrc/src/github.com/kubernetes-incubator/cri-o/conmon/conmon" args=[-c default-podsandbox1-0-infra -r /usr/local/sbin/runc -l /var/log/ocid/pods/5322d4b3450a79798e1a34f9c676fc2005
bac828d09fbf7ba3821b5cac6fc075/d35e7bc3a1bd8111f6c904dd72a5c65c6ffbde65ab170f08118391b2a7d8c46d.log] 
# [conmon:i]: container PID: 7612
# [conmon:e] Failed to add stdin to epoll Operation not permitted

mrunalp · 2016-10-19T18:22:13Z

pod create/ctr create/ctr start work fine. However I don't see any logs in the container.log file :/
Also ctr stop hangs..

cyphar · 2016-10-20T01:43:55Z

@mrunalp How did you run it? You need to give an absolute path to the log directory in your configs -- otherwise odd things happen to the cwd that I haven't figured out yet. For example, the test data is currently broken (though I think I included a fix) because it sets the log_directory to ".".

mrunalp · 2016-10-20T03:51:07Z

@cyphar I'll try with setting the log_directory. Thanks!

cyphar · 2016-10-20T08:03:59Z

@mrunalp Note: if you set log_directory: "" then my code will automatically put all of the logs inside /var/log/ocid/pod/<pod id>/<ctr id>.log. The kubelet will always tell us a log path, but I thought this might be useful just for our own testing.

mrunalp · 2016-10-20T17:14:01Z

@cyphar I set pod log_directory to /var/log/podsandbox1 and container log_path to container.log The files do get created but I don't see any logs in /var/log/podsandbox1/container.log.

cyphar · 2016-10-21T13:37:11Z

@mrunalp Just to make sure, you did this right (I actually made this mistake a few times):

% sbox_id=$(sudo ./ocic pod create --config test/testdata/sandbox_config.json --name sandbox1)
% ctr_id=$(sudo ./ocic ctr create --config test/testdata/container_config.json --pod $sbox_id)
% sudo ./ocic ctr start --id $ctr_id # I forgot to do this a few times.

After making the changes you mentioned. It works for me:

% sudo cat /var/log/ocid/podsandbox1/container.log
bin
boot
data
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

mrunalp · 2016-10-21T14:26:23Z

@cyphar I have been testing container_redis.json. I'll try with the other config and check whether redis is writing to stdout or stderr.

mrunalp · 2016-10-25T22:22:09Z

@cyphar I just got around to testing this again. Your example with ls works fine but redis-server doesn't work with or without tty.

cyphar · 2016-11-04T02:03:34Z

@mrunalp I've got a branch with the --console-socket implementation, and found some bugs in the runC --console-socket stuff (nothing major). Currently this branch works with conmon if you run it by itself, but for some reason it doesn't work under ocid.

mrunalp · 2016-11-09T23:53:24Z

Okay, just retested this. redis works without tty but no logs with tty.

runcom · 2016-11-24T18:31:24Z

Ping @cyphar any update?

mrunalp · 2016-11-24T20:06:09Z

We are close to getting console rewrite merged in runc which worked last I tested so can just wait for that.

…

On Nov 24, 2016, at 10:31 AM, Antonio Murdaca ***@***.***> wrote: Ping @cyphar any update? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

runcom · 2016-11-24T20:13:57Z

@mrunalp thanks, wasn't aware of that

feiskyer · 2016-12-06T03:58:17Z

What's the status of this?

cyphar · 2016-12-06T04:41:34Z

The same as above. There is a console rewrite PR in runC pending to be merged (opencontainers/runc#1018) -- when you apply that patch to runC this PR works properly. Otherwise there are issues with tty: false containers and I don't really want to debug old code that is going to be replaced soon.

cyphar · 2016-12-07T23:31:22Z

opencontainers/runc#1018 has been merged. I will rebase this soon.

runcom · 2016-12-17T13:32:45Z

@cyphar any update on this one 👼

cyphar · 2016-12-18T17:42:02Z

@runcom Sorry, umoci got in the way. I will work on this today / tomorrow. Promise. If not, I give you permission to spam me on twitter. 😸

cyphar · 2016-12-18T20:40:05Z

Hmmm. I need to figure out where all of my logDir code changes need to be moved to now.

mrunalp · 2017-04-04T14:43:58Z

@sameo Can you retest with etcd?

mrunalp · 2017-04-04T14:48:58Z

okay, retested with latest and looks good 👍 :) will wait for tests to go green and @sameo retest.

sameo · 2017-04-04T14:50:48Z

@mrunalp re-testing...

sameo · 2017-04-04T14:55:51Z

@mrunalp @cyphar Works with etcd and an nginx deployment.

mrunalp · 2017-04-04T14:57:57Z

LGTM (we can merge once tests are green)

runcom · 2017-04-04T15:02:22Z

works for me as well

cyphar · 2017-04-04T15:27:27Z

EUGH. I made a typo in the lint command. The actual tests pass though. I'll push a fix.

runcom · 2017-04-04T15:53:07Z

(Travis isn't giving a single fu*k about our tests apparently)

cyphar · 2017-04-04T15:54:52Z

I think it's because Travis limits the number of jobs for each organisation not repo, so there's some other job being run on behalf of @kubernetes-incubator that is blocking us. FFS.

cyphar · 2017-04-04T16:44:50Z

Oh goddammit. go vet changed their output format between Go releases. Why are lint tests breaking things...

checking lint oci/oci.go:372::warning: declaration of err shadows declaration at oci.go:240: (vetshadow) oci/oci.go:265:15:warning: error return value not checked (os.RemoveAll(logPath)) (errcheck) Signed-off-by: Aleksa Sarai <[email protected]>

The runtimeversion test was incorrectly written and would fail for no good reason if setup_ocid happened to run a command that failed (even if it was handled). Signed-off-by: Aleksa Sarai <[email protected]>

This is necessary for --console-socket. Signed-off-by: Aleksa Sarai <[email protected]>

This adds a very simple implementation of logging within conmon, where every buffer read from the masterfd of the container is also written to the log file (with errors during writing to the log file ignored). Signed-off-by: Aleksa Sarai <[email protected]>

While pipes have their downsides, it turns out that socketpair(2) will break any program that tries to open /dev/std{out,err} for writing (because they're symlinked to /proc/1/fd/{1,2} which will cause lots of fun issues with sockets). Signed-off-by: Mrunal Patel <[email protected]> Signed-off-by: Aleksa Sarai <[email protected]>

CRI requires us to timestamp our logs line-by-line by specifying whether the line came from std{out,err} and the time at which the log was recieved. This is a preliminary implementation of said behaviour (without explicit newline handling at the moment). Signed-off-by: Mrunal Patel <[email protected]> Signed-off-by: Aleksa Sarai <[email protected]>

This is necessary, as otherwise ocid will use its own current directory as a log_directory, which is not the best idea in the world. The same applies for log_path. Signed-off-by: Aleksa Sarai <[email protected]>

The main purpose of these tests is to make sure that the log actually contains output from the container. We don't test the timestamps or the stream that's stated at the moment. Signed-off-by: Aleksa Sarai <[email protected]>

Because kubelet will create broken symlinks for logPath it is necessary to remove those symlinks before we attempt to write to them. This is a temporary workaround while the issue is fixed upstream. Ref: https://issues.k8s.io/44043 Signed-off-by: Aleksa Sarai <[email protected]>

cyphar · 2017-04-04T18:49:46Z

🎉 The tests are green! 🦎 💚

cyphar · 2017-04-04T18:51:10Z

Merging.

@sameo

Closes #162 LGTMs: @sameo @runcom @mrunalp @cyphar

mrunalp · 2017-04-04T18:52:19Z

@cyphar Yay 🎉

runcom · 2017-04-04T21:24:08Z

🎉

rhatdan · 2017-04-08T20:09:47Z

Does this help our k8s test matrix?

mrunalp · 2017-04-08T20:42:49Z

Yes. This and other patches in progress for logging helped pass a lot of the tests.

…

On Apr 8, 2017, at 1:09 PM, Daniel J Walsh ***@***.***> wrote: Does this help our k8s test matrix? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

obs: qemu-vanilla: point to qemu repository.

cyphar mentioned this pull request Oct 21, 2016

Prototype conmon with --console-socket #171

Closed

cyphar changed the title ~~[wip] conmon: implement logging~~ conmon: implement logging Nov 4, 2016

cyphar mentioned this pull request Nov 17, 2016

Add support for streaming requests (attach/exec/port-forward) #200

Closed

4 tasks

runcom modified the milestones: 0.0.1, 0.0.2 Dec 17, 2016

k8s-ci-robot added the cncf-cla: yes label Dec 18, 2016

cyphar and others added 9 commits April 5, 2017 02:45

oci: ignore silly lint errors

8f280dc

checking lint oci/oci.go:372::warning: declaration of err shadows declaration at oci.go:240: (vetshadow) oci/oci.go:265:15:warning: error return value not checked (os.RemoveAll(logPath)) (errcheck) Signed-off-by: Aleksa Sarai <[email protected]>

test: fix runtimeversion test

00589b3

The runtimeversion test was incorrectly written and would fail for no good reason if setup_ocid happened to run a command that failed (even if it was handled). Signed-off-by: Aleksa Sarai <[email protected]>

runc: update to 31980a53ae7887b2c8f8715d13c3eb486c27b6cf

266c757

This is necessary for --console-socket. Signed-off-by: Aleksa Sarai <[email protected]>

test: unset log_* in configurations

c31f2cf

This is necessary, as otherwise ocid will use its own current directory as a log_directory, which is not the best idea in the world. The same applies for log_path. Signed-off-by: Aleksa Sarai <[email protected]>

test: add logging tests

65527da

The main purpose of these tests is to make sure that the log actually contains output from the container. We don't test the timestamps or the stream that's stated at the moment. Signed-off-by: Aleksa Sarai <[email protected]>

cyphar merged commit 7679a84 into cri-o:master Apr 4, 2017

cyphar added a commit that referenced this pull request Apr 4, 2017

merge branch 'pr-162'

7f660a2

Closes #162 LGTMs: @sameo @runcom @mrunalp @cyphar

cyphar deleted the conmon-logging branch April 4, 2017 18:51

wking mentioned this pull request Jun 27, 2018

flake: “The binary conntrack is not installed” #1473

Closed

egernst pushed a commit to egernst/cri-o that referenced this pull request Nov 26, 2018

Merge pull request cri-o#162 from jcvenegas/fix-qemu-vanilla-rng

c2c7c1c

obs: qemu-vanilla: point to qemu repository.

conmon: implement logging #162

conmon: implement logging #162

Uh oh!

Conversation

cyphar commented Oct 18, 2016

Uh oh!

cyphar commented Oct 18, 2016

Uh oh!

mrunalp commented Oct 18, 2016

Uh oh!

cyphar commented Oct 19, 2016

Uh oh!

mrunalp commented Oct 19, 2016

Uh oh!

mrunalp commented Oct 19, 2016

Uh oh!

cyphar commented Oct 20, 2016

Uh oh!

mrunalp commented Oct 20, 2016

Uh oh!

cyphar commented Oct 20, 2016

Uh oh!

mrunalp commented Oct 20, 2016

Uh oh!

cyphar commented Oct 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrunalp commented Oct 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrunalp commented Oct 25, 2016

Uh oh!

cyphar commented Nov 4, 2016

Uh oh!

mrunalp commented Nov 9, 2016

Uh oh!

runcom commented Nov 24, 2016

Uh oh!

mrunalp commented Nov 24, 2016 via email

Uh oh!

runcom commented Nov 24, 2016

Uh oh!

feiskyer commented Dec 6, 2016

Uh oh!

cyphar commented Dec 6, 2016

Uh oh!

cyphar commented Dec 7, 2016

Uh oh!

runcom commented Dec 17, 2016

Uh oh!

cyphar commented Dec 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cyphar commented Dec 18, 2016

Uh oh!

mrunalp commented Apr 4, 2017

Uh oh!

mrunalp commented Apr 4, 2017

Uh oh!

sameo commented Apr 4, 2017

Uh oh!

sameo commented Apr 4, 2017

Uh oh!

mrunalp commented Apr 4, 2017

Uh oh!

runcom commented Apr 4, 2017

Uh oh!

cyphar commented Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

runcom commented Apr 4, 2017

Uh oh!

cyphar commented Apr 4, 2017

Uh oh!

cyphar commented Apr 4, 2017

Uh oh!

cyphar commented Apr 4, 2017

Uh oh!

cyphar commented Apr 4, 2017

Uh oh!

cyphar commented Oct 21, 2016 •

edited

Loading

mrunalp commented Oct 21, 2016 •

edited

Loading

cyphar commented Dec 18, 2016 •

edited

Loading

cyphar commented Apr 4, 2017 •

edited

Loading