Storage in Image Management #189

nalind · 2016-11-03T17:52:01Z

Picking up from #28, this PR builds on #210 to pull images using containers/image, and to put images, pod sandboxes, and containers in space managed by containers/storage.

Close #298

runcom · 2016-11-03T17:53:25Z

I'm wondering why Travis disappeared, @mrunalp do you know anything?

mrunalp · 2016-11-04T17:58:20Z

Needs rebase.

nalind · 2016-11-04T20:40:08Z

Ready. Recently-landed changes meant that some of the functionality that was implemented in the storage cri branch was no longer needed, so it's been pruned out.

mrunalp · 2016-11-11T17:46:40Z

There is a lint failure:

checking lint
cmd/server/main.go:88:3:warning: should use for range instead of for { select {} } (gosimple)
make: *** [Makefile:35: lint] Error 1

mrunalp · 2016-11-11T18:01:21Z

From quick manual testing it works fine :) Now running tests locally and reviewing.

mrunalp · 2016-11-11T19:19:59Z

server/container.go

+		return nil, err
+	}
+
+	mountPoint, err := s.storage.StartContainer(containerID)


What is the difference between Create and Start in storage? Creation of r/w layer?

At the storage level, "Start" and "Stop" are more or less equivalent to "Mount" and "Umount". The r/w layer is created in s.storage.CreatePodSandbox or s.storage.CreateContainer. CreatePodSandbox also "Starts" (mounts) the infra container's layer.

nalind · 2016-11-11T23:21:01Z

Rebased to master to resolve conflicts, fixed the lint error, and introduced a proposed default_transport configuration option which, while it's not really an accurate name, hopefully points people in the right direction: if an image name in a pull request can't be parsed as a pull source, we'll prepend the value and try again.

There's some additional logic to convert references to images in other transports into references suitable for use with the local storage when creating images and containers, so some of the self-tests still refer to images using the docker:// prefix, and some don't, hopefully to better exercise the logic.

Reporting of image sizes is kind of broken - if you get an answer at all, it is probably wrong if any of the layers were compressed.

mrunalp · 2016-11-12T01:58:02Z

server/server.go

+		return nil, err
+	}
+
+	imageService, err := scri.GetStorageImageService(store, config.DefaultTransport)


Can we implement the image service in this repo rather than delegating it compelety?

We could. I think it would actually be cleaner in terms of dependencies to move the entirety of containers/storage/cri into this repo, but I also want it to be in a place where other projects and reuse the logic instead of having to reimplement it themselves, and I'm not sure that's within the scope of what this repo wants to be.

Turning the image functions into members of an interface that isn't exactly runtime.ImageServiceServer would let us move some of its API glue here, and let containers/storage lose its hard dependency on k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/runtime, which pulls in quite a few other things, so maybe that'd work better.

We could. I think it would actually be cleaner in terms of dependencies to move the entirety of containers/storage/cri into this repo, but I also want it to be in a place where other projects and reuse the logic instead of having to reimplement it themselves, and I'm not sure that's within the scope of what this repo wants to be.

I agree. IMO storage library shouldn't have any dependencies on the CRI API. That actually makes it harder to use from other projects. You could have convenience APIs to create a Rootfs for an image and a container ID but higher coupling than that will make it harder to reuse.

Okay, just pushed changes which I think move us closer. We still pull in kube by way of containers/image's support for openshift, but we're insulated from changes in the runtime API, at least.

mrunalp · 2016-11-21T19:58:36Z

When I run make install on this PR, I see a bunch of SELinux related errors:

[root@dhcp-16-129 cri-o]# make install
install -D -m 755 ocid /usr/bin/ocid
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found
install -D -m 755 ocic /usr/bin/ocic
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found
install -D -m 755 conmon/conmon /usr/libexec/ocid/conmon
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found
install -D -m 755 pause/pause /usr/libexec/ocid/pause
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found
install -d -m 755 /usr/share/man/man{8,5}
install -m 644 docs/ocid.8 -t /usr/share/man/man8
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found
install -m 644 docs/ocid.conf.5 -t /usr/share/man/man5
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found
install -D -m 644 ocid.conf /etc/ocid.conf
/etc/selinux/targeted/contexts/files/file_contexts.bin:  line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin:  line 1 error due to: Non-ASCII characters found

cc: @rhatdan

mrunalp · 2016-11-21T19:59:50Z

We have a lot of lint errors in container/storage vendoring.

rhatdan · 2016-11-21T20:07:58Z

Looks like you got garbage in
/etc/selinux/targeted/contexts/files/file_contexts.bin: line 1 error due to: Non-ASCII characters found
/etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin: line 1 error due to: Non-ASCII characters found

Just remove these files, they should get recreated.

mrunalp · 2016-11-21T20:17:47Z

@rhatdan That fixed it for me. Thanks!

Signed-off-by: Antonio Murdaca <[email protected]>

The CRI doesn't expect us to implicitly pull an image if it isn't already present before we're asked to use it to create a container, and the tests no longer depend on us doing so, either. Limit the logic which attempts to pull an image, if it isn't present, to only pulling the configured "pause" image, since our use of that image for running pod sandboxes is an implementation detail that our clients can't be expected to know or care about. Include the name of the image that we didn't pull in the error we return when we don't pull one. Signed-off-by: Nalin Dahyabhai <[email protected]>

Signed-off-by: Antonio Murdaca <[email protected]>

nalind · 2017-01-18T15:35:56Z

Rebased, adding debug statements in the handlers for image-related requests that log the results. Updated vendoring results now also pull in github.com/pkg/errors and github.com/opencontainers/go-digest, and the updated github.com/opencontainers/runtime-tools generator's AddProcessEnv method now takes two arguments.

nalind · 2017-01-18T15:40:22Z

Hmm, I'm not that familiar with travis integration, but it doesn't seem to have run for this last rebase. https://www.traviscistatus.com/ says everything should be working, though. Any advice?

runcom · 2017-01-18T15:43:08Z

Hmm, I'm not that familiar with travis integration, but it doesn't seem to have run for this last rebase. https://www.traviscistatus.com/ says everything should be working, though. Any advice?

mmm https://travis-ci.org/kubernetes-incubator/cri-o/pull_requests seems like anything has been triggered from this PR. Could you amend the last commit and re-push? If that doesn't work, it should be something on their end definitely and we can just wait.

nalind · 2017-01-18T15:43:27Z

Sure.

runcom · 2017-01-18T16:43:18Z

@nalind CI triggered, you can remove that test commit now.

nalind · 2017-01-18T19:09:36Z

Okay, CI seems to be back.

mrunalp · 2017-01-18T20:44:09Z

LGTM

sameo · 2017-01-18T21:33:21Z

LGTM

runcom · 2017-01-18T22:02:02Z

LGTM

I'm merging this one right now - there are still issues around devicemapper though and maybe c/storage, we'll tackle these in follow up since I don't believe there's anything left to do as part of this PR (it's better to iterate on simple changes from now on) - for instance, I had to increase the timeout to 15m to be able to run 2 pods spawned at the same time, there seems to be some kind of weird locking which prevents even k8s itself to ping the CRI-O, as said, let's tackle this kind of stuff in follow up. Thanks a lot @nalind and everyone who helped with this :)

jawnsy · 2017-01-18T22:34:48Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉
🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉
🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

Awesome! Great to see this, looking forward to giving it a spin 😄

There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Also consider both Entrypoint and Cmd in the image config when populating the runtime config. The image-spec wording for this is in [1]. This will keep the kubernetes/pause image working, because that image sets Entrypoint [2]. Down the road we might be able to use image-tools image -> runtime config converter [3]. It's currently a private method with public access restricted to full bundle unpacks. I'm not sure we need to check for nil podContainer.Config. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. For this commit, I've assumed it was important and I just clear Process in those cases (which the runtime spec allows [4]). If 'start' is called on those containers and errors (as the spec requires [5]), I think that's a "image was broken" problem and not a "cri-o didn't guess the right value" problem. [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [5]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 Signed-off-by: W. Trevor King <[email protected]>

There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Also consider both Entrypoint and Cmd in the image config when populating the runtime config. The image-spec wording for this is in [1]. This will keep the kubernetes/pause image working, because that image sets Entrypoint [2]. Down the road we might be able to use image-tools image -> runtime config converter [3]. It's currently a private method with public access restricted to full bundle unpacks. I'm not sure we need to check for nil podContainer.Config. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. For this commit, I've assumed it was important and I just clear Process in those cases (which the runtime spec allows [4]). If 'start' is called on those containers and errors (as the spec requires [5]), I think that's a "image was broken" problem and not a "cri-o didn't guess the right value" problem. I've also added a process-clearing check for image configs which set neither Entrypoint nor Cmd (both are optional [6,7]). [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [5]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 [6]: https://github.com/opencontainers/image-spec/blame/v1.0.1/config.md#L138 [7]: https://github.com/opencontainers/image-spec/blame/v1.0.1/config.md#L143 Signed-off-by: W. Trevor King <[email protected]>

There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Also consider both Entrypoint and Cmd in the image config when populating the runtime config. The image-spec wording for this is in [1]. This will keep the kubernetes/pause image working, because that image sets Entrypoint [2]. I'm not sure we need to check for nil podContainer.Config. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. For this commit, I've assumed it was important and I just clear Process in those cases (which the runtime spec allows [3]). If 'start' is called on those containers and errors (as the spec requires [4]), I think that's a "image was broken" problem and not a "cri-o didn't guess the right value" problem. I've also added a process-clearing check for image configs which set neither Entrypoint nor Cmd (both are optional [5,6]). Down the road we might be able to use image-tools image -> runtime config converter [7]. It's currently a private method with public access restricted to full bundle unpacks. [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 [5]: https://github.com/opencontainers/image-spec/blame/v1.0.1/config.md#L138 [6]: https://github.com/opencontainers/image-spec/blame/v1.0.1/config.md#L143 [7]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 Signed-off-by: W. Trevor King <[email protected]>

Use buildOCIProcessArgs to calculate process.args considering PauseCommand (if it was set) and the image's configured Entrypoint and Cmd. This improves the previous implementation, which had been ignoring Entrypoint, despite the image-spec recommendations to consider it [1]. There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Now that we're considering Entrypoint, kubernetes/pause (which sets Entrypoint [2]) no longer needs a PauseCommand "override". I've dropped the podContainer.Config nil check because later code is using it without a guard (e.g. podContainer.Config.Config.StopSignal) and that hasn't caused problems. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. When processArgs is empty, I'm setting Process nil (which the runtime spec allows [3]) following createSandboxContainer. If 'start' is called on those containers and errors (as the spec requires [4]), I think that's an "image was broken" problem and not a "cri-o didn't guess the right value" problem. I've shifted the process.args code down within RunPodSandbox where the possible Process nil can clobber the Process object created by SetProcessSelinuxLabel. I didn't see any later configuration that looked like it would create Process (which would require process.args [5]). Down the road we might be able to use image-tools image -> runtime config converter [6]. It's currently a private method with public access restricted to full bundle unpacks. [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 [5]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L157 [6]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 Signed-off-by: W. Trevor King <[email protected]>

Use buildOCIProcessArgs to calculate process.args considering PauseCommand (if it was set) and the image's configured Entrypoint and Cmd. This improves the previous implementation, which had been ignoring Entrypoint, despite the image-spec recommendations to consider it [1]. There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Now that we're considering Entrypoint, kubernetes/pause (which sets Entrypoint [2]) no longer needs a PauseCommand "override". I've dropped the podContainer.Config nil check because later code is using it without a guard (e.g. podContainer.Config.Config.StopSignal) and that hasn't caused problems. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. When processArgs is empty, I'm setting Process nil (which the runtime spec allows [3]). If 'start' is called on those containers and errors (as the spec requires [4]), I think that's an "image was broken" problem and not a "cri-o didn't guess the right value" problem. I've shifted the process.args code down within RunPodSandbox where the possible Process nil can clobber the Process object created by SetProcessSelinuxLabel. I didn't see any later configuration that looked like it would create Process (which would require process.args [5]). I've shifted the empty-args error from buildOCIProcessArgs into createSandboxContainer, so createSandboxContainer still gets that check but the new RunPodSandbox avoids it. I'd prefer to drop it from createSandboxContainer too, but Mrunal feels like worker containers will always be started and prefers to fail fast in those cases [6]. Down the road we might be able to use image-tools image -> runtime config converter [7]. It's currently a private method with public access restricted to full bundle unpacks. [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 [5]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L157 [6]: cri-o#1265 (comment) [7]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 Signed-off-by: W. Trevor King <[email protected]>

Use buildOCIProcessArgs to calculate process.args considering PauseCommand (if it was set) and the image's configured Entrypoint and Cmd. This improves the previous implementation, which had been ignoring Entrypoint, despite the image-spec recommendations to consider it [1]. There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Now that we're considering Entrypoint, kubernetes/pause (which sets Entrypoint [2]) no longer needs a PauseCommand "override". I've dropped the podContainer.Config nil check because later code is using it without a guard (e.g. podContainer.Config.Config.StopSignal) and that hasn't caused problems. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. When processArgs is empty, I'm setting Process nil (which the runtime spec allows [3]). If 'start' is called on those containers and errors (as the spec requires [4]), I think that's an "image was broken" problem and not a "cri-o didn't guess the right value" problem. I've shifted the process.args code down within RunPodSandbox where the possible Process nil can clobber the Process object created by SetProcessSelinuxLabel. I didn't see any later configuration that looked like it would create Process (which would require process.args [5]). I've shifted the empty-args error from buildOCIProcessArgs into createSandboxContainer, so createSandboxContainer still gets that check but the new RunPodSandbox consumer avoids it. I'd prefer to drop it from createSandboxContainer too, but Mrunal feels like worker containers will always be started and prefers to fail fast in those cases [6]. Down the road we might be able to use image-tools image -> runtime config converter [7]. It's currently a private method with public access restricted to full bundle unpacks. [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 [5]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L157 [6]: cri-o#1265 (comment) [7]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 Signed-off-by: W. Trevor King <[email protected]>

unable to prepare context, unable to evaluate symlinks in context path when building target test-packaging-tools on ppc64le. Fixes: cri-o#189 Signed-off-by: Nitesh Konkar [email protected]

Use buildOCIProcessArgs to calculate process.args considering PauseCommand (if it was set) and the image's configured Entrypoint and Cmd. This improves the previous implementation, which had been ignoring Entrypoint, despite the image-spec recommendations to consider it [1]. There's no reason to assume the pause image contains a /pause command, and it seems more sane to assume that the pause container has configured Entrypoint and Cmd appropriately without our having to repair anything. Now that we're considering Entrypoint, kubernetes/pause (which sets Entrypoint [2]) no longer needs a PauseCommand "override". I've dropped the podContainer.Config nil check because later code is using it without a guard (e.g. podContainer.Config.Config.StopSignal) and that hasn't caused problems. The check was added in c0333b1 (Integrate containers/storage, 2016-10-18, cri-o#189), but there was too much going on for a motivational comment in the commit message. When processArgs is empty, I'm setting Process nil (which the runtime spec allows [3]). If 'start' is called on those containers and errors (as the spec requires [4]), I think that's an "image was broken" problem and not a "cri-o didn't guess the right value" problem. I've shifted the process.args code down within RunPodSandbox where the possible Process nil can clobber the Process object created by SetProcessSelinuxLabel. I didn't see any later configuration that looked like it would create Process (which would require process.args [5]). I've shifted the empty-args error from buildOCIProcessArgs into createSandboxContainer, so createSandboxContainer still gets that check but the new RunPodSandbox consumer avoids it. I'd prefer to drop it from createSandboxContainer too, but Mrunal feels like worker containers will always be started and prefers to fail fast in those cases [6]. Down the road we might be able to use image-tools image -> runtime config converter [7]. It's currently a private method with public access restricted to full bundle unpacks. [1]: https://github.com/opencontainers/image-spec/blob/v1.0.1/conversion.md#verbatim-fields [2]: https://github.com/kubernetes/kubernetes/blob/v1.9.2/build/pause/Dockerfile#L18 [3]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L145 [4]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/runtime.md#L119 [5]: https://github.com/opencontainers/runtime-spec/blame/v1.0.1/config.md#L157 [6]: cri-o#1265 (comment) [7]: https://github.com/opencontainers/image-tools/blob/v0.3.0/image/config.go#L68 Signed-off-by: W. Trevor King <[email protected]>

nalind force-pushed the storage branch 8 times, most recently from ccbac72 to e9eb4c6 Compare November 3, 2016 21:04

nalind force-pushed the storage branch from e9eb4c6 to e8b377d Compare November 4, 2016 20:37

mrunalp mentioned this pull request Nov 10, 2016

Populate image status #178

Closed

mrunalp reviewed Nov 11, 2016

View reviewed changes

nalind force-pushed the storage branch from e8b377d to a3e2333 Compare November 11, 2016 23:20

mrunalp reviewed Nov 12, 2016

View reviewed changes

nalind force-pushed the storage branch 7 times, most recently from 44c91c2 to 301fbfc Compare November 18, 2016 16:33

runcom and others added 3 commits January 18, 2017 10:23

server: cleanup on failed restore

749d24f

Signed-off-by: Antonio Murdaca <[email protected]>

server: fix ImagePullResponse

0e3ff61

Signed-off-by: Antonio Murdaca <[email protected]>

nalind force-pushed the storage branch from 794c01c to 0e3ff61 Compare January 18, 2017 15:35

nalind force-pushed the storage branch from 4e04ff8 to 0e3ff61 Compare January 18, 2017 17:06

runcom merged commit b9dc097 into cri-o:master Jan 18, 2017

mrunalp mentioned this pull request Jan 20, 2017

[Refactor] Move core logical from server to manager #269

Closed

Storage in Image Management #189

Storage in Image Management #189

Uh oh!

Conversation

nalind commented Nov 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

runcom commented Nov 3, 2016

Uh oh!

mrunalp commented Nov 4, 2016

Uh oh!

nalind commented Nov 4, 2016

Uh oh!

mrunalp commented Nov 11, 2016

Uh oh!

mrunalp commented Nov 11, 2016

Uh oh!

mrunalp Nov 11, 2016

Choose a reason for hiding this comment

Uh oh!

nalind Nov 11, 2016

Choose a reason for hiding this comment

Uh oh!

nalind commented Nov 11, 2016

Uh oh!

mrunalp Nov 12, 2016

Choose a reason for hiding this comment

Uh oh!

nalind Nov 15, 2016

Choose a reason for hiding this comment

Uh oh!

mrunalp Nov 15, 2016

Choose a reason for hiding this comment

Uh oh!

nalind Nov 15, 2016

Choose a reason for hiding this comment

Uh oh!

mrunalp commented Nov 21, 2016

Uh oh!

mrunalp commented Nov 21, 2016

Uh oh!

rhatdan commented Nov 21, 2016

Uh oh!

mrunalp commented Nov 21, 2016

Uh oh!

nalind commented Jan 18, 2017

Uh oh!

nalind commented Jan 18, 2017

Uh oh!

runcom commented Jan 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nalind commented Jan 18, 2017

Uh oh!

runcom commented Jan 18, 2017

Uh oh!

nalind commented Jan 18, 2017

Uh oh!

mrunalp commented Jan 18, 2017

Uh oh!

sameo commented Jan 18, 2017

Uh oh!

runcom commented Jan 18, 2017

Uh oh!

jawnsy commented Jan 18, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

nalind commented Nov 3, 2016 •

edited

Loading

runcom commented Jan 18, 2017 •

edited

Loading