Add simple env var resolution #7936
Conversation
|
If people are okay with this, I will write doc for the API. |
pkg/kubelet/kubelet_test.go
Outdated
There was a problem hiding this comment.
Don't know why we would transform this?
There was a problem hiding this comment.
@smarterclayton os.Env transforms ${ into the empty string, even if it's unmatched.
There was a problem hiding this comment.
Can we fork os.Env to fix this and escaping?
There was a problem hiding this comment.
Fwiw i have some unpushed code with more variants and dusty corners of
os.Expand's behavior explored.
On Fri, May 8, 2015 at 2:06 PM Brian Grant [email protected] wrote:
In pkg/kubelet/kubelet_test.go
#7936 (comment)
:
Value: "literal-${TEST_LITERAL}",},{Name: "SERVICE_VAR_TEST",Value: "${TEST_SERVICE_HOST}:${TEST_SERVICE_PORT}",},},},masterServiceNs: "nothing",nilLister: false,expectedEnvs: util.NewStringSet("TEST_LITERAL=test-test-test","POD_NAME=dapi-test-pod-name","POD_NAME_TEST=test-dapi-test-pod-name","POD_NAME_TEST2=test2-dapi-test-pod-name","POD_NAME_TEST3=test3-POD_NAME",Can we fork os.Env to fix this and escaping?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7936/files#r29962317
.
|
Do we really need to do this? It's one step away from evaluating Command and Args in the same way. I argued that Docker should not do this but they did anyway. Can we just depend on that? @jonboulle this is not part of appc right now... |
|
Docker does not do this, or at least, I can't make it work in 1.6. Command and Args are almost useless in practice without a ton of bash inside my container, which reduces the utility of env vars (service or field ref) enough where it's very difficult to use them for anything. If we don't make resolution possible, we essentially are saying "don't use our env vars to do anything interesting".
|
|
Caveat, I'm not advocating it must be automatic. But it's very difficult to use our env for anything real (esp building urls to services)
|
|
Oops, I was going to have a flag in the kibelet that turned this on and
|
|
I'm not sure whether this is an API thing or a kubelet thing. If it's optional, why would it be optional at any layer except the API?
|
|
@thockin I don't understand how this relates to appc, isn't the proposal here to just change the contract between the kubelet and the runtime? |
|
@smarterclayton I'm not sure what optional at the API server means in this case -- if it's not allowed, is it a validation error? If so we might break people that are passing who knows what in. I guess the API server could have two config items:
|
|
I'm talking about an option on the pod or container. If the concern is resolution is unexpected or not always desired, then it's probably a Boolean on the container. The question is whether you ever use ${} inside an environment variable and not want it resolved. I lean towards no, but there's room to be cautious about that statement.
|
|
@smarterclayton One other thing we could do is have an apiVersion: v1beta3
kind: Pod
metadata:
name: expansion-pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: A_NEW_SERVICE_URL
valueFrom:
expansion: "glorp://${FOO_SERVICE_HOST}:${FOO_SERVICE_PORT}"
restartPolicy: Never |
|
The |
|
So I expect expansion to be common. Doing a quick review of the existing examples, here's a list:
That was just me going down the list of our examples in order until I got bored (about 5 seconds). And they all needed it to even boot their thing up on Kube, in a very simple example. So the question is, as soon as someone needs namespace to make path segment unique on a shared disk, where will they get it? Pod name to get an identity? What if you want it to be shorter? Build something real that talks to multiple services? I could live with envvarsource, I just don't think it's practical to abuse our users that way. |
|
@jonboulle yeah, sorry, I got carried away. I thought Docker was doing On Thu, May 7, 2015 at 6:45 PM, Jonathan Boulle [email protected]
|
|
The examples I looked at all need it in argv, not in env vars. Expanding On Thu, May 7, 2015 at 9:13 PM, Clayton Coleman [email protected]
|
|
You mean the ones using code? Or writing shell wrappers just to mutate env? Why would we force people to expand in Argv? That's forcing the user to do something awkward. Can you articulate why controlled env expansion is bad?
|
|
Not that we shouldn't allow expansion in Argv. I just thing Argv is reserved for arguments (some of which can be expanded) , and Env should be reserved for Env (some of which can be expanded). Taking env into Argv in order to get it back into env strikes me as a workaround. Likewise, I want all docker shell wrappers to disappear if they have _SERVICE_HOST or the pod namespace or the host IP (which we should expose via fieldref). Having it in two places (shell and pod) is more fragile and less flexible.
|
Yes. One other thing, you can't do the expansion in argv if your image doesn't have a shell. |
|
I think I will re-implement this with an env var source -- I would really prefer not to suddenly change the behavior of the existing Value field. |
|
Before you go burn more time, let's really figure out if we want to do this On Thu, May 7, 2015 at 10:30 PM, Paul Morie [email protected]
|
|
My preference would be to eventually require service fieldrefs in order to get service env vars (eventually), so requiring a field ref in order to expand is not unreasonable. In that model we can nuke auto injection (eventually) or make it optional. They should be lower in priority than the env provided in the pod as long as they are magic. Let's marshal the arguments on both sides here and reach a decision. We've been dancing around this for months and it's something every non-trivial use case on Kube has already put in as a "hack" (lit: a change required solely to make X work on Kube). Paul or I will do the "pro" side
|
In #7739:
|
|
Don't get me wrong. I think it's cute. I am just not convinced it is cute The Cons:
On Fri, May 8, 2015 at 5:51 AM, Clayton Coleman [email protected]
|
|
Here's my own attempt at an accounting of pros and cons. Feel free to remix or add your own unique twist to express your feels on this subject: CONS
PROS (I'm writing this including possible
|
zing! |
Agree. I honk the justification is real users and use cases (constructing urls to other services in pod.json vs creating custom docker images to do that).
Defaults would be required to be specified as part of the container env set I think (in the short term there are no defaults). Since its evaluated all at once.
Which I believe we already said blindly injection them was a mistake? Which I was hoping to fix by making it explicit here.
Either we don't expand things that aren't defined (no escaping) or we have the value be defined as a variable itself that doesn't expand and we don't double expand. I believe this is solvable.
|
|
I think the way to correct that is to have you try to take a real non-Kube app that needs to either known it's own URL or get another persons. I think you'd agree with me after doing it. But I'm an optimist :)
|
Good point |
|
@derekwaynecarr I think this is ready for review. |
|
Thanks for the heads up. Sent from my iPhone
|
|
I recommend the following:
|
|
@derekwaynecarr Added examples and a plan for how to support events. PTAL and let me know your thoughts on the events stuff. I think we may need to add an interface in the third party code that mirrors my proposed |
|
Example use the wrong syntax... |
|
Whoops, thanks 4 catch |
|
LGTM, will merge when shippable/travis is happy. |
b8137cf to
7769617
Compare
There was a problem hiding this comment.
nit, but I think you mean "not consumed" both here and below?
da279f1 to
b29740a
Compare
|
LGTM |
|
⌛ ⌛ ⌛ ^ My impression of travis-ci |
|
Shippable going green is good enough. Thanks @pmorie! |
Add simple env var resolution
|
👍 |
|
Created #8465 as a follow-up to use byte buffer |
WIP for #7739
@smarterclayton PTAL