Allow nodename to be != hostname, use AWS instance ID on AWS #9728
Conversation
|
GCE e2e build/test passed for commit e0d5b6a8a5fc51bfb7c814b9e8453e072b9f0664. |
|
I broke out the fourth commit into #9747; we should be doing that anyway, and I suspect that should merge faster than this. |
|
@dchen1107 what do you think? This should be low-impact/no-impact other than to AWS. I would really like to get this into 1.0, for two reasons:
If it's just too late to get this into 1.0, I can probably rework #9720 to work in terms of the existing instance names, but I was pretty careful to make it really safe for other clouds (and I think it might even make the code a little clearer to draw a distinction between node name and host name) |
|
@justinsb I like what you proposed here distinguishing nodename from hostname. I am reviewing it now. |
|
ok to test Also this one breaks contrib/mesos. cc/ @jdef |
There was a problem hiding this comment.
why do we still want to keep hostname? why not replace hostname with nodename completely? but still keep hostname_override flag? and marked it to be deprecated? then introducing another one called nodename_override, and it not specified, make it equals to hostname_override?
There was a problem hiding this comment.
I think you are breaking ppl using hostname_override now. Should we treat the value of flag hostname_override as node_name here?
There was a problem hiding this comment.
I don't think I am breaking hostname_override, except maybe on AWS, where if it is broken I want to break it :-)
The logic is this: we build Hostname, just as before: hostname from the OS, but can be overridden by hostname_override. If we have a CloudProvider, we ask it to determine the node-name, but all the implementations apart from AWS simply echo back the maybe-overridden hostname.
That is why we pass hostname in to CurrentNodeName. It is admittedly a weird argument to pass in - it feels a little out of place. Maybe it would be cleaner if we didn't pass in anything, but only changed the nodeName if CurrentNodeName returned non-empty? But then that makes the function even more specialized. I think post V1 we will want a function which returns the CurrentNode() (see https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/kubelet.go#L721-L735)
hostname (& hostname_override) do determine the SSL self-signed cert subject, so I think we have to keep this.
I do agree that we will likely end up with a nodename_override (or just nodename?) but I'm hoping that maybe we can avoid it - that it will only be needed on bare-metal, and on bare-metal the hostname is the logical identifier for nodes.
This is possibly optimistic, but I'd rather avoid introducing another flag until we have someone that needs it!
If you'd prefer, I can create nodename_override, but I don't think anyone will have to use it today.
There was a problem hiding this comment.
[EDIT] in contrib/mesos/pkg/executor/service/service.go set kcfg.NodeName = kcfg.Hostname.
ignore prior statement
also, some unit tests may need updating in contrib/mesos/pkg/executor/executor_test.go
There was a problem hiding this comment.
@justinsb I agreed we shouldn't need nodename_override. The only reason I suggested it above is that I thought we could completely remove hostname* term from our source base, and I think it is very misleading. But you are right that today we need it for SSL cert.
|
I reviewed first two commits which related to kubelet and gce cloud provider, and I think they are pretty sane to me except above two comments. |
|
@dchen1107 posted a slightly rambling reply, but I think we need hostname for the SSL cert, and nodename will be == hostname (as today) everywhere except AWS, where I want it to be different. I agree that nodename_override will probably happen eventually, but am hoping we don't need it. @jdef I'm not particularly familiar with the mesos code, but if you point me in the right direction of what code this might impact I can try to fix it / avoid problems in future. |
There was a problem hiding this comment.
need similar change in contrib/mesos/pkg/executor/service/service.go call to NewMainKubelet
|
you can build/test the |
|
I think there's probably room for improvement w/ respect to the cloudprovider APIs. The API's are used by apiserver, kube-controller-manager and now kubelet. Things seem a bit over-exposed: cloudprovider is a pretty broad grab-bag. Adding For example, perhaps it makes sense to more cleanly separate the APIs that expose "everything that I might want to do-with/know-about the cloud I'm on" from "things I want to know about the node I'm on". |
|
GCE e2e build/test failed for commit 4058adf3e170f64c76f497304e8a11e2ef36afe0. |
|
cc/ @roberthbailey |
|
The implication of not doing this PR-set right now is that the v1.0 release will use "bad" names for nodes, but will anything actually not work? I'm afraid we will have to cross the bridge of "how to upgrade when the upgrade uses different names for nodes" anyway, which takes some of the urgency off this largish patch set. Convince me this is worth the risk at this point? |
|
I think the risk is low for non-AWS clouds: I've tried to separate it out and make it (I hope) clearly a no-op for anything non-AWS. Though there are two variables (nodename and hostname), they are equal everywhere but on AWS. For AWS, there is obviously more risk, but the upside is that we get to use the natural identifier (for AWS users, who expect the instance ID), and that we dump matching-by-primary-dns-name. I believe I have made that work, but it is matching on a secondary index without the same uniqueness constraints as the instance id has. There are some other PRs that also rely on this, including the big one: running the minion nodes in an auto scaling group (#9720 then #9921). I have tried to make it acceptable risk to other clouds. It is higher risk on AWS, but we unlock huge pieces of (last-minute) functionality. We can also have those PRs without this, though I have to rework them to not be based on this, they would have to go through the translation layer. If we can't have this one, but the others are a possibility, let's decide that quickly so I can "de-base" them :-) |
This will allow us to use a nodeName that is not the hostname, for example on clouds where the hostname is not the natural identifier for a node.
The EC2 instance id is the canonical node name on EC2.
|
Rebased to trigger up-to-date e2e |
|
GCE e2e build/test passed for commit 77e1bd3. |
This removes dependency on kubernetes#9728
|
I am also work on a "plan B": breaking the dependency of dynamic CIDRs on this PR, in #9940 I probably should have done that in the first place! |
|
More background in #7775 (comment) |
|
Without having looked at the details in this PR, I'm strongly in favor of clearly distinguishing the node name from the address/hostname the apiserver can use to reach the node for the proxy functionality. The current --hostname_override flag is problematic. Regarding changing node naming schemes later: I'd guess this would require removing a node from the cluster, upgrading its Kubelet to a version using the new naming scheme, and then allowing it to self-register again. |
|
I agreed with @justinsb this pr has low risk for non-aws cloud providers, and it is align with our effort to distinguish node identifier from the hostname too. |
|
LGTM, but still prefer @roberthbailey or @cjcullen take a final look before merge |
|
This seems reasonable to me. |
|
@roberthbailey Thanks. @satnam6502 You can merge this one now. Thanks! |
Allow nodename to be != hostname, use AWS instance ID on AWS
|
I am going to revert this to see if it fixes the unit tests. |
|
I think this PR causes our unit tests to fail. |
|
Sorry about the problem - not sure how that got past my testing, but I guess Shippable wasn't a flake after all :-( Fix coming soon |
|
Actually that was a different commit where I thought Shippable was a flake. (Which I am now double-checking ;-) ) Trying to figure it out... |
This is a partial reversion of kubernetes#9728, and should fix kubernetes#10612. 9728 used the AWS instance id as the node name. But proxy, logs and exec all used the node name as the host name for contacting the minion. It is possible to resolve a host to the IP, and this fixes logs. But exec and proxy also require an SSL certificate match on the hostname, and this is harder to fix. So the sensible fix seems to be a minimal reversion of the changes in kubernetes#9728, and we can revisit this post 1.0.
Most of our communications from apiserver -> nodes used nodutil.GetNodeHostIP, but a few places didn't - and this meant that the node name needed to be resolvable _and_ we needed to populate valid IP addresses. Fix the last few places that used the NodeName. Issue kubernetes#18525 Issue kubernetes#9451 Issue kubernetes#9728 Issue kubernetes#17643 Issue kubernetes#11543 Issue kubernetes#22063 Issue kubernetes#2462 Issue kubernetes#22109 Issue kubernetes#22770 Issue kubernetes#32286
We currently assume that a node's name is the hostname; that is natural on GCE where the name of an instance becomes the hostname, and where you refer to instances by name. But other clouds (notably AWS) typically have synthetic instance names, and it makes more sense on those not to assume that nodename == hostname.
This has been a long time coming because we assumed in a lot of places that node-name was resolvable; but there have all been fixed now (I believe!)
The first commit here should be the most controversial: it differentiates between a nodename and a hostname.
The second commit allows a cloud provider to specify a different name for its own node, though nobody actually uses this.
The third commit wires up the previous commit to work with AWS; so AWS nodes use the instance id for the name. This also means we don't need to query for the PublicDnsName any more, but can just retrieve instances by id.
The fourth commit just adds AWS to the list of clouds where the master kubelet does not register as a node.