Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Allow nodename to be != hostname, use AWS instance ID on AWS #9728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 18, 2015
Merged

Allow nodename to be != hostname, use AWS instance ID on AWS #9728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c28cdfb
For kubelet, differentiate between the nodeName and the hostname
justinsb Jun 12, 2015
efaead8
Allow cloud providers to return a node identifier different from the …
justinsb Jun 12, 2015
c89b0cd
AWS: Use the instance id as the node name
justinsb Jun 12, 2015
77e1bd3
NodeName != HostName: Fixes for contrib/mesos
justinsb Jun 16, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 28 additions & 4 deletions cmd/kubelet/app/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -565,8 +565,30 @@ func SimpleKubelet(client *client.Client,
// Eventually, #2 will be replaced with instances of #3
func RunKubelet(kcfg *KubeletConfig, builder KubeletBuilder) error {
kcfg.Hostname = nodeutil.GetHostname(kcfg.HostnameOverride)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we still want to keep hostname? why not replace hostname with nodename completely? but still keep hostname_override flag? and marked it to be deprecated? then introducing another one called nodename_override, and it not specified, make it equals to hostname_override?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are breaking ppl using hostname_override now. Should we treat the value of flag hostname_override as node_name here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I am breaking hostname_override, except maybe on AWS, where if it is broken I want to break it :-)

The logic is this: we build Hostname, just as before: hostname from the OS, but can be overridden by hostname_override. If we have a CloudProvider, we ask it to determine the node-name, but all the implementations apart from AWS simply echo back the maybe-overridden hostname.

That is why we pass hostname in to CurrentNodeName. It is admittedly a weird argument to pass in - it feels a little out of place. Maybe it would be cleaner if we didn't pass in anything, but only changed the nodeName if CurrentNodeName returned non-empty? But then that makes the function even more specialized. I think post V1 we will want a function which returns the CurrentNode() (see https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/kubelet.go#L721-L735)

hostname (& hostname_override) do determine the SSL self-signed cert subject, so I think we have to keep this.

I do agree that we will likely end up with a nodename_override (or just nodename?) but I'm hoping that maybe we can avoid it - that it will only be needed on bare-metal, and on bare-metal the hostname is the logical identifier for nodes.

This is possibly optimistic, but I'd rather avoid introducing another flag until we have someone that needs it!

If you'd prefer, I can create nodename_override, but I don't think anyone will have to use it today.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[EDIT] in contrib/mesos/pkg/executor/service/service.go set kcfg.NodeName = kcfg.Hostname.
ignore prior statement

also, some unit tests may need updating in contrib/mesos/pkg/executor/executor_test.go

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justinsb I agreed we shouldn't need nodename_override. The only reason I suggested it above is that I thought we could completely remove hostname* term from our source base, and I think it is very misleading. But you are right that today we need it for SSL cert.


if kcfg.NodeName == "" {
// Query the cloud provider for our node name, default to Hostname
nodeName := kcfg.Hostname
if kcfg.Cloud != nil {
var err error
instances, ok := kcfg.Cloud.Instances()
if !ok {
return fmt.Errorf("failed to get instances from cloud provider")
}

nodeName, err = instances.CurrentNodeName(kcfg.Hostname)
if err != nil {
return fmt.Errorf("error fetching current instance name from cloud provider: %v", err)
}

glog.V(2).Infof("cloud provider determined current node name to be %s", nodeName)
}

kcfg.NodeName = nodeName
}

eventBroadcaster := record.NewBroadcaster()
kcfg.Recorder = eventBroadcaster.NewRecorder(api.EventSource{Component: "kubelet", Host: kcfg.Hostname})
kcfg.Recorder = eventBroadcaster.NewRecorder(api.EventSource{Component: "kubelet", Host: kcfg.NodeName})
eventBroadcaster.StartLogging(glog.Infof)
if kcfg.KubeClient != nil {
glog.V(4).Infof("Sending events to api server.")
Expand Down Expand Up @@ -625,17 +647,17 @@ func makePodSourceConfig(kc *KubeletConfig) *config.PodConfig {
// define file config source
if kc.ConfigFile != "" {
glog.Infof("Adding manifest file: %v", kc.ConfigFile)
config.NewSourceFile(kc.ConfigFile, kc.Hostname, kc.FileCheckFrequency, cfg.Channel(kubelet.FileSource))
config.NewSourceFile(kc.ConfigFile, kc.NodeName, kc.FileCheckFrequency, cfg.Channel(kubelet.FileSource))
}

// define url config source
if kc.ManifestURL != "" {
glog.Infof("Adding manifest url: %v", kc.ManifestURL)
config.NewSourceURL(kc.ManifestURL, kc.Hostname, kc.HTTPCheckFrequency, cfg.Channel(kubelet.HTTPSource))
config.NewSourceURL(kc.ManifestURL, kc.NodeName, kc.HTTPCheckFrequency, cfg.Channel(kubelet.HTTPSource))
}
if kc.KubeClient != nil {
glog.Infof("Watching apiserver")
config.NewSourceApiserver(kc.KubeClient, kc.Hostname, cfg.Channel(kubelet.ApiserverSource))
config.NewSourceApiserver(kc.KubeClient, kc.NodeName, cfg.Channel(kubelet.ApiserverSource))
}
return cfg
}
Expand All @@ -656,6 +678,7 @@ type KubeletConfig struct {
FileCheckFrequency time.Duration
HTTPCheckFrequency time.Duration
Hostname string
NodeName string
PodInfraContainerImage string
SyncFrequency time.Duration
RegistryPullQPS float64
Expand Down Expand Up @@ -715,6 +738,7 @@ func createAndInitKubelet(kc *KubeletConfig) (k KubeletBootstrap, pc *config.Pod
pc = makePodSourceConfig(kc)
k, err = kubelet.NewMainKubelet(
kc.Hostname,
kc.NodeName,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need similar change in contrib/mesos/pkg/executor/service/service.go call to NewMainKubelet

kc.DockerClient,
kubeClient,
kc.RootDirectory,
Expand Down
3 changes: 3 additions & 0 deletions contrib/mesos/pkg/executor/service/service.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,8 @@ func (s *KubeletExecutorServer) Run(hks hyperkube.Interface, _ []string) error {
DockerExecHandler: dockerExecHandler,
}

kcfg.NodeName = kcfg.Hostname

err = app.RunKubelet(&kcfg, app.KubeletBuilder(func(kc *app.KubeletConfig) (app.KubeletBootstrap, *kconfig.PodConfig, error) {
return s.createAndInitKubelet(kc, hks, clientConfig, shutdownCloser)
}))
Expand Down Expand Up @@ -319,6 +321,7 @@ func (ks *KubeletExecutorServer) createAndInitKubelet(

klet, err := kubelet.NewMainKubelet(
kc.Hostname,
kc.NodeName,
kc.DockerClient,
kubeClient,
kc.RootDirectory,
Expand Down
100 changes: 41 additions & 59 deletions pkg/cloudprovider/aws/aws.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,14 @@ func (self *AWSCloud) AddSSHKeyToAllInstances(user string, keyData []byte) error
return errors.New("unimplemented")
}

func (a *AWSCloud) CurrentNodeName(hostname string) (string, error) {
selfInstance, err := a.getSelfAWSInstance()
if err != nil {
return "", err
}
return selfInstance.awsID, nil
}

// Implementation of EC2.Instances
func (self *awsSdkEC2) DescribeInstances(request *ec2.DescribeInstancesInput) ([]*ec2.Instance, error) {
// Instances are paged
Expand Down Expand Up @@ -547,7 +555,7 @@ func (aws *AWSCloud) Routes() (cloudprovider.Routes, bool) {

// NodeAddresses is an implementation of Instances.NodeAddresses.
func (aws *AWSCloud) NodeAddresses(name string) ([]api.NodeAddress, error) {
instance, err := aws.getInstanceByDnsName(name)
instance, err := aws.getInstanceById(name)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -581,7 +589,8 @@ func (aws *AWSCloud) NodeAddresses(name string) ([]api.NodeAddress, error) {

// ExternalID returns the cloud provider ID of the specified instance (deprecated).
func (aws *AWSCloud) ExternalID(name string) (string, error) {
inst, err := aws.getInstanceByDnsName(name)
// TODO: Do we need to verify it exists, or can we just return name
inst, err := aws.getInstanceById(name)
if err != nil {
return "", err
}
Expand All @@ -590,7 +599,8 @@ func (aws *AWSCloud) ExternalID(name string) (string, error) {

// InstanceID returns the cloud provider ID of the specified instance.
func (aws *AWSCloud) InstanceID(name string) (string, error) {
inst, err := aws.getInstanceByDnsName(name)
// TODO: Do we need to verify it exists, or can we just construct it knowing our AZ (or via caching?)
inst, err := aws.getInstanceById(name)
if err != nil {
return "", err
}
Expand All @@ -599,46 +609,6 @@ func (aws *AWSCloud) InstanceID(name string) (string, error) {
return "/" + orEmpty(inst.Placement.AvailabilityZone) + "/" + orEmpty(inst.InstanceID), nil
}

// Return the instances matching the relevant private dns name.
func (s *AWSCloud) getInstanceByDnsName(name string) (*ec2.Instance, error) {
filters := []*ec2.Filter{
newEc2Filter("private-dns-name", name),
}
filters = s.addFilters(filters)
request := &ec2.DescribeInstancesInput{
Filters: filters,
}

instances, err := s.ec2.DescribeInstances(request)
if err != nil {
return nil, err
}

matchingInstances := []*ec2.Instance{}
for _, instance := range instances {
// TODO: Push running logic down into filter?
if !isAlive(instance) {
continue
}

if orEmpty(instance.PrivateDNSName) != name {
// TODO: Should we warn here? - the filter should have caught this
// (this will happen in the tests if they don't fully mock the EC2 API)
continue
}

matchingInstances = append(matchingInstances, instance)
}

if len(matchingInstances) == 0 {
return nil, fmt.Errorf("no instances found for host: %s", name)
}
if len(matchingInstances) > 1 {
return nil, fmt.Errorf("multiple instances found for host: %s", name)
}
return matchingInstances[0], nil
}

// Check if the instance is alive (running or pending)
// We typically ignore instances that are not alive
func isAlive(instance *ec2.Instance) bool {
Expand Down Expand Up @@ -698,16 +668,9 @@ func (s *AWSCloud) getInstancesByRegex(regex string) ([]string, error) {
continue
}

privateDNSName := orEmpty(instance.PrivateDNSName)
if privateDNSName == "" {
glog.V(2).Infof("skipping EC2 instance (no PrivateDNSName): %s",
orEmpty(instance.InstanceID))
continue
}

for _, tag := range instance.Tags {
if orEmpty(tag.Key) == "Name" && re.MatchString(orEmpty(tag.Value)) {
matchingInstances = append(matchingInstances, privateDNSName)
matchingInstances = append(matchingInstances, orEmpty(instance.InstanceID))
break
}
}
Expand All @@ -724,7 +687,7 @@ func (aws *AWSCloud) List(filter string) ([]string, error) {

// GetNodeResources implements Instances.GetNodeResources
func (aws *AWSCloud) GetNodeResources(name string) (*api.NodeResources, error) {
instance, err := aws.getInstanceByDnsName(name)
instance, err := aws.getInstanceById(name)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -1185,7 +1148,7 @@ func (aws *AWSCloud) getAwsInstance(instanceName string) (*awsInstance, error) {
return nil, fmt.Errorf("error getting self-instance: %v", err)
}
} else {
instance, err := aws.getInstanceByDnsName(instanceName)
instance, err := aws.getInstanceById(instanceName)
if err != nil {
return nil, fmt.Errorf("error finding instance: %v", err)
}
Expand Down Expand Up @@ -1652,7 +1615,7 @@ func (s *AWSCloud) CreateTCPLoadBalancer(name, region string, publicIP net.IP, p
return nil, fmt.Errorf("publicIP cannot be specified for AWS ELB")
}

instances, err := s.getInstancesByDnsNames(hosts)
instances, err := s.getInstancesByIds(hosts)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -2064,7 +2027,7 @@ func (s *AWSCloud) EnsureTCPLoadBalancerDeleted(name, region string) error {

// UpdateTCPLoadBalancer implements TCPLoadBalancer.UpdateTCPLoadBalancer
func (s *AWSCloud) UpdateTCPLoadBalancer(name, region string, hosts []string) error {
instances, err := s.getInstancesByDnsNames(hosts)
instances, err := s.getInstancesByIds(hosts)
if err != nil {
return err
}
Expand Down Expand Up @@ -2139,21 +2102,40 @@ func (s *AWSCloud) UpdateTCPLoadBalancer(name, region string, hosts []string) er
}

// TODO: Make efficient
func (a *AWSCloud) getInstancesByDnsNames(names []string) ([]*ec2.Instance, error) {
func (a *AWSCloud) getInstancesByIds(ids []string) ([]*ec2.Instance, error) {
instances := []*ec2.Instance{}
for _, name := range names {
instance, err := a.getInstanceByDnsName(name)
for _, id := range ids {
instance, err := a.getInstanceById(id)
if err != nil {
return nil, err
}
if instance == nil {
return nil, fmt.Errorf("unable to find instance " + name)
return nil, fmt.Errorf("unable to find instance " + id)
}
instances = append(instances, instance)
}
return instances, nil
}

// Returns the instance with the specified ID
func (a *AWSCloud) getInstanceById(instanceID string) (*ec2.Instance, error) {
request := &ec2.DescribeInstancesInput{
InstanceIDs: []*string{&instanceID},
}

instances, err := a.ec2.DescribeInstances(request)
if err != nil {
return nil, err
}
if len(instances) == 0 {
return nil, fmt.Errorf("no instances found for instance: %s", instanceID)
}
if len(instances) > 1 {
return nil, fmt.Errorf("multiple instances found for instance: %s", instanceID)
}
return instances[0], nil
}

// Add additional filters, to match on our tags
// This lets us run multiple k8s clusters in a single EC2 AZ
func (s *AWSCloud) addFilters(filters []*ec2.Filter) []*ec2.Filter {
Expand Down
28 changes: 14 additions & 14 deletions pkg/cloudprovider/aws/aws_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ func TestList(t *testing.T) {
Value: aws.String("foo"),
}
instance0.Tags = []*ec2.Tag{&tag0}
instance0.PrivateDNSName = aws.String("instance1")
instance0.InstanceID = aws.String("instance0")
state0 := ec2.InstanceState{
Name: aws.String("running"),
}
Expand All @@ -424,7 +424,7 @@ func TestList(t *testing.T) {
Value: aws.String("bar"),
}
instance1.Tags = []*ec2.Tag{&tag1}
instance1.PrivateDNSName = aws.String("instance2")
instance1.InstanceID = aws.String("instance1")
state1 := ec2.InstanceState{
Name: aws.String("running"),
}
Expand All @@ -436,7 +436,7 @@ func TestList(t *testing.T) {
Value: aws.String("baz"),
}
instance2.Tags = []*ec2.Tag{&tag2}
instance2.PrivateDNSName = aws.String("instance3")
instance2.InstanceID = aws.String("instance2")
state2 := ec2.InstanceState{
Name: aws.String("running"),
}
Expand All @@ -448,7 +448,7 @@ func TestList(t *testing.T) {
Value: aws.String("quux"),
}
instance3.Tags = []*ec2.Tag{&tag3}
instance3.PrivateDNSName = aws.String("instance4")
instance3.InstanceID = aws.String("instance3")
state3 := ec2.InstanceState{
Name: aws.String("running"),
}
Expand All @@ -462,8 +462,8 @@ func TestList(t *testing.T) {
expect []string
}{
{"blahonga", []string{}},
{"quux", []string{"instance4"}},
{"a", []string{"instance2", "instance3"}},
{"quux", []string{"instance3"}},
{"a", []string{"instance1", "instance2"}},
}

for _, item := range table {
Expand Down Expand Up @@ -493,7 +493,7 @@ func TestNodeAddresses(t *testing.T) {
var instance1 ec2.Instance

//0
instance0.PrivateDNSName = aws.String("instance1")
instance0.InstanceID = aws.String("instance-same")
instance0.PrivateIPAddress = aws.String("192.168.0.1")
instance0.PublicIPAddress = aws.String("1.2.3.4")
instance0.InstanceType = aws.String("c3.large")
Expand All @@ -503,7 +503,7 @@ func TestNodeAddresses(t *testing.T) {
instance0.State = &state0

//1
instance1.PrivateDNSName = aws.String("instance1")
instance1.InstanceID = aws.String("instance-same")
instance1.PrivateIPAddress = aws.String("192.168.0.2")
instance1.InstanceType = aws.String("c3.large")
state1 := ec2.InstanceState{
Expand All @@ -514,19 +514,19 @@ func TestNodeAddresses(t *testing.T) {
instances := []*ec2.Instance{&instance0, &instance1}

aws1 := mockInstancesResp([]*ec2.Instance{})
_, err1 := aws1.NodeAddresses("instance")
_, err1 := aws1.NodeAddresses("instance-mismatch")
if err1 == nil {
t.Errorf("Should error when no instance found")
}

aws2 := mockInstancesResp(instances)
_, err2 := aws2.NodeAddresses("instance1")
_, err2 := aws2.NodeAddresses("instance-same")
if err2 == nil {
t.Errorf("Should error when multiple instances found")
}

aws3 := mockInstancesResp(instances[0:1])
addrs3, err3 := aws3.NodeAddresses("instance1")
addrs3, err3 := aws3.NodeAddresses("instance-same")
if err3 != nil {
t.Errorf("Should not error when instance found")
}
Expand Down Expand Up @@ -562,23 +562,23 @@ func TestGetResources(t *testing.T) {
var instance2 ec2.Instance

//0
instance0.PrivateDNSName = aws.String("m3.medium")
instance0.InstanceID = aws.String("m3.medium")
instance0.InstanceType = aws.String("m3.medium")
state0 := ec2.InstanceState{
Name: aws.String("running"),
}
instance0.State = &state0

//1
instance1.PrivateDNSName = aws.String("r3.8xlarge")
instance1.InstanceID = aws.String("r3.8xlarge")
instance1.InstanceType = aws.String("r3.8xlarge")
state1 := ec2.InstanceState{
Name: aws.String("running"),
}
instance1.State = &state1

//2
instance2.PrivateDNSName = aws.String("unknown.type")
instance2.InstanceID = aws.String("unknown.type")
instance2.InstanceType = aws.String("unknown.type")
state2 := ec2.InstanceState{
Name: aws.String("running"),
Expand Down
Loading