This repository was archived by the owner on Nov 15, 2017. It is now read-only.
This repository was archived by the owner on Nov 15, 2017. It is now read-only.
Kubernetes AWS problems with multiple security groups due to tags #144
Description
kubernetes/kubernetes#23339, kubernetes/kubernetes#26787
The Kubernetes Controller manages AWS resources by filtering on aws resource tags like KubernetesCluster:ClusterName. Unfortunately it does this inconsistently for different things.
8527 2292 log_handler.go:33] AWS request: elasticloadbalancing DescribeLoadBalancers
3961 2292 aws_loadbalancer.go:191] Deleting removed load balancer listeners
4035 2292 log_handler.go:33] AWS request: elasticloadbalancing DeleteLoadBalancerListeners
1501 2292 aws_loadbalancer.go:203] Creating added load balancer listeners
1592 2292 log_handler.go:33] AWS request: elasticloadbalancing CreateLoadBalancerListeners
3129 2292 log_handler.go:33] AWS request: elasticloadbalancing DescribeLoadBalancerAttributes
3214 2292 log_handler.go:33] AWS request: elasticloadbalancing ModifyLoadBalancerAttributes
4591 2292 log_handler.go:33] AWS request: elasticloadbalancing DescribeLoadBalancers
9882 2292 log_handler.go:33] AWS request: ec2 DescribeSecurityGroups
1322 2292 log_handler.go:33] AWS request: ec2 DescribeSecurityGroups
8421 2292 aws.go:2731] Error opening ingress rules for the load balancer to the instances: Multiple tagged security groups found for instance i-04bd9c4c8aa; ensure only the k8s security group is tagged
8469 2292 servicecontroller.go:754] Failed to process service. Retrying in 5m0s: Failed to create load balancer for service default/pushgateway: Mutiple tagged security groups found for instance i-04bd9c4c8aa36270e; ensure only the k8s security group is tagged
8480 2292 servicecontroller.go:724] Finished syncing service "default/pushgateway" (419.263237ms)
lines 201-224
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go#L2783
// Returns the first security group for an instance, or nil
// We only create instances with one security group, so we don't expect multiple security groups.
// However, if there are multiple security groups, we will choose the one tagged with our cluster filter.
// Otherwise we will return an error.
The security groups in my case are:
k8s-minions-cncfdemo, k8s-masters-cncfdemo
They are both tagged with the cluster filter. Not expecting multiple security groups seems like a wrong (not to mentioned undocumented!) assumption.
Bit of a head scratcher.