Make k8s Ingress class used by Linkerd configurable (#1433) #1584

pcalcado · 2017-08-15T12:21:54Z

As per #1433 , sometimes one has many different Ingress Resource and Ingress Controller deployed. One common example seems to be when you have one Controller provided by a cloud (e.g. GCE), another one for Linkerd, and a third one provided by Istio. Linkerd, like most controllers, will ignore any Ingress Resource not meant for it, and the way to indicate which one is meant for Linkerds is by using the kubernetes.io/ingress.class annotation.

Initially, Linkerd would always look for any Ingress annotated as linkerd. This change allows for users to configure what annotation value Linkerd should look for.

hawkw

⭐️ Looks good to me. Thanks for working on improving test coverage!

hawkw · 2017-08-15T14:03:46Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+import com.twitter.util.Future
 import io.buoyant.test.Awaits
 import org.scalatest.FunSuite
+import com.twitter.finagle.http.{Request => FRequest, Response => FResponse}


TIOLI, but as far as I can tell, the renaming import shouldn't be necessary here?

Indeed. I'm happy with keeping the name, but something for consideration: Because there are so many types called things like Request, Response, Logger, and Service in the classpath, at SoundCloud we had a convention that every time we imported one of those we'd rename it so that we could identify at a glance where it came from. e.g. we'd always do

import com.twitter.finagle.http.{Request => FinagleRequest, Response => FinagleResponse} import com.soundcloud.ourinternalstuff.http.{Request => ScRequest, Response => ScResponse}

But if we don't follow anything like this convention I'm happy to simplify.

In general, I think our convention is to rename imports only when necessary. If you were importing both finagle.http.Request and a type from Linkerd called Request, then I think we would rename the type from Finagle. But in most of our code that I've seen where we don't have to deal with namespace clashes, we tend not to rename imports by default.

adleong

General approach looks good.

adleong · 2017-08-15T18:34:53Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+      fail(s"unexpected request for [${req.uri}]: $req")
+  }
+
+  test("builds cache with the existing ingress if only one configured") {


Are these test names backwards? This test has many ingresses configured, right?

adleong · 2017-08-15T18:35:36Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+    assert(await(cache.matchPath(host, "/some-path")).get.svc == "echo")
+    assert(await(cache.matchPath(host, "/non-existing-path")).isEmpty)
+  }
+


should we also test the case where there are multiple ingresses with the linkerd class configured? Or is that tested elsewhere?

Uh... good catch. I'll add the test.

esbie

thanks for improving our test coverage!

esbie · 2017-08-15T18:38:06Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+    val service = mkIngressApiServiceReturning(ingressResourceListWithManyAnnotatedIngresses)
+    val cache = new IngressCache(None, service, annotationClass)
+    assert(await(cache.matchPath(host, "/istio-only")).isEmpty)
+    assert(await(cache.matchPath(host, "/ngix-only")).isEmpty)


I'm a little confused why this path isn't matched, since I don't see a "kubernetes.io/ingress.class" specified for nginx-ingress, and we should still use the ingress if no ingress.class is specified.

My understanding is that the way this should work is:

If there is only one ingress resource, we use it

If there are more than one ingress resources, we only use the one annotated with 'linkerd' (or whatever we configure it to be)

Do you think that on (2) we should also add and any other ingress resources without annotations?

I don't see anything in the ingress resource spec that dictates controller annotation behavior. In lieu of a good spec, I try to follow other ingress controllers' behavior. So for traefik https://docs.traefik.io/user-guide/kubernetes/#excluding-an-ingress-from-trfik

You can control which ingress Træfik cares about by using the kubernetes.io/ingress.class annotation. By default if the annotation is not set at all Træfik will include the ingress. If the annotation is set to anything other than traefik or a blank string Træfik will ignore it.

and nginx https://github.com/kubernetes/ingress/tree/master/controllers/nginx

Setting the annotation kubernetes.io/ingress.class to any value other than "nginx" or the empty string, will force the NGINX Ingress controller to ignore your Ingress. Do this if you wish to use one of the other Ingress controllers at the same time as the NGINX controller.

esbie · 2017-08-15T18:39:39Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+    assert(await(cache.matchPath(host, "/linkerd-only")).get.svc == "echo")
+  }
+
+  test("builds cache with only valid ingresses when more than one configured") {


I'm a bit confused about these test names. It seems like this is the test that tests a single configured ingress, whereas the test above is the one with "more than one [ingress class] configured".

Yeah, I've flipped these by mistake :P

Can't believe I missed that...thanks for fixing this, @pcalcado !

adleong

⭐ 🍣 Looks good to me!

esbie · 2017-08-15T22:16:46Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+    val service = mkIngressApiServiceReturning(ingressResourceListWithManyAnnotatedIngresses)
+    val cache = new IngressCache(None, service, annotationClass)
+    assert(await(cache.matchPath(host, "/istio-only")).isEmpty)
+    assert(await(cache.matchPath(host, "/ngix-only")).isEmpty)


looking at this line for another 10 seconds, I see there's a typo in nginx

esbie · 2017-08-15T22:23:29Z

If you're interested in adding tests for the config, I'd recommend writing a ConfigTest. We have a bunch of examples of this, eg https://github.com/linkerd/linkerd/blob/a9c1627d389fab206faac44d1f674695207ff5cd/namerd/storage/consul/src/test/scala/io/buoyant/namerd/storage/consul/ConsulConfigTest.scala

esbie · 2017-08-16T18:22:31Z

k8s/src/test/scala/io/buoyant/k8s/IngressCacheTest.scala

+    assert(await(cache.matchPath(host, "/linkerd-only")).get.svc == "echo")
+  }
+
+  test("when multiple ingress resources with desired annotation class, only adds the first one") {


This does not describe the behavior I expect to see. Linkerd IngressCache will add as many ingress resources as have correct annotations. Whether linkerd chooses to match to echo1 or echo2 is undefined in this case

Aha! Now I understand what you meant earlier, thanks! I've changed it to match specific routes and to be more liberal on the shared one, given which Controller would be picked is undefined. wdyt?

This looks great, thanks for all the updates!

esbie

woop woop 🌟

## 1.2.0 2017-09-07 * **Breaking Change**: `io.l5d.mesh`, `io.l5d.thriftNameInterpreter`, linkerd admin, and namerd admin now serve on 127.0.0.1 by default (instead of 0.0.0.0). * **Breaking Change**: Removed support for PKCS#1-formatted keys. PKCS#1 formatted keys must be converted to PKCS#8 format. * Added experimental `io.l5d.dnssrv` namer for DNS SRV records (#1611) * Kubernetes * Added an experimental `io.l5d.k8s.configMap` interpreter for reading dtabs from a Kubernetes ConfigMap (#1603). This interpreter will respond to changes in the ConfigMap, allowing for dynamic dtab updates without the need to run Namerd. * Made ingress controller's ingress class annotation configurable (#1584). * Fixed an issue where Linkerd would continue routing traffic to endpoints of a service after that service was removed (#1622). * Major refactoring and performance improvements to `io.l5d.k8s` and `io.l5d.k8s.ns` namers (#1603). * Ingress controller now checks all available ingress resources before using a default backend (#1607). * Ingress controller now correctly routes requests with host headers that contain ports (#1607). * HTTP/2 * Fixed an issue where long-running H2 streams would eventually hang (#1598). * Fixed a memory leak on long-running H2 streams (#1598) * Added a user-friendly error message when a HTTP/2 router receives a HTTP/1 request (#1618) * HTTP/1 * Removed spurious `ReaderDiscarded` exception logged on HTTP/1 retries (#1609) * Consul * Added support for querying Consul by specific service health states (#1601) * Consul namers and Dtab store now fall back to a last known good state on Consul observation errors (#1597) * Improved log messages for Consul observation errors (#1597) * TLS * Removed support for PKCS#1 keys (#1590) * Added validation to prevent incompatible `disableValidation: true` and `clientAuth` settings in TLS client configurations (#1621) * Changed `io.l5d.mesh`, `io.l5d.thriftNameInterpreter`, linkerd admin, and namerd admin to serve on 127.0.0.1 by default (instead of 0.0.0.0) (#1366) * Deprecated `io.l5d.statsd` telemeter.

* Use url query params for tap/top form filters * Add comment explaining react-url-query onChange handlers Signed-off-by: Kevin Lingerfelt <[email protected]>

hawkw approved these changes Aug 15, 2017

View reviewed changes

pcalcado force-pushed the phil/1433-make-k8s-ingress-controller-class-configurable branch from c343ed7 to 0006c5d Compare August 15, 2017 15:53

pcalcado added rfc trivial labels Aug 15, 2017

adleong reviewed Aug 15, 2017

View reviewed changes

esbie reviewed Aug 15, 2017

View reviewed changes

pcalcado force-pushed the phil/1433-make-k8s-ingress-controller-class-configurable branch 2 times, most recently from 86cbfbd to 7de636e Compare August 15, 2017 21:30

adleong approved these changes Aug 15, 2017

View reviewed changes

esbie reviewed Aug 15, 2017

View reviewed changes

pcalcado force-pushed the phil/1433-make-k8s-ingress-controller-class-configurable branch 4 times, most recently from 76f58b9 to 221d343 Compare August 16, 2017 17:18

pcalcado mentioned this pull request Aug 16, 2017

Make k8s ingress controller class configurable #1433

Closed

pcalcado added reviewable and removed rfc labels Aug 16, 2017

pcalcado force-pushed the phil/1433-make-k8s-ingress-controller-class-configurable branch from 221d343 to 6cac3df Compare August 16, 2017 17:54

esbie reviewed Aug 16, 2017

View reviewed changes

Phil Calcado added 2 commits August 16, 2017 16:37

Add test for anottated ingresses (#1433)

9a3c395

Make ingress class configurable (#1433)

a3f82bb

pcalcado force-pushed the phil/1433-make-k8s-ingress-controller-class-configurable branch from 6cac3df to a3f82bb Compare August 16, 2017 20:38

esbie approved these changes Aug 16, 2017

View reviewed changes

siggy assigned pcalcado Aug 17, 2017

pcalcado merged commit 2b863a7 into master Aug 17, 2017

pcalcado deleted the phil/1433-make-k8s-ingress-controller-class-configurable branch August 17, 2017 18:16

hawkw added a commit that referenced this pull request Sep 7, 2017

Update changelog entry for #1584

b6ba1b1

hawkw mentioned this pull request Sep 7, 2017

1.2.0 #1629

Merged

Make k8s Ingress class used by Linkerd configurable (#1433) #1584

Make k8s Ingress class used by Linkerd configurable (#1433) #1584

Uh oh!

Conversation

pcalcado commented Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hawkw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcalcado Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adleong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esbie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adleong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esbie commented Aug 15, 2017

Uh oh!

esbie Aug 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esbie left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pcalcado commented Aug 15, 2017 •

edited

Loading

pcalcado Aug 15, 2017 •

edited

Loading

esbie Aug 16, 2017 •

edited

Loading