Invalid dtab ConfigMap prevents further updates #1639
Description
Hello,
Using the io.l5d.k8s.configMap interpreter I provide a dtab file with a syntax error, which causes an java.lang.IllegalArgumentException in Finagle:
at com.twitter.finagle.NameTreeParsers.illegal(NameTreeParsers.scala:28)
--
| at com.twitter.finagle.NameTreeParsers.illegal(NameTreeParsers.scala:38)
| at com.twitter.finagle.NameTreeParsers.eat(NameTreeParsers.scala:54)
| at com.twitter.finagle.NameTreeParsers.parseDentryPrefix(NameTreeParsers.scala:148)
| at com.twitter.finagle.NameTreeParsers.parseDentry(NameTreeParsers.scala:256)
| at com.twitter.finagle.NameTreeParsers.parseDtab(NameTreeParsers.scala:270)
| at com.twitter.finagle.NameTreeParsers.parseAllDtab(NameTreeParsers.scala:307)
| at com.twitter.finagle.NameTreeParsers$.parseDtab(NameTreeParsers.scala:11)
| at com.twitter.finagle.Dtab$.read(Dtab.scala:355)
| at io.buoyant.interpreter.k8s.ConfigMapInterpreterConfig.getDtab(ConfigMapInterpreterInitializer.scala:80)
| at io.buoyant.interpreter.k8s.ConfigMapInterpreterConfig.$anonfun$extractDtab$1(ConfigMapInterpreterInitializer.scala:69)
| at scala.Option.map(Option.scala:146)
| at io.buoyant.interpreter.k8s.ConfigMapInterpreterConfig.extractDtab(ConfigMapInterpreterInitializer.scala:69)
| at io.buoyant.interpreter.k8s.ConfigMapInterpreterConfig.$anonfun$act$1(ConfigMapInterpreterInitializer.scala:52)
| at io.buoyant.k8s.Watchable.$anonfun$activity$3(Watchable.scala:206)
| at io.buoyant.k8s.Watchable.$anonfun$activity$3$adapted(Watchable.scala:205)
| at com.twitter.util.Future.$anonfun$onSuccess$1(Future.scala:1205)
| at com.twitter.util.Future.$anonfun$onSuccess$1$adapted(Future.scala:1204)
| at com.twitter.util.Promise$Monitored.apply(Promise.scala:202)
| at com.twitter.util.Promise$Monitored.apply(Promise.scala:193)
| at com.twitter.util.Promise$$anon$7.run(Promise.scala:545)
| at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
| at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
| at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)
| at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108)
| at com.twitter.util.Promise.runq(Promise.scala:520)
| at com.twitter.util.Promise.updateIfEmpty(Promise.scala:883)
| at com.twitter.util.Promise.update(Promise.scala:857)
| at com.twitter.util.Promise.setValue(Promise.scala:833)
| at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)
| at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$1.channelRead(ChannelTransport.scala:181)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:33)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
| at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
| at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
| at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
| at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
| at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
| at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
| at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24)
| at java.lang.Thread.run(Thread.java:748)
After this exception all further updates of the ConfigMap are not taken into account: linkerd routes using the last good dtab before the corrupted one.
Furthermore, if I update the page of delegator playground before doing any routing requests, the playground stops working showing "The request to namerd has timed out. Please ensure your config is correct and try again." And never comes back, even after submitting a good dtab. Moreover linkerd stops replying. Then it may reply once and stops again.
I see also
E 0914 15:15:57.061 UTC THREAD35 TraceId:9bd2b8cb39306110: service failure: com.twitter.finagle.CancelledConnectionException