Strange class cast exception #1472
Description
Hello,
we are using linkerd successfully for about 2 months now. Strangely since a few days we get lot (like hundreds per hour) of these exceptions:
java.lang.ClassCastException: Transport.cast failed. Expected type io.netty.handler.codec.http2.Http2Frame but found io.netty.handler.codec.http.DefaultHttpRequest
at com.twitter.finagle.transport.Transport$$anon$3.$anonfun$readFn$1 (Transport.scala:321)
at com.twitter.util.Future.$anonfun$flatMap$1 (Future.scala:1089)
at com.twitter.util.Promise$Transformer.liftedTree1$1 (Promise.scala:107)
at com.twitter.util.Promise$Transformer.k (Promise.scala:107)
at com.twitter.util.Promise$Transformer.apply (Promise.scala:117)
at com.twitter.util.Promise$Transformer.apply (Promise.scala:98)
at com.twitter.util.Promise$$anon$1.run (Promise.scala:421)
at com.twitter.concurrent.LocalScheduler$Activation.run (Scheduler.scala:200)
at com.twitter.concurrent.LocalScheduler$Activation.submit (Scheduler.scala:158)
at com.twitter.concurrent.LocalScheduler.submit (Scheduler.scala:272)
at com.twitter.concurrent.Scheduler$.submit (Scheduler.scala:108)
at com.twitter.util.Promise.runq (Promise.scala:406)
at com.twitter.util.Promise.updateIfEmpty (Promise.scala:801)
at com.twitter.util.Promise.update (Promise.scala:775)
at com.twitter.util.Promise.setValue (Promise.scala:751)
at com.twitter.concurrent.AsyncQueue.offer (AsyncQueue.scala:123)
at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$1.channelRead (ChannelTransport.scala:181)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead (ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead (ChannelInboundHandlerAdapter.java:86)
at com.twitter.finagle.netty4.channel.ChannelRequestStatsHandler.channelRead (ChannelRequestStatsHandler.scala:42)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead (MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead (CombinedChannelDuplexHandler.java:438)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead (ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead (ByteToMessageDecoder.java:267)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead (CombinedChannelDuplexHandler.java:253)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at com.twitter.finagle.buoyant.h2.netty4.ServerUpgradeHandler.channelRead (ServerUpgradeHandler.scala:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at com.twitter.finagle.netty4.channel.AnyToHeapInboundHandler$.channelRead (AnyToHeapInboundHandler.scala:36)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead (ChannelInboundHandlerAdapter.java:86)
at com.twitter.finagle.netty4.channel.ChannelStatsHandler.channelRead (ChannelStatsHandler.scala:92)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead (AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead (DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead (DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read (AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey (NioEventLoop.java:624)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized (NioEventLoop.java:559)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys (NioEventLoop.java:476)
at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:438)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run (SingleThreadEventExecutor.java:858)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run (BlockingTimeTrackingThreadFactory.scala:24)
at java.lang.Thread.run (Thread.java:745)
The services we are running are mostly implemented in go and therefore try to use http/2. However they should fall back to http/1.1 in this case, since the configured router in linkerd is configured as http router, not h/2.
Honestly I have no idea where to start looking for the root cause of this issue. We have about 10-20 deployments of services per day, so a lot of changes, but we haven't touched the linkerd config in weeks.
I can provide further information if I know what information will be useful.
Thanks in advance.