[Bug] ShutdownHook was not called when SparkSQLEngine exited gracefully #4280
Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I have searched in the issues and found no similar issues.
Describe the bug
SparkSQLEngine did not call ShutdownHook when it was launched in connection share level and stopped gracefully when session was closed.
This problem appeared in SparkSQLEngine launched in spark deploy mode local and yarn-client. Other spark deploy mode has not been checked.
Affects Version(s)
1.6.0/1.6.1
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLSessionManager: Session stopped due to shared level is Connection.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
23/02/09 11:56:51 WARN [main-EventThread] ZookeeperDiscoveryClient: This Kyuubi instance 10.221.99.69:50439 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] EngineServiceDiscovery: Clean up discovery service due to this is connection share level.
23/02/09 11:56:51 INFO [Curator-Framework-0] CuratorFrameworkImpl: backgroundOperationsLoop exiting
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] ZooKeeper: Session: 0x10004e31d840004 closed
23/02/09 11:56:51 INFO [main-EventThread] ClientCnxn: EventThread shut down for session: 0x10004e31d840004
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLEngine: Service: [SparkSQLBackendService] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLBackendService: Service[SparkSQLBackendService] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkSQLEngine: Service[SparkSQLEngine] is stopped.
23/02/09 11:56:51 INFO [SparkTBinaryFrontendHandler-Pool: Thread-79] SparkTBinaryFrontendService: Finished closing SessionHandle [b98cec81-6444-4803-8431-7dd6dc56a4e5]
23/02/09 11:56:51 INFO [main] SparkUI: Stopped Spark web UI at http://10.221.99.69:50428Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
If SparkSQLEngine was killed by command, or was running in other share level, ShutdownHook was called.
23/02/09 11:58:26 ERROR [SIGTERM handler] SparkSQLEngine: RECEIVED SIGNAL 15: TERM
23/02/09 11:58:26 INFO [Thread-1] SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
23/02/09 11:58:26 INFO [Thread-1] EngineServiceDiscovery: Clean up discovery service due to this is connection share level.
23/02/09 11:58:26 WARN [main-EventThread] ZookeeperDiscoveryClient: This Kyuubi instance 10.221.99.69:50499 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/02/09 11:58:26 INFO [Curator-Framework-0] CuratorFrameworkImpl: backgroundOperationsLoop exiting
23/02/09 11:58:26 INFO [main-EventThread] EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
23/02/09 11:58:26 INFO [Thread-1] ZooKeeper: Session: 0x10004e31d840006 closed
23/02/09 11:58:26 INFO [Thread-1] EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
23/02/09 11:58:26 INFO [Thread-1] SparkSQLEngine: Service: [SparkSQLBackendService] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLBackendService: Service[SparkSQLBackendService] is stopped.
23/02/09 11:58:26 INFO [Thread-1] SparkSQLEngine: Service[SparkSQLEngine] is stopped.
23/02/09 11:58:26 INFO [Thread-1] DiskBlockManager: Shutdown hook called
23/02/09 11:58:26 INFO [main] SparkUI: Stopped Spark web UI at http://10.221.99.69:50487
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Shutdown hook called
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Deleting directory /private/var/folders/yj/25xqj6_52n51xmctftgl_77c0000gn/T/spark-92945c05-f7e3-4fae-86fe-f1a409f141fa
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Deleting directory /private/var/folders/yj/25xqj6_52n51xmctftgl_77c0000gn/T/spark-dd00d8b3-d228-4750-83dd-191059e842a7
23/02/09 11:58:26 INFO [Thread-1] ShutdownHookManager: Deleting directory /private/var/folders/yj/25xqj6_52n51xmctftgl_77c0000gn/T/spark-92945c05-f7e3-4fae-86fe-f1a409f141fa/userFiles-676192b3-9da0-4bd0-8814-84d82abd1ad1
23/02/09 11:58:26 INFO [dispatcher-event-loop-3] MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
Are you willing to submit PR?
- Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- No. I cannot submit a PR at this time.