Correctly shutdown current active LoggerConfigurator on app (re)load in DEV mode #10939

mkurz · 2021-07-26T19:06:53Z

(IMHO this PR fixes a potential memory leak in dev mode)

Before I come to the problem this PR fixes, let's first have a look at how, in a default Play app, the LoggerConfigurator gets set up:

When an app gets build via Guice, at some point the injector gets created. For this the applicationModule() is needed:

playframework/core/play-guice/src/main/scala/play/api/inject/guice/GuiceInjectorBuilder.scala

Lines 193 to 200 in 44886b0

    
           def injector(): PlayInjector = { 
        
             try { 
        
               val stage = environment.mode match { 
        
                 case Mode.Prod    => Stage.PRODUCTION 
        
                 case _ if eagerly => Stage.PRODUCTION 
        
                 case _            => Stage.DEVELOPMENT 
        
               } 
        
               val guiceInjector = Guice.createInjector(stage, applicationModule())

The applicationModule() method now configures the LoggerFactory:

playframework/core/play-guice/src/main/scala/play/api/inject/guice/GuiceApplicationBuilder.scala

Lines 108 to 112 in 44886b0

    
           override def applicationModule(): GuiceModule = { 
        
             val initialConfiguration = loadConfiguration(environment) 
        
             val appConfiguration     = configuration.withFallback(initialConfiguration) 
        
             val loggerFactory = configureLoggerFactory(appConfiguration)

And this is done by "configuring" the LoggerConfigurator (lc.configure(...)):

playframework/core/play-guice/src/main/scala/play/api/inject/guice/GuiceApplicationBuilder.scala

Lines 134 to 140 in 44886b0

    
           def configureLoggerFactory(configuration: Configuration): ILoggerFactory = { 
        
             val loggerFactory: ILoggerFactory = LoggerConfigurator(environment.classLoader) 
        
               .map { lc => 
        
                 lc.configure(environment, configuration, Map.empty) 
        
                 lc.loggerFactory 
        
               } 
        
               .getOrElse(org.slf4j.LoggerFactory.getILoggerFactory)

Even when not using Guice, we tell people they should call lc.configure(...) themselves. See here, here, here and here.

OK, so now we know that each time we create a new app, a LoggerConfigurator gets set up by calling lc.configure(...).

Now, let's have a look at PROD mode:

In prod mode, a single app gets created and a server (netty or akka-http) gets started afterwards. Now when shutting down the "application" (usually the os process), it means the http server and the app get shutdown together.
Now, when that shutdown happens, basically the last thing that happens is that also the LoggerConfigurator (that was configured during bootstrapping the app) gets shut down by the http server (at the point this happens in the below stop method the app was shut down already):

playframework/transport/server/play-server/src/main/scala/play/core/server/Server.scala

Lines 48 to 52 in 44886b0

    
           def stop(): Unit = { 
        
             applicationProvider.get.foreach { app => 
        
               LoggerConfigurator(app.classloader).foreach(_.shutdown()) 
        
             } 
        
           }

So great, the app and http-server get started, a LoggerConfigurator gets configured during bootstrapping the app, and when the whole thing gets shut down, the LoggerConfigurator will be shut down as well. On re-start, the same happens again. So no problem here.

But now, let's have a look at DEV mode:

Dev mode is different. In dev mode, unlike prod, an http server get's started first, and afterwards an app gets build, and usually (if you change a file), that app gets shut down, a new app will be build again, and so forth. Only when you quit dev mode, when you hit the return key on your keyboard, the current running app gets shut down and finally also the http server will be shut down as well.
The problems now are:

During this dev reload cycles, the current active LoggerConfigurator never gets shutdown. Even worse, every time a new app gets created, of course, a new LoggerConfigurator gets configured... Only at the very end, when you hit return and the http-server gets shutdown, the LoggerConfigurator of the last app finally gets shut down by the http-server's stop method (like described above in prod mode). Any LoggerConfigurators set up before however never cleaned up their resources...

Since the dev server starts an http-server before an app gets build, it sets up a special LoggerConfigurator before the first app gets build. And again, also that special LoggerConfigurator never gets shutdown... When the first app gets started, that first app then just goes ahead and configures its own LoggerConfigurator... The special LoggerConfigurator set up is done here:

playframework/transport/server/play-server/src/main/scala/play/core/server/DevServerStart.scala

Lines 150 to 159 in 44886b0

    
           // Configure the logger for the first time. 
        
           // This is usually done by Application itself when it's instantiated, which for other types of ApplicationProviders, 
        
           // is usually instantiated along with or before the provider.  But in dev mode, no application exists initially, so 
        
           // configure it here. 
        
           LoggerConfigurator(classLoader) match { 
        
             case Some(loggerConfigurator) => 
        
               loggerConfigurator.init(path, Mode.Dev) 
        
             case None => 
        
               println("No play.logger.configurator found: logging must be configured entirely by the application.") 
        
           }

So you see, in dev mode the current active LoggerConfigurator never cleanly gets shutdown, which could lead to memory leaks or other problems with the logging infrastructure.

And there is even one more unfortunate side effect because of this behaviour:
The LogbackLoggerConfigurator (which is the default) sets the application mode when it gets configured:

playframework/core/play-logback/src/main/scala/play/api/libs/logback/LogbackLoggerConfigurator.scala

Lines 71 to 72 in 44886b0

    
           // Set the global application mode for logging 
        
           play.api.Logger.setApplicationMode(env.mode)

and unsets it again on shut down:

playframework/core/play-logback/src/main/scala/play/api/libs/logback/LogbackLoggerConfigurator.scala

Lines 144 to 145 in 44886b0

    
           // Unset the global application mode for logging 
        
           play.api.Logger.unsetApplicationMode()

Now look what those methods do:

playframework/core/play/src/main/scala/play/api/Logger.scala

Lines 275 to 303 in 44886b0

    
             /** 
        
              * Set the global application mode used for logging. Used when the Play application starts. 
        
              */ 
        
             def setApplicationMode(mode: Mode): Unit = { 
        
               val appsRunning = _appsRunning.incrementAndGet() 
        
               applicationMode.foreach { currentMode => 
        
                 if (currentMode != mode) { 
        
                   log.warn(s"Setting logging mode to $mode when it was previously set to $currentMode") 
        
                   log.warn(s"There are currently $appsRunning applications running.") 
        
                 } 
        
               } 
        
               _mode = Some(mode) 
        
             } 
        
             /** 
        
              * Unset the global application mode. Used when the application shuts down. 
        
              * 
        
              * If multiple applications are running 
        
              */ 
        
             def unsetApplicationMode(): Unit = { 
        
               val appsRunning = _appsRunning.decrementAndGet() 
        
               if (appsRunning == 0) { 
        
                 _mode = None 
        
               } else if (appsRunning < 0) { 
        
                 log.warn("Cannot unset application mode because none was previously set") 
        
                 _mode = None 
        
                 _appsRunning.incrementAndGet() 
        
               } 
        
             }

If in dev mode, when many LoggerConfigurators will be set up (but never shut down) because of reload cycles, the _appsRunning counter in this code increases more and more, but never decreases anymore. Actually, in normal dev mode there is always just one running app at a time, even though different apps run one after another, but here the _appsRunning counter just increases. That also means, that when the http-server gets shut down and finally the unsetApplicationMode method gets called once, the counter is way to high, and _mode will never be set to None, although there is no app running anymore. Also in between reload cycles, when briefly no app is running, _mode will not be set to None.

With this PR all of the above problems are fixed. I did a lot of testing by e.g. adding breakpoints to these [unset]setApplicationMode methods and also to the shutdown, init and configure methods of the LogbackLoggerConfigurator and can confirm that now there is always just one active LoggerConfigurator that cleanly gets shut down before the next LoggerConfigurator gets configured. Also the app counter increases to 1 and gets set back to 0 when an app shuts down and then again 1 -> 0 -> 1 -> 0 ,etc. which means also _mode gets set to None correctly when no app is running.

Just before an app gets build the first time

Just before the app that will be build configures its own

mkurz · 2021-07-26T19:10:48Z

core/play-logback/src/main/scala/play/api/libs/logback/LogbackLoggerConfigurator.scala

   * Initialize the Logger when there's no application ClassLoader available.
   */
  def init(rootPath: java.io.File, mode: Mode): Unit = {
+    initializedWithoutApp = true


As written in the javadocs, the init method gets called when there is no app - like that is the case in DevServer before an app gets loaded. Therefore that init method does not call the setApplicationMode method.

mkurz · 2021-07-26T19:12:22Z

core/play-logback/src/main/scala/play/api/libs/logback/LogbackLoggerConfigurator.scala

+    if (!initializedWithoutApp) {
+      // Unset the global application mode for logging
+      play.api.Logger.unsetApplicationMode()
+    }


This if check is needed, because when init was called, it means there was no app and therefore no app mode was set, therefore we now also don't want to unset an app mode (otherwise the _appsRunning counter would become negative)

mkurz · 2021-07-26T19:13:34Z

@Mergifyio backport 2.8.x

mergify · 2021-07-26T19:14:11Z

backport 2.8.x

☑️ Nothing to do

Details

merged [📌 backport requirement]

mkurz · 2021-07-26T19:16:15Z

transport/server/play-server/src/main/scala/play/core/server/DevServerStart.scala

+                // to shut down that old app's LoggerConfigurator, just before the app that will be build configures its own
+                lastState.foreach(app => LoggerConfigurator(app.classloader).foreach(lc => lc.shutdown()))
+                // FYI: initialLoggerConfigurator and lastState will never both be set at the same time, so in the lines above at most only one shutdown() gets called
+


I put this code just before loader.load(...) builds the new application.

mkurz · 2021-07-28T09:24:02Z

One more thing: Would it also make sense to shut down the LoggerConfigurator in the various test helpers Plays provides? E.g.
here, after the app was stopped?

playframework/testkit/play-test/src/main/java/play/test/WithApplication.java

Lines 52 to 58 in 44886b0

    
           @After 
        
           public void stopPlay() { 
        
             if (app != null) { 
        
               Helpers.stop(app); 
        
               app = null; 
        
             } 
        
           }

marcospereira

Some comments. :-)

marcospereira · 2021-08-04T16:21:21Z

core/play-logback/src/main/scala/play/api/libs/logback/LogbackLoggerConfigurator.scala

+  // _before_ an app gets build. In that case the init(...) method below gets called. We track that call with this boolean flag
+  // to avoid unsetting the app mode when the LoggerConfigurator gets shut down.
+  // Also see play.core.server.DevServerStart (where loggerConfigurator.init(...) gets called)
+  private var initializedWithoutApp = false


Hum, it looks like to me that since this is a special case for dev mode, this state should not be internal to the LoggerConfigurator, but instead, it should be handled there, in DevServerStart. A good reason not to track state here is that it is possible to have a LoggerConfigurator that is not LogbackLoggerConfigurator.

marcospereira · 2021-08-04T16:29:10Z

transport/server/play-server/src/main/scala/play/core/server/DevServerStart.scala

+
+                // Before building a new app _the first time_, we make sure to shutdown the (Logback)LoggerConfigurator created initally by the dev server above,
+                // because the app that will be build configures its own LoggerConfigurator later (but then by using the app's classloader)
+                initialLoggerConfigurator.foreach(_.shutdown())
+                initialLoggerConfigurator = None
+                // However, if we build an app _not the first time_, but there was an app running before (= lastState is success), we make sure
+                // to shut down that old app's LoggerConfigurator, just before the app that will be build configures its own
+                lastState.foreach(app => LoggerConfigurator(app.classloader).foreach(lc => lc.shutdown()))
+                // FYI: initialLoggerConfigurator and lastState will never both be set at the same time, so in the lines above at most only one shutdown() gets called


I wonder if it would be better to make this all happen using newApplication's CoordinatedShutdown.

marcospereira · 2021-08-04T16:32:51Z

One more thing: Would it also make sense to shut down the LoggerConfigurator in the various test helpers Plays provides? E.g.
here, after the app was stopped?

This makes sense to me.

wsargent · 2021-08-09T17:18:30Z

It's a Logback best practice, even:

In order to release the resources used by logback-classic, it is always a good idea to stop the logback context. Stopping the context will close all appenders attached to loggers defined by the context and stop any active threads in an orderly way. Please also read the section on "shutdown hooks" just below.

http://logback.qos.ch/manual/configuration.html#stopContext

janani-reddy9 · 2024-01-30T03:54:52Z

Hello @mkurz, I see a lot of work done here and this might be probably stopped due to other priority things. I am thinking to work on this PR. Could you please let me know if it is ok to work on this?

mkurz · 2024-06-12T20:58:26Z

@janani-reddy9 sorry I must have missed your message totally. If you are still interested feel free to work in this issue.

mkurz · 2024-06-14T09:49:00Z

When looking into this issue here one should also check:

Maybe those two issues got resolved along the way, but not sure yet.

mkurz added 2 commits July 26, 2021 15:45

Shutdown the DevServer's initial LoggerConfigurator

e60b46f

Just before an app gets build the first time

On reload in DEV, shut down the old app's LoggerConfigurator

93d0c75

Just before the app that will be build configures its own

mkurz commented Jul 26, 2021

View reviewed changes

mkurz added this to the 2.8.9 milestone Jul 26, 2021

mkurz commented Jul 26, 2021

View reviewed changes

This was referenced Jul 28, 2021

[failed] java.lang.OutOfMemoryError: Metaspace when running sbt scripted test #10676

Closed

Memory leak in tests #5144

Open

marcospereira reviewed Aug 4, 2021

View reviewed changes

mkurz modified the milestones: 2.8.9, 2.9.0 Nov 24, 2021

mkurz modified the milestones: 2.9.0, Play.next Feb 11, 2022

This was referenced Nov 10, 2022

[2.8.x] Do not use/recommend shutdownHook in Logback, Play handles that #11532

Merged

NoClassDefFoundError: ch/qos/logback/core/LifeCycleManager playframework/play-ebean#310

Closed

mkurz mentioned this pull request Jun 1, 2023

Logback initialized twice when using custom application loader #10342

Open

mkurz modified the milestones: Play.next, 2.10.0 Jun 3, 2023

This was referenced Nov 13, 2023

DependencyInjectedRoutingDslTest and other tests are failed by LoggerContext #12204

Closed

Set shouldRunSequentially to false by default or just remove runSynchronized #12214

Closed

ihostage mentioned this pull request Mar 6, 2024

Gradle Plugin: Documentation of a few gotchas playframework/twirl#745

Closed

	def injector(): PlayInjector = {
	try {
	val stage = environment.mode match {
	case Mode.Prod => Stage.PRODUCTION
	case _ if eagerly => Stage.PRODUCTION
	case _ => Stage.DEVELOPMENT
	}
	val guiceInjector = Guice.createInjector(stage, applicationModule())

	override def applicationModule(): GuiceModule = {
	val initialConfiguration = loadConfiguration(environment)
	val appConfiguration = configuration.withFallback(initialConfiguration)

	val loggerFactory = configureLoggerFactory(appConfiguration)

	def configureLoggerFactory(configuration: Configuration): ILoggerFactory = {
	val loggerFactory: ILoggerFactory = LoggerConfigurator(environment.classLoader)
	.map { lc =>
	lc.configure(environment, configuration, Map.empty)
	lc.loggerFactory
	}
	.getOrElse(org.slf4j.LoggerFactory.getILoggerFactory)

	def stop(): Unit = {
	applicationProvider.get.foreach { app =>
	LoggerConfigurator(app.classloader).foreach(_.shutdown())
	}
	}

	// Configure the logger for the first time.
	// This is usually done by Application itself when it's instantiated, which for other types of ApplicationProviders,
	// is usually instantiated along with or before the provider. But in dev mode, no application exists initially, so
	// configure it here.
	LoggerConfigurator(classLoader) match {
	case Some(loggerConfigurator) =>
	loggerConfigurator.init(path, Mode.Dev)
	case None =>
	println("No play.logger.configurator found: logging must be configured entirely by the application.")
	}

	// Set the global application mode for logging
	play.api.Logger.setApplicationMode(env.mode)

	// Unset the global application mode for logging
	play.api.Logger.unsetApplicationMode()

	/**
	* Set the global application mode used for logging. Used when the Play application starts.
	*/
	def setApplicationMode(mode: Mode): Unit = {
	val appsRunning = _appsRunning.incrementAndGet()
	applicationMode.foreach { currentMode =>
	if (currentMode != mode) {
	log.warn(s"Setting logging mode to $mode when it was previously set to $currentMode")
	log.warn(s"There are currently $appsRunning applications running.")
	}
	}
	_mode = Some(mode)
	}

	/**
	* Unset the global application mode. Used when the application shuts down.
	*
	* If multiple applications are running
	*/
	def unsetApplicationMode(): Unit = {
	val appsRunning = _appsRunning.decrementAndGet()
	if (appsRunning == 0) {
	_mode = None
	} else if (appsRunning < 0) {
	log.warn("Cannot unset application mode because none was previously set")
	_mode = None
	_appsRunning.incrementAndGet()
	}
	}

Uh oh!

Correctly shutdown current active LoggerConfigurator on app (re)load in DEV mode #10939

Are you sure you want to change the base?

Correctly shutdown current active LoggerConfigurator on app (re)load in DEV mode #10939

Uh oh!

Conversation

mkurz commented Jul 26, 2021

Uh oh!

mkurz Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

mkurz Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

mkurz commented Jul 26, 2021

Uh oh!

mergify bot commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

☑️ Nothing to do

Uh oh!

mkurz Jul 26, 2021

Choose a reason for hiding this comment

Uh oh!

mkurz commented Jul 28, 2021

Uh oh!

marcospereira left a comment

Choose a reason for hiding this comment

Uh oh!

marcospereira Aug 4, 2021

Choose a reason for hiding this comment

Uh oh!

marcospereira Aug 4, 2021

Choose a reason for hiding this comment

Uh oh!

marcospereira commented Aug 4, 2021

Uh oh!

wsargent commented Aug 9, 2021

Uh oh!

janani-reddy9 commented Jan 30, 2024

Uh oh!

mkurz commented Jun 12, 2024

Uh oh!

mkurz commented Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mergify bot commented Jul 26, 2021 •

edited

Loading

mkurz commented Jun 14, 2024 •

edited

Loading