Errors running multiple independent ScriptingContainers in parallel #6218
Description
jruby 9.2.11.1 using the jruby-complete-9.2.11.1.jar
% java -version
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.4+11)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.4+11, mixed mode)
% uname -a
Linux localhost 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
% lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.6.1810 (Core)
Release: 7.6.1810
Codename: Core
Our environment is ScriptingContainer only. We had a ruby on rails web app that was ported to pure java but we have ~100 ERB templates and we use embedded jruby to evaluate them. In production, we have basically very few instances and threads using a ScriptingContainer. However, our unit tests use gradle and many tests touch the ERB pipeline and start ScirptingContainers.
We are finally updating to java11 from java8 and that has forced us to upgrade from jruby-1.6.7 (from 2011!) to the latest jruby (jruby-9.2.11.1). We have been using the same embedded architecture for 3-4 years and never had this problem with jruby-1.6.7.
Our environment is Linux and Windows and we see the same error on both platforms. In fact, probably more on Windows than Linux. I'm showing this error with CentOS 7.6 and java11 but it also repros with other versions CentOS and Windows and on java8.
This multi-threaded issue isn't super common in our CI/CD environment but I crafted a test case to make it common. I start X ScriptingContainers on Y threads where X=500 and Y=40. Again, not something we'd ever do, but it shows the behavior easily. Each ScriptingContainer is only accessed by the single thread that created it. Here is the java test case:
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.Executors;
import org.jruby.embed.LocalContextScope;
import org.jruby.embed.LocalVariableBehavior;
import org.jruby.embed.ScriptingContainer;
import org.junit.Test;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.common.util.concurrent.ListeningExecutorService;
import com.google.common.util.concurrent.MoreExecutors;
public class RubyTests {
@Test
public void testParallelInitizations() throws Exception {
final int howManyThreads = 40;
final int howManyRubies = 500;
ListeningExecutorService executor = MoreExecutors.listeningDecorator(
Executors.newFixedThreadPool(howManyThreads));
System.err.println("starting");
try {
Collection<Callable<Void>> threadFuncs = new ArrayList<>();
for(int ii=0; ii< howManyRubies; ++ii) {
threadFuncs.add(() -> {
ScriptingContainer s = new ScriptingContainer(LocalContextScope.SINGLETHREAD,
LocalVariableBehavior.PERSISTENT);
s.runScriptlet("require 'erb'");
s.terminate();
return null;
});
}
@SuppressWarnings({ "unchecked", "rawtypes" })
List<ListenableFuture<Void>> futures = (List) executor.invokeAll(threadFuncs);
Futures.allAsList(futures).get();
} finally {
executor.shutdownNow();
}
}
}
I think it should be fine to do what is done here. All the errors are related to reading the contents of the jruby-complete jar. I believe if we used the jruby.jar and stuck the contents of jruby.home on disk and set -Djruby.home to point there, it would also solve the problem. I can additionally solve the problem by synchronizing the contents of each thread with a global static lock.
Attached is a log file showing 18 errors among the 500 rubies. We normally see a MUCH lower incidence of this. It's probably because we are not using that many in any JVM. Calling terminate() exacerbates the problem. I think that's because if you don't call terminate(), then GC will do it for you and that doesn't happen that often.
Here are some example stack traces from the above file:
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- erb
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974)
at RUBY.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54)
at RUBY.<main>(<script>:1)
org.jruby.exceptions.NameError: (NameError) uninitialized constant Gem::Dependency
at org.jruby.RubyModule.const_missing(org/jruby/RubyModule.java:3760)
at RUBY.gem(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_gem.rb:48)
at RUBY.<main>(uri:classloader:/jruby/kernel/gem_prelude.rb:8)
at org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1009)
at RUBY.<main>(file:/home/user/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/9.2.11.1/78aa284f9b011173dc2b72bc1c8593a3606aacf3/jruby-complete-9.2.11.1.jar!/jruby/preludes.rb:4)
org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- erb
at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:131)
at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1295)
at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1288)
at com.megacorp.RubyTest.lambda$testParallelInitizations$0(RubyTests.java:208)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
org.jruby.exceptions.SystemCallError: (SystemCallError) Unknown error (SystemCallError) - uri:classloader://META-INF/jruby.home/lib/ruby/gems/shared/specifications/did_you_mean-1.2.0.gemspec
at org.jruby.RubyIO.read(org/jruby/RubyIO.java:3061)
at org.jruby.RubyIO.read(org/jruby/RubyIO.java:3049)
at org.jruby.RubyIO.read(org/jruby/RubyIO.java:3781)
at RUBY.load(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/specification.rb:1158)
at RUBY.to_spec(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/stub_specification.rb:192)
at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2577)
at RUBY.matching_specs(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/dependency.rb:280)
at RUBY.to_specs(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/dependency.rb:303)
at RUBY.to_spec(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/dependency.rb:323)
at RUBY.gem(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_gem.rb:65)
at RUBY.<main>(uri:classloader:/jruby/kernel/gem_prelude.rb:8)
at org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1009)
at RUBY.<main>(file:/home/user/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/9.2.11.1/78aa284f9b011173dc2b72bc1c8593a3606aacf3/jruby-complete-9.2.11.1.jar!/jruby/preludes.rb:4)