In jruby/jruby#8920 a user reported a bug in JRuby's keyword arguments handling that only takes effect after a call site is traversed at least two times. This is due to our current invokedynamic-based call sites performing a slower unoptimized call during the first invocation, only switching to the optimized inlined call logic for subsequent calls. Similar behavior may be found even outside of invokedynamic logic, such as simple call sites that may do lookup only on the first invocation (caching thereafter), and potentially also in other Ruby implementations that behave differently during initial calls at a given call site compared to subsequent calls.

Testing for these "n + 1" bugs would require either writing specs specifically to make calls twice (using the same call site), or modifying mspec itself to allow for re-running the same specs repeatedly using the same call sites without re-compiling. I use the former pattern in a spec added to jruby/jruby#8922 but this solution does not scale. I believe mspec needs a way to re-run the same loaded specs repeatedly.

Given the various forms of test setup and state management in mspec, I'm not sure if this can be done in a generic way. All specs should be free-standing and capable of being re-run repeatedly without reloading their spec file, but there's bound to be specs that make changes to global state expecting those changes to only run once. Such specs should be fixed, of course, but this may stand in the way of an easy mspec enhancement to run specs N times.

All three currently-maintained implementations have JIT optimizations that are deferred until later, and at least JRuby and TruffleRuby both have profiled optimizations that may take many iterations to trigger. Such optimized cases are poorly exercised by current test and spec harnesses that only run code once.

I would like to hear from other implementations about how they are testing optimizations that may take two or more invocations at a given call site to trigger.