Add some compute Opcode benchmarks #9069

siladu · 2025-08-12T13:21:16Z

Heavily based on @daniellehrner's PR #8917

Allow excludes option, e.g. -Pexcludes=TransientStorage,BlockHash

Also gixes includes option to work with comma-separated list, e.g. ./gradlew :ethereum:core:jmh -Pexcludes=Add,Mul,SMod,baseline -Pincludes=Mod,SDiv --rerun-tasks

Remove noisy logging during jmhJar task

m6a.2xlarge:

./gradlew --no-daemon :ethereum:core:jmh -Pexcludes=TransientStorage,BlockHash,CountLeadingZeros,Keccak

Benchmark                                      Mode  Cnt     Score     Error  Units
AddModOperationBenchmark.baseline              avgt   15    16.026 ±   0.065  ns/op
AddModOperationBenchmark.executeOperation      avgt   15   440.741 ±  16.199  ns/op
AddOperationBenchmark.baseline                 avgt   15    11.474 ±   0.025  ns/op
AddOperationBenchmark.executeOperation         avgt   15   157.531 ±   1.428  ns/op
DivOperationBenchmark.baseline                 avgt   15    11.649 ±   0.725  ns/op
DivOperationBenchmark.executeOperation         avgt   15   186.930 ±   3.952  ns/op
EqOperationBenchmark.baseline                  avgt   15    11.435 ±   0.015  ns/op
EqOperationBenchmark.executeOperation          avgt   15    22.425 ±   0.132  ns/op
ExpOperationBenchmark.baseline                 avgt   15    11.527 ±   0.105  ns/op
ExpOperationBenchmark.executeOperation         avgt   15  6277.520 ± 121.047  ns/op
GtOperationBenchmark.baseline                  avgt   15    11.472 ±   0.026  ns/op
GtOperationBenchmark.executeOperation          avgt   15    29.215 ±   0.298  ns/op
LtOperationBenchmark.baseline                  avgt   15    11.969 ±   0.784  ns/op
LtOperationBenchmark.executeOperation          avgt   15    29.089 ±   0.075  ns/op
ModOperationBenchmark.baseline                 avgt   15    11.547 ±   0.161  ns/op
ModOperationBenchmark.executeOperation         avgt   15   290.468 ±   9.521  ns/op
MulModOperationBenchmark.baseline              avgt   15    15.972 ±   0.011  ns/op
MulModOperationBenchmark.executeOperation      avgt   15   533.690 ±  12.484  ns/op
MulOperationBenchmark.baseline                 avgt   15    11.461 ±   0.027  ns/op
MulOperationBenchmark.executeOperation         avgt   15   189.798 ±   4.696  ns/op
SLtOperationBenchmark.baseline                 avgt   15    11.468 ±   0.032  ns/op
SLtOperationBenchmark.executeOperation         avgt   15   102.967 ±   2.811  ns/op
SModOperationBenchmark.baseline                avgt   15    11.570 ±   0.179  ns/op
SModOperationBenchmark.executeOperation        avgt   15   292.092 ±  10.313  ns/op
SignExtendOperationBenchmark.baseline          avgt   15    11.466 ±   0.022  ns/op
SignExtendOperationBenchmark.executeOperation  avgt   15    48.605 ±   1.681  ns/op
SubOperationBenchmark.baseline                 avgt   15    11.451 ±   0.012  ns/op
SubOperationBenchmark.executeOperation         avgt   15   176.670 ±   6.154  ns/op

Allow excludes option, e.g. -Pexcludes=TransientStorage,BlockHash Remove noisy logging during jmhJar task Signed-off-by: Simon Dudley <[email protected]>

lu-pinto · 2025-08-21T14:23:44Z

build.gradle

      jmhVersion = '1.37'
      fork = 3
      includes = _strListCmdArg('includes', [''])
+      excludes = _strListCmdArg('excludes', [])


good addition!

lu-pinto · 2025-08-21T15:29:20Z

build.gradle

        ]
      }
-      duplicateClassesStrategy = DuplicatesStrategy.WARN
+      duplicateClassesStrategy = DuplicatesStrategy.EXCLUDE


is it safe to do this? I can't see why would we end up with duplicate entries in the classpath for a clean project. I think it might be inheriting from our normal classpath? Some cleanup might be needed, maybe it's worth opening a ticket.
I tried adding includeTests = false which isolates jmh and test and got a bunch of missing dependencies, so it's not as easy as that.

Good callout. I played around with this a bit but doing this properly would be quite involved and involve the shadowJar plugin.
I'm not 100% sure it's safe to leave as EXCLUDE: seems like main issue could be if there were multiple implementations of service files (META-INF/services) that got excluded and so we used the wrong class at runtime. Seems unlikely to happen in the EVM code but benchmarks aren't limited to just that.

I think we can change to INCLUDE as it's the same behaviour as WARN but without the logs... we're not acting on the warnings, so don't think there's an issue with this?

dcd032c (#9069)

Maybe @fab-10 would know more. I think even though it's repeated, by running ./gradlew :ethereum:core:dependencies --configuration jmhRuntimeClasspath I see that the duplicates are of the same version. Anyway, no need to block the PR for this, we can do either way - I agree it generates too much output.

ethereum/core/src/jmh/java/org/hyperledger/besu/ethereum/vm/operations/BenchmarkHelper.java

.../core/src/jmh/java/org/hyperledger/besu/ethereum/vm/operations/BinaryOperationBenchmark.java

lu-pinto · 2025-08-21T16:11:05Z

.../core/src/jmh/java/org/hyperledger/besu/ethereum/vm/operations/BinaryOperationBenchmark.java

+    frame.pushStackItem(bPool[i]);
+    frame.pushStackItem(aPool[i]);
+
+    blackhole.consume(invoke(frame));


nit: thought compilation blackholes were enabled by default?

I think it is but not 100% sure this isn't still needed?
e.g. this isn't used in place of a return at the end of the benchmark.

It might be "belts and braces" but I'd favour this explicit approach rather than accidentally eliminating code. Doesn't the overhead of this call get removed either way?

I think you will have overhead if you leave the call, though very small, from what I can infer from this article from Shipilev

I'll leave it in for now since these benchmarks aren't tiny (yet!) but would like to confirm this one way or the other.

...eum/core/src/jmh/java/org/hyperledger/besu/ethereum/vm/operations/ExpOperationBenchmark.java

...core/src/jmh/java/org/hyperledger/besu/ethereum/vm/operations/TernaryOperationBenchmark.java

lu-pinto

Looks fine just a couple of comments to address

Signed-off-by: Luis Pinto <[email protected]>

Signed-off-by: Simon Dudley <[email protected]>

siladu · 2025-08-26T10:26:15Z

Cranked up the warmup and iterations to 5x 2 seconds to check this PR's warmup was sufficient (compared to run in desc)...

Benchmark                                      Mode  Cnt     Score     Error  Units
AddModOperationBenchmark.baseline              avgt   15    15.946 ±   0.047  ns/op
AddModOperationBenchmark.executeOperation      avgt   15   433.681 ±   4.554  ns/op
AddOperationBenchmark.baseline                 avgt   15    11.174 ±   0.036  ns/op
AddOperationBenchmark.executeOperation         avgt   15   158.972 ±   0.782  ns/op
DivOperationBenchmark.baseline                 avgt   15    11.146 ±   0.037  ns/op
DivOperationBenchmark.executeOperation         avgt   15   185.480 ±   0.749  ns/op
EqOperationBenchmark.baseline                  avgt   15    11.200 ±   0.024  ns/op
EqOperationBenchmark.executeOperation          avgt   15    23.099 ±   0.222  ns/op
ExpOperationBenchmark.baseline                 avgt   15    11.063 ±   0.036  ns/op
ExpOperationBenchmark.executeOperation         avgt   15  6369.370 ± 118.572  ns/op
GtOperationBenchmark.baseline                  avgt   15    11.165 ±   0.052  ns/op
GtOperationBenchmark.executeOperation          avgt   15    29.343 ±   0.180  ns/op
LtOperationBenchmark.baseline                  avgt   15    11.431 ±   0.365  ns/op
LtOperationBenchmark.executeOperation          avgt   15    32.357 ±   4.570  ns/op
ModOperationBenchmark.baseline                 avgt   15    11.201 ±   0.046  ns/op
ModOperationBenchmark.executeOperation         avgt   15   289.706 ±   5.090  ns/op
MulModOperationBenchmark.baseline              avgt   15    15.906 ±   0.042  ns/op
MulModOperationBenchmark.executeOperation      avgt   15   534.985 ±   4.255  ns/op
MulOperationBenchmark.baseline                 avgt   15    11.258 ±   0.212  ns/op
MulOperationBenchmark.executeOperation         avgt   15   190.524 ±   0.513  ns/op
SDivOperationBenchmark.baseline                avgt   15    11.174 ±   0.023  ns/op
SDivOperationBenchmark.executeOperation        avgt   15   204.226 ±   3.263  ns/op
SLtOperationBenchmark.baseline                 avgt   15    11.150 ±   0.027  ns/op
SLtOperationBenchmark.executeOperation         avgt   15   101.700 ±   1.478  ns/op
SModOperationBenchmark.baseline                avgt   15    11.442 ±   0.406  ns/op
SModOperationBenchmark.executeOperation        avgt   15   289.852 ±   3.517  ns/op
SignExtendOperationBenchmark.baseline          avgt   15    11.187 ±   0.053  ns/op
SignExtendOperationBenchmark.executeOperation  avgt   15    47.911 ±   1.069  ns/op
SubOperationBenchmark.baseline                 avgt   15    11.400 ±   0.280  ns/op
SubOperationBenchmark.executeOperation         avgt   15   171.094 ±   1.666  ns/op

Perhaps there's some reduction in a few error margins, but nothing signficant worth increasing warmup for IMO.

Signed-off-by: Simon Dudley <[email protected]>

lu-pinto · 2025-08-26T10:42:40Z

Cranked up the warmup and iterations to 5x 2 seconds to check this PR's warmup was sufficent...

Also think it should be enough at 1sec you can run 157010 iterations for the worst case which is EXP. It's well above the compilation tier thresholds

Signed-off-by: Simon Dudley <[email protected]>

Adds an excludes option, e.g. -Pexcludes=TransientStorage,BlockHash and also includes option now works with comma-separated list, e.g. ./gradlew :ethereum:core:jmh -Pexcludes=Add,Mul,SMod,baseline -Pincludes=Mod,SDiv --rerun-tasks Remove noisy logging during jmhJar task Signed-off-by: Simon Dudley <[email protected]> * add SDiv benchmark Signed-off-by: Luis Pinto <[email protected]> --------- Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Luis Pinto <[email protected]> Co-authored-by: Luis Pinto <[email protected]>

ahamlat · 2025-08-28T08:00:06Z

build.gradle

      description = "Usage: gradle jmh -Pincludes=MyBench -PasyncProfiler=<libPath> -PasyncProfilerOptions=<options>\n" +
        "\nRun JMH benchmarks in each of the projects. Allows for controlling JMH execution directly from the command line.\n" +
        "\t-Pincludes=<includePattern>\tInclude pattern (regular expression) for benchmarks to be executed. Defaults to including all benchmarks.\n" +
+        "\t-Pexcludes=<excludePattern>\tComma-separated exclude pattern (regular expression) for benchmarks to not be executed. Defaults to including all benchmarks.\n" +


Adds an excludes option, e.g. -Pexcludes=TransientStorage,BlockHash and also includes option now works with comma-separated list, e.g. ./gradlew :ethereum:core:jmh -Pexcludes=Add,Mul,SMod,baseline -Pincludes=Mod,SDiv --rerun-tasks Remove noisy logging during jmhJar task Signed-off-by: Simon Dudley <[email protected]> * add SDiv benchmark Signed-off-by: Luis Pinto <[email protected]> --------- Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Luis Pinto <[email protected]> Co-authored-by: Luis Pinto <[email protected]>

Adds an excludes option, e.g. -Pexcludes=TransientStorage,BlockHash and also includes option now works with comma-separated list, e.g. ./gradlew :ethereum:core:jmh -Pexcludes=Add,Mul,SMod,baseline -Pincludes=Mod,SDiv --rerun-tasks Remove noisy logging during jmhJar task Signed-off-by: Simon Dudley <[email protected]> * add SDiv benchmark Signed-off-by: Luis Pinto <[email protected]> --------- Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Luis Pinto <[email protected]> Co-authored-by: Luis Pinto <[email protected]> Signed-off-by: jflo <[email protected]>

Adds an excludes option, e.g. -Pexcludes=TransientStorage,BlockHash and also includes option now works with comma-separated list, e.g. ./gradlew :ethereum:core:jmh -Pexcludes=Add,Mul,SMod,baseline -Pincludes=Mod,SDiv --rerun-tasks Remove noisy logging during jmhJar task Signed-off-by: Simon Dudley <[email protected]> * add SDiv benchmark Signed-off-by: Luis Pinto <[email protected]> --------- Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Luis Pinto <[email protected]> Co-authored-by: Luis Pinto <[email protected]> Signed-off-by: georgereuben <[email protected]>

Add some compute Opcode benchmarks

145ccdb

Allow excludes option, e.g. -Pexcludes=TransientStorage,BlockHash Remove noisy logging during jmhJar task Signed-off-by: Simon Dudley <[email protected]>