
jvican in /data/rw/code/experiments/compiler-benchmark                                  [19:28:12] 
> $ for SV in "2.12.5--bin-dbd90495c2-SNAPSHOT"; do /usr/bin/sbt "set scalaVersion in ThisBuild := \"$SV\"" "hot -psource=scalap -f1 -wi 10 -i 8 -vEXTRA" 2>&1; done

[info] Benchmark                                   (corpusVersion)  (extraArgs)  (resident)  (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest                    false    scalap  sample   87   975.417 ± 12.725  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest                    false    scalap  sample        918.553           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest                    false    scalap  sample        967.836           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest                    false    scalap  sample       1024.249           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest                    false    scalap  sample       1046.479           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest                    false    scalap  sample       1084.228           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest                    false    scalap  sample       1084.228           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest                    false    scalap  sample       1084.228           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest                    false    scalap  sample       1084.228           ms/op

jvican in /data/rw/code/experiments/compiler-benchmark                                                                                                                                            [19:19:47] 
> $ for SV in "2.12.5--bin-dbd90495c2-SNAPSHOT" "2.12.5-bin-f18e3c59fd-SNAPSHOT"; do /usr/bin/sbt "set scalaVersion in ThisBuild := \"$SV\"" "hot -psource=scalap -f1 -wi 10 -i 8 -pextraArgs=-Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar -vEXTRA" 2>&1; done

[info] Benchmark                                   (corpusVersion)                                                                                       (extraArgs)  (resident)  (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample   73  1137.892 ± 17.863  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1060.110           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1130.365           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1200.829           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1221.172           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1270.874           ms/op

[info] Benchmark                                   (corpusVersion)                                                                                       (extraArgs)  (resident)  (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample   86   985.844 ± 10.220  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample        937.427           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample        981.467           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1027.919           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1033.162           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest  -Xplugin:/home/jvican/.ivy2/cache/org.scalamacros/paradise_2.12.4/jars/paradise_2.12.4-2.1.0.jar       false    scalap  sample       1077.936           ms/op

Summary

Scenario	Scala version	Compilation time (ms)	Factor vs baseline
no plugin (baseline)	`2.12.5-bin-dbd90495c2-SNAPSHOT`	975
dynamically loaded	`2.12.5-bin-dbd90495c2-SNAPSHOT`	1137	1.17
dynamically loaded	`2.12.5-bin-f18e3c59fd-SNAPSHOT`	985	1.01

Where f18e3c59fd is the commit hash of
https://github.com/scalacenter/scala/tree/ticket/sd-458, and dbd90495c2 the
latest commit in the 2.12.5 branch to this date.

These results confirm that caching dynamically loaded compiler plugins works,
where the caching improvements behave as well as the baseline. Before the patch,
dynamically loading a compiler plugin slowed down a warmed up compiler by 17%.

Machine

cat /proc/cpuinfo:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping	: 3
microcode	: 0x88
cpu MHz		: 3213.470
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown
bogomips	: 5618.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping	: 3
microcode	: 0x88
cpu MHz		: 3225.341
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown
bogomips	: 5618.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

uname -a:

Linux tribox 4.14.13-1-ARCH #1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018 x86_64 GNU/Linux

Processor details

Hyper threading is disabled, NUMA is disabled, Turbo boost is disabled, cpu
frequency is set to 3.2GHz. intel_pstate is used.

Implications

Dynamically loaded compiler plugins are the norm, so more or less all builds
that use them will benefit from these speedups.

I'm working on collecting benchmarking results for macro libraries. I will
report them soon.

Summary

Machine

Processor details

Implications

Next

Investigate compiler performance problems related to dynamic classloading of compiler plugins #458

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions