[runtime] Default llvm to self-init+direct native calls #13312

alexanderkyte · 2019-03-05T19:42:53Z

In order to improve the generation of call sites on LLVM, this PR moves us away from using a PLT call and to doing direct calls of the methods. In order to make sure that our lazy init still works, we need to emit init checks into methods that need them.

This PR does that, and puts a disable option behind MONO_DEBUG.

mono/mini/mini-llvm.c

mono/mini/aot-compiler.c

mono/mini/mini-llvm.h

alexanderkyte · 2019-03-07T18:21:52Z

Gnyah
....................................................................................................................................................................System.ApplicationException
System.IO.IOException
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................False
..........................................................d:2
d:2
d:2
d:2
d:2
d:2
d:2
d:2
....................................................
Tests run: 8134, Passed: 8090, Errors: 0, Failures: 0, Inconclusive: 0
  Not run: 44, Invalid: 0, Ignored: 44, Skipped: 0

Locally this is passing the corlib tests while doing the direct call fixups for my synthetic benchmark.

NITED_BB2
  br label %BB3

BB3:                                              ; preds = %CONT_BB4
  br label %BB2

BB2:                                              ; preds = %BB3
  notail call monocc void @NoThrowInline_MainClass_ThrowIfNull_string(i64* %arg_a)
  notail call monocc void @NoThrowInline_MainClass_ThrowIfNull_string(i64* %arg_b)
  notail call monocc void @NoThrowInline_MainClass_ThrowIfNull_string(i64* %arg_c)
  notail call monocc void @NoThrowInline_MainClass_ThrowIfNull_string(i64* %arg_d)
  %3 = bitcast i64* %arg_a to i32*
  %4 = getelementptr i32, i32* %3, i32 4
  %t50 = load volatile i32, i32* %4
  %5 = bitcast i64* %arg_b to i32*

Exciting.

luhenry · 2019-03-07T19:18:15Z

mono/mini/mini-llvm.c

Why do we need that?

Feel free to discard the review whenever the question is answered, this is not requested changes but merely a question for documentation

I was seeing a lot of weird behavior in the corlib tests that suggested a few edge cases to consider. I think that enabling inlining for the majority of methods is a good idea, but that fitting that into this PR would make it hard to reason about.

In the past, we weren’t inlining any methods because we were obfuscating all of the call sites. This preserves that behavior.

Also not sure how letting LLVM do inlining impacts things like non-C++ EH. There are a number of correctness hurdles there.

Adding a comment to the source to that effect.

As for the reason to disable this on llvmonly, it's clear that this is currently working on llvmonly without it (because this is already the default on llvmonly). I didn't want to change any of the behavior in this PR besides the intended change. I'm pretty sure that things should build on bitcode without that conditional. It shouldn't result in any optimization losses, but I figured I'd wait until I took on inlining in general to look into that.

Comments added

alexanderkyte · 2019-03-07T21:17:50Z

Seeing failures that do not reproduce:

MONO_PATH=/Users/akyte/mono_llvm_opt/mcs/class/lib/testing_aot_full /Users/akyte/mono_llvm_opt/runtime/mono-wrapper --aot-path=/Users/akyte/mono_llvm_opt/mcs/class/lib/net_4_x --aot=llvm,full dim-sharedgenerics.exe
Mono Ahead of Time compiler - compiling assembly /Users/akyte/mono_llvm_opt/mono/tests/dim-sharedgenerics.exe
AOTID D003538B-CB51-89BA-EDBC-F0667A35061B
Executing opt: "opt" -f -O2 -disable-tail-calls -o "mono_aot_r2UzEl/temp.opt.bc" "mono_aot_r2UzEl/temp.bc"
Executing llc: "llc"  -march=x86-64 -mattr=sse4.1 -asm-verbose=false -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=_mono_aot_sharedgenerics_eh_frame -disable-tail-calls -no-x86-call-frame-opt -relocation-model=pic -filetype=obj -o "mono_aot_r2UzEl/temp-llvm.o" "mono_aot_r2UzEl/temp.opt.bc"
Compiled: 12/12
Executing the native assembler: "clang"  -c -x assembler -o /var/folders/00/v8q_733j1hd8br7nxhxs41d80000gn/T/mono_aot_rlQR8R.o /var/folders/00/v8q_733j1hd8br7nxhxs41d80000gn/T/mono_aot_rlQR8R
Executing the native linker: clang --shared -o /Users/akyte/mono_llvm_opt/mono/tests/dim-sharedgenerics.exe.dylib.tmp "mono_aot_r2UzEl/temp-llvm.o" /var/folders/00/v8q_733j1hd8br7nxhxs41d80000gn/T/mono_aot_rlQR8R.o 
Executing dsymutil: dsymutil "/Users/akyte/mono_llvm_opt/mono/tests/dim-sharedgenerics.exe.dylib"
JIT time: 9 ms, Generation time: 234 ms, Assembly+Link time: 292 ms.
akyte@Alexanders-MacBook-Pro-3 ~/mono_llvm_opt/mono/tests (init_transition*) $ MONO_PATH=/Users/akyte/mono_llvm_opt/mcs/class/lib/testing_aot_full /Users/akyte/mono_llvm_opt/runtime/mono-wrapper --aot-path=/Users/akyte/mono_llvm_opt/mcs/class/lib/net_4_x  dim-sharedgenerics.exe 
Calling IFoo<string>.Foo on FooBar<string, object> - expecting default method IFoo<string>.Foo
At IFoo.Foo:Arg=ABC, TypeOf(T)=System.String
PASS
Calling IBar<string[]>.Foo on FooBar<string, object> - expecting default method IBar<object>.Foo
At IBar.Bar:Arg=System.String[], TypeOf(T)=System.Object
PASS
akyte@Alexanders-MacBook-Pro-3 ~/mono_llvm_opt/mono/tests (init_transition*) $ MONO_PATH=/Users/akyte/mono_llvm_opt/mcs/class/lib/testing_aot_full /Users/akyte/mono_llvm_opt/runtime/mono-wrapper --aot-path=/Users/akyte/mono_llvm_opt/mcs/class/lib/net_4_x  dim-sharedgenerics.exe
Calling IFoo<string>.Foo on FooBar<string, object> - expecting default method IFoo<string>.Foo
At IFoo.Foo:Arg=ABC, TypeOf(T)=System.String
PASS
Calling IBar<string[]>.Foo on FooBar<string, object> - expecting default method IBar<object>.Foo
At IBar.Bar:Arg=System.String[], TypeOf(T)=System.Object
PASS
akyte@Alexanders-MacBook-Pro-3 ~/mono_llvm_opt/mono/tests (init_transition*) $ MONO_PATH=/Users/akyte/mono_llvm_opt/mcs/class/lib/testing_aot_full /Users/akyte/mono_llvm_opt/runtime/mono-wrapper --aot-path=/Users/akyte/mono_llvm_opt/mcs/class/lib/net_4_x  dim-sharedgenerics.exe
Calling IFoo<string>.Foo on FooBar<string, object> - expecting default method IFoo<string>.Foo
At IFoo.Foo:Arg=ABC, TypeOf(T)=System.String
PASS
Calling IBar<string[]>.Foo on FooBar<string, object> - expecting default method IBar<object>.Foo
At IBar.Bar:Arg=System.String[], TypeOf(T)=System.Object
PASS

alexanderkyte · 2019-03-07T21:19:24Z

Ah, they repro with --full-aot

alexanderkyte · 2019-03-07T22:35:47Z

Oh, interesting. These are all tests that don't run on bitcode.

alexanderkyte · 2019-03-07T22:36:08Z

MONO_PATH=/Users/akyte/mono_master_llvm_unmodified/mcs/class/lib/net_4_x-macos/ /Users/akyte/mono_master_llvm_unmodified/runtime/mono-wrapper --aot-path=/Users/akyte/mono_master_llvm_unmodified/mcs/class/lib/testing_aot_full --full-aot --llvmonly classinit.exe
* Assertion at driver.c:2856, condition `!inited' not met

alexanderkyte · 2019-03-14T16:45:25Z

Example:
https://gist.github.com/alexanderkyte/ad49291156bb5b378175a83573211a9e

Note:

init_methods referenced from method address fetching function
direct calls, rather than plt calls
Visibility changes to init_method functions

alexanderkyte · 2019-03-14T17:55:25Z

Hmm, since I am using the managed wrapper, maybe I should drop the nmethods change

mono/mini/aot-compiler.c

jaykrell · 2019-03-17T01:36:59Z

Very nice to be calling functions not via pointers.

lewurm · 2019-03-15T21:45:37Z

mono/metadata/marshal.h

does this comment add value?

If it seems obvious that the below field is used as a primary key, I can remove the comment. The data definition seemed pretty far away from the actual usage to me, hence the comment.

lewurm · 2019-03-17T20:31:11Z

mono/mini/mini-llvm.c

this comment seems to belong to LLVMSetInstructionCallConv (). move it there. the comment above belongs to set_call_preserveall_cc ()

lewurm · 2019-03-17T20:34:44Z

mono/mini/mini-runtime.h

the option name already contains a negation, so this is hard to read imho. could you rephrase it to if the flag is set, then [...]. if the flag isn't set then [...]? maybe use bullet points.

Github doesn't show this as out of date, but this one is

lewurm · 2019-03-17T20:35:37Z

mono/mini/llvmonly-runtime.c

how can this code be reached? shouldn't this method only be called if mono_llvm_only is set?

This method is going to be pretty hot code for all LLVM-using JIT modes.

Every single method will do a self-init through this function. If that init fails, we will throw here.

ok. that was not obvious to me because of the filename (llvmonly-runtime.c) and function name (init_llvmonly_method ()), but it makes sense due to the nature of this change. we should change that, but can be done later in order to avoid noise in this PR.

Yes, these should be moved somewhere else later, they were only used by llvmonly previously.

I renamed these functions. Let me know if you'd like for me to move them.

Now the init code is behind llvm_plt_init. Setting --aot=llvm-plt will bring back the legacy llvm+plt behavior

We emitted all of the lazy checks and init around a specific static method, but still were initializing it when we initialized methods that access it. This made static cctors run before they were supposed to. The problem was that the call site here has method == NULL, because we don't compute it. That's expected, as we don't compute the method along this path. This initial "&&" looks like it was supposed to avoid calling the method on a null MonoMethod reference. This preserves that, while fixing the control flow issues.

* [runtime] Default llvm to self-init Now the init code is behind llvm_plt_init. Setting --aot=llvm-plt will bring back the legacy llvm+plt behavior * [llvm] Mark managed methods no-inline by llvm * [runtime] Clean up self-disabling machinery * [llvm] Skip self-init on cctors * [runtime] Always make ctors call their cctors * [runtime] Improve mono_print_unwind_info, add cases * [runtime] Fix too-early eager calling of cctors We emitted all of the lazy checks and init around a specific static method, but still were initializing it when we initialized methods that access it. This made static cctors run before they were supposed to. The problem was that the call site here has method == NULL, because we don't compute it. That's expected, as we don't compute the method along this path. This initial "&&" looks like it was supposed to avoid calling the method on a null MonoMethod reference. This preserves that, while fixing the control flow issues. * [runtime] Associate init_method native code with MonoMethod * [runtime] Future-proof against aot init wrapper subtype mixups * [runtime] Fix goto in AOT-init change for c++ * [runtime] Move docs for llvm-disable-self-init into runtime * [runtime] Fix bitcode, remove unneeded wrapper around init icall * [llvm] Include global prefix in init name * [comment] Describe debugging usage of llvm self-init disable flag * [aot] Consolidate mangler string append lines * [llvm] Make self-init documentation changes * [llvm] Rename self-init funcs, mini_llvmonly_ to mini_llvm_ * [runtime] Enable inlining for methods without init

alexanderkyte · 2019-04-01T18:00:52Z

Perf differences for this are captured in new benchmark run this weekend: #13104 (comment)

lewurm · 2019-04-02T08:12:18Z

@alexanderkyte could you summarize the impact of this PR? it's not exactly obvious when looking at the raw benchmark results 🙂

alexanderkyte added do-not-merge epic: LLVM CodeGen Quality labels Mar 5, 2019

alexanderkyte requested a review from vargaz March 5, 2019 19:42

alexanderkyte requested review from lewurm and luhenry as code owners March 5, 2019 19:42

alexanderkyte force-pushed the init_transition branch 2 times, most recently from 999049c to 738fe65 Compare March 5, 2019 23:30

vargaz reviewed Mar 6, 2019

View reviewed changes

mono/mini/mini-llvm.c Outdated Show resolved Hide resolved

vargaz reviewed Mar 6, 2019

View reviewed changes

mono/mini/aot-compiler.c Outdated Show resolved Hide resolved

vargaz reviewed Mar 6, 2019

View reviewed changes

mono/mini/mini-llvm.h Outdated Show resolved Hide resolved

alexanderkyte force-pushed the init_transition branch from 738fe65 to ebfb035 Compare March 7, 2019 18:10

alexanderkyte requested a review from marek-safar as a code owner March 7, 2019 18:10

alexanderkyte removed the do-not-merge label Mar 7, 2019

alexanderkyte force-pushed the init_transition branch 2 times, most recently from 7f550ff to 7fc43a3 Compare March 7, 2019 18:33

luhenry previously requested changes Mar 7, 2019

View reviewed changes

alexanderkyte force-pushed the init_transition branch from 7fc43a3 to 90a2ca5 Compare March 7, 2019 19:27

alexanderkyte requested a review from lambdageek as a code owner March 14, 2019 16:33

alexanderkyte force-pushed the init_transition branch from c6390c5 to 121a556 Compare March 14, 2019 16:34

alexanderkyte changed the title ~~[runtime] Default llvm to self-init~~ [runtime] Default llvm to self-init+direct native calls Mar 14, 2019

alexanderkyte force-pushed the init_transition branch 2 times, most recently from d790d8f to 3113cc8 Compare March 14, 2019 17:04

vargaz approved these changes Mar 16, 2019

View reviewed changes

jaykrell reviewed Mar 17, 2019

View reviewed changes

mono/mini/aot-compiler.c Outdated Show resolved Hide resolved

lewurm reviewed Mar 17, 2019

View reviewed changes

alexanderkyte added 17 commits March 18, 2019 12:49

[runtime] Default llvm to self-init

8102fe8

Now the init code is behind llvm_plt_init. Setting --aot=llvm-plt will bring back the legacy llvm+plt behavior

[llvm] Mark managed methods no-inline by llvm

6f14275

[runtime] Clean up self-disabling machinery

0c4a048

[llvm] Skip self-init on cctors

480232b

[runtime] Always make ctors call their cctors

7415bd2

[runtime] Improve mono_print_unwind_info, add cases

7bd3353

[runtime] Associate init_method native code with MonoMethod

ca36ca0

[runtime] Fix goto in AOT-init change for c++

278d01d

[runtime] Future-proof against aot init wrapper subtype mixups

5489ebb

[runtime] Move docs for llvm-disable-self-init into runtime

9a8729d

[runtime] Fix bitcode, remove unneeded wrapper around init icall

297664a

[llvm] Include global prefix in init name

4726bbb

[comment] Describe debugging usage of llvm self-init disable flag

a2e9c74

[aot] Consolidate mangler string append lines

4c64986

[llvm] Make self-init documentation changes

60366a0

[llvm] Rename self-init funcs, mini_llvmonly_ to mini_llvm_

e90639c

alexanderkyte force-pushed the init_transition branch from b6d56fa to e90639c Compare March 18, 2019 17:48

alexanderkyte requested a review from thaystg as a code owner March 18, 2019 17:48

[runtime] Enable inlining for methods without init

f8df225

alexanderkyte merged commit 04d65f2 into mono:master Mar 19, 2019

alexanderkyte mentioned this pull request Mar 25, 2019

Returning tuple broken when enabling fullAOT+LLVM #13610

Closed

marek-safar mentioned this pull request Apr 4, 2019

LLVM IR improvements #13844

Open

11 tasks

[runtime] Default llvm to self-init+direct native calls #13312

[runtime] Default llvm to self-init+direct native calls #13312

Uh oh!

Conversation

alexanderkyte commented Mar 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexanderkyte commented Mar 7, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexanderkyte commented Mar 7, 2019

Uh oh!

alexanderkyte commented Mar 7, 2019

Uh oh!

alexanderkyte commented Mar 7, 2019

Uh oh!

alexanderkyte commented Mar 7, 2019

Uh oh!

alexanderkyte commented Mar 14, 2019

Uh oh!

alexanderkyte commented Mar 14, 2019

Uh oh!

Uh oh!

jaykrell commented Mar 17, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexanderkyte commented Apr 1, 2019

Uh oh!

lewurm commented Apr 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

alexanderkyte commented Mar 5, 2019 •

edited

Loading