Build on CBL-mariner host with rootfs #84148

sbomer · 2023-03-30T21:30:36Z

This changes the linux builds to use the CBL-mariner build images added in dotnet/dotnet-buildtools-prereqs-docker#832. All of these builds are now cross-builds with a rootfs (including x64, and x64 musl).

The new images intentionally don't have binutils, so our build jobs have been updated to use llvm-based tools. This includes lld when we run nativeaot over our tests. This leads to a slight size regression, I believe due to ILCompiler not supporting --gc-sections with lld (see Add LinkerFlavor to NativeAOT #83558 (comment)). cc @MichalStrehovsky
This change turns off PGO optimization because it was hitting a compiler crash when consuming and old version of profile data produced by clang9 instrumented builds (https://bugzilla.redhat.com/show_bug.cgi?id=1827282). The intention is to quickly flow the change into our optimization repo, then collect and ingest the new data so we can turn optimization back on.
The Mono LLVMAot test build is kept on the old build images, because they run the aot compiler which depends on binutils. We can probably move it to the same plan as the other legs, but I'm not familiar with the context here. cc @akoeplinger. We would at least need to fix AS_NAME here:

runtime/src/mono/mono/mini/aot-compiler.c

Line 13030 in 1d2cd20

#define AS_NAME "as"
Mono arm64 product build is kept on the old images due to Unable to compile mono arm64 with clang12 #84503.

Contributes to #83428.

This change enables cross-building for Alpine x64 on other x64 distro.

Necessary because pkg-config on mariner is a shim for pkgconf, which doesn't support override variables.

- Use lld for LTO instead of ld.gold - Set cross toolchain options for x64 mono build - Prefer llvm-objcopy for mono symbol stripping

Our custom LLVM build seems to have different defaults when targeting ARM than the clang-12 available in ubuntu. Probably related to https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/b7363248b115339c4fb838fcd3ae43671eedae0a. Passing the -march explicitly should not hurt.

Allow text relocations. The lld defaults differ from ld, which will add text relocations on demand by default. See https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities

+ more cleanup

To locate gcc toolchain in target sysroot, when host arch matches target arch.

ghost · 2023-03-30T21:30:46Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	sbomer
Assignees:	sbomer
Labels:	`area-Infrastructure-coreclr`
Milestone:	-

eng/Subsets.props

eng/common/cross/build-rootfs.sh

src/coreclr/pgosupport.cmake

Co-authored-by: Adeel Mujahid <[email protected]>

@BruceForstall

Fixes dotnet/runtime#84780. Tested by hooking up the changes locally with runtime's `jitformat.py` and running `python3 ./src/coreclr/scripts/jitformat.py -c /runtime/src/coreclr -o linux -a x64` in our mariner build container. This might break scenarios where jitutils are used outside of ci on a non-mariner build image. Per discussion with @BruceForstall let's make a quick fix to unblock ci, and if this is problematic we could for example add an extra argument that controls whether the `-cross` argument is passed to the runtime build. For context, the `-cross` argument is necessary after dotnet/runtime#84148 because our linux build images have change from centos 7 to CBL-mariner, and the linux builds are now cross-builds that use a rootfs (even x64 and x64-musl).

rzikm · 2023-04-19T07:55:40Z

This PR (or a related one) breaks System.Net.Security.Enterprise.Test builds (https://dev.azure.com/dnceng-public/public/_build/results?buildId=243440&view=logs&j=03b95830-93c2-599b-7e4d-1efb940adc28&t=9abf97cd-b4a3-5a26-6cf2-d05ce865f2ef&l=3907).

clang : error : invalid linker name in argument '-fuse-ld=lld' [/repo/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

The docker image used (mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04) has clang-9 but no lld binary installed. Will that be fixed by the TODO at
https://github.com/sbomer/runtime/blob/92d1f2217649103e3f3e5121b8ad35dca5a81bbb/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj#L62? Or is some additional detection needed?

To reflect the plan in dotnet/runtime#83428, and changes made in dotnet/runtime#84148.

janvorli and others added 20 commits February 9, 2023 11:19

Enable x64 Alpine crossbuild

8735c94

This change enables cross-building for Alpine x64 on other x64 distro.

Set PKG_CONFIG_SYSROOT_DIR

0084966

Necessary because pkg-config on mariner is a shim for pkgconf, which doesn't support override variables.

More build fixes

bb39763

- Use lld for LTO instead of ld.gold - Set cross toolchain options for x64 mono build - Prefer llvm-objcopy for mono symbol stripping

test

a5f2dbd

Print full clang command to show selected linker

db7bf3e

Build scripts

597a304

Fix x64 -> x64 musl cross-build

3e5a366

Allow cross compiling for x86

612c686

Enable x86 build with lld

a9fd083

Allow text relocations. The lld defaults differ from ld, which will add text relocations on demand by default. See https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities

Fix x86 mono cross-build

d0a1560

Some cleanup

2abee9b

Add gcc-toolchain arg when aot-ing crossgen

54dc641

+ more cleanup

Merge remote-tracking branch 'origin/main' into marinerTest

2da99a9

Fix AOT compilation of ILCompiler and crossgen2

825a1d9

Only pass --gcc-toolchain where required

c7468f6

To locate gcc toolchain in target sysroot, when host arch matches target arch.

Cleanup

ae6c6ab

Fix symbol stripping on ci builds

3d70352

Fix nativeaot test build

2f12a9a

Pick up mariner images, set cross-build arguments

f6bd3e5

ghost added the area-Infrastructure-coreclr label Mar 30, 2023

ghost assigned sbomer Mar 30, 2023

sbomer mentioned this pull request Mar 30, 2023

Add 'su' to mariner images dotnet/dotnet-buildtools-prereqs-docker#835

Merged

build-analysis bot mentioned this pull request Mar 31, 2023

WasmTestOnBrowser-System.* test failures in CI #83655

Closed

am11 reviewed Mar 31, 2023

View reviewed changes

eng/Subsets.props Outdated Show resolved Hide resolved

am11 reviewed Mar 31, 2023

View reviewed changes

eng/common/cross/build-rootfs.sh Show resolved Hide resolved

am11 reviewed Mar 31, 2023

View reviewed changes

src/coreclr/pgosupport.cmake Outdated Show resolved Hide resolved

sbomer and others added 2 commits March 31, 2023 16:27

Don't use lld on macos

e9f17fb

Update src/coreclr/pgosupport.cmake

26621ab

Co-authored-by: Adeel Mujahid <[email protected]>

This was referenced Apr 13, 2023

Decide on linker to use in runtime builds #84794

Closed

Fix enterprise linux build #84795

Merged

Add runtime build dependencies to ubuntu 22.04 image dotnet/dotnet-buildtools-prereqs-docker#843

Merged

eerhardt mentioned this pull request Apr 13, 2023

[main] Update dependencies from dotnet/runtime dotnet/aspnetcore#47693

Merged

sbomer mentioned this pull request Apr 13, 2023

Add Azure DevOps dependencies to mariner fpm image dotnet/dotnet-buildtools-prereqs-docker#844

Merged

This was referenced Apr 14, 2023

jitrolling build is broken #84822

Closed

jitformat failing on linux x64 #84780

Closed

sbomer mentioned this pull request Apr 14, 2023

Pass "-cross" arg on linux builds dotnet/jitutils#369

Merged

sbomer mentioned this pull request Apr 14, 2023

Adjust managed type system for new function pointer handling #84819

Merged

clamp03 mentioned this pull request Apr 17, 2023

[RISC-V] Disable corelib crossgen2 #84781

Merged

BruceForstall mentioned this pull request Apr 18, 2023

Update coredistools dotnet/jitutils#372

Closed

This was referenced Apr 19, 2023

Add TargetHostName to QuicConnection #84976

Merged

System.Net.Http.Enterprise.Tests build failing in CI #85035

Closed

sbomer mentioned this pull request Apr 20, 2023

dotnet --info hitting /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found on Latest net8.0 dotnet/sdk#31893

Closed

antonfirsov mentioned this pull request Apr 25, 2023

[HttpStress] [SslStress] Linux stress test builds are broken #85338

Closed

sbomer added a commit to dotnet/core that referenced this pull request May 2, 2023

Update .NET 8 libc compat (#8401)

5da7900

To reflect the plan in dotnet/runtime#83428, and changes made in dotnet/runtime#84148.

sbomer mentioned this pull request May 3, 2023

Run NativeAot tests on arm64 #82307

Merged

ghost locked as resolved and limited conversation to collaborators May 19, 2023

sbomer deleted the marinerBuild branch August 7, 2025 17:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build on CBL-mariner host with rootfs #84148

Build on CBL-mariner host with rootfs #84148

Uh oh!

sbomer commented Mar 30, 2023 •

edited

Loading

Uh oh!

ghost commented Mar 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rzikm commented Apr 19, 2023 •

edited

Loading

Uh oh!

Uh oh!

Build on CBL-mariner host with rootfs #84148

Build on CBL-mariner host with rootfs #84148

Uh oh!

Conversation

sbomer commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Mar 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rzikm commented Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sbomer commented Mar 30, 2023 •

edited

Loading

rzikm commented Apr 19, 2023 •

edited

Loading