Hi @zero9178 thank you for the project again!

I tried one more time with your fix of #1 , there is still one problem related with Mach-O codegen:

Assertion failed: ((!CreatedADWARFSection || Section == getContext().getObjectFileInfo()->getStackMapSection()) && "Creating regular section after DWARF"), function changeSection, file MCMachOStreamer.cpp, line 178.
PLEASE submit a bug report to https://github.com/zero9178/Pylir
Stack dump:
0.	Program arguments: /Users/proffan/Projects/Development/VISION/SymFG/Pylir/build/bin/pylir /Users/proffan/Projects/Development/VISION/SymFG/Pylir/unittests/pylir/Runtime/empty.py -fpie -c -o unittests/pylir/Runtime/builtins.o -I /Users/proffan/Projects/Development/VISION/SymFG/Pylir/src/python
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  pylir                    0x0000000106ecf4b4 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  pylir                    0x0000000106ece3a8 llvm::sys::RunSignalHandlers() + 112
2  pylir                    0x0000000106ecfb94 SignalHandler(int) + 344
3  libsystem_platform.dylib 0x0000000184c6f4a4 _sigtramp + 56
4  libsystem_pthread.dylib  0x0000000184c57ee0 pthread_kill + 288
5  libsystem_c.dylib        0x0000000184b92340 abort + 168
6  libsystem_c.dylib        0x0000000184b91754 err + 0
7  pylir                    0x0000000106ad6d48 (anonymous namespace)::MCMachOStreamer::changeSection(llvm::MCSection*, llvm::MCExpr const*) + 1356
8  pylir                    0x0000000106af290c llvm::MCStreamer::switchSection(llvm::MCSection*, llvm::MCExpr const*) + 92
9  pylir                    0x0000000105ca194c (anonymous namespace)::PylirGCMetaDataPrinter::switchToPointerAlignedReadOnly(llvm::MCStreamer&, llvm::AsmPrinter&) + 256
10 pylir                    0x0000000105ca0ecc (anonymous namespace)::PylirGCMetaDataPrinter::writeStackMap(llvm::StackMaps&, llvm::AsmPrinter&) + 160
11 pylir                    0x0000000105ca0de0 (anonymous namespace)::PylirGCMetaDataPrinter::emitStackMaps(llvm::StackMaps&, llvm::AsmPrinter&) + 44
12 pylir                    0x000000010474e374 llvm::AsmPrinter::emitStackMaps(llvm::StackMaps&) + 152
13 pylir                    0x00000001045d3450 (anonymous namespace)::AArch64AsmPrinter::emitEndOfAsmFile(llvm::Module&) + 4080
14 pylir                    0x0000000104749e7c llvm::AsmPrinter::doFinalization(llvm::Module&) + 4388
15 pylir                    0x0000000106cfbd8c llvm::FPPassManager::doFinalization(llvm::Module&) + 68
16 pylir                    0x0000000106cf51b8 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 3144
17 pylir                    0x00000001044c33c8 pylir::CompilerInvocation::compilation(llvm::opt::Arg*, pylir::cli::CommandLine&, pylir::Toolchain const&, pylir::CompilerInvocation::Action, pylir::Diag::DiagnosticsManager&) + 11860
18 pylir                    0x00000001044bfe14 pylir::CompilerInvocation::executeAction(llvm::opt::Arg*, pylir::cli::CommandLine&, pylir::Toolchain const&, pylir::CompilerInvocation::Action, pylir::Diag::DiagnosticsManager&) + 676
19 pylir                    0x000000010444c620 pylir::main(int, char**) + 3076
20 pylir                    0x000000010444b95c main + 96
21 dyld                     0x000000010c15d08c start + 520

Bitcode (although this may not be the issue)
test.bc.zip

Commenting out both switchToPointerAlignedReadOnly(os, printer); appears to make the error disappear and let the compiler go to the linking stage, which then fails with some linking errors:

[100%] Linking CXX executable ../../../../bin/markAndSweep_tests
Undefined symbols for architecture arm64:
  "builtins.bool", referenced from:
      pylir::rt::PyObject::operator==(pylir::rt::PyObject&) in libPylirRuntime.a(Objects.cpp.o)
  "builtins.dict", referenced from:
      bool pylir::rt::PyObject::isa<pylir::rt::PyDict>() in libPylirRuntime.a(Objects.cpp.o)
      decltype(auto) pylir::rt::details::AllocType<pylir::rt::Builtins::Dict>::operator()<>() const in libPylirRuntime.a(Objects.cpp.o)
      bool pylir::rt::PyObject::isa<pylir::rt::PyDict>() in libPylirMarkAndSweep.a(MarkAndSweep.cpp.o)
      decltype(auto) pylir::rt::details::AllocType<pylir::rt::Builtins::Dict>::operator()<>() const in libPylirRuntime.a(Support.cpp.o)
  "builtins.function", referenced from:
      bool pylir::rt::PyObject::isa<pylir::rt::PyFunction>() in libPylirRuntime.a(Objects.cpp.o)
      bool pylir::rt::PyObject::isa<pylir::rt::PyFunction>() in libPylirRuntime.a(Support.cpp.o)
  "builtins.int", referenced from:
      bool pylir::rt::PyObject::isa<pylir::rt::PyInt>() in libPylirRuntime.a(Objects.cpp.o)
      bool pylir::rt::PyObject::isa<pylir::rt::PyInt>() in libPylirRuntime.a(Support.cpp.o)
  "builtins.list", referenced from:
      bool pylir::rt::PyObject::isa<pylir::rt::PyList>() in libPylirMarkAndSweep.a(MarkAndSweep.cpp.o)
  "builtins.str", referenced from:
      bool pylir::rt::PyObject::isa<pylir::rt::PyString>() in libPylirRuntime.a(Objects.cpp.o)
  "builtins.tuple", referenced from:
      ____C_A_T_C_H____T_E_S_T____0() in bestFitTree_tests.cpp.o
      ____C_A_T_C_H____T_E_S_T____2() in bestFitTree_tests.cpp.o
      decltype(auto) pylir::rt::details::AllocType<pylir::rt::Builtins::Tuple>::operator()<>(unsigned long) const in libPylirRuntime.a(Objects.cpp.o)
      bool pylir::rt::PyObject::isa<pylir::rt::PyTuple>() in libPylirMarkAndSweep.a(MarkAndSweep.cpp.o)
      decltype(auto) pylir::rt::details::AllocType<pylir::rt::Builtins::Tuple>::operator()<>(unsigned long) const in libPylirRuntime.a(Support.cpp.o)
ld: symbol(s) not found for architecture arm64

That assertion seems very concerning and might need some changes in either LLVM or the general approach of how the GC plugin works.

The linker errors are probably due to the use of some asm directives. I believe these are interpreted litteraly by the compiler and are therefore missing the platform mangling.
Could you try going to https://github.com/zero9178/Pylir/blob/deafce8703ad82ceb437241a0973e003088d44d7/src/pylir/Runtime/Builtins.hpp#L23
and changing it to:

#ifdef __APPLE__
    #define MANGLING "_"
#else
    #define MANGLING ""
#endif
#define BUILTIN(name, symbol, _, Type, ...) extern Type name asm(MANGLING symbol);

I have posted https://reviews.llvm.org/D132708 to fix the first assert.

I've been thinking quite a bit about the testability of this PR and how to best review it once it's not WIP anymore and I got some questions and possible suggestions.
I don't know a lot about the Mac ecosystem so please do correct me when I say something wrong.

This PR tries to port two things at the moment. First, it adds the Darwin toolchain which should allow setting target specific options and enable linking executables via calls to ld64, and second, it ports the LLVM codegen to work on Arm.

Since I am assuming you're working on an Arm Mac machine, is it possible (via Rosetta or so) to split the Darwin toolchain into a separate reviewable and testable patch so that x86_64 Mac would work?
I am assuming that the Darwin toolchain would require only very minimal to no changes to work for both x86_64 and arm and would also be immediately testable for x86_64 since the codegen support for it already exists.
I could then add a GitHub actions runner that would run on Mac OS to also test trunk against regressions.

The codegen support for arm64 could then be added later in a separate PR. Luckily it's ABI is not a lot different to Linuxs arm ABI making it easier to collaborate on since I don't have any Mac machines to work with.

I have posted https://reviews.llvm.org/D132708 to fix the first assert.

This patch has landed in LLVM now and I have also bumped the version to build Pylir with (and that is used in CI) to include it.

After a few fixes it now is able to generate object files without problem. However, the generated Mach-O files cannot run, because the Apple DYLD fails at the fixup step. Very little documentation exist, so it's really hard to debug.

test_macho.zip

generated mach-o for just one line print("test!").

Can you try if the markAndSweep_tests can be run? That executable also contains code generated py pylir but is linked via your system compiler (aka clang). That would narrow down the problem to the linker invocation.

Additionally you could then try compiling a python source file to an object file with -c and then manually invoking clang++ with -v on the object file and the paths to libPylirMarkAndSweep.a, libPylirRuntime.a and libPylirRuntimeMain.a and see if that links. You could then also compare the linker command clang generates to the one you currently have implemented.

Can you try if the markAndSweep_tests can be run? That executable also contains code generated py pylir but is linked via your system compiler (aka clang). That would narrow down the problem to the linker invocation.

Additionally you could then try compiling a python source file to an object file with -c and then manually invoking clang++ with -v on the object file and the paths to libPylirMarkAndSweep.a, libPylirRuntime.a and libPylirRuntimeMain.a and see if that links. You could then also compare the linker command clang generates to the one you currently have implemented.

Yup, it fails for the same error

My next guess would be something to do with PIC or similar. Clang also defaults to PIC [0] so possibly this is required by the platform (I haven't checked for Darwin, but Windows eg. mandates it as well).

I don't think that will fix the issue however, as the cmake script currently always passes -fpie to the compiler which ought to generate the same kind of relocations that would make at least markAndSweep_tests succeed.
Next best thing is probably to check the disassembly/relocations output of the object file, with one generated by clang (for simple things like calls and referencing globals) and possibly figure out the right knob in LLVM.

[0] https://github.com/llvm/llvm-project/blob/1b3a78d1d534550b2f85a81b2e9ac6d7a94a478e/clang/lib/Driver/ToolChains/Darwin.cpp#L2970

Finally found the culprit, quoting stack overflow: https://stackoverflow.com/questions/73382860/why-cant-i-assemble-absolute-addresses-in-the-text-section-on-arm64-macos/73382861#73382861

Absolute addresses in the text section seem to be unsupported by ARM64 MacOS. As best I can tell, the dynamic linker tries to apply a relocation/fixup to store the actual runtime address of dest into vector, but crashes because .text is already mapped read-only.

So if you have an object that needs to be initialized with the address of a label or other object, then you need to put it in a data section, even if it is read-only. This is what clang does when compiling C/C++ code. For instance, if you write in C

const int * const ptr1 = &i;
const int * const ptr2 = NULL;
const int * const ptr3 = (const int *)0xdeadbeefcafed00dUL;

then i, ptr2, ptr3 all get preceded by .section __TEXT,__const, but ptr1 is preceded by .section __DATA,__const. The latter section is also read-only at runtime, but apparently is mapped read-write while relocation is done.

And you simply cannot use ldr x1, =label at all. Use instead adr or adrp / add as appropriate to generate the address in x1 if label is in the text section, or otherwise load it from the global offset table as appropriate.

It would be nice if the linker would detect this and warn you, instead of going ahead and building an executable that mysteriously crashes.

Only need one line of change:

        if (triple.isOSBinFormatMachO())
        {
            constSectionPrefix = "__DATA,";
            dataSectionPrefix = "__DATA,";
        }

Note this is likely a solution that sacrifices some binary safety.

Now that the thing is working, next step is to clean this mess up and possibly set up CI

Current lit-test result:
output.txt

Amazing work thank you!
A few of those lit tests failing is just sloppy work on my side. Some of those should be marked as requiring x86 so that they are simply not run if your LLVM does not support X86.
The most important and interesting tests to start off with are the Execution tests however, since they fully exercise the whole compiler pipeline. I'd be very interested if these would work with LLVMs lld linker linked in. Could you rebuild your LLVM to also build lld and then build Pylir with -DPYLIR_EMBED_LLD=ON? I am currently also debating whether to make this the default in the future.

A few more things after looking at the output.txt more closely. You'll have to update the globalValueOp.mlir tests with the sections you had to change.
Next I am wondering whether the explicit -lunwind is needed on MacOS? I have very shortly used darling to emulate a Mac environment and I think I remember seeing it be part of libc++.

bbd50cb should fix the tests failing due to X86 not being compiled into your LLVM

Hello @ProfFan! I recently did some progress in this area and added support for MacOS that at least works on x86 machines here 110d2e3
I believe the only thing that currently stops M1 support should now be the ARM64 ABI Support.
The runtime functions don't actually make use of any complex C ABI features yet, so for the time being, a simple implementation that can only handle pointers and pointer sized integer would suffice I believe (which really just means forwarding to LLVM 1:1).

If you have time, I'd be curiouss if all these changes are enough for the compiler to work on the M1 chip.

There is a linking error with Catch2 and fmtlib that I haven't got time to debug:

Undefined symbols for architecture arm64:
  "fmt::v9::detail::assert_fail(char const*, int, char const*)", referenced from:
      std::__1::make_unsigned<long>::type fmt::v9::detail::to_unsigned<long>(long) in libParser.a(Dumper.cpp.o)

That is a very odd and confusing error. Especially since it is about fmt version 9 (based on the namespace), when the repo currently vendors version 7.1.3.

Nevertheless, thank you for checking!

Maybe it's because your CMake did not specify the vendor version and it is finding my system lib?