Add modules support (with order resolution, etc.) and 'import std' support for Meson (initial draft, tested in Clang) #14989
Conversation
8ccd919 to
b5c9469
Compare
b5c9469 to
4bea51a
Compare
|
I'm just offering some input as a user, but adding a project-wide option doesn't seem like it has any path to being usable for not-std modules in the future. I would imagine something like this would be more extensible for the future: Where for a first step, declare_cpp_module() would just have some automagic for std, and later you can add arguments for sources for other modules. And what needs to happen is the bmi is built once for lib_foo with lib_foo's cpp_args, and again for bar_exe with bar_exe's cpp_args. Any implementation that doesn't tie a bmi to the cpp_args is broken, isn't it? |
It would not be used for those. They would have some different mechanism, because they are a completely different thing (because they affect things like compilation step build orders etc). Also note that those options can be set per-target. This would be equivalent to defining it at the top level: project('foo', 'cpp',
default_options: ['cpp_std=c++26'])
executable('importstd', 'importstd.cpp',
override_options: 'cpp_modules=std') |
Currently you can do this: which is global. But I plan to add per-target overrides, as @jpakkane mentioned: By the way, looking at Clang documentation there is something I am not sure if I am doing wrong (because it works): it seems that every BMI file generates a .o file. However, right now I am just adding a target that generates a BMI for both gcc and clang and I compile by adding that custom target that generates a BMI but not a .o file to the c++ targets when the option 'cpp_import_std' is active (with the flags for building the BMI itself aligning with debug/release flags from the build type). But the library that is linked at the end of the process into the c++ targets seems to be the stock libc++ (EDIT: confirmed it is, since adding -nostdlib++ gives me link errors) included in Clang. Same for GCC. I just generate the std.gcm file in gcm.cache. Should I generate a .o file for every BMI file and link it, and remove the std library linking altogether? So in this case I would have to add another custom target that compiles the .o explicitly from the BMI and add it to the link phase. So I am not sure if there is a potential mismatch in compilation flags, etc. could anyone shed some light on this? Maybe the correct way here is to:
How about libraries? I guess the final link step is the only one that would need std.o? EDIT: experiment compiling std.o for std.pcm in Clang gives me a 664B file with a single symbol!? I am confused :D I suspect the correct way is to link the std object file? As per what Microsoft documented here (though compilers might differ): https://learn.microsoft.com/en-us/cpp/cpp/tutorial-import-stl-named-module?view=msvc-170 |
|
According to https://youtu.be/-p9lvvV8F-w?feature=shared&t=2117 you can tell the compiler to not generate the .o file. The BMI compatability issues at minute 50 are also important to review. Apologies for not trying your PR yet, but does import std.compat work, or is it just solely std? |
Tested on Clang 19 on MacOS.
BEWARE, WARNING: This is a drafty implementation
whose only goal was to see module resolution working
in a projet. This is not representative of the
quality of the code I would be willing to submit,
but just a fast test.
That said: support for modules has been added consisting of:
- a global (not per target) scan step via clang-scan-deps.
- the scan + accumulate step are done at once via
clang-scan-deps.
- dependency order resolution for module imports should
work.
- the strategy to activate scanning right now is global,
even subprojects are scanned, based on compile_commands.json
generated. You need clang 17 and above though only
Clang 19 has been tested.
The script depscanaccumulate.py *currently has the path to
clang-scan-deps hardcoded* and will not work unless you fix it
to point to your clang-scan-deps until I do some cleanups.
Please find attached a small project to work with
that I converted in the Github issue:
mesonbuild#14989.
The project is a NES emulator that *does almost nothing* since it was
an abandoned side-project, but for the sake of converting to modules
it has been invaluable.
There are many strategies to do the scanning:
- per file.
- per target.
- globally (as it is currently done with this prototype)
Also, we could make a differentiation to split the files that
are modules from the normal source files, but I would not
recommend it. I think doing per-target scans all-or-nothing
per target should be enough.
Currently, the .pcm files are put at the top-level of the
build directory. This should go in some global per-project
cache to the best of my understanding.
There is a small trick now, which is not optimal, but it does
the job: the cpp_COMPILE rule is added -fmodule-output={the-pcm-file}.pcm
-fprebuilt-module-path={the-build-dir} and since not all files are
modules, this is surrounded by --start-no-unused-arguments and
--end-no-unused-arguments. This avoids the need to generate more
fancy or dynamic stuff, which I do not know even if it is possible
at all at the moment.
This method of compiling generates a .o file and a .pcm as a
side-effect. There is another way, potentially more parallel,
to build separately (via --precompile flag) a .pcm and a .o,
but again, this will do as a first step I guess...
I would expect that for big projects, unless clang-scan-deps is
really fast, there might be other strategies that are better for
incremental builds but something like this should do for now.
I would even expect that Meson could choose in a smart way
what strategy to adopt? (maybe)
Your targets are what you are already familiar with, this is
a transparent feature except for:
1. the name of your build target should have the name of the module.
2. you need to have a module interface file called target.name.cppm.
Control to opt-out from module scanning, when the strategy is not
global scanning, should probably be added, but since I used
compile_commands.json + global scanning I did not get bothered, which
whould have deviated me from my goal.
I updated and activated the rule for should_use_dyndeps() in the
ninja backend that already existed in the code and generated
the depscanaccumulate rule.
Check the build.ninja file to see what it is generated.
- I generate every target with a dependency on std
(using cpp_import_std feature, which is active).
- A single deps.dd file is generated every time a file is touched.
- any file you touch should trigger a deps.dd scan.
We could make a differentiation to split the files that
are modules from the normal source files to do less agressive scanning,
but I would not recommend it. I think doing per-target scans
all-or-nothing per target should be enough since these tools should do
bulk updates and not be invoked once per file anyway.
Currently, the .pcm files are put at the top-level of the
build directory. This should go in some global per-project
cache to the best of my understanding.
There is a small trick now, which is not optimal, but it does
the job: the cpp_COMPILE rule is added -fmodule-output={the-pcm-file}.pcm
-fprebuilt-module-path={the-build-dir} and since not all files are
modules, this is surrounded by --start-no-unused-arguments and
--end-no-unused-arguments. If the file is just a normal translation unit,
a .pcm will not be generated even if it was specified.
This avoids the need to generate more fancy or dynamic stuff, which I do
not know even if it is possible at all at the moment.
Please find attached a small project to work with
that I converted. The project is a NES emulator that
*does almost nothing* since it was an abandoned side-project,
but for the sake of converting to modules has been invaluable.
-------------------------------------------------------------------------
Also add 'import std' support for Meson Clang/GCC.
This commit adds the ability for Meson targets for:
- clang (tested in clang 19 homebrew MacOS)
- gcc (tested in a Docker container with Linx GCC15)
The feature can be used by setting a new option 'cpp_import_std'
globally, which can be true or false.
By default the option is conservatively set to false.
If enabled, during the configuration phase, Meson checks that the
compiler supports building the std library as a module via compiler
version checks.
At build time, it adds a custom target to build the std bmi. This bmi is
wrapped in a dependency and included in build targets that are
executables, shared libraries and libraries if the target uses c++
language and it is not in a subproject. See build_target function.
A best effort has been done to align debug/release flags with the built
version of the std library, though this needs review.
Note that in order to use the feature, the minimum version is C++23,
since import std is a C++23 feature.
An example of how to use it can be found in test '1 import std'.
Smoke-tested in:
- Clang 19 homebrew, in my Mac.
- Gcc 15 in Docker container.
- Tested with a multi-file project that uses subprojects
by replacing all std library with 'import std'.
ROADMAP:
- Add override to build targets to avoid using 'import std'?
|
I added a draft for modules support (not import std only, but modules support) via clang and clang-scan-deps with global project scanning (which clang-scan-deps does, basically) and attached a project you can play with. Please, read through, there is some hardcoded stuff and this is absolutely not representative in terms of code quality, I will sort it out as I keep getting some feedback. |
This commit adds support for implementation partitions
and modifies file conventions for a modularized target.
The conventions are:
- your primary interface unit for your module should always
be called 'module.cppm'. It should export module <target-name>.
- your interface partitions can have any name supported by a module.
For example: MyThings.cppm. The interface partition
should declare 'export module <target-name>:MyThings'.
- Your *importable* interface implementation units should end with
'Impl.cppm'. For example 'MyThingsImpl.cppm'. This should
'module <target-name>:MyThings' (without export).
- Your *non-importable* implementation can have any name *with an
extension .cpp*, not a .cppm, since implementation units are
not importable. It is highly recommended, though, that if you
have a single implementation file, it is called moduleImpl.cpp.
You can see the project attached in Github draft PR:
mesonbuild#14989 as an example that uses
interface partitions, primary interface units, an implementation
module (but not an implementation module partition, though it should
work), multiple targets and import std.
It can run a couple of tests when fully compiled and it resolves
dependency order via scanner with Clang.
4bea51a to
09b1d0a
Compare
|
Ideally I'd like to see meson use clang's two-phase compilation model (see https://clang.llvm.org/docs/StandardCPlusPlusModules.html#how-to-produce-a-bmi ) Have you considered this already?
BMIs are not linkable artifacts and do not replace object files. In the one phase model, clang creates both the BMI and object file. In the two phase model, these are separate steps.
no, neither gcc nor clang require doing anything of the likes for their module implementation. |
I am aware of it, but that complicates things given what is already in, so as a first step I took the path of generating a bmi as a side-effect of compiling a .o file.
Yes.
Actually what I mean is: if I generate a bmi, should I generate also a library artifact and link against that libc++ instead of the original provided by the OS and -nostdlib++? Otherwise, my bmi should match the flags of the stock-provided OS or I am not sure how compatible it is. However, when I compiled the object file, it hardly included any code...
Read above for my concerns about this very thing. I know bmi are "binary header files" in some way, but compiling a library includes also code. So I was wondering about how import std should be used correctly (it works for me, but not sure it is correct or can lead to some kind of UB). |
Oops sorry, I just saw you mention it in your opening post.
The std module contains all the std headers, not the entire standard library. Accordingly, the object file you get from the std pcm is not the entire standard library, but just some inline symbols defined in the headers. Meanwhile, the pcm will contain all the declarations, including template bodies. In general, the object file built from a module is NOT in any way equal to the library itself. You can think of the pcm object file as all symbols that get defined, not just declared, when including the related header. |
EDIT: updated top comment. I will keep it up to date with changes added. Please read again since there are changes since yesterday. LAST UPDATE: september, the 8th, 2025
Changes: 8th september 2025
Test project is here: Nesecidad-0.0.2.zip
Add module partitions and implementation modules.
This commit adds support for implementation partitions
and modifies file conventions for a modularized target.
The conventions are:
your primary interface unit for your module should always
be called 'module.cppm'. It should export module 'target-name'.
your interface partitions can have any name supported by a module.
For example: MyThings.cppm. The interface partition
should declare 'export module target-name:MyThings'.
Your importable interface implementation units should end with
'Impl.cppm'. For example 'MyThingsImpl.cppm'. This should
'module target-name:MyThings' (without export).
Your non-importable implementation can have any name with an
extension .cpp, not a .cppm, since implementation units are
not importable. It is highly recommended, though, that if you
have a single implementation file, it is called moduleImpl.cpp.
You can see the project attached in Github draft PR:
#14989 as an example that uses
interface partitions, primary interface units, an implementation
module (but not an implementation module partition, though it should
work), multiple targets and import std.
It can run a couple of tests when fully compiled and it resolves
dependency order via scanner with Clang.
Changes 7th september:
Test project is here: Nesecidad-0.0.1.tar.gz
DRAFT: Add modules support via clang-scan-deps.
Tested on Clang 19 on MacOS.
BEWARE, WARNING: This is a drafty implementation
whose only goal was to see module resolution working
in a project. This is not representative of the
quality of the code I would be willing to submit,
but just a fast test.
That said: support for modules has been added consisting of:
clang-scan-deps.
work.
even subprojects are scanned, based on compile_commands.json
generated. You need clang 17 and above though only
Clang 19 has been tested.
Only primary interface units have been tested so far (no partitions, etc),
but the dependency order resolution is already there. Probably some
problems could arise in name conventions when adding these but the
order resolution should work already.
The script depscanaccumulate.py currently has the path to
clang-scan-deps hardcoded and will not work unless you fix it
to point to your clang-scan-deps until I do some cleanups.
Please find attached a small project to work with
that I converted in the Github issue:
#14989.
The project is a NES emulator that does almost nothing since it was
an abandoned side-project, but for the sake of converting to modules
it has been invaluable.
There are many strategies to do the scanning:
Also, we could make a differentiation to split the files that
are modules from the normal source files, but I would not
recommend it. I think doing per-target scans all-or-nothing
per target should be enough.
Currently, the .pcm files are put at the top-level of the
build directory. This should go in some global per-project
cache to the best of my understanding.
There is a small trick now, which is not optimal, but it does
the job: the cpp_COMPILER rule is added -fmodule-output={the-pcm-file}.pcm
-fprebuilt-module-path={the-build-dir} in its ARGS and since not all files are
modules, this is surrounded by --start-no-unused-arguments and
--end-no-unused-arguments. This avoids the need to generate more
fancy or dynamic stuff, which I do not know even if it is possible
at all at the moment.
This method of compiling generates a .o file and a .pcm as a
side-effect. There is another way, potentially more parallel,
to build separately (via --precompile flag) a .pcm and a .o,
but again, this will do as a first step I guess...
I would expect that for big projects, unless clang-scan-deps is
really fast, there might be other strategies that are better for
incremental builds but something like this should do for now.
I would even expect that Meson could choose in a smart way
what strategy to adopt? (maybe)
Your targets are what you are already familiar with, this is
a transparent feature except for:
Control to opt-out from module scanning, when the strategy is not
global scanning, should probably be added, but since I used
compile_commands.json + global scanning I did not get bothered, which
whould have deviated me from my goal.
I updated and activated the rule for should_use_dyndeps() in the
ninja backend that already existed in the code and generated
the depscanaccumulate rule.
Check the build.ninja file to see what it is generated.
I generate every target with a dependency on std
(using cpp_import_std feature, which is active).
A single deps.dd file is generated every time a file is touched.
any file you touch should trigger a deps.dd scan.
We could make a differentiation to split the files that
are modules from the normal source files to do less agressive scanning,
but I would not recommend it. I think doing per-target scans
all-or-nothing per target should be enough since these tools should do
bulk updates and not be invoked once per file anyway.
Currently, the .pcm files are put at the top-level of the
build directory. This should go in some global per-project
cache to the best of my understanding.
There is a small trick now, which is not optimal, but it does
the job: the cpp_COMPILE rule is added -fmodule-output={the-pcm-file}.pcm
-fprebuilt-module-path={the-build-dir} and since not all files are
modules, this is surrounded by --start-no-unused-arguments and
--end-no-unused-arguments. If the file is just a normal translation unit,
a .pcm will not be generated even if it was specified.
This avoids the need to generate more fancy or dynamic stuff, which I do
not know even if it is possible at all at the moment.
Please find attached a small project to work with
that I converted. The project is a NES emulator that
does almost nothing since it was an abandoned side-project,
but for the sake of converting to modules has been invaluable.
Also add 'import std' support for Meson Clang/GCC.
This commit adds the ability for Meson targets for:
The feature can be used by setting a new option 'cpp_import_std'
globally, which can be true or false.
By default the option is conservatively set to false.
If enabled, during the configuration phase, Meson checks that the
compiler supports building the std library as a module via compiler
version checks.
At build time, it adds a custom target to build the std bmi. This bmi is
wrapped in a dependency and included in build targets that are
executables, shared libraries and libraries if the target uses c++
language and it is not in a subproject. See build_target function.
A best effort has been done to align debug/release flags with the built
version of the std library, though this needs review.
Note that in order to use the feature, the minimum version is C++23,
since import std is a C++23 feature.
An example of how to use it can be found in test '1 import std'.
Smoke-tested in:
by replacing all std library with 'import std'.
ROADMAP: