Hi @robUx4

Thanks for this wave of PR's, #1591 to #1600. Before processing them one by one, I would like to discuss here a few topics which apply to all of them.

If I understand correctly, all PR's are related to building TSDuck with clang or gcc on Windows, isn't it?

So far, I always avoided such mixture. It does not mean that I don't want to support it. However, by experience, mixing "alien" tools and libraries always come at the price of endless problems. That is why I have always carefully used the "native" tools and libraries:

• Build system: GNU Make on UNIX, MSBuild on Windows.
• Compiler: MSVC on Windows, clang on macOS, FreeBSD, OpenBSD, gcc on all other UNIX systems.
• Crypto: OpenSSL on UNIX, BCrypt on Windows.
• HTTP: libcurl on UNIX, WinInet on Windows.
• etc...

As a concrete example, TSDuck uses libsrt and libsrt uses OpenSSL for cryptographic functions on all platforms. As a result, OpenSSL brings a lot of packaging and building issues on Windows and I try to push the SRT maintainers to use BCrypt on Windows instead of OpenSSL (see here and here).

TSDuck represents a huge amount of code and I don't want to lose time on recurrent issues which can be avoided by using the "right tool on the right platform".

So, my first question is: what is the rationale behind using clang and gcc on Windows? It builds correctly with the "native" Windows compiler, so why looking for problems with other "alien" compilers?

The second question is: what are the versions of clang and gcc that you are using? Where do they come from? MinGW, MSys2, Visual Studio (which includes a version of clang)?

As explained here, TSDuck now requires a fully compliant C++20 compiler. All three compilers, gcc, clang, msvc, were buggy in the first versions where they claim to support C++20. The minimal working versions are gcc 13 and clang 16 (Apple) or 17 (upstream).

I am specifically skeptical about #1598. Reporting a type error in a false branch in an if constexpr is an error I got in most "broken C++20 compilers", ie. versions of the three compilers which claimed to support C++17 and C++20 but which failed to do it properly. This code now compiles correctly on all compilers. I am not a C++ authority but, AFAIK, a false branch in an if constexpr is syntactically evaluated but further steps (semantics and code generation) are discarded. In practice, some semantics are checked but type checking is relaxed. The if constexpr is meant to be a better replacement of some conditional compilations and most SFINAE. So, it would be useless if full type checking was required.

We would need a C++ authority to be more precise. This stackoverflow post mentions that "the if constexpr statement [...] does not require the discarded substatement to be well-formed". I know that stackoverflow is not the C++ standard. However, it is crowded with C++ zealots who enter a fight at the slightest misconception and the above statement hasn't been rebutted. So, I consider it valid, which is confirmed by all three compilers (recent versions).

There is another strange thing. In #1599 and #1600, you need to de-inline public methods which call a private method or object. There is nothing wrong with calling a private method from an inlined public method. This is even a recommended practice because it encapsulates private information without the overhead of an additional level of calls. So, if you had to make these modifications, this is the symptom of some other problem that needs to be specifically fixed. Especially because it works with MSVC and all required symbols must have been correctly exported.

As a last point, which build system are you using to compile TSDuck with clang or gcc on Windows? I guess that the existing MSBuild and GNU Make file don't work. I need to reproduce your build to validate the PR and maintain that configuration later (if it is worth doing). So, I need to reproduce your build.

Hi @lelegard, thanks for your prompt and lengthy response.

If I understand correctly, all PR's are related to building TSDuck with clang or gcc on Windows, isn't it?

Yes. That includes the clang that is available in MSVC (clang-cl.exe which mimics MSVC and clang.exe which is GCC like) using the official Windows SDK. The more common usage is with mingw-w64 which is what people use to cross-compile on Linux (including for WINE).

I have more patches pending, these are the most straightforward. I also made a CMake project to build everything I need, allowing switching between various OS and compilers. This is related to a project that should end up integrating tsduck in VLC. The logical way for us is to use the contrib system we already have in place to use external code, either with autoconf, CMake or meson projects. VLC does not support building with MSVC for Windows builds and we usually cross-compile from Linux. We often end up having to patch mingw-w64 to add missing things needed by our code and contrib code. So any differences with the Windows SDK tend to go away over time.

Using CMake also allows getting external librairies within the tsduck project in some cases (OpenSSL seems to be tricky to cross compile).

TSDuck represents a huge amount of code and I don't want to lose time on recurrent issues which can be avoided by using the "right tool on the right platform".

I understand. I started with an older version and had to substantially edit the CMake project after your recent refactoring. You probably don't want extra work for that. There are ways to do that. For example with CMake enabled in the CI, it should not be blocking the merge of a PR if it fails to build. I know it's possible on Gitlab, not sure about GitHub.

On the plus side, compiling with more compilers allow fixing issues that other compilers don't see.

The second question is: what are the versions of clang and gcc that you are using? Where do they come from? MinGW, MSys2, Visual Studio (which includes a version of clang)?

I've been testing the CMake build with MSVC 17.12.3, clang-cl 18.8, clang.exe 18.1.8 (from Visual Studio) on Windows and mingw-w64 clang 20.0 on Windows Subsystem for Linux (WSL).

I am specifically skeptical about #1598. Reporting a type error in a false branch in an if constexpr is an error I got in most "broken C++20 compilers", ie. versions of the three compilers which claimed to support C++17 and C++20 but which failed to do it properly. This code now compiles correctly on all compilers. I am not a C++ authority but, AFAIK, a false branch in an if constexpr is syntactically evaluated but further steps (semantics and code generation) are discarded. In practice, some semantics are checked but type checking is relaxed. The if constexpr is meant to be a better replacement of some conditional compilations and most SFINAE. So, it would be useless if full type checking was required.

We would need a C++ authority to be more precise. This stackoverflow post mentions that "the if constexpr statement [...] does not require the discarded substatement to be well-formed". I know that stackoverflow is not the C++ standard. However, it is crowded with C++ zealots who enter a fight at the slightest misconception and the above statement hasn't been rebutted. So, I consider it valid, which is confirmed by all three compilers (recent versions).

I get the same issue even with MSVC. Here is a sample code that shows the issue: https://godbolt.org/z/cb4fzMa7Y. You can switch between the compiler versions to show which one allows the compilation, although it should be a mingw where the pathes should be in wchar_t. I think on Linux the code is fine as it's usually using UTF-8. I also got this error with clang-cl 18.1.8 and llvm-mingw 20.
I'm not sure sure why it works in Visual Studio or maybe it's only with newer MSVC which are not available in godbolt.

Looking at cppreference there is a section that says "The discarded statement cannot be ill-formed for every possible specialization". In this case I think that's automatically converting a const wchar_t* to a const char*.

There is another strange thing. In #1599 and #1600, you need to de-inline public methods which call a private method or object. There is nothing wrong with calling a private method from an inlined public method. This is even a recommended practice because it encapsulates private information without the overhead of an additional level of calls. So, if you had to make these modifications, this is the symptom of some other problem that needs to be specifically fixed. Especially because it works with MSVC and all required symbols must have been correctly exported.

In this case that may be an issue with LLVM, even version 20 and even the one provided by Microsoft. The behavior of mixing dllexport/dllimport with inline code is properly defined for MSVC. The inlining is disabled in that case. This doesn't seem to be the case for LLVM. I haven't tried GCC. It may be a bit trickier because these methods don't have an explicit dllexport but the whole class is and therefore inlining calls should be disabled for all the methods ?

As a last point, which build system are you using to compile TSDuck with clang or gcc on Windows? I guess that the existing MSBuild and GNU Make file don't work. I need to reproduce your build to validate the PR and maintain that configuration later (if it is worth doing). So, I need to reproduce your build.

I'm using CMake which should be easy to integrate in the CI. I haven't sent that Pull Request yet because I need to collect all clean patches before it can be used to build successfully.

Hi @robUx4

This is related to a project that should end up integrating tsduck in VLC.

This is a great idea. We can collaborate on this if you like. You may email me using thierry at lelegard dot fr if you wish to discuss this. It is quite common to use TSDuck and ffmpeg together, much less with VLC. I am interested in how and why TSDuck and VLC can be integrated. Maybe this could influence future features in TSDuck.

Interestingly, reading the relevant part of the VLC source code helped me quite a lot when I implemented the class ts::SinkFilter to interface the horrible DirectShow framework on Windows (while the LinuxTV API is so simple in contrast).

I also made a CMake project to build everything I need, allowing switching between various OS and compilers.

A couple of years ago, I planned to move TSDuck to CMake, using "Modern CMake" as it is called. I quickly realized that it created more problems than it solved, even on UNIX alone.

One example, quite important for me: I don't want to list all source files. There are 3700+ source files and 500+ klocs in TSDuck. New source files regularly appear. I am the only maintainer. There are also a couple of regular contributors on specific parts of the source code but I am the only one with the burden of code organization, build and release management (not even part of my paid job). So, I have no time to loose, everything must be automatic. No redundancy allowed. All features are self-registering. The simple presence of a source file somewhere in the directory tree must be sufficient to define how to build it and where to integrate it. No list of source files. No way.

GNU Make and MSBuild can handle that gracefully.

With CMake, it seemed possible but, after some discussions with CMake experts, those zealots (who probably never managed a project with more than 10 source files) explained that this was a bad way of managing projects and that maintaining (redundant) lists of source files was "obviously" the right way to manage a project. I have no time to loose with that kind of guys and I immediately dropped the idea of using CMake.

Using CMake also allows getting external librairies within the tsduck project in some cases (OpenSSL seems to be tricky to cross compile).

This is the theory. In practice, CMake pretends to hide integration problems but they usually blow up in your face on Windows. Not only it does not solve the problems, but it also removes the low-level control to fix them. This is typically the issue that SRT maintainers face with OpenSSL on Windows. The librist, another library which is used by TSDuck, uses meson and it is another kind of nightmare on Windows (see my build script), especially on Arm64 architecture. All those "magic" tools never work correctly on Windows, they are always late compared to Visual Studio and Windows, they cannot anticipate the zillions of ways to integrate third-party libraries.

On the other hand, Visual Studio and MSBuild are always up-to-date and provide all levels of control you need. All problems are directly fixed from the grounds, instead of endlessly search for some indirect workarounds in intermediate tools with little to no added value.

VLC does not support building with MSVC for Windows builds and we usually cross-compile from Linux.

While TSDuck uses the exact opposite...

On the plus side, compiling with more compilers allow fixing issues that other compilers don't see.

In principle, I definitely agree and this is why I always compile with the three compilers, with all warnings on.

However, there is a point where it becomes counter-productive. This is typically what happens here. None of the build issues you encountered and for which you submitted PR's address a problem in the application. They all address integration issues between gcc/clang and Windows. They address the "rubber joint" between two environments which were never designed to work together, despite the considerable amount of work that thousands of people produced to cross compile Linux tools for Windows. In short, this is a waste of time. Both the application code and the developers would have benefited from a native Windows compilation using MSVC.

I have experienced building ffmpeg and a few other tools on Windows and this was an absolute nightmare. This is why I stick to native tools, the "right tools on the right platform". Building TSDuck on any basic Windows system with Visual Studio is as simple as running two scripts, install-prerequisites.ps1 and build-installer.ps1, full stop. No more ffmpeg-like build, never!

I understand that, for VLC, ffmpeg and the like, it is too late because there is just too much legacy build environment on them. However, the recurrent price to pay is very high. Another explanation is that open source developers are ideologically biased toward Linux, deliberately develop for Linux, and let other people with the porting dirt. This is very short-minded. The version 1 of TSDuck (not named TSDuck at the time, not open source either) was developed in C for Linux only. With version 2, I scrapped the code and rewrote everything in C++ with portability in mind, using only native tools on Linux and Windows. The benefits were huge.

Returning to the PR's:

• I am ready to integrate cosmetic definitions or warning suppressions to support clang on Windows.
• About the inlined or implicit public methods calling methods of private inner classes, I prefer to add a TSDUCKDLL attribute to the inner classes. This is already done with public inner classes (because it is required).
• I am reluctant to integrate blind pointer casts. This is a typical case where using more compilers clearly becomes counter-productive. The casts hide potential pointer mismatch on valid platforms just to hide problems on "heretic" compiler/OS configurations. I solved a similar kind of issue in a different way in file tsIPUtils.h to address the WinSocket API which is mostly compatible with the traditional socket API, but carefully incompatible in specific cases.

I will integrate some of the PR's and propose alternatives for others. The problem is that I cannot test your build configuration and I cannot support it in practice.

I will push this on the develop branch only. Let me know if your cases are solved.

The PR has been merged. Because of the rebase before merge, GitHub does not show the PR as "merged" and I need to manually close the PR.

I am interested in how and why TSDuck and VLC can be integrated. Maybe this could influence future features in TSDuck.

I can't share too many details but that's injecting things in a TS stream and detect it and display it on the client side (including mobile devices).

This is the theory. In practice, CMake pretends to hide integration problems but they usually blow up in your face on Windows. Not only it does not solve the problems, but it also removes the low-level control to fix them.

I used to not like CMake but it grew on me. Unlike meson which doesn't allow any hack (no equivalent of file(GLOB) and no way to hack around it for example). With all major IDE's having integrated CMake support it becomes easy to use.

In principle, I definitely agree and this is why I always compile with the three compilers, with all warnings on.

You may have noticed I disabled -Werror and -Weverything for now as it wouldn't build for me. This is also dangerous in general. New compilers tend to add new error detection and find new issues. With this turned on by default the code that builds now may not build with a future compiler, even if you didn't change any line. That should be off by default and only enabled in development mode and in the CI. We do that in VLC or in other libraries.

In short, this is a waste of time

Not for us. We often patch mingw-w64 to add missing parts, and the patch are usually upstreamed (or fixed). We can definitely add the missing headers to mingw-w64 so in the future it builds without patching. For example this patch is not even needed on a recent mingw-w64 since this patch.

Both the application code and the developers would have benefited from a native Windows compilation using MSVC.

You may try to build your Windows code in MSVC in Universal Windows Platform mode (UWP), there are likely forbidden calls here and there in the code. That's the kind of thing we will also need to fix before VLC integration.

As for building VLC with MSVC, I'm all for it. But the Microsoft compiler is lacking some key features of C17 that we use in VLC. So until they fix their compiler it's not possible. However it's possible to use clang and clang-cl with the Windows SDK within Visual Studio to build VLC. So the possibility of switching the compiler with MSVC would be welcome.