-
Notifications
You must be signed in to change notification settings - Fork 548
Bye bye winthreads; relating to #2385 #2409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
I just noticed I botched commit messages on ilmbase and openEXR; they're versions 2.3.0, not 2.0.3. |
You can always reword and force-push, PRs are temporary branches in this project. A few other comments:
|
I was going to do that yesterday but was tired and forgot, actually.
Haven't tested all of them, probably. Any specific ones I should test?
I make my patches using
Will do. Should I use tabs or spaces? Or it doesn't matter as long as each file is consistent with itself? |
It is very similar - the comments give future maintainers context around what can removed from the patch (especially if those have been submitted upstream). In some cases it provides something close to a workflow where mxe patches are just a series of commits that can be rebased.
Always spaces in
It doesn't matter per-se, review the diffs the PR is generating and avoid generating whitespace changes (unless you are certain). |
I really need to get more familiar with MXE targets. I ended up running a regex search for boost among deps and left it running. By default I build most things as shared though so I'll need to try a static build also later. Anyway, if there are build failures you can expect me to either look into them and/or update the problematic packages also. I've always preferred tabs but as I said my editor can go from one to the other easily so I'll fix that later also. |
|
@tonytheodore I was working further on this over the weekend. I had to make some more changes to these and a few other packages but so far most things are building fine, including pcl and wt. I'm currently facing some issues with cgal and libtorrent-rasterbar possibly among a few others. Will rebase and push again once I've got them sorted. Sent with GitHawk |
|
P.S. Jesus. The format patch tool makes work so much more straightforward and easier. I can't believe I was working the badwrong™️ way until now. Sent with GitHawk |
|
Using the format-patch style enables a workflow like: Once you get used to it, git handles lots of things nicely. |
Yeah that's what I've been doing now that I realized how it works. |
69e5f60 to
dfbc7d5
Compare
|
This should be good to go now. I've tested, fixed and updated pretty much every thing that depends on boost. I've also updated cmake and even made some fixes (OpenEXR and IlmBase) I've committed PRs upstream for. It's conflicting with a couple of your updates though, I'm not sure whose commits fixing those issues are better. For openthreads and openscenegraph, I changed the TRY_RUN to TRY_COMPILE... for openal, I remove and added patches as appropriate, but I also fixed one (mmdevapi is actually an old name for wasapi; I'm not sure if you caught that) |
1060538 to
4f6d14d
Compare
|
I know this PR is humongous but think you could take a look at merging it at some point? |
5b02524 to
ee0dd78
Compare
|
@tonytheodore think somebody could take a serious look at this PR? even if you don't end-up merging all of it (some parts already might have become obsolete but some solve problems outstanding on master or a few that were solved already but in different ways) but a lot of work went into making everything play nice with pthreads; most of the updates after that are me having noticed certain broken downstream packages and fixing them. And, there are many plain-old package updates in here also. |
|
Thanks for the updates @Lord-Kamina, I do appreciate the effort that you've put into it and will have a look soon - it's been a long time coming. Large PRs are hard to merge upfront, even when most the changes could be merged individually without much review. Over time, the only (sane) option is to use A couple of pthreads-related cleanups I'll add from a toolchain perspective:
|
|
Keep in mind I've been actively using this branch (or at least an offshoot of it), so I can work to solve current conflicts with master. |
|
There's probably four phases I would approach this in:
It's probably not worth your effort to resolve conflicts since there's also whitespace and style changes, and I'll most likely leave the boost update as a new open PR. |
Too late. Also, keep the following in mind: The reason that this PR is so big is that I began changing stuff and seeing downstream packages broken; so I began fixing those and that eventually turned into updating those many times. The boost update will break many things if you isolate it. |
|
The one thing you'll most likely want to pick-out (unless you changed your mind since I last asked) is that this PR updates the core gcc to gcc8. |
c3ae74d to
ed1865d
Compare
src/boost.mk
Outdated
| newlib=`echo \`basename $${lib}\` | $(AWK) '{gsub(/-mt-x[0-9]{2}/,"-mt"); gsub(/_pthread/,"")}1'`; \ | ||
| echo ln -sf "$${lib}" "$(PREFIX)/$(TARGET)/lib/$${newlib}"; \ | ||
| ln -sf "$${lib}" "$(PREFIX)/$(TARGET)/lib/$${newlib}"; \ | ||
| done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't rename the libs, it's failing here (x86_64-apple-darwin19.5.0):
for lib in `ls "/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib"/libboost_*.a | tr "\n" " "`; do newlib=`echo \`basename ${lib}\` | /opt/local/bin/gawk '{gsub(/-mt-x[0-9]{2}/,"-mt"); gsub(/_pthread/,"")}1'`; echo ln -sf "${lib}" "/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/${newlib}"; ln -sf "${lib}" "/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/${newlib}"; done
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_atomic-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_atomic-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_chrono-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_chrono-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_container-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_container-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_context-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_context-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_contract-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_contract-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_coroutine-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_coroutine-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_date_time-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_date_time-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_exception-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_exception-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_filesystem-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_filesystem-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_graph-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_graph-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_iostreams-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_iostreams-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_locale-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_locale-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_log-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_log-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_log_setup-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_log_setup-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_c99-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_c99-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_c99f-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_c99f-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_c99l-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_c99l-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_tr1-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_tr1-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_tr1f-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_tr1f-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_tr1l-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_math_tr1l-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_prg_exec_monitor-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_prg_exec_monitor-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_program_options-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_program_options-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_random-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_random-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_regex-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_regex-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_serialization-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_serialization-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_stacktrace_basic-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_stacktrace_basic-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_stacktrace_noop-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_stacktrace_noop-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_system-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_system-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_test_exec_monitor-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_test_exec_monitor-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_thread_pthread-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_thread-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_timer-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_timer-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_type_erasure-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_type_erasure-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_unit_test_framework-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_unit_test_framework-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_wave-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_wave-mt-s-x32.a
ln -sf /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_wserialization-mt-s-x32.a /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib/libboost_wserialization-mt-s-x32.a
# setup cmake toolchain
printf "set(Boost_THREADAPI "pthread")\\n set (Boost_USE_STATIC_LIBS ON)\\n set (Boost_USE_MULTITHREADED ON)\\n set (Boost_NO_BOOST_CMAKE OFF)\\n set (Boost_USE_STATIC_RUNTIME ON)\\n" > '/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/share/cmake/mxe-conf.d/boost.cmake'
'i686-w64-mingw32.static-g++' -W -Wall -Werror -ansi -U__STRICT_ANSI__ -pedantic '/Users/tonyt/dev/mxe/src/boost-test.cpp' -o '/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/bin/test-boost.exe' -std='c++11' -lboost_serialization-mt -lboost_thread-mt -lboost_system-mt -lboost_chrono-mt -lboost_context-mt -L'/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.static/lib'
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.static-ld: cannot find -lboost_serialization-mt
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.static-ld: cannot find -lboost_thread-mt
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.static-ld: cannot find -lboost_system-mt
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.static-ld: cannot find -lboost_chrono-mt
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.static-ld: cannot find -lboost_context-mt
collect2: error: ld returned 1 exit status
gmake[1]: *** [Makefile:846: build-only-boost_i686-w64-mingw32.static] Error 1
gmake[1]: Leaving directory '/Users/tonyt/dev/mxe'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Goddamn it, this is a pain in the ass; I changed it like four times last night and eventually left the one that built fine here. Keep in mind, I'm not renaming, I'm adding symlinks.
But I see the issue, I think; the problem is the "s" static suffix; I think we do in fact need to rename them to remove that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd be happy to remove the manual test altogether and instruct users to use the provided cmake routines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would probably be a more robust solution in the long-term.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If only Boost had decent pkg-config support.
src/boost.mk
Outdated
| $(PKG)_TARGETS := $(BUILD) $(MXE_TARGETS) | ||
| $(PKG)_DEPS := cc bzip2 expat zlib | ||
| $(PKG)_DEPS := cc bzip2 expat zlib xz | ||
| AWK = $(shell which $(shell gawk --help >/dev/null 2>&1 && echo g)awk) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add such variables near the top of the main Makefile.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't fix it until tomorrow but will do.
src/ffmpeg.mk
Outdated
| --disable-debug \ | ||
| --disable-pthreads \ | ||
| --enable-w32threads \ | ||
| --disable-$(if $(findstring posix, $(MXE_GCC_THREADS)),w32threads,pthreads) \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replace MXE_GCC_THREADS test with either WIN32_THREADS or POSIX_THREADS tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tonytheodore What about cases where I just want to check threading against no threading? Should I use $(or) or will MXE_GCC_THREADS be good for those?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or will MXE_GCC_THREADS be good for those
MXE_GCC_THREADS is compiler-specific, consider the case where it isn't set.
Short-circuit $(or) is ideal, though nested ifs may be more readable in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're unsure, default to pthreads.
I'm pretty sure most w32threads are placeholders.
src/ilmbase.mk
Outdated
| $(PKG)_GH_CONF := AcademySoftwareFoundation/openexr/releases,v | ||
| $(PKG)_FILE := openexr-v$($(PKG)_VERSION).tar.gz | ||
| $(PKG)_SUBDIR := openexr-$($(PKG)_VERSION) | ||
| $(PKG)_DEPS := cc $(BUILD)~cmake |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$(BUILD)~cmake is added automatically
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok
src/nsis.mk
Outdated
| $(PKG)_VERSION := 3.05 | ||
| $(PKG)_CHECKSUM := b6e1b309ab907086c6797618ab2879cb95387ec144dab36656b0b5fb77e97ce9 | ||
| $(PKG)_VERSION := 3.04 | ||
| $(PKG)_CHECKSUM := 609536046c50f35cfd909dd7df2ab38f2e835d0da3c1048aa0d48c59c5a4f4f5 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this and the scons to a new PR and base your changes after the parallel build fix.
3.3.0 failed to compile against OpenEXR 2.5.1
There are still a few left, I only did it for the ones I’d already modified in this PR
In a nutshell, the problems were two: 1) The test was building with gcc, and assimp is now using c++11, which was producing a myriad issues; I tried explicitly compiling as C11 but still no-go, so I changed the test to use the c++ API. 2) IrrXML is now built as a separate library, and that was also producing linking errors.
Also build tests with C++11 and make sure to pass the TryRun results.
Decided to go with this because this thing has been a bug upstream for years in several different forms. Point is, I don’t think they can be trusted with IrrXML and since it’s a small library, it’s just cleaner to add it to mxe and declare it as a dependency for assimp.
What a bitch it was to get this working properly; and all so probably nobody will ever use it. -_-
Now that gtk2 shared builds work, so do gtkmm2 shared builds.
The parentheses patch was fixed upstream in 3.23.0, but I’d rather not upgrade to an unstable release, and upgrading to the 3.24 series would require updating gdk-pixbuf as well; maybe some other time.
I’m not entirely sure any longer that the addition to LIBS is relevant, but I don’t want to rebuild it YET AGAIN. Most likely the underlying error had to do with gtk3’s parentheses issue.
By skipping download, CMake instead calls pkg-config, which is what we want.
Not strictly related to the PR; updated attempting to fix an error that turned out to be ccache's fault.
Not strictly related to the PR; updated attempting to fix an error that turned out to be ccache's fault.
d29ff9e to
c233ad0
Compare
|
Conflicts solved again. |
|
I agree. This needs to be merged. EDIT: But it has file overlap with my GTK update PR for sure. Since I was migrating the whole GTK stack to the meson build system. See: melroy89#1 |
So, here's a big one...
I removed essentially all instances of hardcoded win32 threads; adding the necessary patches where required.
Additionally, I updated some of the packages as well, most notably boost which is now at 1.71.0 and building properly against pthreads. 🎉