Tags: vapor/vapor
Tags
Fix parallel build failures on platforms with Glibc (#3393) ## Motivation Since adding `MemberImportVisibility`, when Vapor is compiled in highly parallel environments it fails with high probability: ```console % git rev-parse HEAD ac3aeb7 % rm -rf .build ~/.cache/org.swift.swiftpm/manifests/ && swift build -j 64 ... Building for debugging... /pwd/Sources/Vapor/Utilities/String+IsIPAddress.swift:10:24: error: initializer 'init()' is not available due to missing import of defining module 'CNIOLinux' [#Membe rImportVisibility] 1 | import Foundation 2 | import NIOCore 3 | #if canImport(Android) | `- note: add import of module 'CNIOLinux' 4 | import Android 5 | #endif : 8 | func isIPAddress() -> Bool { 9 | // We need some scratch space to let inet_pton write into. 10 | var ipv4Addr = in_addr() | `- error: initializer 'init()' is not available due to missing import of defining module 'CNIOLinux' [#MemberImportVisibility] 11 | var ipv6Addr = in6_addr() 12 | ---[ similar error for in6_addr too ]--- ``` Building with `-j 1` always succeeds. I've spent quite some time looking into this and it appears that we're hitting a combination of: 1. The Glibc module map is broken. It only declares a top-level `SwiftGlibc.h`, which then `#include`s many of the system headers—relevant to this this issue, it includes `in.h`. As a result of this investigation, @al45tair has filed swiftlang/swift#85427, but we'll need a solution in the interim. 2. The CNIOLinux non-product C target of Swift NIO also has an umbrella `CNIOLinux.h` header, which also `#include`s many headers, including a transitive include of `in.h`. 3. Vapor has started building with `MemberImportVisibility`. We can see in the above error that the Swift compiler is suggesting to add `import CNIOLinux`, which is a dubious suggestion, since you'd expect the fix to be to add an `import Glibc` or similar. However, if we poke around at the failed build output we can see that `in.h` is _only_ part of the `CNIOLinux` and _absent_ from `Glibc`: ```console % find .build/aarch64-unknown-linux-gnu/debug/ModuleCache/ -name '*.pcm' | grep -e NIOLinux -e Glibc | xargs -n 1 -t clang -cc1 -module-file-info 2>&1 | grep -e "clang -cc1" -e "Input file:.*/in\\.h" clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/SwiftGlibc-IGSGJKVMPVR1.pcm clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/CNIOLinux-1MNQXZXDCO5H5.pcm Input file: /usr/include/linux/in.h [System] clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/SwiftGlibc-IGSGJKVMPVR1.pcm clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/CNIOLinux-1MNQXZXDCO5H5.pcm Input file: /usr/include/linux/in.h [System] clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1O3SENRIBS9OC/SwiftGlibc-IGSGJKVMPVR1.pcm ``` Compare this to the build output of a successful build (`-j 1`), where we can see that `in.h` is part of `Glibc`: ```console % find .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/ -name '*.pcm' | grep Glibc | xargs -n 1 -t clang -cc1 -module-file-info 2>&1 | grep -e "clang -cc1" -e "Input file:.*/in\\.h" clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/SwiftGlibc-IGSGJKVMPVR1.pcm Input file: /usr/include/netinet/in.h [System] Input file: /usr/include/bits/in.h [System] clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/SwiftGlibc-IGSGJKVMPVR1.pcm Input file: /usr/include/netinet/in.h [System] Input file: /usr/include/bits/in.h [System] clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/1O3SENRIBS9OC/SwiftGlibc-IGSGJKVMPVR1.pcm Input file: /usr/include/netinet/in.h [System] Input file: /usr/include/bits/in.h [System] ``` What's happening is that a header file can only belong to exactly one Clang module and how header files are attributed to modules is non-deterministic in some cases. `CNIOLinux` is an implicit module——it does _not_ have a `.modulemap`——and is inferred from the presence and contents of `Sources/CNIOLinux/include/CNIOLinux.h`. `Glibc` is an explicit module——it _does_ have a `.modulemap`——but its module map does not explicitly list all the headers that belong to the module. It only lists `SwiftGlibc.h`. Header files that are explicitly listed in a module map are deterministically attributed to that module, but header files that are transitively included using `#include` are attributed to modules on a first-come-first-attributed basis. To illustrate this, I can make the `-j64` build succeed with the following hot-patch of the `Glibc.modulemap` in my container: ```diff module SwiftGlibc [system] { // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 23) link "m" // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 26) link "pthread" // FIXME: util contains rarely used functions and not usually needed. Unfortunately // link directive doesn't work in the submodule yet. // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 30) link "util" // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 33) // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 35) link "dl" // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 37) // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 43) header "SwiftGlibc.h" + header "netinet/in.h" // <assert.h>'s use of NDEBUG requires textual inclusion. textual header "assert.h" export * } ``` We'll have to track swiftlang/swift#85427 for a real fix for this issue, but in the meantime, we'll need a workaround in Vapor if we want it to work with the `MemberImportVisibility` compiler setting. As it happens, it looks like the compiler hint——`import CNIOLinux`——might be the best we can do for today, and it turns out that there are other projects that are downstream of NIO that added this when they added `MemberImportVisibilty` too: - apple/swift-nio-ssl#497 - apple/swift-nio-extras#240 - swift-server/async-http-client#794 ## Modifications Add missing imports, with a link to the Swift issue to track. ## Result Vapor will now reliably build when built in parallel.
Fix a couple of import issues (#3390) * Fix an error on some versions of Linux with imports * Fix Sendable issue on old macOS SDK * Try just NIOCore * Update Sources/Vapor/Security/OTP.swift Co-authored-by: Gwynne Raskind <[email protected]> --------- Co-authored-by: Gwynne Raskind <[email protected]>
Expose peer certificates in request handlers (#3362) * Expose peer certificates in request handlers Additional certificate information can be relevant in mTLS deployments. Exposing the validated certificate chain to the peer makes them available per request. This PR exposes the validated peer certificate chain, i.e., the certificates that establish trust of the peer identity (from the leaf to and including the root certificate). * Address API breakage * Add Swift ASN1 to server tests --------- Co-authored-by: Tim Condon <[email protected]>
Update minimum Swift version to 6.0 (#3365) * Update package minimum Swift version to 6.0. Update source code accordingly, and add theming to API docs. * Brutally hack around SymmetricKey not being Sendable in SwiftCrypto * Update dependabot.yml * Sendable stinks. * There, that should do it * Update ServerTests.swift * Update URITests.swift --------- Co-authored-by: Tim Condon <[email protected]>
PreviousNext