Handle fetching IBM MQ libraries in CI package build jobs outside of CMake#21862
Open
Ferroin wants to merge 3 commits intonetdata:masterfrom
Open
Handle fetching IBM MQ libraries in CI package build jobs outside of CMake#21862Ferroin wants to merge 3 commits intonetdata:masterfrom
Ferroin wants to merge 3 commits intonetdata:masterfrom
Conversation
This allows us to handle fetching them as a separate step in CI, letting us better handle the errors resulting from IBM’s questioanble infrastructure.
This lets us better handle infrastructure issues when trying to fetch the libraries.
Contributor
There was a problem hiding this comment.
1 issue found across 8 files
Confidence score: 3/5
- There is a concrete configuration failure risk:
packaging/cmake/Modules/NetdataIBMPlugin.cmakeusesCOMMANDin an ExternalProject_Add context, which is not a valid option and can break the CMake configure step. - Given the medium severity (6/10) and direct impact on build/configuration, this carries some user-facing risk despite being localized.
- Pay close attention to
packaging/cmake/Modules/NetdataIBMPlugin.cmake- ensure the external project step uses a supported option like PATCH_COMMAND.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packaging/cmake/Modules/NetdataIBMPlugin.cmake">
<violation number="1" location="packaging/cmake/Modules/NetdataIBMPlugin.cmake:23">
P2: FetchContent_Declare passes arguments to ExternalProject_Add, but "COMMAND" is not a valid ExternalProject_Add option, so CMake will fail configuring this external project. Use PATCH_COMMAND (or another supported step) for the extra copy operation.</violation>
</file>
Architecture diagram
sequenceDiagram
participant CI as CI Runner (GHA)
participant Script as prep-ibm-mq-libs.sh
participant Cache as Artifact Cache
participant IBM as IBM CDN
participant CMake as CMake (FetchContent)
participant Build as Plugin Build (CGO)
Note over CI, Build: NEW: IBM MQ Library Prefetching Flow
CI->>Script: Execute prep script
Script->>Script: Read URL/SHA256 from packaging/vendor/
Script->>Cache: Check for cached tarball
alt Cache Miss or Invalid Hash
loop NEW: Retry up to 5 times
Script->>IBM: GET IBM MQ Redistributable
IBM-->>Script: Tarball data
end
Script->>Script: Verify SHA256 hash
Script->>Cache: Update cache with valid tarball
else Cache Hit
Cache-->>Script: Provide cached tarball
end
Script->>Script: Extract to tmp/ibm_mq
Note over CI, CMake: Start Build Stage
CI->>CMake: Invoke with -DFETCHCONTENT_SOURCE_DIR_IBM_MQ=tmp/ibm_mq
activate CMake
alt CHANGED: Pre-populated source provided
CMake->>CMake: NEW: Copy libs from tmp/ibm_mq to build dir
Note right of CMake: Skips network download
else Standard build
CMake->>IBM: Fetch via FetchContent_MakeAvailable
end
CMake->>Build: Configure CGO_CFLAGS/LDFLAGS with local paths
deactivate CMake
Build->>Build: Compile ibm.d.plugin using local headers/libs
Build-->>CI: Build Success
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| SOURCE_DIR "${IBM_MQ_BUILD_DIR}" | ||
| CONFIGURE_COMMAND "${CMAKE_COMMAND} -E true" | ||
| CONFIGURE_COMMAND "${CMAKE_COMMAND} -E copy_directory bin gskit8 inc java lap lib lib64 licenses msg samp swidtag ${IBM_MQ_BUILD_DIR}" | ||
| COMMAND "${CMAKE_COMMAND} -E copy MANIFEST MANIFEST.Redist README.Redist ${IBM_MQ_BUILD_DIR}" |
Contributor
There was a problem hiding this comment.
P2: FetchContent_Declare passes arguments to ExternalProject_Add, but "COMMAND" is not a valid ExternalProject_Add option, so CMake will fail configuring this external project. Use PATCH_COMMAND (or another supported step) for the extra copy operation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packaging/cmake/Modules/NetdataIBMPlugin.cmake, line 23:
<comment>FetchContent_Declare passes arguments to ExternalProject_Add, but "COMMAND" is not a valid ExternalProject_Add option, so CMake will fail configuring this external project. Use PATCH_COMMAND (or another supported step) for the extra copy operation.</comment>
<file context>
@@ -12,19 +13,37 @@ macro(add_ibm_plugin_target)
SOURCE_DIR "${IBM_MQ_BUILD_DIR}"
CONFIGURE_COMMAND "${CMAKE_COMMAND} -E true"
+ CONFIGURE_COMMAND "${CMAKE_COMMAND} -E copy_directory bin gskit8 inc java lap lib lib64 licenses msg samp swidtag ${IBM_MQ_BUILD_DIR}"
+ COMMAND "${CMAKE_COMMAND} -E copy MANIFEST MANIFEST.Redist README.Redist ${IBM_MQ_BUILD_DIR}"
BUILD_COMMAND "${CMAKE_COMMAND} -E true"
INSTALL_COMMAND "${CMAKE_COMMAND} -E true"
</file context>
Suggested change
| COMMAND "${CMAKE_COMMAND} -E copy MANIFEST MANIFEST.Redist README.Redist ${IBM_MQ_BUILD_DIR}" | |
| PATCH_COMMAND "${CMAKE_COMMAND} -E copy MANIFEST MANIFEST.Redist README.Redist ${IBM_MQ_BUILD_DIR}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This uses CMake’s
FETCHCONTENT_SOURCE_DIR_*functionality plus a bit of additional trickery to allow usage of a pre-populated copy of the IBM MQ libraries in builds, and then leverages this to fetch the IBM MQ libraries before the build in our native DEB/RPM package builds in CI.This should completely eliminate (or almost completely eliminate) the intermittent failures we’re seeing on these CI jobs resulting from failures to fetch these libraries.
Test Plan
CI passes on this PR.
Summary by cubic
Move IBM MQ library fetching out of CMake and into CI for DEB/RPM builds to reduce flaky package builds. We prefetch, verify, cache, and extract the redistributable, and point CMake at a pre-populated source when present.
Refactors
Bug Fixes
Written for commit 923e857. Summary will update on new commits.