-
Notifications
You must be signed in to change notification settings - Fork 33
feat: refactor TFTP/DHCP handling and add pxelinux.cfg support for RPI/U-Boot #443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #443 +/- ##
==========================================
- Coverage 52.89% 52.02% -0.88%
==========================================
Files 107 114 +7
Lines 7177 7524 +347
==========================================
+ Hits 3796 3914 +118
- Misses 3095 3314 +219
- Partials 286 296 +10 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Hey @appkins. Thanks for this. This is a lot though and has many significant changes. Would you mind breaking out the changes into individual PRs, please? I recommend a PR for the CI/CD changes, one for the goreleaser changes, one for the Hook OCI downloading and file serving, one for the DHCP changes, and one for the TFTP changes. Also, there is only a single commit for all these changes. We don't squash PRs when merging to main. This mean that bisecting issues for all these changes would be quite difficult. I'm not suggesting tiny commits for every little thing but the magnitude of the changes aren't reflected in the number of commits. Maybe with all the different features broken out into individual PRs will help. This is going to need some thorough documentation too. It would be nice to understand some things like what this enables, the supported hardware, how to get it all working, walkthroughs, diagrams, info on dependencies like OCI registries, etc. |
…Boot Major changes: TFTP/DHCP Enhancements: - Add dynamic hardware type handling in DHCP option 209 (PXELinuxConfigFile) - Implement TFTP handler for pxelinux.cfg/<hwtype>-<mac> format - Add U-Boot client support (option 60) and architecture handling - Make Raspberry Pi detection public (IsRaspberryPI) - Add proper architecture-specific bootfile handling (ARM64, ARM32) TFTP Server Refactoring: - Extract TFTP multiplexer to smee/internal/tftp package - Add pattern-based routing with regex support - Implement middleware for logging and request tracking - Add hook file handler for serving from cache directory - Add TFTP configuration options (Anticipate, CacheDir) - Consolidate TFTP handlers with HTTP handlers Hook Service (NEW): - Add hook/hook.go for OCI image management - Support for pulling and serving hook files via HTTP/TFTP - Background image pulling with caching - Comprehensive test coverage (765 lines) Code Quality & Testing: - Add extensive tests for TFTP pattern matching (tftp_test.go) - Fix test compatibility issues (macOS loopback interface) - Add tests for hardware type validation and error handling - Improve shell script robustness (proper quoting, shellcheck compliance) Protobuf & Formatting: - Regenerate protobuf files with proper import ordering - Fix indentation in .proto files (edition 2023 format) - Standardize code formatting across shell scripts Minor Fixes: - Fix naked return statements in xff.go - Clean up unused code and improve error handling - Add proper platform-agnostic loopback interface detection - Fix syslog test for available port usage Breaking Changes: - TFTP server now requires explicit handler registration - Binary TFTP handler moved from binary/tftp.go to binary.Handler.HandleTFTP - Script TFTP handler now supports pxelinux.cfg format This refactoring improves modularity, adds U-Boot support, and prepares the codebase for more flexible boot configuration scenarios including ARM devices and network appliances. Signed-off-by: tinkerbell-community <[email protected]>
Lol don't worry I'm fully aware. I put it in as a draft PR so it's on the radar and we can get a discussion moving forward. There's a lot of stuff I tackled in this PR that directly or indirectly solve some of the roadmap items. It will take me quite a while to split everything out and thoroughly document it. If there's a feature that you're particularly interested in having sooner, I'd be happy to split that out into a PR.
I'm happy to close this PR and break it up into a few others, but I'm also actively developing the platform to suit my environment. If we can identify some elements that can go in with lower risk, I'd be happy to submit a PR, merge them, and not experience rebase hell - the reason for the single commit.
Sorry about all the workflow changes in these PRs. I had to change some things to get my fork set up for testing and then went crazy. To keep the initial response relevant and clear, I'm going to post additional info in another comment below |
Additional ContextI'm using this effectively as a playground for my home data center that consists of well over 30 nodes that I like to re-provision frequently. After years of research and development across a wide range of cloud native platforms, I'm settling on this one for its elegance, simplicity, and clarity. That being said, I have a very wide set of features to share back to this project if you will have them. I've been developing in a real world environment, which is what prompted many of these changes. My aim with these changes is to provide the breadth of HookOS and tink workflows to the DHCP, TFTP and HTTP components. Feature set:
Overall, the changes are stable and efficient in my testing environment. Extended Ecosystem Features
Anyway, as I do work full time in the Cloud Native space it will be hard to find time to coordinate on triaging all of this. If you could give me a general order of priorities from the feature list, that would help me commit to getting a few of the PRs ironed out |
|
Important detail - When I break up the PRs, I would like them to be locked to save hours of my time. I'm happy to cherry pick the relevant elements, but would prefer to manage all rebasing/merging in that isolated branch. What I want to avoid is having to merge the PR into my development branch to properly test the changes, then cherry pick again, then another potential merge from Dependabot. If the staff has availability to assist with follow up on the PRs, that would be appreciated. |
|
This is, wow, a lot. @appkins you should join out Slack and the TB community meeting, maybe we can dissect this a bit. Exciting, as I've been experimenting on TB + u-boot/pxelinux.cfg myself. |
Description
Major changes:
TFTP/DHCP Enhancements:
TFTP Server Refactoring:
Hook Service (NEW):
Code Quality & Testing:
Protobuf & Formatting:
Minor Fixes:
Breaking Changes:
This refactoring improves modularity, adds U-Boot support, and prepares the codebase for more flexible boot configuration scenarios including ARM devices and network appliances.
Fixes: #
How Has This Been Tested?
How are existing users impacted? What migration steps/scripts do we need?
Checklist:
I have:
This pull request makes significant improvements to the CI/CD pipeline, build tooling, and containerization for the project. The changes modernize the workflow by introducing GoReleaser for builds and releases, streamline binary management, improve caching and linting, and enhance Docker image creation with statically linked dependencies. Additionally, the Makefile and configuration files have been refactored for clarity and maintainability.
CI/CD Workflow Overhaul
.github/workflows/ci.yamlwith GoReleaser for building, packaging, and releasing binaries and container images. This includes signing images with Cosign and simplifying the workflow steps and dependencies. [1] [2]lint,generate,validation, andvalidate-helm-chartjobs; improves caching strategies and removes redundant cache restore/save steps. [1] [2]release-notesjob and its artifact handling, further simplifying the release process.Build Tooling and Makefile Improvements
Makefile, with new variables and targets for tool binaries, and updates the default build target to use GoReleaser.bindirectory and updating related variables.Docker Image and Binary Management
Dockerfile.tinkerbellto use a multi-stage build for statically compilingipmitool, ensuring the final image contains a static binary and required certificates.Dockerfile.agentto use the new binary path structure for copying agent binaries.Configuration and Linting Adjustments
.golangci.ymlto increase the minimum complexity threshold for thegocognitlinter.Makefileto use a dynamic repository owner.These changes collectively modernize and simplify the build and release process, improve maintainability, and ensure consistent, statically linked binaries and container images.
This pull request introduces significant improvements to the build, release, and CI/CD pipeline for the project. The main changes include a migration to GoReleaser for building and publishing binaries and container images, a refactor of the GitHub Actions workflow to streamline jobs and caching, and an update to the Dockerfiles for more reliable static builds. It also introduces UPX compression for binaries and updates Makefile tooling and variables for better maintainability.
Build and Release Automation:
.goreleaser.yamlconfiguration to automate building, compressing, and publishing binaries and container images using GoReleaser, including UPX compression and static linking for Linux binaries.CI/CD Workflow Refactor:
.github/workflows/ci.yamlto replace custom build and cache steps with GoReleaser, streamlined job structure (lint, generate, validation, build/publish), and improved caching and artifact handling. [1] [2] [3] [4] [5]Container/Docker Improvements:
Dockerfile.tinkerbellto build a statically linkedipmitoolbinary in a multi-stage process for improved reliability and reproducibility.Dockerfile.agentto use the new build output directory structure for binaries.Configuration Updates:
gocognitlinter in.golangci.ymlfor more robust code analysis.