Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@shykes
Copy link
Contributor

@shykes shykes commented Nov 7, 2025

The time has finally come to break up our CI monolith into cleanly separated toolchains. This will make our CI simpler, smaller, and hopefully faster! It will also finally allow us to show our own CI as a "golden example" of how to use Dagger. It has been a long road getting to this moment... 😭

Goals:

  1. Faster CI
  2. Simpler CI
  3. A golden example of Dagger best practices

TODO

LATER

  • Open issue in dagger/dagger fix toolchain config (default values can only be strings)
  • Open issue to allow shadowing core types with a toolchain (for dagger call engine instead of dagger call engine-dev)
  • Convert docs/recorder to a toolchain (wip @eunomie)
  • Translate more modules to dang for load speed
  • Remove .gitignore from our modules; add a generate function; stop generating on-the-fly on go lint (faster!)
  • Ditch golangci-lint, we don't need a dumb meta-linter, we have dagger to glue stuff together already. Use staticcheck
  • [core] Rename "toolchain configurations" to "toolchain customizations"

DONE

  • Configure docker auth in CI environment
  • Check that all modules actually work (harder because of lazy loading) (should be fixed by feat: add eager-mod-loading flag #11413)
  • Re-enable bump in elixir-sdk and typescript-sdk
  • Fix timeouts in engine tests. example
  • Harmonize dependency pins for load speed
  • Make sure all checks are exposed as such (wip @shykes )
  • Fx error in typescript-sdk:typecheck
  • Vendor all external module dependencies
  • Fix display of toolchain descriptions (in dagger functions and dagger toolchain list) (done @eunomie)
  • Fix error in typescript-sdk:test
  • Spin out elixir-sdk-dev and absorb native toolchain (wip @kpenfound)
  • 'dagger develop' wipes toolchain config #11433
  • Fix error in php-sdk-dev:generate (done @TomChv)
  • all-sdk toolchain: actually include all SDKs (done @TomChv)
  • Fix commented-out code in typescript SDK
  • Fix module loading error in .github/workflows/publish.yml
  • Change CI configuration to call dagger checks
  • Test with new dagger checks Switch checks to @check pragma, improve UI #11374
  • Fix missing files error in dagger call php-sdk generate (wip @shykes)
  • Fix broken json-schema generation in docs-dev (move it to engine-dev, for correct source access)
  • Fix toolchain configuration for go lint and go check-tidy
  • Fix performance bottleneck when engine loads toolchains
  • Fix display of toolchains list, too verbose
  • Spin out engine-dev
  • Spin out cli-dev
  • Spin out docs-dev
  • Spin out ci
  • Spin out Go toolchain
  • Spin out go-sdk-dev
  • Spin out python-sdk-dev and absorb native toolchain
  • Spin out typescript-sdk-dev and absorb native toolchain
  • Spin out php-sdk-dev and absorb native toolchain
  • Spin out rust-sdk-dev and absorb native toolchain
  • Spin out java-sdk-dev and absorb native toolchain
  • Spin out dotnet-sdk-dev and absorb native toolchain
  • Spin out helm-dev
  • Spin out scripts to installers
  • Convert .changes to a toolchain (changelog)
  • Convert releaser to a toolchain (release)
  • Convert scan.go to a toolchain (security)
  • Remove top-level constructors
  • Remove all non-builtin SDKs, for load time

@shykes shykes force-pushed the ci-faster-load branch 2 times, most recently from 79a732a to 9148da7 Compare November 12, 2025 07:31
@TomChv

This comment was marked as resolved.

@shykes

This comment was marked as resolved.

@wingyplus
Copy link
Contributor

May I know the reason why convert the sdk-dev to Go? I remember that it’s use as a dogfood the sdk itself.

However, I can help converting Elixir to Go if you preffered.

@shykes

This comment was marked as resolved.

@shykes shykes force-pushed the ci-faster-load branch 3 times, most recently from abe1f6d to 7912565 Compare November 13, 2025 08:04
@TomChv

This comment was marked as resolved.

@TomChv

This comment was marked as resolved.

@kpenfound

This comment was marked as resolved.

@shykes

This comment was marked as resolved.

@shykes

This comment was marked as resolved.

@TomChv

This comment was marked as resolved.

@TomChv TomChv mentioned this pull request Nov 17, 2025
6 tasks
@shykes shykes force-pushed the ci-faster-load branch 3 times, most recently from 0bf1b89 to 850882a Compare November 18, 2025 00:40
@shykes shykes marked this pull request as ready for review November 18, 2025 00:41
@shykes shykes requested review from a team as code owners November 18, 2025 00:41
@shykes shykes changed the title Split up the CI monolith to make it faster Kill the monolith Nov 18, 2025
kpenfound and others added 23 commits November 24, 2025 09:25
Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Solomon Hykes <[email protected]>
This avoids triggering an unresolved issue with 3 layers of nested dagger engines (dagger-in-dagger-in-dagger).

Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Yves Brissaud <[email protected]>
Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Solomon Hykes <[email protected]>
…engine-dev installclient

Signed-off-by: Solomon Hykes <[email protected]>
@shykes
Copy link
Contributor Author

shykes commented Nov 24, 2025

All green, let's do this!

@shykes shykes merged commit b842a45 into dagger:main Nov 24, 2025
65 checks passed
raykao pushed a commit to raykao/dagger that referenced this pull request Dec 9, 2025
* CI: kill the monolith

Signed-off-by: Solomon Hykes <[email protected]>

* feat: merge php sdk  dev inside toolchain

Signed-off-by: Tom Chauveau <[email protected]>

* feat: add dotnet-sdk-dev toolchain

Signed-off-by: Tom Chauveau <[email protected]>

* feat: add dotnet SDK

Signed-off-by: Tom Chauveau <[email protected]>

* chore: remove old dotnet SDK

Signed-off-by: Tom Chauveau <[email protected]>

* chore: convert dotnet-sdk-dev to dang

Signed-off-by: Tom Chauveau <[email protected]>

* feat: add rust-sdk-dev toolchain

Signed-off-by: Tom Chauveau <[email protected]>

* adding java-sdk-dev toolchain

Signed-off-by: Yves Brissaud <[email protected]>

* WIP: appease go linter

Signed-off-by: Solomon Hykes <[email protected]>

* go mod tidy

Signed-off-by: Solomon Hykes <[email protected]>

* fix docs toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* modules/go: customize display of linting "sub-checks"

Signed-off-by: Solomon Hykes <[email protected]>

* fix dependency issue in monolith

Signed-off-by: Solomon Hykes <[email protected]>

* fix linting across toolchains

Signed-off-by: kpenfound <[email protected]>

* new toolchain: release

Signed-off-by: Solomon Hykes <[email protected]>

* Change Github Actions to use 'dagger checks' instead of 'dagger call'

Signed-off-by: Solomon Hykes <[email protected]>
Co-authored-by: Kyle Penfound <[email protected]>

* Harmonize version of dang SDK

NOTE: this version does not work with dagger v0.19.6 and below.

Signed-off-by: Solomon Hykes <[email protected]>

* Annotate missing checks

Signed-off-by: Solomon Hykes <[email protected]>

* remove dead dependencies

Signed-off-by: Solomon Hykes <[email protected]>

* wip elixir dev module

Signed-off-by: kpenfound <[email protected]>

* toolchain: typescript-sdk: uncomment docs lint check

Signed-off-by: Solomon Hykes <[email protected]>

* Fix typescript-sdk toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* CI: disable test-loading of removed modules

Signed-off-by: Solomon Hykes <[email protected]>

* improve Python SDK toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* Polish the Go SDK toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* chore: fix php sdk dev

Signed-off-by: Tom Chauveau <[email protected]>

* chore: add missing SDK to all-sdks module list

Signed-off-by: Tom Chauveau <[email protected]>

* chore: fix the rust SDK dev

Signed-off-by: Tom Chauveau <[email protected]>

* elixir-dev: use File.withReplaced for publish and syncImage

Signed-off-by: kpenfound <[email protected]>

* chore: push fix attempt on php sdk (not working yet)

Signed-off-by: Tom Chauveau <[email protected]>

* chore: cleanup php sdk dev

Signed-off-by: Tom Chauveau <[email protected]>

* wip: attach dev engine to typescript SDK tests

Signed-off-by: Solomon Hykes <[email protected]>

* monolith: comment code

Signed-off-by: Solomon Hykes <[email protected]>

* typescript SDK toolchain: fix bun tests

Signed-off-by: Solomon Hykes <[email protected]>

* typescript-sdk toolchain: expose bunjs-dev-container

Signed-off-by: Solomon Hykes <[email protected]>

* gha: remove concurrency group in checks

Signed-off-by: Solomon Hykes <[email protected]>

* python-sdk toolchain: expose python dev container for debugging

Signed-off-by: Solomon Hykes <[email protected]>

* python-sdk toolchain: remove typecheck() from exposed checks

Signed-off-by: Solomon Hykes <[email protected]>

* go mod tidy

Signed-off-by: Solomon Hykes <[email protected]>

* ci: run checks on latest main commit

Signed-off-by: Solomon Hykes <[email protected]>

* fix toolchain description display

Signed-off-by: Yves Brissaud <[email protected]>
Co-authored-by: Solomon Hykes <[email protected]>

* fix: lint issues

Signed-off-by: Yves Brissaud <[email protected]>

* fix(modules/go): optional include/exclude for `modules` list

Signed-off-by: Yves Brissaud <[email protected]>

* fix: left over from previous lint commit

Signed-off-by: Yves Brissaud <[email protected]>

* go mod tidy without integration ignored

Signed-off-by: kpenfound <[email protected]>

* only ignore core/integration/testdata for go toolchain

Signed-off-by: kpenfound <[email protected]>

* ignore only testdata modules

Signed-off-by: kpenfound <[email protected]>

* helm-dev toolchain: vendor k3s dependency

Signed-off-by: Solomon Hykes <[email protected]>

* python-sdk-dev toolchain: vendor dockerd dependency

Signed-off-by: Solomon Hykes <[email protected]>

* fix testdata linting

Signed-off-by: kpenfound <[email protected]>

* the key to the universe

Signed-off-by: kpenfound <[email protected]>

* unvendor dang and fix linting

Signed-off-by: kpenfound <[email protected]>

* elixir-sdk: implement bump()

Signed-off-by: Solomon Hykes <[email protected]>

* typescript sdk: implement bump()

Signed-off-by: Solomon Hykes <[email protected]>

* dotnet SDK: add (no-op) bump function

Signed-off-by: Solomon Hykes <[email protected]>

* rust sdk: fix bump()

Signed-off-by: Solomon Hykes <[email protected]>

* ci: run linters in bootstrap, instead of SDK tests

This avoids triggering an unresolved issue with 3 layers of nested dagger engines (dagger-in-dagger-in-dagger).

Signed-off-by: Solomon Hykes <[email protected]>

* sdk/java: add fmt command

Signed-off-by: Yves Brissaud <[email protected]>

* convert toolchains/security to dang

Signed-off-by: Yves Brissaud <[email protected]>

* convert toolchains/changelog to dang

Signed-off-by: Yves Brissaud <[email protected]>

* python-sdk toolchain: remove over-complicated "dockerfile extractor", just hardcode uv image

Signed-off-by: Solomon Hykes <[email protected]>

* go-sdk: expand ignore filter to include new codegen dependencies

Signed-off-by: Solomon Hykes <[email protected]>

* Caught stealthy typo which caused ignore filters to be... ignored

Signed-off-by: Solomon Hykes <[email protected]>

* CI: .env.gha: env-specific configuration for our Github Actions CI environment

Signed-off-by: Solomon Hykes <[email protected]>

* engine-dev toolchain: relax source filter to un-break the build

Signed-off-by: Solomon Hykes <[email protected]>

* CI: update "outer engine" pinned commit

Signed-off-by: Solomon Hykes <[email protected]>

* CI: hardcode absolute path of docker config

Signed-off-by: Solomon Hykes <[email protected]>

* typescript sdk: remove duplicate code, fix weird dang error, simplify

Signed-off-by: Solomon Hykes <[email protected]>

* unbreak rebase

Signed-off-by: Solomon Hykes <[email protected]>

* go toolchain: fancier output + limited parallelism of 'check-tidy' function

Signed-off-by: Solomon Hykes <[email protected]>

* CI: bump outer engine version

Signed-off-by: Solomon Hykes <[email protected]>

* fix wrong ignore filter, which caused seemingly unrelated shell auto-complete error

Signed-off-by: Solomon Hykes <[email protected]>

* tentative fix for php-sdk:generate error caused by missing secret in engine-dev installclient

Signed-off-by: Solomon Hykes <[email protected]>

---------

Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Tom Chauveau <[email protected]>
Signed-off-by: Yves Brissaud <[email protected]>
Signed-off-by: kpenfound <[email protected]>
Signed-off-by: Yves Brissaud <[email protected]>
Co-authored-by: Tom Chauveau <[email protected]>
Co-authored-by: Yves Brissaud <[email protected]>
Co-authored-by: kpenfound <[email protected]>
Co-authored-by: Yves Brissaud <[email protected]>
raykao pushed a commit to raykao/dagger that referenced this pull request Dec 14, 2025
* CI: kill the monolith

Signed-off-by: Solomon Hykes <[email protected]>

* feat: merge php sdk  dev inside toolchain

Signed-off-by: Tom Chauveau <[email protected]>

* feat: add dotnet-sdk-dev toolchain

Signed-off-by: Tom Chauveau <[email protected]>

* feat: add dotnet SDK

Signed-off-by: Tom Chauveau <[email protected]>

* chore: remove old dotnet SDK

Signed-off-by: Tom Chauveau <[email protected]>

* chore: convert dotnet-sdk-dev to dang

Signed-off-by: Tom Chauveau <[email protected]>

* feat: add rust-sdk-dev toolchain

Signed-off-by: Tom Chauveau <[email protected]>

* adding java-sdk-dev toolchain

Signed-off-by: Yves Brissaud <[email protected]>

* WIP: appease go linter

Signed-off-by: Solomon Hykes <[email protected]>

* go mod tidy

Signed-off-by: Solomon Hykes <[email protected]>

* fix docs toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* modules/go: customize display of linting "sub-checks"

Signed-off-by: Solomon Hykes <[email protected]>

* fix dependency issue in monolith

Signed-off-by: Solomon Hykes <[email protected]>

* fix linting across toolchains

Signed-off-by: kpenfound <[email protected]>

* new toolchain: release

Signed-off-by: Solomon Hykes <[email protected]>

* Change Github Actions to use 'dagger checks' instead of 'dagger call'

Signed-off-by: Solomon Hykes <[email protected]>
Co-authored-by: Kyle Penfound <[email protected]>

* Harmonize version of dang SDK

NOTE: this version does not work with dagger v0.19.6 and below.

Signed-off-by: Solomon Hykes <[email protected]>

* Annotate missing checks

Signed-off-by: Solomon Hykes <[email protected]>

* remove dead dependencies

Signed-off-by: Solomon Hykes <[email protected]>

* wip elixir dev module

Signed-off-by: kpenfound <[email protected]>

* toolchain: typescript-sdk: uncomment docs lint check

Signed-off-by: Solomon Hykes <[email protected]>

* Fix typescript-sdk toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* CI: disable test-loading of removed modules

Signed-off-by: Solomon Hykes <[email protected]>

* improve Python SDK toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* Polish the Go SDK toolchain

Signed-off-by: Solomon Hykes <[email protected]>

* chore: fix php sdk dev

Signed-off-by: Tom Chauveau <[email protected]>

* chore: add missing SDK to all-sdks module list

Signed-off-by: Tom Chauveau <[email protected]>

* chore: fix the rust SDK dev

Signed-off-by: Tom Chauveau <[email protected]>

* elixir-dev: use File.withReplaced for publish and syncImage

Signed-off-by: kpenfound <[email protected]>

* chore: push fix attempt on php sdk (not working yet)

Signed-off-by: Tom Chauveau <[email protected]>

* chore: cleanup php sdk dev

Signed-off-by: Tom Chauveau <[email protected]>

* wip: attach dev engine to typescript SDK tests

Signed-off-by: Solomon Hykes <[email protected]>

* monolith: comment code

Signed-off-by: Solomon Hykes <[email protected]>

* typescript SDK toolchain: fix bun tests

Signed-off-by: Solomon Hykes <[email protected]>

* typescript-sdk toolchain: expose bunjs-dev-container

Signed-off-by: Solomon Hykes <[email protected]>

* gha: remove concurrency group in checks

Signed-off-by: Solomon Hykes <[email protected]>

* python-sdk toolchain: expose python dev container for debugging

Signed-off-by: Solomon Hykes <[email protected]>

* python-sdk toolchain: remove typecheck() from exposed checks

Signed-off-by: Solomon Hykes <[email protected]>

* go mod tidy

Signed-off-by: Solomon Hykes <[email protected]>

* ci: run checks on latest main commit

Signed-off-by: Solomon Hykes <[email protected]>

* fix toolchain description display

Signed-off-by: Yves Brissaud <[email protected]>
Co-authored-by: Solomon Hykes <[email protected]>

* fix: lint issues

Signed-off-by: Yves Brissaud <[email protected]>

* fix(modules/go): optional include/exclude for `modules` list

Signed-off-by: Yves Brissaud <[email protected]>

* fix: left over from previous lint commit

Signed-off-by: Yves Brissaud <[email protected]>

* go mod tidy without integration ignored

Signed-off-by: kpenfound <[email protected]>

* only ignore core/integration/testdata for go toolchain

Signed-off-by: kpenfound <[email protected]>

* ignore only testdata modules

Signed-off-by: kpenfound <[email protected]>

* helm-dev toolchain: vendor k3s dependency

Signed-off-by: Solomon Hykes <[email protected]>

* python-sdk-dev toolchain: vendor dockerd dependency

Signed-off-by: Solomon Hykes <[email protected]>

* fix testdata linting

Signed-off-by: kpenfound <[email protected]>

* the key to the universe

Signed-off-by: kpenfound <[email protected]>

* unvendor dang and fix linting

Signed-off-by: kpenfound <[email protected]>

* elixir-sdk: implement bump()

Signed-off-by: Solomon Hykes <[email protected]>

* typescript sdk: implement bump()

Signed-off-by: Solomon Hykes <[email protected]>

* dotnet SDK: add (no-op) bump function

Signed-off-by: Solomon Hykes <[email protected]>

* rust sdk: fix bump()

Signed-off-by: Solomon Hykes <[email protected]>

* ci: run linters in bootstrap, instead of SDK tests

This avoids triggering an unresolved issue with 3 layers of nested dagger engines (dagger-in-dagger-in-dagger).

Signed-off-by: Solomon Hykes <[email protected]>

* sdk/java: add fmt command

Signed-off-by: Yves Brissaud <[email protected]>

* convert toolchains/security to dang

Signed-off-by: Yves Brissaud <[email protected]>

* convert toolchains/changelog to dang

Signed-off-by: Yves Brissaud <[email protected]>

* python-sdk toolchain: remove over-complicated "dockerfile extractor", just hardcode uv image

Signed-off-by: Solomon Hykes <[email protected]>

* go-sdk: expand ignore filter to include new codegen dependencies

Signed-off-by: Solomon Hykes <[email protected]>

* Caught stealthy typo which caused ignore filters to be... ignored

Signed-off-by: Solomon Hykes <[email protected]>

* CI: .env.gha: env-specific configuration for our Github Actions CI environment

Signed-off-by: Solomon Hykes <[email protected]>

* engine-dev toolchain: relax source filter to un-break the build

Signed-off-by: Solomon Hykes <[email protected]>

* CI: update "outer engine" pinned commit

Signed-off-by: Solomon Hykes <[email protected]>

* CI: hardcode absolute path of docker config

Signed-off-by: Solomon Hykes <[email protected]>

* typescript sdk: remove duplicate code, fix weird dang error, simplify

Signed-off-by: Solomon Hykes <[email protected]>

* unbreak rebase

Signed-off-by: Solomon Hykes <[email protected]>

* go toolchain: fancier output + limited parallelism of 'check-tidy' function

Signed-off-by: Solomon Hykes <[email protected]>

* CI: bump outer engine version

Signed-off-by: Solomon Hykes <[email protected]>

* fix wrong ignore filter, which caused seemingly unrelated shell auto-complete error

Signed-off-by: Solomon Hykes <[email protected]>

* tentative fix for php-sdk:generate error caused by missing secret in engine-dev installclient

Signed-off-by: Solomon Hykes <[email protected]>

---------

Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Tom Chauveau <[email protected]>
Signed-off-by: Yves Brissaud <[email protected]>
Signed-off-by: kpenfound <[email protected]>
Signed-off-by: Yves Brissaud <[email protected]>
Co-authored-by: Tom Chauveau <[email protected]>
Co-authored-by: Yves Brissaud <[email protected]>
Co-authored-by: kpenfound <[email protected]>
Co-authored-by: Yves Brissaud <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants