Kustomize and Helm diff support #820

willejs · 2021-01-29T12:13:02Z

willejs
Jan 29, 2021

Usually, one really valuable step in your CI against a PR is to do a kubectl/helm diff to show users what would change in your cluster (minus secrets).

A great example of this is probably how Terraform Enterprise/Spacelift/Atlantis do this.
They will run a terraform plan and post the diff of the plan back to your PR as a comment and link through to the plan phase for you to inspect. This is really helpful when reviewing PRs, catching errors early on, and communicating the impact of your PRs. In the CiOps world, it is common for people to use helmfile to do a similar thing too.

Currently, you could probably hack this into your CI, reading a helmRelease/kustomize resource and trying to construct diff commands. This would end up devaluing one of the main benefits of gitops, not having k8s credentials in CI tasks. Additionally, it would also mean you have to try and replicate the same logic that the helm (merging values) and kustomize (sops secrets), and even source (repackaging helm values files) controllers carry out.

I am wondering if there is a way to implement this in flux2. Getting the source controller to create an artifact for a PR, letting the controllers do the diff, send a notification to a github pr, and store the diff output in a resource, cleaned up on PR close? Just an idea.

Really, I am wondering if this is on anyone else's radar, and If this could become a feature request?

stefanprodan · 2021-01-29T12:18:03Z

stefanprodan
Jan 29, 2021
Maintainer

Posting the diff of decrypted data(Kubernetes secrets) to GitHub or any other external system it’s a no-go IMO.

15 replies

shibumi Mar 12, 2021

Hi, sorry for reviving this thread. I think the mistake is already to have plaintext secrets in any YAML configuration. As far as I know people use Hashicorp Vault or Bitnami's Sealed-Secrets Operator for running around this issue. So flux should always show the diff and any plaintext secrets should not be visible, because SealedSecrets or Vault should handle the rest. Everything else is a cluster misconfiguration IMHO.

william-salt-cko Mar 16, 2021

@stefanprodan Have you had any further thoughts or discussions about this?

I am currently doing a flux1 -> flux2 migration and lack of diff support has created a quite complex shell script that diffs just helm releases, which doesn't work quite in the same way as the controller.
We have also on one occasion changed some kustomize and affected a dozen or so releases in production which isn't obvious just from a git diff, whereas a kubectl diff would indicate actually what would happen.

We have other ideas for workarounds, but we would really like some functionality akin to TF, ArgoCD, or Helm in Flux2, or at least indication there is a plan for it. 🙂

RichiCoder1 Apr 15, 2021

This is something I love about both terraform and ArgoCD, and it's a bit painful having to maintain a massive diff script just to accomplish this for PRs

william-salt-cko May 18, 2021

@stefanprodan Do you think you could revisit this now? The PR for kubectl diff hiding secrets support has been merged in from the original post. Helm diff already supports this too...

morremeyer Jun 3, 2021

I think we are now in a state where we can take a look at how to do this. The PR is merged as @william-salt-cko mentioned, so this will be part of Kubernetes 1.22.

I started #820 (comment) to gather ideas.

messiahUA · 2021-05-12T18:37:15Z

messiahUA
May 12, 2021

I think that diff functionality is a must for production use. For example, I've made a mistake in configuration files and flux completely deleted everything including itself with prune enabled. Of course I can disable pruning, but in this case I would need to do it for everything and it defeats the purpose. These mistakes can be prevented only by checking everything before applying...

0 replies

morremeyer · 2021-06-03T14:10:26Z

morremeyer
Jun 3, 2021

kubectl will have support for hiding secrets in 1.22, see this PR.
I think we could start discussing how this can be implemented to support different git flows, so here is something that I would find useful to have:

In most git flows, there is a production branch (most often named production or main) that flux be configured to apply resources from. If functionality was added that allows the configuration of a staging branch to diff against (e.g. staging or pre-releaes), this would allow to:

Use gitflow and gitlabflow natively
Not add any „blocking“ logic to the deployment from the production branch as the diff and the apply workflows would/could be completely seperate from one another

I have not yet found an idea about how to report the output that I am really happy with. My only idea so far would be to add a URL field to specify a webhook to which the diff could then be posted.

1 reply

morremeyer Jun 3, 2021

Another idea I had is for flux to track all branches of the repository and diff changes - this would also enable the usage of githubflow, but probably make the implementation more complex.

However, it would probably also enable easier tracking of diffs by being able to track changes back to a PR (or MR for e.g. GitLab).

stefanprodan · 2022-02-04T08:36:34Z

stefanprodan
Feb 4, 2022
Maintainer

In Flux 0.26 we've introduced the flux diff kustomization --path my/local/overlay command which uses the same server-side apply engine as kustomize-controller. The flux diff command detects SOPS encrypted secrets and masks the secret values in the diff output.

9 replies

mckennajones Jan 5, 2023

Hey there @stefanprodan is the above linked issue in helm-controller still a prereq for implementing a flux diff hr command?

ghobadimhd Oct 13, 2023

Hi, is there any further update about flux diff HelmRelease? Is this feature going to be added to flux or not? I've seen that the above issue has been completed
@stefanprodan

onedr0p Dec 26, 2023

The helm controller just got a pretty big update so hopefully we're not too far away from this functionality.

TaylorChristie May 1, 2024

Wondering if there is still any interest on the flux diff helmreleases command? Would be quite nice to have native support as currently I have a hacky bash script that doesn't quite get me all the way there. Having output similar to helm diff with the https://github.com/databus23/helm-diff plugin would be ideal

mo-rieger Jun 23, 2025

Are there any blockers left that prevent implementing flux diff helmrelease command? I am also looking for a better alternative to our bash script.

julienFlowdesk · 2022-02-18T13:59:55Z

julienFlowdesk
Feb 18, 2022

The flux diff kustomization --path X is an amazing command. I just wanted to add some early feedbacks:

In my case, I am trying to do flux diffing within a github-action with a specific service account having only read-only access to all kubernetes objects within my cluster to limit risks.

I was expecting flux to be able to do diff by reading only the current cluster objects but in fact it looks like it need more than that:

.  Kustomization diffing...: running dry-run
 errors occurred:
	* Namespace/x dry-run failed, reason: Forbidden, error: namespaces "x" is forbidden: User "github-action@xxxx" cannot patch resource "namespaces" in API group "" in the namespace "x": requires one of ["container.namespaces.update"] permission(s).

Maybe it is the awaited behaviour but I feel like the diff command shouldn't require more than the view roles within kubernetes to output something.

2 replies

stefanprodan Feb 18, 2022
Maintainer

Please raise this with the Kubernetes team, Flux performs a server-side apply dry-run, why dry-run requires write access I have no clue.

julienFlowdesk Feb 18, 2022

For posterity purposes: the kubernetes related issues :

kubernetes/kubernetes#95449
kubernetes/kubectl#981

Preisschild · 2025-09-03T13:50:12Z

Preisschild
Sep 3, 2025

Is there any way to show objects that would be created in a new Kustomization?

Currently, it just shows ► Kustomization/flux-system/<name> created

1 reply

stefanprodan Sep 3, 2025
Maintainer

Running flux build ks will show all objects, and you can use yq to selector the one from the diff.

Kustomize and Helm diff support #820

Uh oh!

Uh oh!

Replies: 6 comments · 28 replies

Uh oh!

stefanprodan Jan 29, 2021 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanprodan Feb 4, 2022 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanprodan Feb 18, 2022 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanprodan Sep 3, 2025 Maintainer

Replies: 6 comments 28 replies

stefanprodan
Jan 29, 2021
Maintainer

stefanprodan
Feb 4, 2022
Maintainer

stefanprodan Feb 18, 2022
Maintainer

stefanprodan Sep 3, 2025
Maintainer