-
Notifications
You must be signed in to change notification settings - Fork 431
Kueueviz: Helm values.yaml for ingress annotations, tls and environment variables #6682
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kueueviz: Helm values.yaml for ingress annotations, tls and environment variables #6682
Conversation
✅ Deploy Preview for kubernetes-sigs-kueue canceled.
|
|
Welcome @Smuger! |
Hi @Smuger. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
/ok-to-test |
0104c99
to
f6bc463
Compare
/retest |
@mbobrovskyi |
/area dashboard |
@mbobrovskyi |
annotations: | ||
nginx.ingress.kubernetes.io/rewrite-target: / | ||
nginx.ingress.kubernetes.io/ssl-redirect: "true" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand that Helm charts aren't needed, but can we create a patch to retain it in the manifests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively, revert the config changes and just update the annotations in the charts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mbobrovskyi
Sorry, I'm not sure if I understand. nginx annotations are still the default in values.yaml. Are you suggesting keeping the hardcoded nginx annotations in config and then overwriting them from values.yaml? I'm a bit worried about the added complexity
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you configured the default value for the Helm charts — that’s good. But what about the Kustomize manifests? Many users install Kueue using manifests instead of Helm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can check the difference between manifests using make artifacts
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mbobrovskyi
Thanks, I get it now. Kustomize patch added
charts/kueue/tests/kueue_test.yaml
Outdated
env: [] | ||
asserts: | ||
- notExists: | ||
path: spec.template.spec.containers[0].env[0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: please add empty line at the end of this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mbobrovskyi
Added empty line
# -- Environment variables for KueueViz backend deployment | ||
env: | ||
- name: KUEUEVIZ_ALLOWED_ORIGINS | ||
value: "frontend.kueueviz.local" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add the same for frontend?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mbobrovskyi
Added environment variables for frontend
/test all |
8d2dfdd
to
0f6f140
Compare
@Smuger: The following commands are available to trigger required jobs:
Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
config/kueueviz/kustomization.yaml
Outdated
|
||
patches: | ||
# Add ingress annotations for nginx backend ingress controller | ||
- path: backend_ingress_annotations_patch.yaml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about frontend ingress?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added patch for frontend ingress
- name: frontend | ||
env: | ||
- name: REACT_APP_WEBSOCKET_URL | ||
value: 'wss://backend.kueueviz.local' No newline at end of file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: new line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
new line added
config/kueueviz/kustomization.yaml
Outdated
# Add ingress annotations for nginx backend ingress controller | ||
- path: backend_ingress_annotations_patch.yaml | ||
# Add environment variables for frontend deployment | ||
- path: frontend_deployment_env_patch.yaml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about tls changes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added tls patch
/approve |
@mimowo: once the present PR merges, I will cherry-pick it on top of In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mimowo, Smuger The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
imagePullSecrets: [] | ||
# -- Enable PriorityClass for KueueViz dashboard backend deployments | ||
priorityClassName: | ||
# -- Environment variables for KueueViz backend deployment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# -- Environment variables for KueueViz backend deployment | |
# -- Environment variables for KueueViz frontend deployment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm guessing you mean the error here env frontend
Sorry for such a silly miss. Fixed
| enableVisibilityAPF | bool | `false` | Enable API Priority and Fairness configuration for the visibility API | | ||
| fullnameOverride | string | `""` | Override the resource name | | ||
| kubernetesClusterDomain | string | `"cluster.local"` | Kubernetes cluster's domain | | ||
| kueueViz.backend.env | list | `[{"name":"KUEUEVIZ_ALLOWED_ORIGINS","value":"frontend.kueueviz.local"}]` | Environment variables for KueueViz backend deployment | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| kueueViz.backend.env | list | `[{"name":"KUEUEVIZ_ALLOWED_ORIGINS","value":"frontend.kueueviz.local"}]` | Environment variables for KueueViz backend deployment | | |
| kueueViz.backend.env | list | `[{"name":"KUEUEVIZ_ALLOWED_ORIGINS","value":"frontend.kueueviz.local"}]` | Environment variables for KueueViz frontend deployment | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| kueueViz.backend.env | list | `[{"name":"KUEUEVIZ_ALLOWED_ORIGINS","value":"frontend.kueueviz.local"}]` | Environment variables for KueueViz backend deployment | | |
| kueueViz.backend.env | list | `[{"name":"KUEUEVIZ_ALLOWED_ORIGINS","value":"backend.kueueviz.local"}]` | Environment variables for KueueViz backend deployment | |
It look like still not correct. Could you please take a look?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might be completly wrong so do let me know if I'm misunderstanding this.
Isn't setting env variable of backend deployment KUEUEVIZ_ALLOWED_ORIGINS to "frontend.kueueviz.local" correct? I can set it back to '*' if you want
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes sense to me, wdyt @mbobrovskyi ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I see.
…ined in Helm values; Add environment variables via values.yaml in frontend and backend deployment; kustomize patch ingress annotations and frontend deployment environment variables
f1249b8
to
121624f
Compare
/lgtm |
LGTM label has been added. Git tree hash: 5f867f5dae8c80a852d506270dc22ee38bde9661
|
@mimowo: new pull request created: #6934 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
@mimowo: #6682 failed to apply on top of branch "release-0.12":
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
@Smuger we cherrypicked for 0.13 using the robot, but there are conflicts on 0.12. If you want to cherrypick there the PR needs to be prepared manually. Otherwise we can just skip CP to 0.12. |
| enableVisibilityAPF | bool | `false` | Enable API Priority and Fairness configuration for the visibility API | | ||
| fullnameOverride | string | `""` | Override the resource name | | ||
| kubernetesClusterDomain | string | `"cluster.local"` | Kubernetes cluster's domain | | ||
| kueueViz.backend.env | list | `[{"name":"KUEUEVIZ_ALLOWED_ORIGINS","value":"frontend.kueueviz.local"}]` | Environment variables for KueueViz backend deployment | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIUC, this .local
top level domain is a bit dangerous since .local
is reserved by mDNS usage and some ecosystem rely on that like zeroConf.
I hope that no one deploys this default value to their production.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would definitly like to avoid this. I was unsure how to handle KUEUEVIZ_ALLOWED_ORIGINS from the start.
Should I set KUEUEVIZ_ALLOWED_ORIGINS in values.yaml to "" and let middleware inform the user that they need to explicilty set it in production?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How was KUEUEVIZ_ALLOWED_ORIGINS set before this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tenzen-y can you clarify why this is dangerous? Do you mean (1.) someone could hack KueueViz if deployed this way, or that (2.) it wouldn't work for environments not using "local".
IIUC it may not work on some environements, and require manual change, but so would empty "" also not work. In that case it seems fair to leave it using "local" as a default. wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Assuming (2.) - no attack risk, just "doesn't work", I'm ok to either leave it empty or set as frontend.kueueviz.local
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1Q: In my opinion we should handle CORS like this
Development:
Should be set to: KUEUEVIZ_ALLOWED_ORIGINS = "*" by default making every origin valid
Production:
Should be set to KUEUEVIZ_ALLOWED_ORIGINS = "" by default making no origin valid. Expecting user to setup
their own domain
I'm ok with that in a follow up.
2Q I'm also for removing .local domain.
I'm ok for that in a follow up too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1A: Specifying a dedicated CORS domain should be better in the production env. If not, some security risks occur like CSRF and XSS.
Yes, so it is better not to use *. Actually * was used before this PR.
IIUC, the *
CORS origin is used only when development mode. So, I think that it is not bigger problem.
2A: .local domain is a reserved name. So, if we use this domain here, sometimes the intranet will break based on their network settings.
Yes, but if this is the case admins will be able to change. If this works in most setups maybe no need to change the default. It does not seem to have any risk - it may just not work.
I meant the affection potentially could reach across their entire internal networks. So, I mentioned that "I hope ... this default value to their production."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1Q: In my opinion we should handle CORS like this
Development:
Should be set to: KUEUEVIZ_ALLOWED_ORIGINS = "*" by default making every origin valid
Production:
Should be set to KUEUEVIZ_ALLOWED_ORIGINS = "" by default making no origin valid. Expecting user to setup their own domain
2Q I'm also for removing .local domain.
Should we just use localhost by default and tell people to use port-forwarding? Kubernetes-Dashboard does that
@Saumya40-codes Thanks for summarizing that. Both look good to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIUC, the * CORS origin is used only when development mode. So, I think that it is not bigger problem.
I think this "developer" more is actually what we were releasing in 0.13 by default TBH. So, I think this PR making it "frontend.kueueviz.local" is making this more strict anyway.
I meant the affection potentially could reach across their entire internal networks. So, I mentioned that "I hope ... this default value to their production."
Can you clarify what you mean by that?
Basically, I think KueueViz needs to work OOTB just by enableKueueViz, otherwise users will hit issues.
So IIUC "frontend.kueueviz.local" is very useful as default setting. Sure, we can educate users they recommend changing (I'm still don't understand why / how).
Similarly, we recommend users to use HA mode for Kueue, but at least it works OOTB, I think we need to make KueueViz also works OOTB.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we want to make KueueViz OOTB, *
could be used as a default parameter with development mode.
But, if we want to make KueueViz deployable to production, avoiding both *
and .local
would be better.
Both *
and .local
have the same risk "level", but the "directions" are different.
*
indicates security risk, .local
indicates network disruption risk.
We might consider the .localhost
domain, but we're not sure if it works correctly.
To be clear, my comment is "hope" for production. So, I don't try to enforce any approach. Some solutions are safe, but others are inconvenient.
/release-note-edit
|
What type of PR is this?
/kind feature
What this PR does / why we need it:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
I'm setting KUEUEVIZ_ALLOWED_ORIGINS to "frontend.kueueviz.local" by default. Let me know if there is a better option.

Does this PR introduce a user-facing change?