Feedback thread for set_pipeline: self
#5732
Replies: 19 comments 10 replies
-
|
I find this functionality very useful. It was pretty straightforward to implement thanks to this step. |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, I was using Thanks for your help, really liked this feature. |
Beta Was this translation helpful? Give feedback.
-
|
I have been using While this is very understandable - updating the contents of a running pipeline would probably do more harm than good - I would like to inquire whether there is a possibility of "resetting" such a build, or cancelling it and triggering a re-run with the same inputs. |
Beta Was this translation helpful? Give feedback.
-
Our pipelines are widely using In our practice, we use Triggered a re-run with the same inputs after |
Beta Was this translation helpful? Give feedback.
-
|
I use this on the couple of pipelines that I manage and I love that it takes away some of the uncertainty around maintaining them. There's no longer any possibility of checking in a definition change to a repo where the pipeline accidentally wasn't updated, likewise pipeline changes don't get made without the definitions being checked in (nor without explanation via commit messages). It's now just: if you want to change a pipeline, change the definition and check it in. One source of truth, etc. And everyone gets visibility of it in the repo |
Beta Was this translation helpful? Give feedback.
-
|
I very much like this feature and find it useful, so I would like to keep it around for sure. ❤️ |
Beta Was this translation helpful? Give feedback.
-
|
This has been a game-changer for reducing the maintainance burden when it comes to updating pipelines. I've also been able to implement some orchestration pipelines to configure downstream pipelines which has been good. Thank you very much! ❤️ Something that I use when using Would it be possible to extend the |
Beta Was this translation helpful? Give feedback.
-
|
Plus 1. I've been working on the oneflow release process and |
Beta Was this translation helpful? Give feedback.
-
|
Just wanted to say that we use this on all my group's pipelines at our organization and I agree with others that this is a game-changer. Please do not remove this feature because it will send us back to the stone ages where conflicting pipeline overwrites were commonplace. |
Beta Was this translation helpful? Give feedback.
-
|
@vito or others, could we possibly get an idea of how "concrete" this experimental feature is? It's still widely warned in the UI. I'm not looking for exacts, but more of a "we don't expect it to be removed, but maybe changed" or "we have plans to deprecate due to X potential feature", etc. I also understand that things can change and that there are no promises, but I'd like to find out roughly where things are heading.. Reason I ask is, although this is experimental, the benefit of using |
Beta Was this translation helpful? Give feedback.
-
|
We're probably gonna keep it. There doesn't seem to be any foot-gun'ing going on so I personally don't see any reason to remove it. |
Beta Was this translation helpful? Give feedback.
-
|
I really like this feature, it helps keeping the manual work at a minimum. One thig that I stepped into which is not documented (but clear for most users i think) is the fact, that the set_pipeline: self step should be in its own job at the begining of the pipeline. Or more clear, that only jobs AFTER the set_pipeline: self step are updated. Not much of a problem but I walked right into it by only updating the pipeline in the beginning of a job and then wondering why the job itself did not update. |
Beta Was this translation helpful? Give feedback.
-
|
As others have said, this feature is truly a game changer. We have a git resource that watches the pipeline configuration file and pipeline variable files for commits. Upon a new commit to any of them, the pipeline immediately updates itself. This also allows us to have very dynamic, rapidly changing pipelines when paired with Jinja2 templating for the pipeline configuration file. Upon new commits to the pipeline's files, the pipeline configuration file is templated using Jinja2 in a task, and then it is passed to the |
Beta Was this translation helpful? Give feedback.
-
|
Would like to find out if it is possible to use this feature and optionally mark a set_pipeline task so that it will not be culled when it is not updated. Thus we have a single scanner for each team to scan all repos in our organisation for a 'desire' to create a pipeline. once created each repo will then maintain their own pipelines using set_pipeline (in this part fo the use-case the set_pipeline and culling works well). |
Beta Was this translation helpful? Give feedback.
-
|
ie my request is an option flag to disable the auto delete of the pipeline that has been generated with set_pipeline |
Beta Was this translation helpful? Give feedback.
-
|
We've been experimenting with the parent pipeline --> child pipeline pattern described by @taylorsilva in this blog post and have been loving it. It was particularly useful while we experimented with how we were hosting Concourse since all we had to do was set a single pipeline and, like a phoenix from the ashes, our entire configuration would get re-created in a new Concourse cluster. However, if both parent & child are triggered by the same resource (e.g., some git repo) then the child can kick off a set of builds and might get updated halfway through. What I'd want is for the child pipeline to first get updated, THEN for it to run. We've managed to address this by adding a |
Beta Was this translation helpful? Give feedback.
-
|
We use this pattern too and it works to avoid the child pipeline running with out-of-date configuration. Though there seems to be some check in place to avoid two Not sure how we work around that error. Would using an |
Beta Was this translation helpful? Give feedback.
-
|
Using this in every pipeline. Please don't remove. |
Beta Was this translation helpful? Give feedback.
-
|
Use the feature extensively, however I feel like there needs to be something similar to the |
Beta Was this translation helpful? Give feedback.
-
|
Agreed, a dry-run flag would be extremely useful for validating changes. |
Beta Was this translation helpful? Give feedback.
-
|
Loving this feature. In my use case, I have 50+ pipelines that I have to manage in an "Phoenix" style environment where Concourse needs bootstrapping of the current pipelines and takes over the commissioning process of an environment. Also using this functionality to treat the pipelines as attached to specific feature commits. Incredibly useful feature. |
Beta Was this translation helpful? Give feedback.
-
@ericgravelle If you can a little more regarding this particular use case that would be great, I tried but couldn't figure it out. |
Beta Was this translation helpful? Give feedback.
-
|
Here's a resource that's useful for this use-case (creating a pipeline per PR): https://github.com/aoldershaw/github-pr-resource#example We've been using the pattern shown in the example there to run unit tests on PRs and it's worked mint for us so far! |
Beta Was this translation helpful? Give feedback.
-
|
It should get a parameter to tell whether the pipeline should be created/updated paused, and there should be another step that should allow pausing/unpausing. Reason: sometimes you don't want to just create a pipeline from a template, you want to create a pipeline and set up things around the pipeline so the new pipeline can run. If you create pipelines unpaused, they start running right away, before you can create what else they need to run properly. If you do the setup first, on update, the previous versions of the pipelines start running before you are able to create the new versions. |
Beta Was this translation helpful? Give feedback.
-
|
Please do not remove this feature it is great! |
Beta Was this translation helpful? Give feedback.
-
|
Essential for me! Enabled me to be not dependent on GitHub/Gitea/GitLab/ForgeJo Actions anymore! |
Beta Was this translation helpful? Give feedback.
-
|
How do people use this? Do you have to manually re-run the task? What if you don't grant users more than viewer privileges (to enforce configuration as code)? I'm not sure how to make it so when I run it against an SCM revision it deterministically brings it into the requested state. If I make any changes to the pipeline they do not actually occur until the second revision after that change, even if I put it in a separate job and mark them both serial. I'm starting to think its best not to update pipelines in pipelines at all because of how poor the execution model supports it, I'm thinking maybe have a pipeline trigger a remote bazel job that updates the pipelines, that way no pipeline is self updating except for the bootstrap pipeline itself which does not execute anything after updating itself. Maybe another workaround is to make another resource that just acts like a cron job to subvert concourse's usual execution model so that it's eventually consistent. |
Beta Was this translation helpful? Give feedback.
-
|
Figured it out by reading: https://blog.concourse-ci.org/2020/08/24/gitops-for-pipelines.html It seems the one way to actually re-run a pipeline on the new version is by using the "passed" semantic. Can we get some discussion of this on the docs for set_pipeline and/or the docs for get? It doesn't seem obvious. |
Beta Was this translation helpful? Give feedback.
-
|
This thread is no longer monitored as |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
This thread is to gather feedback for the use of
set_pipeline: self.(I'll circle back here and add some example questions/prompts, this is just a placeholder for now so we can link to it from the PR that adds it.)
Beta Was this translation helpful? Give feedback.
All reactions