Hey @vito, I've been playing around with fly commands for instanced pipeline and I think I've found a way to simplify the fly UX.

Instead of adding --instance-var with a colon (:) for partial filtering to almost all fly commands, I could modify the flaghelpers.*Flag to have an optional coma separated list of key:value pairs as suffix.

e.g.:

fly get-pipeline -p pipeline-group/version:6.0.0,os:windows
fly builds -j pipeline-group/job-name/version:6.0.0,os:windows
fly check-resource -r pipeline-group/resource-name/version:6.0.0,os:windows

Then :

• The flaghelpers.PipelineFlag: could be like PIPELINE_GROUP/PIPELINE_INSTANCE_VARS
• The flaghelpers.JobFlag could be like PIPELINE_GROUP/JOB/PIPELINE_INSTANCE_VARS
• The flaghelpers.ResourceFlag could be like PIPELINE_GROUP/RESOURCE/PIPELINE_INSTANCE_VARS

This also simplifies group operations (like pausing all instances of a group). By just splitting the flag's value on "/", if there's an instance vars suffix we can easily distinguish between instanced and group operations.

What do you think?

@mouellet Good idea! Seems like a good use of the flag system, since in these cases the intent of the flag is to identify a single pipeline, and specifying the vars is necessary in order to do that.

One suggestion: the order of PIPELINE/JOB/PIPELINE_VARS feels unintuitive - could we do PIPELINE/PIPELINE_VARS/JOB instead?

Sorry for the lack of feedback on this @mouellet. It looks really good so far! Was playing around with setting some pipelines, and noticed a couple things:

1. You can create both a group of instanced pipelines and an individual pipeline with the same name:

$ fly -t dev sp -n -p prs -c pipeline.yml -i number=5896
... (succeeds)
$ fly -t dev sp -n -p prs -c pipeline.yml -i number=5897
... (succeeds)
$ fly -t dev sp -n -p prs -c pipeline.yml
... (succeeds)

I suspect we want to disallow the third fly sp (i.e. a team + pipeline name either refers to a single pipeline or to a group of pipeline instances). Is that true @vito?

2. Not sure what practical implications this has, but saving a pipeline without instance_vars doesn't result in p.instance_vars being NULL in Postgres - instead, it's the jsonb value of null. At the very least, I think this UNIQUE INDEX won't be used at all:

I suspect we want to disallow the third fly sp (i.e. a team + pipeline name either refers to a single pipeline or to a group of pipeline instances). Is that true @vito?

Yeah that's a good point, the RFC doesn't really make this clear. 🤔 I think we should disallow it; seems pretty sneaky.

I suspect we want to disallow the third fly sp (i.e. a team + pipeline name either refers to a single pipeline or to a group of pipeline instances). Is that true @vito?

I had the same question, but for now I see all pipelines as pipeline instances whether they have instance vars or not... The Pipeline Groups concept hasn't been introduced yet and I think it's somewhat related to that. BTW, we should have a discussion about that soon!

1. Not sure what practical implications this has, but saving a pipeline without instance_vars doesn't result in p.instance_vars being NULL in Postgres - instead, it's the jsonb value of null. At the very least, I think this UNIQUE INDEX won't be used at all:

Expect this to be working in the next commits (still WIP...)

I had the same question, but for now I see all pipelines as pipeline instances whether they have instance vars or not

Hmm, that's interesting! All of the UI designs that @matthewpereira has mocked up are under the assumption that all pipeline instances sharing a name would have the same set of instance variables, but I'm realizing now that isn't necessarily the case. Pipeline instances sharing a name may not have the same pipeline configuration, either. Granted, the use case of the across step + set_pipeline with instance vars will result in uniform pipeline instances - but this isn't the only way to create pipeline instances, clearly. I wonder if this changes anything from a design perspective

I wonder if this changes anything from a design perspective

Hey @aoldershaw , right now I'm focusing on replacing the pipelineName param for (almost) all method and api calls with a new atc.PipelineRef parameter object. This way we'll be able to query, visualize and take actions on individual pipeline instances (e.g. watch the build of a pipeline instance...).

It does allow for everything that you said, and it alone doesn't resolve the rfc... But I think it'll make it flexible enough for which ever design we pick fo the Pipeline Groups as well as pipeline group operations.

Maybe the fly set-pipeline command should remain as-is and a new fly command is needed for setting pipeline groups or templates..?

Hey @mouellet, sorry, I was mostly just thinking out loud as it's something I hadn't really thought of (and wasn't sure if our designer was aware of it or not).

I think the approach you're taking now makes sense!

One suggestion: the order of PIPELINE/JOB/PIPELINE_VARS feels unintuitive - could we do PIPELINE/PIPELINE_VARS/JOB instead?

Hey @vito, I implemented this and it works great! Definitely more intuitive.

There is one thing however, unmarshalling the flag is a little tricky...

Both pipeline name and job/resource name should be valid identifiers, but not the instance vars with its colon. And dealing with the cases where someone forgot to add the job/resource name at the end (e.g. pipeline-name/branch:master)...

So I added an error when the job/resource name is not a valid identifier even if we only emit warnings for now. Is that ok or should we wait for the valid identifiers to be enforced?

So I added an error when the job/resource name is not a valid identifier even if we only emit warnings for now. Is that ok or should we wait for the valid identifiers to be enforced?

Hmm, good question. I think we might have to leave it as a warning for now, since we don't want it to break for someone who has an existing job with invalid characters and was able to use the foo/bar notation up until this point. 🤔

@vito, should I also update the logging, tracing and metrics w/ the pipeline instance vars?

We could use the whole atc.PipelineRef with lager.Data (or anything that takes and interface{} or json) and it would look something like this:

{...,"data":{"pipeline":{"name":"p1","instance_vars":{"branch":"master"}},...}}

Maybe could use the dot notations when it's only a string, like with tracing.Attrs...

What do you think?

@mouellet Hmm I wonder if it would be better to just log IDs. Logging pipeline names implies we need to also log e.g. team name/ID, project name/ID (later), and other things to disambiguate since names are non-unique at various levels.

It adds an extra step if you're e.g. trying to search for logs of pipeline XYZ, but on the other hand it's precise, less verbose, and won't involve false positives or additional filtering. 🤔

What do you think?

Side note: maybe we should start a pattern of adding a method like LogData() lager.Data to objects so there's a single place to change this and to make sure everything's using consistent data field names? (e.g. pipeline_id instead of pipeline) Then we can do e.g. log.WithData(pipeline.LogData()) everywhere pipeline-related.

@vito, maybe this is out of scope for the current PR? It's already quite large and feature flagged...

I suggest we open an issue for that, replacing all names with IDs. In logging, at the API level, etc.

@mouellet 👍 yeah sounds wise, that would probably be a bigger change