IE-498: added support to migrate pipelines #5
Conversation
|
✅ Gitleaks Findings: No secrets detected. Safe to proceed! |
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for migrating pipelines from a source vault to a target vault, enabling cross-environment pipeline transfers with datastore configuration overrides.
- Implements core pipeline migration logic with datastore validation and transformation
- Adds comprehensive test coverage for migration functionality
- Provides GitHub Actions workflow for automated pipeline migration
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| migrate_pipelines.py | Implements pipeline migration logic including API calls, datastore validation, and payload transformation |
| tests/test_migrate_pipelines.py | Provides comprehensive test suite covering validation, transformation, and error handling |
| README.md | Documents pipeline migration parameters, usage, and configuration examples |
| .github/workflows/migrate_pipelines.yml | Defines automated workflow for executing pipeline migrations with configurable parameters |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
✅ Gitleaks Findings: No secrets detected. Safe to proceed! |
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| type: choice | ||
| default: "Source: SANDBOX, Target: PRODUCTION" | ||
| options: | ||
| - "Source: SANDBOX, Target: PRODUCTION" |
There was a problem hiding this comment.
Just a suggestion, but instead of giving rigid options with source and target specified, would it not be better to provide individual options for them? This way, there wont be a need to maintain rigid options and source and target values could be selected with much more flexibility.
This would of course require different inputs for both source and target instead of a single input.
Something like this:
inputs:
source:
description: "Select source env"
type: choice
default: "SANDBOX"
options:
- "SANDBOX"
- "PRODUCTION"
target:
description: "Select target env"
type: choice
default: "PRODUCTION"
options:
- "SANDBOX"
- "PRODUCTION"
There was a problem hiding this comment.
GitHub workflows have a limit on no.of inputs it can take.
There was a problem hiding this comment.
I do agree that the suggested input options are more intuitive. If we are approaching a limit on the number of options github actions will accept, then I wonder if we are trying to include too many configurations into the inputs.
however, not going to block progress on this. just want to call it out.
|
✅ Gitleaks Findings: No secrets detected. Safe to proceed! |
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if 'ID' in transformed_resource: | ||
| del transformed_resource['ID'] # remove pipeline ID |
There was a problem hiding this comment.
[nitpick] Use double quotes for string literals to maintain consistency with the rest of the codebase. Change 'ID' to \"ID\" on both lines.
| if 'ID' in transformed_resource: | |
| del transformed_resource['ID'] # remove pipeline ID | |
| if "ID" in transformed_resource: | |
| del transformed_resource["ID"] # remove pipeline ID |
|
|
||
|
|
||
| def test_validate_ftp_server_success(monkeypatch): | ||
| module = load_module(monkeypatch) |
There was a problem hiding this comment.
I am not super familiar with testing standards in python, but my intuition is we should be handling a number of permutations of ftp fields that could be provided. But if this unit test provides full coverage of the validation function then we are good.
| headers=SOURCE_ACCOUNT_HEADERS, | ||
| ) | ||
| response.raise_for_status() | ||
| return response.json()["pipeline"] |
There was a problem hiding this comment.
can we put all of the magic strings of json field names into consts? this way if the request/response bodies ever change and we need to update this code, we only need to make the change in one place.
There was a problem hiding this comment.
in the long term, ideally we would create generated python code out of our protos so we don't need the magic strings. but that is certainly outside the scope of this.
| } | ||
|
|
||
|
|
||
| def test_replace_datastore_input_replaces_only_datastore(monkeypatch): |
There was a problem hiding this comment.
another question due to my lack of python expertise: is it best practice to have tests that all relate to test_replace_datastore_input as individual defs? is there any convention for organizing these related tests into a single func? I ask because it is somewhat difficult to track all of the cases being tested when they are not grouped in a meaningful way.
however if this is the idiomatic python way of writing unit tests I'll accept it :)
IE-498: added support to migrate pipelines