Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@kd7lxl
Copy link
Contributor

@kd7lxl kd7lxl commented Jun 23, 2023

By defining a primary key, the deterministic cq id feature can be used so that multiple syncs are able to associate the same resource.

@github-actions
Copy link

github-actions bot commented Jun 23, 2023

This PR has the following changes to source plugin(s) tables:

  • ⚠️ BREAKING CHANGE: Table aws_ec2_eips: primary key constraint added to column account_id
  • ⚠️ BREAKING CHANGE: Table aws_ec2_eips: primary key constraint added to column allocation_id
  • ⚠️ BREAKING CHANGE: Table aws_ec2_eips: primary key constraint added to column region
  • ⚠️ BREAKING CHANGE: Table aws_ec2_eips: primary key constraint removed from column _cq_id

Copy link
Member

@erezrokah erezrokah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @kd7lxl 🙏 I think you should be able to use the transformers.WithPrimaryKeys option instead of redefining the column. See example

Transform: transformers.TransformWithStruct(&types.Image{}, transformers.WithPrimaryKeys("Arn")),

What do you think?

@bbernays
Copy link
Collaborator

@kd7lxl - I agree that this is the right fix, if you need the cq_id to be consistent across multiple syncs you can use the deterministic_cq_id parameter in the source spec... More information about that can be found here

@kd7lxl
Copy link
Contributor Author

kd7lxl commented Jun 26, 2023

Thanks for the PR @kd7lxl 🙏 I think you should be able to use the transformers.WithPrimaryKeys option instead of redefining the column. See example

Transform: transformers.TransformWithStruct(&types.Image{}, transformers.WithPrimaryKeys("Arn")),

What do you think?

Looks like that worked great. Thanks for the hint.

@kd7lxl
Copy link
Contributor Author

kd7lxl commented Jun 26, 2023

@kd7lxl - I agree that this is the right fix, if you need the cq_id to be consistent across multiple syncs you can use the deterministic_cq_id parameter in the source spec... More information about that can be found here

Indeed, that's exactly how I arrived at this solution. I tried using deterministic cq id but it had no effect on this table. Then I realized it was because this table was missing a primary key.

@hermanschaaf hermanschaaf changed the title feat(aws): Define primary key for eips feat(aws)!: Define primary key for eips Jun 27, 2023
@bbernays bbernays added the automerge Automatically merge once required checks pass label Jul 12, 2023
@kodiakhq kodiakhq bot merged commit fa48d4a into cloudquery:main Jul 12, 2023
kodiakhq bot pushed a commit that referenced this pull request Jul 12, 2023
🤖 I have created a release *beep* *boop*
---


## [20.0.0](plugins-source-aws-v19.2.0...plugins-source-aws-v20.0.0) (2023-07-12)


### ⚠ BREAKING CHANGES

* **aws:** Define primary key for eips ([#11728](#11728))
* Upgrades the awspricing source plugin to use plugin-sdk v4. This version does not contain any user-facing breaking changes, but because it is now using CloudQuery gRPC protocol v3, it does require use of a destination plugin that also supports protocol v3. All recent destination plugin versions support this.

### This Release has the Following Changes to Tables
- Table `aws_ec2_eips`: primary key constraint added to column `account_id` (:warning: breaking)
- Table `aws_ec2_eips`: primary key constraint added to column `allocation_id` (:warning: breaking)
- Table `aws_ec2_eips`: primary key constraint added to column `region` (:warning: breaking)
- Table `aws_ec2_eips`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `aws_networkmanager_global_networks` was added
- Table `aws_networkmanager_links` was added
- Table `aws_networkmanager_sites` was added
- Table `aws_networkmanager_transit_gateway_registrations` was added

### Features

* Add table_options support for aws_securityhub_findings table ([#11955](#11955)) ([c9eff12](c9eff12))
* **aws-policies:** Add in AWS security account contact query ([#11729](#11729)) ([c9d7294](c9d7294))
* **aws-policies:** Add sns logging of delivery status to AWS Policies ([#12074](#12074)) ([80f0b88](80f0b88))
* **aws-policies:** Update sqs encryption for aws foundational security policies ([#11777](#11777)) ([30d415c](30d415c))
* **aws-policies:** Update ssm queries for aws policies ([#12067](#12067)) ([2b9180f](2b9180f))
* **aws-services:** Support newly added regions ([#11922](#11922)) ([6680d7a](6680d7a))
* **aws-services:** Support newly added regions ([#12120](#12120)) ([15ea38c](15ea38c))
* **aws:** Add Support for `ecs:ListTasks` in `table_options` ([#11986](#11986)) ([3016c16](3016c16)), closes [#11981](#11981)
* **aws:** Define primary key for eips ([#11728](#11728)) ([fa48d4a](fa48d4a))
* **aws:** Support networkmanager resources ([#12123](#12123)) ([a642ce0](a642ce0))
* Upgrades the awspricing source plugin to use plugin-sdk v4. This version does not contain any user-facing breaking changes, but because it is now using CloudQuery gRPC protocol v3, it does require use of a destination plugin that also supports protocol v3. All recent destination plugin versions support this. ([7d50d29](7d50d29))


### Bug Fixes

* **aws:** Skip fetching tags for `aws_kafka_cluster_operations` ([#11973](#11973)) ([2b62ba4](2b62ba4))
* **aws:** Validate table relations not just top level table ([#12121](#12121)) ([e13d931](e13d931))
* **deps:** Update github.com/apache/arrow/go/v13 digest to 5a06b2e ([#11857](#11857)) ([43c2f5f](43c2f5f))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 0a52533 ([#12091](#12091)) ([927cefa](927cefa))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to a2a76eb ([#12104](#12104)) ([311f474](311f474))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to df3b664 ([#11882](#11882)) ([9635b22](9635b22))
* **deps:** Update github.com/cockroachdb/cockroachdb-parser digest to c9c144e ([#11863](#11863)) ([1547efd](1547efd))
* **deps:** Update github.com/cockroachdb/logtags digest to 21c5414 ([#11864](#11864)) ([da48b1f](da48b1f))
* **deps:** Update github.com/gocarina/gocsv digest to 99d496c ([#11865](#11865)) ([c3de686](c3de686))
* **deps:** Update github.com/golang/geo digest to 6adc566 ([#11866](#11866)) ([edb7ed8](edb7ed8))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/networkfirewall to v1.28.3 ([#12079](#12079)) ([a27fa21](a27fa21))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/securityhub to v1.33.2 ([#12081](#12081)) ([e77f93e](e77f93e))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/servicediscovery to v1.21.7 ([#12082](#12082)) ([01f8b59](01f8b59))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.5.0 ([#11850](#11850)) ([3255857](3255857))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.6.0 ([#11916](#11916)) ([421e752](421e752))
* **postgresql:** Rerun release please ([#12002](#12002)) ([9d12843](9d12843))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants