feat: Add table_options support for aws_securityhub_findings table #11955

VictorCovalski · 2023-07-04T17:12:37Z

Summary

This PR adds support for table_options for the table aws_securityhub_findings. This allows users to specify filters before fetching findings from Security Hub.

Dougoc · 2023-07-04T17:24:58Z

👍

bbernays

Thank you for this PR, it looks really good! A few comments. And then if you could also add this to the docs here

plugins/source/aws/client/tableoptions/securityhub.go

plugins/source/aws/resources/services/securityhub/findings.go

hermanschaaf

Looks great, thanks for the contribution! One small thing to fix, otherwise looks good to me 👍

plugins/source/aws/client/tableoptions/securityhub.go

Co-authored-by: Herman Schaaf <[email protected]>

VictorCovalski · 2023-07-05T16:25:45Z

@hermanschaaf if MaxResults is not set in the configuration file then it will have the default value of 0 in the GetFindingsInput struct, which is not within valid range [1,100].

Would it be better to have this range go from [0,100]? Or perhaps we should let the AWS API validate this?

As you can see, MaxResults is not validated in Cloudtrail LookupEvents table options.

cloudquery/plugins/source/aws/client/tableoptions/cloudtrail.go

Lines 34 to 41 in 074aec2

    
           func (c *CloudtrailAPIs) validateLookupEvents() error { 
        
           	for _, opt := range c.LookupEventsOpts { 
        
           		if aws.ToString(opt.NextToken) != "" { 
        
           			return errors.New("invalid input: cannot set NextToken in LookupEvents") 
        
           		} 
        
           	} 
        
           	return nil 
        
           }

hermanschaaf

👍

hermanschaaf · 2023-07-05T19:17:03Z

I think setting a default of 100 in the validation function makes sense for now, we generally want it to be the maximum 👍 Later on we can probably split it out and create a SetDefaults function that gets called before Validate for all table options, but we don't have to do that in this PR.

🤖 I have created a release *beep* *boop* --- ## [20.0.0](plugins-source-aws-v19.2.0...plugins-source-aws-v20.0.0) (2023-07-12) ### ⚠ BREAKING CHANGES * **aws:** Define primary key for eips ([#11728](#11728)) * Upgrades the awspricing source plugin to use plugin-sdk v4. This version does not contain any user-facing breaking changes, but because it is now using CloudQuery gRPC protocol v3, it does require use of a destination plugin that also supports protocol v3. All recent destination plugin versions support this. ### This Release has the Following Changes to Tables - Table `aws_ec2_eips`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_ec2_eips`: primary key constraint added to column `allocation_id` (:warning: breaking) - Table `aws_ec2_eips`: primary key constraint added to column `region` (:warning: breaking) - Table `aws_ec2_eips`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `aws_networkmanager_global_networks` was added - Table `aws_networkmanager_links` was added - Table `aws_networkmanager_sites` was added - Table `aws_networkmanager_transit_gateway_registrations` was added ### Features * Add table_options support for aws_securityhub_findings table ([#11955](#11955)) ([c9eff12](c9eff12)) * **aws-policies:** Add in AWS security account contact query ([#11729](#11729)) ([c9d7294](c9d7294)) * **aws-policies:** Add sns logging of delivery status to AWS Policies ([#12074](#12074)) ([80f0b88](80f0b88)) * **aws-policies:** Update sqs encryption for aws foundational security policies ([#11777](#11777)) ([30d415c](30d415c)) * **aws-policies:** Update ssm queries for aws policies ([#12067](#12067)) ([2b9180f](2b9180f)) * **aws-services:** Support newly added regions ([#11922](#11922)) ([6680d7a](6680d7a)) * **aws-services:** Support newly added regions ([#12120](#12120)) ([15ea38c](15ea38c)) * **aws:** Add Support for `ecs:ListTasks` in `table_options` ([#11986](#11986)) ([3016c16](3016c16)), closes [#11981](#11981) * **aws:** Define primary key for eips ([#11728](#11728)) ([fa48d4a](fa48d4a)) * **aws:** Support networkmanager resources ([#12123](#12123)) ([a642ce0](a642ce0)) * Upgrades the awspricing source plugin to use plugin-sdk v4. This version does not contain any user-facing breaking changes, but because it is now using CloudQuery gRPC protocol v3, it does require use of a destination plugin that also supports protocol v3. All recent destination plugin versions support this. ([7d50d29](7d50d29)) ### Bug Fixes * **aws:** Skip fetching tags for `aws_kafka_cluster_operations` ([#11973](#11973)) ([2b62ba4](2b62ba4)) * **aws:** Validate table relations not just top level table ([#12121](#12121)) ([e13d931](e13d931)) * **deps:** Update github.com/apache/arrow/go/v13 digest to 5a06b2e ([#11857](#11857)) ([43c2f5f](43c2f5f)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 0a52533 ([#12091](#12091)) ([927cefa](927cefa)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to a2a76eb ([#12104](#12104)) ([311f474](311f474)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to df3b664 ([#11882](#11882)) ([9635b22](9635b22)) * **deps:** Update github.com/cockroachdb/cockroachdb-parser digest to c9c144e ([#11863](#11863)) ([1547efd](1547efd)) * **deps:** Update github.com/cockroachdb/logtags digest to 21c5414 ([#11864](#11864)) ([da48b1f](da48b1f)) * **deps:** Update github.com/gocarina/gocsv digest to 99d496c ([#11865](#11865)) ([c3de686](c3de686)) * **deps:** Update github.com/golang/geo digest to 6adc566 ([#11866](#11866)) ([edb7ed8](edb7ed8)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/networkfirewall to v1.28.3 ([#12079](#12079)) ([a27fa21](a27fa21)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/securityhub to v1.33.2 ([#12081](#12081)) ([e77f93e](e77f93e)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/servicediscovery to v1.21.7 ([#12082](#12082)) ([01f8b59](01f8b59)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.5.0 ([#11850](#11850)) ([3255857](3255857)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.6.0 ([#11916](#11916)) ([421e752](421e752)) * **postgresql:** Rerun release please ([#12002](#12002)) ([9d12843](9d12843)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

VictorCovalski added 3 commits July 4, 2023 13:57

first commit

d2059b5

small refactor

2808bc4

check return value for error

6618aac

VictorCovalski requested review from hermanschaaf and yevgenypats as code owners July 4, 2023 17:12

cq-bot added the aws label Jul 4, 2023

VictorCovalski changed the title ~~feat: Add support for table options for table aws_securityhub_findings~~ feat: Add support for table options for aws_securityhub_findings table Jul 4, 2023

bbernays reviewed Jul 4, 2023

View reviewed changes

plugins/source/aws/client/tableoptions/securityhub.go Show resolved Hide resolved

plugins/source/aws/resources/services/securityhub/findings.go Outdated Show resolved Hide resolved

VictorCovalski added 4 commits July 4, 2023 17:54

refactor and bug fix

8143078

fix error string in test

284471f

add unexported types

8f6bfa1

update docs

2a7e3a4

cq-bot added the website label Jul 4, 2023

vercel bot deployed to Preview July 4, 2023 21:26 View deployment

hermanschaaf requested changes Jul 5, 2023

View reviewed changes

plugins/source/aws/client/tableoptions/securityhub.go Outdated Show resolved Hide resolved

VictorCovalski and others added 2 commits July 5, 2023 11:21

Fix validation logic

0264c34

Co-authored-by: Herman Schaaf <[email protected]>

Merge branch 'main' into feat-sechub-table-options

e25dab8

vercel bot deployed to Preview July 5, 2023 14:29 View deployment

Merge branch 'main' into feat-sechub-table-options

cd91d20

vercel bot deployed to Preview July 5, 2023 17:06 View deployment

add default value for MaxResults

d495b23

VictorCovalski requested a review from hermanschaaf July 5, 2023 18:05

hermanschaaf approved these changes Jul 5, 2023

View reviewed changes

Merge branch 'main' into feat-sechub-table-options

1aa74e6

vercel bot deployed to Preview July 5, 2023 19:20 View deployment

VictorCovalski and others added 2 commits July 5, 2023 16:35

Merge branch 'main' into feat-sechub-table-options

2015272

modify TestGetFindings to work around fake object

cd95299

VictorCovalski changed the title ~~feat: Add support for table options for aws_securityhub_findings table~~ feat: Add table_options support for aws_securityhub_findings table Jul 5, 2023

bbernays added the automerge Automatically merge once required checks pass label Jul 5, 2023

kodiakhq bot merged commit c9eff12 into cloudquery:main Jul 5, 2023

cq-bot mentioned this pull request Jul 5, 2023

chore(main): Release plugins-source-aws v20.0.0 #11853

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add table_options support for aws_securityhub_findings table #11955

feat: Add table_options support for aws_securityhub_findings table #11955

Uh oh!

VictorCovalski commented Jul 4, 2023

Uh oh!

Dougoc commented Jul 4, 2023

Uh oh!

bbernays left a comment

Uh oh!

Uh oh!

Uh oh!

hermanschaaf left a comment

Uh oh!

Uh oh!

VictorCovalski commented Jul 5, 2023

Uh oh!

hermanschaaf left a comment

Uh oh!

hermanschaaf commented Jul 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

feat: Add table_options support for aws_securityhub_findings table #11955

feat: Add table_options support for aws_securityhub_findings table #11955

Uh oh!

Conversation

VictorCovalski commented Jul 4, 2023

Summary

Uh oh!

Dougoc commented Jul 4, 2023

Uh oh!

bbernays left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hermanschaaf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

VictorCovalski commented Jul 5, 2023

Uh oh!

hermanschaaf left a comment

Choose a reason for hiding this comment

Uh oh!

hermanschaaf commented Jul 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants