[google_scc] Add support for Security Command Center API v2 and update finding data stream to support Cloud Detection and Response (CDR) workflow #14629

brijesh-elastic · 2025-07-22T06:39:24Z

Proposed commit message

google_scc: Add support for Security Command Center API v2 and update finding data stream to support
Cloud Detection and Response (CDR) workflow

Some fields inside "google_scc.finding.resource" are moved into "google_scc.finding.resource.gcp_metadata"
as per v2 API schema. This change allows to choose between v1 and v2 (recommended) APIs to fetch
findings. Location-based findings can be fetched to support Security Command Center data
residency feature. Field "google_scc.finding.vulnerability.cve.cvssv3.base_score" data
type is updated to "double" to better suit data. In dashboards, ECS fields are used
in preference to custom fields.

Added "do_not_log_failure: true" to the asset and source data streams to prevent logging of expected
pagination end failures, which have been causing system test failures from version 8.19 onwards.
This issue arose due to the addition of Fleet health status updates in the httpjson input.

Add ECS mappings and latest transform to finding data stream to help with the
Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflow.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html
[2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management

Note

To Reviewers:

Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/google_scc directory.
Run the following command to run tests.

elastic-package test

Related issues

elasticmachine · 2025-07-22T06:39:28Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-07-22T07:35:20Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

…DR) workflow

packages/google_scc/changelog.yml

packages/google_scc/data_stream/finding/fields/fields.yml

packages/google_scc/data_stream/asset/agent/stream/httpjson.yml.hbs

packages/google_scc/data_stream/finding/manifest.yml

packages/google_scc/data_stream/finding/_dev/test/pipeline/test-finding.log-expected.json

packages/google_scc/elasticsearch/transform/latest_cdr_misconfigurations/fields/base-fields.yml

packages/google_scc/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml

kcreddy · 2025-07-30T12:55:26Z

packages/google_scc/img/google-scc-finding-screenshot.png

May I know what are the changes made to the dashboards?

The main change is that dashboards are exported to version 8.19, along with below minor field changes :

Finding Dashboard

Resource Name Filter (google_scc.finding.resource.name -> resource.id)

Total Finding Metric (change decimal from 2 to 0)

Overview Dashboard

Distribution of Vulnerabilities by Severity (google_scc.finding.severity -> vulnerability.severity)

Top 5 Projects by Severity for Vulnerabilities (google_scc.finding.severity -> vulnerability.severity)

Top 5 Projects by Severity for Vulnerabilities (google_scc.finding.severity -> vulnerability.severity, google_scc.finding.resource.project.name -> cloud.project.id)

Top 5 Projects by Severity for Threats (google_scc.finding.resource.project.name -> cloud.project.id)

Finding Essentials Details

organization.id - google_scc.finding.name + resource.id google_scc.finding.category google_scc.finding.class google_scc.finding.severity - google_scc.finding.resource.project.name - event.created - google_scc.finding.security_marks.name - google_scc.finding.parent + cloud.project.id + google_scc.finding.security_marks.name

It seems like mainly ECS fields are used inside dashboards over custom fields. Could you add that to commit message?

packages/google_scc/_dev/build/docs/README.md

…leet health status updates in the httpjson input. Adding do_not_log_failure: true helps prevent the logging of expected failures, which were previously resulting in test failures

brijesh-elastic · 2025-07-31T06:57:10Z

/test

maxcold · 2025-07-31T10:55:29Z

packages/google_scc/elasticsearch/transform/latest_cdr_misconfigurations/fields/resource.yml

+    - name: id
+      type: keyword
+      description: The ID of the resource.
+    - name: name


@kcreddy @brijesh-elastic for example this I believe is covered by ecs@mappings even though resource is not a part of ECS itself. In the component template dynamic templates are being used which match by *.name for example

The problem is with pipeline tests. Any non-ECS field inside the *expected.json file must be explicitly defined. Otherwise they fail.

got it, I wonder if it's smth we want to improve going forward, or is it an expected safety net? It just doesn't match well the dynamic approach taken by ecs@mappings

Safety and also to do with preferring explicit mappings over dynamic mappings for Elasticsearch performanc.e

maxcold · 2025-07-31T10:56:57Z

packages/google_scc/elasticsearch/transform/latest_cdr_misconfigurations/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - event.id


I wonder if we need event.id here. Isn't a combination of rule.uuid and resource.id unique?

Yes, it will be required.
rule.uuid is not present in most of the cases. So, we must have to add event.id to be unique.

maxcold · 2025-07-31T10:59:56Z

packages/google_scc/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - event.id


same here, the combination of resource.id, package.name and package.version should be unique enough. This way we make sure that we don't have duplicated findings, eg. if event.id is generated for each change in the finding (eg. change of the status) we might have N finding documents for each (resource.id, package.name,package.version)

Yes, in here too, it will be required.
There are cases that combination of resource.id, package.name, package.version isn't unique.

maxcold

Looks good, great work! Tested on the provided env, CDR flows work as expected.
Before approving I want to clarify the usage of event.id for the transform uniqueness

… the unique field in visualizations

kcreddy

nits

packages/google_scc/_dev/build/docs/README.md

kcreddy

@brijesh-elastic, since the CI is failing on a known issue elastic/beats#45664, can you skip failing system tests for source and asset data streams where its failing? We can add them back after fix is made by updating kibana version.

Example:

integrations/packages/aws/data_stream/securityhub_findings/_dev/test/system/test-default-config.yml

Lines 2 to 5 in 9bd4e70

    
           skip: 
        
             reason: "Support backward compatibility of Current AWS package." 
        
             link: https://github.com/elastic/integrations/issues/3695 
        
           service: securityhub

…#45664

elasticmachine · 2025-08-06T06:49:05Z

💚 Build Succeeded

Buildkite Build
Commit: f1334d6

History

💔 Build #29509 failed eb06c1d
💔 Build #29409 failed 550781c
💚 Build #29369 succeeded 58721ab
💔 Build #29355 failed ccce426
💔 Build #29317 failed ccce426

cc @brijesh-elastic

elastic-sonarqube · 2025-08-06T06:49:12Z

Quality Gate failed

Failed conditions
53.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

kcreddy

LGTM

elastic-vault-github-plugin-prod · 2025-08-08T12:39:13Z

Package google_scc - 2.0.0 containing this change is available at https://epr.elastic.co/package/google_scc/2.0.0/

Add support for Findings v2 API

f76c8f7

brijesh-elastic self-assigned this Jul 22, 2025

brijesh-elastic requested a review from a team as a code owner July 22, 2025 06:39

brijesh-elastic marked this pull request as draft July 22, 2025 06:41

brijesh-elastic added 2 commits July 22, 2025 12:12

Update changelog entry

5d65a15

Add support for Security Command Center API v2 in source datastream

00031e4

brijesh-elastic changed the title ~~[google_scc] Add support for Findings v2 API~~ [google_scc] Add support for Security Command Center API v2 Jul 22, 2025

Update finding data stream to support Cloud Detection and Response (C…

db931d4

…DR) workflow

brijesh-elastic marked this pull request as ready for review July 30, 2025 06:21

brijesh-elastic changed the title ~~[google_scc] Add support for Security Command Center API v2~~ [google_scc] Add support for Security Command Center API v2 and update finding data stream to support Cloud Detection and Response (CDR) workflow Jul 30, 2025

brijesh-elastic added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Jul 30, 2025

resolve build failure

64b2680

kcreddy reviewed Jul 30, 2025

View reviewed changes

System tests are failing (from 8.19 onwards) due to the addition of F…

ccce426

…leet health status updates in the httpjson input. Adding do_not_log_failure: true helps prevent the logging of expected failures, which were previously resulting in test failures

brijesh-elastic force-pushed the google_scc-2.0.0 branch from 9d761be to ccce426 Compare July 30, 2025 13:58

address pr comments

58721ab

brijesh-elastic requested a review from kcreddy July 31, 2025 10:28

maxcold reviewed Jul 31, 2025

View reviewed changes

Increase initial_interval to 2160 hours (90 days) and use event.id as…

550781c

… the unique field in visualizations

kcreddy reviewed Aug 1, 2025

View reviewed changes

packages/google_scc/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/google_scc/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/google_scc/_dev/build/docs/README.md Outdated Show resolved Hide resolved

Resolve comments on Readme doc

eb06c1d

brijesh-elastic requested a review from kcreddy August 4, 2025 09:12

maxcold self-requested a review August 4, 2025 16:26

maxcold approved these changes Aug 4, 2025

View reviewed changes

kcreddy reviewed Aug 6, 2025

View reviewed changes

skip system test for asset and source datastream due to elastic/beats…

f1334d6

…#45664

kcreddy approved these changes Aug 6, 2025

View reviewed changes

kcreddy mentioned this pull request Aug 8, 2025

sonarqube: Update exclusions paths #14880

Closed

brijesh-elastic merged commit 9110393 into elastic:main Aug 8, 2025
8 of 9 checks passed

	skip:
	reason: "Support backward compatibility of Current AWS package."
	link: https://github.com/elastic/integrations/issues/3695
	service: securityhub

[google_scc] Add support for Security Command Center API v2 and update finding data stream to support Cloud Detection and Response (CDR) workflow #14629

[google_scc] Add support for Security Command Center API v2 and update finding data stream to support Cloud Detection and Response (CDR) workflow #14629

Uh oh!

Conversation

brijesh-elastic commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jul 22, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 22, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Finding Dashboard

Overview Dashboard

Finding Essentials Details

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

brijesh-elastic commented Jul 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brijesh-elastic Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold left a comment

Choose a reason for hiding this comment

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 6, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Aug 6, 2025

Quality Gate failed

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 8, 2025

Uh oh!

Uh oh!

brijesh-elastic commented Jul 22, 2025 •

edited

Loading

brijesh-elastic Jul 31, 2025 •

edited

Loading