[Cloud Security] [Findings Ingest Pipeline] Remove cloud.account.id and cloud.account.name empty fields from the latest findings document #9470
Conversation
🚀 Benchmarks reportTo see the full report comment with |
| description: 'Backward compatibility cloudbeat version < 8.8' | ||
| ignore_empty_value: true | ||
| if: ctx.orchestrator?.cluster?.id == null | ||
| - remove: |
There was a problem hiding this comment.
:aha moment
So the integration for cloud security posture handles the transformation of data collected from cloudbeat to the latest findings index.
:knowledge question
Is this rule in the ingestion pipeline removing the field from the document before it ends up the in the findings?
I am assuming this because the expected test results shows an empty object for the account
There was a problem hiding this comment.
So the integration for cloud security posture handles the transformation of data collected from cloudbeat to the latest findings index.
:knowledge question
Is this rule in the ingestion pipeline removing the field from the document before it ends up the in the findings?
I am assuming this because the expected test results shows an empty object for the account
Yes, and It's directly on the data stream index logs-cloud_security_posture.findings-* actually, (so it's before the transform runs to that aggregates the data into the logs-cloud_security_posture.findings_latest-* index.
The Ingest pipeline runs at the moment the data going to be inserted into the findings index
And how it's knows that it needs to run this pipeline for that index is defined with Index Templates, it's where the findings mappings are defined as well the default pipeline:
Co-authored-by: seanrathier <[email protected]>
💚 Build Succeeded
History
|
|
100.0% Coverage on New Code
0.0% Duplication on New Code|
Package cloud_security_posture - 1.9.0-preview03 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture |
|
Package cloud_security_posture - 1.9.0 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture |
Summary
It fixes elastic/kibana#178904
This PR updates the Findings Ingest Pipeline to remove empty
cloud.account.idandcloud.account.namefields in a Findings document. This fixes issues in Kibana when those fields are empty.This PR also introduces Ingest Pipelines tests and adds tests to cover pipeline processors in a findings document. It can be tested following the documentation here
Screenshot
Pipeline Tests
Kibana Tests
Before changes:
Fields with empty
cloud.account.nameBug in group by cloud account visualization, result is hidden due to empty field #178904
After changes:
No more fields with empty
cloud.account.nameFields are instead missing (expected behaviour)
Group by cloud account visualization shows "No cloud account" (expected behaviour)
