Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[multiple integrations] Lowercase host.name field#6057

Merged
kcreddy merged 51 commits into
elastic:mainfrom
LaZyDK:ecs_host_name
May 24, 2023
Merged

[multiple integrations] Lowercase host.name field#6057
kcreddy merged 51 commits into
elastic:mainfrom
LaZyDK:ecs_host_name

Conversation

@LaZyDK
Copy link
Copy Markdown
Contributor

@LaZyDK LaZyDK commented May 2, 2023 •
edited
Loading

What does this PR do?

As defined in ECS 8.7, host.name is preferably lowercase.

It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host.

The below integrations have been edited to support this.

Cisco Secure Endpoint

  • Lowercase host.name

Cisco Umbrella

  • Lowercase host.name

PANW Cortex XDR

  • Lowercase host.name

PANW

  • Lowercase host.name

Fortinet Fortigate

  • Add event.type: denied when action is deny
  • Add host field
  • Lowercase host.name
  • Fix MAC address format
  • Add test

Microsoft DHCP

  • Lowercase host.name
  • remove dynamic_fields

Microsoft Defender Endpoint

  • Lowercase host.name
  • remove dynamic_fields

M365 Defender

  • Incident
  1. Add tests
  2. Add host fields
  3. Copy deviceDnsName to host.name
  4. Lowercase host.name
  • Alert:
  1. Add host fields
  2. Copy DeviceName to host.name in pipelines: alert, app and identity, device
  3. Lowercase host.name
  • Log:
  1. Add host fields
  2. Copy deviceDnsName to host.name
  3. Lowercase host.name

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

@LaZyDK LaZyDK requested a review from a team as a code owner May 2, 2023 10:01
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 2, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-05-24T08:11:27.659+0000

  • Duration: 18 min 51 sec

Test stats 🧪

Test Results
Failed 0
Passed 129
Skipped 0
Total 129

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@LaZyDK
Copy link
Copy Markdown
Contributor Author

LaZyDK commented May 2, 2023

Ready for discussions and test

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 3, 2023

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@kcreddy
Copy link
Copy Markdown
Contributor

kcreddy commented May 5, 2023

/test

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 5, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (10/10) 💚
Files 100.0% (33/33) 💚
Classes 100.0% (33/33) 💚
Methods 95.139% (274/288) 👍 3.139
Lines 92.535% (11453/12377) 👎 -1.894
Conditionals 100.0% (0/0) 💚

kcreddy
kcreddy reviewed May 5, 2023
View reviewed changes
Comment thread packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread ...soft_defender_endpoint/data_stream/log/_dev/test/pipeline/test-defenderatp.log-expected.json
Comment thread ...point/kibana/dashboard/microsoft_defender_endpoint-65402c30-ca6a-11ea-9d4d-9737a63aaa55.json
Comment thread packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/globalprotect.yml Outdated
Comment thread packages/panw_cortex_xdr/changelog.yml Outdated
LaZyDK and others added 3 commits May 5, 2023 12:00
@kcreddy
Copy link
Copy Markdown
Contributor

kcreddy commented May 5, 2023

/test

@LaZyDK
Copy link
Copy Markdown
Contributor Author

LaZyDK commented May 5, 2023 •
edited
Loading

Well that's some awkward test results for panw.

I am getting this locally:

--- Test results for package: panw - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                    │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────┼────────┼──────────────┤
│ panw    │ panos       │ pipeline  │ test-panw-panos-authentication-sample.log    │ PASS   │   48.51725ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-config-sample.log            │ PASS   │    6.00225ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-correlated-events-sample.log │ PASS   │  15.830709ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-decryption-sample.log        │ PASS   │  18.005458ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-globalprotect-sample.log     │ PASS   │     34.052ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-gtp-sample.log               │ PASS   │   5.512625ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-hipmatch-sample.log          │ PASS   │   4.934625ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-other-sample.log         │ PASS   │  17.633458ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-threat-sample.log        │ PASS   │  85.674084ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-traffic-sample.log       │ PASS   │  93.949667ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-traffic.json             │ PASS   │  10.136167ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-ip-tag-sample.log            │ PASS   │   3.899834ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-sctp-sample.log              │ PASS   │   3.654917ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-system-sample.log            │ PASS   │   3.894459ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-threat-sample.log            │ PASS   │   103.6335ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-traffic-sample.log           │ PASS   │  91.255583ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-tunnel-inspection-sample.log │ PASS   │   5.151458ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-userid-sample.log            │ PASS   │  16.124375ms │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: panw - END   ---
Done

--- Test results for package: panw - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ panw    │ panos       │ system    │ logfile   │ PASS   │ 38.838056667s │
│ panw    │ panos       │ system    │ tcp       │ PASS   │ 24.517048291s │
│ panw    │ panos       │ system    │ udp       │ PASS   │ 24.565975292s │
│ panw    │ panos       │ system    │ udp-tz    │ PASS   │ 24.312194666s │
│ panw    │ panos       │ system    │ tls       │ PASS   │ 27.659064584s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: panw - END   ---

@LaZyDK
Copy link
Copy Markdown
Contributor Author

LaZyDK commented May 5, 2023

Found the error. Generating new test results and will commit when done.

@LaZyDK
Copy link
Copy Markdown
Contributor Author

LaZyDK commented May 5, 2023

Ready for another test

@LaZyDK @efd6
Update packages/microsoft_defender_endpoint/data_stream/log/elasticse…
2cf2e84 
…arch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <[email protected]>
@kcreddy
Copy link
Copy Markdown
Contributor

kcreddy commented May 17, 2023

/test

@efd6
Copy link
Copy Markdown
Contributor

efd6 commented May 22, 2023

/test

@jamiehynds
Copy link
Copy Markdown

jamiehynds commented May 22, 2023

LGTM

@kcreddy
Copy link
Copy Markdown
Contributor

kcreddy commented May 23, 2023

/test

@P1llus
Copy link
Copy Markdown
Member

P1llus commented May 24, 2023

Just as a note for the reviewers, this will temporarily start creating duplicate host entries, because hostnames that was not previously lowercase, will now become that, which will conflict with things like SIEM rules that aggregates on hostname etc, for a short time period. This is however a healthy change for the future of the users. Should get a thumbs up from @jamiehynds as well.

@P1llus when you say a short period of time, do we know how long any impacted hostnames (i.e. not currently lowercase) will remain an issue? It's short term pain for long term gain, but might be worth creating a brief article for Support, to get ahead of issues that arise from the change. FYI @jamesspi based on the SIEM impact.

Hard to say, depends on the usecase and how long its used:

  1. Transforms without any maximum age.
  2. ML trained models for entity analytics or anything host based.
  3. Anything custom (rules, transforms etc).

I feel we should still merge, but its just good to know. The current status before we merge, is that it causes issues already due to the mix of lower/upper case, so its fixing much more.

@P1llus
Copy link
Copy Markdown
Member

P1llus commented May 24, 2023

/test

kcreddy
kcreddy approved these changes May 24, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kcreddy kcreddy merged commit 8a0bb78 into elastic:main May 24, 2023
@LaZyDK LaZyDK deleted the ecs_host_name branch May 24, 2023 09:05
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package cisco_secure_endpoint - 2.12.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_endpoint

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package cisco_umbrella - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=cisco_umbrella

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package fortinet_fortigate - 1.11.0 containing this change is available at https://epr.elastic.co/search?package=fortinet_fortigate

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package m365_defender - 1.9.0 containing this change is available at https://epr.elastic.co/search?package=m365_defender

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package microsoft_defender_endpoint - 2.11.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_defender_endpoint

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package microsoft_dhcp - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_dhcp

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package panw - 3.8.0 containing this change is available at https://epr.elastic.co/search?package=panw

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 24, 2023

Package panw_cortex_xdr - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=panw_cortex_xdr

@andrewkroh andrewkroh added Integration:cisco_secure_endpoint Cisco Secure Endpoint Integration:cisco_umbrella Cisco Umbrella Integration:fortinet_fortigate Fortinet FortiGate Firewall Logs Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint Integration:microsoft_dhcp Microsoft DHCP Integration:panw Palo Alto Next-Gen Firewall Integration:panw_cortex_xdr Palo Alto Cortex XDR labels Jul 22, 2024
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@LaZyDK @kcreddy @efd6
[multiple integrations] Lowercase host.name field (elastic#6057)
158ba7a 
* Remove empty events and anonymize test data

* Fixup fields

* Bump version

* Fix panw_cortex_xdr

* Fix panw

* Fix cisco_secure_endpoint

* Fix cisco_umbrella

* Fix fortinet_fortigate

* Fix microsoft_dhcp

* Fix microsoft_defender_endpoint

* Bump versions

* Add system test for microsoft_dhcp

* Remove questionmark

Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>

* Check for host.name

* Invert host.name check

* Commit panw tests

* Fixup panw and system tests

* Update packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/panw/manifest.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/panw/changelog.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_dhcp/manifest.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_dhcp/changelog.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/cisco_umbrella/manifest.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/cisco_umbrella/changelog.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/cisco_secure_endpoint/manifest.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/cisco_secure_endpoint/changelog.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/fortinet_fortigate/changelog.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_dhcp/data_stream/log/elasticsearch/ingest_pipeline/dhcp.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/fortinet_fortigate/manifest.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_defender_endpoint/changelog.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_defender_endpoint/manifest.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update packages/microsoft_dhcp/data_stream/log/elasticsearch/ingest_pipeline/dhcpv6.yml

Co-authored-by: Dan Kortschak <[email protected]>

* Update fortigate changelog

* Bump versions

* System tests for panw_xdr

* System tests for microsoft_defender_endpoint

* Fix m365_defender

* System tests for m365_defender

* System tests for cisco_secure_endpoint

* System tests for panw

* System tests for fortinet_fortigate

* Add m365_defender incident test

* Update packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <[email protected]>

* System test for correct elastic version

---------

Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>
Co-authored-by: Dan Kortschak <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 left review comments

@kcreddy kcreddy kcreddy approved these changes

Assignees

No one assigned

Labels

Integration:cisco_secure_endpoint Cisco Secure Endpoint Integration:cisco_umbrella Cisco Umbrella Integration:fortinet_fortigate Fortinet FortiGate Firewall Logs Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint Integration:microsoft_dhcp Microsoft DHCP Integration:panw_cortex_xdr Palo Alto Cortex XDR Integration:panw Palo Alto Next-Gen Firewall

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@LaZyDK @elasticmachine @kcreddy @efd6 @P1llus @jamiehynds @jamesspi @andrewkroh