Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

mohitjha-elastic
Copy link
Collaborator

Type of change

  • Enhancement

Proposed Commit Message

Add support of Ihealth, system, BOT and DOS logs.
Add support of new fields of ASM.
Handle N/A values in all the pipelines.
Disabled all the input type by default as per new best practice.
Removed event.type and event.category from main pipeline and set in individual pipeline.
Add tags in all the pipelines.
Add support of one more date format in event_timestamp field of ltm pipeline.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/f5_bigip directory.
Run the following command to run tests.
elastic-package test -v

Related issues

Automated Test

F5_Big-IP.log

Screenshots

F5_Big-IP1
F5_Big-IP2

1. Add support of new fields of ASM.
2. Handle N/A values in all the pipelines.
3. Disabled all the input type by default as per new best practice.
4. Removed event.type and event.category from main pipeline and set in individual pipeline.
5. Add tags in all the pipelines.
6. Add support of one more date format in event_timestamp field of ltm pipeline.
@mohitjha-elastic mohitjha-elastic requested a review from a team as a code owner July 3, 2024 15:51
@efd6
Copy link
Contributor

efd6 commented Jul 4, 2024

/test

- **AFM** is designed to reduce the hardware and extra hops required when ADC's are paired with traditional firewalls and helps to protect traffic destined for the user's data center. For more information, refer to the link [here](https://www.f5.com/products/security/advanced-firewall-manager).
- **APM** provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link [here](https://www.f5.com/products/security/access-policy-manager).
- **ASM** is F5's web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link [here](https://www.f5.com/pdf/products/big-ip-application-security-manager-overview.pdf).
- **BOT and DOS** provides detailes about the bot events and distributed dos attacks events related to F5 BIG-IP. For more information on BOT, refer to the link [here](https://my.f5.com/manage/s/article/K17680287). For more information on DOS, refer to the link [here](https://my.f5.com/manage/s/article/K75699030).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **BOT and DOS** provides detailes about the bot events and distributed dos attacks events related to F5 BIG-IP. For more information on BOT, refer to the link [here](https://my.f5.com/manage/s/article/K17680287). For more information on DOS, refer to the link [here](https://my.f5.com/manage/s/article/K75699030).
- **BOT and DOS** provides details about the bot events and distributed DOS attacks events related to F5 BIG-IP ASM. For more information on BOT, refer to the knowledge base article [here](https://my.f5.com/manage/s/article/K17680287). For more information on DOS, refer to the article [here](https://my.f5.com/manage/s/article/K75699030).

(added ASM because it's not clear from the text that this is a sub-item under that heading)

Comment on lines 31 to 32
- **System Information** provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#system-information).
- **IHealth Information** highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#ihealth-information-request).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **System Information** provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#system-information).
- **IHealth Information** highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#ihealth-information-request).
- **System Information** provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the documentation [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#system-information).
- **IHealth Information** highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the documentation [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#ihealth-information-request).

},
"f5_bigip": {
"log": {
"diagnostics": [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This really feels to me like it should be split into documents.

tag: json_event_original
target_field: json
ignore_failure: true
if: ctx.event?.original != null && !ctx.event.original.contains('device_product="Application Security Module"') && !ctx.event.original.contains('device_product=ASM')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.event?.original != null && !ctx.event.original.contains('device_product="Application Security Module"') && !ctx.event.original.contains('device_product=ASM')
if: >-
ctx.event?.original != null && !(
ctx.event.original.contains('device_product="Application Security Module"') ||
ctx.event.original.contains('device_product=ASM')
)

- pipeline:
name: '{{ IngestPipeline "pipeline_bigip_bot_and_dos" }}'
tag: pipeline_bigip_bot_and_dos
if: ctx.event?.original != null && ctx.event.original.contains('device_product="Application Security Module"') || ctx.event.original.contains('device_product=ASM')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.event?.original != null && ctx.event.original.contains('device_product="Application Security Module"') || ctx.event.original.contains('device_product=ASM')
if: >-
ctx.event?.original != null && (
ctx.event.original.contains('device_product="Application Security Module"') ||
ctx.event.original.contains('device_product=ASM')
)

field: event.original
tag: remove_event_original
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))
if: ctx.tags == null || !ctx.tags.contains('preserve_original_event')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, these are being removed elsewhere, so let's not add this (ref: #10417).

- _ingest._value.summary
tag: remove_duplicate_custom_fields_from_diagnostics_array
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')

ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
- remove:
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')

- _ingest._value.filename
tag: remove_duplicate_custom_fields_from_filename
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has not been done.

- f5_bigip.log.version
tag: remove_custom_duplicate_fields
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has not been done.

@kcreddy kcreddy added Integration:f5_bigip F5 BIG-IP Crest Contributions from Crest developement team. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jul 4, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@kcreddy kcreddy added the enhancement New feature or request label Jul 4, 2024
1. Update the readme.
2. Update the default.yml with some best practice and indentation.
3. Add some ecs mappings.
4. Change observer.product mapping to the services.
5. Remove redundant fields from the pipeline.
@mohitjha-elastic mohitjha-elastic requested review from efd6 and kcreddy July 9, 2024 13:42
description: Drops NA and 'NA' values recursively.
tag: script_to_remove_NA_values
lang: painless
source:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
source:
source: |

field: event.original
tag: remove_event_original
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, these are being removed elsewhere, so let's not add this (ref: #10417).

ignore_missing: true
- dissect:
field: f5_bigip.log.request.detail
pattern: "%{f5_bigip.log.request.method} %{f5_bigip.log.request.path} %{f5_bigip.log.request.protocol}\\r\\nHost: %{f5_bigip.log.request.host}\\r\\nConnection: %{f5_bigip.log.request.connection}\\r\\nCache-Control: %{f5_bigip.log.request.cache_control}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works for me.

diff --git a/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml b/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml
index 0b388080e6..e36cc043b9 100644
--- a/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml
+++ b/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml
@@ -335,9 +335,10 @@ processors:
       ignore_missing: true
   # Two dissect patterns are used to handle both single and multiple escape characters.
   - dissect:
+      # The literal escapes here are intentional. These appear in real-world data.
       field: f5_bigip.log.request.detail
       description: Pattern to handle multiple escape character.
-      pattern: "%{f5_bigip.log.request.method} %{f5_bigip.log.request.path} %{f5_bigip.log.request.protocol}\\r\\nHost: %{f5_bigip.log.request.host}\\r\\nConnection: %{f5_bigip.log.request.connection}\\r\\nCache-Control: %{f5_bigip.log.request.cache_control}"
+      pattern: '%{f5_bigip.log.request.method} %{f5_bigip.log.request.path} %{f5_bigip.log.request.protocol}\r\nHost: %{f5_bigip.log.request.host}\r\nConnection: %{f5_bigip.log.request.connection}\r\nCache-Control: %{f5_bigip.log.request.cache_control}'
       ignore_failure: true
   - dissect:
       field: f5_bigip.log.request.detail

description: Script to extract http_request
tag: script_to_extract_http_request
lang: painless
source:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
source:
source: |

int endIndex = message.indexOf('"', startIndex);
if (startIndex >= 0 && endIndex >= 0) {
ctx.kv.http_request = message.substring(startIndex, endIndex);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
}
}

1. Remove the removal of event.original from all the pipelines.
2. Update comment in the dissect processor.
3. Use pipe in the scripts for the source code.
@mohitjha-elastic mohitjha-elastic requested a review from efd6 July 10, 2024 12:15
- _ingest._value.filename
tag: remove_duplicate_custom_fields_from_filename
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has not been done.

- f5_bigip.log.version
tag: remove_custom_duplicate_fields
ignore_missing: true
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has not been done.

ignore_missing: true
- dissect:
field: f5_bigip.log.http.request
pattern: "%{f5_bigip.log.http.method} %{f5_bigip.log.http.path} %{f5_bigip.log.http.version}\\r\\nHost: %{f5_bigip.log.http.host}\\r\\nConnection: %{f5_bigip.log.http.connection}\\r\\nPragma: %{f5_bigip.log.http.pragma}\\r\\nCache-Control: %{f5_bigip.log.http.cache_control}\\r\\nUser-Agent: %{f5_bigip.log.http.user_agent}\\r\\n%{f5_bigip.log.http.other_headers}\\r\\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pattern: "%{f5_bigip.log.http.method} %{f5_bigip.log.http.path} %{f5_bigip.log.http.version}\\r\\nHost: %{f5_bigip.log.http.host}\\r\\nConnection: %{f5_bigip.log.http.connection}\\r\\nPragma: %{f5_bigip.log.http.pragma}\\r\\nCache-Control: %{f5_bigip.log.http.cache_control}\\r\\nUser-Agent: %{f5_bigip.log.http.user_agent}\\r\\n%{f5_bigip.log.http.other_headers}\\r\\n"
pattern: '%{f5_bigip.log.http.method} %{f5_bigip.log.http.path} %{f5_bigip.log.http.version}\r\nHost: %{f5_bigip.log.http.host}\r\nConnection: %{f5_bigip.log.http.connection}\r\nPragma: %{f5_bigip.log.http.pragma}\r\nCache-Control: %{f5_bigip.log.http.cache_control}\r\nUser-Agent: %{f5_bigip.log.http.user_agent}\r\n%{f5_bigip.log.http.other_headers}\r\n'

1. Add event.category and event.type for system event.
2. Change the double quote to single quote in dissect pattern.
3. Remove extra parenthesis.
@mohitjha-elastic mohitjha-elastic requested a review from efd6 July 11, 2024 09:30
@efd6
Copy link
Contributor

efd6 commented Jul 11, 2024

/test

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but wait for @kcreddy

@elasticmachine
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few suggestions.

"network"
],
"created": "2018-04-15T11:30:26.000Z",
"kind": "event",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

event.kind should be alert

1. Replace all foreach with the script.
2. Update event.kind of bot and dos pipeline to alert.
3. Use uri parts on the url constructued in bigasm.
4. set source.ip from client.ip in bigasm pipeline.
@mohitjha-elastic mohitjha-elastic requested a review from kcreddy July 15, 2024 09:04
@kcreddy
Copy link
Contributor

kcreddy commented Jul 15, 2024

/test

1 similar comment
@kcreddy
Copy link
Contributor

kcreddy commented Jul 15, 2024

/test

@elasticmachine
Copy link

💚 Build Succeeded

History

Copy link

Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼
Thanks!

@kcreddy kcreddy merged commit 6cdfbc9 into elastic:main Jul 15, 2024
@elasticmachine
Copy link

Package f5_bigip - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=f5_bigip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Crest Contributions from Crest developement team. enhancement New feature or request Integration:f5_bigip F5 BIG-IP Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement/Fix] - F5 BIG-IP Azure hosted (Virtual Edition) - N/A values [F5] Requesting new fields for ASM & APM
4 participants