-
Notifications
You must be signed in to change notification settings - Fork 489
[F5 BIG-IP] Add Support of IHealth, System, BOT and DOS Events #10355
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
1. Add support of new fields of ASM. 2. Handle N/A values in all the pipelines. 3. Disabled all the input type by default as per new best practice. 4. Removed event.type and event.category from main pipeline and set in individual pipeline. 5. Add tags in all the pipelines. 6. Add support of one more date format in event_timestamp field of ltm pipeline.
/test |
- **AFM** is designed to reduce the hardware and extra hops required when ADC's are paired with traditional firewalls and helps to protect traffic destined for the user's data center. For more information, refer to the link [here](https://www.f5.com/products/security/advanced-firewall-manager). | ||
- **APM** provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link [here](https://www.f5.com/products/security/access-policy-manager). | ||
- **ASM** is F5's web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link [here](https://www.f5.com/pdf/products/big-ip-application-security-manager-overview.pdf). | ||
- **BOT and DOS** provides detailes about the bot events and distributed dos attacks events related to F5 BIG-IP. For more information on BOT, refer to the link [here](https://my.f5.com/manage/s/article/K17680287). For more information on DOS, refer to the link [here](https://my.f5.com/manage/s/article/K75699030). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- **BOT and DOS** provides detailes about the bot events and distributed dos attacks events related to F5 BIG-IP. For more information on BOT, refer to the link [here](https://my.f5.com/manage/s/article/K17680287). For more information on DOS, refer to the link [here](https://my.f5.com/manage/s/article/K75699030). | |
- **BOT and DOS** provides details about the bot events and distributed DOS attacks events related to F5 BIG-IP ASM. For more information on BOT, refer to the knowledge base article [here](https://my.f5.com/manage/s/article/K17680287). For more information on DOS, refer to the article [here](https://my.f5.com/manage/s/article/K75699030). |
(added ASM because it's not clear from the text that this is a sub-item under that heading)
- **System Information** provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#system-information). | ||
- **IHealth Information** highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#ihealth-information-request). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- **System Information** provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#system-information). | |
- **IHealth Information** highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the link [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#ihealth-information-request). | |
- **System Information** provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the documentation [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#system-information). | |
- **IHealth Information** highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the documentation [here](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/output-example.html#ihealth-information-request). |
}, | ||
"f5_bigip": { | ||
"log": { | ||
"diagnostics": [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This really feels to me like it should be split into documents.
tag: json_event_original | ||
target_field: json | ||
ignore_failure: true | ||
if: ctx.event?.original != null && !ctx.event.original.contains('device_product="Application Security Module"') && !ctx.event.original.contains('device_product=ASM') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.event?.original != null && !ctx.event.original.contains('device_product="Application Security Module"') && !ctx.event.original.contains('device_product=ASM') | |
if: >- | |
ctx.event?.original != null && !( | |
ctx.event.original.contains('device_product="Application Security Module"') || | |
ctx.event.original.contains('device_product=ASM') | |
) |
- pipeline: | ||
name: '{{ IngestPipeline "pipeline_bigip_bot_and_dos" }}' | ||
tag: pipeline_bigip_bot_and_dos | ||
if: ctx.event?.original != null && ctx.event.original.contains('device_product="Application Security Module"') || ctx.event.original.contains('device_product=ASM') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.event?.original != null && ctx.event.original.contains('device_product="Application Security Module"') || ctx.event.original.contains('device_product=ASM') | |
if: >- | |
ctx.event?.original != null && ( | |
ctx.event.original.contains('device_product="Application Security Module"') || | |
ctx.event.original.contains('device_product=ASM') | |
) |
field: event.original | ||
tag: remove_event_original | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event')) | |
if: ctx.tags == null || !ctx.tags.contains('preserve_original_event') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, these are being removed elsewhere, so let's not add this (ref: #10417).
- _ingest._value.summary | ||
tag: remove_duplicate_custom_fields_from_diagnostics_array | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) | |
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields') |
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) | ||
- remove: | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) | |
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields') |
- _ingest._value.filename | ||
tag: remove_duplicate_custom_fields_from_filename | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) | |
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has not been done.
- f5_bigip.log.version | ||
tag: remove_custom_duplicate_fields | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) | |
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has not been done.
packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipsystem.yml
Show resolved
Hide resolved
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigip_bot_and_dos.yml
Outdated
Show resolved
Hide resolved
packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipsystem.yml
Outdated
Show resolved
Hide resolved
...ges/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-system.log-expected.json
Outdated
Show resolved
Hide resolved
...ges/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-system.log-expected.json
Outdated
Show resolved
Hide resolved
...ges/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-system.log-expected.json
Outdated
Show resolved
Hide resolved
1. Update the readme. 2. Update the default.yml with some best practice and indentation. 3. Add some ecs mappings. 4. Change observer.product mapping to the services. 5. Remove redundant fields from the pipeline.
packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
description: Drops NA and 'NA' values recursively. | ||
tag: script_to_remove_NA_values | ||
lang: painless | ||
source: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
source: | |
source: | |
field: event.original | ||
tag: remove_event_original | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, these are being removed elsewhere, so let's not add this (ref: #10417).
ignore_missing: true | ||
- dissect: | ||
field: f5_bigip.log.request.detail | ||
pattern: "%{f5_bigip.log.request.method} %{f5_bigip.log.request.path} %{f5_bigip.log.request.protocol}\\r\\nHost: %{f5_bigip.log.request.host}\\r\\nConnection: %{f5_bigip.log.request.connection}\\r\\nCache-Control: %{f5_bigip.log.request.cache_control}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This works for me.
diff --git a/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml b/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml
index 0b388080e6..e36cc043b9 100644
--- a/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml
+++ b/packages/f5_bigip/data_stream/log/elasticsearch/ingest_pipeline/pipeline_bigipasm.yml
@@ -335,9 +335,10 @@ processors:
ignore_missing: true
# Two dissect patterns are used to handle both single and multiple escape characters.
- dissect:
+ # The literal escapes here are intentional. These appear in real-world data.
field: f5_bigip.log.request.detail
description: Pattern to handle multiple escape character.
- pattern: "%{f5_bigip.log.request.method} %{f5_bigip.log.request.path} %{f5_bigip.log.request.protocol}\\r\\nHost: %{f5_bigip.log.request.host}\\r\\nConnection: %{f5_bigip.log.request.connection}\\r\\nCache-Control: %{f5_bigip.log.request.cache_control}"
+ pattern: '%{f5_bigip.log.request.method} %{f5_bigip.log.request.path} %{f5_bigip.log.request.protocol}\r\nHost: %{f5_bigip.log.request.host}\r\nConnection: %{f5_bigip.log.request.connection}\r\nCache-Control: %{f5_bigip.log.request.cache_control}'
ignore_failure: true
- dissect:
field: f5_bigip.log.request.detail
description: Script to extract http_request | ||
tag: script_to_extract_http_request | ||
lang: painless | ||
source: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
source: | |
source: | |
int endIndex = message.indexOf('"', startIndex); | ||
if (startIndex >= 0 && endIndex >= 0) { | ||
ctx.kv.http_request = message.substring(startIndex, endIndex); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
} | |
} |
1. Remove the removal of event.original from all the pipelines. 2. Update comment in the dissect processor. 3. Use pipe in the scripts for the source code.
- _ingest._value.filename | ||
tag: remove_duplicate_custom_fields_from_filename | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has not been done.
- f5_bigip.log.version | ||
tag: remove_custom_duplicate_fields | ||
ignore_missing: true | ||
if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has not been done.
ignore_missing: true | ||
- dissect: | ||
field: f5_bigip.log.http.request | ||
pattern: "%{f5_bigip.log.http.method} %{f5_bigip.log.http.path} %{f5_bigip.log.http.version}\\r\\nHost: %{f5_bigip.log.http.host}\\r\\nConnection: %{f5_bigip.log.http.connection}\\r\\nPragma: %{f5_bigip.log.http.pragma}\\r\\nCache-Control: %{f5_bigip.log.http.cache_control}\\r\\nUser-Agent: %{f5_bigip.log.http.user_agent}\\r\\n%{f5_bigip.log.http.other_headers}\\r\\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pattern: "%{f5_bigip.log.http.method} %{f5_bigip.log.http.path} %{f5_bigip.log.http.version}\\r\\nHost: %{f5_bigip.log.http.host}\\r\\nConnection: %{f5_bigip.log.http.connection}\\r\\nPragma: %{f5_bigip.log.http.pragma}\\r\\nCache-Control: %{f5_bigip.log.http.cache_control}\\r\\nUser-Agent: %{f5_bigip.log.http.user_agent}\\r\\n%{f5_bigip.log.http.other_headers}\\r\\n" | |
pattern: '%{f5_bigip.log.http.method} %{f5_bigip.log.http.path} %{f5_bigip.log.http.version}\r\nHost: %{f5_bigip.log.http.host}\r\nConnection: %{f5_bigip.log.http.connection}\r\nPragma: %{f5_bigip.log.http.pragma}\r\nCache-Control: %{f5_bigip.log.http.cache_control}\r\nUser-Agent: %{f5_bigip.log.http.user_agent}\r\n%{f5_bigip.log.http.other_headers}\r\n' |
1. Add event.category and event.type for system event. 2. Change the double quote to single quote in dissect pattern. 3. Remove extra parenthesis.
/test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM but wait for @kcreddy
🚀 Benchmarks reportTo see the full report comment with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few suggestions.
packages/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-asm.log-expected.json
Show resolved
Hide resolved
packages/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-asm.log-expected.json
Outdated
Show resolved
Hide resolved
packages/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-asm.log-expected.json
Show resolved
Hide resolved
"network" | ||
], | ||
"created": "2018-04-15T11:30:26.000Z", | ||
"kind": "event", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
event.kind
should be alert
...5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-dos-and-bot.log-expected.json
Show resolved
Hide resolved
1. Replace all foreach with the script. 2. Update event.kind of bot and dos pipeline to alert. 3. Use uri parts on the url constructued in bigasm. 4. set source.ip from client.ip in bigasm pipeline.
/test |
1 similar comment
/test |
💚 Build Succeeded
History
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍🏼
Thanks!
Package f5_bigip - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=f5_bigip |
Type of change
Proposed Commit Message
Checklist
changelog.yml
file.How to test this PR locally
Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/f5_bigip directory.
Run the following command to run tests.
elastic-package test -v
Related issues
Automated Test
F5_Big-IP.log
Screenshots