Hi there,

I apologize for this big monolithic PR. Since we needed many new functionalities fast to get our automation pipelines running for many customers, we decided to continue implementing this in our fork. Now we have a good set of features and fixes that we would like to contribute back.

I hope that is also somehow beneficial for you and the main repo. At least a lot of time and resources were spent implementing all those new features. In the best case, you can benefit from the changes, and we can benefit from you continuing to maintain it.

We can also arrange a meeting to discuss the changes if you like :)

Features and Fixes

• kibana export-rules / and some features are also applicable for the import-rules-to-repo command
  • Supports exporting timeline templates that are referenced by rules we export to toml files
  • Supports exporting value lists that are referenced in exception lists we export as plain value list files
  • --strip-dates support to not include dates that change regularly during export import loops in files on disk
  • --strip-exception-list-id to not include the ids of exception lists, since list_id is the identifier and is used to identify a list in the cluster, the id changes regularly and can be excluded this way in files on disk
• kibana import-rules / and some features are also applicable for the export-rules-from-repo command
  • Supports importing timeline templates that are referenced by rules we import, --overwrite-timeline-templates to trigger overwrite if desired
  • Supports importing value lists that are referenced in exception lists we import, --overwrite-value-lists to trigger overwrite if desired
  • --rule-name can import rules based on rule-name which is parsed from files on disk
  • --exclude-exceptions flag to exclude exceptions with wildcards from being imported
• Tests
  • If the bypass_optional_elastic_validation flag is set in the config, the default test selection is nicely tuned to work with custom rules now. Also, auto-skip tests that rely on the origin/main branch if that's not available.
  • Added a mini test for the new timeline templates
• Mini feature: if no directory is specified via the -d flag, by default the first entry of the rule dirs in the config is used now
• Mini feature: default_author and strip_version can also be set via config now

Insights into some files that changed a lot:

• The kibane import-rules function changed the most, because of the new features and the complexity of importing rules with references to timelines and value lists. It was refactored completely to be more maintainable. It includes many checks for missing, skipped, or failed imports and tries to address many of the sync edge cases which can happen when importing a set of rules with references to a Kibana instance that is not empty.
• The lib/kibana library changed a lot to support all the new API calls and error handling for the different APIs and responses.

Testing and Known Issues:

• Most testing was done for the kibana export-rules and import-rules commands, but always on a limited set of rules. Would be beneficial to test with a larger set of rules and different rule types.
• The export function stays basically the same when not specifying the new flags to include timelines and value lists. The import function changed more significantly, but should also work as before when no value lists and templates are available in the repo. It will simply issue messages that no timelines or value lists were found to import.
• Different cases when overwriting timelines and value lists were tested, but the performance is far from optimal yet for large sets, and the API calls need to be optimized in the future. However, this shouldn't be a problem if no value lists or timelines are used since the main rules, exceptions, and actions import logic is still the same.
• Introduced new features, and flags were synced to all relevant help texts and the docs and example config files, etc. as good as possible.
• There are most likely some edge cases that are not covered yet and there might be some bugs whith certain CLI commands or arguments we simply didn't use yet.
• And there are most likely some coding practices that could be improved and are not yet up to your standards.

I went through the whole diff again and checked each file and line once more, and at least I think that I could still explain the purpose of each change if you need help understanding something.

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?