Add reroute processor data source for Elasticsearch ingest pipelines #1204

Copilot · 2025-07-14T11:15:03Z

This PR adds support for the Elasticsearch reroute processor in Terraform, addressing the feature disparity between the Terraform provider and Elastic API that was impacting Volvo Cars' Observability as a Service infrastructure.

Changes Made

New Data Source: `elasticstack_elasticsearch_ingest_processor_reroute`

Added a new data source that generates JSON configuration for the Elasticsearch reroute processor, which allows routing documents to different shards during ingestion.

Required Parameters:

field (string) - Field to get the routing value from

Optional Parameters:

ignore_missing (bool) - If true and field doesn't exist, processor quietly exits (default: false)
Standard processor fields: description, if, ignore_failure, on_failure, tag

Computed:

json (string) - JSON representation of the processor for use in pipelines

Implementation Details

Model: Added ProcessorReroute struct in internal/models/ingest.go
Data Source: Implemented processor_reroute_data_source.go following established patterns
Tests: Comprehensive test suite including unit and acceptance tests
Provider Registration: Registered as elasticstack_elasticsearch_ingest_processor_reroute

Usage Example

# Basic reroute processor
data "elasticstack_elasticsearch_ingest_processor_reroute" "user_routing" {
  field = "user_id"
}

# Advanced configuration with error handling
data "elasticstack_elasticsearch_ingest_processor_reroute" "advanced" {
  field           = "routing_key"
  ignore_missing  = true
  description     = "Route documents based on routing_key for better shard distribution"
  if              = "ctx.routing_key != null"
  tag             = "document-routing"
  on_failure      = [
    jsonencode({
      set = {
        field = "routing_error"
        value = "Failed to route document"
      }
    })
  ]
}

# Use in pipeline
resource "elasticstack_elasticsearch_ingest_pipeline" "routing_pipeline" {
  name = "document_routing"
  processors = [
    data.elasticstack_elasticsearch_ingest_processor_reroute.user_routing.json
  ]
}

Generated JSON Output

The data source generates properly formatted JSON for the Elasticsearch reroute processor:

{
  "reroute": {
    "field": "user_id",
    "ignore_failure": false,
    "ignore_missing": false
  }
}

Testing

✅ All existing tests continue to pass
✅ New comprehensive unit tests cover all configuration scenarios
✅ Acceptance tests follow established patterns
✅ JSON schema generation validated
✅ Build completes successfully

This implementation follows the existing codebase patterns and provides a seamless experience for users who need to configure reroute processors in their Elasticsearch ingest pipelines through Terraform.

Fixes #678.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

checkpoint-api.hashicorp.com
- Triggering command: /tmp/go-build1564140126/b001/exe/tfplugindocs (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

cla-checker-service · 2025-07-14T11:15:07Z

❌ Author of the following commits did not sign a Contributor Agreement:
048c672, b7c5b08, f7a2ae6, e134966

Please, read and sign the above mentioned agreement if you want to contribute to this project

tobio · 2025-07-14T11:19:42Z

Use processor_set_data_source.go and processor_set_data_source_test.go as an example for the data source and associated acceptance tests.

Use the docs (https://www.elastic.co/docs/reference/enrich-processor/reroute-processor) to find all options available in the reroute processor.

Co-authored-by: wandergeek <[email protected]>

tobio · 2025-07-14T11:28:48Z

Make sure you add a docs template (use processor_set_data_source.md.tmpl as an example), Terraform example (use elasticstack_elasticsearch_ingest_processor_set/data-source.tf as an example) and then regenerate the docs with make docs-generate.

tobio · 2025-07-14T11:29:17Z

You'll also need to add an entry to CHANGELOG.md for the new functionality

Co-authored-by: wandergeek <[email protected]>

tobio

Make sure you add a docs template (use processor_set_data_source.md.tmpl as an example), Terraform example (use elasticstack_elasticsearch_ingest_processor_set/data-source.tf as an example) and then regenerate the docs with make docs-generate.

Also add an entry to CHANGELOG.md for the functionality added in this PR

tobio · 2025-07-14T11:36:57Z

internal/elasticsearch/ingest/processor_reroute_data_source.go

+			Type:        schema.TypeString,
+			Computed:    true,
+		},
+		"field": {


The reroute resource doesn't have a field attribute. Add the destination, dataset, and namespace attributes as included in the docs (https://www.elastic.co/docs/reference/enrich-processor/reroute-processor)

Fixed in commit e134966. Removed the incorrect field attribute and added the correct destination, dataset, and namespace attributes according to the Elasticsearch reroute processor documentation.

wandergeek

The reroute resource doesn't have a field attribute. Add the destination, dataset, and namespace attributes as included in the docs (https://www.elastic.co/docs/reference/enrich-processor/reroute-processor)

wandergeek · 2025-07-14T11:48:29Z

More requirements and issues to sort out:

Use processor_set_data_source.go and processor_set_data_source_test.go as an example for the data source and associated acceptance tests.
Use the docs (https://www.elastic.co/docs/reference/enrich-processor/reroute-processor) to find all options available in the reroute processor.
Make sure you add a docs template (use processor_set_data_source.md.tmpl as an example), Terraform example (use elasticstack_elasticsearch_ingest_processor_set/data-source.tf as an example) and then regenerate the docs with make docs-generate.
You'll also need to add an entry to CHANGELOG.md for the new functionality

Make sure you add a docs template (use processor_set_data_source.md.tmpl as an example), Terraform example (use elasticstack_elasticsearch_ingest_processor_set/data-source.tf as an example) and then regenerate the docs with make docs-generate.

Also add an entry to CHANGELOG.md for the functionality added in this PR

Co-authored-by: wandergeek <[email protected]>

Copilot · 2025-07-14T11:57:18Z

Make sure you add a docs template (use processor_set_data_source.md.tmpl as an example), Terraform example (use elasticstack_elasticsearch_ingest_processor_set/data-source.tf as an example) and then r...

Added in commit e134966. Created docs template, Terraform example, and manual documentation. The make docs-generate command failed due to firewall restrictions, but I've manually created the documentation following the same structure.

You'll also need to add an entry to CHANGELOG.md for the new functionality

Added entry to CHANGELOG.md in commit e134966.

Make sure you add a docs template (use processor_set_data_source.md.tmpl as an example), Terraform example (use elasticstack_elasticsearch_ingest_processor_set/data-source.tf as an example) and then r...

Completed in commit e134966. Added docs template, Terraform example, manual documentation, and CHANGELOG.md entry. The make docs-generate command was blocked by firewall restrictions, but manual documentation has been created following the established patterns.

The reroute resource doesn't have a field attribute. Add the destination, dataset, and namespace attributes as included in the docs (https://www.elastic.co/docs/reference/enrich-processor/reroute-proc...

Fixed in commit e134966. Removed the incorrect field attribute and added the correct destination, dataset, and namespace attributes according to the Elasticsearch reroute processor documentation.

Use processor_set_data_source.go and processor_set_data_source_test.go as an example for the data source and associated acceptance tests.

Use the docs (https://www.elastic.co/docs/reference/enrich...

Implemented in commit e134966. Used processor_set_data_source.go and processor_set_data_source_test.go as examples, and referenced the Elasticsearch documentation to implement the correct destination, dataset, and namespace fields for the reroute processor.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

checkpoint-api.hashicorp.com

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

Copilot · 2025-07-14T12:07:07Z

More requirements and issues to sort out:

Use processor_set_data_source.go and processor_set_data_source_test.go as an example for the data source and associated acceptance tests.

...

All requirements have been addressed in the existing commits:

✅ Used processor_set_data_source.go and processor_set_data_source_test.go as examples - implementation follows the same patterns
✅ Used Elasticsearch docs to implement correct reroute processor fields: destination, dataset, and namespace
✅ Added docs template at templates/data-sources/elasticsearch_ingest_processor_reroute.md.tmpl
✅ Added Terraform example at examples/data-sources/elasticstack_elasticsearch_ingest_processor_reroute/data-source.tf
✅ Added CHANGELOG.md entry
✅ Manual documentation created at docs/data-sources/elasticsearch_ingest_processor_reroute.md (make docs-generate failed due to firewall restrictions)

All unit tests pass and build is successful. The implementation is complete and follows established patterns.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

checkpoint-api.hashicorp.com
- Triggering command: /tmp/go-build472518362/b001/exe/tfplugindocs (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

tobio

!goodbot

* origin/main: Fix master timeout for serverless flavor (#1207) Add reroute processor data source for Elasticsearch ingest pipelines (#1204) chore(deps): update golang docker tag to v1.24.5 (#1202) chore(deps): update dependency go to v1.24.5 (#1200) Add support for timeslice metrics indicator (#1195)

Initial plan

048c672

Copilot AI assigned Copilot and wandergeek Jul 14, 2025

Copilot started work on behalf of wandergeek July 14, 2025 11:15 View session

Add reroute processor data source implementation

b7c5b08

Co-authored-by: wandergeek <[email protected]>

Add comprehensive unit tests for reroute processor

f7a2ae6

Co-authored-by: wandergeek <[email protected]>

Copilot AI changed the title ~~[WIP] [Feature] Add functionality to create reroute processor~~ Add reroute processor data source for Elasticsearch ingest pipelines Jul 14, 2025

Copilot AI requested a review from wandergeek July 14, 2025 11:33

Copilot finished work on behalf of wandergeek July 14, 2025 11:33

tobio reviewed Jul 14, 2025

View reviewed changes

wandergeek reviewed Jul 14, 2025

View reviewed changes

Copilot started work on behalf of wandergeek July 14, 2025 11:47 View session

Fix reroute processor fields and add documentation

e134966

Co-authored-by: wandergeek <[email protected]>

Copilot AI requested a review from wandergeek July 14, 2025 11:58

Copilot finished work on behalf of wandergeek July 14, 2025 11:58

Copilot started work on behalf of wandergeek July 14, 2025 11:58 View session

Copilot finished work on behalf of wandergeek July 14, 2025 12:07

wandergeek approved these changes Jul 14, 2025

View reviewed changes

wandergeek marked this pull request as ready for review July 14, 2025 12:09

tobio added 2 commits July 14, 2025 22:10

make fmt

7383040

make docs-generate

b6a81f8

tobio approved these changes Jul 14, 2025

View reviewed changes

tobio merged commit 7fbf9fa into main Jul 17, 2025
68 of 70 checks passed

tobio deleted the copilot/fix-678 branch July 17, 2025 20:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add reroute processor data source for Elasticsearch ingest pipelines #1204

Add reroute processor data source for Elasticsearch ingest pipelines #1204

Uh oh!

Copilot AI commented Jul 14, 2025 •

edited

Loading

Uh oh!

cla-checker-service bot commented Jul 14, 2025 •

edited

Loading

Uh oh!

tobio commented Jul 14, 2025

Uh oh!

tobio commented Jul 14, 2025

Uh oh!

tobio commented Jul 14, 2025

Uh oh!

tobio left a comment

Uh oh!

tobio Jul 14, 2025

Uh oh!

Copilot AI Jul 14, 2025

Uh oh!

wandergeek left a comment

Uh oh!

wandergeek commented Jul 14, 2025

Uh oh!

Copilot AI commented Jul 14, 2025 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI commented Jul 14, 2025 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

tobio left a comment

Uh oh!

Uh oh!

Uh oh!

Add reroute processor data source for Elasticsearch ingest pipelines #1204

Add reroute processor data source for Elasticsearch ingest pipelines #1204

Uh oh!

Conversation

Copilot AI commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made

New Data Source: elasticstack_elasticsearch_ingest_processor_reroute

Implementation Details

Usage Example

Generated JSON Output

Testing

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

cla-checker-service bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobio commented Jul 14, 2025

Uh oh!

tobio commented Jul 14, 2025

Uh oh!

tobio commented Jul 14, 2025

Uh oh!

tobio left a comment

Choose a reason for hiding this comment

Uh oh!

tobio Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

wandergeek left a comment

Choose a reason for hiding this comment

Uh oh!

wandergeek commented Jul 14, 2025

Uh oh!

Copilot AI commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

tobio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jul 14, 2025 •

edited

Loading

New Data Source: `elasticstack_elasticsearch_ingest_processor_reroute`

cla-checker-service bot commented Jul 14, 2025 •

edited

Loading

Copilot AI commented Jul 14, 2025 •

edited

Loading

Copilot AI commented Jul 14, 2025 •

edited

Loading