Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ti_*: Allow transforms to run unattended mode#16535

Merged
kcreddy merged 5 commits into
elastic:mainfrom
kcreddy:ti_all-add-unattended
Dec 18, 2025
Merged

ti_*: Allow transforms to run unattended mode#16535
kcreddy merged 5 commits into
elastic:mainfrom
kcreddy:ti_all-add-unattended

Conversation

@kcreddy
Copy link
Copy Markdown
Contributor

@kcreddy kcreddy commented Dec 12, 2025

Proposed commit message

ti_*: Allow transforms to run unattended mode

Right now transforms require manual intervention to
restart them whenever there is network instability. 
This PR adds "settings.unattended = true" for all "ti_*" 
transforms to allow them auto-recover. Although the 
ideal way to solve this is by setting
 "settings.num_failure_retries: -1", it is not allowed 
by Transform API or package-spec yet.


[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml); do
    yq -i '.settings.unattended = true' "$transform"
done

for upd_transform in $(git diff --name-only packages/ | grep 'transform\.yml$'); do
    current_version=$(yq '._meta.fleet_transform_version' "$upd_transform")
    new_version=$(echo "$current_version" | awk -F. '{printf "%d.%d.%d", $1, $2+1, 0}')
    yq -i "._meta.fleet_transform_version = \"$new_version\"" "$upd_transform"
    echo "Updated $upd_transform: $current_version -> $new_version"
done

for package in $(git diff --name-only packages/ | cut -d'/' -f1,2 | sort -u) ; do
    cd $package && elastic-package changelog add --description "Allow transforms to run unattended mode." --type enhancement --next minor --link https://github.com/elastic/integrations/pull/16535
    cd ../../
done

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

@andrewkroh andrewkroh added Integration:ti_threatq ThreatQuotient (Partner supported) Integration:ti_cybersixgill Cybersixgill Integration:ti_opencti OpenCTI Integration:ti_cif3 Collective Intelligence Framework v3 (Community supported) Integration:ti_custom Custom Threat Intelligence Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:ti_abusech abuse.ch Integration:ti_maltiverse Maltiverse (Partner supported) Integration:ti_eset ESET Threat Intelligence (Partner supported) Integration:ti_otx AlienVault OTX Integration:ti_threatconnect ThreatConnect (Partner supported) Integration:ti_anomali Anomali ThreatStream Integration:ti_misp MISP Integration:ti_eclecticiq EclecticIQ (Partner supported) Integration:ti_recordedfuture Recorded Future Integration:ti_domaintools DomainTools Feeds (Partner supported) labels Dec 12, 2025
@kcreddy kcreddy self-assigned this Dec 12, 2025
@kcreddy kcreddy marked this pull request as ready for review December 12, 2025 13:11
@kcreddy kcreddy requested a review from a team as a code owner December 12, 2025 13:11
@kcreddy kcreddy added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 12, 2025
@elasticmachine
Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Copy link
Copy Markdown
Contributor

@chrisberkhout chrisberkhout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Just needs the conflict to be resolved, ideally with git-generate still applying cleanly.

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

History

cc @kcreddy

@kcreddy kcreddy merged commit 54fefd3 into elastic:main Dec 18, 2025
8 checks passed
@kcreddy kcreddy deleted the ti_all-add-unattended branch December 18, 2025 03:49
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_abusech - 3.5.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/3.5.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_anomali - 2.3.0 containing this change is available at https://epr.elastic.co/package/ti_anomali/2.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_cif3 - 1.18.0 containing this change is available at https://epr.elastic.co/package/ti_cif3/1.18.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_custom - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_custom/1.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_cybersixgill - 1.34.0 containing this change is available at https://epr.elastic.co/package/ti_cybersixgill/1.34.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_domaintools - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_domaintools/1.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_eclecticiq - 1.5.0 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.5.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_eset - 1.9.0 containing this change is available at https://epr.elastic.co/package/ti_eset/1.9.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_maltiverse - 1.6.0 containing this change is available at https://epr.elastic.co/package/ti_maltiverse/1.6.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_misp - 1.40.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.40.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_opencti - 2.12.0 containing this change is available at https://epr.elastic.co/package/ti_opencti/2.12.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_otx - 1.29.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.29.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_rapid7_threat_command - 2.6.0 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.6.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_recordedfuture - 2.4.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/2.4.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_threatconnect - 1.12.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/1.12.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package ti_threatq - 1.36.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.36.0/

kcreddy added a commit to kcreddy/integrations that referenced this pull request Apr 15, 2026
Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (elastic#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

Requires elastic/package-spec#1124 (add num_failure_retries to the
transform settings schema).

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "elastic/security-team#14926"
    cd ../../
done

Made-with: Cursor
kcreddy added a commit to kcreddy/integrations that referenced this pull request Apr 15, 2026
Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (elastic#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

Requires elastic/package-spec#1124 (add num_failure_retries to the
transform settings schema).

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "elastic#18404"
    cd ../../
done

Made-with: Cursor
kcreddy added a commit to kcreddy/integrations that referenced this pull request May 11, 2026
Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (elastic#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "elastic#18404"
    cd ../../
done

Co-authored-by: Cursor <[email protected]>
kcreddy added a commit that referenced this pull request May 12, 2026
Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "#18404"
    cd ../../
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:ti_abusech abuse.ch Integration:ti_anomali Anomali ThreatStream Integration:ti_cif3 Collective Intelligence Framework v3 (Community supported) Integration:ti_custom Custom Threat Intelligence Integration:ti_cybersixgill Cybersixgill Integration:ti_domaintools DomainTools Feeds (Partner supported) Integration:ti_eclecticiq EclecticIQ (Partner supported) Integration:ti_eset ESET Threat Intelligence (Partner supported) Integration:ti_maltiverse Maltiverse (Partner supported) Integration:ti_misp MISP Integration:ti_opencti OpenCTI Integration:ti_otx AlienVault OTX Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:ti_recordedfuture Recorded Future Integration:ti_threatconnect ThreatConnect (Partner supported) Integration:ti_threatq ThreatQuotient (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants