ti_*: Allow transforms to run unattended mode#16535
Conversation
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
🚀 Benchmarks reportTo see the full report comment with |
chrisberkhout
left a comment
There was a problem hiding this comment.
Nice. Just needs the conflict to be resolved, ideally with git-generate still applying cleanly.
💚 Build Succeeded
History
cc @kcreddy |
|
Package ti_abusech - 3.5.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/3.5.0/ |
|
Package ti_anomali - 2.3.0 containing this change is available at https://epr.elastic.co/package/ti_anomali/2.3.0/ |
|
Package ti_cif3 - 1.18.0 containing this change is available at https://epr.elastic.co/package/ti_cif3/1.18.0/ |
|
Package ti_custom - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_custom/1.3.0/ |
|
Package ti_cybersixgill - 1.34.0 containing this change is available at https://epr.elastic.co/package/ti_cybersixgill/1.34.0/ |
|
Package ti_domaintools - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_domaintools/1.3.0/ |
|
Package ti_eclecticiq - 1.5.0 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.5.0/ |
|
Package ti_eset - 1.9.0 containing this change is available at https://epr.elastic.co/package/ti_eset/1.9.0/ |
|
Package ti_maltiverse - 1.6.0 containing this change is available at https://epr.elastic.co/package/ti_maltiverse/1.6.0/ |
|
Package ti_misp - 1.40.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.40.0/ |
|
Package ti_opencti - 2.12.0 containing this change is available at https://epr.elastic.co/package/ti_opencti/2.12.0/ |
|
Package ti_otx - 1.29.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.29.0/ |
|
Package ti_rapid7_threat_command - 2.6.0 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.6.0/ |
|
Package ti_recordedfuture - 2.4.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/2.4.0/ |
|
Package ti_threatconnect - 1.12.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/1.12.0/ |
|
Package ti_threatq - 1.36.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.36.0/ |
Replace settings.unattended: true with settings.num_failure_retries: -1 in all ti_* managed transforms. Unlike unattended mode which retries all failures indefinitely (masking irrecoverable errors), num_failure_retries: -1 retries only recoverable failures while still surfacing genuinely irrecoverable ones to users. Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added after the original unattended PR (elastic#16535) had no failure resilience at all and now get num_failure_retries: -1 added. Requires elastic/package-spec#1124 (add num_failure_retries to the transform settings schema). [git-generate] for transform in $(find packages/ti_*/ -type f -name transform.yml \ -path '*/elasticsearch/transform/*'); do yq -i 'del(.settings.unattended)' "$transform" yq -i '.settings.num_failure_retries = -1' "$transform" done for transform in $(git diff --name-only packages/ | \ grep 'transform\.yml$'); do current=$(yq '._meta.fleet_transform_version' "$transform") next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}') yq -i "._meta.fleet_transform_version = \"$next\"" "$transform" done for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \ sort -u); do cd "$pkg" elastic-package changelog add \ --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \ --type enhancement --next minor \ --link "elastic/security-team#14926" cd ../../ done Made-with: Cursor
Replace settings.unattended: true with settings.num_failure_retries: -1 in all ti_* managed transforms. Unlike unattended mode which retries all failures indefinitely (masking irrecoverable errors), num_failure_retries: -1 retries only recoverable failures while still surfacing genuinely irrecoverable ones to users. Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added after the original unattended PR (elastic#16535) had no failure resilience at all and now get num_failure_retries: -1 added. Requires elastic/package-spec#1124 (add num_failure_retries to the transform settings schema). [git-generate] for transform in $(find packages/ti_*/ -type f -name transform.yml \ -path '*/elasticsearch/transform/*'); do yq -i 'del(.settings.unattended)' "$transform" yq -i '.settings.num_failure_retries = -1' "$transform" done for transform in $(git diff --name-only packages/ | \ grep 'transform\.yml$'); do current=$(yq '._meta.fleet_transform_version' "$transform") next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}') yq -i "._meta.fleet_transform_version = \"$next\"" "$transform" done for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \ sort -u); do cd "$pkg" elastic-package changelog add \ --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \ --type enhancement --next minor \ --link "elastic#18404" cd ../../ done Made-with: Cursor
Replace settings.unattended: true with settings.num_failure_retries: -1 in all ti_* managed transforms. Unlike unattended mode which retries all failures indefinitely (masking irrecoverable errors), num_failure_retries: -1 retries only recoverable failures while still surfacing genuinely irrecoverable ones to users. Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added after the original unattended PR (elastic#16535) had no failure resilience at all and now get num_failure_retries: -1 added. [git-generate] for transform in $(find packages/ti_*/ -type f -name transform.yml \ -path '*/elasticsearch/transform/*'); do yq -i 'del(.settings.unattended)' "$transform" yq -i '.settings.num_failure_retries = -1' "$transform" done for transform in $(git diff --name-only packages/ | \ grep 'transform\.yml$'); do current=$(yq '._meta.fleet_transform_version' "$transform") next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}') yq -i "._meta.fleet_transform_version = \"$next\"" "$transform" done for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \ sort -u); do cd "$pkg" elastic-package changelog add \ --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \ --type enhancement --next minor \ --link "elastic#18404" cd ../../ done Co-authored-by: Cursor <[email protected]>
Replace settings.unattended: true with settings.num_failure_retries: -1 in all ti_* managed transforms. Unlike unattended mode which retries all failures indefinitely (masking irrecoverable errors), num_failure_retries: -1 retries only recoverable failures while still surfacing genuinely irrecoverable ones to users. Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added after the original unattended PR (#16535) had no failure resilience at all and now get num_failure_retries: -1 added. [git-generate] for transform in $(find packages/ti_*/ -type f -name transform.yml \ -path '*/elasticsearch/transform/*'); do yq -i 'del(.settings.unattended)' "$transform" yq -i '.settings.num_failure_retries = -1' "$transform" done for transform in $(git diff --name-only packages/ | \ grep 'transform\.yml$'); do current=$(yq '._meta.fleet_transform_version' "$transform") next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}') yq -i "._meta.fleet_transform_version = \"$next\"" "$transform" done for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \ sort -u); do cd "$pkg" elastic-package changelog add \ --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \ --type enhancement --next minor \ --link "#18404" cd ../../ done
Proposed commit message
Checklist
changelog.ymlfile.Related issues