Rewrite Env2yaml in java instead of Go #18423
Conversation
🤖 GitHub commentsJust comment with:
|
|
This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
|
|
/run exhaustive tests |
|
Failure looks suspiciously related to a |
Managing a Go toolchain for persisting ENV vars in logstash container artifacts has become cumbersome. We already manage a java runtime so this commit presents a path forward to use that instead of Go. The Go binary is faster than java (in my testing Go would complete in around less than 200ms while java takes over 300ms). Given the container startup time is on the order of magnitute of seconds this change should be inperceptable to consumers. The benefit from consolidating in Java is worth the slightly lower performance.
1a3eb07 to
aa61b56
Compare
|
Double checked what copilot came up with. Also to show how insignificant the performance is i did this simple ruby test: #!/usr/bin/env ruby
GO_IMAGE = "docker.elastic.co/logstash/logstash:9.2.0"
JAVA_IMAGE = "772bb6342461"
def measure_env2yaml(image, runs = 100)
times = []
runs.times do
start = Time.now
system("docker run --rm -e PIPELINE_WORKERS=4 --entrypoint='' #{image} env2yaml /usr/share/logstash/config/logstash.yml > /dev/null 2>&1")
times << ((Time.now - start) * 1000).to_i
end
times.sum / times.size
end
puts "=== env2yaml execution time (#{100} runs) ==="
puts "Go: #{measure_env2yaml(GO_IMAGE)}ms avg"
puts "Java: #{measure_env2yaml(JAVA_IMAGE)}ms avg"example run: |
|
Still need to do some more |
|
Still working through how ironbank is different. Will push an update when possible. The java re-write is ready for review (and works in the other container), just making sure to get all the golang stuff out of the repo. |
|
/run exhaustive tests |
|
ironbank has been sorted out and i think i've got all the other references to the golang verions removed. Kicked off exhaustive test run https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/2897 |
|
Exhaustive tests for packages (unrelated to container only env2yaml) are failing due to a separate issue #18424 |
| @@ -0,0 +1,203 @@ | |||
| import org.yaml.snakeyaml.Yaml; | |||
There was a problem hiding this comment.
could we use snakeyaml-engine instaed of snakeyaml? couple of reasons:
- we don't need a lot of the higher level snakeyaml machinery for this very simple yaml parsing
- reduces surface for vulnerabilities and cve findings, given most are typically found in snakeyaml itself and not snakeyaml-engine
There was a problem hiding this comment.
I didnt even know about this 😅 I see it ships with jruby. I pushed a commit using that instead. The only difference i can spot is that it seems to want to quote values (which in my testing does not matter). Take a look, we can always revert back to snakeyaml if needed (assuming we continue to ship that with logstash).
| # The .class file is compiled during Docker build and placed in /usr/local/bin | ||
|
|
||
| exec /usr/share/logstash/jdk/bin/java \ | ||
| -cp "/usr/share/logstash/logstash-core/lib/jars/*:/usr/local/bin" \ |
There was a problem hiding this comment.
I have tested a series of JVM flags to make startup faster. overall I can see ever-so-slight benefits with -XX:+UseSerialGC -Xms16m -Xmx16m, I tried messing with tiered complation settings but no obvious benefit. best tests are likely run on a very limited container.
There was a problem hiding this comment.
Nice, updated with these. I think 16m is probably fine, do you feel comfortable with that or should we bump it a bit?
Use snakeyaml-engine and some java flags for faster execution
💚 Build Succeeded
History
|
Release notes
[rn:skip]
What does this PR do?
Managing a Go toolchain for persisting ENV vars in logstash container artifacts has become cumbersome. We already manage a java runtime so this commit presents a path forward to use that instead of Go. The Go binary is faster than java (in my testing Go would complete in around less than 200ms while java takes over 300ms). Given the container startup time is on the order of magnitute of seconds this change should be inperceptable to consumers. The benefit from consolidating in Java is worth the slightly lower performance.
Why is it important/What is the impact to the user?
This should not be noticeable, though technically starting logstash in a container will take about 200ms longer.
Checklist
How to test this PR locally
Build container:
Run env2yaml directly or check at startup.
Example: