fix: Ensure configs are sanitized for parsing #3819
Conversation
|
This has surfaced some bugs in the test suite, I'll need to investigate those tomorrow. |
|
Is this WIP? If so, please mark as draft - otherwise I'll start to review. |
These lines will be not be processed by `read`, emit a warning to raise awareness.
This functionality was handled in `accounts.sh` via a similar sed command (that the linked references also offer). `printf` is better for this, no shellcheck comment required either. We additionally don't attempt to modify files that are read-only.
Likewise, this runtime fix was only covering two config files. It now applies to all callers of this method.
This feature should have been using the helper to avoid user error from their config updates accidentally introducing subtle breakage implicitly (due to CRLF or missing final newline).
313a776 to
936d43c
Compare
This temporary directory is created and removed implicitly. Even after a test failure.
This was introduced in 2018, there should be no one needing to rely on this anymore?
936d43c to
60017bc
Compare
It's ready for review now 👍 |
georglauterbach
left a comment
There was a problem hiding this comment.
Looks good overall, but I have left a few comments to address still.
casperklein
left a comment
There was a problem hiding this comment.
Only some minor things I saw while reviewing.
| function _get_valid_lines_from_file() { | ||
| _convert_crlf_to_lf_if_necessary "${1}" | ||
| _append_final_newline_if_missing "${1}" |
There was a problem hiding this comment.
From a logical POV: A "get" function, should not alter the provided source file.
There was a problem hiding this comment.
Yeah, if you have a suggestion to change it to a pipe instead that only transforms the text being parsed into lines instead of needing to write it, I'm all good for that 👍
There was a problem hiding this comment.
Minimal effort: Rename the function to _fix_file_and_get_content
|
|
||
| if [[ -w ${1} ]]; then | ||
| _log 'debug' 'Converting CRLF to LF' | ||
| sed -i 's|\r||g' "${1}" |
There was a problem hiding this comment.
A file change is expected, let's ensure that with:
sedfile --strict -i
There was a problem hiding this comment.
I originally did use sedfile here 🤓 but the tests run outside of the container where sedfile is not a binary on the host so it failed, not sure how to approach that?
There was a problem hiding this comment.
Okay, that actually makes sense!
There was a problem hiding this comment.
I don't understand this. So when writing scripts that are meant to run inside our container, we must take into account, that they also run on any host?
Like any other dependency, it must be present on the host. If not, sedfile should either be copied to /usr/bin or included in the PATH env: PATH=$PATH:target/bin/sedfile
For example, parallel is also a dependency that is not available on a default debian installation and have to be explicit installed in order to run tests. But that's no reason for not using parallel.
What do I miss here? Why are container scripts run the host? Why not run them in the container, where all dependencies are satisfied?
There was a problem hiding this comment.
What do I miss here? Why are container scripts run the host? Why not run them in the container, where all dependencies are satisfied?
Not missing anything, I think it was just how it was handled before and we just haven't considered moving the tests to be run within the container?
Description
This PR is to address a parsing concern with
postfix-master.cf(no final newline ignores last line of file), which happens to be the only config we loop through withreadto process individual lines from but wasn't using the common helper method.New helper methods added replace very old corrections (that we had only for
postfix-accounts.cf):sed -i -e '$a\' /tmp/postfix/accounts.cf(2015, associated issue)sed -i 's/\r//g' /tmp/docker-mailserver/postfix-accounts.cf(2016, associated issue)sed -i -e "s|, |,|g" -e "s|,$||g" "${DATABASE_VIRTUAL}"(2018, removed as this should definitely not be a concern anymore?)This is also a follow-up PR to #2820 (which provides more details on those replaced / removed sed calls)
Commit history is fairly clean with additional context in commit messages if needed.
sed '$a\'(seddocs) in particular didn't seem to offer any extra value than just aprintf '\n'.I've included several links in comment for the helper as context. That could be removed too, but that sed example was presumably taken from there (top answer), while another better clarifies why
tail -c 1+wc -lis chosen for detection.I could drop the
-wread-only conditional, I just recallsetup.bashin tests having a concern about read-only mount support. Not that our inputs should really cause any concerns there with this change. Let me know, happy to drop that guard (might affect those eventual plans for error handling magic I've seen discussed 😝 ).Type of change
Checklist:
CHANGELOG.md