• I have verified with docker run --env-file where the same issue associated to podman-compose occurs.
• docker compose with a compose.yaml using env_file parses the ENV file properly.

Without quotes the trailing white-space is still part of the value, but only for docker run --env-file.

# Trailing white-space kept with `docker run --env-file`:
SA_SPAM_SUBJECT=***SPAM*** 

# `docker compose` strips the white-space and expects quote wrapping to preserve it, like a shell variable:
SA_SPAM_SUBJECT='***SPAM*** '

We could comment this out instead, or deviate from the default value (or change to undef for v14?)

I'm fine with changing the default to undef.

The equivalent --env-file with docker run however does not parse the key/value pairs this way, which likewise introduces the single quotes into the ENV value string.

Good finding 👍

SA_SPAM_SUBJECT=undef

With this change, the actual Subject will be 'undef', instead of '***SPAM*** '

I propose one of these solutions (prefer the last one):

SA_SPAM_SUBJECT=

or

#SA_SPAM_SUBJECT='***SPAM*** '

Both work fine with docker run / docker compose and doesn't change our default value.

Since this seems to be rather "old" code, undef seems to be a special value:

and according to our own docs, undef "opt[s]-out of modifying the subject for mail marked as spam".

Since this seems to be rather "old" code, undef seems to be a special value:

Thanks for pointing that out 👍

I still think the current SPAM subject line is useful and should not be emptied by setting undef.

undef opts-out of this feature (useful since by default spam is delivered to junk folder already

This is only true when using IMAP. In POP3, there are no folders.

Simple question: Can't we just insert the space after the spam prefix in our scripts? This way, you can write whatever you want in there, we could even leave the ENV file as is, and we'd only need to

1. leave this change as @polarathene proposes it in this PR as is, and
2. insert a space in our scripts.

Done.

Alternatively, I'd go with

SA_SPAM_SUBJECT=

I'll leave it up to @polarathene to decide what to do with this - I'll approve as I think we have multiple valid solutions.

TL;DR:

• POP3 does not retrieve spam that gets sent to the junk folder by default. This was my assumption and I've now verified it. Thus SA_SPAM_SUBJECT isn't providing value until opt-out of the spam to junk feature.
• I don't know of any major/popular services personally that modify the subject line of mail when they isolate spam to a junk folder.
• Some POP3 clients like ThunderBird can identify spam from the mail spam headers DMS adds, and can move that into a local junk folder. Reducing the value of SA_SPAM_SUBJECT.
• Just because you find no users changing SA_SPAM_SUBJECT to be undef does not mean users want the subject line modification. They may just not care about it or notice it with our defaults, additionally our docs weren't the best at communicating awareness of the feature.

There is no explicit need to disable it. When enabled, it just have no effect, because there are no Folders in POP3.

# Run a basic DMS instance:
docker run --rm -it \
  --hostname mail.example.test \
  --env PERMIT_DOCKER=container \
  --env ENABLE_SPAMASSASSIN=1 \
  --env SA_TAG2=3 \
  --env ENABLE_POP3=1
  --name dms \
  -p 110:110 -p 143:143 -p 587:587 \
  mailserver/docker-mailserver

# In the container create an account and send a legit mail and a spam:
setup email add [email protected] bad-password
# This will be legit, spam score was around 1.2:
swaks --server 0.0.0.0 --to [email protected] --body 'hello, your code is 42'
# This will trigger spam, score was around 3-4:
swaks --server 0.0.0.0 --to [email protected] --body 'spam' --data 'Date: %DATE%\nTo: %TO_ADDRESS%\nFrom: %FROM_ADDRESS%\nSubject: test\n%NEW_HEADERS%\n%BODY%\n' --header
'Message-Id: <[email protected]>'

Setup ThunderBird with POP3 account, I have no custom DNS here, just the DMS container as above running from WSL2 and ThunderBird connecting to it over localhost port without TLS (None was required to avoid TLS issue).

Right-click on the account address/name on the sidebar and "Get Messages", only the mail that didn't land in the Junk folder is available:

Yet the DMS logs show both mails sent were delivered and stored as expected:

Jan 14 21:33:41 mail dovecot: lmtp([email protected])<2300><cg/dHjVTpGX8CAAAUi6ngw>: sieve: msgid=<[email protected]>: stored mail into mailbox 'INBOX'
Jan 14 21:33:47 mail dovecot: lmtp([email protected])<2300><sNGlNztTpGX8CAAAUi6ngw>: sieve: msgid=<[email protected]>: fileinto action: stored mail into mailbox 'Junk'

This meets the assumption I had. POP3 users won't get such mail by default regardless. They would need to opt-out of the spam to junk defaults we have. Hence it's probably worth documenting that, as SA_SPAM_SUBJECT does not have any relevance to them by default until they opt-out of the junk folder.

SA_SPAM_SUBJECT is mostly redundant then, when you have all mail that would have that marking separated into the junk folder by default. That's why SA_SPAM_SUBJECT=undef made sense to me.

You assume, that IMAP users don't want the spam subject. I am not sure if that is really the case.
I looked through various bug reports with their posted compose.yml / mailserver.env files and couldn't find a single one, where someone used SA_SPAM_SUBJECT= or SA_SPAM_SUBJECT=undef.

The lack of it doesn't mean that users were happy with SA_SPAM_SUBJECT default. They may not have cared either way, or weren't aware that it could be easily customized. I only recently improved docs on this to help communicate that better.

Did you find anyone who actually customized the default to something else? I can only imagine it's very small amount, since there should be at least one from someone who wanted/implemented the feature in the first place. Along with the others that added additional contributions to the original wiki with advice on how to get white-space separator or the spam score. This is understandable since they are users actively interested in the feature.

Since such mail goes to the junk folder by default, I'd wager many users just don't think about it or see it much that it concerns them? Even by looking at the junk folder, they'll have all mails listed with a consistent prefix, so it's less likely to stand out or look odd to them. That doesn't mean they actually want that and would actively opt-in to that if we defaulted to undef.

I think the majority prefer the spam subject. If some don't want that, they can change it with less effort.
Yes, the same would go for POP3 users to enable that.
But DMS should work for all users out of the box with sane defaults.

We can either document the better defaults for POP3, or you can conditionally adjust defaults 🤷‍♂️

I'd just like to know the value of the subject prefix adding SPAM to mail that gets lumped into a junk folder by default and as shown above does not get retrieved when POP3 is used.

Regardless of IMAP/POP3, marking SPAM messages for all users is what I consider sane.
Everyone is free to opt out. This is better instead of every POP3 user must opt in to mark spam mails.

POP3 is already an opt-in feature. It's not a big ask (especially given the above findings) to inform the user of other ENV to change. SA_SPAM_SUBJECT will be more relevant to the POP3 user, so it also makes more sense for them to customize that to what they prefer.

I don't mind the default being adjusted from undef when POP3 is enabled (or rather spam to junk is disabled). That makes more sense as sane defaults to me 🤔

How to you come to this confidence, that the change will better suits for IMAP users? Altering the subject is a common practice and I've never heard that this is a bad one.

I only use Gmail and Outlook/Live, but neither of these alter the subject line AFAIK? What popular services are doing this for you when they isolate detected spam mail to a junk folder?

In the case of ThunderBird, here are the Junk settings it offers which would be relevant to POP3 users. It has it's own spam analysis and moving spam to junk folder features. Notably (disabled by default) is that "Trust junk mail headers" feature, which will identify spam when DMS marks it with spam headers... that's much nicer, it's implicit and works just as well without modifying a subject line.

Other POP3 clients perhaps aren't as capable of such features, so having a clearly communicated SPAM prefix to your mails subject line can make sense. I'm just showing that it's not necessarily important to POP3 clients either.

Simple question: Can't we just insert the space after the spam prefix in our scripts? This way, you can write whatever you want in there

What if a user wants no trailing white-space? 🤷‍♂️

Probably unlikely though, so that would be nice... however the feature is already in use and this could introduce additional white-space that's already been accounted for, or you add maintenance burden with extra conditional logic :\

I'll leave it up to @polarathene to decide what to do with this - I'll approve as I think we have multiple valid solutions.

I personally think SA_SPAM_SUBJECT=undef makes sense, it'd be more consistent with rspamd which will replace SA AFAIK (unless there are plans to use the integration available with rspamd?), so this sort of breaking change is probably going to land eventually anyway..

I'm not a fan of mailserver.env, I'd rather remove it entirely but I'll resolve this by commenting out the ENV.

Thanks for taking the time and doing extended testing!

POP3 does not retrieve spam that gets sent to the junk folder by default. This was my assumption and I've now verified it. Thus SA_SPAM_SUBJECT isn't providing value until opt-out of the spam to junk feature.

I didn't know that, I had always MOVE_SPAM_TO_JUNK=0. That's why I've never had problems and got the spam mails. So in a POP3 only setup, the user does not have access to the junk folder. Mails ending up there, are not accessable. That means the user is not aware of mails tagged as spam + misses false/positives. This could be fixed by printing a warning if IMAP=0 and POP3=1 and MOVE_SPAM_TO_JUNK=1

Just because you find no users changing SA_SPAM_SUBJECT to be undef does not mean users want the subject line modification.

100% ack. But the same goes for the opposite :-D

I'm not a fan of mailserver.env, I'd rather remove it entirely but I'll resolve this by commenting out the ENV.

Same here, I apply my ENVs via the compose file. Only while testing, I found it sometimes useful, because I didn't have to open the online documentation, when I am not exactly sure, what a ENV does. If we decide to remove the .env file, we could alternatively provide an additional compose.override.yml file (like here) with just all ENVs listed. That way, you can see all ENVs available + work arround the quoting issues with docker run .. using an .env file

I only use Gmail and Outlook/Live, but neither of these alter the subject line AFAIK?

Just guessing: Probably because they want their users to use IMAP? As we found out, modifying the subject is mostly useful with POP3. But for security reasons, not everyone is fine with IMAP. Personally, I don't want my >20 years of mail history to be stored on a public server, regardless if its my own or run by one of the big players. That's the main point of IMAP: storing all mails on a central server.
However, within a company network, I fully agree, that it makes sense to store all mails at central servers.

Thanks for taking the time and doing extended testing!

❤️

That means the user is not aware of mails tagged as spam + misses false/positives. This could be fixed by printing a warning if IMAP=0 and POP3=1 and MOVE_SPAM_TO_JUNK=1

A warning is probably a good addition to raising that awareness yep 👍

If we decide to remove the .env file, we could alternatively provide an additional compose.override.yml file (like here) with just all ENVs listed

I think that is a slight improvement, but my main gripe is maintaining both. IIRC they're not in sync with the docs (which can also better convey extra information). I also loathe user reports that dump the whole file in the report where we're left to figure out what they explicitly changed from the defaults (since many would assign a default value)

Only while testing, I found it sometimes useful, because I didn't have to open the online documentation, when I am not exactly sure, what a ENV does

I can understand that convenience, but really it's the difference of browsing a file (via some app or terminal) vs a docs page.

The main issue with the docs page is there's a bit more noise going on, it doesn't feel as nice to navigate/explore for an overview of ENV.

We could better organize that further, and perhaps throw in a table overview at the top with key/value columns to show the defaults and each key linking to the respective ENV setting for more information.

Probably because they want their users to use IMAP? As we found out, modifying the subject is mostly useful with POP3. But for security reasons, not everyone is fine with IMAP.

👍 I was just asking as you gave the impression it was common practice even with a Junk folder storing the spam 😝