X-Original-DMARC-Record header field may break message

[balert]

Version

sympa 6.2.60-1.el7 on CentOS Linux release 7.9.2009 (Core)

Installation method

rpm

Expected behavior

Correct display of german umlauts in mails from the list that have configured a message_footer.

Actual behavior

German umlauts (öäüÖÄÜß​) get corrupted in the main email body (not the message_footer!) whenever we use any form of message_footer.
Example: Sending "test öäüÖÄÜß​" as the email content will end up as "test öäüÖÄÜß​" in the email coming back from the sympa list.

Additional information

Changing footer_type seems not to help, the umlauts get corrupted in both cases (append and mime).
As soon as we remove the message_footer (clearing the message_footer field in the message_templates settings section) the umlauts get displayed correctly.

[racke]

I couldn't reproduce that problem on a 6.2.60 instance (Debian 10).

Can you please post a message that you sent to a Sympa list (where the umlauts are intact) as EML file which caused the corrupted message going out from the Sympa list?

[balert]

Sure, find attached one eml-file which caused the problem. Please note, that I replaced the original listname and my sender address AFTER saving the eml-file with a text editor (sublime-text) for privacy reasons.
test.zip

[ikedas]

@balert , also, can you show attach the message coming back from Sympa? Confidential things (List name, sender address, host name, IP address, ...) may be replaced as you have done on original message.

[balert]

Sure, see attachment.
received.zip

[ikedas]

@balert , thank you for information.

As far as I can see, corrupt message was modified twice. I'm not sure the former modification is due to Sympa. However, I think the latter modification is not due to Sympa, because the format of the message is not what Sympa's function may generate.

I'd like to investigate further. If possible, could you attach these things also?

• the message_footer file that was used when the message was corrupt.
• the messge coming back when you "removed the message_footer".

[balert]

Hi, thanks, what do you mean exactly by modification? Maybe some virus or spam filter? Find attached the requested files. message_footer is the footer that causes the corruption. The eml-file is now WITHOUT this footer, and a working example with correct characters.
files.zip

[ikedas]

Hi, thanks, what do you mean exactly by modification? Maybe some virus or spam filter?

Sort of. The modification I found out is exactly:

1. Converting message body to UTF-8 charset, mistakenly assuming the body is encoded by Latin-1 charset.
2. Converting again with the same manner.
3. Change transfer-encoding from 8bit (no transfer-encoding) to quoted-printable.

By these conversion, the original message

<<snip>>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit

test öäüÖÄÜß

was modified to

<<snip>>
Content-Transfer-Encoding: quoted-printable

test =C3=83=C2=B6=C3=83=C2=A4=C3=83=C2=BC=C3=83=C2=96=C3=83=C2=84=C3=83=C2=
=9C=C3=83=C2=9F=C3=A2=C2=80=C2=8B
test

(the last line "test" is the content of message_footer).

Among above, 3 is not due to Sympa, because new Content-Transfer-Encoding field was appended (Sympa won't do such). For instance, I have seen several installation of Sendmail MTA doing the conversion like 3.

I'm not sure whether the conversion 1 and 2 were done by Sympa or the other part.

[ikedas]

@balert, can you check if the message is changed before when it is received by Sympa? To do it,

• Make the list in question moderated (set send parameter to editorkey).
• Send the original message (it will be held for moderation).
• Check the content of moderation spool (/var/spool/sympa/moderation).

Thanks.

[balert]

thanks for the advice, I tried that out and I believe sympa is not causing it but something else instead. Sorry. The following is what I see in /var/spool/sympa/moderation:

Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit

test öäüÖÄÜß
-- 

[ikedas]

It proves that any part before Sympa didn't modify the message. Then what'll happen if you approved that moderated message and let it be sent?

[balert]

I've created a new example with one eml file before moderation and then after delivery to the user. I've already talked to our email system administrator, she claims all the systems involved should not alter the message in general.
moderation.zip

[ikedas]

I've created a new example with one eml file before moderation and then after delivery to the user. I've already talked to our email system administrator, she claims all the systems involved should not alter the message in general.

However, in the first example, a new header field and a Content-Transfer-Encoding field were appended to the header. In particular former new field should not be added by Sympa: It is considered being added by any part after Sympa, either administered by her or not.

I’ll check the attachment later. Thanks!

[ikedas]

I've created a new example with one eml file before moderation and then after delivery to the user.

1. How did you get the file held-back-for-moderation.eml? This doesn't look being picked from moderation spool.
2. It's obvious that delivered-to-user.eml has been modified by the other party after Sympa. Because, modification by Sympa is adding the fields from X-Validation-by: to X-Original-DMARC-Record, and the rest of modification that is adding the fields from X-PMX-Spam to Content-Transfer-Encoding is not due to Sympa.

On 1., please get the message held in the moderation spool directory.

On 2., please investigate the system that is adding X-PMX-Spam field.

[racke]

It looks like Sophos is adding the X-PMX-Spam header: https://docs.sophos.com/msg/pmx/help/en-us/msg/pmx/concepts/AdmDevTutorialCustomizingPolicy-DetectSpam.html

[balert]

Oh, I just used the eml attached to the moderation request email I've received from sympa as list moderator, sorry for that. Find attached the eml file from /var/spool/sympa/moderation. Curiously the email seems to circle around dfn.de systems a lot, I will in parallel further investigate that with our email system admins.

moderated.eml.zip

[racke]

My educated guess would be that Sympa is not the culprit here.

[ikedas]

I agree: At least Sophos PureMessage alters message body, but CommuniGate Pro and AMaViSd-new look not (It's unsure about Sympa by now).

What I feel curious is that, as I wrote before, alterations look occurred twice.
Anyway, you'd be better to investigate PureMessage where the message is probably broken at least once.

[balert]

yeah, thanks a lot for all your support, feel free to close this issue.

[balert]

I did some further investigation:
one of the list members is my private mail account, also in this case the email body is broken, but in this case PMX is not in the chain, so we can exclude PMX from the list of possible candidates for breaking the body. Only the changing Content-Transfer-Encoding from 8bit to quoted-printable is possibly PMX's fault, but not the whole deal.

My private account still receives the email with Content-Transfer-Encoding 8bit unchanged from the original, but still the body is broken. What I also noticed is that the email from the moderation folder does not yet contain the footer. So the footer is added afterwards, and also changes the email to multi-part mime. Afaik the email was not multi-part mime before addind the footer. So I guess it could be possible that this step messes things up. What do you guys think?

I've just had a lengthy phone call with our email administrator, and we came to the conclusion, that it is highly unlikely that our systems cause this. Still I will also get in contact with our external whatsoever (entries with dfn.de in the headers) and ask them what ideas they might have.

tl;dr: unfortunately I'm not (anymore) convinced that sympa is completely innocent here, even though definitely not responsible for all of the curious things going on.

[ikedas]

I don't think Sympa is exactly innocent, however also am not sure that PureMessage will never alter the message. Because we don't have evidence, either negative or positive.

Below is the picture of message delivery. Can you get the message just before and after it goes through PureMessage?

---

• 192.168.178.123 (sender's client machine)

  Message is sent by the user.

• tu-braunschweig.de (CommuniGate Pro SMTP 6.2.15)

  • DKIM signing

    DKIM-Signature added

• b1991.mx.srv.dfn.de (Postfix)

  • 127.0.0.1 (amavisd-new, port 20114)

    • AMaViSd-new

      Authentication-Results & X-DFN-Virus-Scanned added

  • b1991.mx.srv.dfn.de (Postfix)

• sympa.rz.tu-bs.de (Postfix)

  Delivered-To & X-Original-To added

  • Sympa

    --- held by moderation --- ==> Moderation notice including original message (held-back-for-moderation.eml in moderation.zip --- Altered once) is sent.

    Message is held in moderation spool ([email protected]_eacbab1c2f443f96047fc0f92d245132 in moderated.eml.zip: --- Not altered)

    --- approved to resume processing ---

    X-Validation-by to X-Original-DMARC-Record appended

    Footer appended --- It's unsure if original body was altered.

  • sympa.rz.tu-bs.de (Postfix, from userid 1000)

• a1991.mx.srv.dfn.de (Postfix)

  • 127.0.0.1 (amavisd-new, port 20114)

    • AMaViSd-new

      Authentication-Results & X-DFN-Virus-Scanned added

  • a1991.mx.srv.dfn.de (Postfix)

• pmx3.rz.tu-bs.de (Postfix)

  • localhost (Postfix)

    Delivered-To added

    • Sophos PureMessage

      X-PMX-Version & X-PMX-Spam appended

      Probably body altered

      Mofidied Content-Transfer-Encoding appended

  • pmx3.rz.tu-bs.de (Postfix)

• tu-braunschweig.de (CommuniGate Pro SMTP 6.2.15)

  • DKIM signing

    DKIM-Signature added

• Disposed by the user. (delivered-to-user.eml in moderation.zip --- Altered twice)

[balert]

look at the attached example, no PMX involved, but still broken body. This is delivered to my private mailbox, not business, therefore PMX is not involved here. It's different, so PMX might not be innocent, but still without PMX not everything is fine, therefore lets concentrate on whats left over first. We will investigate PMX in parallel, this is not your responsibility :-)

[list-name] test äöüÄÖÜß.zip

[ikedas]

Can you set the list moderated and get the message in moderation spool?

And, can you get the message just before it is sent to SpamAssassin?

[ikedas]

If the message will prove the brakage of message is due to Sympa, please do these:

1. Update system:

   # yum update --exclude='sympa*' 

2. Check if the problem won't be solved.
3. If it won't, please show the list of updated packages:

    # yum deplist sympa

   I'll check it and prepare a test environment.

[balert]

I have already provided an example from the moderation spool. My sending client for all the tests and the test message content was always the same, so I hardly believe there will be a difference if I create another example here, no?

Regarding SpamAssassin, are you supposing I reconfigure my private mail server to circumvent SpamAssassin? No idea how I should catch the mail before SpamAssassin, and touching my mailserver is off the table, running well for too long :-) But I could add the email address of your choice (so the email provider of your choice as well) to the test list and send another test?

[ikedas]

But I could add the email address of your choice (so the email provider of your choice as well) to the test list and send another test?

OK. Add my email.

[balert]

done

[ikedas]

OK. I confirmed conversion 1 and 2 I described were done by your Sympa. The message was not altered as 3.

Could you please check another instruction by me?

[balert]

I did, didn't solve the issue though. Attached is deplist.

deplist.zip

[ikedas]

I couldn't reproduce on my side. See attached result.

issue-1113_ikedas_01.zip

[balert]

ok, I've created another example. I've sent the email directly to our sympa host, then I've added my googlemail address to the list, so in the example attached you can clearly see that sympa is the only host under our control involved and still the email content is broken. I count this as a proof that it is Sympa's fault.

google.zip

[ikedas]

I see. I confirmed that the modification 3 described in my comment is not due to Sympa.
However, is this really sent by Thunderbird? I think it is unlikely that it sends the message with unencoded header (subject).

BTW, could you please show the following configuration inbformation related to the list in question? I'll try reproducing the problem using them.

• The content of list-name directory under $EXPLDIR (e.g. /var/lib/sympa/list_data)
• The all entries related to list-name list in aliases file (location is configured by sendmail_aliases parameter)
• The all configuration files of Postfix related to Sympa (main.cf, master.cf and related map files)

EDIT: I forgot:

• sympa.conf

If you have problem disclosing those information, please send them to my address.

[balert]

I'm running 78.7.1 (64-Bit) on Archlinux, this is what I've used to send the email. I'll send you the requested information via email. Thanks for looking into it.

[ikedas]

I couldn't reproduce the problem using your configurations.

@balert , If possible, I want to confirm one last time:

1. Copy a shell script attached in below into your mailing list server as appropriate name, e.g. sendmail-tee.
2. Make it executable: Run "chmod 755 /path/to/sendmail-tee".
3. Add this line to sympa.conf:
   sendmail /path/to/sendmail-tee
4. Restart sympa service (and wwsympa service).

By this, Sympa will append verbatim copy of any messages sent by Sympa itself to the file sendmail-output . Post the test message and check content of the file.

sendmail-tee:

#! /bin/sh

# Change this!
OUTPUT=/var/tmp/sendmail-output

args=
for arg; do
    case "$arg" in
    *\'*)
        arg=`cat <<EOF | sed -e "s/'/'\\''/g"`
$arg
EOF
        ;;
    *)
        ;;
    esac
    args="$args '$arg'"
done

echo ---- >> $OUTPUT
LANG=C date >> $OUTPUT
echo sendmail $args >> $OUTPUT
echo ---- >> $OUTPUT
tee -i -a $OUTPUT | eval "/usr/sbin/sendmail $args"
rc=$?
echo '' >> $OUTPUT
exit $rc

[balert]

Sorry, that took me a while due to other tasks. Find attached the sendmail-output.
sendmail-tee.txt

[ikedas]

Hi @balert,
Could please show us a complete set of reproducible data? It doesn't matter if it takes a long time.

The process:

1. Make the list in question moderated (set send parameter to editorkey).
2. Send the original message using mailer (it will be held for moderation). *
3. Check the content of moderation spool (/var/spool/sympa/moderation). *
4. Approve the held message (it will be delivered). *
5. Check how the delivered message is displayed by your mailer. *
6. Get the raw message you received. *

The results:

• A copy of the message you sent at 2. It would be saved in "Sent" folder of your mailer.
• A file held in the moderation spool at 3.
• A copy of delivered message at 4. It would be saved by sendmail-tee.
• A screen shot of your mailer displaying the message you received at 5.
• A message exported from your mailer at 6.

Note: Don't copy the message by copying & pasting its content: Inappropriate setting of console might break the texts. Instead, copy the file itself.

A set of five files above is the result of one test case. If possible, could you please try also the other cases such as below?

• Use the other mailer to send original message.
• Enable or disable setting for signature by the mailer (the text appended to body by mailer).
• Use the other mailer to view delivered message.

and by each case, could you please get a set of files above?

[balert]

i've consolidated a complete set of files for you. It appears that the corruption happens between the moderation spool and sendmail. What can I do to solve this? Maybe I've just misconfigured something?
complete.zip

[ikedas]

I wrote:

A set of five files above is the result of one test case. If possible, could you please try also the other cases such as below?

• Use the other mailer to send original message.
• Enable or disable setting for signature by the mailer (the text appended to body by mailer).
• Use the other mailer to view delivered message.

and by each case, could you please get a set of files above?

Additionally, by each case, could you please show us where and how you changed it?

[balert]

I've tested Evolution, Thunderbird and hPronto (CGP) to send the test message. Then I opened each mail with each of those three clients and every combination of sender and receiver application showed corrupted characters.

According to my already provided examples the problem is clearly caused by Sympa. My users are freaking out already, since they suffer from the problem for several months already.

Have you had the chance to look at the Sympa code to find out what might cause this? Could you point out thing I could check on the Linux servers running Sympa? Maybe we've got a locales problem here?

[ikedas]

I and @racke have not yet been able to reproduce the behavior you reported. So I guess anything depending on your environment affects Sympa to make messages broken.

Could you please check the other cases, and additionally, the other cases such as configuration with another option (e.g. changing footer_type, removing message_footer, ...)?

[ikedas]

@balert ,
Could you please apply this patch and check if the behavior will change?

[balert]

thanks @ikedas. I tried this, but the patch did not help. However, I found out if I disable DMARC (dmarc_protection: mode=none) then it works! So the bug must be somewhere in the DMARC related procedures. Maybe this helps you further track down the bug.

[ikedas]

Did you restart Sympa services?
Exactly, I found the logic of DMARC broke the messages (footer itself doesn’t matter). If possible, could you please check if the patch will fix the problem?

[balert]

Yes, that was it, I hadn't restarted Sympa, my bad. Now it works, thank you so much for your help. Great work!

[racke]

Terrific good work, both of you!! ❤️