Bounce processing via POP3 is not working

[citrus-it]

Version:

• listmonk: 2.0.0
• OS: OmniOS

Description of the bug and steps to reproduce:

I've set up a listmonk server on OmniOS and I am testing bounce processing by sending a campaign to a small list with a couple of bad addresses in there. The bounces end up in the POP3 box as expected but listmonk does not handle them.

I have two bounces messages, generated by different software.

The first is from Sendmail and looks like the following. The campaign and subscriber UUIDs are present but they are not in the message headers. They are in the text/rfc822-headers MIME part of the bounce. Since listmonk only retrieves the headers of the bounce, it does not see these.

Return-Path: <>
Received: from mail.example.com (mail.example.com [10.1.2.101])
         by mailstore.example.com with LMTPA;
         Wed, 29 Sep 2021 08:20:33 +0000
Date: Wed, 29 Sep 2021 08:20:34 GMT
From: Mail Delivery Subsystem <[email protected]>
Message-Id: <[email protected]>
To: [email protected]
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="18T8KYXY004619.1632903634/mail.example.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

This is a MIME-encapsulated message

--18T8KYXY004619.1632903634/mail.example.com

The original message was received at Wed, 29 Sep 2021 08:20:33 GMT
from mail.example.com [10.1.2.101]

* Sendmail is experiencing difficulty in delivering your message.

   ----- The following addresses had permanent fatal errors -----
<[email protected]>
    (reason: 550 Host unknown)

   ----- Transcript of session follows -----
550 5.1.2 <[email protected]>... Host unknown (Name server: asdfijasdeidd.ch.
: host not found)

--18T8KYXY004619.1632903634/mail.example.com
Content-Type: message/delivery-status

Reporting-MTA: dns; mail.example.com
Received-From-MTA: DNS; mail.example.com
Arrival-Date: Wed, 29 Sep 2021 08:20:33 GMT

Final-Recipient: RFC822; [email protected]
Action: failed
Status: 5.1.2
Remote-MTA: DNS; asdfijasdeidd.ch
Diagnostic-Code: SMTP; 550 Host unknown
Last-Attempt-Date: Wed, 29 Sep 2021 08:20:34 GMT

--18T8KYXY004619.1632903634/mail.example.com
Content-Type: text/rfc822-headers

Return-Path: <[email protected]>
Received: from mail.example.com (mail.example.com [10.1.2.101])
        by mail.example.com with ESMTP id 18T8KXXY004617
        for <[email protected]>; Wed, 29 Sep 2021 08:20:33 GMT
X-Listmonk-Subscriber: 73d11b6b-3da2-47d5-9dd6-7b3c96a8e15b
Message-Id: <1632903633748563659.927.5595426720521411472@examplece>
Mime-Version: 1.0
From: "OmniOS News" <[email protected]>
Date: Wed, 29 Sep 2021 08:20:33 +0000
X-Listmonk-Campaign: 730438ad-a86a-424d-8604-288f6f6f1831
List-Unsubscribe-Post: List-Unsubscribe=One-Click

--18T8KYXY004619.1632903634/mail.example.com--

Here's another example generated by zimbra. In this case, the UUIDs that listmonk needs are in a message/rfc822 section contained within the bounce message. I'm just showing this section.

--A040461442.1632903634/zimbra.example.com
Content-Description: Undelivered Message
Content-Type: message/rfc822

Return-Path: <[email protected]>
X-Listmonk-Campaign: 730438ad-a86a-424d-8604-288f6f6f1831
Content-Type: multipart/alternative;
 boundary=c65d1d1684fcb7f61e6faf1a7aa51aa246d8948ed2bb03eb27d7dbc95235
From: "OmniOS News" <[email protected]>
Date: Wed, 29 Sep 2021 08:20:33 +0000
Mime-Version: 1.0
List-Unsubscribe-Post: List-Unsubscribe=One-Click
To: <[email protected]>
Subject: Hello Bouncy
Message-Id: <1632903633346478703.927.424505366807004780@omniosce>
X-Listmonk-Subscriber: 7c3e7431-f035-45de-8f35-06854108b74f

It seems that listmonk needs to retrieve the entire message, and not just the headers, and then look through for a MIME part matching message/rfc822 or text/rfc822-headers (there may be others) then parse that into a message before extracting the UUID headers.

[knadh]

Thanks for describing this well. Will work on a fix.

[Apreche]

POP3 bounce mailbox is not working for me either, but I have a different error. This is all that shows up in the logs.

2021/10/11 02:58:03error scanning bounce mailbox: unknown response. Neither -ERR, nor +OK.

For reference, the POP3 mailbox is on a GMail account. I confirmed the POP3 settings were correct and working by testing them using a different stand alone email client, so that is not the issue.

[jackdanielux]

Not sure, if I should open 1 more issue for this - Pls advise.

If a user enters an incorrect email ID while subscribing, should that as well now automatically get captured in bounce? Because technically, if I am adding users in double opt in - then the user post verification will hardly bounce.

But If a user is added to non double opt in list, then the best practice might be to outright block the email from retriggering an email.

[knadh]

If a user enters an incorrect email ID while subscribing, should that as well now automatically get captured in bounce?

There is no foolproof way to confirm the validity of an e-mail address without actually sending (and receiving) an e-mail.

[jackdanielux]

@knadh No, I wasnt saying that it should auto identify if the entered email is valid, I was talking exactly what you mentioned - that is we actually send the email and if the first email itself bounced (if it's a double opt in) then that should be captured in bounce. I don't think it's happening at the moment.

I tried with an incorrect email ID in double opt in list, and it allowed me to enter it as many times as I wanted (post deleting from the list as many times).

Ideal scenario could have been, I entered it once, it bounced and got added into the bounce list and 2nd time if that email is added to any list, it could throw an error.

[knadh]

if the first email itself bounced (if it's a double opt in) then that should be captured in bounce

Ah yes, this makes sense. Will explore this.

[cantoute]

Server side I get

Nov  7 00:50:51 orion dovecot: pop3-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<5MMFbifQuqZ/AAAB>

Seems user name is not passed.

[cantoute]

Could it be capitalisation ?

listmonk/internal/bounce/mailbox/pop.go

Line 42 in 0bd13fe

if err := c.Auth(p.opt.Username, p.opt.Password); err != nil {

		if err := c.Auth(p.opt.username, p.opt.password); err != nil {

[knadh]

Could it be capitalisation ?

Don't think so. "none" is an enum that's passed from the UI.

I'm unable to reproduce his locally (testing with the InBucket server).

[cantoute]

Could it be capitalisation ?

Don't think so. "none" is an enum that's passed from the UI.

I don't see anything like 'none' in this line

opt.Username == opt.username ?

[knadh]

opt.Username == opt.username ?

Ah, definitely not this. Variable names are case-sensitive. If they're incorrect, the app won't compile. That's why I assumed you were referring to the line just above with "none".