Memory leak when Archive Download fails

[dpc22]

Version

6.2.64

Installation method

My own rpm, derived from official source RPM. Redhat Enterprise Server 7.

Expected behavior

wwsympa.pl should not leak 4 GBytes of memory when an archive download fails.

Actual behavior

(Output from Linux top command):

16817 sympa     20   0 4916564   4.2g   6612 S   0.0 26.8   4:30.05 wwsympa.fcgi                                                      

4.2g is 4.2 Gytes of Residient Set Size. That starts as about 200 MBytes.

Additional Information

One of my list owners attempted to download a 1.3 GBytes ZIP archive of their mail archives (about 3 to 4 GBytes of raw data).

Apache timed out after (I think) 60 seconds: it is possible I need to increase the Apache default timeout. That left the wwsympa.fcgi process in question using 4.2 GBytes RSS, which does not appear to be returned to the system.

The list owner then repeated the process multiple times until all of the wwsympa processes were 4.2 GBytes in size.

I think that there might also be slow memory leaks in other Sympa daemons, but this largely explains why a virtual machine with 16 GBytes RAM was swapping like mad this morning until I restarted wwsympa. It normally only uses about 6 GBytes, with the rest spare/as buffer cache. OOM killer had stepped in, which is never a good sign on Linux.

I don't think that this memory leak happens if the download is successful. However I think that I will disable list archive downloads for the time being...

[dpc22]

Possibly worth adding: the Linux "zip" command can build these archives much faster than the Perl modules Sympa is using.

[ikedas]

@dpc22 , can you check this patch?
This fix requires a new module Archive-Zip-SimpleZip.

[dpc22]

The patch applies, but Archive-Zip-SimpleZip isn't available from the RHEL base or EPEL repositories, so I will need to install it using CPAN or such like. All of the other Perl dependencies for Sympa are available from EPEL. I'm also a bit dubious about : "Module version: 0.040", although I guess version numbers are fairly arbitrary.

wwsympa is creating a temporary .zip file in /var/spool/sympa/tmp. Is there a particular reason that we can't use /usr/bin/zip to generate that file if it is available? I would expect the command line tool to be much more readily available than Archive-Zip-SimpleZip.

[racke]

For example, zip is not present on a basic Debian installation.

[dpc22]

Presumably you can install it using "apt get zip" though?

I don't think that Archive-Zip-SimpleZip is available on Debian/Ubuntu either unless it is called something other than simplezip

[racke]

Yes, you can install it of course. Just saying that we can't assume that zip is installed.

[ikedas]

@dpc22, for the presnt, you have to install it manually, e.g. using cpanm (see the description).

I think the problem is caused by the fact that Archive::Zip keeps the entire generated archive in memory. And Archive::Zip::SimpleZip seems to work around this problem.

If Archive::Zip::SimpleZip would solve the problem, we will add it to dependency (see cpanfile file), and the packagers working for each distribution (Debian, RHEL/EPEL, FreeBSD, ...) will request thier community to include it into their distribution.

[dpc22]

I seem to have ended up in Perl dependency hell:

lists-2[dpc22:Archive-Zip-SimpleZip-0.040]$ perl Makefile.PL 
Warning: prerequisite IO::Compress::Base 2.096 not found. We have 2.061.
Warning: prerequisite IO::Compress::Bzip2 2.096 not found. We have 2.061.
Warning: prerequisite IO::Uncompress::Base 2.096 not found. We have 2.061.

lists-2[dpc22:Archive-Zip-SimpleZip-0.021]$ perl Makefile.PL 
Warning: prerequisite IO::Compress::Base 2.081 not found. We have 2.061.
Writing Makefile for Archive::Zip::SimpleZip

RHEL7 currently provides perl-IO-Compress-2.061
RHEL8 currently provides perl-IO-Compress-2.081

I will set up a throwaway VM to test the patch, but I don't want to fight the Redhat package manager on my production servers, which are carefully managed using Ansible. Archive-Zip-SimpleZip-0.021 looks plausible when we upgrade from RHEL 7 to RHEL 8 (probably next summer). In the mean time noone has actually complained that I have disabled archive downloads.

[dpc22]

Okay, I can confirm that the patch plus Archive-Zip-SimpleZip definitely fixes the issue that I reported. It takes about 25 seconds for Sympa to generate and a Web browser to download:

-rw-rw-r-- 1 dpc22 dpc22 301M Oct  4 12:57  test-dpc22_archives.zip

compared to about 20 seconds for the command line "zip" command to generate the raw archive, so not shabby at all. wwsympa.pl doesn't appear to have leaked substantial amounts of memory in the process. If I remove Archive::Zip::SimpleZip, the same archive download times out and leaks lots of memory just as before, but it has transparently fallen back to Archive::Zip.

In order to install Archive::Zip::SimpleZip, cpanm needed to upgrade or install the following on RHEL7. Not as bad as I had feared, particularly given that cpanm seems to have installed its packages to /usr/local/share/perl5/ rather than /usr/share/perl5, which makes it simpler to revert.

[root@lists-2 ~]# cpanm install Archive::Zip::SimpleZip
--> Working on install
Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/install-0.01.tar.gz ... OK
Configuring install-0.01 ... OK
Building and testing install-0.01 ... OK
Successfully installed install-0.01
--> Working on Archive::Zip::SimpleZip
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/Archive-Zip-SimpleZip-0.040.tar.gz ... OK
Configuring Archive-Zip-SimpleZip-0.040 ... OK
==> Found dependencies: IO::Uncompress::Base, IO::Compress::Base, IO::Compress::Bzip2
--> Working on IO::Uncompress::Base
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/IO-Compress-2.102.tar.gz ... OK
Configuring IO-Compress-2.102 ... OK
==> Found dependencies: Compress::Raw::Bzip2, Compress::Raw::Zlib
--> Working on Compress::Raw::Bzip2
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/Compress-Raw-Bzip2-2.101.tar.gz ... OK
Configuring Compress-Raw-Bzip2-2.101 ... OK
Building and testing Compress-Raw-Bzip2-2.101 ... OK
Successfully installed Compress-Raw-Bzip2-2.101 (upgraded from 2.061)
--> Working on Compress::Raw::Zlib
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/Compress-Raw-Zlib-2.101.tar.gz ... OK
Configuring Compress-Raw-Zlib-2.101 ... OK
Building and testing Compress-Raw-Zlib-2.101 ... OK
Successfully installed Compress-Raw-Zlib-2.101 (upgraded from 2.061)
Building and testing IO-Compress-2.102 ... OK
Successfully installed IO-Compress-2.102 (upgraded from 2.061)
Building and testing Archive-Zip-SimpleZip-0.040 ... OK
Successfully installed Archive-Zip-SimpleZip-0.040
5 distributions installed

[dpc22]

Now deployed on the live server. I will let you know if there turn out to be any complications. Thanks.

[ikedas]

@dpc22, thanks for confirming!

This PR will be merged for now, in some days. However if you notice anything, please let us know.