7 days Long running test failed on 4.3.0 server. (OutOfMemoryError: Java heap space )

[MohitMaliFtechiz]

##Describe.
7 days Long running test failed on 4.3.0 server.

Server breaks after running 2 to 3 days. Able to replicate it on 8 gb with load and 4gb without load system.

logs

11/Aug/2021:11:51:57 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 11 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Asynchronous Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Asynchronous Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space ()

Config

Gluu-version 4.3.0
OS:- Ubuntu 20.04

[devrimyatar]

@nynymike @yuriyz @yurem I need guide on how to this problem.

[MohitMaliFtechiz]

hi @mbaser ,

I have placed 4 gb system with 8k user enteries with 4k user login, the system crashes within 2 hours.

[MohitMaliFtechiz]

hi @nynymike ,

I turn the audit logs but after turning it off i am not seeing anylogs on opendj logs.

this is how i turn it off.

go to /opt/opendj/bin
run dsconfig 
Specify OpenDJ LDAP connection parameters
select log publisher
select view and edit an existing  log publishers
select File-Based Audit Logger
select enabled to false

[yuriyz]

@MohitMaliFtechiz does it crash with disabled audit logger ? From message in ticket description it looks like some big data chunk goes to logger which it was not able to process.

[yuriyz]

@MohitMaliFtechiz To circle it back, lets do following :

1. enable logging back
2. add option to create dump file on OOM (e.g. -Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/home/user/dump")
3. Send dump file to me.