Exception java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.5-1: Read-only file system [in thread "default-nioEventLoopGroup-1-4"]

[Dapengsechs]

When I install the version 0.25.0 and 0.25.1 of akhq using helm, I alway got this error, and akhq will not start. Following is the complete log after installing.

grep: /app/config/jvm/akhq_jvmoptions.yaml: No such file or directory
2024-07-25 13:31:30,400 INFO  main         Context$RuntimeConfiguredEnvironment Established active environments: [k8s, cloud]
2024-07-25 13:31:34,294 WARN  main         o.a.c.security.JwtSecurityWarning 
2024-07-25 13:31:34,294 WARN  main         o.a.c.security.JwtSecurityWarning ##############################################################
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning #                      SECURITY WARNING                      #
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning ##############################################################
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning 
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning You have set a security group config but did not set the jwt secret.
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning This means that the API request will not be checked against the security group config.
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning Please set the 'micronaut.security.token.jwt.signatures.secret.generator.secret' configuration, or ask your administrator to do it !
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning 
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning ##############################################################
2024-07-25 13:31:34,295 WARN  main         o.a.c.security.JwtSecurityWarning 
2024-07-25 13:31:38,318 INFO  main         io.micronaut.runtime.Micronaut Startup completed in 12207ms. Server Running: http://akhq-557dcb9f89-4wrgq:8080
2024-07-25 13:31:44,010 WARN  default-nioEventLoopGroup-1-4 io.netty.channel.ChannelInitializer Failed to initialize a channel. Closing: [id: 0x164ee301, L:/10.19.5.33:28081 - R:/10.19.5.24:60114]
java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.5-1: Read-only file system
 at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
 at java.base/java.io.File.createTempFile(Unknown Source)
 at com.github.luben.zstd.util.Native.load(Native.java:129)
 at com.github.luben.zstd.util.Native.load(Native.java:85)
 at com.github.luben.zstd.Zstd.<clinit>(Zstd.java:13)
 at io.netty.handler.codec.compression.ZstdConstants.<clinit>(ZstdConstants.java:25)
 at io.netty.handler.codec.compression.ZstdOptions.<clinit>(ZstdOptions.java:41)
 at io.netty.handler.codec.compression.StandardCompressionOptions.zstd(StandardCompressionOptions.java:54)
 at io.micronaut.http.server.netty.handler.Compressor.<init>(Compressor.java:62)
 at io.micronaut.http.server.netty.handler.PipeliningServerHandler.setCompressionStrategy(PipeliningServerHandler.java:142)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertMicronautHandlers(HttpPipelineBuilder.java:622)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertHttp1DownstreamHandlers(HttpPipelineBuilder.java:638)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.configureForHttp1(HttpPipelineBuilder.java:380)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.initChannel(HttpPipelineBuilder.java:299)
 at io.micronaut.http.server.netty.NettyHttpServer$Listener.initChannel(NettyHttpServer.java:892)
 at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
 at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
 at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130)
 at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
 at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
 at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
 at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
 at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
 at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
 at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
 at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
 at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.base/java.lang.Thread.run(Unknown Source)
2024-07-25 13:31:44,010 WARN  default-nioEventLoopGroup-1-3 io.netty.channel.ChannelInitializer Failed to initialize a channel. Closing: [id: 0x6c67a861, L:/10.19.5.33:28081 - R:/10.19.5.24:60100]
java.lang.NoClassDefFoundError: Could not initialize class io.netty.handler.codec.compression.ZstdOptions
 at io.netty.handler.codec.compression.StandardCompressionOptions.zstd(StandardCompressionOptions.java:54)
 at io.micronaut.http.server.netty.handler.Compressor.<init>(Compressor.java:62)
 at io.micronaut.http.server.netty.handler.PipeliningServerHandler.setCompressionStrategy(PipeliningServerHandler.java:142)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertMicronautHandlers(HttpPipelineBuilder.java:622)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertHttp1DownstreamHandlers(HttpPipelineBuilder.java:638)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.configureForHttp1(HttpPipelineBuilder.java:380)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.initChannel(HttpPipelineBuilder.java:299)
 at io.micronaut.http.server.netty.NettyHttpServer$Listener.initChannel(NettyHttpServer.java:892)
 at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
 at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
 at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130)
 at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
 at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
 at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
 at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
 at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
 at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
 at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
 at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
 at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.5-1: Read-only file system [in thread "default-nioEventLoopGroup-1-4"]
 at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
 at java.base/java.io.File.createTempFile(Unknown Source)
 at com.github.luben.zstd.util.Native.load(Native.java:129)
 at com.github.luben.zstd.util.Native.load(Native.java:85)
 at com.github.luben.zstd.Zstd.<clinit>(Zstd.java:13)
 at io.netty.handler.codec.compression.ZstdConstants.<clinit>(ZstdConstants.java:25)
 at io.netty.handler.codec.compression.ZstdOptions.<clinit>(ZstdOptions.java:41)
 ... 27 common frames omitted
2024-07-25 13:31:53,795 WARN  default-nioEventLoopGroup-1-3 io.netty.channel.ChannelInitializer Failed to initialize a channel. Closing: [id: 0x3ab83817, L:/10.19.5.33:28081 - R:/10.19.5.24:33772]
java.lang.NoClassDefFoundError: Could not initialize class io.netty.handler.codec.compression.ZstdOptions
 at io.netty.handler.codec.compression.StandardCompressionOptions.zstd(StandardCompressionOptions.java:54)
 at io.micronaut.http.server.netty.handler.Compressor.<init>(Compressor.java:62)
 at io.micronaut.http.server.netty.handler.PipeliningServerHandler.setCompressionStrategy(PipeliningServerHandler.java:142)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertMicronautHandlers(HttpPipelineBuilder.java:622)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$StreamPipeline.insertHttp1DownstreamHandlers(HttpPipelineBuilder.java:638)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.configureForHttp1(HttpPipelineBuilder.java:380)
 at io.micronaut.http.server.netty.HttpPipelineBuilder$ConnectionPipeline.initChannel(HttpPipelineBuilder.java:299)
 at io.micronaut.http.server.netty.NettyHttpServer$Listener.initChannel(NettyHttpServer.java:892)
 at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
 at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) 

[Dapengsechs]

Using 0.24.0 everything works fine, no such error.

[AlexisSouquiere]

The error you have seems to be linked to a readonly filesystem in the container that is preventing Micronaut to start. Micronaut needs to unpack the ZSTD library and a RO FS prevents it.
If you don't have it in 0.24.0 it's probably because this requirement was introduced recently in Micronaut.

Don't you have a readOnlyRootFilesystem in your Helm values ?

[djanm]

I'm facing the same problem. Our k8s cluster is running with a readOnlyRootFilesystem policy only

[AlexisSouquiere]

Then you should probably try to override the JVM options (https://akhq.io/docs/configuration/docker.html#run-with-another-jvm-options-file) and add the java.io.tmpdir property to put a folder having read write permissions. Right now AKHQ will use the default value which /tmp