AKHQ UI is not able to display schemas

[ritambanik]

Hello,

We are running AKHQ in k8s. Our Confluent Schema Registry is also running in k8s with basic auth enabled. As per the AKHQ logs, it's able to pull the schemas from the Confluent Schema Registry but not able to display in the UI. Below is the AKHQ config and also attached a screenshot of the chrome developer network view when 'Schema Registry' is clicked on the UI. Can you please help me understand what could be the issue?

Log Snippet

{"sequenceNumber":0,"timestamp":1723127499851,"nanoseconds":851609599,"level":"DEBUG","threadName":"io-executor-thread-29","loggerName":"io.confluent.kafka.schemaregistry.client.rest.RestService","context":{"name":"default","birthdate":1723049832133,"properties":{}},"mdc": {},"message":"Sending GET with input null to https://uk-schema-registry-np-wales.yyy.cloud/config/m_27460_xxx_any_uk_xxxio_logging-event-value?defaultToGlobal=true","throwable":null}

{"sequenceNumber":0,"timestamp":1723127502079,"nanoseconds":79244492,"level":"DEBUG","threadName":"default-nioEventLoopGroup-1-23","loggerName":"io.micronaut.context.event.ApplicationEventPublisher","context":{"name":"default","birthdate":1723049832133,"properties":{}},"mdc": {},"message":"Publishing event: {}","arguments": ["io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]"],"throwable":null} {"sequenceNumber":0,"timestamp":1723127502355,"nanoseconds":355522398,"level":"DEBUG","threadName":"default-nioEventLoopGroup-1-3","loggerName":"io.micronaut.http.server.netty.NettyRequestLifecycle","context":{"name":"default","birthdate":1723049832133,"properties":{}},"mdc": {},"message":"Request {} {}","arguments": ["GET","/api/uk-kafka-wales-sandbox/schema?&search=&page=1"],"throwable":null} {"sequenceNumber":0,"timestamp":1723127502355,"nanoseconds":355796391,"level":"DEBUG","threadName":"default-nioEventLoopGroup-1-3","loggerName":"io.micronaut.security.token.reader.HttpHeaderTokenReader","context":{"name":"default","birthdate":1723049832133,"properties":{}},"mdc": {},"message":"Looking for bearer token in {} header","arguments": ["Authorization"],"throwable":null}

Thanks!

[AlexisSouquiere]

For me there is nothing indicating that AKHQ was able to get the schemas from the registry. We can see in your logs that it sent a request to the registry but I'm not sure it means that it had an answer. Your screenshot shows an HTTP 504 - Gateway Timeout so I would think about a communication issue.
Can you open a shell into the AKHQ container and do a curl to the registry to see if you get an answer ?

[ritambanik]

Yes. The connectivity is fine. Attaching the logs from the k8s and local docker-compose. In k8s, the same subject is fetched thrice whereas in local, it's exactly once. Also the below log is missing in k8s.

{"sequenceNumber":0,"timestamp":1723216850064,"nanoseconds":64515762,"level":"INFO","threadName":"io-executor-thread-6","loggerName":"org.akhq.log.access","context":{"name":"default","birthdate":1723216726064,"properties":{}},"mdc": {},"message":"[Date: 2024-08-09T15:19:20.290853762Z] [Duration: 89774 ms] [Url: GET /api/docker-kafka-server/schema] [Status: 200] [Ip: /192.168.65.1] [User: dtxxxxxxx]","throwable":null}

[
local-docker-compose.log
k8s.log
]

[AlexisSouquiere]

I think you are reaching a timeout in K8S. I can see Duration: 89774 ms in your previous message from your local AKHQ. If it's the same in K8S you are probably reaching a timeout coming from the ingress controller.
Is it slow to slow the schema locally ?

[ritambanik]

What is meant by 'slow to slow the schema locally'? The local container running AKHQ is hitting the same ingress for the Schema Registry as the pod running AKHQ in k8s.

[AlexisSouquiere]

Sorry, slow to show the schema on your local AKHQ. According to your log it takes 1min 30s (89774ms) to show the schemas on your local AKHQ. When you run AKHQ in K8S, you are getting a HTTP 504 which means Gateway timeout issue. So for me there is clearly a link between the time the registry takes to answer and the HTTP 504 error. The issue is not on AKHQ itself

[ritambanik]

Thanks for pointing to the root cause. I'll investigate the latency. Since the timeout is occuring after 3 min, is there a config in AKHQ to increase the timeout?