Suspected deadlock in Loki integration

[bsod90]

Describe the bug

I'm trying to configure Falco <> Loki integration, but it doesn't seem to work. DEBUG logs indicate that Loki client has been configured, but then it seems to be just hanging when attempting to send Loki payload.
I have to specify tenant to connect to my Loki instance, and I suspect that there's a mistake in the code that adds the X-Org-ID handler here:

falcosidekick/outputs/loki.go

Line 71 in 63d23f9

defer c.httpClientLock.Unlock()

I'm not a Go expert, but won't this cause a deadlock because of the other lock acquisition a few lines below?

Also, the custom headers block doesn't seem to be locking the client at all.
I wish I could just test it locally and contribute a fix, but I don't see a quick way to go through all the steps of building the binary, then docker, then Helm to finally get it onto our infra.

I'd be super grateful if someone with an already configured dev-environment could help. Thank you!

How to reproduce it

Configure falcosidekick (via Helm chart) like this:

falcosidekick:
  enabled: true
  webui:
    enabled: false
  config:
    debug: true
    slack:
      minimumpriority: "notice"
      webhookurl: https://hooks.slack.com/services/*********
    loki:
      hostport: "https://loki.monitoring.****"
      user: loki
      apikey: ******
      endpoint: /loki/api/v1/push
      tenant: falco
      minimumpriority: info

Expected behaviour

Receive Falco alerts both in Slack and Loki

Actual behaviour

Only Slack alerts are being delivered

Screenshots

Environment

Falcosidekick 2.28.0 installed via Helm on GKE

[Issif]

Thank for your issue, I'll take a look at the root cause and your fix.

For the test env, no need a k8s cluster, go can easily run the binary locally and use a container aside for Loki.

[bsod90]

Yeah, but I don't even have Falco running locally to generate alerts :)
Anyway, I was trying to prove my theory by manually supplying X-Scope-OrgID via customHeaders instead of the tenant (custom headers part just doesn't lock the client for the second time). However, it seems like that part doesn't work either. I suspect it's because Loki.CustomHeaders has to be mentioned here

falcosidekick/config.go

Line 498 in 63d23f9

v.GetStringMapString("Webhook.CustomHeaders")

, but I'm too unfamiliar with the codebase and could easily be wrong.

Thanks again for helping with this, @Issif ! 🤗

[Issif]

I will check. For generating test events, you can call falcosidekick, see itself https://github.com/falcosecurity/falcosidekick#quicktest

[bsod90]

Ok, I was able to verify that the fix indeed works