400 Headers Missing When Firing Multiple Alerts via HTTP Post

[bc-sb]

Describe the bug
When testing the falco/spyderbat integration we were seeing 400's header missing

How to reproduce it
setup a simple http service with a nonzero delay and a required header and point the webhook integration at it and fire 2 test events

Expected behaviour
200 responses for both request

Screenshots

2022/10/25 15:51:50 [INFO]  : Webhook - Post OK (200)
2022/10/25 15:51:51 [ERROR] : Webhook - header missing (400): {"message":"Failed to get expected header"}
2022/10/25 15:51:51 [ERROR] : WebHook - header missing

Environment
Saw first in aws hosted kubernetes, but able to reproduce locally

• Falco version:

GitVersion:    2.26.0-36-g76a29e9-dirty
GitCommit:     76a29e9f15dff55c9019d8ba1f9e8dee75c4da50
GitTreeState:  dirty
BuildDate:     '2022-10-21T11:45:18Z'
GoVersion:     go1.19.1
Compiler:      gc
Platform:      linux/amd64

• System info:

• Cloud provider or hardware configuration:
• OS:

• Kernel:

• Installation method:

Kubernetes helm chart and local build from source

Additional context

[bc-sb]

Simple http server used as aws lambda:

'use strict';

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

module.exports.hello = async (event) => {
  console.log(event)
  await sleep(5000)
  if(event.headers["x-beau-api"] === "this-should-always-be-here") {
    return {
      statusCode: 200,
      body: JSON.stringify(
        {
          message: 'Go Serverless v3.0! Your function executed successfully!',
          input: event,
        },
        null,
        2
      ),
    };
  } else {
    return { statusCode: 400,
    body: JSON.stringify({
      message: "Failed to get expected header"
    })}
  }
};

Config used:

webhook:
  address: https://random-lambda-url-via-serverless.lambda-url.us-east-2.on.aws/
  customHeaders:
    X-Beau-API: this-should-always-be-here

[Issif]

I'm not sure the Lock solved the issue after all, I'm updating the Loki output to work with Grafana Logs (SaaS), and I'm facing this issue:

2022/11/02 12:47:48 [ERROR] : Loki - authentication error (401): {"status":"error","error":"authentication error: no credentials provided"}
2022/11/02 12:47:48 [ERROR] : Loki - authentication error
2022/11/02 12:47:48 [INFO]  : Loki - Post OK (204)
2022/11/02 12:47:48 [INFO]  : Loki - Post OK (204)

[bc-sb]

It might need a refactor at this point. It seems like because there are 2 methods that can manipulate the headers there is definitely a non zero chance one might beat the other. Wondering if the post call can accept the headers and just do it locally in scope. If you move the lock unlock into the Loki client I assume it fixes it?

…

On Wed, Nov 2, 2022 at 7:49 AM Thomas Labarussias ***@***.***> wrote: I'm not sure the Lock solved the issue after all, I'm updating the Loki output to work with Grafana Logs (SaaS), and I'm facing this issue: 2022/11/02 12:47:48 [ERROR] : Loki - authentication error (401): {"status":"error","error":"authentication error: no credentials provided"} 2022/11/02 12:47:48 [ERROR] : Loki - authentication error 2022/11/02 12:47:48 [INFO] : Loki - Post OK (204) 2022/11/02 12:47:48 [INFO] : Loki - Post OK (204) — Reply to this email directly, view it on GitHub <#375 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2XKELHBT5T5M6SEWHV4JZLWGJILPANCNFSM6AAAAAAROEFM6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Issif]

Yes, your first solution by locking at output level seems the only way for now. Sorry for having led you in a wrong direction.

[Issif]

@bc-sb can you submit a new PR or do you want me to do it?

[bc-sb]

@Issif sorry...last week got away from me...so you want the original version I posted put up for PR? If so, I can certainly quickly get that done.

[Issif]

I think it's the better for now, adding the mutex in each concerned output. Another solution would take much more time and refacto. I adding new outputs since your first draft, take care. Thanks