Initializing the index mapping for the elasticsearch's output

[idrissneumann]

Motivation

Make the search easier with Elasticsearch and let falcosidekick define the index mapping using an autocreateindex configuration.

Feature

Having a elasticsearch.autocreateindex boolean configuration. If it's set to true, falcosidekick will push an index mapping for the payloads, exactly the same way it has been done for Quickwit here: #736

Additional context

Already discussed with @Issif here: https://github.com/falcosecurity/falcosidekick/pull/736/files#discussion_r1464786855

[Issif]

Even if we can't match all possible output_fields, especially because of the plugins which have their own and the possibility in falcosidekick to inject custom fields in the payload. Do you think it is worth to create the mapping with all "basic" fields Falco handles: https://falco.org/docs/reference/rules/supported-fields/ ?

[idrissneumann]

@Issif I think it's a little bit less important than Quickwit because Elasticsearch can handle a dynamic mapping from the first ingested document. So it's important if there's some concurrent types with the same key that will be ingested, for example a field that can contain a date or something else which is not a date or timestamp, if the first ingested document containing the key is interpreted as date, the other following can be ignored and not indexed at all.

So if we know that some key must be indexed as raw text in advance, it might be worth it to define the mapping of those subfields at the init to be sure the first ingested documents will not be interpreted as a more restricted type.

[Issif]

We're on the same page 😉

[Issif]

For helping you, you can see how I did it in falco talon, it creates an index template at init https://github.com/Falco-Talon/falco-talon/tree/main/notifiers/elasticsearch

[Issif]

@idrissneumann what's the status on your side? do you need help?

[Issif]

Will be fixed in the upcoming 2.29