Log aggregation

[mweinelt]

Is your feature request related to a problem? Please describe.

Querying the journal on rhea.nixos.org is annoying when it is under load. Just grepping through the last 24 hours takes a long time, and journalctl blocks a whole core, likely due to iowait.

Describe the solution you'd like

Stream the logs elsewhere, where we can store and query them.

Could be a simple as remote syslog, but could also be loki/promtail and allow integration and exposure via Grafana.

Describe alternatives you've considered

Additional context

n/a

[RaitoBezarius]

Relevant to the context: netdata is working on a bunch of systemd-journald optimizations. (I unfortunately lack of time to detail but I will comment when I have more time with precise details.)