feat: Enable prometheus tendermint metrics

[DavidNix]

Closes #19

[DavidNix]

@nourspace With the federated monitoring, am I missing anything?

I've set the typical prometheus annotations for each pod. The pod already has a port mapping to 26660.

[nourspace]

@nourspace With the federated monitoring, am I missing anything?

I've set the typical prometheus annotations for each pod. The pod already has a port mapping to 26660.

The new Prometheus Operator uses resources like PodMonitor and ServiceMonitor where we define selectors for the pods or monitors we want to be monitored. These monitors can define paths and ports among other settings like job relabeling, which we might need to distinguish instances across clusters.

You can refer to the cosmos-node-pod-monitor.yaml which I'm adding as part of the sub-monitoring stack deployment. We could also manage that monitor by the operator.

I'm using PodMonitor because we don't define a Prometheus service that loadbalances between pods. There is a long discussion on the difference.
prometheus-operator/prometheus-operator#3119

TLDR from one comment

PodMonitor - no service
ServieMonitor - must has service

The end result is the same. The pod endpoints will be the ones scraped and each pod adds its ip as an instance label to the scrapped metric.

API reference
https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1.PodMonitor

Labels are no longer supported by the operator
prometheus-operator/kube-prometheus#16 (comment)

[DavidNix]

@nourspace thanks for the detail and background. I'll remove the annotations. The user can customize pod annotations if they want to use this style.

[DavidNix]

@nourspace or @agouin This is ready for review again. Small one.