How to get etcd metrics?

[CRASH-Tech]

It's not a pod, and I can't scape metrics from the cluster.
Where is metrics endpoint?

[frezbo]

You'd set the extraArgs for etcd here: https://www.talos.dev/v1.4/reference/configuration/#etcdconfig

and add a listen-metric-urls

[CRASH-Tech]

Set to 0.0.0.0 and scrape from cp node ip's?

[frezbo]

yes, you can also set it to the node ip

[CRASH-Tech]

Thanks!

[gerhard]

So what is the recommended way of configuring etcd metrics scraping again?

I was not able to get the above recommendation to work. I am running:

• Talos v1.5.3 managed by Omni v0.18.1
• K8s v1.28.2
• kube-prometheus-stack v48.3.6

After applying the following cluster patch:

cluster:
  etcd:
    extraArgs:
      listen-metrics-url: 0.0.0.0

etcd continues to listen on IPv6 only:

# netstat -ltnp | grep etcd
tcp6       0      0 :::2380                 :::*                    LISTEN      2061/etcd
tcp6       0      0 :::2379                 :::*                    LISTEN      2061/etcd

---

FWIW:

# curl -6 "http://[::1]:2380/metrics" -v
*   Trying [::1]:2380...
* Connected to ::1 (::1) port 2380 (#0)
> GET /metrics HTTP/1.1
> Host: [::1]:2380
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

# curl -6 "http://[::1]:2379/metrics" -v
*   Trying [::1]:2379...
* Connected to ::1 (::1) port 2379 (#0)
> GET /metrics HTTP/1.1
> Host: [::1]:2379
> User-Agent: curl/7.88.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

[gerhard]

I was able to reboot it from Omni after the manual reboot.

[gerhard]

All working, thanks everyone!

kube-prometheus-stack.yaml

# ↑ 1557 rows of config before

kubeEtcd:
  enabled: true

  ## If your etcd is not deployed as a pod, specify IPs it can be found on
  ##
  endpoints:
  - 192.168.88.12

  ## Etcd service. If using kubeEtcd.endpoints only the port and targetPort are used
  ##
  service:
    enabled: true
    port: 2381
    targetPort: 2381

# ↓ 2482 lines of config after

patches/etcd-metrics.yaml (for Omni)

# ⚠️ Requires reboot after changing!
cluster:
  etcd:
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381

[R-Studio]

Is there no way to automate this (prevent hardcoded IP's in kube-prometheus-stack.yaml)?

[abueschel]

@R-Studio I found a (admittedly bit ugly) workaround to this. Under the assumption that every controlplane node runs the same controlplane services (etcd, kube-controller-manager etc.) you can modify the selector of the headless servicekube-prometheus-stack-kube-etcd that kube-prometheus-stack uses for the serviceMonitor resource:

kube-prometheus-stack.yaml:

kubeEtcd:
  service:
    selector:
      k8s-app: kube-controller-manager # or any other service like kube-proxy or kube-scheduler

This populates the endpoints of the service with the controlplane server IPs, because kube-controller-manager pods are running in host network mode. You might want to do some additional relabeling, as the serviceMonitor will add some wrong labels for e.g. pod (etcd isn't running in a pod...)

Alternatively you could also add a simple deployment manifest to your Talos machine configuration for the controlplane. Just run a "pause" container in host network mode and you could pick a more fitting name.

[R-Studio]

@abueschel nice this workaround works perfectly (all of our controlplane nodes runs the same controlplane services).
My kube-prometheus-stack Helm values for etcd looks like this:

kubeEtcd:
  enabled: true
  service:
    selector:
      k8s-app: kube-controller-manager # Fix selector for kube-etcd for Talos (set itentionally to kube-controller-manager because all master nodes has the same roles)
  serviceMonitor:
    relabelings:   # Add nodename label
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: ^(.*)$
        targetLabel: nodename
        replacement: $1
        action: replace
    metricRelabelings:   # Remove pod label
      - action: labeldrop
        regex: pod

[Charlotte-br560]

You can retrieve etcd metrics directly from the etcd server using its HTTP API. The metrics endpoint typically resides at http://<etcd-server>:<port>/metrics. Ensure you have access to the etcd server and its port to fetch metrics.