Skip to content

ETCD

Jump to bottom
Vonng edited this page Feb 11, 2023 · 3 revisions

ETCD

ETCD is a distributed, reliable key-value store for the most critical data of a distributed system

Pigsty use etcd as DCS: Distributed configuration storage (or distributed consensus service). Which is critical to PostgreSQL High-Availability & Auto-Failover.

You have to install ETCD module before installing any PGSQL modules, since patroni & vip-manager will use etcd as DCS. You have to install it after NODE module is installed, because etcd require the trusted CA to work.

Check ETCD Administration for details.

Playbook

There's a built-in playbook: etcd.yml for installing etcd cluster. But you have to define it first.

./etcd.yml    # install etcd cluster on group 'etcd'

Here are available sub tasks:

  • etcd_assert : generate minio identity
  • etcd_install : install etcd rpm packages
  • etcd_clean : cleanup existing etcd
    • etcd_check : check etcd instance is running
    • etcd_purge : remove running etcd instance & data
  • etcd_dir : create etcd data & conf dir
  • etcd_dir : create etcd directories
  • etcd_config : generate etcd config
    • etcd_conf : generate etcd main config
    • etcd_cert : generate etcd ssl cert
  • etcd_launch : launch etcd service
  • etcd_register : register etcd to prometheus

If etcd_safeguard is enabled, or etcd_clean is false, the playbook will abort if any running etcd instance exists to prevent purge etcd accidently.

Configuration

You have to define an etcd cluster before deploying it. There some parameters about etcd.

It is recommending to have 3 instances at least. Single node etcd is NOT Reliable enough for a serious production HA deployment.

Single Node

Define a group etcd in the inventory

# etcd cluster for ha postgres
etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

It will create a singleton etcd instance.

This is good enough for devbox, testing & demonstration, but not recommended in serious production environment.

Three Nodes

You can define etcd cluster with multiple nodes.

Remember to use odd number for cluster size.

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # odd number please
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging
    etcd_clean: true # purge etcd during init process

Administration

Cluster Management

Environment

Here's an example of client environment config.

Pigsty use etcd v3 API by default.

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

CRUD

You can do CRUD with following commands.

e put a 10 ; e get a; e del a ; # V3 API

Parameters

There are 10 parameters about ETCD module.

Parameter Type Level Comment
etcd_seq int I etcd instance identifier, REQUIRED
etcd_cluster string C etcd cluster & group name, etcd by default
etcd_safeguard bool G/C/A prevent purging running etcd instance?
etcd_clean bool G/C/A purging existing etcd during initialization?
etcd_data path C etcd data directory, /data/etcd by default
etcd_port port C etcd client port, 2379 by default
etcd_peer_port port C etcd peer port, 2380 by default
etcd_init enum C etcd initial cluster state, new or existing
etcd_election_timeout int C etcd election timeout, 1000ms by default
etcd_heartbeat_interval int C etcd heartbeat interval, 100ms by default
Clone this wiki locally