Skip to content
forked from crewjam/etcd-aws

tools for building a robust etcd cluster in AWS

Notifications You must be signed in to change notification settings

topochan/etcd-aws

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

etcd-aws

This repository contains tools for building a robust etcd cluster in AWS.

It uses CloudFormation to establish a three node autoscaling group of etcd instances. In case of the failure of a single node, the cluster remains available and the replacement nodes are integrated automatically into the cluster. Each node in the cluster can be replaced by a new node, one at a time, and the cluster remains available. In the event of failure of all nodes simultaneously, the cluster recovers from the backup stored in S3 without intervention.

Please see this blog post for more on how this little utility came to be.

Invoking the etcd-aws program will configure and launch etcd based on the current autoscaling group:

etcd-aws

It is also available as a Docker container:

/usr/bin/docker run --name etcd-aws \
  -p 2379:2379 -p 2380:2380 \
  -v /var/lib/etcd2:/var/lib/etcd2 \
  -e ETCD_BACKUP_BUCKET=my-etcd-backups \
  --rm crewjam/etcd-aws

CloudFormation

The program etcd-aws-cfn generates and deploys a CloudFormation template:

go install ./...
etcd-aws-cfn -key-pair my-key

You can also generate the CloudFormation template and deploy it yourself:

etcd-aws-cfn -key-pair my-key -dry-run > etcd.template

The template consists of:

  • A VPC containing three subnets across three availability zones.
  • An autoscaling group of CoreOS instances running etcd with an initial size of 3.
  • An internal load balancer that routes etcd client requests to the autoscaling group.
  • A lifecycle hook that monitors the autoscaling group and sends termination events to an SQS queue.
  • An S3 bucket that stores the backup.
  • CloudWatch alarms that monitor the health of the cluster and that the backup is happening.

Cluster Discovery

The program etcd-aws discovers other cluster members by looking for EC2 instances that are part of the same autoscaling group. It invokes etcd with appropriate configuration settings based on the result of cluster discovery.

When adding nodes to an existing cluster, etcd-aws automatically registers the node before it is launched.

The program monitors an AWS AutoScaling Lifecycle Hook to detect when nodes are terminated and removes them from the cluster. This is important because the terminated nodes no longer count against the etcd quorum calculation.

Backup

Periodically, etcd-aws writes a file to S3 containing the value of all the keys in the etcd database.

When creating the first node of a cluster, etcd-aws checks for an existing backup and automatically restores it. In this way, an etcd-aws cluster can recover from failure of all nodes in the cluster.

Load Balancer

The CloudFormation template creates a load balancer which can be used by etcd clients to discover cluster members. Etcd clients tend to be cluster aware -- they discover the cluster members on initial contact. You can configure an etcd client to connect to the load balancer, which will provide the initial node list, and then the client will connect directly to the current nodes in the cluster. This avoids the need for clients to maintain and update a list of etcd nodes.

About

tools for building a robust etcd cluster in AWS

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 96.9%
  • Makefile 3.1%