Skip to content

2021 08 28 (Saturday) Deployment

Jump to bottom
Mike Marcotte edited this page Aug 29, 2021 · 6 revisions

General Notes

This deployment includes one bugfix and batch 7. Batch 7 is mainly readying search for Court users as well as a few more deployment improvements. The feature work has been tested in the mig environment, and the bugfix has been tested in the test environment.

Until we can accomplish a more seamless user experience for deployments, we will be performing deployments on weekends or after midnight on weekdays.

Bugfixes

Feature Stories

Timeline

  • 00:28 - ran environment specific
Terraform will perform the following actions:

  # aws_iam_role_policy.lambda_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "lambda_policy" {
        id     = "lambda_role_prod:lambda_policy_prod"
        name   = "lambda_policy_prod"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                    # (6 unchanged elements hidden)
                    {
                        Action   = [
                            "dynamodb:BatchGetItem",
                            "dynamodb:BatchWriteItem",
                            "dynamodb:DeleteItem",
                            "dynamodb:DescribeStream",
                            "dynamodb:DescribeTable",
                            "dynamodb:GetItem",
                            "dynamodb:GetRecords",
                            "dynamodb:GetShardIterator",
                            "dynamodb:ListShards",
                            "dynamodb:ListStreams",
                            "dynamodb:PutItem",
                            "dynamodb:Query",
                            "dynamodb:UpdateItem",
                        ]
                        Effect   = "Allow"
                        Resource = [
                            "arn:aws:dynamodb:us-east-1:980423577122:table/efcms-prod-*",
                            "arn:aws:dynamodb:us-west-1:980423577122:table/efcms-prod-*",
                        ]
                    },
                  ~ {
                      ~ Action   = [
                          + "dynamodb:GetItem",
                            "dynamodb:DescribeTable",
                        ]
                        # (2 unchanged elements hidden)
                    },
                    {
                        Action   = [
                            "ses:SendBulkTemplatedEmail",
                        ]
                        Effect   = "Allow"
                        Resource = [
                            "arn:aws:ses:us-east-1:980423577122:identity/[email protected]",
                        ]
                    },
                    # (3 unchanged elements hidden)
                ]
                # (1 unchanged element hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }
  • 00:31 - made the Pull Request
  • 00:32 - ran the $ ./scripts/setup-order-search-flag.sh prod script to inject the toggle value into the deploy table.
  • 00:32 - merged PR ✅; CircleCI build
  • 00:42 - tests pass ✅
  • 00:57 - deploy step fails. Perhaps I forgot to delete the west table that was no longer in use.
Error: error initially creating DynamoDB Table (efcms-****-beta) replicas: error creating DynamoDB Table (efcms-****-beta) replica (us-west-1): ValidationException: Failed to create a the new replica of table with name: ‘efcms-****-beta’ because one or more replicas already existed as tables.
│ 	status code: 400, request id: U7D3BJ9AJJV5D5ARUUJSH0DJ63VV4KQNSO5AEMVJF66Q9ASUAAJG
│ 
│   with module.ef-cms_apis.module.dynamo_table_beta.aws_dynamodb_table.efcms-table-east,
│   on ../template/dynamo-table/dynamo-migration.tf line 15, in resource "aws_dynamodb_table" "efcms-table-east":
│   15: resource "aws_dynamodb_table" "efcms-table-east" {
  • 00:59 - deleted tables efcms-prod-beta east & west
  • 01:01 - rerun from failed CircleCI build
  • 01:15 - deploy completed successfully
  • 01:20 - initial testing looks good; switching colors
  • 01:25 - observed that search is still disabled despite https://github.com/flexion/ef-cms/issues/8544. It appears that it's hardcoded to not work in production:
const getIsFeatureEnabled = (featureName, user, env) => {
  const features = {
    advanced_document_search: (() => {
      const isProduction = env === 'prod';
      const isInternalUser = User.isInternalUser(user.role);

      return !isProduction && isInternalUser;
    })(),
  };

  return !!features[featureName];
};
  • 02:00 - created a new branch to remove this hardcoded measure, and merged it.
  • 02:10 - created a PR to prod and merged it to kickoff the deployment inside of the deployment window. CircleCI build
  • 02:17 - created an additional test to test for environment = prod and user is not internal, and merged it.
  • 02:20 - tests pass
  • 02:33 - deploy completed
  • 02:35 - switched colors
  • 02:36 - looks like search is working in prod now! 👍 conducting some more testing throughout the system before calling it a night.

Search is working in prod!

search is live

AWS environment configuration

Clone this wiki locally