2021 08 14 (Saturday) Deployment

General Notes

This is fairly large deployment including two batches of work completed by the vendor. It includes some data migrations to fix some data and consolidate some unnecessary records. It also notably includes some more automation for the migration process. We will be paying close attention to how these procedures perform on their maiden voyage.

Due to the high amount of code changes, we will be performing this deployment after hours and on the weekend. It has been tested in the dev, mig, and test environments.

Timeline

• 22:00 -  made the Pull Request

• 22:10 - ran ./docker-to-ecr.sh latest

• 22:15 - ran ./docker-to-ecr.sh prod

  NOTE: There exists some confusion about what the correct procedure is for this, and I have received conflicting guidance. Created this ticket to track that and help bring clarity to me and the team about its usage.

• 22:18 - ran ./clamav-docker-to-ecr.sh prod

• 22:24 - ran account specific

  NOTE: had to add -upgrade flag to download the latest provider

  NOTE: Using 8 nodes in prod for the Kibana cluster until make snapshots and are comfortable running without all of the logs to search upon.

• 22:26 - created missing log groups create-missing-log-groups.sh

• 22:27 - reran account specific ✅ deploy:account-specific

• 22:28 - ran environment-specific ✅ deploy:environment-specific

• 22:30 - merged PR ✅

• 22:45 - tests pass ✅

• 22:48 - deploy fails; 👎 error: expected the efcms-****-beta table to have been deleted from us-east-1 before running migration

  NOTE: I believe we have an improvement to this where if the table exists (and is empty) to not fail here and instead just use the empty DDB table.

• 22:50 - deleted tables

• 23:00 - restart from failed

• 23:05 - observed setup of migration worked! ✅ 😎

• 23:45 - Migration kicked off ✅

• 00:40 - Severe weather alert for flooding 😨 ⛈️

• 00:46 - migration finished; ✅ indexing begins 🕐

• 03:00 - clusters quieted, testing looks good, switched colors

  Comparison between ES Clusters (Beta was destination; Alpha was source)

  Discrepancy with work items is due to the migration script to consolidate those records Discrepancy with the docket entry, user, and user-case is due to activity after switch colors

  indexName	countAlpha	countBeta	diff
  'efcms-case'	1990810	1990810	0
  'efcms-case-deadline'	16318	16318	0
  'efcms-docket-entry'	18253964	18253968	4
  'efcms-message'	291752	291752	0
  'efcms-user'	2001222	2001226	4
  'efcms-user-case'	1979220	1979228	8
  'efcms-work-item'	775992	740948	35044

  Total Difference: 35028 (25274250/25309278) 99.86%