-
Notifications
You must be signed in to change notification settings - Fork 46
2021 08 14 (Saturday) Deployment
This is fairly large deployment including two batches of work completed by the vendor. It includes some data migrations to fix some data and consolidate some unnecessary records. It also notably includes some more automation for the migration process. We will be paying close attention to how these procedures perform on their maiden voyage.
Due to the high amount of code changes, we will be performing this deployment after hours and on the weekend. It has been tested in the dev
, mig
, and test
environments.
-
22:00 - made the Pull Request
-
22:10 - ran
./docker-to-ecr.sh latest
-
22:15 - ran
./docker-to-ecr.sh prod
NOTE: There exists some confusion about what the correct procedure is for this, and I have received conflicting guidance. Created this ticket to track that and help bring clarity to me and the team about its usage.
-
22:18 - ran
./clamav-docker-to-ecr.sh prod
-
22:24 - ran account specific
NOTE: had to add
-upgrade
flag to download the latest providerNOTE: Using 8 nodes in prod for the Kibana cluster until make snapshots and are comfortable running without all of the logs to search upon.
-
22:26 - created missing log groups
create-missing-log-groups.sh
-
22:27 - reran account specific ✅
deploy:account-specific
-
22:28 - ran environment-specific ✅
deploy:environment-specific
-
22:30 - merged PR ✅
-
22:45 - tests pass ✅
-
22:48 - deploy fails; 👎
error: expected the efcms-****-beta table to have been deleted from us-east-1 before running migration
NOTE: I believe we have an improvement to this where if the table exists (and is empty) to not fail here and instead just use the empty DDB table.
-
22:50 - deleted tables
-
23:00 - restart from failed
-
23:05 - observed setup of migration worked! ✅ 😎
-
23:45 - Migration kicked off ✅
-
00:40 - Severe weather alert for flooding 😨 ⛈️
-
00:46 - migration finished; ✅ indexing begins 🕐
-
03:00 - clusters quieted, testing looks good, switched colors
Comparison between ES Clusters (Beta was destination; Alpha was source)
Discrepancy with work items is due to the migration script to consolidate those records Discrepancy with the docket entry, user, and user-case is due to activity after switch colors
indexName countAlpha countBeta diff 'efcms-case' 1990810 1990810 0 'efcms-case-deadline' 16318 16318 0 'efcms-docket-entry' 18253964 18253968 4 'efcms-message' 291752 291752 0 'efcms-user' 2001222 2001226 4 'efcms-user-case' 1979220 1979228 8 'efcms-work-item' 775992 740948 35044 Total Difference: 35028 (25274250/25309278) 99.86%