[DPE-1177] Change 'master' to 'primary' in Patroni leader role #532

marceloneppel · 2024-06-26T14:45:53Z

Issue

Patroni sets some labels in the pods to indicate which unit is the primary and which are the replicas. Unfortunately, in older versions, it's using the master value for the role label.

Solution

We upgraded to use the primary value for the role label because we wanted to use better language. It's now possible to use custom values for the labels.

Notes:

The version constraints were upgraded on src/dependency.json to later, in the next stable after the one that will contain these changes, we can remove the upgrade logic by updating the version constraints again.
src/charm.py don't patch the charm custom k8s services during an upgrade to avoid creating moments of downtime (like, for example, if there is a unit still with a role label equal to master and we already switch to the value equal to primary in the k8s service).
src/upgrade.py contains temporary logic to handle the label value update during the upgrade process (moving to the primary label only when the last unit - unit 0 - will be upgraded to avoid downtime).
tests/integration/ha_tests/test_rollback_to_master_label.py and tests/integration/ha_tests/test_upgrade_to_primary_label.py are almost a copy from tests/integration/ha_tests/test_upgrade.py, but they should be removed later, as the intent is only to validate the role label value migration at this point.
More unit tests will be added through another PR because this PR is already over 500 lines.

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

codecov · 2024-06-26T14:47:41Z

Codecov Report

Attention: Patch coverage is 63.04348% with 17 lines in your changes missing coverage. Please review.

Project coverage is 70.74%. Comparing base (9ea09b7) to head (289e392).

Files	Patch %	Lines
src/relations/async_replication.py	30.00%	7 Missing ⚠️
src/upgrade.py	30.00%	5 Missing and 2 partials ⚠️
src/charm.py	88.46%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #532      +/-   ##
==========================================
- Coverage   70.92%   70.74%   -0.18%     
==========================================
  Files          10       10              
  Lines        2817     2834      +17     
  Branches      525      531       +6     
==========================================
+ Hits         1998     2005       +7     
- Misses        720      727       +7     
- Partials       99      102       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

…first unit upgraded) Signed-off-by: Marcelo Henrique Neppel <[email protected]>

…first unit upgraded) in the rollback test Signed-off-by: Marcelo Henrique Neppel <[email protected]>

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

dragomirp · 2024-07-11T15:10:24Z

tests/integration/ha_tests/test_rollback_to_master_label.py

+        ops_test.model.deploy(
+            DATABASE_APP_NAME,
+            num_units=3,
+            channel="14/stable",


Maybe we should use a specific revision here? New releases would use the new label by default.

Great idea. I updated it on f02e964.

lucasgameiroborges · 2024-07-11T18:13:41Z

tests/integration/ha_tests/test_rollback_to_master_label.py

+    logger.info("Wait for applications to become active")
+    async with ops_test.fast_forward():
+        await ops_test.model.wait_for_idle(
+            apps=[DATABASE_APP_NAME, APPLICATION_NAME], status="active"


Optional of course, but if you want to avoid https://warthogs.atlassian.net/browse/DPE-4855 you can add raise_on_error=False in this specific wait.

Cool! Thanks for letting me know about that! I updated this PR with that on f02e964.

lucasgameiroborges · 2024-07-11T18:14:04Z

tests/integration/ha_tests/test_upgrade_to_primary_label.py

+    logger.info("Wait for applications to become active")
+    async with ops_test.fast_forward():
+        await ops_test.model.wait_for_idle(
+            apps=[DATABASE_APP_NAME, APPLICATION_NAME], status="active"


Updated this PR with that on f02e964.

lucasgameiroborges · 2024-07-11T18:21:21Z

src/charm.py

+            try:
+                self._create_services()
+            except ApiError:
+                logger.exception("failed to create k8s services")


More of a question here, sorry 😅 but will log.exception (without the error as argument) display the error info for debugging purposes (eg. message, stack trace, etc)? I imagine so, just want to confirm so this doesn´t become a mistery later on!

Yes, it does. I did a quick local test to double-check it.

lucasgameiroborges

LGTM! 🚀

dragomirp · 2024-07-12T08:03:36Z

tests/integration/ha_tests/test_upgrade.py

The writes_increasing checks seem to be failing consistently on juju 3.

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

marceloneppel · 2024-07-15T12:33:46Z

tests/integration/ha_tests/test_rollback_to_master_label.py

+        ops_test.model.deploy(
+            DATABASE_APP_NAME,
+            num_units=3,
+            channel="14/stable",


Great idea. I updated it on f02e964.

marceloneppel · 2024-07-15T12:35:13Z

tests/integration/ha_tests/test_rollback_to_master_label.py

+    logger.info("Wait for applications to become active")
+    async with ops_test.fast_forward():
+        await ops_test.model.wait_for_idle(
+            apps=[DATABASE_APP_NAME, APPLICATION_NAME], status="active"


Cool! Thanks for letting me know about that! I updated this PR with that on f02e964.

marceloneppel · 2024-07-15T12:35:41Z

tests/integration/ha_tests/test_upgrade_to_primary_label.py

+    logger.info("Wait for applications to become active")
+    async with ops_test.fast_forward():
+        await ops_test.model.wait_for_idle(
+            apps=[DATABASE_APP_NAME, APPLICATION_NAME], status="active"


Updated this PR with that on f02e964.

marceloneppel · 2024-07-15T12:50:33Z

src/charm.py

+            try:
+                self._create_services()
+            except ApiError:
+                logger.exception("failed to create k8s services")


Yes, it does. I did a quick local test to double-check it.

marceloneppel · 2024-07-22T19:56:13Z

lib/charms/postgresql_k8s/v0/postgresql.py

+        connection = psycopg2.connect(
+            f"dbname='{database if database else self.database}' user='{self.user}' host='{host}'"
+            f"password='{self.password}' connect_timeout=1"
+        )


Reverted due to failures in the async replication test (timeouts).

Change 'master' to 'primary' in Patroni leader role

ec7ffa4

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

github-actions bot added the Libraries: Out of sync label Jun 26, 2024

marceloneppel added 9 commits June 26, 2024 18:42

Handle upgrade and rollback

512f157

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

2fae815

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Add integration tests

b73b428

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

c8dc6e5

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Check labels and whether writes are increasing (the latter after the …

7b1fb65

…first unit upgraded) Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Check labels and whether writes are increasing (the latter after the …

959ce3c

…first unit upgraded) in the rollback test Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

38498d3

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Fix rollback revision and labels check

5c4e089

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Fix standby cluster initialisation

48c1ad7

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

marceloneppel marked this pull request as ready for review July 11, 2024 12:28

marceloneppel requested review from delgod, dragomirp and lucasgameiroborges July 11, 2024 12:28

dragomirp reviewed Jul 11, 2024

View reviewed changes

delgod approved these changes Jul 11, 2024

View reviewed changes

lucasgameiroborges reviewed Jul 11, 2024

View reviewed changes

lucasgameiroborges approved these changes Jul 11, 2024

View reviewed changes

dragomirp reviewed Jul 12, 2024

View reviewed changes

marceloneppel added 7 commits July 12, 2024 10:18

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

9279600

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Fix upgrade test

c80c62e

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Avoid errors on deploy and set specific revisions

f02e964

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

d8fbeb1

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

87452d5

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Merge remote-tracking branch 'origin/main' into dpe-1177-primary-role…

7975d74

…-label Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Add charm series

5bfbfc3

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

Remove Retrying

289e392

Signed-off-by: Marcelo Henrique Neppel <[email protected]>

marceloneppel commented Jul 22, 2024

View reviewed changes

marceloneppel merged commit 988d3ca into main Jul 22, 2024
84 checks passed

marceloneppel deleted the dpe-1177-primary-role-label branch July 22, 2024 19:56

taurus-forever mentioned this pull request Jul 29, 2024

Fix wrong wording #101

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DPE-1177] Change 'master' to 'primary' in Patroni leader role #532

[DPE-1177] Change 'master' to 'primary' in Patroni leader role #532

marceloneppel commented Jun 26, 2024 •

edited

Loading

codecov bot commented Jun 26, 2024 •

edited

Loading

dragomirp Jul 11, 2024

marceloneppel Jul 15, 2024

lucasgameiroborges Jul 11, 2024

marceloneppel Jul 15, 2024

lucasgameiroborges Jul 11, 2024

marceloneppel Jul 15, 2024

lucasgameiroborges Jul 11, 2024 •

edited

Loading

marceloneppel Jul 15, 2024

lucasgameiroborges left a comment

dragomirp Jul 12, 2024

marceloneppel Jul 15, 2024

marceloneppel Jul 15, 2024

marceloneppel Jul 15, 2024

marceloneppel Jul 15, 2024

marceloneppel Jul 22, 2024

[DPE-1177] Change 'master' to 'primary' in Patroni leader role #532

[DPE-1177] Change 'master' to 'primary' in Patroni leader role #532

Conversation

marceloneppel commented Jun 26, 2024 • edited Loading

Issue

Solution

codecov bot commented Jun 26, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucasgameiroborges Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucasgameiroborges left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marceloneppel commented Jun 26, 2024 •

edited

Loading

codecov bot commented Jun 26, 2024 •

edited

Loading

lucasgameiroborges Jul 11, 2024 •

edited

Loading