-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DPE-4839] Point In Time Recovery #554
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #554 +/- ##
==========================================
- Coverage 70.94% 68.74% -2.21%
==========================================
Files 10 10
Lines 2853 2972 +119
Branches 536 563 +27
==========================================
+ Hits 2024 2043 +19
- Misses 726 811 +85
- Partials 103 118 +15 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks @Zvirovyi!
One last change is needed to make on of the backup tests pass.
In tests/integration/test_backups.py
's test_restore_on_new_cluster()
, you need to replace the following code (starting at line 290):
# Wait for the restore to complete.
async with ops_test.fast_forward():
await wait_for_idle_on_blocked(
ops_test,
database_app_name,
0,
S3_INTEGRATOR_APP_NAME,
ANOTHER_CLUSTER_REPOSITORY_ERROR_MESSAGE,
)
With:
# Wait for the restore to complete.
async with ops_test.fast_forward():
await wait_for_idle_on_blocked(
ops_test,
database_app_name,
0,
S3_INTEGRATOR_APP_NAME,
MOVE_RESTORED_CLUSTER_TO_ANOTHER_BUCKET, # <--- The constant needs to be changed to this one.
)
The other backup tests are already passing.
# Conflicts: # src/charm.py
This is work on porting the same functionality from the VM.
Resolves #553.
Point In Time Recovery
Add 'restore-to-time' parameter for 'restore' action to be able to do point-in-time-recovery. Also, add 'test_pitr_backup' integration test.
Overview
Whether PostgreSQL charm is active, it writes write-ahead log (WAL) - log of all the transactions with corresponding times. Periodically (mostly based on minimum size constraint) new WAL file is generated and then is archived to the stanza through s3-integrator relation. This WAL can be useful, if you want to restore not only backup, but transactions occurred after that - this is named Point-In-Time-Recovery.
Use-case example
Let's assume, that you have model with PostgreSQL and s3-integrator units, both are configured and integrated between each other.
Firstly, you will need standard backup - it will be base for our later WAL applying:
juju run postgresql/leader create-backup
Then, you need to do some transactions and switch WAL file manually (for showcase):
Shortly after that, you can start your PITR:
juju run postgresql/leader restore restore-to-time="2024-04-22 06:54:42.280896+00"
Also, in this example as target you can use second timestamp, but not third, as third was selected after all the transactions. Basically, DB knows only the time of transactions, and time outside of them is unknown. So, if you try to run PITR with third timestamp, it will fail with error "cannot reach PITR target".
For that cases when you need to apply all of the transactions in the WAL, you can use
juju run postgresql/leader restore restore-to-time="latest"
.After any restore (even ordinary full backup), database will restore to the status "Move restored cluster to another S3 bucket".
Must be known
Move restored cluster to another S3 bucket
Database will now become in "Move restored cluster to another S3 bucket" status after every recovery. This is made with purpose to warn user: restoring backup will overwrite current WAL timeline, and this will separate tree of events starting from the backup starting point.
For example:
day 1 - day 5
day 5 - day 8
day 2 - day 5
period, as new timeline became the latest timelineTechnically,
day 2 - day 5
period will still be in WAL archive - only opportunity to use it will be lost.So, requiring user to change stanza, ensures no such data lost.
For failed PITR,
juju debug-log
will report latest transaction timeDifferences from VM version:
pebble logs postgresql
command instead of calling it through API (PostgresqlOperatorCharm.is_pitr_failed
), but it's not critical and can be improved later with adding required method to the pebble client.