You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This happened in prod, and I haven't reproduced in a local env.
At some point replicas go out of sync and try to restore pgdata from the primary, but fail with this error:
2024-04-29 09:16:53 UTC [15]: ERROR: Could not rename data directory /var/lib/postgresql/data/pgdata
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 1314, in remove_data_directory
shutil.rmtree(self._data_dir)
File "/usr/lib/python3.10/shutil.py", line 731, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python3.10/shutil.py", line 729, in rmtree
os.rmdir(path)
PermissionError: [Errno 13] Permission denied: '/var/lib/postgresql/data/pgdata'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 1287, in move_data_directory
os.rename(self._data_dir, new_name)
PermissionError: [Errno 13] Permission denied: '/var/lib/postgresql/data/pgdata' -> '/var/lib/postgresql/data/pgdata.failed'
Expected behavior
Replicas retrieve correctly the wal entries from the primary and restore their state.
Actual behavior
Replicas fail to get pgdata and fail. This in turn causes the primary wal to increase the size until it can't function properly.
Steps to reproduce
This happened in prod, and I haven't reproduced in a local env.
At some point replicas go out of sync and try to restore pgdata from the primary, but fail with this error:
Expected behavior
Replicas retrieve correctly the wal entries from the primary and restore their state.
Actual behavior
Replicas fail to get pgdata and fail. This in turn causes the primary wal to increase the size until it can't function properly.
Versions
Operating system:
Juju CLI: 2.9.49
Juju agent: 3.1.8
Charm revision: 14/edge 198
microk8s: v1.26.15
Log output
Juju debug log:
Additional context
To resolve this issue I used these steps: https://matrix.to/#/!BukWfnyOTgQSKAxdtT:ubuntu.com/$C-iLZEXS39xBD8vFV40EVBWFefNjlvUQmxFDxNcS2p0?via=ubuntu.com&via=matrix.org
but the I am not sure how to prevent this issue in the future.
The text was updated successfully, but these errors were encountered: