Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to cr-sqlite 0.16.0 + a few performance improvements #75

Merged
merged 12 commits into from
Sep 29, 2023

Conversation

jeromegn
Copy link
Member

Upgrading to cr-sqlite 0.16.0 (unreleased, for now) gives a significant disk space reduction by adding a "lookaside" table for primary keys. Instead of repeating a whole primary key for each row in the clock table (there's 1 row per column + 1 sentinel row), this assigns an integer that can be looked up.

For consul services and checks, the savings are significant (about 30% in our case).

This PR also adds a few performance improvements:

  • Process all "empty" (cleared) db versions out of band in bigger transactions
  • Buffer more data when synchronizing change before sending them

@jeromegn jeromegn merged commit 4b723ea into main Sep 29, 2023
4 checks passed
@jeromegn jeromegn deleted the crsql-compact-pks branch September 29, 2023 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant