Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move in-memory bookkeeping into SQLite #52

Open
jeromegn opened this issue Sep 18, 2023 · 0 comments
Open

Move in-memory bookkeeping into SQLite #52

jeromegn opened this issue Sep 18, 2023 · 0 comments

Comments

@jeromegn
Copy link
Member

Corrosion track bookkeeping in memory, as well as in SQLite. It uses more efficient data structures in memory. This makes accessing the state of Corrosion out-of-band very fast and easy.

The reason for its existence is: finding "gaps" in SQLite tables is hard to optimize and the queries are therefore resource intensive. Being able to quickly figure out if Corrosion has "seen" a version, without a DB lookup, before checking out a write connection from the pool is efficient.

Unfortunately, it also comes with a lot of downsides:

  • The in-memory bookkeeping structure needs to be protected via locking as it can be used from any threads.
  • Since we're locking, we can't easily bundle up many change insertions into one transaction as we require locking on a structure before inserting.
  • We have to periodically compact the in-memory representation based on what's in the database to keep memory usage low

It might be possible to get rid of it entirely or design it in a way that it's not a requirement for inserts. Could likely be achieved by devising queries that looks at bookkeeping as Corrosion tries to insert data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant