ENG-2572: VectorClockIds are a tuple. Editing changesets are gone #3988
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
bors try |
github-actions
bot
added
A-sdf
tryBuild failed: |
|
bors try |
tryBuild failed: |
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
from
9f6710a
to
70b8a3b
Compare
|
bors try |
tryBuild failed: |
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
4 times, most recently
from
b7e532a
to
5f8fe8b
Compare
|
bors try |
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
from
5f8fe8b
to
2794b81
Compare
tryBuild succeeded: |
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
from
2794b81
to
055406c
Compare
zacharyhamm
changed the title
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
3 times, most recently
from
5059dfd
to
41a8c60
Compare
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
5 times, most recently
from
4345b56
to
4f21b25
Compare
✅ Deploy Preview for systeminit-tools ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
from
bfc952c
to
a21e3b1
Compare
Previously, vector clock ids were the same as change set ids. And, we generated a "editing change set" anytime we mutated the graph. This changeset was ephemeral, and not connected to a real "change set" in the system. In addition, for conflict detection to work correctly, the node write clocks have to store every vector clock write that has *ever* happened to them. This meant these clocks would grow indefinitely, since they have to store every ephemeral "editing change set" in the node, forever. This change transforms the vector clock id into a tuple of the real ChangeSetId and the UserPk/ActorId of the current user. In the context of system actors, like Pinga and the Rebaser, the WorkspacePk is used in place of the UserPk and removes editing change sets entirely. Now the bound on the vector clock write clocks in node weights is the number of change sets and users in the system, which will grow much more slowly than the editing change sets. This is a breaking change, since it changes both Node and Edge weight data strucutres. Migration must be in place before this can be deployed.
Whenever we don't have HistoryActor, generate an actor id that lasts for the current DalContext and use that for the vector clock id's actor id. But, when the rebaser writes out the final snapshot, use the workspace pk for the actor id. Co-Authored-By: Jacob Helwig <[email protected]>
Co-Authored-By: Jacob Helwig <[email protected]>
Co-Authored-By: Jacob Helwig <[email protected]>
On SDF boot, attempt to automatically migrate all snapshots for a deployment, beginning with the builtin workspace's snapshot. Follows the "based_on_change_set_id" paths, treating the snapshots as a dependency graph, so that shared clock ids are migrated to the new clock ids correctly. Once this code is deployed, SDF will panic if it encounters a 'legacy' snapshot. Co-Authored-By: Jacob Helwig <[email protected]>
zacharyhamm
force-pushed
the
zack/ENG-2572-vector-clock-is-a-tuple
branch
from
a21e3b1
to
6bd2768
Compare
If a table's structure changes, cached query plans against that table need to be invalidated, or postgresql will return an error. This change prevents that error after migrating the database in a production system running pb_bouncer, which holds on to connections and reuses them even if our services are restarted. We could avoid needing to discard plans by selecting exactly the columns we need instead of SELECT * (unless the column type changes!) This issue never hit us before because we haven't changed table structures much since adding pg_bouncer to the stack. Co-Authored-By: Jacob Helwig <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change mitigates the unbounded growth of vector clocks by using the change set id + the actor id as the vector clock id. Ephemeral "editing change sets" are gone, although system services get a randomly generated actor id for the duration of their DalContext. It also prunes more vector clocks, now including write clocks, on rebase, ensuring our vector clock entries remain small.
WorkspaceSnapshot method signatures have been refactored to take the vector clock id directly, instead of assuming the change set and vector clock id are the same.
All snapshots for change sets that are not applied will be migrated on SDF boot, so deploying this to prod may cause some downtime.