Change API (Feature Request)

[eraserhd]

Rough Idea

A generic mechanism that can be hooked to be notified of any change to a buffer, along with enough information to understand how the buffer changed; e.g. was it an insert, a delete, or a change? If an insert, where was the cursor, and what is the range of the inserted text? If a delete, what was the deleted text's range and content? If a change, what range was changed, what was its old value, and what is its new range?

Motivation

parinfer-rust uses diffing to determine what has changed each buffer iterations, as does kak-lsp. At conference, I had a long discussion with Shaun Lebron, the author of the parinfer algorithm that I ported, and he pointed out several ways in which diffing fails to isolate the actual changes, making parinfer not work correctly.

1. Parinfer's "smart" mode needs to know which parens are inserted and which are moved to determine which parens to balance and which to push. In cases where one inserts the middle paren in something like (((, diffing algorithms choose an arbitrary paren, and that causes us to do the wrong thing.
2. When making complex changes, diffing is even worse. Since Kakoune relies heavily on multiple cursors, we are hopefully making complex changes more often than Vim.

In addition, having a change API will reduce the amount of data sent to parinfer-rust, which currently uses two environment variables with the entire "previous" and "current" contents of the file so it can diff them. This hits environment variable size limitations on Mac OS. We'd be able to, instead, send one whole copy via piping, and only delta information.

Some changes - ones triggered by external agents with kak -p are not currently detectable except via NormalIdle and diffing.

Possible Methods of Implementing

1. A hook which triggers for every change.
2. A buffer-local variable which contains description of each change that can be checked at NormalIdle (so multiple changes can be processed when needed).
3. A command which takes a revision number and a variable name and sets the variable to a description of the change.
4. ???

I'm happy to do the implementation work here.

[Screwtapello]

I don't want to scope-creep too much, but it seems like there's a chance this could help address issues like #2021 and #745.

Also note that if diffing is a problem, Kakoune sometimes uses diffing to determine what changed, so you may still be stuck.

[eraserhd]

It looks like exposing the history tree the way mentioned in #2021 would resolve this.

[eraserhd]

Reopening this because of discussion on IRC a while back. The history tree mechanism doesn't quite work, and Kakoune stores a better mechanism that "linearizes" all changes, and we can expose this. Since it is linearized, it doesn't require walking up the history tree and then back down in plugin code when the user jumps to a part in the history tree and a plugin wants to catch up.

[Screwtapello]

Could you describe in more detail why the history tree mechanism doesn't quite work, and which part of Kakoune is the "better mechanism"?

[eraserhd]

@Screwtapello Exposing Buffer::m_changes. History ids are just indexes into this, so we can keep track of the last history id we processed, then process changes forward to the end and store the new history id. I'm told this is how highlighters keep in sync.

Exposing the history tree is still useful for plugins to serialize history, but this is much easier to work with for plugins which need to keep up-to-date on buffer contents, as this changes even when the user is jumping around in history but not making changes.

[mawww]

History id are not indexes into the changes array, timestamps are. Exposing this would make sense, but this probably gets quite big, so we likely would want a way to get changes since a given timestamp. The problem being that we cannot resolve say $kak_changes_since_12345 without having it visible in the shell expansion source.