Narek/streaming hnsw index selects #322
Open
+347
−227
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
- Renamed and reorganized SQL update files for better structure
- Updated triggers and functions for async tasks and vector search
- Modified HNSW index scan logic with
continue_searchvariable - Updated
third_party/usearchsubmodule to a new commit hash - Added empty
sql/updates/0.3.0--0.3.1.sqlfile for version tracking
src/hnsw/scan.c
Outdated
| @@ -181,6 +181,7 @@ bool ldb_amgettuple(IndexScanDesc scan, ScanDirection dir) | |||
|
|
|||
| int ef = ldb_hnsw_ef_search; // 0 if not set, but we pass it into usearch_custom_ef anyway since 0 is also a | |||
| // sentinel value there | |||
| bool continue_search = false; | |||
There was a problem hiding this comment.
Introduced continue_search to manage search state.
src/hnsw/scan.c
Outdated
| @@ -225,6 +226,7 @@ bool ldb_amgettuple(IndexScanDesc scan, ScanDirection dir) | |||
| scanstate->usearch_scalar, | |||
| k, | |||
| ef, | |||
| continue_search, | |||
There was a problem hiding this comment.
Passed continue_search to usearch_search_ef.
| @@ -252,6 +254,8 @@ bool ldb_amgettuple(IndexScanDesc scan, ScanDirection dir) | |||
| return false; | |||
There was a problem hiding this comment.
Set continue_search to true for subsequent searches.
src/hnsw/scan.c
Outdated
| @@ -272,12 +276,14 @@ bool ldb_amgettuple(IndexScanDesc scan, ScanDirection dir) | |||
| scanstate->usearch_scalar, | |||
| k, | |||
| ef, | |||
| continue_search, | |||
There was a problem hiding this comment.
Passed continue_search to usearch_search_ef.
| scanstate->labels, | ||
| scanstate->distances, | ||
| &error); | ||
| ldb_wal_retriever_area_reset(scanstate->retriever_ctx); | ||
|
|
||
| scanstate->count = num_returned; | ||
| scanstate->current = 0; |
There was a problem hiding this comment.
Reset scanstate->current to 0 after search.
third_party/usearch
Outdated
| @@ -1 +1 @@ | |||
| Subproject commit a20193e1f3e10e18c0a8edd860727b11bf424e44 | |||
| Subproject commit ece8908a58597ffa6570ca6fc5ef043103d05038 | |||
There was a problem hiding this comment.
Updated submodule commit from a20193e1f3e10e18c0a8edd860727b11bf424e44 to ece8908a58597ffa6570ca6fc5ef043103d05038.
Ngalstyan4
force-pushed
the
narek/streaming-hnsw-index-selects
branch
from
7302bbb
to
c29f9aa
Compare
Default to streaming search for index pagination This used to have a GUC option. I removed streaming search GUC and defaulted to it becasue the old approach had an issue and could simetimes skip some index values and never returned them when paginating
Ngalstyan4
force-pushed
the
narek/streaming-hnsw-index-selects
branch
from
53bb3bb
to
194e0f1
Compare
Ngalstyan4
force-pushed
the
narek/streaming-hnsw-index-selects
branch
from
194e0f1
to
87aa88a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.