deps,lib,src: add experimental web storage #52435

cjihrig · 2024-04-09T15:45:23Z

Reopening of #50169, which cannot be reopened. Opening as a draft because the original PR had a block, so the TSC would need to make a decision (#50169 (comment)).

This commit introduces an experimental implementation of the Web Storage API using SQLite as the backing data store.

nodejs-github-bot · 2024-04-09T15:45:28Z

Review requested:

@nodejs/gyp
@nodejs/security-wg
@nodejs/web-standards

anonrig · 2024-04-09T18:00:17Z

@nodejs/tsc can we add this to tomorrows meeting?

GeoffreyBooth · 2024-04-09T18:24:55Z

Looking at #50169 (review), I think all of those concerns are addressed by making SQLite an internal dependency that is not exposed in any way to the user; is that the case in this PR?

If so, then I think the block has been addressed, and there’s nothing to override.

ronag · 2024-04-09T19:02:52Z

Due to the history of this PR I propose we just quickly bring it up and approve it at the TSC to make sure there is no further frustration.

github-actions · 2024-04-10T02:15:51Z

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @anonrig.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

GeoffreyBooth · 2024-04-10T02:25:57Z

I’m all for adding localStorage support, but this PR isn’t exactly ready to land as is. The previous PR had no approvals, just the one block; and there was at least one note that hasn’t been addressed as far as I’m aware, of where to save the data (or how the user should specify where to save the data).

cjihrig · 2024-04-10T02:49:04Z

I think all of those concerns are addressed by making SQLite an internal dependency that is not exposed in any way to the user; is that the case in this PR?

This is currently the same implementation as #50169.

If so, then I think the block has been addressed, and there’s nothing to override.

The specific review containing the objection is not really relevant in my opinion. However, not trying to "keep up with the Joneses" (referring to Deno, txiki.js, and potentially Bun) was mentioned, as was a sentiment opposing browser APIs.

but this PR isn’t exactly ready to land as is

This is indeed not ready to land yet. I'm not going to invest any more time on it though until I know the TSC supports landing it in principle. I do believe the PR is far enough along for that decision to be made fairly easily.

mcollina

I’m in favor of landing this.

lib/internal/webstorage.js

jasnell · 2024-04-10T15:39:52Z

The @nodejs/tsc discussed this today and there is consensus on having web storage land. Obviously there are technical details to work through but there are no objections to this moving forward. There is also positive consensus on adding sqllite as an internal dependency in support of the implementation. Let's get this done :-)

jasnell · 2024-04-10T16:20:53Z

src/node_webstorage.cc

+    return;
+  }
+  CHECK(domexception_ctor_val->IsFunction());
+  Local<Function> domexception_ctor = domexception_ctor_val.As<Function>();


Non-blocking Nit: The code necessary to create a DOMException should likely be moved out into a separate utility method.

src/node_webstorage.h

src/node_webstorage.cc

cjihrig · 2024-04-10T18:39:54Z

The @nodejs/tsc discussed this today and there is consensus on having web storage land.

I appreciate it. I'll start addressing the feedback from this PR and the old one soon.

As a side note, I noticed on the TSC call that a few people had some technical questions, such as "why SQLite, specifically instead of writing to disk." I tried to answer this and other questions in #50169 (comment).

doc/api/globals.md

benjamingr

In general I am +1 on eventually landing this and like other experimental/new stuff I am in favor of landing this sooner rather than later and then iterating on it.

There is a lot of work to be done to make this stable but I think iterating usually is. abetter strategy than big changes and a lot of criticism.

My understanding is the TSC discussed this and approved the idea in principle so this is me +1ing (I couldn't attend due to a prod incident)

benjamingr · 2024-04-11T08:02:48Z

Note to self: create tickets after this lands with all the issues (e.g. range queries and others) so they can be addressed or discussed before this is stable.

mcollina · 2024-04-12T13:03:32Z

@cjihrig did you consider using:

leveldb
lmdb
berkeley db

Would any of the above provide slightly better characteristics (performance, binary size, source code size, manutenibility)?

Note that this does not matter much, and we can change the storage mechanism later (before it becomes stable).

cjihrig · 2024-04-12T13:12:08Z

did you consider using

Please see #50169 (comment) for my answer to why I chose SQLite.

benjamingr · 2024-04-12T14:49:54Z

I'd like to publicly agree with Colin on SQLite here.

I've also done a few projects with various options for embedded databases and have found SQLite to be an excellent choice due to its ease of integration, ultra-permissive license (public domain), omnipresence across hardware which means good support etc.

localStorage + sharing readers across workers seems right up its alley. A file with a global lock would work but would lead to a lot less extensibility and would make us solve problems in the future SQLite would avoid.

(If anything, we should eventully consider exposing SQLite the same way Python does)

mcollina · 2024-04-12T21:41:46Z

There might be a slight issue when adding SQLite as a dependency. It uses a non-approved license that is not OSI.

We would need to seek legal+board approval to land this. @nodejs/tsc can you weight in?

As an alternative, we could use libsql that is MIT license (but is a fork of sqlite).

nodejs-github-bot · 2024-06-13T04:01:50Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59745/ 🟡

jasnell

Now this is a decently sized PR ;-)

Ethan-Arrowood

love this! I'm not experienced enough with C++ to give that part a proper review, and I just scanned the WPT as I assume if they are passing then they are good to go.

I'll keep an eye on some of the other changes being introduced if they change.

Great work and thank you!

H4ad · 2024-06-13T16:05:14Z

src/node_webstorage.cc

+      // max_size is 10MB. This can be made configurable in the future.
+      "  max_size INTEGER NOT NULL DEFAULT 10485760,"


Does this only apply to sessionStoage or to localStorage too? If yes, in the docs, you only mentioned the limit for sessionStorage.

mcollina

lgtm with two nits/questions

mcollina · 2024-06-13T16:17:35Z

src/node_webstorage.cc

+        std::to_string(kCurrentSchemaVersion) + ";";
+    r = sqlite3_exec(db, set_user_version_sql.c_str(), 0, 0, nullptr);
+    CHECK_ERROR_OR_THROW(env(), r, SQLITE_OK, false);
+  }


I have a rough feeling we would need to implement a migration framework here. Is that risk mitigated somewhere?

The idea is to use the schema version for managing any required migrations.

mcollina · 2024-06-13T16:19:30Z

doc/api/globals.md

+> Stability: 1.0 - Early development.
+
+A browser-compatible implementation of [`Storage`][]. Enable this API with the
+[`--experimental-webstorage`][] CLI flag.


I think we should note that this is backed by SQLite and all actions are synchronous and blocking as they might cause the event loop to stall.

For the synchronous bit - this is a spec'ed API, so I'm not sure if it makes sense to explain that type of detail. I'd also prefer not to go through the whole CI process again for minor docs things that can more easily be done in a followup PR.

nodejs-github-bot · 2024-06-14T18:40:08Z

Landed in 340cb68...15b3902

This commit introduces an experimental implementation of the Web Storage API using SQLite as the backing data store. PR-URL: #52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This is equivalent to the following upstream change: https://sqlite.org/src/info/6c103aee6f146869 Original commit message: Change constant expressions to pre-computed constants, because apparently MSVC on ARM requires that. [forum:/forumpost/4feb1685cced0a8e|Forum thread 4feb1685cced0a8e]. FossilOrigin-Name: 6c103aee6f146869a3e0c48694592f2e4c6b57ecdb4450f46e762c38b4e686f1 PR-URL: #52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This commit introduces an experimental implementation of the Web Storage API using SQLite as the backing data store. PR-URL: #52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This is equivalent to the following upstream change: https://sqlite.org/src/info/6c103aee6f146869 Original commit message: Change constant expressions to pre-computed constants, because apparently MSVC on ARM requires that. [forum:/forumpost/4feb1685cced0a8e|Forum thread 4feb1685cced0a8e]. FossilOrigin-Name: 6c103aee6f146869a3e0c48694592f2e4c6b57ecdb4450f46e762c38b4e686f1 PR-URL: #52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This commit introduces an experimental implementation of the Web Storage API using SQLite as the backing data store. PR-URL: nodejs#52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This is equivalent to the following upstream change: https://sqlite.org/src/info/6c103aee6f146869 Original commit message: Change constant expressions to pre-computed constants, because apparently MSVC on ARM requires that. [forum:/forumpost/4feb1685cced0a8e|Forum thread 4feb1685cced0a8e]. FossilOrigin-Name: 6c103aee6f146869a3e0c48694592f2e4c6b57ecdb4450f46e762c38b4e686f1 PR-URL: nodejs#52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This commit introduces an experimental implementation of the Web Storage API using SQLite as the backing data store. PR-URL: nodejs#52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

This is equivalent to the following upstream change: https://sqlite.org/src/info/6c103aee6f146869 Original commit message: Change constant expressions to pre-computed constants, because apparently MSVC on ARM requires that. [forum:/forumpost/4feb1685cced0a8e|Forum thread 4feb1685cced0a8e]. FossilOrigin-Name: 6c103aee6f146869a3e0c48694592f2e4c6b57ecdb4450f46e762c38b4e686f1 PR-URL: nodejs#52435 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ethan Arrowood <[email protected]>

Notable changes: deps,lib,src: * (SEMVER-MINOR) add experimental web storage (Colin Ihrig) #52435 doc: * move `node --run` stability to rc (Yagiz Nizipli) #53433 * mark WebSocket as stable (Matthew Aitken) #53352 * mark --heap-prof and related flags stable (Joyee Cheung) #53343 * mark --cpu-prof and related flags stable (Joyee Cheung) #53343 * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 lib: * (SEMVER-MINOR) add diagnostics_channel events to module loading (RafaelGSS) #44340 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: #53583

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Apr 9, 2024

cjihrig mentioned this pull request Apr 9, 2024

deps,doc,lib,src,test: add experimental web storage #50169

Closed

anonrig added the tsc-agenda Issues and PRs to discuss during the meetings of the TSC. label Apr 9, 2024

anonrig approved these changes Apr 10, 2024

View reviewed changes

anonrig added the notable-change PRs with changes that should be highlighted in changelogs. label Apr 10, 2024

mcollina reviewed Apr 10, 2024

View reviewed changes

lib/internal/webstorage.js Outdated Show resolved Hide resolved

ShogunPanda mentioned this pull request Apr 10, 2024

Node.js Technical Steering Committee (TSC) Meeting 2024-04-10 nodejs/TSC#1526

Closed

jasnell reviewed Apr 10, 2024

View reviewed changes

src/node_webstorage.h Outdated Show resolved Hide resolved

jasnell reviewed Apr 10, 2024

View reviewed changes

src/node_webstorage.cc Outdated Show resolved Hide resolved

jasnell reviewed Apr 10, 2024

View reviewed changes

src/node_webstorage.cc Outdated Show resolved Hide resolved

jasnell reviewed Apr 10, 2024

View reviewed changes

src/node_webstorage.cc Outdated Show resolved Hide resolved

GeoffreyBooth reviewed Apr 10, 2024

View reviewed changes

doc/api/globals.md Outdated Show resolved Hide resolved

doc/api/globals.md Outdated Show resolved Hide resolved

doc/api/globals.md Outdated Show resolved Hide resolved

jasnell removed the tsc-agenda Issues and PRs to discuss during the meetings of the TSC. label Apr 10, 2024

benjamingr reviewed Apr 11, 2024

View reviewed changes

jasnell approved these changes Jun 13, 2024

View reviewed changes

Ethan-Arrowood approved these changes Jun 13, 2024

View reviewed changes

H4ad reviewed Jun 13, 2024

View reviewed changes

mcollina approved these changes Jun 13, 2024

View reviewed changes

cjihrig added commit-queue Add this label to land a pull request using GitHub Actions. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. labels Jun 14, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 14, 2024

nodejs-github-bot closed this Jun 14, 2024

cjihrig deleted the webstorage-bak branch June 14, 2024 18:40

targos mentioned this pull request Jun 25, 2024

v22.4.0 release proposal #53583

Merged

Karliz24 mentioned this pull request Jul 9, 2024

## 2024-07-02, Version 22.4.0 (Current), @targos Karliz24/Marley-#16

Open

marco-ippolito added the baking-for-lts PRs that need to wait before landing in a LTS release. label Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps,lib,src: add experimental web storage #52435

deps,lib,src: add experimental web storage #52435

cjihrig commented Apr 9, 2024

nodejs-github-bot commented Apr 9, 2024

anonrig commented Apr 9, 2024

GeoffreyBooth commented Apr 9, 2024

ronag commented Apr 9, 2024

github-actions bot commented Apr 10, 2024

GeoffreyBooth commented Apr 10, 2024

cjihrig commented Apr 10, 2024

mcollina left a comment

jasnell commented Apr 10, 2024

jasnell Apr 10, 2024

cjihrig commented Apr 10, 2024

benjamingr left a comment

benjamingr commented Apr 11, 2024

mcollina commented Apr 12, 2024

cjihrig commented Apr 12, 2024

benjamingr commented Apr 12, 2024

mcollina commented Apr 12, 2024

nodejs-github-bot commented Jun 13, 2024 •

edited by cjihrig

Loading

jasnell left a comment

Ethan-Arrowood left a comment

H4ad Jun 13, 2024

mcollina left a comment

mcollina Jun 13, 2024

cjihrig Jun 14, 2024

mcollina Jun 13, 2024

cjihrig Jun 14, 2024

nodejs-github-bot commented Jun 14, 2024

		// max_size is 10MB. This can be made configurable in the future.
		" max_size INTEGER NOT NULL DEFAULT 10485760,"

deps,lib,src: add experimental web storage #52435

deps,lib,src: add experimental web storage #52435

Conversation

cjihrig commented Apr 9, 2024

nodejs-github-bot commented Apr 9, 2024

anonrig commented Apr 9, 2024

GeoffreyBooth commented Apr 9, 2024

ronag commented Apr 9, 2024

github-actions bot commented Apr 10, 2024

GeoffreyBooth commented Apr 10, 2024

cjihrig commented Apr 10, 2024

mcollina left a comment

Choose a reason for hiding this comment

jasnell commented Apr 10, 2024

jasnell Apr 10, 2024

Choose a reason for hiding this comment

cjihrig commented Apr 10, 2024

benjamingr left a comment

Choose a reason for hiding this comment

benjamingr commented Apr 11, 2024

mcollina commented Apr 12, 2024

cjihrig commented Apr 12, 2024

benjamingr commented Apr 12, 2024

mcollina commented Apr 12, 2024

nodejs-github-bot commented Jun 13, 2024 • edited by cjihrig Loading

jasnell left a comment

Choose a reason for hiding this comment

Ethan-Arrowood left a comment

Choose a reason for hiding this comment

H4ad Jun 13, 2024

Choose a reason for hiding this comment

mcollina left a comment

Choose a reason for hiding this comment

mcollina Jun 13, 2024

Choose a reason for hiding this comment

cjihrig Jun 14, 2024

Choose a reason for hiding this comment

mcollina Jun 13, 2024

Choose a reason for hiding this comment

cjihrig Jun 14, 2024

Choose a reason for hiding this comment

nodejs-github-bot commented Jun 14, 2024

nodejs-github-bot commented Jun 13, 2024 •

edited by cjihrig

Loading