Javascript Dev: Our naming conventions cause issues with require()

[joshuafontany]

Something I have been struggling with recently.

Our conventions of placing type: application/javascript tiddlers inside of plugins, and of using $:/plugins/$author$/$name$ as the prefix for most-if-not-all tiddlers contained within the plugin has caused some frustration when trying to use 3rd party javascript libraries.

TW's module system allows requiring modules with a relative path, i.e. "./name" and uses this to build a path from the current context/module. It then tries to load code from that tiddler. Most plugins use the tiddlywiki.files method to add $:/plugins/$author$/$name$/ to all their javascript tiddlers....

In node, a required(name) call will try to load from a tiddler of that name (& relative path), and if it fails default to loading from a node_module of that name (if available). But this causes issues when I need to use a library in node and browser contexts. I am currently running the "npm dist" command for all of the Yjs libraries I am experimenting with, and then I have to manually edit many files to change the require('somelibrary') calls to require('./somelibrary') .

I could simply name the Yjs library title: yjs and all of the required sub-libraries as title: lib0/observable, etc, etc. This would remove them from the "system namespace", and possibly spam code into a user's search results when using the wiki (unless filtered by !is[shadow]).

I was considering experimenting by changing boot.js to always concatenate the current module's name to the context (boot.js line 760) and checking for that first, then the raw moduleName, then if Node the available node_module.

I think this would really simplify how people can structure their plugins, so that even if they use the tiddlywiki.files method, they would not have to manually change any requires involved.

What do you all think?

[joshuafontany]

Hrm, that causes a bunch of issues... scratch that change....

[joshuafontany]

My primary issue is that I have 3rd party libraries that depend on other 3rd party libraries. Any time I require('./yjs.cjs'), there are multiple calls to

var observable = require('lib0/dist/observable.cjs');
var array = require('lib0/dist/array.cjs');
var math = require('lib0/dist/math.cjs');
var map = require('lib0/dist/map.cjs');

etc, etc.

Same thing with the WS (websockets) library. I can import these no problem, but if I use the standard tiddlywiki.files convention of appending the plugin name to all script files, all these references break...

[joshuafontany]

I have figured out one of the issues. I have gone to just naming the tiddlers after their base Node.js/NPM package name in the tiddlywiki.files setup.

So, instead of $:/plugins/commons/yjswebsockets/ws just 'ws'. This is so that the require calls inside ws should work. But TW uses it own internal require call that does some wierd path checking. All modules are loaded by checking the module name against the "rootModule" that loads it, and this uses the $tw.utils.resolvePath() function. This fuction expects the rootpath variable to end in a / if a directory, and always removes the last part of the path before appending the current relative path.

This breaks the Node.js style requires in a lot of 3rd party packages. @Jermolene any ideas here? can we adjust $tw.utils.resolvePath()?

[joshuafontany]

Reading through this as it seems related: #2817

I am trying to structure a package that will be able to use npm modules as sources, using well structured tiddlywiki.files settings to grab code from a local <node_modules> directory. Should each TW plugin have a dependencies listing in it's package.json file? I could then npm install from a terminal within the plugin folder to have npm install the built version of various packages (I am manually building and copy/pasting into the plugin folder, and that is not sustainable)...

[joshuafontany]

I'm learning more about node modules, how package.json works and other things that are helping with this. I may end up making my realtime multiplayer server-code a stand-along node package that requires TW as a dependency.

[linonetwo]

I think you should use rollup or esbuild or webpack, and use import statement. Then all import will only happened at buold time, and wont affect tw runtime.

[joshuafontany]

This discussion can be "closed". I actually learned a lot, and have managed to implement installing TW plugins as NPM modules, with their own sub-folder that has a package.json file, which can be triggered to install the dependencies that I need to bundle up into TW plugin-modules. Then, I can leave all the node-only dependencies as things to install from the command line.

[saqimtiaz]

This is my approach as well for TW plugins with npm dependencies.

[pmario]

@joshuafontany ... You can close the issue too