Is there an advantage of using "$param$" instead of <<__param__>> and "$(nonlocalvar)$" instead of <<nonlocalvar>> ?

[nilslindemann]

Hi people, I have a question.

Macro Definitions in WikiText documents four types of variables: <<__param__>>, "$param$", <<nonlocalvar>>, and "$(nonlocalvar)$".

It seems <<__param__>> should be preferred over "$param$", as it avoids some bugs. Otherwise, it has the same effect. <<nonlocalvar>> has the same effect as "$(nonlocalvar)$".

• Would this be right?
• If so, why are there four types? Is it due to backwards compatibility? Did "$param$" and "$(nonlocalvar)$" exist before <<__param__>> and <<nonlocalvar>>?

Greetings, Nils

[nilslindemann]

I just see, they seem to be substituted at different points in time. $param$ before the macrobody is parsed, <<__param__>> anyhow, while it gets parsed.

[nilslindemann]

I am starting to grasp it

\define weirdcall(param)
<$macrocall $name="show" param=<<__param__>>/>

<$macrocall $name="show" param=$param$/>

\end

\define show(param param2)
|$param$|$param2$|
\end

<<weirdcall '"bar" param2="baz"'>>

I am not sure if I like this.

[kookma]

I am not sure if I like this.

I do not like it too! I am very much in favor of simple and concise syntax! and I regret to use tricky rules and syntax!

In my own workflow, except where I need a text substitution, I always use <<__para__>> It may be amazing to know you can use this form of parameter in nested macros!

\define main(par)
Hello $par$<br>
Call nested macro: <<nested>>
\end

\define nested()
This is the same parameter <<__par__>>.
\end


<<main "Nils">>

The result is:

Hello Nils
Call nested macro: This is the same parameter Nils.

[nilslindemann]

I do not like it too! I am very much in favor of simple and concise syntax! and I regret to use tricky rules and syntax!

I am currently working on a Parser based on Peggy.js (which is why I asked this question), which has a quite simple syntax. It will take a while until this is done. An example of what already works:

-- single line comment

(-
	multiline
	(- chained -)
	comment
-)

"Hello World, "
	'just ' 
		'text. '
		'Whitespace outside of strings is irrelevant.'

hr

self-closing-element br img
self-closing-element [with='attribs']
element-with-closing-tag()
element-with-closing-tag [with="attribs"] (
	and (
		more child elements 'and ' "some more text"
	)
)

hr

-- Thats a widget. Same syntax as elements, but name is written in BIG LETTERS
BUTTON [normal="attribute" DOLLAR-ATTRIBUTE='value'] (
	"Hello Button"
)

hr

TEXT [text=tw/themes/tiddlywiki/white/metrics/sidebarbreakpoint] -- indirect text reference attribute value

hr

-- currently working on this (I go through the core files and look
-- what is there and then I implement it in my syntax).
BUTTON [class="tc-btn-invisible"] (
	WIKIFY [
		name = "site-title"
		text = tw/config/SaveWikiButton/Filename
	] (
		ACTION-SENDMESSAGE [
			MESSAGE = "tm-download-file"
			PARAM = tw/config/SaveWikiButton/Template
			(-
			-- Macro call syntax, TBD
			filename = !site-title[foo="bar" "baz"]
			-)
		]
	)

	(-
	-- tiddler widget with transclusion child widget, TBD
	tw/core/images/download-button
	-)
	" Save snapshot for offline use"
)

The above code generates a valid parse tree and the proper output.

Edit: This is exactly the generated Html (forgot that I already pimped the Html output):

Hello World, just text. Whitespace outside of strings is irrelevant.<hr><self-closing-element></self-closing-element><br><img><self-closing-element with="attribs"></self-closing-element><element-with-closing-tag></element-with-closing-tag><element-with-closing-tag with="attribs"><and><more></more><child></child><elements></elements>and some more text</and></element-with-closing-tag><hr><button class="">Hello Button</button><hr>960px<hr><button class="tc-btn-invisible"> Save snapshot for offline use</button>

As you see, the Html is maximal compressed.

The parser has error reporting, but it needs to be polished. For example, if I remove the first dash in the first line, the preview output window shows a pre containing:

Line 1, column 1:

- single line comment
^

Expected "'", "(-", "--", "\"", [ ,\t,\r,\n], [A-Z,a-z], or end of input but "-" found.

---

In my own workflow, except where I need a text substitution, I always use <<__para__>> It may be amazing to know you can use this form of parameter in nested macros!

\define main(par)
Hello $par$<br>
Call nested macro: <<nested>>
\end

\define nested()
This is the same parameter <<__par__>>.
\end


<<main "Nils">>

The result is:

Hello Nils
Call nested macro: This is the same parameter Nils.

Interesting. That shows that consecutive macro definitions are just nested Set widgets. This will be interesting to implement in my syntax.

[Jermolene]

I am currently working on a Parser based on Peggy.js (which is why I asked this question), which has a quite simple syntax. It will take a while until this is done. An example of what already works:

Very exciting!

I hope in the future to refactor the core such that the parser is written in wikitext and filters. Things like Peggy.js show that all that is required is a sufficiently flexible programming language to concisely express the parse rules, and wikitext/filters are getting steadily more capable every year.

[nilslindemann]

I hope in the future to refactor the core such that the parser is written in wikitext and filters.

Things like Peggy.js show that all that is required is a sufficiently flexible programming language to concisely express the parse rules, and wikitext/filters are getting steadily more capable every year.

That sounds like an ambitious goal. I have no idea how you would do this. Didn't know that filters are that powerful. So, we are going towards Turing-completeness :-)

Yes, indeed. It is all about languages. A well-designed language and a lovely editor, that's all one needs.

[pmario]

If so, why are there four types?

Evolution. <<__macro-param__>> was added last, to avoid the problem with """parms with nested """tripple""" double quotes""", which doesn't work ...

Is it due to backwards compatibility?

Partially yes, <<__param__>> is preferable, but there are some functions which are easier with $param$ eg: text concatenation.

\define concat(a b) $a$/$b$

If the macro is called with <<concat "$:/temp" "test">> the core will basically do a "search / replace" with the params and the macro will be substituted as a single text string...

The same is true for \define conat-var() $(var1)$/-$(var2)$ if var1 and var2 are text strings. ...

Did "$param$" and "$(nonlocalvar)$" exist before <<param>> and <>?

As I wrote <<__param__>> was added last. The rest was created at the same time.

Technically macro definition and variables are almost the same thing. eg: \define test() some text will internally be translated to <$set name=test value="some text"> with a scope of the whole tiddler and an internal flag: isMacroDefinition.

Hope that helps.

[Jermolene]

Hi @nilslindemann

The history was roughly in the following order.

Macros were first implemented with parameters based on text substitution (ie, the $param$ syntax). The implementation was and is very simple: before using the value of the macro a search and replace is performed for each specified $param$ with the corresponding value (the value specified when invoking the macro, or the default value if the parameter was not specified in the invocation).

The weakness of that approach became clear very quickly. For a long time the core had problems with tiddler titles that ended in double quotes because of situations like this:

\define mymacro(param)
<$macrocall $name="anothermacro" something="""$param$"""/>
\end

There are situations where text substitution is a good mental model: assembling URLs or tiddler titles from fragments are good examples.

Back to the history: next we implemented the $(var)$ syntax which gave a tiny bit more flexibility to the text substitution approach.

Much, much later, about 2018 I believe, we added the __param__ syntax for accessing the parameters with the same semantics as variables. It was a breakthrough that finally enabled us to fix the last few places where the core would break with special characters in titles

It seems <<__param__>> should be preferred over "$param$", as it avoids some bugs. Otherwise, it has the same effect. <<nonlocalvar>> has the same effect as "$(nonlocalvar)$".

• Would this be right?

Broadly. It's better to think of macros in two different ways:

• For macros whose output is not going to be wikified (eg because it is intended to be used as an attribute value attr=<<var>>), then only text substitution is available
• For macros that are intended to be wikified, it generally makes sense to use the __param__ syntax

• If so, why are there four types? Is it due to backwards compatibility? Did "$param$" and "$(nonlocalvar)$" exist before <<__param__>> and <<nonlocalvar>>?

See above!

Perhaps some of the above might be usefully folded into the docs? @btheado – might you kindly be able to have a look at this?

[nilslindemann]

Thank you, @Jermolene and @pmario, for these helpful answers. I throw a coin and mark Jermolenes answer :-)

[btheado]

Perhaps some of the above might be usefully folded into the docs? @btheado – might you kindly be able to have a look at this?

Sure, I can take a look.

[nilslindemann]

@btheado

Here are some notes which may help to resolve your questions. Maybe you already know all that, as you already mentioned a few things, which I recall here. In that case, it is hopefully appealing for other readers.

Parsing starts in wiki.js → parseText() – when this function gets called, and its first parameter resolves to "text/vnd.tiddlywiki" (the WikiText mime type), then its second parameter is a string containing wikitext. That string then gets converted to a parse tree. The parsing is flat, it just parses the current "wikitext level"; contained macros are parsed, but not called. The bodies of macro definitions are still raw strings. One can see that when inspecting the parse tree for e.g \define m(param) $param$ <<__param__>> $(outer)$ <<outer>> in the tiddler edit preview.

When parseText() gets called from a module inside core/modules/parsers, we are in the parse tree generation phase, and when it gets called from a module inside core/modules/widgets, we are in the widget tree generation/rendering phase. So, again, not all things are expanded during the parse tree generation phase, especially not macro definition bodies.

parseText() is called in very few locations in the TiddlyWiki codebase (if we ignore test-widget.js). The locations are:

(In the following, '(P)' means 'parse tree generation phase' and '(W)' means 'widget tree generation/rendering phase')

• in wiki.js → parseTiddler() – Here, parseText() parses the text of a tiddler. (triggers P for this tiddler)
• in macrocall.js → execute() – Here, it parses the body of a macro definition. (W) So, the macro body is parsed when the macro calling widget is executed, not when the macro definition is parsed. Short after, in this function, also __param__ variables are created, these are the variables which the <<__param__>> syntax uses (so, this syntax is just an inner macro call). At the end of the function, the generation of child widgets is triggered, by calling widget.js → makeChildWidgets(), passing it the modified parse tree, which may result in further parsing of bodies of macros which are called from the current macro body – recursive expansion. Or, in other words, during the process of converting the nodes of a parse tree to widget nodes, we trigger creations of sub parse trees for macro bodies, which are merged as childs of the current parse node, and then converted to widget nodes on the child levels.
• in wikify.js → execute() – Here, it parses the content of the "text" attribute of the wikify widget. (W)
• in widget.js → invokeActionString() – Here, it parses wikitext strings containing action widgets, which can be values of an action attribute, or normal wikitext outside of attributes. (W)
• in typedblock.js → parse() – Here, it parses the body of typed blocks. (P)
• in list.js → getEmptyMessage() – Here, it parses the content of the emptyMessage attribute of the list widget. (W)
• in wiki.js → parseTextReference() – Here, it parses the text of a transcluded Tiddler. (W?)
• in wiki.js → renderText() – Here, a string is parsed according to its mime type, and then rendered to a specified output type - html or text. renderText() is only called during W, namely in link.js, macrocall.js and in four functions in view.js, which are all called from view.js → execute(). (all W?)
• in backlinks-index.js → _getLinks() – Here, the parsing is done to afterwards extract the backlinks from the parse tree (W?). This is the only time when we are not inside core/modules/parsers or core/modules/widgets or inside wiki.js.

A macro call as attribute value (myattrib = <<mymacro>>) (but not myattrib = "<<mymacro>>") is parsed in parseutils.js → parseAttribute(). (P)

That macro call then gets executed in widget.js → computeAttribute(). Except from let.js, which does special stuff, computeAttribute() gets only called from its plural version computeAttributes(). computeAttributes() gets called from every module under core/modules/widgets (except log.js and raw.js). (W)

computeAttribute() calls getVariableInfo() and here happens the replacement of $param$ variables and, if variable.isMacroDefinition, the replacement of outer $(var)$ variables (via call to substituteVariableReferences()).

That would be my current knowledge of the mechanism. Anybody correct me, when things are wrong.

[btheado]

@nilslindemann, this is fantastic. Thanks for sharing.

One question I didn't see answered in the above (maybe I missed it?) is which parsing rules will run further parsing. I think searching for parseInlineRun answers that question:

$ find ./core/modules/parsers/wikiparser/rules -type f | xargs grep -l parseInlineRun
./core/modules/parsers/wikiparser/rules/hardlinebreaks.js
./core/modules/parsers/wikiparser/rules/styleinline.js
./core/modules/parsers/wikiparser/rules/emphasis/subscript.js
./core/modules/parsers/wikiparser/rules/emphasis/italic.js
./core/modules/parsers/wikiparser/rules/emphasis/bold.js
./core/modules/parsers/wikiparser/rules/emphasis/strikethrough.js
./core/modules/parsers/wikiparser/rules/emphasis/superscript.js
./core/modules/parsers/wikiparser/rules/emphasis/underscore.js
./core/modules/parsers/wikiparser/rules/quoteblock.js
./core/modules/parsers/wikiparser/rules/list.js
./core/modules/parsers/wikiparser/rules/heading.js
./core/modules/parsers/wikiparser/rules/html.js
./core/modules/parsers/wikiparser/rules/table.js

Another question: how do the variable references and transclusions in filter runs work? I didn't look, but I'm guessing the filter parsing code has calls to getVariableInfo and parseTextReference.

[nilslindemann]

@btheado

One question I didn't see answered in the above (maybe I missed it?) is which parsing rules will run further parsing. I think searching for parseInlineRun answers that question: (...)

parseInlineRun() only means that the following text will be parsed with inline rules. Its counterpart is parseBlocks() which triggers parsing with block rules. See my tutorial How The Wiki Parser Works, which explains inline and block rules (near the start).

But these functions do not trigger a new parsing process with a different parser, or another wikitext parser. The only parse rule, which triggers a new parsing process at this moment – aka, while generating the parse tree – is the typedblock rule. It does so by calling parseText(), and it does so, because the content of the typed block usually has a different mime type than the surrounding wikitext, and therefore needs to get parsed with a different parser.

All other rules, including those for macro definitions, macro calls (1, 2), transclusions (1, 2), filter transclusions (1, 2), imports, and images do not yet resolve what they call/transclude/import/reference. They just prepare things (like for example parsing the macro name and parameters, or the filter expression, or a referencing URL) for the actual resolving, which happens later, when the parse tree is converted to a widget tree (and when that generates the DOM).

Another question: how do the variable references and transclusions in filter runs work? I didn't look, but I'm guessing the filter parsing code has calls to getVariableInfo and parseTextReference.

It takes a few curves, but we finally end up there.

First, it works like with macro bodies and macro calls. During parse tree generation, the whole filter expression, including variables and transclusions, gets parsed into a string, but stays unexpanded.

Then later, when the parse tree gets converted to a widget tree, we have a call stack like this:

• A widget calls computeAttributes()
• That calls computeAttribute()
• That calls filterTiddlers()
• That calls compileFilter()
• That calls parseFilter()
• That calls parseFilterOperation()
• That, if we have a variable reference or a transclusion, sets operand.variable or operand.indirect to true
• Later, back in compileFilter(), if for example operand.variable is true, calls parseutils.js → parseFilterVariable() with the content of the variable (without '<' and '>')
• That parses the variable name and maybe the parameters, if there are any (I am not going deeper here) and returns an object containing this info
• (Back in compileFilter()) calls getVariable() with the variable name and parameters, taken from the returned info object. I assume, it is this getVariable(), (is this correct @Jermolene? There are other definitions of this function)
• That, finally, calls getVariableInfo().

I guess parseTextReference() is also reached anyhow via this path, but I did not check.

[btheado]

parseInlineRun() only means that the following text will be parsed with inline rules. Its counterpart is parseBlocks() which triggers parsing with block rules. See my tutorial How The Wiki Parser Works, which explains inline and block rules (near the start).

@nilslindemann, thanks. I hadn't noticed the parseBlock call.

But these functions do not trigger a new parsing process with a different parser, or another wikitext parser.

Sure and the intent behind my question is from a user perspective how to know which syntax will have its enclosed text parsed. Identifying all the places where a new parsing process is triggered is not enough. In addition all the parseInlineRun and parseBlocks calls must be identified.

About variable references and transclusions in filter runs:

It takes a few curves, but we finally end up there.

Thanks for all those details. Incredibly useful.

[nilslindemann]

Sure and the intent behind my question is from a user perspective how to know which syntax will have its enclosed text parsed.

Yes, I completely misunderstood your intention here. Weird.