LitREPL is a command-line tool that brings together the benefits of literate programming and read-eval-print-loop coding. LitREPL comes bundled with an interface Vim plugin, integrating it into the editor.
- Document formats
Markdown (Example [MD]) | LaTeX (Examples [TEX][PDF]) - Interpreters
Python | IPython | Aicli - Editor integration
Vim (plugin included)
- POSIX-compatible OS, typically a Linux. The tool relies on POSIX pipes and depends on certain shell commands.
- Python packages:
lark-parser
,psutil
(Required). - Command line tools:
GNU socat
(Optional)
- Installation
- Usage
- Development
- Gallery
- Technical details
- Limitations
- Related projects
- Third-party issues
This repository includes the Litrepl tool in Python and an interface Vim plugin.
The Python part might be installed with pip install .
run from the project
folder. The Vim part requires hand-copying ./vim/plugin/litrepl.vim
to the
~/.vim
config folder or using any Vim plugin manager, e.g. Vim-Plug.
Release versions from pypi and vim.org
pip install litrepl
- Download the
litrepl.vim
from the vim.org script page and put it into your~/.vim/plugin
folder.
Latest versions using pip-install and Vim-Plug
- Install the
litrepl
Python package with pip:$ pip install --user git+https://github.com/sergei-mironov/litrepl $ litrepl --version
- Install the Vim plugin by adding the following line between the
plug#begin
andplug#end
lines of your.vimrc
file:Note:Plug 'https://github.com/sergei-mironov/litrepl' , { 'rtp': 'vim' }
rtp
sets the custom vim-plugin source directory of the plugin.
Development environment using Nix
The repository also includes a set of Nix expressions that automate installation on Nix-enabled systems.
Nix supports
configurable Vim expressions.
To enable the Litrepl plugin, add the vim-litrepl.vim-litrepl-release
to the
list of Vim plugins and put this version of vim into your Nix profile. Litrepl
and its dependencies will be installed automatically.
{ litrepl }:
...
vim_configurable.customize {
name = "vim";
vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; {
start = [
...
litrepl.vim-litrepl-release
...
];
};
}
Note: vim-demo
expression from the default.nix provides
an example Vim configuration. Use nix build '.#vim-demo'
to build it and then
./result/bin/vim-demo
to run the editor.
See the Development section for more details.
The tool sends verbatim sections from a document to external interpreters, receiving the evaluated results in return. Litrepl currently supports two flavors of Python and the Aicli interpreter.
Litrepl recognises verbatim code sections followed by zero or more result
sections. In Markdown documents, the code is any triple-quoted section labeled
as python
. The result is any triple-quoted result
section. In LaTeX
documents, sections are marked with \begin{python}\end{python}
and
\begin{result}\end{result}
environments correspondingly.
litrepl eval-sections
is the main command evaluating the formatted document.
To run the evaluation, send the file to the input of the shell command.
For example:
$ cat file.md
``` python
print('Hello Markdown!')
```
``` result
```
$ cat file.md | litrepl eval-sections
.. would produce a Markdown document containing the properly filled result section.
``` python
print('Hello Markdown!')
```
``` result
Hello Markdown!
```
Below we also show what the relevant LaTeX part would look like:
\begin{python}
print('Hello LaTeX!')
\end{python}
\begin{result}
Hello LaTeX!
\end{result}
- The equivalent Vim command is
:LEval
.:LEval
accepts optional argument denoting the range:all
,above
(the cursor),below
(the cursor), section number, etc. - Litrepl expects Markdown formatting by default. Add
--filetype=tex
for Tex documents. Vim plugin does this automatically based on thefiletype
variable. - Both command-line and Vim versions of the command accept code section indices. Everything is evaluated by default.
- LaTeX documents need a preamble introducing python/result tags to the Tex processor. For details, see:
- For more information on Markdown formatting, see
litrepl start
, litrepl stop
and litrepl restart
manage the interpreter
sessions. The commands also accepts the type of the interpreter to operation on.
IPython interpreter is assumed by default.
litrepl status
queries the information about the interpreters running in
the background. The command reveals the process PID and the command-line arguments.
$ litrepl status
# Format:
# TYP PID EXITCODE CMD
python 3900919 - python3 -m IPython --config=/tmp/litrepl_1000_a2732d/python/litrepl_ipython_config.py --colors=NoColor -i
ai 3904696 - aicli --readline-prompt=
- The interpreters are associates with the directory they were started in.
- The corresponding Vim commands are
:LStart
,:LStop
,:LRestart
and:LStatus
Litrepl can produce output document earlier than the interpreter reports the completion. In cases where the evaluation takes longer to finish, LitREPL will leave a marker that allows it to pick up where it left off on subsequent executions.
litrepl --timeout=3.5 eval-sections
changes the reading timeout from the
default infinity the specified number of seconds. The output would be:
``` python
from tqdm import tqdm
from time import sleep
for i in tqdm(range(10)):
sleep(1)
```
``` result
30%|███ | 3/10 [00:03<00:07, 1.00s/it]
[BG:/tmp/nix-shell.vijcH0/litrepl_1000_a2732d/python/litrepl_eval_5503542553591491252.txt]
```
When re-executing this document, LitREPL will resume the reading. Once the evaluation is complete, it will remove the continuation marker from the output section.
litrepl interrupt
will send interrupt signal to the interpreter so it return
the control earlier (with an exception).
- The corresponding Vim commands are
:LEvalAsyn
(with the timeout set to 0.5 seconds by default) and:LInterrupt
. - Vim plugin defines
:LEvalMon
command that enables repeated code evaluation without any delay. Interrupting this process using Ctrl+C will cause Litrepl to return control to the editor while leaving the evaluation in the background.
litrepl repl
"manually" attaches to the interpreter session allowing us to
examine its internal state:
$ litrepl repl
Opening the interpreter terminal (NO PROMPTS, USE `Ctrl+D` TO DETACH)
W = 'Hello from repl'
^D
- Python prompts are disabled internally, no
>>>
symbols are going to appear. - The corresponding Vim command is
:LTerm
litrepl eval-code
might be used to pipe the code through the interpreter. The
W
variable now resides in memory so we can query it as we would do in a
regular IPython session.
$ echo 'W' | litrepl eval-code
'Hello from repl'
Litrepl experimentally supports
Aicli allowing users to
query local LLMs. In order to try it, install the interpreter and use ai
as
the name for code sections. For low-speed models it would be convenient to use
:LEvalMon
command for evaluation.
``` ai
/model "~/.local/share/nomic.ai/GPT4All/Meta-Llama-3-8B-Instruct.Q4_0.gguf"
Hi chat! What is your name?
```
``` result
I'm LLaMA, a large language model trained by Meta AI. I'm here to help answer
any questions you might have and provide information on a wide range of topics.
How can I assist you today?
```
For AI sections, Litrepl can paste text from other sections of the document in place of reference markers. The markers have the following format:
>>RX<<
, whereX
is a number - references a section numberX
(starting from zero).^^RX^^
, whereX
is a number - references the sectionX
times above the current one.vvRXvv
, whereX
is a number - references the sectionX
times below the current one.
``` ai
AI, what do you think the following text means?
^^R1^^
```
``` result
Another interesting piece of text!
This is an example of a chatbot introduction or "hello message." It appears to
be written in a friendly, approachable tone, with the goal of establishing a
connection with users.
```
Where
T
type of the interpreter:python
orai
(some commands also acceptall
)F
Path to a Markdown or LaTeX fileP
Path to a Python scriptN
number of code section to evaluate, starting from 0.L:C
denotes line:column of the cursor.
D
type of the document:tex
ormarkdown
(the default).B
interpreter binary to use, defaults toauto
which guesses the best one.FLOAT
should be formatted as1
or1.1
orinf
. Note: command line argument also accepts a pair of timeouts.
usage: litrepl [-h] [-v] [--filetype STR] [--python-interpreter EXE]
[--ai-interpreter EXE] [--timeout F[,F]] [--propagate-sigint]
[-d INT] [--verbose] [--auxdir DIR] [-C DIR]
[--pending-exit INT] [--exception-exit INT] [--foreground]
[--map-cursor LINE:COL:FILE] [--result-textwidth NUM]
{start,stop,restart,status,parse,parse-print,eval-sections,eval-code,repl,interrupt}
...
positional arguments:
{start,stop,restart,status,parse,parse-print,eval-sections,eval-code,repl,interrupt}
Commands to execute
start Start the background interpreter.
stop Stop the background interpreters.
restart Restart the background interpreters.
status Print background interpreter's status.
parse Parse the input file (diagnostics).
parse-print Parse and print the input file back
(diagnostics, no changes are made).
eval-sections Parse stdin, evaluate the sepcified sections (by
default - all available sections), print the
resulting file to stdout.
eval-code Evaluate the code snippet.
repl Connect to the background terminal using GNU
socat.
interrupt Send SIGINT to the background interpreter.
options:
-h, --help show this help message and exit
-v, --version Print version.
--filetype STR Specify the type of input formatting
(markdown|[la]tex).
--python-interpreter EXE Python interpreter to use (python|ipython|auto)
--ai-interpreter EXE AI interpreter to use (aicli|auto).
--timeout F[,F] Timeouts for initial evaluation and for pending
checks, in seconds. If the latter is omitted, it
is considered to be equal to the former one.
--propagate-sigint If set, litrepl will relay SIGINT signals to the
running interpreter. Otherwise it just handles
them by itself.
-d INT, --debug INT Enable (a lot of) debug messages.
--verbose Be more verbose (used in status).
--auxdir DIR Directory to store auxilary session files. By
default, the name of the aux. dir is derived
from the name of the current dir.
-C DIR, --workdir DIR Directory to run from, the current directory is
used by default.
--pending-exit INT Return this error code if whenever section hits
timeout.
--exception-exit INT Return this error code at exception, if any.
Note: it takes affect only for newly-started
interpreters.
--foreground Start a separate session and stop it when the
evaluation is done.
--map-cursor LINE:COL:FILE Calculate the new position of a cursor at
LINE:COL and write it to FILE.
--result-textwidth NUM Wrap result lines longer than NUM symbols.
To evaluate code section in a document:
$ cat doc/example.md | litrepl eval-sections >output.md
To evaluate a Python script:
$ cat script.py | litrepl eval-code
Note that both commands above share the same background interpreter session.
For batch processing of documents, it may be necessary to have an on-demand interpreter session available, which would exist solely for the duration of the evaluation process.
$ cat document.md.in
``` python
raise Exception("D'oh!")
```
$ cat document.md.in | litrepl --foreground --exception-exit=200 eval-sections >document.md
$ echo $?
200
Here, the --foreground
argument tells Litrepl to run a new interpreter session
and then stop it before exiting, --exception-exit=200
sets the exit code
returned in the case of unhandled exceptions.
A typical Makefile recipe updating README.md might look as follows:
.stamp_readme: $(SRC) Makefile
cp README.md _README.md.in
cat _README.md.in | \
litrepl --foreground --exception-exit=100 --result-textwidth=100 \
eval-sections >README.md
touch $@
.PHONY: readme
readme: .stamp_readme
Where $(SRC)
is supposed to contain valuable source filenames.
The plugin does not define any keybindings, but users could do it by themselves, for example:
nnoremap <F5> :LEval<CR>
nnoremap <F6> :LEvalAsync<CR>
Below we define :C
command inserting new sections.
command! -buffer -nargs=0 C normal 0i``` python<CR>```<CR><CR>``` result<CR>```<Esc>4k
We define the :LR
command running first section after the restart.
command! -nargs=0 LR LRestart | LEval 0
Thanks to IPython features, we can use exclamation to run shell commands directly from Python code sections.
``` python
!cowsay "Hello, Litrepl!"
```
``` result
_________________
< Hello, Litrepl! >
-----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
```
This project uses Nix as a primary development framework. flake.nix handles the source-level Nix dependencies while the default.nix defines the common build targets including Pypi and Vim packages, demo Vim configurations, development shells, etc.
The default development shell is defined in the ./default.nix
as a Nix
expression named shell
which is the default name for development shells.
Running
$ nix develop
will ask Nix to install the development dependencies and open the shell.
Another shell which might be useful is shell-screencast
. This would build the
full set of Litrepl tools and makes sure that the screencasting software is
available. To enter it, specify its Nix-flake path as follows:
$ nix develop '.#shell-screencast'
To build individual Nix expressions, run nix build '.#NAME'
passing the
name of Nix-expression to build. If succeeded, Nix publishes the last build'
results under the ./result
symlink.
$ nix build '.#vim-demo'
$ ./result/bin/vim-demo # Run the pre-configured demo instance of Vim
The list of Nix build targets includes:
litrepl-release
- Litrepl script and Python liblitrepl-release-pypi
- Litrepl script and Python libvim-litrepl-release
- Vim with locally built litrepl pluginvim-litrepl-release-pypi
- Vim with litrepl plugin built from PYPIvim-test
- A minimalistic Vim with a single litrepl pluginvim-demo
- Vim configured to use litrepl suitable for recording screencastsvim-plug
- Vim configured to use litrepl via the Plug managershell-dev
- The development shellshell-screencast
- The shell for recording demonstrations, includesvim-demo
.
See Nix flakes manual for other Nix-related details.
The top-level Makefile encodes common development workflows:
[LitREPL-develop] $ make help
LitREPL is a macroprocessing Python library for Litrate programming and code execution
Build targets:
help: Print help
test: Run the test script (./sh/test.sh)
wheel: Build Python wheel (the DEFAULT target)
version: Print the version
upload: Upload wheel to Pypi.org (./_token.pypi is required)
Basic usage
Note: the below screencasts are outdated.
Using LitREPL in combination with the Vimtex plugin to edit Latex documents on the fly.
demo-vimtex.mp4
Asynchronous code execution
The following events should normally happen after users type the :LitEval1
command:
- On the first run, LitREPL starts the Python interpreter in the background. Its standard input and output are redirected into UNIX pipes in the current directory.
- LitREPL runs the whole document through the express Markdown/Latex parser determining the start/stop positions of code and result sections. The cursor position is also available and the code from the right code section can reach the interpreter.
- The process which reads the interpreter's response is forked out of the main LitREPL process. The output goes to the temporary file.
- If the interpreter reports the completion quickly, the output is pasted to the resulting document immediately. Otherwise, the temporary results are pasted.
- Re-evaluating sections with temporary results causes LitREPL to update these results.
- Formatting: Nested code sections are not supported.
Formatting: Special symbols in the Python output could invalidate the document.- Interpreter: Extra newline is required after Python function definitions.
- Interpreter: Stdout and stderr are joined together.
Interpreter: Evaluation of a code section locks the editor.- Interpreter: Tweaking
os.ps1
/os.ps2
prompts of the Python interpreter could break the session. Interpreter: No asynchronous code execution.Interpreter: Background Python interpreter couldn't be interrupted
Edititng:
- https://github.com/lervag/vimtex (LaTeX editing, LaTeX preview)
- https://github.com/shime/vim-livedown (Markdown preview)
- https://github.com/preservim/vim-markdown (Markdown editing)
Code execution:
- Vim-medieval https://github.com/gpanders/vim-medieval
- Evaluates Markdown code sections
- Pyluatex https://www.ctan.org/pkg/pyluatex
- Magma-nvim https://github.com/dccsillag/magma-nvim
- Codi https://github.com/metakirby5/codi.vim
- Pythontex https://github.com/gpoore/pythontex
- Evaluates Latex code sections
- Codebraid https://github.com/gpoore/codebraid
- Vim-ipython-cell https://github.com/hanschen/vim-ipython-cell
- Vim-ipython https://github.com/ivanov/vim-ipython
- Jupytext https://github.com/goerz/jupytext.vim
- Alternative? https://github.com/mwouts/jupytext
- Ipython-vimception https://github.com/ivanov/ipython-vimception
Useful Vim plugins:
- https://github.com/sergei-grechanik/vim-terminal-images (Graphics in vim terminals)
Useful tools:
- Vim-plug junegunn/vim-plug#1010 (comment)
- Pandoc jgm/pandoc#8598
- Jupytext mwouts/jupytext#220 (comment)
- Vim-LSC natebosch/vim-lsc#469
- Bad PDF fonts in Firefox