Benchmarks.md

Benchmarking pazi

Pazi is benchmarked against other autojump utilities. Currently, it's benchmarked against fasd, autojump, and z.

The benchmarks are maintained under the 'tests' directory in this repository.

Benchmark methodology

Test system

These benchmarks were all run on a machine with an i5-4670K and a Samsung 850 EVO SSD. The following versions of relevant software were installed:

software	version
bash	GNU bash, version 4.4.12
zsh	zsh 5.5.1
autojump	commit 6a529f4
fasd	commit 90b531a
pazi	commit e56d571
z	commit ea5ec78
rust	1.29.0-nightly

Procedure

Each benchmark was run once to warm up, and then once to get the value used. Benchmarks were run from the repository via:

$ cargo build --release && cd tests
$ cargo bench --features=nightly,cgroups2 <benchmark name>

Analysis is based on my understanding of the benchmarks and software under analysis.

Benchmarks

The below is a list of benchmarks which I think can be meaningfully interpreted.

cd bench

The cd_bench_normal benchmark tries to evaluate the cost of the autojump program maintaining its frecency list during regular usage. The benchmark accesses directories, causing the autojump program under test to add the directory to its history.

This benchmark also includes a benchmark for performing the same operations with no autojump program installed. This provides a baseline for how slow the shell+vte+test harness are without any additional autojump accounting.

Without further adieu, the results:

Autojump Program	Shell	cd/s	ms/cd
None	bash	10870	.092
None	zsh	32258	.031
autojump	bash	60	16.771
autojump	zsh	56	17.765
fasd	bash	66	15.218
fasd	zsh	73	13.770
pazi	bash	599	1.669
pazi	zsh	702	1.423
z	bash	754	1.327

Using an autojump program clearly has a significant impact on a shell's performance. However, with pazi that impact is clearly very small.

Using pazi introduces around 1.5 milliseconds of delay between a command completing and a new shell prompt being available. This 1.5 milliseconds will often not be noticible, especially since 60hz displays are only refreshing every 16ms.

The fastest autojumper in this specific benchmark is z, which is neck-and-neck with pazi. Unfortunately, under zsh, z exhibits a serious bug often enough that it cannot be benchmarked in this method.

This test does not show all of the differences between the impact these programs have though. In addition to the visible "time to next command prompt" this benchmark shows, there's the "time to directory processed". This time is the same for pazi and fasd. However, autojump and z both fork work into the background during cds. Even if a new shell prompt is visible in a millisecond with z, it continues to consume resources for a short while more.

The next benchmark tries to measure that difference as well.

cd sync bench

The cd_bench benchmark above, simply measures how long it takes a new shell prompt to be available for use after running a cd command. This does not capture any work an autojumper might do in the background. The cd_sync benchmark is meant to measure that as well. It operates by running cd, and then waiting for a shell to be visible and for no additional background processes to be running under the shell.

This captures how long it takes autojumpers that fork into the background to quit using resources.

The results for this benchmark are:

Autojump Program	Shell	cd/s	ms/cd
None	bash	10000	.100
None	zsh	32258	.031
autojump	bash	19	52.293
autojump	zsh	19	52.123
fasd	bash	66	15.237
fasd	zsh	73	13.728
pazi	bash	644	1.552
pazi	zsh	733	1.364
z	bash	262	3.813
z	zsh	319	3.133

For this benchmark, we can run z under both bash and zsh since the bugs which prevented it from working in the previous cd benchmark happen to be caused by multiple processes forking and modifying its data file in parallel.

This benchmark shows that, when taking into account processes forking into the background, pazi is clearly the fastest. Vroom vroom.

jump bench

The jump_bench benchmark attempts to measure how long it takes from typing z <directory> to having your shell's CWD changed to that directory.

Notably, this benchmark is performed with only a minimal number of entries in the benchmark program's database, meaning the autojump software need not load much data from disk.

Autojump Program	Shell	jumps/s	ms/jump
autojump	bash	18	55.875
autojump	zsh	18	56.440
fasd	bash	32	31.463
fasd	zsh	33	29.888
pazi	bash	502	1.993
pazi	zsh	348	2.873
z	bash	242	4.126

We can see that pazi is twice as quick as the alternatives, though again we can't measure z on zsh due to bugs.

Both autojump and fasd end up over 10 times slower. I suspect autojump's slowness her, and elsewhere, is in no small part due to loading the python interpreter. fasd's speed, I suspect, is limited by its awk program used to parse its database, though I have not done in depth profiling to be sure.

Large database jump bench

The jump_large_db benchmark measures the same thing as the previous jump benchmark, but after filling the database with 1000 entries.

This measures more typical jump performance since the database will usually have quite a few entries.

Autojump Program	Shell	jumps/s	ms/jump
autojump	bash	22	95.099
autojump	zsh	21	94.594
fasd	bash	49	35.512
fasd	zsh	40	34.214
pazi	bash	49	17.763
pazi	zsh	40	22.021
z	bash	40	17.811

Pazi does better than the competition here, but its performance degrades significantly worse than fasds. This is a clear sign that pazi's use of msgpack may be non-optimial. Further investigation will be done to improve this performance in Pazi.

Pazi is also close enough to z we might as well call that comparison a tie.