Code for the benchmark study described in this blog post.
Neo4j version | Kùzu version | Python version |
---|---|---|
5.22.0 (community) | 0.6.0 | 3.12.4 |
Kùzu is an in-process (embedded) graph database management system (GDBMS) written in C++. It is blazing fast 🔥, and is optimized for handling complex join-heavy analytical workloads on very large graphs. Kùzu's goal is to do in the graph database world what DuckDB has done in the world of relational databases -- that is, to provide a fast, lightweight, embeddable graph database for analytics (OLAP) use cases, while being heavily focused on usability and developer productivity.
This study has the following goals:
- Generate an artificial social network dataset, including persons, interests and locations
- You can scale up the size of the artificial dataset using the scripts provided and test query performance on larger graphs
- Ingest the dataset into two graph databases: Kùzu and Neo4j (community edition)
- Run a set of queries in Cypher on either DB to:
- (1) Verify that the data is ingested correctly and that the results from either DB are consistent with one another
- (2) Compare the query performance on a suite of queries that involve multi-hop traversals and aggregations
Python (and the associated client APIs for either DB) are used to orchestrate the pipelines throughout.
Activate a Python virtual environment and install the dependencies as follows.
# Assuming that the uv package manager is installed
# https://github.com/astral-sh/uv
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
An artificial social network dataset is generated specifically for this exercise, via the Faker Python library.
A shell script generate_data.sh
is provided in the root directory of this repo that sequentially runs the Python scripts, generating the data for the nodes and edges for the social network. This is the recommended way to generate the data. A single positional argument is provided to the shell script: The number of person profiles to generate -- this is specified as an integer, as shown below.
# Generate data with 100K persons and ~2.4M edges
bash generate_data.sh 100000
Running this command generates a series of files in the output
directory, following which we can proceed to ingesting the data into a graph database.
See ./data/README.md for more details on each script that is run sequentially to generate the data.
The following graph schema is used for the social network dataset.
Person
nodeFOLLOWS
anotherPerson
nodePerson
nodeLIVES_IN
aCity
nodePerson
nodeHAS_INTEREST
towards anInterest
nodeCity
node isCITY_IN
aState
nodeState
node isSTATE_IN
aCountry
node
Navigate to the neo4j and the kuzudb directories to see the instructions on how to ingest the data into each database.
The generated graph is a well-connected graph, and a sample of Person
-Person
connections as visualized in the Neo4j browser is shown below. Certain groups of persons form a clique, and some others are central hubs with many connections, and each person can have many interests, but only one primary residence city.
Some sample queries are run in each DB to verify that the data is ingested correctly, and that the results are consistent with one another.
The following questions are asked of both graphs:
- Query 1: Who are the top 3 most-followed persons?
- Query 2: In which city does the most-followed person live?
- Query 3: Which 5 cities in a particular country have the lowest average age in the network?
- Query 4: How many persons between ages 30-40 are there in each country?
- Query 5: How many men in London, United Kingdom have an interest in fine dining?
- Query 6: Which city has the maximum number of women that like Tennis?
- Query 7: Which U.S. state has the maximum number of persons between the age 23-30 who enjoy photography?
- Query 8: How many second-degree paths exist in the graph?
- Query 9: How many paths exist in the graph through persons age 50 to persons above age 25?
The run times for both ingestion and queries are compared.
- For ingestion, KùzuDB is consistently faster than Neo4j by a factor of ~18x for a graph size of 100K nodes and ~2.4M edges.
- For OLAP queries, KùzuDB is significantly faster than Neo4j, especially for ones that involve multi-hop queries via nodes with many-to-many relationships.
The benchmark is run M3 Macbook Pro with 36 GB RAM.
Case | Neo4j (sec) | Kùzu (sec) | Speedup factor |
---|---|---|---|
Nodes | 2.33 | 0.11 | 21.2x |
Edges | 31.08 | 0.42 | 74.0x |
Total | 33.41 | 0.53 | 63.0x |
Nodes are ingested significantly faster in Kùzu, and using its community edition, Neo4j's node ingestion
remains of the order of seconds
despite setting constraints on the ID fields as per their best practices. The speedup factors shown
are expected to be even higher as the dataset gets larger and larger using this approach, and
the only way to speed up Neo4j data ingestion is to use admin-import
instead (however, this means
you lose the ability to work in Python and have to switch languages).
The full benchmark numbers are in the README.md
pages for respective directories for neo4j
and kuzudb
, with the high-level summary shown below.
The benchmarks are run via the pytest-benchmark
library for the query scripts for either DB. pytest-benchmark
, which is built on top of pytest
, attaches each set of runs to a timer. It uses the Python time module's time.perf_counter
, which has a resolution of 500 ns, smaller than the run time of the fastest query in this dataset.
- 5 warmup runs are performed to ensure byte code compilation and to warm up the cache prior to measuring run times
- Each query is run for a minimum of 5 rounds, so the run times shown in each section below as the average over a minimum of 5 rounds, or upwards of 50 rounds.
- Long-running queries (where the total run time exceeds 1 sec) are run for at least 5 rounds.
- Short-running queries (of the order of milliseconds) will run as many times as fits into a period of 1 second, so the fastest queries can run upwards of 50 times.
- Python's own GC overhead can obscure true run times, so the
benchamrk-disable-gc
argument is enabled.
See the pytest-benchmark
docs to see how they calibrate their timer and group the rounds.
The following table shows the run times for each query (averaged over the number of rounds run, guaranteed to be a minimum of 5 runs) and the speedup factor of Kùzu over Neo4j when Kùzu is limited to execute queries on a single thread.
Query | Neo4j (sec) | Kùzu (sec) | Speedup factor |
---|---|---|---|
1 | 1.375 | 0.216 | 6.4x |
2 | 0.567 | 0.253 | 2.2x |
4 | 0.047 | 0.008 | 5.9x |
3 | 0.052 | 0.006 | 8.7x |
5 | 0.012 | 0.181 | 0.1x |
6 | 0.024 | 0.059 | 0.4x |
7 | 0.155 | 0.013 | 11.9x |
8 | 2.988 | 0.064 | 46.7x |
9 | 3.755 | 0.170 | 22.1x |
KùzuDB (by default) supports multi-threaded execution of queries. The following results are for the same queries as above, but allowing Kùzu to choose the optimal number of threads for each query. Again, the run times for each query (averaged over the number of rounds run, guaranteed to be a minimum of 5 runs) are shown.
Query | Neo4j (sec) | Kùzu (sec) | Speedup factor |
---|---|---|---|
1 | 1.375 | 0.251 | 5.5x |
2 | 0.567 | 0.283 | 2.0x |
3 | 0.052 | 0.011 | 4.7x |
4 | 0.047 | 0.008 | 5.9x |
5 | 0.012 | 0.017 | 0.7x |
6 | 0.024 | 0.061 | 0.4x |
7 | 0.155 | 0.014 | 11.1x |
8 | 2.988 | 0.064 | 46.7x |
9 | 3.755 | 0.142 | 26.5x |
🔥 The second-degree path-finding queries (8 and 9) show the biggest speedup over Neo4j, due to innovations in KùzuDB's query planner and execution engine.
You can attempt to generate a much larger artificial dataset of ~100M nodes and ~2.5B edges, and see how the performance of Kùzu and Neo4j compare, if you're interested.
# Generate data with 100M persons and ~2.5B edges (takes a long time in Python!)
bash generate_data.sh 100000000
The above script can take really long to run in Python. Here's an example
of using the fake-rs
crate in Rust to do this much faster.
The queries 1-9 in this benchmark are all on node properties. You can add relationship properties in the dataset
to see how the two DBs compare when aggregating on them. For example, add a since
date property on the
Follows
edges to run filter queries on how long a person has been following another person.