Support opening the database in exclusive mode

[shekhirin]

Problem

Now, if you try to open a database file located in a remote filesystem such as NFS, it will fail with error 15:

2024-02-20T02:50:15.969537Z  INFO reth::cli: reth 0.1.0-alpha.18 (11dbd08) starting
2024-02-20T02:50:15.969564Z  INFO reth::cli: Opening database path="/data/db"
2024-02-20T02:50:15.973550Z ERROR reth::cli: shutting down due to error
Error: failed to open the database: unknown error code (15)

Location:
    /project/crates/storage/db/src/lib.rs:113:18
I have no name!@reth-snapshot2-execution-0:/$ ls -la /data/db
total 2508195548
drwxrwsr-x 2 10001 10001          4096 Feb 20 01:49 .
drwxrwsr-x 8 10001 10001          4096 Feb 20 02:06 ..
-rw-rw-r-- 1 10001 10001             1 Feb 20 01:49 database.version
-rw-rw-r-- 1 10001 10001 2568390443008 Feb 20 01:49 mdbx.dat
-rw-rw-r-- 1 10001 10001       1028096 Feb 20 01:49 mdbx.lck

Error 15 maps into ENOTBLK GNU kernel error, which MDBX re-uses as MDBX_EREMOTE

reth/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.h

Line 1969 in a554c1f

MDBX_EREMOTE = ENOTBLK

And returns in cases when remote filesystem is being accessed with no MDBX_EXCLUSIVE set https://github.com/paradigmxyz/reth/blob/a554c1fbf889af0087093a28b9efe93917f51004/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.c#L31925-L31933

Solution

We have the exclusive flag inside our Rust struct

reth/crates/storage/libmdbx-rs/src/flags.rs

Lines 152 to 154 in a554c1f

	if self.exclusive {
	flags |= ffi::MDBX_EXCLUSIVE;
	}

but there's no way to set it when opening a new database environment. We need to be able to pass this flag through DatabaseArguments

reth/crates/storage/db/src/implementation/mdbx/mod.rs

Lines 59 to 66 in a554c1f

	/// Arguments for database initialization.
	#[derive(Debug, Default, Clone, Copy)]
	pub struct DatabaseArguments {
	/// Database log level. If [None], the default value is used.
	log_level: Option<LogLevel>,
	/// Maximum duration of a read transaction. If [None], the default value is used.
	max_read_transaction_duration: Option<MaxReadTransactionDuration>,
	}

and then use it when setting up the environment

reth/crates/storage/db/src/implementation/mdbx/mod.rs

Lines 244 to 251 in a554c1f

	inner_env.set_flags(EnvironmentFlags {
	mode,
	// We disable readahead because it improves performance for linear scans, but
	// worsens it for random access (which is our access pattern outside of sync)
	no_rdahead: true,
	coalesce: true,
	..Default::default()
	});

[loocapro]

@shekhirin , does this imply that once resolved, it would be feasible to host the database file on NFS to support multiple readers alongside a single node writing to it?

Can i take this?

[argakiig]

I believe the read from many, write from one works but there are some caveats

attaching info from reth:

Ok, So i patched in exclusive and accede flags when the db is openned and was able to open the DB from the NFS(R/W) mount and everything is working properly there
Then out of curiosity I attempted the same patch and ran the db-access example with the the DB being NFS(RO) mounted. This returned an error 30 similar to #6354 as it was unable to do anything with the existing lck file due to the RO mount>
Symlinking the db to a writeable path and it will open the db just fine but also creates a lck file.

[argakiig]

applogies, i was fat fingering a path when i was getting error 30 here

argakiig@reth-restored-disk-vm-1:~/reth/examples$ RETH_DB_PATH=/data2/db cargo run --example db-access
    Finished dev [unoptimized + debuginfo] target(s) in 0.71s
     Running `/home/argakiig/reth/target/debug/examples/db-access`
Error: Could not open database at path: /data2/db

Caused by:
    failed to open the database: unknown error code (30)

Location:
    crates/storage/db/src/lib.rs:128:14

using the correct path

RETH_DB_PATH=/data2/ cargo run --example db-access
    Finished dev [unoptimized + debuginfo] target(s) in 1.23s
     Running `/home/argakiig/reth/target/debug/examples/db-access`
Error: Could not open database at path: /data2/

Caused by:
    failed to open the database: unknown error code (2)

Location:
    crates/storage/db/src/lib.rs:128:14

and then using the symlink

 RETH_DB_PATH=/home/argakiig/data/ cargo run --example db-access
    Finished dev [unoptimized + debuginfo] target(s) in 0.69s
     Running `/home/argakiig/reth/target/debug/examples/db-access`
^C

argakiig@reth-restored-disk-vm-1:~/reth/examples$ ls -la /home/argakiig/data/
total 8
drwxr-xr-x 2 argakiig argakiig 4096 Feb 21 16:58 .
drwxr-xr-x 8 argakiig argakiig 4096 Feb 21 16:57 ..
lrwxrwxrwx 1 argakiig argakiig   18 Feb 21 16:57 mdbx.dat -> /data2/db/mdbx.dat
-rw-r--r-- 1 argakiig argakiig    0 Feb 21 16:58 mdbx.lck