Git object hashing in libstore #8918

Ericson2314 · 2023-09-04T14:19:17Z

Motivation

Part of RFC 133, tracking issue #8919

Method

The libnixutil interfaces for working with NARs have been resused for a new implementation that works with Git tree+blob Merkle DAGs.
libnixstore's ContentAddress and friends have been augmented to also express git hashes. There are new store paths accordingly.
libfetcher's git fetcher has been agumented to also support this. Any fetcher could create git-hashed objects in principle, but the git fetcher is the best candidate since the source-side objects are also hashed this way; this allows us to fetch by tree hash and thus only need a single hash while still being pure.

The most notable detail is that when transferring git-hashed files over the wire, we create a store object for every subdir and file. This is the easiest way to make building the Merkle Dag safe and easy.

Context

Extracted from our old IPFS branches.

~~depends on #9294~~
~~depends on #9325~~
~~depends on #9815~~

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
documentation in the internal API docs
code and comments are self-explanatory
commit message explains why the change was made
new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

nixos-discourse · 2023-09-04T14:25:56Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/64

bobvanderlinden · 2023-09-04T17:23:16Z

I think it's RFC133 👍 very cool to see the progress in this space!

fricklerhandwerk · 2023-09-18T19:30:43Z

Discussed in Nix team meeting 2023-09-15:

The first four commits are independent cleanups, can be split out.

nixos-discourse · 2023-09-18T19:36:06Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-15-nix-team-meeting-minutes-86/33171/1

dpulls · 2024-02-22T16:15:37Z

🎉 All dependencies have been resolved !

edolstra · 2024-02-23T13:50:52Z

tests/functional/git-hashing/simple.sh

+echo Run Hello World! > $TEST_ROOT/dummy3/dir/executable
+path3=$(nix store add --mode git --hash-algo sha1 $TEST_ROOT/dummy3)
+hash3=$(nix-store -q --hash $path3)
+test "$hash3" = "sha256:08y3nm3mvn9qvskqnf13lfgax5lh73krxz4fcjd5cp202ggpw9nv"


This needs some tests for what happens with store paths that have an executable file or symlink at top-level.

nixos-discourse · 2024-02-26T08:02:26Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-02-03-nix-team-meeting-127/40309/1

src/libstore/binary-cache-store.cc

edolstra · 2024-02-27T14:40:01Z

src/libstore/binary-cache-store.hh

@@ -147,7 +148,7 @@ public:

    void narFromPath(const StorePath & path, Sink & sink) override;

-    ref<SourceAccessor> getFSAccessor(bool requireValidPath) override;
+    ref<SourceAccessor> getFSAccessor(bool requireValidPath = true) override;


Specifying defaults on overrides should be superfluous, so I think it's best to omit them.

Actually see https://en.cppreference.com/w/cpp/language/default_arguments

The overriders of virtual functions do not acquire the default arguments from the base class declarations, and when the virtual function call is made, the default arguments are decided based on the static type of the object.

src/libutil/git.hh

src/libstore/content-address.cc

edolstra · 2024-02-27T14:54:38Z

src/libutil/file-content-address.hh

+ * An enumeration of the ways we can ingest file system
+ * objects, producing a hash or digest.
+ */
+enum struct FileIngestionMethod : uint8_t {


Maybe we can get rid of FileSerialisationMethod by having isFlat() and isRecursive() methods on FileIngestionMethod.

I wish, but that only helps when we are returning FileSerializationMethod, dumpPath and other things that take FileSerialisationMethod as an argument really want to statically, provably, exclude the "git" case.

They were getting skipped for the test-against checks.

Part of RFC 133 Extracted from our old IPFS branches. Co-Authored-By: Matthew Bauer <[email protected]> Co-Authored-By: Carlo Nucera <[email protected]> Co-authored-by: Robert Hensing <[email protected]> Co-authored-by: Florian Klink <[email protected]>

Instead, serialize as NAR and send that over, then rehash sever side. This is alorithmically simpler, but comes at the cost of a newer parameter to `Store::addToStoreFromDump`. Co-authored-by: Eelco Dolstra <[email protected]>

roberth · 2024-02-28T01:59:51Z

This seems to now cause a failure in the daemon compatibility test in #9546.
Could you have a look @Ericson2314?

thufschmitt · 2024-02-28T06:12:25Z

This seems to now cause a failure in the daemon compatibility test in #9546.

This seems to be a wrong bound in requireDaemonNewerThan, should be fixed by 6147d27

Ericson2314 · 2024-02-28T14:30:34Z

Thanks, both of you!

github-actions bot added new-cli Relating to the "nix" command with-tests Issues related to testing. PRs with tests have some priority store Issues and pull requests concerning the Nix store fetching Networking with the outside (non-Nix) world, input locking labels Sep 4, 2023

Ericson2314 added the RFC Related to an accepted RFC label Sep 4, 2023

Ericson2314 force-pushed the git-objects branch 2 times, most recently from 3b44d2d to 06a23cd Compare September 4, 2023 15:32

This was referenced Sep 4, 2023

Tracking issue for RFC 133: Git hashing and Git-hashing-based remote stores #8919

Open

Git hashing as a new file ingestion method --- contains #3754 #3635

Closed

Ericson2314 force-pushed the git-objects branch 4 times, most recently from 3c52443 to 5fdf6e2 Compare September 8, 2023 03:16

Ericson2314 force-pushed the git-objects branch 2 times, most recently from 79fc9f6 to ef57479 Compare September 21, 2023 15:23

Ericson2314 mentioned this pull request Sep 21, 2023

Shuffle ParseSink code in preparation for git hashing support #9024

Merged

github-actions bot removed the fetching Networking with the outside (non-Nix) world, input locking label Sep 21, 2023

Ericson2314 mentioned this pull request Sep 21, 2023

Git hashing for Git fetching #9025

Draft

Ericson2314 changed the base branch from master to 2.18-maintenance September 25, 2023 15:40

Ericson2314 changed the base branch from 2.18-maintenance to master September 25, 2023 15:40

Ericson2314 force-pushed the git-objects branch from ef57479 to aa9d55b Compare September 25, 2023 15:41

Ericson2314 marked this pull request as ready for review September 25, 2023 15:42

Ericson2314 requested review from thufschmitt and edolstra as code owners September 25, 2023 15:42

Ericson2314 force-pushed the git-objects branch 2 times, most recently from fb0c11b to a37bd98 Compare October 6, 2023 16:43

Ericson2314 force-pushed the git-objects branch 2 times, most recently from 79c9cc9 to 6578193 Compare January 19, 2024 06:11

Ericson2314 marked this pull request as ready for review January 19, 2024 06:12

Ericson2314 requested a review from roberth as a code owner January 19, 2024 06:12

Ericson2314 force-pushed the git-objects branch 5 times, most recently from 36e5ff3 to aab19fa Compare January 22, 2024 15:30

Ericson2314 force-pushed the git-objects branch from aab19fa to 7629584 Compare February 13, 2024 15:57

Ericson2314 force-pushed the git-objects branch from 7629584 to bf8b9ab Compare February 22, 2024 17:09

edolstra reviewed Feb 23, 2024

View reviewed changes

Ericson2314 force-pushed the git-objects branch 2 times, most recently from 42373e5 to 6becfa8 Compare February 26, 2024 19:11

thufschmitt mentioned this pull request Feb 27, 2024

Revert "Remove dead Git code" #10093

Merged

edolstra approved these changes Feb 27, 2024

View reviewed changes

Ericson2314 force-pushed the git-objects branch from 6becfa8 to 3673dd7 Compare February 27, 2024 16:16

Ericson2314 and others added 3 commits February 27, 2024 11:27

Merge nativeCheckInputs into nativeBuildInputs

04836c7

They were getting skipped for the test-against checks.

Avoid creating temporary store object for git over the wire

d4ad1fc

Instead, serialize as NAR and send that over, then rehash sever side. This is alorithmically simpler, but comes at the cost of a newer parameter to `Store::addToStoreFromDump`. Co-authored-by: Eelco Dolstra <[email protected]>

Ericson2314 force-pushed the git-objects branch from d32f528 to d4ad1fc Compare February 27, 2024 17:10

Ericson2314 merged commit f489a6e into NixOS:master Feb 28, 2024
8 checks passed

Ericson2314 deleted the git-objects branch February 28, 2024 00:02

Ericson2314 mentioned this pull request Feb 28, 2024

Support symlinks properly with git-hashing experimental feature #10111

Merged

Ericson2314 added this to the git-hashing stabilisation milestone Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Git object hashing in libstore #8918

Git object hashing in libstore #8918

Ericson2314 commented Sep 4, 2023 •

edited

Loading

nixos-discourse commented Sep 4, 2023

bobvanderlinden commented Sep 4, 2023

fricklerhandwerk commented Sep 18, 2023

nixos-discourse commented Sep 18, 2023

dpulls bot commented Feb 22, 2024

edolstra Feb 23, 2024

nixos-discourse commented Feb 26, 2024

edolstra Feb 27, 2024

Ericson2314 Feb 27, 2024

edolstra Feb 27, 2024

Ericson2314 Feb 27, 2024

roberth commented Feb 28, 2024

thufschmitt commented Feb 28, 2024

Ericson2314 commented Feb 28, 2024

Git object hashing in libstore #8918

Git object hashing in libstore #8918

Conversation

Ericson2314 commented Sep 4, 2023 • edited Loading

Motivation

Method

Context

Checklist for maintainers

Priorities

nixos-discourse commented Sep 4, 2023

bobvanderlinden commented Sep 4, 2023

fricklerhandwerk commented Sep 18, 2023

nixos-discourse commented Sep 18, 2023

dpulls bot commented Feb 22, 2024

edolstra Feb 23, 2024

Choose a reason for hiding this comment

nixos-discourse commented Feb 26, 2024

edolstra Feb 27, 2024

Choose a reason for hiding this comment

Ericson2314 Feb 27, 2024

Choose a reason for hiding this comment

edolstra Feb 27, 2024

Choose a reason for hiding this comment

Ericson2314 Feb 27, 2024

Choose a reason for hiding this comment

roberth commented Feb 28, 2024

thufschmitt commented Feb 28, 2024

Ericson2314 commented Feb 28, 2024

Ericson2314 commented Sep 4, 2023 •

edited

Loading