Metadata

[lorenzleutgeb]

I'd like to suggest to add metadata to the packages:

mkDerivation {
  # ...
  meta = {
    # The usual, standard meta-attributes, like `url`, `description`, `license`, ...
    # See <https://nixos.org/manual/nixpkgs/stable/#chap-meta>.
    description = "...";

    # Proposed custom meta-attribute:
    ngi = {
      # `project` refers to a URL of the form "https://nlnet.nl/project/${project}/"
      # the purpose is to establish a link to an NGI 
      project = "Example";

      # `main` is `true` if this package is the "main" package of the
      # project.
      main = true;

      # `options` designates which options (and their child options)
      # can be used to configure the project.
      # It is a list of paths, each of which is a list.
      options = [
        ["services" "example"]
      ];
    };
  };
}

What this enables:

• Indexing of NGIpkgs by project.
• Linking of projects to packages and modules (or rather, their options).
• A "per-project view" of NGIpkgs.

Alternative: Instead of encoding this information as meta-attributes of derivations, we could also have a "registry", e.g. files projects/${project}.nix or so which contains a reference to the packages that are associated with the project and modules/options in a similar way as above. This might give us more flexibility, but would require one more file to maintain per project.

I think that configurations should also be exposed, but they are not mentioned in the above proposal yet.

[fricklerhandwerk]

Discussed this with @lorenzleutgeb today. One thing we realised is that NixOS/rfcs#140 with package.nix and its future-work-idea to have a module.nix is a bit problematic: modules are essentially language values without inherent semantics. dream2nix and NixOS both use modules, and there is no "true" or "default" module class.

Similarly, as I've been pointing out for a while now, the notion of a package is practically nonexistent in the Nix ecosystem, and what mkDerivation in package.nix produces is really, well, just a derivation (even if it's stuffed full of language-level metadata that coexists with the regular builtins.derivation attributes). An idea that I only came to appreciate too late was introduced in a nix.dev article by @zmitchell: What packages.nix contains at the moment is what we'd want to run nix-build on, so make a top-level default.nix that exposes a build attribute. Same for a shell.

Therefore an alternative proposal for pkgs/by-name:

Each directory has the following structure:

• default.nix
  • Exposes the files in the directory as attributes, e.g. build = import ./build.nix;
  • A bit more thinking may need to be applied, but the stupid simple approach to require the result to be callPackaged may actually be enough
  • Adds arbitrary metadata in a meta attribute
• build.nix
  • The thing to run nix-build on
  • Evaluates to a derivation
• shell.nix (optional)
  • The thing to run nix-shell on
  • Evaluates to whatever can be used as a development shell for the package (currently the magic unbuildable mkShell derivation)
• dream2.nix, nixos.nix, ... (optional)
  • Other values with their own semantics
• examples directory (optional)
  • These could be Nix expressions or markdown files
  • Directory contents are exposed in default.nix as literal files (possibly within meta)
• README.md (optional)
  • Contributor-oriented documentation for the package

This follows @infinisil's direction that directory structure should not matter when accessed from the outside (it's an implementation detail that only contributors should have to care about), and to instead emphasize the language-level interfaces.

[fricklerhandwerk]

Discussed with @lorenzleutgeb

Some updates on these ideas (not final, but we seem to be converging):

• Have all package recipes (those functions that take derivations and return a derivation) in pkgs/by-name

  Otherwise one has to think about every recipe where it belongs, this is cognitive overhead that we want to avoid for contributors.

• Each project has its own directory in /projects that corresponds to the project key at https://nlnet.nl/project/

  # /projects/Flarum/default.nix
  {
    pkgs ? import ../., # Nixpkgs + NGIpkgs
    sources ? (import ../../.).sources, # attrset of values that string-interpolate to file system paths, denotes external dependencies
  }:
  {
     packages = {
       inherit (pkgs) flarum;
     };
    nixos.modules = {
      service = ./modules/flarum.nix; # the module system checks equality via file paths
      # there may be other things that one may want to import separately to avoid eval overhead
    };
    nixos.tests = {}; # attrset of NixOS tests, e.g. to be consumed by CI
    nixos.examples = {}; # attrset of attrset: { path = <path to config>;  description = <string>; } (could be just files with doc comments in the future, with post processing using something like nixdoc)
    # space for extra stuff, such as project-level meta-data (e.g. source, tags), documentation

  The sources pattern is not great, but at least it encapsulates external dependencies and makes them overrideable in principle. The ideal end state would be to allow taking the project directory in isolation and have it work, but that's currently not possible with existing tooling in the Nix ecosystem. (See my elaboration on the problem of module system modularity.) We'll go for an approximation and can still fix forward when there are better solutions. These "compositional" interfaces should not be public anyway, for now.

• NixOS tests are per project, and thus live in the project's directory