Skip to content

Commit

Permalink
treewide: Refactor to per-project structure
Browse files Browse the repository at this point in the history
  • Loading branch information
lorenzleutgeb committed Apr 16, 2024
1 parent 2b45422 commit 3864fa8
Show file tree
Hide file tree
Showing 48 changed files with 315 additions and 139 deletions.
50 changes: 33 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,38 +4,49 @@
of software projects funded by
the [Next Generation Internet][NGI] (NGI) initiative of the European Commission

[Nix]: https://nixos.org/manual/nix
[NixOS]: https://nixos.org/manual/nixos
[NGI]: https://www.ngi.eu

## Structure of NGIpkgs

The software in NGIpkgs can be divided into two broad categories: Nix packages, and NixOS modules.

Nix packages can theoretically be built and run on any operating system that runs the Nix package manager. The output of building a Nix package is often a usable library or executable and most if not all of its dependencies. In NGIpkgs, these packages are all contained in the `pkgs` directory. For simple package definitions, we use `pkgs/by-name/<pname>/package.nix`, inspired by [Nix RFC 140](https://github.com/NixOS/rfcs/blob/c8569f6719356009204133cd00d92010889ed56d/rfcs/0140-simple-package-paths.md). Otherwise, packages are added in `pkgs/<pname>/default.nix` imported in `pkgs/default.nix`.
Nix packages can theoretically be built and run on any operating system that runs the Nix package manager.
The output of building a Nix package is often a usable library or executable and most if not all of its dependencies.
In NGIpkgs, these packages are all contained in the `pkgs` directory.
For simple package definitions, we use `pkgs/by-name/<pname>/package.nix`, inspired by [Nix RFC 140][rfc-140].
Otherwise, packages are added in `pkgs/<pname>/default.nix` imported in `pkgs/default.nix`.

Corresponding to [funded projects](https://nlnet.nl/project/) there are per-project subdirectories within the `projects` directory.
These per-project directories contain a `default.nix` which
(a) picks packages associated with the project from those defined in `pkgs` and Nixpkgs,
(b) exposes NixOS modules, tests and configurations which are also contained in the per-project directory,
(c) may contain additional metadata about the project.

NixOS modules are components that can be easily integrated into NixOS. Usually they enrich Nix packages with configuration parameters. Many of them represent services that map to one or more systemd service(s) that are designed to, run persistently on NixOS. These modules are defined in the `modules` directory of NGIpkgs, and they are ready to be deployed to a new NixOS system (such as a container, VM, or physical machine). Templates in `configs` are a good starting point for anyone interested in using modules, and they are also used for testing.
NixOS modules are components that can be easily integrated into NixOS.
Usually they enrich Nix packages with configuration parameters.
Many of them represent services that map to one or more systemd service(s) that are designed to, run persistently on NixOS.
These modules are ready to be deployed to a new NixOS system (such as a container, VM, or physical machine).
Templates configurations found in the corresponding per-project directory are a good starting point for anyone interested in using modules, and they are also used for testing.

```
.
├── flake.nix
├── pkgs
│   ├── by-name
│ │   └── ... # directories of packages that are added `by-name`
│ ├── default.nix # imports all packages that are not in `by-name`
│   └── ... # directories for packages
├── modules
│   ── ... # add module files here
├── README.md # this file
── configs
│   ── all-configurations.nix # import configuration files here
│   └── ... # add configuration directories here
└── ...
│ │   └── # directories of packages that are added `by-name`
│ ├── default.nix # imports all packages that are not in `by-name`
│   └── # directories for packages
├── projects
│   ── <project-name> # names match with those at https://nlnet.nl/project
│ │ ├── default.nix # project definition
│   │ └── … # files of the project (e.g. NixOS module, configuration, tests, etc.)
│   ──
── README.md # this file
└──
```

## Continuous Builds of Packages with Hydra

All packages in the main branch of NGIpkgs are automatically built by a [Hydra](https://github.com/NixOS/hydra) server. The results of these builds can be seen at <https://hydra.ngi0.nixos.org/jobset/NGIpkgs/main#tabs-jobs>
All packages in the main branch of NGIpkgs are automatically built by a [Hydra](https://github.com/NixOS/hydra) server.
The results of these builds can be seen at <https://hydra.ngi0.nixos.org/jobset/NGIpkgs/main#tabs-jobs>

## Reasoning for Creation of the NGIpkgs Monorepo

Expand All @@ -46,3 +57,8 @@ All packages in the main branch of NGIpkgs are automatically built by a [Hydra](
## Contributing to NGIpkgs

Please see [`CONTRIBUTING.md`](CONTRIBUTING.md)

[Nix]: https://nixos.org/manual/nix
[NixOS]: https://nixos.org/manual/nixos
[NGI]: https://www.ngi.eu
[rfc-140]: https://github.com/NixOS/rfcs/blob/c8569f6719356009204133cd00d92010889ed56d/rfcs/0140-simple-package-paths.md
27 changes: 0 additions & 27 deletions configs/all-configurations.nix

This file was deleted.

File renamed without changes.
61 changes: 35 additions & 26 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,17 @@
dream2nix,
...
}: let
# Take Nixpkgs' lib and update it with the definitions in ./lib.nix
lib = nixpkgs.lib.recursiveUpdate nixpkgs.lib (import ./lib.nix {inherit (nixpkgs) lib;});

inherit
(builtins)
readDir
mapAttrs
attrValues
;

inherit
(nixpkgs.lib)
(lib)
concatMapAttrs
mapAttrs'
foldr
Expand All @@ -45,26 +47,27 @@
nixosSystem
filterAttrs
attrByPath
mapAttrByPath
flattenAttrsSlash
;

importProjects = {
pkgs ? {},
sources ? {
configurations = rawNixosConfigs;
modules = extendedModules;
},
}:
import ./projects {inherit lib pkgs sources;};

pickNixosModules = mapAttrByPath ["nixos" "modules"] {};

pickNixosTests = mapAttrByPath ["nixos" "tests"] {};

pickNixosConfigurations = x: mapAttrs (_: v: mapAttrs (_: v: v.path) v) (mapAttrByPath ["nixos" "configurations"] {} x);

importPackages = pkgs: let
nixosTests = let
dir = ./tests;
testDirs = readDir dir;

dirToTest = name: _: let
mkTestModule = import "${dir}/${name}";

testModule = mkTestModule {
inherit pkgs;
inherit (pkgs) lib;
modules = extendedModules;
configurations = rawNixosConfigs;
};
in
pkgs.nixosTest testModule;
in
mapAttrs dirToTest testDirs;
nixosTests = pickNixosTests (importProjects {pkgs = pkgs // allPackages;});

callPackage = pkgs.newScope (
allPackages // {inherit callPackage nixosTests;}
Expand All @@ -87,10 +90,13 @@
importNixpkgs = system: overlays:
import nixpkgs {inherit system overlays;};

rawNixosConfigs = import ./configs/all-configurations.nix;
rawNixosConfigs = flattenAttrsSlash (pickNixosConfigurations (importProjects {}));

loadTreefmt = pkgs: treefmt-nix.lib.evalModule pkgs ./treefmt.nix;

# Overlays a package set (e.g. Nixpkgs) with the packages defined in this flake.
overlay = final: prev: importPackages prev;

# Attribute set containing all modules obtained via `inputs` and defined
# in this flake towards definition of `nixosConfigurations` and `nixosTests`.
extendedModules =
Expand All @@ -101,13 +107,13 @@

nixosConfigurations =
mapAttrs
(_: config: nixosSystem {modules = [config ./configs/dummy.nix] ++ attrValues extendedModules;})
(_: config: nixosSystem {modules = [config ./dummy.nix] ++ attrValues extendedModules;})
rawNixosConfigs;

eachDefaultSystemOutputs = flake-utils.lib.eachDefaultSystem (system: let
pkgs = importNixpkgs system [rust-overlay.overlays.default];
treefmtEval = loadTreefmt pkgs;
toplevel = name: config: nameValuePair "nixosConfigs/${name}" config.config.system.build.toplevel;
toplevel = name: config: nameValuePair "nixosConfigurations/${name}" config.config.system.build.toplevel;
in {
packages = importPackages pkgs;
formatter = treefmtEval.config.build.wrapper;
Expand Down Expand Up @@ -143,7 +149,10 @@
nonBrokenPkgs;
in {
packages.${system} = nonBrokenPkgs;
tests.${system} = passthruTests;
tests.${system} = {
passthru = passthruTests;
nixos = pickNixosTests (importProjects {pkgs = pkgs // nonBrokenPkgs;});
};

nixosConfigurations.${system} =
mapAttrs
Expand All @@ -157,14 +166,14 @@

nixosModules =
(import ./modules/all-modules.nix)
// (flattenAttrsSlash (pickNixosModules (importProjects {})))
// {
# The default module adds the default overlay on top of nixpkgs.
# The default module adds the default overlay on top of Nixpkgs.
# This is so that `ngipkgs` can be used alongside `nixpkgs` in a configuration.
default.nixpkgs.overlays = [self.overlays.default];
};

# Overlays a package set (e.g. nixpkgs) with the packages defined in this flake.
overlays.default = final: prev: importPackages prev;
overlays.default = overlay;
};
in
foldr recursiveUpdate {} [
Expand Down
50 changes: 50 additions & 0 deletions lib.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Functions to lay on top of Nixpkgs' lib for convenience.
{lib}: let
inherit
(builtins)
mapAttrs
isAttrs
concatStringsSep
;

inherit
(lib)
attrByPath
concatMapAttrs
;
in rec {
# Takes an attrset of arbitrary nesting (attrset containing attrset)
# and flattens it into an attrset that is *not* nested, i.e., does
# *not* contain attrsets.
# This is done by concatenating the names of nested values using a
# separator.
#
# Type: flattenAttrs :: string -> [string] -> AttrSet -> AttrSet
#
# Example:
# flattenAttrs "~" ["1" "2"] { a = { b = "x"; }; c = { d = { e = "y"; }; }; f = "z"; }
# => { "1~2~a~b" = "x"; "1~2~c~d~e" = "y"; "1~2~f" = "z"; }
flattenAttrs =
# Separator to use to join names of different nesting levels.
separator:
# Prefix to be prepended to all names in the generated attrset,
# as a list that is joined by the separator.
prefix: let
initPath =
if prefix == []
then ""
else (concatStringsSep separator prefix) + separator;
f = path:
concatMapAttrs (
name: value:
if isAttrs value
then f (path + name + separator) value
else {${path + name} = value;}
);
in
f initPath;

flattenAttrsSlash = flattenAttrs "/" [];

mapAttrByPath = attrPath: default: mapAttrs (_: attrByPath attrPath default);
}
File renamed without changes.
7 changes: 0 additions & 7 deletions modules/all-modules.nix
Original file line number Diff line number Diff line change
@@ -1,10 +1,3 @@
{
# LiberaForms is intentionally disabled.
# Refer to <https://github.com/ngi-nix/ngipkgs/issues/40>.
#liberaforms = import ./liberaforms.nix;
flarum = import ./flarum.nix;
kbin = import ./kbin.nix;
mcaptcha = import ./mcaptcha.nix;
pretalx = import ./pretalx.nix;
unbootable = import ./unbootable.nix;
}
6 changes: 3 additions & 3 deletions pkgs/by-name/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -25,18 +25,18 @@

packageDirectories = concatMapAttrs names (readDir baseDirectory);

callModule = moduleDir: let
callModule = module: let
evaluated = lib.evalModules {
specialArgs = {
inherit dream2nix;
packageSets.nixpkgs = pkgs;
};
modules = [
moduleDir
module
{
paths.projectRoot = ../..;
paths.projectRootFile = "flake.nix";
paths.package = moduleDir;
paths.package = module;
paths.lockFile = "lock.json";
}
];
Expand Down
46 changes: 20 additions & 26 deletions pkgs/by-name/flarum/package.nix
Original file line number Diff line number Diff line change
@@ -1,33 +1,27 @@
{
fetchFromGitHub,
fetchurl,
lib,
php,
}: let
inherit
(lib)
licenses
;
in
php.buildComposerProject (finalAttrs: {
pname = "flarum";
version = "1.8.0";
}:
php.buildComposerProject (finalAttrs: {
pname = "flarum";
version = "1.8.0";

src = fetchFromGitHub {
owner = "flarum";
repo = "flarum";
rev = "v${finalAttrs.version}";
hash = "sha256-xadZIdyH20mxfxCyiDRtSRSrPj8DWXpuup61WSsjgWw=";
};
src = fetchFromGitHub {
owner = "flarum";
repo = "flarum";
rev = "v${finalAttrs.version}";
hash = "sha256-xadZIdyH20mxfxCyiDRtSRSrPj8DWXpuup61WSsjgWw=";
};

composerLock = ./composer.lock;
composerStrictValidation = false;
vendorHash = "sha256-G/EPHcvcppuyAC0MAzE11ZjlOSTlphQrHnO3yS4+j5g=";
composerLock = ./composer.lock;
composerStrictValidation = false;
vendorHash = "sha256-G/EPHcvcppuyAC0MAzE11ZjlOSTlphQrHnO3yS4+j5g=";

meta = {
changelog = "https://github.com/flarum/framework/blob/main/CHANGELOG.md";
description = "Flarum is a delightfully simple discussion platform for your website";
homepage = "https://github.com/flarum/flarum";
license = licenses.mit;
};
})
meta = {
changelog = "https://github.com/flarum/framework/blob/main/CHANGELOG.md";
description = "Flarum is a delightfully simple discussion platform for your website";
homepage = "https://github.com/flarum/flarum";
license = lib.licenses.mit;
};
})
8 changes: 7 additions & 1 deletion pkgs/by-name/kbin/package.nix
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{
lib,
runCommand,
kbin-frontend,
kbin-backend,
Expand All @@ -10,9 +11,14 @@ runCommand "kbin" {
kbin-backend.passthru
// {
tests = {
inherit (nixosTests) kbin;
inherit (nixosTests.Kbin) kbin;
};
};
meta = {
license = lib.licenses.agpl3Only;
homepage = "https://kbin.pub/";
description = "/kbin is a modular, decentralized content aggregator and microblogging platform running on the Fediverse network.";
};
} ''
# As of 2023-10-09, there is no way to just symlink
# backend and frontend (using `lndir`):
Expand Down
Loading

0 comments on commit 3864fa8

Please sign in to comment.