-
-
Notifications
You must be signed in to change notification settings - Fork 13.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bring all packaging layers into a single fixpoint #273815
Comments
This is not complete, as described in the comment. You can now do: nix-repl> :b derivation (stdenv.makeDerivationArgument { name = "hi"; dontUnpack = true; buildPhase = "echo hi > $out"; dontInstall = true; }) or even: nix-repl> :b x // { type = "derivation"; } error: derivation name missing nix-repl> x.drvPath "/nix/store/zfnznggsbm48y2g130wg1h891gwbf6s4-hi.drv" nix-repl> :b x // { name = "okay"; type = "derivation"; } This derivation produced the following outputs: out -> /nix/store/szky5bcmil93q0jjqprzc52mia5nfgsc-hi The latter is obviously NOT recommended, because it does not produce a standard package attribute set. Point is, new code could implement that logic without duplicating what's in Nix's built-in derivation.nix file. Among many other improvements. - For instance, no cmake/meson overhead where it's not needed. - Single fixpoint NixOS#273815
This shows the `deps` pattern, merging the following into a single fixpoint, from NixOS#273815 - package function args (`this.deps`) - mkDerivation args (`this.setup`) - derivation args (`this.drvAttrs`) - package attributes (`this.public`)
This shows the `deps` pattern, merging the following into a single fixpoint, from NixOS#273815 - package function args (`this.deps`) - mkDerivation args (`this.setup`) - derivation args (`this.drvAttrs`) - package attributes (`this.public`)
This shows the `deps` pattern, merging the following into a single fixpoint, from NixOS#273815 - package function args (`this.deps`) - mkDerivation args (`this.setup`) - derivation args (`this.drvAttrs`) - package attributes (`this.public`)
First attempt: |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/is-this-the-current-state-of-things/45944/13 |
It might be a good idea to mention the idea of Pure Object Prototypes since that idea was also brought up at one point to solve some of the same problems. I haven't looked into this or POPs enough to know how similar they are so I can't say for certain, but this does seem at least somewhat similar from what I know. @roberth sorry for the ping, but is that concept(NixOS/rfcs#91) something you know the details about? |
@purepani No worries! Thanks for highlighting POP. It seems to be a good object system, but the added complexity seems to be a problem, both in terms of contributors having to learn yet another composition mechanism (even if it may be worth learning), and for the possible performance issue raised here, especially as I do expect it to be amplified by this. |
Note
This proposal is scoped to more or less
mkDerivation
, so individual packages, and it does not affect package sets in any significant way.Describe the problem
This issue proposes a solution to a number of problems.
mkDerivation
mkDerivation
mkDerivation
has significant pressure to grow, become huge, incomprehensible, slow, and mass-rebuild inducingWhen packaging, we have to keep six+ layers of attribute sets in mind for various purposes.
That is quite a lot, and it leads to issues when one needs to access information from a particular layer in another layer.
For instance, overriding functions are generally only available for ~3 of the layers, and using some may revert the effect of previous overrides.
As an example, the layers of a Python package are:
callPackage
,.override
)pkg.dev != pkg
, but is very similar)Proposed solution
The layers themselves are valid, mostly, but the way they are composed, by "ad hoc" functions, is the cause of aforementioned problems. Instead, we may compose them in a manner that may be somewhat familiar from the module system.
Use of the module system has been explored extensively and successfully by @DavHau and dream2nix. However, Nixpkgs has such a scale that we need to care far more about even a constant factor overhead, such as would be imposed by the module system. Experimentally, we have seen that this pretty much rules out wide use of such a feature rich system.
However, that does not mean that we need to reject what I would consider the core features of the module system: a fixpoint of a monoid. What does that mean?
Fixpoint: we declare things using functions, where the argument is the "final" result. This recursion allows access to "variables" or "option" in a way that might feel similar to fields in an object oriented language.
Monoid: we don't use a single function, but multiple, and their results are merged in some way.
In the module system, this merging operation is very elaborate, and therefore a source of evaluation overhead. This must be avoided.
A possible, lighter weight version of this is the merge operation in minimod. An even lighter alternative is that of overlays: little more than
//
, also known asattrsets.merge
, which is non-recursive and minimally helpful.The exact merging semantics is to be decided. It will be easier to do so when we have a prototype that we can benchmark and play around with.
And that brings us to the final crucial element, which is overriding. The module system uses numeric priority markers to specify which definitions win, whereas with overlays, the last overlay composition operation wins. That latter is more efficient, but harder to use, as any merging needs to be specified by hand, e.g.
buildInputs = o.buildInputs or [] ++ ...
etc.Finally, we may consider leaving all merging behavior up to the user, and let them pick between such methods. This would be most flexible, but imposes more complexity on package authors.
What might this look like? Assuming we go with a limited amount of merging, and a last-wins overriding system as described, a package might as follows. Some required attributes, such as
setup.name
are omitted.An override may look like:
The "replacement" of
callPackage
may look likeImplementation
In order to implement this functionality in a sustainable manner, we need to disentangle the
setup -> derivation
transformation that is currently implemented inmake-derivation.nix
, mixed with parts of the implementation ofoverrideAttrs
.Taking this apart will greatly benefit readability, but more importantly allow us to improve the overriding mechanisms without unsustainable code duplication with
mkDerivation
.The first layer to implement is the package attrs layer. This layer is fairly simple. It is responsible for taking the
public
attribute from the fixpoint and returning it.The next layer to implement is the
derivation
layer. It writes the result ofbuiltins.strictDerivation
topublic
, perhaps with minor tweaks. This is more efficient thanbuiltins.derivation
and it implements a separation between a package's public, supported interface, and attributes that are implementation details. See #217243.Then the disentangled
mkDerivation
logic can be applied, taking arguments fromsetup
(or some other name; TBD), and returning into thederivation
attribute of the fixpoint. This should not include redundant features such aspassthru
oroverrideAttrs
, whose implementations stay behind inmkDerivation
.For the multi-outputs wrapper functionality that presents alternate outputs as full blown package attrsets, I would suggest to provide an alternative attribute
outputs = { bin = "<store path string>"; dev = "<store path string>"; }
, which is sufficient for almost all usages, and more efficient to evaluate. Legacy output attributes may still be provided, but should be avoided by library code and builder functions ifoutputs
is an attrset.I am confident that we can combine all layers except perhaps cross splicing into a single fixed point. Cross splicing does not represent a 1:1 function invocation, so may need to remain similar to today, although I wouldn't exclude the possibility of making improvements in that area.
For the details of the
deps
pattern, refer to dream2nix.At this point, the
mkDerivation
functionality has been replicated in a better way that solves most of the problems. Porting the language infrastructures to the new style will take further effort. As similar approach to that withmkDerivation
can be applied.Additional context
__cleanAttrs
: Clean packages that don't expose all internals #217243 (comment)Notify maintainers
@infinisil @DavHau may already be somewhat familiar with the concepts.
Metadata
Please run
nix-shell -p nix-info --run "nix-info -m"
and paste the result.Add a 👍 reaction to issues you find important.
The text was updated successfully, but these errors were encountered: