[RFC 0089] Collect non-source package meta attribute

[risicle]

Don't know if there's a way of reserving a number - I just picked 0088 as it looked like the latest.

Edit: oh - is it the PR number? Fixed.

See https://github.com/risicle/rfcs/blob/ris-from-source/rfcs/0089-collect-non-source-package-meta.md

[dotlambda]

There should also be allowNonSourcePredicate.

[dotlambda]

This doesn't seem flexible enough:

• Along the lines of the unfreeRedistributableFirmware license, there should be a way to label binary packages used in bootstrapping as such. Most users who don't want binary packages will be okay with their use in bootstrapping.
• Binary fonts (mentioned below) or graphics might also be acceptable.

Do you think allowNonSourcePredicate is enough for the latter?

[risicle]

Well, this hints slightly towards making it non-boolean, but if so, what should the options be? I think I considered finer classifications but saw it as something this could evolve into if/once we come to understand the problem better.

[samueldr]

A list of "types"? Because a package might contain different types!

{
  meta.sourceProvenance = with lib.sourceTypes; [
    sourceCode
    sourceFirmware
  ];
}

For an hypothetical driver with a code part, and a binary firmware part.

{
  meta.sourceProvenance = with lib.sourceTypes; [
    sourcePrebuiltFont
  ];
}

For a pre-built font.

(The naming here is clunky, I hope if gets the point across.)

---

EDIT: as a bonus, collecting what provenances a closure uses is a matter of adding all the lists together, and then getting the unique elements. And it is closer to the mechanisms used for licenses.

[dotlambda]

I think a list of types is usually too complex. Hence I suggest usually not using them, even though some packages might require them.
We sometimes have lists of licenses, though without clear semantics. For this attribute, a list should be interpreted as being built from all of those types combined.

I would call the attribute meta.builtFrom.
A normal xyz-bin package would, even if parts of it are built from source, have

meta.builtFrom = lib.builtFrom.binary

A binary font could have

meta.builtFrom = lib.builtFrom.binaryAsset

And a bootstrapping package would have

meta.builtFrom = lib.builtFrom.binaryBootstrap

The default would be

meta.builtFrom = lib.builtFrom.source

[samueldr]

While I agree that they are more complex, I don't think it's an issue.

Otherwise we're pushing the complexity into pre-declaring all possible combinations as part of the library representing the different types.

We need granularity here. Otherwise we'll have issues deciding which descriptor to use, and end up using the wrong generic "it's a binary", which will make filtering against or for a specific descriptor needlessly harder.

[kevincox]

We can do something "weird" and have lib.builtFrom.source = ["source"] so that you can easily do one or ++ two together. Or maybe better is lib.buildFrom.source = {source = true;} and then you can // them together but it is a bit easier for a predicate to filter the ones that you are okay with (and removes the irrelevant ordering).

Also I think instead of builtFrom it probably makes sense to call it nonSourceComponents and list the types of things that were not built from source where "types" could be things like code, assets, docs and similar.

I see this is somewhat against the "Why not isBinary?" below but I think it kinda agrees. nonSourceComponents = {} is the "purest" form and well understood. Then you just have to document the deviations.

[alyssais]

I really like samueldr's idea here. Another benefit of it would be that I'd be able to say "I don't want proprietary software on my computer, but it's okay if data files are CC-BY-ND" or something.

[risicle]

It's nice, my worry is just what we'd actually end up with given humans being humans, most people probably not that invested in understanding a complex tagging scheme well enough to perfectly represent the situation for a particular package.

This concern probably comes from my background in openstreetmap and the long, heated discussions that take place on the "tagging" mailing list debating a scheme that can perfectly represent most every situation, yet which ends up bearing little relation to what people actually map with in reality, because it's too complex and verbose.

I certainly think it's important to make the common cases have very concise representations, and also allow both coarse and fine granularity. If only fine granularities are allowed, it will deter most people from bothering to add an annotation at all. Coarse data is better than nothing and can always be refined.

[asymmetric]

I tend to agree with @risicle and @dotlambda here, in that I think we should at least start with a rougher schema, and wait until we've seen if we actually need a more refined one. I resonate with the risk @risicle sees, otherwise.

@dotlambda's proposal matches what I would do:

• If a package has both binary and source it will be marked as binary (i.e. we prefer a rougher schema, and err on the side of overapplying the "binary" label)
• it special cases bootstrap binaries and fonts
• Adds an allowNonSourcePredicate

I think this would be a great first step, and it would address the 2 points raised in the RFC:

• supply chain attacks
• patching

Other concerns, like wanting to only run free software + special casing some data files based on their license, I would leave out of this RFC, and potentially address in a follow-up one.

[marius851000]

Will this also concerned vendored code ? If the option of using a list for the provenance of source information, might it be relevant to include information if the source have vendored code (I here mean code that isn't created by the author of the source file, and that is unmodified, so that may be compiled in another unit, like for example rust package, with the VendorSha256 entry). It might be relevant to indicate this, as it mean overriding the library in nixpkgs may not update other package that vendor the library.

[risicle]

I think vendored code is beyond what I'd consider scope creep for this proposal.

[spacekookie]

This RFC is open for shepherd nominations now btw!

[alyssais]

Nominate myself.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/vision-for-nixpkgs/11950/9

[Mic92]

We still need more nomination for this RFC to proceed.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcs-looking-for-shepherds/14203/1

[asymmetric]

I’d like to be a shepherd on this one.

[aforemny]

I nominate myself

[Mic92]

We decided on the shepherd team: @asymmetric, @aforemny and @alyssais
with @asymmetric as shepherd leader.
@asymmetric could you propose a meeting (i.e. with https://www.when2meet.com/)

[asymmetric]

Thanks @risicle for the work you're putting into this. the RFC reads great!

Just to clarify, apart from getting this RFC through the finish line, are you still willing to go through (parts of) nixpkgs and mark non-exclusively-source packages as such?

I don't think this is a requirement, by the way - just wondering what practical outcomes we can expect to come out of this RFC. Even just arriving at consensus on the new attribute would be great.

[asymmetric]

What does opaque type mean in this context?

[risicle]

I think I meant "you shouldn't expect any specific behaviour from it other than being able to check for equality". At time of writing I think I was leaving open the possibility of having values which were able to understand their own hierarchy to some degree, though the more I think of it, it would probably be better to embed all notions of hierarchy in the allowNonSourcePredicate function to allow customization there too.

[asymmetric]

I would then rather change this to remove the reference to opaqueness, since it's rather.. opaque like this.

Do you agree?

[risicle]

Yes, I'm willing to do a non-comprehensive pass of nixpkgs looking for the low-hanging fruit and more obvious cases.

[Mic92]

@asymmetric with the updated rfc text, do you think it's time for a new meeting?

[asymmetric]

I think that, apart from the nit I mentioned in this review, this is good to go.

@risicle can you make a quick edit addressing this point? I think after that, we can move ahead.

[asymmetric]

I would then rather change this to remove the reference to opaqueness, since it's rather.. opaque like this.

Do you agree?

[asymmetric]

Also, sorry for the delay everyone, life took over.

[risicle]

Done.

[asymmetric]

I propose this RFC move to FCP, with intention to merge.

[aforemny]

I propose this RFC move to FCP, with intention to merge.

I second that! :-)

[alyssais]

I agree, let's go to FCP!

[asymmetric]

@NixOS/rfc-steering-committee all shepherds have signaled their intention to move to FCP.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-0089-fcp-collect-non-source-package-meta-attribute/17350/1

[risicle]

An initial implementation: NixOS/nixpkgs#161098

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/packages-marked-as-broken-should-come-with-an-explanation/19187/26