[RFC 0146] Meta.Categories, not Filesystem Directory Trees

[AndersonTorres]

This RFC proposes a new meta.categories attribute in order to categorize Nix packages, instead of using the filesystem dirtree.

Rendered!

[AndersonTorres]

The idea of this RFC was triggered by the RFC #140, by the way.

Ref.: #140 (comment)

[PetarKirov]

One of the advantages of meta.categories, not directly mentioned in the RFC (though could be thought of as a consequence of solving drawback 7) is that users will be able to browse for packages in search.nixos.org (and similar tools based meta) much more easily. Currently the file system hierarchy is only visible for people browsing the nixpkgs repo, while being a hidden implementation detail for all other users.

[jtojnar]

Another alternative might be an open taxonomy.

[fgaz]

+1. It would allow us to import the existing categories as they are, and thanks to nix we can map them to freedesktop categories in the category definition, like the current license-spdx mapping

[AndersonTorres]

Well, I believe I can produce some preliminar code on this regard.
Basically I can verbatim copy-paste the categories of Freedesktop, just for start.

[roberth]

Here's a largely uncontrolled categorization, although skewed by a built in scaffolding tool that only suggests some 20 categories: Haskell packages by category

My takeaways:

• listing an uncurated set of categories does not appear to be useful
  • curating and restricting are different ideas!
• some categories highlight library ecosystems (e.g. the Scotty category) - this could alternatively be captured by a related packages attribute (presumably out of scope, but worth considering)
• we might want to embed these categories into ours - if we go with some open categories

[AndersonTorres]

I have proposed something like "related categories" on the preliminary code:

NixOS/nixpkgs#230439

[ghost]

This is awesome.

[ghost]

Allowing to add, remove, or split categories would mean that any bikeshedding wouldn't block other processes, which is a good thing.

[bew]

What about ease/speed of finding all packages of a specific category or a set of categories ?

I mean: by using a list of categories it means that to find all compiler packages we would have to go through every packages categories list, and compare every elements of that list.
There is no fast hash lookup like meta.categories.compiler.

(not 100% sure about it) One way to make single category lookup fast could be to generate a lookup attrset like:

{
  meta.categories_lookup = { compiler = lib.categories.compiler; . . . };
}

Another way could be to NOT use a list of categories in the first place and directly use an attrset like that one above. And get the added benerit of inherit to easily write it (avoiding one more usage of with x; y):

{
  meta.categories = { inherit (lib.categories) compiler; . . . };
}

The only downside I see is that we loose the ordering of categories (not mentioned yet, but the first one could mark the pkg's primary category) on the other hand it removes a bikeshed waiting to happen (how to order categories).
I don't think ordering is that important though, and if we want to mark the primary category of a pkg there are other ways, like using a special category name meta.categories.primary = lib.categories.foobar.

To check if a package has a set of categories I don't think there's a better way than to check if the pkg is in the first category, then the second, etc..

What do you think?

[AndersonTorres]

Hum, the problem I can see is that it would be not orthogonal. Look:

{
. . .
  meta = {
    maintainers = [ maintainers.MeMyselfAndI ];
    categories = { inherit (lib.categories) compiler; }
  };
. . .
}

Why maintainers is a list while categories is a strange dictionary?

[bjornfor]

Why maintainers is a list while categories is a strange dictionary?

That can be solved by migrating maintainers list to a dictionary.

[AndersonTorres]

And meta.platfoms too?

nix-repl> :l <nixpkgs>
Added 16546 variables.

nix-repl> pkgs.bochs.meta.platforms
[ "i686-cygwin" "x86_64-cygwin" "x86_64-darwin" "i686-darwin" "aarch64-darwin" "armv7a-darwin" "i686-freebsd" "x86_64-freebsd" "x86_64-solaris" "aarch64-linux" "armv5tel-linux" "armv6l-linux" "armv7a-linux" "armv7l-linux" "i686-linux" "m68k-linux" "mipsel-linux" "mips64el-linux" "powerpc64-linux" "powerpc64le-linux" "riscv32-linux" "riscv64-linux" "s390-linux" "s390x-linux" "x86_64-linux" "aarch64-netbsd" "armv6l-netbsd" "armv7a-netbsd" "armv7l-netbsd" "i686-netbsd" "m68k-netbsd" "mipsel-netbsd" "powerpc-netbsd" "riscv32-netbsd" "riscv64-netbsd" "x86_64-netbsd" "i686-openbsd" "x86_64-openbsd" "x86_64-redox" ]

nix-repl> 

[7c6f434c]

One could switch meta to attrubute sets, but currently it's all lists, yeah, so makes sense to follow this tradition

[piegamesde]

The idea of this RFC was triggered by the RFC #140, by the way.

I'd like to have a section in this RFC discussing its relationship and synergies with RFC 140. Does one of them depend on the other? Or are they completely orthogonal to each other?

[AndersonTorres]

Completely orthogonal.

Certainly if this RFC solves a corner problem of that other #140 (losing the classification), but they have no dependencies.
This RFC merely decouples the classification from the filesystem.

[suhr]

For initial categorization one could extract categories from current package paths. So for example applications/editors/emacs get meta.categories = [application editor];

[KFearsoff]

Also, all-packages.nix is HUGE. 40306 lines on the master at the time of me writing this comment. No person can ever go through that manually, which means that one would have to use file searching functionality (which is very inconvenient for stuff like python3, which is often also an argument to a lot of other packages, and very error-prone because sometimes stuff has weird names). It also takes a while to load and parse.

[AndersonTorres]

For initial categorization one could extract categories from current package paths.

For now I have made a preliminary code‚ copying the Freedesktop.org menu entries. It is already in the PR queue for Nixpkgs acceptance.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-0146-fcp-meta-categories/47818/1

[7c6f434c]

So we are starting the Final Comment Period with disposition to accept, running until 2024-07-08

Discourse announcement: https://discourse.nixos.org/t/rfc-0146-fcp-meta-categories/47818

We are also now looking for volunteers for the category list maintenance team!

[tomodachi94]

I volunteer to be part of the Categorization Team. I have a strong background in ontology from my work on Wikidata, so I feel that I could meaningfully contribute here.

(This is the correct spot, right? 🙂)

[jopejoe1]

Would be intressted in joing the catagory maintenance team.

[Aleksanaa]

Would be intressted in joing the catagory maintenance team.

Me too.

[7c6f434c]

Re: team — @GetPsyched over at Discourse https://discourse.nixos.org/t/rfc-0146-fcp-meta-categories/47818/2 (as the majority of team joining interest comments are here, I make a reference not to forget)

[GetPsyched]

Apologies, I didn't want to create more noise on the RFC itself; however since we're nearing a merge soon, I guess this is fine.

[7c6f434c]

What you did is perfectly good, but my duty as a shepherd leader is to keep the situation easier to summarise!

(And yeah, we don't have and don't expect much else going here right now)

[GetPsyched]

RFCSC:

@7c6f434c, the FCP period has passed, can you confirm if this RFC is good to be merged?

[7c6f434c]

Yes, I think there have been no blocking objections (and some local criticisms have been addressed acceptably)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-07-08/48678/1