Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0146] Meta.Categories, not Filesystem Directory Trees #146

Merged
merged 39 commits into from
Jul 8, 2024
Merged
Changes from 35 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
610c110
Meta.Categories, not Filesystem Directory Trees
AndersonTorres Apr 23, 2023
415c97a
Whitespace cleanup
AndersonTorres Apr 23, 2023
a908922
Add a short answer to the bikeshedding problem
AndersonTorres Apr 23, 2023
f2eac8d
Add a short line on "Do nothing" alternative
AndersonTorres Apr 23, 2023
5444d90
Extend an answer for the "Ignore/nuke" alternative
AndersonTorres Apr 23, 2023
93ef176
Add "update ci" to future work
AndersonTorres May 6, 2023
2bf1c25
Add a repl-like interaction example
AndersonTorres May 6, 2023
e29a413
Add more arguments for categorization and against its nuking
AndersonTorres May 10, 2023
79b92e9
small rewording
AndersonTorres May 10, 2023
004fc01
Add an option of category data structure
AndersonTorres May 10, 2023
6289abe
reorder arguments against nuking
AndersonTorres May 12, 2023
602e521
add argument for usefulness of categorization
AndersonTorres May 12, 2023
fd1d6af
add drawback
AndersonTorres May 20, 2023
c30ff6c
rework nuke argument
AndersonTorres May 20, 2023
907d510
update metainfo - shepherd team and leader
AndersonTorres Aug 24, 2023
4d02382
Add prior art section
AndersonTorres Oct 26, 2023
0a868c6
typo
AndersonTorres May 24, 2024
e258926
Categorization Team
AndersonTorres May 24, 2024
98a8348
Remove the optional data structure
AndersonTorres May 24, 2024
cc8caa3
typo
AndersonTorres May 24, 2024
618dc67
reword the creation of a team
AndersonTorres May 24, 2024
abbf7fd
Debtags FAQ
AndersonTorres May 24, 2024
0e21673
Update rules and duties of categorization team
AndersonTorres May 24, 2024
21ec340
The team shall have authority to carry out their duties
AndersonTorres May 24, 2024
a18c24a
A section for the team
AndersonTorres May 24, 2024
f771094
Appstream as prior art
AndersonTorres May 24, 2024
eb079f6
Section for code implementation
AndersonTorres May 24, 2024
0ad89e7
Move categorization team to implementation
AndersonTorres May 24, 2024
fa75eec
Update future work
AndersonTorres May 24, 2024
ee8633c
A hybrid approach to be considered by the future team
AndersonTorres May 24, 2024
2dcb0ed
extra duties for the team
AndersonTorres May 24, 2024
e0ee96f
reword duties from team
AndersonTorres May 24, 2024
a029a22
typo
AndersonTorres May 24, 2024
172cc8e
A semantic detail: treat the first element of meta.categories as most…
AndersonTorres May 24, 2024
fdc2424
Move hybrid approach to alternatives section
AndersonTorres May 24, 2024
11a7757
identify AndersonTorres' tag
AndersonTorres Jun 28, 2024
bcc2444
Suggestions from FCP
AndersonTorres Jun 28, 2024
9979bef
remove infinisil from shepherd team
AndersonTorres Jun 28, 2024
f643103
add an extra reference to the categorization team
AndersonTorres Jul 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
354 changes: 354 additions & 0 deletions rfcs/0146-meta-categories.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,354 @@
---
feature: Decouple filesystem from categorization
start-date: 2023-04-23
author: Anderson Torres
co-authors:
shepherd-team: @7c6f434c @natsukium @fgaz @infinisil
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
shepherd-leader: @7c6f434c
related-issues: (will contain links to implementation PRs)
---

# Summary
[summary]: #summary

Deploy a new method of categorization for the packages maintained by Nixpkgs,
not relying on filesystem idiosyncrasies.

# Motivation
[motivation]: #motivation

Currently, Nixpkgs uses the filesystem, or more accurately, the directory tree
layout in order to informally categorize the softwares it packages, as described
in the [Hierarchy](https://nixos.org/manual/nixpkgs/stable/#sec-hierarchy)
section of Nixpkgs manual.

This is a simple, easy to understand and consecrated-by-use method of
categorization, partially employed by many other package managers like GNU Guix
and NetBSD pkgsrc.

However this system of categorization has serious problems:

1. It is bounded by the constraints imposed by the filesystem.

- Restrictions on filenames, subdirectory tree depth, permissions, inodes,
quotas, and many other things.
- Some of these restrictions are not well documented and are found simply
by "bumping" on them.
- The restrictions can vary on an implementation basis.
- Some filesystems have more restrictions or less features than others,
forcing an uncomfortable lowest common denominator.
- Some operating systems can impose additional constraints over otherwise
full-featured filesystems because of backwards compatibility (8 dot
3, anyone?).

2. It requires a local checkout of the tree.

Certainly this checkout can be "cached" using some form of `find . >
/tmp/pkgs-listing.txt`, or more sophisticated solutions like `locate +
updatedb`. Nonetheless such solutions still require access to a fresh,
updated copy of the Nixpkgs tree.

3. The creation of a new category - and more generally the manipulation of
categories - requires an unpleaseant task of renaming and eventually patching
many seemingly unrelated files.

- Moving files around Nixpkgs codebase requires updating their forward and
backward references.
- Especially in some auxiliary tools like editor plugins, testing suites,
autoupdate scripts and so on.
- Rewriting `all-packages.nix` can be error-prone (even using Metapad) and it

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, all-packages.nix is HUGE. 40306 lines on the master at the time of me writing this comment. No person can ever go through that manually, which means that one would have to use file searching functionality (which is very inconvenient for stuff like python3, which is often also an argument to a lot of other packages, and very error-prone because sometimes stuff has weird names). It also takes a while to load and parse.

can generate huge, noisy patches.

4. There is no convenient way to use multivalued categorization.

A piece of software can fulfill many categories; e.g.
- an educational game
- a console emulator (vs. a PC emulator)
- and a special-purpose programming language (say, a smart-contracts one).

The current one-size-fits-all restriction is artificial, imposes unreasonable
limitations and results in incomplete and confusing information.

- No, symlinks or hardlinks are not convenient for this purpose; not all
environments support them (falling on the "less features than others"
problem expressed before) and they convey nothing besides confusion - just
think about writing the corresponding entry in `all-packages.nix`.

5. It puts over the (possibly human) package writer the mental load of where to
put the files on the filesystem hierarchy, deviating them from the job of
really writing them.

- Or just taking the shortest path and throw it on a folder under `misc`.

6. It "locks" the filesystem, preventing its usage for other, more sensible
purposes.

7. The most important: the categorization is not discoverable via Nix language
infrastructure.

Indeed there is no higher level way to query about such categories besides
the one described in the bullet 2 above.
Comment on lines +86 to +90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idk, this might be a bit off topic, but one use case this RFC would enable is to have something like nix-shell -p but for launcher like rofi, dmenu, etc... because I would imagine that there would be a tag for graphical applications and then one can filter for those and dynamically use nix to load them like one would with a nix-shell.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how to word such a use case carefully, given that many packages contain multiple GUI applications that can be launched without arguments, and then some more programs where command-line arguments are expected.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it looks like an extra application that uses the Nixpkgs metadata as a filter. Something like nix-index with extras as a backend for a launcher.


In light of such a bunch of problems, this RFC proposes a novel alternative to
the above mess: new `meta` attributes.

AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
# Detailed design
[design]: #detailed-design

## Code Implementation
[code-implementation]: #code-implementation

A new attribute, `meta.categories`, will be included for every Nix expression
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
living inside Nixpkgs.

This attribute will be a list, whose elements are one of the possible elements
of the `lib.categories` set.

A typical snippet of `lib.categories` will be similar to:

```nix
{
assembler = {
name = "Assembler";
description = ''
A program that converts text written in assembly language to binary code.
'';
};

compiler = {
name = "Compiler";
description = ''
A program that converts a source from a language to another, usually from
a higher, human-readable level to a lower, machine level.
'';
};

font = {
name = "Font";
description = ''
A set of files that defines a set of graphically-related glyphs.
'';
};

game = {
name = "Game";
description = ''
A program developed with entertainment in mind.
'';
};

interpreter = {
name = "Interpreter";
description = ''
A program that directly executes instructions written in a programming
language, without requiring compilation into the native machine language.
'';
};

```

### Semantic Details
[semantic-details]: #semantic-details

Given that `meta.categories` is implemented as a list, it is interesting to
treat the first element of this list as the "most important" categorization, the
one that mostly identifies with the software being classified.

## Categorization Team
[categorization-team]: #categorization-team

Given the typical complexities that arise from categorization, and expecting
that regular maintainers are not expected to understand its minuteness
(according to the experience from [Debtags
Team](https://wiki.debian.org/Debtags/FAQ#Why_don.27t_you_just_ask_the_maintainers_to_tag_their_own_packages.3F)),
it is strongly recommended the creation of a team entrusted with authority to
manage issues related to categorization and carry their corresponding duties.

# Examples and Interactions
[examples-and-interactions]: #examples-and-interactions

In file bochs/default.nix:

```nix
stdenv.mkDerivation {

. . .

meta = {
. . .
categories = with lib.categories; [ emulator debugger ];
. . .
};
};
}

```

In a `nix repl`:

```
nix-repl> :l <nixpkgs>
Added XXXXXX variables.

nix-repl> pkgs.bochs.meta.categories
[ { ... } ]

nix-repl> map (z: z.name) pkgs.bochs.meta.categories
[ "debugger" "emulator" ]
```

# Drawbacks
[drawbacks]: #drawbacks
Copy link

@bew bew May 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about ease/speed of finding all packages of a specific category or a set of categories ?

I mean: by using a list of categories it means that to find all compiler packages we would have to go through every packages categories list, and compare every elements of that list.
There is no fast hash lookup like meta.categories.compiler.

(not 100% sure about it) One way to make single category lookup fast could be to generate a lookup attrset like:

{
  meta.categories_lookup = { compiler = lib.categories.compiler; . . . };
}

Another way could be to NOT use a list of categories in the first place and directly use an attrset like that one above. And get the added benerit of inherit to easily write it (avoiding one more usage of with x; y):

{
  meta.categories = { inherit (lib.categories) compiler; . . . };
}

The only downside I see is that we loose the ordering of categories (not mentioned yet, but the first one could mark the pkg's primary category) on the other hand it removes a bikeshed waiting to happen (how to order categories).
I don't think ordering is that important though, and if we want to mark the primary category of a pkg there are other ways, like using a special category name meta.categories.primary = lib.categories.foobar.

To check if a package has a set of categories I don't think there's a better way than to check if the pkg is in the first category, then the second, etc..

What do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, the problem I can see is that it would be not orthogonal. Look:

{
. . .
  meta = {
    maintainers = [ maintainers.MeMyselfAndI ];
    categories = { inherit (lib.categories) compiler; }
  };
. . .
}

Why maintainers is a list while categories is a strange dictionary?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why maintainers is a list while categories is a strange dictionary?

That can be solved by migrating maintainers list to a dictionary.

Copy link
Member Author

@AndersonTorres AndersonTorres May 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And meta.platfoms too?

nix-repl> :l <nixpkgs>
Added 16546 variables.

nix-repl> pkgs.bochs.meta.platforms
[ "i686-cygwin" "x86_64-cygwin" "x86_64-darwin" "i686-darwin" "aarch64-darwin" "armv7a-darwin" "i686-freebsd" "x86_64-freebsd" "x86_64-solaris" "aarch64-linux" "armv5tel-linux" "armv6l-linux" "armv7a-linux" "armv7l-linux" "i686-linux" "m68k-linux" "mipsel-linux" "mips64el-linux" "powerpc64-linux" "powerpc64le-linux" "riscv32-linux" "riscv64-linux" "s390-linux" "s390x-linux" "x86_64-linux" "aarch64-netbsd" "armv6l-netbsd" "armv7a-netbsd" "armv7l-netbsd" "i686-netbsd" "m68k-netbsd" "mipsel-netbsd" "powerpc-netbsd" "riscv32-netbsd" "riscv64-netbsd" "x86_64-netbsd" "i686-openbsd" "x86_64-openbsd" "x86_64-redox" ]

nix-repl> 

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One could switch meta to attrubute sets, but currently it's all lists, yeah, so makes sense to follow this tradition


The most immediate drawbacks are:

1. A huge treewide edit of Nixpkgs

On the other hand, this is easily sprintable and amenable to automation.

2. Bikeshedding

How many and which categories we should create? Can we expand them later?
Comment on lines +209 to +211
Copy link

@ghost ghost May 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allowing to add, remove, or split categories would mean that any bikeshedding wouldn't block other processes, which is a good thing.


For start, we can follow/take inspiration from many of the already existing
categories sets and add extra ones when the needs arise. Indeed, it is way
easier to create such categories using Nix language when compared to other
software collections.

3. Superfluous

It can be argued that there are other ways to discover similar or related
package sets, like Repology.

However, this argument is a bit circular, because e.g. the classification
shown by Repology effectively replicates the classification done by the many
software collections in its catalog. Therefore, relying in Repology merely
transfers the question to external sources.

Further it becomes more pronounced when we take into account the fact Nixpkgs
is top 1 of most Repology statistics. The expected outcome, therefore, should
be precisely the opposite: Nixpkgs being _the_ source of structured metainfo
for other software collections.

# Alternatives
[alternatives]: #alternatives

1. Do nothing

This will exacerbate the problems already listed.

2. Ignore/nuke the categorization completely

This is an alternative worthy of some consideration. After all,
categorization is not without its problems, as shown above. Removing or
ignoring classification removes all problems.

However, there are good reasons to keep the categorization:

- The complete removal of categorization is too harsh. A solution that keeps
and enhances the categorization is way more preferrable than one that nukes
it completely.

- As said before, the categorization is already present; this RFC proposes to
expose it to a higher level, in a structured, more discoverable format.

- Categorization is very traditional among software collections. Many of them
are doing this just fine for years on end, and Nixpkgs can imitate them
easily - and even surpass them, given the benefits of Nix language
machinery.

- Categorization is useful in many scenarios and use cases - indeed they
are ubiquitous in software world:
- specialized search engines (from Repology to MELPA)
- code forges, from Sourceforge to Gitlab
- as said above, software collections from pkgsrc to slackbuilds
- organization and preservation (as Software Heritage)

3. Debtags/Appstream hybrid approach

A hybrid approach for code implementation would be implement two meta
attributes, namely

- `meta.categories` for Appstream-based categories
- the corresponding `lib.categories` should follow Appstream closely, with
few room to custom/extra categories
- `meta.tags` for Debtags-like tags
- while being inspired from the venerable Debtags work, the corresponding
`lib.tags` is completely free to modify and even divert from Debtags,
following its own way
- generally speaking, `lib.tags` should be less bureaucratic than
`lib.categories`

However, this approach arguably elevates the complexity of the whole work, and
adds too much redundancy.

# Prior art
[prior-art]: #prior-art

As said above, categorization is very traditional among software collections. It
is not hard to cite examples in this arena; the most interesting ones I have
found are listed below (linked at [references section](#references)):

- FreeBSD Ports;
- Debtags;
- Appstream Project;

AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another alternative might be an open taxonomy.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. It would allow us to import the existing categories as they are, and thanks to nix we can map them to freedesktop categories in the category definition, like the current license-spdx mapping

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I believe I can produce some preliminar code on this regard.
Basically I can verbatim copy-paste the categories of Freedesktop, just for start.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a largely uncontrolled categorization, although skewed by a built in scaffolding tool that only suggests some 20 categories: Haskell packages by category

My takeaways:

  • listing an uncurated set of categories does not appear to be useful
    • curating and restricting are different ideas!
  • some categories highlight library ecosystems (e.g. the Scotty category) - this could alternatively be captured by a related packages attribute (presumably out of scope, but worth considering)
  • we might want to embed these categories into ours - if we go with some open categories

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have proposed something like "related categories" on the preliminary code:

NixOS/nixpkgs#230439

# Unresolved questions
[unresolved]: #unresolved-questions

There are some still unsolved issues:
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

- What data structure is suitable to represent a category?

- For now we stick to the most natural: a set `{ name, description }`.

- Should we have a set of primary, "most important" categories with mandatory
status, in the sense each package should set at least one of them?
- The answer is most certainly positive.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

# Future work
[future]: #future-work

- Create the [categorization team](#categorization-team)
- Carry out the duties correlated to categorization, including but not limited
to:

- Decide between possibilities of implementation;
- Documentation updates;
- Category curation, integration and updates;
- Continuous Integration updates and adaptations;
- Coordinaton of efforts to import, integrate and update categorization of
packages;
- Litigations and disputations:
- Solve them, especially in corner cases;
- Enforce implementation issues
- Decide when a CI check should be converted to block

# References
[references]: #references

- [Desktop Menu
Specification](https://specifications.freedesktop.org/menu-spec/latest/);
specifically,
- [Main
categories](https://specifications.freedesktop.org/menu-spec/latest/apa.html)
- [Additional
categories](https://specifications.freedesktop.org/menu-spec/latest/apas02.html)
- [Reserved
categories](https://specifications.freedesktop.org/menu-spec/latest/apas03.html)

- [Appstream](https://www.freedesktop.org/wiki/Distributions/AppStream/)

- [Debtags](https://wiki.debian.org/Debtags)

- [Debtags FAQ](https://wiki.debian.org/Debtags/FAQ)

- [NetBSD pkgsrc guide](https://www.netbsd.org/docs/pkgsrc/)
- Especially, [Chapter 12, Section
1](https://www.netbsd.org/docs/pkgsrc/components.html#components.Makefile)
contains a short list of CATEGORIES.

- [FreeBSD Porters
Handbook](https://docs.freebsd.org/en/books/porters-handbook/)
- Especially
[Categories](https://docs.freebsd.org/en/books/porters-handbook/makefiles/#porting-categories)