RFC 140 migration plan

[infinisil]

Here's my current plan to implement part 2 of RFC 140:

0. Implement a strategy to make it easy to increase the strictness of the pkgs/by-name CI check:
   • A strategy for evolving the pkgs/by-name CI checks #256788
   • tests.nixpkgs-check-by-name: Implement gradual empty arg check migration #272395
1. Require packages that use pkgs/by-name to not be defined with an { } argument in all-packages.nix:
   • Implement check to disallow pkgs/by-name package definitions in all-packages.nix with an empty argument #256792
   • tests.nixpkgs-check-by-name: Implement gradual empty arg check migration #272395
   • pkgs/by-name: Enable gradual migration checks and add run-local.sh #274591
2. Require new packages using callPackage to use pkgs/by-name
   • check-by-name: Refactor to prepare for enforcing pkgs/by-name, make --base required #278805
   • [RFC 140 2a] pkgs/by-name: Enforce for new packages #275539
3. Try to migrate all packages that don't cause failures or output path changes
   • [RFC 140 2b] pkgs/by-name: Automated migration #211832

Ping the Nixpkgs Architecture Team, @roberth @Ericson2314 @tomberek @aakropotkin @DavHau @phaer

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-10-03-nixpkgs-architecture-team-meeting-44/33798/1

[SuperSandro2000]

What if I don't want my packages to be trapped in this experimental feature? It doesn't solve any problem for me personally, so it can only create new ones like breaking this fancy prepare-commit-msg hook I am using for a year (turns out I was confusing things) or it will likely break brittle update scripts (like the one for ArchiSteamFarm which are already pretty hard to test and I honestly don't know if it would work for r-rynatm even.

[phaer]

What if I don't want my packages to be trapped in this experimental feature?

Then you would have had about 7 months of time between the initial proposal and the end of FCP to add your concerns to the extensive discussion. You did so just now, so I would like to kindly ask you to keep this kind of the debate out of this thread which explicitly is about the implementation only. :)

[ghost]

What if I don't want my packages

We don't have package owners. Everybody is allowed to submit PRs against any package.

to be trapped in this experimental feature?

This isn't an experimental feature; it's the future and it's long overdue.

I have plenty of automation that gets broken constantly because there is no reliable way to map from attrnames to files. Overlays get silently disabled when I rebase nixpkgs because there's no restrictions on what code can see which files -- I can't count the number of times my override-patches have been silently dropped because somebody rewired all-packages.nix and found a new way to reference the package's definition without going through the attrname I'd overridden. It's madness.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-28-nixpkgs-architecture-team-meeting-46/36171/1

[SuperSandro2000]

I have plenty of automation that gets broken constantly because there is no reliable way to map from attrnames to files.

There will be now for packages that are in this but there will be still plenty of interesting packages left, especially the more complicated ones.

Overlays get silently disabled when I rebase nixpkgs because there's no restrictions on what code can see which files -- I can't count the number of times my override-patches have been silently dropped because somebody rewired all-packages.nix and found a new way to reference the package's definition without going through the attrname I'd overridden. It's madness.

Except for versioned attrs people should create aliases which should make this manageable.

I also have a good forest of extra commits and overlays and I mostly touch them when things changed in the exact same files I was changing. Changing some things with overrides is a bit more complicated than directly touching the code especially if you plan to upstream it anyway.

With this change still everything sees everything. I don't see how that definitely solves your problems.

We don't have package owners. Everybody is allowed to submit PRs against any package.

If I tell that some people about some of their beloved packages then they would walk up the next wall.

Also this contradicts what you wrote below. If anyone can change anything at will then sure things break for others break left, right and center. Be it your overlays or my commits.

[ehmry]

What happens to libsForQt5.callPackage ././. { } calls in all-packages.nix?

[infinisil]

@ehmry Nothing, see https://github.com/NixOS/rfcs/blob/master/rfcs/0140-simple-package-paths.md#drawbacks or https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name#limitations

[infinisil]

This PR makes progress on points 0. and 1.: #272395

[blaggacao]

@infinisil Maybe there's a way to reconcile the concerns about breaking update scripts. A regex fishing for ../.. patterns could with some probablility identify update scripts that break out of their unit folder-boundary and would thus likely break after migration. These could then be put in another shortlist and scheduled for manual migration. Would that be a viable way out for keeping the gameplan and reasonably addressing @SuperSandro2000 's concerns?

[infinisil]

@blaggacao I think with the gradual transition enabled by approaches like #272395 that's totally a possibility. Once I get to running the automation, I'll do some quick grepping for ../.., update.sh and updateScript to hopefully catch most. Those would then just be excluded from the automated migration and have to be fixed manually.

[infinisil]

Two new PRs:

• pkgs/by-name: Enable gradual migration checks and add run-local.sh #274591 is ready to be reviewed/merged. It enables tests.nixpkgs-check-by-name: Implement gradual empty arg check migration #272395 in CI
• [RFC 140 2a] pkgs/by-name: Enforce for new packages #275539 enforces new packages to use pkgs/by-name

[katexochen]

3. Try to migrate all packages that don't cause failures or output path changes
   [RFC 140 2b] pkgs/by-name: Automated migration #211832

Is the automatic migration still planned? Is this the next step, or do we need anything else?

[infinisil]

@katexochen Indeed I'm still planning to do that, I'll try to get it done as soon as possible. It's the next and final step 😃

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-merge-bot-testing-and-plan/39824/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-54/39990/1

[Atemu]

Hi, what's the progress on this?

[infinisil]

@Atemu In recent weeks we moved the tooling to a separate repo and got more people involved in contributions, so development speed is going faster now: https://github.com/NixOS/nixpkgs-check-by-name

Still not done, but work in on-going :)

[Aleksanaa]

What about the automatic migration? Should we still wait now, or slowly migrate these packages during the update process?

[infinisil]

Yeah that's still planned too, so I'd still wait. There's no real harm in delaying that until after, when we'll see the ones that can't be automatically migrated.

[infinisil]

I just wrote down this thorough description of the current state and plan regarding the implementation of the automated migration: NixOS/nixpkgs-check-by-name#56

tl;dr: It's quite tricky but doable!