-
-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC 0099] Change default shell from bash to oil #99
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
--- | ||
feature: oil-shell | ||
start-date: 2021-08-11 | ||
author: Raphael Megzari (happysalada) | ||
co-authors: (find a buddy later to help out with the RFC) | ||
shepherd-team: Eelco Dolstra (edolstra), Kevin Cox (kevincox), Las Safin (L-as) | ||
shepherd-leader: Kevin Cox (kevincox) | ||
related-issues: | ||
--- | ||
|
||
# Summary | ||
|
||
[summary]: #summary | ||
|
||
Use oil as the stdenv shell in nixos. Oil aims to be compatible with bash while trying to bring real programming language features (like hashmap or dicts) to the shell. | ||
|
||
## Motivation | ||
|
||
[motivation]: #motivation | ||
|
||
Two primary motivations | ||
|
||
- Removing the footguns in bash that even experienced programmers find painful | ||
- Making shell scripts more powerfull to remove the need to bring additional languages for scripting. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What occasions are you referring to here where we bring in other languages for scripting? As far as I am aware stdenv and setup hooks are 100% bash. Some core nixpkgs builders are implemented in Perl, the question here is, what we'd gain from rewriting them in oil. As it stands the user base of Oil is quite small. In any case, I'd appreciate a clarification here, maybe even some examples! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The main thing I'm referring to is the One more point I have against bash is the error handling. If we had a better way of handling errors, it wouldn't be so bad to try to debug why a modification in stdenv breaks something. I'm not 100% that oil will be able to solve this, but normal programming language give you a little easier interface to work with error handling. The last point I have is about trying to restrict the numbers of things you need to learn by heart on a language. There are so many pitfals, that we run into experienced programmers making mistakes in nixos modules or packages. I'm hoping that there are less things to learn with oil. (less rules, make it harder to mess up). |
||
|
||
## Detailed design | ||
|
||
[design]: #detailed-design | ||
|
||
The [oil shell](https://www.oilshell.org/) has two parts osh and the oil language. Osh is compatible with bash and posix and it's goal is to run existing shell scripts. The oil language is a brand new incompatible language. The idea is to fix more than four decades of accumulated warts in the Unix shell. Many Unix users are angry that shell is so difficult, and Oil aims to fix that. Those definitions were taken verbatim from [reference](https://www.oilshell.org/blog/2021/01/why-a-new-shell.html) | ||
|
||
Regarding oil | ||
|
||
A high level overview is that it has a syntax similar to python, it brings dictionaries (hashmaps). For more details check the following posts | ||
|
||
- [This post](https://www.oilshell.org/blog/2021/01/why-a-new-shell.html) aims at describing the goals in trying to create a new shell and the alternatives. | ||
- [This post](https://www.oilshell.org/blog/2020/01/simplest-explanation.html) aims to provide a simple explanation of what oil is. | ||
- [This post](https://www.oilshell.org/release/latest/doc/idioms.html) shows different oil idioms and how they fix some of bash problems. | ||
|
||
There would be two steps to make the transition from bash to oil. | ||
|
||
- From bash to osh. There is one incompatibility that came up so far. osh does not handle the `-i` flag on local and declare. The transition involves rewriting some shell scripts to get rid of the `-i` flag. There are about 5-10 uses in stdenv, and I've started PRs to remove them. After those changes, it would be possible to switch from bash to osh and test this for a while to verify there are no regression. | ||
- From osh to oil. It will require to modify stdenv to be able to run with the `strict_errexit` flag. The changes will be a little more involved, but as long as nobody uses oil specific features, we could retain compatibility with bash and posix. The exact quantity of work involved is unknown. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps you can clarify the meaning of this statement, as the way it is currently written, it seems to defeat the purpose of this whole idea. Why rewrite the standard env if we don't actually wish to leverage any of oils specific features? If that's not what you meant to convey, then perhaps this segment should just be removed? (I'm referring to line 43 specifically) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What I mean by this is that the changes to make the switch possible are a little more involved.
|
||
|
||
One more things to consider here is that changes to stdenv are relatively slow and costly. Even for some seemingly simple changes they can break things in unexpected ways. One example was trying to split buildFlagsArray on whitespace, some packages use newlines for the split. The tests seemed to pass and breakage was discovered much later. For this reason, changes in stdenv can only be made sure to work by triggering a full hydra rebuild which takes several days. Even after that because of flaky tests, it's hard to be sure nothing was broken by the actual changes. | ||
|
||
[reference](https://github.com/oilshell/oil/wiki/Migration-Guide) | ||
|
||
Some more background context. Originally this is coming from this [PR](https://github.com/NixOS/nixpkgs/pull/105233) by zimbatm. After seeing this, I thought I would implement the changes needed to make the switch and generate a discussion in a PR. Somebody brought to my attention that this change is significant enought that it needs to go through an RFC. | ||
|
||
## Examples and Interactions | ||
|
||
[examples-and-interactions]: #examples-and-interactions | ||
|
||
Some extracts from the post [why use oil](https://www.oilshell.org/why.html) | ||
|
||
- better error handling. One of the confusing things about bash is how to handle errors in scripts. `err_exit` was a try at improving that but it has many pitfalls. Oil aims to have more straightforward error handling. Any error will exit, if you do not want that behavior there is the `try my_function` form. | ||
|
||
`mkdir /tmp/dest && cp foo /tmp/dest` becomes simply | ||
|
||
```Shell | ||
mkdir /tmp/dest | ||
cp foo /tmp/dest | ||
``` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The passage above demonstrates that the semantics of the former and the latter statements provided are equivalent with oilshell, correct? This could use some clarification - 'x becomes y', I don't think, is sufficiently clear. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You're right this isn't super clear. |
||
|
||
- Introduce functions that need to explicitely define their parameters | ||
|
||
```Shell | ||
proc f(first, @rest) { # @ means "the rest of the arguments" | ||
write -- $first | ||
write -- @rest # @ means "splice this array" | ||
} | ||
``` | ||
|
||
instead of the traditional | ||
|
||
```Shell | ||
f() { | ||
echo $1 | ||
shift 1 | ||
echo "$@" | ||
} | ||
``` | ||
|
||
those functions are called procs and their variables don't mess with their outer scope. | ||
|
||
- Oil has proper associative arrays (dictionaries or hashmaps) that don't have the problems that bash's have. | ||
|
||
- No more quotes to prevent splitting | ||
|
||
## Drawbacks | ||
|
||
[drawbacks]: #drawbacks | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What I find a benefit of rougly-POSIX-shell, and maybe even Bash outright, is that upstream build system has a high chance of including some shell pieces, and sometimes we need to understand it, and switching between that and a similar but different language stdenv code sounds like a path to confusion — maybe slightly more confusion than using a conspiciously different language like Python or Lua. (But that would have higher migration costs) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As stated earlier in the proposal, there is no need to modify most bash scripts used for build systems. Think of Oil as a superset of POSIX - it supports everything Bash does with the same semantics, then introduces new, useful language features without regressing on legacy code. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How is Oil's architecture support? I know there is some interest in running Nix on chips different from aarch-64 and x86-64, so we should reasonably ensure that there is interest in actively testing for and supporting select alternative architectures going forward (i.e. RISC-V) so as not to disparage those valuable efforts. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I understand the confusing part.
Regarding the architecture support I've never seen anything on it. As the code moves to c++ the story should get better in that regard. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One more thing that is probably not a big issue but I want just to mention it.. We know that bash → osh has few «loud» issues like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Your concern makes sense. I think we can only make changes by testing the rebuild of the whole of stdenv to make sure nothing breaks silently. It won't provide any guarantees that nothing will break in the future, however, it should be a pretty good sign that the status quo is good. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just |
||
|
||
- Currently oilshell has a bus factor of 1. While Andrew has been very motivated and responsive thus far, this could change in the future. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IIRC @zimbatm suggested to try out oil as language for the stdenv rather than bash and I actually like the idea. However I think that this is the biggest risk we have. Shifts of interest or (unfortunately also) burnout do happen in open source. The worst case is that the "heart" of the nixpkgs project depends on an unmaintained, effectively dead language at some point in the future. That said, I think that we should discuss how we can minimize this risk, e.g. by finding people who are willing to help out / can (co-)maintain this project or by sponsoring this project via the NixOS foundation (these are just two ideas from the top of my head, perhaps someone else has better ideas :)) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One thing to mitigate this could be that we could have a "long" period of keeping compatibility with both language and only make the switch when more people have been involved with oil. After the flag feature was proposed, I think enabling package maintainers early to choose another shell for evaluation of their package could be an interesting experiment. As oil's user base increases, contributors should increase as well. We could make it a condition to switch stdenv only when a "sufficent" numbers of contributors have appeared. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. An argument could be made for being worth the switch even if it gets stuck in some state — it's not like we eagerly adopt new features of Bash immediately. But then indeed there should be evaluation from this point of view (maybe including some estimation of the burden of maintaining buildability) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree that this is the biggest concern. Sure, if Oil disappears we won't be "screwed" overnight but over time bugs will be found and security issues will need to be fixed. Oil's dependencies will need to be updated, including its build tools. Sure, in theory we can keep whatever old deps Oil needs forever but that is a significant maintenance burden. I don't want Oil to be forcing us to keep python 3 around when the world is on python 5. There are alternatives to consider. ZSH avoids some of the biggest issues by not requiring variables to be quoted, but it isn't as nice as Oil. I think it is a clear improvement from bash but is it worth the migration? Oil seems clearly worth the migration given the current state of the project but again, I don't want to bet on a 1-dev project and need to migrate back to bash (or zsh) one day. I also don't think that long-term multi-shell support is a great option. I think if we are going to change the shell we need to expect to complete the migration without any significant "waiting" period. I'm not saying that it needs to be done overnight, or that we can't change our mind and rollback, but I am against the long-term multi shell state due to the confusion and maintenance burden it causes. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Well, we are not talking CGI or user shell, intentional attacks on stdenv are a pretty niche consideration.
Current track record of PyPy suggests that if there is nothing CPython-only, the risk is limited. But of course even just a big mass of C++ having lost maintenance can become an annoying liability. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Completely correct. We just have to weigh the potential cost of carrying this project vs the savings of having a better shell that helps make nixpkgs more reliable and productive. In my mind it is a very tough call, but I think I am leaning in favour. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's unclear to me whether a better backwards compatible shell replacement exists - reading the oil shell blog reveals a ton of time and research sunk into the POSIX spec and the behavior of common shells (not always POSIX compliant!) to optimize for backwards compatibility above all else. I don't see a better alternative shell for Nix. That said, picking up Oil with a single developer and before a 1.0 release is a significant risk, so I agree that we should somehow ensure that development is funded and actively contributed to. Is the current developer aware of Nix and our interest in the project? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the alias andychu that commented on this RFC is the person behind oil. There are also some nix users who have contributed to oil https://github.com/abathur is one of them. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this objection is reasonable. However, at the same time, GNU Bash has very few maintainers (I cannot say if there are more than one due to the unclear/opaque dev process and git commits). The main advantage with Bash in this respect is that so many people rely on it that it is likely some maintenance will happen indefinitely regardless of who does it. I don't think this is necessarily a major problem. |
||
- Oil shell is pre 1.0 so the syntax might change. | ||
- Parts of oil are rewritten in c++, while this could be good for performance, new user might discover bugs that are really hard to debug. | ||
- Oil has had little usage so far, there might be bugs that will just be discovered by having a large user base. | ||
- Using osh, could be straightforward, but using oil might take more work as basically the existing bash code need to rewritten with `strict_errexit`. | ||
- The oil documentation is lacking at the moment. | ||
- Oil has really lofty goals, which I think it's good, but some potential solutions are not documented or not implemented. One example is eggex which are meant to be a replacement for reggexes. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How is the platform support for Oil? Does it work on all platforms that are supported to some extent in Nixpkgs? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Linux and darwin are both supported.
|
||
## Alternatives | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This appears to be proposing a "flag-day" kind of switch to some degree. While it is expected that there is a period of oil+bash compatibility my understanding is that there is no true guarantee of this and at the end of the day once we think everything is compatible the shell will be flipped. I wonder if we could somewhat ease this by adding a shell flag to stdenv which can be set to bash or oil. It would start defaulting to bash and we can slowly start moving some packages to oil. Once we have a substantial number of packages moved that we are fairly confident that the stdenv machinery and common hooks work we can flip the default. However for packages that fail on oil we can temporarily add the flag to switch back to bash. This way we can get better, continuous, testing before the switch. Then we have a route to deal with the last remaining bugs without continuously flipping the shell between oil and bash for all packages. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I really like the idea of the flag. I would even want to have it for specific packages for people to use their favorite shell. The only thing would be that if people can start using their own shell for a package and they use specific features of that shell, then we will never be able to remove that flag again (I don't necessarily think it's a bad idea). Perhaps you were thinking of a flag that isn't changeable by individual developpers ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would hesitate about "for specific packages for people to use their favorite shell". This puts long-term maintenance burden on stdenv and hooks to work with as many shells as possible, it also means that we can't take advantage of the convenience and safety features of oil. I don't think that is worth it. I see it purely as a migration tool between
The ensures that each step is small and safe and enforces forward progress through the migration (accidental regressions on either shell can't happen as some builds will fail). Again, I recommend this only as a migration tool. I think the long-term maintenance of multiple shells is likely more effort than it is worth. |
||
|
||
[alternatives]: #alternatives | ||
|
||
- I'm not aware of any other alternative shell that could enable a smooth transition with bash. [Here](https://github.com/oilshell/oil/wiki/Alternative-Shells) is a list of alternative shells. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think there are two alternatives to this which should at least be seriously considered: More provocative maybe than realistic, but POSIX shell could also be an alternative. It would also allow for a transition period where we could keep using bash and make our scripts POSIX compatible (granted, the changes necessary would be far greater than for osh). The two reasons for this would be:
While I don't think me or anyone here actually wants to switch to POSIX shell (it's just not worth the effort as there's little to gain), I do think that answering this question could help us and the RFC to clarify what we want from The second alternative would be to rely less on the shell in the future. The bulk of shell scripts subject to this RFC or discussion is not the build scripts which are quite simple execute-commands-in-sequence affairs, but the setup hooks and the many wrappers (binutils, cc, pkg-config, ...) contained in nixpkgs. setup hooks have always been considered an implementation detail of nixpkgs and consequently, we could consider replacing these shell scripts with an alternative approach, e. g. a mechanism implemented in pure Nix or using a DSL for manipulating the environment (which is the main thing the wrappers / setup hooks are doing after all). Should we want to take such a direction in the future, it may not be considered worth it to invest time into Oil support in the short / medium term. |
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think you need a Bash-compatible programming language (not necessarily shell language) to obtain compatibility with the existing code, e.g., what is a function in stdenv could likely be replaced by an external program. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think having some kind of bash compatibility would be critical, as it would allow us to incrementally port the existing code. Otherwise we would be forced to rewrite everything at once which is a lot of work and would probably lead to a situation where it is harder to catch and debug regressions. Also I don't think current shell functions could be replaced with external programs as we rely on bash-features like arrays which generally don't translate well to anything non-bash. |
||
## Unresolved questions | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would like to see some sort of performance estimate. I suspect it will be trivial but it would be nice to do a sanity check with some number of packages just to validate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I think some kind of performance eval is in order. As noted in https://matthewbauer.us/blog/avoid-subshells.html, this can really add up in the stdenv. |
||
|
||
[unresolved]: #unresolved-questions | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One aspect is that it complicates the system bootstrap as it is now part of the low-level dependencies needed for the stdenv. It would be good to get an idea of how much that represents, and how hard it would be to port to various systems. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you mean in terms of size? In terms of various systems, do you mean linux and darwin ? Or different architectures ? (the PR includes what I know for linux and darwin). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Size in terms of number of additional dependencies. nixpkgs only officially supports 4(?) (kernel, cpu) combinations but there are efforts out there to port it to other types of systems. Each new tool added to the stdenv closure can introduce more friction for these porting efforts. Even though those systems are not fully supported, we usually try to keep them in mind for high-impact dependencies like the stdenv. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Only 4? Reasonable way to get native stdenv should be a concern where we pay attention even to Tier 4, I think — and at that level we have quite a few more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From #99 (comment) I gather that currently python is required for Oil, this means that as it stands we'd need to redo all bootstrap tarballs to also include Python. OTOH this could also serve as a good indicator of the earliest viable point to switch to Oil: As soon as |
||
- What are the pittfalls and bugs in oilshell? | ||
- How much work exactly is required to make the switch? | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another aspect is speed. What time impact on a mass rebuild would the Oil adoption cause? Ideally that's something that can be answered before this RFC gets accepted. |
||
## Future work | ||
|
||
[future]: #future-work | ||
|
||
- Remove the `-i` flag uses in stdenv. An example can be found in [PR](https://github.com/NixOS/nixpkgs/pull/130597) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The whole space where bash gets massaged to be compatible with Oil is probably fine to do outside of RFC scope. I don't think it would be too controversial. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree with you, I think it can even be considered a benefit (having less potential bug in stdenv). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How a shell 100% compatible with bash can remove bash footguns?