Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC 0099] Change default shell from bash to oil #99
[RFC 0099] Change default shell from bash to oil #99
Changes from 1 commit
f4e238a
5cd85be
fa4ee8d
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How a shell 100% compatible with bash can remove bash footguns?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What occasions are you referring to here where we bring in other languages for scripting? As far as I am aware stdenv and setup hooks are 100% bash.
Some core nixpkgs builders are implemented in Perl, the question here is, what we'd gain from rewriting them in oil. As it stands the user base of Oil is quite small.
In any case, I'd appreciate a clarification here, maybe even some examples!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main thing I'm referring to is the
findInputs()
in stdenv.that function is super involved and is doing a very weird recursion with dynamic references all over the place. I feel that the best was done with the tools that bash has to offer, I keep on thinking that it would have been written differently with a programming language that gives you a little more power. Perhaps that function will appear crystal clear to you in that case, feel free to ignore this first point :-)
One more point I have against bash is the error handling. If we had a better way of handling errors, it wouldn't be so bad to try to debug why a modification in stdenv breaks something. I'm not 100% that oil will be able to solve this, but normal programming language give you a little easier interface to work with error handling.
The last point I have is about trying to restrict the numbers of things you need to learn by heart on a language. There are so many pitfals, that we run into experienced programmers making mistakes in nixos modules or packages. I'm hoping that there are less things to learn with oil. (less rules, make it harder to mess up).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do osh and oil interoperate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have two binaries, and depending on the one you use, you have different features. osh can run bash script almost as is. In order for oil to run bash scripts, there needs to run scripts, those need to be reworked to tolerate the
strict err_exit
. Running a script with oil enables you to use additional features of the oil language. Where scripts run with osh, only have access to bash shell features. Let me know if this doesn't answer your question.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So you can access functions defined in an Oil script I assume?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the same way you can access bash function defined in a script.
oil introduce procs which are functions that don't mess with their outer scope and take parameters. I don't know if the same interoperability would apply to those. The way to call a proc is to pass it arguments. I think those would be fine too but this is something that has to be confirmed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify, "oilshell" is a standardization of the POSIX shell protocol, and "Oil" (the programming language) is oilshell augmented with Oil language features, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
osh is a bit more as it is meant to run bash things which have some non POSIX behaviours in it. But yes, oil is osh augmented with new language features.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps you can clarify the meaning of this statement, as the way it is currently written, it seems to defeat the purpose of this whole idea. Why rewrite the standard env if we don't actually wish to leverage any of oils specific features? If that's not what you meant to convey, then perhaps this segment should just be removed? (I'm referring to line 43 specifically)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I mean by this is that the changes to make the switch possible are a little more involved.
The changes to osh only include removing a
-i
flag so far (potentially a couple of lines change in stdenv).The changes to enable a switch to oil are more involved (hard to know the extent yet).
The referring to maintaining compatibility is related to having a period where things are in-between and we can evaluate if we want to switch to oil or not. Potentially retain compatibility for some time and when we feel more comfortable then start using oil exclusive feature.
I do agree with you, it's not very well formulated, how would the following work ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The passage above demonstrates that the semantics of the former and the latter statements provided are equivalent with oilshell, correct? This could use some clarification - 'x becomes y', I don't think, is sufficiently clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right this isn't super clear.
Here is what I intended to convey.
The top example is "idiomatic" bash. The bottom example is the equivalent "idiom" in oil.
Let me know if that is too vague
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I find a benefit of rougly-POSIX-shell, and maybe even Bash outright, is that upstream build system has a high chance of including some shell pieces, and sometimes we need to understand it, and switching between that and a similar but different language stdenv code sounds like a path to confusion — maybe slightly more confusion than using a conspiciously different language like Python or Lua. (But that would have higher migration costs)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As stated earlier in the proposal, there is no need to modify most bash scripts used for build systems. Think of Oil as a superset of POSIX - it supports everything Bash does with the same semantics, then introduces new, useful language features without regressing on legacy code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is Oil's architecture support? I know there is some interest in running Nix on chips different from aarch-64 and x86-64, so we should reasonably ensure that there is interest in actively testing for and supporting select alternative architectures going forward (i.e. RISC-V) so as not to disparage those valuable efforts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand the confusing part.
flagsArray=(${distFlagsArray+"${distFlagsArray[@]}"})
mean (just a compatibility between 3 to 4). Those are being removed and so hopefully the bash code feels less foreign as it gets massaged.Regarding the architecture support I've never seen anything on it. As the code moves to c++ the story should get better in that regard.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One more thing that is probably not a big issue but I want just to mention it..
We know that bash → osh has few «loud» issues like
-i
. But are there no «quiet» issues where osh attempted to make most sense and ended up slightly incompatible by accident? Probably not so many that we'd hit one. However, we could want either to decide on the scope of our testing of the corner cases, or on the expectations of evidence that other people have checked whether differences are documented.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your concern makes sense. I think we can only make changes by testing the rebuild of the whole of stdenv to make sure nothing breaks silently. It won't provide any guarantees that nothing will break in the future, however, it should be a pretty good sign that the status quo is good.
I'll make sure to mention anything that is found and that cannot be resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just
stdenv
per se is not much; there needs to be something cross-platform, and maybe ideally at least one three-platform test? Not sure what are the other corner cases, maybe someone will remember some more.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC @zimbatm suggested to try out oil as language for the stdenv rather than bash and I actually like the idea. However I think that this is the biggest risk we have.
Shifts of interest or (unfortunately also) burnout do happen in open source. The worst case is that the "heart" of the nixpkgs project depends on an unmaintained, effectively dead language at some point in the future.
That said, I think that we should discuss how we can minimize this risk, e.g. by finding people who are willing to help out / can (co-)maintain this project or by sponsoring this project via the NixOS foundation (these are just two ideas from the top of my head, perhaps someone else has better ideas :))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing to mitigate this could be that we could have a "long" period of keeping compatibility with both language and only make the switch when more people have been involved with oil.
After the flag feature was proposed, I think enabling package maintainers early to choose another shell for evaluation of their package could be an interesting experiment. As oil's user base increases, contributors should increase as well. We could make it a condition to switch stdenv only when a "sufficent" numbers of contributors have appeared.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
An argument could be made for being worth the switch even if it gets stuck in some state — it's not like we eagerly adopt new features of Bash immediately. But then indeed there should be evaluation from this point of view (maybe including some estimation of the burden of maintaining buildability)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that this is the biggest concern. Sure, if Oil disappears we won't be "screwed" overnight but over time bugs will be found and security issues will need to be fixed. Oil's dependencies will need to be updated, including its build tools. Sure, in theory we can keep whatever old deps Oil needs forever but that is a significant maintenance burden. I don't want Oil to be forcing us to keep python 3 around when the world is on python 5.
There are alternatives to consider. ZSH avoids some of the biggest issues by not requiring variables to be quoted, but it isn't as nice as Oil. I think it is a clear improvement from bash but is it worth the migration? Oil seems clearly worth the migration given the current state of the project but again, I don't want to bet on a 1-dev project and need to migrate back to bash (or zsh) one day. I also don't think that long-term multi-shell support is a great option. I think if we are going to change the shell we need to expect to complete the migration without any significant "waiting" period. I'm not saying that it needs to be done overnight, or that we can't change our mind and rollback, but I am against the long-term multi shell state due to the confusion and maintenance burden it causes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, we are not talking CGI or user shell, intentional attacks on stdenv are a pretty niche consideration.
Current track record of PyPy suggests that if there is nothing CPython-only, the risk is limited.
But of course even just a big mass of C++ having lost maintenance can become an annoying liability.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Completely correct. We just have to weigh the potential cost of carrying this project vs the savings of having a better shell that helps make nixpkgs more reliable and productive. In my mind it is a very tough call, but I think I am leaning in favour.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's unclear to me whether a better backwards compatible shell replacement exists - reading the oil shell blog reveals a ton of time and research sunk into the POSIX spec and the behavior of common shells (not always POSIX compliant!) to optimize for backwards compatibility above all else.
I don't see a better alternative shell for Nix.
That said, picking up Oil with a single developer and before a 1.0 release is a significant risk, so I agree that we should somehow ensure that development is funded and actively contributed to. Is the current developer aware of Nix and our interest in the project?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the alias andychu that commented on this RFC is the person behind oil.
There are also some nix users who have contributed to oil https://github.com/abathur is one of them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this objection is reasonable. However, at the same time, GNU Bash has very few maintainers (I cannot say if there are more than one due to the unclear/opaque dev process and git commits). The main advantage with Bash in this respect is that so many people rely on it that it is likely some maintenance will happen indefinitely regardless of who does it.
I don't think this is necessarily a major problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is the platform support for Oil? Does it work on all platforms that are supported to some extent in Nixpkgs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Linux and darwin are both supported.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This appears to be proposing a "flag-day" kind of switch to some degree. While it is expected that there is a period of oil+bash compatibility my understanding is that there is no true guarantee of this and at the end of the day once we think everything is compatible the shell will be flipped.
I wonder if we could somewhat ease this by adding a shell flag to stdenv which can be set to bash or oil. It would start defaulting to bash and we can slowly start moving some packages to oil. Once we have a substantial number of packages moved that we are fairly confident that the stdenv machinery and common hooks work we can flip the default. However for packages that fail on oil we can temporarily add the flag to switch back to bash.
This way we can get better, continuous, testing before the switch. Then we have a route to deal with the last remaining bugs without continuously flipping the shell between oil and bash for all packages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really like the idea of the flag. I would even want to have it for specific packages for people to use their favorite shell. The only thing would be that if people can start using their own shell for a package and they use specific features of that shell, then we will never be able to remove that flag again (I don't necessarily think it's a bad idea). Perhaps you were thinking of a flag that isn't changeable by individual developpers ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would hesitate about "for specific packages for people to use their favorite shell". This puts long-term maintenance burden on stdenv and hooks to work with as many shells as possible, it also means that we can't take advantage of the convenience and safety features of oil. I don't think that is worth it. I see it purely as a migration tool between
bash
andosh
. It could be set per-package so that the migration looks something like this.shell = "oil"
for a small number of packages after updating them and stdenv to work.shell = "bash"
to any packages that this breaks.shell = "bash"
packages to zero.shell
flag from stdenv. ("inline" oil as the default shell)The ensures that each step is small and safe and enforces forward progress through the migration (accidental regressions on either shell can't happen as some builds will fail).
Again, I recommend this only as a migration tool. I think the long-term maintenance of multiple shells is likely more effort than it is worth.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there are two alternatives to this which should at least be seriously considered:
More provocative maybe than realistic, but POSIX shell could also be an alternative. It would also allow for a transition period where we could keep using bash and make our scripts POSIX compatible (granted, the changes necessary would be far greater than for osh). The two reasons for this would be:
*FlagsArray
attributes whichsetup.sh
understands will not work as expected.While I don't think me or anyone here actually wants to switch to POSIX shell (it's just not worth the effort as there's little to gain), I do think that answering this question could help us and the RFC to clarify what we want from
stdenv.shell
.The second alternative would be to rely less on the shell in the future. The bulk of shell scripts subject to this RFC or discussion is not the build scripts which are quite simple execute-commands-in-sequence affairs, but the setup hooks and the many wrappers (binutils, cc, pkg-config, ...) contained in nixpkgs. setup hooks have always been considered an implementation detail of nixpkgs and consequently, we could consider replacing these shell scripts with an alternative approach, e. g. a mechanism implemented in pure Nix or using a DSL for manipulating the environment (which is the main thing the wrappers / setup hooks are doing after all).
Should we want to take such a direction in the future, it may not be considered worth it to invest time into Oil support in the short / medium term.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you need a Bash-compatible programming language (not necessarily shell language) to obtain compatibility with the existing code, e.g., what is a function in stdenv could likely be replaced by an external program.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think having some kind of bash compatibility would be critical, as it would allow us to incrementally port the existing code. Otherwise we would be forced to rewrite everything at once which is a lot of work and would probably lead to a situation where it is harder to catch and debug regressions.
Also I don't think current shell functions could be replaced with external programs as we rely on bash-features like arrays which generally don't translate well to anything non-bash.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like to see some sort of performance estimate. I suspect it will be trivial but it would be nice to do a sanity check with some number of packages just to validate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I think some kind of performance eval is in order. As noted in https://matthewbauer.us/blog/avoid-subshells.html, this can really add up in the stdenv.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One aspect is that it complicates the system bootstrap as it is now part of the low-level dependencies needed for the stdenv. It would be good to get an idea of how much that represents, and how hard it would be to port to various systems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean in terms of size?
In the following PR, I've made the initial additions to stdenv in terms of allowedRequisites
NixOS/nixpkgs#131676
I haven't checked the size though. (this PR will fail until the
local -i
changes are merged to staging, so I don't know how much information it will give you.)In terms of various systems, do you mean linux and darwin ? Or different architectures ? (the PR includes what I know for linux and darwin).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Size in terms of number of additional dependencies.
nixpkgs only officially supports 4(?) (kernel, cpu) combinations but there are efforts out there to port it to other types of systems. Each new tool added to the stdenv closure can introduce more friction for these porting efforts. Even though those systems are not fully supported, we usually try to keep them in mind for high-impact dependencies like the stdenv.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only 4? Reasonable way to get native stdenv should be a concern where we pay attention even to Tier 4, I think — and at that level we have quite a few more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From #99 (comment) I gather that currently python is required for Oil, this means that as it stands we'd need to redo all bootstrap tarballs to also include Python. OTOH this could also serve as a good indicator of the earliest viable point to switch to Oil: As soon as
oil-native
is ready, it would be way easier to add it to the bootstrap. Up to that point we could provide an oil-based stdenv, but wouldn't be able to bootstrap nixpkgs using oil.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another aspect is speed. What time impact on a mass rebuild would the Oil adoption cause?
Ideally that's something that can be answered before this RFC gets accepted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The whole space where bash gets massaged to be compatible with Oil is probably fine to do outside of RFC scope. I don't think it would be too controversial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with you, I think it can even be considered a benefit (having less potential bug in stdenv).
Only the final PR of either adding a flag to enable conditional switching or switching altogether would have to go through the RFC.
I wanted to give people an idea of what work needs to be done. The idea is that it's not so much work and it's valid in both bash and osh. But perhaps it wasn't very clear from the sentence. Let me know if you think I can remove this.