[RFC 0099] Change default shell from bash to oil

[happysalada]

This proposal is around changing the default shell of bash to an alternative shell called oil.

The main point is that it seems the changes should be small and the benefits look promising

Rendered version

[L-as]

There are some unclear parts.

[L-as]

How do osh and oil interoperate?

[happysalada]

You have two binaries, and depending on the one you use, you have different features. osh can run bash script almost as is. In order for oil to run bash scripts, there needs to run scripts, those need to be reworked to tolerate the strict err_exit. Running a script with oil enables you to use additional features of the oil language. Where scripts run with osh, only have access to bash shell features. Let me know if this doesn't answer your question.

[L-as]

So you can access functions defined in an Oil script I assume?

[happysalada]

In the same way you can access bash function defined in a script.
oil introduce procs which are functions that don't mess with their outer scope and take parameters. I don't know if the same interoperability would apply to those. The way to call a proc is to pass it arguments. I think those would be fine too but this is something that has to be confirmed.

[jakeisnt]

To clarify, "oilshell" is a standardization of the POSIX shell protocol, and "Oil" (the programming language) is oilshell augmented with Oil language features, correct?

[happysalada]

osh is a bit more as it is meant to run bash things which have some non POSIX behaviours in it. But yes, oil is osh augmented with new language features.

[L-as]

I don't think you need a Bash-compatible programming language (not necessarily shell language) to obtain compatibility with the existing code, e.g., what is a function in stdenv could likely be replaced by an external program.

[sternenseemann]

I think having some kind of bash compatibility would be critical, as it would allow us to incrementally port the existing code. Otherwise we would be forced to rewrite everything at once which is a lot of work and would probably lead to a situation where it is harder to catch and debug regressions.

Also I don't think current shell functions could be replaced with external programs as we rely on bash-features like arrays which generally don't translate well to anything non-bash.

[L-as]

How is the platform support for Oil? Does it work on all platforms that are supported to some extent in Nixpkgs?

[happysalada]

Linux and darwin are both supported.

• There seems to be no problem so far with the different architectures
• Darwin has some heisenbugs related to the python implementation it seems.

[L-as]

Small nitpick: All the other PRs are called [RFC XXXX] text

[Ma27]

IIRC @zimbatm suggested to try out oil as language for the stdenv rather than bash and I actually like the idea. However I think that this is the biggest risk we have.

Shifts of interest or (unfortunately also) burnout do happen in open source. The worst case is that the "heart" of the nixpkgs project depends on an unmaintained, effectively dead language at some point in the future.

That said, I think that we should discuss how we can minimize this risk, e.g. by finding people who are willing to help out / can (co-)maintain this project or by sponsoring this project via the NixOS foundation (these are just two ideas from the top of my head, perhaps someone else has better ideas :))

[happysalada]

One thing to mitigate this could be that we could have a "long" period of keeping compatibility with both language and only make the switch when more people have been involved with oil.

After the flag feature was proposed, I think enabling package maintainers early to choose another shell for evaluation of their package could be an interesting experiment. As oil's user base increases, contributors should increase as well. We could make it a condition to switch stdenv only when a "sufficent" numbers of contributors have appeared.

[7c6f434c]

An argument could be made for being worth the switch even if it gets stuck in some state — it's not like we eagerly adopt new features of Bash immediately. But then indeed there should be evaluation from this point of view (maybe including some estimation of the burden of maintaining buildability)

[kevincox]

I agree that this is the biggest concern. Sure, if Oil disappears we won't be "screwed" overnight but over time bugs will be found and security issues will need to be fixed. Oil's dependencies will need to be updated, including its build tools. Sure, in theory we can keep whatever old deps Oil needs forever but that is a significant maintenance burden. I don't want Oil to be forcing us to keep python 3 around when the world is on python 5.

There are alternatives to consider. ZSH avoids some of the biggest issues by not requiring variables to be quoted, but it isn't as nice as Oil. I think it is a clear improvement from bash but is it worth the migration? Oil seems clearly worth the migration given the current state of the project but again, I don't want to bet on a 1-dev project and need to migrate back to bash (or zsh) one day. I also don't think that long-term multi-shell support is a great option. I think if we are going to change the shell we need to expect to complete the migration without any significant "waiting" period. I'm not saying that it needs to be done overnight, or that we can't change our mind and rollback, but I am against the long-term multi shell state due to the confusion and maintenance burden it causes.

[7c6f434c]

security issues

Well, we are not talking CGI or user shell, intentional attacks on stdenv are a pretty niche consideration.

to keep python 3

Current track record of PyPy suggests that if there is nothing CPython-only, the risk is limited.

But of course even just a big mass of C++ having lost maintenance can become an annoying liability.

[kevincox]

Completely correct. We just have to weigh the potential cost of carrying this project vs the savings of having a better shell that helps make nixpkgs more reliable and productive. In my mind it is a very tough call, but I think I am leaning in favour.

[jakeisnt]

It's unclear to me whether a better backwards compatible shell replacement exists - reading the oil shell blog reveals a ton of time and research sunk into the POSIX spec and the behavior of common shells (not always POSIX compliant!) to optimize for backwards compatibility above all else.

I don't see a better alternative shell for Nix.

That said, picking up Oil with a single developer and before a 1.0 release is a significant risk, so I agree that we should somehow ensure that development is funded and actively contributed to. Is the current developer aware of Nix and our interest in the project?

[happysalada]

the alias andychu that commented on this RFC is the person behind oil.

There are also some nix users who have contributed to oil https://github.com/abathur is one of them.

[lf-]

I think this objection is reasonable. However, at the same time, GNU Bash has very few maintainers (I cannot say if there are more than one due to the unclear/opaque dev process and git commits). The main advantage with Bash in this respect is that so many people rely on it that it is likely some maintenance will happen indefinitely regardless of who does it.

I don't think this is necessarily a major problem.

[andir]

I like the idea of moving to osh but:

A year ago I did toy with a package set from scratch that used oil shell instead of bash for everything. The result wasn't great. I still had to call a ton of shell scripts (that come as part of packages) with a bash version as osh would simply fail or not do the expected thing. While it helps us structure our stdenv better we will not get rid of bash. It is probably very likely that stdenv should still contain a bash exectuable just to be backwards compatible.

In addition to that we should think about a migration plan. Simply flipping a switch in a huge PR that does "all" of the work isn't a great outlook. We should ensure that we can run the same stdenv (with minimal changes, where required) with both bash and osh. Ideally this happens after we adopted content addressed builds. Ideally the build outputs would be identical between both shells. Also for the migration time we need a way to assert that changes in expression work with both shells. Something like shellcheck that tests compatibility would be needed.

[kevincox]

This appears to be proposing a "flag-day" kind of switch to some degree. While it is expected that there is a period of oil+bash compatibility my understanding is that there is no true guarantee of this and at the end of the day once we think everything is compatible the shell will be flipped.

I wonder if we could somewhat ease this by adding a shell flag to stdenv which can be set to bash or oil. It would start defaulting to bash and we can slowly start moving some packages to oil. Once we have a substantial number of packages moved that we are fairly confident that the stdenv machinery and common hooks work we can flip the default. However for packages that fail on oil we can temporarily add the flag to switch back to bash.

This way we can get better, continuous, testing before the switch. Then we have a route to deal with the last remaining bugs without continuously flipping the shell between oil and bash for all packages.

[happysalada]

I really like the idea of the flag. I would even want to have it for specific packages for people to use their favorite shell. The only thing would be that if people can start using their own shell for a package and they use specific features of that shell, then we will never be able to remove that flag again (I don't necessarily think it's a bad idea). Perhaps you were thinking of a flag that isn't changeable by individual developpers ?

[kevincox]

I would hesitate about "for specific packages for people to use their favorite shell". This puts long-term maintenance burden on stdenv and hooks to work with as many shells as possible, it also means that we can't take advantage of the convenience and safety features of oil. I don't think that is worth it. I see it purely as a migration tool between bash and osh. It could be set per-package so that the migration looks something like this.

1. Set shell = "oil" for a small number of packages after updating them and stdenv to work.
2. Expand this list of packages until we believe that just about everything in stdenv and popular hooks are working on oil.
3. Flip the default shell and add shell = "bash" to any packages that this breaks.
4. Burn down the list of shell = "bash" packages to zero.
5. Remove the shell flag from stdenv. ("inline" oil as the default shell)
6. We can now take advantage of oil features!

The ensures that each step is small and safe and enforces forward progress through the migration (accidental regressions on either shell can't happen as some builds will fail).

Again, I recommend this only as a migration tool. I think the long-term maintenance of multiple shells is likely more effort than it is worth.

[kevincox]

I would like to see some sort of performance estimate. I suspect it will be trivial but it would be nice to do a sanity check with some number of packages just to validate.

[bhipple]

Yes, I think some kind of performance eval is in order. As noted in https://matthewbauer.us/blog/avoid-subshells.html, this can really add up in the stdenv.

[zimbatm]

Nice. Even if Oil doesn't end up getting accepted, I believe that massaging the stdenv has its benefits on its own.

[zimbatm]

The whole space where bash gets massaged to be compatible with Oil is probably fine to do outside of RFC scope. I don't think it would be too controversial.

[happysalada]

I agree with you, I think it can even be considered a benefit (having less potential bug in stdenv).
Only the final PR of either adding a flag to enable conditional switching or switching altogether would have to go through the RFC.
I wanted to give people an idea of what work needs to be done. The idea is that it's not so much work and it's valid in both bash and osh. But perhaps it wasn't very clear from the sentence. Let me know if you think I can remove this.

[zimbatm]

One aspect is that it complicates the system bootstrap as it is now part of the low-level dependencies needed for the stdenv. It would be good to get an idea of how much that represents, and how hard it would be to port to various systems.

[happysalada]

Do you mean in terms of size?
In the following PR, I've made the initial additions to stdenv in terms of allowedRequisites
NixOS/nixpkgs#131676
I haven't checked the size though. (this PR will fail until the local -i changes are merged to staging, so I don't know how much information it will give you.)

In terms of various systems, do you mean linux and darwin ? Or different architectures ? (the PR includes what I know for linux and darwin).

[zimbatm]

Size in terms of number of additional dependencies.

nixpkgs only officially supports 4(?) (kernel, cpu) combinations but there are efforts out there to port it to other types of systems. Each new tool added to the stdenv closure can introduce more friction for these porting efforts. Even though those systems are not fully supported, we usually try to keep them in mind for high-impact dependencies like the stdenv.

[7c6f434c]

Only 4? Reasonable way to get native stdenv should be a concern where we pay attention even to Tier 4, I think — and at that level we have quite a few more.

[sternenseemann]

From #99 (comment) I gather that currently python is required for Oil, this means that as it stands we'd need to redo all bootstrap tarballs to also include Python. OTOH this could also serve as a good indicator of the earliest viable point to switch to Oil: As soon as oil-native is ready, it would be way easier to add it to the bootstrap. Up to that point we could provide an oil-based stdenv, but wouldn't be able to bootstrap nixpkgs using oil.

[zimbatm]

Another aspect is speed. What time impact on a mass rebuild would the Oil adoption cause?

Ideally that's something that can be answered before this RFC gets accepted.

[happysalada]

@andir @kevincox Regarding the switch. I think it's not really clear from that PR, but the changes involved in switching are just making changes to the current stdenv that would still be valid bash.
Changes for osh are minimal and would be 6 lines that need to change in stdenv to enable the switch. I'm hoping once those changes are made to make a PR that enables the switch to osh so that people can try it for a while on their local.

The changes required to enable oil would be similar, just more in number.
Things like

• Separating assignment with subshell (local my_var=$(...) should really be 2 lines)
• Making sure variables are defined before assigning to them (related to the -u flag)
• Refactoring complicated logic inside conditional.
  ...

The point is that after all those changes, stdenv is still valid bash. So having a flag like suggested, makes perfect sense. Some people can choose to experiment with it for a while before any decision is made.
I like the idea of the flag for another reason. If people could choose for their packages which shell it would be evaluated with, it could give a chance to other people to experiment. It would be much less involved than switching stdenv.
The only point of no-return comes when you start using features only available to oil. (when you want the benefits really).

The one thing that I keep on thinking is how to introduce some tests for some of the functions we have in stdenv. If there were some way to introduce some amount of testing, I think we could be more confident of the changes we make to stdenv. Some of the stuff in stdenv is relatively old and mysterious. Recently I was struggling with the function consumeEntries. This function could be renamed failOnBinary, as it's purpose is just to fail loudly if you feed it with an executable for example. I haven't really figure out how to add tests.

Regarding the shellcheck proposal, somebody else mentioned it. At the moment, there are many shellcheck violation, but adding a hook somewhere that checks for those would certainly help. I also don't know here how to make this a reality.

Let me know if this is still vague.

[kevincox]

I agree that the changes are likely small but there will be a long tail of derivations that have incompatibilities, even if stdenv itself supports both perfectly. This is why I think the flag is critical. If we just stay on bash until every incompatibility is fixed it will be painful as we will continuously accidentally break oil. If we don't have the opt-out then we can't enable oil until everything (or close to it) works. The flag allows both ensuring that stdenv is being exercised in both shells and preventing backslides for the already migrated derivations. For these reasons I think it is critical to note this in the RFC.

I think we could update the stdenv builder build to have a checkPhase to do whatever testing we want. As long as the tests are stable I don't think there would be much opposition to that (probably doesn't need an RFC) and would likely be some nice prep-work for this change.

[7c6f434c]

What I find a benefit of rougly-POSIX-shell, and maybe even Bash outright, is that upstream build system has a high chance of including some shell pieces, and sometimes we need to understand it, and switching between that and a similar but different language stdenv code sounds like a path to confusion — maybe slightly more confusion than using a conspiciously different language like Python or Lua. (But that would have higher migration costs)

[jakeisnt]

As stated earlier in the proposal, there is no need to modify most bash scripts used for build systems. Think of Oil as a superset of POSIX - it supports everything Bash does with the same semantics, then introduces new, useful language features without regressing on legacy code.

[jakeisnt]

How is Oil's architecture support? I know there is some interest in running Nix on chips different from aarch-64 and x86-64, so we should reasonably ensure that there is interest in actively testing for and supporting select alternative architectures going forward (i.e. RISC-V) so as not to disparage those valuable efforts.

[happysalada]

I understand the confusing part.

• One thing that should improve the situation is that at the same time all the version compatibility tricks for bash will be slowly removed. Nobody should have to know again what flagsArray=(${distFlagsArray+"${distFlagsArray[@]}"}) mean (just a compatibility between 3 to 4). Those are being removed and so hopefully the bash code feels less foreign as it gets massaged.
• Oil itself feels very different, so hopefully when we actually use it, the confusion should lessen

Regarding the architecture support I've never seen anything on it. As the code moves to c++ the story should get better in that regard.

[andychu]

(author of Oil here) Hi and thanks @happysalada for bringing this up!

I commented on the Oil Zulip [1] but I'll summarize my thoughts here. The risks / issues that are brought up are definitely valid, like platform support (Darwin issues) and the current performance (due to the Python / C++ split, or "executable spec" strategy). I bring up the latter in http://www.oilshell.org/why.html#why-not-use-it

What I would say is that there is little risk to START doing this work. To run bash and Oil in parallel. This might be for a long time! But I believe it will improve the shell scripts.

I mean making the shell scripts work on 2 different shells IMO will improve things, regardless of whether that shell is Oil. However Oil is designed to run real bash scripts, and it has for over 3.5 years, so this will be EASIER than it is with any other shell. Oil is the most bash-compatible shell by a mile.

• Success with Aboriginal, Alpine, and Debian Linux - January 2018. Oil is designed for the foundation of distros and Nix is absolutely within scope! In fact I have changed the OSH language based on feedback from multiple Nix users, like Travis Everett, @happysalada, and a few others. OSH is basically a "rational" version of bash, as bash has details that change from version to version and that are undocumented.
• Oil 0.8pre4 - The Biggest Shell Programs in the World - April 2020. We made a ton of progress toward more compatibility in 2.5 years, and even more in the last year!

So what I would say is:

• I would not expect this to be a short or easy project :) However I believe it will improve your bash scripts, even if you never switch to Oil.
• If anyone disagrees about that then I'm open to feedback, and it's likely that feedback based on real use cases will change the project. Actually Oil needs more feedback from production code, not "nice to have" suggestions from people who haven't really tried it! I prioritize issues motivated by real code.
• I believe it's absolutely possible for Oil to meet its goals given enough effort. I'm going to write a "Five Years of Oil Milestones" blog post about this. However I don't claim I can do it alone :) I will write more about that. But note that OSH is quite mature and stable, and the Oil language is the more "risky" part. I think you will get benefit just from using OSH, and maybe when people see the benefits they'll be motivated to help out :)
• The Python vs. C++ stuff is probably confusing to people. The short answer is that Oil is written as an "executable spec" in Python, and then it's translated to C++ automatically -- BUT that part isn't done yet. So oil-native is not a working shell yet. But oil-native does pass something like 1000 out of 1900 spec tests, so I believe the strategy will work. This is all on the blog, but feel free to ask questions and I will clarify / points to the right sections.

Thanks for this discussion, and I do think there is a strong philosophical overlap between Oil and Nix, because we're both focused on correctness! I basically wanted to make a shell with the documented semantics of a programming language AND that runs real bash programs, and that is actually possible! (though it's not obvious that it is). And I also want to add more features like structured data, declarative configuration, etc. but that part isn't done yet :) The designs are strong though, feel free again to ask questions.

[1] https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/Oil.20as.20default.20shell.20on.20Nix (log in with Github)

[andychu]

Also note the differences between OSH and bash are documented in detail, and I'm open to PRs / suggestions on docs:

https://www.oilshell.org/release/latest/doc/known-differences.html

This doc is also relevant, and makes reference to the "common subset principle".

https://github.com/oilshell/oil/wiki/What-Is-Expected-to-Run-Under-OSH

The common subset of bash and OSH is very large and "rational". In fact our own shell scripts (thousands of lines) all live in that subset. It is dogfooded a lot on real shell code, and again open to feedback.

[jakeisnt]

The passage above demonstrates that the semantics of the former and the latter statements provided are equivalent with oilshell, correct? This could use some clarification - 'x becomes y', I don't think, is sufficiently clear.

[happysalada]

You're right this isn't super clear.
Here is what I intended to convey.
The top example is "idiomatic" bash. The bottom example is the equivalent "idiom" in oil.
Let me know if that is too vague

[L-as]

@andychu What platforms does Oil support? I tried looking for a list of supported platforms, but couldn't find anything.

Of the ones that aren't supported, how hard would it be for us to add support for them?

(scraped from (import <nixpkgs> {}).lib.platforms.all)

Some notable OSes (excluding the obvious ones):

• Darwin
• FreeBSD
• Genode
• Solaris
• NetBSD
• Windows
• OpenBSD

Some notable CPU architectures (excluding the obvious ones):

• aarch64
• armv7
• armv6
• armv5
• i686
• s390
• riscv32
• riscv64
• wasm32
• wasm64

[sternenseemann]

What occasions are you referring to here where we bring in other languages for scripting? As far as I am aware stdenv and setup hooks are 100% bash.

Some core nixpkgs builders are implemented in Perl, the question here is, what we'd gain from rewriting them in oil. As it stands the user base of Oil is quite small.

In any case, I'd appreciate a clarification here, maybe even some examples!

[happysalada]

The main thing I'm referring to is the findInputs() in stdenv.
that function is super involved and is doing a very weird recursion with dynamic references all over the place. I feel that the best was done with the tools that bash has to offer, I keep on thinking that it would have been written differently with a programming language that gives you a little more power. Perhaps that function will appear crystal clear to you in that case, feel free to ignore this first point :-)

One more point I have against bash is the error handling. If we had a better way of handling errors, it wouldn't be so bad to try to debug why a modification in stdenv breaks something. I'm not 100% that oil will be able to solve this, but normal programming language give you a little easier interface to work with error handling.

The last point I have is about trying to restrict the numbers of things you need to learn by heart on a language. There are so many pitfals, that we run into experienced programmers making mistakes in nixos modules or packages. I'm hoping that there are less things to learn with oil. (less rules, make it harder to mess up).

[sternenseemann]

I think there are two alternatives to this which should at least be seriously considered:

More provocative maybe than realistic, but POSIX shell could also be an alternative. It would also allow for a transition period where we could keep using bash and make our scripts POSIX compatible (granted, the changes necessary would be far greater than for osh). The two reasons for this would be:

• Nix's env setup (without structuredAttrs) can't make use of any non-standard bash features like arrays. This leads to surprising behavior where setting *FlagsArray attributes which setup.sh understands will not work as expected.
• Portability: There are far more POSIX shell implementation or POSIX shell compatible shells than bash compatible ones.

While I don't think me or anyone here actually wants to switch to POSIX shell (it's just not worth the effort as there's little to gain), I do think that answering this question could help us and the RFC to clarify what we want from stdenv.shell.

The second alternative would be to rely less on the shell in the future. The bulk of shell scripts subject to this RFC or discussion is not the build scripts which are quite simple execute-commands-in-sequence affairs, but the setup hooks and the many wrappers (binutils, cc, pkg-config, ...) contained in nixpkgs. setup hooks have always been considered an implementation detail of nixpkgs and consequently, we could consider replacing these shell scripts with an alternative approach, e. g. a mechanism implemented in pure Nix or using a DSL for manipulating the environment (which is the main thing the wrappers / setup hooks are doing after all).

Should we want to take such a direction in the future, it may not be considered worth it to invest time into Oil support in the short / medium term.

[sternenseemann]

From #99 (comment) I gather that currently python is required for Oil, this means that as it stands we'd need to redo all bootstrap tarballs to also include Python. OTOH this could also serve as a good indicator of the earliest viable point to switch to Oil: As soon as oil-native is ready, it would be way easier to add it to the bootstrap. Up to that point we could provide an oil-based stdenv, but wouldn't be able to bootstrap nixpkgs using oil.

[andychu]

@L-as

@andychu What platforms does Oil support? I tried looking for a list of supported platforms, but couldn't find anything.

Of the ones that aren't supported, how hard would it be for us to add support for them?

There is no real notion of "platform support" at the moment. In theory it should be trivial because Oil uses the oldest and most portable APIs in Unix: fork(), exec(). Not only it is a goal for Oil to be portable, but it's a goal for it to have close to zero #ifdefs!

Compare Lua vs. CPython -- both are highly portable from the outside, but Lua has almost zero #ifdefs, and CPython has a ton. I didn't quite realize this before hacking on CPython.

In practice there are some platform issues precisely because of this CPython issue, which in theory will go away with oil-native, but as mentioned that's not done yet.

I develop and test Oil on Linux (Ubuntu/Debian), and all the platform support has come from contributors. Back in 2018 someone patched Oil with the ability to run on OS X. I have run it on OS X and so have a few other people, but there is a weird input reading bug I don't understand, and don't have time to look into.

Also a contributor named Josh made OS X run on FreeBSD.

Someone else reported that it runs on WSL.

Again I think this would be entirely unremarkable for oil-native -- it would run everywhere simply because fork(), open(), and stat() work the same way everywhere. There are perhaps some interactive features of shell that are less portable across Unixes than that, but I imagine Nix won't use those since I assume it's all for build automation and not interactive use.

tl;dr I don't make any promises of platform support because it would be a lie. I prefer to underpromise / overdeliver, etc. If a bunch of people get involved, then it might be accurate to make promises.

However there already have been a bunch of people involved, so that's a sign that the codebase is not unapproachable.

---

As many people up the thread mentioned, I think it's a hard requirement to have some sort of migration path and not a "flag day".

That is, you don't want to pile up a whole bunch of changes in the hopes that Oil will be a big improvement in some unspecified future.

Instead you want to make small changes to existing scripts to make them run under OSH (many of them should run unmodified). There should be some kind of testing process that runs with both bash and Oil, and compares the two results. When there is a difference, report a bug against Oil and I will prioritize it for sure.

There should also be people reviewing the changes for OSH to make sure that everyone agrees with the changes even if you never make Oil the default shell (very important) !!!! I don't want there to be one contributor making changes for Oil, and another contributor complaining that this is messing up the codebase.

I don't believe that will happen because I've designed the OSH language to avoid it and be "rational". But if it happens I want to know.

---

As Rafael mentioned probably one of the bigger issues is -i. That is documented here:

https://www.oilshell.org/release/latest/doc/known-differences.html#values-are-tagged-with-types-not-cells

Although maybe that explanation is not clear to those who aren't very familiar with bash. Bash has an unusual model where locations for values are tagged with types, no values themselves, which leads to behavior nobody can explain, even if you ask the bash mailing list. You will basically get somebody reading the source code to you, and it changes from time to time.

For example try converting an indexed array to an associative array in bash, or vice versa, and try to predict the results.

Oil intentionally avoids this ill specified and undocumented behavior in favor of a simpler model that has a common subset with bash, but is not identical bug-for-bug with bash.

As mentioned, thousands of lines of shell scripts (unmodified and lightly modified) already run under this common subset.

tl;dr I would say it's a good idea for Nix to start using this large common subset, and I want feedback on whether everyone actually agrees with the changes! I think a bottleneck here though is that not everyone understands the bash code deep in the bowels of Nix. I have looked myself and it's pretty confusing.

However the whole point of Oil is to make sense of this obscure bash code. I think another issue was "nameref" or declare -n support, especially with dynamic parsing of a[i].

[happysalada]

One last thing to put things in perspective. I've had a look at upgrading to bash 5. there was a completely undocumented behavior change of read -N 0. I'm not the one who found the change nor the solution
NixOS/nixpkgs#108101
We were lucky that somebody already did the work, it just wasn't documented anywhere. It's a really small change, but it's stuff like this that make it really hard to upgrade bash. It's not life threatening to be stuck on 4.4 for a while, but I just want to put that in perspective with potential maintenance difficulties of oil.

[andychu]

That's a great point and something I was going to bring up. If Nix build scripts worked with bash 4.4 and bash 5.1 simultaneously, then they would be closer to working with bash and OSH simultaneously.

However I think one of the main principles of Nix is that you can pin exact versions, so this kind of portability is less important. That is, Nix's version stability means that people can write whatever hairy code they want that depends on the specific bash version, including undocumented behavior, and it will work.

---

In practice I've noticed that maintainers just hack until things work (in general, not sure about Nix), which is rational / reasonable given time constraints. But OSH is trying to make shell more "literate" and more like a language you can explain.

[happysalada]

This RFC involves a change in stdenv shell scripts, they need to be massaged into compatibility with several shells (this should be an improvement and make them more robust). That first part is less controversial and can be started right now (the decision whether to make the switch or not, can be made later). However in order for that first part to succeed, the following would be needed.

• Somebody that likes shell, and knows or has a willingness to learn. That person can review shell changes and proposes improovement or make sure the changes made make sense. I can make the changes myself, but I am biased and not the most knowledgeable, so somebody available for review would help i think.
• Somebody with some hydra knowledge (or on the hydra team). Changes to stdenv should ideally go through a hydra rebuild, since breakage can be quite subtle. In previous PR, I naively merged after the tests where passing and then we discovered in staging-next that there were some breakage. I am trying to level up on hydra and checking if I can't run some builds on my server.

At the moment, the PRs for shell changes are blocked because of lack of the second person. (so far vcunat has been very generous with his time in helping with hydra rebuilds). If think progress toward completion of this RFC is going to be conditioned on finding two profiles like detailed above. I'm not sure who might be a good fit and who to ask.

[happysalada]

The first round of shell changes has been tested on hydra and there has been some pkgsCross checks.
NixOS/nixpkgs#132490
Those are 5 PRs that implement some simple bash fixes and one PR that removes the -i flag. Just in case anybody is interested to review. (at the moment, those are pending more reviews).
After those PRs are merged, I plan to test again the dummy PR I have to migrate to osh to check the next failures.
The idea is to try to batch the changes in group of 5-6 to make it worthwhile to test on hydra.
(note that most of these changes have nothing to do with a migration to osh, they are just some shellcheck fixes. If the shellcheck fixes generate interest, I will make some more. Most likely though, the next batch of changes will include osh related changes.)

[bhipple]

Are we really sure we want to tackle this? The stdenv may be ugly, but it's already here, it already works, and it's using the universally known bash shell. Nix already has a steep learning curve. Do we want to toss a different shell onto the pile of things to learn to understand nix?

It seems likely that we'll end up supporting both bash and oil. Even if we don't experience contributor burnout mid-migration, there's a long tail of weird scripts and package behavior out there to support. Seems like a big risk that we'll make a mess and end up with a hybrid half-and-half situation that's harder to understand than what we started with.

[andychu]

Are we really sure we want to tackle this?

Seems like a big risk that we'll make a mess and end up with a hybrid half-and-half situation that's harder to understand than what we started with.

See my comment above. I think this question is asssuming that there is some big difference between bash and OSH. OSH is designed to run bash scripts. When there is a difference, changing to use the large common subset of bash and OSH should improve your scripts. If people feel that's not the case then I want to know.

It seems like the biggest issue is that not many understand the shell in Nix. I have seen this in other projects like bash-completion -- the maintainers don't understand the shell code they're maintaining. Which is perfectly reasonable since it may have been written a long time ago by someone else.

So that has to be weighed in the cost benefit and I don't know what the answer is.

I would say the cost of starting is low except for:

1. figuring out to test with both bash and OSH.

2. getting people to review any shell changes necessary to run under OSH. Depending on the program, there may be zero changes.

I wouldn't think of it as "should we tackle this huge project?" You can stop any time and still be better off, or at least in the same position.

How much code are we talking about? Sorry if I missed it

[andychu]

To clarify, I'm saying that if there are resources to do the work, there's no risk to migrating to OSH (the compatible shell).

That is a prerequisite to migrating to Oil, which can and should be discussed as a separate issue. When you start writing Oil, then obviously the code won't work with bash, and that's indeed a risk. But that may not happen for many months or ever, depending on what it takes to work under OSH.

[Ericson2314]

Can Oil or Osh work on Windows or other not-so-POSIX-y things? We've discussed what to do about Nixpkgs if the Nix Windows port was finished. I always thought Oil was a nice sounding thing with good goals, but if it would help us here that would be a very good concrete benefit.

[nyabinary]

There hasn't been a lot of talk on this RFC, what the status?

[happysalada]

The change is very involved and opposed by a non trivial number of people from the community. If there was agreement , this is something that could be done. But its something that doesnt come without tradeoff.
Maybe this can be revisited in a few years if/when oil is more developped and has more features.