Deprecate latest for toolkit & add a relevant actionable diagnostic

[armanbilge]

Version(s)
1.0.4

Describe the bug

The default versions of Scala Native (and Scala.js) are behind the latest releases. Because they are not guaranteed to be forward-compatible, that means that libraries and toolkits built with the latest Native/JS releases may not work with scala-cli out-of-the-box.

To Reproduce

//> using toolkit typelevel:latest
//> using platform native

import cats.effect.*

object App extends IOApp.Simple:
  def run = IO.unit

$ scala-cli foo.scala 
Compiling project (Scala 3.3.0, Scala Native)
Compiled project (Scala 3.3.0, Scala Native)
[info] Linking (11491 ms)
[info] Checking intermediate code (quick) (300 ms)
[info] Discovered 1744 classes and 11262 methods
[info] Optimizing (debug mode) (6478 ms)
[info] Generating intermediate code (6400 ms)
[info] Produced 16 files
[info] Compiling to native code (4600 ms)
[error] /usr/bin/ld: /workspace/sandbox/.scala-build/sandbox_05adbf4d1c_919f107979-b0bbeb3ef8/native/native-code-nativelib_native0.4_3-0.4.15-1/scala-native/gc/immix/Marker.c.o: in function `Object_IsWeakReference':
[error] Marker.c:(.text+0x88): undefined reference to `__weak_ref_ids_min'
[error] /usr/bin/ld: Marker.c:(.text+0xa2): undefined reference to `__weak_ref_ids_max'
[error] clang: error: linker command failed with exit code 1 (use -v to see invocation)
Exception in thread "main" scala.scalanative.build.BuildException: Failed to link /workspace/sandbox/.scala-build/sandbox_05adbf4d1c_919f107979-b0bbeb3ef8/native/main
        at scala.scalanative.build.LLVM$.link(LLVM.scala:131)
        at scala.scalanative.build.Build$.$anonfun$build$1(Build.scala:106)
        at scala.scalanative.build.Logger.time(Logger.scala:35)
        at scala.scalanative.build.Logger.time$(Logger.scala:32)
        at scala.scalanative.cli.utils.FilteredLogger.time(FilteredLogger.scala:7)
        at scala.scalanative.build.Build$.build(Build.scala:61)
        at scala.scalanative.cli.ScalaNativeLd$.$anonfun$runLd$3(ScalaNativeLd.scala:73)
        at scala.scalanative.util.Scope$.apply(Scope.scala:32)
        at scala.scalanative.cli.ScalaNativeLd$.runLd(ScalaNativeLd.scala:72)
        at scala.scalanative.cli.ScalaNativeLd$.main(ScalaNativeLd.scala:42)
        at scala.scalanative.cli.ScalaNativeLd.main(ScalaNativeLd.scala)
Error: scala.build.errors.ScalaNativeBuildError: Error compiling with Scala Native
For more details, please see '/workspace/sandbox/.scala-build/stacktraces/1695712231-10233067241258950535.log'

Adding //> using nativeVersion 0.4.15 fixes it.

Expected behaviour

IMHO a latest toolkit should Just Work:tm: with latest scala-cli. If a user has to explicitly lookup and add a version then it undermines the convenience factor and generally breaks user expectation.

• Compiling for scala-native throws scala.MatchError typelevel/toolkit#49

There are two possible strategies to address this.

1. scala-cli should update its defaults to latest Scala.js and Scala Native versions as soon as they are released.

2. Toolkits should not release with libraries that require newer Native/JS versions than what scala-cli ships by default. In practice, this means the libraries aggregated in toolkits should also hold back on JS/Native updates, otherwise it might be impossible to release a toolkit with an important bug fix to one of those libraries.

cc @TonioGela

[SethTisue]

related: #2394

[armanbilge]

FYI the official Scala Toolkit is now also broken.

• Toolkit is broken on Native scala/toolkit#36

[SethTisue]

it appears that 1.0.5 is imminent: #2469 was a suitable fix already merged, such that it will be included in the release?

[armanbilge]

Probably there will be the band-aid fix in that release (i.e. bump to latest Scala Native toolchain). But we will keep hitting this problem unless a more permanent solution is arrived at (which could be as simple as releasing a new scala-cli every time there is a new Scala Native or Scala.js release).

[TonioGela]

(which could be as simple as releasing a new scala-cli every time there is a new Scala Native or Scala.js release).

or Scala release

[bishabosha]

was a suitable fix already merged, such that it will be included in the release?

looks like in #2465 it is specifically now testing toolkit:latest compatibility with default platform versions

[tgodzik]

The biggest problem with this is that if we release Scala CLI with certain default versions, we don't want people to have them suddenly updated, which might also break things. This is why toolkit latest is a problem. Maybe we should only have a default version of toolkit for each Scala CLI version? We could have toolkit default as the main way of configuring it

[bishabosha]

we don't want people to have them suddenly updated, which might also break things. This is why toolkit latest is a problem

even if someone doesnt change their Scala CLI version and then toolkit:latest suddenly breaks from a newer version, I guess there's not really a record anywhere of which version it used to resolve to before it broke, (so should we add lock files?)

[sjrd]

For the love of everything that is good and beautiful in this world, please don't solve a self-inflicted problem (latest/default versions actually written down and persisted in files) by another problem (lock files).

latest-style versions should only exist in "transient" states, like a command that does scala install toolkit.latest and then it adds in the file a using toolkit 1.2.3 if that's the latest version at the time of executing scala install.

---

Edit: elaborating at @bishabosha's request.

I've said this before, in a number of occasions: there should be no reason to get to the place where I want to write "latest". Either I'm in a command line, and I type in scala install toolkit, and it adds //> using toolkit 1.2.3 in my file. Or I'm in an IDE, and I start typing

//> using toolkit

and there the IDE's autocompletion suggests to complete into

//> using toolkit 1.2.3

There's no reasonable workflow where I would even write the word "latest". The tools should preemptively get the actual latest version and write it for me.

[bishabosha]

(latest/default versions actually written down and persisted in files) by another problem (lock files).

translation: code should not compile if it says //> using toolkit latest, or at least it should come with a warning and quick fix to substitute the actual version, explaining that you shouldn't share the code/commit to git etc.

[armanbilge]

Cross-linking to my other open issue 🙂

• A way to generate/rewrite directives to pin all the things for a stable, reproducible build #2145

[armanbilge]

There's no reasonable workflow where I would even write the word "latest". The tools should preemptively get the actual latest version and write it for me.

If we follow this thread (and I think it's a good thread to follow) then it seems to me that we need to encourage a lot more version pinning besides the toolkit version? We must pin either the version of scala-cli itself, or if not that, then the versions of Scala and Scala.js and Scala Native. Because otherwise even with pinned toolkit, the rug could be pulled under your feet by upgrading scala-cli.

[tgodzik]

Ok, so out approach would be to deprecate latest (remove from docs etc.) and also:

• add suggestion (actionable diagnostic) to switch to explicit latest version, like we do in case of outdated deps, when user wrote latest
• add option for default instead of latests, this might be useful for snippets etc. and would use the default for a specific version of ScalaCLI. We could also suggest to switch default to explicit version. This could be //> using toolkit default or just //> using toolkit like we had discussed in the past.
• add in Metals autocomplete for toolkit with the actual latest version (as suggested by Seb)

[SethTisue]

add option for default instead of latests, this might be useful for snippets etc

super useful, yes, and very useful when simply firing up a REPL. (but yes, sure, totally okay to change it to defaulting to "default" semantics rather than "latest" semantics)

[szymon-rd]

latest could be latest working for a given target. I agree that latest should be replaced with a fixed version in any script or project as soon as it becomes possible that we will use it more than one time. It only brings value when you start a REPL or write a one-use script to, for example, traverse directories, modify the files, etc. But it's a common case, so the value is there. I use latest 90% of the time because that's what I use the toolkit for.

[armanbilge]

I use latest 90% of the time because that's what I use the toolkit for.

In that case, do we care about the latest version of toolkit staying compatible with the current release of scala-cli? The issue is primarily with Scala.js and Native, which is probably not a priority for REPL or one-use scripts. In theory it does also affect Scala versions, but I guess that breakage happens less frequently and can be better coordinated.

[armanbilge]

scala-cli should update its defaults to latest Scala.js and Scala Native versions as soon as they are released.

Coming back to this ...

So we release the Typelevel Toolkit every time one of the libraries bundled in the toolkit is updated. IMHO that's the point: an easy dependency to bring in the latest stuff.

So far the Scala Toolkit hasn't really done this, but it might consider this strategy as well.

Following from there: arguably scala-cli is morally a "toolkit" itself. It's not bundling libraries, but it is bundling compilers, toolchains, and plugins. So it should release frequently to keep this bundle up-to-date.

[tgodzik]

Following from there: arguably scala-cli is morally a "toolkit" itself. It's not bundling libraries, but it is bundling compilers, toolchains, and plugins. So it should release frequently to keep this bundle up-to-date.

We should release more frequently for sure,we've been just having some issues with the CI recently.