Support mill init from an existing SBT project (2000USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 2000USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

We should be able to run ./mill init to do a best effort conversion of an existing SBT project to Mill:

• Code-generate a build.mill containing a MavenModulefor the root build.sbt, the package.mill for any subfolder build.sbts or SBT subprojects in the root build.sbt
• Convert the main and test dependencies on third-party libraries to ivyDeps
• Convert inter-subproject dependencies to moduleDeps
• Set any necessary javacOptions
• Implement PublishModule with relevant metadata

Such a conversion will never be 100%, but if we can automate 80% of the conversion that will already be a huge help for people migrating to Mill or even just trying it out

[Dylanb-dev]

hello, I am having a look at this. I haven't used java much (just a bit of clojure and kotlin) so might take a couple days for PR. If I don't have anything in 48 hours happy for anyone else to take this.

[steinybot]

I'll take a crack at this.

[lihaoyi]

@steinybot go for it

[Dylanb-dev]

Hi, I have been looking at this.

I am using scala-meta to parse the SBT files, can I confirm some test examples?

Test 1 - basic sbt from sbt documentation

  ThisBuild / scalaVersion := "2.13.12"
  ThisBuild / organization := "com.example"

  lazy val hello = project
    .in(file("."))
    .settings(
      name := "Hello",
      libraryDependencies ++= Seq(
        "org.scala-lang" %% "toolkit" % "0.1.7",
        "org.scala-lang" %% "toolkit-test" % "0.1.7" % Test
      )
    )

produces ./build.sc with

import mill._, scalalib._

object Hello extends MavenModule {
  def scalaVersion = "2.13.12"
  def organization = "com.example"

  def ivyDeps = Agg(
    ivy"org.scala-lang::toolkit:0.1.7"
  )

  object test extends ScalaTests with TestModule.Munit {
    def ivyDeps = Agg(
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  }
} 

test 2 - more complex realworld sbt with publishing

import Dependencies._

ThisBuild / scalaVersion := "2.12.19"
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / organization := "com.example"
ThisBuild / organizationName := "example"

lazy val root = (project in file("."))
  .settings(
    name := "Hello",
    libraryDependencies += scalaTest % Test
  )
libraryDependencies += "org.scalameta" %% "scalameta" % "4.9.9"
// libraryDependencies += "com.typesafe.play" %% "play-json" % "2.9.2"

produces this mill build.sc

import mill._, scalalib._, publish._
import Dependencies._

object Hello extends MavenModule with PublishModule {
  def scalaVersion = "2.12.19"
  def publishVersion = "0.1.0-SNAPSHOT"
  
  def ivyDeps = Agg(
    ivy"org.scala-lang::toolkit:0.1.7"
  )

  def pomSettings = PomSettings(
    name = "example",
    organization = "com.example",
  )

  object test extends ScalaTests with TestModule.Munit {
    def ivyDeps = Agg(
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  }
}

test 3 - build.sbt with subprojects

ThisBuild / version      := "0.1.0"
ThisBuild / scalaVersion := "2.13.6"
ThisBuild / organization := "com.example"

val scalaTest = "org.scalatest" %% "scalatest" % "3.2.7"
val gigahorse = "com.eed3si9n" %% "gigahorse-okhttp" % "0.5.0"
val playJson  = "com.typesafe.play" %% "play-json" % "2.9.2"

lazy val hello = (project in file("."))
  .aggregate(helloCore)
  .dependsOn(helloCore)
  .enablePlugins(JavaAppPackaging)
  .settings(
    name := "Hello",
    libraryDependencies += scalaTest % Test,
  )

lazy val helloCore = (project in file("core"))
  .settings(
    name := "Hello Core",
    libraryDependencies ++= Seq(gigahorse, playJson),
    libraryDependencies += scalaTest % Test,
  )

Produces the following build.sc

import mill._, scalalib._

object Hello extends MavenModule with PublishModule {
  def scalaVersion = "2.13.6"
  def publishVersion = "0.1.0"
  
  def ivyDeps = Agg(
      ivy"com.eed3si9n::gigahorse-okhttp:0.5.0",
      ivy"com.typesafe.play::play-json:2.9.2",
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )

  def pomSettings = PomSettings(
    name = "Hello",
    organization = "com.example",
  )

  object test extends ScalaTests with TestModule.Munit {
    def ivyDeps = Agg(
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  }
}

with the following in ./core/package.mill

package build.core

import mill._, scalalib._

object `package` extends MavenModule with ScalaModule {
  def name = "Hello Core"
  def ivyDeps = Agg(
      ivy"com.eed3si9n::gigahorse-okhttp:0.5.0",
      ivy"com.typesafe.play::play-json:2.9.2",
      ivy"org.scala-lang::toolkit-test:0.1.7"
    )
  object test extends ScalaTests with TestModule.Munit {
      def ivyDeps = Agg(
        ivy"org.scala-lang::toolkit-test:0.1.7"
      )
}

[lihaoyi]

Those examples look reasonable. One issue is that Scala-Meta parsing has a pretty low ceiling of how sophisticated builds it can handle. e.g. once people start having variables or helper methods, which is very common, it no longer works.

I'd say that to do this well, we want to actually run SBT to evaluate the SBT build files, and export the metadata that results from this evaluation. That way you don't care how much indirection exists in the SBT build: SBT has to be able to handle it, and in the end convert it to a "dumb metadata" format that will be much easier for Mill to process

[Dylanb-dev]

Sounds reasonable, I checked sbt for meta data output and https://github.com/sbt/sbt-buildinfo but it seemed too limited. I will take a closer look.

[lihaoyi]

There's an SBT -> Bazel converter that I think goes through SBT to dump its metadata https://github.com/stripe-archive/sbt-bazel

[lihaoyi]

This may be relevant as well https://stackoverflow.com/a/62767456/871202

[sideeffffect]

@Dylanb-dev https://github.com/oyvindberg/bleep can import sbt builds too. AFAIK it works by exporting sbt build to BSP (the .bsp/ directory) and then importing information to Bleep from it.
Maybe you could use this trick too.

[llvee]

@Dylanb-dev Did you complete this?

If not please let me know in the next 24 hours as I am interested in potentially attempting this.

[Dylanb-dev]

@llvee

Go ahead, although it might be better to wait for #3449 to be finished which will help make this issue much easier to complete.

[llvee]

@Dylanb-dev Thank you for mentioning that, I will review #3449.

[lihaoyi]

@ajaychandran's PR #3449 has landed, in case anyone wants to take a crack at this. We should be able to re-use all the rendering logic, and the only issue them becomes how to extract the relevant metadata from an SBT build to feed into the renderer

[lihaoyi]

Bumping the bounty on this from 1500 -> 2000USD

[Friendseeker]

I am currently working on this bounty. I have a concrete plan (use sbt-structure to parse the sbt settings & use #3449 to transpile the parsed settings), however I have other commitments in my life so my progress is slow.

I am down teaming up with others. I do not aim to take a monetary cut (I plan to ask Haoyi for information as reward, instead of monetary reward). If you would like to team up with me, feel free to join Mill's discord channel and ping me.

[lihaoyi]

Haha if you want information just email me and ask, no need to wait for completing the ticket. Can answer whatever questions you have today to the best of my abilities