Building Scala 3 on Windows ↩

Source code of the Dotty project is hosted on Github and continuous delivery is performed on the Dotty CI server ¹ from LAMP/EPFL.
This document describes changes we made to the lampepfl/dotty repository in order to successfully execute the CI build/test steps on a Windows runner (GitHub-hosted) and to reproduce the same build/test steps on a local Windows machine.

This document is part of a series of topics related to Scala 3 on Microsoft Windows:

Running Scala 3 on Windows
Building Scala 3 on Windows ▼
Data Sharing and Scala 3 on Windows
OpenJDK and Scala 3 on Windows

Ada, Akka, C++, Dart, Deno, Flix, Golang, GraalSqueak, GraalVM, Kotlin, LLVM, Node.js, Rust, Spark, Spring, TruffleSqueak and, Wix Toolset and Zig are other topics we are continuously investigating.

Project dependencies

Our Dotty fork depends on the following external software for the Microsoft Windows platform:

Git 2.46 (release notes)
sbt 1.10 (requires Java 8) (release notes)
Temurin OpenJDK 11 ² (release notes)

Optionally one may also install the following software:

Temurin OpenJDK 17 (release notes)

☛ Installation policy
When possible we install software from a Zip archive rather than via a Windows installer. In our case we defined C:\opt\ as the installation directory for optional software tools (in reference to the /opt/ directory on Unix).

For instance our development environment looks as follows (September 2024):

C:\opt\Git\                        (388 MB)
C:\opt\jdk-temurin-1.8.0u412-b08\  (189 MB)
C:\opt\jdk-temurin-11.0.24_8\      (303 MB)
C:\opt\jdk-temurin-17.0.12_7\      (301 MB)
C:\opt\sbt\                        (137 MB)

🔎 Git for Windows provides a BASH emulation used to run git from the command line (as well as over 250 Unix commands like awk, diff, file, grep, more, mv, rmdir, sed and wc).

Directory structure ▴

The directory structure of the Dotty repository ³ is quite complex but fortunately we only have to deal with three subdirectories. Concretely our additions affect two groups of directories :

In directories dotty\bin\ and dotty\project\scripts\ we add the batch files to support our build process on a Windows machine.

dotty\bin\common.bat
dotty\bin\scala.bat
dotty\bin\scalac.bat
dotty\bin\scaladoc.bat
dotty\project\scripts\bootstrapCmdTests.bat
dotty\project\scripts\cmdTests.bat
dotty\project\scripts\cmdTestsCommon.inc.bat.bat
dotty\project\scripts\genDocs.bat

In directory dotty\dist\bin\ we add the batch files to be included in the generated software distributions.
```
dotty\dist\bin\common.bat
dotty\dist\bin\scala.bat
dotty\dist\bin\scalac.bat
dotty\dist\bin\scaladoc.bat
```
🔎 With PR#13006 those batch files have been added to the 3.0.2 branch of Scala 3.

In the next section we give a brief description of the batch files present in those directories.

Batch/Bash commands ▴

We distinguish several sets of batch commands:

Directory bin\3.0\ - This directory contains the shell scripts and batch files to be added unchanged to a Scala 3 software distribution.

🔎 With the merge of PR#13006 into the 3.0.2 branch of the Dotty project the copy operation below is no more required.
```
> cp bin\3.0\*.bat dotty\dist\bin
> dir /b dotty\dist\bin
common
common.bat
scala
scala.bat
scalac
scalac.bat
scaladoc
scaladoc.bat
```

build.bat/build.sh - Both commands perform on a Windows machine the same build/test steps as specified in file ci.yaml and executed on the Dotty CI server.

🔎 Scala users working in a Linux-like environment such as Cygwin or MSYS2 can perform the same tasks with the bash script ./build.sh.

> cp bin\dotty\build.* dotty
> cp bin\dotty\project\scripts\*.bat dotty\project\scripts\
> cd dotty

Command build.bat help display the help message.

> cd
W:\dotty
 
> build help
Usage: build { <option> | <subcommand> }
 
  Options:
    -timer                 print the total build time
    -verbose               print progress messages
 
  Subcommands:
    arch[ives]             generate gz/zip archives (after bootstrap)
    boot[strap]            generate+test bootstrapped compiler (after compile)
    clean                  clean project
    clone                  update submodules
    compile                generate+test 1st stage compiler (after clone)
    community              test community-build
    doc[umentation]        generate documentation (after bootstrap)
    help                   print this help message
    sbt                    test sbt-dotty (after bootstrap)
    update                 fetch/merge upstream repository
 
  Advanced subcommands (no deps):
    arch[ives]-only        generate ONLY gz/zip archives
    boot[strap]-only       generate+test ONLY bootstrapped compiler
    compile-only           generate+test ONLY 1st stage compiler
    doc[umentation]-only   generate ONLY documentation
    sbt-only               test ONLY sbt-dotty

Subcommands obey the following dependency rules for their execution:

A depends on B	Execution time^a)	Output from A
`clean` → ∅	<1 min
`clone` → ∅	<1 min
`compile` → `clone`	~24 min	`compiler\target\` `library\target` `sbt-bridge\target\`
`bootstrap` → `compile`	~45 min
`community` → ∅
`archives` → `bootstrap`		`dist-bootstrapped\target\.gz,.zip`
`documentation` → `bootstrap`		`docs\_site\.html` `docs\docs\.md`
`sbt` → `bootstrap`

_{^a) Average execution time measured on a i7-i8550U laptop with 16 GB of memory.}

🔎 Subcommands whose name ends with -only help us to execute one single step without running again the precedent ones. They are listed as Advanced subcommands by command build help and should never be used in an automatic build.

Subcommand Execution time Output

compile-only ~ 24 min

bootstrap-only ~ 26 min

archives-only < 1 min dist-bootstrapped\target\*.gz,*.zip

documentation-only < 3 min docs\_site\*.html
docs\docs\*.md

sbt-only

In particular we have the following equivalences:

Command Equivalent command

build compile build clone compile-only

build bootstrap build compile bootstrap-only

build archives build bootstrap archives-only

build documentation build bootstrap documentation-only

build sbt build bootstrap sbt-only

Directory bin\dotty\bin\ - This directory contains batch files used internally during the build process (see the bootstrapCmdTests.bat command).
```
> cp bin\dotty\bin\*.bat dotty\bin
> dir /b dotty\bin
common
common.bat
scala
scala.bat
scalac
scalac.bat
scaladoc
scaladoc.bat
```
bin\dotty\project\scripts\ - This directory contains bash files to performs test steps on a Windows machine in a similar manner to the shell scripts on the Dotty CI server (see console output in section Usage examples).

Batch file (build.bat) Bash script (./build.sh)

cmdTests.bat cmdTests

bootstrapCmdTests.bat bootstrapCmdTests

genDocs.bat genDocs

🔎 From me reported issues and pull requests are listed in document CONTRIBUTIONS.md.

Usage examples ▴

Command build.bat consists of ~400 lines of batch/Powershell code and features the following subcommands:

`build.bat clean`

Command build.bat clean removes all generated and untracked files/directories from our Dotty fork.
Internally, build clean executes the command sbt clean (one may use git clean -xdf to remove all untracked directories/files, including build products).

> build clean
[...(sbt)...]
Removing .vscode/
Removing HelloWorld$.class
Removing HelloWorld.class
Removing HelloWorld.tasty
Removing compiler/target/
Removing dist-bootstrapped/
Removing doc-tool/target/
Removing dotty-bootstrapped/
Removing interfaces/target/
Removing library/target/
Removing out/
Removing project/project/project/
Removing project/project/target/
Removing project/target/
Removing sbt-bridge/target/
Removing scala-compiler/
Removing scala-library/
Removing scala-reflect/
Removing scalap/
Removing setenv.bat
Removing target/
Removing testlogs/

Command build -verbose clean also displays the tool paths/options and the current Git branch:

> build -verbose clean
Tool paths
   "GIT_CMD=C:\opt\Git\bin\git.exe"
   "JAVA_CMD=C:\opt\jdk-openjdk-11.0.23_9\bin\java.exe"
   "SBT_CMD=C:\opt\sbt\bin\sbt.bat"
Tool options
   JAVA_OPTS=-Xmx2048m -XX:ReservedCodeCacheSize=2048m -XX:MaxMetaspaceSize=1024m
   SBT_OPTS=-Ddotty.drone.mem=4096m -Dsbt.ivy.home=U:\.ivy2\ -Dsbt.log.noformat=true
Current Git branch
   master
 
[...(sbt)...]

`build.bat compile`

Command build.bat compile generates the "1st stage compiler" for Scala 3 and executes the relevant test suites.

> build compile
sbt compile and sbt test
[...]
[info] Done compiling.
[...]
[info] Done packaging.
[...]
[info] Test run started
[info] Test dotty.tools.dottydoc.TestWhitelistedCollections.arrayAndImmutableHasDocumentation started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 21.918s
[info] Test run started
[...]
8 suites passed, 0 failed, 8 total
[...]
[info] Test run started
[...]
2 suites passed, 0 failed, 2 total
[...]
[info] Test run started
[...]
11 suites passed, 0 failed, 11 total
[...]
[info] Test run started
[...]
[info] Passed: Total 73, Failed 0, Errors 0, Passed 73
[info] Passed: Total 290, Failed 0, Errors 0, Passed 288, Skipped 2
[success] Total time: 1063 s, completed 16 nov. 2018 15:39:19
testing sbt scalac and scala
hello world
testing sbt scalac -from-tasty and scala -classpath
hello world
testing sbt scalac -decompile
[...]
testing sbt scala with no -classpath
hello world
testing loading tasty from .tasty file in jar
[...]

`build.bat bootstrap`

Command build.bat bootstrap works as follows: if execution of the compile subcommand was successful the bootstrap subcommand generates the "bootstrap compiler" for Scala 3 and executes the relevant test suites.

> build bootstrap
[...]

`build.bat community`

Command build.bat community generates subprojects from community-build\community-projects\:

> build community
[...]

`build.bat archives`

Command build.bat archives works as follows: if execution of the bootstrap subcommand was successful the archives subcommand generates the gz/zip archives.
Below we execute the arch-only subcommand for the sake of brievity (previous steps are assumed to be successful):

> build arch-only
[...]
 
> dir /a-d /b dist\target
scala3-3.4.0-RC1-bin-SNAPSHOT.tar.gz
scala3-3.4.0-RC1-bin-SNAPSHOT.zip

`build.bat documentation`

Command build.bat documentation works as follows: if execution of the bootstrap subcommand was successful the documentation subcommand generates the Scala 3 website and the online Scala 3 documentation.
Below we execute the doc-only subcommand for the sake of brievity (previous operations are assumed to be successful):

> build -timer doc-only
Working directory: W:\dotty
[...]
[info] Running (fork) dotty.tools.dottydoc.Main -siteroot docs -project Dotty -project-version 3.3.1-bin-SNAPSHOT -project-url https://github.com/lampepfl/dotty ...
Compiling (1/406): AlternateConstructorsPhase.scala
[...]
Compiling (406/406): package.scala
[...]
28 warnings found
there were 3987 feature warning(s); re-run with -feature for details
[doc info] Generating doc page for: dotty.tools.dotc.plugins
[...]
[doc info] Generating doc page for: dotty.tools.dotc.core.unpickleScala2.Scala2Unpickler$.TempPolyType$
=================================================================================
Dottydoc summary report for project `Dotty`
=================================================================================
Documented members in public API:
[...]
Summary:
 
public members with docstrings:    5181/14606 (35%)
protected members with docstrings: 164/537 (30%)
=================================================================================
 
Documented members in internal API:
[...]
Summary internal API:
 
public members with docstrings:    154/601 (25%)
protected members with docstrings: 6/60 (10%)
private members with docstrings:   464/2450 (18%)
total warnings with regards to compilation and documentation: 29
[success] Total time: 146 s, completed 29 nov. 2018 11:49:22
Total execution time: 00:02:36

Output directory docs\_site\ contains the files of the online Scala 3 documentation:

> dir /b docs\_site
.gitignore
api
blog
css
docs
images
index.html
js
sidebar.yml
versions
> dir /a-d /b /s docs\_site\*.html | wc -l
2551
> dir /a-d /b /s docs\_site\*.jpg docs\_site\*.png docs\_site\*.svg | wc -l
23
> dir /a-d /b /s docs\_site\*.js | wc -l
9

Output directory docs\docs\ contains the Markdown files of the Scala 3 website:

> dir /b docs\docs
contributing
index.md  
internals 
reference 
release-notes
resources   
typelevel.md
usage
> dir /a-d /b /s docs\docs\*.md | wc -l
88

`cmdTests.bat`

Command project\scripts\cmdTests.bat performs several tests running Scala 3 commands from sbt. In the normal case, command cmdTests is called by command build compile but may also be called directly.

> cmdTests
testing sbt scalac and scala
hello world
testing sbt scalac -from-tasty and scala -classpath
hello world
testing sbt scalac -decompile
[info] Loading project definition from W:\dotty\project\project
[info] Loading project definition from W:\dotty\project
  def main(args: scala.Array[scala.Predef.String]): scala.Unit = scala.Predef.println("hello world")
testing sbt scalac -decompile from file
[info] Loading project definition from W:\dotty\project\project
[info] Loading project definition from W:\dotty\project
  def main(args: scala.Array[scala.Predef.String]): scala.Unit = scala.Predef.println("hello world")
testing sbt scala with no -classpath
hello world
testing loading tasty from .tasty file in jar
[info] Loading project definition from W:\dotty\project\project
[info] Loading project definition from W:\dotty\project
  def main(args: scala.Array[scala.Predef.String]): scala.Unit = scala.Predef.println("hello world")
[...]

`bootstrapCmdTests.bat`

Command project\scripts\bootstrapCmdTests.bat performs several benchmarks and generates the documentation page for the tests\pos\HelloWorld.scala program. In the normal case, command bootstrapCmdTests is called by command build bootstrap but may also be called directly.

> bootstrapCmdTests
[...]
[info] Updating dotty-bench...
[...]
[info] Running (fork) dotty.tools.benchmarks.Bench 1 1 tests/pos/alias.scala
# JMH version: 1.35
# VM version: JDK 11.0.18, VM 11.0.18+10
# VM invoker: C:\opt\jdk-temurin-11.0.20_8\bin\java.exe
# VM options: -Xms2G -Xmx2G
# Warmup: 1 iterations, 1 s each
# Measurement: 1 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: dotty.tools.benchmarks.Worker.compile

# Run progress: 0.00% complete, ETA 00:00:02
# Fork: 1 of 1
# Warmup Iteration   1: 3011.972 ms/op
Iteration   1: 533.625 ms/op


Result "dotty.tools.benchmarks.Worker.compile":
  533.625 ms/op


# Run complete. Total time: 00:00:05

Benchmark       Mode  Cnt    Score   Error  Units
Worker.compile  avgt       533.625          ms/op
[success] Total time: 21 s, completed 3 déc. 2018 09:44:07
[...]
[info] Updating dotty-bench-bootstrapped...
[...]
[info] Running (fork) dotty.tools.benchmarks.Bench 1 1 tests/pos/alias.scala
# JMH version: 1.35
# VM version: JDK 11.0.18, VM 11.0.18+10
# VM invoker: C:\opt\jdk-temurin-11.0.20_8\bin\java.exe
# VM options: -Xms2G -Xmx2G
# Warmup: 1 iterations, 1 s each
# Measurement: 1 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: dotty.tools.benchmarks.Worker.compile

# Run progress: 0.00% complete, ETA 00:00:02
# Fork: 1 of 1
# Warmup Iteration   1: 2359.948 ms/op
Iteration   1: 361.619 ms/op


Result "dotty.tools.benchmarks.Worker.compile":
  361.619 ms/op


# Run complete. Total time: 00:00:04

Benchmark       Mode  Cnt    Score   Error  Units
Worker.compile  avgt       361.619          ms/op
[success] Total time: 21 s, completed 3 déc. 2018 09:44:42
[...]
[info] Running (fork) dotty.tools.benchmarks.Bench 1 1 -with-compiler compiler/src/dotty/tools/dotc/core/Types.scala
# JMH version: 1.35
# VM version: JDK 11.0.18, VM 11.0.18+10
# VM invoker: C:\opt\jdk-temurin-11.0.20_8\bin\java.exe
# VM options: -Xms2G -Xmx2G
# Warmup: 1 iterations, 1 s each
# Measurement: 1 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: dotty.tools.benchmarks.Worker.compile

# Run progress: 0.00% complete, ETA 00:00:02
# Fork: 1 of 1
# Warmup Iteration   1: 13858.101 ms/op
Iteration   1: 5828.334 ms/op


Result "dotty.tools.benchmarks.Worker.compile":
  5828.334 ms/op


# Run complete. Total time: 00:00:20

Benchmark       Mode  Cnt     Score   Error  Units
Worker.compile  avgt       5828.334          ms/op
[success] Total time: 28 s, completed 3 déc. 2018 09:45:23
testing scala.quoted.Expr.run from sbt scala
[...]
[info] [dist-bootstrapped] Creating a distributable package in dist-bootstrapped\target\pack
[...]
[info] [dist-bootstrapped] done.
[success] Total time: 8 s, completed 3 déc. 2018 09:46:13
testing ./bin/scalac and ./bin/scala
testing ./bin/sclac -from-tasty and scala -classpath
testing ./bin/scalad
Compiling (1/1): HelloWorld.scala
[doc info] Generating doc page for: 
[doc info] Generating doc page for: .HelloWorld$
[doc info] Generating doc page for: .HelloWorld$
[...]
public members with docstrings:    0
protected members with docstrings: 0
private members with docstrings:   0

`genDocs.bat`

Command genDocs.bat generates the documentation page for program tests\pos\HelloWorld.scala.

> genDocs
Working directory: W:\dotty
[..(sbt)..]       
[info] Running (fork) dotty.tools.dottydoc.Main -siteroot docs -project Dotty -project-version 
OT -project-url https://github.com/lampepfl/dotty -classpath ...
[...]
Summary:

public members with docstrings:    5187/14614 (35%)
protected members with docstrings: 165/538 (30%)
================================================================================
[...]
Summary internal API:

public members with docstrings:    156/604 (25%)
protected members with docstrings: 6/60 (10%)
private members with docstrings:   466/2454 (18%)
total warnings with regards to compilation and documentation: 29
[success] Total time: 135 s, completed 3 déc. 2018 15:05:04

Footnotes ▴

[1] Continuous Integration/Delivery (CI/CD) ↩

Steps are: Checkout → Compile → Test → Deploy.

Software	CI/CD service	Hosting
Scala 3 (Dotty)	Actions ⁽¹⁾	EPFL in Lausanne, Switzerland
Scala 2	Jenkins ⁽²⁾ Travis CI ⁽³⁾	Lightbend in San-Francisco, USA Travis in Berlin, Germany
Oracle OpenJDK	Jenkins ⁽²⁾	Oracle
IBM OpenJ9	Jenkins ⁽²⁾	IBM

_{⁽¹⁾ Self-hosted Github runners, ⁽²⁾ Written in Java, ⁽³⁾ Written in Ruby.}

[2] Java LTS (2018-11-18) ↩

Oracle annonces in his Java SE Support Roadmap he will stop public updates of Java SE 8 for commercial use after January 2019. Launched in March 2014 Java SE 8 is classified an LTS release in the new time-based system and Java SE 11, released in September 2018, is the current LTS release.
(see also Java 11 keynote from Hendrik Ebbers at JVM-Con 2018).

[3] Git master repository ↩

Nowadays we have experienced two times the error Server does not allow request for unadvertised object.. when synchronizing our fork with the lampepfl/dotty repository:

> git fetch upstream master
> git merge upstream/master
[...]
Error: Server does not allow request for unadvertised object ...

That error is caused by one of the subprojects in directory community-build\community-projects\ and can be solved with the following commands:

> git submodule sync
> git submodule update --depth 50

mics/September 2024 ▲

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUILD.md

BUILD.md

Building Scala 3 on Windows ↩

Project dependencies

Directory structure ▴

Batch/Bash commands ▴

Usage examples ▴

`build.bat clean`

`build.bat compile`

`build.bat bootstrap`

`build.bat community`

`build.bat archives`

`build.bat documentation`

`cmdTests.bat`

`bootstrapCmdTests.bat`

`genDocs.bat`

Footnotes ▴

Subcommand	Execution time	Output
`compile-only`	~ 24 min
`bootstrap-only`	~ 26 min
`archives-only`	< 1 min	`dist-bootstrapped\target\.gz,.zip`
`documentation-only`	< 3 min	`docs\_site\.html` `docs\docs\.md`
`sbt-only`

Command	Equivalent command
`build compile`	`build clone compile-only`
`build bootstrap`	`build compile bootstrap-only`
`build archives`	`build bootstrap archives-only`
`build documentation`	`build bootstrap documentation-only`
`build sbt`	`build bootstrap sbt-only`

Batch file (`build.bat`)	Bash script (`./build.sh`)
`cmdTests.bat`	`cmdTests`
`bootstrapCmdTests.bat`	`bootstrapCmdTests`
`genDocs.bat`	`genDocs`

Files

BUILD.md

Latest commit

History

BUILD.md

File metadata and controls

Building Scala 3 on Windows ↩

Project dependencies

Directory structure ▴

Batch/Bash commands ▴

Usage examples ▴

build.bat clean

build.bat compile

build.bat bootstrap

build.bat community

build.bat archives

build.bat documentation

cmdTests.bat

bootstrapCmdTests.bat

genDocs.bat

Footnotes ▴

`build.bat clean`

`build.bat compile`

`build.bat bootstrap`

`build.bat community`

`build.bat archives`

`build.bat documentation`

`cmdTests.bat`

`bootstrapCmdTests.bat`

`genDocs.bat`