'Composite: true + Incremental: true' excessively increases compilation times

[kwonoj]

Bug Report

• Background

We're in attempt to establish project references by splitting codes into multiple tsconfig projects. First migration attempt trying to make small project, then reference existing monolithic codebase. Enabling composite option to existing tsconfig however, makes overall compilation time excessively longer which we can't try this approach.

• tsconfig
  Please note few names are redacted.

{
	"include": [
		"bin/**/*",
		"js/**/*",
		"webpack/**/*"
	],
	"exclude": ["**/node_modules", "**/.*", "js/ts-plugins/**/*"],
	"compilerOptions": {
		"allowJs": true,
		"allowSyntheticDefaultImports": true,
		"baseUrl": ".",
		"esModuleInterop": true,
		"importHelpers": true,
		"jsx": "react",
		"lib": ["dom", "es2020"],
		"downlevelIteration": true,
		"module": "esnext",
		"moduleResolution": "node",
		"target": "es5",
		"composite": true,
		"incremental": true,
		"outDir": "./.ts",
		"resolveJsonModule": true,
		"paths": {
			"@libs/*": ["js/libs/*"],
			"@docs/*": ["js/${...}/app/docs/*"],
                         ...//few other pathMappings
		},
		"noImplicitAny": true,
		"noImplicitReturns": true,
		"noImplicitThis": true,
		"noUnusedLocals": true,
		"noUnusedParameters": true,
		"strictNullChecks": true,
		"strictPropertyInitialization": true,
		"plugins": [
			{
				"name": "../../js/ts-plugins/dist/custom-plugin/index"
			}
		]
	}
}

• Compilation time

Compilation executed via

node --max-old-space-size=16384 ./node_modules/typescript/bin/tsc -p ./tsconfig.json --extendedDiagnostics

due to there are some memory pressure causes OOM without increasing heap size.

composite: false, incremental: false

Files:                        15454
Lines:                      2360040
Nodes:                      8724975
Identifiers:                2858297
Symbols:                    5053080
Types:                      2160538
Instantiations:             6523057
Memory used:               7364254K
Assignability cache size:   1029646
Identity cache size:          80555
Subtype cache size:           98716
Strict subtype cache size:   125795
Tracing time:               222.07s
I/O Read time:                1.33s
Parse time:                   9.58s
ResolveModule time:           9.42s
ResolveTypeReference time:    0.08s
Program time:                29.52s
Bind time:                    6.74s
Check time:                 478.03s
transformTime time:          11.61s
commentTime time:            63.43s
I/O Write time:               5.99s
printTime time:             228.94s
Emit time:                  228.94s
Dump types time:            185.83s
Total time:                 743.23s

composite: true, incremental: true

Files:                         15454
Lines:                       2360040
Nodes:                       8724975
Identifiers:                 2858297
Symbols:                     5067727
Types:                       2208351
Instantiations:              6654654
Memory used:                5015179K
Assignability cache size:    1030212
Identity cache size:          131336
Subtype cache size:            98687
Strict subtype cache size:    125733
Tracing time:                259.28s
I/O Read time:                 3.18s
Parse time:                   10.27s
ResolveModule time:           10.52s
ResolveTypeReference time:     0.11s
Program time:                 34.22s
Bind time:                     6.90s
Check time:                  562.21s
transformTime time:        14012.56s
commentTime time:            261.38s
printTime time:            14893.35s
Emit time:                 14894.61s
I/O Write time:               13.35s
Dump types time:             377.50s
Total time:                15497.94s

When enabled, other stats are nearly similar but only transformTime takes excessively longer on same machine.
Below's an example file from generated trace shows single file's transformNodes duration differences.

composite: false, incremental: false

{
  "pid": 1,
  "tid": 1,
  "ph": "X",
  "cat": "emit",
  "ts": 560591824.5819993,
  "name": "transformNodes",
  "dur": 1967.9180011749268,
  "args": {
    "path": "/users/.../${redacted}.js"
  }
}

composite: true, incremental: true

{
  "pid": 1,
  "tid": 1,
  "ph": "X",
  "cat": "emit",
  "ts": 10553221859.568998,
  "name": "transformNodes",
  "dur": 134794797.68300056,
  "args": {
    "path": "/users/.../${redacted}.js"
  }
}

This looks like mostly occurring around .js files since we have allowJs: true and majority of files are still .js, but some of .ts and .tsx are large offenders around long transformTime as well.

It looks like #39969 is somewhat similar while it doesn't look like exact same.

🔎 Search Terms

Incremental build slow

🕗 Version & Regression Information

• Experiencing tsc >= 4.1.3

⏯ Playground Link

💻 Code

// We can quickly address your report if:
//  - The code sample is short. Nearly all TypeScript bugs can be demonstrated in 20-30 lines of code!
//  - It doesn't use external libraries. These are often issues with the type definitions rather than TypeScript bugs.
//  - The incorrectness of the behavior is readily apparent from reading the sample.
// Reports are slower to investigate if:
//  - We have to pare too much extraneous code.
//  - We have to clone a large repo and validate that the problem isn't elsewhere.
//  - The sample is confusing or doesn't clearly demonstrate what's wrong.

Due to internal policies, it is not possible to share actual code unfortunately. (and it's large)

🙁 Actual behavior

Whole compilation time takes more than hours

🙂 Expected behavior

Compilation time would increase, but within range of original compile time which takes less than 10 min in worst case.

[sheetalkamat]

Seems like you will get same behavior if you enabled declaration. #42960 will postpone the cost to do declaration emit for shape to later time and #41219 will help you avoid that cost at expense of making every change seem like a change to non local file.

[kwonoj]

Interesting, thanks for the pointer. It looks like somewhat related and would like to try out those changes.