Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize scan with a lookup table to reduce branching #59106

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

DanielRosenwasser
Copy link
Member

@DanielRosenwasser DanielRosenwasser commented Jul 2, 2024

This change introduces two lookup tables to the lexical scanner/tokenizer. One is a simple array-based lookup for entities in the ASCII range, and the other is a Map for wider codepoints.

These tables assign categories to codepoints to indicate how they should be handled. For example, all line breaks, single-line whitespace, and single-character tokens can be handled generically in their own respective fashion. Doing so allows us to perform fewer individual checks to jump into or skip over a given branch.

One additional optimization is the way in which simple single-character tokens are handled is that the token itself is encoded in the lower 8 bits the category value. This allows us to avoid any branching for such cases.

This all comes at the expense of some memory overhead, along with some overhead from fetching from the table. To avoid hitting the table too often, spaces, tabs, and carriage returns, and line feeds are currently handled separately.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Jul 2, 2024
@DanielRosenwasser
Copy link
Member Author

It's probably faster to actually check the common case of spaces, tabs, and newlines, but the current draft always does the lookup.

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 193,967k (± 1.03%) 192,827k (± 0.77%) ~ 192,123k 195,863k p=0.810 n=6
Parse Time 1.57s (± 1.04%) 1.54s (± 0.76%) -0.03s (- 2.12%) 1.53s 1.56s p=0.010 n=6
Bind Time 0.85s (± 0.48%) 0.85s (± 0.96%) ~ 0.83s 0.85s p=1.000 n=6
Check Time 11.16s (± 0.73%) 11.22s (± 0.58%) ~ 11.17s 11.35s p=0.261 n=6
Emit Time 3.25s (± 1.13%) 3.26s (± 0.54%) ~ 3.23s 3.28s p=0.802 n=6
Total Time 16.84s (± 0.48%) 16.86s (± 0.44%) ~ 16.80s 17.01s p=0.574 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,360k (± 0.00%) 1,218,350k (± 0.00%) ~ 1,218,273k 1,218,452k p=0.575 n=6
Parse Time 7.97s (± 0.63%) 7.92s (± 0.48%) ~ 7.86s 7.97s p=0.064 n=6
Bind Time 2.22s (± 0.18%) 2.23s (± 0.87%) ~ 2.20s 2.25s p=0.473 n=6
Check Time 35.84s (± 0.39%) 35.73s (± 0.59%) ~ 35.51s 36.09s p=0.173 n=6
Emit Time 16.22s (± 0.31%) 16.14s (± 0.59%) ~ 16.01s 16.28s p=0.148 n=6
Total Time 62.25s (± 0.24%) 62.03s (± 0.40%) ~ 61.76s 62.39s p=0.148 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,136,980 2,136,980 ~ ~ ~ p=1.000 n=6
Types 929,117 929,117 ~ ~ ~ p=1.000 n=6
Memory used 2,117,767k (± 0.00%) 2,117,708k (± 0.01%) ~ 2,117,559k 2,117,943k p=0.298 n=6
Parse Time 9.71s (± 0.16%) 9.63s (± 0.63%) ~ 9.55s 9.73s p=0.063 n=6
Bind Time 3.39s (± 0.58%) 3.40s (± 1.27%) ~ 3.34s 3.45s p=0.571 n=6
Check Time 102.35s (± 1.25%) 102.33s (± 0.99%) ~ 100.38s 103.20s p=0.936 n=6
Emit Time 0.19s (± 4.29%) 0.19s (± 2.81%) ~ 0.19s 0.20s p=0.855 n=6
Total Time 115.64s (± 1.09%) 115.55s (± 0.87%) ~ 113.64s 116.47s p=0.423 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,226,737 +1,456 (+ 0.12%) ~ ~ p=0.001 n=6
Types 261,342 261,620 +278 (+ 0.11%) ~ ~ p=0.001 n=6
Memory used 2,341,362k (± 0.05%) 2,341,478k (± 0.06%) ~ 2,339,978k 2,343,685k p=1.000 n=6
Parse Time 7.52s (± 0.57%) 7.47s (± 1.18%) ~ 7.38s 7.59s p=0.378 n=6
Bind Time 2.78s (± 0.87%) 2.80s (± 0.69%) ~ 2.78s 2.83s p=0.191 n=6
Check Time 49.65s (± 0.65%) 49.63s (± 0.54%) ~ 49.39s 50.16s p=1.000 n=6
Emit Time 3.90s (± 5.28%) 4.07s (± 6.58%) ~ 3.75s 4.38s p=0.298 n=6
Total Time 63.88s (± 0.45%) 63.98s (± 0.59%) ~ 63.47s 64.49s p=0.689 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,226,737 +1,456 (+ 0.12%) ~ ~ p=0.001 n=6
Types 261,342 261,620 +278 (+ 0.11%) ~ ~ p=0.001 n=6
Memory used 2,424,311k (± 1.02%) 2,415,497k (± 0.05%) ~ 2,414,486k 2,417,424k p=0.471 n=6
Parse Time 6.28s (± 0.93%) 6.25s (± 0.84%) ~ 6.20s 6.35s p=0.298 n=6
Bind Time 2.02s (± 0.74%) 2.02s (± 0.60%) ~ 2.00s 2.03s p=0.517 n=6
Check Time 40.72s (± 0.37%) 40.95s (± 0.63%) ~ 40.66s 41.23s p=0.230 n=6
Emit Time 3.20s (± 6.13%) 3.22s (± 5.34%) ~ 3.04s 3.53s p=0.688 n=6
Total Time 52.23s (± 0.46%) 52.44s (± 0.54%) ~ 52.07s 52.79s p=0.173 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 259,650 +1,456 (+ 0.56%) ~ ~ p=0.001 n=6
Types 104,737 105,015 +278 (+ 0.27%) ~ ~ p=0.001 n=6
Memory used 427,489k (± 0.01%) 428,192k (± 0.01%) +704k (+ 0.16%) 428,150k 428,249k p=0.005 n=6
Parse Time 3.29s (± 0.57%) 3.27s (± 0.32%) ~ 3.26s 3.29s p=0.370 n=6
Bind Time 1.33s (± 0.31%) 1.32s (± 0.96%) ~ 1.30s 1.33s p=0.214 n=6
Check Time 17.86s (± 0.42%) 17.82s (± 0.31%) ~ 17.77s 17.92s p=0.335 n=6
Emit Time 1.25s (± 1.40%) 1.26s (± 0.96%) ~ 1.25s 1.28s p=0.317 n=6
Total Time 23.72s (± 0.41%) 23.68s (± 0.20%) ~ 23.62s 23.75s p=0.195 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,514k (± 0.04%) 369,435k (± 0.02%) ~ 369,334k 369,557k p=0.298 n=6
Parse Time 2.76s (± 0.67%) 2.76s (± 1.11%) ~ 2.72s 2.79s p=0.747 n=6
Bind Time 1.58s (± 0.52%) 1.59s (± 1.17%) ~ 1.57s 1.61s p=0.445 n=6
Check Time 15.44s (± 0.24%) 15.41s (± 0.44%) ~ 15.31s 15.49s p=0.573 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.77s (± 0.27%) 19.75s (± 0.44%) ~ 19.65s 19.88s p=0.936 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,975 2,879,975 ~ ~ ~ p=1.000 n=6
Types 975,327 975,327 ~ ~ ~ p=1.000 n=6
Memory used 3,042,839k (± 0.00%) 3,042,871k (± 0.00%) ~ 3,042,771k 3,042,936k p=0.298 n=6
Parse Time 13.73s (± 0.35%) 13.73s (± 0.40%) ~ 13.64s 13.78s p=1.000 n=6
Bind Time 4.24s (± 1.91%) 4.26s (± 2.49%) ~ 4.18s 4.40s p=0.935 n=6
Check Time 74.09s (± 2.34%) 73.73s (± 0.43%) ~ 73.21s 74.19s p=0.471 n=6
Emit Time 23.67s (± 6.50%) 24.12s (± 1.92%) ~ 23.22s 24.50s p=0.936 n=6
Total Time 115.72s (± 0.26%) 115.84s (± 0.36%) ~ 115.22s 116.41s p=0.810 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,511k (± 0.01%) 411,543k (± 0.01%) ~ 411,475k 411,623k p=0.378 n=6
Parse Time 3.18s (± 0.59%) 3.15s (± 0.53%) ~ 3.13s 3.17s p=0.062 n=6
Bind Time 1.41s (± 1.22%) 1.41s (± 0.59%) ~ 1.41s 1.43s p=0.934 n=6
Check Time 14.16s (± 0.24%) 14.19s (± 0.33%) ~ 14.14s 14.25s p=0.688 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.75s (± 0.23%) 18.75s (± 0.25%) ~ 18.69s 18.82s p=0.936 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 462,921k (± 0.07%) 462,714k (± 0.08%) ~ 462,378k 463,125k p=0.471 n=6
Parse Time 3.18s (± 0.37%) 3.08s (± 0.67%) -0.09s (- 2.94%) 3.06s 3.12s p=0.005 n=6
Bind Time 1.17s 1.17s (± 0.76%) ~ 1.16s 1.18s p=1.000 n=6
Check Time 17.89s (± 0.36%) 17.95s (± 0.71%) ~ 17.82s 18.18s p=0.470 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.24s (± 0.29%) 22.20s (± 0.64%) ~ 22.04s 22.46s p=0.378 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,294ms (± 0.67%) 2,287ms (± 0.64%) ~ 2,265ms 2,310ms p=0.298 n=6
Req 2 - geterr 5,059ms (± 0.89%) 5,059ms (± 0.55%) ~ 5,008ms 5,087ms p=1.000 n=6
Req 3 - references 264ms (± 0.24%) 264ms (± 0.21%) ~ 263ms 264ms p=0.201 n=6
Req 4 - navto 228ms (± 0.18%) 227ms (± 0.90%) ~ 223ms 228ms p=0.218 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 89ms (± 0.94%) 90ms (± 3.58%) ~ 88ms 96ms p=1.000 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,424ms (± 0.56%) 2,397ms (± 0.69%) -27ms (- 1.11%) 2,382ms 2,418ms p=0.030 n=6
Req 2 - geterr 3,791ms (± 0.52%) 3,794ms (± 0.28%) ~ 3,778ms 3,810ms p=1.000 n=6
Req 3 - references 278ms (± 0.20%) 278ms (± 0.63%) ~ 275ms 279ms p=0.508 n=6
Req 4 - navto 226ms (± 0.28%) 230ms (± 2.60%) +4ms (+ 1.70%) 227ms 242ms p=0.008 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 71ms (± 3.83%) 83ms (± 5.31%) 🔻+12ms (+16.12%) 74ms 85ms p=0.007 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,184ms (± 0.87%) 5,140ms (± 0.30%) -44ms (- 0.85%) 5,120ms 5,156ms p=0.030 n=6
Req 2 - geterr 1,133ms (± 1.61%) 1,125ms (± 1.33%) ~ 1,111ms 1,153ms p=0.423 n=6
Req 3 - references 78ms (± 4.99%) 77ms (± 4.67%) ~ 75ms 82ms p=0.344 n=6
Req 4 - navto 457ms (± 0.23%) 458ms (± 0.93%) ~ 456ms 467ms p=0.605 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 832ms (± 2.15%) 842ms (± 1.25%) ~ 830ms 857ms p=0.297 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 156.97ms (± 0.16%) 156.76ms (± 0.21%) -0.20ms (- 0.13%) 155.79ms 162.86ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 230.76ms (± 0.13%) 230.73ms (± 0.13%) ~ 229.50ms 233.81ms p=0.615 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 225.94ms (± 0.14%) 226.15ms (± 0.13%) +0.21ms (+ 0.09%) 224.77ms 229.96ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 225.80ms (± 0.16%) 225.89ms (± 0.13%) +0.08ms (+ 0.04%) 224.41ms 227.84ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@DanielRosenwasser
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 193,968k (± 1.03%) 193,382k (± 0.97%) ~ 192,131k 195,906k p=0.936 n=6
Parse Time 1.58s (± 0.40%) 1.55s (± 1.35%) -0.03s (- 1.90%) 1.51s 1.57s p=0.005 n=6
Bind Time 0.85s (± 0.60%) 0.86s (± 1.47%) ~ 0.85s 0.88s p=0.418 n=6
Check Time 11.17s (± 0.39%) 11.17s (± 0.45%) ~ 11.10s 11.25s p=1.000 n=6
Emit Time 3.29s (± 0.17%) 3.27s (± 0.59%) ~ 3.25s 3.30s p=0.100 n=6
Total Time 16.89s (± 0.31%) 16.84s (± 0.28%) ~ 16.80s 16.92s p=0.170 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,354k (± 0.00%) 1,218,334k (± 0.00%) ~ 1,218,307k 1,218,364k p=0.471 n=6
Parse Time 7.96s (± 1.13%) 7.86s (± 0.61%) -0.09s (- 1.19%) 7.78s 7.92s p=0.043 n=6
Bind Time 2.23s (± 0.52%) 2.22s (± 0.68%) ~ 2.21s 2.25s p=0.401 n=6
Check Time 35.74s (± 0.30%) 35.75s (± 0.36%) ~ 35.59s 35.97s p=0.936 n=6
Emit Time 16.16s (± 0.44%) 16.12s (± 0.29%) ~ 16.05s 16.18s p=0.377 n=6
Total Time 62.08s (± 0.21%) 61.95s (± 0.17%) ~ 61.81s 62.14s p=0.065 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,136,980 2,136,980 ~ ~ ~ p=1.000 n=6
Types 929,117 929,117 ~ ~ ~ p=1.000 n=6
Memory used 2,117,728k (± 0.01%) 2,117,653k (± 0.01%) ~ 2,117,562k 2,117,860k p=0.378 n=6
Parse Time 7.89s (± 0.19%) 7.77s (± 0.29%) -0.12s (- 1.52%) 7.75s 7.81s p=0.005 n=6
Bind Time 2.76s (± 0.37%) 2.76s (± 0.70%) ~ 2.73s 2.79s p=1.000 n=6
Check Time 83.71s (± 0.25%) 83.61s (± 0.39%) ~ 83.29s 84.12s p=0.470 n=6
Emit Time 0.16s (± 4.65%) 0.16s (± 3.95%) ~ 0.15s 0.17s p=0.718 n=6
Total Time 94.52s (± 0.22%) 94.31s (± 0.34%) ~ 93.98s 94.78s p=0.336 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,410 +129 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,443 +101 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,350,713k (± 1.06%) 2,341,270k (± 0.04%) ~ 2,339,826k 2,342,288k p=0.378 n=6
Parse Time 7.52s (± 0.49%) 7.42s (± 0.98%) -0.09s (- 1.26%) 7.34s 7.55s p=0.045 n=6
Bind Time 2.78s (± 0.49%) 2.79s (± 0.77%) ~ 2.75s 2.81s p=0.569 n=6
Check Time 49.78s (± 0.71%) 49.77s (± 0.38%) ~ 49.51s 50.03s p=1.000 n=6
Emit Time 3.94s (± 5.23%) 3.85s (± 3.21%) ~ 3.78s 4.10s p=0.688 n=6
Total Time 64.05s (± 0.74%) 63.84s (± 0.55%) ~ 63.48s 64.46s p=0.689 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,410 +129 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,443 +101 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,415,021k (± 0.04%) 2,416,113k (± 0.05%) ~ 2,415,052k 2,417,912k p=0.128 n=6
Parse Time 7.80s (± 0.99%) 7.66s (± 1.02%) -0.13s (- 1.69%) 7.52s 7.73s p=0.020 n=6
Bind Time 2.51s (± 0.39%) 2.52s (± 1.05%) ~ 2.48s 2.55s p=0.870 n=6
Check Time 50.40s (± 0.64%) 50.43s (± 0.51%) ~ 50.20s 50.86s p=1.000 n=6
Emit Time 3.99s (± 4.97%) 3.85s (± 2.53%) ~ 3.73s 3.99s p=0.336 n=6
Total Time 64.71s (± 0.41%) 64.47s (± 0.37%) ~ 64.26s 64.89s p=0.173 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,323 +129 (+ 0.05%) ~ ~ p=0.001 n=6
Types 104,737 104,838 +101 (+ 0.10%) ~ ~ p=0.001 n=6
Memory used 427,490k (± 0.01%) 427,609k (± 0.01%) +119k (+ 0.03%) 427,547k 427,695k p=0.005 n=6
Parse Time 3.30s (± 0.70%) 3.23s (± 0.54%) -0.07s (- 1.97%) 3.21s 3.26s p=0.005 n=6
Bind Time 1.32s (± 0.91%) 1.31s (± 1.42%) ~ 1.28s 1.33s p=0.096 n=6
Check Time 17.86s (± 0.30%) 17.84s (± 0.29%) ~ 17.78s 17.91s p=0.471 n=6
Emit Time 1.26s (± 0.83%) 1.26s (± 0.82%) ~ 1.24s 1.27s p=0.801 n=6
Total Time 23.74s (± 0.27%) 23.63s (± 0.22%) -0.11s (- 0.46%) 23.58s 23.69s p=0.037 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,602k (± 0.05%) 369,472k (± 0.01%) ~ 369,409k 369,567k p=0.298 n=6
Parse Time 2.78s (± 1.50%) 2.72s (± 0.72%) -0.05s (- 1.92%) 2.70s 2.75s p=0.036 n=6
Bind Time 1.57s (± 0.40%) 1.59s (± 1.11%) ~ 1.56s 1.61s p=0.081 n=6
Check Time 15.44s (± 0.14%) 15.45s (± 0.19%) ~ 15.42s 15.49s p=0.625 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.79s (± 0.22%) 19.76s (± 0.18%) ~ 19.71s 19.80s p=0.147 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,975 2,879,975 ~ ~ ~ p=1.000 n=6
Types 975,327 975,327 ~ ~ ~ p=1.000 n=6
Memory used 3,042,832k (± 0.00%) 3,042,846k (± 0.00%) ~ 3,042,776k 3,042,921k p=0.872 n=6
Parse Time 13.70s (± 0.25%) 13.63s (± 0.29%) -0.07s (- 0.51%) 13.57s 13.68s p=0.020 n=6
Bind Time 4.24s (± 2.07%) 4.20s (± 0.18%) ~ 4.19s 4.21s p=0.210 n=6
Check Time 73.44s (± 0.29%) 74.72s (± 2.03%) +1.28s (+ 1.75%) 73.36s 77.27s p=0.045 n=6
Emit Time 24.17s (± 1.22%) 23.06s (± 6.83%) ~ 20.45s 24.52s p=0.335 n=6
Total Time 115.55s (± 0.40%) 115.61s (± 0.22%) ~ 115.30s 116.02s p=1.000 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,574k (± 0.02%) 411,568k (± 0.01%) ~ 411,519k 411,630k p=1.000 n=6
Parse Time 3.81s (± 0.59%) 3.75s (± 0.31%) -0.07s (- 1.75%) 3.73s 3.76s p=0.005 n=6
Bind Time 1.70s (± 0.62%) 1.69s (± 0.58%) ~ 1.68s 1.71s p=0.547 n=6
Check Time 16.74s (± 0.28%) 16.76s (± 0.25%) ~ 16.70s 16.83s p=0.421 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.25s (± 0.22%) 22.19s (± 0.25%) ~ 22.11s 22.28s p=0.106 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 462,752k (± 0.09%) 462,922k (± 0.07%) ~ 462,313k 463,152k p=0.936 n=6
Parse Time 2.65s (± 0.82%) 2.56s (± 0.40%) 🟩-0.08s (- 3.15%) 2.55s 2.57s p=0.004 n=6
Bind Time 0.98s (± 0.42%) 0.97s (± 0.42%) -0.01s (- 0.68%) 0.97s 0.98s p=0.034 n=6
Check Time 15.18s (± 0.33%) 15.19s (± 0.33%) ~ 15.10s 15.24s p=0.748 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.81s (± 0.20%) 18.72s (± 0.30%) -0.08s (- 0.43%) 18.62s 18.78s p=0.013 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,299ms (± 0.49%) 2,280ms (± 0.70%) -19ms (- 0.84%) 2,253ms 2,295ms p=0.045 n=6
Req 2 - geterr 5,060ms (± 0.47%) 5,056ms (± 0.55%) ~ 5,012ms 5,085ms p=0.936 n=6
Req 3 - references 264ms (± 0.28%) 264ms (± 0.62%) ~ 261ms 265ms p=0.800 n=6
Req 4 - navto 227ms (± 1.31%) 229ms (± 0.33%) ~ 228ms 230ms p=0.081 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 89ms (± 0.84%) 86ms (± 7.46%) ~ 78ms 93ms p=0.564 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,409ms (± 0.50%) 2,375ms (± 0.37%) -34ms (- 1.40%) 2,366ms 2,391ms p=0.005 n=6
Req 2 - geterr 3,796ms (± 0.43%) 3,801ms (± 0.20%) ~ 3,794ms 3,814ms p=0.173 n=6
Req 3 - references 276ms (± 0.42%) 276ms (± 0.54%) ~ 275ms 279ms p=0.804 n=6
Req 4 - navto 227ms (± 0.18%) 227ms (± 0.18%) ~ 226ms 227ms p=1.000 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 73ms (± 7.73%) 76ms (± 8.76%) ~ 69ms 85ms p=0.372 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,204ms (± 0.84%) 5,137ms (± 0.78%) -68ms (- 1.30%) 5,086ms 5,200ms p=0.031 n=6
Req 2 - geterr 1,128ms (± 1.34%) 1,130ms (± 0.68%) ~ 1,123ms 1,143ms p=0.936 n=6
Req 3 - references 78ms (± 4.93%) 77ms (± 5.21%) ~ 74ms 83ms p=0.681 n=6
Req 4 - navto 459ms (± 0.79%) 457ms (± 0.29%) ~ 455ms 459ms p=0.246 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 841ms (± 2.00%) 835ms (± 1.56%) ~ 823ms 851ms p=0.689 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 156.99ms (± 0.17%) 156.89ms (± 0.17%) -0.10ms (- 0.07%) 155.87ms 159.53ms p=0.001 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 230.67ms (± 0.13%) 230.82ms (± 0.14%) +0.16ms (+ 0.07%) 229.42ms 235.56ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 226.45ms (± 0.13%) 226.53ms (± 0.13%) +0.08ms (+ 0.04%) 225.23ms 229.04ms p=0.009 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 225.73ms (± 0.13%) 225.97ms (± 0.13%) +0.25ms (+ 0.11%) 224.25ms 228.07ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@DanielRosenwasser
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@DanielRosenwasser
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 193,962k (± 1.02%) 193,453k (± 0.92%) ~ 192,161k 195,765k p=0.936 n=6
Parse Time 1.57s (± 0.74%) 1.52s (± 0.77%) -0.05s (- 2.98%) 1.50s 1.53s p=0.005 n=6
Bind Time 0.85s (± 0.89%) 0.85s (± 0.60%) ~ 0.85s 0.86s p=0.241 n=6
Check Time 11.23s (± 0.43%) 11.21s (± 0.34%) ~ 11.17s 11.27s p=0.519 n=6
Emit Time 3.27s (± 1.09%) 3.28s (± 0.89%) ~ 3.24s 3.33s p=0.519 n=6
Total Time 16.92s (± 0.43%) 16.86s (± 0.21%) ~ 16.83s 16.93s p=0.171 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,327k (± 0.00%) 1,218,367k (± 0.00%) ~ 1,218,289k 1,218,410k p=0.066 n=6
Parse Time 7.96s (± 0.47%) 7.88s (± 0.33%) -0.08s (- 0.96%) 7.85s 7.91s p=0.008 n=6
Bind Time 2.23s (± 0.54%) 2.23s (± 0.54%) ~ 2.22s 2.25s p=1.000 n=6
Check Time 35.79s (± 0.58%) 35.84s (± 0.21%) ~ 35.75s 35.92s p=0.198 n=6
Emit Time 16.18s (± 0.29%) 16.23s (± 0.73%) ~ 16.11s 16.43s p=0.520 n=6
Total Time 62.16s (± 0.35%) 62.18s (± 0.20%) ~ 62.04s 62.35s p=0.574 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,136,980 2,136,980 ~ ~ ~ p=1.000 n=6
Types 929,117 929,117 ~ ~ ~ p=1.000 n=6
Memory used 2,117,610k (± 0.00%) 2,117,706k (± 0.01%) ~ 2,117,502k 2,117,823k p=0.093 n=6
Parse Time 9.71s (± 0.57%) 9.54s (± 0.35%) -0.17s (- 1.73%) 9.51s 9.59s p=0.005 n=6
Bind Time 3.38s (± 0.76%) 3.37s (± 0.50%) ~ 3.35s 3.39s p=0.568 n=6
Check Time 102.45s (± 0.35%) 101.87s (± 0.99%) ~ 99.94s 102.67s p=0.261 n=6
Emit Time 0.19s (± 4.22%) 0.20s (± 4.15%) ~ 0.19s 0.21s p=0.389 n=6
Total Time 115.73s (± 0.29%) 114.98s (± 0.85%) ~ 113.12s 115.78s p=0.078 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,412 +131 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,447 +105 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,341,897k (± 0.06%) 2,342,141k (± 0.03%) ~ 2,341,329k 2,343,133k p=0.936 n=6
Parse Time 6.05s (± 0.90%) 5.93s (± 0.86%) -0.12s (- 1.96%) 5.89s 6.02s p=0.013 n=6
Bind Time 2.27s (± 0.66%) 2.25s (± 0.61%) ~ 2.24s 2.27s p=0.197 n=6
Check Time 40.19s (± 0.46%) 40.33s (± 0.44%) ~ 40.18s 40.56s p=0.128 n=6
Emit Time 3.04s (± 5.65%) 3.03s (± 2.33%) ~ 2.95s 3.15s p=0.471 n=6
Total Time 51.57s (± 0.61%) 51.55s (± 0.50%) ~ 51.33s 51.89s p=0.810 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,412 +131 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,447 +105 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,415,583k (± 0.03%) 2,415,755k (± 0.01%) ~ 2,415,449k 2,416,417k p=0.575 n=6
Parse Time 7.76s (± 0.37%) 7.59s (± 1.04%) -0.17s (- 2.19%) 7.46s 7.68s p=0.005 n=6
Bind Time 2.52s (± 0.89%) 2.52s (± 1.54%) ~ 2.47s 2.57s p=0.809 n=6
Check Time 50.21s (± 0.47%) 50.38s (± 0.29%) ~ 50.19s 50.58s p=0.173 n=6
Emit Time 3.93s (± 4.63%) 3.92s (± 1.48%) ~ 3.82s 4.00s p=0.470 n=6
Total Time 64.41s (± 0.49%) 64.42s (± 0.25%) ~ 64.15s 64.60s p=0.810 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,325 +131 (+ 0.05%) ~ ~ p=0.001 n=6
Types 104,737 104,842 +105 (+ 0.10%) ~ ~ p=0.001 n=6
Memory used 427,480k (± 0.00%) 427,683k (± 0.01%) +203k (+ 0.05%) 427,630k 427,729k p=0.005 n=6
Parse Time 3.28s (± 0.66%) 3.23s (± 0.73%) -0.05s (- 1.42%) 3.20s 3.26s p=0.013 n=6
Bind Time 1.31s (± 0.96%) 1.30s (± 1.41%) ~ 1.29s 1.33s p=0.397 n=6
Check Time 17.79s (± 0.29%) 17.82s (± 0.38%) ~ 17.76s 17.94s p=0.870 n=6
Emit Time 1.25s (± 0.71%) 1.25s (± 1.01%) ~ 1.23s 1.27s p=1.000 n=6
Total Time 23.63s (± 0.18%) 23.60s (± 0.21%) ~ 23.54s 23.68s p=0.377 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,562k (± 0.03%) 369,530k (± 0.04%) ~ 369,434k 369,814k p=0.575 n=6
Parse Time 2.78s (± 0.71%) 2.73s (± 0.46%) -0.05s (- 1.68%) 2.71s 2.74s p=0.005 n=6
Bind Time 1.58s (± 0.98%) 1.59s (± 0.94%) ~ 1.58s 1.62s p=0.084 n=6
Check Time 15.48s (± 0.34%) 15.45s (± 0.42%) ~ 15.34s 15.53s p=0.422 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.84s (± 0.39%) 19.78s (± 0.45%) ~ 19.64s 19.89s p=0.336 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,975 2,879,975 ~ ~ ~ p=1.000 n=6
Types 975,327 975,327 ~ ~ ~ p=1.000 n=6
Memory used 3,042,831k (± 0.00%) 3,042,773k (± 0.00%) ~ 3,042,693k 3,042,884k p=0.093 n=6
Parse Time 13.74s (± 0.52%) 13.54s (± 0.26%) -0.20s (- 1.44%) 13.50s 13.60s p=0.005 n=6
Bind Time 4.24s (± 1.92%) 4.22s (± 1.96%) ~ 4.17s 4.39s p=0.373 n=6
Check Time 73.51s (± 0.35%) 74.09s (± 1.81%) ~ 73.20s 76.16s p=0.810 n=6
Emit Time 24.23s (± 0.42%) 23.60s (± 4.01%) ~ 22.20s 24.27s p=0.128 n=6
Total Time 115.72s (± 0.20%) 115.46s (± 0.33%) ~ 115.07s 116.08s p=0.228 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,528k (± 0.01%) 411,534k (± 0.02%) ~ 411,456k 411,618k p=0.936 n=6
Parse Time 3.18s (± 0.54%) 3.13s (± 0.90%) -0.05s (- 1.63%) 3.10s 3.16s p=0.013 n=6
Bind Time 1.41s (± 0.53%) 1.41s (± 0.94%) ~ 1.39s 1.43s p=0.548 n=6
Check Time 14.20s (± 0.34%) 14.20s (± 0.24%) ~ 14.15s 14.25s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.79s (± 0.28%) 18.74s (± 0.25%) ~ 18.66s 18.80s p=0.172 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 462,528k (± 0.06%) 462,641k (± 0.08%) ~ 462,297k 463,096k p=0.575 n=6
Parse Time 3.18s (± 0.32%) 3.08s (± 0.45%) 🟩-0.10s (- 3.20%) 3.06s 3.10s p=0.004 n=6
Bind Time 1.17s (± 0.47%) 1.17s (± 0.35%) ~ 1.16s 1.17s p=0.282 n=6
Check Time 17.88s (± 0.45%) 17.88s (± 0.18%) ~ 17.84s 17.91s p=0.468 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.22s (± 0.39%) 22.12s (± 0.17%) ~ 22.07s 22.17s p=0.066 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,291ms (± 0.18%) 2,242ms (± 0.62%) -49ms (- 2.14%) 2,227ms 2,262ms p=0.005 n=6
Req 2 - geterr 5,057ms (± 0.58%) 5,071ms (± 0.31%) ~ 5,045ms 5,087ms p=0.471 n=6
Req 3 - references 265ms (± 0.15%) 264ms (± 0.20%) -1ms (- 0.44%) 263ms 264ms p=0.007 n=6
Req 4 - navto 228ms 228ms (± 0.37%) ~ 226ms 228ms p=0.176 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 88ms (± 0.46%) 88ms (± 0.46%) ~ 88ms 89ms p=1.000 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,419ms (± 0.30%) 2,386ms (± 1.10%) ~ 2,366ms 2,425ms p=0.065 n=6
Req 2 - geterr 3,793ms (± 0.38%) 3,792ms (± 0.34%) ~ 3,777ms 3,812ms p=0.748 n=6
Req 3 - references 276ms (± 0.27%) 277ms (± 0.48%) ~ 275ms 279ms p=0.131 n=6
Req 4 - navto 226ms (± 0.28%) 226ms (± 0.23%) ~ 226ms 227ms p=0.386 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 72ms (± 3.83%) 72ms (± 8.71%) ~ 67ms 84ms p=0.412 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,182ms (± 0.45%) 5,126ms (± 0.19%) -56ms (- 1.08%) 5,112ms 5,137ms p=0.005 n=6
Req 2 - geterr 1,120ms (± 1.44%) 1,137ms (± 1.43%) ~ 1,118ms 1,162ms p=0.148 n=6
Req 3 - references 77ms (± 2.95%) 79ms (± 4.88%) ~ 75ms 82ms p=0.673 n=6
Req 4 - navto 459ms (± 0.70%) 459ms (± 1.26%) ~ 456ms 471ms p=0.505 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 833ms (± 1.53%) 841ms (± 1.18%) ~ 830ms 851ms p=0.469 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 157.08ms (± 0.17%) 157.15ms (± 0.17%) +0.08ms (+ 0.05%) 156.02ms 160.72ms p=0.004 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 230.51ms (± 0.14%) 230.61ms (± 0.19%) +0.10ms (+ 0.04%) 229.34ms 246.30ms p=0.008 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 226.09ms (± 0.13%) 226.31ms (± 0.17%) +0.22ms (+ 0.10%) 225.05ms 236.63ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 225.82ms (± 0.16%) 225.89ms (± 0.17%) +0.07ms (+ 0.03%) 224.42ms 235.68ms p=0.034 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 193,433k (± 0.93%) 194,534k (± 0.94%) ~ 192,152k 195,805k p=0.810 n=6
Parse Time 1.58s (± 1.09%) 1.50s (± 0.73%) 🟩-0.08s (- 4.96%) 1.49s 1.52s p=0.005 n=6
Bind Time 0.85s (± 1.21%) 0.86s (± 1.36%) ~ 0.84s 0.87s p=0.456 n=6
Check Time 11.17s (± 0.37%) 11.22s (± 0.45%) ~ 11.16s 11.28s p=0.196 n=6
Emit Time 3.29s (± 0.85%) 3.28s (± 0.45%) ~ 3.26s 3.30s p=0.935 n=6
Total Time 16.89s (± 0.17%) 16.86s (± 0.26%) ~ 16.80s 16.91s p=0.295 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,400k (± 0.00%) 1,218,372k (± 0.00%) ~ 1,218,309k 1,218,440k p=0.230 n=6
Parse Time 6.67s (± 0.92%) 6.57s (± 0.54%) -0.10s (- 1.47%) 6.50s 6.60s p=0.016 n=6
Bind Time 1.87s (± 0.44%) 1.87s (± 0.22%) ~ 1.87s 1.88s p=0.206 n=6
Check Time 30.60s (± 0.36%) 30.62s (± 0.67%) ~ 30.43s 31.02s p=1.000 n=6
Emit Time 13.58s (± 0.40%) 13.58s (± 0.50%) ~ 13.51s 13.69s p=1.000 n=6
Total Time 52.72s (± 0.38%) 52.65s (± 0.51%) ~ 52.42s 53.15s p=0.470 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,136,980 2,136,980 ~ ~ ~ p=1.000 n=6
Types 929,117 929,117 ~ ~ ~ p=1.000 n=6
Memory used 2,117,784k (± 0.01%) 2,117,749k (± 0.00%) ~ 2,117,569k 2,117,854k p=0.810 n=6
Parse Time 7.88s (± 0.25%) 7.74s (± 0.28%) -0.14s (- 1.76%) 7.72s 7.78s p=0.005 n=6
Bind Time 2.73s (± 0.43%) 2.74s (± 0.68%) ~ 2.71s 2.76s p=0.101 n=6
Check Time 83.32s (± 0.38%) 83.25s (± 0.24%) ~ 82.99s 83.49s p=0.471 n=6
Emit Time 0.15s (± 3.53%) 0.16s (± 3.29%) ~ 0.15s 0.16s p=0.640 n=6
Total Time 94.08s (± 0.35%) 93.89s (± 0.20%) ~ 93.66s 94.12s p=0.336 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,412 +131 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,448 +106 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,342,656k (± 0.02%) 2,342,024k (± 0.04%) ~ 2,340,664k 2,343,136k p=0.378 n=6
Parse Time 6.03s (± 0.77%) 5.98s (± 0.93%) ~ 5.90s 6.05s p=0.297 n=6
Bind Time 2.26s (± 0.91%) 2.26s (± 0.54%) ~ 2.24s 2.27s p=0.625 n=6
Check Time 40.29s (± 0.56%) 40.33s (± 0.13%) ~ 40.24s 40.38s p=0.936 n=6
Emit Time 2.95s (± 1.41%) 3.04s (± 2.52%) +0.09s (+ 2.99%) 2.96s 3.17s p=0.045 n=6
Total Time 51.54s (± 0.54%) 51.63s (± 0.17%) ~ 51.51s 51.72s p=0.936 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,412 +131 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,448 +106 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,415,148k (± 0.05%) 2,415,902k (± 0.04%) ~ 2,414,963k 2,417,577k p=0.128 n=6
Parse Time 7.74s (± 0.76%) 7.63s (± 1.08%) ~ 7.56s 7.73s p=0.066 n=6
Bind Time 2.52s (± 1.29%) 2.53s (± 0.65%) ~ 2.51s 2.56s p=0.574 n=6
Check Time 50.30s (± 0.52%) 50.54s (± 0.79%) ~ 50.18s 51.24s p=0.575 n=6
Emit Time 3.92s (± 3.49%) 3.87s (± 1.94%) ~ 3.73s 3.94s p=0.689 n=6
Total Time 64.49s (± 0.56%) 64.59s (± 0.79%) ~ 64.12s 65.49s p=1.000 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,325 +131 (+ 0.05%) ~ ~ p=0.001 n=6
Types 104,737 104,843 +106 (+ 0.10%) ~ ~ p=0.001 n=6
Memory used 427,524k (± 0.03%) 427,685k (± 0.01%) ~ 427,642k 427,717k p=0.065 n=6
Parse Time 3.30s (± 0.56%) 3.22s (± 0.47%) -0.07s (- 2.17%) 3.21s 3.25s p=0.005 n=6
Bind Time 1.33s (± 0.61%) 1.31s (± 1.79%) ~ 1.28s 1.34s p=0.171 n=6
Check Time 17.89s (± 0.30%) 17.84s (± 0.35%) ~ 17.77s 17.91s p=0.230 n=6
Emit Time 1.26s (± 1.30%) 1.25s (± 1.20%) ~ 1.24s 1.28s p=0.287 n=6
Total Time 23.77s (± 0.21%) 23.62s (± 0.27%) -0.15s (- 0.63%) 23.51s 23.67s p=0.005 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,547k (± 0.05%) 369,476k (± 0.02%) ~ 369,393k 369,583k p=0.810 n=6
Parse Time 2.75s (± 1.00%) 2.73s (± 1.06%) ~ 2.69s 2.76s p=0.225 n=6
Bind Time 1.57s (± 0.48%) 1.58s (± 0.67%) ~ 1.56s 1.59s p=0.611 n=6
Check Time 15.44s (± 0.33%) 15.45s (± 0.22%) ~ 15.42s 15.51s p=0.871 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.76s (± 0.31%) 19.74s (± 0.22%) ~ 19.70s 19.81s p=0.686 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,880,017 2,880,017 ~ ~ ~ p=1.000 n=6
Types 975,336 975,336 ~ ~ ~ p=1.000 n=6
Memory used 3,042,842k (± 0.00%) 3,042,791k (± 0.00%) ~ 3,042,683k 3,042,922k p=0.229 n=6
Parse Time 11.45s (± 0.38%) 11.32s (± 0.51%) -0.13s (- 1.12%) 11.27s 11.43s p=0.010 n=6
Bind Time 3.50s (± 0.33%) 3.51s (± 0.43%) ~ 3.49s 3.53s p=0.618 n=6
Check Time 64.76s (± 1.98%) 65.23s (± 1.96%) ~ 63.50s 66.91s p=0.575 n=6
Emit Time 19.25s (± 6.63%) 18.70s (± 6.37%) ~ 17.27s 20.15s p=0.471 n=6
Total Time 98.97s (± 0.40%) 98.75s (± 0.22%) ~ 98.48s 99.02s p=0.173 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,571k (± 0.02%) 411,585k (± 0.01%) ~ 411,507k 411,672k p=0.575 n=6
Parse Time 3.83s (± 0.64%) 3.75s (± 0.62%) -0.08s (- 1.96%) 3.72s 3.78s p=0.005 n=6
Bind Time 1.70s (± 0.71%) 1.69s (± 0.32%) ~ 1.69s 1.70s p=0.865 n=6
Check Time 16.78s (± 0.29%) 16.75s (± 0.30%) ~ 16.67s 16.80s p=0.469 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.31s (± 0.26%) 22.20s (± 0.30%) -0.11s (- 0.48%) 22.11s 22.26s p=0.016 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 462,791k (± 0.08%) 462,975k (± 0.02%) ~ 462,906k 463,149k p=0.423 n=6
Parse Time 2.65s (± 0.31%) 2.56s (± 0.59%) 🟩-0.08s (- 3.15%) 2.54s 2.58s p=0.004 n=6
Bind Time 0.98s (± 0.53%) 0.98s (± 1.12%) ~ 0.97s 1.00s p=0.784 n=6
Check Time 15.19s (± 0.16%) 15.24s (± 0.48%) ~ 15.13s 15.30s p=0.335 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.82s (± 0.15%) 18.78s (± 0.41%) ~ 18.67s 18.85s p=0.688 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,301ms (± 0.29%) 2,242ms (± 0.71%) -59ms (- 2.56%) 2,221ms 2,259ms p=0.005 n=6
Req 2 - geterr 5,060ms (± 0.37%) 5,075ms (± 0.60%) ~ 5,023ms 5,107ms p=0.335 n=6
Req 3 - references 264ms (± 0.15%) 266ms (± 1.18%) ~ 264ms 272ms p=0.248 n=6
Req 4 - navto 227ms (± 1.48%) 226ms (± 1.40%) ~ 222ms 229ms p=0.607 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 89ms (± 2.30%) 89ms (± 6.61%) ~ 78ms 94ms p=0.531 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,419ms (± 0.55%) 2,360ms (± 0.65%) -60ms (- 2.46%) 2,345ms 2,384ms p=0.005 n=6
Req 2 - geterr 3,797ms (± 0.41%) 3,802ms (± 0.21%) ~ 3,792ms 3,815ms p=0.378 n=6
Req 3 - references 277ms (± 0.30%) 277ms (± 0.15%) ~ 276ms 277ms p=0.285 n=6
Req 4 - navto 226ms (± 0.33%) 226ms (± 0.53%) ~ 224ms 227ms p=0.485 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 73ms (± 8.26%) 73ms (± 8.07%) ~ 69ms 84ms p=1.000 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 6,217ms (± 0.43%) 6,249ms (± 4.61%) ~ 6,101ms 6,836ms p=0.066 n=6
Req 2 - geterr 1,577ms (±10.42%) 1,582ms (±11.03%) ~ 1,353ms 1,710ms p=0.936 n=6
Req 3 - references 118ms (± 4.38%) 112ms (± 7.33%) ~ 99ms 122ms p=0.213 n=6
Req 4 - navto 606ms (± 2.37%) 599ms (± 2.44%) ~ 580ms 620ms p=0.378 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 1,268ms (± 2.05%) 1,255ms (± 1.67%) ~ 1,234ms 1,284ms p=0.377 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 188.04ms (± 0.17%) 188.07ms (± 0.17%) ~ 186.53ms 192.34ms p=0.188 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 230.59ms (± 0.12%) 230.73ms (± 0.12%) +0.14ms (+ 0.06%) 229.43ms 233.17ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 225.92ms (± 0.13%) 226.09ms (± 0.15%) +0.16ms (+ 0.07%) 224.82ms 232.69ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 226.00ms (± 0.13%) 226.46ms (± 0.14%) +0.46ms (+ 0.20%) 224.95ms 231.90ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@DanielRosenwasser
Copy link
Member Author

I'm actually not sure if that's much better or not, it could just be a better run.

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
pack this ✅ Started ✅ Results

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Hey @DanielRosenwasser, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/162560/artifacts?artifactName=tgz&fileId=4AA9667F9E1F7B8C6212CF52192D9CDF3B283A368A7D0AD742ABB748FCC2402602&fileName=/typescript-5.6.0-insiders.20240702.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/pr-build@5.6.0-pr-59106-14".;

@DanielRosenwasser DanielRosenwasser changed the title Experiments with token scanning Optimize scan with a lookup table to reduce branching Jul 2, 2024
@DanielRosenwasser
Copy link
Member Author

DanielRosenwasser commented Jul 2, 2024

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
pack this ✅ Started ✅ Results

@DanielRosenwasser
Copy link
Member Author

@typescript-bot test top400
@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
test top400 ✅ Started ✅ Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 2, 2024

Hey @DanielRosenwasser, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/162561/artifacts?artifactName=tgz&fileId=E35843EEA10E387D43B75BB02FEF6B9CF5F7BA42782FB52C26B1840A976DBB6102&fileName=/typescript-5.6.0-insiders.20240702.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/pr-build@5.6.0-pr-59106-19".;

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 193,427k (± 0.92%) 192,159k (± 0.02%) ~ 192,113k 192,244k p=0.298 n=6
Parse Time 1.31s (± 0.75%) 1.24s (± 0.83%) 🟩-0.07s (- 5.21%) 1.23s 1.26s p=0.004 n=6
Bind Time 0.71s 0.71s ~ ~ ~ p=1.000 n=6
Check Time 9.43s (± 0.61%) 9.45s (± 0.26%) ~ 9.40s 9.47s p=0.519 n=6
Emit Time 2.75s (± 0.88%) 2.73s (± 1.37%) ~ 2.66s 2.76s p=0.255 n=6
Total Time 14.20s (± 0.45%) 14.13s (± 0.34%) ~ 14.05s 14.19s p=0.064 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,317k (± 0.00%) 1,218,356k (± 0.00%) ~ 1,218,316k 1,218,402k p=0.092 n=6
Parse Time 7.99s (± 0.52%) 7.88s (± 0.69%) -0.11s (- 1.34%) 7.81s 7.95s p=0.006 n=6
Bind Time 2.22s (± 0.34%) 2.22s (± 0.55%) ~ 2.20s 2.23s p=0.503 n=6
Check Time 35.89s (± 0.71%) 35.81s (± 0.49%) ~ 35.55s 36.04s p=0.688 n=6
Emit Time 16.19s (± 0.34%) 16.19s (± 0.36%) ~ 16.08s 16.24s p=0.809 n=6
Total Time 62.29s (± 0.45%) 62.10s (± 0.28%) ~ 61.88s 62.36s p=0.230 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,136,939 2,136,939 ~ ~ ~ p=1.000 n=6
Types 929,123 929,123 ~ ~ ~ p=1.000 n=6
Memory used 2,117,686k (± 0.01%) 2,117,782k (± 0.00%) ~ 2,117,617k 2,117,884k p=0.230 n=6
Parse Time 9.70s (± 0.34%) 9.54s (± 0.38%) -0.16s (- 1.60%) 9.50s 9.59s p=0.005 n=6
Bind Time 3.38s (± 0.51%) 3.40s (± 0.37%) +0.02s (+ 0.64%) 3.39s 3.42s p=0.035 n=6
Check Time 102.88s (± 0.57%) 101.50s (± 1.84%) ~ 99.72s 103.42s p=0.378 n=6
Emit Time 0.20s (± 4.15%) 0.20s (± 4.47%) ~ 0.19s 0.21s p=0.550 n=6
Total Time 116.15s (± 0.51%) 114.64s (± 1.60%) ~ 112.87s 116.51s p=0.298 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,412 +131 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,448 +106 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,341,955k (± 0.05%) 2,341,458k (± 0.03%) ~ 2,340,732k 2,342,696k p=0.230 n=6
Parse Time 6.01s (± 0.91%) 5.93s (± 0.81%) ~ 5.89s 5.99s p=0.061 n=6
Bind Time 2.26s (± 0.93%) 2.25s (± 0.76%) ~ 2.23s 2.27s p=0.809 n=6
Check Time 40.21s (± 0.31%) 40.29s (± 0.78%) ~ 40.08s 40.92s p=0.936 n=6
Emit Time 3.06s (± 4.86%) 3.06s (± 2.89%) ~ 2.88s 3.11s p=0.574 n=6
Total Time 51.57s (± 0.44%) 51.56s (± 0.67%) ~ 51.22s 52.21s p=0.689 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,412 +131 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,448 +106 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,415,657k (± 0.02%) 2,415,756k (± 0.04%) ~ 2,414,302k 2,417,226k p=0.810 n=6
Parse Time 7.74s (± 0.38%) 7.60s (± 0.79%) -0.14s (- 1.77%) 7.54s 7.68s p=0.005 n=6
Bind Time 2.51s (± 0.70%) 2.51s (± 0.91%) ~ 2.47s 2.53s p=0.628 n=6
Check Time 50.15s (± 0.31%) 50.46s (± 0.63%) +0.31s (+ 0.62%) 50.09s 51.05s p=0.045 n=6
Emit Time 3.82s (± 2.13%) 3.90s (± 2.95%) ~ 3.73s 4.08s p=0.149 n=6
Total Time 64.23s (± 0.21%) 64.48s (± 0.63%) ~ 63.95s 65.17s p=0.109 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,325 +131 (+ 0.05%) ~ ~ p=0.001 n=6
Types 104,737 104,843 +106 (+ 0.10%) ~ ~ p=0.001 n=6
Memory used 427,519k (± 0.00%) 427,684k (± 0.01%) +164k (+ 0.04%) 427,650k 427,708k p=0.005 n=6
Parse Time 3.31s (± 1.05%) 3.21s (± 0.65%) 🟩-0.10s (- 3.02%) 3.19s 3.24s p=0.005 n=6
Bind Time 1.32s (± 0.31%) 1.29s (± 1.16%) -0.03s (- 2.14%) 1.28s 1.32s p=0.010 n=6
Check Time 17.82s (± 0.39%) 17.84s (± 0.39%) ~ 17.73s 17.91s p=0.810 n=6
Emit Time 1.25s (± 1.17%) 1.25s (± 0.94%) ~ 1.24s 1.27s p=0.680 n=6
Total Time 23.71s (± 0.33%) 23.60s (± 0.33%) -0.11s (- 0.46%) 23.48s 23.68s p=0.045 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,504k (± 0.02%) 369,545k (± 0.03%) ~ 369,396k 369,689k p=0.575 n=6
Parse Time 2.77s (± 1.64%) 2.71s (± 0.77%) -0.06s (- 2.11%) 2.67s 2.73s p=0.029 n=6
Bind Time 1.58s (± 0.96%) 1.58s (± 0.33%) ~ 1.57s 1.58s p=0.351 n=6
Check Time 15.48s (± 0.24%) 15.48s (± 0.25%) ~ 15.41s 15.51s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.83s (± 0.22%) 19.77s (± 0.28%) -0.07s (- 0.34%) 19.66s 19.82s p=0.037 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,780 2,879,780 ~ ~ ~ p=1.000 n=6
Types 975,260 975,260 ~ ~ ~ p=1.000 n=6
Memory used 3,042,725k (± 0.00%) 3,042,710k (± 0.00%) ~ 3,042,618k 3,042,860k p=0.810 n=6
Parse Time 13.69s (± 0.56%) 13.54s (± 0.36%) -0.16s (- 1.16%) 13.47s 13.62s p=0.008 n=6
Bind Time 4.19s (± 0.25%) 4.26s (± 2.08%) +0.07s (+ 1.71%) 4.18s 4.41s p=0.037 n=6
Check Time 73.60s (± 0.73%) 74.17s (± 1.36%) ~ 73.00s 75.93s p=0.298 n=6
Emit Time 24.11s (± 2.28%) 23.59s (± 4.16%) ~ 21.91s 24.44s p=0.687 n=6
Total Time 115.59s (± 0.33%) 115.56s (± 0.24%) ~ 115.18s 115.86s p=0.810 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,550k (± 0.02%) 411,617k (± 0.02%) ~ 411,549k 411,731k p=0.093 n=6
Parse Time 3.18s (± 0.37%) 3.12s (± 0.43%) -0.06s (- 1.78%) 3.10s 3.14s p=0.004 n=6
Bind Time 1.41s (± 0.37%) 1.41s (± 0.59%) ~ 1.39s 1.41s p=0.070 n=6
Check Time 14.21s (± 0.25%) 14.21s (± 0.26%) ~ 14.14s 14.24s p=0.936 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.81s (± 0.22%) 18.73s (± 0.26%) -0.08s (- 0.40%) 18.64s 18.78s p=0.009 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 463,001k (± 0.07%) 463,009k (± 0.05%) ~ 462,622k 463,281k p=0.810 n=6
Parse Time 2.66s (± 0.39%) 2.56s (± 0.67%) 🟩-0.09s (- 3.52%) 2.54s 2.58s p=0.005 n=6
Bind Time 0.98s (± 0.77%) 0.97s (± 0.86%) ~ 0.97s 0.99s p=0.172 n=6
Check Time 15.19s (± 0.26%) 15.25s (± 0.37%) ~ 15.20s 15.34s p=0.126 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.82s (± 0.23%) 18.79s (± 0.34%) ~ 18.74s 18.91s p=0.334 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,294ms (± 0.34%) 2,235ms (± 0.76%) -60ms (- 2.59%) 2,223ms 2,265ms p=0.005 n=6
Req 2 - geterr 5,061ms (± 0.50%) 5,077ms (± 0.46%) ~ 5,050ms 5,116ms p=0.298 n=6
Req 3 - references 265ms (± 1.25%) 264ms (± 0.28%) ~ 263ms 265ms p=0.931 n=6
Req 4 - navto 227ms (± 0.53%) 227ms (± 1.18%) ~ 222ms 230ms p=0.931 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 87ms (± 3.88%) 88ms (± 6.22%) ~ 78ms 93ms p=0.391 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,431ms (± 0.56%) 2,364ms (± 0.24%) -67ms (- 2.77%) 2,357ms 2,372ms p=0.005 n=6
Req 2 - geterr 3,798ms (± 0.18%) 3,791ms (± 0.42%) ~ 3,774ms 3,816ms p=0.378 n=6
Req 3 - references 277ms (± 0.29%) 276ms (± 0.46%) ~ 275ms 278ms p=0.082 n=6
Req 4 - navto 227ms (± 0.28%) 229ms (± 2.83%) ~ 225ms 242ms p=0.438 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 79ms (± 9.11%) 78ms (± 8.10%) ~ 69ms 85ms p=0.677 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 6,342ms (± 5.02%) 6,109ms (± 0.55%) 🟩-233ms (- 3.67%) 6,056ms 6,160ms p=0.005 n=6
Req 2 - geterr 1,597ms (±10.84%) 1,585ms (±10.42%) ~ 1,364ms 1,721ms p=0.689 n=6
Req 3 - references 113ms (± 4.02%) 114ms (± 4.48%) ~ 110ms 121ms p=0.561 n=6
Req 4 - navto 597ms (± 2.78%) 596ms (± 1.51%) ~ 586ms 609ms p=0.810 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 1,251ms (± 1.60%) 1,254ms (± 1.32%) ~ 1,234ms 1,274ms p=0.873 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 157.17ms (± 0.20%) 157.21ms (± 0.16%) +0.03ms (+ 0.02%) 156.22ms 160.96ms p=0.047 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 230.32ms (± 0.21%) 230.43ms (± 0.14%) +0.11ms (+ 0.05%) 229.08ms 236.21ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 226.63ms (± 0.14%) 226.87ms (± 0.14%) +0.24ms (+ 0.11%) 225.59ms 231.38ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 275.26ms (± 0.28%) 276.27ms (± 0.73%) +1.01ms (+ 0.37%) 269.29ms 308.09ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@DanielRosenwasser DanielRosenwasser marked this pull request as ready for review July 2, 2024 20:30
@typescript-bot
Copy link
Collaborator

@DanielRosenwasser Here are the results of running the top 400 repos with tsc comparing main and refs/pull/59106/merge:

Everything looks good!

@DanielRosenwasser
Copy link
Member Author

@dragomirtitian maybe this balances out the parse-time hit from #58928. 😄

@DanielRosenwasser
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 3, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 192,850k (± 0.77%) 193,503k (± 0.93%) ~ 192,149k 195,866k p=0.261 n=6
Parse Time 1.58s (± 0.48%) 1.50s (± 1.09%) 🟩-0.08s (- 4.85%) 1.49s 1.53s p=0.005 n=6
Bind Time 0.86s (± 1.41%) 0.85s (± 1.16%) ~ 0.84s 0.86s p=0.246 n=6
Check Time 11.18s (± 0.60%) 11.21s (± 0.17%) ~ 11.18s 11.23s p=0.520 n=6
Emit Time 3.28s (± 0.76%) 3.29s (± 0.95%) ~ 3.25s 3.32s p=0.808 n=6
Total Time 16.90s (± 0.19%) 16.85s (± 0.19%) ~ 16.82s 16.89s p=0.065 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,371k (± 0.00%) 1,218,388k (± 0.00%) ~ 1,218,311k 1,218,446k p=0.575 n=6
Parse Time 7.96s (± 0.64%) 7.90s (± 0.49%) -0.07s (- 0.82%) 7.84s 7.93s p=0.030 n=6
Bind Time 2.24s (± 1.08%) 2.22s (± 0.66%) ~ 2.21s 2.25s p=0.145 n=6
Check Time 35.73s (± 0.40%) 35.87s (± 0.43%) ~ 35.65s 36.13s p=0.171 n=6
Emit Time 16.11s (± 0.32%) 16.18s (± 0.22%) ~ 16.14s 16.23s p=0.052 n=6
Total Time 62.04s (± 0.33%) 62.17s (± 0.27%) ~ 61.90s 62.43s p=0.229 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,154,949 2,154,949 ~ ~ ~ p=1.000 n=6
Types 931,473 931,473 ~ ~ ~ p=1.000 n=6
Memory used 2,125,930k (± 0.00%) 2,126,089k (± 0.00%) +159k (+ 0.01%) 2,126,001k 2,126,145k p=0.005 n=6
Parse Time 9.77s (± 0.34%) 9.56s (± 0.33%) -0.21s (- 2.20%) 9.52s 9.61s p=0.005 n=6
Bind Time 3.39s (± 0.37%) 3.38s (± 0.51%) ~ 3.36s 3.40s p=0.415 n=6
Check Time 103.96s (± 0.62%) 103.99s (± 0.33%) ~ 103.33s 104.29s p=1.000 n=6
Emit Time 0.19s (± 2.13%) 0.20s 🔻+0.01s (+ 4.35%) ~ ~ p=0.007 n=6
Total Time 117.31s (± 0.55%) 117.12s (± 0.31%) ~ 116.41s 117.43s p=0.297 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,413 +132 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,448 +106 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,342,013k (± 0.03%) 2,341,774k (± 0.04%) ~ 2,340,479k 2,343,198k p=0.471 n=6
Parse Time 6.03s (± 1.12%) 5.99s (± 1.09%) ~ 5.91s 6.09s p=0.336 n=6
Bind Time 2.26s (± 0.87%) 2.27s (± 1.02%) ~ 2.23s 2.29s p=0.686 n=6
Check Time 40.22s (± 0.60%) 40.42s (± 0.49%) ~ 40.11s 40.63s p=0.128 n=6
Emit Time 3.05s (± 1.85%) 3.11s (± 6.04%) ~ 2.93s 3.47s p=0.873 n=6
Total Time 51.56s (± 0.52%) 51.79s (± 0.47%) ~ 51.37s 52.11s p=0.128 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,281 1,225,413 +132 (+ 0.01%) ~ ~ p=0.001 n=6
Types 261,342 261,448 +106 (+ 0.04%) ~ ~ p=0.001 n=6
Memory used 2,414,808k (± 0.05%) 2,415,265k (± 0.03%) ~ 2,414,227k 2,416,053k p=0.689 n=6
Parse Time 7.79s (± 0.85%) 7.64s (± 0.36%) -0.15s (- 1.95%) 7.61s 7.69s p=0.008 n=6
Bind Time 2.51s (± 0.48%) 2.52s (± 1.03%) ~ 2.48s 2.55s p=0.518 n=6
Check Time 50.32s (± 0.43%) 50.45s (± 0.31%) ~ 50.17s 50.62s p=0.298 n=6
Emit Time 3.94s (± 2.65%) 3.99s (± 4.57%) ~ 3.89s 4.36s p=0.298 n=6
Total Time 64.58s (± 0.34%) 64.61s (± 0.40%) ~ 64.20s 64.99s p=0.936 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,326 +132 (+ 0.05%) ~ ~ p=0.001 n=6
Types 104,737 104,843 +106 (+ 0.10%) ~ ~ p=0.001 n=6
Memory used 427,488k (± 0.01%) 427,701k (± 0.01%) +213k (+ 0.05%) 427,650k 427,748k p=0.005 n=6
Parse Time 3.28s (± 0.60%) 3.26s (± 0.19%) -0.02s (- 0.71%) 3.25s 3.27s p=0.048 n=6
Bind Time 1.31s (± 1.12%) 1.32s (± 1.82%) ~ 1.29s 1.35s p=0.514 n=6
Check Time 17.76s (± 0.29%) 17.77s (± 0.35%) ~ 17.69s 17.87s p=1.000 n=6
Emit Time 1.25s (± 1.31%) 1.25s (± 1.20%) ~ 1.24s 1.27s p=0.458 n=6
Total Time 23.60s (± 0.30%) 23.60s (± 0.37%) ~ 23.51s 23.75s p=0.810 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,475k (± 0.04%) 369,514k (± 0.04%) ~ 369,393k 369,750k p=0.298 n=6
Parse Time 2.76s (± 0.88%) 2.74s (± 1.41%) ~ 2.68s 2.79s p=0.376 n=6
Bind Time 1.59s (± 0.94%) 1.59s (± 1.30%) ~ 1.58s 1.62s p=0.666 n=6
Check Time 15.46s (± 0.26%) 15.49s (± 0.36%) ~ 15.42s 15.56s p=0.469 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.81s (± 0.18%) 19.83s (± 0.30%) ~ 19.78s 19.93s p=0.747 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,791 2,879,791 ~ ~ ~ p=1.000 n=6
Types 975,122 975,122 ~ ~ ~ p=1.000 n=6
Memory used 3,043,017k (± 0.00%) 3,042,990k (± 0.00%) ~ 3,042,960k 3,043,023k p=0.298 n=6
Parse Time 13.71s (± 0.75%) 13.56s (± 0.41%) -0.14s (- 1.06%) 13.47s 13.62s p=0.016 n=6
Bind Time 4.30s (± 2.90%) 4.19s (± 0.30%) ~ 4.18s 4.21s p=0.369 n=6
Check Time 73.88s (± 2.34%) 74.11s (± 2.25%) ~ 73.21s 77.49s p=0.092 n=6
Emit Time 23.67s (± 6.92%) 23.66s (± 6.60%) ~ 20.50s 24.48s p=0.810 n=6
Total Time 115.55s (± 0.26%) 115.52s (± 0.34%) ~ 114.87s 115.91s p=1.000 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,562k (± 0.02%) 411,552k (± 0.01%) ~ 411,496k 411,643k p=0.936 n=6
Parse Time 3.18s (± 0.67%) 3.14s (± 0.20%) -0.04s (- 1.21%) 3.13s 3.15s p=0.004 n=6
Bind Time 1.41s 1.42s (± 0.39%) ~ 1.41s 1.42s p=0.071 n=6
Check Time 14.20s (± 0.31%) 14.16s (± 0.31%) ~ 14.09s 14.22s p=0.127 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.79s (± 0.27%) 18.71s (± 0.26%) -0.08s (- 0.41%) 18.64s 18.78s p=0.037 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 462,855k (± 0.08%) 462,947k (± 0.04%) ~ 462,673k 463,208k p=0.810 n=6
Parse Time 3.93s (± 0.57%) 3.83s (± 0.26%) -0.10s (- 2.58%) 3.82s 3.84s p=0.004 n=6
Bind Time 1.46s (± 1.28%) 1.46s (± 0.71%) ~ 1.44s 1.47s p=1.000 n=6
Check Time 22.15s (± 0.28%) 22.17s (± 0.28%) ~ 22.09s 22.27s p=0.463 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 27.54s (± 0.24%) 27.47s (± 0.26%) ~ 27.37s 27.57s p=0.127 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,295ms (± 0.54%) 2,244ms (± 0.54%) -50ms (- 2.19%) 2,230ms 2,259ms p=0.005 n=6
Req 2 - geterr 5,068ms (± 0.37%) 5,072ms (± 0.32%) ~ 5,056ms 5,100ms p=0.689 n=6
Req 3 - references 263ms (± 0.44%) 264ms (± 0.50%) ~ 262ms 266ms p=0.216 n=6
Req 4 - navto 229ms (± 0.81%) 227ms (± 0.72%) ~ 224ms 228ms p=0.177 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 84ms (± 8.24%) 87ms (± 5.91%) ~ 78ms 94ms p=0.432 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,414ms (± 0.41%) 2,387ms (± 0.59%) -27ms (- 1.13%) 2,370ms 2,406ms p=0.008 n=6
Req 2 - geterr 3,794ms (± 0.37%) 3,795ms (± 0.33%) ~ 3,780ms 3,809ms p=1.000 n=6
Req 3 - references 277ms (± 0.29%) 277ms (± 0.35%) ~ 275ms 278ms p=0.604 n=6
Req 4 - navto 226ms (± 0.33%) 227ms (± 0.18%) ~ 226ms 227ms p=0.100 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 75ms (± 7.21%) 77ms (±10.31%) ~ 69ms 85ms p=1.000 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,182ms (± 0.20%) 5,116ms (± 0.28%) -66ms (- 1.28%) 5,100ms 5,135ms p=0.005 n=6
Req 2 - geterr 1,129ms (± 0.88%) 1,123ms (± 0.18%) ~ 1,120ms 1,125ms p=0.196 n=6
Req 3 - references 77ms (± 4.67%) 75ms (± 0.54%) ~ 74ms 75ms p=0.114 n=6
Req 4 - navto 460ms (± 1.04%) 457ms (± 0.11%) ~ 457ms 458ms p=1.000 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 834ms (± 1.52%) 832ms (± 1.05%) ~ 823ms 848ms p=0.748 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 157.43ms (± 0.16%) 157.38ms (± 0.17%) ~ 156.21ms 159.84ms p=0.074 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 230.50ms (± 0.15%) 230.58ms (± 0.15%) +0.08ms (+ 0.04%) 229.33ms 236.14ms p=0.025 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 226.02ms (± 0.15%) 226.16ms (± 0.17%) +0.14ms (+ 0.06%) 224.63ms 232.34ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 225.25ms (± 0.15%) 225.33ms (± 0.13%) +0.08ms (+ 0.04%) 224.05ms 227.31ms p=0.003 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

}

const tokenCategoryLookup: TokenCategory[] = [];
const tokenCategoryLookupUncommon = new Map<CharacterCodes, TokenCategory>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does map lookup performance and memory usage compare to branching ifs with comparison operators?

[CharacterCodes.replacementCharacter, TokenCategory.RecognizedMisc],
]
) {
if (key < tokenCategoryLookup.length) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do this in a branch? The uncommon lookups could just be passed to the Map constructor.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you be more specific? In the first two commits, I split this out from a map to an array and map to avoid the map lookup.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that, since we know ahead of time which characters belong in the Map, we can just construct the map with those character codes rather than looping and comparing. It's not that important, and if it makes more sense to have them visually grouped together, that's perfectly fine.

}

if (!(tokenCategory & TokenCategory.RecognizedMisc)) Debug.fail(`Unhandled token category ${tokenCategory}`);
switch (ch) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the switch here is optimized in any way or if it is doing case-by-case comparisons against each character code. If the latter, maybe it would be worthwhile to analyze what that most common character codes in JS are and change the order of the cases so that the more frequent cases are earlier in the list. I expect the perf improvement here actually comes from handling digit and identifier characters earlier than the switch, since those are the most frequent characters in a given file. If that's the case, some of the complexity and memory overhead could be reduced by just optimizing for those two cases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is typically case-by-case comparisons. You're absolutely right that doing the checks earlier on for common classes; however, I think that checking for all the identifier starts isn't practical (or requires the same sort of fast-path/slow-path separation where we check for ranges), so that's why the token categorization is helpful here. By dismissing all the other types of categories, we can immediately launch into the fast path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming CharacterCodes is a const enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the majority of identifier/keyword characters are ASCII characters, then it's fairly trivial to check for those major cases without an array or Map using a simple comparison:

if (
    (ch | 32) >= CharacterCodes.a && (ch | 32) <= CharacterCodes.z ||
    ch === CharacterCodes._ ||
    ch === CharacterCodes.$
) { /* [A-Za-z_$] */ }

If the array/map lookups are faster than this for the identifier case, then by all means keep it as is. I only point this out as you explicitly avoid hitting the table for line terminators and spaces. If that was an important optimization, then it might make sense to do the same kind of optimization for the most frequent (and easy to optimize for) characters in a source file, e.g., ASCII letters.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming CharacterCodes is a const enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.

We've generally found that not to be the case, which is why we've had to implement our own jump tables for functions like forEachChild. I'm not sure what heuristics V8 uses for switch optimization, if any.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the majority of identifier/keyword characters are ASCII characters, then it's fairly trivial to check for those major cases without an array or Map using a simple comparison:

Fair, and this is what @ahejlsberg also suggested. The same way whitespace is prioritized, ASCII range is also very likely and worth prioritizing as a quicker up-front case instead of hitting the lookup table. It would mean there's some code duplication but maybe it's fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming CharacterCodes is a const enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.

We've generally found that not to be the case, which is why we've had to implement our own jump tables for functions like forEachChild. I'm not sure what heuristics V8 uses for switch optimization, if any.

Especially in the case where these character codes are so sparse - I am very doubtful that V8 would do such an optimization, and at best I'd guess we'd end up with something like a binary search.


if (tokenCategory & TokenCategory.SimpleToken) {
pos++;
return token = tokenCategory & TokenCategory.SimpleTokenMask;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment here would be good for future readers as it isn't obvious at first glance how a TokenCategory becomes a token.

Comment on lines +2067 to +2101
if (ch === CharacterCodes._0) {
if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.X || charCodeUnchecked(pos + 1) === CharacterCodes.x)) {
pos += 2;
tokenValue = scanMinimumNumberOfHexDigits(1, /*canHaveSeparators*/ true);
if (!tokenValue) {
error(Diagnostics.Hexadecimal_digit_expected);
tokenValue = "0";
}
else {
pos++;
tokenValue = "0x" + tokenValue;
tokenFlags |= TokenFlags.HexSpecifier;
return token = checkBigIntSuffix();
}
else if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.B || charCodeUnchecked(pos + 1) === CharacterCodes.b)) {
pos += 2;
tokenValue = scanBinaryOrOctalDigits(/* base */ 2);
if (!tokenValue) {
error(Diagnostics.Binary_digit_expected);
tokenValue = "0";
}
return token = SyntaxKind.NewLineTrivia;
}
case CharacterCodes.tab:
case CharacterCodes.verticalTab:
case CharacterCodes.formFeed:
case CharacterCodes.space:
case CharacterCodes.nonBreakingSpace:
case CharacterCodes.ogham:
case CharacterCodes.enQuad:
case CharacterCodes.emQuad:
case CharacterCodes.enSpace:
case CharacterCodes.emSpace:
case CharacterCodes.threePerEmSpace:
case CharacterCodes.fourPerEmSpace:
case CharacterCodes.sixPerEmSpace:
case CharacterCodes.figureSpace:
case CharacterCodes.punctuationSpace:
case CharacterCodes.thinSpace:
case CharacterCodes.hairSpace:
case CharacterCodes.zeroWidthSpace:
case CharacterCodes.narrowNoBreakSpace:
case CharacterCodes.mathematicalSpace:
case CharacterCodes.ideographicSpace:
case CharacterCodes.byteOrderMark:
if (skipTrivia) {
pos++;
continue;
tokenValue = "0b" + tokenValue;
tokenFlags |= TokenFlags.BinarySpecifier;
return token = checkBigIntSuffix();
}
else {
while (pos < end && isWhiteSpaceSingleLine(charCodeUnchecked(pos))) {
pos++;
else if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.O || charCodeUnchecked(pos + 1) === CharacterCodes.o)) {
pos += 2;
tokenValue = scanBinaryOrOctalDigits(/* base */ 8);
if (!tokenValue) {
error(Diagnostics.Octal_digit_expected);
tokenValue = "0";
}
return token = SyntaxKind.WhitespaceTrivia;
tokenValue = "0o" + tokenValue;
tokenFlags |= TokenFlags.OctalSpecifier;
return token = checkBigIntSuffix();
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (ch === CharacterCodes._0) {
if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.X || charCodeUnchecked(pos + 1) === CharacterCodes.x)) {
pos += 2;
tokenValue = scanMinimumNumberOfHexDigits(1, /*canHaveSeparators*/ true);
if (!tokenValue) {
error(Diagnostics.Hexadecimal_digit_expected);
tokenValue = "0";
}
else {
pos++;
tokenValue = "0x" + tokenValue;
tokenFlags |= TokenFlags.HexSpecifier;
return token = checkBigIntSuffix();
}
else if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.B || charCodeUnchecked(pos + 1) === CharacterCodes.b)) {
pos += 2;
tokenValue = scanBinaryOrOctalDigits(/* base */ 2);
if (!tokenValue) {
error(Diagnostics.Binary_digit_expected);
tokenValue = "0";
}
return token = SyntaxKind.NewLineTrivia;
}
case CharacterCodes.tab:
case CharacterCodes.verticalTab:
case CharacterCodes.formFeed:
case CharacterCodes.space:
case CharacterCodes.nonBreakingSpace:
case CharacterCodes.ogham:
case CharacterCodes.enQuad:
case CharacterCodes.emQuad:
case CharacterCodes.enSpace:
case CharacterCodes.emSpace:
case CharacterCodes.threePerEmSpace:
case CharacterCodes.fourPerEmSpace:
case CharacterCodes.sixPerEmSpace:
case CharacterCodes.figureSpace:
case CharacterCodes.punctuationSpace:
case CharacterCodes.thinSpace:
case CharacterCodes.hairSpace:
case CharacterCodes.zeroWidthSpace:
case CharacterCodes.narrowNoBreakSpace:
case CharacterCodes.mathematicalSpace:
case CharacterCodes.ideographicSpace:
case CharacterCodes.byteOrderMark:
if (skipTrivia) {
pos++;
continue;
tokenValue = "0b" + tokenValue;
tokenFlags |= TokenFlags.BinarySpecifier;
return token = checkBigIntSuffix();
}
else {
while (pos < end && isWhiteSpaceSingleLine(charCodeUnchecked(pos))) {
pos++;
else if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.O || charCodeUnchecked(pos + 1) === CharacterCodes.o)) {
pos += 2;
tokenValue = scanBinaryOrOctalDigits(/* base */ 8);
if (!tokenValue) {
error(Diagnostics.Octal_digit_expected);
tokenValue = "0";
}
return token = SyntaxKind.WhitespaceTrivia;
tokenValue = "0o" + tokenValue;
tokenFlags |= TokenFlags.OctalSpecifier;
return token = checkBigIntSuffix();
}
}
if (ch === CharacterCodes._0 && pos + 2 < end) {
switch (charCodeUnchecked(pos + 1)) {
case CharacterCodes.X:
case CharacterCodes.x:
pos += 2;
tokenValue = scanMinimumNumberOfHexDigits(1, /*canHaveSeparators*/ true);
if (!tokenValue) {
error(Diagnostics.Hexadecimal_digit_expected);
tokenValue = "0";
}
tokenValue = "0x" + tokenValue;
tokenFlags |= TokenFlags.HexSpecifier;
return token = checkBigIntSuffix();
case CharacterCodes.B:
case CharacterCodes.b:
pos += 2;
tokenValue = scanBinaryOrOctalDigits(/* base */ 2);
if (!tokenValue) {
error(Diagnostics.Binary_digit_expected);
tokenValue = "0";
}
tokenValue = "0b" + tokenValue;
tokenFlags |= TokenFlags.BinarySpecifier;
return token = checkBigIntSuffix();
case CharacterCodes.O:
case CharacterCodes.o:
pos += 2;
tokenValue = scanBinaryOrOctalDigits(/* base */ 8);
if (!tokenValue) {
error(Diagnostics.Octal_digit_expected);
tokenValue = "0";
}
tokenValue = "0o" + tokenValue;
tokenFlags |= TokenFlags.OctalSpecifier;
return token = checkBigIntSuffix();
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's hard to parse out what the suggestion is - can you explain it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that's fair, I don't know why GitHub insists on showing the old code in the diff rather than diffing against your PR version of the code.

Suggestion is to promote the pos + 2 < end check to the same if that checks for CharacterCodes._0, since all branches beneath it use that condition, then replace the nested if statements with a single switch (charCodeUnchecked(pos + 1)) with grouped case statements for each of the radix markers.

}

if (!(tokenCategory & TokenCategory.RecognizedMisc)) Debug.fail(`Unhandled token category ${tokenCategory}`);
switch (ch) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming CharacterCodes is a const enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.

@DanielRosenwasser
Copy link
Member Author

#59244 tries to just check for identifiers up-front, and moves checks for space earlier. It is faster, but not as fast as the lookup approach here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants