TypeScript 3.9 to 4.0 performance regression

[brieb]

TypeScript Version: 4.0.3

Search Terms: performance regression, upgrade

Related Issues:
#34801

// Continuing the conversation from #41369 (comment)

Upon upgrading to the latest version of TypeScript, we noticed some slowdowns in build performance. In CI, we build 8M lines of TypeScript. We cache our TS output directory between CI builds so that we can incrementally build from it on subsequent builds. We invalidate our cache whenever node_modules changes (#38648). So, when we do a TS upgrade we do a cold build. I was looking at these cold build timings to get a sense of the raw performance of the upgraded version, and we also want to generally keep these full-rebuilds under a 15 min budget for developers who make changes to the package.json.

We've seen these cold build times trending upwards as part of the last few upgrades. In these tests, we use the same snapshot of the code, vary the TS version, and run a few trials to get an average.

We did some timings when upgrading from 3.7 to 3.9 and found
TS3.7: ~9-12 m
TS3.8: ~16-19 m
TS3.9: ~15-18 m

Since then, we've added/migrated more TS code, parallelized our build, and sharded it across five machines. The maximum time of one of the shards represents how long a developer needs to wait until the overall build is finished. On average, we found the longest shard to take 1m 14s more time.
TS3.9: ~17m 11s
TS4.0: ~18m 25s

Here is a replica of our repo setup: https://github.com/brieb/ts-monorepo

Do you have any thoughts on what might have caused the TS3.9 -> TS4.0 slowdown we observed? Any advice on changes we can make to our setup to avoid the perf regressions introduced in 4, so we could proceed with the upgrade? We haven't merged the upgrade yet, because we wanted to avoid slowing down our CI jobs.

Also, do you have any advice for catching this more proactively, in a way that would be beneficial to the TS team? Perhaps we could set up an automated build process that pulls down the latest TS snapshots to pinpoint when slowdowns happen, or come up with a representative example monorepo to add to your profiling suite?

Thank you!

[brieb]

Perhaps we could also try the new profiling mode mentioned in #40124 to identify slow types we may be using.

[DanielRosenwasser]

Thanks for reporting this - have you seen any specific regressions in the replica?

I think that the profiling mode may help, but I don't know if it can scale easily on that much code. @amcasey will have a better clue.

[amcasey]

Hey @brieb, thanks for kicking off this thread!

We did some timings when upgrading from 3.7 to 3.9

I'm used to seeing version to version regressions (things tend to slow down as we add code and features), but those are bigger than I would expect. (I'd say we're usually within 5% of our old numbers.)

Perhaps we could also try the new profiling mode

The new functionality is described here. It's not very user-friendly at the moment, but I'm actively working on it and would love to pair with you on making it work for your project.

As Daniel said, out of the box, it will almost certainly produce unusably large files. I have some scripts for post-processing away the less interesting bits but, again, it would take some patience on your end. 😉

Perhaps we could set up an automated build process that pulls down the latest TS snapshots to pinpoint when slowdowns happen

That would be amazing! I would guess it would be pretty expensive though as you'd probably need to do multiple cold builds per nightly release to get stable numbers.

come up with a representative example monorepo to add to your profiling suite

This is probably more practical, since we'll be able to do multiple runs at a commit granularity. It would likely need to match both your project structure and dependencies (one cause of version-to-version slowdown is suddenly picking up new, more expensive typings).

Let me know how I can help!

[NoPhaseNoKill]

I would be really interested in this, particularly surrounding a base-line of performance on each version. Considering the amount of adoption we're seeing across the open-source community for TS - type-checking/build times will become exponentially more important for companies migrating from JS.

I know that the below may not be feasible, but as an open-source contributor we should be able to find the below information. To summarise I'd be looking for:

1. Baseline stats from 'Performance Tracing' to LoC (e.g estimated 20s:10,000 LoC for time spent type-checking).

2. Regression stats per version/across versions. This would include one major version (i.e 3.7 -> 3.8 has a regression of 5% for type-checking), as well as, across major versions (e.g 3.7 -> 4.1, has a regression of 18% for type-checking)

3. Somewhere that documents contributors to "large/semi-large" (definition of these are subjective) changes to any of these stats

Reasons these may be useful:

1. Identification of bottlenecks in compilation/type-checking speeds become far easier. These can then be opened as 'issues' - and potentially opened up to the community to help with speed increases. For example: you now a series of commits which are responsible for regressions, and one new function may be the sole/main contributor of this. Note: This is something I'd like to help contribute to, but don't know where to start. Could someone point me in the right direction? Basically want to help with build speeds :)

2. Idenitifcation of bugs that cause regression in these times also becomes much easier. For example: you put in the release notes that I'm expecting a regression of 5%. I upgrade to the new version, and notice a 20% slow down. Makes it pretty obvious that something dun' goofed

3. It will help developers make decisions on whether or not it's feasible for them to upgrade to a new major version, or whether they simply don't want to incur the increased cost of build/compilation times

Please let me know if this is the wrong thread for this conversation, I'm just more looking for a way to help this oh so wonderful project in the long-run :)

edit: Added another example

[amcasey]

@NoPhaseNoKill this issue was primarily intended to provide support to @brieb, so ordinarily I'd suggest opening a new one. In this case, however, this answer is (unfortunately) short, so I'll just put it here.

Basically, I spent a bunch of time (weeks) trying to build what you described and ended up convincing myself that it was impossible - the state space of possible TypeScript programs is just too large. Increasing the LoC doesn't directly slow down TS (beyond the obvious linear increase), complexity does. Of course, large programs tend to be more complex.

We do already have a suite of large programs that we use to track our performance over time. Obviously, if we notice a meaningful regression in our suite, we address it (or, in rare cases, conclude that the feature or bug fix is more valuable than the perf lost). In spite of that, there are version over version regressions in large programs like Brie's. That's why we're constantly looking for "representative" programs to add to our suite (while also keeping the size of the suite manageable).

[brieb]

Happy new year!

Just to report back on this, we're looking for resourcing internally for picking up this work - to either build nightlies internally on our end or to contribute a representative sample project to TypeScript's profiling suite (agree this seems more practical). We'll report back here when we know more.

We haven't tried the 4.1 profiling mode just yet. We ran into some issues with 4.1 (errors in the actual compiler vs your typical TS compiler errors). I'll formulate a small-scale repro and will file a separate issue on that. But, once we get past that and fixup our Did you forget to include 'void' in your type argument to 'Promise'?s we'd be eager to give it a whirl.

For the TS4.0 upgrade, we figured out some optimizations we could make to our custom parallel build we use in CI that we think will sufficiently offset the perf regression we observed, such that we can proceed with the upgrade and still be below our 15 min CI budget.

Thanks again for your willingness to partner on this, so we can help catch these kinds of regressions earlier going forward!

[brieb]

Just to report back with another update. We have been steadily making performance improvements to the point that we could land the 4.0.3 upgrade while still staying under our 15 min CI budget. We subsequently upgraded to 4.1.3 and performance improved. So, overall the performance impact was roughly neutral. We're so happy to be on the latest version to get all the new features. 😃

We're still figuring out whether someone could pick up the nightly build work. But, it's becoming less of an issue for us now with our other perf improvements and soon moving to Bazel. Many of our ideas have come from your wiki page https://github.com/microsoft/TypeScript/wiki/Performance, so thank you for putting that together.

@lencioni on our team tried the new profiling mode. It helped us see which files were slow to compile. Though we did find it challenging from the output to understand which types were causing the slowdowns and why. You had mentioned that you're planning on making usability improvements. So, we're looking forward to trying those out as part of future TS releases.

Thanks again for your help and insights with this issue. Will go ahead and close this out, now that we've been able to successfully land our 4.1.3 upgrade!

[amcasey]

Fantastic! Thanks for the update, @brieb! Let me know if you want to set up a call about how to read the traces and how to make them more readable.