Implicit Referencing - opting in and opting out

[johnnyreilly]

Hi,

I've a question about Visual Studio's Implicit Referencing. I've looked around in vain for a definitive answer and so I thought it might make sense to raise the question here.

Previously, all TypeScript files in a project had to reference each other explicitly. With 0.9.1, they now implicitly reference all other TypeScript files in the project.

Since 0.9.1 I've used this feature pretty happily and purged my files of the mass of /// <reference path= entries.

I've recently been wondering if there are any hidden "gotchas" to using /// <reference path= in a Visual Studio context? My understanding is that if there are any /// <reference path= instances in a file then implicit referencing goes out the window and only the specified references are used. Is that correct? (So use of any references in the head of a ts file is effectively the same as opting that file out of Implicit Referencing for Visual Studio.)

@mhegazy - this might be a question for you? Similar ground was covered here.

[mhegazy]

Short answer is no. /// references is honored regardless of how the context is built.

/// references are considered "dependencies" and the compiler/Language service will honor these dependencies while building the compilation context. where a compilation context is all files part of a single compilation unit (i.e. a single invocation to tsc.exe). Building a context is similar in all supported modes of interaction with VS and the compiler. Here is how a context is built (bold signals differences).

1. Command line compiler

• Start with files passed on the command line, in order
• For each file, examine its /// references and import = require() statements
  • If the file has not been seen before add it to the list, and recursively examine its references

2. Visual Studio, with a project referencing the file

• Start with all files in the project with build action === "TypeScriptCompile", in order of reference in the project file
• For each file, examine its /// references and import = require() statements
  • If the file has not been seen before add it to the list, and recursively examine its references

3. Visual Studio, with No project referencing the file (loose files)

• Assume all loose files currently open as part of one project "Miscellaneous"
• Start with all open files, that do not belong to a project (see above)
• For each file, examine its /// references and import = require() statements
  • If the file has not been seen before add it to the list, and recursively examine its references

There is not "implicit references" per se, its detecting the "compilation context". In a project the msbuild task will take all the TypeScriptCompile items and pass them to the compiler on the command line, the behavior here should be identical between VS and the command line. With loose files, we have no way of knowing how you are going to build this file, so we make an assumption that all the files currently open are part of one compilation unit and will all be passed on the command line to the compiler.

in all cases, you can use /// references to pull in files into your context that you have not specified earlier.

Now, some caveats:

1. MSBuild Incremental build
   Problem: MSBuild incremental build support relies on checking the time stamp of the inputs against the outputs, if you are in a project context, inputs are all TypeScriptCompiler items, if you have a file that is not references in the project, but pulled in as a /// reference, MSBuild has no way to know about it, so it will not check its time stamp. so if you update only that file, and build, the build will be skipped as MSBuild thinks the project is up-to-date.
   Solution: Include all your files in your project, use /// references for ordering or documenting dependency, but not to pull in new items.
2. Ordering output with --out
   Problem: The compiler will generate outputs in the order of occurrence in the dependency graph. The dependency graph is built using the logic stated earlier. In the batch compiler case, you can change your order on the command line, but in VS, manually editing the project file to enforce order is neither convenient nor productive.
   Solution: There is support to a special file "_references.ts" at the root of your project (has to be at the root, if anywhere else it will be treated like a normal file). If this file exists, the file is guaranteed to be passed as the first file in the list, both in VS and in the MSBuild task. that allows you to specify order through adding /// references in _references.ts. Remember that the graph is built in order, and following /// references in order, so the order of /// references in _references.ts will dictate the order of files in the output, in absence of any circular references.

Diagnosis:

Here are some tips to diagnose problems related to context building:

1. Enable "Virtual Project" view:
   Go to Tools > Options > Text Editor > TypeScript > Project, and enable the two check boxes for Virtual Projects (see screen snapshot)
   
2. Understanding Virtual Projects:
   In the screen shot below, I have 3 files open:
3. app.ts part of a project (TypeScriptHTMLApp14)
4. NotnProject1.ts, pulled in through a /// reference from app.ts
5. NotInProject2.ts, not part of any project, nor referenced from any file that belongs to a project, just open (loose file)
6. NotInProject3.ts, another loose file, but closed.

From the screenshot you can see a new TypeScript Virtual Projects node under the solution. this shows you how VS have computed the contexts. as you can see there are two "Compilation Context" or Projects:

• Project#1 -- corresponds to my real project, TypeScriptHTMLApp14, and was built by first including app.ts, then walking its references to include NotInProject1.ts
• Project#2 -- Miscellaneous, corresponds to all other "loose files" that were not part of a project. We start by open files, NotInProject2.ts and follow its /// references to load a closed file "NotInProject3.ts"

[johnnyreilly]

Hi @mhegazy,

Thank you for the fully detailed response! Very helpful.

First important take home:

• Context is added to not replaced when using /// references. So it's not possible to wipe existing references in the project just by adding a /// reference to the top of a file. Perfectly clear - thanks.

Second important take home:

There is not "implicit references" per se, its detecting the "compilation context"

Thanks for detailing the 3 different ways context can be constructed. That's really helpful. I wasn't aware of the details of the third listed way:

Visual Studio, with No project referencing the file (loose files)

I’d love you to clarify this a little if you could. It sounds like this scenario covers where someone is just using Visual Studio as an IDE for TypeScript. Presumably this means 1 of 2 things; either

1. There is no csproj at all.
2. Or there is a csproj but it does not refererence *.ts files.

Question about 1.

If 1. is the case how does Visual Studio decide the compiler switches for TypeScript? What I mean is, even the Visual Studio TypeScript template solution ships with a csproj file that says <TypeScriptToolsVersion>1.0</TypeScriptToolsVersion>, <TypeScriptRemoveComments>false</TypeScriptRemoveComments> etc. Without out these specified how does tsc get invoked? Just as is? (So specifying –noImplicitAny or similar is not possible in this scenario?)

Either way, the implication is that /// references should be used when using loose *.ts files to ensure context is built correctly.

Question about _references.ts

I know _references.ts was originally introduced to allow the ordering of the files (particularly important for having the correct overload resolution when there's a million jQuery plugins). Presumably in the “no project referencing the file” scenario it could also serve as a replacement for having the csproj listing all the build action === "TypeScriptCompile" files?

What I mean is, you could use _references.ts to control your context construction / implicit referencing. This would be achieved by listing all TS files in _references.ts and either by having every TS file reference _references.ts in turn (or by always keeping _references.ts open in VS – which probably isn’t as practical).

Is there any reason this shouldn't work? I'm assuming that if _references.ts was located at the highest level in the world of scripts it will considered as being in the root and so will still be passed as the first file to tsc.

Question about ASP.Net vNext world

Given that in the ASP.Net vNext world I understand that csproj files are to disappear what is the plan for context building in this sort of project? Is it loose files all the way?

[mhegazy]

Question 1:
That is a great question. In loose files we have no way of knowing how the user is building the files. as you mentioned they could be using it as a text editor, and they have a different way to manage their build. so we make no assumptions. as for configuration, you get the default configurations from tsc.exe with some exceptions:

• target === ES5 // basically the latest
• toolsVersion === Latest // again, latest

There are two options that you can control through tools/options (see screenshot below, namely:

• compile on save
• module (amd or commonjs -- default unspecified)

Note that these configurations only affect loose files. for files in projects, you need to set them for every project.

Question 2:
one note, _references.ts is only special in the project scenario. in the loose file scenario it is just a normal file.
One common practice that we have seen is using an index.ts file that lists all the files in your project, and all the files in your project reference index.ts; this way you open a file and you get the whole project loaded correctly all the time.
you could definitely use _references.ts to do that. my recommendation is not to mix projects and /// references if you can, to ensure your incremental compilation works as expected.

Question 3:
I have not played with the new ASP.net projects in a while. but i do not think there is much change in the fundamentals since i looked at it. The ASP.net project still provide a project system interface to VS, though you do not have a project file. The TypeScript language service asks the project system about the files in the project. as long as the project system gives us the typescript files, we should be good.
The problem is the configuration. but if you are using a build tool like grunt or bower, there is a lot of specific knowlege that needs to be backed into the language service, which i do not think we will be able to support.

[johnnyreilly]

Thanks @mhegazy. That clarifies things a great deal.

my recommendation is not to mix projects and /// references if you can, to ensure your incremental compilation works as expected.

This totally makes sense. However it raises an issue about having a TypeScript project which some people may be using Visual Studio to work on and some people may be using another IDE (eg WebStorm). In this situation it would be necessary to use /// references for the people not using Visual Studio.

In order to avoid confusion in this situation the ideal would seem to be having a way of opting out of VS project support so everyone has the same experience regardless of IDE. So everyone would depend on using /// references to build context.

I don't think there is a way to do this at present. Is this something that could made an option? eg. Have a Visual Studio setting that disabled implicit referencing?

[johnnyreilly]

@jonathandturner did discuss the possibility of providing this option back on Oct 23, 2013 by the way:

https://typescript.codeplex.com/workitem/1471#CommentContainer7

This discussion took a number of different turns after that and the option was never really returned to. Just thought I'd provide this as a reference.

[mhegazy]

@johnnyreilly i am not sure i understand the desired behavior. is the problem the project file? what if you just forget about the project file and open the files in VS and let /// reference resolution figure out your context?

if this is the case, just add after your import to Microsoft.TypeScript.targets. just note that it will ignore your project file all together. that means files, and configuration, and will treat all files as if they did not belong to a project.

  <PropertyGroup>
    <!-- Indicates to the language service that this project supports TypeScript -->
    <TypeScriptEnabled>false</TypeScriptEnabled>
  </PropertyGroup>

[johnnyreilly]

@mhegazy,

I'm guess the question is, what's the best way to set up a TypeScript project that can be worked on regardless of IDE (so whether it's Visual Studio, WebStorm etc)?

As you say, you could have both the project file figuring out context for Visual Studio users and /// references in place for non-Visual Studio users. But I'm mindful that you'll have the situation of trying to keep both the /// references and the TypeScriptCompile references in the project file in line. Also, it's not very DRY 😄 Your comment seems to express a similar concern?:

my recommendation is not to mix projects and /// references if you can, to ensure your incremental compilation works as expected.

For that reason I thought it might be simpler to abandon the project file approach entirely. That way the only maintenance is just the /// references and all developers (regardless of IDE) should need to do the same maintenance to keep the code compiling. No danger that a Visual Studio user might add a file to the project but forget to add the appropriate /// reference if you see what I mean.

I appreciate this might sound like an odd thing to make a fuss about but I'm very keen to reduce "developer friction". When I've been introducing people to TypeScript I've had them complain that they feel the tooling is fighting against them on occasion. I want to reduce that as much as possible. Whilst using the project file to build context is a really handy shortcut for Visual Studio users it also kind of erects a barrier for TypeScript users between the Visual Studio world and those outside it. (I'm sure it wasn't intended to be that - but that's kind of a side effect I guess.)

I'll try out the <TypeScriptEnabled> approach you suggest and report back.

[johnnyreilly]

Hi @mhegazy,

I put together a little test project using the TypeScriptEnabled approach suggested. You can see it here:

https://github.com/johnnyreilly/proverb-without-implicit-referencing

It has the csproj change you suggest:

  <PropertyGroup>
    <!-- Indicates to the language service that this project supports TypeScript -->
    <TypeScriptEnabled>false</TypeScriptEnabled>
  </PropertyGroup>

I created a file I called _references.d.ts which is just a list of the TypeScript files and definition files my project uses:

https://github.com/johnnyreilly/proverb-without-implicit-referencing/blob/master/Proverb.Web/app/_references.d.ts

And each TypeScript file has a /// <reference path="_references.d.ts" /> (with appropriate path tweaks) at it's head. Seems to work just fine. Is this what you were proposing?

So am I right in thinking that <TypeScriptEnabled>false</TypeScriptEnabled> in actual fact means "deactivate implicit referencing" or is there more to it than that? I hadn't heard of this flag before....

[mhegazy]

@johnnyreilly, increasing developer productivity is the reason typescript exists, so friction is not something desired.

The part that I find confusing is why you need the project file. If you just open one of your files by itself, /// references will cause you to load the all your files, similar to other editors. I am assuming you are using gulp to build anyways.

Either ways, figuring out files is not the only issue, compiler options are the other piece of the puzzle. What tells the language service is to not to load anything from the project file.

[johnnyreilly]

@mhegazy,

Yes I am using Gulp to build.

The project file is kind of extraneous with this approach. I suppose I'd imagined that someone using Visual Studio would automatically be using a project file as they would in a non-TypeScript project. And if they were, I was keen to see if there was a way to use the project file without opting into the implicit file referencing. I think we've established this is possible with <TypeScriptEnabled>false</TypeScriptEnabled>?

Though you raise an interesting point - I'm guessing you're thinking that if the user is using Visual Studio just as an editor that they'd rely on the TypeScript Virtual Project instead? That way they can entirely bin the project file.

As you rightly say, compiler options are the other piece of the puzzle. From what you've said earlier, without a project file you have no control over compiler options. That's a shame - I'm a --noImplicitAny kind of chap and I'd certainly miss that.

From what I understand, a cross IDE project using a common approach is pretty much do-able in the following ways:

1. With <TypeScriptEnabled>false</TypeScriptEnabled> in a project file. This approach requires that all developers (regardless of IDE) use /// references to build context. Compiler options in VS can be controlled using the project file as is.
2. Using Visual Studio without any csproj tweaks. This approach requires that all /// references are maintained to build context outside of Visual Studio. It's possible that /// references and the csproj could get out of line - care required to avoid this. Compiler options in VS can be controlled using the project file as is.
3. Using just files in Visual Studio with /// references to build context. This scenario also requires that all developers (regardless of IDE) use /// references to build context. In Visual Studio there will be no control over compiler options. 😢

Are there any plans to allow the control of "default" TypeScript compiler options in Visual Studio in the absence of a project file? This would help with approach 3.

I think approach 1 might be the best cross-IDE approach available at present in my view.

[mhegazy]

@johnnyreilly i think you summed up nicely. thanks for the write up.

I do not see a problem with us exposing more options as global settings for loose files, e.g. --noImplicitAny, and --target. I have created #1223 to track that. but there are ones that we know we can not support globally, e.g. out, outDir, declarations. maproot, sourceRoot. There is room here for a proposal :)

[ghost]

A good counter example is GitHub-Electon or similar background-worker script/dom-script situations where definintions might only be available in certain contexts. For Electron, there are two modules called 'ipc' that are loaded in different contexts only ("renderer" vs "main" contexts). With Typescript automatically including ALL available files in the project, I cannot be selective about which d.ts contexts to include for a particular script. Instead I have to split it into multiple projects - maybe not a bad thing, I'm not sure. (also, the 'ipc' conflict doesn't appear in the GitHub-Electon*.d.ts files in their current release, but would once those definitions mature)

[mhegazy]

@GITGIDDY tsconfig.json (with full support in next VS2015 public release) should solve this issue for you.

[agoodwin]

Problem: The compiler will generate outputs in the order of occurrence in the dependency graph. The dependency graph is built using the logic stated earlier. In the batch compiler case, you can change your order on the command line, but in VS, manually editing the project file to enforce order is neither convenient nor productive.
Solution: There is support to a special file "_references.ts" at the root of your project (has to be at the root, if anywhere else it will be treated like a normal file). If this file exists, the file is guaranteed to be passed as the first file in the list, both in VS and in the MSBuild task. that allows you to specify order through adding /// references in _references.ts. Remember that the graph is built in order, and following /// references in order, so the order of /// references in _references.ts will dictate the order of files in the output, in absence of any circular references.

@mhegazy I've just come across this problem, and after finding multiple issues on the subject this seems to be the most appropriate place to get some clarification.

First let me explain my situation a bit:

I've got a Visual Studio project, with its TypeScript properties set to combine the compiled TypeScript into a single .js file, and among many other .ts files it has a base class in one .ts file and a child class in another .ts file. Until I added the base class I had assumed that I didn't need to use /// references at all while using a Visual Studio project, because I thought Visual Studio was figuring out TypeScript dependencies on its own. After reading this issue I now realise that it's not, and in my case my output .js file has the base class declared after the child class, resulting in the "TypeError: b is undefined in __extends" error (#4341).

So, based on your responses to this and other issues, it sounds like I should use a _references.ts file or just use /// reference directives in my source files.

The problem I have with these options is that they're manual processes, but the overall issue seems to be something that could have an automated solution. For example, when you described this process (regarding how files are passed to the TypeScript compiler by Visual Studio):

2. Visual Studio, with a project referencing the file

• Start with all files in the project with build action === "TypeScriptCompile", in order of reference in the project file
• For each file, examine its /// references and import = require() statements

  • If the file has not been seen before add it to the list, and recursively examine its references
    

Wouldn't it be possible for an extra step to be added here so that, instead of initially ordering the files based on their order within the project file, the initial ordering of the files is based on their dependencies?

I mean, doesn't Visual Studio already have this information at hand? (I assume this because IntelliSense and Go To Definition work in TypeScript, so it must surely be quite easy to figure out that a file containing a base class needs to be compiled before files containing child classes).

Having said that though, I realise that in reality there's probably a huge amount of work involved in implementing this dependency checking and ordering functionality – and if it were to be implemented it would probably belong in the TypeScript compiler itself, not in Visual Studio.

So I guess my only real point of confusion is, why does Visual Studio do anything around a "compilation context" at all? I guess I'm with @johnnyreilly when he says:

For that reason I thought it might be simpler to abandon the project file approach entirely. That way the only maintenance is just the /// references and all developers (regardless of IDE) should need to do the same maintenance to keep the code compiling. No danger that a Visual Studio user might add a file to the project but forget to add the appropriate /// reference if you see what I mean.

The current behaviour just seems a bit misleading to me. The way that Visual Studio figures out the TypeScript references without explicitly having any /// directives, and then enables IntelliSense etc, makes it appear as though your code will run when it won't necessarily. This mainly seems to be an issue around inheritance, because of how extends compiles to JavaScript that is run at the point in time when a child class is declared. I'm not aware of many other situations where you get this out-of-order compilation happening.

I realise this whole comment probably seems a bit aimless, so I guess my actual point can be boiled down to this:

If you are working in a project within Visual Studio, and:

• You are compiling your TypeScript files into a single JavaScript file
• You are making use of inheritance

Then there is a high likelihood that you will still need to use /// reference directives or a _references.ts file in order to have your output JavaScript run without errors. With that being the case, wouldn't it be better to require developers to use /// reference directives within Visual Studio (like they have to when using other IDEs) rather than giving the appearance of handling this automatically? I realise I'm simply repeating @johnnyreilly's point, and like him I understand that it might seem like a small thing to be fussing about, but I do think it's misleading the way things are. Am I missing something obvious, or would I be right in saying that there's not really anything gained by having Visual Studio detect the TypeScript compilation context?

[mhegazy]

It is a design choice not to reorder output. the compiler will generate code that looks like your input. The compiler can not generate code that will always work in all cases, e.g. if you have a cyclic dependency between files.

The solution we believe is correct is to issue an error when you have your base before your derived, see #5207, the PR for it is in, #4343, we need to update the PR though.

[agoodwin]

Thanks for the quick reply,

I don't want to keep bothering you about this for too long, but if it's a design choice to not reorder the output, what was behind the design choice to have all of a project's .ts files automatically passed to the TypeScript compiler (in project reference order) in the first place?

Would I be right in saying that this was because there used to be no support for a tsconfig.json file (before TypeScript 1.5)? So it was the only way to have Visual Studio compile TypeScript files in a project automatically? I guess the alternative would have been for the user to add a pre or post build step to run tsc.exe, which wouldn't have been very nice.

If that's the case, will things be moving more towards tsconfig.json in the future (even when using a project file)? I've been reading through #3983 and its related issues. It kind of sounds to me like the TypeScript properties in a *proj file are on their way out, and instead we will always be using tsconfig.json in future. This is certainly already the case for ASP.NET 5 projects, so is this the intent for other kinds of projects as well?

[mhegazy]

what was behind the design choice to have all of a project's .ts files automatically passed to the TypeScript compiler (in project reference order) in the first place?

Well. the ts files need to be compiled to js files to be able to run, we would assume you want to compile all the files in your project, so we pass them all to the compiler. this is the same behavior as C#, VB, etc..
Now, for the order, it is the order that MSBuild decides when we ask for TypeScriptCompile items in the file. no special sorting or ordering is going on.

If that's the case, will things be moving more towards tsconfig.json in the future (even when using a project file)?

yes. this is the intention. tsconfig.json has been easier for developers to reason about than the xml project file, and fits with other JS tooling patterns.

This is certainly already the case for ASP.NET 5 projects, so is this the intent for other kinds of projects as well?

Starting with TS 1.8, if you have a tsconfig.json in your project file it will be used as the source of configurations for all project types. see https://github.com/Microsoft/TypeScript/wiki/What%27s-new-in-TypeScript#improved-support-for-tsconfigjson-in-visual-studio-2015

[johnnyreilly]

I can confirm that tsconfig.json rocks and will improve your life by at least 23%. It's that good. 😄

[agoodwin]

Excellent, thanks for clearing that up. I didn't realise that tsconfig.json was already supported for all project types - I should have just tried it.

I'm already using TypeScript 1.8, so I'll switch to using tsconfig.json right away. Thanks for your help.

[wclr]

@mhegazy
Could you please elaborate what is current (or upcoming due to TS 2.0) state of need of triple slash reference API for types dependency management? When/where it still should be used and should be be avoided if possible?