External module resolution logic

[vladima]

Problem

Current module resolution logic is roughly based on Node module loading logic however not all aspects of Node specific module loading were implemented. Also this approach does not really play well with scenarios like RequireJS\ES6 style module loading where resolution of relative files names is performed deterministically using the base url without needing the folder walk. Also current process does not allow user to specify extra locations for module resolution.

Proposal

Instead of using one hybrid way to resolve modules, have two implementations, one for out-of-browser workflows (i.e Node) and one for in-browser versions (ES6). These implementations should closely mimic its runtime counterparts to avoid runtime failures when design time module resolution succeeded and vice versa.

Node Resolution Algorithm

Resolution logic should use the following algorithm (originally taken from Modules all toghether):

require(X) from module at path Y

If exists ambient external module named X {
  return the ambient external module 
}
else if X begins with './' or '../' or it is rooted path {
  try LOAD_AS_FILE(Y + X, loadOnlyDts=false)
  try LOAD_AS_DIRECTORY(Y + X, loadOnlyDts=false)
}
else {
  LOAD_NODE_MODULES(X, dirname(Y))
}
THROW "not found"

function LOAD_AS_FILE(X, loadOnlyDts) {
  if loadOnlyDts then load X.d.ts 
  else { 
    if  X.ts is a file, load X.ts
    else if X.tsx is a file, load X.tsx
    else If X.d.ts is a file, load X.d.ts
  }
}

function LOAD_AS_DIRECTORY(X, loadOnlyDts) {
  If X/package.json is a file {
    Parse X/package.json, and look for "typings" field.
    if parsed json has field "typings": 
    let M = X + (json "typings" field)
    LOAD_AS_FILE(M, loadOnlyDts).
  }
  LOAD_AS_FILE(X/index, loadOnlyDts)
}

function LOAD_NODE_MODULES(X, START) {
  let DIRS=NODE_MODULES_PATHS(START)
  for each DIR in DIRS {
    LOAD_AS_FILE(DIR/X, loadOnlyDts=true)
    LOAD_AS_DIRECTORY(DIR/X, loadOnlyDts=true)
  }
}

function NODE_MODULES_PATHS(START) {
  let PARTS = path split(START)
  let I = count of PARTS - 1
  let DIRS = []
  while I >= 0 {
    if PARTS[I] = "node_modules" CONTINUE
    DIR = path join(PARTS[0 .. I] + "node_modules")
    DIRS = DIRS + DIR
    let I = I - 1
  }
  return DIRS
}

RequireJS/ES6 module loader

• If module name starts with './' - then name is relative to the file that imports module or calls require.
• If module name is a relative path (i.e. 'a/b/c') - it is resolved using the base folder.

Base folder can be either specified explicitly via command line option or can be inferred:

• if compiler can uses 'tsconfig.json' to determine files and compilation options then location of 'tsconfig.json' is the base folder
• otherwise base folder is common subpath for all explicitly provided files

Path mappings can be used to customize module resolution process. In 'package.json' these mappings can be represented as JSON object with a following structure:

  {
    "*.ts":"project/ts/*.ts",
    "annotations": "/common/core/annotations"
  }

Property name represents a pattern that might contain zero or one asterisk (which acts as a capture group). Property value represents a substitution that might contain zero or one asterisk - here it marks the location where captured content will be spliced. For example mapping above for a path 'assert.ts' will produce a string 'project/ts/assert.ts'. Effectively this logic is the same with the implementation of locate function in System.js.

With path mappings in mind module resolution can be described as:

for (var path in [relative_path, relative_path + '.ts', relative_path + "d.ts"]) {
    var mappedPath = apply_path_mapping(path);
    var candidatePath = isPathRooted(mappedPath) ? mappedPath : combine(baseFolder, mappedPath);
    if (fileExists(candidatePath)) {
        return candidatePath
    }
}
return undefined

With path mappings it becomes trivial to resolve some module names to files located on network share or some location on the disk outside the project folder.

{
    "*.ts": "project/scripts/*.ts",
    "shared/*": "q:/shared/*.ts"
}

Using this mapping relative path 'shared/core' will be mapped to absolute path 'q:/shared/core.ts'.

We can apply the same resolution rules for both modules and tripleslash references though for the latter onces its is not strictly necessary since they do not implact runtime in any way.

[vladima]

@mprobst and @jrieken - can you please check if this proposal covers your scenarios?

[danquirk]

Need to cross reference with comments here too #247

[mprobst]

One question:
"2. If X begins with './' or '/' or '../' [...]"

What does it mean when X is e.g. '/foo/bar' and you calculate Y + X? Do you resolve X against Y, e.g. like a browser would resolve a URL, so that '/foo/bar' would result in an absolute path?

Regarding the path mapping, this approach would not quite solve our problem. What I want to express is that a source file should first be searched in location A, then in location B, then C, and so on. That should be true for all source files, not just for a specific subset (as you do with the pattern matching). The source code of the including file should not care where the included file is located.

I presume the harder problem is establishing what the path that is searched in those locations is exactly. If a file is loaded relative to a base URL, we could first resolve all paths relative to that file's relative URL to the base, and then use the result to look up in the search paths.

Given a file Y, a require('X'):

1. let path be resolve(Y, X). Question here: default to relative paths or default to absolutes?
2. for each include path i (passed on command line, current working directory is the implicit first entry)
   1. let effective path be resolve(i, path)
   2. if effective path exists, return effective path
   3. continue
3. throw not found.

The initially passed 'Y' from the command line would also be resolved against the include/search paths.

For example, for a file lib/a.ts and running tsc -I first/path -I /second/path lib/a.ts in a path /c/w/d, where a.ts contains a 'require("other/b")', the locations searched for b would be, assuming default to absolute paths:

1. /c/w/d/other/b.ts (+.d.ts, +maybe /index.ts and /index.d.ts)
2. /c/w/d/first/path/other/b.ts
3. /second/path/other/b.ts

If you specified ./other/b, the locations searched would be:

1. /c/w/d/lib/other/b.ts (+.d.ts, +maybe /index.ts and /index.d.ts)
2. /c/w/d/first/path/lib/other/b.ts
3. /second/path/lib/other/b.ts

This would allow us to "overlay" the working directory of the user over an arbitrary number of include paths. I think this is essentially the same as e.g. C++ -I works, Java's classpath, how the Python system search path works, Ruby's $LOAD_PATH etc.

[mprobst]

... oh and obviously, I mean this as a suggestion to be incorporated into your more complete design that also handles node modules etc.

[vladima]

What does it mean when X is e.g. '/foo/bar'?

Yes, module name that starts with '/' is an absolute path to the file

I think path mappings can solve the problem if we allow one entry of it to be mapped to the set of locations

{
    "*": [ "first/path/*", "/second/path/*" ] 
}

Having this update module resolution process will look like:

var moduleName;
if (moduleName.startsWith(../) || moduleName.startsWith('../')) {
    // module name is relative to the file that calls require
   return makeAbsolutePath(currentFilePath, moduleName); 
}
else {
    for(var path of [ moduleName, moduleName + '.ts', moduleName + '.d.ts']) {
        var mappedPaths = applyPathMapping(path);
        for(var mappedPath in mappedPaths) {
            var candidate = isPathRooted(mappedPath) ? mappedPath : makeAbsolute(baseFolder, mappedPath);
            if (fileExists(candidate)) {
                  return candidate;
            }
        }
    }
}
throw PathNotFound;

Note:
I do see a certain value of having a path mappings, since it allows with a reasonably low cost easily express things like: part of files that I'm using are outside of my repository so I'd like to load the from some another location. However if it turns out that all use-cases that we have involve remapping of all files in project and path mappings degrade to just include directories - then let's use include directories.

Out of curiosity, do you have many cases when code like require('../../module') should be resolved in include directories and not in relatively to file that contains this call?

[mprobst]

Re the code example, I think even relative paths should be resolved against the mapped paths. Imagine you have a part of your code base in a different physical location (repository), but still use the conceptually relative path. In general, I think it might be a good idea to have a logical level of paths that get resolved, and then those are matched against physical locations, but the two concepts are orthogonal otherwise - that is, you can have relative or absolute logical paths mapping to any physical location.

Our particular use case is that we currently exclusively use absolute rooted include paths (require('my/module') where my is resolved to the logical root of the source repository). Relative paths could be useful if you have deep directory structures, but would need to be clearly marked so that there's no ambiguity, e.g. by using ./relative/path.

I see that your example of path mappings is strictly more powerful, but at least from where I stand, I think include directories cover all we need, and might be simpler for tooling to understand & implement. YMMV.

[basarat]

Would be great if typescript.definition was supported. #2829

I am open to different suggestions if you want.

[csnover]

For path mapping AMD/ES6 could you follow the syntax already used by AMD paths common configuration? It maps module ID prefixes, from most-specific to least specific, so e.g.:

paths: {
  'foo': '/path/to/foo',
  'foo/baz': '/path/to/baz'
}

'foo/bar/blah' -> '/path/to/foo/bar/blah.{ts,d.ts}'
'foo/baz' -> '/path/to/baz.{ts,d.ts}'

In so doing, this leaves open the possibility of the compiler being able to simply consume the same AMD configurations used by an app at runtime, instead of introducing an incompatible equivalent syntax.

[3nsoft]

I will give my vote for having typescript field in package.json, into which declarations can be placed, or some other configs, relevant to compiler.

This will keep things simpler (no yet another config), and it will clearly indicate that package can be used with TS.

[3nsoft]

When package produces single entity (ES6' export default), I assume, declaration file, hopefully generated by compiler, will be something like

exports = {
    foo: ...;
    bar: ...;
}

[mhegazy]

When package produces single entity (ES6' export default), I assume, declaration file, hopefully generated by compiler, will be something like

@3nsoft, i am not sure i understand the question/comment

[3nsoft]

@mhegazy I have in mind package that is a single object or function. How will this be done?
The exports = Foo; is analogy from declarations like

declare module 'q' {
    exports = Q;
}

In given setting the declare module 'q' { is no longer needed, and we are left with exports = Foo;, I guess. I do not know if such line is currently valid, or if something else will be chosen.
I just want to point out this use case, analogous to ES6's export default. As having every single-function module to export function makeFactory(): Factory; is not very convenient.

[mhegazy]

Typescript has a non-ES6 export syntax export = id. so your module would look like:

declare module 'q' {
    function makeFactory(): Factory;
    export = makeFactory;
}

and you would import it as:

import q = require("q");
var f = q();

[poelstra]

@vladima For the Node case, would it make sense to automatically detect the typing to use based not only on a typings property, but also on the main field?

LOAD_AS_DIRECTORY(X)

1. If X/package.json is a file,
   • Parse X/package.json
   • If there is a "typings" field
     • let M = X + (json 'typings' field)
     • LOAD_AS_FILE(M)
   • If there is a "main" field
     • let M = X + (json 'main' field, with optional ".js" extension stripped off)
     • LOAD_AS_FILE(M)
2. If X/index.ts is a file, load X/index.ts. STOP
3. If X/index.d.ts is a file, load X/index.d.ts. STOP

This way, I won't have to specify e.g.:

{
  // ...
  "main": "./dist/foo.js",
  "typings": "./dist/foo.d.ts"
}

but a simple main suffices.

BTW FWIW, I think I also prefer typings over typescript.definitions, as it seems more people (Flow) are using the .d.ts format, so it's not necessarily Typescript-specific.

[3nsoft]

@poelstra I'd recall Python's wisdom of "explicit is better".
It is better to explicitly have types-related field(s).

If others are using .d.ts, and format becomes non-Typescript-specific, then it is a good reason to have a separate typescript place, which compiler will be assured is related to TS, and, therefore, can be reliably used.

Here are other fields, which might be useful in the future:

• typescript.src :string
• typescript.src-maps :string

These and other goodies will benefit from having an agreed-upon place in package.json, i.e. typescript object in package.json

[poelstra]

I think it would be useful to consider adding an extra lookup to the Node case, to get smooth support for non-Typescript packages too (i.e. all the good stuff on DefinitelyTyped).

Didn't want to highjack this thread, so created #2839 for it. Curious what you guys think about it, though.

[basarat]

If X.d.ts is a file, load X.d.ts. STOP

I didn't notice the .d.ts stuff in there. Thanks @poelstra for the example : https://github.com/basarat/typescript-node/tree/master/poelstra2 👍 !

[basarat]

Having looked through @poelstra's example I definitely agree about calling it typings. This is because it is different from typescript.definition. typescript.definition would import the stuff into the global namespace (and that is a bad idea) whereas typings is used to resolve a require call and do a local import of the exported stuff.

[Pajn]

What is the current status for a package that exports multiple modules?
For example I want to add typings for both my-package and my-package/react.
I can't seem to use declare module...

[avesus]

Module resolution should work on the underlaying JavaScript level, and not on transpiled language. Stop doing wrong work. Babel never trying to traverse dependencies, that's the purpose of a dependencies bundler which will work on compiled JavaScript level.

WebPack and TypeScript guys really do very bad design choices about separation of concerns and isolation of responsibilites.

[basarat]

Babel doesn't do type checking. To check imports you need to know their location 🌹

[masaeedu]

@avesus Module resolution is a platform/runtime concern, not a language concern. ES6 specifies a syntax for describing and importing modules, but it does not specify a uniform module loader or module resolution strategy that is implemented by all environments where JS runs. While NodeJS and browsers do not yet natively implement a module loader, TypeScript will need to emit different stuff to support different module loaders.

[avesus]

Dear Asad,

NodeJS natively implements module loader. And I mean not LOADING process,
but MODULES COMPOSITION process. De-facto standard for that - NPM Node
modules.

And that standard is f**_ly simple: EXPORT YOUR JAVASCRIPT to get it easily
imported by require() or import declaration specified in f**_ng JAVASCRIPT
code. Not in TypeScript, not in CoffeeScript, not in ClosureScript.

This f**ng ClosureScript and TypeScrypt allow to export more than one
module from a single file.

Mr. Asad, you have to know that Node Modules system along with NPM was best
invention in world of composability.

When you write import 'module.ts'; and it's compiled into LOADING of a
typescript file instead of JavaScript, that's a crap.

Babel compiles any JSX and other files without trying to directly load
dependencies AS LANGUAGE OF DEPENDENT CODE, but transforms them to
require(JAVASCRIPT).

Exporting multiple modules from a single typescript file enforces to create
multiple JavaScript files - this breaks npm/node modularity.

When WebPack (f_**ng too) allows you to import css, coffee, JSX and a lot
of other s_*t, YOUR FILE CANNOT BE COMPILED INTO A MODULE WITH EXPORTS.
Trying to import external dependencies EVEN ON A FILE LEVEL WITHIN THE SAME
PROJECT HIERARCHY in the language of implementation instead of JavaScript
makes your project fragile AND INCOMPATIBLE WITH NODE ECOSYSTEM.

Hope @sindresorhus and @tj agree with me. It will be very interesting to
hear expert point of view on the topic of corruption of NPM/Node best parts
of the world's best practices made by Microsoft who has no idea what NPM
ecosystem is or intentionally trying to break it and mr. Sokra who trying
to solve all problems in the world supporting importing of any s**t thru
loaders system.

Mighty Lamers can break all best things made by professionals. Angular2
along with TypeScript teams have to bear a lot of responsibility thinking
about what influence they made. Supporting AMD modules? SystemJS? Importing
TypeScript in import declarations instead of JavaScript?

You're lamers and you're trying to break NPM world of JavaScript magic.

On Tuesday, July 5, 2016, Asad Saeeduddin notifications@github.com wrote:

@avesus https://github.com/avesus Module resolution is a
platform/runtime concern, not a language concern. ES6 specifies a syntax
for describing and importing modules, but it does not specify a uniform
module loader or module resolution strategy that is implemented by all
environments where JS runs. While NodeJS and browsers do not yet natively
implement a module loader, TypeScript will need to emit different stuff to
support different module loaders.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2338 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AD9MI5VDAv6g3baOL4vMT69XtR0OB00zks5qSuGigaJpZM4DuGom
.

Best Regards,
Ivan Borisenko

[hdavidzhu]

@avesus I understand where you are coming from, however we don't have to get so passionate and personal over something like this. We're all just trying to make the best solutions for developers.

[mhegazy]

@avesus this is a forum for technical discussions and issue reporting for a programming language and a compiler, and not a political or social forum. The way you express your opinions and the language you have used in this thread are not inducive to a constructive discussion. If you want to contribute to this project, and have an interest in future of the TS/JS tooling, please refrain from using such language, and avoid directing insults to the community members and/or the core team.

[avesus]

I apologise for super emotional tone of course. Hope future of npm will be
better because of the very high role TypeScript developers have got and how
they can influence the future of web development.

Only one simple prayer: please be responsive and smart. Think more before
making decisions and when do explain to community.

I support the language itself and have nothing against angular or webpack
in their principles, but what they do with modularity is hell.

Sorry for the emotional tone.

On Wednesday, July 6, 2016, Mohamed Hegazy notifications@github.com wrote:

@avesus https://github.com/avesus this is a forum for technical
discussions and issue reporting for a programming language and a compiler,
and not a political or social forum. The way you express your opinions and
the language you have used in this thread are not inducive to a
constructive discussion. If you want to contribute to this project, and
have an interest in future of the TS/JS tooling, please refrain from using
such language, and avoid directing insults to the community members and/or
the core team.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2338 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AD9MI0bvI-BacJw1JVvk0EPz5sI9O8_qks5qS-jBgaJpZM4DuGom
.

Best Regards,
Ivan Borisenko

[avesus]

"Babel doesn't do type checking. To check imports you need to know their
location" - and that's great. It is necessary to invent another way to do
type checkings of imported TypeScript files (my advice is: never import
TypeScript files directly, follow the Babel's approach).

On Wednesday, July 6, 2016, Ivan Borisenko avesus8@gmail.com wrote:

I apologise for super emotional tone of course. Hope future of npm will be
better because of the very high role TypeScript developers have got and how
they can influence the future of web development.

Only one simple prayer: please be responsive and smart. Think more before
making decisions and when do explain to community.

I support the language itself and have nothing against angular or webpack
in their principles, but what they do with modularity is hell.

Sorry for the emotional tone.

On Wednesday, July 6, 2016, Mohamed Hegazy <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@avesus https://github.com/avesus this is a forum for technical
discussions and issue reporting for a programming language and a compiler,
and not a political or social forum. The way you express your opinions and
the language you have used in this thread are not inducive to a
constructive discussion. If you want to contribute to this project, and
have an interest in future of the TS/JS tooling, please refrain from using
such language, and avoid directing insults to the community members and/or
the core team.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2338 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AD9MI0bvI-BacJw1JVvk0EPz5sI9O8_qks5qS-jBgaJpZM4DuGom
.

Best Regards,
Ivan Borisenko

Best Regards,
Ivan Borisenko

[avesus]

What I advice is to compare modules idea with object files / linker system in C world. TypeScript - compiler. Webpack - linker.

If disable TypeScript modules resolution, each *.ts file will be compiled to a *.js file and that *.js file will import dependencies. It allows superfast *.ts files compilation.

The command line I use:

 tsc --watch --isolatedModules --pretty --skipDefaultLibCheck --listFiles --target es5 \
   --moduleResolution node --module commonjs --inlineSourceMap --inlineSources \
   --noResolve --jsx react --removeComments --strictNullChecks \
   --experimentalDecorators --emitDecoratorMetadata \
   --project . --outDir . --rootDir src

It is necessary to have ambient declarations, for example,

• ambient.d.ts:

declare module 'angular2/core' {
  var Component:any;
  export { Component };
}

declare module 'angular2/platform/browser' {
  var bootstrap: any;
  export { bootstrap };
}

declare var module: any;
declare var require: any;

declare interface Window {
  MSStream: any;
  webkitURL: any;
  Worker: any;
}

Declare reference in *.ts files:

/// <reference path="ambient.d.ts"/>

So, package-name/src/index.ts becomes package-name/index.js and package-name/src/lib/abc.ts becomes package-name/lib/abc.ts. And if index.ts imports abc.ts, when bundling it shouldn't be imported directly into the *.ts file but referenced from index.js.

It allows to author files written in multiple different languages within one package and export correct require()'able node exports consumable by any JavaScript code.

All I ask from community is to support this approach widely because it is base of npm modularity and success.