Refactor frontend pt. 1

[tbekolay]

This PR is similar to #673 in that it refactors a relatively large part of the codebase, but this time it's all frontend stuff. Frontend meaning anything that the browser touches (mostly JavaScript, but also CSS and HTML). This is a work in progress -- I haven't even implemented anything yet, but since this PR will involve introducing some new tools / concepts, and frontend development moves fast and none of us are on the cutting edge, I'm making the PR now so I can document the process in addition to doing the actual refactoring.

I'll be posting updates frequently throughout the next week or two or however long this takes. If this sounds annoying to you, keep in mind that you can mute individual issues / PRs with this little button on right side of this page:

That's not the actual button, but that's what it looks like. It's over on the right. Just saying.

[tbekolay]

Webpack

Webpack is the first tool that I'm looking into. Everyone that uses it seems to love it, but the documentation is really rough. After looking into this in some detail today, webpack seems well suited for our current situation as we have a ton of assets that get loaded in page.html. With webpack, we will have two -- one css file and one js file. While loading multiple assets is not a huge deal when loading them locally, this will make a hug difference in page load times when accessing a remote server.

Link dump for future me, and for those looking to learn more about webpack:

• 5 minute introduction to webpack: This is the most clear, practical introduction that I've seen; worth taking a look if you only have 5 minutes to spare.
• 30 minute talk + 10 minutes of questions on how instagram.com works: (spoiler: it uses webpack) This is a pretty decent talk that motivates why something like webpack is needed; not many details on how to use it in the talk itself, but it's worth watching for the motivation alone.
• webpack-howto: A practical how-to companion piece to the instagram.com talk. Definitely one of the better tutorials I came across.
• Comprehensive webpack intro: Another intro tutorial that looks good. Haven't read this fully yet.
• webpack docs: For completeness' sake, and so you can see for yourself how confusing the docs are. My favorite part: there's a disqus comment thread on each page, and it's filled to the brim with people complaining about the docs.

Modules

The point of webpack is to manage modules, where modules might refer to JS files, CSS, or something that transpiles to JS, CSS, etc. Unfortunately there is not one but three standards for how to define modules: CommonJS, AMD, and ES6 (a.k.a ECMA Script 6, a.k.a. ECMA Script 2015) modules.

The consensus I got early on was that CommonJS is the one to use, even though ES6 will be incorporated in browsers eventually. Unfortunately, TypeScript (which, at the moment, I think we should use) uses ES6 modules, and webpack doesn't support ES6 module natively yet (though it will in webpack 2, whenever that gets released).

So, at the moment, I'm thinking I'll have to know both CommonJS and ES6 syntax. Hopefully this will be a temporary thing and not the permanent state of affairs, but neither syntax is too complicated, so whatever!

• ES6 module syntax: Clear guide; also includes best practices, which is helpful. From the TypeScript documentation.
• CommonJS module syntax: Another clear guide; includes some performance discussion also.
• Using ES6 modules with TypeScript and webpack: Some information about using ES6 modules with webpack.

---

That's all for today; I think I'm getting webpack enough to start actually implementing stuff tomorrow. The biggest decisions that I will have to mull over will be: Where should webpack.config.js live? Where should the JS / CSS modules live, should they be moved to simplify the directory structure?

Bonus link for those who feel like raw JS is fine: ES6 syntax cheatsheet, which we would get access to through TypeScript. Default function arguments!

[tbekolay]

Node Package Manager

Okay, I need to introduce one more tool before I actually start making code changes. With this, however, I think we'll be able to radically simplify the nengo_gui file tree. It will have no impact on the end user, but a few additional steps for developers. But they should be pretty straightforward for anyone who's used to pip.

The JavaScript ecosystem is weird. The majority of JavaScript is run in people's browsers, through whatever JS engine is used by the browser; Firefox has SpiderMonkey, Safari has JavaScriptCore, and Chrome has V8. Since these JS engines are super-mega-optimized so that people get their webpages 20 ms faster (and get to see 8 more ads), and some of the are open source, someone had the idea to add a few things to the V8 JavaScript engine and make JavaScript a Legit Programming Language that you can run on the desktop, without a browser. The desktop usage of JavaScript is under the node.js banner.

Being a Legit Programming Language means that you can do general purpose stuff like make files and FTP clients and all that kind of stuff. An ecosystem of node.js packages spring up, and they want to use those packages to make new packages, so you need a way of managing JS packages. From that comes the Node Package Manager (npm) and CommonJS modules to interact with packages installed with npm.

npm and pip have a lot in common; they solve the same basic problem of managing metadata about packages, resolving dependencies, and interacting with package repositories (e.g., pypi, npmjs.com). However, there are a few key differences which are worth pointing out for those that are familiar with pip:

1. While you can install node packages globally (with npm install -g) there is no "user install"; instead, npm is designed to be package-specific (kind of like using virtual environments in Python). That means that for most node.js projects, you git clone and then npm install. This gfives you a node_modules directory which contains all of the node project's dependencies. Even if you're in a subdirectory of node-project, npm will go up the directory tree to find node_modules, meaning that if you do npm install some-other-project anywhere in the node-project tree, it will install to node-project/node_modules.
2. The metadata associated with a node package is represented in JSON format, not an executable script like in Python. The npm equivalent of setup.py is the package.json file, which contains metadata and a list of dependencies. As we'll see later, webpack uses a JavaScript file for its configuration, like pip/setuptools does.

There are other differences, like explicitly tracking dependencies and developer dependencies, but we don't have to concern ourselves with those differences very much. We will only have developer dependencies, for example.

Oh, one last thing to mention about the oddities of the JavaScript ecosystem. As I mentioned at the start, most JS is executed in the browser, yet node.js is designed for the desktop and managing desktop JS packages. Well, since it ended up being the de facto way to deal with JavaScript packages, JS packages that should only ever be used in the browser (e.g., Bootstrap, Ace editor, D3.js) are also installable through npm. Which is basically the reason that we're using npm.

NPM + webpack

One thing that might not be clear at this point is the relationship between NPM and webpack. Their relationship is somewhat like the relationship between pip and setuptools, though I think the responsibilities are more clear cut with NPM and webpack. Put simply:

• NPM is a package manager that tracks nengo GUI's frontend dependencies and lets us install them from trusted repositories rather than keeping them in our directory tree.
• webpack is a build tool that takes all of the frontend dependencies and our own modules (i.e. JS and CSS files) and packages them up efficiently to be the files that we end up using in nengo GUI.

Once we use these tools, the experience for the end-user will be unchanged, aside from faster startup times and (hopefully) less buggy code. For developers, the main change will be an additional step when first setting up nengo GUI for development. Instead of just git clone and python setup.py develop, you will now also need to do npm install.

Note that since setup.py can run arbitrary code, we can choose to automatically run npm install whenever python setup.py develop in invoked. However, I think that we should probably not do that, as people will have to install node.js and npm if they don't already have that installed, and we can't reasonably write something to install those tools on an arbitrary machine. Better to put it in the README and point people to the right place; installing node.js has been trivial in all the machines I've tried, and npm comes with node.js.

[jgosmann]

If python setup.py develop requires npm install, is that also the case for python setup.py install? What about installing nengo_gui with pip?

[jgosmann]

On the dangers of relying on npm. ;)

But in all seriousness, it probably makes sense to use it.

[tbekolay]

is that also the case for python setup.py install?

Basically, if you're changing any of the frontend files (JS, CSS) then you'll need to do npm install. While it is a build artifact, for simplicity I think it makes sense to commit the output of the webpack build, so unless you're changing frontend files and therefore needing to run a webpack build, you shouldn't need to use npm.

On the dangers of relying on npm.

Indeed, we should also stop using pip, apt, conda, etc etc and deliver all code printed out on reams of paper ;)

[tbekolay]

Okay, so the update for this afternoon is that I've resolved all of our dependencies through npm. They all exist in the node_modules directory in a form that we can use them. I had hoped that this would be enough to get the GUI in a workable state by symlinking the right files, but the npm versions include require statements that work in node.js but not in a browser. On the bright side, now I know what browserify is for, and why it's named "browserify"! Of course, webpack is a more general and more powerful tool than browserify, but still an interesting aha! moment.

Aside from that fact that I'll need to do the webpack stuff before I can get the GUI in a workable state again (about half done at this point), there was one other issue that I ran into today: some of the versions of JS packages that we're using aren't installable with npm, though later versions are.

For now, I'm depending on the updated versions that are installable with npm to see if they work; if they don't, then we can put the old versions back in this repo.

Specifically, here are the forced upgrades I've done so far:

• ace editor: I can't tell what version we were using (doesn't seem to be in the minified source at all), but now we're using 1.2.2
• jqueryfiletree: 1.0.1 -> 2.1.4
• jquery-ui: 1.9.2 -> 1.10.4

Note that unlike all the other packages, jqueryfiletree is not available through npmjs.com, so I'm using an npm feature that installs packages directly from Github. By default, npm tries to do this through SSH (with a git:// URL), which doesn't work for me as I haven't uploaded SSH keys to Github. If that's the case for you as well, setting the following git configuration options fixed the issue for me, and forces npm to clone through https instead:

git config --global url."https://".insteadOf git://
git config --global url."https://github.com/".insteadOf git@github.com:

Finally, for interest, here's the package.json that I've got going so far. I don't think this will change much (unlike my half-finished webpack.config.json):

{
  "name": "nengo",
  "version": "0.2.1",
  "description": "Frontend components for Nengo GUI",
  "author": "Nengo developers",
  "license": "Free for non-commercial use",
  "main": "nengo_gui/www/nengo.js",
  "repository": {
    "type": "git",
    "url": "https://github.com/nengo/nengo_gui.git"
  },
  "bugs": {
    "url": "https://github.com/nengo/nengo_gui/issues"
  },
  "private": true,
  "scripts": {
    "build": "webpack"
  },
  "devDependencies": {
    "ace-builds": "^1.2.2",
    "bootstrap": "^3.3.4",
    "bootstrap-validator": "^0.8.1",
    "d3": "^3.5.3",
    "interact.js": "^1.2.4",
    "jquery": "^2.1.3",
    "jquery-ui": "^1.10.4",
    "jqueryfiletree": "jqueryfiletree/jqueryfiletree#2.1.4",
    "webpack": "^1.13.1"
  }
}