Skip to content
/ metapensiero.pj Public
forked from metapensiero/metapensiero.pj

Javascript for refined palates: a Python 3 to ES6 Javascript translator

License

View license
0 stars 73 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

sudip358/metapensiero.pj

BranchesTags
 
 

Repository files navigation

JavaScripthon: a Python 3 to ES6 JavaScript translator

tests status tests coverage Join the chat at https://gitter.im/javascripthon/Lobby
JavaScripthon
(image courtesy of fossBytes)

It is based on previous work by Andrew Schaaf.

author:Alberto Berti
contact:alberto@metapensiero.it
license:GNU General Public License version 3 or later

Table of Contents

Introduction

JavaScripthon is a small and simple Python 3.5+ translator to JavaScript which aims to be able to translate most of the Python's core semantics without providing a full python-in-js environment, as most existing translators do. It tries to emit code which is simple to read and check. It does so by switching to ES6 construct when possible/required. This allows to simplify the needs of polyfills for many of the expected Python behaviors.

It is designed to be the first step in a pipeline that translates your Pyhton code into something that a browser can understand. Usually it is used with tools like BabelJS and Webpack to prepare the final bundle that will be served to the browser. The steps from the source code to the bundle are the following:

  1. JavaScripthon converts your Python 3.5+ code to ES6 JavaScript modules;
  2. the BabelJS loader (configured inside Webpack or standalone) translates the ES6 JavaScript to ES5 so that the browser can understand it;
  3. Webpack parses the resulting source code and packages your source code with its dependencies by analyzing import statements and emits a bundle.js ready to be served to the browser.

Along this process the corresponding source maps are read and integrated at every step, allowing you to place breakpoints on your original Python source files when working with the developer tools of your browser.

An example of such setup is provided in the examples directory.

In addition to that, you can choose to do most these steps without using external JS tools. It comes with an embedded js interpreter that loads a standalone version of BabelJS and converts your code to ES5 JavaScript without the need to install anything else. In fact most of the the test you can find in tests/test_evaljs.py use the embedded interpreter to dual evaluate the source code (one time in Python, one time in JavaScript) and simply check that the results are the same.

Thanks to that, JavaScripthon can also be used as a server-side library to translate single functions or classes that you want your browser to load and evaluate.

The interface with the JS world is completely flat, just import the modules or use the expected globals (window, document, etc...) as you would do in JavaScript.

Brief list of the supported Python semantics

The fact that JavaScripthon doesn't reinvent the wheel by reimplementing in Python many of the features available with JavaScript translators/transpilers allows it to be lean while implementing quite a decent set of the core Python semanticts. These are, briefly:

  • Misc

    • list slices;

    • list's append();

    • dict's copy(), update();

    • len();

    • print();

    • str();

    • type(instance);

    • yield and yield from;

    • async and await;

    • import and from...import to use any JS module (see import statements);

    • callable();

    • hasattr(), getattr(), setattr();

    • template literals with tmpl('a string with ${substitution}');

    • simple Python 3.6+ f-strings (see Strings);

    • template literals and tagged_templates (see Strings);

    • names starting with d_ and dd_ will have that part replaced with $ and $$, respectively;

    • names ending with an underscore will have it removed. Useful for example with the AVA ES6 test runner which has a check named is;

    • __instancecheck__ to [Symbol.hasInstance];

    • int to parseInt;

    • float to parseFloat;

    • dictionary keys are unanbiguous when ES6 translation is enabled. For example the following code gets translated correctly:

      a = 'foo'
d = {a: 1}
print(d[a])

      prints 1 in both Python and JavaScript, while it prints undefined when translated and evaluated in JavaScript without ES6.

  • Comparisons (see section Simple stuff for the details)
    • most of the basics;
    • isinstance() and issubclass();
    • element in container for use with lists, objects, strings and the new ES6 collections like Map, Set and so on;
    • identity checks: foo is bar;
    • chained comparisons like x < y <= z;
  • Statements (see section Simple stuff and for statement for the details)
    • if...elif...else;
    • while loop;
    • for over list, over range, over plain js objects, over iterables (JS iterables);
    • try...except...finally with pythonesque behavior (see try...except...finally statement section for the details);
    • assert statement;
  • Functions (see Functions section)
    • standard functions, generator functions, async functions;
    • parameters defaults;
    • keyword parameters;
    • parameters accumulators (*args and **kwargs), with some restrictions;
    • functions in methods are usually converted to "arrow functions" (the new ES6 syntax like (foo, bar) => foo * bar;) because they automatically keep this from the enclosing scope. Appending _fn to a function declaration will force the translation to a normal function;
  • Classes (see Classes section)
    • single inheritance;
    • Exception classes for use with except statement;
    • class decorators and method decorators;
    • property descriptors;
    • special handling of property and classmethod descriptors;
    • async methods, generator methods;
    • non-function body members (i.e. member_of_class_Foo = bar);

License

This package is covered by the GNU General Public License version 3 or later. The code produced by it (i.e. the transpiled JavaScript) is your code, and you are free to choose whatever license you like. The only runtime that exists is the file snippets.py from which some utility functions are picked when necessary and transpiled together with your code. While it's distributed with the same license as the other source code, in its transpiled form will have the license you choose.

So, to summarize, the license of the this tool is GPL, but it doesn't extends to the products of this tool, on which you are free to decide.

Installation

Python 3.5 is required because Python's AST has changed between 3.4 and 3.5 and as of now supporting multiple Python versions is not one of my priorities.

To install the package execute the following command:

$ pip install javascripthon

or, if you want install it from sources:

$ git clone https://github.com/azazel75/metapensiero.pj
$ pip install -r metapensiero.pj/requirements.txt
$ pip install metapensiero.pj

Usage

To compile or transpile a python source module, use the commandline:

$ python -m metapensiero.pj source.py

or:

$ python -m metapensiero.pj -5 source.py

to transpile.

A pj console script is also automatically installed:

$ pj --help
usage: pj [-h] [--disable-es6] [--disable-stage3] [-5] [--transform-runtime]
          [-o OUTPUT] [-d] [--pdb] [-s STRING] [-e]
          [file [file ...]]

A Python 3.5+ to ES6 JavaScript compiler

positional arguments:
  file                  Python source file(s) or directory(ies) to convert.
                        When it is a directory it will be converted
                        recursively

optional arguments:
  -h, --help            show this help message and exit
  --disable-es6         Disable ES6 features during conversion (Ignored if
                        --es5 is specified)
  --disable-stage3      Disable ES7 stage3 features during conversion
  -5, --es5             Also transpile to ES5 using BabelJS.
  --transform-runtime   Add trasform runtime as plugin during transpile
  -o OUTPUT, --output OUTPUT
                        Output file/directory where to save the generated code
  -d, --debug           Enable error reporting
  --pdb                 Enter post-mortem debug when an error occurs
  -s STRING, --string STRING
                        Convert a string, useful for small snippets. If the
                        string is '-' will be read from the standard input.
  -e, --eval            Evaluate the string supplied with the -s using the
                        embedded interpreter and return the last result. This
                        will convert the input string with all the extensions
                        enabled (comparable to adding the '-5' option) and so
                        it will take some time because of BabelJS load times.

This offers many ways to test the framework, both the string conversion and the evaluation using the embedded JavaScript interpreter are very handy. For example:

$ pj -s '"foo" if True else "bar"'
(true ? "foo" : "bar");

and evaluating the same statement:

$ pj -s '"foo" if True else "bar"' -e
foo

You can even try more fancy ES6 features, like destructuring assignment:

$ pj -s "a, b, c = (2, 3, 5) \na+b+c" -e
10

Reporting Bugs

The main development repository is the one on gitlab, the one on github is just a mirror so please report issues and feature requests there.

Conversions Rosetta Stone

Here is a brief list of examples of the conversions the tool applies, just some, but not all.

Simple stuff

Most are obvious
Python JavaScript 
x < y <= z < 5
 
((x < y) && (y <= z) && (z < 5))
 
def foo():
    return [True, False, None, 1729,
            "foo", r"foo\bar", {}]
 
function foo() {
    return [true, false, null, 1729,
            "foo", "foo\\bar", {}];
}
 
while len(foo) > 0:
    print(foo.pop())
 
while ((foo.length > 0)) {
    console.log(foo.pop());
}
 
if foo > 0:
    ....
elif foo < 0:
    ....
else:
    ....
 
if ((foo > 0)) {
    ....
} else {
    if ((foo < 0)) {
        ....
    } else {
        ....
    }
}
 
str(x)
 
x.toString()
 
yield foo
yield from foo
 
yield foo
yield* foo

Then there are special cases. Here you can see some of these conversions. JavaScripthon cannot do a full trace of the sources, so some shortcuts are taken about the conversion of some core, specific Python's semantics. For example Python's self is always converted to JavaScript's this, no matter where it's found. Or len(foo) is always translated to foo.length. Albeit this an API specific of just some objects (Strings, Arrays, etc...), it is considered wide adopted and something the user may consider obvious.

The rules of thumb to treat things especially are:

  • Is it possible to think of a conversion that covers most of the use cases?
  • Is ts possible to find a convention widely used on the Python world to express this special case?
There are special cases
Python JavaScript 
==
 
===
 
!=
 
!==
 
2**3
 
Math.pow(2, 3)
 
'docstring'
 
/* docstring */
 
self
 
this
 
len(...)
 
(...).length
 
print(...)
 
console.log(...)
 
isinstance(x, y)
isinstance(x, (y, z))
 
(x instanceof y)
(x instanceof y || x instanceof z)
 
typeof(x)
 
(typeof x)
 
type(x)
 
Object.getPrototypeOf(x)
 
FirstCharCapitalized(...)
new(any_function(...))
 
new FirstCharCapitalized(...)
new any_function(...)
 
foo in bar
 
var _pj;
function _pj_snippets(container) {
    function in_es6(left, right) {
        if (((right instanceof Array) || ((typeof right) === "string"))) {
            return (right.indexOf(left) > (- 1));
        } else {
            if (((right instanceof Map) || (right instanceof Set)
                || (right instanceof WeakMap)
                || (right instanceof WeakSet))) {
                return right.has(left);
            } else {
                return (left in right);
            }
        }
    }
    container["in_es6"] = in_es6;
    return container;
}
_pj = {};
_pj_snippets(_pj);
_pj.in_es6(foo, bar);
 
foo[3:]
foo[:3]
 
foo.slice(3);
foo.slice(0, 3);
 
list(foo).append(bar)
 
foo.push(bar);
 
dict(foo).update(bar)
 
Object.assign(foo, bar);
 
dict(foo).copy()
 
Object.assign({}, foo);

for statement

The for statement by default is translated as if the object of the cycle is a list but has two special cases:

for loops
Python JavaScript notes 
for el in dict(a_dict):
    print(el)
 
var _pj_a = a_dict;
for (var el in _pj_a) {
    if (_pj_a.hasOwnProperty(el)) {
        console.log(el);
    }
}
 With this kind of loop if you use dict(a_dict, True) the check on hasOwnProperty() will not be added, so the loop will include inherited (and enumerable) properties. 
for el in an_array:
    print(el)
 
for (var el, _pj_c = 0, _pj_a = an_array, _pj_b = _pj_a.length;
      (_pj_c < _pj_b); _pj_c += 1) {
    el = _pj_a[_pj_c];
    console.log(el);
}
   
for i in range(5):
    print(i)
 
for (var i = 0, _pj_a = 5; (i < _pj_a); i += 1) {
    console.log(i);
}
   
for el in iterable(a_set):
    print(el)
 
var _pj_a = a_set;
for (var el of  _pj_a) {
    console.log(el);
}
 This will loop over all the iterables, like instances of Array, Map, Set, etc. but not over normal objects.

Functions

Functions are very well supported. This should be obvious, you can say. Really it is not so simple, if we mean functions in their broader meaning, including the async functions and generator functions.

The various types of functions at play
Python JavaScript notes 
def foo(a, b, c):
    pass
 
function foo(a, b, c) {
}
 Normal functions 
def foo(a, b, c):
    for i in range(a, b, c):
        yield i

for i in iterable(foo(0, 5, 2)):
    print(i)
 
function* foo(a, b, c) {
    for ... { // loop control omitted for brevity
        yield i;
    }
}

for (var i of foo(0, 5, 2)) {
    console.log(i);
}
 Generator functions. They return an iterable and to correctly loop over it you should use the iterable(...) call, so that the Python's for...in will be converted into a for...of 
async def foo(a, b, c):
    await some_promise_based_async
 
async function foo(a, b, c) {
    await some_promised_base_async;
}
 Async functions. They make use of the new Promise class, which is also available.

Function's args and call parameters

Parmeters defaults and keyword parameters are supported and so is *foo accumulator, which is translated into the ES6 rest expression (...foo).

The only caveat is that JS support for keyword args sucks, so you will have to remember to fill in all the arguments before specifying keywords.

On function definitions, **kwargs is supported if it's alone, i.e. without either keyword arguments or *args.

function's args and call parameters
Python JavaScript 
def foo(a=2, b=3, *args):
    pass
 
function foo(a = 2, b = 3, ...args) {
}
 
def bar(c, d, *, zoo=2):
    pass
 
function bar(c, d, {zoo = 2}={}) {
}
 
foo(5, *a_list)
 
foo(5, ...a_list);
 
bar('a', 'b', zoo=5, another='c')
 
bar("a", "b", {zoo: 5, another: "c"});
 
def zoo(e, **kwargs):
    print(kwargs['bar'])
 
function zoo(e, kwargs = {}) {
    console.log(kwargs['bar'])
}
 
zoo(4, bar=6)
 
zoo(4, {bar: 6})

Classes

Classes are translated to ES6 classes as much as they can support. This means:

  • no direct support multi-class inheritance, you have to come up with your own solution for now. Many established frameworks support this in a way or another so just use those facilities for now. I've read of some attempts, see for example the suggestion on Mozilla developer or the other about simple mixins on Exploring ES6.

  • external implementation for class-level non assignment members. Assignment members are those on the body of a class which are defined with: a_label = an_expression like:

    class Foo:

    bar = 'zoo' # or any kind of expression

    These members are removed from the translated body and submitted to a snippet of code that will run after class creation in JS land. This serves two purposes: if the value is simple, i.e. it isn't an instance of Object, it will be setup as a data descriptor, and it will work mostly like you are used to in Python. The most noticeable caveat is that it will not be accessible through the class as it is in Python, you will have to access the class' prototype, so in the case above i mean Foo.prototype.bar.

    The other purpose is to check for accessor descriptors. If the value on the right side of the assignment implements a get function, it will be installed as a property as-is, and its get and set members will be used to manage the value with the bar name.

  • external implementation for method decorators whose name is different from property or classmethod (more on these later on), because these are already supported by the ES6 class notation.

  • external implementation for class decorators. One caveat here is that the return value of the decorator has always to be a function with a prototype: unfortunately a new statement seems not to be delegable in any way. So for example a class decorator implemented like the following:

    def test_class_deco():

    counter = 0

    def deco(cls):
        def wrapper(self, *args):
            counter += 1 # side effect
            return cls(*args)
        return wrapper

    @deco
    class Foo:
        pass

    will never work. This will work instead:

    def deco(cls):
    def wrapper(self, *args):
        counter += 1 # side effect
        return cls.prototype.constructor.call(self, *args)
    wrapper.prototype = cls.prototype
    return wrapper

    So either return the original class or setup the wrapper appropriately.

Methods can be functions or async-functions although the latters aren't officially supported yet by the JavaScript specification. You can disable them adding a --disable-stage3 to the command line utility.

Python`s super() calls are converted accordingly to the type of their surrounding method: super().__init__(foo) becomes super(foo) in constructors.

Functions inside methods are translated to arrow functions so that they keep the this of the surrounding method.

@property and @a_property.setter are translated to ES6 properties.

Methods decorated with @classmethod are translated to static methods.

Special methods __str__ and __len__ are translated to toString() method and get length() property, respectively.

Arrow method expression to retain the this at method level aren't implemented yet.

Classes
Python JavaScript 
class Foo(bar):
    def __init__(self, zoo):
        super().__init__(zoo)


    def meth(self, zoo):
        super().meth(zoo)
        def cool(a, b, c):
            print(self.zoo)


    async def something(self, a_promise):
        result = await a_promise

    def generator_method(self):
        yield something

    @property
    def foo(self):
        return self._foo


    @foo.setter
    def foo(self, value):
        self._foo = value


    @classmethod
    def bar(self, val):
        do_something()


    def __len__(self):
        return 1


    def __str__(self):
        return 'Foo instance'
 
class Foo extends bar {
    constructor(zoo) {
        super(zoo);
    }

    meth(zoo) {
        super.meth(zoo);
        var cool;
        cool = (a, b, c) => {
            console.log(this.zoo);
        };
    }

    async something(a_promise) {
        var result;
        result = await a_promise;
    }

    * generator_method() {
        yield something;
    }

    get foo() {
        return this._foo;
    }

    set foo(value) {
        self._foo = value;
    }

    static bar(val) {
        do_something()
    }

    get length() {
        return 1;
    }

    toString() {
        return "Foo instance";
    }
}

Only direct descendants of Exception are threated especially, but just for them to be meaningful in JS land and to be detectable with instanceof in catch statements.

Exceptions
Python JavaScript 
class MyError(Exception):
    pass

raise MyError("An error occurred")
 
function MyError(message) {
    this.name = "MyError";
    this.message = (message || "Custom error MyError");
    if (((typeof Error.captureStackTrace) === "function")) {
        Error.captureStackTrace(this, this.constructor);
    } else {
        this.stack = new Error(message).stack;
    }
}
MyError.prototype = Object.create(Error.prototype);
MyError.prototype.constructor = MyError;
throw new MyError("An error occurred");

try...except...finally statement

The conversion of this statement is mostly obvious with the only exception of the except part: it translates to a catch part containing one if statement for each non catchall except. If a catchall except isn't present, the error will be re-thrown, to mimic Python's behavior.

try...catch...finally statement
Python JavaScript 
try:
    foo.bar()
except MyError:
    recover()
except MyOtherError:
    recover_bad()
finally:
    foo.on_end()
 
try {
    foo.bar();
} catch(e) {
    if ((e instanceof MyError)) {
        recover();
    } else {
        if ((e instanceof MyOtherError)) {
            recover_bad()
        } else {
            throw e;
        }
    }
} finally {
    foo.on_end();
}

import statements

import and from ... import statements are converted to ES6 imports, and the declaration of an __all__ member on the module top level is translated to ES6 named exports.

import and exports
Python JavaScript 
import foo, bar
import foo.bar as b
from foo.bar import hello as h, bye as bb
from ..foo.zoo import bar
from . import foo
from .foo import bar

from foo__bar import zoo

from __foo.zoo import bar

from foo import __default__ as bar

from __globals__ import test_name

# this should not trigger variable definition
test_name = 2

# this instead should do it
test_foo = True

__all__ = ['test_name', 'test_foo']
__default__ = 'test_name'
 
var test_foo;

import * as foo from 'foo';
import * as bar from 'bar';
import * as b from 'foo/bar';
import {hello as h, bye as bb} from 'foo/bar';
import {bar} from '../foo/zoo';
import * as foo from './foo';
import {bar} from './foo';

import {zoo} from 'foo-bar';

import {bar} from '@foo/zoo';

import bar from 'foo';

test_name = 2;
test_foo = true;

export {test_name, test_foo};
export default test_name;

About JS default export and import

If you want to export something as default export in your modules, declare a __default__ member and assign to it the string of the symbol you want to export. To clarify:

foo = 42
bar = "hello"

__all__ = ['foo', 'bar']  # foo and bar will be exported as named exports
__default__ = 'bar'  # bar will also be exported as the *default*

This becomes:

var bar, foo;

foo = 42;
bar = "hello";

export {foo, bar};
export default bar;

For what concerns the import, you can import the default export of a module using the default name, as defined by the ES6 spec. However, as there were some issues reported to me with bundlers not supporting the named import of the default export, a special import statement using __default__ as name has been added that directly translates to the more common form of ES6 default import. So:

from foo import default as bar
from foo import __default__ as zoo

Translates to:

import {default as bar} from 'foo';
import zoo from 'foo';

The two imports should work the same, see exploring js section and the linked spec. But if you encounter problems with the former use the latter instead. Keep in mind that you cannot mix the __default__ import with others (i.e. it needs to be on a line of its own) and that you always need to specify an ... as name ... part.

Strings

Javascripthon supports converting Python 3.6+ f-strings to ES6 template literals. The expression in the braces gets converted, but neither conversion nor format_spec are supported: f"Value of {a}" becomes `Value of ${a}` and f"Value of {self.foo}" becomes `Value of ${this.foo}`.

You can also write raw template literals by using the function tmpl() it does only a conversion of the string markers, from those used in Python's literal string notation to template literal notation.

There is also the way to express tagged templates, template literals that are parsed using a provided function. This is done by using the function __. So for example:

__('A template ${string} with foo', bar)

gets translated to:

bar`A template ${string} with foo`

bar will be executed with the value of ${string} as a parameter, see the link for template literals for help.

Verbatim JS

You can intermix Python and JS by using the JS('Your-JS-code-here') marker function. It will not be touched by the ES6 transcoder but if you choose to also transpile it to ES5, il will be considered by Babel.

Examples

Execute make inside the examples directory.

Testing

To run the tests you should run the following at the package root:

python setup.py test

How to contribute

So you like this project and want to contribute? Good!

These are the terse guidelines:

There are some TODO points in the readme, or even the issue #6 is
quite simple to fix. Feel free to pick what you like.

The guidelines are to follow PEP8 for coding where possible, so use
CamelCase for classes and snake_case for variables, functions and
members, and UPPERCASE for constants.

An exception to this rules are the function names inside
``metapensiero.pj.transformations`` subpackage. Those are matched
against names of the AST objects coming from the ``ast`` module in
standard lib, so they have to to match even in case.

Try to keep lines lengths under 79 chars, more or less ;-)

The workflow is to fork the project, do your stuff, maybe add a test
for it and then submit a pull request.

Have fun

Contributing

Any contribution is welcome, drop me a line or file a pull request.

External contributions

Todo

This is a brief list of what needs to be done:

  • refactor the comprehensions conversion to use the snippets facility;
  • refactor snippets rendering to write them as a module and import them in the module when tree conversion is enabled;
  • convert dict() calls to ES6 Map object creation;
  • convert set literals to ES6 Set objects. Also, update "foo in bar" to use bar.has(foo) for sets;

Done

Stuff that was previously in the todo:

  • translate import statements to ES6;
  • translate __all__ definition to ES6 module exports;
  • write a command line interface to expose the api;
  • make try...except work again and implement try...finally;
  • convert async and await to the same proposed features for js (see BabelJS documentation);
  • convert argument defaults on functions to ES6;
  • convert call keyword arguments;
  • convert *iterable syntax to ES6 destructuring;
  • use arrow functions for functions created in functions;
  • properties to ES6 properties (getter and setter);
  • take advantage of new duckpy features to use a JS execution context that lasts multiple calls. This way the BabelJS bootstrap affects only the initial execution;
  • class and method decorators;
  • implement yield, yield from and generator functions;
  • update "foo in bar" to use bar.has(foo) for maps;

External documentation

A good documentation and explanation of ES6 features can be found on the book Exploring ES6 by Axel Rauschmayer (donate if you can).

An extensive documentation about Python's AST objects, very handy.

Tools

Have a look at ECMAScript 6 Tools by Addy Osmani.

To debug source maps have a look at source-map-visualization and its package on npm.

Still i found these links to be helpful:

Here is an example of the latter tool showing code generated by JavaScripthon, have fun!

Notes

  • A post about proposed solutions to use ES6 classes with Backbone. See also the bug open on github.
  • A story about ES6 crazyest stuff... symbols

About

Javascript for refined palates: a Python 3 to ES6 Javascript translator

Resources

Readme

License

View license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

10 tags

Packages

No packages published

Languages

  • Python 96.4%
  • JavaScript 3.4%
  • Makefile 0.2%