Python import functionality for py5 imported mode code

[hx2A]

Intro

py5's "imported mode" code allows coders to write py5 code that omits the py5. prefix before accessing py5's functionality.

py5 has some special code to support "imported mode" that involves some Python AST transformations. You can execute "imported mode" code using the run_sketch command line utility, with the Thonny plugin, or with the py5 Jupyter Notebook kernel.

Up until now, all of your "imported mode" code had to be in one file. This is a bit limiting for new coders who are ready to start using multiple files. A py5 enhancement idea from @villares was to allow imported mode code to import other Python packages that also contain "imported mode" code.

I have implemented a prototype of this new feature and would like feedback on some of the design decisions.

Demo

Here's how it works.

Let's say you have this "imported mode" code:

import test1

def setup():
    size(200, 200)

def draw():
    test1.draw_squares()

run_sketch()

In test1.py we have:

# test1.py
# PY5 IMPORTED MODE CODE

def draw_squares():
    rect(mouse_x, mouse_y, 10, 10)
    rect(random_int(width), random_int(height), 10, 10)

Note the single line comment # PY5 IMPORTED MODE CODE. This is a marker of sorts that flags the code as imported mode code. The custom import hook will make sure this is imported correctly and will ensure the dynamic variables mouse_x, etc work (this was the biggest technical challenge for this feature).

The marker is necessary to make sure that the custom import hook does not interfere with other non-py5 packages. Note we can't just assume that Python code in the current working directory or not in site-packages is imported mode code. Among other things, a user might be building a Sketch that incorporates code they downloaded from github or elsewhere.

Installing the Test Build

I created a test environment with Anaconda & installed the test build with the below terminal commands. The last line will install the test package from a test github branch.

conda env create -n py5test -f http://py5coding.org/files/install/py5_environment.yml
conda activate py5test
pip install git+https://github.com/hx2A/py5_fork.git@import_test#egg=py5

Here's some more information.

In addition to single file imports, you can import a small "module" of imported mode code. Let's say I create a directory called test2 and add files __init__.py and code.py that contain the following:

# test2/__init__.py
# PY5 IMPORTED MODE CODE

from . import code

# test2/code.py

def draw_squares():
    rect(mouse_x, mouse_y, 10, 10)
    rect(random_int(400), random_int(400), 10, 10)

That can be run with:

import test2

def setup():
    size(200, 200)

def draw():
    test2.code.draw_squares()

run_sketch()

In a multi-file module of imported mode code, there is no need to add that marker to every single file. It should only need to go in the top-level __init__.py file, as you see in the above example. All of the submodules will get the same special handling.

Questions

What do you think of the marker # PY5 IMPORTED MODE CODE?

Alternative and not implemented design ideas are:

import numpy as np

# ACTIVATE PY5 IMPORTED MODE IMPORTS
import test1

def setup():
    size(200, 200)

def draw():
    test1.draw_squares()

run_sketch()

Here, the # ACTIVATE PY5 IMPORTED MODE IMPORTS marker would mean that everything after the marker must be imported mode code and everything before it is not. Users will need to sort their imports appropriately. With this idea, there's no need to add a marker # PY5 IMPORTED MODE CODE to the code that is imported.

Another idea is to leverage the Thonny editor's ability to "know" what code the user is writing with its editor. Thonny allows you to open multiple tabs of code (correct?), much like the PDE. Possibly the Thonny pluggin can pass a list of files and directories to the run_sketch utility, like this:

run_sketch sketch.py --imports test1.py test2/

I have no idea how feasible this is but it might work and I wanted to throw it out there. @tabreturn , what do you think of this? If this could work, there would be no need for markers in any files and this would all be seamless to the users.

This last idea wouldn't work at all for the py5 Jupyter Notebook kernel. We don't have to go with only one idea here though. The special import hook can be activated more than one way.

[hx2A]

@villares previously talked to me about a 2x2 matrix of "mixed-mode" code importing each other. Here is a discussion of the 4 permutations:

"imported mode" code importing code using "imported mode"

This is what is implemented with the described enhancement. It works great!

"imported mode" code importing code using "module mode"

This doesn't work, and for technical reasons, can't work. The issue has to do with the dynamic variables like mouse_x, etc. Both "imported mode" and "module mode" code use special machinery for the dynamic variables to work. That special machinery is not the same for those two modes and they are not compatible.

Note that if you attempt this the import command will work but when the Sketch is run, if it uses the dynamic variables, the user will get strange errors. I don't think there is anything I can do to protect against this outcome, except perhaps somehow detect and disallow importing "module mode" code into an "imported mode" sketch. This protection is probably possible. Should I implement it?

"module mode" code importing code using "module mode"

This has always worked in py5.

"module mode" code importing code using "imported mode"

This doesn't work, but in my tests, seems like it could work if we wanted it to. Should it??? Given that "imported mode" code importing code using "module mode" can't work, it seemed to me that perhaps it would be confusing to coders if this was allowed and the reverse was not. Feel free to disagree with me on this point. If you want to try it, use this code:

from py5_tools import import_hook
import_hook.activate_py5_import_hook()

what about "class mode"?

Interfering with @villares 's neatly organized 2x2 matrix is "class mode". "Class mode", however, actually provides a clean approach to writing importable code that works for all modes, addressing the limitations from above. Class mode doesn't employ any special machinery for the dynamic variables mouse_x, etc. Consider the following function:

import py5

def draw_squares(size, *, sketch: py5.Sketch=None):
    s = sketch or py5.get_current_sketch()
    s.rect(s.mouse_x, s.mouse_y, size, size)
    s.rect(s.random_int(s.width), s.random_int(s.height), size, size)

That sketch param is optional, and in this example, is a keyword only param. When it is not used, it gets the current sketch from py5. It can be used in an "imported mode" Sketch like this:

import test5

def setup():
    size(200, 200)

def draw():
    test5.draw_squares(10)

It can also be used by coders using static mode, module mode, and class mode. This is the best way to write py5 code that is universally importable by everyone.

Bottom line, if you want to write code that coders programming in "imported mode" and "static mode" can use, you should write that code in either "imported mode" or you should use the approach outlined above that uses class mode.

[villares]

This all looks fantastic, very good indeed!

I would only argue that "imported mode" code importing code using "module mode" kind of works already, with the limitation that you can't use the dynamic variables, well you can if you know how to, but anyway, it can be quite useful, take this example, my helper function to draw shapely objects. I can import this both in module mode and imported mode sketches:

def draw_shapely(shp):
    """Tries to draw a shapely object with py5."""
    from shapely import Polygon, MultiPolygon
    from shapely import LineString, MultiLineString
    from shapely import Point, MultiPoint
    from py5 import begin_closed_shape, begin_shape, begin_contour
    from py5 import  vertex, push_style, no_fill, point
    
    if isinstance(shp, (MultiPolygon, MultiLineString, MultiPoint)):
        for p in shp.geoms:
            draw_shapely(p)
    elif isinstance(shp, Polygon):
        with begin_closed_shape():
            for x, y in shp.exterior.coords:
                vertex(x, y)
            for hole in shp.interiors:
                with begin_contour():
                    for x, y in hole.coords:
                        vertex(x, y)
    elif isinstance(shp, LineString):
        with push_style():
            no_fill()
            with begin_shape():
                for x, y in shp.coords:
                    vertex(x, y)
    elif isinstance(shp, Point):
        point(*shp.coords[0])
    else:
        print(f"Unable to draw: {shp}")

I guess I could use something "almost class mode" if I needed the dynamic variables, like this:

def draw_circle_on_canvas_center():
    import py5
    s = py5.get_current_sketch()
    py5.circle(s.width / 2, s.height / 2, s.width / 20)

And then, for full "class mode" I should adopt the strategy shown:

def draw_circle_on_canvas_center(sketch: py5.Sketch=None):
    import py5
    s = sketch or py5.get_current_sketch()
    s.circle(s.width / 2, s.height / 2, s.width / 20)

Other comments

• I don't see much sense in importing "imported mode" code in a "module mode" sketch, but maybe this is just a prejudice.

• I like the idea of the marker comments on the [to be] imported module, I wondered if it should mimic the "hashbang" (#!) form used to indicate the Python interpreter... I then thought if a #! could be used to trigger the sketch_runner command line tool also, maybe all that would be too confusing...

• I have no strong arguments, but I don't think using Thonny's py5mode plugin and open tabs status would be a robust solution (it would be easy to close some tabs and mess things up).

• Could the marker of the not implemented idea (activating behavior on the importing file) be something like this?

  import test1 # PY5 IMPORTED MODE

[hx2A]

This all looks fantastic, very good indeed!

Thank you!!

I would only argue that "imported mode" code importing code using "module mode" kind of works already, with the limitation that you can't use the dynamic variables

OK, so I won't do anything to restrict it. However, when I document this new feature, I am going to talk about importing imported mode code into imported mode sketches and how to write code that is universal across all modes. I don't want to confuse anyone with extra information that will work only if they don't use the dynamic variables.

Side note: I like your draw_shapely function and one day will use it for a new convert_shape() method, which takes various inputs and converts it to a Py5Shape object. This would be analogous to convert_image(), which can convert various inputs to a Py5Image object.

I don't see much sense in importing "imported mode" code in a "module mode" sketch, but maybe this is just a prejudice.

I agree. I won't concern myself with why it doesn't work.

I have no strong arguments, but I don't think using Thonny's py5mode plugin and open tabs status would be a robust solution (it would be easy to close some tabs and mess things up).

Yes, I was afraid those kinds of problems might occur, but thought perhaps there is something I don't know about Thonny that could address that concern.

Could the marker of the not implemented idea (activating behavior on the importing file) be something like this?

import test1 # PY5 IMPORTED MODE

Theoretically, but it would work differently. There would have to be some kind of pre-processing that would convert that line to something like this:

py5_tools.imported_mode_import('test1')

This would be harder to add to the py5 kernel, and who knows how it would work if user wanted something like from test1 import function1. It also might have robustness problems. I think we should stick with the approach used in the prototype.

Have you had a chance to test this and experiment with the feature? I was hoping you would have some time to test it.

I was thinking of doing a release after this feature is finished and maybe include a few other small changes. Smaller releases are easier to manage.

[villares]

I'll try to test it a bit more! I loved the conda env install instructions, thank you!

[hx2A]

@villares , you mentioned on mastodon about using the hashbang #! form for the marker. That gave me the idea to test the following as a runable script:

#!/home/jim/INSTALL/anaconda3/envs/py5/bin/run_sketch

import test5
from test3 import code


def setup():
    size(200, 200)


def draw():
    code.draw_squares()

That, plus chmod +x sketch.py to change the permission, will be a runable file.

On my computer I needed the full run_sketch path for it to work. I'm not sure why, but that might be fixable.

[Maltiec]

I'll try this env as soon as I find some free time. I prefer working in Spyder, so it will be another sort of testing. )

[hx2A]

Thank you @Maltiec, your feedback will be much appreciated!

[hx2A]

I've been testing this and uncovered a small bug / shortcoming related to code validation in imported mode code. Currently there are "py5 reserved words" such as rect, mouse_x, etc and you are not allowed to define variables that use these keywords. However, I noticed I could define new functions with names that use the reserved words. The user should get a warning here also. Now the user will see this:

ImportError: There are 3 problems with the code in /Users/jim/Projects/ITP/pythonprocessing/py5projects/importing/test3/code.py.
====================================================================================================================
Defining a function named after py5 reserved word "mouse_x" on line 17 is discouraged and may causes errors in your sketch.
def mouse_x():
^
Defining a function named after py5 reserved word "square" on line 21 is discouraged and may causes errors in your sketch.
def square():
^
Defining a function named after py5 reserved word "vertex" on line 25 is discouraged and may causes errors in your sketch.
def vertex():
^

Other than that, this new feature is pretty solid. I am going to do a bit more testing and then merge this branch.

[hx2A]

I've been testing this and uncovered a small bug / shortcoming related to code validation in imported mode code. Currently there are "py5 reserved words" such as rect, mouse_x, etc and you are not allowed to define variables that use these keywords. However, I noticed I could define new functions with names that use the reserved words. The user should get a warning here also. Now the user will see this:

ImportError: There are 3 problems with the code in /Users/jim/Projects/ITP/pythonprocessing/py5projects/importing/test3/code.py.
====================================================================================================================
Defining a function named after py5 reserved word "mouse_x" on line 17 is discouraged and may causes errors in your sketch.
def mouse_x():
^
Defining a function named after py5 reserved word "square" on line 21 is discouraged and may causes errors in your sketch.
def square():
^
Defining a function named after py5 reserved word "vertex" on line 25 is discouraged and may causes errors in your sketch.
def vertex():
^

Other than that, this new feature is pretty solid. I am going to do a bit more testing and then merge this branch.

[villares]

While you are working with these warnings, could you please disable the interdition of attributing to g?

[hx2A]

You mean changing py5 so that g is no longer a py5 reserved word?

That is a reasonable, single character reserved words can be problematic.

[hx2A]

@villares , just made this change

[villares]

Thanks , it is common in class to make RGB examples using r, g & b variables.

[hx2A]

OK! I have finished the documentation for this feature as well as all of the documentation changes for this release. You can read the documentation for this feature here:

http://dev.py5coding.org/content/importing_py5_code.html

While writing this documentation, I made two changes. First, the marker # PY5 IMPORTED MODE CODE is now case insensitive. Second, users will get a helpful error message if py5 detects what looks to be py5 Imported Mode code that was not imported correctly:

File "/tmp/test/helper_functions.py", line 13, in draw_squares
    12   def draw_squares():
--> 13       rect(mouse_x, mouse_y, 10, 10)
    14       rect(random_int(width), random_int(height), 10, 10)

NameError: The name "rect" is not defined. Your Sketch is also running in Imported Mode.
If the code throwing this exception was imported into your main py5 Sketch code, please
ensure the py5 Imported Mode marker "# PY5 IMPORTED MODE CODE" has been properly added
to the module.

You can test the release candidate using the same commands as before:

conda env create -n py5test -f http://py5coding.org/files/install/py5_environment.yml
conda activate py5test
pip install git+https://github.com/hx2A/py5_fork.git@import_test#egg=py5

[hx2A]

I made a small code change for #314 and pushed a second release candidate to the same branch. Test it with this:

conda env create -n py5test -f http://py5coding.org/files/install/py5_environment.yml
conda activate py5test
pip install git+https://github.com/hx2A/py5_fork.git@import_test#egg=py5

This is a small change and I am confident I didn't break anything with this but to be safe I will leave some extra time to do the release so myself and others can test.

[villares]

Looks good to me!

[hx2A]

Awesome, thank you @villares for installing and testing!