pyc-ng - creating serializations for modules that lends itself well to memory mapping and lazy loading

[RonnyPfannschmidt]

i want to propose creating a opt-in feature for python, that makes a module whose attributes are all lazy computed (like lazy computed type annotations

that way api defining modules as well as quite a number of utility modules would be able to defer their execution similar to how for example the mercurial scm manages delayed imports to ensure quick startup for commands

additionally those "lazy modules" could possibly handle some degrees of import cycles for typing and other details easily

for example the importation of one "lazy type" into another lazy module could still defer the making of the type until it is actually used

in addition all code objects could ideally just use buffers into a memory map instead of reading strings into the normal heap
which would enable sharing a lot more of a python module across multiple processes

[gvanrossum]

Lazy modules are an old idea. I'm not sure what your point is with "lazy types". When your shift key is fixed can you perhaps walk us through an example?

[RonnyPfannschmidt]

The idea would be to apply what https://www.python.org/dev/peps/pep-0649/ does for annotations to all top-level declarations of a participating module
with the caveat that decorators/class decorators may either be deferred as well and/or extra support for them has to be added

i hope to complete a larger example later today

[ericsnowcurrently]

@nascheme worked on something exactly like this a few years back

[gvanrossum]

I think @nascheme explained it here: #32 (comment)

The idea also reminds me of Instagram/Cinder's Strict Modules: https://github.com/facebookincubator/cinder#strict-modules