gh-104240: make _PyCompile_CodeGen support different compilation modes

[iritkatriel]

With this PR we can do this:

from _testinternalcapi import compiler_codegen, optimize_cfg, assemble_code_object
import ast
import opcode
import textwrap
import types

a = ast.parse("(x+y)/2", mode='eval')
filename = "myfile.py"
eval_mode = 1  # 0=exec, 1=eval, 2=single

insts = compiler_codegen(a, filename, 0, eval_mode)

consts = list(range(5))
insts = optimize_cfg(insts, consts)

metadata = {   
    'name'     : 'avg',
    'qualname' : 'stats.avg',
    'consts'   : [2],
    'argcount' : 0,
    'names'    : {'x' : 0, 'y' : 1}, 
    'varnames' : {},
}

from test.test_compiler_assemble import IsolatedAssembleTests
IsolatedAssembleTests().complete_metadata(metadata)

co = assemble_code_object(filename, insts, metadata)

for x,y in [(3, 4), (-100, 200), (10, 18)]:
    f = types.FunctionType(co, {'x': x, 'y': y}) 
    print(f'avg of {x} and {y} is {f()}')

• Issue: combine compilation steps in a pipeline #104240

[carljm]

I'm a bit confused by this; how does this not leave us with one fewer compiler_exit_scope than we had previously? We still have the same number of compiler_enter_scope (it's just moved from above into both callers of compiler_codegen), and both callers still do the same number of compiler_exit_scope that they did before (1), but we eliminate this compiler_exit_scope. Why?

[iritkatriel]

I think currently we may have one too many - one exit here and then we return to the caller which exits again.

[iritkatriel]

Actually I'm wrong in the case of compiler_mod, I'll fix it there.

[carljm]

What is the purpose of moving this from here out into the (only) two callers of this function? It duplicates this line of code, but it doesn't seem like it would make any other difference, AFAICT.

[iritkatriel]

The purpose is to be able to make a distinction between compiler_enter_scope failed, in which case we don't need to exit the scope, and the case where compiler_enter_scope succeeded but compiler_codegen failed later, in which case we do need to exit the scope. I don't think this is currently correct in all cases in main.

[carljm]

That makes sense! Beyond adding the compilation modes option, I wasn't clear where this PR was aiming to maintain behavior vs fixing problems in existing behavior.

Looking at the definition of compiler_exit_scope, it seems like if we have one too many it would leave things in an observably broken state (like too small c_nestlevel?) Is there somewhere we can add asserts to show that things are broken today, but not after this fix?

Also, it seems like it would be simplest to get the enter/exit logic correct if we do it just once, inside compiler_codegen, rather than in both callers?

[iritkatriel]

I'm looking into it, I think it's a bigger mess than I previously thought. There are functions like compile_class which always exit the scope (which they did not enter). And VISIT_SEQ/VISIT_SEQ_IN_SCOPE also call compiler_exit_scope.

Also, it seems like it would be simplest to get the enter/exit logic correct if we do it just once, inside compiler_codegen, rather than in both callers?

I don't think we can exit the scope in compiler_codegen, because we need it until after optimize_and_assemble.

[iritkatriel]

I'm going to revert the commit with the enter/exit changes for now - I probably only needed it in a broken state when I was sending bad inputs to the unit tests.

[carljm]

Looks good!