feat(quil-py): support extern call instructions

[erichulburd]

Review Guidance

High-level overview

This introduces support for CALL and PRAGMA EXTERN instructions. The former is supported directly as an Instruction and I've introduced a ReservedPragma instruction to accommodate the latter (see source code comment below for discussion on this particular choice).

There are three main aspects to this support:

• Parsing of call in crate::parser::command::parse_call and parsing of ExternSignature in crate::parser::pragma_extern.
• Rust representation of these instructions in crate::instruction::extern_call.
• Resolution of memory accesses in crate::instruction::extern_call and crate::program::Program::get_memory_accesses.

This functionality was also ported to Python in quil-py/src/instruction/extern_call.rs.

Public API Changes

There are two breaking public API changes from adding EXTERN / CALL support:

1. Two One new enum variant on Instruction: Call and ReservedPragma (see source code comment below for the choice of ReservedPragma).
2. Instruction::get_memory_accesses is fallible now. This reflects the fact that a CALL instruction cannot know its memory accesses until it has been resolved to an ExternSignature (ie it has to know the mutability of its different arguments.

[erichulburd]

I considered three alternatives here:

1. Just have a Vec<CallArgument> where each CallArgument has a resolution: Option<Resolution> attribute.
2. Do not mutate the CallArguments and instead return a struct that represents the resolution, roughly of the structure ("name", instruction_index) => Vec (we need the index to refer to index of the CALL instruction within the program).
3. Add a type parameter to the CALL instructions and then to the program itself. Resolution would then return a program of a different type (eg into_call_resolved_program(self) -> Result<Program<ResolveCallArgument>, ProgramError>).

I landed on the de facto implementation because:

1. Within a given instruction, call arguments are all resolved at the same time. Either all arguments are resolved or none are.
2. There are existing patterns within Quil that mutate the program, such as resolve_placeholders.
3. Tracking instruction indices seems fairly unergonomic and brittle.
4. I did not want to add type complexity to the Program struct which is pretty easy to use.
5. I wanted to avoid the user resolving the program more than once for efficiency's sake.

I definitely feel like this resolution functionality belongs in quil-rs and not separately in downstream compilers. After going back and forth, I think this is the right implementation, but am open to input here.

[antalsz]

This is indeed a thorny question! I agree that alternatives #2 and (to a lesser extent) #1 are inferior, but I think there's a lot to be said for #3. I've talked to you about this offline, but to expand here:

I think this raises the question of if we've hit the point where we want to separate the parsed AST from the "typechecked"/"resolved" AST. I think there's a lot of merit to doing so, as it allows for capturing invariants much more cleanly. But I'm not sure if this PR is the place to do it.

I think I might favor a design where we parameterize all our types by the stage of compilation, passing that down to where we need to make a decision, and default that type parameter to the stage that corresponds to the existing situation. Something like the following:

pub trait Stage {
    type CallArgument;
}

pub enum Parsed;

impl Stage for Parsed {
    type CallArgument = UnresolvedCallArgument;
}

pub enum Resolved;

impl Stage for Resolved {
    type CallArgument = ResolvedCallArgument;
}

pub struct Program<S: Stage = Resolved> {
    // …
    instructions: Instruction<S>,
    // …
}

pub enum Instruction<S: Stage = Resolved> {
    // …
    Call(Call<S>),
    // …
}

pub struct Call<S: Stage = Resolved> {
    pub name: String,
    pub arguments: Vec<S::CallArgument>,
}

The upside to this is that we can bundle all the types we need to parameterize by together; the downside is the extra trait. I think the upside is likely to be worth it, but it's not obvious.

Another limitation of this approach is that it forces the ASTs to be almost identical. This can be good or bad. Another approach would be to have

pub trait Stage {
    type AST;
}

even if Parsed::AST = Program<α> and Resolved::AST = Program<β> for now.

We can also hide some of the complexity by having the parser return a Program<Parsed>, a function fn resolve(parsed: Program<Parsed>) -> Result<Program<Resolved>, ResolutionError>, and then exposing at the top level a function that simply combines the two, returning a Result<Program, …>, so the user is not confronted with this new API.

That said: this adds extra complexity! I think that may be worth it, but it's not obvious. This mutate-and-resolve approach is not a bad one, and it might even be the implementation behind the more complex version I outline above.

[erichulburd]

Just spoke with Kalan about this. We landed somewhere around here:

• Pragma doesn't need to be an enum. We can add something like extern_definitions: HashMap<String, ExternDefinition> to Program.
• struct Call will only have arguments: Vec<UnresolvedCallArgument>. Call.resolve(...) returns Result<Vec<ResolvedCallArgument>, ...>. This will be a public function for the purposes of translating.

There's a bit of inefficiency here WRT resolving in get_memory_accesses (still fallible) and then resolving for translation. I may think a bit more about that, but otherwise, this seems tenable to me.

[antalsz]

So is the idea here that resolution doesn't need to produce a new Program, just store the extern_definitions? And then any processing that needs to consume a resolved Call will just call .resolve in situ when necessary? I think this seems fine, if a bit of kicking the can down the road wrt a new AST – but as I said above, this PR is likely not the right place for that anyway, so I don't think that's an issue.

[erichulburd]

Yup, you got it. Here I'm just going for consistency with the existing implementation but I'm onboard with your general vision for generating a separate "validated and resolved" AST. We'll keep the conversation going.

[antalsz]

This is indeed a thorny question! I agree that alternatives #2 and (to a lesser extent) #1 are inferior, but I think there's a lot to be said for #3. I've talked to you about this offline, but to expand here:

I think this raises the question of if we've hit the point where we want to separate the parsed AST from the "typechecked"/"resolved" AST. I think there's a lot of merit to doing so, as it allows for capturing invariants much more cleanly. But I'm not sure if this PR is the place to do it.

I think I might favor a design where we parameterize all our types by the stage of compilation, passing that down to where we need to make a decision, and default that type parameter to the stage that corresponds to the existing situation. Something like the following:

pub trait Stage {
    type CallArgument;
}

pub enum Parsed;

impl Stage for Parsed {
    type CallArgument = UnresolvedCallArgument;
}

pub enum Resolved;

impl Stage for Resolved {
    type CallArgument = ResolvedCallArgument;
}

pub struct Program<S: Stage = Resolved> {
    // …
    instructions: Instruction<S>,
    // …
}

pub enum Instruction<S: Stage = Resolved> {
    // …
    Call(Call<S>),
    // …
}

pub struct Call<S: Stage = Resolved> {
    pub name: String,
    pub arguments: Vec<S::CallArgument>,
}

The upside to this is that we can bundle all the types we need to parameterize by together; the downside is the extra trait. I think the upside is likely to be worth it, but it's not obvious.

Another limitation of this approach is that it forces the ASTs to be almost identical. This can be good or bad. Another approach would be to have

pub trait Stage {
    type AST;
}

even if Parsed::AST = Program<α> and Resolved::AST = Program<β> for now.

We can also hide some of the complexity by having the parser return a Program<Parsed>, a function fn resolve(parsed: Program<Parsed>) -> Result<Program<Resolved>, ResolutionError>, and then exposing at the top level a function that simply combines the two, returning a Result<Program, …>, so the user is not confronted with this new API.

That said: this adds extra complexity! I think that may be worth it, but it's not obvious. This mutate-and-resolve approach is not a bad one, and it might even be the implementation behind the more complex version I outline above.

[erichulburd]

then exposing at the top level a function that simply combines the two, returning a Result<Program, …>, so the user is not confronted with this new API.

The one caveat here is program mutation (either mutating a parsed program or just some Program::new()). If the user deletes an EXTERN pragma, or adds another call, then our resolution is invalid. So we'll need to either expose the resolution method to the user, or re-resolve after every relevant mutation (less efficient and more complex, but also could help us guarantee correctness opaquely). My knee-jerk preference is the former option.

[antalsz]

I overall really like this representation, and I think it's a big improvement. I have some questions around exactly where we perform resolution, and I think you may be able to punt some (or most?) of them away with "we'll figure this out when we figure out what broader program checking looks like". I also have various specific comments.

One general note: I wonder if we should provide a way to trim out unused PRAGMA EXTERNs? I believe quil-rs provides that for unused calibrations etc. from the CLI; we should add removing unused PRAGMA EXTERNs to that, I think.

[antalsz]

Thanks for the detailed response, Eric, this looks great – I have a couple of tiny comments (one comment typo and some desire for code deduplication), but other than that I think it looks ready to go!