feat: Add Quoted::tokens

[Thunkar]

Description

Problem*

While fmtstr.quoted_contents is great for 99% of use cases, sometimes the raw str (including double quotes) is needed. Consider this case (fn signature generation in aztec.nr)

pub(crate) comptime fn compute_fn_selector(f: FunctionDefinition) -> Field {
    let fn_name = f.name();
    let args_signatures = f.parameters().map(
        | (_, typ): (Quoted, Type) | {
        signature_of_type(typ)
    }
    ).join(quote {,});
    let signature = f"{fn_name}({args_signatures})".quoted_contents();
    // from_signature takes a str, quoted_contents is no good here!
    let computation_quote = quote {
        protocol_types::abis::function_selector::FunctionSelector::from_signature($signature).to_field()
    };
    unquote!(computation_quote)
}

We have a bunch of Quoted values obtained during comptime that we have to string (ha) together. The result is then needed to compute a hash.

Summary*

This just adds a builtin that "resolves" the fmtstr and then turns it into a Token::Str

It's possible to create a string from quoted values by doing:

   let signature = f"{name}({args})";
   let signature_as_str = unquote!(quote { $signature });

But this includes whitespaces around the tokens. To get rid of them, ended up going with the approach suggested by @jfecher, which certainly complicates user code, but avoids the "eating whitespaces" problem:

Going forward I think we'll need to instead control the formatting of Quoted values more. The compiler itself can never guess what exact formatting users will want of them since spaces are already not present. So I'm leaning towards a Quoted::tokens(self) -> [Quoted] method to return a slice of each individual token in the Quoted value. Then users could iterate over it and apply (or not apply) spacing in between each token as they see fit.

Also, the returned values are str<_>, since the compiler has no idea of the returned length after formatting. This means values can only be used by unquoting them and not directly.

Additional Context

My first intuition was to do

let signature = f"\"{quotedvalues}\""

But it doesn't work =/

Documentation*

Check one:

• No documentation needed.
• Documentation included in this PR.
• [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

• I have tested the changes locally.
• I have formatted the changes with Prettier and/or cargo fmt on default settings.

[github-actions]

Changes to Brillig bytecode sizes

Generated at commit: 2b6b630553c2eb7127aa4d89fa31031bd6449cd2, compared to commit: b84009ca428a5790acf53a6c027146b706170574

There are no changes in circuit sizes

[github-actions]

🚀 Deployed on https://66d9de3fc84f8d381f9adb6f--noir-docs.netlify.app

[jfecher]

This is a string formatting issue, not a quoting format strings issue.

As discussed on slack the issue is that formatting quoted values into a format string will add spaces between each token which breaks your use case. fmtstr_quoted here will lex the entire format string, then append each token together as a string with no separator. This would remove all the spaces in the format string, even for the static string contents which haven't been formatted. E.g. f" ".quoted() would return an empty string. Two identifiers: f"foo bar".quoted() would also be collapsed to just one with this method: `"foobar"

Going forward I think we'll need to instead control the formatting of Quoted values more. The compiler itself can never guess what exact formatting users will want of them since spaces are already not present. So I'm leaning towards a Quoted::tokens(self) -> [Quoted] method to return a slice of each individual token in the Quoted value. Then users could iterate over it and apply (or not apply) spacing in between each token as they see fit.

[asterite]

@jfecher I discussed this a bit with Grego over Slack. The main issue is that he needs signature to be a string. Here's a small program:

    comptime
    {
        let name = "hello";
        let fn_name_as_str_quoted = f"{name}";
        let fn_name_len: u32 = unquote!(quote { $fn_name_as_str_quoted.as_bytes().len()});
    }

Here unquote! ends up generating this:

hello.len()

but he wants to get this instead:

"hello".len()

One thing that I thought would work was doing something like this:

f"\"{name}\""

but it seems \" isn't allowed in format strings. But even then, it could break if name has double quotes in it.

So the idea of fmtstr::quoted is to produce a quoted string value.

[asterite]

Now that I finished writing the above, I realized this can be done like this:

use std::meta::unquote;

fn main() {
comptime
    {
        let name = "hello";
        let fn_name_as_str_quoted = f"{name}";
        let fn_name_len: u32 = unquote!(quote { "$fn_name_as_str_quoted".as_bytes().len()});
    }
}

/cc @Thunkar

[jfecher]

@asterite You don't need the as_bytes trick, you can just quote and unquote the format string:

fn main() {
    comptime {
        let name = quote { add };
        let args = quote { 1, 2 };
        let signature = f"{name}({args})";
        print_str(std::meta::unquote!(quote { $signature }));
    }
}

fn print_str<let N: u32>(s: str<N>) {
    println(s);
}

When you unquote a format string, it's lowered as a normal string literal since all the formatting is already done.

I wanted this to be a method on fmtstr since it's a bit niche but there's no way to specify the size of the return string in the type system:

impl<let N: u32, Env> fmtstr<N, Env> {
    comptime fn quoted_as_str(self) -> str<?> {
        std::meta::unquote!(quote { $self })
    }
}

It isn't N since that's the length of the fmtstr with the {foo} placeholders. Expanding out foo may make it longer.

[jfecher]

Looks like this is a corner case of lowering versus unquoting where the two differ.

• Unquoting: When we use $foo in a quote to splice in a value into the token stream
• Lowering: When we have a comptime value and lower it into "runtime" code. Lowering also applies to higher-comptimed items. E.g. the unquote! macro calls when we're already in a comptime context still perform lowering since that is how macros are inserted.

I'll make an edit to the docs to include this case.

[jfecher]

👍
We can explore ways to make this easier in the future

[github-actions]

FYI @noir-lang/developerrelations on Noir doc changes.

[asterite]

I think there's still no way to generate strings in quote 🤔

For example, how to do this?

fn main() {
    comptime
    {
        let values = [quote { a }, quote { b }];

        // Generate this body from the above values:
        quote {
            println("a, b");
        }
    }
}

Or at least I can't find a way to generate a string literal with content from quoted values.

There is a way to geneate a quoted value from other quoted values, using f"...", but there's no way to turn that into a string literal.

I think the first commit in this PR did that: allow turning a quoted value into a string token with.

As a side note, I found out that f"..." inside quote doesn't work well:

fn main() {
    comptime
    {
        println(quote {
            f"Hello"
        })
    }
}

The above prints quote { fHello }.

/cc @jfecher

[jfecher]

@asterite The first example can be done by moving the format string:

comptime fn foo() -> Quoted {
    let values = [quote { a }, quote { b }];

    let a = values[0];
    let b = values[1];
    let s = f"{a}, {b}";
    quote {
        println($s);
    }
}

Although it seems like there is a bug preventing me from calling foo from main:

fn main() {
    // error: Comptime-only type `Quoted` used in runtime code
    foo!();
}

Which is odd since we use a macro call to unquote it. I'd think this would be caught in one of the existing macro tests so I'm unsure what this is.

Formatting with an arbitrary number of slice elements is more difficult. IIRC Grego had a rather hacky solution to it. We probably should add a compile-time String type.

The format string printing issue is a result of our change to print strings without their surrounding quotes. Not sure what to do there, maybe we just print them with quotes when we're within a quoted value.

[asterite]

Formatting with an arbitrary number of slice elements is more difficult. IIRC Grego had a rather hacky solution to it. We probably should add a compile-time String type.

Right, a compile-time String might solve this.

Until them, I'm a bit blocked. The context is that I need to generate a function signature from its parameters, and I get these extra spaces. I guess for now I'll go with the extra spaces and we can fix that once we have compile-time strings.