Stop tokens for Java do not allow completions that produce several top-level methods.

[PootieT]

example Java signature:

import java.util.*;
import java.lang.reflect.*;
import org.javatuples.*;
import java.security.*;
import java.math.*;
import java.io.*;
import java.util.stream.*;
class Problem {
    public static boolean hasCloseElements(ArrayList<Float> numbers, float threshold) {

The current stop sequence for Java is

"\n    }\n"

semantically it means to stop after having finished generating the current function. However, for the case of HumanEval at least, there are multiple instances where the canonical python solution contain nested functions. This means that for Java (where nested functions are illegal), the model may need to generate helper functions in addition to the main function.

I found the following stop tokens to work better, stops when it needs to, while allowing the model to generate multiple functions

"public static void main"
"\n}"

This way, model stops either when it begins to generate the main function, or the end of the Problem class.

This however, may break the philosophy of the testing, if we were to only expect one function to be generated? In which case, the problems with nested functions may just be much more difficult for models to generate (In Java).

[arjunguha]

(Well, you can nest lambdas and create inner classes. ) I agree that a canonical solution in Java may require multiple top-level functions. But, that may also be the case for Python, though the OpenAI stop sequences would prevent that.

I don't think I want MultiPL-E to generate multiple top-level functions. It is an artificial restriction, but it creates consistency across languages (for what that's worth).