Is it possible to evaluate llama.cpp perplexity using llama-cpp-python?

[oobabooga]

I am working here on a "llamacpp_HF" wrapper that allows llama.cpp to be treated like a transformers model, giving it access to the exact same samplers as models in that library. The text generation is currently functional but slow.

It works by making the forward call inside the wrapper like this:

    def __call__(self, *args, **kwargs):
        seq = kwargs['input_ids'][0].tolist()
        seq_tensor = torch.tensor(seq)
        if self.cache is None or not torch.equal(self.cache, seq_tensor[:-1]):
            self.model.model.reset()
            self.model.model.eval(seq)
        else:
            self.model.model.eval([seq[-1]])

        self.cache = seq_tensor
        logits = torch.tensor(self.model.model.eval_logits).view(1, 1, -1).to(kwargs['input_ids'].device)

I create a self.cache variable to be able to tell whether I need to call reset() on the llama.cpp model or not based on the provided input ids.

The next step would be to use this wrapper for perplexity evaluation. This would make a direct comparison against transformers or AutoGPTQ possible. The problem is that the call

logits = torch.tensor(self.model.model.eval_logits).view(1, 1, -1).to(kwargs['input_ids'].device)

returns a tensor with shape torch.Size([1, 1, 32000]), while my existing evaluation code, as well as a few alternative implementations that I have tried, always expect the forward call to return a tensor with shape torch.Size([1, 1200, 32000]), where the second number is the context size being used for the evaluation.

Can anyone see an obvious solution to this?