createCompletion returns an uncompleted answer

[akudo7]

System Info

• node 18.16.0
• gpt4all 3.1.0

The modules below are not related with this problem, I guess. But just in case.

• nodemon 3.0.2
• ts-node 10.9.1
• typescript 5.3.3

Information

• The official example notebooks/scripts
• My own modified scripts

Reproduction

1. Let us confirm setting values of the UI.

2. Let us ask the question below.

Find the bugs in the code, and show the improvements as the improved code.:

 import { Injectable, NestMiddleware } from '@nestjs/common';
 import { Request, Response, NextFunction } from 'express';
 @Injectable() export class SingletonMiddleware implements NestMiddleware {
    constructor() {
         console.log('SingletonMiddleware::constructor()');
     }
     public async use(req: Request, res: Response, next: NextFunction): Promise<void> {
          console.log('SingletonMiddleware::use()');
          req['user'] = 'Akira Kudo'; // req.body['user']
          next();
     }
 }

3. Let us run the code below. Unfortunately, createCompletion() returns an uncompleted answer.

import { createCompletion, loadModel, PromptMessage } from "gpt4all";

async function main() {
  try {
    // TODO: FAiled to load the 'gpt4all-falcon-q4_0.gguf'
    // https://github.com/nomic-ai/gpt4all/issues/1631
    const model = await loadModel("mistral-7b-openorca.Q4_0.gguf", {
      modelPath:
        "/Users/akirakudo/Library/Application Support/nomic.ai/GPT4All",
      librariesPath: "./node_modules/gpt4all/runtimes/darwin/native",
      device: "cpu",
      verbose: true,
    });

    const messages: PromptMessage[] = [
      { role: "system", content: "You are the NestJS expert." },
      {
        role: "user",
        content:
          "Find the bugs in the code, and show the improvements as the improved code.: \n ```typescript\n import { Injectable, NestMiddleware } from '@nestjs/common';\n import { Request, Response, NextFunction } from 'express';\n @Injectable() export class SingletonMiddleware implements NestMiddleware {\n   constructor() {\n     console.log('SingletonMiddleware::constructor()');\n   }\n   public async use(req: Request, res: Response, next: NextFunction): Promise<void> {\n     console.log('SingletonMiddleware::use()');\n      req['user'] = 'Akira Kudo'; // req.body['user']\n     next();\n   }\n }\n ```",
      },
    ];

    const events = await createCompletion(model, messages, {
      /*
      tokensSize: 200,
      temp: 0.7,
      topK: 40,
      topP: 0.4,
      repeatPenalty: 1.18,
      repeatLastN: 64,
      nBatch: 8,
      */
      systemPromptTemplate: '%1'
    });

    let response = "";
    for (const choice of events.choices) {
      const delta = choice.message.content;
      if (delta !== undefined) {
        response += delta;
      }
    }
    console.log("response:", response);
  } catch (error) {
    console.error(error);
  }
}

main();

Expected behavior

Need to get the same answer with the UI using NodeJS. At least, a completed answer.

[jacoobes]

Try adjusting the settings. nPredict option in createCompletion can be increased to have a longer output to the llm's response.

[akudo7]

Hi jacoobes,
Thanks for your reply.

Using the nPredict like blow makes a complete answer.

    const events = await createCompletion(model, messages, {
      nPredict: 1000,
      systemPromptTemplate: '%1'
    });

What number does the UI use for it?

GPT4ALL manual only says ...

NPREDICT
The number of tokens to predict.

Happy Holidays!
Thanks.
Akira

[hwoitke]

Saved my day, thanks @akudo7 :)

[cebtenzzre]

What number does the UI use for it?

Sorry for the late reply, but the chat UI defaults to a limit of 4096 predicted tokens.