[Feature Request/Improvement] Fast unload of model in vram constrained environments for Ollama inference via keep_alive parameter in request

[Deathproof76]

Hello!👋 😊

Would it be possible to add the keep_alive parameter for the Ollama requests and possible expose it as an env for hoarder? I've noticed that my vram is still occupied for some time after a hoarding stuff (most likely related to the default env settings of my olllama instance).
So far I couldn't see a keep_alive parameter in the inference.ts.

Maybe it could work something like this:
Disclaimer: I used an llm to generate the code

class OllamaInferenceClient implements InferenceClient {
  ollama: Ollama;

  constructor() {
    this.ollama = new Ollama({
      host: serverConfig.inference.ollamaBaseUrl,
    });
  }

  async runModel(model: string, prompt: string, image?: string, keepAlive: number = 10) {
    const chatCompletion = await this.ollama.chat({
      model: model,
      format: "json",
      stream: true,
      messages: [
        { role: "user", content: prompt, images: image ? [image] : undefined },
      ],
      keep_alive: keepAlive,  // <-- Set keep_alive to 10 seconds
    });

    let totalTokens = 0;
    let response = "";
    try {
      for await (const part of chatCompletion) {
        response += part.message.content;
        if (!isNaN(part.eval_count)) {
          totalTokens += part.eval_count;
        }
        if (!isNaN(part.prompt_eval_count)) {
          totalTokens += part.prompt_eval_count;
        }
      }
    } catch (e) {
      totalTokens = NaN;
      logger.warn(
        `Got an exception from ollama, will still attempt to deserialize the response we got so far: ${e}`,
      );
    }

    return { response, totalTokens };
  }

  async inferFromText(prompt: string, keepAlive: number = 10): Promise<InferenceResponse> {
    return await this.runModel(serverConfig.inference.textModel, prompt, undefined, keepAlive);
  }

  async inferFromImage(
    prompt: string,
    _contentType: string,
    image: string,
    keepAlive: number = 10
  ): Promise<InferenceResponse> {
    return await this.runModel(
      serverConfig.inference.imageModel,
      prompt,
      image,
      keepAlive,
    );
  }
}

And for example in the compose evironment something like INFERENCE_OLLAMA_KEEPALIFE: 10 and the model would be purged within 10 Seconds from vram after being done generating tags.

Thank you for reading and kind regards!

[MohamedBassem]

Hey, thanks for the suggestion. Today I learned about ollama's keep alive! Sure, I can do that.