Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added a few helpful Ollama connection settings #1

Open
Steff81 opened this issue Dec 1, 2024 · 0 comments
Open

Added a few helpful Ollama connection settings #1

Steff81 opened this issue Dec 1, 2024 · 0 comments

Comments

@Steff81
Copy link

Steff81 commented Dec 1, 2024

Hey Ben, I used your script for connection to my private Ollama instance. Thanks for that.
Anyway i figured out that the connection was not working because curl closed the connection directly and did not wait for the response of the AI server. I tried with curl-timeout without success for whatever reason.
Took me a while but then I made the changes below to the code and added a loop with a timeout.
Not 100% perfect as the loop always adds another complete second, which is maybe not needed but
after adding the loop everything works as expected.
Up to you if you want to add it to the code or leave it here so it will maybe help someone else.
I tried to keep it configurable and added it only for ollama for now.

`private function execOllama() {

	// a few more ollama settings that maybe are helpful

	// This option determines whether curl verifies the authenticity of the peer's certificate. A value of 1 means curl verifies; 0 (zero) means it does not.
	$sslVerifyPeer = 1;
	// Set this option to zero to switch to the default built-in connection timeout - 300 seconds. But you can also set this higher
	$curlInitialConnectionTimeout = 0;
	// This parameter is used for 2 things. It sets the CURLOPT_TIMEOUT to this seconds variable. But as this did not help solving my problem it will also be the waiting time in the loop for the response from the AI-Server. Sometimes CURLOPT_TIMEOUT is not working as expected as it looks like. Time is in seconds. I set the default to 30 seconds.
	$aiWaitTimeout = 30;

	$post_data = [
		"model"       => $this->Model,
		// "max_tokens"  => $this->MaxTokens, // not used in ollama
	  "options" => [
	    "temperature" => $this->Temperature
	  ],
		'stream'      => false,
		"messages"    => [
			[	"role" => "system", "content" => '', ],
			[	"role" => "user",   "content" => $this->Prompt, ]
		],
	];
	$post_fields = json_encode($post_data);

	curl_setopt($this->curl, CURLOPT_URL, $this->reqUrl);
	curl_setopt($this->curl, CURLOPT_POST, true);
	curl_setopt($this->curl, CURLOPT_POSTFIELDS, $post_fields);
	curl_setopt($this->curl, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($this->curl, CURLOPT_SSL_VERIFYPEER, $sslVerifyPeer);
	curl_setopt($this->curl, CURLOPT_TIMEOUT, $aiWaitTimeout);
	curl_setopt($this->curl, CURLOPT_CONNECTTIMEOUT, $curlInitialConnectionTimeout); 
	curl_setopt($this->curl, CURLOPT_HTTPHEADER, [
		'Content-Type: application/json',
		'Authorization: Bearer ' . $this->ApiKey,
	]);

	//intitialise the timer
	$aiWaitTimeoutCounter = 0;

	do {
	    $this->rsp->Raw        = curl_exec($this->curl);
	    $responseCode = curl_getinfo($this->curl, CURLINFO_RESPONSE_CODE);

	    // keep in mind that sleep returns 0 on success,
	    // that would evaluate to false without strict comparison
	    if ($aiWaitTimeoutCounter == $aiWaitTimeout) {break;}
	    // add 1 second to the aiWaitTimeoutCounter
	    $aiWaitTimeoutCounter++;
	    // sleep for another second
	} while ($responseCode !== 200 && sleep(1) !== false);

	curl_close($this->curl);

	$this->rsp->RawDecoded = json_decode($this->rsp->Raw, true);
	if (isset($this->rsp->RawDecoded['message']['content'])) {
		$this->rsp->Data = $this->rsp->RawDecoded['message']['content'];
	} else {
		$this->rsp->Err = true;
	}
}`

Kind regards
Steffen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant