Skip to content

Commit

Permalink
update version 0.2.1: support azure api
Browse files Browse the repository at this point in the history
  • Loading branch information
HowieHwong committed Jan 29, 2024
1 parent 91a6991 commit 0dd2e84
Show file tree
Hide file tree
Showing 4 changed files with 34 additions and 5 deletions.
1 change: 1 addition & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ hide:

- **Support LLMs in [replicate](https://replicate.com/) and [deepinfra](https://deepinfra.com/)**
- **Support easy pipeline for evaluation**
- **Support [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) API**

## **Version 0.2.0**

Expand Down
12 changes: 11 additions & 1 deletion docs/guides/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,24 @@
Before starting the evaluation, you need to first set up your [OpenAI API](https://openai.com/product) (GPT-4-turbo) and [Perspective API](https://developers.perspectiveapi.com/s/docs-get-started?language=en_US) (used for measuring toxicity).

```python

from trustllm import config

config.openai_key = 'your-openai-api-key'

config.perspective_key = 'your-perspective-api-key'
```

If you're using OpenAI API through [Azure](https://azure.microsoft.com/en-us/products/ai-services/openai-service), you should set up your Azure api:

```python
config.azure_openai = True

config.azure_engine = "your-azure-engine-name"

config.azure_api_base = "your-azure-api-url (openai.base_url)"
```



### Easy Pipeline

Expand Down
6 changes: 5 additions & 1 deletion trustllm_pkg/trustllm/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,8 @@
ernie_api = None
claude_api = None
palm_api = None
replicate_api = None
replicate_api = None
azure_openai = False
azure_engine = None
azure_api_version = "2023-08-01-preview"
azure_api_base = None
20 changes: 17 additions & 3 deletions trustllm_pkg/trustllm/utils/gpt_auto_eval.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import openai
from tenacity import retry, wait_random_exponential, stop_after_attempt
from trustllm.utils import file_process
from tqdm import tqdm
import logging
import os
import trustllm
Expand Down Expand Up @@ -29,6 +28,22 @@ def get_res(string, model='gpt-4-1106-preview', temp=0):
Raises:
ValueError: If the API response is null or an empty string.
"""

if trustllm.config.azure_openai:
openai.api_type = "azure"
openai.api_base = trustllm.config.api_base
openai.api_version = "2023-08-01-preview"

completion = openai.ChatCompletion.create(
engine=trustllm.config.azure_engine,
messages=[{"role": "user", "content": string}],
temperature=temp
)
if not completion.choices[0].message['content']:
raise ValueError("The response from the API is NULL or an empty string!")

return completion.choices[0].message['content']

completion = openai.ChatCompletion.create(
model=model,
messages=[{"role": "user", "content": string}],
Expand Down Expand Up @@ -70,7 +85,6 @@ def save_progress(self, data, filename='auto_eval.json'):
file_process.save_json(data, save_path)
logging.info("Progress saved to %s", save_path)


def evaluate(self, data, task, resume=False, progress_filename='eval_progress.json', concat=True):
"""
Evaluate a given dataset using a specified task.
Expand Down Expand Up @@ -146,4 +160,4 @@ def process_item(item, el):
concurrent.futures.wait(futures)

self.save_progress(data, filename=progress_filename)
return data
return data

0 comments on commit 0dd2e84

Please sign in to comment.