Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add claude-v2 #2571

Merged
merged 3 commits into from
Oct 17, 2023
Merged

feat: add claude-v2 #2571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2475141
feat: add claude-v2
congchan Oct 16, 2023
262375f
fix: rename Anthropic models names
congchan Oct 17, 2023
3671f09
fix: rename to ANTHROPIC_MODEL_LIST
congchan Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions fastchat/llm_judge/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import openai
import anthropic

from fastchat.model.model_adapter import get_conversation_template
from fastchat.model.model_adapter import get_conversation_template, ANTHROPIC_MODEL_LIST

# API setting constants
API_MAX_RETRY = 16
Expand Down Expand Up @@ -161,7 +161,7 @@ def run_judge_single(question, answer, judge, ref_answer, multi_turn=False):

if model in ["gpt-3.5-turbo", "gpt-4"]:
judgment = chat_compeletion_openai(model, conv, temperature=0, max_tokens=2048)
elif model in ["claude-v1", "claude-instant-v1"]:
elif model in ANTHROPIC_MODEL_LIST:
judgment = chat_compeletion_anthropic(
model, conv, temperature=0, max_tokens=1024
)
Expand Down Expand Up @@ -265,7 +265,7 @@ def run_judge_pair(question, answer_a, answer_b, judge, ref_answer, multi_turn=F
if model in ["gpt-3.5-turbo", "gpt-4"]:
conv.set_system_message(system_prompt)
judgment = chat_compeletion_openai(model, conv, temperature=0, max_tokens=2048)
elif model in ["claude-v1", "claude-instant-v1"]:
elif model in ANTHROPIC_MODEL_LIST:
if system_prompt != "You are a helpful assistant.":
user_prompt = "[Instruction]\n" + system_prompt + "\n\n" + user_prompt
conv.messages[0][1] = user_prompt
Expand Down
4 changes: 2 additions & 2 deletions fastchat/llm_judge/gen_api_answer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
chat_compeletion_palm,
)
from fastchat.llm_judge.gen_model_answer import reorg_answer_file
from fastchat.model.model_adapter import get_conversation_template
from fastchat.model.model_adapter import get_conversation_template, ANTHROPIC_MODEL_LIST


def get_answer(
Expand All @@ -44,7 +44,7 @@ def get_answer(
conv.append_message(conv.roles[0], question["turns"][j])
conv.append_message(conv.roles[1], None)

if model in ["claude-v1", "claude-instant-v1"]:
if model in ANTHROPIC_MODEL_LIST:
output = chat_compeletion_anthropic(
model, conv, temperature, max_tokens
)
Expand Down
9 changes: 8 additions & 1 deletion fastchat/model/model_adapter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@
)


ANTHROPIC_MODEL_LIST = (
"claude-1",
"claude-2",
Comment on lines +53 to +54
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's indeed claude-v1 and claude-2. I know it's confusing... could you test and confirm this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, my current resource could only test the AWS agile's Anthropic service, and -v1 and -v2 model ids works for me, but not -1 and -2. But I think the claude-2 should works as it is the official name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fastchat has been using "claude-v1", "claude-instant-v1" in llm_judge.common.py, and "claude-2", "claude-instant-1" in model/model_adapter.py. Seems that there have been different users using different sources services.

Would you consider accept all these names and let users to decide which key they choose to pass?

For reference the official Anthropic's doc shows

Family Latest major version Latest full version
Claude Instant claude-instant-1 claude-instant-1.2
Claude claude-2 claude-2.0

Copy link
Member

@infwinston infwinston Oct 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just confirmed that claude-1 works. thanks. then we're good now. I think v1 has become legacy name now.

"claude-instant-1",
)


class BaseModelAdapter:
"""The base and the default model adapter."""

Expand Down Expand Up @@ -936,7 +943,7 @@ class ClaudeAdapter(BaseModelAdapter):
"""The model adapter for Claude"""

def match(self, model_path: str):
return model_path in ["claude-2", "claude-instant-1"]
return model_path in ANTHROPIC_MODEL_LIST

def load_model(self, model_path: str, from_pretrained_kwargs: dict):
raise NotImplementedError()
Expand Down