Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Hermes 2.5 to OpenOrcaAdapter #2634

Closed
wants to merge 7 commits into from
Closed

Add Hermes 2.5 to OpenOrcaAdapter #2634

wants to merge 7 commits into from

Conversation

teknium1
Copy link
Contributor

@teknium1 teknium1 commented Nov 3, 2023

They use the same prompt format

Why are these changes needed?

This adds support for the OpenHermes 2.5 model, I added it under OpenOrca's because they use the same format, though you may prefer it have it's own. Let me know.

They use the same prompt format
@infwinston
Copy link
Member

could you resolve conflict?

@teknium1
Copy link
Contributor Author

teknium1 commented Nov 3, 2023

Updated

@infwinston
Copy link
Member

infwinston commented Nov 3, 2023

looks like some special token <|end_of_turn|> is not correctly filtered out. Have you tested it on your end with
python3 -m fastchat.serve.cli --model-path teknium/OpenHermes-2.5-Mistral-7B --debug?

image

@teknium1
Copy link
Contributor Author

teknium1 commented Nov 3, 2023

Okay, it seems the actual setting of the template moved places after my original commit, which wasnt obvious when having to resolve the conflict at first, and it had to be reimplemented in the final call of that method.

@teknium1
Copy link
Contributor Author

teknium1 commented Nov 3, 2023

well I tried it now but found that it has a preset system prompt:
{'conv_template': 'mistral-7b-openorca', 'prompt': '<|im_start|>system\nYou are MistralOrca, a large language model trained by Alignment Lab AI. Write out your reasoning step-by-step to be sure you get the right answers!<|im_end|>

Not ideal for my model so I will redo a new one :D

Add "hermes2" template
add Hermes2Adapter model adapter
@teknium1
Copy link
Contributor Author

teknium1 commented Nov 3, 2023

Okay, it should be good to go now

Hermes2Adapter :)

@teknium1
Copy link
Contributor Author

teknium1 commented Nov 3, 2023

Reversed my changes to the OpenOrca mistral adapter as well

fix model_path.lower call
@teknium1
Copy link
Contributor Author

teknium1 commented Nov 3, 2023

Am now getting [INST] format - not seeing exactly why..
image

@infwinston
Copy link
Member

@teknium1 any luck getting it work?

@teknium1
Copy link
Contributor Author

teknium1 commented Nov 5, 2023

No not yet I had to focus on other things, will return to it on mon or tues

@infwinston
Copy link
Member

@teknium1 any update? we'd love to host your model on Arena!

@teknium1
Copy link
Contributor Author

I am just not sure what it needs to work - adding the Hermes adapter did not result in it using the format, instead defaulted to llama-2-chat format. The only issue is I can use orca mistral 7b format but it has a very bad system prompt that I dont want to use

@merrymercy
Copy link
Member

@teknium1

  1. You need to register your adapter, similar to this line
    register_model_adapter(MicrosoftOrcaAdapter)
  2. You can start debugging by adding some print here
    def get_conversation_template(model_path: str) -> Conversation:
    """Get the default conversation template."""
    adapter = get_model_adapter(model_path)
    return adapter.get_default_conv_template(model_path)

@152334H 152334H mentioned this pull request Nov 23, 2023
@teknium1
Copy link
Contributor Author

We rebased and restarted, new PR is here: #2725

Closing this one

@teknium1 teknium1 closed this Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants