Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Llama 2 with Llama 3 as an AI model #38071

Closed
Tracked by #38195
I-I-IT opened this issue May 5, 2024 · 7 comments · Fixed by brave/brave-core#23907 or brave/brave-core#23911
Closed
Tracked by #38195

Replace Llama 2 with Llama 3 as an AI model #38071

I-I-IT opened this issue May 5, 2024 · 7 comments · Fixed by brave/brave-core#23907 or brave/brave-core#23911
Assignees
Labels
browser-ai OS/Android Fixes related to Android browser functionality OS/Desktop priority/P2 A bad problem. We might uplift this to the next planned release. QA Pass - Android ARM QA Pass-macOS QA/Yes release-notes/include

Comments

@I-I-IT
Copy link

I-I-IT commented May 5, 2024

This seems obvious, but haven't seem an issue about adding the new more accurate and more responsive Meta Llama 3 8b as a the unlimited model to replace Llama 2 13b. Llama 3 8b is better than even Llama 2 70b, according to Arena LMSYS leaderboard<

Test plan

All llama2 strings now use llama3. Query functionality still works with the llama model selection.

@I-I-IT I-I-IT added OS/Android Fixes related to Android browser functionality OS/Desktop labels May 5, 2024
@mattmcalister mattmcalister added priority/P2 A bad problem. We might uplift this to the next planned release. QA/Yes release-notes/include labels May 9, 2024
@mattmcalister
Copy link

Thanks @TechFanTheo . It is indeed coming and we need an issue for it.

@kjozwiak
Copy link
Member

kjozwiak commented Jun 3, 2024

The above requires 1.67.105 or higher for 1.67.x verification 👍

@I-I-IT
Copy link
Author

I-I-IT commented Jun 3, 2024

I have latest Nightly and tested it. Says Connection error with Llama 3 but it works fine with other models.

Did you enable the Llama 3 servers yet ?
VideoCapture_20240603-192613
VideoCapture_20240603-192622
VideoCapture_20240603-192633
VideoCapture_20240603-192641
VideoCapture_20240603-192644

@Uni-verse Uni-verse added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Jun 3, 2024
@I-I-IT
Copy link
Author

I-I-IT commented Jun 4, 2024

Fixed.

@stephendonner
Copy link

stephendonner commented Jun 6, 2024

Verification PASSED using

Brave | 1.67.107 Chromium: 126.0.6478.26 (Official Build) beta (x86_64)
-- | --
Revision | 2831369c7540d93e98f38d9b463f6762430a7982
OS | macOS Version 11.7.10 (Build 20G1427)

Steps:

  1. installed 1.67.107
  2. launched Brave
  3. chose Llama 3 8b for Default model for new conversations in brave://settings/leo-assistant
  4. loaded https://www.vox.com/technology/352849/openai-chatgpt-google-meta-artificial-intelligence-vox-media-chatbots
  5. clicked on Leo in the sidebar panel
  6. clicked on Summarize this page
  7. clicked I understand
  8. waited
  9. also clicked on Suggest questions...
  10. confirmed questions loaded

Confirmed Llama 3 8b correctly/quickly summarized the page contents and rendered/answered Suggest questions... as well

example example example example example example example
Screen Shot 2024-06-06 at 10 18 56 AM Screen Shot 2024-06-06 at 10 19 07 AM Screen Shot 2024-06-06 at 10 29 27 AM Screen Shot 2024-06-06 at 10 30 29 AM Screen Shot 2024-06-06 at 10 30 36 AM Screen Shot 2024-06-06 at 10 32 38 AM Screen Shot 2024-06-06 at 10 33 03 AM

@stephendonner stephendonner added QA/In-Progress Indicates that QA is currently in progress for that particular issue QA Pass-macOS and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Jun 6, 2024
@Uni-verse
Copy link
Contributor

Uni-verse commented Jun 6, 2024

Verified on Samsung Galaxy S21 using the following version:

Brave	1.67.112 Chromium: 126.0.6478.40 (Official Build) (64-bit) 
Revision	4190f7ced064ec338b9dedc915d70b50e28c843e
OS	Android 13; Build/TP1A.220624.014; 33; REL

  • Ensured Llama 3 8b has replaced Llama 2 in the language models menu for new and existing users.
  • Ensured Llama model is updated in the Default model in Advanced Settings
  • Ensured query functionality with and without page context for free and premium users.
  • Ensured model is not rate limited.
  • Ensured users upgrading with Llama 2 set as default will default to Llama 3.
  • Ensured summarize this page and suggestions are working with Llama.
  • Ensured Ask Leo is working with Llama 3 from the omnibox.
Example Example Example Example
Screenshot 2024-06-11 at 5 45 07 PM Screenshot 2024-06-11 at 6 12 07 PM Screenshot 2024-06-11 at 6 12 15 PM Screenshot 2024-06-11 at 6 14 13 PM
Screenshot 2024-06-11 at 6 16 36 PM Screenshot 2024-06-11 at 6 16 57 PM Screenshot 2024-06-11 at 6 42 44 PM Screenshot 2024-06-11 at 6 43 56 PM

@Uni-verse Uni-verse added QA Pass - Android ARM and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Jun 11, 2024
@kjozwiak
Copy link
Member

The above requires 1.67.121 or higher for 1.67.x verification on iOS 👍 CCing @hffvld @Uni-verse

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
browser-ai OS/Android Fixes related to Android browser functionality OS/Desktop priority/P2 A bad problem. We might uplift this to the next planned release. QA Pass - Android ARM QA Pass-macOS QA/Yes release-notes/include
Projects
Status: Done
7 participants