Discussions

• 1 You must be logged in to vote

  🙏

  GGML tensors

  moha657 asked Nov 9, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  CUDA OOM in the middle of prompt processing

  jurov asked Nov 9, 2024 in Q&A · Unanswered

  

  

  4
• 2 You must be logged in to vote

  🙏

  Llama.cpp RPC over Ethernet strangely slow

  becky-soda asked Aug 22, 2024 in Q&A · Unanswered

  

  

  

  

  4
• 2 You must be logged in to vote

  💡

  Add a no-gpu CLI arg to llama-cli

  WilliamTambellini started Aug 6, 2024 in Ideas

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  How to extend inference ctx by yarn out of training data?

  FNsi asked Nov 9, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Is there a way to silence log/debug output?

  oelna asked Jun 8, 2023 in Q&A · Answered

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  What does DGGML_NATIVE build option do?

  abitrolly asked Nov 9, 2024 in Q&A · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to use KV cache?

  manimathma asked Nov 9, 2024 in Q&A · Unanswered

  

  

  0
• 11 You must be logged in to vote

  💬

  LLM inference server performances comparison llama.cpp / TGI / vLLM

  performance Speed related topics

  phymbert started Apr 17, 2024 in General · Closed

  

  

  

  

  37
• 2 You must be logged in to vote

  💬

  What is ggml.org business plan ?

  ExtReMLapin started Nov 6, 2024 in General

  

  

  2
• 4 You must be logged in to vote

  💡

  QTIP: Quantization with Trellises and Incoherence Processing

  sorasoras started Nov 1, 2024 in Ideas

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  Can someone explain what ggml_backend_sched_t do?

  Zijie-Tian asked Nov 5, 2024 in Q&A · Closed · Answered

  

  

  6
• 13 You must be logged in to vote

  🙏

  Need help to understand q4_0, q4_1, q4_2, q4_3 quantization

  santapo asked Apr 22, 2023 in Q&A · Unanswered

  

  

  

  

  8
• 32 You must be logged in to vote

  🙌

  Performance of llama.cpp on Apple Silicon M-series

  performance Speed related topics need feedback Testing and feedback with results are needed

  ggerganov started Nov 22, 2023 in Show and tell

  

  

  

  

  132
• 34 You must be logged in to vote

  💡

  Neural Engine Support

  BrianSemiglia started Mar 20, 2023 in Ideas

  

  

  

  

  30
• 2 You must be logged in to vote

  🙌

  Nutshell 🐿️ - an AI meeting assistant powered by llama.cpp

  NutshellAI started Nov 8, 2024 in Show and tell

  

  

  0
• 2 You must be logged in to vote

  🙏

  Cuda build and --n-gpu-layers set to 0

  clarinevong asked Nov 7, 2024 in Q&A · Answered

  

  

  

  

  3
• 1 You must be logged in to vote

  🙏

  Directly use low bit gguf for continuous training?

  FNsi asked Nov 7, 2024 in Q&A · Answered

  

  

  2
• 1 You must be logged in to vote

  🙏

  Is there a way to verify model correctness in a new backend?

  luo-yicong asked Nov 4, 2024 in Q&A · Unanswered

  

  

  3
• 7 You must be logged in to vote

  💡

  Microsoft's TMAC (LUT BASED; lookup table based) Much Faster Performance.

  joseph777111 started Aug 23, 2024 in Ideas

  

  

  

  

  3
• 52 You must be logged in to vote

  🙌

  Tutorial: How to convert HuggingFace model to GGUF format

  samos123 started Sep 1, 2023 in Show and tell

  

  

  

  

  94
• 1 You must be logged in to vote

  🙏

  How do I know whether I should override modify_tensors to change tensors layout or not???

  FdyCN asked Sep 24, 2024 in Q&A · Unanswered

  

  

  1
• 7 You must be logged in to vote

  🙌

  tutorial : compute embeddings using llama.cpp

  ggerganov started Jun 3, 2024 in Show and tell

  

  

  

  

  7
• 14 You must be logged in to vote

  🙌

  Performance of llama.cpp on Apple Silicon A-series

  ggerganov started Dec 17, 2023 in Show and tell

  

  

  

  

  35
• 0 You must be logged in to vote

  🙏

  Specific int8 operators for conv and fully connected?

  mikhilg10 asked Nov 3, 2024 in Q&A · Answered

  

  

  2