Infini-AI-Lab

All

13 repositories

MagicPIG
Public
MagicPIG: LSH Sampling for Efficient LLM Generation
decoding lsh-algorithm llm-inference gpu-cpu
Python
•
Apache License 2.0
•6•129•2•0•Updated Dec 12, 2024Dec 12, 2024
S2FT-Page
Public
JavaScript
•0•0•0•0•Updated Dec 10, 2024Dec 10, 2024
S2FT
Public
Python
•0•4•0•0•Updated Dec 9, 2024Dec 9, 2024
MagicDec
Public
Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
Python
•
Apache License 2.0
•4•100•6•0•Updated Dec 4, 2024Dec 4, 2024
MagicPIG-Page
Public
JavaScript
•1•0•0•0•Updated Dec 2, 2024Dec 2, 2024
Factor
Public
0•1•0•0•Updated Nov 7, 2024Nov 7, 2024
MagicDec-part1
Public
Speculative decoding for high-throughput long-context inference
JavaScript
•
Apache License 2.0
•0•0•0•0•Updated Sep 10, 2024Sep 10, 2024
Sirius
Public
Sirius, an efficient correction mechanism, which significantly boosts Contextual Sparsity models on reasoning tasks while maintaining its efficiency gain.
Python
•4•20•0•0•Updated Sep 10, 2024Sep 10, 2024
MagicDec-part2
Public
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Contexts with Speculative Decoding
JavaScript
•
Apache License 2.0
•0•0•0•0•Updated Sep 5, 2024Sep 5, 2024
TriForce
Public
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
acceleration efficiency inference llm long-context llm-inference speculative-decoding
Python
•13•234•7•0•Updated Aug 31, 2024Aug 31, 2024
Sequoia
Public
scalable and robust tree-based speculative decoding algorithm
efficiency inference llm speculative-decoding
Python
•36•322•7•3•Updated Aug 13, 2024Aug 13, 2024
lm-evaluation-harness
Public
A framework for few-shot evaluation of language models.
Python
•
MIT License
•1.9k•0•0•0•Updated Jun 10, 2024Jun 10, 2024
Sequoia-Page
Public
JavaScript
•1•0•0•0•Updated May 21, 2024May 21, 2024