speculative-decoding

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3.

#计算机科学#Large-scale LLM inference engine

#大语言模型#scalable and robust tree-based speculative decoding algorithm

#大语言模型#Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024

#大语言模型#[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

REST: Retrieval-Based Speculative Decoding, NAACL 2024

LLM Inference on consumer devices

#大语言模型#Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

[ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation

#大语言模型#[NeurIPS'23] Speculative Decoding with Big Little Decoder

#大语言模型#Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.

[ICLR 2025] SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

[ICML 2025] Reward-guided Speculative Decoding (RSD) for efficiency and effectiveness.

Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)

#自然语言处理#Pretty and simple to use implementation of speculative decoding algorithm eagle which is extrapolation algorithm for greater language model efficiency 🦅

Official Implementation of SAM-Decoding: Speculative Decoding via Suffix Automaton

#大语言模型#PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation

#大语言模型#minimal C implementation of speculative decoding based on llama2.c

Official Implementation of LANTERN (ICLR'25) and LANTERN++(ICLRW-SCOPE'25)