llm-serving · GitHub Topics

#大语言模型#A high-throughput and memory-efficient inference and serving engine for LLMs

gpt 大语言模型 PyTorch llmops mlops model-serving transformer llm-serving inference llama amd rocm CUDA inferentia trainium tpu xpu hpu deepseek qwen

Python 49.62 k

20 小时前

ray-project / ray

#大语言模型#Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Python 37.52 k

14 小时前

liguodongiot / llm-action

#大语言模型#本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

大语言模型 llm-inference llm-serving llm-training llmops

HTML 18.54 k

17 小时前

sgl-project / sglang

#大语言模型#SGLang is a fast serving framework for large language models and vision language models.

CUDA inference llama llava 大语言模型 llm-serving moe PyTorch transformer vlm llama3 llama3-1 deepseek deepseek-llm deepseek-v3 deepseek-r1 deepseek-r1-zero qwen3 llama4

Python 15.14 k

7 小时前

bentoml / OpenLLM

#大语言模型#Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

大语言模型 llmops model-inference fine-tuning llm-serving llama vicuna bentoml llama2 llm-inference llm-ops mistral mlops llama3-1

Python 11.35 k

6 天前

skypilot-org / skypilot

#计算机科学#SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 16+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

Python 8.23 k

4 天前

bentoml / BentoML

#大语言模型#The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

model-serving mlops llmops generative-ai llm-inference 深度学习 llm-serving 机器学习 Python multimodal ml-engineering 大语言模型

Python 7.78 k

3 天前

superduper-io / superduper

#向量搜索引擎#Superduper: End-to-end framework for building custom AI applications and agents.

人工智能 mlops torch transformers MongoDB Python PyTorch 机器学习数据库 data inference llm-inference pretrained-models 聊天机器人 semantic-search llm-serving llmops vector-search rag

Python 5.08 k

3 天前

PaddlePaddle / FastDeploy

#大语言模型#Large Language Model Deployment Toolkit

serving ernie 大语言模型 qwen wenxinyiyan inference llm-serving

Cuda 3.21 k

5 天前

predibase / lorax

#大语言模型#Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

fine-tuning gpt llama 大语言模型 llm-inference llm-serving llmops lora model-serving PyTorch transformers

Python 3.01 k

25 天前

gpustack / gpustack

#大语言模型#Simple, scalable AI model deployment on GPU clusters

ascend CUDA deepseek distributed-inference genai inference llama llamacpp 大语言模型 maas metal openai qwen rocm vllm mindie llm-inference llm-serving local-ai heterogeneous-cluster

Python 2.92 k

3 天前

microsoft / aici

#大语言模型#AICI: Prompts as (Wasm) Programs

人工智能 Rust WebAssembly wasmtime inference language-model 大语言模型 llm-framework llm-inference llm-serving llmops model-serving transformer

Rust 2.03 k

5 个月前

MoonshotAI / MoBA

#大语言模型#MoBA: Mixture of Block Attention for Long-Context LLMs

flash-attention 大语言模型 llm-serving llm-training moe PyTorch transformer

Python 1.8 k

2 个月前

ray-project / ray-llm

#大语言模型#RayLLM - LLMs on Ray (Archived). Read README for more info.

ray 大语言模型 llm-serving

Python 1.26 k

3 个月前

thu-pacman / chitu

#大语言模型#High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

deepseek gpu 大语言模型 PyTorch llm-serving model-serving

Python 1.14 k

6 天前

zhihu / ZhiLight

#大语言模型#A highly optimized LLM inference acceleration engine for Llama and its variants.

inference-engine 大语言模型 CUDA gpt llama llm-serving PyTorch llm-inference model-serving deepseek-r1

C++ 891

1 个月前

mosecorg / mosec

#大语言模型#A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

model-serving 深度学习机器学习 nerual-network mlops Hacktoberfest gpu Python PyTorch Tensorflow 大语言模型 jax llm-serving Rust cv mxnet tts

Python 842

5 天前

efeslab / Nanoflow

#大语言模型#A throughput-oriented high-performance serving framework for LLMs

CUDA inference llama2 大语言模型 llm-serving model-serving

Jupyter Notebook 820

11 天前

alibaba / rtp-llm

#大语言模型#RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

gpt inference llama 大语言模型 llm-serving llmops model-serving

C++ 788

12 天前

vllm-project / vllm-ascend

#大语言模型#Community maintained hardware plugin for vLLM on Ascend

ascend inference 大语言模型 llm-serving llmops mlops model-serving transformer vllm

Python 758

20 小时前