llm-inference · GitHub Topics

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

C++ 73.61 k

19 天前

#大语言模型#Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Python 37.52 k

10 小时前

gitleaks / gitleaks

#大语言模型#Gitleaks 是一个开源SAST（静态应用安全测试）命令行工具，用于检测Git 仓库以防止把密码、API 密钥和访问令牌等机密信息硬编码到代码中

安全 Git Go secret gitleaks devsecops Hacktoberfest CI/CD 命令行界面 data-loss-prevention dlp Open Source ai-powered 大语言模型 llm-inference llm-training

Go 20.21 k

7 天前

liguodongiot / llm-action

#大语言模型#本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

大语言模型 llm-inference llm-serving llm-training llmops

HTML 18.54 k

12 小时前

Lightning-AI / litgpt

#大语言模型#20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

人工智能深度学习 large-language-models 大语言模型 llm-inference

Python 12.26 k

3 天前

bentoml / OpenLLM

#大语言模型#Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

大语言模型 llmops model-inference fine-tuning llm-serving llama vicuna bentoml llama2 llm-inference llm-ops mistral mlops llama3-1

Python 11.35 k

6 天前

mistralai / mistral-inference

#大语言模型#Official inference library for Mistral models

大语言模型 llm-inference mistralai

Jupyter Notebook 10.29 k

3 个月前

openvinotoolkit / openvino

#自然语言处理#OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

inference 深度学习 openvino 人工智能机器视觉 diffusion-models generative-ai llm-inference 自然语言处理 performance-boost speech-recognition stable-diffusion deploy-ai optimize-ai transformers yolo recommendation-system good-first-issue

C++ 8.43 k

16 小时前

SJTU-IPADS / PowerInfer

#大语言模型#PowerInfer 是一个快速的、可运行在消费级GPU、个人电脑上的大模型服务

large-language-models llama 大语言模型 llm-inference local-inference

C++ 8.22 k

4 个月前

bentoml / BentoML

#大语言模型#The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

model-serving mlops llmops generative-ai llm-inference 深度学习 llm-serving 机器学习 Python multimodal ml-engineering 大语言模型

Python 7.78 k

2 天前

InternLM / lmdeploy

#大语言模型#LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

cuda-kernels deepspeed fastertransformer llm-inference turbomind internlm llama 大语言模型 codellama llama2 llama3

Python 6.52 k

2 天前

superduper-io / superduper

#向量搜索引擎#Superduper: End-to-end framework for building custom AI applications and agents.

人工智能 mlops torch transformers MongoDB Python PyTorch 机器学习数据库 data inference llm-inference pretrained-models 聊天机器人 semantic-search llm-serving llmops vector-search rag

Python 5.08 k

3 天前

kserve / kserve

#计算机科学#Standardized Serverless ML Inference Platform on Kubernetes

knative 机器学习 model-interpretability model-serving istio kubeflow 人工智能 Tensorflow PyTorch scikit-learn xgboost Kubernetes service-mesh kserve Hacktoberfest mlops genai llm-inference

Python 4.24 k

4 天前

xlite-dev / Awesome-LLM-Inference

📚A curated list of Awesome LLM Inference Papers with Codes.

flash-attention tensorrt-llm vllm llm-inference deepseek deepseek-v3 deepseek-r1 qwen3

Python 4.12 k

7 天前

FellouAI / eko

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

TypeScript 3.99 k

1 天前

NVIDIA / GenerativeAIExamples

#大语言模型#Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

gpu-acceleration large-language-models 大语言模型 llm-inference 微服务 nemo rag retrieval-augmented-generation tensorrt triton-inference-server

Jupyter Notebook 3.18 k

5 天前

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving

gpu CUDA PyTorch llm-inference jit attention Nvidia

Cuda 3.17 k

2 天前

neuralmagic / deepsparse

#自然语言处理#Sparsity-aware deep learning inference runtime for CPUs

机器学习 onnx inference 机器视觉 object-detection pruning quantization pretrained-models 自然语言处理 cpus sparsification llm-inference performance

Python 3.15 k

13 天前

predibase / lorax

#大语言模型#Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

fine-tuning gpt llama 大语言模型 llm-inference llm-serving llmops lora model-serving PyTorch transformers

Python 3.01 k

25 天前

gpustack / gpustack

#大语言模型#Simple, scalable AI model deployment on GPU clusters

ascend CUDA deepseek distributed-inference genai inference llama llamacpp 大语言模型 maas metal openai qwen rocm vllm mindie llm-inference llm-serving local-ai heterogeneous-cluster

Python 2.92 k

3 天前