A high-throughput and memory-efficient inference and serving engine for LLMs

Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

SGLang is a high-performance serving framework for large language models and multimodal models.

Faster Whisper transcription with CTranslate2

Machine Learning Engineering Open Book

Large Language Model Text Generation Inference

Nano vLLM

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Easily fine-tune, evaluate and deploy gpt-oss, Qwen3, DeepSeek-R1, or any open source LLM / VLM!

💎1MB lightweight face detection model (1MB轻量级人脸检测模型)

Supercharge Your LLM with the Fastest KV Cache Layer

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Superduper: End-to-end framework for building custom AI applications and agents.

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

An easy to use PyTorch to TensorRT converter

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式

Performance-Optimized AI Inference on Your GPUs. Unlock it by selecting and tuning the optimal inference engine for your model.

Pre-trained Deep Learning models and demos (high quality and extremely fast)