LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
- Updated
Jul 29, 2026 - Python
inference-server - 技术专题
相关标签
Here are 129 public repositories matching this topic...
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
rust machine-learning transformers api-server developer-tools llama command-line-tool inference-server webgpu rust-crate huggingface huggingface-transformers huggingface-models webgpu-shaders llamacpp llm-inference local-ai gguf ollama-api openai-compatible
- Updated
Jul 29, 2026 - Rust
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
- Updated
Jul 28, 2026 - Python
Turn any computer or edge device into a command center for your computer vision projects.
python docker machine-learning computer-vision deployment inference classification object-detection vit agents inference-server jetson tensorrt instance-segmentation onnx inference-api yolov5 yolov8 yolo11 yolov12
- Updated
Jul 29, 2026 - Python
Open-source inference server and production cluster for all the models your agent needs.
python nlp natural-language-processing information-retrieval deep-learning retrieval ml inference embeddings bge semantic-search inference-server data-pipeline reranking mlops vector-search colbert llm splade retrieval-augmented-generation
- Updated
Jul 28, 2026 - Python
The simplest way to serve AI/ML models in production
open-source machine-learning packaging artificial-intelligence falcon easy-to-use whisper inference-server model-serving inference-api stable-diffusion wizardlm
- Updated
Jul 28, 2026 - Python
An open-source computer vision framework to build and deploy apps in minutes
python machine-learning cloud video computer-vision deep-learning gstreamer ffmpeg multimedia inference pipeline-framework artificial-intelligence stream-processing video-processing perception yolo object-detection inference-server vision-framework multimedia-applications
- Updated
May 8, 2024 - Rust
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
ocr grpc nvidia text-recognition text-detection inference-server fp16 tensorrt rag fastapi pdf-extraction paddleocr easyocr document-ai document-parsing qwen-vl gpu-ocr
- Updated
Jul 20, 2026 - C++
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
python nlp data-science machine-learning ai computer-vision deep-learning tensorflow transformers inference pytorch artificial-intelligence inference-server predict paddlepaddle model-deployment model-serving serving huggingface modelserver
- Updated
Feb 14, 2023 - Python
A REST API for Caffe using Docker and Go
- Updated
Jul 20, 2018 - C++
PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.
macos machine-learning ai deep-learning metal transformers tui lora quantization ane inference-server mlx fine-tuning distillation apple-silicon llm llm-training llm-inference qlora gguf
- Updated
Jun 5, 2026 - Rust
Work with LLMs on a local environment using containers
- Updated
Jul 28, 2026 - TypeScript
This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.
api docker dockerfile computer-vision deep-learning neural-network gpu rest-api inference yolo deeplearning inference-server bounding-boxes no-code yolov3 alexeyab-darknet yolo-gui detection-inference-api inference-gui yolov4
- Updated
Jun 28, 2022 - Python
Auto-tuned launcher for GGUF models on llama.cpp / ik_llama.cpp — OpenAI-compatible server with multi-GPU tensor-split, MoE expert placement, measured flag tuning (AI Tune), hardware-matched HuggingFace downloads, and crash recovery. An Ollama alternative for multi-GPU rigs.
golang metal vulkan cuda self-hosted moe inference-server multi-gpu openai-api llm llamacpp llama-cpp local-llm gguf speculative-decoding localllama ollama-alternative
- Updated
Jul 28, 2026 - Go
This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.
api docker opencv deep-neural-networks cpu computer-vision deep-learning neural-network rest-api inference object-detection inference-server bounding-boxes no-code yolov3 detection-inference-api cpu-inference-api inference-gui yolov4 yolov4-darknet
- Updated
Jun 28, 2022 - Python
ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.
machine-learning ai deep-learning cuda inference-server nueral-networks contributions-welcome onnx onnxruntime
- Updated
Jul 25, 2026 - C++
This is a repository for an object detection inference API using the Tensorflow framework.
api docker cpu computer-vision deep-learning docker-container tensorflow rest-api docker-image inference deeplearning object-detection tensorflow-framework predictions inference-server bounding-boxes computervision docker-ce inference-engine detection-inference-api
- Updated
Jun 28, 2022 - Python
Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints
inference pmml inference-server onnx onnx-models ai-serving pmml-model onnx-inference onnx-rest pmml-deployment pmml-rest pmml-grpc onnx-grpc pmml-realtime onnx-realtime pmml-inference
- Updated
Feb 24, 2026 - Scala
Orkhon: ML Inference Framework and Server Runtime
- Updated
Feb 1, 2021 - Rust
Deploy DL/ ML inference pipelines with minimal extra code.
python docker deep-learning websocket gunicorn pytorch falcon http-server triton gevent inference-server tensorflow-serving streaming-audio model-deployment model-serving serving tf-serving torchserve triton-inference-server triton-server
- Updated
Feb 10, 2026 - Python