More

LocalAI

LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.

• Overview | LocalAI documentation
• GitHub: https://github.com/mudler/LocalAI 

OpenAI Proxy

Proxy Server to call 100+ LLMs in a unified interface & track spend, set budgets per virtual key/user

Features:

Unified Interface: Calling 100+ LLMs Huggingface/Bedrock/TogetherAI/etc. in the OpenAI ChatCompletions & Completions format Cost tracking: Authentication, Spend Tracking & Budgets Virtual Keys Load Balancing: between Multiple Models + Deployments of the same model - LiteLLM proxy can handle 1.5k+ requests/second during load tests.

企業在導入 LLM 時，可能會用到多種不同的模型，這些包含商用授權與開源授權以及來自不同的服務商。為了統一管理及開發應用這些各類不同模型，建議使用 OpenAI Proxy 這個平台來解決，以達到下列目的：

統一 API 介接入口與格式 成本追蹤 平衡負載

Doc: https://docs.litellm.ai/docs/simple_proxy 

Xinference

Xorbits Inference (Xinference) is an open-source platform to streamline the operation and integration of a wide array of AI models. With Xinference, you’re empowered to run inference using any open-source LLMs, embedding models, and multimodal models either in the cloud or on your own premises, and create robust AI-driven applications.

• Welcome to Xinference! — Xinference
• GitHub: https://github.com/xorbitsai/inference 

NVIDIA NIM

Explore the latest community-built AI models with an API optimized and accelerated by NVIDIA, then deploy anywhere with NVIDIA NIM inference microservices.

• NVIDIA NIM for Deploying Generative AI | NVIDIA
• Doc: Introduction - NVIDIA Docs
• Models: google / gemma-7b
• YT: Self-Host and Deploy Local LLAMA-3 with NIMs - YouTube

text-generation-webui

A Gradio web UI for Large Language Models.

只能執行本地模型，不支援外部模型 API。

支援以下多重功能的 AI 平台

• Chat
• Fine-Tune Model
• Multiple model backends: Transformers, llama.cpp (through llama-cpp-python), ExLlamaV2, AutoGPTQ, AutoAWQ, GPTQ-for-LLaMa, QuIP#.
• OpenAI-compatible API server with Chat and Completions endpoints

教學

• GitHub: https://github.com/oobabooga/text-generation-webui 
• GitHub: https://github.com/Atinoda/text-generation-webui-docker 
• 大型語言模型LLMs課程教學 課程大綱 (三) - HackMD
  
  • YOUTUBE [啟動 TextGen]
  • YOUTUBE [上架大型語言模型]
  • YOUTUBE [指派AI人設]
  • YOUTUBE [模型微調]
  • YOUTUBE [上架微調模型]
  • 程式碼 Z01_TextGen_Colab.ipynb
  • 預設密碼在程式碼裡面 (account:nchc password:nchc) 請自行修改

koboldcpp

• GitHub: https://github.com/LostRuins/koboldcpp 
• 單機版運行/免安裝/無相依姓套件
• 支援 GGUF 模型
• 支援 OuteTTS (Text-To-Speech), Whisper (Speech-To-Text), 文字/圖像生成
• 內建 KoboldAI Lite UI