# More

##### LocalAI

LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.

- [Overview | LocalAI documentation](https://localai.io/)
- GitHub: [https://github.com/mudler/LocalAI](https://github.com/mudler/LocalAI)

##### Xinference

Xorbits Inference (Xinference) is an open-source platform to streamline the operation and integration of a wide array of AI models. With Xinference, you’re empowered to run inference using any open-source LLMs, embedding models, and multimodal models either in the cloud or on your own premises, and create robust AI-driven applications.

- [Welcome to Xinference! — Xinference](https://inference.readthedocs.io/en/latest/)
- GitHub: [https://github.com/xorbitsai/inference](https://github.com/xorbitsai/inference)

##### NVIDIA NIM

Explore the latest community-built AI models with an API optimized and accelerated by NVIDIA, then deploy anywhere with NVIDIA NIM inference microservices.

- [NVIDIA NIM for Deploying Generative AI | NVIDIA](https://www.nvidia.com/en-us/ai/?ncid=ref-inpa-611229)
- Doc: [Introduction - NVIDIA Docs](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html)
- Models: [google / gemma-7b](https://docs.api.nvidia.com/nim/reference/google-gemma7b)
- YT: [Self-Host and Deploy Local LLAMA-3 with NIMs - YouTube](https://www.youtube.com/watch?v=OuQBxBrO2ms)

##### text-generation-webui

A Gradio web UI for Large Language Models.

<p class="callout info">只能執行本地模型，不支援外部模型 API。</p>

支援以下多重功能的 AI 平台

- Chat
- Fine-Tune Model
- Multiple model backends: Transformers, llama.cpp (through llama-cpp-python), ExLlamaV2, AutoGPTQ, AutoAWQ, GPTQ-for-LLaMa, QuIP#.
- OpenAI-compatible API server with Chat and Completions endpoints

教學

- GitHub: [https://github.com/oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui)
- GitHub: [https://github.com/Atinoda/text-generation-webui-docker](https://github.com/Atinoda/text-generation-webui-docker)
- [大型語言模型LLMs課程教學 課程大綱 (三) - HackMD](https://hackmd.io/@whYPD8MBSHWRZV6y-ymFwQ/HkHz7qvw6)  
    
    - YOUTUBE \[[啟動 TextGen](https://youtu.be/I-oLK-Olh7o)\]
    - YOUTUBE \[[上架大型語言模型](https://youtu.be/6Ual2nd4ezU)\]
    - YOUTUBE \[[指派AI人設](https://www.youtube.com/watch?v=Vx1tjNs9q-k)\]
    - YOUTUBE \[[模型微調](https://www.youtube.com/watch?v=LyWJtdJ9olc)\]
    - YOUTUBE \[[上架微調模型](https://www.youtube.com/watch?v=p2k1RjtJeS4)\]
    - 程式碼 [Z01\_TextGen\_Colab.ipynb](https://colab.research.google.com/github/c00cjz00/llm_course_20231228/blob/main/Z01_TextGen_Colab.ipynb)
    - 預設密碼在程式碼裡面 (account:nchc password:nchc) 請自行修改

##### koboldcpp

- GitHub: [https://github.com/LostRuins/koboldcpp](https://github.com/LostRuins/koboldcpp)
- 單機版運行/免安裝/無相依姓套件
- 支援 GGUF 模型
- 支援 OuteTTS (Text-To-Speech), Whisper (Speech-To-Text), 文字/圖像生成
- 內建 KoboldAI Lite UI