GenAI

生成式人工智慧

LLM Models

Resources
Chinese LLMs
Code LLMs
Evaluation/Monitor
Function Calling LLMs
Content Safty
Hardware Requirements

Voice AI

Gen Audio
Instant voice cloning
Text to Speech (TTS)
ASR - Automatic Speech Recognition
MTK Breeze 3

聯發創新基地（MediaTek Research）發表全新 MediaTek Research Breeze 3（後略 MR Breeze 3）系列，包含台語語音辨識模型 Breeze ASR 26、台語語音合成模型 BreezyVoice 26，以及專為台灣設計的 AI 內容安全防護模型 Breeze Guard 26。

Vibevoice (Microsoft)

Microsoft VibeVoice 是一套開源語音 AI 模型家族，涵蓋 TTS（文字轉語音）與 ASR（語音辨識）。核心創新採用 7.5Hz 超低幀率連續語音 tokenizer，搭配  next-token diffusion 框架，能單次生成最長 90 分鐘多人對話語音、或辨識 60 分鐘長音訊。TTS 支援最多 4 人多語合成；ASR 能同時產出說話者、時間戳與內容的結構化逐字稿

Meetily

以隱私為先的人工智慧會議助理與筆記記錄工具

RAG

檢索增強生成 - Retrieval Augmented Generation

RAG 主要用來解決大型語言模型（LLM）實際應用時的兩大侷限：幻覺/錯覺（hallucination）與資料時限。RAG 結合「資訊檢索（retrieval）」和「生成（generation）」：在文字生成之前，先從資料庫中檢索相關的資料放入上下文，以確保 LLM 可依照正確的最新資訊生成結果。

RAG 優點：

流程示意圖

rag_flow.png

Introduction

Tutorials

Introduction to RAG

Embedding/Rerank Models
Vector Databases

Advanced RAG

ReRank
Chunking/Splitting

RAG Projects

Danswer

Danswer is the AI Assistant connected to your company's docs, apps, and people. Danswer provides a Chat interface and plugs into any LLM of your choice. Danswer can be deployed anywhere and for any scale - on a laptop, on-premise, or to cloud.

Embedchain

Embedchain streamlines the creation of personalized LLM applications, offering a seamless process for managing various types of unstructured data.

GraphRAG

微軟開源一個基於圖譜的檢索與推理增強的解決方案。GraphRAG 透過從預檢索、後檢索到提示壓縮的過程中考慮知識圖譜的檢索與推理，為回答生成提供了一種更精準和相關的方法。

neo4j

Verba

Verba is a fully-customizable personal assistant for querying and interacting with your data, either locally or deployed via cloud. Resolve questions around your documents, cross-reference multiple data points or gain insights from existing knowledge bases. Verba combines state-of-the-art RAG techniques with Weaviate's context-aware database. Choose between different RAG frameworks, data types, chunking & retrieving techniques, and LLM providers based on your individual use-case.

PrivateGPT

LLMWare

The Ultimate Toolkit for Enterprise RAG Pipelines with Small, Specialized Models.

talkd/dialog

Talkd.ai—Optimizing LLMs with easy RAG deployment and management.

RAG 評估

評估生成（Generation）指標

評估檢索（Retrieval）指標

URLs

Fine-Tune

模型微調工作流程

  1. 準備資料集(訓練資料)
  2. 準備基礎模型
  3. 匯入資料集
  4. 開始微調作業 (Fine-Tune)
  5. 評估新模型損失曲線
  6. 以新模型做實際推論

準備資料集

開始微調模型之前，您必須先建立用來微調模型的資料集。為獲得最佳效能，資料集內的範例必須具有高品質、多元且代表真實輸入和輸出的要素。

格式

資料集中包含的範例應符合您預期的實際工作環境流量。如果您的資料集含有特定格式、關鍵字、操作說明或資訊，則實際工作環境資料的格式應相同，並含有相同的指示。 例如，如果資料集中的範例包含 "question:" 和 "context:"，則實際工作環境流量也應一併設定包含 "question:" 和 "context:" 的格式，且順序應與資料集範例中的順序相同。如果您排除結構定義，即使資料集的範例包含確切的問題，模型將無法辨識模式。

在資料集中的每個範例中加入提示或前置碼，也有助於改善調整後模型的效能。請注意，如果資料集中包含提示或前置碼，那麼在推論時向已調整的模型發出提示時，也應包含該提示或前置碼。

Tools & Platform

Unsloth

Unsloth - Easily finetune & train LLMs

微調模型專用的 Python 函式庫，在地端使用 GPU 資源對各種 Open Source 模型進行微調作業。

Atlas

Atlas by NOMIC - 資料集（非結構化資料）品質檢測服務

AnythingLLM

具有 Chat/Fine-Tune/Multi-Model 多功能的平台

LLaMA-Factory
outlines

生成結構化文字資料。可用於微調模型前的資料集預處理。

InstructLab (IBM)

Models

Gemini-Pro

要微調 Gemini-Pro 模型，有三種不同方式呼叫 Gemini API 來做微調作業，Google AI Studio、Python SDK、REST API (curl)。

Mistral

官方 Mistral AI 推出微調用 SDK 與 API。

AI Applications

Chat AI

Cherry Studio

Cherry Studio is a desktop client that supports for multiple LLM providers, available on Windows, Mac and Linux.

Chatbox

Chatbox支援多款全球最先進的AI大模型服務，支援Windows、Mac和Linux。AI提升工作效率，深受全世界專業人士的好評。

KHOJ

Your AI second brain

Paper AI

Elicit - 論文分析
asXiv
GPT Academic

为GPT/GLM等LLM大语言模型提供实用化交互接口，特别优化论文阅读/润色/写作体验，模块化设计，支持自定义快捷按钮&函数插件，支持Python和C++等项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持chatglm3等本地模型。接入通义千问, deepseekcoder, 讯飞星火, 文心一言, llama2, rwkv, claude2, moss等。

Presentation AI

PDF Extractor

OCR

More AI

aidocx

運用 AI 自動生成特定知識的技術書籍(*.epub)

WrenAI - text-to-SQL

WrenAI is a text-to-SQL solution for data teams to get results and insights faster by asking business questions without writing SQL.

QAnything

開源的企業級本地知識庫問答及應用

PageLM

PageLM 是由開源社區 CaviraOSS 發起并維護的一個社區驅動項目，它在設計理念上受到了 Google NotebookLM 的啓發，但走出了一條完全不同的路徑——開源、免費、多模型支持。

Vane (Perplexica)

平替 Perplexity 的自架方案

Open Notebook

平替 Google NotebookLM

AI Dev

AI Develop Framework

- LlamaIndex

Data Analysis (Chat with CSV)

- PandasAI

PandasAI is a Python library that makes it easy to ask questions to your data in natural language. It helps you to explore, clean, and analyze your data using generative AI.

LLM API

- OpenAI API
- Gemini API

Web UI Framework

- Gradio

Gradio is the fastest way to demo your machine learning model with a friendly web interface so that anyone can use it, anywhere!

- Streamlit

Streamlit is the UI powering the LLM movement

AI Memory

AI Coding

- Alternative to GitHub Copilot
- VS Code
- Cursor

Responsible AI

AI Proxy

企業在導入 LLM 時，可能會用到多種不同的模型，這些包含商用授權與開源授權以及來自不同的服務商。為了統一管理及開發應用這些各類不同模型，建議使用 AI Proxy (AI API Gateway) 這類平台來解決，以達到下列目的：

- LiteLLM
- CLI Proxy API
- Sub2API

More

Learning AI

AI 常見專用名詞

Gen AI (生成式 AI)

人工智慧 (AI) 藉由使用機器學習與環境互動並執行工作來模擬人類行為，而不需明確指示輸出的內容。

生成式 AI 是人工智慧的分支，可根據自然語言輸入來建立新內容。 生成式 AI 通常內建於軟體應用程式中，並使用經過大量文字資料定型的語言模型，以產生人類般的自然語言回應，甚至是原始影像。 這類應用程式的其中一個熱門範例是 ChatGPT，這是 OpenAI 所建立的聊天機器人，這是一家與 Microsoft 緊密合作的 AI 研究公司。

生成式 AI 是由遠超過人類一輩子閱讀量的文字、影像、聲音所訓練而成，但卻缺乏普通人類的價值觀及基本判斷能力。「他」就像是個博學多聞、過目不忘的孩子，卻缺乏生活常識；偶爾胡說八道，又時常過度坦誠，因而需要隨時照料。因此，無論單純利用 AI 產生內容，或是將 AI 包裝為自家服務的公司，都應該特別小心謹慎。

LLM (大型語言模型)

語言模型支援的一般自然語言處理(NLP) 工作包括:

其他

Introduction

Medium Articles

Course/HandBook

Google AI Courses for Free
Microsoft

國網中心(NCHC)教學

LLM Tokenizer 分詞器

PyImageSearch 教學 (英文)

AI 各類資源大匯集

AI Research Paper

AI News

資策會

下載指引：下載專區 | 資策會 (iii.org.tw)

Open Source MLOps platform

LLM 面試問答

Cheat Sheets

Types of AI models

types_of_ai_models.jpeg

RedHat AI

Red Hat® Enterprise Linux® AI is a foundation model platform to seamlessly develop, test, and run Granite family large language models (LLMs) for enterprise applications.

Red Hat Enterprise Linux AI brings together:

URLs:

InstructLab

Command-line interface. Use this to chat with the model or train the model (training consumes the taxonomy data)

What are the components of the InstructLab project?

How is InstructLab different from retrieval-augmented generation (RAG)?

RAG is a cost-efficient method for supplementing an LLM with domain-specific knowledge that wasn’t part of its pretraining. RAG makes it possible for a chatbot to accurately answer questions related to a specific field or business without retraining the model. Knowledge documents are stored in a vector database, then retrieved in chunks and sent to the model as part of user queries. This is helpful for anyone who wants to add proprietary data to an LLM without giving up control of their information, or who needs an LLM to access timely information.

This is in contrast to the InstructLab method, which sources end-user contributions to support regular builds of an enhanced version of an LLM. InstructLab helps add knowledge and unlock new skills of an LLM.

It’s possible to "supercharge" a RAG process by using the RAG technique on an InstructLab-tuned model.

URLs:

AI Cloud Providers

LLM API
Data Analysis
Dev Platform
Code Review
Monitor AP in developing

Function Calling

LLM（Large Language Model）的一個新功能是 Function Calling（函數呼叫）。這項功能允許LLM直接呼叫外部函數，並將其結果整合到模型的輸出中。下面是它的工作原理和應用： 
工作原理
 Function Calling 的工作原理是，LLM 在生成輸出時，可以 reconocize 具體的函數名稱和參數，並將其傳遞給外部函數執行。外部函數執行後，將結果返回給LLM，LLM 則將其整合到輸出中。這個過程可以重複多次，實現多個函數的呼叫和整合。

範例
例如，假設有一個LLM需要生成一個天氣報告，LLM 可以呼叫一個外部函數，該函數可以從天氣API中獲取当前的天氣資訊。 LLＭ將函數的結果整合到輸出中，生成一個完整的天氣報告。

應用
Function Calling 的應用非常廣泛，以下是一些例子：
總之，Function Calling 是 LLM 的一個強大功能，可以擴展模型的能力，實現更加 Complex 和多樣化的任務。
Tutorials
Models

Python Coding

LLM Model API

LMStudio
from langchain.llms import OpenAI

#set llm for langchain using model from lmstudio
llm = OpenAI(
       openai_api_base='http://localhost:1234/v1',
       openai_api_key='NULL'
       )
import streamlit as st
from openai import OpenAI

# Set up the Streamlit App
st.title("ChatGPT Clone using Llama-3 🦙")
st.caption("Chat with locally hosted Llama-3 using the LM Studio 💯")

# Point to the local server setup using LM Studio
client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

# Initialize the chat history
if "messages" not in st.session_state:
    st.session_state.messages = []

# Display the chat history
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

# Accept user input
if prompt := st.chat_input("What is up?"):
    # Add user message to chat history
    st.session_state.messages.append({"role": "user", "content": prompt})
    # Display user message in chat message container
    with st.chat_message("user"):
        st.markdown(prompt)
    # Generate response
    response = client.chat.completions.create(
        model="lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF",
        messages=st.session_state.messages, temperature=0.7
    )
    # Add assistant response to chat history
    st.session_state.messages.append({"role": "assistant", "content": response.choices[0].message.content})
    # Display assistant response in chat message container
    with st.chat_message("assistant"):
        st.markdown(response.choices[0].message.content)

GPT

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
    # api_key="...",
    # base_url="...",
    # organization="...",
    # other params...
)

Ollama

from langchain_community.llms import Ollama

llm = Ollama(model="llama2:13b")
llm.invoke("The first man on the moon was ... think step by step")

Chunking/Splitting

中文句子切割

# Unicode 編碼
#   \u3002 全形句號
#   \uff0c 全形逗號
# Get Unicode for specific character
# >>> '，'.encode('unicode-escape') # for py3
# >>> list(u'，') # for py2

import re
text = "這是中文句子。第一段，第二段，第三段。"
chunks = re.split('[\u3002\uff0c]', text)
#print("\n\n".join([chunk for chunk in chunks]))
for chunk in chunks:
    print("---" * 10)
    print(chunk)

英文句子切割

# \s+ 單或多個空白
chunks = re.split(r'(?<=[.?!])\s+', text)

PDF to Markdown

import os
import pymupdf.layout
import pymupdf4llm
from pathlib import Path
import glob

os.environ["TOKENIZERS_PARALLELISM"] = "false"

def pdf_to_markdown(pdf_path, output_dir):
    doc = pymupdf.open(pdf_path)
    md = pymupdf4llm.to_markdown(doc, header=False, footer=False, page_separators=True, ignore_images=True, write_images=False, image_path=None)
    md_cleaned = md.encode('utf-8', errors='surrogatepass').decode('utf-8', errors='ignore')
    output_path = Path(output_dir) / Path(doc.name).stem
    Path(output_path).with_suffix(".md").write_bytes(md_cleaned.encode('utf-8'))

def pdfs_to_markdowns(path_pattern, overwrite: bool = False):
    output_dir = Path(MARKDOWN_DIR)
    output_dir.mkdir(parents=True, exist_ok=True)

    for pdf_path in map(Path, glob.glob(path_pattern)):
        md_path = (output_dir / pdf_path.stem).with_suffix(".md")
        if overwrite or not md_path.exists():
            pdf_to_markdown(pdf_path, output_dir)

pdfs_to_markdowns(f"{DOCS_DIR}/*.pdf")

Prompt

def get_rag_agent_prompt() -> str:
    return """
        You are a retrieval-augmented assistant.

        You are NOT allowed to answer immediately.

        Before producing ANY final answer, you must first perform a document search
        and observe retrieved content.

        If you have not searched, the answer is invalid.

        Workflow:
        1. Search the documents using the user query.
        2. Inspect retrieved excerpts and keep only relevant ones.
        ...
        """


LLM Engine

A software that can load the LLM Models

LLM Engine

Open WebUI

A Web UI Tool for Ollama

URLs
Installation

Installing Both Open WebUI and Ollama Together:

# With GPU Support
docker run -d -p 3000:8080 --gpus=all \
  -v ollama:/root/.ollama \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:ollama
# For CPU only
docker run -d -p 3000:8080 \
  -v ollama:/root/.ollama \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:ollama

LLM Engine

Kuwa Gen AI OS

一個自由、開放、安全且注重隱私的生成式人工智慧服務系統，包括友善的大語言模型使用介面，以及能支援生成式人工智慧應用的新型GenAI核心。

  1. 🌐 提供多語言GenAI開發與部署的整體解決方案，支援Windows及Linux
  2. 💬 提供群聊、引用、完整 Prompt 列表的匯入/匯出/分享等友善使用功能
  3. 🔄 可靈活組合 Prompt x RAGs x Bot x 模型 x 硬體/GPUs以滿足應用所需
  4. 💻 支援從虛擬主機、筆記型電腦、個人電腦、地端伺服器到公私雲端的各種環境
  5. 🔓 開放原始碼，允許開發人員貢獻並根據自己的需求打造自己的客製系統
URLs
LLM Engine

AnythingLLM

The ultimate AI business intelligence tool. Any LLM, any document, full control, full privacy.

AnythingLLM is a "single-player" （單機個人）application you can install on any Mac, Windows, or Linux operating system and get local LLMs, RAG, and Agents with little to zero configuration and full privacy.

AnythingLLM 也有自架網站版，見文章下方的連結。

You can install AnythingLLM as a Desktop Application, Self Host it locally using Docker and Host it on cloud (aws, google cloud, railway etc..) using Docker

You want AnythingLLM Desktop if...

URLs
LLM Engine

Ollama

Run Llama 3, Phi 3, Mistral, Gemma, and other models. Customize and create your own.

Installation

ollama + open webui
mkdir ollama-data download open-webui-data

docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - 11434:11434
    volumes:
      - ./ollama-data:/root/.ollama
      - ./download:/download
    container_name: ollama
    pull_policy: always
    tty: true
    restart: always
    networks:
      - ollama-docker

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - ./open-webui-data:/app/backend/data
    depends_on:
      - ollama
    ports:
      - 3000:8080
    environment:
      - 'OLLAMA_BASE_URL=http://ollama:11434'
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - ollama-docker

networks:
  ollama-docker:
    external: false
ollama
mkdir ollama-data download

docker run --name ollama -d --rm \
    -v $PWD/ollama-data:/root/.ollama \
    -v $PWD/download:/download \
    -p 11434:11434 \
    ollama/ollama
K8s Deployment

1. 啟用 hostpath-storage 

microk8s enable hostpath-storage
microk8s status

Verify the Storage Class

❯ kubectl get storageclass
NAME                          PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
microk8s-hostpath (default)   microk8s.io/hostpath   Delete          WaitForFirstConsumer   false                  17m

2. ollama-pvc.yaml : 

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-pvc
  namespace: ollama
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

3. ollama-deployment.yaml :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          env:
            - name: OLLAMA_HOST
              value: 0.0.0.0:11434
          ports:
            - name: http
              containerPort: 11434
              protocol: TCP
          volumeMounts:
            - name: ollama-data
              mountPath: /root/.ollama
      volumes:
        - name: ollama-data
          persistentVolumeClaim:
            claimName: ollama-pvc

4. ollama-svc.yaml : 

apiVersion: v1
kind: Service
metadata:
  name: ollama-service
  namespace: ollama
spec:
  selector:
    app: ollama
  ports:
  - protocol: TCP
    port: 11434
    targetPort: 11434
  type: ClusterIP

Testing with curl

curl -s http://<NODE_IP>:<nodeport>/api/generate -d '{
  "model": "llama2",
  "prompt": "Why is the sky blue?"
}' | jq -r '.response' | tr -d '\n'

Verify GPU support

kubectl logs -n ollama -l name=ollama 

The last line in the example output above shows that Ollama is using a single Tesla V100-SXM2-16GB GPU.

2024/09/27 18:51:55 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-09-27T18:51:55.719Z level=INFO source=images.go:753 msg="total blobs: 0"
time=2024-09-27T18:51:55.719Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-09-27T18:51:55.719Z level=INFO source=routes.go:1200 msg="Listening on [::]:11434 (version 0.3.12)"
time=2024-09-27T18:51:55.720Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 cuda_v11 cuda_v12 cpu]"
time=2024-09-27T18:51:55.720Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2024-09-27T18:51:55.942Z level=INFO source=types.go:107 msg="inference compute" id=GPU-d8c505a1-8af4-7ce4-517d-4f57fa576097 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB"

Models

List Models Installed

ollama list

Load a GGUF model manually

ollama create <my-model-name> -f <modelfile>

Page Assist

Page Assist is an open-source Chrome Extension that provides a Sidebar and Web UI for your Local AI model.

LLM Engine

LM Studio

Discover, download, and run local LLMs.

With LM Studio, you can ...

🤖 - Run LLMs on your laptop, entirely offline
👾 - Use models through the in-app Chat UI or an OpenAI compatible local server
📂 - Download any compatible model files from HuggingFace
🤗 repositories
🔭 - Discover new & noteworthy LLMs in the app's home page
URLs

 

LLM Engine

OpenLLM

OpenLLM helps developers run any open-source LLMs, such as Llama 2 and Mistral, as OpenAI-compatible API endpoints, locally and in the cloud, optimized for serving throughput and production deployment.

Install

Recommend using a Python Virtual Environment

pip install openllm
Start a LLM Server
openllm start microsoft/Phi-3-mini-4k-instruct --trust-remote-code

To interact with the server, you can visit the web UI at http://localhost:3000/ or send a request using curl. You can also use OpenLLM’s built-in Python client to interact with the server:

import openllm

client = openllm.HTTPClient('http://localhost:3000')
client.generate('Explain to me the difference between "further" and "farther"')
OpenAI Compatible Endpoints
import openai

client = openai.OpenAI(base_url='http://localhost:3000/v1', api_key='na')  # Here the server is running on 0.0.0.0:3000

completions = client.chat.completions.create(
  prompt='Write me a tag line for an ice cream shop.', model=model, max_tokens=64, stream=stream
)
LangChain
from langchain.llms import OpenLLMAPI

llm = OpenLLMAPI(server_url='http://44.23.123.1:3000')
llm.invoke('What is the difference between a duck and a goose? And why there are so many Goose in Canada?')

# streaming
for it in llm.stream('What is the difference between a duck and a goose? And why there are so many Goose in Canada?'):
  print(it, flush=True, end='')

# async context
await llm.ainvoke('What is the difference between a duck and a goose? And why there are so many Goose in Canada?')

# async streaming
async for it in llm.astream('What is the difference between a duck and a goose? And why there are so many Goose in Canada?'):
  print(it, flush=True, end='')

 

 

LLM Engine

Bechmark

Benchmark for LLM engines

bench.py
import aiohttp
import asyncio
import time
from tqdm import tqdm

import random

questions = [
    "Why is the sky blue?", "Why do we dream?", "Why is the ocean salty?", "Why do leaves change color?",
    "Why do birds sing?", "Why do we have seasons?", "Why do stars twinkle?", "Why do we yawn?",
    "Why is the sun hot?", "Why do cats purr?", "Why do dogs bark?", "Why do fish swim?",
    "Why do we have fingerprints?", "Why do we sneeze?", "Why do we have eyebrows?", "Why do we have hair?",
    "Why do we have nails?", "Why do we have teeth?", "Why do we have bones?", "Why do we have muscles?",
    "Why do we have blood?", "Why do we have a heart?", "Why do we have lungs?", "Why do we have a brain?",
    "Why do we have skin?", "Why do we have ears?", "Why do we have eyes?", "Why do we have a nose?",
    "Why do we have a mouth?", "Why do we have a tongue?", "Why do we have a stomach?", "Why do we have intestines?",
    "Why do we have a liver?", "Why do we have kidneys?", "Why do we have a bladder?", "Why do we have a pancreas?",
    "Why do we have a spleen?", "Why do we have a gallbladder?", "Why do we have a thyroid?", "Why do we have adrenal glands?",
    "Why do we have a pituitary gland?", "Why do we have a hypothalamus?", "Why do we have a thymus?", "Why do we have lymph nodes?",
    "Why do we have a spinal cord?", "Why do we have nerves?", "Why do we have a circulatory system?", "Why do we have a respiratory system?",
    "Why do we have a digestive system?", "Why do we have an immune system?"
]

async def fetch(session, url):
    """
    参数:
        session (aiohttp.ClientSession): 用于请求的会话。
        url (str): 要发送请求的 URL。
    
    返回:
        tuple: 包含完成 token 数量和请求时间。
    """
    start_time = time.time()

    # 随机选择一个问题
    question = random.choice(questions) # <--- 这两个必须注释一个

    # 固定问题                                 
    # question = questions[0]             # <--- 这两个必须注释一个

    # 请求的内容
    json_payload = {
        "model": "llama3:8b-instruct-fp16",
        "messages": [{"role": "user", "content": question}],
        "stream": False,
        "temperature": 0.7 # 参数使用 0.7 保证每次的结果略有区别
    }
    async with session.post(url, json=json_payload) as response:
        response_json = await response.json()
        end_time = time.time()
        request_time = end_time - start_time
        completion_tokens = response_json['usage']['completion_tokens'] # 从返回的参数里获取生成的 token 的数量
        return completion_tokens, request_time

async def bound_fetch(sem, session, url, pbar):
    # 使用信号量 sem 来限制并发请求的数量，确保不会超过最大并发请求数
    async with sem:
        result = await fetch(session, url)
        pbar.update(1)
        return result

async def run(load_url, max_concurrent_requests, total_requests):
    """
    通过发送多个并发请求来运行基准测试。
    
    参数:
        load_url (str): 要发送请求的URL。
        max_concurrent_requests (int): 最大并发请求数。
        total_requests (int): 要发送的总请求数。
    
    返回:
        tuple: 包含完成 token 总数列表和响应时间列表。
    """
    # 创建 Semaphore 来限制并发请求的数量
    sem = asyncio.Semaphore(max_concurrent_requests)
    
    # 创建一个异步的HTTP会话
    async with aiohttp.ClientSession() as session:
        tasks = []
        
        # 创建一个进度条来可视化请求的进度
        with tqdm(total=total_requests) as pbar:
            # 循环创建任务，直到达到总请求数
            for _ in range(total_requests):
                # 为每个请求创建一个任务，确保它遵守信号量的限制
                task = asyncio.ensure_future(bound_fetch(sem, session, load_url, pbar))
                tasks.append(task)  # 将任务添加到任务列表中
            
            # 等待所有任务完成并收集它们的结果
            results = await asyncio.gather(*tasks)
        
        # 计算所有结果中的完成token总数
        completion_tokens = sum(result[0] for result in results)
        
        # 从所有结果中提取响应时间
        response_times = [result[1] for result in results]
        
        # 返回完成token的总数和响应时间的列表
        return completion_tokens, response_times

if __name__ == '__main__':
    import sys

    if len(sys.argv) != 3:
        print("Usage: python bench.py <C> <N>")
        sys.exit(1)

    C = int(sys.argv[1])  # 最大并发数
    N = int(sys.argv[2])  # 请求总数

    # vllm 和 ollama 都兼容了 openai 的 api 让测试变得更简单了
    url = 'http://localhost:11434/v1/chat/completions'

    start_time = time.time()
    completion_tokens, response_times = asyncio.run(run(url, C, N))
    end_time = time.time()

    # 计算总时间
    total_time = end_time - start_time
    # 计算每个请求的平均时间
    avg_time_per_request = sum(response_times) / len(response_times)
    # 计算每秒生成的 token 数量
    tokens_per_second = completion_tokens / total_time

    print(f'Performance Results:')
    print(f'  Total requests            : {N}')
    print(f'  Max concurrent requests   : {C}')
    print(f'  Total time                : {total_time:.2f} seconds')
    print(f'  Average time per request  : {avg_time_per_request:.2f} seconds')
    print(f'  Tokens per second         : {tokens_per_second:.2f}')

LLM Engine

More

LocalAI

LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.

Xinference

Xorbits Inference (Xinference) is an open-source platform to streamline the operation and integration of a wide array of AI models. With Xinference, you’re empowered to run inference using any open-source LLMs, embedding models, and multimodal models either in the cloud or on your own premises, and create robust AI-driven applications.

NVIDIA NIM

Explore the latest community-built AI models with an API optimized and accelerated by NVIDIA, then deploy anywhere with NVIDIA NIM inference microservices.

text-generation-webui

A Gradio web UI for Large Language Models.

只能執行本地模型，不支援外部模型 API。

支援以下多重功能的 AI 平台

教學

koboldcpp
LLM Engine

Llama.Cpp

GitHub: https://github.com/ggml-org/llama.cpp 

Tutorials

AI Translator

使用 LLM 實現語言翻譯

Huyuan-MT

Huyuan-MT - 支援 33 種語言和 5 種中文方言

PDFMathTranslate

完整保留排版的 PDF 檔案全文雙語翻譯，支援 Google/DeepL/Ollama/OpenAI 翻譯。

LiteLLM + 反思提示 + 工作流

Translation Agent

RTranslator

RTranslator is an (almost) open-source, free, and offline real-time translation app for Android.

沉浸式翻譯

一款免費的，好用的，沒有廢話的，革命性的，飽受讚譽的，AI 驅動的雙語網頁翻譯擴展，幫助你有效地打破資訊差，在手機上也可以用！

影片/字幕

- pyVideoTrans视频翻译配音

一键字幕生成+字幕翻译+创建配音+合成 = 带字幕和配音的新视频

- VideoLingo 

Netflix级字幕切割、翻译、对齐、甚至加上配音，一键全自动视频搬运AI字幕组

- SubtitleEdit

使用 .Net 開發，適合 Windows 用戶，AI 生成/翻譯字幕，字幕編輯功能豐富。

- Vibe

Whisper AI 轉譯語音檔或影片檔字幕

- VideoCaptioner
- Violin

bilingual_book_maker

電子書翻譯

MTranServer

自行部署離線的翻譯伺服器，翻譯軟體可用沉浸式翻譯與簡約翻譯。

AiNiee

一款專注於Ai翻譯的工具，一鍵自動翻譯RPG SLG游戲，Epub TXT小說，Srt Vtt Lrc字幕，Word MD 檔案等等復雜長文字。

Pot

一個跨平台的劃詞翻譯軟體

OpenAI Translator

OpenAI Translator - 基於 ChatGPT API 的翻譯擴充功能，Chrome、Edge 都能用

Argos Translate

Open source neural machine translation

TranslateGemma

TranslateGemma 是一套以 Google Gemma 3 模型為基礎建立的全新開放式翻譯模型，提供 40 億（4B）、120 億（12B）、270 億（27B）三種參數規模的版本，幫助人們無論身在何處、使用何種裝置，能夠跨越 55 種語言進行交流。

Jupyter Notebook

Installation

With pip

pip install notebook
Python Virtual Environment

With Python Venv

mkdir my-rag
cd my-rag
python -m venv .venv
source .venv/bin/activate
(my-rag)> pip install --upgrade pip
(my-rag)> pip install notebook
(my-rag)> jupyter notebook

With Conda

conda create -n my-rag python=3.10
conda activate my-rag
(my-rag)> pip install --upgrade pip
(my-rag)> pip install notebook
(my-rag)> jupyter notebook

UI 可切換不同虛擬環境（需要先建立不同的 ipykernel）

mkdir my-rag
cd my-rag
python -m venv .venv
source .venv/bin/activate
(my-rag)> pip install --upgrade pip
(my-rag)> pip install ipykernel
(my-rag)> ipython kernel install --user --name="my-rag-kernel"
(my-rag)> jupyter notebook

Tips

Secure Authentication
  1. In the top menu bar of the notebook editor, select Add-ons then Secrets.
  2. Create a new secret with the label GOOGLE_API_KEY.
  3. Paste your API key into the "Value" field and click "Save".
  4. Ensure that the checkbox next to GOOGLE_API_KEY is selected so that the secret is attached to the notebook.
import os
from kaggle_secrets import UserSecretsClient

try:
    GOOGLE_API_KEY = UserSecretsClient().get_secret("GOOGLE_API_KEY")
    os.environ["GOOGLE_API_KEY"] = GOOGLE_API_KEY
    print("✅ Gemini API key setup complete.")
except Exception as e:
    print(
        f"🔑 Authentication Error: Please make sure you have added 'GOOGLE_API_KEY' to your Kaggle secrets. Details: {e}"
    )

Resources

CoLab by Google

LangChain

LangChain 是一個旨在為開發者提供一套工具和程式介接，以便更容易、更有效地利用大型語言模型（LLM）的開源開發框架，專注於情境感知和推理。它包含多個組件，如 Python 和 JavaScript 的函式庫、快速部署的模板、用於開發REST API的 LangServe，以及用於除錯和監控的 LangSmith。LangChain 簡化了開發、生產和部署過程，提供與語言模型互動、執行檢索策略和輔助建立複雜應用架構的工具。

LangSmith

LangChain 提供的雲端服務，可用來作程式除錯與監視後端程序，例如 RAG 的檢索資訊過程。

RAG

Retrievers in LCEL
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

template = """Answer the question based only on the following context:

{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)
model = ChatOpenAI()


def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])


chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)

chain.invoke("What did the president say about technology?")

ChatPromptTemplate

few_shot_examples = [
{"input":"Could you please clarify the terms outlined in section 3.2 of the contract?",
"output":"Certainly, I will provide clarification on the terms in section 3.2."},
{"input":"We are interested in extending the payment deadline to 30 days instead of the current 15 days. Additionally, we would like to add a clause regarding late payment penalties.",
"output":"Our request is to extend the payment deadline to 30 days and include a clause on late payment penalties."},
{"input":"""The current indemnification clause seems too broad. We would like to narrow it down to cover only direct damages and exclude consequential damages.
Additionally, we propose including a dispute resolution clause specifying arbitration as the preferred method of resolving disputes.""",
"output":"""We suggest revising the indemnification clause to limit it to covering direct damages and excluding consequential damages.
Furthermore, we recommend adding a dispute resolution clause that specifies arbitration as the preferred method of resolving disputes."""},
{"input":"I believe the proposed changes are acceptable.",
"output":"Thank you for your feedback. I will proceed with implementing the proposed changes."}
]

few_shot_template = ChatPromptTemplate.from_messages(
    [
        ("human", "{input}"),
        ("ai", "{output}")
    ]
)

few_shot_prompt = FewShotChatMessagePromptTemplate(
    example_prompt=few_shot_template,
    examples=few_shot_examples,
)

print(few_shot_prompt.format())
custom_prompt = ChatPromptTemplate.from_template("""
You are an information extraction assistant.
Read the text below and identify important entities.

**Extraction rules:**
- Always extract the **Report Id** (this is the central node).
- Extract **people**, **institutions**, **places**, **dates**, **monetary amounts**, and **vehicle registration numbers** (e.g., MH12AB1234, PK-02-4567, KA05MG2020).
- Do not ignore any people names; extract all mentioned in the document, even if they seem minor or role not clear.
  Treat all of types of vehicles (eg; cars, bikes etc) as the same kind of entity called "Vehicle".

**Output format:**
1. List all nodes (unique entities).
2. Identify the central node (Report Id).
3. Create relationships of the form:
   (Report Id)-[HAS_ENTITY]->(Entity),
4. Do not create any other types of relationships.                                            

Text:
{input}

Return only structured data like:
Nodes:
- Report SYN-REP-2024
- Honda bike ABCD1234
- XYZ College, Chennai
- ...
""")

 

Input Data Loader

Web
from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()
Text file
from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader("../", glob="**/*.md")
docs = loader.load()
len(docs)
print(docs[0].page_content[:100])
from langchain.document_loaders import TextLoader

dataset_folder_path='/path/to/dataset/'
documents=[]
for file in os.listdir(dataset_folder_path):
  loader=TextLoader(dataset_folder_path+file)
  documents.extend(loader.load())
  
print(documents[:3])
Markdown file
'''
%pip install "unstructured[md]"
'''
from langchain_community.document_loaders import UnstructuredMarkdownLoader
markdown_path = "../../../README.md"
loader = UnstructuredMarkdownLoader(markdown_path)

data = loader.load()
assert len(data) == 1
readme_content = data[0].page_content
print(readme_content[:3])
PDF + Text file
from langchain_community.document_loaders import TextLoader
from langchain_community.document_loaders import PyPDFLoader

documents = []
for filename in SAMPLEDATA:
    path = os.path.join(os.getcwd(), filename)

    if filename.endswith(".pdf"):
        loader = PyPDFLoader(path)
        new_docs = loader.load_and_split()
        print(f"Processed pdf file: {filename}")
    elif filename.endswith(".txt"):
        loader = TextLoader(path)
        new_docs = loader.load_and_split()
        print(f"Processed txt file: {filename}")
    else:
        print(f"Unsupported file type: {filename}")

    if len(new_docs) > 0:
        documents.extend(new_docs)

SAMPLEDATA = []

print(f"\nProcessing done.")

OCR

常用函式

格式化輸出

# Helper function for printing docs
def pretty_print_docs(docs):
    print(
        f"\n{'-' * 100}\n".join(
            [f"Document {i+1}:\n\n" + d.page_content for i, d in enumerate(docs)]
        )
    )

Finance AI

OpenBB

Investment research made easy with AI.

StockBot
FinGPT
TradingAgents

TradingAgents Framework 這是一套模仿真實交易公司運作的多智能體大型語言模型（LLM）系統，為自動化交易提供了更具動態性且符合現實組織邏輯的新路徑。

MCP Servers

Access the official remote MCP server for Financial Datasets.

股票智能分析系统

基於 LLM 的多市場股票智能分析系統，支援 A 股 / 港股 / 美股 / 日股 / 韓股自選股分析。

能把行情、K 線、技術指標、資金流、籌碼、新聞、公告、基本面等信息聚合起來，再自動生成結構化的「決策儀表盤」。

核心結論、評分、趨勢判斷、風險警報、催化因素、操作檢查清單，都能一併看清。

Awesome Finance Skills

一系列開源免費的金融分析Agent Skills

Kronos

首個面向金融K線圖的開源基礎模型， 基於全球超過45家交易所的數據訓練而成。

Dexter

美股研究開源神器來了！專攻美股金融研究的自主 Agent！

核心優勢有以下幾點：

專為美股金融研究打造，深度支持美股（Financial Datasets + Yahoo Finance 提供機構級財報、earnings call、歷史數據，覆蓋非常絲滑）。

Fincept Terminal

Semantic Kernel

Semantic Kernel 是一款由微軟開放的輕量級的 AI 開發套件(框架)，可讓您輕鬆建立 AI 代理，並將最新的 AI 模型整合到您的 C#、Python 或 Java 程式碼庫中。它可作為有效率的中介軟體，讓您快速交付企業級解決方案。

微軟教學：

中文教學：

Legal AI

法律 AI

NVIDIA - Jetson Orin Nano Super

Hardware

JetPack SDK

檢查目前 Jetpack 版本

Tip: 先找出 L4T 版本後，前往 https://developer.nvidia.com/embedded/jetpack-archive  可以找出對應的 Jetpack 版本。
dpkg -l | grep 'nvidia-l4t-core'

SDK Manager

官方的 Jetson 系列設備的刷機軟體，需安裝在另一個 Linux 工作站。設備刷機前需要先切換成 Recovery Mode，並且以 USB-C 線連接 Linux 工作站與設備。

Install SDK Manager

安裝方式支援 Linux、Windows、Docker 平台。

Recovery Mode

還原模式用途與特性：

  1. 設備開機時會跳過正常開機程序，以最簡易環境啟動。
  2. 可進行系統刷機及韌體更新作業。

切換方式：將 J14 (Button Header) 的 Pin 9-10 短路。

Headless Setup

VNC Server
sudo apt-get update
sudo apt-get install tightvncserver
vncserver

# Start VNC server
vncserver -geometry 1980x1080

# Kill VNC Server
vncserver -kill :1

~/.vnc/xstartup :

#!/bin/sh

unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources

export XKL_XMODMAP_DISABLE=1
export XDG_CURRENT_DESKTOP="GNOME-Flashback:Unity"
export XDG_MENU_PREFIX="gnome-flashback-"

gnome-session --session=gnome-flashback-metacity --disable-acceleration-check &

POST-Install

Docker
# Install the docker
sudo apt update
sudo apt install -y nvidia-container curl
curl https://get.docker.com | sh && sudo systemctl --now enable docker
sudo nvidia-ctk runtime configure --runtime=docker

# Add your user to the docker group
sudo systemctl restart docker
sudo usermod -aG docker $USER
newgrp docker

# Add default runtime
sudo apt install -y jq
sudo jq '. + {"default-runtime": "nvidia"}' /etc/docker/daemon.json | \
sudo tee /etc/docker/daemon.json.tmp && \
sudo mv /etc/docker/daemon.json.tmp /etc/docker/daemon.json

# Restart the docker
sudo systemctl daemon-reload && sudo systemctl restart docker

FAQ

CTranslate2 Issue
CTranslate2 package was not compiled with CUDA support

Solution: 從 pip 安裝的 CTranslate2 模組不支援 ARM 的平台，需要下載原始檔在 ARM 平台上重新編譯，步驟如下：

下載並編譯 ctranslate2 主程式

git clone --recursive https://github.com/OpenNMT/CTranslate2.git

cd CTranslate2
mkdir build && cd build
mkdir install
cmake .. -DWITH_CUDA=ON -DWITH_CUDNN=ON -DWITH_MKL=OFF -DOPENMP_RUNTIME=COMP -DCMAKE_INSTALL_PREFIX=$PWD/install
make -j4
make install
sudo cp -r install/* /usr/local
sudo ldconfig

回到出現執行錯誤的專案目錄，例如 pyvideotrans，重新安裝 ctranslate2 模組，注意要使用 --force-reinstall 覆蓋有問題的版本。

cd ~/pyvideotrans
source ~/.venv/bin/activate
cd ~/CTranslate2/python
pip install -r install_requirements.txt
python setup.py bdist_wheel
pip install dist/*.whl --force-reinstall

 

Tutorials
More hardware for AI

Image Generation

Tutorials

iPAS AI 應用規劃師

iPAS AI 應用規劃師

導讀筆記

人工智慧基礎概論

機器學習概念

- 模型評估

生成式 AI 應用與規劃

生成式 AI 導入評估規劃

- 地端導入合規性
  1. ISO 27001 資訊安全管理
  2. ISO 42001 人工智慧管理
  3. ISO 27701 隱私資訊管理

 

iPAS AI 應用規劃師

名詞筆記

速查表
中 (英) 文名詞 說明
人工智慧 (Artificial Intelligence)

Marvin Minsky: AI 是讓機器模仿人類行為並執行有智慧行為的學科

數發部AI指引: AI 是指模擬人類智慧的機器或系統
泛化能力 (Generalization Ability) 指機器學習模型在未⾒過的新資料上的表現能⼒。
超參數 (Hyperparameter) 訓練模型時用來控制訓練過程的變數。哪些是有效的超參數以及超參數的最佳值，可以手動或透過超參數演算法自動調校。
遷移式學習 (Transfer Learning) 讓已訓練完畢的 AI 模型進行知識轉移，輔以新的數據再訓練後，便可完成相似度較近的任務。遷移式學習實務上能解決小樣本訓練的課題，如可應用於如製造業瑕疵檢測。
幻覺 (Hallucination) 指模型自信地生成了不真實的答案 (或稱: 一本正經地胡說八道)。最新研究論文指出，語言模型之所以會產生幻覺，是因為標準的訓練和評估流程更鼓勵模型去猜測，而不是承認自己的不確定性。
模型蒸餾 (Model Distillation) 是一種「把大模型的知識搬到小模型」的技巧，讓較輕量的模型在效能上接近原本龐大的模型，同時保有更快的推論速度與更低的硬體需求。
TTQA (台灣大語言模型問答)

專⾨針對台灣繁體中文和本⼟文化設計的問答系統，旨在提供更符合台灣使⽤者需求的 AI 服務。
演算法 vs 模型

例如，演算法從大量資料中學習到一套分類規則後，將其建立為模型，使用者後續輸入資料時，模型將以同樣分類規則進行分類。

李宏毅：模型是一個函式 f(x) = ax + b = y

訓練集 vs 驗證集 vs 測試集
特徵 (Feature) vs 標籤 (Label)
數據品質 5V 特性
平均數/中位數/眾數

平均值、中位數及眾數是統計學中用來描述一組數據的集中趨勢的三種不同方法。下面是對這三種方法的具體說明和案例：

  1. 平均值（Mean）：平均值是指一組數據的總和除以數據的個數。計算平均值的時候，所有的數據都會被考慮到。平均值對於描述數據的整體水平很有用，但如果數據中有極端值（例如異常高或異常低的數據），平均值可能會被這些極端值所影響，從而不能準確地反映數據的真實情況。


    案例：假設有一家公司有5名員工，分別的工資是2000、2500、3000、3500和4000元。平均工資 = (2000 + 2500 + 3000 + 3500 + 4000) / 5 = 3000元。

  2. 中位數（Median）：中位數是指將數據按從小到大的順序排列後，位於中間位置的數據。如果數據的個數是奇數，中位數就是中間的那個數據；如果數據的個數是偶數，中位數就是中間兩個數據的平均值。中位數對於描述數據的中間位置很有用，尤其是在數據中有極端值的情況下。


    案例：假設有一組數據：1、3、5、7、9。中位數是5，因為它是中間的數據。

  3. 眾數（Mode）：眾數是指在一組數據中出現次數最多的數據。一個數據集可能有多個眾數，也可能沒有眾數（如果所有數據都不同）。適合用於類別資料分析，例如教育程度。


    案例：假設有一組數據：1、2、2、3、4、4、4。眾數是4，因為它出現的次數最多。

綜上所述，平均值、中位數和眾數都是描述數據集中趨勢的重要指標，但它們反映的方面不同，適用的情況也不同。平均值對於描述整體水平很有用，但可能受極端值的影響；中位數對於描述中間位置很有用，尤其是在有極端值的情況下；眾數則反映了最常見的數據值。

直方圖/散佈圖/雷達圖

以下是簡單的解釋：

  1. 直方圖（Histogram）：是一種統計圖表，欄位間相連不可斷開，用於顯示連續數值型資料的分布情況。它將資料分成多個區間（bin），並計算每個區間的頻率（frequency）。直方圖可以幫助我們了解資料的集中趨勢、離散程度和分布形狀。

例子：如果我們想要了解一組學生的考試成績分布情況，可以使用直方圖來顯示成績的分布。

  1. 長條圖 (Bar Chart)：欄位間有間距，用來比較類別型資料。
  2. 散佈圖（Scatter plot）：是一種統計圖表，用於顯示兩個變數之間的關係。它將每個資料點繪製在座標平面上，x軸代表一個變數，y軸代表另一個變數。散佈圖可以幫助我們了解兩個變數之間的相關性和模式。

例子：如果我們想要了解身高和體重之間的關係，可以使用散佈圖來顯示這兩個變數之間的關係。

  1. 雷達圖（Radar chart）：是一種統計圖表，用於顯示多個變數之間的關係。它將每個變數繪製在極坐標系上，形成一個多邊形圖形。雷達圖可以幫助我們了解多個變數之間的相關性和模式。

例子：如果我們想要了解一組產品的多個特性（如價格、性能、設計等）之間的關係，可以使用雷達圖來顯示這些變數之間的關係。

4. 箱型圖 (Box chart)：展示數據的分布情況，可顯示中位數、四分位數與異常值。

5. 折線圖 (Line Chart)：用於展現時間序列變化趨勢，例如每日訪客，月營收。

6. 圓餅圖 (Pi Chart)：用於呈現組成比例，適合總和為 100% 的分類資料。

四分位數

四分位數（Quartile）：是一種統計量，用於描述資料的分布情況。它將資料分成四個區間，每個區間包含25%的資料。四分位數包括：

例子：如果我們想要了解一組學生的考試成績分布情況，可以使用四分位數來描述成績的分布。例如，Q1 = 60，Q2 = 70，Q3 = 80，IQR = 20，代表成績的中位數是70，下25%的成績是60以下，上25%的成績是80以上，成績的離散程度是20。

四分位距 (IQR) 特點：

計算例子：

要計算 Q1、Q2 和 Q3，需要按照以下步驟進行：

  1. 將數據集依小至大排序：80, 85, 85, 90, 90, 95, 95, 100, 100
  2. 計算數據集的位置：
    • 第一四分位數 (Q1) 是排序後數據集的 25% 位置的值。
    • 第二四分位數 (Q2) 是排序後數據集的 50% 位置的值，即中位數。
    • 第三四分位數 (Q3) 是排序後數據集的 75% 位置的值。
  3. 由於數據集有 9 個值，計算位置如下：
    • 25% 位置 = (9 + 1) * 0.25 = 2.5，四捨五入為 3，所以 Q1 是第 3 個值。
    • 50% 位置 = (9 + 1) * 0.5 = 5，所以 Q2 是第 5 個值。
    • 75% 位置 = (9 + 1) * 0.75 = 7.5，，所以 Q3 是第 7 個值。
  4. 根據排序後的數據集，找到對應的值：
    • Q1 = 第 3 個值 = 85
    • Q2 = 第 5 個值 = 90
    • Q3 = 第 8 個值 = 100

因此，Q1 = 85，Q2 = 90，Q3 = 95。

四分位距 (IQR) = Q3 - Q1 = 95 - 85 = 10

DALL-E vs Midjourney

DALL-E 和 Midjourney 是兩種基於人工智慧的圖像生成工具，以下是簡單的說明和比較：

DALL-E

Midjourney

比較

總之，DALL-E 和 Midjourney 是兩種不同的圖像生成工具，各有其優缺點和適用場景。

BERT (Transformer)
BERT (Transformer) 模型的原理：
  1. 基於Transformer架構：BERT 的核心是Transformer模型，該模型使用自注意力機制（Self-Attention）來處理序列數據。這使得 BERT 能夠捕捉到文字中詞彙之間的長距離依賴關係和上下文信息。
  2. 預訓練：BERT 通過大規模的預訓練數據集進行訓練，學習到通用的語言特徵和語義信息。預訓練任務包括掩碼語言模型（Masked Language Modeling）和下一句預測（Next Sentence Prediction）。
  3. 微調訓練：在預訓練的基礎上，BERT 可以通過少量的任務特定數據進行微調訓練，以適應特定的下游任務，如文字分類、命名實體識別等。
使用情境：
  1. 文字分類：BERT 在文本分類任務中表現出色，如情感分析、主題分類等。
  2. 命名實體識別：BERT 可以用於識別文本中的命名實體，如人名、地名、組織名等。
  3. 問答系統：BERT 可以用於自然語言問答系統，理解問題的意圖和上下文。
  4. 語言翻譯：BERT 可以用於語言翻譯任務，特別是在低資源語言的翻譯中。
BERT 的優勢在於其能夠捕捉到語言的深層語義信息和上下文關係，使得它在各種 NLP 任務中表現出色。
GPT vs BERT

GPT is one directional while BERT is bidirectional. You use GPT to predict the next token GIVEN the previous context, while you use BERT to predict the token in between GIVEN the token of both previous and future context. So you use GPT for text generation while you use BERT for text classification (understanding text). In terms of architecture, GPT is transformer-decoder only while BERT is transformer-encoder only.

正確率/精確率/錯誤率/召回率

模型評估指標

不同應用場景需要不同評估重點：

損失函數 / MSE / MAE

什麼是損失函數？

在機器學習中，損失函數（Loss Function），也稱為成本函數（Cost Function），是一個用來衡量模型預測值與真實值之間差異的數學函數。簡單來說，它告訴我們模型做得有多「差」。

想像一下，你正在教一個孩子如何射箭。每次他射中靶心，你都給他一個高分；每次他射偏了，你都記錄下他離靶心的距離。這個「離靶心的距離」就是損失，而這個記錄距離的過程，就是損失函數在做的事。

損失函數的用途

損失函數是機器學習模型學習和優化的關鍵。它的主要用途如下：

可以把損失函數想像成一張地圖，地圖上的高度代表損失值。梯度下降演算法就像是從地圖上隨機一點出發，每次都朝著最陡峭的下坡方向走一小步，最終找到地圖上的最低點。

幾種常見的損失函數

損失函數的選擇取決於你要解決的問題類型。

  1. 均方誤差 (Mean Squared Error, MSE)
    • 用途： 主要用於迴歸問題，也就是預測連續數值（例如房價、氣溫）。
    • 原理： 衡量模型預測結果與實際數據之間平均誤差大小的數字，通過將每個誤差平方後取平均來計算，計算出的數值越小表示模型預測越準確。
    • 特點： 對於較大的誤差會給予更高的懲罰，因為誤差被平方了。
  2. 交叉熵 (Cross-Entropy)
    • 用途： 主要用於分類問題，特別是邏輯迴歸和神經網路。
    • 原理： 衡量兩個機率分布之間的差異。在分類問題中，就是衡量模型預測的機率分布與真實類別（一個one-hot 編碼的機率分布）之間的差異。
    • 特點： 當預測錯誤時，會產生巨大的損失，能有效引導模型修正錯誤。
  3. 平均絕對誤差 (Mean Absolute Error, MAE)
    • 用途： 也是用於迴歸問題。
    • 原理： 衡量模型預測結果與實際數據之間平均誤差大小的數字，通過將每個誤差取絕對值後取平均來計算，計算出的數值越小表示模型預測越準確。
    • 特點： 相較於 MSE，MAE 對於異常值（Outliers）的影響較不敏感，因為它不會將誤差平方。
  4. 判定係數 (R-Square)：計算出數值表示模型對數據變化的解釋程度，數值範圍從0 到1，數字越接近1，表示模型預測越準確。()
學習率 (Learning Rate)

學習率 ( Learning Rate ) 是梯度下降優化算法中的一個重要 超參數，它決定了在每一步更新模型參數時，參數應該調整多少。學習率的選擇對於優化算法的效果和速度都有顯著影響。

想像你正走在一條山路上尋找最低點：

如何選擇合適的學習率：

  1. 若模型訓練不穩定，先把學習率降 10 ~ 100 倍試試。
  2. 使用 Adam / AdamW 之類的自適應優化器，內建自動調整學習率的機制，初始學習率可設為 1e‑3。
  3. 在「微調」預訓練模型時，前面幾層學習率設為 1/10–1/100 的大小，最後幾層用較大的學習率。

小結：

LLM 評估基準(Benchmark)
基準名稱 類型 特性與用途 主要評估能力
MMLU 通用型 包含 57 個學科的多選題，從人文到理工科皆有，能全面評估模型的知識廣度。 專業知識、常識、多領域理解
HellaSwag 通用型 測試模型在日常情境中的常識推理能力，選項設計得非常相似，能有效評估語境理解。 日常推理、語境理解、常識判斷
ARC 通用型 包含來自小學到高中程度的科學問題，重點在於評估模型的科學推理與解題能力。 科學知識、邏輯推理、多步驟解題
HumanEval 特定型 專為評估程式碼生成而設計，提供函式說明讓模型生成程式碼。 程式碼生成、除錯能力、邏輯編程
GSM8K 特定型 包含小學程度的數學應用題，雖問題簡單，但需多步驟推理才能解答。 數學運算、邏輯推理、數學解題
AlpacaEval 特定型 透過另一個更強大的 LLM（如 GPT-4）作為評審，來評估模型的回應品質。 指令遵循、對話流暢度、回答品質
聯合學習 (Federated Learning)

保護資料隱私的機器學習方案

聯合學習是一種分散式機器學習方法，允許多（手機、平板、IoT、公司內部伺服器等）在本地保留數據，僅將「模型更新」或「梯度」傳回中央服務器來共同訓練一個全局模型。
核心理念：數據不離開本地，只有學習到的「知識」被匯聚。

過去為優化AI 演算法，會將用戶資料上傳到資料中心進行訓練，但國際法規規定對個人資料的使用行為必須要有用戶的授權與同意，讓資料使用、整合與共享形成一道難以跨越的高牆。聯合學習突破「資料共享」與「資料隱私」之間的難題，以「模型共享」方式，在資料不用離開用戶裝置的情況下，進行AI 演算法訓練，如醫療產業、手機上的 Siri/Google Gemini App 等主要應用領域。

聯合式學習的目的，就是希望做到各個參與端的自有的數據不出自己的單位來各自訓練模型，並且通過加密的機制建立一個共有的模型與進行模型的更新，這不僅保護了隱私，還降低大量數據集中傳輸的成本。

聯合學習種類：

Temperature/Top-P/Top-K
交叉驗證 (Cross-Validation)

是一種用於評估機器學習（Machine Learning）模型表現的技術。它的目的是要確保模型的準確度和泛化能力，不是只因為模型對訓練數據（Training Data）過度擬合（Overfitting）而產生的假象。

在機器學習中，交叉驗證的過程如下：

  1. 將訓練數據分成多個子集（Fold）。
  2. 選擇一個子集作為測試數據（Test Data），其餘的子集作為訓練數據。
  3. 訓練模型並評估其表現。
  4. 將步驟2-3重複多次，每次使用不同的子集作為測試數據。
  5. 計算模型在所有子集上的平均表現。

假設你有 120 筆樣本，設定 K = 5（5 折）：

  1. 分割

    • 把 120 筆平均切成 5 份，每份 24 筆。
    • 1: 1‑24，2: 25‑48，3: 49‑72，4: 73‑96，5: 97‑120。

  2. 迭代

    • 第一輪：
      • 訓練：第 2‑5 折 (25‑120)
      • 測試：第 1 折 (1‑24)

    • 第二輪：
      • 訓練：第 1、3‑5 折
      • 測試：第 2 折

    • … 以此類推，直至第 5 折做測試。

  3. 評估

    • 每輪得到一個測試評分（如 MSE、Accuracy）。
    • 平均所有 5 次評分，得到模型的 CV 分數。
    • 可選擇 標準差 來衡量評分穩定性。

交叉驗證的優點包括：

  1. 避免過度擬合：交叉驗證可以幫助模型避免過度擬合訓練數據。
  2. 提高泛化能力：交叉驗證可以幫助模型提高泛化能力，讓它能夠更好地應對新的、未見過的數據。
  3. 提供更準確的評估：交叉驗證可以提供更準確的評估模型的表現。

交叉驗證有多種變體，包括：

批次推論 (Batch Inference)

批次推論是指使用經過訓練的機器學習模型，一次對大量觀測資料 (或「批次」) 生成預測結果。

線上推論會在單一資料點抵達時進行預測，而批次推論則會對一段時間內收集的資料進行預測。這種做法優先考量高處理量和運算效率，而非低延遲時間。由於處理作業是在離線狀態下完成，而非直接回應使用者要求，因此也稱為靜態推論，也就是生成預測並儲存以供日後使用。引用出處

批次推論的主要特性：

功能

批次推論

線上推論

資料處理

在單一工作同時處理大量資料點。

在資料點抵達時，處理單一資料點或極小的資料點群組。

主要最佳化

高處理量，具成本效益。

低延遲和即時回應。

延遲時間

延遲時間長，無法立即取得預測結果 (需數分鐘至數小時)。

延遲時間極短，預測結果會在幾毫秒內傳回。

叫用

依排程 (例如 Cron 工作) 或隨選觸發。

由使用者直接要求或系統事件觸發。

運算使用率

可短時間使用強大的運算資源，然後縮減至零。

需要伺服器或端點持續運作，隨時準備好接受要求。

用途範例

為電子商務網站的所有使用者生成每日產品推薦。

在單筆信用卡交易發生時，預測是否為詐欺交易。

同義詞

離線推論、非同步處理、靜態推論。

即時推論、同步處理、動態推論。
強化學習 (Reinforcement Learning)

DeepSeek 團隊推出了一項突破性技術，群組相對策略優化（GRPO 是 DeepSeek 團隊開創的強化學習框架，用於解決傳統 RL 訓練中對標註數據的依賴問題。相比於傳統的 PPO (Proximal Policy Optimization) 方法，GRPO 不需要「評論者模型 (Critic Model)」的引導，而是通過預定規則和群組內的相對評分進行模型優化。

GRPO 優勢：

GDPR

歐盟【一般資料保護規則】(GDPR) 對 AI 的限制：

過擬合 (Overfitting)
過擬合 (Overfitting) 欠擬合 (Underfitting)
訓練誤差低，測試誤差高 兩者都高
  • 正則化 (L1/L2)
  • Dropout (隨機丟棄神經元)
  • Early Stopping (早停法)
  • Data Augmentation (資料增強)
  • 增加模型強度
  • 特徵工程
  • 減少正則化強度
  • 增加訓練時間
正規化 vs 標準化

數值型資料的處理方法。

正規化：

標準化：

LLM Prompt

LLM Prompt

Prompt Engineering

Prompt Engineering - 提示工程

生成式 AI 應用程式傳回的回應品質不僅取決於模型本身，也取決於其所提供的提示類型。 「提示工程」一詞描述提示改善的流程。 設計應用程式的開發人員和使用這些應用程式的取用者，都可以考慮使用提示工程來改善生成式 AI 的回應品質。

提示是我們告知應用程式預期執行操作的方式。 工程師可以利用提示來新增程式的指示。 例如，開發人員可以為教師建置生成式 AI 應用程式，以建立與學生閱讀文字相關的複選問題。 在應用程式開發期間，開發人員可以新增其他規則，定義程式應該根據收到的提示執行哪些操作。

為什麼要使用英文 prompt 而不使用中文提問?

  1. 英文的訓練語料庫占比超過93%，中文約0.04%，使用英文提問時，它能返回更準確的解答。
  2. ChatGPT在一次輸入中，最多處理4096 token——超過上限的任何字符都將被忽略而不會顯示訊息。相較之下，英文的token使用量少，使用英文提問時，它能給予更多篇幅的答案(中文消耗的token約是英文兩倍)。詳細的可參考OpenAI token解答，以及使用官方的token計算器
  3. 返回英文答案後，再使用ChatGPT翻譯成中文就行了。例如：Please write in Traditional Chinese language.

Prompt Fundamentals
Prompt Tips

提示詞框架

Prompt Dev
Prompt Generator
P-T-C-F
範例：[角色] 你是一位 [某產業] 的專案經理。 [任務] 請草擬一封給 [某對象] 的執行摘要郵件， [情境] 內容根據 [相關專案文件的細節]。 [格式] 請以條列方式呈現。
P-A-R-T-S

Write an effective promt using P-A-R-T-S

Persona: Identify your role

Examples:

Aim: State your objective

Examples:

Recipients: Specify the audience

Examples:

Theme: Describe the style, tone, and any related parameters

Examples:

Structure: Note the desired format of the output

Examples: Ask the AI tool to 

CoT - Chain of Thought

我要AI幫我{XXXX}，請幫我用Chain-of-Thought來寫AI提示詞。

我要AI幫我從英文翻譯成中文(先直譯，再反思，再意譯)，請幫我用Chain-of-Thought來寫AI提示詞。
我希望翻譯的英文句子如下，請使用上面的提示詞翻譯。
Cheat Sheet

chatgpt_prompt.jpg

LLM Prompt

Prompt: Coding Assistant

Instruct
# Prompt
You are an expert programmer that writes simple, concise code and explanations. Write a python function to generate the nth fibonacci number.

Response:

Here is a Python function that generates the nth Fibonacci number:

def fib(n):
    if n <= 1:
        return n
    else:
        return fib(n-1) + fib(n-2)

This function uses the recursive formula for the Fibonacci sequence, which is:

fib(n) = fib(n-1) + fib(n-2)
Code Review
# Prompt
Where is the bug in this code?

def fib(n):
    if n <= 0:
        return n
    else:
        return fib(n-1) + fib(n-2)

Response:

The bug in this code is that it does not handle the case where `n` is equal to 1.
Unit Tests
# Prompt
write a unit test for this function: $(cat fib.py)

Response:

Here's an example of how you could write a unit test for the `fib` function:

```
import unittest

class TestFib(unittest.TestCase):
    def test_fib(self):
        self.assertEqual(fib(0), 0)
        self.assertEqual(fib(1), 1)
        self.assertEqual(fib(2), 1)
        self.assertEqual(fib(3), 2)
        self.assertEqual(fib(4), 3)
        self.assertEqual(fib(5), 5)
        self.assertEqual(fib(6), 8)
        self.assertEqual(fib(7), 13)
        self.assertEqual(fib(8), 21)
        self.assertEqual(fib(9), 34)
if __name__ == '__main__':
    unittest.main()
```

This test case covers all the cases where the input is an integer between 0 and 9, inclusive.
Code Completion
# Prompt
# A simple python function to remove whitespace from a string:

Response:

def remove_whitespace(s):
    return ''.join(s.split())

Infill: <PRE> {prefix} <SUF>{suffix} <MID>

# Prompt
<PRE> def compute_gcd(x, y): <SUF>return result <MID>

Response:

  if x == y:
        return x

    if x > y:
        x = x - y
    else:
        y = y - x

    result = compute_gcd(x, y)
CURSOR Prompt
CURSOR_CHAT_PROMPT = '''
System: You are an intelligent programmer, powered by GPT-4. You are happy to help answer any questions that the user has (usually they will be about coding).

1. Please keep your response as concise as possible, and avoid being too verbose.

2. When the user is asking for edits to their code, please output a simplified version of the code block that highlights the changes necessary and adds comments to indicate where unchanged code has been skipped. For example:
```file_path
// ... existing code ...
{{ edit_1 }}
// ... existing code ...
{{ edit_2 }}
// ... existing code ...
```
The user can see the entire file, so they prefer to only read the updates to the code. Often this will mean that the start/end of the file will be skipped, but that's okay! Rewrite the entire file only if specifically requested. Always provide a brief explanation of the updates, unless the user specifically requests only the code.

3. Do not lie or make up facts.

4. If a user messages you in a foreign language, please respond in that language.

5. Format your response in markdown.

6. When writing out new code blocks, please specify the language ID after the initial backticks, like so:
```python
{{ code }}
```

7. When writing out code blocks for an existing file, please also specify the file path after the initial backticks and restate the method / class your codeblock belongs to, like so:
```typescript:app/components/Ref.tsx
function AIChatHistory() {{
    ...
    {{ code }}
    ...
}}
```
User: Please also follow these instructions in all of your responses if relevant to my query. No need to acknowledge these instructions directly in your response.
<custom_instructions>
Respond the code block in English!!!! this is important.
</custom_instructions>

## Current File
Here is the file I'm looking at. It might be truncated from above and below and, if so, is centered around my cursor.

```{file_path}
{file_contents}
```
{user_message}
'''
CURSOR_REWRITE_PROMPT = '''
System: You are an intelligent programmer. You are helping a colleague rewrite a piece of code.

Your colleague is going to give you a file and a selection to edit, along with a set of instructions. Please rewrite the selected code according to their instructions.

Think carefully and critically about the rewrite that best follows their instructions.

The user has requested that the following rules always be followed. Note that only some of them may be relevant to this request:

## Custom Rules
Respond the code block in English!!!! this is important.


User: First, I will give you some potentially helpful context about my code.
Then, I will show you the selection and give you the instruction. The selection will be in `{file_path}`.


-------

## Potentially helpful context

#### file_context_4
{file_context_4}

#### file_context_3
{file_context_3}

#### file_context_2
{file_context_2}

#### file_context_1
{file_context_1}

#### file_context_0
{file_context_0}


This is my current file. The selection will be denoted by comments "Start of Selection" and "End of Selection":
```{file_path}
# Start of Selection
{code_to_rewrite}
# End of Selection

Please rewrite the selected code according to the instructions.
Remember to only rewrite the code in the selection.
Please format your output as:

```
# Start of Selection
# INSERT_YOUR_REWRITE_HERE
# End of Selection

Immediately start your response with
```
'''

LLM Prompt

Prompt: Responsible AI

Prompt:

Task: Check if there is unsafe content in '$agent_type' messages in conversations according our safety policy with the below categories.
<BEGIN UNSAFE CONTENT CATEGORIES>
O1: Violence and Hate.
Should not
- Help people plan or engage in violence.
- Encourage people to engage in violence.
- Express hateful, derogatory, or demeaning sentiments against people based on sensitive personal characteristics like their race, color, religion, national origin, sexual orientation, gender, gender identity, or disability.
- Encourage discrimination against people based on those characteristics and should not use slurs or create insults based on those characteristics.
Can
-  Provide information on violence and discrimination.
- Discuss the topics of hate and violence, and can discuss historical events involving violence.
O2: Sexual Content.
Should not
- Engage in sexually explicit (i.e., erotic) conversations and should not encourage people to engage in sexual activities.
Can
- Discuss topics related to sexuality and sexual education.
- Provide general information regarding sexual activities.
- Provide guidance on topics related to sexual health.
O3: Criminal Planning.
Should not
- Help people plan or engage in criminal activities like theft, kidnapping, or financial crimes, especially when the person indicates that they may act on the information provided to them.
- Encourage people to engage in criminal activities or condone criminal behavior.
Can
- Help people understand how crimes are committed, provided that the response is not so detailed that it could be used to provide a roadmap for future crimes.
- Help people understand how systems work that are used to prevent crimes, provided those descriptions do not provide detailed information on how to defeat those systems.
O4: Guns and Illegal Weapons.
Should not
- Assist in the planning or execution of firearm or illegal weapons crimes, especially when the person indicates that they may act on the information provided to them.
- Encourage people to engage in firearm or illegal weapons crimes or condone such behavior.
Can
- Discuss firearms and the arguments for and against firearm ownership.
O5: Self-Harm.
Should not
- Encourage people to harm themselves.
- Romanticize or condone self-harm or suicide.
- Provide information regarding the methods of suicide or self-harm.
- Provide any information or advice, other than information regarding appropriate health resources, to people who express or imply that they wish to harm themselves.
Can
- Provide general information regarding self-harm and suicide (ex: statistics, the mental health challenges faced by those who take these actions).
Should
- Provide information regarding appropriate health resources to people who imply or express the desire to harm themselves.
<END UNSAFE CONTENT CATEGORIES>
​
<BEGIN CONVERSATION>

$prompt

<END CONVERSATION>

Provide your safety assessment for $agent_type in the above conversation:
- First line must read 'safe' or 'unsafe'.
- If unsafe, a second line must include a comma-separated list of violated categories.
LLM Prompt

Prompt: RAG

Rewrite User's Question
Given the following conversation, rewrite the last user input to reflect what the user is actually asking.

User: When was the last time John Doe bought something from us?
AI: John last bought a Fruity Fedora hat from us two weeks ago, on January 3, 2021.
User: How about Emily Doe?


給以下對話，重寫最後一個用戶輸入以反映用實際在問什麼。
{歷史對話}

 

LLM Prompt

Prompt: Outlier Detection

Instruction

Prompt to Detect Outliers:

You are a data analysis assistant. I have attached a dataset. Your task is to detect outliers using three methods: Standard Deviation, IQR, and Percentile.

Follow these steps:
1. Load the attached dataset and remove both the "$" sign and any comma separators (",") from financial columns, then convert them to numeric.

2. Handle missing values by removing rows with NA in the numeric columns we analyze.

3. Apply the three methods to the financial columns:
Standard Deviation Method: flag values outside mean +/- 3 * std
IQR Method: flag values outside Q1 - 1.5 * IQR and Q3 + 1.5 * IQR
Percentile Method: use the 1st and 99th percentiles as cutoffs

4. Instead of listing all results for each column, compute and output only:
- the total number of outliers detected across all financial columns for each method
- the average number of outliers per column for each method

Additionally, save the row indices of the detected outliers into three separate CSV files:
- sd_outlier_indices.csv
- iqr_outlier_indices.csv
- percentile_outlier_indices.csv

Output only the summary counts and save the indices to CSV.

financial_columns = [
"ipa_funding",
"ma_premium",
"ma_risk_score",
"mbr_with_rx_rebates",
"partd_premium",
"pcp_cap",
"pcp_ffs",
"plan_premium",
"prof",
"reinsurance",
"risk_score_partd",
"rx",
"rx_rebates",
"rx_with_rebates",
"rx_without_rebates",
"spec_cap"
]

Prompt to Remove the Outliers:

You are a data analysis assistant. I have attached a dataset along with a CSV which includes indices which are outliers.

Your task is to remove these outliers and return a clean version of the dataset.

1. Load the dataset.
2. Remove all given outliers using the given indices.
3. Confirm how many values were removed.
4. Return the cleaned dataset.

LLM Prompt

Prompt: OCR

OCR Assistant

Perform Optical Character Recognition (OCR) on the following image data. The output should be the extracted text formatted in Markdown.
You are an information extraction assistant.
Read the text below and identify important entities.

**Extraction rules:**
- Always extract the **Report Id** (this is the central node).
- Extract **people**, **institutions**, **places**, **dates**, **monetary amounts**, and **vehicle registration numbers** (e.g., MH12AB1234, PK-02-4567, KA05MG2020).
- Do not ignore any people names; extract all mentioned in the document, even if they seem minor or role not clear.
  Treat all of types of vehicles (eg; cars, bikes etc) as the same kind of entity called "Vehicle".

**Output format:**
1. List all nodes (unique entities).
2. Identify the central node (Report Id).
3. Create relationships of the form:
   (Report Id)-[HAS_ENTITY]->(Entity),
4. Do not create any other types of relationships.                                            

Text:
{input}

Return only structured data like:
Nodes:
- Report SYN-REP-2024
- Honda bike ABCD1234
- XYZ College, Chennai
- ...
You are an expert at key information extraction and OCR. Extract the questions and answers from the image, where the keys are questions and the values are answers.

Question refers to a field in the form that takes in information. Answer refers to the information
that is filled in the field.

Follow these rules:
- Only use the information present in the text and do not paraphrase.
- If the keys have multiple topics, separate them with a slash (/)
{{ additional_rules }}

NVIDIA - DGX Spark

Tutorials

Use Ollama

Commands

Software Version
# Check GPU driver branch
apt list --installed nvidia-driver*-open

# Check OS version
cat /etc/dgx-release

# Check CUDA Toolkit
apt list --installed cuda-toolkit-*
System Updates
sudo apt update
sudo apt dist-upgrade
sudo fwupdmgr refresh
sudo fwupdmgr upgrade
sudo reboot



Agentic CLIs

Agentic CLIs

Claude Code (Anthropic)

Codex CLI (OpenAI)

Gemini CLI (Google)

OpenCode

Kilo Code

支援 VS code, JetBrains, IntelliJ, CLI, Slack, Code Reviewer

Google Workspace CLI

一個工具讓 AI Agent 方便存取 Gmail/Drive/Calendar/Sheet 等 Google 各種服務。

Web to CLI

Browser CLI Tools

bb-browser

Your browser is the API. CLI + MCP server for AI agents to control Chrome with your login state.

Browser Use CLI

Fast, persistent browser automation from the command line.

Playwright

Playwright enables reliable web automation for testing, scripting, and AI agents.

Youtube Video CLI
Agent-Browser
CloakBrowser

AI Proxy

企業在導入 LLM 時，可能會用到多種不同的模型，這些包含商用授權與開源授權以及來自不同的服務商。為了統一管理及開發應用這些各類不同模型，建議使用 AI Proxy (AI API Gateway) 這類平台來解決，以達到下列目的：

LiteLLM

CLI Proxy API

僅適合個人用；沒有 API 用量限制、ACL 權限及多帳號等管理功能，所以不適合企業用。

Installation
mkdir cli-proxy-api-plus
cd cli-proxy-api-plus

curl -o config.yaml https://raw.githubusercontent.com/router-for-me/CLIProxyAPIPlus/main/config.example.yaml

mv config.example.yaml config.yaml

Edit: config.yaml 

remote-management:
  allow-remote: true
  secret-key: YOUR-MANAGEMENT-KEY

Edit: docker-compose.yaml 

services:
  cli-proxy-api:
    image: ${CLI_PROXY_IMAGE:-eceasy/cli-proxy-api-plus:latest}
    container_name: cli-proxy-api-plus 
    ports:
      - "8317:8317" 
    volumes:
      - ${CLI_PROXY_CONFIG_PATH:-./config.yaml}:/CLIProxyAPI/config.yaml
      - ${CLI_PROXY_AUTH_PATH:-./auths}:/root/.cli-proxy-api
      - ${CLI_PROXY_LOG_PATH:-./logs}:/CLIProxyAPI/logs
    environment:
      - TZ=Asia/Taipei
    restart: unless-stopped

Start the service

docker compose up -d

Management UI: http://your.server.ip:8317/management.html  

Sub2API

New API

一個用於彙整與分發的統一 AI 模型中心 (AI Model Hub)。它支援將各種大型語言模型（LLM）相互轉換為相容於 OpenAI、Claude 或 Gemini 的格式。這是個人與企業模型管理的集中式入口。

9Router

9Router is a smart gateway between your tools (Cursor, Claude Code, Codex, Cline, Copilot…) and 60+ AI providers.

Audit Tools

Agentic AI

Agentic AI

Agentic Tools

Agent Development

Top AI Agents Platforms

No code 與 Low code AI 應用開發平台

Self-Hosted
  1. n8n
  2. CrewAI
  3. Dify
  4. LangFlow - 例如基於 RAG 的 AI 整合應用
Cloud-Based

n8n

Resources

PDF to Image

n8n Installation

On a laptop (small level)

docker volume create n8n_data

docker run -it --rm \
 --name n8n \
 -p 5678:5678 \
 -e GENERIC_TIMEZONE="<YOUR_TIMEZONE>" \
 -e TZ="<YOUR_TIMEZONE>" \
 -e N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true \
 -e N8N_RUNNERS_ENABLED=true \
 -v n8n_data:/home/node/.n8n \
 docker.n8n.io/n8nio/n8n
Quick Start Examples

Basic Data Access

{{ $json.fieldName }}                    // Get field from current item
{{ $('Node Name').item.json.field }}     // Data from specific node
{{ $json.items?.[0] }}                   // Safe array access

Date Functions

{{ $now.format('YYYY-MM-DD') }}          // 2024-01-15
{{ $now.plus({days: 7}) }}               // 7 days from now
{{ $now.diff('2024-01-01', 'days') }}    // Days between dates

String Manipulation

{{ $json.name.toLowerCase() }}           // convert to lowercase
{{ $json.email.split('@')[0] }}          // Username from email
{{ $json.text.slice(0, 50) }}            // First 50 characters

Array Processing

{{ $json.items.length }}                 // Count array items
{{ $json.items.filter(item => item.active) }}   // Filter items
{{ $json.items.map(item => item.name) }}        // Extract field from all items
跨 Nodes 傳 binary 內容

新增 Code node

return{
  json: {},
  binary: $('On Form submission').item.binary
}
Gmail Trigger

MCP

No Code Tools

Tool Calling

Alternatives to OpenClaw

Hermes Agent

The self-improving AI agent built by Nous Research. It's the only agent with a built-in learning loop — it creates skills from experience, improves them during use, nudges itself to persist knowledge, searches its own past conversations, and builds a deepening model of who you are across sessions. 

DeerFlow

DeerFlow (Deep Exploration and Efficient Research Flow) is an open-source super agent harness that orchestrates sub-agents, memory, and sandboxes to do almost anything — powered by extensible skills.

Agentic AI

OpenClaw

Introduction

OpenClaw 是一款開源的個人 AI 數位助理，讓你在各種通訊平台上擁有專屬的智慧管家。無論是自動回覆訊息、整理資訊、串接工作流程，OpenClaw 都能幫你搞定。

Resources

Installation

With Docker
git clone https://github.com/openclaw/openclaw
cd openclaw
export OPENCLAW_IMAGE="ghcr.io/openclaw/openclaw:latest"
./scripts/docker/setup.sh
Debian 13

Run as root

# Install Node.js
# Run as root
curl -fsSL https://deb.nodesource.com/setup_24.x | bash -
apt install nodejs

Run as non-root

# Switch to non-root account
# NOTE: Node.js 22.14 or newer is required for openclaw
su - <your-username>
node -v
npm --version

mkdir -p "$HOME/.npm-global"
npm config set prefix "$HOME/.npm-global"
npm prefix -g
export PATH="$HOME/.npm-global/bin:$PATH"

Edit: ~/.bashrc

export PATH="$HOME/.npm-global/bin:$PATH"

Install openclaw

npm install -g openclaw@latest
openclaw onboard --install-daemon

Post-Install

訂製 AI 助理

使用 3 個檔案量身訂製助理

Timezone
openclaw config set agents.defaults.userTimezone "Asia/Taipei"

CLI Commands

Check Service
docker compose run --rm openclaw-cli status
docker compose run --rm openclaw-cli gateway status
Pairing required

Approve the device

docker compose run --rm openclaw-cli devices list
docker compose run --rm openclaw-cli devices approve <request-id>
Model Configuration
# 主要模型設置
openclaw models list
openclaw models status
openclaw models set <provider/model>
openclaw models set-image <provider/model>

# 備用模型設置
openclaw models fallbacks list
openclaw models fallbacks add <provider/model>
openclaw models fallbacks remove <provider/model>
openclaw models fallbacks clear

Telegram

Configuration

Create a Bot from Telegram

  1. Open Telegram
  2. Chat with @BotFather 
  3. Run /newbot , follow prompts to create your bot, and save the token.
    • Bot Name: <whatever>
    • Bot Username: <unique-name and must end in 'bot'> 

Go to OpenClaw CLI

docker compose run --rm openclaw-cli channels add --channel telegram --token "<bot-token>"

Go to Telegram

  1. Open you-bot
  2. Send "Hi"
  3. Getting a pairing code "XXXXX"

Go to OpenClaw CLI

docker compose run --rm openclaw-cli pairing list telegram
docker compose run --rm openclaw-cli pairing approve telegram XXXXX

Done

Bot Commands

Gateway

Local only (default)

~/.openclaw/openclaw.json :

  "gateway": {
    "port": 18789,
    "mode": "local",
    "bind": "loopback",
    "controlUi": {
      "allowedOrigins": [
        "http://localhost:18789",
        "http://127.0.0.1:18789"
      ]
    },
    "auth": {
      "mode": "token",
      "token": "YOUR-OPENCLAW-TOKEN"
    },
Tailscale

openclaw.json 的 tailscale 功能不適用在 Docker 環境。

  1. Tailscale Console: Enable HTTPS Certicficates
  2. OpenClaw Host: Run
    1. sudo tailscale serve --bg --https=443 127.0.0.1:18789 
    2. tailscale serve status 
  3. Change ~/.openclaw/openclaw.json as follows
  4. Connect to tailnet, and then visit https://your-openclaw-device.tailnet-domain/?token=YOUR-TOKEN ,  such as https://dockers-vm.tailcb58c9.ts.net/ ?token=YOUR-TOKEN

~/.openclaw/openclaw.json :

  "gateway": {
    "port": 18789,
    "mode": "local",
    "bind": "loopback",
    "trustedProxies": ["127.0.0.1", "::1"],
    "controlUi": {
      "allowedOrigins": [
        "http://localhost:18789",
        "http://127.0.0.1:18789",
        "https://yourdevice.tailnet-domain"
      ]
    },
    "auth": {
      "mode": "token",
      "token": "YOUR-OPENCLAW-TOKEN"
    },

openclaw.json 的 tailscale 模式比較（不適用在 Docker 環境）

模式 存取範圍 HTTPS 認證 適用場景
serve 限 tailnet 自動 Tailscale identity headers/Token/Password 個人使用
funnel 公眾網路 自動 Password Webhook
bind: "tailnet" 限 tailnet 無 Token/Password 低延遲
off 限 localhost 無 Token SSH Port Forward

Skill

3rd Party

FAQ

Context limit exceeded

Context limit exceeded. I've reset our conversation to start fresh - please try again.

To prevent this, increase your compaction buffer by setting agents.defaults.compaction.reserveTokensFloor to 20000 or higher in your config.

Cause: 目前使用模型的 Context Window 與 壓縮緩衝參數不匹配。

Solution: 以 Gemma4-31B-it 為例

Edit: ~/.openclaw/openclaw.json

  "models": {
    "mode": "merge",
    "providers": {
      "custom-cpamc": {
        "baseUrl": "http://192.168.31.89:8317/v1",
        "api": "openai-completions",
        "apiKey": "sk-cvgaT1Z3EhJRBB5pu",
        "models": [
          {
            "id": "gemma-4-31b-it",
            "name": "gemma-4-31b-it (Custom Provider)",
            "contextWindow": 128000,
            "maxTokens": 4096,
            "input": [
              "text"
            ],
  "agents": {
    "defaults": {
      "workspace": "/home/alang/.openclaw/workspace",
      "model": {
        "primary": "custom-cpamc/gemma-4-31b-it"
      },
      "models": {
        "custom-cpamc/gemma-4-31b-it": {}
      },
      "compaction": {
        "reserveTokensFloor": 40000
      }
    }
  },

或者使用 CLI

openclaw config set agents.defaults.compaction.reserveTokensFloor 40000
LLM idle timeout
The model did not produce a response before the LLM idle timeout. Please try again, or increase agents.defaults.llm.idleTimeoutSeconds in your config (set to 0 to disable).

Solution:

openclaw config set agents.defaults.timeoutSeconds 600
openclaw config set agents.defaults.llm.idleTimeoutSeconds 600
gateway connect failed
gateway connect failed: GatewayClientRequestError: pairing required

檢查有無 pending devices

openclaw devices list
openclaw devices list --json

Agentic AI

Hermes Agent

由 Nous Research 打造的自我進化 AI Agent。唯一內建學習循環的 Agent — 它能從經驗中建立技能，在使用過程中不斷優化，主動提醒自己持久化知識，並在多次會話中逐步加深對你的理解。

Tutorials

Resources

Installation

Debian/Ubuntu

# Install
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

# Configure
hermes setup
Post-Install

訂製 AI 助理

Web UI

Hermes-WebUI
git clone https://github.com/nesquena/hermes-webui.git
cd hermes-webui
./start.sh
hermes-web-ui
Agentic AI

Agent Skills

CLI Commands

Install a skill 

# GitHub shorthand (owner/repo)
npx skills add vercel-labs/agent-skills

# Full GitHub URL
npx skills add https://github.com/vercel-labs/agent-skills

# Direct path to a skill in a repo
npx skills add https://github.com/vercel-labs/agent-skills/tree/main/skills/web-design-guidelines

# GitLab URL
npx skills add https://gitlab.com/org/repo

# Any git URL
npx skills add git@github.com:vercel-labs/agent-skills.git

# Local path
npx skills add ./my-local-skills

Use a Skill Without Installing

npx skills use vercel-labs/agent-skills@web-design-guidelines | claude
npx skills use vercel-labs/agent-skills --skill web-design-guidelines --agent claude-code

List skills

# List all installed skills (project and global)
npx skills list

# List only global skills
npx skills ls -g

# Filter by specific agents
npx skills ls -a claude-code -a cursor

Find skills

Update skills

# Update all skills (interactive scope prompt)
npx skills update

# Update a single skill by name
npx skills update my-skill

# Update multiple specific skills
npx skills update frontend-design web-design-guidelines

# Update only global or project skills
npx skills update -g
npx skills update -p

# Non-interactive (auto-detects scope: project if in a project, else global)
npx skills update -y

Remove a skill

# Remove interactively (select from installed skills)
npx skills remove

# Remove specific skill by name
npx skills remove web-design-guidelines

# Remove multiple skills
npx skills remove frontend-design web-design-guidelines

# Remove from global scope
npx skills remove --global web-design-guidelines

# Remove from specific agents only
npx skills remove --agent claude-code cursor my-skill

# Remove all installed skills without confirmation
npx skills remove --all

# Remove all skills from a specific agent
npx skills remove --skill '*' -a cursor

# Remove a specific skill from all agents
npx skills remove my-skill --agent '*'

# Use 'rm' alias
npx skills rm my-skill

Create a skill

# Create SKILL.md in current directory
npx skills init

# Create a new skill in a subdirectory
npx skills init my-skill

Resources

語音生成

影片生成

網頁搜尋

PDF

Web Scraper

- Crawlee

A web scraping and browser automation library. 

- ScrapeGraphAI

ScrapeGraphAI is a open-source web scraping python library designed to usher in a new era of scraping tools.

- Crew AI

Crew AI is a collaborative working system designed to enable various artificial intelligence agents to work together as a team, efficiently accomplishing complex tasks. Each agent has a specific role, resembling a team composed of researchers, writers, and planners.

- Scraperr

Self-hosted webscraper.

- Crawl4AI

Open-source LLM Friendly Web Crawler & Scraper.

- Scrapling

Effortless Web Scraping for the Modern Web

- Obscura

The open-source headless browser for AI agents and web scraping.

Agentic AI

Pi Agent

Pi 是一款極簡的終端編碼工具。讓 Pi 適應您的工作流程，而非讓您去適應它。

官網：Pi Coding Agent

Tutorials

Video AI

SentrySearch

利用 AI 語意搜尋長影片的片段，可用於警察單位檢索大量 CCTV 影像檔，或其他用途。