2026.6.25 update 更新

官方暂未支持，推荐用ollama编译的二进制文件启动

部署链接参考：https://github.com/AuditAIH/audit-tool-skills/tree/main/Rerank

手动启动固定端口

# 手动拉取模型
ollama pull AuditAid/Qwen3_Reranker:0.6B_Q8
#手动指定GPU启动，并启动在rerank端口
export GGML_BACKEND_PATH=/usr/local/lib/ollama/cuda_v13/libggml-cuda.so
export LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13
export CUDA_VISIBLE_DEVICES=0
/usr/local/lib/ollama/llama-server \
  --model /usr/share/ollama/.ollama/models/blobs/sha256-22c9979ce4fbcdc5acdc310c6641c32797eff1aa980b8f7a2db8a8ea23429a48 \
  --host 0.0.0.0 \
  --port 11435 \
  --no-webui \
  --rerank \
  --ctx-size 8192

快速开始 （Quick Start）

# 下载脚本到本地文件 startup_llama.cpp.sh
#如需代理请在github的http前面追加https://gh-proxy.org/https://raw.git...
wget -q "https://raw.githubusercontent.com/AuditAIH/audit-tool-skills/main/Rerank/startup_llama.cpp.sh" -O startup_llama.cpp.sh
# 运行脚本
bash startup_llama.cpp.sh

请求方式：

curl -X POST http://localhost:11435/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-reranker",
    "query": "Apple",
    "documents": [
      "apple",
      "banana",
      "fruit",
      "vegetable"
    ]
  }'

预期响应

{
"model":"Qwen3-reranker","object":"list",
"usage":{"prompt_tokens":296,"total_tokens":296},
"results":
[{"index":0,"relevance_score":0.9830306172370911},{"index":2,"relevance_score":0.001323841861449182},{"index":1,"relevance_score":0.0005493499338626862},{"index":3,"relevance_score":0.0002995904360432178}]
}