feat: add real-time caption CLI prototype with Qwen3-ASR
Add a C++/Qt6/CMake CLI prototype that captures the default sink monitor through libpulse and transcribes it in-process with Qwen3-ASR via libllama/libmtmd. Add phase 0/1 verification docs. 新增实时字幕命令行原型：libpulse 采集默认输出 monitor，进程内用 libllama/libmtmd 跑 Qwen3-ASR 推理。附阶段 0/1 验证文档。
This commit is contained in:
blumia committed
commit
18dbc9c8d3
18 files changed
+1682
No files matched your search
@@ -0,0 +1,72 @@
|
||||
# Phase 0 验证实验
|
||||
|
||||
目标：在动手写 C++/Qt 之前，先证实三个关键假设。
|
||||
|
||||
1. **系统播放音频可采集**：默认 sink 的 `monitor` 源能被读到。
|
||||
2. **Qwen3-ASR 能在这套系统库上跑起来**：发行版自带的 llama.cpp 支持 Qwen3-ASR。
|
||||
3. **CPU 实时性够用**：0.6B 量化模型在这台没有独显的机器上能否做到近实时。
|
||||
|
||||
## 环境（实测）
|
||||
|
||||
| 项 | 值 |
|
||||
|---|---|
|
||||
| 系统 | Deepin 25 (crimson), X11 |
|
||||
| CPU | i5-13420H, 8 核 12 线程 |
|
||||
| GPU | Intel UHD（无 CUDA） |
|
||||
| 音频 | PipeWire 1.6.4 + pipewire-pulse，`libpulse` 17.0 |
|
||||
| llama.cpp | 系统包 `9951+dfsg-1deepin1`（`libllama0` / `libmtmd` / `libggml0`）|
|
||||
|
||||
## 关键结论（细节见 ../../docs/phase0-verification.md）
|
||||
|
||||
- 主模型 GGUF 的 `general.architecture = qwen3vl` → 系统 `libllama` 有 `llama_model_qwen3vl`，**能加载**。
|
||||
- mmproj 的 `clip.audio.projector_type = qwen3a` → 系统 `libmtmd` 有 `clip_graph_qwen3a` / `mtmd_audio_preprocessor_qwen3a`，**音频编码器也支持**。
|
||||
- 转写中英均正确，`language <X><asr_text>...` 前缀里带语言识别。
|
||||
- 稳态 RTF ≈ 0.19（纯推理），实时绰绰有余。
|
||||
|
||||
## 复现步骤
|
||||
|
||||
### 1. 拿到 llama.cpp 的 CLI（不需要 root）
|
||||
|
||||
系统只装了库，没装工具，且本机没有免密 sudo。直接从 deb 解包：
|
||||
|
||||
```bash
|
||||
cd /tmp/opencode
|
||||
apt-get download llama.cpp-tools
|
||||
mkdir -p llama-tools && dpkg-deb -x llama.cpp-tools_*.deb llama-tools
|
||||
# 运行时必须指定 LD_LIBRARY_PATH（包含 /usr/lib/x86_64-linux-gnu/llama）
|
||||
```
|
||||
|
||||
### 2. 下载模型
|
||||
|
||||
网络只通了 `hf-mirror.com`，用 aria2c 多连接（单连接约 3MB/s，多连接可到 19MB/s）：
|
||||
|
||||
```bash
|
||||
B=https://hf-mirror.com/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main
|
||||
aria2c -x16 -s16 -k4M -o Qwen3-ASR-0.6B-Q8_0.gguf "$B/Qwen3-ASR-0.6B-Q8_0.gguf"
|
||||
aria2c -x16 -s16 -k4M -o mmproj-Qwen3-ASR-0.6B-Q8_0.gguf "$B/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf"
|
||||
```
|
||||
|
||||
> 换模型时 **主模型 + mmproj 必须成对替换**。
|
||||
|
||||
### 3. 验证
|
||||
|
||||
```bash
|
||||
# 音频采集链路（播放的同时录 monitor）
|
||||
./loopback_test.sh <某个音频> /tmp/captured_48k.wav
|
||||
ffmpeg -y -i /tmp/captured_48k.wav -ar 16000 -ac 1 -c:a pcm_s16le /tmp/captured_16k.wav
|
||||
|
||||
# 转写 + 测 RTF
|
||||
./bench.sh /tmp/captured_16k.wav
|
||||
|
||||
# 换量化模型对比
|
||||
MODEL=/tmp/opencode/models/Qwen3-ASR-0.6B-Q4_K_M.gguf ./bench.sh some.wav
|
||||
|
||||
# 只读 GGUF 头部元数据（不下载整个文件）
|
||||
python3 gguf_head.py
|
||||
```
|
||||
|
||||
## 已知坑
|
||||
|
||||
- **`llama-cli` 会挂住**：不给 `-st` / 不让 stdin EOF 时，生成完会回到交互读取循环。`bench.sh` 已用 `-st --simple-io </dev/null` + 外层 `timeout` 处理。
|
||||
- **输出需清洗**：结果形如 `language English<asr_text>正文`，接入时要把语言名与正文分开（语言名正好喂给 UI 的语言标签）。
|
||||
- **发行版 CMake config 路径非标准**：`/usr/lib/x86_64-linux-gnu/llama/cmake-private/llama-config.cmake`，`find_package(llama)` 默认找不到；它有配套的 `pkgconfig/llama.pc`，走 pkg-config 更稳。
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Phase 0 验证脚本：用 llama.cpp + Qwen3-ASR 转写一段音频并测 RTF
|
||||
#
|
||||
# 依赖（本机验证环境：Deepin 25 / Qt 6.8 / x86_64）：
|
||||
# libllama0, libllama-dev, libggml0, libggml-dev, libmtmd （系统自带 9951+dfsg-1deepin1）
|
||||
# llama.cpp-tools 的二进制（本脚本从 deb 解包到 LLAMA_TOOLS，不需要 root 安装）
|
||||
#
|
||||
# 用法: ./bench.sh <audio.wav> [prompt] # prompt 默认 <__media__>
|
||||
# 可选环境变量: THREADS / MODEL / MMPROJ / LLAMA_TOOLS / LLAMA_LIBDIR / TMO
|
||||
set -uo pipefail
|
||||
|
||||
LLAMA_TOOLS="${LLAMA_TOOLS:-/tmp/opencode/llama-tools}"
|
||||
LLAMA_LIBDIR="${LLAMA_LIBDIR:-/usr/lib/x86_64-linux-gnu/llama}"
|
||||
MODEL="${MODEL:-/tmp/opencode/models/Qwen3-ASR-0.6B-Q8_0.gguf}"
|
||||
MMPROJ="${MMPROJ:-/tmp/opencode/models/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf}"
|
||||
AUDIO="${1:?usage: bench.sh <audio> [prompt]}"
|
||||
PROMPT="${2:-<__media__>}"
|
||||
THREADS="${THREADS:-8}"
|
||||
CLI="$LLAMA_TOOLS/usr/bin/llama-cli"
|
||||
|
||||
export LD_LIBRARY_PATH="$LLAMA_LIBDIR:$LLAMA_TOOLS/usr/lib/x86_64-linux-gnu/llama"
|
||||
|
||||
dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$AUDIO")
|
||||
printf '音频: %s (%ss) 模型: %s 线程: %s\n' "$AUDIO" "$dur" "$(basename "$MODEL")" "$THREADS"
|
||||
|
||||
# 关键: stdin 必须接 /dev/null 并在外层套 timeout
|
||||
# 否则 llama-cli 生成完会回到 stdin 读取循环而挂住
|
||||
t0=$(date +%s.%N)
|
||||
timeout "${TMO:-300}" "$CLI" -m "$MODEL" --mmproj "$MMPROJ" --audio "$AUDIO" \
|
||||
-p "$PROMPT" -n 256 --temp 0 -t "$THREADS" -st --simple-io \
|
||||
</dev/null >/tmp/lc_out.log 2>/tmp/lc_err.log
|
||||
rc=$?
|
||||
t1=$(date +%s.%N)
|
||||
|
||||
[ "$rc" = 124 ] && echo "(警告: 被 timeout 杀掉)"
|
||||
echo "---- 转写结果 ----"
|
||||
cat /tmp/lc_out.log
|
||||
echo "---- 计时 ----"
|
||||
python3 -c "print(f'墙钟总耗时: {$t1-$t0:.2f}s 音频: $dur s → RTF(含加载) ≈ {($t1-$t0)/$dur:.3f}')"
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""只抓 GGUF 头部（HTTP Range）读元数据，不下载整个文件。
|
||||
用法: gguf_head.py <url> [<url> ...]"""
|
||||
import sys, struct, urllib.request
|
||||
|
||||
TYPES = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:None,9:None,10:8,11:8,12:8}
|
||||
NAMES = {0:"u8",1:"i8",2:"u16",3:"i16",4:"u32",5:"i32",6:"f32",7:"bool",8:"str",9:"arr",10:"u64",11:"i64",12:"f64"}
|
||||
|
||||
def readstr(b,o):
|
||||
n,=struct.unpack_from("<Q",b,o); o+=8
|
||||
return b[o:o+n].decode("utf-8","replace"), o+n
|
||||
|
||||
def peek(url, limit=4_000_000):
|
||||
req=urllib.request.Request(url, headers={"Range":f"bytes=0-{limit}"})
|
||||
b=urllib.request.urlopen(req, timeout=60).read()
|
||||
assert b[:4]==b"GGUF", "not a GGUF"
|
||||
n_kv,=struct.unpack_from("<Q",b,16); o=24
|
||||
out={}
|
||||
for _ in range(n_kv):
|
||||
k,o=readstr(b,o)
|
||||
t,=struct.unpack_from("<I",b,o); o+=4
|
||||
if t==8:
|
||||
v,o=readstr(b,o)
|
||||
elif t==9:
|
||||
et,=struct.unpack_from("<I",b,o); o+=4
|
||||
n,=struct.unpack_from("<Q",b,o); o+=8
|
||||
if et==8:
|
||||
v=[readstr(b,o)[0] for _ in range(min(n,4))]
|
||||
for _ in range(n): _,o=readstr(b,o)
|
||||
else:
|
||||
o+=n*TYPES[et]; v=f"<{n}x{NAMES[et]}>"
|
||||
else:
|
||||
sz=TYPES[t]; v=int.from_bytes(b[o:o+sz],"little",signed=t in (1,3,5,11)); o+=sz
|
||||
out[k]=v
|
||||
if k=="general.architecture":
|
||||
break
|
||||
return out
|
||||
|
||||
for url in sys.argv[1:]:
|
||||
print(f"=== {url.split('/resolve/')[0].split('/')[-2] + '/' + url.split('/')[-1]} ===")
|
||||
try:
|
||||
m=peek(url)
|
||||
print(" architecture =", m.get("general.architecture"), "| name =", m.get("general.name"))
|
||||
for k,v in m.items():
|
||||
if "clip" in k or "projector" in k or "audio" in k:
|
||||
print(f" {k} = {str(v)[:70]}")
|
||||
except Exception as e:
|
||||
print(" ERR", e)
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Phase 0: 验证"系统播放音频 → monitor 源采集"链路
|
||||
# 用法: ./loopback_test.sh <要播放的音频文件> <输出wav>
|
||||
set -euo pipefail
|
||||
SRC="${1:?usage: loopback_test.sh <file> <out.wav>}"
|
||||
OUT="${2:?usage: loopback_test.sh <file> <out.wav>}"
|
||||
|
||||
echo "默认 sink : $(pactl get-default-sink)"
|
||||
echo "默认 sink 的 monitor: $(pactl get-default-sink).monitor"
|
||||
|
||||
# 1) 后台开始录制默认输出的 monitor（48k 立体声 float32，与设备速率一致）
|
||||
parec -d @DEFAULT_MONITOR@ --format=float32le --rate=48000 --channels=2 \
|
||||
--file-format=wav "$OUT" &
|
||||
REC=$!
|
||||
sleep 0.6 # 等录制真正开始
|
||||
|
||||
# 2) 播放测试音频到默认输出
|
||||
paplay --device="$(pactl get-default-sink)" "$SRC"
|
||||
|
||||
sleep 0.6
|
||||
kill "$REC" 2>/dev/null || true
|
||||
wait "$REC" 2>/dev/null || true
|
||||
|
||||
echo "采集完成: $OUT"
|
||||
ffprobe -v error -show_entries format=duration -of csv=p=0 "$OUT" | awk '{printf "采集时长: %.2f s\n",$1}'
|
||||
Reference in new issue
Block a user