feat(stream): add append-only streaming mode with flat cost
Add an append-only streaming path that feeds each audio chunk as a new conversation turn, so nothing is ever recomputed: the KV grows by append, no prefix is re-decoded and no audio is re-encoded. Per-step cost becomes flat instead of growing with the segment (RTF 1.06 -> 0.26 on 62.6s of continuous speech), but accuracy requires ~8s chunks because the audio encoder needs its 8s attention window, so text only appears every 8s. The incremental mode stays the default. 新增追加式流式:每块音频作为一个新的对话轮追加,前缀不重算、旧音频不重 编码,单步成本不再随段落增长(连续 62.6s 语音 RTF 从 1.06 降到 0.26)。 但准确率要求 chunk≈8s,否则会被截断,所以出字粒度是 8s;增量式仍为默认。
This commit is contained in:
1 parent
7e5257429a
commit
85b206f1cd
10 files changed
+326
-14
No files matched your search
+4
-3
@@ -72,12 +72,13 @@ int runStreamFile(lc::Qwen3Asr & asr, const QString & path, const lc::StreamOpti
|
|||||||
const size_t chunkSamples = (size_t) (sop.chunkSec * asr.sampleRate());
|
const size_t chunkSamples = (size_t) (sop.chunkSec * asr.sampleRate());
|
||||||
|
|
||||||
const auto onStep = [](const lc::StreamingAsr::Step & s) {
|
const auto onStep = [](const lc::StreamingAsr::Step & s) {
|
||||||
std::printf("\n[chunk %2d] 窗口 %5.1fs / 编码 %5.2fs / 本步 %5.2fs%s\n",
|
std::printf("\n[chunk %2d] 累计 %5.1fs | 音频 %5.2fs | 前缀 %5.2fs | 生成 %5.2fs (%d tok) "
|
||||||
s.chunkIndex, s.windowSeconds, s.encodeSeconds, s.inferSeconds,
|
"| 合计 %5.2fs%s\n",
|
||||||
|
s.chunkIndex, s.windowSeconds, s.encodeSeconds, s.prefillSeconds,
|
||||||
|
s.genSeconds, s.newTokens, s.inferSeconds,
|
||||||
s.ok ? "" : " <失败>");
|
s.ok ? "" : " <失败>");
|
||||||
if (!s.ok) std::printf(" error: %s\n", s.error.c_str());
|
if (!s.ok) std::printf(" error: %s\n", s.error.c_str());
|
||||||
std::printf(" 已提交: %s\n", s.committed.c_str());
|
std::printf(" 已提交: %s\n", s.committed.c_str());
|
||||||
std::printf(" 含尾部: %s\n", s.full.c_str());
|
|
||||||
std::fflush(stdout);
|
std::fflush(stdout);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -44,7 +44,8 @@ inline void addStreamOptions(QCommandLineParser & p)
|
|||||||
p.addOption(makeOption("stream-segment", "单段最长时长,单位秒;0=不限", "sec", "30"));
|
p.addOption(makeOption("stream-segment", "单段最长时长,单位秒;0=不限", "sec", "30"));
|
||||||
p.addOption(makeOption("unfixed-tokens", "前缀回滚丢弃的 token 数", "n", "5"));
|
p.addOption(makeOption("unfixed-tokens", "前缀回滚丢弃的 token 数", "n", "5"));
|
||||||
p.addOption(makeOption("unfixed-chunks", "冷启动不带前缀的 chunk 数", "n", "2"));
|
p.addOption(makeOption("unfixed-chunks", "冷启动不带前缀的 chunk 数", "n", "2"));
|
||||||
p.addOption(makeOption("stream-mode", "流式后端: incremental 或 window", "mode", "incremental"));
|
p.addOption(makeOption("stream-mode", "流式后端: incremental / append / window", "mode",
|
||||||
|
"incremental"));
|
||||||
}
|
}
|
||||||
|
|
||||||
inline AsrOptions asrOptions(const QCommandLineParser & p)
|
inline AsrOptions asrOptions(const QCommandLineParser & p)
|
||||||
@@ -67,7 +68,8 @@ inline StreamOptions streamOptions(const QCommandLineParser & p)
|
|||||||
o.unfixedTokenNum = p.value("unfixed-tokens").toInt();
|
o.unfixedTokenNum = p.value("unfixed-tokens").toInt();
|
||||||
o.unfixedChunkNum = p.value("unfixed-chunks").toInt();
|
o.unfixedChunkNum = p.value("unfixed-chunks").toInt();
|
||||||
o.maxTokensPerStep = p.value("max-tokens").toInt();
|
o.maxTokensPerStep = p.value("max-tokens").toInt();
|
||||||
o.incremental = p.value("stream-mode") != QLatin1String("window");
|
o.incremental = p.value("stream-mode") == QLatin1String("incremental");
|
||||||
|
o.appendOnly = p.value("stream-mode") == QLatin1String("append");
|
||||||
return o;
|
return o;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,13 +2,66 @@
|
|||||||
|
|
||||||
状态:**完成**。准确率与离线转写一致,单步计算稳定在 2s 预算内。
|
状态:**完成**。准确率与离线转写一致,单步计算稳定在 2s 预算内。
|
||||||
|
|
||||||
|
## 两种流式模式(实测对比)
|
||||||
|
|
||||||
|
代码里实现了两条流式路径,用 `--stream-mode` 切换。它们对应一个**无法同时满足的三选二**:
|
||||||
|
|
||||||
|
| 目标 | `incremental`(默认) | `append`(追加式) |
|
||||||
|
|---|---|---|
|
||||||
|
| 成本不随说话时长增长 | ❌ | ✅ |
|
||||||
|
| CPU 效率 | RTF 1.06(无段落上限时) | **RTF 0.26** |
|
||||||
|
| 1–2s 就出字 | ✅ | ❌ 需 8s chunk |
|
||||||
|
| 准确率 | ✅(2s 提交 + 8s 编码切片) | ✅(仅 chunk≈8s 时) |
|
||||||
|
|
||||||
|
同一段 62.6s 连续语音(不设段落上限):
|
||||||
|
|
||||||
|
| 模式 | 墙钟 | RTF | 首步 | 末步 | 最慢步 |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `incremental` | 66.3s | 1.06 | 0.62s | 3.14s | 4.06s |
|
||||||
|
| `append` chunk=2 | 32.6s | 0.52 | 0.70s | 0.67s | 1.32s |
|
||||||
|
| `append` chunk=8 | **16.0s** | **0.26** | — | — | 2.52s |
|
||||||
|
|
||||||
|
`append` 的每步耗时几乎不随总长变化;`incremental` 则线性增长(尾步是最慢的)。
|
||||||
|
|
||||||
|
### 为什么 `append` 必须用大 chunk
|
||||||
|
|
||||||
|
准确率对 **chunk 长度**极其敏感,因为编码器(AuT)用的是 **8 秒注意力窗口**(`n_window_infer = 800` 帧 @100Hz)。每块只用自己的长度去编码时:
|
||||||
|
|
||||||
|
| chunk | 中文实测结果 |
|
||||||
|
|---|---|
|
||||||
|
| 2s | ❌ `不排除购房者可以在互。`(每块都被截断,`互联网上下单` 全丢) |
|
||||||
|
| 4s | ⚠️ `不排除购房者可以在互联网上下单,不排除。`(基本对,边界多一个词) |
|
||||||
|
| 8s | ✅ 与离线一致 |
|
||||||
|
|
||||||
|
所以「便宜/平坦」和「1–2s 出字」对 Qwen3-ASR 是互斥的:要准确就得给编码器 8s 上下文,给了就必然 8s 才出一次字。
|
||||||
|
|
||||||
|
### 追加式为什么便宜
|
||||||
|
|
||||||
|
`append` 把每块音频当作一个新的对话轮追加到 KV 末尾:
|
||||||
|
|
||||||
|
```
|
||||||
|
[user: audio_1][assistant: text_1][user: audio_2][assistant: text_2]...
|
||||||
|
```
|
||||||
|
|
||||||
|
纯追加 —— **前缀完全不重算、旧音频不重编码、KV 永不回退**,所以单步成本与音频总长无关。
|
||||||
|
|
||||||
|
而 `incremental` 每步都要把「音频尾部 + 到目前为止的全部文本前缀」按新的音频上下文重算一遍(因果注意力决定了:把新音频插到文本之前,文本之后的 KV 全部作废)。实测拆解显示这一步的代价是 O(N²):
|
||||||
|
|
||||||
|
| 累计音频 | 音频(编码+写KV) | 文本前缀 prefill | 生成 | 合计 |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 2.0s | 0.35s | 0.03s | 0.19s | 0.58s |
|
||||||
|
| 46.0s | 1.23s | 1.20s | 0.38s | 2.81s |
|
||||||
|
| 62.6s | 1.46s | **1.53s** | 0.16s | 3.15s |
|
||||||
|
|
||||||
|
「文本前缀 prefill」从 0.03s 涨到 1.53s,占末尾步耗时的一半。`append` 模式把它降到了 **0.06s**。
|
||||||
|
|
||||||
## 结论
|
## 结论
|
||||||
|
|
||||||
| 指标 | 结果 |
|
| 数据 | 结果 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 准确率 | 流式输出与离线转写逐字一致(中/英各自验证) |
|
| 准确率 | 流式输出与离线转写逐字一致(中/英各自验证) |
|
||||||
| 单步计算 | 最慢 1.83s(预算 2.0s,提交粒度 2s) |
|
| 单步计算(incremental,segment≤20s) | 最慢 1.84s(预算 2.0s,提交粒度 2s) |
|
||||||
| 整体 RTF | 约 0.66 |
|
| 整体 RTF | 约 0.59 |
|
||||||
| 实现 | 纯 C++,libllama + libmtmd,无 Python |
|
| 实现 | 纯 C++,libllama + libmtmd,无 Python |
|
||||||
|
|
||||||
## 算法
|
## 算法
|
||||||
@@ -74,15 +127,19 @@
|
|||||||
## 用法
|
## 用法
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 流式转写(增量 + 8s 切片 + 2s 提交)
|
# 流式转写(默认 incremental:8s 切片 + 前缀回滚 + 2s 提交)
|
||||||
./build/live-caption-cli --model MODEL --mmproj MMPROJ \
|
./build/live-caption-cli --model MODEL --mmproj MMPROJ \
|
||||||
--stream-file audio_16k_mono.wav [--stream-chunk 2] [--stream-segment 30]
|
--stream-file audio_16k_mono.wav [--stream-chunk 2] [--stream-segment 30]
|
||||||
|
|
||||||
# 对比用:旧的"重编码整个窗口"实现
|
# 追加式:成本平坦、省 CPU,但需要大 chunk 才准确
|
||||||
|
./build/live-caption-cli --model MODEL --mmproj MMPROJ \
|
||||||
|
--stream-file audio_16k_mono.wav --stream-mode append --stream-chunk 8
|
||||||
|
|
||||||
|
# 对比用:最早的"重编码整个窗口"实现
|
||||||
./build/live-caption-cli ... --stream-mode window --stream-window 4
|
./build/live-caption-cli ... --stream-mode window --stream-window 4
|
||||||
```
|
```
|
||||||
|
|
||||||
相关参数:`--stream-chunk`(提交粒度)、`--stream-window`(仅 window 模式)、`--stream-segment`(单段上限)、`--unfixed-tokens`、`--unfixed-chunks`。
|
相关参数:`--stream-mode`(incremental / append / window)、`--stream-chunk`(提交粒度)、`--stream-segment`(单段上限)、`--stream-window`(仅 window 模式)、`--unfixed-tokens`、`--unfixed-chunks`。
|
||||||
|
|
||||||
## 遗留
|
## 遗留
|
||||||
|
|
||||||
|
|||||||
+11
-2
@@ -24,8 +24,10 @@ struct AsrStreamStep {
|
|||||||
std::string newText; // 本步新生成的文本(不含前缀,未 trim)
|
std::string newText; // 本步新生成的文本(不含前缀,未 trim)
|
||||||
int newTokens = 0;
|
int newTokens = 0;
|
||||||
int audioTokens = 0;
|
int audioTokens = 0;
|
||||||
double encodeSeconds = 0.0;
|
double encodeSeconds = 0.0; // 音频编码 + 写入 KV
|
||||||
double inferSeconds = 0.0;
|
double prefillSeconds = 0.0; // 文本前缀(tail + prefix)prefill
|
||||||
|
double genSeconds = 0.0; // 贪心生成
|
||||||
|
double inferSeconds = 0.0; // 以上合计
|
||||||
};
|
};
|
||||||
|
|
||||||
// 把推理后端抽象出来,好处是流式策略(StreamingAsr)和实时管线(LivePipeline)
|
// 把推理后端抽象出来,好处是流式策略(StreamingAsr)和实时管线(LivePipeline)
|
||||||
@@ -44,6 +46,13 @@ public:
|
|||||||
// 流式:只编码新音频,已解码的音频留在 KV cache 里复用
|
// 流式:只编码新音频,已解码的音频留在 KV cache 里复用
|
||||||
virtual AsrStreamStep streamAppend(const float * pcm, size_t nSamples,
|
virtual AsrStreamStep streamAppend(const float * pcm, size_t nSamples,
|
||||||
const std::string & prefix, int maxTokens) = 0;
|
const std::string & prefix, int maxTokens) = 0;
|
||||||
|
|
||||||
|
// 追加式轮次(append-only):把这一小段音频当作一个新的对话轮,
|
||||||
|
// 追加到已有 KV 的末尾,不重算任何前缀、也不重编码旧音频。
|
||||||
|
// 代价是每块只用自己的长度做编码(没有 8s 上下文),准确率需要实测确认。
|
||||||
|
virtual AsrStreamStep streamAppendTurn(const float * pcm, size_t nSamples,
|
||||||
|
int maxTokens) = 0;
|
||||||
|
|
||||||
virtual void streamReset() = 0;
|
virtual void streamReset() = 0;
|
||||||
|
|
||||||
// 用词表把 text 编码成 token、丢掉最后 dropLast 个、再还原成文本。
|
// 用词表把 text 编码成 token、丢掉最后 dropLast 个、再还原成文本。
|
||||||
|
|||||||
@@ -38,6 +38,13 @@ struct Qwen3Asr::Impl {
|
|||||||
size_t stableSamples = 0; // 已固化进 KV 的音频样本数(8s 对齐)
|
size_t stableSamples = 0; // 已固化进 KV 的音频样本数(8s 对齐)
|
||||||
llama_pos nPastStable = 0; // stable 音频末尾的 KV 位置
|
llama_pos nPastStable = 0; // stable 音频末尾的 KV 位置
|
||||||
|
|
||||||
|
// 追加式轮次(append-only)状态
|
||||||
|
bool turnReady = false;
|
||||||
|
std::vector<llama_token> firstHead; // 第 1 轮的轮首文本(含 <|audio_start|>)
|
||||||
|
std::vector<llama_token> contHead; // 后续轮的轮首文本(<|im_end|>\n<|im_start|>user\n<|audio_start|>)
|
||||||
|
llama_pos nPastTurn = 0;
|
||||||
|
int turnCount = 0;
|
||||||
|
|
||||||
std::string buildPrompt(const std::string & marker) const
|
std::string buildPrompt(const std::string & marker) const
|
||||||
{
|
{
|
||||||
const char * tmpl = llama_model_chat_template(model, nullptr);
|
const char * tmpl = llama_model_chat_template(model, nullptr);
|
||||||
@@ -112,6 +119,48 @@ struct Qwen3Asr::Impl {
|
|||||||
nPast = np;
|
nPast = np;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 把带 media marker 的 prompt 拆成「音频前文本」和「音频后文本」两组 token
|
||||||
|
bool splitPrompt(const std::string & skeleton,
|
||||||
|
std::vector<llama_token> & head,
|
||||||
|
std::vector<llama_token> & tail,
|
||||||
|
std::string * err)
|
||||||
|
{
|
||||||
|
std::vector<float> dummy(1600, 0.0f); // 0.1s 占位,只为触发音频分块
|
||||||
|
mtmd_bitmap * bmp = mtmd_bitmap_init_from_audio(dummy.size(), dummy.data());
|
||||||
|
mtmd_input_chunks * chunks = mtmd_input_chunks_init();
|
||||||
|
const mtmd_input_text itext{ skeleton.c_str(), true, true };
|
||||||
|
const mtmd_bitmap * bms[1] = { bmp };
|
||||||
|
|
||||||
|
if (mtmd_tokenize(mctx, chunks, &itext, bms, 1) != 0) {
|
||||||
|
mtmd_bitmap_free(bmp);
|
||||||
|
mtmd_input_chunks_free(chunks);
|
||||||
|
if (err) *err = "mtmd_tokenize 失败";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
mtmd_bitmap_free(bmp);
|
||||||
|
|
||||||
|
bool seenAudio = false;
|
||||||
|
for (size_t i = 0; i < mtmd_input_chunks_size(chunks); ++i) {
|
||||||
|
const mtmd_input_chunk * c = mtmd_input_chunks_get(chunks, i);
|
||||||
|
if (mtmd_input_chunk_get_type(c) == MTMD_INPUT_CHUNK_TYPE_AUDIO) {
|
||||||
|
seenAudio = true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
size_t n = 0;
|
||||||
|
const llama_token * toks = mtmd_input_chunk_get_tokens_text(c, &n);
|
||||||
|
if (!toks || n == 0) continue;
|
||||||
|
std::vector<llama_token> & dst = seenAudio ? tail : head;
|
||||||
|
dst.insert(dst.end(), toks, toks + n);
|
||||||
|
}
|
||||||
|
mtmd_input_chunks_free(chunks);
|
||||||
|
|
||||||
|
if (head.empty()) {
|
||||||
|
if (err) *err = "prompt 里没有音频之前的文本";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
Qwen3Asr::Qwen3Asr() : d(new Impl) {}
|
Qwen3Asr::Qwen3Asr() : d(new Impl) {}
|
||||||
@@ -313,6 +362,109 @@ void Qwen3Asr::streamReset()
|
|||||||
d->streamAudio.clear();
|
d->streamAudio.clear();
|
||||||
d->stableSamples = 0;
|
d->stableSamples = 0;
|
||||||
d->nPastStable = 0;
|
d->nPastStable = 0;
|
||||||
|
|
||||||
|
d->turnReady = false;
|
||||||
|
d->firstHead.clear();
|
||||||
|
d->contHead.clear();
|
||||||
|
d->nPastTurn = 0;
|
||||||
|
d->turnCount = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 追加式轮次:把每小段音频当作一个新的对话轮追加到 KV 末尾。
|
||||||
|
// 序列形如 [user: audio_1][assistant: text_1][user: audio_2][assistant: text_2]...
|
||||||
|
// 好处是纯追加:不重算前缀、不重编码旧音频,单步成本与音频总长无关。
|
||||||
|
bool Qwen3Asr::prepareTurn(std::string * error)
|
||||||
|
{
|
||||||
|
if (d->turnReady) return true;
|
||||||
|
|
||||||
|
const std::string marker = mtmd_default_marker();
|
||||||
|
// 第 1 轮是完整的 user/assistant 结构;后续轮接在上一轮文本之后开新一轮。
|
||||||
|
const std::string s1 = "<|im_start|>user\n" + marker +
|
||||||
|
"<|im_end|>\n<|im_start|>assistant\n";
|
||||||
|
const std::string sc = "<|im_end|>\n<|im_start|>user\n" + marker +
|
||||||
|
"<|im_end|>\n<|im_start|>assistant\n";
|
||||||
|
|
||||||
|
std::vector<llama_token> tail1, tailC;
|
||||||
|
if (!d->splitPrompt(s1, d->firstHead, tail1, error)) return false;
|
||||||
|
if (!d->splitPrompt(sc, d->contHead, tailC, error)) return false;
|
||||||
|
|
||||||
|
// 两种轮尾应当一致
|
||||||
|
d->tailTokens = tail1;
|
||||||
|
|
||||||
|
llama_memory_clear(llama_get_memory(d->ctx), true);
|
||||||
|
d->nPastTurn = 0;
|
||||||
|
d->turnCount = 0;
|
||||||
|
d->turnReady = true;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
AsrStreamStep Qwen3Asr::streamAppendTurn(const float * pcm, size_t nSamples, int maxTokens)
|
||||||
|
{
|
||||||
|
AsrStreamStep res;
|
||||||
|
if (!d->model || !d->ctx || !d->mctx) { res.error = "模型未加载"; return res; }
|
||||||
|
if (nSamples == 0) { res.error = "音频为空"; return res; }
|
||||||
|
|
||||||
|
std::string err;
|
||||||
|
if (!prepareTurn(&err)) { res.error = err; return res; }
|
||||||
|
|
||||||
|
const auto t0 = std::chrono::steady_clock::now();
|
||||||
|
|
||||||
|
// 1) 轮首文本(纯追加,位置接在 KV 末尾)
|
||||||
|
const std::vector<llama_token> & head = (d->turnCount == 0) ? d->firstHead : d->contHead;
|
||||||
|
llama_pos p = d->nPastTurn;
|
||||||
|
if (!d->decodeTokens(head.data(), (int32_t) head.size(), p, false)) {
|
||||||
|
res.error = "解码轮首失败";
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
p += (llama_pos) head.size();
|
||||||
|
|
||||||
|
// 2) 这一小段音频:只编码一次,写进 KV 就永久保留
|
||||||
|
const llama_pos beforeAudio = p;
|
||||||
|
if (!d->appendAudio(pcm, nSamples, p, &err)) {
|
||||||
|
res.error = "音频编码/解码失败: " + err;
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
res.audioTokens = (int) (p - beforeAudio);
|
||||||
|
const auto tEnc = std::chrono::steady_clock::now();
|
||||||
|
|
||||||
|
// 3) 轮尾文本 + 开始生成
|
||||||
|
if (!d->decodeTokens(d->tailTokens.data(), (int32_t) d->tailTokens.size(), p, true)) {
|
||||||
|
res.error = "解码轮尾失败";
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
p += (llama_pos) d->tailTokens.size();
|
||||||
|
|
||||||
|
const auto tPre = std::chrono::steady_clock::now();
|
||||||
|
|
||||||
|
llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
|
||||||
|
llama_sampler_chain_add(smpl, llama_sampler_init_greedy());
|
||||||
|
|
||||||
|
std::string out;
|
||||||
|
for (int i = 0; i < maxTokens; ++i) {
|
||||||
|
llama_token id = llama_sampler_sample(smpl, d->ctx, -1);
|
||||||
|
if (llama_vocab_is_eog(d->vocab, id)) break;
|
||||||
|
|
||||||
|
char buf[256];
|
||||||
|
const int32_t n = llama_token_to_piece(d->vocab, id, buf, sizeof(buf), 0, true);
|
||||||
|
if (n > 0) out.append(buf, (size_t) n);
|
||||||
|
++res.newTokens;
|
||||||
|
|
||||||
|
if (!d->decodeTokens(&id, 1, p, true)) { res.error = "生成时解码失败"; break; }
|
||||||
|
++p;
|
||||||
|
}
|
||||||
|
llama_sampler_free(smpl);
|
||||||
|
|
||||||
|
d->nPastTurn = p;
|
||||||
|
++d->turnCount;
|
||||||
|
|
||||||
|
const auto t1 = std::chrono::steady_clock::now();
|
||||||
|
res.encodeSeconds = std::chrono::duration<double>(tEnc - t0).count();
|
||||||
|
res.prefillSeconds = std::chrono::duration<double>(tPre - tEnc).count();
|
||||||
|
res.genSeconds = std::chrono::duration<double>(t1 - tPre).count();
|
||||||
|
res.inferSeconds = std::chrono::duration<double>(t1 - t0).count();
|
||||||
|
res.newText = out;
|
||||||
|
res.ok = res.error.empty();
|
||||||
|
return res;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool Qwen3Asr::prepareStream(std::string * error)
|
bool Qwen3Asr::prepareStream(std::string * error)
|
||||||
@@ -452,6 +604,8 @@ AsrStreamStep Qwen3Asr::streamAppend(const float * pcm, size_t nSamples,
|
|||||||
llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
|
llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
|
||||||
llama_sampler_chain_add(smpl, llama_sampler_init_greedy());
|
llama_sampler_chain_add(smpl, llama_sampler_init_greedy());
|
||||||
|
|
||||||
|
const auto tPre = std::chrono::steady_clock::now();
|
||||||
|
|
||||||
std::string out;
|
std::string out;
|
||||||
for (int i = 0; i < maxTokens; ++i) {
|
for (int i = 0; i < maxTokens; ++i) {
|
||||||
llama_token id = llama_sampler_sample(smpl, d->ctx, -1);
|
llama_token id = llama_sampler_sample(smpl, d->ctx, -1);
|
||||||
@@ -469,6 +623,8 @@ AsrStreamStep Qwen3Asr::streamAppend(const float * pcm, size_t nSamples,
|
|||||||
|
|
||||||
const auto t1 = std::chrono::steady_clock::now();
|
const auto t1 = std::chrono::steady_clock::now();
|
||||||
res.encodeSeconds = std::chrono::duration<double>(tEnc - t0).count();
|
res.encodeSeconds = std::chrono::duration<double>(tEnc - t0).count();
|
||||||
|
res.prefillSeconds = std::chrono::duration<double>(tPre - tEnc).count();
|
||||||
|
res.genSeconds = std::chrono::duration<double>(t1 - tPre).count();
|
||||||
res.inferSeconds = std::chrono::duration<double>(t1 - t0).count();
|
res.inferSeconds = std::chrono::duration<double>(t1 - t0).count();
|
||||||
res.newText = out;
|
res.newText = out;
|
||||||
res.ok = res.error.empty();
|
res.ok = res.error.empty();
|
||||||
|
|||||||
@@ -38,12 +38,14 @@ public:
|
|||||||
|
|
||||||
AsrStreamStep streamAppend(const float * pcm, size_t nSamples,
|
AsrStreamStep streamAppend(const float * pcm, size_t nSamples,
|
||||||
const std::string & prefix, int maxTokens) override;
|
const std::string & prefix, int maxTokens) override;
|
||||||
|
AsrStreamStep streamAppendTurn(const float * pcm, size_t nSamples, int maxTokens) override;
|
||||||
void streamReset() override;
|
void streamReset() override;
|
||||||
|
|
||||||
std::string dropLastTokens(const std::string & text, int dropLast) const override;
|
std::string dropLastTokens(const std::string & text, int dropLast) const override;
|
||||||
|
|
||||||
private:
|
private:
|
||||||
bool prepareStream(std::string * error);
|
bool prepareStream(std::string * error);
|
||||||
|
bool prepareTurn(std::string * error);
|
||||||
|
|
||||||
struct Impl;
|
struct Impl;
|
||||||
Impl * d = nullptr;
|
Impl * d = nullptr;
|
||||||
|
|||||||
@@ -94,15 +94,33 @@ void StreamingAsr::runStep(const std::vector<float> & audio, bool finalStep,
|
|||||||
s.chunkIndex = chunkIndex_;
|
s.chunkIndex = chunkIndex_;
|
||||||
|
|
||||||
std::string newText;
|
std::string newText;
|
||||||
if (opt_.incremental) {
|
if (opt_.appendOnly) {
|
||||||
|
// 追加式:这一块作为一个新的对话轮,文本随即定稿,不做前缀回滚
|
||||||
|
AsrStreamStep r = asr_.streamAppendTurn(audio.data(), audio.size(),
|
||||||
|
opt_.maxTokensPerStep);
|
||||||
|
s.ok = r.ok;
|
||||||
|
s.error = r.error;
|
||||||
|
s.encodeSeconds = r.encodeSeconds;
|
||||||
|
s.prefillSeconds = r.prefillSeconds;
|
||||||
|
s.genSeconds = r.genSeconds;
|
||||||
|
s.inferSeconds = r.inferSeconds;
|
||||||
|
s.audioTokens = r.audioTokens;
|
||||||
|
s.newTokens = r.newTokens;
|
||||||
|
newText = r.newText;
|
||||||
|
totalSeconds_ += (double) audio.size() / asr_.sampleRate();
|
||||||
|
s.windowSeconds = totalSeconds_;
|
||||||
|
} else if (opt_.incremental) {
|
||||||
// 只把这一块交给增量引擎;音频总长不影响单步成本
|
// 只把这一块交给增量引擎;音频总长不影响单步成本
|
||||||
AsrStreamStep r =
|
AsrStreamStep r =
|
||||||
asr_.streamAppend(audio.data(), audio.size(), prefix, opt_.maxTokensPerStep);
|
asr_.streamAppend(audio.data(), audio.size(), prefix, opt_.maxTokensPerStep);
|
||||||
s.ok = r.ok;
|
s.ok = r.ok;
|
||||||
s.error = r.error;
|
s.error = r.error;
|
||||||
s.encodeSeconds = r.encodeSeconds;
|
s.encodeSeconds = r.encodeSeconds;
|
||||||
|
s.prefillSeconds = r.prefillSeconds;
|
||||||
|
s.genSeconds = r.genSeconds;
|
||||||
s.inferSeconds = r.inferSeconds;
|
s.inferSeconds = r.inferSeconds;
|
||||||
s.audioTokens = r.audioTokens;
|
s.audioTokens = r.audioTokens;
|
||||||
|
s.newTokens = r.newTokens;
|
||||||
newText = r.newText;
|
newText = r.newText;
|
||||||
totalSeconds_ += (double) audio.size() / asr_.sampleRate();
|
totalSeconds_ += (double) audio.size() / asr_.sampleRate();
|
||||||
s.windowSeconds = totalSeconds_;
|
s.windowSeconds = totalSeconds_;
|
||||||
@@ -126,6 +144,23 @@ void StreamingAsr::runStep(const std::vector<float> & audio, bool finalStep,
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (s.ok) {
|
if (s.ok) {
|
||||||
|
if (opt_.appendOnly) {
|
||||||
|
// 追加式:每轮文本直接定稿,追加进历史,没有"不稳定尾部"
|
||||||
|
const ParsedTranscript pTurn = parseAsrOutput(newText);
|
||||||
|
if (!pTurn.language.empty()) language_ = pTurn.language;
|
||||||
|
historyText_ += pTurn.text;
|
||||||
|
s.rawFull = newText;
|
||||||
|
s.full = historyText_;
|
||||||
|
s.committed = historyText_;
|
||||||
|
s.tail.clear();
|
||||||
|
s.language = language_;
|
||||||
|
lastRawFull_.clear();
|
||||||
|
committedRaw_.clear();
|
||||||
|
++chunkIndex_;
|
||||||
|
if (onStep) onStep(s);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// 拼接要用未经 trim 的文本:流式续写时开头那个空格是有意义的
|
// 拼接要用未经 trim 的文本:流式续写时开头那个空格是有意义的
|
||||||
s.rawFull = prefix + newText;
|
s.rawFull = prefix + newText;
|
||||||
lastRawFull_ = s.rawFull;
|
lastRawFull_ = s.rawFull;
|
||||||
|
|||||||
@@ -21,6 +21,11 @@ struct StreamOptions {
|
|||||||
// false = 每次重编码整个窗口(Phase 2 早期的做法,仅用于对比)
|
// false = 每次重编码整个窗口(Phase 2 早期的做法,仅用于对比)
|
||||||
bool incremental = true;
|
bool incremental = true;
|
||||||
|
|
||||||
|
// true = 追加式轮次(append-only):每块音频作为新一轮追加,前缀完全不重算、
|
||||||
|
// 旧音频不重编码,单步成本与总长无关。代价是每块只用自己的长度编码
|
||||||
|
// (没有 8s 上下文),准确率需实测确认。
|
||||||
|
bool appendOnly = false;
|
||||||
|
|
||||||
// 单段最长时长(秒):超过就提交并重置上下文(0 = 不限)。
|
// 单段最长时长(秒):超过就提交并重置上下文(0 = 不限)。
|
||||||
// 真实场景应由静音检测决定分段;这里是一个兜底上界,
|
// 真实场景应由静音检测决定分段;这里是一个兜底上界,
|
||||||
// 因为解码器对 KV 的注意力随总长增长,段太长单步会超预算。
|
// 因为解码器对 KV 的注意力随总长增长,段太长单步会超预算。
|
||||||
@@ -42,8 +47,11 @@ public:
|
|||||||
std::string rawFull; // 未 trim 的完整文本(含 language 头)
|
std::string rawFull; // 未 trim 的完整文本(含 language 头)
|
||||||
double windowSeconds = 0.0;
|
double windowSeconds = 0.0;
|
||||||
double encodeSeconds = 0.0;
|
double encodeSeconds = 0.0;
|
||||||
|
double prefillSeconds = 0.0;
|
||||||
|
double genSeconds = 0.0;
|
||||||
double inferSeconds = 0.0;
|
double inferSeconds = 0.0;
|
||||||
int audioTokens = 0;
|
int audioTokens = 0;
|
||||||
|
int newTokens = 0;
|
||||||
bool ok = true;
|
bool ok = true;
|
||||||
std::string error;
|
std::string error;
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ public:
|
|||||||
|
|
||||||
// 记录(供断言用)
|
// 记录(供断言用)
|
||||||
std::vector<std::string> prefixesSeen;
|
std::vector<std::string> prefixesSeen;
|
||||||
|
std::vector<std::string> turnsSeen;
|
||||||
std::vector<size_t> samplesSeen;
|
std::vector<size_t> samplesSeen;
|
||||||
int resets = 0;
|
int resets = 0;
|
||||||
|
|
||||||
@@ -52,6 +53,20 @@ public:
|
|||||||
return s;
|
return s;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
lc::AsrStreamStep streamAppendTurn(const float *, size_t nSamples, int) override
|
||||||
|
{
|
||||||
|
samplesSeen.push_back(nSamples);
|
||||||
|
|
||||||
|
lc::AsrStreamStep s;
|
||||||
|
s.ok = true;
|
||||||
|
if (call_ < scripted.size()) s.newText = scripted[call_];
|
||||||
|
++call_;
|
||||||
|
turnsSeen.push_back(s.newText);
|
||||||
|
s.newTokens = 1;
|
||||||
|
s.inferSeconds = 0.001;
|
||||||
|
return s;
|
||||||
|
}
|
||||||
|
|
||||||
void streamReset() override { ++resets; }
|
void streamReset() override { ++resets; }
|
||||||
|
|
||||||
std::string dropLastTokens(const std::string & text, int dropLast) const override
|
std::string dropLastTokens(const std::string & text, int dropLast) const override
|
||||||
|
|||||||
@@ -150,6 +150,33 @@ private slots:
|
|||||||
QVERIFY2(all.contains("hello there world"), qPrintable(all));
|
QVERIFY2(all.contains("hello there world"), qPrintable(all));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 追加式:每块音频作为新一轮追加,文本立即定稿,不回喂任何前缀
|
||||||
|
void appendModeCommitsEachChunkImmediately()
|
||||||
|
{
|
||||||
|
FakeAsr fake;
|
||||||
|
fake.scripted = {"language English<asr_text>one two",
|
||||||
|
" language English<asr_text> three four"};
|
||||||
|
|
||||||
|
lc::StreamOptions o = baseOptions();
|
||||||
|
o.appendOnly = true;
|
||||||
|
|
||||||
|
lc::StreamingAsr stream(fake, o);
|
||||||
|
std::vector<lc::StreamingAsr::Step> steps;
|
||||||
|
const auto cb = [&steps](const lc::StreamingAsr::Step & s) { steps.push_back(s); };
|
||||||
|
|
||||||
|
const auto c = chunk(16000);
|
||||||
|
stream.feed(c.data(), c.size(), cb);
|
||||||
|
stream.feed(c.data(), c.size(), cb);
|
||||||
|
|
||||||
|
QCOMPARE(fake.turnsSeen.size(), size_t(2));
|
||||||
|
QVERIFY2(fake.prefixesSeen.empty(), "追加式不应该回喂前缀");
|
||||||
|
|
||||||
|
QCOMPARE(steps.size(), size_t(2));
|
||||||
|
QCOMPARE(q(steps[0].committed), QString("one two"));
|
||||||
|
QVERIFY(steps[0].tail.empty()); // 追加式没有"不稳定尾部"
|
||||||
|
QVERIFY(q(steps[1].committed).contains("three four"));
|
||||||
|
}
|
||||||
|
|
||||||
// 同一次 feed 里跨多块也要正确切分
|
// 同一次 feed 里跨多块也要正确切分
|
||||||
void splitsMultipleChunksInOneFeed()
|
void splitsMultipleChunksInOneFeed()
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in new issue
Block a user