feat(ui): add floating caption window with live transcription
Add a frameless always-on-top caption window wired to the live capture pipeline. Silence-based segmentation gates the audio so the model never sees silence, and the panel background is painted in paintEvent with a configurable opacity instead of a style sheet. 新增悬浮字幕窗（无边框、置顶、可折叠/展开），接入实时采集与流式转写。 分段由静音检测驱动，静音期间不送音频，避免模型对静音产生幻觉。底板 改为在 paintEvent 中绘制，不透明度可调。
This commit is contained in:
blumia committed
1 parent
21d68e2294
commit
4ed4e23282
14 files changed
+803
-259
No files matched your search
+6
-1
@@ -25,10 +25,12 @@ foreach(_p
|
||||
endif()
|
||||
endforeach()
|
||||
|
||||
find_package(Qt6 REQUIRED COMPONENTS Core)
|
||||
find_package(Qt6 REQUIRED COMPONENTS Core Widgets)
|
||||
find_package(PkgConfig REQUIRED)
|
||||
find_package(Threads REQUIRED)
|
||||
|
||||
set(CMAKE_AUTOMOC ON)
|
||||
|
||||
pkg_check_modules(SAMPLERATE REQUIRED IMPORTED_TARGET samplerate)
|
||||
pkg_check_modules(PULSE REQUIRED IMPORTED_TARGET libpulse)
|
||||
|
||||
@@ -67,12 +69,15 @@ add_executable(live-caption-cli
|
||||
src/qwen3asr.cpp
|
||||
src/streamingasr.cpp
|
||||
src/audio.cpp
|
||||
src/livetranscriber.cpp
|
||||
src/captionwindow.cpp
|
||||
)
|
||||
|
||||
target_include_directories(live-caption-cli PRIVATE src)
|
||||
|
||||
target_link_libraries(live-caption-cli PRIVATE
|
||||
Qt6::Core
|
||||
Qt6::Widgets
|
||||
Threads::Threads
|
||||
PkgConfig::SAMPLERATE
|
||||
PkgConfig::PULSE
|
||||
|
||||
@@ -10,7 +10,7 @@ Linux 桌面实时字幕：采集系统正在播放的声音，实时转写成
|
||||
| Phase 0 | 可行性验证（采集 / 模型 / CPU 实时性） | ✅ 见 [docs/phase0-verification.md](docs/phase0-verification.md) |
|
||||
| Phase 1 | 命令行原型（进程内推理 + monitor 采集 + 滑窗 + 稳定前缀） | ✅ 见 [experiments/phase1/README.md](experiments/phase1/README.md) |
|
||||
| Phase 2 | 原生流式解码（8s 切片 + 增量 KV 复用 + 前缀回滚） | ✅ 见 [docs/phase2-streaming.md](docs/phase2-streaming.md) |
|
||||
| Phase 3 | Qt 悬浮字幕窗（折叠/展开、语言标签、置顶） | 待做 |
|
||||
| Phase 3 | Qt 悬浮字幕窗（折叠/展开、语言标签、置顶、实时链路） | ✅ 见 [docs/phase3-caption-window.md](docs/phase3-caption-window.md) |
|
||||
| Phase 4 | 设置界面（模型切换、语言、字体、位置） | 待做 |
|
||||
| Phase 5 | Wayland 支持、打包、模型分发 | 待做 |
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
@@ -0,0 +1,77 @@
|
||||
# Phase 3：Qt 悬浮字幕窗
|
||||
|
||||
状态：**完成**。悬浮窗可用，已接入实时采集 + 流式转写。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 已实现
|
||||
|
||||
- **悬浮窗**：无边框、半透明深色圆角、始终置顶、不接受焦点（`Qt::FramelessWindowHint | WindowStaysOnTopHint | Tool | WindowDoesNotAcceptFocus`），可拖动。
|
||||
- **折叠 / 展开**：折叠两行、展开八行，底部箭头切换（折叠态 `⌄`、展开态 `⌃`）。
|
||||
- **语言标签**：左上角圆角 pill，显示识别到的语言（英语 / 中文 …）；点击会发 `languageClicked` 信号，设置界面留到 Phase 4。
|
||||
- **稳定/不稳定区分**：已稳定文本白色，不稳定尾部灰色 —— 与 CLI 版一致，也是 Chrome 那种"字长出来"的观感。
|
||||
- **背景**：**黑色 + 可调不透明度**（不是全透明；纯黑底才能保证任何桌面背景下文字都可读）。默认 84%，范围限制 20~100，`--opacity` 或 `setPanelOpacity()` 可调，Phase 4 接到设置里即可。
|
||||
- **关闭**：右上角 `✕` 退出。
|
||||
- **实时链路**：采集默认输出的 monitor → 流式转写 → 信号推给窗口，全部在主线程外的工作线程里跑。
|
||||
|
||||
## 实时分段
|
||||
|
||||
Phase 2 的结论是段不能无限长（解码器对 KV 的注意力随总长增长）。Phase 3 用**静音检测**决定段边界：
|
||||
|
||||
- 能量 VAD（噪声底用"快降慢升"的最小值跟踪，见 Phase 1 的教训）；
|
||||
- 检测到语音才开始把音频送进引擎，并保留 250ms 前导避免吃掉词头；
|
||||
- **静音期间完全不送音频**：早期版本会把静音也喂进模型，模型对静音产生了幻觉（英文语音前冒出一个 `嗯。`）；
|
||||
- 静音超过 0.8s 就收束这一段（提交文本、清空 KV），下一段重新开始。
|
||||
|
||||
`--stream-segment`（默认 30s）仍作为兜底上界。
|
||||
|
||||
## 代码结构
|
||||
|
||||
```
|
||||
src/
|
||||
├─ captionwindow.{h,cpp} # 悬浮窗
|
||||
├─ livetranscriber.{h,cpp} # 采集 + VAD + 分段 + 流式引擎（QObject）
|
||||
├─ streamingasr.{h,cpp} # 流式策略（前缀回滚、8s 切片、分段）
|
||||
├─ qwen3asr.{h,cpp} # 进程内推理（libllama + libmtmd）
|
||||
├─ audio.{h,cpp} # WAV 读取 + monitor 采集 + 重采样
|
||||
└─ transcript.{h,cpp} # 输出解析与语言名映射
|
||||
```
|
||||
|
||||
## 过程中踩的坑
|
||||
|
||||
1. **静音幻觉**：第一版把采集到的静音也喂给了模型，模型对静音产生幻觉，英文语音前面冒出一个 `嗯。`。改为静音期间完全不送音频。
|
||||
2. **自定义 QWidget 子类的样式表背景不生效**：`CaptionWindow { background-color: rgba(0,0,0,215) }` 这条规则在真实渲染时**根本没画**，面板全透明，只有文字和按钮浮在桌面上。原因是 Qt 不会为自定义 `QWidget` 子类绘制样式表背景（需要 `WA_StyledBackground`，或自己画）。改成在 `paintEvent()` 里用 `QPainter::drawRoundedRect` 画底板 —— 这也是 `WA_TranslucentBackground` 顶层窗口的推荐做法。
|
||||
3. **验证方法骗了自己**：`QWidget::grab()` 的产物当时是合成到**深灰底色**上看的，「黑色 + 84%」和「全透明」在深色背景上几乎看不出区别，于是把第 2 条盖住了。改成**直接检查像素 alpha**（面板中央应为 `(0,0,0,≈214)`、圆角外应为 `(0,0,0,0)`），并合成到亮色背景上复核。
|
||||
4. **DPI 警告不关本程序的事**：`setHighDpiScaleFactorRoundingPolicy must be called before creating the QGuiApplication instance` 连 5 行的空白 Qt 程序也会打印 —— 是 Deepin 会话环境设了 `QT_SCALE_FACTOR_ROUNDING_POLICY` 导致的。
|
||||
|
||||
## 用法
|
||||
|
||||
```bash
|
||||
# 默认就是悬浮窗模式
|
||||
./build/live-caption-cli --model MODEL --mmproj MMPROJ
|
||||
|
||||
# 调整背景不透明度（20-100）
|
||||
./build/live-caption-cli --model MODEL --mmproj MMPROJ --opacity 60
|
||||
|
||||
# 无窗口、纯命令行实时（便于在无显示环境调试）
|
||||
./build/live-caption-cli --model MODEL --mmproj MMPROJ --cli-live
|
||||
|
||||
# 自截图（验证渲染，不依赖外部截图工具）；会依次截折叠态与展开态
|
||||
./build/live-caption-cli --model MODEL --mmproj MMPROJ \
|
||||
--screenshot out.png --screenshot-delay 12
|
||||
```
|
||||
|
||||
> 自截图产物带 alpha 通道；要判断不透明度是否正确，**直接看像素 alpha**，不要靠肉眼看合成效果。
|
||||
|
||||
## 已知限制 / 后续
|
||||
|
||||
- X11 下置顶与定位都正常；Wayland 下需要 layer-shell，属于 Phase 5。
|
||||
- 语言标签点击暂无设置界面（Phase 4）。
|
||||
- 采集跟随的是**启动时**的默认输出设备；设备切换（例如切到 HDMI）暂未处理。
|
||||
- 界面尚未做 HiDPI 之外的适配（当前主屏宽度 72%、最大 900 逻辑像素）。
|
||||
@@ -0,0 +1,238 @@
|
||||
#include "captionwindow.h"
|
||||
|
||||
#include <QApplication>
|
||||
#include <QFontMetrics>
|
||||
#include <QHBoxLayout>
|
||||
#include <QMouseEvent>
|
||||
#include <QPainter>
|
||||
#include <QPushButton>
|
||||
#include <QScreen>
|
||||
#include <QTextCursor>
|
||||
#include <QTextEdit>
|
||||
#include <QVBoxLayout>
|
||||
|
||||
namespace lc {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kCollapsedLines = 2;
|
||||
constexpr int kExpandedLines = 8;
|
||||
constexpr int kMargin = 12;
|
||||
constexpr int kRadius = 10;
|
||||
|
||||
QString escapeHtml(const QString & s)
|
||||
{
|
||||
return s.toHtmlEscaped().replace(QLatin1Char('\n'), QLatin1String("<br/>"));
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
CaptionWindow::CaptionWindow(QWidget * parent)
|
||||
: QWidget(parent)
|
||||
{
|
||||
// 无边框 + 置顶 + 不抢焦点（X11 下都有对应实现）
|
||||
setWindowFlags(Qt::FramelessWindowHint | Qt::WindowStaysOnTopHint | Qt::Tool
|
||||
| Qt::WindowDoesNotAcceptFocus);
|
||||
setAttribute(Qt::WA_TranslucentBackground);
|
||||
setAttribute(Qt::WA_ShowWithoutActivating);
|
||||
setWindowTitle(QStringLiteral("实时字幕"));
|
||||
|
||||
auto * root = new QVBoxLayout(this);
|
||||
root->setContentsMargins(kMargin, kMargin, kMargin, kMargin);
|
||||
root->setSpacing(6);
|
||||
|
||||
// ---- 顶部：语言标签 + 关闭 ----
|
||||
auto * top = new QHBoxLayout();
|
||||
top->setSpacing(6);
|
||||
|
||||
langPill_ = new QPushButton(this);
|
||||
langPill_->setCursor(Qt::PointingHandCursor);
|
||||
langPill_->setFocusPolicy(Qt::NoFocus);
|
||||
langPill_->setToolTip(tr("识别语言（设置界面将在后续阶段接入）"));
|
||||
connect(langPill_, &QPushButton::clicked, this, &CaptionWindow::languageClicked);
|
||||
|
||||
closeBtn_ = new QPushButton(QStringLiteral("\u2715"), this); // ✕
|
||||
closeBtn_->setCursor(Qt::PointingHandCursor);
|
||||
closeBtn_->setFocusPolicy(Qt::NoFocus);
|
||||
closeBtn_->setFixedSize(22, 22);
|
||||
connect(closeBtn_, &QPushButton::clicked, this, &CaptionWindow::closed);
|
||||
|
||||
top->addWidget(langPill_);
|
||||
top->addStretch(1);
|
||||
top->addWidget(closeBtn_);
|
||||
root->addLayout(top);
|
||||
|
||||
// ---- 中间：字幕文本 ----
|
||||
text_ = new QTextEdit(this);
|
||||
text_->setReadOnly(true);
|
||||
text_->setFrameShape(QFrame::NoFrame);
|
||||
text_->setVerticalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
|
||||
text_->setHorizontalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
|
||||
text_->setTextInteractionFlags(Qt::NoTextInteraction); // 让拖动可以穿透到窗口
|
||||
text_->setFocusPolicy(Qt::NoFocus);
|
||||
text_->viewport()->setAutoFillBackground(false);
|
||||
root->addWidget(text_, 1);
|
||||
|
||||
// ---- 底部：展开/收起 ----
|
||||
toggleBtn_ = new QPushButton(this);
|
||||
toggleBtn_->setCursor(Qt::PointingHandCursor);
|
||||
toggleBtn_->setFocusPolicy(Qt::NoFocus);
|
||||
connect(toggleBtn_, &QPushButton::clicked, this, [this] { setCollapsed(!collapsed_); });
|
||||
|
||||
auto * bottom = new QHBoxLayout();
|
||||
bottom->addStretch(1);
|
||||
bottom->addWidget(toggleBtn_);
|
||||
bottom->addStretch(1);
|
||||
root->addLayout(bottom);
|
||||
|
||||
setStyleSheet(QString()); // 实际样式由 applyStyle() 生成（透明度可调）
|
||||
applyStyle();
|
||||
|
||||
langPill_->setObjectName(QStringLiteral("langPill"));
|
||||
toggleBtn_->setObjectName(QStringLiteral("toggleBtn"));
|
||||
|
||||
relayout();
|
||||
setLanguage(QString());
|
||||
setCaption(QString(), QString());
|
||||
|
||||
// 默认停在主屏下方居中
|
||||
if (QScreen * screen = QApplication::primaryScreen()) {
|
||||
const QRect avail = screen->availableGeometry();
|
||||
const int w = qMin(900, (int) (avail.width() * 0.72));
|
||||
resize(w, height());
|
||||
move(avail.center().x() - w / 2, avail.bottom() - height() - 48);
|
||||
}
|
||||
}
|
||||
|
||||
void CaptionWindow::applyStyle()
|
||||
{
|
||||
// 注意：这里不设 CaptionWindow 自己的 background。
|
||||
// 自定义 QWidget 子类默认不绘制样式表背景，而且带 WA_TranslucentBackground 的
|
||||
// 顶层窗口本来就该自己画底板 —— 见 paintEvent()。
|
||||
setStyleSheet(QStringLiteral(R"(
|
||||
QPushButton {
|
||||
color: #f0f0f0;
|
||||
background-color: transparent;
|
||||
border: none;
|
||||
padding: 2px 10px;
|
||||
font-size: 12px;
|
||||
}
|
||||
QPushButton:hover { background-color: rgba(255,255,255,40); border-radius: 9px; }
|
||||
#langPill {
|
||||
border: 1px solid rgba(255,255,255,140);
|
||||
border-radius: 10px;
|
||||
padding: 1px 10px;
|
||||
}
|
||||
#toggleBtn { color: #bbbbbb; font-size: 15px; padding: 0px; }
|
||||
QTextEdit {
|
||||
color: #ffffff;
|
||||
background: transparent;
|
||||
font-size: 15px;
|
||||
}
|
||||
)"));
|
||||
}
|
||||
|
||||
void CaptionWindow::paintEvent(QPaintEvent *)
|
||||
{
|
||||
// 背景是"黑色 + 一定不透明度"，不是全透明：这样才能保证任何桌面背景下文字都可读。
|
||||
const int alpha = qBound(20, opacityPercent_, 100) * 255 / 100;
|
||||
|
||||
QPainter p(this);
|
||||
p.setRenderHint(QPainter::Antialiasing, true);
|
||||
p.setPen(Qt::NoPen);
|
||||
p.setBrush(QColor(0, 0, 0, alpha));
|
||||
p.drawRoundedRect(rect(), kRadius, kRadius);
|
||||
}
|
||||
|
||||
void CaptionWindow::setPanelOpacity(int percent)
|
||||
{
|
||||
const int clamped = qBound(20, percent, 100);
|
||||
if (clamped == opacityPercent_) return;
|
||||
opacityPercent_ = clamped;
|
||||
applyStyle();
|
||||
}
|
||||
|
||||
int CaptionWindow::lineHeight() const
|
||||
{
|
||||
return QFontMetrics(text_->font()).lineSpacing();
|
||||
}
|
||||
|
||||
int CaptionWindow::textHeight() const
|
||||
{
|
||||
return lineHeight() * (collapsed_ ? kCollapsedLines : kExpandedLines)
|
||||
+ (int) text_->document()->documentMargin() * 2;
|
||||
}
|
||||
|
||||
void CaptionWindow::relayout()
|
||||
{
|
||||
text_->setFixedHeight(textHeight());
|
||||
// 折叠态显示向下的箭头（点击展开），展开态显示向上的箭头（点击收起）
|
||||
toggleBtn_->setText(collapsed_ ? QStringLiteral("\u2304") // ⌄
|
||||
: QStringLiteral("\u2303")); // ⌃
|
||||
toggleBtn_->setToolTip(collapsed_ ? tr("展开") : tr("收起"));
|
||||
const int h = kMargin * 2 + 22 + 6 + textHeight() + 6 + 24;
|
||||
setFixedHeight(h);
|
||||
setMinimumWidth(320);
|
||||
scrollToEnd();
|
||||
}
|
||||
|
||||
void CaptionWindow::scrollToEnd()
|
||||
{
|
||||
QTextCursor c = text_->textCursor();
|
||||
c.movePosition(QTextCursor::End);
|
||||
text_->setTextCursor(c);
|
||||
text_->ensureCursorVisible();
|
||||
}
|
||||
|
||||
void CaptionWindow::setCollapsed(bool collapsed)
|
||||
{
|
||||
if (collapsed_ == collapsed) return;
|
||||
collapsed_ = collapsed;
|
||||
relayout();
|
||||
}
|
||||
|
||||
void CaptionWindow::setLanguage(const QString & language)
|
||||
{
|
||||
language_ = language;
|
||||
langPill_->setText(language.isEmpty() ? tr("自动") : language);
|
||||
langPill_->adjustSize();
|
||||
}
|
||||
|
||||
void CaptionWindow::setCaption(const QString & committed, const QString & tail)
|
||||
{
|
||||
if (committed == committed_ && tail == tail_) return;
|
||||
committed_ = committed;
|
||||
tail_ = tail;
|
||||
|
||||
// 已稳定部分正常色，不稳定尾部暗色 —— 和 CLI 版渲染一致
|
||||
const QString html =
|
||||
QStringLiteral("<div>%1<span style=\"color:#9a9a9a\">%2</span></div>")
|
||||
.arg(escapeHtml(committed_), escapeHtml(tail_));
|
||||
text_->setHtml(html);
|
||||
scrollToEnd();
|
||||
}
|
||||
|
||||
void CaptionWindow::mousePressEvent(QMouseEvent * e)
|
||||
{
|
||||
if (e->button() == Qt::LeftButton) {
|
||||
dragging_ = true;
|
||||
dragOffset_ = e->globalPosition().toPoint() - frameGeometry().topLeft();
|
||||
e->accept();
|
||||
}
|
||||
}
|
||||
|
||||
void CaptionWindow::mouseMoveEvent(QMouseEvent * e)
|
||||
{
|
||||
if (dragging_ && (e->buttons() & Qt::LeftButton)) {
|
||||
move(e->globalPosition().toPoint() - dragOffset_);
|
||||
e->accept();
|
||||
}
|
||||
}
|
||||
|
||||
void CaptionWindow::mouseReleaseEvent(QMouseEvent * e)
|
||||
{
|
||||
dragging_ = false;
|
||||
e->accept();
|
||||
}
|
||||
|
||||
} // namespace lc
|
||||
@@ -0,0 +1,63 @@
|
||||
#pragma once
|
||||
|
||||
#include <QPoint>
|
||||
#include <QWidget>
|
||||
|
||||
class QLabel;
|
||||
class QPushButton;
|
||||
class QTextEdit;
|
||||
|
||||
namespace lc {
|
||||
|
||||
// 悬浮字幕窗（对应 Chrome 实时字幕那种样式）：
|
||||
// 无边框、置顶、不接受焦点；折叠时两行，展开后多行。
|
||||
class CaptionWindow : public QWidget {
|
||||
Q_OBJECT
|
||||
|
||||
public:
|
||||
explicit CaptionWindow(QWidget * parent = nullptr);
|
||||
|
||||
// committed 为已稳定文本，tail 为不稳定尾部（暗色显示）
|
||||
void setCaption(const QString & committed, const QString & tail);
|
||||
void setLanguage(const QString & language);
|
||||
|
||||
bool isCollapsed() const { return collapsed_; }
|
||||
void setCollapsed(bool collapsed);
|
||||
|
||||
// 背景不透明度（百分比）。范围限制在 20~100：
|
||||
// 太低会让字幕在任何桌面背景下都读不清，所以不给到全透明。
|
||||
void setPanelOpacity(int percent);
|
||||
int panelOpacity() const { return opacityPercent_; }
|
||||
|
||||
Q_SIGNALS:
|
||||
void languageClicked();
|
||||
void closed();
|
||||
|
||||
protected:
|
||||
void mousePressEvent(QMouseEvent * e) override;
|
||||
void mouseMoveEvent(QMouseEvent * e) override;
|
||||
void mouseReleaseEvent(QMouseEvent * e) override;
|
||||
void paintEvent(QPaintEvent * e) override;
|
||||
|
||||
private:
|
||||
int lineHeight() const;
|
||||
int textHeight() const; // 按折叠/展开计算文本区高度
|
||||
void relayout();
|
||||
void scrollToEnd();
|
||||
void applyStyle();
|
||||
|
||||
QPushButton * langPill_ = nullptr;
|
||||
QPushButton * closeBtn_ = nullptr;
|
||||
QPushButton * toggleBtn_ = nullptr;
|
||||
QTextEdit * text_ = nullptr;
|
||||
|
||||
bool collapsed_ = true;
|
||||
int opacityPercent_ = 84; // 背景不透明度（百分比）
|
||||
bool dragging_ = false;
|
||||
QPoint dragOffset_;
|
||||
QString language_;
|
||||
QString committed_;
|
||||
QString tail_;
|
||||
};
|
||||
|
||||
} // namespace lc
|
||||
@@ -0,0 +1,178 @@
|
||||
#include "livetranscriber.h"
|
||||
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
|
||||
#include "qwen3asr.h"
|
||||
|
||||
namespace lc {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kFrameSamples = 320; // 20ms @16k
|
||||
constexpr float kSilenceTimeout = 0.8f; // 静音多久算一句结束
|
||||
constexpr float kPrerollSec = 0.25f; // 语音开始前保留的一段，避免吃掉词头
|
||||
|
||||
// 能量 VAD。噪声底用"快降慢升"的最小值跟踪（不能用均值，否则轻音频会被全判成静音）。
|
||||
class Vad {
|
||||
public:
|
||||
bool feed(const float * frame, size_t n)
|
||||
{
|
||||
double acc = 0.0;
|
||||
for (size_t i = 0; i < n; ++i) acc += (double) frame[i] * frame[i];
|
||||
const float rms = std::sqrt(acc / (double) n);
|
||||
|
||||
if (rms < floor_) floor_ = rms;
|
||||
else floor_ = floor_ * 0.9999f + rms * 0.0001f;
|
||||
|
||||
return rms > std::max(floor_ * 3.0f, 0.0015f);
|
||||
}
|
||||
|
||||
private:
|
||||
float floor_ = 0.01f;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
LiveTranscriber::LiveTranscriber(Qwen3Asr & asr, QObject * parent)
|
||||
: QObject(parent), asr_(asr) {}
|
||||
|
||||
LiveTranscriber::~LiveTranscriber()
|
||||
{
|
||||
stop();
|
||||
}
|
||||
|
||||
QString LiveTranscriber::deviceName() const
|
||||
{
|
||||
return QString::fromStdString(capture_.deviceName());
|
||||
}
|
||||
|
||||
bool LiveTranscriber::start(const StreamOptions & opts, QString * error)
|
||||
{
|
||||
opts_ = opts;
|
||||
|
||||
std::string err;
|
||||
const bool ok = capture_.start(asr_.sampleRate(),
|
||||
[this](const float * s, size_t n) {
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
pending_.insert(pending_.end(), s, s + n);
|
||||
}
|
||||
cv_.notify_one();
|
||||
},
|
||||
&err);
|
||||
if (!ok) {
|
||||
if (error) *error = QString::fromStdString(err);
|
||||
return false;
|
||||
}
|
||||
|
||||
worker_ = std::thread([this] { workerLoop(); });
|
||||
return true;
|
||||
}
|
||||
|
||||
void LiveTranscriber::stop()
|
||||
{
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
quit_ = true;
|
||||
}
|
||||
cv_.notify_all();
|
||||
if (worker_.joinable()) worker_.join();
|
||||
|
||||
capture_.stop();
|
||||
}
|
||||
|
||||
void LiveTranscriber::workerLoop()
|
||||
{
|
||||
Vad vad;
|
||||
StreamingAsr stream(asr_, opts_);
|
||||
|
||||
float silenceAcc = 0.0f;
|
||||
bool inSegment = false;
|
||||
bool flushNow = false;
|
||||
|
||||
std::vector<float> buffer; // 跨批次的残留样本
|
||||
std::vector<float> preroll; // 语音开始前保留的一小段，避免吃掉词头
|
||||
std::vector<float> segAudio; // 本段待送进引擎的音频
|
||||
|
||||
const size_t prerollSamples = (size_t) (kPrerollSec * asr_.sampleRate());
|
||||
|
||||
const auto emitStep = [this](const StreamingAsr::Step & s) {
|
||||
if (!s.ok) return;
|
||||
Q_EMIT caption(QString::fromStdString(s.committed),
|
||||
QString::fromStdString(s.tail),
|
||||
QString::fromStdString(s.language));
|
||||
};
|
||||
|
||||
const auto feedSegment = [&] {
|
||||
if (!segAudio.empty()) stream.feed(segAudio.data(), segAudio.size(), emitStep);
|
||||
segAudio.clear();
|
||||
};
|
||||
|
||||
for (;;) {
|
||||
std::vector<float> chunk;
|
||||
{
|
||||
std::unique_lock<std::mutex> lk(mutex_);
|
||||
cv_.wait_for(lk, std::chrono::milliseconds(50),
|
||||
[this] { return quit_ || !pending_.empty(); });
|
||||
if (quit_ && pending_.empty()) break;
|
||||
chunk.swap(pending_);
|
||||
}
|
||||
|
||||
// 不足一帧的残留要接上，否则会持续丢样本
|
||||
buffer.insert(buffer.end(), chunk.begin(), chunk.end());
|
||||
|
||||
size_t off = 0;
|
||||
for (; off + kFrameSamples <= buffer.size(); off += kFrameSamples) {
|
||||
const float * frame = buffer.data() + off;
|
||||
const bool speech = vad.feed(frame, kFrameSamples);
|
||||
|
||||
if (!inSegment) {
|
||||
// 静音期间什么都不送给模型：否则模型会对静音产生幻觉
|
||||
if (speech) {
|
||||
inSegment = true;
|
||||
silenceAcc = 0.0f;
|
||||
segAudio.insert(segAudio.end(), preroll.begin(), preroll.end());
|
||||
preroll.clear();
|
||||
segAudio.insert(segAudio.end(), frame, frame + kFrameSamples);
|
||||
} else {
|
||||
preroll.insert(preroll.end(), frame, frame + kFrameSamples);
|
||||
if (preroll.size() > prerollSamples) {
|
||||
preroll.erase(preroll.begin(),
|
||||
preroll.begin() + (long) (preroll.size() - prerollSamples));
|
||||
}
|
||||
}
|
||||
} else {
|
||||
segAudio.insert(segAudio.end(), frame, frame + kFrameSamples);
|
||||
if (speech) silenceAcc = 0.0f;
|
||||
else silenceAcc += (float) kFrameSamples / asr_.sampleRate();
|
||||
if (silenceAcc >= kSilenceTimeout) {
|
||||
flushNow = true;
|
||||
off += kFrameSamples; // 这一帧已经并入 segAudio
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!segAudio.empty()) feedSegment();
|
||||
|
||||
// 静音够久 → 收束这一段（提交文本并重置上下文）
|
||||
if (flushNow) {
|
||||
stream.flush(emitStep);
|
||||
stream.reset();
|
||||
inSegment = false;
|
||||
silenceAcc = 0.0f;
|
||||
flushNow = false;
|
||||
preroll.clear();
|
||||
}
|
||||
|
||||
if (off > 0) buffer.erase(buffer.begin(), buffer.begin() + (long) off);
|
||||
}
|
||||
|
||||
if (inSegment) {
|
||||
feedSegment();
|
||||
stream.flush(emitStep);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace lc
|
||||
@@ -0,0 +1,52 @@
|
||||
#pragma once
|
||||
|
||||
#include <QObject>
|
||||
#include <QString>
|
||||
|
||||
#include <condition_variable>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "audio.h"
|
||||
#include "streamingasr.h"
|
||||
|
||||
namespace lc {
|
||||
|
||||
class Qwen3Asr;
|
||||
|
||||
// 实时字幕：采集默认输出 → 流式转写 → 用信号把文本推给 UI。
|
||||
// 分段由静音检测驱动：句子之间的自然停顿就收束一段、清空上下文，
|
||||
// 这样解码器的上下文不会无限增长（见 docs/phase2-streaming.md）。
|
||||
class LiveTranscriber : public QObject {
|
||||
Q_OBJECT
|
||||
|
||||
public:
|
||||
explicit LiveTranscriber(Qwen3Asr & asr, QObject * parent = nullptr);
|
||||
~LiveTranscriber() override;
|
||||
|
||||
bool start(const StreamOptions & opts, QString * error = nullptr);
|
||||
void stop();
|
||||
|
||||
QString deviceName() const;
|
||||
|
||||
Q_SIGNALS:
|
||||
// committed = 已稳定文本；tail = 不稳定尾部（UI 用暗色显示）
|
||||
void caption(const QString & committed, const QString & tail, const QString & language);
|
||||
void failed(const QString & message);
|
||||
|
||||
private:
|
||||
void workerLoop();
|
||||
|
||||
Qwen3Asr & asr_;
|
||||
MonitorCapture capture_;
|
||||
StreamOptions opts_;
|
||||
std::thread worker_;
|
||||
std::mutex mutex_;
|
||||
std::condition_variable cv_;
|
||||
std::vector<float> pending_;
|
||||
bool quit_ = false;
|
||||
};
|
||||
|
||||
} // namespace lc
|
||||
+151
-249
@@ -1,214 +1,29 @@
|
||||
#include <atomic>
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <condition_variable>
|
||||
#include <cstdio>
|
||||
#include <mutex>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include <QApplication>
|
||||
#include <QCommandLineParser>
|
||||
#include <QCoreApplication>
|
||||
#include <QGuiApplication>
|
||||
#include <QScopedPointer>
|
||||
#include <QTimer>
|
||||
|
||||
#include "audio.h"
|
||||
#include "captionwindow.h"
|
||||
#include "livetranscriber.h"
|
||||
#include "qwen3asr.h"
|
||||
#include "streamingasr.h"
|
||||
#include "transcript.h"
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kFrameSamples = 320; // 20ms @ 16k
|
||||
constexpr float kSilenceTimeout = 0.8f; // 静音多久算一句结束
|
||||
constexpr float kMinSpeech = 0.5f; // 一句话至少多长才值得转写
|
||||
constexpr float kMaxWindow = 20.0f; // 窗口上限，超过强制收尾
|
||||
|
||||
// 两个字符串的最长公共前缀，且保证切在 UTF-8 字符边界上
|
||||
size_t utf8Lcp(const std::string & a, const std::string & b)
|
||||
{
|
||||
const size_t n = std::min(a.size(), b.size());
|
||||
size_t i = 0;
|
||||
while (i < n && a[i] == b[i]) ++i;
|
||||
// 回退到不是 UTF-8 续字节(10xxxxxx)的位置
|
||||
while (i > 0 && (static_cast<unsigned char>(a[i]) & 0xC0) == 0x80) --i;
|
||||
return i;
|
||||
}
|
||||
|
||||
// 简单的能量 VAD。
|
||||
// 噪声底必须用"最小值跟踪"（快降慢升），不能用均值：
|
||||
// 否则遇到本来就很轻的音频（例如 AISHELL 峰值 RMS 只有 0.03），
|
||||
// 噪声底会一路抬升、阈值超过语音电平，把话音全判成静音。
|
||||
class Vad {
|
||||
public:
|
||||
// 返回该帧是否为语音
|
||||
bool feed(const float * frame, size_t n)
|
||||
{
|
||||
double acc = 0.0;
|
||||
for (size_t i = 0; i < n; ++i) acc += (double) frame[i] * frame[i];
|
||||
const float rms = std::sqrt(acc / (double) n);
|
||||
|
||||
if (rms < floor_) {
|
||||
floor_ = rms; // 快速下探
|
||||
} else {
|
||||
floor_ = floor_ * 0.9999f + rms * 0.0001f; // 极慢回升
|
||||
}
|
||||
|
||||
const float threshold = std::max(floor_ * 3.0f, 0.0015f);
|
||||
speech_ = rms > threshold;
|
||||
return speech_;
|
||||
}
|
||||
|
||||
bool speaking() const { return speech_; }
|
||||
|
||||
private:
|
||||
float floor_ = 0.01f;
|
||||
bool speech_ = false;
|
||||
};
|
||||
|
||||
// 实时转写：采集线程 -> 队列 -> 工作线程（VAD + 滑窗 + 转写）
|
||||
class LiveTranscriber {
|
||||
public:
|
||||
LiveTranscriber(lc::Qwen3Asr & asr, int sampleRate) : asr_(asr), sampleRate_(sampleRate) {}
|
||||
|
||||
void onAudio(const float * samples, size_t n)
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
pending_.insert(pending_.end(), samples, samples + n);
|
||||
cv_.notify_one();
|
||||
}
|
||||
|
||||
void start() { worker_ = std::thread([this] { run(); }); }
|
||||
|
||||
void stop()
|
||||
{
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
quit_ = true;
|
||||
}
|
||||
cv_.notify_all();
|
||||
if (worker_.joinable()) worker_.join();
|
||||
}
|
||||
|
||||
private:
|
||||
void emitPartial(const lc::Qwen3Asr::Result & r)
|
||||
{
|
||||
// 只把"跨多次转写都不变"的前缀当成已定文本，尾部标为不稳定预览。
|
||||
// 窗口在词中间被切断时，模型会给尾部编一个结尾，这样能避免它上屏。
|
||||
committed_ = r.transcript.text.substr(0, utf8Lcp(committed_, r.transcript.text));
|
||||
const std::string tail = r.transcript.text.substr(committed_.size());
|
||||
|
||||
std::lock_guard<std::mutex> lk(printMutex_);
|
||||
std::printf("\r\033[K~ [%s] %s\033[2m%s\033[0m",
|
||||
lc::languageDisplayName(r.transcript.language).c_str(),
|
||||
committed_.c_str(), tail.c_str());
|
||||
std::fflush(stdout);
|
||||
}
|
||||
|
||||
void emitFinal(const lc::Qwen3Asr::Result & r)
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(printMutex_);
|
||||
std::printf("\r\033[K= [%s] %s\n",
|
||||
lc::languageDisplayName(r.transcript.language).c_str(),
|
||||
r.transcript.text.c_str());
|
||||
std::fflush(stdout);
|
||||
committed_.clear();
|
||||
}
|
||||
|
||||
lc::Qwen3Asr::Result transcribeWindow(const std::vector<float> & window)
|
||||
{
|
||||
lc::Qwen3Asr::Result r = asr_.transcribe(window.data(), window.size());
|
||||
const double dur = (double) window.size() / sampleRate_;
|
||||
std::lock_guard<std::mutex> lk(printMutex_);
|
||||
std::fprintf(stderr, " [infer] %.2fs 音频 / %.2fs 计算 (rtf %.3f)\n",
|
||||
dur, r.inferSeconds, dur > 0 ? r.inferSeconds / dur : 0.0);
|
||||
return r;
|
||||
}
|
||||
|
||||
void run()
|
||||
{
|
||||
Vad vad;
|
||||
std::vector<float> window; // 当前这句话累积的音频
|
||||
std::vector<float> buffer; // 跨批次残留，保证不丢样本
|
||||
float silenceAcc = 0.0f;
|
||||
float lastPartialAt = 0.0f; // 上次做 partial 时窗口长度（秒）
|
||||
|
||||
for (;;) {
|
||||
std::vector<float> chunk;
|
||||
{
|
||||
std::unique_lock<std::mutex> lk(mutex_);
|
||||
cv_.wait_for(lk, std::chrono::milliseconds(50), [this] { return quit_ || !pending_.empty(); });
|
||||
if (quit_ && pending_.empty()) break;
|
||||
chunk.swap(pending_);
|
||||
}
|
||||
|
||||
// 关键：把上一轮不足一帧的残留接上，否则会持续丢样本
|
||||
buffer.insert(buffer.end(), chunk.begin(), chunk.end());
|
||||
|
||||
size_t off = 0;
|
||||
for (; off + kFrameSamples <= buffer.size(); off += kFrameSamples) {
|
||||
const float * frame = buffer.data() + off;
|
||||
const bool speech = vad.feed(frame, kFrameSamples);
|
||||
|
||||
if (speech) {
|
||||
if (window.empty()) lastPartialAt = 0.0f;
|
||||
window.insert(window.end(), frame, frame + kFrameSamples);
|
||||
silenceAcc = 0.0f;
|
||||
} else if (!window.empty()) {
|
||||
window.insert(window.end(), frame, frame + kFrameSamples);
|
||||
silenceAcc += (float) kFrameSamples / sampleRate_;
|
||||
}
|
||||
|
||||
if (window.empty()) continue;
|
||||
|
||||
const float windowSec = (float) window.size() / sampleRate_;
|
||||
const float newAudioSincePartial = windowSec - lastPartialAt;
|
||||
const float needed = std::max(0.8f, 0.35f * windowSec);
|
||||
|
||||
// 静音够久 -> 收尾
|
||||
if (silenceAcc >= kSilenceTimeout && windowSec >= kMinSpeech) {
|
||||
emitFinal(transcribeWindow(window));
|
||||
window.clear();
|
||||
silenceAcc = 0.0f;
|
||||
lastPartialAt = 0.0f;
|
||||
continue;
|
||||
}
|
||||
|
||||
// 窗口过长 -> 强制收尾，避免延迟无限增长
|
||||
if (windowSec >= kMaxWindow) {
|
||||
emitFinal(transcribeWindow(window));
|
||||
window.clear();
|
||||
silenceAcc = 0.0f;
|
||||
lastPartialAt = 0.0f;
|
||||
continue;
|
||||
}
|
||||
|
||||
// 定期输出 partial（预览）
|
||||
if (newAudioSincePartial >= needed) {
|
||||
emitPartial(transcribeWindow(window));
|
||||
lastPartialAt = windowSec;
|
||||
}
|
||||
}
|
||||
|
||||
if (off > 0) buffer.erase(buffer.begin(), buffer.begin() + (long) off);
|
||||
}
|
||||
|
||||
if (!window.empty() && (float) window.size() / sampleRate_ >= kMinSpeech) {
|
||||
emitFinal(transcribeWindow(window));
|
||||
}
|
||||
}
|
||||
|
||||
lc::Qwen3Asr & asr_;
|
||||
int sampleRate_;
|
||||
std::thread worker_;
|
||||
std::mutex mutex_;
|
||||
std::condition_variable cv_;
|
||||
std::vector<float> pending_;
|
||||
bool quit_ = false;
|
||||
std::mutex printMutex_;
|
||||
std::string committed_; // 当前这句已提交的稳定前缀
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 文件模式
|
||||
// ---------------------------------------------------------------------------
|
||||
int runFile(lc::Qwen3Asr & asr, const QString & path)
|
||||
{
|
||||
std::vector<float> pcm;
|
||||
@@ -238,6 +53,9 @@ int runFile(lc::Qwen3Asr & asr, const QString & path)
|
||||
return 0;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 流式文件模式（把文件当音频流按块喂进去）
|
||||
// ---------------------------------------------------------------------------
|
||||
int runStreamFile(lc::Qwen3Asr & asr, const QString & path, const lc::StreamOptions & sop)
|
||||
{
|
||||
std::vector<float> pcm;
|
||||
@@ -248,10 +66,9 @@ int runStreamFile(lc::Qwen3Asr & asr, const QString & path, const lc::StreamOpti
|
||||
}
|
||||
|
||||
const double dur = (double) pcm.size() / asr.sampleRate();
|
||||
std::fprintf(stderr, "流式转写: %s (%.2fs) chunk=%.1fs window=%.1fs "
|
||||
"unfixed_token=%d unfixed_chunk=%d\n",
|
||||
qUtf8Printable(path), dur, sop.chunkSec, sop.windowSec,
|
||||
sop.unfixedTokenNum, sop.unfixedChunkNum);
|
||||
std::fprintf(stderr, "流式转写: %s (%.2fs) chunk=%.1fs segment=%.1fs mode=%s\n",
|
||||
qUtf8Printable(path), dur, sop.chunkSec, sop.maxUtteranceSec,
|
||||
sop.incremental ? "incremental" : "window");
|
||||
|
||||
lc::StreamingAsr stream(asr, sop);
|
||||
const size_t chunkSamples = (size_t) (sop.chunkSec * asr.sampleRate());
|
||||
@@ -260,15 +77,12 @@ int runStreamFile(lc::Qwen3Asr & asr, const QString & path, const lc::StreamOpti
|
||||
std::printf("\n[chunk %2d] 窗口 %5.1fs / 编码 %5.2fs / 本步 %5.2fs%s\n",
|
||||
s.chunkIndex, s.windowSeconds, s.encodeSeconds, s.inferSeconds,
|
||||
s.ok ? "" : " <失败>");
|
||||
if (!s.ok) {
|
||||
std::printf(" error: %s\n", s.error.c_str());
|
||||
}
|
||||
if (!s.ok) std::printf(" error: %s\n", s.error.c_str());
|
||||
std::printf(" 已提交: %s\n", s.committed.c_str());
|
||||
std::printf(" 含尾部: %s\n", s.full.c_str());
|
||||
std::fflush(stdout);
|
||||
};
|
||||
|
||||
// 模拟音频按 chunk 到达
|
||||
for (size_t off = 0; off < pcm.size(); off += chunkSamples) {
|
||||
const size_t n = std::min(chunkSamples, pcm.size() - off);
|
||||
stream.feed(pcm.data() + off, n, onStep);
|
||||
@@ -281,74 +95,162 @@ int runStreamFile(lc::Qwen3Asr & asr, const QString & path, const lc::StreamOpti
|
||||
return 0;
|
||||
}
|
||||
|
||||
int runLive(lc::Qwen3Asr & asr)
|
||||
// ---------------------------------------------------------------------------
|
||||
// 命令行实时模式（无窗口，便于在无显示环境下调试）
|
||||
// ---------------------------------------------------------------------------
|
||||
int runCliLive(lc::Qwen3Asr & asr, const lc::StreamOptions & sop)
|
||||
{
|
||||
LiveTranscriber live(asr, asr.sampleRate());
|
||||
lc::MonitorCapture capture;
|
||||
lc::LiveTranscriber live(asr);
|
||||
|
||||
std::string err;
|
||||
if (!capture.start(asr.sampleRate(), [&live](const float * s, size_t n) { live.onAudio(s, n); }, &err)) {
|
||||
std::fprintf(stderr, "启动采集失败: %s\n", err.c_str());
|
||||
QObject::connect(&live, &lc::LiveTranscriber::caption,
|
||||
[](const QString & committed, const QString & tail, const QString & lang) {
|
||||
// 用 \r 原地刷新：稳定部分正常显示，尾部加括号表示未定
|
||||
std::printf("\r\033[K[%s] %s\033[2m%s\033[0m",
|
||||
lc::languageDisplayName(lang.toStdString()).c_str(),
|
||||
qUtf8Printable(committed), qUtf8Printable(tail));
|
||||
std::fflush(stdout);
|
||||
});
|
||||
QObject::connect(&live, &lc::LiveTranscriber::failed, [](const QString & m) {
|
||||
std::fprintf(stderr, "\n采集失败: %s\n", qUtf8Printable(m));
|
||||
});
|
||||
|
||||
QString err;
|
||||
if (!live.start(sop, &err)) {
|
||||
std::fprintf(stderr, "启动采集失败: %s\n", qUtf8Printable(err));
|
||||
return 1;
|
||||
}
|
||||
std::fprintf(stderr, "正在采集: %s (Ctrl+C 退出)\n", qUtf8Printable(live.deviceName()));
|
||||
return QCoreApplication::exec();
|
||||
}
|
||||
|
||||
std::fprintf(stderr, "正在采集: %s (%d Hz)\n按 Ctrl+C 退出\n\n",
|
||||
capture.deviceName().c_str(), asr.sampleRate());
|
||||
// ---------------------------------------------------------------------------
|
||||
// 图形界面模式：悬浮字幕窗
|
||||
// ---------------------------------------------------------------------------
|
||||
int runGui(lc::Qwen3Asr & asr, const lc::StreamOptions & sop,
|
||||
const QString & screenshotPath, int screenshotDelaySec, int opacityPercent)
|
||||
{
|
||||
lc::CaptionWindow window;
|
||||
window.setPanelOpacity(opacityPercent);
|
||||
lc::LiveTranscriber live(asr);
|
||||
|
||||
live.start();
|
||||
// 简单起见：主线程等待，Ctrl+C 结束进程
|
||||
for (;;) std::this_thread::sleep_for(std::chrono::seconds(1));
|
||||
return 0;
|
||||
QObject::connect(&live, &lc::LiveTranscriber::caption, &window,
|
||||
[&window](const QString & committed, const QString & tail,
|
||||
const QString & lang) {
|
||||
window.setLanguage(
|
||||
QString::fromStdString(lc::languageDisplayName(lang.toStdString())));
|
||||
window.setCaption(committed, tail);
|
||||
});
|
||||
QObject::connect(&live, &lc::LiveTranscriber::failed, &window, [](const QString & m) {
|
||||
qWarning("采集失败: %s", qUtf8Printable(m));
|
||||
});
|
||||
QObject::connect(&window, &lc::CaptionWindow::closed, qApp, &QCoreApplication::quit);
|
||||
QObject::connect(&window, &lc::CaptionWindow::languageClicked, &window, [&window] {
|
||||
qInfo() << "语言标签被点击（设置界面将在 Phase 4 接入）:" << window.isCollapsed();
|
||||
});
|
||||
|
||||
QString err;
|
||||
if (!live.start(sop, &err)) {
|
||||
qCritical("启动采集失败: %s", qUtf8Printable(err));
|
||||
return 1;
|
||||
}
|
||||
qInfo("正在采集: %s", qUtf8Printable(live.deviceName()));
|
||||
|
||||
window.show();
|
||||
|
||||
if (!screenshotPath.isEmpty()) {
|
||||
// 自截图：用于验证渲染（不依赖外部截图工具）
|
||||
QTimer::singleShot(screenshotDelaySec * 1000, &window, [&window, screenshotPath] {
|
||||
const bool ok1 = window.grab().save(screenshotPath);
|
||||
|
||||
// 再截一张展开态的
|
||||
window.setCollapsed(false);
|
||||
const QString expanded = screenshotPath;
|
||||
QString expandedPath = expanded;
|
||||
expandedPath.replace(QStringLiteral(".png"), QStringLiteral("-expanded.png"));
|
||||
const bool ok2 = window.grab().save(expandedPath);
|
||||
|
||||
std::fprintf(stderr, "截图: 折叠态 %s / 展开态 %s\n",
|
||||
ok1 ? "成功" : "失败", ok2 ? "成功" : "失败");
|
||||
QCoreApplication::quit();
|
||||
});
|
||||
}
|
||||
|
||||
return QCoreApplication::exec();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char ** argv)
|
||||
{
|
||||
QCoreApplication app(argc, argv);
|
||||
QCoreApplication::setApplicationName("live-caption");
|
||||
// 文件类模式不需要图形环境；实时模式（默认）需要 QApplication
|
||||
bool wantsWidgets = true;
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
const QString a = QString::fromLocal8Bit(argv[i]);
|
||||
if (a == QLatin1String("--file") || a == QLatin1String("--stream-file")) {
|
||||
wantsWidgets = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
std::unique_ptr<QCoreApplication> app;
|
||||
if (wantsWidgets) {
|
||||
// 说明意图：DPI 取整策略必须在 QGuiApplication 之前定。
|
||||
// （Deepin 的会话环境已经设了 QT_SCALE_FACTOR_ROUNDING_POLICY，
|
||||
// 所以本机仍会看到 Qt 的那条提醒，那不是本程序的问题。）
|
||||
QGuiApplication::setHighDpiScaleFactorRoundingPolicy(
|
||||
Qt::HighDpiScaleFactorRoundingPolicy::PassThrough);
|
||||
app.reset(new QApplication(argc, argv));
|
||||
} else {
|
||||
app.reset(new QCoreApplication(argc, argv));
|
||||
}
|
||||
QCoreApplication::setApplicationName(QStringLiteral("live-caption"));
|
||||
|
||||
QCommandLineParser parser;
|
||||
parser.setApplicationDescription("实时字幕原型（Qwen3-ASR + llama.cpp）");
|
||||
parser.setApplicationDescription(QStringLiteral("实时字幕（Qwen3-ASR + llama.cpp）"));
|
||||
parser.addHelpOption();
|
||||
|
||||
QCommandLineOption optModel("model", "主模型 GGUF 路径（架构须为 qwen3vl）", "path");
|
||||
QCommandLineOption optMmproj("mmproj", "mmproj GGUF 路径（clip + qwen3a）", "path");
|
||||
QCommandLineOption optFile("file", "转写一个 wav 文件后退出（不开启实时采集）", "path");
|
||||
QCommandLineOption optFile("file", "转写一个 wav 文件后退出", "path");
|
||||
QCommandLineOption optStreamFile("stream-file", "以流式模式转写 wav 文件后退出", "path");
|
||||
QCommandLineOption optCliLive("cli-live", "实时转写但只用命令行输出（不开窗口）");
|
||||
QCommandLineOption optThreads("threads", "推理线程数", "n", "8");
|
||||
QCommandLineOption optMaxTokens("max-tokens", "单段最多生成 token 数", "n", "256");
|
||||
QCommandLineOption optWarmup("warmup", "mtmd 初始化后做一次预热编码");
|
||||
QCommandLineOption optVerbose("verbose", "输出 llama.cpp/mtmd 的详细日志");
|
||||
QCommandLineOption optStreamFile("stream-file", "以流式模式（chunk + 前缀回滚）转写 wav", "path");
|
||||
QCommandLineOption optStreamChunk("stream-chunk", "流式 chunk 秒数", "sec", "2.0");
|
||||
QCommandLineOption optStreamWindow("stream-window", "流式滑窗上限秒数（0=不滑窗）", "sec", "16");
|
||||
QCommandLineOption optStreamWindow("stream-window", "滑窗上限秒数（仅 window 模式）", "sec", "16");
|
||||
QCommandLineOption optStreamSegment("stream-segment", "单段最长秒数（0=不限）", "sec", "30");
|
||||
QCommandLineOption optUnfixedTokens("unfixed-tokens", "前缀回滚丢弃的 token 数", "n", "5");
|
||||
QCommandLineOption optUnfixedChunks("unfixed-chunks", "冷启动不带前缀的 chunk 数", "n", "2");
|
||||
QCommandLineOption optStreamMode("stream-mode", "流式后端: incremental（增量 KV 复用）或 window（重编码整个窗口）", "mode", "incremental");
|
||||
QCommandLineOption optStreamSegment("stream-segment", "单段最长秒数，超过则提交并重置上下文（0=不限）", "sec", "30");
|
||||
QCommandLineOption optStreamMode("stream-mode", "流式后端: incremental 或 window", "mode", "incremental");
|
||||
QCommandLineOption optVerbose("verbose", "输出 llama.cpp/mtmd 的详细日志");
|
||||
QCommandLineOption optScreenshot("screenshot", "启动后自动截图并退出（验证渲染用）", "path");
|
||||
QCommandLineOption optScreenshotDelay("screenshot-delay", "截图前等待秒数", "sec", "6");
|
||||
QCommandLineOption optOpacity("opacity", "字幕背景不透明度（20-100，百分比）", "n", "84");
|
||||
|
||||
parser.addOption(optModel);
|
||||
parser.addOption(optMmproj);
|
||||
parser.addOption(optFile);
|
||||
parser.addOption(optStreamFile);
|
||||
parser.addOption(optCliLive);
|
||||
parser.addOption(optThreads);
|
||||
parser.addOption(optMaxTokens);
|
||||
parser.addOption(optWarmup);
|
||||
parser.addOption(optVerbose);
|
||||
parser.addOption(optStreamFile);
|
||||
parser.addOption(optStreamChunk);
|
||||
parser.addOption(optStreamWindow);
|
||||
parser.addOption(optStreamSegment);
|
||||
parser.addOption(optUnfixedTokens);
|
||||
parser.addOption(optUnfixedChunks);
|
||||
parser.addOption(optStreamMode);
|
||||
parser.addOption(optStreamSegment);
|
||||
parser.process(app);
|
||||
parser.addOption(optVerbose);
|
||||
parser.addOption(optScreenshot);
|
||||
parser.addOption(optScreenshotDelay);
|
||||
parser.addOption(optOpacity);
|
||||
parser.process(*app);
|
||||
|
||||
lc::AsrOptions asrOpt;
|
||||
asrOpt.modelPath = parser.value(optModel).toStdString();
|
||||
asrOpt.mmprojPath = parser.value(optMmproj).toStdString();
|
||||
asrOpt.threads = parser.value(optThreads).toInt();
|
||||
asrOpt.maxTokens = parser.value(optMaxTokens).toInt();
|
||||
asrOpt.warmup = parser.isSet(optWarmup);
|
||||
asrOpt.verbose = parser.isSet(optVerbose);
|
||||
|
||||
if (asrOpt.modelPath.empty() || asrOpt.mmprojPath.empty()) {
|
||||
@@ -356,31 +258,31 @@ int main(int argc, char ** argv)
|
||||
return 1;
|
||||
}
|
||||
|
||||
lc::StreamOptions sop;
|
||||
sop.chunkSec = parser.value(optStreamChunk).toDouble();
|
||||
sop.windowSec = parser.value(optStreamWindow).toDouble();
|
||||
sop.maxUtteranceSec = parser.value(optStreamSegment).toDouble();
|
||||
sop.unfixedTokenNum = parser.value(optUnfixedTokens).toInt();
|
||||
sop.unfixedChunkNum = parser.value(optUnfixedChunks).toInt();
|
||||
sop.maxTokensPerStep = asrOpt.maxTokens;
|
||||
sop.incremental = parser.value(optStreamMode) != QLatin1String("window");
|
||||
|
||||
lc::Qwen3Asr asr;
|
||||
std::string err;
|
||||
const auto tl0 = std::chrono::steady_clock::now();
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
if (!asr.load(asrOpt, &err)) {
|
||||
std::fprintf(stderr, "加载失败: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
std::fprintf(stderr, "模型加载完成: %.2fs (采样率 %d)\n",
|
||||
std::chrono::duration<double>(std::chrono::steady_clock::now() - tl0).count(),
|
||||
std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count(),
|
||||
asr.sampleRate());
|
||||
|
||||
if (parser.isSet(optFile)) {
|
||||
return runFile(asr, parser.value(optFile));
|
||||
}
|
||||
|
||||
if (parser.isSet(optStreamFile)) {
|
||||
lc::StreamOptions sop;
|
||||
sop.chunkSec = parser.value(optStreamChunk).toDouble();
|
||||
sop.windowSec = parser.value(optStreamWindow).toDouble();
|
||||
sop.unfixedTokenNum = parser.value(optUnfixedTokens).toInt();
|
||||
sop.unfixedChunkNum = parser.value(optUnfixedChunks).toInt();
|
||||
sop.incremental = parser.value(optStreamMode) != QLatin1String("window");
|
||||
sop.maxUtteranceSec = parser.value(optStreamSegment).toDouble();
|
||||
return runStreamFile(asr, parser.value(optStreamFile), sop);
|
||||
}
|
||||
|
||||
return runLive(asr);
|
||||
if (parser.isSet(optFile)) return runFile(asr, parser.value(optFile));
|
||||
if (parser.isSet(optStreamFile)) return runStreamFile(asr, parser.value(optStreamFile), sop);
|
||||
if (parser.isSet(optCliLive)) return runCliLive(asr, sop);
|
||||
return runGui(asr, sop,
|
||||
parser.value(optScreenshot),
|
||||
parser.value(optScreenshotDelay).toInt(),
|
||||
parser.value(optOpacity).toInt());
|
||||
}
|
||||
+34
-7
@@ -50,12 +50,33 @@ void StreamingAsr::feed(const float * pcm, size_t n,
|
||||
|
||||
void StreamingAsr::flush(const std::function<void(const Step &)> & onStep)
|
||||
{
|
||||
if (pending_.empty()) return;
|
||||
if (!pending_.empty()) {
|
||||
std::vector<float> chunk;
|
||||
chunk.swap(pending_);
|
||||
// 流结束：没有更多音频可以修正尾部了，这一步全部提交
|
||||
runStep(chunk, true, onStep);
|
||||
return;
|
||||
}
|
||||
|
||||
std::vector<float> chunk;
|
||||
chunk.swap(pending_);
|
||||
// 流结束：没有更多音频可以修正尾部了，这一步全部提交
|
||||
runStep(chunk, true, onStep);
|
||||
// 没有待处理音频：直接把当前已稳定文本提交为最终
|
||||
if (committedRaw_.empty()) return;
|
||||
|
||||
historyText_ += parseAsrOutput(committedRaw_).text;
|
||||
committedRaw_.clear();
|
||||
|
||||
Step s;
|
||||
s.chunkIndex = chunkIndex_;
|
||||
s.language = language_;
|
||||
s.ok = true;
|
||||
s.full = historyText_;
|
||||
s.committed = historyText_;
|
||||
if (onStep) onStep(s);
|
||||
|
||||
// 收束这一段
|
||||
asr_.streamReset();
|
||||
buffer_.clear();
|
||||
totalSeconds_ = 0.0;
|
||||
chunkIndex_ = 0;
|
||||
}
|
||||
|
||||
void StreamingAsr::runStep(const std::vector<float> & audio, bool finalStep,
|
||||
@@ -117,11 +138,17 @@ void StreamingAsr::runStep(const std::vector<float> & audio, bool finalStep,
|
||||
historyText_ += pSeg.text;
|
||||
s.full = historyText_;
|
||||
s.committed = historyText_;
|
||||
s.tail.clear();
|
||||
} else {
|
||||
// 最后 unfixedTokenNum 个 token 视为不稳定，不提交也不回喂
|
||||
committedRaw_ = asr_.dropLastTokens(s.rawFull, opt_.unfixedTokenNum);
|
||||
s.full = historyText_ + pSeg.text;
|
||||
s.committed = historyText_ + parseAsrOutput(committedRaw_).text;
|
||||
const std::string committedSeg = parseAsrOutput(committedRaw_).text;
|
||||
s.full = historyText_ + pSeg.text;
|
||||
s.committed = historyText_ + committedSeg;
|
||||
// 不稳定尾部 = 本段文本去掉已稳定的那部分
|
||||
s.tail = s.full.size() > s.committed.size()
|
||||
? s.full.substr(s.committed.size())
|
||||
: std::string();
|
||||
}
|
||||
s.language = language_;
|
||||
}
|
||||
|
||||
+3
-1
@@ -37,6 +37,7 @@ public:
|
||||
int chunkIndex = 0;
|
||||
std::string language;
|
||||
std::string committed; // 稳定部分
|
||||
std::string tail; // 不稳定尾部（committed 之后的部分）
|
||||
std::string full; // 含不稳定尾部
|
||||
std::string rawFull; // 未 trim 的完整文本（含 language 头）
|
||||
double windowSeconds = 0.0;
|
||||
@@ -51,7 +52,8 @@ public:
|
||||
|
||||
// 喂入 16k 单声道音频；每凑够 chunkSec 就跑一步并回调
|
||||
void feed(const float * pcm, size_t n, const std::function<void(const Step &)> & onStep);
|
||||
// 处理不足一块的残留音频，并作为最后一步提交全部文本
|
||||
// 处理不足一块的残留音频，并作为最后一步提交全部文本。
|
||||
// 若没有待处理音频，也把当前已稳定文本提交为最终（静音分段时会用到）。
|
||||
void flush(const std::function<void(const Step &)> & onStep);
|
||||
|
||||
// 结束当前这段（例如检测到静音）：清空 KV 与状态，下一段重新开始
|
||||
|
||||
Reference in new issue
Block a user