feat(web): 语音输入(讯飞语音听写 IAT)(bump 0.47.0)
- core/asr_xfyun.py:IAT wss 客户端(HMAC-SHA256 签名 / 9600B 帧 / vad_eos=10000),整段 PCM→文本 - web/app.py:POST /v1/asr/transcribe(裸 PCM 16k/16bit/mono body,>60s→413,未配 env→501,讯飞错→502) - 前端「🎙 语音」按钮:Web Audio 采 PCM 降采样 16k → 转写 → insertText 填输入框(接 undo 栈,不自动发送);60s 自动停;渠道只读镜像锁定 - 凭据走 env XFYUN_APPID/API_KEY/API_SECRET(.env 当前为测试 key,待换正式) - scripts/diag_asr.py 链路诊断;requirements 显式加 websockets - 已实测:TTS 合成 5.2s 中文经端点转写逐字正确;企微语音(AMR 需 ffmpeg)留二期 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
9ae1040faf
commit
a36a8f1da4
|
|
@ -23,6 +23,7 @@
|
|||
|
||||
### 2026-07
|
||||
|
||||
- **07-07 / 0.47.0**:web 端语音输入(讯飞语音听写 IAT)——聊天输入区新增「🎙 语音」按钮:点击录音(Web Audio ScriptProcessor 采 PCM → 前端线性插值降采样 16k/16bit/mono)→ 再点停止 → `POST /v1/asr/transcribe`(裸 PCM body,require_user)→ 后端 `core/asr_xfyun.py` 连讯飞 wss(HMAC-SHA256 签名,9600B/帧 40ms 节奏,vad_eos=10000 防句间停顿截尾,未开 dwa 整段拼接)→ 文本 insertText 插输入框光标处(接原生 undo 栈,**不自动发送**)。60s 上限自动停(讯飞单会话限制);录音需 HTTPS/localhost;渠道只读镜像 task 按钮锁定;`XFYUN_APPID/API_KEY/API_SECRET` env 未配返 501(**当前 .env 为测试 key,待换正式**)。诊断 `scripts/diag_asr.py`(静音冒烟 / 带 PCM 文件真测);requirements 显式加 `websockets`。已实测:TTS 合成中文 5.2s 音频经端点转写逐字正确。企微语音消息(AMR 需 ffmpeg 转码)留待二期;将来流式逐字上屏可复用 `build_auth_url` 签临时 URL。
|
||||
- **07-07 / 0.46.2**:消息分页窗口对齐轮次边界(纯前端,`chat.js`)——原按固定 30 条翻页/跳转,窗口边界常切在 assistant 工具调用序列中间(顶部出现孤儿 tool 消息),点目录圆点用 `before_idx=idx+11` 启发式(长轮拉不全、短轮混无关上文)。业界调研确认:分页单位保持消息条数是共识(Discord/OpenAI 均按条 + 每页上限,轮次大小无上界不能当传输单位),轮次归组属渲染层职责。实现:idx 连续 + outline 已含全部 user 轮次起点 → 前端可精确算区间,后端 API 零改动。① 向上翻页 `alignedEarlierLimit`:从窗口顶往前按整轮累加攒够 30 条,新窗口顶恒为 user 消息;② 圆点跳转:`after_idx=起点-1` + 相邻边界差精确拉整轮(短轮补到 30 铺满、超大轮按 `MSG_FETCH_MAX=100` 截断,后续向上加载逐步自愈对齐);③ outline 未加载/idx 不在 outline → 回退旧按条行为。后端 limit 无 clamp,前端以 100 硬上限自律。
|
||||
- **07-07 / 0.46.1**:撤除企微欢迎推送机制——实际部署采用聊天优先布局(不配应用主页 + 菜单「工作台」按钮)后,点应用直进会话,冷启动问题不复存在,0.46.0 的欢迎推送成了死重:删 `wecom_welcomes` 表(migration 0021 撤下,DB 已 downgrade 回 0020,表建后无一行写入)、`maybe_send_wecom_welcome`、免登回调欢迎钩子、群发脚本、`list_visible_userids`。**保留**:未绑定入站回绑定指引(聊天优先下必需)、`core/branding.py` 品牌单一源。RUN.md 企微段只留推荐布局 + 未绑定指引两条。
|
||||
- **07-07 / 0.46.0**:企业微信会话冷启动 + 未绑定入站回指引——背景:应用配了「应用主页」后点应用只进主页,应用会话要等应用发过第一条消息才出现在成员消息列表(首次登录的员工没入口和 bot 打字)。最终采用**聊天优先布局**(企微后台不配主页、菜单放「工作台」按钮指向 `/v1/wecom/entry`,点应用直进会话),代码侧:① 回调对**未绑定成员**的 text/图片/文件消息不再静默,每条回一句绑定指引(聊天优先下新员工上来就打字,静默会被当成 bot 坏了;事件不回);② 新表 `wecom_welcomes`(0021,wecom_userid PK,未绑定成员也能去重)+ `service.maybe_send_wecom_welcome`(两层去重:已推过 / 已绑定且有 chat task;claim-first INSERT 并发安全,发送失败释放占坑),免登入口回调对首次进入的成员自动推欢迎语(菜单「工作台」/主页两条路都触发);③ 可选工具 `scripts/wecom_welcome_broadcast.py`:配主页布局下给存量成员批量"种"会话入口(`--dry-run` 先看名单;可见范围枚举 `wecom.list_visible_userids`:agent/get + 部门/标签展开);④ 品牌名单一事实源挪到 `core/branding.py`(web 层外的欢迎语也要用,app.py BRAND 引它)。RUN.md 企微段同步推荐布局。
|
||||
|
|
@ -207,6 +208,7 @@ core/memory.py 81 ← per-user `.memory/` dotfile
|
|||
core/export_docx.py 383
|
||||
core/storage/{__init__,engine,models,usage,utils}.py ← 4 表(0004-0007 演进);record_chat/image_usage
|
||||
core/ark_client.py 105 ← 火山方舟 HTTP 客户端
|
||||
core/asr_xfyun.py 170 ← 讯飞语音听写 IAT wss 客户端(整段 PCM→文本;web 语音输入用,diag: scripts/diag_asr.py)
|
||||
core/agent_builder.py 340 ← 装配 lib(有 ARK_API_KEY 才挂 SeedreamTool);build_skill_registry 装两来源
|
||||
core/executor.py / sandbox/{network,pool}.py / executor_docker.py ← Executor ABC + Docker per-user 容器池
|
||||
tools/{base,fs,shell,run_python,skill_tool,skill_authoring,seedream,seedance,web_search,web_fetch,documents,materials_project}.py ← skill_authoring=save_skill/fork_skill(host-side 写 user .skills)
|
||||
|
|
|
|||
8
RUN.md
8
RUN.md
|
|
@ -85,6 +85,12 @@
|
|||
# 用户即可在企业微信里直接和 zcbot 对话(回调 URL = <公网 base>/v1/wecom/callback)。
|
||||
# WECOM_CALLBACK_TOKEN=... # 接收消息 Token(企微后台生成)
|
||||
# WECOM_CALLBACK_AESKEY=... # EncodingAESKey(43 字符,企微后台生成)
|
||||
# 讯飞语音听写(web 端「🎙 语音」麦克风输入,core/asr_xfyun.py):可选,三件套齐才可用,
|
||||
# 未配时点按钮转写返 501。控制台 https://console.xfyun.cn/ 开通「语音听写(流式版)」取凭据。
|
||||
# 验证链路:`.venv/Scripts/python.exe scripts/diag_asr.py`(静音冒烟)或带 PCM 文件参数真测。
|
||||
# XFYUN_APPID=...
|
||||
# XFYUN_API_KEY=...
|
||||
# XFYUN_API_SECRET=...
|
||||
# 对外品牌名:zcbot 是内部代号,所有用户可见文案(页面标题/顶栏/登录卡/微信·企微推送与
|
||||
# 提示页)统一用品牌名。/healthz 返回 brand 字段,前端 boot 拉取覆盖静态页默认值。
|
||||
# ZCBOT_BRAND_NAME=总院科研辅助助手 # 可选,默认即此值
|
||||
|
|
@ -226,6 +232,7 @@ curl --noproxy '*' -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8765/v1/ta
|
|||
| `POST /v1/tasks/{id}/cancel` | 协作式 cancel;`run_status != running` → 409;LLM 走 streaming,chunk 间 poll cancel — 延迟 100ms 级,基本秒退 | 必填 |
|
||||
| `POST /v1/tasks/{id}/clear` | 清空当前 task 全部 messages + reset `tasks.tokens_prompt/completion/cost_cny` 三列累计 + `run_status='idle'`;`usage_events`(账单记账)**不动**,只 `message_id` 列变 NULL;run 活跃中(running/cancelling)→ 409(先 cancel);FS 文件保留 | 必填 |
|
||||
| `POST /v1/tasks/{id}/optimize_prompt` | body `{text(req, ≤4000), image_model?=""}`;同步调当前 task model 润色草稿,返 `{optimized, model_profile, tokens_in, tokens_out, cost_cny}`;**不**写 messages、**不**累计 task 三列(顶栏数字不污染),只在 `usage_events` 写一行 `kind="prompt_optimize"`(对账可见);不与主对话 run 互斥(允许 streaming 中并行润色) | 必填 |
|
||||
| `POST /v1/asr/transcribe` | body 为裸 PCM(16kHz/16bit/单声道/小端,`application/octet-stream`)→ 讯飞 IAT 整段转写,返 `{text}`;>60s → 413;`XFYUN_*` env 未配 → 501;讯飞侧错误 → 502(带错误码提示)。前端「🎙 语音」按钮用 | 必填 |
|
||||
| `GET /v1/files?path=` | 列 user_root 下条目 + 面包屑;dotfile 隐藏 | 必填 |
|
||||
| `GET /v1/files/download?path=` | 下单文件 | 必填 |
|
||||
| `POST /v1/files/upload` | multipart 上传到 `<user_root>/<path>/`;路径不存在自动 mkdir,重名覆盖 | 必填 |
|
||||
|
|
@ -816,6 +823,7 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_<user_uuid>" /opt
|
|||
| Export 报 "无可导出内容" | task 没 messages(只 system 不算);先发条消息再 export |
|
||||
| `NoSubtaskError: working_dir ... 前缀嵌套` | §7.4 no-subtask:同 user 不允许 working_dir 嵌套(child / parent)。**同项目多对话**用**完全相同**的 working_dir;否则改成 sibling(平级) |
|
||||
| `main.py web` 启动后 curl 连不上 | 检查 proxy(`HTTP_PROXY` / `HTTPS_PROXY`):本地服务 127.0.0.1,系统 proxy 拦截会 502。临时 `unset HTTP_PROXY HTTPS_PROXY` 或 `curl --noproxy '*'`。验通:`curl --noproxy '*' http://127.0.0.1:8765/healthz` |
|
||||
| 「🎙 语音」点了没反应 / 提示"此环境不支持录音" | 浏览器只在**安全上下文**(HTTPS 或 localhost)开放麦克风;http://IP 访问必现。上 HTTPS 或本机 localhost 试。凭据/链路问题跑 `scripts/diag_asr.py` 定位(11200=讯飞服务未开通或量用完,10105/10313=appid 错) |
|
||||
| SSE 卡住不流(经 nginx) | 反代要关 buffering — 后端响应头已带 `X-Accel-Buffering: no`(nginx ≥1.5.6 默认认)。仍卡看 nginx 配 `proxy_buffering off; proxy_read_timeout 3600s;` |
|
||||
| platform CORS preflight 失败 | 本地 dev `allow_origins=["*"]` 应该没事;部署后看是否按 platform 域名收紧过头 |
|
||||
| `POST /v1/tasks/{id}/messages` 返 409 `task already has an active run` | 上一条消息的 BG run 还没跑完;等流式 done 或点 stop / `POST .../cancel`;服务异常下 `run_status` 卡 `running`/`cancelling`,启动 reaper 会清 |
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
# zcbot 版本号单一事实源:web/app.py 的 FastAPI version、/healthz 返回、前端展示都引这里。
|
||||
# 改版本只动这一行。
|
||||
__version__ = "0.46.2"
|
||||
__version__ = "0.47.0"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,157 @@
|
|||
"""讯飞语音听写(IAT,流式版)WebSocket 客户端 — 整段音频转写。
|
||||
|
||||
接口文档: https://www.xfyun.cn/doc/asr/voicedictation/API.html
|
||||
- 鉴权: HMAC-SHA256 签名拼进 wss URL query(authorization/date/host)
|
||||
- 音频: 仅收 16kHz/16bit/单声道 PCM(raw),单会话上限 60 秒
|
||||
- 凭据: env XFYUN_APPID / XFYUN_API_KEY / XFYUN_API_SECRET(换正式 key 只改 .env)
|
||||
|
||||
本模块只做"整段 PCM → 最终文本"(web 语音输入按钮用);未开 dwa 动态修正 ——
|
||||
那是实时逐字上屏才需要的,整段转写直接按序拼接 result.ws 即可。将来做流式
|
||||
上屏时复用 build_auth_url() 签临时 URL 给前端直连。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from contextlib import suppress
|
||||
from urllib.parse import urlencode
|
||||
from wsgiref.handlers import format_date_time
|
||||
|
||||
XFYUN_HOST = "iat-api.xfyun.cn"
|
||||
XFYUN_PATH = "/v2/iat"
|
||||
|
||||
SAMPLE_RATE = 16000
|
||||
BYTES_PER_SECOND = SAMPLE_RATE * 2 # 16bit mono
|
||||
MAX_SECONDS = 60 # 讯飞单会话硬上限
|
||||
MAX_PCM_BYTES = MAX_SECONDS * BYTES_PER_SECOND
|
||||
|
||||
# 每帧 9600B(=300ms 音频),base64 后 12800B < 讯飞单帧上限 13000B;
|
||||
# 帧间 40ms 为文档推荐节奏,整段上传比实时快 7.5 倍(60s 音频 ≈ 8s 传完)
|
||||
_FRAME_BYTES = 9600
|
||||
_FRAME_INTERVAL = 0.04
|
||||
_RECV_TIMEOUT = 30 # 单次等响应上限;讯飞 10s 无数据也会主动断
|
||||
|
||||
# 常见错误码 → 给用户/运维看的中文提示(全量见文档错误码页)
|
||||
_ERR_HINTS = {
|
||||
10105: "appid 不合法(检查 XFYUN_APPID)",
|
||||
10107: "参数值非法",
|
||||
10110: "无授权许可(服务未开通或已过期)",
|
||||
10114: "会话超时(单次音频超 60 秒)",
|
||||
10163: "业务参数校验失败",
|
||||
10165: "会话句柄无效",
|
||||
10313: "appid 不能为空或与 apikey 不匹配",
|
||||
11200: "无权限或用量已用完(检查讯飞控制台服务量)",
|
||||
11201: "当日流控超限",
|
||||
}
|
||||
|
||||
|
||||
class XfyunASRError(RuntimeError):
|
||||
"""转写失败(讯飞返回错误码 / 连接异常),code 为讯飞错误码(连接类异常为 None)。"""
|
||||
|
||||
def __init__(self, message: str, code: int | None = None):
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
class XfyunASRNotConfigured(XfyunASRError):
|
||||
"""XFYUN_* env 未配置 — 上层应返回 501 而非 502。"""
|
||||
|
||||
|
||||
def build_auth_url(api_key: str, api_secret: str) -> str:
|
||||
"""按讯飞规范生成带签名的 wss URL(date 用 RFC1123 GMT)。"""
|
||||
date = format_date_time(time.time())
|
||||
signature_origin = f"host: {XFYUN_HOST}\ndate: {date}\nGET {XFYUN_PATH} HTTP/1.1"
|
||||
signature = base64.b64encode(
|
||||
hmac.new(api_secret.encode(), signature_origin.encode(), hashlib.sha256).digest()
|
||||
).decode()
|
||||
authorization_origin = (
|
||||
f'api_key="{api_key}", algorithm="hmac-sha256", '
|
||||
f'headers="host date request-line", signature="{signature}"'
|
||||
)
|
||||
authorization = base64.b64encode(authorization_origin.encode()).decode()
|
||||
query = urlencode({"authorization": authorization, "date": date, "host": XFYUN_HOST})
|
||||
return f"wss://{XFYUN_HOST}{XFYUN_PATH}?{query}"
|
||||
|
||||
|
||||
def _load_credentials() -> tuple[str, str, str]:
|
||||
appid = (os.getenv("XFYUN_APPID") or "").strip()
|
||||
api_key = (os.getenv("XFYUN_API_KEY") or "").strip()
|
||||
api_secret = (os.getenv("XFYUN_API_SECRET") or "").strip()
|
||||
if not (appid and api_key and api_secret):
|
||||
raise XfyunASRNotConfigured(
|
||||
"语音识别未配置:需在 .env 设 XFYUN_APPID / XFYUN_API_KEY / XFYUN_API_SECRET"
|
||||
)
|
||||
return appid, api_key, api_secret
|
||||
|
||||
|
||||
async def transcribe(pcm: bytes, *, language: str = "zh_cn") -> str:
|
||||
"""整段 PCM(16k/16bit/mono, 小端)→ 识别文本。空音频/纯静音返回 ""。"""
|
||||
appid, api_key, api_secret = _load_credentials()
|
||||
if not pcm:
|
||||
return ""
|
||||
if len(pcm) > MAX_PCM_BYTES:
|
||||
raise XfyunASRError(f"音频超长:最多 {MAX_SECONDS} 秒")
|
||||
|
||||
import websockets # uvicorn[standard] 附带;requirements 亦显式声明
|
||||
|
||||
audio_meta = {"format": f"audio/L16;rate={SAMPLE_RATE}", "encoding": "raw"}
|
||||
|
||||
async def _send_frames(ws) -> None:
|
||||
first = True
|
||||
for off in range(0, len(pcm), _FRAME_BYTES):
|
||||
chunk = base64.b64encode(pcm[off : off + _FRAME_BYTES]).decode()
|
||||
frame: dict = {"data": {"status": 1 if not first else 0, **audio_meta, "audio": chunk}}
|
||||
if first:
|
||||
frame["common"] = {"app_id": appid}
|
||||
# vad_eos 拉到上限 10s:整段转写不希望句间停顿被判"说完了"截尾
|
||||
frame["business"] = {
|
||||
"language": language,
|
||||
"domain": "iat",
|
||||
"accent": "mandarin",
|
||||
"vad_eos": 10000,
|
||||
"ptt": 1,
|
||||
}
|
||||
first = False
|
||||
await ws.send(json.dumps(frame))
|
||||
await asyncio.sleep(_FRAME_INTERVAL)
|
||||
await ws.send(json.dumps({"data": {"status": 2, **audio_meta, "audio": ""}}))
|
||||
|
||||
pieces: list[str] = []
|
||||
try:
|
||||
async with websockets.connect(build_auth_url(api_key, api_secret), open_timeout=10) as ws:
|
||||
send_task = asyncio.create_task(_send_frames(ws))
|
||||
try:
|
||||
while True:
|
||||
raw = await asyncio.wait_for(ws.recv(), timeout=_RECV_TIMEOUT)
|
||||
msg = json.loads(raw)
|
||||
code = msg.get("code")
|
||||
if code:
|
||||
hint = _ERR_HINTS.get(code, msg.get("message") or "未知错误")
|
||||
raise XfyunASRError(f"讯飞识别失败({code}):{hint}", code=code)
|
||||
data = msg.get("data") or {}
|
||||
result = data.get("result") or {}
|
||||
pieces.append(
|
||||
"".join(
|
||||
cw.get("w", "")
|
||||
for w in (result.get("ws") or [])
|
||||
for cw in (w.get("cw") or [])
|
||||
)
|
||||
)
|
||||
if data.get("status") == 2:
|
||||
break
|
||||
finally:
|
||||
send_task.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await send_task
|
||||
except XfyunASRError:
|
||||
raise
|
||||
except asyncio.TimeoutError:
|
||||
raise XfyunASRError("讯飞识别超时(30s 无响应)")
|
||||
except Exception as e: # 握手 401/连接断开等
|
||||
raise XfyunASRError(f"讯飞连接失败:{type(e).__name__}: {e}")
|
||||
return "".join(pieces).strip()
|
||||
|
|
@ -20,6 +20,9 @@ markitdown[pdf,docx,pptx,xlsx]>=0.0.1
|
|||
httpx>=0.27.0
|
||||
html2text>=2024.0
|
||||
|
||||
# 语音听写(core/asr_xfyun.py 连讯飞 IAT wss;uvicorn[standard] 也附带,这里显式声明直接依赖)
|
||||
websockets>=12.0
|
||||
|
||||
# 定时任务(§8.5 scheduled_jobs):cron 串 → next_run_at 计算,正确处理 dom/dow OR 语义 + 时区
|
||||
croniter>=2.0
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,67 @@
|
|||
"""讯飞语音听写(IAT)诊断:验证 XFYUN_* 凭据 + wss 协议链路是否通。
|
||||
|
||||
用法(.env 在仓库根):
|
||||
.venv/Scripts/python.exe scripts/diag_asr.py # 2s 静音冒烟(验鉴权/协议,预期空文本)
|
||||
.venv/Scripts/python.exe scripts/diag_asr.py <pcm文件> # 真实音频(16k/16bit/mono 裸 PCM)
|
||||
|
||||
ASCII 输出(Windows 控制台 GBK,不打 emoji);secret 不打印。
|
||||
常见错误码:10105/10313=appid 错;11200=服务未开通或量用完;11201=当日超限。
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
if _ROOT not in sys.path:
|
||||
sys.path.insert(0, _ROOT)
|
||||
|
||||
|
||||
def _load_env(path: str) -> None:
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv(path)
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
k, v = line.split("=", 1)
|
||||
os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
_load_env(os.path.join(_ROOT, ".env"))
|
||||
|
||||
from core.asr_xfyun import SAMPLE_RATE, XfyunASRError, transcribe # noqa: E402
|
||||
|
||||
|
||||
def main() -> int:
|
||||
appid = os.getenv("XFYUN_APPID", "")
|
||||
print(f"[env] XFYUN_APPID={appid or '(missing)'} "
|
||||
f"API_KEY={'set' if os.getenv('XFYUN_API_KEY') else '(missing)'} "
|
||||
f"API_SECRET={'set' if os.getenv('XFYUN_API_SECRET') else '(missing)'}")
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
pcm = open(sys.argv[1], "rb").read()
|
||||
print(f"[audio] file={sys.argv[1]} bytes={len(pcm)} "
|
||||
f"({len(pcm) / (SAMPLE_RATE * 2):.1f}s @16k/16bit/mono)")
|
||||
else:
|
||||
pcm = b"\x00" * (SAMPLE_RATE * 2 * 2) # 2s 静音:只验鉴权+协议,预期识别为空
|
||||
print("[audio] builtin 2s silence (auth/protocol smoke, empty text expected)")
|
||||
|
||||
try:
|
||||
text = asyncio.run(transcribe(pcm))
|
||||
except XfyunASRError as e:
|
||||
print(f"[FAIL] code={e.code} {e}")
|
||||
return 1
|
||||
print(f"[OK] text={text!r}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
33
web/app.py
33
web/app.py
|
|
@ -2850,6 +2850,39 @@ def create_app() -> FastAPI:
|
|||
"cost_cny": float(cost_cny),
|
||||
}
|
||||
|
||||
# ───────────── ASR 语音转写 ─────────────
|
||||
|
||||
@app.post("/v1/asr/transcribe", tags=["asr"])
|
||||
async def asr_transcribe(
|
||||
request: Request,
|
||||
user_id: UUID = Depends(require_user),
|
||||
):
|
||||
"""整段语音转写:body 为 raw PCM(16kHz/16bit/单声道/小端),返回 {"text"}。
|
||||
|
||||
前端麦克风按钮录音(Web Audio 采样降到 16k)→ 本端点 → 讯飞 IAT websocket
|
||||
(core/asr_xfyun.py)。上限 60 秒(讯飞单会话限制);XFYUN_* env 未配 → 501。
|
||||
"""
|
||||
from core.asr_xfyun import (
|
||||
MAX_PCM_BYTES,
|
||||
MAX_SECONDS,
|
||||
XfyunASRError,
|
||||
XfyunASRNotConfigured,
|
||||
transcribe,
|
||||
)
|
||||
|
||||
pcm = await request.body()
|
||||
if not pcm:
|
||||
raise HTTPException(400, "empty audio")
|
||||
if len(pcm) > MAX_PCM_BYTES:
|
||||
raise HTTPException(413, f"音频超长:单次最多 {MAX_SECONDS} 秒")
|
||||
try:
|
||||
text = await transcribe(pcm)
|
||||
except XfyunASRNotConfigured as e:
|
||||
raise HTTPException(501, str(e))
|
||||
except XfyunASRError as e:
|
||||
raise HTTPException(502, str(e))
|
||||
return {"text": text}
|
||||
|
||||
# ───────────── SSE events ─────────────
|
||||
|
||||
@app.get("/v1/tasks/{task_id}/events", tags=["tasks"])
|
||||
|
|
|
|||
|
|
@ -890,6 +890,9 @@
|
|||
#chat-form textarea { flex: 1; }
|
||||
/* 微信渠道 task 只读镜像:输入框置灰禁手输,引导去微信对话 */
|
||||
#chat-input.readonly-locked { background: #f0f0f0; color: var(--muted); cursor: not-allowed; }
|
||||
/* 语音输入按钮:录音中红色呼吸提醒(别忘了停) */
|
||||
#chat-mic.recording { color: #fff; background: #d33; border-color: #d33; animation: mic-pulse 1.6s ease-in-out infinite; }
|
||||
@keyframes mic-pulse { 50% { opacity: .65; } }
|
||||
#chat-form .hint { font-size: 11px; color: var(--muted); }
|
||||
/* 附件托盘:粘贴 / 拖拽的文件 chip 独占一行,累积展示,不与状态文字抢容器 */
|
||||
#chat-attach { display: none; flex-wrap: wrap; gap: 4px 0; align-items: center; }
|
||||
|
|
@ -1564,6 +1567,7 @@
|
|||
<div class="row">
|
||||
<span class="hint" id="chat-hint">就绪</span>
|
||||
<span style="flex:1;"></span>
|
||||
<button type="button" class="small" id="chat-mic" title="语音输入:点击开始录音,再点结束并转写填入输入框(最长 60 秒)">🎙 语音</button>
|
||||
<button type="button" class="small" id="chat-optimize" disabled title="用当前对话模型润色草稿(参考已选生图模型偏好)— 替换为更清晰可执行的 prompt,Ctrl+Z 可撤销">✨ 润色</button>
|
||||
<button type="submit" class="primary" id="chat-action">发送</button>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -1406,6 +1406,8 @@ function applyChannelComposerLock(meta) {
|
|||
const opt = $("chat-optimize");
|
||||
if (opt) opt.disabled = true;
|
||||
}
|
||||
const mic = $("chat-mic");
|
||||
if (mic) mic.disabled = !!cfg; // 只读镜像没有发送入口,语音输入同样锁掉
|
||||
}
|
||||
|
||||
// 润色:同步调后端,把 textarea 内容替成优化后文本。用 execCommand('insertText')
|
||||
|
|
@ -1465,6 +1467,113 @@ async function optimizePrompt() {
|
|||
|
||||
$("chat-optimize").onclick = optimizePrompt;
|
||||
|
||||
// ── 语音输入(讯飞 IAT 整段转写)──────────────────────────────────────────────
|
||||
// 点击开始录音(Web Audio 采 PCM → 降采样 16k/16bit)→ 再点停止 → POST
|
||||
// /v1/asr/transcribe → 文本插到输入框光标处(insertText 接原生 undo 栈,不自动
|
||||
// 发送,用户确认后再发)。录音需安全上下文(HTTPS / localhost);60s 自动停
|
||||
// (讯飞单会话上限)。
|
||||
const ASR_MAX_SECONDS = 60;
|
||||
let _voice = null; // {stream, ctx, source, node, chunks, rate, t0, timer} — 非空 = 录音中
|
||||
|
||||
// 线性插值降采样到 16k 并量化 Int16 小端(讯飞只收 16k/16bit/mono raw)
|
||||
function _pcm16kFromChunks(chunks, srcRate) {
|
||||
let total = 0;
|
||||
for (const c of chunks) total += c.length;
|
||||
const src = new Float32Array(total);
|
||||
let off = 0;
|
||||
for (const c of chunks) { src.set(c, off); off += c.length; }
|
||||
const ratio = srcRate / 16000;
|
||||
const out = new Int16Array(Math.floor(src.length / ratio));
|
||||
for (let i = 0; i < out.length; i++) {
|
||||
const pos = i * ratio, i0 = Math.floor(pos), i1 = Math.min(i0 + 1, src.length - 1);
|
||||
const s = src[i0] + (src[i1] - src[i0]) * (pos - i0);
|
||||
out[i] = Math.max(-1, Math.min(1, s)) * 0x7fff;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function startVoiceInput() {
|
||||
const btn = $("chat-mic");
|
||||
if (!(navigator.mediaDevices && navigator.mediaDevices.getUserMedia)) {
|
||||
$("chat-hint").textContent = "此环境不支持录音(需 HTTPS 或 localhost 访问)";
|
||||
return;
|
||||
}
|
||||
let stream;
|
||||
try {
|
||||
stream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
|
||||
});
|
||||
} catch (e) {
|
||||
$("chat-hint").textContent = "无法使用麦克风:" +
|
||||
(e.name === "NotAllowedError" ? "请在浏览器里允许麦克风权限" : e.message);
|
||||
return;
|
||||
}
|
||||
const ctx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
const source = ctx.createMediaStreamSource(stream);
|
||||
// ScriptProcessor 已标废弃但全浏览器可用;AudioWorklet 要独立模块文件,纯静态 JS 下不值当
|
||||
const node = ctx.createScriptProcessor(4096, 1, 1);
|
||||
const chunks = [];
|
||||
node.onaudioprocess = (ev) => { chunks.push(new Float32Array(ev.inputBuffer.getChannelData(0))); };
|
||||
source.connect(node);
|
||||
node.connect(ctx.destination); // 不接 destination 部分浏览器不触发 onaudioprocess;输出恒为静音无回声
|
||||
_voice = { stream, ctx, source, node, chunks, rate: ctx.sampleRate, t0: Date.now(), timer: 0 };
|
||||
btn.classList.add("recording");
|
||||
const tick = () => {
|
||||
if (!_voice) return;
|
||||
const sec = Math.floor((Date.now() - _voice.t0) / 1000);
|
||||
btn.textContent = `■ 停止 ${Math.floor(sec / 60)}:${String(sec % 60).padStart(2, "0")}`;
|
||||
if (sec >= ASR_MAX_SECONDS) { stopVoiceInput(); return; }
|
||||
_voice.timer = setTimeout(tick, 500);
|
||||
};
|
||||
tick();
|
||||
$("chat-hint").textContent = "录音中…再点一次结束并转写(最长 60 秒)";
|
||||
}
|
||||
|
||||
async function stopVoiceInput() {
|
||||
const v = _voice;
|
||||
if (!v) return;
|
||||
_voice = null;
|
||||
clearTimeout(v.timer);
|
||||
const btn = $("chat-mic");
|
||||
btn.classList.remove("recording");
|
||||
btn.textContent = "🎙 语音";
|
||||
try { v.source.disconnect(); v.node.disconnect(); } catch (e) {}
|
||||
v.stream.getTracks().forEach((t) => t.stop());
|
||||
try { await v.ctx.close(); } catch (e) {}
|
||||
const pcm = _pcm16kFromChunks(v.chunks, v.rate);
|
||||
if (pcm.length < 16000 / 4) { // < 0.25s:多半是误触
|
||||
$("chat-hint").textContent = "录音太短,已取消";
|
||||
return;
|
||||
}
|
||||
btn.disabled = true;
|
||||
$("chat-hint").textContent = "转写中…";
|
||||
try {
|
||||
const r = await fetch("/v1/asr/transcribe", {
|
||||
method: "POST",
|
||||
headers: { "Authorization": "Bearer " + state.token, "Content-Type": "application/octet-stream" },
|
||||
body: pcm.buffer,
|
||||
});
|
||||
if (r.status === 401) { logout(); return; }
|
||||
const data = await r.json().catch(() => null);
|
||||
if (!r.ok) throw new Error((data && data.detail) || r.status + " " + r.statusText);
|
||||
const text = ((data && data.text) || "").trim();
|
||||
if (!text) { $("chat-hint").textContent = "没有识别到内容,请重试"; return; }
|
||||
const ta = $("chat-input");
|
||||
ta.focus();
|
||||
// 光标处插入(不覆盖已有草稿);execCommand 接 textarea 原生 undo 栈,Ctrl+Z 可撤销
|
||||
const ok = document.execCommand("insertText", false, text);
|
||||
if (!ok) ta.value += text;
|
||||
syncOptimizeBtn();
|
||||
$("chat-hint").textContent = "已转写,确认后发送";
|
||||
} catch (e) {
|
||||
$("chat-hint").textContent = "转写失败:" + e.message;
|
||||
} finally {
|
||||
btn.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
$("chat-mic").onclick = () => { if (_voice) stopVoiceInput(); else startVoiceInput(); };
|
||||
|
||||
// 对话流里 artifact chip / 内联 img 点击委托 — 复用右栏文件预览 modal(modal 内自带"下载")。
|
||||
// 视频走原生 <video controls>:点击=播放/暂停,全屏走浏览器自带按钮,不进 modal —
|
||||
// 弹个 modal 反而打断播放,不如交给浏览器。
|
||||
|
|
|
|||
Loading…
Reference in New Issue