diff --git a/PROGRESS.md b/PROGRESS.md index 76dc642..8a6010d 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -23,6 +23,7 @@ ### 2026-07 +- **07-07 / 0.47.0**:web 端语音输入(讯飞语音听写 IAT)——聊天输入区新增「🎙 语音」按钮:点击录音(Web Audio ScriptProcessor 采 PCM → 前端线性插值降采样 16k/16bit/mono)→ 再点停止 → `POST /v1/asr/transcribe`(裸 PCM body,require_user)→ 后端 `core/asr_xfyun.py` 连讯飞 wss(HMAC-SHA256 签名,9600B/帧 40ms 节奏,vad_eos=10000 防句间停顿截尾,未开 dwa 整段拼接)→ 文本 insertText 插输入框光标处(接原生 undo 栈,**不自动发送**)。60s 上限自动停(讯飞单会话限制);录音需 HTTPS/localhost;渠道只读镜像 task 按钮锁定;`XFYUN_APPID/API_KEY/API_SECRET` env 未配返 501(**当前 .env 为测试 key,待换正式**)。诊断 `scripts/diag_asr.py`(静音冒烟 / 带 PCM 文件真测);requirements 显式加 `websockets`。已实测:TTS 合成中文 5.2s 音频经端点转写逐字正确。企微语音消息(AMR 需 ffmpeg 转码)留待二期;将来流式逐字上屏可复用 `build_auth_url` 签临时 URL。 - **07-07 / 0.46.2**:消息分页窗口对齐轮次边界(纯前端,`chat.js`)——原按固定 30 条翻页/跳转,窗口边界常切在 assistant 工具调用序列中间(顶部出现孤儿 tool 消息),点目录圆点用 `before_idx=idx+11` 启发式(长轮拉不全、短轮混无关上文)。业界调研确认:分页单位保持消息条数是共识(Discord/OpenAI 均按条 + 每页上限,轮次大小无上界不能当传输单位),轮次归组属渲染层职责。实现:idx 连续 + outline 已含全部 user 轮次起点 → 前端可精确算区间,后端 API 零改动。① 向上翻页 `alignedEarlierLimit`:从窗口顶往前按整轮累加攒够 30 条,新窗口顶恒为 user 消息;② 圆点跳转:`after_idx=起点-1` + 相邻边界差精确拉整轮(短轮补到 30 铺满、超大轮按 `MSG_FETCH_MAX=100` 截断,后续向上加载逐步自愈对齐);③ outline 未加载/idx 不在 outline → 回退旧按条行为。后端 limit 无 clamp,前端以 100 硬上限自律。 - **07-07 / 0.46.1**:撤除企微欢迎推送机制——实际部署采用聊天优先布局(不配应用主页 + 菜单「工作台」按钮)后,点应用直进会话,冷启动问题不复存在,0.46.0 的欢迎推送成了死重:删 `wecom_welcomes` 表(migration 0021 撤下,DB 已 downgrade 回 0020,表建后无一行写入)、`maybe_send_wecom_welcome`、免登回调欢迎钩子、群发脚本、`list_visible_userids`。**保留**:未绑定入站回绑定指引(聊天优先下必需)、`core/branding.py` 品牌单一源。RUN.md 企微段只留推荐布局 + 未绑定指引两条。 - **07-07 / 0.46.0**:企业微信会话冷启动 + 未绑定入站回指引——背景:应用配了「应用主页」后点应用只进主页,应用会话要等应用发过第一条消息才出现在成员消息列表(首次登录的员工没入口和 bot 打字)。最终采用**聊天优先布局**(企微后台不配主页、菜单放「工作台」按钮指向 `/v1/wecom/entry`,点应用直进会话),代码侧:① 回调对**未绑定成员**的 text/图片/文件消息不再静默,每条回一句绑定指引(聊天优先下新员工上来就打字,静默会被当成 bot 坏了;事件不回);② 新表 `wecom_welcomes`(0021,wecom_userid PK,未绑定成员也能去重)+ `service.maybe_send_wecom_welcome`(两层去重:已推过 / 已绑定且有 chat task;claim-first INSERT 并发安全,发送失败释放占坑),免登入口回调对首次进入的成员自动推欢迎语(菜单「工作台」/主页两条路都触发);③ 可选工具 `scripts/wecom_welcome_broadcast.py`:配主页布局下给存量成员批量"种"会话入口(`--dry-run` 先看名单;可见范围枚举 `wecom.list_visible_userids`:agent/get + 部门/标签展开);④ 品牌名单一事实源挪到 `core/branding.py`(web 层外的欢迎语也要用,app.py BRAND 引它)。RUN.md 企微段同步推荐布局。 @@ -207,6 +208,7 @@ core/memory.py 81 ← per-user `.memory/` dotfile core/export_docx.py 383 core/storage/{__init__,engine,models,usage,utils}.py ← 4 表(0004-0007 演进);record_chat/image_usage core/ark_client.py 105 ← 火山方舟 HTTP 客户端 +core/asr_xfyun.py 170 ← 讯飞语音听写 IAT wss 客户端(整段 PCM→文本;web 语音输入用,diag: scripts/diag_asr.py) core/agent_builder.py 340 ← 装配 lib(有 ARK_API_KEY 才挂 SeedreamTool);build_skill_registry 装两来源 core/executor.py / sandbox/{network,pool}.py / executor_docker.py ← Executor ABC + Docker per-user 容器池 tools/{base,fs,shell,run_python,skill_tool,skill_authoring,seedream,seedance,web_search,web_fetch,documents,materials_project}.py ← skill_authoring=save_skill/fork_skill(host-side 写 user .skills) diff --git a/RUN.md b/RUN.md index dcf5c50..febe8f0 100644 --- a/RUN.md +++ b/RUN.md @@ -85,6 +85,12 @@ # 用户即可在企业微信里直接和 zcbot 对话(回调 URL = <公网 base>/v1/wecom/callback)。 # WECOM_CALLBACK_TOKEN=... # 接收消息 Token(企微后台生成) # WECOM_CALLBACK_AESKEY=... # EncodingAESKey(43 字符,企微后台生成) + # 讯飞语音听写(web 端「🎙 语音」麦克风输入,core/asr_xfyun.py):可选,三件套齐才可用, + # 未配时点按钮转写返 501。控制台 https://console.xfyun.cn/ 开通「语音听写(流式版)」取凭据。 + # 验证链路:`.venv/Scripts/python.exe scripts/diag_asr.py`(静音冒烟)或带 PCM 文件参数真测。 + # XFYUN_APPID=... + # XFYUN_API_KEY=... + # XFYUN_API_SECRET=... # 对外品牌名:zcbot 是内部代号,所有用户可见文案(页面标题/顶栏/登录卡/微信·企微推送与 # 提示页)统一用品牌名。/healthz 返回 brand 字段,前端 boot 拉取覆盖静态页默认值。 # ZCBOT_BRAND_NAME=总院科研辅助助手 # 可选,默认即此值 @@ -226,6 +232,7 @@ curl --noproxy '*' -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8765/v1/ta | `POST /v1/tasks/{id}/cancel` | 协作式 cancel;`run_status != running` → 409;LLM 走 streaming,chunk 间 poll cancel — 延迟 100ms 级,基本秒退 | 必填 | | `POST /v1/tasks/{id}/clear` | 清空当前 task 全部 messages + reset `tasks.tokens_prompt/completion/cost_cny` 三列累计 + `run_status='idle'`;`usage_events`(账单记账)**不动**,只 `message_id` 列变 NULL;run 活跃中(running/cancelling)→ 409(先 cancel);FS 文件保留 | 必填 | | `POST /v1/tasks/{id}/optimize_prompt` | body `{text(req, ≤4000), image_model?=""}`;同步调当前 task model 润色草稿,返 `{optimized, model_profile, tokens_in, tokens_out, cost_cny}`;**不**写 messages、**不**累计 task 三列(顶栏数字不污染),只在 `usage_events` 写一行 `kind="prompt_optimize"`(对账可见);不与主对话 run 互斥(允许 streaming 中并行润色) | 必填 | +| `POST /v1/asr/transcribe` | body 为裸 PCM(16kHz/16bit/单声道/小端,`application/octet-stream`)→ 讯飞 IAT 整段转写,返 `{text}`;>60s → 413;`XFYUN_*` env 未配 → 501;讯飞侧错误 → 502(带错误码提示)。前端「🎙 语音」按钮用 | 必填 | | `GET /v1/files?path=` | 列 user_root 下条目 + 面包屑;dotfile 隐藏 | 必填 | | `GET /v1/files/download?path=` | 下单文件 | 必填 | | `POST /v1/files/upload` | multipart 上传到 `//`;路径不存在自动 mkdir,重名覆盖 | 必填 | @@ -816,6 +823,7 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_" /opt | Export 报 "无可导出内容" | task 没 messages(只 system 不算);先发条消息再 export | | `NoSubtaskError: working_dir ... 前缀嵌套` | §7.4 no-subtask:同 user 不允许 working_dir 嵌套(child / parent)。**同项目多对话**用**完全相同**的 working_dir;否则改成 sibling(平级) | | `main.py web` 启动后 curl 连不上 | 检查 proxy(`HTTP_PROXY` / `HTTPS_PROXY`):本地服务 127.0.0.1,系统 proxy 拦截会 502。临时 `unset HTTP_PROXY HTTPS_PROXY` 或 `curl --noproxy '*'`。验通:`curl --noproxy '*' http://127.0.0.1:8765/healthz` | +| 「🎙 语音」点了没反应 / 提示"此环境不支持录音" | 浏览器只在**安全上下文**(HTTPS 或 localhost)开放麦克风;http://IP 访问必现。上 HTTPS 或本机 localhost 试。凭据/链路问题跑 `scripts/diag_asr.py` 定位(11200=讯飞服务未开通或量用完,10105/10313=appid 错) | | SSE 卡住不流(经 nginx) | 反代要关 buffering — 后端响应头已带 `X-Accel-Buffering: no`(nginx ≥1.5.6 默认认)。仍卡看 nginx 配 `proxy_buffering off; proxy_read_timeout 3600s;` | | platform CORS preflight 失败 | 本地 dev `allow_origins=["*"]` 应该没事;部署后看是否按 platform 域名收紧过头 | | `POST /v1/tasks/{id}/messages` 返 409 `task already has an active run` | 上一条消息的 BG run 还没跑完;等流式 done 或点 stop / `POST .../cancel`;服务异常下 `run_status` 卡 `running`/`cancelling`,启动 reaper 会清 | diff --git a/core/__init__.py b/core/__init__.py index 4a159a8..57e690b 100644 --- a/core/__init__.py +++ b/core/__init__.py @@ -1,3 +1,3 @@ # zcbot 版本号单一事实源:web/app.py 的 FastAPI version、/healthz 返回、前端展示都引这里。 # 改版本只动这一行。 -__version__ = "0.46.2" +__version__ = "0.47.0" diff --git a/core/asr_xfyun.py b/core/asr_xfyun.py new file mode 100644 index 0000000..e8e7143 --- /dev/null +++ b/core/asr_xfyun.py @@ -0,0 +1,157 @@ +"""讯飞语音听写(IAT,流式版)WebSocket 客户端 — 整段音频转写。 + +接口文档: https://www.xfyun.cn/doc/asr/voicedictation/API.html +- 鉴权: HMAC-SHA256 签名拼进 wss URL query(authorization/date/host) +- 音频: 仅收 16kHz/16bit/单声道 PCM(raw),单会话上限 60 秒 +- 凭据: env XFYUN_APPID / XFYUN_API_KEY / XFYUN_API_SECRET(换正式 key 只改 .env) + +本模块只做"整段 PCM → 最终文本"(web 语音输入按钮用);未开 dwa 动态修正 —— +那是实时逐字上屏才需要的,整段转写直接按序拼接 result.ws 即可。将来做流式 +上屏时复用 build_auth_url() 签临时 URL 给前端直连。 +""" +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import hmac +import json +import os +import time +from contextlib import suppress +from urllib.parse import urlencode +from wsgiref.handlers import format_date_time + +XFYUN_HOST = "iat-api.xfyun.cn" +XFYUN_PATH = "/v2/iat" + +SAMPLE_RATE = 16000 +BYTES_PER_SECOND = SAMPLE_RATE * 2 # 16bit mono +MAX_SECONDS = 60 # 讯飞单会话硬上限 +MAX_PCM_BYTES = MAX_SECONDS * BYTES_PER_SECOND + +# 每帧 9600B(=300ms 音频),base64 后 12800B < 讯飞单帧上限 13000B; +# 帧间 40ms 为文档推荐节奏,整段上传比实时快 7.5 倍(60s 音频 ≈ 8s 传完) +_FRAME_BYTES = 9600 +_FRAME_INTERVAL = 0.04 +_RECV_TIMEOUT = 30 # 单次等响应上限;讯飞 10s 无数据也会主动断 + +# 常见错误码 → 给用户/运维看的中文提示(全量见文档错误码页) +_ERR_HINTS = { + 10105: "appid 不合法(检查 XFYUN_APPID)", + 10107: "参数值非法", + 10110: "无授权许可(服务未开通或已过期)", + 10114: "会话超时(单次音频超 60 秒)", + 10163: "业务参数校验失败", + 10165: "会话句柄无效", + 10313: "appid 不能为空或与 apikey 不匹配", + 11200: "无权限或用量已用完(检查讯飞控制台服务量)", + 11201: "当日流控超限", +} + + +class XfyunASRError(RuntimeError): + """转写失败(讯飞返回错误码 / 连接异常),code 为讯飞错误码(连接类异常为 None)。""" + + def __init__(self, message: str, code: int | None = None): + super().__init__(message) + self.code = code + + +class XfyunASRNotConfigured(XfyunASRError): + """XFYUN_* env 未配置 — 上层应返回 501 而非 502。""" + + +def build_auth_url(api_key: str, api_secret: str) -> str: + """按讯飞规范生成带签名的 wss URL(date 用 RFC1123 GMT)。""" + date = format_date_time(time.time()) + signature_origin = f"host: {XFYUN_HOST}\ndate: {date}\nGET {XFYUN_PATH} HTTP/1.1" + signature = base64.b64encode( + hmac.new(api_secret.encode(), signature_origin.encode(), hashlib.sha256).digest() + ).decode() + authorization_origin = ( + f'api_key="{api_key}", algorithm="hmac-sha256", ' + f'headers="host date request-line", signature="{signature}"' + ) + authorization = base64.b64encode(authorization_origin.encode()).decode() + query = urlencode({"authorization": authorization, "date": date, "host": XFYUN_HOST}) + return f"wss://{XFYUN_HOST}{XFYUN_PATH}?{query}" + + +def _load_credentials() -> tuple[str, str, str]: + appid = (os.getenv("XFYUN_APPID") or "").strip() + api_key = (os.getenv("XFYUN_API_KEY") or "").strip() + api_secret = (os.getenv("XFYUN_API_SECRET") or "").strip() + if not (appid and api_key and api_secret): + raise XfyunASRNotConfigured( + "语音识别未配置:需在 .env 设 XFYUN_APPID / XFYUN_API_KEY / XFYUN_API_SECRET" + ) + return appid, api_key, api_secret + + +async def transcribe(pcm: bytes, *, language: str = "zh_cn") -> str: + """整段 PCM(16k/16bit/mono, 小端)→ 识别文本。空音频/纯静音返回 ""。""" + appid, api_key, api_secret = _load_credentials() + if not pcm: + return "" + if len(pcm) > MAX_PCM_BYTES: + raise XfyunASRError(f"音频超长:最多 {MAX_SECONDS} 秒") + + import websockets # uvicorn[standard] 附带;requirements 亦显式声明 + + audio_meta = {"format": f"audio/L16;rate={SAMPLE_RATE}", "encoding": "raw"} + + async def _send_frames(ws) -> None: + first = True + for off in range(0, len(pcm), _FRAME_BYTES): + chunk = base64.b64encode(pcm[off : off + _FRAME_BYTES]).decode() + frame: dict = {"data": {"status": 1 if not first else 0, **audio_meta, "audio": chunk}} + if first: + frame["common"] = {"app_id": appid} + # vad_eos 拉到上限 10s:整段转写不希望句间停顿被判"说完了"截尾 + frame["business"] = { + "language": language, + "domain": "iat", + "accent": "mandarin", + "vad_eos": 10000, + "ptt": 1, + } + first = False + await ws.send(json.dumps(frame)) + await asyncio.sleep(_FRAME_INTERVAL) + await ws.send(json.dumps({"data": {"status": 2, **audio_meta, "audio": ""}})) + + pieces: list[str] = [] + try: + async with websockets.connect(build_auth_url(api_key, api_secret), open_timeout=10) as ws: + send_task = asyncio.create_task(_send_frames(ws)) + try: + while True: + raw = await asyncio.wait_for(ws.recv(), timeout=_RECV_TIMEOUT) + msg = json.loads(raw) + code = msg.get("code") + if code: + hint = _ERR_HINTS.get(code, msg.get("message") or "未知错误") + raise XfyunASRError(f"讯飞识别失败({code}):{hint}", code=code) + data = msg.get("data") or {} + result = data.get("result") or {} + pieces.append( + "".join( + cw.get("w", "") + for w in (result.get("ws") or []) + for cw in (w.get("cw") or []) + ) + ) + if data.get("status") == 2: + break + finally: + send_task.cancel() + with suppress(asyncio.CancelledError): + await send_task + except XfyunASRError: + raise + except asyncio.TimeoutError: + raise XfyunASRError("讯飞识别超时(30s 无响应)") + except Exception as e: # 握手 401/连接断开等 + raise XfyunASRError(f"讯飞连接失败:{type(e).__name__}: {e}") + return "".join(pieces).strip() diff --git a/requirements.txt b/requirements.txt index 288406c..25c97dd 100644 --- a/requirements.txt +++ b/requirements.txt @@ -20,6 +20,9 @@ markitdown[pdf,docx,pptx,xlsx]>=0.0.1 httpx>=0.27.0 html2text>=2024.0 +# 语音听写(core/asr_xfyun.py 连讯飞 IAT wss;uvicorn[standard] 也附带,这里显式声明直接依赖) +websockets>=12.0 + # 定时任务(§8.5 scheduled_jobs):cron 串 → next_run_at 计算,正确处理 dom/dow OR 语义 + 时区 croniter>=2.0 diff --git a/scripts/diag_asr.py b/scripts/diag_asr.py new file mode 100644 index 0000000..0b8a6f5 --- /dev/null +++ b/scripts/diag_asr.py @@ -0,0 +1,67 @@ +"""讯飞语音听写(IAT)诊断:验证 XFYUN_* 凭据 + wss 协议链路是否通。 + +用法(.env 在仓库根): + .venv/Scripts/python.exe scripts/diag_asr.py # 2s 静音冒烟(验鉴权/协议,预期空文本) + .venv/Scripts/python.exe scripts/diag_asr.py # 真实音频(16k/16bit/mono 裸 PCM) + +ASCII 输出(Windows 控制台 GBK,不打 emoji);secret 不打印。 +常见错误码:10105/10313=appid 错;11200=服务未开通或量用完;11201=当日超限。 +""" +import asyncio +import os +import sys + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if _ROOT not in sys.path: + sys.path.insert(0, _ROOT) + + +def _load_env(path: str) -> None: + try: + from dotenv import load_dotenv + load_dotenv(path) + return + except Exception: + pass + try: + with open(path, encoding="utf-8") as f: + for line in f: + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + k, v = line.split("=", 1) + os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'")) + except FileNotFoundError: + pass + + +_load_env(os.path.join(_ROOT, ".env")) + +from core.asr_xfyun import SAMPLE_RATE, XfyunASRError, transcribe # noqa: E402 + + +def main() -> int: + appid = os.getenv("XFYUN_APPID", "") + print(f"[env] XFYUN_APPID={appid or '(missing)'} " + f"API_KEY={'set' if os.getenv('XFYUN_API_KEY') else '(missing)'} " + f"API_SECRET={'set' if os.getenv('XFYUN_API_SECRET') else '(missing)'}") + + if len(sys.argv) > 1: + pcm = open(sys.argv[1], "rb").read() + print(f"[audio] file={sys.argv[1]} bytes={len(pcm)} " + f"({len(pcm) / (SAMPLE_RATE * 2):.1f}s @16k/16bit/mono)") + else: + pcm = b"\x00" * (SAMPLE_RATE * 2 * 2) # 2s 静音:只验鉴权+协议,预期识别为空 + print("[audio] builtin 2s silence (auth/protocol smoke, empty text expected)") + + try: + text = asyncio.run(transcribe(pcm)) + except XfyunASRError as e: + print(f"[FAIL] code={e.code} {e}") + return 1 + print(f"[OK] text={text!r}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/web/app.py b/web/app.py index 5ff55f8..f72b650 100644 --- a/web/app.py +++ b/web/app.py @@ -2850,6 +2850,39 @@ def create_app() -> FastAPI: "cost_cny": float(cost_cny), } + # ───────────── ASR 语音转写 ───────────── + + @app.post("/v1/asr/transcribe", tags=["asr"]) + async def asr_transcribe( + request: Request, + user_id: UUID = Depends(require_user), + ): + """整段语音转写:body 为 raw PCM(16kHz/16bit/单声道/小端),返回 {"text"}。 + + 前端麦克风按钮录音(Web Audio 采样降到 16k)→ 本端点 → 讯飞 IAT websocket + (core/asr_xfyun.py)。上限 60 秒(讯飞单会话限制);XFYUN_* env 未配 → 501。 + """ + from core.asr_xfyun import ( + MAX_PCM_BYTES, + MAX_SECONDS, + XfyunASRError, + XfyunASRNotConfigured, + transcribe, + ) + + pcm = await request.body() + if not pcm: + raise HTTPException(400, "empty audio") + if len(pcm) > MAX_PCM_BYTES: + raise HTTPException(413, f"音频超长:单次最多 {MAX_SECONDS} 秒") + try: + text = await transcribe(pcm) + except XfyunASRNotConfigured as e: + raise HTTPException(501, str(e)) + except XfyunASRError as e: + raise HTTPException(502, str(e)) + return {"text": text} + # ───────────── SSE events ───────────── @app.get("/v1/tasks/{task_id}/events", tags=["tasks"]) diff --git a/web/static/dev.html b/web/static/dev.html index 2995e9e..eee12ed 100644 --- a/web/static/dev.html +++ b/web/static/dev.html @@ -890,6 +890,9 @@ #chat-form textarea { flex: 1; } /* 微信渠道 task 只读镜像:输入框置灰禁手输,引导去微信对话 */ #chat-input.readonly-locked { background: #f0f0f0; color: var(--muted); cursor: not-allowed; } + /* 语音输入按钮:录音中红色呼吸提醒(别忘了停) */ + #chat-mic.recording { color: #fff; background: #d33; border-color: #d33; animation: mic-pulse 1.6s ease-in-out infinite; } + @keyframes mic-pulse { 50% { opacity: .65; } } #chat-form .hint { font-size: 11px; color: var(--muted); } /* 附件托盘:粘贴 / 拖拽的文件 chip 独占一行,累积展示,不与状态文字抢容器 */ #chat-attach { display: none; flex-wrap: wrap; gap: 4px 0; align-items: center; } @@ -1564,6 +1567,7 @@
就绪 +
diff --git a/web/static/js/chat.js b/web/static/js/chat.js index dec2a82..6435808 100644 --- a/web/static/js/chat.js +++ b/web/static/js/chat.js @@ -1406,6 +1406,8 @@ function applyChannelComposerLock(meta) { const opt = $("chat-optimize"); if (opt) opt.disabled = true; } + const mic = $("chat-mic"); + if (mic) mic.disabled = !!cfg; // 只读镜像没有发送入口,语音输入同样锁掉 } // 润色:同步调后端,把 textarea 内容替成优化后文本。用 execCommand('insertText') @@ -1465,6 +1467,113 @@ async function optimizePrompt() { $("chat-optimize").onclick = optimizePrompt; +// ── 语音输入(讯飞 IAT 整段转写)────────────────────────────────────────────── +// 点击开始录音(Web Audio 采 PCM → 降采样 16k/16bit)→ 再点停止 → POST +// /v1/asr/transcribe → 文本插到输入框光标处(insertText 接原生 undo 栈,不自动 +// 发送,用户确认后再发)。录音需安全上下文(HTTPS / localhost);60s 自动停 +// (讯飞单会话上限)。 +const ASR_MAX_SECONDS = 60; +let _voice = null; // {stream, ctx, source, node, chunks, rate, t0, timer} — 非空 = 录音中 + +// 线性插值降采样到 16k 并量化 Int16 小端(讯飞只收 16k/16bit/mono raw) +function _pcm16kFromChunks(chunks, srcRate) { + let total = 0; + for (const c of chunks) total += c.length; + const src = new Float32Array(total); + let off = 0; + for (const c of chunks) { src.set(c, off); off += c.length; } + const ratio = srcRate / 16000; + const out = new Int16Array(Math.floor(src.length / ratio)); + for (let i = 0; i < out.length; i++) { + const pos = i * ratio, i0 = Math.floor(pos), i1 = Math.min(i0 + 1, src.length - 1); + const s = src[i0] + (src[i1] - src[i0]) * (pos - i0); + out[i] = Math.max(-1, Math.min(1, s)) * 0x7fff; + } + return out; +} + +async function startVoiceInput() { + const btn = $("chat-mic"); + if (!(navigator.mediaDevices && navigator.mediaDevices.getUserMedia)) { + $("chat-hint").textContent = "此环境不支持录音(需 HTTPS 或 localhost 访问)"; + return; + } + let stream; + try { + stream = await navigator.mediaDevices.getUserMedia({ + audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true }, + }); + } catch (e) { + $("chat-hint").textContent = "无法使用麦克风:" + + (e.name === "NotAllowedError" ? "请在浏览器里允许麦克风权限" : e.message); + return; + } + const ctx = new (window.AudioContext || window.webkitAudioContext)(); + const source = ctx.createMediaStreamSource(stream); + // ScriptProcessor 已标废弃但全浏览器可用;AudioWorklet 要独立模块文件,纯静态 JS 下不值当 + const node = ctx.createScriptProcessor(4096, 1, 1); + const chunks = []; + node.onaudioprocess = (ev) => { chunks.push(new Float32Array(ev.inputBuffer.getChannelData(0))); }; + source.connect(node); + node.connect(ctx.destination); // 不接 destination 部分浏览器不触发 onaudioprocess;输出恒为静音无回声 + _voice = { stream, ctx, source, node, chunks, rate: ctx.sampleRate, t0: Date.now(), timer: 0 }; + btn.classList.add("recording"); + const tick = () => { + if (!_voice) return; + const sec = Math.floor((Date.now() - _voice.t0) / 1000); + btn.textContent = `■ 停止 ${Math.floor(sec / 60)}:${String(sec % 60).padStart(2, "0")}`; + if (sec >= ASR_MAX_SECONDS) { stopVoiceInput(); return; } + _voice.timer = setTimeout(tick, 500); + }; + tick(); + $("chat-hint").textContent = "录音中…再点一次结束并转写(最长 60 秒)"; +} + +async function stopVoiceInput() { + const v = _voice; + if (!v) return; + _voice = null; + clearTimeout(v.timer); + const btn = $("chat-mic"); + btn.classList.remove("recording"); + btn.textContent = "🎙 语音"; + try { v.source.disconnect(); v.node.disconnect(); } catch (e) {} + v.stream.getTracks().forEach((t) => t.stop()); + try { await v.ctx.close(); } catch (e) {} + const pcm = _pcm16kFromChunks(v.chunks, v.rate); + if (pcm.length < 16000 / 4) { // < 0.25s:多半是误触 + $("chat-hint").textContent = "录音太短,已取消"; + return; + } + btn.disabled = true; + $("chat-hint").textContent = "转写中…"; + try { + const r = await fetch("/v1/asr/transcribe", { + method: "POST", + headers: { "Authorization": "Bearer " + state.token, "Content-Type": "application/octet-stream" }, + body: pcm.buffer, + }); + if (r.status === 401) { logout(); return; } + const data = await r.json().catch(() => null); + if (!r.ok) throw new Error((data && data.detail) || r.status + " " + r.statusText); + const text = ((data && data.text) || "").trim(); + if (!text) { $("chat-hint").textContent = "没有识别到内容,请重试"; return; } + const ta = $("chat-input"); + ta.focus(); + // 光标处插入(不覆盖已有草稿);execCommand 接 textarea 原生 undo 栈,Ctrl+Z 可撤销 + const ok = document.execCommand("insertText", false, text); + if (!ok) ta.value += text; + syncOptimizeBtn(); + $("chat-hint").textContent = "已转写,确认后发送"; + } catch (e) { + $("chat-hint").textContent = "转写失败:" + e.message; + } finally { + btn.disabled = false; + } +} + +$("chat-mic").onclick = () => { if (_voice) stopVoiceInput(); else startVoiceInput(); }; + // 对话流里 artifact chip / 内联 img 点击委托 — 复用右栏文件预览 modal(modal 内自带"下载")。 // 视频走原生