diff --git a/PROGRESS.md b/PROGRESS.md index 1667cc8..07f95d1 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,7 +2,7 @@ > 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 1-2 句:做了啥 + 关键判断;细节查 `git log` / `git diff` / `DESIGN §7.9`。 -最后更新:2026-07-10(后台进程 bg proc + seedance 续查 + LLM 显式超时,bump 0.56.0) +最后更新:2026-07-11(sandbox pids-limit 256→1024 修 mermaid 渲染必崩,bump 0.58.2) --- @@ -23,7 +23,7 @@ ### 2026-07 -- **07-10 / 0.58.1**:**unifyllm 文本模型再加 2 个进 pro 档**——`gpt55`(gpt-5.5)/`gpt54_mini`(gpt-5.4-mini),参数照 gpt56_sol 模板(temp 硬约束 1.0),mini 档 tool_calling/code_quality 记 good、max_iterations 100;网关 `GET /v1/models` 确认模型 ID + diag 冒烟(tool-calling)两模型均过 + `ModelCapabilities.load` 线上路径验证;`diag_unifyllm.py` 默认清单 5→7 个(RUN.md 同步)。 +- **07-11 / 0.58.2**:**sandbox `pids_limit` 256→1024,修容器内 mermaid 渲染必崩**(task 9dcae061 排查):DB 全量扫描证实近 90 天 mmdc 渲图 0 成功(26 次 launch fail + 67 次 timeout),根因 `--pids-limit` 把线程也计数——chromium headless 一次启动 ~150-200 线程,叠加 shell 超时只杀 host 侧 docker CLI、容器内 mmdc+chromium 树残留(executor_docker 已知 Cancel limitation),256 打满 → `pthread_create EAGAIN` → chromium SIGTRAP(dbus 报错是干扰项)。改 agent.yaml + pool.py 默认值两处;RUN.md 故障兜底补条目。**待服务器验证**:重启 web + 容器回收后跑 `probe_mermaid.sh`,若 chromium 150(07-08 镜像重建带进)仍秒崩则另查宿主 docker/seccomp;彻底修法是 Step 3b PGID kill 协议(超时清残留),此次未动。附带 4 个 diag 脚本(scripts/diag_task_9dcae061.py 等)未入库。**unifyllm 文本模型再加 2 个进 pro 档**——`gpt55`(gpt-5.5)/`gpt54_mini`(gpt-5.4-mini),参数照 gpt56_sol 模板(temp 硬约束 1.0),mini 档 tool_calling/code_quality 记 good、max_iterations 100;网关 `GET /v1/models` 确认模型 ID + diag 冒烟(tool-calling)两模型均过 + `ModelCapabilities.load` 线上路径验证;`diag_unifyllm.py` 默认清单 5→7 个(RUN.md 同步)。 - **07-10 / 0.58.0**:**第二图像后端 gpt_image(unifyllm 网关,pro 档)**——媒体 provider 从豆包单家泛化为多家:新 `config/media/unifyllm.yaml`(image 段 `gpt_image` variant,model=gpt-5.6-sol 渠道)+ 新 tool `tools/gpt_image.py`(仅 prompt 参数——网关实测忽略 size/quality,尺寸上游自动定,不支持 i2i;b64 直落 `figures/`,banner 协议同 seedream,价目未知期 cost 记 0、usage tokens 进 units 留对账)。装配泛化:`ArkConfig.load` 支持通用键名 `api_key_env`/`base_url`(ark_* 旧键兜底);`agent_builder._choose_image_variant` 跨 provider 选 image variant(豆包在前 = 默认仍 seedream_5,本 run 只挂选中的一个图像工具),`_MEDIA_TOOLS_BLOCK` 拆分段按实挂工具拼装(`_media_tools_block`,gpt_image 段自带"不支持改图/尺寸→建议切回 Seedream"红线);web `_list_media_variants` 扫 `config/media/*.yaml`(image/video 清单、admin 门控 catalog 同步泛化,variant key 跨文件唯一、撞名先读生效);`media.js` 三处白名单加 gpt_image(banner/产物 chip/摘要行);prompt 润色 hint 无 default_size 时不再谎报 2048。`gpt_image` 进 pro 档(与 seedream 共享 images_per_day 配额,kind=image 同口径)。手工验证 `scripts/test_gpt_image_manual.py`(variant 选择 4 例 + prompt 块 3 例 + 真调网关落图)全过,unittest 全量 201 过。 - **07-10 / 0.57.0**:**unifyllm 网关接入(5 个国际旗舰模型进 pro 档)**:新档案 `config/models/unifyllm.yaml`(fable5/opus48/sonnet46/gpt56_sol/gemini31_pro,litellm `openai/` 前缀 + `api_base=https://unifyllm.ai/v1` + `UNIFYLLM_API_KEY`,零代码改动),`agent.yaml model_tiers.pro` 加 5 条。实测坑(全记在 yaml 头注释):gpt-5.x 被 litellm 硬拦 temp≠1、gemini-3.1-pro temp=0.3 返回空 → 两者 `optimal_temperature: 1.0`;`parallel_tool_calls` 参数网关接受(claude 实测 2 并行);流式 + include_usage 5 模型全通。新 `scripts/diag_unifyllm.py`(--stream/--temp,列模型 GET /v1/models)。**部署注意:服务器直连 unifyllm.ai TLS 握手失败,需代理出口(HTTPS_PROXY),部署前先跑 diag 验证**。价格字段暂 0(网关价目未知,成本先记 0)。 - **07-10 / 0.56.0**:**后台进程 bg proc(DESIGN §8.12)**——解掉"长脚本被工具超时掐死 / 占死 run / 蓝绿切换陪葬"三连:`shell`/`run_python` 加 `background=true`(模型按预计时长自选,前台超时报错里提示改走后台;默认上限 7200s、cap 86400s、每用户并发 3 个 `ZCBOT_MAX_BG_PROCS`),新 `check_process` 工具查状态/日志尾部/kill。状态协议纯文件(`/.zcbot_procs///{proc.json,output.log,exit_code}`,dotfile 用户不可见,exit_code 出现即终态),无 DB 无队列组件。host 模式 detach 独立 wrapper(`core/proc_wrapper.py`,stdlib-only,限时+杀树+截尾 10MB);docker 模式**专用容器** `zcbot-proc-`(`pool.run_proc_container`,同款硬化/iptables init,product=proc + 无 instance label → 与 sandbox 的 idle reaper/shutdown_all 生命周期解耦,dockerd 托管扛蓝绿)。回收:check_process 见终态顺手 rm 容器 + web lifespan 每小时 `procs.sweep`(终态目录 7d TTL、exited 孤儿容器)。**前端可视 + 通知**:`GET /v1/procs`(用户级)+ `POST /v1/tasks/{id}/procs/{pid}/kill` 两端点(纯文件读取无 DB);`procs.js` 把 `[Background]` 工具结果卡活化(spinner+summary 跳秒+停止按钮,与前台工具卡同体验;历史重渲同样恢复活态/终态定格 exit·耗时)+ running→终态弹 toast(右下角,跨 task 也提醒、点击跳转对应任务)+ **对话锁**(bg proc 运行期 composer 发送→停止、Enter 拦截,观感与前台执行一致,完成的那次轮询解锁;锁仅前端,服务端不 409——停止入口须可达);轮询式(仅有 running proc 时 5s 一拉,触发点:登录/选 task/run 收尾/[Background] 工具结果)——不走 SSE,proc 完成时刻往往没有活跃 run。LLM 超时最终定为**显式化默认 600s + env 可调**(长工具已走后台,LLM 单调用无正当 600s 静默;被掐再调 `ZCBOT_LLM_TIMEOUT_S`)。手工验证 `scripts/test_bgproc_manual.py` 全过(win host 模式:启动/探活/exit 0/日志/kill 137/sweep),unittest 全量 201 过。同批带上:**seedance `resume_task_id` 续查**(超时/中断后拿 cgt_id 跳过提交直接续轮询,不过配额闸不重复计费;fast/pro poll_timeout 统一 1200s)+ **LLM 超时显式化**(`core/llm.py` 显式传 litellm timeout,env `ZCBOT_LLM_TIMEOUT_S` 可调,见后)。 diff --git a/RUN.md b/RUN.md index 3897717..4a9e7a6 100644 --- a/RUN.md +++ b/RUN.md @@ -848,6 +848,7 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_" /opt | 容器内 shell 写工作目录报 `Permission denied`(but `sandbox check` ⑤ HOST_UID aligned ok) | DockerExecutor 写死了 `--user 1000:1000` 不会自动跟 build 的 HOST_UID 同步(改 `--user zcbot` 后已修)。仍报错检查镜像内 `docker run --rm --entrypoint id zcbot-sandbox:latest zcbot` 输出 uid 是否 = `id -u $(whoami)` | | 容器内 mmdc 渲 mermaid 报 `Failed to launch chromium` / `No usable sandbox` | chromium 在 `--cap-drop=ALL` 下自家 setuid sandbox 起不来,要 `--no-sandbox`。镜像已落 `/sandbox/puppeteer-config.json` + 给 mmdc 套了 wrapper 自动 `-p` 注入 ── **裸调 `mmdc -i x -o y` 就该成**。仍跪:`docker exec ... cat /usr/local/bin/mmdc` 看 wrapper 在不在(老镜像没 rebuild 则没有);或显式 `mmdc -p /sandbox/puppeteer-config.json -i x -o y` | | 容器内 mmdc 渲图卡到 **timeout** 而非报错 | chromium 默认用 `/dev/shm`,docker 不传 `--shm-size` 时只 64MB → 起不来一直挂。已在 `pool.py` 给 `docker run` 加 `--shm-size`(默 512m,env `ZCBOT_SANDBOX_SHM_SIZE` / yaml `sandbox.shm_size`)。**已 running 的旧容器不会自动生效**,重启 web + 等 idle 回收(或 `docker rm -f zcbot-sandbox-`)后新起的才带。实测脚本 `deploy/sandbox/probe_mermaid.sh` | +| 容器内 mmdc 报 `Failed to launch the browser process: Code: null`(伴满屏 dbus 报错),wrapper / `-p` 配置都给了照崩;或直接跑 chromium `Trace/breakpoint trap (core dumped)` | **`--pids-limit` 打满**(线程也计数):chromium headless 一次启动 ~150-200 线程,加上 shell 超时残留在容器里的 mmdc+chromium 树(超时只杀 host 侧 docker CLI),256 秒穿 → `pthread_create: Resource temporarily unavailable (11)` → chromium CHECK 崩(SIGTRAP)。2026-07-11 起默认已调 1024(yaml `sandbox.pids_limit` / env `ZCBOT_SANDBOX_PIDS_LIMIT`)。**已 running 的旧容器不生效**,重启 web + 等 idle 回收(或 `docker rm -f zcbot-sandbox-`)。dbus 报错是干扰项(成功时也打);验证跑 `deploy/sandbox/probe_mermaid.sh`,仍崩则查宿主 docker 版本 / seccomp | | 模型不渲本地 mmdc、反复试 `mermaid.ink` 等在线渲图服务还失败 | 容器**有外网**(bridge+NAT),但 `mermaid.ink` 等**境外服务易被墙/不稳**,渲图不该依赖出站。docker backend 的 system prompt「运行环境」段(`agent_builder.py` 注入)已写明"渲图走本地 mmdc、别调在线服务";撞到多半是 prompt 没更新 / 跑在 host backend。渲 mermaid 一律 `mmdc -i x.md -o x.png` | | Export 报 "无可导出内容" | task 没 messages(只 system 不算);先发条消息再 export | | `NoSubtaskError: working_dir ... 前缀嵌套` | §7.4 no-subtask:同 user 不允许 working_dir 嵌套(child / parent)。**同项目多对话**用**完全相同**的 working_dir;否则改成 sibling(平级) | diff --git a/config/agent.yaml b/config/agent.yaml index 4a69e9a..621b114 100644 --- a/config/agent.yaml +++ b/config/agent.yaml @@ -71,7 +71,9 @@ shutdown: sandbox: memory: 2g # --memory (env: ZCBOT_SANDBOX_MEMORY) cpus: 1.0 # --cpus (env: ZCBOT_SANDBOX_CPUS) - pids_limit: 256 # --pids-limit (env: ZCBOT_SANDBOX_PIDS_LIMIT) + pids_limit: 1024 # --pids-limit (env: ZCBOT_SANDBOX_PIDS_LIMIT);线程也计数, + # chromium headless 一次 ~150-200 线程 + 超时残留进程,256 会被 + # 打满致 mmdc 渲 mermaid 必崩(pthread_create EAGAIN) shm_size: 512m # --shm-size (env: ZCBOT_SANDBOX_SHM_SIZE);chromium/mmdc 渲 mermaid 的 /dev/shm,默 64MB 不够会挂 # 容器 DNS server 显式配置(docker run --dns,容器 /etc/resolv.conf 直接写, # 绕过 docker daemon 上游 DNS 探测路径;腾讯云轻量 / 部分云上 daemon 探测 diff --git a/core/__init__.py b/core/__init__.py index 2264dc8..4a056eb 100644 --- a/core/__init__.py +++ b/core/__init__.py @@ -1,3 +1,3 @@ # zcbot 版本号单一事实源:web/app.py 的 FastAPI version、/healthz 返回、前端展示都引这里。 # 改版本只动这一行。 -__version__ = "0.58.1" +__version__ = "0.58.2" diff --git a/core/sandbox/pool.py b/core/sandbox/pool.py index 071aae7..45a6397 100644 --- a/core/sandbox/pool.py +++ b/core/sandbox/pool.py @@ -61,7 +61,12 @@ DEFAULT_IDLE_TTL_SECONDS = 300 # 容器资源限制默认值(可被 yaml `sandbox.*` / env override,详 SandboxPool ctor) DEFAULT_MEMORY = "2g" DEFAULT_CPUS = "1.0" -DEFAULT_PIDS_LIMIT = 256 +# pids-limit 把**线程**也计数:chromium headless 一次启动(browser+GPU+network+renderer +# 多进程)就 ~150-200 线程,256 本来就贴边;shell 超时又只杀 host 侧 docker CLI,容器内 +# mmdc+chromium 树留着(见 executor_docker.py 头注 Cancel limitation),残留几棵就把配额 +# 吃穿 → 后续 chromium 全部 pthread_create EAGAIN 崩(2026-07 生产实测,mermaid 渲染 +# 连挂近一个月)。1024 给足 2-3 次并存 chromium + 正常任务余量,仍防 fork 炸弹。 +DEFAULT_PIDS_LIMIT = 1024 # chromium(mmdc 渲 mermaid / puppeteer)默认走 /dev/shm,docker 不传 --shm-size 时 # 只给 64MB,起不来就一直挂到 timeout。镜像备的 puppeteer-config 有 --disable-dev-shm-usage, # 但模型不一定用那份;这里从根上把 /dev/shm 撑到够用,任何 chromium 路径都不再挂。 diff --git a/scripts/diag_mmdc_history2.py b/scripts/diag_mmdc_history2.py new file mode 100644 index 0000000..b9f3036 --- /dev/null +++ b/scripts/diag_mmdc_history2.py @@ -0,0 +1,72 @@ +"""90-day mmdc history with proper exit-code classification + detail of 06-23 failure. (ASCII safe)""" +import os +from pathlib import Path + +env = Path(__file__).resolve().parent.parent / ".env" +for line in env.read_text(encoding="utf-8").splitlines(): + if line.strip().startswith("ZCBOT_DB_URL="): + os.environ["ZCBOT_DB_URL"] = line.split("=", 1)[1].strip() +from sqlalchemy import create_engine, text # noqa: E402 + +import builtins # noqa: E402 + +_out = open(Path(__file__).resolve().parent / "_mmdc_history2.txt", "w", encoding="utf-8") + + +def print(*a, **k): # noqa: A001 + builtins.print(*a, **k, file=_out) + + +engine = create_engine(os.environ["ZCBOT_DB_URL"]) + +with engine.connect() as conn: + rows = conn.execute(text(""" + select m.task_id, m.idx, m.created_at, m.payload->>'content' as c + from messages m + where m.payload->>'role' = 'tool' + and m.payload->>'content' like '%Generating single mermaid chart%' + and m.created_at > now() - interval '90 days' + order by m.created_at + """)).fetchall() + print(f"[MMDC RUNS last 90d] count={len(rows)}") + for tid, idx, ca, c in rows: + if "Failed to launch the browser" in c: + v = "LAUNCH_FAIL" + elif "[exit 0]" in c and "Error" not in c: + v = "EXIT0" + elif "timed out" in c: + v = "TIMEOUT" + else: + v = "OTHER" + print(f" {ca} task={str(tid)[:8]} idx={idx} {v}") + if v in ("EXIT0", "OTHER"): + print(" head:", c[:300].replace("\n", " | ")) + + # also count timeouts of mmdc commands (result may lack the Generating line) + rows2 = conn.execute(text(""" + select m.task_id, m.idx, m.created_at + from messages m + join messages a on a.task_id = m.task_id and a.idx = m.idx - 1 + where m.payload->>'role' = 'tool' + and m.payload->>'content' like '%timed out%' + and a.payload::text like '%mmdc%' + and m.created_at > now() - interval '90 days' + order by m.created_at + """)).fetchall() + print(f"\n[MMDC TIMEOUTS last 90d] count={len(rows2)}") + for tid, idx, ca in rows2: + print(f" {ca} task={str(tid)[:8]} idx={idx}") + + # detail: 06-23 c4f03d56 failure command+error + print("\n[DETAIL c4f03d56 idx 84-86]") + rows3 = conn.execute(text(""" + select idx, payload from messages + where task_id::text like 'c4f03d56%' and idx between 83 and 86 order by idx + """)).fetchall() + for idx, p in rows3: + print(f"[{idx}] {p.get('role')}") + for tc in p.get("tool_calls") or []: + fn = tc.get("function") or {} + print(f" CALL {fn.get('name')}: {str(fn.get('arguments'))[:600]}") + if p.get("role") == "tool": + print(f" RESULT: {str(p.get('content'))[:1200]}") diff --git a/scripts/diag_mmdc_ok_case.py b/scripts/diag_mmdc_ok_case.py new file mode 100644 index 0000000..29c95ba --- /dev/null +++ b/scripts/diag_mmdc_ok_case.py @@ -0,0 +1,36 @@ +"""Inspect the single OK mmdc render (task 839a7c14 idx~112) vs a failing neighbor. (ASCII safe)""" +import os +from pathlib import Path + +env = Path(__file__).resolve().parent.parent / ".env" +for line in env.read_text(encoding="utf-8").splitlines(): + if line.strip().startswith("ZCBOT_DB_URL="): + os.environ["ZCBOT_DB_URL"] = line.split("=", 1)[1].strip() +from sqlalchemy import create_engine, text # noqa: E402 + +import builtins # noqa: E402 + +_out = open(Path(__file__).resolve().parent / "_mmdc_ok_case.txt", "w", encoding="utf-8") + + +def print(*a, **k): # noqa: A001 + builtins.print(*a, **k, file=_out) + + +engine = create_engine(os.environ["ZCBOT_DB_URL"]) + +with engine.connect() as conn: + rows = conn.execute(text(""" + select idx, payload from messages + where task_id::text like '839a7c14%' and idx between 105 and 118 + order by idx + """)).fetchall() + for idx, p in rows: + role = p.get("role") + print(f"[{idx}] {role}") + for tc in p.get("tool_calls") or []: + fn = tc.get("function") or {} + print(f" CALL {fn.get('name')}: {str(fn.get('arguments'))[:800]}") + if role == "tool": + print(f" RESULT: {str(p.get('content'))[:800]}") + print() diff --git a/scripts/diag_task_9dcae061.py b/scripts/diag_task_9dcae061.py new file mode 100644 index 0000000..9e3acfa --- /dev/null +++ b/scripts/diag_task_9dcae061.py @@ -0,0 +1,73 @@ +"""Dump task 9dcae061 full row + messages to diagnose render failure (ASCII safe).""" +import json +import os +import sys +from pathlib import Path + +env = Path(__file__).resolve().parent.parent / ".env" +for line in env.read_text(encoding="utf-8").splitlines(): + if line.strip().startswith("ZCBOT_DB_URL="): + os.environ["ZCBOT_DB_URL"] = line.split("=", 1)[1].strip() +from sqlalchemy import create_engine, text # noqa: E402 + +import builtins # noqa: E402 + +_out = open(Path(__file__).resolve().parent / "_task_9dcae061.txt", "w", encoding="utf-8") + + +def print(*a, **k): # noqa: A001 + builtins.print(*a, **k, file=_out) + + +engine = create_engine(os.environ["ZCBOT_DB_URL"]) +TID = "9dcae061-684e-4d1a-8287-4ef6e32e7183" + + +def s(x, n=4000): + t = str(x or "") + return t if len(t) <= n else t[:n] + f"...[+{len(t)-n} chars]" + + +with engine.connect() as conn: + row = conn.execute(text("select * from tasks where task_id=:t"), {"t": TID}).mappings().fetchone() + if not row: + print("[NO TASK]", TID) + sys.exit(1) + print("[TASK ROW]") + for k, v in dict(row).items(): + print(f" {k} = {s(v, 2000)}") + + msgs = conn.execute( + text("select idx,payload,model_profile,tokens_in,tokens_out,created_at from messages " + "where task_id=:t order by idx"), + {"t": TID}, + ).fetchall() + print(f"\n[MESSAGES] count={len(msgs)}\n") + for idx, p, mp, ti, to, ca in msgs: + if isinstance(p, str): + try: + p = json.loads(p) + except Exception as e: + print(f"[{idx}] !! payload JSON parse fail: {e}; raw head: {s(p, 500)}") + continue + role = p.get("role") + head = f"[{idx}] {role}" + if mp: + head += f" ({mp})" + if ti or to: + head += f" tok={ti}/{to}" + head += f" at={ca}" + print(head) + content = p.get("content") + if content is not None: + print(f" content type={type(content).__name__}") + print(" content:", s(content, 3000)) + extra_keys = set(p.keys()) - {"role", "content", "tool_calls", "name", "tool_call_id"} + if extra_keys: + print(" extra keys:", sorted(extra_keys)) + for tc in p.get("tool_calls") or []: + fn = tc.get("function") or {} + print(f" CALL {fn.get('name')}({s(fn.get('arguments'), 1500)})") + if role == "tool": + print(f" TOOL[{p.get('name')}] id={p.get('tool_call_id')}:", s(content, 2000)) + print()