feat(deploy): 蓝绿双实例无感部署落地——零 503 窗口 + SSE 不断流(bump 0.41.0)
- deploy/update_bluegreen.sh:复用 update.sh --prepare-only 做准备 → 起空闲色
→ healthz 验 instance → nginx upstream 切流 → 轮询 DB 等旧色 run 自然清空
(DRAIN_WAIT_SECONDS 默 600s,期间旧 SSE 继续直播)→ stop 旧色 → 残留
running 行标 error → enable 新色/disable 旧色;--switch 半途恢复调流
- deploy/zcbot@.service 模板 unit + deploy/instances/{blue,green}.env +
deploy/nginx/zcbot.conf.example(upstream 单行文件由脚本改写)
- tasks.run_owner(migration 0020):起 run 写 ZCBOT_INSTANCE,启动 reaper
只收自己色的孤儿,不误杀另一实例在跑的 run
- sandbox 容器名/label 带实例色,shutdown_all 只清自己色,并存期互不清扫
- 微信 ClawBot 长轮询 PG advisory lock 选主(detach 连接持锁 + 30s keepalive,
断连即让位),蓝绿并存不抢消息;定时任务 claim 本就 SKIP LOCKED 免改
- /healthz 加 instance 字段;update.sh 加 --prepare-only
- 单实例部署(ZCBOT_INSTANCE 未设)行为全部不变;RUN/DESIGN/PROGRESS 同步
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
c52f399f07
commit
1d62670366
|
|
@ -125,6 +125,8 @@ Session = 消息列表,ORM 直写 PG `messages`(append-only,jsonb 存 LiteLLM
|
|||
|
||||
同一份 web `/v1` 服务换部署位置:Storage 都是 PG(换 `ZCBOT_DB_URL`);working_dir/memory 换外层根目录;Sandbox 本地 subprocess → SaaS per-user 容器;Auth 邮箱密码 + platform_key→JWT → 未来 OIDC(邮箱密码长期并存)。`workspace/` 仅存产物,state/messages 全在 PG。
|
||||
|
||||
**无感部署(蓝绿,0.41)**:生产双 systemd 实例(`ZCBOT_INSTANCE=blue/green`)+ nginx upstream 切流,取代单实例 restart 的 503 窗口。取舍:**event broker 刻意留在进程内**(不上 Redis pub/sub)——代价只是"切换瞬间刷新页面的用户看不到旧实例上 run 的直播"(run 照常落库),当前规模不值得为此引基础设施。双实例并存的互踩由三件事消解:`tasks.run_owner`(0020)让 reaper 只收自己色、sandbox 容器名/label 带实例色各管各的、微信长轮询 PG advisory lock 选主;定时任务 claim 本就 SKIP LOCKED 天然安全。部署编排在 `deploy/update_bluegreen.sh`(流程/兜底见 RUN)。
|
||||
|
||||
### 7.1 心智模型:Task 一等公民 + Dir 文件副视图
|
||||
|
||||
两个并列入口,正交不嵌套:**Task list**(主,"我的对话历史")+ **Dir tree**(辅,"我的文件资产")。类比 Finder + 最近使用。dir **不是 task 的父容器**、无 DB 实体、path 即标识。同 working_dir 多 task 共享 = "同一项目多对话",无需"项目"实体;前缀嵌套拒(no-subtask)。skill 产物全落 working_dir,不引 artifacts 表。空 dir 正常展示(上传本身是有效行为)。多 task 并发写由软警告兜底(§7.9)。
|
||||
|
|
@ -164,6 +166,7 @@ tasks(task_id pk, user_id fk, name NOT NULL, working_dir NOT NULL, skill, descri
|
|||
model_profile, tokens_*, cost_usd,
|
||||
channel default 'web', -- 0013 渠道来源,仅 INSERT 写定
|
||||
run_status default 'idle', run_error, -- 0004 合并 runs 表
|
||||
run_owner, -- 0020 蓝绿实例归属,reaper 只收自己色;单实例 NULL
|
||||
scheduled_job_id, -- 0017 定时任务归属,普通列表过滤
|
||||
context_base_idx, -- 0019 §8.8 软重置窗口起点
|
||||
deleted_at, -- 0010 软删
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 1-2 句:做了啥 + 关键判断;细节查 `git log` / `git diff` / `DESIGN §7.9`。
|
||||
|
||||
最后更新:2026-07-06(DESIGN/PROGRESS 两文档精简,bump 0.39.1)
|
||||
最后更新:2026-07-06(蓝绿无感部署 B 档落地,bump 0.41.0)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -23,6 +23,7 @@
|
|||
|
||||
### 2026-07
|
||||
|
||||
- **07-06 / 0.41.0 蓝绿无感部署(RUN.md B 档落地)**:生产部署零 503 窗口 + 在跑对话 SSE 直播不断。`zcbot@blue/green` 双 systemd 实例(模板 unit + `deploy/instances/*.env`)+ nginx upstream 单行文件切流,编排在 `deploy/update_bluegreen.sh`(复用 `update.sh --prepare-only`;切流后轮询 DB 等旧色 run 自然清空再 stop——直接 stop 会被 uvicorn `timeout_graceful_shutdown=5` 掐 SSE)。双实例并存互踩三修:`tasks.run_owner`(0020)reaper 只收自己色、sandbox 容器名/label 带实例色各清各的、微信 ClawBot 长轮询 PG advisory lock 选主(detach 连接持锁,断连即让位);定时任务 claim 本就 SKIP LOCKED 免改。关键判断:broker 刻意不出进程(不上 Redis),接受"切换瞬间刷新的用户看不到旧实例直播"这一边缘。单实例部署 `ZCBOT_INSTANCE` 未设全部退回原行为,`update.sh` 保留不废。
|
||||
- **07-06 / 0.40.0 ppt 伪导出根治(三层)**:0.35.1 复发(模型照禁令里的"配方"装 cairosvg 手搓整页贴图导出)复盘——①平台层机检门落地(`core/pptx_guard.py` + loop 钩子:shell/run_python 新产出 .pptx 命中"≥80% 页整版位图 + 全 deck 零原生文本"即注入 `[产物机检 ERROR]` 当场逼返工;判定刻意保守,`tests/test_pptx_guard.py` 8 例);②禁令去菜谱:ppt/brief SKILL.md 否定式指令改"正面窄门"(唯一入口 + 报错就修/上报),删掉可照做的违规配方;③官方脚本 5 处 `pip install` hint 改环境说明(pptx_media/svg_preview/crop_images/fix_image_aspect/pptx_cli)——打印给 agent 的安装指令就是执行指令。详 DESIGN §8.9。
|
||||
- **07-06 / 0.39.2 research 对接 paper_server API Key**:paper_server GET 接口收紧要 Key 后,`skills/research/paper.py` 全部请求带 `?api_key=`(env `PAPER_SERVER_API_KEY`),认证失败(paper_server 异常处理器压成 HTTP 400 + err_code)转成带指引的 RuntimeError;这把低价值只读 key 是唯一刻意放进 sandbox 的凭证——`run_python` env 过滤器加 allowlist 放行、docker 模式 `_sandbox_env()` 透传进容器(顺带修掉 `PAPER_SERVER_URL` 覆盖不进 docker 的缺口)。
|
||||
- **07-06 / 0.39.1 文档精简**:PROGRESS(865→188 行)与 DESIGN(797→218 行)按各自边界契约压缩——PROGRESS 每条收成一行(做了啥+关键判断,细节查 git log),DESIGN 保留全部决策/取舍/硬协议清单/升级信号、砍实现复述(协议细节指向代码与 probe 脚本),顺手把 §2 架构树补齐实际模块。
|
||||
|
|
|
|||
75
RUN.md
75
RUN.md
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> 怎么把 zcbot 跑起来。env / 常用命令 / 故障兜底。设计看 `DESIGN.md`,进度看 `PROGRESS.md`。
|
||||
|
||||
最后更新:2026-06-12(admin 角色 + /static/admin.html 管理后台:user role CLI / 建用户带 --role / 顶栏"管理"入口)
|
||||
最后更新:2026-07-06(蓝绿双实例无感部署 B 档落地:`deploy/update_bluegreen.sh` + `zcbot@.service` 模板 unit + nginx upstream 切流;`tasks.run_owner`(0020)/ 实例色 sandbox 容器 / 微信长轮询 PG advisory lock 选主)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -299,9 +299,7 @@ sudo systemctl stop zcbot
|
|||
|
||||
下面两挡是另一个维度(REST / SSE 抖动平滑),与 drain 正交:
|
||||
|
||||
zcbot 现在 5 人级 + SSE 长连接,**严格"零中断"**(蓝绿 + nginx + SSE 客户端 reconnect 设计)代价高,不值得。有性价比的两挡:
|
||||
|
||||
**A. 简易档:`--reload`**(推荐当前规模)
|
||||
**A. 简易档:`--reload`**(dogfood / 单实例机可用)
|
||||
ExecStart 加 `--reload`,`git pull` 后 uvicorn 监听到文件变动自动重起子进程,REST 抖动 < 1s。**代价**:SSE 连接被切断,**dev.html 自动重连 3 次(1s/2s/4s 退避)**;若新进程已被启动 reaper 标 `run_status=error`,重连立即收 done,卡片末尾追加红色"连接已断开,请重发"。期间 LLM 吐的 delta 丢失(broker 不持久化 event,接受)。3 次仍失败 → 同上提示,用户重发即可。
|
||||
|
||||
```ini
|
||||
|
|
@ -310,18 +308,66 @@ ExecStart=/opt/zcbot/.venv/bin/python main.py web --host 0.0.0.0 --port 8765 --r
|
|||
|
||||
`sudo systemctl restart zcbot` 一次生效。之后**只 `git pull` 即可**,不用再 restart;改 unit 文件本身才需 daemon-reload + restart。
|
||||
|
||||
**B. 真无感档:nginx + 蓝绿双实例**(将来流量上来再上)
|
||||
两个 systemd 实例 `zcbot@blue` / `zcbot@green`(模板 unit,`--port 8765` / `--port 8766`),nginx upstream 在中间切。流程:
|
||||
1. 部署到空闲实例(假设 green):`sudo systemctl restart zcbot@green`
|
||||
2. `curl 127.0.0.1:8766/healthz` 验新版起来
|
||||
3. 改 nginx upstream 指向 green,`nginx -s reload` — **新 REST 走 green,旧 SSE 还连在 blue 不断**
|
||||
4. 等 blue SSE 自然清空(`ss -tnp | grep :8765` 为空)再关 blue
|
||||
### 蓝绿部署(B 档,已实现,生产推荐)
|
||||
|
||||
**zcbot 端额外要做的事**:消息 broker 当前在 task 进程内(`web/broker.py`),蓝绿期间同 task 不同进程会丢事件。nginx 侧用 `hash $arg_task_id consistent` 保同 task 落同实例可以缓解,但 task 创建分布是另一回事。要做这条得先把 broker 改成 Redis pub/sub。10 人内**不推**,留到真有需要再上。
|
||||
两个 systemd 实例 `zcbot@blue`(127.0.0.1:8765)/ `zcbot@green`(127.0.0.1:8766),nginx upstream(`/etc/nginx/zcbot_upstream.conf`,单行 `server` 指令)在中间切流量。日常部署一条命令:
|
||||
|
||||
```bash
|
||||
sudo bash /opt/zcbot/deploy/update_bluegreen.sh
|
||||
# --skip-build 透传给 update.sh(跳过 sandbox 镜像重建)
|
||||
# --switch blue|green 只切 nginx 流量到指定色(不拉码不重启;半途恢复 / 手动调流)
|
||||
# DRAIN_WAIT_SECONDS=N 等旧色在跑 run 自然收尾的上限(默 600s)
|
||||
```
|
||||
|
||||
脚本流程:`update.sh --prepare-only`(拉码 / pip / migration / build,全复用单实例脚本)→ 起空闲色实例 → healthz 验活(核对 `instance` 字段)→ 改写 upstream + nginx reload(**新请求全走新实例,零 503**)→ **轮询 DB 等旧色在跑 run 自然清空**(上限 `DRAIN_WAIT_SECONDS`,期间旧 SSE 继续直播不断流 —— 直接 stop 的话 uvicorn `timeout_graceful_shutdown=5` 会 5s 掐长连接)→ stop 旧实例(残留 run 走 SIGTERM drain / 协作式 cancel)→ 旧色 SIGKILL 残留的 running 行标 error → enable 新色 / disable 旧色(重启机器只拉起 live 色)。
|
||||
|
||||
效果 vs 单实例 `update.sh`:**没有 503 窗口,在跑对话的 SSE 直播也不断**。已知边缘(接受):用户恰在切换后刷新页面、而他的 run 还在旧实例上跑 → 新实例 broker 没有该 run 的事件,直播看不到(run 照常在旧实例跑完落库,刷新看历史即可)。broker 仍是进程内(`web/broker.py`)不跨实例 —— 刻意取舍,当前规模不值得上 Redis pub/sub。
|
||||
|
||||
蓝绿并存期的互踩问题已在代码层解决(单实例部署 `ZCBOT_INSTANCE` 未设,以下全部退回原行为):
|
||||
- **启动 reaper**:`tasks.run_owner`(migration 0020)记 run 归属实例,实例启动只收自己色(+ NULL 遗留)的孤儿 run,不误杀另一实例在跑的
|
||||
- **sandbox 容器**:容器名 / label 带实例色(`zcbot-sandbox-blue-<uid>` + `zcbot.instance` label),`shutdown_all` 只清自己色;切换窗口同 user 可能短暂双容器,idle reaper 会收
|
||||
- **定时任务**:`claim_due_jobs` 本就 `FOR UPDATE SKIP LOCKED` 原子认领,双实例不重复触发,无需改
|
||||
- **微信 ClawBot 长轮询**:PG advisory lock 选主(getupdates 是消费型,双拉会抢消息),同时只有一个实例拉;旧实例停 → 锁随连接释放 → 新实例 ≤15s 接管
|
||||
|
||||
**一次性 bootstrap(从单实例切到蓝绿)** —— 这一步本身仍有一次普通重启窗口,"无感"从下一次部署开始:
|
||||
|
||||
```bash
|
||||
cd /opt/zcbot
|
||||
# 1) 先用旧脚本把含蓝绿支持的代码 + migration 0020 部署上去(最后一次有 503 窗口的部署)
|
||||
sudo bash deploy/update.sh
|
||||
|
||||
# 2) 装模板 unit(实例 env 已随仓库在 deploy/instances/{blue,green}.env)
|
||||
sudo cp deploy/zcbot@.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# 3) 装 nginx + 反代配置 + 初始 upstream 指 blue(conf 里按需改 server_name / TLS / 对外端口)
|
||||
sudo apt-get install -y nginx
|
||||
sudo cp deploy/nginx/zcbot.conf.example /etc/nginx/conf.d/zcbot.conf
|
||||
echo 'server 127.0.0.1:8765;' | sudo tee /etc/nginx/zcbot_upstream.conf
|
||||
|
||||
# 4) 停旧单实例服务(优雅 drain)→ 清无实例 label 的旧 sandbox 容器 → 起 blue → nginx 生效
|
||||
sudo systemctl disable --now zcbot
|
||||
sudo docker rm -f $(sudo docker ps -aq -f label=zcbot.product=sandbox) 2>/dev/null || true
|
||||
sudo systemctl enable --now zcbot@blue
|
||||
curl -fsS http://127.0.0.1:8765/healthz # 应含 "instance":"blue"
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
|
||||
# 5) 之后日常部署一律:
|
||||
sudo bash deploy/update_bluegreen.sh
|
||||
```
|
||||
|
||||
注意:
|
||||
- **对外端口变了**:客户端从直连 8765 改走 nginx(80/443)。platform 嵌入 / EMBED.md 对接方同步换 URL;实例只绑 127.0.0.1 不再直接对外。
|
||||
- **回滚 ≠ 切回旧色**:代码目录两实例共享,旧实例重启后跑的也是新代码。回滚 = `git revert` 后重跑 `update_bluegreen.sh`。`--switch` 只用于半途恢复(如新实例切完流量后挂了,切回还活着的那个色)。
|
||||
- `deploy/update.sh`(单实例版)保留不废:蓝绿脚本内部复用它 `--prepare-only`;单实例部署机(host backend / 内网机)继续用它,行为不变。
|
||||
- unit 的 `TimeoutStopSec=90` 与 `config/agent.yaml` shutdown 段(drain 30s + cancel 15s)的约束同单实例:改 yaml 数值时模板 unit 跟着调。
|
||||
|
||||
### 部署 SOP
|
||||
|
||||
**一把梭(推荐):`deploy/update.sh`**
|
||||
> 蓝绿部署机(生产,已 bootstrap)一律用 `sudo bash deploy/update_bluegreen.sh`(见上节)。
|
||||
> 下面的 `update.sh` 是单实例版:host backend 机 / 内网机 / 未切蓝绿的部署继续用。
|
||||
|
||||
**一把梭(单实例):`deploy/update.sh`**
|
||||
|
||||
```bash
|
||||
# 在服务器上,root 跑(脚本内部文件操作降到 zcbot 用户)
|
||||
|
|
@ -787,6 +833,11 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_<user_uuid>" /opt
|
|||
| dev.html 显示 "load failed" 立刻回登录页 | token 过期或 JWT_SECRET 服务端变了。已自动跳登录页,按上次 tab 重登 |
|
||||
| dev.html 顶栏出现"连接断开,重连中…(N/3)" | SSE 流被切(`--reload` 重启 / nginx 切换 / 网络抖)。客户端自动重连,1s/2s/4s 退避;新进程已 reaper 标 error 则立即收 done + 卡片末尾"请重发"提示;若服务端还活着会继续看后续 delta(断开期间的丢失,broker 不持久化) |
|
||||
| 对话里偶发 `[Error] invalid JSON arguments` / `[Error] bad arguments to write: ... missing required` | deepseek-v4-flash **大参数工具调用(大 write/run_python,≈7K+ 字符)偶发把内容碎片错位粘进 arguments 或退化成空 `{}`**(上游流式抖动)。`core/loop.py` 已自动兜底:畸形参数丢弃整轮重 roll(≤3 次)+ 最后一次降级非流式。仍频繁撞 → 引导模型**把大文件拆小 / 用 run_python 分块写**,或换 `deepseek_v4.pro`。前端看到 warn「工具调用参数损坏…重试」即此机制在生效 |
|
||||
| 蓝绿脚本报 `旧单实例服务 zcbot.service 还在跑` | 还没做蓝绿 bootstrap(旧服务占 8765 与 `zcbot@blue` 冲突)。走 RUN.md「蓝绿部署」bootstrap 段:`disable --now zcbot` 后起 `zcbot@blue` |
|
||||
| 蓝绿 wait_health 报 `instance 不是 blue/green` | 实例 env 没生效:`/etc/systemd/system/zcbot@.service` 未装 / 没 `daemon-reload` / `deploy/instances/<色>.env` 缺失。`systemctl cat zcbot@blue` 核对 `EnvironmentFile` 两行都在 |
|
||||
| 蓝绿切换后用户直播不动、但任务其实在跑 | 用户恰在切换窗口刷新页面,run 还在旧实例上而 SSE 连到了新实例(broker 进程内不跨实例)。run 照常跑完落库,等旧实例日志安静后让用户刷新看历史;非 bug,已知边缘 |
|
||||
| 蓝绿并存期微信 ClawBot 没反应 | 入站长轮询走 PG advisory lock 选主。两实例 journalctl 里找 `[wechat] inbound leader lock acquired`——都没有 → 检查 DB 连通;锁在已停实例上不可能(连接断锁即释),新实例最长 15s 接管 |
|
||||
| 蓝绿部署后旧色 sandbox 容器残留 | 正常:旧实例 stop 时 `shutdown_all` 只清自己色,失败/超时的靠下次该色启动兜底。手动清:`docker rm -f $(docker ps -aq -f label=zcbot.instance=<色>)` |
|
||||
| 长任务跑到一半停下、提示「已达单轮步数上限…回复『继续』可接着跑」 | **预期行为非崩溃**:单个 run 自主步数到 backstop(`config/models/*.yaml` 的 `max_iterations`,flash 120 / pro 150)就主动停,回 `[reached max iterations]`。直接回复「继续」即续跑。经常撞顶=任务确实大,可调高对应 variant 的 `max_iterations` 或换 pro |
|
||||
| 任务停下提示「连续 N 步无净产出…已自动停止」(回 `[stopped: no progress]`) | **空转熔断**:连续 `_STALL_LIMIT`(loop.py,默认 8)步所有 tool 都只返 `[Error]`/重复结果/被拦 = 没在推进,主动停以免空烧 token。说明模型卡死在某个错上——**换思路 / 补充信息再回复「继续」**,别原样重发(会再次撞同一墙) |
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
# zcbot 版本号单一事实源:web/app.py 的 FastAPI version、/healthz 返回、前端展示都引这里。
|
||||
# 改版本只动这一行。
|
||||
__version__ = "0.40.0"
|
||||
__version__ = "0.41.0"
|
||||
|
|
|
|||
|
|
@ -2,7 +2,10 @@
|
|||
|
||||
命名:`zcbot-sandbox-<user_id>`(user_id = UUID 标准串带 dash,与 bind mount
|
||||
源路径 `<workspace>/users/<user_id>/` 对齐 ── `docker ps` 看到容器名能直接 grep
|
||||
出 workspace 目录)。
|
||||
出 workspace 目录)。蓝绿双实例(RUN.md B 档,`ZCBOT_INSTANCE=blue/green`)时名字
|
||||
变 `zcbot-sandbox-<instance>-<user_id>` + 追加 `zcbot.instance` label,shutdown_all
|
||||
只清自己实例的容器 ── 并存期两实例各管各的,互不清扫(代价:切换窗口同一 user 可能
|
||||
短暂双容器,idle reaper 会收)。单实例部署不设 env,命名/清扫行为不变。
|
||||
|
||||
生命周期:
|
||||
- `ensure(user_id)`:per-user `threading.Lock` 串行化 → `docker inspect` 探测 →
|
||||
|
|
@ -41,6 +44,10 @@ CONTAINER_NAME_PREFIX = "zcbot-sandbox-"
|
|||
LABEL_PRODUCT_KEY = "zcbot.product"
|
||||
LABEL_PRODUCT_VALUE = "sandbox"
|
||||
LABEL_USER_ID_KEY = "zcbot.user_id"
|
||||
LABEL_INSTANCE_KEY = "zcbot.instance"
|
||||
|
||||
# 蓝绿实例名(与 web/app.py 同款读法);进程生命周期内不变,import 时读一次
|
||||
INSTANCE = os.getenv("ZCBOT_INSTANCE", "").strip()
|
||||
|
||||
DEFAULT_IMAGE = "zcbot-sandbox:latest"
|
||||
DEFAULT_IDLE_TTL_SECONDS = 300
|
||||
|
|
@ -57,6 +64,8 @@ DEFAULT_SHM_SIZE = "512m"
|
|||
|
||||
|
||||
def container_name(user_id: UUID) -> str:
|
||||
if INSTANCE:
|
||||
return f"{CONTAINER_NAME_PREFIX}{INSTANCE}-{user_id}"
|
||||
return f"{CONTAINER_NAME_PREFIX}{user_id}"
|
||||
|
||||
|
||||
|
|
@ -201,6 +210,7 @@ class SandboxPool:
|
|||
"--name", name,
|
||||
"--label", f"{LABEL_PRODUCT_KEY}={LABEL_PRODUCT_VALUE}",
|
||||
"--label", f"{LABEL_USER_ID_KEY}={user_id}",
|
||||
*(["--label", f"{LABEL_INSTANCE_KEY}={INSTANCE}"] if INSTANCE else []),
|
||||
"--network", NETWORK_NAME,
|
||||
# §7.5 硬限制(任一缺失视为 hardening 未完成)
|
||||
"--read-only", # rootfs read-only
|
||||
|
|
@ -273,13 +283,18 @@ class SandboxPool:
|
|||
return removed
|
||||
|
||||
def shutdown_all(self) -> List[str]:
|
||||
"""杀所有 label=zcbot.product=sandbox 的容器。
|
||||
"""杀本实例名下所有 sandbox 容器。
|
||||
|
||||
典型用途:① app 启动时清前驱进程留下的孤儿 ② 测试 / 维护手动调。
|
||||
蓝绿(ZCBOT_INSTANCE 在)时按 zcbot.instance label 再过滤一层,只清自己
|
||||
实例的 ── 另一实例正在跑的容器碰不到;单实例部署清全部(含无 instance
|
||||
label 的历史遗留)。
|
||||
"""
|
||||
filters = ["--filter", f"label={LABEL_PRODUCT_KEY}={LABEL_PRODUCT_VALUE}"]
|
||||
if INSTANCE:
|
||||
filters += ["--filter", f"label={LABEL_INSTANCE_KEY}={INSTANCE}"]
|
||||
list_r = subprocess.run(
|
||||
["docker", "ps", "-aq", "--filter",
|
||||
f"label={LABEL_PRODUCT_KEY}={LABEL_PRODUCT_VALUE}"],
|
||||
["docker", "ps", "-aq", *filters],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
if list_r.returncode != 0 or not list_r.stdout.strip():
|
||||
|
|
|
|||
|
|
@ -85,6 +85,10 @@ class Task(Base):
|
|||
# 只有 error 是持久终态(下次起新 run 时由 post_message 清掉)
|
||||
run_status: Mapped[str] = mapped_column(Text, nullable=False, default="idle")
|
||||
run_error: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||
# in-flight run 归属实例(0020,蓝绿双实例 RUN.md B 档):起 run 时写 ZCBOT_INSTANCE
|
||||
# (blue/green),启动 reaper 只收自己实例的孤儿。单实例部署恒 NULL,行为不变。
|
||||
# 只在 running/cancelling 期间有意义,run 收尾不清(reaper 只看活跃行)。
|
||||
run_owner: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||
# 喂给模型的上下文窗口起点(0019,channel 长会话软重置)。Session.load 只把 idx >=
|
||||
# context_base_idx 的消息装进 LLM 上下文;之前的历史仍全量留 messages 表(web 翻得到)。
|
||||
# web 普通任务恒 0 = 喂全量;channel 入站按 gap / 「新话题」推进。详 DESIGN §8.7。
|
||||
|
|
|
|||
|
|
@ -0,0 +1,32 @@
|
|||
"""tasks.run_owner 列(蓝绿双实例部署,RUN.md B 档).
|
||||
|
||||
Revision ID: 0020
|
||||
Revises: 0019
|
||||
Create Date: 2026-07-06
|
||||
|
||||
给 tasks 加 run_owner(TEXT NULL):当前 in-flight run 是哪个实例(ZCBOT_INSTANCE,
|
||||
如 blue/green)起的。启动 stale-run reaper 据此只收自己实例的孤儿 run,不误杀蓝绿
|
||||
并存期另一实例正在跑的 run。
|
||||
|
||||
单实例部署(ZCBOT_INSTANCE 未设)写 NULL,reaper 行为不变(全量收)。additive,
|
||||
无数据迁移;存量 running 行 run_owner=NULL,会被任一实例的 reaper 兜底收掉(与
|
||||
迁移前行为一致)。
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision: str = "0020"
|
||||
down_revision: Union[str, None] = "0019"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("tasks", sa.Column("run_owner", sa.Text(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("tasks", "run_owner")
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
# zcbot@blue 实例配置(systemd 模板 unit EnvironmentFile,RUN.md B 档)
|
||||
# 端口/实例名是蓝绿脚本与 nginx upstream 的契约,别改;业务配置进 /opt/zcbot/.env
|
||||
ZCBOT_PORT=8765
|
||||
ZCBOT_INSTANCE=blue
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
# zcbot@green 实例配置(systemd 模板 unit EnvironmentFile,RUN.md B 档)
|
||||
# 端口/实例名是蓝绿脚本与 nginx upstream 的契约,别改;业务配置进 /opt/zcbot/.env
|
||||
ZCBOT_PORT=8766
|
||||
ZCBOT_INSTANCE=green
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
# zcbot nginx 反代示例(蓝绿双实例,RUN.md §无感更新 B 档)。
|
||||
#
|
||||
# 安装(一次性):
|
||||
# sudo cp deploy/nginx/zcbot.conf.example /etc/nginx/conf.d/zcbot.conf # 按需改 server_name/端口/TLS
|
||||
# echo 'server 127.0.0.1:8765;' | sudo tee /etc/nginx/zcbot_upstream.conf # 初始指 blue
|
||||
# sudo nginx -t && sudo systemctl reload nginx
|
||||
#
|
||||
# 之后 zcbot_upstream.conf 由 deploy/update_bluegreen.sh 改写 + reload nginx 切流量,
|
||||
# 本文件不再动。nginx reload 对已建立连接零影响:旧 SSE 留在旧实例自然收尾,新请求走新实例。
|
||||
|
||||
upstream zcbot_backend {
|
||||
# 只含一行 server 127.0.0.1:8765|8766;由 update_bluegreen.sh 维护
|
||||
include /etc/nginx/zcbot_upstream.conf;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
# TODO 按部署域名改;上 TLS 时换 listen 443 ssl + 证书(certbot 等)
|
||||
server_name _;
|
||||
|
||||
# 上传接口(/v1/files/upload)会传大文件;与后端无硬限制,给足余量
|
||||
client_max_body_size 512m;
|
||||
|
||||
location / {
|
||||
proxy_pass http://zcbot_backend;
|
||||
proxy_http_version 1.1;
|
||||
# keepalive upstream 需要清掉 Connection 头
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# ★ SSE(GET /v1/tasks/{id}/events):关 buffering、拉长 read 超时。
|
||||
# 后端响应头已带 X-Accel-Buffering: no(对 proxy_buffering 同效),这里显式
|
||||
# 再关一层作双保险;30s 服务端 ping 心跳 < read_timeout,长连接不会被掐。
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
|
|
@ -13,6 +13,8 @@
|
|||
#
|
||||
# 怎么跑:在部署机 `sudo deploy/update.sh`(需 root 跑 systemctl;文件操作降到 $APP_USER)。
|
||||
# --skip-build 跳过第 4 步 sandbox 镜像重建(host backend 机器 / 临时不想动 docker)
|
||||
# --prepare-only 只跑 1-4(拉码/依赖/migration/build),不 restart 不验活 ——
|
||||
# 蓝绿部署 update_bluegreen.sh 复用本脚本做准备,切换自己管
|
||||
# 一次性 bootstrap(useradd / 写 systemd unit / enable)不在这里 —— 见 RUN.md §部署 一次性。
|
||||
#
|
||||
# 可调 env(不传则用默认 / Dockerfile 默认):
|
||||
|
|
@ -32,11 +34,13 @@ set -euo pipefail
|
|||
# 原始参数先存一份 —— pull 后若脚本自身有更新要 exec 新版本重跑,得把参数原样传回。
|
||||
ORIG_ARGS=("$@")
|
||||
SKIP_BUILD=0
|
||||
PREPARE_ONLY=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--skip-build) SKIP_BUILD=1; shift ;;
|
||||
-h|--help) echo "用法:sudo $0 [--skip-build]"; exit 0 ;;
|
||||
*) echo "[deploy] ERROR: 未知参数:$1(只支持 --skip-build)" >&2; exit 1 ;;
|
||||
--prepare-only) PREPARE_ONLY=1; shift ;;
|
||||
-h|--help) echo "用法:sudo $0 [--skip-build] [--prepare-only]"; exit 0 ;;
|
||||
*) echo "[deploy] ERROR: 未知参数:$1(只支持 --skip-build / --prepare-only)" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
|
|
@ -151,6 +155,12 @@ else
|
|||
"$APP_DIR"
|
||||
fi
|
||||
|
||||
# ── 4.5) prepare-only:蓝绿部署由 update_bluegreen.sh 接管切换 ──
|
||||
if [ "$PREPARE_ONLY" = "1" ]; then
|
||||
log "--prepare-only:代码/依赖/migration/镜像已就绪($NEW_HEAD),不 restart"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── 5) 重启 service(在 build 之后!)───────────────────────────
|
||||
# 优雅 drain:若有在跑的 run,restart 会先等它们收尾(≤ unit TimeoutStopSec,
|
||||
# 见 RUN.md / config/agent.yaml shutdown 段)再换新版 —— 这步"看着卡住"几十秒是正常的。
|
||||
|
|
|
|||
|
|
@ -0,0 +1,204 @@
|
|||
#!/usr/bin/env bash
|
||||
# zcbot 蓝绿无感部署(RUN.md §无感更新 B 档)。
|
||||
#
|
||||
# 拓扑:zcbot@blue(127.0.0.1:8765)/ zcbot@green(127.0.0.1:8766)双 systemd 实例,
|
||||
# nginx upstream(/etc/nginx/zcbot_upstream.conf,单行 server 指令)在中间切流量。
|
||||
#
|
||||
# 干什么(顺序写死):
|
||||
# 1) update.sh --prepare-only 拉码 / pip / db migration / docker build(复用单实例脚本)
|
||||
# 2) systemctl restart zcbot@<空闲色> 空闲实例起新代码
|
||||
# 3) curl 空闲实例 /healthz 验活 + 核对 instance 字段
|
||||
# 4) 改写 nginx upstream → reload 新 REST/SSE 全走新实例;旧 SSE 留旧实例不断
|
||||
# 5) 等旧实例在跑 run 自然清空 轮询 DB run_owner=旧色 的活跃行,归零才停
|
||||
# (上限 DRAIN_WAIT_SECONDS 默 600;期间旧 SSE
|
||||
# 继续直播不断流 —— 这步是"无感"的关键)
|
||||
# 6) systemctl stop zcbot@<旧live色> 此时通常已无在跑 run,秒停;超上限的残留 run
|
||||
# 走 SIGTERM drain → 协作式 cancel(标 idle 可重发)
|
||||
# 7) DB 收尾 旧色 SIGKILL 兜底残留的 running 行标 error
|
||||
# 8) enable 新 / disable 旧 重启机器只拉起当前 live 色
|
||||
#
|
||||
# 效果:部署全程无 503 窗口(对比单实例 update.sh 的几十秒 drain 窗口),在跑对话的
|
||||
# SSE 直播也不断(对比直接 stop:uvicorn timeout_graceful_shutdown=5 会 5s 掐长连接)。
|
||||
#
|
||||
# 怎么跑:sudo bash /opt/zcbot/deploy/update_bluegreen.sh
|
||||
# --skip-build 透传给 update.sh(跳过 sandbox 镜像重建)
|
||||
# --switch blue|green 只切 nginx 流量到指定色(不拉码不重启;半途恢复 / 手动调流用)
|
||||
#
|
||||
# 一次性 bootstrap(装模板 unit / nginx 配置 / 停旧单实例服务)见 RUN.md §蓝绿部署。
|
||||
# 回滚注意:代码目录两实例共享,旧实例停了再起跑的也是新代码 —— 回滚 = git revert 后
|
||||
# 重跑本脚本,不是把流量切回旧色。
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ORIG_ARGS=("$@")
|
||||
SKIP_BUILD_ARG=()
|
||||
SWITCH_TO=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--skip-build) SKIP_BUILD_ARG=(--skip-build); shift ;;
|
||||
--switch) SWITCH_TO="${2:-}"; shift 2 ;;
|
||||
-h|--help) echo "用法:sudo $0 [--skip-build] [--switch blue|green]"; exit 0 ;;
|
||||
*) echo "[deploy] ERROR: 未知参数:$1" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SCRIPT_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
|
||||
APP_DIR="${APP_DIR:-$(dirname "$SCRIPT_DIR")}"
|
||||
APP_USER="${APP_USER:-$(stat -c %U "$APP_DIR")}"
|
||||
PY="$APP_DIR/.venv/bin/python"
|
||||
UPSTREAM_FILE="${UPSTREAM_FILE:-/etc/nginx/zcbot_upstream.conf}"
|
||||
# 切流量后等旧实例在跑 run 自然收尾的上限(超了就 stop,残留 run 走协作式 cancel)
|
||||
DRAIN_WAIT_SECONDS="${DRAIN_WAIT_SECONDS:-600}"
|
||||
|
||||
declare -A PORT=([blue]=8765 [green]=8766)
|
||||
|
||||
log() { echo "[deploy] $*"; }
|
||||
fail() { echo "[deploy] ERROR: $*" >&2; exit 1; }
|
||||
asuser() { sudo -u "$APP_USER" "$@"; }
|
||||
|
||||
# ── 0) 前置检查 ────────────────────────────────────────────────
|
||||
[ "$(id -u)" -eq 0 ] || fail "需 root(systemctl / 写 nginx 配置),请用:sudo $0"
|
||||
[ -d "$APP_DIR/.git" ] || fail "$APP_DIR 不是 git 仓库"
|
||||
[ -x "$PY" ] || fail "找不到 venv python:$PY"
|
||||
[ -f "$APP_DIR/.env" ] || fail "缺 $APP_DIR/.env"
|
||||
command -v nginx >/dev/null || fail "nginx 未安装(蓝绿依赖 nginx 反代,见 RUN.md §蓝绿部署 bootstrap)"
|
||||
[ -f "$UPSTREAM_FILE" ] || fail "缺 $UPSTREAM_FILE(nginx upstream 未初始化,见 RUN.md §蓝绿部署 bootstrap)"
|
||||
systemctl cat zcbot@blue >/dev/null 2>&1 || fail "模板 unit 未安装(sudo cp deploy/zcbot@.service /etc/systemd/system/ && daemon-reload)"
|
||||
if systemctl is-active --quiet zcbot 2>/dev/null; then
|
||||
fail "旧单实例服务 zcbot.service 还在跑(占 8765 与 zcbot@blue 冲突)—— 先走 RUN.md §蓝绿部署 bootstrap 停掉它"
|
||||
fi
|
||||
DB_URL="$(grep -E '^[[:space:]]*ZCBOT_DB_URL=' "$APP_DIR/.env" | tail -n1 | cut -d= -f2-)"
|
||||
[ -n "$DB_URL" ] || fail ".env 里没读到 ZCBOT_DB_URL"
|
||||
|
||||
# ── 当前 live 色:从 nginx upstream 文件反推(单一事实源,不另立状态文件)──
|
||||
detect_live() {
|
||||
if grep -q ":${PORT[blue]}" "$UPSTREAM_FILE"; then echo blue
|
||||
elif grep -q ":${PORT[green]}" "$UPSTREAM_FILE"; then echo green
|
||||
else fail "$UPSTREAM_FILE 内容不认识(应含 127.0.0.1:${PORT[blue]} 或 :${PORT[green]}):$(cat "$UPSTREAM_FILE")"
|
||||
fi
|
||||
}
|
||||
|
||||
wait_health() { # $1=color;实例 /healthz 返 ok 且 instance 字段对得上
|
||||
local color="$1" url="http://127.0.0.1:${PORT[$color]}/healthz" body=""
|
||||
for _ in $(seq 1 30); do
|
||||
body="$(curl -fsS --max-time 2 "$url" 2>/dev/null || true)"
|
||||
if echo "$body" | grep -q '"ok"'; then
|
||||
echo "$body" | grep -q "\"instance\":\"$color\"" \
|
||||
|| fail "$url 返回的 instance 不是 $color(实例 env 没生效?):$body"
|
||||
log "zcbot@$color 健康:$body"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "[deploy] ----- journalctl -u zcbot@$color -n 50 -----" >&2
|
||||
journalctl -u "zcbot@$color" -n 50 --no-pager >&2 || true
|
||||
fail "$url 30s 内没返回 ok(看上面日志)"
|
||||
}
|
||||
|
||||
switch_nginx() { # $1=color;改 upstream + 校验 + reload,校验失败回滚原文件
|
||||
local color="$1" backup
|
||||
backup="$(cat "$UPSTREAM_FILE")"
|
||||
echo "server 127.0.0.1:${PORT[$color]};" > "$UPSTREAM_FILE"
|
||||
if ! nginx -t 2>&1; then
|
||||
echo "$backup" > "$UPSTREAM_FILE"
|
||||
fail "nginx -t 校验失败,upstream 已回滚"
|
||||
fi
|
||||
systemctl reload nginx 2>/dev/null || nginx -s reload
|
||||
log "nginx upstream → zcbot@$color(127.0.0.1:${PORT[$color]})"
|
||||
}
|
||||
|
||||
# ── --switch:只调流量,不部署 ─────────────────────────────────
|
||||
if [ -n "$SWITCH_TO" ]; then
|
||||
[[ "$SWITCH_TO" == "blue" || "$SWITCH_TO" == "green" ]] || fail "--switch 只接受 blue|green"
|
||||
systemctl is-active --quiet "zcbot@$SWITCH_TO" || fail "zcbot@$SWITCH_TO 没在跑,先 systemctl start"
|
||||
wait_health "$SWITCH_TO"
|
||||
switch_nginx "$SWITCH_TO"
|
||||
log "OK:流量已切到 zcbot@$SWITCH_TO(另一实例未动,自行决定 stop/disable)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
LIVE="$(detect_live)"
|
||||
if [ "$LIVE" = "blue" ]; then IDLE="green"; else IDLE="blue"; fi
|
||||
log "当前 live=$LIVE,本次部署到 idle=$IDLE"
|
||||
|
||||
# ── 1) 准备:拉码 / 依赖 / migration / sandbox 镜像(复用 update.sh)──
|
||||
# migration 先于切换跑 —— 项目纪律要求 schema 改动平滑兼容存量代码,旧实例(旧代码)
|
||||
# 在新 schema 上继续 drain 是安全的。
|
||||
OLD_HEAD="$(asuser git -C "$APP_DIR" rev-parse HEAD)"
|
||||
bash "$APP_DIR/deploy/update.sh" --prepare-only "${SKIP_BUILD_ARG[@]}"
|
||||
NEW_HEAD="$(asuser git -C "$APP_DIR" rev-parse HEAD)"
|
||||
|
||||
# 本脚本自身被 pull 更新 → 用新版本重跑(同 update.sh 的自更新逻辑;里面的
|
||||
# prepare 已幂等,重跑秒过)。ZCBOT_UPDATE_BG_REEXEC 防死循环。
|
||||
if [ "$OLD_HEAD" != "$NEW_HEAD" ] && [ "${ZCBOT_UPDATE_BG_REEXEC:-0}" != "1" ] \
|
||||
&& ! asuser git -C "$APP_DIR" diff --quiet "$OLD_HEAD" "$NEW_HEAD" -- deploy/update_bluegreen.sh; then
|
||||
log "update_bluegreen.sh 自身有更新 —— 用新版本重跑..."
|
||||
exec env ZCBOT_UPDATE_BG_REEXEC=1 bash "$APP_DIR/deploy/update_bluegreen.sh" "${ORIG_ARGS[@]}"
|
||||
fi
|
||||
|
||||
# ── 2) 空闲实例起新代码 ────────────────────────────────────────
|
||||
log "systemctl restart zcbot@$IDLE ..."
|
||||
systemctl restart "zcbot@$IDLE"
|
||||
|
||||
# ── 3) 验活(healthz + instance 字段)──────────────────────────
|
||||
wait_health "$IDLE"
|
||||
|
||||
# ── 4) nginx 切流量:新请求全走 $IDLE,旧 SSE 留 $LIVE 上不断 ──
|
||||
switch_nginx "$IDLE"
|
||||
|
||||
# ── 5) 等旧实例在跑的 run 自然清空(SSE 直播不断的关键)──────────
|
||||
# 新流量已全走新实例;旧实例上的 run 继续跑、SSE 继续推(nginx reload 不影响已建立
|
||||
# 连接)。直接 stop 的话 uvicorn timeout_graceful_shutdown=5 会 5s 掐掉长连接 ——
|
||||
# run 能跑完但用户看到断流,所以先等 DB 里 run_owner=旧色 的活跃行归零再停。
|
||||
# 超 DRAIN_WAIT_SECONDS 的转 stop 的 SIGTERM drain(协作式 cancel,标 idle 可重发)。
|
||||
count_live_runs() {
|
||||
asuser env ZCBOT_DB_URL="$DB_URL" "$PY" - "$LIVE" <<'PYEOF'
|
||||
import os, sys
|
||||
from sqlalchemy import create_engine, text
|
||||
eng = create_engine(os.environ["ZCBOT_DB_URL"])
|
||||
with eng.connect() as c:
|
||||
print(c.execute(text(
|
||||
"SELECT count(*) FROM tasks "
|
||||
"WHERE run_status IN ('running','cancelling') AND run_owner=:o"
|
||||
), {"o": sys.argv[1]}).scalar())
|
||||
PYEOF
|
||||
}
|
||||
log "等 zcbot@$LIVE 在跑的 run 自然收尾(上限 ${DRAIN_WAIT_SECONDS}s)..."
|
||||
waited=0
|
||||
while [ "$waited" -lt "$DRAIN_WAIT_SECONDS" ]; do
|
||||
n="$(count_live_runs)"
|
||||
if [ "$n" = "0" ]; then
|
||||
log "zcbot@$LIVE 已无在跑 run"
|
||||
break
|
||||
fi
|
||||
log " $LIVE 还有 $n 个 run 在跑,${waited}s/${DRAIN_WAIT_SECONDS}s ..."
|
||||
sleep 10
|
||||
waited=$((waited + 10))
|
||||
done
|
||||
|
||||
# ── 6) 停旧实例(通常已空,秒停;残留的走 SIGTERM drain)────────
|
||||
log "systemctl stop zcbot@$LIVE ..."
|
||||
systemctl stop "zcbot@$LIVE"
|
||||
|
||||
# ── 7) DB 收尾:旧色被 SIGKILL 兜底砍掉的 run(没来得及收尾的)标 error ──
|
||||
# 正常路径这里是 0 行;非 0 说明有 run 超上限被硬停,对应 task 重新可发消息。
|
||||
asuser env ZCBOT_DB_URL="$DB_URL" "$PY" - "$LIVE" <<'PYEOF'
|
||||
import os, sys
|
||||
from sqlalchemy import create_engine, text
|
||||
color = sys.argv[1]
|
||||
eng = create_engine(os.environ["ZCBOT_DB_URL"])
|
||||
with eng.begin() as c:
|
||||
r = c.execute(text(
|
||||
"UPDATE tasks SET run_status='error', "
|
||||
"run_error='instance stopped before run finished' "
|
||||
"WHERE run_status IN ('running','cancelling') AND run_owner=:o"
|
||||
), {"o": color})
|
||||
print(f"[deploy] reaped {r.rowcount} leftover run(s) owned by {color}")
|
||||
PYEOF
|
||||
|
||||
# ── 8) 开机自启只留新 live 色 ──────────────────────────────────
|
||||
systemctl enable "zcbot@$IDLE" >/dev/null 2>&1 || true
|
||||
systemctl disable "zcbot@$LIVE" >/dev/null 2>&1 || true
|
||||
|
||||
log "OK:蓝绿切换完成($NEW_HEAD),live=$IDLE,旧实例 $LIVE 已停"
|
||||
log "看实时日志:journalctl -u zcbot@$IDLE -f"
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# zcbot 蓝绿双实例模板 unit(RUN.md §无感更新 B 档)。
|
||||
# 安装:sudo cp deploy/zcbot@.service /etc/systemd/system/ && sudo systemctl daemon-reload
|
||||
# 实例:zcbot@blue(port 8765)/ zcbot@green(port 8766),端口与实例名由
|
||||
# deploy/instances/<实例>.env 注入(ZCBOT_PORT / ZCBOT_INSTANCE)。
|
||||
# nginx upstream 在两实例间切流量,部署走 deploy/update_bluegreen.sh。
|
||||
[Unit]
|
||||
Description=zcbot web (FastAPI/uvicorn) - instance %i
|
||||
After=network-online.target postgresql.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=zcbot
|
||||
WorkingDirectory=/opt/zcbot
|
||||
# 先装 .env(业务配置),再装实例 env(ZCBOT_PORT/ZCBOT_INSTANCE,后装覆盖前者同名项)
|
||||
EnvironmentFile=/opt/zcbot/.env
|
||||
EnvironmentFile=/opt/zcbot/deploy/instances/%i.env
|
||||
# 只绑 127.0.0.1:对外统一走 nginx,实例端口不直接暴露
|
||||
ExecStart=/opt/zcbot/.venv/bin/python main.py web --host 127.0.0.1 --port ${ZCBOT_PORT}
|
||||
Restart=on-failure
|
||||
RestartSec=2
|
||||
KillSignal=SIGTERM
|
||||
# ★ 优雅 drain:SIGTERM 后先拒新 run(503)、等在跑的 run 收尾(config/agent.yaml
|
||||
# `shutdown` 段:drain_timeout 30s + cancel_grace 15s)。TimeoutStopSec 必须 >
|
||||
# drain_timeout + cancel_grace + 余量(sandbox 容器清扫 + uvicorn graceful 5s),
|
||||
# 否则 systemd 中途 SIGKILL 把 drain 砍掉。改 agent.yaml 那两个数值时这里跟着调。
|
||||
TimeoutStopSec=90
|
||||
KillMode=mixed
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
113
web/app.py
113
web/app.py
|
|
@ -32,7 +32,7 @@ from fastapi import Depends, FastAPI, File, Form, HTTPException, Request, Upload
|
|||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import FileResponse, RedirectResponse, StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import BigInteger, cast, func, select, update
|
||||
from sqlalchemy import BigInteger, cast, func, or_, select, update
|
||||
from starlette.background import BackgroundTask
|
||||
|
||||
from core import __version__
|
||||
|
|
@ -63,6 +63,12 @@ from .sinks import WebEventSink
|
|||
from .static_files import NoCacheStaticFiles
|
||||
|
||||
|
||||
# 蓝绿双实例部署(RUN.md B 档):实例名(blue/green),由 systemd 模板 unit 的
|
||||
# per-instance env 注入;单实例部署不设 = ""。用途:① 起 run 时写 tasks.run_owner,
|
||||
# 启动 reaper 只收自己实例的孤儿 ② sandbox 容器名/label 带实例色互不清扫(pool.py 同款
|
||||
# 读法)③ /healthz 返回,蓝绿切换时验证 nginx 已指到新实例。
|
||||
INSTANCE = os.getenv("ZCBOT_INSTANCE", "").strip()
|
||||
|
||||
STATUS_FILTERS = ("active", "completed", "abandoned")
|
||||
STATUS_WRITABLE = ("completed", "abandoned") # web 不让从 web 端切回 active(走 CLI)
|
||||
# 渠道镜像 task 的 channel 取值(每用户每渠道一条常驻只读对话):从普通任务列表排除,
|
||||
|
|
@ -570,7 +576,7 @@ async def _run_channel_conversation(app, uid, text, attachments, *, channel):
|
|||
if row.run_status in ("running", "cancelling"):
|
||||
return "上一条还在处理中,请稍候再发。"
|
||||
s.execute(update(Task).where(Task.task_id == tid).values(
|
||||
run_status="running", run_error=None))
|
||||
run_status="running", run_error=None, run_owner=INSTANCE or None))
|
||||
|
||||
broker.start(tid)
|
||||
runner = asyncio.create_task(asyncio.to_thread(
|
||||
|
|
@ -788,12 +794,17 @@ def create_app() -> FastAPI:
|
|||
|
||||
# Stale-run reaper:上次进程 crash 留下的 "running" / "cancelling" 已无 BG 线程
|
||||
# 继续,启动时标 error,让对应 task 重新可发消息(否则 gate 永挂)。
|
||||
# TODO 真生产 multi-worker:换 heartbeat / lease,只 reap 自家 worker 的孤儿。
|
||||
# 蓝绿双实例(RUN.md B 档):ZCBOT_INSTANCE 在时只收 run_owner=自己(上次本色
|
||||
# 实例的孤儿)或 NULL(0020 前的遗留行)的,不动另一实例正在跑的 run;单实例
|
||||
# 部署 INSTANCE="" → 全量收,行为不变。
|
||||
with session_scope() as s:
|
||||
stmt = update(Task).where(Task.run_status.in_(("running", "cancelling")))
|
||||
if INSTANCE:
|
||||
stmt = stmt.where(
|
||||
or_(Task.run_owner == INSTANCE, Task.run_owner.is_(None))
|
||||
)
|
||||
result = s.execute(
|
||||
update(Task)
|
||||
.where(Task.run_status.in_(("running", "cancelling")))
|
||||
.values(
|
||||
stmt.values(
|
||||
run_status="error",
|
||||
run_error="server restarted before run finished",
|
||||
)
|
||||
|
|
@ -939,7 +950,8 @@ def create_app() -> FastAPI:
|
|||
print(f"[scheduler] job {str(job_id)[:8]} skipped (task busy)")
|
||||
return
|
||||
s.execute(update(Task).where(Task.task_id == tid).values(
|
||||
run_status="running", run_error=None))
|
||||
run_status="running", run_error=None,
|
||||
run_owner=INSTANCE or None))
|
||||
|
||||
message = build_run_message(snap)
|
||||
broker.start(tid)
|
||||
|
|
@ -1046,11 +1058,86 @@ def create_app() -> FastAPI:
|
|||
|
||||
if clawbot_enabled():
|
||||
from core.wechat.inbound import run_inbound_manager
|
||||
wechat_task = asyncio.create_task(
|
||||
|
||||
async def _wechat_leader() -> None:
|
||||
"""ClawBot 入站长轮询单实例互斥(蓝绿 B 档,RUN.md)。
|
||||
|
||||
getupdates 长轮询是消费型的:蓝绿并存期两实例同时拉会抢消息 / 同一条
|
||||
入站双跑 agent。用 PG advisory lock 选主:持锁实例才跑 inbound manager,
|
||||
另一实例每 15s 重试;持锁连接断(DB 重启 / 网络抖 = 锁已自动释放)→
|
||||
停管理器重新竞争。单实例部署恒拿到锁,行为与之前完全一致。
|
||||
"""
|
||||
from core.storage import get_engine
|
||||
engine = get_engine()
|
||||
# pg_try_advisory_lock(int4, int4) 的固定 key:('zc'=0x7A63, 1=wechat-inbound)
|
||||
lock_sql = "SELECT pg_try_advisory_lock(31331, 1)"
|
||||
|
||||
def _acquire():
|
||||
conn = engine.connect()
|
||||
# 从连接池摘出独占:advisory lock 是 DBAPI 连接级的,不 detach 的话
|
||||
# close() 只是还池,锁挂在池化连接上既不释放也没人持有
|
||||
conn.detach()
|
||||
try:
|
||||
if conn.exec_driver_sql(lock_sql).scalar():
|
||||
return conn
|
||||
conn.close()
|
||||
return None
|
||||
except Exception:
|
||||
conn.close()
|
||||
raise
|
||||
|
||||
async def _wait(seconds: float) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(wechat_stop.wait(), timeout=seconds)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
|
||||
while not wechat_stop.is_set():
|
||||
conn = None
|
||||
mgr = None
|
||||
try:
|
||||
conn = await asyncio.to_thread(_acquire)
|
||||
if conn is None:
|
||||
await _wait(15)
|
||||
continue
|
||||
print("[wechat] inbound leader lock acquired, manager starting")
|
||||
mgr = asyncio.create_task(
|
||||
run_inbound_manager(_run_wechat_message, wechat_stop),
|
||||
name="wechat-inbound",
|
||||
)
|
||||
print("[wechat] ClawBot inbound manager enabled")
|
||||
# 30s keepalive ping 锁连接;ping 挂 = 连接断 = 锁已丢 → 重选
|
||||
while not wechat_stop.is_set() and not mgr.done():
|
||||
await _wait(30)
|
||||
if wechat_stop.is_set() or mgr.done():
|
||||
break
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
lambda: conn.exec_driver_sql("SELECT 1").scalar()
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"[wechat] leader lock lost "
|
||||
f"({type(e).__name__}), re-electing")
|
||||
break
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"[wechat] leader loop error: {type(e).__name__}: {e}")
|
||||
await _wait(15)
|
||||
finally:
|
||||
if mgr is not None and not mgr.done():
|
||||
mgr.cancel()
|
||||
try:
|
||||
await mgr
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
if conn is not None:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
wechat_task = asyncio.create_task(_wechat_leader(), name="wechat-leader")
|
||||
print("[wechat] ClawBot inbound enabled (PG advisory lock leader election)")
|
||||
|
||||
# Sandbox pool(§7.5):仅当 ZCBOT_SANDBOX_BACKEND=docker 时启用。
|
||||
# 启动钩子:① init_pool(创建 docker network + pool 实例)② shutdown_all 清
|
||||
|
|
@ -1205,7 +1292,8 @@ def create_app() -> FastAPI:
|
|||
|
||||
@app.get("/healthz", tags=["misc"])
|
||||
def healthz():
|
||||
return {"status": "ok", "version": __version__}
|
||||
# instance:蓝绿部署时验证打到的是哪个实例(blue/green);单实例为 null
|
||||
return {"status": "ok", "version": __version__, "instance": INSTANCE or None}
|
||||
|
||||
@app.get("/WW_verify_{token}.txt", include_in_schema=False)
|
||||
def wecom_domain_verify(token: str):
|
||||
|
|
@ -2400,7 +2488,10 @@ def create_app() -> FastAPI:
|
|||
f"task already has an active run (status={row.run_status}); "
|
||||
f"wait for it to finish or cancel",
|
||||
)
|
||||
values: dict = {"run_status": "running", "run_error": None}
|
||||
values: dict = {
|
||||
"run_status": "running", "run_error": None,
|
||||
"run_owner": INSTANCE or None,
|
||||
}
|
||||
# 档位门控:存量 task 的模型已不在用户档位内(如管理员下调了档位)→ 本次起
|
||||
# 持久落回 flash(基线必含),UI 下拉随之显示 flash。不报错、不打断会话历史,
|
||||
# 符合"老 task 下次发消息直接切 flash"。当前 task 模型仍在档内则原样不动。
|
||||
|
|
|
|||
Loading…
Reference in New Issue