diff --git a/CHANGELOG.md b/CHANGELOG.md index 21fde22..f1ec9f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ > 开发中的用户文案可先写入 `## Unreleased`；该区不会被前端解析，正式发布时再替换为数字版本和日期。 > 工程口径的完整记录见 `PROGRESS.md` / git log。 +## Unreleased + +- GLM-5.3 Flash 替换旧版 GLM 并向默认档位开放；上传图片会直接交给主模型理解，长工具任务可延续既有分析状态。 + ## 0.69.1 — 2026-08-27 - 专业软件任务的进度与结果改为在当前对话左侧集中堆叠，不再用多条结果消息打断正文；可直接打开 Blender 等软件的多张预览，手机端从“软件结果”入口统一查看。 diff --git a/DESIGN.md b/DESIGN.md index d90db3a..bca5a59 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -136,7 +136,7 @@ Eval 与生产 core 解耦，通过现有 `/v1` API 创建专用任务、监听 默认 `deepseek_v4.flash`;复杂 bug / 终稿升 pro + reasoning_effort=max;fallback 手动切 Claude。成本量级:修 bug flash ~$0.01 / 完整申报书 flash ~$0.30(pro-max ~$1.5,Opus ~$10+)。99% 任务 flash 够用。 -模型思考参数由 profile 统一表达：`thinking_enabled` 只表示开关，`thinking_transport` 只表示已验证的传输协议，`reasoning_effort` 只表示开启后的推理强度，`reasoning_replay` 只表示历史 reasoning 的 provider 回传策略（`none` / `tool_calls` / `all`）；`core/llm_params.py` 是请求参数构造唯一入口，`core/context.py` 是历史消息清洗唯一入口。原始 assistant 响应仍完整落库供展示与导出，发模型前才按 profile 裁剪，且上下文统计、压缩与折叠都使用裁剪后的请求视图。DeepSeek、GLM、方舟当前共享 `extra_body` 协议，DeepSeek V4 仅为带工具调用的 assistant 消息保留 reasoning，未验证网关明确用 `none`、不猜参数协议，主循环不再按 family 分支。`/v1/models` 只返回语义明确的 `thinking_enabled`。 +模型思考参数由 profile 统一表达：`thinking_enabled` 只表示开关，`thinking_transport` 只表示已验证的传输协议，`reasoning_effort` 只表示开启后的推理强度，`thinking_clear` 表示 provider 是否清除历史思考，`reasoning_replay` 表示状态生命周期（`none` / `tool_turn` / `conversation` / `provider_managed`）；`core/llm_params.py` 是请求参数构造唯一入口，`core/context.py` 是历史消息清洗唯一入口。原始 assistant 响应完整落库，provider-bound 副本只向相同生产模型回放未改写的 reasoning：DeepSeek V4 仅保留当前用户轮次的工具链状态，GLM-5.3 Flash 在同模型会话内保留完整状态，未验证网关明确用 `none`，未来签名/加密 block 走 `provider_managed`。模型切换、上下文折叠和普通压缩都在同一入口应用隔离；上下文统计使用裁剪后的请求视图，原生图片 token 不反向污染 chars/token 校准。 --- @@ -184,7 +184,7 @@ Eval 与生产 core 解耦，通过现有 `/v1` API 创建专用任务、监听 **对话产物与生命周期(0025/0028/0033)**:真实文件仍是内容事实源；`artifacts` 表记录已发布产物的稳定身份和生命周期，包含 user-root 相对当前路径、来源 task、复制来源、可空的 `software_job_id`、哈希/大小及 active/deleted、回收路径。`software_job_id` 不设外键，非空即表示该正式产物由对应专业软件 Job 生成；复制品保留该来源，Job 或 task 生命周期结束也不抹除来源标识。新 `messages.artifact_refs` 使用 `{version:2, artifact_id, scope:"working_dir", path:"reports/a.pdf", label?:"最终报告"}`；`path` 是兼容快照，预览/下载优先按 `artifact_id` 找当前路径，因此移动或重命名后历史卡片仍有效。version 1 和 `NULL` 旧消息继续走原 task-scoped 兼容链。普通源码树、中间文件、`.meta/` 技术信息和配套资源不登记；agent 仅用 `publish_artifacts` 显式提升少量最终文件。移动保持身份，复制为每个副本创建新身份并记录直接来源；删除将文件移入 `.zcbot_artifact_trash/` 并软删记录，普通文件仍物理删除。 -**用户消息附件(0031)**:`messages.attachment_refs` 与助手产物分开表达输入附件，元素为 `{version:1, scope:"working_dir", path, label, kind, media_type, size_bytes}`；文件仍是事实源，不登记为已发布 artifact，也不承诺独立生命周期。`payload.content` 只保存用户自然语言，模型上下文在内存中按附件类型补兼容提示，避免 UI 协议污染正文。`NULL` 表示旧客户端/旧历史，前端继续解析正文标记；`[]` 表示新消息明确无附件。独立 attachment 表暂缓，只有出现跨消息复用、稳定身份、版本快照或附件级审计需求时再抽象 `file_assets + message_attachments`。 +**用户消息附件(0031)**:`messages.attachment_refs` 与助手产物分开表达输入附件，元素为 `{version:1, scope:"working_dir", path, label, kind, media_type, size_bytes}`；文件仍是事实源，不登记为已发布 artifact，也不承诺独立生命周期。`payload.content` 只保存用户自然语言，Session 内存补兼容路径提示并保留私有附件引用；发模型前按 `input_modalities` 决定纯文本退化或把图片物化为 OpenAI-compatible Base64 `image_url` block，Base64 永不落库、物化仍经过 working_dir/user_root 越界与大小校验。原生看图模型不注册 `look_at_image`，`read_document` 继续承担扫描 PDF 等文档摄取。`NULL` 表示旧客户端/旧历史，`[]` 表示新消息明确无附件；独立 attachment 表仍待跨消息复用、稳定身份、版本快照或附件级审计需求出现后再抽象。 ### 7.2 资源模型(/v1) @@ -316,7 +316,7 @@ scheduled_jobs(§8.5) channel_bindings(§8.7,判别列+JSONB) ### 8.1 图像理解 + Seedream i2i(✅ 2026-06-16) -缺口:主模型纯文本;t2i 无法"改已生成图"或"读上传图"。**选 E+C 组合**:seedream 加 `reference_images` 走 i2i + 新增 `look_at_image`(Doubao Seed 2.0 Lite,一次读图 <¥0.01)让 DeepSeek 自决何时"借眼睛"。**不选 A**(主模型换多模态:V4 code/tool-calling 是核心,换=降能力+改 loop 引 multimodal,工程 5×);**不选 B**(每条消息隐式 vision 路由:烧 token+失 agentic 控制权)。关键实测:ARK 接受 base64 data URL → 内网无需对象存储。**升级到 A 的信号**:用户要"贴图直接对话读图"成高频——当前假设"图是工具调用对象"而非"对话内容"。 +历史基线是纯文本主模型配 `look_at_image` 借眼睛，生图改图独立走 `reference_images`。2026-08-27 随 GLM-5.3 Flash 原生多模态进入默认档位，升级信号已满足：附件输入改为 capability-driven，原生图片模型直接收到用户上传图片，纯文本模型继续使用路径提示与辅助视觉工具；不会把所有文件或工具产物隐式送入模型，避免无关 token、越界读取和缓存污染。生图/改图仍是显式外部动作，原生理解不替代图像生成工具；运行中产生的新图片若未作为后续用户附件加入，不自动注入下一轮。 ### 8.2 Token 优化与上下文治理(✅ 2026-06-04 起) diff --git a/PROGRESS.md b/PROGRESS.md index d5485ac..2f8887a 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,7 +2,7 @@ > 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 1-2 句:做了啥 + 关键判断;细节查 `git log` / `git diff` / `DESIGN §7.9`。 -最后更新:2026-08-27(专业软件结果轨与 Web 终态收敛，bump 0.69.1) +最后更新:2026-08-27(GLM-5.3 Flash、原生图片输入与 reasoning 生命周期统一；未发版) --- @@ -20,6 +20,8 @@ --- ## 已完成关键能力 +- **08-27 / Unreleased / GLM-5.3 Flash + 通用 reasoning/多模态能力**：下架 GLM-5.1/5.2 的可选入口并以隐藏别名将存量 `glm.pro/pro52` 统一解析到 `glm.flash53`，Flash53 加入默认与专业档位；模型能力新增原生输入模态和 `none/tool_turn/conversation/provider_managed` reasoning 生命周期，历史状态仅向同一生产模型回放，DeepSeek 限当前工具轮、GLM 保留同模型会话并发送 `clear_thinking=false`。结构化图片附件仍以文件引用为事实源，只在 provider 请求边界安全物化为 Base64 `image_url`，GLM 不再注册 `look_at_image`，文档读取与生图/视频工具保持独立；无 schema/migration/依赖变化，未调用真实模型或连接生产数据库。 + - **08-27 / 0.69.1 / 专业软件结果轨与终态收敛**：当前对话的专业软件 Job 按独立状态点堆叠在聊天左侧，悬停展示名称与进度，成功的 Workspace 任务可直接打开多预览画廊，移动端收敛为“软件结果”入口；主消息流只保留用户与 Agent 叙事，默认 `report` 完成后只更新 Job 账本和视图，显式 `analyze` 才重新进入 Agent 对话。Web SSE 终态统一延后到任务状态落库之后发送，结果刷新会清理不再活动的 live card，避免软件已出图但“思考中”被重新点亮。既有历史软件报告消息继续兼容显示，无 schema、migration 或 HTTP API 变化。 - **08-26 / 0.69.0 / Origin 科研分析 MVP**：新增非默认的 `origin.analysis@v1` / adapter 0.1.0，与稳定的 `origin.plot@v2` 分离；首批提供数据体检、最大值/面积/参考点归一化、固定局部多项式平滑、一/二阶导数、区间/累计梯形积分和带 95% 置信区间的线性拟合，固定输出可编辑 OPJU、CSV、诊断、分析规格与溯源，可选 XLSX，线性拟合工程同时保留 Origin 原生报告。六个合成真值用例在 Origin 2024 / originpro 1.1.15 / numpy 2.2.6 上完成生产 Worker、数值 oracle、OPJU 重开和进程释放门；前三例来自组合执行的逐例通过记录，后三例分别通过，目标机 COM 完整退出约需 2 分钟。Origin/合同/Job/Node 专项 159 项 unittest、Release .NET build、完整 Node publish 与独立 adapter ZIP 校验通过，未连接或写入数据库。 diff --git a/RUN.md b/RUN.md index 1530dd2..c108538 100644 --- a/RUN.md +++ b/RUN.md @@ -215,7 +215,7 @@ python -m venv .venv ```bash # 模型能力对账(费 token) .venv/Scripts/python.exe main.py probe --model deepseek_v4.flash -.venv/Scripts/python.exe main.py probe --model glm.pro # 智谱 GLM-5.1(走 litellm zai provider + 国内站 bigmodel.cn) +.venv/Scripts/python.exe main.py probe --model glm.flash53 # 智谱 GLM-5.3 Flash(原生多模态，国内站 bigmodel.cn) .venv/Scripts/python.exe main.py probe --model local.r1 # 内网 DeepSeek-R1(满血,128K),涉密任务用;需 .env 设 LOCAL_LLM_API_KEY .venv/Scripts/python.exe main.py probe --model local.qwen3 # 内网 Qwen3-30B-A3B(MoE,原生 32K);共享 LOCAL_LLM_API_KEY diff --git a/config/agent.yaml b/config/agent.yaml index 621b114..8c781a5 100644 --- a/config/agent.yaml +++ b/config/agent.yaml @@ -13,6 +13,7 @@ model_tiers: default: # 基线:所有未分配档位的用户(= 公测期默认可用) - deepseek_v4.flash - deepseek_v4.pro + - glm.flash53 - local.r1 # 内网模型(涉密任务) - local.qwen3 - seedream_5 # 图(config/media/doubao.yaml image 段) @@ -26,8 +27,7 @@ model_tiers: - doubao.turbo - doubao.pro - doubao.evolving - - glm.pro - - glm.pro52 + - glm.flash53 - unifyllm.fable5 # unifyllm 网关(海外旗舰,走代理出口) - unifyllm.opus48 - unifyllm.sonnet46 diff --git a/config/models/deepseek_v4.yaml b/config/models/deepseek_v4.yaml index 3a5d6aa..7e0f119 100644 --- a/config/models/deepseek_v4.yaml +++ b/config/models/deepseek_v4.yaml @@ -16,7 +16,7 @@ variants: thinking_transport: extra_body reasoning_effort_levels: [low, high, max] default_reasoning_effort: high - reasoning_replay: tool_calls # 普通跨轮剥离；工具调用 reasoning 按官方协议完整回传 + reasoning_replay: tool_turn # 只在当前用户轮次的工具链内原样回传 reasoning code_quality: good enable_run_python: true max_iterations: 120 # backstop 兜底,非"轮"预算;真正的空转防护是 loop 的无进展熔断 + _RepeatGuard @@ -42,7 +42,7 @@ variants: thinking_transport: extra_body reasoning_effort_levels: [low, medium, high, max] default_reasoning_effort: medium - reasoning_replay: tool_calls + reasoning_replay: tool_turn code_quality: excellent enable_run_python: true max_iterations: 150 # backstop 兜底,非"轮"预算;真正的空转防护是 loop 的无进展熔断 + _RepeatGuard diff --git a/config/models/glm.yaml b/config/models/glm.yaml index 8064906..aa9f158 100644 --- a/config/models/glm.yaml +++ b/config/models/glm.yaml @@ -1,55 +1,43 @@ # 智谱 GLM 模型档案 -# 走 litellm 原生 zai provider(1.83+ 内置)。litellm 默认 api_base 是国际站 api.z.ai; -# 国内站 bigmodel.cn 通过 yaml 的 api_base 字段覆盖。两站 API key 不通用,env 也分开命名。 -# thinking 走统一 extra_body 协议,由本档 `thinking_enabled` 决定开关。当前均 false=禁用 —— -# 因网关侧默认开 thinking,重任务上会把输出预算烧在 reasoning_content 上撞满 65536 上限 -# 被截断→空响应(task 35744bea 案);禁用后线上探针实测 reasoning_content 归零、正文照常。 +# 走 LiteLLM 原生 zai provider。国内站 bigmodel.cn 与国际站 API key 不通用。 +# GLM-5.3-Flash 强制 thinking；zcbot 保留同模型完整 reasoning 以维持长工具链连续性。 family: glm variants: - pro: - display_name: GLM 5.1 - model_id: zai/glm-5.1 + flash53: + display_name: GLM-5.3 Flash + model_id: zai/glm-5.3-flash api_base: https://open.bigmodel.cn/api/paas/v4 api_key_env: ZHIPUAI_API_KEY - max_context: 200000 - reliable_context: 100000 - max_output: 8192 - parallel_tools: false - tool_calling_quality: good - thinking_enabled: false - thinking_transport: extra_body - reasoning_effort_levels: [] - default_reasoning_effort: "" - code_quality: good - enable_run_python: true - max_iterations: 50 - optimal_temperature: 0.3 - prompt_caching: false - extended_thinking: false - - # GLM 5.2:与 5.1 并存(新增 variant,不动 glm.pro,线上 task 仍引 5.1 不受影响)。 - # 旗舰基座,真正可用的 1M 上下文,适合大仓库/长链路工程任务。thinking 同 pro 留 false(协议同 5.1)。 - pro52: - display_name: GLM 5.2 - model_id: zai/glm-5.2 - api_base: https://open.bigmodel.cn/api/paas/v4 - api_key_env: ZHIPUAI_API_KEY - max_context: 1000000 # 真 1M + max_context: 1000000 reliable_context: 262144 - max_output: 8192 + max_output: 131072 + input_modalities: [text, image] parallel_tools: false - tool_calling_quality: good - thinking_enabled: false + tool_calling_quality: excellent + thinking_enabled: true thinking_transport: extra_body - reasoning_effort_levels: [] - default_reasoning_effort: "" + thinking_clear: false + reasoning_effort_levels: [low, high, max] + default_reasoning_effort: high + reasoning_replay: conversation code_quality: excellent enable_run_python: true - max_iterations: 50 - optimal_temperature: 0.3 + max_iterations: 120 + optimal_temperature: 1.0 prompt_caching: false extended_thinking: false - input_cny_per_mtoken: 8.0 - output_cny_per_mtoken: 28.0 - cache_hit_cny_per_mtoken: 2.0 + # 官方国际站标价按 1 USD ~= 7.2 CNY 折算；促销价不写入长期兜底。 + input_cny_per_mtoken: 1.08 + output_cny_per_mtoken: 3.6 + cache_hit_cny_per_mtoken: 0.216 + + # 隐藏兼容别名：真实调用、能力和记账身份全部归 glm.flash53。 + # 待一个兼容版本确认无旧 profile 活跃后删除。 + pro: + hidden: true + alias_of: flash53 + + pro52: + hidden: true + alias_of: flash53 diff --git a/core/agent_builder.py b/core/agent_builder.py index 8347303..078dbda 100644 --- a/core/agent_builder.py +++ b/core/agent_builder.py @@ -56,10 +56,12 @@ _MEDIA_LOOK_SEG = """\ - **何时调**:用户消息里出现 `[用户上传的参考图] <路径>` 且需要据图内容回答(问"这图里写了啥 / 是什么 / 表格数据多少");或要基于 task 内某张图(`figures/xxx.png`)的**实际内容**做事(不是改图,改图走 seedream)。传 `image=<路径>`，并且**必须在 `question` 里一次写清完成当前任务所需的具体信息**，别分多次看。 - **问题范围**:只问完成用户任务所需内容。例如问仪表读数就只读数，问某列数据就只提取该列；仅当用户明确要求"全文 OCR / 转文字 / 完整识别整张图"时，才要求逐字识别全部文字。不要无条件同时要求画面描述、全文 OCR 和图表解析。 - **何时不调**:用户只是要改图(走 seedream i2i)/ 只要文件名不关心内容 / 图是你自己刚生成的且 prompt 已知(无需再读)/ 图的内容对当前任务可有可无。**绝不对同一张图反复看**(每看一次都是几十秒等待)。""" +_MEDIA_NATIVE_LOOK_SEG = """\ +- **原生看图**:你可以直接理解用户随消息上传的图片；图片已作为多模态内容随请求送入，直接据图分析，**不要调用辅助看图工具**。消息中的 `[用户上传的参考图]` 路径仅用于定位同一附件或交给改图工具。""" _MEDIA_READDOC_SEG = """\ - `read_document` —— 读 PDF(豆包 Seed 2.0 Lite 文档理解),**专治扫描件**:markitdown 对某 PDF 转出**空 / 近空**(纯图页无文本层)→ 用它逐页 OCR 成 markdown。每页约 1-2 厘钱,单次上限 100 页(更长先拆分卷)。 - **多页整本 OCR 必传 `save_md`**(如 `save_md='source/xxx.md'`):全文落文件、只返回预览,后续直接 read 那个 md;只问局部内容("第3章指标是什么")传 `question` 即可不落盘。 - - **何时不调**:PDF 有文本层且 markitdown 已转出正文(直接用那份 md,别重复花钱);DOCX/PPTX/XLSX(走 markitdown);单张图片(走 look_at_image)。""" + - **何时不调**:PDF 有文本层且 markitdown 已转出正文(直接用那份 md,别重复花钱);DOCX/PPTX/XLSX(走 markitdown);单张图片由主模型原生读取或走本轮实际提供的看图能力。""" # PDF 摄取是跨 skill 的基础入口,始终注入,不能依赖 agent 是否正确加载 paper / proposal。 # read_document 只在配了 ARK 时挂载;没挂时扫描件明确停在“需 OCR”,不诱导现场装包或 @@ -87,15 +89,20 @@ _MEDIA_SEEDANCE_SEG = """\ - 兜底硬约束:用户没主动要视频就别装饰性生成(比生图更严重的红线);同一目的不满意**绝不连发**(1 次错 = ¥4+60s,连发 2 次 = ¥8+2min);固定开场图传 `image`,多张主体/产品/场景参考图传 `reference_images`；当前不支持 video-to-video。""" -def _media_tools_block(has_ark: bool, image_tool: str) -> str: +def _media_tools_block( + has_ark: bool, image_tool: str, native_image_input: bool = False +) -> str: """按本 run 实际挂载的媒体工具拼指引段;一个都没挂 → 返 ""(prompt 不出现媒体段)。 image_tool: "seedream" / "gpt_image" / ""(无图像工具)。look_at_image 与 seedance 只跟豆包(ARK_API_KEY)走;图像工具二选一,prompt 只描述实际存在的那个。 """ segs: list[str] = [] + if native_image_input: + segs.append(_MEDIA_NATIVE_LOOK_SEG) if has_ark: - segs.append(_MEDIA_LOOK_SEG) + if not native_image_input: + segs.append(_MEDIA_LOOK_SEG) segs.append(_MEDIA_READDOC_SEG) if image_tool == "seedream": segs.append(_MEDIA_SEEDREAM_SEG) @@ -107,7 +114,7 @@ def _media_tools_block(has_ark: bool, image_tool: str) -> str: segs.append(_MEDIA_SEEDANCE_SEG) if not segs: return "" - return "## 媒体工具(图像生成 / 视频生成 / 看图)\n" + "\n".join(segs) + return "## 媒体能力(图像生成 / 视频生成 / 看图)\n" + "\n".join(segs) def _choose_image_variant( @@ -525,6 +532,7 @@ def build_agent( model = cfg["default_model"] caps = ModelCapabilities.load(model, ROOT / cfg["models_dir"]) + model = caps.profile llm = LLM(caps) workspace_dir = resolve_workspace(workspace, cfg) @@ -603,7 +611,9 @@ def build_agent( system_prompt = _build_system_prompt( cfg, skills, workspace_dir, tool_base, working_dir_path, uid, task_id, task_state.name, task_state.skill, - media_block=_media_tools_block(ark_cfg is not None, image_tool_name), + media_block=_media_tools_block( + ark_cfg is not None, image_tool_name, caps.native_image_input + ), allow_working_dir_rename=not scheduled_run, office_to_pdf_available=office_pdf_available, ) @@ -671,6 +681,7 @@ def build_agent( if current_profile.split(".", 1)[0] == "unifyllm": return None new_caps = ModelCapabilities.load(profile, ROOT / cfg["models_dir"]) + profile = new_caps.profile new_llm = LLM(new_caps) # 缺 api key 会 raise → loop 捕获转 warn,不切换 from sqlalchemy import update as _update from core.storage import session_scope as _scope diff --git a/core/attachments.py b/core/attachments.py index 333a55b..cd0ec2a 100644 --- a/core/attachments.py +++ b/core/attachments.py @@ -1,7 +1,9 @@ """Structured user-message attachments and model-context compatibility rendering.""" from __future__ import annotations +import base64 import mimetypes +from copy import deepcopy from collections.abc import Iterable from pathlib import Path from typing import Any @@ -10,6 +12,14 @@ from .artifacts import ArtifactPathError, resolve_artifact_path ATTACHMENT_REF_VERSION = 1 MAX_ATTACHMENTS_PER_MESSAGE = 10 +MAX_NATIVE_IMAGE_BYTES = 10 * 1024 * 1024 +_NATIVE_IMAGE_MIME = { + ".gif": "image/gif", + ".jpeg": "image/jpeg", + ".jpg": "image/jpeg", + ".png": "image/png", + ".webp": "image/webp", +} _IMAGE_EXTENSIONS = { ".avif", ".bmp", ".gif", ".heic", ".heif", ".jpeg", ".jpg", ".png", ".svg", ".tif", ".tiff", ".webp", @@ -74,3 +84,60 @@ def content_for_model(content: str, refs: Iterable[dict], working_dir_name: str) return text suffix = "\n".join(lines) return f"{text}\n\n{suffix}" if text else suffix + + +def materialize_native_images( + messages: Iterable[dict], + *, + enabled: bool, + working_dir: Path, + user_root: Path, +) -> tuple[list[dict], int]: + """把内存附件引用按需物化为 OpenAI-compatible image_url blocks。 + + Base64 只存在于本次 provider-bound 副本，不写数据库。历史文件已删除、越界、 + 格式不受支持或超过上限时保留原有文字路径提示并跳过图片块，不能让旧附件阻断续聊。 + 所有 ``_`` 内部字段在返回前剥离。 + """ + output: list[dict] = [] + image_count = 0 + for message in messages: + new_msg = deepcopy(message) + refs = list(new_msg.pop("_attachment_refs", []) or []) + new_msg.pop("_model_profile", None) + if not enabled or new_msg.get("role") != "user" or not refs: + output.append(new_msg) + continue + + blocks: list[dict] = [] + text = new_msg.get("content") + if isinstance(text, str) and text: + blocks.append({"type": "text", "text": text}) + message_image_count = 0 + for ref in refs: + if not isinstance(ref, dict) or ref.get("kind") != "image": + continue + raw_path = str(ref.get("path") or "") + try: + path, _ = resolve_artifact_path( + raw_path, + working_dir=working_dir, + user_root=user_root, + require_file=True, + ) + mime = _NATIVE_IMAGE_MIME.get(path.suffix.lower()) + if mime is None or path.stat().st_size > MAX_NATIVE_IMAGE_BYTES: + continue + data = base64.b64encode(path.read_bytes()).decode("ascii") + except (ArtifactPathError, OSError): + continue + blocks.append({ + "type": "image_url", + "image_url": {"url": f"data:{mime};base64,{data}"}, + }) + image_count += 1 + message_image_count += 1 + if message_image_count: + new_msg["content"] = blocks + output.append(new_msg) + return output, image_count diff --git a/core/capabilities.py b/core/capabilities.py index 98aee6b..ea9aa0b 100644 --- a/core/capabilities.py +++ b/core/capabilities.py @@ -3,13 +3,26 @@ from __future__ import annotations from dataclasses import dataclass, field, fields from pathlib import Path -from typing import List +from typing import List, Optional import yaml from .llm_params import THINKING_TRANSPORTS -REASONING_REPLAY_POLICIES = {"none", "tool_calls", "all"} +REASONING_REPLAY_POLICIES = { + "none", + "tool_turn", + "conversation", + "provider_managed", +} + + +def model_profile_of(caps: object) -> str: + """返回能力对象的 canonical profile；兼容测试和旧调用方的鸭子类型。""" + profile = getattr(caps, "profile", "") + if profile: + return str(profile) + return f"{getattr(caps, 'family', '')}.{getattr(caps, 'variant', '')}".strip(".") @dataclass @@ -34,10 +47,16 @@ class ModelCapabilities: thinking_transport: str = "none" reasoning_effort_levels: List[str] = field(default_factory=list) default_reasoning_effort: str = "" + # 是否清除历史 thinking。None=不发送该 provider 可选字段；False=保留历史推理。 + thinking_clear: Optional[bool] = None # 历史 reasoning 发回模型的策略。持久化原始响应与 provider-bound 输入分离： - # none=全部剥离；tool_calls=仅工具调用 assistant 保留；all=完整原样回传。 + # none=剥离；tool_turn=仅当前工具链；conversation=同模型会话； + # provider_managed=同模型不透明签名/block。 reasoning_replay: str = "none" + # 原生输入模态。附件仍以文件引用落库，发模型前才物化成 provider content blocks。 + input_modalities: List[str] = field(default_factory=lambda: ["text"]) + # 代码 / 沙盒 code_quality: str = "good" enable_run_python: bool = False @@ -81,6 +100,16 @@ class ModelCapabilities: ) var = dict(variants[variant]) + # 已下架 profile 可保留为一版隐藏别名，确保存量 task / scheduled job 能续跑； + # 真正能力和记账身份统一归目标 variant，不维持旧模型双轨。 + seen_aliases = {variant} + while var.get("alias_of"): + target = str(var["alias_of"]).strip() + if not target or target in seen_aliases or target not in variants: + raise ValueError(f"档案 {path} 的 variant={variant} alias_of 无效") + seen_aliases.add(target) + variant = target + var = dict(variants[target]) valid_keys = {f.name for f in fields(cls)} kwargs = {k: v for k, v in var.items() if k in valid_keys} kwargs["family"] = data.get("family", family) @@ -109,3 +138,11 @@ class ModelCapabilities: f"{caps.default_reasoning_effort!r} 不在 reasoning_effort_levels 中" ) return caps + + @property + def profile(self) -> str: + return f"{self.family}.{self.variant}" + + @property + def native_image_input(self) -> bool: + return "image" in self.input_modalities diff --git a/core/context.py b/core/context.py index 5b966fd..a6fac1d 100644 --- a/core/context.py +++ b/core/context.py @@ -14,6 +14,8 @@ from typing import Any, List import json import re +from .capabilities import REASONING_REPLAY_POLICIES + # chars↔tokens 粗折算(CJK+代码+json 混合保守按 ~2.5 char/token)与压缩触发比例。 # loop(50% 压缩门槛)与 context_fold(85% 折叠门槛)共用,单一事实源;压缩/折叠是 # 成本/安全优化、非正确性关键,估算粗糙无妨。 @@ -40,29 +42,41 @@ _REASONING_FIELDS = { def filter_reasoning_for_replay( messages: List[dict[str, Any]], policy: str = "none", + target_model_profile: str = "", ) -> tuple[List[dict[str, Any]], dict[str, int]]: """构造 provider-bound 消息副本，按能力档案裁剪历史 reasoning。 原始 Session/DB payload 不动，继续供 UI、导出和排障使用。工具调用需要 reasoning - 连续性的 provider 可用 ``tool_calls``；未声明能力的 provider 默认 ``none``，避免 - 跨模型发送私有状态。保留时把 LiteLLM 可能放在 provider_specific_fields 里的 - reasoning_content 提升到 DeepSeek/OpenAI-compatible 所需的顶层字段。 + ``tool_turn`` 只保留当前用户轮次内工具链的状态；``conversation`` 保留同模型 + 会话状态；``provider_managed`` 保留同模型的不透明签名/block。未声明能力默认 + ``none``。只要历史消息的生产模型与目标模型不同，就无条件剥离私有状态。 """ - if policy not in {"none", "tool_calls", "all"}: + if policy not in REASONING_REPLAY_POLICIES: raise ValueError(f"unsupported reasoning replay policy: {policy!r}") + last_user = max( + (i for i, msg in enumerate(messages) if msg.get("role") == "user"), + default=-1, + ) + filtered: List[dict[str, Any]] = [] stripped_messages = 0 stripped_chars = 0 - for message in messages: + for idx, message in enumerate(messages): new_msg = deepcopy(message) - if policy == "all": - filtered.append(new_msg) - continue + source_profile = str(new_msg.pop("_model_profile", "") or "") + compatible = not target_model_profile or source_profile == target_model_profile preserve = ( - policy == "tool_calls" + compatible and new_msg.get("role") == "assistant" - and bool(new_msg.get("tool_calls")) + and ( + policy in {"conversation", "provider_managed"} + or ( + policy == "tool_turn" + and idx > last_user + and bool(new_msg.get("tool_calls")) + ) + ) ) if preserve: provider_fields = new_msg.get("provider_specific_fields") @@ -242,6 +256,7 @@ def prepare_messages_for_llm( old_tool_chars: int = 2_000, compact_threshold_chars: int = 0, reasoning_replay: str = "none", + target_model_profile: str = "", ) -> List[dict[str, Any]]: """返回发给 LLM 的 messages 副本。 @@ -257,6 +272,7 @@ def prepare_messages_for_llm( old_tool_chars=old_tool_chars, compact_threshold_chars=compact_threshold_chars, reasoning_replay=reasoning_replay, + target_model_profile=target_model_profile, ) return prepared @@ -268,6 +284,7 @@ def prepare_messages_with_stats( old_tool_chars: int = 2_000, compact_threshold_chars: int = 0, reasoning_replay: str = "none", + target_model_profile: str = "", ) -> tuple[List[dict[str, Any]], dict[str, int]]: """返回发给 LLM 的 messages 副本和压缩统计。 @@ -279,7 +296,9 @@ def prepare_messages_with_stats( """ if keep_recent < 0: keep_recent = 0 - messages, reasoning_stats = filter_reasoning_for_replay(messages, reasoning_replay) + messages, reasoning_stats = filter_reasoning_for_replay( + messages, reasoning_replay, target_model_profile + ) # 先补齐被中断 run 留下的悬空 tool_calls(否则原样发给模型会被拒,见函数注释)。 messages, repaired_tool_calls = _repair_dangling_tool_calls(messages) original_chars = sum(_message_chars(m) for m in messages) diff --git a/core/context_fold.py b/core/context_fold.py index 56f0efd..2479e30 100644 --- a/core/context_fold.py +++ b/core/context_fold.py @@ -25,6 +25,7 @@ from uuid import UUID from sqlalchemy import update +from .capabilities import model_profile_of from .context import ( CHARS_PER_TOKEN, COMPACT_CONTEXT_RATIO, @@ -144,7 +145,10 @@ def maybe_fold( """ msgs = session.messages reasoning_replay = getattr(caps, "reasoning_replay", "none") - provider_msgs, _ = filter_reasoning_for_replay(msgs, reasoning_replay) + target_profile = model_profile_of(caps) + provider_msgs, _ = filter_reasoning_for_replay( + msgs, reasoning_replay, target_profile + ) before_chars = sum(_message_chars(m) for m in provider_msgs) est_tokens = estimate_window_tokens(provider_msgs, session.last_measured_usage()) if est_tokens < caps.reliable_context * FOLD_TRIGGER_RATIO: @@ -164,6 +168,7 @@ def maybe_fold( msgs, compact_threshold_chars=compact_threshold, reasoning_replay=reasoning_replay, + target_model_profile=target_profile, ) p_cut = _kth_user_index(prepared, _user_ordinal(msgs, cutoff)) if p_cut is None: # 理论不可达,防御 @@ -200,7 +205,7 @@ def maybe_fold( persist_fold(session.task_id, new_base, summary) session.apply_fold(cutoff, summary) after_provider_msgs, _ = filter_reasoning_for_replay( - session.messages, reasoning_replay + session.messages, reasoning_replay, target_profile ) after_chars = sum(_message_chars(m) for m in after_provider_msgs) stats = { diff --git a/core/llm.py b/core/llm.py index 9703ec8..83676be 100644 --- a/core/llm.py +++ b/core/llm.py @@ -54,9 +54,15 @@ class LLM: parallel_tool_calls: Optional[bool], reasoning_effort: Optional[str], ) -> dict: + # Session 私有元数据永不进入 provider 请求。原生图片由主 loop 在此之前 + # 物化；折叠/标题等旁路则安全退化为纯文本路径提示。 + provider_messages = [ + {k: v for k, v in message.items() if not str(k).startswith("_")} + for message in messages + ] kwargs: dict = { "model": self.caps.model_id, - "messages": messages, + "messages": provider_messages, "temperature": self.caps.optimal_temperature, "api_key": self.api_key, "timeout": _REQUEST_TIMEOUT_S, @@ -72,6 +78,7 @@ class LLM: enabled=self.caps.thinking_enabled, transport=self.caps.thinking_transport, reasoning_effort=reasoning_effort, + clear_thinking=self.caps.thinking_clear, ) ) if self.caps.prompt_caching: diff --git a/core/llm_params.py b/core/llm_params.py index 6ec1b87..e709ede 100644 --- a/core/llm_params.py +++ b/core/llm_params.py @@ -7,7 +7,11 @@ THINKING_TRANSPORTS = {"none", "extra_body"} def build_thinking_kwargs( - *, enabled: bool, transport: str, reasoning_effort: str | None + *, + enabled: bool, + transport: str, + reasoning_effort: str | None, + clear_thinking: bool | None = None, ) -> dict[str, Any]: """把统一的 thinking 配置转换为 LiteLLM 调用参数。 @@ -22,9 +26,12 @@ def build_thinking_kwargs( f"不支持的 thinking_transport={transport!r};可选: {sorted(THINKING_TRANSPORTS)}" ) - body: dict[str, Any] = { - "thinking": {"type": "enabled" if enabled else "disabled"} + thinking: dict[str, Any] = { + "type": "enabled" if enabled else "disabled" } + if enabled and clear_thinking is not None: + thinking["clear_thinking"] = clear_thinking + body: dict[str, Any] = {"thinking": thinking} if enabled and reasoning_effort: body["reasoning_effort"] = reasoning_effort return {"extra_body": body} diff --git a/core/loop.py b/core/loop.py index 9466077..9c509d8 100644 --- a/core/loop.py +++ b/core/loop.py @@ -22,7 +22,8 @@ import litellm from . import pptx_guard from .artifacts import MAX_ARTIFACTS_PER_MESSAGE -from .capabilities import ModelCapabilities +from .attachments import materialize_native_images +from .capabilities import ModelCapabilities, model_profile_of from .context import ( CHARS_PER_TOKEN, COMPACT_CONTEXT_RATIO, @@ -312,20 +313,21 @@ class AgentLoop: assert response is not None msg = response.choices[0].message tool_calls = getattr(msg, "tool_calls", None) or [] + model_profile = model_profile_of(self.caps) asst_msg_id = self.session.append( msg, artifact_refs=(list(self._pending_artifact_refs) if not tool_calls else None), + source_model_profile=model_profile, ) usage_details = extract_usage_details(getattr(response, "usage", None)) pt, ct = usage_details["tokens_in"], usage_details["tokens_out"] # 用本轮实报 prompt_tokens 刷新 chars/token 校准比值(下一轮门槛/占用环即用)。 - if pt > 0 and self._last_sent_chars > 0: + if pt > 0 and self._last_sent_chars > 0 and not self._last_had_native_images: self._ctx_chars_per_token = clamp_ratio(self._last_sent_chars / pt) # 记账(0006):一行 usage_event + 回填 messages.tokens_in/out + model_profile。 # 任何失败都吞掉(litellm cost map miss / DB 异常),不阻塞主 loop; # message 仍在 session/DB 里,后续重启不影响。 - model_profile = f"{self.caps.family}.{self.caps.variant}" try: record_chat_usage( task_id=self.session.task_id, @@ -456,6 +458,7 @@ class AgentLoop: # 无实测时的回退(详 context.py 常数注释)。 _ctx_chars_per_token: Optional[float] = None _last_sent_chars: int = 0 + _last_had_native_images: bool = False def _context_ratio(self) -> float: """当前窗口的 chars/token 换算比值(校准态)。 @@ -470,6 +473,7 @@ class AgentLoop: provider_messages, _ = filter_reasoning_for_replay( self.session.messages, getattr(self.caps, "reasoning_replay", "none"), + model_profile_of(self.caps), ) self._ctx_chars_per_token = calibrated_chars_per_token( provider_messages, self.session.last_measured_usage() @@ -514,7 +518,18 @@ class AgentLoop: self.session.messages, compact_threshold_chars=compact_threshold, reasoning_replay=getattr(self.caps, "reasoning_replay", "none"), + target_model_profile=model_profile_of(self.caps), ) + native_images = 0 + if self.user_root is not None: + llm_messages, native_images = materialize_native_images( + llm_messages, + enabled=self.caps.native_image_input, + working_dir=self.working_dir, + user_root=self.user_root, + ) + self._last_had_native_images = native_images > 0 + context_stats["native_images"] = native_images self._last_sent_chars = context_stats.get("sent_chars", 0) llm_start_event = { "type": "llm_start", diff --git a/core/model_access.py b/core/model_access.py index dd8a7dc..1a43ceb 100644 --- a/core/model_access.py +++ b/core/model_access.py @@ -6,7 +6,7 @@ - 某档成员里出现 `"*"` → 该档全开(含未来新增模型),给内部档用。 模型 id 约定(与 list 端点 / resolve 校验一致): -- 文本模型 = `family.variant`(config/models/.yaml),如 `doubao.pro`、`glm.pro52` +- 文本模型 = `family.variant`(config/models/.yaml),如 `doubao.pro`、`glm.flash53` - 图 / 视频模型 = variant key(config/media/doubao.yaml),如 `seedream_5`、`seedance_2_fast` 两者命名不冲突(文本带点、媒体 variant 不带点),同一档集合里混放即可。 diff --git a/core/session.py b/core/session.py index bf27713..56916e3 100644 --- a/core/session.py +++ b/core/session.py @@ -65,6 +65,7 @@ class Session: msg: Any, *, artifact_refs: Optional[list[dict]] = None, + source_model_profile: str = "", ) -> Optional[UUID]: """追加消息;非 system 落 DB,system 仅内存。返回新落库行的 message_id。 @@ -91,7 +92,10 @@ class Session: f"[markdown:fence-warning] task={self.task_id} unclosed=1", flush=True, ) - self.messages.append(msg_dict) + memory_msg = dict(msg_dict) + if source_model_profile and memory_msg.get("role") == "assistant": + memory_msg["_model_profile"] = source_model_profile + self.messages.append(memory_msg) if msg_dict.get("role") == "system": return None @@ -236,6 +240,9 @@ class Session: attachment_refs, wd_name, ) + payload["_attachment_refs"] = list(attachment_refs) + if payload.get("role") == "assistant" and row.model_profile: + payload["_model_profile"] = row.model_profile if payload.get("role") == "assistant" and isinstance(payload.get("content"), str): # 历史行不回写生产库；只在重建 LLM 上下文时应用同一窄修复， # 与 Web 展示层保持一致，避免旧坏围栏继续污染后续轮次。 diff --git a/core/tool_registry.py b/core/tool_registry.py index 0e426cd..a1c223c 100644 --- a/core/tool_registry.py +++ b/core/tool_registry.py @@ -269,17 +269,24 @@ def build_tools(ctx: ToolContext) -> dict[str, Any]: cancel_check=ctx.cancel_check, daily_limit=videos_per_day, **base, )] - def _vision() -> list: + def _vision_kwargs() -> Optional[dict]: # 看图 + 文档理解共用 vision variant(读图不分档,固定第一个)。 key, v = _pick_variant(ctx.ark_cfg.raw.get("vision") or {}) if v is None: - return [] - kw = dict( + return None + return dict( ark_cfg=ctx.ark_cfg, vision_variant_cfg=v, variant_key=key, working_dir=ctx.working_dir_path, task_id=ctx.task_id, user_id=ctx.uid, **base, ) - return [LookAtImageTool(**kw), ReadDocumentTool(**kw)] + + def _look_at_image() -> list: + kw = _vision_kwargs() + return [LookAtImageTool(**kw)] if kw is not None else [] + + def _read_document() -> list: + kw = _vision_kwargs() + return [ReadDocumentTool(**kw)] if kw is not None else [] def _transcribe() -> list: # 录音文件转写(讯飞 LFASR,与 IAT 听写是两个服务两套 key)。 @@ -316,7 +323,13 @@ def build_tools(ctx: ToolContext) -> dict[str, Any]: ("office_to_pdf", lambda: ctx.office_to_pdf_available, _office_to_pdf), ("image", lambda: ctx.img_cfg is not None, _image), ("video", lambda: ctx.ark_cfg is not None, _video), - ("vision", lambda: ctx.ark_cfg is not None, _vision), + ( + "look_at_image", + lambda: ctx.ark_cfg is not None + and not getattr(ctx.caps, "native_image_input", False), + _look_at_image, + ), + ("read_document", lambda: ctx.ark_cfg is not None, _read_document), ("transcribe_audio", lfasr_configured, _transcribe), ("web_search", lambda: BochaConfig.load() is not None, _web_search), ] diff --git a/tests/test_attachments.py b/tests/test_attachments.py index 2655b40..3fed8fa 100644 --- a/tests/test_attachments.py +++ b/tests/test_attachments.py @@ -3,7 +3,11 @@ import unittest from pathlib import Path from core.artifacts import ArtifactPathError -from core.attachments import content_for_model, normalize_attachment_refs +from core.attachments import ( + content_for_model, + materialize_native_images, + normalize_attachment_refs, +) class AttachmentRefTests(unittest.TestCase): @@ -47,3 +51,27 @@ class AttachmentRefTests(unittest.TestCase): content_for_model("", refs, "实验分析"), "[用户上传的参考图] 实验分析/figures/a.png", ) + + def test_materializes_native_image_without_persisting_internal_fields(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + wd = root / "任务" + wd.mkdir() + (wd / "a.png").write_bytes(b"png") + messages = [{ + "role": "user", + "content": "请分析\n\n[用户上传的参考图] 任务/a.png", + "_attachment_refs": [{"path": "a.png", "kind": "image"}], + }] + prepared, count = materialize_native_images( + messages, enabled=True, working_dir=wd, user_root=root, + ) + self.assertEqual(count, 1) + self.assertNotIn("_attachment_refs", prepared[0]) + self.assertEqual(prepared[0]["content"][0]["type"], "text") + self.assertEqual(prepared[0]["content"][1]["type"], "image_url") + self.assertTrue( + prepared[0]["content"][1]["image_url"]["url"].startswith( + "data:image/png;base64," + ) + ) diff --git a/tests/test_context_compaction.py b/tests/test_context_compaction.py index ec9f3c8..57fb66c 100644 --- a/tests/test_context_compaction.py +++ b/tests/test_context_compaction.py @@ -58,7 +58,7 @@ class ContextCompactionTests(unittest.TestCase): prepared, stats = prepare_messages_with_stats( messages, - reasoning_replay="tool_calls", + reasoning_replay="tool_turn", compact_threshold_chars=10_000, ) @@ -78,7 +78,7 @@ class ContextCompactionTests(unittest.TestCase): }, {"role": "tool", "tool_call_id": "tc1", "content": "ok"}] prepared = prepare_messages_for_llm( - messages, reasoning_replay="tool_calls", + messages, reasoning_replay="tool_turn", ) self.assertEqual(prepared[0]["reasoning_content"], "nested reasoning") @@ -91,10 +91,54 @@ class ContextCompactionTests(unittest.TestCase): "provider_specific_fields": {"signature": "opaque"}, } - prepared = prepare_messages_for_llm([message], reasoning_replay="all") + prepared = prepare_messages_for_llm( + [message], reasoning_replay="conversation" + ) self.assertEqual(prepared[0], message) + def test_conversation_policy_strips_reasoning_from_other_model(self) -> None: + messages = [{ + "role": "assistant", + "content": "answer", + "reasoning_content": "deepseek reasoning", + "_model_profile": "deepseek_v4.flash", + }] + prepared = prepare_messages_for_llm( + messages, + reasoning_replay="conversation", + target_model_profile="glm.flash53", + ) + self.assertNotIn("reasoning_content", prepared[0]) + self.assertNotIn("_model_profile", prepared[0]) + + def test_tool_turn_keeps_only_reasoning_after_latest_user(self) -> None: + call = lambda cid, reasoning: { + "role": "assistant", "content": None, + "reasoning_content": reasoning, + "_model_profile": "deepseek_v4.flash", + "tool_calls": [{ + "id": cid, "type": "function", + "function": {"name": "shell", "arguments": "{}"}, + }], + } + messages = [ + {"role": "user", "content": "old"}, + call("old", "old reasoning"), + {"role": "tool", "tool_call_id": "old", "content": "old result"}, + {"role": "assistant", "content": "done"}, + {"role": "user", "content": "new"}, + call("new", "new reasoning"), + {"role": "tool", "tool_call_id": "new", "content": "new result"}, + ] + prepared = prepare_messages_for_llm( + messages, + reasoning_replay="tool_turn", + target_model_profile="deepseek_v4.flash", + ) + self.assertNotIn("reasoning_content", prepared[1]) + self.assertEqual(prepared[5]["reasoning_content"], "new reasoning") + def test_preserves_system_and_recent_messages(self) -> None: messages = [ {"role": "system", "content": "rules"}, diff --git a/tests/test_context_fold.py b/tests/test_context_fold.py index 284b5ee..4d27486 100644 --- a/tests/test_context_fold.py +++ b/tests/test_context_fold.py @@ -146,7 +146,7 @@ class _FakeLLM: _FAKE_CAPS = SimpleNamespace( reliable_context=100, # 极小上限 → 体量必然超阈值,不用造几十万字符 family="deepseek_v4", variant="flash", - reasoning_replay="tool_calls", + reasoning_replay="tool_turn", input_cny_per_mtoken=1.0, output_cny_per_mtoken=2.0, ) @@ -227,7 +227,7 @@ class MaybeFoldTests(unittest.TestCase): self.assertEqual(sess.messages, before) # 内存零污染 self.assertTrue(any(e.get("type") == "warn" for e in events)) - def test_fold_input_strips_final_reasoning_but_keeps_tool_reasoning(self) -> None: + def test_fold_input_strips_reasoning_from_completed_turns(self) -> None: sess = self._session() sess.messages[2]["reasoning_content"] = "final reasoning" sess.messages[4] = { @@ -254,7 +254,7 @@ class MaybeFoldTests(unittest.TestCase): final_msg = next(m for m in sent if m.get("content", "").startswith("a000")) tool_msg = next(m for m in sent if m.get("tool_calls")) self.assertNotIn("reasoning_content", final_msg) - self.assertEqual(tool_msg["reasoning_content"], "tool reasoning") + self.assertNotIn("reasoning_content", tool_msg) if __name__ == "__main__": diff --git a/tests/test_llm_kwargs.py b/tests/test_llm_kwargs.py index cbc69ed..80d136a 100644 --- a/tests/test_llm_kwargs.py +++ b/tests/test_llm_kwargs.py @@ -75,6 +75,22 @@ class LLMKwargsTests(unittest.TestCase): kwargs["extra_body"], {"thinking": {"type": "enabled"}} ) + def test_glm_preserved_thinking_is_nested_in_thinking_body(self) -> None: + caps = ModelCapabilities( + family="glm", variant="flash53", model_id="zai/glm-5.3-flash", + api_key_env="TEST_LLM_API_KEY", thinking_enabled=True, + thinking_transport="extra_body", thinking_clear=False, + default_reasoning_effort="high", + ) + with patch.dict(os.environ, {"TEST_LLM_API_KEY": "test-key"}): + kwargs = LLM(caps)._build_kwargs( + [{"role": "user", "content": "hello"}], None, None, "high" + ) + self.assertEqual(kwargs["extra_body"], { + "thinking": {"type": "enabled", "clear_thinking": False}, + "reasoning_effort": "high", + }) + def test_invalid_transport_is_rejected_before_request(self) -> None: llm = self._llm( family="test", thinking_enabled=True, thinking_transport="unknown" @@ -97,7 +113,7 @@ class LLMKwargsTests(unittest.TestCase): self.assertEqual(caps.output_cny_per_mtoken, 2.0) self.assertEqual(caps.cache_hit_cny_per_mtoken, 0.02) self.assertEqual(caps.thinking_transport, "extra_body") - self.assertEqual(caps.reasoning_replay, "tool_calls") + self.assertEqual(caps.reasoning_replay, "tool_turn") def test_other_controllable_profiles_declare_transport(self) -> None: models_dir = Path(__file__).resolve().parents[1] / "config" / "models" @@ -105,8 +121,12 @@ class LLMKwargsTests(unittest.TestCase): glm = ModelCapabilities.load("glm.pro52", models_dir) doubao = ModelCapabilities.load("doubao.turbo", models_dir) - self.assertFalse(glm.thinking_enabled) + self.assertEqual(glm.profile, "glm.flash53") + self.assertTrue(glm.thinking_enabled) self.assertEqual(glm.thinking_transport, "extra_body") + self.assertFalse(glm.thinking_clear) + self.assertEqual(glm.reasoning_replay, "conversation") + self.assertTrue(glm.native_image_input) self.assertTrue(doubao.thinking_enabled) self.assertEqual(doubao.thinking_transport, "extra_body") self.assertEqual(doubao.default_reasoning_effort, "") diff --git a/tests/test_look_at_image.py b/tests/test_look_at_image.py index e7b5bb0..808b109 100644 --- a/tests/test_look_at_image.py +++ b/tests/test_look_at_image.py @@ -1,9 +1,12 @@ """look_at_image 的问题收敛与兼容兜底测试（不碰网络和数据库）。""" from __future__ import annotations +import tempfile import unittest import uuid +from contextlib import ExitStack from pathlib import Path +from types import SimpleNamespace from unittest import mock from core.ark_client import ArkConfig @@ -27,6 +30,69 @@ class LookAtImageQuestionTests(unittest.TestCase): user_root=Path("."), ) + def test_native_vision_model_omits_look_tool_but_keeps_document_ocr(self): + from core.tool_registry import ToolContext, build_tools + + ark = ArkConfig( + api_key="test", + base_url="https://example.invalid", + raw={"vision": {"test": {"model_id": "vision-test"}}}, + ) + + def build(native_image_input: bool) -> dict: + with tempfile.TemporaryDirectory() as tmp, ExitStack() as stack: + root = Path(tmp) + stack.enter_context(mock.patch( + "core.tool_registry._external_system_status_available", + return_value=False, + )) + stack.enter_context(mock.patch( + "core.tool_registry._external_systems_available", + return_value=False, + )) + stack.enter_context(mock.patch( + "core.tool_registry.smtp_configured", return_value=False, + )) + stack.enter_context(mock.patch( + "core.tool_registry.wechat_push_available", return_value=False, + )) + stack.enter_context(mock.patch( + "core.tool_registry.lfasr_configured", return_value=False, + )) + stack.enter_context(mock.patch( + "core.tool_registry.BochaConfig.load", return_value=None, + )) + return build_tools(ToolContext( + tool_base=root, + ur_path=root, + working_dir_path=root, + task_id=uuid.uuid4(), + uid=uuid.uuid4(), + cfg={}, + caps=SimpleNamespace( + enable_run_python=False, + native_image_input=native_image_input, + ), + skills=SimpleNamespace(skills={}), + cancel_check=None, + scheduled_run=True, + deferred_actions=SimpleNamespace(), + ark_cfg=ark, + img_provider="", + img_key="", + img_cfg=None, + img_provider_cfg=None, + video_variant="", + office_to_pdf_available=False, + )) + + text_only = build(False) + native = build(True) + self.assertIn("look_at_image", text_only) + self.assertIn("read_document", text_only) + self.assertNotIn("look_at_image", native) + self.assertIn("read_document", native) + def _execute_and_question(self, question=None) -> str: captured = {} diff --git a/web/admin.py b/web/admin.py index d8f79fd..32186e1 100644 --- a/web/admin.py +++ b/web/admin.py @@ -155,7 +155,9 @@ def _model_catalog() -> list[dict]: except Exception: continue family = data.get("family") or path.stem - for variant in (data.get("variants") or {}).keys(): + for variant, variant_cfg in (data.get("variants") or {}).items(): + if isinstance(variant_cfg, dict) and variant_cfg.get("hidden"): + continue profile = f"{family}.{variant}" try: caps = ModelCapabilities.load(profile, models_dir) diff --git a/web/model_gate.py b/web/model_gate.py index 39ed5af..e95baa6 100644 --- a/web/model_gate.py +++ b/web/model_gate.py @@ -70,8 +70,9 @@ def resolve_model_profile(profile: str, user_id: Optional[UUID] = None) -> tuple caps = ModelCapabilities.load(name, ROOT / cfg["models_dir"]) except (FileNotFoundError, ValueError) as e: raise HTTPException(400, f"invalid model_profile {name!r}: {e}") - assert_model_allowed(name, user_id, "text") - return name, caps.model_id + canonical = caps.profile + assert_model_allowed(canonical, user_id, "text") + return canonical, caps.model_id def skill_pinned_profiles() -> set: diff --git a/web/routers/messages.py b/web/routers/messages.py index c764a2a..3c19034 100644 --- a/web/routers/messages.py +++ b/web/routers/messages.py @@ -261,6 +261,16 @@ def register_message_routes(app, *, require_user) -> None: cur_profile = task.model_profile or "" if cur_profile: from core.model_access import is_allowed + # 旧模型 profile 先解析成隐藏兼容别名的正式目标，避免被误判成档外 + # 后降到 DeepSeek；本次持久化 canonical profile，后续直接走新模型。 + try: + canonical_profile, canonical_model = resolve_model_profile(cur_profile) + except HTTPException: + canonical_profile, canonical_model = cur_profile, task.model + if canonical_profile != cur_profile: + values["model_profile"] = canonical_profile + values["model"] = canonical_model + cur_profile = canonical_profile urow = s.execute( select(User.plan, User.role).where(User.user_id == user_id) ).first() @@ -511,6 +521,10 @@ def register_message_routes(app, *, require_user) -> None: cfg = load_config() chosen_profile = task_model_profile or cfg["default_model"] + try: + chosen_profile, _ = resolve_model_profile(chosen_profile) + except HTTPException: + pass # 档位门控:task 存量模型已不在用户档位内 → 润色也落回 flash(与 send 路径一致, # 不持久改 task,仅本次润色调用降级)。 if chosen_profile and not model_allowed_for_user(chosen_profile, user_id): diff --git a/web/routers/models.py b/web/routers/models.py index ddf5559..0898df7 100644 --- a/web/routers/models.py +++ b/web/routers/models.py @@ -41,7 +41,9 @@ def register_model_routes(app, *, require_user) -> None: except Exception: continue family = data.get("family") or path.stem - for variant in (data.get("variants") or {}).keys(): + for variant, variant_cfg in (data.get("variants") or {}).items(): + if isinstance(variant_cfg, dict) and variant_cfg.get("hidden"): + continue profile = f"{family}.{variant}" if allowed is not None and profile not in allowed: continue diff --git a/web/routers/tasks.py b/web/routers/tasks.py index 9c4bc06..43fdce2 100644 --- a/web/routers/tasks.py +++ b/web/routers/tasks.py @@ -322,6 +322,7 @@ def register_task_routes(app, *, require_user) -> None: window_messages, compact_threshold_chars=compact_threshold, reasoning_replay=caps.reasoning_replay, + target_model_profile=caps.profile, ) window_chars = context_stats["original_chars"] except Exception: diff --git a/web/static/js/chat.js b/web/static/js/chat.js index 1876707..990b328 100644 --- a/web/static/js/chat.js +++ b/web/static/js/chat.js @@ -950,7 +950,7 @@ function renderModelDropdown(t) { let opts = state.models.map(m => `` ).join(""); - // 当前模型不在该用户档位清单里(skill 定向模型,如 ppt → glm.pro52)→ 补一项, + // 当前模型不在该用户档位清单里(skill 定向模型)→ 补一项, // 下拉如实显示;用户仍可切回档内模型。 if (cur && !state.models.some(m => m.profile === cur)) { opts += ``;