refactor(platform): isolate managed source providers

This commit is contained in:
caoqianming 2026-08-26 11:29:44 +08:00
parent 604f4da373
commit 631a635eab
27 changed files with 603 additions and 189 deletions

View File

@ -8,7 +8,7 @@
## Unreleased
- 文献检索现在可以在服务端直接限定书籍、章节、论文等出版物类型,书籍类查询更准确;平台文献源的访问凭据也不再进入任务沙箱。
- 平台托管的文献、内部材料库和 Materials Project 来源现在使用来源明确的工具,并可在单个来源不可用时继续使用其他来源;访问凭据不进入任务沙箱。
## 0.68.0 — 2026-08-24

View File

@ -39,7 +39,8 @@ zcbot/
│ ├── wechat/ # 渠道:ilink / wecom / service / inbound(§8.7)
│ ├── sandbox/ + executor*.py # Executor ABC + Docker per-user 容器池(§7.5)
│ └── agent_builder.py # 装配 lib:build_agent / system prompt
├── tools/ # fs / shell / run_python / skill / 媒体(共享原语 media_common)/ 检索 / host-side 域工具
├── platform_sources/ # 平台托管共享只读来源、显式 Provider registry 与 typed tools(§3.4)
├── tools/ # fs / shell / run_python / skill / 媒体(共享原语 media_common)
├── skills/<name>/ # SKILL.md + references / scripts / assets
├── rendering/ # 平台渲染层 md→docx/pdf(§8.6;块收集器/inline 切分单一事实源在 common)
├── prompts/system/general_v1.md
@ -87,10 +88,16 @@ yaml 是手填的,probe 用真实调用对账(basic_chat/parallel_tools/thinking
### 3.4 工具系统(Hybrid 范式)
**JSON tool call** 管离散操作;**run_python**(tmp .py + subprocess + 敏感 env 过滤)管批处理/生成文档。`edit` **唯一匹配**(old_str 重复即报错);工具按**原子操作**切分,不做 `make_pptx()` 式高级封装。持 key 的能力一律 host-side tool、仅对应 env 存在才注册(§7.5 #7)。
**平台托管来源 bounded context**(2026-08-26):管理员随部署配置、平台统一凭据和共享只读数据源统一归顶层 `platform_sources/`;选择顶层包而非 `core/platform_sources + tools/platform_sources`,是为了让 client/runtime、来源专用 typed Tool adapter、显式 Provider registry 和生命周期保持在同一领域边界,同时让 `core/tool_registry.py` 只依赖唯一 `build_platform_source_tools` 入口。依赖方向固定为 `core/tool_registry → platform_sources → tools.base``platform_sources` 与 `core/external_systems` 互不依赖;后者仍只负责用户身份连接、用户凭据和动态 OpenAPI/MCP。
Provider 的最小契约只有 `source_id`、`capabilities`、`available()`、`build_tools(context)`,统一注册和生命周期,不假设每个来源都有相同 search/get/fetch 业务接口。可信 Provider 使用代码内显式列表,逐来源隔离配置和装配失败,诊断只记录来源标识与异常类型;模型面继续暴露 `paper_server_*`、`materials_library_*`、`materials_project_*` 强类型工具,不提供万能弱类型 RPC。平台凭据只在宿主 control plane 读取,不进 prompt、Tool schema/result、日志、`run_python` 或 Docker sandboxpaper_server 保留同源下载、100 MiB 上限、认证错误脱敏和 `.part` 原子落盘。只抽取已稳定重复的安全 HTTP/下载原语Materials Project 的 SDK、化学式、`material_id` 与 CIF/entries 语义留在专用 adapter。
当前定义和凭据仍以部署环境变量为事实源,修改后重启生效,不建数据库。若未来出现用户级凭据或 per-user grant`external_systems`;若需要平台来源在线动态 definition、revision/reverify 或 OAuth则另行增加 `platform_sources` 控制面并重新评估持久化,不能把动态连接状态塞进当前静态 Provider registry。
### 3.5 Skill 系统(Anthropic 渐进披露)
三层加载:Discovery(name+description,几百 token)→ Activation(`load_skill` 完整 SKILL.md)→ Execution(references 按需拉)。写 WHY+WHAT 不写 Step 1/2/3;description 决定触发。
**出版物检索统一入口**(2026-08-25):`literature` 按用户任务而非数据源划分 skill统一论文、书籍、章节、会议论文、学位论文、报告、标准、专利、预印本及未来出版物的发现、获取、跨源去重与证据核验`paper_server`、内部材料知识库及未来来源只是可替换后端。出版物类型与来源正交,同一出版物可有多条来源记录;跨源机械层只自动合并 DOI、ISBN+版次、标准号、公开号等精确键或题名+主要责任者+年份完全一致项,疑似重复、不同版次、章节、标准修订版和专利族成员保守保留。题录、摘要、片段、全文四级证据分开标注,下载成功不等于全文已经核验。取舍是不为每个数据库新增 skill也不把检索编排固化成统一高级工具skill 负责路由与判断,来源能力保持各自工具边界,确定性标准化与精确去重下沉到可重建脚本。`paper_server` 自 2026-08-26 起使用按配置注册的 host-side `paper_search` / `paper_get` / `paper_fetch`API Key 不再进入 sandbox来源原始 `type` 可服务端过滤,统一类型映射仍归出版物模型与合并脚本。
**出版物检索统一入口**(2026-08-25):`literature` 按用户任务而非数据源划分 skill统一论文、书籍、章节、会议论文、学位论文、报告、标准、专利、预印本及未来出版物的发现、获取、跨源去重与证据核验`paper_server`、内部材料知识库及未来来源只是可替换后端。出版物类型与来源正交,同一出版物可有多条来源记录;跨源机械层只自动合并 DOI、ISBN+版次、标准号、公开号等精确键或题名+主要责任者+年份完全一致项,疑似重复、不同版次、章节、标准修订版和专利族成员保守保留。题录、摘要、片段、全文四级证据分开标注,下载成功不等于全文已经核验。取舍是不为每个数据库新增 skill也不把检索编排固化成统一高级工具skill 负责路由与判断,来源能力保持各自工具边界,确定性标准化与精确去重下沉到可重建脚本。`paper_server` 自 2026-08-26 起使用按配置注册的 host-side `paper_server_search` / `paper_server_get` / `paper_server_fetch`API Key 不再进入 sandbox来源原始 `type` 可服务端过滤,统一类型映射仍归出版物模型与合并脚本。
**用户私有 skill**(2026-06-11):registry 收有序来源列表——内置 `ROOT/skills`(只读)+ 用户 `user_root/.skills`(可写)。取舍:① **user wins 同名覆盖**(核心用例是"copy 内置再改",覆盖只作用于本人会话,blast radius 锁死),覆盖显式标注不静默;② **创作走 host-side `save_skill`/`fork_skill`**——fs 工具的 base_dir 跨 backend 够不到 `user_root/.skills`,host 工具一个落点两模式通吃;③ 用户 skill 加载失败收进 `load_errors` 注入 prompt 提示修,不崩整次扫描。
@ -109,7 +116,7 @@ Session = 消息列表,ORM 直写 PG `messages`(append-only,jsonb 存 LiteLLM
### 3.8 个人知识库(`core/kb.py` + `core/kb_ingest.py`,✅ 2026-07-22)
用户自建资料(规范/报告/标准/内部文档)的长期查阅层,与 §3.7 记忆同范式:**纯文件 + prompt 注入契约,无向量无 DB**(判据同"真实文件为准":个人库几十~百余文件,agentic search 足够;索引若引入只能是可重建派生缓存)。**两层格局**:zcbot 内建 `.kb/` 私有小库(本节)+ 院检索服务共享大库(document_search,zcbot 只当客户端)——分工标准 = 文件数 × 查询频次,路由靠各自工具/契约描述自然分流,不在 kb 契约里点名 document_search(2026-07-22 收窄:契约只管自己怎么用,少一层耦合)。
用户自建资料(规范/报告/标准/内部文档)的长期查阅层,与 §3.7 记忆同范式:**纯文件 + prompt 注入契约,无向量无 DB**(判据同"真实文件为准":个人库几十~百余文件,agentic search 足够;索引若引入只能是可重建派生缓存)。**两层格局**:zcbot 内建 `.kb/` 私有小库(本节)+ 院检索服务共享大库(`materials_library_search`,zcbot 只当客户端)——分工标准 = 文件数 × 查询频次,路由靠各自工具/契约描述自然分流,不在 kb 契约里点名平台材料库(2026-07-22 收窄:契约只管自己怎么用,少一层耦合)。
- **做成机制而非 skill**(判据:有独立于会话的持久状态需用户管理 → 机制):落盘 `user_root/.kb/<库名>/`(INDEX.md + docs/ 转换后 md + sources/ 原件)。**"已入库"判据 = INDEX.md 有条目**,sources 有而 INDEX 无 = 待入库 → 入库幂等、崩溃可恢复、零 migration。dotfile 命名同 `.memory` 双向防呆,GET /v1/files 天然隐藏。
- **入库管线**(`core/kb_ingest.py`,上传即触发 + 手动兜底):markitdown Python API 转 md → 扫描件 PDF(文本近零)走方舟文档理解 OCR 兜底(§8.13 同通道)→ deepseek flash 单次 chat 写 标题/摘要/关键词(失败降级文件名+正文开头,不阻塞)→ 追加 INDEX 行。编排照定时执行器:create_task + to_thread;**写并发收口为共享 FS advisory lock**(`.kb/.locks/<hash>.lock`):Web 上传/删除、后台入库与 agent `write/edit` 对同一库共用一把跨进程锁,蓝绿实例间只允许一个写者,锁占用返 409/工具可重试;进程退出由 OS 自动释锁,不靠清理锁文件。文档正文、原件与 INDEX 全走同目录临时文件 + fsync + `os.replace` 原子发布,读者只会看到完整旧版或完整新版。进度详情仍以内存保存细节,但会探测跨进程锁补出 `running`;崩了靠 FS 判据续跑。
@ -378,13 +385,13 @@ scheduled_jobs(§8.5) channel_bindings(§8.7,判别列+JSONB)
### 8.11 最小子循环 delegate:上下文隔离而非多 agent 编排(design,2026-07-08,按诊断数据触发)
**根因**:检索/扫文类工作(文献 brief、document_search、批量读文件)的形态是"中间数据量大、最终只要结论"——在主循环里跑,中间数据必然流经主上下文,污染 + 膨胀是**架构性的**。已踩实例:38 篇 abstract 反复 dump 烧 2.5M token、`document_search` 同参调 122 次不收敛。现有缓解(`_RepeatGuard` 熔断、§8.2 context 压缩、brief skill 的 context 纪律)全是**行为约束**——劝模型别乱来,不改变"中间数据必须过主上下文"这个结构;同 8.9 的教训,提示层纪律挡不住结构性问题。
**根因**:检索/扫文类工作(文献 brief、`materials_library_search`、批量读文件)的形态是"中间数据量大、最终只要结论"——在主循环里跑,中间数据必然流经主上下文,污染 + 膨胀是**架构性的**。已踩实例:38 篇 abstract 反复 dump 烧 2.5M token、材料库检索同参调 122 次不收敛。现有缓解(`_RepeatGuard` 熔断、§8.2 context 压缩、brief skill 的 context 纪律)全是**行为约束**——劝模型别乱来,不改变"中间数据必须过主上下文"这个结构;同 8.9 的教训,提示层纪律挡不住结构性问题。
**决策**:§6"不做 subagent"针对的是**编排型多 agent**(并行、状态共享、agent 间通信、任务分解),该结论不变。本条预留的是**一个工具**:`delegate(instruction) -> str 摘要`——复用现成 `AgentLoop` 起一个全新空上下文的子循环,只注只读工具(检索/读文件类白名单),硬轮数上限(~20),跑完只把文字摘要返回主循环。主循环视角就是一次普通 tool call,零状态共享、零并行、零 agent 间协议;实现量约一个文件。
**触发条件(无信号不实施)**:RepeatGuard + brief 纪律上线后,`scripts/diag_*.py` 数据仍显示检索型 task 是烧 token 大户 / 检索中间数据占上下文大头。数据收敛则本条永久搁置。
**⚠️ 实现后撤回(0.58.30 落地 → 0.58.31 revert,记录教训)**:2026-07-15 曾完整实现(FilteredExecutor + 内存态子 Session + loop 拦截 + 降 flash),又整体撤回。撤回原因不是机制错(设计是对的、对标 Claude Code subagent 也成立),而是**触发信号没坐实**:自评时回看 `diag_search_args.py` 数据,motivating 案子 `document_search` 122 次呈"一批批不同材料体系并行搜"形态,**更像批量扇出而非结果驱动的探索**——若属实,它的正解和 mp_search 一样是**批量工具 `document_search_batch`**(便宜、可预测、可诊断),delegate 对它是过度设计。加上子循环 transcript 不落盘(诊断驱动的功能反而不可诊断)、20 轮上限对 122 次负载可能偏低、强制 flash 对难检索可能降质——一堆未验证的坑。**重建的前置条件收紧为**:先用 diag 确认某检索型 task 的 query 序列是**真探索**(query 依赖前序结果、无法一次性列全),而非可批量枚举;是批量就走批量工具,只有真探索才值得 delegate。**教训**:§5"无信号不实施"要落到"信号已被数据证伪 batchable 的可能性"这一步,不能凭"看起来像探索"就上 agentic 子循环——同 mp_search 的判断(批量扇出≠检索发散)。批量工具那条(0.58.28)验证过、保留。
**⚠️ 实现后撤回(0.58.30 落地 → 0.58.31 revert,记录教训)**:2026-07-15 曾完整实现(FilteredExecutor + 内存态子 Session + loop 拦截 + 降 flash),又整体撤回。撤回原因不是机制错(设计是对的、对标 Claude Code subagent 也成立),而是**触发信号没坐实**:自评时回看 `diag_search_args.py` 数据,motivating 案子材料库检索 122 次呈"一批批不同材料体系并行搜"形态,**更像批量扇出而非结果驱动的探索**——若属实,它的正解`materials_library_search(queries=[...])` 这类批量工具(便宜、可预测、可诊断),delegate 对它是过度设计。加上子循环 transcript 不落盘(诊断驱动的功能反而不可诊断)、20 轮上限对 122 次负载可能偏低、强制 flash 对难检索可能降质——一堆未验证的坑。**重建的前置条件收紧为**:先用 diag 确认某检索型 task 的 query 序列是**真探索**(query 依赖前序结果、无法一次性列全),而非可批量枚举;是批量就走批量工具,只有真探索才值得 delegate。**教训**:§5"无信号不实施"要落到"信号已被数据证伪 batchable 的可能性"这一步,不能凭"看起来像探索"就上 agentic 子循环——同 Materials Project 批量查询的判断(批量扇出≠检索发散)。批量工具那条(0.58.28)验证过、保留。
**不选**:
- 完整多 agent 编排:状态管理爆炸(§6),且 zcbot 无真实并行需求——用户没有"同时审 3 篇"诉求,职责隔离已由 skill 体系覆盖。

View File

@ -2,7 +2,7 @@
> 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 1-2 句:做了啥 + 关键判断;细节查 `git log` / `git diff` / `DESIGN §7.9`
最后更新:2026-08-26(paper_server 类型过滤与 host-side 凭据隔离,未发版)
最后更新:2026-08-26(platform_sources 独立平台托管来源,未发版)
---
@ -20,9 +20,11 @@
---
## 已完成关键能力
- **08-26 / Unreleased / paper_server 宿主侧受控访问**:新增按配置注册的 `paper_search` / `paper_get` / `paper_fetch` typed tools`publication_type` 直接映射后端 `type` 过滤且保留开放原始类型API Key 从 `run_python` 和 Docker sandbox 透传中移除,下载增加同源 URL、100 MiB 上限和原子落盘约束。literature / brief 指引、smoke、RUN、Skill 清单及回归测试同步迁移,线上只读 `type=book` 验证通过,不提升版本号
- **08-26 / Unreleased / platform_sources bounded context**:将 paper_server、内部材料库和 Materials Project 收敛为独立顶层包,以显式可信 Provider 列表统一可用性与生命周期,单来源配置/装配失败不阻断其他来源;`core/tool_registry.py` 只保留一个构建入口,与用户连接 `external_systems` 完全独立。模型工具统一改为来源明确的 `paper_server_*`、`materials_library_*`、`materials_project_*`保留材料库批量检索、MP 专用 SDK/CIF/entries 语义及 paper_server 安全下载边界;部署环境变量仍是事实源,无数据库迁移、无版本提升
- **08-25 / Unreleased / literature 统一出版物检索入口**:删除按数据源拆分的 `research` / `documents` skill新增 `literature` 统一论文、书籍、章节及未来学术与技术出版物的发现、获取、跨源去重和证据核验paper_server helper 归入新 skill内部材料库客户端移到 host-side `tools/` 保持密钥隔离。新增开放出版物模型、来源与证据 references以及只自动合并精确键、保守标记疑似重复的 `merge_publications.py``brief/paper/review/rebuttal/patent/standard/analyze` 和 Skill 清单统一改走新入口,不提升版本号。
- **08-26 / Unreleased / paper_server 宿主侧受控访问**:新增按配置注册的 `paper_server_search` / `paper_server_get` / `paper_server_fetch` typed tools`publication_type` 直接映射后端 `type` 过滤且保留开放原始类型API Key 从 `run_python` 和 Docker sandbox 透传中移除,下载增加同源 URL、100 MiB 上限和原子落盘约束。literature / brief 指引、smoke、RUN、Skill 清单及回归测试同步迁移,线上只读 `type=book` 验证通过,不提升版本号。
- **08-25 / Unreleased / literature 统一出版物检索入口**:删除按数据源拆分的 `research` / `documents` skill新增 `literature` 统一论文、书籍、章节及未来学术与技术出版物的发现、获取、跨源去重和证据核验paper_server helper 归入新 skill内部材料库通过宿主侧平台来源保持密钥隔离。新增开放出版物模型、来源与证据 references以及只自动合并精确键、保守标记疑似重复的 `merge_publications.py``brief/paper/review/rebuttal/patent/standard/analyze` 和 Skill 清单统一改走新入口,不提升版本号。
- **08-24 / 0.68.0 / Origin 小提琴分布图**`origin.plot@v2` / adapter 1.3.0 新增 `violin`,每条 `input/y` 系列作为一组原始观测Worker 复制到受控全 Y 展示表并通过 Origin 2024 原生 `Violin.otpu` 一次性生成分类分布图;默认隐藏冗余图例,保留分类标签并为 X 轴标题预留底部空间。首版不开放逐系列 style、分裂/半小提琴及带箱线变体,避免把未稳定映射的模板属性写入公共契约。固定 QA 新增三组抗压强度分布、中位数 oracle、OPJU 重开与全格式复导Origin 2024 / originpro 1.1.15 数据、视觉与进程释放门禁通过Origin/合同/Job/Node 专项 126 项 unittest、Ruff 致命规则、py_compile、diff 检查与独立 adapter 打包通过,未连接或写入数据库。
- **08-24 / 0.68.0 / Origin 材料谱图错位叠加**`origin.plot@v2` / adapter 1.2.0 新增 `stacked_line`,面向 XRD、XPS 和光谱多曲线比较请求显式声明间隔百分比Worker 保留原始工作表并生成基线对齐的受控展示表,按全体最大谱幅确定逐条偏移,允许不同 X 采样点且不依赖节点模板。默认使用曲线右端直标并按画布定位标题,显式图例请求仍兼容。固定 QA harness 同步修正 OriginPro 列式读取、空白工作簿及短/长名称和有效列标签识别Origin 2024 / originpro 1.1.15 真机生成、OPJU 重开、PNG/SVG/PDF 复导、偏移基线 oracle、视觉检查与进程释放通过。Origin/合同/Job/Node 专项 124 项 unittest、ruff 致命规则、py_compile 与独立 adapter 打包校验通过,未连接或写入数据库。
@ -253,7 +255,7 @@
- **07-15 / 0.58.32**:空响应防御——provider 吐空 tool_calls + 空正文原被当"答完"静默 done(隐蔽卡死);`_is_empty_response` 接 attempt 循环走非流式重试,耗尽仍空发**可见 warn**「已重试仍空,回复继续可重试」+ toolfail 聚集留痕,不引入终态 error。刻意不做内容嗅探式重试(假阳性更糟)。
- **07-15 / 0.58.31**:撤回 §8.11 delegate(0.58.30 曾整体落地,revert + force-push 抹除)——机制没错但触发信号没坐实(motivating 案更像批量扇出而非探索检索);重建前先用 diag 确认是真探索(依赖前序结果、无法一次列全)才 delegate,是批量走批量工具。只留验证过的批量工具(0.58.28)。
- **07-15 / 0.58.29**:修对话正文文件锚点两回归——① `media.js` `_TAIL_CLS` 误排间隔号 `·`(文件名合法分隔符)致含 `·` 的 pptx 丢 chip;② 正文相对路径链接点击整页 404,`chat.js` 加 `.msg .body a[href]` 拦截(外链新标签 / 内部 openFilePreview)。
- **07-15 / 0.58.28**:`mp_search_summary` 加批量入参 `formulas=[...]`(检索型烧 token 头号定点解)——pymatgen task 562 次逐式调用(91% 重复),批量化并成 1 次工具调用、复用单 MPRester 会话、中间数据落盘不进对话、只回紧凑摘要 + 路径;单查询路径字节不变向后兼容。"已知清单扇出"正解是工具批量化,非 delegate 子循环。
- **07-15 / 0.58.28**:`materials_project_search` 的批量入参 `formulas=[...]`(检索型烧 token 头号定点解)——pymatgen task 562 次逐式调用(91% 重复),批量化并成 1 次工具调用、复用单 MPRester 会话、中间数据落盘不进对话、只回紧凑摘要 + 路径。"已知清单扇出"正解是工具批量化,非 delegate 子循环。
- **07-15 / 0.58.27**:对话卡片悬停「复制」按钮(仿 GPT/DeepSeek)——取正文 Markdown 原文(据 `data-idx``loadedMessages`,不从渲染 HTML 反推),clipboard 主路径 + execCommand 降级;范围收敛只做复制(后端纯追加无重跑/截断语义,重生成/编辑留待后端支持)。
- **07-15 / 0.58.26**:用户停止后「已停止」持久提示(补 cancel 终态,与 error 对称)——cancel 提升为一等持久终态 `run_status="cancelled"`(Text 列无 migration),前端 `renderPersistedRunTerminal` 认 error/cancelled、刷新/切任务仍在;删被收尾 loadMessages 冲掉的 live badge。蓝绿旧实例读到当良性态优雅退化。
- **07-15 / 0.58.25**:skill 定向模型豁免——current profile family==`unifyllm`(已选国际旗舰)时不切走 pin 的国产档,尊重用户显式选择;判据放 DB 写入前无副作用。DESIGN §3.5 记。

19
RUN.md
View File

@ -2,7 +2,7 @@
> 怎么把 zcbot 跑起来。env / 常用命令 / 故障兜底。设计看 `DESIGN.md`,进度看 `PROGRESS.md`
最后更新:2026-08-10(外部系统统一为通用 OpenAPI/MCP definition)
最后更新:2026-08-26(platform_sources 独立平台托管来源)
---
@ -20,19 +20,21 @@
# 3) 视频生成 seedance tool(Seedance 2.0 Fast,文生/单图首帧/最多9张多图参考,480p 4s ¥1.86 ~ 720p 15s ¥12+,异步等 30-90s)
# 未设:豆包文本模型选不了,seedream/seedance 两个 tool 都不出现
ARK_API_KEY=...
# documents skill(内部知识库 document_search API):可选。设了后注册
# document_list_kb / document_search / document_download 三个 host-side tool;
# 内部材料知识库:可选。设了后注册 materials_library_list /
# materials_library_search / materials_library_fetch 三个 host-side tool;
# key 只留宿主后端,sandbox/run_python 不读取。
DOCUMENT_SEARCH_API_KEY=...
# 可选:覆盖默认 base_url(默认 https://ai.ctc-zc.com:8100/api)
# DOCUMENT_SEARCH_URL=https://ai.ctc-zc.com:8100/api
# pymatgen skill 的 Materials Project 接入:可选。设了后注册
# mp_search_summary / mp_get_structure / mp_get_entries 三个 host-side tool;
# materials_project_search / materials_project_get_structure /
# materials_project_get_entries 三个 host-side tool;
# 离线分析(CIF / POSCAR + SpacegroupAnalyzer + XRDCalculator + CEMENT_PHASES)仍在 sandbox 跑。
# 申请 https://materialsproject.org/api(免费)
MP_API_KEY=...
# literature skill 的 paper_server 文献库:GET 接口要 API Key(paper_server Django admin
# 里建 ApiKey)。设了后注册 paper_search / paper_get / paper_fetch 三个 host-side tool;
# 里建 ApiKey)。设了后注册 paper_server_search / paper_server_get /
# paper_server_fetch 三个 host-side tool;
# key 不进入 sandbox。未设时本轮不注册这三个工具,literature 会降级使用其他来源。
PAPER_SERVER_API_KEY=...
# 可选:覆盖 paper_server 地址(host 与 docker 模式均生效)
@ -139,6 +141,7 @@
# ZCBOT_CREDENTIAL_MASTER_KEY=<至少 32 字符随机串>
```
> litellm 在 import 时副作用加载 .env;入口走 `main.py`,`.env` 自动生效。直跑 `python -c "from core.storage import ..."` 不经 litellm 链路时记得自己 `import litellm` 触发,或手动 `export ZCBOT_DB_URL=...`
- **平台托管来源配置**:`PAPER_SERVER_*`、`DOCUMENT_SEARCH_*`、`MP_API_KEY` 由宿主部署环境提供,修改后重启 web 生效不进数据库、用户连接、prompt、tool result、`run_python` 或 Docker sandbox。若需求升级为用户级凭据/per-user grant接入 `external_systems`;若需要在线动态 definition、revision/reverify 或 OAuth先设计独立 `platform_sources` 控制面,不直接扩展当前静态 env registry。
- **依赖**:`pip install -r requirements.txt`(已在 `.venv` 里;含 `bcrypt`、`segno`、`cryptography`)。
- **微信接入(ClawBot,§8.7)**:① `main.py db upgrade head` 带上 migration `0012`;② `.env``ZCBOT_WECHAT_BOT_ENABLED=1` + `ZCBOT_WECHAT_SECRET_KEY=<串>`;③ 用户登录后点**左栏 rail「微信」按钮**(`/static/wechat_bind.html` 仍保留作独立/嵌入入口)扫码绑定(需个人微信 8.0.70+ 且灰度到 ClawBot 插件)。绑定后在微信「微信 ClawBot」对话即走 zcbot;**主动推送需用户近 24h 在微信开口过一次**(冷启动/超期推不出,退邮件兜底)。**支持语音消息**(voice_item SILK v3 → pilk 解码 → 讯飞 IAT 转写进对话,回执「🎤 已识别:…」;需 `XFYUN_*` 三件套 + ffmpeg + pilk,pilk 随 requirements 装)。
- **企业微信(渠道 B,纯推送,§8.7)**:① 管理员建自建应用 → 填 `WECOM_CORPID/AGENTID/SECRET`(+ 可见范围含目标用户);② `main.py db upgrade head`。**绑定两条路,任选**:
@ -965,7 +968,7 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_<user_uuid>" /opt
| DB 级单测把行写进了生产库 | 测试只认 `ZCBOT_TEST_DB_URL`(见「环境」段);别把它指向 `.env` 里的隧道 URL。已发生的:按测试专属 email(`test-*@invalid.local`)过滤清理 usage_events/tasks/scheduled_jobs/users |
| `ModuleNotFoundError: litellm` | 用了全局 `python`,改 `.venv/Scripts/python.exe ...` |
| Windows 控制台 emoji 崩 | Python stdout 是 GBK。用 `[OK]` / `[ng]` 等 ASCII 标签(见 memory) |
| `paper_search` 等工具未出现或报 paper_server 认证失败 | `.env``PAPER_SERVER_API_KEY=`(paper_server Django admin 里建 ApiKey)并重启 web 进程工具在宿主侧调用key 不进入 sandbox |
| `paper_server_search` 等工具未出现或报 paper_server 认证失败 | `.env``PAPER_SERVER_API_KEY=`(paper_server Django admin 里建 ApiKey)并重启 web 进程工具在宿主侧调用key 不进入 sandbox |
| `db upgrade``column already exists` | DB 已被改过,`db current` 确认 revision,必要时手 ALTER 或 `db downgrade base` 重来 |
| Resume 找不到 task | dev SPA 左侧 task 列表看 task_id 是否在;或 `curl /v1/tasks` 拉 |
| task 删了文件还在 | 现在 `DELETE /v1/tasks/{id}` 是**软删**,本就不动任何磁盘文件(留作语料 + 可恢复);要清磁盘走 `POST /v1/files/delete`。彻底物理删 task(及 messages)留给将来的管理员清理工具;当前如需手动:`psql> DELETE FROM tasks WHERE task_id=...`(messages/usage_events CASCADE) |
@ -1007,10 +1010,10 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_<user_uuid>" /opt
| `[startup] reaped N stale active run(s)` | 上次 web 进程未正常 finish 留下 N 个孤儿 run,启动 lifespan 自动标 error。info 级,无需处理 |
| `seedream` tool 没出现在对话里 | `.env` 没设 `ARK_API_KEY`,build_agent 跳过注册。设了重启 web 即可;无需迁移、无需 DB 改动 |
| 图像模型选了「GPT 生图」却仍走 seedream / 报错 | `.env` 没设 `UNIFYLLM_API_KEY`(gpt_image tool 未注册,静默 fallback 豆包);或服务器没代理出口(直连 unifyllm.ai TLS 失败)。跑 `scripts/diag_unifyllm.py` 验证连通后重启 web |
| `document_*` tool 没出现在对话里 | `.env` 没设 `DOCUMENT_SEARCH_API_KEY`,build_agent 跳过注册。设了重启 web 即可;key 不进入 sandbox。 |
| `materials_library_*` tool 没出现在对话里 | `.env` 没设 `DOCUMENT_SEARCH_API_KEY`,build_agent 跳过注册。设了重启 web 即可;key 不进入 sandbox。 |
| 文件区点 `.pptx` 弹"服务器未装 LibreOffice"/ `office_to_pdf` 提示缺 Writer/Calc/Impress | web host(非 sandbox)缺对应 LibreOffice 组件。`sudo apt-get install -y --no-install-recommends libreoffice-writer libreoffice-calc libreoffice-impress fonts-noto-cjk` 后**重启 web**。dev(Windows)`winget install TheDocumentFoundation.LibreOffice`。验:`soffice --version`,再分别拿一个 `.docx/.xlsx/.pptx``office_to_pdf` 冒烟。 |
| `.pptx` 预览首次慢几秒 | 正常 —— soffice 冷启 + 转换 ~2-4s,转完缓存到源同目录 `.preview/<stem>.<hash>.pdf`,再点即时。源文件一改(mtime/size 变)hash 变、自动重转 |
| `mp_*` tool 没出现在对话里 | `.env` 没设 `MP_API_KEY`,build_agent 跳过注册。设了重启 web 即可;Materials Project 联网查询走 host-side tool,离线 pymatgen 不受影响。 |
| `materials_project_*` tool 没出现在对话里 | `.env` 没设 `MP_API_KEY`,build_agent 跳过注册。设了重启 web 即可;Materials Project 联网查询走 host-side tool,离线 pymatgen 不受影响。 |
| 豆包调价了 | 改 `config/media/doubao.yaml``price_cny_per_image` 一行 → 重启 web。**历史 usage_events 不受影响**(units jsonb 里有当时单价 snapshot,聚合查仍按旧价);新写入按新价。涨价瞬间到改 YAML 中间这段记账偏低,开发期接受 |
| `kill -HUP <pid>``/openapi.json` 没新接口 | uvicorn **不响应 SIGHUP**(没装 handler,落 Python 默认终止;Windows 上信号本身无效)。Ubuntu 上用 `systemctl restart zcbot`,或 unit 加 `--reload` 让 uvicorn 监听文件自动重起(见"部署"段)。验证:`curl -s http://127.0.0.1:8765/openapi.json \| python3 -c 'import sys,json;print([p for p in json.load(sys.stdin)["paths"] if "auth" in p])'` |
| `systemctl restart zcbot` 要等几十秒才退 | 正常 —— 优雅 drain 在等在跑的 run 收尾(`shutdown.drain_timeout` 默 30s),没在跑 run 时秒退。journal 出现 `[shutdown] draining N in-flight run(s)` 即正常。真急(不在乎杀掉在跑 run):`systemctl kill -s KILL zcbot` |

View File

@ -1,7 +1,7 @@
# zcbot Skill 清单
服务对象:中国建筑材料科学研究总院 —— 无机非金属材料 R&D(水泥 / 混凝土 / 玻璃 / 陶瓷 / 耐火 / 新型建材)
最后更新:2026-08-26(literature 的 paper_server 改用宿主侧受控工具并支持出版物类型前置过滤)
最后更新:2026-08-26(平台托管来源统一改用来源明确的宿主侧工具)
Skill 总数:17
zcbot 的"skill"是一份可加载的工作流脚本(`skills/<name>/SKILL.md` + 配套 templates / scripts / Python helper),模型在识别用户意图后挂载对应 skill,按其内置的阶段化流程产出可交付物。本文档面向**使用方 / 协作方**,按"做什么、什么时候用、什么时候别用、典型产物"组织。
@ -264,7 +264,7 @@ zcbot 的"skill"是一份可加载的工作流脚本(`skills/<name>/SKILL.md` +
证据统一标记为 `metadata_only` / `abstract_verified` / `snippet_verified` / `fulltext_verified`。下载原件但未实际读取,不算全文已核对;出版物真实存在也不代表它支持当前论断。
**主要能力**:host-side `paper_search` / `paper_get` / `paper_fetch`(支持 `publication_type` 服务端过滤),以及 `document_list_kb` / `document_search` / `document_download``merge_publications.py` 负责确定性的标准化、精确合并和疑似重复标记。
**主要能力**:host-side `paper_server_search` / `paper_server_get` / `paper_server_fetch`(支持 `publication_type` 服务端过滤),以及 `materials_library_list` / `materials_library_search` / `materials_library_fetch``merge_publications.py` 负责确定性的标准化、精确合并和疑似重复标记。
**典型产物**:多类型出版物候选清单、规范化 `publications.json`、可核验摘要或全文、引文与论断证据台账。
@ -327,7 +327,7 @@ zcbot 的"skill"是一份可加载的工作流脚本(`skills/<name>/SKILL.md` +
- 正向算 XRD pattern → `XRDCalculator`,跟实测谱对比
- 物相对称性 → `SpacegroupAnalyzer`
- 凝胶 / 水化产物热力学稳定性 → `PhaseDiagram` + `PDEntry`
- 从 MP 拉已知结构 / 性质 → `mp_search_summary` / `mp_get_structure` / `mp_get_entries`
- 从 MP 拉已知结构 / 性质 → `materials_project_search` / `materials_project_get_structure` / `materials_project_get_entries`
---

View File

@ -100,7 +100,7 @@ class _RepeatGuard:
"""检测「同名同参 + 无产出」的病理性重复调用,断掉死循环。
背景(2026-06-08 DB 实测):高轮数烧 token task ,单个工具被用**完全相同的
参数**重复调用几十上百次(`document_search` 122 空参数 `shell{}` 51 反复
参数**重复调用几十上百次(`materials_library_search` 122 空参数 `shell{}` 51 反复
`glob` 同一个不存在的路径)loop 原本对此零防护,照单全收直到撞 max_iterations
命门是只惩罚无产出重复,绝不误伤正常迭代:

View File

@ -16,7 +16,6 @@
"""
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Optional
@ -24,7 +23,6 @@ from uuid import UUID
from tools.ask_user import AskUserTool
from tools.check_process import CheckProcessTool
from tools.documents import DocumentDownloadTool, DocumentListKbTool, DocumentSearchTool
from tools.external_systems import (
ExternalSystemCallTool,
ExternalSystemListTool,
@ -35,13 +33,7 @@ from tools.external_systems import (
from tools.fs import EditTool, GlobTool, GrepTool, ReadTool, WriteTool
from tools.gpt_image import GptImageTool
from tools.look_at_image import LookAtImageTool
from tools.materials_project import (
MaterialsProjectGetEntriesTool,
MaterialsProjectGetStructureTool,
MaterialsProjectSearchSummaryTool,
)
from tools.office_to_pdf import OfficeToPdfTool
from tools.paper_server import PaperFetchTool, PaperGetTool, PaperSearchTool
from tools.read_document import ReadDocumentTool
from tools.register_artifact import RegisterArtifactTool
from tools.rename_working_dir import RenameWorkingDirTool
@ -72,6 +64,7 @@ from tools.wechat_bot import WechatPushTool, wechat_push_available
from core.asr_lfasr import is_configured as lfasr_configured
from core.bocha_client import BochaConfig
from core.task_actions import DeferredTaskActions
from platform_sources import PlatformSourceContext, build_platform_source_tools
@dataclass
@ -98,10 +91,6 @@ class ToolContext:
office_to_pdf_available: bool # backend host 是否可调用 LibreOffice
def _env_set(name: str) -> Callable[[], bool]:
return lambda: bool(os.getenv(name, "").strip())
def _pick_variant(section: dict, preferred: str = "") -> tuple[str, Optional[dict]]:
"""从 yaml 段选 variant:preferred 命中优先,否则第一个 dict 条目;无 → ("", None)。"""
if preferred:
@ -164,26 +153,14 @@ def build_tools(ctx: ToolContext) -> dict[str, Any]:
),
]
def _document_search() -> list:
return [
DocumentListKbTool(**base),
DocumentSearchTool(**base),
DocumentDownloadTool(working_dir=ctx.working_dir_path, **base),
]
def _materials_project() -> list:
return [
MaterialsProjectSearchSummaryTool(working_dir=ctx.working_dir_path, **base),
MaterialsProjectGetStructureTool(working_dir=ctx.working_dir_path, **base),
MaterialsProjectGetEntriesTool(working_dir=ctx.working_dir_path, **base),
]
def _paper_server() -> list:
return [
PaperSearchTool(**base),
PaperGetTool(**base),
PaperFetchTool(working_dir=ctx.working_dir_path, **base),
]
def _platform_sources() -> list:
return build_platform_source_tools(
PlatformSourceContext(
base_dir=ctx.tool_base,
user_root=ctx.ur_path,
working_dir=ctx.working_dir_path,
)
)
def _external_system_status() -> list:
return [ExternalSystemListTool(ctx.uid, **base)]
@ -320,11 +297,8 @@ def build_tools(ctx: ToolContext) -> dict[str, Any]:
("core", lambda: True, _core),
# 当前 working_dir 只能延迟到交互 run 收尾后改名;定时 run 不允许自行改目录。
("task_actions", lambda: not ctx.scheduled_run, _task_actions),
# Secret-bearing 域工具一律 host-side、仅对应 env 存在才注册(§7.5 #7):
# key 绝不进 run_python / 沙箱。
("document_search", _env_set("DOCUMENT_SEARCH_API_KEY"), _document_search),
("materials_project", _env_set("MP_API_KEY"), _materials_project),
("paper_server", _env_set("PAPER_SERVER_API_KEY"), _paper_server),
# 平台托管来源由独立 bounded context 统一装配;来源配置和隔离不在 core 展开。
("platform_sources", lambda: True, _platform_sources),
(
"external_system_status",
lambda: _external_system_status_available(ctx.uid),

View File

@ -0,0 +1,13 @@
"""平台托管只读数据源的独立 bounded context。"""
from .registry import (
PlatformSourceContext,
PlatformSourceProvider,
build_platform_source_tools,
)
__all__ = [
"PlatformSourceContext",
"PlatformSourceProvider",
"build_platform_source_tools",
]

View File

@ -1,4 +1,4 @@
"""Host-side document_search tools.
"""Host-side internal materials library tools.
These tools intentionally keep DOCUMENT_SEARCH_API_KEY on the host side. The
sandbox receives only business arguments and trimmed results / saved paths.
@ -9,12 +9,13 @@ from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Optional
from . import document_client as doc_client
from . import materials_library_client as client
from .security import safe_error_text
from .base import Tool
from tools.base import Tool
_MAX_QUERIES = 8 # document_search 单次批量 query 上限
_MAX_DOWNLOADS = 10 # document_download 单次批量 item 上限
_MAX_QUERIES = 8 # materials_library_search 单次批量 query 上限
_MAX_DOWNLOADS = 10 # materials_library_fetch 单次批量 item 上限
_CONCURRENCY = 6
@ -36,19 +37,20 @@ def _dedup_keep_order(items: list[str]) -> list[str]:
return out
class DocumentListKbTool(Tool):
name = "document_list_kb"
class MaterialsLibraryListTool(Tool):
name = "materials_library_list"
description = (
"List internal materials knowledge bases available in document_search. "
"Use before document_search when the user did not specify a materials domain."
"List internal materials knowledge bases available to materials_library_search. "
"Use before searching when the user did not specify a materials domain."
)
parameters = {"type": "object", "properties": {}}
def execute(self) -> str:
try:
kbs = doc_client.list_kb()
kbs = client.list_kb()
except Exception as e:
return f"[Error] document_list_kb failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("DOCUMENT_SEARCH_API_KEY",))
return f"[Error] materials_library_list failed: {type(e).__name__}: {detail}"
if not kbs:
return "(no knowledge bases returned)"
lines = ["Knowledge bases:"]
@ -64,8 +66,8 @@ class DocumentListKbTool(Tool):
return "\n".join(lines)
class DocumentSearchTool(Tool):
name = "document_search"
class MaterialsLibrarySearchTool(Tool):
name = "materials_library_search"
description = (
"Search the internal materials document knowledge base with one OR MORE queries at once. "
"Pass every distinct query you want in a single `queries` list instead of calling this tool "
@ -85,7 +87,7 @@ class DocumentSearchTool(Tool):
"kb_names": {
"type": "array",
"items": {"type": "string"},
"description": "Optional knowledge-base names from document_list_kb (applies to all queries).",
"description": "Optional knowledge-base names from materials_library_list (applies to all queries).",
},
"classification_ids": {
"type": "array",
@ -116,14 +118,15 @@ class DocumentSearchTool(Tool):
) -> str:
"""搜单个 query,返回格式化文本块或 [Error ...];绝不抛异常(供并发安全调用)。"""
try:
docs = doc_client.search(
docs = client.search(
query=query,
kb_names=kb_names or None,
classification_ids=classification_ids or None,
max_documents=max_documents,
)
except Exception as e:
return f"[Error] document_search failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("DOCUMENT_SEARCH_API_KEY",))
return f"[Error] materials_library_search failed: {type(e).__name__}: {detail}"
if not docs:
return f"(no documents found for query: {query!r})"
@ -188,13 +191,13 @@ class DocumentSearchTool(Tool):
return out
class DocumentDownloadTool(Tool):
name = "document_download"
class MaterialsLibraryFetchTool(Tool):
name = "materials_library_fetch"
description = (
"Download one OR MORE original documents from document_search into task_dir/documents/. "
"Download one OR MORE original documents from materials_library_search into task_dir/documents/. "
"Pass every document you want in a single `items` list instead of calling this tool repeatedly — "
"downloads run concurrently and one failing item does not abort the others. "
"Use the file_name and kb_name returned by document_search."
"Use the file_name and kb_name returned by materials_library_search."
)
parameters = {
"type": "object",
@ -204,8 +207,8 @@ class DocumentDownloadTool(Tool):
"items": {
"type": "object",
"properties": {
"file_name": {"type": "string", "description": "Original file_name or md_filename returned by document_search."},
"kb_name": {"type": "string", "description": "Knowledge-base name returned by document_search."},
"file_name": {"type": "string", "description": "Original file_name or md_filename returned by materials_library_search."},
"kb_name": {"type": "string", "description": "Knowledge-base name returned by materials_library_search."},
"preview": {"type": "boolean", "description": "Request inline preview disposition. Usually false."},
},
"required": ["file_name", "kb_name"],
@ -235,14 +238,15 @@ class DocumentDownloadTool(Tool):
if not file_name or not kb_name:
return f"[Error] file_name / kb_name 不可为空: {item!r}"
try:
rel = doc_client.download(
rel = client.download(
file_name=file_name,
kb_name=kb_name,
working_dir=str(self.working_dir),
preview=bool(item.get("preview", False)),
)
except Exception as e:
return f"[Error] download {file_name!r} failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("DOCUMENT_SEARCH_API_KEY",))
return f"[Error] download {file_name!r} failed: {type(e).__name__}: {detail}"
return f"saved: {self._display(self.working_dir / rel)}"
def execute(self, items: list[dict] | dict) -> str:

View File

@ -1,4 +1,4 @@
"""内部材料知识库的 host-side 客户端。
"""platform_sources 内部材料知识库的 host-side 客户端。
base_url / api_key env:
DOCUMENT_SEARCH_URL 默认 https://ai.ctc-zc.com:8100/api

View File

@ -11,7 +11,9 @@ import os
from pathlib import Path
from typing import Any, Optional
from .base import Tool
from tools.base import Tool
from .security import safe_error_text
try: # patched in tests; missing dependency should produce a clean tool error.
from mp_api.client import MPRester # type: ignore
@ -59,15 +61,16 @@ def _mpr():
return MPRester(_mp_key())
class MaterialsProjectSearchSummaryTool(Tool):
name = "mp_search_summary"
class MaterialsProjectSearchTool(Tool):
name = "materials_project_search"
description = (
"Search Materials Project summary data using the host MP_API_KEY. "
"Single query: pass formula / material_ids / elements → returns trimmed JSON. "
"Batch: pass `formulas` (a list) to look up MANY formulae in ONE tool call — "
"full results are written to task_dir/materials/*.json and only a compact "
"per-formula digest is returned. Always prefer batch over calling this tool "
"once per formula. Use mp_get_structure to save a CIF for offline pymatgen analysis."
"once per formula. Use materials_project_get_structure to save a CIF for "
"offline pymatgen analysis."
)
parameters = {
"type": "object",
@ -125,7 +128,8 @@ class MaterialsProjectSearchSummaryTool(Tool):
num_chunks=1, chunk_size=limit, **kwargs
)
except Exception as e:
return f"[Error] mp_search_summary failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("MP_API_KEY",))
return f"[Error] materials_project_search failed: {type(e).__name__}: {detail}"
plain = [_to_plain(d) for d in list(docs)[:limit]]
return json.dumps(plain, ensure_ascii=False, indent=2)
@ -148,7 +152,8 @@ class MaterialsProjectSearchSummaryTool(Tool):
try:
session = _mpr()
except Exception as e:
return f"[Error] mp_search_summary batch failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("MP_API_KEY",))
return f"[Error] materials_project_search batch failed: {type(e).__name__}: {detail}"
agg: list[dict[str, Any]] = []
n_ok = 0
# 单个 formula 出错不连坐整批;复用一个 MPRester 会话避免逐条重认证。
@ -164,13 +169,14 @@ class MaterialsProjectSearchSummaryTool(Tool):
n_ok += 1
agg.append({"formula": f, "n": len(results), "results": results})
except Exception as e:
agg.append({"formula": f, "error": f"{type(e).__name__}: {e}"})
detail = safe_error_text(e, ("MP_API_KEY",))
agg.append({"formula": f, "error": f"{type(e).__name__}: {detail}"})
return self._render_batch(agg, fields, n_ok)
def _render_batch(self, agg: list[dict[str, Any]], fields: list[str], n_ok: int) -> str:
empties = [a["formula"] for a in agg if "error" in a or a.get("n", 0) == 0]
header = (
f"mp_search_summary batch: {len(agg)} 个化学式 → {n_ok} 个有结果,"
f"materials_project_search batch: {len(agg)} 个化学式 → {n_ok} 个有结果,"
f"{len(empties)} 个空/错误。\n"
f"每条记录字段: {', '.join(fields)}\n"
)
@ -192,7 +198,8 @@ class MaterialsProjectSearchSummaryTool(Tool):
)
except Exception as e:
body = (
f"[warn] 写文件失败({type(e).__name__}: {e}),改为内联返回完整 JSON:\n"
f"[warn] 写文件失败({type(e).__name__}: "
f"{safe_error_text(e, ('MP_API_KEY',))}),改为内联返回完整 JSON:\n"
+ json.dumps(agg, ensure_ascii=False, indent=2)
+ "\n"
)
@ -207,7 +214,7 @@ class MaterialsProjectSearchSummaryTool(Tool):
class MaterialsProjectGetStructureTool(Tool):
name = "mp_get_structure"
name = "materials_project_get_structure"
description = (
"Download a Materials Project structure by material_id and save it as CIF in task_dir/materials/."
)
@ -244,12 +251,13 @@ class MaterialsProjectGetStructureTool(Tool):
dest.parent.mkdir(parents=True, exist_ok=True)
struct.to(filename=str(dest))
except Exception as e:
return f"[Error] mp_get_structure failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("MP_API_KEY",))
return f"[Error] materials_project_get_structure failed: {type(e).__name__}: {detail}"
return f"saved: {self._display(dest)}"
class MaterialsProjectGetEntriesTool(Tool):
name = "mp_get_entries"
name = "materials_project_get_entries"
description = (
"Fetch Materials Project computed entries for a chemical system and save trimmed JSON to task_dir/materials/. "
"Downloads the FULL chemical system (all sub-systems) — volume grows fast with element count; call sparingly and reuse the saved file rather than re-querying."
@ -297,5 +305,6 @@ class MaterialsProjectGetEntriesTool(Tool):
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
except Exception as e:
return f"[Error] mp_get_entries failed: {type(e).__name__}: {e}"
detail = safe_error_text(e, ("MP_API_KEY",))
return f"[Error] materials_project_get_entries failed: {type(e).__name__}: {detail}"
return f"saved: {self._display(dest)}"

View File

@ -15,7 +15,9 @@ from urllib.parse import urljoin, urlparse
import httpx
from .base import Tool
from tools.base import Tool
from .security import safe_error_text
_DEFAULT_BASE_URL = "http://paper.xxhhcty.xyz:8080"
_TIMEOUT = 30.0
@ -153,8 +155,8 @@ def _media_url(raw_url: str, base_url: str) -> str:
return url
class PaperSearchTool(Tool):
name = "paper_search"
class PaperServerSearchTool(Tool):
name = "paper_server_search"
description = (
"Search the platform paper_server metadata collection. "
"Use publication_type for server-side source-type filtering, e.g. 'book', "
@ -245,15 +247,16 @@ class PaperSearchTool(Tool):
_base_url, api_url, api_key = _config()
papers = _results(_get_json(api_url + "/", api_key=api_key, params=params))
except Exception as exc:
return f"[Error] paper_search failed:{type(exc).__name__}:{exc}"
detail = safe_error_text(exc, ("PAPER_SERVER_API_KEY",))
return f"[Error] paper_server_search failed:{type(exc).__name__}:{detail}"
trimmed = [
{key: paper.get(key) for key in _LIST_FIELDS} for paper in papers[:limit]
]
return json.dumps(trimmed, ensure_ascii=False, indent=2)
class PaperGetTool(Tool):
name = "paper_get"
class PaperServerGetTool(Tool):
name = "paper_server_get"
description = (
"Get one complete paper_server metadata record by internal id or exact DOI."
)
@ -269,15 +272,17 @@ class PaperGetTool(Tool):
try:
paper = _get_paper(id_or_doi)
except Exception as exc:
return f"[Error] paper_get failed:{type(exc).__name__}:{exc}"
detail = safe_error_text(exc, ("PAPER_SERVER_API_KEY",))
return f"[Error] paper_server_get failed:{type(exc).__name__}:{detail}"
return json.dumps(paper, ensure_ascii=False, indent=2)
class PaperFetchTool(Tool):
name = "paper_fetch"
class PaperServerFetchTool(Tool):
name = "paper_server_fetch"
description = (
"Download an available PDF or XML from paper_server into the current task's "
"papers/ directory. Use the format actually reported by paper_search/paper_get."
"papers/ directory. Use the format actually reported by "
"paper_server_search/paper_server_get."
)
parameters = {
"type": "object",
@ -358,5 +363,6 @@ class PaperFetchTool(Tool):
except OSError:
pass
except Exception as exc:
return f"[Error] paper_fetch failed:{type(exc).__name__}:{exc}"
detail = safe_error_text(exc, ("PAPER_SERVER_API_KEY",))
return f"[Error] paper_server_fetch failed:{type(exc).__name__}:{detail}"
return f"saved:{self._display(destination)}"

View File

@ -0,0 +1,159 @@
"""平台托管来源的显式注册表和统一生命周期入口。
Provider 只统一来源标识能力声明可用性和 Tool 装配各来源的业务 API
保持专用 typed contract不在这里抽象成 search/get/fetch 万能接口
"""
from __future__ import annotations
import logging
import os
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
from urllib.parse import urlparse
from tools.base import Tool
from .materials_library import (
MaterialsLibraryFetchTool,
MaterialsLibraryListTool,
MaterialsLibrarySearchTool,
)
from .materials_project import (
MPRester,
MaterialsProjectGetEntriesTool,
MaterialsProjectGetStructureTool,
MaterialsProjectSearchTool,
)
from .paper_server import (
PaperServerFetchTool,
PaperServerGetTool,
PaperServerSearchTool,
)
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class PlatformSourceContext:
"""Provider 装配 Tool 所需的非敏感宿主上下文。"""
base_dir: Path
user_root: Path
working_dir: Path
class PlatformSourceProvider(Protocol):
source_id: str
capabilities: frozenset[str]
def available(self) -> bool: ...
def build_tools(self, context: PlatformSourceContext) -> list[Tool]: ...
def _env_set(name: str) -> bool:
return bool(os.environ.get(name, "").strip())
def _require_valid_http_url(name: str, default: str) -> None:
value = os.environ.get(name, default).strip()
parsed = urlparse(value)
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
raise ValueError(f"invalid {name}")
class PaperServerProvider:
source_id = "paper_server"
capabilities = frozenset({"search", "metadata", "download"})
def available(self) -> bool:
if not _env_set("PAPER_SERVER_API_KEY"):
return False
_require_valid_http_url("PAPER_SERVER_URL", "http://paper.xxhhcty.xyz:8080")
return True
def build_tools(self, context: PlatformSourceContext) -> list[Tool]:
common = {"base_dir": context.base_dir, "user_root": context.user_root}
return [
PaperServerSearchTool(**common),
PaperServerGetTool(**common),
PaperServerFetchTool(working_dir=context.working_dir, **common),
]
class MaterialsLibraryProvider:
source_id = "materials_library"
capabilities = frozenset({"catalog", "batch_search", "batch_download"})
def available(self) -> bool:
if not _env_set("DOCUMENT_SEARCH_API_KEY"):
return False
_require_valid_http_url(
"DOCUMENT_SEARCH_URL", "https://ai.ctc-zc.com:8100/api"
)
return True
def build_tools(self, context: PlatformSourceContext) -> list[Tool]:
common = {"base_dir": context.base_dir, "user_root": context.user_root}
return [
MaterialsLibraryListTool(**common),
MaterialsLibrarySearchTool(**common),
MaterialsLibraryFetchTool(working_dir=context.working_dir, **common),
]
class MaterialsProjectProvider:
source_id = "materials_project"
capabilities = frozenset({"summary_search", "structure", "entries"})
def available(self) -> bool:
if not _env_set("MP_API_KEY"):
return False
if MPRester is None:
raise RuntimeError("Materials Project SDK unavailable")
return True
def build_tools(self, context: PlatformSourceContext) -> list[Tool]:
common = {"base_dir": context.base_dir, "user_root": context.user_root}
return [
MaterialsProjectSearchTool(working_dir=context.working_dir, **common),
MaterialsProjectGetStructureTool(
working_dir=context.working_dir, **common
),
MaterialsProjectGetEntriesTool(
working_dir=context.working_dir, **common
),
]
# 显式可信列表:不扫描目录、不动态 import 部署侧 definition。
TRUSTED_PROVIDERS: tuple[PlatformSourceProvider, ...] = (
PaperServerProvider(),
MaterialsLibraryProvider(),
MaterialsProjectProvider(),
)
def build_platform_source_tools(
context: PlatformSourceContext,
*,
providers: Sequence[PlatformSourceProvider] = TRUSTED_PROVIDERS,
) -> list[Tool]:
"""逐来源隔离装配;失败只跳过该来源,日志不含配置值或异常正文。"""
built: list[Tool] = []
for provider in providers:
source_id = getattr(provider, "source_id", "unknown")
try:
if not provider.available():
continue
built.extend(provider.build_tools(context))
except Exception as exc:
logger.warning(
"platform source %s registration failed (%s)",
source_id,
type(exc).__name__,
)
return built

View File

@ -0,0 +1,21 @@
"""平台来源错误脱敏原语。"""
from __future__ import annotations
import os
import re
from collections.abc import Iterable
_QUERY_SECRET_RE = re.compile(
r"(?i)(api[_-]?key|token|secret|password)(=|%3[dD])([^&\s]+)"
)
def safe_error_text(exc: BaseException, secret_env_names: Iterable[str]) -> str:
"""返回可诊断但不包含部署凭据的异常文本。"""
text = str(exc)
for name in secret_env_names:
value = os.environ.get(name, "")
if value:
text = text.replace(value, "[REDACTED]")
return _QUERY_SECRET_RE.sub(r"\1\2[REDACTED]", text)

View File

@ -35,7 +35,11 @@ if env_file.exists():
import json
from tools.paper_server import PaperFetchTool, PaperGetTool, PaperSearchTool
from platform_sources.paper_server import (
PaperServerFetchTool,
PaperServerGetTool,
PaperServerSearchTool,
)
_BASE_URL = os.environ.get("PAPER_SERVER_URL", "http://paper.xxhhcty.xyz:8080").rstrip("/")
@ -48,15 +52,15 @@ def _json_result(raw: str):
def search(**kwargs) -> list[dict]:
return _json_result(PaperSearchTool().execute(**kwargs))
return _json_result(PaperServerSearchTool().execute(**kwargs))
def get_paper(id_or_doi: str) -> dict:
return _json_result(PaperGetTool().execute(id_or_doi=id_or_doi))
return _json_result(PaperServerGetTool().execute(id_or_doi=id_or_doi))
def _fetch(id_or_doi: str, working_dir: str, file_format: str) -> str:
raw = PaperFetchTool(working_dir=Path(working_dir)).execute(
raw = PaperServerFetchTool(working_dir=Path(working_dir)).execute(
id_or_doi=id_or_doi, format=file_format
)
if raw.startswith("[Error]"):

View File

@ -4,11 +4,11 @@
依赖:`pip install pymatgen mp-api scikit-learn statsmodels`(PyMC 可选,装了就测)
不依赖网络默认情况下(MP_API_KEY 没配则跳过 mp_rester 联网那一)
不依赖网络默认情况下(MP_API_KEY 没配则跳过 Materials Project 联网)
不动 DB / workspace,产物落系统临时目录,跑完即丢
skill 顺序 4 :
step A pymatgen import + CEMENT_PHASES 几个查询 + mp_rester 未配 key
step A pymatgen import + CEMENT_PHASES 几个查询 + host tool 未配 key
step B stats_ml 三库装包 + OLS / RandomForest smoke
step C plot_pub apply_pub_style + 最小 XRD-like 图出 PNG
step D (可选)MP_API_KEY 配了就联网拉一条 Materials Project 数据
@ -137,14 +137,14 @@ def step_a_pymatgen() -> None:
_info("MP_API_KEY 已配置,skip 缺 key 报错验证(下面 step D 测真实查询)")
else:
try:
from tools.materials_project import MaterialsProjectSearchSummaryTool
out = MaterialsProjectSearchSummaryTool().execute(formula="Ca3SiO5")
from platform_sources.materials_project import MaterialsProjectSearchTool
out = MaterialsProjectSearchTool().execute(formula="Ca3SiO5")
if out.startswith("[Error]") and "MP_API_KEY" in out:
_ok("mp_search_summary 未配 key 返回 [Error] 含 MP_API_KEY 提示")
_ok("materials_project_search 未配 key 返回 [Error] 含 MP_API_KEY 提示")
else:
_fail(f"未配 key 应返回 [Error] 含 MP_API_KEY,实际: {out[:120]}")
except Exception as e:
_fail(f"mp_search_summary 未配 key 应返回 [Error] 而非抛异常: {type(e).__name__}: {e}")
_fail(f"materials_project_search 未配 key 应返回 [Error] 而非抛异常: {type(e).__name__}: {e}")
def step_b_stats_ml() -> None:
@ -271,24 +271,24 @@ def step_d_mp_online() -> None:
try:
import json
from skills.pymatgen.materials import lookup_phase
from tools.materials_project import MaterialsProjectSearchSummaryTool
from platform_sources.materials_project import MaterialsProjectSearchTool
formula = lookup_phase("C3S") # Ca3SiO5
t0 = time.time()
out = MaterialsProjectSearchSummaryTool().execute(
out = MaterialsProjectSearchTool().execute(
formula=formula,
fields=["material_id", "formula_pretty", "energy_above_hull"],
limit=3,
)
dt = (time.time() - t0) * 1000
if out.startswith("[Error]"):
_fail(f"mp_search_summary{formula}: {out[:160]}")
_fail(f"materials_project_search 查 {formula}: {out[:160]}")
return
docs = json.loads(out)
_ok(f"mp_search_summary{formula}: 返回 {len(docs)} 条 in {dt:.0f}ms")
_ok(f"materials_project_search 查 {formula}: 返回 {len(docs)} 条 in {dt:.0f}ms")
for d in docs[:3]:
print(f" {d.get('material_id')} {d.get('formula_pretty')} ehull={d.get('energy_above_hull')}")
except Exception as e:
_fail(f"mp_search_summary 联网查询: {type(e).__name__}: {e}")
_fail(f"materials_project_search 联网查询: {type(e).__name__}: {e}")
def main() -> int:

View File

@ -43,10 +43,10 @@ description: 生成科研方向简报(research direction briefing / 重要文献
**先读 `references/journals.md`**。**中文方向先转专业英文术语**(库主语料英文):低碳水泥→low-carbon cement / clinker substitution;SCM→supplementary cementitious materials / fly ash / GGBFS / calcined clay;LC3→limestone calcined clay cement;碳化养护→CO2 curing / carbonation。缩写与全称都试。
**literature / paper_server(逐刊取最新 Elsevier 论文 + DOI)** —— 用宿主侧 `paper_search`。按 `publication_name` 精确匹配并带时间窗;需要限定论文时传 `publication_type="article"`。list 自带 abstract看前 200400 字判切题与分量。目标期刊包括 `Cement and Concrete Research`、`Cement and Concrete Composites`、`Construction and Building Materials`、`Journal of Cleaner Production`。
**literature / paper_server(逐刊取最新 Elsevier 论文 + DOI)** —— 用宿主侧 `paper_server_search`。按 `publication_name` 精确匹配并带时间窗;需要限定论文时传 `publication_type="article"`。list 自带 abstract看前 200400 字判切题与分量。目标期刊包括 `Cement and Concrete Research`、`Cement and Concrete Composites`、`Construction and Building Materials`、`Journal of Cleaner Production`。
某刊精确名 0 命中 → 换 `keyword=<方向英文术语>` 再搜,从返回里挑 `publication_name` 命中目标刊的;仍空记"该刊本窗口库内无收录"。
**literature / 内部材料库(取全文,材料类首选)** —— host-side tool `document_search`,中英 query 都行(后端跨语言语义检索);胶凝材料库 `classification_id=1`。取 `md_content` 既做候选也供引文核验抓锚点最顺。
**literature / 内部材料库(取全文,材料类首选)** —— host-side tool `materials_library_search`,中英 query 都行(后端跨语言语义检索);胶凝材料库 `classification_id=1`。取 `md_content` 既做候选也供引文核验抓锚点最顺。
**web search(取动向)** —— 政策(双碳/碳配额)、标准(新国标/团标)、行业会议、企业产线中试。**单列"其他动向",不混进论文列表与总结**。

View File

@ -25,7 +25,7 @@ description: 检索、获取、合并与核验各类学术和技术出版物,
## 来源选择
- 材料主题、性能、配方、工艺、表征或全文语义检索:优先内部材料知识库。
- DOI、题名、作者、期刊、年份、出版物类型或跨学科发现`paper_search` 可用时优先 `paper_server`
- DOI、题名、作者、期刊、年份、出版物类型或跨学科发现`paper_server_search` 可用时优先 `paper_server`
- 系统调研、综述、重要引用或关键论断:并查可用来源。
- 用户指定来源:只查指定来源,除非该来源无法满足用户要求并需要说明降级方案。
- 某一路不可用:继续使用其他来源,并明确实际覆盖范围。

View File

@ -4,9 +4,9 @@
## 工具
- `document_list_kb()`:列出当前有效知识库。用户未指定材料方向且需要缩窄范围时调用。
- `document_search(queries, kb_names=None, classification_ids=None, max_documents=6, content_chars_per_doc=1200)`:批量检索并返回元数据和截断的 `md_content`
- `document_download(items)`:把选定原件下载到当前任务的 `documents/` 目录。
- `materials_library_list()`:列出当前有效知识库。用户未指定材料方向且需要缩窄范围时调用。
- `materials_library_search(queries, kb_names=None, classification_ids=None, max_documents=6, content_chars_per_doc=1200)`:批量检索并返回元数据和截断的 `md_content`
- `materials_library_fetch(items)`:把选定原件下载到当前任务的 `documents/` 目录。
查询可使用中文或英文;复杂专业术语通常使用英文更精确。先规划互不重复的查询并批量调用,不逐个近义词反复搜索。

View File

@ -6,9 +6,9 @@
使用宿主侧 typed tools凭据不会进入 sandbox
- `paper_search(keyword="", publication_type="", year=None, year_gte=None, year_lte=None, doi="", first_author="", publication_name="", has_pdf=None, is_oa=None, limit=10)`:搜索题录并返回摘要和全文可用状态。能确定来源类型时用 `publication_type` 做服务端过滤,例如 `book`、`book-chapter`、`article`
- `paper_get(id_or_doi)`:按内部 ID 或 DOI 获取完整记录。
- `paper_fetch(id_or_doi, format)`:把记录实际提供的 `pdf``xml` 保存到任务的 `papers/` 目录。
- `paper_server_search(keyword="", publication_type="", year=None, year_gte=None, year_lte=None, doi="", first_author="", publication_name="", has_pdf=None, is_oa=None, limit=10)`:搜索题录并返回摘要和全文可用状态。统一出版物类型 `book` 映射为来源原始值 `book``book_chapter` 映射为 `book-chapter`;能确定来源类型时用 `publication_type` 做服务端过滤
- `paper_server_get(id_or_doi)`:按内部 ID 或 DOI 获取完整记录。
- `paper_server_fetch(id_or_doi, format)`:把记录实际提供的 `pdf``xml` 保存到任务的 `papers/` 目录。
工具由平台自动使用 `PAPER_SERVER_API_KEY` 和可选的 `PAPER_SERVER_URL`未配置时本轮不会注册这些工具。只通过这些工具访问由它们处理认证、URL、响应裁剪和下载路径。

View File

@ -18,7 +18,7 @@
每条引文先确认"这篇文献真实存在":
1. 用 `literature` 检索:材料类优先内部材料库语义检索,精确 DOI 使用 paper_server 的 `paper_search(doi=...)` / `paper_get(id_or_doi=...)`
1. 用 `literature` 检索:材料类优先内部材料库语义检索,精确 DOI 使用 paper_server 的 `paper_server_search(doi=...)` / `paper_server_get(id_or_doi=...)`
2. 命中 → 记下真实 DOI / 作者 / 年份 / 期刊 / 卷期页;**以库里返回为准**,不沿用记忆里的字段
3. 两个库都查不到 → 标 `[未核实]`,**不得编造条目**;告诉用户"这条找不到来源,请提供 PDF/DOI 或删去该论断"
@ -34,7 +34,7 @@
最容易翻车的一层:文献存在,但**并不支撑你写的那句话**。逐条做:
1. 抓回该文献的 `md_content`(内部材料库正文片段),或用 paper_server 的 `paper_fetch(format="xml"|"pdf")` 获取原件
1. 抓回该文献的 `md_content`(内部材料库正文片段),或用 paper_server 的 `paper_server_fetch(format="xml"|"pdf")` 获取原件
2. 在原文里定位与论断相关的**锚点证据**:一句 ≤25 词的原文引语 + 出现的段落/小节位置
3. 判定支撑度三档:
- **support**:原文明确支撑该论断 → 通过

View File

@ -7,7 +7,7 @@ description: 无机材料计算工具(晶体结构 I/O、XRD 模拟、相图、
无机材料计算的核心库,服务建材院的水泥 / 混凝土 / 玻璃 / 陶瓷 / 耐火材料场景。离线计算在 sandbox 里用 pymatgen 官方 API;联网查 Materials Project 走 host-side tool。**本 skill 提供一个轻量 helper**:`CEMENT_PHASES` 常量(中文相名→化学式映射)。
> ⚠️ **配置条件**:只有宿主后端配置了 `MP_API_KEY` 时,`mp_search_summary` / `mp_get_structure` / `mp_get_entries` 才会出现在可用工具列表里。没有这些 tool 时,Materials Project 联网查询不可用;离线全部正常:用户给 `.cif` / `POSCAR``Structure.from_file()` / `SpacegroupAnalyzer` / `XRDCalculator`(对已有 Structure)/ `CEMENT_PHASES` 查表 / 格式转换 / `MPRelaxSet`。要 mp 拿结构 → 让用户从 https://materialsproject.org 下个 CIF 丢 task 目录;**不要脑补晶格 / 原子坐标**。
> ⚠️ **配置条件**:只有宿主后端配置了 `MP_API_KEY` 时,`materials_project_search` / `materials_project_get_structure` / `materials_project_get_entries` 才会出现在可用工具列表里。没有这些 tool 时,Materials Project 联网查询不可用;离线全部正常:用户给 `.cif` / `POSCAR``Structure.from_file()` / `SpacegroupAnalyzer` / `XRDCalculator`(对已有 Structure)/ `CEMENT_PHASES` 查表 / 格式转换 / `MPRelaxSet`。要 mp 拿结构 → 让用户从 https://materialsproject.org 下个 CIF 丢 task 目录;**不要脑补晶格 / 原子坐标**。
## 何时用
@ -53,9 +53,9 @@ from pymatgen.analysis.phase_diagram import PhaseDiagram, PDEntry
API key 只在宿主后端读取,不要在 `run_python` 里读 env。可用 tool:
- `mp_search_summary(formula?, formulas?, material_ids?, elements?, fields?, limit=10)` —— 查 summary,返回裁剪 JSON。**要查多个化学式时传 `formulas=[...]` 一次批量查完**(结果落 `materials/mp_search_batch_*.json`,只回摘要),不要一个式子调一次 —— 逐条调既慢又把中间数据全灌进对话
- `mp_get_structure(material_id, filename?)` —— 把结构保存到 task_dir `materials/*.cif`,再用 `Structure.from_file()` 离线计算
- `mp_get_entries(elements, filename?, limit=200)` —— 把 chemsys entries 保存到 task_dir `materials/*.json`
- `materials_project_search(formula?, formulas?, material_ids?, elements?, fields?, limit=10)` —— 查 summary,返回裁剪 JSON。**要查多个化学式时传 `formulas=[...]` 一次批量查完**(结果落 `materials/mp_search_batch_*.json`,只回摘要),不要一个式子调一次 —— 逐条调既慢又把中间数据全灌进对话
- `materials_project_get_structure(material_id, filename?)` —— 把结构保存到 task_dir `materials/*.cif`,再用 `Structure.from_file()` 离线计算
- `materials_project_get_entries(elements, filename?, limit=200)` —— 把 chemsys entries 保存到 task_dir `materials/*.json`
`MP_API_KEY` 没配 → 上述 tool 不会出现,告诉用户配置或手动从 Materials Project 下载 CIF。
@ -64,7 +64,7 @@ API key 只在宿主后端读取,不要在 `run_python` 里读 env。可用 tool
### A. 实测 XRD 比对(谁是这个峰)
1. 用户给疑似相清单(中文 / 英文 / 简写都行)
2. 各相分别:`CEMENT_PHASES` 查化学式 → `mp_search_summary(formula=...)` 找 material_id → `mp_get_structure(material_id=...)` 保存 CIF → `XRDCalculator().get_pattern(structure)` 算理论谱
2. 各相分别:`CEMENT_PHASES` 查化学式 → `materials_project_search(formula=...)` 找 material_id → `materials_project_get_structure(material_id=...)` 保存 CIF → `XRDCalculator().get_pattern(structure)` 算理论谱
3. 把各相理论谱跟实测谱(用户给的 xy 数据)叠图(走 `plot_pub`)
4. 报"x° 这个峰最可能是 C3S 的 (h k l) 衍射"
@ -72,7 +72,7 @@ API key 只在宿主后端读取,不要在 `run_python` 里读 env。可用 tool
from pymatgen.analysis.diffraction.xrd import XRDCalculator
xrd = XRDCalculator(wavelength="CuKa") # 默认 Cu Kα
# 先用 mp_search_summary / mp_get_structure tool 保存 CIF,再:
# 先用 materials_project_search / materials_project_get_structure tool 保存 CIF,再:
struct = Structure.from_file("materials/mp-xxxx.cif")
pattern = xrd.get_pattern(struct, two_theta_range=(5, 80))
# pattern.x = 2θ 列表, pattern.y = 强度, pattern.hkls = (h,k,l) 列表
@ -94,7 +94,7 @@ prim = sga.get_primitive_standard_structure() # 简约胞
### C. 凝胶 / 水化产物相图稳定性
先用 `mp_get_entries(elements=["Ca", "Si", "O", "H"])` 保存 JSON。第一版 host tool 的 entries JSON 主要用于审阅 / 归档;若要直接构造 `PhaseDiagram`,优先使用用户提供的本地 entry 数据或后续补一个专门的 host-side 相图 helper。不要在 sandbox 里绕过 tool 直接连 MP。
先用 `materials_project_get_entries(elements=["Ca", "Si", "O", "H"])` 保存 JSON。第一版 host tool 的 entries JSON 主要用于审阅 / 归档;若要直接构造 `PhaseDiagram`,优先使用用户提供的本地 entry 数据或后续补一个专门的 host-side 相图 helper。不要在 sandbox 里绕过 tool 直接连 MP。
### D. 格式转换(给计算所做 VASP 输入)
@ -130,4 +130,4 @@ mp-api>=0.41.0
MP_API_KEY=your_key_from_materialsproject_org
```
配置后重启 web,`mp_*` tools 才会注册。sandbox / `run_python` 不读取这个 key。
配置后重启 web,`materials_project_*` tools 才会注册。sandbox / `run_python` 不读取这个 key。

View File

@ -2,7 +2,7 @@
pymatgen skill helpers 建材院无机材料场景常用映射(中文相名 化学式)
LLM sandbox 中只应使用 `CEMENT_PHASES` / `lookup_phase` 和离线 pymatgen
Materials Project 联网查询走 host-side `mp_*` tools,不要在 sandbox 里读 MP_API_KEY
Materials Project 联网查询走 host-side `materials_project_*` tools,不要在 sandbox 里读 MP_API_KEY
"""
from __future__ import annotations

View File

@ -71,10 +71,10 @@ class TestRepeatGuard(unittest.TestCase):
g = _RepeatGuard()
unprods = []
for _ in range(_RepeatGuard.SOFT + 1):
unprods.append(g.record("document_search", {"queries": ["x"]}, "(no documents found)")[0])
unprods.append(g.record("materials_library_search", {"queries": ["x"]}, "(no documents found)")[0])
# 累计达到 SOFT(此时应注入软提示),但还没到 HARD 拦截
self.assertGreaterEqual(max(unprods), _RepeatGuard.SOFT)
self.assertFalse(g.should_block("document_search", {"queries": ["x"]}))
self.assertFalse(g.should_block("materials_library_search", {"queries": ["x"]}))
def test_record_returns_productive_signal(self):
"""record 第二个返回值喂全局无进展熔断:新非错结果=有产出,[Error]/重复=无产出。"""
@ -94,10 +94,10 @@ class TestRepeatGuard(unittest.TestCase):
def test_distinct_args_tracked_separately(self):
g = _RepeatGuard()
_simulate(g, "document_search", {"queries": ["a"]}, ["[Error] e"] * 8)
_simulate(g, "materials_library_search", {"queries": ["a"]}, ["[Error] e"] * 8)
# 不同参数互不影响
self.assertTrue(g.should_block("document_search", {"queries": ["a"]}))
self.assertFalse(g.should_block("document_search", {"queries": ["b"]}))
self.assertTrue(g.should_block("materials_library_search", {"queries": ["a"]}))
self.assertFalse(g.should_block("materials_library_search", {"queries": ["b"]}))
class TestErrStreak(unittest.TestCase):

View File

@ -7,7 +7,11 @@ from types import SimpleNamespace
from unittest.mock import patch
from uuid import uuid4
from tools.paper_server import PaperFetchTool, PaperSearchTool, _media_url
from platform_sources.paper_server import (
PaperServerFetchTool,
PaperServerSearchTool,
_media_url,
)
from tools.run_python import RunPythonTool
@ -22,16 +26,16 @@ class PaperServerToolTests(unittest.TestCase):
with (
patch(
"tools.paper_server._config",
"platform_sources.paper_server._config",
return_value=(
"https://paper.test",
"https://paper.test/api/resm/paper",
"secret",
),
),
patch("tools.paper_server._get_json", side_effect=fake_get_json),
patch("platform_sources.paper_server._get_json", side_effect=fake_get_json),
):
result = PaperSearchTool().execute(publication_type="book", limit=20)
result = PaperServerSearchTool().execute(publication_type="book", limit=20)
self.assertEqual(captured["type"], "book")
self.assertEqual(captured["page_size"], 20)
@ -41,22 +45,22 @@ class PaperServerToolTests(unittest.TestCase):
def test_search_accepts_open_raw_type_without_closed_enum(self):
with (
patch(
"tools.paper_server._config",
"platform_sources.paper_server._config",
return_value=(
"https://paper.test",
"https://paper.test/api/resm/paper",
"secret",
),
),
patch("tools.paper_server._get_json", return_value=[]) as request,
patch("platform_sources.paper_server._get_json", return_value=[]) as request,
):
result = PaperSearchTool().execute(publication_type="future-type")
result = PaperServerSearchTool().execute(publication_type="future-type")
self.assertEqual(request.call_args.kwargs["params"]["type"], "future-type")
self.assertEqual(result, "[]")
def test_search_rejects_unsafe_raw_type(self):
with patch("tools.paper_server._get_json") as request:
result = PaperSearchTool().execute(publication_type="book&api_key=leak")
with patch("platform_sources.paper_server._get_json") as request:
result = PaperServerSearchTool().execute(publication_type="book&api_key=leak")
request.assert_not_called()
self.assertTrue(result.startswith("[Error]"))
@ -70,7 +74,7 @@ class PaperServerToolTests(unittest.TestCase):
def test_fetch_schema_limits_format(self):
with tempfile.TemporaryDirectory() as tmp:
tool = PaperFetchTool(working_dir=Path(tmp))
tool = PaperServerFetchTool(working_dir=Path(tmp))
self.assertEqual(
tool.parameters["properties"]["format"]["enum"], ["pdf", "xml"]
)
@ -97,9 +101,9 @@ class PaperServerToolTests(unittest.TestCase):
}
with (
tempfile.TemporaryDirectory() as tmp,
patch("tools.paper_server._get_paper", return_value=paper),
patch("platform_sources.paper_server._get_paper", return_value=paper),
patch(
"tools.paper_server._config",
"platform_sources.paper_server._config",
return_value=(
"https://paper.test",
"https://paper.test/api/resm/paper",
@ -107,10 +111,10 @@ class PaperServerToolTests(unittest.TestCase):
),
),
patch(
"tools.paper_server.httpx.stream", return_value=FakeStreamResponse()
"platform_sources.paper_server.httpx.stream", return_value=FakeStreamResponse()
) as stream,
):
tool = PaperFetchTool(working_dir=Path(tmp))
tool = PaperServerFetchTool(working_dir=Path(tmp))
first = tool.execute(id_or_doi="p1", format="pdf")
second = tool.execute(id_or_doi="p1", format="pdf")
destination = Path(tmp) / "papers" / "10.1_example.pdf"
@ -195,8 +199,10 @@ class PaperServerToolTests(unittest.TestCase):
enter_common_patches(stack)
with_key = build_tools(ToolContext(**common))
self.assertNotIn("paper_search", without_key)
self.assertTrue({"paper_search", "paper_get", "paper_fetch"}.issubset(with_key))
self.assertNotIn("paper_server_search", without_key)
self.assertTrue(
{"paper_server_search", "paper_server_get", "paper_server_fetch"}.issubset(with_key)
)
if __name__ == "__main__":

View File

@ -0,0 +1,206 @@
from __future__ import annotations
import os
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from core.executor_docker import _sandbox_env
from platform_sources.registry import (
MaterialsLibraryProvider,
MaterialsProjectProvider,
PaperServerProvider,
PlatformSourceContext,
build_platform_source_tools,
)
from platform_sources.materials_library import MaterialsLibrarySearchTool
from platform_sources.materials_project import MaterialsProjectSearchTool
from platform_sources.paper_server import PaperServerSearchTool
from tools.run_python import RunPythonTool
class _Provider:
capabilities = frozenset({"test"})
def __init__(self, source_id, *, available=True, tools=None, failure=None):
self.source_id = source_id
self._available = available
self._tools = list(tools or [])
self._failure = failure
def available(self):
if self._failure == "available":
raise RuntimeError("secret-value")
return self._available
def build_tools(self, context):
if self._failure == "build":
raise RuntimeError("secret-value")
return self._tools
class _Tool:
name = "survivor"
class PlatformSourceRegistryTests(unittest.TestCase):
def _context(self, root: Path) -> PlatformSourceContext:
return PlatformSourceContext(root, root, root)
def test_provider_failure_isolated_and_diagnostic_is_secret_free(self):
with tempfile.TemporaryDirectory() as tmp, self.assertLogs(
"platform_sources.registry", level="WARNING"
) as logs:
built = build_platform_source_tools(
self._context(Path(tmp)),
providers=(
_Provider("bad_available", failure="available"),
_Provider("bad_build", failure="build"),
_Provider("good", tools=[_Tool()]),
),
)
self.assertEqual([tool.name for tool in built], ["survivor"])
output = "\n".join(logs.output)
self.assertIn("bad_available", output)
self.assertIn("bad_build", output)
self.assertIn("RuntimeError", output)
self.assertNotIn("secret-value", output)
def test_each_env_gate_is_independent(self):
providers = (
PaperServerProvider(),
MaterialsLibraryProvider(),
MaterialsProjectProvider(),
)
empty = {
"PAPER_SERVER_API_KEY": "",
"DOCUMENT_SEARCH_API_KEY": "",
"MP_API_KEY": "",
}
with patch.dict(os.environ, empty, clear=False), patch(
"platform_sources.registry.MPRester", object()
):
self.assertEqual([provider.available() for provider in providers], [False] * 3)
for env_name, index in (
("PAPER_SERVER_API_KEY", 0),
("DOCUMENT_SEARCH_API_KEY", 1),
("MP_API_KEY", 2),
):
values = dict(empty)
values[env_name] = "configured"
with patch.dict(os.environ, values, clear=False):
available = [provider.available() for provider in providers]
self.assertTrue(available[index])
self.assertEqual(sum(available), 1)
def test_invalid_source_config_does_not_block_other_source(self):
with tempfile.TemporaryDirectory() as tmp, patch.dict(
os.environ,
{
"PAPER_SERVER_API_KEY": "paper-secret",
"PAPER_SERVER_URL": "not-a-url",
"DOCUMENT_SEARCH_API_KEY": "library-secret",
"MP_API_KEY": "",
},
clear=False,
), self.assertLogs("platform_sources.registry", level="WARNING") as logs:
tools = build_platform_source_tools(self._context(Path(tmp)))
names = {tool.name for tool in tools}
self.assertEqual(
names,
{
"materials_library_list",
"materials_library_search",
"materials_library_fetch",
},
)
output = "\n".join(logs.output)
self.assertIn("paper_server", output)
self.assertIn("ValueError", output)
self.assertNotIn("paper-secret", output)
def test_registered_tool_names_are_source_specific(self):
with tempfile.TemporaryDirectory() as tmp, patch.dict(
os.environ,
{
"PAPER_SERVER_API_KEY": "paper-secret",
"DOCUMENT_SEARCH_API_KEY": "library-secret",
"MP_API_KEY": "mp-secret",
},
clear=False,
), patch("platform_sources.registry.MPRester", object()):
tools = build_platform_source_tools(self._context(Path(tmp)))
names = {tool.schema["function"]["name"] for tool in tools}
self.assertEqual(
names,
{
"paper_server_search",
"paper_server_get",
"paper_server_fetch",
"materials_library_list",
"materials_library_search",
"materials_library_fetch",
"materials_project_search",
"materials_project_get_structure",
"materials_project_get_entries",
},
)
self.assertFalse(
names
& {
"paper_search",
"paper_get",
"paper_fetch",
"document_list_kb",
"document_search",
"document_download",
"mp_search_summary",
"mp_get_structure",
"mp_get_entries",
"platform_source_call",
}
)
def test_platform_credentials_do_not_enter_execution_environments(self):
secrets = {
"PAPER_SERVER_API_KEY": "paper-secret",
"DOCUMENT_SEARCH_API_KEY": "library-secret",
"MP_API_KEY": "mp-secret",
}
with patch.dict(os.environ, secrets, clear=False):
host_env = RunPythonTool()._filtered_env()
docker_env = _sandbox_env()
for name in secrets:
self.assertNotIn(name, host_env)
self.assertNotIn(name, docker_env)
def test_platform_credentials_are_redacted_from_tool_results(self):
secrets = {
"PAPER_SERVER_API_KEY": "paper-secret",
"DOCUMENT_SEARCH_API_KEY": "library-secret",
"MP_API_KEY": "mp-secret",
}
with patch.dict(os.environ, secrets, clear=False), patch(
"platform_sources.paper_server._config",
side_effect=RuntimeError("paper-secret"),
), patch(
"platform_sources.materials_library.client.search",
side_effect=RuntimeError("api_key=library-secret"),
), patch(
"platform_sources.materials_project._mpr",
side_effect=RuntimeError("mp-secret"),
):
results = (
PaperServerSearchTool().execute(keyword="cement"),
MaterialsLibrarySearchTool().execute(queries=["cement"]),
MaterialsProjectSearchTool().execute(formula="Ca3SiO5"),
)
joined = "\n".join(results)
for value in secrets.values():
self.assertNotIn(value, joined)
self.assertIn("[REDACTED]", joined)
if __name__ == "__main__":
unittest.main()

View File

@ -10,9 +10,9 @@ from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
class TestDocumentHostTools(unittest.TestCase):
def test_document_search_truncates_content_without_requiring_key_arg(self):
from tools.documents import DocumentSearchTool
class TestMaterialsLibraryHostTools(unittest.TestCase):
def test_materials_library_search_truncates_content_without_key_arg(self):
from platform_sources.materials_library import MaterialsLibrarySearchTool
docs = [
{
@ -22,9 +22,9 @@ class TestDocumentHostTools(unittest.TestCase):
"md_content": "A" * 200,
}
]
with patch("tools.documents.doc_client.search", return_value=docs) as search:
with patch("platform_sources.materials_library.client.search", return_value=docs) as search:
# 单 query 批量:queries 列表只一条时,缩量逻辑不动用户给的参数
out = DocumentSearchTool().execute(
out = MaterialsLibrarySearchTool().execute(
queries=["cement hydration"],
max_documents=3,
content_chars_per_doc=20,
@ -41,8 +41,8 @@ class TestDocumentHostTools(unittest.TestCase):
self.assertIn("A" * 20, out)
self.assertIn("truncated", out)
def test_document_search_batches_queries_concurrently_and_dedups(self):
from tools.documents import DocumentSearchTool
def test_materials_library_search_batches_and_dedups(self):
from platform_sources.materials_library import MaterialsLibrarySearchTool
calls: list[str] = []
@ -50,8 +50,8 @@ class TestDocumentHostTools(unittest.TestCase):
calls.append(query)
return [{"file_name": f"{query}.md", "kb_name": "mu_1", "md_content": "x"}]
with patch("tools.documents.doc_client.search", side_effect=fake_search):
out = DocumentSearchTool().execute(
with patch("platform_sources.materials_library.client.search", side_effect=fake_search):
out = MaterialsLibrarySearchTool().execute(
queries=["q1", "q2", "q1"], # 含重复 → 去重成 q1/q2
)
@ -61,17 +61,17 @@ class TestDocumentHostTools(unittest.TestCase):
self.assertIn("'q1'", out)
self.assertIn("'q2'", out)
def test_document_download_uses_constructor_working_dir(self):
from tools.documents import DocumentDownloadTool
def test_materials_library_fetch_uses_constructor_working_dir(self):
from platform_sources.materials_library import MaterialsLibraryFetchTool
with tempfile.TemporaryDirectory() as tmp:
working_dir = Path(tmp) / "task"
working_dir.mkdir()
with patch(
"tools.documents.doc_client.download",
"platform_sources.materials_library.client.download",
return_value="documents/paper.pdf",
) as download:
tool = DocumentDownloadTool(
tool = MaterialsLibraryFetchTool(
working_dir=working_dir,
base_dir=working_dir,
user_root=Path(tmp),
@ -86,8 +86,8 @@ class TestDocumentHostTools(unittest.TestCase):
)
self.assertIn("saved: task/documents/paper.pdf", out)
def test_document_download_batches_items_isolating_failure(self):
from tools.documents import DocumentDownloadTool
def test_materials_library_fetch_batches_items_isolating_failure(self):
from platform_sources.materials_library import MaterialsLibraryFetchTool
with tempfile.TemporaryDirectory() as tmp:
working_dir = Path(tmp) / "task"
@ -98,8 +98,8 @@ class TestDocumentHostTools(unittest.TestCase):
raise RuntimeError("404")
return f"documents/{file_name}"
with patch("tools.documents.doc_client.download", side_effect=fake_download):
tool = DocumentDownloadTool(
with patch("platform_sources.materials_library.client.download", side_effect=fake_download):
tool = MaterialsLibraryFetchTool(
working_dir=working_dir, base_dir=working_dir, user_root=Path(tmp)
)
out = tool.execute(items=[
@ -114,8 +114,8 @@ class TestDocumentHostTools(unittest.TestCase):
class TestMaterialsProjectHostTools(unittest.TestCase):
def test_mp_search_summary_uses_host_key_and_returns_json(self):
from tools.materials_project import MaterialsProjectSearchSummaryTool
def test_materials_project_search_uses_host_key_and_returns_json(self):
from platform_sources.materials_project import MaterialsProjectSearchTool
class FakeDoc:
material_id = "mp-1"
@ -145,10 +145,10 @@ class TestMaterialsProjectHostTools(unittest.TestCase):
return False
with patch.dict("os.environ", {"MP_API_KEY": "host-secret"}, clear=False), patch(
"tools.materials_project.MPRester",
"platform_sources.materials_project.MPRester",
FakeMPRester,
):
out = MaterialsProjectSearchSummaryTool().execute(
out = MaterialsProjectSearchTool().execute(
formula="Ca3SiO5",
fields=["material_id", "formula_pretty", "energy_above_hull"],
limit=2,
@ -164,8 +164,8 @@ class TestMaterialsProjectHostTools(unittest.TestCase):
self.assertEqual(captured["num_chunks"], 1)
self.assertEqual(captured["chunk_size"], 2)
def test_mp_search_summary_batch_writes_file_and_isolates_failure(self):
from tools.materials_project import MaterialsProjectSearchSummaryTool
def test_materials_project_search_batch_writes_file_and_isolates_failure(self):
from platform_sources.materials_project import MaterialsProjectSearchTool
class FakeDoc:
def __init__(self, mid, formula, hull):
@ -204,9 +204,9 @@ class TestMaterialsProjectHostTools(unittest.TestCase):
wd = Path(td) / "proj"
wd.mkdir()
with patch.dict("os.environ", {"MP_API_KEY": "host-secret"}, clear=False), patch(
"tools.materials_project.MPRester", FakeMPRester
"platform_sources.materials_project.MPRester", FakeMPRester
):
tool = MaterialsProjectSearchSummaryTool(working_dir=wd)
tool = MaterialsProjectSearchTool(working_dir=wd)
out = tool.execute(
formulas=["Ca3SiO5", "Ca3SiO5", "Empty1", "BadX"],
fields=["material_id", "formula_pretty", "energy_above_hull"],