refactor(platform): isolate managed source providers
This commit is contained in:
parent
604f4da373
commit
631a635eab
|
|
@ -8,7 +8,7 @@
|
|||
|
||||
## Unreleased
|
||||
|
||||
- 文献检索现在可以在服务端直接限定书籍、章节、论文等出版物类型,书籍类查询更准确;平台文献源的访问凭据也不再进入任务沙箱。
|
||||
- 平台托管的文献、内部材料库和 Materials Project 来源现在使用来源明确的工具,并可在单个来源不可用时继续使用其他来源;访问凭据不进入任务沙箱。
|
||||
|
||||
## 0.68.0 — 2026-08-24
|
||||
|
||||
|
|
|
|||
17
DESIGN.md
17
DESIGN.md
|
|
@ -39,7 +39,8 @@ zcbot/
|
|||
│ ├── wechat/ # 渠道:ilink / wecom / service / inbound(§8.7)
|
||||
│ ├── sandbox/ + executor*.py # Executor ABC + Docker per-user 容器池(§7.5)
|
||||
│ └── agent_builder.py # 装配 lib:build_agent / system prompt
|
||||
├── tools/ # fs / shell / run_python / skill / 媒体(共享原语 media_common)/ 检索 / host-side 域工具
|
||||
├── platform_sources/ # 平台托管共享只读来源、显式 Provider registry 与 typed tools(§3.4)
|
||||
├── tools/ # fs / shell / run_python / skill / 媒体(共享原语 media_common)
|
||||
├── skills/<name>/ # SKILL.md + references / scripts / assets
|
||||
├── rendering/ # 平台渲染层 md→docx/pdf(§8.6;块收集器/inline 切分单一事实源在 common)
|
||||
├── prompts/system/general_v1.md
|
||||
|
|
@ -87,10 +88,16 @@ yaml 是手填的,probe 用真实调用对账(basic_chat/parallel_tools/thinking
|
|||
### 3.4 工具系统(Hybrid 范式)
|
||||
**JSON tool call** 管离散操作;**run_python**(tmp .py + subprocess + 敏感 env 过滤)管批处理/生成文档。`edit` **唯一匹配**(old_str 重复即报错);工具按**原子操作**切分,不做 `make_pptx()` 式高级封装。持 key 的能力一律 host-side tool、仅对应 env 存在才注册(§7.5 #7)。
|
||||
|
||||
**平台托管来源 bounded context**(2026-08-26):管理员随部署配置、平台统一凭据和共享只读数据源统一归顶层 `platform_sources/`;选择顶层包而非 `core/platform_sources + tools/platform_sources`,是为了让 client/runtime、来源专用 typed Tool adapter、显式 Provider registry 和生命周期保持在同一领域边界,同时让 `core/tool_registry.py` 只依赖唯一 `build_platform_source_tools` 入口。依赖方向固定为 `core/tool_registry → platform_sources → tools.base`,`platform_sources` 与 `core/external_systems` 互不依赖;后者仍只负责用户身份连接、用户凭据和动态 OpenAPI/MCP。
|
||||
|
||||
Provider 的最小契约只有 `source_id`、`capabilities`、`available()`、`build_tools(context)`,统一注册和生命周期,不假设每个来源都有相同 search/get/fetch 业务接口。可信 Provider 使用代码内显式列表,逐来源隔离配置和装配失败,诊断只记录来源标识与异常类型;模型面继续暴露 `paper_server_*`、`materials_library_*`、`materials_project_*` 强类型工具,不提供万能弱类型 RPC。平台凭据只在宿主 control plane 读取,不进 prompt、Tool schema/result、日志、`run_python` 或 Docker sandbox;paper_server 保留同源下载、100 MiB 上限、认证错误脱敏和 `.part` 原子落盘。只抽取已稳定重复的安全 HTTP/下载原语,Materials Project 的 SDK、化学式、`material_id` 与 CIF/entries 语义留在专用 adapter。
|
||||
|
||||
当前定义和凭据仍以部署环境变量为事实源,修改后重启生效,不建数据库。若未来出现用户级凭据或 per-user grant,归 `external_systems`;若需要平台来源在线动态 definition、revision/reverify 或 OAuth,则另行增加 `platform_sources` 控制面并重新评估持久化,不能把动态连接状态塞进当前静态 Provider registry。
|
||||
|
||||
### 3.5 Skill 系统(Anthropic 渐进披露)
|
||||
三层加载:Discovery(name+description,几百 token)→ Activation(`load_skill` 完整 SKILL.md)→ Execution(references 按需拉)。写 WHY+WHAT 不写 Step 1/2/3;description 决定触发。
|
||||
|
||||
**出版物检索统一入口**(2026-08-25):`literature` 按用户任务而非数据源划分 skill,统一论文、书籍、章节、会议论文、学位论文、报告、标准、专利、预印本及未来出版物的发现、获取、跨源去重与证据核验;`paper_server`、内部材料知识库及未来来源只是可替换后端。出版物类型与来源正交,同一出版物可有多条来源记录;跨源机械层只自动合并 DOI、ISBN+版次、标准号、公开号等精确键或题名+主要责任者+年份完全一致项,疑似重复、不同版次、章节、标准修订版和专利族成员保守保留。题录、摘要、片段、全文四级证据分开标注,下载成功不等于全文已经核验。取舍是不为每个数据库新增 skill,也不把检索编排固化成统一高级工具:skill 负责路由与判断,来源能力保持各自工具边界,确定性标准化与精确去重下沉到可重建脚本。`paper_server` 自 2026-08-26 起使用按配置注册的 host-side `paper_search` / `paper_get` / `paper_fetch`,API Key 不再进入 sandbox;来源原始 `type` 可服务端过滤,统一类型映射仍归出版物模型与合并脚本。
|
||||
**出版物检索统一入口**(2026-08-25):`literature` 按用户任务而非数据源划分 skill,统一论文、书籍、章节、会议论文、学位论文、报告、标准、专利、预印本及未来出版物的发现、获取、跨源去重与证据核验;`paper_server`、内部材料知识库及未来来源只是可替换后端。出版物类型与来源正交,同一出版物可有多条来源记录;跨源机械层只自动合并 DOI、ISBN+版次、标准号、公开号等精确键或题名+主要责任者+年份完全一致项,疑似重复、不同版次、章节、标准修订版和专利族成员保守保留。题录、摘要、片段、全文四级证据分开标注,下载成功不等于全文已经核验。取舍是不为每个数据库新增 skill,也不把检索编排固化成统一高级工具:skill 负责路由与判断,来源能力保持各自工具边界,确定性标准化与精确去重下沉到可重建脚本。`paper_server` 自 2026-08-26 起使用按配置注册的 host-side `paper_server_search` / `paper_server_get` / `paper_server_fetch`,API Key 不再进入 sandbox;来源原始 `type` 可服务端过滤,统一类型映射仍归出版物模型与合并脚本。
|
||||
|
||||
**用户私有 skill**(2026-06-11):registry 收有序来源列表——内置 `ROOT/skills`(只读)+ 用户 `user_root/.skills`(可写)。取舍:① **user wins 同名覆盖**(核心用例是"copy 内置再改",覆盖只作用于本人会话,blast radius 锁死),覆盖显式标注不静默;② **创作走 host-side `save_skill`/`fork_skill`**——fs 工具的 base_dir 跨 backend 够不到 `user_root/.skills`,host 工具一个落点两模式通吃;③ 用户 skill 加载失败收进 `load_errors` 注入 prompt 提示修,不崩整次扫描。
|
||||
|
||||
|
|
@ -109,7 +116,7 @@ Session = 消息列表,ORM 直写 PG `messages`(append-only,jsonb 存 LiteLLM
|
|||
|
||||
### 3.8 个人知识库(`core/kb.py` + `core/kb_ingest.py`,✅ 2026-07-22)
|
||||
|
||||
用户自建资料(规范/报告/标准/内部文档)的长期查阅层,与 §3.7 记忆同范式:**纯文件 + prompt 注入契约,无向量无 DB**(判据同"真实文件为准":个人库几十~百余文件,agentic search 足够;索引若引入只能是可重建派生缓存)。**两层格局**:zcbot 内建 `.kb/` 私有小库(本节)+ 院检索服务共享大库(document_search,zcbot 只当客户端)——分工标准 = 文件数 × 查询频次,路由靠各自工具/契约描述自然分流,不在 kb 契约里点名 document_search(2026-07-22 收窄:契约只管自己怎么用,少一层耦合)。
|
||||
用户自建资料(规范/报告/标准/内部文档)的长期查阅层,与 §3.7 记忆同范式:**纯文件 + prompt 注入契约,无向量无 DB**(判据同"真实文件为准":个人库几十~百余文件,agentic search 足够;索引若引入只能是可重建派生缓存)。**两层格局**:zcbot 内建 `.kb/` 私有小库(本节)+ 院检索服务共享大库(`materials_library_search`,zcbot 只当客户端)——分工标准 = 文件数 × 查询频次,路由靠各自工具/契约描述自然分流,不在 kb 契约里点名平台材料库(2026-07-22 收窄:契约只管自己怎么用,少一层耦合)。
|
||||
|
||||
- **做成机制而非 skill**(判据:有独立于会话的持久状态需用户管理 → 机制):落盘 `user_root/.kb/<库名>/`(INDEX.md + docs/ 转换后 md + sources/ 原件)。**"已入库"判据 = INDEX.md 有条目**,sources 有而 INDEX 无 = 待入库 → 入库幂等、崩溃可恢复、零 migration。dotfile 命名同 `.memory` 双向防呆,GET /v1/files 天然隐藏。
|
||||
- **入库管线**(`core/kb_ingest.py`,上传即触发 + 手动兜底):markitdown Python API 转 md → 扫描件 PDF(文本近零)走方舟文档理解 OCR 兜底(§8.13 同通道)→ deepseek flash 单次 chat 写 标题/摘要/关键词(失败降级文件名+正文开头,不阻塞)→ 追加 INDEX 行。编排照定时执行器:create_task + to_thread;**写并发收口为共享 FS advisory lock**(`.kb/.locks/<hash>.lock`):Web 上传/删除、后台入库与 agent `write/edit` 对同一库共用一把跨进程锁,蓝绿实例间只允许一个写者,锁占用返 409/工具可重试;进程退出由 OS 自动释锁,不靠清理锁文件。文档正文、原件与 INDEX 全走同目录临时文件 + fsync + `os.replace` 原子发布,读者只会看到完整旧版或完整新版。进度详情仍以内存保存细节,但会探测跨进程锁补出 `running`;崩了靠 FS 判据续跑。
|
||||
|
|
@ -378,13 +385,13 @@ scheduled_jobs(§8.5) channel_bindings(§8.7,判别列+JSONB)
|
|||
|
||||
### 8.11 最小子循环 delegate:上下文隔离而非多 agent 编排(design,2026-07-08,按诊断数据触发)
|
||||
|
||||
**根因**:检索/扫文类工作(文献 brief、document_search、批量读文件)的形态是"中间数据量大、最终只要结论"——在主循环里跑,中间数据必然流经主上下文,污染 + 膨胀是**架构性的**。已踩实例:38 篇 abstract 反复 dump 烧 2.5M token、`document_search` 同参调 122 次不收敛。现有缓解(`_RepeatGuard` 熔断、§8.2 context 压缩、brief skill 的 context 纪律)全是**行为约束**——劝模型别乱来,不改变"中间数据必须过主上下文"这个结构;同 8.9 的教训,提示层纪律挡不住结构性问题。
|
||||
**根因**:检索/扫文类工作(文献 brief、`materials_library_search`、批量读文件)的形态是"中间数据量大、最终只要结论"——在主循环里跑,中间数据必然流经主上下文,污染 + 膨胀是**架构性的**。已踩实例:38 篇 abstract 反复 dump 烧 2.5M token、材料库检索同参调 122 次不收敛。现有缓解(`_RepeatGuard` 熔断、§8.2 context 压缩、brief skill 的 context 纪律)全是**行为约束**——劝模型别乱来,不改变"中间数据必须过主上下文"这个结构;同 8.9 的教训,提示层纪律挡不住结构性问题。
|
||||
|
||||
**决策**:§6"不做 subagent"针对的是**编排型多 agent**(并行、状态共享、agent 间通信、任务分解),该结论不变。本条预留的是**一个工具**:`delegate(instruction) -> str 摘要`——复用现成 `AgentLoop` 起一个全新空上下文的子循环,只注只读工具(检索/读文件类白名单),硬轮数上限(~20),跑完只把文字摘要返回主循环。主循环视角就是一次普通 tool call,零状态共享、零并行、零 agent 间协议;实现量约一个文件。
|
||||
|
||||
**触发条件(无信号不实施)**:RepeatGuard + brief 纪律上线后,`scripts/diag_*.py` 数据仍显示检索型 task 是烧 token 大户 / 检索中间数据占上下文大头。数据收敛则本条永久搁置。
|
||||
|
||||
**⚠️ 实现后撤回(0.58.30 落地 → 0.58.31 revert,记录教训)**:2026-07-15 曾完整实现(FilteredExecutor + 内存态子 Session + loop 拦截 + 降 flash),又整体撤回。撤回原因不是机制错(设计是对的、对标 Claude Code subagent 也成立),而是**触发信号没坐实**:自评时回看 `diag_search_args.py` 数据,motivating 案子 `document_search` 122 次呈"一批批不同材料体系并行搜"形态,**更像批量扇出而非结果驱动的探索**——若属实,它的正解和 mp_search 一样是**批量工具 `document_search_batch`**(便宜、可预测、可诊断),delegate 对它是过度设计。加上子循环 transcript 不落盘(诊断驱动的功能反而不可诊断)、20 轮上限对 122 次负载可能偏低、强制 flash 对难检索可能降质——一堆未验证的坑。**重建的前置条件收紧为**:先用 diag 确认某检索型 task 的 query 序列是**真探索**(query 依赖前序结果、无法一次性列全),而非可批量枚举;是批量就走批量工具,只有真探索才值得 delegate。**教训**:§5"无信号不实施"要落到"信号已被数据证伪 batchable 的可能性"这一步,不能凭"看起来像探索"就上 agentic 子循环——同 mp_search 的判断(批量扇出≠检索发散)。批量工具那条(0.58.28)验证过、保留。
|
||||
**⚠️ 实现后撤回(0.58.30 落地 → 0.58.31 revert,记录教训)**:2026-07-15 曾完整实现(FilteredExecutor + 内存态子 Session + loop 拦截 + 降 flash),又整体撤回。撤回原因不是机制错(设计是对的、对标 Claude Code subagent 也成立),而是**触发信号没坐实**:自评时回看 `diag_search_args.py` 数据,motivating 案子材料库检索 122 次呈"一批批不同材料体系并行搜"形态,**更像批量扇出而非结果驱动的探索**——若属实,它的正解是 `materials_library_search(queries=[...])` 这类批量工具(便宜、可预测、可诊断),delegate 对它是过度设计。加上子循环 transcript 不落盘(诊断驱动的功能反而不可诊断)、20 轮上限对 122 次负载可能偏低、强制 flash 对难检索可能降质——一堆未验证的坑。**重建的前置条件收紧为**:先用 diag 确认某检索型 task 的 query 序列是**真探索**(query 依赖前序结果、无法一次性列全),而非可批量枚举;是批量就走批量工具,只有真探索才值得 delegate。**教训**:§5"无信号不实施"要落到"信号已被数据证伪 batchable 的可能性"这一步,不能凭"看起来像探索"就上 agentic 子循环——同 Materials Project 批量查询的判断(批量扇出≠检索发散)。批量工具那条(0.58.28)验证过、保留。
|
||||
|
||||
**不选**:
|
||||
- 完整多 agent 编排:状态管理爆炸(§6),且 zcbot 无真实并行需求——用户没有"同时审 3 篇"诉求,职责隔离已由 skill 体系覆盖。
|
||||
|
|
|
|||
10
PROGRESS.md
10
PROGRESS.md
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 1-2 句:做了啥 + 关键判断;细节查 `git log` / `git diff` / `DESIGN §7.9`。
|
||||
|
||||
最后更新:2026-08-26(paper_server 类型过滤与 host-side 凭据隔离,未发版)
|
||||
最后更新:2026-08-26(platform_sources 独立平台托管来源,未发版)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -20,9 +20,11 @@
|
|||
---
|
||||
## 已完成关键能力
|
||||
|
||||
- **08-26 / Unreleased / paper_server 宿主侧受控访问**:新增按配置注册的 `paper_search` / `paper_get` / `paper_fetch` typed tools,`publication_type` 直接映射后端 `type` 过滤且保留开放原始类型;API Key 从 `run_python` 和 Docker sandbox 透传中移除,下载增加同源 URL、100 MiB 上限和原子落盘约束。literature / brief 指引、smoke、RUN、Skill 清单及回归测试同步迁移,线上只读 `type=book` 验证通过,不提升版本号。
|
||||
- **08-26 / Unreleased / platform_sources bounded context**:将 paper_server、内部材料库和 Materials Project 收敛为独立顶层包,以显式可信 Provider 列表统一可用性与生命周期,单来源配置/装配失败不阻断其他来源;`core/tool_registry.py` 只保留一个构建入口,与用户连接 `external_systems` 完全独立。模型工具统一改为来源明确的 `paper_server_*`、`materials_library_*`、`materials_project_*`,保留材料库批量检索、MP 专用 SDK/CIF/entries 语义及 paper_server 安全下载边界;部署环境变量仍是事实源,无数据库迁移、无版本提升。
|
||||
|
||||
- **08-25 / Unreleased / literature 统一出版物检索入口**:删除按数据源拆分的 `research` / `documents` skill,新增 `literature` 统一论文、书籍、章节及未来学术与技术出版物的发现、获取、跨源去重和证据核验;paper_server helper 归入新 skill,内部材料库客户端移到 host-side `tools/` 保持密钥隔离。新增开放出版物模型、来源与证据 references,以及只自动合并精确键、保守标记疑似重复的 `merge_publications.py`;`brief/paper/review/rebuttal/patent/standard/analyze` 和 Skill 清单统一改走新入口,不提升版本号。
|
||||
- **08-26 / Unreleased / paper_server 宿主侧受控访问**:新增按配置注册的 `paper_server_search` / `paper_server_get` / `paper_server_fetch` typed tools,`publication_type` 直接映射后端 `type` 过滤且保留开放原始类型;API Key 从 `run_python` 和 Docker sandbox 透传中移除,下载增加同源 URL、100 MiB 上限和原子落盘约束。literature / brief 指引、smoke、RUN、Skill 清单及回归测试同步迁移,线上只读 `type=book` 验证通过,不提升版本号。
|
||||
|
||||
- **08-25 / Unreleased / literature 统一出版物检索入口**:删除按数据源拆分的 `research` / `documents` skill,新增 `literature` 统一论文、书籍、章节及未来学术与技术出版物的发现、获取、跨源去重和证据核验;paper_server helper 归入新 skill,内部材料库通过宿主侧平台来源保持密钥隔离。新增开放出版物模型、来源与证据 references,以及只自动合并精确键、保守标记疑似重复的 `merge_publications.py`;`brief/paper/review/rebuttal/patent/standard/analyze` 和 Skill 清单统一改走新入口,不提升版本号。
|
||||
|
||||
- **08-24 / 0.68.0 / Origin 小提琴分布图**:`origin.plot@v2` / adapter 1.3.0 新增 `violin`,每条 `input/y` 系列作为一组原始观测,Worker 复制到受控全 Y 展示表并通过 Origin 2024 原生 `Violin.otpu` 一次性生成分类分布图;默认隐藏冗余图例,保留分类标签并为 X 轴标题预留底部空间。首版不开放逐系列 style、分裂/半小提琴及带箱线变体,避免把未稳定映射的模板属性写入公共契约。固定 QA 新增三组抗压强度分布、中位数 oracle、OPJU 重开与全格式复导;Origin 2024 / originpro 1.1.15 数据、视觉与进程释放门禁通过,Origin/合同/Job/Node 专项 126 项 unittest、Ruff 致命规则、py_compile、diff 检查与独立 adapter 打包通过,未连接或写入数据库。
|
||||
- **08-24 / 0.68.0 / Origin 材料谱图错位叠加**:`origin.plot@v2` / adapter 1.2.0 新增 `stacked_line`,面向 XRD、XPS 和光谱多曲线比较;请求显式声明间隔百分比,Worker 保留原始工作表并生成基线对齐的受控展示表,按全体最大谱幅确定逐条偏移,允许不同 X 采样点且不依赖节点模板。默认使用曲线右端直标并按画布定位标题,显式图例请求仍兼容。固定 QA harness 同步修正 OriginPro 列式读取、空白工作簿及短/长名称和有效列标签识别;Origin 2024 / originpro 1.1.15 真机生成、OPJU 重开、PNG/SVG/PDF 复导、偏移基线 oracle、视觉检查与进程释放通过。Origin/合同/Job/Node 专项 124 项 unittest、ruff 致命规则、py_compile 与独立 adapter 打包校验通过,未连接或写入数据库。
|
||||
|
|
@ -253,7 +255,7 @@
|
|||
- **07-15 / 0.58.32**:空响应防御——provider 吐空 tool_calls + 空正文原被当"答完"静默 done(隐蔽卡死);`_is_empty_response` 接 attempt 循环走非流式重试,耗尽仍空发**可见 warn**「已重试仍空,回复继续可重试」+ toolfail 聚集留痕,不引入终态 error。刻意不做内容嗅探式重试(假阳性更糟)。
|
||||
- **07-15 / 0.58.31**:撤回 §8.11 delegate(0.58.30 曾整体落地,revert + force-push 抹除)——机制没错但触发信号没坐实(motivating 案更像批量扇出而非探索检索);重建前先用 diag 确认是真探索(依赖前序结果、无法一次列全)才 delegate,是批量走批量工具。只留验证过的批量工具(0.58.28)。
|
||||
- **07-15 / 0.58.29**:修对话正文文件锚点两回归——① `media.js` `_TAIL_CLS` 误排间隔号 `·`(文件名合法分隔符)致含 `·` 的 pptx 丢 chip;② 正文相对路径链接点击整页 404,`chat.js` 加 `.msg .body a[href]` 拦截(外链新标签 / 内部 openFilePreview)。
|
||||
- **07-15 / 0.58.28**:`mp_search_summary` 加批量入参 `formulas=[...]`(检索型烧 token 头号定点解)——pymatgen task 562 次逐式调用(91% 重复),批量化并成 1 次工具调用、复用单 MPRester 会话、中间数据落盘不进对话、只回紧凑摘要 + 路径;单查询路径字节不变向后兼容。"已知清单扇出"正解是工具批量化,非 delegate 子循环。
|
||||
- **07-15 / 0.58.28**:`materials_project_search` 的批量入参 `formulas=[...]`(检索型烧 token 头号定点解)——pymatgen task 562 次逐式调用(91% 重复),批量化并成 1 次工具调用、复用单 MPRester 会话、中间数据落盘不进对话、只回紧凑摘要 + 路径。"已知清单扇出"正解是工具批量化,非 delegate 子循环。
|
||||
- **07-15 / 0.58.27**:对话卡片悬停「复制」按钮(仿 GPT/DeepSeek)——取正文 Markdown 原文(据 `data-idx` 回 `loadedMessages`,不从渲染 HTML 反推),clipboard 主路径 + execCommand 降级;范围收敛只做复制(后端纯追加无重跑/截断语义,重生成/编辑留待后端支持)。
|
||||
- **07-15 / 0.58.26**:用户停止后「已停止」持久提示(补 cancel 终态,与 error 对称)——cancel 提升为一等持久终态 `run_status="cancelled"`(Text 列无 migration),前端 `renderPersistedRunTerminal` 认 error/cancelled、刷新/切任务仍在;删被收尾 loadMessages 冲掉的 live badge。蓝绿旧实例读到当良性态优雅退化。
|
||||
- **07-15 / 0.58.25**:skill 定向模型豁免——current profile family==`unifyllm`(已选国际旗舰)时不切走 pin 的国产档,尊重用户显式选择;判据放 DB 写入前无副作用。DESIGN §3.5 记。
|
||||
|
|
|
|||
19
RUN.md
19
RUN.md
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> 怎么把 zcbot 跑起来。env / 常用命令 / 故障兜底。设计看 `DESIGN.md`,进度看 `PROGRESS.md`。
|
||||
|
||||
最后更新:2026-08-10(外部系统统一为通用 OpenAPI/MCP definition)
|
||||
最后更新:2026-08-26(platform_sources 独立平台托管来源)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -20,19 +20,21 @@
|
|||
# 3) 视频生成 seedance tool(Seedance 2.0 Fast,文生/单图首帧/最多9张多图参考,480p 4s ¥1.86 ~ 720p 15s ¥12+,异步等 30-90s)
|
||||
# 未设:豆包文本模型选不了,seedream/seedance 两个 tool 都不出现
|
||||
ARK_API_KEY=...
|
||||
# documents skill(内部知识库 document_search API):可选。设了后注册
|
||||
# document_list_kb / document_search / document_download 三个 host-side tool;
|
||||
# 内部材料知识库:可选。设了后注册 materials_library_list /
|
||||
# materials_library_search / materials_library_fetch 三个 host-side tool;
|
||||
# key 只留宿主后端,sandbox/run_python 不读取。
|
||||
DOCUMENT_SEARCH_API_KEY=...
|
||||
# 可选:覆盖默认 base_url(默认 https://ai.ctc-zc.com:8100/api)
|
||||
# DOCUMENT_SEARCH_URL=https://ai.ctc-zc.com:8100/api
|
||||
# pymatgen skill 的 Materials Project 接入:可选。设了后注册
|
||||
# mp_search_summary / mp_get_structure / mp_get_entries 三个 host-side tool;
|
||||
# materials_project_search / materials_project_get_structure /
|
||||
# materials_project_get_entries 三个 host-side tool;
|
||||
# 离线分析(CIF / POSCAR + SpacegroupAnalyzer + XRDCalculator + CEMENT_PHASES)仍在 sandbox 跑。
|
||||
# 申请 https://materialsproject.org/api(免费)
|
||||
MP_API_KEY=...
|
||||
# literature skill 的 paper_server 文献库:GET 接口要 API Key(paper_server Django admin
|
||||
# 里建 ApiKey)。设了后注册 paper_search / paper_get / paper_fetch 三个 host-side tool;
|
||||
# 里建 ApiKey)。设了后注册 paper_server_search / paper_server_get /
|
||||
# paper_server_fetch 三个 host-side tool;
|
||||
# key 不进入 sandbox。未设时本轮不注册这三个工具,literature 会降级使用其他来源。
|
||||
PAPER_SERVER_API_KEY=...
|
||||
# 可选:覆盖 paper_server 地址(host 与 docker 模式均生效)
|
||||
|
|
@ -139,6 +141,7 @@
|
|||
# ZCBOT_CREDENTIAL_MASTER_KEY=<至少 32 字符随机串>
|
||||
```
|
||||
> litellm 在 import 时副作用加载 .env;入口走 `main.py`,`.env` 自动生效。直跑 `python -c "from core.storage import ..."` 不经 litellm 链路时记得自己 `import litellm` 触发,或手动 `export ZCBOT_DB_URL=...`。
|
||||
- **平台托管来源配置**:`PAPER_SERVER_*`、`DOCUMENT_SEARCH_*`、`MP_API_KEY` 由宿主部署环境提供,修改后重启 web 生效;不进数据库、用户连接、prompt、tool result、`run_python` 或 Docker sandbox。若需求升级为用户级凭据/per-user grant,接入 `external_systems`;若需要在线动态 definition、revision/reverify 或 OAuth,先设计独立 `platform_sources` 控制面,不直接扩展当前静态 env registry。
|
||||
- **依赖**:`pip install -r requirements.txt`(已在 `.venv` 里;含 `bcrypt`、`segno`、`cryptography`)。
|
||||
- **微信接入(ClawBot,§8.7)**:① `main.py db upgrade head` 带上 migration `0012`;② `.env` 设 `ZCBOT_WECHAT_BOT_ENABLED=1` + `ZCBOT_WECHAT_SECRET_KEY=<串>`;③ 用户登录后点**左栏 rail「微信」按钮**(`/static/wechat_bind.html` 仍保留作独立/嵌入入口)扫码绑定(需个人微信 8.0.70+ 且灰度到 ClawBot 插件)。绑定后在微信「微信 ClawBot」对话即走 zcbot;**主动推送需用户近 24h 在微信开口过一次**(冷启动/超期推不出,退邮件兜底)。**支持语音消息**(voice_item SILK v3 → pilk 解码 → 讯飞 IAT 转写进对话,回执「🎤 已识别:…」;需 `XFYUN_*` 三件套 + ffmpeg + pilk,pilk 随 requirements 装)。
|
||||
- **企业微信(渠道 B,纯推送,§8.7)**:① 管理员建自建应用 → 填 `WECOM_CORPID/AGENTID/SECRET`(+ 可见范围含目标用户);② `main.py db upgrade head`。**绑定两条路,任选**:
|
||||
|
|
@ -965,7 +968,7 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_<user_uuid>" /opt
|
|||
| DB 级单测把行写进了生产库 | 测试只认 `ZCBOT_TEST_DB_URL`(见「环境」段);别把它指向 `.env` 里的隧道 URL。已发生的:按测试专属 email(`test-*@invalid.local`)过滤清理 usage_events/tasks/scheduled_jobs/users |
|
||||
| `ModuleNotFoundError: litellm` | 用了全局 `python`,改 `.venv/Scripts/python.exe ...` |
|
||||
| Windows 控制台 emoji 崩 | Python stdout 是 GBK。用 `[OK]` / `[ng]` 等 ASCII 标签(见 memory) |
|
||||
| `paper_search` 等工具未出现或报 paper_server 认证失败 | `.env` 配 `PAPER_SERVER_API_KEY=`(paper_server Django admin 里建 ApiKey)并重启 web 进程;工具在宿主侧调用,key 不进入 sandbox |
|
||||
| `paper_server_search` 等工具未出现或报 paper_server 认证失败 | `.env` 配 `PAPER_SERVER_API_KEY=`(paper_server Django admin 里建 ApiKey)并重启 web 进程;工具在宿主侧调用,key 不进入 sandbox |
|
||||
| `db upgrade` 报 `column already exists` | DB 已被改过,`db current` 确认 revision,必要时手 ALTER 或 `db downgrade base` 重来 |
|
||||
| Resume 找不到 task | dev SPA 左侧 task 列表看 task_id 是否在;或 `curl /v1/tasks` 拉 |
|
||||
| task 删了文件还在 | 现在 `DELETE /v1/tasks/{id}` 是**软删**,本就不动任何磁盘文件(留作语料 + 可恢复);要清磁盘走 `POST /v1/files/delete`。彻底物理删 task(及 messages)留给将来的管理员清理工具;当前如需手动:`psql> DELETE FROM tasks WHERE task_id=...`(messages/usage_events CASCADE) |
|
||||
|
|
@ -1007,10 +1010,10 @@ sudo xfs_quota -x -c "limit -p bhard=10g zcbot_<user_uuid>" /opt
|
|||
| `[startup] reaped N stale active run(s)` | 上次 web 进程未正常 finish 留下 N 个孤儿 run,启动 lifespan 自动标 error。info 级,无需处理 |
|
||||
| `seedream` tool 没出现在对话里 | `.env` 没设 `ARK_API_KEY`,build_agent 跳过注册。设了重启 web 即可;无需迁移、无需 DB 改动 |
|
||||
| 图像模型选了「GPT 生图」却仍走 seedream / 报错 | `.env` 没设 `UNIFYLLM_API_KEY`(gpt_image tool 未注册,静默 fallback 豆包);或服务器没代理出口(直连 unifyllm.ai TLS 失败)。跑 `scripts/diag_unifyllm.py` 验证连通后重启 web |
|
||||
| `document_*` tool 没出现在对话里 | `.env` 没设 `DOCUMENT_SEARCH_API_KEY`,build_agent 跳过注册。设了重启 web 即可;key 不进入 sandbox。 |
|
||||
| `materials_library_*` tool 没出现在对话里 | `.env` 没设 `DOCUMENT_SEARCH_API_KEY`,build_agent 跳过注册。设了重启 web 即可;key 不进入 sandbox。 |
|
||||
| 文件区点 `.pptx` 弹"服务器未装 LibreOffice"/ `office_to_pdf` 提示缺 Writer/Calc/Impress | web host(非 sandbox)缺对应 LibreOffice 组件。`sudo apt-get install -y --no-install-recommends libreoffice-writer libreoffice-calc libreoffice-impress fonts-noto-cjk` 后**重启 web**。dev(Windows)`winget install TheDocumentFoundation.LibreOffice`。验:`soffice --version`,再分别拿一个 `.docx/.xlsx/.pptx` 调 `office_to_pdf` 冒烟。 |
|
||||
| `.pptx` 预览首次慢几秒 | 正常 —— soffice 冷启 + 转换 ~2-4s,转完缓存到源同目录 `.preview/<stem>.<hash>.pdf`,再点即时。源文件一改(mtime/size 变)hash 变、自动重转 |
|
||||
| `mp_*` tool 没出现在对话里 | `.env` 没设 `MP_API_KEY`,build_agent 跳过注册。设了重启 web 即可;Materials Project 联网查询走 host-side tool,离线 pymatgen 不受影响。 |
|
||||
| `materials_project_*` tool 没出现在对话里 | `.env` 没设 `MP_API_KEY`,build_agent 跳过注册。设了重启 web 即可;Materials Project 联网查询走 host-side tool,离线 pymatgen 不受影响。 |
|
||||
| 豆包调价了 | 改 `config/media/doubao.yaml` 的 `price_cny_per_image` 一行 → 重启 web。**历史 usage_events 不受影响**(units jsonb 里有当时单价 snapshot,聚合查仍按旧价);新写入按新价。涨价瞬间到改 YAML 中间这段记账偏低,开发期接受 |
|
||||
| `kill -HUP <pid>` 后 `/openapi.json` 没新接口 | uvicorn **不响应 SIGHUP**(没装 handler,落 Python 默认终止;Windows 上信号本身无效)。Ubuntu 上用 `systemctl restart zcbot`,或 unit 加 `--reload` 让 uvicorn 监听文件自动重起(见"部署"段)。验证:`curl -s http://127.0.0.1:8765/openapi.json \| python3 -c 'import sys,json;print([p for p in json.load(sys.stdin)["paths"] if "auth" in p])'` |
|
||||
| `systemctl restart zcbot` 要等几十秒才退 | 正常 —— 优雅 drain 在等在跑的 run 收尾(`shutdown.drain_timeout` 默 30s),没在跑 run 时秒退。journal 出现 `[shutdown] draining N in-flight run(s)` 即正常。真急(不在乎杀掉在跑 run):`systemctl kill -s KILL zcbot` |
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# zcbot Skill 清单
|
||||
|
||||
服务对象:中国建筑材料科学研究总院 —— 无机非金属材料 R&D(水泥 / 混凝土 / 玻璃 / 陶瓷 / 耐火 / 新型建材)
|
||||
最后更新:2026-08-26(literature 的 paper_server 改用宿主侧受控工具并支持出版物类型前置过滤)
|
||||
最后更新:2026-08-26(平台托管来源统一改用来源明确的宿主侧工具)
|
||||
Skill 总数:17
|
||||
|
||||
zcbot 的"skill"是一份可加载的工作流脚本(`skills/<name>/SKILL.md` + 配套 templates / scripts / Python helper),模型在识别用户意图后挂载对应 skill,按其内置的阶段化流程产出可交付物。本文档面向**使用方 / 协作方**,按"做什么、什么时候用、什么时候别用、典型产物"组织。
|
||||
|
|
@ -264,7 +264,7 @@ zcbot 的"skill"是一份可加载的工作流脚本(`skills/<name>/SKILL.md` +
|
|||
|
||||
证据统一标记为 `metadata_only` / `abstract_verified` / `snippet_verified` / `fulltext_verified`。下载原件但未实际读取,不算全文已核对;出版物真实存在也不代表它支持当前论断。
|
||||
|
||||
**主要能力**:host-side `paper_search` / `paper_get` / `paper_fetch`(支持 `publication_type` 服务端过滤),以及 `document_list_kb` / `document_search` / `document_download`;`merge_publications.py` 负责确定性的标准化、精确合并和疑似重复标记。
|
||||
**主要能力**:host-side `paper_server_search` / `paper_server_get` / `paper_server_fetch`(支持 `publication_type` 服务端过滤),以及 `materials_library_list` / `materials_library_search` / `materials_library_fetch`;`merge_publications.py` 负责确定性的标准化、精确合并和疑似重复标记。
|
||||
|
||||
**典型产物**:多类型出版物候选清单、规范化 `publications.json`、可核验摘要或全文、引文与论断证据台账。
|
||||
|
||||
|
|
@ -327,7 +327,7 @@ zcbot 的"skill"是一份可加载的工作流脚本(`skills/<name>/SKILL.md` +
|
|||
- 正向算 XRD pattern → `XRDCalculator`,跟实测谱对比
|
||||
- 物相对称性 → `SpacegroupAnalyzer`
|
||||
- 凝胶 / 水化产物热力学稳定性 → `PhaseDiagram` + `PDEntry`
|
||||
- 从 MP 拉已知结构 / 性质 → `mp_search_summary` / `mp_get_structure` / `mp_get_entries`
|
||||
- 从 MP 拉已知结构 / 性质 → `materials_project_search` / `materials_project_get_structure` / `materials_project_get_entries`
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -100,7 +100,7 @@ class _RepeatGuard:
|
|||
"""检测「同名同参 + 无产出」的病理性重复调用,断掉死循环。
|
||||
|
||||
背景(2026-06-08 DB 实测):高轮数烧 token 的 task 里,单个工具被用**完全相同的
|
||||
参数**重复调用几十上百次(`document_search` 122 次、空参数 `shell{}` 51 次、反复
|
||||
参数**重复调用几十上百次(`materials_library_search` 122 次、空参数 `shell{}` 51 次、反复
|
||||
`glob` 同一个不存在的路径)。loop 原本对此零防护,照单全收直到撞 max_iterations。
|
||||
|
||||
命门是只惩罚「无产出」重复,绝不误伤正常迭代:
|
||||
|
|
|
|||
|
|
@ -16,7 +16,6 @@
|
|||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Optional
|
||||
|
|
@ -24,7 +23,6 @@ from uuid import UUID
|
|||
|
||||
from tools.ask_user import AskUserTool
|
||||
from tools.check_process import CheckProcessTool
|
||||
from tools.documents import DocumentDownloadTool, DocumentListKbTool, DocumentSearchTool
|
||||
from tools.external_systems import (
|
||||
ExternalSystemCallTool,
|
||||
ExternalSystemListTool,
|
||||
|
|
@ -35,13 +33,7 @@ from tools.external_systems import (
|
|||
from tools.fs import EditTool, GlobTool, GrepTool, ReadTool, WriteTool
|
||||
from tools.gpt_image import GptImageTool
|
||||
from tools.look_at_image import LookAtImageTool
|
||||
from tools.materials_project import (
|
||||
MaterialsProjectGetEntriesTool,
|
||||
MaterialsProjectGetStructureTool,
|
||||
MaterialsProjectSearchSummaryTool,
|
||||
)
|
||||
from tools.office_to_pdf import OfficeToPdfTool
|
||||
from tools.paper_server import PaperFetchTool, PaperGetTool, PaperSearchTool
|
||||
from tools.read_document import ReadDocumentTool
|
||||
from tools.register_artifact import RegisterArtifactTool
|
||||
from tools.rename_working_dir import RenameWorkingDirTool
|
||||
|
|
@ -72,6 +64,7 @@ from tools.wechat_bot import WechatPushTool, wechat_push_available
|
|||
from core.asr_lfasr import is_configured as lfasr_configured
|
||||
from core.bocha_client import BochaConfig
|
||||
from core.task_actions import DeferredTaskActions
|
||||
from platform_sources import PlatformSourceContext, build_platform_source_tools
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -98,10 +91,6 @@ class ToolContext:
|
|||
office_to_pdf_available: bool # backend host 是否可调用 LibreOffice
|
||||
|
||||
|
||||
def _env_set(name: str) -> Callable[[], bool]:
|
||||
return lambda: bool(os.getenv(name, "").strip())
|
||||
|
||||
|
||||
def _pick_variant(section: dict, preferred: str = "") -> tuple[str, Optional[dict]]:
|
||||
"""从 yaml 段选 variant:preferred 命中优先,否则第一个 dict 条目;无 → ("", None)。"""
|
||||
if preferred:
|
||||
|
|
@ -164,26 +153,14 @@ def build_tools(ctx: ToolContext) -> dict[str, Any]:
|
|||
),
|
||||
]
|
||||
|
||||
def _document_search() -> list:
|
||||
return [
|
||||
DocumentListKbTool(**base),
|
||||
DocumentSearchTool(**base),
|
||||
DocumentDownloadTool(working_dir=ctx.working_dir_path, **base),
|
||||
]
|
||||
|
||||
def _materials_project() -> list:
|
||||
return [
|
||||
MaterialsProjectSearchSummaryTool(working_dir=ctx.working_dir_path, **base),
|
||||
MaterialsProjectGetStructureTool(working_dir=ctx.working_dir_path, **base),
|
||||
MaterialsProjectGetEntriesTool(working_dir=ctx.working_dir_path, **base),
|
||||
]
|
||||
|
||||
def _paper_server() -> list:
|
||||
return [
|
||||
PaperSearchTool(**base),
|
||||
PaperGetTool(**base),
|
||||
PaperFetchTool(working_dir=ctx.working_dir_path, **base),
|
||||
]
|
||||
def _platform_sources() -> list:
|
||||
return build_platform_source_tools(
|
||||
PlatformSourceContext(
|
||||
base_dir=ctx.tool_base,
|
||||
user_root=ctx.ur_path,
|
||||
working_dir=ctx.working_dir_path,
|
||||
)
|
||||
)
|
||||
|
||||
def _external_system_status() -> list:
|
||||
return [ExternalSystemListTool(ctx.uid, **base)]
|
||||
|
|
@ -320,11 +297,8 @@ def build_tools(ctx: ToolContext) -> dict[str, Any]:
|
|||
("core", lambda: True, _core),
|
||||
# 当前 working_dir 只能延迟到交互 run 收尾后改名;定时 run 不允许自行改目录。
|
||||
("task_actions", lambda: not ctx.scheduled_run, _task_actions),
|
||||
# Secret-bearing 域工具一律 host-side、仅对应 env 存在才注册(§7.5 #7):
|
||||
# key 绝不进 run_python / 沙箱。
|
||||
("document_search", _env_set("DOCUMENT_SEARCH_API_KEY"), _document_search),
|
||||
("materials_project", _env_set("MP_API_KEY"), _materials_project),
|
||||
("paper_server", _env_set("PAPER_SERVER_API_KEY"), _paper_server),
|
||||
# 平台托管来源由独立 bounded context 统一装配;来源配置和隔离不在 core 展开。
|
||||
("platform_sources", lambda: True, _platform_sources),
|
||||
(
|
||||
"external_system_status",
|
||||
lambda: _external_system_status_available(ctx.uid),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,13 @@
|
|||
"""平台托管只读数据源的独立 bounded context。"""
|
||||
|
||||
from .registry import (
|
||||
PlatformSourceContext,
|
||||
PlatformSourceProvider,
|
||||
build_platform_source_tools,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"PlatformSourceContext",
|
||||
"PlatformSourceProvider",
|
||||
"build_platform_source_tools",
|
||||
]
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""Host-side document_search tools.
|
||||
"""Host-side internal materials library tools.
|
||||
|
||||
These tools intentionally keep DOCUMENT_SEARCH_API_KEY on the host side. The
|
||||
sandbox receives only business arguments and trimmed results / saved paths.
|
||||
|
|
@ -9,12 +9,13 @@ from concurrent.futures import ThreadPoolExecutor
|
|||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from . import document_client as doc_client
|
||||
from . import materials_library_client as client
|
||||
from .security import safe_error_text
|
||||
|
||||
from .base import Tool
|
||||
from tools.base import Tool
|
||||
|
||||
_MAX_QUERIES = 8 # document_search 单次批量 query 上限
|
||||
_MAX_DOWNLOADS = 10 # document_download 单次批量 item 上限
|
||||
_MAX_QUERIES = 8 # materials_library_search 单次批量 query 上限
|
||||
_MAX_DOWNLOADS = 10 # materials_library_fetch 单次批量 item 上限
|
||||
_CONCURRENCY = 6
|
||||
|
||||
|
||||
|
|
@ -36,19 +37,20 @@ def _dedup_keep_order(items: list[str]) -> list[str]:
|
|||
return out
|
||||
|
||||
|
||||
class DocumentListKbTool(Tool):
|
||||
name = "document_list_kb"
|
||||
class MaterialsLibraryListTool(Tool):
|
||||
name = "materials_library_list"
|
||||
description = (
|
||||
"List internal materials knowledge bases available in document_search. "
|
||||
"Use before document_search when the user did not specify a materials domain."
|
||||
"List internal materials knowledge bases available to materials_library_search. "
|
||||
"Use before searching when the user did not specify a materials domain."
|
||||
)
|
||||
parameters = {"type": "object", "properties": {}}
|
||||
|
||||
def execute(self) -> str:
|
||||
try:
|
||||
kbs = doc_client.list_kb()
|
||||
kbs = client.list_kb()
|
||||
except Exception as e:
|
||||
return f"[Error] document_list_kb failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("DOCUMENT_SEARCH_API_KEY",))
|
||||
return f"[Error] materials_library_list failed: {type(e).__name__}: {detail}"
|
||||
if not kbs:
|
||||
return "(no knowledge bases returned)"
|
||||
lines = ["Knowledge bases:"]
|
||||
|
|
@ -64,8 +66,8 @@ class DocumentListKbTool(Tool):
|
|||
return "\n".join(lines)
|
||||
|
||||
|
||||
class DocumentSearchTool(Tool):
|
||||
name = "document_search"
|
||||
class MaterialsLibrarySearchTool(Tool):
|
||||
name = "materials_library_search"
|
||||
description = (
|
||||
"Search the internal materials document knowledge base with one OR MORE queries at once. "
|
||||
"Pass every distinct query you want in a single `queries` list instead of calling this tool "
|
||||
|
|
@ -85,7 +87,7 @@ class DocumentSearchTool(Tool):
|
|||
"kb_names": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Optional knowledge-base names from document_list_kb (applies to all queries).",
|
||||
"description": "Optional knowledge-base names from materials_library_list (applies to all queries).",
|
||||
},
|
||||
"classification_ids": {
|
||||
"type": "array",
|
||||
|
|
@ -116,14 +118,15 @@ class DocumentSearchTool(Tool):
|
|||
) -> str:
|
||||
"""搜单个 query,返回格式化文本块或 [Error ...];绝不抛异常(供并发安全调用)。"""
|
||||
try:
|
||||
docs = doc_client.search(
|
||||
docs = client.search(
|
||||
query=query,
|
||||
kb_names=kb_names or None,
|
||||
classification_ids=classification_ids or None,
|
||||
max_documents=max_documents,
|
||||
)
|
||||
except Exception as e:
|
||||
return f"[Error] document_search failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("DOCUMENT_SEARCH_API_KEY",))
|
||||
return f"[Error] materials_library_search failed: {type(e).__name__}: {detail}"
|
||||
if not docs:
|
||||
return f"(no documents found for query: {query!r})"
|
||||
|
||||
|
|
@ -188,13 +191,13 @@ class DocumentSearchTool(Tool):
|
|||
return out
|
||||
|
||||
|
||||
class DocumentDownloadTool(Tool):
|
||||
name = "document_download"
|
||||
class MaterialsLibraryFetchTool(Tool):
|
||||
name = "materials_library_fetch"
|
||||
description = (
|
||||
"Download one OR MORE original documents from document_search into task_dir/documents/. "
|
||||
"Download one OR MORE original documents from materials_library_search into task_dir/documents/. "
|
||||
"Pass every document you want in a single `items` list instead of calling this tool repeatedly — "
|
||||
"downloads run concurrently and one failing item does not abort the others. "
|
||||
"Use the file_name and kb_name returned by document_search."
|
||||
"Use the file_name and kb_name returned by materials_library_search."
|
||||
)
|
||||
parameters = {
|
||||
"type": "object",
|
||||
|
|
@ -204,8 +207,8 @@ class DocumentDownloadTool(Tool):
|
|||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file_name": {"type": "string", "description": "Original file_name or md_filename returned by document_search."},
|
||||
"kb_name": {"type": "string", "description": "Knowledge-base name returned by document_search."},
|
||||
"file_name": {"type": "string", "description": "Original file_name or md_filename returned by materials_library_search."},
|
||||
"kb_name": {"type": "string", "description": "Knowledge-base name returned by materials_library_search."},
|
||||
"preview": {"type": "boolean", "description": "Request inline preview disposition. Usually false."},
|
||||
},
|
||||
"required": ["file_name", "kb_name"],
|
||||
|
|
@ -235,14 +238,15 @@ class DocumentDownloadTool(Tool):
|
|||
if not file_name or not kb_name:
|
||||
return f"[Error] file_name / kb_name 不可为空: {item!r}"
|
||||
try:
|
||||
rel = doc_client.download(
|
||||
rel = client.download(
|
||||
file_name=file_name,
|
||||
kb_name=kb_name,
|
||||
working_dir=str(self.working_dir),
|
||||
preview=bool(item.get("preview", False)),
|
||||
)
|
||||
except Exception as e:
|
||||
return f"[Error] download {file_name!r} failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("DOCUMENT_SEARCH_API_KEY",))
|
||||
return f"[Error] download {file_name!r} failed: {type(e).__name__}: {detail}"
|
||||
return f"saved: {self._display(self.working_dir / rel)}"
|
||||
|
||||
def execute(self, items: list[dict] | dict) -> str:
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""内部材料知识库的 host-side 客户端。
|
||||
"""platform_sources 内部材料知识库的 host-side 客户端。
|
||||
|
||||
base_url / api_key 走 env:
|
||||
DOCUMENT_SEARCH_URL 默认 https://ai.ctc-zc.com:8100/api
|
||||
|
|
@ -11,7 +11,9 @@ import os
|
|||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from .base import Tool
|
||||
from tools.base import Tool
|
||||
|
||||
from .security import safe_error_text
|
||||
|
||||
try: # patched in tests; missing dependency should produce a clean tool error.
|
||||
from mp_api.client import MPRester # type: ignore
|
||||
|
|
@ -59,15 +61,16 @@ def _mpr():
|
|||
return MPRester(_mp_key())
|
||||
|
||||
|
||||
class MaterialsProjectSearchSummaryTool(Tool):
|
||||
name = "mp_search_summary"
|
||||
class MaterialsProjectSearchTool(Tool):
|
||||
name = "materials_project_search"
|
||||
description = (
|
||||
"Search Materials Project summary data using the host MP_API_KEY. "
|
||||
"Single query: pass formula / material_ids / elements → returns trimmed JSON. "
|
||||
"Batch: pass `formulas` (a list) to look up MANY formulae in ONE tool call — "
|
||||
"full results are written to task_dir/materials/*.json and only a compact "
|
||||
"per-formula digest is returned. Always prefer batch over calling this tool "
|
||||
"once per formula. Use mp_get_structure to save a CIF for offline pymatgen analysis."
|
||||
"once per formula. Use materials_project_get_structure to save a CIF for "
|
||||
"offline pymatgen analysis."
|
||||
)
|
||||
parameters = {
|
||||
"type": "object",
|
||||
|
|
@ -125,7 +128,8 @@ class MaterialsProjectSearchSummaryTool(Tool):
|
|||
num_chunks=1, chunk_size=limit, **kwargs
|
||||
)
|
||||
except Exception as e:
|
||||
return f"[Error] mp_search_summary failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("MP_API_KEY",))
|
||||
return f"[Error] materials_project_search failed: {type(e).__name__}: {detail}"
|
||||
plain = [_to_plain(d) for d in list(docs)[:limit]]
|
||||
return json.dumps(plain, ensure_ascii=False, indent=2)
|
||||
|
||||
|
|
@ -148,7 +152,8 @@ class MaterialsProjectSearchSummaryTool(Tool):
|
|||
try:
|
||||
session = _mpr()
|
||||
except Exception as e:
|
||||
return f"[Error] mp_search_summary batch failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("MP_API_KEY",))
|
||||
return f"[Error] materials_project_search batch failed: {type(e).__name__}: {detail}"
|
||||
agg: list[dict[str, Any]] = []
|
||||
n_ok = 0
|
||||
# 单个 formula 出错不连坐整批;复用一个 MPRester 会话避免逐条重认证。
|
||||
|
|
@ -164,13 +169,14 @@ class MaterialsProjectSearchSummaryTool(Tool):
|
|||
n_ok += 1
|
||||
agg.append({"formula": f, "n": len(results), "results": results})
|
||||
except Exception as e:
|
||||
agg.append({"formula": f, "error": f"{type(e).__name__}: {e}"})
|
||||
detail = safe_error_text(e, ("MP_API_KEY",))
|
||||
agg.append({"formula": f, "error": f"{type(e).__name__}: {detail}"})
|
||||
return self._render_batch(agg, fields, n_ok)
|
||||
|
||||
def _render_batch(self, agg: list[dict[str, Any]], fields: list[str], n_ok: int) -> str:
|
||||
empties = [a["formula"] for a in agg if "error" in a or a.get("n", 0) == 0]
|
||||
header = (
|
||||
f"mp_search_summary batch: {len(agg)} 个化学式 → {n_ok} 个有结果,"
|
||||
f"materials_project_search batch: {len(agg)} 个化学式 → {n_ok} 个有结果,"
|
||||
f"{len(empties)} 个空/错误。\n"
|
||||
f"每条记录字段: {', '.join(fields)}\n"
|
||||
)
|
||||
|
|
@ -192,7 +198,8 @@ class MaterialsProjectSearchSummaryTool(Tool):
|
|||
)
|
||||
except Exception as e:
|
||||
body = (
|
||||
f"[warn] 写文件失败({type(e).__name__}: {e}),改为内联返回完整 JSON:\n"
|
||||
f"[warn] 写文件失败({type(e).__name__}: "
|
||||
f"{safe_error_text(e, ('MP_API_KEY',))}),改为内联返回完整 JSON:\n"
|
||||
+ json.dumps(agg, ensure_ascii=False, indent=2)
|
||||
+ "\n"
|
||||
)
|
||||
|
|
@ -207,7 +214,7 @@ class MaterialsProjectSearchSummaryTool(Tool):
|
|||
|
||||
|
||||
class MaterialsProjectGetStructureTool(Tool):
|
||||
name = "mp_get_structure"
|
||||
name = "materials_project_get_structure"
|
||||
description = (
|
||||
"Download a Materials Project structure by material_id and save it as CIF in task_dir/materials/."
|
||||
)
|
||||
|
|
@ -244,12 +251,13 @@ class MaterialsProjectGetStructureTool(Tool):
|
|||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
struct.to(filename=str(dest))
|
||||
except Exception as e:
|
||||
return f"[Error] mp_get_structure failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("MP_API_KEY",))
|
||||
return f"[Error] materials_project_get_structure failed: {type(e).__name__}: {detail}"
|
||||
return f"saved: {self._display(dest)}"
|
||||
|
||||
|
||||
class MaterialsProjectGetEntriesTool(Tool):
|
||||
name = "mp_get_entries"
|
||||
name = "materials_project_get_entries"
|
||||
description = (
|
||||
"Fetch Materials Project computed entries for a chemical system and save trimmed JSON to task_dir/materials/. "
|
||||
"Downloads the FULL chemical system (all sub-systems) — volume grows fast with element count; call sparingly and reuse the saved file rather than re-querying."
|
||||
|
|
@ -297,5 +305,6 @@ class MaterialsProjectGetEntriesTool(Tool):
|
|||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
except Exception as e:
|
||||
return f"[Error] mp_get_entries failed: {type(e).__name__}: {e}"
|
||||
detail = safe_error_text(e, ("MP_API_KEY",))
|
||||
return f"[Error] materials_project_get_entries failed: {type(e).__name__}: {detail}"
|
||||
return f"saved: {self._display(dest)}"
|
||||
|
|
@ -15,7 +15,9 @@ from urllib.parse import urljoin, urlparse
|
|||
|
||||
import httpx
|
||||
|
||||
from .base import Tool
|
||||
from tools.base import Tool
|
||||
|
||||
from .security import safe_error_text
|
||||
|
||||
_DEFAULT_BASE_URL = "http://paper.xxhhcty.xyz:8080"
|
||||
_TIMEOUT = 30.0
|
||||
|
|
@ -153,8 +155,8 @@ def _media_url(raw_url: str, base_url: str) -> str:
|
|||
return url
|
||||
|
||||
|
||||
class PaperSearchTool(Tool):
|
||||
name = "paper_search"
|
||||
class PaperServerSearchTool(Tool):
|
||||
name = "paper_server_search"
|
||||
description = (
|
||||
"Search the platform paper_server metadata collection. "
|
||||
"Use publication_type for server-side source-type filtering, e.g. 'book', "
|
||||
|
|
@ -245,15 +247,16 @@ class PaperSearchTool(Tool):
|
|||
_base_url, api_url, api_key = _config()
|
||||
papers = _results(_get_json(api_url + "/", api_key=api_key, params=params))
|
||||
except Exception as exc:
|
||||
return f"[Error] paper_search failed:{type(exc).__name__}:{exc}"
|
||||
detail = safe_error_text(exc, ("PAPER_SERVER_API_KEY",))
|
||||
return f"[Error] paper_server_search failed:{type(exc).__name__}:{detail}"
|
||||
trimmed = [
|
||||
{key: paper.get(key) for key in _LIST_FIELDS} for paper in papers[:limit]
|
||||
]
|
||||
return json.dumps(trimmed, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
class PaperGetTool(Tool):
|
||||
name = "paper_get"
|
||||
class PaperServerGetTool(Tool):
|
||||
name = "paper_server_get"
|
||||
description = (
|
||||
"Get one complete paper_server metadata record by internal id or exact DOI."
|
||||
)
|
||||
|
|
@ -269,15 +272,17 @@ class PaperGetTool(Tool):
|
|||
try:
|
||||
paper = _get_paper(id_or_doi)
|
||||
except Exception as exc:
|
||||
return f"[Error] paper_get failed:{type(exc).__name__}:{exc}"
|
||||
detail = safe_error_text(exc, ("PAPER_SERVER_API_KEY",))
|
||||
return f"[Error] paper_server_get failed:{type(exc).__name__}:{detail}"
|
||||
return json.dumps(paper, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
class PaperFetchTool(Tool):
|
||||
name = "paper_fetch"
|
||||
class PaperServerFetchTool(Tool):
|
||||
name = "paper_server_fetch"
|
||||
description = (
|
||||
"Download an available PDF or XML from paper_server into the current task's "
|
||||
"papers/ directory. Use the format actually reported by paper_search/paper_get."
|
||||
"papers/ directory. Use the format actually reported by "
|
||||
"paper_server_search/paper_server_get."
|
||||
)
|
||||
parameters = {
|
||||
"type": "object",
|
||||
|
|
@ -358,5 +363,6 @@ class PaperFetchTool(Tool):
|
|||
except OSError:
|
||||
pass
|
||||
except Exception as exc:
|
||||
return f"[Error] paper_fetch failed:{type(exc).__name__}:{exc}"
|
||||
detail = safe_error_text(exc, ("PAPER_SERVER_API_KEY",))
|
||||
return f"[Error] paper_server_fetch failed:{type(exc).__name__}:{detail}"
|
||||
return f"saved:{self._display(destination)}"
|
||||
|
|
@ -0,0 +1,159 @@
|
|||
"""平台托管来源的显式注册表和统一生命周期入口。
|
||||
|
||||
Provider 只统一来源标识、能力声明、可用性和 Tool 装配;各来源的业务 API
|
||||
保持专用 typed contract,不在这里抽象成 search/get/fetch 万能接口。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from tools.base import Tool
|
||||
|
||||
from .materials_library import (
|
||||
MaterialsLibraryFetchTool,
|
||||
MaterialsLibraryListTool,
|
||||
MaterialsLibrarySearchTool,
|
||||
)
|
||||
from .materials_project import (
|
||||
MPRester,
|
||||
MaterialsProjectGetEntriesTool,
|
||||
MaterialsProjectGetStructureTool,
|
||||
MaterialsProjectSearchTool,
|
||||
)
|
||||
from .paper_server import (
|
||||
PaperServerFetchTool,
|
||||
PaperServerGetTool,
|
||||
PaperServerSearchTool,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PlatformSourceContext:
|
||||
"""Provider 装配 Tool 所需的非敏感宿主上下文。"""
|
||||
|
||||
base_dir: Path
|
||||
user_root: Path
|
||||
working_dir: Path
|
||||
|
||||
|
||||
class PlatformSourceProvider(Protocol):
|
||||
source_id: str
|
||||
capabilities: frozenset[str]
|
||||
|
||||
def available(self) -> bool: ...
|
||||
|
||||
def build_tools(self, context: PlatformSourceContext) -> list[Tool]: ...
|
||||
|
||||
|
||||
def _env_set(name: str) -> bool:
|
||||
return bool(os.environ.get(name, "").strip())
|
||||
|
||||
|
||||
def _require_valid_http_url(name: str, default: str) -> None:
|
||||
value = os.environ.get(name, default).strip()
|
||||
parsed = urlparse(value)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
|
||||
raise ValueError(f"invalid {name}")
|
||||
|
||||
|
||||
class PaperServerProvider:
|
||||
source_id = "paper_server"
|
||||
capabilities = frozenset({"search", "metadata", "download"})
|
||||
|
||||
def available(self) -> bool:
|
||||
if not _env_set("PAPER_SERVER_API_KEY"):
|
||||
return False
|
||||
_require_valid_http_url("PAPER_SERVER_URL", "http://paper.xxhhcty.xyz:8080")
|
||||
return True
|
||||
|
||||
def build_tools(self, context: PlatformSourceContext) -> list[Tool]:
|
||||
common = {"base_dir": context.base_dir, "user_root": context.user_root}
|
||||
return [
|
||||
PaperServerSearchTool(**common),
|
||||
PaperServerGetTool(**common),
|
||||
PaperServerFetchTool(working_dir=context.working_dir, **common),
|
||||
]
|
||||
|
||||
|
||||
class MaterialsLibraryProvider:
|
||||
source_id = "materials_library"
|
||||
capabilities = frozenset({"catalog", "batch_search", "batch_download"})
|
||||
|
||||
def available(self) -> bool:
|
||||
if not _env_set("DOCUMENT_SEARCH_API_KEY"):
|
||||
return False
|
||||
_require_valid_http_url(
|
||||
"DOCUMENT_SEARCH_URL", "https://ai.ctc-zc.com:8100/api"
|
||||
)
|
||||
return True
|
||||
|
||||
def build_tools(self, context: PlatformSourceContext) -> list[Tool]:
|
||||
common = {"base_dir": context.base_dir, "user_root": context.user_root}
|
||||
return [
|
||||
MaterialsLibraryListTool(**common),
|
||||
MaterialsLibrarySearchTool(**common),
|
||||
MaterialsLibraryFetchTool(working_dir=context.working_dir, **common),
|
||||
]
|
||||
|
||||
|
||||
class MaterialsProjectProvider:
|
||||
source_id = "materials_project"
|
||||
capabilities = frozenset({"summary_search", "structure", "entries"})
|
||||
|
||||
def available(self) -> bool:
|
||||
if not _env_set("MP_API_KEY"):
|
||||
return False
|
||||
if MPRester is None:
|
||||
raise RuntimeError("Materials Project SDK unavailable")
|
||||
return True
|
||||
|
||||
def build_tools(self, context: PlatformSourceContext) -> list[Tool]:
|
||||
common = {"base_dir": context.base_dir, "user_root": context.user_root}
|
||||
return [
|
||||
MaterialsProjectSearchTool(working_dir=context.working_dir, **common),
|
||||
MaterialsProjectGetStructureTool(
|
||||
working_dir=context.working_dir, **common
|
||||
),
|
||||
MaterialsProjectGetEntriesTool(
|
||||
working_dir=context.working_dir, **common
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
# 显式可信列表:不扫描目录、不动态 import 部署侧 definition。
|
||||
TRUSTED_PROVIDERS: tuple[PlatformSourceProvider, ...] = (
|
||||
PaperServerProvider(),
|
||||
MaterialsLibraryProvider(),
|
||||
MaterialsProjectProvider(),
|
||||
)
|
||||
|
||||
|
||||
def build_platform_source_tools(
|
||||
context: PlatformSourceContext,
|
||||
*,
|
||||
providers: Sequence[PlatformSourceProvider] = TRUSTED_PROVIDERS,
|
||||
) -> list[Tool]:
|
||||
"""逐来源隔离装配;失败只跳过该来源,日志不含配置值或异常正文。"""
|
||||
|
||||
built: list[Tool] = []
|
||||
for provider in providers:
|
||||
source_id = getattr(provider, "source_id", "unknown")
|
||||
try:
|
||||
if not provider.available():
|
||||
continue
|
||||
built.extend(provider.build_tools(context))
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"platform source %s registration failed (%s)",
|
||||
source_id,
|
||||
type(exc).__name__,
|
||||
)
|
||||
return built
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
"""平台来源错误脱敏原语。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
|
||||
_QUERY_SECRET_RE = re.compile(
|
||||
r"(?i)(api[_-]?key|token|secret|password)(=|%3[dD])([^&\s]+)"
|
||||
)
|
||||
|
||||
|
||||
def safe_error_text(exc: BaseException, secret_env_names: Iterable[str]) -> str:
|
||||
"""返回可诊断但不包含部署凭据的异常文本。"""
|
||||
|
||||
text = str(exc)
|
||||
for name in secret_env_names:
|
||||
value = os.environ.get(name, "")
|
||||
if value:
|
||||
text = text.replace(value, "[REDACTED]")
|
||||
return _QUERY_SECRET_RE.sub(r"\1\2[REDACTED]", text)
|
||||
|
|
@ -35,7 +35,11 @@ if env_file.exists():
|
|||
|
||||
import json
|
||||
|
||||
from tools.paper_server import PaperFetchTool, PaperGetTool, PaperSearchTool
|
||||
from platform_sources.paper_server import (
|
||||
PaperServerFetchTool,
|
||||
PaperServerGetTool,
|
||||
PaperServerSearchTool,
|
||||
)
|
||||
|
||||
|
||||
_BASE_URL = os.environ.get("PAPER_SERVER_URL", "http://paper.xxhhcty.xyz:8080").rstrip("/")
|
||||
|
|
@ -48,15 +52,15 @@ def _json_result(raw: str):
|
|||
|
||||
|
||||
def search(**kwargs) -> list[dict]:
|
||||
return _json_result(PaperSearchTool().execute(**kwargs))
|
||||
return _json_result(PaperServerSearchTool().execute(**kwargs))
|
||||
|
||||
|
||||
def get_paper(id_or_doi: str) -> dict:
|
||||
return _json_result(PaperGetTool().execute(id_or_doi=id_or_doi))
|
||||
return _json_result(PaperServerGetTool().execute(id_or_doi=id_or_doi))
|
||||
|
||||
|
||||
def _fetch(id_or_doi: str, working_dir: str, file_format: str) -> str:
|
||||
raw = PaperFetchTool(working_dir=Path(working_dir)).execute(
|
||||
raw = PaperServerFetchTool(working_dir=Path(working_dir)).execute(
|
||||
id_or_doi=id_or_doi, format=file_format
|
||||
)
|
||||
if raw.startswith("[Error]"):
|
||||
|
|
|
|||
|
|
@ -4,11 +4,11 @@
|
|||
|
||||
依赖:`pip install pymatgen mp-api scikit-learn statsmodels`(PyMC 可选,装了就测)。
|
||||
|
||||
不依赖网络默认情况下(MP_API_KEY 没配则跳过 mp_rester 联网那一段)。
|
||||
不依赖网络默认情况下(MP_API_KEY 没配则跳过 Materials Project 联网段)。
|
||||
不动 DB / workspace,产物落系统临时目录,跑完即丢。
|
||||
|
||||
按 skill 顺序 4 段:
|
||||
step A — pymatgen import + CEMENT_PHASES 几个查询 + mp_rester 未配 key 抛错
|
||||
step A — pymatgen import + CEMENT_PHASES 几个查询 + host tool 未配 key 报错
|
||||
step B — stats_ml 三库装包 + 小 OLS / RandomForest smoke
|
||||
step C — plot_pub apply_pub_style + 最小 XRD-like 图出 PNG
|
||||
step D —(可选)MP_API_KEY 配了就联网拉一条 Materials Project 数据
|
||||
|
|
@ -137,14 +137,14 @@ def step_a_pymatgen() -> None:
|
|||
_info("MP_API_KEY 已配置,skip 缺 key 报错验证(下面 step D 测真实查询)")
|
||||
else:
|
||||
try:
|
||||
from tools.materials_project import MaterialsProjectSearchSummaryTool
|
||||
out = MaterialsProjectSearchSummaryTool().execute(formula="Ca3SiO5")
|
||||
from platform_sources.materials_project import MaterialsProjectSearchTool
|
||||
out = MaterialsProjectSearchTool().execute(formula="Ca3SiO5")
|
||||
if out.startswith("[Error]") and "MP_API_KEY" in out:
|
||||
_ok("mp_search_summary 未配 key 返回 [Error] 含 MP_API_KEY 提示")
|
||||
_ok("materials_project_search 未配 key 返回 [Error] 含 MP_API_KEY 提示")
|
||||
else:
|
||||
_fail(f"未配 key 应返回 [Error] 含 MP_API_KEY,实际: {out[:120]}")
|
||||
except Exception as e:
|
||||
_fail(f"mp_search_summary 未配 key 应返回 [Error] 而非抛异常: {type(e).__name__}: {e}")
|
||||
_fail(f"materials_project_search 未配 key 应返回 [Error] 而非抛异常: {type(e).__name__}: {e}")
|
||||
|
||||
|
||||
def step_b_stats_ml() -> None:
|
||||
|
|
@ -271,24 +271,24 @@ def step_d_mp_online() -> None:
|
|||
try:
|
||||
import json
|
||||
from skills.pymatgen.materials import lookup_phase
|
||||
from tools.materials_project import MaterialsProjectSearchSummaryTool
|
||||
from platform_sources.materials_project import MaterialsProjectSearchTool
|
||||
formula = lookup_phase("C3S") # Ca3SiO5
|
||||
t0 = time.time()
|
||||
out = MaterialsProjectSearchSummaryTool().execute(
|
||||
out = MaterialsProjectSearchTool().execute(
|
||||
formula=formula,
|
||||
fields=["material_id", "formula_pretty", "energy_above_hull"],
|
||||
limit=3,
|
||||
)
|
||||
dt = (time.time() - t0) * 1000
|
||||
if out.startswith("[Error]"):
|
||||
_fail(f"mp_search_summary 查 {formula}: {out[:160]}")
|
||||
_fail(f"materials_project_search 查 {formula}: {out[:160]}")
|
||||
return
|
||||
docs = json.loads(out)
|
||||
_ok(f"mp_search_summary 查 {formula}: 返回 {len(docs)} 条 in {dt:.0f}ms")
|
||||
_ok(f"materials_project_search 查 {formula}: 返回 {len(docs)} 条 in {dt:.0f}ms")
|
||||
for d in docs[:3]:
|
||||
print(f" {d.get('material_id')} {d.get('formula_pretty')} ehull={d.get('energy_above_hull')}")
|
||||
except Exception as e:
|
||||
_fail(f"mp_search_summary 联网查询: {type(e).__name__}: {e}")
|
||||
_fail(f"materials_project_search 联网查询: {type(e).__name__}: {e}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
|
|
|
|||
|
|
@ -43,10 +43,10 @@ description: 生成科研方向简报(research direction briefing / 重要文献
|
|||
|
||||
**先读 `references/journals.md`**。**中文方向先转专业英文术语**(库主语料英文):低碳水泥→low-carbon cement / clinker substitution;SCM→supplementary cementitious materials / fly ash / GGBFS / calcined clay;LC3→limestone calcined clay cement;碳化养护→CO2 curing / carbonation。缩写与全称都试。
|
||||
|
||||
**literature / paper_server(逐刊取最新 Elsevier 论文 + DOI)** —— 用宿主侧 `paper_search`。按 `publication_name` 精确匹配并带时间窗;需要限定论文时传 `publication_type="article"`。list 自带 abstract,看前 200–400 字判切题与分量。目标期刊包括 `Cement and Concrete Research`、`Cement and Concrete Composites`、`Construction and Building Materials`、`Journal of Cleaner Production`。
|
||||
**literature / paper_server(逐刊取最新 Elsevier 论文 + DOI)** —— 用宿主侧 `paper_server_search`。按 `publication_name` 精确匹配并带时间窗;需要限定论文时传 `publication_type="article"`。list 自带 abstract,看前 200–400 字判切题与分量。目标期刊包括 `Cement and Concrete Research`、`Cement and Concrete Composites`、`Construction and Building Materials`、`Journal of Cleaner Production`。
|
||||
某刊精确名 0 命中 → 换 `keyword=<方向英文术语>` 再搜,从返回里挑 `publication_name` 命中目标刊的;仍空记"该刊本窗口库内无收录"。
|
||||
|
||||
**literature / 内部材料库(取全文,材料类首选)** —— host-side tool `document_search`,中英 query 都行(后端跨语言语义检索);胶凝材料库 `classification_id=1`。取 `md_content` 既做候选也供引文核验抓锚点最顺。
|
||||
**literature / 内部材料库(取全文,材料类首选)** —— host-side tool `materials_library_search`,中英 query 都行(后端跨语言语义检索);胶凝材料库 `classification_id=1`。取 `md_content` 既做候选也供引文核验抓锚点最顺。
|
||||
|
||||
**web search(取动向)** —— 政策(双碳/碳配额)、标准(新国标/团标)、行业会议、企业产线中试。**单列"其他动向",不混进论文列表与总结**。
|
||||
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ description: 检索、获取、合并与核验各类学术和技术出版物,
|
|||
## 来源选择
|
||||
|
||||
- 材料主题、性能、配方、工艺、表征或全文语义检索:优先内部材料知识库。
|
||||
- DOI、题名、作者、期刊、年份、出版物类型或跨学科发现:`paper_search` 可用时优先 `paper_server`。
|
||||
- DOI、题名、作者、期刊、年份、出版物类型或跨学科发现:`paper_server_search` 可用时优先 `paper_server`。
|
||||
- 系统调研、综述、重要引用或关键论断:并查可用来源。
|
||||
- 用户指定来源:只查指定来源,除非该来源无法满足用户要求并需要说明降级方案。
|
||||
- 某一路不可用:继续使用其他来源,并明确实际覆盖范围。
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@
|
|||
|
||||
## 工具
|
||||
|
||||
- `document_list_kb()`:列出当前有效知识库。用户未指定材料方向且需要缩窄范围时调用。
|
||||
- `document_search(queries, kb_names=None, classification_ids=None, max_documents=6, content_chars_per_doc=1200)`:批量检索并返回元数据和截断的 `md_content`。
|
||||
- `document_download(items)`:把选定原件下载到当前任务的 `documents/` 目录。
|
||||
- `materials_library_list()`:列出当前有效知识库。用户未指定材料方向且需要缩窄范围时调用。
|
||||
- `materials_library_search(queries, kb_names=None, classification_ids=None, max_documents=6, content_chars_per_doc=1200)`:批量检索并返回元数据和截断的 `md_content`。
|
||||
- `materials_library_fetch(items)`:把选定原件下载到当前任务的 `documents/` 目录。
|
||||
|
||||
查询可使用中文或英文;复杂专业术语通常使用英文更精确。先规划互不重复的查询并批量调用,不逐个近义词反复搜索。
|
||||
|
||||
|
|
|
|||
|
|
@ -6,9 +6,9 @@
|
|||
|
||||
使用宿主侧 typed tools,凭据不会进入 sandbox:
|
||||
|
||||
- `paper_search(keyword="", publication_type="", year=None, year_gte=None, year_lte=None, doi="", first_author="", publication_name="", has_pdf=None, is_oa=None, limit=10)`:搜索题录并返回摘要和全文可用状态。能确定来源类型时用 `publication_type` 做服务端过滤,例如 `book`、`book-chapter`、`article`。
|
||||
- `paper_get(id_or_doi)`:按内部 ID 或 DOI 获取完整记录。
|
||||
- `paper_fetch(id_or_doi, format)`:把记录实际提供的 `pdf` 或 `xml` 保存到任务的 `papers/` 目录。
|
||||
- `paper_server_search(keyword="", publication_type="", year=None, year_gte=None, year_lte=None, doi="", first_author="", publication_name="", has_pdf=None, is_oa=None, limit=10)`:搜索题录并返回摘要和全文可用状态。统一出版物类型 `book` 映射为来源原始值 `book`,`book_chapter` 映射为 `book-chapter`;能确定来源类型时用 `publication_type` 做服务端过滤。
|
||||
- `paper_server_get(id_or_doi)`:按内部 ID 或 DOI 获取完整记录。
|
||||
- `paper_server_fetch(id_or_doi, format)`:把记录实际提供的 `pdf` 或 `xml` 保存到任务的 `papers/` 目录。
|
||||
|
||||
工具由平台自动使用 `PAPER_SERVER_API_KEY` 和可选的 `PAPER_SERVER_URL`;未配置时本轮不会注册这些工具。只通过这些工具访问,由它们处理认证、URL、响应裁剪和下载路径。
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@
|
|||
|
||||
每条引文先确认"这篇文献真实存在":
|
||||
|
||||
1. 用 `literature` 检索:材料类优先内部材料库语义检索,精确 DOI 使用 paper_server 的 `paper_search(doi=...)` / `paper_get(id_or_doi=...)`
|
||||
1. 用 `literature` 检索:材料类优先内部材料库语义检索,精确 DOI 使用 paper_server 的 `paper_server_search(doi=...)` / `paper_server_get(id_or_doi=...)`
|
||||
2. 命中 → 记下真实 DOI / 作者 / 年份 / 期刊 / 卷期页;**以库里返回为准**,不沿用记忆里的字段
|
||||
3. 两个库都查不到 → 标 `[未核实]`,**不得编造条目**;告诉用户"这条找不到来源,请提供 PDF/DOI 或删去该论断"
|
||||
|
||||
|
|
@ -34,7 +34,7 @@
|
|||
|
||||
最容易翻车的一层:文献存在,但**并不支撑你写的那句话**。逐条做:
|
||||
|
||||
1. 抓回该文献的 `md_content`(内部材料库正文片段),或用 paper_server 的 `paper_fetch(format="xml"|"pdf")` 获取原件
|
||||
1. 抓回该文献的 `md_content`(内部材料库正文片段),或用 paper_server 的 `paper_server_fetch(format="xml"|"pdf")` 获取原件
|
||||
2. 在原文里定位与论断相关的**锚点证据**:一句 ≤25 词的原文引语 + 出现的段落/小节位置
|
||||
3. 判定支撑度三档:
|
||||
- **support**:原文明确支撑该论断 → 通过
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ description: 无机材料计算工具(晶体结构 I/O、XRD 模拟、相图、
|
|||
|
||||
无机材料计算的核心库,服务建材院的水泥 / 混凝土 / 玻璃 / 陶瓷 / 耐火材料场景。离线计算在 sandbox 里用 pymatgen 官方 API;联网查 Materials Project 走 host-side tool。**本 skill 提供一个轻量 helper**:`CEMENT_PHASES` 常量(中文相名→化学式映射)。
|
||||
|
||||
> ⚠️ **配置条件**:只有宿主后端配置了 `MP_API_KEY` 时,`mp_search_summary` / `mp_get_structure` / `mp_get_entries` 才会出现在可用工具列表里。没有这些 tool 时,Materials Project 联网查询不可用;离线全部正常:用户给 `.cif` / `POSCAR` → `Structure.from_file()` / `SpacegroupAnalyzer` / `XRDCalculator`(对已有 Structure)/ `CEMENT_PHASES` 查表 / 格式转换 / `MPRelaxSet`。要 mp 拿结构 → 让用户从 https://materialsproject.org 下个 CIF 丢 task 目录;**不要脑补晶格 / 原子坐标**。
|
||||
> ⚠️ **配置条件**:只有宿主后端配置了 `MP_API_KEY` 时,`materials_project_search` / `materials_project_get_structure` / `materials_project_get_entries` 才会出现在可用工具列表里。没有这些 tool 时,Materials Project 联网查询不可用;离线全部正常:用户给 `.cif` / `POSCAR` → `Structure.from_file()` / `SpacegroupAnalyzer` / `XRDCalculator`(对已有 Structure)/ `CEMENT_PHASES` 查表 / 格式转换 / `MPRelaxSet`。要 mp 拿结构 → 让用户从 https://materialsproject.org 下个 CIF 丢 task 目录;**不要脑补晶格 / 原子坐标**。
|
||||
|
||||
## 何时用
|
||||
|
||||
|
|
@ -53,9 +53,9 @@ from pymatgen.analysis.phase_diagram import PhaseDiagram, PDEntry
|
|||
|
||||
API key 只在宿主后端读取,不要在 `run_python` 里读 env。可用 tool:
|
||||
|
||||
- `mp_search_summary(formula?, formulas?, material_ids?, elements?, fields?, limit=10)` —— 查 summary,返回裁剪 JSON。**要查多个化学式时传 `formulas=[...]` 一次批量查完**(结果落 `materials/mp_search_batch_*.json`,只回摘要),不要一个式子调一次 —— 逐条调既慢又把中间数据全灌进对话
|
||||
- `mp_get_structure(material_id, filename?)` —— 把结构保存到 task_dir `materials/*.cif`,再用 `Structure.from_file()` 离线计算
|
||||
- `mp_get_entries(elements, filename?, limit=200)` —— 把 chemsys entries 保存到 task_dir `materials/*.json`
|
||||
- `materials_project_search(formula?, formulas?, material_ids?, elements?, fields?, limit=10)` —— 查 summary,返回裁剪 JSON。**要查多个化学式时传 `formulas=[...]` 一次批量查完**(结果落 `materials/mp_search_batch_*.json`,只回摘要),不要一个式子调一次 —— 逐条调既慢又把中间数据全灌进对话
|
||||
- `materials_project_get_structure(material_id, filename?)` —— 把结构保存到 task_dir `materials/*.cif`,再用 `Structure.from_file()` 离线计算
|
||||
- `materials_project_get_entries(elements, filename?, limit=200)` —— 把 chemsys entries 保存到 task_dir `materials/*.json`
|
||||
|
||||
`MP_API_KEY` 没配 → 上述 tool 不会出现,告诉用户配置或手动从 Materials Project 下载 CIF。
|
||||
|
||||
|
|
@ -64,7 +64,7 @@ API key 只在宿主后端读取,不要在 `run_python` 里读 env。可用 tool
|
|||
### A. 实测 XRD 比对(谁是这个峰)
|
||||
|
||||
1. 用户给疑似相清单(中文 / 英文 / 简写都行)
|
||||
2. 各相分别:`CEMENT_PHASES` 查化学式 → `mp_search_summary(formula=...)` 找 material_id → `mp_get_structure(material_id=...)` 保存 CIF → `XRDCalculator().get_pattern(structure)` 算理论谱
|
||||
2. 各相分别:`CEMENT_PHASES` 查化学式 → `materials_project_search(formula=...)` 找 material_id → `materials_project_get_structure(material_id=...)` 保存 CIF → `XRDCalculator().get_pattern(structure)` 算理论谱
|
||||
3. 把各相理论谱跟实测谱(用户给的 xy 数据)叠图(走 `plot_pub`)
|
||||
4. 报"x° 这个峰最可能是 C3S 的 (h k l) 衍射"
|
||||
|
||||
|
|
@ -72,7 +72,7 @@ API key 只在宿主后端读取,不要在 `run_python` 里读 env。可用 tool
|
|||
from pymatgen.analysis.diffraction.xrd import XRDCalculator
|
||||
|
||||
xrd = XRDCalculator(wavelength="CuKa") # 默认 Cu Kα
|
||||
# 先用 mp_search_summary / mp_get_structure tool 保存 CIF,再:
|
||||
# 先用 materials_project_search / materials_project_get_structure tool 保存 CIF,再:
|
||||
struct = Structure.from_file("materials/mp-xxxx.cif")
|
||||
pattern = xrd.get_pattern(struct, two_theta_range=(5, 80))
|
||||
# pattern.x = 2θ 列表, pattern.y = 强度, pattern.hkls = (h,k,l) 列表
|
||||
|
|
@ -94,7 +94,7 @@ prim = sga.get_primitive_standard_structure() # 简约胞
|
|||
|
||||
### C. 凝胶 / 水化产物相图稳定性
|
||||
|
||||
先用 `mp_get_entries(elements=["Ca", "Si", "O", "H"])` 保存 JSON。第一版 host tool 的 entries JSON 主要用于审阅 / 归档;若要直接构造 `PhaseDiagram`,优先使用用户提供的本地 entry 数据或后续补一个专门的 host-side 相图 helper。不要在 sandbox 里绕过 tool 直接连 MP。
|
||||
先用 `materials_project_get_entries(elements=["Ca", "Si", "O", "H"])` 保存 JSON。第一版 host tool 的 entries JSON 主要用于审阅 / 归档;若要直接构造 `PhaseDiagram`,优先使用用户提供的本地 entry 数据或后续补一个专门的 host-side 相图 helper。不要在 sandbox 里绕过 tool 直接连 MP。
|
||||
|
||||
### D. 格式转换(给计算所做 VASP 输入)
|
||||
|
||||
|
|
@ -130,4 +130,4 @@ mp-api>=0.41.0
|
|||
MP_API_KEY=your_key_from_materialsproject_org
|
||||
```
|
||||
|
||||
配置后重启 web,`mp_*` tools 才会注册。sandbox / `run_python` 不读取这个 key。
|
||||
配置后重启 web,`materials_project_*` tools 才会注册。sandbox / `run_python` 不读取这个 key。
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
pymatgen skill helpers — 建材院无机材料场景常用映射(中文相名 → 化学式)。
|
||||
|
||||
LLM 在 sandbox 中只应使用 `CEMENT_PHASES` / `lookup_phase` 和离线 pymatgen。
|
||||
Materials Project 联网查询走 host-side `mp_*` tools,不要在 sandbox 里读 MP_API_KEY。
|
||||
Materials Project 联网查询走 host-side `materials_project_*` tools,不要在 sandbox 里读 MP_API_KEY。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
|
|||
|
|
@ -71,10 +71,10 @@ class TestRepeatGuard(unittest.TestCase):
|
|||
g = _RepeatGuard()
|
||||
unprods = []
|
||||
for _ in range(_RepeatGuard.SOFT + 1):
|
||||
unprods.append(g.record("document_search", {"queries": ["x"]}, "(no documents found)")[0])
|
||||
unprods.append(g.record("materials_library_search", {"queries": ["x"]}, "(no documents found)")[0])
|
||||
# 累计达到 SOFT(此时应注入软提示),但还没到 HARD 拦截
|
||||
self.assertGreaterEqual(max(unprods), _RepeatGuard.SOFT)
|
||||
self.assertFalse(g.should_block("document_search", {"queries": ["x"]}))
|
||||
self.assertFalse(g.should_block("materials_library_search", {"queries": ["x"]}))
|
||||
|
||||
def test_record_returns_productive_signal(self):
|
||||
"""record 第二个返回值喂全局无进展熔断:新非错结果=有产出,[Error]/重复=无产出。"""
|
||||
|
|
@ -94,10 +94,10 @@ class TestRepeatGuard(unittest.TestCase):
|
|||
|
||||
def test_distinct_args_tracked_separately(self):
|
||||
g = _RepeatGuard()
|
||||
_simulate(g, "document_search", {"queries": ["a"]}, ["[Error] e"] * 8)
|
||||
_simulate(g, "materials_library_search", {"queries": ["a"]}, ["[Error] e"] * 8)
|
||||
# 不同参数互不影响
|
||||
self.assertTrue(g.should_block("document_search", {"queries": ["a"]}))
|
||||
self.assertFalse(g.should_block("document_search", {"queries": ["b"]}))
|
||||
self.assertTrue(g.should_block("materials_library_search", {"queries": ["a"]}))
|
||||
self.assertFalse(g.should_block("materials_library_search", {"queries": ["b"]}))
|
||||
|
||||
|
||||
class TestErrStreak(unittest.TestCase):
|
||||
|
|
|
|||
|
|
@ -7,7 +7,11 @@ from types import SimpleNamespace
|
|||
from unittest.mock import patch
|
||||
from uuid import uuid4
|
||||
|
||||
from tools.paper_server import PaperFetchTool, PaperSearchTool, _media_url
|
||||
from platform_sources.paper_server import (
|
||||
PaperServerFetchTool,
|
||||
PaperServerSearchTool,
|
||||
_media_url,
|
||||
)
|
||||
from tools.run_python import RunPythonTool
|
||||
|
||||
|
||||
|
|
@ -22,16 +26,16 @@ class PaperServerToolTests(unittest.TestCase):
|
|||
|
||||
with (
|
||||
patch(
|
||||
"tools.paper_server._config",
|
||||
"platform_sources.paper_server._config",
|
||||
return_value=(
|
||||
"https://paper.test",
|
||||
"https://paper.test/api/resm/paper",
|
||||
"secret",
|
||||
),
|
||||
),
|
||||
patch("tools.paper_server._get_json", side_effect=fake_get_json),
|
||||
patch("platform_sources.paper_server._get_json", side_effect=fake_get_json),
|
||||
):
|
||||
result = PaperSearchTool().execute(publication_type="book", limit=20)
|
||||
result = PaperServerSearchTool().execute(publication_type="book", limit=20)
|
||||
|
||||
self.assertEqual(captured["type"], "book")
|
||||
self.assertEqual(captured["page_size"], 20)
|
||||
|
|
@ -41,22 +45,22 @@ class PaperServerToolTests(unittest.TestCase):
|
|||
def test_search_accepts_open_raw_type_without_closed_enum(self):
|
||||
with (
|
||||
patch(
|
||||
"tools.paper_server._config",
|
||||
"platform_sources.paper_server._config",
|
||||
return_value=(
|
||||
"https://paper.test",
|
||||
"https://paper.test/api/resm/paper",
|
||||
"secret",
|
||||
),
|
||||
),
|
||||
patch("tools.paper_server._get_json", return_value=[]) as request,
|
||||
patch("platform_sources.paper_server._get_json", return_value=[]) as request,
|
||||
):
|
||||
result = PaperSearchTool().execute(publication_type="future-type")
|
||||
result = PaperServerSearchTool().execute(publication_type="future-type")
|
||||
self.assertEqual(request.call_args.kwargs["params"]["type"], "future-type")
|
||||
self.assertEqual(result, "[]")
|
||||
|
||||
def test_search_rejects_unsafe_raw_type(self):
|
||||
with patch("tools.paper_server._get_json") as request:
|
||||
result = PaperSearchTool().execute(publication_type="book&api_key=leak")
|
||||
with patch("platform_sources.paper_server._get_json") as request:
|
||||
result = PaperServerSearchTool().execute(publication_type="book&api_key=leak")
|
||||
request.assert_not_called()
|
||||
self.assertTrue(result.startswith("[Error]"))
|
||||
|
||||
|
|
@ -70,7 +74,7 @@ class PaperServerToolTests(unittest.TestCase):
|
|||
|
||||
def test_fetch_schema_limits_format(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tool = PaperFetchTool(working_dir=Path(tmp))
|
||||
tool = PaperServerFetchTool(working_dir=Path(tmp))
|
||||
self.assertEqual(
|
||||
tool.parameters["properties"]["format"]["enum"], ["pdf", "xml"]
|
||||
)
|
||||
|
|
@ -97,9 +101,9 @@ class PaperServerToolTests(unittest.TestCase):
|
|||
}
|
||||
with (
|
||||
tempfile.TemporaryDirectory() as tmp,
|
||||
patch("tools.paper_server._get_paper", return_value=paper),
|
||||
patch("platform_sources.paper_server._get_paper", return_value=paper),
|
||||
patch(
|
||||
"tools.paper_server._config",
|
||||
"platform_sources.paper_server._config",
|
||||
return_value=(
|
||||
"https://paper.test",
|
||||
"https://paper.test/api/resm/paper",
|
||||
|
|
@ -107,10 +111,10 @@ class PaperServerToolTests(unittest.TestCase):
|
|||
),
|
||||
),
|
||||
patch(
|
||||
"tools.paper_server.httpx.stream", return_value=FakeStreamResponse()
|
||||
"platform_sources.paper_server.httpx.stream", return_value=FakeStreamResponse()
|
||||
) as stream,
|
||||
):
|
||||
tool = PaperFetchTool(working_dir=Path(tmp))
|
||||
tool = PaperServerFetchTool(working_dir=Path(tmp))
|
||||
first = tool.execute(id_or_doi="p1", format="pdf")
|
||||
second = tool.execute(id_or_doi="p1", format="pdf")
|
||||
destination = Path(tmp) / "papers" / "10.1_example.pdf"
|
||||
|
|
@ -195,8 +199,10 @@ class PaperServerToolTests(unittest.TestCase):
|
|||
enter_common_patches(stack)
|
||||
with_key = build_tools(ToolContext(**common))
|
||||
|
||||
self.assertNotIn("paper_search", without_key)
|
||||
self.assertTrue({"paper_search", "paper_get", "paper_fetch"}.issubset(with_key))
|
||||
self.assertNotIn("paper_server_search", without_key)
|
||||
self.assertTrue(
|
||||
{"paper_server_search", "paper_server_get", "paper_server_fetch"}.issubset(with_key)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -0,0 +1,206 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from core.executor_docker import _sandbox_env
|
||||
from platform_sources.registry import (
|
||||
MaterialsLibraryProvider,
|
||||
MaterialsProjectProvider,
|
||||
PaperServerProvider,
|
||||
PlatformSourceContext,
|
||||
build_platform_source_tools,
|
||||
)
|
||||
from platform_sources.materials_library import MaterialsLibrarySearchTool
|
||||
from platform_sources.materials_project import MaterialsProjectSearchTool
|
||||
from platform_sources.paper_server import PaperServerSearchTool
|
||||
from tools.run_python import RunPythonTool
|
||||
|
||||
|
||||
class _Provider:
|
||||
capabilities = frozenset({"test"})
|
||||
|
||||
def __init__(self, source_id, *, available=True, tools=None, failure=None):
|
||||
self.source_id = source_id
|
||||
self._available = available
|
||||
self._tools = list(tools or [])
|
||||
self._failure = failure
|
||||
|
||||
def available(self):
|
||||
if self._failure == "available":
|
||||
raise RuntimeError("secret-value")
|
||||
return self._available
|
||||
|
||||
def build_tools(self, context):
|
||||
if self._failure == "build":
|
||||
raise RuntimeError("secret-value")
|
||||
return self._tools
|
||||
|
||||
|
||||
class _Tool:
|
||||
name = "survivor"
|
||||
|
||||
|
||||
class PlatformSourceRegistryTests(unittest.TestCase):
|
||||
def _context(self, root: Path) -> PlatformSourceContext:
|
||||
return PlatformSourceContext(root, root, root)
|
||||
|
||||
def test_provider_failure_isolated_and_diagnostic_is_secret_free(self):
|
||||
with tempfile.TemporaryDirectory() as tmp, self.assertLogs(
|
||||
"platform_sources.registry", level="WARNING"
|
||||
) as logs:
|
||||
built = build_platform_source_tools(
|
||||
self._context(Path(tmp)),
|
||||
providers=(
|
||||
_Provider("bad_available", failure="available"),
|
||||
_Provider("bad_build", failure="build"),
|
||||
_Provider("good", tools=[_Tool()]),
|
||||
),
|
||||
)
|
||||
self.assertEqual([tool.name for tool in built], ["survivor"])
|
||||
output = "\n".join(logs.output)
|
||||
self.assertIn("bad_available", output)
|
||||
self.assertIn("bad_build", output)
|
||||
self.assertIn("RuntimeError", output)
|
||||
self.assertNotIn("secret-value", output)
|
||||
|
||||
def test_each_env_gate_is_independent(self):
|
||||
providers = (
|
||||
PaperServerProvider(),
|
||||
MaterialsLibraryProvider(),
|
||||
MaterialsProjectProvider(),
|
||||
)
|
||||
empty = {
|
||||
"PAPER_SERVER_API_KEY": "",
|
||||
"DOCUMENT_SEARCH_API_KEY": "",
|
||||
"MP_API_KEY": "",
|
||||
}
|
||||
with patch.dict(os.environ, empty, clear=False), patch(
|
||||
"platform_sources.registry.MPRester", object()
|
||||
):
|
||||
self.assertEqual([provider.available() for provider in providers], [False] * 3)
|
||||
for env_name, index in (
|
||||
("PAPER_SERVER_API_KEY", 0),
|
||||
("DOCUMENT_SEARCH_API_KEY", 1),
|
||||
("MP_API_KEY", 2),
|
||||
):
|
||||
values = dict(empty)
|
||||
values[env_name] = "configured"
|
||||
with patch.dict(os.environ, values, clear=False):
|
||||
available = [provider.available() for provider in providers]
|
||||
self.assertTrue(available[index])
|
||||
self.assertEqual(sum(available), 1)
|
||||
|
||||
def test_invalid_source_config_does_not_block_other_source(self):
|
||||
with tempfile.TemporaryDirectory() as tmp, patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"PAPER_SERVER_API_KEY": "paper-secret",
|
||||
"PAPER_SERVER_URL": "not-a-url",
|
||||
"DOCUMENT_SEARCH_API_KEY": "library-secret",
|
||||
"MP_API_KEY": "",
|
||||
},
|
||||
clear=False,
|
||||
), self.assertLogs("platform_sources.registry", level="WARNING") as logs:
|
||||
tools = build_platform_source_tools(self._context(Path(tmp)))
|
||||
names = {tool.name for tool in tools}
|
||||
self.assertEqual(
|
||||
names,
|
||||
{
|
||||
"materials_library_list",
|
||||
"materials_library_search",
|
||||
"materials_library_fetch",
|
||||
},
|
||||
)
|
||||
output = "\n".join(logs.output)
|
||||
self.assertIn("paper_server", output)
|
||||
self.assertIn("ValueError", output)
|
||||
self.assertNotIn("paper-secret", output)
|
||||
|
||||
def test_registered_tool_names_are_source_specific(self):
|
||||
with tempfile.TemporaryDirectory() as tmp, patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"PAPER_SERVER_API_KEY": "paper-secret",
|
||||
"DOCUMENT_SEARCH_API_KEY": "library-secret",
|
||||
"MP_API_KEY": "mp-secret",
|
||||
},
|
||||
clear=False,
|
||||
), patch("platform_sources.registry.MPRester", object()):
|
||||
tools = build_platform_source_tools(self._context(Path(tmp)))
|
||||
names = {tool.schema["function"]["name"] for tool in tools}
|
||||
self.assertEqual(
|
||||
names,
|
||||
{
|
||||
"paper_server_search",
|
||||
"paper_server_get",
|
||||
"paper_server_fetch",
|
||||
"materials_library_list",
|
||||
"materials_library_search",
|
||||
"materials_library_fetch",
|
||||
"materials_project_search",
|
||||
"materials_project_get_structure",
|
||||
"materials_project_get_entries",
|
||||
},
|
||||
)
|
||||
self.assertFalse(
|
||||
names
|
||||
& {
|
||||
"paper_search",
|
||||
"paper_get",
|
||||
"paper_fetch",
|
||||
"document_list_kb",
|
||||
"document_search",
|
||||
"document_download",
|
||||
"mp_search_summary",
|
||||
"mp_get_structure",
|
||||
"mp_get_entries",
|
||||
"platform_source_call",
|
||||
}
|
||||
)
|
||||
|
||||
def test_platform_credentials_do_not_enter_execution_environments(self):
|
||||
secrets = {
|
||||
"PAPER_SERVER_API_KEY": "paper-secret",
|
||||
"DOCUMENT_SEARCH_API_KEY": "library-secret",
|
||||
"MP_API_KEY": "mp-secret",
|
||||
}
|
||||
with patch.dict(os.environ, secrets, clear=False):
|
||||
host_env = RunPythonTool()._filtered_env()
|
||||
docker_env = _sandbox_env()
|
||||
for name in secrets:
|
||||
self.assertNotIn(name, host_env)
|
||||
self.assertNotIn(name, docker_env)
|
||||
|
||||
def test_platform_credentials_are_redacted_from_tool_results(self):
|
||||
secrets = {
|
||||
"PAPER_SERVER_API_KEY": "paper-secret",
|
||||
"DOCUMENT_SEARCH_API_KEY": "library-secret",
|
||||
"MP_API_KEY": "mp-secret",
|
||||
}
|
||||
with patch.dict(os.environ, secrets, clear=False), patch(
|
||||
"platform_sources.paper_server._config",
|
||||
side_effect=RuntimeError("paper-secret"),
|
||||
), patch(
|
||||
"platform_sources.materials_library.client.search",
|
||||
side_effect=RuntimeError("api_key=library-secret"),
|
||||
), patch(
|
||||
"platform_sources.materials_project._mpr",
|
||||
side_effect=RuntimeError("mp-secret"),
|
||||
):
|
||||
results = (
|
||||
PaperServerSearchTool().execute(keyword="cement"),
|
||||
MaterialsLibrarySearchTool().execute(queries=["cement"]),
|
||||
MaterialsProjectSearchTool().execute(formula="Ca3SiO5"),
|
||||
)
|
||||
joined = "\n".join(results)
|
||||
for value in secrets.values():
|
||||
self.assertNotIn(value, joined)
|
||||
self.assertIn("[REDACTED]", joined)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -10,9 +10,9 @@ from unittest.mock import patch
|
|||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
|
||||
class TestDocumentHostTools(unittest.TestCase):
|
||||
def test_document_search_truncates_content_without_requiring_key_arg(self):
|
||||
from tools.documents import DocumentSearchTool
|
||||
class TestMaterialsLibraryHostTools(unittest.TestCase):
|
||||
def test_materials_library_search_truncates_content_without_key_arg(self):
|
||||
from platform_sources.materials_library import MaterialsLibrarySearchTool
|
||||
|
||||
docs = [
|
||||
{
|
||||
|
|
@ -22,9 +22,9 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
"md_content": "A" * 200,
|
||||
}
|
||||
]
|
||||
with patch("tools.documents.doc_client.search", return_value=docs) as search:
|
||||
with patch("platform_sources.materials_library.client.search", return_value=docs) as search:
|
||||
# 单 query 批量:queries 列表只一条时,缩量逻辑不动用户给的参数
|
||||
out = DocumentSearchTool().execute(
|
||||
out = MaterialsLibrarySearchTool().execute(
|
||||
queries=["cement hydration"],
|
||||
max_documents=3,
|
||||
content_chars_per_doc=20,
|
||||
|
|
@ -41,8 +41,8 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
self.assertIn("A" * 20, out)
|
||||
self.assertIn("truncated", out)
|
||||
|
||||
def test_document_search_batches_queries_concurrently_and_dedups(self):
|
||||
from tools.documents import DocumentSearchTool
|
||||
def test_materials_library_search_batches_and_dedups(self):
|
||||
from platform_sources.materials_library import MaterialsLibrarySearchTool
|
||||
|
||||
calls: list[str] = []
|
||||
|
||||
|
|
@ -50,8 +50,8 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
calls.append(query)
|
||||
return [{"file_name": f"{query}.md", "kb_name": "mu_1", "md_content": "x"}]
|
||||
|
||||
with patch("tools.documents.doc_client.search", side_effect=fake_search):
|
||||
out = DocumentSearchTool().execute(
|
||||
with patch("platform_sources.materials_library.client.search", side_effect=fake_search):
|
||||
out = MaterialsLibrarySearchTool().execute(
|
||||
queries=["q1", "q2", "q1"], # 含重复 → 去重成 q1/q2
|
||||
)
|
||||
|
||||
|
|
@ -61,17 +61,17 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
self.assertIn("'q1'", out)
|
||||
self.assertIn("'q2'", out)
|
||||
|
||||
def test_document_download_uses_constructor_working_dir(self):
|
||||
from tools.documents import DocumentDownloadTool
|
||||
def test_materials_library_fetch_uses_constructor_working_dir(self):
|
||||
from platform_sources.materials_library import MaterialsLibraryFetchTool
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
working_dir = Path(tmp) / "task"
|
||||
working_dir.mkdir()
|
||||
with patch(
|
||||
"tools.documents.doc_client.download",
|
||||
"platform_sources.materials_library.client.download",
|
||||
return_value="documents/paper.pdf",
|
||||
) as download:
|
||||
tool = DocumentDownloadTool(
|
||||
tool = MaterialsLibraryFetchTool(
|
||||
working_dir=working_dir,
|
||||
base_dir=working_dir,
|
||||
user_root=Path(tmp),
|
||||
|
|
@ -86,8 +86,8 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
)
|
||||
self.assertIn("saved: task/documents/paper.pdf", out)
|
||||
|
||||
def test_document_download_batches_items_isolating_failure(self):
|
||||
from tools.documents import DocumentDownloadTool
|
||||
def test_materials_library_fetch_batches_items_isolating_failure(self):
|
||||
from platform_sources.materials_library import MaterialsLibraryFetchTool
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
working_dir = Path(tmp) / "task"
|
||||
|
|
@ -98,8 +98,8 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
raise RuntimeError("404")
|
||||
return f"documents/{file_name}"
|
||||
|
||||
with patch("tools.documents.doc_client.download", side_effect=fake_download):
|
||||
tool = DocumentDownloadTool(
|
||||
with patch("platform_sources.materials_library.client.download", side_effect=fake_download):
|
||||
tool = MaterialsLibraryFetchTool(
|
||||
working_dir=working_dir, base_dir=working_dir, user_root=Path(tmp)
|
||||
)
|
||||
out = tool.execute(items=[
|
||||
|
|
@ -114,8 +114,8 @@ class TestDocumentHostTools(unittest.TestCase):
|
|||
|
||||
|
||||
class TestMaterialsProjectHostTools(unittest.TestCase):
|
||||
def test_mp_search_summary_uses_host_key_and_returns_json(self):
|
||||
from tools.materials_project import MaterialsProjectSearchSummaryTool
|
||||
def test_materials_project_search_uses_host_key_and_returns_json(self):
|
||||
from platform_sources.materials_project import MaterialsProjectSearchTool
|
||||
|
||||
class FakeDoc:
|
||||
material_id = "mp-1"
|
||||
|
|
@ -145,10 +145,10 @@ class TestMaterialsProjectHostTools(unittest.TestCase):
|
|||
return False
|
||||
|
||||
with patch.dict("os.environ", {"MP_API_KEY": "host-secret"}, clear=False), patch(
|
||||
"tools.materials_project.MPRester",
|
||||
"platform_sources.materials_project.MPRester",
|
||||
FakeMPRester,
|
||||
):
|
||||
out = MaterialsProjectSearchSummaryTool().execute(
|
||||
out = MaterialsProjectSearchTool().execute(
|
||||
formula="Ca3SiO5",
|
||||
fields=["material_id", "formula_pretty", "energy_above_hull"],
|
||||
limit=2,
|
||||
|
|
@ -164,8 +164,8 @@ class TestMaterialsProjectHostTools(unittest.TestCase):
|
|||
self.assertEqual(captured["num_chunks"], 1)
|
||||
self.assertEqual(captured["chunk_size"], 2)
|
||||
|
||||
def test_mp_search_summary_batch_writes_file_and_isolates_failure(self):
|
||||
from tools.materials_project import MaterialsProjectSearchSummaryTool
|
||||
def test_materials_project_search_batch_writes_file_and_isolates_failure(self):
|
||||
from platform_sources.materials_project import MaterialsProjectSearchTool
|
||||
|
||||
class FakeDoc:
|
||||
def __init__(self, mid, formula, hull):
|
||||
|
|
@ -204,9 +204,9 @@ class TestMaterialsProjectHostTools(unittest.TestCase):
|
|||
wd = Path(td) / "proj"
|
||||
wd.mkdir()
|
||||
with patch.dict("os.environ", {"MP_API_KEY": "host-secret"}, clear=False), patch(
|
||||
"tools.materials_project.MPRester", FakeMPRester
|
||||
"platform_sources.materials_project.MPRester", FakeMPRester
|
||||
):
|
||||
tool = MaterialsProjectSearchSummaryTool(working_dir=wd)
|
||||
tool = MaterialsProjectSearchTool(working_dir=wd)
|
||||
out = tool.execute(
|
||||
formulas=["Ca3SiO5", "Ca3SiO5", "Empty1", "BadX"],
|
||||
fields=["material_id", "formula_pretty", "energy_above_hull"],
|
||||
|
|
|
|||
Loading…
Reference in New Issue