diff --git a/CHANGELOG.md b/CHANGELOG.md index ecd9fb9..60c8513 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,8 @@ - 专业软件任务完成后,输出文件会立即显示在当前对话的文件面板;任务中心收起时也不会再遮挡发送按钮。 - 专业软件生成的正式文件会保存在当前对话的正确目录并记录来源;运行参数和溯源信息收纳到隐藏元数据目录,减少产物列表噪声。 - 专业软件任务即使在完成回执时短暂断线,也不会持续占用 Windows Node;后续任务可继续执行,已完成结果会在后台自动确认。 +- Windows Node 窗口新增本机任务列表,可查看任务标题、输入文件、执行阶段、进度、时间、Job ID 和失败详情;任务状态会在本机持久保存,断线或重启后仍可查看。 +- Origin 绘图任务支持同时绑定多个 CSV、XLSX 或 JSON 输入、从不同输入中选择数据系列,并显式选择需要生成的工程文件和多种图形格式。 ## 0.65.2 — 2026-08-13 diff --git a/DESIGN.md b/DESIGN.md index 4a7d4a1..737eb3a 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -458,21 +458,21 @@ scheduled_jobs(§8.5) channel_bindings(§8.7,判别列+JSONB) ### 8.16 Windows Node 内网 MVP(implementation,2026-08-12) -第一阶段以 `docs/windows-node-mvp-intranet.md` 为实现契约:Windows Node 只作为受控执行节点,通过出站 HTTP/WS 主动连接 zcbot;首批能力固定为 `origin.plot@v1`。长期方案中的 mTLS、Service/DesktopRunner 双进程、完整租约与多节点调度暂不进入 MVP,但 URL path、Node ID、Bearer Header 和任务协议保留原位升级空间。 +第一阶段以 `docs/windows-node-mvp-intranet.md` 为实现契约:Windows Node 只作为受控执行节点,通过出站 HTTP/WS 主动连接 zcbot;首批能力固定为 Origin 绘图。长期方案中的 mTLS、Service/DesktopRunner 双进程、完整租约与多节点调度暂不进入 MVP,但 URL path、Node ID、Bearer Header 和任务协议保留原位升级空间。 云端控制面使用独立的 `software_node_enrollments` 与 `software_nodes`,不复用用户外部系统连接。管理员创建的一次性注册码具有 128 bit 随机熵,数据库只保存 SHA-256 摘要;节点注册在行锁事务中校验有效期、预期名称和允许能力,成功后原子消费。每个节点获得独立高熵 Token,数据库只保存 bcrypt 强哈希,明文仅在注册响应出现一次。 Node 通过 `Authorization: Bearer` 与 `X-Node-Id` 建立 `/v1/software-nodes/connect` WebSocket。进程内 Connection Manager 保证同一节点单活,新连接关闭旧连接;`hello`/`heartbeat` 更新版本、容量、软件健康与最后在线时间。管理员禁用节点时先持久化禁用态,再关闭现有连接;断线收尾不得覆盖禁用态。当前单活只覆盖单 Web 进程,生产启用多实例前必须增加 Redis/PG fencing 或将 Node API 固定路由到单一控制面实例。 -第二阶段已增加 `software_jobs`(专业软件任务)账本与 `origin.plot@v1` 的 offer/accept 骨架。用户只能在本人 task 下以幂等键提交固定 schema;云端规范化请求并记录 SHA-256,按当前进程真实在线、能力匹配、健康且有空闲 slot 的 Node 创建短期 offer。Node 再次校验 schema、图形类型和输出格式,使用 write-through、flush 与原子 rename 先落本机任务目录,再回 `job_accept`;重复 job 只有 digest 一致才接受。过期或发送失败的 offer 回到队列,lease、Node 和 digest 不匹配的响应被拒绝。Node 接收后云端进入 `dispatched` 而非 `running`,并将 slot 降为 0;只有固定 Worker 真正启动后才进入 `origin_running`。 +第二阶段已增加 `software_jobs`(专业软件任务)账本与 `origin.plot@v2` 的 offer/accept 骨架。用户只能在本人 task 下以幂等键提交固定 schema;云端规范化请求并记录 SHA-256,按当前进程真实在线、能力匹配、健康且有空闲 slot 的 Node 创建短期 offer。Node 再次校验 schema、图形类型和输出格式,使用 write-through、flush 与原子 rename 先落本机任务目录,再回 `job_accept`;重复 job 只有 digest 一致才接受。过期或发送失败的 offer 回到队列,lease、Node 和 digest 不匹配的响应被拒绝。Node 接收后云端进入 `dispatched` 而非 `running`,并将 slot 降为 0;只有固定 Worker 真正启动后才进入软件无关的 `software_running`,具体软件和操作由 capability/request 表达。 -第三阶段补齐输入下载与恢复状态协议:`input_id` 固定为 artifact UUID。已有 artifact 可直接提交;普通 task 文件先调用 `register_artifact(path)` 登记稳定身份,该动作不把文件发布为聊天交付物。提交时快照文件名、大小和 SHA-256,只允许 CSV/XLSX/JSON 且不超过 100 MiB。Node 以自身 Bearer 身份访问任务绑定的只读下载端点,流式写入本 job 的 `input/`,同时限制声明大小并校验 SHA-256,完成后原子 rename;不暴露工作区路径。Node 会原子读取/补报 `terminal.json`,断线后云端把活动任务标记 `disconnected` 并保留 Node/lease,重连按 job、lease、digest 恢复下载或幂等补报终态,不自动重派。 +第三阶段补齐多输入下载与恢复状态协议:请求使用通用 `inputs[]` 绑定 1–16 个 artifact,并由 `operation.plot.series[]` 以输入 key 引用各自的 X/Y 列;单文件不超过 100 MiB、总量不超过 512 MiB。已有 artifact 可直接提交;普通 task 文件先逐个调用 `register_artifact(path)` 登记稳定身份。Node 以自身 Bearer 身份访问每个任务绑定的只读下载端点,流式写入本 job 的 `input//`,同时校验大小与 SHA-256;不暴露工作区路径。Node 会原子读取/补报 `terminal.json`,断线后云端把活动任务标记 `disconnected` 并保留 Node/lease,重连按 job、lease、digest 恢复下载或幂等补报终态,不自动重派。Node UI 的“本机任务”只读取已经 accept 到本机的任务目录,不查询云端未派发 Job;每个任务以原子 `state.json` 持久化 `accepted/downloading_inputs/ready_to_run/software_running/uploading_outputs/succeeded/failed/cancelled` 通用阶段,窗口再与 request、terminal、upload-complete 合并成可恢复视图。 第四阶段落地固定 Origin Worker:Node 仅从管理员安装的固定 Python 运行时启动随程序发布的 `worker.py`,参数只有本机 job 目录;请求不能指定脚本、解释器或文件路径。Worker 使用 `originpro` 生成 OPJU、PNG、SVG、PDF、plot spec 和 provenance,校验产物签名并原子写入终态;当前受控图形仅含 line、scatter、line_scatter 和双栏出版布局。进程内 pipeline 按 job 去重,并脱离单次 WebSocket 的取消令牌运行;连接中断只延迟状态/终态上报。Node 进程若在 Worker 启动后重启,则保守失败而不重复驱动 Origin,避免无法证明的双执行。 第五阶段完成输出上传与发布:Node 只按固定 manifest ID 逐项流式 PUT,并携带 Node、lease、request digest 与内容摘要;云端重新绑定任务身份,不信 Node 提供的路径或媒体类型。文件先进入用户根下隐藏暂存区,固定文件名、单文件/总大小和 SHA-256 全部验证后,把 plot spec、provenance 整理进 `.meta/`,再将完整目录原子移动到 `/origin//`。PNG/SVG/PDF/OPJU 等正式输出登记平台 artifact UUID 和 `software_job_id`,`.meta/` 只落真实文件;成功状态返回 task-relative `output_dir`,Agent 以该目录为起点按需搜索。重复 PUT、complete 和重连均按摘要幂等;部分上传不可见,只有完整集合才能发布。Origin 执行槽与上传确认是两个正交状态:本地已有终态且固定 Worker 已退出时即释放软件执行槽,成功但尚无 `upload-complete.json` 的任务继续后台补传;若云端已经是 succeeded,重复 PUT/complete 必须按数据库持久化 manifest 校验并直接确认,不得按新版本目录规则重新发布旧 Job。 -第六阶段增加用户级 Job 中心与 Agent typed tools。`software_capability_list` 只暴露固定能力及当前在线空闲节点数,`software_job_submit/status/cancel` 在构造时绑定当前 user/task,模型不能跨用户或跨对话指定归属。`software_job_submit` 的模型可见唯一入口为 `input_id + plot + output`;`register_artifact` 是普通文件获得输入身份的唯一入口,并明确返回 UUID。内部完整 `request` 形式仅保留执行层兼容,不进入工具 schema。右下角 Job 中心按用户聚合各对话任务,活动期短轮询、空闲期降频;终态变化通知用户,成功任务可回到原对话发起分析。取消采用协作协议:未派发任务直接终止,已派发任务先进入 `cancelling`,云端通过 WebSocket 发送并在心跳时重放 `job_cancel`,Node 杀死固定 Worker 进程树后回报 `cancelled`;终态写入仍由云端账本裁决。 +第六阶段增加用户级 Job 中心与 Agent typed tools。`software_capability_list` 只暴露固定能力及当前在线空闲节点数,`software_job_submit/status/cancel` 在构造时绑定当前 user/task,模型不能跨用户或跨对话指定归属。`software_job_submit` 的唯一入口为通用 `inputs[] + operation + outputs[]`;输入和输出使用任务内稳定 key,具体 selector、type、format 和 options 由 capability 校验。`register_artifact` 是普通文件获得输入身份的唯一入口,并明确返回 UUID。右下角 Job 中心按用户聚合各对话任务,活动期短轮询、空闲期降频;终态变化通知用户,成功任务可回到原对话发起分析。取消采用协作协议:未派发任务直接终止,已派发任务先进入 `cancelling`,云端通过 WebSocket 发送并在心跳时重放 `job_cancel`,Node 杀死固定 Worker 进程树后回报 `cancelled`;终态写入仍由云端账本裁决。 后续仍需实现 Token 轮换;不得以任意命令或脚本接口临时代替。当前 Job 中心采用轮询而非用户事件推送,单活与 offer 选择仍只覆盖单 Web 进程;生产启用多实例前必须增加 Redis/PG fencing 或固定路由到单一控制面实例。 diff --git a/PROGRESS.md b/PROGRESS.md index 7b66b9a..b844f61 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -22,6 +22,8 @@ ### 2026-08-14 +- **08-14 / Unreleased / Windows Node 本机任务监控 + Origin 多输入多输出 v2**:配置窗口新增只读本机任务列表与详情,每秒从 `%ProgramData%/Zcbot/WindowsNode/jobs` 合并 request/state/terminal/upload-complete,展示标题、全部输入、阶段、执行耗时、时间、Job ID 与错误,重启后可恢复且不查询云端未派发 Job。按用户明确授权直接切换到 `origin.plot@v2`:通用 `inputs[] + operation + outputs[]` 支持 1–16 个输入、跨输入系列和多个显式产物,Node 下载到 keyed 目录,Origin Worker 建立多工作表后合并绘图;本地阶段统一为 `downloading_inputs/software_running`,不兼容 v1 请求或旧本地任务。相关专项 71 项 unittest、Python/JavaScript 语法、Ruff 致命规则、.NET build 与 diff 检查通过;完整测试 593 项中非数据库测试通过,3 个数据库集成模块因显式测试库未迁移、缺少 `users` 表而未通过,未连接或写入生产 DB。 + - **08-14 / Unreleased / 专业软件执行槽与上传恢复解耦**:修正服务端已完成但 Node 本地 `upload-complete.json` 未落盘时整节点长期上报 0 slot 的活性故障;Origin 终态且 Worker 退出后即释放执行槽,成功输出继续后台重传,服务端对 succeeded Job 的重复 PUT/complete 以已持久化 manifest 幂等确认并兼容旧目录布局。相关 43 项 unittest 与 .NET build 通过;只读核查生产任务状态,未写入生产 DB。 - **08-14 / Unreleased / 专业软件输出目录归一 + artifact 来源**:修正 `tasks.working_dir` 被二次拼入 user root 导致 Job 输出错位的问题,正式输出原子发布到 `/origin//` 并以 nullable `software_job_id` 记录来源;plot spec/provenance 改入 `.meta/` 且不再登记 artifact。新增 0033 migration 和只认显式迁移库地址、默认 dry-run、冲突即停的存量目录修复脚本;相关 Python 63 项、Node 前端 26 项、编译、Alembic 单 head、Ruff 致命规则及 diff 检查通过,未连接或写入生产 DB。 diff --git a/RUN.md b/RUN.md index b92d8ef..548a137 100644 --- a/RUN.md +++ b/RUN.md @@ -1100,7 +1100,7 @@ install-windows-node.bat 若 Python 未加入 PATH,可把绝对路径作为第一个参数,例如 `install-windows-node.bat "C:\Python312\python.exe"`。默认解释器为 `%ProgramData%\Zcbot\WindowsNode\runtimes\origin\Scripts\python.exe`。如需使用其他受管解释器,设置机器级 `ZCBOT_ORIGIN_PYTHON` 为绝对 `python.exe` 路径后重启 Node。`node.json`、可恢复任务和 runtime 集中保存在 `%ProgramData%\Zcbot\WindowsNode\`,不会因替换程序目录而丢失。运行时固定依赖见发布目录的 `origin-worker/requirements.txt`;任务请求无权选择解释器、脚本或路径。当前 Worker 支持 CSV/XLSX/JSON 输入,`line`、`scatter`、`line_scatter` 与 OPJU/PNG/SVG/PDF 输出。成功产物由 Node 流式上传,全部校验通过后发布到任务工作目录 `origin//`,plot spec 与 provenance 位于其 `.meta/`;上传中断会在重连时幂等续传。 -Web 用户登录后,文件栏 Job 中心会聚合本人最近任务。活动任务约 4 秒刷新一次,空闲时降为约 30 秒;停止已派发任务是协作取消,状态先显示“正在停止”,Node 在线时立即接收,断线后在下次连接或心跳时重放。Agent 可调用 `software_capability_list`、`register_artifact`、`software_job_submit`、`software_job_status` 和 `software_job_cancel`。Origin 输入必须是 artifact:已有 UUID 可直接提交,普通 task 文件先用相对路径登记;登记不会发布聊天交付卡片。提交工具接收 `input_id`、`plot`、`output`,只创建固定 schema 的持久任务,不会阻塞当前对话等待完成。成功状态提供 `output_dir`,Agent可在该目录内搜索并分析;正式输出的 artifact 带 `software_job_id`,供结果卡和产物详情展示来源。 +Web 用户登录后,文件栏 Job 中心会聚合本人最近任务。活动任务约 4 秒刷新一次,空闲时降为约 30 秒;停止已派发任务是协作取消,状态先显示“正在停止”,Node 在线时立即接收,断线后在下次连接或心跳时重放。Agent 可调用 `software_capability_list`、`register_artifact`、`software_job_submit`、`software_job_status` 和 `software_job_cancel`。Origin 输入必须是 artifact:已有 UUID 可直接提交,普通 task 文件先逐个用相对路径登记;登记不会发布聊天交付卡片。提交工具接收 `inputs`、`operation`、`outputs`,支持 1–16 个输入、跨输入系列和多个显式输出,只创建固定 v2 schema 的持久任务,不会阻塞当前对话等待完成。成功状态提供 `output_dir`,Agent可在该目录内搜索并分析;正式输出的 artifact 带 `software_job_id`,供结果卡和产物详情展示来源。 注册配置写入 `%ProgramData%\Zcbot\WindowsNode\node.json`;Token 使用 DPAPI `LocalMachine` 加密,ACL 仅允许注册账号和 `SYSTEM`。应始终用同一专用 Windows 账号执行统一安装器、注册并运行 Node。当前 MVP 以该账号的登录后计划任务启动,不安装 Windows Service。 diff --git a/core/software_jobs.py b/core/software_jobs.py index fa1e0c5..780ce53 100644 --- a/core/software_jobs.py +++ b/core/software_jobs.py @@ -19,9 +19,10 @@ OFFER_SECONDS = 60 ALLOWED_PLOT_TYPES = frozenset( {"line", "scatter", "line_scatter"} ) -ALLOWED_OUTPUT_FORMATS = frozenset({"opju", "png", "svg", "pdf"}) ALLOWED_INPUT_SUFFIXES = frozenset({".csv", ".xlsx", ".json"}) MAX_INPUT_BYTES = 100 * 1024 * 1024 +MAX_INPUTS = 16 +MAX_INPUT_TOTAL_BYTES = 512 * 1024 * 1024 MAX_OUTPUT_ARTIFACT_BYTES = 256 * 1024 * 1024 class SoftwareJobError(Exception): pass @@ -37,6 +38,12 @@ OUTPUT_ARTIFACTS = { "provenance": ("provenance.json", "application/json", None), } SOFTWARE_JOB_METADATA_IDS = frozenset({"plot_spec", "provenance"}) +ORIGIN_OUTPUT_IDENTITIES = { + ("project", "opju"): "project", + ("figure", "png"): "figure_png", + ("figure", "svg"): "figure_svg", + ("figure", "pdf"): "figure_pdf", +} def software_job_output_path(output_id: str) -> str: @@ -52,29 +59,50 @@ def _has_only(value: dict, fields: set[str]) -> bool: return set(value).issubset(fields) -def _canonical_request(request: dict) -> tuple[dict, str]: - if not isinstance(request, dict) or set(request) != {"schema_version", "input", "plot", "output"}: +def _canonical_origin_plot_request(request: dict) -> tuple[dict, str]: + if not isinstance(request, dict) or set(request) != { + "schema_version", "inputs", "operation", "outputs" + }: raise SoftwareJobError("invalid origin plot request fields") - if request.get("schema_version") != 1: + if request.get("schema_version") != 2: raise SoftwareJobError("unsupported origin plot schema version") - input_spec = request.get("input") - plot = request.get("plot") - output = request.get("output") - if not all(isinstance(item, dict) for item in (input_spec, plot, output)): + inputs = request.get("inputs") + operation = request.get("operation") + outputs = request.get("outputs") + if not isinstance(inputs, list) or not 1 <= len(inputs) <= MAX_INPUTS: + raise SoftwareJobError("inputs must contain 1 to 16 artifact bindings") + if not isinstance(operation, dict) or set(operation) != {"plot"}: + raise SoftwareJobError("origin operation must contain exactly plot") + plot = operation.get("plot") + if not isinstance(plot, dict): raise SoftwareJobError("origin plot request sections must be objects") - if not _has_only(input_spec, {"input_id", "sheet"}): - raise SoftwareJobError("unsupported origin input fields") - try: - UUID(str(input_spec.get("input_id") or "")) - except ValueError as exc: - raise SoftwareJobError("input.input_id must be an artifact UUID") from exc - if "sheet" in input_spec and ( - not isinstance(input_spec["sheet"], str) or not 1 <= len(input_spec["sheet"]) <= 128 - ): - raise SoftwareJobError("input.sheet must be a string") + input_keys: list[str] = [] + for input_spec in inputs: + if not isinstance(input_spec, dict) or set(input_spec) not in ( + {"key", "artifact_id"}, {"key", "artifact_id", "selector"} + ): + raise SoftwareJobError("invalid origin input binding fields") + input_key = input_spec.get("key") + if not isinstance(input_key, str) or not re.fullmatch(r"[a-z][a-z0-9_]{0,31}", input_key): + raise SoftwareJobError("input key must match [a-z][a-z0-9_]{0,31}") + try: + UUID(str(input_spec.get("artifact_id") or "")) + except ValueError as exc: + raise SoftwareJobError("inputs[].artifact_id must be an artifact UUID") from exc + selector = input_spec.get("selector") + if selector is not None and ( + not isinstance(selector, dict) + or set(selector) != {"sheet"} + or not isinstance(selector.get("sheet"), str) + or not 1 <= len(selector["sheet"]) <= 128 + ): + raise SoftwareJobError("origin input selector must contain a valid sheet") + input_keys.append(input_key) + if len(input_keys) != len(set(input_keys)): + raise SoftwareJobError("input keys must be unique") if not _has_only( plot, - {"type", "x", "y", "template", "title", "x_axis", "y_axis", "legend", "error_bars"}, + {"type", "series", "template", "title", "x_axis", "y_axis", "legend", "error_bars"}, ): raise SoftwareJobError("unsupported origin plot fields") if plot.get("type") not in ALLOWED_PLOT_TYPES: @@ -85,19 +113,42 @@ def _canonical_request(request: dict) -> tuple[dict, str]: raise SoftwareJobError("plot.title must be a string") if plot.get("template", "publication_double_column") != "publication_double_column": raise SoftwareJobError("unsupported origin plot template") - x_column = plot.get("x") - y_columns = plot.get("y") - if not isinstance(x_column, str) or not 1 <= len(x_column) <= 128: - raise SoftwareJobError("plot.x must be a column name") - if isinstance(y_columns, str): - y_columns = [y_columns] - if ( - not isinstance(y_columns, list) - or not 1 <= len(y_columns) <= 16 - or len(y_columns) != len(set(y_columns)) - or any(not isinstance(item, str) or not 1 <= len(item) <= 128 for item in y_columns) - ): - raise SoftwareJobError("plot.y must contain 1 to 16 unique column names") + series = plot.get("series") + if not isinstance(series, list) or not 1 <= len(series) <= 16: + raise SoftwareJobError("plot.series must contain 1 to 16 series") + identities: list[tuple[str, str, str]] = [] + used_input_keys: set[str] = set() + series_labels: dict[tuple[str, str], str] = {} + for item in series: + if not isinstance(item, dict) or not _has_only(item, {"input", "x", "y", "label"}): + raise SoftwareJobError("invalid plot series fields") + if not {"input", "x", "y"}.issubset(item): + raise SoftwareJobError("plot series requires input, x, and y") + input_key = item.get("input") + x_column = item.get("x") + y_column = item.get("y") + if input_key not in input_keys: + raise SoftwareJobError("plot series references an unknown input") + if any( + not isinstance(value, str) or not 1 <= len(value) <= 128 + for value in (x_column, y_column) + ): + raise SoftwareJobError("plot series x and y must be column names") + if "label" in item and ( + not isinstance(item["label"], str) or not 1 <= len(item["label"]) <= 200 + ): + raise SoftwareJobError("plot series label must be a string") + label_key = (input_key, y_column) + effective_label = item.get("label", y_column) + if label_key in series_labels and series_labels[label_key] != effective_label: + raise SoftwareJobError("series sharing an input Y column must use one label") + series_labels[label_key] = effective_label + used_input_keys.add(input_key) + identities.append((input_key, x_column, y_column)) + if len(identities) != len(set(identities)): + raise SoftwareJobError("plot series must be unique") + if used_input_keys != set(input_keys): + raise SoftwareJobError("every input must be referenced by a plot series") for axis_name in ("x_axis", "y_axis"): axis = plot.get(axis_name) if axis is not None and ( @@ -120,27 +171,37 @@ def _canonical_request(request: dict) -> tuple[dict, str]: ): raise SoftwareJobError("invalid plot.legend") if plot.get("error_bars") is not None: - raise SoftwareJobError("error bars are not supported in origin.plot@v1") - if not _has_only(output, {"formats", "dpi", "capture_screenshots", "record_video"}): - raise SoftwareJobError("unsupported origin output fields") - if any( - name in output and not isinstance(output[name], bool) - for name in ("capture_screenshots", "record_video") - ): - raise SoftwareJobError("origin output capture flags must be boolean") - if output.get("record_video", False): - raise SoftwareJobError("origin video recording is not supported") - dpi = output.get("dpi", 300) - if not isinstance(dpi, int) or isinstance(dpi, bool) or not 72 <= dpi <= 1200: - raise SoftwareJobError("output.dpi must be between 72 and 1200") - formats = output.get("formats") - if ( - not isinstance(formats, list) - or not formats - or len(formats) != len(set(formats)) - or any(item not in ALLOWED_OUTPUT_FORMATS for item in formats) - ): - raise SoftwareJobError("output.formats contains unsupported values") + raise SoftwareJobError("error bars are not supported") + if not isinstance(outputs, list) or not 1 <= len(outputs) <= 16: + raise SoftwareJobError("outputs must contain 1 to 16 declarations") + output_keys: list[str] = [] + output_identities: list[tuple[str, str]] = [] + for output in outputs: + if not isinstance(output, dict) or set(output) not in ( + {"key", "type", "format"}, {"key", "type", "format", "options"} + ): + raise SoftwareJobError("invalid origin output declaration fields") + output_type = output.get("type") + output_format = output.get("format") + expected_key = ORIGIN_OUTPUT_IDENTITIES.get((output_type, output_format)) + if output.get("key") != expected_key: + raise SoftwareJobError("origin output key, type, and format do not match") + options = output.get("options") + if output_format == "png": + if options is not None and ( + not isinstance(options, dict) + or set(options) != {"dpi"} + or not isinstance(options.get("dpi"), int) + or isinstance(options.get("dpi"), bool) + or not 72 <= options["dpi"] <= 1200 + ): + raise SoftwareJobError("PNG output options must contain a valid dpi") + elif options is not None: + raise SoftwareJobError("output options are only supported for PNG") + output_keys.append(expected_key) + output_identities.append((output_type, output_format)) + if len(output_keys) != len(set(output_keys)) or len(output_identities) != len(set(output_identities)): + raise SoftwareJobError("outputs must be unique") encoded = json.dumps(request, ensure_ascii=False, sort_keys=True, separators=(",", ":")) if len(encoded.encode("utf-8")) > 256 * 1024: raise SoftwareJobError("origin plot request is too large") @@ -148,6 +209,18 @@ def _canonical_request(request: dict) -> tuple[dict, str]: return normalized, sha256(encoded.encode("utf-8")).hexdigest() +REQUEST_VALIDATORS = { + "origin.plot@v2": _canonical_origin_plot_request, +} + + +def _canonical_request(capability: str, request: dict) -> tuple[dict, str]: + validator = REQUEST_VALIDATORS.get(capability) + if validator is None: + raise SoftwareJobError("unsupported capability") + return validator(request) + + def _job_dict(row: SoftwareJob) -> dict: return { "job_id": str(row.job_id), @@ -233,16 +306,16 @@ def list_jobs( def _request_summary(job: SoftwareJob) -> dict: - plot = job.request.get("plot") or {} - output = job.request.get("output") or {} + plot = (job.request.get("operation") or {}).get("plot") or {} + outputs = job.request.get("outputs") or [] return { "display_name": ( "Origin 科研绘图" - if job.capability == "origin.plot@v1" + if job.capability == "origin.plot@v2" else job.capability ), "title": str(plot.get("title") or ""), - "formats": list(output.get("formats") or []), + "formats": [item.get("format") for item in outputs if isinstance(item, dict)], } @@ -309,40 +382,55 @@ def create_job( raise SoftwareJobError("idempotency_key must contain 1 to 200 characters") if capability not in SUPPORTED_CAPABILITIES: raise SoftwareJobError("unsupported capability") - normalized, digest = _canonical_request(request) + normalized, digest = _canonical_request(capability, request) with session_scope() as session: task = session.execute( select(Task.task_id).where(Task.task_id == task_id, Task.user_id == user_id) ).first() if task is None: raise SoftwareJobError("task not found") - artifact_id = UUID(normalized["input"]["input_id"]) - artifact = session.execute( + artifact_ids = [UUID(item["artifact_id"]) for item in normalized["inputs"]] + artifacts = session.execute( select(Artifact).where( - Artifact.artifact_id == artifact_id, + Artifact.artifact_id.in_(set(artifact_ids)), Artifact.user_id == user_id, Artifact.status == "active", ) - ).scalar_one_or_none() - if artifact is None: - raise SoftwareJobError("input artifact not found") - suffix = "." + artifact.current_path.rsplit(".", 1)[-1].lower() if "." in artifact.current_path else "" - if suffix not in ALLOWED_INPUT_SUFFIXES: - raise SoftwareJobError("input artifact type is not supported") - if ( - artifact.size_bytes is None - or artifact.size_bytes < 0 - or artifact.size_bytes > MAX_INPUT_BYTES - or not artifact.content_sha256 - or len(artifact.content_sha256) != 64 - ): - raise SoftwareJobError("input artifact metadata is incomplete or too large") - input_manifest = { - "artifact_id": str(artifact.artifact_id), - "filename": artifact.current_path.replace("\\", "/").rsplit("/", 1)[-1], - "size_bytes": artifact.size_bytes, - "sha256": artifact.content_sha256, - } + ).scalars().all() + artifacts_by_id = {artifact.artifact_id: artifact for artifact in artifacts} + input_manifest: list[dict] = [] + total_input_bytes = 0 + for binding, artifact_id in zip(normalized["inputs"], artifact_ids, strict=True): + artifact = artifacts_by_id.get(artifact_id) + if artifact is None: + raise SoftwareJobError("input artifact not found") + suffix = ( + "." + artifact.current_path.rsplit(".", 1)[-1].lower() + if "." in artifact.current_path else "" + ) + if suffix not in ALLOWED_INPUT_SUFFIXES: + raise SoftwareJobError("input artifact type is not supported") + if ( + artifact.size_bytes is None + or artifact.size_bytes < 0 + or artifact.size_bytes > MAX_INPUT_BYTES + or not artifact.content_sha256 + or len(artifact.content_sha256) != 64 + ): + raise SoftwareJobError("input artifact metadata is incomplete or too large") + total_input_bytes += artifact.size_bytes + item = { + "key": binding["key"], + "artifact_id": str(artifact.artifact_id), + "filename": artifact.current_path.replace("\\", "/").rsplit("/", 1)[-1], + "size_bytes": artifact.size_bytes, + "sha256": artifact.content_sha256, + } + if binding.get("selector") is not None: + item["selector"] = binding["selector"] + input_manifest.append(item) + if total_input_bytes > MAX_INPUT_TOTAL_BYTES: + raise SoftwareJobError("job inputs exceed the total size limit") existing = session.execute( select(SoftwareJob).where( SoftwareJob.user_id == user_id, @@ -471,15 +559,20 @@ def offer_next_job(node_ids: set[UUID]) -> dict | None: "capability": job.capability, "request_digest": job.request_digest, "request": job.request, - "input_transfer": { - **job.input_manifest, - "download_path": f"/v1/software-jobs/{job.job_id}/input", - }, + "input_transfers": [ + { + **item, + "download_path": ( + f"/v1/software-jobs/{job.job_id}/inputs/{item['key']}" + ), + } + for item in job.input_manifest + ], }, } -def get_job_input(node_id: UUID, job_id: UUID) -> dict | None: +def get_job_input(node_id: UUID, job_id: UUID, input_key: str) -> dict | None: """返回任务绑定的 artifact 定位信息;调用方仍需在 user_root 内安全解析。""" with session_scope() as session: job = session.execute( @@ -491,7 +584,13 @@ def get_job_input(node_id: UUID, job_id: UUID) -> dict | None: ).scalar_one_or_none() if job is None: return None - artifact_id = UUID(job.input_manifest["artifact_id"]) + manifest = next( + (item for item in job.input_manifest if item.get("key") == input_key), + None, + ) + if manifest is None: + return None + artifact_id = UUID(manifest["artifact_id"]) artifact = session.execute( select(Artifact).where( Artifact.artifact_id == artifact_id, @@ -504,7 +603,7 @@ def get_job_input(node_id: UUID, job_id: UUID) -> dict | None: return { "user_id": job.user_id, "current_path": artifact.current_path, - **job.input_manifest, + **manifest, } @@ -539,12 +638,11 @@ def get_job_output_context(node_id: UUID, job_id: UUID, lease_id: UUID, digest: def validate_output_manifest(request: dict, manifest: object) -> list[dict]: if not isinstance(manifest, list): raise SoftwareJobError("job artifact manifest must be a list") - requested_formats = set(request.get("output", {}).get("formats") or []) expected_ids = {"plot_spec", "provenance"} expected_ids.update( - artifact_id - for artifact_id, (_, _, output_format) in OUTPUT_ARTIFACTS.items() - if output_format in requested_formats + item.get("key") + for item in request.get("outputs", []) + if isinstance(item, dict) and isinstance(item.get("key"), str) ) if len(manifest) != len(expected_ids): raise SoftwareJobError("job artifact manifest is incomplete") @@ -717,7 +815,7 @@ def update_job_state(node_id: UUID, payload: dict) -> None: raise SoftwareJobError("job state cannot advance from its current status") job.status = ( "dispatched" - if stage in {"accepted", "waiting_input", "ready_to_run"} + if stage in {"accepted", "downloading_inputs", "ready_to_run"} else "running" ) job.stage = stage diff --git a/core/software_nodes.py b/core/software_nodes.py index f5e49fd..595c0cb 100644 --- a/core/software_nodes.py +++ b/core/software_nodes.py @@ -13,7 +13,7 @@ from sqlalchemy import select from core.storage.engine import session_scope from core.storage.models import SoftwareNode, SoftwareNodeEnrollment -SUPPORTED_CAPABILITIES = frozenset({"origin.plot@v1"}) +SUPPORTED_CAPABILITIES = frozenset({"origin.plot@v2"}) MAX_ENROLLMENT_FAILURES = 5 @@ -44,7 +44,7 @@ def create_enrollment( capabilities: list[str] | None = None, ttl_seconds: int = 600, ) -> dict: - allowed = list(dict.fromkeys(capabilities or ["origin.plot@v1"])) + allowed = list(dict.fromkeys(capabilities or ["origin.plot@v2"])) if not allowed or any(item not in SUPPORTED_CAPABILITIES for item in allowed): raise SoftwareNodeError("unsupported capability") if not 60 <= ttl_seconds <= 3600: diff --git a/docs/windows-node-design.md b/docs/windows-node-design.md index a8b0886..3130315 100644 --- a/docs/windows-node-design.md +++ b/docs/windows-node-design.md @@ -435,39 +435,46 @@ C# COM 只作未覆盖能力或旧版本备用,并与实际 Origin 版本匹 ### 12.2 首期 capability -只开放 `origin.plot@v1`,接受 CSV、XLSX 或规范化 JSON,不接受任意 Python、LabTalk、模板文件或宏。 +只开放 `origin.plot@v2`,接受 1–16 个 CSV、XLSX 或规范化 JSON 输入,不接受任意 Python、LabTalk、模板文件或宏。 ```json { - "schema_version": 1, - "input": {"input_id": "5af18255-c859-43f3-a424-220bc68b7db2", "sheet": "Sheet1"}, - "plot": { - "type": "line_scatter", - "x": "Temperature", - "y": ["Strength_7d", "Strength_28d"], - "template": "publication_double_column", - "title": "温度对抗压强度的影响", - "x_axis": {"title": "温度", "unit": "°C", "scale": "linear"}, - "y_axis": {"title": "抗压强度", "unit": "MPa", "scale": "linear"}, - "legend": {"enabled": true, "position": "top_right"}, - "error_bars": null + "schema_version": 2, + "inputs": [ + {"key": "seven_day", "artifact_id": "5af18255-c859-43f3-a424-220bc68b7db2", "selector": {"sheet": "Sheet1"}}, + {"key": "twenty_eight_day", "artifact_id": "6bf29366-d960-54f4-b535-331cd79c8ec3"} + ], + "operation": { + "plot": { + "type": "line_scatter", + "series": [ + {"input": "seven_day", "x": "Temperature", "y": "Strength", "label": "7 d"}, + {"input": "twenty_eight_day", "x": "Temperature", "y": "Strength", "label": "28 d"} + ], + "template": "publication_double_column", + "title": "温度对抗压强度的影响", + "x_axis": {"title": "温度", "unit": "°C", "scale": "linear"}, + "y_axis": {"title": "抗压强度", "unit": "MPa", "scale": "linear"}, + "legend": {"enabled": true, "position": "top_right"}, + "error_bars": null + } }, - "output": { - "formats": ["opju", "png", "svg", "pdf"], - "dpi": 600, - "capture_screenshots": true, - "record_video": false - } + "outputs": [ + {"key": "project", "type": "project", "format": "opju"}, + {"key": "figure_png", "type": "figure", "format": "png", "options": {"dpi": 600}}, + {"key": "figure_svg", "type": "figure", "format": "svg"}, + {"key": "figure_pdf", "type": "figure", "format": "pdf"} + ] } ``` 当前图形类型:line、scatter、line_scatter。后续增加 grouped_bar、box、histogram、heatmap、误差棒组合、三元图、等高线、三维曲面、XRD 堆叠图、热分析联图和多面板布局。 上述 JSON 同时是提交工具与云端账本、Windows Node 之间的 artifact 输入契约。 -Agent 优先复用已有 artifact UUID;普通 task 文件先调用 -`register_artifact(path)` 获得 UUID,再以 `input_id`、`plot`、`output` 调用 -`software_job_submit`。登记输入不会把文件发布为聊天交付物。旧的完整 `request` -参数仅作为执行层兼容入口保留,不暴露在模型可见 schema 中。 +Agent 优先复用已有 artifact UUID;普通 task 文件先逐个调用 +`register_artifact(path)` 获得 UUID,再以 `inputs`、`operation`、`outputs` 调用 +`software_job_submit`。登记输入不会把文件发布为聊天交付物。v2 是当前唯一协议, +不读取 v1 请求或旧本地任务格式。 ### 12.3 模板与产物 @@ -476,7 +483,7 @@ Agent 优先复用已有 artifact UUID;普通 task 文件先调用 ```text D:\ZcbotNode\jobs\\ ├── request\request.json -├── input\ +├── input\\ ├── work\ ├── logs\events.jsonl ├── screenshots\ @@ -607,7 +614,7 @@ Windows 基线:固定版本并受控更新;禁止休眠和自动锁屏;固 ## 18. 首期验收标准 1. Windows 不开放业务入站端口,Node 能通过 mTLS WSS 注册、重连和心跳。 -2. 云端能发现 `origin.plot@v1`、软件版本、健康和 slot。 +2. 云端能发现 `origin.plot@v2`、软件版本、健康和 slot。 3. 同一幂等键不会创建两个任务。 4. 双连接时只有最新 epoch 能接受任务。 5. zcbot 能用 CSV 生成指定模板图并获得 `job_id`。 diff --git a/docs/windows-node-mvp-intranet.md b/docs/windows-node-mvp-intranet.md index 51e7b7e..e4616a1 100644 --- a/docs/windows-node-mvp-intranet.md +++ b/docs/windows-node-mvp-intranet.md @@ -2,7 +2,7 @@ > **当前有效的第一阶段开发依据。**本文件取代 `windows-node-mvp.md` 中的 MVP 通信与注册方案。 > 长期演进边界见 `windows-node-design.md`。 -> 首批能力:`origin.plot@v1`。 +> 首批能力:`origin.plot@v2`。 ## 1. 适用边界 @@ -82,7 +82,7 @@ Content-Type: application/json "install_id": "019...", "node_version": "0.1.0", "os_version": "Windows 11 Enterprise 24H2", - "capabilities": ["origin.plot@v1"] + "capabilities": ["origin.plot@v2"] } ``` @@ -200,12 +200,12 @@ Node 断线且本地任务可能仍在执行时标记 `disconnected`,不得自 ## 7. Origin 任务闭环 -当前实现进度:云端任务账本、幂等提交、短期 offer、Node 本地原子保存与 accept/reject 已落地。输入以任务绑定的 artifact UUID 下载,Node 流式校验大小和 SHA-256 后原子保存。固定 Worker 使用管理员安装的隔离 Python 运行时与随程序发布的 `worker.py` 驱动 Origin,生成 OPJU、PNG、SVG、PDF、plot spec、provenance 和原子 `terminal.json`;运行不绑定单次 WebSocket,断线后继续执行。同一进程按 job 去重,Node 重启后不重复启动已留启动标记但无可信终态的任务。成功产物逐项流式上传到云端隐藏暂存区,云端复核任务身份、固定文件名、大小和 SHA-256 后,一次性发布到 `/origin//` 并登记平台 artifact UUID;Node 以 `upload-complete.json` 恢复中断上传。 +当前实现进度:云端任务账本、幂等提交、短期 offer、Node 本地原子保存与 accept/reject 已落地。请求以 `inputs[]` 绑定多个 artifact、以 `outputs[]` 声明多个交付物,Node 逐项流式校验大小和 SHA-256 后将输入保存到 `input//`。固定 Worker 使用管理员安装的隔离 Python 运行时与随程序发布的 `worker.py` 驱动 Origin,按 `operation.plot.series[]` 从不同输入选择系列,并按输出声明生成 OPJU、PNG、SVG、PDF;plot spec、provenance 和原子 `terminal.json` 是系统强制元数据。运行不绑定单次 WebSocket,断线后继续执行。同一进程按 job 去重,Node 重启后不重复启动已留启动标记但无可信终态的任务。成功产物逐项流式上传到云端隐藏暂存区,云端复核任务身份、固定文件名、大小和 SHA-256 后,一次性发布到 `/origin//` 并登记平台 artifact UUID;Node 以 `upload-complete.json` 恢复中断上传。 ```text -用户上传 CSV/XLSX +用户上传一个或多个 CSV/XLSX/JSON → zcbot 生成受控 plot spec -→ Node 接收 origin.plot@v1 +→ Node 接收 origin.plot@v2 → 先持久化 job,再启动 Origin Worker → originpro 生成 OPJU/PNG/SVG/PDF → terminal.json 原子记录终态 diff --git a/tests/test_origin_worker.py b/tests/test_origin_worker.py index af1770e..a8c5168 100644 --- a/tests/test_origin_worker.py +++ b/tests/test_origin_worker.py @@ -6,7 +6,6 @@ import tempfile import unittest from pathlib import Path - WORKER_PATH = ( Path(__file__).resolve().parents[1] / "windows-node" / "origin-worker" / "worker.py" ) @@ -41,6 +40,39 @@ class OriginWorkerUnitTests(unittest.TestCase): self.assertEqual(worker._axis_title({"title": "Stress", "unit": "MPa"}, "Y"), "Stress (MPa)") self.assertEqual(worker._axis_title(None, "Time"), "Time") + def test_keyed_input_directory_requires_exactly_one_file(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + keyed = root / "input" / "sample" + keyed.mkdir(parents=True) + path = keyed / "data.csv" + path.write_text("x,y\n1,2\n", encoding="utf-8") + self.assertEqual(worker._input_file(root, "sample"), path) + (keyed / "extra.csv").write_text("x,y\n3,4\n", encoding="utf-8") + with self.assertRaisesRegex(ValueError, "INPUT_FILE_COUNT_INVALID:sample"): + worker._input_file(root, "sample") + + def test_series_are_resolved_against_each_input(self) -> None: + input_data = { + "first": (["time", "strength"], [[1, 10]]), + "second": (["temperature", "value"], [[20, 30]]), + } + series = [ + {"input": "first", "x": "time", "y": "strength", "label": "7 d"}, + {"input": "second", "x": "temperature", "y": "value", "label": "28 d"}, + ] + resolved, labels = worker._resolve_series(input_data, series) + self.assertEqual(resolved, [("first", 0, 1, "7 d"), ("second", 0, 1, "28 d")]) + self.assertEqual(labels, {("first", 1): "7 d", ("second", 1): "28 d"}) + + def test_shared_y_column_rejects_conflicting_labels(self) -> None: + input_data = {"sample": (["x1", "x2", "y"], [[1, 2, 3]])} + with self.assertRaisesRegex(ValueError, "SERIES_LABEL_CONFLICT"): + worker._resolve_series(input_data, [ + {"input": "sample", "x": "x1", "y": "y", "label": "First"}, + {"input": "sample", "x": "x2", "y": "y", "label": "Second"}, + ]) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_software_job_tools.py b/tests/test_software_job_tools.py index ad0523e..a42aa59 100644 --- a/tests/test_software_job_tools.py +++ b/tests/test_software_job_tools.py @@ -22,12 +22,12 @@ class SoftwareJobToolTests(unittest.TestCase): nodes = [ { "status": "online", - "capabilities": ["origin.plot@v1"], + "capabilities": ["origin.plot@v2"], "runtime": {"available_slots": 1}, }, { "status": "offline", - "capabilities": ["origin.plot@v1"], + "capabilities": ["origin.plot@v2"], "runtime": {"available_slots": 1}, }, ] @@ -43,36 +43,53 @@ class SoftwareJobToolTests(unittest.TestCase): tool = SoftwareJobSubmitTool(self.user_id, self.task_id) with patch("tools.software_jobs.create_job", return_value=(created, True)) as create: result = json.loads(tool.execute( - "origin.plot@v1", - input_id=str(artifact_id), - plot={"type": "scatter", "x": "x", "y": ["y"]}, - output={"formats": ["opju", "png"], "dpi": 300}, + "origin.plot@v2", + inputs=[{"key": "sample", "artifact_id": str(artifact_id)}], + operation={"plot": { + "type": "scatter", + "series": [{"input": "sample", "x": "x", "y": "y"}], + }}, + outputs=[ + {"key": "project", "type": "project", "format": "opju"}, + {"key": "figure_png", "type": "figure", "format": "png", "options": {"dpi": 300}}, + ], )) self.assertTrue(result["created"]) self.assertEqual(create.call_args.args[:2], (self.user_id, self.task_id)) - self.assertEqual(create.call_args.kwargs["capability"], "origin.plot@v1") + self.assertEqual(create.call_args.kwargs["capability"], "origin.plot@v2") self.assertEqual( create.call_args.kwargs["request"], { - "schema_version": 1, - "input": {"input_id": str(artifact_id)}, - "plot": {"type": "scatter", "x": "x", "y": ["y"]}, - "output": {"formats": ["opju", "png"], "dpi": 300}, + "schema_version": 2, + "inputs": [{"key": "sample", "artifact_id": str(artifact_id)}], + "operation": {"plot": { + "type": "scatter", + "series": [{"input": "sample", "x": "x", "y": "y"}], + }}, + "outputs": [ + {"key": "project", "type": "project", "format": "opju"}, + {"key": "figure_png", "type": "figure", "format": "png", "options": {"dpi": 300}}, + ], }, ) def test_submit_requires_artifact_uuid(self): result = SoftwareJobSubmitTool(self.user_id, self.task_id).execute( - "origin.plot@v1", - input_id="data/input.csv", - plot={"type": "scatter", "x": "x", "y": ["y"]}, - output={"formats": ["png"]}, + "origin.plot@v2", + inputs=[{"key": "sample", "artifact_id": "data/input.csv"}], + operation={"plot": { + "type": "scatter", + "series": [{"input": "sample", "x": "x", "y": "y"}], + }}, + outputs=[{"key": "figure_png", "type": "figure", "format": "png"}], ) self.assertIn("call register_artifact first", result) def test_submit_schema_requires_artifact_input(self): required = SoftwareJobSubmitTool.parameters["required"] - self.assertIn("input_id", required) + self.assertIn("inputs", required) + self.assertIn("operation", required) + self.assertIn("outputs", required) self.assertNotIn("input_path", SoftwareJobSubmitTool.parameters["properties"]) self.assertNotIn("request", SoftwareJobSubmitTool.parameters["properties"]) diff --git a/tests/test_software_nodes.py b/tests/test_software_nodes.py index 4f7868a..3fc8d9a 100644 --- a/tests/test_software_nodes.py +++ b/tests/test_software_nodes.py @@ -233,7 +233,7 @@ class SoftwareJobProtocolTests(unittest.TestCase): session = session_scope.return_value.__enter__.return_value job = type("Job", (), {})() job.job_id = uuid4(); job.user_id = uuid4(); job.task_id = uuid4() - job.capability = "origin.plot@v1"; job.request_digest = "a" * 64 + job.capability = "origin.plot@v2"; job.request_digest = "a" * 64 job.node_id = None; job.lease_id = None; job.status = "queued"; job.stage = "" job.progress = 0; job.metrics = {}; job.error = {}; job.artifact_manifest = [] job.created_at = None; job.started_at = None; job.terminal_at = None @@ -248,9 +248,9 @@ class SoftwareJobProtocolTests(unittest.TestCase): session = session_scope.return_value.__enter__.return_value job = type("Job", (), {})() job.job_id = uuid4(); job.user_id = uuid4(); job.task_id = uuid4() - job.capability = "origin.plot@v1"; job.request_digest = "b" * 64 + job.capability = "origin.plot@v2"; job.request_digest = "b" * 64 job.node_id = uuid4(); job.lease_id = uuid4(); job.status = "running" - job.stage = "origin_running"; job.progress = 10; job.metrics = {}; job.error = {} + job.stage = "software_running"; job.progress = 10; job.metrics = {}; job.error = {} job.artifact_manifest = []; job.created_at = None; job.started_at = None; job.terminal_at = None session.execute.return_value.scalar_one_or_none.return_value = job result, node_message = request_job_cancel(job.user_id, job.job_id) @@ -262,58 +262,88 @@ class SoftwareJobProtocolTests(unittest.TestCase): def test_job_list_is_enriched_for_job_center(self, session_scope) -> None: session = session_scope.return_value.__enter__.return_value job = type("Job", (), {})() - job.job_id = uuid4(); job.task_id = uuid4(); job.capability = "origin.plot@v1" + job.job_id = uuid4(); job.task_id = uuid4(); job.capability = "origin.plot@v2" job.request_digest = "c" * 64; job.node_id = uuid4(); job.status = "running" - job.stage = "origin_running"; job.progress = 20; job.metrics = {}; job.error = {} - job.artifact_manifest = []; job.input_manifest = {"filename": "input.xlsx"} - job.request = {"plot": {"title": "Test"}, "output": {"formats": ["png"]}} + job.stage = "software_running"; job.progress = 20; job.metrics = {}; job.error = {} + job.artifact_manifest = []; job.input_manifest = [{"key": "sample", "filename": "input.xlsx"}] + job.request = {"operation": {"plot": {"title": "Test"}}, "outputs": [ + {"key": "figure_png", "type": "figure", "format": "png"} + ]} job.created_at = None; job.started_at = None; job.terminal_at = None session.execute.return_value.all.return_value = [(job, "材料仿真", "LAB-01")] results = list_jobs(uuid4(), limit=10) self.assertEqual(results[0]["task_name"], "材料仿真") self.assertEqual(results[0]["node_name"], "LAB-01") self.assertEqual(results[0]["request_summary"]["display_name"], "Origin 科研绘图") + self.assertEqual(results[0]["request_summary"]["formats"], ["png"]) self.assertEqual(results[0]["output_dir"], f"origin/{job.job_id}") def test_origin_request_is_canonical_and_rejects_extra_fields(self) -> None: request = { - "schema_version": 1, - "input": {"input_id": str(uuid4()), "sheet": "Sheet1"}, - "plot": {"type": "line", "x": "x", "y": ["y"]}, - "output": {"formats": ["png", "opju"]}, + "schema_version": 2, + "inputs": [ + {"key": "first", "artifact_id": str(uuid4()), "selector": {"sheet": "Sheet1"}}, + {"key": "second", "artifact_id": str(uuid4())}, + ], + "operation": {"plot": { + "type": "line", + "series": [ + {"input": "first", "x": "x", "y": "y"}, + {"input": "second", "x": "time", "y": "value", "label": "Second"}, + ], + }}, + "outputs": [ + {"key": "figure_png", "type": "figure", "format": "png", "options": {"dpi": 600}}, + {"key": "project", "type": "project", "format": "opju"}, + ], } - normalized, digest = _canonical_request(request) + normalized, digest = _canonical_request("origin.plot@v2", request) self.assertEqual(normalized, request) self.assertEqual(len(digest), 64) with self.assertRaisesRegex(Exception, "invalid origin plot request fields"): - _canonical_request({**request, "script": "anything"}) + _canonical_request("origin.plot@v2", {**request, "script": "anything"}) with self.assertRaisesRegex(Exception, "unsupported origin plot fields"): - _canonical_request({**request, "plot": {**request["plot"], "script": "anything"}}) - with self.assertRaisesRegex(Exception, "input.input_id must be an artifact UUID"): - _canonical_request({**request, "input": {"input_id": "C:\\data.csv"}}) + _canonical_request("origin.plot@v2", {**request, "operation": {"plot": { + **request["operation"]["plot"], "script": "anything" + }}}) + with self.assertRaisesRegex(Exception, "artifact UUID"): + _canonical_request("origin.plot@v2", {**request, "inputs": [{"key": "first", "artifact_id": "C:\\data.csv"}]}) def test_origin_request_rejects_unimplemented_plot_semantics(self) -> None: request = { - "schema_version": 1, - "input": {"input_id": str(uuid4())}, - "plot": {"type": "scatter", "x": "time", "y": ["a", "b"]}, - "output": {"formats": ["png"], "dpi": 300}, + "schema_version": 2, + "inputs": [{"key": "sample", "artifact_id": str(uuid4())}], + "operation": {"plot": {"type": "scatter", "series": [ + {"input": "sample", "x": "time", "y": "a"}, + {"input": "sample", "x": "time", "y": "b"}, + ]}}, + "outputs": [{"key": "figure_png", "type": "figure", "format": "png", "options": {"dpi": 300}}], } - for plot, message in ( - ({**request["plot"], "template": "custom"}, "unsupported origin plot template"), - ({**request["plot"], "x_axis": {"scale": "log10"}}, "invalid x_axis"), - ({**request["plot"], "legend": {"enabled": False}}, "invalid plot.legend"), - ({**request["plot"], "y": ["a", "a"]}, "plot.y must contain"), + base_plot = request["operation"]["plot"] + for case_plot, message in ( + ({**base_plot, "template": "custom"}, "unsupported origin plot template"), + ({**base_plot, "x_axis": {"scale": "log10"}}, "invalid x_axis"), + ({**base_plot, "legend": {"enabled": False}}, "invalid plot.legend"), + ({**base_plot, "series": [base_plot["series"][0], base_plot["series"][0]]}, "plot series must be unique"), ): with self.subTest(message=message), self.assertRaisesRegex(Exception, message): - _canonical_request({**request, "plot": plot}) - with self.assertRaisesRegex(Exception, "video recording is not supported"): + _canonical_request("origin.plot@v2", {**request, "operation": {"plot": case_plot}}) + with self.assertRaisesRegex(Exception, "unknown input"): + _canonical_request("origin.plot@v2", {**request, "operation": {"plot": { + **base_plot, + "series": [{"input": "missing", "x": "time", "y": "a"}], + }}}) + with self.assertRaisesRegex(Exception, "key, type, and format do not match"): _canonical_request( - {**request, "output": {"formats": ["png"], "record_video": True}} + "origin.plot@v2", + {**request, "outputs": [{"key": "project", "type": "figure", "format": "png"}]} ) - def test_output_manifest_matches_exact_requested_formats(self) -> None: - request = {"output": {"formats": ["opju", "png"]}} + def test_output_manifest_matches_exact_requested_outputs(self) -> None: + request = {"outputs": [ + {"key": "project", "type": "project", "format": "opju"}, + {"key": "figure_png", "type": "figure", "format": "png"}, + ]} manifest = [ {"artifact_id": "project", "filename": "project.opju", "media_type": "application/x-origin-project", "size_bytes": 10, "sha256": "a" * 64}, {"artifact_id": "figure_png", "filename": "figure.png", "media_type": "image/png", "size_bytes": 20, "sha256": "b" * 64}, @@ -375,6 +405,8 @@ class SoftwareJobProtocolTests(unittest.TestCase): ).read_text(encoding="utf-8") self.assertIn("digest = sha256()", source) self.assertIn('digest.hexdigest() != item["sha256"]', source) + self.assertIn('context["request"].get("outputs", [])', source) + self.assertIn("artifact_id not in requested_ids", source) @patch("core.software_jobs.session_scope") def test_job_state_restores_disconnected_job(self, session_scope) -> None: @@ -393,12 +425,12 @@ class SoftwareJobProtocolTests(unittest.TestCase): "job_id": str(uuid4()), "lease_id": str(lease_id), "request_digest": digest, - "stage": "waiting_input", + "stage": "downloading_inputs", "progress": 0, "metrics": {}, }) self.assertEqual(job.status, "dispatched") - self.assertEqual(job.stage, "waiting_input") + self.assertEqual(job.stage, "downloading_inputs") @patch("core.software_jobs.session_scope") def test_ready_to_run_is_not_reported_as_running(self, session_scope) -> None: diff --git a/tests/test_static_vendor.py b/tests/test_static_vendor.py index 3d8ab6e..534c6db 100644 --- a/tests/test_static_vendor.py +++ b/tests/test_static_vendor.py @@ -62,7 +62,7 @@ class StaticVendorTests(unittest.TestCase): self.assertIn('id="node-enrollment-modal" class="modal"', html) self.assertIn("生成 Windows Node 注册码", html) self.assertIn('"/v1/admin/software-node-enrollments"', admin_js) - self.assertIn('capabilities: ["origin.plot@v1"]', admin_js) + self.assertIn('capabilities: ["origin.plot@v2"]', admin_js) self.assertIn('origin.health === "ready"', admin_js) self.assertIn("ttl_seconds: 600", admin_js) self.assertIn("navigator.clipboard.writeText(value)", admin_js) diff --git a/tests/test_windows_node_source.py b/tests/test_windows_node_source.py index 0ac4773..2d502ff 100644 --- a/tests/test_windows_node_source.py +++ b/tests/test_windows_node_source.py @@ -26,7 +26,7 @@ class WindowsNodeSourceTests(unittest.TestCase): 'SetRequestHeader("X-Node-Id"', "DataProtectionScope.LocalMachine", "SetAccessRuleProtection(isProtected: true", - '"origin.plot@v1"', + '"origin.plot@v2"', "NotifyIcon", "ConfigurationForm", "TrayIconFactory.Create", @@ -71,16 +71,42 @@ class WindowsNodeSourceTests(unittest.TestCase): self.assertIn("注册并连接", form) self.assertIn("ContentWidth = 760", form) self.assertIn("Origin 绘图", form) - self.assertIn('CreateCapabilityRow("Origin 绘图", "origin.plot@v1")', form) + self.assertIn('CreateCapabilityRow("Origin 绘图", "origin.plot@v2")', form) self.assertIn('CreateButton("立即重连", 112, primary: true)', form) self.assertIn("ReconnectRequested?.Invoke()", form) self.assertIn("registrationCard.Visible = !registered", form) self.assertIn("reconnect.Visible = registered", form) self.assertIn("resetIdentity.Visible = registered", form) + self.assertIn('CreateSectionTitle("本机任务")', form) + self.assertIn("DataGridView", form) + self.assertIn("jobInbox.ReadJobSnapshots()", form) + self.assertIn('"software_running" => "软件执行中"', form) + self.assertNotIn("HttpClient", form) tray = (PROJECT / "TrayApplicationContext.cs").read_text(encoding="utf-8") self.assertIn("form.ReconnectRequested += RestartConnection", tray) + def test_local_job_monitor_is_persisted_and_software_neutral(self) -> None: + inbox = (PROJECT / "JobInboxStore.cs").read_text(encoding="utf-8") + connection = (PROJECT / "NodeConnectionLoop.cs").read_text(encoding="utf-8") + models = (PROJECT / "JobMonitorModels.cs").read_text(encoding="utf-8") + self.assertIn('"state.json"', inbox) + self.assertIn("ReadJobSnapshots", inbox) + self.assertIn("AtomicWrite(path, content, overwrite: true)", inbox) + for stage in ( + "accepted", + "downloading_inputs", + "ready_to_run", + "software_running", + "uploading_outputs", + "succeeded", + "failed", + "cancelled", + ): + self.assertIn(f'"{stage}"', inbox + connection) + self.assertIn("JobDisplaySnapshot", models) + self.assertNotIn("origin_running", inbox + connection + models) + def test_unified_installer_registers_login_startup(self) -> None: launcher = (ROOT / "install-windows-node.bat").read_text(encoding="utf-8") self.assertNotIn("powershell", launcher.lower()) @@ -141,7 +167,7 @@ class WindowsNodeSourceTests(unittest.TestCase): self.assertIn('AutomationProgId = @"Origin.ApplicationSI\\CLSID"', probe) self.assertIn("RegistryHive.LocalMachine", probe) self.assertIn("RegistryHive.CurrentUser", probe) - self.assertIn('new("OriginPro", version, "0.2.0", health, detail)', probe) + self.assertIn('new("OriginPro", version, "0.3.0", health, detail)', probe) self.assertIn( "&& !jobInbox.HasPendingOriginJobs", connection, @@ -164,9 +190,12 @@ class WindowsNodeSourceTests(unittest.TestCase): def test_job_offer_is_persisted_before_acceptance(self) -> None: inbox = (PROJECT / "JobInboxStore.cs").read_text(encoding="utf-8") connection = (PROJECT / "NodeConnectionLoop.cs").read_text(encoding="utf-8") - self.assertIn('capabilityValue.GetString() != "origin.plot@v1"', inbox) + self.assertIn('capabilityValue.GetString() != "origin.plot@v2"', inbox) + self.assertIn('root.TryGetProperty("input_transfers"', inbox) + self.assertIn('"input", key, filename', inbox) self.assertIn("PlotTypes.Contains", inbox) - self.assertIn("OutputFormats.Contains", inbox) + self.assertIn("IsValidOutputs", inbox) + self.assertIn('("figure", "png") => "figure_png"', inbox) self.assertIn("FileOptions.WriteThrough", inbox) self.assertIn("stream.Flush(flushToDisk: true)", inbox) new_record = inbox.split("var record =", 1)[1].split("private static JsonElement?", 1)[0] @@ -183,9 +212,11 @@ class WindowsNodeSourceTests(unittest.TestCase): self.assertIn("StartJobPipeline(socket, acceptedJob)", connection) self.assertIn("inputDownloader.DownloadAsync(job, CancellationToken.None)", connection) self.assertIn('stage = "uploading_outputs"', connection) + self.assertIn('stage = "software_running"', connection) + self.assertNotIn('stage = "origin_running"', connection) self.assertIn("&& !job.UploadComplete", connection) self.assertIn("StartJobPipeline(socket, job)", connection) - self.assertIn('stage = "waiting_input"', connection) + self.assertIn('stage = "downloading_inputs"', connection) self.assertIn('Path.Combine(jobDirectory, "terminal.json")', inbox) self.assertIn("AtomicWrite(requestPath, updated, overwrite: true)", inbox) diff --git a/tools/software_jobs.py b/tools/software_jobs.py index 60c7ddc..fd9c2ac 100644 --- a/tools/software_jobs.py +++ b/tools/software_jobs.py @@ -32,7 +32,7 @@ class SoftwareCapabilityListTool(_SoftwareJobTool): nodes = list_nodes() items = [{ "capability": item, - "display_name": "Origin 科研绘图" if item == "origin.plot@v1" else item, + "display_name": "Origin 科研绘图" if item == "origin.plot@v2" else item, "available_nodes": sum( 1 for node in nodes @@ -47,8 +47,8 @@ class SoftwareCapabilityListTool(_SoftwareJobTool): class SoftwareJobSubmitTool(_SoftwareJobTool): name = "software_job_submit" description = ( - "Submit an Origin plot job using a registered CSV, XLSX, or JSON artifact. " - "Call register_artifact first when the input is only a workspace file. " + "Submit an Origin plot job using one or more registered CSV, XLSX, or JSON artifacts. " + "Call register_artifact first for each workspace file. " "Return immediately with job_id; do not poll continuously or wait for completion." ) _axis_schema = { @@ -64,19 +64,22 @@ class SoftwareJobSubmitTool(_SoftwareJobTool): "type": "object", "properties": { "type": {"type": "string", "enum": ["line", "scatter", "line_scatter"]}, - "x": { - "type": "string", - "minLength": 1, - "maxLength": 128, - "description": "Exact X column name in the input file.", - }, - "y": { + "series": { "type": "array", "minItems": 1, "maxItems": 16, - "uniqueItems": True, - "items": {"type": "string", "minLength": 1, "maxLength": 128}, - "description": "One to sixteen unique Y column names.", + "items": { + "type": "object", + "properties": { + "input": {"type": "string", "pattern": "^[a-z][a-z0-9_]{0,31}$"}, + "x": {"type": "string", "minLength": 1, "maxLength": 128}, + "y": {"type": "string", "minLength": 1, "maxLength": 128}, + "label": {"type": "string", "minLength": 1, "maxLength": 200}, + }, + "required": ["input", "x", "y"], + "additionalProperties": False, + }, + "description": "Series with explicit input key and X/Y columns.", }, "template": {"type": "string", "enum": ["publication_double_column"]}, "title": {"type": "string", "maxLength": 500}, @@ -92,84 +95,112 @@ class SoftwareJobSubmitTool(_SoftwareJobTool): }, "error_bars": {"type": "null"}, }, - "required": ["type", "x", "y"], + "required": ["type", "series"], + "additionalProperties": False, + } + _input_schema = { + "type": "object", + "properties": { + "key": {"type": "string", "pattern": "^[a-z][a-z0-9_]{0,31}$"}, + "artifact_id": { + "type": "string", + "description": "Artifact UUID returned by register_artifact.", + }, + "selector": { + "type": "object", + "properties": { + "sheet": {"type": "string", "minLength": 1, "maxLength": 128}, + }, + "required": ["sheet"], + "additionalProperties": False, + }, + }, + "required": ["key", "artifact_id"], "additionalProperties": False, } _output_schema = { "type": "object", "properties": { - "formats": { - "type": "array", - "minItems": 1, - "uniqueItems": True, - "items": {"type": "string", "enum": ["opju", "png", "svg", "pdf"]}, + "key": { + "type": "string", + "enum": ["project", "figure_png", "figure_svg", "figure_pdf"], + }, + "type": {"type": "string", "enum": ["project", "figure"]}, + "format": {"type": "string", "enum": ["opju", "png", "svg", "pdf"]}, + "options": { + "type": "object", + "properties": {"dpi": {"type": "integer", "minimum": 72, "maximum": 1200}}, + "required": ["dpi"], + "additionalProperties": False, + "description": "Only valid for the PNG figure output.", }, - "dpi": {"type": "integer", "minimum": 72, "maximum": 1200}, - "capture_screenshots": {"type": "boolean"}, - "record_video": {"type": "boolean", "enum": [False]}, }, - "required": ["formats"], + "required": ["key", "type", "format"], "additionalProperties": False, } parameters = { "type": "object", "properties": { "capability": {"type": "string", "enum": sorted(SUPPORTED_CAPABILITIES)}, - "input_id": { - "type": "string", - "description": "Artifact UUID returned by register_artifact or another artifact-producing flow.", + "inputs": { + "type": "array", + "minItems": 1, + "maxItems": 16, + "items": _input_schema, }, - "sheet": { - "type": "string", - "minLength": 1, - "maxLength": 128, - "description": "Optional XLSX worksheet name.", + "operation": { + "type": "object", + "properties": {"plot": _plot_schema}, + "required": ["plot"], + "additionalProperties": False, + }, + "outputs": { + "type": "array", + "minItems": 1, + "maxItems": 16, + "uniqueItems": True, + "items": _output_schema, + "description": ( + "Requested deliverables. Use project/project/opju, " + "figure_png/figure/png, figure_svg/figure/svg, or figure_pdf/figure/pdf." + ), }, - "plot": _plot_schema, - "output": _output_schema, "idempotency_key": { "type": "string", "description": "Stable unique key for this exact submission; omit to generate one.", }, }, - "required": ["capability", "input_id", "plot", "output"], + "required": ["capability", "inputs", "operation", "outputs"], "additionalProperties": False, } def execute( self, capability: str, - input_id: str = "", - plot: dict | None = None, - output: dict | None = None, - sheet: str = "", - request: dict | None = None, + inputs: list[dict] | None = None, + operation: dict | None = None, + outputs: list[dict] | None = None, idempotency_key: str = "", ) -> str: try: - if isinstance(input_id, dict) and request is None: - request = input_id - input_id = "" - if request is not None: - if input_id or plot is not None or output is not None or sheet: - return "[Error] request cannot be combined with input_id, sheet, plot, or output" - normalized_request = request - else: - if not input_id or plot is None or output is None: - return "[Error] input_id, plot, and output are required for an Origin job" + if not inputs or operation is None or not outputs: + return "[Error] inputs, operation, and outputs are required for a software job" + canonical_inputs = [] + for item in inputs: + if not isinstance(item, dict): + return "[Error] each input must be an object" try: - canonical_input_id = str(UUID(str(input_id))) + artifact_id = str(UUID(str(item.get("artifact_id") or ""))) except ValueError: - return "[Error] input_id must be an artifact UUID; call register_artifact first" - input_spec = {"input_id": canonical_input_id} - if sheet: - input_spec["sheet"] = sheet - normalized_request = { - "schema_version": 1, - "input": input_spec, - "plot": plot, - "output": output, - } + return "[Error] every artifact_id must be a UUID; call register_artifact first" + canonical = {**item, "artifact_id": artifact_id} + canonical_inputs.append(canonical) + normalized_request = { + "schema_version": 2, + "inputs": canonical_inputs, + "operation": operation, + "outputs": outputs, + } job, created = create_job( self.user_id, self.task_id, diff --git a/web/routers/software_nodes.py b/web/routers/software_nodes.py index 954a5e9..82e2f96 100644 --- a/web/routers/software_nodes.py +++ b/web/routers/software_nodes.py @@ -281,9 +281,10 @@ def register_software_node_routes(app, *, require_user, require_admin) -> None: except SoftwareNodeError as exc: raise HTTPException(400, str(exc)) from exc - @app.get("/v1/software-jobs/{job_id}/input", tags=["software-nodes"]) + @app.get("/v1/software-jobs/{job_id}/inputs/{input_key}", tags=["software-nodes"]) def download_software_job_input( job_id: UUID, + input_key: str, authorization: str | None = Header(default=None), x_node_id: str = Header(default=""), ): @@ -292,7 +293,7 @@ def register_software_node_routes(app, *, require_user, require_admin) -> None: authenticate_node(node_id, _bearer(authorization)) except (ValueError, SoftwareNodeError) as exc: raise HTTPException(401, "invalid node credentials") from exc - item = get_job_input(node_id, job_id) + item = get_job_input(node_id, job_id, input_key) if item is None: raise HTTPException(404, "software job input not found") target = safe_join(load_user_root(item["user_id"]), item["current_path"]) @@ -340,9 +341,13 @@ def register_software_node_routes(app, *, require_user, require_admin) -> None: metadata = OUTPUT_ARTIFACTS.get(artifact_id) if metadata is None: raise HTTPException(400, "unsupported output artifact identity") - filename, _, output_format = metadata - requested_formats = set(context["request"].get("output", {}).get("formats") or []) - if output_format is not None and output_format not in requested_formats: + filename, _, _ = metadata + requested_ids = { + item.get("key") + for item in context["request"].get("outputs", []) + if isinstance(item, dict) + } | SOFTWARE_JOB_METADATA_IDS + if artifact_id not in requested_ids: raise HTTPException(400, "output artifact was not requested") if not 1 <= x_content_length <= MAX_OUTPUT_ARTIFACT_BYTES: raise HTTPException(400, "output artifact size is invalid") diff --git a/web/schemas.py b/web/schemas.py index 6d009f4..982d7df 100644 --- a/web/schemas.py +++ b/web/schemas.py @@ -118,7 +118,7 @@ class ExternalSystemCredentialsRequest(BaseModel): class SoftwareEnrollmentCreateRequest(BaseModel): expected_name: str = "" - capabilities: list[str] = Field(default_factory=lambda: ["origin.plot@v1"]) + capabilities: list[str] = Field(default_factory=lambda: ["origin.plot@v2"]) ttl_seconds: int = 600 @@ -137,5 +137,5 @@ class SoftwareNodeDisableRequest(BaseModel): class SoftwareJobCreateRequest(BaseModel): idempotency_key: str - capability: str = "origin.plot@v1" + capability: str = "origin.plot@v2" request: dict = Field(default_factory=dict) diff --git a/web/static/js/admin.js b/web/static/js/admin.js index 7ac2421..8771ff8 100644 --- a/web/static/js/admin.js +++ b/web/static/js/admin.js @@ -312,7 +312,7 @@ async function createNodeEnrollment(e) { try { const result = await apiSend("POST", "/v1/admin/software-node-enrollments", { expected_name: $("node-expected-name").value.trim(), - capabilities: ["origin.plot@v1"], + capabilities: ["origin.plot@v2"], ttl_seconds: 600, }); $("node-enrollment-code").textContent = result.enrollment_code || ""; diff --git a/web/static/js/software_jobs.js b/web/static/js/software_jobs.js index 4d6f85f..14eba62 100644 --- a/web/static/js/software_jobs.js +++ b/web/static/js/software_jobs.js @@ -25,8 +25,8 @@ const statusLabel = { succeeded: "已完成", failed: "失败", cancelled: "已取消", }; const stageLabel = { - accepted: "节点已接收", waiting_input: "正在下载输入文件", ready_to_run: "准备软件环境", - origin_running: "Origin 正在生成图表", uploading_outputs: "正在上传结果", + accepted: "节点已接收", downloading_inputs: "正在下载输入文件", ready_to_run: "准备软件环境", + software_running: "专业软件正在执行", uploading_outputs: "正在上传结果", cancel_requested: "停止请求已发送", terminal: "任务已结束", }; @@ -165,7 +165,8 @@ function onListScroll(event) { function jobCard(job) { const summary = job.request_summary || {}; - const input = job.input || {}; + const inputs = Array.isArray(job.input) ? job.input : []; + const inputNames = inputs.map((item) => item.filename).filter(Boolean).join("、"); const active = ACTIVE.has(job.status); const progress = Math.max(0, Math.min(100, Number(job.progress || 0))); const detail = stageLabel[job.stage] || statusLabel[job.status] || "正在处理"; @@ -177,7 +178,7 @@ function jobCard(job) { ${statusIcon(job.status)}${escapeHtml(statusLabel[job.status] || job.status)}
${icons.activity}${escapeHtml(detail)}${error ? ` · ${escapeHtml(error)}` : ""}
${active ? `
` : ""} -
${icons.clock}${escapeHtml(job.task_name || "未命名对话")}${input.filename ? ` · ${escapeHtml(input.filename)}` : ""}${openedText ? ` · 开启于 ${escapeHtml(openedText)}` : ""}
+
${icons.clock}${escapeHtml(job.task_name || "未命名对话")}${inputNames ? ` · ${escapeHtml(inputNames)}` : ""}${openedText ? ` · 开启于 ${escapeHtml(openedText)}` : ""}
${job.status === "succeeded" ? `` : ""} diff --git a/windows-node/README.md b/windows-node/README.md index 65d9ea1..f0a0983 100644 --- a/windows-node/README.md +++ b/windows-node/README.md @@ -2,7 +2,7 @@ 内网 MVP 的 Windows 执行节点,目标运行环境为 Windows 11 Enterprise + .NET 10 SDK 10.0.303。仓库根目录 `global.json` 固定 SDK patch;客户端只使用 .NET Windows Desktop Framework,不依赖第三方 NuGet 包。 -当前实现托盘状态角标、小型配置窗口、注册、DPAPI/ACL 配置保存、WebSocket `hello`/心跳和退避重连,并只读探测 Origin/OriginPro 安装版本、COM 自动化组件与桌面会话状态。Node 可以接收受控的 `origin.plot@v1` offer,在本机任务目录原子保存请求后回报 accept/reject;随后以 Node 身份流式下载任务绑定的 CSV/XLSX/JSON,校验大小与 SHA-256 后原子保存。固定 Origin Worker 独立于单次 WebSocket 执行,断线不终止已启动绘图。成功产物按 manifest 逐项流式上传并由云端复核大小与 SHA-256,全部完成后原子发布到任务工作目录的 `origin//`;中断后按本地 `upload-complete.json` 幂等续传。 +当前实现托盘状态角标、配置与本机任务窗口、注册、DPAPI/ACL 配置保存、WebSocket `hello`/心跳和退避重连,并只读探测 Origin/OriginPro 安装版本、COM 自动化组件与桌面会话状态。本机任务列表只读取已派发到该 Node 的持久化目录,展示标题、全部输入、通用执行阶段、进度、时间、Job ID 和错误,不查询云端未派发队列。Node 可以接收受控的 `origin.plot@v2` offer,在本机任务目录原子保存请求后回报 accept/reject;随后以 Node 身份逐个流式下载任务绑定的 CSV/XLSX/JSON 到 `input//`,校验大小与 SHA-256 后原子保存。固定 Origin Worker 独立于单次 WebSocket 执行,断线不终止已启动绘图。成功产物按 manifest 逐项流式上传并由云端复核大小与 SHA-256,全部完成后原子发布到任务工作目录的 `origin//`;中断后按本地 `upload-complete.json` 幂等续传。 在仓库根目录执行一条命令生成可分发 ZIP: @@ -31,7 +31,7 @@ windows-node/Zcbot.WindowsNode/bin/Debug/net10.0-windows/Zcbot.WindowsNode.exe e windows-node/Zcbot.WindowsNode/bin/Debug/net10.0-windows/Zcbot.WindowsNode.exe ``` -直接双击 EXE 默认启动托盘 UI;红点表示未注册或身份失效,黄点表示正在连接,绿点表示在线。双击托盘图标打开配置窗口。无界面运行使用: +直接双击 EXE 默认启动托盘 UI;红点表示未注册或身份失效,黄点表示正在连接,绿点表示在线。双击托盘图标打开窗口,可在“本机任务”中查看最近 50 条已接收任务及执行详情。无界面运行使用: ```powershell Zcbot.WindowsNode.exe run --headless diff --git a/windows-node/Zcbot.WindowsNode/ConfigurationForm.cs b/windows-node/Zcbot.WindowsNode/ConfigurationForm.cs index 7df4525..af2fff4 100644 --- a/windows-node/Zcbot.WindowsNode/ConfigurationForm.cs +++ b/windows-node/Zcbot.WindowsNode/ConfigurationForm.cs @@ -25,6 +25,11 @@ internal sealed class ConfigurationForm : Form private readonly Label detail = CreateBodyLabel(); private readonly Label identity = CreateBodyLabel(); private readonly Label capabilitySummary = CreateBodyLabel(); + private readonly Label jobSummary = CreateBodyLabel(); + private readonly Label jobDetail = CreateBodyLabel(); + private readonly DataGridView jobGrid = CreateJobGrid(); + private readonly JobInboxStore jobInbox = new(NodePaths.ForCurrentMachine().JobsDirectory); + private readonly System.Windows.Forms.Timer jobRefreshTimer = new() { Interval = 1000 }; private readonly TableLayoutPanel registrationCard; private bool changingStartup; @@ -63,7 +68,7 @@ internal sealed class ConfigurationForm : Form AutoSizeMode = AutoSizeMode.GrowAndShrink, Dock = DockStyle.Top, ColumnCount = 1, - RowCount = 5, + RowCount = 6, BackColor = BackColor, }; page.ColumnStyles.Add(new ColumnStyle(SizeType.Percent, 100)); @@ -92,7 +97,7 @@ internal sealed class ConfigurationForm : Form statusCard.Controls.Add(detail); statusCard.Controls.Add(identity); statusCard.Controls.Add(CreateDivider()); - statusCard.Controls.Add(CreateCapabilityRow("Origin 绘图", "origin.plot@v1")); + statusCard.Controls.Add(CreateCapabilityRow("Origin 绘图", "origin.plot@v2")); statusCard.Controls.Add(capabilitySummary); var resetActions = CreateActions(); resetActions.Controls.Add(reconnect); @@ -100,6 +105,15 @@ internal sealed class ConfigurationForm : Form statusCard.Controls.Add(resetActions); page.Controls.Add(statusCard); + var jobsCard = CreateCard(); + jobsCard.Controls.Add(CreateSectionTitle("本机任务")); + jobsCard.Controls.Add(CreateHint( + "仅显示已经派发到本机的任务;状态来自本地持久化记录,断线或重启后仍可查看。")); + jobsCard.Controls.Add(jobSummary); + jobsCard.Controls.Add(jobGrid); + jobsCard.Controls.Add(jobDetail); + page.Controls.Add(jobsCard); + registrationCard = CreateCard(); registrationCard.Controls.Add(CreateSectionTitle("首次注册")); registrationCard.Controls.Add(CreateHint( @@ -137,6 +151,11 @@ internal sealed class ConfigurationForm : Form } }; startAtLogin.Checked = StartupRegistration.IsEnabled; + jobGrid.SelectionChanged += (_, _) => ShowSelectedJob(); + jobRefreshTimer.Tick += (_, _) => RefreshJobs(); + jobRefreshTimer.Start(); + Disposed += (_, _) => jobRefreshTimer.Dispose(); + RefreshJobs(); ApplyStatus(NodeStatus.Create(NodeState.NotRegistered, "尚未注册"), null); } @@ -188,6 +207,106 @@ internal sealed class ConfigurationForm : Form return $"{state} · {version}\n{origin.Detail}"; } + private void RefreshJobs() + { + Guid? selectedId = jobGrid.SelectedRows.Count > 0 + && jobGrid.SelectedRows[0].Tag is JobDisplaySnapshot selected + ? selected.JobId + : null; + IReadOnlyList snapshots; + try + { + snapshots = jobInbox.ReadJobSnapshots(); + } + catch (Exception exception) when ( + exception is IOException or UnauthorizedAccessException) + { + jobSummary.Text = $"无法读取本机任务:{exception.Message}"; + return; + } + jobGrid.Rows.Clear(); + DataGridViewRow? rowToSelect = null; + foreach (var snapshot in snapshots) + { + var index = jobGrid.Rows.Add( + FormatJobStage(snapshot.Stage), + snapshot.Title, + snapshot.InputFilename, + FormatProgress(snapshot), + snapshot.AcceptedAt.LocalDateTime.ToString("MM-dd HH:mm"), + snapshot.JobId.ToString("N")[..8]); + var row = jobGrid.Rows[index]; + row.Tag = snapshot; + row.DefaultCellStyle.ForeColor = snapshot.Stage switch + { + "failed" => Color.Firebrick, + "cancelled" => Color.DimGray, + "succeeded" => Color.ForestGreen, + _ => Color.FromArgb(30, 41, 59), + }; + if (snapshot.JobId == selectedId) + { + rowToSelect = row; + } + } + var active = snapshots.Count(item => item.IsActive); + jobSummary.Text = snapshots.Count == 0 + ? "暂无本机任务" + : $"活动任务 {active} 个 · 最近记录 {snapshots.Count} 条"; + if (rowToSelect is not null) + { + rowToSelect.Selected = true; + jobGrid.CurrentCell = rowToSelect.Cells[0]; + } + else if (jobGrid.Rows.Count > 0) + { + jobGrid.Rows[0].Selected = true; + jobGrid.CurrentCell = jobGrid.Rows[0].Cells[0]; + } + else + { + jobDetail.Text = "选择任务后可查看执行阶段、更新时间和完整 Job ID。"; + } + ShowSelectedJob(); + } + + private void ShowSelectedJob() + { + if (jobGrid.SelectedRows.Count == 0 + || jobGrid.SelectedRows[0].Tag is not JobDisplaySnapshot snapshot) + { + return; + } + jobDetail.Text = string.Join("\n", [ + $"{FormatJobStage(snapshot.Stage)} · {snapshot.Detail}", + $"能力:{snapshot.Capability}", + $"Job ID:{snapshot.JobId}", + $"更新时间:{snapshot.UpdatedAt.LocalDateTime:yyyy-MM-dd HH:mm:ss}", + ]); + } + + private static string FormatProgress(JobDisplaySnapshot snapshot) => + snapshot.Stage == "software_running" + ? $"已执行 {FormatElapsed(DateTimeOffset.UtcNow - snapshot.UpdatedAt)}" + : $"{snapshot.Progress}%"; + + private static string FormatElapsed(TimeSpan elapsed) => elapsed.TotalHours >= 1 + ? $"{(int)elapsed.TotalHours}:{elapsed.Minutes:00}:{elapsed.Seconds:00}" + : $"{elapsed.Minutes:00}:{elapsed.Seconds:00}"; + + private static string FormatJobStage(string stage) => stage switch + { + "accepted" => "等待处理", + "downloading_inputs" => "下载输入", + "ready_to_run" => "准备执行", + "software_running" => "软件执行中", + "uploading_outputs" => "上传结果", + "succeeded" => "成功", + "failed" => "失败", + "cancelled" => "已取消", + _ => stage, + }; + private void ResetIdentity() { var answer = MessageBox.Show( @@ -289,6 +408,65 @@ internal sealed class ConfigurationForm : Form Margin = new Padding(0, 2, 0, 4), }; + private static DataGridView CreateJobGrid() + { + var grid = new DataGridView + { + Height = 238, + Dock = DockStyle.Top, + Margin = new Padding(0, 8, 0, 8), + BackgroundColor = Color.White, + BorderStyle = BorderStyle.FixedSingle, + AllowUserToAddRows = false, + AllowUserToDeleteRows = false, + AllowUserToResizeRows = false, + AutoSizeRowsMode = DataGridViewAutoSizeRowsMode.None, + ColumnHeadersHeight = 34, + EnableHeadersVisualStyles = false, + MultiSelect = false, + ReadOnly = true, + RowHeadersVisible = false, + RowTemplate = { Height = 34 }, + SelectionMode = DataGridViewSelectionMode.FullRowSelect, + }; + grid.ColumnHeadersDefaultCellStyle.BackColor = Color.FromArgb(241, 245, 249); + grid.ColumnHeadersDefaultCellStyle.ForeColor = Color.FromArgb(51, 65, 85); + grid.DefaultCellStyle.SelectionBackColor = Color.FromArgb(219, 234, 254); + grid.DefaultCellStyle.SelectionForeColor = Color.FromArgb(30, 64, 175); + grid.Columns.Add(new DataGridViewTextBoxColumn + { + HeaderText = "状态", + Width = 105, + }); + grid.Columns.Add(new DataGridViewTextBoxColumn + { + HeaderText = "任务", + AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill, + MinimumWidth = 170, + }); + grid.Columns.Add(new DataGridViewTextBoxColumn + { + HeaderText = "输入", + Width = 125, + }); + grid.Columns.Add(new DataGridViewTextBoxColumn + { + HeaderText = "进度", + Width = 95, + }); + grid.Columns.Add(new DataGridViewTextBoxColumn + { + HeaderText = "接收时间", + Width = 100, + }); + grid.Columns.Add(new DataGridViewTextBoxColumn + { + HeaderText = "Job ID", + Width = 76, + }); + return grid; + } + private static Panel CreateDivider() => new() { Height = 1, diff --git a/windows-node/Zcbot.WindowsNode/EnrollmentClient.cs b/windows-node/Zcbot.WindowsNode/EnrollmentClient.cs index a740c40..60728b7 100644 --- a/windows-node/Zcbot.WindowsNode/EnrollmentClient.cs +++ b/windows-node/Zcbot.WindowsNode/EnrollmentClient.cs @@ -6,7 +6,7 @@ namespace Zcbot.WindowsNode; internal static class EnrollmentClient { - private static readonly string[] Capabilities = ["origin.plot@v1"]; + private static readonly string[] Capabilities = ["origin.plot@v2"]; internal static async Task EnrollAsync( EnrollOptions options, NodeConfigStore store, CancellationToken cancellationToken) diff --git a/windows-node/Zcbot.WindowsNode/JobInboxStore.cs b/windows-node/Zcbot.WindowsNode/JobInboxStore.cs index 5a69959..6a488f3 100644 --- a/windows-node/Zcbot.WindowsNode/JobInboxStore.cs +++ b/windows-node/Zcbot.WindowsNode/JobInboxStore.cs @@ -8,7 +8,17 @@ internal sealed class JobInboxStore(string jobsDirectory) private static readonly JsonSerializerOptions JsonOptions = new() { WriteIndented = true }; private static readonly HashSet PlotTypes = ["line", "scatter", "line_scatter"]; - private static readonly HashSet OutputFormats = ["opju", "png", "svg", "pdf"]; + private static readonly HashSet JobStages = + [ + "accepted", + "downloading_inputs", + "ready_to_run", + "software_running", + "uploading_outputs", + "succeeded", + "failed", + "cancelled", + ]; // Origin 执行槽只由尚无终态的任务占用。成功但上传确认尚未落盘的任务会由 // 心跳恢复管线继续重传;上传不使用 Origin,不能反向阻塞新的绘图任务。 @@ -41,8 +51,8 @@ internal sealed class JobInboxStore(string jobsDirectory) jobId, leaseId, requestDigest, - root.TryGetProperty("input_transfer", out var transfer) - ? transfer.Clone() : null, + root.TryGetProperty("input_transfers", out var transfers) + ? transfers.Clone() : null, ReadTerminal(Path.Combine(jobDirectory, "terminal.json")), File.Exists(Path.Combine(jobDirectory, "upload-complete.json")))); } @@ -54,6 +64,127 @@ internal sealed class JobInboxStore(string jobsDirectory) return jobs; } + internal IReadOnlyList ReadJobSnapshots(int limit = 50) + { + if (!Directory.Exists(jobsDirectory)) + { + return []; + } + var snapshots = new List(); + foreach (var requestPath in Directory.EnumerateFiles( + jobsDirectory, "request.json", SearchOption.AllDirectories)) + { + try + { + using var requestDocument = JsonDocument.Parse(File.ReadAllBytes(requestPath)); + var root = requestDocument.RootElement; + if (!TryReadGuid(root, "job_id", out var jobId)) + { + continue; + } + var jobDirectory = Directory.GetParent( + Directory.GetParent(requestPath)!.FullName)!.FullName; + var acceptedAt = ReadDate(root, "accepted_at") + ?? new DateTimeOffset(File.GetCreationTimeUtc(requestPath)); + var capability = ReadString(root, "capability", "unknown"); + var title = "未命名任务"; + if (root.TryGetProperty("request", out var request) + && request.TryGetProperty("operation", out var operation) + && operation.TryGetProperty("plot", out var plot)) + { + title = ReadString(plot, "title", title); + } + var inputFilename = root.TryGetProperty("input_transfers", out var transfers) + && transfers.ValueKind == JsonValueKind.Array + ? string.Join(", ", transfers.EnumerateArray() + .Select(item => ReadString(item, "filename", "-"))) + : "-"; + var state = ReadState(Path.Combine(jobDirectory, "state.json")); + var terminal = ReadTerminal(Path.Combine(jobDirectory, "terminal.json")); + var uploadPath = Path.Combine(jobDirectory, "upload-complete.json"); + var uploadComplete = File.Exists(uploadPath); + var stage = state?.Stage ?? "accepted"; + var progress = state?.Progress ?? 0; + var detail = state?.Detail ?? "任务已由本机接收"; + var updatedAt = state is not null && state.UpdatedAt != DateTimeOffset.MinValue + ? state.UpdatedAt + : acceptedAt; + if (terminal is JsonElement terminalValue) + { + var terminalStatus = ReadString(terminalValue, "status", "failed"); + if (terminalStatus == "succeeded" && !uploadComplete) + { + stage = "uploading_outputs"; + progress = Math.Max(progress, 90); + detail = state?.Stage == "uploading_outputs" + ? state.Detail + : "软件执行完成,等待上传结果"; + } + else + { + stage = terminalStatus; + progress = terminalStatus == "succeeded" ? 100 : progress; + detail = TerminalDetail(terminalValue, terminalStatus, detail); + } + updatedAt = LatestWrite(updatedAt, Path.Combine(jobDirectory, "terminal.json")); + } + if (uploadComplete) + { + stage = "succeeded"; + progress = 100; + detail = "结果已上传并由云端确认"; + updatedAt = LatestWrite(updatedAt, uploadPath); + } + snapshots.Add(new JobDisplaySnapshot( + jobId, + capability, + title, + inputFilename, + stage, + Math.Clamp(progress, 0, 100), + detail, + acceptedAt, + updatedAt, + uploadComplete)); + } + catch (Exception exception) when ( + exception is JsonException or IOException or UnauthorizedAccessException) + { + } + } + return snapshots + .OrderByDescending(item => item.IsActive) + .ThenByDescending(item => item.UpdatedAt) + .Take(Math.Max(1, limit)) + .ToArray(); + } + + internal void WriteState(RecoverableJob job, string stage, int progress, string detail) + { + if (!JobStages.Contains(stage)) + { + throw new InvalidDataException("Unsupported local job stage."); + } + var path = Path.Combine( + jobsDirectory, job.JobId.ToString("D"), "state.json"); + var content = JsonSerializer.SerializeToUtf8Bytes(new + { + stage, + progress = Math.Clamp(progress, 0, 100), + detail = detail[..Math.Min(detail.Length, 500)], + updated_at = DateTimeOffset.UtcNow, + }, JsonOptions); + try + { + AtomicWrite(path, content, overwrite: true); + } + catch (Exception exception) when ( + exception is IOException or UnauthorizedAccessException) + { + Console.Error.WriteLine($"[WARN] Local job state update failed: {exception.Message}"); + } + } + internal JobOfferResult Accept(JsonElement payload) { if (!TryReadGuid(payload, "job_id", out var jobId) @@ -62,11 +193,11 @@ internal sealed class JobInboxStore(string jobsDirectory) || digestValue.ValueKind != JsonValueKind.String || digestValue.GetString() is not { Length: 64 } requestDigest || !payload.TryGetProperty("capability", out var capabilityValue) - || capabilityValue.GetString() != "origin.plot@v1" + || capabilityValue.GetString() != "origin.plot@v2" || !payload.TryGetProperty("request", out var request) || request.ValueKind != JsonValueKind.Object - || !payload.TryGetProperty("input_transfer", out var inputTransfer) - || !IsValidInputTransfer(inputTransfer)) + || !payload.TryGetProperty("input_transfers", out var inputTransfers) + || !IsValidInputTransfers(inputTransfers, jobId)) { return JobOfferResult.Reject("invalid_offer"); } @@ -74,7 +205,10 @@ internal sealed class JobInboxStore(string jobsDirectory) { return JobOfferResult.Reject("unsupported_request"); } - + if (!InputsMatchTransfers(request, inputTransfers)) + { + return JobOfferResult.Reject("invalid_offer"); + } var directory = Path.Combine(jobsDirectory, jobId.ToString("D")); var requestDirectory = Path.Combine(directory, "request"); var requestPath = Path.Combine(requestDirectory, "request.json"); @@ -99,13 +233,14 @@ internal sealed class JobInboxStore(string jobsDirectory) job_id = jobId, lease_id = leaseId, request_digest = requestDigest, - capability = "origin.plot@v1", + capability = "origin.plot@v2", accepted_at = DateTimeOffset.UtcNow, request, - input_transfer = inputTransfer, + input_transfers = inputTransfers, }, JsonOptions); AtomicWrite(requestPath, updated, overwrite: true); } + EnsureAcceptedState(jobId); return JobOfferResult.Accept(jobId, leaseId, requestDigest); } catch (JsonException) @@ -119,14 +254,15 @@ internal sealed class JobInboxStore(string jobsDirectory) job_id = jobId, lease_id = leaseId, request_digest = requestDigest, - capability = "origin.plot@v1", + capability = "origin.plot@v2", accepted_at = DateTimeOffset.UtcNow, request, - input_transfer = inputTransfer, + input_transfers = inputTransfers, }, JsonOptions); try { AtomicWrite(requestPath, record, overwrite: false); + EnsureAcceptedState(jobId); return JobOfferResult.Accept(jobId, leaseId, requestDigest); } catch (IOException) @@ -145,6 +281,101 @@ internal sealed class JobInboxStore(string jobsDirectory) return document.RootElement.Clone(); } + private void EnsureAcceptedState(Guid jobId) + { + var path = Path.Combine(jobsDirectory, jobId.ToString("D"), "state.json"); + if (File.Exists(path)) + { + return; + } + var content = JsonSerializer.SerializeToUtf8Bytes(new + { + stage = "accepted", + progress = 0, + detail = "任务已由本机接收", + updated_at = DateTimeOffset.UtcNow, + }, JsonOptions); + try + { + AtomicWrite(path, content, overwrite: false); + } + catch (IOException) when (File.Exists(path)) + { + } + catch (Exception exception) when ( + exception is IOException or UnauthorizedAccessException) + { + Console.Error.WriteLine($"[WARN] Initial local job state failed: {exception.Message}"); + } + } + + private static LocalJobState? ReadState(string path) + { + if (!File.Exists(path)) + { + return null; + } + try + { + using var document = JsonDocument.Parse(File.ReadAllBytes(path)); + var root = document.RootElement; + var stage = ReadString(root, "stage", "accepted"); + var progress = root.TryGetProperty("progress", out var progressValue) + && progressValue.TryGetInt32(out var parsedProgress) + ? parsedProgress + : 0; + return new LocalJobState( + stage, + progress, + ReadString(root, "detail", ""), + ReadDate(root, "updated_at") ?? DateTimeOffset.MinValue); + } + catch (Exception exception) when ( + exception is JsonException or IOException or UnauthorizedAccessException) + { + return null; + } + } + + private static string TerminalDetail(JsonElement terminal, string status, string fallback) + { + if (terminal.TryGetProperty("error", out var error)) + { + var detail = ReadString(error, "detail", ""); + if (!string.IsNullOrWhiteSpace(detail)) + { + return detail; + } + } + return status switch + { + "succeeded" => "软件任务执行成功", + "cancelled" => "任务已取消", + "failed" => "任务执行失败", + _ => fallback, + }; + } + + private static DateTimeOffset LatestWrite(DateTimeOffset current, string path) + { + var writtenAt = new DateTimeOffset(File.GetLastWriteTimeUtc(path)); + return writtenAt > current ? writtenAt : current; + } + + private static string ReadString(JsonElement value, string name, string fallback) => + value.TryGetProperty(name, out var property) + && property.ValueKind == JsonValueKind.String + && !string.IsNullOrWhiteSpace(property.GetString()) + ? property.GetString()! + : fallback; + + private static DateTimeOffset? ReadDate(JsonElement value, string name) => + value.TryGetProperty(name, out var property) + && property.ValueKind == JsonValueKind.String + && DateTimeOffset.TryParse(property.GetString(), out var parsed) + ? parsed + : null; + private static void AtomicWrite(string path, byte[] content, bool overwrite) { Directory.CreateDirectory(Path.GetDirectoryName(path)!); @@ -166,77 +397,184 @@ internal sealed class JobInboxStore(string jobsDirectory) } } - private static bool IsValidRequest(JsonElement request) => - HasOnlyProperties(request, "schema_version", "input", "plot", "output") - && request.TryGetProperty("schema_version", out var schemaVersion) - && schemaVersion.TryGetInt32(out var version) - && version == 1 - && request.TryGetProperty("input", out var input) - && input.ValueKind == JsonValueKind.Object - && HasOnlyProperties(input, "input_id", "sheet") - && input.TryGetProperty("input_id", out var inputId) - && inputId.ValueKind == JsonValueKind.String - && !string.IsNullOrWhiteSpace(inputId.GetString()) - && request.TryGetProperty("plot", out var plot) - && plot.ValueKind == JsonValueKind.Object - && HasOnlyProperties( - plot, "type", "x", "y", "template", "title", "x_axis", "y_axis", "legend", "error_bars") - && plot.TryGetProperty("type", out var plotType) - && plotType.ValueKind == JsonValueKind.String - && PlotTypes.Contains(plotType.GetString() ?? "") - && (!plot.TryGetProperty("title", out var title) - || title.ValueKind == JsonValueKind.String && title.GetString()!.Length <= 500) - && plot.TryGetProperty("x", out var x) - && IsColumnName(x) - && plot.TryGetProperty("y", out var y) - && IsValidYColumns(y) - && (!plot.TryGetProperty("template", out var template) - || template.GetString() == "publication_double_column") - && IsValidAxis(plot, "x_axis") - && IsValidAxis(plot, "y_axis") - && IsValidLegend(plot) - && !plot.TryGetProperty("error_bars", out _) - && request.TryGetProperty("output", out var output) - && output.ValueKind == JsonValueKind.Object - && HasOnlyProperties(output, "formats", "dpi", "capture_screenshots", "record_video") - && output.TryGetProperty("formats", out var formats) - && formats.ValueKind == JsonValueKind.Array - && formats.GetArrayLength() > 0 - && IsValidFormats(formats) - && (!output.TryGetProperty("dpi", out var dpi) - || dpi.TryGetInt32(out var dpiValue) && dpiValue is >= 72 and <= 1200) - && IsOptionalBoolean(output, "capture_screenshots") - && IsOptionalBoolean(output, "record_video") - && (!output.TryGetProperty("record_video", out var recordVideo) - || recordVideo.ValueKind == JsonValueKind.False); + private static bool IsValidRequest(JsonElement request) + { + if (!HasOnlyProperties(request, "schema_version", "inputs", "operation", "outputs") + || !request.TryGetProperty("schema_version", out var schemaVersion) + || !schemaVersion.TryGetInt32(out var version) + || version != 2 + || !request.TryGetProperty("inputs", out var inputs) + || !IsValidInputBindings(inputs) + || !request.TryGetProperty("operation", out var operation) + || operation.ValueKind != JsonValueKind.Object + || !HasOnlyProperties(operation, "plot") + || !operation.TryGetProperty("plot", out var plot) + || !IsValidPlot(plot, inputs) + || !request.TryGetProperty("outputs", out var outputs)) + { + return false; + } + return IsValidOutputs(outputs); + } + + private static bool IsValidOutputs(JsonElement outputs) + { + if (outputs.ValueKind != JsonValueKind.Array || outputs.GetArrayLength() is < 1 or > 16) + { + return false; + } + var keys = new HashSet(StringComparer.Ordinal); + var identities = new HashSet(StringComparer.Ordinal); + foreach (var output in outputs.EnumerateArray()) + { + if (output.ValueKind != JsonValueKind.Object + || !HasOnlyProperties(output, "key", "type", "format", "options") + || !output.TryGetProperty("key", out var keyValue) + || keyValue.GetString() is not { } key + || !output.TryGetProperty("type", out var typeValue) + || typeValue.GetString() is not { } outputType + || !output.TryGetProperty("format", out var formatValue) + || formatValue.GetString() is not { } format) + { + return false; + } + var expectedKey = (outputType, format) switch + { + ("project", "opju") => "project", + ("figure", "png") => "figure_png", + ("figure", "svg") => "figure_svg", + ("figure", "pdf") => "figure_pdf", + _ => "", + }; + if (key != expectedKey || !keys.Add(key) || !identities.Add($"{outputType}\0{format}")) + { + return false; + } + if (format == "png") + { + if (output.TryGetProperty("options", out var options) + && (options.ValueKind != JsonValueKind.Object + || !HasOnlyProperties(options, "dpi") + || !options.TryGetProperty("dpi", out var dpi) + || !dpi.TryGetInt32(out var dpiValue) + || dpiValue is < 72 or > 1200)) + { + return false; + } + } + else if (output.TryGetProperty("options", out _)) + { + return false; + } + } + return true; + } + + private static bool IsValidInputBindings(JsonElement inputs) + { + if (inputs.ValueKind != JsonValueKind.Array || inputs.GetArrayLength() is < 1 or > 16) + { + return false; + } + var keys = new HashSet(StringComparer.Ordinal); + foreach (var input in inputs.EnumerateArray()) + { + if (input.ValueKind != JsonValueKind.Object + || !HasOnlyProperties(input, "key", "artifact_id", "selector") + || !input.TryGetProperty("key", out var keyValue) + || keyValue.GetString() is not { } key + || !IsInputKey(key) + || !keys.Add(key) + || !input.TryGetProperty("artifact_id", out var artifactId) + || !Guid.TryParse(artifactId.GetString(), out _) + || input.TryGetProperty("selector", out var selector) + && !IsValidSelector(selector)) + { + return false; + } + } + return true; + } + + private static bool IsValidPlot(JsonElement plot, JsonElement inputs) + { + if (plot.ValueKind != JsonValueKind.Object + || !HasOnlyProperties( + plot, "type", "series", "template", "title", "x_axis", "y_axis", "legend", "error_bars") + || !plot.TryGetProperty("type", out var plotType) + || !PlotTypes.Contains(plotType.GetString() ?? "") + || plot.TryGetProperty("title", out var title) + && (title.ValueKind != JsonValueKind.String || title.GetString()!.Length > 500) + || !plot.TryGetProperty("series", out var series) + || series.ValueKind != JsonValueKind.Array + || series.GetArrayLength() is < 1 or > 16 + || plot.TryGetProperty("template", out var template) + && template.GetString() != "publication_double_column" + || !IsValidAxis(plot, "x_axis") + || !IsValidAxis(plot, "y_axis") + || !IsValidLegend(plot) + || plot.TryGetProperty("error_bars", out _)) + { + return false; + } + var inputKeys = inputs.EnumerateArray() + .Select(item => item.GetProperty("key").GetString()!) + .ToHashSet(StringComparer.Ordinal); + var identities = new HashSet(StringComparer.Ordinal); + var usedInputs = new HashSet(StringComparer.Ordinal); + var labels = new Dictionary(StringComparer.Ordinal); + foreach (var item in series.EnumerateArray()) + { + if (item.ValueKind != JsonValueKind.Object + || !HasOnlyProperties(item, "input", "x", "y", "label") + || !item.TryGetProperty("input", out var input) + || input.GetString() is not { } inputKey + || !inputKeys.Contains(inputKey) + || !item.TryGetProperty("x", out var x) + || !IsColumnName(x) + || !item.TryGetProperty("y", out var y) + || !IsColumnName(y) + || item.TryGetProperty("label", out var label) + && (label.ValueKind != JsonValueKind.String + || label.GetString()!.Length is < 1 or > 200) + || !identities.Add($"{inputKey}\0{x.GetString()}\0{y.GetString()}")) + { + return false; + } + usedInputs.Add(inputKey); + var labelKey = $"{inputKey}\0{y.GetString()}"; + var effectiveLabel = item.TryGetProperty("label", out var seriesLabel) + ? seriesLabel.GetString()! + : y.GetString()!; + if (labels.TryGetValue(labelKey, out var existingLabel) + && existingLabel != effectiveLabel) + { + return false; + } + labels[labelKey] = effectiveLabel; + } + return usedInputs.SetEquals(inputKeys); + } + + private static bool IsValidSelector(JsonElement selector) => + selector.ValueKind == JsonValueKind.Object + && HasOnlyProperties(selector, "sheet") + && selector.TryGetProperty("sheet", out var sheet) + && sheet.ValueKind == JsonValueKind.String + && sheet.GetString()!.Length is >= 1 and <= 128; + + private static bool IsInputKey(string key) => + key.Length is >= 1 and <= 32 + && key[0] is >= 'a' and <= 'z' + && key.All(character => + character is >= 'a' and <= 'z' + || character is >= '0' and <= '9' + || character == '_'); private static bool IsColumnName(JsonElement value) => value.ValueKind == JsonValueKind.String && value.GetString() is { Length: >= 1 and <= 128 }; - private static bool IsValidFormats(JsonElement formats) - { - var values = formats.EnumerateArray().ToArray(); - return values.All(item => - item.ValueKind == JsonValueKind.String - && OutputFormats.Contains(item.GetString() ?? "")) - && values.Select(item => item.GetString()).Distinct(StringComparer.Ordinal).Count() - == values.Length; - } - - private static bool IsValidYColumns(JsonElement value) - { - if (IsColumnName(value)) return true; - if (value.ValueKind != JsonValueKind.Array - || value.GetArrayLength() is < 1 or > 16) - { - return false; - } - var names = value.EnumerateArray().Select(item => item.GetString()).ToArray(); - return value.EnumerateArray().All(IsColumnName) - && names.Distinct(StringComparer.Ordinal).Count() == names.Length; - } - private static bool IsValidAxis(JsonElement plot, string name) { if (!plot.TryGetProperty(name, out var axis)) return true; @@ -258,36 +596,94 @@ internal sealed class JobInboxStore(string jobsDirectory) || position.GetString() == "top_right"); } - private static bool IsOptionalBoolean(JsonElement value, string name) => - !value.TryGetProperty(name, out var property) - || property.ValueKind is JsonValueKind.True or JsonValueKind.False; - - private static bool IsValidInputTransfer(JsonElement transfer) => - transfer.ValueKind == JsonValueKind.Object - && HasOnlyProperties(transfer, "artifact_id", "filename", "size_bytes", "sha256", "download_path") - && transfer.TryGetProperty("artifact_id", out var artifactId) - && Guid.TryParse(artifactId.GetString(), out _) - && transfer.TryGetProperty("filename", out var filename) - && filename.ValueKind == JsonValueKind.String - && Path.GetFileName(filename.GetString()) == filename.GetString() - && transfer.TryGetProperty("size_bytes", out var size) - && size.TryGetInt64(out var sizeBytes) - && sizeBytes is >= 0 and <= 104_857_600 - && transfer.TryGetProperty("sha256", out var sha) - && sha.GetString() is { Length: 64 } - && transfer.TryGetProperty("download_path", out var downloadPath) - && downloadPath.GetString()?.StartsWith("/v1/software-jobs/", StringComparison.Ordinal) == true - && downloadPath.GetString()?.EndsWith("/input", StringComparison.Ordinal) == true; - - internal string InputPath(RecoverableJob job) + private static bool IsValidInputTransfers(JsonElement transfers, Guid jobId) { - if (job.InputTransfer is not JsonElement transfer - || !IsValidInputTransfer(transfer)) + if (transfers.ValueKind != JsonValueKind.Array + || transfers.GetArrayLength() is < 1 or > 16) { - throw new InvalidDataException("Stored input transfer is invalid."); + return false; } + var keys = new HashSet(StringComparer.Ordinal); + long total = 0; + foreach (var transfer in transfers.EnumerateArray()) + { + if (transfer.ValueKind != JsonValueKind.Object + || !HasOnlyProperties( + transfer, "key", "artifact_id", "filename", "size_bytes", "sha256", + "selector", "download_path") + || !transfer.TryGetProperty("key", out var keyValue) + || keyValue.GetString() is not { } key + || !IsInputKey(key) + || !keys.Add(key) + || !transfer.TryGetProperty("artifact_id", out var artifactId) + || !Guid.TryParse(artifactId.GetString(), out _) + || !transfer.TryGetProperty("filename", out var filename) + || filename.ValueKind != JsonValueKind.String + || Path.GetFileName(filename.GetString()) != filename.GetString() + || !transfer.TryGetProperty("size_bytes", out var size) + || !size.TryGetInt64(out var sizeBytes) + || sizeBytes is < 0 or > 104_857_600 + || !transfer.TryGetProperty("sha256", out var sha) + || sha.GetString() is not { Length: 64 } + || transfer.TryGetProperty("selector", out var selector) + && !IsValidSelector(selector) + || !transfer.TryGetProperty("download_path", out var downloadPath) + || downloadPath.GetString() + != $"/v1/software-jobs/{jobId:D}/inputs/{key}") + { + return false; + } + total += sizeBytes; + } + return total <= 536_870_912; + } + + private static bool InputsMatchTransfers(JsonElement request, JsonElement transfers) + { + var bindings = request.GetProperty("inputs").EnumerateArray().ToDictionary( + item => item.GetProperty("key").GetString()!, StringComparer.Ordinal); + if (bindings.Count != transfers.GetArrayLength()) + { + return false; + } + foreach (var transfer in transfers.EnumerateArray()) + { + var key = transfer.GetProperty("key").GetString()!; + if (!bindings.TryGetValue(key, out var binding) + || binding.GetProperty("artifact_id").GetString() + != transfer.GetProperty("artifact_id").GetString()) + { + return false; + } + var hasBindingSelector = binding.TryGetProperty("selector", out var bindingSelector); + var hasTransferSelector = transfer.TryGetProperty("selector", out var transferSelector); + if (hasBindingSelector != hasTransferSelector + || hasBindingSelector + && bindingSelector.GetProperty("sheet").GetString() + != transferSelector.GetProperty("sheet").GetString()) + { + return false; + } + } + return true; + } + + internal IReadOnlyList InputTransfers(RecoverableJob job) + { + if (job.InputTransfers is not JsonElement transfers + || !IsValidInputTransfers(transfers, job.JobId)) + { + throw new InvalidDataException("Stored input transfers are invalid."); + } + return transfers.EnumerateArray().Select(item => item.Clone()).ToArray(); + } + + internal string InputPath(RecoverableJob job, JsonElement transfer) + { + var key = transfer.GetProperty("key").GetString()!; var filename = transfer.GetProperty("filename").GetString()!; - return Path.Combine(jobsDirectory, job.JobId.ToString("D"), "input", filename); + return Path.Combine( + jobsDirectory, job.JobId.ToString("D"), "input", key, filename); } internal void WriteTerminal( @@ -330,7 +726,7 @@ internal sealed record RecoverableJob( Guid JobId, Guid LeaseId, string RequestDigest, - JsonElement? InputTransfer, + JsonElement? InputTransfers, JsonElement? Terminal, bool UploadComplete); diff --git a/windows-node/Zcbot.WindowsNode/JobInputDownloader.cs b/windows-node/Zcbot.WindowsNode/JobInputDownloader.cs index d17b0d6..9a072fb 100644 --- a/windows-node/Zcbot.WindowsNode/JobInputDownloader.cs +++ b/windows-node/Zcbot.WindowsNode/JobInputDownloader.cs @@ -8,20 +8,32 @@ internal sealed class JobInputDownloader(NodeConfig config, JobInboxStore inbox) { internal async Task DownloadAsync(RecoverableJob job, CancellationToken cancellationToken) { - if (job.InputTransfer is not JsonElement transfer) + var transfers = inbox.InputTransfers(job); + for (var index = 0; index < transfers.Count; index++) { - throw new InvalidDataException("Job input transfer is missing."); + inbox.WriteState( + job, + "downloading_inputs", + Math.Max(0, index * 5 / transfers.Count), + $"正在下载输入 {index + 1}/{transfers.Count}"); + await DownloadOneAsync(job, transfers[index], cancellationToken); } + } + + private async Task DownloadOneAsync( + RecoverableJob job, + JsonElement transfer, + CancellationToken cancellationToken) + { var downloadPath = transfer.GetProperty("download_path").GetString()!; if (!downloadPath.StartsWith("/v1/software-jobs/", StringComparison.Ordinal) - || !downloadPath.EndsWith("/input", StringComparison.Ordinal) || !Uri.TryCreate(downloadPath, UriKind.Relative, out var relativeUri)) { throw new InvalidDataException("Job input download path is invalid."); } var expectedSize = transfer.GetProperty("size_bytes").GetInt64(); var expectedSha256 = transfer.GetProperty("sha256").GetString()!; - var destination = inbox.InputPath(job); + var destination = inbox.InputPath(job, transfer); if (File.Exists(destination)) { await VerifyExistingAsync(destination, expectedSize, expectedSha256, cancellationToken); diff --git a/windows-node/Zcbot.WindowsNode/JobMonitorModels.cs b/windows-node/Zcbot.WindowsNode/JobMonitorModels.cs new file mode 100644 index 0000000..80383f2 --- /dev/null +++ b/windows-node/Zcbot.WindowsNode/JobMonitorModels.cs @@ -0,0 +1,22 @@ +namespace Zcbot.WindowsNode; + +internal sealed record LocalJobState( + string Stage, + int Progress, + string Detail, + DateTimeOffset UpdatedAt); + +internal sealed record JobDisplaySnapshot( + Guid JobId, + string Capability, + string Title, + string InputFilename, + string Stage, + int Progress, + string Detail, + DateTimeOffset AcceptedAt, + DateTimeOffset UpdatedAt, + bool UploadComplete) +{ + internal bool IsActive => Stage is not ("succeeded" or "failed" or "cancelled"); +} diff --git a/windows-node/Zcbot.WindowsNode/NodeConfigStore.cs b/windows-node/Zcbot.WindowsNode/NodeConfigStore.cs index c6e28ec..1fc13ea 100644 --- a/windows-node/Zcbot.WindowsNode/NodeConfigStore.cs +++ b/windows-node/Zcbot.WindowsNode/NodeConfigStore.cs @@ -57,7 +57,7 @@ internal sealed class NodeConfigStore(NodePaths paths) || string.IsNullOrWhiteSpace(stored.NodeName) || string.IsNullOrWhiteSpace(token) || stored.Capabilities.Count == 0 - || stored.Capabilities.Any(item => item != "origin.plot@v1")) + || stored.Capabilities.Any(item => item != "origin.plot@v2")) { throw new NodeConfigurationException("Node configuration contains an invalid identity or capability."); } diff --git a/windows-node/Zcbot.WindowsNode/NodeConnectionLoop.cs b/windows-node/Zcbot.WindowsNode/NodeConnectionLoop.cs index ec0e292..32446a3 100644 --- a/windows-node/Zcbot.WindowsNode/NodeConnectionLoop.cs +++ b/windows-node/Zcbot.WindowsNode/NodeConnectionLoop.cs @@ -163,7 +163,7 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? job_id = job.JobId, lease_id = job.LeaseId, request_digest = job.RequestDigest, - stage = "waiting_input", + stage = "downloading_inputs", progress = 0, metrics = new { }, }, cancellationToken); @@ -242,7 +242,7 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? job_id = offerResult.JobId, lease_id = offerResult.LeaseId, request_digest = offerResult.RequestDigest, - stage = "waiting_input", + stage = "downloading_inputs", progress = 0, metrics = new { }, }, cancellationToken); @@ -322,6 +322,7 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? { if (job.Terminal is null) { + jobInbox.WriteState(job, "downloading_inputs", 0, "正在下载并校验输入文件"); await inputDownloader.DownloadAsync(job, CancellationToken.None); var afterDownload = jobInbox.ReadRecoverableJobs() .Single(item => item.JobId == job.JobId); @@ -330,6 +331,7 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? await TrySendAsync(socket, "job_terminal", cancelledTerminal); return; } + jobInbox.WriteState(job, "ready_to_run", 5, "输入文件已就绪,准备启动软件"); await TrySendAsync(socket, "job_state", new { job_id = job.JobId, @@ -337,14 +339,20 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? request_digest = job.RequestDigest, stage = "ready_to_run", progress = 5, - metrics = new { input_bytes = job.InputTransfer?.GetProperty("size_bytes").GetInt64() }, + metrics = new + { + input_count = jobInbox.InputTransfers(job).Count, + input_bytes = jobInbox.InputTransfers(job) + .Sum(item => item.GetProperty("size_bytes").GetInt64()), + }, }); + jobInbox.WriteState(job, "software_running", 10, "Origin 正在生成图形"); await TrySendAsync(socket, "job_state", new { job_id = job.JobId, lease_id = job.LeaseId, request_digest = job.RequestDigest, - stage = "origin_running", + stage = "software_running", progress = 10, metrics = new { }, }); @@ -358,9 +366,17 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? } if (terminal.GetProperty("status").GetString() != "succeeded") { + var terminalStatus = terminal.GetProperty("status").GetString() ?? "failed"; + var terminalDetail = terminal.TryGetProperty("error", out var error) + && error.TryGetProperty("detail", out var errorDetail) + ? errorDetail.GetString() ?? "任务执行失败" + : "任务执行失败"; + jobInbox.WriteState( + refreshed, terminalStatus, terminalStatus == "cancelled" ? 0 : 10, terminalDetail); await TrySendAsync(socket, "job_terminal", terminal); return; } + jobInbox.WriteState(refreshed, "uploading_outputs", 90, "软件执行完成,正在上传结果"); await TrySendAsync(socket, "job_state", new { job_id = job.JobId, @@ -371,6 +387,7 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? metrics = new { }, }); await outputUploader.UploadAsync(refreshed); + jobInbox.WriteState(refreshed, "succeeded", 100, "结果已上传并由云端确认"); } catch (Exception exception) when ( exception is HttpRequestException @@ -384,6 +401,11 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? if (current.Terminal is JsonElement terminal && terminal.GetProperty("status").GetString() == "succeeded") { + jobInbox.WriteState( + current, + "uploading_outputs", + 90, + $"结果上传暂缓:{exception.Message}"); Console.Error.WriteLine($"[WARN] Output upload deferred: {exception.Message}"); return; } @@ -392,6 +414,7 @@ internal sealed class NodeConnectionLoop(NodeConfig config, Action? "failed", "INPUT_DOWNLOAD_FAILED", exception.Message[..Math.Min(exception.Message.Length, 500)]); + jobInbox.WriteState(job, "failed", 0, exception.Message); var failedTerminal = jobInbox.ReadRecoverableJobs() .Single(item => item.JobId == job.JobId).Terminal; if (failedTerminal is JsonElement payload) diff --git a/windows-node/Zcbot.WindowsNode/OriginRuntimeProbe.cs b/windows-node/Zcbot.WindowsNode/OriginRuntimeProbe.cs index ee14d30..1d2ba7f 100644 --- a/windows-node/Zcbot.WindowsNode/OriginRuntimeProbe.cs +++ b/windows-node/Zcbot.WindowsNode/OriginRuntimeProbe.cs @@ -54,7 +54,7 @@ internal static class OriginRuntimeProbe } private static OriginRuntimeStatus Status(string? version, string health, string detail) => - new("OriginPro", version, "0.2.0", health, detail); + new("OriginPro", version, "0.3.0", health, detail); private static string? FindInstalledVersion() { diff --git a/windows-node/origin-worker/worker.py b/windows-node/origin-worker/worker.py index 51947e7..d5cb2ed 100644 --- a/windows-node/origin-worker/worker.py +++ b/windows-node/origin-worker/worker.py @@ -1,4 +1,4 @@ -"""Fixed Origin adapter for origin.plot@v1. +"""Fixed Origin adapter for origin.plot@v2. This process accepts exactly one argument: a Node-created job directory. It never installs packages, evaluates user code, downloads data, or resolves paths from the @@ -104,6 +104,35 @@ def _axis_title(axis: Any, fallback: str) -> str: return f"{title} ({unit})" if unit else title +def _input_file(job_dir: Path, key: str) -> Path: + directory = job_dir / "input" / key + files = [path for path in directory.iterdir() if path.is_file() and not path.name.startswith(".")] + if len(files) != 1: + raise ValueError(f"INPUT_FILE_COUNT_INVALID:{key}") + return files[0] + + +def _resolve_series( + input_data: dict[str, tuple[list[str], list[list[Any]]]], + series_specs: list[dict[str, Any]], +) -> tuple[list[tuple[str, int, int, str | None]], dict[tuple[str, int], str]]: + resolved: list[tuple[str, int, int, str | None]] = [] + labels: dict[tuple[str, int], str] = {} + for series in series_specs: + input_key = series["input"] + headers, _ = input_data[input_key] + x_index = _column_index(headers, series["x"], "x") + y_index = _column_index(headers, series["y"], "y") + label = series.get("label") + label_key = (input_key, y_index) + effective_label = label or series["y"] + if label_key in labels and labels[label_key] != effective_label: + raise ValueError("SERIES_LABEL_CONFLICT") + labels[label_key] = effective_label + resolved.append((input_key, x_index, y_index, label)) + return resolved, labels + + def _validate_artifact(path: Path, extension: str) -> None: if not path.is_file() or path.stat().st_size == 0: raise RuntimeError(f"{extension.upper()}_EXPORT_EMPTY") @@ -122,21 +151,21 @@ def run(job_dir: Path) -> list[dict[str, Any]]: job_dir = job_dir.resolve(strict=True) request_record = json.loads((job_dir / "request" / "request.json").read_text(encoding="utf-8")) request = request_record["request"] - input_files = [path for path in (job_dir / "input").iterdir() if path.is_file() and not path.name.startswith(".")] - if len(input_files) != 1: - raise ValueError("INPUT_FILE_COUNT_INVALID") - headers, rows = _read_rows(input_files[0], request["input"].get("sheet")) - plot_spec = request["plot"] + input_specs = request["inputs"] + input_files = {item["key"]: _input_file(job_dir, item["key"]) for item in input_specs} + input_data = { + item["key"]: _read_rows( + input_files[item["key"]], + (item.get("selector") or {}).get("sheet"), + ) + for item in input_specs + } + plot_spec = request["operation"]["plot"] plot_type = plot_spec["type"] if plot_type not in PLOT_TYPES: raise ValueError("PLOT_TYPE_NOT_IMPLEMENTED") - x_index = _column_index(headers, plot_spec.get("x"), "x") - y_names = plot_spec.get("y") - if isinstance(y_names, str): - y_names = [y_names] - if not isinstance(y_names, list) or not y_names: - raise ValueError("Y_COLUMNS_REQUIRED") - y_indexes = [_column_index(headers, name, "y") for name in y_names] + series_specs = plot_spec["series"] + resolved_series, labels = _resolve_series(input_data, series_specs) import originpro as op @@ -145,22 +174,37 @@ def run(job_dir: Path) -> list[dict[str, Any]]: op.set_show(False) try: op.new() - worksheet = op.new_sheet("w", lname="Data") - for index, header in enumerate(headers): - worksheet.from_list(index, [row[index] if index < len(row) else None for row in rows], lname=header) + worksheets: dict[str, Any] = {} + for input_spec in input_specs: + input_key = input_spec["key"] + headers, rows = input_data[input_key] + worksheet = op.new_sheet("w", lname=input_key) + worksheets[input_key] = worksheet + for index, header in enumerate(headers): + column_label = labels.get((input_key, index), header) + worksheet.from_list( + index, + [row[index] if index < len(row) else None for row in rows], + lname=column_label, + ) graph = op.new_graph(template={"line": "line", "scatter": "scatter", "line_scatter": "linesymb"}[plot_type]) layer = graph[0] - for y_index in y_indexes: - layer.add_plot(worksheet, coly=y_index, colx=x_index, type=PLOT_TYPES[plot_type]) + for input_key, x_index, y_index, _ in resolved_series: + layer.add_plot( + worksheets[input_key], coly=y_index, colx=x_index, type=PLOT_TYPES[plot_type] + ) layer.rescale() - layer.axis("x").title = _axis_title(plot_spec.get("x_axis"), str(plot_spec.get("x") or "X")) + layer.axis("x").title = _axis_title( + plot_spec.get("x_axis"), str(series_specs[0].get("x") or "X") + ) layer.axis("y").title = _axis_title(plot_spec.get("y_axis"), "Y") if plot_spec.get("title"): title = layer.add_label(str(plot_spec["title"])) title.set_int("fsize", 18) title.set_int("left", 2200) title.set_int("top", 120) - formats = request["output"]["formats"] + requested_outputs = request["outputs"] + formats = [item["format"] for item in requested_outputs] if any(item not in FORMATS for item in formats): raise ValueError("OUTPUT_FORMAT_UNSUPPORTED") artifacts: list[dict[str, Any]] = [] @@ -170,7 +214,8 @@ def run(job_dir: Path) -> list[dict[str, Any]]: _validate_artifact(project, "opju") artifacts.append(_manifest(project, "application/x-origin-project")) media = {"png": "image/png", "svg": "image/svg+xml", "pdf": "application/pdf"} - dpi = request["output"].get("dpi", 300) + png_output = next((item for item in requested_outputs if item["format"] == "png"), None) + dpi = (png_output.get("options") or {}).get("dpi", 300) if png_output else 300 if not isinstance(dpi, int) or isinstance(dpi, bool) or not 72 <= dpi <= 1200: raise ValueError("OUTPUT_DPI_INVALID") pixel_width = round(dpi * 160 / 25.4) @@ -192,12 +237,20 @@ def run(job_dir: Path) -> list[dict[str, Any]]: except PackageNotFoundError: originpro_version = "embedded" provenance = { - "adapter_version": "0.2.0", + "adapter_version": "0.3.0", "originpro_version": originpro_version, "request_digest": request_record["request_digest"], - "input_sha256": _file_sha256(input_files[0]), - "requested_dpi": dpi, - "png_pixel_width": pixel_width, + "inputs": [ + { + "key": item["key"], + "filename": input_files[item["key"]].name, + "sha256": _file_sha256(input_files[item["key"]]), + } + for item in input_specs + ], + "outputs": requested_outputs, + "requested_dpi": dpi if png_output else None, + "png_pixel_width": pixel_width if png_output else None, } plot_spec_path = output / "plot-spec.json" provenance_path = output / "provenance.json"