diff --git a/CHANGELOG.md b/CHANGELOG.md index e44f8a2..556ff0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ ## Unreleased +- Origin 新增受控科研分析，可进行数据体检、归一化、平滑、求导、积分和线性拟合，并交付结果表、诊断信息及可继续检查的 Origin 工程。 + - 平台托管的文献、内部材料库和 Materials Project 来源现在使用来源明确的工具，并可在单个来源不可用时继续使用其他来源；访问凭据不进入任务沙箱。 ## 0.68.0 — 2026-08-24 diff --git a/DESIGN.md b/DESIGN.md index 866c0c1..8c211f9 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -483,6 +483,8 @@ Node 通过 `Authorization: Bearer` 与 `X-Node-Id` 建立 `/v1/software-nodes/c 第四阶段落地固定 Origin Worker：Node 仅从管理员安装的固定 Python 运行时启动随程序发布的 `worker.py`，参数只有本机 job 目录；请求不能指定脚本、解释器或文件路径。Worker 使用 `originpro` 生成 OPJU、PNG、SVG、PDF、plot spec 和 provenance，校验产物签名并原子写入终态。`origin.plot@v2` 保持单一外层契约，旧单图 `series[]` 以 `x/y/z/y_error` 统一表达数据角色，再按 `plot.type` 判别必需角色；当前覆盖折线、散点、线点、柱/条形、分组柱形、Y 误差棒、等高线、三维曲面、三元图和规则网格热图，并以可选的 `canvas`、轴排版、图例、标题和 series style 统一表达出版级尺寸与样式。新增复合图不继续枚举 `dual_axis_line` 等组合类型，而使用 `multi_panel` feature：`panels[]` 显式描述 1–4 个二维 panel，每条系列用 `kind` 选择基础二维图型、用 `y_axis` 绑定左右轴，并可提供对称 `x_error/y_error`；`layout` 只允许 1×1、1×2、2×1、2×2 四种受控布局。Worker 从固定基础二维图创建受控原生图层网格及 right-Y 图层，不依赖带隐式主题分组的 panel 模板，也不接受请求传入模板名；Origin 不支持同一 3D 图窗多图层，因此 contour、surface、ternary 和 heatmap 不混入本阶段 panel。旧 XY 请求原样兼容。进程内 pipeline 按 job 去重，并脱离单次 WebSocket 的取消令牌运行；连接中断只延迟状态/终态上报。Node 进程若在 Worker 启动后重启，则保守失败而不重复驱动 Origin，避免无法证明的双执行。 +Origin 科研分析以独立 `origin.analysis@v1` capability 承载，不把数据处理、拟合和统计继续塞入 `origin.plot@v2`。首批白名单 operation 为 `data_check/normalize/smooth/differentiate/integrate/linear_fit`：合同绑定单个 CSV/XLSX/JSON artifact 和显式 X/Y 列，固定 Worker 使用版本化数值算法生成结果表与机器可读诊断，再把原始数据、结果和诊断写入可编辑 OPJU；线性拟合额外调用 Origin 内置 `LinearFit` 生成原生报告。标准化数值结果以独立 oracle 为发布依据，Origin 报告用于工程内继续检查和编辑，避免把正确性绑定到本机语言相关的层级报告表解析。输出固定包含 OPJU、CSV、diagnostics、analysis spec 与 provenance，可选 XLSX；分析 capability 为无状态一次性 Job，结果 CSV/XLSX 可登记为后续 plot Job 输入，不与绘图混在同一请求或 Workspace 事务中。`data_check` 命中请求指定的问题时返回 `diagnostics.passed=false` 而非让 Job 失败，使 Agent 能先体检再决定是否占用后续分析资源。请求仍不能提供 Python、LabTalk、函数表达式、模板或路径；两份 Origin adapter 共享同一受管 `origin` runtime，并由整机单执行槽保守串行。 + Recipe 是专业软件的声明式目标状态，不是任意脚本或逐次鼠标命令。`origin.plot@v2` 以新增 `plot.type=recipe + recipe_version=1` 复用已验证的 `layout/panels/series/axis/legend` 组合结构；旧 `multi_panel` 与 Recipe 进入同一个受控执行器，单图和特殊统计图继续保留兼容入口。模型可以组合契约允许的原子图形能力，但不能提供 Python、LabTalk、解释器、模板名或文件路径。其他软件可采用同一通用 Job 外壳和各自的声明式 Recipe，不建设跨软件万能 DSL；API/SDK executor 优先，未来的 UI Automation 或 Computer Use 仅作为 adapter 内部执行后端，不改变云端 Recipe。 Blender 入口为 `blender.scene.author@v3`，唯一场景类型是 `recipe`；它是 Blender 专属的声明式静态场景目标状态，不是跨软件万能 DSL，也不是 Python、逐步按键或任意节点脚本。Recipe v1 组合带稳定 ID、父子层级和集合归属的 geometry/generator/text/empty 对象：geometry 覆盖 box/cylinder/cone/sphere/torus/plane、二维轮廓拉伸、轴向旋转、路径扫掠和有壁厚直管，受控 modifier 覆盖 boolean/bevel/mirror/array/solidify，并统一声明材质、变换、灯光、世界背景和 1–8 个相机视图；`rotary_kiln` 只是 generator 组件，可与其他对象处于同一场景，不再占据顶层 capability。adapter 1.0.0 把合同校验后的 Recipe 编译为固定 `bpy` 调用，限制请求 256 KiB、200 个声明对象、展开后 1000 个 Blender 对象、100 万 mesh polygon、材质/灯光/视图和修改器数量；Worker 还机械校验 ID 唯一性、引用、父子关系、布尔目标、管壁和请求预览的一致性。输出固定包含 `.blend`、首张 PNG、场景清单和溯源，可再选择最多七张 PNG 与 GLB；预览用 `preview_01`–`preview_08` 稳定槽位，视图业务名称记录在 manifest。 diff --git a/PROGRESS.md b/PROGRESS.md index 231d8e3..99e6487 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -20,6 +20,8 @@ --- ## 已完成关键能力 +- **08-26 / Unreleased / Origin 科研分析 MVP**：新增非默认的 `origin.analysis@v1` / adapter 0.1.0，与稳定的 `origin.plot@v2` 分离；首批提供数据体检、最大值/面积/参考点归一化、固定局部多项式平滑、一/二阶导数、区间/累计梯形积分和带 95% 置信区间的线性拟合，固定输出可编辑 OPJU、CSV、诊断、分析规格与溯源，可选 XLSX，线性拟合工程同时保留 Origin 原生报告。六个合成真值用例在 Origin 2024 / originpro 1.1.15 / numpy 2.2.6 上完成生产 Worker、数值 oracle、OPJU 重开和进程释放门；前三例来自组合执行的逐例通过记录，后三例分别通过，目标机 COM 完整退出约需 2 分钟。Origin/合同/Job/Node 专项 159 项 unittest、Release .NET build、完整 Node publish 与独立 adapter ZIP 校验通过，未连接或写入数据库。 + - **08-26 / Unreleased / platform_sources bounded context**：将 paper_server、内部材料库和 Materials Project 收敛为独立顶层包，以显式可信 Provider 列表统一可用性与生命周期，单来源配置/装配失败不阻断其他来源；`core/tool_registry.py` 只保留一个构建入口，与用户连接 `external_systems` 完全独立。模型工具统一改为来源明确的 `paper_server_*`、`materials_library_*`、`materials_project_*`，保留材料库批量检索、MP 专用 SDK/CIF/entries 语义及 paper_server 安全下载边界；部署环境变量仍是事实源，无数据库迁移、无版本提升。 - **08-26 / Unreleased / paper_server 宿主侧受控访问**：新增按配置注册的 `paper_server_search` / `paper_server_get` / `paper_server_fetch` typed tools，`publication_type` 直接映射后端 `type` 过滤且保留开放原始类型；API Key 从 `run_python` 和 Docker sandbox 透传中移除，下载增加同源 URL、100 MiB 上限和原子落盘约束。literature / brief 指引、smoke、RUN、Skill 清单及回归测试同步迁移，线上只读 `type=book` 验证通过，不提升版本号。 diff --git a/RUN.md b/RUN.md index 6a2525b..1530dd2 100644 --- a/RUN.md +++ b/RUN.md @@ -1160,6 +1160,12 @@ Web 用户登录后，文件栏 Job 中心会聚合本人最近任务。活动 Origin adapter 1.1.0 起随包提供固定真机 QA harness，但暂不接入 Node UI。在专用测试节点确认没有活动 Job 后，从 PowerShell 使用 Origin 卡片已安装的受管解释器运行：`\runtimes\origin\Scripts\python.exe \adapters\origin.plot@v2\acceptance.py --work-root \origin-acceptance-YYYYMMDD`。`--work-root` 必须指向尚不存在的新目录；adapter 1.3.0 默认依次执行 annotations、recipe_2x2、heatmap、surface_3d、stacked、stacked_line、violin、band，也可重复传入 `--case ` 只跑指定用例。每个用例先通过生产 Worker 子进程生成 OPJU/PNG/SVG/PDF，再由独立 Origin 会话重开工程，核对图层、系列、工作表/矩阵和数值 oracle，复导三种图件，并等待 Origin 进程退出；必要时可用 `--release-wait ` 调整释放等待时间。最终报告位于 `\acceptance-report.json`。输入均为脚本生成的合成数据，但 PNG/SVG/PDF 仍需人工完成视觉 gate 后才能判定目标机器通过。 +`origin.analysis@v1` 是与绘图分离的无状态科研分析入口，首批 operation 为 `data_check`、`normalize`、`smooth`、`differentiate`、`integrate`、`linear_fit`。请求只绑定一个 CSV/XLSX/JSON artifact；除 `data_check` 可只检查整表外，其余操作必须显式提供 `input/x/y`。数据体检通过 `parameters.fail_on` 指定需要阻断后续流程的问题，命中时 Job 仍成功并在 `diagnostics.json` 返回 `passed=false`；调用方应读取诊断后再决定是否提交分析。归一化的 `max_abs` 使用最大绝对值，`area` 使用有符号梯形积分，`reference` 在数据范围内按 X 线性插值；平滑固定为完整奇数窗口的局部多项式，求导支持一/二阶且接受非等间距 X，积分输出累计值及可选 `from/to` 区间值，线性拟合输出参数、95% t 置信区间、R²、RMSE、拟合值和残差，并在 OPJU 内保留 Origin 原生线性拟合报告。所有数值操作拒绝缺失、非有限、重复或非严格递增的 X/Y 数据，不做隐式排序或删行。 + +分析任务固定发布 `analysis.opju`、`result-table.csv` 和 `diagnostics.json`，隐藏保存完整 analysis spec 与 provenance；请求 `outputs=[{"key":"result_workbook","type":"table","format":"xlsx"}]` 时再增加 XLSX。CSV/XLSX 可登记为后续 `origin.plot@v2` Job 输入，但分析与绘图不合并成同一个请求。`origin.analysis@v1` 与 `origin.plot@v2` 共享 Origin 软件位置及 `runtimes\origin`，不会在 UI 生成第二张软件卡；完整 Node 包会同时携带两份 adapter。只更新分析 adapter 时运行 `windows-node\package-origin-analysis-adapter.bat`，退出 Node 后整体替换 `adapters\origin.analysis@v1\` 再启动；该独立包假定目标节点已经安装 Origin 受管 runtime。 + +分析 adapter 自带六个固定真机用例。运行：`\runtimes\origin\Scripts\python.exe \adapters\origin.analysis@v1\acceptance.py --work-root \origin-analysis-acceptance-YYYYMMDD --release-wait 180`。可用 `--case ` 逐例执行；每例通过生产 Worker 生成标准产物，再用独立 Origin 会话重开 OPJU，核对必需工作表、结果摘要和数值 oracle，并等待新 Origin 进程完全退出。目标测试机观察到 COM 退出可能接近 2 分钟，因此整套六例需预留较长命令时间；查询进程列表的瞬时失败会固定重试三次。最终以 `acceptance-report.json` 的 `passed=true`、各 case 的 `oracle_validation.passed=true` 和 `origin_processes_released=true` 共同作为通过条件。 + Blender adapter 可独立运行 `windows-node\package-blender-adapter.bat` 打成 `dist\blender.scene.author@v3-adapter.zip`，替换规则与 Origin 相同。升级时先从托盘退出 Node，删除程序目录中的 `adapters\blender.scene.author@v1\`、`adapters\blender.scene.author@v2\`，整体放入 v3 目录后重启；数据根目录中的旧 Workspace 不在清理范围。Blender 的 `requirements.txt` 当前不安装第三方包，`bpy` 不安装到轻量启动 runtime。 注册配置写入 `\node.json`；Token 使用 DPAPI `LocalMachine` 加密，ACL 仅允许注册账号和 `SYSTEM`。应始终用同一专用 Windows 账号配置、注册并运行 Node。当前 MVP 可由 UI 写入该账号的登录启动项，不安装 Windows Service。 diff --git a/software-contracts/origin.analysis.v1.json b/software-contracts/origin.analysis.v1.json new file mode 100644 index 0000000..4161a68 --- /dev/null +++ b/software-contracts/origin.analysis.v1.json @@ -0,0 +1,298 @@ +{ + "capability": "origin.analysis@v1", + "display_name": "Origin 科研分析", + "default_enrollment": false, + "output_namespace": "origin_analysis", + "input_policy": { + "suffixes": [".csv", ".xlsx", ".json"], + "max_count": 1, + "max_bytes": 104857600, + "max_total_bytes": 104857600 + }, + "outputs": { + "project": { + "filename": "analysis.opju", + "media_type": "application/x-origin-project", + "relative_path": "analysis.opju", + "publish": true, + "required": true + }, + "result_table": { + "filename": "result-table.csv", + "media_type": "text/csv", + "relative_path": "result-table.csv", + "publish": true, + "required": true + }, + "result_workbook": { + "filename": "result-table.xlsx", + "media_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + "relative_path": "result-table.xlsx", + "publish": true, + "required": false + }, + "diagnostics": { + "filename": "diagnostics.json", + "media_type": "application/json", + "relative_path": "diagnostics.json", + "publish": true, + "required": true + }, + "analysis_spec": { + "filename": "analysis-spec.json", + "media_type": "application/json", + "relative_path": ".meta/analysis-spec.json", + "publish": false, + "required": true + }, + "provenance": { + "filename": "provenance.json", + "media_type": "application/json", + "relative_path": ".meta/provenance.json", + "publish": false, + "required": true + } + }, + "feature_path": ["operation", "analysis", "type"], + "features": { + "data_check": "0.1.0", + "normalize": "0.1.0", + "smooth": "0.1.0", + "differentiate": "0.1.0", + "integrate": "0.1.0", + "linear_fit": "0.1.0" + }, + "summary": { + "title_path": ["operation", "analysis", "title"] + }, + "legacy_runtime": null, + "workspace": null, + "request_schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "x-maxBytes": 65536, + "required": ["schema_version", "inputs", "operation", "outputs"], + "additionalProperties": false, + "properties": { + "schema_version": {"const": 1}, + "inputs": { + "type": "array", + "minItems": 1, + "maxItems": 1, + "items": { + "type": "object", + "required": ["key", "artifact_id"], + "additionalProperties": false, + "properties": { + "key": {"type": "string", "pattern": "^[a-z][a-z0-9_]{0,31}$"}, + "artifact_id": {"type": "string", "format": "uuid"}, + "selector": { + "type": "object", + "required": ["sheet"], + "additionalProperties": false, + "properties": { + "sheet": {"type": "string", "minLength": 1, "maxLength": 128} + } + } + } + } + }, + "operation": { + "type": "object", + "required": ["analysis"], + "additionalProperties": false, + "properties": { + "analysis": { + "type": "object", + "required": ["type", "title", "data", "parameters"], + "additionalProperties": false, + "properties": { + "type": { + "enum": [ + "data_check", + "normalize", + "smooth", + "differentiate", + "integrate", + "linear_fit" + ] + }, + "title": {"type": "string", "minLength": 1, "maxLength": 200}, + "data": {"$ref": "#/$defs/data_binding"}, + "parameters": {"type": "object"} + }, + "allOf": [ + { + "if": {"properties": {"type": {"const": "data_check"}}}, + "then": { + "properties": { + "parameters": {"$ref": "#/$defs/data_check_parameters"} + } + } + }, + { + "if": {"properties": {"type": {"const": "normalize"}}}, + "then": { + "properties": { + "data": {"$ref": "#/$defs/xy_binding"}, + "parameters": {"$ref": "#/$defs/normalize_parameters"} + } + } + }, + { + "if": {"properties": {"type": {"const": "smooth"}}}, + "then": { + "properties": { + "data": {"$ref": "#/$defs/xy_binding"}, + "parameters": {"$ref": "#/$defs/smooth_parameters"} + } + } + }, + { + "if": {"properties": {"type": {"const": "differentiate"}}}, + "then": { + "properties": { + "data": {"$ref": "#/$defs/xy_binding"}, + "parameters": {"$ref": "#/$defs/differentiate_parameters"} + } + } + }, + { + "if": {"properties": {"type": {"const": "integrate"}}}, + "then": { + "properties": { + "data": {"$ref": "#/$defs/xy_binding"}, + "parameters": {"$ref": "#/$defs/integrate_parameters"} + } + } + }, + { + "if": {"properties": {"type": {"const": "linear_fit"}}}, + "then": { + "properties": { + "data": {"$ref": "#/$defs/xy_binding"}, + "parameters": {"$ref": "#/$defs/linear_fit_parameters"} + } + } + } + ] + } + } + }, + "outputs": { + "type": "array", + "maxItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "required": ["key", "type", "format"], + "additionalProperties": false, + "properties": { + "key": {"const": "result_workbook"}, + "type": {"const": "table"}, + "format": {"const": "xlsx"} + } + } + } + }, + "$defs": { + "data_binding": { + "type": "object", + "required": ["input"], + "additionalProperties": false, + "properties": { + "input": {"type": "string", "pattern": "^[a-z][a-z0-9_]{0,31}$"}, + "x": {"type": "string", "minLength": 1, "maxLength": 128}, + "y": {"type": "string", "minLength": 1, "maxLength": 128} + } + }, + "xy_binding": { + "type": "object", + "required": ["input", "x", "y"], + "additionalProperties": false, + "properties": { + "input": {"type": "string", "pattern": "^[a-z][a-z0-9_]{0,31}$"}, + "x": {"type": "string", "minLength": 1, "maxLength": 128}, + "y": {"type": "string", "minLength": 1, "maxLength": 128} + } + }, + "data_check_parameters": { + "type": "object", + "description": "数据体检始终返回诊断产物；命中 fail_on 时 diagnostics.passed=false，但 Job 本身成功完成，供调用方决定是否继续分析。", + "required": ["fail_on"], + "additionalProperties": false, + "properties": { + "fail_on": { + "type": "array", + "uniqueItems": true, + "items": { + "enum": [ + "missing", + "non_finite", + "duplicate_x", + "non_monotonic_x", + "uneven_spacing" + ] + } + } + } + }, + "normalize_parameters": { + "type": "object", + "description": "max_abs 使用最大绝对值；area 使用有符号梯形积分；reference 在数据范围内按 X 线性插值得到归一化除数。", + "required": ["method"], + "additionalProperties": false, + "properties": { + "method": {"enum": ["max_abs", "area", "reference"]}, + "reference_x": {"type": "number"} + }, + "allOf": [ + { + "if": {"properties": {"method": {"const": "reference"}}}, + "then": {"required": ["reference_x"]}, + "else": {"not": {"required": ["reference_x"]}} + } + ] + }, + "smooth_parameters": { + "type": "object", + "description": "固定局部多项式平滑；窗口必须为奇数，边界使用平移后的完整窗口，不补造数据点。", + "required": ["method", "window", "polynomial_order"], + "additionalProperties": false, + "properties": { + "method": {"const": "savitzky_golay"}, + "window": {"type": "integer", "minimum": 5, "maximum": 101}, + "polynomial_order": {"type": "integer", "minimum": 2, "maximum": 5} + } + }, + "differentiate_parameters": { + "type": "object", + "description": "使用支持非等间距 X 的二阶边界数值梯度，按 order 重复求导。", + "required": ["order"], + "additionalProperties": false, + "properties": { + "order": {"type": "integer", "enum": [1, 2]} + } + }, + "integrate_parameters": { + "type": "object", + "description": "使用梯形积分输出从首个 X 开始的累计积分；from/to 省略时报告完整范围积分。", + "additionalProperties": false, + "properties": { + "from": {"type": "number"}, + "to": {"type": "number"} + } + }, + "linear_fit_parameters": { + "type": "object", + "description": "固定最小二乘线性拟合并输出 95% t 置信区间、R²、RMSE 和残差；同时在 OPJU 中生成 Origin 原生线性拟合报告。", + "required": ["include_intercept", "confidence_level"], + "additionalProperties": false, + "properties": { + "include_intercept": {"type": "boolean"}, + "confidence_level": {"const": 0.95} + } + } + } + } +} diff --git a/tests/test_origin_analysis.py b/tests/test_origin_analysis.py new file mode 100644 index 0000000..8ee4e2e --- /dev/null +++ b/tests/test_origin_analysis.py @@ -0,0 +1,309 @@ +from __future__ import annotations + +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +import numpy as np + +from core.software_contracts import get_contract + +ROOT = Path(__file__).resolve().parents[1] +ADAPTER_ROOT = ROOT / "windows-node" / "adapters" / "origin.analysis@v1" +WORKER_PATH = ADAPTER_ROOT / "worker.py" +SPEC = importlib.util.spec_from_file_location("zcbot_origin_analysis_worker", WORKER_PATH) +assert SPEC and SPEC.loader +worker = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(worker) + +ACCEPTANCE_PATH = ADAPTER_ROOT / "acceptance.py" +ACCEPTANCE_SPEC = importlib.util.spec_from_file_location( + "zcbot_origin_analysis_acceptance", ACCEPTANCE_PATH +) +assert ACCEPTANCE_SPEC and ACCEPTANCE_SPEC.loader +acceptance = importlib.util.module_from_spec(ACCEPTANCE_SPEC) +ACCEPTANCE_SPEC.loader.exec_module(acceptance) + + +def _request(operation: str, parameters: dict, *, data: dict | None = None) -> dict: + return { + "schema_version": 1, + "inputs": [ + { + "key": "sample", + "artifact_id": "f4186347-65cc-4f07-9c26-bf11992beef8", + } + ], + "operation": { + "analysis": { + "type": operation, + "title": f"Test {operation}", + "data": data or {"input": "sample", "x": "x", "y": "y"}, + "parameters": parameters, + } + }, + "outputs": [], + } + + +class OriginAnalysisContractTests(unittest.TestCase): + def test_contract_exposes_six_guarded_operations_and_fixed_outputs(self) -> None: + contract = get_contract("origin.analysis@v1") + self.assertFalse(contract.default_enrollment) + self.assertEqual( + set(contract.features), + { + "data_check", + "normalize", + "smooth", + "differentiate", + "integrate", + "linear_fit", + }, + ) + request = _request("normalize", {"method": "max_abs"}) + normalized, digest = contract.normalize_request(request) + self.assertEqual(normalized, request) + self.assertEqual(len(digest), 64) + self.assertEqual( + set(contract.expected_outputs(request)), + {"project", "result_table", "diagnostics", "analysis_spec", "provenance"}, + ) + + workbook_request = json.loads(json.dumps(request)) + workbook_request["outputs"] = [ + {"key": "result_workbook", "type": "table", "format": "xlsx"} + ] + normalized, _ = contract.normalize_request(workbook_request) + self.assertEqual(normalized, workbook_request) + self.assertIn("result_workbook", contract.expected_outputs(workbook_request)) + + def test_contract_rejects_scripts_and_operation_specific_parameter_drift(self) -> None: + contract = get_contract("origin.analysis@v1") + request = _request("smooth", { + "method": "savitzky_golay", + "window": 7, + "polynomial_order": 3, + }) + contract.normalize_request(request) + + request["operation"]["analysis"]["parameters"]["script"] = "anything" + with self.assertRaises(ValueError): + contract.normalize_request(request) + + reference = _request("normalize", {"method": "reference"}) + with self.assertRaises(ValueError): + contract.normalize_request(reference) + + def test_worker_and_manifest_versions_match(self) -> None: + manifest = json.loads((ADAPTER_ROOT / "adapter.json").read_text(encoding="utf-8")) + self.assertEqual(worker.ADAPTER_VERSION, manifest["adapter_version"]) + + +class OriginAnalysisNumericalTests(unittest.TestCase): + def test_data_check_profiles_columns_and_reports_x_issues(self) -> None: + headers = ["x", "y", "label"] + rows = [[1, 2, "a"], [1, None, "b"], [3.5, float("inf"), "c"]] + spec = _request( + "data_check", + {"fail_on": ["missing", "non_finite", "duplicate_x", "uneven_spacing"]}, + )["operation"]["analysis"] + result_headers, result_rows, diagnostics = worker.analyze(headers, rows, spec) + self.assertEqual(result_headers[0], "column") + self.assertEqual(len(result_rows), 3) + self.assertFalse(diagnostics["passed"]) + self.assertEqual( + set(diagnostics["failed_checks"]), + {"missing", "non_finite", "duplicate_x"}, + ) + + def test_normalize_supports_max_area_and_interpolated_reference(self) -> None: + x = np.array([0.0, 1.0, 2.0]) + y = np.array([0.0, 2.0, 4.0]) + _, rows, diagnostics = worker._normalize(x, y, {"method": "max_abs"}) + self.assertEqual([item[2] for item in rows], [0.0, 0.5, 1.0]) + self.assertEqual(diagnostics["divisor"], 4.0) + + _, _, diagnostics = worker._normalize(x, y, {"method": "area"}) + self.assertEqual(diagnostics["divisor"], 4.0) + + _, rows, diagnostics = worker._normalize( + x, y, {"method": "reference", "reference_x": 1.5} + ) + self.assertEqual(diagnostics["divisor"], 3.0) + self.assertAlmostEqual(rows[-1][2], 4 / 3) + + def test_smooth_reproduces_quadratic_and_rejects_even_window(self) -> None: + x = np.arange(7, dtype=float) + y = x**2 + 2 * x + 3 + _, rows, diagnostics = worker._smooth( + x, + y, + {"method": "savitzky_golay", "window": 5, "polynomial_order": 2}, + ) + np.testing.assert_allclose([item[2] for item in rows], y, atol=1e-10) + self.assertAlmostEqual(diagnostics["rmse"], 0.0, places=10) + with self.assertRaisesRegex(ValueError, "SMOOTH_WINDOW_MUST_BE_ODD"): + worker._smooth( + x, + y, + {"method": "savitzky_golay", "window": 6, "polynomial_order": 2}, + ) + + def test_differentiate_and_integrate_match_polynomial_oracles(self) -> None: + x = np.arange(5, dtype=float) + y = x**2 + _, derivative_rows, _ = worker._differentiate(x, y, {"order": 1}) + np.testing.assert_allclose([item[2] for item in derivative_rows], 2 * x) + + _, integral_rows, diagnostics = worker._integrate(x, 2 * x, {}) + np.testing.assert_allclose([item[2] for item in integral_rows], x**2) + self.assertEqual(diagnostics["full_integral"], 16.0) + _, _, diagnostics = worker._integrate(x, 2 * x, {"from": 1.0, "to": 3.0}) + self.assertEqual(diagnostics["integral"], 8.0) + + def test_linear_fit_returns_parameters_intervals_and_residuals(self) -> None: + x = np.arange(1, 7, dtype=float) + y = 2.5 * x - 1.25 + headers, rows, diagnostics = worker._linear_fit( + x, + y, + {"include_intercept": True, "confidence_level": 0.95}, + ) + self.assertEqual(headers, ["x", "y", "fitted_y", "residual"]) + self.assertAlmostEqual(diagnostics["slope"], 2.5) + self.assertAlmostEqual(diagnostics["intercept"], -1.25) + self.assertAlmostEqual(diagnostics["r_squared"], 1.0) + np.testing.assert_allclose([item[3] for item in rows], 0, atol=1e-12) + self.assertLessEqual( + diagnostics["slope_confidence_interval"][0], diagnostics["slope"] + ) + self.assertGreaterEqual( + diagnostics["slope_confidence_interval"][1], diagnostics["slope"] + ) + + def test_xy_values_require_finite_strictly_increasing_x(self) -> None: + binding = {"input": "sample", "x": "x", "y": "y"} + with self.assertRaisesRegex(ValueError, "X_VALUES_DUPLICATED"): + worker._xy_values(["x", "y"], [[1, 2], [1, 3]], binding) + with self.assertRaisesRegex(ValueError, "Y_VALUE_NOT_FINITE"): + worker._xy_values(["x", "y"], [[1, 2], [2, float("nan")]], binding) + + def test_csv_json_and_xlsx_outputs_are_deterministically_readable(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + csv_path = root / "input.csv" + csv_path.write_text("x,y\n1,2\n3,4\n", encoding="utf-8") + self.assertEqual( + worker._read_rows(csv_path, None), + (["x", "y"], [["1", "2"], ["3", "4"]]), + ) + json_path = root / "input.json" + json_path.write_text( + json.dumps([{"x": 1, "y": 2}, {"x": 3, "y": 4}]), + encoding="utf-8", + ) + self.assertEqual( + worker._read_rows(json_path, None), (["x", "y"], [[1, 2], [3, 4]]) + ) + + result_path = root / "result-table.csv" + worker._atomic_csv(result_path, ["x", "y"], [[1, 2], [3, 4]]) + self.assertTrue(result_path.read_bytes().startswith(b"\xef\xbb\xbf")) + workbook_path = root / "result-table.xlsx" + worker._atomic_xlsx(workbook_path, ["x", "y"], [[1, 2], [3, 4]]) + self.assertTrue(workbook_path.is_file()) + + def test_run_builds_fixed_manifest_without_origin_side_effects_in_request(self) -> None: + with tempfile.TemporaryDirectory() as directory: + job_dir = Path(directory) + (job_dir / "request").mkdir() + (job_dir / "input" / "sample").mkdir(parents=True) + (job_dir / "input" / "sample" / "sample.csv").write_text( + "x,y\n0,0\n1,2\n2,4\n", encoding="utf-8" + ) + request = _request("normalize", {"method": "max_abs"}) + record = { + "job_id": "job", + "lease_id": "lease", + "request_digest": "a" * 64, + "request": request, + } + worker._atomic_json(job_dir / "request" / "request.json", record) + + def fake_project(path, *_args): + path.write_bytes(b"origin-project" * 16) + + with patch.object(worker, "_write_origin_project", side_effect=fake_project): + artifacts = worker.run(job_dir) + + self.assertEqual( + {item["artifact_id"] for item in artifacts}, + {"project", "result_table", "diagnostics", "analysis_spec", "provenance"}, + ) + diagnostics = json.loads( + (job_dir / "output" / "diagnostics.json").read_text(encoding="utf-8") + ) + self.assertEqual(diagnostics["divisor"], 4.0) + provenance = json.loads( + (job_dir / "output" / "provenance.json").read_text(encoding="utf-8") + ) + self.assertEqual(provenance["request_digest"], "a" * 64) + + +class OriginAnalysisAcceptanceTests(unittest.TestCase): + def test_fixed_acceptance_cases_are_contract_valid(self) -> None: + contract = get_contract("origin.analysis@v1") + cases = acceptance._cases() + self.assertEqual( + list(cases), + [ + "data_check", + "normalize", + "smooth", + "differentiate", + "integrate", + "linear_fit", + ], + ) + for name, case in cases.items(): + normalized, digest = contract.normalize_request(case["request"]) + self.assertEqual(normalized, case["request"], name) + self.assertEqual(len(digest), 64) + + def test_acceptance_oracles_are_independently_recomputed(self) -> None: + for name, case in acceptance._cases().items(): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "sample.csv" + path.write_text(case["csv"], encoding="utf-8", newline="") + headers, rows = worker._read_rows(path, None) + result_headers, result_rows, diagnostics = worker.analyze( + headers, rows, case["request"]["operation"]["analysis"] + ) + validation = acceptance._validate_oracle( + name, + diagnostics, + result_headers, + result_rows, + case["oracle"], + ) + self.assertTrue(validation["passed"], name) + + def test_stage_writes_keyed_input_and_stable_request_digest(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + case = acceptance._cases()["linear_fit"] + job_dir = acceptance._stage(root, "linear_fit", case) + record = json.loads( + (job_dir / "request" / "request.json").read_text(encoding="utf-8") + ) + _, digest = acceptance._canonical_request(case["request"]) + self.assertEqual(record["request_digest"], digest) + self.assertTrue((job_dir / "input" / "sample" / "sample.csv").is_file()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_software_nodes.py b/tests/test_software_nodes.py index 9839e14..3a94bbd 100644 --- a/tests/test_software_nodes.py +++ b/tests/test_software_nodes.py @@ -82,6 +82,7 @@ class SoftwareNodeSecurityTests(unittest.TestCase): self.assertEqual( set(enrollment.allowed_capabilities), { + "origin.analysis@v1", "origin.plot@v2", "ansys.geometry.inspect@v1", "ansys.mechanical.static_structural@v2", diff --git a/tests/test_windows_node_source.py b/tests/test_windows_node_source.py index c1d5c15..1b725e4 100644 --- a/tests/test_windows_node_source.py +++ b/tests/test_windows_node_source.py @@ -304,6 +304,8 @@ class WindowsNodeSourceTests(unittest.TestCase): self.assertIn("adapters\\%(RecursiveDir)%(Filename)%(Extension)", project) self.assertIn("..\\..\\software-contracts\\origin.plot.v2.json", project) self.assertIn("adapters\\origin.plot@v2\\origin.plot.v2.json", project) + self.assertIn("..\\..\\software-contracts\\origin.analysis.v1.json", project) + self.assertIn("adapters\\origin.analysis@v1\\origin.analysis.v1.json", project) self.assertIn( "..\\..\\software-contracts\\ansys.geometry.inspect.v1.json", project, @@ -541,6 +543,24 @@ class WindowsNodeSourceTests(unittest.TestCase): tray, ) + def test_origin_analysis_adapter_can_be_packaged_without_building_node(self) -> None: + script = (ROOT / "package-origin-analysis-adapter.bat").read_text(encoding="utf-8") + self.assertIn('"adapters\\origin.analysis@v1\\adapter.json"', script) + self.assertIn('"adapters\\origin.analysis@v1\\worker.py"', script) + self.assertIn('"adapters\\origin.analysis@v1\\acceptance.py"', script) + self.assertIn('"..\\software-contracts\\origin.analysis.v1.json"', script) + self.assertNotIn("dotnet", script.lower()) + + worker = ( + ROOT / "adapters" / "origin.analysis@v1" / "worker.py" + ).read_text(encoding="utf-8") + for forbidden in ("eval(", "exec(", "os.system"): + self.assertNotIn(forbidden, worker) + self.assertIn("op.LinearFit()", worker) + self.assertIn("op.save", worker) + self.assertIn('if op.oext:', worker) + self.assertIn("op.exit()", worker) + def test_blender_adapter_can_be_packaged_without_building_node(self) -> None: script = (ROOT / "package-blender-adapter.bat").read_text(encoding="utf-8") self.assertIn("adapters\\blender.scene.author@v3", script) diff --git a/windows-node/Zcbot.WindowsNode/Zcbot.WindowsNode.csproj b/windows-node/Zcbot.WindowsNode/Zcbot.WindowsNode.csproj index 58c1ab2..6f8d744 100644 --- a/windows-node/Zcbot.WindowsNode/Zcbot.WindowsNode.csproj +++ b/windows-node/Zcbot.WindowsNode/Zcbot.WindowsNode.csproj @@ -25,6 +25,11 @@ PreserveNewest PreserveNewest + + adapters\origin.analysis@v1\origin.analysis.v1.json + PreserveNewest + PreserveNewest + adapters\ansys.geometry.inspect@v1\ansys.geometry.inspect.v1.json PreserveNewest diff --git a/windows-node/adapters/origin.analysis@v1/acceptance.py b/windows-node/adapters/origin.analysis@v1/acceptance.py new file mode 100644 index 0000000..192a823 --- /dev/null +++ b/windows-node/adapters/origin.analysis@v1/acceptance.py @@ -0,0 +1,397 @@ +"""Run fixed origin.analysis@v1 acceptance cases on a dedicated Windows node.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import importlib.util +import json +import locale +import os +import platform +import re +import subprocess +import sys +import time +from datetime import datetime, timezone +from importlib.metadata import PackageNotFoundError, version +from pathlib import Path +from typing import Any +from uuid import NAMESPACE_URL, uuid5 + +ADAPTER_DIR = Path(__file__).resolve().parent +WORKER_PATH = ADAPTER_DIR / "worker.py" +_WORKER_SPEC = importlib.util.spec_from_file_location( + "zcbot_origin_analysis_acceptance_worker", WORKER_PATH +) +if _WORKER_SPEC is None or _WORKER_SPEC.loader is None: + raise RuntimeError("ORIGIN_ANALYSIS_WORKER_IMPORT_FAILED") +worker = importlib.util.module_from_spec(_WORKER_SPEC) +_WORKER_SPEC.loader.exec_module(worker) + +REPORT_SCHEMA_VERSION = 1 +ORIGIN_PROCESS_PATTERN = r"^origin(?:\d+)?(?:_?\d+)?(?:64)?\.exe$" + + +def _artifact_id(case_name: str) -> str: + return str(uuid5(NAMESPACE_URL, f"zcbot-origin-analysis-acceptance:{case_name}")) + + +def _request(case_name: str, operation: str, parameters: dict[str, Any]) -> dict[str, Any]: + return { + "schema_version": 1, + "inputs": [{"key": "sample", "artifact_id": _artifact_id(case_name)}], + "operation": { + "analysis": { + "type": operation, + "title": f"Origin analysis acceptance: {operation}", + "data": {"input": "sample", "x": "x", "y": "y"}, + "parameters": parameters, + } + }, + "outputs": ( + [{"key": "result_workbook", "type": "table", "format": "xlsx"}] + if operation == "linear_fit" + else [] + ), + } + + +def _cases() -> dict[str, dict[str, Any]]: + data_check = _request( + "data_check", + "data_check", + {"fail_on": ["missing", "non_finite", "duplicate_x", "uneven_spacing"]}, + ) + return { + "data_check": { + "csv": "x,y\n0,1\n1,\n1,inf\n3,4\n", + "request": data_check, + "oracle": { + "passed": False, + "failed_checks": ["duplicate_x", "missing", "non_finite", "uneven_spacing"], + }, + }, + "normalize": { + "csv": "x,y\n0,0\n1,2\n2,4\n", + "request": _request( + "normalize", "normalize", {"method": "reference", "reference_x": 1.5} + ), + "oracle": {"divisor": 3.0, "last_value": 4 / 3}, + }, + "smooth": { + "csv": "x,y\n" + "\n".join(f"{x},{x*x + 2*x + 3}" for x in range(7)) + "\n", + "request": _request( + "smooth", + "smooth", + {"method": "savitzky_golay", "window": 5, "polynomial_order": 2}, + ), + "oracle": {"rmse": 0.0}, + }, + "differentiate": { + "csv": "x,y\n" + "\n".join(f"{x},{x*x}" for x in range(5)) + "\n", + "request": _request("differentiate", "differentiate", {"order": 1}), + "oracle": {"derived": [0.0, 2.0, 4.0, 6.0, 8.0]}, + }, + "integrate": { + "csv": "x,y\n" + "\n".join(f"{x},{2*x}" for x in range(5)) + "\n", + "request": _request("integrate", "integrate", {"from": 1.0, "to": 3.0}), + "oracle": {"integral": 8.0, "full_integral": 16.0}, + }, + "linear_fit": { + "csv": "x,y\n" + "\n".join(f"{x},{2.5*x - 1.25}" for x in range(1, 7)) + "\n", + "request": _request( + "linear_fit", + "linear_fit", + {"include_intercept": True, "confidence_level": 0.95}, + ), + "oracle": {"slope": 2.5, "intercept": -1.25, "r_squared": 1.0}, + }, + } + + +def _canonical_request(request: dict[str, Any]) -> tuple[str, str]: + encoded = json.dumps(request, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return encoded, hashlib.sha256(encoded.encode("utf-8")).hexdigest() + + +def _stage(root: Path, case_name: str, case: dict[str, Any]) -> Path: + job_dir = root / case_name + request_dir = job_dir / "request" + input_dir = job_dir / "input" / "sample" + request_dir.mkdir(parents=True) + input_dir.mkdir(parents=True) + (input_dir / "sample.csv").write_text(case["csv"], encoding="utf-8", newline="") + _, digest = _canonical_request(case["request"]) + record = { + "job_id": str(uuid5(NAMESPACE_URL, f"zcbot-origin-analysis-job:{case_name}")), + "lease_id": str(uuid5(NAMESPACE_URL, f"zcbot-origin-analysis-lease:{case_name}")), + "request_digest": digest, + "request": case["request"], + } + worker._atomic_json(request_dir / "request.json", record) + return job_dir + + +def _read_result(path: Path) -> tuple[list[str], list[list[float | str | None]]]: + with path.open("r", encoding="utf-8-sig", newline="") as handle: + rows = list(csv.reader(handle)) + converted: list[list[float | str | None]] = [] + for row in rows[1:]: + values: list[float | str | None] = [] + for value in row: + if value == "": + values.append(None) + continue + try: + values.append(float(value)) + except ValueError: + values.append(value) + converted.append(values) + return rows[0], converted + + +def _assert_close(actual: float, expected: float, tolerance: float = 1e-9) -> None: + if abs(float(actual) - float(expected)) > tolerance: + raise RuntimeError(f"NUMERICAL_ORACLE_MISMATCH:{actual}:{expected}") + + +def _validate_oracle( + case_name: str, + diagnostics: dict[str, Any], + result_headers: list[str], + result_rows: list[list[Any]], + oracle: dict[str, Any], +) -> dict[str, Any]: + if case_name == "data_check": + if diagnostics["passed"] is not oracle["passed"]: + raise RuntimeError("DATA_CHECK_PASS_STATE_MISMATCH") + if diagnostics["failed_checks"] != oracle["failed_checks"]: + raise RuntimeError("DATA_CHECK_CODES_MISMATCH") + elif case_name == "normalize": + _assert_close(diagnostics["divisor"], oracle["divisor"]) + _assert_close(result_rows[-1][result_headers.index("normalized_y")], oracle["last_value"]) + elif case_name == "smooth": + _assert_close(diagnostics["rmse"], oracle["rmse"], 1e-10) + elif case_name == "differentiate": + values = [row[result_headers.index("derivative_order_1")] for row in result_rows] + for actual, expected in zip(values, oracle["derived"], strict=True): + _assert_close(actual, expected) + elif case_name == "integrate": + _assert_close(diagnostics["integral"], oracle["integral"]) + _assert_close(diagnostics["full_integral"], oracle["full_integral"]) + elif case_name == "linear_fit": + for name in ("slope", "intercept", "r_squared"): + _assert_close(diagnostics[name], oracle[name]) + return {"passed": True, "oracle": oracle} + + +def _validate_manifest(job_dir: Path, case: dict[str, Any]) -> dict[str, Any]: + terminal = json.loads((job_dir / "terminal.json").read_text(encoding="utf-8")) + if terminal.get("status") != "succeeded": + raise RuntimeError(f"WORKER_FAILED:{terminal.get('error')}") + expected = {"project", "result_table", "diagnostics", "analysis_spec", "provenance"} + if case["request"]["outputs"]: + expected.add("result_workbook") + manifest = terminal.get("artifact_manifest") or [] + by_id = {item["artifact_id"]: item for item in manifest} + if set(by_id) != expected: + raise RuntimeError(f"ARTIFACT_MANIFEST_MISMATCH:{sorted(by_id)}") + output = job_dir / "output" + paths = {artifact_id: output / item["filename"] for artifact_id, item in by_id.items()} + for artifact_id, path in paths.items(): + if not path.is_file() or path.stat().st_size != by_id[artifact_id]["size_bytes"]: + raise RuntimeError(f"ARTIFACT_INVALID:{artifact_id}") + if worker._file_sha256(path) != by_id[artifact_id]["sha256"]: + raise RuntimeError(f"ARTIFACT_DIGEST_MISMATCH:{artifact_id}") + diagnostics = json.loads(paths["diagnostics"].read_text(encoding="utf-8")) + headers, rows = _read_result(paths["result_table"]) + oracle = _validate_oracle(case["request"]["operation"]["analysis"]["type"], diagnostics, headers, rows, case["oracle"]) + return { + "artifacts": { + key: {"size_bytes": path.stat().st_size, "sha256": worker._file_sha256(path)} + for key, path in sorted(paths.items()) + }, + "oracle_validation": oracle, + } + + +def _reopen_project(job_dir: Path) -> dict[str, Any]: + import originpro as op + + project = job_dir / "output" / "analysis.opju" + op.set_show(False) + try: + if not op.open(str(project), readonly=True): + raise RuntimeError("OPJU_REOPEN_FAILED") + books = list(op.pages("w")) + sheets = [sheet for book in books for sheet in book] + names = [ + str(sheet.lname or getattr(book, "lname", "") or getattr(book, "name", "") or "") + for book in books + for sheet in book + ] + for required in ("RawData", "Result", "Diagnostics"): + if required not in names: + raise RuntimeError(f"OPJU_SHEET_MISSING:{required}") + return {"worksheet_count": len(sheets), "worksheet_names": names} + finally: + if op.oext: + op.exit() + + +def _environment_fingerprint() -> dict[str, Any]: + try: + originpro_version = version("originpro") + except PackageNotFoundError: + originpro_version = "unknown" + return { + "adapter_version": worker.ADAPTER_VERSION, + "algorithm_version": worker.ALGORITHM_VERSION, + "originpro_version": originpro_version, + "numpy_version": worker.np.__version__, + "python_version": platform.python_version(), + "platform": platform.platform(), + "locale": locale.getlocale(), + "execution_mode": "hidden", + "configured_origin_executable": bool(os.environ.get("ZCBOT_ORIGIN_EXE")), + } + + +def _origin_processes() -> dict[int, str]: + completed = None + for attempt in range(3): + completed = subprocess.run( + ["tasklist.exe", "/fo", "csv", "/nh"], + check=False, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + if completed.returncode == 0: + break + if attempt < 2: + time.sleep(1) + if completed is None or completed.returncode != 0: + detail = completed.stderr[-500:] if completed is not None else "not started" + raise RuntimeError(f"TASKLIST_FAILED:{detail}") + result: dict[int, str] = {} + for row in csv.reader(completed.stdout.splitlines()): + name = row[0].casefold() if row else "" + if len(row) >= 2 and re.fullmatch(ORIGIN_PROCESS_PATTERN, name): + result[int(row[1])] = row[0] + return result + + +def _wait_for_origin_release(baseline: dict[int, str], timeout_seconds: int) -> None: + deadline = time.monotonic() + timeout_seconds + while True: + remaining = { + pid: name for pid, name in _origin_processes().items() if pid not in baseline + } + if not remaining: + return + if time.monotonic() >= deadline: + raise RuntimeError(f"ORIGIN_PROCESS_REMAINS:{remaining}") + time.sleep(2) + + +def _run_case( + root: Path, + case_name: str, + case: dict[str, Any], + baseline: dict[int, str], + release_wait: int, +) -> dict[str, Any]: + job_dir = _stage(root, case_name, case) + started = time.monotonic() + completed = subprocess.run( + [sys.executable, str(WORKER_PATH), str(job_dir)], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + check=False, + ) + if completed.returncode != 0: + raise RuntimeError(f"WORKER_PROCESS_FAILED:{completed.stderr[-500:]}") + validation = _validate_manifest(job_dir, case) + reopened = _reopen_project(job_dir) + _wait_for_origin_release(baseline, release_wait) + _, digest = _canonical_request(case["request"]) + return { + "case": case_name, + "elapsed_seconds": round(time.monotonic() - started, 3), + "request_digest": digest, + "validation": validation, + "reopen": reopened, + "origin_processes_released": True, + } + + +def run_suite( + root: Path, + selected: list[str] | None = None, + *, + release_wait: int = 60, +) -> dict[str, Any]: + cases = _cases() + names = selected or list(cases) + unknown = sorted(set(names) - cases.keys()) + if unknown: + raise ValueError(f"UNKNOWN_ACCEPTANCE_CASES:{','.join(unknown)}") + root.mkdir(parents=True, exist_ok=False) + baseline = _origin_processes() + report = { + "schema_version": REPORT_SCHEMA_VERSION, + "started_at": datetime.now(timezone.utc).isoformat(), + "environment": _environment_fingerprint(), + "selected_cases": names, + "cases": [], + "passed": False, + } + report_path = root / "acceptance-report.json" + try: + for index, name in enumerate(names, start=1): + report["cases"].append( + _run_case(root, name, cases[name], baseline, release_wait) + ) + worker._atomic_json(report_path, report) + print(f"[OK] Origin analysis acceptance {index}/{len(names)}: {name}") + report["passed"] = True + return report + except Exception as exc: + report["failure"] = { + "type": type(exc).__name__, + "detail": str(exc)[:1000], + "completed_cases": len(report["cases"]), + } + raise + finally: + report["completed_at"] = datetime.now(timezone.utc).isoformat() + worker._atomic_json(report_path, report) + + +def main() -> int: + parser = argparse.ArgumentParser(description="Run fixed Origin analysis acceptance cases.") + parser.add_argument("--work-root", type=Path, required=True) + parser.add_argument("--release-wait", type=int, default=60) + parser.add_argument( + "--case", action="append", choices=tuple(_cases()), dest="cases" + ) + args = parser.parse_args() + if sys.platform != "win32": + raise RuntimeError("Origin analysis acceptance requires Windows") + if args.release_wait < 1: + raise ValueError("release-wait must be positive") + root = args.work_root.resolve() + report = run_suite(root, args.cases, release_wait=args.release_wait) + print(f"[OK] Origin analysis acceptance passed. Report: {root / 'acceptance-report.json'}") + print(f"[INFO] Cases: {len(report['cases'])}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/windows-node/adapters/origin.analysis@v1/adapter.json b/windows-node/adapters/origin.analysis@v1/adapter.json new file mode 100644 index 0000000..8a20a1a --- /dev/null +++ b/windows-node/adapters/origin.analysis@v1/adapter.json @@ -0,0 +1,10 @@ +{ + "capability": "origin.analysis@v1", + "adapter_version": "0.1.0", + "runtime": "python", + "runtime_id": "origin", + "entrypoint": "worker.py", + "contract": "origin.analysis.v1.json", + "worker_timeout_minutes": 30, + "running_detail": "Origin 正在执行科研分析" +} diff --git a/windows-node/adapters/origin.analysis@v1/requirements.txt b/windows-node/adapters/origin.analysis@v1/requirements.txt new file mode 100644 index 0000000..9002075 --- /dev/null +++ b/windows-node/adapters/origin.analysis@v1/requirements.txt @@ -0,0 +1,3 @@ +originpro==1.1.15 +openpyxl==3.1.5 +numpy==2.2.6 diff --git a/windows-node/adapters/origin.analysis@v1/worker.py b/windows-node/adapters/origin.analysis@v1/worker.py new file mode 100644 index 0000000..53ed742 --- /dev/null +++ b/windows-node/adapters/origin.analysis@v1/worker.py @@ -0,0 +1,792 @@ +"""Fixed Origin adapter for origin.analysis@v1. + +The worker accepts one Node-created job directory. It performs only versioned, +declarative analysis operations, writes the raw data and results into an Origin +project, and never evaluates user code or accepts user-controlled paths. +""" + +from __future__ import annotations + +import csv +import hashlib +import json +import math +import os +import sys +from datetime import datetime, timezone +from importlib.metadata import PackageNotFoundError, version +from itertools import pairwise +from pathlib import Path +from typing import Any + +import numpy as np + +ADAPTER_VERSION = "0.1.0" +ALGORITHM_VERSION = "origin-analysis-mvp-1" +OPERATIONS = { + "data_check", + "normalize", + "smooth", + "differentiate", + "integrate", + "linear_fit", +} +OUTPUT_MEDIA = { + "analysis.opju": ("project", "application/x-origin-project"), + "result-table.csv": ("result_table", "text/csv"), + "result-table.xlsx": ( + "result_workbook", + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + ), + "diagnostics.json": ("diagnostics", "application/json"), + "analysis-spec.json": ("analysis_spec", "application/json"), + "provenance.json": ("provenance", "application/json"), +} + + +def _probe() -> int: + health = "ready" + detail = "Origin COM 与科研分析运行时可用" + software_version = None + try: + if sys.platform != "win32": + raise RuntimeError("Origin analysis adapter requires Windows") + import winreg + + import originpro + + with winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, r"Origin.ApplicationSI\CLSID"): + pass + originpro_version = version("originpro") + numpy_version = version("numpy") + detail = ( + "Origin COM 与科研分析运行时可用" + f"（originpro {originpro_version}，numpy {numpy_version}）" + ) + del originpro + except ( + FileNotFoundError, + ImportError, + OSError, + PackageNotFoundError, + RuntimeError, + ) as exc: + health = "unavailable" + detail = str(exc) + print( + json.dumps( + { + "adapter_version": ADAPTER_VERSION, + "software": "OriginPro", + "software_version": software_version, + "health": health, + "detail": detail, + }, + ensure_ascii=False, + ) + ) + return 0 + + +def _atomic_json(path: Path, value: Any) -> None: + temporary = path.with_name(path.name + ".tmp-" + os.urandom(8).hex()) + try: + with temporary.open("w", encoding="utf-8", newline="\n") as handle: + json.dump(value, handle, ensure_ascii=False, indent=2, allow_nan=False) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def _atomic_csv(path: Path, headers: list[str], rows: list[list[Any]]) -> None: + temporary = path.with_name(path.name + ".tmp-" + os.urandom(8).hex()) + try: + with temporary.open("w", encoding="utf-8-sig", newline="") as handle: + writer = csv.writer(handle, lineterminator="\n") + writer.writerow(headers) + writer.writerows(rows) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def _atomic_xlsx(path: Path, headers: list[str], rows: list[list[Any]]) -> None: + from openpyxl import Workbook + + temporary = path.with_name(path.name + ".tmp-" + os.urandom(8).hex() + ".xlsx") + try: + workbook = Workbook(write_only=True) + worksheet = workbook.create_sheet("Result") + worksheet.append(headers) + for row in rows: + worksheet.append(row) + workbook.save(temporary) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def _read_rows(path: Path, sheet: str | None) -> tuple[list[str], list[list[Any]]]: + suffix = path.suffix.lower() + if suffix == ".csv": + with path.open("r", encoding="utf-8-sig", newline="") as handle: + rows = list(csv.reader(handle)) + if len(rows) < 2: + raise ValueError("CSV_INPUT_EMPTY") + return [str(item) for item in rows[0]], rows[1:] + if suffix == ".json": + value = json.loads(path.read_text(encoding="utf-8")) + if isinstance(value, list) and value and all(isinstance(item, dict) for item in value): + headers = list(value[0]) + if any(set(item) != set(headers) for item in value): + raise ValueError("JSON_OBJECT_COLUMNS_MISMATCH") + return headers, [[item.get(name) for name in headers] for item in value] + if ( + isinstance(value, dict) + and value + and all(isinstance(item, list) for item in value.values()) + ): + headers = list(value) + length = max(len(value[name]) for name in headers) + return headers, [ + [value[name][index] if index < len(value[name]) else None for name in headers] + for index in range(length) + ] + raise ValueError("JSON_INPUT_SHAPE_UNSUPPORTED") + if suffix == ".xlsx": + from openpyxl import load_workbook + + workbook = load_workbook(path, read_only=True, data_only=True) + try: + worksheet = workbook[sheet] if sheet else workbook.active + rows = list(worksheet.iter_rows(values_only=True)) + finally: + workbook.close() + if len(rows) < 2: + raise ValueError("XLSX_INPUT_EMPTY") + return [str(item or "") for item in rows[0]], [list(row) for row in rows[1:]] + raise ValueError("INPUT_TYPE_UNSUPPORTED") + + +def _input_file(job_dir: Path, key: str) -> Path: + directory = job_dir / "input" / key + files = [ + path for path in directory.iterdir() if path.is_file() and not path.name.startswith(".") + ] + if len(files) != 1: + raise ValueError(f"INPUT_FILE_COUNT_INVALID:{key}") + return files[0] + + +def _file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _manifest(path: Path) -> dict[str, Any]: + artifact_id, media_type = OUTPUT_MEDIA[path.name] + return { + "artifact_id": artifact_id, + "filename": path.name, + "media_type": media_type, + "size_bytes": path.stat().st_size, + "sha256": _file_sha256(path), + } + + +def _missing(value: Any) -> bool: + return value is None or (isinstance(value, str) and not value.strip()) + + +def _finite_number(value: Any, role: str, row_number: int) -> float: + if isinstance(value, bool) or _missing(value): + raise ValueError(f"{role.upper()}_VALUE_NOT_NUMERIC:row={row_number}") + try: + result = float(value) + except (TypeError, ValueError) as exc: + raise ValueError(f"{role.upper()}_VALUE_NOT_NUMERIC:row={row_number}") from exc + if not math.isfinite(result): + raise ValueError(f"{role.upper()}_VALUE_NOT_FINITE:row={row_number}") + return result + + +def _column_index(headers: list[str], name: Any, role: str) -> int: + if not isinstance(name, str) or name not in headers: + raise ValueError(f"{role.upper()}_COLUMN_NOT_FOUND") + return headers.index(name) + + +def _xy_values( + headers: list[str], rows: list[list[Any]], binding: dict[str, Any] +) -> tuple[np.ndarray, np.ndarray]: + x_index = _column_index(headers, binding.get("x"), "x") + y_index = _column_index(headers, binding.get("y"), "y") + x = np.array( + [ + _finite_number(row[x_index] if x_index < len(row) else None, "x", index) + for index, row in enumerate(rows, start=2) + ], + dtype=float, + ) + y = np.array( + [ + _finite_number(row[y_index] if y_index < len(row) else None, "y", index) + for index, row in enumerate(rows, start=2) + ], + dtype=float, + ) + if len(x) < 2: + raise ValueError("ANALYSIS_REQUIRES_AT_LEAST_TWO_ROWS") + differences = np.diff(x) + if np.any(differences == 0): + raise ValueError("X_VALUES_DUPLICATED") + if np.any(differences < 0): + raise ValueError("X_VALUES_NOT_STRICTLY_INCREASING") + return x, y + + +def _json_number(value: float) -> float | None: + return float(value) if math.isfinite(float(value)) else None + + +def _column_profile(name: str, values: list[Any]) -> dict[str, Any]: + missing = sum(_missing(item) for item in values) + numeric: list[float] = [] + non_numeric = 0 + non_finite = 0 + for item in values: + if _missing(item): + continue + if isinstance(item, bool): + non_numeric += 1 + continue + try: + number = float(item) + except (TypeError, ValueError): + non_numeric += 1 + continue + if not math.isfinite(number): + non_finite += 1 + continue + numeric.append(number) + return { + "column": name, + "row_count": len(values), + "non_missing": len(values) - missing, + "missing": missing, + "numeric": len(numeric), + "non_numeric": non_numeric, + "non_finite": non_finite, + "minimum": min(numeric) if numeric else None, + "maximum": max(numeric) if numeric else None, + "mean": float(np.mean(numeric)) if numeric else None, + } + + +def _data_check( + headers: list[str], rows: list[list[Any]], spec: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + profiles = [ + _column_profile( + header, + [row[index] if index < len(row) else None for row in rows], + ) + for index, header in enumerate(headers) + ] + issues: list[dict[str, Any]] = [] + if any(item["missing"] for item in profiles): + issues.append({"code": "missing", "detail": "输入包含缺失值。"}) + if any(item["non_finite"] for item in profiles): + issues.append({"code": "non_finite", "detail": "输入包含非有限数值。"}) + binding = spec["data"] + if "x" in binding: + x_index = _column_index(headers, binding["x"], "x") + x_values: list[float] = [] + x_complete = True + for row_number, row in enumerate(rows, start=2): + try: + x_values.append( + _finite_number( + row[x_index] if x_index < len(row) else None, + "x", + row_number, + ) + ) + except ValueError: + x_complete = False + break + if x_complete and len(x_values) >= 2: + differences = np.diff(np.array(x_values, dtype=float)) + if np.any(differences == 0): + issues.append({"code": "duplicate_x", "detail": "X 列包含重复值。"}) + if np.any(differences < 0): + issues.append( + {"code": "non_monotonic_x", "detail": "X 列不是严格递增。"} + ) + positive = differences[differences > 0] + if len(positive) >= 2: + relative_span = float(np.ptp(positive) / np.mean(positive)) + if relative_span > 1e-6: + issues.append( + { + "code": "uneven_spacing", + "detail": "X 列采样间隔不均匀。", + "relative_spacing_span": relative_span, + } + ) + fail_on = set(spec["parameters"]["fail_on"]) + failed_codes = sorted({item["code"] for item in issues} & fail_on) + result_headers = [ + "column", + "row_count", + "non_missing", + "missing", + "numeric", + "non_numeric", + "non_finite", + "minimum", + "maximum", + "mean", + ] + result_rows = [[item.get(header) for header in result_headers] for item in profiles] + diagnostics = { + "operation": "data_check", + "passed": not failed_codes, + "failed_checks": failed_codes, + "issues": issues, + "row_count": len(rows), + "column_count": len(headers), + } + return result_headers, result_rows, diagnostics + + +def _normalize( + x: np.ndarray, y: np.ndarray, parameters: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + method = parameters["method"] + selected_x = None + if method == "max_abs": + divisor = float(np.max(np.abs(y))) + elif method == "area": + divisor = float(np.trapezoid(y, x)) + elif method == "reference": + reference_x = float(parameters["reference_x"]) + if reference_x < x[0] or reference_x > x[-1]: + raise ValueError("REFERENCE_X_OUTSIDE_DATA_RANGE") + divisor = float(np.interp(reference_x, x, y)) + selected_x = reference_x + else: + raise ValueError("NORMALIZE_METHOD_UNSUPPORTED") + scale = max(1.0, float(np.max(np.abs(y)))) + if abs(divisor) <= np.finfo(float).eps * scale: + raise ValueError("NORMALIZE_DIVISOR_IS_ZERO") + normalized = y / divisor + rows = [[float(a), float(b), float(c)] for a, b, c in zip(x, y, normalized, strict=True)] + return ["x", "y", "normalized_y"], rows, { + "operation": "normalize", + "method": method, + "divisor": divisor, + "reference_x": selected_x, + "row_count": len(rows), + } + + +def _smooth( + x: np.ndarray, y: np.ndarray, parameters: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + window = int(parameters["window"]) + polynomial_order = int(parameters["polynomial_order"]) + if window % 2 == 0: + raise ValueError("SMOOTH_WINDOW_MUST_BE_ODD") + if polynomial_order >= window: + raise ValueError("SMOOTH_POLYNOMIAL_ORDER_INVALID") + if window > len(x): + raise ValueError("SMOOTH_WINDOW_EXCEEDS_ROW_COUNT") + half = window // 2 + smoothed = np.empty_like(y) + for index in range(len(x)): + start = min(max(index - half, 0), len(x) - window) + stop = start + window + local_x = x[start:stop] - x[index] + coefficients = np.polynomial.polynomial.polyfit( + local_x, y[start:stop], polynomial_order + ) + smoothed[index] = coefficients[0] + residual = y - smoothed + rows = [ + [float(a), float(b), float(c), float(d)] + for a, b, c, d in zip(x, y, smoothed, residual, strict=True) + ] + return ["x", "y", "smoothed_y", "residual"], rows, { + "operation": "smooth", + "method": "savitzky_golay", + "window": window, + "polynomial_order": polynomial_order, + "rmse": float(np.sqrt(np.mean(residual**2))), + "boundary_handling": "shifted_full_window_local_polynomial", + "row_count": len(rows), + } + + +def _differentiate( + x: np.ndarray, y: np.ndarray, parameters: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + if len(x) < 3: + raise ValueError("DIFFERENTIATE_REQUIRES_AT_LEAST_THREE_ROWS") + order = int(parameters["order"]) + derivative = y.copy() + for _ in range(order): + derivative = np.gradient(derivative, x, edge_order=2) + rows = [ + [float(a), float(b), float(c)] + for a, b, c in zip(x, y, derivative, strict=True) + ] + return ["x", "y", f"derivative_order_{order}"], rows, { + "operation": "differentiate", + "order": order, + "method": "numpy_gradient_nonuniform", + "edge_order": 2, + "row_count": len(rows), + } + + +def _cumulative_trapezoid(x: np.ndarray, y: np.ndarray) -> np.ndarray: + cumulative = np.zeros_like(y) + cumulative[1:] = np.cumsum(np.diff(x) * (y[:-1] + y[1:]) / 2) + return cumulative + + +def _integrate( + x: np.ndarray, y: np.ndarray, parameters: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + cumulative = _cumulative_trapezoid(x, y) + lower = float(parameters.get("from", x[0])) + upper = float(parameters.get("to", x[-1])) + if lower >= upper: + raise ValueError("INTEGRATE_RANGE_INVALID") + if lower < x[0] or upper > x[-1]: + raise ValueError("INTEGRATE_RANGE_OUTSIDE_DATA") + integral = float(np.interp(upper, x, cumulative) - np.interp(lower, x, cumulative)) + rows = [ + [float(a), float(b), float(c)] + for a, b, c in zip(x, y, cumulative, strict=True) + ] + return ["x", "y", "cumulative_integral"], rows, { + "operation": "integrate", + "method": "trapezoid", + "from": lower, + "to": upper, + "integral": integral, + "full_integral": float(cumulative[-1]), + "row_count": len(rows), + } + + +_T_CRITICAL_95 = ( + 12.706, + 4.303, + 3.182, + 2.776, + 2.571, + 2.447, + 2.365, + 2.306, + 2.262, + 2.228, + 2.201, + 2.179, + 2.160, + 2.145, + 2.131, + 2.120, + 2.110, + 2.101, + 2.093, + 2.086, + 2.080, + 2.074, + 2.069, + 2.064, + 2.060, + 2.056, + 2.052, + 2.048, + 2.045, + 2.042, +) + + +def _t_critical_95(degrees_of_freedom: int) -> float: + if degrees_of_freedom <= 0: + raise ValueError("LINEAR_FIT_DEGREES_OF_FREEDOM_INVALID") + if degrees_of_freedom <= len(_T_CRITICAL_95): + return _T_CRITICAL_95[degrees_of_freedom - 1] + anchors = ((30, 2.042), (40, 2.021), (60, 2.000), (120, 1.980), (10**9, 1.960)) + for (left_df, left_value), (right_df, right_value) in pairwise(anchors): + if degrees_of_freedom <= right_df: + ratio = (degrees_of_freedom - left_df) / (right_df - left_df) + return left_value + ratio * (right_value - left_value) + return 1.960 + + +def _linear_fit( + x: np.ndarray, y: np.ndarray, parameters: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + include_intercept = bool(parameters["include_intercept"]) + design = np.column_stack((x, np.ones_like(x))) if include_intercept else x[:, None] + coefficients, _, rank, _ = np.linalg.lstsq(design, y, rcond=None) + if rank != design.shape[1]: + raise ValueError("LINEAR_FIT_DESIGN_RANK_DEFICIENT") + fitted = design @ coefficients + residual = y - fitted + parameter_count = design.shape[1] + degrees_of_freedom = len(x) - parameter_count + if degrees_of_freedom <= 0: + raise ValueError("LINEAR_FIT_INSUFFICIENT_ROWS") + sum_squared_error = float(residual @ residual) + mean_squared_error = sum_squared_error / degrees_of_freedom + covariance = mean_squared_error * np.linalg.inv(design.T @ design) + standard_errors = np.sqrt(np.diag(covariance)) + slope = float(coefficients[0]) + intercept = float(coefficients[1]) if include_intercept else 0.0 + slope_se = float(standard_errors[0]) + intercept_se = float(standard_errors[1]) if include_intercept else 0.0 + t_critical = _t_critical_95(degrees_of_freedom) + slope_ci = [slope - t_critical * slope_se, slope + t_critical * slope_se] + intercept_ci = ( + [intercept - t_critical * intercept_se, intercept + t_critical * intercept_se] + if include_intercept + else [0.0, 0.0] + ) + total_sum_squares = ( + float(np.sum((y - np.mean(y)) ** 2)) + if include_intercept + else float(y @ y) + ) + r_squared = 1 - sum_squared_error / total_sum_squares if total_sum_squares else 1.0 + rows = [ + [float(a), float(b), float(c), float(d)] + for a, b, c, d in zip(x, y, fitted, residual, strict=True) + ] + return ["x", "y", "fitted_y", "residual"], rows, { + "operation": "linear_fit", + "include_intercept": include_intercept, + "confidence_level": 0.95, + "slope": slope, + "intercept": intercept, + "slope_standard_error": slope_se, + "intercept_standard_error": intercept_se, + "slope_confidence_interval": slope_ci, + "intercept_confidence_interval": intercept_ci, + "t_critical": t_critical, + "degrees_of_freedom": degrees_of_freedom, + "r_squared": r_squared, + "rmse": math.sqrt(mean_squared_error), + "sum_squared_error": sum_squared_error, + "row_count": len(rows), + } + + +def analyze( + headers: list[str], rows: list[list[Any]], analysis: dict[str, Any] +) -> tuple[list[str], list[list[Any]], dict[str, Any]]: + operation = analysis["type"] + if operation not in OPERATIONS: + raise ValueError("ANALYSIS_TYPE_NOT_IMPLEMENTED") + if operation == "data_check": + return _data_check(headers, rows, analysis) + x, y = _xy_values(headers, rows, analysis["data"]) + parameters = analysis["parameters"] + functions = { + "normalize": _normalize, + "smooth": _smooth, + "differentiate": _differentiate, + "integrate": _integrate, + "linear_fit": _linear_fit, + } + return functions[operation](x, y, parameters) + + +def _origin_value(value: Any) -> Any: + if value is None or isinstance(value, (str, int, float, bool)): + return value + return json.dumps(value, ensure_ascii=False, sort_keys=True) + + +def _write_origin_project( + project: Path, + headers: list[str], + rows: list[list[Any]], + result_headers: list[str], + result_rows: list[list[Any]], + diagnostics: dict[str, Any], + analysis: dict[str, Any], +) -> None: + import originpro as op + + op.set_show(False) + try: + op.new() + raw_sheet = op.new_sheet("w", lname="RawData") + for index, header in enumerate(headers): + raw_sheet.from_list( + index, + [row[index] if index < len(row) else None for row in rows], + lname=header, + ) + result_sheet = op.new_sheet("w", lname="Result") + for index, header in enumerate(result_headers): + result_sheet.from_list( + index, + [row[index] if index < len(row) else None for row in result_rows], + lname=header, + ) + diagnostic_sheet = op.new_sheet("w", lname="Diagnostics") + diagnostic_items = list(diagnostics.items()) + diagnostic_sheet.from_list(0, [item[0] for item in diagnostic_items], lname="metric") + diagnostic_sheet.from_list( + 1, + [_origin_value(item[1]) for item in diagnostic_items], + lname="value", + ) + if analysis["type"] == "linear_fit": + binding = analysis["data"] + x_index = _column_index(headers, binding["x"], "x") + y_index = _column_index(headers, binding["y"], "y") + fit = op.LinearFit() + fit.set_data(raw_sheet, x_index, y_index) + if not analysis["parameters"]["include_intercept"]: + fit.fix_intercept(0) + fit.report(3) + op.save(str(project)) + finally: + if op.oext: + op.exit() + + +def _validate_project(path: Path) -> None: + if not path.is_file() or path.stat().st_size < 128: + raise RuntimeError("ORIGIN_PROJECT_INVALID") + + +def run(job_dir: Path) -> list[dict[str, Any]]: + job_dir = job_dir.resolve(strict=True) + request_record = json.loads( + (job_dir / "request" / "request.json").read_text(encoding="utf-8") + ) + request = request_record["request"] + input_spec = request["inputs"][0] + analysis = request["operation"]["analysis"] + if analysis["data"]["input"] != input_spec["key"]: + raise ValueError("ANALYSIS_INPUT_KEY_NOT_BOUND") + input_path = _input_file(job_dir, input_spec["key"]) + headers, rows = _read_rows(input_path, (input_spec.get("selector") or {}).get("sheet")) + if not headers or len(set(headers)) != len(headers) or any(not item for item in headers): + raise ValueError("INPUT_HEADERS_INVALID") + result_headers, result_rows, diagnostics = analyze(headers, rows, analysis) + + output = job_dir / "output" + output.mkdir(exist_ok=True) + result_table = output / "result-table.csv" + diagnostics_path = output / "diagnostics.json" + analysis_spec_path = output / "analysis-spec.json" + provenance_path = output / "provenance.json" + project_path = output / "analysis.opju" + _atomic_csv(result_table, result_headers, result_rows) + _atomic_json(diagnostics_path, diagnostics) + _atomic_json(analysis_spec_path, request) + _write_origin_project( + project_path, + headers, + rows, + result_headers, + result_rows, + diagnostics, + analysis, + ) + _validate_project(project_path) + try: + originpro_version = version("originpro") + except PackageNotFoundError: + originpro_version = "embedded" + provenance = { + "adapter_version": ADAPTER_VERSION, + "algorithm_version": ALGORITHM_VERSION, + "originpro_version": originpro_version, + "numpy_version": np.__version__, + "request_digest": request_record["request_digest"], + "input": { + "key": input_spec["key"], + "filename": input_path.name, + "sha256": _file_sha256(input_path), + "sheet": (input_spec.get("selector") or {}).get("sheet"), + }, + "operation": analysis["type"], + "origin_linear_fit_report": analysis["type"] == "linear_fit", + } + _atomic_json(provenance_path, provenance) + + artifacts = [ + _manifest(project_path), + _manifest(result_table), + _manifest(diagnostics_path), + _manifest(analysis_spec_path), + _manifest(provenance_path), + ] + if any(item["key"] == "result_workbook" for item in request["outputs"]): + workbook_path = output / "result-table.xlsx" + _atomic_xlsx(workbook_path, result_headers, result_rows) + artifacts.append(_manifest(workbook_path)) + return artifacts + + +def main() -> int: + if sys.argv[1:] == ["--probe"]: + return _probe() + if len(sys.argv) != 2: + print("[ERR] Usage: worker.py ", file=sys.stderr) + return 2 + job_dir = Path(sys.argv[1]) + request_record: dict[str, Any] = {} + try: + request_record = json.loads( + (job_dir / "request" / "request.json").read_text(encoding="utf-8") + ) + artifacts = run(job_dir) + terminal = { + "job_id": request_record["job_id"], + "lease_id": request_record["lease_id"], + "request_digest": request_record["request_digest"], + "status": "succeeded", + "error": {}, + "artifact_manifest": artifacts, + "terminal_at": datetime.now(timezone.utc).isoformat(), + } + _atomic_json(job_dir / "artifacts.json", artifacts) + _atomic_json(job_dir / "terminal.json", terminal) + print("[OK] Origin analysis job completed.") + return 0 + except Exception as exception: # noqa: BLE001 - terminalize every worker failure + terminal = { + "job_id": request_record.get("job_id", ""), + "lease_id": request_record.get("lease_id", ""), + "request_digest": request_record.get("request_digest", ""), + "status": "failed", + "error": {"code": type(exception).__name__, "detail": str(exception)[:500]}, + "artifact_manifest": [], + "terminal_at": datetime.now(timezone.utc).isoformat(), + } + _atomic_json(job_dir / "terminal.json", terminal) + print(f"[ERR] {type(exception).__name__}: {exception}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/windows-node/package-origin-analysis-adapter.bat b/windows-node/package-origin-analysis-adapter.bat new file mode 100644 index 0000000..d4058fd --- /dev/null +++ b/windows-node/package-origin-analysis-adapter.bat @@ -0,0 +1,47 @@ +@echo off +setlocal EnableExtensions EnableDelayedExpansion +cd /d "%~dp0" + +set "OUTPUT_ROOT=%~f1" +if "%~1"=="" set "OUTPUT_ROOT=%~dp0dist" +set "ADAPTER_DIR_NAME=origin.analysis@v1" +set "ARCHIVE_NAME=origin.analysis@v1-adapter.zip" +set "PACKAGE_DIR=!OUTPUT_ROOT!\!ADAPTER_DIR_NAME!" +set "ARCHIVE_PATH=!OUTPUT_ROOT!\!ARCHIVE_NAME!" +set "HASH_PATH=!ARCHIVE_PATH!.sha256" + +where.exe tar.exe >nul 2>&1 +if errorlevel 1 goto :missing_tool +where.exe certutil.exe >nul 2>&1 +if errorlevel 1 goto :missing_tool +if not exist "!OUTPUT_ROOT!" mkdir "!OUTPUT_ROOT!" +if exist "!PACKAGE_DIR!" rmdir /s /q "!PACKAGE_DIR!" +if exist "!ARCHIVE_PATH!" del /f /q "!ARCHIVE_PATH!" +if exist "!HASH_PATH!" del /f /q "!HASH_PATH!" +mkdir "!PACKAGE_DIR!" + +copy /y "adapters\origin.analysis@v1\adapter.json" "!PACKAGE_DIR!\adapter.json" >nul +copy /y "adapters\origin.analysis@v1\worker.py" "!PACKAGE_DIR!\worker.py" >nul +copy /y "adapters\origin.analysis@v1\acceptance.py" "!PACKAGE_DIR!\acceptance.py" >nul +copy /y "adapters\origin.analysis@v1\requirements.txt" "!PACKAGE_DIR!\requirements.txt" >nul +copy /y "..\software-contracts\origin.analysis.v1.json" "!PACKAGE_DIR!\origin.analysis.v1.json" >nul +if errorlevel 1 goto :failed + +tar.exe -a -c -f "!ARCHIVE_PATH!" -C "!OUTPUT_ROOT!" "!ADAPTER_DIR_NAME!" +if errorlevel 1 goto :failed +set "SHA256=" +for /f "tokens=*" %%H in ('certutil.exe -hashfile "!ARCHIVE_PATH!" SHA256 ^| findstr.exe /R /C:"^[0-9A-Fa-f][0-9A-Fa-f ]*[0-9A-Fa-f]$"') do set "SHA256=%%H" +set "SHA256=!SHA256: =!" +if not defined SHA256 goto :failed +>"!HASH_PATH!" echo !SHA256! !ARCHIVE_NAME! +echo [OK] Adapter package: !ARCHIVE_PATH! +echo [OK] SHA256: !SHA256! +exit /b 0 + +:missing_tool +echo [ERR] Windows tar.exe and certutil.exe are required. +exit /b 1 + +:failed +echo [ERR] Origin analysis adapter packaging failed. +exit /b 1