feat(skill): research list 加 pdf_url / xml_url 直链 + 新增 fetch_xml + smoke 扩 trgm/xml 步
paper.py: _LIST_FIELDS 扩到 16(加 publication_date / has_fulltext_xml / pdf_url / xml_url),新加 fetch_xml(id_or_doi, working_dir) helper 走 paper_server media 静态直链(从 paper.xml_url 读,paper_pdf_view 不支持 XML),抽出 _stream_to 共用;fetch_pdf 行为不变。 SKILL.md: 工作流加 "XML 优先 PDF" 原则(已结构化标签 vs OCR 抽取),四函数清单 + 错误处理表更新 fetch_xml / xml_url 空场景。 smoke: 加 step 0 验 pg_trgm 索引速度(>5s 警告 migration 没生效)+ step 4 fetch_xml 多候选轮询 + 复用,step 1 字段集 expected 同步扩到 16。 paper_server 侧改动(serializers pdf_url/xml_url + migration 0006 pg_trgm)见 paper_server 仓库 6a5a5d7b。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b480147fb2
commit
a1c0e71703
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 2-4 句:做了啥 + 关键判断 + 没动什么;细节查 `git log` / `git diff`。
|
||||
|
||||
最后更新:2026-05-21(顶栏 token 累计修 — 5/20 切流式后 `LLM.token_counter` 不再被更新,task 级 tokens_prompt/completion 一直 0;改 `sync_task_tokens` 走 messages SUM,删 TokenCounter 这个冗余内存计数器)
|
||||
最后更新:2026-05-21(research skill 二次迭代:list 加 pdf_url / xml_url 直链 + paper.py 加 fetch_xml + paper_server 加 pg_trgm GIN 索引根治 SearchFilter ILIKE 全表扫;顶栏 token 累计修;同 wd 多 task 并发软警告)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -23,6 +23,8 @@
|
|||
|
||||
### 2026-05-21
|
||||
|
||||
- **research skill 二次迭代:list 端点加 pdf_url / xml_url 直链 + 新增 fetch_xml + pg_trgm GIN 索引根治 SearchFilter timeout**:首版 redeploy 后跑 `scripts/smoke_paper_skill.py` 发现两个 paper_server 既有问题 — P1 `?search=cement` 不带 filter 30s timeout(`title ILIKE '%xxx%' OR ...` 跨 3 列前后通配,B-tree 索引救不了),P2 5/5 候选 paper_pdf_view 全 404(`has_fulltext_pdf=True` 但磁盘文件缺失,DB/disk 不一致)。用户指出 `media/papers/<Y>/<M>/<D>/<safe_doi>.{pdf,xml}` 静态 URL 可直接访问,**且 XML 也走同样路径**(原 paper_pdf_view 只支持 PDF,XML 没任何 API 暴露)。**paper_server 改动**:① `apps/resm/serializers.py` 加两个 SerializerMethodField `pdf_url` / `xml_url`,基于 `obj.publication_date.{year,month,day}` + `safe_doi` 后端拼,用 `request.build_absolute_uri()` 给完整 URL;`has_fulltext_pdf/xml=False` / `publication_date=None`(unknown 目录)返空串避免 LLM 拿到 URL 就 404。② 新 migration `0006_pg_trgm_index.py`:`CREATE EXTENSION IF NOT EXISTS pg_trgm` + 3 列 GIN 索引(`title` / `first_author` / `first_author_institution`),把 ILIKE '%xxx%' 全表扫降到几十 ms。**zcbot 改动**:① `paper.py` `_LIST_FIELDS` 扩到 16 字段(加 `publication_date` / `has_fulltext_xml` / `pdf_url` / `xml_url`);② 新加 `fetch_xml(id_or_doi, working_dir) -> str` helper,对称 `fetch_pdf` 但**走 paper_server media 静态直链**(从 `paper["xml_url"]` 读),`has_fulltext_xml=False` 或 `xml_url` 空都抛 `RuntimeError`;`fetch_pdf` 抽出 `_stream_to(url, dest)` 共用 helper;③ `SKILL.md` 工作流加 "XML 优先 PDF" 原则(XML 已结构化:章节/参考文献/caption 有标签,LLM 友好,免 OCR / PDF 文本抽取)+ fetch_xml 文档 + 错误处理表更新;④ `smoke_paper_skill.py` 新增 step 0(trgm 速度验证,>5s 警告 migration 没跑生效)+ step 4(fetch_xml 链路 ×2 复用 + 多关键词候选轮询)+ step 1 字段集 expected 同步扩到 16。**对比方案**:① abstract 从 list 移除以缓解 P1 —— 复盘 P1 根因是 SearchFilter 跨列 ILIKE 跟 abstract 字段无关,移了不解决问题反失收益,**不动**;② pdf_url 走 paper_pdf_view 端点 —— 仍依赖 `has_fulltext_pdf` 预检 DB 标记,直链跳过 DB 标记,disk 在就能下 + 同时让 XML 走同一范式;③ pg_trgm 索引用 `CONCURRENTLY` 不锁表 —— Django migration 默认 atomic 不支持,且 dev 期表行数小普通 CREATE INDEX 秒级,不必引入 `atomic = False` 复杂度。**Tradeoffs**:① URL 直链字段是后端拼,paper_server media 路径若改 → serializer 同步改一处,前向兼容性强于客户端拼;② pg_trgm 扩展需 superuser 权限一次性安装,普通用户运行 migration 报权限错时手动 `psql -c "CREATE EXTENSION pg_trgm;"` 即可;③ fetch_xml 不走 paper_pdf_view 范式 —— 因为没对应 API,且静态 URL 不依赖 `has_fulltext_xml` DB 标记预检(LLM 能跳过 stale 标记拿到 disk 上的文件);④ P2 数据一致性问题(DB 标 has_fulltext_pdf=True 但 disk 文件丢)**本次不修**,属于 paper_server 抓取 pipeline 的事,smoke 里已加多候选轮询逻辑容忍。**没动**:Paper model / views.py(retrieve mixin / select_related / filterset_class 已在前一条改完)/ urls / paper_pdf_view / `search_fields` / `_resolve_to_id` / DESIGN / RUN。**遗留**:paper_server 两文件(serializers.py + migration 0006)落地,由用户 redeploy + 跑 `python manage.py migrate resm`(migration 0006 会装 pg_trgm 扩展 + 建 3 个 GIN 索引)。
|
||||
|
||||
- **顶栏 token 累计修(`sync_task_tokens` 改走 messages SUM,删 `LLM.TokenCounter`)**:用户报"token 计数一直 0"。复盘:5/20 把 loop 切流式后,`LLM.token_counter.add()` 只在同步 `chat()` 路径里调,新走的 `chat_stream()` 路径从来不更新它;`agent_builder.sync_task_tokens(task_state, llm)` 每轮 run 后从 `llm.token_counter.{prompt,completion}_tokens` 读累计 UPDATE 进 `tasks.tokens_prompt/completion` — 内存计数器永远 0 → tasks 行 0/0 → `_task_dict` 顶栏数字 0。DB 验证:5/20 08:55 之前最后一个 task 4568/934,之后所有 task 0/0;但每条 assistant message 的 `tokens_in/out` 都是对的(3223/1014 这种,`record_chat_usage` 在 loop 里写),所以 source-of-truth 在 messages 表完好,只是 task 级冗余汇总列没同步。**修法**:删 `LLM.TokenCounter` 整个类 + `.token_counter` 属性 + `chat()` 里那行 `.add()` 调用;`sync_task_tokens` 改签名为 `(task_state)`(不再要 llm),内部 `SELECT coalesce(sum(tokens_in),0), coalesce(sum(tokens_out),0) FROM messages WHERE task_id=?` 现算后 UPDATE。`ConsoleEventSink` 同步删 `token_counter` 回调参数 + spinner fmt 的 `ctx N tok` 尾巴(CLI 旁路,改动小)。`web/app.py:273` 调用点改 `sync_task_tokens(task_state)`。**对比方案**:① 一行补丁在 `core/loop.py` 拼回 response 后补 `self.llm.token_counter.add(response.usage)` — 最小,但留着 TokenCounter 这个"内存计数器 vs DB 真相"双写源头不解决根因;② 当前方案改 4 文件去掉冗余,符合"开发期以最优实现为准不留兼容层"。**性能**:`(task_id)` FK + `uq_messages_task_idx` 复合索引,单 task 行数顶天几百,SUM 两 int 亚毫秒,在刚跑完几秒 LLM 的 round-trip 噪声里。**Backfill**:`SELECT task_id, SUM(...) GROUP BY task_id` 一次性把现有 0/0 行修对,4 个 task 累计补正(52943/6593、26191/8687、10138/427、6399/1069)。**没动**:DB schema(`tasks.tokens_prompt/completion` 列保留作汇总展示,只是数据源改 messages 现算)、`record_chat_usage`(per-message 写入逻辑就是真相源)、loop / streaming 流程、DESIGN(§348 描述 `sync_task_tokens 维护` 仍准确,只是实现细节变,不属于架构/schema/API 语义变化)。
|
||||
|
||||
- **同 wd 并发软警告 banner + task header `📁 wd` 仅在 name≠wdName 时显示 + `/v1/tasks` 加 `run_status` 筛选**:用户问"task + working_dir 设计如何 / 同 wd 多 task 并发咋处理",评估了 γ(同 wd 单活 gate)/ short_id 全产物隔离 / clone task 三个方案均判定过度工程 —— dogfood 经验同 wd 基本不并发,定了 Claude Code 同款"信任 + 软警告 + 承认 limitation"。**后端**:`GET /v1/tasks` 加 `run_status` query 参数(逗号分隔,allowlist `idle/running/cancelling/error`,非法静默忽略),拼 `Task.run_status.in_(set)` 条件;复用现有 `(user_id, working_dir)` 索引,同 wd 活跃 task 常态 0/1 行近零开销。**前端**:① `refreshConcurrentWarnings()` 在 `selectTask` + SSE 收尾两点 fire-and-forget 拉 `working_dir=<末段名>&run_status=running,cancelling&page_size=10`,过滤 task_id != current 后存 `state.concurrentWarnings`;② `renderConcurrentWarning()` 在 `#chat-meta` 后 / `#chat-stream` 前插 `#wd-concurrent-warn` 黄底 banner(⚠ + 项目名 + 邻居 task name + run_status + "等 N 个"),非阻塞不挡发送;③ `renderChatMeta` 把 `📁 wdName` 改为"仅 wdName !== taskName 时显示"(留空 fallback 多数场景 name == wd,显示是噪音;不同时显示提示项目归属)。**对比方案**:γ 硬挡破坏对话切换流畅性,short_id 全产物破坏 §7.1 扁平共享语义 + SKILL.md 改造成本,clone task 工程量对零频场景过重;软警告 ~80 行(后端 10 + 前端 70)实现意图,真高频再升级。**没动**:不轮询(接受"邻居 task 在我浏览时收尾,banner 短暂 stale 几秒"边界 — 同 wd 并发本就近 0)、警告无"不再提示"开关、不点击跳目标 task、宪法文件 short_id 命名约定保留(§7.9 2026-05-20 不动)、不加 clone / gate / 物理隔离。**文档**:DESIGN §7.8 风险表"文件级悲观锁"(本就未实现)行替换为"软警告 + known limitation";§7.9 新增 2026-05-21 取舍条说明 γ/short_id/clone 三方案为何都不选 + 引 Claude Code 同款设计。
|
||||
|
|
|
|||
|
|
@ -0,0 +1,308 @@
|
|||
"""Smoke: paper_server → zcbot research skill 三步链路。
|
||||
|
||||
跑法: .venv/Scripts/python.exe scripts/smoke_paper_skill.py
|
||||
|
||||
依赖:paper_server 已 redeploy(list 端点返 abstract / retrieve 端点可用 / filterset_class 生效)。
|
||||
不动 DB,不动 workspace;PDF 落到系统临时目录,跑完即丢。
|
||||
|
||||
三步:
|
||||
1) search(keyword="cement", limit=5) — 验 list shape + abstract 字段
|
||||
2) get_paper(<step 1 拿到的第一条 doi>) — 验 retrieve 端点(原 viewset 没挂 mixin 是 404 bug)
|
||||
3) fetch_pdf(<有 PDF 的那条>, tmp_dir) ×2 — 验文件落盘 + 复用(第二次应跳过下载直接复用)
|
||||
|
||||
任一步异常都打印后 continue 下一步,保证整条链路看一遍。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
# 读 .env 注入(PAPER_SERVER_URL 可在这里覆盖,默认 http://paper.xxhhcty.xyz:8080)
|
||||
env_file = ROOT / ".env"
|
||||
if env_file.exists():
|
||||
for line in env_file.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
k, _, v = line.partition("=")
|
||||
os.environ.setdefault(k.strip(), v.strip())
|
||||
|
||||
from skills.research.paper import _BASE_URL, fetch_pdf, fetch_xml, get_paper, search
|
||||
|
||||
|
||||
def _hr(title: str) -> None:
|
||||
print()
|
||||
print("=" * 60)
|
||||
print(f"[{title}]")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def _truncate(s: str, n: int = 200) -> str:
|
||||
if not s:
|
||||
return "(empty)"
|
||||
s = s.replace("\n", " ")
|
||||
return s if len(s) <= n else s[:n] + "...(+%d chars)" % (len(s) - n)
|
||||
|
||||
|
||||
def step0_trgm_speed() -> None:
|
||||
_hr("step 0: search(keyword='cement', limit=3) 不带 filter — 验证 pg_trgm GIN 索引生效")
|
||||
print("note: 加 pg_trgm 索引前会 30s+ timeout(ILIKE 跨 3 列全表扫);加后应几百 ms 内返")
|
||||
t0 = time.time()
|
||||
try:
|
||||
results = search(keyword="cement", limit=3)
|
||||
dt = (time.time() - t0) * 1000
|
||||
print(f"[OK] returned {len(results)} in {dt:.0f}ms")
|
||||
if dt > 5000:
|
||||
print(f"[WARN] >5s,pg_trgm 索引可能没建对 — 检查 migration 0006_pg_trgm_index 是否跑了")
|
||||
elif dt < 1000:
|
||||
print("[OK] <1s,pg_trgm 索引生效")
|
||||
except Exception as e:
|
||||
print(f"[FAIL] {type(e).__name__}: {e}")
|
||||
print(" (若是 ReadTimeout,pg_trgm 没生效;请确认 migration 跑了 + DB 有 superuser 装扩展)")
|
||||
|
||||
|
||||
def step1_search() -> list[dict]:
|
||||
_hr("step 1: search(keyword='cement', has_pdf=True, limit=5)")
|
||||
print(f"PAPER_SERVER_URL = {_BASE_URL}")
|
||||
print("note: 带 has_pdf=True 走 has_fulltext_pdf 索引先过滤,纯 keyword 大词会 ILIKE 全表扫导致 30s timeout (paper_server 既有问题)")
|
||||
t0 = time.time()
|
||||
try:
|
||||
results = search(keyword="cement", has_pdf=True, limit=5)
|
||||
except Exception as e:
|
||||
print(f"[FAIL] {type(e).__name__}: {e}")
|
||||
return []
|
||||
dt = (time.time() - t0) * 1000
|
||||
print(f"[OK] returned {len(results)} papers in {dt:.0f}ms")
|
||||
if not results:
|
||||
print("[WARN] 0 results — keyword 'cement' + has_pdf=True 居然没命中?后两步会用空集 fallback")
|
||||
return []
|
||||
# 验返回 shape:必含 16 字段
|
||||
expected = {
|
||||
"id", "doi", "title", "first_author", "first_author_institution",
|
||||
"publication_year", "publication_date", "publication_name",
|
||||
"has_fulltext_pdf", "has_fulltext_xml", "has_abstract", "is_oa",
|
||||
"type", "abstract", "pdf_url", "xml_url",
|
||||
}
|
||||
actual = set(results[0].keys())
|
||||
missing = expected - actual
|
||||
extra = actual - expected
|
||||
if missing:
|
||||
print(f"[FAIL] 字段缺失:{missing}")
|
||||
if extra:
|
||||
print(f"[INFO] 字段冗余(应被 _LIST_FIELDS 过滤掉):{extra}")
|
||||
if not missing and not extra:
|
||||
print("[OK] 16 字段完整,无冗余")
|
||||
# 抽样打前 3 条
|
||||
print()
|
||||
for i, p in enumerate(results[:3], 1):
|
||||
print(f"--- paper {i} ---")
|
||||
print(f" id = {p['id']}")
|
||||
print(f" doi = {p['doi']}")
|
||||
print(f" title = {_truncate(p['title'], 100)}")
|
||||
print(f" year = {p['publication_year']} / date={p['publication_date']} / type={p['type']} / is_oa={p['is_oa']}")
|
||||
print(f" has_abstract = {p['has_abstract']}")
|
||||
print(f" has_fulltext_pdf = {p['has_fulltext_pdf']} / has_fulltext_xml = {p['has_fulltext_xml']}")
|
||||
print(f" pdf_url = {p['pdf_url'] or '(empty)'}")
|
||||
print(f" xml_url = {p['xml_url'] or '(empty)'}")
|
||||
print(f" abstract = {_truncate(p['abstract'], 150)}")
|
||||
return results
|
||||
|
||||
|
||||
def step1b_abstract_filled() -> None:
|
||||
_hr("step 1b: search(has_abstract=True, limit=3) — 验证 abstract 字段在 list 里确实能装文本")
|
||||
try:
|
||||
t0 = time.time()
|
||||
results = search(keyword="hydration", has_pdf=True, limit=3)
|
||||
dt = (time.time() - t0) * 1000
|
||||
print(f"[OK] returned {len(results)} in {dt:.0f}ms")
|
||||
if not results:
|
||||
print("[WARN] hydration+has_pdf 也是 0 结果,跳过")
|
||||
return
|
||||
for i, p in enumerate(results, 1):
|
||||
print(f"--- paper {i} --- has_abstract={p['has_abstract']}")
|
||||
print(f" title = {_truncate(p['title'], 80)}")
|
||||
print(f" abstract = {_truncate(p['abstract'], 200)}")
|
||||
any_filled = any(p["abstract"] for p in results if p.get("has_abstract"))
|
||||
if any_filled:
|
||||
print("\n[OK] abstract 字段实际装载了文本(non-empty)— list 加 abstract 改动真实生效")
|
||||
else:
|
||||
print('\n[INFO] 本批候选 has_abstract 全为 False / abstract 全空;不算 fail,只说明本批走的 serializer 的 "无 PaperAbstract 行返空串" 分支')
|
||||
except Exception as e:
|
||||
print(f"[FAIL] {type(e).__name__}: {e}")
|
||||
|
||||
|
||||
def step2_get_paper(papers: list[dict]) -> None:
|
||||
_hr("step 2: get_paper(<step 1 第一条 doi>)")
|
||||
if not papers:
|
||||
print("[SKIP] step 1 无结果,无法挑 doi")
|
||||
return
|
||||
doi = papers[0]["doi"]
|
||||
print(f"querying doi = {doi}")
|
||||
try:
|
||||
t0 = time.time()
|
||||
paper = get_paper(doi)
|
||||
dt = (time.time() - t0) * 1000
|
||||
print(f"[OK] retrieve in {dt:.0f}ms")
|
||||
except Exception as e:
|
||||
print(f"[FAIL] {type(e).__name__}: {e}")
|
||||
print(" (若是 404,可能 redeploy 没生效 / retrieve mixin 没挂;原 bug 就是这个)")
|
||||
return
|
||||
print(f" id = {paper.get('id')}")
|
||||
print(f" title = {_truncate(paper.get('title') or '', 100)}")
|
||||
print(f" has_abstract = {paper.get('has_abstract')}")
|
||||
print(f" abstract = {_truncate(paper.get('abstract') or '', 200)}")
|
||||
if paper.get("id") != papers[0]["id"]:
|
||||
print(f"[WARN] retrieve 返回的 id ({paper.get('id')}) ≠ list 里的 id ({papers[0]['id']})")
|
||||
|
||||
|
||||
def step3_fetch_pdf(papers: list[dict]) -> None:
|
||||
_hr("step 3: fetch_pdf(轮询候选直到命中真实可下载, tmp_dir) ×2")
|
||||
candidates = [p for p in papers if p.get("has_fulltext_pdf")]
|
||||
if not candidates:
|
||||
print("[SKIP] step 1 候选里没有 has_fulltext_pdf=True,跳过(P1 timeout 也会落到这里)")
|
||||
return
|
||||
|
||||
import httpx as _httpx
|
||||
tmp_dir = Path(tempfile.mkdtemp(prefix="zcbot_smoke_paper_"))
|
||||
print(f"tmp working_dir = {tmp_dir}")
|
||||
print(f"候选数: {len(candidates)}")
|
||||
success = False
|
||||
for i, paper in enumerate(candidates, 1):
|
||||
pid = paper["id"]
|
||||
print(f"\n--- 尝试候选 {i}/{len(candidates)}: id={pid} doi={paper['doi']} ---")
|
||||
try:
|
||||
t0 = time.time()
|
||||
rel1 = fetch_pdf(pid, str(tmp_dir))
|
||||
dt1 = (time.time() - t0) * 1000
|
||||
abs_path = tmp_dir / rel1
|
||||
size = abs_path.stat().st_size
|
||||
print(f"[OK] 第 1 次下载: rel={rel1} size={size/1024:.1f}KB in {dt1:.0f}ms")
|
||||
t0 = time.time()
|
||||
rel2 = fetch_pdf(pid, str(tmp_dir))
|
||||
dt2 = (time.time() - t0) * 1000
|
||||
print(f"[OK] 第 2 次复用: rel={rel2} in {dt2:.0f}ms (期望 <100ms)")
|
||||
if dt2 > 1000:
|
||||
print(f"[WARN] 第 2 次 >1s,可能没走复用分支?")
|
||||
if rel1 != rel2:
|
||||
print(f"[FAIL] 两次返回路径不一致:{rel1} vs {rel2}")
|
||||
success = True
|
||||
break
|
||||
except _httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
print(f"[SKIP] paper_pdf_view 404 — DB has_fulltext_pdf=True 但磁盘文件缺失(paper_server 数据一致性问题,继续下一个)")
|
||||
continue
|
||||
print(f"[FAIL] HTTPStatusError {e.response.status_code}: {e}")
|
||||
break
|
||||
except RuntimeError as e:
|
||||
print(f"[INFO] {e}")
|
||||
continue
|
||||
except Exception as e:
|
||||
print(f"[FAIL] {type(e).__name__}: {e}")
|
||||
break
|
||||
if not success:
|
||||
print(f"\n[WARN] {len(candidates)} 个候选全部 404 — paper_server 侧 DB/disk 不一致问题严重,fetch_pdf 链路逻辑未真实验证(本地实现按 has_fulltext_pdf=True 是放行的,服务器侧最终又 404 拦下)")
|
||||
print(f"(tmp_dir 留着自查:{tmp_dir})")
|
||||
|
||||
|
||||
def step4_fetch_xml() -> None:
|
||||
_hr("step 4: fetch_xml(<has_fulltext_xml=True 候选>, tmp_dir) ×2 — 验证 XML 直链 + 复用")
|
||||
try:
|
||||
candidates = search(keyword="cement", limit=20)
|
||||
except Exception as e:
|
||||
print(f"[FAIL] 拉候选失败: {type(e).__name__}: {e}")
|
||||
return
|
||||
xml_candidates = [p for p in candidates if p.get("has_fulltext_xml") and p.get("xml_url")]
|
||||
print(f"候选 {len(candidates)} 条中,has_fulltext_xml=True 且 xml_url 非空: {len(xml_candidates)} 条")
|
||||
if not xml_candidates:
|
||||
print("[INFO] 库里 cement 主题没 has_fulltext_xml 候选,试别的关键词")
|
||||
for kw in ("hydration", "concrete", "polymer"):
|
||||
try:
|
||||
more = search(keyword=kw, limit=20)
|
||||
xml_candidates = [p for p in more if p.get("has_fulltext_xml") and p.get("xml_url")]
|
||||
if xml_candidates:
|
||||
print(f" '{kw}' 拉到 {len(xml_candidates)} 条 XML 候选")
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
if not xml_candidates:
|
||||
print("[SKIP] 多关键词试了都没 XML 候选;fetch_xml 链路未验证")
|
||||
return
|
||||
tmp_dir = Path(tempfile.mkdtemp(prefix="zcbot_smoke_xml_"))
|
||||
print(f"tmp working_dir = {tmp_dir}")
|
||||
import httpx as _httpx
|
||||
success = False
|
||||
for i, paper in enumerate(xml_candidates[:5], 1):
|
||||
pid = paper["id"]
|
||||
print(f"\n--- 尝试候选 {i}: id={pid} doi={paper['doi']} xml_url={paper['xml_url']} ---")
|
||||
try:
|
||||
t0 = time.time()
|
||||
rel1 = fetch_xml(pid, str(tmp_dir))
|
||||
dt1 = (time.time() - t0) * 1000
|
||||
size = (tmp_dir / rel1).stat().st_size
|
||||
print(f"[OK] 第 1 次下载: rel={rel1} size={size/1024:.1f}KB in {dt1:.0f}ms")
|
||||
t0 = time.time()
|
||||
rel2 = fetch_xml(pid, str(tmp_dir))
|
||||
dt2 = (time.time() - t0) * 1000
|
||||
print(f"[OK] 第 2 次复用: rel={rel2} in {dt2:.0f}ms (期望 <100ms)")
|
||||
if rel1 != rel2:
|
||||
print(f"[FAIL] 两次路径不一致:{rel1} vs {rel2}")
|
||||
success = True
|
||||
break
|
||||
except _httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
print(f"[SKIP] media 静态 URL 404 — paper_server disk 文件缺失,继续下一个")
|
||||
continue
|
||||
print(f"[FAIL] HTTPStatusError {e.response.status_code}: {e}")
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"[FAIL] {type(e).__name__}: {e}")
|
||||
break
|
||||
if not success:
|
||||
print(f"\n[WARN] 5 个候选全部 disk 缺失;fetch_xml 客户端代码本身简单(直链 stream + 复用),不阻塞改动验证")
|
||||
print(f"(tmp_dir 留着自查:{tmp_dir})")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
print("=" * 60)
|
||||
print("zcbot research skill smoke")
|
||||
print("=" * 60)
|
||||
try:
|
||||
step0_trgm_speed()
|
||||
except Exception as e:
|
||||
print(f"[FAIL step 0] {type(e).__name__}: {e}")
|
||||
try:
|
||||
papers = step1_search()
|
||||
except Exception as e:
|
||||
print(f"[FAIL step 1] {type(e).__name__}: {e}")
|
||||
papers = []
|
||||
try:
|
||||
step1b_abstract_filled()
|
||||
except Exception as e:
|
||||
print(f"[FAIL step 1b] {type(e).__name__}: {e}")
|
||||
try:
|
||||
step2_get_paper(papers)
|
||||
except Exception as e:
|
||||
print(f"[FAIL step 2] {type(e).__name__}: {e}")
|
||||
try:
|
||||
step3_fetch_pdf(papers)
|
||||
except Exception as e:
|
||||
print(f"[FAIL step 3] {type(e).__name__}: {e}")
|
||||
try:
|
||||
step4_fetch_xml()
|
||||
except Exception as e:
|
||||
print(f"[FAIL step 4] {type(e).__name__}: {e}")
|
||||
print()
|
||||
print("=" * 60)
|
||||
print("smoke done")
|
||||
print("=" * 60)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -5,88 +5,131 @@ description: 查 paper_server 文献库(基于 OpenAlex 元数据 + Sci-Hub 下
|
|||
|
||||
# Research
|
||||
|
||||
paper_server 是内部部署的 Django 文献库:元数据来自 OpenAlex,PDF 由 Sci-Hub 异步抓取。本 skill 给你三个 helper(`search` / `get_paper` / `fetch_pdf`),用 `run_python` 调用,**不要**自己 `httpx` 裸调 API。
|
||||
paper_server 是内部部署的 Django 文献库:元数据来自 OpenAlex,PDF / XML 由 Sci-Hub / OpenAlex 异步抓取。**库里主语料是英文**(OpenAlex 主索引英文文献),少量中文。本 skill 给你四个 helper(`search` / `get_paper` / `fetch_pdf` / `fetch_xml`),用 `run_python` 调用,**不要**自己 `httpx` 裸调 API。
|
||||
|
||||
## 何时用
|
||||
|
||||
- 用户要查 / 找 / 看 / 推荐文献
|
||||
- 要 DOI、要某篇 PDF、要 abstract
|
||||
- 写申报书 / 研究方案 / 调研报告的"国内外现状"段需要真实文献支撑
|
||||
- 配合 `proposal` skill 的「立项依据」起草 —— 先 `search` 拿候选,`get_paper` 看 abstract 决定要不要引
|
||||
- 配合 `proposal` skill 的「立项依据」起草 —— 先 `search` 拿候选,看 abstract 决定要不要引
|
||||
|
||||
## 何时不用
|
||||
|
||||
- 用户只问通识(直接答即可,不需要文献支撑)
|
||||
- 用户已经给了具体文献清单(直接用,不要二次校验)
|
||||
- paper_server 没覆盖的领域(本库主打中文工程 / 材料 / 土木 / 化学;前沿 AI / 数学 / 纯理论 paper 命中率低 —— 命中 0 条直接告诉用户,不要瞎编)
|
||||
|
||||
## 准备
|
||||
|
||||
```python
|
||||
from skills.research.paper import search, get_paper, fetch_pdf
|
||||
from skills.research.paper import search, get_paper, fetch_pdf, fetch_xml
|
||||
```
|
||||
|
||||
(import 路径由 `run_python` 注入的 `PYTHONPATH` 提供,直接写就行,不必折腾 `sys.path`)
|
||||
|
||||
## 三个函数
|
||||
## 关键:keyword 优先用英文
|
||||
|
||||
### `search(keyword="", year=None, doi="", has_pdf=None, limit=10) -> list[dict]`
|
||||
`search(keyword=...)` 走 paper_server SearchFilter,模糊匹配 **title / first_author / first_author_institution**(目前不含 abstract)。库里 95%+ 文献 title 是英文,中文 keyword 命中率很低。
|
||||
|
||||
搜文献,返回精简列表(每条只含 id / doi / title / first_author / publication_year / publication_name / has_fulltext_pdf / has_abstract / type)。
|
||||
**用户输入中文 → 先转成专业英文术语再 search**:
|
||||
|
||||
- `keyword`: paper_server 的 search 字段,匹配 title / first_author / first_author_institution
|
||||
- `year`: 精确年份(目前只支持 exact)
|
||||
| 用户原话 | 不要这样 | 这样 |
|
||||
|---|---|---|
|
||||
| 水泥水化 | `search("水泥水化")` | `search("cement hydration")` |
|
||||
| 钢筋锈蚀 | `search("钢筋锈蚀")` | `search("steel reinforcement corrosion")` |
|
||||
| 混凝土碳化 | `search("混凝土碳化")` | `search("concrete carbonation")` |
|
||||
| 锂离子电池电解液 | `search("锂离子电池电解液")` | `search("lithium-ion battery electrolyte")` |
|
||||
|
||||
转译策略:
|
||||
- 用领域内标准英文术语(查不准就先英文 keyword 试一次看返回 title 是不是相关)
|
||||
- 多词术语用空格分隔(`SearchFilter` 默认空格视作 AND,要更宽用单词)
|
||||
- 不确定时同义词都试一遍:`search("CO2 absorption")` 没结果 → 试 `search("carbon dioxide capture")`
|
||||
- 中文期刊 paper 也可中文 keyword 单独搜一次(`search("水泥") + filter 中文期刊` 命中率不算低,但远不如英文主搜)
|
||||
|
||||
## 四个函数
|
||||
|
||||
### `search(keyword="", year=None, year_gte=None, year_lte=None, doi="", first_author="", publication_name="", has_pdf=None, is_oa=None, limit=10) -> list[dict]`
|
||||
|
||||
搜文献,返回精简列表(每条含 16 字段:id / doi / title / first_author / first_author_institution / publication_year / publication_date / publication_name / has_fulltext_pdf / has_fulltext_xml / has_abstract / is_oa / type / **abstract** / **pdf_url** / **xml_url**)。
|
||||
|
||||
- `keyword`: SearchFilter,匹配 title / first_author / first_author_institution(**英文为主**,见上节)
|
||||
- `year` / `year_gte` / `year_lte`: 精确年份 / 范围(做"近 5 年文献"用 `year_gte=2020`)
|
||||
- `doi`: 精确 DOI(命中 0 / 1 条)
|
||||
- `has_pdf=True` 仅返已下好 PDF 的;`False` 仅返没 PDF 的;`None` 都返
|
||||
- `first_author` / `publication_name`: 精确作者 / 期刊
|
||||
- `has_pdf=True` 仅返 PDF 已下好的;`False` 仅返没 PDF 的;`None` 都返
|
||||
- `is_oa=True` 仅返开放获取(OA);`False` 仅返非 OA;`None` 都返
|
||||
- `limit`: 默认 10,上限 50
|
||||
|
||||
```python
|
||||
# 找最近 5 篇关于"水泥水化"且已下好 PDF 的
|
||||
papers = search(keyword="水泥水化", has_pdf=True, limit=5)
|
||||
# 找近 5 年水泥水化研究,且 PDF 已下好
|
||||
papers = search(keyword="cement hydration", year_gte=2020, has_pdf=True, limit=10)
|
||||
for p in papers:
|
||||
print(p["title"], p["doi"], p["publication_year"])
|
||||
print(p["title"], p["publication_year"])
|
||||
if p["abstract"]:
|
||||
print(p["abstract"][:200]) # 看摘要前 200 字判断是否切题
|
||||
```
|
||||
|
||||
### `get_paper(id_or_doi) -> dict`
|
||||
|
||||
取单条完整 metadata + abstract。`id_or_doi` 既接受 paper_server 内部 id,也接受 DOI(自动解析)。
|
||||
|
||||
**list 端点已带 abstract,正常工作流不需要调本函数** —— 仅在用户给单个 id / DOI 想拿全字段(含 OpenAlex 原始字段如 `o_keywords` 等)时用。
|
||||
|
||||
```python
|
||||
paper = get_paper("10.1016/j.cemconres.2020.106156")
|
||||
print(paper["title"])
|
||||
print(paper["abstract"]) # 没 abstract 时是空串,不会抛
|
||||
print(paper["abstract"])
|
||||
```
|
||||
|
||||
### `fetch_pdf(id_or_doi, working_dir) -> str`
|
||||
|
||||
下载 PDF 到 `<working_dir>/papers/<safe_doi>.pdf`,返回相对路径 `papers/<safe_doi>.pdf`(safe_doi 把 `/` 换成 `_`)。已存在跳过下载直接复用。
|
||||
下载 PDF 到 `<working_dir>/papers/<safe_doi>.pdf`,返回相对路径 `papers/<safe_doi>.pdf`(safe_doi 把 `/` 换成 `_`)。已存在跳过下载直接复用。走 paper_server 的 `/resm/paper/<id>/pdf/` 端点(有 `has_fulltext_pdf` 预检)。
|
||||
|
||||
`has_fulltext_pdf=False` 时抛 `RuntimeError` —— 服务器侧还没下到 PDF。
|
||||
|
||||
```python
|
||||
rel = fetch_pdf("10.1016/j.cemconres.2020.106156", working_dir=r"D:/projects/zcbot/workspace/users/<uid>/<wd>")
|
||||
# rel == "papers/10.1016_j.cemconres.2020.106156.pdf"
|
||||
```
|
||||
|
||||
`has_fulltext_pdf=False` 时抛 `RuntimeError` —— 这时该 paper 服务器侧还没下到 PDF,告诉用户不要硬等。
|
||||
### `fetch_xml(id_or_doi, working_dir) -> str`
|
||||
|
||||
下载 XML 到 `<working_dir>/papers/<safe_doi>.xml`,对称 `fetch_pdf`。**走 paper_server 的 media 静态直链**(由 list/retrieve 返回的 `xml_url` 字段提供),paper_pdf_view 只覆盖 PDF,XML 没对应 API。已存在跳过下载直接复用。
|
||||
|
||||
`has_fulltext_xml=False` 或 `xml_url` 空(publication_date 缺失时会空)→ 抛 `RuntimeError`。
|
||||
|
||||
**为什么 XML 优先 PDF**:XML 已结构化 —— 章节标题 / 摘要 / 段落 / 参考文献 / 图表 caption 都有标签,LLM 读取无需 OCR 或 PDF 文本抽取的不确定性;文献综述 / 引文清单 / 章节定位场景比 PDF 友好得多。能拿 XML 就别拿 PDF;只有需要看具体公式 / 图表内容 / 表格数据时才下 PDF。
|
||||
|
||||
## 标准工作流
|
||||
|
||||
1. **search**:按关键词 / 年份缩窄候选(`limit=10` 起,不够再扩)
|
||||
2. **筛选**:扫返回列表,挑相关的(看 title + first_author + year),`has_fulltext_pdf=True` 的优先
|
||||
3. **get_paper**:对每篇候选拿 abstract,确认确实切题再继续
|
||||
4. **fetch_pdf**:确定要全文细读时才下载(用户没说要全文就别下,abstract 已足够覆盖大多数引用场景)
|
||||
5. **read PDF**:`fetch_pdf` 返回相对路径,用主 agent 的 `read` 工具读 PDF 提取关键段(zcbot 已内置 PDF 文本提取)
|
||||
1. **(若用户输入中文)转专业英文术语**
|
||||
2. **search**:按 keyword + filter(年份范围 / OA / has_pdf 等)缩窄候选,`limit=10` 起
|
||||
3. **直接看返回里的 abstract**(list 端点已带):
|
||||
- abstract 非空 → 看前 200-400 字判断切题
|
||||
- abstract 空(`has_abstract=False`)→ 仅凭 title + 期刊 + 年份判断,信号弱时下条候选
|
||||
4. **下全文(若需要)** —— **优先级**:
|
||||
- `has_fulltext_xml=True` → `fetch_xml`(LLM 友好,结构化)
|
||||
- 仅 `has_fulltext_pdf=True` → `fetch_pdf`(回退,需要 PDF 文本抽取)
|
||||
- 两者都 False → 仅凭 abstract 写综述,告诉用户哪几篇没全文
|
||||
5. **read 全文**:fetch 返回相对路径,用主 agent 的 `read` 工具读取(zcbot 已内置 PDF 文本抽取;XML 直接当文本读)
|
||||
|
||||
## 错误处理
|
||||
|
||||
- 网络超时 / paper_server 不可达:`httpx.ConnectError` / `httpx.TimeoutException` —— 直接告诉用户"paper_server 暂时连不上",不要重试堆栈刷屏
|
||||
- `doi 未命中` / `doi 命中多条`:`get_paper` / `fetch_pdf` 内部 `_resolve_to_id` 抛 `ValueError` —— DOI 拼写错或库里真没收录,改 keyword 重搜
|
||||
- `has_fulltext_pdf=False`:`fetch_pdf` 抛 `RuntimeError(reason=...)` —— 服务器还没下到 PDF;告诉用户"这篇 paper_server 还没下到 PDF,可读 abstract 或换一篇"
|
||||
- 网络超时 / paper_server 不可达:`httpx.ConnectError` / `httpx.TimeoutException` —— 告诉用户"paper_server 暂时连不上",不要重试堆栈刷屏
|
||||
- `doi 未命中` / `doi 命中多条`:`get_paper` / `fetch_pdf` / `fetch_xml` 内部 `_resolve_to_id` 抛 `ValueError` —— DOI 拼写错或库里没收录,改 keyword 重搜
|
||||
- `has_fulltext_pdf=False` / `has_fulltext_xml=False`:`fetch_pdf` / `fetch_xml` 抛 `RuntimeError` —— 服务器还没下到对应格式;若另一格式存在则换用,都没就只能用 abstract
|
||||
- `xml_url` 空(publication_date 缺失,paper 落到 unknown 目录):`fetch_xml` 抛 `RuntimeError(xml_url unavailable...)` —— 改试 `fetch_pdf`
|
||||
- 文件 disk 缺失(`has_fulltext_pdf=True` 但 paper_server 侧文件丢了 → HTTP 404):helper 透传 `httpx.HTTPStatusError`,告诉用户换一篇
|
||||
- `abstract` 字段为空字符串:正常情况,不是错;告诉用户"这篇没收录摘要"即可
|
||||
- search 命中 0 条:先尝试同义词 / 缩短 keyword / 放宽 filter,3 次还是 0 条告诉用户库里没覆盖,**不要凭训练数据脑补文献**
|
||||
|
||||
## 反模式
|
||||
|
||||
- 用 `httpx` / `requests` 裸调 paper_server API(走 helper,免得 base_url / auth / 字段名漂移时四处改)
|
||||
- `search(limit=50)` 一次拉满后扔给 LLM 全文 dump(只 print 前 5-10 条精简就够,要全部让用户自己 `print(papers)`)
|
||||
- 没看 abstract 就 `fetch_pdf`(80% 场景 abstract 够用,下载 PDF 慢且费带宽)
|
||||
- 用户输中文直接 `search(中文)` —— 转英文术语,见上节
|
||||
- `search(limit=50)` 一次拉满后 dump 给 LLM 全文(只 print 前 5-10 条精简就够,要全部让用户自己 `print(papers)`)
|
||||
- 已经看到 abstract 还 `get_paper` 一遍(list 已带 abstract,重复调白费 roundtrip)
|
||||
- 没看 abstract 就 `fetch_pdf` / `fetch_xml`(80% 场景 abstract 够用,下载全文慢且费带宽)
|
||||
- `has_fulltext_xml=True` 时盲走 `fetch_pdf`(XML 对 LLM 更友好,先试 fetch_xml)
|
||||
- 编造 DOI / title / 作者 —— 不在 paper_server 库里就**明确告诉用户"未命中"**,不要凭训练数据脑补
|
||||
- 把 `fetch_pdf` 返回的相对路径当绝对路径用(它是相对 `working_dir` 的)
|
||||
|
|
|
|||
|
|
@ -17,11 +17,18 @@ _LIST_FIELDS = (
|
|||
"doi",
|
||||
"title",
|
||||
"first_author",
|
||||
"first_author_institution",
|
||||
"publication_year",
|
||||
"publication_date",
|
||||
"publication_name",
|
||||
"has_fulltext_pdf",
|
||||
"has_fulltext_xml",
|
||||
"has_abstract",
|
||||
"is_oa",
|
||||
"type",
|
||||
"abstract",
|
||||
"pdf_url",
|
||||
"xml_url",
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -52,17 +59,27 @@ def _resolve_to_id(id_or_doi: str) -> str:
|
|||
def search(
|
||||
keyword: str = "",
|
||||
year: Optional[int] = None,
|
||||
year_gte: Optional[int] = None,
|
||||
year_lte: Optional[int] = None,
|
||||
doi: str = "",
|
||||
first_author: str = "",
|
||||
publication_name: str = "",
|
||||
has_pdf: Optional[bool] = None,
|
||||
is_oa: Optional[bool] = None,
|
||||
limit: int = 10,
|
||||
) -> list[dict]:
|
||||
"""搜文献,返回精简列表。
|
||||
"""搜文献,返回精简列表(每条含 abstract 字段,有就是文本,没就是空串)。
|
||||
|
||||
keyword: paper_server search 字段,匹配 title / first_author / first_author_institution
|
||||
year: 精确年份(paper_server 当前只支持 exact)
|
||||
doi: 精确 DOI(命中 0/1 条)
|
||||
has_pdf: True 仅返已下好 PDF;False 仅返没 PDF;None 都返
|
||||
limit: 默认 10,上限 50
|
||||
keyword: paper_server SearchFilter,模糊匹配 title / first_author / first_author_institution
|
||||
库里主语料是英文,**优先英文关键词**(用户中文输入要先转专业英文术语)
|
||||
year: 精确年份
|
||||
year_gte/year_lte: 年份范围(做"近 N 年文献"用)
|
||||
doi: 精确 DOI(命中 0/1 条)
|
||||
first_author: 精确作者名
|
||||
publication_name: 精确期刊名
|
||||
has_pdf: True 仅返 PDF 已下好的;False 仅返没 PDF 的;None 都返
|
||||
is_oa: True 仅返 OA 的;False 仅返非 OA;None 都返
|
||||
limit: 默认 10,上限 50
|
||||
"""
|
||||
if limit > 50:
|
||||
limit = 50
|
||||
|
|
@ -71,12 +88,24 @@ def search(
|
|||
params["search"] = keyword
|
||||
if year is not None:
|
||||
params["publication_year"] = year
|
||||
if year_gte is not None:
|
||||
params["publication_year_gte"] = year_gte
|
||||
if year_lte is not None:
|
||||
params["publication_year_lte"] = year_lte
|
||||
if doi:
|
||||
params["doi"] = doi
|
||||
if first_author:
|
||||
params["first_author"] = first_author
|
||||
if publication_name:
|
||||
params["publication_name"] = publication_name
|
||||
if has_pdf is True:
|
||||
params["has_fulltext"] = "true"
|
||||
params["has_fulltext_pdf"] = "true"
|
||||
elif has_pdf is False:
|
||||
params["has_fulltext"] = "false"
|
||||
params["has_fulltext_pdf"] = "false"
|
||||
if is_oa is True:
|
||||
params["is_oa"] = "true"
|
||||
elif is_oa is False:
|
||||
params["is_oa"] = "false"
|
||||
r = httpx.get(_API + "/", params=params, timeout=_TIMEOUT)
|
||||
r.raise_for_status()
|
||||
data = r.json()
|
||||
|
|
@ -85,9 +114,9 @@ def search(
|
|||
|
||||
|
||||
def get_paper(id_or_doi: str) -> dict:
|
||||
"""取单条 metadata + abstract(走 retrieve 端点)。
|
||||
"""取单条完整 metadata + abstract。
|
||||
|
||||
abstract 字段由 paper_server retrieve serializer 提供;无 PaperAbstract 行时返空串。
|
||||
list 端点已带 abstract,正常工作流不需要调本函数;仅在用户给单个 id/DOI 想拿全字段时用。
|
||||
"""
|
||||
pid = _resolve_to_id(id_or_doi)
|
||||
r = httpx.get(f"{_API}/{pid}/", timeout=_TIMEOUT)
|
||||
|
|
@ -95,6 +124,15 @@ def get_paper(id_or_doi: str) -> dict:
|
|||
return r.json()
|
||||
|
||||
|
||||
def _stream_to(url: str, dest: Path) -> None:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
with httpx.stream("GET", url, timeout=60.0) as resp:
|
||||
resp.raise_for_status()
|
||||
with open(dest, "wb") as f:
|
||||
for chunk in resp.iter_bytes(chunk_size=64 * 1024):
|
||||
f.write(chunk)
|
||||
|
||||
|
||||
def fetch_pdf(id_or_doi: str, working_dir: str) -> str:
|
||||
"""下载 PDF 到 <working_dir>/papers/<safe_doi>.pdf,返回相对路径 'papers/<safe_doi>.pdf'。
|
||||
|
||||
|
|
@ -109,10 +147,31 @@ def fetch_pdf(id_or_doi: str, working_dir: str) -> str:
|
|||
dest = Path(working_dir) / rel
|
||||
if dest.exists() and dest.stat().st_size > 0:
|
||||
return rel
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
with httpx.stream("GET", f"{_PDF}/{paper['id']}/pdf/", timeout=60.0) as resp:
|
||||
resp.raise_for_status()
|
||||
with open(dest, "wb") as f:
|
||||
for chunk in resp.iter_bytes(chunk_size=64 * 1024):
|
||||
f.write(chunk)
|
||||
_stream_to(f"{_PDF}/{paper['id']}/pdf/", dest)
|
||||
return rel
|
||||
|
||||
|
||||
def fetch_xml(id_or_doi: str, working_dir: str) -> str:
|
||||
"""下载 XML 到 <working_dir>/papers/<safe_doi>.xml,返回相对路径 'papers/<safe_doi>.xml'。
|
||||
|
||||
XML 走 paper_server media 静态直链(由 list/retrieve 返回的 xml_url 字段提供);
|
||||
paper_pdf_view 只覆盖 PDF,XML 没对应 API。
|
||||
paper.has_fulltext_xml=False / xml_url 空 → 抛 RuntimeError。
|
||||
已存在跳过下载直接复用。
|
||||
"""
|
||||
paper = get_paper(id_or_doi)
|
||||
if not paper.get("has_fulltext_xml"):
|
||||
raise RuntimeError(f"paper has no XML: id={paper.get('id')}")
|
||||
xml_url = paper.get("xml_url") or ""
|
||||
if not xml_url:
|
||||
# publication_date 缺失(unknown 目录)→ paper_server 没暴露这层 media URL
|
||||
raise RuntimeError(
|
||||
f"paper xml_url unavailable (likely missing publication_date): id={paper.get('id')}"
|
||||
)
|
||||
safe = _safe_doi(paper["doi"])
|
||||
rel = f"papers/{safe}.xml"
|
||||
dest = Path(working_dir) / rel
|
||||
if dest.exists() and dest.stat().st_size > 0:
|
||||
return rel
|
||||
_stream_to(xml_url, dest)
|
||||
return rel
|
||||
|
|
|
|||
Loading…
Reference in New Issue