feat(media): look_at_image 图像理解(豆包 Seed 2.0 Lite vision)+ bump 0.16.0
DESIGN §8.1 C 路落地 —— 主模型 DeepSeek V4 纯文本无视觉,挂 look_at_image 工具按需读图(OCR / 描述 / 读图表),模型自决何时调。 - 选型:设计时的 Seed 1.6 vision 已过时,改用 Doubao Seed 2.0 Lite (doubao-seed-2-0-lite-260428,全模态 SOTA 细粒度感知)。token 计费 输入 ¥0.6 / 输出 ¥3.6 /Mtok,一次读图 < ¥0.01 - 后端:tools/look_at_image.py(/chat/completions base64 单图+问题→文本解读); doubao.yaml 加 vision 段;usage.py 加 record_vision_usage(kind=vision, 按 token,无需 migration——kind 自由文本);agent_builder 注册 + media prompt 段 - 图片路径解析与 i2i 共用 tools/image_ref.py - 验证:scripts/smoke_look_at_image.py 真机 OCR 通过(实测 ¥0.0011) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
be813629b2
commit
0d69ae86e2
10
DESIGN.md
10
DESIGN.md
|
|
@ -526,11 +526,12 @@ create index on usage_events (model_profile, created_at);
|
|||
|
||||
> 实施细节(步骤清单 / 验收项)进 PROGRESS + git;此处只留缺口、选型与取舍。
|
||||
|
||||
### 8.1 图像理解 + Seedream i2i(2026-05-29 设计;i2i ✅ 2026-06-16 落地,look_at_image 仍待做)
|
||||
### 8.1 图像理解 + Seedream i2i(2026-05-29 设计;✅ 2026-06-16 i2i + look_at_image 双双落地)
|
||||
|
||||
**缺口**:DeepSeek V4 主模型纯文本无视觉;`seedream` 只 t2i;"基于已生成图二次修改" / "上传外部参考图让 agent 据此干活"两条路径未覆盖。
|
||||
|
||||
**选 E + C 组合**:`seedream` 加 `reference_images` 走 i2i(改已生成图,像素级)+ 新增 `look_at_image` 走豆包 Seed 1.6 vision 单图理解(读外部图,DeepSeek 自决何时调)。改动面=2 tool + 1 prompt 段 + 1 yaml 段,不动 loop / llm / capabilities / DB / 前端。
|
||||
**选 E + C 组合**:`seedream` 加 `reference_images` 走 i2i(改已生成图,像素级)+ 新增 `look_at_image` 走豆包视觉单图理解(读外部图,DeepSeek 自决何时调)。改动面=2 tool + 1 prompt 段 + 1 yaml 段,不动 loop / llm / capabilities / DB / 前端。
|
||||
> **模型选型更新(2026-06-16)**:设计时写的 Seed 1.6 vision 已过时,落地用 **Doubao Seed 2.0 Lite**(`doubao-seed-2-0-lite-260428`,2026-02 发布、05 升级为全模态理解 SOTA 细粒度感知)。Seed 2.0 全系文本模型已原生支持图片输入 → A 路(主模型换多模态)门槛降低,但主模型 DeepSeek V4 的 code/tool-calling 仍是核心,**维持 C 路(vision 当工具)不变**。token 计费(输入 ¥0.6 / 输出 ¥3.6 / Mtok),一次读图 < ¥0.01。
|
||||
- **不选 A(主模型换多模态)**:V4 的 code / tool calling 是主路径核心,换豆包当主 chat 降能力 + 要改 loop/memory 引 multimodal,工程 5× 且破坏架构。
|
||||
- **不选 B(后台 vision 路由)**:每条消息隐式 vision 描述 = 多烧 token + 1 跳延迟 + 失去 agentic 控制权 + debug 难。
|
||||
|
||||
|
|
@ -538,7 +539,10 @@ create index on usage_events (model_profile, created_at);
|
|||
|
||||
**风险 / 边界**:v1 只支持单张参考(multi-ref 角色定义靠 prompt,留 v2);base64 ARK 未承诺长期稳定(收紧则降级走 TOS 上传换 URL)。
|
||||
|
||||
**i2i 落地实况(2026-06-16,详 PROGRESS)**:`seedream` 加 `reference_images`(v1 单图,传 >1 报错);路径解析强制落 user_root 内防越界;前端 `chat.js` 补 paste 路径注入(把粘贴图路径作 `[用户上传的参考图]` 行进正文,修了"粘贴路径到不了模型"的既有缺口)。E 路(改图)完成;C 路(`look_at_image` 看图)仍待做。
|
||||
**落地实况(2026-06-16,详 PROGRESS)**:
|
||||
- **E 路(改图)**:`seedream` 加 `reference_images`(v1 单图,传 >1 报错);路径解析强制落 user_root 内防越界;前端 `chat.js` 补 paste 路径注入(把粘贴图路径作 `[用户上传的参考图]` 行进正文,修了"粘贴路径到不了模型"的既有缺口)。
|
||||
- **C 路(看图)**:`look_at_image` tool(`tools/look_at_image.py`)走 Seed 2.0 Lite `/chat/completions`,base64 单图 + 问题 → 文本解读;`doubao.yaml` 加 `vision:` 段;`usage.py` 加 `record_vision_usage`(kind="vision",token 计费);agent_builder 注册 + media prompt 段。图片解析与 i2i 共用 `tools/image_ref.py`。真机 smoke(`scripts/smoke_look_at_image.py`)OCR 验证通过。
|
||||
- 两路均不动 loop / llm / DB schema(`usage_events.kind` 自由文本,vision 无需 migration)。
|
||||
**升级到 A 的信号**:用户要"贴图同时说话模型直接读图回话",或多轮带图成高频 —— 当前假设"图是工具调用对象"而非"对话内容"。
|
||||
|
||||
### 8.2 Token 优化与上下文治理(2026-06-04,✅ 已落地,详 PROGRESS)
|
||||
|
|
|
|||
12
PROGRESS.md
12
PROGRESS.md
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> 配合 `DESIGN.md`。本文件只记 phase 状态、决策偏差、文件量、下一步。每条 1-2 句:做了啥 + 关键判断;细节查 `git log` / `git diff` / `DESIGN §7.9`。
|
||||
|
||||
最后更新:2026-06-16(seedream 加 i2i 改图 + 前端 paste 路径注入)
|
||||
最后更新:2026-06-16(图像理解 look_at_image + seedream i2i 改图,DESIGN §8.1 双路落地)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -21,12 +21,20 @@
|
|||
|
||||
## 已完成关键能力
|
||||
|
||||
### 2026-06-16 / look_at_image 图像理解(DESIGN §8.1 C 路落地)
|
||||
|
||||
- 需求:DeepSeek V4 主模型纯文本无视觉,挂 `look_at_image` 工具按需"借眼睛"读图(OCR / 描述 / 读图表),模型自决何时调。
|
||||
- **模型选型**:设计时的 Seed 1.6 vision 已过时(联网核实),改用 **Doubao Seed 2.0 Lite**(`doubao-seed-2-0-lite-260428`,全模态 SOTA 细粒度感知)。token 计费输入 ¥0.6 / 输出 ¥3.6 / Mtok,一次读图 < ¥0.01。否决「换主模型走 A 路」——DeepSeek 的 code/tool-calling 仍是核心,vision 当工具更稳。
|
||||
- 后端:新建 `tools/look_at_image.py`(`/chat/completions` OpenAI 兼容,base64 单图 + question → 文本解读,默认 question 覆盖描述+OCR+图表读数);`config/media/doubao.yaml` 加 `vision:` 段;`core/storage/usage.py` 加 `record_vision_usage`(kind="vision",按 token,单价 snapshot 进 units);`agent_builder.py` 注册(yaml 有 vision 段才挂)+ media prompt 段教「何时调 / 何时别调」。`usage_events.kind` 自由文本,vision **无需 migration**。
|
||||
- 重构:图片路径解析 + base64 抽到 `tools/image_ref.py`,seedream(i2i)与 look_at_image 共用(三形态路径 + user_root 边界 + 扩展名/大小校验)。
|
||||
- 验证:真机 smoke `scripts/smoke_look_at_image.py` 合成含已知文字图 → OCR 准确读出 + usage_events 落 kind=vision(实测 ¥0.0011)。bump 0.15.0 → 0.16.0。
|
||||
|
||||
### 2026-06-16 / seedream i2i 改图(DESIGN §8.1 E 路落地)+ 前端 paste 路径注入
|
||||
|
||||
- 需求:覆盖「基于已生成 / 上传的图做修改」(像素级),核心循环=文生图 → 用户"改成 X" → i2i 改那张(不重画)。base64 通路 probe 2026-05-29 已验,本次落 tool。
|
||||
- 后端 `tools/seedream.py`:加 `reference_images` 数组参数(**v1 单图**,传 >1 直接报错不静默截断)。路径解析走共享 `tools/image_ref.py`(与 look_at_image 同一套)——依次试 `working_dir/rel` → `user_root/rel` → 绝对,**强制结果落在 user_root 子树内**(防越界读任意文件),吃三种路径形态(`figures/x.png` / saved 形态 `<taskname>/figures/x.png` / 绝对);校验存在 + 图片扩展名(png/jpg/jpeg/webp/gif)+ ≤10MB;读 base64 → data URL → ARK body `image_urls`。**不传 reference_images = 文生图,行为 100% 不变(向后兼容)**。banner 加 `· mode=i2i` + `reference=` 行(前端正则兼容),meta.json 记 `mode` / `reference_images`(派生链可追溯)。
|
||||
- 前端 `web/static/js/chat.js`:`sendMessage` 发送时 `takePastedRels()` 收集 `chat-hint` 的 paste-chip 路径,作 `[用户上传的参考图] <rel>` 行注入正文 + 清 chip ——**修了既有缺口**(之前粘贴的图路径根本到不了模型)。这样"上传外部图 → 改图 / 看图"才能定位到文件。
|
||||
- 引导:`skills/imagegen/SKILL.md` 删旧「不接图像输入」结论 + 加「改图(i2i)」专段(最易踩错=该 i2i 却重新 t2i 丢构图);`agent_builder.py` 媒体 block 提 i2i + paste 注入约定;`SKILL_LIST.md` 同步。`look_at_image`(看图)仍待做。bump 0.14.0 → 0.15.0。
|
||||
- 引导:`skills/imagegen/SKILL.md` 删旧「不接图像输入」结论 + 加「改图(i2i)」专段(最易踩错=该 i2i 却重新 t2i 丢构图);`agent_builder.py` 媒体 block 提 i2i + paste 注入约定;`SKILL_LIST.md` 同步。bump 0.14.0 → 0.15.0(看图 look_at_image 同日落地见上一条)。
|
||||
|
||||
### 2026-06-16 / ask_user:回复里渲染可点击「方案确认」选项卡(Claude 式)
|
||||
|
||||
|
|
|
|||
|
|
@ -25,6 +25,23 @@ image:
|
|||
default_search: false # web search 额外加价 ~¥0.05/张;默认关
|
||||
request_timeout_s: 60 # 出图慢于此判超时
|
||||
|
||||
vision:
|
||||
# 图像理解(看图 / OCR / 读图表)。DeepSeek V4 主模型纯文本无视觉,挂 `look_at_image`
|
||||
# 工具走豆包 Seed 2.0 Lite(全模态理解)按需读图。OpenAI 兼容 /chat/completions,
|
||||
# 单图走 base64 data URL(同 seedream i2i,内网无需对象存储)。
|
||||
# 价格 last_updated: 2026-06-16,源 https://www.volcengine.com/docs/82379/1544106
|
||||
seed_2_lite:
|
||||
model_id: doubao-seed-2-0-lite-260428
|
||||
display_name: 豆包 Seed 2.0 Lite(视觉理解)
|
||||
endpoint: /chat/completions
|
||||
# token 计费(元/百万 tokens):输入 0.6 / 输出 3.6 / 缓存命中 0.12。
|
||||
# 一次读图 ≈ 图 1-2K + 输出几百 token → 成本 < ¥0.01,故不设每日配额(够便宜)。
|
||||
price_cny_per_mtoken_input: 0.6
|
||||
price_cny_per_mtoken_output: 3.6
|
||||
price_cny_per_mtoken_cache_hit: 0.12
|
||||
max_image_mb: 10 # 单图上限(超出 tool 侧直接报错,不发请求)
|
||||
request_timeout_s: 60 # 读图慢于此判超时
|
||||
|
||||
video:
|
||||
# fast 放第一个 → 默认 variant(成本敏感场景优先);开通了 Pro 的用户从顶栏下拉切。
|
||||
seedance_2_fast:
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
# zcbot 版本号单一事实源:web/app.py 的 FastAPI version、/healthz 返回、前端展示都引这里。
|
||||
# 改版本只动这一行。
|
||||
__version__ = "0.15.0"
|
||||
__version__ = "0.16.0"
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ from tools.materials_project import (
|
|||
MaterialsProjectGetStructureTool,
|
||||
MaterialsProjectSearchSummaryTool,
|
||||
)
|
||||
from tools.look_at_image import LookAtImageTool
|
||||
from tools.run_python import RunPythonTool
|
||||
from tools.seedance import SeedanceTool
|
||||
from tools.seedream import SeedreamTool
|
||||
|
|
@ -65,7 +66,10 @@ from core.bocha_client import BochaConfig
|
|||
# 也不该背这段红线。文案与 base 模板里其余工具表平级,放在 _build_system_prompt 里按需拼。
|
||||
_MEDIA_TOOLS_BLOCK = """\
|
||||
|
||||
## 媒体生成工具(seedream 图 / seedance 视频)
|
||||
## 媒体工具(seedream 图 / seedance 视频 / look_at_image 看图)
|
||||
- `look_at_image` —— 看图 / 读图(豆包 Seed 2.0 Lite 视觉)。**你(主模型)是纯文本看不见图,要"看"图就调它**:OCR 文字、描述画面、读图表/表格/示意图、识别物体。每次很便宜(按 token,通常 < ¥0.01)。
|
||||
- **何时调**:用户消息里出现 `[用户上传的参考图] <路径>` 且需要据图内容回答(问"这图里写了啥 / 是什么 / 表格数据多少");或要基于 task 内某张图(`figures/xxx.png`)的**实际内容**做事(不是改图,改图走 seedream)。传 `image=<路径>` + 可选 `question`。
|
||||
- **何时不调**:用户只是要改图(走 seedream i2i)/ 只要文件名不关心内容 / 图是你自己刚生成的且 prompt 已知(无需再读)。别对同一张图无意义反复看(每次都烧 token)。
|
||||
- `seedream` —— 豆包图像生成 / 改图。产物自动落 `<task_dir>/figures/`。每次 **¥0.22**(联网 `search=true` 加 ¥0.05)。
|
||||
- **文生图**(不传 `reference_images`):从零按 prompt 画。**改图 i2i**(传 `reference_images=["figures/xxx.png"]`):在已有图上做像素级修改。**用户对刚生成 / 上传的图说"改成 X / 换个颜色 / 去掉某处" → 必须走改图(reference_images 指那张图),绝不重新文生图**(重画 = 完全不同的图,丢原构图)。v1 改图仅支持单张参考。
|
||||
- **调用前必须先 `load_skill('imagegen')`** —— skill 里有「何时该用 / 该不该用 mermaid 替代 / 用户描述模糊度诊断 / 一次性追问范式 / prompt 装配 / 改图(i2i)范式 / 失败解药」全套引导。**不要拿用户原话直接当 prompt 调 tool** —— 容易烧 ¥0.22 在错的方向上。
|
||||
|
|
@ -615,6 +619,27 @@ def build_agent(
|
|||
)
|
||||
tools[seedance_tool.name] = seedance_tool
|
||||
|
||||
# 图像理解 tool(look_at_image / 豆包 Seed 2.0 Lite vision):仅当 yaml 有 vision 段才挂。
|
||||
# 无 variant 选择维度(读图不分档,固定第一个 variant),与 image/video 的"用户可切档"不同。
|
||||
vision_cfg = (ark_cfg.raw.get("vision") or {})
|
||||
vis_key, vis_variant = "", None
|
||||
for variant_key, variant_cfg in vision_cfg.items():
|
||||
if isinstance(variant_cfg, dict):
|
||||
vis_key, vis_variant = variant_key, variant_cfg
|
||||
break
|
||||
if vis_variant is not None:
|
||||
look_tool = LookAtImageTool(
|
||||
ark_cfg=ark_cfg,
|
||||
vision_variant_cfg=vis_variant,
|
||||
variant_key=vis_key,
|
||||
working_dir=working_dir_path,
|
||||
task_id=task_id,
|
||||
user_id=uid,
|
||||
base_dir=tool_base,
|
||||
user_root=ur_path,
|
||||
)
|
||||
tools[look_tool.name] = look_tool
|
||||
|
||||
# 博查联网搜索:仅当 BOCHA_API_KEY 设了才挂
|
||||
bocha_cfg = BochaConfig.load()
|
||||
if bocha_cfg is not None:
|
||||
|
|
|
|||
|
|
@ -249,6 +249,54 @@ def record_video_usage(
|
|||
return cost_cny
|
||||
|
||||
|
||||
def record_vision_usage(
|
||||
*,
|
||||
task_id: UUID,
|
||||
user_id: UUID,
|
||||
model_profile: str,
|
||||
prompt_tokens: int,
|
||||
completion_tokens: int,
|
||||
input_cny_per_mtoken: float,
|
||||
output_cny_per_mtoken: float,
|
||||
extra_units: Optional[Mapping[str, Any]] = None,
|
||||
) -> Decimal:
|
||||
"""记一次图像理解(看图 / OCR):写 usage_events(kind=vision)。
|
||||
|
||||
vision 是 token 计费(同 chat,不是 per-call),`cost_cny = in*in_price/1e6 + out*out_price/1e6`。
|
||||
tokens 由 caller 从 ARK /chat/completions 响应的 usage 字段取(没有则传 0,cost=0 不阻塞)。
|
||||
单价(CNY/Mtok)同步 snapshot 进 units jsonb,日后调价不影响历史对账。
|
||||
`model_profile` 形如 `"doubao.seed_2_lite"`(family.variant 风格,跟 chat/image 对齐)。
|
||||
|
||||
**失败任务不要走这里** —— ARK 失败不计费,失败的 tool 调用直接返 [Error] 不写 usage。
|
||||
"""
|
||||
inp = Decimal(str(input_cny_per_mtoken or 0))
|
||||
out = Decimal(str(output_cny_per_mtoken or 0))
|
||||
cost_cny = (
|
||||
Decimal(str(int(prompt_tokens))) * inp / Decimal("1000000")
|
||||
+ Decimal(str(int(completion_tokens))) * out / Decimal("1000000")
|
||||
).quantize(Decimal("0.000001"))
|
||||
units: dict[str, Any] = {
|
||||
"tokens_in": int(prompt_tokens),
|
||||
"tokens_out": int(completion_tokens),
|
||||
"input_cny_per_mtoken": float(input_cny_per_mtoken or 0),
|
||||
"output_cny_per_mtoken": float(output_cny_per_mtoken or 0),
|
||||
}
|
||||
if extra_units:
|
||||
units.update(extra_units)
|
||||
|
||||
with session_scope() as s:
|
||||
s.add(UsageEvent(
|
||||
user_id=user_id,
|
||||
task_id=task_id,
|
||||
message_id=None, # vision tool 在 tool execute 时调用,message 还未落库
|
||||
kind="vision",
|
||||
model_profile=model_profile,
|
||||
units=units,
|
||||
cost_cny=cost_cny,
|
||||
))
|
||||
return cost_cny
|
||||
|
||||
|
||||
def check_daily_quota(*, user_id: UUID, kind: str, limit: int) -> tuple[int, bool]:
|
||||
"""每账号每日 kind=image/video 调用配额检查。返回 (今日已用次数, 是否超额)。
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,130 @@
|
|||
"""Smoke: look_at_image(豆包 Seed 2.0 Lite 视觉)端到端走通 + OCR 验证。
|
||||
|
||||
跑法: .venv/Scripts/python.exe scripts/smoke_look_at_image.py
|
||||
依赖 .env 里 ARK_API_KEY / ZCBOT_DB_URL。**会真的调豆包 vision API,产生 < ¥0.01 费用**。
|
||||
|
||||
校验:
|
||||
1. ArkConfig.load() + yaml vision 段存在
|
||||
2. 合成一张含已知文字 "ZCBOT-VISION-8848" 的 PNG → LookAtImageTool.execute 能 OCR 出该串
|
||||
3. 返回串首行 banner 含 model/tokens/cost
|
||||
4. usage_events 多出一行 kind="vision",units 含 tokens_in/out + 单价 snapshot
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
# Windows 控制台默认 GBK,打印 ¥ / 中文结果会崩 → 强制 stdout UTF-8(只影响本脚本打印)
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 读 .env
|
||||
env_file = ROOT / ".env"
|
||||
if env_file.exists():
|
||||
for line in env_file.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
k, _, v = line.partition("=")
|
||||
os.environ.setdefault(k.strip(), v.strip())
|
||||
|
||||
from PIL import Image, ImageDraw
|
||||
from sqlalchemy import text
|
||||
|
||||
from core.ark_client import ArkConfig
|
||||
from core.storage import session_scope
|
||||
from core.storage.models import Task, User
|
||||
from tools.look_at_image import LookAtImageTool
|
||||
|
||||
MAGIC = "ZCBOT-VISION-8848"
|
||||
|
||||
|
||||
def make_text_png(dest: Path) -> None:
|
||||
"""白底大号黑字 PNG(放大 4x 让默认位图字体也清晰可 OCR)。"""
|
||||
small = Image.new("RGB", (260, 60), (255, 255, 255))
|
||||
d = ImageDraw.Draw(small)
|
||||
d.text((10, 22), MAGIC, fill=(0, 0, 0))
|
||||
big = small.resize((260 * 4, 60 * 4), Image.NEAREST)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
big.save(dest)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
cfg = ArkConfig.load()
|
||||
if cfg is None:
|
||||
print("[SKIP] ARK_API_KEY 未设(或 doubao.yaml 缺失),无法测真接口")
|
||||
return 0
|
||||
vision_cfg = (cfg.raw.get("vision") or {})
|
||||
if not vision_cfg:
|
||||
print("[SKIP] doubao.yaml 无 vision 段")
|
||||
return 0
|
||||
variant_key, variant_cfg = next(iter(vision_cfg.items()))
|
||||
print(f"[setup] variant={variant_key} model={variant_cfg.get('model_id')} "
|
||||
f"price_in={variant_cfg.get('price_cny_per_mtoken_input')} "
|
||||
f"price_out={variant_cfg.get('price_cny_per_mtoken_output')}")
|
||||
|
||||
uid = uuid.uuid4()
|
||||
tid = uuid.uuid4()
|
||||
ws_user = ROOT / "workspace" / "users" / str(uid)
|
||||
wd = ws_user / "smoke_vision"
|
||||
img = wd / "figures" / "magic.png"
|
||||
make_text_png(img)
|
||||
print(f"[setup] 合成测试图 {img.name}(含文字 {MAGIC!r})")
|
||||
|
||||
with session_scope() as s: # User 先单独落库,再建 Task(FK 顺序保险)
|
||||
s.add(User(user_id=uid))
|
||||
with session_scope() as s:
|
||||
s.add(Task(task_id=tid, user_id=uid, name="smoke_vision", working_dir=str(wd)))
|
||||
|
||||
tool = LookAtImageTool(
|
||||
ark_cfg=cfg,
|
||||
vision_variant_cfg=variant_cfg,
|
||||
variant_key=variant_key,
|
||||
working_dir=wd,
|
||||
task_id=tid,
|
||||
user_id=uid,
|
||||
base_dir=wd,
|
||||
user_root=ws_user,
|
||||
)
|
||||
|
||||
print("[call] question='把图中的文字逐字 OCR 出来'")
|
||||
result = tool.execute(image="figures/magic.png", question="把图中的文字逐字 OCR 出来。")
|
||||
print(f"[tool result]\n{result}\n")
|
||||
if result.startswith("[Error]"):
|
||||
print("[FAIL] tool 返回错误")
|
||||
return 2
|
||||
|
||||
# OCR 命中(容忍模型加空格/大小写差异,去掉分隔比对)
|
||||
norm = result.replace(" ", "").replace("\n", "").upper()
|
||||
if MAGIC.replace("-", "") in norm.replace("-", ""):
|
||||
print(f"[OK] OCR 命中魔术串 {MAGIC}")
|
||||
else:
|
||||
print(f"[WARN] 未在结果里精确匹配 {MAGIC} —— 人工核对上面 result(模型可能换了排版)")
|
||||
|
||||
with session_scope() as s:
|
||||
rows = s.execute(text(
|
||||
"SELECT kind, model_profile, units, cost_cny FROM usage_events "
|
||||
"WHERE task_id = :tid"
|
||||
), {"tid": str(tid)}).all()
|
||||
assert len(rows) == 1, f"usage_events 行数应 1,实际 {len(rows)}"
|
||||
row = rows[0]
|
||||
assert row.kind == "vision", f"kind 应 vision,实际 {row.kind}"
|
||||
assert row.model_profile == f"doubao.{variant_key}", f"model_profile={row.model_profile}"
|
||||
for k in ("tokens_in", "tokens_out", "input_cny_per_mtoken", "output_cny_per_mtoken"):
|
||||
assert k in row.units, f"units 缺 {k}"
|
||||
print(f"[OK] usage_events: kind={row.kind} model={row.model_profile} "
|
||||
f"cost_cny={row.cost_cny} units={row.units}")
|
||||
|
||||
print("\n[PASS] smoke_look_at_image 全部通过")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,180 @@
|
|||
"""look_at_image: 给 DeepSeek V4 主模型(纯文本)借一双眼睛。
|
||||
|
||||
主模型无视觉,这个 tool 走豆包 Seed 2.0 Lite(全模态理解)读单图 —— OCR / 描述画面 /
|
||||
读图表表格 / 识别物体。模型自决何时调(用户贴了图问"这写的啥" / "图里是什么")。
|
||||
|
||||
模型 ID + 单价全在 `config/media/doubao.yaml` 的 vision 段,本 tool 只装配。
|
||||
计费:token 计费(同 chat),成功后写一行 usage_events(kind="vision")。
|
||||
图片路径解析 + base64 复用 tools/image_ref(与 seedream i2i 同一套边界/校验)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
from uuid import UUID
|
||||
|
||||
from core.ark_client import ArkClient, ArkConfig, ArkError
|
||||
from core.storage.usage import record_vision_usage
|
||||
|
||||
from .base import Tool, compact_tool_output
|
||||
from .image_ref import load_image_as_data_url
|
||||
|
||||
# 不带 question 时的默认提问:全覆盖(描述 + OCR + 图表读数),让模型一次把图里能用的信息都吐出来
|
||||
_DEFAULT_QUESTION = (
|
||||
"请仔细看这张图并回答:"
|
||||
"1) 完整描述画面内容(主体/场景/关键元素);"
|
||||
"2) 如果图中有任何文字,逐字准确 OCR 出来,尽量保留原排版与换行;"
|
||||
"3) 如果是图表/表格/示意图,把其中的数据、坐标轴、图例、结构关系读出来。"
|
||||
)
|
||||
|
||||
|
||||
class LookAtImageTool(Tool):
|
||||
name = "look_at_image"
|
||||
description = (
|
||||
"Read/understand an image using Doubao Seed 2.0 Lite vision (the main model is text-only). "
|
||||
"Use to OCR text, describe a picture, read charts/tables/diagrams, or identify objects in an "
|
||||
"image the user uploaded (look for a `[用户上传的参考图] <path>` line in their message) or that "
|
||||
"was generated/saved in the task. Pass the image path; optionally a specific question "
|
||||
"(default: describe + OCR everything). Cheap (token-based, usually < ¥0.01). "
|
||||
"Returns the model's textual reading of the image."
|
||||
)
|
||||
parameters = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"image": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"要看的图片相对路径(task_dir 内,如 'figures/xxx.png',或用户消息里 "
|
||||
"`[用户上传的参考图]` 行给的路径,或某工具上次返回的 saved 路径)。"
|
||||
),
|
||||
},
|
||||
"question": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"想从图里知道什么(可选)。如「这张图里的表格数据是多少」「图中仪表读数」"
|
||||
"「把这页文字 OCR 出来」。不传则默认全面描述 + OCR 全部文字。"
|
||||
),
|
||||
},
|
||||
},
|
||||
"required": ["image"],
|
||||
}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
ark_cfg: ArkConfig,
|
||||
vision_variant_cfg: dict,
|
||||
variant_key: str,
|
||||
working_dir: Path,
|
||||
task_id: UUID,
|
||||
user_id: UUID,
|
||||
base_dir: Optional[Path] = None,
|
||||
user_root: Optional[Path] = None,
|
||||
) -> None:
|
||||
super().__init__(base_dir, user_root=user_root)
|
||||
self.ark_cfg = ark_cfg
|
||||
self.cfg = vision_variant_cfg
|
||||
self.variant_key = variant_key # 'seed_2_lite' → usage_events.model_profile = "doubao.seed_2_lite"
|
||||
self.working_dir = Path(working_dir)
|
||||
self.task_id = task_id
|
||||
self.user_id = user_id
|
||||
|
||||
def execute(self, image: str, question: Optional[str] = None) -> str:
|
||||
if not (image or "").strip():
|
||||
return "[Error] image(图片路径)不能为空"
|
||||
|
||||
cfg = self.cfg
|
||||
max_bytes = int(float(cfg.get("max_image_mb", 10)) * 1024 * 1024)
|
||||
data_url, disp, err = load_image_as_data_url(
|
||||
image.strip(),
|
||||
working_dir=self.working_dir,
|
||||
user_root=self.user_root,
|
||||
display_fn=self._display,
|
||||
max_bytes=max_bytes,
|
||||
)
|
||||
if err:
|
||||
return err
|
||||
|
||||
q = (question or "").strip() or _DEFAULT_QUESTION
|
||||
model_id = cfg["model_id"]
|
||||
timeout_s = float(cfg.get("request_timeout_s", 60))
|
||||
endpoint = cfg.get("endpoint", "/chat/completions")
|
||||
|
||||
body: dict[str, Any] = {
|
||||
"model": model_id,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": q},
|
||||
{"type": "image_url", "image_url": {"url": data_url}},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
try:
|
||||
with ArkClient(self.ark_cfg, timeout_s=timeout_s) as client:
|
||||
resp = client.post_json(endpoint, body, timeout_s=timeout_s)
|
||||
except ArkError as e:
|
||||
return f"[Error] look_at_image API: {e}"
|
||||
|
||||
answer = self._extract_answer(resp)
|
||||
if not answer:
|
||||
return (
|
||||
"[Error] vision 响应缺内容(模型未返回文本)。"
|
||||
"可能图片格式异常或模型暂不可用,稍后重试。"
|
||||
)
|
||||
|
||||
usage = resp.get("usage") or {}
|
||||
tin = int(usage.get("prompt_tokens", 0) or 0)
|
||||
tout = int(usage.get("completion_tokens", 0) or 0)
|
||||
|
||||
# 记账;失败不阻塞 tool 返回(沿用 seedream 兜底)
|
||||
cost_cny = 0.0
|
||||
try:
|
||||
cost = record_vision_usage(
|
||||
task_id=self.task_id,
|
||||
user_id=self.user_id,
|
||||
model_profile=f"doubao.{self.variant_key}",
|
||||
prompt_tokens=tin,
|
||||
completion_tokens=tout,
|
||||
input_cny_per_mtoken=float(cfg.get("price_cny_per_mtoken_input", 0)),
|
||||
output_cny_per_mtoken=float(cfg.get("price_cny_per_mtoken_output", 0)),
|
||||
extra_units={"image": disp},
|
||||
)
|
||||
cost_cny = float(cost)
|
||||
except Exception as e:
|
||||
print(f"[look_at_image] record_vision_usage failed: {type(e).__name__}: {e}", flush=True)
|
||||
|
||||
# 第一行 banner(key=value · 分隔,与 seedream/seedance 同协议,便于前端/对账)
|
||||
banner = (
|
||||
f"[look_at_image] model={model_id} · image={disp} · "
|
||||
f"tokens={tin}+{tout} · cost=¥{cost_cny:.4f}"
|
||||
)
|
||||
# 图片解读正文可能很长(整页 OCR),压一下防爆上下文(保头尾)
|
||||
return f"{banner}\n\n{compact_tool_output(answer)}"
|
||||
|
||||
@staticmethod
|
||||
def _extract_answer(resp: dict) -> str:
|
||||
"""OpenAI 兼容 chat 响应取文本: choices[0].message.content。
|
||||
|
||||
content 可能是 str,也可能是 list[{type:text,text:...}](多模态返回形态),都兜住。
|
||||
"""
|
||||
choices = resp.get("choices")
|
||||
if not (isinstance(choices, list) and choices):
|
||||
return ""
|
||||
msg = choices[0].get("message") if isinstance(choices[0], dict) else None
|
||||
if not isinstance(msg, dict):
|
||||
return ""
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str):
|
||||
return content.strip()
|
||||
if isinstance(content, list):
|
||||
parts = [
|
||||
c.get("text", "")
|
||||
for c in content
|
||||
if isinstance(c, dict) and c.get("type") == "text"
|
||||
]
|
||||
return "\n".join(p for p in parts if p).strip()
|
||||
return ""
|
||||
Loading…
Reference in New Issue