feat(ops): 工具失败聚集巡检,同签名报错自动冒头(bump 0.58.11)
反 9dcae061 结构性缺口(mmdc 挂 90 天 0 成功无人知,靠人工扫 DB 才发现): - core/toolfail.py:扫近 7 天 role=tool 错误([Error/timeout/exit!=0]), 按 工具名+归一化签名(路径/数字/uuid 抹平) 聚合,>=5 次且跨 >=2 task 判聚集 - web/app.py _toolfail_scanner:每天跑(仿 _disk_scanner),聚集发 ZCBOT_DEVELOPER_EMAIL(复用 SMTP_*,未配则日志);签名进程内去重 - GET /v1/admin/tool-failures + admin 页"工具失败"表(低阈值看全量) 生产 DB 冒烟:一次扫出 8 类真聚集(沙箱缺 file 命令 x13 跨 13 task、 web_fetch 断网 x7、mmdc 超时批次 x6 等)。无新表无 migration。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
73d07c9560
commit
62bfb07312
|
|
@ -23,6 +23,7 @@
|
|||
|
||||
### 2026-07
|
||||
|
||||
- **07-13 / 0.58.11**:**工具失败聚集巡检**(反 9dcae061 "挂 90 天无人知"的结构性缺口):新模块 `core/toolfail.py`——扫近 7 天 messages 里 role=tool 的错误结果(`[Error` 开头 / `command timed out` / 尾部 `[exit 非0]`),按 (工具名+归一化签名:路径/数字/uuid 抹平) 聚合,同签名 ≥5 次且跨 ≥2 task 判聚集(单 task 内试错自愈不触发)。三个出口:① web/app.py `_toolfail_scanner` 周期任务(仿 _disk_scanner,每天跑,签名进程内去重)→ 发 `ZCBOT_DEVELOPER_EMAIL`(复用 SMTP_*,未配则日志);② `GET /v1/admin/tool-failures`;③ admin 页新"工具失败"表(低阈值 3 次/1 task 看全量,悬浮看样例)。生产 DB 冒烟:一次扫出 21 类真聚集(沙箱缺 `file` 命令 x13 跨 13 task、web_fetch Network unreachable x7、mmdc 超时批次等)。env:`ZCBOT_DEVELOPER_EMAIL` / `ZCBOT_TOOLFAIL_SCAN_INTERVAL`(RUN.md 环境段已记)。无新表无 migration;重启后同签名重推一次(聚集还在=问题还在,可接受)。
|
||||
- **07-12 / 0.58.10**:**mermaid 渲染真·终案:init.sh loopback DROP 掐死 puppeteer↔chromium**(task 9dcae061 四轮收官):0.58.8 部署后(镜像已 chromium .100、全新容器)复测仍恒定 ~2m15s 超时 + `[object Object]` + user CPU 近 0 → 复刻 pool 全款硬限制容器 bisect 定位:**init.sh `127.0.0.0/8 DROP` 挡掉 127.0.0.1 上的 DevTools WebSocket**(puppeteer 连 chromium 的控制通道),SYN DROP → TCP 重试打满 ~127s;容器内放行 `-o lo` 后小图 3.9s、真实大图原参数(`-w 2800 -s 2`)5.2s 出图——"1 vCPU 渲大图慢"亦为误判。**90 天 67 次 timeout 真凶是它**(chromium 坏版本只解释 26 次 launch fail);之前所有 probe 因 `--entrypoint bash` 绕过 init.sh 无 iptables,全绿误判三轮。修复:init.sh `apply_blocklist` 首条 `iptables -A OUTPUT -o lo -j ACCEPT`(v6 同款),原 127.0.0.11:53 DNS 豁免收编,`127.0.0.0/8`/`::1` DROP 降为 belt-and-suspenders 并修正"防回头打宿主"错误注释(容器 127.0.0.1 是自己 netns 的 lo,到不了宿主)。DESIGN §7.5 #1 红线段同步语义修正;RUN.md 故障兜底加终案条目(含 probe 大坑);probe_mermaid.sh 头部加 entrypoint-bypass 警示;CHANGELOG 面向用户补记。生效条件:镜像重建(init.sh 是 COPY 层,秒过)+ 回收存量容器。
|
||||
- **07-12 / 0.58.9**:**probe_mermaid.sh 用法注释修正**:临时容器跑法(case B)的 `docker run` 少了 `-i` —— 不带则容器 stdin 为空,bash 读 `/dev/stdin` 拿到空脚本**静默退出零输出**(服务器验证 0.58.8 时实测踩中,误以为探针跑完无异常)。仅注释修正,脚本本体不变。
|
||||
- **07-11 / 0.58.8**:**mermaid 渲染必崩最终定案 + 根治**(9dcae061 三轮排查收官):真凶 = 7-08 镜像重建时 apt 烤进的 chromium 点版本 `150.0.7871.46` 自带启动即崩 bug(int3 自杀、零日志、strace 无失败 syscall;上游 `.100` 已修,容器内 `--only-upgrade chromium` 实测复活出图)。排查中被逐一洗清的:沙箱 hardening 六件套、seccomp、AppArmor(Ubuntu 24.04 userns 限制在开但无关)、overcommit/ulimit/max_map_count、QEMU EPYC vCPU、镜像环境(纯净 trixie 对照 rc=0 反转定位)。修复两件套:Dockerfile ① `ARG CHROMIUM_REFRESH=<日期>` cache-bust 旋钮(chromium 出问题改日期重建,pip 大层不动);② **build 末尾渲染 canary**(真渲一张中文最小图,chromium/字体链坏则 build 当场失败——有它 7-08 就不会把坏版本带上线)。RUN.md 故障兜底条目改写为最终版(两根因 + 判别特征 + "别再猜 hardening")、CHANGELOG 面向用户记修复。部署:服务器重建镜像(新 ARG 自然打穿 chromium 层 cache)+ 回收存量容器。
|
||||
|
|
|
|||
5
RUN.md
5
RUN.md
|
|
@ -69,6 +69,11 @@
|
|||
# SMTP_PASSWORD=<授权码/应用专用密码,非登录密码>
|
||||
# SMTP_FROM=you@qq.com # 可选,默认取 SMTP_USER
|
||||
# SMTP_FROM_NAME=总院科研辅助智能体 # 可选,发件人显示名,默"总院科研辅助智能体"(不暴露内部代号)
|
||||
# 工具失败聚集巡检(0.58.11,反 9dcae061"挂 90 天无人知"):web 进程每天扫近 7 天
|
||||
# tool 错误消息,同签名 >=5 次且跨 >=2 task 判聚集 → 发下面邮箱(复用上面 SMTP_*;
|
||||
# 未配邮箱或 SMTP 则只打日志 [toolfail] 行)。admin 页"工具失败"表可随时看低阈值全量。
|
||||
# ZCBOT_DEVELOPER_EMAIL=dev@example.com
|
||||
# ZCBOT_TOOLFAIL_SCAN_INTERVAL=86400 # 秒,默 86400;<=0 关掉巡检
|
||||
# broker 外置(DESIGN §7.0,蓝绿双实例跨进程 SSE/cancel):可选。设了 → event/cancel
|
||||
# 走 Redis pub/sub(蓝绿两实例读同一份 .env,天然同 URL);不设 → 进程内 broker
|
||||
# (dev / 单实例,零依赖不用起 redis)。设了但连不上 → 启动 fail-fast(报错见故障兜底)。
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
# zcbot 版本号单一事实源:web/app.py 的 FastAPI version、/healthz 返回、前端展示都引这里。
|
||||
# 改版本只动这一行。
|
||||
__version__ = "0.58.10"
|
||||
__version__ = "0.58.11"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,156 @@
|
|||
"""工具失败聚集巡检:同签名的工具报错反复出现 → 主动冒头。
|
||||
|
||||
背景(2026-07,task 9dcae061 终案的结构性教训):mermaid 渲染在生产挂了 90 天
|
||||
0 成功(67 次超时 + 26 次 launch fail、烧掉数十万 token),没有任何机制发现,
|
||||
靠人工扫 DB 才挖出来。本模块把「失败聚集」变成信号:扫 messages 里 role=tool
|
||||
的错误结果,按 (工具名 + 归一化错误签名) 聚合,超阈值即算聚集。
|
||||
|
||||
纯只读查询、无新表无状态;告警通道由调用方决定(web/app.py 的巡检 loop 发
|
||||
开发者邮箱,admin API 直接返给前端表格)。
|
||||
|
||||
失败判定(tool content 的三类标记,形态见 executor_docker/_host):
|
||||
- `[Error` 开头 —— 执行器/工具层报错([Error]、[Error executing ...])
|
||||
- `command timed out` —— shell/run_python 超时
|
||||
- 尾部 `[exit N]` 且 N != 0 —— shell 非零退出
|
||||
`[exit 0]` 但语义失败(如 "No mermaid charts found")不判 —— 无通用判据,不猜。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from core.storage import session_scope
|
||||
|
||||
# 签名归一:同一类错误在不同 task/参数下的差异(路径/数字/uuid/十六进制)抹平,
|
||||
# 让 "figures/a.png doesn't exist" 和 "figures/b.png doesn't exist" 聚成一条。
|
||||
_RE_UUID = re.compile(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}")
|
||||
_RE_HEX = re.compile(r"0x[0-9a-fA-F]+")
|
||||
_RE_PATH = re.compile(r"(?:[A-Za-z]:)?(?:[/\\][\w.\-一-鿿*]+){2,}")
|
||||
_RE_NUM = re.compile(r"\d+")
|
||||
_RE_WS = re.compile(r"\s+")
|
||||
|
||||
_EXIT_TAIL = re.compile(r"\[exit (\d+)\]\s*$")
|
||||
_STREAM_MARKS = ("[stdout]", "[stderr]")
|
||||
|
||||
|
||||
def _normalize(s: str) -> str:
|
||||
s = _RE_UUID.sub("<id>", s)
|
||||
s = _RE_HEX.sub("<hex>", s)
|
||||
s = _RE_PATH.sub("<path>", s)
|
||||
s = _RE_NUM.sub("N", s)
|
||||
s = _RE_WS.sub(" ", s).strip()
|
||||
return s[:120]
|
||||
|
||||
|
||||
def _classify(content: str) -> Optional[Tuple[str, str]]:
|
||||
"""返回 (kind, 原始签名行) 或 None(不算失败)。"""
|
||||
head = content.lstrip()
|
||||
# 超时判定在前:超时结果形如 "[Error] command timed out after 30s",
|
||||
# 让它归 timeout 而不是被 [Error 前缀截走(kind 对排查方向有指示意义)
|
||||
if "command timed out" in content:
|
||||
return "timeout", "command timed out"
|
||||
if head.startswith("[Error"):
|
||||
return "error", head.splitlines()[0]
|
||||
m = _EXIT_TAIL.search(content)
|
||||
if m and m.group(1) != "0":
|
||||
# 签名取 [exit N] 前最后一行有实际内容的输出(通常是真正的报错行)
|
||||
lines = [
|
||||
ln.strip() for ln in content.splitlines()[:-1]
|
||||
if ln.strip() and ln.strip() not in _STREAM_MARKS
|
||||
]
|
||||
return "exit", (lines[-1] if lines else f"exit {m.group(1)}")
|
||||
return None
|
||||
|
||||
|
||||
def scan_tool_failures(
|
||||
days: float = 7,
|
||||
min_count: int = 5,
|
||||
min_tasks: int = 2,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""扫近 `days` 天的 tool 错误消息,返回超阈值的聚集(count 降序)。
|
||||
|
||||
阈值语义:同签名 >= min_count 次 且 跨 >= min_tasks 个 task —— 单 task 内
|
||||
模型试错几次就自愈的正常噪音不触发;跨 task 复现的才是平台性问题。
|
||||
同步阻塞(DB 查询),asyncio 调用方放 to_thread/executor。
|
||||
"""
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=days)
|
||||
rows = []
|
||||
with session_scope() as s:
|
||||
rows = s.execute(
|
||||
text(
|
||||
"select m.task_id, t.user_id, m.created_at, "
|
||||
" m.payload->>'name' as tool_name, "
|
||||
" m.payload->>'content' as content "
|
||||
"from messages m join tasks t on t.task_id = m.task_id "
|
||||
"where m.created_at >= :cutoff "
|
||||
" and m.payload->>'role' = 'tool' "
|
||||
" and (m.payload->>'content' like '[Error%' "
|
||||
" or m.payload->>'content' like '%command timed out%' "
|
||||
" or m.payload->>'content' like '%[exit %')"
|
||||
),
|
||||
{"cutoff": cutoff},
|
||||
).fetchall()
|
||||
|
||||
agg: Dict[Tuple[str, str], Dict[str, Any]] = {}
|
||||
for task_id, user_id, created_at, tool_name, content in rows:
|
||||
if not content:
|
||||
continue
|
||||
hit = _classify(content)
|
||||
if hit is None:
|
||||
continue
|
||||
kind, sig_line = hit
|
||||
key = (tool_name or "?", _normalize(sig_line))
|
||||
c = agg.get(key)
|
||||
if c is None:
|
||||
c = agg[key] = {
|
||||
"tool": key[0],
|
||||
"signature": key[1],
|
||||
"kind": kind,
|
||||
"count": 0,
|
||||
"tasks": set(),
|
||||
"users": set(),
|
||||
"last_at": created_at,
|
||||
"sample": content[:300],
|
||||
}
|
||||
c["count"] += 1
|
||||
c["tasks"].add(task_id)
|
||||
c["users"].add(user_id)
|
||||
if created_at > c["last_at"]:
|
||||
c["last_at"] = created_at
|
||||
c["sample"] = content[:300]
|
||||
|
||||
out = []
|
||||
for c in agg.values():
|
||||
if c["count"] < min_count or len(c["tasks"]) < min_tasks:
|
||||
continue
|
||||
out.append({
|
||||
"tool": c["tool"],
|
||||
"signature": c["signature"],
|
||||
"kind": c["kind"],
|
||||
"count": c["count"],
|
||||
"task_count": len(c["tasks"]),
|
||||
"user_count": len(c["users"]),
|
||||
"last_at": c["last_at"].isoformat(),
|
||||
"sample": c["sample"],
|
||||
})
|
||||
out.sort(key=lambda x: x["count"], reverse=True)
|
||||
return out
|
||||
|
||||
|
||||
def format_alert(clusters: List[Dict[str, Any]], days: float) -> str:
|
||||
"""聚集列表 → 告警邮件正文(纯文本)。"""
|
||||
lines = [f"近 {days:g} 天内检出 {len(clusters)} 类工具失败聚集:", ""]
|
||||
for c in clusters:
|
||||
lines.append(
|
||||
f"- [{c['tool']}/{c['kind']}] x{c['count']}"
|
||||
f"(task {c['task_count']} 个 / 用户 {c['user_count']} 人,"
|
||||
f"最近 {c['last_at']})"
|
||||
)
|
||||
lines.append(f" 签名: {c['signature']}")
|
||||
lines.append(f" 样例: {c['sample'][:200]}")
|
||||
lines.append("")
|
||||
lines.append("排查入口:RUN.md 故障兜底表;历史案例:mermaid loopback DROP(0.58.10)。")
|
||||
return "\n".join(lines)
|
||||
16
web/admin.py
16
web/admin.py
|
|
@ -388,6 +388,22 @@ def register_admin_routes(app: FastAPI, require_admin) -> None:
|
|||
with session_scope() as s:
|
||||
return _storage_page(s, page, page_size)
|
||||
|
||||
@app.get("/v1/admin/tool-failures", tags=["admin"])
|
||||
def admin_tool_failures(
|
||||
days: int = 7, min_count: int = 3, min_tasks: int = 1,
|
||||
user_id: UUID = Depends(require_admin),
|
||||
):
|
||||
"""工具失败聚集(core/toolfail 同款扫描,页面用低阈值看全量;
|
||||
巡检推送走 5 次/2 task 的高阈值)。admin-only。"""
|
||||
from core.toolfail import scan_tool_failures
|
||||
days = min(90, max(1, days))
|
||||
return {
|
||||
"days": days,
|
||||
"clusters": scan_tool_failures(
|
||||
days=days, min_count=max(1, min_count), min_tasks=max(1, min_tasks)
|
||||
),
|
||||
}
|
||||
|
||||
@app.get("/v1/admin/tiers", tags=["admin"])
|
||||
def admin_tiers(user_id: UUID = Depends(require_admin)):
|
||||
"""模型档位定义 + 全模型目录。admin-only。
|
||||
|
|
|
|||
50
web/app.py
50
web/app.py
|
|
@ -954,6 +954,50 @@ def create_app() -> FastAPI:
|
|||
|
||||
stats_logger_task = asyncio.create_task(_stats_logger(), name="stats-logger")
|
||||
|
||||
# ── 工具失败聚集巡检(0.58.11)── 反 9dcae061 教训(mmdc 在生产挂 90 天
|
||||
# 0 成功无人知,靠人工扫 DB 才发现):每天扫近 7 天 tool 错误消息,同签名
|
||||
# >=5 次且跨 >=2 task 判聚集,发 ZCBOT_DEVELOPER_EMAIL(复用 SMTP_*;未配
|
||||
# 邮箱/SMTP 则只打日志)。签名进程内去重 —— 重启/蓝绿切换后同签名会再推
|
||||
# 一次(聚集还在=问题还在,重提醒可接受)。ZCBOT_TOOLFAIL_SCAN_INTERVAL
|
||||
# 秒,默 86400,<=0 整体关掉。
|
||||
dev_email = os.getenv("ZCBOT_DEVELOPER_EMAIL", "").strip()
|
||||
toolfail_interval = int(os.getenv("ZCBOT_TOOLFAIL_SCAN_INTERVAL", "").strip() or 86400)
|
||||
|
||||
async def _toolfail_scanner() -> None:
|
||||
from core.toolfail import format_alert, scan_tool_failures
|
||||
from tools.send_email import send_email_smtp, smtp_configured
|
||||
loop = asyncio.get_running_loop()
|
||||
alerted: set = set()
|
||||
while True:
|
||||
try:
|
||||
clusters = await loop.run_in_executor(None, scan_tool_failures)
|
||||
fresh = [c for c in clusters
|
||||
if (c["tool"], c["signature"]) not in alerted]
|
||||
if fresh:
|
||||
alerted.update((c["tool"], c["signature"]) for c in fresh)
|
||||
for c in fresh:
|
||||
print(f"[toolfail] {c['tool']}/{c['kind']} x{c['count']} "
|
||||
f"tasks={c['task_count']}: {c['signature'][:80]}")
|
||||
if dev_email and smtp_configured():
|
||||
await loop.run_in_executor(
|
||||
None, send_email_smtp, dev_email,
|
||||
f"[zcbot] 工具失败聚集 {len(fresh)} 类",
|
||||
format_alert(fresh, 7),
|
||||
)
|
||||
else:
|
||||
print("[toolfail] ZCBOT_DEVELOPER_EMAIL/SMTP 未配,仅日志")
|
||||
await asyncio.sleep(toolfail_interval)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"[toolfail] error: {type(e).__name__}: {e}")
|
||||
await asyncio.sleep(toolfail_interval)
|
||||
|
||||
toolfail_task = (
|
||||
asyncio.create_task(_toolfail_scanner(), name="toolfail-scanner")
|
||||
if toolfail_interval > 0 else None
|
||||
)
|
||||
|
||||
# ── 定时任务守护循环(§8.5)── 仿 _disk_scanner 的 plain-asyncio 范式,不引
|
||||
# APScheduler/Celery。每 ~10s 认领到点 job(claim+advance next_run 防重复触发),
|
||||
# 复用 _run_agent_bg 起 run,跑完确定性兜底投递 + 回写 last_*。间隔只决定最坏延迟
|
||||
|
|
@ -1333,6 +1377,12 @@ def create_app() -> FastAPI:
|
|||
await stats_logger_task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
if toolfail_task is not None:
|
||||
toolfail_task.cancel()
|
||||
try:
|
||||
await toolfail_task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
if scheduler_task is not None:
|
||||
scheduler_task.cancel()
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ const SORT_OPTS = [["cost", "按成本"], ["tokens", "按用量"]];
|
|||
const SECTIONS = [
|
||||
["s-runtime", "运行态"], ["s-tasks", "任务"], ["s-usage", "用户与用量"],
|
||||
["s-models", "按模型"], ["s-users", "各用户用量"], ["s-storage", "存储"],
|
||||
["s-toolfail", "工具失败"],
|
||||
];
|
||||
|
||||
const $ = (id) => document.getElementById(id);
|
||||
|
|
@ -287,6 +288,26 @@ function renderStorage(d) {
|
|||
wirePager("st", page, maxPage, (p) => loadStorage(p));
|
||||
}
|
||||
|
||||
// 工具失败聚集(近7天,低阈值看全量;巡检邮件走 core/toolfail 高阈值)。
|
||||
// 签名已在后端归一(路径/数字抹平),悬浮 title 给最近一次的原始样例。
|
||||
function renderToolFailures(d) {
|
||||
const rows = d.clusters || [];
|
||||
const body = rows.map(c => `<tr>`
|
||||
+ `<td>${escapeHtml(c.tool)}</td>`
|
||||
+ `<td>${escapeHtml(c.kind)}</td>`
|
||||
+ `<td class="email" title="${escapeHtml(c.sample || "")}">${escapeHtml(c.signature)}</td>`
|
||||
+ `<td class="num">${c.count}</td>`
|
||||
+ `<td class="num">${c.task_count}</td>`
|
||||
+ `<td class="num">${c.user_count}</td>`
|
||||
+ `<td>${c.last_at ? fmtTime(c.last_at) : "—"}</td>`
|
||||
+ `</tr>`).join("")
|
||||
|| `<tr><td colspan="7" class="empty">近 ${d.days || 7} 天无失败聚集</td></tr>`;
|
||||
$("s-toolfail").innerHTML = `<div class="card"><h2>工具失败聚集（近 ${d.days || 7} 天，同签名 ≥3 次）</h2>`
|
||||
+ `<div class="scroll-x"><table>`
|
||||
+ `<thead><tr><th>工具</th><th>类型</th><th>签名（悬浮看样例）</th><th>次数</th><th>任务数</th><th>用户数</th><th>最近</th></tr></thead>`
|
||||
+ `<tbody>${body}</tbody></table></div></div>`;
|
||||
}
|
||||
|
||||
function pagerHTML(prefix, page, maxPage, from, to, total) {
|
||||
return `<div class="pager">`
|
||||
+ `<button id="${prefix}-prev" ${page <= 0 ? "disabled" : ""}>上一页</button>`
|
||||
|
|
@ -437,7 +458,14 @@ async function loadStorage(page) {
|
|||
} catch (e) { /* 同上 */ }
|
||||
}
|
||||
|
||||
// overview(固定指标)轮询:拿到后建骨架、渲指标,再顺手刷新三个独立表(保持各自状态)
|
||||
async function loadToolFailures() {
|
||||
try {
|
||||
const d = await apiGet("/v1/admin/tool-failures?days=7&min_count=3&min_tasks=1");
|
||||
renderToolFailures(d);
|
||||
} catch (e) { /* 同上 */ }
|
||||
}
|
||||
|
||||
// overview(固定指标)轮询:拿到后建骨架、渲指标,再顺手刷新四个独立表(保持各自状态)
|
||||
async function refresh() {
|
||||
try {
|
||||
const d = await apiGet("/v1/admin/overview");
|
||||
|
|
@ -446,6 +474,7 @@ async function refresh() {
|
|||
loadModels();
|
||||
loadUserUsage(userPage);
|
||||
loadStorage(storagePage);
|
||||
loadToolFailures();
|
||||
} catch (e) {
|
||||
if (e.code !== "auth") showMsg(`加载失败：${escapeHtml(e.message || String(e))}`);
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue