Compare commits

...

3 Commits

Author SHA1 Message Date
sinohqb
595409487b docs(release): v0.4「联」发布说明与里程碑收尾
Some checks failed
CI / test (push) Failing after 38s
- 新增 release-notes-v0.4.md(事故排查 + 功能总结 + v0.5 候选方向)
- AGENT.md 里程碑表更新至 v0.4 + 鉴权配置说明
- README / plan-v0.4 状态同步
2026-07-28 17:41:20 +08:00
sinohqb
9c564b575e feat(frontend): v0.4 login page, dashboard rebuild, reports UX, keep-alive refresh
- 登录页 + App 鉴权门 + X-Auth-Token 拦截器 + 菜单头部退出按钮
- 仪表盘重构:6 指标卡 / 趋势图 + 场景表现 / 最近记录 + 快捷操作 + 来源分布
- Reports 页重做:场景筛选、富选项下拉、allowClear、一键重置、同场景对比约束
- useOnTabActive:标签页激活自动刷新(根治 AI 助手评测记录"消失")
- ModelConfigs 12 列合并为 6 列;来源 Tag;chunk 告警阈值修正并记录原因
2026-07-28 17:41:08 +08:00
sinohqb
739d586aec feat(backend): v0.4 triggered_by tracking, login gate, compare guard, dashboard stats
- EvalRun.triggered_by 全链路(manual/ai_assistant/cli)+ 迁移 b7d4e6f81c22
- 标准 agenteval-run SKILL.md 纳入版本管理,deploy 脚本同步 + API Key 注入
- 简单登录:AGENTEVAL_ADMIN_PASSWORD + HMAC 会话 token,require_auth 双凭据
- 对比报告限同场景(400)+ 空 results 误判修复
- /api/stats/dashboard 扩展聚合;/api/runs 返回场景/对象名
- 测试 218 → 232
2026-07-28 17:40:54 +08:00
42 changed files with 1441 additions and 326 deletions

View File

@ -6,6 +6,10 @@
# In production, set a strong random token; clients must send it as X-API-Key.
AGENTEVAL_API_KEY=
# If set, the web UI requires login with this password (session-level, stored
# in sessionStorage). Leave empty to disable the login gate (dev default).
AGENTEVAL_ADMIN_PASSWORD=
# Fernet key used to encrypt model API keys stored in SQLite.
# Generate once with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# Keep this value backed up with the database. Do not commit a real key.

View File

@ -6,7 +6,7 @@
**AgentEvalTool** 是智能体质量评估工具集平台,评估 AI 数字员工(大模型 + RAG)和 AI 助手(OpenClaw)的服务质量。核心闭环:定义评估 → 执行评估 → 分析结果 → 改进优化。
- **当前版本**: v0.3.0-dev(2026-07-17 发布至 t480,里程碑「拓」)
- **当前版本**: v0.4.0-dev(2026-07-28 发布至 t480,里程碑「联」)
- **运行环境**: t480 测试服务器(192.168.8.145:8001)
- **语言**: Python 3.11(后端) + TypeScript/React 18(前端)
@ -16,10 +16,11 @@
|---|---|---|---|---|
| v0.1 | MVP | 2026-07-09 | 核心闭环跑通 | ✅ 已发布 |
| v0.2 | 「稳」 | 2026-07-14 | async 引擎 + 安全基线 + 测试 + 部署规范化 | ✅ 已发布(t480) |
| **v0.3** | **「拓」** | **2026-07-17** | **多通道 + 规则扩展 + 模型配置中心** | **✅ 已发布(t480)** |
| v0.4 | — | 规划中 | 前端测试 / 报告增强 / OpenClaw 深度集成二期 / 多模态用例 | 📋 待规划 |
| v0.3 | 「拓」 | 2026-07-17 | 多通道 + 规则扩展 + 模型配置中心 | ✅ 已发布(t480) |
| **v0.4** | **「联」** | **2026-07-28** | **AI 助手标准化(triggered_by) + 登录 + 仪表盘重构** | **✅ 已发布(t480)** |
| v0.5 | — | 规划中 | 前端测试 / 报告聚合看板 / OpenClaw 二期 / 多模态 | 📋 待规划 |
详细内容见 [docs/release-notes-v0.3.md](docs/release-notes-v0.3.md)。
详细内容见 [docs/release-notes-v0.4.md](docs/release-notes-v0.4.md)。
## 技术栈
@ -138,6 +139,12 @@ curl http://192.168.8.145:8001/api/health
- `backend/` → `/app/backend:ro`(代码热更新)
- `.env` → `/app/.env:ro`(Pydantic Settings 配置, 见 `.env.example`)
### 鉴权(v0.4 起)
- `AGENTEVAL_ADMIN_PASSWORD`:设置后 Web UI 需登录(会话 token, 关浏览器失效);不设则无登录门
- `AGENTEVAL_API_KEY`:机器调用凭据(X-API-Key);deploy 脚本会把它写入 openclaw 工作区
`data/openclaw/agenteval-api-key`, agenteval-run skill 自动读取
- 两个凭据任一匹配即放行(`web/deps.py: require_auth`);`/api/health` 与 `/api/auth/*` 始终开放
### 镜像 tag 约定
- `t480-agenteval:<version>` — 不可变, 对应某次具体构建
- `t480-agenteval:latest` — 可变, 总是指向最新部署

View File

@ -1,6 +1,6 @@
# AgentEvalTool
> **当前开发版本**: v0.3.0-dev 「拓」 · 已发布至 t480 · 发布说明见 [docs/release-notes-v0.3.md](docs/release-notes-v0.3.md)
> **当前开发版本**: v0.4.0-dev 「联」 · 已发布至 t480 · 发布说明见 [docs/release-notes-v0.4.md](docs/release-notes-v0.4.md)
智能体质量评估工具集平台。
@ -11,19 +11,23 @@
- 异步评测执行引擎(真并发 / 真取消 / 可配置超时 / WS 实时进度)
- 评测通道:tutu-api / HTTP / OpenClaw
- 评估规则:关键词、响应时间、LLM 评分、语义相似度、JSON Schema、安全检测 + all/any/weighted 组合
- 模型配置中心(统一凭据管理 + 加密落库 + OpenAI/Anthropic/DashScope/Gemini 协议)
- 报告生成(JSON / HTML / Markdown / 对比报告)+ Webhook 通知
- 模型配置中心(统一凭据管理 + Fernet 加密 + OpenAI/Anthropic/DashScope/Gemini 协议)
- 报告生成(JSON / HTML / Markdown / 同场景对比报告)+ Webhook 通知
- 触发来源标记(手动 / AI 助手 / CLI)+ AI 助手标准化评测链路
- 简单登录(访问密码 + 会话 token,未配置零打扰)
- 运营仪表盘(指标卡 / 趋势 / 场景表现 / 触发来源分布)
- 文件管理(分类树 + 上传下载)
- OpenClaw 免登录 iframe 集成 + HTTP Skill
- OpenClaw 免登录 iframe 集成 + 标准 agenteval-run 技能
- Web 管理后台(FastAPI + React)
## v0.3 亮点
## v0.4 亮点
- **模型配置中心**:模型连接统一管理,场景只引用配置 ID,API Key 加密落库、响应零泄露
- **ModelGateway**:chat / embed / moderate 统一入口,4 种协议适配器
- **3 通道 + 6 规则**:工厂/注册表模式,插件式扩展
- **测试基线**:218 个测试,82% 覆盖率
- **报告增强**:Markdown 导出、run 对比、Webhook 通知
- **AI 助手标准化**:`triggered_by` 全链路标记 + 标准 SKILL.md 版本管理与自动同步 + API Key 自动注入
- **简单登录**:`AGENTEVAL_ADMIN_PASSWORD` 一键启用访问控制,机器调用(X-API-Key)不受影响
- **仪表盘重构**:6 指标卡 + 趋势 + 场景表现排行 + 触发来源分布
- **报告页重做**:场景筛选、富选项下拉、一键重置、对比报告限同场景
- **keep-alive 刷新修复**:标签页激活自动刷新,根治"评测记录消失"问题
- **测试基线**:232 个测试
## 快速开始

View File

@ -33,6 +33,13 @@ class Settings(BaseSettings):
default=None,
description="If set, all /api/* endpoints require the X-API-Key header to match.",
)
admin_password: Optional[str] = Field(
default=None,
description=(
"If set, the web UI requires login with this password. "
"Leave empty to disable the login gate (dev default)."
),
)
secret_key: Optional[str] = Field(
default=None,
description="Fernet key used to encrypt model provider API keys at rest.",

View File

@ -24,6 +24,7 @@ from agenteval.models import (
ModelPurpose,
RuleLogic,
RunStatus,
RunTrigger,
Scenario,
Turn,
)
@ -75,6 +76,7 @@ class EvalEngine:
cancel_token: Optional[asyncio.Event] = None,
timeout_config: Optional[TimeoutConfig] = None,
max_concurrent_cases: int = 1,
triggered_by: RunTrigger = RunTrigger.MANUAL,
):
self.target = target
self.scenario = scenario
@ -84,6 +86,7 @@ class EvalEngine:
self.result_repo = result_repo or ResultRepository(self.session)
self.cancel_token = cancel_token or asyncio.Event()
self.timeout_config = timeout_config or TimeoutConfig()
self.triggered_by = triggered_by
self._case_semaphore = asyncio.Semaphore(max(1, max_concurrent_cases))
# Collects fatal case-level errors (e.g. dynamic message generation
# failures) so their cause is persisted into run.summary — not just
@ -116,6 +119,7 @@ class EvalEngine:
target_id=self.target.id or "",
scenario_id=self.scenario.id or "",
status=RunStatus.RUNNING,
triggered_by=self.triggered_by,
started_at=utc_now(),
)
run = self.run_repo.create(run)

View File

@ -169,10 +169,15 @@ def generate_report(run_id: str, session=None) -> dict[str, Any]:
def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[str, Any]:
"""Build a side-by-side comparison dict for two runs."""
"""Build a side-by-side comparison dict for two runs of the same scenario."""
report_a = generate_report(run_id_1, session)
report_b = generate_report(run_id_2, session)
# Cross-scenario case_ids never overlap, so every case would be flagged
# "changed" and the diff would be meaningless — reject early.
if report_a.get("scenario_id") != report_b.get("scenario_id"):
raise ValueError("compare report requires both runs to use the same scenario")
def _summary_delta(key: str) -> float:
return report_b["summary"][key] - report_a["summary"][key]
@ -189,7 +194,11 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
def _case_passed(c):
if not c:
return None
return all(r["passed"] for r in c.get("results", []))
results = c.get("results", [])
if not results:
# No rule results (e.g. errored case) must not count as passed.
return None
return all(r["passed"] for r in results)
case_diffs.append(
{

View File

@ -148,6 +148,12 @@ class RunStatus(str, Enum):
FAILED = "failed"
class RunTrigger(str, Enum):
MANUAL = "manual"
AI_ASSISTANT = "ai_assistant"
CLI = "cli"
class EvalRun(BaseModel):
"""A single evaluation run."""
@ -155,6 +161,7 @@ class EvalRun(BaseModel):
target_id: str
scenario_id: str
status: RunStatus = RunStatus.PENDING
triggered_by: RunTrigger = RunTrigger.MANUAL
started_at: Optional[datetime] = None
completed_at: Optional[datetime] = None
summary: Optional[dict[str, Any]] = None

View File

@ -168,6 +168,7 @@ class EvalRunDB(SQLModel, table=True):
target_id: Optional[str] = Field(default=None, foreign_key="eval_targets.id")
scenario_id: Optional[str] = Field(default=None, foreign_key="scenarios.id")
status: str = "pending"
triggered_by: str = Field(default="manual")
started_at: Optional[datetime] = Field(default_factory=utc_now)
completed_at: Optional[datetime] = None
summary: Optional[str] = None

View File

@ -82,6 +82,7 @@ def _run_to_db(run: EvalRun) -> EvalRunDB:
target_id=run.target_id,
scenario_id=run.scenario_id,
status=run.status.value,
triggered_by=run.triggered_by.value,
started_at=run.started_at,
completed_at=run.completed_at,
)
@ -96,6 +97,7 @@ def _run_from_db(db: EvalRunDB) -> EvalRun:
target_id=db.target_id,
scenario_id=db.scenario_id,
status=db.status,
triggered_by=db.triggered_by or "manual",
started_at=db.started_at,
completed_at=db.completed_at,
summary=db.get_summary(),

View File

@ -11,7 +11,7 @@ from agenteval.config import get_settings
from agenteval.storage.db import init_db
from agenteval.version import get_build_info, get_version
from agenteval.web.deps import require_api_key
from agenteval.web.routers import files, model_configs, proxy, reports, runs, scenarios, stats, targets
from agenteval.web.routers import auth, files, model_configs, proxy, reports, runs, scenarios, stats, targets
from agenteval.web.websocket import ws_manager
@ -40,6 +40,8 @@ app.add_middleware(
_api_deps = [Depends(require_api_key)]
# Login endpoints must stay open — they are how the client obtains credentials.
app.include_router(auth.router, prefix="/api/auth", tags=["auth"])
app.include_router(targets.router, prefix="/api/targets", tags=["targets"], dependencies=_api_deps)
app.include_router(scenarios.router, prefix="/api/scenarios", tags=["scenarios"], dependencies=_api_deps)
app.include_router(runs.router, prefix="/api/runs", tags=["runs"], dependencies=_api_deps)

View File

@ -21,21 +21,37 @@ def get_db():
session.close()
def require_api_key(x_api_key: Optional[str] = Header(default=None)) -> None:
"""Enforce X-API-Key header when AGENTEVAL_API_KEY is configured.
def require_auth(
x_api_key: Optional[str] = Header(default=None),
x_auth_token: Optional[str] = Header(default=None),
) -> None:
"""Enforce authentication when any credential is configured.
When the setting is empty (default), the check is a no-op so local dev keeps
working without extra configuration.
Accepted credentials (either one passes):
- X-API-Key matching AGENTEVAL_API_KEY (machine callers, OpenClaw skill)
- X-Auth-Token matching the web session token (browser after login)
When neither AGENTEVAL_API_KEY nor AGENTEVAL_ADMIN_PASSWORD is set, the
check is a no-op so local dev keeps working without extra configuration.
"""
configured = get_settings().api_key
if not configured:
settings = get_settings()
if not settings.api_key and not settings.admin_password:
return
if not x_api_key or x_api_key != configured:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or missing X-API-Key",
)
if settings.api_key and x_api_key == settings.api_key:
return
if settings.admin_password:
from agenteval.web.routers.auth import verify_session_token
if x_auth_token and verify_session_token(x_auth_token):
return
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or missing credentials",
)
# Backwards-compatible alias: existing routers depend on require_api_key.
require_api_key = require_auth
# Convenience alias used by routers via ``dependencies=[Depends(auth_required)]``.
auth_required = Depends(require_api_key)
auth_required = Depends(require_auth)

View File

@ -0,0 +1,52 @@
"""Session login for the web UI.
A single shared password (AGENTEVAL_ADMIN_PASSWORD) gates the SPA. On success
the client receives a stateless HMAC token it sends as X-Auth-Token. Machine
callers (e.g. the OpenClaw skill) keep using X-API-Key and are unaffected.
"""
import hmac
import secrets
from fastapi import APIRouter, Header, HTTPException, status
from pydantic import BaseModel
from agenteval.config import get_settings
router = APIRouter()
_TOKEN_CONTEXT = b"agenteval-web-session"
def session_token() -> str:
"""Derive the stateless session token from configured secrets."""
settings = get_settings()
key = (settings.secret_key or settings.admin_password or "").encode()
return hmac.new(key, _TOKEN_CONTEXT, "sha256").hexdigest()
def verify_session_token(token: str) -> bool:
return bool(token) and hmac.compare_digest(token, session_token())
class LoginRequest(BaseModel):
password: str
@router.post("/login")
def login(request: LoginRequest) -> dict:
configured = get_settings().admin_password
if not configured:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="登录未启用")
if not secrets.compare_digest(request.password, configured):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="密码错误")
return {"token": session_token()}
@router.get("/status")
def auth_status(x_auth_token: str = Header(default="")) -> dict:
auth_required = bool(get_settings().admin_password)
return {
"auth_required": auth_required,
"authenticated": (not auth_required) or verify_session_token(x_auth_token),
}

View File

@ -23,10 +23,14 @@ def get_compare_report(
session: Session = Depends(get_db),
) -> dict:
repo = RunRepository(session)
if not repo.get(run1):
run_a = repo.get(run1)
run_b = repo.get(run2)
if not run_a:
raise HTTPException(status_code=404, detail=f"run not found: {run1}")
if not repo.get(run2):
if not run_b:
raise HTTPException(status_code=404, detail=f"run not found: {run2}")
if run_a.scenario_id != run_b.scenario_id:
raise HTTPException(status_code=400, detail="对比报告要求两个运行使用相同场景")
return generate_compare_report(run1, run2, session)

View File

@ -8,7 +8,7 @@ from pydantic import BaseModel
from sqlmodel import Session
from agenteval.evaluation.engine import EvalEngine
from agenteval.models import EvalRun, RunStatus
from agenteval.models import EvalRun, RunStatus, RunTrigger
from agenteval.storage.db import get_session, iso_utc
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from agenteval.utils.llm import extract_reply_text
@ -22,6 +22,7 @@ router = APIRouter()
class StartRunRequest(BaseModel):
target_id: str
scenario_id: str
triggered_by: RunTrigger = RunTrigger.MANUAL
# ── Task registry for live evaluation runs ─────────────────────────────
@ -69,7 +70,16 @@ async def _run_evaluation(run_id: str, target_id: str, scenario_id: str) -> None
@router.get("")
async def list_runs(session: Session = Depends(get_db)) -> list[dict]:
return [r.model_dump() for r in RunRepository(session).list_all()]
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
target_names = {t.id: t.name for t in TargetRepository(session).list_all()}
return [
{
**r.model_dump(),
"scenario_name": scenario_names.get(r.scenario_id),
"target_name": target_names.get(r.target_id),
}
for r in RunRepository(session).list_all()
]
@router.post("")
@ -82,7 +92,11 @@ async def start_run(
if not target or not scenario:
raise HTTPException(status_code=404, detail="target or scenario not found")
run = EvalRun(target_id=request.target_id, scenario_id=request.scenario_id)
run = EvalRun(
target_id=request.target_id,
scenario_id=request.scenario_id,
triggered_by=request.triggered_by,
)
run = RunRepository(session).create(run)
task = asyncio.create_task(

View File

@ -1,34 +1,93 @@
"""API routes for statistics and dashboard data."""
from collections import defaultdict
from datetime import datetime, timezone
from fastapi import APIRouter, Depends
from sqlmodel import Session
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from agenteval.web.deps import get_db
router = APIRouter()
def _ts(dt: datetime | None) -> float:
"""Sortable timestamp tolerant of naive/aware mixes in legacy rows."""
if dt is None:
return 0.0
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.timestamp()
@router.get("/dashboard")
def dashboard(session: Session = Depends(get_db)) -> dict:
targets = TargetRepository(session).list_all()
scenarios = ScenarioRepository(session).list_all()
runs = RunRepository(session).list_all()
model_configs = ModelConfigRepository(session).list_all()
scenario_names = {s.id: s.name for s in scenarios}
target_names = {t.id: t.name for t in targets}
completed_runs = [r for r in runs if r.status == "completed" and r.summary]
pass_rates = [r.summary.get("pass_rate", 0) for r in completed_runs if isinstance(r.summary, dict)]
overall_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else None
recent_runs = sorted(runs, key=lambda r: r.started_at or "", reverse=True)[:10]
today = datetime.now(timezone.utc).date()
today_runs = 0
running_count = 0
trigger_breakdown: dict[str, int] = defaultdict(int)
for r in runs:
if r.started_at:
started = r.started_at
if started.tzinfo is None:
started = started.replace(tzinfo=timezone.utc)
if started.date() == today:
today_runs += 1
if r.status in ("running", "pending"):
running_count += 1
trigger_breakdown[r.triggered_by.value] += 1
# Per-scenario aggregation over completed runs.
by_scenario: dict[str, list] = defaultdict(list)
for r in completed_runs:
by_scenario[r.scenario_id].append(r)
scenario_stats = []
for sid, sruns in by_scenario.items():
rates = [r.summary.get("pass_rate", 0) for r in sruns if isinstance(r.summary, dict)]
last_run = max(sruns, key=lambda r: _ts(r.started_at))
scenario_stats.append({
"scenario_id": sid,
"scenario_name": scenario_names.get(sid, sid[:8]),
"run_count": len(sruns),
"avg_pass_rate": round(sum(rates) / len(rates), 4) if rates else None,
"last_run_at": last_run.started_at.isoformat() if last_run.started_at else None,
})
scenario_stats.sort(key=lambda s: s["run_count"], reverse=True)
recent_runs = sorted(runs, key=lambda r: _ts(r.started_at), reverse=True)[:10]
return {
"targets_count": len(targets),
"scenarios_count": len(scenarios),
"runs_count": len(runs),
"model_configs_count": len(model_configs),
"today_runs": today_runs,
"running_count": running_count,
"overall_pass_rate": overall_pass_rate,
"recent_runs": [r.model_dump() for r in recent_runs],
"trigger_breakdown": dict(trigger_breakdown),
"scenario_stats": scenario_stats,
"recent_runs": [
{
**r.model_dump(),
"scenario_name": scenario_names.get(r.scenario_id),
"target_name": target_names.get(r.target_id),
}
for r in recent_runs
],
}

View File

@ -5,6 +5,7 @@ from typing import Any
import typer
from agenteval.evaluation.engine import EvalEngine
from agenteval.models import RunTrigger
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from rich.console import Console
from rich.table import Table
@ -44,7 +45,7 @@ def start_run(
raise typer.Exit(1)
console.print(f"开始评测: 对象={target.name}, 场景={scenario.name}")
engine = EvalEngine(target=target, scenario=scenario)
engine = EvalEngine(target=target, scenario=scenario, triggered_by=RunTrigger.CLI)
try:
run = asyncio.run(engine.run(progress_callback=_progress))
except Exception as exc:

View File

@ -1,57 +1,45 @@
# OpenClaw 插件 for AgentEvalTool
本目录提供 OpenClaw 调用 AgentEvalTool 的示例技能,使 OpenClaw 能够自闭环完成评测策略设计。
本目录提供 OpenClaw 调用 AgentEvalTool 的标准接入方式,使 OpenClaw(AI 助手)能够自闭环完成评测。
## 设计原则
- AgentEvalTool 作为独立的 CLI 工具集存在,负责:
- AgentEvalTool 负责:
- 评测对象管理
- 评测场景管理
- 评测执行与原始数据收集
- 评估分析与报告生成
- OpenClaw 通过本插件调用 `agenteval` CLI,负责:
- OpenClaw 负责:
- 评测策略编排(选择对象、场景、触发时机)
- 定时调度(OpenClaw 自身的 cron + skill 机制)
- 结果通知与后续动作
## 前置条件
## 唯一标准接入方式:HTTP API
1. 已安装 AgentEvalTool CLI:
```bash
pip install -e /path/to/AgentEvalTool
```
2. `agenteval` 命令已在 PATH 中可用。
3. 已通过 `agenteval target add` 注册评测对象。
4. 已通过 `agenteval scenario import` 导入评测场景。
**所有 OpenClaw 触发的评测必须走 AgentEvalTool 的 HTTP API**(`POST /api/runs` 等),
并在启动评测时携带 `"triggered_by": "ai_assistant"`,以便平台区分手动触发与 AI 助手触发。
## 接入方式
> ⚠️ 禁止在 OpenClaw 中通过 subprocess 调用 `agenteval` CLI。
> CLI 直接写入其运行环境下的本地 SQLite(路径硬编码),与 Web 后台的数据库不共享,
> 会导致评测记录"消失"——Web 页面永远查不到。CLI 仅用于开发者在 agenteval
> 服务同一环境下的手工调试。
### 方式一:直接调用 CLI(推荐)
### 方式一:agenteval-run 技能(推荐)
在 OpenClaw Skill 中直接调用系统命令:
`skills/agenteval-run/SKILL.md` 是标准技能文件,指导 AI 助手用 `curl` 走 API 全流程
(列对象/场景 → 启动评测 → 轮询状态 → 取报告)。
```python
import subprocess
部署脚本(`scripts/deploy-t480.sh`)会自动把它同步到 OpenClaw 工作区:
`data/openclaw/workspace/skills/agenteval-run/SKILL.md`。
# 触发评测
subprocess.run([
"agenteval", "run", "start",
"--target-id", "<target-id>",
"--scenario-id", "<scenario-id>",
], check=True)
### 方式二:Python Skill 封装
# 获取报告(需要在运行输出中解析 run_id)
report = subprocess.check_output([
"agenteval", "report", "show", "<run-id>", "--format", "json",
])
```
### 方式二:使用本目录封装的 Skill
参考 `agenteval_skill.py`,在 OpenClaw 中注册技能时传入配置:
参考 `agenteval_skill.py`(httpx 调 API),在 OpenClaw 中注册技能时传入配置:
```json
{
"api_base_url": "http://agenteval:8000",
"api_key": "<可选,对应 AGENTEVAL_API_KEY>",
"target_id": "<target-id>",
"scenario_id": "<scenario-id>",
"report_format": "json"
@ -66,20 +54,21 @@ OpenClaw 的 cron 配置(具体格式以 OpenClaw 平台为准):
skill: agenteval_skill
schedule: "0 9 * * *" # 每天早上 9 点执行
config:
api_base_url: "http://agenteval:8000"
target_id: "<target-id>"
scenario_id: "<scenario-id>"
report_format: "json"
```
## CLI 返回的 run_id 解析
## API 速查
`agenteval run start` 成功后会输出:
| 操作 | 请求 |
|---|---|
| 启动评测 | `POST /api/runs` body: `{"target_id", "scenario_id", "triggered_by": "ai_assistant"}` |
| 查询状态 | `GET /api/runs/{run_id}`(轮询至 completed/failed) |
| 获取报告 | `GET /api/reports/{run_id}` |
```text
评测完成: run_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, status=completed
```
OpenClaw Skill 需要从 stdout 中提取 `run_id`,然后调用 `agenteval report show <run_id>` 获取报告。
如果平台设置了 `AGENTEVAL_API_KEY`,所有请求需带 `X-API-Key` 头。
## 扩展建议

View File

@ -44,6 +44,7 @@ class AgentEvalSkill:
json={
"target_id": self.target_id,
"scenario_id": self.scenario_id,
"triggered_by": "ai_assistant",
},
)
if resp.status_code != 200:

View File

@ -0,0 +1,54 @@
---
name: agenteval-run
description: 触发 AgentEvalTool 评测任务并获取报告
---
当用户要求执行评测时,使用 exec 工具运行以下命令。
所有评测必须走 AgentEvalTool 标准 HTTP API(禁止直接调 CLI 或操作数据库)。
平台可能启用了 API Key 鉴权。每次执行命令前先读取密钥(文件不存在则为空,不影响未启用鉴权的环境):
```bash
KEY=$(cat ~/.openclaw/agenteval-api-key 2>/dev/null)
```
以下所有 curl 命令都必须带 `-H "X-API-Key: $KEY"`。
## 查看评测对象列表
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/targets | python3 -m json.tool
```
## 查看评测场景列表
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/scenarios | python3 -m json.tool
```
## 启动评测
必须携带 `"triggered_by": "ai_assistant"`,平台以此区分手动触发与 AI 助手触发:
```bash
curl -s -X POST http://agenteval:8000/api/runs \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"target_id": "<target_id>", "scenario_id": "<scenario_id>", "triggered_by": "ai_assistant"}'
```
响应中的 `id` 字段即 run_id,请务必回报给用户。
## 查看运行状态
轮询直到 `status` 变为 `completed` 或 `failed`(建议每 5 秒一次):
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/runs/<run_id> | python3 -m json.tool
```
## 获取报告
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/reports/<run_id> | python3 -m json.tool
```
请将 <target_id>、<scenario_id>、<run_id> 替换为实际值。
完成后向用户汇报:run_id、状态、通过率(summary.pass_rate)以及失败用例摘要。

70
docs/plan-v0.4.md Normal file
View File

@ -0,0 +1,70 @@
# AgentEvalTool v0.4.0 「联」开发计划
**开发版本**: v0.4.0-dev
**制定日期**: 2026-07-27
**代码基线**: `92f98c3`(v0.3.0-dev 已发布 t480)
**核心目标**: AI 助手评测链路标准化 + 报告查询体验优化
**状态**: ✅ 已发布(t480, 2026-07-28),发布说明见 [release-notes-v0.4.md](release-notes-v0.4.md)
---
## 一、背景:run 95ee8738 "消失"事故排查
用户通过 AI 助手触发评测(run `95ee8738-11e7-4959-a3cf-45e3536b28eb`),但在"评测执行"和"评测报告"页面均查不到。
**排查结论(2026-07-27)**:
| 检查项 | 结果 |
|---|---|
| t480 数据库 `eval_runs` | ✅ run 存在,status=completed |
| `GET /api/runs` | ✅ 正常返回,列表第一条 |
| t480 SKILL.md(openclaw 工作区) | ✅ 走标准 `POST /api/runs` |
| 前端展示 | ❌ **根因所在** |
**根因**:前端采用 keep-alive 标签页架构(页面挂载后永不卸载),Runs / Reports 页只在**首次挂载**时加载一次列表,切回标签页不刷新。AI 助手评测发生在页面已挂载之后 → 两个页面全是旧数据。叠加因素:Runs 页默认时间过滤为"今天"、Reports 页只显示 completed 状态。
**数据与 API 完全正常,纯前端展示层问题。**
## 二、需求与实现
### 需求 1:触发来源标记 + AI 助手标准化链路
- `EvalRun.triggered_by` 枚举字段(`manual` / `ai_assistant` / `cli`,默认 manual),全链路透传:models → EvalRunDB → repository → engine → CLI → API
- 迁移 `b7d4e6f81c22`:`eval_runs` 加列,存量回填 `manual`
- `POST /api/runs` 接受可选 `triggered_by`(非法值 422)
- 标准 skill 纳入版本管理:`backend/plugins/openclaw/skills/agenteval-run/SKILL.md`(要求带 `triggered_by: "ai_assistant"`),deploy 脚本自动同步到 openclaw 工作区
- `plugins/openclaw/README.md` 移除 CLI subprocess 模式(CLI 写孤立库,禁止 OpenClaw 使用)
- 前端:RunList 与 Reports 下拉显示来源 Tag(AI 助手=紫色 / CLI=蓝色 / 手动不显示)
### 需求 1.5(排查衍生,本次事故的直接修复):标签页激活刷新
- 新 hook `useOnTabActive(path, cb)`:监听 tabStore.activeKey,标签页重新激活时触发刷新(跳过首次挂载)
- Runs、Reports 两页接入
### 需求 2:报告页查询体验优化
- 后端 `GET /api/runs` 返回冗余 `scenario_name` / `target_name`
- Reports 页选择区重做:
- 场景筛选器(含各场景 run 数量统计)
- run 下拉富选项:场景名 · 对象名 · 时间 · 通过率 · 来源 Tag,支持关键字搜索
- 修复旧版 `.reverse()` 导致最旧记录排最前的问题
### 需求 3:对比报告限制同场景
- 后端 `GET /api/reports/compare`:场景不同返回 `400 对比报告要求两个运行使用相同场景`
- `generate_compare_report` 入口双保险(ValueError)
- 顺带修复:case 无规则结果时 `all([]) == True` 被误判为通过 → 改为 None
- 前端:报告 B 下拉只列同场景 run;A 变更后 B 场景不符自动清空
## 三、测试
- 新增 6 个测试:triggered_by 默认值/透传持久化/非法值 422、list 含名称字段、compare 跨场景 400/ValueError
- 全量 224 通过(v0.3 基线 218 → 224)
## 四、发布验证清单(t480)
1. `scripts/deploy-t480.sh` 部署(自动同步 SKILL.md 到 openclaw 工作区)
2. `/api/health` 返回 version=0.4.0-dev
3. 手动触发一次评测 → Runs 页显示无来源 Tag(manual)
4. 让 AI 助手触发一次评测 → 切回 Runs/Reports 页**不刷新浏览器**即可看到新记录,带"AI 助手"标记
5. 对比报告:跨场景 run 不可选、同场景可正常对比

View File

@ -0,0 +1,95 @@
# AgentEvalTool v0.4 版本发布说明
**版本**: v0.4.0-dev
**日期**: 2026-07-27 ~ 2026-07-28
**状态**: 已发布(t480 开发线)
**代号**: 「联」(Integration milestone)
**作者**: AgentEval Team
---
## 一、版本定位
v0.4 的主题是**打通 AI 助手与平台的标准化联动**,并完成生产化前的最后两块拼图:访问控制与运营视图。起因是一次真实事故:用户通过 AI 助手触发的评测(run `95ee8738`)在页面上"消失"。
```
v0.1 (2026-07-09) MVP 闭环
v0.2 (2026-07-14) 「稳」async 引擎 + 安全基线 + 部署规范化
v0.3 (2026-07-17) 「拓」多通道 + 规则扩展 + 模型配置中心
v0.4 (2026-07-28) ← 你在这里:「联」AI 助手标准化 + 登录 + 仪表盘重构
```
## 二、事故排查:run 95ee8738 "消失"
| 检查项 | 结果 |
|---|---|
| t480 数据库 / API | ✅ run 存在、completed、列表第一条 |
| OpenClaw skill 调用链 | ✅ 走标准 POST /api/runs |
| 前端展示 | ❌ **根因** |
**根因**:keep-alive 标签页架构下页面永不重新挂载,Runs / Reports 只在首次挂载时加载列表,切回标签页不刷新;叠加 Runs 页默认"今天"时间过滤、Reports 页只显示 completed。**纯前端展示层问题,数据无损**。
**修复**:新增 `useOnTabActive` hook — 标签页重新激活时自动刷新,已接入仪表盘 / Runs / Reports 三页。
## 三、主要功能
### 3.1 触发来源标记(triggered_by)
- `EvalRun.triggered_by` 枚举(manual / ai_assistant / cli)全链路:模型 → DB(迁移 `b7d4e6f81c22`,存量回填 manual)→ API → CLI → 引擎
- `POST /api/runs` 接受可选 `triggered_by`,非法值 422
- 前端 Runs 列表、Reports 下拉、仪表盘显示来源 Tag(AI 助手=紫 / CLI=蓝)
### 3.2 AI 助手链路标准化
- 标准 skill `backend/plugins/openclaw/skills/agenteval-run/SKILL.md` 纳入版本管理,deploy 脚本自动同步到 openclaw 工作区
- skill 自动从 `~/.openclaw/agenteval-api-key` 读取 API Key(deploy 脚本从远端 .env 提取生成),启用鉴权后 AI 助手不受影响
- `plugins/openclaw/README.md` 废除 CLI subprocess 模式(CLI 直写本地 SQLite 孤立库,是"记录消失"类事故的高危路径)
### 3.3 评测报告页体验重做
- `GET /api/runs` 返回冗余 `scenario_name` / `target_name`
- 两行表头(模式+场景筛选+重置/刷新 | 报告选择+操作),对比模式不再换行
- 场景筛选器(含计数)、富选项下拉(场景·对象·时间·通过率·来源)、全下拉 allowClear、一键重置
- **对比报告限同场景**:后端 400 + generate 层双保险 + 前端 B 下拉只列同场景;顺带修复空 results 被 `all([])` 误判通过
### 3.4 简单登录(访问控制)
- `.env` 配 `AGENTEVAL_ADMIN_PASSWORD` 即启用;未配置零打扰
- `POST /api/auth/login` 换取无状态 HMAC 会话 token(sessionStorage,关浏览器失效);401 自动踢回登录页;菜单头部图标退出
- `require_api_key` 升级 `require_auth`:X-API-Key(机器)与 X-Auth-Token(浏览器)双凭据并行
### 3.5 仪表盘全面重构
- `/api/stats/dashboard` 扩展:模型配置数、今日执行、运行中、触发来源分布、场景聚合统计、富化最近记录
- 三行布局:6 指标卡 / 趋势图+场景表现排行 / 最近记录+快捷操作+来源分布
- 页面请求从 3 个全量列表收敛为 2 个聚合接口
### 3.6 UI 与工程优化
- 模型配置页 12 列 → 6 列合并,取消横向滚动(Endpoint/完整 Token 数移入悬浮提示)
- 仪表盘滚动修复(flex:1 在非 flex 父容器失效 → height:100%)
- vite `chunkSizeWarningLimit: 2200`:注释记录 vendor 2.12MB 不可拆分原因(@antv 强拆触发 TDZ 崩溃、rollup 拒绝 async 归并的实测结论),阈值仅在真实回归时报警
## 四、测试与质量
| 指标 | v0.3 | v0.4 |
|---|---|---|
| 测试数量 | 218 | **232** |
| 新增 | — | triggered_by×4、compare 场景限制×2、auth×5、stats×3 |
CI 三件套(v0.3 收尾时补齐)在本版本首次实战:pre-push hook 拦截推送前跑版本一致性 + ruff + pytest。
## 五、部署与配置变更
- **t480 `.env` 新增**(需手动维护,陷阱 #5):`AGENTEVAL_ADMIN_PASSWORD`、`AGENTEVAL_API_KEY`
- deploy-t480.sh 新增:SKILL.md 同步 + API Key 文件注入 openclaw 工作区
- t480 验证:登录门 401/200 全链路、AI 助手带 key 调用 200、仪表盘聚合数据正确
## 六、已知问题
1. t480 访问密码为临时值,需自行更换
2. 前端仍无自动化测试(仅 tsc)
3. `.env` 远端手动维护未根治(陷阱 #5)
4. volcengine-102 生产线尚未同步 v0.4(含登录配置)
## 七、v0.5 候选方向
- 前端测试基线(vitest)
- 报告趋势分析 / 多 run 聚合看板
- OpenClaw 深度集成二期(插件双向调用)
- 多模态评测用例
- 配置同步自动化(消灭陷阱 #5)

View File

@ -1,12 +1,12 @@
{
"name": "agenteval-web",
"version": "0.3.0-dev",
"version": "0.4.0-dev",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "agenteval-web",
"version": "0.3.0-dev",
"version": "0.4.0-dev",
"dependencies": {
"@ant-design/charts": "^2.6.7",
"@ant-design/icons": "^6.3.2",

View File

@ -1,6 +1,6 @@
{
"name": "agenteval-web",
"version": "0.3.0-dev",
"version": "0.4.0-dev",
"private": true,
"type": "module",
"scripts": {

View File

@ -1,6 +1,6 @@
import { lazy, Suspense, useEffect } from 'react'
import { lazy, Suspense, useEffect, useState } from 'react'
import { useLocation, useNavigate } from 'react-router-dom'
import { Layout, Menu, Spin } from 'antd'
import { Layout, Menu, Spin, Button, Tooltip } from 'antd'
import type { MenuProps } from 'antd'
import {
DashboardOutlined,
@ -11,8 +11,11 @@ import {
RobotOutlined,
FolderOpenOutlined,
CloudServerOutlined,
LogoutOutlined,
} from '@ant-design/icons'
import TabBar from './components/TabBar'
import LoginPage from './pages/Login'
import { authApi, authToken, setOnUnauthorized } from './api'
import { useTabStore, type TabItem } from './stores/tabStore'
import { colors } from './tokens'
import type { ReactNode } from 'react'
@ -68,10 +71,30 @@ const menuItems: MenuProps['items'] = routeConfigs.map((r) => ({
label: r.name,
}))
type AuthState = 'checking' | 'login' | 'ready'
function App() {
const location = useLocation()
const navigate = useNavigate()
const { tabs, activeKey, openTab, setActiveTab } = useTabStore()
const [authState, setAuthState] = useState<AuthState>('checking')
const [authRequired, setAuthRequired] = useState(false)
useEffect(() => {
setOnUnauthorized(() => setAuthState('login'))
authApi.status()
.then((res) => {
setAuthRequired(res.data.auth_required)
setAuthState(res.data.authenticated ? 'ready' : 'login')
})
.catch(() => setAuthState('ready')) // 后端不可达时不锁死界面,由具体请求报错
return () => setOnUnauthorized(null)
}, [])
const logout = () => {
authToken.clear()
setAuthState('login')
}
useEffect(() => {
const config = routeConfigs.find((r) => r.path === location.pathname)
@ -102,6 +125,18 @@ function App() {
navigate(key)
}
if (authState === 'checking') {
return (
<div style={{ height: '100vh', display: 'flex', alignItems: 'center', justifyContent: 'center' }}>
<Spin size="large" />
</div>
)
}
if (authState === 'login') {
return <LoginPage onSuccess={() => setAuthState('ready')} />
}
return (
<Layout style={{ height: '100vh', overflow: 'hidden' }}>
<Layout.Sider
@ -120,7 +155,7 @@ function App() {
height: 56,
display: 'flex',
alignItems: 'center',
padding: '0 20px',
padding: '0 12px 0 20px',
flexShrink: 0,
borderBottom: '1px solid rgba(255,255,255,0.06)',
}}
@ -128,6 +163,17 @@ function App() {
<span style={{ color: '#fff', fontWeight: 700, fontSize: 16, letterSpacing: '0.3px' }}>
AgentEvalTool
</span>
{authRequired && (
<Tooltip title="退出登录" placement="right">
<Button
type="text"
size="small"
icon={<LogoutOutlined />}
onClick={logout}
style={{ color: colors.siderText, marginLeft: 'auto' }}
/>
</Tooltip>
)}
</div>
<Menu
theme="dark"

View File

@ -1,14 +1,38 @@
import axios from 'axios'
import { message } from 'antd'
const AUTH_TOKEN_KEY = 'agenteval_auth_token'
export const authToken = {
get: () => sessionStorage.getItem(AUTH_TOKEN_KEY),
set: (token: string) => sessionStorage.setItem(AUTH_TOKEN_KEY, token),
clear: () => sessionStorage.removeItem(AUTH_TOKEN_KEY),
}
/** Registered by App to switch back to the login screen on 401. */
let onUnauthorized: (() => void) | null = null
export const setOnUnauthorized = (handler: (() => void) | null) => { onUnauthorized = handler }
const api = axios.create({
baseURL: '/api',
timeout: 30000,
})
api.interceptors.request.use((config) => {
const token = authToken.get()
if (token) config.headers['X-Auth-Token'] = token
return config
})
api.interceptors.response.use(
(response) => response,
(error) => {
const isLoginCall = error.config?.url?.includes('/auth/login')
if (error.response?.status === 401 && !isLoginCall && onUnauthorized) {
authToken.clear()
onUnauthorized()
return Promise.reject(error)
}
const msg = error.response?.data?.detail || error.message || '请求失败'
message.error(msg)
return Promise.reject(error)
@ -17,6 +41,16 @@ api.interceptors.response.use(
export default api
export interface AuthStatus {
auth_required: boolean
authenticated: boolean
}
export const authApi = {
status: () => api.get<AuthStatus>('/auth/status'),
login: (password: string) => api.post<{ token: string }>('/auth/login', { password }),
}
export interface Target {
id: string
name: string
@ -100,21 +134,39 @@ export interface ModelConfigReference {
purpose: string
}
export type RunTrigger = 'manual' | 'ai_assistant' | 'cli'
export interface Run {
id: string
target_id: string
scenario_id: string
status: string
triggered_by?: RunTrigger
scenario_name?: string | null
target_name?: string | null
started_at: string
completed_at: string | null
summary: Record<string, unknown> | null
}
export interface ScenarioStat {
scenario_id: string
scenario_name: string
run_count: number
avg_pass_rate: number | null
last_run_at: string | null
}
export interface DashboardStats {
targets_count: number
scenarios_count: number
runs_count: number
model_configs_count: number
today_runs: number
running_count: number
overall_pass_rate: number | null
trigger_breakdown: Partial<Record<RunTrigger, number>>
scenario_stats: ScenarioStat[]
recent_runs: Run[]
}

View File

@ -1,10 +1,10 @@
import { useMemo, useState } from 'react'
import { Empty, Input, Pagination, Progress, Segmented, Skeleton, Tooltip } from 'antd'
import { Empty, Input, Pagination, Progress, Segmented, Skeleton, Tag, Tooltip } from 'antd'
import { FileTextOutlined, SearchOutlined } from '@ant-design/icons'
import dayjs from 'dayjs'
import type { Run, Scenario, Target } from '../api'
import type { RunProgress } from '../hooks/useRunSession'
import { colors, statusColors, statusLabels } from '../tokens'
import { colors, statusColors, statusLabels, triggerColors, triggerLabels } from '../tokens'
import { elapsedStr, shortDateTime, toDate } from '../utils/date'
const PAGE_SIZE = 10
@ -292,6 +292,14 @@ function RunRow({ r, selected, targetName, scenarioName, onSelect, onOpenReport,
<span>{shortDateTime(r.started_at)}</span>
<span>·</span>
<span>{elapsedStr(r.started_at, r.completed_at)}</span>
{r.triggered_by && r.triggered_by !== 'manual' && (
<Tag
color={triggerColors[r.triggered_by] ?? 'default'}
style={{ marginLeft: 2, marginRight: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}
>
{triggerLabels[r.triggered_by] ?? r.triggered_by}
</Tag>
)}
{onOpenReport && status === 'completed' && (
<Tooltip title="查看报告">
<FileTextOutlined

View File

@ -0,0 +1,23 @@
import { useEffect, useRef } from 'react'
import { useTabStore } from '../stores/tabStore'
/**
* Keep-alive tabs never remount, so "load on mount" data goes stale.
* Fires `callback` every time the given tab path becomes active again
* (skips the initial mount, which pages already handle themselves).
*/
export function useOnTabActive(path: string, callback: () => void) {
const activeKey = useTabStore((s) => s.activeKey)
const first = useRef(true)
const cbRef = useRef(callback)
cbRef.current = callback
useEffect(() => {
if (activeKey !== path) return
if (first.current) {
first.current = false
return
}
cbRef.current()
}, [activeKey, path])
}

View File

@ -1,56 +1,39 @@
import { useEffect, useState } from 'react'
import { Button, Col, Row, Table, Tag, Spin, Card, Tooltip } from 'antd'
import { Button, Card, Col, Empty, Progress, Row, Spin, Tag, Tooltip } from 'antd'
import { Line } from '@ant-design/charts'
import { useNavigate } from 'react-router-dom'
import {
AimOutlined,
BarChartOutlined,
CheckCircleOutlined,
CloudServerOutlined,
FileTextOutlined,
PlayCircleOutlined,
CheckCircleOutlined,
PlusOutlined,
ReloadOutlined,
RobotOutlined,
SyncOutlined,
} from '@ant-design/icons'
import { runsApi, scenariosApi, targetsApi, type Run, type TrendPoint } from '../api'
import { statsApi, type DashboardStats, type Run, type TrendPoint } from '../api'
import PageWrapper from '../components/PageWrapper'
import StatCard from '../components/StatCard'
import { useOnTabActive } from '../hooks/useOnTabActive'
import { colors, statusColors, statusLabels, triggerColors, triggerLabels } from '../tokens'
import { formatDateTime } from '../utils/date'
import { passRateColor } from '../utils/colors'
export default function HomePage() {
const navigate = useNavigate()
const [loading, setLoading] = useState(true)
const [counts, setCounts] = useState({ targets: 0, scenarios: 0, runs: 0 })
const [recentRuns, setRecentRuns] = useState<Run[]>([])
const [stats, setStats] = useState<DashboardStats | null>(null)
const [trend, setTrend] = useState<TrendPoint[]>([])
const [passRate, setPassRate] = useState<number | null>(null)
const loadData = async () => {
setLoading(true)
try {
const [t, s, r] = await Promise.all([
targetsApi.list(),
scenariosApi.list(),
runsApi.list(),
])
const runs = r.data
setCounts({
targets: t.data.length,
scenarios: s.data.length,
runs: runs.length,
})
setRecentRuns(runs.slice(-10).reverse())
const completedRuns = runs.filter((run) => run.status === 'completed' && run.summary)
if (completedRuns.length > 0) {
const totalRate = completedRuns.reduce(
(sum, run) => sum + ((run.summary as any)?.pass_rate || 0),
0,
)
setPassRate(totalRate / completedRuns.length)
}
const trendData: TrendPoint[] = completedRuns.map((run) => ({
date: run.started_at?.split('T')[0] || '',
pass_rate: ((run.summary as any)?.pass_rate || 0) * 100,
run_count: 1,
}))
setTrend(trendData)
const [d, t] = await Promise.all([statsApi.dashboard(), statsApi.trend(30)])
setStats(d.data)
setTrend(t.data)
} catch {
// errors handled by interceptor
} finally {
@ -59,39 +42,25 @@ export default function HomePage() {
}
useEffect(() => { loadData() }, [])
useOnTabActive('/', loadData)
const statusColor: Record<string, string> = {
completed: 'success',
running: 'processing',
pending: 'default',
failed: 'error',
}
const columns = [
{ title: 'Run ID', dataIndex: 'id', key: 'id', ellipsis: true, width: 240 },
{ title: '状态', dataIndex: 'status', key: 'status', width: 100,
render: (s: string) => <Tag color={statusColor[s] || 'default'}>{s}</Tag>,
},
{ title: '开始时间', dataIndex: 'started_at', key: 'started_at', width: 180,
render: (t: string) => t ? formatDateTime(t) : '-',
},
{ title: '通过率', key: 'pass_rate', width: 100,
render: (_: any, record: Run) => {
const rate = (record.summary as any)?.pass_rate
return rate != null ? `${(rate * 100).toFixed(0)}%` : '-'
},
},
]
const passRate = stats?.overall_pass_rate ?? null
const trendConfig = {
data: trend,
xField: 'date',
yField: 'pass_rate',
smooth: true,
height: 200,
height: 240,
yAxis: { label: { formatter: (v: string) => `${v}%` }, min: 0, max: 100 },
point: { size: 3 },
color: '#1677ff',
tooltip: {
formatter: (datum: TrendPoint) => ({
name: '通过率',
value: `${datum.pass_rate}%(${datum.run_count ?? '-'} 次执行)`,
}),
},
}
return (
@ -106,58 +75,163 @@ export default function HomePage() {
</Tooltip>
}
>
{/* 内容区 — 可滚动 */}
<div style={{ flex: 1, minHeight: 0, overflowY: 'auto', padding: '0 16px 16px' }}>
<div style={{ height: '100%', overflowY: 'auto', padding: '0 16px 16px' }}>
<Spin spinning={loading}>
{/* 指标卡行 */}
<Row gutter={[16, 16]}>
<Col xs={24} sm={12} lg={6}>
<StatCard icon={<AimOutlined />} color="blue" title="评测对象" value={counts.targets} />
<Col xs={12} sm={8} xl={4}>
<StatCard icon={<AimOutlined />} color="blue" title="评测对象" value={stats?.targets_count ?? 0} />
</Col>
<Col xs={24} sm={12} lg={6}>
<StatCard icon={<FileTextOutlined />} color="green" title="评测场景" value={counts.scenarios} />
<Col xs={12} sm={8} xl={4}>
<StatCard icon={<FileTextOutlined />} color="green" title="评测场景" value={stats?.scenarios_count ?? 0} />
</Col>
<Col xs={24} sm={12} lg={6}>
<StatCard icon={<PlayCircleOutlined />} color="orange" title="评测执行" value={counts.runs} />
<Col xs={12} sm={8} xl={4}>
<StatCard icon={<CloudServerOutlined />} color="purple" title="模型配置" value={stats?.model_configs_count ?? 0} />
</Col>
<Col xs={24} sm={12} lg={6}>
<Col xs={12} sm={8} xl={4}>
<StatCard
icon={<PlayCircleOutlined />} color="orange" title="累计执行"
value={stats?.runs_count ?? 0}
suffix={stats?.today_runs ? ` / 今日 ${stats.today_runs}` : undefined}
/>
</Col>
<Col xs={12} sm={8} xl={4}>
<StatCard
icon={<SyncOutlined spin={(stats?.running_count ?? 0) > 0} />}
color={(stats?.running_count ?? 0) > 0 ? 'blue' : 'green'}
title="运行中"
value={stats?.running_count ?? 0}
valueStyle={(stats?.running_count ?? 0) > 0 ? { color: '#1677ff' } : undefined}
/>
</Col>
<Col xs={12} sm={8} xl={4}>
<StatCard
icon={<CheckCircleOutlined />}
color="purple"
color={passRate != null && passRate >= 0.8 ? 'green' : 'red'}
title="平均通过率"
value={passRate != null ? passRate * 100 : 0}
suffix="%"
value={passRate != null ? passRate * 100 : '-'}
suffix={passRate != null ? '%' : undefined}
precision={1}
valueStyle={{ color: passRate != null && passRate >= 0.8 ? '#52c41a' : '#ff4d4f' }}
valueStyle={{ color: passRate != null ? passRateColor(passRate) : colors.textMuted }}
/>
</Col>
</Row>
{/* 图表行 */}
<Row gutter={[16, 16]} style={{ marginTop: 16 }}>
<Col xs={24} lg={14}>
<Card title="最近评测记录">
<Table
dataSource={recentRuns}
columns={columns}
rowKey="id"
size="small"
pagination={false}
/>
</Card>
</Col>
<Col xs={24} lg={10}>
<Card title="通过率趋势">
<Card title="通过率趋势(近 30 天)" size="small" styles={{ body: { minHeight: 260 } }}>
{trend.length > 0 ? (
<Line {...trendConfig} />
) : (
<div style={{ height: 200, display: 'flex', alignItems: 'center', justifyContent: 'center', color: '#9ca3af' }}>
暂无趋势数据
</div>
<Empty style={{ paddingTop: 60 }} description="暂无趋势数据" />
)}
</Card>
</Col>
<Col xs={24} lg={10}>
<Card title="场景表现" size="small" styles={{ body: { minHeight: 260, overflowY: 'auto', maxHeight: 300 } }}>
{stats?.scenario_stats.length ? stats.scenario_stats.map((s) => (
<div
key={s.scenario_id}
onClick={() => navigate('/reports')}
style={{ padding: '8px 4px', cursor: 'pointer', borderBottom: `1px solid ${colors.border}` }}
>
<div style={{ display: 'flex', justifyContent: 'space-between', marginBottom: 4 }}>
<span style={{ fontSize: 13, fontWeight: 500 }}>{s.scenario_name}</span>
<span style={{ fontSize: 12, color: colors.textSecondary }}>
{s.run_count} 次 · {s.last_run_at ? formatDateTime(s.last_run_at) : '-'}
</span>
</div>
<Progress
percent={s.avg_pass_rate != null ? Math.round(s.avg_pass_rate * 100) : 0}
size="small"
strokeColor={s.avg_pass_rate != null ? passRateColor(s.avg_pass_rate) : colors.textMuted}
/>
</div>
)) : <Empty style={{ paddingTop: 60 }} description="暂无已完成的评测" />}
</Card>
</Col>
</Row>
{/* 动态行 */}
<Row gutter={[16, 16]} style={{ marginTop: 16 }}>
<Col xs={24} lg={14}>
<Card title="最近评测记录" size="small">
{stats?.recent_runs.length ? stats.recent_runs.map((r) => (
<RecentRunRow key={r.id} run={r} onOpen={() => navigate(r.status === 'completed' ? `/reports?run=${r.id}` : '/runs')} />
)) : <Empty description="还没有评测记录" />}
</Card>
</Col>
<Col xs={24} lg={10}>
<Card title="快捷操作" size="small" style={{ marginBottom: 16 }}>
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr 1fr', gap: 8 }}>
<Button icon={<PlayCircleOutlined />} onClick={() => navigate('/runs')}>发起评测</Button>
<Button icon={<PlusOutlined />} onClick={() => navigate('/scenarios')}>新建场景</Button>
<Button icon={<RobotOutlined />} onClick={() => navigate('/openclaw')}>AI 助手</Button>
</div>
</Card>
<Card title="触发来源分布" size="small">
{stats && Object.keys(stats.trigger_breakdown).length > 0 ? (
<div style={{ display: 'flex', flexDirection: 'column', gap: 10 }}>
{(Object.entries(stats.trigger_breakdown) as Array<[string, number]>).map(([k, count]) => (
<div key={k} style={{ display: 'flex', alignItems: 'center', gap: 8 }}>
<Tag color={triggerColors[k] ?? 'default'} style={{ margin: 0, width: 72, textAlign: 'center' }}>
{triggerLabels[k] ?? k}
</Tag>
<Progress
percent={stats.runs_count ? Math.round((count / stats.runs_count) * 100) : 0}
size="small"
style={{ flex: 1, margin: 0 }}
/>
<span style={{ fontSize: 12, color: colors.textSecondary, width: 40, textAlign: 'right' }}>{count} 次</span>
</div>
))}
</div>
) : <Empty image={Empty.PRESENTED_IMAGE_SIMPLE} description="暂无数据" />}
</Card>
</Col>
</Row>
</Spin>
</div>
</PageWrapper>
)
}
function RecentRunRow({ run, onOpen }: { run: Run; onOpen: () => void }) {
const rate = (run.summary as Record<string, unknown>)?.pass_rate as number | undefined
const dotColor = statusColors[run.status] ?? colors.textMuted
const trigger = run.triggered_by ?? 'manual'
return (
<div
className="run-row"
onClick={onOpen}
style={{
display: 'flex', alignItems: 'center', gap: 10,
padding: '8px 4px', cursor: 'pointer',
borderBottom: `1px solid ${colors.border}`,
}}
>
<span style={{ width: 8, height: 8, borderRadius: '50%', background: dotColor, flexShrink: 0 }} />
<div style={{ flex: 1, minWidth: 0, overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap', fontSize: 13 }}>
<span style={{ fontWeight: 500 }}>{run.scenario_name || run.scenario_id.slice(0, 8)}</span>
<span style={{ color: colors.textMuted }}> · {run.target_name || run.target_id.slice(0, 8)}</span>
</div>
{trigger !== 'manual' && (
<Tag color={triggerColors[trigger] ?? 'default'} style={{ margin: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}>
{triggerLabels[trigger] ?? trigger}
</Tag>
)}
<span style={{ fontSize: 12, color: dotColor, width: 48, flexShrink: 0 }}>{statusLabels[run.status] ?? run.status}</span>
<span style={{
fontSize: 12, fontWeight: 600, width: 44, textAlign: 'right', flexShrink: 0,
color: rate != null ? passRateColor(rate) : colors.textMuted,
}}>
{rate != null ? `${Math.round(rate * 100)}%` : '-'}
</span>
<span style={{ fontSize: 12, color: colors.textMuted, width: 130, textAlign: 'right', flexShrink: 0 }}>
{run.started_at ? formatDateTime(run.started_at) : '-'}
</span>
<BarChartOutlined style={{ color: colors.textMuted }} />
</div>
)
}

View File

@ -0,0 +1,52 @@
import { useState } from 'react'
import { Button, Card, Form, Input, message } from 'antd'
import { LockOutlined } from '@ant-design/icons'
import { authApi, authToken } from '../api'
import { colors } from '../tokens'
export default function LoginPage({ onSuccess }: { onSuccess: () => void }) {
const [loading, setLoading] = useState(false)
const submit = async ({ password }: { password: string }) => {
setLoading(true)
try {
const res = await authApi.login(password)
authToken.set(res.data.token)
message.success('登录成功')
onSuccess()
} catch {
// 错误提示由 axios 拦截器统一弹出(如"密码错误")
} finally {
setLoading(false)
}
}
return (
<div style={{
height: '100vh', display: 'flex', alignItems: 'center', justifyContent: 'center',
background: `linear-gradient(160deg, ${colors.sider} 0%, #2e3250 100%)`,
}}>
<Card style={{ width: 360, boxShadow: '0 8px 32px rgba(0,0,0,0.35)' }}>
<div style={{ textAlign: 'center', marginBottom: 24 }}>
<div style={{ fontSize: 22, fontWeight: 700, color: colors.text }}>AgentEvalTool</div>
<div style={{ fontSize: 13, color: colors.textSecondary, marginTop: 4 }}>
智能体质量评估平台
</div>
</div>
<Form onFinish={submit} autoComplete="off">
<Form.Item name="password" rules={[{ required: true, message: '请输入访问密码' }]}>
<Input.Password
size="large"
prefix={<LockOutlined style={{ color: colors.textMuted }} />}
placeholder="访问密码"
autoFocus
/>
</Form.Item>
<Button type="primary" htmlType="submit" size="large" block loading={loading}>
登录
</Button>
</Form>
</Card>
</div>
)
}

View File

@ -33,13 +33,6 @@ const modalityLabels: Record<ModelModality, string> = {
video: '视频',
}
const modalityColors: Record<ModelModality, string> = {
text: 'default',
image: 'cyan',
audio: 'magenta',
video: 'purple',
}
const modalityOptions = Object.entries(modalityLabels).map(([value, label]) => ({ value, label }))
const featureLabels: Array<[keyof Pick<
@ -54,6 +47,13 @@ const featureLabels: Array<[keyof Pick<
const formatTokens = (value: number | null) => value ? new Intl.NumberFormat('zh-CN').format(value) : '-'
const compactTokens = (value: number | null) => {
if (!value) return '-'
if (value >= 1000000 && value % 100000 === 0) return `${value / 1000000}M`
if (value >= 1000) return `${Math.round(value / 1000)}K`
return String(value)
}
const protocolOptions: Record<ModelProtocol, {
label: string
capabilities: ModelCapability[]
@ -207,7 +207,8 @@ export default function ModelConfigsPage() {
const columns = [
{
title: '配置名称', dataIndex: 'name', key: 'name', width: 220,
// 配置名称 + 默认/厂商/区域 + Endpoint 摘要
title: '配置名称', dataIndex: 'name', key: 'name',
render: (value: string, record: ModelConfig) => (
<div style={{ minWidth: 0 }}>
<Space size={6}>
@ -216,84 +217,96 @@ export default function ModelConfigsPage() {
</Tooltip>
{record.is_default && <Tag color="gold" style={{ margin: 0 }}>默认</Tag>}
</Space>
{(record.vendor_name || record.region) && (
<div style={{ color: '#8c8c8c', fontSize: 12, marginTop: 3 }}>
{[record.vendor_name, record.region].filter(Boolean).join(' · ')}
<Tooltip title={record.endpoint_url}>
<div style={{
color: '#8c8c8c', fontSize: 12, marginTop: 3,
overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap', maxWidth: 260,
}}>
{[record.vendor_name, record.region].filter(Boolean).join(' · ') || record.endpoint_url}
</div>
)}
</Tooltip>
</div>
),
},
{
title: '协议', dataIndex: 'provider', key: 'provider', width: 150,
render: (value: ModelProtocol) => <Tag>{protocolOptions[value]?.label || value}</Tag>,
},
{
title: '能力', dataIndex: 'capability', key: 'capability', width: 110,
render: (value: ModelCapability) => <Tag color={capabilityColors[value]}>{capabilityLabels[value]}</Tag>,
},
{
title: '输入 / 输出', key: 'modalities', width: 230,
// 模型名 + 协议
title: '模型 / 协议', key: 'model', width: 200,
render: (_: unknown, record: ModelConfig) => (
<Space direction="vertical" size={3}>
<Space size={4} wrap>
<span style={{ color: '#8c8c8c', fontSize: 12, width: 28 }}>输入</span>
{record.input_modalities.map((value) => (
<Tag key={`input-${value}`} color={modalityColors[value]} style={{ margin: 0 }}>
{modalityLabels[value]}
</Tag>
))}
</Space>
<Space size={4} wrap>
<span style={{ color: '#8c8c8c', fontSize: 12, width: 28 }}>输出</span>
{record.output_modalities.map((value) => (
<Tag key={`output-${value}`} color={modalityColors[value]} style={{ margin: 0 }}>
{modalityLabels[value]}
</Tag>
))}
</Space>
</Space>
),
},
{
title: '模型', dataIndex: 'model_name', key: 'model_name', width: 180, ellipsis: true,
render: (value: string | null) => value || '-',
},
{
title: '规格', key: 'limits', width: 170,
render: (_: unknown, record: ModelConfig) => (
<div style={{ fontSize: 12, lineHeight: 1.7 }}>
<div><span style={{ color: '#8c8c8c' }}>上下文 </span>{formatTokens(record.context_window)}</div>
<div><span style={{ color: '#8c8c8c' }}>最大输出 </span>{formatTokens(record.max_output_tokens)}</div>
<div style={{ minWidth: 0 }}>
<div style={{
fontWeight: 500, overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap', maxWidth: 180,
}}>
{record.model_name || '-'}
</div>
<div style={{ color: '#8c8c8c', fontSize: 12, marginTop: 3 }}>
{protocolOptions[record.provider]?.label || record.provider}
</div>
</div>
),
},
{
title: '特性', key: 'features', width: 210,
// 能力 + 输入/输出模态
title: '能力 / 模态', key: 'capability', width: 200,
render: (_: unknown, record: ModelConfig) => {
const enabledFeatures = featureLabels.filter(([key]) => record[key])
return enabledFeatures.length ? (
<Space size={[4, 4]} wrap>
{enabledFeatures.map(([key, label]) => <Tag key={key} style={{ margin: 0 }}>{label}</Tag>)}
const modalityLine = (values: ModelModality[]) =>
values.map((v) => modalityLabels[v]).join('、')
return (
<Space size={6} align="start">
<Tag color={capabilityColors[record.capability]} style={{ margin: 0 }}>
{capabilityLabels[record.capability]}
</Tag>
<span style={{ color: '#8c8c8c', fontSize: 12, lineHeight: '22px' }}>
{modalityLine(record.input_modalities)} → {modalityLine(record.output_modalities)}
</span>
</Space>
) : '-'
)
},
},
{ title: 'Endpoint', dataIndex: 'endpoint_url', key: 'endpoint_url', width: 260, ellipsis: true },
{
title: '凭据', dataIndex: 'has_api_key', key: 'has_api_key', width: 90,
render: (value: boolean) => value ? <Tag color="success">已配置</Tag> : <Tag>无</Tag>,
// 上下文/最大输出 + 特性
title: '规格 / 特性', key: 'spec', width: 210,
render: (_: unknown, record: ModelConfig) => {
const enabledFeatures = featureLabels.filter(([key]) => record[key])
return (
<div style={{ fontSize: 12 }}>
<Tooltip title={`上下文 ${formatTokens(record.context_window)} / 最大输出 ${formatTokens(record.max_output_tokens)}`}>
<span>
<span style={{ color: '#8c8c8c' }}>上下文 </span>{compactTokens(record.context_window)}
<span style={{ color: '#8c8c8c' }}> 输出 </span>{compactTokens(record.max_output_tokens)}
</span>
</Tooltip>
{enabledFeatures.length > 0 && (
<div style={{ marginTop: 4, display: 'flex', gap: 4, flexWrap: 'wrap' }}>
{enabledFeatures.map(([key, label]) => (
<Tag key={key} style={{ margin: 0, fontSize: 11, lineHeight: '18px', padding: '0 5px' }}>{label}</Tag>
))}
</div>
)}
</div>
)
},
},
{
title: '状态', dataIndex: 'enabled', key: 'enabled', width: 90,
render: (value: boolean) => value ? <Tag color="success">启用</Tag> : <Tag>停用</Tag>,
// 状态 + 凭据 + 更新时间
title: '状态', key: 'status', width: 150,
render: (_: unknown, record: ModelConfig) => (
<div>
<Space size={4}>
{record.enabled ? <Tag color="success" style={{ margin: 0 }}>启用</Tag> : <Tag style={{ margin: 0 }}>停用</Tag>}
{record.has_api_key
? <Tag color="blue" style={{ margin: 0 }}>有凭据</Tag>
: <Tag style={{ margin: 0 }}>无凭据</Tag>}
</Space>
<Tooltip title={`更新于 ${formatDateTime(record.updated_at)}`}>
<div style={{ color: '#8c8c8c', fontSize: 12, marginTop: 3 }}>
{formatDateTime(record.updated_at)}
</div>
</Tooltip>
</div>
),
},
{
title: '更新时间', dataIndex: 'updated_at', key: 'updated_at', width: 170,
render: (value: string | null) => formatDateTime(value),
},
{
title: '操作', key: 'actions', width: 145, fixed: 'right' as const,
title: '操作', key: 'actions', width: 110,
render: (_: unknown, record: ModelConfig) => (
<Space size={2}>
<Tooltip title="连接测试">
@ -355,8 +368,9 @@ export default function ModelConfigsPage() {
rowKey="id"
loading={loading}
pagination={{ pageSize: 15, showSizeChanger: true, showTotal: (total) => `共 ${total} 个` }}
scroll={{ x: 1800, y: 'calc(100vh - 200px)' }}
scroll={{ y: 'calc(100vh - 200px)' }}
style={{ height: '100%' }}
tableLayout="auto"
/>
</div>

View File

@ -1,17 +1,18 @@
import { useEffect, useState } from 'react'
import { useEffect, useMemo, useState } from 'react'
import { useSearchParams } from 'react-router-dom'
import {
Button, Card, Col, Collapse, Descriptions, Empty, Row, Segmented,
Select, Space, Spin, Statistic, Table, Tag, Tooltip, Badge,
Select, Space, Spin, Statistic, Table, Tag, Tooltip, Badge, message,
} from 'antd'
import {
CheckCircleOutlined, CloseCircleOutlined,
CheckCircleOutlined, CloseCircleOutlined, ClearOutlined,
DownloadOutlined, UserOutlined, RobotOutlined, ReloadOutlined,
DiffOutlined, FileMarkdownOutlined,
} from '@ant-design/icons'
import { reportsApi, runsApi, type Run } from '../api'
import { colors } from '../tokens'
import { colors, triggerColors, triggerLabels } from '../tokens'
import { formatDateTime } from '../utils/date'
import { useOnTabActive } from '../hooks/useOnTabActive'
interface TurnData {
sent_text: string
@ -72,6 +73,7 @@ export default function ReportsPage() {
const runQuery = searchParams.get('run') ?? ''
const [runs, setRuns] = useState<Run[]>([])
const [scenarioFilter, setScenarioFilter] = useState<string>('')
const [selectedRunId, setSelectedRunId] = useState<string>('')
const [compareRunId, setCompareRunId] = useState<string>('')
const [report, setReport] = useState<Report | null>(null)
@ -93,8 +95,8 @@ export default function ReportsPage() {
const loadRuns = async () => {
const res = await runsApi.list()
const completed = res.data.filter((r) => r.status === 'completed').reverse()
setRuns(completed)
// API 已按 started_at DESC 排序,最新的在前
setRuns(res.data.filter((r) => r.status === 'completed'))
}
useEffect(() => {
@ -104,6 +106,10 @@ export default function ReportsPage() {
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [])
// Keep-alive tabs never remount: refresh whenever this tab is re-activated
// so runs triggered elsewhere (e.g. AI assistant) show up.
useOnTabActive('/reports', loadRuns)
useEffect(() => {
if (runQuery && runQuery !== selectedRunId) {
loadReport(runQuery)
@ -114,6 +120,10 @@ export default function ReportsPage() {
const handleView = (runId: string) => {
setSearchParams({ run: runId })
loadReport(runId)
// 对比报告要求同场景:A 变更后若 B 场景不同则清空
const a = runs.find((r) => r.id === runId)
const b = runs.find((r) => r.id === compareRunId)
if (a && b && a.scenario_id !== b.scenario_id) setCompareRunId('')
}
const handleCompare = async () => {
@ -122,15 +132,82 @@ export default function ReportsPage() {
try {
const res = await reportsApi.compare(selectedRunId, compareRunId)
setCompareResult(res.data as CompareResult)
} catch (e: any) {
const detail = e?.response?.data?.detail
if (detail) message.error(detail)
} finally {
setLoading(false)
}
}
const runSelectOptions = runs.map((r) => ({
value: r.id,
label: `${r.id.slice(0, 8)}… | ${r.started_at ? formatDateTime(r.started_at) : ''}`,
}))
const clearSelection = () => {
setSelectedRunId('')
setCompareRunId('')
setReport(null)
setCompareResult(null)
setSearchParams({})
}
const resetFilters = () => {
setScenarioFilter('')
clearSelection()
}
// 场景筛选项:从 run 列表聚合(含各场景 run 数量)
const scenarioOptions = useMemo(() => {
const map = new Map<string, { name: string; count: number }>()
for (const r of runs) {
const entry = map.get(r.scenario_id)
if (entry) entry.count += 1
else map.set(r.scenario_id, { name: r.scenario_name || r.scenario_id.slice(0, 8), count: 1 })
}
return [...map.entries()].map(([id, s]) => ({ value: id, label: `${s.name}(${s.count})` }))
}, [runs])
const filteredRuns = useMemo(
() => (scenarioFilter ? runs.filter((r) => r.scenario_id === scenarioFilter) : runs),
[runs, scenarioFilter],
)
const buildOption = (r: Run) => {
const passRate = (r.summary as Record<string, unknown>)?.pass_rate as number | undefined
const pct = passRate != null ? `${Math.round(passRate * 100)}%` : '-'
const scenario = r.scenario_name || r.scenario_id.slice(0, 8)
const target = r.target_name || r.target_id.slice(0, 8)
const time = r.started_at ? formatDateTime(r.started_at) : ''
const trigger = r.triggered_by ?? 'manual'
return {
value: r.id,
searchText: `${scenario} ${target} ${time} ${r.id} ${triggerLabels[trigger] ?? trigger}`.toLowerCase(),
label: (
<span style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
<span style={{ fontWeight: 500, overflow: 'hidden', textOverflow: 'ellipsis' }}>{scenario}</span>
<span style={{ color: colors.textMuted }}>· {target}</span>
<span style={{ color: colors.textMuted, fontSize: 12 }}>{time}</span>
<span style={{
color: passRate != null && passRate >= 0.8 ? '#3f8600' : '#cf1322',
fontSize: 12, fontWeight: 600,
}}>{pct}</span>
{trigger !== 'manual' && (
<Tag color={triggerColors[trigger] ?? 'default'} style={{ marginRight: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}>
{triggerLabels[trigger] ?? trigger}
</Tag>
)}
</span>
),
}
}
const runSelectOptions = filteredRuns.map(buildOption)
// 报告 B 只能选与报告 A 同场景的 run
const selectedRun = runs.find((r) => r.id === selectedRunId)
const compareOptions = runs
.filter((r) => r.id !== selectedRunId && selectedRun && r.scenario_id === selectedRun.scenario_id)
.map(buildOption)
const optionFilter = (input: string, opt?: { searchText?: string }) =>
(opt?.searchText ?? '').includes(input.toLowerCase())
return (
<div style={{ height: '100%', display: 'flex', flexDirection: 'column', overflow: 'hidden' }}>
@ -144,72 +221,96 @@ export default function ReportsPage() {
<span style={{ fontSize: 13, color: colors.textSecondary }}>查看评测结果详情与统计分析</span>
</div>
{/* 选择栏 */}
{/* 选择栏:两行布局,避免对比模式下控件换行错乱 */}
<div style={{
padding: '8px 16px', flexShrink: 0,
borderBottom: `1px solid ${colors.border}`,
background: colors.bgSubtle,
display: 'flex', alignItems: 'center', gap: 12, flexWrap: 'wrap',
display: 'flex', flexDirection: 'column', gap: 8,
}}>
<Segmented
value={viewMode}
onChange={(v) => { setViewMode(v as ViewMode); setCompareResult(null) }}
options={[
{ label: '单次报告', value: 'single' },
{ label: '对比报告', value: 'compare', icon: <DiffOutlined /> },
]}
/>
{/* 第一行:视图模式 + 场景筛选 + 重置/刷新 */}
<div style={{ display: 'flex', alignItems: 'center', gap: 12 }}>
<Segmented
value={viewMode}
onChange={(v) => { setViewMode(v as ViewMode); setCompareResult(null) }}
options={[
{ label: '单次报告', value: 'single' },
{ label: '对比报告', value: 'compare', icon: <DiffOutlined /> },
]}
/>
<Select
style={{ width: 320 }}
value={scenarioFilter || undefined}
placeholder={`全部场景(${runs.length})`}
allowClear
onChange={(v) => setScenarioFilter(v ?? '')}
options={scenarioOptions}
popupMatchSelectWidth={false}
/>
<div style={{ marginLeft: 'auto', display: 'flex', gap: 8 }}>
<Tooltip title="重置筛选与选择">
<Button size="middle" icon={<ClearOutlined />} onClick={resetFilters}>重置</Button>
</Tooltip>
<Tooltip title="刷新列表">
<Button size="middle" icon={<ReloadOutlined />} onClick={loadRuns} />
</Tooltip>
</div>
</div>
<Select
style={{ width: 340 }}
placeholder={viewMode === 'compare' ? '选择报告 A' : '选择已完成的评测记录'}
value={selectedRunId || undefined}
onChange={handleView}
showSearch
filterOption={(input, opt) =>
(opt?.label as string ?? '').toLowerCase().includes(input.toLowerCase())}
options={runSelectOptions}
/>
{/* 第二行:报告选择 + 操作 */}
<div style={{ display: 'flex', alignItems: 'center', gap: 8 }}>
<Select
style={{ flex: 1, minWidth: 320, maxWidth: 680 }}
placeholder={viewMode === 'compare' ? '选择报告 A' : '选择已完成的评测记录'}
value={selectedRunId || undefined}
onChange={(v) => (v ? handleView(v) : clearSelection())}
allowClear
showSearch
filterOption={optionFilter}
options={runSelectOptions}
popupMatchSelectWidth={false}
/>
{viewMode === 'compare' && (
<>
<span style={{ color: colors.textMuted, fontSize: 12 }}>vs</span>
<Select
style={{ width: 340 }}
placeholder="选择报告 B"
value={compareRunId || undefined}
onChange={setCompareRunId}
showSearch
filterOption={(input, opt) =>
(opt?.label as string ?? '').toLowerCase().includes(input.toLowerCase())}
options={runSelectOptions.filter((o) => o.value !== selectedRunId)}
/>
<Button
type="primary"
icon={<DiffOutlined />}
disabled={!selectedRunId || !compareRunId}
onClick={handleCompare}
>
对比
</Button>
</>
)}
{viewMode === 'compare' && (
<>
<span style={{ color: colors.textMuted, fontSize: 12, flexShrink: 0 }}>vs</span>
<Tooltip title={selectedRunId ? '仅可选择与报告 A 相同场景的记录' : '请先选择报告 A'}>
<Select
style={{ flex: 1, minWidth: 320, maxWidth: 680 }}
placeholder="选择报告 B(同场景)"
value={compareRunId || undefined}
onChange={(v) => { setCompareRunId(v ?? ''); if (!v) setCompareResult(null) }}
allowClear
showSearch
disabled={!selectedRunId}
filterOption={optionFilter}
options={compareOptions}
popupMatchSelectWidth={false}
notFoundContent={<Empty image={Empty.PRESENTED_IMAGE_SIMPLE} description="没有同场景的其他评测记录" />}
/>
</Tooltip>
<Button
type="primary"
icon={<DiffOutlined />}
disabled={!selectedRunId || !compareRunId}
onClick={handleCompare}
style={{ flexShrink: 0 }}
>
对比
</Button>
</>
)}
{viewMode === 'single' && report && (
<Space>
<Button icon={<DownloadOutlined />} onClick={() => window.open(`/api/reports/${selectedRunId}/html`, '_blank')}>
导出 HTML
</Button>
<Button icon={<FileMarkdownOutlined />} onClick={() => window.open(reportsApi.markdownUrl(selectedRunId), '_blank')}>
导出 MD
</Button>
</Space>
)}
<div style={{ marginLeft: 'auto' }}>
<Tooltip title="刷新列表">
<Button size="middle" icon={<ReloadOutlined />} onClick={loadRuns} />
</Tooltip>
{viewMode === 'single' && report && (
<Space style={{ flexShrink: 0 }}>
<Button icon={<DownloadOutlined />} onClick={() => window.open(`/api/reports/${selectedRunId}/html`, '_blank')}>
导出 HTML
</Button>
<Button icon={<FileMarkdownOutlined />} onClick={() => window.open(reportsApi.markdownUrl(selectedRunId), '_blank')}>
导出 MD
</Button>
</Space>
)}
</div>
</div>

View File

@ -13,6 +13,7 @@ import RuleOverview from '../components/RuleOverview'
import CaseDetail from '../components/CaseDetail'
import { useRunSession } from '../hooks/useRunSession'
import { useTicker } from '../hooks/useTicker'
import { useOnTabActive } from '../hooks/useOnTabActive'
import { colors, statusColors, statusLabels } from '../tokens'
import { formatDateTime, elapsedStr } from '../utils/date'
import { passRateColor } from '../utils/colors'
@ -52,6 +53,10 @@ export default function RunsPage() {
useEffect(() => { loadLists() }, [])
// Keep-alive tabs never remount: refresh the list whenever the tab is
// re-activated so runs triggered elsewhere (e.g. AI assistant) show up.
useOnTabActive('/runs', loadLists)
useEffect(() => {
if (!session.isLive) return
const id = window.setInterval(() => {

View File

@ -43,6 +43,18 @@ export const statusLabels: Record<string, string> = {
cancelled: '已取消',
}
export const triggerLabels: Record<string, string> = {
manual: '手动',
ai_assistant: 'AI 助手',
cli: 'CLI',
}
export const triggerColors: Record<string, string> = {
manual: 'default',
ai_assistant: 'purple',
cli: 'blue',
}
export const spacing = {
xs: 4,
sm: 8,

View File

@ -22,6 +22,14 @@ export default defineConfig({
},
},
build: {
// Accepted oversized chunks (measured 2026-07-28):
// vendor ~2.12 MB min / ~650 kB gzip — react + @ant-design/charts
// (charts CANNOT be split out: forced separate chunk creates
// a circular reference and a runtime TDZ crash, and charts
// loads on the default Home route anyway)
// vendor-antd ~620 kB min / ~173 kB gzip
// Warning fires only above this limit, i.e. on a real size regression.
chunkSizeWarningLimit: 2200,
rollupOptions: {
output: {
manualChunks(id) {
@ -37,8 +45,8 @@ export default defineConfig({
// NOTE: do NOT split @ant-design/charts into its own chunk — its deep
// @antv dependency graph creates a circular chunk reference that causes
// a runtime TDZ crash ("Cannot access 'd' before initialization").
// charts is loaded on the Home page (default route) anyway, so a
// separate chunk buys almost nothing.
// Returning undefined for @antv doesn't help either: rollup still
// merges it into vendor (verified 2026-07-28).
if (id.includes('node_modules')) {
return 'vendor'
}

View File

@ -0,0 +1,29 @@
"""add triggered_by to eval_runs
Revision ID: b7d4e6f81c22
Revises: a64b2f8c9d10
Create Date: 2026-07-27
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "b7d4e6f81c22"
down_revision: Union[str, Sequence[str], None] = "a64b2f8c9d10"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("eval_runs") as batch_op:
batch_op.add_column(
sa.Column("triggered_by", sa.String(), nullable=False, server_default="manual")
)
def downgrade() -> None:
with op.batch_alter_table("eval_runs") as batch_op:
batch_op.drop_column("triggered_by")

View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "agenteval"
version = "0.3.0-dev"
version = "0.4.0-dev"
description = "智能体质量评估工具集平台"
readme = "README.md"
requires-python = ">=3.10"

View File

@ -92,6 +92,23 @@ run rsync -az --delete \
--exclude='*.db*' \
./ "$HOST:$REMOTE_DIR/"
# ── OpenClaw skill sync ────────────────────────────────────────────────
# The agenteval-run skill lives in the openclaw workspace volume (data/),
# which rsync excludes. Sync the versioned copy explicitly so the AI
# assistant always follows the standard HTTP API flow.
SKILL_SRC="backend/plugins/openclaw/skills/agenteval-run/SKILL.md"
SKILL_DST="$REMOTE_DIR/data/openclaw/workspace/skills/agenteval-run/SKILL.md"
log "sync OpenClaw skill → $SKILL_DST"
run ssh "$HOST" "mkdir -p $(dirname "$SKILL_DST")"
run rsync -az "$SKILL_SRC" "$HOST:$SKILL_DST"
# Provision the API key file the skill reads (~/.openclaw/agenteval-api-key in
# the openclaw container = data/openclaw/agenteval-api-key on the host). Sourced
# from AGENTEVAL_API_KEY in the remote .env; removed when the key is unset.
run ssh "$HOST" "KEY=\$(grep -E '^AGENTEVAL_API_KEY=.+' $REMOTE_DIR/.env 2>/dev/null | cut -d= -f2-); \
if [ -n \"\$KEY\" ]; then printf '%s' \"\$KEY\" > $REMOTE_DIR/data/openclaw/agenteval-api-key; \
else rm -f $REMOTE_DIR/data/openclaw/agenteval-api-key; fi"
# ── image rebuild ──────────────────────────────────────────────────────
if [[ "$SKIP_BUILD" == "1" ]]; then
warn "skipping image rebuild (--skip-build)"

View File

@ -0,0 +1,89 @@
"""Integration tests for the login gate: /api/auth/* and require_auth."""
import pytest
from fastapi.testclient import TestClient
from sqlmodel import Session, SQLModel, create_engine
from agenteval.config.settings import Settings
from agenteval.web.app import app
from agenteval.web.deps import get_db
@pytest.fixture()
def client_with_db(tmp_path):
from agenteval.storage.db import ( # noqa: F401
EvalResultDB, EvalRunDB, EvalTargetDB, ScenarioDB, TurnDB,
)
engine = create_engine(
f"sqlite:///{tmp_path / 'auth_api.db'}",
connect_args={"check_same_thread": False},
)
SQLModel.metadata.create_all(engine)
session = Session(engine)
def override_get_db():
try:
yield session
finally:
pass
app.dependency_overrides[get_db] = override_get_db
client = TestClient(app)
yield client
app.dependency_overrides.clear()
session.close()
def _patch_settings(monkeypatch, **kwargs):
settings = Settings(_env_file=None, **kwargs)
from agenteval.web import deps as deps_module
from agenteval.web.routers import auth as auth_module
monkeypatch.setattr(deps_module, "get_settings", lambda: settings)
monkeypatch.setattr(auth_module, "get_settings", lambda: settings)
return settings
def test_no_credentials_configured_is_open(client_with_db, monkeypatch):
_patch_settings(monkeypatch)
assert client_with_db.get("/api/targets").status_code == 200
status = client_with_db.get("/api/auth/status").json()
assert status == {"auth_required": False, "authenticated": True}
def test_login_disabled_returns_400(client_with_db, monkeypatch):
_patch_settings(monkeypatch)
resp = client_with_db.post("/api/auth/login", json={"password": "whatever"})
assert resp.status_code == 400
def test_admin_password_gates_api(client_with_db, monkeypatch):
_patch_settings(monkeypatch, admin_password="s3cret")
# No credentials → 401.
assert client_with_db.get("/api/targets").status_code == 401
# Wrong password → 401.
assert client_with_db.post("/api/auth/login", json={"password": "nope"}).status_code == 401
# Correct password → token that unlocks the API.
token = client_with_db.post("/api/auth/login", json={"password": "s3cret"}).json()["token"]
resp = client_with_db.get("/api/targets", headers={"X-Auth-Token": token})
assert resp.status_code == 200
status = client_with_db.get("/api/auth/status", headers={"X-Auth-Token": token}).json()
assert status == {"auth_required": True, "authenticated": True}
assert client_with_db.get("/api/auth/status").json()["authenticated"] is False
def test_api_key_still_works_alongside_login(client_with_db, monkeypatch):
_patch_settings(monkeypatch, admin_password="s3cret", api_key="machine-key")
assert client_with_db.get("/api/targets").status_code == 401
assert client_with_db.get("/api/targets", headers={"X-API-Key": "machine-key"}).status_code == 200
assert client_with_db.get("/api/targets", headers={"X-API-Key": "wrong"}).status_code == 401
def test_health_stays_open(client_with_db, monkeypatch):
_patch_settings(monkeypatch, admin_password="s3cret")
assert client_with_db.get("/api/health").status_code == 200

View File

@ -39,7 +39,7 @@ def client_with_db(tmp_path):
session.close()
def _seed_run(session: Session, name_suffix: str = "") -> str:
def _seed_run(session: Session, name_suffix: str = "", scenario_id: str | None = None) -> str:
target = EvalTarget(
name=f"target{name_suffix}",
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
@ -49,15 +49,17 @@ def _seed_run(session: Session, name_suffix: str = "") -> str:
)
target = TargetRepository(session).create(target)
scenario = Scenario(
name=f"scenario{name_suffix}",
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
)
scenario = ScenarioRepository(session).create(scenario)
if scenario_id is None:
scenario = Scenario(
name=f"scenario{name_suffix}",
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
)
scenario = ScenarioRepository(session).create(scenario)
scenario_id = scenario.id
run = EvalRun(
target_id=target.id,
scenario_id=scenario.id,
scenario_id=scenario_id,
status=RunStatus.COMPLETED,
)
run = RunRepository(session).create(run)
@ -143,7 +145,8 @@ def test_get_markdown_report_attachment_header(client_with_db):
def test_compare_report(client_with_db):
client, session = client_with_db
run_id_a = _seed_run(session, "A")
run_id_b = _seed_run(session, "B")
sid = RunRepository(session).get(run_id_a).scenario_id
run_id_b = _seed_run(session, "B", scenario_id=sid)
resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}")
assert resp.status_code == 200
data = resp.json()
@ -153,6 +156,15 @@ def test_compare_report(client_with_db):
assert "cases" in data
def test_compare_report_different_scenarios_400(client_with_db):
client, session = client_with_db
run_id_a = _seed_run(session, "A")
run_id_b = _seed_run(session, "B") # separate scenario
resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}")
assert resp.status_code == 400
assert "相同场景" in resp.json()["detail"]
def test_compare_report_run_not_found(client_with_db):
client, session = client_with_db
run_id = _seed_run(session)

View File

@ -196,3 +196,45 @@ async def test_start_run_missing_target(client, seeded_db):
"target_id": "does-not-exist", "scenario_id": "s-1",
})
assert resp.status_code == 404
# ── triggered_by ─────────────────────────────────────────────────────────
async def test_start_run_default_triggered_by_manual(client, seeded_db, mock_channel):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1",
})
assert resp.status_code == 200
assert resp.json()["triggered_by"] == "manual"
async def test_start_run_ai_assistant_triggered_by(client, seeded_db, mock_channel):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "ai_assistant",
})
assert resp.status_code == 200
run_id = resp.json()["id"]
assert resp.json()["triggered_by"] == "ai_assistant"
# Persisted, not just echoed.
got = (await client.get(f"/api/runs/{run_id}")).json()
assert got["triggered_by"] == "ai_assistant"
async def test_start_run_invalid_triggered_by_422(client, seeded_db):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "robot",
})
assert resp.status_code == 422
async def test_list_runs_includes_names_and_trigger(client, seeded_db, mock_channel):
await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "ai_assistant",
})
listing = (await client.get("/api/runs")).json()
assert len(listing) == 1
row = listing[0]
assert row["scenario_name"] == "mock-scenario"
assert row["target_name"] == "mock-target"
assert row["triggered_by"] == "ai_assistant"

View File

@ -0,0 +1,106 @@
"""Integration tests for /api/stats/dashboard aggregation."""
import pytest
from fastapi.testclient import TestClient
from sqlmodel import Session, SQLModel, create_engine
from agenteval.models import (
Case, CaseType, ChannelType, EvalRun, EvalTarget, PlatformType,
RunStatus, RunTrigger, Scenario, TargetStatus,
)
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from agenteval.web.app import app
from agenteval.web.deps import get_db
@pytest.fixture()
def client_with_db(tmp_path):
from agenteval.storage.db import ( # noqa: F401
EvalResultDB, EvalRunDB, EvalTargetDB, ModelConfigDB, ScenarioDB, TurnDB,
)
engine = create_engine(
f"sqlite:///{tmp_path / 'stats_api.db'}",
connect_args={"check_same_thread": False},
)
SQLModel.metadata.create_all(engine)
session = Session(engine)
def override_get_db():
try:
yield session
finally:
pass
app.dependency_overrides[get_db] = override_get_db
client = TestClient(app)
yield client, session
app.dependency_overrides.clear()
session.close()
def _seed(session: Session) -> None:
target = TargetRepository(session).create(EvalTarget(
name="对象A", platform=PlatformType.AI_DIGITAL_EMPLOYEE,
channel_type=ChannelType.TUTU_API, channel_config={}, status=TargetStatus.ACTIVE,
))
scenario = ScenarioRepository(session).create(Scenario(
name="场景A", cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
))
repo = RunRepository(session)
for pass_rate, trigger in [(1.0, RunTrigger.MANUAL), (0.5, RunTrigger.AI_ASSISTANT)]:
run = repo.create(EvalRun(
target_id=target.id, scenario_id=scenario.id,
status=RunStatus.COMPLETED, triggered_by=trigger,
))
run.summary = {"total_cases": 2, "passed_cases": 1, "failed_cases": 1,
"total_rules": 2, "passed_rules": 1, "pass_rate": pass_rate}
repo.update(run)
repo.create(EvalRun(
target_id=target.id, scenario_id=scenario.id,
status=RunStatus.RUNNING, triggered_by=RunTrigger.MANUAL,
))
def test_dashboard_aggregates(client_with_db):
client, session = client_with_db
_seed(session)
data = client.get("/api/stats/dashboard").json()
assert data["targets_count"] == 1
assert data["scenarios_count"] == 1
assert data["runs_count"] == 3
assert data["model_configs_count"] == 0
assert data["running_count"] == 1
assert data["today_runs"] == 3
assert data["overall_pass_rate"] == pytest.approx(0.75)
assert data["trigger_breakdown"] == {"manual": 2, "ai_assistant": 1}
assert len(data["scenario_stats"]) == 1
stat = data["scenario_stats"][0]
assert stat["scenario_name"] == "场景A"
assert stat["run_count"] == 2
assert stat["avg_pass_rate"] == pytest.approx(0.75)
assert stat["last_run_at"] is not None
assert len(data["recent_runs"]) == 3
assert data["recent_runs"][0]["scenario_name"] == "场景A"
assert data["recent_runs"][0]["target_name"] == "对象A"
assert "triggered_by" in data["recent_runs"][0]
def test_dashboard_empty_db(client_with_db):
client, _ = client_with_db
data = client.get("/api/stats/dashboard").json()
assert data["runs_count"] == 0
assert data["overall_pass_rate"] is None
assert data["scenario_stats"] == []
assert data["recent_runs"] == []
def test_trend_returns_daily_points(client_with_db):
client, session = client_with_db
_seed(session)
points = client.get("/api/stats/trend").json()
assert len(points) == 1
assert points[0]["run_count"] == 2
assert points[0]["pass_rate"] == pytest.approx(75.0)

View File

@ -34,7 +34,13 @@ def report_session(tmp_path):
engine.dispose()
def _seed_run(session: Session, *, pass_rate: float = 1.0, n_cases: int = 1) -> str:
def _seed_run(
session: Session,
*,
pass_rate: float = 1.0,
n_cases: int = 1,
scenario_id: str | None = None,
) -> str:
"""Create a minimal completed run with real data in the DB and return run_id."""
target = EvalTarget(
name="测试对象",
@ -45,15 +51,17 @@ def _seed_run(session: Session, *, pass_rate: float = 1.0, n_cases: int = 1) ->
)
target = TargetRepository(session).create(target)
scenario = Scenario(
name="测试场景",
cases=[Case(id=f"c{i}", type=CaseType.SINGLE, messages=["hi"]) for i in range(n_cases)],
)
scenario = ScenarioRepository(session).create(scenario)
if scenario_id is None:
scenario = Scenario(
name="测试场景",
cases=[Case(id=f"c{i}", type=CaseType.SINGLE, messages=["hi"]) for i in range(n_cases)],
)
scenario = ScenarioRepository(session).create(scenario)
scenario_id = scenario.id
run = EvalRun(
target_id=target.id,
scenario_id=scenario.id,
scenario_id=scenario_id,
status=RunStatus.COMPLETED,
)
run = RunRepository(session).create(run)
@ -148,9 +156,14 @@ def test_generate_report_not_found_raises(report_session):
# ── generate_compare_report ───────────────────────────────────────────────
def _scenario_of(session: Session, run_id: str) -> str:
return RunRepository(session).get(run_id).scenario_id
def test_compare_report_structure(report_session):
run_id_a = _seed_run(report_session, pass_rate=1.0, n_cases=2)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session)
assert "run_a" in result
@ -163,14 +176,16 @@ def test_compare_report_structure(report_session):
def test_compare_report_delta(report_session):
run_id_a = _seed_run(report_session, pass_rate=0.5, n_cases=2)
run_id_b = _seed_run(report_session, pass_rate=1.0, n_cases=2)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, pass_rate=1.0, n_cases=2, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session)
assert result["delta"]["pass_rate"] > 0 # B improved over A
def test_compare_report_changed_cases(report_session):
run_id_a = _seed_run(report_session, pass_rate=1.0, n_cases=2)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session)
# At least one case changed (A all-pass vs B half-pass)
assert result["changed_cases"] >= 1
@ -178,7 +193,8 @@ def test_compare_report_changed_cases(report_session):
def test_compare_report_case_level(report_session):
run_id_a = _seed_run(report_session, n_cases=1)
run_id_b = _seed_run(report_session, n_cases=1)
sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, n_cases=1, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session)
assert len(result["cases"]) >= 1
case = result["cases"][0]
@ -187,6 +203,13 @@ def test_compare_report_case_level(report_session):
assert "changed" in case
def test_compare_report_different_scenarios_rejected(report_session):
run_id_a = _seed_run(report_session, n_cases=1)
run_id_b = _seed_run(report_session, n_cases=1) # separate scenario
with pytest.raises(ValueError, match="same scenario"):
generate_compare_report(run_id_a, run_id_b, report_session)
# ── render_markdown_report ────────────────────────────────────────────────
def test_render_markdown_contains_header(report_session):