Compare commits

...

3 Commits

Author SHA1 Message Date
sinohqb
595409487b docs(release): v0.4「联」发布说明与里程碑收尾
Some checks failed
CI / test (push) Failing after 38s
- 新增 release-notes-v0.4.md(事故排查 + 功能总结 + v0.5 候选方向)
- AGENT.md 里程碑表更新至 v0.4 + 鉴权配置说明
- README / plan-v0.4 状态同步
2026-07-28 17:41:20 +08:00
sinohqb
9c564b575e feat(frontend): v0.4 login page, dashboard rebuild, reports UX, keep-alive refresh
- 登录页 + App 鉴权门 + X-Auth-Token 拦截器 + 菜单头部退出按钮
- 仪表盘重构:6 指标卡 / 趋势图 + 场景表现 / 最近记录 + 快捷操作 + 来源分布
- Reports 页重做:场景筛选、富选项下拉、allowClear、一键重置、同场景对比约束
- useOnTabActive:标签页激活自动刷新(根治 AI 助手评测记录"消失")
- ModelConfigs 12 列合并为 6 列;来源 Tag;chunk 告警阈值修正并记录原因
2026-07-28 17:41:08 +08:00
sinohqb
739d586aec feat(backend): v0.4 triggered_by tracking, login gate, compare guard, dashboard stats
- EvalRun.triggered_by 全链路(manual/ai_assistant/cli)+ 迁移 b7d4e6f81c22
- 标准 agenteval-run SKILL.md 纳入版本管理,deploy 脚本同步 + API Key 注入
- 简单登录:AGENTEVAL_ADMIN_PASSWORD + HMAC 会话 token,require_auth 双凭据
- 对比报告限同场景(400)+ 空 results 误判修复
- /api/stats/dashboard 扩展聚合;/api/runs 返回场景/对象名
- 测试 218 → 232
2026-07-28 17:40:54 +08:00
42 changed files with 1441 additions and 326 deletions

View File

@ -6,6 +6,10 @@
# In production, set a strong random token; clients must send it as X-API-Key. # In production, set a strong random token; clients must send it as X-API-Key.
AGENTEVAL_API_KEY= AGENTEVAL_API_KEY=
# If set, the web UI requires login with this password (session-level, stored
# in sessionStorage). Leave empty to disable the login gate (dev default).
AGENTEVAL_ADMIN_PASSWORD=
# Fernet key used to encrypt model API keys stored in SQLite. # Fernet key used to encrypt model API keys stored in SQLite.
# Generate once with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" # Generate once with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# Keep this value backed up with the database. Do not commit a real key. # Keep this value backed up with the database. Do not commit a real key.

View File

@ -6,7 +6,7 @@
**AgentEvalTool** 是智能体质量评估工具集平台,评估 AI 数字员工(大模型 + RAG)和 AI 助手(OpenClaw)的服务质量。核心闭环:定义评估 → 执行评估 → 分析结果 → 改进优化。 **AgentEvalTool** 是智能体质量评估工具集平台,评估 AI 数字员工(大模型 + RAG)和 AI 助手(OpenClaw)的服务质量。核心闭环:定义评估 → 执行评估 → 分析结果 → 改进优化。
- **当前版本**: v0.3.0-dev(2026-07-17 发布至 t480,里程碑「拓」) - **当前版本**: v0.4.0-dev(2026-07-28 发布至 t480,里程碑「联」)
- **运行环境**: t480 测试服务器(192.168.8.145:8001) - **运行环境**: t480 测试服务器(192.168.8.145:8001)
- **语言**: Python 3.11(后端) + TypeScript/React 18(前端) - **语言**: Python 3.11(后端) + TypeScript/React 18(前端)
@ -16,10 +16,11 @@
|---|---|---|---|---| |---|---|---|---|---|
| v0.1 | MVP | 2026-07-09 | 核心闭环跑通 | ✅ 已发布 | | v0.1 | MVP | 2026-07-09 | 核心闭环跑通 | ✅ 已发布 |
| v0.2 | 「稳」 | 2026-07-14 | async 引擎 + 安全基线 + 测试 + 部署规范化 | ✅ 已发布(t480) | | v0.2 | 「稳」 | 2026-07-14 | async 引擎 + 安全基线 + 测试 + 部署规范化 | ✅ 已发布(t480) |
| **v0.3** | **「拓」** | **2026-07-17** | **多通道 + 规则扩展 + 模型配置中心** | **✅ 已发布(t480)** | | v0.3 | 「拓」 | 2026-07-17 | 多通道 + 规则扩展 + 模型配置中心 | ✅ 已发布(t480) |
| v0.4 | — | 规划中 | 前端测试 / 报告增强 / OpenClaw 深度集成二期 / 多模态用例 | 📋 待规划 | | **v0.4** | **「联」** | **2026-07-28** | **AI 助手标准化(triggered_by) + 登录 + 仪表盘重构** | **✅ 已发布(t480)** |
| v0.5 | — | 规划中 | 前端测试 / 报告聚合看板 / OpenClaw 二期 / 多模态 | 📋 待规划 |
详细内容见 [docs/release-notes-v0.3.md](docs/release-notes-v0.3.md)。 详细内容见 [docs/release-notes-v0.4.md](docs/release-notes-v0.4.md)。
## 技术栈 ## 技术栈
@ -138,6 +139,12 @@ curl http://192.168.8.145:8001/api/health
- `backend/` → `/app/backend:ro`(代码热更新) - `backend/` → `/app/backend:ro`(代码热更新)
- `.env` → `/app/.env:ro`(Pydantic Settings 配置, 见 `.env.example`) - `.env` → `/app/.env:ro`(Pydantic Settings 配置, 见 `.env.example`)
### 鉴权(v0.4 起)
- `AGENTEVAL_ADMIN_PASSWORD`:设置后 Web UI 需登录(会话 token, 关浏览器失效);不设则无登录门
- `AGENTEVAL_API_KEY`:机器调用凭据(X-API-Key);deploy 脚本会把它写入 openclaw 工作区
`data/openclaw/agenteval-api-key`, agenteval-run skill 自动读取
- 两个凭据任一匹配即放行(`web/deps.py: require_auth`);`/api/health` 与 `/api/auth/*` 始终开放
### 镜像 tag 约定 ### 镜像 tag 约定
- `t480-agenteval:<version>` — 不可变, 对应某次具体构建 - `t480-agenteval:<version>` — 不可变, 对应某次具体构建
- `t480-agenteval:latest` — 可变, 总是指向最新部署 - `t480-agenteval:latest` — 可变, 总是指向最新部署

View File

@ -1,6 +1,6 @@
# AgentEvalTool # AgentEvalTool
> **当前开发版本**: v0.3.0-dev 「拓」 · 已发布至 t480 · 发布说明见 [docs/release-notes-v0.3.md](docs/release-notes-v0.3.md) > **当前开发版本**: v0.4.0-dev 「联」 · 已发布至 t480 · 发布说明见 [docs/release-notes-v0.4.md](docs/release-notes-v0.4.md)
智能体质量评估工具集平台。 智能体质量评估工具集平台。
@ -11,19 +11,23 @@
- 异步评测执行引擎(真并发 / 真取消 / 可配置超时 / WS 实时进度) - 异步评测执行引擎(真并发 / 真取消 / 可配置超时 / WS 实时进度)
- 评测通道:tutu-api / HTTP / OpenClaw - 评测通道:tutu-api / HTTP / OpenClaw
- 评估规则:关键词、响应时间、LLM 评分、语义相似度、JSON Schema、安全检测 + all/any/weighted 组合 - 评估规则:关键词、响应时间、LLM 评分、语义相似度、JSON Schema、安全检测 + all/any/weighted 组合
- 模型配置中心(统一凭据管理 + 加密落库 + OpenAI/Anthropic/DashScope/Gemini 协议) - 模型配置中心(统一凭据管理 + Fernet 加密 + OpenAI/Anthropic/DashScope/Gemini 协议)
- 报告生成(JSON / HTML / Markdown / 对比报告)+ Webhook 通知 - 报告生成(JSON / HTML / Markdown / 同场景对比报告)+ Webhook 通知
- 触发来源标记(手动 / AI 助手 / CLI)+ AI 助手标准化评测链路
- 简单登录(访问密码 + 会话 token,未配置零打扰)
- 运营仪表盘(指标卡 / 趋势 / 场景表现 / 触发来源分布)
- 文件管理(分类树 + 上传下载) - 文件管理(分类树 + 上传下载)
- OpenClaw 免登录 iframe 集成 + HTTP Skill - OpenClaw 免登录 iframe 集成 + 标准 agenteval-run 技能
- Web 管理后台(FastAPI + React) - Web 管理后台(FastAPI + React)
## v0.3 亮点 ## v0.4 亮点
- **模型配置中心**:模型连接统一管理,场景只引用配置 ID,API Key 加密落库、响应零泄露 - **AI 助手标准化**:`triggered_by` 全链路标记 + 标准 SKILL.md 版本管理与自动同步 + API Key 自动注入
- **ModelGateway**:chat / embed / moderate 统一入口,4 种协议适配器 - **简单登录**:`AGENTEVAL_ADMIN_PASSWORD` 一键启用访问控制,机器调用(X-API-Key)不受影响
- **3 通道 + 6 规则**:工厂/注册表模式,插件式扩展 - **仪表盘重构**:6 指标卡 + 趋势 + 场景表现排行 + 触发来源分布
- **测试基线**:218 个测试,82% 覆盖率 - **报告页重做**:场景筛选、富选项下拉、一键重置、对比报告限同场景
- **报告增强**:Markdown 导出、run 对比、Webhook 通知 - **keep-alive 刷新修复**:标签页激活自动刷新,根治"评测记录消失"问题
- **测试基线**:232 个测试
## 快速开始 ## 快速开始

View File

@ -33,6 +33,13 @@ class Settings(BaseSettings):
default=None, default=None,
description="If set, all /api/* endpoints require the X-API-Key header to match.", description="If set, all /api/* endpoints require the X-API-Key header to match.",
) )
admin_password: Optional[str] = Field(
default=None,
description=(
"If set, the web UI requires login with this password. "
"Leave empty to disable the login gate (dev default)."
),
)
secret_key: Optional[str] = Field( secret_key: Optional[str] = Field(
default=None, default=None,
description="Fernet key used to encrypt model provider API keys at rest.", description="Fernet key used to encrypt model provider API keys at rest.",

View File

@ -24,6 +24,7 @@ from agenteval.models import (
ModelPurpose, ModelPurpose,
RuleLogic, RuleLogic,
RunStatus, RunStatus,
RunTrigger,
Scenario, Scenario,
Turn, Turn,
) )
@ -75,6 +76,7 @@ class EvalEngine:
cancel_token: Optional[asyncio.Event] = None, cancel_token: Optional[asyncio.Event] = None,
timeout_config: Optional[TimeoutConfig] = None, timeout_config: Optional[TimeoutConfig] = None,
max_concurrent_cases: int = 1, max_concurrent_cases: int = 1,
triggered_by: RunTrigger = RunTrigger.MANUAL,
): ):
self.target = target self.target = target
self.scenario = scenario self.scenario = scenario
@ -84,6 +86,7 @@ class EvalEngine:
self.result_repo = result_repo or ResultRepository(self.session) self.result_repo = result_repo or ResultRepository(self.session)
self.cancel_token = cancel_token or asyncio.Event() self.cancel_token = cancel_token or asyncio.Event()
self.timeout_config = timeout_config or TimeoutConfig() self.timeout_config = timeout_config or TimeoutConfig()
self.triggered_by = triggered_by
self._case_semaphore = asyncio.Semaphore(max(1, max_concurrent_cases)) self._case_semaphore = asyncio.Semaphore(max(1, max_concurrent_cases))
# Collects fatal case-level errors (e.g. dynamic message generation # Collects fatal case-level errors (e.g. dynamic message generation
# failures) so their cause is persisted into run.summary — not just # failures) so their cause is persisted into run.summary — not just
@ -116,6 +119,7 @@ class EvalEngine:
target_id=self.target.id or "", target_id=self.target.id or "",
scenario_id=self.scenario.id or "", scenario_id=self.scenario.id or "",
status=RunStatus.RUNNING, status=RunStatus.RUNNING,
triggered_by=self.triggered_by,
started_at=utc_now(), started_at=utc_now(),
) )
run = self.run_repo.create(run) run = self.run_repo.create(run)

View File

@ -169,10 +169,15 @@ def generate_report(run_id: str, session=None) -> dict[str, Any]:
def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[str, Any]: def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[str, Any]:
"""Build a side-by-side comparison dict for two runs.""" """Build a side-by-side comparison dict for two runs of the same scenario."""
report_a = generate_report(run_id_1, session) report_a = generate_report(run_id_1, session)
report_b = generate_report(run_id_2, session) report_b = generate_report(run_id_2, session)
# Cross-scenario case_ids never overlap, so every case would be flagged
# "changed" and the diff would be meaningless — reject early.
if report_a.get("scenario_id") != report_b.get("scenario_id"):
raise ValueError("compare report requires both runs to use the same scenario")
def _summary_delta(key: str) -> float: def _summary_delta(key: str) -> float:
return report_b["summary"][key] - report_a["summary"][key] return report_b["summary"][key] - report_a["summary"][key]
@ -189,7 +194,11 @@ def generate_compare_report(run_id_1: str, run_id_2: str, session=None) -> dict[
def _case_passed(c): def _case_passed(c):
if not c: if not c:
return None return None
return all(r["passed"] for r in c.get("results", [])) results = c.get("results", [])
if not results:
# No rule results (e.g. errored case) must not count as passed.
return None
return all(r["passed"] for r in results)
case_diffs.append( case_diffs.append(
{ {

View File

@ -148,6 +148,12 @@ class RunStatus(str, Enum):
FAILED = "failed" FAILED = "failed"
class RunTrigger(str, Enum):
MANUAL = "manual"
AI_ASSISTANT = "ai_assistant"
CLI = "cli"
class EvalRun(BaseModel): class EvalRun(BaseModel):
"""A single evaluation run.""" """A single evaluation run."""
@ -155,6 +161,7 @@ class EvalRun(BaseModel):
target_id: str target_id: str
scenario_id: str scenario_id: str
status: RunStatus = RunStatus.PENDING status: RunStatus = RunStatus.PENDING
triggered_by: RunTrigger = RunTrigger.MANUAL
started_at: Optional[datetime] = None started_at: Optional[datetime] = None
completed_at: Optional[datetime] = None completed_at: Optional[datetime] = None
summary: Optional[dict[str, Any]] = None summary: Optional[dict[str, Any]] = None

View File

@ -168,6 +168,7 @@ class EvalRunDB(SQLModel, table=True):
target_id: Optional[str] = Field(default=None, foreign_key="eval_targets.id") target_id: Optional[str] = Field(default=None, foreign_key="eval_targets.id")
scenario_id: Optional[str] = Field(default=None, foreign_key="scenarios.id") scenario_id: Optional[str] = Field(default=None, foreign_key="scenarios.id")
status: str = "pending" status: str = "pending"
triggered_by: str = Field(default="manual")
started_at: Optional[datetime] = Field(default_factory=utc_now) started_at: Optional[datetime] = Field(default_factory=utc_now)
completed_at: Optional[datetime] = None completed_at: Optional[datetime] = None
summary: Optional[str] = None summary: Optional[str] = None

View File

@ -82,6 +82,7 @@ def _run_to_db(run: EvalRun) -> EvalRunDB:
target_id=run.target_id, target_id=run.target_id,
scenario_id=run.scenario_id, scenario_id=run.scenario_id,
status=run.status.value, status=run.status.value,
triggered_by=run.triggered_by.value,
started_at=run.started_at, started_at=run.started_at,
completed_at=run.completed_at, completed_at=run.completed_at,
) )
@ -96,6 +97,7 @@ def _run_from_db(db: EvalRunDB) -> EvalRun:
target_id=db.target_id, target_id=db.target_id,
scenario_id=db.scenario_id, scenario_id=db.scenario_id,
status=db.status, status=db.status,
triggered_by=db.triggered_by or "manual",
started_at=db.started_at, started_at=db.started_at,
completed_at=db.completed_at, completed_at=db.completed_at,
summary=db.get_summary(), summary=db.get_summary(),

View File

@ -11,7 +11,7 @@ from agenteval.config import get_settings
from agenteval.storage.db import init_db from agenteval.storage.db import init_db
from agenteval.version import get_build_info, get_version from agenteval.version import get_build_info, get_version
from agenteval.web.deps import require_api_key from agenteval.web.deps import require_api_key
from agenteval.web.routers import files, model_configs, proxy, reports, runs, scenarios, stats, targets from agenteval.web.routers import auth, files, model_configs, proxy, reports, runs, scenarios, stats, targets
from agenteval.web.websocket import ws_manager from agenteval.web.websocket import ws_manager
@ -40,6 +40,8 @@ app.add_middleware(
_api_deps = [Depends(require_api_key)] _api_deps = [Depends(require_api_key)]
# Login endpoints must stay open — they are how the client obtains credentials.
app.include_router(auth.router, prefix="/api/auth", tags=["auth"])
app.include_router(targets.router, prefix="/api/targets", tags=["targets"], dependencies=_api_deps) app.include_router(targets.router, prefix="/api/targets", tags=["targets"], dependencies=_api_deps)
app.include_router(scenarios.router, prefix="/api/scenarios", tags=["scenarios"], dependencies=_api_deps) app.include_router(scenarios.router, prefix="/api/scenarios", tags=["scenarios"], dependencies=_api_deps)
app.include_router(runs.router, prefix="/api/runs", tags=["runs"], dependencies=_api_deps) app.include_router(runs.router, prefix="/api/runs", tags=["runs"], dependencies=_api_deps)

View File

@ -21,21 +21,37 @@ def get_db():
session.close() session.close()
def require_api_key(x_api_key: Optional[str] = Header(default=None)) -> None: def require_auth(
"""Enforce X-API-Key header when AGENTEVAL_API_KEY is configured. x_api_key: Optional[str] = Header(default=None),
x_auth_token: Optional[str] = Header(default=None),
) -> None:
"""Enforce authentication when any credential is configured.
When the setting is empty (default), the check is a no-op so local dev keeps Accepted credentials (either one passes):
working without extra configuration. - X-API-Key matching AGENTEVAL_API_KEY (machine callers, OpenClaw skill)
- X-Auth-Token matching the web session token (browser after login)
When neither AGENTEVAL_API_KEY nor AGENTEVAL_ADMIN_PASSWORD is set, the
check is a no-op so local dev keeps working without extra configuration.
""" """
configured = get_settings().api_key settings = get_settings()
if not configured: if not settings.api_key and not settings.admin_password:
return
if settings.api_key and x_api_key == settings.api_key:
return
if settings.admin_password:
from agenteval.web.routers.auth import verify_session_token
if x_auth_token and verify_session_token(x_auth_token):
return return
if not x_api_key or x_api_key != configured:
raise HTTPException( raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or missing X-API-Key", detail="Invalid or missing credentials",
) )
# Backwards-compatible alias: existing routers depend on require_api_key.
require_api_key = require_auth
# Convenience alias used by routers via ``dependencies=[Depends(auth_required)]``. # Convenience alias used by routers via ``dependencies=[Depends(auth_required)]``.
auth_required = Depends(require_api_key) auth_required = Depends(require_auth)

View File

@ -0,0 +1,52 @@
"""Session login for the web UI.
A single shared password (AGENTEVAL_ADMIN_PASSWORD) gates the SPA. On success
the client receives a stateless HMAC token it sends as X-Auth-Token. Machine
callers (e.g. the OpenClaw skill) keep using X-API-Key and are unaffected.
"""
import hmac
import secrets
from fastapi import APIRouter, Header, HTTPException, status
from pydantic import BaseModel
from agenteval.config import get_settings
router = APIRouter()
_TOKEN_CONTEXT = b"agenteval-web-session"
def session_token() -> str:
"""Derive the stateless session token from configured secrets."""
settings = get_settings()
key = (settings.secret_key or settings.admin_password or "").encode()
return hmac.new(key, _TOKEN_CONTEXT, "sha256").hexdigest()
def verify_session_token(token: str) -> bool:
return bool(token) and hmac.compare_digest(token, session_token())
class LoginRequest(BaseModel):
password: str
@router.post("/login")
def login(request: LoginRequest) -> dict:
configured = get_settings().admin_password
if not configured:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="登录未启用")
if not secrets.compare_digest(request.password, configured):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="密码错误")
return {"token": session_token()}
@router.get("/status")
def auth_status(x_auth_token: str = Header(default="")) -> dict:
auth_required = bool(get_settings().admin_password)
return {
"auth_required": auth_required,
"authenticated": (not auth_required) or verify_session_token(x_auth_token),
}

View File

@ -23,10 +23,14 @@ def get_compare_report(
session: Session = Depends(get_db), session: Session = Depends(get_db),
) -> dict: ) -> dict:
repo = RunRepository(session) repo = RunRepository(session)
if not repo.get(run1): run_a = repo.get(run1)
run_b = repo.get(run2)
if not run_a:
raise HTTPException(status_code=404, detail=f"run not found: {run1}") raise HTTPException(status_code=404, detail=f"run not found: {run1}")
if not repo.get(run2): if not run_b:
raise HTTPException(status_code=404, detail=f"run not found: {run2}") raise HTTPException(status_code=404, detail=f"run not found: {run2}")
if run_a.scenario_id != run_b.scenario_id:
raise HTTPException(status_code=400, detail="对比报告要求两个运行使用相同场景")
return generate_compare_report(run1, run2, session) return generate_compare_report(run1, run2, session)

View File

@ -8,7 +8,7 @@ from pydantic import BaseModel
from sqlmodel import Session from sqlmodel import Session
from agenteval.evaluation.engine import EvalEngine from agenteval.evaluation.engine import EvalEngine
from agenteval.models import EvalRun, RunStatus from agenteval.models import EvalRun, RunStatus, RunTrigger
from agenteval.storage.db import get_session, iso_utc from agenteval.storage.db import get_session, iso_utc
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from agenteval.utils.llm import extract_reply_text from agenteval.utils.llm import extract_reply_text
@ -22,6 +22,7 @@ router = APIRouter()
class StartRunRequest(BaseModel): class StartRunRequest(BaseModel):
target_id: str target_id: str
scenario_id: str scenario_id: str
triggered_by: RunTrigger = RunTrigger.MANUAL
# ── Task registry for live evaluation runs ───────────────────────────── # ── Task registry for live evaluation runs ─────────────────────────────
@ -69,7 +70,16 @@ async def _run_evaluation(run_id: str, target_id: str, scenario_id: str) -> None
@router.get("") @router.get("")
async def list_runs(session: Session = Depends(get_db)) -> list[dict]: async def list_runs(session: Session = Depends(get_db)) -> list[dict]:
return [r.model_dump() for r in RunRepository(session).list_all()] scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
target_names = {t.id: t.name for t in TargetRepository(session).list_all()}
return [
{
**r.model_dump(),
"scenario_name": scenario_names.get(r.scenario_id),
"target_name": target_names.get(r.target_id),
}
for r in RunRepository(session).list_all()
]
@router.post("") @router.post("")
@ -82,7 +92,11 @@ async def start_run(
if not target or not scenario: if not target or not scenario:
raise HTTPException(status_code=404, detail="target or scenario not found") raise HTTPException(status_code=404, detail="target or scenario not found")
run = EvalRun(target_id=request.target_id, scenario_id=request.scenario_id) run = EvalRun(
target_id=request.target_id,
scenario_id=request.scenario_id,
triggered_by=request.triggered_by,
)
run = RunRepository(session).create(run) run = RunRepository(session).create(run)
task = asyncio.create_task( task = asyncio.create_task(

View File

@ -1,34 +1,93 @@
"""API routes for statistics and dashboard data.""" """API routes for statistics and dashboard data."""
from collections import defaultdict from collections import defaultdict
from datetime import datetime, timezone
from fastapi import APIRouter, Depends from fastapi import APIRouter, Depends
from sqlmodel import Session from sqlmodel import Session
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from agenteval.web.deps import get_db from agenteval.web.deps import get_db
router = APIRouter() router = APIRouter()
def _ts(dt: datetime | None) -> float:
"""Sortable timestamp tolerant of naive/aware mixes in legacy rows."""
if dt is None:
return 0.0
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.timestamp()
@router.get("/dashboard") @router.get("/dashboard")
def dashboard(session: Session = Depends(get_db)) -> dict: def dashboard(session: Session = Depends(get_db)) -> dict:
targets = TargetRepository(session).list_all() targets = TargetRepository(session).list_all()
scenarios = ScenarioRepository(session).list_all() scenarios = ScenarioRepository(session).list_all()
runs = RunRepository(session).list_all() runs = RunRepository(session).list_all()
model_configs = ModelConfigRepository(session).list_all()
scenario_names = {s.id: s.name for s in scenarios}
target_names = {t.id: t.name for t in targets}
completed_runs = [r for r in runs if r.status == "completed" and r.summary] completed_runs = [r for r in runs if r.status == "completed" and r.summary]
pass_rates = [r.summary.get("pass_rate", 0) for r in completed_runs if isinstance(r.summary, dict)] pass_rates = [r.summary.get("pass_rate", 0) for r in completed_runs if isinstance(r.summary, dict)]
overall_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else None overall_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else None
recent_runs = sorted(runs, key=lambda r: r.started_at or "", reverse=True)[:10] today = datetime.now(timezone.utc).date()
today_runs = 0
running_count = 0
trigger_breakdown: dict[str, int] = defaultdict(int)
for r in runs:
if r.started_at:
started = r.started_at
if started.tzinfo is None:
started = started.replace(tzinfo=timezone.utc)
if started.date() == today:
today_runs += 1
if r.status in ("running", "pending"):
running_count += 1
trigger_breakdown[r.triggered_by.value] += 1
# Per-scenario aggregation over completed runs.
by_scenario: dict[str, list] = defaultdict(list)
for r in completed_runs:
by_scenario[r.scenario_id].append(r)
scenario_stats = []
for sid, sruns in by_scenario.items():
rates = [r.summary.get("pass_rate", 0) for r in sruns if isinstance(r.summary, dict)]
last_run = max(sruns, key=lambda r: _ts(r.started_at))
scenario_stats.append({
"scenario_id": sid,
"scenario_name": scenario_names.get(sid, sid[:8]),
"run_count": len(sruns),
"avg_pass_rate": round(sum(rates) / len(rates), 4) if rates else None,
"last_run_at": last_run.started_at.isoformat() if last_run.started_at else None,
})
scenario_stats.sort(key=lambda s: s["run_count"], reverse=True)
recent_runs = sorted(runs, key=lambda r: _ts(r.started_at), reverse=True)[:10]
return { return {
"targets_count": len(targets), "targets_count": len(targets),
"scenarios_count": len(scenarios), "scenarios_count": len(scenarios),
"runs_count": len(runs), "runs_count": len(runs),
"model_configs_count": len(model_configs),
"today_runs": today_runs,
"running_count": running_count,
"overall_pass_rate": overall_pass_rate, "overall_pass_rate": overall_pass_rate,
"recent_runs": [r.model_dump() for r in recent_runs], "trigger_breakdown": dict(trigger_breakdown),
"scenario_stats": scenario_stats,
"recent_runs": [
{
**r.model_dump(),
"scenario_name": scenario_names.get(r.scenario_id),
"target_name": target_names.get(r.target_id),
}
for r in recent_runs
],
} }

View File

@ -5,6 +5,7 @@ from typing import Any
import typer import typer
from agenteval.evaluation.engine import EvalEngine from agenteval.evaluation.engine import EvalEngine
from agenteval.models import RunTrigger
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from rich.console import Console from rich.console import Console
from rich.table import Table from rich.table import Table
@ -44,7 +45,7 @@ def start_run(
raise typer.Exit(1) raise typer.Exit(1)
console.print(f"开始评测: 对象={target.name}, 场景={scenario.name}") console.print(f"开始评测: 对象={target.name}, 场景={scenario.name}")
engine = EvalEngine(target=target, scenario=scenario) engine = EvalEngine(target=target, scenario=scenario, triggered_by=RunTrigger.CLI)
try: try:
run = asyncio.run(engine.run(progress_callback=_progress)) run = asyncio.run(engine.run(progress_callback=_progress))
except Exception as exc: except Exception as exc:

View File

@ -1,57 +1,45 @@
# OpenClaw 插件 for AgentEvalTool # OpenClaw 插件 for AgentEvalTool
本目录提供 OpenClaw 调用 AgentEvalTool 的示例技能,使 OpenClaw 能够自闭环完成评测策略设计。 本目录提供 OpenClaw 调用 AgentEvalTool 的标准接入方式,使 OpenClaw(AI 助手)能够自闭环完成评测。
## 设计原则 ## 设计原则
- AgentEvalTool 作为独立的 CLI 工具集存在,负责: - AgentEvalTool 负责:
- 评测对象管理 - 评测对象管理
- 评测场景管理 - 评测场景管理
- 评测执行与原始数据收集 - 评测执行与原始数据收集
- 评估分析与报告生成 - 评估分析与报告生成
- OpenClaw 通过本插件调用 `agenteval` CLI,负责: - OpenClaw 负责:
- 评测策略编排(选择对象、场景、触发时机) - 评测策略编排(选择对象、场景、触发时机)
- 定时调度(OpenClaw 自身的 cron + skill 机制) - 定时调度(OpenClaw 自身的 cron + skill 机制)
- 结果通知与后续动作 - 结果通知与后续动作
## 前置条件 ## 唯一标准接入方式:HTTP API
1. 已安装 AgentEvalTool CLI: **所有 OpenClaw 触发的评测必须走 AgentEvalTool 的 HTTP API**(`POST /api/runs` 等),
```bash 并在启动评测时携带 `"triggered_by": "ai_assistant"`,以便平台区分手动触发与 AI 助手触发。
pip install -e /path/to/AgentEvalTool
```
2. `agenteval` 命令已在 PATH 中可用。
3. 已通过 `agenteval target add` 注册评测对象。
4. 已通过 `agenteval scenario import` 导入评测场景。
## 接入方式 > ⚠️ 禁止在 OpenClaw 中通过 subprocess 调用 `agenteval` CLI。
> CLI 直接写入其运行环境下的本地 SQLite(路径硬编码),与 Web 后台的数据库不共享,
> 会导致评测记录"消失"——Web 页面永远查不到。CLI 仅用于开发者在 agenteval
> 服务同一环境下的手工调试。
### 方式一:直接调用 CLI(推荐) ### 方式一:agenteval-run 技能(推荐)
在 OpenClaw Skill 中直接调用系统命令: `skills/agenteval-run/SKILL.md` 是标准技能文件,指导 AI 助手用 `curl` 走 API 全流程
(列对象/场景 → 启动评测 → 轮询状态 → 取报告)。
```python 部署脚本(`scripts/deploy-t480.sh`)会自动把它同步到 OpenClaw 工作区:
import subprocess `data/openclaw/workspace/skills/agenteval-run/SKILL.md`。
# 触发评测 ### 方式二:Python Skill 封装
subprocess.run([
"agenteval", "run", "start",
"--target-id", "<target-id>",
"--scenario-id", "<scenario-id>",
], check=True)
# 获取报告(需要在运行输出中解析 run_id) 参考 `agenteval_skill.py`(httpx 调 API),在 OpenClaw 中注册技能时传入配置:
report = subprocess.check_output([
"agenteval", "report", "show", "<run-id>", "--format", "json",
])
```
### 方式二:使用本目录封装的 Skill
参考 `agenteval_skill.py`,在 OpenClaw 中注册技能时传入配置:
```json ```json
{ {
"api_base_url": "http://agenteval:8000",
"api_key": "<可选,对应 AGENTEVAL_API_KEY>",
"target_id": "<target-id>", "target_id": "<target-id>",
"scenario_id": "<scenario-id>", "scenario_id": "<scenario-id>",
"report_format": "json" "report_format": "json"
@ -66,20 +54,21 @@ OpenClaw 的 cron 配置(具体格式以 OpenClaw 平台为准):
skill: agenteval_skill skill: agenteval_skill
schedule: "0 9 * * *" # 每天早上 9 点执行 schedule: "0 9 * * *" # 每天早上 9 点执行
config: config:
api_base_url: "http://agenteval:8000"
target_id: "<target-id>" target_id: "<target-id>"
scenario_id: "<scenario-id>" scenario_id: "<scenario-id>"
report_format: "json" report_format: "json"
``` ```
## CLI 返回的 run_id 解析 ## API 速查
`agenteval run start` 成功后会输出: | 操作 | 请求 |
|---|---|
| 启动评测 | `POST /api/runs` body: `{"target_id", "scenario_id", "triggered_by": "ai_assistant"}` |
| 查询状态 | `GET /api/runs/{run_id}`(轮询至 completed/failed) |
| 获取报告 | `GET /api/reports/{run_id}` |
```text 如果平台设置了 `AGENTEVAL_API_KEY`,所有请求需带 `X-API-Key` 头。
评测完成: run_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, status=completed
```
OpenClaw Skill 需要从 stdout 中提取 `run_id`,然后调用 `agenteval report show <run_id>` 获取报告。
## 扩展建议 ## 扩展建议

View File

@ -44,6 +44,7 @@ class AgentEvalSkill:
json={ json={
"target_id": self.target_id, "target_id": self.target_id,
"scenario_id": self.scenario_id, "scenario_id": self.scenario_id,
"triggered_by": "ai_assistant",
}, },
) )
if resp.status_code != 200: if resp.status_code != 200:

View File

@ -0,0 +1,54 @@
---
name: agenteval-run
description: 触发 AgentEvalTool 评测任务并获取报告
---
当用户要求执行评测时,使用 exec 工具运行以下命令。
所有评测必须走 AgentEvalTool 标准 HTTP API(禁止直接调 CLI 或操作数据库)。
平台可能启用了 API Key 鉴权。每次执行命令前先读取密钥(文件不存在则为空,不影响未启用鉴权的环境):
```bash
KEY=$(cat ~/.openclaw/agenteval-api-key 2>/dev/null)
```
以下所有 curl 命令都必须带 `-H "X-API-Key: $KEY"`。
## 查看评测对象列表
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/targets | python3 -m json.tool
```
## 查看评测场景列表
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/scenarios | python3 -m json.tool
```
## 启动评测
必须携带 `"triggered_by": "ai_assistant"`,平台以此区分手动触发与 AI 助手触发:
```bash
curl -s -X POST http://agenteval:8000/api/runs \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"target_id": "<target_id>", "scenario_id": "<scenario_id>", "triggered_by": "ai_assistant"}'
```
响应中的 `id` 字段即 run_id,请务必回报给用户。
## 查看运行状态
轮询直到 `status` 变为 `completed` 或 `failed`(建议每 5 秒一次):
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/runs/<run_id> | python3 -m json.tool
```
## 获取报告
```bash
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/reports/<run_id> | python3 -m json.tool
```
请将 <target_id>、<scenario_id>、<run_id> 替换为实际值。
完成后向用户汇报:run_id、状态、通过率(summary.pass_rate)以及失败用例摘要。

70
docs/plan-v0.4.md Normal file
View File

@ -0,0 +1,70 @@
# AgentEvalTool v0.4.0 「联」开发计划
**开发版本**: v0.4.0-dev
**制定日期**: 2026-07-27
**代码基线**: `92f98c3`(v0.3.0-dev 已发布 t480)
**核心目标**: AI 助手评测链路标准化 + 报告查询体验优化
**状态**: ✅ 已发布(t480, 2026-07-28),发布说明见 [release-notes-v0.4.md](release-notes-v0.4.md)
---
## 一、背景:run 95ee8738 "消失"事故排查
用户通过 AI 助手触发评测(run `95ee8738-11e7-4959-a3cf-45e3536b28eb`),但在"评测执行"和"评测报告"页面均查不到。
**排查结论(2026-07-27)**:
| 检查项 | 结果 |
|---|---|
| t480 数据库 `eval_runs` | ✅ run 存在,status=completed |
| `GET /api/runs` | ✅ 正常返回,列表第一条 |
| t480 SKILL.md(openclaw 工作区) | ✅ 走标准 `POST /api/runs` |
| 前端展示 | ❌ **根因所在** |
**根因**:前端采用 keep-alive 标签页架构(页面挂载后永不卸载),Runs / Reports 页只在**首次挂载**时加载一次列表,切回标签页不刷新。AI 助手评测发生在页面已挂载之后 → 两个页面全是旧数据。叠加因素:Runs 页默认时间过滤为"今天"、Reports 页只显示 completed 状态。
**数据与 API 完全正常,纯前端展示层问题。**
## 二、需求与实现
### 需求 1:触发来源标记 + AI 助手标准化链路
- `EvalRun.triggered_by` 枚举字段(`manual` / `ai_assistant` / `cli`,默认 manual),全链路透传:models → EvalRunDB → repository → engine → CLI → API
- 迁移 `b7d4e6f81c22`:`eval_runs` 加列,存量回填 `manual`
- `POST /api/runs` 接受可选 `triggered_by`(非法值 422)
- 标准 skill 纳入版本管理:`backend/plugins/openclaw/skills/agenteval-run/SKILL.md`(要求带 `triggered_by: "ai_assistant"`),deploy 脚本自动同步到 openclaw 工作区
- `plugins/openclaw/README.md` 移除 CLI subprocess 模式(CLI 写孤立库,禁止 OpenClaw 使用)
- 前端:RunList 与 Reports 下拉显示来源 Tag(AI 助手=紫色 / CLI=蓝色 / 手动不显示)
### 需求 1.5(排查衍生,本次事故的直接修复):标签页激活刷新
- 新 hook `useOnTabActive(path, cb)`:监听 tabStore.activeKey,标签页重新激活时触发刷新(跳过首次挂载)
- Runs、Reports 两页接入
### 需求 2:报告页查询体验优化
- 后端 `GET /api/runs` 返回冗余 `scenario_name` / `target_name`
- Reports 页选择区重做:
- 场景筛选器(含各场景 run 数量统计)
- run 下拉富选项:场景名 · 对象名 · 时间 · 通过率 · 来源 Tag,支持关键字搜索
- 修复旧版 `.reverse()` 导致最旧记录排最前的问题
### 需求 3:对比报告限制同场景
- 后端 `GET /api/reports/compare`:场景不同返回 `400 对比报告要求两个运行使用相同场景`
- `generate_compare_report` 入口双保险(ValueError)
- 顺带修复:case 无规则结果时 `all([]) == True` 被误判为通过 → 改为 None
- 前端:报告 B 下拉只列同场景 run;A 变更后 B 场景不符自动清空
## 三、测试
- 新增 6 个测试:triggered_by 默认值/透传持久化/非法值 422、list 含名称字段、compare 跨场景 400/ValueError
- 全量 224 通过(v0.3 基线 218 → 224)
## 四、发布验证清单(t480)
1. `scripts/deploy-t480.sh` 部署(自动同步 SKILL.md 到 openclaw 工作区)
2. `/api/health` 返回 version=0.4.0-dev
3. 手动触发一次评测 → Runs 页显示无来源 Tag(manual)
4. 让 AI 助手触发一次评测 → 切回 Runs/Reports 页**不刷新浏览器**即可看到新记录,带"AI 助手"标记
5. 对比报告:跨场景 run 不可选、同场景可正常对比

View File

@ -0,0 +1,95 @@
# AgentEvalTool v0.4 版本发布说明
**版本**: v0.4.0-dev
**日期**: 2026-07-27 ~ 2026-07-28
**状态**: 已发布(t480 开发线)
**代号**: 「联」(Integration milestone)
**作者**: AgentEval Team
---
## 一、版本定位
v0.4 的主题是**打通 AI 助手与平台的标准化联动**,并完成生产化前的最后两块拼图:访问控制与运营视图。起因是一次真实事故:用户通过 AI 助手触发的评测(run `95ee8738`)在页面上"消失"。
```
v0.1 (2026-07-09) MVP 闭环
v0.2 (2026-07-14) 「稳」async 引擎 + 安全基线 + 部署规范化
v0.3 (2026-07-17) 「拓」多通道 + 规则扩展 + 模型配置中心
v0.4 (2026-07-28) ← 你在这里:「联」AI 助手标准化 + 登录 + 仪表盘重构
```
## 二、事故排查:run 95ee8738 "消失"
| 检查项 | 结果 |
|---|---|
| t480 数据库 / API | ✅ run 存在、completed、列表第一条 |
| OpenClaw skill 调用链 | ✅ 走标准 POST /api/runs |
| 前端展示 | ❌ **根因** |
**根因**:keep-alive 标签页架构下页面永不重新挂载,Runs / Reports 只在首次挂载时加载列表,切回标签页不刷新;叠加 Runs 页默认"今天"时间过滤、Reports 页只显示 completed。**纯前端展示层问题,数据无损**。
**修复**:新增 `useOnTabActive` hook — 标签页重新激活时自动刷新,已接入仪表盘 / Runs / Reports 三页。
## 三、主要功能
### 3.1 触发来源标记(triggered_by)
- `EvalRun.triggered_by` 枚举(manual / ai_assistant / cli)全链路:模型 → DB(迁移 `b7d4e6f81c22`,存量回填 manual)→ API → CLI → 引擎
- `POST /api/runs` 接受可选 `triggered_by`,非法值 422
- 前端 Runs 列表、Reports 下拉、仪表盘显示来源 Tag(AI 助手=紫 / CLI=蓝)
### 3.2 AI 助手链路标准化
- 标准 skill `backend/plugins/openclaw/skills/agenteval-run/SKILL.md` 纳入版本管理,deploy 脚本自动同步到 openclaw 工作区
- skill 自动从 `~/.openclaw/agenteval-api-key` 读取 API Key(deploy 脚本从远端 .env 提取生成),启用鉴权后 AI 助手不受影响
- `plugins/openclaw/README.md` 废除 CLI subprocess 模式(CLI 直写本地 SQLite 孤立库,是"记录消失"类事故的高危路径)
### 3.3 评测报告页体验重做
- `GET /api/runs` 返回冗余 `scenario_name` / `target_name`
- 两行表头(模式+场景筛选+重置/刷新 | 报告选择+操作),对比模式不再换行
- 场景筛选器(含计数)、富选项下拉(场景·对象·时间·通过率·来源)、全下拉 allowClear、一键重置
- **对比报告限同场景**:后端 400 + generate 层双保险 + 前端 B 下拉只列同场景;顺带修复空 results 被 `all([])` 误判通过
### 3.4 简单登录(访问控制)
- `.env` 配 `AGENTEVAL_ADMIN_PASSWORD` 即启用;未配置零打扰
- `POST /api/auth/login` 换取无状态 HMAC 会话 token(sessionStorage,关浏览器失效);401 自动踢回登录页;菜单头部图标退出
- `require_api_key` 升级 `require_auth`:X-API-Key(机器)与 X-Auth-Token(浏览器)双凭据并行
### 3.5 仪表盘全面重构
- `/api/stats/dashboard` 扩展:模型配置数、今日执行、运行中、触发来源分布、场景聚合统计、富化最近记录
- 三行布局:6 指标卡 / 趋势图+场景表现排行 / 最近记录+快捷操作+来源分布
- 页面请求从 3 个全量列表收敛为 2 个聚合接口
### 3.6 UI 与工程优化
- 模型配置页 12 列 → 6 列合并,取消横向滚动(Endpoint/完整 Token 数移入悬浮提示)
- 仪表盘滚动修复(flex:1 在非 flex 父容器失效 → height:100%)
- vite `chunkSizeWarningLimit: 2200`:注释记录 vendor 2.12MB 不可拆分原因(@antv 强拆触发 TDZ 崩溃、rollup 拒绝 async 归并的实测结论),阈值仅在真实回归时报警
## 四、测试与质量
| 指标 | v0.3 | v0.4 |
|---|---|---|
| 测试数量 | 218 | **232** |
| 新增 | — | triggered_by×4、compare 场景限制×2、auth×5、stats×3 |
CI 三件套(v0.3 收尾时补齐)在本版本首次实战:pre-push hook 拦截推送前跑版本一致性 + ruff + pytest。
## 五、部署与配置变更
- **t480 `.env` 新增**(需手动维护,陷阱 #5):`AGENTEVAL_ADMIN_PASSWORD`、`AGENTEVAL_API_KEY`
- deploy-t480.sh 新增:SKILL.md 同步 + API Key 文件注入 openclaw 工作区
- t480 验证:登录门 401/200 全链路、AI 助手带 key 调用 200、仪表盘聚合数据正确
## 六、已知问题
1. t480 访问密码为临时值,需自行更换
2. 前端仍无自动化测试(仅 tsc)
3. `.env` 远端手动维护未根治(陷阱 #5)
4. volcengine-102 生产线尚未同步 v0.4(含登录配置)
## 七、v0.5 候选方向
- 前端测试基线(vitest)
- 报告趋势分析 / 多 run 聚合看板
- OpenClaw 深度集成二期(插件双向调用)
- 多模态评测用例
- 配置同步自动化(消灭陷阱 #5)

View File

@ -1,12 +1,12 @@
{ {
"name": "agenteval-web", "name": "agenteval-web",
"version": "0.3.0-dev", "version": "0.4.0-dev",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "agenteval-web", "name": "agenteval-web",
"version": "0.3.0-dev", "version": "0.4.0-dev",
"dependencies": { "dependencies": {
"@ant-design/charts": "^2.6.7", "@ant-design/charts": "^2.6.7",
"@ant-design/icons": "^6.3.2", "@ant-design/icons": "^6.3.2",

View File

@ -1,6 +1,6 @@
{ {
"name": "agenteval-web", "name": "agenteval-web",
"version": "0.3.0-dev", "version": "0.4.0-dev",
"private": true, "private": true,
"type": "module", "type": "module",
"scripts": { "scripts": {

View File

@ -1,6 +1,6 @@
import { lazy, Suspense, useEffect } from 'react' import { lazy, Suspense, useEffect, useState } from 'react'
import { useLocation, useNavigate } from 'react-router-dom' import { useLocation, useNavigate } from 'react-router-dom'
import { Layout, Menu, Spin } from 'antd' import { Layout, Menu, Spin, Button, Tooltip } from 'antd'
import type { MenuProps } from 'antd' import type { MenuProps } from 'antd'
import { import {
DashboardOutlined, DashboardOutlined,
@ -11,8 +11,11 @@ import {
RobotOutlined, RobotOutlined,
FolderOpenOutlined, FolderOpenOutlined,
CloudServerOutlined, CloudServerOutlined,
LogoutOutlined,
} from '@ant-design/icons' } from '@ant-design/icons'
import TabBar from './components/TabBar' import TabBar from './components/TabBar'
import LoginPage from './pages/Login'
import { authApi, authToken, setOnUnauthorized } from './api'
import { useTabStore, type TabItem } from './stores/tabStore' import { useTabStore, type TabItem } from './stores/tabStore'
import { colors } from './tokens' import { colors } from './tokens'
import type { ReactNode } from 'react' import type { ReactNode } from 'react'
@ -68,10 +71,30 @@ const menuItems: MenuProps['items'] = routeConfigs.map((r) => ({
label: r.name, label: r.name,
})) }))
type AuthState = 'checking' | 'login' | 'ready'
function App() { function App() {
const location = useLocation() const location = useLocation()
const navigate = useNavigate() const navigate = useNavigate()
const { tabs, activeKey, openTab, setActiveTab } = useTabStore() const { tabs, activeKey, openTab, setActiveTab } = useTabStore()
const [authState, setAuthState] = useState<AuthState>('checking')
const [authRequired, setAuthRequired] = useState(false)
useEffect(() => {
setOnUnauthorized(() => setAuthState('login'))
authApi.status()
.then((res) => {
setAuthRequired(res.data.auth_required)
setAuthState(res.data.authenticated ? 'ready' : 'login')
})
.catch(() => setAuthState('ready')) // 后端不可达时不锁死界面,由具体请求报错
return () => setOnUnauthorized(null)
}, [])
const logout = () => {
authToken.clear()
setAuthState('login')
}
useEffect(() => { useEffect(() => {
const config = routeConfigs.find((r) => r.path === location.pathname) const config = routeConfigs.find((r) => r.path === location.pathname)
@ -102,6 +125,18 @@ function App() {
navigate(key) navigate(key)
} }
if (authState === 'checking') {
return (
<div style={{ height: '100vh', display: 'flex', alignItems: 'center', justifyContent: 'center' }}>
<Spin size="large" />
</div>
)
}
if (authState === 'login') {
return <LoginPage onSuccess={() => setAuthState('ready')} />
}
return ( return (
<Layout style={{ height: '100vh', overflow: 'hidden' }}> <Layout style={{ height: '100vh', overflow: 'hidden' }}>
<Layout.Sider <Layout.Sider
@ -120,7 +155,7 @@ function App() {
height: 56, height: 56,
display: 'flex', display: 'flex',
alignItems: 'center', alignItems: 'center',
padding: '0 20px', padding: '0 12px 0 20px',
flexShrink: 0, flexShrink: 0,
borderBottom: '1px solid rgba(255,255,255,0.06)', borderBottom: '1px solid rgba(255,255,255,0.06)',
}} }}
@ -128,6 +163,17 @@ function App() {
<span style={{ color: '#fff', fontWeight: 700, fontSize: 16, letterSpacing: '0.3px' }}> <span style={{ color: '#fff', fontWeight: 700, fontSize: 16, letterSpacing: '0.3px' }}>
AgentEvalTool AgentEvalTool
</span> </span>
{authRequired && (
<Tooltip title="退出登录" placement="right">
<Button
type="text"
size="small"
icon={<LogoutOutlined />}
onClick={logout}
style={{ color: colors.siderText, marginLeft: 'auto' }}
/>
</Tooltip>
)}
</div> </div>
<Menu <Menu
theme="dark" theme="dark"

View File

@ -1,14 +1,38 @@
import axios from 'axios' import axios from 'axios'
import { message } from 'antd' import { message } from 'antd'
const AUTH_TOKEN_KEY = 'agenteval_auth_token'
export const authToken = {
get: () => sessionStorage.getItem(AUTH_TOKEN_KEY),
set: (token: string) => sessionStorage.setItem(AUTH_TOKEN_KEY, token),
clear: () => sessionStorage.removeItem(AUTH_TOKEN_KEY),
}
/** Registered by App to switch back to the login screen on 401. */
let onUnauthorized: (() => void) | null = null
export const setOnUnauthorized = (handler: (() => void) | null) => { onUnauthorized = handler }
const api = axios.create({ const api = axios.create({
baseURL: '/api', baseURL: '/api',
timeout: 30000, timeout: 30000,
}) })
api.interceptors.request.use((config) => {
const token = authToken.get()
if (token) config.headers['X-Auth-Token'] = token
return config
})
api.interceptors.response.use( api.interceptors.response.use(
(response) => response, (response) => response,
(error) => { (error) => {
const isLoginCall = error.config?.url?.includes('/auth/login')
if (error.response?.status === 401 && !isLoginCall && onUnauthorized) {
authToken.clear()
onUnauthorized()
return Promise.reject(error)
}
const msg = error.response?.data?.detail || error.message || '请求失败' const msg = error.response?.data?.detail || error.message || '请求失败'
message.error(msg) message.error(msg)
return Promise.reject(error) return Promise.reject(error)
@ -17,6 +41,16 @@ api.interceptors.response.use(
export default api export default api
export interface AuthStatus {
auth_required: boolean
authenticated: boolean
}
export const authApi = {
status: () => api.get<AuthStatus>('/auth/status'),
login: (password: string) => api.post<{ token: string }>('/auth/login', { password }),
}
export interface Target { export interface Target {
id: string id: string
name: string name: string
@ -100,21 +134,39 @@ export interface ModelConfigReference {
purpose: string purpose: string
} }
export type RunTrigger = 'manual' | 'ai_assistant' | 'cli'
export interface Run { export interface Run {
id: string id: string
target_id: string target_id: string
scenario_id: string scenario_id: string
status: string status: string
triggered_by?: RunTrigger
scenario_name?: string | null
target_name?: string | null
started_at: string started_at: string
completed_at: string | null completed_at: string | null
summary: Record<string, unknown> | null summary: Record<string, unknown> | null
} }
export interface ScenarioStat {
scenario_id: string
scenario_name: string
run_count: number
avg_pass_rate: number | null
last_run_at: string | null
}
export interface DashboardStats { export interface DashboardStats {
targets_count: number targets_count: number
scenarios_count: number scenarios_count: number
runs_count: number runs_count: number
model_configs_count: number
today_runs: number
running_count: number
overall_pass_rate: number | null overall_pass_rate: number | null
trigger_breakdown: Partial<Record<RunTrigger, number>>
scenario_stats: ScenarioStat[]
recent_runs: Run[] recent_runs: Run[]
} }

View File

@ -1,10 +1,10 @@
import { useMemo, useState } from 'react' import { useMemo, useState } from 'react'
import { Empty, Input, Pagination, Progress, Segmented, Skeleton, Tooltip } from 'antd' import { Empty, Input, Pagination, Progress, Segmented, Skeleton, Tag, Tooltip } from 'antd'
import { FileTextOutlined, SearchOutlined } from '@ant-design/icons' import { FileTextOutlined, SearchOutlined } from '@ant-design/icons'
import dayjs from 'dayjs' import dayjs from 'dayjs'
import type { Run, Scenario, Target } from '../api' import type { Run, Scenario, Target } from '../api'
import type { RunProgress } from '../hooks/useRunSession' import type { RunProgress } from '../hooks/useRunSession'
import { colors, statusColors, statusLabels } from '../tokens' import { colors, statusColors, statusLabels, triggerColors, triggerLabels } from '../tokens'
import { elapsedStr, shortDateTime, toDate } from '../utils/date' import { elapsedStr, shortDateTime, toDate } from '../utils/date'
const PAGE_SIZE = 10 const PAGE_SIZE = 10
@ -292,6 +292,14 @@ function RunRow({ r, selected, targetName, scenarioName, onSelect, onOpenReport,
<span>{shortDateTime(r.started_at)}</span> <span>{shortDateTime(r.started_at)}</span>
<span>·</span> <span>·</span>
<span>{elapsedStr(r.started_at, r.completed_at)}</span> <span>{elapsedStr(r.started_at, r.completed_at)}</span>
{r.triggered_by && r.triggered_by !== 'manual' && (
<Tag
color={triggerColors[r.triggered_by] ?? 'default'}
style={{ marginLeft: 2, marginRight: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}
>
{triggerLabels[r.triggered_by] ?? r.triggered_by}
</Tag>
)}
{onOpenReport && status === 'completed' && ( {onOpenReport && status === 'completed' && (
<Tooltip title="查看报告"> <Tooltip title="查看报告">
<FileTextOutlined <FileTextOutlined

View File

@ -0,0 +1,23 @@
import { useEffect, useRef } from 'react'
import { useTabStore } from '../stores/tabStore'
/**
* Keep-alive tabs never remount, so "load on mount" data goes stale.
* Fires `callback` every time the given tab path becomes active again
* (skips the initial mount, which pages already handle themselves).
*/
export function useOnTabActive(path: string, callback: () => void) {
const activeKey = useTabStore((s) => s.activeKey)
const first = useRef(true)
const cbRef = useRef(callback)
cbRef.current = callback
useEffect(() => {
if (activeKey !== path) return
if (first.current) {
first.current = false
return
}
cbRef.current()
}, [activeKey, path])
}

View File

@ -1,56 +1,39 @@
import { useEffect, useState } from 'react' import { useEffect, useState } from 'react'
import { Button, Col, Row, Table, Tag, Spin, Card, Tooltip } from 'antd' import { Button, Card, Col, Empty, Progress, Row, Spin, Tag, Tooltip } from 'antd'
import { Line } from '@ant-design/charts' import { Line } from '@ant-design/charts'
import { useNavigate } from 'react-router-dom'
import { import {
AimOutlined, AimOutlined,
BarChartOutlined,
CheckCircleOutlined,
CloudServerOutlined,
FileTextOutlined, FileTextOutlined,
PlayCircleOutlined, PlayCircleOutlined,
CheckCircleOutlined, PlusOutlined,
ReloadOutlined, ReloadOutlined,
RobotOutlined,
SyncOutlined,
} from '@ant-design/icons' } from '@ant-design/icons'
import { runsApi, scenariosApi, targetsApi, type Run, type TrendPoint } from '../api' import { statsApi, type DashboardStats, type Run, type TrendPoint } from '../api'
import PageWrapper from '../components/PageWrapper' import PageWrapper from '../components/PageWrapper'
import StatCard from '../components/StatCard' import StatCard from '../components/StatCard'
import { useOnTabActive } from '../hooks/useOnTabActive'
import { colors, statusColors, statusLabels, triggerColors, triggerLabels } from '../tokens'
import { formatDateTime } from '../utils/date' import { formatDateTime } from '../utils/date'
import { passRateColor } from '../utils/colors'
export default function HomePage() { export default function HomePage() {
const navigate = useNavigate()
const [loading, setLoading] = useState(true) const [loading, setLoading] = useState(true)
const [counts, setCounts] = useState({ targets: 0, scenarios: 0, runs: 0 }) const [stats, setStats] = useState<DashboardStats | null>(null)
const [recentRuns, setRecentRuns] = useState<Run[]>([])
const [trend, setTrend] = useState<TrendPoint[]>([]) const [trend, setTrend] = useState<TrendPoint[]>([])
const [passRate, setPassRate] = useState<number | null>(null)
const loadData = async () => { const loadData = async () => {
setLoading(true) setLoading(true)
try { try {
const [t, s, r] = await Promise.all([ const [d, t] = await Promise.all([statsApi.dashboard(), statsApi.trend(30)])
targetsApi.list(), setStats(d.data)
scenariosApi.list(), setTrend(t.data)
runsApi.list(),
])
const runs = r.data
setCounts({
targets: t.data.length,
scenarios: s.data.length,
runs: runs.length,
})
setRecentRuns(runs.slice(-10).reverse())
const completedRuns = runs.filter((run) => run.status === 'completed' && run.summary)
if (completedRuns.length > 0) {
const totalRate = completedRuns.reduce(
(sum, run) => sum + ((run.summary as any)?.pass_rate || 0),
0,
)
setPassRate(totalRate / completedRuns.length)
}
const trendData: TrendPoint[] = completedRuns.map((run) => ({
date: run.started_at?.split('T')[0] || '',
pass_rate: ((run.summary as any)?.pass_rate || 0) * 100,
run_count: 1,
}))
setTrend(trendData)
} catch { } catch {
// errors handled by interceptor // errors handled by interceptor
} finally { } finally {
@ -59,39 +42,25 @@ export default function HomePage() {
} }
useEffect(() => { loadData() }, []) useEffect(() => { loadData() }, [])
useOnTabActive('/', loadData)
const statusColor: Record<string, string> = { const passRate = stats?.overall_pass_rate ?? null
completed: 'success',
running: 'processing',
pending: 'default',
failed: 'error',
}
const columns = [
{ title: 'Run ID', dataIndex: 'id', key: 'id', ellipsis: true, width: 240 },
{ title: '状态', dataIndex: 'status', key: 'status', width: 100,
render: (s: string) => <Tag color={statusColor[s] || 'default'}>{s}</Tag>,
},
{ title: '开始时间', dataIndex: 'started_at', key: 'started_at', width: 180,
render: (t: string) => t ? formatDateTime(t) : '-',
},
{ title: '通过率', key: 'pass_rate', width: 100,
render: (_: any, record: Run) => {
const rate = (record.summary as any)?.pass_rate
return rate != null ? `${(rate * 100).toFixed(0)}%` : '-'
},
},
]
const trendConfig = { const trendConfig = {
data: trend, data: trend,
xField: 'date', xField: 'date',
yField: 'pass_rate', yField: 'pass_rate',
smooth: true, smooth: true,
height: 200, height: 240,
yAxis: { label: { formatter: (v: string) => `${v}%` }, min: 0, max: 100 }, yAxis: { label: { formatter: (v: string) => `${v}%` }, min: 0, max: 100 },
point: { size: 3 }, point: { size: 3 },
color: '#1677ff', color: '#1677ff',
tooltip: {
formatter: (datum: TrendPoint) => ({
name: '通过率',
value: `${datum.pass_rate}%(${datum.run_count ?? '-'} 次执行)`,
}),
},
} }
return ( return (
@ -106,58 +75,163 @@ export default function HomePage() {
</Tooltip> </Tooltip>
} }
> >
{/* 内容区 — 可滚动 */} <div style={{ height: '100%', overflowY: 'auto', padding: '0 16px 16px' }}>
<div style={{ flex: 1, minHeight: 0, overflowY: 'auto', padding: '0 16px 16px' }}>
<Spin spinning={loading}> <Spin spinning={loading}>
{/* 指标卡行 */}
<Row gutter={[16, 16]}> <Row gutter={[16, 16]}>
<Col xs={24} sm={12} lg={6}> <Col xs={12} sm={8} xl={4}>
<StatCard icon={<AimOutlined />} color="blue" title="评测对象" value={counts.targets} /> <StatCard icon={<AimOutlined />} color="blue" title="评测对象" value={stats?.targets_count ?? 0} />
</Col> </Col>
<Col xs={24} sm={12} lg={6}> <Col xs={12} sm={8} xl={4}>
<StatCard icon={<FileTextOutlined />} color="green" title="评测场景" value={counts.scenarios} /> <StatCard icon={<FileTextOutlined />} color="green" title="评测场景" value={stats?.scenarios_count ?? 0} />
</Col> </Col>
<Col xs={24} sm={12} lg={6}> <Col xs={12} sm={8} xl={4}>
<StatCard icon={<PlayCircleOutlined />} color="orange" title="评测执行" value={counts.runs} /> <StatCard icon={<CloudServerOutlined />} color="purple" title="模型配置" value={stats?.model_configs_count ?? 0} />
</Col> </Col>
<Col xs={24} sm={12} lg={6}> <Col xs={12} sm={8} xl={4}>
<StatCard
icon={<PlayCircleOutlined />} color="orange" title="累计执行"
value={stats?.runs_count ?? 0}
suffix={stats?.today_runs ? ` / 今日 ${stats.today_runs}` : undefined}
/>
</Col>
<Col xs={12} sm={8} xl={4}>
<StatCard
icon={<SyncOutlined spin={(stats?.running_count ?? 0) > 0} />}
color={(stats?.running_count ?? 0) > 0 ? 'blue' : 'green'}
title="运行中"
value={stats?.running_count ?? 0}
valueStyle={(stats?.running_count ?? 0) > 0 ? { color: '#1677ff' } : undefined}
/>
</Col>
<Col xs={12} sm={8} xl={4}>
<StatCard <StatCard
icon={<CheckCircleOutlined />} icon={<CheckCircleOutlined />}
color="purple" color={passRate != null && passRate >= 0.8 ? 'green' : 'red'}
title="平均通过率" title="平均通过率"
value={passRate != null ? passRate * 100 : 0} value={passRate != null ? passRate * 100 : '-'}
suffix="%" suffix={passRate != null ? '%' : undefined}
precision={1} precision={1}
valueStyle={{ color: passRate != null && passRate >= 0.8 ? '#52c41a' : '#ff4d4f' }} valueStyle={{ color: passRate != null ? passRateColor(passRate) : colors.textMuted }}
/> />
</Col> </Col>
</Row> </Row>
{/* 图表行 */}
<Row gutter={[16, 16]} style={{ marginTop: 16 }}> <Row gutter={[16, 16]} style={{ marginTop: 16 }}>
<Col xs={24} lg={14}> <Col xs={24} lg={14}>
<Card title="最近评测记录"> <Card title="通过率趋势(近 30 天)" size="small" styles={{ body: { minHeight: 260 } }}>
<Table
dataSource={recentRuns}
columns={columns}
rowKey="id"
size="small"
pagination={false}
/>
</Card>
</Col>
<Col xs={24} lg={10}>
<Card title="通过率趋势">
{trend.length > 0 ? ( {trend.length > 0 ? (
<Line {...trendConfig} /> <Line {...trendConfig} />
) : ( ) : (
<div style={{ height: 200, display: 'flex', alignItems: 'center', justifyContent: 'center', color: '#9ca3af' }}> <Empty style={{ paddingTop: 60 }} description="暂无趋势数据" />
暂无趋势数据
</div>
)} )}
</Card> </Card>
</Col> </Col>
<Col xs={24} lg={10}>
<Card title="场景表现" size="small" styles={{ body: { minHeight: 260, overflowY: 'auto', maxHeight: 300 } }}>
{stats?.scenario_stats.length ? stats.scenario_stats.map((s) => (
<div
key={s.scenario_id}
onClick={() => navigate('/reports')}
style={{ padding: '8px 4px', cursor: 'pointer', borderBottom: `1px solid ${colors.border}` }}
>
<div style={{ display: 'flex', justifyContent: 'space-between', marginBottom: 4 }}>
<span style={{ fontSize: 13, fontWeight: 500 }}>{s.scenario_name}</span>
<span style={{ fontSize: 12, color: colors.textSecondary }}>
{s.run_count} 次 · {s.last_run_at ? formatDateTime(s.last_run_at) : '-'}
</span>
</div>
<Progress
percent={s.avg_pass_rate != null ? Math.round(s.avg_pass_rate * 100) : 0}
size="small"
strokeColor={s.avg_pass_rate != null ? passRateColor(s.avg_pass_rate) : colors.textMuted}
/>
</div>
)) : <Empty style={{ paddingTop: 60 }} description="暂无已完成的评测" />}
</Card>
</Col>
</Row>
{/* 动态行 */}
<Row gutter={[16, 16]} style={{ marginTop: 16 }}>
<Col xs={24} lg={14}>
<Card title="最近评测记录" size="small">
{stats?.recent_runs.length ? stats.recent_runs.map((r) => (
<RecentRunRow key={r.id} run={r} onOpen={() => navigate(r.status === 'completed' ? `/reports?run=${r.id}` : '/runs')} />
)) : <Empty description="还没有评测记录" />}
</Card>
</Col>
<Col xs={24} lg={10}>
<Card title="快捷操作" size="small" style={{ marginBottom: 16 }}>
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr 1fr', gap: 8 }}>
<Button icon={<PlayCircleOutlined />} onClick={() => navigate('/runs')}>发起评测</Button>
<Button icon={<PlusOutlined />} onClick={() => navigate('/scenarios')}>新建场景</Button>
<Button icon={<RobotOutlined />} onClick={() => navigate('/openclaw')}>AI 助手</Button>
</div>
</Card>
<Card title="触发来源分布" size="small">
{stats && Object.keys(stats.trigger_breakdown).length > 0 ? (
<div style={{ display: 'flex', flexDirection: 'column', gap: 10 }}>
{(Object.entries(stats.trigger_breakdown) as Array<[string, number]>).map(([k, count]) => (
<div key={k} style={{ display: 'flex', alignItems: 'center', gap: 8 }}>
<Tag color={triggerColors[k] ?? 'default'} style={{ margin: 0, width: 72, textAlign: 'center' }}>
{triggerLabels[k] ?? k}
</Tag>
<Progress
percent={stats.runs_count ? Math.round((count / stats.runs_count) * 100) : 0}
size="small"
style={{ flex: 1, margin: 0 }}
/>
<span style={{ fontSize: 12, color: colors.textSecondary, width: 40, textAlign: 'right' }}>{count} 次</span>
</div>
))}
</div>
) : <Empty image={Empty.PRESENTED_IMAGE_SIMPLE} description="暂无数据" />}
</Card>
</Col>
</Row> </Row>
</Spin> </Spin>
</div> </div>
</PageWrapper> </PageWrapper>
) )
} }
function RecentRunRow({ run, onOpen }: { run: Run; onOpen: () => void }) {
const rate = (run.summary as Record<string, unknown>)?.pass_rate as number | undefined
const dotColor = statusColors[run.status] ?? colors.textMuted
const trigger = run.triggered_by ?? 'manual'
return (
<div
className="run-row"
onClick={onOpen}
style={{
display: 'flex', alignItems: 'center', gap: 10,
padding: '8px 4px', cursor: 'pointer',
borderBottom: `1px solid ${colors.border}`,
}}
>
<span style={{ width: 8, height: 8, borderRadius: '50%', background: dotColor, flexShrink: 0 }} />
<div style={{ flex: 1, minWidth: 0, overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap', fontSize: 13 }}>
<span style={{ fontWeight: 500 }}>{run.scenario_name || run.scenario_id.slice(0, 8)}</span>
<span style={{ color: colors.textMuted }}> · {run.target_name || run.target_id.slice(0, 8)}</span>
</div>
{trigger !== 'manual' && (
<Tag color={triggerColors[trigger] ?? 'default'} style={{ margin: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}>
{triggerLabels[trigger] ?? trigger}
</Tag>
)}
<span style={{ fontSize: 12, color: dotColor, width: 48, flexShrink: 0 }}>{statusLabels[run.status] ?? run.status}</span>
<span style={{
fontSize: 12, fontWeight: 600, width: 44, textAlign: 'right', flexShrink: 0,
color: rate != null ? passRateColor(rate) : colors.textMuted,
}}>
{rate != null ? `${Math.round(rate * 100)}%` : '-'}
</span>
<span style={{ fontSize: 12, color: colors.textMuted, width: 130, textAlign: 'right', flexShrink: 0 }}>
{run.started_at ? formatDateTime(run.started_at) : '-'}
</span>
<BarChartOutlined style={{ color: colors.textMuted }} />
</div>
)
}

View File

@ -0,0 +1,52 @@
import { useState } from 'react'
import { Button, Card, Form, Input, message } from 'antd'
import { LockOutlined } from '@ant-design/icons'
import { authApi, authToken } from '../api'
import { colors } from '../tokens'
export default function LoginPage({ onSuccess }: { onSuccess: () => void }) {
const [loading, setLoading] = useState(false)
const submit = async ({ password }: { password: string }) => {
setLoading(true)
try {
const res = await authApi.login(password)
authToken.set(res.data.token)
message.success('登录成功')
onSuccess()
} catch {
// 错误提示由 axios 拦截器统一弹出(如"密码错误")
} finally {
setLoading(false)
}
}
return (
<div style={{
height: '100vh', display: 'flex', alignItems: 'center', justifyContent: 'center',
background: `linear-gradient(160deg, ${colors.sider} 0%, #2e3250 100%)`,
}}>
<Card style={{ width: 360, boxShadow: '0 8px 32px rgba(0,0,0,0.35)' }}>
<div style={{ textAlign: 'center', marginBottom: 24 }}>
<div style={{ fontSize: 22, fontWeight: 700, color: colors.text }}>AgentEvalTool</div>
<div style={{ fontSize: 13, color: colors.textSecondary, marginTop: 4 }}>
智能体质量评估平台
</div>
</div>
<Form onFinish={submit} autoComplete="off">
<Form.Item name="password" rules={[{ required: true, message: '请输入访问密码' }]}>
<Input.Password
size="large"
prefix={<LockOutlined style={{ color: colors.textMuted }} />}
placeholder="访问密码"
autoFocus
/>
</Form.Item>
<Button type="primary" htmlType="submit" size="large" block loading={loading}>
登录
</Button>
</Form>
</Card>
</div>
)
}

View File

@ -33,13 +33,6 @@ const modalityLabels: Record<ModelModality, string> = {
video: '视频', video: '视频',
} }
const modalityColors: Record<ModelModality, string> = {
text: 'default',
image: 'cyan',
audio: 'magenta',
video: 'purple',
}
const modalityOptions = Object.entries(modalityLabels).map(([value, label]) => ({ value, label })) const modalityOptions = Object.entries(modalityLabels).map(([value, label]) => ({ value, label }))
const featureLabels: Array<[keyof Pick< const featureLabels: Array<[keyof Pick<
@ -54,6 +47,13 @@ const featureLabels: Array<[keyof Pick<
const formatTokens = (value: number | null) => value ? new Intl.NumberFormat('zh-CN').format(value) : '-' const formatTokens = (value: number | null) => value ? new Intl.NumberFormat('zh-CN').format(value) : '-'
const compactTokens = (value: number | null) => {
if (!value) return '-'
if (value >= 1000000 && value % 100000 === 0) return `${value / 1000000}M`
if (value >= 1000) return `${Math.round(value / 1000)}K`
return String(value)
}
const protocolOptions: Record<ModelProtocol, { const protocolOptions: Record<ModelProtocol, {
label: string label: string
capabilities: ModelCapability[] capabilities: ModelCapability[]
@ -207,7 +207,8 @@ export default function ModelConfigsPage() {
const columns = [ const columns = [
{ {
title: '配置名称', dataIndex: 'name', key: 'name', width: 220, // 配置名称 + 默认/厂商/区域 + Endpoint 摘要
title: '配置名称', dataIndex: 'name', key: 'name',
render: (value: string, record: ModelConfig) => ( render: (value: string, record: ModelConfig) => (
<div style={{ minWidth: 0 }}> <div style={{ minWidth: 0 }}>
<Space size={6}> <Space size={6}>
@ -216,84 +217,96 @@ export default function ModelConfigsPage() {
</Tooltip> </Tooltip>
{record.is_default && <Tag color="gold" style={{ margin: 0 }}>默认</Tag>} {record.is_default && <Tag color="gold" style={{ margin: 0 }}>默认</Tag>}
</Space> </Space>
{(record.vendor_name || record.region) && ( <Tooltip title={record.endpoint_url}>
<div style={{
color: '#8c8c8c', fontSize: 12, marginTop: 3,
overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap', maxWidth: 260,
}}>
{[record.vendor_name, record.region].filter(Boolean).join(' · ') || record.endpoint_url}
</div>
</Tooltip>
</div>
),
},
{
// 模型名 + 协议
title: '模型 / 协议', key: 'model', width: 200,
render: (_: unknown, record: ModelConfig) => (
<div style={{ minWidth: 0 }}>
<div style={{
fontWeight: 500, overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap', maxWidth: 180,
}}>
{record.model_name || '-'}
</div>
<div style={{ color: '#8c8c8c', fontSize: 12, marginTop: 3 }}> <div style={{ color: '#8c8c8c', fontSize: 12, marginTop: 3 }}>
{[record.vendor_name, record.region].filter(Boolean).join(' · ')} {protocolOptions[record.provider]?.label || record.provider}
</div>
</div>
),
},
{
// 能力 + 输入/输出模态
title: '能力 / 模态', key: 'capability', width: 200,
render: (_: unknown, record: ModelConfig) => {
const modalityLine = (values: ModelModality[]) =>
values.map((v) => modalityLabels[v]).join('、')
return (
<Space size={6} align="start">
<Tag color={capabilityColors[record.capability]} style={{ margin: 0 }}>
{capabilityLabels[record.capability]}
</Tag>
<span style={{ color: '#8c8c8c', fontSize: 12, lineHeight: '22px' }}>
{modalityLine(record.input_modalities)} → {modalityLine(record.output_modalities)}
</span>
</Space>
)
},
},
{
// 上下文/最大输出 + 特性
title: '规格 / 特性', key: 'spec', width: 210,
render: (_: unknown, record: ModelConfig) => {
const enabledFeatures = featureLabels.filter(([key]) => record[key])
return (
<div style={{ fontSize: 12 }}>
<Tooltip title={`上下文 ${formatTokens(record.context_window)} / 最大输出 ${formatTokens(record.max_output_tokens)}`}>
<span>
<span style={{ color: '#8c8c8c' }}>上下文 </span>{compactTokens(record.context_window)}
<span style={{ color: '#8c8c8c' }}> 输出 </span>{compactTokens(record.max_output_tokens)}
</span>
</Tooltip>
{enabledFeatures.length > 0 && (
<div style={{ marginTop: 4, display: 'flex', gap: 4, flexWrap: 'wrap' }}>
{enabledFeatures.map(([key, label]) => (
<Tag key={key} style={{ margin: 0, fontSize: 11, lineHeight: '18px', padding: '0 5px' }}>{label}</Tag>
))}
</div> </div>
)} )}
</div> </div>
), )
},
}, },
{ {
title: '协议', dataIndex: 'provider', key: 'provider', width: 150, // 状态 + 凭据 + 更新时间
render: (value: ModelProtocol) => <Tag>{protocolOptions[value]?.label || value}</Tag>, title: '状态', key: 'status', width: 150,
},
{
title: '能力', dataIndex: 'capability', key: 'capability', width: 110,
render: (value: ModelCapability) => <Tag color={capabilityColors[value]}>{capabilityLabels[value]}</Tag>,
},
{
title: '输入 / 输出', key: 'modalities', width: 230,
render: (_: unknown, record: ModelConfig) => ( render: (_: unknown, record: ModelConfig) => (
<Space direction="vertical" size={3}> <div>
<Space size={4} wrap> <Space size={4}>
<span style={{ color: '#8c8c8c', fontSize: 12, width: 28 }}>输入</span> {record.enabled ? <Tag color="success" style={{ margin: 0 }}>启用</Tag> : <Tag style={{ margin: 0 }}>停用</Tag>}
{record.input_modalities.map((value) => ( {record.has_api_key
<Tag key={`input-${value}`} color={modalityColors[value]} style={{ margin: 0 }}> ? <Tag color="blue" style={{ margin: 0 }}>有凭据</Tag>
{modalityLabels[value]} : <Tag style={{ margin: 0 }}>无凭据</Tag>}
</Tag>
))}
</Space> </Space>
<Space size={4} wrap> <Tooltip title={`更新于 ${formatDateTime(record.updated_at)}`}>
<span style={{ color: '#8c8c8c', fontSize: 12, width: 28 }}>输出</span> <div style={{ color: '#8c8c8c', fontSize: 12, marginTop: 3 }}>
{record.output_modalities.map((value) => ( {formatDateTime(record.updated_at)}
<Tag key={`output-${value}`} color={modalityColors[value]} style={{ margin: 0 }}> </div>
{modalityLabels[value]} </Tooltip>
</Tag>
))}
</Space>
</Space>
),
},
{
title: '模型', dataIndex: 'model_name', key: 'model_name', width: 180, ellipsis: true,
render: (value: string | null) => value || '-',
},
{
title: '规格', key: 'limits', width: 170,
render: (_: unknown, record: ModelConfig) => (
<div style={{ fontSize: 12, lineHeight: 1.7 }}>
<div><span style={{ color: '#8c8c8c' }}>上下文 </span>{formatTokens(record.context_window)}</div>
<div><span style={{ color: '#8c8c8c' }}>最大输出 </span>{formatTokens(record.max_output_tokens)}</div>
</div> </div>
), ),
}, },
{ {
title: '特性', key: 'features', width: 210, title: '操作', key: 'actions', width: 110,
render: (_: unknown, record: ModelConfig) => {
const enabledFeatures = featureLabels.filter(([key]) => record[key])
return enabledFeatures.length ? (
<Space size={[4, 4]} wrap>
{enabledFeatures.map(([key, label]) => <Tag key={key} style={{ margin: 0 }}>{label}</Tag>)}
</Space>
) : '-'
},
},
{ title: 'Endpoint', dataIndex: 'endpoint_url', key: 'endpoint_url', width: 260, ellipsis: true },
{
title: '凭据', dataIndex: 'has_api_key', key: 'has_api_key', width: 90,
render: (value: boolean) => value ? <Tag color="success">已配置</Tag> : <Tag>无</Tag>,
},
{
title: '状态', dataIndex: 'enabled', key: 'enabled', width: 90,
render: (value: boolean) => value ? <Tag color="success">启用</Tag> : <Tag>停用</Tag>,
},
{
title: '更新时间', dataIndex: 'updated_at', key: 'updated_at', width: 170,
render: (value: string | null) => formatDateTime(value),
},
{
title: '操作', key: 'actions', width: 145, fixed: 'right' as const,
render: (_: unknown, record: ModelConfig) => ( render: (_: unknown, record: ModelConfig) => (
<Space size={2}> <Space size={2}>
<Tooltip title="连接测试"> <Tooltip title="连接测试">
@ -355,8 +368,9 @@ export default function ModelConfigsPage() {
rowKey="id" rowKey="id"
loading={loading} loading={loading}
pagination={{ pageSize: 15, showSizeChanger: true, showTotal: (total) => `共 ${total} 个` }} pagination={{ pageSize: 15, showSizeChanger: true, showTotal: (total) => `共 ${total} 个` }}
scroll={{ x: 1800, y: 'calc(100vh - 200px)' }} scroll={{ y: 'calc(100vh - 200px)' }}
style={{ height: '100%' }} style={{ height: '100%' }}
tableLayout="auto"
/> />
</div> </div>

View File

@ -1,17 +1,18 @@
import { useEffect, useState } from 'react' import { useEffect, useMemo, useState } from 'react'
import { useSearchParams } from 'react-router-dom' import { useSearchParams } from 'react-router-dom'
import { import {
Button, Card, Col, Collapse, Descriptions, Empty, Row, Segmented, Button, Card, Col, Collapse, Descriptions, Empty, Row, Segmented,
Select, Space, Spin, Statistic, Table, Tag, Tooltip, Badge, Select, Space, Spin, Statistic, Table, Tag, Tooltip, Badge, message,
} from 'antd' } from 'antd'
import { import {
CheckCircleOutlined, CloseCircleOutlined, CheckCircleOutlined, CloseCircleOutlined, ClearOutlined,
DownloadOutlined, UserOutlined, RobotOutlined, ReloadOutlined, DownloadOutlined, UserOutlined, RobotOutlined, ReloadOutlined,
DiffOutlined, FileMarkdownOutlined, DiffOutlined, FileMarkdownOutlined,
} from '@ant-design/icons' } from '@ant-design/icons'
import { reportsApi, runsApi, type Run } from '../api' import { reportsApi, runsApi, type Run } from '../api'
import { colors } from '../tokens' import { colors, triggerColors, triggerLabels } from '../tokens'
import { formatDateTime } from '../utils/date' import { formatDateTime } from '../utils/date'
import { useOnTabActive } from '../hooks/useOnTabActive'
interface TurnData { interface TurnData {
sent_text: string sent_text: string
@ -72,6 +73,7 @@ export default function ReportsPage() {
const runQuery = searchParams.get('run') ?? '' const runQuery = searchParams.get('run') ?? ''
const [runs, setRuns] = useState<Run[]>([]) const [runs, setRuns] = useState<Run[]>([])
const [scenarioFilter, setScenarioFilter] = useState<string>('')
const [selectedRunId, setSelectedRunId] = useState<string>('') const [selectedRunId, setSelectedRunId] = useState<string>('')
const [compareRunId, setCompareRunId] = useState<string>('') const [compareRunId, setCompareRunId] = useState<string>('')
const [report, setReport] = useState<Report | null>(null) const [report, setReport] = useState<Report | null>(null)
@ -93,8 +95,8 @@ export default function ReportsPage() {
const loadRuns = async () => { const loadRuns = async () => {
const res = await runsApi.list() const res = await runsApi.list()
const completed = res.data.filter((r) => r.status === 'completed').reverse() // API 已按 started_at DESC 排序,最新的在前
setRuns(completed) setRuns(res.data.filter((r) => r.status === 'completed'))
} }
useEffect(() => { useEffect(() => {
@ -104,6 +106,10 @@ export default function ReportsPage() {
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
}, []) }, [])
// Keep-alive tabs never remount: refresh whenever this tab is re-activated
// so runs triggered elsewhere (e.g. AI assistant) show up.
useOnTabActive('/reports', loadRuns)
useEffect(() => { useEffect(() => {
if (runQuery && runQuery !== selectedRunId) { if (runQuery && runQuery !== selectedRunId) {
loadReport(runQuery) loadReport(runQuery)
@ -114,6 +120,10 @@ export default function ReportsPage() {
const handleView = (runId: string) => { const handleView = (runId: string) => {
setSearchParams({ run: runId }) setSearchParams({ run: runId })
loadReport(runId) loadReport(runId)
// 对比报告要求同场景:A 变更后若 B 场景不同则清空
const a = runs.find((r) => r.id === runId)
const b = runs.find((r) => r.id === compareRunId)
if (a && b && a.scenario_id !== b.scenario_id) setCompareRunId('')
} }
const handleCompare = async () => { const handleCompare = async () => {
@ -122,15 +132,82 @@ export default function ReportsPage() {
try { try {
const res = await reportsApi.compare(selectedRunId, compareRunId) const res = await reportsApi.compare(selectedRunId, compareRunId)
setCompareResult(res.data as CompareResult) setCompareResult(res.data as CompareResult)
} catch (e: any) {
const detail = e?.response?.data?.detail
if (detail) message.error(detail)
} finally { } finally {
setLoading(false) setLoading(false)
} }
} }
const runSelectOptions = runs.map((r) => ({ const clearSelection = () => {
setSelectedRunId('')
setCompareRunId('')
setReport(null)
setCompareResult(null)
setSearchParams({})
}
const resetFilters = () => {
setScenarioFilter('')
clearSelection()
}
// 场景筛选项:从 run 列表聚合(含各场景 run 数量)
const scenarioOptions = useMemo(() => {
const map = new Map<string, { name: string; count: number }>()
for (const r of runs) {
const entry = map.get(r.scenario_id)
if (entry) entry.count += 1
else map.set(r.scenario_id, { name: r.scenario_name || r.scenario_id.slice(0, 8), count: 1 })
}
return [...map.entries()].map(([id, s]) => ({ value: id, label: `${s.name}(${s.count})` }))
}, [runs])
const filteredRuns = useMemo(
() => (scenarioFilter ? runs.filter((r) => r.scenario_id === scenarioFilter) : runs),
[runs, scenarioFilter],
)
const buildOption = (r: Run) => {
const passRate = (r.summary as Record<string, unknown>)?.pass_rate as number | undefined
const pct = passRate != null ? `${Math.round(passRate * 100)}%` : '-'
const scenario = r.scenario_name || r.scenario_id.slice(0, 8)
const target = r.target_name || r.target_id.slice(0, 8)
const time = r.started_at ? formatDateTime(r.started_at) : ''
const trigger = r.triggered_by ?? 'manual'
return {
value: r.id, value: r.id,
label: `${r.id.slice(0, 8)}… | ${r.started_at ? formatDateTime(r.started_at) : ''}`, searchText: `${scenario} ${target} ${time} ${r.id} ${triggerLabels[trigger] ?? trigger}`.toLowerCase(),
})) label: (
<span style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
<span style={{ fontWeight: 500, overflow: 'hidden', textOverflow: 'ellipsis' }}>{scenario}</span>
<span style={{ color: colors.textMuted }}>· {target}</span>
<span style={{ color: colors.textMuted, fontSize: 12 }}>{time}</span>
<span style={{
color: passRate != null && passRate >= 0.8 ? '#3f8600' : '#cf1322',
fontSize: 12, fontWeight: 600,
}}>{pct}</span>
{trigger !== 'manual' && (
<Tag color={triggerColors[trigger] ?? 'default'} style={{ marginRight: 0, fontSize: 10, lineHeight: '16px', padding: '0 4px' }}>
{triggerLabels[trigger] ?? trigger}
</Tag>
)}
</span>
),
}
}
const runSelectOptions = filteredRuns.map(buildOption)
// 报告 B 只能选与报告 A 同场景的 run
const selectedRun = runs.find((r) => r.id === selectedRunId)
const compareOptions = runs
.filter((r) => r.id !== selectedRunId && selectedRun && r.scenario_id === selectedRun.scenario_id)
.map(buildOption)
const optionFilter = (input: string, opt?: { searchText?: string }) =>
(opt?.searchText ?? '').includes(input.toLowerCase())
return ( return (
<div style={{ height: '100%', display: 'flex', flexDirection: 'column', overflow: 'hidden' }}> <div style={{ height: '100%', display: 'flex', flexDirection: 'column', overflow: 'hidden' }}>
@ -144,13 +221,15 @@ export default function ReportsPage() {
<span style={{ fontSize: 13, color: colors.textSecondary }}>查看评测结果详情与统计分析</span> <span style={{ fontSize: 13, color: colors.textSecondary }}>查看评测结果详情与统计分析</span>
</div> </div>
{/* 选择栏 */} {/* 选择栏:两行布局,避免对比模式下控件换行错乱 */}
<div style={{ <div style={{
padding: '8px 16px', flexShrink: 0, padding: '8px 16px', flexShrink: 0,
borderBottom: `1px solid ${colors.border}`, borderBottom: `1px solid ${colors.border}`,
background: colors.bgSubtle, background: colors.bgSubtle,
display: 'flex', alignItems: 'center', gap: 12, flexWrap: 'wrap', display: 'flex', flexDirection: 'column', gap: 8,
}}> }}>
{/* 第一行:视图模式 + 场景筛选 + 重置/刷新 */}
<div style={{ display: 'flex', alignItems: 'center', gap: 12 }}>
<Segmented <Segmented
value={viewMode} value={viewMode}
onChange={(v) => { setViewMode(v as ViewMode); setCompareResult(null) }} onChange={(v) => { setViewMode(v as ViewMode); setCompareResult(null) }}
@ -159,36 +238,63 @@ export default function ReportsPage() {
{ label: '对比报告', value: 'compare', icon: <DiffOutlined /> }, { label: '对比报告', value: 'compare', icon: <DiffOutlined /> },
]} ]}
/> />
<Select <Select
style={{ width: 340 }} style={{ width: 320 }}
value={scenarioFilter || undefined}
placeholder={`全部场景(${runs.length})`}
allowClear
onChange={(v) => setScenarioFilter(v ?? '')}
options={scenarioOptions}
popupMatchSelectWidth={false}
/>
<div style={{ marginLeft: 'auto', display: 'flex', gap: 8 }}>
<Tooltip title="重置筛选与选择">
<Button size="middle" icon={<ClearOutlined />} onClick={resetFilters}>重置</Button>
</Tooltip>
<Tooltip title="刷新列表">
<Button size="middle" icon={<ReloadOutlined />} onClick={loadRuns} />
</Tooltip>
</div>
</div>
{/* 第二行:报告选择 + 操作 */}
<div style={{ display: 'flex', alignItems: 'center', gap: 8 }}>
<Select
style={{ flex: 1, minWidth: 320, maxWidth: 680 }}
placeholder={viewMode === 'compare' ? '选择报告 A' : '选择已完成的评测记录'} placeholder={viewMode === 'compare' ? '选择报告 A' : '选择已完成的评测记录'}
value={selectedRunId || undefined} value={selectedRunId || undefined}
onChange={handleView} onChange={(v) => (v ? handleView(v) : clearSelection())}
allowClear
showSearch showSearch
filterOption={(input, opt) => filterOption={optionFilter}
(opt?.label as string ?? '').toLowerCase().includes(input.toLowerCase())}
options={runSelectOptions} options={runSelectOptions}
popupMatchSelectWidth={false}
/> />
{viewMode === 'compare' && ( {viewMode === 'compare' && (
<> <>
<span style={{ color: colors.textMuted, fontSize: 12 }}>vs</span> <span style={{ color: colors.textMuted, fontSize: 12, flexShrink: 0 }}>vs</span>
<Tooltip title={selectedRunId ? '仅可选择与报告 A 相同场景的记录' : '请先选择报告 A'}>
<Select <Select
style={{ width: 340 }} style={{ flex: 1, minWidth: 320, maxWidth: 680 }}
placeholder="选择报告 B" placeholder="选择报告 B(同场景)"
value={compareRunId || undefined} value={compareRunId || undefined}
onChange={setCompareRunId} onChange={(v) => { setCompareRunId(v ?? ''); if (!v) setCompareResult(null) }}
allowClear
showSearch showSearch
filterOption={(input, opt) => disabled={!selectedRunId}
(opt?.label as string ?? '').toLowerCase().includes(input.toLowerCase())} filterOption={optionFilter}
options={runSelectOptions.filter((o) => o.value !== selectedRunId)} options={compareOptions}
popupMatchSelectWidth={false}
notFoundContent={<Empty image={Empty.PRESENTED_IMAGE_SIMPLE} description="没有同场景的其他评测记录" />}
/> />
</Tooltip>
<Button <Button
type="primary" type="primary"
icon={<DiffOutlined />} icon={<DiffOutlined />}
disabled={!selectedRunId || !compareRunId} disabled={!selectedRunId || !compareRunId}
onClick={handleCompare} onClick={handleCompare}
style={{ flexShrink: 0 }}
> >
对比 对比
</Button> </Button>
@ -196,7 +302,7 @@ export default function ReportsPage() {
)} )}
{viewMode === 'single' && report && ( {viewMode === 'single' && report && (
<Space> <Space style={{ flexShrink: 0 }}>
<Button icon={<DownloadOutlined />} onClick={() => window.open(`/api/reports/${selectedRunId}/html`, '_blank')}> <Button icon={<DownloadOutlined />} onClick={() => window.open(`/api/reports/${selectedRunId}/html`, '_blank')}>
导出 HTML 导出 HTML
</Button> </Button>
@ -205,11 +311,6 @@ export default function ReportsPage() {
</Button> </Button>
</Space> </Space>
)} )}
<div style={{ marginLeft: 'auto' }}>
<Tooltip title="刷新列表">
<Button size="middle" icon={<ReloadOutlined />} onClick={loadRuns} />
</Tooltip>
</div> </div>
</div> </div>

View File

@ -13,6 +13,7 @@ import RuleOverview from '../components/RuleOverview'
import CaseDetail from '../components/CaseDetail' import CaseDetail from '../components/CaseDetail'
import { useRunSession } from '../hooks/useRunSession' import { useRunSession } from '../hooks/useRunSession'
import { useTicker } from '../hooks/useTicker' import { useTicker } from '../hooks/useTicker'
import { useOnTabActive } from '../hooks/useOnTabActive'
import { colors, statusColors, statusLabels } from '../tokens' import { colors, statusColors, statusLabels } from '../tokens'
import { formatDateTime, elapsedStr } from '../utils/date' import { formatDateTime, elapsedStr } from '../utils/date'
import { passRateColor } from '../utils/colors' import { passRateColor } from '../utils/colors'
@ -52,6 +53,10 @@ export default function RunsPage() {
useEffect(() => { loadLists() }, []) useEffect(() => { loadLists() }, [])
// Keep-alive tabs never remount: refresh the list whenever the tab is
// re-activated so runs triggered elsewhere (e.g. AI assistant) show up.
useOnTabActive('/runs', loadLists)
useEffect(() => { useEffect(() => {
if (!session.isLive) return if (!session.isLive) return
const id = window.setInterval(() => { const id = window.setInterval(() => {

View File

@ -43,6 +43,18 @@ export const statusLabels: Record<string, string> = {
cancelled: '已取消', cancelled: '已取消',
} }
export const triggerLabels: Record<string, string> = {
manual: '手动',
ai_assistant: 'AI 助手',
cli: 'CLI',
}
export const triggerColors: Record<string, string> = {
manual: 'default',
ai_assistant: 'purple',
cli: 'blue',
}
export const spacing = { export const spacing = {
xs: 4, xs: 4,
sm: 8, sm: 8,

View File

@ -22,6 +22,14 @@ export default defineConfig({
}, },
}, },
build: { build: {
// Accepted oversized chunks (measured 2026-07-28):
// vendor ~2.12 MB min / ~650 kB gzip — react + @ant-design/charts
// (charts CANNOT be split out: forced separate chunk creates
// a circular reference and a runtime TDZ crash, and charts
// loads on the default Home route anyway)
// vendor-antd ~620 kB min / ~173 kB gzip
// Warning fires only above this limit, i.e. on a real size regression.
chunkSizeWarningLimit: 2200,
rollupOptions: { rollupOptions: {
output: { output: {
manualChunks(id) { manualChunks(id) {
@ -37,8 +45,8 @@ export default defineConfig({
// NOTE: do NOT split @ant-design/charts into its own chunk — its deep // NOTE: do NOT split @ant-design/charts into its own chunk — its deep
// @antv dependency graph creates a circular chunk reference that causes // @antv dependency graph creates a circular chunk reference that causes
// a runtime TDZ crash ("Cannot access 'd' before initialization"). // a runtime TDZ crash ("Cannot access 'd' before initialization").
// charts is loaded on the Home page (default route) anyway, so a // Returning undefined for @antv doesn't help either: rollup still
// separate chunk buys almost nothing. // merges it into vendor (verified 2026-07-28).
if (id.includes('node_modules')) { if (id.includes('node_modules')) {
return 'vendor' return 'vendor'
} }

View File

@ -0,0 +1,29 @@
"""add triggered_by to eval_runs
Revision ID: b7d4e6f81c22
Revises: a64b2f8c9d10
Create Date: 2026-07-27
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "b7d4e6f81c22"
down_revision: Union[str, Sequence[str], None] = "a64b2f8c9d10"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("eval_runs") as batch_op:
batch_op.add_column(
sa.Column("triggered_by", sa.String(), nullable=False, server_default="manual")
)
def downgrade() -> None:
with op.batch_alter_table("eval_runs") as batch_op:
batch_op.drop_column("triggered_by")

View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "agenteval" name = "agenteval"
version = "0.3.0-dev" version = "0.4.0-dev"
description = "智能体质量评估工具集平台" description = "智能体质量评估工具集平台"
readme = "README.md" readme = "README.md"
requires-python = ">=3.10" requires-python = ">=3.10"

View File

@ -92,6 +92,23 @@ run rsync -az --delete \
--exclude='*.db*' \ --exclude='*.db*' \
./ "$HOST:$REMOTE_DIR/" ./ "$HOST:$REMOTE_DIR/"
# ── OpenClaw skill sync ────────────────────────────────────────────────
# The agenteval-run skill lives in the openclaw workspace volume (data/),
# which rsync excludes. Sync the versioned copy explicitly so the AI
# assistant always follows the standard HTTP API flow.
SKILL_SRC="backend/plugins/openclaw/skills/agenteval-run/SKILL.md"
SKILL_DST="$REMOTE_DIR/data/openclaw/workspace/skills/agenteval-run/SKILL.md"
log "sync OpenClaw skill → $SKILL_DST"
run ssh "$HOST" "mkdir -p $(dirname "$SKILL_DST")"
run rsync -az "$SKILL_SRC" "$HOST:$SKILL_DST"
# Provision the API key file the skill reads (~/.openclaw/agenteval-api-key in
# the openclaw container = data/openclaw/agenteval-api-key on the host). Sourced
# from AGENTEVAL_API_KEY in the remote .env; removed when the key is unset.
run ssh "$HOST" "KEY=\$(grep -E '^AGENTEVAL_API_KEY=.+' $REMOTE_DIR/.env 2>/dev/null | cut -d= -f2-); \
if [ -n \"\$KEY\" ]; then printf '%s' \"\$KEY\" > $REMOTE_DIR/data/openclaw/agenteval-api-key; \
else rm -f $REMOTE_DIR/data/openclaw/agenteval-api-key; fi"
# ── image rebuild ────────────────────────────────────────────────────── # ── image rebuild ──────────────────────────────────────────────────────
if [[ "$SKIP_BUILD" == "1" ]]; then if [[ "$SKIP_BUILD" == "1" ]]; then
warn "skipping image rebuild (--skip-build)" warn "skipping image rebuild (--skip-build)"

View File

@ -0,0 +1,89 @@
"""Integration tests for the login gate: /api/auth/* and require_auth."""
import pytest
from fastapi.testclient import TestClient
from sqlmodel import Session, SQLModel, create_engine
from agenteval.config.settings import Settings
from agenteval.web.app import app
from agenteval.web.deps import get_db
@pytest.fixture()
def client_with_db(tmp_path):
from agenteval.storage.db import ( # noqa: F401
EvalResultDB, EvalRunDB, EvalTargetDB, ScenarioDB, TurnDB,
)
engine = create_engine(
f"sqlite:///{tmp_path / 'auth_api.db'}",
connect_args={"check_same_thread": False},
)
SQLModel.metadata.create_all(engine)
session = Session(engine)
def override_get_db():
try:
yield session
finally:
pass
app.dependency_overrides[get_db] = override_get_db
client = TestClient(app)
yield client
app.dependency_overrides.clear()
session.close()
def _patch_settings(monkeypatch, **kwargs):
settings = Settings(_env_file=None, **kwargs)
from agenteval.web import deps as deps_module
from agenteval.web.routers import auth as auth_module
monkeypatch.setattr(deps_module, "get_settings", lambda: settings)
monkeypatch.setattr(auth_module, "get_settings", lambda: settings)
return settings
def test_no_credentials_configured_is_open(client_with_db, monkeypatch):
_patch_settings(monkeypatch)
assert client_with_db.get("/api/targets").status_code == 200
status = client_with_db.get("/api/auth/status").json()
assert status == {"auth_required": False, "authenticated": True}
def test_login_disabled_returns_400(client_with_db, monkeypatch):
_patch_settings(monkeypatch)
resp = client_with_db.post("/api/auth/login", json={"password": "whatever"})
assert resp.status_code == 400
def test_admin_password_gates_api(client_with_db, monkeypatch):
_patch_settings(monkeypatch, admin_password="s3cret")
# No credentials → 401.
assert client_with_db.get("/api/targets").status_code == 401
# Wrong password → 401.
assert client_with_db.post("/api/auth/login", json={"password": "nope"}).status_code == 401
# Correct password → token that unlocks the API.
token = client_with_db.post("/api/auth/login", json={"password": "s3cret"}).json()["token"]
resp = client_with_db.get("/api/targets", headers={"X-Auth-Token": token})
assert resp.status_code == 200
status = client_with_db.get("/api/auth/status", headers={"X-Auth-Token": token}).json()
assert status == {"auth_required": True, "authenticated": True}
assert client_with_db.get("/api/auth/status").json()["authenticated"] is False
def test_api_key_still_works_alongside_login(client_with_db, monkeypatch):
_patch_settings(monkeypatch, admin_password="s3cret", api_key="machine-key")
assert client_with_db.get("/api/targets").status_code == 401
assert client_with_db.get("/api/targets", headers={"X-API-Key": "machine-key"}).status_code == 200
assert client_with_db.get("/api/targets", headers={"X-API-Key": "wrong"}).status_code == 401
def test_health_stays_open(client_with_db, monkeypatch):
_patch_settings(monkeypatch, admin_password="s3cret")
assert client_with_db.get("/api/health").status_code == 200

View File

@ -39,7 +39,7 @@ def client_with_db(tmp_path):
session.close() session.close()
def _seed_run(session: Session, name_suffix: str = "") -> str: def _seed_run(session: Session, name_suffix: str = "", scenario_id: str | None = None) -> str:
target = EvalTarget( target = EvalTarget(
name=f"target{name_suffix}", name=f"target{name_suffix}",
platform=PlatformType.AI_DIGITAL_EMPLOYEE, platform=PlatformType.AI_DIGITAL_EMPLOYEE,
@ -49,15 +49,17 @@ def _seed_run(session: Session, name_suffix: str = "") -> str:
) )
target = TargetRepository(session).create(target) target = TargetRepository(session).create(target)
if scenario_id is None:
scenario = Scenario( scenario = Scenario(
name=f"scenario{name_suffix}", name=f"scenario{name_suffix}",
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])], cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
) )
scenario = ScenarioRepository(session).create(scenario) scenario = ScenarioRepository(session).create(scenario)
scenario_id = scenario.id
run = EvalRun( run = EvalRun(
target_id=target.id, target_id=target.id,
scenario_id=scenario.id, scenario_id=scenario_id,
status=RunStatus.COMPLETED, status=RunStatus.COMPLETED,
) )
run = RunRepository(session).create(run) run = RunRepository(session).create(run)
@ -143,7 +145,8 @@ def test_get_markdown_report_attachment_header(client_with_db):
def test_compare_report(client_with_db): def test_compare_report(client_with_db):
client, session = client_with_db client, session = client_with_db
run_id_a = _seed_run(session, "A") run_id_a = _seed_run(session, "A")
run_id_b = _seed_run(session, "B") sid = RunRepository(session).get(run_id_a).scenario_id
run_id_b = _seed_run(session, "B", scenario_id=sid)
resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}") resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}")
assert resp.status_code == 200 assert resp.status_code == 200
data = resp.json() data = resp.json()
@ -153,6 +156,15 @@ def test_compare_report(client_with_db):
assert "cases" in data assert "cases" in data
def test_compare_report_different_scenarios_400(client_with_db):
client, session = client_with_db
run_id_a = _seed_run(session, "A")
run_id_b = _seed_run(session, "B") # separate scenario
resp = client.get(f"/api/reports/compare?run1={run_id_a}&run2={run_id_b}")
assert resp.status_code == 400
assert "相同场景" in resp.json()["detail"]
def test_compare_report_run_not_found(client_with_db): def test_compare_report_run_not_found(client_with_db):
client, session = client_with_db client, session = client_with_db
run_id = _seed_run(session) run_id = _seed_run(session)

View File

@ -196,3 +196,45 @@ async def test_start_run_missing_target(client, seeded_db):
"target_id": "does-not-exist", "scenario_id": "s-1", "target_id": "does-not-exist", "scenario_id": "s-1",
}) })
assert resp.status_code == 404 assert resp.status_code == 404
# ── triggered_by ─────────────────────────────────────────────────────────
async def test_start_run_default_triggered_by_manual(client, seeded_db, mock_channel):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1",
})
assert resp.status_code == 200
assert resp.json()["triggered_by"] == "manual"
async def test_start_run_ai_assistant_triggered_by(client, seeded_db, mock_channel):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "ai_assistant",
})
assert resp.status_code == 200
run_id = resp.json()["id"]
assert resp.json()["triggered_by"] == "ai_assistant"
# Persisted, not just echoed.
got = (await client.get(f"/api/runs/{run_id}")).json()
assert got["triggered_by"] == "ai_assistant"
async def test_start_run_invalid_triggered_by_422(client, seeded_db):
resp = await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "robot",
})
assert resp.status_code == 422
async def test_list_runs_includes_names_and_trigger(client, seeded_db, mock_channel):
await client.post("/api/runs", json={
"target_id": "t-1", "scenario_id": "s-1", "triggered_by": "ai_assistant",
})
listing = (await client.get("/api/runs")).json()
assert len(listing) == 1
row = listing[0]
assert row["scenario_name"] == "mock-scenario"
assert row["target_name"] == "mock-target"
assert row["triggered_by"] == "ai_assistant"

View File

@ -0,0 +1,106 @@
"""Integration tests for /api/stats/dashboard aggregation."""
import pytest
from fastapi.testclient import TestClient
from sqlmodel import Session, SQLModel, create_engine
from agenteval.models import (
Case, CaseType, ChannelType, EvalRun, EvalTarget, PlatformType,
RunStatus, RunTrigger, Scenario, TargetStatus,
)
from agenteval.storage.repository import RunRepository, ScenarioRepository, TargetRepository
from agenteval.web.app import app
from agenteval.web.deps import get_db
@pytest.fixture()
def client_with_db(tmp_path):
from agenteval.storage.db import ( # noqa: F401
EvalResultDB, EvalRunDB, EvalTargetDB, ModelConfigDB, ScenarioDB, TurnDB,
)
engine = create_engine(
f"sqlite:///{tmp_path / 'stats_api.db'}",
connect_args={"check_same_thread": False},
)
SQLModel.metadata.create_all(engine)
session = Session(engine)
def override_get_db():
try:
yield session
finally:
pass
app.dependency_overrides[get_db] = override_get_db
client = TestClient(app)
yield client, session
app.dependency_overrides.clear()
session.close()
def _seed(session: Session) -> None:
target = TargetRepository(session).create(EvalTarget(
name="对象A", platform=PlatformType.AI_DIGITAL_EMPLOYEE,
channel_type=ChannelType.TUTU_API, channel_config={}, status=TargetStatus.ACTIVE,
))
scenario = ScenarioRepository(session).create(Scenario(
name="场景A", cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
))
repo = RunRepository(session)
for pass_rate, trigger in [(1.0, RunTrigger.MANUAL), (0.5, RunTrigger.AI_ASSISTANT)]:
run = repo.create(EvalRun(
target_id=target.id, scenario_id=scenario.id,
status=RunStatus.COMPLETED, triggered_by=trigger,
))
run.summary = {"total_cases": 2, "passed_cases": 1, "failed_cases": 1,
"total_rules": 2, "passed_rules": 1, "pass_rate": pass_rate}
repo.update(run)
repo.create(EvalRun(
target_id=target.id, scenario_id=scenario.id,
status=RunStatus.RUNNING, triggered_by=RunTrigger.MANUAL,
))
def test_dashboard_aggregates(client_with_db):
client, session = client_with_db
_seed(session)
data = client.get("/api/stats/dashboard").json()
assert data["targets_count"] == 1
assert data["scenarios_count"] == 1
assert data["runs_count"] == 3
assert data["model_configs_count"] == 0
assert data["running_count"] == 1
assert data["today_runs"] == 3
assert data["overall_pass_rate"] == pytest.approx(0.75)
assert data["trigger_breakdown"] == {"manual": 2, "ai_assistant": 1}
assert len(data["scenario_stats"]) == 1
stat = data["scenario_stats"][0]
assert stat["scenario_name"] == "场景A"
assert stat["run_count"] == 2
assert stat["avg_pass_rate"] == pytest.approx(0.75)
assert stat["last_run_at"] is not None
assert len(data["recent_runs"]) == 3
assert data["recent_runs"][0]["scenario_name"] == "场景A"
assert data["recent_runs"][0]["target_name"] == "对象A"
assert "triggered_by" in data["recent_runs"][0]
def test_dashboard_empty_db(client_with_db):
client, _ = client_with_db
data = client.get("/api/stats/dashboard").json()
assert data["runs_count"] == 0
assert data["overall_pass_rate"] is None
assert data["scenario_stats"] == []
assert data["recent_runs"] == []
def test_trend_returns_daily_points(client_with_db):
client, session = client_with_db
_seed(session)
points = client.get("/api/stats/trend").json()
assert len(points) == 1
assert points[0]["run_count"] == 2
assert points[0]["pass_rate"] == pytest.approx(75.0)

View File

@ -34,7 +34,13 @@ def report_session(tmp_path):
engine.dispose() engine.dispose()
def _seed_run(session: Session, *, pass_rate: float = 1.0, n_cases: int = 1) -> str: def _seed_run(
session: Session,
*,
pass_rate: float = 1.0,
n_cases: int = 1,
scenario_id: str | None = None,
) -> str:
"""Create a minimal completed run with real data in the DB and return run_id.""" """Create a minimal completed run with real data in the DB and return run_id."""
target = EvalTarget( target = EvalTarget(
name="测试对象", name="测试对象",
@ -45,15 +51,17 @@ def _seed_run(session: Session, *, pass_rate: float = 1.0, n_cases: int = 1) ->
) )
target = TargetRepository(session).create(target) target = TargetRepository(session).create(target)
if scenario_id is None:
scenario = Scenario( scenario = Scenario(
name="测试场景", name="测试场景",
cases=[Case(id=f"c{i}", type=CaseType.SINGLE, messages=["hi"]) for i in range(n_cases)], cases=[Case(id=f"c{i}", type=CaseType.SINGLE, messages=["hi"]) for i in range(n_cases)],
) )
scenario = ScenarioRepository(session).create(scenario) scenario = ScenarioRepository(session).create(scenario)
scenario_id = scenario.id
run = EvalRun( run = EvalRun(
target_id=target.id, target_id=target.id,
scenario_id=scenario.id, scenario_id=scenario_id,
status=RunStatus.COMPLETED, status=RunStatus.COMPLETED,
) )
run = RunRepository(session).create(run) run = RunRepository(session).create(run)
@ -148,9 +156,14 @@ def test_generate_report_not_found_raises(report_session):
# ── generate_compare_report ─────────────────────────────────────────────── # ── generate_compare_report ───────────────────────────────────────────────
def _scenario_of(session: Session, run_id: str) -> str:
return RunRepository(session).get(run_id).scenario_id
def test_compare_report_structure(report_session): def test_compare_report_structure(report_session):
run_id_a = _seed_run(report_session, pass_rate=1.0, n_cases=2) run_id_a = _seed_run(report_session, pass_rate=1.0, n_cases=2)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2) sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session) result = generate_compare_report(run_id_a, run_id_b, report_session)
assert "run_a" in result assert "run_a" in result
@ -163,14 +176,16 @@ def test_compare_report_structure(report_session):
def test_compare_report_delta(report_session): def test_compare_report_delta(report_session):
run_id_a = _seed_run(report_session, pass_rate=0.5, n_cases=2) run_id_a = _seed_run(report_session, pass_rate=0.5, n_cases=2)
run_id_b = _seed_run(report_session, pass_rate=1.0, n_cases=2) sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, pass_rate=1.0, n_cases=2, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session) result = generate_compare_report(run_id_a, run_id_b, report_session)
assert result["delta"]["pass_rate"] > 0 # B improved over A assert result["delta"]["pass_rate"] > 0 # B improved over A
def test_compare_report_changed_cases(report_session): def test_compare_report_changed_cases(report_session):
run_id_a = _seed_run(report_session, pass_rate=1.0, n_cases=2) run_id_a = _seed_run(report_session, pass_rate=1.0, n_cases=2)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2) sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, pass_rate=0.5, n_cases=2, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session) result = generate_compare_report(run_id_a, run_id_b, report_session)
# At least one case changed (A all-pass vs B half-pass) # At least one case changed (A all-pass vs B half-pass)
assert result["changed_cases"] >= 1 assert result["changed_cases"] >= 1
@ -178,7 +193,8 @@ def test_compare_report_changed_cases(report_session):
def test_compare_report_case_level(report_session): def test_compare_report_case_level(report_session):
run_id_a = _seed_run(report_session, n_cases=1) run_id_a = _seed_run(report_session, n_cases=1)
run_id_b = _seed_run(report_session, n_cases=1) sid = _scenario_of(report_session, run_id_a)
run_id_b = _seed_run(report_session, n_cases=1, scenario_id=sid)
result = generate_compare_report(run_id_a, run_id_b, report_session) result = generate_compare_report(run_id_a, run_id_b, report_session)
assert len(result["cases"]) >= 1 assert len(result["cases"]) >= 1
case = result["cases"][0] case = result["cases"][0]
@ -187,6 +203,13 @@ def test_compare_report_case_level(report_session):
assert "changed" in case assert "changed" in case
def test_compare_report_different_scenarios_rejected(report_session):
run_id_a = _seed_run(report_session, n_cases=1)
run_id_b = _seed_run(report_session, n_cases=1) # separate scenario
with pytest.raises(ValueError, match="same scenario"):
generate_compare_report(run_id_a, run_id_b, report_session)
# ── render_markdown_report ──────────────────────────────────────────────── # ── render_markdown_report ────────────────────────────────────────────────
def test_render_markdown_contains_header(report_session): def test_render_markdown_contains_header(report_session):