feat(comparison): v0.8 周期对比 — 计划指纹自动基线配对、机械指标 diff 与 LLM 演进叙述

正式线活动分析完成后自动链式生成对比;报告抽屉新增「周期对比」区块
(趋势徽章、指标 delta 表、问题演变、建议追踪,无自动基线时手动选择)。
版本号升至 0.8.0。
This commit is contained in:
sinohqb 2026-08-03 13:32:48 +08:00
parent fca3d64270
commit 1e55a21649
21 changed files with 2244 additions and 7 deletions

View File

@ -0,0 +1,13 @@
# 01 — 自动基线配对 + 机械指标 diff(GET 先行)
**What to build:** 周期对比的确定性半边。计划指纹纯函数(对象 + 计划条目集合[场景/偏移/次数] + 窗口,条目顺序无关);自动基线解析:双方均正式线、指纹相等、完成时间早于本期、已有 completed 分析,取最近一期,无则 None。机械 diff 纯函数:两期 `generate_campaign_report` dict 进,整窗 + 分场景(按 scenario_id 并集)的通过率/可用性/时延 delta 出,任一侧 None 则 delta None。`GET /api/campaigns/{id}/comparison` 先行返回 `{auto_baseline: {id, name, completed_at} | null, metric_diff: {...} | null}`(自动基线存在才算 diff),404 活动不存在。
**Blocked by:** 无 — 可立即开始
**Status:** done
- [x] 计划指纹纯函数:同计划不同条目顺序相等;改窗口/对象/场景/偏移/次数任一不等
- [x] 自动基线:指纹相等取最近一期;跳过无分析/分析非 completed/加速线候选;本期为加速线时返回 None;无匹配返回 None
- [x] 机械 diff:整窗与分场景 delta 数值正确;并集场景缺一侧 delta None;None 指标传播
- [x] GET 返回自动基线 + 现算 diff;无基线时两个 null;404
- [x] `pytest` + `ruff` 全绿

View File

@ -0,0 +1,13 @@
# 02 — 对比叙述生成(表 + Agent + POST)
**What to build:** 周期对比的 LLM 半边与完整生命周期。新表 `campaign_period_comparisons`(当前活动 id 唯一 upsert、baseline_campaign_id、generating/completed/failed、result JSON、model_config_id 快照、error、triggered_by、时间戳)+ 迁移。单次 LLM 调用编排:输入两期分析 JSON + 机械 diff,复用 `resolve_analysis_model` 与 ChatClient 注入 seam;输出 schema 见 spec(trend/summary/problem_evolution/suggestion_tracking);scenario_ids 白名单剔除;非法枚举归一(trend→stable、演化→persisting、建议→unaddressed);解析失败落 failed。`POST /api/campaigns/{id}/comparison`:非终态 400;本期无 completed 分析 400;body 可带 `baseline_campaign_id`(缺省走自动基线,仍无 → 400 引导;手动基线须有 completed 分析);无模型 400 引导;成功 → 后台任务 + `{"status": "generating"}`。GET 扩展合并:状态行字段 + 机械 diff + 叙述 + 基线信息。
**Blocked by:** 01(GET 与配对复用)
**Status:** done
- [x] 迁移:`campaign_period_comparisons` 表(campaign_id 唯一约束)+ 迁移测试
- [x] 仓储 upsert / get_by_campaign
- [x] 编排:FakeChatClient 断言输入含两期分析与 diff;解析失败→failed;白名单剔除虚构 scenario_ids;非法枚举归一
- [x] POST 各 400 分支 + 成功分支 + 重跑 upsert 不新增行;GET 合并返回
- [x] `pytest` + `ruff` 全绿

View File

@ -0,0 +1,16 @@
# 03 — 报告抽屉「周期对比」区块
**What to build:** 报告抽屉在「智能分析」之下新增「周期对比」区块。状态行:生成中(spinner + 5s 轮询)、失败(错误原因 + 重试)、未生成(「生成对比」按钮;无自动基线时显示基线选择器——同对象历史终态活动下拉,选定后生成)。已生成依次渲染:趋势徽章(improving 绿 / stable 蓝 / regressing 红)+ 总结 → 指标变化表(整窗行 + 分场景行,delta 正绿负红、None 显示 —)→ 问题演化列表(新增红 / 持续橙 / 消解绿 Tag + 标题 + 说明 + 场景 Tag)→ 建议追踪(已落实绿 / 部分橙 / 未落实灰 / 新增蓝)。展示基线活动名与完成时间、所用分析模型名与生成时间。生成按钮仅终态活动可用。
**Blocked by:** 02(API 与结果结构)
**Status:** done
- [x] `api.ts`:对比类型 + `getComparison` / `generateComparison(id, baselineId?)`
- [x] 区块按状态渲染(生成中轮询复用 `usePolling`)
- [x] 已生成渲染:趋势徽章 + 总结 / 指标 delta 表(正绿负红)/ 问题演化 Tag / 建议追踪状态
- [x] 无自动基线时的基线选择器(同对象终态活动下拉)
- [x] 基线活动名 + 完成时间 + 模型名 + 生成时间展示;按钮终态门控
- [x] `npx tsc --noEmit` + `npm run build` 通过;浏览器实操验证
(注:本会话无浏览器工具,以 `curl` 对 8010 端口本地后端做了 GET 端点冒烟:
无基线活动返回 `{"status":"none",...}`、未知活动 404;交互式浏览器验证待用户本地确认)

View File

@ -0,0 +1,12 @@
# 04 — 正式线自动链(分析完成 → 对比生成)
**What to build:** `execute_campaign_analysis` 成功落 completed 后,活动为正式线且存在自动基线(含 completed 分析)→ 自动 enqueue 对比生成(`triggered_by=auto`);加速线、无自动基线、无分析模型均静默跳过,不影响分析任务本身。注意:analysis 模块与 comparison 模块互相依赖(对比复用 `resolve_analysis_model`),用函数内延迟导入避免循环 import。
**Blocked by:** 02(生成入口)
**Status:** done
- [x] 分析完成后自动链:正式线 + 自动基线 → spy 断言 enqueue(`triggered_by=auto`)
- [x] 加速线 / 无自动基线 / 无模型三分支静默跳过
- [x] 自动链异常不冒泡(警告日志),分析结果不受影响
- [x] `pytest` + `ruff` 全绿

94
.scratch/v0.8/spec.md Normal file
View File

@ -0,0 +1,94 @@
# v0.8 — 周期对比(Period Comparison)
## Problem Statement
运营者用评估活动对同一对象做周期性考察,每期末尾拿到一份智能分析(总体结论、问题诊断、改善建议)。但单期分析是孤立的:这一期的问题到底是新冒出来的、持续未愈的、还是已经消解的?上一期的改善建议落实了吗?要回答这些,人得自己打开两期报告逐条对照自由文本——而这正是平台该替人做的元层研判。
## Solution
在相邻两期活动之间提供**周期对比**:机械 diff 给出两期聚合指标的客观变化(整窗与分场景的通过率/可用性/时延 delta,ADR-0004 口径,确定性计算),LLM 在两期分析结论之上产出结构化演进叙述(趋势判断、问题演化、建议追踪)。报告抽屉内新增「周期对比」区块,与智能分析并列。正式线活动全链自动(活动完成 → 分析 → 对比),加速调试线手动按需。
## User Stories
1. 作为运营者,我希望打开本期活动报告时看到"与上一期相比"的指标变化表(整窗通过率/可用性/时延及分场景 delta),以便快速判断走势。
2. 作为运营者,我希望看到一个趋势徽章(改善/平稳/退化),以便不看数字也能感知方向。
3. 作为运营者,我希望看到问题演化列表(新增/持续/消解,含涉及场景与说明),以便知道该担心什么、可以放下什么。
4. 作为运营者,我希望看到上期建议的追踪状态(已落实/部分落实/未落实/新增),以便评估改进动作的有效性。
5. 作为运营者,我希望正式线活动的对比在分析完成后自动生成,以便零操作拿到结论。
6. 作为调试者,我希望加速线活动能手动点「生成对比」,以便验证功能而不等完整周期。
7. 作为运营者,当本期活动计划与历史不一致(无自动基线)时,我希望能手动挑选一个历史活动作基线,以便换计划后仍能对比。
8. 作为运营者,我希望对比生成失败时看到原因并可重试,以便区分模型问题与数据问题。
9. 作为运营者,我希望对比叙述与指标 diff 分开呈现(叙述是 LLM 判断、数字是确定性计算),以便知道哪部分可以较真。
10. 作为管理员,我希望对比复用「分析默认」模型(活动覆盖 ?? 全局默认),以便不引入新的配置面。
11. 作为运营者,我希望对比中引用的场景是真实存在的(白名单过滤),以便不被模型虚构误导。
## Implementation Decisions
### 配对(计划指纹 + 自动基线)
- **计划指纹** = 评测对象 + 计划条目集合(场景 id、偏移秒数、次数)+ 窗口秒数。指纹相等的正式线活动构成活动串
- **自动基线** = 同活动串中、完成时间早于本期、且已有 completed 分析的最近一期活动
- 指纹不一致 → 无自动基线;`POST /comparison` 接受可选 `baseline_campaign_id` 手动指定(跨串允许,可比性由用户负责)
- 基线活动被删或分析被重置 → 视为无基线
### 产出(混合:机械 diff + LLM 叙述)
- **机械 diff**:纯函数,输入两期 `generate_campaign_report` 结果,输出整窗 + 分场景的通过率/可用性/时延 delta(读时现算,不存储)
- **LLM 叙述**:单次调用(不做两阶段——输入只有两份已消化的分析 JSON + 机械 diff,无原始样本),输入上限轻量;输出结构化 JSON:
```json
{
"trend": "improving | stable | regressing",
"summary": "总体演进结论",
"problem_evolution": [{"status": "new | persisting | resolved", "title", "detail", "scenario_ids": []}],
"suggestion_tracking": [{"text", "status": "addressed | partial | unaddressed | new", "note"}]
}
```
- 叙述中的 `scenario_ids` 落库前过白名单(同 v0.7);非法 `trend`/`status` 枚举归一(trend→stable,status→persisting / unaddressed)
- LLM 解析失败 → 行落 failed + error
### 存储与生命周期
- 新表 `campaign_period_comparisons`:当前活动 id 唯一 upsert、baseline_campaign_id、generating/completed/failed、result(叙述 JSON)、model_config_id 快照、error、triggered_by(auto/manual)、时间戳——镜像 `campaign_analyses` 模式
- 模型解析复用 v0.7 `resolve_analysis_model`(活动覆盖 ?? 全局分析默认);不可解析 → 自动链静默跳过 / 手动 400 引导
- 自动链:`execute_campaign_analysis` 成功落 completed 后,活动为正式线且存在自动基线 → enqueue 对比生成;任一前提不满足静默跳过
- 重新生成(手动 POST)= 同活动 upsert 覆盖,可借此更换基线重算
### API
- `GET /api/campaigns/{id}/comparison`:无行 → 空态 `{"status": "none"}`;有行 → 机械 diff(现算)+ 叙述行合并返回;附 `baseline_campaign_id` 与自动配对结果,前端据此渲染或引导选基线
- `POST /api/campaigns/{id}/comparison`:body 可选 `{"baseline_campaign_id": "..."}`;非终态活动 400;当前活动无 completed 分析 400(先生成分析);无模型 400 引导;成功 → 后台任务 + `{"status": "generating"}`
- 两期都有 completed 分析是生成前提(基线无分析 → 400 提示先生成基线分析)
### 前端(报告抽屉「周期对比」区块)
- 位置:「智能分析」区块之下,复用区块状态行范式(生成中 5s 轮询 / 失败重试 / 未生成按钮)
- 已生成:趋势徽章 + 总结 → 指标变化表(整窗 + 分场景 delta,正绿负红)→ 问题演化(新增红/持续橙/消解绿 Tag)→ 建议追踪(已落实绿/部分橙/未落实灰/新增蓝)
- 无自动基线:基线选择器(同对象历史终态活动下拉)+ 生成按钮
- 展示基线活动名与完成时间、所用分析模型名与生成时间
## Testing Decisions
好测试只测外部行为,不测实现细节;全部落在既有最高层 seam:
1. **机械 diff 纯函数**:手搓两份报告 dict,断言 delta 结构与数值、空桶/None 处理(先例如 `test_report_render.py`)
2. **计划指纹与自动基线**:内存 SQLite,指纹相等/不等、跳过无分析活动、跳过非正式线、取最近一期(先例如 `test_campaign_analysis.py` 的仓储测试)
3. **LLM 叙述编排**:FakeChatClient(v0.7 seam)覆盖成功编排、解析失败→AnalysisError 式失败、白名单剔除虚构 scenario_ids、非法枚举归一
4. **API 集成**:镜像 `test_campaign_analysis_api.py`——GET 空态、POST 非终态 400、无分析 400、无基线 400、手动指定基线成功、重跑 upsert 不新增行
5. **自动链**:spy 对比入口,断言分析完成后正式线自动触发、加速线不触发、无基线不触发(镜像 `test_campaign_analysis_auto_trigger.py`)
6. **迁移测试**:表存在 + campaign_id 唯一约束(先例如 `test_campaign_analyses_migration.py`)
## Out of Scope
- 跨活动对比(多对象横切)——词汇表中另一个轴,本规格不涉及
- 活动列表趋势标记(↑↓)——后续增强
- 对比结果纳入 Markdown 导出——看使用情况再定
- 基线活动的选取 UI 优化(如按指纹分组展示)
- 三期以上的趋势序列(只对比相邻两期)
## Further Notes
- 设计决策源自 `/grill-with-docs` 访谈(6 问 6 决):配对 A、产出 C 混合、呈现 A 抽屉区块、生命周期 A 新表+联动触发、输入 A 轻量、输出 A 结构化
- 「周期对比」「正式线/加速调试线」已入 `CONTEXT.md` 词汇表
- 对比不重新消化失败样本:样本已被两期分析消化,对比是元层研判

View File

@ -74,6 +74,14 @@ _Avoid_: 排程、日程表
活动周期报告中的一个维度:某时段(或整窗)内**正常完成**的子 Run 占比(completed / 已派生)。与通过率正交——通过率反映"回答质量"(用例级、含故障判不通过,ADR-0002),可用性反映"服务是否可达/执行是否成功"。通道故障导致的失败子 Run 拉低可用性。
_Avoid_: 在线率、健康度
**周期对比(Period Comparison)**:
同一活动串(同评测对象 + 同计划指纹:场景集合、偏移、次数、窗口完全一致)中,相邻两期正式线活动的对照,回答"这一期比上一期好了还是坏了"。自动按完成时间取前一个已有 completed 分析的活动为基线;计划指纹不一致即无自动基线,可手动另选。区别于"跨活动对比"(多对象并行活动的横切对照)与"对比报告"(Run 级逐项对照)。
_Avoid_: 环比(暗示固定自然周期,活动窗口可任意长)
**正式线 / 加速调试线(Production-line / Accelerated-line)**:
活动的两种运行档位,由时间倍速区分:`time_scale == 1` 为正式线(窗口按真实时长走完,代表真实服务周期,享受自动分析、自动周期对比等全链自动化);`time_scale > 1` 为加速调试线(压缩窗口用于快速验证流程,自动化能力默认关闭,仅手动按需触发)。
_Avoid_: 真实线、快速模式
## 模型配置
**模型能力(Capability)**:

View File

@ -239,6 +239,29 @@ async def analyze_campaign(
}
def _maybe_enqueue_period_comparison(campaign: Campaign, session: Session) -> None:
"""正式线活动分析完成后自动链到周期对比(v0.8)。
前提:正式线(time_scale == 1)、分析模型可解析、存在自动基线
(同活动串且已有 completed 分析)。任一不满足静默跳过;异常仅告警,
不影响刚落库的分析结果。
"""
try:
if campaign.time_scale != 1:
return
if resolve_analysis_model(campaign, session) is None:
return
# 延迟导入:comparison 顶层依赖 analysis(resolve_analysis_model),
# 反向导入会成环。
from agenteval.evaluation import comparison as comparison_module
if comparison_module.resolve_auto_baseline(campaign, session) is None:
return
comparison_module.start_campaign_comparison(campaign.id, triggered_by="auto")
except Exception as exc:
_logger.warning("活动 %s 自动周期对比跳过: %s", campaign.id, exc)
def _gateway_chat_client(runtime: ModelRuntimeConfig) -> ChatClient:
gateway = ModelGateway(timeout=180.0)
@ -298,6 +321,7 @@ async def execute_campaign_analysis(
campaign_id, status="completed", result=result,
model_config_id=runtime.id, triggered_by=triggered_by,
)
_maybe_enqueue_period_comparison(campaign, session)
finally:
session.close()

View File

@ -0,0 +1,419 @@
"""Period comparison — baseline pairing, metric diff and LLM narrative (周期对比).
机械半边:计划指纹(同活动串的判据)、自动基线解析、两期报告的指标
diff——全部确定性计算(ADR-0004 口径,经 ``generate_campaign_report`` 的
既有聚合结果现算)。叙述半边:单次 LLM 调用,在两期已消化的分析 JSON 与
机械 diff 之上产出结构化演进叙述(CONTEXT.md「周期对比」)。
"""
import asyncio
import json
import logging
from datetime import datetime, timezone
from typing import Any, Optional
from sqlmodel import Session
from agenteval.evaluation.analysis import ChatClient, resolve_analysis_model
from agenteval.evaluation.report import generate_campaign_report
from agenteval.model_gateway import ModelGateway
from agenteval.models import Campaign, EvalRun
from agenteval.services.model_configs import ModelRuntimeConfig
from agenteval.storage.db import get_session, iso_utc, utc_now
from agenteval.storage.repository import (
CampaignAnalysisRepository,
CampaignPeriodComparisonRepository,
CampaignRepository,
RunRepository,
ScenarioRepository,
)
from agenteval.utils.llm import parse_json_from_llm_text
_logger = logging.getLogger("agenteval")
_SAME_MOMENT_EPS = 1e-3
_VALID_TRENDS = {"improving", "stable", "regressing"}
_VALID_EVOLUTION_STATUSES = {"new", "persisting", "resolved"}
_VALID_SUGGESTION_STATUSES = {"addressed", "partial", "unaddressed", "new"}
class ComparisonError(RuntimeError):
"""对比生成失败(数据缺失或模型输出无法解析),可重试。"""
def campaign_plan_fingerprint(campaign: Campaign) -> str:
"""活动串判据:评测对象 + 计划条目集合 + 窗口秒数(条目顺序无关)。
指纹相等的活动构成同一活动串(CONTEXT.md「周期对比」)。档位
(time_scale)不参与指纹——正式/加速由基线解析另行过滤。
"""
entries = sorted(
((entry.scenario_id, entry.offset_seconds, entry.count) for entry in campaign.plan),
key=lambda e: (e[0], e[1], e[2]),
)
return json.dumps(
{"target": campaign.target_id, "window": campaign.window_seconds, "plan": entries},
separators=(",", ":"),
)
def _to_utc(dt: Optional[datetime]) -> Optional[datetime]:
if dt is None:
return None
return dt.replace(tzinfo=timezone.utc) if dt.tzinfo is None else dt
def resolve_auto_baseline(campaign: Campaign, session: Session) -> Optional[Campaign]:
"""自动基线:同活动串中最近一期已完成的正式线活动(须有 completed 分析)。
条件(全部满足才可作基线):指纹相等、双方均为正式线
(time_scale == 1)、候选完成时间早于本期、候选已有 completed 分析。
本期为加速线或无匹配候选时返回 None。
"""
if campaign.time_scale != 1:
return None
reference = _to_utc(campaign.completed_at) or utc_now()
fingerprint = campaign_plan_fingerprint(campaign)
best: Optional[Campaign] = None
best_completed: Optional[datetime] = None
for candidate in CampaignRepository(session).list_all():
if candidate.id == campaign.id or candidate.time_scale != 1:
continue
if campaign_plan_fingerprint(candidate) != fingerprint:
continue
completed = _to_utc(candidate.completed_at)
if completed is None:
continue
if (reference - completed).total_seconds() < _SAME_MOMENT_EPS:
continue
analysis_row = CampaignAnalysisRepository(session).get_by_campaign(candidate.id)
if analysis_row is None or analysis_row.status != "completed":
continue
if best_completed is None or completed > best_completed:
best = candidate
best_completed = completed
return best
def _delta(base: Optional[float], current: Optional[float], ndigits: int) -> Optional[float]:
if base is None or current is None:
return None
return round(current - base, ndigits)
def compute_metric_diff(
baseline_report: dict[str, Any],
current_report: dict[str, Any],
) -> dict[str, Any]:
"""两期 ``generate_campaign_report`` 结果 → 整窗 + 分场景指标 delta。
通过率/可用性按 4 位小数、时延按 1 位小数取 delta;任一侧指标为 None
或该侧缺失该场景时 delta 为 None(只保留存在的 baseline/current 值)。
纯函数,无 I/O。
"""
def _pair(base: Optional[float], current: Optional[float], ndigits: int) -> dict[str, Optional[float]]:
return {"baseline": base, "current": current, "delta": _delta(base, current, ndigits)}
base_summary = baseline_report.get("summary") or {}
cur_summary = current_report.get("summary") or {}
overall = {
"pass_rate": _pair(base_summary.get("overall_pass_rate"), cur_summary.get("overall_pass_rate"), 4),
"availability": _pair(base_summary.get("overall_availability"), cur_summary.get("overall_availability"), 4),
"avg_latency_ms": _pair(base_summary.get("avg_latency_ms"), cur_summary.get("avg_latency_ms"), 1),
}
base_scenarios = {s.get("scenario_id"): s for s in baseline_report.get("capability_summary") or []}
cur_scenarios = {s.get("scenario_id"): s for s in current_report.get("capability_summary") or []}
scenarios = []
for sid in sorted(set(base_scenarios) | set(cur_scenarios), key=lambda x: x or ""):
base = base_scenarios.get(sid)
cur = cur_scenarios.get(sid)
entry: dict[str, Any] = {
"scenario_id": sid,
"scenario_name": (cur or base or {}).get("scenario_name") or "",
"pass_rate": _pair(
base.get("pass_rate") if base else None,
cur.get("pass_rate") if cur else None,
4,
),
"availability": _pair(
base.get("availability") if base else None,
cur.get("availability") if cur else None,
4,
),
"avg_latency_ms": _pair(
base.get("avg_latency_ms") if base else None,
cur.get("avg_latency_ms") if cur else None,
1,
),
}
scenarios.append(entry)
return {"overall": overall, "scenarios": scenarios}
def build_campaign_report_dict(campaign: Campaign, session: Session) -> dict[str, Any]:
"""为周期对比现算一期活动的报告 dict(读路径,不重算聚合口径)。"""
runs: list[EvalRun] = RunRepository(session).list_by_campaign(campaign.id)
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
return generate_campaign_report(campaign, runs, scenario_names=scenario_names)
def build_comparison_payload(campaign: Campaign, session: Session) -> dict[str, Any]:
"""GET 返回体:自动基线信息 + 机械 diff(无基线时两者均为 null)。"""
baseline = resolve_auto_baseline(campaign, session)
if baseline is None:
return {"auto_baseline": None, "metric_diff": None}
diff = compute_metric_diff(
build_campaign_report_dict(baseline, session),
build_campaign_report_dict(campaign, session),
)
return {
"auto_baseline": {
"id": baseline.id,
"name": baseline.name,
"completed_at": iso_utc(baseline.completed_at),
},
"metric_diff": diff,
}
# ── 叙述半边:单次 LLM 调用编排 ─────────────────────────────────────────
async def narrate_period_comparison(
*,
baseline_analysis: dict[str, Any],
current_analysis: dict[str, Any],
metric_diff: dict[str, Any],
valid_scenario_ids: set[str],
chat_client: ChatClient,
) -> dict[str, Any]:
"""在两期分析结论与机械 diff 之上产出结构化演进叙述(单次调用)。
输出 schema:trend / summary / problem_evolution / suggestion_tracking。
模型虚构的 scenario_id 按白名单剔除;非法枚举归一(trend→stable、
演化 status→persisting、建议 status→unaddressed);解析失败抛
``ComparisonError``(由调用方落 failed 状态)。
"""
system_prompt = (
"你是智能客服质量评估平台的趋势研判专家,负责对照相邻两期评估活动的"
"智能分析结论,产出结构化的演进叙述。只输出一个 JSON 对象:"
'{"trend": "improving|stable|regressing", '
'"summary": "总体演进结论(一段话)", '
'"problem_evolution": [{"status": "new|persisting|resolved", "title": "...", '
'"detail": "该问题的演化说明", "scenario_ids": ["涉及场景 id"]}], '
'"suggestion_tracking": [{"text": "上期建议原文", '
'"status": "addressed|partial|unaddressed|new", "note": "落实情况的说明"}]}'
";problem_evolution 覆盖上期问题在本期的新增/持续/消解,"
"suggestion_tracking 追踪上期每条建议的落实情况并列出本期新增建议。"
"全部使用中文。"
)
user_prompt = json.dumps(
{
"上期分析": baseline_analysis,
"本期分析": current_analysis,
"指标变化(确定性计算,可信)": metric_diff,
},
ensure_ascii=False,
)
content = await chat_client(
[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
]
)
try:
parsed = parse_json_from_llm_text(content)
except Exception as exc:
raise ComparisonError(f"对比叙述输出解析失败: {exc}") from exc
if not isinstance(parsed, dict):
raise ComparisonError("对比叙述输出不是 JSON 对象")
summary = parsed.get("summary")
if not isinstance(summary, str) or not summary.strip():
raise ComparisonError("对比叙述缺少 summary")
trend = parsed.get("trend")
if trend not in _VALID_TRENDS:
trend = "stable"
problem_evolution = []
for item in parsed.get("problem_evolution") or []:
if not isinstance(item, dict):
continue
status = item.get("status")
problem_evolution.append(
{
"status": status if status in _VALID_EVOLUTION_STATUSES else "persisting",
"title": str(item.get("title", "")),
"detail": str(item.get("detail", "")),
"scenario_ids": [s for s in item.get("scenario_ids") or [] if s in valid_scenario_ids],
}
)
suggestion_tracking = []
for item in parsed.get("suggestion_tracking") or []:
if not isinstance(item, dict):
continue
status = item.get("status")
suggestion_tracking.append(
{
"text": str(item.get("text", "")),
"status": status if status in _VALID_SUGGESTION_STATUSES else "unaddressed",
"note": str(item.get("note", "")),
}
)
return {
"trend": trend,
"summary": summary,
"problem_evolution": problem_evolution,
"suggestion_tracking": suggestion_tracking,
}
def _gateway_chat_client(runtime: ModelRuntimeConfig) -> ChatClient:
gateway = ModelGateway(timeout=180.0)
async def _chat(messages: list[dict[str, str]]) -> str:
return await gateway.chat(runtime, messages, temperature=0.2)
return _chat
async def execute_campaign_comparison(
campaign_id: str,
*,
triggered_by: str,
baseline_campaign_id: Optional[str] = None,
chat_client: Optional[ChatClient] = None,
) -> None:
"""后台执行体:generating → completed/failed 状态机(upsert,每活动一行)。
基线优先取显式指定(手动重选),缺省走自动基线。与 Runs 同款后台任务
约定:自持 Session、try/finally 关闭、失败落 error。
"""
session = get_session()
try:
comparisons = CampaignPeriodComparisonRepository(session)
analyses = CampaignAnalysisRepository(session)
campaign = CampaignRepository(session).get(campaign_id)
if not campaign:
return
runtime = resolve_analysis_model(campaign, session)
if runtime is None:
comparisons.upsert(
campaign_id,
status="failed",
triggered_by=triggered_by,
error="未配置分析模型:请在模型配置中心将某个 chat 配置设为「分析默认」",
)
return
if baseline_campaign_id:
baseline = CampaignRepository(session).get(baseline_campaign_id)
if baseline is None:
comparisons.upsert(
campaign_id,
status="failed",
triggered_by=triggered_by,
error=f"基线活动不存在: {baseline_campaign_id}",
)
return
else:
baseline = resolve_auto_baseline(campaign, session)
if baseline is None:
comparisons.upsert(
campaign_id,
status="failed",
triggered_by=triggered_by,
error="未找到可用的基线活动(计划指纹不一致或历史活动无已完成分析)",
)
return
baseline_analysis = analyses.get_by_campaign(baseline.id)
if baseline_analysis is None or baseline_analysis.status != "completed":
comparisons.upsert(
campaign_id,
status="failed",
baseline_campaign_id=baseline.id,
model_config_id=runtime.id,
triggered_by=triggered_by,
error="基线活动没有已完成的智能分析,请先生成基线活动的分析",
)
return
current_analysis = analyses.get_by_campaign(campaign_id)
if current_analysis is None or current_analysis.status != "completed":
comparisons.upsert(
campaign_id,
status="failed",
baseline_campaign_id=baseline.id,
model_config_id=runtime.id,
triggered_by=triggered_by,
error="本期活动没有已完成的智能分析,请先生成智能分析",
)
return
comparisons.upsert(
campaign_id,
status="generating",
baseline_campaign_id=baseline.id,
model_config_id=runtime.id,
triggered_by=triggered_by,
)
try:
diff = compute_metric_diff(
build_campaign_report_dict(baseline, session),
build_campaign_report_dict(campaign, session),
)
valid_scenario_ids = {s["scenario_id"] for s in diff["scenarios"]}
result = await narrate_period_comparison(
baseline_analysis=baseline_analysis.get_result(),
current_analysis=current_analysis.get_result(),
metric_diff=diff,
valid_scenario_ids=valid_scenario_ids,
chat_client=chat_client or _gateway_chat_client(runtime),
)
except Exception as exc:
_logger.warning("活动 %s 周期对比失败: %s", campaign_id, exc)
comparisons.upsert(
campaign_id,
status="failed",
baseline_campaign_id=baseline.id,
model_config_id=runtime.id,
error=str(exc)[:500],
triggered_by=triggered_by,
)
return
comparisons.upsert(
campaign_id,
status="completed",
baseline_campaign_id=baseline.id,
result=result,
model_config_id=runtime.id,
triggered_by=triggered_by,
)
finally:
session.close()
def start_campaign_comparison(
campaign_id: str,
*,
triggered_by: str,
baseline_campaign_id: Optional[str] = None,
) -> asyncio.Task:
"""以后台任务启动对比生成(fire-and-forget;状态经 campaign_period_comparisons 表观测)。"""
return asyncio.create_task(
execute_campaign_comparison(
campaign_id,
triggered_by=triggered_by,
baseline_campaign_id=baseline_campaign_id,
)
)

View File

@ -224,6 +224,33 @@ class CampaignAnalysisDB(SQLModel, table=True):
self.result = _json_dumps(result)
class CampaignPeriodComparisonDB(SQLModel, table=True):
"""One row per campaign holding its period-comparison (周期对比) state.
Mirrors ``campaign_analyses`` (one row, upserted per campaign), plus the
baseline campaign this comparison was computed against.
"""
__tablename__ = "campaign_period_comparisons"
id: Optional[str] = Field(default_factory=new_uuid, primary_key=True)
campaign_id: str = Field(unique=True, index=True)
baseline_campaign_id: str = Field(index=True)
status: str = "generating"
result: Optional[str] = None
model_config_id: Optional[str] = None
error: Optional[str] = None
triggered_by: str = "manual"
created_at: Optional[datetime] = Field(default_factory=utc_now)
updated_at: Optional[datetime] = Field(default_factory=utc_now)
def get_result(self) -> Optional[dict[str, Any]]:
return _json_loads(self.result) if self.result else None
def set_result(self, result: dict[str, Any]) -> None:
self.result = _json_dumps(result)
class EvalRunDB(SQLModel, table=True):
"""Database table for evaluation runs."""

View File

@ -9,6 +9,7 @@ from agenteval.services.model_configs import ModelConfigService
from agenteval.storage.db import (
CampaignAnalysisDB,
CampaignDB,
CampaignPeriodComparisonDB,
EvalResultDB,
EvalRunDB,
EvalTargetDB,
@ -433,6 +434,52 @@ class CampaignAnalysisRepository:
return row
class CampaignPeriodComparisonRepository:
"""Repository for period-comparison rows (one per campaign, upserted)."""
def __init__(self, session: Optional[Session] = None):
self.session = session or get_session()
def get_by_campaign(self, campaign_id: str) -> Optional[CampaignPeriodComparisonDB]:
statement = select(CampaignPeriodComparisonDB).where(
CampaignPeriodComparisonDB.campaign_id == campaign_id
)
return self.session.exec(statement).first()
def upsert(
self,
campaign_id: str,
*,
status: str,
baseline_campaign_id: Optional[str] = None,
result: Optional[dict] = None,
model_config_id: Optional[str] = None,
error: Optional[str] = None,
triggered_by: str = "manual",
) -> CampaignPeriodComparisonDB:
row = self.get_by_campaign(campaign_id)
if row is None:
row = CampaignPeriodComparisonDB(
campaign_id=campaign_id,
baseline_campaign_id=baseline_campaign_id or "",
)
row.status = status
if baseline_campaign_id is not None:
row.baseline_campaign_id = baseline_campaign_id
if result is not None:
row.set_result(result)
else:
row.result = None
row.model_config_id = model_config_id
row.error = error
row.triggered_by = triggered_by
row.updated_at = utc_now()
self.session.add(row)
self.session.commit()
self.session.refresh(row)
return row
class ResultRepository:
"""Repository for evaluation results."""

View File

@ -7,12 +7,19 @@ the live window position and spawned/completed Run counts, and a campaign can be
cancelled mid-flight.
"""
from fastapi import APIRouter, Depends, HTTPException, Response
from fastapi import APIRouter, Body, Depends, HTTPException, Response
from pydantic import BaseModel, Field
from sqlmodel import Session
from agenteval.evaluation.analysis import resolve_analysis_model, start_campaign_analysis
from agenteval.evaluation.campaign_runner import campaign_progress, request_cancel, start_campaign
from agenteval.evaluation.comparison import (
build_campaign_report_dict,
build_comparison_payload,
compute_metric_diff,
resolve_auto_baseline,
start_campaign_comparison,
)
from agenteval.evaluation.report import (
build_campaign_timeline,
generate_campaign_report,
@ -24,6 +31,7 @@ from agenteval.storage.db import iso_utc, utc_now
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import (
CampaignAnalysisRepository,
CampaignPeriodComparisonRepository,
CampaignRepository,
RunRepository,
ScenarioRepository,
@ -181,6 +189,101 @@ async def trigger_campaign_analysis(campaign_id: str, session: Session = Depends
return {"status": "generating"}
@router.get("/{campaign_id}/comparison")
async def get_campaign_comparison(campaign_id: str, session: Session = Depends(get_db)) -> dict:
"""周期对比(v0.8):自动基线配对 + 叙述行状态 + 机械 diff(现算,不存储)。
``metric_diff`` 对应「生效基线」:已有对比行时为该行记录的基线,
否则为自动基线;两者都没有时为 null。
"""
campaign = CampaignRepository(session).get(campaign_id)
if not campaign:
raise HTTPException(status_code=404, detail="campaign not found")
payload = build_comparison_payload(campaign, session)
row = CampaignPeriodComparisonRepository(session).get_by_campaign(campaign_id)
if row is None:
return {"status": "none", "comparison": None, **payload}
baseline = CampaignRepository(session).get(row.baseline_campaign_id)
metric_diff = (
compute_metric_diff(
build_campaign_report_dict(baseline, session),
build_campaign_report_dict(campaign, session),
)
if baseline is not None
else None
)
comparison = {
"baseline_campaign_id": row.baseline_campaign_id,
"baseline": (
{
"id": baseline.id,
"name": baseline.name,
"completed_at": iso_utc(baseline.completed_at),
}
if baseline
else None
),
"result": row.get_result(),
"error": row.error,
"model_config_id": row.model_config_id,
"triggered_by": row.triggered_by,
"updated_at": iso_utc(row.updated_at),
}
return {
"status": row.status,
"comparison": comparison,
"auto_baseline": payload["auto_baseline"],
"metric_diff": metric_diff,
}
class GenerateComparisonRequest(BaseModel):
baseline_campaign_id: str | None = None
@router.post("/{campaign_id}/comparison")
async def trigger_campaign_comparison(
campaign_id: str,
request: GenerateComparisonRequest = Body(default=None),
session: Session = Depends(get_db),
) -> dict:
campaign = CampaignRepository(session).get(campaign_id)
if not campaign:
raise HTTPException(status_code=404, detail="campaign not found")
if campaign.status in (CampaignStatus.PLANNED, CampaignStatus.RUNNING):
raise HTTPException(status_code=400, detail="活动完成后才能生成周期对比")
analyses = CampaignAnalysisRepository(session)
current_analysis = analyses.get_by_campaign(campaign_id)
if current_analysis is None or current_analysis.status != "completed":
raise HTTPException(status_code=400, detail="请先生成本期活动的智能分析")
if resolve_analysis_model(campaign, session) is None:
raise HTTPException(
status_code=400,
detail="未配置分析模型:请在模型配置中心将某个 chat 配置设为「分析默认」,或为该活动指定分析模型",
)
explicit_baseline_id = request.baseline_campaign_id if request else None
if explicit_baseline_id:
baseline = CampaignRepository(session).get(explicit_baseline_id)
if baseline is None:
raise HTTPException(status_code=400, detail=f"基线活动不存在: {explicit_baseline_id}")
baseline_analysis = analyses.get_by_campaign(explicit_baseline_id)
if baseline_analysis is None or baseline_analysis.status != "completed":
raise HTTPException(status_code=400, detail="基线活动没有已完成的智能分析,请先生成基线活动的分析")
else:
baseline = resolve_auto_baseline(campaign, session)
if baseline is None:
raise HTTPException(
status_code=400,
detail="未找到自动基线:历史活动中没有同计划指纹且已完成分析的活动,可手动选择基线活动",
)
start_campaign_comparison(campaign_id, triggered_by="manual", baseline_campaign_id=baseline.id)
return {"status": "generating"}
@router.get("/{campaign_id}")
async def get_campaign(campaign_id: str, session: Session = Depends(get_db)) -> dict:
campaign = CampaignRepository(session).get(campaign_id)

View File

@ -1,12 +1,12 @@
{
"name": "agenteval-web",
"version": "0.7.0",
"version": "0.8.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "agenteval-web",
"version": "0.7.0",
"version": "0.8.0",
"dependencies": {
"@ant-design/charts": "^2.6.7",
"@ant-design/icons": "^6.3.2",

View File

@ -1,6 +1,6 @@
{
"name": "agenteval-web",
"version": "0.7.0",
"version": "0.8.0",
"private": true,
"type": "module",
"scripts": {

View File

@ -429,6 +429,74 @@ export interface CreateCampaignPayload {
analysis_model_config_id?: string | null
}
// ── Period comparison (v0.8) ────────────────────────────────────
export type ComparisonTrend = 'improving' | 'stable' | 'regressing'
export interface MetricDeltaPair {
baseline: number | null
current: number | null
delta: number | null
}
export interface MetricDiffScenario {
scenario_id: string
scenario_name: string
pass_rate: MetricDeltaPair
availability: MetricDeltaPair
avg_latency_ms: MetricDeltaPair
}
export interface MetricDiff {
overall: {
pass_rate: MetricDeltaPair
availability: MetricDeltaPair
avg_latency_ms: MetricDeltaPair
}
scenarios: MetricDiffScenario[]
}
export interface ComparisonBaselineInfo {
id: string
name: string
completed_at: string | null
}
export interface ComparisonProblemEvolution {
status: string
title: string
detail: string
scenario_ids: string[]
}
export interface ComparisonSuggestionTracking {
text: string
status: string
note: string
}
export interface ComparisonResult {
trend: ComparisonTrend
summary: string
problem_evolution: ComparisonProblemEvolution[]
suggestion_tracking: ComparisonSuggestionTracking[]
}
export interface CampaignComparison {
status: CampaignAnalysisStatus
auto_baseline: ComparisonBaselineInfo | null
metric_diff: MetricDiff | null
comparison: {
baseline_campaign_id: string
baseline: ComparisonBaselineInfo | null
result: ComparisonResult | null
error: string | null
model_config_id: string | null
triggered_by: string
updated_at: string | null
} | null
}
export const campaignsApi = {
list: () => api.get<CampaignListItem[]>('/campaigns'),
get: (id: string) => api.get<Campaign>(`/campaigns/${id}`),
@ -439,6 +507,12 @@ export const campaignsApi = {
api.get<{ entries: CampaignTimelineEntry[] }>(`/campaigns/${id}/timeline`),
getAnalysis: (id: string) => api.get<CampaignAnalysis>(`/campaigns/${id}/analysis`),
generateAnalysis: (id: string) => api.post<{ status: string }>(`/campaigns/${id}/analysis`),
getComparison: (id: string) => api.get<CampaignComparison>(`/campaigns/${id}/comparison`),
generateComparison: (id: string, baselineCampaignId?: string) =>
api.post<{ status: string }>(
`/campaigns/${id}/comparison`,
baselineCampaignId ? { baseline_campaign_id: baselineCampaignId } : {},
),
downloadReport: async (id: string) => {
const res = await api.get(`/campaigns/${id}/report/markdown`, { responseType: 'blob' })
const url = URL.createObjectURL(res.data as Blob)

View File

@ -8,7 +8,7 @@ import {
PlusOutlined, ReloadOutlined, StopOutlined, BarChartOutlined,
FileMarkdownOutlined, MinusCircleOutlined, QuestionCircleOutlined,
RocketOutlined, CheckCircleOutlined, SafetyOutlined, ClockCircleOutlined,
RobotOutlined,
RobotOutlined, SwapOutlined,
} from '@ant-design/icons'
import { Line, Bar } from '@ant-design/charts'
import PageWrapper from '../components/PageWrapper'
@ -17,6 +17,7 @@ import {
campaignsApi, targetsApi, scenariosApi, runsApi, modelConfigsApi,
type CampaignListItem, type CampaignReport, type Target, type Scenario, type Run,
type CampaignTimelineEntry, type ModelConfig, type CampaignAnalysis,
type CampaignComparison, type MetricDiff,
} from '../api'
import { passRateColor } from '../utils/colors'
import { shortDateTime, toDate } from '../utils/date'
@ -42,6 +43,25 @@ const SEVERITY_META: Record<string, { label: string; color: string }> = {
low: { label: '低', color: 'blue' },
}
const TREND_META: Record<string, { label: string; color: string }> = {
improving: { label: '改善', color: 'green' },
stable: { label: '平稳', color: 'blue' },
regressing: { label: '退化', color: 'red' },
}
const EVOLUTION_META: Record<string, { label: string; color: string }> = {
new: { label: '新增', color: 'red' },
persisting: { label: '持续', color: 'orange' },
resolved: { label: '消解', color: 'green' },
}
const TRACKING_META: Record<string, { label: string; color: string }> = {
addressed: { label: '已落实', color: 'green' },
partial: { label: '部分落实', color: 'orange' },
unaddressed: { label: '未落实', color: 'default' },
new: { label: '新增', color: 'blue' },
}
const WINDOW_OPTIONS = [6, 12, 24, 48, 72].map((h) => ({ label: `${h} 小时`, value: h * 3600 }))
const UNIT_OPTIONS = [
@ -125,6 +145,9 @@ export default function CampaignsPage() {
const [reportTimeline, setReportTimeline] = useState<CampaignTimelineEntry[]>([])
const [analysis, setAnalysis] = useState<CampaignAnalysis | null>(null)
const [analysisBusy, setAnalysisBusy] = useState(false)
const [comparison, setComparison] = useState<CampaignComparison | null>(null)
const [comparisonBusy, setComparisonBusy] = useState(false)
const [baselineChoice, setBaselineChoice] = useState<string | undefined>(undefined)
const [expandedIds, setExpandedIds] = useState<string[]>([])
const [timelines, setTimelines] = useState<Record<string, CampaignTimelineEntry[]>>({})
@ -233,18 +256,22 @@ export default function CampaignsPage() {
setReportRuns([])
setReportTimeline([])
setAnalysis(null)
setComparison(null)
setBaselineChoice(undefined)
}
try {
const [rep, runs, tl, ana] = await Promise.all([
const [rep, runs, tl, ana, cmp] = await Promise.all([
campaignsApi.report(campaignId),
runsApi.list(),
campaignsApi.timeline(campaignId),
campaignsApi.getAnalysis(campaignId),
campaignsApi.getComparison(campaignId),
])
setReport(rep.data)
setReportRuns(runs.data.filter((r) => r.campaign_id === campaignId))
setReportTimeline(tl.data.entries)
setAnalysis(ana.data)
setComparison(cmp.data)
} finally {
if (!silent) setReportLoading(false)
}
@ -288,6 +315,28 @@ export default function CampaignsPage() {
activeKey === '/campaigns' && reportOpen && !!reportId && analysis?.status === 'generating',
)
const generateComparison = async (baselineCampaignId?: string) => {
if (!reportId) return
setComparisonBusy(true)
try {
await campaignsApi.generateComparison(reportId, baselineCampaignId)
const res = await campaignsApi.getComparison(reportId)
setComparison(res.data)
} finally {
setComparisonBusy(false)
}
}
// 对比生成中同样轮询,直到 completed/failed。
usePolling(
() => {
if (!reportId) return
void campaignsApi.getComparison(reportId).then((res) => setComparison(res.data))
},
POLL_INTERVAL_MS,
activeKey === '/campaigns' && reportOpen && !!reportId && comparison?.status === 'generating',
)
// Grow the expanded timeline of any still-running campaign as new child Runs
// spawn. Completed/cancelled campaigns are fetched once on expand.
const activeExpandedIds = expandedIds.filter(
@ -636,6 +685,231 @@ export default function CampaignsPage() {
)
}
// ── 周期对比区块 ─────────────────────────────────────────────────────
const comparisonReady = analysis?.status === 'completed'
const comparisonModelName = comparison?.comparison?.model_config_id
? (data?.modelConfigs ?? []).find((m) => m.id === comparison.comparison?.model_config_id)?.name
?? comparison.comparison.model_config_id.slice(0, 8)
: null
const baselineCandidates = report
? campaigns
.filter((c) => c.target_id === report.target_id
&& c.id !== report.campaign_id
&& !isActiveStatus(c.status))
.slice()
.sort((a, b) => (b.completed_at ?? '').localeCompare(a.completed_at ?? ''))
: []
const comparisonButton = (label: string, baselineCampaignId?: string, requireBaseline = false) => (
<Tooltip title={
!analysisTerminal ? '活动完成后可生成'
: !comparisonReady ? '请先生成智能分析'
: requireBaseline && !baselineCampaignId ? '请先选择基线活动'
: undefined
}>
<Button
size="small" type="primary" ghost
loading={comparisonBusy}
disabled={!analysisTerminal || !comparisonReady || (requireBaseline && !baselineCampaignId)}
onClick={() => void generateComparison(baselineCampaignId)}
>
{label}
</Button>
</Tooltip>
)
const fmtMetricValue = (metric: 'pass_rate' | 'availability' | 'avg_latency_ms', v: number | null): string => {
if (v == null) return '—'
return metric === 'avg_latency_ms' ? `${Math.round(v)}ms` : fmtPct(v)
}
const fmtMetricDelta = (metric: 'pass_rate' | 'availability' | 'avg_latency_ms', d: number | null): string => {
if (d == null) return '—'
const sign = d > 0 ? '+' : ''
return metric === 'avg_latency_ms' ? `${sign}${Math.round(d * 10) / 10}ms` : `${sign}${(d * 100).toFixed(1)}pp`
}
const deltaColor = (metric: 'pass_rate' | 'availability' | 'avg_latency_ms', d: number | null): string => {
if (d == null || d === 0) return colors.textMuted
// 时延下降是好事;比率上升是好事。
const good = metric === 'avg_latency_ms' ? d < 0 : d > 0
return good ? '#52c41a' : '#ff4d4f'
}
const metricDiffTable = (diff: MetricDiff) => {
const rows = [
{ key: 'overall', name: '整窗(总体)', ...diff.overall },
...diff.scenarios.map((s) => ({
key: s.scenario_id,
name: s.scenario_name || scenarioNames[s.scenario_id] || s.scenario_id.slice(0, 8),
pass_rate: s.pass_rate, availability: s.availability, avg_latency_ms: s.avg_latency_ms,
})),
]
const metricCols = (['pass_rate', 'availability', 'avg_latency_ms'] as const).map((metric) => ({
title: metric === 'pass_rate' ? '通过率' : metric === 'availability' ? '可用性' : '平均时延',
key: metric,
render: (_: unknown, r: typeof rows[number]) => {
const pair = r[metric]
return (
<Space size={4}>
<span style={{ color: colors.textMuted }}>{fmtMetricValue(metric, pair.baseline)}</span>
<span style={{ color: colors.textMuted }}>→</span>
<span>{fmtMetricValue(metric, pair.current)}</span>
<span style={{ color: deltaColor(metric, pair.delta), fontWeight: 500 }}>
{fmtMetricDelta(metric, pair.delta)}
</span>
</Space>
)
},
}))
return (
<Table
rowKey="key"
size="small"
dataSource={rows}
pagination={false}
columns={[{ title: '维度', dataIndex: 'name', key: 'name' }, ...metricCols]}
style={{ marginBottom: 12 }}
/>
)
}
const renderComparisonSection = () => {
const status = comparison?.status ?? 'none'
if (status === 'generating') {
return (
<div style={analysisBoxStyle}>
<Spin size="small" />
<span style={{ marginLeft: 10, color: colors.textSecondary }}>
正在生成周期对比叙述,通常需要几十秒…
</span>
</div>
)
}
if (status === 'failed') {
return (
<div style={{ marginBottom: 16 }}>
<Alert
type="error" showIcon message="周期对比生成失败"
description={comparison?.comparison?.error || '未知错误'}
action={comparisonButton('重试', comparison?.comparison?.baseline_campaign_id)}
/>
</div>
)
}
const entry = comparison?.comparison ?? null
if (!entry || !entry.result) {
const autoBaseline = comparison?.auto_baseline ?? null
return (
<div style={{ ...analysisBoxStyle, background: colors.bgSubtle }}>
<div style={{ color: colors.textSecondary, marginBottom: 8 }}>
{autoBaseline
? <>已找到自动基线「{autoBaseline.name}」({shortDateTime(autoBaseline.completed_at)}),可生成对比叙述。</>
: '未找到自动基线(需同计划、正式线、更早完成且已有智能分析的历史活动)。可手动选择基线。'}
</div>
<Space size={12} wrap>
{autoBaseline
? comparisonButton('生成对比')
: (
<>
<Select
size="small"
style={{ minWidth: 260 }}
placeholder="选择基线活动"
value={baselineChoice}
onChange={(v) => setBaselineChoice(v)}
options={baselineCandidates.map((c) => ({
label: `${c.name}(完成于 ${shortDateTime(c.completed_at)})`,
value: c.id,
}))}
notFoundContent="没有可选的历史活动"
/>
{comparisonButton('生成对比', baselineChoice, true)}
</>
)}
{!comparisonReady && (
<span style={{ fontSize: 12, color: colors.warning }}>请先完成智能分析后再生成对比</span>
)}
</Space>
{comparison?.metric_diff && metricDiffTable(comparison.metric_diff)}
</div>
)
}
const result = entry.result
const baseline = entry.baseline
const trendMeta = TREND_META[result.trend] ?? TREND_META.stable
return (
<div style={analysisBoxStyle}>
<div style={{
display: 'flex', justifyContent: 'space-between', alignItems: 'center',
fontSize: 12, color: colors.textSecondary, marginBottom: 10,
}}>
<span>
基线「{baseline?.name ?? entry.baseline_campaign_id.slice(0, 8)}」
{baseline?.completed_at && <>(完成于 {shortDateTime(baseline.completed_at)})</>}
{' · '}模型 {comparisonModelName ?? '未知'}
{entry.updated_at && <> · 生成于 {shortDateTime(entry.updated_at)}</>}
</span>
{comparisonButton('重新生成', entry.baseline_campaign_id)}
</div>
<div style={{
background: '#e6f4ff', border: '1px solid #91caff', borderRadius: 8,
padding: '10px 14px', marginBottom: 12,
}}>
<Space size={8} align="start">
<Tag color={trendMeta.color} style={{ margin: 0 }}>{trendMeta.label}</Tag>
<span><b>总体趋势:</b>{result.summary}</span>
</Space>
</div>
{comparison?.metric_diff && metricDiffTable(comparison.metric_diff)}
{result.problem_evolution.length > 0 && (
<div style={{ marginBottom: 12 }}>
<div style={{ fontWeight: 500, marginBottom: 8 }}>问题演变</div>
<Space direction="vertical" size={8} style={{ width: '100%' }}>
{result.problem_evolution.map((p, i) => {
const meta = EVOLUTION_META[p.status] ?? EVOLUTION_META.persisting
return (
<div key={i} style={{ border: `1px solid ${colors.border}`, borderRadius: 8, padding: '8px 12px' }}>
<Space size={8} wrap>
<Tag color={meta.color} style={{ margin: 0 }}>{meta.label}</Tag>
<b>{p.title}</b>
{p.scenario_ids.map((sid) => (
<Tag key={sid} style={{ margin: 0 }}>{scenarioNames[sid] ?? sid.slice(0, 8)}</Tag>
))}
</Space>
<div style={{ color: colors.textSecondary, marginTop: 6 }}>{p.detail}</div>
</div>
)
})}
</Space>
</div>
)}
{result.suggestion_tracking.length > 0 && (
<div>
<div style={{ fontWeight: 500, marginBottom: 8 }}>建议落实情况</div>
<Space direction="vertical" size={6} style={{ width: '100%' }}>
{result.suggestion_tracking.map((s, i) => {
const meta = TRACKING_META[s.status] ?? TRACKING_META.unaddressed
return (
<div key={i}>
<Space size={8} wrap>
<Tag color={meta.color} style={{ margin: 0 }}>{meta.label}</Tag>
<span>{s.text}</span>
</Space>
{s.note && (
<div style={{ color: colors.textSecondary, fontSize: 12, margin: '2px 0 0 22px' }}>{s.note}</div>
)}
</div>
)
})}
</Space>
</div>
)}
</div>
)
}
return (
<PageWrapper
title="评估活动"
@ -942,6 +1216,9 @@ export default function CampaignsPage() {
<SectionTitle><RobotOutlined /> 智能分析</SectionTitle>
{renderAnalysisSection()}
<SectionTitle><SwapOutlined /> 周期对比</SectionTitle>
{renderComparisonSection()}
<SectionTitle>过程时间轴</SectionTitle>
<div style={{
border: `1px solid ${colors.border}`, borderRadius: 8,

View File

@ -0,0 +1,51 @@
"""add campaign_period_comparisons table
Revision ID: f2a9b7c34d18
Revises: c8f5e4b13d26
Create Date: 2026-08-03
"""
from typing import Sequence, Union
import sqlalchemy as sa
import sqlmodel # noqa: F401
from alembic import op
revision: str = "f2a9b7c34d18"
down_revision: Union[str, Sequence[str], None] = "c8f5e4b13d26"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"campaign_period_comparisons",
sa.Column("id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("campaign_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("baseline_campaign_id", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("status", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("result", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("model_config_id", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("error", sqlmodel.sql.sqltypes.AutoString(), nullable=True),
sa.Column("triggered_by", sqlmodel.sql.sqltypes.AutoString(), nullable=False),
sa.Column("created_at", sa.DateTime(), nullable=True),
sa.Column("updated_at", sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("campaign_id", name="uq_campaign_period_comparisons_campaign_id"),
)
op.create_index(
"ix_campaign_period_comparisons_campaign_id",
"campaign_period_comparisons",
["campaign_id"],
)
op.create_index(
"ix_campaign_period_comparisons_baseline_campaign_id",
"campaign_period_comparisons",
["baseline_campaign_id"],
)
def downgrade() -> None:
op.drop_index("ix_campaign_period_comparisons_baseline_campaign_id")
op.drop_index("ix_campaign_period_comparisons_campaign_id")
op.drop_table("campaign_period_comparisons")

View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "agenteval"
version = "0.7.0"
version = "0.8.0"
description = "智能体质量评估工具集平台"
readme = "README.md"
requires-python = ">=3.10"

View File

@ -0,0 +1,409 @@
"""Integration tests for /api/campaigns/{id}/comparison (v0.8 tickets 01+02)."""
from datetime import timedelta
import pytest
from agenteval.models import (
CampaignStatus,
Case,
CaseType,
ChannelType,
EvalRun,
EvalTarget,
PlatformType,
RunStatus,
RunSummary,
Scenario,
TargetStatus,
)
from agenteval.storage.db import CampaignAnalysisDB, CampaignPeriodComparisonDB, ModelConfigDB, utc_now
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import CampaignRepository, RunRepository, ScenarioRepository, TargetRepository
from agenteval.web.app import app
from httpx import ASGITransport, AsyncClient
from sqlmodel import select
@pytest.fixture()
def seeded_db(db_session, monkeypatch):
from agenteval.storage import db as db_module
from agenteval.storage import repository as repo_module
from agenteval.web import app as app_module
from agenteval.web.routers import campaigns as campaigns_module
monkeypatch.setattr(app_module, "init_db", lambda: None)
monkeypatch.setattr(campaigns_module, "start_campaign", lambda *a, **k: None)
monkeypatch.setattr(db_module, "get_session", lambda: db_session)
monkeypatch.setattr(repo_module, "get_session", lambda: db_session)
from agenteval.web.deps import get_db
def _test_get_db():
try:
yield db_session
finally:
pass
app.dependency_overrides[get_db] = _test_get_db
TargetRepository(db_session).create(EvalTarget(
id="t-1", name="mock-target",
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
channel_type=ChannelType.TUTU_API,
channel_config={"base_url": "http://mock", "token": "x"},
status=TargetStatus.ACTIVE,
))
ScenarioRepository(db_session).create(Scenario(
id="s-1", name="mock-scenario",
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
))
yield db_session
app.dependency_overrides.clear()
@pytest.fixture()
async def client():
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as c:
yield c
def _payload(name: str = "24h-cycle") -> dict:
return {
"name": name,
"target_id": "t-1",
"window_seconds": 86400,
"time_scale": 1.0,
"plan": [{"scenario_id": "s-1", "offset_seconds": 0, "count": 1}],
}
async def _create_completed_campaign(client, session, name: str, *, completed_at) -> str:
campaign_id = (await client.post("/api/campaigns", json=_payload(name))).json()["id"]
repo = CampaignRepository(session)
campaign = repo.get(campaign_id)
campaign.status = CampaignStatus.COMPLETED
campaign.completed_at = completed_at
repo.update(campaign)
return campaign_id
def _complete_analysis(session, campaign_id: str) -> None:
session.add(CampaignAnalysisDB(campaign_id=campaign_id, status="completed"))
session.commit()
def _seed_run(session, run_id: str, campaign_id: str, *, pass_rate: float, latency: float) -> None:
RunRepository(session).create(EvalRun(
id=run_id, target_id="t-1", scenario_id="s-1", campaign_id=campaign_id,
status=RunStatus.COMPLETED, started_at=utc_now(),
summary=RunSummary(total_cases=2, pass_rate=pass_rate, avg_latency_ms=latency),
))
async def test_get_comparison_missing_campaign_404(client, seeded_db):
resp = await client.get("/api/campaigns/nope/comparison")
assert resp.status_code == 404
async def test_get_comparison_without_baseline_returns_nulls(client, seeded_db):
campaign_id = await _create_completed_campaign(
client, seeded_db, "第一期", completed_at=utc_now() - timedelta(hours=1),
)
resp = await client.get(f"/api/campaigns/{campaign_id}/comparison")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "none"
assert body["comparison"] is None
assert body["auto_baseline"] is None
assert body["metric_diff"] is None
async def test_get_comparison_accelerated_line_has_no_baseline(client, seeded_db):
completed = utc_now() - timedelta(hours=1)
await _create_completed_campaign(client, seeded_db, "上期正式线", completed_at=completed - timedelta(hours=2))
# 本期为加速调试线:即便有同指纹正式线也不自动配对
payload = _payload("本期加速线")
payload["time_scale"] = 10.0
campaign_id = (await client.post("/api/campaigns", json=payload)).json()["id"]
repo = CampaignRepository(seeded_db)
campaign = repo.get(campaign_id)
campaign.status = CampaignStatus.COMPLETED
campaign.completed_at = completed
repo.update(campaign)
resp = await client.get(f"/api/campaigns/{campaign_id}/comparison")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "none"
assert body["auto_baseline"] is None
assert body["metric_diff"] is None
async def test_get_comparison_returns_baseline_and_metric_diff(client, seeded_db):
completed_now = utc_now()
baseline_id = await _create_completed_campaign(
client, seeded_db, "上期", completed_at=completed_now - timedelta(hours=2),
)
_complete_analysis(seeded_db, baseline_id)
current_id = await _create_completed_campaign(
client, seeded_db, "本期", completed_at=completed_now,
)
_seed_run(seeded_db, "run-base", baseline_id, pass_rate=0.5, latency=800.0)
_seed_run(seeded_db, "run-cur", current_id, pass_rate=0.9, latency=500.0)
resp = await client.get(f"/api/campaigns/{current_id}/comparison")
assert resp.status_code == 200
body = resp.json()
assert body["auto_baseline"]["id"] == baseline_id
assert body["auto_baseline"]["name"] == "上期"
assert body["auto_baseline"]["completed_at"]
diff = body["metric_diff"]
assert diff["overall"]["pass_rate"] == {"baseline": 0.5, "current": 0.9, "delta": 0.4}
assert diff["overall"]["availability"]["baseline"] == 1.0
assert diff["overall"]["availability"]["delta"] == 0.0
assert diff["overall"]["avg_latency_ms"] == {"baseline": 800.0, "current": 500.0, "delta": -300.0}
assert [s["scenario_id"] for s in diff["scenarios"]] == ["s-1"]
assert diff["scenarios"][0]["pass_rate"]["delta"] == 0.4
async def test_get_comparison_baseline_without_analysis_is_skipped(client, seeded_db):
completed_now = utc_now()
baseline_id = await _create_completed_campaign(
client, seeded_db, "上期无分析", completed_at=completed_now - timedelta(hours=2),
)
_seed_run(seeded_db, "run-base", baseline_id, pass_rate=0.5, latency=800.0)
current_id = await _create_completed_campaign(
client, seeded_db, "本期", completed_at=completed_now,
)
_seed_run(seeded_db, "run-cur", current_id, pass_rate=0.9, latency=500.0)
resp = await client.get(f"/api/campaigns/{current_id}/comparison")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "none"
assert body["auto_baseline"] is None
assert body["metric_diff"] is None
# ── ticket 02: POST + GET 合并 ────────────────────────────────────────────
def _seed_analysis_default(session) -> None:
ModelConfigRepository(session).create(ModelConfigDB(
id="mc-1", name="analysis-cfg", provider="openai_compatible", capability="chat",
endpoint_url="https://models.example.com/v1/chat/completions", model_name="m",
is_analysis_default=True,
))
def _complete_analysis_with_result(session, campaign_id: str) -> None:
row = CampaignAnalysisDB(campaign_id=campaign_id, status="completed")
row.set_result({
"overall": "整体达标",
"problems": [],
"scenario_narratives": [{"scenario_id": "s-1", "narrative": "表现稳定"}],
"suggestions": [{"priority": 1, "text": "保持"}],
})
existing = session.exec(
select(CampaignAnalysisDB).where(CampaignAnalysisDB.campaign_id == campaign_id)
).first()
if existing:
session.delete(existing)
session.commit()
session.add(row)
session.commit()
def _complete_comparison_row(session, campaign_id: str, baseline_id: str) -> None:
row = CampaignPeriodComparisonDB(
campaign_id=campaign_id, baseline_campaign_id=baseline_id,
status="completed", model_config_id="mc-1", triggered_by="manual",
)
row.set_result({
"trend": "improving",
"summary": "通过率提升",
"problem_evolution": [{"status": "resolved", "title": "答非所问", "detail": "已缓解", "scenario_ids": ["s-1"]}],
"suggestion_tracking": [{"text": "保持", "status": "addressed", "note": "已落实"}],
})
existing = session.exec(
select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == campaign_id)
).first()
if existing:
session.delete(existing)
session.commit()
session.add(row)
session.commit()
async def _two_period_setup(client, seeded_db):
"""上期(completed + 分析)+ 本期(completed + 分析),返回 (baseline_id, current_id)。"""
completed_now = utc_now()
baseline_id = await _create_completed_campaign(
client, seeded_db, "上期", completed_at=completed_now - timedelta(hours=2),
)
_complete_analysis_with_result(seeded_db, baseline_id)
current_id = await _create_completed_campaign(
client, seeded_db, "本期", completed_at=completed_now,
)
_complete_analysis_with_result(seeded_db, current_id)
return baseline_id, current_id
async def test_post_comparison_rejects_non_terminal(client, seeded_db):
_seed_analysis_default(seeded_db)
payload = _payload("运行中")
campaign_id = (await client.post("/api/campaigns", json=payload)).json()["id"]
repo = CampaignRepository(seeded_db)
campaign = repo.get(campaign_id)
campaign.status = CampaignStatus.RUNNING
repo.update(campaign)
resp = await client.post(f"/api/campaigns/{campaign_id}/comparison")
assert resp.status_code == 400
async def test_post_comparison_requires_current_analysis(client, seeded_db):
_seed_analysis_default(seeded_db)
campaign_id = await _create_completed_campaign(
client, seeded_db, "无分析本期", completed_at=utc_now(),
)
resp = await client.post(f"/api/campaigns/{campaign_id}/comparison")
assert resp.status_code == 400
assert "智能分析" in resp.json()["detail"]
async def test_post_comparison_requires_model(client, seeded_db):
baseline_id, current_id = await _two_period_setup(client, seeded_db)
# 不配置分析默认模型
resp = await client.post(f"/api/campaigns/{current_id}/comparison")
assert resp.status_code == 400
assert "分析" in resp.json()["detail"]
async def test_post_comparison_without_baseline_guides_manual_selection(client, seeded_db):
_seed_analysis_default(seeded_db)
# 本期有分析但无同指纹历史活动
campaign_id = await _create_completed_campaign(
client, seeded_db, "孤立本期", completed_at=utc_now(),
)
_complete_analysis_with_result(seeded_db, campaign_id)
resp = await client.post(f"/api/campaigns/{campaign_id}/comparison")
assert resp.status_code == 400
assert "基线" in resp.json()["detail"]
async def test_post_comparison_auto_baseline_success(client, seeded_db, monkeypatch):
from agenteval.web.routers import campaigns as campaigns_module
_seed_analysis_default(seeded_db)
baseline_id, current_id = await _two_period_setup(client, seeded_db)
_seed_run(seeded_db, "run-base", baseline_id, pass_rate=0.5, latency=800.0)
_seed_run(seeded_db, "run-cur", current_id, pass_rate=0.9, latency=500.0)
monkeypatch.setattr(
campaigns_module,
"start_campaign_comparison",
lambda cid, *, triggered_by, baseline_campaign_id=None: _complete_comparison_row(
seeded_db, cid, baseline_campaign_id
),
)
resp = await client.post(f"/api/campaigns/{current_id}/comparison")
assert resp.status_code == 200
assert resp.json() == {"status": "generating"}
got = (await client.get(f"/api/campaigns/{current_id}/comparison")).json()
assert got["status"] == "completed"
assert got["comparison"]["baseline_campaign_id"] == baseline_id
assert got["comparison"]["baseline"]["name"] == "上期"
assert got["comparison"]["result"]["trend"] == "improving"
assert got["comparison"]["model_config_id"] == "mc-1"
# metric_diff 按生效基线(=存储基线)现算
assert got["metric_diff"]["overall"]["pass_rate"]["delta"] == 0.4
async def test_post_comparison_manual_baseline_across_fingerprints(client, seeded_db, monkeypatch):
from agenteval.web.routers import campaigns as campaigns_module
_seed_analysis_default(seeded_db)
# 本期:24h 窗口(无同指纹历史活动 → 无自动基线)
current_id = await _create_completed_campaign(client, seeded_db, "本期", completed_at=utc_now())
_complete_analysis_with_result(seeded_db, current_id)
# 手动基线:不同窗口(不同指纹)的历史活动,有 completed 分析
other_payload = _payload("异指纹基线")
other_payload["window_seconds"] = 3600
other_id = (await client.post("/api/campaigns", json=other_payload)).json()["id"]
repo = CampaignRepository(seeded_db)
other = repo.get(other_id)
other.status = CampaignStatus.COMPLETED
other.completed_at = utc_now() - timedelta(hours=2)
repo.update(other)
_complete_analysis_with_result(seeded_db, other_id)
monkeypatch.setattr(
campaigns_module,
"start_campaign_comparison",
lambda cid, *, triggered_by, baseline_campaign_id=None: _complete_comparison_row(
seeded_db, cid, baseline_campaign_id
),
)
# 不指定基线 → 无自动基线 400
resp = await client.post(f"/api/campaigns/{current_id}/comparison")
assert resp.status_code == 400
# 手动指定跨指纹基线 → 成功
resp = await client.post(
f"/api/campaigns/{current_id}/comparison",
json={"baseline_campaign_id": other_id},
)
assert resp.status_code == 200
got = (await client.get(f"/api/campaigns/{current_id}/comparison")).json()
assert got["comparison"]["baseline_campaign_id"] == other_id
async def test_post_comparison_manual_baseline_without_analysis_400(client, seeded_db):
_seed_analysis_default(seeded_db)
baseline_id, current_id = await _two_period_setup(client, seeded_db)
# 再建一个无分析的历史活动作为手动基线
ghost_id = await _create_completed_campaign(
client, seeded_db, "无分析基线", completed_at=utc_now() - timedelta(hours=3),
)
resp = await client.post(
f"/api/campaigns/{current_id}/comparison",
json={"baseline_campaign_id": ghost_id},
)
assert resp.status_code == 400
assert "基线" in resp.json()["detail"]
async def test_post_comparison_rerun_upserts_without_new_row(client, seeded_db, monkeypatch):
from agenteval.web.routers import campaigns as campaigns_module
_seed_analysis_default(seeded_db)
baseline_id, current_id = await _two_period_setup(client, seeded_db)
monkeypatch.setattr(
campaigns_module,
"start_campaign_comparison",
lambda cid, *, triggered_by, baseline_campaign_id=None: _complete_comparison_row(
seeded_db, cid, baseline_campaign_id
),
)
await client.post(f"/api/campaigns/{current_id}/comparison")
await client.post(f"/api/campaigns/{current_id}/comparison")
rows = seeded_db.exec(
select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == current_id)
).all()
assert len(rows) == 1

View File

@ -0,0 +1,171 @@
"""Auto-chain tests: analysis completed → period comparison enqueued (v0.8 ticket 04).
Only 正式线 campaigns with a resolvable model and an auto baseline enqueue the
comparison task; all other branches skip silently without touching the analysis
row. The comparison entry point is spied, not executed.
"""
import json
from datetime import timedelta
from types import SimpleNamespace
import pytest
from agenteval.evaluation import analysis as analysis_module
from agenteval.evaluation import comparison as comparison_module
from agenteval.evaluation.analysis import execute_campaign_analysis
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, EvalRun, RunStatus, RunSummary
from agenteval.storage.db import CampaignAnalysisDB, utc_now
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import CampaignRepository, RunRepository
from sqlmodel import select
T0 = utc_now().replace(tzinfo=None) - timedelta(hours=3)
STAGE1 = json.dumps({"narrative": "售前场景表现稳定", "problems": []})
STAGE2 = json.dumps({"overall": "整体达标", "problems": [], "suggestions": []})
class FakeChatClient:
def __init__(self, *responses):
self._responses = list(responses)
async def __call__(self, messages: list[dict]) -> str:
return self._responses.pop(0)
def _campaign(campaign_id: str, *, time_scale: float = 1.0, completed_at) -> Campaign:
return Campaign(
id=campaign_id, name=f"campaign-{campaign_id}", target_id="t-1",
window_seconds=86400, time_scale=time_scale,
plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=1)],
status=CampaignStatus.COMPLETED, completed_at=completed_at,
)
def _seed_config(session) -> None:
from agenteval.storage.db import ModelConfigDB
ModelConfigRepository(session).create(ModelConfigDB(
id="mc-1", name="analysis-cfg", provider="openai_compatible", capability="chat",
endpoint_url="https://models.example.com/v1/chat/completions", model_name="m",
is_analysis_default=True,
))
def _seed_baseline(session, completed_at) -> None:
CampaignRepository(session).create(_campaign("camp-base", completed_at=completed_at))
row = CampaignAnalysisDB(campaign_id="camp-base", status="completed")
row.set_result({"overall": "上期达标", "problems": [], "scenario_narratives": [], "suggestions": []})
session.add(row)
session.commit()
def _seed_current(session, *, time_scale: float = 1.0, completed_at) -> None:
CampaignRepository(session).create(
_campaign("camp-cur", time_scale=time_scale, completed_at=completed_at)
)
RunRepository(session).create(EvalRun(
id="run-cur", target_id="t-1", scenario_id="s-1", campaign_id="camp-cur",
status=RunStatus.COMPLETED, started_at=utc_now(),
summary=RunSummary(total_cases=2, pass_rate=1.0, avg_latency_ms=300),
))
@pytest.fixture()
def comparison_spy(monkeypatch):
calls: list[tuple[str, str]] = []
def _spy(campaign_id, *, triggered_by, baseline_campaign_id=None):
calls.append((campaign_id, triggered_by))
monkeypatch.setattr(comparison_module, "start_campaign_comparison", _spy)
return calls
async def test_production_line_with_baseline_enqueues_comparison(db_session, monkeypatch, comparison_spy):
monkeypatch.setattr(analysis_module, "get_session", lambda: db_session)
_seed_config(db_session)
_seed_baseline(db_session, T0)
_seed_current(db_session, completed_at=T0 + timedelta(hours=2))
await execute_campaign_analysis(
"camp-cur", triggered_by="auto", chat_client=FakeChatClient(STAGE1, STAGE2),
)
analysis_row = db_session.exec(
select(CampaignAnalysisDB).where(CampaignAnalysisDB.campaign_id == "camp-cur")
).one()
assert analysis_row.status == "completed"
assert comparison_spy == [("camp-cur", "auto")]
async def test_accelerated_line_does_not_enqueue(db_session, monkeypatch, comparison_spy):
monkeypatch.setattr(analysis_module, "get_session", lambda: db_session)
_seed_config(db_session)
_seed_baseline(db_session, T0)
_seed_current(db_session, time_scale=24.0, completed_at=T0 + timedelta(hours=2))
await execute_campaign_analysis(
"camp-cur", triggered_by="auto", chat_client=FakeChatClient(STAGE1, STAGE2),
)
assert comparison_spy == []
async def test_no_auto_baseline_does_not_enqueue(db_session, monkeypatch, comparison_spy):
monkeypatch.setattr(analysis_module, "get_session", lambda: db_session)
_seed_config(db_session)
_seed_current(db_session, completed_at=T0 + timedelta(hours=2)) # 无历史活动
await execute_campaign_analysis(
"camp-cur", triggered_by="auto", chat_client=FakeChatClient(STAGE1, STAGE2),
)
assert comparison_spy == []
async def test_missing_model_skips_comparison(db_session, monkeypatch, comparison_spy):
monkeypatch.setattr(analysis_module, "get_session", lambda: db_session)
_seed_config(db_session)
_seed_baseline(db_session, T0)
_seed_current(db_session, completed_at=T0 + timedelta(hours=2))
# 分析本体解析得到模型(成功),自动链再次解析时模型已不可用 → 静默跳过
calls = {"n": 0}
def _resolve(campaign, session):
calls["n"] += 1
return SimpleNamespace(id="mc-1") if calls["n"] == 1 else None
monkeypatch.setattr(analysis_module, "resolve_analysis_model", _resolve)
await execute_campaign_analysis(
"camp-cur", triggered_by="auto", chat_client=FakeChatClient(STAGE1, STAGE2),
)
analysis_row = db_session.exec(
select(CampaignAnalysisDB).where(CampaignAnalysisDB.campaign_id == "camp-cur")
).one()
assert analysis_row.status == "completed"
assert comparison_spy == []
async def test_comparison_enqueue_error_does_not_affect_analysis(db_session, monkeypatch):
monkeypatch.setattr(analysis_module, "get_session", lambda: db_session)
_seed_config(db_session)
_seed_baseline(db_session, T0)
_seed_current(db_session, completed_at=T0 + timedelta(hours=2))
def _boom(campaign_id, *, triggered_by, baseline_campaign_id=None):
raise RuntimeError("enqueue failed")
monkeypatch.setattr(comparison_module, "start_campaign_comparison", _boom)
await execute_campaign_analysis(
"camp-cur", triggered_by="auto", chat_client=FakeChatClient(STAGE1, STAGE2),
)
analysis_row = db_session.exec(
select(CampaignAnalysisDB).where(CampaignAnalysisDB.campaign_id == "camp-cur")
).one()
assert analysis_row.status == "completed" # 异常不冒泡,分析结果不受影响

View File

@ -0,0 +1,448 @@
"""Auto-baseline pairing + mechanical metric diff + narrative orchestration (v0.8)."""
import json
from datetime import timedelta
import pytest
from agenteval.evaluation.comparison import (
ComparisonError,
campaign_plan_fingerprint,
compute_metric_diff,
execute_campaign_comparison,
narrate_period_comparison,
resolve_auto_baseline,
)
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, EvalRun, RunStatus, RunSummary
from agenteval.storage.db import (
CampaignAnalysisDB,
CampaignPeriodComparisonDB,
ModelConfigDB,
utc_now,
)
from agenteval.storage.model_config_repository import ModelConfigRepository
from agenteval.storage.repository import CampaignRepository, RunRepository
from sqlmodel import select
T0 = utc_now().replace(tzinfo=None) - timedelta(hours=1)
def _campaign(
campaign_id: str = "camp-1",
*,
target_id: str = "t-1",
window_seconds: int = 86400,
time_scale: float = 1.0,
plan: list[CampaignPlanEntry] | None = None,
status: CampaignStatus = CampaignStatus.COMPLETED,
completed_at=T0 + timedelta(hours=1),
) -> Campaign:
return Campaign(
id=campaign_id,
name=f"campaign-{campaign_id}",
target_id=target_id,
window_seconds=window_seconds,
time_scale=time_scale,
plan=plan or [
CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=2),
CampaignPlanEntry(scenario_id="s-2", offset_seconds=3600, count=1),
],
status=status,
completed_at=completed_at,
)
# ── 计划指纹 ──────────────────────────────────────────────────────────────
def test_fingerprint_ignores_plan_entry_order():
a = _campaign()
b = _campaign(plan=list(reversed(a.plan)))
assert campaign_plan_fingerprint(a) == campaign_plan_fingerprint(b)
def test_fingerprint_changes_with_any_plan_dimension():
base = campaign_plan_fingerprint(_campaign())
assert campaign_plan_fingerprint(_campaign(target_id="t-2")) != base
assert campaign_plan_fingerprint(_campaign(window_seconds=3600)) != base
assert campaign_plan_fingerprint(
_campaign(plan=[CampaignPlanEntry(scenario_id="s-9", offset_seconds=0, count=2),
CampaignPlanEntry(scenario_id="s-2", offset_seconds=3600, count=1)])
) != base
assert campaign_plan_fingerprint(
_campaign(plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=60, count=2),
CampaignPlanEntry(scenario_id="s-2", offset_seconds=3600, count=1)])
) != base
assert campaign_plan_fingerprint(
_campaign(plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=3),
CampaignPlanEntry(scenario_id="s-2", offset_seconds=3600, count=1)])
) != base
def test_fingerprint_ignores_time_scale():
assert campaign_plan_fingerprint(_campaign()) == campaign_plan_fingerprint(_campaign(time_scale=24.0))
# ── 自动基线解析 ──────────────────────────────────────────────────────────
def _seed_baseline(session, campaign: Campaign, *, analysis_status: str | None = "completed") -> None:
CampaignRepository(session).create(campaign)
if analysis_status is not None:
session.add(CampaignAnalysisDB(campaign_id=campaign.id, status=analysis_status))
session.commit()
def test_baseline_picks_most_recent_with_completed_analysis(db_session):
old = _campaign("camp-old", completed_at=T0)
recent = _campaign("camp-recent", completed_at=T0 + timedelta(minutes=30))
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_baseline(db_session, old)
_seed_baseline(db_session, recent)
resolved = resolve_auto_baseline(current, db_session)
assert resolved is not None and resolved.id == "camp-recent"
def test_baseline_skips_candidates_without_completed_analysis(db_session):
no_analysis = _campaign("camp-no", completed_at=T0)
failed_analysis = _campaign("camp-failed", completed_at=T0 + timedelta(minutes=30))
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_baseline(db_session, no_analysis, analysis_status=None)
_seed_baseline(db_session, failed_analysis, analysis_status="failed")
assert resolve_auto_baseline(current, db_session) is None
def test_baseline_skips_accelerated_candidates_and_accelerated_current(db_session):
accelerated = _campaign("camp-fast", time_scale=24.0, completed_at=T0)
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_baseline(db_session, accelerated)
assert resolve_auto_baseline(current, db_session) is None
# 本期自身为加速线 → 无自动基线
production = _campaign("camp-prod", completed_at=T0)
_seed_baseline(db_session, production)
assert resolve_auto_baseline(_campaign("camp-cur", time_scale=12.0), db_session) is None
def test_baseline_skips_different_fingerprint(db_session):
other_plan = _campaign("camp-other", window_seconds=3600, completed_at=T0)
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_baseline(db_session, other_plan)
assert resolve_auto_baseline(current, db_session) is None
def test_baseline_skips_later_or_same_moment_completion(db_session):
later = _campaign("camp-later", completed_at=T0 + timedelta(hours=3))
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_baseline(db_session, later)
assert resolve_auto_baseline(current, db_session) is None
same_moment = _campaign("camp-same", completed_at=current.completed_at)
_seed_baseline(db_session, same_moment)
assert resolve_auto_baseline(current, db_session) is None
def test_baseline_skips_uncompleted_candidates(db_session):
uncompleted = _campaign("camp-run", status=CampaignStatus.RUNNING, completed_at=None)
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_baseline(db_session, uncompleted)
assert resolve_auto_baseline(current, db_session) is None
# ── 机械指标 diff ─────────────────────────────────────────────────────────
def _report(summary: dict, capability: list[dict]) -> dict:
return {"summary": summary, "capability_summary": capability}
def test_metric_diff_computes_overall_and_scenario_deltas():
baseline = _report(
{"overall_pass_rate": 0.5, "overall_availability": 1.0, "avg_latency_ms": 800.0},
[
{"scenario_id": "s-1", "scenario_name": "售前", "pass_rate": 0.4,
"availability": 1.0, "avg_latency_ms": 700.0},
{"scenario_id": "s-2", "scenario_name": "售后", "pass_rate": 0.6,
"availability": 1.0, "avg_latency_ms": 900.0},
],
)
current = _report(
{"overall_pass_rate": 0.8, "overall_availability": 0.75, "avg_latency_ms": 500.0},
[
{"scenario_id": "s-1", "scenario_name": "售前", "pass_rate": 0.9,
"availability": 0.5, "avg_latency_ms": 400.0},
{"scenario_id": "s-2", "scenario_name": "售后", "pass_rate": 0.6,
"availability": 1.0, "avg_latency_ms": 900.0},
],
)
diff = compute_metric_diff(baseline, current)
assert diff["overall"]["pass_rate"] == {"baseline": 0.5, "current": 0.8, "delta": 0.3}
assert diff["overall"]["availability"] == {"baseline": 1.0, "current": 0.75, "delta": -0.25}
assert diff["overall"]["avg_latency_ms"] == {"baseline": 800.0, "current": 500.0, "delta": -300.0}
scenarios = {s["scenario_id"]: s for s in diff["scenarios"]}
assert scenarios["s-1"]["pass_rate"] == {"baseline": 0.4, "current": 0.9, "delta": 0.5}
assert scenarios["s-1"]["availability"] == {"baseline": 1.0, "current": 0.5, "delta": -0.5}
assert scenarios["s-1"]["avg_latency_ms"] == {"baseline": 700.0, "current": 400.0, "delta": -300.0}
assert scenarios["s-2"]["pass_rate"]["delta"] == 0.0
assert [s["scenario_name"] for s in diff["scenarios"]] == ["售前", "售后"]
def test_metric_diff_union_of_scenarios_missing_side_yields_none():
baseline = _report(
{"overall_pass_rate": 0.5, "overall_availability": 1.0, "avg_latency_ms": 800.0},
[{"scenario_id": "s-1", "scenario_name": "售前", "pass_rate": 0.4,
"availability": 1.0, "avg_latency_ms": 700.0}],
)
current = _report(
{"overall_pass_rate": 0.8, "overall_availability": 1.0, "avg_latency_ms": 500.0},
[{"scenario_id": "s-2", "scenario_name": "售后", "pass_rate": 0.9,
"availability": 1.0, "avg_latency_ms": 400.0}],
)
diff = compute_metric_diff(baseline, current)
scenarios = {s["scenario_id"]: s for s in diff["scenarios"]}
assert set(scenarios) == {"s-1", "s-2"}
# s-1 仅基线有:current/delta 为 None
assert scenarios["s-1"]["pass_rate"] == {"baseline": 0.4, "current": None, "delta": None}
assert scenarios["s-1"]["avg_latency_ms"]["delta"] is None
# s-2 仅本期有:baseline/delta 为 None
assert scenarios["s-2"]["pass_rate"] == {"baseline": None, "current": 0.9, "delta": None}
assert scenarios["s-2"]["availability"]["baseline"] is None
def test_metric_diff_none_metrics_propagate():
baseline = _report(
{"overall_pass_rate": None, "overall_availability": None, "avg_latency_ms": None},
[{"scenario_id": "s-1", "scenario_name": "售前", "pass_rate": None,
"availability": None, "avg_latency_ms": None}],
)
current = _report(
{"overall_pass_rate": 0.8, "overall_availability": 1.0, "avg_latency_ms": 500.0},
[{"scenario_id": "s-1", "scenario_name": "售前", "pass_rate": 0.9,
"availability": 1.0, "avg_latency_ms": 400.0}],
)
diff = compute_metric_diff(baseline, current)
assert diff["overall"]["pass_rate"]["delta"] is None
assert diff["scenarios"][0]["pass_rate"]["delta"] is None
assert diff["scenarios"][0]["pass_rate"]["current"] == 0.9
def test_metric_diff_empty_capability_summaries():
diff = compute_metric_diff(
_report({"overall_pass_rate": None, "overall_availability": None, "avg_latency_ms": None}, []),
_report({"overall_pass_rate": None, "overall_availability": None, "avg_latency_ms": None}, []),
)
assert diff["scenarios"] == []
assert diff["overall"]["pass_rate"]["delta"] is None
# ── 叙述编排(单次 LLM 调用) ─────────────────────────────────────────────
class FakeChatClient:
"""Queued-response fake for the comparison LLM seam."""
def __init__(self, *responses):
self._responses = list(responses)
self.calls: list[list[dict]] = []
async def __call__(self, messages: list[dict]) -> str:
self.calls.append(messages)
if not self._responses:
raise AssertionError("unexpected extra LLM call")
item = self._responses.pop(0)
if isinstance(item, Exception):
raise item
return item
def _analysis(overall: str) -> dict:
return {
"overall": overall,
"problems": [{"severity": "high", "title": "答非所问", "scenario_ids": ["s-1"]}],
"suggestions": [{"priority": 1, "text": "补充意图语料"}],
}
def _diff() -> dict:
return compute_metric_diff(
{"summary": {"overall_pass_rate": 0.5, "overall_availability": 1.0, "avg_latency_ms": 800.0},
"capability_summary": [{"scenario_id": "s-1", "pass_rate": 0.5, "availability": 1.0, "avg_latency_ms": 800.0}]},
{"summary": {"overall_pass_rate": 0.9, "overall_availability": 1.0, "avg_latency_ms": 500.0},
"capability_summary": [{"scenario_id": "s-1", "pass_rate": 0.9, "availability": 1.0, "avg_latency_ms": 500.0}]},
)
NARRATION = json.dumps({
"trend": "improving",
"summary": "整体通过率显著提升,售前答非所问问题缓解",
"problem_evolution": [
{"status": "resolved", "title": "答非所问", "detail": "语料补充后恢复",
"scenario_ids": ["s-1", "ghost-scenario"]},
{"status": "nonsense", "title": "新问题", "detail": "...", "scenario_ids": []},
],
"suggestion_tracking": [
{"text": "补充意图语料", "status": "addressed", "note": "已落实"},
{"text": "排查上游", "status": "nonsense", "note": "状态未知"},
{"text": "本期新增建议", "status": "new", "note": ""},
],
})
async def test_narration_input_carries_both_analyses_and_diff():
client = FakeChatClient(NARRATION)
await narrate_period_comparison(
baseline_analysis=_analysis("上期整体不达标"),
current_analysis=_analysis("本期整体改善"),
metric_diff=_diff(),
valid_scenario_ids={"s-1"},
chat_client=client,
)
assert len(client.calls) == 1
prompt = str(client.calls[0])
assert "上期整体不达标" in prompt
assert "本期整体改善" in prompt
assert "pass_rate" in prompt # 机械 diff 进入输入
async def test_narration_normalizes_and_whitelists():
client = FakeChatClient(NARRATION)
result = await narrate_period_comparison(
baseline_analysis=_analysis("a"),
current_analysis=_analysis("b"),
metric_diff=_diff(),
valid_scenario_ids={"s-1"},
chat_client=client,
)
assert result["trend"] == "improving"
assert result["summary"].startswith("整体通过率显著提升")
resolved = result["problem_evolution"][0]
assert resolved["status"] == "resolved"
assert resolved["scenario_ids"] == ["s-1"] # ghost-scenario 剔除
assert result["problem_evolution"][1]["status"] == "persisting" # 非法枚举归一
statuses = [s["status"] for s in result["suggestion_tracking"]]
assert statuses == ["addressed", "unaddressed", "new"] # nonsense → unaddressed
async def test_narration_normalizes_invalid_trend_to_stable():
client = FakeChatClient(json.dumps({"trend": "wild", "summary": "结论"}))
result = await narrate_period_comparison(
baseline_analysis={}, current_analysis={}, metric_diff={},
valid_scenario_ids=set(), chat_client=client,
)
assert result["trend"] == "stable"
assert result["problem_evolution"] == []
assert result["suggestion_tracking"] == []
async def test_unparseable_narration_raises_comparison_error():
client = FakeChatClient("这不是 JSON")
with pytest.raises(ComparisonError):
await narrate_period_comparison(
baseline_analysis={}, current_analysis={}, metric_diff={},
valid_scenario_ids=set(), chat_client=client,
)
async def test_narration_missing_summary_raises():
client = FakeChatClient(json.dumps({"trend": "stable"}))
with pytest.raises(ComparisonError):
await narrate_period_comparison(
baseline_analysis={}, current_analysis={}, metric_diff={},
valid_scenario_ids=set(), chat_client=client,
)
# ── 后台执行状态机 ───────────────────────────────────────────────────────
def _seed_config(session, config_id: str) -> None:
ModelConfigRepository(session).create(ModelConfigDB(
id=config_id, name=f"cfg-{config_id}", provider="openai_compatible", capability="chat",
endpoint_url="https://models.example.com/v1/chat/completions", model_name="m",
is_analysis_default=True,
))
def _seed_campaign_with_analysis(session, campaign: Campaign, analysis_result: dict) -> None:
CampaignRepository(session).create(campaign)
row = CampaignAnalysisDB(campaign_id=campaign.id, status="completed")
row.set_result(analysis_result)
session.add(row)
session.commit()
RunRepository(session).create(EvalRun(
id=f"run-{campaign.id}", target_id="t-1", scenario_id="s-1",
campaign_id=campaign.id, status=RunStatus.COMPLETED, started_at=utc_now(),
summary=RunSummary(total_cases=2, pass_rate=0.5, avg_latency_ms=700),
))
async def test_execute_writes_completed_row_with_baseline_snapshot(db_session, monkeypatch):
from agenteval.evaluation import comparison as comparison_module
monkeypatch.setattr(comparison_module, "get_session", lambda: db_session)
_seed_config(db_session, "mc-default")
baseline = _campaign("camp-base", completed_at=T0)
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_campaign_with_analysis(db_session, baseline, _analysis("上期"))
_seed_campaign_with_analysis(db_session, current, _analysis("本期"))
await execute_campaign_comparison(
"camp-cur", triggered_by="manual", chat_client=FakeChatClient(NARRATION),
)
row = db_session.exec(
select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == "camp-cur")
).one()
assert row.status == "completed"
assert row.baseline_campaign_id == "camp-base"
assert row.model_config_id == "mc-default"
assert row.get_result()["trend"] == "improving"
async def test_execute_records_failure_on_unparseable_output(db_session, monkeypatch):
from agenteval.evaluation import comparison as comparison_module
monkeypatch.setattr(comparison_module, "get_session", lambda: db_session)
_seed_config(db_session, "mc-default")
baseline = _campaign("camp-base", completed_at=T0)
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
_seed_campaign_with_analysis(db_session, baseline, _analysis("上期"))
_seed_campaign_with_analysis(db_session, current, _analysis("本期"))
await execute_campaign_comparison(
"camp-cur", triggered_by="manual", chat_client=FakeChatClient("garbage"),
)
row = db_session.exec(
select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == "camp-cur")
).one()
assert row.status == "failed"
assert row.error
assert row.baseline_campaign_id == "camp-base"
async def test_execute_fails_without_baseline_analysis(db_session, monkeypatch):
from agenteval.evaluation import comparison as comparison_module
monkeypatch.setattr(comparison_module, "get_session", lambda: db_session)
_seed_config(db_session, "mc-default")
baseline = _campaign("camp-base", completed_at=T0)
current = _campaign("camp-cur", completed_at=T0 + timedelta(hours=2))
CampaignRepository(db_session).create(baseline) # 基线无分析行
_seed_campaign_with_analysis(db_session, current, _analysis("本期"))
await execute_campaign_comparison(
"camp-cur", triggered_by="manual", chat_client=FakeChatClient(NARRATION),
)
row = db_session.exec(
select(CampaignPeriodComparisonDB).where(CampaignPeriodComparisonDB.campaign_id == "camp-cur")
).one()
assert row.status == "failed"
assert "基线" in row.error

View File

@ -0,0 +1,31 @@
"""Verify the campaign_period_comparisons migration creates the table with a unique campaign_id."""
import importlib
import sqlalchemy as sa
from alembic.migration import MigrationContext
from alembic.operations import Operations
def test_period_comparisons_migration_creates_table(tmp_path, monkeypatch):
engine = sa.create_engine(f"sqlite:///{tmp_path / 'comparisons.db'}")
with engine.begin() as connection:
operations = Operations(MigrationContext.configure(connection))
migration = importlib.import_module(
"migrations.versions.f2a9b7c34d18_add_campaign_period_comparisons"
)
monkeypatch.setattr(migration, "op", operations)
migration.upgrade()
inspector = sa.inspect(connection)
assert "campaign_period_comparisons" in inspector.get_table_names()
columns = {c["name"] for c in inspector.get_columns("campaign_period_comparisons")}
assert {
"id", "campaign_id", "baseline_campaign_id", "status", "result",
"model_config_id", "error", "triggered_by", "created_at", "updated_at",
} <= columns
uniques = inspector.get_unique_constraints("campaign_period_comparisons")
assert any(uc["column_names"] == ["campaign_id"] for uc in uniques)