Compare commits
18 Commits
3e485bfbe6
...
a665b496b0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a665b496b0 | ||
|
|
b26c432f3a | ||
|
|
ef4c094082 | ||
|
|
936640fb36 | ||
|
|
7f68afd765 | ||
|
|
9abf572949 | ||
|
|
958cefc380 | ||
|
|
140bb1eb89 | ||
|
|
fc167a8e4e | ||
|
|
3c97e6e1ba | ||
|
|
62f7031b5b | ||
|
|
b21dce73c5 | ||
|
|
a12eb09da3 | ||
|
|
51980016a4 | ||
|
|
2484c207af | ||
|
|
9cf64ab0e2 | ||
|
|
8ed4221b0f | ||
|
|
2285a25009 |
@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
**Blocked by:** 01(会话关闭事件与对话数据)
|
**Blocked by:** 01(会话关闭事件与对话数据)
|
||||||
|
|
||||||
**Status:** ready-for-agent
|
**Status:** completed
|
||||||
|
|
||||||
- [ ] 会话关闭后自动触发抽样复核,结果结构化落库
|
- [x] 会话关闭后自动触发抽样复核,结果结构化落库
|
||||||
- [ ] 抽样上限生效;无模型配置时静默跳过
|
- [x] 抽样上限生效;无模型配置时静默跳过
|
||||||
- [ ] 复核失败落错误,会话仍为 completed
|
- [x] 复核失败落错误,会话仍为 completed
|
||||||
- [ ] 假客户端测试覆盖编排、解析失败、无模型分支(先例:v0.7 分析测试)
|
- [x] 假客户端测试覆盖编排、解析失败、无模型分支(先例:v0.7 分析测试)
|
||||||
|
|||||||
@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
**Blocked by:** 01(聚合主料是体验记录);04 若已就位则复核结论一并纳入,未就位不阻塞
|
**Blocked by:** 01(聚合主料是体验记录);04 若已就位则复核结论一并纳入,未就位不阻塞
|
||||||
|
|
||||||
**Status:** ready-for-agent
|
**Status:** completed
|
||||||
|
|
||||||
- [ ] 活动报告聚合含探索发现维度,达成率与问题清单口径正确
|
- [x] 活动报告聚合含探索发现维度,达成率与问题清单口径正确
|
||||||
- [ ] 分析输入含探索摘要且不含全量对话
|
- [x] 分析输入含探索摘要且不含全量对话
|
||||||
- [ ] Markdown 导出追加探索附录;无数据时逐字节不变(缺则无痕)
|
- [x] Markdown 导出追加探索附录;无数据时逐字节不变(缺则无痕)
|
||||||
- [ ] 纯函数渲染与聚合单测 + 导出集成测试(先例:report_render 单测、活动导出测试)
|
- [x] 纯函数渲染与聚合单测 + 导出集成测试(先例:report_render 单测、活动导出测试)
|
||||||
|
|||||||
@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
**Blocked by:** 05(报告聚合与查询端点就绪)
|
**Blocked by:** 05(报告聚合与查询端点就绪)
|
||||||
|
|
||||||
**Status:** ready-for-agent
|
**Status:** completed(a12eb09 + 9abf572 回复文本修复;Playwright 浏览器验证通过)
|
||||||
|
|
||||||
- [ ] 报告抽屉呈现探索发现区块,数据与后端口径一致
|
- [x] 报告抽屉呈现探索发现区块,数据与后端口径一致
|
||||||
- [ ] 会话详情展示完整对话(角色、内容、延迟)与体验记录
|
- [x] 会话详情展示完整对话(角色、内容、延迟)与体验记录
|
||||||
- [ ] 无探索数据时区块隐藏
|
- [x] 无探索数据时区块隐藏
|
||||||
- [ ] tsc 通过;人工在浏览器验证展开/下钻交互
|
- [x] tsc 通过;浏览器验证展开/下钻交互(Playwright 在 t480 实测:区块渲染、问题清单、judge 复核、会话下钻气泡与体验卡片均正常,截图 /tmp/ticket06)
|
||||||
|
|||||||
@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
**Blocked by:** 01(会话实体与状态机)
|
**Blocked by:** 01(会话实体与状态机)
|
||||||
|
|
||||||
**Status:** ready-for-agent
|
**Status:** completed
|
||||||
|
|
||||||
- [ ] 活动终态结算时 running 会话转 expired,后续消息 409
|
- [x] 活动终态结算时 running 会话转 expired,后续消息 409
|
||||||
- [ ] 加速线自动触发被拒并附原因;手动触发可用
|
- [x] 加速线自动触发被拒并附原因;手动触发可用
|
||||||
- [ ] 正式线两种触发来源均可用
|
- [x] 正式线两种触发来源均可用
|
||||||
- [ ] 集成测试覆盖结算与档位分支(先例:活动自动触发类测试)
|
- [x] 集成测试覆盖结算与档位分支(先例:活动自动触发类测试)
|
||||||
|
|||||||
@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
**Blocked by:** 03(巡检 API 契约就绪才能写说明书)
|
**Blocked by:** 03(巡检 API 契约就绪才能写说明书)
|
||||||
|
|
||||||
**Status:** ready-for-agent
|
**Status:** completed(skill fc167a8;常驻作业 e6f19ac8 已注册并重启存活;E2E 活动 e1f3bdde 自动派发会话 3e46b6a3,体验记录落库,活动已收尾取消)
|
||||||
|
|
||||||
- [ ] skill 文档覆盖巡检决策、会话驱动、体验记录提交、409 收敛与衍生留痕
|
- [x] skill 文档覆盖巡检决策、会话驱动、体验记录提交、409 收敛与衍生留痕
|
||||||
- [ ] 部署同步管线分发 skill(复用现有同步步骤)
|
- [x] 部署同步管线分发 skill(复用现有同步步骤)
|
||||||
- [ ] 常驻巡检作业注册成功,重启后不丢(依赖 OpenClaw 调度持久化)
|
- [x] 常驻巡检作业注册成功,重启后不丢(依赖 OpenClaw 调度持久化)
|
||||||
- [ ] t480 端到端验证:正式线活动窗口内出现自动派发的探索会话并有体验记录落库
|
- [x] t480 端到端验证:正式线活动窗口内出现自动派发的探索会话并有体验记录落库
|
||||||
|
|||||||
@ -51,7 +51,10 @@
|
|||||||
- `POST /api/exploration/sessions`:body = campaign_id、persona、goal、seed_ref(可空);平台硬校验(活动存在且 running、正式线仅接受 auto 或 manual 触发;加速线仅接受 manual;预算余量)→ 创建 running 会话;超限 409 + 原因
|
- `POST /api/exploration/sessions`:body = campaign_id、persona、goal、seed_ref(可空);平台硬校验(活动存在且 running、正式线仅接受 auto 或 manual 触发;加速线仅接受 manual;预算余量)→ 创建 running 会话;超限 409 + 原因
|
||||||
- `POST /api/exploration/sessions/{id}/messages`:body = content;平台转发到目标的通道(复用 ChannelFactory)、持久化双方轮次、返回回复与延迟;单会话轮数超限 409;会话非 running 状态 409
|
- `POST /api/exploration/sessions/{id}/messages`:body = content;平台转发到目标的通道(复用 ChannelFactory)、持久化双方轮次、返回回复与延迟;单会话轮数超限 409;会话非 running 状态 409
|
||||||
- `POST /api/exploration/sessions/{id}/close`:body = 体验记录(goal_achieved: bool、blockers[]、misled[]、emotion、notes);结构校验 + 非法值归一(沿 v0.7 白名单经验);会话转 completed
|
- `POST /api/exploration/sessions/{id}/close`:body = 体验记录(goal_achieved: bool、blockers[]、misled[]、emotion、notes);结构校验 + 非法值归一(沿 v0.7 白名单经验);会话转 completed
|
||||||
- `GET /api/campaigns/{id}/exploration`:会话列表 + 探索发现聚合(会话数、达成率、问题清单)
|
- 探索发现读出口(实现裁决:不设独立 `/api/campaigns/{id}/exploration`,避免与报告链重复聚合):
|
||||||
|
- `GET /api/campaigns/{id}/report` 的 `exploration` 键携带探索发现聚合(会话数、达成率、问题清单、judge 复核),无探索数据时缺则无痕
|
||||||
|
- `GET /api/exploration/campaigns/{id}/sessions`:会话列表(下钻用)
|
||||||
|
- `GET /api/exploration/sessions/{id}/messages`:单会话对话记录(下钻用)
|
||||||
- 活动窗口 finalize 时:仍 running 的会话转 expired,不再接受消息(在 `resolve_finalize` 同处挂接)
|
- 活动窗口 finalize 时:仍 running 的会话转 expired,不再接受消息(在 `resolve_finalize` 同处挂接)
|
||||||
|
|
||||||
### 护栏(平台硬执行,不信任客户端自律)
|
### 护栏(平台硬执行,不信任客户端自律)
|
||||||
@ -62,11 +65,11 @@
|
|||||||
### 判定双证据线
|
### 判定双证据线
|
||||||
|
|
||||||
- 体验判定:close 接口收结构化自报,为第一手证据
|
- 体验判定:close 接口收结构化自报,为第一手证据
|
||||||
- judge 抽样复核:会话结束后平台对对话抽样(默认 ≤3 段,控 token),经 judge 岗位模型产出质量维度复核;ChatClient 可注入(沿 v0.7 分析 seam);异步后台执行,失败落错误不阻塞
|
- judge 抽样复核:会话结束后平台对对话抽样(实现裁决:按「轮」抽样,默认 ≤3 轮即最多 6 条消息、每条截断 500 字符控 token),经 judge 岗位模型产出质量维度复核;ChatClient 可注入(沿 v0.7 分析 seam);异步后台执行,失败落错误不阻塞
|
||||||
|
|
||||||
### 报告 / 分析 / 导出
|
### 报告 / 分析 / 导出
|
||||||
|
|
||||||
- 活动报告聚合新增"探索发现"维度(会话数、目标达成率、问题清单来自体验记录聚合);周期对比口径不变(探索数据不参与)
|
- 活动报告聚合新增"探索发现"维度(会话数、目标达成率、问题清单来自体验记录聚合);judge 复核结论一并纳入(实现裁决:findings 全量收各档发现、poor 档排前,而非只收 poor);周期对比口径不变(探索数据不参与)
|
||||||
- v0.7 分析输入追加探索**摘要**(问题清单 + 达成统计,非全量对话)
|
- v0.7 分析输入追加探索**摘要**(问题清单 + 达成统计,非全量对话)
|
||||||
- Markdown 导出:存在探索数据时追加「探索发现」附录(缺则无痕),附录顺序:智能分析 → 周期对比 → 探索发现
|
- Markdown 导出:存在探索数据时追加「探索发现」附录(缺则无痕),附录顺序:智能分析 → 周期对比 → 探索发现
|
||||||
|
|
||||||
|
|||||||
@ -14,6 +14,7 @@ from typing import Any, Awaitable, Callable, Optional
|
|||||||
from sqlmodel import Session
|
from sqlmodel import Session
|
||||||
|
|
||||||
from agenteval.evaluation.report import generate_campaign_report
|
from agenteval.evaluation.report import generate_campaign_report
|
||||||
|
from agenteval.exploration.summary import summarize_campaign_exploration
|
||||||
from agenteval.model_gateway import ModelGateway
|
from agenteval.model_gateway import ModelGateway
|
||||||
from agenteval.models import Campaign, ModelCapability, RunStatus
|
from agenteval.models import Campaign, ModelCapability, RunStatus
|
||||||
from agenteval.services.model_configs import (
|
from agenteval.services.model_configs import (
|
||||||
@ -152,6 +153,7 @@ async def _synthesize(
|
|||||||
report: dict[str, Any],
|
report: dict[str, Any],
|
||||||
stage1: list[dict[str, Any]],
|
stage1: list[dict[str, Any]],
|
||||||
chat_client: ChatClient,
|
chat_client: ChatClient,
|
||||||
|
exploration_summary: Optional[dict[str, Any]] = None,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""阶段二:汇总各场景产出,产总体结论 + 跨场景问题 + 优先级建议。"""
|
"""阶段二:汇总各场景产出,产总体结论 + 跨场景问题 + 优先级建议。"""
|
||||||
system_prompt = (
|
system_prompt = (
|
||||||
@ -163,17 +165,18 @@ async def _synthesize(
|
|||||||
'"suggestions": [{"priority": 1, "text": "可执行的改善建议"}]}'
|
'"suggestions": [{"priority": 1, "text": "可执行的改善建议"}]}'
|
||||||
";问题按严重度从高到低排列,建议按优先级排列。全部使用中文。"
|
";问题按严重度从高到低排列,建议按优先级排列。全部使用中文。"
|
||||||
)
|
)
|
||||||
user_prompt = json.dumps(
|
payload: dict[str, Any] = {
|
||||||
{
|
|
||||||
"活动": {
|
"活动": {
|
||||||
"名称": campaign.name,
|
"名称": campaign.name,
|
||||||
"窗口秒数": campaign.window_seconds,
|
"窗口秒数": campaign.window_seconds,
|
||||||
"总体指标": report.get("summary", {}),
|
"总体指标": report.get("summary", {}),
|
||||||
},
|
},
|
||||||
"各场景诊断": stage1,
|
"各场景诊断": stage1,
|
||||||
},
|
}
|
||||||
ensure_ascii=False,
|
if exploration_summary is not None:
|
||||||
)
|
# 探索式评测证据线:只给统计与问题清单,不含全量对话
|
||||||
|
payload["探索发现"] = exploration_summary
|
||||||
|
user_prompt = json.dumps(payload, ensure_ascii=False)
|
||||||
parsed = _parse_stage(
|
parsed = _parse_stage(
|
||||||
await chat_client([
|
await chat_client([
|
||||||
{"role": "system", "content": system_prompt},
|
{"role": "system", "content": system_prompt},
|
||||||
@ -194,6 +197,7 @@ async def analyze_campaign(
|
|||||||
failure_samples: dict[str, list[dict[str, str]]],
|
failure_samples: dict[str, list[dict[str, str]]],
|
||||||
valid_run_ids: set[str],
|
valid_run_ids: set[str],
|
||||||
chat_client: ChatClient,
|
chat_client: ChatClient,
|
||||||
|
exploration_summary: Optional[dict[str, Any]] = None,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""两阶段编排:阶段一按场景并行诊断,阶段二综合研判。
|
"""两阶段编排:阶段一按场景并行诊断,阶段二综合研判。
|
||||||
|
|
||||||
@ -209,7 +213,7 @@ async def analyze_campaign(
|
|||||||
_analyze_scenario(entry, failure_samples.get(entry["scenario_id"], []), chat_client)
|
_analyze_scenario(entry, failure_samples.get(entry["scenario_id"], []), chat_client)
|
||||||
for entry in capability
|
for entry in capability
|
||||||
])
|
])
|
||||||
stage2 = await _synthesize(campaign, report, list(stage1), chat_client)
|
stage2 = await _synthesize(campaign, report, list(stage1), chat_client, exploration_summary=exploration_summary)
|
||||||
|
|
||||||
valid_scenario_ids = {entry["scenario_id"] for entry in capability}
|
valid_scenario_ids = {entry["scenario_id"] for entry in capability}
|
||||||
problems = []
|
problems = []
|
||||||
@ -310,6 +314,7 @@ async def execute_campaign_analysis(
|
|||||||
failure_samples=collect_failure_samples(campaign_id, session),
|
failure_samples=collect_failure_samples(campaign_id, session),
|
||||||
valid_run_ids={r.id for r in runs if r.id},
|
valid_run_ids={r.id for r in runs if r.id},
|
||||||
chat_client=client,
|
chat_client=client,
|
||||||
|
exploration_summary=summarize_campaign_exploration(session, campaign_id),
|
||||||
)
|
)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
_logger.warning("活动 %s 智能分析失败: %s", campaign_id, exc)
|
_logger.warning("活动 %s 智能分析失败: %s", campaign_id, exc)
|
||||||
|
|||||||
@ -30,6 +30,7 @@ from agenteval.evaluation.campaign_scheduler import (
|
|||||||
resolve_finalize,
|
resolve_finalize,
|
||||||
)
|
)
|
||||||
from agenteval.evaluation.engine import EvalEngine
|
from agenteval.evaluation.engine import EvalEngine
|
||||||
|
from agenteval.exploration.settlement import settle_campaign_sessions
|
||||||
from agenteval.models import (
|
from agenteval.models import (
|
||||||
Campaign,
|
Campaign,
|
||||||
CampaignStatus,
|
CampaignStatus,
|
||||||
@ -201,6 +202,18 @@ def _auto_start_analysis(campaign: Campaign, session: Session) -> None:
|
|||||||
_logger.warning("活动 %s 自动分析触发失败(已跳过): %s", campaign.id, exc)
|
_logger.warning("活动 %s 自动分析触发失败(已跳过): %s", campaign.id, exc)
|
||||||
|
|
||||||
|
|
||||||
|
def _settle_exploration(campaign_id: str, session: Session) -> None:
|
||||||
|
"""Expire dangling exploration sessions on completion.
|
||||||
|
|
||||||
|
Never blocks completion: a settlement failure is logged and skipped, the
|
||||||
|
same non-blocking semantics as ``_auto_start_analysis``.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
settle_campaign_sessions(campaign_id, session)
|
||||||
|
except Exception as exc:
|
||||||
|
_logger.warning("活动 %s 探索会话结算失败(已跳过): %s", campaign_id, exc)
|
||||||
|
|
||||||
|
|
||||||
# ── durable scheduler loop ──────────────────────────────────────────────────
|
# ── durable scheduler loop ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
@ -250,6 +263,7 @@ async def run_campaign_loop(
|
|||||||
current.status = CampaignStatus.COMPLETED
|
current.status = CampaignStatus.COMPLETED
|
||||||
current.completed_at = utc_now()
|
current.completed_at = utc_now()
|
||||||
repo.update(current)
|
repo.update(current)
|
||||||
|
_settle_exploration(campaign_id, session)
|
||||||
_auto_start_analysis(current, session)
|
_auto_start_analysis(current, session)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|||||||
@ -347,19 +347,63 @@ def _window_line(report: dict[str, Any]) -> str:
|
|||||||
return f"**窗口**: {_offset(window)}(加速调试线 ×{scale:g}{wall})"
|
return f"**窗口**: {_offset(window)}(加速调试线 ×{scale:g}{wall})"
|
||||||
|
|
||||||
|
|
||||||
|
def _render_exploration_lines(exploration: dict[str, Any]) -> list[str]:
|
||||||
|
"""探索发现附录:会话统计 + 问题清单 + judge 复核发现(有才渲染)。"""
|
||||||
|
lines: list[str] = [
|
||||||
|
"",
|
||||||
|
"## 探索发现",
|
||||||
|
"",
|
||||||
|
"| 指标 | 数值 |",
|
||||||
|
"|------|------|",
|
||||||
|
f"| 探索会话数 | {exploration.get('session_count', 0)} |",
|
||||||
|
f"| 有体验记录会话数 | {exploration.get('sessions_with_experience', 0)} |",
|
||||||
|
f"| 目标达成率 | {_pct(exploration.get('goal_achievement_rate'))} |",
|
||||||
|
"",
|
||||||
|
"### 问题清单",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
issues = exploration.get("issues") or []
|
||||||
|
misled = exploration.get("misled") or []
|
||||||
|
if not issues and not misled:
|
||||||
|
lines.append("无")
|
||||||
|
else:
|
||||||
|
for item in issues:
|
||||||
|
lines.append(f"- {item['issue']} ×{item['count']}")
|
||||||
|
for item in misled:
|
||||||
|
lines.append(f"- (被误导){item['issue']} ×{item['count']}")
|
||||||
|
judge = exploration.get("judge_review")
|
||||||
|
if judge:
|
||||||
|
lines += [
|
||||||
|
"",
|
||||||
|
"### judge 复核",
|
||||||
|
"",
|
||||||
|
f"已复核 {judge.get('reviewed_sessions', 0)} 个会话:",
|
||||||
|
]
|
||||||
|
findings = judge.get("findings") or []
|
||||||
|
if findings:
|
||||||
|
for item in findings:
|
||||||
|
lines.append(f"- [{item.get('dimension')}/{item.get('rating')}] {item.get('comment')}")
|
||||||
|
else:
|
||||||
|
lines.append("- 未发现问题")
|
||||||
|
for summary_text in judge.get("summaries") or []:
|
||||||
|
lines.append(f"- 复核结论:{summary_text}")
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
def render_campaign_markdown(
|
def render_campaign_markdown(
|
||||||
report: dict[str, Any],
|
report: dict[str, Any],
|
||||||
*,
|
*,
|
||||||
analysis: Optional[dict[str, Any]] = None,
|
analysis: Optional[dict[str, Any]] = None,
|
||||||
comparison: Optional[dict[str, Any]] = None,
|
comparison: Optional[dict[str, Any]] = None,
|
||||||
|
exploration: Optional[dict[str, Any]] = None,
|
||||||
target_name: Optional[str] = None,
|
target_name: Optional[str] = None,
|
||||||
scenario_names: Optional[dict[str, str]] = None,
|
scenario_names: Optional[dict[str, str]] = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Render a dual-axis campaign report dict as Markdown.
|
"""Render a dual-axis campaign report dict as Markdown.
|
||||||
|
|
||||||
``analysis`` and ``comparison`` are the stored 智能分析 / 周期对比 results
|
``analysis`` and ``comparison`` are the stored 智能分析 / 周期对比 results
|
||||||
(completed only); when absent the corresponding appendix is omitted
|
(completed only); ``exploration`` is the on-the-fly 探索发现 aggregate;
|
||||||
entirely.
|
when absent the corresponding appendix is omitted entirely (缺则无痕).
|
||||||
"""
|
"""
|
||||||
s = report["summary"]
|
s = report["summary"]
|
||||||
status = _CAMPAIGN_STATUS_LABELS.get(report.get("status"), report.get("status") or "-")
|
status = _CAMPAIGN_STATUS_LABELS.get(report.get("status"), report.get("status") or "-")
|
||||||
@ -409,4 +453,6 @@ def render_campaign_markdown(
|
|||||||
lines += _render_analysis_lines(analysis, scenario_names or {})
|
lines += _render_analysis_lines(analysis, scenario_names or {})
|
||||||
if comparison:
|
if comparison:
|
||||||
lines += _render_comparison_lines(comparison, scenario_names or {})
|
lines += _render_comparison_lines(comparison, scenario_names or {})
|
||||||
|
if exploration:
|
||||||
|
lines += _render_exploration_lines(exploration)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
192
backend/agenteval/exploration/judge.py
Normal file
192
backend/agenteval/exploration/judge.py
Normal file
@ -0,0 +1,192 @@
|
|||||||
|
"""Judge sampling review for exploration sessions (judge 岗位抽样复核).
|
||||||
|
|
||||||
|
会话关闭后,平台对该会话对话抽样(默认 ≤3 段,控 token),经 judge 岗位模型
|
||||||
|
独立复核,产出质量维度结论(态度、专业性、幻觉)落入会话的 ``judge_review``。
|
||||||
|
复核是异步后台执行:失败落错误不阻塞会话状态,也不影响体验记录这条第一手
|
||||||
|
证据线;未配置模型时静默跳过。LLM 调用沿 v0.7 分析的 ``ChatClient`` 接缝
|
||||||
|
注入,测试用假客户端覆盖。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
from sqlmodel import Session
|
||||||
|
|
||||||
|
from agenteval.evaluation.analysis import (
|
||||||
|
ChatClient,
|
||||||
|
gateway_chat_client,
|
||||||
|
resolve_analysis_model,
|
||||||
|
)
|
||||||
|
from agenteval.exploration.models import ExplorationMessage, ExplorationSession
|
||||||
|
from agenteval.models import Campaign
|
||||||
|
from agenteval.services.model_configs import ModelRuntimeConfig
|
||||||
|
from agenteval.storage.db import get_session, iso_utc, utc_now
|
||||||
|
from agenteval.storage.repository import (
|
||||||
|
CampaignRepository,
|
||||||
|
ExplorationMessageRepository,
|
||||||
|
ExplorationSessionRepository,
|
||||||
|
)
|
||||||
|
from agenteval.utils.llm import parse_json_from_llm_text
|
||||||
|
|
||||||
|
_logger = logging.getLogger("agenteval")
|
||||||
|
|
||||||
|
MAX_JUDGE_SAMPLES = 3
|
||||||
|
SAMPLE_TEXT_LIMIT = 500
|
||||||
|
VALID_DIMENSIONS = ("attitude", "professionalism", "hallucination")
|
||||||
|
VALID_RATINGS = ("good", "acceptable", "poor")
|
||||||
|
|
||||||
|
JUDGE_SYSTEM_PROMPT = """你是评测平台的 judge 岗位模型,负责独立复核一段「虚拟用户」与被评对象的探索对话。
|
||||||
|
只依据给定对话抽样判断,不要臆测抽样之外的内容。
|
||||||
|
输出必须是合法 JSON,且仅包含以下结构:
|
||||||
|
{
|
||||||
|
"dimensions": [
|
||||||
|
{
|
||||||
|
"dimension": "attitude | professionalism | hallucination",
|
||||||
|
"rating": "good | acceptable | poor",
|
||||||
|
"comment": "一句话依据"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"summary": "一句话总体结论"
|
||||||
|
}
|
||||||
|
维度说明:attitude=服务态度;professionalism=专业性(流程与答复正确性);
|
||||||
|
hallucination=幻觉(编造事实、政策或能力),无幻觉时 rating 为 good。
|
||||||
|
三个维度必须各出现一次。"""
|
||||||
|
|
||||||
|
|
||||||
|
class JudgeReviewError(RuntimeError):
|
||||||
|
"""judge 复核失败(模型输出无法解析或调用失败),不影响会话状态。"""
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_judge_model(campaign: Campaign, session: Session) -> Optional[ModelRuntimeConfig]:
|
||||||
|
"""解析 judge 岗位模型:沿用活动分析模型口径(活动覆盖 ?? 全局分析默认)。"""
|
||||||
|
return resolve_analysis_model(campaign, session)
|
||||||
|
|
||||||
|
|
||||||
|
def sample_round_indexes(rounds: list[int], limit: int = MAX_JUDGE_SAMPLES) -> list[int]:
|
||||||
|
"""超限时均匀取样(含首尾),保证抽样可复现。"""
|
||||||
|
if len(rounds) <= limit:
|
||||||
|
return list(rounds)
|
||||||
|
span = len(rounds) - 1
|
||||||
|
return [rounds[round(i * span / (limit - 1))] for i in range(limit)]
|
||||||
|
|
||||||
|
|
||||||
|
def build_judge_messages(
|
||||||
|
session_obj: ExplorationSession, samples: list[ExplorationMessage]
|
||||||
|
) -> list[dict[str, str]]:
|
||||||
|
transcript = []
|
||||||
|
for message in samples:
|
||||||
|
speaker = "虚拟用户" if message.role == "user" else "被评对象"
|
||||||
|
content = str(message.content)[:SAMPLE_TEXT_LIMIT]
|
||||||
|
transcript.append(f"[第 {message.round_index} 轮] {speaker}: {content}")
|
||||||
|
payload = {
|
||||||
|
"persona": session_obj.persona,
|
||||||
|
"goal": session_obj.goal,
|
||||||
|
"transcript": transcript,
|
||||||
|
}
|
||||||
|
return [
|
||||||
|
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
|
||||||
|
{"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_judge_review(raw: Any) -> dict[str, Any]:
|
||||||
|
"""白名单归一:未知维度丢弃,非法档位归 acceptable(沿 v0.7 白名单经验)。"""
|
||||||
|
dims = raw.get("dimensions") if isinstance(raw, dict) else None
|
||||||
|
dims = dims if isinstance(dims, list) else []
|
||||||
|
normalized = []
|
||||||
|
for item in dims:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
name = item.get("dimension")
|
||||||
|
if name not in VALID_DIMENSIONS:
|
||||||
|
continue
|
||||||
|
rating = item.get("rating")
|
||||||
|
normalized.append(
|
||||||
|
{
|
||||||
|
"dimension": name,
|
||||||
|
"rating": rating if rating in VALID_RATINGS else "acceptable",
|
||||||
|
"comment": str(item.get("comment") or "")[:SAMPLE_TEXT_LIMIT],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
summary = str(raw.get("summary") or "")[:SAMPLE_TEXT_LIMIT] if isinstance(raw, dict) else ""
|
||||||
|
return {"dimensions": normalized, "summary": summary}
|
||||||
|
|
||||||
|
|
||||||
|
async def judge_conversation(
|
||||||
|
session_obj: ExplorationSession,
|
||||||
|
samples: list[ExplorationMessage],
|
||||||
|
*,
|
||||||
|
chat_client: ChatClient,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
text = await chat_client(build_judge_messages(session_obj, samples))
|
||||||
|
try:
|
||||||
|
raw = parse_json_from_llm_text(text)
|
||||||
|
except Exception as exc:
|
||||||
|
raise JudgeReviewError(f"judge 输出解析失败: {exc}") from exc
|
||||||
|
if not isinstance(raw, dict):
|
||||||
|
raise JudgeReviewError("judge 输出解析失败: 输出不是 JSON 对象")
|
||||||
|
review = normalize_judge_review(raw)
|
||||||
|
review["sampled_rounds"] = sorted({m.round_index for m in samples})
|
||||||
|
return review
|
||||||
|
|
||||||
|
|
||||||
|
async def execute_judge_review(
|
||||||
|
exploration_session_id: str,
|
||||||
|
*,
|
||||||
|
chat_client: Optional[ChatClient] = None,
|
||||||
|
) -> None:
|
||||||
|
"""后台执行体:抽样 → judge 复核 → judge_review 落库。
|
||||||
|
|
||||||
|
与 Runs 同款后台任务约定:自持 Session、try/finally 关闭、失败落 error
|
||||||
|
不阻塞会话状态;未配置模型时静默跳过。
|
||||||
|
"""
|
||||||
|
session = get_session()
|
||||||
|
try:
|
||||||
|
repo = ExplorationSessionRepository(session)
|
||||||
|
session_obj = repo.get(exploration_session_id)
|
||||||
|
if not session_obj:
|
||||||
|
return
|
||||||
|
campaign = CampaignRepository(session).get(session_obj.campaign_id)
|
||||||
|
runtime = resolve_judge_model(campaign, session) if campaign else None
|
||||||
|
if runtime is None:
|
||||||
|
_logger.info("探索会话 %s 跳过 judge 复核:未配置模型", exploration_session_id)
|
||||||
|
return
|
||||||
|
|
||||||
|
messages = ExplorationMessageRepository(session).list_by_session(session_obj.id)
|
||||||
|
round_indexes = sorted({m.round_index for m in messages})
|
||||||
|
sampled = set(sample_round_indexes(round_indexes))
|
||||||
|
samples = [m for m in messages if m.round_index in sampled]
|
||||||
|
if not samples:
|
||||||
|
return
|
||||||
|
|
||||||
|
client = chat_client or gateway_chat_client(runtime)
|
||||||
|
reviewed_at = iso_utc(utc_now())
|
||||||
|
try:
|
||||||
|
review = await judge_conversation(session_obj, samples, chat_client=client)
|
||||||
|
review.update(status="completed", model_config_id=runtime.id, reviewed_at=reviewed_at)
|
||||||
|
except Exception as exc:
|
||||||
|
_logger.warning("探索会话 %s judge 复核失败: %s", exploration_session_id, exc)
|
||||||
|
review = {
|
||||||
|
"status": "failed",
|
||||||
|
"model_config_id": runtime.id,
|
||||||
|
"reviewed_at": reviewed_at,
|
||||||
|
"error": str(exc)[:SAMPLE_TEXT_LIMIT],
|
||||||
|
}
|
||||||
|
session_obj.judge_review = review
|
||||||
|
repo.update(session_obj)
|
||||||
|
finally:
|
||||||
|
session.close()
|
||||||
|
|
||||||
|
|
||||||
|
_BACKGROUND_TASKS: set[asyncio.Task] = set()
|
||||||
|
|
||||||
|
|
||||||
|
def start_judge_review(exploration_session_id: str) -> asyncio.Task:
|
||||||
|
"""以后台任务启动抽样复核(fire-and-forget;结果经会话 judge_review 观测)。"""
|
||||||
|
task = asyncio.create_task(execute_judge_review(exploration_session_id))
|
||||||
|
# 强引用防止未持有引用的任务被 GC(asyncio 已知坑)
|
||||||
|
_BACKGROUND_TASKS.add(task)
|
||||||
|
task.add_done_callback(_BACKGROUND_TASKS.discard)
|
||||||
|
return task
|
||||||
29
backend/agenteval/exploration/settlement.py
Normal file
29
backend/agenteval/exploration/settlement.py
Normal file
@ -0,0 +1,29 @@
|
|||||||
|
"""Exploration session settlement on campaign finalize (活动终态结算).
|
||||||
|
|
||||||
|
活动窗口结束或被取消进入终态时,仍 running 的探索会话转 expired——
|
||||||
|
不留悬挂会话。结算后消息端点按既有状态机拒收(409)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from sqlmodel import Session
|
||||||
|
|
||||||
|
from agenteval.exploration.models import ExplorationSessionStatus
|
||||||
|
from agenteval.storage.db import utc_now
|
||||||
|
from agenteval.storage.repository import ExplorationSessionRepository
|
||||||
|
|
||||||
|
|
||||||
|
def settle_campaign_sessions(campaign_id: str, session: Session) -> int:
|
||||||
|
"""Expire every still-running exploration session of a finalized campaign.
|
||||||
|
|
||||||
|
Returns the number of sessions expired. Completed/failed sessions keep
|
||||||
|
their evidence untouched.
|
||||||
|
"""
|
||||||
|
repo = ExplorationSessionRepository(session)
|
||||||
|
expired = 0
|
||||||
|
for session_obj in repo.list_by_campaign(campaign_id):
|
||||||
|
if session_obj.status != ExplorationSessionStatus.RUNNING:
|
||||||
|
continue
|
||||||
|
session_obj.status = ExplorationSessionStatus.EXPIRED
|
||||||
|
session_obj.closed_at = utc_now()
|
||||||
|
repo.update(session_obj)
|
||||||
|
expired += 1
|
||||||
|
return expired
|
||||||
84
backend/agenteval/exploration/summary.py
Normal file
84
backend/agenteval/exploration/summary.py
Normal file
@ -0,0 +1,84 @@
|
|||||||
|
"""Exploration findings aggregation (探索发现聚合).
|
||||||
|
|
||||||
|
把探索会话的体验记录聚合成探索摘要:会话数、目标达成率、问题清单
|
||||||
|
(blockers / misled 按出现次数降序)、judge 复核结论(已就位才纳入;
|
||||||
|
findings 全量收各档发现,poor 档排前)。这是体验记录这条第一手证据线
|
||||||
|
进入报告 / 分析 / 导出三个出口前的唯一聚合口径;只含统计与问题清单,
|
||||||
|
不含全量对话。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections import Counter
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
from sqlmodel import Session
|
||||||
|
|
||||||
|
from agenteval.exploration.models import ExplorationSession
|
||||||
|
|
||||||
|
_RATING_SEVERITY = {"poor": 0, "acceptable": 1, "good": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def _issue_list(counter: Counter) -> list[dict[str, Any]]:
|
||||||
|
return [{"issue": issue, "count": count} for issue, count in counter.most_common()]
|
||||||
|
|
||||||
|
|
||||||
|
def summarize_exploration(sessions: list[ExplorationSession]) -> Optional[dict[str, Any]]:
|
||||||
|
"""聚合一组探索会话为探索摘要;无会话返回 None(缺则无痕)。"""
|
||||||
|
if not sessions:
|
||||||
|
return None
|
||||||
|
|
||||||
|
with_experience = [s for s in sessions if s.experience is not None]
|
||||||
|
blockers: Counter = Counter()
|
||||||
|
misled: Counter = Counter()
|
||||||
|
achieved = 0
|
||||||
|
for session_obj in with_experience:
|
||||||
|
experience = session_obj.experience
|
||||||
|
if experience.get("goal_achieved"):
|
||||||
|
achieved += 1
|
||||||
|
blockers.update(experience.get("blockers") or [])
|
||||||
|
misled.update(experience.get("misled") or [])
|
||||||
|
|
||||||
|
judge = _summarize_judge_reviews(sessions)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"session_count": len(sessions),
|
||||||
|
"sessions_with_experience": len(with_experience),
|
||||||
|
"goal_achieved_count": achieved,
|
||||||
|
"goal_achievement_rate": round(achieved / len(with_experience), 4) if with_experience else None,
|
||||||
|
"issues": _issue_list(blockers),
|
||||||
|
"misled": _issue_list(misled),
|
||||||
|
"judge_review": judge,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def summarize_campaign_exploration(db_session: Session, campaign_id: str) -> Optional[dict[str, Any]]:
|
||||||
|
"""取数 + 聚合一步完成:报告 / 分析 / 导出三个出口共用的探索摘要取法。"""
|
||||||
|
from agenteval.storage.repository import ExplorationSessionRepository
|
||||||
|
|
||||||
|
return summarize_exploration(ExplorationSessionRepository(db_session).list_by_campaign(campaign_id))
|
||||||
|
|
||||||
|
|
||||||
|
def _summarize_judge_reviews(sessions: list[ExplorationSession]) -> Optional[dict[str, Any]]:
|
||||||
|
"""只纳入复核完成的会话;findings 全量收各档发现(poor 档排前),
|
||||||
|
summaries 收复核总体结论。"""
|
||||||
|
findings: list[dict[str, Any]] = []
|
||||||
|
summaries: list[str] = []
|
||||||
|
reviewed = 0
|
||||||
|
for session_obj in sessions:
|
||||||
|
review = session_obj.judge_review
|
||||||
|
if not review or review.get("status") != "completed":
|
||||||
|
continue
|
||||||
|
reviewed += 1
|
||||||
|
if review.get("summary"):
|
||||||
|
summaries.append(str(review["summary"]))
|
||||||
|
for item in review.get("dimensions") or []:
|
||||||
|
findings.append(
|
||||||
|
{
|
||||||
|
"dimension": item.get("dimension"),
|
||||||
|
"rating": item.get("rating"),
|
||||||
|
"comment": item.get("comment") or "",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if reviewed == 0:
|
||||||
|
return None
|
||||||
|
findings.sort(key=lambda f: _RATING_SEVERITY.get(f["rating"], len(_RATING_SEVERITY)))
|
||||||
|
return {"reviewed_sessions": reviewed, "findings": findings, "summaries": summaries}
|
||||||
@ -26,6 +26,8 @@ from agenteval.evaluation.report import (
|
|||||||
summarize_campaign_progress,
|
summarize_campaign_progress,
|
||||||
)
|
)
|
||||||
from agenteval.evaluation.report_render import render_campaign_markdown
|
from agenteval.evaluation.report_render import render_campaign_markdown
|
||||||
|
from agenteval.exploration.settlement import settle_campaign_sessions
|
||||||
|
from agenteval.exploration.summary import summarize_campaign_exploration
|
||||||
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, ExplorationBudgetConfig, ExplorationSeeds
|
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, ExplorationBudgetConfig, ExplorationSeeds
|
||||||
from agenteval.storage.db import iso_utc, utc_now
|
from agenteval.storage.db import iso_utc, utc_now
|
||||||
from agenteval.storage.model_config_repository import ModelConfigRepository
|
from agenteval.storage.model_config_repository import ModelConfigRepository
|
||||||
@ -120,6 +122,7 @@ async def cancel_campaign(campaign_id: str, session: Session = Depends(get_db))
|
|||||||
campaign.completed_at = utc_now()
|
campaign.completed_at = utc_now()
|
||||||
repo.update(campaign)
|
repo.update(campaign)
|
||||||
request_cancel(campaign_id)
|
request_cancel(campaign_id)
|
||||||
|
settle_campaign_sessions(campaign_id, session)
|
||||||
return campaign.model_dump()
|
return campaign.model_dump()
|
||||||
|
|
||||||
|
|
||||||
@ -130,7 +133,11 @@ async def get_campaign_report(campaign_id: str, session: Session = Depends(get_d
|
|||||||
raise HTTPException(status_code=404, detail="campaign not found")
|
raise HTTPException(status_code=404, detail="campaign not found")
|
||||||
runs = RunRepository(session).list_by_campaign(campaign_id)
|
runs = RunRepository(session).list_by_campaign(campaign_id)
|
||||||
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
|
scenario_names = {s.id: s.name for s in ScenarioRepository(session).list_all()}
|
||||||
return generate_campaign_report(campaign, runs, scenario_names=scenario_names)
|
report = generate_campaign_report(campaign, runs, scenario_names=scenario_names)
|
||||||
|
exploration = summarize_campaign_exploration(session, campaign_id)
|
||||||
|
if exploration is not None:
|
||||||
|
report["exploration"] = exploration
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{campaign_id}/report/markdown")
|
@router.get("/{campaign_id}/report/markdown")
|
||||||
@ -174,6 +181,7 @@ async def get_campaign_report_markdown(campaign_id: str, session: Session = Depe
|
|||||||
generate_campaign_report(campaign, runs, scenario_names=scenario_names),
|
generate_campaign_report(campaign, runs, scenario_names=scenario_names),
|
||||||
analysis=analysis,
|
analysis=analysis,
|
||||||
comparison=comparison,
|
comparison=comparison,
|
||||||
|
exploration=summarize_campaign_exploration(session, campaign_id),
|
||||||
target_name=target_name,
|
target_name=target_name,
|
||||||
scenario_names=scenario_names,
|
scenario_names=scenario_names,
|
||||||
)
|
)
|
||||||
|
|||||||
@ -10,6 +10,7 @@ with 409 plus a readable reason, so the rejection itself is feedback to the
|
|||||||
resident agent.
|
resident agent.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@ -20,6 +21,7 @@ from sqlmodel import Session
|
|||||||
from agenteval.channels.factory import ChannelFactory
|
from agenteval.channels.factory import ChannelFactory
|
||||||
from agenteval.config import get_settings
|
from agenteval.config import get_settings
|
||||||
from agenteval.evaluation.report import generate_campaign_report
|
from agenteval.evaluation.report import generate_campaign_report
|
||||||
|
from agenteval.exploration.judge import start_judge_review
|
||||||
from agenteval.exploration.models import (
|
from agenteval.exploration.models import (
|
||||||
ExplorationBudget,
|
ExplorationBudget,
|
||||||
ExplorationMessage,
|
ExplorationMessage,
|
||||||
@ -60,6 +62,21 @@ class CloseSessionRequest(BaseModel):
|
|||||||
experience: dict[str, Any]
|
experience: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce_reply_text(content: Any) -> str:
|
||||||
|
"""Flatten a reply payload to text; tutu returns msgBody as a parsed object,
|
||||||
|
and str(dict) would leak a Python repr into the view."""
|
||||||
|
if isinstance(content, str):
|
||||||
|
return content
|
||||||
|
if isinstance(content, dict):
|
||||||
|
for key in ("content", "text", "message"):
|
||||||
|
value = content.get(key)
|
||||||
|
if isinstance(value, str) and value:
|
||||||
|
return value
|
||||||
|
if content is None:
|
||||||
|
return ""
|
||||||
|
return json.dumps(content, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
def _check_creation_guardrails(
|
def _check_creation_guardrails(
|
||||||
campaign: Campaign,
|
campaign: Campaign,
|
||||||
triggered_by: ExplorationTrigger,
|
triggered_by: ExplorationTrigger,
|
||||||
@ -200,6 +217,28 @@ async def create_session(
|
|||||||
return repo.create(session_obj).model_dump(mode="json")
|
return repo.create(session_obj).model_dump(mode="json")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/campaigns/{campaign_id}/sessions")
|
||||||
|
async def list_campaign_sessions(
|
||||||
|
campaign_id: str,
|
||||||
|
session: Session = Depends(get_db),
|
||||||
|
) -> dict:
|
||||||
|
if not CampaignRepository(session).get(campaign_id):
|
||||||
|
raise HTTPException(status_code=404, detail="campaign not found")
|
||||||
|
sessions = ExplorationSessionRepository(session).list_by_campaign(campaign_id)
|
||||||
|
return {"sessions": [s.model_dump(mode="json") for s in sessions]}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/sessions/{session_id}/messages")
|
||||||
|
async def list_session_messages(
|
||||||
|
session_id: str,
|
||||||
|
session: Session = Depends(get_db),
|
||||||
|
) -> dict:
|
||||||
|
if not ExplorationSessionRepository(session).get(session_id):
|
||||||
|
raise HTTPException(status_code=404, detail="exploration session not found")
|
||||||
|
messages = ExplorationMessageRepository(session).list_by_session(session_id)
|
||||||
|
return {"messages": [m.model_dump(mode="json") for m in messages]}
|
||||||
|
|
||||||
|
|
||||||
@router.post("/sessions/{session_id}/messages")
|
@router.post("/sessions/{session_id}/messages")
|
||||||
async def send_session_message(
|
async def send_session_message(
|
||||||
session_id: str,
|
session_id: str,
|
||||||
@ -260,7 +299,7 @@ async def send_session_message(
|
|||||||
|
|
||||||
received_at = utc_now()
|
received_at = utc_now()
|
||||||
latency_ms = int((received_at - sent_at).total_seconds() * 1000)
|
latency_ms = int((received_at - sent_at).total_seconds() * 1000)
|
||||||
reply_text = str(reply.content)
|
reply_text = _coerce_reply_text(reply.content)
|
||||||
message_repo.save_message(
|
message_repo.save_message(
|
||||||
ExplorationMessage(
|
ExplorationMessage(
|
||||||
session_id=session_obj.id,
|
session_id=session_obj.id,
|
||||||
@ -290,4 +329,6 @@ async def close_session(
|
|||||||
session_obj.experience = normalize_experience(request.experience)
|
session_obj.experience = normalize_experience(request.experience)
|
||||||
session_obj.status = ExplorationSessionStatus.COMPLETED
|
session_obj.status = ExplorationSessionStatus.COMPLETED
|
||||||
session_obj.closed_at = utc_now()
|
session_obj.closed_at = utc_now()
|
||||||
return repo.update(session_obj).model_dump(mode="json")
|
updated = repo.update(session_obj)
|
||||||
|
start_judge_review(session_obj.id)
|
||||||
|
return updated.model_dump(mode="json")
|
||||||
|
|||||||
@ -32,7 +32,17 @@
|
|||||||
部署脚本(`scripts/deploy-t480.sh`)会自动把它同步到 OpenClaw 工作区:
|
部署脚本(`scripts/deploy-t480.sh`)会自动把它同步到 OpenClaw 工作区:
|
||||||
`data/openclaw/workspace/skills/agenteval-run/SKILL.md`。
|
`data/openclaw/workspace/skills/agenteval-run/SKILL.md`。
|
||||||
|
|
||||||
### 方式二:Python Skill 封装
|
### 方式二:agenteval-patrol 巡检技能(探索式评测,v0.9)
|
||||||
|
|
||||||
|
`skills/agenteval-patrol/SKILL.md` 指导 AI 助手作为常驻巡检代理完成探索闭环:
|
||||||
|
调巡检 API(`GET /api/exploration/patrol`)→ 研判新结果与预算余量 → 决定是否派发
|
||||||
|
探索会话 → 以虚拟用户身份驱动会话对话 → 提交体验记录关闭会话。说明书内含收到
|
||||||
|
409 时的收敛行为与衍生变体的种子回溯(`seed_ref`)要求。
|
||||||
|
|
||||||
|
部署脚本会把 `backend/plugins/openclaw/skills/` 下的每个技能同步到
|
||||||
|
`data/openclaw/workspace/skills/<skill-name>/SKILL.md`。
|
||||||
|
|
||||||
|
### 方式三:Python Skill 封装
|
||||||
|
|
||||||
参考 `agenteval_skill.py`(httpx 调 API),在 OpenClaw 中注册技能时传入配置:
|
参考 `agenteval_skill.py`(httpx 调 API),在 OpenClaw 中注册技能时传入配置:
|
||||||
|
|
||||||
@ -60,6 +70,21 @@ config:
|
|||||||
report_format: "json"
|
report_format: "json"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 常驻巡检作业(探索式评测)
|
||||||
|
|
||||||
|
巡检代理需注册为全局常驻作业,默认节拍 1 小时;调度状态由 OpenClaw 自身持久化,
|
||||||
|
重启后不丢:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
skill: agenteval-patrol
|
||||||
|
schedule: "0 * * * *" # 每小时整点巡检一次
|
||||||
|
config:
|
||||||
|
api_base_url: "http://agenteval:8000"
|
||||||
|
```
|
||||||
|
|
||||||
|
巡检 API 自带水位(`last_patrolled_at`),每个节拍只报增量结果,重复触发不会
|
||||||
|
重复上报;预算与间隔护栏在服务端强制执行,代理收到 409 即收敛。
|
||||||
|
|
||||||
## API 速查
|
## API 速查
|
||||||
|
|
||||||
| 操作 | 请求 |
|
| 操作 | 请求 |
|
||||||
@ -67,6 +92,10 @@ config:
|
|||||||
| 启动评测 | `POST /api/runs` body: `{"target_id", "scenario_id", "triggered_by": "ai_assistant"}` |
|
| 启动评测 | `POST /api/runs` body: `{"target_id", "scenario_id", "triggered_by": "ai_assistant"}` |
|
||||||
| 查询状态 | `GET /api/runs/{run_id}`(轮询至 completed/failed) |
|
| 查询状态 | `GET /api/runs/{run_id}`(轮询至 completed/failed) |
|
||||||
| 获取报告 | `GET /api/reports/{run_id}` |
|
| 获取报告 | `GET /api/reports/{run_id}` |
|
||||||
|
| 巡检探索 | `GET /api/exploration/patrol`(无参数,报增量新结果与预算余量) |
|
||||||
|
| 派发探索会话 | `POST /api/exploration/sessions` body: `{"campaign_id", "persona", "goal", "seed_ref?", "triggered_by"}` |
|
||||||
|
| 驱动会话对话 | `POST /api/exploration/sessions/{session_id}/messages` body: `{"content"}` |
|
||||||
|
| 提交体验记录 | `POST /api/exploration/sessions/{session_id}/close` body: `{"experience"}` |
|
||||||
|
|
||||||
如果平台设置了 `AGENTEVAL_API_KEY`,所有请求需带 `X-API-Key` 头。
|
如果平台设置了 `AGENTEVAL_API_KEY`,所有请求需带 `X-API-Key` 头。
|
||||||
|
|
||||||
|
|||||||
110
backend/plugins/openclaw/skills/agenteval-patrol/SKILL.md
Normal file
110
backend/plugins/openclaw/skills/agenteval-patrol/SKILL.md
Normal file
@ -0,0 +1,110 @@
|
|||||||
|
---
|
||||||
|
name: agenteval-patrol
|
||||||
|
description: 常驻巡检:巡检 AgentEvalTool 评测活动,研判新结果与预算余量,自主派发并驱动探索式评测会话
|
||||||
|
---
|
||||||
|
|
||||||
|
你是平台的常驻巡检代理。本 skill 指导你按「巡检 → 研判 → 派发 → 驱动 → 提交」的闭环行动。
|
||||||
|
所有操作必须走 AgentEvalTool 标准 HTTP API(禁止直接调 CLI 或操作数据库)。
|
||||||
|
|
||||||
|
平台可能启用了 API Key 鉴权。每次执行命令前先读取密钥(文件不存在则为空,不影响未启用鉴权的环境):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
KEY=$(cat ~/.openclaw/agenteval-api-key 2>/dev/null)
|
||||||
|
```
|
||||||
|
|
||||||
|
以下所有 curl 命令都必须带 `-H "X-API-Key: $KEY"`。
|
||||||
|
|
||||||
|
## 巡检(闭环第一步)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/exploration/patrol | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
返回所有参与探索的进行中正式线活动。每个活动包含:
|
||||||
|
|
||||||
|
- `new_results`:自上次巡检以来的新评测结果(null 表示无新增)。有值时先消化它——通过率下滑的能力项是本次探索的优先方向。
|
||||||
|
- `budget`:探索预算台账。`remaining_sessions` 为剩余可派发会话数;`seconds_since_last_session` 与 `min_interval_seconds` 用于判断会话间隔是否已满足。
|
||||||
|
- 该接口自带水位:每次调用只报增量,无需自己记录上次结果。
|
||||||
|
|
||||||
|
## 研判与派发决策
|
||||||
|
|
||||||
|
对每个活动,按以下规则决定是否派发探索会话:
|
||||||
|
|
||||||
|
1. `remaining_sessions` 为 0 → 不派发(预算耗尽)。
|
||||||
|
2. `seconds_since_last_session` 非 null 且小于 `min_interval_seconds` → 不派发(间隔未到,等下个节拍)。
|
||||||
|
3. 其余情况可派发一个会话。persona 与 goal 必须取自活动的种子集:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -H "X-API-Key: $KEY" http://agenteval:8000/api/campaigns/<campaign_id> | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
读取响应中的 `exploration_seeds.personas` 与 `exploration_seeds.goals`,各取其一组合。允许在种子上做衍生变体(如把「查账单」衍生为「查账单并开发票」),但必须在 `seed_ref` 里回溯来源。
|
||||||
|
|
||||||
|
## 派发探索会话
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST http://agenteval:8000/api/exploration/sessions \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"campaign_id": "<campaign_id>",
|
||||||
|
"persona": {"name": "<人设名>", "traits": ["<特质1>", "<特质2>"]},
|
||||||
|
"goal": "<探索目标>",
|
||||||
|
"seed_ref": {"persona_seed": "<所用 persona 种子原文>", "goal_seed": "<所用 goal 种子原文>", "derived": false},
|
||||||
|
"triggered_by": "auto"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
- 直接使用种子时 `derived` 为 false;衍生变体时为 true,并追加 `"derived_from": "<原种子>"` 与 `"variant": "<变异点说明>"`——留痕是探索数据可回溯的底线。
|
||||||
|
- 响应中的 `id` 即 session_id,后续所有步骤都要用它。
|
||||||
|
|
||||||
|
## 驱动会话对话
|
||||||
|
|
||||||
|
以 persona 的身份向目标推进 goal,每轮发送一条用户消息并同步收到回复:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST http://agenteval:8000/api/exploration/sessions/<session_id>/messages \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"content": "<本轮用户消息>"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
- 根据 `reply` 决定下一轮:追问、纠正、换路径,直到目标达成或确认走不通。
|
||||||
|
- 单会话默认最多 12 轮,以 patrol 返回的 `budget.max_turns` 为准,接近上限时尽快收尾。
|
||||||
|
- 502 表示通道异常;连续两次 502 就结束会话,把异常写进体验记录备注。
|
||||||
|
|
||||||
|
## 提交体验记录(闭环收尾)
|
||||||
|
|
||||||
|
会话必须显式关闭,提交第一人称体验记录——这是探索证据链的源头,务必如实:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST http://agenteval:8000/api/exploration/sessions/<session_id>/close \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"experience": {
|
||||||
|
"goal_achieved": true,
|
||||||
|
"blockers": ["<遇到的障碍,逐条列出>"],
|
||||||
|
"misled": ["<被误导的经历:回答把你带偏的地方>"],
|
||||||
|
"emotion": "positive",
|
||||||
|
"notes": "<整体体验简述>"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
- `emotion` 只能是 positive / neutral / confused / frustrated 之一,按真实体感选择。
|
||||||
|
- `blockers` 与 `misled` 写具体事实(如「让我重复提供订单号三次」),不写空泛评价。
|
||||||
|
|
||||||
|
## 收到 409 的收敛行为
|
||||||
|
|
||||||
|
409 是平台账本的明确拒绝,一律停止当前动作、不重试:
|
||||||
|
|
||||||
|
- 创建会话 409(预算耗尽 / 间隔不足 / 活动非进行中)→ 本活动本轮放弃派发,记录原因,等下个巡检节拍。
|
||||||
|
- 发消息 409(会话不在进行中 / 轮数超限)→ 立即走关闭流程提交已有体验记录;若关闭也返回 409,说明会话已被平台收口(如活动终态结算),放弃并记录原因。
|
||||||
|
- 404(活动或会话不存在)→ 对象已消失,跳过。
|
||||||
|
|
||||||
|
## 汇报要求
|
||||||
|
|
||||||
|
每个巡检节拍结束后向用户汇报:巡检到的活动数、派发的会话数(含被 409 拦截的原因)、关闭的会话及其目标达成情况与关键问题。
|
||||||
|
|
||||||
|
请将 <campaign_id>、<session_id> 等占位符替换为实际值。
|
||||||
@ -324,6 +324,69 @@ export interface ExplorationBudgetConfig {
|
|||||||
min_interval_seconds?: number | null
|
min_interval_seconds?: number | null
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface ExplorationIssueCount {
|
||||||
|
issue: string
|
||||||
|
count: number
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExplorationJudgeReview {
|
||||||
|
reviewed_sessions: number
|
||||||
|
findings: { dimension: string; rating: string; comment: string }[]
|
||||||
|
summaries: string[]
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExplorationSummary {
|
||||||
|
session_count: number
|
||||||
|
sessions_with_experience: number
|
||||||
|
goal_achieved_count: number
|
||||||
|
goal_achievement_rate: number | null
|
||||||
|
issues: ExplorationIssueCount[]
|
||||||
|
misled: ExplorationIssueCount[]
|
||||||
|
judge_review: ExplorationJudgeReview | null
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExplorationExperience {
|
||||||
|
goal_achieved: boolean
|
||||||
|
blockers: string[]
|
||||||
|
misled: string[]
|
||||||
|
emotion: string
|
||||||
|
notes: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExplorationSession {
|
||||||
|
id: string
|
||||||
|
campaign_id: string
|
||||||
|
target_id: string
|
||||||
|
persona: Record<string, unknown>
|
||||||
|
goal: string
|
||||||
|
seed_ref: Record<string, unknown> | null
|
||||||
|
status: string
|
||||||
|
triggered_by: string
|
||||||
|
experience: ExplorationExperience | null
|
||||||
|
judge_review: Record<string, unknown> | null
|
||||||
|
turn_count: number
|
||||||
|
error: string | null
|
||||||
|
created_at: string | null
|
||||||
|
closed_at: string | null
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExplorationMessage {
|
||||||
|
id: string
|
||||||
|
session_id: string
|
||||||
|
round_index: number
|
||||||
|
role: string
|
||||||
|
content: string
|
||||||
|
latency_ms: number | null
|
||||||
|
created_at: string | null
|
||||||
|
}
|
||||||
|
|
||||||
|
export const explorationApi = {
|
||||||
|
listSessions: (campaignId: string) =>
|
||||||
|
api.get<{ sessions: ExplorationSession[] }>(`/exploration/campaigns/${campaignId}/sessions`),
|
||||||
|
listMessages: (sessionId: string) =>
|
||||||
|
api.get<{ messages: ExplorationMessage[] }>(`/exploration/sessions/${sessionId}/messages`),
|
||||||
|
}
|
||||||
|
|
||||||
export interface CampaignProgress {
|
export interface CampaignProgress {
|
||||||
current_offset_seconds: number
|
current_offset_seconds: number
|
||||||
spawned_runs: number
|
spawned_runs: number
|
||||||
@ -394,6 +457,7 @@ export interface CampaignReport {
|
|||||||
}
|
}
|
||||||
time_trend: CampaignTrendBucket[]
|
time_trend: CampaignTrendBucket[]
|
||||||
capability_summary: CampaignCapability[]
|
capability_summary: CampaignCapability[]
|
||||||
|
exploration?: ExplorationSummary | null
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CampaignTimelineEntry {
|
export interface CampaignTimelineEntry {
|
||||||
|
|||||||
@ -7,9 +7,11 @@ interface ChatBubbleProps {
|
|||||||
content: ReactNode
|
content: ReactNode
|
||||||
meta?: ReactNode
|
meta?: ReactNode
|
||||||
pending?: boolean
|
pending?: boolean
|
||||||
|
/** 镜像布局:探索下钻里虚拟用户居左、被评对象居右(与常规聊天相反) */
|
||||||
|
mirrored?: boolean
|
||||||
}
|
}
|
||||||
|
|
||||||
export default function ChatBubble({ role, content, meta, pending }: ChatBubbleProps) {
|
export default function ChatBubble({ role, content, meta, pending, mirrored }: ChatBubbleProps) {
|
||||||
const isUser = role === 'user'
|
const isUser = role === 'user'
|
||||||
const isAgent = role === 'agent'
|
const isAgent = role === 'agent'
|
||||||
const isSystem = role === 'system'
|
const isSystem = role === 'system'
|
||||||
@ -17,8 +19,9 @@ export default function ChatBubble({ role, content, meta, pending }: ChatBubbleP
|
|||||||
const bg = isUser ? colors.chatUser : isAgent ? colors.chatAgent : colors.chatSystemError
|
const bg = isUser ? colors.chatUser : isAgent ? colors.chatAgent : colors.chatSystemError
|
||||||
const border = isSystem ? '#ffccc7' : 'transparent'
|
const border = isSystem ? '#ffccc7' : 'transparent'
|
||||||
const textColor = isSystem ? '#cf1322' : colors.text
|
const textColor = isSystem ? '#cf1322' : colors.text
|
||||||
const align = isUser ? 'flex-end' : 'flex-start'
|
const alignRight = isSystem ? false : isUser ? !mirrored : !!mirrored
|
||||||
const radius = isUser ? '10px 10px 2px 10px' : '10px 10px 10px 2px'
|
const align = alignRight ? 'flex-end' : 'flex-start'
|
||||||
|
const radius = alignRight ? '10px 10px 2px 10px' : '10px 10px 10px 2px'
|
||||||
|
|
||||||
const icon = isUser
|
const icon = isUser
|
||||||
? <UserOutlined style={{ color: colors.primary }} />
|
? <UserOutlined style={{ color: colors.primary }} />
|
||||||
@ -29,7 +32,7 @@ export default function ChatBubble({ role, content, meta, pending }: ChatBubbleP
|
|||||||
|
|
||||||
return (
|
return (
|
||||||
<div style={{ display: 'flex', justifyContent: align, marginBottom: 6 }}>
|
<div style={{ display: 'flex', justifyContent: align, marginBottom: 6 }}>
|
||||||
<div style={{ maxWidth: '82%', display: 'flex', flexDirection: 'column', alignItems: isUser ? 'flex-end' : 'flex-start', gap: 2 }}>
|
<div style={{ maxWidth: '82%', display: 'flex', flexDirection: 'column', alignItems: alignRight ? 'flex-end' : 'flex-start', gap: 2 }}>
|
||||||
<div style={{ display: 'flex', gap: 4, alignItems: 'center', fontSize: 11, color: colors.textMuted }}>
|
<div style={{ display: 'flex', gap: 4, alignItems: 'center', fontSize: 11, color: colors.textMuted }}>
|
||||||
{!isUser && icon}
|
{!isUser && icon}
|
||||||
<span>{label}</span>
|
<span>{label}</span>
|
||||||
|
|||||||
227
frontend/web/src/components/ExplorationSection.tsx
Normal file
227
frontend/web/src/components/ExplorationSection.tsx
Normal file
@ -0,0 +1,227 @@
|
|||||||
|
import { useEffect, useState } from 'react'
|
||||||
|
import { Collapse, Empty, Space, Spin, Tag } from 'antd'
|
||||||
|
import ChatBubble from './ChatBubble'
|
||||||
|
import {
|
||||||
|
explorationApi,
|
||||||
|
type ExplorationMessage,
|
||||||
|
type ExplorationSession,
|
||||||
|
type ExplorationSummary,
|
||||||
|
} from '../api'
|
||||||
|
import { colors } from '../tokens'
|
||||||
|
import { shortDateTime } from '../utils/date'
|
||||||
|
|
||||||
|
const SESSION_STATUS: Record<string, { label: string; color: string }> = {
|
||||||
|
running: { label: '进行中', color: 'processing' },
|
||||||
|
completed: { label: '已完成', color: 'success' },
|
||||||
|
failed: { label: '失败', color: 'error' },
|
||||||
|
expired: { label: '已过期', color: 'default' },
|
||||||
|
}
|
||||||
|
|
||||||
|
const EMOTION_LABELS: Record<string, string> = {
|
||||||
|
positive: '满意',
|
||||||
|
neutral: '平静',
|
||||||
|
confused: '困惑',
|
||||||
|
frustrated: '沮丧',
|
||||||
|
}
|
||||||
|
|
||||||
|
const DIMENSION_LABELS: Record<string, string> = {
|
||||||
|
attitude: '服务态度',
|
||||||
|
professionalism: '专业度',
|
||||||
|
hallucination: '幻觉',
|
||||||
|
}
|
||||||
|
|
||||||
|
const RATING_TAG: Record<string, { label: string; color: string }> = {
|
||||||
|
poor: { label: '差', color: 'red' },
|
||||||
|
acceptable: { label: '一般', color: 'orange' },
|
||||||
|
good: { label: '优', color: 'green' },
|
||||||
|
}
|
||||||
|
|
||||||
|
function pct(rate: number | null): string {
|
||||||
|
return rate == null ? '—' : `${(rate * 100).toFixed(1)}%`
|
||||||
|
}
|
||||||
|
|
||||||
|
function IssueList({ label, items, misled }: {
|
||||||
|
label: string
|
||||||
|
items: { issue: string; count: number }[]
|
||||||
|
misled?: boolean
|
||||||
|
}) {
|
||||||
|
if (items.length === 0) return null
|
||||||
|
return (
|
||||||
|
<div style={{ marginBottom: 10 }}>
|
||||||
|
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 6 }}>{label}</div>
|
||||||
|
<Space size={8} wrap>
|
||||||
|
{items.map((item, i) => (
|
||||||
|
<Tag key={i} color={misled ? 'orange' : 'red'}>
|
||||||
|
{misled ? `(被误导)${item.issue}` : item.issue} ×{item.count}
|
||||||
|
</Tag>
|
||||||
|
))}
|
||||||
|
</Space>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
function SessionDetail({ session, messages }: {
|
||||||
|
session: ExplorationSession
|
||||||
|
messages: ExplorationMessage[] | null
|
||||||
|
}) {
|
||||||
|
const experience = session.experience
|
||||||
|
return (
|
||||||
|
<div>
|
||||||
|
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 10 }}>
|
||||||
|
目标:{session.goal}
|
||||||
|
{session.created_at && <> · 开始于 {shortDateTime(session.created_at)}</>}
|
||||||
|
{session.closed_at && <> · 结束于 {shortDateTime(session.closed_at)}</>}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{!messages && <Spin size="small" />}
|
||||||
|
{messages && messages.length === 0 && (
|
||||||
|
<Empty description="暂无对话记录" image={Empty.PRESENTED_IMAGE_SIMPLE} />
|
||||||
|
)}
|
||||||
|
{messages && messages.map((m) => (
|
||||||
|
<ChatBubble
|
||||||
|
key={m.id}
|
||||||
|
role={m.role === 'user' ? 'user' : 'agent'}
|
||||||
|
mirrored
|
||||||
|
content={m.content}
|
||||||
|
meta={m.role !== 'user' && m.latency_ms != null ? (
|
||||||
|
<span style={{ color: colors.textMuted, fontSize: 11 }}>{m.latency_ms}ms</span>
|
||||||
|
) : undefined}
|
||||||
|
/>
|
||||||
|
))}
|
||||||
|
|
||||||
|
{experience && (
|
||||||
|
<div style={{
|
||||||
|
marginTop: 8, border: `1px solid ${colors.border}`, borderRadius: 8,
|
||||||
|
background: colors.bgSubtle, padding: '10px 12px', fontSize: 13, lineHeight: 1.8,
|
||||||
|
}}>
|
||||||
|
<div style={{ fontWeight: 500, marginBottom: 4 }}>体验记录</div>
|
||||||
|
<Space size={8} wrap>
|
||||||
|
{experience.goal_achieved
|
||||||
|
? <Tag color="success">目标达成</Tag>
|
||||||
|
: <Tag color="error">目标未达成</Tag>}
|
||||||
|
<Tag>{EMOTION_LABELS[experience.emotion] ?? experience.emotion}</Tag>
|
||||||
|
</Space>
|
||||||
|
{experience.blockers.length > 0 && (
|
||||||
|
<div style={{ color: colors.textSecondary }}>障碍:{experience.blockers.join(';')}</div>
|
||||||
|
)}
|
||||||
|
{experience.misled.length > 0 && (
|
||||||
|
<div style={{ color: colors.warning }}>被误导:{experience.misled.join(';')}</div>
|
||||||
|
)}
|
||||||
|
{experience.notes && (
|
||||||
|
<div style={{ color: colors.textSecondary }}>备注:{experience.notes}</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{session.error && (
|
||||||
|
<div style={{ marginTop: 8, fontSize: 12, color: colors.textMuted }}>会话异常:{session.error}</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function ExplorationSection({ campaignId, summary }: {
|
||||||
|
campaignId: string
|
||||||
|
summary: ExplorationSummary
|
||||||
|
}) {
|
||||||
|
const [sessions, setSessions] = useState<ExplorationSession[] | null>(null)
|
||||||
|
const [loading, setLoading] = useState(false)
|
||||||
|
const [messageCache, setMessageCache] = useState<Record<string, ExplorationMessage[]>>({})
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let cancelled = false
|
||||||
|
setLoading(true)
|
||||||
|
explorationApi.listSessions(campaignId)
|
||||||
|
.then((res) => { if (!cancelled) setSessions(res.data.sessions) })
|
||||||
|
.catch(() => undefined) // 拦截器已弹错;避免未处理 rejection
|
||||||
|
.finally(() => { if (!cancelled) setLoading(false) })
|
||||||
|
return () => { cancelled = true }
|
||||||
|
}, [campaignId])
|
||||||
|
|
||||||
|
const onExpand = (keys: string | string[]) => {
|
||||||
|
for (const key of Array.isArray(keys) ? keys : [keys]) {
|
||||||
|
if (key in messageCache) continue
|
||||||
|
void explorationApi.listMessages(key)
|
||||||
|
.then((res) => { setMessageCache((prev) => ({ ...prev, [key]: res.data.messages })) })
|
||||||
|
.catch(() => undefined) // 拦截器已弹错;收起再展开可重试
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const judge = summary.judge_review
|
||||||
|
return (
|
||||||
|
<div style={{
|
||||||
|
border: `1px solid ${colors.border}`, borderRadius: 8,
|
||||||
|
padding: '12px 14px', marginBottom: 16,
|
||||||
|
}}>
|
||||||
|
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 10 }}>
|
||||||
|
探索会话 {summary.session_count} 个 · 有体验记录 {summary.sessions_with_experience} 个
|
||||||
|
{' · '}目标达成率 {pct(summary.goal_achievement_rate)}({summary.goal_achieved_count}/{summary.sessions_with_experience})
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<IssueList label="问题清单(体验记录)" items={summary.issues} />
|
||||||
|
<IssueList label="被误导清单(体验记录)" items={summary.misled} misled />
|
||||||
|
{summary.issues.length === 0 && summary.misled.length === 0 && (
|
||||||
|
<div style={{ fontSize: 12, color: colors.textMuted, marginBottom: 10 }}>体验记录未报告问题</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{judge && (
|
||||||
|
<div style={{ marginBottom: 10 }}>
|
||||||
|
<div style={{ fontSize: 12, color: colors.textSecondary, marginBottom: 6 }}>
|
||||||
|
judge 复核({judge.reviewed_sessions} 个会话抽样)
|
||||||
|
</div>
|
||||||
|
{judge.findings.length === 0
|
||||||
|
? <div style={{ fontSize: 13, color: colors.textSecondary }}>无复核发现</div>
|
||||||
|
: (
|
||||||
|
<Space direction="vertical" size={4} style={{ width: '100%' }}>
|
||||||
|
{judge.findings.map((f, i) => (
|
||||||
|
<div key={i} style={{ fontSize: 13 }}>
|
||||||
|
<Tag color={RATING_TAG[f.rating]?.color ?? 'default'} style={{ margin: 0 }}>
|
||||||
|
{DIMENSION_LABELS[f.dimension] ?? f.dimension}
|
||||||
|
{' · '}
|
||||||
|
{RATING_TAG[f.rating]?.label ?? f.rating}
|
||||||
|
</Tag>
|
||||||
|
{' '}{f.comment}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</Space>
|
||||||
|
)}
|
||||||
|
{judge.summaries.map((s, i) => (
|
||||||
|
<div key={i} style={{ fontSize: 13, color: colors.textSecondary, marginTop: 4 }}>
|
||||||
|
复核结论:{s}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{loading && <Spin size="small" />}
|
||||||
|
{!loading && sessions && sessions.length === 0 && (
|
||||||
|
<Empty description="暂无探索会话" image={Empty.PRESENTED_IMAGE_SIMPLE} />
|
||||||
|
)}
|
||||||
|
{sessions && sessions.length > 0 && (
|
||||||
|
<Collapse
|
||||||
|
size="small"
|
||||||
|
onChange={onExpand}
|
||||||
|
items={sessions.map((s) => {
|
||||||
|
const meta = SESSION_STATUS[s.status] ?? SESSION_STATUS.running
|
||||||
|
const personaName = typeof s.persona?.name === 'string' ? s.persona.name : s.id.slice(0, 8)
|
||||||
|
return {
|
||||||
|
key: s.id,
|
||||||
|
label: (
|
||||||
|
<Space size={8} wrap>
|
||||||
|
<span style={{ fontWeight: 500 }}>{personaName}</span>
|
||||||
|
<Tag color={meta.color}>{meta.label}</Tag>
|
||||||
|
<Tag>{s.turn_count} 轮</Tag>
|
||||||
|
{s.experience?.goal_achieved != null && (
|
||||||
|
s.experience.goal_achieved
|
||||||
|
? <Tag color="success">目标达成</Tag>
|
||||||
|
: <Tag color="error">目标未达成</Tag>
|
||||||
|
)}
|
||||||
|
</Space>
|
||||||
|
),
|
||||||
|
children: <SessionDetail session={s} messages={messageCache[s.id] ?? null} />,
|
||||||
|
}
|
||||||
|
})}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
@ -8,7 +8,7 @@ import {
|
|||||||
PlusOutlined, ReloadOutlined, StopOutlined, BarChartOutlined,
|
PlusOutlined, ReloadOutlined, StopOutlined, BarChartOutlined,
|
||||||
FileMarkdownOutlined, MinusCircleOutlined, QuestionCircleOutlined,
|
FileMarkdownOutlined, MinusCircleOutlined, QuestionCircleOutlined,
|
||||||
RocketOutlined, CheckCircleOutlined, SafetyOutlined, ClockCircleOutlined,
|
RocketOutlined, CheckCircleOutlined, SafetyOutlined, ClockCircleOutlined,
|
||||||
RobotOutlined, SwapOutlined,
|
RobotOutlined, SwapOutlined, CompassOutlined,
|
||||||
} from '@ant-design/icons'
|
} from '@ant-design/icons'
|
||||||
import { Line, Bar } from '@ant-design/charts'
|
import { Line, Bar } from '@ant-design/charts'
|
||||||
import PageWrapper from '../components/PageWrapper'
|
import PageWrapper from '../components/PageWrapper'
|
||||||
@ -25,6 +25,7 @@ import { deriveTimeScale, acceleratedDuration, formatScale } from '../utils/camp
|
|||||||
import WindowTimeline, { type TimelineMarker } from '../components/WindowTimeline'
|
import WindowTimeline, { type TimelineMarker } from '../components/WindowTimeline'
|
||||||
import CampaignRunTimeline from '../components/CampaignRunTimeline'
|
import CampaignRunTimeline from '../components/CampaignRunTimeline'
|
||||||
import PeriodComparisonSection from '../components/PeriodComparisonSection'
|
import PeriodComparisonSection from '../components/PeriodComparisonSection'
|
||||||
|
import ExplorationSection from '../components/ExplorationSection'
|
||||||
import { useResource } from '../hooks/useResource'
|
import { useResource } from '../hooks/useResource'
|
||||||
import { usePolling } from '../hooks/usePolling'
|
import { usePolling } from '../hooks/usePolling'
|
||||||
import { useTabStore } from '../stores/tabStore'
|
import { useTabStore } from '../stores/tabStore'
|
||||||
@ -1055,6 +1056,17 @@ export default function CampaignsPage() {
|
|||||||
onGenerate={generateComparison}
|
onGenerate={generateComparison}
|
||||||
/>
|
/>
|
||||||
|
|
||||||
|
{report.exploration && (
|
||||||
|
<>
|
||||||
|
<SectionTitle><CompassOutlined /> 探索发现</SectionTitle>
|
||||||
|
<ExplorationSection
|
||||||
|
key={report.campaign_id}
|
||||||
|
campaignId={report.campaign_id}
|
||||||
|
summary={report.exploration}
|
||||||
|
/>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
|
||||||
<SectionTitle>过程时间轴</SectionTitle>
|
<SectionTitle>过程时间轴</SectionTitle>
|
||||||
<div style={{
|
<div style={{
|
||||||
border: `1px solid ${colors.border}`, borderRadius: 8,
|
border: `1px solid ${colors.border}`, borderRadius: 8,
|
||||||
|
|||||||
@ -93,14 +93,16 @@ run rsync -az --delete \
|
|||||||
./ "$HOST:$REMOTE_DIR/"
|
./ "$HOST:$REMOTE_DIR/"
|
||||||
|
|
||||||
# ── OpenClaw skill sync ────────────────────────────────────────────────
|
# ── OpenClaw skill sync ────────────────────────────────────────────────
|
||||||
# The agenteval-run skill lives in the openclaw workspace volume (data/),
|
# OpenClaw skills live in the openclaw workspace volume (data/), which
|
||||||
# which rsync excludes. Sync the versioned copy explicitly so the AI
|
# rsync excludes. Sync every versioned skill explicitly so the AI
|
||||||
# assistant always follows the standard HTTP API flow.
|
# assistant always follows the standard HTTP API flows.
|
||||||
SKILL_SRC="backend/plugins/openclaw/skills/agenteval-run/SKILL.md"
|
for SKILL_SRC in backend/plugins/openclaw/skills/*/SKILL.md; do
|
||||||
SKILL_DST="$REMOTE_DIR/data/openclaw/workspace/skills/agenteval-run/SKILL.md"
|
SKILL_NAME=$(basename "$(dirname "$SKILL_SRC")")
|
||||||
log "sync OpenClaw skill → $SKILL_DST"
|
SKILL_DST="$REMOTE_DIR/data/openclaw/workspace/skills/$SKILL_NAME/SKILL.md"
|
||||||
run ssh "$HOST" "mkdir -p $(dirname "$SKILL_DST")"
|
log "sync OpenClaw skill → $SKILL_DST"
|
||||||
run rsync -az "$SKILL_SRC" "$HOST:$SKILL_DST"
|
run ssh "$HOST" "mkdir -p $(dirname "$SKILL_DST")"
|
||||||
|
run rsync -az "$SKILL_SRC" "$HOST:$SKILL_DST"
|
||||||
|
done
|
||||||
|
|
||||||
# Provision the API key file the skill reads (~/.openclaw/agenteval-api-key in
|
# Provision the API key file the skill reads (~/.openclaw/agenteval-api-key in
|
||||||
# the openclaw container = data/openclaw/agenteval-api-key on the host). Sourced
|
# the openclaw container = data/openclaw/agenteval-api-key on the host). Sourced
|
||||||
|
|||||||
@ -302,6 +302,61 @@ async def test_campaign_report_markdown_export(client, seeded_db):
|
|||||||
assert "## 能力汇总" in resp.text
|
assert "## 能力汇总" in resp.text
|
||||||
|
|
||||||
|
|
||||||
|
# ── exploration findings in report/export (v0.9 ticket 05) ───────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_exploration_session(seeded_db, campaign_id: str) -> None:
|
||||||
|
from agenteval.exploration.models import ExplorationSession
|
||||||
|
from agenteval.storage.repository import ExplorationSessionRepository
|
||||||
|
|
||||||
|
repo = ExplorationSessionRepository(seeded_db)
|
||||||
|
session_obj = repo.create(
|
||||||
|
ExplorationSession(campaign_id=campaign_id, target_id="t-1", persona={"name": "x"}, goal="缴费")
|
||||||
|
)
|
||||||
|
session_obj.experience = {
|
||||||
|
"goal_achieved": True,
|
||||||
|
"blockers": ["缴费入口难找"],
|
||||||
|
"misled": [],
|
||||||
|
"emotion": "neutral",
|
||||||
|
"notes": "",
|
||||||
|
}
|
||||||
|
repo.update(session_obj)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_campaign_report_includes_exploration_findings(client, seeded_db):
|
||||||
|
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
|
||||||
|
_seed_exploration_session(seeded_db, campaign_id)
|
||||||
|
|
||||||
|
report = (await client.get(f"/api/campaigns/{campaign_id}/report")).json()
|
||||||
|
exploration = report["exploration"]
|
||||||
|
assert exploration["session_count"] == 1
|
||||||
|
assert exploration["goal_achievement_rate"] == 1.0
|
||||||
|
assert exploration["issues"] == [{"issue": "缴费入口难找", "count": 1}]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_campaign_report_without_exploration_omits_key(client, seeded_db):
|
||||||
|
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
|
||||||
|
report = (await client.get(f"/api/campaigns/{campaign_id}/report")).json()
|
||||||
|
assert "exploration" not in report
|
||||||
|
|
||||||
|
|
||||||
|
async def test_markdown_export_appends_exploration_appendix(client, seeded_db):
|
||||||
|
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
|
||||||
|
_seed_exploration_session(seeded_db, campaign_id)
|
||||||
|
|
||||||
|
resp = await client.get(f"/api/campaigns/{campaign_id}/report/markdown")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
assert "## 探索发现" in resp.text
|
||||||
|
assert "缴费入口难找 ×1" in resp.text
|
||||||
|
|
||||||
|
|
||||||
|
async def test_markdown_export_without_exploration_leaves_no_trace(client, seeded_db):
|
||||||
|
campaign_id = (await client.post("/api/campaigns", json=_valid_payload())).json()["id"]
|
||||||
|
resp = await client.get(f"/api/campaigns/{campaign_id}/report/markdown")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
assert "探索发现" not in resp.text
|
||||||
|
|
||||||
|
|
||||||
# ── campaign timeline endpoint (ticket 06) ───────────────────────────────────
|
# ── campaign timeline endpoint (ticket 06) ───────────────────────────────────
|
||||||
|
|
||||||
async def test_campaign_timeline_structure(client, seeded_db):
|
async def test_campaign_timeline_structure(client, seeded_db):
|
||||||
|
|||||||
@ -54,6 +54,11 @@ def seeded_db(db_session, monkeypatch):
|
|||||||
|
|
||||||
app.dependency_overrides[get_db] = _test_get_db
|
app.dependency_overrides[get_db] = _test_get_db
|
||||||
|
|
||||||
|
# 隔离后台 judge 复核任务:API 测试不真正派发后台任务
|
||||||
|
from agenteval.web.routers import exploration as exploration_router
|
||||||
|
|
||||||
|
monkeypatch.setattr(exploration_router, "start_judge_review", lambda session_id: None)
|
||||||
|
|
||||||
target = EvalTarget(
|
target = EvalTarget(
|
||||||
id="t-1",
|
id="t-1",
|
||||||
name="mock-target",
|
name="mock-target",
|
||||||
@ -170,6 +175,37 @@ async def test_full_lifecycle_create_message_close(seeded_db, mock_channel, clie
|
|||||||
assert closed["closed_at"] is not None
|
assert closed["closed_at"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_dict_reply_content_is_flattened_to_text(seeded_db, monkeypatch, client):
|
||||||
|
"""通道回复 content 为对象(如 tutu msgBody)时应提取文本而非存 str(dict)。"""
|
||||||
|
from agenteval.channels.base import Reply
|
||||||
|
from tests.unit.mock_channel import MockChannel
|
||||||
|
|
||||||
|
class _DictReplyChannel(MockChannel):
|
||||||
|
async def poll_reply(self, question_msg_id, timeout=30.0, poll_interval=1.0):
|
||||||
|
return Reply(
|
||||||
|
question_msg_id=question_msg_id,
|
||||||
|
content={"content": "您好,我是客服"},
|
||||||
|
raw_message={},
|
||||||
|
)
|
||||||
|
|
||||||
|
_stub_channel_factory(monkeypatch, _DictReplyChannel(reply_text="unused"))
|
||||||
|
|
||||||
|
resp = await _create_session(client)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
session_id = resp.json()["id"]
|
||||||
|
|
||||||
|
resp = await client.post(
|
||||||
|
f"/api/exploration/sessions/{session_id}/messages",
|
||||||
|
json={"content": "你好"},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
assert resp.json()["reply"] == "您好,我是客服"
|
||||||
|
|
||||||
|
messages_resp = await client.get(f"/api/exploration/sessions/{session_id}/messages")
|
||||||
|
assistant = [m for m in messages_resp.json()["messages"] if m["role"] == "assistant"]
|
||||||
|
assert assistant[0]["content"] == "您好,我是客服"
|
||||||
|
|
||||||
|
|
||||||
async def test_create_requires_existing_running_campaign(seeded_db, client):
|
async def test_create_requires_existing_running_campaign(seeded_db, client):
|
||||||
resp = await _create_session(client, campaign_id="nope")
|
resp = await _create_session(client, campaign_id="nope")
|
||||||
assert resp.status_code == 404
|
assert resp.status_code == 404
|
||||||
@ -330,6 +366,80 @@ async def test_close_twice_rejected(seeded_db, mock_channel, client):
|
|||||||
assert resp.status_code == 409
|
assert resp.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
async def test_close_triggers_judge_review(seeded_db, mock_channel, client, monkeypatch):
|
||||||
|
from agenteval.web.routers import exploration as exploration_router
|
||||||
|
|
||||||
|
started: list[str] = []
|
||||||
|
monkeypatch.setattr(exploration_router, "start_judge_review", started.append)
|
||||||
|
|
||||||
|
session_id = (await _create_session(client)).json()["id"]
|
||||||
|
resp = await client.post(
|
||||||
|
f"/api/exploration/sessions/{session_id}/close", json={"experience": {"goal_achieved": True}}
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
assert started == [session_id]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- read endpoints (ticket 06)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_list_campaign_sessions_returns_lifecycle_fields(seeded_db, mock_channel, client):
|
||||||
|
resp = await _create_session(client)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
session_id = resp.json()["id"]
|
||||||
|
|
||||||
|
await client.post(f"/api/exploration/sessions/{session_id}/messages", json={"content": "查账单"})
|
||||||
|
resp = await client.post(
|
||||||
|
f"/api/exploration/sessions/{session_id}/close",
|
||||||
|
json={"experience": {"goal_achieved": True, "blockers": [], "misled": ["跳转误导"], "notes": "绕了三圈"}},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
|
||||||
|
resp = await client.get("/api/exploration/campaigns/c-1/sessions")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
sessions = resp.json()["sessions"]
|
||||||
|
assert len(sessions) == 1
|
||||||
|
entry = sessions[0]
|
||||||
|
assert entry["id"] == session_id
|
||||||
|
assert entry["status"] == "completed"
|
||||||
|
assert entry["turn_count"] == 1
|
||||||
|
assert entry["goal"] == "查询本月账单并完成缴费"
|
||||||
|
assert entry["persona"]["name"] == "急性子用户"
|
||||||
|
assert entry["experience"]["goal_achieved"] is True
|
||||||
|
assert entry["experience"]["misled"] == ["跳转误导"]
|
||||||
|
assert entry["closed_at"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_list_sessions_empty_campaign(seeded_db, client):
|
||||||
|
resp = await client.get("/api/exploration/campaigns/c-1/sessions")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
assert resp.json() == {"sessions": []}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_list_session_messages_returns_conversation(seeded_db, mock_channel, client):
|
||||||
|
session_id = (await _create_session(client)).json()["id"]
|
||||||
|
await client.post(f"/api/exploration/sessions/{session_id}/messages", json={"content": "我要查账单"})
|
||||||
|
|
||||||
|
resp = await client.get(f"/api/exploration/sessions/{session_id}/messages")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
messages = resp.json()["messages"]
|
||||||
|
assert len(messages) == 2
|
||||||
|
user_msg, agent_msg = messages
|
||||||
|
assert user_msg["role"] == "user"
|
||||||
|
assert user_msg["content"] == "我要查账单"
|
||||||
|
assert user_msg["round_index"] == 1
|
||||||
|
assert agent_msg["role"] == "assistant"
|
||||||
|
assert agent_msg["content"] == "您好,请问有什么可以帮您?"
|
||||||
|
assert agent_msg["round_index"] == 1
|
||||||
|
assert isinstance(agent_msg["latency_ms"], int)
|
||||||
|
assert user_msg["created_at"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_list_messages_unknown_session_returns_404(seeded_db, client):
|
||||||
|
resp = await client.get("/api/exploration/sessions/nope/messages")
|
||||||
|
assert resp.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- migration
|
# ---------------------------------------------------------------- migration
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
189
tests/integration/test_exploration_settlement.py
Normal file
189
tests/integration/test_exploration_settlement.py
Normal file
@ -0,0 +1,189 @@
|
|||||||
|
"""Integration tests for exploration settlement on campaign finalize (v0.9 票据 07).
|
||||||
|
|
||||||
|
Covers both terminal transitions: window completion (durable loop COMPLETE)
|
||||||
|
and mid-flight cancellation. Running sessions must turn expired and reject
|
||||||
|
further messages (409), leaving no dangling sessions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from agenteval.evaluation import campaign_runner as runner_module
|
||||||
|
from agenteval.evaluation.campaign_runner import run_campaign_loop
|
||||||
|
from agenteval.exploration.models import ExplorationSessionStatus
|
||||||
|
from agenteval.models import (
|
||||||
|
Campaign,
|
||||||
|
CampaignPlanEntry,
|
||||||
|
Case,
|
||||||
|
CaseType,
|
||||||
|
ChannelType,
|
||||||
|
EvalTarget,
|
||||||
|
PlatformType,
|
||||||
|
Scenario,
|
||||||
|
TargetStatus,
|
||||||
|
)
|
||||||
|
from agenteval.storage.db import utc_now
|
||||||
|
from agenteval.storage.repository import (
|
||||||
|
CampaignRepository,
|
||||||
|
ExplorationSessionRepository,
|
||||||
|
ScenarioRepository,
|
||||||
|
TargetRepository,
|
||||||
|
)
|
||||||
|
from agenteval.web.app import app
|
||||||
|
from httpx import ASGITransport, AsyncClient
|
||||||
|
|
||||||
|
|
||||||
|
def _session_payload(campaign_id: str) -> dict:
|
||||||
|
return {
|
||||||
|
"campaign_id": campaign_id,
|
||||||
|
"persona": {"name": "巡检用户", "traits": ["耐心"]},
|
||||||
|
"goal": "查询账单",
|
||||||
|
"triggered_by": "manual",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def seeded_db(db_session, monkeypatch):
|
||||||
|
"""Point every get_session consumer at the test session and seed the
|
||||||
|
target + scenario needed by both the loop path and the cancel path."""
|
||||||
|
from agenteval.channels import factory as factory_module
|
||||||
|
from agenteval.evaluation import engine as engine_module
|
||||||
|
from agenteval.storage import db as db_module
|
||||||
|
from agenteval.storage import repository as repo_module
|
||||||
|
from agenteval.web import app as app_module
|
||||||
|
from agenteval.web.routers import exploration as exploration_router
|
||||||
|
|
||||||
|
monkeypatch.setattr(app_module, "init_db", lambda: None)
|
||||||
|
|
||||||
|
def _test_get_session():
|
||||||
|
return db_session
|
||||||
|
|
||||||
|
monkeypatch.setattr(db_module, "get_session", _test_get_session)
|
||||||
|
monkeypatch.setattr(repo_module, "get_session", _test_get_session)
|
||||||
|
monkeypatch.setattr(engine_module, "get_session", _test_get_session)
|
||||||
|
monkeypatch.setattr(runner_module, "get_session", _test_get_session)
|
||||||
|
monkeypatch.setattr(runner_module, "_auto_start_analysis", lambda campaign, session: None)
|
||||||
|
|
||||||
|
from tests.unit.mock_channel import MockChannel
|
||||||
|
|
||||||
|
channel = MockChannel(reply_delay=0.0)
|
||||||
|
monkeypatch.setattr(factory_module.ChannelFactory, "create", lambda target: channel)
|
||||||
|
monkeypatch.setattr(exploration_router, "start_judge_review", lambda session_id: None)
|
||||||
|
|
||||||
|
from agenteval.web.deps import get_db
|
||||||
|
|
||||||
|
def _test_get_db():
|
||||||
|
try:
|
||||||
|
yield db_session
|
||||||
|
finally:
|
||||||
|
pass
|
||||||
|
|
||||||
|
app.dependency_overrides[get_db] = _test_get_db
|
||||||
|
|
||||||
|
target = EvalTarget(
|
||||||
|
id="t-1",
|
||||||
|
name="mock-target",
|
||||||
|
platform=PlatformType.AI_DIGITAL_EMPLOYEE,
|
||||||
|
channel_type=ChannelType.TUTU_API,
|
||||||
|
channel_config={"base_url": "http://mock", "token": "x"},
|
||||||
|
status=TargetStatus.ACTIVE,
|
||||||
|
)
|
||||||
|
TargetRepository(db_session).create(target)
|
||||||
|
scenario = Scenario(
|
||||||
|
id="s-1",
|
||||||
|
name="mock-scenario",
|
||||||
|
cases=[Case(id="c1", type=CaseType.SINGLE, messages=["hi"])],
|
||||||
|
)
|
||||||
|
ScenarioRepository(db_session).create(scenario)
|
||||||
|
|
||||||
|
yield db_session
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def _make_campaign(db_session, campaign_id: str, *, time_scale: float) -> Campaign:
|
||||||
|
return CampaignRepository(db_session).create(
|
||||||
|
Campaign(
|
||||||
|
id=campaign_id,
|
||||||
|
name=f"campaign-{campaign_id}",
|
||||||
|
target_id="t-1",
|
||||||
|
window_seconds=3600,
|
||||||
|
time_scale=time_scale,
|
||||||
|
plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=1)],
|
||||||
|
status="running",
|
||||||
|
started_at=utc_now(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _create_running_session(db_session, campaign_id: str) -> str:
|
||||||
|
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
|
||||||
|
resp = await c.post("/api/exploration/sessions", json=_session_payload(campaign_id))
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
session_id = resp.json()["id"]
|
||||||
|
assert ExplorationSessionRepository(db_session).get(session_id).status == (ExplorationSessionStatus.RUNNING)
|
||||||
|
return session_id
|
||||||
|
|
||||||
|
|
||||||
|
async def test_window_completion_expires_running_sessions(seeded_db):
|
||||||
|
campaign = _make_campaign(seeded_db, "c-done", time_scale=3600.0)
|
||||||
|
session_id = await _create_running_session(seeded_db, "c-done")
|
||||||
|
|
||||||
|
# time_scale 3600 → 1 real second covers the whole window; the loop
|
||||||
|
# completes on its first tick and settles exploration sessions.
|
||||||
|
cancel = asyncio.Event()
|
||||||
|
await asyncio.wait_for(run_campaign_loop(campaign.id, cancel, tick_seconds=0.05), timeout=5)
|
||||||
|
|
||||||
|
assert CampaignRepository(seeded_db).get("c-done").status.value == "completed"
|
||||||
|
settled = ExplorationSessionRepository(seeded_db).get(session_id)
|
||||||
|
assert settled.status == ExplorationSessionStatus.EXPIRED
|
||||||
|
assert settled.closed_at is not None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_cancel_expires_running_sessions_and_rejects_messages(seeded_db):
|
||||||
|
_make_campaign(seeded_db, "c-1", time_scale=1.0)
|
||||||
|
session_id = await _create_running_session(seeded_db, "c-1")
|
||||||
|
|
||||||
|
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||||
|
resp = await client.post("/api/campaigns/c-1/cancel")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
assert resp.json()["status"] == "cancelled"
|
||||||
|
|
||||||
|
settled = ExplorationSessionRepository(seeded_db).get(session_id)
|
||||||
|
assert settled.status == ExplorationSessionStatus.EXPIRED
|
||||||
|
assert settled.closed_at is not None
|
||||||
|
|
||||||
|
resp = await client.post(f"/api/exploration/sessions/{session_id}/messages", json={"content": "还在吗"})
|
||||||
|
assert resp.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
async def test_completed_session_survives_settlement(seeded_db):
|
||||||
|
_make_campaign(seeded_db, "c-1", time_scale=1.0)
|
||||||
|
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||||
|
resp = await client.post("/api/exploration/sessions", json=_session_payload("c-1"))
|
||||||
|
session_id = resp.json()["id"]
|
||||||
|
resp = await client.post(
|
||||||
|
f"/api/exploration/sessions/{session_id}/close",
|
||||||
|
json={"experience": {"goal_achieved": True}},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200
|
||||||
|
|
||||||
|
await client.post("/api/campaigns/c-1/cancel")
|
||||||
|
|
||||||
|
assert ExplorationSessionRepository(seeded_db).get(session_id).status == (ExplorationSessionStatus.COMPLETED)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_production_line_manual_trigger_also_settles(seeded_db):
|
||||||
|
_make_campaign(seeded_db, "c-1", time_scale=1.0)
|
||||||
|
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||||
|
resp = await client.post(
|
||||||
|
"/api/exploration/sessions",
|
||||||
|
json={**_session_payload("c-1"), "triggered_by": "manual"},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
session_id = resp.json()["id"]
|
||||||
|
assert resp.json()["triggered_by"] == "manual"
|
||||||
|
|
||||||
|
# 正式线手动会话同样纳入终态结算,触发来源不影响收口。
|
||||||
|
await client.post("/api/campaigns/c-1/cancel")
|
||||||
|
|
||||||
|
assert ExplorationSessionRepository(seeded_db).get(session_id).status == (ExplorationSessionStatus.EXPIRED)
|
||||||
@ -152,6 +152,45 @@ async def test_scenario_without_failure_samples_still_gets_narrative():
|
|||||||
assert len(result["scenario_narratives"]) == 2
|
assert len(result["scenario_narratives"]) == 2
|
||||||
|
|
||||||
|
|
||||||
|
async def test_analysis_input_includes_exploration_summary():
|
||||||
|
"""v0.9 票据 05:阶段二输入追加探索摘要(问题清单 + 达成统计,非全量对话)。"""
|
||||||
|
exploration_summary = {
|
||||||
|
"session_count": 2,
|
||||||
|
"sessions_with_experience": 2,
|
||||||
|
"goal_achieved_count": 1,
|
||||||
|
"goal_achievement_rate": 0.5,
|
||||||
|
"issues": [{"issue": "缴费入口难找", "count": 2}],
|
||||||
|
"misled": [],
|
||||||
|
"judge_review": None,
|
||||||
|
}
|
||||||
|
client = FakeChatClient(STAGE1_A, STAGE1_B, STAGE2)
|
||||||
|
await analyze_campaign(
|
||||||
|
campaign=_campaign(),
|
||||||
|
report=_report(),
|
||||||
|
failure_samples={},
|
||||||
|
valid_run_ids=set(),
|
||||||
|
chat_client=client,
|
||||||
|
exploration_summary=exploration_summary,
|
||||||
|
)
|
||||||
|
stage2_prompt = json.dumps(client.calls[2], ensure_ascii=False)
|
||||||
|
assert "探索发现" in stage2_prompt
|
||||||
|
assert "缴费入口难找" in stage2_prompt
|
||||||
|
assert "goal_achievement_rate" in stage2_prompt or "0.5" in stage2_prompt
|
||||||
|
|
||||||
|
|
||||||
|
async def test_analysis_without_exploration_omits_section():
|
||||||
|
client = FakeChatClient(STAGE1_A, STAGE1_B, STAGE2)
|
||||||
|
await analyze_campaign(
|
||||||
|
campaign=_campaign(),
|
||||||
|
report=_report(),
|
||||||
|
failure_samples={},
|
||||||
|
valid_run_ids=set(),
|
||||||
|
chat_client=client,
|
||||||
|
)
|
||||||
|
stage2_prompt = json.dumps(client.calls[2], ensure_ascii=False)
|
||||||
|
assert "探索发现" not in stage2_prompt
|
||||||
|
|
||||||
|
|
||||||
# ── 分析模型解析 ─────────────────────────────────────────────────────────
|
# ── 分析模型解析 ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def _seed_config(session, config_id: str, *, analysis_default: bool = False, enabled: bool = True) -> None:
|
def _seed_config(session, config_id: str, *, analysis_default: bool = False, enabled: bool = True) -> None:
|
||||||
|
|||||||
235
tests/unit/test_campaign_report_format.py
Normal file
235
tests/unit/test_campaign_report_format.py
Normal file
@ -0,0 +1,235 @@
|
|||||||
|
"""Report aggregation logic & output format validation (报告聚合与格式校验).
|
||||||
|
|
||||||
|
补充 test_campaign_report.py(数值口径)与 test_metrics.py(单点聚合)之外:
|
||||||
|
- 聚合边界:用户取消不计分母(ADR-0004)、窗口外/窗口前运行钳制、
|
||||||
|
SQLite naive 时间戳与 aware 等价;
|
||||||
|
- 格式校验:报告 dict 的键集合、类型、取值范围、时间序列连续性、
|
||||||
|
能力面排序与 ISO-UTC 时间戳格式。generate_campaign_report 是纯函数,
|
||||||
|
直接构造模型对象,不落库。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
from agenteval.evaluation.metrics import aggregate_runs
|
||||||
|
from agenteval.evaluation.report import generate_campaign_report
|
||||||
|
from agenteval.models import Campaign, CampaignPlanEntry, CampaignStatus, EvalRun, RunStatus
|
||||||
|
|
||||||
|
T0 = datetime(2026, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def _campaign(*, window_seconds: int = 12, time_scale: float = 1.0, started_at=T0, completed_at=None) -> Campaign:
|
||||||
|
return Campaign(
|
||||||
|
id="camp-fmt",
|
||||||
|
name="格式校验活动",
|
||||||
|
target_id="t-1",
|
||||||
|
window_seconds=window_seconds,
|
||||||
|
time_scale=time_scale,
|
||||||
|
plan=[CampaignPlanEntry(scenario_id="s-a", offset_seconds=0, count=1)],
|
||||||
|
status=CampaignStatus.RUNNING,
|
||||||
|
started_at=started_at,
|
||||||
|
completed_at=completed_at,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run(
|
||||||
|
scenario_id: str = "s-a",
|
||||||
|
*,
|
||||||
|
status: RunStatus = RunStatus.COMPLETED,
|
||||||
|
offset_seconds: float = 0.0,
|
||||||
|
pass_rate: float | None = None,
|
||||||
|
latency: float | None = None,
|
||||||
|
error: dict | None = None,
|
||||||
|
started_at: datetime | None = None,
|
||||||
|
) -> EvalRun:
|
||||||
|
summary = None
|
||||||
|
if pass_rate is not None or latency is not None or error is not None:
|
||||||
|
summary = {}
|
||||||
|
if pass_rate is not None:
|
||||||
|
summary["pass_rate"] = pass_rate
|
||||||
|
if latency is not None:
|
||||||
|
summary["avg_latency_ms"] = latency
|
||||||
|
if error is not None:
|
||||||
|
summary["error"] = error
|
||||||
|
return EvalRun(
|
||||||
|
target_id="t-1",
|
||||||
|
scenario_id=scenario_id,
|
||||||
|
campaign_id="camp-fmt",
|
||||||
|
status=status,
|
||||||
|
started_at=started_at if started_at is not None else T0 + timedelta(seconds=offset_seconds),
|
||||||
|
summary=summary,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _report(runs, **kwargs):
|
||||||
|
return generate_campaign_report(_campaign(), runs, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
# ── 格式校验:顶层结构 ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_top_level_schema_keys_and_types():
|
||||||
|
report = _report([_run(pass_rate=1.0, latency=100)])
|
||||||
|
assert set(report) == {
|
||||||
|
"campaign_id",
|
||||||
|
"name",
|
||||||
|
"target_id",
|
||||||
|
"status",
|
||||||
|
"window_seconds",
|
||||||
|
"time_scale",
|
||||||
|
"started_at",
|
||||||
|
"completed_at",
|
||||||
|
"summary",
|
||||||
|
"time_trend",
|
||||||
|
"capability_summary",
|
||||||
|
}
|
||||||
|
assert report["campaign_id"] == "camp-fmt"
|
||||||
|
assert isinstance(report["window_seconds"], int) and report["window_seconds"] > 0
|
||||||
|
assert isinstance(report["time_scale"], float)
|
||||||
|
assert report["status"] == CampaignStatus.RUNNING.value
|
||||||
|
assert isinstance(report["time_trend"], list)
|
||||||
|
assert isinstance(report["capability_summary"], list)
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_schema_types_and_bounds():
|
||||||
|
report = _report(
|
||||||
|
[
|
||||||
|
_run(pass_rate=1.0, latency=100),
|
||||||
|
_run(scenario_id="s-b", status=RunStatus.FAILED),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
summary = report["summary"]
|
||||||
|
assert set(summary) == {
|
||||||
|
"total_runs",
|
||||||
|
"completed_runs",
|
||||||
|
"overall_pass_rate",
|
||||||
|
"overall_availability",
|
||||||
|
"avg_latency_ms",
|
||||||
|
}
|
||||||
|
assert isinstance(summary["total_runs"], int) and summary["total_runs"] == 2
|
||||||
|
assert isinstance(summary["completed_runs"], int) and summary["completed_runs"] == 1
|
||||||
|
for key in ("overall_pass_rate", "overall_availability"):
|
||||||
|
value = summary[key]
|
||||||
|
assert value is None or (isinstance(value, float) and 0.0 <= value <= 1.0), key
|
||||||
|
assert summary["avg_latency_ms"] is None or summary["avg_latency_ms"] >= 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_matches_single_aggregation_seam():
|
||||||
|
"""summary 数值必须与单点聚合口径 aggregate_runs 完全一致(ADR-0004)。"""
|
||||||
|
runs = [
|
||||||
|
_run(pass_rate=1.0, latency=100),
|
||||||
|
_run(pass_rate=0.5, latency=300),
|
||||||
|
_run(status=RunStatus.FAILED),
|
||||||
|
]
|
||||||
|
report = _report(runs)
|
||||||
|
agg = aggregate_runs(runs)
|
||||||
|
summary = report["summary"]
|
||||||
|
assert summary["overall_pass_rate"] == agg["pass_rate"]
|
||||||
|
assert summary["overall_availability"] == agg["availability"]
|
||||||
|
assert summary["avg_latency_ms"] == agg["avg_latency_ms"]
|
||||||
|
|
||||||
|
|
||||||
|
# ── 格式校验:时间趋势序列 ────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_time_trend_contiguous_buckets_with_custom_count():
|
||||||
|
report = generate_campaign_report(_campaign(window_seconds=12), [_run(pass_rate=1.0)], bucket_count=4)
|
||||||
|
trend = report["time_trend"]
|
||||||
|
assert len(trend) == 4
|
||||||
|
for idx, bucket in enumerate(trend):
|
||||||
|
assert bucket["bucket_index"] == idx
|
||||||
|
assert bucket["start_seconds"] == idx * 3.0 # window / bucket_count
|
||||||
|
assert bucket["end_seconds"] == (idx + 1) * 3.0
|
||||||
|
assert {"run_count", "pass_rate", "availability", "avg_latency_ms"} <= set(bucket)
|
||||||
|
# 相邻桶首尾相接
|
||||||
|
for prev, cur in zip(trend, trend[1:]):
|
||||||
|
assert prev["end_seconds"] == cur["start_seconds"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_time_trend_bucket_types_and_bounds():
|
||||||
|
report = _report([_run(pass_rate=0.8, latency=120)])
|
||||||
|
for bucket in report["time_trend"]:
|
||||||
|
assert isinstance(bucket["run_count"], int) and bucket["run_count"] >= 0
|
||||||
|
if bucket["run_count"] == 0:
|
||||||
|
assert bucket["pass_rate"] is None and bucket["availability"] is None
|
||||||
|
else:
|
||||||
|
assert 0.0 <= bucket["pass_rate"] <= 1.0
|
||||||
|
assert 0.0 <= bucket["availability"] <= 1.0
|
||||||
|
|
||||||
|
|
||||||
|
# ── 格式校验:能力面汇总 ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_capability_summary_schema_and_sorted_by_run_count_desc():
|
||||||
|
runs = [
|
||||||
|
_run("s-a", pass_rate=1.0),
|
||||||
|
_run("s-a", pass_rate=0.5),
|
||||||
|
_run("s-a", pass_rate=0.0),
|
||||||
|
_run("s-b", pass_rate=1.0),
|
||||||
|
]
|
||||||
|
report = _report(runs, scenario_names={"s-a": "夜间问诊"})
|
||||||
|
caps = report["capability_summary"]
|
||||||
|
assert [c["scenario_id"] for c in caps] == ["s-a", "s-b"] # run_count 降序
|
||||||
|
for cap in caps:
|
||||||
|
assert {"scenario_id", "scenario_name", "run_count", "pass_rate", "availability", "avg_latency_ms"} <= set(cap)
|
||||||
|
assert cap["run_count"] >= 1
|
||||||
|
assert caps[0]["scenario_name"] == "夜间问诊"
|
||||||
|
assert caps[1]["scenario_name"] == "s-b"[:8] # 无映射时回落 id 前缀
|
||||||
|
|
||||||
|
|
||||||
|
# ── 格式校验:时间戳 ISO-UTC ──────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_timestamps_serialize_as_iso_utc_or_none():
|
||||||
|
report = generate_campaign_report(_campaign(completed_at=T0 + timedelta(hours=1)), [_run()])
|
||||||
|
assert report["started_at"].endswith("Z")
|
||||||
|
assert report["completed_at"].endswith("Z")
|
||||||
|
assert "T" in report["started_at"]
|
||||||
|
|
||||||
|
unfinished = generate_campaign_report(_campaign(), [_run()])
|
||||||
|
assert unfinished["completed_at"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_naive_campaign_timestamps_still_serialize_utc():
|
||||||
|
naive_start = datetime(2026, 1, 1, 0, 0, 0) # SQLite round-trip drops tzinfo
|
||||||
|
report = generate_campaign_report(_campaign(started_at=naive_start), [_run()])
|
||||||
|
assert report["started_at"].endswith("Z")
|
||||||
|
|
||||||
|
|
||||||
|
# ── 聚合边界:取消 / 钳制 / naive 时间 ────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_cancelled_run_excluded_from_summary_but_counted():
|
||||||
|
runs = [
|
||||||
|
_run(pass_rate=1.0, latency=100),
|
||||||
|
_run(status=RunStatus.FAILED, error={"code": "cancelled_by_user", "message": "stop"}),
|
||||||
|
]
|
||||||
|
summary = _report(runs)["summary"]
|
||||||
|
assert summary["total_runs"] == 2 # 发生过的事仍然可见
|
||||||
|
assert summary["completed_runs"] == 1
|
||||||
|
assert summary["overall_pass_rate"] == 1.0 # 取消不进分母(ADR-0004)
|
||||||
|
assert summary["overall_availability"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_started_before_window_clamped_to_first_bucket():
|
||||||
|
report = _report([_run(pass_rate=1.0, started_at=T0 - timedelta(seconds=10))])
|
||||||
|
assert report["time_trend"][0]["run_count"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_started_after_window_clamped_to_last_bucket():
|
||||||
|
report = _report([_run(pass_rate=1.0, started_at=T0 + timedelta(seconds=100))])
|
||||||
|
assert report["time_trend"][-1]["run_count"] == 1
|
||||||
|
assert sum(b["run_count"] for b in report["time_trend"]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_naive_run_timestamps_bucket_same_as_aware():
|
||||||
|
aware = _run(pass_rate=1.0, started_at=T0 + timedelta(seconds=6))
|
||||||
|
naive = _run(pass_rate=1.0, started_at=datetime(2026, 1, 1, 0, 0, 6)) # SQLite 读回的 naive UTC
|
||||||
|
trend_aware = _report([aware])["time_trend"]
|
||||||
|
trend_naive = _report([naive])["time_trend"]
|
||||||
|
assert trend_aware == trend_naive
|
||||||
|
assert trend_naive[6]["run_count"] == 1 # window 12s / 12 buckets → offset 6 落第 6 桶
|
||||||
|
|
||||||
|
|
||||||
|
def test_unstarted_run_sits_at_offset_zero():
|
||||||
|
report = _report([_run(pass_rate=1.0, started_at=None)])
|
||||||
|
assert report["time_trend"][0]["run_count"] == 1
|
||||||
239
tests/unit/test_exploration_judge.py
Normal file
239
tests/unit/test_exploration_judge.py
Normal file
@ -0,0 +1,239 @@
|
|||||||
|
"""Judge sampling review for exploration sessions (v0.9 票据 04).
|
||||||
|
|
||||||
|
会话关闭后,平台抽样对话(≤3 段)经 judge 岗位模型独立复核,质量维度结论
|
||||||
|
结构化落入会话的 judge_review。复核是异步后台执行:失败落错误不阻塞会话
|
||||||
|
状态;无模型配置时静默跳过。LLM 调用经可注入 ChatClient 接缝(沿 v0.7
|
||||||
|
分析 seam),测试用假客户端覆盖(先例:test_campaign_analysis.py)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from agenteval.exploration.judge import (
|
||||||
|
MAX_JUDGE_SAMPLES,
|
||||||
|
execute_judge_review,
|
||||||
|
normalize_judge_review,
|
||||||
|
sample_round_indexes,
|
||||||
|
)
|
||||||
|
from agenteval.exploration.models import (
|
||||||
|
ExplorationMessage,
|
||||||
|
ExplorationSession,
|
||||||
|
ExplorationSessionStatus,
|
||||||
|
)
|
||||||
|
from agenteval.models import Campaign, CampaignPlanEntry
|
||||||
|
from agenteval.storage.db import ModelConfigDB
|
||||||
|
from agenteval.storage.model_config_repository import ModelConfigRepository
|
||||||
|
from agenteval.storage.repository import (
|
||||||
|
CampaignRepository,
|
||||||
|
ExplorationMessageRepository,
|
||||||
|
ExplorationSessionRepository,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FakeChatClient:
|
||||||
|
"""Queued-response fake for the judge LLM seam."""
|
||||||
|
|
||||||
|
def __init__(self, *responses):
|
||||||
|
self._responses = list(responses)
|
||||||
|
self.calls: list[list[dict]] = []
|
||||||
|
|
||||||
|
async def __call__(self, messages: list[dict]) -> str:
|
||||||
|
self.calls.append(messages)
|
||||||
|
if not self._responses:
|
||||||
|
raise AssertionError("unexpected extra LLM call")
|
||||||
|
item = self._responses.pop(0)
|
||||||
|
if isinstance(item, Exception):
|
||||||
|
raise item
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
REVIEW_JSON = json.dumps(
|
||||||
|
{
|
||||||
|
"dimensions": [
|
||||||
|
{"dimension": "attitude", "rating": "good", "comment": "态度友好"},
|
||||||
|
{"dimension": "professionalism", "rating": "acceptable", "comment": "流程基本正确"},
|
||||||
|
{"dimension": "hallucination", "rating": "poor", "comment": "编造了不存在的政策"},
|
||||||
|
],
|
||||||
|
"summary": "服务态度好但存在幻觉",
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _campaign() -> Campaign:
|
||||||
|
return Campaign(
|
||||||
|
id="camp-1",
|
||||||
|
name="24h 正式线",
|
||||||
|
target_id="t-1",
|
||||||
|
window_seconds=86400,
|
||||||
|
time_scale=1.0,
|
||||||
|
plan=[CampaignPlanEntry(scenario_id="s-1", offset_seconds=0, count=1)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_config(db_session, config_id: str = "mc-default") -> None:
|
||||||
|
ModelConfigRepository(db_session).create(
|
||||||
|
ModelConfigDB(
|
||||||
|
id=config_id,
|
||||||
|
name=f"cfg-{config_id}",
|
||||||
|
provider="openai_compatible",
|
||||||
|
capability="chat",
|
||||||
|
endpoint_url="https://models.example.com/v1/chat/completions",
|
||||||
|
model_name="m",
|
||||||
|
enabled=True,
|
||||||
|
is_analysis_default=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_session_with_messages(db_session, rounds: int = 5) -> str:
|
||||||
|
CampaignRepository(db_session).create(_campaign())
|
||||||
|
repo = ExplorationSessionRepository(db_session)
|
||||||
|
session_obj = repo.create(
|
||||||
|
ExplorationSession(
|
||||||
|
campaign_id="camp-1",
|
||||||
|
target_id="t-1",
|
||||||
|
persona={"name": "急性子用户"},
|
||||||
|
goal="查询账单并缴费",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
session_obj.status = ExplorationSessionStatus.COMPLETED
|
||||||
|
repo.update(session_obj)
|
||||||
|
|
||||||
|
message_repo = ExplorationMessageRepository(db_session)
|
||||||
|
for i in range(1, rounds + 1):
|
||||||
|
message_repo.save_message(
|
||||||
|
ExplorationMessage(session_id=session_obj.id, round_index=i, role="user", content=f"用户消息-{i}")
|
||||||
|
)
|
||||||
|
message_repo.save_message(
|
||||||
|
ExplorationMessage(session_id=session_obj.id, round_index=i, role="assistant", content=f"回复内容-{i}")
|
||||||
|
)
|
||||||
|
return session_obj.id
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def judge_env(db_session, monkeypatch):
|
||||||
|
from agenteval.exploration import judge as judge_module
|
||||||
|
|
||||||
|
monkeypatch.setattr(judge_module, "get_session", lambda: db_session)
|
||||||
|
return db_session
|
||||||
|
|
||||||
|
|
||||||
|
# ── 抽样 ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_sample_round_indexes_returns_all_within_cap():
|
||||||
|
assert sample_round_indexes([1, 2]) == [1, 2]
|
||||||
|
assert sample_round_indexes([1, 2, 3]) == [1, 2, 3]
|
||||||
|
|
||||||
|
|
||||||
|
def test_sample_round_indexes_spreads_evenly_over_cap():
|
||||||
|
assert sample_round_indexes([1, 2, 3, 4, 5]) == [1, 3, 5]
|
||||||
|
assert len(sample_round_indexes(list(range(1, 11)))) == MAX_JUDGE_SAMPLES
|
||||||
|
|
||||||
|
|
||||||
|
# ── 归一化 ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_judge_review_applies_whitelists():
|
||||||
|
raw = {
|
||||||
|
"dimensions": [
|
||||||
|
{"dimension": "attitude", "rating": "good", "comment": "ok"},
|
||||||
|
{"dimension": "神秘维度", "rating": "good", "comment": "drop me"},
|
||||||
|
{"dimension": "hallucination", "rating": "离谱", "comment": "bad rating"},
|
||||||
|
],
|
||||||
|
"summary": "结论",
|
||||||
|
}
|
||||||
|
review = normalize_judge_review(raw)
|
||||||
|
dims = {d["dimension"]: d for d in review["dimensions"]}
|
||||||
|
assert set(dims) == {"attitude", "hallucination"}
|
||||||
|
assert dims["attitude"]["rating"] == "good"
|
||||||
|
assert dims["hallucination"]["rating"] == "acceptable" # 非法档位归一
|
||||||
|
assert review["summary"] == "结论"
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_judge_review_rejects_non_list_dimensions():
|
||||||
|
review = normalize_judge_review({"dimensions": "不是列表", "summary": 123})
|
||||||
|
assert review["dimensions"] == []
|
||||||
|
assert review["summary"] == "123"
|
||||||
|
|
||||||
|
|
||||||
|
# ── 后台执行编排 ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
async def test_execute_persists_structured_review(judge_env):
|
||||||
|
session_id = _seed_session_with_messages(judge_env, rounds=5)
|
||||||
|
_seed_config(judge_env)
|
||||||
|
|
||||||
|
client = FakeChatClient(REVIEW_JSON)
|
||||||
|
await execute_judge_review(session_id, chat_client=client)
|
||||||
|
|
||||||
|
session_obj = ExplorationSessionRepository(judge_env).get(session_id)
|
||||||
|
assert session_obj.status == ExplorationSessionStatus.COMPLETED
|
||||||
|
review = session_obj.judge_review
|
||||||
|
assert review["status"] == "completed"
|
||||||
|
assert review["model_config_id"] == "mc-default"
|
||||||
|
assert review["sampled_rounds"] == [1, 3, 5]
|
||||||
|
assert len(review["dimensions"]) == 3
|
||||||
|
assert review["summary"] == "服务态度好但存在幻觉"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_execute_prompt_carries_only_sampled_rounds(judge_env):
|
||||||
|
session_id = _seed_session_with_messages(judge_env, rounds=5)
|
||||||
|
_seed_config(judge_env)
|
||||||
|
|
||||||
|
client = FakeChatClient(REVIEW_JSON)
|
||||||
|
await execute_judge_review(session_id, chat_client=client)
|
||||||
|
|
||||||
|
payload = json.dumps(client.calls[0], ensure_ascii=False)
|
||||||
|
assert "用户消息-1" in payload and "用户消息-3" in payload and "用户消息-5" in payload
|
||||||
|
assert "用户消息-2" not in payload and "用户消息-4" not in payload
|
||||||
|
|
||||||
|
|
||||||
|
async def test_execute_records_error_on_unparseable_output(judge_env):
|
||||||
|
session_id = _seed_session_with_messages(judge_env)
|
||||||
|
_seed_config(judge_env)
|
||||||
|
|
||||||
|
await execute_judge_review(session_id, chat_client=FakeChatClient("这不是 JSON"))
|
||||||
|
|
||||||
|
session_obj = ExplorationSessionRepository(judge_env).get(session_id)
|
||||||
|
assert session_obj.status == ExplorationSessionStatus.COMPLETED # 复核失败不阻塞会话
|
||||||
|
review = session_obj.judge_review
|
||||||
|
assert review["status"] == "failed"
|
||||||
|
assert review["error"]
|
||||||
|
assert review["model_config_id"] == "mc-default"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_execute_records_error_on_client_exception(judge_env):
|
||||||
|
session_id = _seed_session_with_messages(judge_env)
|
||||||
|
_seed_config(judge_env)
|
||||||
|
|
||||||
|
await execute_judge_review(session_id, chat_client=FakeChatClient(RuntimeError("模型网关超时")))
|
||||||
|
|
||||||
|
review = ExplorationSessionRepository(judge_env).get(session_id).judge_review
|
||||||
|
assert review["status"] == "failed"
|
||||||
|
assert "模型网关超时" in review["error"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_execute_skips_silently_without_model_config(judge_env):
|
||||||
|
session_id = _seed_session_with_messages(judge_env)
|
||||||
|
|
||||||
|
client = FakeChatClient(REVIEW_JSON)
|
||||||
|
await execute_judge_review(session_id, chat_client=client)
|
||||||
|
|
||||||
|
assert client.calls == []
|
||||||
|
assert ExplorationSessionRepository(judge_env).get(session_id).judge_review is None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_execute_skips_session_without_messages(judge_env):
|
||||||
|
CampaignRepository(judge_env).create(_campaign())
|
||||||
|
session_obj = ExplorationSessionRepository(judge_env).create(
|
||||||
|
ExplorationSession(campaign_id="camp-1", target_id="t-1", persona={}, goal="g")
|
||||||
|
)
|
||||||
|
_seed_config(judge_env)
|
||||||
|
|
||||||
|
client = FakeChatClient(REVIEW_JSON)
|
||||||
|
await execute_judge_review(session_obj.id, chat_client=client)
|
||||||
|
|
||||||
|
assert client.calls == []
|
||||||
|
assert ExplorationSessionRepository(judge_env).get(session_obj.id).judge_review is None
|
||||||
106
tests/unit/test_exploration_summary.py
Normal file
106
tests/unit/test_exploration_summary.py
Normal file
@ -0,0 +1,106 @@
|
|||||||
|
"""Exploration summary aggregation (v0.9 票据 05).
|
||||||
|
|
||||||
|
体验记录聚合成探索摘要:会话数、目标达成率、问题清单(blockers/misled
|
||||||
|
按出现次数降序)、judge 复核结论(已就位才纳入;findings 全量收各档发现,
|
||||||
|
poor 档排前)。缺则无痕:无会话返回 None。纯函数,不落库。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from agenteval.exploration.models import ExplorationSession, ExplorationSessionStatus
|
||||||
|
from agenteval.exploration.summary import summarize_exploration
|
||||||
|
|
||||||
|
|
||||||
|
def _session(experience=None, judge_review=None, status=ExplorationSessionStatus.COMPLETED) -> ExplorationSession:
|
||||||
|
return ExplorationSession(
|
||||||
|
campaign_id="c-1",
|
||||||
|
target_id="t-1",
|
||||||
|
persona={"name": "x"},
|
||||||
|
goal="查询账单",
|
||||||
|
status=status,
|
||||||
|
experience=experience,
|
||||||
|
judge_review=judge_review,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _experience(goal_achieved: bool, blockers=None, misled=None) -> dict:
|
||||||
|
return {
|
||||||
|
"goal_achieved": goal_achieved,
|
||||||
|
"blockers": blockers or [],
|
||||||
|
"misled": misled or [],
|
||||||
|
"emotion": "neutral",
|
||||||
|
"notes": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_sessions_returns_none():
|
||||||
|
assert summarize_exploration([]) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_aggregates_counts_rate_and_issue_lists():
|
||||||
|
sessions = [
|
||||||
|
_session(_experience(True, blockers=["缴费入口难找"])),
|
||||||
|
_session(_experience(False, blockers=["缴费入口难找", "验证码收不到"], misled=["被误导选了错误套餐"])),
|
||||||
|
_session(_experience(False, blockers=["验证码收不到"])),
|
||||||
|
]
|
||||||
|
summary = summarize_exploration(sessions)
|
||||||
|
assert summary["session_count"] == 3
|
||||||
|
assert summary["sessions_with_experience"] == 3
|
||||||
|
assert summary["goal_achieved_count"] == 1
|
||||||
|
assert summary["goal_achievement_rate"] == round(1 / 3, 4)
|
||||||
|
# 问题清单按出现次数降序(同频按首次出现顺序)
|
||||||
|
assert summary["issues"] == [
|
||||||
|
{"issue": "缴费入口难找", "count": 2},
|
||||||
|
{"issue": "验证码收不到", "count": 2},
|
||||||
|
]
|
||||||
|
assert summary["misled"] == [{"issue": "被误导选了错误套餐", "count": 1}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_sessions_without_experience_excluded_from_rate_denominator():
|
||||||
|
sessions = [
|
||||||
|
_session(_experience(True)),
|
||||||
|
_session(), # 未关闭/无体验记录
|
||||||
|
]
|
||||||
|
summary = summarize_exploration(sessions)
|
||||||
|
assert summary["session_count"] == 2
|
||||||
|
assert summary["sessions_with_experience"] == 1
|
||||||
|
assert summary["goal_achievement_rate"] == 1.0
|
||||||
|
assert summary["issues"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_experience_at_all_rate_is_none():
|
||||||
|
summary = summarize_exploration([_session(), _session()])
|
||||||
|
assert summary["goal_achievement_rate"] is None
|
||||||
|
assert summary["issues"] == [] and summary["misled"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_judge_review_included_only_when_completed():
|
||||||
|
good_review = {
|
||||||
|
"status": "completed",
|
||||||
|
"dimensions": [
|
||||||
|
{"dimension": "attitude", "rating": "good", "comment": "友好"},
|
||||||
|
{"dimension": "professionalism", "rating": "acceptable", "comment": "基本准确"},
|
||||||
|
{"dimension": "hallucination", "rating": "poor", "comment": "编造了政策"},
|
||||||
|
],
|
||||||
|
"summary": "存在幻觉",
|
||||||
|
}
|
||||||
|
failed_review = {"status": "failed", "error": "解析失败"}
|
||||||
|
sessions = [
|
||||||
|
_session(_experience(True), judge_review=good_review),
|
||||||
|
_session(_experience(False), judge_review=failed_review),
|
||||||
|
_session(_experience(True)), # 未复核
|
||||||
|
]
|
||||||
|
summary = summarize_exploration(sessions)
|
||||||
|
judge = summary["judge_review"]
|
||||||
|
assert judge["reviewed_sessions"] == 1
|
||||||
|
# 复核发现全量纳入,poor 档排前(票据 05:复核结论一并纳入)
|
||||||
|
assert judge["findings"] == [
|
||||||
|
{"dimension": "hallucination", "rating": "poor", "comment": "编造了政策"},
|
||||||
|
{"dimension": "professionalism", "rating": "acceptable", "comment": "基本准确"},
|
||||||
|
{"dimension": "attitude", "rating": "good", "comment": "友好"},
|
||||||
|
]
|
||||||
|
# 复核总体结论一并纳入
|
||||||
|
assert judge["summaries"] == ["存在幻觉"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_judge_review_absent_when_none_reviewed():
|
||||||
|
summary = summarize_exploration([_session(_experience(True))])
|
||||||
|
assert summary["judge_review"] is None
|
||||||
@ -346,3 +346,70 @@ def test_render_campaign_markdown_comparison_comes_after_analysis():
|
|||||||
def test_render_campaign_markdown_without_comparison_unchanged():
|
def test_render_campaign_markdown_without_comparison_unchanged():
|
||||||
md = render_campaign_markdown(_campaign_report())
|
md = render_campaign_markdown(_campaign_report())
|
||||||
assert "周期对比" not in md
|
assert "周期对比" not in md
|
||||||
|
|
||||||
|
|
||||||
|
# ── render_campaign_markdown: 探索发现附录(v0.9 票据 05)────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _exploration() -> dict:
|
||||||
|
return {
|
||||||
|
"session_count": 3,
|
||||||
|
"sessions_with_experience": 3,
|
||||||
|
"goal_achieved_count": 1,
|
||||||
|
"goal_achievement_rate": 0.3333,
|
||||||
|
"issues": [
|
||||||
|
{"issue": "缴费入口难找", "count": 2},
|
||||||
|
{"issue": "验证码收不到", "count": 1},
|
||||||
|
],
|
||||||
|
"misled": [{"issue": "被误导选了错误套餐", "count": 1}],
|
||||||
|
"judge_review": {
|
||||||
|
"reviewed_sessions": 1,
|
||||||
|
"findings": [{"dimension": "hallucination", "rating": "poor", "comment": "编造了不存在的政策"}],
|
||||||
|
"summaries": ["服务态度好但存在幻觉"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_campaign_markdown_appends_exploration_appendix():
|
||||||
|
md = render_campaign_markdown(_campaign_report(), exploration=_exploration())
|
||||||
|
assert "## 探索发现" in md
|
||||||
|
assert "| 探索会话数 | 3 |" in md
|
||||||
|
assert "| 目标达成率 | 33.3% |" in md
|
||||||
|
assert "缴费入口难找 ×2" in md
|
||||||
|
assert "验证码收不到 ×1" in md
|
||||||
|
assert "被误导选了错误套餐 ×1" in md
|
||||||
|
assert "编造了不存在的政策" in md
|
||||||
|
assert "复核结论:服务态度好但存在幻觉" in md
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_campaign_markdown_exploration_without_judge_or_issues():
|
||||||
|
md = render_campaign_markdown(
|
||||||
|
_campaign_report(),
|
||||||
|
exploration={
|
||||||
|
"session_count": 1,
|
||||||
|
"sessions_with_experience": 1,
|
||||||
|
"goal_achieved_count": 1,
|
||||||
|
"goal_achievement_rate": 1.0,
|
||||||
|
"issues": [],
|
||||||
|
"misled": [],
|
||||||
|
"judge_review": None,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert "## 探索发现" in md
|
||||||
|
assert "无" in md # 空问题清单回落文案
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_campaign_markdown_exploration_comes_after_comparison():
|
||||||
|
md = render_campaign_markdown(
|
||||||
|
_campaign_report(), analysis=_analysis(), comparison=_comparison(), exploration=_exploration()
|
||||||
|
)
|
||||||
|
assert md.index("## 智能分析") < md.index("## 周期对比") < md.index("## 探索发现")
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_campaign_markdown_without_exploration_byte_identical():
|
||||||
|
base = render_campaign_markdown(_campaign_report(), analysis=_analysis(), comparison=_comparison())
|
||||||
|
with_none = render_campaign_markdown(
|
||||||
|
_campaign_report(), analysis=_analysis(), comparison=_comparison(), exploration=None
|
||||||
|
)
|
||||||
|
assert base == with_none
|
||||||
|
assert "探索发现" not in base
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user